"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "src/parallel_alternatives.7" between
parallel-20210122.tar.bz2 and parallel-20210222.tar.bz2

About: GNU Parallel is a shell tool for executing jobs in parallel using multiple CPU cores and/or multiple computers.

parallel_alternatives.7  (parallel-20210122.tar.bz2):parallel_alternatives.7  (parallel-20210222.tar.bz2)
skipping to change at line 2433 skipping to change at line 2433
Summary (see legend above): - - - I4 - - - - - - - - M6 O1 - O3 - - - O7 O8 - - E1 - - - - - - - - - - - Summary (see legend above): - - - I4 - - - - - - - - M6 O1 - O3 - - - O7 O8 - - E1 - - - - - - - - - - -
- - - - - - - - - - - -
bthread takes around 1 sec per MB of output. The maximal output line leng th is 1073741759. bthread takes around 1 sec per MB of output. The maximal output line leng th is 1073741759.
You cannot quote space in the command, so you cannot run composed command s like sh -c "echo a; echo b". You cannot quote space in the command, so you cannot run composed command s like sh -c "echo a; echo b".
https://gitlab.com/netikras/bthread (Last checked: 2021-01) https://gitlab.com/netikras/bthread (Last checked: 2021-01)
DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel
Summary (see legend above): I1 - - - - - I7 M1 - - - - M6 - O2 O3 - - O6 Summary (see legend above): I1 - - - - - I7 M1 - - - - M6 - O2 O3 - - O6
- N/A N/A O10 E1 - - - - - - - - - x x O10 E1 - - - - - - - - - -
- - - - - - - - - - - - - - - -
EXAMPLES FROM simple_gpu_scheduler MANUAL EXAMPLES FROM simple_gpu_scheduler MANUAL
1$ simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt 1$ simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt
1$ parallel -j3 --shuf \ 1$ parallel -j3 --shuf \
CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt
2$ simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs} " \ 2$ simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs} " \
-p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 |
skipping to change at line 2505 skipping to change at line 2505
Being a Python module parasweep integrates tighter with Python than GNU p arallel. You get the parameters Being a Python module parasweep integrates tighter with Python than GNU p arallel. You get the parameters
directly in a Python data structure. With GNU parallel you can use the JS ON or CSV output format to get directly in a Python data structure. With GNU parallel you can use the JS ON or CSV output format to get
something similar, but you would have to read the output. something similar, but you would have to read the output.
parasweep has a filtering method to ignore parameter combinations you do not need. parasweep has a filtering method to ignore parameter combinations you do not need.
Instead of calling the jobs directly, parasweep can use Python's Distribu ted Resource Management Instead of calling the jobs directly, parasweep can use Python's Distribu ted Resource Management
Application API to make jobs run with different cluster software. Application API to make jobs run with different cluster software.
Future support in GNU Parallel GNU parallel --tmpl supports templates with replacement strings. Such as:
Future versions of GNU parallel may support templates with replacement st
rings. Such as:
Xval: {x} Xval: {x}
Yval: {y} Yval: {y}
FixedValue: 9 FixedValue: 9
# x with 2 decimals # x with 2 decimals
DecimalX: {=x $_=sprintf("%.2f",$_) =} DecimalX: {=x $_=sprintf("%.2f",$_) =}
TenX: {=x $_=$_*10 =} TenX: {=x $_=$_*10 =}
RandomVal: {= $_=rand() =} RandomVal: {=1 $_=rand() =}
that can be used like: that can be used like:
parallel --header : --tmpl my.tmpl {#}.t myprog {#}.t \ parallel --header : --tmpl my.tmpl={#}.t myprog {#}.t \
::: x 1 2 3 ::: y 1 2 3 ::: x 1 2 3 ::: y 1 2 3
Filtering may also be supported as: Filtering is supported as:
parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3 parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3
which will basically do: https://github.com/eviatarbach/parasweep (Last checked: 2021-01)
parallel echo '{= $arg[1] > $arg[2] and skip() =}' ::: 1 2 3 ::: 1 2 3 DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
Summary (see legend above): I1 I2 - - - - - - - M3 - - M6 - O2 O3 - O5 O6
- O8 x O10 E1 - - - - - - - - -
- - - - - - - -
https://github.com/eviatarbach/parasweep (Last checked: 2021-01) parallel-bash is written in pure bash. It is really fast (overhead of ~0.
05 ms/job compared to GNU
parallel's ~3 ms/job). So if your jobs are extremely short lived, and you
can live with the quite limited
command, this may be useful.
Todo parallel-bash will not start the first job, until it has read all input.
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash. The input can at most be 20935
html lines and the lines cannot be all be empty.
https://github.com/Nukesor/pueue Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend running j
obs.
EXAMPLES FROM parallel-bash
1$ some_input | parallel-bash -p 5 -c echo
1$ some_input | parallel -j 5 echo
2$ parallel-bash -p 5 -c echo < some_file
2$ parallel -j 5 echo < some_file
3$ parallel-bash -p 5 -c echo <<< 'some string'
3$ parallel -j 5 -c echo <<< 'some string'
4$ something | parallel-bash -p 5 -c echo {} {}
PASH: Light-touch Data-Parallel Shell Processing 4$ something | parallel -j 5 echo {} {}
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.
html (Last checked: 2021-02)
DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
bash-concurrent is more an alternative to make than to GNU parallel. Its
input is very similar to a
Makefile, where jobs depend on other jobs.
It has a nice progress indicator where you can see which jobs completed s
uccessfully, which jobs are
currently running, which jobs failed, and which jobs were skipped due to
a depending job failed. The
indicator does not deal well with resizing the window.
Output is cached in tempfiles on disk, but is only shown if there is an e
rror, so it is not meant to be
part of a UNIX pipeline. If bash-concurrent crashes these tempfiles are n
ot removed.
It uses an O(n*n) algorithm, so if you have 1000 independent jobs it take
s 22 seconds to start it.
https://github.com/themattrix/bash-concurrent (Last checked: 2021-02)
Todo
https://github.com/Nukesor/pueue
https://arxiv.org/pdf/2012.15443.pdf KumQuat https://arxiv.org/pdf/2012.15443.pdf KumQuat
https://arxiv.org/pdf/2007.09436.pdf https://arxiv.org/pdf/2007.09436.pdf PaSH: Light-touch Data-Parallel Shel l Processing
https://github.com/JeiKeiLim/simple_distribute_job https://github.com/JeiKeiLim/simple_distribute_job
https://github.com/Akianonymus/parallel-bash https://github.com/reggi/pkgrun - not obvious how to use
https://github.com/reggi/pkgrun
https://github.com/benoror/better-npm-run - not obvious how to use https://github.com/benoror/better-npm-run - not obvious how to use
https://github.com/bahmutov/with-package https://github.com/bahmutov/with-package
https://github.com/xuchenCN/go-pssh https://github.com/xuchenCN/go-pssh
https://github.com/flesler/parallel https://github.com/flesler/parallel
https://github.com/Julian/Verge https://github.com/Julian/Verge
http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
http://vicerveza.homeunix.net/~viric/soft/ts/
TESTING OTHER TOOLS TESTING OTHER TOOLS
There are certain issues that are very common on parallelizing tools. Her e are a few stress tests. Be There are certain issues that are very common on parallelizing tools. Her e are a few stress tests. Be
warned: If the tool is badly coded it may overload your machine. warned: If the tool is badly coded it may overload your machine.
MIX: Output mixes MIX: Output mixes
Output from 2 jobs should not mix. If the output is not used, this does n ot matter; but if the output is Output from 2 jobs should not mix. If the output is not used, this does n ot matter; but if the output is
used then it is important that you do not get half a line from one job fo llowed by half a line from used then it is important that you do not get half a line from one job fo llowed by half a line from
another job. another job.
If the tool does not buffer, output will most likely mix now and then. If the tool does not buffer, output will most likely mix now and then.
skipping to change at line 2841 skipping to change at line 2881
A copy of the full license is included in the file as cc-by-sa.txt. A copy of the full license is included in the file as cc-by-sa.txt.
DEPENDENCIES DEPENDENCIES
GNU parallel uses Perl, and the Perl modules Getopt::Long, IPC::Open3, Sy mbol, IO::File, POSIX, and GNU parallel uses Perl, and the Perl modules Getopt::Long, IPC::Open3, Sy mbol, IO::File, POSIX, and
File::Temp. For remote usage it also uses rsync with ssh. File::Temp. For remote usage it also uses rsync with ssh.
SEE ALSO SEE ALSO
find(1), xargs(1), make(1), pexec(1), ppss(1), xjobs(1), prll(1), dxargs( 1), mdm(1) find(1), xargs(1), make(1), pexec(1), ppss(1), xjobs(1), prll(1), dxargs( 1), mdm(1)
20201222 2021-01-19 PARALLEL_ALTERNATIVES(7) 20210122 2021-02-20 PARALLEL_ALTERNATIVES(7)
 End of changes. 15 change blocks. 
22 lines changed or deleted 72 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)