"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "src/parallel_alternatives.pod" between
parallel-20210122.tar.bz2 and parallel-20210222.tar.bz2

About: GNU Parallel is a shell tool for executing jobs in parallel using multiple CPU cores and/or multiple computers.

parallel_alternatives.pod  (parallel-20210122.tar.bz2):parallel_alternatives.pod  (parallel-20210222.tar.bz2)
skipping to change at line 2719 skipping to change at line 2719
You cannot quote space in the command, so you cannot run composed You cannot quote space in the command, so you cannot run composed
commands like B<sh -c "echo a; echo b">. commands like B<sh -c "echo a; echo b">.
https://gitlab.com/netikras/bthread (Last checked: 2021-01) https://gitlab.com/netikras/bthread (Last checked: 2021-01)
=head2 DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel =head2 DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel
Summary (see legend above): Summary (see legend above):
I1 - - - - - I7 I1 - - - - - I7
M1 - - - - M6 M1 - - - - M6
- O2 O3 - - O6 - N/A N/A O10 - O2 O3 - - O6 - x x O10
E1 - - - - - - E1 - - - - - -
- - - - - - - - - - - - - - - - - -
- - - -
=head3 EXAMPLES FROM simple_gpu_scheduler MANUAL =head3 EXAMPLES FROM simple_gpu_scheduler MANUAL
1$ simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt 1$ simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt
1$ parallel -j3 --shuf \ 1$ parallel -j3 --shuf \
CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt
skipping to change at line 2802 skipping to change at line 2802
format to get something similar, but you would have to read the format to get something similar, but you would have to read the
output. output.
B<parasweep> has a filtering method to ignore parameter combinations B<parasweep> has a filtering method to ignore parameter combinations
you do not need. you do not need.
Instead of calling the jobs directly, B<parasweep> can use Python's Instead of calling the jobs directly, B<parasweep> can use Python's
Distributed Resource Management Application API to make jobs run with Distributed Resource Management Application API to make jobs run with
different cluster software. different cluster software.
=head3 Future support in GNU Parallel GNU B<parallel> B<--tmpl> supports templates with replacement
strings. Such as:
B<Future> versions of GNU B<parallel> may support templates
with replacement strings. Such as:
Xval: {x} Xval: {x}
Yval: {y} Yval: {y}
FixedValue: 9 FixedValue: 9
# x with 2 decimals # x with 2 decimals
DecimalX: {=x $_=sprintf("%.2f",$_) =} DecimalX: {=x $_=sprintf("%.2f",$_) =}
TenX: {=x $_=$_*10 =} TenX: {=x $_=$_*10 =}
RandomVal: {= $_=rand() =} RandomVal: {=1 $_=rand() =}
that can be used like: that can be used like:
parallel --header : --tmpl my.tmpl {#}.t myprog {#}.t \ parallel --header : --tmpl my.tmpl={#}.t myprog {#}.t \
::: x 1 2 3 ::: y 1 2 3 ::: x 1 2 3 ::: y 1 2 3
Filtering may also be supported as: Filtering is supported as:
parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3 parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3
which will basically do: https://github.com/eviatarbach/parasweep (Last checked: 2021-01)
parallel echo '{= $arg[1] > $arg[2] and skip() =}' ::: 1 2 3 ::: 1 2 3 =head2 DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
https://github.com/eviatarbach/parasweep (Last checked: 2021-01) Summary (see legend above):
I1 I2 - - - - -
- - M3 - - M6
- O2 O3 - O5 O6 - O8 x O10
E1 - - - - - -
- - - - - - - - -
- -
=head2 Todo B<parallel-bash> is written in pure bash. It is really fast (overhead
of ~0.05 ms/job compared to GNU B<parallel>'s ~3 ms/job). So if your
jobs are extremely short lived, and you can live with the quite
limited command, this may be useful.
B<parallel-bash> will not start the first job, until it has read all
input. The input can at most be 20935 lines and the lines cannot be
all be empty.
Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend
running jobs.
=head3 EXAMPLES FROM parallel-bash
1$ some_input | parallel-bash -p 5 -c echo
1$ some_input | parallel -j 5 echo
2$ parallel-bash -p 5 -c echo < some_file
2$ parallel -j 5 echo < some_file
3$ parallel-bash -p 5 -c echo <<< 'some string'
3$ parallel -j 5 -c echo <<< 'some string'
4$ something | parallel-bash -p 5 -c echo {} {}
4$ something | parallel -j 5 echo {} {}
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html
(Last checked: 2021-02)
https://github.com/Nukesor/pueue =head2 DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
B<bash-concurrent> is more an alternative to B<make> than to GNU
B<parallel>. Its input is very similar to a Makefile, where jobs
depend on other jobs.
It has a nice progress indicator where you can see which jobs
completed successfully, which jobs are currently running, which jobs
failed, and which jobs were skipped due to a depending job failed.
The indicator does not deal well with resizing the window.
Output is cached in tempfiles on disk, but is only shown if there is
an error, so it is not meant to be part of a UNIX pipeline. If
B<bash-concurrent> crashes these tempfiles are not removed.
It uses an O(n*n) algorithm, so if you have 1000 independent jobs it
takes 22 seconds to start it.
PASH: Light-touch Data-Parallel Shell Processing https://github.com/themattrix/bash-concurrent
(Last checked: 2021-02)
=head2 Todo
https://github.com/Nukesor/pueue
https://arxiv.org/pdf/2012.15443.pdf KumQuat https://arxiv.org/pdf/2012.15443.pdf KumQuat
https://arxiv.org/pdf/2007.09436.pdf https://arxiv.org/pdf/2007.09436.pdf PaSH: Light-touch Data-Parallel Shell Proce ssing
https://github.com/JeiKeiLim/simple_distribute_job https://github.com/JeiKeiLim/simple_distribute_job
https://github.com/Akianonymus/parallel-bash https://github.com/reggi/pkgrun - not obvious how to use
https://github.com/reggi/pkgrun
https://github.com/benoror/better-npm-run - not obvious how to use https://github.com/benoror/better-npm-run - not obvious how to use
https://github.com/bahmutov/with-package https://github.com/bahmutov/with-package
https://github.com/xuchenCN/go-pssh https://github.com/xuchenCN/go-pssh
https://github.com/flesler/parallel https://github.com/flesler/parallel
https://github.com/Julian/Verge https://github.com/Julian/Verge
http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
http://vicerveza.homeunix.net/~viric/soft/ts/
=head1 TESTING OTHER TOOLS =head1 TESTING OTHER TOOLS
There are certain issues that are very common on parallelizing There are certain issues that are very common on parallelizing
tools. Here are a few stress tests. Be warned: If the tool is badly tools. Here are a few stress tests. Be warned: If the tool is badly
coded it may overload your machine. coded it may overload your machine.
=head2 MIX: Output mixes =head2 MIX: Output mixes
Output from 2 jobs should not mix. If the output is not used, this Output from 2 jobs should not mix. If the output is not used, this
does not matter; but if the output I<is> used then it is important does not matter; but if the output I<is> used then it is important
 End of changes. 15 change blocks. 
18 lines changed or deleted 74 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)