"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "src/parallel_alternatives.texi" between
parallel-20210122.tar.bz2 and parallel-20210222.tar.bz2

About: GNU Parallel is a shell tool for executing jobs in parallel using multiple CPU cores and/or multiple computers.

parallel_alternatives.texi  (parallel-20210122.tar.bz2):parallel_alternatives.texi  (parallel-20210222.tar.bz2)
skipping to change at line 96 skipping to change at line 96
* DIFFERENCES BETWEEN Tollef's parallel (moreutils) AND GNU Parallel:: * DIFFERENCES BETWEEN Tollef's parallel (moreutils) AND GNU Parallel::
* DIFFERENCES BETWEEN rargs AND GNU Parallel:: * DIFFERENCES BETWEEN rargs AND GNU Parallel::
* DIFFERENCES BETWEEN threader AND GNU Parallel:: * DIFFERENCES BETWEEN threader AND GNU Parallel::
* DIFFERENCES BETWEEN runp AND GNU Parallel:: * DIFFERENCES BETWEEN runp AND GNU Parallel::
* DIFFERENCES BETWEEN papply AND GNU Parallel:: * DIFFERENCES BETWEEN papply AND GNU Parallel::
* DIFFERENCES BETWEEN async AND GNU Parallel:: * DIFFERENCES BETWEEN async AND GNU Parallel::
* DIFFERENCES BETWEEN pardi AND GNU Parallel:: * DIFFERENCES BETWEEN pardi AND GNU Parallel::
* DIFFERENCES BETWEEN bthread AND GNU Parallel:: * DIFFERENCES BETWEEN bthread AND GNU Parallel::
* DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel:: * DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel::
* DIFFERENCES BETWEEN parasweep AND GNU Parallel:: * DIFFERENCES BETWEEN parasweep AND GNU Parallel::
* DIFFERENCES BETWEEN parallel-bash AND GNU Parallel::
* DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel::
* Todo:: * Todo::
@end menu @end menu
@node SUMMARY LEGEND @node SUMMARY LEGEND
@section SUMMARY LEGEND @section SUMMARY LEGEND
The following features are in some of the comparable tools: The following features are in some of the comparable tools:
@strong{Inputs} @strong{Inputs}
I1. Arguments can be read from stdin I1. Arguments can be read from stdin
skipping to change at line 3288 skipping to change at line 3290
commands like @strong{sh -c "echo a; echo b"}. commands like @strong{sh -c "echo a; echo b"}.
https://gitlab.com/netikras/bthread (Last checked: 2021-01) https://gitlab.com/netikras/bthread (Last checked: 2021-01)
@node DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel @node DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel
@section DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel @section DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel
Summary (see legend above): Summary (see legend above):
I1 - - - - - I7 I1 - - - - - I7
M1 - - - - M6 M1 - - - - M6
- O2 O3 - - O6 - N/A N/A O10 - O2 O3 - - O6 - x x O10
E1 - - - - - - E1 - - - - - -
- - - - - - - - - - - - - - - - - -
- - - -
@menu @menu
* EXAMPLES FROM simple_gpu_scheduler MANUAL:: * EXAMPLES FROM simple_gpu_scheduler MANUAL::
@end menu @end menu
@node EXAMPLES FROM simple_gpu_scheduler MANUAL @node EXAMPLES FROM simple_gpu_scheduler MANUAL
@subsection EXAMPLES FROM simple_gpu_scheduler MANUAL @subsection EXAMPLES FROM simple_gpu_scheduler MANUAL
skipping to change at line 3381 skipping to change at line 3383
format to get something similar, but you would have to read the format to get something similar, but you would have to read the
output. output.
@strong{parasweep} has a filtering method to ignore parameter combinations @strong{parasweep} has a filtering method to ignore parameter combinations
you do not need. you do not need.
Instead of calling the jobs directly, @strong{parasweep} can use Python's Instead of calling the jobs directly, @strong{parasweep} can use Python's
Distributed Resource Management Application API to make jobs run with Distributed Resource Management Application API to make jobs run with
different cluster software. different cluster software.
@menu GNU @strong{parallel} @strong{--tmpl} supports templates with replacement
* Future support in GNU Parallel:: strings. Such as:
@end menu
@node Future support in GNU Parallel
@subsection Future support in GNU Parallel
@strong{Future} versions of GNU @strong{parallel} may support templates
with replacement strings. Such as:
@verbatim @verbatim
Xval: {x} Xval: {x}
Yval: {y} Yval: {y}
FixedValue: 9 FixedValue: 9
# x with 2 decimals # x with 2 decimals
DecimalX: {=x $_=sprintf("%.2f",$_) =} DecimalX: {=x $_=sprintf("%.2f",$_) =}
TenX: {=x $_=$_*10 =} TenX: {=x $_=$_*10 =}
RandomVal: {= $_=rand() =} RandomVal: {=1 $_=rand() =}
@end verbatim @end verbatim
that can be used like: that can be used like:
@verbatim @verbatim
parallel --header : --tmpl my.tmpl {#}.t myprog {#}.t \ parallel --header : --tmpl my.tmpl={#}.t myprog {#}.t \
::: x 1 2 3 ::: y 1 2 3 ::: x 1 2 3 ::: y 1 2 3
@end verbatim @end verbatim
Filtering may also be supported as: Filtering is supported as:
@verbatim @verbatim
parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3 parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3
@end verbatim @end verbatim
which will basically do: https://github.com/eviatarbach/parasweep (Last checked: 2021-01)
@node DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
@section DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
Summary (see legend above):
I1 I2 - - - - -
- - M3 - - M6
- O2 O3 - O5 O6 - O8 x O10
E1 - - - - - -
- - - - - - - - -
- -
@strong{parallel-bash} is written in pure bash. It is really fast (overhead
of ~0.05 ms/job compared to GNU @strong{parallel}'s ~3 ms/job). So if your
jobs are extremely short lived, and you can live with the quite
limited command, this may be useful.
@strong{parallel-bash} will not start the first job, until it has read all
input. The input can at most be 20935 lines and the lines cannot be
all be empty.
Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend
running jobs.
@menu
* EXAMPLES FROM parallel-bash::
@end menu
@node EXAMPLES FROM parallel-bash
@subsection EXAMPLES FROM parallel-bash
@verbatim @verbatim
parallel echo '{= $arg[1] > $arg[2] and skip() =}' ::: 1 2 3 ::: 1 2 3 1$ some_input | parallel-bash -p 5 -c echo
1$ some_input | parallel -j 5 echo
2$ parallel-bash -p 5 -c echo < some_file
2$ parallel -j 5 echo < some_file
3$ parallel-bash -p 5 -c echo <<< 'some string'
3$ parallel -j 5 -c echo <<< 'some string'
4$ something | parallel-bash -p 5 -c echo {} {}
4$ something | parallel -j 5 echo {} {}
@end verbatim @end verbatim
https://github.com/eviatarbach/parasweep (Last checked: 2021-01) https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html
(Last checked: 2021-02)
@node DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
@section DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
@strong{bash-concurrent} is more an alternative to @strong{make} than to GNU
@strong{parallel}. Its input is very similar to a Makefile, where jobs
depend on other jobs.
It has a nice progress indicator where you can see which jobs
completed successfully, which jobs are currently running, which jobs
failed, and which jobs were skipped due to a depending job failed.
The indicator does not deal well with resizing the window.
Output is cached in tempfiles on disk, but is only shown if there is
an error, so it is not meant to be part of a UNIX pipeline. If
@strong{bash-concurrent} crashes these tempfiles are not removed.
It uses an O(n*n) algorithm, so if you have 1000 independent jobs it
takes 22 seconds to start it.
https://github.com/themattrix/bash-concurrent
(Last checked: 2021-02)
@node Todo @node Todo
@section Todo @section Todo
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html
https://github.com/Nukesor/pueue https://github.com/Nukesor/pueue
PASH: Light-touch Data-Parallel Shell Processing
https://arxiv.org/pdf/2012.15443.pdf KumQuat https://arxiv.org/pdf/2012.15443.pdf KumQuat
https://arxiv.org/pdf/2007.09436.pdf https://arxiv.org/pdf/2007.09436.pdf PaSH: Light-touch Data-Parallel Shell Proce ssing
https://github.com/JeiKeiLim/simple_distribute_job https://github.com/JeiKeiLim/simple_distribute_job
https://github.com/Akianonymus/parallel-bash https://github.com/reggi/pkgrun - not obvious how to use
https://github.com/reggi/pkgrun
https://github.com/benoror/better-npm-run - not obvious how to use https://github.com/benoror/better-npm-run - not obvious how to use
https://github.com/bahmutov/with-package https://github.com/bahmutov/with-package
https://github.com/xuchenCN/go-pssh https://github.com/xuchenCN/go-pssh
https://github.com/flesler/parallel https://github.com/flesler/parallel
https://github.com/Julian/Verge https://github.com/Julian/Verge
http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
http://vicerveza.homeunix.net/~viric/soft/ts/
@node TESTING OTHER TOOLS @node TESTING OTHER TOOLS
@chapter TESTING OTHER TOOLS @chapter TESTING OTHER TOOLS
There are certain issues that are very common on parallelizing There are certain issues that are very common on parallelizing
tools. Here are a few stress tests. Be warned: If the tool is badly tools. Here are a few stress tests. Be warned: If the tool is badly
coded it may overload your machine. coded it may overload your machine.
@menu @menu
* MIX@asis{:} Output mixes:: * MIX@asis{:} Output mixes::
* STDERRMERGE@asis{:} Stderr is merged with stdout:: * STDERRMERGE@asis{:} Stderr is merged with stdout::
 End of changes. 14 change blocks. 
24 lines changed or deleted 84 lines changed or added

Home  |  About  |  Features  |  All  |  Newest  |  Dox  |  Diffs  |  RSS Feeds  |  Screenshots  |  Comments  |  Imprint  |  Privacy  |  HTTP(S)