"Fossies" - the Fresh Open Source Software Archive  

Source code changes of the file "src/parallel_alternatives.7" between
parallel-20210122.tar.bz2 and parallel-20210222.tar.bz2

About: GNU Parallel is a shell tool for executing jobs in parallel using multiple CPU cores and/or multiple computers.

parallel_alternatives.7  (parallel-20210122.tar.bz2)	:	parallel_alternatives.7  (parallel-20210222.tar.bz2)
skipping to change at line 2433		skipping to change at line 2433
Summary (see legend above): - - - I4 - - - - - - - - M6 O1 - O3 - - - O7 O8 - - E1 - - - - - - - - - - -		Summary (see legend above): - - - I4 - - - - - - - - M6 O1 - O3 - - - O7 O8 - - E1 - - - - - - - - - - -
- - - - - -		- - - - - -
bthread takes around 1 sec per MB of output. The maximal output line leng th is 1073741759.		bthread takes around 1 sec per MB of output. The maximal output line leng th is 1073741759.
You cannot quote space in the command, so you cannot run composed command s like sh -c "echo a; echo b".		You cannot quote space in the command, so you cannot run composed command s like sh -c "echo a; echo b".
https://gitlab.com/netikras/bthread (Last checked: 2021-01)		https://gitlab.com/netikras/bthread (Last checked: 2021-01)
DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel		DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel
Summary (see legend above): I1 - - - - - I7 M1 - - - - M6 - O2 O3 - - O6		Summary (see legend above): I1 - - - - - I7 M1 - - - - M6 - O2 O3 - - O6
- N/A N/A O10 E1 - - - - - - - -		- x x O10 E1 - - - - - - - - - -
- - - - - - - - -		- - - - - - -
EXAMPLES FROM simple_gpu_scheduler MANUAL		EXAMPLES FROM simple_gpu_scheduler MANUAL
1$ simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt		1$ simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt
1$ parallel -j3 --shuf \		1$ parallel -j3 --shuf \
CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt		CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt
2$ simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs} " \		2$ simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs} " \
-p lr 0.001 0.0005 0.0001 -p bs 32 64 128 |		-p lr 0.001 0.0005 0.0001 -p bs 32 64 128 |
skipping to change at line 2505		skipping to change at line 2505
Being a Python module parasweep integrates tighter with Python than GNU p arallel. You get the parameters		Being a Python module parasweep integrates tighter with Python than GNU p arallel. You get the parameters
directly in a Python data structure. With GNU parallel you can use the JS ON or CSV output format to get		directly in a Python data structure. With GNU parallel you can use the JS ON or CSV output format to get
something similar, but you would have to read the output.		something similar, but you would have to read the output.
parasweep has a filtering method to ignore parameter combinations you do not need.		parasweep has a filtering method to ignore parameter combinations you do not need.
Instead of calling the jobs directly, parasweep can use Python's Distribu ted Resource Management		Instead of calling the jobs directly, parasweep can use Python's Distribu ted Resource Management
Application API to make jobs run with different cluster software.		Application API to make jobs run with different cluster software.
Future support in GNU Parallel		GNU parallel --tmpl supports templates with replacement strings. Such as:
Future versions of GNU parallel may support templates with replacement st
rings. Such as:
Xval: {x}		Xval: {x}
Yval: {y}		Yval: {y}
FixedValue: 9		FixedValue: 9
# x with 2 decimals		# x with 2 decimals
DecimalX: {=x $_=sprintf("%.2f",$_) =}		DecimalX: {=x $_=sprintf("%.2f",$_) =}
TenX: {=x $_=$_*10 =}		TenX: {=x $_=$_*10 =}
RandomVal: {= $_=rand() =}		RandomVal: {=1 $_=rand() =}
that can be used like:		that can be used like:
parallel --header : --tmpl my.tmpl {#}.t myprog {#}.t \		parallel --header : --tmpl my.tmpl={#}.t myprog {#}.t \
::: x 1 2 3 ::: y 1 2 3		::: x 1 2 3 ::: y 1 2 3
Filtering may also be supported as:		Filtering is supported as:
parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3		parallel --filter '{1} > {2}' echo ::: 1 2 3 ::: 1 2 3
which will basically do:		https://github.com/eviatarbach/parasweep (Last checked: 2021-01)
parallel echo '{= $arg[1] > $arg[2] and skip() =}' ::: 1 2 3 ::: 1 2 3		DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
		Summary (see legend above): I1 I2 - - - - - - - M3 - - M6 - O2 O3 - O5 O6
		- O8 x O10 E1 - - - - - - - - -
		- - - - - - - -
https://github.com/eviatarbach/parasweep (Last checked: 2021-01)		parallel-bash is written in pure bash. It is really fast (overhead of ~0.
		05 ms/job compared to GNU
		parallel's ~3 ms/job). So if your jobs are extremely short lived, and you
		can live with the quite limited
		command, this may be useful.
Todo		parallel-bash will not start the first job, until it has read all input.
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.		The input can at most be 20935
html		lines and the lines cannot be all be empty.
https://github.com/Nukesor/pueue		Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend running j
		obs.
		EXAMPLES FROM parallel-bash
		1$ some_input | parallel-bash -p 5 -c echo
		1$ some_input | parallel -j 5 echo
		2$ parallel-bash -p 5 -c echo < some_file
		2$ parallel -j 5 echo < some_file
		3$ parallel-bash -p 5 -c echo <<< 'some string'
		3$ parallel -j 5 -c echo <<< 'some string'
		4$ something | parallel-bash -p 5 -c echo {} {}
PASH: Light-touch Data-Parallel Shell Processing		4$ something | parallel -j 5 echo {} {}
		https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.
		html (Last checked: 2021-02)
		DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
		bash-concurrent is more an alternative to make than to GNU parallel. Its
		input is very similar to a
		Makefile, where jobs depend on other jobs.
		It has a nice progress indicator where you can see which jobs completed s
		uccessfully, which jobs are
		currently running, which jobs failed, and which jobs were skipped due to
		a depending job failed. The
		indicator does not deal well with resizing the window.
		Output is cached in tempfiles on disk, but is only shown if there is an e
		rror, so it is not meant to be
		part of a UNIX pipeline. If bash-concurrent crashes these tempfiles are n
		ot removed.
		It uses an O(n*n) algorithm, so if you have 1000 independent jobs it take
		s 22 seconds to start it.
		https://github.com/themattrix/bash-concurrent (Last checked: 2021-02)
		Todo
		https://github.com/Nukesor/pueue
https://arxiv.org/pdf/2012.15443.pdf KumQuat		https://arxiv.org/pdf/2012.15443.pdf KumQuat
https://arxiv.org/pdf/2007.09436.pdf		https://arxiv.org/pdf/2007.09436.pdf PaSH: Light-touch Data-Parallel Shel l Processing
https://github.com/JeiKeiLim/simple_distribute_job		https://github.com/JeiKeiLim/simple_distribute_job
https://github.com/Akianonymus/parallel-bash		https://github.com/reggi/pkgrun - not obvious how to use
https://github.com/reggi/pkgrun
https://github.com/benoror/better-npm-run - not obvious how to use		https://github.com/benoror/better-npm-run - not obvious how to use
https://github.com/bahmutov/with-package		https://github.com/bahmutov/with-package
https://github.com/xuchenCN/go-pssh		https://github.com/xuchenCN/go-pssh
https://github.com/flesler/parallel		https://github.com/flesler/parallel
https://github.com/Julian/Verge		https://github.com/Julian/Verge
		http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
		http://vicerveza.homeunix.net/~viric/soft/ts/
TESTING OTHER TOOLS		TESTING OTHER TOOLS
There are certain issues that are very common on parallelizing tools. Her e are a few stress tests. Be		There are certain issues that are very common on parallelizing tools. Her e are a few stress tests. Be
warned: If the tool is badly coded it may overload your machine.		warned: If the tool is badly coded it may overload your machine.
MIX: Output mixes		MIX: Output mixes
Output from 2 jobs should not mix. If the output is not used, this does n ot matter; but if the output is		Output from 2 jobs should not mix. If the output is not used, this does n ot matter; but if the output is
used then it is important that you do not get half a line from one job fo llowed by half a line from		used then it is important that you do not get half a line from one job fo llowed by half a line from
another job.		another job.
If the tool does not buffer, output will most likely mix now and then.		If the tool does not buffer, output will most likely mix now and then.
skipping to change at line 2841		skipping to change at line 2881
A copy of the full license is included in the file as cc-by-sa.txt.		A copy of the full license is included in the file as cc-by-sa.txt.
DEPENDENCIES		DEPENDENCIES
GNU parallel uses Perl, and the Perl modules Getopt::Long, IPC::Open3, Sy mbol, IO::File, POSIX, and		GNU parallel uses Perl, and the Perl modules Getopt::Long, IPC::Open3, Sy mbol, IO::File, POSIX, and
File::Temp. For remote usage it also uses rsync with ssh.		File::Temp. For remote usage it also uses rsync with ssh.
SEE ALSO		SEE ALSO
find(1), xargs(1), make(1), pexec(1), ppss(1), xjobs(1), prll(1), dxargs( 1), mdm(1)		find(1), xargs(1), make(1), pexec(1), ppss(1), xjobs(1), prll(1), dxargs( 1), mdm(1)
20201222 2021-01-19 PARALLEL_ALTERNATIVES(7)		20210122 2021-02-20 PARALLEL_ALTERNATIVES(7)
End of changes. 15 change blocks.
22 lines changed or deleted		72 lines changed or added

---