\input texinfo @setfilename parallel_alternatives.info @documentencoding utf-8 @settitle parallel_alternatives - Alternatives to GNU parallel @node Top @top parallel_alternatives @menu * NAME:: * DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES:: * AUTHOR:: * LICENSE:: * DEPENDENCIES:: * SEE ALSO:: @end menu @node NAME @chapter NAME parallel_alternatives - Alternatives to GNU @strong{parallel} @node DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES @chapter DIFFERENCES BETWEEN GNU Parallel AND ALTERNATIVES There are a lot programs with some of the functionality of GNU @strong{parallel}. GNU @strong{parallel} strives to include the best of the functionality without sacrificing ease of use. @menu * SUMMARY TABLE:: * DIFFERENCES BETWEEN xargs AND GNU Parallel:: * DIFFERENCES BETWEEN find -exec AND GNU Parallel:: * DIFFERENCES BETWEEN make -j AND GNU Parallel:: * DIFFERENCES BETWEEN ppss AND GNU Parallel:: * DIFFERENCES BETWEEN pexec AND GNU Parallel:: * DIFFERENCES BETWEEN xjobs AND GNU Parallel:: * DIFFERENCES BETWEEN prll AND GNU Parallel:: * DIFFERENCES BETWEEN dxargs AND GNU Parallel:: * DIFFERENCES BETWEEN mdm/middleman AND GNU Parallel:: * DIFFERENCES BETWEEN xapply AND GNU Parallel:: * DIFFERENCES BETWEEN AIX apply AND GNU Parallel:: * DIFFERENCES BETWEEN paexec AND GNU Parallel:: * DIFFERENCES BETWEEN map AND GNU Parallel:: * DIFFERENCES BETWEEN ladon AND GNU Parallel:: * DIFFERENCES BETWEEN jobflow AND GNU Parallel:: * DIFFERENCES BETWEEN gargs AND GNU Parallel:: * DIFFERENCES BETWEEN orgalorg AND GNU Parallel:: * DIFFERENCES BETWEEN Rust parallel AND GNU Parallel:: * DIFFERENCES BETWEEN Rush AND GNU Parallel:: * DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel:: @end menu @node SUMMARY TABLE @section SUMMARY TABLE The following features are in some of the comparable tools: Inputs I1. Arguments can be read from stdin I2. Arguments can be read from a file I3. Arguments can be read from multiple files I4. Arguments can be read from command line I5. Arguments can be read from a table I6. Arguments can be read from the same file using #! (shebang) I7. Line oriented input as default (Quoting of special chars not needed) Manipulation of input M1. Composed command M2. Multiple arguments can fill up an execution line M3. Arguments can be put anywhere in the execution line M4. Multiple arguments can be put anywhere in the execution line M5. Arguments can be replaced with context M6. Input can be treated as the complete command line Outputs O1. Grouping output so output from different jobs do not mix O2. Send stderr (standard error) to stderr (standard error) O3. Send stdout (standard output) to stdout (standard output) O4. Order of output can be same as order of input O5. Stdout only contains stdout (standard output) from the command O6. Stderr only contains stderr (standard error) from the command Execution E1. Running jobs in parallel E2. List running jobs E3. Finish running jobs, but do not start new jobs E4. Number of running jobs can depend on number of cpus E5. Finish running jobs, but do not start new jobs after first failure E6. Number of running jobs can be adjusted while running Remote execution R1. Jobs can be run on remote computers R2. Basefiles can be transferred R3. Argument files can be transferred R4. Result files can be transferred R5. Cleanup of transferred files R6. No config files needed R7. Do not run more than SSHD's MaxStartups can handle R8. Configurable SSH command R9. Retry if connection breaks occasionally Semaphore S1. Possibility to work as a mutex S2. Possibility to work as a counting semaphore Legend - = no x = not applicable ID = yes As every new version of the programs are not tested the table may be outdated. Please file a bug-report if you find errors (See REPORTING BUGS). parallel: I1 I2 I3 I4 I5 I6 I7 M1 M2 M3 M4 M5 M6 O1 O2 O3 O4 O5 O6 E1 E2 E3 E4 E5 E6 R1 R2 R3 R4 R5 R6 R7 R8 R9 S1 S2 xargs: I1 I2 - - - - - - M2 M3 - - - - O2 O3 - O5 O6 E1 - - - - - - - - - - x - - - - - find -exec: - - - x - x - - M2 M3 - - - - - O2 O3 O4 O5 O6 - - - - - - - - - - - - - - - - x x make -j: - - - - - - - - - - - - - O1 O2 O3 - x O6 E1 - - - E5 - - - - - - - - - - - - ppss: I1 I2 - - - - I7 M1 - M3 - - M6 O1 - - x - - E1 E2 ?E3 E4 - - R1 R2 R3 R4 - - ?R7 ? ? - - pexec: I1 I2 - I4 I5 - - M1 - M3 - - M6 O1 O2 O3 - O5 O6 E1 - - E4 - E6 R1 - - - - R6 - - - S1 - xjobs, prll, dxargs, mdm/middelman, xapply, paexec, ladon, jobflow, ClusterSSH: TODO - Please file a bug-report if you know what features they support (See REPORTING BUGS). @node DIFFERENCES BETWEEN xargs AND GNU Parallel @section DIFFERENCES BETWEEN xargs AND GNU Parallel @strong{xargs} offers some of the same possibilities as GNU @strong{parallel}. @strong{xargs} deals badly with special characters (such as space, \, ' and "). To see the problem try this: @verbatim touch important_file touch 'not important_file' ls not* | xargs rm mkdir -p "My brother's 12\" records" ls | xargs rmdir touch 'c:\windows\system32\clfs.sys' echo 'c:\windows\system32\clfs.sys' | xargs ls -l @end verbatim You can specify @strong{-0}, but many input generators are not optimized for using @strong{NUL} as separator but are optimized for @strong{newline} as separator. E.g @strong{head}, @strong{tail}, @strong{awk}, @strong{ls}, @strong{echo}, @strong{sed}, @strong{tar -v}, @strong{perl} (@strong{-0} and \0 instead of \n), @strong{locate} (requires using @strong{-0}), @strong{find} (requires using @strong{-print0}), @strong{grep} (requires user to use @strong{-z} or @strong{-Z}), @strong{sort} (requires using @strong{-z}). GNU @strong{parallel}'s newline separation can be emulated with: @strong{cat | xargs -d "\n" -n1 @emph{command}} @strong{xargs} can run a given number of jobs in parallel, but has no support for running number-of-cpu-cores jobs in parallel. @strong{xargs} has no support for grouping the output, therefore output may run together, e.g. the first half of a line is from one process and the last half of the line is from another process. The example @strong{Parallel grep} cannot be done reliably with @strong{xargs} because of this. To see this in action try: @verbatim parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} \ ::: a b c d e f ls -l a b c d e f parallel -kP4 -n1 grep 1 > out.par ::: a b c d e f echo a b c d e f | xargs -P4 -n1 grep 1 > out.xargs-unbuf echo a b c d e f | \ xargs -P4 -n1 grep --line-buffered 1 > out.xargs-linebuf echo a b c d e f | xargs -n1 grep 1 > out.xargs-serial ls -l out* md5sum out* @end verbatim @strong{xargs} has no support for keeping the order of the output, therefore if running jobs in parallel using @strong{xargs} the output of the second job cannot be postponed till the first job is done. @strong{xargs} has no support for running jobs on remote computers. @strong{xargs} has no support for context replace, so you will have to create the arguments. If you use a replace string in @strong{xargs} (@strong{-I}) you can not force @strong{xargs} to use more than one argument. Quoting in @strong{xargs} works like @strong{-q} in GNU @strong{parallel}. This means composed commands and redirection require using @strong{bash -c}. @verbatim ls | parallel "wc {} >{}.wc" ls | parallel "echo {}; ls {}|wc" @end verbatim becomes (assuming you have 8 cores) @verbatim ls | xargs -d "\n" -P8 -I {} bash -c "wc {} >{}.wc" ls | xargs -d "\n" -P8 -I {} bash -c "echo {}; ls {}|wc" @end verbatim @node DIFFERENCES BETWEEN find -exec AND GNU Parallel @section DIFFERENCES BETWEEN find -exec AND GNU Parallel @strong{find -exec} offer some of the same possibilities as GNU @strong{parallel}. @strong{find -exec} only works on files. So processing other input (such as hosts or URLs) will require creating these inputs as files. @strong{find -exec} has no support for running commands in parallel. @node DIFFERENCES BETWEEN make -j AND GNU Parallel @section DIFFERENCES BETWEEN make -j AND GNU Parallel @strong{make -j} can run jobs in parallel, but requires a crafted Makefile to do this. That results in extra quoting to get filename containing newline to work correctly. @strong{make -j} computes a dependency graph before running jobs. Jobs run by GNU @strong{parallel} does not depend on eachother. (Very early versions of GNU @strong{parallel} were coincidently implemented using @strong{make -j}). @node DIFFERENCES BETWEEN ppss AND GNU Parallel @section DIFFERENCES BETWEEN ppss AND GNU Parallel @strong{ppss} is also a tool for running jobs in parallel. The output of @strong{ppss} is status information and thus not useful for using as input for another command. The output from the jobs are put into files. The argument replace string ($ITEM) cannot be changed. Arguments must be quoted - thus arguments containing special characters (space '"&!*) may cause problems. More than one argument is not supported. File names containing newlines are not processed correctly. When reading input from a file null cannot be used as a terminator. @strong{ppss} needs to read the whole input file before starting any jobs. Output and status information is stored in ppss_dir and thus requires cleanup when completed. If the dir is not removed before running @strong{ppss} again it may cause nothing to happen as @strong{ppss} thinks the task is already done. GNU @strong{parallel} will normally not need cleaning up if running locally and will only need cleaning up if stopped abnormally and running remote (@strong{--cleanup} may not complete if stopped abnormally). The example @strong{Parallel grep} would require extra postprocessing if written using @strong{ppss}. For remote systems PPSS requires 3 steps: config, deploy, and start. GNU @strong{parallel} only requires one step. @menu * EXAMPLES FROM ppss MANUAL:: @end menu @node EXAMPLES FROM ppss MANUAL @subsection EXAMPLES FROM ppss MANUAL Here are the examples from @strong{ppss}'s manual page with the equivalent using GNU @strong{parallel}: @strong{1} ./ppss.sh standalone -d /path/to/files -c 'gzip ' @strong{1} find /path/to/files -type f | parallel gzip @strong{2} ./ppss.sh standalone -d /path/to/files -c 'cp "$ITEM" /destination/dir ' @strong{2} find /path/to/files -type f | parallel cp @{@} /destination/dir @strong{3} ./ppss.sh standalone -f list-of-urls.txt -c 'wget -q ' @strong{3} parallel -a list-of-urls.txt wget -q @strong{4} ./ppss.sh standalone -f list-of-urls.txt -c 'wget -q "$ITEM"' @strong{4} parallel -a list-of-urls.txt wget -q @{@} @strong{5} ./ppss config -C config.cfg -c 'encode.sh ' -d /source/dir -m 192.168.1.100 -u ppss -k ppss-key.key -S ./encode.sh -n nodes.txt -o /some/output/dir --upload --download ; ./ppss deploy -C config.cfg ; ./ppss start -C config @strong{5} # parallel does not use configs. If you want a different username put it in nodes.txt: user@@hostname @strong{5} find source/dir -type f | parallel --sshloginfile nodes.txt --trc @{.@}.mp3 lame -a @{@} -o @{.@}.mp3 --preset standard --quiet @strong{6} ./ppss stop -C config.cfg @strong{6} killall -TERM parallel @strong{7} ./ppss pause -C config.cfg @strong{7} Press: CTRL-Z or killall -SIGTSTP parallel @strong{8} ./ppss continue -C config.cfg @strong{8} Enter: fg or killall -SIGCONT parallel @strong{9} ./ppss.sh status -C config.cfg @strong{9} killall -SIGUSR2 parallel @node DIFFERENCES BETWEEN pexec AND GNU Parallel @section DIFFERENCES BETWEEN pexec AND GNU Parallel @strong{pexec} is also a tool for running jobs in parallel. @menu * EXAMPLES FROM pexec MANUAL:: @end menu @node EXAMPLES FROM pexec MANUAL @subsection EXAMPLES FROM pexec MANUAL Here are the examples from @strong{pexec}'s info page with the equivalent using GNU @strong{parallel}: @strong{1} pexec -o sqrt-%s.dat -p "$(seq 10)" -e NUM -n 4 -c -- \ 'echo "scale=10000;sqrt($NUM)" | bc' @strong{1} seq 10 | parallel -j4 'echo "scale=10000;sqrt(@{@})" | bc > sqrt-@{@}.dat' @strong{2} pexec -p "$(ls myfiles*.ext)" -i %s -o %s.sort -- sort @strong{2} ls myfiles*.ext | parallel sort @{@} ">@{@}.sort" @strong{3} pexec -f image.list -n auto -e B -u star.log -c -- \ 'fistar $B.fits -f 100 -F id,x,y,flux -o $B.star' @strong{3} parallel -a image.list \ 'fistar @{@}.fits -f 100 -F id,x,y,flux -o @{@}.star' 2>star.log @strong{4} pexec -r *.png -e IMG -c -o - -- \ 'convert $IMG $@{IMG%.png@}.jpeg ; "echo $IMG: done"' @strong{4} ls *.png | parallel 'convert @{@} @{.@}.jpeg; echo @{@}: done' @strong{5} pexec -r *.png -i %s -o %s.jpg -c 'pngtopnm | pnmtojpeg' @strong{5} ls *.png | parallel 'pngtopnm < @{@} | pnmtojpeg > @{@}.jpg' @strong{6} for p in *.png ; do echo $@{p%.png@} ; done | \ pexec -f - -i %s.png -o %s.jpg -c 'pngtopnm | pnmtojpeg' @strong{6} ls *.png | parallel 'pngtopnm < @{@} | pnmtojpeg > @{.@}.jpg' @strong{7} LIST=$(for p in *.png ; do echo $@{p%.png@} ; done) pexec -r $LIST -i %s.png -o %s.jpg -c 'pngtopnm | pnmtojpeg' @strong{7} ls *.png | parallel 'pngtopnm < @{@} | pnmtojpeg > @{.@}.jpg' @strong{8} pexec -n 8 -r *.jpg -y unix -e IMG -c \ 'pexec -j -m blockread -d $IMG | \ jpegtopnm | pnmscale 0.5 | pnmtojpeg | \ pexec -j -m blockwrite -s th_$IMG' @strong{8} Combining GNU @strong{parallel} and GNU @strong{sem}. @strong{8} ls *jpg | parallel -j8 'sem --id blockread cat @{@} | jpegtopnm |' \ 'pnmscale 0.5 | pnmtojpeg | sem --id blockwrite cat > th_@{@}' @strong{8} If reading and writing is done to the same disk, this may be faster as only one process will be either reading or writing: @strong{8} ls *jpg | parallel -j8 'sem --id diskio cat @{@} | jpegtopnm |' \ 'pnmscale 0.5 | pnmtojpeg | sem --id diskio cat > th_@{@}' @node DIFFERENCES BETWEEN xjobs AND GNU Parallel @section DIFFERENCES BETWEEN xjobs AND GNU Parallel @strong{xjobs} is also a tool for running jobs in parallel. It only supports running jobs on your local computer. @strong{xjobs} deals badly with special characters just like @strong{xargs}. See the section @strong{DIFFERENCES BETWEEN xargs AND GNU Parallel}. Here are the examples from @strong{xjobs}'s man page with the equivalent using GNU @strong{parallel}: @strong{1} ls -1 *.zip | xjobs unzip @strong{1} ls *.zip | parallel unzip @strong{2} ls -1 *.zip | xjobs -n unzip @strong{2} ls *.zip | parallel unzip >/dev/null @strong{3} find . -name '*.bak' | xjobs gzip @strong{3} find . -name '*.bak' | parallel gzip @strong{4} ls -1 *.jar | sed 's/\(.*\)/\1 > \1.idx/' | xjobs jar tf @strong{4} ls *.jar | parallel jar tf @{@} '>' @{@}.idx @strong{5} xjobs -s script @strong{5} cat script | parallel @strong{6} mkfifo /var/run/my_named_pipe; xjobs -s /var/run/my_named_pipe & echo unzip 1.zip >> /var/run/my_named_pipe; echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe @strong{6} mkfifo /var/run/my_named_pipe; cat /var/run/my_named_pipe | parallel & echo unzip 1.zip >> /var/run/my_named_pipe; echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe @node DIFFERENCES BETWEEN prll AND GNU Parallel @section DIFFERENCES BETWEEN prll AND GNU Parallel @strong{prll} is also a tool for running jobs in parallel. It does not support running jobs on remote computers. @strong{prll} encourages using BASH aliases and BASH functions instead of scripts. GNU @strong{parallel} supports scripts directly, functions if they are exported using @strong{export -f}, and aliases if using @strong{env_parallel}. @strong{prll} generates a lot of status information on stderr (standard error) which makes it harder to use the stderr (standard error) output of the job directly as input for another program. Here is the example from @strong{prll}'s man page with the equivalent using GNU @strong{parallel}: @verbatim prll -s 'mogrify -flip $1' *.jpg parallel mogrify -flip ::: *.jpg @end verbatim @node DIFFERENCES BETWEEN dxargs AND GNU Parallel @section DIFFERENCES BETWEEN dxargs AND GNU Parallel @strong{dxargs} is also a tool for running jobs in parallel. @strong{dxargs} does not deal well with more simultaneous jobs than SSHD's MaxStartups. @strong{dxargs} is only built for remote run jobs, but does not support transferring of files. @node DIFFERENCES BETWEEN mdm/middleman AND GNU Parallel @section DIFFERENCES BETWEEN mdm/middleman AND GNU Parallel middleman(mdm) is also a tool for running jobs in parallel. Here are the shellscripts of http://mdm.berlios.de/usage.html ported to GNU @strong{parallel}: @verbatim seq 19 | parallel buffon -o - | sort -n > result cat files | parallel cmd find dir -execdir sem cmd {} \; @end verbatim @node DIFFERENCES BETWEEN xapply AND GNU Parallel @section DIFFERENCES BETWEEN xapply AND GNU Parallel @strong{xapply} can run jobs in parallel on the local computer. Here are the examples from @strong{xapply}'s man page with the equivalent using GNU @strong{parallel}: @strong{1} xapply '(cd %1 && make all)' */ @strong{1} parallel 'cd @{@} && make all' ::: */ @strong{2} xapply -f 'diff %1 ../version5/%1' manifest | more @strong{2} parallel diff @{@} ../version5/@{@} < manifest | more @strong{3} xapply -p/dev/null -f 'diff %1 %2' manifest1 checklist1 @strong{3} parallel --link diff @{1@} @{2@} :::: manifest1 checklist1 @strong{4} xapply 'indent' *.c @strong{4} parallel indent ::: *.c @strong{5} find ~ksb/bin -type f ! -perm -111 -print | xapply -f -v 'chmod a+x' - @strong{5} find ~ksb/bin -type f ! -perm -111 -print | parallel -v chmod a+x @strong{6} find */ -... | fmt 960 1024 | xapply -f -i /dev/tty 'vi' - @strong{6} sh <(find */ -... | parallel -s 1024 echo vi) @strong{6} find */ -... | parallel -s 1024 -Xuj1 vi @strong{7} find ... | xapply -f -5 -i /dev/tty 'vi' - - - - - @strong{7} sh <(find ... |parallel -n5 echo vi) @strong{7} find ... |parallel -n5 -uj1 vi @strong{8} xapply -fn "" /etc/passwd @strong{8} parallel -k echo < /etc/passwd @strong{9} tr ':' '\012' < /etc/passwd | xapply -7 -nf 'chown %1 %6' - - - - - - - @strong{9} tr ':' '\012' < /etc/passwd | parallel -N7 chown @{1@} @{6@} @strong{10} xapply '[ -d %1/RCS ] || echo %1' */ @strong{10} parallel '[ -d @{@}/RCS ] || echo @{@}' ::: */ @strong{11} xapply -f '[ -f %1 ] && echo %1' List | ... @strong{11} parallel '[ -f @{@} ] && echo @{@}' < List | ... @node DIFFERENCES BETWEEN AIX apply AND GNU Parallel @section DIFFERENCES BETWEEN AIX apply AND GNU Parallel @strong{apply} can build command lines based on a template and arguments - very much like GNU @strong{parallel}. @strong{apply} does not run jobs in parallel. @strong{apply} does not use an argument separator (like @strong{:::}); instead the template must be the first argument. Here are the examples from https://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds1/apply.htm 1. To obtain results similar to those of the @strong{ls} command, enter: @verbatim apply echo * parallel echo ::: * @end verbatim 2. To compare the file named @strong{a1} to the file named @strong{b1}, and the file named @strong{a2} to the file named @strong{b2}, enter: @verbatim apply -2 cmp a1 b1 a2 b2 parallel -N2 cmp ::: a1 b1 a2 b2 @end verbatim 3. To run the @strong{who} command five times, enter: @verbatim apply -0 who 1 2 3 4 5 parallel -N0 who ::: 1 2 3 4 5 @end verbatim 4. To link all files in the current directory to the directory @strong{/usr/joe}, enter: @verbatim apply 'ln %1 /usr/joe' * parallel ln {} /usr/joe ::: * @end verbatim @node DIFFERENCES BETWEEN paexec AND GNU Parallel @section DIFFERENCES BETWEEN paexec AND GNU Parallel @strong{paexec} can run jobs in parallel on both the local and remote computers. @strong{paexec} requires commands to print a blank line as the last output. This means you will have to write a wrapper for most programs. @strong{paexec} has a job dependency facility so a job can depend on another job to be executed successfully. Sort of a poor-man's @strong{make}. Here are the examples from @strong{paexec}'s example catalog with the equivalent using GNU @strong{parallel}: @table @asis @item 1_div_X_run: @anchor{1_div_X_run:} @verbatim ../../paexec -s -l -c "`pwd`/1_div_X_cmd" -n +1 <<EOF [...] parallel echo {} '|' `pwd`/1_div_X_cmd <<EOF [...] @end verbatim @item all_substr_run: @anchor{all_substr_run:} @verbatim ../../paexec -lp -c "`pwd`/all_substr_cmd" -n +3 <<EOF [...] parallel echo {} '|' `pwd`/all_substr_cmd <<EOF [...] @end verbatim @item cc_wrapper_run: @anchor{cc_wrapper_run:} @verbatim ../../paexec -c "env CC=gcc CFLAGS=-O2 `pwd`/cc_wrapper_cmd" \ -n 'host1 host2' \ -t '/usr/bin/ssh -x' <<EOF [...] parallel echo {} '|' "env CC=gcc CFLAGS=-O2 `pwd`/cc_wrapper_cmd" \ -S host1,host2 <<EOF [...] # This is not exactly the same, but avoids the wrapper parallel gcc -O2 -c -o {.}.o {} \ -S host1,host2 <<EOF [...] @end verbatim @item toupper_run: @anchor{toupper_run:} @verbatim ../../paexec -lp -c "`pwd`/toupper_cmd" -n +10 <<EOF [...] parallel echo {} '|' ./toupper_cmd <<EOF [...] # Without the wrapper: parallel echo {} '| awk {print\ toupper\(\$0\)}' <<EOF [...] @end verbatim @end table @node DIFFERENCES BETWEEN map AND GNU Parallel @section DIFFERENCES BETWEEN map AND GNU Parallel @strong{map} sees it as a feature to have less features and in doing so it also handles corner cases incorrectly. A lot of GNU @strong{parallel}'s code is to handle corner cases correctly on every platform, so you will not get a nasty surprise if a user for example saves a file called: @emph{My brother's 12" records.txt} @strong{map}'s example showing how to deal with special characters fails on special characters: @verbatim echo "The Cure" > My\ brother\'s\ 12\"\ records ls | \ map 'echo -n `gzip < "%" | wc -c`; echo -n '*100/'; wc -c < "%"' | bc @end verbatim It works with GNU @strong{parallel}: @verbatim ls | \ parallel 'echo -n `gzip < {} | wc -c`; echo -n '*100/'; wc -c < {}' | bc @end verbatim And you can even get the file name prepended: @verbatim ls | \ parallel --tag '(echo -n `gzip < {} | wc -c`'*100/'; wc -c < {}) | bc' @end verbatim @strong{map} has no support for grouping. So this gives the wrong results without any warnings: @verbatim parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} \ ::: a b c d e f ls -l a b c d e f parallel -kP4 -n1 grep 1 > out.par ::: a b c d e f map -p 4 'grep 1' a b c d e f > out.map-unbuf map -p 4 'grep --line-buffered 1' a b c d e f > out.map-linebuf map -p 1 'grep --line-buffered 1' a b c d e f > out.map-serial ls -l out* md5sum out* @end verbatim The documentation shows a workaround, but not only does that mix stdout (standard output) with stderr (standard error) it also fails completely for certain jobs (and may even be considered less readable): @verbatim parallel echo -n {} ::: 1 2 3 map -p 4 'echo -n % 2>&1 | sed -e "s/^/$$:/"' 1 2 3 | sort | cut -f2- -d: @end verbatim @strong{map}s replacement strings (% %D %B %E) can be simulated in GNU @strong{parallel} by putting this in @strong{~/.parallel/config}: @verbatim --rpl '%' --rpl '%D $_=::shell_quote(::dirname($_));' --rpl '%B s:.*/::;s:\.[^/.]+$::;' --rpl '%E s:.*\.::' @end verbatim @strong{map} cannot handle bundled options: @strong{map -vp 0 echo this fails} @strong{map} does not have an argument separator on the command line, but uses the first argument as command. This makes quoting harder which again may affect readability. Compare: @verbatim map -p 2 perl\\\ -ne\\\ \\\'/^\\\\S+\\\\s+\\\\S+\\\$/\\\ and\\\ print\\\ \\\$ARGV,\\\"\\\\n\\\"\\\' * parallel -q perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"' ::: * @end verbatim @strong{map} can do multiple arguments with context replace, but not without context replace: @verbatim parallel --xargs echo 'BEGIN{'{}'}END' ::: 1 2 3 @end verbatim @strong{map} does not set exit value according to whether one of the jobs failed: @verbatim parallel false ::: 1 || echo Job failed map false 1 || echo Never run @end verbatim @strong{map} requires Perl v5.10.0 making it harder to use on old systems. @strong{map} has no way of using % in the command (GNU Parallel has -I to specify another replacement string than @strong{@{@}}). By design @strong{map} is option incompatible with @strong{xargs}, it does not have remote job execution, a structured way of saving results, multiple input sources, progress indicator, configurable record delimiter (only field delimiter), logging of jobs run with possibility to resume, keeping the output in the same order as input, --pipe processing, and dynamically timeouts. @node DIFFERENCES BETWEEN ladon AND GNU Parallel @section DIFFERENCES BETWEEN ladon AND GNU Parallel @strong{ladon} can run multiple jobs on files in parallel. @strong{ladon} only works on files and the only way to specify files is using a quoted glob string (such as \*.jpg). It is not possible to list the files manually. As replacement strings it uses FULLPATH DIRNAME BASENAME EXT RELDIR RELPATH These can be simulated using GNU @strong{parallel} by putting this in @strong{~/.parallel/config}: @verbatim --rpl 'FULLPATH $_=::shell_quote($_);chomp($_=qx{readlink -f $_});' --rpl 'DIRNAME $_=::shell_quote(::dirname($_));chomp($_=qx{readlink -f $_});' --rpl 'BASENAME s:.*/::;s:\.[^/.]+$::;' --rpl 'EXT s:.*\.::' --rpl 'RELDIR $_=::shell_quote($_);chomp(($_,$c)=qx{readlink -f $_;pwd});s:\Q$c/\E::;$_=::dirname($_);' --rpl 'RELPATH $_=::shell_quote($_);chomp(($_,$c)=qx{readlink -f $_;pwd});s:\Q$c/\E::;' @end verbatim @strong{ladon} deals badly with filenames containing " and newline, and it fails for output larger than 200k: @verbatim ladon '*' -- seq 36000 | wc @end verbatim @menu * EXAMPLES FROM ladon MANUAL:: @end menu @node EXAMPLES FROM ladon MANUAL @subsection EXAMPLES FROM ladon MANUAL It is assumed that the '--rpl's above are put in @strong{~/.parallel/config} and that it is run under a shell that supports '**' globbing (such as @strong{zsh}): @strong{1} ladon "**/*.txt" -- echo RELPATH @strong{1} parallel echo RELPATH ::: **/*.txt @strong{2} ladon "~/Documents/**/*.pdf" -- shasum FULLPATH >hashes.txt @strong{2} parallel shasum FULLPATH ::: ~/Documents/**/*.pdf >hashes.txt @strong{3} ladon -m thumbs/RELDIR "**/*.jpg" -- convert FULLPATH -thumbnail 100x100^ -gravity center -extent 100x100 thumbs/RELPATH @strong{3} parallel mkdir -p thumbs/RELDIR\; convert FULLPATH -thumbnail 100x100^ -gravity center -extent 100x100 thumbs/RELPATH ::: **/*.jpg @strong{4} ladon "~/Music/*.wav" -- lame -V 2 FULLPATH DIRNAME/BASENAME.mp3 @strong{4} parallel lame -V 2 FULLPATH DIRNAME/BASENAME.mp3 ::: ~/Music/*.wav @node DIFFERENCES BETWEEN jobflow AND GNU Parallel @section DIFFERENCES BETWEEN jobflow AND GNU Parallel @strong{jobflow} can run multiple jobs in parallel. Just like @strong{xargs} output from @strong{jobflow} jobs running in parallel mix together by default. @strong{jobflow} can buffer into files (placed in /run/shm), but these are not cleaned up - not even if @strong{jobflow} dies unexpectently. If the total output is big (in the order of RAM+swap) it can cause the system to run out of memory. @strong{jobflow} gives no error if the command is unknown, and like @strong{xargs} redirection requires wrapping with @strong{bash -c}. @strong{jobflow} makes it possible to set ressource limits on the running jobs. This can be emulated by GNU @strong{parallel} using @strong{bash}'s @strong{ulimit}: @verbatim jobflow -limits=mem=100M,cpu=3,fsize=20M,nofiles=300 myjob parallel 'ulimit -v 102400 -t 3 -f 204800 -n 300 myjob' @end verbatim @menu * EXAMPLES FROM jobflow README:: @end menu @node EXAMPLES FROM jobflow README @subsection EXAMPLES FROM jobflow README @strong{1} cat things.list | jobflow -threads=8 -exec ./mytask @{@} @strong{1} cat things.list | parallel -j8 ./mytask @{@} @strong{2} seq 100 | jobflow -threads=100 -exec echo @{@} @strong{2} seq 100 | parallel -j100 echo @{@} @strong{3} cat urls.txt | jobflow -threads=32 -exec wget @{@} @strong{3} cat urls.txt | parallel -j32 wget @{@} @strong{4} find . -name '*.bmp' | jobflow -threads=8 -exec bmp2jpeg @{.@}.bmp @{.@}.jpg @strong{4} find . -name '*.bmp' | parallel -j8 bmp2jpeg @{.@}.bmp @{.@}.jpg @node DIFFERENCES BETWEEN gargs AND GNU Parallel @section DIFFERENCES BETWEEN gargs AND GNU Parallel @strong{gargs} can run multiple jobs in parallel. It caches output in memory. This causes it to be extremely slow when the output is larger than the physical RAM, and can cause the system to run out of memory. See more details on this in @strong{man parallel_design}. Output to stderr (standard error) is changed if the command fails. Here are the two examples from @strong{gargs} website. @strong{1} seq 12 -1 1 | gargs -p 4 -n 3 "sleep @{0@}; echo @{1@} @{2@}" @strong{1} seq 12 -1 1 | parallel -P 4 -n 3 "sleep @{1@}; echo @{2@} @{3@}" @strong{2} cat t.txt | gargs --sep "\s+" -p 2 "echo '@{0@}:@{1@}-@{2@}' full-line: \'@{@}\'" @strong{2} cat t.txt | parallel --colsep "\\s+" -P 2 "echo '@{1@}:@{2@}-@{3@}' full-line: \'@{@}\'" @node DIFFERENCES BETWEEN orgalorg AND GNU Parallel @section DIFFERENCES BETWEEN orgalorg AND GNU Parallel @strong{orgalorg} can run the same job on multiple machines. This is related to @strong{--onall} and @strong{--nonall}. @strong{orgalorg} supports entering the SSH password - provided it is the same for all servers. GNU @strong{parallel} advocates using @strong{ssh-agent} instead, but it is possible to emulate @strong{orgalorg}'s behavior by setting SSHPASS and by using @strong{--ssh "sshpass ssh"}. To make the emulation easier, make a simple alias: @verbatim alias par_emul="parallel -j0 --ssh 'sshpass ssh' --nonall --tag --linebuffer" @end verbatim If you want to supply a password run: @verbatim SSHPASS=`ssh-askpass` @end verbatim or set the password directly: @verbatim SSHPASS=P4$$w0rd! @end verbatim If the above is set up you can then do: @verbatim orgalorg -o frontend1 -o frontend2 -p -C uptime par_emul -S frontend1 -S frontend2 uptime orgalorg -o frontend1 -o frontend2 -p -C top -bid 1 par_emul -S frontend1 -S frontend2 top -bid 1 orgalorg -o frontend1 -o frontend2 -p -er /tmp -n 'md5sum /tmp/bigfile' -S bigfile par_emul -S frontend1 -S frontend2 --basefile bigfile --workdir /tmp md5sum /tmp/bigfile @end verbatim @strong{orgalorg} has a progress indicator for the transferring of a file. GNU @strong{parallel} does not. @node DIFFERENCES BETWEEN Rust parallel AND GNU Parallel @section DIFFERENCES BETWEEN Rust parallel AND GNU Parallel Rust parallel focuses on speed. It is almost as fast as @strong{xargs}. It implements a few features from GNU @strong{parallel}, but lacks many functions. All these fail: @verbatim # Show what would be executed parallel --dry-run echo ::: a # Read arguments from file parallel -a file echo # Changing the delimiter parallel -d _ echo ::: a_b_c_ @end verbatim These do something different from GNU @strong{parallel} @verbatim # Read more arguments at a time -n parallel -n 2 echo ::: 1 a 2 b # -q to protect quoted $ and space parallel -q perl -e '$a=shift; print "$a"x10000000' ::: a b c # Generation of combination of inputs parallel echo {1} {2} ::: red green blue ::: S M L XL XXL # {= perl expression =} replacement string parallel echo '{= s/new/old/ =}' ::: my.new your.new # --pipe seq 100000 | parallel --pipe wc # linked arguments parallel echo ::: S M L :::+ small medium large ::: R G B :::+ red green blue # Run different shell dialects zsh -c 'parallel echo \={} ::: zsh && true' csh -c 'parallel echo \$\{\} ::: shell && true' bash -c 'parallel echo \$\({}\) ::: pwd && true' # Rust parallel does not start before the last argument is read (seq 10; sleep 5; echo 2) | time parallel -j2 'sleep 2; echo' tail -f /var/log/syslog | parallel echo @end verbatim Rust parallel has no remote facilities. It uses /tmp/parallel for tmp files and does not clean up if terminated abrubtly. If another user on the system uses Rust parallel, then /tmp/parallel will have the wrong permissions and Rust parallel will fail. A malicious user can setup the right permissions and symlink the output file to one of the user's files and next time the user uses Rust parallel it will overwrite this file. If /tmp/parallel runs full during the run, Rust parallel does not report this, but finishes with success - thereby risking data loss. @node DIFFERENCES BETWEEN Rush AND GNU Parallel @section DIFFERENCES BETWEEN Rush AND GNU Parallel Rush (https://github.com/shenwei356/rush) is written in Go and based on gargs. @node DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel @section DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel ClusterSSH solves a different problem than GNU @strong{parallel}. ClusterSSH opens a terminal window for each computer and using a master window you can run the same command on all the computers. This is typically used for administrating several computers that are almost identical. GNU @strong{parallel} runs the same (or different) commands with different arguments in parallel possibly using remote computers to help computing. If more than one computer is listed in @strong{-S} GNU @strong{parallel} may only use one of these (e.g. if there are 8 jobs to be run and one computer has 8 cores). GNU @strong{parallel} can be used as a poor-man's version of ClusterSSH: @strong{parallel --nonall -S server-a,server-b do_stuff foo bar} @node AUTHOR @chapter AUTHOR When using GNU @strong{parallel} for a publication please cite: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47. This helps funding further development; and it won't cost you a cent. If you pay 10000 EUR you should feel free to use GNU Parallel without citing. Copyright (C) 2007-10-18 Ole Tange, http://ole.tange.dk Copyright (C) 2008,2009,2010 Ole Tange, http://ole.tange.dk Copyright (C) 2010,2011,2012,2013,2014,2015,2016,2017 Ole Tange, http://ole.tange.dk and Free Software Foundation, Inc. Parts of the manual concerning @strong{xargs} compatibility is inspired by the manual of @strong{xargs} from GNU findutils 4.4.2. @node LICENSE @chapter LICENSE Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017 Free Software Foundation, Inc. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or at your option any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>. @menu * Documentation license I:: * Documentation license II:: @end menu @node Documentation license I @section Documentation license I Permission is granted to copy, distribute and/or modify this documentation under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the file fdl.txt. @node Documentation license II @section Documentation license II You are free: @table @asis @item @strong{to Share} @anchor{@strong{to Share}} to copy, distribute and transmit the work @item @strong{to Remix} @anchor{@strong{to Remix}} to adapt the work @end table Under the following conditions: @table @asis @item @strong{Attribution} @anchor{@strong{Attribution}} You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). @item @strong{Share Alike} @anchor{@strong{Share Alike}} If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. @end table With the understanding that: @table @asis @item @strong{Waiver} @anchor{@strong{Waiver}} Any of the above conditions can be waived if you get permission from the copyright holder. @item @strong{Public Domain} @anchor{@strong{Public Domain}} Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license. @item @strong{Other Rights} @anchor{@strong{Other Rights}} In no way are any of the following rights affected by the license: @itemize @item Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations; @item The author's moral rights; @item Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. @end itemize @end table @table @asis @item @strong{Notice} @anchor{@strong{Notice}} For any reuse or distribution, you must make clear to others the license terms of this work. @end table A copy of the full license is included in the file as cc-by-sa.txt. @node DEPENDENCIES @chapter DEPENDENCIES GNU @strong{parallel} uses Perl, and the Perl modules Getopt::Long, IPC::Open3, Symbol, IO::File, POSIX, and File::Temp. For remote usage it also uses rsync with ssh. @node SEE ALSO @chapter SEE ALSO @strong{find}(1), @strong{xargs}(1), @strong{make}(1), @strong{pexec}(1), @strong{ppss}(1), @strong{xjobs}(1), @strong{prll}(1), @strong{dxargs}(1), @strong{mdm}(1) @bye