Comments on: xargs parallel execution http://matpalm.com/blog/2009/11/06/xargs-parallel-execution/ thoughts from a data scientist wannabe Sun, 06 Jun 2010 19:38:46 -0400 http://wordpress.org/?v=2.9.2 hourly 1 By: matpalm http://matpalm.com/blog/2009/11/06/xargs-parallel-execution/comment-page-1/#comment-52 matpalm Thu, 28 Jan 2010 08:45:12 +0000 http://matpalm.com/blog/?p=217#comment-52 thanks ole, had never heard of parallel before will have to add it to my must-install list mat thanks ole,
had never heard of parallel before
will have to add it to my must-install list
mat

]]> By: Ole Tange http://matpalm.com/blog/2009/11/06/xargs-parallel-execution/comment-page-1/#comment-51 Ole Tange Thu, 28 Jan 2010 07:23:28 +0000 http://matpalm.com/blog/?p=217#comment-51 As you have discovered you need to make temporary files to avoid mixing xargs' output for different jobs. Parallel <a href="https://savannah.nongnu.org/projects/parallel/" rel="nofollow">https://savannah.nongnu.org/projects/parallel/</a> does not suffer from this: ls sample*gz | parallel -j+0 -k 'zcat {} | ./script.rb' > output -k makes sure the output is in the same order as input. -j+0 runs number_of_cores jobs in parallel. As you have discovered you need to make temporary files to avoid mixing xargs’ output for different jobs.

Parallel https://savannah.nongnu.org/projects/parallel/ does not suffer from this:

ls sample*gz | parallel -j+0 -k ‘zcat {} | ./script.rb’ > output

-k makes sure the output is in the same order as input.

-j+0 runs number_of_cores jobs in parallel.

]]>