2015/03/19

Parallel shell (bash)

find /path/to/files | xargs -n1 -P8 sh -c 'foo $0 && bar $0 || baz $0'

The -n is how many inputs to give each process and the -P indicates the number of processes to run in parallel.
Also important to be aware of is that such a parallel pipeline doesn't guarantee delivery order, but this isn't a problem if you are used to dealing with distributed processing systems.

xargs --max-procs=$(getconf _NPROCESSORS_ONLN)

No comments:

Post a Comment