Parallel execution with find and xargs

build.sh

#! /usr/bin/env bash

find ./src/bookmarklets -name "*.js"  -print0 | xargs -I FILE -0 -P "$(nproc)" npm run rollup -- --input "FILE" --file "output/FILE"

# EDIT: Per my latest comment - I now prefer this:
for bookmarklet in ./src/bookmarklets/**/*.js
do
  npm run rollup -- --input "$bookmarklet" --file "output/$bookmarklet" &
done
wait

-I FILE tells xargs to use FILE as a placeholder in the command... similar to the use of {} during find/--exec

-P "$(nproc)" tells xargs to run the commands in parallel, with a max of N parallel threads...
In this script I just use the result of nproc command to pass in a number (on my current system, it comes back with 8).

Then, we just run our command as normal (in this case, npm run rollup with some args).

The use of FILE placeholder will get substituted with the value that passed in from find

I ran into a behavior of rollup that I didn't want. Specifically, it was trying to require other modules instead of inlining them.

After a few minutes of searching, I had not found a way to force inlining all code other than to disable multi-file input/output and process each file on its' own.

My first solution was to use find/--exec on each file I wanted to process.

It was taking 3+ seconds per file, which isn't a lot but also isn't super quick. So, I was able to process all files in parallel with find/xargs -P N.

Since my command consumes the filename in multiple spots, I needed to use the placeholder -I FILE.

https://serverfault.com/questions/193319/a-better-unix-find-with-parallel-processing
https://remysharp.com/2016/12/16/tricks-with-xargs
https://www.cyberciti.biz/faq/check-how-many-cpus-are-there-in-linux-system/

After some time to stew on this, and reading some alternatives, I feel that using a loop with background processes is more readable.

for thing in glob
do
  npm run doThingWithArg -- --file=thing & # & sends proc to bg
done
wait # waits for bg procs to complete