Running commands in Workspaces

running-cmds-in-workspaces.md

• RFC TODAY: npm fund --workspace=a
• RFC TODO CHANGE: npm ws fund --workspace=a --workspace=b --workspace=group-name

Adding a top-level workspaces|ws command should abstract enough the implementation to make it flexible enough to accomodate future tweaks in the workspace installing algorithm.

Where to start?

• Add new workspaces|ws command/alias
• Add new folder ./lib/workspaces/*.js
• Add default behavior that sets prefix to top-level commands under ./lib/workspaces/default.js
• Add ws run-script
• Add ws install (ie. scoped installs)
• Add ws install <pkg>
• Add ws ci
• Add ws update
• Add ws uninstall
• Add ws outdated (ie. cd ./ && npm outdated, might need tweaking Arborist to only load from a specific tree node)
• Add ws ls (ie. cd ./ && npm ls)
• Add ``

What should happen?

• Should run commands over multiple workspaces
  • (no args) run command across all workspaces
  • (-w=<workspace-name> named option) filter by only defined names
• Should support --parallel (defaults to --serial)
• ref. https://www.npmjs.com/package/npm-run-all

What happens under the hood?

./lib/workspaces.js `npm workspaces|ws`
./lib/workspaces/default.js <- default... tries `prefix` + <command> / warn if we couldn't do anything...
./lib/workspaces/install.js <- some cmds require special logic
./lib/workspaces/publish.js <- would set `prefix` & then include ./lib/publish.js
...

# The default behavior is to run the command setting the prefix to workspace realpath, e.g:
npm ws publish -w name
# Might be effectively the same as:
npm publish --prefix=<workspace-name>
# Assuming `npm publish` is a command that won't need special tweaks/impl

npm ws install -w name
#      ^--- "scoped install": *only* reify the packages for the workspace defined, e.g:

root:
dependencies:
  d@1.0.0
workspaces:
  a -> foo@^1.0.0 -> c@1
  b -> foo@^1.0.1 -> c@2
  
$ npm ws install -w a
node_modules
+- a -> ../a
+- c@2
+- foo@1.0.1
# NOTE: just be mindful of deduping (ie. you'd get c@2 if all workspaces
# were being installed... you should still get it if you only specify `a`)
# NOTE2: arborist will not place `d` within `node_modules` for
# a "scoped install"

# Adding a new dep to a workspace:
$ npm ws install -w <workspace-name> <pkg> -> ./lib/workspaces/install.js 
#                                  ^--- <pkg> will be installed as a
#                                       dep of workspace-name

Adding a new dep to a workspace:

npm install <pkg>
Arborist
root:
  - <pkg> <-- add user request
 
npm ws install <pkg> -w <workspace-name>
Arborist
root:
  - workspace-name:
    - <pkg> <-- add user request under workspace-name instead

API:

npm ws <command> -w|--workspace=<pkg-name|group-alias>

Groups:

A simple way to refer to a set of workspace by using a single name, e.g:

.
+- core
  +- foo
+- plugins
  +- lorem
  +- ipsum

With a root package.json defining both workspaces packages and groups:

{
    "name": "workspace-example",
    "version": "1.0.0",
    "workspaces": {
        "groups": {
            "plugins": ["lorem", "ipsum"],
            "common": ["foo"]
        },
        "packages": [
            "core/*",
            "plugins/*"
        ]
    }
}

Running: npm ws install abbrev -w plugins effectively means adding abbrev as a dep to both lorem and ipsum and reifying the tree.

Arborist needs API for:

• all reify options (add/rm/update), operating on a given workspace node (so, "update all deps of this workspace", or "add this pkg spec to the workspace node's deps, rather than to root", etc.), updating the package.json for that workspace project
• reify limited to a given workspace node
• lockfile consideration: if we still only have a single lockfile in the project root, we still need a {complete:true} buildIdealTree for everything other than the workspace(s) being reified, so that we can write the package-lock.json fully.

This (ie, the CLI) needs at least stub API for:

• add workspace
  • folder exists?
• list workspaces (list the workspaces themselves, different from npm ws list --workspace=a, which runs npm ls under workspace a)
• add to groups? list groups? (maybe punt on groups for now, just say "you can use a group alias to stand in for all workspaces in that group", and leave "how to create/modify groups" for a later RFC?)
• what does this do? npm ws i -w=core/*? Should we support the paths/globs as well as the name?
• what is npm_command environment var set to when I run npm ws test? "ws" or "test"?

Idea, for when we do dig into ws groups: maybe the folder is the group, and we just have that be the opinionated convention? Ie, if you have packages: [ "core/*", "plugins/*" ] then you can use -w=core and that'll pick up all the packages under core/*.

For the future, nested workspaces? Could we ever imagine npm ws -w foo ws -w bar install? Would run the install in packages/foo/packages/bar, where bar is a workspace nested under the foo workspace.

• if -w is ambiguous between name and path, use name, and print a warning, because you can always do npm ws i -w core/ or -w ./core to make it only match the path.

This will never work: npm ws i -w core/*, because the shell will expand that to npm ws i -w core/foo core/bar core/baz. But npm ws i -w=core/* or npm i -w "core/*" would work fine.

So, suggestion:

• if a -w arg matches a workspace name, use that workspace by that name. -w=foo will match only core/foo (where package is named foo)
• otherwise, expand glob, include all workspaces under the found path(s). -w=core/* will match ./core/foo and ./core/bar. -w=./core will also match ./core/foo and ./core/bar, because it's a partial path.
• -w core/* will not work, and we aren't going to try to reverse engineer from -w core/foo core/bar to what the user might have actually typed into the shell. (nopt is weird and broken, but at least it's weird and broken in one place.)

Action item: @isaacs write RFC for parsing cli options more strictly finally, with warnings in npm v8 for what will break in npm v9.

• API does seem a bit better to only do npm i -w=a instead of npm ws i -w=a, just go through the appropriate code path when --workspaces config is set, and reserve npm ws for management of workspaces config. npm ws add, npm ws ls, npm ws rm, etc. Implementation-wise, that's a little bit trickier though, maybe? Might need Arborist to be able to just take a workspaces config, and pass it through without even looking at it, in some cases?

regarding globbing, there are differences and footguns here when using different shells. in fish npm ws -w=core/* you'll actually get an error about there being no matches for the glob.

IMO we should get the commands working first on manually specified workspace names and have a separate RFC later for how we specify groups of workspaces to act on

Wait, WAT? Fish has failglob on by default? (Digs a little deeper into my comfortable "bourne again shell forever" ditch.)

IMO we should get the commands working first on manually specified workspace names and have a separate RFC later for how we specify groups of workspaces to act on

I agree, but if we say "the arguments are (partial) paths", then we get groups for free by convention, without having extra bookkeeping. If someone changes the name of their package, or moves it from one folder to another, the groups will always match what's on the filesystem, because that is the groups.

We can still layer other grouping UX on top of that, of course, but if we make the -w argument limited to package names, we force ourselves to add more abstractions on top of that abstraction to get that functionality. (Maybe that's what we'll want to do anyway, but I'm not convinced without more careful thinking about it.)

On the other hand, if we can accept either package names or paths, we leave the door open for fs-based or name-based grouping, and maintain symmetry with the npm explain command. (Speaking of which, it'd be really nice to be able to do npm update node_modules/foo, without updating every instance of foo in the tree.)