Nimble RFC for project local dependencies

README.md

Following is the RFC for providing project local dependencies in Nimble. This is being tracked in issue 131 and discussed in the Nim forum.

Definitions:

• local deps mode - this RFC to enable project local dependencies where $prj/nimbledeps is used by Nimble on a per project basis
• user deps mode - current Nimble behavior of using ~/.nimble since dependencies are usable across projects for a specific user
• global deps mode - tracked in issue 80 where a system-wide folder can be used to store dependencies across users and projects. It is not covered in this RFC.

Motivations:

• The main value of local deps mode is to provide dependency isolation from other projects. Isolation simply makes development easier and avoids any conflicts caused by mixing dependencies across projects.
• local deps mode is simply a mode of operation during development and is not expected to be checked into source control. The intention is not to push vendoring or an alternative for lock files. It is not aimed at solving any distribution related challenges or enabling reproducible builds.
• If a user wants local deps mode for a particular project, it implies they want complete isolation. As a result, Nimble will no longer consider user or global dependencies and commands will all act as if $prj/nimbledeps is the only directory Nimble uses.
• Once a working configuration is reached regardless of deps mode, the user could then generate a lock file that improves on distribution. That design will be discussed separately and not distract this RFC.
• This RFC mainly describes Nimble behavior at this time. The changes required for Nim to understand local deps mode will be addressed later.

All behaviors below that drive actual code changes are highlighted with *.

Behavior:

• All nimble commands will highlight when in local deps mode so that it is amply clear to the user *
• An explicit --nimbleDir:xxx overrides local deps mode.
• If $prj/nimbledeps directory exists, set local deps mode by forcing nimbleDir = $prj/nimbledeps *
• Else, continue in default user deps mode

Command-line behavior in local deps mode:

• nimble install
  • Install all the project's dependencies into $prj/nimbledeps
  • Build binaries if applicable
  • Do not copy the project itself into $prj/nimbledeps - inform user *
• nimble install -d | --depsOnly - install all the project's dependencies into $prj/nimbledeps
• nimble install url|pkg - install dependency into $prj/nimbledeps
• nimble uninstall pkg - remove pkg from $prj/nimbledeps
• nimble uninstall pkg -i - remove pkg and packages that depend on it from $prj/nimbledeps
• nimble build - install deps into $prj/nimbledeps in the processDeps step
• nimble develop
  • Install all deps into $prj/nimbledeps in the processDeps step
  • Do not create any links to itself it in $prj/nimbledeps - inform user *
• nimble develop prj2
  • Clone prj2 into sub-directory within prj1 and add a link into $prj/nimbledeps
• nimble develop prj2 --nimbleDir:$prj1/nimbledeps
  • Clone prj2 into directory outside prj1 and link into $prj1/nimbledeps
• To use an existing project $prj2 as a dependency in local deps mode project $prj1:
  • cd $prj2
  • nimble develop --nimbleDir:$prj1/nimbledeps

local deps mode is a mode of operation during development and not for influencing distribution. While it is not recommended, users can save local deps mode in source control by:

• Checking in $prj/nimbledeps/empty.txt
• Checking in $prj/nimbledeps/* - vendoring or poor man's lock files
• Need to verify that nimble install of such a project does not break - Nimble should not get confused what the true nimbleDir is when processing such deps

No changes:

• All commands will continue to run like they do today when in user deps mode
• Dependencies will need to be added to the .nimble file in the requires section per usual to get installed from scratch regardless of deps mode

Caveats:

• Checking in $prj/nimbledeps/* into source control will work as expected for the project when checked back out but if the project is itself installed by a parent project, these checked-in deps should not be inherited. The dependencies will get pulled from ~/.nimble or $prj/nimbledeps of that parent project depending on its configuration. This is similar to how lock files work for libraries.
• If a project with local deps has a dep which builds binaries, it will install into $prj/nimbledeps which means the binaries will not be in ~/.nimble/bin, typically in $PATH. User is expected to add $prj/nimbledeps/bin to the $PATH.

If uninstall isn't going to by default remove things that depend on the uninstall target, it should at least warn the user, or maybe prompt for permission to remove the orphaned packages. Also, this behavior suggests the need for a command to remove or list orphaned packages. Personally, I'd prefer if it just logged notice and went ahead and removed the orphans by default, and a special flag was required to keep them.

It seems @disruptek is agreeing with me a lot here. You're wanting to add a lot of implicit behaviour before it's even obvious that this feature is going to work.

My main question is:

• What is the aim of this feature? Is it to make development easier or is it to be a replacement for lock files?

All that needs to be done is to change the name from deps to nimbledeps in order to avoid surprises. Consider that many know how Nimble is setup and so if the spec amounts to

If proj/nimbledeps exists, this is used instead of $nimbledir for every command.

Then it's super easy to understand and predict what Nimble does.

What is the aim of this feature? Is it to make development easier or is it to be a replacement for lock files?

It has been answered already, multiple times. It's to avoid bugs when nim c picks the wrong version of a package for a build. It should make lock files easier to implement / redundant but that's not the main motivation.

There needs to be some way for nimble install to do something install-like when run from localdeps. I don't really care what it does, but it should not be identical to nimble build.

Replies to some of the comments here.

At this point, I think adding any behavior that isn't explicit is a mistake. - @disruptek

This is being addressed in two ways - one is for nimble to be vocal about local deps mode in every command so that user knows where deps are being installed and coming from.

Second, local deps is opt in - you need to do something for it to be in effect. If you checkout someone else's project that uses local deps, nimble will remind you as noted above.

If uninstall isn't going to by default remove things that depend on the uninstall target, it should at least warn the user, or maybe prompt for permission to remove the orphaned packages. - @matt2000

nimble uninstall of a package that other packages depend on fails altogether and displays the list. You need to use -i to force remove it and all the others too.

If proj/nimbledeps exists, this is used instead of $nimbledir for every command. Then it's super easy to understand and predict what Nimble does. - @Araq

As I have noted above, nimble install and nimble develop can be made to behave in a variety of ways depending on this change. Some of them are not compatible with ~/.nimble at all which means a project with local deps cannot then be used by other global projects on the system. This is highlighted by @disruptek's last comment. Maybe that's okay but it isn't obvious. We need a clear definition of how things change with this feature.

There needs to be some way for nimble install to do something install-like when run from localdeps. - @disruptek

Whatever it is, it needs to be obvious and consistent. If you want to install or link (develop) in ~/.nimble then deps also need to be installed in ~/.nimble. It could mean that these commands do both project local and ~/.nimble management simultaneously but seems wasteful and confusing. Maybe there should be a flag to force ~/.nimble when local deps are in effect. Perhaps a --user or something. This seems more sensible since it will be a less frequently used and could do fine with requiring the user to be deliberate.

I don't want to mix up --global in this proposal cause there's a different issue for that.

What is the aim of this feature? Is it to make development easier or is it to be a replacement for lock files? - @dom96

Replying to this separately. I think this is a legitimate question.

npm uses project local deps by default. You need to specify -g | --global to be explicit about using the global repo. Nimble doesn't have this -g feature (being tracked in issue 80). It instead uses a per-user global. Now this is definitely cooler since the user is able to manage this repo in any way he chooses unlike npm's global which requires admin/root. pip allows a global location that needs admin as well as a per user site-packages but resorts to virtual envs to provide isolation.

The problem local deps alleviate is that a project only has real control over its immediate dependencies. If a dependency pulls in additional dependencies, there is a likelihood of packages conflicts causing issues. Isolating each project's dependencies into a local dep directory definitely reduces these conflicts. It does not completely eliminate them since A -> B -> C@v1 and A -> D -> C@v2 is still possible but at least you can manage that for a specific project without having to worry about other stuff in ~/.nimble.

So in some sense, yes, local deps are for making development easier by providing project isolation. --nimbleDir and --path can be used today but requires a bunch of manual work. This proposal is just to make this easier. It is not a supplement for lock files but perhaps a foundation. There is also the question of how to inform Nim of local deps and that will be the next step of this proposal.

Lock files solve a distribution issue so that other users can achieve reproducible builds. Once a user has a working configuration, ~/.nimble based or local deps based, he can generate and check in a lock file. That design should be discussed separately and not distract this proposal. There are other steps as well to make life better such as nimble install adding entries into .nimble and perhaps others.

I'm talking only about nimble behavior at this point, not the compiler. Nimble passes --noNimblePath to avoid ~/.nimble with the compiler.

We will address Nim behavior next.

As I have noted above, nimble install and nimble develop can be made to behave in a variety of ways depending on this change.

Ok, tbh I was confused about nimble install vs nimble install url. IMHO nimble install doesn't have to work with nimbledeps at all for now, later on it can simply copy the project itself to $nimbledir and its local dependencies to $nimbledir too (checking for duplicated directories). The same applies to nimble develop.

I've made multiple changes to the gist:

• Added definitions and motivations section to address some questions
• Added -u | --user flag to break out of local deps mode
• Added ability to link projects in local deps mode

My $0.02 of bikeshedding: instead of nimbledeps/ name the local dir nimble/, which is more in line with ~/.nimble/

The problem local deps alleviate is that a project only has real control over its immediate dependencies. If a dependency pulls in additional dependencies, there is a likelihood of packages conflicts causing issues. Isolating each project's dependencies into a local dep directory definitely reduces these conflicts.

I still do not understand. When you run nimble anytask, nimble will actually add to the path only the project dependencies and their transititive dependencies. So the project is already isolated.

The difference is that presently, the isolation is only "virtual": actually all dependencies reside on disk in the ~/.nimble directory, and nimble takes care of crafting the path so that - from the point of view of the project - only the relevant dependencies at the correct version can be seen.

In a way, what nimble does is just an optimization over the current proposal - after all, packages at the same version can be shared. This is quite common: for instance all dependency managers on the JVM rely on a shared folder like ~/.ivy2.

The only issue is that nim is not aware of the way nimble tracks dependencies (this is why we have two different tools after all), so running nim c will select the wrong dependencies.

Is my understanding correct or there is more to this proposal?

When you run nimble anytask, nimble will actually add to the path only the project dependencies and their transititive dependencies. So the project is already isolated. - @andreaferretti

This isn't totally true. Here's the scenario we are improving with local deps mode:

• User is working on project 1 and adds / removes / upgrades / downgrades dependencies till project works correctly
• User switches to to project 2 and works on it until it works correctly
• User resumes working on project 1 and it is broken since some dep changes in project 2 affected project 1

Nimble simply picks the latest unless the dep version is fixed in the requires section. And you don't typically hard code versions since it is too much work to maintain manually. We also don't have lock files yet which automates this process. I believe Nim does the same as well with picking latest.

Local deps mode is basically making it possible to develop multiple projects without interference from each other. In theory, lock files solve this once they are created but that's a much bigger lift to implement and you need to get to the point where a lock file can be created in the first place. Isolation just gets you there sooner.

In theory, lock files solve this once they are created but that's a much bigger lift to implement and you need to get to the point where a lock file can be created in the first place. Isolation just gets you there sooner.

Ok, so this is an alternative to lock files.

I don't know what is easier to implement - I would have assumed lock files were easier. I mean, as soon as you run nimble anything, nimble downloads your dependencies, at specific version, and adds a long string of intructions --path:~/.nimble/pkgs/dep1/v1, --path:~/.nimble/pkgs/dep2/v2, and so on. It should be trivial to just dump this list to a file, and for later invocations use the same path strings.

Whereas all the dance between project and user configuration seems more difficult to implement. But again, I do not know much about nimble internals.

I may be missing something, possibly implementing lock files requires some more subtlety? :-?

I don't want to call this an alternative but a foundation for lock files. It will make dev. and dependency debugging easier. Lock files will be easier to build on this foundation. I don't think the code changes for this RFC are very drastic. Lot of the functionality can be used as is including leveraging --nimbleDir. It is just a question of detecting, notifying and documenting.

Whereas with lock files, there's much more nuance on user workflow and impact on commands. I see that npm ran into some confusion while implementing lock files. It might be easy too but I cannot claim to understand lock files end to end, let alone how nimble will implement it. I know others are looking into it as well which is why I don't want to mix this with lock files.

Lock files will be easier to build on this foundation.

I don't see how that is the case. In what way does this provide a foundation for lock files? In my mind this is a replacement for them.

It has been answered already, multiple times. It's to avoid bugs when nim c picks the wrong version of a package for a build. It should make lock files easier to implement / redundant but that's not the main motivation.

Implementing this won't solve these bugs. It'll just add another way to manage dependencies with Nimble and yet another special edge case for Nim's package lookup rules. I'd rather go the other way and make nim print an error whenever you try to import a Nimble package telling you to run nimble c instead.

Your argument will be "but what about nimsuggest?". Just make it call nimble to get the paths, it's an IDE tool, IDEs usually call the package managers anyway.

In what way does this provide a foundation for lock files? In my mind this is a replacement for them. - @dom96

I have explained this in the Motivations section in the proposal. local deps mode is for project isolation. I have given an example of how that's of value in this comment.

Once you have a working configuration, you can then create/update the lock file which can then be used for distribution. That's why it is perhaps a foundation.

Your argument will be "but what about nimsuggest?". Just make it call nimble to get the paths, it's an IDE tool, IDEs usually call the package managers anyway.

nim check is also affected and so nim calls into nimble which calls into nim for .nimble file evaluation... But you don't listen, so these discussions are futile.

nim check is also affected

That could be subsumed by nimble check. On the other hand, I just tried to nimble check a project of mine and it just spits out Success: $PROJECT is valid! without actually checking anything, so if it is meant to be a drop-in replacement it doesn't work so well...

Nim is a set of tools around nim.cfg, the fact that it's nim doc and nim check and yet nimsuggest is more of an implementation detail. So Nimble should edit nim.cfg in order to play nice. It's the best solution by far. But it's all been discussed to death already, so I'm out.

Usually nim.cfg contains general configuration and is committed in version control. Nimble could edit nim.cfg with specific paths, but then the result would only work on the local machine

1. We could come up with nimble.paths that is taken into account by Nim's tooling. I don't like it as there is no reason nim.cfg wouldn't work just as well.
2. Relative paths exist.

The last few comments are around how Nimble will communicate path info to Nim. This comment is most appropriate for that.

Making nimble check and nimble doc work is my immediate thought too. I do listen to you @Araq, the main reason I oppose having Nimble edit the nim.cfg file is because this will always require two steps:

• Edit project.nimble file to add a dependency
• Run nimble <something> (some nimble command that will change the nim.cfg file)
• Now nim c works...

This will be a PITA which is why you might as well just use Nimble for every Nim subcommand. Maybe I'm missing something though, so please let me know if that is the case.

We don't know if it really will be a PITA. If I add a dependency to my .nimble file, I already need to do nimble bump (which really should be a thing...) or manually git tag it. Not immediately afterwards but soon enough. So already I cannot "just" edit my .nimble file and be done. But let's assume you're right, then we can make nim (including nimsuggest, nim doc etc etc) evaluate the .nimble file, it's already what it does, Nimble delegates this task to Nim already.

If I add a dependency to my .nimble file, I already need to do nimble bump (which really should be a thing...) or manually git tag it. Not immediately afterwards but soon enough.

Huh? You only need to do this when releasing a new version, which shouldn't happen often at all.

Likewise adding/removing dependencies doesn't happen all that often in practice.

Updated this RFC - removed --user since local deps mode is all about isolation. Adding support to break out is unnecessary complexity. This greatly simplifies this RFC.