Extras handling in micropip

notes.md

Explanation of micropip's extras handling

From my understanding, a transaction corresponds to a single micropip.install() command.

The method add_requirement_inner() gets called for each package that we will be installing as part of the current microppip transaction.

So, consider the following:

• pkga with optional dependency marker, all which will bring in packages, depa and depb
• pkgb with optional dependency marker, optional_feature which will bring in package, depc

Based on the above understanding and assumptions, as part of the change in the proposed pull request at pyodide/pyodide#2584, when we invoke, micropip.install(["pkg[all]", "pkgb[opt_feature]"]) which results in 4 calls to add_requirement_inner(). This is how the values of the key attributes/variables look like in each add_requirement_inner() call progressively:

1. Called with pkga, the values of key attributes are:

   • req.extras = extras: {'all'}
   • req.marker = None
   • self.ctx = {'implementation_name': 'cpython', 'implementation_version': '3.10.2', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.18.0-1-MANJARO ', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 9 07:56:17 UTC 2022', 'python_full_version': '3.10.2', 'platform_python_implementation': 'CPython', 'python _version': '3.10', 'sys_platform': 'linux'}
   • self.ctx_extras: []

2. Called with pkgb, the values of key attributes now are:

   • req.extras = extras: {'opt_feature'}
   • req.marker = None
   • self.ctx = {'implementation_name': 'cpython', 'implementation_version': '3.10.2', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.18.0-1-MANJARO ', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 9 07:56:17 UTC 2022', 'python_full_version': '3.10.2', 'platform_python_implementation': 'CPython', 'python _version': '3.10', 'sys_platform': 'linux'}
   • self.ctx_extras: [{'extra': 'all'}] # We populated this in the previous step

3. Called with depb, the values of key attributes now are:

   • req.extras = ()
   • req.marker = extra == "all" # this attribute is how we can find out that this package is being installed due to the all optional feature of one of the other packages
   • self.ctx = {'implementation_name': 'cpython', 'implementation_version': '3.10.2', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.18.0-1-MANJARO ', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 9 07:56:17 UTC 2022', 'python_full_version': '3.10.2', 'platform_python_implementation': 'CPython', 'python _version': '3.10', 'sys_platform': 'linux'}
   • self.ctx_extras: [{'extra': 'all'}, {'extra': 'opt_feature'}] # Now consists of both the extras of pkgaandpkgb`

4. Called with depa, the values of key attributes now are:

   • req.extras = ()
   • req.marker = extra == "all" # this attribute is how we can find out that this package is being installed due to the all optional feature of one of the other packages, note that this is same for depb and depa as they are both being installed as part of the same primary package's optional dependency
   • self.ctx = {'implementation_name': 'cpython', 'implementation_version': '3.10.2', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.18.0-1-MANJARO ', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 9 07:56:17 UTC 2022', 'python_full_version': '3.10.2', 'platform_python_implementation': 'CPython', 'python _version': '3.10', 'sys_platform': 'linux'}
   • self.ctx_extras: [{'extra': 'all'}, {'extra': 'opt_feature'}] # Now consists of both the extras of pkgaandpkgb`

5. Called with depa, the values of key attributes now are:

   • req.extras = ()
   • req.marker = extra == "opt_feature" # this attribute is how we can find out that this package is being installed due to the opt_feature optional feature of one of the other package
   • self.ctx = {'implementation_name': 'cpython', 'implementation_version': '3.10.2', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.18.0-1-MANJARO ', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 9 07:56:17 UTC 2022', 'python_full_version': '3.10.2', 'platform_python_implementation': 'CPython', 'python _version': '3.10', 'sys_platform': 'linux'}
   • self.ctx_extras: [{'extra': 'all'}, {'extra': 'opt_feature'}] # Now consists of both the extras of pkgaandpkgb`

Marker evaluation

Two key points from above which drove the implementation of the fix in the PR are:

• The marker attribute is set for the optional dependencies and the evaluation result determines whether the dependency will be installed or not
• The evaluation of the marker requires the extra key in self.ctx to have the right value. For example, for depa and depb, there must be a {'extra': 'all'} object in self.ctx.
• The extras attribute is only set for the primary packages and hence has to be available during the above evaluation
• Hence, we use the self.ctx_extras attribute to store all the extra values we come across during the transaction and attempt the marker evaluation for all of these values. If any of the evaluations return true we include the dependency.

if req.marker:
        def eval_marker(e: dict[str, str]) -> bool:
            self.ctx.update(e)
            # need the assertion here to make mypy happy:
            # https://github.com/python/mypy/issues/4805
            assert req.marker is not None
            return req.marker.evaluate(self.ctx)

    self.ctx.update({"extra": ""})
    if not req.marker.evaluate(self.ctx) and not any(
             [eval_marker(e) for e in self.ctx_extras]
    ):
        return

Why do we have both req.marker.evaluate(self.ctx) and eval_marker(e) for e in self.ctx_extras? The reason is the current package may have been brought into the transaction without any of the optional requirement specification, but has another marker, such as implementation_name. In this scenario, self.ctx_extras is empty and hence the eval_marker() function will not be called at all.