About imports from a local python module on Windows using conda

python_module_import.md

If one wants to import from a python module or a package that is stored or is being developed locally to other local module to have no import problems and not to have to play manually with PATH or PYTHONPATH or deal with absolute/relative import issues one should install it. These are my notes on how to do it successfully on Windows using conda environment.

The problem:

To import local python module to a sibling or a totally different (unrelated) directory.

To set up a drama scene:

There are two modules: module_a and module_b. Both modules are stored locally, but in different, unrelated directories. One wants to import a some_function() from module_a to module_b. The drama takes place on Windows operating system, also a conda environment is to be used.

Solution:

In short: one needs to install module_a locally, in development (editable) mode: pip install -e. To do so:

1. Create a virtual environment, for sanity reasons, install pip in it (important!) and activate it.
   In my case it's a conda environment.
   
   Important note for conda environments: conda and pip can play together but with a caution. One should remember that all pip installs should be after all the conda packages installs are made. If one has to conda install in an environment that has some pip installs already in it it is better to (1) remove whole conda environment and (2) re-create it from the scratch, including this new conda package, and then do the neccessary pip installs.
   
   Documentation on conda and pip: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#pip-in-env
   
   Also see in: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html:
   

   It is best to install all packages at once, so that all of the dependencies are installed at the same time.(...)
   To gain the benefits of conda integration, be sure to install pip inside the currently active conda environment and then install packages with that instance of pip. The command conda list shows packages installed this way, with a label showing that they were installed with pip.
   

2. Prepare a proper directory structure for module_a.
   I used a src-layout: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#src-layout.
3. To be able to install the module_a as a package, create suitable __init__.py and pyproject.toml and/or setup.cfg, setup.py files.
   Package name for this use case would be modulea.
4. EDIT: Before taking this step check the UPDATE below this numbered list - alternative is possible.
   
   Important for Windows: Run Powershell (preferably Anaconda Powershell in case of using conda) as administrator.
   
   If you don't instead of installing in conda environment pip could rather install the package somewhere else, f.e. in base conda environment or even in your global python interpreter. You will know that such event took place, when after pip install -e you get an info: Defaulting to user installation because normal site-packages is not writeable. This message indicates that pip could not write to conda environment directory.
5. In Powershell: go to the directory where those previously (pt.3.) created package config files are stored.
6. Check if you are using pip instance installed in your virtual environment. It's crucial for conda environment (see: https://bitsofanalytics.org/posts/pip-conda-local-dev/pip_conda_local_dev).
   
   Commands to check it: for UNIX/Linux it would be which pip, for Windows Powershell it is where.exe pip. As a result you will get full path to pip.
   
   Confirm that it is in your virtual environment's directory. If it is so, copy this full path.
7. In Powershell: paste that copied path and then
   full\path\to\your\conda\env\pip.exe install -e .
   
   (note: this dot at the end of command is important! and indicates that you are installing the project configured in the directory you are in (pt. 5 above))
8. Successfull installation message should be similar to:
   

   Obtaining file:///full/dir/to/package/modulea
     Preparing metadata (setup.py) ... done
   Installing collected packages: modulea
     Running setup.py develop for modulea
   Successfully installed modulea-0.1.0
   

   Instead of modulea there would be your package name of course.
   
9. To confirm that the package was installed in the conda environment not somewhere else, in Powershell:
   conda list --> it will list all the packages installed in the conda environment. Your package should be on that list.
   pip list --> it will list all the packages installed in the environment by pip. The package should be there too.
   
10. In the package\src directory there will also be new folder created, which name is: package_name.egg-info (f.e. modulea.egg-info in this use case).

UPDATE - alternative possibility:

Instead of running Powershell as administrator I managed to install succesfully, without the Defaulting to user installation because normal site-packages is not writeable issue (see pt. 4 above), by running pip via conda run as follows:

Go through points 1 - 3 and 5 - 6 as stated above. Then in VSCode Powershell terminal or other Powershell (preferably Anaconda Powershell), with your conda environment activated:

7. conda run -n yourEnvName full\path\to\your\conda\env\pip.exe install -e .

It should behave exactly as stated in points 8 - 10.

Now, when using this virtual environment and its python's interpreter you can simply import modulea or from modulea import some_function to any python module easily!

Note on using pip install.

Take into consideration, that it is oftes argued, that using python -m pip install package is batter / safer than plain pip install package or pip3 install package. More on this:
https://snarky.ca/why-you-should-use-python-m-pip/
https://stackoverflow.com/questions/25749621/whats-the-difference-between-pip-install-and-python-m-pip-install

Used software info:

Windows 11
python 3.12
pip 23.3
conda 23.9
VS Code 1.85.1 (user setup)

Additional reference:

About python imports, modules/packages and virtual environments.

Documentation:

https://packaging.python.org/en/latest/tutorials/installing-packages/
https://pip.pypa.io/en/stable/topics/local-project-installs/
https://setuptools.pypa.io/en/latest/userguide/quickstart.html

Stackoverflow:

https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time
https://stackoverflow.com/questions/1896918/running-unittest-with-typical-test-directory-structure
https://stackoverflow.com/a/50193944/19716626

Others:

https://realpython.com/python-import/#create-and-install-a-local-package
https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure
https://snarky.ca/how-virtual-environments-work/

Very important here from Bits of Analytics:

https://bitsofanalytics.org/posts/pip-conda-local-dev/pip_conda_local_dev