Python project setup, packaging, and documentation for relatively new users.

python_project_packaging.md

Notes to create a python package and publish the documentation using Sphinx and ReadTheDocs

Create a project folder

I use Linux with Bash Shell, so this is the general meaning of the command prompt. Please refer to it if you are confused as to which folder to run the commands from.

[username@localhost currentfolder]$ 

So, for example, I have a folder called repos for saving my projects. I will create a project folder called pygeospt in repos. I like to organize the packages according to their names.

[username@localhost repos]$ mkdir pygeospt 

Change into the project folder, pygeospt, and create a package folder of the same name pygeospt. You may choose to do things differently but for now let us stick to this.

I will add a README.md file and create two additional directories for tests and examples.

If you have a .venv virtual environment, pipreqs returns a bunch of packages. In such cases use --ignore .venv.

[username@localhost pygeospt]$ mkdir examples
[username@localhost pygeospt]$ mkdir tests
[username@localhost pygeospt]$ nano README.md 

Now the initial project tree look like this:

pygeospt/  <-- This is the project directory
├── docs
├── examples
├── pygeospt  <-- This is the package directory
├── README.md
└── tests

Now add your code inside the package directory pygeospt. Also, add __init__.py and __about__.py. I wrote my code inside spt.py.

./  <-- This is the package directory root.
├── __about__.py
├── __init__.py
└── spt.py

Inside the spt.py I have a class called SPT. So to be able to import the SPT class from the spt modulue as from pygeospt import SPT, put the following import in __init__.py file.

from .spt import SPT

Without the above import in the __init__.py file we would have to import the SPT class as from pygeospt.spt import SPT

Importing your own package while developing before publishing it in PyPI.

This is one of the things I could not figure out for a long time, until recently. How to import your own package for testing, and while still adding things. I had generally done this by adding the path to the package directory to the path as sys.path.append(module_path). But only recently I found out about local install using pip.

1. Make a virtural environtment .venv directory in the project folder, not the package folder. At least I have it set this way.

[username@localhost pygeospt]$ python -m venv .venv

2. Create a requirements.txt which has a list of all the package dependencies for your package. I will use pipreqs. Again from the project directory.

[username@localhost pygeospt]$ pipreqs ./ --ignore .venv

In case a requirements.txt already exists, use --force.

3. Activate the virtual environment

[username@localhost pygeospt]$ source .venv/bin/activate

4. Install the requirements, notice the change in the prompt. When you see that `(.venv) in your prompt, you are in the virtual environment.

(.venv) [username@localhost pygeospt]$ python -m pip install -r requirements.txt

5. Create a pyproject.toml file which should look similar to this:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
# name of the package
name="pygeospt"

# version
version="0.0.1"

# small description about the module
description="A minimal code for SPT correction."

keywords = ["Soil Mechanics", "Geotechnical Engineering", "SPT", "SPT Correction"]

# author name and email
authors = [
    {name = "Firstname Lastname", email = "author@gmail.com"},
]

license = {text = "GPLV3"}

classifiers = [
    "Programming Language :: Python :: 3",
    "Development Status :: 1 - Planning",
    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
    "Operating System :: OS Independent",
]

dependencies = ["numpy==1.24.2", "pyvstress==0.0.1"]

# [project.urls]
# homepage = "https://gitlab.com/"
# documentation = "https://l"
# repository = "https://"

[tool.setuptools.packages]
# Scan the project directory with the default parameters
find = {}

5. Install the package locally. This will install the package to the virtural environment. Any changes made to the source code will be reflected in the package.

(.venv) [username@localhost pygeospt]$ pip install -e .

Now your project tree should look like this:

./pygeospt
├── examples
├── pygeospt
├── pygeospt.egg-info
├── pyproject.toml
├── README.md
├── requirements.txt
├── tests
└── .venv

Build system

Creater a pyproject.toml file. Refer to Setuptools Quickstart Userguide

• Place a pyproject.toml in the root directory of the project. Check the file for this package in the project root directory.

• Install setuptools, and build using pip in the virtual environment .venv.

• Run python -m build in the package root directory. This will create .whl and .tar.gz in the /build directory for distribution.

Upload to an online repository

Use twine to upload your package to online repository

twine upload

To test your package you can upload to PyPI with twine

twine upload --repository dist/*

You will need to create a PyPI account before uploading.

Publishing

These notes are for creating the documentation using the docstrings with Sphinx and publish those in Read the docs. This is based on docstrings so please document your code accordingly.

1. Install sphinx and sphinx-rtd-theme, in the same environment in which you are building your package.

(.venv) [username@localhost pygeospt]$ pip install sphinx
(.venv) [username@localhost pygeospt]$ pip install sphinx-rtd-theme

2. Create docs directory in your package directory to hold the documentation files.

(.venv) [username@localhost pygeospt]$ mkdir docs

Your project tree will look as follows:

./pygeospt
├── docs
├── examples
├── pygeospt
├── pygeospt.egg-info
├── pyproject.toml
├── README.md
├── requirements.txt
├── tests
└── .venv

3. Run sphinx-quickstart in the docs folder. Provide the information and when asked about creating a separate folder for build type y.

(.venv) [username@localhost pygeospt]$ cd docs
(.venv) [username@localhost docs]$ sphinx-quickstart

The docs folder will have the following files and folders:

./docs
├── build
├── make.bat
├── Makefile
└── source

Within the source folder there will be two files:

conf.py
index.rst

4. Run the Sphinx API documentation command sphinx-apidoc to generate Sphinx source files.

(.venv) [username@localhost docs]$ sphinx-apidoc -o source/ ../pygeospt

pygeospt is the package folder. Sphinx will convert the docstrings in your code in that package directory to create the documentation.

5. Edit the conf.py file.

   • a. Change the html_theme to the one of your liking, html_theme = 'sphinx_rtd_theme.

   • b. For autocode generation from docstrings in the code, specifiy the directory of source code.

   import os
   import sys
   sys.path.insert(0, os.path.abspath('../../pygeospt/'))
   

   • c. Also add some extensions

   extensions = ['sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc']
   

At this point your docs folder should look similar to this:

./docs
├── build
├── make.bat
├── Makefile
└── source
    ├── conf.py
    ├── index.rst
    ├── modules.rst
    ├── pygeospt.rst
    ├── _static
    └── _templates

6. To build the documentation, html files in the build/ directory.

(.venv) [username@localhost docs]$ make html

7. To view the documentation in your browser

(.venv) [username@localhost docs]$ firefox ./build/html/index.html

Publishing your documentation on the ReadTheDocs

1. First you need to have an online git repository. There maybe otherway to do this but for now I know this particular way.

2. Create a account in ReadTheDocs and log into ReadTheDocs and click on Import Project. Fill out the fields. For the repository URL use https://gitlab.com/geosharma/pygeospt.git. This can be got from CLONE -> Clone with HTPPS URL.

3. Then in Projects -> Admin -> Integration, click on Add Integration and choose GitLab incoming webhook.

4. Then copy the URL starting with readthedocs.org/api/v2/webhook/...../...... and paste it into GitLab Project Settings -> Webhooks

5. Also copy the Secret token and paste it in Gitlab project Webhooks -> Secret token field.

References:

• Read the docs
• Sphinx-RTD-Tutorial
• A idiot's guide to Python documentation with Sphinx and ReadTheDocs
• Read the Docs VSC Integration
• Deplyoing a Sphinx project online -Making Readthedocs for a Python package -An introduction to Sphinx and Read the Docs for Technical Writers
• Python Packaging