Sharing code using a setup.py

A 101 on how to share code across a Python project

Photo by AbsolutVision on Unsplash

You have the following project structure:

├── src  ├── common
├── __init__.py
└── utils.py
├── config
└── requirements.txt
├── project1
├── __init__.py
└── main.py
├── project2
├── __init__.py
└── main.py
├── README.md

Where both sub-projects <project1> and <project2> have a main.py file that use similar code.

You (want to) store this shared code inside a utils.py file that sits in a third directory <common>.

To use any function stored in utils.py you attempt to write the following import statement into the main.py files of <project1> and <project2>:

from common.utils import <function1>, ...

and you get the following ModuleNotFoundError:

ModuleNotFoundError: No module named 'common'

You then attempt to modify the import statement to:

from .utils import <function1>

or

from ..utils import <function1>

returning an ImportError:

ImportError: attempted relative import with no known parent package

In this article, I will explain why you receive these errors and how to avoid this issue by using a setup.py file that creates a Python package.

Firstly…

What are the Errors?

ModuleNotFoundError: No module named ‘common’

This error occurs when you are trying to import a Python package/module that is not installed in your (virtual) environment. If you run the command:

pip list

in your terminal (mac) or command line prompt (windows) the module that you are trying to install will not be in the list outputted to the screen. For example, running the pip listcommand in my virtual environment (venv) after receiving the ModuleNotFoundError I get:

Package        Version
-------------- -------
pip 21.3.1
setuptools 59.0.1

This does not contain any reference to the utils.py inside the <common> package folder.

ImportError: attempted relative import with no known parent package

Relative import errors arise when you attempt to import a module from a parent package but Python has a different parent package stored in __main__. Essentially, you are telling Python to import from a package that it is not aware of.

A great explanation of how Python stores information in its __main__ variable can be found in the [Script vs. Module] StackOverflow answer here.

Note: Relative imports are not pythonic and should be avoided if possible to avoid confusion and improve the readability of your code.

Some possible non-pythonic solutions you may have tried…

1. sys.path()

You can often bandage both of these errors by adding the desired parent path to the sys.path variable before the erroring imports. For example, adding the following lines into the main.py file:

However, in general, I would suggest avoiding this approach for the following reasons:

  • messy: It is messy.
  • Error Propagation: It needs to be coded properly to avoid errors on re-use by others due to different folder locations.
  • Code Duplication: It needs to be added to each .py file with the import errors.

2. Restructuring Project

You can restructure the project and have each sub-project’s main.py file at the root, enabling the imports to follow the hierarchical directory structure downwards.

Two key reasons to avoid adopting this solution:

  • messy: As your project grows, this approach will become disorganized, unclear, and messy.
  • Customization: It doesn’t allow you to structure your project in a readable and unique way.
  • Name: You are not allowed name files the same as they are all in the same directory meaning you will have to conjure up unique names such as, main1.py and main2.py.

setup.py to the rescue…

A good rule of thumb is that if you have a lot of code in a project that can be reused by many modules, make it into a Python package.

What is a Python Package?

A folder that contains an __init__.py will be recognised by Python as a package. A package folder usually contains multiple modules.

Packages that are installed in your (virtual) environment can be identified using pip list command.

Setting up the setup.py

  • A setup.py file provides information on how to create a custom Python package. It leverages Python’s setuptools library and a basic file looks something like this:
basic setup.py file

The main three kwargs of setup() are:

  • name — The name of your Python package.
  • install_requires — Points the setup.py at a requirements.txt file that contains the Python libraries that are needed by the Python modules that are going to be a part of the package. Lines 3 and 4 let the setup.py read in the package required Python packages from a specified location.
  • packages — The packages that you want setup.py to include in the custom Python package. You can use setuptools.find_packages() to locate all of the packages in your project or enter them manually. find_packages() will identify all of the folders that contain a __init__.py. For the example project structure, running find_packages() returns the following as packages (as they are folders that contain an __init__.py)
['common', 'project1', 'project2']
  • We only want to package the code inside <common> and so we add project1 and project2 to the exclude kwarg:
find_packages(exclude=["project1", "project2"])

Package Installation

Considering the project setup in “The Problem” section of this article, the following is an updated structure when using the setup.py screenshotted above.

├── src  ├── common
├── __init__.py
└── utils.py
├── config
└── requirements.txt
├── project1
├── __init__.py
└── main.py
├── project2
├── __init__.py
└── main.py
├── setup.py <--- Added in setup.py to the /src folder├── README.md

Note: The setup.py file needs to be in the location of the packages that are being used to build the custom package (in the setup() packages kwarg).

To create the package, you need to go to the parent folder dir of the setup.py file and run the following command:

pip install -e ./<root of setup.py dir>

For the example project in this article, you would run pip install -e ./src:

terminal stdout when running pip install -e ./src

The -e Runs the package installing in editable mode (dynamically) which automatically detects any changes you make to the code when developing, avoiding the need to continually re-install the package.

Once installed, you can check that it was successfully installed by re-running:

pip list

You should now see your package as well as those listed in the requirements.txt in the listed packages:

Package        Version Editable project location
-------------- ------- -----------------------------
DateTime 4.4
pip 21.3.1
pytz 2022.1
setuptools 59.0.1
simple-package 1.0.0 /Users/danielseal/git_local/Sharing_Code_Example/src
zope.interface 5.4.0

Notice the Editable project location part of the pip list output.

If you want to hard install the package (statically) ie for a stable version where you are not going to make any changes you can simply drop the -e and run pip install ./src:

terminal stdout when running pip install ./src

This installation is more explicit, returning the following output after running pip list:

Package        Version
-------------- -------
DateTime 4.4
pip 21.3.1
pytz 2022.1
setuptools 59.0.1
simple-package 1.0.0 <--- this is the custom package
zope.interface 5.4.0
wheel 0.37.1. <--- see 2nd note below

Dropping the -e will also build a wheel for the packing in your project in a /build folder as you are not in editable mode.

Note:

  • Both static and dynamic installations will add a .egg-info file into your project.
  • before running pip install ./src it is advised to install the wheel package using pip install wheel to avoid a warning that you do not have wheel installed.
Project structure after running pip intall ./src (dropped the -e to show the explicit install).

At this step, you have successfully installed a custom Python package.

Now when you run the previously erroring imports, you should not get any errors…

stdout when running project1/main.py and project2/main.py after package installation.

Comments

  • Warning: If you installed the packaged statically (without adding -ein install command) you will need to re-run the installation command, adding --upgrade or -uat the end, if you make changes to the code inside the package folder you are installing ( <common> for this example):
pip install ./src --upgrade
  • Editable: As mentioned, installing the package using pip install -e ./src will install the package in editable mode. It will not add a /build folder to the project
  • Version Control: If you have implemented a new feature or a fix to the Python package, it is advised to update the version kwarg in setup() to promote good version control.

This StackOverflow feed on updating a local Python package with pip explains the use of -u (upgrade) and -e (editable) really well.

Leave a Comment