Capturing Python Project Module Requirements

I began working on a python script and wanted to capture all the python modules required for the script.  Coming from npm, ruby and other languages with package managers I began searching for a way to do this.  Python is a bit unique from the other languages I mentioned in that to capture the project's modules you first create a python virtual environment.

Our project's dependencies can be kept in a file called requirements.txt.  Python's package manager, pip, can use that file to install modules:

pip install -r requirements.txt

If that file is packaged with your project you'll be able to pip install all the modules your project requires.

A requirements.txt file is very easy to write by hand but we should be scripting these things.  You can generate it using the command pip freeze but you'll get a list of all system installed packages.  We need to restrict the list to only our project's modules.  Enter python virtual environments.

Python virtual environments is documented very well.  Using virtual environments you'll be able to lock your project to a specific python version and install modules only for the project.  This will keep your system python clean and provide consistency for your project.

Use python virtual environments to create a python for your project.  Then switch to using your project's python, install a module and generate a requirements.txt file.

Capturing Python Project Module Requirements Step by Step

Step by step walk through to create a python project with its own environment, install a module and create a requirements.txt file.

Requirements

• Python3
• Internet access to download modules
• Linux/macOS shell. Maybe works on Windows Subsystem for Linux (WSL)?

Create the Environment

# create a project directory

mkdir google_search && cd google_search

# create a virtual environment. Note the directories created.
python3.7 -m venv  ./
ls

bin                     lib                   
include                 pyvenv.cfg              webmaster-quickstart.py

# my python before switching to the project's environment
which python
${HOME}/.pyenv/shims/python

# activate the project's python environment
source ./bin/activate

# my python after switching to the project's environment
which python

${HOME}/website/google_search/bin/python

Install and Inspect the Modules

# install a module
pip install google-api-python-client

# look at all these modules!
ls ./lib/python3.7/site-packages/
__pycache__                                    pip-19.2.3.dist-info
apiclient                                      pkg_resources
cachetools                                     pyasn1
cachetools-3.1.1.dist-info                     pyasn1-0.4.8.dist-info
easy_install.py                                pyasn1_modules
google                                         pyasn1_modules-0.2.7.dist-info
google_api_python_client-1.7.11-py3.7.egg-info rsa
google_auth-1.7.1-py3.6-nspkg.pth              rsa-4.0.dist-info
google_auth-1.7.1.dist-info                    setuptools
google_auth_httplib2-0.0.3.dist-info           setuptools-41.2.0.dist-info
google_auth_httplib2.py                        six-1.13.0.dist-info
googleapiclient                                six.py
httplib2                                       uritemplate
httplib2-0.14.0.dist-info                      uritemplate-3.0.0.dist-info
pip

Generate and Inspect requirements.txt

# generate a list of installed modules for our project
pip freeze > requirements.txt
cat requirements.txt
cachetools==3.1.1
google-api-python-client==1.7.11
google-auth==1.7.1
google-auth-httplib2==0.0.3
httplib2==0.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
rsa==4.0
six==1.13.0
uritemplate==3.0.0

Notes

You should add the directories bin, lib and include to your .gitignore.  These should not be included in revision control.