I began working on a python script and wanted to capture all the python modules required for the script. Coming from npm, ruby and other languages with package managers I began searching for a way to do this. Python is a bit unique from the other languages I mentioned in that to capture the project's modules you first create a python virtual environment.
Our project's dependencies can be kept in a file called requirements.txt. Python's package manager, pip, can use that file to install modules:
If that file is packaged with your project you'll be able to pip install all the modules your project requires.
A requirements.txt file is very easy to write by hand but we should be scripting these things. You can generate it using the command pip freeze but you'll get a list of all system installed packages. We need to restrict the list to only our project's modules. Enter python virtual environments.
Python virtual environments is documented very well. Using virtual environments you'll be able to lock your project to a specific python version and install modules only for the project. This will keep your system python clean and provide consistency for your project.
Use python virtual environments to create a python for your project. Then switch to using your project's python, install a module and generate a requirements.txt file.
# create a project directory
# create a virtual environment. Note the directories created.
python3.7 -m venv ./
ls
bin lib
include pyvenv.cfg webmaster-quickstart.py
# my python before switching to the project's environment
which python
${HOME}/.pyenv/shims/python
# activate the project's python environment
source ./bin/activate
# my python after switching to the project's environment
which python
# install a module
pip install google-api-python-client
# look at all these modules!
ls ./lib/python3.7/site-packages/
__pycache__ pip-19.2.3.dist-info
apiclient pkg_resources
cachetools pyasn1
cachetools-3.1.1.dist-info pyasn1-0.4.8.dist-info
easy_install.py pyasn1_modules
google pyasn1_modules-0.2.7.dist-info
google_api_python_client-1.7.11-py3.7.egg-info rsa
google_auth-1.7.1-py3.6-nspkg.pth rsa-4.0.dist-info
google_auth-1.7.1.dist-info setuptools
google_auth_httplib2-0.0.3.dist-info setuptools-41.2.0.dist-info
google_auth_httplib2.py six-1.13.0.dist-info
googleapiclient six.py
httplib2 uritemplate
httplib2-0.14.0.dist-info uritemplate-3.0.0.dist-info
pip
# generate a list of installed modules for our project
pip freeze > requirements.txt
cat requirements.txt
cachetools==3.1.1
google-api-python-client==1.7.11
google-auth==1.7.1
google-auth-httplib2==0.0.3
httplib2==0.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
rsa==4.0
six==1.13.0
uritemplate==3.0.0
Our project's dependencies can be kept in a file called requirements.txt. Python's package manager, pip, can use that file to install modules:
pip install -r requirements.txt
If that file is packaged with your project you'll be able to pip install all the modules your project requires.
A requirements.txt file is very easy to write by hand but we should be scripting these things. You can generate it using the command pip freeze but you'll get a list of all system installed packages. We need to restrict the list to only our project's modules. Enter python virtual environments.
Python virtual environments is documented very well. Using virtual environments you'll be able to lock your project to a specific python version and install modules only for the project. This will keep your system python clean and provide consistency for your project.
Use python virtual environments to create a python for your project. Then switch to using your project's python, install a module and generate a requirements.txt file.
Capturing Python Project Module Requirements Step by Step
Step by step walk through to create a python project with its own environment, install a module and create a requirements.txt file.Requirements
- Python3
- Internet access to download modules
- Linux/macOS shell. Maybe works on Windows Subsystem for Linux (WSL)?
Create the Environment
# create a project directory
mkdir google_search && cd google_search
# create a virtual environment. Note the directories created.
python3.7 -m venv ./
ls
bin lib
include pyvenv.cfg webmaster-quickstart.py
# my python before switching to the project's environment
which python
${HOME}/.pyenv/shims/python
# activate the project's python environment
source ./bin/activate
# my python after switching to the project's environment
which python
${HOME}/website/google_search/bin/python
Install and Inspect the Modules
# install a module
pip install google-api-python-client
# look at all these modules!
ls ./lib/python3.7/site-packages/
__pycache__ pip-19.2.3.dist-info
apiclient pkg_resources
cachetools pyasn1
cachetools-3.1.1.dist-info pyasn1-0.4.8.dist-info
easy_install.py pyasn1_modules
google pyasn1_modules-0.2.7.dist-info
google_api_python_client-1.7.11-py3.7.egg-info rsa
google_auth-1.7.1-py3.6-nspkg.pth rsa-4.0.dist-info
google_auth-1.7.1.dist-info setuptools
google_auth_httplib2-0.0.3.dist-info setuptools-41.2.0.dist-info
google_auth_httplib2.py six-1.13.0.dist-info
googleapiclient six.py
httplib2 uritemplate
httplib2-0.14.0.dist-info uritemplate-3.0.0.dist-info
pip
Generate and Inspect requirements.txt
# generate a list of installed modules for our project
pip freeze > requirements.txt
cat requirements.txt
cachetools==3.1.1
google-api-python-client==1.7.11
google-auth==1.7.1
google-auth-httplib2==0.0.3
httplib2==0.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
rsa==4.0
six==1.13.0
uritemplate==3.0.0