Skyler Burger

Posted on Jun 12, 2020 • Edited on Jul 3, 2020

PVP - A Workflow for Python Projects

#python

I swear I already installed that package twice!!
Which of these environments belong to my project?!
Screw it, I'm installing everything globally!! 😈

Managing Python projects can be quite the task, especially if you have more than one on your machine. There are countless workflow approaches to managing your Python versions, environments, and project dependencies. This guide is an overview of one such approach that emphasizes explicit control of your Python versions and virtual environments. The tools we'll be using are Pyenv, Virtualenv, and Poetry, which is why I refer to this as the PVP workflow.

Requirements

To follow along with this guide you will need pyenv, pyenv-virtualenv, and poetry installed on your machine.

Pyenv Prerequisites

Required for pyenv to build Python interpreters from source.
Pyenv Installer (Recommended approach)

Will install pyenv as well as pyenv-virtualenv.
Poetry Installation Page

Be sure to check out everything poetry can do by skimming the docs.

If you are a Windows user utilizing Windows Subsystem for Linux, be sure that you are installing these tools on the subsystem rather than Windows.

Setting Up New Projects

This section of the guide focuses on quickly setting up a brand new Python project. Here we'll be ensuring we have the version of Python we want to use, creating a virtual environment for our project to play in, and installing project dependencies inside that environment.

Installing Python Versions
Creating Virtual Environments
Installing Project Dependencies
Git Started
New Project Summary

Installing Python Versions

Before creating a virtual environment and installing your project dependencies, we should make sure that the Python version you'd like to utilize is installed on your machine. We'll use Pyenv to install and manage different versions of Python. The following commands can be executed from any location on your file system.

Check the versions of Python currently available on your machine.
```
pyenv versions
```
This will show you a list of each version of Python that Pyenv has installed on your machine. If the version you want is listed you can skip down to the next section of this guide on Creating Virtual Environments.
Check the versions of Python that Pyenv is able to install on your machine.
```
pyenv install --list
```
This command will list out all of the versions of Python that are available for installation. The version you want is most likely one of the simple semantic versioning numbers at the very top of the list (such as 3.8.2).
Install your desired version of Python on your machine.
```
# Template:
pyenv install <python_version>

# Example:
pyenv install 3.8.2

# Verbose Example
pyenv install -v 3.8.2
```
This process can take about 5 mins so sit tight while it's working. If you want visual feedback on the installation process, add the optional -v flag for a verbose printout of the actions Pyenv is taking during installation.
Verify that the desired version was installed.
```
pyenv versions
```
This will list out all of the version of Python that Pyenv has installed on your machine. We should now see the version you just installed on the list.

Creating Virtual Environments

Now that we have our desired version of Python installed on our machine, let's use Virtualenv to create a virtual environment for our project to use. This virtual environment is like a playground containing your desired version of Python and any packages that you install for your project. Within this playground your project can play with reduced fear of interfering with other Python projects on your machine.

Create a directory for your project and move into it.

# Template
mkdir <project_name>
cd <project_name>

# Example
mdkir my-sweet-project
cd my-sweet-project

Create a virtual environment for your project and give it a name that matches your project.
```
# Template
pyenv virtualenv <python_version> <project_name>

# Example
pyenv virtualenv 3.8.2 my-sweet-project
```
Using this naming convention makes it easy to select the virtual environment containing your dependencies in IDEs like VS Code or Jupyter Lab since the virtual environment's name matches your project's name.

If you check pyenv versions after this step you will now see your new virtual environment is listed as an available version of Python.
Set your new virtual environment as the local version of Python.
```
# Template
pyenv local <virtual_environment_name>

# Example
pyenv local my-sweet-project
```
This command will ask Pyenv to create a .python-version file in the current directory. This file tells Pyenv that any time we are in this directory, the version of Python we want to use is located in our virtual environment. Now anytime we navigate to this directory, Pyenv quietly switches the Python version to the virtual environment automatically. Whenever we leave this directory, Pyenv will switch the Python version back to the system default, unless it encounters another .python-version file that specifies which version to use instead.

You can double-check python -V at this point to verify that the version of Python that is currently recognized matches the version you used to create the virtual environment.

Installing Project Dependencies

Now that we have a virtual environment utilizing our desired version of Python, we can start managing our new project's dependencies with Poetry.

Initialize your project.
```
poetry init

# Alternatively, if you want to skip the prompts use:
poetry init -n
```
Poetry will create a pyproject.toml file that is used to keep track of some basic project information as well as any dependencies you install for the project.
Install project dependencies.
```
# Template
poetry add <package_name_1> <package_name_2> <...>

# Example
poetry add requests jupyterlab requests
```
When Poetry installs these packages, it places them within the virtual environment that was set as the local version of Python. Poetry will also add the name of the package to your project's .pyproject.toml file as a dependency.
Activate Poetry's shell.
```
poetry shell
```
To use the dependencies that Poetry has installed, we need to activate Poetry's shell before executing our code.

Sometimes it can be unclear whether the shell is active or not. To verify if Poetry's shell is active you can check your environment variables with the following command, echo $POETRY_ACTIVE. If the Poetry shell is active it will return a 1, otherwise it will print a blank line to your terminal.

Git Started

Now that your project is all set up, you probably want to keep track of your work with a version control system. Here we'll be connecting our project to GitHub for safekeeping.

Navigate to the root directory of your project.
Create a few required files.
```
touch .gitignore README.md
```
At minimum, your README.md should have a top-level header with your project's name so it is rendered on your GitHub repository's page.

Be sure to fill in your .gitignore file with the appropriate file names, file extensions, and directories to be ignored. GitIgnore.io is a great .gitignore generator.
Initialize your project as a Git repository.
```
git init
```

Create your first commit.

git add .
git commit -m "<commit_message>"

Create a new, empty GitHub repository online.

Log in to GitHub and create a new repository. DO NOT create the repo with a license, README.md, or .gitignore file. Once the repo has been created you'll see a page with a bunch of commands you can copy/paste into your terminal to finish setting up the repo. We'll use these in the next step.
Create an upstream remote and push your work to GitHub.

To complete the set up we'll want the two commands listed on your GitHub repository's page under "... or push an existing repository from the command line". You can copy/paste those commands directly into your terminal or type them out if you can spot the HTTPS link to your repo (it ends in .git).
```
git remote add origin <link_from_github.git>
git push -u origin master
```
After this you're ready to proceed with your usual Git flow.

New Project Summary

Create Project Directory
```
mkdir <project_name>
cd <project_name>
```

Create Virtual Environment

pyenv virtualenv <python_version> <project_name>
pyenv local <virtual_environment_name>

Install Dependencies

poetry init -n
poetry add <package_name_1> <package_name_2> <etc>

Activate Poetry's Shell

poetry shell

# (optional) Check if the shell is active
echo $POETRY_ACTIVE

Git Started

# Add your README.md and .gitignore files, then
git init
git add .
git commit -m "<commit_message>"

# Create new, empty repo on GitHub, then
git remote add origin <link_from_github.git>
git push -u origin master

Working with Existing Projects

This section of the guide will focus on how to use the PVP workflow with existing projects. We'll start from the very beginning and assume you are cloning a repository from GitHub on your machine and then move forward from there.

Setting Up Existing Projects
Managing Project Dependencies
Removing Virtual Environments
Removing Python Versions

Setting Up Existing Projects

This setup process assumes that you are cloning a repository that utilized Poetry for dependency management. Mainly we want to make sure that the repository has a pyproject.toml and poetry.lock file which are used to notify Poetry which packages (and more specifically which versions) are required as dependencies for the project.

Clone the repository from GitHub to your macine.
```
git clone <link_from_github.git>
```
Check the required version of Python.
```
cd <repository_name>
cat pyproject.toml
```
The cat command will print out the contents of the pyproject.toml file. Under [tool.poetry.dependencies] you'll see a specification for which versions of Python will work with the project. The example below indicates that the project requires a version of Python that is 3.8 or higher.
```
# Partial example of cat output

[tool.poetry.dependencies]
python = "^3.8"
```
Check that you have the right version of Python installed.
```
pyenv versions
```
Use this command to see which versions of Python are available on your system. If you see a version that meets the project requirements you can move on to the next step.

If you do not see a version of Python that meets the project requirements, please follow the Installing Python Versions section to identify and install a version that is compatible with the project.

Create a virtual environment for the project.

# Template
pyenv virtualenv <python_version> <project_name>

# Example
pyenv virtualenv 3.8.2 my-sweet-project

Set your virtual environment as the local Python version.

# Template
pyenv local <virtual_environment_name>

# Example
pyenv local my-sweet-project

Install the project's dependencies.
```
poetry install
```
Poetry will now read through your pyproject.toml to see which dependencies are required by the project. If the project includes a poetry.lock, Poetry will also examine that file to see which versions of each dependency are required.
Shell up and get to work!
```
poetry shell
```

Managing Project Dependencies

Here are a few basic Poetry commands that will help you install, remove, and inspect dependencies for your projects.

Adding Project Dependencies

# Adding a single package
poetry add <package_name>

# Adding multiple packages
poetry add <package_name_1> <package_name_2> <...>

Removing Project Dependencies

# Remove a single package
poetry remove <package_name>

# Remove multiple packages
poetry remove <package_name_1> <package_name_2> <...>

Listing Installed Dependencies

poetry show

Removing Virtual Environments

When you are no longer examining or working on a project, you may want to perform a little clean up on your machine. Getting rid of the project is as simple as deleting the directory containing the project. It's only one additional step to remove the virtual environment and accompanying installed packages from your machine as well.

# Template
pyenv virtualenv-delete <virtual_environment_name>

# Example
pyenv virtualenv-delete my-sweet-project

You will be prompted to confirm that you want to delete your virtual environment by entering 'y' on the terminal.

Removing Python Versions

Have you installed an experimental build or moved on to the new best thing? At some point you'll probably want to remove a version of Python from your system. Pyenv brought Python into our machine, and so help us, it will take it out.

# Template
pyenv uninstall <python_version>

# Example
pyenv uninstall 3.8.2

Conclusion

This is simply one approach out of many towards managing all of the moving pieces associated with development on a Python project. Use aspects that you like, plug in tools that you prefer, and make it your own! There is no one correct answer and I would love to hear about workflows that others are using.

Top comments (2)

Michael Klengel • Feb 9 '21

Thx for the interesting article.

Why do you add requests twice (found in "2. Install project dependencies") and why don't you use poetry for the virtual environment management?

# Example
poetry add requests jupyterlab requests

Regards
Michael

Skyler Burger • Mar 4 '21

Hi Michael! Adding the 'requests' package twice is a typo, nice catch! I prefer using Pyenv for managing my virtual environments, I feel I have better control over determining the correct Python version and naming of the virtual environment. I'm not the biggest fan of having all of my environments labelled '.venv'.

DEV Community