If you are aware of the idea of a virtual environment but you don't really understand the concept or you can't see how it applies to your circumstances, then this article is for you. Until quite recently, I was the same: the whole concept confused me and, to be honest, I didn't see the point.
I have played around with various forms of computer code in an amateur fashion for more years than I care to remember. A few months ago - during the early stages of the coronavirus lockdown in the UK - I rediscovered my love of programming, reaquainted myself with Python and made the decision that programming in one form or another needed to become a much bigger part of my life.
My Python journey led me to Django and it was only then that I fully appreciated the concept of a virtual environment and understood why every programming project should start with the creation of a one.
By writing this article I hope to demystify the concept for programmers who either don't get it or can't see the point because I've been that person too!
So, what is a virtual environment?
My son is 7 and he has just finished year 2 at school. The official end of term was just 2 or 3 weeks ago but he has spent only 2 days actually in school since March (4 months ago) because the school effectively closed due to covid-19. Those 2 days that he did spend in school were only possible because of a "bubble" system that the school introduced.
My son was in a "bubble" with 14 other pupils from his class. During those 2 days at school he only interracted with people in his bubble. They sat together in class, ate together, and had break time together, all without interracting with anybody who was not in their bubble. The bubble system aimed to remove the risk of cross-infection between bubbles.
Virtual environments are similar to the bubbles in my son's school. Your application only interracts with software packages in its own bubble. It is insulated from packages in any other bubble or, perhaps more importantly, packages at the system level.
What is the issue that virtual environments address?
Imagine you wrote a Python program some time ago, before you ever heard of virtual environments. To build the program you used a couple of third-party Python packages that you installed specifically to do the job. When you installed those packages you will have installed them at the system level. That's not a problem at that stage. You finished the program, it worked really well and you were pretty chuffed!
A little while later you start work on another Python program on the same machine. Your new program uses one of the packages that you used in the first program and you have read about some whizz-bang new features that the latest version of that package offers. You like to live on the cutting edge of technology so you update that package to the latest version. You implement those awesome new features into your new Python program and you are doubly chuffed when it's finished. Well done you!
A few days later you try to run your first Python program only to discover that the new version of that package that you installed (at the system level remember) no longer supports some of the features that your first program uses. Your first Python program no longer runs. How frustrating!
Imagine if that was a live application with multiple users who might even have paid you money for the privilege of using it. Somewhat more than frustrating!
So how do virtual environments help?
Imagine a different scenario where the first thing you did when you started work on Amazing-Python-Project-1 was to set up a virtual environment - a figurative bubble in which your application could live, insulated from other applications and installations on your machine - and you then installed the two python packages into that same virtual environment and not at a system level.
When you move on to Amazing-Python-Project-2, in this scenario you would again set up a dedicated virtual environment for that program, and install the packages, including the latest whizz bang version, into that virtual environment. Those package installatons have no effect on the first virtual environment.
The two programs, and all of the packages required to run them, co-exist independently in their respective bubbles. The updated packages in the second virtual environment have no impact whatsoever on the program and associated packages in the first virtual environment.
Amazing-Python-Project-1 & Amazing-Python-Project-2 can both still run on the same machine. Your users will be unaffected, remain blissfully happy and continue to hand over their cash.
That sounds pretty amazing! But ... how do I do it??
Confusion reigned once again for me at the point of implementaton. I had read a few different books and tutorials offering advice on setting up your Django application - remember, it was Django that got me embroiled in the whole virtual environment thing in the first place - and it turns out that there's more than one way to skin the virtual environment cat. The author of each book or tutorial seemed to favour a different method.
There are third-party "virtual environment management" packages that you have to install on your machine, counterintuitively at system level, or there is a package which has been part of the standard Python library since version 3.4.
After playing around with the different methods ... and getting thoroughly confused and frustrated in the process ... I decided if something is part of the standard Python library then it makes sense to use it.
It's a package called venv. If you too are new to virtual environments, my advice is just stick with that: why introduce further complication when all the functionality you need is already there within Python? You can always try out different methodologies when you're comfortable with the whole concept.
Implementation steps
Ok, we've decided on the tool for the lob: venv. The following steps show how to set up a Python virtual environment on a Mac or Linux in readiness for the start of a new Django project which will live in a new directory called my_project. The virtual environment also needs a name (because it gets its own dedicated directory); let's call it myvenv
In a Terminal window, navigate to the directory where you plan to put the my_project directory and type...
$ mkdir my_project
$ cd my_project
$ python3 -m venv myvenv
The first 2 lines should be familiar Bash commands. A little explanation is probably in order for line 3. Firstly, I have to add the number 3 to the end of Python because I am using a 6 year old Macbook came with Python version 2 installed as default. I always have to explicitly tell the Python interpreter that I want to use version 3.
If you habitually type python3 to activate the Python shell then you need to add the 3 here too. If not, then simply python -m venv myvenv will work.
venv is the name of the virtual environment package that comes pre-installed with Python since version 3.4 and myvenv is the name that we chose to give to the virtual environment.
When the command on line 3 above is run, a new directory called myvenv is created within the my_project directory.
The myvenv directory contains files that venv will use to manage your virtual environment and which will contain files for the packages that are subsequently installed into the virtual environment. That is how those packages are kept within the "bubble". Ingenious eh?
So that has created your new virtual environment called myvenv. Before we install any packages we have to activate it with
$ source myvenv/bin/activate
You will now notice that your Terminal prompt is prefixed with the name of your virtual environment in parentheses.
(myvenv)$
We can now install Django into our virtual environment
(myvenv)$ pip install Django~=3.0
After running that command look in myvenv/lib/python3.8/site-packages and you will see that the latest incarnation of Django version 3 has been installed. And it's safe within your virtual environment untouched by anything outside that "bubble".
You can also run the pip list command in the Terminal...
(myvenv)$ pip list
Package Version
---------- -------
asgiref 3.2.10
Django 3.1
pip 19.2.3
pytz 2020.1
setuptools 41.2.0
sqlparse 0.3.1
In the example shown here, I have installed Django 3.1 and there are some other default packages already installed.
Any other packages that you need for your program should be installed the same way - pip install... into the active virtual environment.
And now for the cherry on the top: requirements.txt. This is a simple text file that keeps track of the various packages that you have installed into the virtual environment. When your program is complete, requirements.txt will contain a blueprint of the specific versions of all of the packages that your program needs in order to run. You can therefore run your program in a virtual environment on another machine by passing that virtual environment a copy of requirements.txt to use as a recipe to concoct the precise mix of packages required. More on that later.
So how do you create the requirements.txt file?
There is another pip command: pip freeze. Running that in the Terminal will list all third-party packages and version numbers.
(myvenv)$ pip freeze
asgiref==3.2.10
Django==3.1
pytz==2020.1
sqlparse==0.3.1
All you need to do now is add a suffix to that same command to instruct it to write the list to file instead of to screen...
(myvenv)$ pip freeze > requirements.txt
You now have a file called requirements.txt that contains precisely the same list as was printed to screen by the pip freeze command without the suffix.
IMPORTANT When you run that command you need to make sure you are in the project root directory - i.e. my_project in this example - because requirements.txt will be created in whatever directory the pip freeze command was run in.
So there you have it. All of the packages that your program needs, safe within their own bubble which has a list of contents on the outside in the form of requirements.txt. Every time you install a new package to use in your program, run the command pip freeze > requirements.txt and your "contents list" is updated.
Take your new requirements.txt file for a test drive
You can really get a feel for how useful requirements.txt is by creating a new virtual environment in a new diectory - say my_other_project - by following the instructions above. You can give the virtual environment the same name - myvenv - if you like; that's up to you.
Then, before installing anything, copy the requirements.txt file that we created above in my_project into your my_other_project directory. It needs to sit in that root directory, alongside myvenv (assuming you chose the same virtual environment name).
Then, with your Terminal in the my_other_project directory, and the virtual environment activated, run...
(myvenv)$ pip install -r requirements.txt
Wait for the magic to happen, then run pip freeze...
(myvenv)$ pip freeze
asgiref==3.2.10
Django==3.1
pytz==2020.1
sqlparse==0.3.1
And you get exactly the same output as in the other project. The environment has been precisely recreated.
Magic eh?
When you have finished working on your project, use the command deactivate in the Terminal to ... well ... deactivate the virtual environment
(myvenv)$ deactivate
$
Conclusion
After reading this, I hope that you now have a better understanding of:
- virtual environments in general;
- the reason they are a good idea;
- and how to set one up for your next Python project.
Image credit: Photo by Lanju Fotografie on Unsplash
Top comments (0)