How to migrate Python 2 projects to 3 & some useful tips

Python 2.7 came to end of life in January 2020 and even before that lot of software migrations to Python 3 have been happening. However, I was a little late to the party and got to work on one such project in late 2020. So in this blog, I'll try and summarize what steps we followed and some of the tips which can be helpful for a fellow developer who may need to migrate a Python project in future.

Before we get started, let me also add a small disclaimer, that I'll not be touching upon the differences of Python 2 & 3, there is a lot of content available on the internet for the same so please feel free to check it out.

We'll be talking about the migration where we were not worried about Python2 compatibility because that was the case with us. However, I'll share some reference links for py2&3 compatible migrations as well. So let's get started and see what are the steps one can follow when migration to Python 3 is to be done. We migrated from Python 2.7 to 3.7 but the latest version is 3.9 so if you are confident enough to choose the latest Python version, you can do that as well.

Steps for migration

• The first step would be to understand some fundamental difference between Python 2 & 3, that will help fix issues
• Check the code coverage of the code and make sure that we have decent test coverage. The more the code coverage, the easier would be to migrate the code
• Set up a virtual environment for both Python2.7 and Python 3.7+ versions

Virtual Env for Python2

python2.7 -m  pip install virtualenv
python2.7 -m virtualenv py27
source py27/bin/activate

Virtual Env for Python3

python3.7 -m virtualenv py3
source py3/bin/activate

• Also, do create a separate branch for Python 3 codebase and maintain these two versions parallelly unless the full migration is completed

• The next steps shall be to figure out the dependencies of the codebase compatible with Python 3, for that I'd recommend using the following tool

  • canisepython3: This helps to find out which existing dependencies are compatible with python3
    

    pip install caniusepython3
    

  • pip-tools: When we are migrating and we may also worried about pinning the dependencies & sub-deps then pip-tools is very handy, do explore it
    

    pip install pip-tools
    

• Once dependencies are sorted then run the test cases and find out the number of failures or any major issues, keep a note of test case failures, that will help you estimate the work up to some level

• After this, we need to move to the conversion of the code, so we already have some auto conversions tools available like 2to3

• In our case we used a library called futurize which is also based on 2to3
  

  pip install future
  

• We can use below command to auto-convert the code to Python 3
  

  futurize --stage1 mypackage/**/*.py
  

• Once the code is auto-converted, run the test cases and check the number of failures and start fixing them

• Once the test cases are fixed, we can proceed with our functional testing, regression testing and CI pipeline upgrades

Note - While fixing the functional test cases, it will be handy to have both Python 2 & Python 3 version of apps running in two diff virtual envs, because then we can go back n forth between two versions and see the behaviour change if we are stuck with fixing a bug

Few Gotchas

As it's mentioned in futurize library's documentation that its auto conversion output may not be fully accurate and it has some issues and bugs as well, so it is important for us to review the automated fixes before merging the code, so from our experience few things can be taken care:

• Auto-conversion will end up adding list inbuilt function to all such cases where we might be using dict.iteritems() , after the conversion it becomes list(dict.items()) because iteritems returns a list in Python 2.7 and in Python 3 it returns <class 'dict_items'> . However, It is an unnecessary call because conversion to list takes time and you can still iterate through dict.items() the same way you do over the converted list
• Futurize can also add a lot of unused imports to your codebase, we can get rid of them if we are not worried about backward compatibility, but be mindful of not removing all unused import and do check and analyze the module before doing it
• If you might have used zip function in Python 2.7 and then tried converting it to dict then futurize might wrap zip with a list function call, you won't need that list() function call in most cases after migrating to Python 3

all_vals = dict(list(zip(array1, array2))) # Unnecessary list function call
all_vals = dict(zip(array1, array2)) # Correct

• Relative imports are no longer supported in Python 3. Any code that used relative imports in Python 2 now has to be modified to support absolute imports
• In Python2, CSV files should be opened in binary mode. but, in Python 3, the files are to be opened in the text mode

Common Error Fixes

So during the migration, we may end up fixing some very basic level errors which may get introduced because of futurize or because of the Python 2 & 3 differences. So I'm listing few of them which we faced and that might help you to quickly get rid of those -

• The old division would not work in Python3, and we can fix such cases like this

from past.utils import old_div
x = y / 1000 # Would give unexpected result in Python 3 and existing test case may fail because of this
x = old_div(y, 1000) # This would result as Python 2

• In all exception handling cases, we might end up getting this common error, wherein Python 2 we had an attribute called message for the exception class and python 3 doesn't have it

# Old case
except Exception as e:
    raise CustomError(message="message: {}".format(e.message))

# How to fix
except Exception as e:
    raise CustomError(message="message: {}".format(e))

• For the same exception handling errors when we remove message attribute from the e variable than in some case we might face below error where we will do concat operation on e and some string, that will cause a failure

# Failure case
try:
    1/0
except Exception as e:
    print("custom_message"+e) # This would give an error
# How to fix
try:
    1/0
except Exception as e:
    print("custom_message"+str(e))

• hasattr(dict, 'iteritems') would return False in Python 3, to fix please use this

hasattr(dict,'items’) 

• In Python 2 we used to use string.replace() as well in somecases to replace string within a string, that will not work in Python 3, hence fix it like this -

a = "asdfghj"
string.replace(a, "asd", "xcv") # Will not work
a.replace("asd", "xcv") # correct

• cmp() function which used to be available in Python 2 is not available in Python 3, so you might need to implement something like this to fix the errors -

def cmp(a, b):
    return (a > b) - (a < b) 

Conclusion

I hope this information was helpful for you and I also hope that this may expedite your migration time. Overall, I must also confess that even though migrations steps seems kind of pretty straight forward but the whole process can be tiring because of fixing some common errors again and again. If you are able to fix these small errors quickly then you can focus on the functional part of the migration which is most important. So which is why I tried to list some of the common issues I faced during the migration, hope it can be useful for someone. Thanks for reading and feel free to share any feedback in the comment section.

References:

https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3
http://python3porting.com/
https://docs.python.org/dev/howto/pyporting.html
https://python-future.org/quickstart.html
https://python-future.org/quickstart.html#to-convert-existing-python-2-code (On python 2 compatibility)

Disclaimer: This is a personal [blog, post, statement, opinion]. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.