Part Two, Upgrading to Python 3.x

Is your team still using Python 2? If you’re not sure, now is a really good time to check. In April, the Python team released version 2.7.18, and it will be the last version of Python 2.x. If security vulnerabilities or other bugs are discovered going forward, they will NOT be fixed. To ensure that your software is secure and functioning properly, it’s imperative to develop a plan for migrating to Python 3. Numerous strategies can be found online. This post outlines an approach that we recommend you take.

How Not to Upgrade

Before we dig into the steps in our recommended upgrade process, let’s spend some time reviewing things to avoid doing.

Don’t stop the presses

It’s possible to upgrade without stopping all of your regular work. While you might need to invest additional resources, such as reallocating staff or hiring some outside help, the long-term health of your codebase will benefit from continuing ongoing maintenance and enhancement-related tasks during the upgrade process. If you need additional help during the upgrade, Corgibytes would love to assist, if you’ll forgive the plug. 🙂

Don’t create a new app from scratch

Python can be upgraded gradually, without replacing your entire codebase with something new in one go. Doing so is way too much work to do all at once anyways, especially since you already have a working application. It’s best to tackle the upgrade a little bit at a time.

Don’t do a hard cutover

Don’t expect that you’ll be able to deploy the migration in a single evening or weekend. Give yourself time, and assume that you’re going to have a few failed attempts. The process that’s outlined here defends against the optimism bias that our human brains so often fall victim to.

The Upgrade Process

Now that we’ve covered what not to do, let’s talk about what to do.

Upgrade to 2.7.18

If you’re still using an older version of 2.7, make sure you upgrade to 2.7.18 right away to benefit from the various security fixes that have been released so far. Doing so will also ensure the upgrade to 3.x is as simple as possible.

Make sure you have enough automated tests

There’s no exact “correct” amount of test coverage that must be achieved before starting an upgrade. You just need to have enough coverage so that you trust your tests to catch most problems. If you’re not at that point already, then you’ll need to add more tests.

ApprovalTest is a great tool for adding a lot of coverage with minimal effort. It takes a “golden master” approach to testing, meaning that it will ensure that your program continues to work the way that it used to prior to the upgrade.

Upgrade your code but keep it working on 2.7

The goal through this process is that your codebase is still deployable using Python 2.7. This will allow us to make your project Python 3 “ready” without having to stop the other work that you’re doing. You won’t have to pause other feature or bug fix work. You also won’t have to worry about a giant python-3 branch getting out of sync with you primary branch.

To make your code work with both Python 2 and Python 3, we recommend using a compatibility library such as six. six helps minimize the differences between Python 2 and Python 3 by providing a consistent API that makes the correct underlying call depending on which version of Python is running.

Pick a starting point

You might be wondering which version of Python 3.x to start with. Our recommendation is to start with the latest stable version. At the time of this writing, that’s Python 3.9.0. You might have to walk that backwards once you start updating your dependencies, but we’re getting ahead of ourselves.

Wrangle your dependencies

Some failures will happen before your test suite even runs. You may be using libraries that don’t yet support Python 3 themselves and may fail to install as a result. In addition to installation issues, your test suite is going to find dependencies that installed successfully using Python 3, but which are still lacking proper Python 3 support. To fix these errors, you’ll need to upgrade to a version of those libraries that supports both Python 2 and Python 3.

Because Python 3 was initially released over a decade ago, there’s a good chance that the latest version of a library may have already dropped support for Python 2 entirely and you’ll need to utilize an interim version of that dependency during the upgrade. There’s also a chance that the interim version of the library won’t work with the newest version of Python 3.x. So you’ll need to temporarily target an older version of Python 3.x in the interest of maintaining cross-compatibility with Python 2.7. Don’t fret about this too much. It’s temporary. At the end of this we’ll get you running on the latest version of Python 3.x.

Run your test suite with both Python 2.7 and Python 3.x

It’s helpful to run your tests against both the old and new versions of Python. tox is a great tool to help you achieve this. For both Python versions, it can run your test suite after installing all of your dependencies into an isolated virtual Python environment. Once you have everything set up, it does all of this from just one command tox.

This will allow you to easily run your test suite against a version of Python 2.7.x and a version of Python 3.x. While the Python 3.x version will initially show a ton of errors, the Python 2.7.x version should have no errors. Meaning that you can safely deploy your application to a Python 2.7 environment and it’ll still work.

At this point in the process you’ll want to have output from tox which looks something like this:

____________________ summary ____________________
py27: commands succeeded
ERROR:   py39: commands failed

Address issues and errors

Running your test suite in both 2.7 and 3.x environments will likely generate quite a long list of failures and errors. This list is now your to-do list. Churn through these items one at a time until all tests pass in both environments. In some cases, fixing an issue in one environment, such as Python 3.9, may cause new issues in the other other environment, such as Python 2.7 and you may need to temporarily revert the initial change while other areas of code are upgraded.

Perform exploratory testing

At this point, you’ve probably found more than 90% of the errors in the codebase. How you handle the last 10% really depends on your individual needs. However, in general, we think that exploratory testing in Python 3 is a really good idea. Doing so will leverage your human brain’s powerful sleuthing skills. These are incredibly hard to replicate and just may uncover several issues that your automated test suite let slip past.

Keep in mind that for every issue that you discover you need to first confirm that it was introduced as a result of the upgrade and didn’t previously exist. Log issues unrelated to the upgrade in the same place you track issues of that type, and stay focused on just the Python 3 specific problems.

Make the switch in production

Once your test suite is passing and you’re satisfied with your exploratory testing, it’s time to make the switch over to Python 3 mode in production. If any issues are found from this point, they should be fixed using a “fail forward” mentality. That is, keep production running on Python 3 and quickly address the problems that are discovered. Reverting back to Python 2 at this point should be seen as a last resort.

For each issue that’s found, make sure to add a corresponding test case, and also make sure that your fix works in both Python 2 and Python 3. Should things go drastically wrong, then you can always revert back to Python 2, but the goal is to have built up enough confidence in your Python 3 support that you can now stick with it.

Drop support for Python 2

Once things are pretty stable on Python 3, you can gradually start removing Python 2 support from your codebase. This will enable you to upgrade your dependencies to the latest stable versions (which may have previously been unavailable as they’d already dropped support for Python 2). Also, if you didn’t initially upgrade to the latest Python version, take this time to make sure your codebase works against the latest stable version of Python 3.x. A number of 3.x versions have reached end-of-life status, so it’s important to keep an eye on the support status for each version and make sure your version will continue to be supported and receive security updates.

Don’t let a green field turn brown

Once you take your software this far, it’ll be pretty close to a new “greenfield” Python 3.x application. The dependencies that you’re using are at their latest. Now, invest a little bit more effort to keep things this way. You can continue to use tox, but instead of making sure your application supports the older versions of Python, start using it to evaluate how well it works with the next version. You can also start to employ tools such as Dependabot to detect when new packages are released and automate upgrading them while also testing how well the newer version works with your application. Keeping your application as updated as possible will keep your technical debt low and will reduce the effort needed to upgrade to newer Python versions going forward.

Share your experience

Have you recently upgraded Python? How’d it go? Did you use our recommended approach? We’d love to hear more about your experiences in the comments below. If you haven’t upgraded yet and need to convince your boss that it is time, checkout part one of this two-part series: Python 2 Sunsetting: What Does This Mean for Your Business? As always, we’re happy to help - just drop us a note!