Continuous testing python code with pytest: how to

In this article, we will look at simple and straightforward ways to set up continuous testing of python code for your local machine. After we complete these steps we will be able to get real time updates to test results in a terminal every time we save a code file.

You can find an example project in the following repository.

Why continuous testing

The main motivator to having continuous testing is, instantly letting us know when a code change has broken some piece of functionality. Continuously testing python code will give us confidence that our code is robust. Furthermore, Continuous testing also lowers the bar for major code refactoring, because we will be operating with the knowledge that tests are continuously running and that if something goes wrong we will know about it right away, before it becomes a problem. It will also motivate us to write code that is unit testable, which, in turn, encourages cleaner code. Let us get started on continuous testing a python project.

Prerequisites

Some knowledge about pytest for context is helpful.

What we will learn

We will learn how to set up continuous testing and reporting of code coverage. To do all that, we will use the following python libraries:

• pytest: is a test runner and test framework which will collect tests from the project and run them, and also gives us tools to write tests more easily.
• pytest-cov: produces a test code coverage report.
• pytest-watch: is a tool for re-running tests with pytest every time a file is changed and saved.

Package installation

Start by installing the packages mentioned in the previous section:

pip install pytest pytest-cov pytest-watch

Don’t forget to add these dependencies to a requirements.txt file:

pip freeze > requirements.txt

Setting up a simple test project

We are going to create a simple project structure with a single python script with a function we can unit test:

/continuous-testing-tutorial
+--src
   + main.py
   + __init__.py
+--tests
  + conftest.py
  + __init__.py
  +--unit
    + test_main.py

The src directory holds our code. In this case main.py.
The tests directory holds our tests and test set utility code.

Notice that the structure in tests is similar to the src directory.
The pytest tool will find any file prefixed with test_ or suffixed with _test.py.

The file conftest.py is automatically picked up by the pytest test runner and contains fixtures that are shared globally. Defining fixtures in conftest.py allows you to use fixtures in test files without having to import them explicitly.

Sample code

Open or create main.py in the src directory and paste in the following code:

def increase_person_age(years_to_increase_by, person):
    person["age"] = person["age"]+years_to_increase_by

def decrease_person_age(years_to_decrease_by, person):
    person["age"] = person["age"]-years_to_decrease_by

Writing tests

We will start by creating a global fixture in tests/conftest.py:

import pytest

@pytest.fixture
def get_person():
    return {
        "age": 23,
        "name": "Joe"
    }

This is just something simple to demonstrate how the global fixture works.

Open tests/unit/test_main.py an copy the following code into it:

from src import main

def test_increase_person_age(get_person):
    person = get_person

    main.increase_person_age(10, person)
    actual_result =  person["age"]

    expected_result = 33
    assert actual_result == expected_result

def test_decrease_person_age(get_person):
    person = get_person

    main.decrease_person_age(1, person)
    actual_result = person["age"]
    expected_result = 22
    assert actual_result == expected_result

Notice that we are using the get_person fixture we defined earlier as a parameter to the tests, however, we’re not explicitly importing it anywhere. That is the magic of conftest.py.

Running tests

We will now run the tests. We could the tests simply using pytest:

pytest

This will run the tests once and list the results. However, we want to run tests continuously and generate a coverage report. That is where pytest-watch and pytest-cov come in. Use the following command to run pytest-watch and have it produce a test coverage report:

ptw -- --cov-report term:skip-covered --cov=src tests

The parameter --cov-report produces the report.
The parameter term:skip-covered hides files from the report that have 100% coverage.
Next --cov=src indicates which folder should be inspected for code to calculate code coverage on.
Finally tests indicates where the tests are located.

The pytest-watch tool will keep running indefinitely and will detect any file changes until it has been canceled.

The report generated by the command is a simple summary of coverage percentages for each file in the project. The pytest-cov tool also has functionality to generate a detailed html report which shows which lines are or are not yet covered.

If we want to generate the detailed report we can use the following command:

pytest --cov=src --cov-report html:cov_html tests

This will generate the report in a directory cov_html in the project’s root directory. We can view it by opening the index.html file in a browser or starting a webserver in the cov_html directory and going to the hosting address.

I recommend not generating the html report continuously with pytest-watch as that will slow down the running of the tests on larger projects.

Conclusion

We can now have our code be tested every time we write new code or make changes to existing code. If we write our code with testing in mind, and keep up with writing tests at the same time, we will also gain confidence that our code has a solid foundation and will not break easily.

You can find the full example of continuous testing for python code in the following repository.

A practical example of using unit testing a flask backend API can be found in the article: Automated testing for a Flask API: useful examples.

Information about mocking and parametrization in tests can be found in the article: Mocking in python tests: practical examples.