Hello, Blog!
If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.
Aside from continuing to work on contributing to Open Source projects, this week's focus was on looking into "managing project complexity through the use of Automated Testing" & starting to write test cases for our CLI tools.
Testing is a fundamental aspect of software development. Tests allow us to refine the code's logic, ensuring each component performs as expected even when handling edge cases. Automated tests make it easier to introduce changes to code without breaking its core logic, which is especially important in the Open Source project (as they rely on collaboration).
Helpful links:
- A Beginner's Guide to Unit Testing with Pytest
- Mastering Unit Tests in Python with pytest: A Comprehensive Guide
- pytest documentation
- pytests-mock documentation
- My Project Repository
ADDCOM
addcom is a CLI source code documenter tool which provides coders with an easy way to add comments to their source code files
Give it a relative/absolute path to your file and it will analyze its contents and add comments using a Large Language Model's chat completion.
See a demo of the functionality on YouTube: addcom-demo
Setup Instructions
Prerequisites
Make sure Python is installed on your system (you can download it here: https://www.python.org/downloads/).
1. After cloning the repo cd into the project folder and simply run:
pip install .
2. Default: Create an account and generate the API key here: https://console.groq.com/
By default, addcom uses the Groq API endpoint for chat completion. However, you can specify a custom endpoint using the --base-url or -u flag option. (If you do this, make sure to obtain an appropriate API key and specify the model supported by the chosen provider using…
I welcome feedback/contributions, so don’t hesitate to take a look and get involved 😊
Starting to write tests in Python
Beforehand, I had only written tests using Jest in one of my Node.js projects, so I had a general understanding of how to approach testing. However, I had no experience with testing in Python.
So, to start, I went "shopping" for the right testing library. After reading a couple of articles and community discussions, I decided to begin writing tests using Pytest in combination with the pytest-mock plugin, which I used to mock the API interactions with the LLM.
Why Pytest?
Pytest is one of the most popular testing frameworks that combines simple syntax (e.g. less boilerplate code compared to Python's built-in unittest library) with powerful features and a vast ecosystem of plugins for nearly every testing scenario.
Prep & Setup
Before diving straight into the documentation, I like to go through a few simple beginner guides to get a quick overview of the tools and their functionality. I did the same with Pytest, reading two beginner guides before jumping into setup and coding (A Beginner's Guide to Unit Testing with Pytest, Mastering Unit Tests in Python with pytest: A Comprehensive Guide).
The setup was close to none, I simply installed the library and the mocking plugin using pip, created a new tests/ folder at the root of my project, and was good to go:
pip install pytest pytest-mock
mkdir tests/
Writing the first test
I began testing with the simplest function in my project: the version_callback() function, which prints the version of the CLI tool when the --version/-v flag is provided.
I covered three test cases:
- Successful version output – Verifying that the version prints correctly to the terminal when the flag is used.
- No flag behaviour – Ensuring that not providing the flag does not cause any unexpected behaviour.
- Handling unset version – Confirming that the function prints an appropriate message if the version is not set.
Writing the tests was straightforward, I mostly followed examples from guides and documentation, with occasional clarifications from an AI chatbot.
I used pytest assertions for stdout comparisons and pytest.raises for exception checks and also got to try out the patching to replace the imported version value with None.
import pytest
from typer import Exit
from app.core.callbacks import version_callback
from app import __version__
def test_version_callback_without_version_defined(mocker, capfd):
"""
Test version callback function when the version is not defined
"""
mocker.patch("app.core.callbacks.__version__", None)
with pytest.raises(Exit):
version_callback(provided=True)
# Capture the output
captured = capfd.readouterr()
# Check that it prints a fallback message instead of the version
assert "version not defined" in captured.out.strip()
See the full code on GitHub
Testing the core functionality/Mocking LLM interactions
The core functionality of the tool revolves around interacting with the LLM, which is handled in the generate_comments() function. To test this, I needed to mock the LLM interaction.
To simulate the OpenAI client interaction, I used the pytest-mock plugin with its mocker fixture. This allowed me to mock the behaviour of the OpenAI client and control the responses returned by its methods during testing.
In
pytest, a fixture is a function that prepares and provides the necessary setup for tests. Fixtures are defined with the@pytest.fixturedecorator and can be automatically passed into test functions as arguments.
@pytest.fixture
def mock_openai_client(mocker):
"""
Fixture to mock the OpenAI client and its responses
"""
# Create a mock object with MagicMock()
mock_client = mocker.MagicMock()
# Replace the client with the mock object whenever
# OpenaAI() is called in the code
mocker.patch("app.core.api.OpenAI", return_value=mock_client)
return mock_client
####################################################################
def test_generate_comments_with_api_key(test_parameters, mocker, mock_openai_client):
"""
Test that the function generates expected output when a valid file path and API key are provided
"""
file_path, content = test_parameters
commented_code = """
def generate_fibonacci(limit):
# Initialize the Fibonacci sequence with the first two numbers
fibonacci_sequence = [0, 1]
# Generate Fibonacci numbers until the next number exceeds the given limit
while True:
# Calculate the next number in the sequence
next_number = fibonacci_sequence[-1] + fibonacci_sequence[-2]
# Stop generating if the next number exceeds the limit
if next_number > limit:
break
# Append the next number to the sequence
fibonacci_sequence.append(next_number)
return fibonacci_sequence
"""
# Replicate the OpenAI client response structure using mock objects
mock_response = mocker.MagicMock()
mock_response.choices = [
mocker.MagicMock(message=mocker.MagicMock(content=commented_code))
]
# Set the result of the mock chat completion to the mock response,
# so when the value is accessed in the code it gives the expected result
mock_openai_client.chat.completions.create.return_value = mock_response
result = generate_comments(
file_path, content, None, "test_api_key", None, None, False
)
assert result == commented_code
See the full code on GitHub
Afterthoughts
From the start, I tried making my code clean and modular by breaking it down into smaller functions. This approach certainly made my project more testable. But, along the way, I realized that I hadn't fully accounted for all possible edge cases ("Wait, does the logic break when this value is None?"). From now on, I will try to look at my code more critically by asking more "what if?" questions.
Writing the first few tests helped me gain a solid understanding of pytest. I learned how to do simple assertions, test exceptions, capture data written to the terminal, use fixtures for setup, and how to replace values and mock different objects using mocker.
Although testing the generate_comments() and version_callback() functions has given me a good foundation, I still need to improve test coverage, explore testing argument parsing and figure out a way to test the main entry point function.
Top comments (0)