Description
This post is about lab 07 for OSD600 course at Seneca College. This week we learned how to test our project. We learned about unit test, e2e(end to end), and integration test.
Progress
First, I needed to decide which test framewokr I am going to use. I asked chatGPT which one is most popular and also googled it and read some people's opinion. Then, I decided to use
pytestframework.Second, I started writing test code with each class that is called on top of the function call stacks. I chose the easist one, which is
load_config.py. It only contained one function.
import os
import toml
def load_config():
# Load config file from the project's root directory
config_path = os.path.expanduser("~/.codemage-config.toml") # Adjust path to project root
if os.path.exists(config_path):
config = toml.load(config_path)
else:
config = {}
return config
This function opens .toml file, and read key-value pairs. I had to mock the config file and its content.
class TestLoadConfig:
@patch("os.path.exists")
@patch("toml.load")
def test_load_config_file_exists(self, mock_toml_load, mock_os_exists):
# Simulate that the file exists
mock_os_exists.return_value = True
# Simulate the contents of the file
mock_toml_load.return_value = {
"language": "Python",
"OPENROUTER_API_KEY": "mock_openrouter_api_key",
}
# Call load_config function
config = load_config()
# Assertions
mock_os_exists.assert_called_once_with(os.path.expanduser("~/.codemage-config.toml"))
mock_toml_load.assert_called_once_with(os.path.expanduser("~/.codemage-config.toml"))
assert config == {"language": "Python", "OPENROUTER_API_KEY": "mock_openrouter_api_key"}
I used @patch to mock the file, file status(exists or not), and the content. Then I checked if the file is called with the path and if file content is correct.
- Third, I tested
Apiclass. Since this is also independent and called on the top of the function stack.
class Api:
def __init__(self, model, config):
self.supported_model = ["groq", "openrouter"]
self.model = model if model is not None else "openrouter"
if self.model not in self.supported_model:
sys.exit(f"{self.model} is not suppored. Model Supported: {self.supported_model}")
# default api_url and api_model
self.api_url = "https://openrouter.ai/api/v1"
self.api_model = "sao10k/l3-euryale-70b"
self.api_key = os.getenv("OPENROUTER_API_KEY") or config.get("OPENROUTER_API_KEY")
# api_url and api_model when the provider is groq
if self.model == "groq":
self.api_url = "https://api.groq.com/openai/v1"
self.api_model = "llama3-8b-8192"
self.api_key = os.getenv("GROQ_API_KEY") or config.get("GROQ_API_KEY")
def call_api(self, target_lang, code, stream_flag=False):
client = OpenAI(
base_url=self.api_url,
api_key=self.api_key,
)
completion = client.chat.completions.create(
extra_headers={},
model=self.api_model,
messages=[
{
"role": "system",
"content": "only display the code without any explanation",
},
{
"role": "user",
"content": f"translate this to {target_lang} language: {code}",
},
],
stream=stream_flag,
)
return completion
I tested the constructor first and then I tested the method call_api. Testing constructor was not difficult. I just needed to make a mock_config, and mock_args. Testing the method was pretty tough. I had to mock the response object.
class TestApiWithEnv:
@pytest.fixture
def mock_config(self):
return {
"OPENROUTER_API_KEY": "fake_openrouter_api_key_from_toml",
"GROQ_API_KEY": "fake_groq_api_key",
}
# Mock OpenAI response
def mock_api_call(self, *args, **kwargs):
# Return a mock response object with a `choices` attribute
mock_response = MagicMock()
mock_response.choices = [{"message": {"content": "Translated code"}}]
return mock_response
# Test when model is supported
@patch.dict(
os.environ,
{
"OPENROUTER_API_KEY": "fake_api_key_from_env",
"GROQ_API_KEY": "fake_groq_api_key_from_env",
},
)
@patch("code_mage.api.OpenAI")
def test_api_with_openrouter(self, mock_openai_class):
# Mock the OpenAI client and its methods
mock_client = MagicMock()
# When the code create OpenAI class inside call_api(), it's replaced by mock_openai_class
mock_openai_class.return_value = mock_client
mock_client.chat.completions.create = MagicMock(return_value=self.mock_api_call())
# Create an instance of the Api with a supported model
api = Api(model="openrouter", config={})
# Simulate an API call
response = api.call_api(target_lang="python", code="some_code")
# Check if the API call was made with the expected arguments
mock_client.chat.completions.create.assert_called_once_with(
extra_headers={},
model="sao10k/l3-euryale-70b",
messages=[
{"role": "system", "content": "only display the code without any explanation"},
{"role": "user", "content": "translate this to python language: some_code"},
],
stream=False,
)
# Assert the response content
assert response.choices[0]["message"]["content"] == "Translated code"
Fortunately, there is a way to mock api call provided. I used
@patch("code_mage.api.OpenAI")
This is one of my Test codes for Api Class.
class TestApiWithEnv:
@pytest.fixture
def mock_config(self):
return {
"OPENROUTER_API_KEY": "fake_openrouter_api_key_from_toml",
"GROQ_API_KEY": "fake_groq_api_key",
}
# Mock OpenAI response
def mock_api_call(self, *args, **kwargs):
# Return a mock response object with a `choices` attribute
mock_response = MagicMock()
mock_response.choices = [{"message": {"content": "Translated code"}}]
return mock_response
# Test when model is supported
@patch.dict(
os.environ,
{
"OPENROUTER_API_KEY": "fake_api_key_from_env",
"GROQ_API_KEY": "fake_groq_api_key_from_env",
},
)
@patch("code_mage.api.OpenAI")
def test_api_with_openrouter(self, mock_openai_class):
# Mock the OpenAI client and its methods
mock_client = MagicMock()
# When the code create OpenAI class inside call_api(), it's replaced by mock_openai_class
mock_openai_class.return_value = mock_client
mock_client.chat.completions.create = MagicMock(return_value=self.mock_api_call())
# Create an instance of the Api with a supported model
api = Api(model="openrouter", config={})
# Simulate an API call
response = api.call_api(target_lang="python", code="some_code")
# Check if the API call was made with the expected arguments
mock_client.chat.completions.create.assert_called_once_with(
extra_headers={},
model="sao10k/l3-euryale-70b",
messages=[
{"role": "system", "content": "only display the code without any explanation"},
{"role": "user", "content": "translate this to python language: some_code"},
],
stream=False,
)
# Assert the response content
assert response.choices[0]["message"]["content"] == "Translated code"
I used ChatGPT; it gave me good hints, but didn't work well, I had to make some changes and it worked. If I am being honest, I am still confused how it works, but it looks like it tests the code as it should. I think I need to study more to fully understand and write test code.
- Last, the most difficult part was testing
Translatorclass. It calls other functions that requires mock environments such as openeing a file from file system and calling api.
This is my test code for translate() function in Translator class:
class TestTranslator:
@pytest.fixture
def mock_args(self):
"""Fixture for simulating command-line arguments."""
return Mock(language=None, model=None, stream=False, token_usage=False, output=None)
def mock_api_call(self, *args, **kwargs):
mock_response = MagicMock()
mock_response.choices = [{"message": {"content": "Translated code"}}]
return mock_response
@patch("builtins.open", new_callable=MagicMock) # Mock open
@patch("code_mage.translator.Api") # Mock Api class
@patch.object(Translator, "_Translator__get_output_filename", return_value="translated_test.py")
def test_translate(self, mock_get_output_filename, mock_api, mock_open, mock_args):
mock_config = {"api_key": "fake_api_key"}
# Initialize Translator instance
translator = Translator(mock_args, mock_config)
mock_file = MagicMock()
mock_open.return_value = mock_file
# Mock the Api's call_api method
mock_api_instance = MagicMock()
mock_api.return_value = mock_api_instance
mock_api_instance.chat.completions.create = MagicMock(return_value=self.mock_api_call())
translator.translate("./example/test.js")
assert mock_api_instance.chat.completions.create.return_value.choices == [
{"message": {"content": "Translated code"}}
]
mock_open.assert_any_call("./example/test.js", "r")
mock_open.assert_any_call("translated_test.py", "w")
What was challenging was that I didn't understand how it knows which mock is for api_call() and which mock is for open() when I made two Mock objects. I tested it by changing the order of arguments, and the name of arguments. I think it maps @patch result to the arguments by order and name. I don't understand how exactly it works, but I learned that the order and name matters and I should make sure that they followes the order in which @patch is called and the name.
Conclusion
At first, I thought that it is going to be easy. However, it was really tough, especially for mocking object, .env, file open(), http request, and http response. It also was kind of fun. From this lab, I learned how to test code thoroughly and how important it is to modulize the code so that I can test the code in a isolated environment. It's not perfect but at least it guarantees a certain level of robustness.
Top comments (0)