We strive to make functions of a single responsibility in our code. You'll notice in the previous post, we had 3 functions that had one responsibility: Grab the webpage, clean up the values, and write a json. We didn't create a mammoth function to do all 3 at once, though our simple script from a couple of posts ago essentially did just that.
As we create our functions, it's good to create tests to make sure each unit in your system is functioning as expected. When a system gets large, it becomes harder and harder to debug where problems are occurring. Edge cases will pop up that your functions won't be able to handle and your script will fail. Unit testing allows us to look at the inputs/outputs of our function, dream up possibilities, and ensure our function can handle it. If we need to make modifications to the function in the future, we can see if the function still is able to handle the scenarios you first presented it.
What does this process look like?
In the order of operations, ideally you:
- Create unit test(s) with different types of inputs and their expected outputs
- Create your function like this:
def my_function(your_inputs):
pass
- Run your unit test(s) and watch them fail.
- Replace
passin the function with what it is supposed to achieve (e.g. grab a website and return it). - Run your unit test(s)
- Alter your function and run your unit test(s) until all of the unit tests are passing
Congratulations! You've just gone through the process of Test Driven Development (TDD).
Let's take a look at the web_call function we created last time and make some unit tests for it. Before I begin though, I have to add an important point about unit tests: They are to be self-contained; no internet connection required.
How can we test a function that makes a web_call without the web?
Through mocking. First, I'm going to create a PyTest "fixture" (reusable) that says we're about to make a get request and instead of actually making the request, this is what I want you to return.
import requests
from unittest.mock import patch
from bs4 import BeautifulSoup
import json
import pytest
from functional_scraper import web_call, get_fund_values, write_json
@pytest.fixture
def mock_requests_get():
with patch.object(requests, 'get') as mock_get:
yield mock_get
@pytest.fixture
def mock_requests_failed_get():
with patch.object(requests, 'get') as mock_get:
mock_get.return_value.status_code = 400
yield mock_get
You'll notice I have 2 mocks: one where I have a successful call and one where I have a call that returns a 400 status code (client-side failure). We could easily make a dozen more of these with different status_codes, but that wouldn't provide any additional value; making these two to cover the two basic scenarios in the conditional should suffice.
What are our unit tests?
def test_web_call(mock_requests_get):
mock_requests_get.return_value.status_code = 200
mock_requests_get.return_value.content = '<html><head></head><body><div></div></body></html>'
result = web_call('http://www.example.com')
assert isinstance(result, BeautifulSoup)
def test_failed_web_call(mock_requests_failed_get):
result = web_call('http://www.example.com')
assert result == 400
You'll see in the first one, we return a status_code and some basic html content. We then make sure the function returned the BeautifulSoup content as it should.
In the second unit test, we test the other side, where we don't get a successful call to the webpage and get a 400 status_code.
Both of these unit tests pass. I have set up a pipeline in Github using Github Actions to run my unit tests every time I push new code into the repo!
Join me again next time when we turn our function into a class.
As always, you can find all of my code here in the Github repo.
Top comments (0)