Spring Boot – Black Box Testing

In this article, I’m showing you

1. What’s the difference between white box and black testing
2. What are the benefits of the latter
3. How you can implement it in your Spring Boot application
4. How to configure the OpenAPI generator to simplify code and reduce duplications

You can find the code examples in this repository.

Domain

We’re developing a restaurant automatization system. There are two domain classes. Fridge and Product. A fridge can have many products, whilst a product belongs to a single fridge. Look at the classes declaration below.

@Entity
@Table(name = "fridge")
public class Fridge {
    @Id
    @GeneratedValue(strategy = IDENTITY)
    private Long id;

    private String name;

    @OneToMany(fetch = LAZY, mappedBy = "fridge")
    private List<Product> products = new ArrayList<>();
}

@Entity
@Table(name = "product")
public class Product {
    @Id
    @GeneratedValue(strategy = IDENTITY)
    private Long id;

    @Enumerated(STRING)
    private Type type;

    private int quantity;

    @ManyToOne(fetch = LAZY)
    @JoinColumn(name = "fridge_id")
    private Fridge fridge;

    public enum Type {
        POTATO, ONION, CARROT
    }
}

I'm using Spring Data JPA as a persistence framework. Therefore, those classes are Hibernate entities.

White box testing

This type of testing makes an assumption that we know some implementation details and may interact them. We have 4 REST API endpoints in the system:

1. Create a new Fridge.
2. Add a new Product.
3. Change the Product quantity.
4. Remove the Product from the Fridge.

Suppose we want to test the one that changes the Product quantity. Take a look at the example of the test below.

@SpringBootTest(webEnvironment = RANDOM_PORT)
class ProductControllerWhiteBoxTest extends IntegrationSuite {
    @Autowired
    private FridgeRepository fridgeRepository;
    @Autowired
    private ProductRepository productRepository;
    @Autowired
    private TestRestTemplate rest;

    @BeforeEach
    void beforeEach() {
        productRepository.deleteAllInBatch();
        fridgeRepository.deleteAllInBatch();
    }

    @Test
    void shouldUpdateProductQuantity() {
        final var fridge = fridgeRepository.save(Fridge.newFridge("someFridge"));
        final var productId = productRepository.save(Product.newProduct(POTATO, 10, fridge)).getId();

        assertDoesNotThrow(() -> rest.put("/api/product/{productId}?newQuantity={newQuantity}", null, Map.of(
            "productId", productId,
            "newQuantity", 20
        )));

        final var product = productRepository.findById(productId).orElseThrow();
        assertEquals(20, product.getQuantity());
    }
}

Let’s examine this piece of code step by step.
We inject repositories to manipulate rows in the database. Then the TestRestTemplate comes into play. This bean is used to send HTTP requests. Then you can see that the @BeforeEach callback deletes all rows from the database. So, each test runs deterministically. And finally, here is the test itself:

1. We create a new Fridge.
2. Then we create a new Product with a quantity of 10 that belongs to the newly created Fridge.
3. Afterwards, we invoke the REST endpoint to increase the Product quantity from 10 to 20.
4. Eventually we select the same Product from the database and check that the quantity has been increased.

The test works fine. Anyway, there are nuances that should be taken into account:

1. Though the test verifies the entire system behavior (aka functional test) there is a coupling on implementation details (i.e. the database).
2. What the test validates is not the actual use case. If somebody wants to interact with our service, they won’t be able to insert and update rows in the database directly.

As a matter of fact, if we want to test the system from the user perspective, we can only use the public API that the service exposes.

Black box testing

This type of testing means loose coupling on the system's implementation details. Therefore, we can only depend on the public API (i.e. REST API).

Check out the previous white box test example. How can we refactor it into the black box kind? Look at the @BeforeEach implementation below.

@BeforeEach
void beforeEach() {
    productRepository.deleteAllInBatch();
    fridgeRepository.deleteAllInBatch();
}

A black box test should not interact with the persistence provider directly. Meaning that there should be a separate REST endpoint clearing all data. Look at the code snippet below.

@RestController
@RequiredArgsConstructor
@Profile("qa")
public class QAController {
    private final FridgeRepository fridgeRepository;
    private final ProductRepository productRepository;

    @DeleteMapping("/api/clearData")
    @Transactional
    public void clearData() {
        productRepository.deleteAllInBatch();
        fridgeRepository.deleteAllInBatch();
    }
}

Now we have a particular controller that encapsulates all the clearing data logic. If the test depends only on this endpoint, then we can safely put changes and refactor the method as the application grows. And our black box tests won’t break. The @Profile(“qa”) annotation is crucial. We don’t want to expose an endpoint that can delete all user data in production or even development environment. So, we register this endpoint, if qa profile is active. We’ll use it only in tests.

The qa abbreviation stands for the quality assurance.

And now we should refactor the test method itself. Have a look at its implementation below again.

@Test
void shouldUpdateProductQuantity() {
    final var fridge = fridgeRepository.save(Fridge.newFridge("someFridge"));
    final var productId = productRepository.save(Product.newProduct(POTATO, 10, fridge)).getId();

    assertDoesNotThrow(() -> rest.put("/api/product/{productId}?newQuantity={newQuantity}", null, Map.of(
        "productId", productId,
        "newQuantity", 20
    )));

    final var product = productRepository.findById(productId).orElseThrow();
    assertEquals(20, product.getQuantity());
}

There are 3 operations that should be replaced with direct REST API invocations. These are:

1. Creating new Fridge.
2. Creating new Product.
3. Checking the the Product quantity has been increased.

Look at the whole black box test example below.

@SpringBootTest(webEnvironment = RANDOM_PORT)
@ActiveProfiles("qa")
class ProductControllerBlackBoxTest extends IntegrationSuite {
    @Autowired
    private TestRestTemplate rest;

    @BeforeEach
    void beforeEach() {
        rest.delete("/api/qa/clearData");
    }

    @Test
    void shouldUpdateProductQuantity() {
        // create new Fridge
        final var fridgeResponse =
            rest.postForEntity("/api/fridge?name={fridgeName}", null, FridgeResponse.class, Map.of(
                "fridgeName", "someFridge"
            ));
        assertTrue(fridgeResponse.getStatusCode().is2xxSuccessful(), "Error during creating new Fridge: " + fridgeResponse.getStatusCode());
        // create new Product
        final var productResponse = rest.postForEntity(
            "/api/product/fridge/{fridgeId}",
            new ProductCreateRequest(
                POTATO,
                10
            ),
            ProductResponse.class,
            Map.of(
                "fridgeId", fridgeResponse.getBody().id()
            )
        );
        assertTrue(productResponse.getStatusCode().is2xxSuccessful(), "Error during creating new Product: " + productResponse.getStatusCode());

        // call the API that should be tested
        assertDoesNotThrow(
            () -> rest.put("/api/product/{productId}?newQuantity={newQuantity}",
                null,
                Map.of(
                    "productId", productResponse.getBody().id(),
                    "newQuantity", 20
                ))
        );

        // get the updated Product by id
        final var updatedProductResponse = rest.getForEntity(
            "/api/product/{productId}",
            ProductResponse.class,
            Map.of(
                "productId", productResponse.getBody().id()
            )
        );
        assertTrue(
            updatedProductResponse.getStatusCode().is2xxSuccessful(),
            "Error during retrieving Product by id: " + updatedProductResponse.getStatusCode()
        );
        // check that the quantity has been changed
        assertEquals(20, updatedProductResponse.getBody().quantity());
    }
}

The benefits of black box testing in comparison to white box testing are:

1. The test checks the path that a user shall do to retrieve the expected result. Therefore, the verification behavior becomes more robust.
2. Black box tests are highly stable against refactoring. As long as the API contract remains the same, the test should not break.
3. If you accidentally break the backward compatibility (e.g. adding a new mandatory parameter to an existing REST endpoint), the black box test will fail and you'll determine the issue way before the artefact is being deployed to any environment.

However, there is a slight problem with the code that you have probably noticed. The test is rather cumbersome. It’s hard to read and maintain. If I didn’t put in the explanatory comments, you would probably spend too much time figuring out what’s going on. Besides, the same endpoints might be called for different scenarios, which can lead to code duplication.

Luckily there is solution.

OpenAPI and code generation

Spring Boot comes with a brilliant OpenAPI support. All you have to do is to add two dependencies. Look at the Gradle configuration below.

implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springdoc:springdoc-openapi-ui:1.6.12'

After adding these dependencies, the OpenAPI specification is available by GET /v3/api-docs endpoint.

The SpringDoc library comes with lots of annotations to tune your REST API specification precisely. Anyway, that's out of context of this article.

If we have the OpenAPI specification, it means that we can generate Java classes to call the endpoints in a type-safe manner. What’s even more exciting is that we can apply those generated classes in our black box tests!

Firstly, let's define the requirements for the upcoming OpenAPI Java client:

1. The generated classes should be put into .gitignore. Otherwise, if you have Checkstyle, PMD, or SonarQube in your project, then generated classes can violate some rules. Besides, if you don't put them into .gitignore, then each pull request might become huge due to the fact that even a slightest fix can lead to lots of changes in the generated classes.
2. Each pull request build should guarantee that generated classes are always up to date with the actual OpenAPI specification.

How can we get the OpenAPI specification itself during the build phase? The easiest way is to write a separate test that creates the web part of the Spring context, invokes the /v3/api-docs endpoint, and put the retrieved specification into build folder (if you are Maven user, then it will be target folder). Take a look at the code example below.

@SpringBootTest(
    webEnvironment = RANDOM_PORT
)
@AutoConfigureTestDatabase
@ActiveProfiles("qa")
public class OpenAPITest {
    @Autowired
    private TestRestTemplate rest;

    @Test
    @SneakyThrows
    void generateOpenApiSpec() {
        final var response = rest.getForEntity("/v3/api-docs", String.class);
        assertTrue(response.getStatusCode().is2xxSuccessful(), "Unexpected status code: " + response.getStatusCode());
        // the specification will be written to 'build/classes/test/open-api.json'
        Files.writeString(
            Path.of(getClass().getResource("/").getPath(), "open-api.json"),
            response.getBody()
        );
    }
}

The @AutoConfigureTestDatabase configures the in-memory database (e.g. H2), if you have one in the classpath. Since the database provider does not affect the result OpenAPI specification, we can make the test run a bit faster by not using Testcontainers.

Now we have the result specification. How can we generate the Java classes based on it? We have another Gradle plugin for that. Take a look at the build.gradle configuration below.

plugins {
    ...
    id "org.openapi.generator" version "6.2.0"
}

...

openApiGenerate {
    inputSpec = "$buildDir/classes/java/test/open-api.json".toString()
    outputDir = "$rootDir/open-api-java-client".toString()
    apiPackage = "com.example.demo.generated"
    invokerPackage = "com.example.demo.generated"
    modelPackage = "com.example.demo.generated"
    configOptions = [
            dateLibrary    : "java8",
            openApiNullable: "false",
    ]
    generatorName = 'java'
    groupId = "com.example.demo"
    globalProperties = [
            modelDocs: "false"
    ]
    additionalProperties = [
            hideGenerationTimestamp: true
    ]
}

In this article, I'm showing you how to configure the corresponding Gradle plugin. Anyway, there is Maven plugin as well and the approach won't be different much.

There is an important detail about OpenAPI generator plugin. It creates the whole Gradle/Maven/SBT project (containing build.gradle, pom.xml, and build.sbt files) but not just Java classes. So, we set the theoutputDir property as $rootDir/open-api-java-client. Therefore, the generated Java classes go into the Gradle subproject.

We should also mark the open-api-java-client directory as a subproject in the settings.gradle. Look at the code snippet below.

rootProject.name = 'demo'
include 'open-api-java-client'

All you have to do to generate OpenAPI Java client is to run these Gradle commands:

gradle test --tests "com.example.demo.controller.OpenAPITest.generateOpenApiSpec"
gradle openApiGenerate

Applying the Java client

Now let’s try our brand new Java client in action. We’ll create a separate @TestComponent for convenience. Look at the code snippet below.

@TestComponent
public class TestRestController {
    @Autowired
    private Environment environment;

    public FridgeControllerApi fridgeController() {
        return new FridgeControllerApi(newApiClient());
    }

    public ProductControllerApi productController() {
        return new ProductControllerApi(newApiClient());
    }

    public QaControllerApi qaController() {
        return new QaControllerApi(newApiClient());
    }

    private ApiClient newApiClient() {
        final var apiClient = new ApiClient();
        apiClient.setBasePath("http://localhost:" + environment.getProperty("local.server.port", Integer.class));
        return apiClient;
    }
}

Finally, we can refactor our black box test. Look at the final version below.

@SpringBootTest(webEnvironment = RANDOM_PORT)
@ActiveProfiles("qa")
@Import(TestRestControllers.class)
class ProductControllerBlackBoxGeneratedClientTest extends IntegrationSuite {
    @Autowired
    private TestRestControllers rest;

    @BeforeEach
    @SneakyThrows
    void beforeEach() {
        rest.qaController().clearData();
    }

    @Test
    void shouldUpdateProductQuantity() {
        final var fridgeResponse = assertDoesNotThrow(
            () -> rest.fridgeController()
                .createNewFridge("someFridge")
        );
        final var productResponse = assertDoesNotThrow(
            () -> rest.productController()
                .createNewProduct(
                    fridgeResponse.getId(),
                    new ProductCreateRequest()
                        .quantity(10)
                        .type(POTATO)
                )
        );

        assertDoesNotThrow(
            () -> rest.productController()
                .updateProductQuantity(productResponse.getId(), 20)
        );

        final var updatedProduct = assertDoesNotThrow(
            () -> rest.productController()
                .getProductById(productResponse.getId())
        );
        assertEquals(20, updatedProduct.getQuantity());
    }
}

As you can see, the test is much more declarative. Moreover, the API contract become statically typed and the parameters validation proceeds during compile time!

The .gitignore caveat and the separate test source

I told that we should put the generated classes to the .gitignore. However, if you mark the open-api-java-client/src directory as the unindexed by Git, then you suddenly realize that your tests do not compile in CI environment. The reason is that the process of generation the OpenAPI specification (i.e. open-api.json file) is an individual test as well. And even if you tell the Gradle to run a single test directly, it will compile everything in the src/test directory. In the end, tests don’t compile successfully.

Thankfully, the issue can be solved easily. Gradle provides source sets. It's a logical group that splits the code into separate modules that you can compile independently.

Firstly, let's add the gradle-testsets plugin and define a separate test source that'll contain the OpenAPITest file. It's the one that generates the open-api.json specification. Take a look at the code example below.

plugins {
    ...
    id "org.unbroken-dome.test-sets" version "4.0.0"
}

...

testSets {
    openApiGenerator
}

tasks.withType(Test) {
    group = 'verification'
    useJUnitPlatform()
    testLogging {
        showExceptions true
        showStandardStreams = false
        showCauses true
        showStackTraces true
        exceptionFormat "full"
        events("skipped", "failed", "passed")
    }
}

openApiGenerator.outputs.upToDateWhen { false }

tasks.named('openApiGenerate') {
    dependsOn 'openApiGenerator'
}

The testSets block declares a new source set called openApiGenerator. Meaning that Gradle treats the src/openApiGenerator directory like another test source.

The tasks.withType(Test) declaration is also important. We need to tell Gradle that every task of Test type (i.e. the test task itself and the openApiGenerator as well) should run with JUnit.

I put the upToDateWhen option for convenience. It means that the test that generates open-api.json file will be always run on demand and never cached.

And the last block defines that before generating the OpenAPI Java client we should update the specification in advance.

Now we just need to move the OpenAPITest to the src/openApiGenerator directory and also make a slight change to the openApiGenerate task in build.gradle. Look at the code snippet below.

openApiGenerate {
    // 'test' directory should be replaced with 'openApiGenerator'
    inputSpec = "$buildDir/classes/java/openApiGenerator/open-api.json".toString()
    ....
}

Finally, you can build the entire project with these two commands.

gradle openApiGenerate
gradle build

Conclusion

The black box testing is a crucial part of the application development process. Try it and you’ll notice that the test scenarios become much more representative. Besides, black box test are also great documentation for the API. You can even apply Spring REST Docs and generate a nice manual that’ll be useful both for the API users and QA engineers.

If you have any questions or suggestion, leave your comments down below. Thanks for reading!

Resources

1. The repository with code examples
2. Spring Data JPA
3. Hibernate
4. Spring Profile
5. Quality Assurance
6. OpenAPI
7. Gradle
8. SpringDoc
9. Checkstyle
10. PMD
11. SonarQube
12. H2 database
13. Testcontainers
14. OpenAPI generator Gradle plugin
15. OpenAPI generator Maven plugin
16. Gradle source sets
17. Gradle testsets plugin
18. Spring REST Docs