Local Testing With Spring Data and GCP Datastore

Developing in the cloud is quite different from traditional development. A recurring challenge is to write tests that work locally or on a CI system without access to the cloud.

This article presents a way to test services that use Spring Boot and Google’s Datastore.
Prerequisites

• Java 11 ( which can easily be installed with sdkman.)
• Docker and docker-compose
• Basic knowledge Maven, Spring Boot and Google Cloud Platform.

Before we start

To see that everything runs locally, log out of gcloud and unset the project.

gcloud auth revoke
gcloud config unset project

The complete code is available on GitHub.

Project Setup

We start with a Spring Boot project which has all the necessary dependencies to use Datastore. You can find all code on GitHub.

The project contains a simple entity ( Article) and a spring-data repository with a custom method ( ArticleRepository ).
Let the test fail.

Checkout the tag 1-let-the-tests-fail if you want to follow this step by step.

If you run mvn clean install in this state, the test will fail since there is no active project:

[...]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.google.cloud.datastore.Datastore]: Factory method ‘datastore’ threw exception; nested exception is java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder.
Caused by: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder

This is easily fixed by adding a mock project id to the application.properties (See the git tag 2-add-project-id-property) :

spring.cloud.gcp.project-id=test-project

Maven (mvn clean install) now fails with the following error:

com.google.cloud.datastore.DatastoreException: Cloud Datastore API has not been used in project … before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/datastore.googleapis.com/overview?project=… then retry.

Spring now tries to connect to Datastore in test-project , which does not exist.
Make the tests green.

Here comes the Datastore emulator into play. You can install it via gcloud, but for portability, we will use a dockerized version.

We add the following docker-compose.yml to the root of the project:

version: '3'

services:

  datastore:
    image: knarz/datastore-emulator
    ports:
      - 8081:8432
    environment:
      CONSISTENCY: 1 # If you omit this, the emulator will emulate eventual consistency.

To ensure that Spring uses the emulator, we must add a property to application.properties that points to it:

spring.cloud.gcp.datastore.host=localhost:8081

(See the git tag 3-datastore-emulator for all code in this step)

Start the emulator with docker-compose up and run the test again with mvn clean install .

Now, the tests are green! You created a test with datastore which runs 100% locally!
Make it more comfortable.

Starting and stopping the container by hand is tedious and error-prone. Luckily, we can configure maven so that this happens automatically.

First, we add maven-failsafe-plugin to our build section in the pom.xml :

...
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.22.2</version>
</plugin>

The main task of the plugin is to run integration tests. You can discuss if our test qualifies as an integration test or not. However, there are no appropriate maven lifecycle phases for unit tests as there are for integration tests (which we’ll need in a second). Feel free to alter this step so that it suits your needs.

In order for maven-failsafe-plugin to execute our test, we have to rename it from ArticleRepositoryTest to ArticleRepositoryIT .

Then, we add exec-maven-plugin which will start the emulator before the integration tests and stop them afterward.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <version>3.0.0</version>
  <executions>
    <execution>
      <id>start-datastore-emulator</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <executable>docker-compose</executable>
        <arguments>
          <argument>up</argument>
          <argument>-d</argument>
        </arguments>
      </configuration>
    </execution>
    <execution>
      <id>stop-datastore-emulator</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <executable>docker-compose</executable>
        <arguments>
          <argument>down</argument>
          <argument>-v</argument>
        </arguments>
      </configuration>
    </execution>
  </executions>
</plugin>

Make sure to stop the docker container now with docker-compose down . Then, run mvn clean install to see that maven starts the emulator, runs the test and stops the emulator afterward:

[INFO] Scanning for projects…
[INFO] 
[INFO] — — — — — — — < com.example:data-datastore-local-testing > — — — — — — — 
[INFO] Building data-datastore-local-testing 0.0.1-SNAPSHOT...[INFO] 
[INFO] — — exec-maven-plugin:3.0.0:exec (start-datastore-emulator) @ data-datastore-local-testing — -
Creating network “data-datastore-local-testing_default” with driver “bridge”
Creating data-datastore-local-testing_datastore_1 … done
[INFO] 
[INFO] — — maven-failsafe-plugin:2.22.2:integration-test (default) @ data-datastore-local-testing — -
[INFO] 
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — -
[INFO] T E S T S
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — -
[INFO] Running com.example.localtesting.repository.ArticleRepositoryIT...[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.849 s — in com.example.localtesting.repository.ArticleRepositoryIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] — — exec-maven-plugin:3.0.0:exec (stop-datastore-emulator) @ data-datastore-local-testing — -
Stopping data-datastore-local-testing_datastore_1 … done
Removing data-datastore-local-testing_datastore_1 … done
Removing network data-datastore-local-testing_default...[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
[INFO] BUILD SUCCESS
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
[INFO] Total time: 18.264 s
[INFO] Finished at: 2021–10–19T16:10:59+02:00
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

Done! Thanks for reading this article. Head over to GitHub to get the complete code.

If this article was helpful for your, please consider to buy me a coffee :-)