DEV Community

Meeshkan profile image Mike Solomon
Mike Solomon for Meeshkan

Posted on • Edited on • Originally published at meeshkan.com

Comparing Prism, Hoverfly, and HMT- Part 2

#api #web #devops #testing

This is the second article of a two-part series comparing Prism, Hoverfly, and HMT. This article focuses on using all three tools to build a mock of the github API v3, whereas the first article focuses on a high level comparison of the tools.

HMT team
Team HMT busy building HMT.

In the previous article, we saw how to create a Prism server from an OpenAPI spec. We also saw how Hoverfly can play back recordings of server traffic. Lastly, we saw how HMT blends OpenAPI specs and recordings into a mock server.

In this article, we will examine how all three tools can mock the Stripe API.

Prism

To mock the Stripe API with Prism, we first download the Stripe V3 OpenAPI spec.



$ wget https://github.com/stripe/openapi/blob/master/openapi/spec3.yaml?raw=true
Enter fullscreen mode Exit fullscreen mode

Then, we spin up a Prism server that will mock the Stripe API according to the spec.



$ prism mock spec3.yaml
Enter fullscreen mode Exit fullscreen mode

Let's inspect the API using some common Stripe endpoints. For example, we can get test the balance endpoint by calling /v1/balance.



$ curl http://localhost:4010/v1/customers
Enter fullscreen mode Exit fullscreen mode

Prism provides useful information in the console as it validates the URI.



$ [7:50:52 PM] » [HTTP SERVER] get /v1/customers i  info      Request
received
[7:50:52 PM] »     [NEGOTIATOR] i  info      Request contains an accept header: */*
[7:50:52 PM] »     [VALIDATOR] ‼  warning   Request did not pass the validation rules
[7:50:52 PM] »     [NEGOTIATOR] √  success   Created a 401 from a default response
[7:50:52 PM] »     [NEGOTIATOR] √  success   Found response 401. I'll try with it.
[7:50:52 PM] »     [NEGOTIATOR] √  success   The response 401 has a
schema. I'll keep going with this one
Enter fullscreen mode Exit fullscreen mode

Unfortunately, this is where I ran into difficulties, as Prism serves the response.



{
  "type": "https://stoplight.io/prism/errors#UNKNOWN",
  "title": "Your schema contains $ref. You must provide specification in the third parameter.",
  "status": 500,
  "detail": ""
}
Enter fullscreen mode Exit fullscreen mode

While Prism is good at mocking most API specs, it struggled when it ran into the Stripe spec. I'm sure the team will fix this in an upcoming version of Prism. In the meantime, let's see how Prism handles the ubiquitous petstore.yml spec.



$ prism mock petstore.yml
Enter fullscreen mode Exit fullscreen mode

Now, when we query /pets, we get the following result.



[
  {
    "id": -9223372036854776000,
    "properties": {
      "isCat": true
    },
    "name": "string",
    "tag": "string"
  }
]
Enter fullscreen mode Exit fullscreen mode

Conclusion: Prism will create an out-of-the-box mock for many APIs. Although it struggled with Stripe, it works for most OpenAPI specs. It also has helpful logs.

Hoverfly

To create a Hoverfly mock, we first have to record the calls to the API we would like to mock. I'll use my Stripe test account, where I've created three mock customers. For example, here is the information for a customer named Jane Doe.

Jane Doe on Stripe

If we call the Stripe /v1/customers endpoint, we get information on all three mock customers.



$ curl -u **redacted** https://api.stripe.com/v1/customers
Enter fullscreen mode Exit fullscreen mode

And here's an abbreviated JSON.



{
  "object": "list",
  "data": [
    {
      "id": "cus_GhmXKD3awekHyV",
      "object": "customer",
      "account_balance": -755,
      "address": {
        "city": "Elk Grove",
        "country": "US",
        "line1": "12 McKenna Dr",
        "line2": "",
        "postal_code": "95757",
        "state": "CA"
      }
      ...
    }
    ...
  ],
  "has_more": false,
  "url": "/v1/customers"
}
Enter fullscreen mode Exit fullscreen mode

Now, let's fire up Hoverfly and record our interactions with some Stripe endpoints. First, we'll need to get the Hoverfly SSL certificate so we can call the Stripe HTTPS endpoint.



$ wget https://raw.githubusercontent.com/SpectoLabs/hoverfly/master/core/cert.pem
Enter fullscreen mode Exit fullscreen mode

We'll record several different outcomes, including failures.



$ hoverctl start
$ hoverctl mode capture
$ curl --proxy localhost:8500 https://api.stripe.com/v1/customers # leave out the key
$ curl --proxy localhost:8500 -u **redacted** https://api.stripe.com/v1/customers # include the key
$ curl --proxy localhost:8500 -u **redacted** https://api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # getting one customer # get one customer
$ curl --proxy localhost:8500 -u **redacted** -X POST -d email="susan@example.com" -d name="Susan McLane" -d description="Our secret favorite customer" https://api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # add one customer
$ curl --proxy localhost:8500 -u **redacted** -X DELETE https://api.stripe.com/v1/customers --cacert cert.pem # try to delete all customers, will fail!
$ curl --proxy localhost:8500 -u **redacted** -X DELETE https://api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # try to delete one customer, will succeed
$ curl --proxy localhost:8500 -u  -X DELETE https://api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # leave out the key, will fail
$ curl --proxy localhost:8500 -u **redacted** -X DELETE https://api.stripe.com/v1/customers/cus_GhmV4PbooARHhM # delete a customer that doesn't exist, will fail
$ hoverctl export stripe-simulation.json
$ hoverctl stop
Enter fullscreen mode Exit fullscreen mode

Digressing for a moment, I'd like to say a word about making recordings for tests. As you saw in the example above, POST or DELETE mutates the real data backing your requests. If you're not careful, this can lead to some pretty bad consequences. That's why I would always recommend recording real server traffic.

Back to Hoverfly, let's now examine how our recordings are transformed into a mock API. We'll start by firing up a Hoverfly server.



$ hoverctl start webserver
$ hoverctl import stripe-simulation.json
Enter fullscreen mode Exit fullscreen mode

Now, let's test retrieving a single customer.



$ curl http://localhost:8500/v1/customers/foobar -u **redacted**
Enter fullscreen mode Exit fullscreen mode

Hoverfly produces a helpful error message indicating what went wrong. It also gives suggestions how to fix the matcher to produce a result. Here is a truncated error message:



Hoverfly Error!

There was an error when matching

Got error: Could not find a match for request, create or record a valid matcher first!

The following request was made, but was not matched by Hoverfly:

{
    "Path": "/v1/customers/foobar",
    "Method": "GET",
    ...
}

Whilst Hoverfly has the following state:

{}

But it did not match on the following fields:

[path]

Which if hit would have given the following response:

{
    "status": 200,
    "body": "...",
}
Enter fullscreen mode Exit fullscreen mode

Let's follow its suggestion and add a glob wildcard to stripe-simulation.json. Before, the file contained this.



[
  {
    "matcher": "exact",
    "value": "/v1/customers/cus_GhmV4PbooARHhM"
  }
]
Enter fullscreen mode Exit fullscreen mode

And now, we will change it to this.



[
  {
    "matcher": "glob",
    "value": "/v1/customers/*"
  }
]
Enter fullscreen mode Exit fullscreen mode

Running our curl request again, we get the recorded response. w00t! Here's a shortened version of the response.



{
  "id": "cus_GhmV4PbooARHhM",
  "object": "customer",
  "account_balance": 0,
  "address": {
    "city": "Melbourne",
    "country": "AU",
    "line1": "1 Batman Way",
    "line2": "",
    "postal_code": "3000",
    "state": "NSW"
  },
  ...
  "tax_info": null,
  "tax_info_verification": null
}
Enter fullscreen mode Exit fullscreen mode

This is the basic flow in Hoverfly. For small APIs, I find this to be a compelling approach. For larger APIs with more complex behavior, OpenAPI-based mocks tend to perform better. This is because the spec will likely contain more outcomes and endpoints than you would see in the wild.

HMT

The first step in creating a mock with the HTTP Mocking Toolkit (HMT) is the same as the Prism mock - we download the OpenAPI spec.



$ wget https://github.com/stripe/openapi/blob/master/openapi/spec3.yaml?raw=true
Enter fullscreen mode Exit fullscreen mode

We'll create a folder called stripe-mock, where we will move the spec.



$ mkdir stripe-spec && mv spec3.yaml ./stripe-spec
Enter fullscreen mode Exit fullscreen mode

Already here, we can start HMT in server mode an examine some results.



$ hmt mock -a ./stripe-spec
$ curl -i -X GET http://localhost:8000/v1/clients
Enter fullscreen mode Exit fullscreen mode

We can see in the gist below that the result is an array of mock users built using Stripe's OpenAPI spec.



{
  "data": [
    {
      "created": 74675021,
      "id": "dolor enim minim culpa ipsum",
      "livemode": true,
      "object": "customer",
      ...
    }
    ...
  ],
  "has_more": true,
  "object": "list",
  "url": "/v1/customers"
}
Enter fullscreen mode Exit fullscreen mode

Now, let's enrich the spec with the same set of recordings we made when building our Hoverfly mock. Blending an OpenAPI spec with recorded server traffic has two main advantages:

  • it allows HMT to pick up on features of the spec not present in the recordings and vice versa.
  • you can alternate between serving rote recordings and synthetic data.

For example, let's record a call to /v1/customers as we did in the previous example.



$ hmt record --log_dir ./stripe-logs
$ curl -u **redacted** http://localhost:8500/https/api.stripe.com/v1/customers
$ curl http://localhost:8000/https/api.stripe.com/v1/customers # leave out the key
$ curl -u **redacted** http://localhost:8000/https/api.stripe.com/v1/customers # include the key
$ curl -u **redacted** http://localhost:8000/https/api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # getting one customer # get one customer
$ curl -u **redacted** -X POST -d email="susan@example.com" -d name="Susan McLane" -d description="Our secret favorite customer" http://localhost:8000/https/api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # add one customer
$ curl -u **redacted** -X DELETE http://localhost:8000/https/api.stripe.com/v1/customers --cacert cert.pem # try to delete all customers, will fail!
$ curl -u **redacted** -X DELETE http://localhost:8000/https/api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # try to delete one customer, will succeed
$ curl -u  -X DELETE http://localhost:8000/https/api.stripe.com/v1/customers/cus_GhmV4PbooARHhM --cacert cert.pem # leave out the key, will fail
$ curl -u **redacted** -X DELETE http://localhost:8000/https/api.stripe.com/v1/customers/cus_GhmV4PbooARHhM # delete a customer that doesn't exist, will fail
Enter fullscreen mode Exit fullscreen mode

Now, let's build a spec using this mock. We'll use gen mode. That means that if our mock runs into a customer it hasn't recorded, it will create a mock customer on the fly.




$ hmt build -i ./stripe-logs/ --mode gen -o build

$ hmt mock -s build/

$ curl http://localhost:8500/v1/customers -u redacted

$ curl http://localhost:8500/v1/customers/cus_x410fdsfxfs -u redacted
Enter fullscreen mode Exit fullscreen mode




Conclusion

In this article, we saw how three mocking tools - Prism, Hoverfly and HMT - can create a mock of the Stripe API. I hope that, in going through this, you got a sense of the differences between the three tools. In short:

  • Prism can create a mock of an OpenAPI spec
  • Hoverfly can serve back recordings
  • HMT can serve a mix of recordings and synthetic data

We're curious to hear your opinion of all three tools. Please leave a comment below letting us know what you think!

Top comments (1)

Collapse
 
vncz profile image
Vincenzo Chianese •

Hey,

Prism author here! I am aware of the bug with the Stripe API and it happens because of a circular references with their models.

I am aware of the situation and yes, we are working to fix that issue!

Cheers,

V.

Read next

av-codes profile image

Performance testing of OpenAI-compatible APIs (K6+Grafana)

AV -

amyereichert profile image

Test Intelligence in the Era of AI: Opportunities and Challenges

Amy E Reichert -

wizzyb profile image

Creating a Linux VM and installing nginx on it

Wisdom Benson -

debcloud profile image

DDoS Attack on Kubernetes: Effective Solutions

Debashis Sikdar -