Seamlessly Validate Your APIs Against Well-Documented Schemas from Swagger and Other API Tools and Standards
(Cover image from pexels.com by Picography)
ACT 1: EXPOSITION
Imagine you hire a contractor to renovate your home. You agree on work and price, documenting everything in a contract. But the contractor uses lower-quality materials and skips renovations. The contract is breached.
What do you do? Obviously, you need a brave fighter for justice—a fearless vigilante—to put that unscrupulous individual in their place. 🙂
Now, think of your API. The API schema is your contract. If the backend delivers unexpected data, you need a competent guardian to fight for your rights and point out all those unjust violations that blindsided you. That’s where a Schema Validator comes in to help you enforce your rights.
When I was creating my first API tests, unfortunately, many of the tests suddenly failed from one day to the next. Why now? I had to dig through those tests, debugging them and figuring out what could have gone wrong: Was it the test? Was the app changed? Was the data changed?
In my opinion, one of the most common reasons why web applications fail (if not the most common) is due to unexpected, uninformed, or undocumented changes in the API. Quite often, the 'QA guy' is the last to know about these changes; however, he is the one who has to sign off on the release. Counterintuitive, right?
That's why API testing exists alongside end-to-end (E2E) testing. One of the most common practices (again... if not the most common) when testing a specific API endpoint is to include assertions over the response to verify that the data obtained has a certain structure or design. Assertions like: this property is of type string, or that other property is an array of integers, the value is not null and is within a min and max range, or the property is present in the response, and so on.
Consequently, our tests for validating the correctness of the data structure in the response end up having quite a large number of lines of code, which we need to change when there are changes in the API. If you have, let's say, 100 tests just for one of your APIs, the time spent on maintenance will be considerable, especially if the API specification is still under development.
However, numerous organizations have well-documented backend APIs using tools like Swagger, following well-defined standard formats such as OpenAPI. Many of these organizations also do a fairly good job of updating these schemas regularly.
So why don't we use these schemas to validate the contract between our backend services and our frontend application? We can use these Schema Validators in our API tests to monitor and flag any injustice inflicted on our beloved API—just like a brave Vigilante.
ACT 2: CONFRONTATION
Contract Testing: Laying Down the Law
But before we get too far ahead of ourselves, let's understand a little bit about what Contract Testing is.
Contract Testing is an approach in software testing where the interactions between different services are tested based on the contract they agree upon. A contract specifies the expected inputs and outputs for a service, ensuring that different components interact correctly and reliably.
There are two main approaches to contract testing: Consumer-Driven Contract Testing and Producer-Driven Contract Testing. The key difference between them lies in who defines the contract.
Consumer-Driven Contract Testing focuses on the needs of the consumer. The contract is defined by the consumers of the API, dictating what they expect from the provider. One of the most popular tools for this approach is Pact. Pact, and specifically PactJS, is a powerful tool that allows consumers to create contracts that providers can then verify, ensuring that their APIs meet consumer expectations.
Producer-Driven Contract Testing, on the other hand, is where the provider defines the contract. This approach is often centered around schema validation. Producers outline the structure and constraints of the data they will provide, ensuring that consumers handle it correctly. Schema validation tools, such as Ajv, are crucial in this approach as they verify that the responses adhere to the predefined schema.
By leveraging these two methods, teams can ensure more robust and reliable communication between API producers and consumers, significantly reducing the risk of integration issues.
About JSON Schemas and Schema Documents: The Rulebook
Let's clarify a couple of key components that we'll be discussing throughout this post:
- JSON Schema: It is a hierarchical, declarative language that describes and validates JSON data.
{
"title": "User",
"type": "object",
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
},
"address": {
"type": "object",
"properties": {
"city": {
"type": "string"
},
"zipCode": {
"type": "string"
}
},
"required": ["city", "zipCode"]
}
},
"required": ["id", "name", "address"]
}
- OpenAPI 3.0.1 and Swagger 2.0 Schema Documents: The OpenAPI Specification (formerly Swagger Specification) are schema documents to describe your entire API (in JSON format or XML format). So a schema document will contain multiple schemas, one for each supported combination of Endpoint - Method - Expected Response Status (also called path) by that API.
{
"openapi": "3.0.1",
"info": {
"title": "User API",
"version": "1.0.0"
},
"paths": {
"/users/{id}": {
"get": {
"summary": "Get a user by ID",
"parameters": [
{
"name": "id",
"in": "path",
"required": true,
"schema": {
"type": "integer"
}
}
],
"responses": {
"200": {
"description": "User details",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/User"
}
}
}
},
"401": {
"description": "Unauthorized"
}
}
}
}
},
"components": {
"schemas": {
"User": {
"type": "object",
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
},
"address": {
"type": "object",
"properties": {
"city": {
"type": "string"
},
"zipCode": {
"type": "string"
}
},
"required": ["city", "zipCode"]
}
},
"required": ["id", "name", "address"]
}
}
}
}
For the JSON above, an example of path would be: "/users/{id}" - "get" - "200".
The Trusty Sidekick: The Ajv JSON Schema Validator
AJV (Another JSON Schema Validator) is a JavaScript library that validates data objects against a JSON Schema structure (Ajv official website).
Don’t be fooled by its name—it’s not just another schema validator. AJV is an fantastic tool!
In my opinion, AJV is a very versatile, powerful, fast, and well-maintained JSON Schema validator. On top of that, it has excellent documentation.
But no matter how amazing a Vigilante might be, they all have their weaknesses.
Setbacks I Experienced Using Ajv Schema Validator Out of the Box
Although Ajv is a plugin that's fairly easy to integrate into your Cypress frameworks, when I implemented schema validation for OpenAPI and Swagger using Ajv for a large number of APIs across multiple Cypress projects, I encountered a few setbacks:
-
The APIs to test were documented with the Swagger tool in OpenAPI 3.0.1 or Swagger 2.0 schema documents, each containing the full specification of the API (endpoints, methods, expected result statuses). However, Ajv requires the specific JSON schema object of the response to validate, so you cannot feed it a full schema document with the entire API definition.
As a result, I found myself repeatedly writing the same custom code and commands across all API test projects to: first, extract the specific schema to validate from the full schema document; and second, "massage" that schema into a format that Ajv could process.
Obviously, this is not very DRY (Don’t Repeat Yourself) principle-friendly!
-
I discovered that the way Ajv returns schema validation errors is not straightforward. Many of these errors are particularly complex to understand.
If you don't believe me, check out this example of schema errors provided by Ajv for a relatively simple schema with just two objects in the data response:
Ugh! Right?
So, what if we take the Ajv Schema Validator to the next level by integrating it seamlessly with Cypress, and providing the schema discrepancies results in a very user-friendly way that is easy to understand? This would enable easier and more robust validation in our testing workflow.
The new Vigilante on the block: The CYPRESS-AJV-SCHEMA-VALIDATOR Plugin
Afterward, it became clear to me that all of this could be turned into a plugin. However, I wanted it to be incredibly easy to use, requiring virtually no configuration.
The new cypress-ajv-schema-validator plugin is available for installation on NPM, and the open source code is available on GitHub.
Main Features
It includes a Cypress command
cy.validateSchema()and a utility functionvalidateSchema()to report JSON Schema validation errors in the response obtained from any network request withcy.request().The command
cy.validateSchema()is chainable and returns the original API response yielded.It supports schemas provided as plain JSON schema, OpenAPI 3.0.1 schema document, and Swagger 2.0 schema document, which can be sourced from a Cypress fixture.
It uses the Ajv JSON Schema Validator as its core engine.
It provides in the Cypress log a summary of the schema errors, as well as a list of the individual errors in the schema validation.
-
By clicking on the summary of schema errors in the Cypress log, the console will output:
- Number of schema errors.
- Full list of schema errors as provided by Ajv.
- A nested tree view of the validated data, clearly indicating the errors and where they occurred in an easy-to-understand format.
⭐⭐⭐⭐⭐
Note: This plugin would complement Filip Hric's cypress-plugin-api and Gleb Bahmutov's @bahmutov/cy-api plugins to perform JSON schema validations.Example usage with these two API plugins:
cy.api('/users/1').validateSchema(schema)
Installation
npm install cypress-ajv-schema-validator
// or
yarn add cypress-ajv-schema-validator
Compatibility
- Cypress 12.0.0 or higher
- Ajv 8.16.0 or higher
- ajv-formats 3.0.1 or higher
Configuration
Add the following lines either to your cypress/support/commands.js to include the custom command and function globally, or directly in the test file that will host the schema validation tests:
- For
cy.validateSchema()Custom Command:
import 'cypress-ajv-schema-validator'
- For
validateSchema()Function:
import validateSchema from 'cypress-ajv-schema-validator'
API Reference
Custom Command cy.validateSchema(schema, path)
Validates the given data against the provided schema.
-
Parameters
-
schema(object): The schema to validate against. Supported formats are plain JSON schema, Swagger, and OpenAPI documents. -
path(object, optional): The path object to the schema definition in a Swagger or OpenAPI document.-
endpoint(string, optional): The endpoint path. -
method(string, optional): The HTTP method. Defaults to 'GET'. -
status(integer, optional): The response status code. Defaults to 200.
-
-
-
Returns
-
Cypress.Chainable: The response object wrapped in a Cypress.Chainable.
-
-
Throws
-
Error: If any of the required parameters are missing or if the schema or schema definition is not found.
-
Example providing a Plain JSON schema:
cy.request('GET', 'https://awesome.api.com/users/1')
.validateSchema(schema);
Example providing an OpenAPI 3.0.1 or Swagger 2.0 schema documents:
cy.request('GET', 'https://awesome.api.com/users/1')
.validateSchema(schema, { endpoint: '/users/{id}', method: 'GET', status: 200 });
Function validateSchema(data, schema, path)
Validates the given data against the provided schema.
-
Parameters
-
data(any): The data to be validated. -
schema(object): The schema to validate against. -
path(object, optional): The path object to the schema definition in a Swagger or OpenAPI document.-
endpoint(string, optional): The endpoint path. -
method(string, optional): The HTTP method. Defaults to 'GET'. -
status(integer, optional): The response status code. Defaults to 200.
-
-
-
Returns
-
Array: An array of validation errors, or null if the data is valid against the schema.
-
-
Throws
-
Error: If any of the required parameters are missing or if the schema or schema definition is not found.
-
Example providing a Plain JSON schema:
cy.request('GET', 'https://awesome.api.com/users/1').then(response => {
const data = response.body
const errors = validateSchema(data, schema);
expect(errors).to.have.length(0); // Assertion to ensure no validation errors
});
Example providing an OpenAPI 3.0.1 or Swagger 2.0 schema documents:
cy.request('GET', 'https://awesome.api.com/users/1').then(response => {
const data = response.body
const errors = validateSchema(data, schema, { endpoint: '/users/{id}', method: 'GET', status: 200 });
expect(errors).to.have.length(0); // Assertion to ensure no validation errors
});
Usage Examples
cy.validateSchema() command with a Plain JSON schema.
describe('API Schema Validation with Plain JSON', () => {
it('should validate the user data using plain JSON schema', () => {
const schema = {
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "number" }
},
"required": ["name", "age"]
};
cy.request('GET', 'https://awesome.api.com/users/1')
.validateSchema(schema)
.then(response => {
// Further assertions
});
});
});
cy.validateSchema() command with a Swagger 2.0 schema document.
describe('API Schema Validation with Swagger 2.0', () => {
it('should validate the user data using Swagger 2.0 schema', () => {
const schema = {
"swagger": "2.0",
"paths": {
"/users/{id}": {
"get": {
"responses": {
"200": {
"schema": { "$ref": "#/definitions/User" }
}
}
}
}
},
"definitions": {
"User": {
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "number" }
},
"required": ["name", "age"]
}
}
};
const path = { endpoint: '/users/{id}', method: 'GET', status: 200 };
cy.request('GET', 'https://awesome.api.com/users/1')
.validateSchema(schema, path)
.then(response => {
// Further assertions
});
});
});
validateSchema() function with an OpenAPI 3.0.1 schema document.
import validateSchema from 'cypress-ajv-schema-validator';
describe('API Schema Validation Function', () => {
it('should validate the user data using validateSchema function', () => {
const schema = {
"openapi": "3.0.1",
"paths": {
"/users/{id}": {
"get": {
"responses": {
"200": {
"content": {
"application/json": {
"schema": { "$ref": "#/components/schemas/User" }
}
}
}
}
}
}
},
"components": {
"schemas": {
"User": {
"type": "object",
"properties": {
"name": { "type": "string" },
"age": { "type": "number" }
},
"required": ["name", "age"]
}
}
}
};
const path = { endpoint: '/users/{id}', method: 'GET', status: 200 };
cy.request('GET', 'https://awesome.api.com/users/1').then(response => {
const errors = validateSchema(response.body, schema, path);
expect(errors).to.have.length(0); // Assertion to ensure no validation errors
});
});
});
For more detailed usage examples, check the document USAGE-EXAMPLES.md in the plugin GitHub repository.
Validation Results
One of the key strengths of the cypress-ajv-schema-validator plugin is its ability to clearly display validation results within the Cypress log. This feature makes debugging much simpler and more efficient, as it provides immediate, actionable insights right where you need them. Whether your tests pass or fail, the plugin ensures that the feedback is both comprehensive and user-friendly. Let's explore how this works.
Test Passed
When a test passes, the Cypress log will show the message: "✔️ PASSED - THE RESPONSE BODY IS VALID AGAINST THE SCHEMA.".
Test Failed
When a test fails, the Cypress log will display the message: "❌ FAILED - THE RESPONSE BODY IS NOT VALID AGAINST THE SCHEMA," indicating the total number of errors: (Number of schema errors: N).
Additionally, the Cypress log will provide entries for each individual schema validation error as identified by AJV. Errors related to missing fields in the validated data are marked with the symbol 🗑️, while other types of errors are flagged with the symbol 👉.
Detailed Error View in the Console
For a more in-depth look, open the Console in your browser's DevTools and click on the summary line for the schema validation errors in the Cypress log. The console will reveal detailed information about all the errors, including:
- The total number of errors.
- The full list of errors as reported by AJV.
- A user-friendly view of the validated data, highlighting the exact location and reason for each validation error.
The plugin continues to provide the list of schema errors in Ajv format, but it also offers the response data that was validated, indicating exactly where the errors are located and precisely what those errors are.
This content can be copied directly from the console and provided elsewhere in a JSON file.
[
{
"id": "👉 \"abc123\" must match format \"uuid\"",
"description": "👉 null must be string",
"createdDate": "👉 \"abc123\" must match format \"date-time\"",
"priority": 3,
"completed": true,
"details": {
"detail1": "👉 789 must be string",
"detail2": "🗑️ Missing property \"detail2\""
}
},
{
"id": "👉 null must be string",
"description": "New Entry",
"completed": "👉 \"false\" must be boolean",
"priority": "👉 true must be integer",
"details": {
"detail2": 260
},
"createdDate": "🗑️ Missing property \"createdDate\""
}
]
By presenting validation results in this clear and structured manner, the cypress-ajv-schema-validator plugin not only makes your testing workflow more robust but also saves you valuable time and effort in identifying and resolving issues.
ACT3: RESOLUTION
What an adventure it's been digging into the world of vigilant testing! Just like a vigilante swooping in to restore order in a chaotic city, the cypress-ajv-schema-validator plugin ensures your testing environment stays neat, efficient, and just. With its precise JSON schema validation and clear, user-friendly error reporting, this plugin is the hero your API testing workflow deserves.
Imagine AJV as your trusty sidekick, always ready to catch even the most elusive validation errors. And with our plugin, navigating the treacherous streets of API testing becomes a lot less daunting and a lot more exhilarating. Whether you're dealing with JSON schemas, or complete OpenAPI and Swagger documents, this plugin steps in to maintain law and order, ensuring that no violation goes unnoticed.
So don your tester's cape, and add cypress-ajv-schema-validator to your arsenal. With this powerful tool, you'll be ready to tackle any challenge that comes your way, maintaining justice and peace in the realm of API testing.
Stay vigilant, stay validated!
Don't forget to leave a comment, give a thumbs up, or follow my Cypress blog if you found this new plugin and this post useful. Your feedback and support are always appreciated!
Top comments (4)
I like this solution better than Optic's verification of e2e suites against the openapi docs.
If you already have some cy, you just slightly enhance what you have and you have schema testing, this is very low cost compared to other solutions.
Here's how I did added it to a suite of tests github.com/muratkeremozcan/pact-js....
We will probably start adopting this at mass at work because we have 40+ services, all having api e2e cy, and all having manually created OpenAPI specs... Oh boi...
Thanks @muratkeremozcan!
Glad that you liked the plugin.
Hey Sebastian, Excellent plugin that truly simplifies API contract validation in Cypress! chainable cy.validateSchema() command is very promising along with support for multiple schema formats (plain JSON, OpenAPI, and Swagger) with examples its a versatile tool for any project. Great job on creating such a user-friendly and efficient solution for ensuring API reliability!
Thanks Dhruv! I had lot of fun implementing it :)