Use Case for Balancing Code Duplication and Complexity for Effective Test Automation
(Cover image from pexels.com by Pixabay)
ACT 1: EXPOSITION
It's interesting how divided the automation world is about the use of loops when implementing tests. Some prefer to put the direct assertions in the test, even if it means repeating the same code again and again with just changes to the asserted data. They believe this approach is clearer. Others are definitely against duplicating code and are firm proponents of the DRY (Don't Repeat Yourself) principle to the fullest extent.
However, I believe that any radical perspective in either approach will never be optimally effective, as each has its justified use cases. In my opinion, for code clarity and easy maintenance of your test suite, overusing the duplication of almost identical assertions can be as counterproductive as having complicated loops that require PhD in Theoretic Computer Science to understand. 🤯
But wouldn’t it be amazing if we could find a way of creating Dynamic Tests in Cypress that would be super easy not just to implement but also to maintain, or even scale effortlessly if needed?
ACT 2: CONFRONTATION
There is one particular case (although not the only one) that I believe would benefit from the approach of Dynamic Tests: testing APIs with multiple possible combinations of inputs that have to be validated.
Our Case:
I have an API where you provide in the request body up to 4 parameters: account, ipAddress, phone, and email.
Depending on certain combinations of valid or invalid values of those parameters, the request will be approved or rejected. The logic implemented in the backend that we need to test is: if the account is valid or the ipAddress is valid or both the phone and email are valid, then the request is approved; otherwise, it is rejected.
So we will need to test various case scenarios (Input -> Output):
- We provide a valid
account-> Approved - We provide an invalid
account-> Rejected - We provide a valid
ipAddress-> Approved - We provide an invalid
ipAddress-> Rejected - We provide a valid
phoneand a validemail-> Approved - We provide an invalid
phoneand a validemail-> Rejected - We provide a valid
phoneand an invalidemail-> Rejected - We provide an invalid
phoneand an invalidemail-> Rejected
Easy, right?
But you really do not know the logic that was implemented in the backend (especially in black box testing).
What should happen if we provide in the same request a valid account and an invalid ipAddress? According to our specification above, it should be approved.
But how can you ensure this is how it was implemented if you do not test it? Maybe, just maybe, the backend developer made an error, and in the event that both parameters are provided in the body (account and ipAddress), both have to be valid in order to approve the request (in other words, they implemented a logic AND instead of an OR).
So we will also need to test:
- We provide a valid
accountand a validipAddress-> Approved - We provide a valid
accountand an invalidipAddress-> Approved - We provide an invalid
accountand a validipAddress-> Approved - We provide an invalid
accountand an invalidipAddress-> Rejected
... and so on. You get it, right?
Let's now write the code using plain assertions without using any kind of loop operation. The code would look something like this:
File: /e2e/apiValidation.cy.js
const method = 'POST'
const url = '/api/endpoint'
describe('API Tests without Loops', () => {
it('should approve with a valid account', () => {
cy.request({
method,
url,
body: { account: 'valid-account#' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'approved');
});
});
it('should reject with an invalid account', () => {
cy.request({
method,
url,
body: { account: 'invalid-account#' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'rejected');
});
});
it('should approve with a valid ipAddress', () => {
cy.request({
method,
url,
body: { ipAddress: 'VA.LID.IP' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'approved');
});
});
it('should reject with an invalid ipAddress', () => {
cy.request({
method,
url,
body: { ipAddress: 'IN.VA.LID.IP' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'rejected');
});
});
it('should approve with a valid phone and a valid email', () => {
cy.request({
method,
url,
body: { phone: '(555) Valid-Phone', email: 'valid@email.com' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'approved');
});
});
it('should reject with an invalid phone and a valid email', () => {
cy.request({
method,
url,
body: { phone: '(555) Invalid-Phone', email: 'valid@email.com' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'rejected');
});
});
it('should reject with a valid phone and an invalid email', () => {
cy.request({
method,
url,
body: { phone: '(555) Valid-Phone', email: 'invalid@email.com' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'rejected');
});
});
it('should reject with an invalid phone and an invalid email', () => {
cy.request({
method,
url,
body: { phone: '(555) Invalid-Phone', email: 'invalid@email.com' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'rejected');
});
});
it('should approve with a valid account and a valid ipAddress', () => {
cy.request({
method,
url,
body: { account: 'valid-account#', ipAddress: 'VA.LID.IP' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'approved');
});
});
it('should approve with a valid account and an invalid ipAddress', () => {
cy.request({
method,
url,
body: { account: 'valid-account#', ipAddress: 'IN.VA.LID.IP' }
}).then((response) => {
expect(response.status).to.eq(200);
expect(response.body).to.have.property('status', 'approved');
});
});
// And many more tests according our test plan
// [...]
});
Now think about maintaining this suite, especially if, for whatever business reasons, "the product" decides to change the validation logic so that when a valid account is provided, a valid ipAddress also needs to be provided to approve the request.
This could become quite a mess. 🧻
What if... we create a data structure that includes the mapped combinations for both approved and rejected requests, along with the names of the use cases they represent? We could also put that data structure in a Cypress fixture. Why not?
I know... but just bear with me for now.
File: /fixtures/apiValidationUseCases.json
[
{
"nameUseCase": "Valid account",
"requestBody": {"account": "valid-account#"},
"responseBody": {"status": "approved"}
},
{
"nameUseCase": "Invalid account",
"requestBody": {"account": "invalid-account#"},
"responseBody": {"status": "rejected"}
},
{
"nameUseCase": "Valid ipAddress",
"requestBody": {"ipAddress": "VA.LID.IP"},
"responseBody": {"status": "approved"}
},
{
"nameUseCase": "Invalid ipAddress",
"requestBody": {"ipAddress": "IN.VA.LID.IP"},
"responseBody": {"status": "rejected"}
},
{
"nameUseCase": "Valid phone and valid email",
"requestBody": {"phone": "(555) Valid-Phone", "email": "valid@email.com"},
"responseBody": {"status": "approved"}
},
{
"nameUseCase": "Invalid phone and valid email",
"requestBody": {"phone": "(555) Invalid-Phone", "email": "valid@email.com"},
"responseBody": {"status": "rejected"}
},
{
"nameUseCase": "Valid phone and invalid email",
"requestBody": {"phone": "(555) Valid-Phone", "email": "invalid@email.com"},
"responseBody": {"status": "rejected"}
},
{
"nameUseCase": "Invalid phone and invalid email",
"requestBody": {"phone": "(555) Invalid-Phone", "email": "invalid@email.com"},
"responseBody": {"status": "rejected"}
},
{
"nameUseCase": "Valid account and valid ipAddress",
"requestBody": {"account": "valid-account#", "ipAddress": "VA.LID.IP"},
"responseBody": {"status": "approved"}
},
{
"nameUseCase": "Valid account and invalid ipAddress",
"requestBody": {"account": "valid-account#", "ipAddress": "IN.VA.LID.IP"},
"responseBody": {"status": "approved"}
},
// And the rest of all our use cases
// [...]
]
If you need to add more use cases to test in the future or even change the logic of the conditions for approval or rejection, it would be fairly simple to do in this JSON data structure, as the human brain is able to process mapped data in a list or tabular form quite easily.
Now we only need to write our test suite in the following manner:
File: /e2e/apiValidationDynamic.cy.js
import useCases from '../fixtures/apiValidationUseCases.json';
const method = 'POST'
const url = '/api/endpoint'
describe('API Tests with Dynamic Fixtures', () => {
for (let { nameUseCase, requestBody, responseBody } of useCases) {
it(`Use Case: ${nameUseCase}`, () => {
cy.request({
method,
url,
body: requestBody
}).then(response => {
expect(response.status).to.eq(200);
expect(response.body).to.deep.equal(responseBody)
});
});
}
});
Notice that the .it() function is within the loop, which is key!
This way, if one use case fails, the test suite will continue running tests for the rest of the use cases instead of aborting at the first test that fails.
Maybe, just maybe, you have just started looking at Dynamic Tests and loops with kinder eyes? 😍
ACT3: RESOLUTION
At this point, the choice of whether to embrace loops or not is in the capable hands of our diligent QA engineers.
While loops can turn a mess into a neatly organized suite of tests, every scenario is unique, and sometimes sticking with plain assertions might feel just right. After all, finding the perfect balance between simplicity and efficiency is an art.
So, whether you decide to loop or not to loop, remember that the ultimate goal is to ensure robust, maintainable tests. And hey, a little bit of code repetition never hurt anyone... except maybe the next person maintaining your code! 😉
Don't forget to follow me, leave a comment, or give a thumbs up if you found this post useful or insightful.
Happy reading!
Top comments (9)
I did something like that in a recent commit:
github.com/wlsf82/EngageSphere/com...
Exactly @walmyrlimaesilv ! It's a super powerful approach!
🙌
This is so cool. I'd love to implement this somehow. Are there any downsides? My first thought is that if you had a large variety of tests, the fixture file could get more hairy than helpful. Have you run into a scenario where you decided that it's better not to loop the tests this way?
Glad you like the article @jmosley.
And you are right, when the test data could become cumbersome to maintain. Although it was not the purpose of the article to tackle that (I think might be a very interesting article it self), it's a very fair point.
The loop article is an over simplification of how you can implement certain type of tests, with the upside that if you have the loop OUTSIDE the it(), failing one assertion does not fail all the others, so you can get the most of your test (and most likely) avoid additional runs and annoying debugs to figure out all the use cases that failed.
Also illustrate that you can create test data in a tabular view (easier to assimilate by our brain), instead of having a huge test suite with code that is basically the same. So at first sight you know that all tests do exactly the same thing (with no exceptions: no need to read all tests in the large suite). I believe because of those things pointed out, this approach by itself is very useful.
Maintaining a large test sample (specially with many permutations) it's its own beat by it self as you know, independently if you use loop or not. So even if you do not use loops an a fixture, you would need to tackle that issue anyway.
If what you need to test is all combinations, as somebody pointed out in a comment you can use multiple loops one inside the other. However this will not work is you are not testing all permutations.
All those are tricks or techniques that the QA will need to evaluate, and probably adapt or tweak depending on what they need to test. The typical one solution that does not fit all. 😏
If you liked the article, please remember to leave a reaction to support it: ❤️, 🦄,... or what ever is your favorite 😄
In my recent project, I was thinking of some way to handle this type of different validations, but I didn't know how to do it reliably, and maintainable.
I will definitely try this!
Thanks @kendall_vargas for your input. Glad the blog can be of help!
This is great. We have a slightly different approach here.
So what we have is a file based API request defined. say fixtures/RestApiFiles/sampleRequest.json
now this file has:
REQUEST_TYPE:
URL:
HEADERS:
BODY:
EXPECTED_STATUS_CODE:
EXPECTED_RESPONSE_BODY:
etc, note the BODY can be a json or it can also reference another json file when body is huge.
and we have a wrapper method defined which takes in this file path fixtures/RestApiFiles/sampleRequest.json as an argument, constructs the request and validates the response.
Now for tests which need input / output modifications what we have is dynamic_changes argument (which is a json type so keys are entirely optional). so if you need to the url to invalid you just do dynamic_changes = {URL: '', EXPECTED_STATUS_CODE: 404} . you get the idea.
The benefit of this approach is you have total control over everything in each request. you can literally change very minute things in each api call.
Yeah, there are many different ways to approach DDT with dynamic tests. I also have some scripts where I have part of the data the headers content (both req and response), the body a more complex element, and 401, 403, 500 expected responses.
Additionally I chain the response with the cypress-ajv-shema-validator to check the response json schema including for failed requests. :)