A Guide to Load Testing Node.js APIs with Artillery

Artillery is an open-source command-line tool purpose-built for load testing and smoke testing web applications. It is written in JavaScript and it supports testing HTTP, Socket.io, and WebSockets APIs.

This article will get you started with load testing your Node.js APIs using Artillery. You'll be able to detect and fix critical performance issues before you deploy code to production.

Before we dive in and set up Artillery for a Node.js app, though, let's first answer the question: what is load testing and why is it important?

Why Should You Do Load Tests in Node.js?

Load testing is essential to quantify system performance and identify breaking points at which an application starts to fail. A load test generally involves simulating user queries to a remote server.

Load tests reproduce real-world workloads to measure how a system responds to a specified load volume over time. You can determine if a system behaves correctly under loads it is designed to handle and how adaptable it is to spikes in traffic. It is closely related to stress testing, which assesses how a system behaves under extreme loads and if it can recover once traffic returns to normal levels.

Load testing can help validate if an application can withstand realistic load scenarios without a degradation in performance. It can also help uncover issues like:

• Increased response times
• Memory leaks
• Poor performance of various system components under load

As well as other design issues that contribute to a suboptimal user experience.

In this article, we'll focus on the free and open-source version of Artillery to explore load testing. However, bear in mind that a pro version of Artillery is also available for those whose needs exceed what can be achieved through the free version. It provides added features for testing at scale and is designed to be usable even if you don't have prior DevOps experience.

Installing Artillery for Node.js

Artillery is an npm package so you can install it through npm or yarn:

$ yarn global add artillery

If this is successful, the artillery program should be accessible from the command line:

$ artillery -V
        ___         __  _ ____                  _
  _____/   |  _____/ /_(_) / /__  _______  __  (_)___  _____
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ / / / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__, (_)_/\____/
                                    /____/

------------ Version Info ------------
Artillery: 1.7.7
Artillery Pro: not installed (https://artillery.io/pro)
Node.js: v16.7.0
OS: linux/x64
--------------------------------------

Basic Artillery Usage

Once you've installed the Artillery CLI, you can start using it to send traffic to a web server. It provides a quick subcommand that lets you run a test without writing a test script first.

You'll need to specify:

• an endpoint
• the rate of virtual users per second or a fixed amount of virtual users
• how many requests should be made per user

$ artillery quick --count 20 --num 10 http://localhost:4000/example

The --count parameter above specifies the total number of virtual users, while --num indicates the number of requests that should be made per user. Therefore, 200 (20*10) GET requests are sent to the specified endpoint. On successful completion of the test, a report is printed out to the console.

All virtual users finished
Summary report @ 14:46:26(+0100) 2021-08-29
  Scenarios launched:  20
  Scenarios completed: 20
  Requests completed:  200
  Mean response/sec: 136.99
  Response time (msec):
    min: 0
    max: 2
    median: 1
    p95: 1
    p99: 2
  Scenario counts:
    0: 20 (100%)
  Codes:
    200: 200

This shows several details about the test run, such as the requests completed, response times, time taken for the test, and more. It also displays the response codes received on each request so that you can determine if your API handles failures gracefully in cases of overload.

While the quick subcommand is handy for performing one-off tests from the command line, it's quite limited in what it can achieve. That's why Artillery provides a way to configure different load testing scenarios through test definition files in YAML or JSON formats. This allows great flexibility to simulate the expected flows at one or more of your application's endpoints.

Writing Your First Artillery Test Script

In this section, I'll demonstrate a basic test configuration that you can apply to any application. If you want to follow along, you can set up a test environment for your project, or run the tests locally so that your production environment is not affected. Ensure you install Artillery as a development dependency so that the version you use is consistent across all deployments.

$ yarn add -D artillery

An Artillery test script consists of two main sections: config and scenarios. config includes the general configuration settings for the test such as the target, response timeouts, default HTTP headers, etc. scenarios consist of the various requests that virtual users should make during a test. Here's a script that tests an endpoint by sending 10 virtual users every second for 30 seconds:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 30
      arrivalRate: 10

scenarios:
  - name: "Retrieve data"
    flow:
      - get:
          url: "/example"

In the above script, the config section defines the base URL for the application that's being tested in the target property. All the endpoints defined later in the script will run against this base URL.

The phases property is then used to set up the number of virtual users generated in a period of time and how frequently these users are sent to specified endpoints.

In this test, duration determines that virtual users will be generated for 30 seconds and arrivalRate determines the number of virtual users sent to the endpoints per second (10 users).

On the other hand, the scenarios section defines the various operations that a virtual user should perform. This is controlled through the flow property, which specifies the exact steps that should be executed in order. In this case, we have a single step: a GET request to the /example endpoint on the base URL. Every virtual user that Artillery generates will make this request.

Now that we've written our first script, let's dive into how to run a load test.

Running a Load Test in Artillery

Save your test script to a file (such as load-test.yml) and execute it through the command below:

$ artillery run path/to/script.yml

This command will start sending virtual users to the specified endpoints at a rate of 10 requests per second. A report will be printed to the console every 10 seconds, informing you of the number of test scenarios launched and completed within the time period, and other statistics such as mean response time, HTTP response codes, and errors (if any).

Once the test concludes, a summary report (identical to the one we examined earlier) is printed out before the command exits.

All virtual users finished
Summary report @ 15:38:48(+0100) 2021-09-02
  Scenarios launched:  300
  Scenarios completed: 300
  Requests completed:  300
  Mean response/sec: 9.87
  Response time (msec):
    min: 0
    max: 1459
    median: 1
    p95: 549.5
    p99: 1370
  Scenario counts:
    Retrieve data: 300 (100%)
  Codes:
    200: 300

How to Create Realistic User Flows

The test script we executed in the previous section is not very different from the quick example in that it makes requests to only a single endpoint. However, you can use Artillery to test more complex user flows in an application.

In a SaaS product, for example, a user flow could be: someone lands on your homepage, checks out the pricing page, and then signs up for a free trial. You'll definitely want to find out how this flow will perform under stress if hundreds or thousands of users are trying to perform these actions at the same time.

Here's how you can define such a user flow in an Artillery test script:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
      name: "Warming up"
    - duration: 240
      arrivalRate: 20
      rampTo: 100
      name: "Ramping up"
    - duration: 500
      arrivalRate: 100
      name: "Sustained load"
  processor: "./processor.js"

scenarios:
  - name: "Sign up flow"
    flow:
      - get:
          url: "/"
      - think: 1
      - get:
          url: "/pricing"
      - think: 2
      - get:
          url: "/signup"
      - think: 3
      - post:
          url: "/signup"
          beforeRequest: generateSignupData
          json:
            email: "{{ email }}"
            password: "{{ password }}"

In the above script, we define three test phases in config.phases:

• The first phase sends 20 virtual users per second to the application for 60 seconds.
• In the second phase, the load will start at 20 users per second and gradually increase to 100 users per second over 240 seconds.
• The third and final phase simulates a sustained load of 100 users per second for 500 seconds.

By providing several phases, you can accurately simulate real-world traffic patterns and test how adaptable your system is to a sudden barrage of requests.

The steps that each virtual user takes in the application are under scenarios.flow. The first request is GET / which leads to the homepage. Afterward, there is a pause for 1 second (configured with think) to simulate user scrolling or reading before making the next GET request to /pricing. After a further delay of 2 seconds, the virtual user makes a GET request to /signup. The last request is POST /signup, which sends a JSON payload in the request body.

The {{ email }} and {{ password }} placeholders are populated through the generateSignupData function, which executes before the request is made. This function is defined in the processor.js file referenced in config.processor. In this way, Artillery lets you specify custom hooks to execute at specific points during a test run. Here are the contents of processor.js:

const Faker = require('faker');

function generateSignupData(requestParams, ctx, ee, next) {
  ctx.vars['email'] = Faker.internet.exampleEmail();
  ctx.vars['password'] = Faker.internet.password(10);

  return next();
}

module.exports = {
  generateSignupData,
};

The generateSignupData function uses methods provided by Faker.js to generate a random email address and password each time it is called. The results are then set on the virtual user's context, and next() is called so that the scenario can continue to execute. You can use this approach to inject dynamic random content into your tests so they're as close as possible to real-world requests.

Note that other hooks are available aside from beforeRequest, including the following:

• afterResponse - Executes one or more functions after a response has been received from the endpoint:

- post:
    url: "/login"
    afterResponse:
      - "logHeaders"
      - "logBody"

• beforeScenario and afterScenario - Used to execute one or more functions before or after each request in a scenario:

scenarios:
  - beforeScenario: "setData"
    afterScenario: "logResults"
    flow:
      - get:
          url: "/auth"

• function - Can execute functions at any point in a scenario:

- post:
    url: "/login"
    function: "doSomething"

Injecting Data from a Payload File

Artillery also lets you inject custom data through a payload file in CSV format. For example, instead of generating fake email addresses and passwords on the fly as we did in the previous section, you can have a predefined list of such data in a CSV file:

[e-mail],rwkWspKUKy
[e-mail],7BaFHbaWga
[e-mail],CWvc6Bznnh
[e-mail],1vlT_02i6h
[e-mail],1N0PRraQU7

To access the data in this file, you need to reference it in the test script through the config.payload.path property. Secondly, you need to specify the names of the fields you'd like to access through config.payload.fields. The config.payload property provides several other options to configure its behavior, and it's also possible to specify multiple payload files in a single script.

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./auth.csv"
    fields:
      - "email"
      - "password"

scenarios:
  - name: "Authenticating users"
    flow:
      - post:
          url: "/login"
          json:
            email: "{{ email }}"
            password: "{{ password }}"

Capturing Response Data From an Endpoint

Artillery makes it easy to capture the response of a request and reuse certain fields in a subsequent request. This is helpful if you're simulating flows with requests that depend on an earlier action's execution.

Let's assume you're providing a geocoding API that accepts the name of a place and returns its longitude and latitude in the following format:

{
  "longitude": -73.935242,
  "latitude": 40.730610
}

You can populate a CSV file with a list of cities:

Seattle
London
Paris
Monaco
Milan

Here's how you can configure Artillery to use each city's longitude and latitude values in another request. For example, you can use the values to retrieve the current weather through another endpoint:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./cities.csv"
    fields:
      - "city"

scenarios:
  - flow:
      - get:
          url: "/geocode?city={{ city }}"
          capture:
            - json: "$.longitude"
              as: "lon"
            - json: "$.latitude"
              as: "lat"
      - get:
          url: "/weather?lon={{ lon }}&lat={{ lat }}"

The capture property above is where all the magic happens. It's where you can access the JSON response of a request and store it in a variable to reuse in subsequent requests. The longitude and latitude properties from the /geocode response body (with the aliases lon and lat, respectively) are then passed on as query parameters to the /weather endpoint.

Using Artillery in a CI/CD Environment

An obvious place to run your load testing scripts is in a CI/CD pipeline so that your application is put through its paces before being deployed to production.

When using Artillery in such environments, it's necessary to set failure conditions that cause the program to exit with a non-zero code. Your deployment should abort if performance objectives are not met. Artillery provides support for this use case through its config.ensure property.

Here's an example that uses the ensure setting to assert that 99% of all requests have an aggregate response time of 150 milliseconds or less and that 1% or less of all requests are allowed to fail:

config:
  target: "https://example.com"
  phases:
    - duration: 60
      arrivalRate: 20
  ensure:
    p99: 150
    maxErrorRate: 1

Once you run the test, it will continue as before, except that assertions are verified at the end of the test and cause the program to exit with a non-zero exit code if requirements are not met. The reason for a test failure is printed at the bottom of the summary report.

All virtual users finished
Summary report @ 07:45:48(+0100) 2021-09-03
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 4
  Response time (msec):
    min: 1
    max: 487
    median: 2
    p95: 443.5
    p99: 487
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20

ensure condition failed: ensure.p99 < 200

Aside from checking the aggregate latency, you can also run assertions on min, max, and median — the minimum, maximum, and median response times, respectively. Here's how to assert that requests never take more than 500 milliseconds to complete during a test run:

config:
  ensure:
    max: 500

The report for a failed test will indicate the reason for failure:

All virtual users finished
Summary report @ 08:29:59(+0100) 2021-09-03
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 3.64
  Response time (msec):
    min: 1
    max: 603
    median: 305.5
    p95: 602.5
    p99: 603
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20

ensure condition failed: ensure.max < 500

Generating Status Reports in Artillery

Artillery prints a summary report for each test run to the standard output, but it's also possible to output detailed statistics for a test run into a JSON file by utilizing the --output flag:

$ artillery run config.yml --output test.json

Once the test completes, its report is placed in a test.json file in the current working directory. This JSON file can be visualized through Artillery's online report viewer or converted into an HTML report through the report subcommand:

$ artillery report --output report.html test.json
Report generated: report.html

You can open the report.html file in your browser to view a full report of the test run. It includes tables and several charts that should give you a good idea of how your application performed under load:

Extending Artillery With Plugins

Artillery's built-in tools for testing HTTP, Socket.io, and Websocket APIs can take you quite far in your load testing process. However, if you have additional requirements, you can search for plugins on NPM to extend Artillery's functionality.

Here are some official Artillery plugins that you might want to check out:

• artillery-plugin-expect: Helps with adding expectations to HTTP requests for functional or acceptance testing.
• artillery-plugin-publish-metrics: Used to send statistics from test runs to some external monitoring and observability systems.
• artillery-plugin-fuzzer: Helps you fuzz test your APIs with random and unexpected payloads to your API endpoints so you can catch errors. It is based on the Big List Of Naughty Strings.
• artillery-plugin-metrics-by-endpoint: Breaks down response time metrics by endpoint rather than displaying aggregate values across all endpoints.

You can also extend Artillery by creating your own plugins.

Use Artillery for Node.js Apps to Avoid Downtime

In this article, we've described how you can set up a load testing workflow for your Node.js applications with Artillery. This setup will ensure that your application performance stays predictable under various traffic conditions. You'll be able to account well for traffic-heavy periods and avoid downtime, even when faced with a sudden influx of users.

We've covered a sizeable chunk of what Artillery can do for you, but there's still lots more to discover.

Ensure you read the Artillery official documentation to learn about the other features on offer.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him through his blog.