Load testing 3scale

I was interested in doing some load testing against 3scale. After a bit of digging around I found that there was an open source load testing toolkit in https://github.com/3scale-labs/perftest-toolkit.

At first look I was unsure what this gave me or what it required. So after playing around with it a while I eventually figured it out along with a few pain points.

What do we need?

Prerequisites:

• An Openshift Kubernetes cluster
• An instance of 3scale on this cluster
• An ec2 instance that can connect a route on your cluster. It also can connect to your local pc via ssh. As your load testing this ec2 instance needs to be resourced to handle the load e.g.c5.xlarge.
• Ansible 2.9.14 (this is used by the repo to deploy resources)
• An endpoint like the echo-api endpoint, the following commands below to deploy an echo-api in an echo-api namespace

oc new-project echo-api
cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    metadata:
      labels:
        app: echo-api
    spec:
      containers:
        - name: echo-api
          image: quay.io/3scale/echoapi:stable
          livenessProbe:
            tcpSocket:
              port: 9292
            initialDelaySeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /test/200
              port: 9292
            initialDelaySeconds: 15
            timeoutSeconds: 1
          ports:
            - containerPort: 9292
              protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: echo-api
spec:
  ports:
  - name: echo-api-port
    port: 9292
    protocol: TCP
    targetPort: 9292
  selector:
    app: echo-api
  type: ClusterIP
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: echo-api
  namespace: echo-api
spec:
  to:
    kind: Service
    name: echo-api
  port:
    targetPort: echo-api-port
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow
  wildcardPolicy: None
---
EOF

What does it give us?

Well the perftest-toolkit creates 3scale resources like backends and products.

It installs Hyperfoil(cli load testing tool)locally and on the remote ec2 instance. This can be accessed via the browser on the ec2 instance public address via port 8090 or directly on the ec2 instances

It uploads a benchmark yaml file like the following

name: 3scale-benchmark
agents:
  ec2-agent: 
    host: ec2-54-93-91-242.eu-central-1.compute.amazonaws.com
    port: 22
http:
- host: https://perf-test-dwxpoent-3scale-apicast-production.apps.aucunnin-ccs.lpi0.s1.devshift.org:443
  sharedConnections: 50
usersPerSec: 228
duration: 3600s
maxDuration: 3600s
scenario:
- testSequence:
  - randomCsvRow:
     file: /tmp/3scale.csv
     skipComments: 'True'
     removeQuotes: 'True'
     columns:
       0: target-host
       1: uri
  - template:
      pattern: ${target-host}:443
      toVar: target-authority
  - httpRequest:
      authority:
        fromVar: target-authority
      GET:
        fromVar: uri
      headers:
        HOST:
          fromVar: target-host

threads: 4

It also uploads a csv with endpoint data to the /tmp directory on the ec2 instance which is consumed by the benchmark above.

cat 3scale.csv 
"perf-test-dwxpoent-3scale-apicast-production.apps.aucunnin-ccs.lpi0.s1.devshift.org","/pets?user_key=<some_user_key_from_3scale>"

It will run the test automatically and returns a html report when done e.g. 0001.html

Pain Points

Firstly a couple of pain points. I needed to install Docker on the ec2 instance as it was a Redhat RHEL 8 AIM I used the following commands on the ec2 instance

sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo &&
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin &&
sudo systemctl start docker &&
sudo usermod -aG docker $USER &&
newgrp docker

The version of Hyperfoil I was using requires id_rsa key pair on the ec2 instance, so I generated them and added the public key to the authorized_keys

cd ~/.ssh/
ssh-keygen -t rsa
cat id_rsa.pub >> authorized_keys

There are few Ansible playbooks to run as part of the perftest-toolkit but I hit a few blockers when trying to run them. In the end I ended up using ansible-galaxy to install the missing collections

# You may not have to do this ansible-galaxy collection list will show you if the are installed or not
ansible-galaxy collection install ansible.posix
ansible-galaxy collection install community.docker
ansible-galaxy collection install community.general

In deployment/ansible.cfg I needed to point the ssh key to allow ansible to access the ec2 instance via ssh so added the [ssh_connection] section. Updating this was in the readme but was missing the key

# project specific configuration for Ansible 

[defaults]
inventory_plugins=inventory
command_warnings=False

[inventory]
enable_plugins=3scale_inventory_plugin,ini

[ssh_connection]
ssh_args = -o ServerAliveInterval=30 -i "~/.ssh/pem_key_name.pem"
pipelining = True

In deployment/roles/platform-setup/vars/RedHat.yml need to set the java version

---
java_package:
  - java-11-openjdk
unzip_package:
  - unzip

One final pain point found that running Hyperfoil for longer runs 4 to 6 hours ended hitting up against some Garbage collection issue

Error [
"ec2-agent: Jitter watchdog was not invoked for 109 ms; check GC settings.",
"ec2-agent: Agent unexpectedly left the cluster."
]

Got around this by increasing the Java memory settings with extras: -XX:+UseShenandoahGC -Xms1G -Xmx4G in the hyperfoil benchmark to the max my ec2 instance could handle in this case 4GB. I updated deployment/benchmark/3scale-benchemark.yaml.j2 as follows

Getting started

Run the requirements playbook to install Hyperfoil locally

git clone https://github.com/3scale-labs/perftest-toolkit
cd deployment
ansible-galaxy install -r requirements.yaml

Next we edit a couple of files. First deployment/hosts we add in the ec2 instances public address

upstream ansible_host=ec2-54-93-91-242.eu-central-1.compute.amazonaws.com ansible_user=ec2-user

[hyperfoil_controller]
ec2 ansible_host=ec2-54-93-91-242.eu-central-1.compute.amazonaws.com ansible_user=ec2-user

[hyperfoil_agent]
ec2-agent ansible_host=ec2-54-93-91-242.eu-central-1.compute.amazonaws.com ansible_user=ec2-user

In the deployment/run.yml we set

• shared_connections (Hyperfoil setting number of concurrent connections to your load testing endpoint. A pain point if set to low Hyperfoil will fail the load test and if set to high can overload the test, you have to play around with it. Found for a 5 million request per day shared_connection of 30 worked where as 100 we see an increase in the 4xx error rate)
• users_per_sec (rate per second e.g. 5 million request per day 5000,000÷60÷60÷24 = 57.87 so round to 58 rps)
• duration_sec (length of load test run in seconds)

---
- hosts: hyperfoil_controller
  roles:
    - hyperfoil.hyperfoil_test
  vars:
    test_name: 3scale-benchmark
    test_files:
      - "{{ csv_dest_file_path }}"
    shared_connections: 50
    users_per_sec: 58
    duration_sec: 3600
- hosts: hyperfoil_controller
  become: yes
  roles:
    - hyperfoil_generate_report
- hosts: hyperfoil_controller
  tasks:
    - name: Retrieve the report
      fetch:
        src: "{{ reports_path }}/{{ test_runid }}.html"
        dest: "{{ playbook_dir }}/{{ toolkit_csv_file_path }}/"
        flat: yes

Next we need to populate the main.yml found in deployments/roles/profiled-traffic-generator/defaults/main.yml
You need to get the 3scale-admin route, 3scale admin access token and the echo-api route

oc get routes -n <3scale-namespace-here> | grep 3scale-admin | awk '{print $2}'
3scale-admin.apps.aucunnin-ccs.lpi0.s1.devshift.org                                          
oc get secret system-seed -n <3scale-namespace-here> -o jsonpath="{.data.ADMIN_ACCESS_TOKEN}"| base64 --decode
<Some_token>
oc get routes -n echo-api | awk '{print $2}'
echo-api-echo-api.apps.aucunnin-ccs.lpi0.s1.devshift.org          

And add them to main.yml as follows

---
# defaults file for profiled-traffic-generator

# URI that includes your password and portal endpoint in the following format: <schema>://<password>@<admin-portal-domain>.
# The <password> can be either the provider key or an access token for the 3scale Account Management API.
# <admin-portal-domain> is the URL used to log into the admin portal.
# Example: https://[e-mail]
threescale_portal_endpoint: https://<Some-token>@3scale-admin.apps.aucunnin-ccs.lpi0.s1.devshift.org/

# Used traffic for performance testing is not real traffic.
# It is synthetically generated traffic based on traffic models.
# Information about available traffic profiles (or test plans) can be found here:
# https://github.com/3scale/perftest-toolkit/blob/master/buddhi/README.md#profiles
# Currently available profiles: [ simple | backend | medium | standard ]
traffic_profile: simple

# Private Base URL
# Make sure your private application behaves like an echo api service
# example: https://echo-api.3scale.net:443
private_base_url: https://echo-api-echo-api.apps.aucunnin-ccs.lpi0.s1.devshift.org

# Public Base URL
# Public address of your API gateway in the production environment.
# Optional. When it is left empty, public base url will be the hosted gateway url
# example: https://gw.example.com:443
public_base_url: 

Note we have selected the simple profile here as it's quickest to setup and is a single product and backend.Other profiles are backend, medium, standard with standard producing the most endpoints and being a slower setup.

So once we have the hosts, run and main set we can run the following playbook which is basically setup and deployment jobs

ansible-playbook -i hosts profiled-injector.yml

Once completed you can run your test with this playbook

ansible-playbook -i hosts -i benchmarks/3scale.csv run.yml

So how do you know it's working?

You will see a polling in the Ansible output

The Ansible output above is showing that this is run/0001 we can hit that endpoint using the ec2 public address and port 8090 and can check the status, below is an example of a complete status

You can ssh to the ec2 instance and access via the Hyperfoil cli in /tmp/hyperfoil/hyperfoile-0.15/bin and run the cli with ./cli.sh

As stated above you should get a run html file each time you run the load test in the deployments/benchmarks directory which can be opened in the browser.

We can then gather things from the report like percentile response times, number of successful/failed request etc.