Expose the metrics in Express.js app
I use Prometheus all the time for metrics and alert monitoring in Kubernetes. I decided to see how to setup monitoring in a Node/Express.js app. A quick search of npmjs and I found these two package prom-client a really detailed Prometheus client and express-prom-bundle which uses prom-client
under the hood, I choose express-prom-bundle
as it was a quick win and was producing metrics with a few lines of code, my repo is here. I installed the following packages in my express app
npm install prom-client express-prom-bundle --save
Then added the Prometheus middleware to all routes
const express = require('express');
const app = express();
const promBundle = require("express-prom-bundle");
// Add the options to the prometheus middleware most option are for http_request_duration_seconds histogram metric
const metricsMiddleware = promBundle({
includeMethod: true,
includePath: true,
includeStatusCode: true,
includeUp: true,
customLabels: {project_name: 'hello_world', project_type: 'test_metrics_labels'},
promClient: {
collectDefaultMetrics: {
}
}
});
// add the prometheus middleware to all routes
app.use(metricsMiddleware)
// default endpoint
app.get("/",(req,res) => res.json({
"GET /": "All Routes",
"GET /hello": "{hello:world}",
"GET /metrics": "Metrics data",
"POST /bye": "POST Request: + post data"
}));
// hello world rest endpoint
app.get("/hello", (req,res) => res.json({hello:"world"}));
app.post("/bye", (req,res) => res.send("POST Request : "+ req));
app.listen(8080, function () {
console.log('Listening at http://localhost:8080');
});
Running the app
npm start
> express-prometheus@1.0.0 start /home/austincunningham/repo/express-prometheus
> node index.js
Listening at http://localhost:8080
# curl the hello world endpoint
curl localhost:8080/hello
{"hello":"world"}%
# curl the metrics endpoint
curl localhost:8080/metrics
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 0.120868
# I cut the metrics output short here as its a lot of text but you get the idea
Setup the Express app on Openshift
I am using crc which is local Kubernetes development environment based on Red Hat Openshift. I create a container for the app based on the following DockerFile
# syntax=docker/dockerfile:1
FROM node:12.18.1
WORKDIR /app
COPY ["package.json", "package-lock.json*", "./"]
RUN npm install
COPY . .
CMD [ "node", "index.js" ]
I then build, test the image locally and push the image
docker build -t quay.io/austincunningham/express-prometheus:v1.0.0 .
docker run -p 8080:8080 quay.io/austincunningham/express-prometheus:v1.0.0
Listening at http://localhost:8080
docker push quay.io/austincunningham/express-prometheus:v1.0.0
I can then deploy this on crc/openshift with the following two files
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-app
spec:
replicas: 3
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
containers:
- name: example-app
image: quay.io/austincunningham/express-prometheus:v1.0.0
ports:
- name: web
containerPort: 8080
service.yaml
kind: Service
apiVersion: v1
metadata:
name: example-app
labels:
app: example-app #--> this is used for scraping the service via the serviceMonitor
spec:
selector:
app: example-app
ports:
- name: web
port: 8080
Apply the files to the default project
oc project default
oc apply -f deployment.yaml
oc apply -f service.yaml
service/example-app created
# create a route to the service so you can access from the browser
oc expose service example-app
route.route.openshift.io/example-app exposed
You can test the route by hitting the /metrics path in the browser you should see
Setup Prometheus Operator on Openshift
I am following the prometheus operator getting started guide. Applied the bundle from the setup on the default namespace
oc project default
oc apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
NOTE: Hit a issue where the prometheus-operator pod was in a crash loop backoff :(
Openshift has an operator hub so I did the following to fix the crashing operator pod. First I deleted the existing prometheus-operator deployment
oc delete deployment prometheus-operator
Logged in to crc/Openshift console as kubeadmin, in the administrator view go to OperatorHub and search for prometheus
Select the Prometheus Operator
tile and continue
then select install
button
Select the default namespace from the drop down and install button again
Phew! that took longer to explain that to do.
Steps to get Prometheus to see the Express.js apps metrics
First we add the Prometheus CR(custom resource) to the default namespace to start the Prometheus instance
prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
team: frontend # --> this is used by prometheus to scrape the serviceMonitor
resources:
requests:
memory: 400Mi
enableAdminAPI: false
And add the service
prometheus-service.yaml
kind: Service
apiVersion: v1
metadata:
name: prometheus-operated
namespace: default
labels:
operated-prometheus: 'true'
spec:
ports:
- name: web
protocol: TCP
port: 9090
targetPort: web
selector:
app: prometheus
Apply the files and create a route
oc apply -f prometheus.yaml
oc apply -f prometheus-service.yaml
oc expose service prometheus-operated
The way Prometheus scrapes metrics is that it uses a service monitor to check a service
for a particular label. We have already created the service when we deployed the example-app with the label app: example-app
in metadata.labels.
Next we create a serviceMonitor in the default namespace and with a selector
for the app: example-app
label. So we create the following file.
service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-app
labels:
team: frontend # --> this should match the serviceMonitorSelector in the prometheus CR
spec:
selector:
matchLabels:
app: example-app # --> this should match the label in the service in example-app
endpoints:
- port: web
NOTE metadata.labels
team: frontend
we will use this later.
We upload the service-monitor.yaml file to the default namespace to create the serviceMonitor
oc apply -f service-monitor.yaml
In the prometheus.yaml CR we have already selected the service monitor this is done via serviceMonitorSelector
label with the label team: frontend
Finally we need some RBAC rules which is Kubernetes version of permissions to allow Prometheus to see everything
Setup a service account, clusterRole and clusterRoleBinding. Create the following files
service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
clusterRole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
clusterRoleBinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: default
Apply the files to the default namespace
oc apply -f service-account.yaml
oc apply -f clusterRole.yaml
oc apply -f clusterRoleBinding.yaml
You should be able to access the route the default namespace
oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
example-app example-app-default.apps-crc.testing example-app web None
prometheus prometheus-default.apps-crc.testing prometheus web None
You can open the Prometheus UI by adding a http:// to the Prometheus HOST/PORT returned from the oc get routes command
So how do you know if its working
It takes a little while for the Prometheus operator to reconcile and to show up the new resources. In the Prometheus ui first check the Status\Service Discovery
you should see example-app show up
NOTE: be patient it can take a while to show up
Then check the Status\Targets
should see the following targets up
You also should be able to see metrics from example-app in the graph tab
That it I may do a follow up on setting up Grafana to use these metrics
Top comments (2)
Nice work on this tutorial! It's a great way to monitor Express stats, especially for people that already use Prometheus and Grafana in a Kubernetes environment, like yourself.
For anyone looking for a simpler and easier way to monitor their Express app, one that doesn't involve deploying & configuring complex monitoring systems, check out Apitally.
Apitally provides insights into API traffic, errors, response times and payload sizes, for the whole API, each endpoint and individual API consumers. It also monitors API uptime & availability, alerting users when their API is down.
To set it up, users just need to install a small dependency in their Express project and add two lines of code. Here's the setup guide for Express.
Thank you so much. I'm starting to discover this new tool.