Implemented a bench marker to compare Go's HTTP Router

This article is a translation of bmf-tech.com - GoのHTTP Routerを比較するベンチマーカーを実装した.

This is day 5 of Makuake Advent Calendar 2022!

Overview

Bench markers were implemented to compare the performance of Go's HTTP Router.

bmf-san/go-router-benchmark

The following HTTP Routers are currently being compared.

• bmf-san/goblin
• julienschmidt/httprouter
• go-chi/chi
• gin-gonic/gin
• uptrace/bunrouter
• dimfeld/httptreemux
• beego/mux
• gorilla/mux
• nissy/bon
• naoina/denco
• labstack/echo
• gocraft/web
• vardius/gorouter
• go-ozzo/ozzo-routing
• lkeix/techbook13-sample

In some test cases, the Go standard net/http#ServeMux is also covered.

Motivation

I have created my own HTTP Router called bmf-san/goblin.

bmf-san/goblin is a simple HTTP Router based on the Trie tree with minimal functionality.

By comparing the performance of bmf-san/goblin with other HTTP Routers, bmf-san/goblin goblin), the motivation is to get hints on how to improve bmf-san/goblin.

Another reason is that I have my own bench marker that can replace julienschmidt/go-http-routing-benchmark. Another motivation was to be able to maintain

The julienschmidt/go-http-routing-benchmark is the julienschmidt/httprouter, but maintenance seemed to have stopped in recent years, so I decided to create my own benchmarker and implement it. I decided to implement bench markers.

About the test design of the bench marker

As a premise, I would like to state that bmf-san/go-router-benchmark is not a complete comparison of HTTP Router performance .

The reasons are as follows.

• Each HTTP Router has different functions and specifications, and it is not realistic to prepare a test case that perfectly covers all of them, so only some of the specifications are taken into account in the comparison.
• Because each HTTP Router has its own data structure and algorithms, which may or may not be good or bad, and therefore the defined routing test cases may not fully evaluate the performance of each HTTP Router.

Therefore, the benchmark test will be a test case for a specific function or specification of HTTP Router only, but it can measure a certain difference in performance.

In bmf-san/go-router-benchmark, the performance is measured for the processing part of routing.
Specifically, it tests the ServeHTTP function in http#Handler.

benchmark_test.go#L21

The part that defines routing is not tested.
The process of defining routing is the process of registering data necessary for the routing process.

package main

import (
    "fmt"
    "net/http" )
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprint(w, "Hello World")
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/", handler) // here
    ListenAndServe(":8080", mux)
}

The following two test cases are available for the routing process.

• Static routes
• Routes with path parameters

Each test case is described below.

Static routes

A static route is a route without variable parameters such as /foo/bar.

The following four input patterns are available for testing this route.

• /
• /foo
• /foo/bar/baz/qux/quux
• /foo/bar/baz/qux/quux/corge/grault/garply/waldo/fred

This test case also compares the Go standard net/http#ServeMux.

Routes with path parameters

A route with path parameters is a route with variable parameters such as /foo/:bar.

In this route test, the following three input patterns are provided.

• /foo/:bar.
• /foo/:bar/:baz/:qux/:quux/:corge.
• /foo/:bar/:baz/:qux/:quux/:corge/:grault/:garply/:waldo/:fred/:plugh

Since some HTTP Routers have different syntaxes for variable parameters, we also take into account the corresponding syntaxes for each.

ex.pathparam.go#L15

Benchmark test results

Benchmark test results are available at go-router-benchmark.

The benchmark test environment is as follows

• go version: go1.19
• goos: darwin
• goarch: amd64
• pkg: github.com/go-router-benchmark
• cpu: VirtualApple @ 2.50GHz

The benchmark results are as follows

• time
  • Number of function executions
  • The higher the number of executions, the better the performance.
• ns/op
  • Time required per function execution
  • The smaller the time, the better the performance
• B/op
  • Size of memory allocated per function execution
  • The smaller the size, the better the performance
• allocs/op
  • Number of memory allocations made per function execution
  • The fewer the number of memory allocations made per execution of the allocs/op function, the better the performance

cf. dev.to - Improving the performance of your code starting with Go

Describe the results of each test case.

Static routes

For static routes, one point of comparison seems to be whether the performance is better than or equal to the standard net/http#ServeMux.
HTTP Router, which is touted for its better performance, is still better than the standard.

time

time	static-routes-root	static-routes-1	static-routes-5	static-routes-10
servemux	24301910	22053468	13324357	8851803
goblin	32296879	16738813	5753088	3111172
httprouter	100000000	100000000	100000000	72498970
chi	5396652	5350285	5353856	5415325
gin	34933861	34088810	34136852	33966028
bunrouter	63478486	54812665	53564055	54345159
httptreemux	6669231	6219157	5278312	4300488
beegomux	22320199	15369320	1000000	577272
gorillamux	1807042	2104210	1904696	1869037
bon	72425132	56830177	59573305	58364338
denco	90249313	92561344	89325312	73905086
echo	41742093	36207878	23962478	12379764
gocraftweb	1284613	1262863	1000000	889360
gorouter	21622920	28592134	15582778	9636147
ozzorouting	31406931	34989970	24825552	19431296
techbook13-sample	8176849	6349896	2684418	1384840

Graph - time

nsop

nsop	static-routes-root	static-routes-1	static-routes-5	static-routes-10
servemux	50.44	54.97	89.81	135.2
goblin	36.63	69.9	205.2	382.7
httprouter	10.65	10.74	10.75	16.42
chi	217.2	220.1	216.7	221.5
gin	34.53	34.91	34.69	35.04
bunrouter	18.77	21.78	22.41	22
httptreemux	178.8	190.9	227.2	277.7
beegomux	55.07	74.69	1080	2046
gorillamux	595.7	572.8	626.5	643.3
bon	15.75	20.17	18.87	19.16
denco	14	13.03	13.4	15.87
echo	28.17	32.83	49.82	96.77
gocraftweb	929.4	948.8	1078	1215
gorouter	55.16	37.64	76.6	124.1
ozzorouting	42.62	34.22	48.12	61.6
techbook13-sample	146.1	188.4	443.5	867.8

Graph - nsop

bop

bop	static-routes-root	static-routes-1	static-routes-5	static-routes-10
servemux	0	0	0	0
goblin	0	16	80	160
httprouter	0	0	0	0
chi	304	304	304	304
gin	0	0	0	0
bunrouter	0	0	0	0
httptreemux	328	328	328	328
beegomux	32	32	32	32
gorillamux	720	720	720	720
bon	0	0	0	0
denco	0	0	0	0
echo	0	0	0	0
gocraftweb	288	288	352	432
gorouter	0	0	0	0
ozzorouting	0	0	0	0
techbook13-sample	304	308	432	872

Graph - bop

allocs

allocs	static-routes-root	static-routes-1	static-routes-5	static-routes-10
servemux	0	0	0	0
goblin	0	1	1	1
httprouter	0	0	0	0
chi	2	2	2	2
gin	0	0	0	0
bunrouter	0	0	0	0
httptreemux	3	3	3	3
beegomux	1	1	1	1
gorillamux	7	7	7	7
bon	0	0	0	0
denco	0	0	0	0
echo	0	0	0	0
gocraftweb	6	6	6	6
gorouter	0	0	0	0
ozzorouting	0	0	0	0
techbook13-sample	2	3	11	21

Graph - allocs

Routes with path parameters

For routes with path parameters, we split the performance into two groups: one with a large degradation in performance as the number of parameters increases, and the other with a modest degradation.

time

time	pathparam-routes-1	pathparam-routes-5	pathparam-routes-10
goblin	1802690	492392	252274
httprouter	25775940	10057874	6060843
chi	4337922	2687157	1772881
gin	29479381	15714673	9586220
bunrouter	37098772	8479642	3747968
httptreemux	2610324	1550306	706356
beegomux	3177818	797472	343969
gorillamux	1364386	470180	223627
bon	6639216	4486780	3285571
denco	20093167	8503317	4988640
echo	30667137	12028713	6721176
gocraftweb	921375	734821	466641
gorouter	4678617	3038450	2136946
ozzorouting	27126000	12228037	7923040
techbook13-sample	3019774	917042	522897

Graph - time

nsop

nsop	pathparam-routes-1	pathparam-routes-5	pathparam-routes-10
goblin	652.4	2341	4504
httprouter	45.73	117.4	204.2
chi	276.4	442.8	677.6
gin	40.21	76.39	124.3
bunrouter	32.52	141.1	317.2
httptreemux	399.7	778.5	1518
beegomux	377.2	1446	3398
gorillamux	850.3	2423	5264
bon	186.5	269.6	364.4
denco	60.47	139.4	238.7
echo	39.36	99.6	175.7
gocraftweb	1181	1540	2280
gorouter	256.4	393	557.6
ozzorouting	43.66	99.52	150.4
techbook13-sample	380.7	1154	2150

Graph - nsop

bop

bop	pathparam-routes-1	pathparam-routes-5	pathparam-routes-10
goblin	409	962	1608
httprouter	32	160	320
chi	304	304	304
gin	0	0	0
bunrouter	0	0	0
httptreemux	680	904	1742
beegomux	672	672	1254
gorillamux	1024	1088	1751
bon	304	304	304
denco	32	160	320
echo	0	0	0
gocraftweb	656	944	1862
gorouter	360	488	648
ozzorouting	0	0	0
techbook13-sample	432	968	1792

Graph - bop

allocs

allocs	pathparam-routes-1	pathparam-routes-5	pathparam-routes-10
goblin	6	13	19
httprouter	1	1	1
chi	2	2	2
gin	0	0	0
bunrouter	0	0	0
httptreemux	6	9	11
beegomux	5	5	6
gorillamux	8	8	9
bon	2	2	2
denco	1	1	1
echo	0	0	0
gocraftweb	9	12	14
gorouter	4	4	4
ozzorouting	0	0	0
techbook13-sample	10	33	59

Graph - allocs

Conclusion.

It can be seen that the HTTP Router with better performance shows less performance degradation in each test case.
This seems to be a clear trend indicating that the implementations are optimized.

When examining some of the better performing HTTP Router implementations, we find that they employ a more sophisticated tree structure.
For example, Echo, gin, httprouter, bon, and chi use the Radix tree (Patricia trie), while denco uses a double array.

Regarding bmf-san/goblin, we found that it is a proprietary extension of the tri-tree, which is not very optimized and has poor performance compared to other HTTP Routers. (We will do our best to improve it...)

On the other hand, it seemed possible that the multifunctionality of some of the HTTP Routers that seemed to perform poorly may have reduced performance.

We felt that we could further obtain performance trends for each HTTP Router by adding test cases, which we will try to address if we have time.

Comments

[Dennis Ping]

This benchmark article deserves more recognition. Very insightful!