So far we’ve learned a lot about how to develop backend web services with gRPC. When it comes to deployment, one important thing we should consider is load balancing.
A large scale gRPC deployment typically has a number of identical backend servers and a number of clients. Load balancing is used for distributing the load from clients optimally across available servers.
Here's the link to the full gRPC course playlist on Youtube
Gitlab repository: pcbook-go and pcbook-java
Table of contents
- Types of load balancing
- Code refactoring
- Update Makefile
- Install Nginx
- Config Nginx for insecure gRPC
- Config Nginx for gRPC with TLS
Types of load balancing
There are 2 main options for gRPC load balancing: server-side and client-side. Deciding which one to use is a primary architectural choice.
Server side load balancing
In server-side load balancing, the client issues RPCs to a load balancer or proxy, such as Nginx or Envoy. The load balancer distributes the RPC call to one of the available backend servers.
It also keeps track of load on each server and implements algorithms for distributing load fairly. The clients themselves do not know about the backend servers.
Client side load balancing
In client-side load balancing, the client is aware of multiple backend servers and chooses one to use for each RPC. Usually, the backend servers register themselves with a service discovery infrastructure, such as Consul or Etcd. Then the client communicates with that infrastructure to know the addresses of the servers.
A thick client implements the load balancing algorithms itself. For example, in a simple configuration, where the server load is not considered, the client can just round-robin between available servers.
Another approach is to use a look-aside load balancer, where the load balancing smarts are implemented in a special load-balancing server. Clients query the look-aside load balancer to get the best server(s) to use. The heavy lifting of keeping server state, service discovery, and implementation of a load balancing algorithm is consolidated in the look-aside load balancer.
Pros and cons
One pros of server-side load balancing is simple client implementation. All client needs to know is the address of the proxy, no more coding is needed. This approach works even for untrusted clients, which means the gRPC service can be open for everyone from the public internet.
However, its cons is adding 1 more extra hop to the call. All RPCs have to go through the proxy before reaching the backend server, thus causing higher latency. Therefore, this server-side load balancing is suitable for the cases where there are many clients, possibly untrusted from the open internet who want to connect to our gRPC servers in a data center.
The client-side load balancing, on the other hand, doesn’t add any extra hop to the call and thus giving us higher performance in general. However, the client implementation now becomes complex, especially for thick client approach. Therefore, it should only be used for trusted clients, or we will need to use a look-aside load balancer to stand in the front of the trust boundary network. Client-side load balancing is often used in a very high traffic system and microservices architecture.
In this article, we will learn how to set up server-side load balancing for our gRPC services with Nginx.
Code refactoring
Since I’m gonna show you different Nginx configurations where TLS can be enabled or disabled on the server and client, let’s update our code a bit to take in a new command-line argument.
Update server
On the server, let's add a new boolean flag enableTLS, which will tell us whether we want to enable TLS on our gRPC server or not. Its default value is false.
func main() {
port := flag.Int("port", 0, "the server port")
enableTLS := flag.Bool("tls", false, "enable SSL/TLS")
flag.Parse()
log.Printf("start server on port %d, TLS = %t", *port, *enableTLS)
...
}
Then let's extract the interceptors to a separate serverOptions variable. We check the enableTLS flag. Only in case it’s enabled then we load the TLS credentials, and append that credentials to the server options slice. Finally we just pass the server options to the grpc.NewServer() function call.
func main() {
...
interceptor := service.NewAuthInterceptor(jwtManager, accessibleRoles())
serverOptions := []grpc.ServerOption{
grpc.UnaryInterceptor(interceptor.Unary()),
grpc.StreamInterceptor(interceptor.Stream()),
}
if *enableTLS {
tlsCredentials, err := loadTLSCredentials()
if err != nil {
log.Fatal("cannot load TLS credentials: ", err)
}
serverOptions = append(serverOptions, grpc.Creds(tlsCredentials))
}
grpcServer := grpc.NewServer(serverOptions...)
...
}
And that’s it for the server. Let’s do similar thing for the client!
Update client
First we add the enableTLS flag to the command line argument. Then we define a transportOption variable with the default value grpc.WithInsecure().
func main() {
serverAddress := flag.String("address", "", "the server address")
enableTLS := flag.Bool("tls", false, "enable SSL/TLS")
flag.Parse()
log.Printf("dial server %s, TLS = %t", *serverAddress, *enableTLS)
transportOption := grpc.WithInsecure()
if *enableTLS {
tlsCredentials, err := loadTLSCredentials()
if err != nil {
log.Fatal("cannot load TLS credentials: ", err)
}
transportOption = grpc.WithTransportCredentials(tlsCredentials)
}
cc1, err := grpc.Dial(*serverAddress, transportOption)
if err != nil {
log.Fatal("cannot dial server: ", err)
}
authClient := client.NewAuthClient(cc1, username, password)
interceptor, err := client.NewAuthInterceptor(authClient, authMethods(), refreshDuration)
if err != nil {
log.Fatal("cannot create auth interceptor: ", err)
}
cc2, err := grpc.Dial(
*serverAddress,
transportOption,
grpc.WithUnaryInterceptor(interceptor.Unary()),
grpc.WithStreamInterceptor(interceptor.Stream()),
)
if err != nil {
log.Fatal("cannot dial server: ", err)
}
laptopClient := client.NewLaptopClient(cc2)
testRateLaptop(laptopClient)
}
Only when the enableTLS flag value is true, we load the TLS credentials from PEM files and change the transportOption to grpc.WithTransportCredentials(tlsCredentials). Finally we pass the transportOption to the grpc connections. And the client is done.
Test the new flag
Now if we run make server, we can see that the server is running with TLS disabled.
And if we run make client, it’s also running with no TLS, and all the RPC calls are successful.
If we add -tls flag to the make server command, and restart it, the TLS will be enabled.
...
server:
go run cmd/server/main.go -port 8080 -tls
...
If we run make client now, the requests will fail:
We have to enable TLS on client side as well, by adding -tls flag to the make client command.
...
client:
go run cmd/client/main.go -address 0.0.0.0:8080 -tls
...
And now we can see the requests are successful again.
Update Makefile
Alright, now the TLS flag is working as we wanted. Before adding Nginx, let's update our Makefile a bit so that we can easily run multiple instances of the server and the client with or without TLS.
I’m gonna remove the -tls flags so that the make server and make client commands will run without TLS. And I will add 2 more make commands to run 2 instances of the server on different ports. Let’s say the 1st server will run on port 50051, and the 2nd sever will run on port 50052.
...
server:
go run cmd/server/main.go -port 8080
client:
go run cmd/client/main.go -address 0.0.0.0:8080
server1:
go run cmd/server/main.go -port 50051
server2:
go run cmd/server/main.go -port 50052
...
Let’s also add 3 more make commands to start the client and servers with TLS. The client-tls command will run the client with TLS. The make server1-tls command will start a TLS server on port 50051, and the make server2-tls command will start another TLS server on port 50052.
...
client-tls:
go run cmd/client/main.go -address 0.0.0.0:8080 -tls
server1-tls:
go run cmd/server/main.go -port 50051 -tls
server2-tls:
go run cmd/server/main.go -port 50052 -tls
...
Install Nginx
The next thing we need to do is to install Nginx. Since I’m on a mac, I can simply use Homebrew:
❯ brew install nginx
After nginx is installed, we can go to this usr/local/etc/nginx folder to config it. Let's open the nginx.conf file with with visual studio code.
❯ cd /usr/local/etc/nginx
❯ code nginx.conf
This is the default configuration:
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 8080;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
include servers/*;
}
There are several things that we don’t need to care about in this tutorial, so let's update this config file.
Config Nginx for insecure gRPC
First, let's remove the user config, uncomment the error log, remove the config for log levels and process id, and let’s say we just need 10 worker connections for now.
One important thing we need to do is to config the right location to store the error log and access log files. In my case, Homebrew has already created a log folder for Nginx at /usr/local/var/log/nginx, so I just go ahead and use it in the error/access log setting.
worker_processes 1;
error_log /usr/local/var/log/nginx/error.log;
events {
worker_connections 10;
}
http {
access_log /usr/local/var/log/nginx/access.log;
server {
listen 8080 http2;
location / {
}
}
}
Now in the server block, we have a listen command to listen to incoming requests from client on port 8080. This is the default config for a normal HTTP server. Since gRPC uses HTTP/2, we should add http2 at the end of this command.
Let’s remove the server name and charset since we don’t need them now. Similar for the access log because we’ve already defined it above. Let’s also delete the config for default root HTML file and everything after the location block as we don’t care about them for now.
OK, now we want to load balance the incoming requests to our 2 server instances. So we should define a upstream for them. I’m gonna call it upstream pcbook_services. Inside this block, we use the server keyword to declare a server instance. The first one is running on localhost port 50051, and the second one is running on port 50052.
worker_processes 1;
error_log /usr/local/var/log/nginx/error.log;
events {
worker_connections 10;
}
http {
access_log /usr/local/var/log/nginx/access.log;
upstream pcbook_services {
server 0.0.0.0:50051;
server 0.0.0.0:50052;
}
server {
listen 8080 http2;
location / {
grpc_pass grpc://pcbook_services;
}
}
}
Then to route all RPC calls to the upstream, in the location block, we use grpc_pass keyword, followed by the grpc:// scheme and the name of the upstream, which is pcbook_services.
And that’s it! The load balancing for our insecure gRPC server is done.
Let’s run nginx in the terminal to start it.
❯ nginx
❯ ps aux | grep nginx
quangpham 9013 0.0 0.0 4408572 800 s000 S+ 6:13PM 0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox nginx
quangpham 9007 0.0 0.0 4562704 1124 ?? S 6:12PM 0:00.00 nginx: worker process
quangpham 9006 0.0 0.0 4422416 612 ?? Ss 6:12PM 0:00.00 nginx: master process nginx
We can check if it’s running or not using the ps and grep command. Let’s check out the log folder:
❯ cd /usr/local/var/log/nginx
❯ ls -l
total 0
-rw-r--r-- 1 quangpham admin 0 Oct 11 18:12 access.log
-rw-r--r-- 1 quangpham admin 0 Oct 11 18:12 error.log
As you can see, 2 log files are generated: access.log and error.log. They’re empty at the moment because we haven’t sent any requests yet.
Now let’s run make server1 to start the first server on port 50051 with TLS = false. Then on another tab, run make server2 to start the second server on port 50052, also with TLS disabled. Finally, we run make client on another new tab.
Looks good. All RPC calls are successful. Let’s check the logs on our servers.
The server2 receives 2 create laptop requests.
And the server1 receives 1 login request and 1 create laptop request. Excellent!
And after a while, there’s another login request comming to this server. That’s because our client is still running, and it periodically calls login to refresh the token.
I hope you still remember the codes that we’ve written in the gRPC interceptors lecture.
OK, now let’s look at the nginx access log file.
You can see first there’s a login call, then 3 create laptop calls, and finally another login call. So everything is working exactly as we expect.
Next I’m gonna show you how to enable SSL/TLS for Nginx.
Config Nginx for gRPC with TLS
In a typical deployment, the gRPC servers are already running inside a trusted network, and only the load balancer (Nginx in this case) is exposed to the public internet. So we can leave our gRPC servers running without TLS as before, and only add TLS to Nginx.
Enable TLS on Nginx but keep gRPC servers insecure
To do that, we will need to copy 3 pem files to the nginx config folder:
- The server’s certificate
- The server’s private key
- And the certificate of the CA who signed client’s certificate if we use mutual TLS.
OK, so now I’m gonna cd to the /usr/local/etc/nginx folder and create a new cert folder. Then I will copy those 3 pem files from our pcbook project to this folder.
❯ cd /usr/local/etc/nginx
❯ mkdir cert
❯ cp ~/Projects/techschool/pcbook-go/cert/server-cert.pem cert
❯ cp ~/Projects/techschool/pcbook-go/cert/server-key.pem cert
❯ cp ~/Projects/techschool/pcbook-go/cert/ca-cert.pem cert
Alright, now all the certificate and key files are ready. Let's go back to our nginx config file.
To enable TLS, we first need to add ssl to the listen command. Then we use the ssl_certificate command to give Nginx the location of the server’s certificate file. And use the ssl_certificate_key command to give it the location of the server’s private key file.
...
server {
listen 8080 ssl http2;
ssl_certificate cert/server-cert.pem;
ssl_certificate_key cert/server-key.pem;
ssl_client_certificate cert/ca-cert.pem;
ssl_verify_client on;
location / {
grpc_pass grpc://pcbook_services;
}
}
...
As we’re using mutual TLS, we also need to use the ssl_client_certificate command to tell nginx the location of the client CA’s certificate file. And finally we set ssl_verify_client to on to tell nginx to verify the authenticity of the certificate that client will send.
And we’re done. Let’s restart nginx. We run nginx -s stop to stop it first. Then we start it with nginx command.
❯ nginx -s stop
❯ nginx
Our server is already running, so let’s run the client!
If we just run make client, it will run without TLS, so the request will fail, because Nginx is now running with TLS enabled.
Now let’s run make client-tls.
This time the client is running with TLS, and all requests are successful.
Keep in mind that our servers are still running without TLS. So basically what happens is: only the connection between client and Nginx is secure, and Nginx is connecting to our backend servers via another insecure connection.
Once Nginx receives the encrypted data from client, it will decrypt the data before forwarding it to the backend servers. Therefore, you should only use this approach if the Nginx and backend servers stay in the same trusted network.
OK, but what if they are not on the same trusted network? Well, in that case, we have no choice but to enable TLS on our backend servers as well, and config nginx to work with it.
Enable TLS on both Nginx and gRPC servers
Let’s stop the current server1 and server2, then restart them with TLS.
❯ make server1-tls
❯ make server2-tls
Now if we run make client-tls immediately, the request will fail.
The reason is, although the TLS handshake between the client and Nginx succeeded, the TLS handshake between Nginx and our backend servers failed since the backend servers are now expecting a secure TLS connection, while Nginx is still using an insecure connection when connecting to the backend servers.
As you can see in the error log, the failure happened when Nginx talked to the upstream servers.
To enable secure TLS connection between nginx and upstream, in the nginx.conf file, we have to change the grpc scheme to grpcs.
...
server {
...
location / {
grpc_pass grpcs://pcbook_services;
}
}
...
This should be enough if we just use server-side TLS. However, in this case, we’re using mutual TLS, so if we just restart Nginx now and rerun make client-tls, the request will still fail because Nginx is not configured to send its certificate to the upstream servers yet.
We’ve got the bad certificate error as you can see in the log.
Let’s see what happens if we go to the server code cmd/server/main.go and change the ClientAuth field from tls.RequireAndVerifyClientCert to tls.NoClientCert, which means we will just use server-side TLS.
func loadTLSCredentials() (credentials.TransportCredentials, error) {
...
// Create the credentials and return it
config := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.NoClientCert,
ClientCAs: certPool,
}
return credentials.NewTLS(config), nil
}
Then restart server1-tls and server2-tls, and run make client-tls again.
❯ make server1-tls
❯ make server2-tls
❯ make client-tls
This time all requests are successful. That's exactly what we expected!
OK now what if we really want mutual TLS between nginx and upstream?
Let’s change the ClientAuth field back to tls.RequireAndVerifyClientCert, restart the 2 TLS backend servers, and get back to our nginx.conf file.
This time, we must instruct Nginx to do mutual TLS with the backend servers by giving it the location of the certificate and private key. We use the grpc_ssl_certificate keyword for the certificate, and the grpc_ssl_certificate_key keyword for the private key.
You can generate a different pair of certificate and private key for Nginx if you want. Here I simply use the same certificate and private key of the servers.
worker_processes 1;
error_log /usr/local/var/log/nginx/error.log;
events {
worker_connections 10;
}
http {
access_log /usr/local/var/log/nginx/access.log;
upstream pcbook_services {
server 0.0.0.0:50051;
server 0.0.0.0:50052;
}
server {
listen 8080 ssl http2;
# Mutual TLS between gRPC client and nginx
ssl_certificate cert/server-cert.pem;
ssl_certificate_key cert/server-key.pem;
ssl_client_certificate cert/ca-cert.pem;
ssl_verify_client on;
location / {
grpc_pass grpcs://pcbook_services;
# Mutual TLS between nginx and gRPC server
grpc_ssl_certificate cert/server-cert.pem;
grpc_ssl_certificate_key cert/server-key.pem;
}
}
}
OK, let’s try it.
First stop the current nginx process, then start a new one. And run make client-tls again.
❯ nginx -s stop
❯ nginx
❯ make client-tls
This time all requests are successful. Perfect!
Multiple routing locations
There’s one more thing I want to show you before we finish.
As you’ve already seen, the login and create-laptop requests are now evenly distributed between our 2 backend servers. But sometimes, we might want to separate the authentication service and the business logic service.
For example, let’s say we want all login requests to go to server 1, and all other requests to go to server 2. In that case, we can also tell Nginx to route the requests based on its path.
worker_processes 1;
error_log /usr/local/var/log/nginx/error.log;
events {
worker_connections 10;
}
http {
access_log /usr/local/var/log/nginx/access.log;
upstream auth_services {
server 0.0.0.0:50051;
server 0.0.0.0:50052;
}
upstream laptop_services {
server 0.0.0.0:50051;
server 0.0.0.0:50052;
}
server {
listen 8080 ssl http2;
# Mutual TLS between gRPC client and nginx
ssl_certificate cert/server-cert.pem;
ssl_certificate_key cert/server-key.pem;
ssl_client_certificate cert/ca-cert.pem;
ssl_verify_client on;
location /techschool.pcbook.AuthService {
grpc_pass grpcs://auth_services;
# Mutual TLS between nginx and gRPC server
grpc_ssl_certificate cert/server-cert.pem;
grpc_ssl_certificate_key cert/server-key.pem;
}
location /techschool.pcbook.LaptopService {
grpc_pass grpcs://laptop_services;
# Mutual TLS between nginx and gRPC server
grpc_ssl_certificate cert/server-cert.pem;
grpc_ssl_certificate_key cert/server-key.pem;
}
}
}
Here I just copy the path /techschool.pcbook.AuthService of the AuthService and paste it to this location. Then I change this upstream name to auth_services. It should only connect to the server1 at port 50051.
Then I add another upstream for the laptop_services and make it connect to only server2 at port 50052. Then duplicate the location block, change the upstream name to laptop_services, and update the path to techschool.pcbook.LaptopService.
OK, let’s try this! We just need to restart Nginx and run make client-tls.
Now we can see only login request goes to server1.
And all other create laptop requests go to server2. Even if we run this make client-tls multiple times.
So it works! And that wraps up our lecture about load balancing gRPC with Nginx.
I’m gonna push this nginx config file to the pcbook-go repository so that you can download and play with it if you like.
Thanks a lot for reading and following the course. Happy coding and see you in the next one!
If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.
If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.
Top comments (6)
This is so great, thank you! I am building a grpc-web client for my grpc service. This seems to imply that I can use Nginx as the proxy instead of Envoy, is that true?
Hi John. I haven't tried Nginx with grpc-web yet, so I don't know if it can replace Envoy in that case or not. Some people on the internet said it works, but some said it doesn't. So I will tell you when I have time to check it carefully.
Thanks! I would love to see some material on deploying all of this on GCP or similar. That would be a great way to round out the series!
Awesome! Thanks for sharing.
That was the brilliant and top class explaint about GRPC.
One of the best articles related to gRPC.
(y)
really well done!