DEV Community

Cover image for Load balancing gRPC service with Nginx
TECH SCHOOL
TECH SCHOOL

Posted on • Edited on

Load balancing gRPC service with Nginx

#go #tutorial #webdev #devops

So far we’ve learned a lot about how to develop backend web services with gRPC. When it comes to deployment, one important thing we should consider is load balancing.

A large scale gRPC deployment typically has a number of identical backend servers and a number of clients. Load balancing is used for distributing the load from clients optimally across available servers.

Here's the link to the full gRPC course playlist on Youtube
Gitlab repository: pcbook-go and pcbook-java

Table of contents

Types of load balancing

There are 2 main options for gRPC load balancing: server-side and client-side. Deciding which one to use is a primary architectural choice.

Server side load balancing

In server-side load balancing, the client issues RPCs to a load balancer or proxy, such as Nginx or Envoy. The load balancer distributes the RPC call to one of the available backend servers.

It also keeps track of load on each server and implements algorithms for distributing load fairly. The clients themselves do not know about the backend servers.

Alt Text

Client side load balancing

In client-side load balancing, the client is aware of multiple backend servers and chooses one to use for each RPC. Usually, the backend servers register themselves with a service discovery infrastructure, such as Consul or Etcd. Then the client communicates with that infrastructure to know the addresses of the servers.

Alt Text

A thick client implements the load balancing algorithms itself. For example, in a simple configuration, where the server load is not considered, the client can just round-robin between available servers.

Alt Text

Another approach is to use a look-aside load balancer, where the load balancing smarts are implemented in a special load-balancing server. Clients query the look-aside load balancer to get the best server(s) to use. The heavy lifting of keeping server state, service discovery, and implementation of a load balancing algorithm is consolidated in the look-aside load balancer.

Pros and cons

One pros of server-side load balancing is simple client implementation. All client needs to know is the address of the proxy, no more coding is needed. This approach works even for untrusted clients, which means the gRPC service can be open for everyone from the public internet.

However, its cons is adding 1 more extra hop to the call. All RPCs have to go through the proxy before reaching the backend server, thus causing higher latency. Therefore, this server-side load balancing is suitable for the cases where there are many clients, possibly untrusted from the open internet who want to connect to our gRPC servers in a data center.

Alt Text

The client-side load balancing, on the other hand, doesn’t add any extra hop to the call and thus giving us higher performance in general. However, the client implementation now becomes complex, especially for thick client approach. Therefore, it should only be used for trusted clients, or we will need to use a look-aside load balancer to stand in the front of the trust boundary network. Client-side load balancing is often used in a very high traffic system and microservices architecture.

In this article, we will learn how to set up server-side load balancing for our gRPC services with Nginx.

Code refactoring

Since I’m gonna show you different Nginx configurations where TLS can be enabled or disabled on the server and client, let’s update our code a bit to take in a new command-line argument.

Update server

On the server, let's add a new boolean flag enableTLS, which will tell us whether we want to enable TLS on our gRPC server or not. Its default value is false.



func main() {
    port := flag.Int("port", 0, "the server port")
    enableTLS := flag.Bool("tls", false, "enable SSL/TLS")

    flag.Parse()
    log.Printf("start server on port %d, TLS = %t", *port, *enableTLS)

    ...
}
Enter fullscreen mode Exit fullscreen mode

Then let's extract the interceptors to a separate serverOptions variable. We check the enableTLS flag. Only in case it’s enabled then we load the TLS credentials, and append that credentials to the server options slice. Finally we just pass the server options to the grpc.NewServer() function call.



func main() {
    ...

    interceptor := service.NewAuthInterceptor(jwtManager, accessibleRoles())
    serverOptions := []grpc.ServerOption{
        grpc.UnaryInterceptor(interceptor.Unary()),
        grpc.StreamInterceptor(interceptor.Stream()),
    }

    if *enableTLS {
        tlsCredentials, err := loadTLSCredentials()
        if err != nil {
            log.Fatal("cannot load TLS credentials: ", err)
        }

        serverOptions = append(serverOptions, grpc.Creds(tlsCredentials))
    }

    grpcServer := grpc.NewServer(serverOptions...)

    ...
}
Enter fullscreen mode Exit fullscreen mode

And that’s it for the server. Let’s do similar thing for the client!

Update client

First we add the enableTLS flag to the command line argument. Then we define a transportOption variable with the default value grpc.WithInsecure().



func main() {
    serverAddress := flag.String("address", "", "the server address")
    enableTLS := flag.Bool("tls", false, "enable SSL/TLS")

    flag.Parse()
    log.Printf("dial server %s, TLS = %t", *serverAddress, *enableTLS)

    transportOption := grpc.WithInsecure()

    if *enableTLS {
        tlsCredentials, err := loadTLSCredentials()
        if err != nil {
            log.Fatal("cannot load TLS credentials: ", err)
        }

        transportOption = grpc.WithTransportCredentials(tlsCredentials)
    }

    cc1, err := grpc.Dial(*serverAddress, transportOption)
    if err != nil {
        log.Fatal("cannot dial server: ", err)
    }

    authClient := client.NewAuthClient(cc1, username, password)
    interceptor, err := client.NewAuthInterceptor(authClient, authMethods(), refreshDuration)
    if err != nil {
        log.Fatal("cannot create auth interceptor: ", err)
    }

    cc2, err := grpc.Dial(
        *serverAddress,
        transportOption,
        grpc.WithUnaryInterceptor(interceptor.Unary()),
        grpc.WithStreamInterceptor(interceptor.Stream()),
    )
    if err != nil {
        log.Fatal("cannot dial server: ", err)
    }

    laptopClient := client.NewLaptopClient(cc2)
    testRateLaptop(laptopClient)
}
Enter fullscreen mode Exit fullscreen mode

Only when the enableTLS flag value is true, we load the TLS credentials from PEM files and change the transportOption to grpc.WithTransportCredentials(tlsCredentials). Finally we pass the transportOption to the grpc connections. And the client is done.

Test the new flag

Now if we run make server, we can see that the server is running with TLS disabled.

Alt Text

And if we run make client, it’s also running with no TLS, and all the RPC calls are successful.

Alt Text

If we add -tls flag to the make server command, and restart it, the TLS will be enabled.



...

server:
    go run cmd/server/main.go -port 8080 -tls

...
Enter fullscreen mode Exit fullscreen mode

Alt Text

If we run make client now, the requests will fail:

Alt Text

We have to enable TLS on client side as well, by adding -tls flag to the make client command.



...

client:
    go run cmd/client/main.go -address 0.0.0.0:8080 -tls

...
Enter fullscreen mode Exit fullscreen mode

Alt Text

And now we can see the requests are successful again.

Update Makefile

Alright, now the TLS flag is working as we wanted. Before adding Nginx, let's update our Makefile a bit so that we can easily run multiple instances of the server and the client with or without TLS.

I’m gonna remove the -tls flags so that the make server and make client commands will run without TLS. And I will add 2 more make commands to run 2 instances of the server on different ports. Let’s say the 1st server will run on port 50051, and the 2nd sever will run on port 50052.



...

server:
    go run cmd/server/main.go -port 8080

client:
    go run cmd/client/main.go -address 0.0.0.0:8080

server1:
    go run cmd/server/main.go -port 50051

server2:
    go run cmd/server/main.go -port 50052

...
Enter fullscreen mode Exit fullscreen mode

Let’s also add 3 more make commands to start the client and servers with TLS. The client-tls command will run the client with TLS. The make server1-tls command will start a TLS server on port 50051, and the make server2-tls command will start another TLS server on port 50052.



...

client-tls:
    go run cmd/client/main.go -address 0.0.0.0:8080 -tls

server1-tls:
    go run cmd/server/main.go -port 50051 -tls

server2-tls:
    go run cmd/server/main.go -port 50052 -tls

...
Enter fullscreen mode Exit fullscreen mode

Install Nginx

The next thing we need to do is to install Nginx. Since I’m on a mac, I can simply use Homebrew:



❯ brew install nginx
Enter fullscreen mode Exit fullscreen mode

After nginx is installed, we can go to this usr/local/etc/nginx folder to config it. Let's open the nginx.conf file with with visual studio code.



❯ cd /usr/local/etc/nginx
❯ code nginx.conf
Enter fullscreen mode Exit fullscreen mode

This is the default configuration:



#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       8080;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }

    include servers/*;
}
Enter fullscreen mode Exit fullscreen mode

There are several things that we don’t need to care about in this tutorial, so let's update this config file.

Config Nginx for insecure gRPC

First, let's remove the user config, uncomment the error log, remove the config for log levels and process id, and let’s say we just need 10 worker connections for now.

One important thing we need to do is to config the right location to store the error log and access log files. In my case, Homebrew has already created a log folder for Nginx at /usr/local/var/log/nginx, so I just go ahead and use it in the error/access log setting.



worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    server {
        listen       8080 http2;

        location / {
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Now in the server block, we have a listen command to listen to incoming requests from client on port 8080. This is the default config for a normal HTTP server. Since gRPC uses HTTP/2, we should add http2 at the end of this command.

Let’s remove the server name and charset since we don’t need them now. Similar for the access log because we’ve already defined it above. Let’s also delete the config for default root HTML file and everything after the location block as we don’t care about them for now.

OK, now we want to load balance the incoming requests to our 2 server instances. So we should define a upstream for them. I’m gonna call it upstream pcbook_services. Inside this block, we use the server keyword to declare a server instance. The first one is running on localhost port 50051, and the second one is running on port 50052.



worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    upstream pcbook_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    server {
        listen       8080 http2;

        location / {
            grpc_pass grpc://pcbook_services;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Then to route all RPC calls to the upstream, in the location block, we use grpc_pass keyword, followed by the grpc:// scheme and the name of the upstream, which is pcbook_services.

And that’s it! The load balancing for our insecure gRPC server is done.

Let’s run nginx in the terminal to start it.



❯ nginx
❯ ps aux | grep nginx
quangpham         9013   0.0  0.0  4408572    800 s000  S+    6:13PM   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox nginx
quangpham         9007   0.0  0.0  4562704   1124   ??  S     6:12PM   0:00.00 nginx: worker process
quangpham         9006   0.0  0.0  4422416    612   ??  Ss    6:12PM   0:00.00 nginx: master process nginx
Enter fullscreen mode Exit fullscreen mode

We can check if it’s running or not using the ps and grep command. Let’s check out the log folder:



❯ cd /usr/local/var/log/nginx
❯ ls -l
total 0
-rw-r--r--  1 quangpham  admin  0 Oct 11 18:12 access.log
-rw-r--r--  1 quangpham  admin  0 Oct 11 18:12 error.log
Enter fullscreen mode Exit fullscreen mode

As you can see, 2 log files are generated: access.log and error.log. They’re empty at the moment because we haven’t sent any requests yet.

Now let’s run make server1 to start the first server on port 50051 with TLS = false. Then on another tab, run make server2 to start the second server on port 50052, also with TLS disabled. Finally, we run make client on another new tab.

Alt Text

Looks good. All RPC calls are successful. Let’s check the logs on our servers.

Alt Text

The server2 receives 2 create laptop requests.

Alt Text

And the server1 receives 1 login request and 1 create laptop request. Excellent!

And after a while, there’s another login request comming to this server. That’s because our client is still running, and it periodically calls login to refresh the token.

Alt Text

I hope you still remember the codes that we’ve written in the gRPC interceptors lecture.

OK, now let’s look at the nginx access log file.

Alt Text

You can see first there’s a login call, then 3 create laptop calls, and finally another login call. So everything is working exactly as we expect.

Next I’m gonna show you how to enable SSL/TLS for Nginx.

Config Nginx for gRPC with TLS

In a typical deployment, the gRPC servers are already running inside a trusted network, and only the load balancer (Nginx in this case) is exposed to the public internet. So we can leave our gRPC servers running without TLS as before, and only add TLS to Nginx.

Enable TLS on Nginx but keep gRPC servers insecure

To do that, we will need to copy 3 pem files to the nginx config folder:

  • The server’s certificate
  • The server’s private key
  • And the certificate of the CA who signed client’s certificate if we use mutual TLS.

OK, so now I’m gonna cd to the /usr/local/etc/nginx folder and create a new cert folder. Then I will copy those 3 pem files from our pcbook project to this folder.



❯ cd /usr/local/etc/nginx
❯ mkdir cert
❯ cp ~/Projects/techschool/pcbook-go/cert/server-cert.pem cert
❯ cp ~/Projects/techschool/pcbook-go/cert/server-key.pem cert
❯ cp ~/Projects/techschool/pcbook-go/cert/ca-cert.pem cert
Enter fullscreen mode Exit fullscreen mode

Alright, now all the certificate and key files are ready. Let's go back to our nginx config file.

To enable TLS, we first need to add ssl to the listen command. Then we use the ssl_certificate command to give Nginx the location of the server’s certificate file. And use the ssl_certificate_key command to give it the location of the server’s private key file.



...

    server {
        listen       8080 ssl http2;

        ssl_certificate cert/server-cert.pem;
        ssl_certificate_key cert/server-key.pem;

        ssl_client_certificate cert/ca-cert.pem;
        ssl_verify_client on;

        location / {
            grpc_pass grpc://pcbook_services;
        }
    }

...
Enter fullscreen mode Exit fullscreen mode

As we’re using mutual TLS, we also need to use the ssl_client_certificate command to tell nginx the location of the client CA’s certificate file. And finally we set ssl_verify_client to on to tell nginx to verify the authenticity of the certificate that client will send.

And we’re done. Let’s restart nginx. We run nginx -s stop to stop it first. Then we start it with nginx command.



❯ nginx -s stop
❯ nginx
Enter fullscreen mode Exit fullscreen mode

Our server is already running, so let’s run the client!

If we just run make client, it will run without TLS, so the request will fail, because Nginx is now running with TLS enabled.

Alt Text

Now let’s run make client-tls.

Alt Text

This time the client is running with TLS, and all requests are successful.

Keep in mind that our servers are still running without TLS. So basically what happens is: only the connection between client and Nginx is secure, and Nginx is connecting to our backend servers via another insecure connection.

Once Nginx receives the encrypted data from client, it will decrypt the data before forwarding it to the backend servers. Therefore, you should only use this approach if the Nginx and backend servers stay in the same trusted network.

OK, but what if they are not on the same trusted network? Well, in that case, we have no choice but to enable TLS on our backend servers as well, and config nginx to work with it.

Enable TLS on both Nginx and gRPC servers

Let’s stop the current server1 and server2, then restart them with TLS.



❯ make server1-tls
❯ make server2-tls
Enter fullscreen mode Exit fullscreen mode

Now if we run make client-tls immediately, the request will fail.

Alt Text

The reason is, although the TLS handshake between the client and Nginx succeeded, the TLS handshake between Nginx and our backend servers failed since the backend servers are now expecting a secure TLS connection, while Nginx is still using an insecure connection when connecting to the backend servers.

Alt Text

As you can see in the error log, the failure happened when Nginx talked to the upstream servers.

To enable secure TLS connection between nginx and upstream, in the nginx.conf file, we have to change the grpc scheme to grpcs.



...

    server {
        ...

        location / {
            grpc_pass grpcs://pcbook_services;
        }
    }

...
Enter fullscreen mode Exit fullscreen mode

This should be enough if we just use server-side TLS. However, in this case, we’re using mutual TLS, so if we just restart Nginx now and rerun make client-tls, the request will still fail because Nginx is not configured to send its certificate to the upstream servers yet.

Alt Text

We’ve got the bad certificate error as you can see in the log.

Let’s see what happens if we go to the server code cmd/server/main.go and change the ClientAuth field from tls.RequireAndVerifyClientCert to tls.NoClientCert, which means we will just use server-side TLS.



func loadTLSCredentials() (credentials.TransportCredentials, error) {
    ...

    // Create the credentials and return it
    config := &tls.Config{
        Certificates: []tls.Certificate{serverCert},
        ClientAuth:   tls.NoClientCert,
        ClientCAs:    certPool,
    }

    return credentials.NewTLS(config), nil
}
Enter fullscreen mode Exit fullscreen mode

Then restart server1-tls and server2-tls, and run make client-tls again.



❯ make server1-tls
❯ make server2-tls
❯ make client-tls
Enter fullscreen mode Exit fullscreen mode

Alt Text

This time all requests are successful. That's exactly what we expected!

OK now what if we really want mutual TLS between nginx and upstream?

Let’s change the ClientAuth field back to tls.RequireAndVerifyClientCert, restart the 2 TLS backend servers, and get back to our nginx.conf file.

This time, we must instruct Nginx to do mutual TLS with the backend servers by giving it the location of the certificate and private key. We use the grpc_ssl_certificate keyword for the certificate, and the grpc_ssl_certificate_key keyword for the private key.

You can generate a different pair of certificate and private key for Nginx if you want. Here I simply use the same certificate and private key of the servers.



worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    upstream pcbook_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    server {
        listen       8080 ssl http2;

        # Mutual TLS between gRPC client and nginx
        ssl_certificate cert/server-cert.pem;
        ssl_certificate_key cert/server-key.pem;

        ssl_client_certificate cert/ca-cert.pem;
        ssl_verify_client on;

        location / {
            grpc_pass grpcs://pcbook_services;

            # Mutual TLS between nginx and gRPC server
            grpc_ssl_certificate cert/server-cert.pem;
            grpc_ssl_certificate_key cert/server-key.pem;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

OK, let’s try it.

First stop the current nginx process, then start a new one. And run make client-tls again.



❯ nginx -s stop
❯ nginx
❯ make client-tls
Enter fullscreen mode Exit fullscreen mode

Alt Text

This time all requests are successful. Perfect!

Multiple routing locations

There’s one more thing I want to show you before we finish.

As you’ve already seen, the login and create-laptop requests are now evenly distributed between our 2 backend servers. But sometimes, we might want to separate the authentication service and the business logic service.

For example, let’s say we want all login requests to go to server 1, and all other requests to go to server 2. In that case, we can also tell Nginx to route the requests based on its path.



worker_processes  1;

error_log  /usr/local/var/log/nginx/error.log;

events {
    worker_connections  10;
}

http {
    access_log  /usr/local/var/log/nginx/access.log;

    upstream auth_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    upstream laptop_services {
        server 0.0.0.0:50051;
        server 0.0.0.0:50052;
    }

    server {
        listen       8080 ssl http2;

        # Mutual TLS between gRPC client and nginx
        ssl_certificate cert/server-cert.pem;
        ssl_certificate_key cert/server-key.pem;

        ssl_client_certificate cert/ca-cert.pem;
        ssl_verify_client on;

        location /techschool.pcbook.AuthService {
            grpc_pass grpcs://auth_services;

            # Mutual TLS between nginx and gRPC server
            grpc_ssl_certificate cert/server-cert.pem;
            grpc_ssl_certificate_key cert/server-key.pem;
        }

        location /techschool.pcbook.LaptopService {
            grpc_pass grpcs://laptop_services;

            # Mutual TLS between nginx and gRPC server
            grpc_ssl_certificate cert/server-cert.pem;
            grpc_ssl_certificate_key cert/server-key.pem;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Here I just copy the path /techschool.pcbook.AuthService of the AuthService and paste it to this location. Then I change this upstream name to auth_services. It should only connect to the server1 at port 50051.

Then I add another upstream for the laptop_services and make it connect to only server2 at port 50052. Then duplicate the location block, change the upstream name to laptop_services, and update the path to techschool.pcbook.LaptopService.

OK, let’s try this! We just need to restart Nginx and run make client-tls.

Alt Text

Now we can see only login request goes to server1.

Alt Text

And all other create laptop requests go to server2. Even if we run this make client-tls multiple times.

So it works! And that wraps up our lecture about load balancing gRPC with Nginx.

I’m gonna push this nginx config file to the pcbook-go repository so that you can download and play with it if you like.

Thanks a lot for reading and following the course. Happy coding and see you in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

Top comments (6)

Collapse
 
jpcorry profile image
John Corry •

This is so great, thank you! I am building a grpc-web client for my grpc service. This seems to imply that I can use Nginx as the proxy instead of Envoy, is that true?

Collapse
 
techschoolguru profile image
TECH SCHOOL •

Hi John. I haven't tried Nginx with grpc-web yet, so I don't know if it can replace Envoy in that case or not. Some people on the internet said it works, but some said it doesn't. So I will tell you when I have time to check it carefully.

Collapse
 
jpcorry profile image
John Corry •

Thanks! I would love to see some material on deploying all of this on GCP or similar. That would be a great way to round out the series!

Collapse
 
koddr profile image
Vic Shóstak •

Awesome! Thanks for sharing.

Collapse
 
aravin profile image
Aravind A •

That was the brilliant and top class explaint about GRPC.

One of the best articles related to gRPC.

(y)

Collapse
 
isparkes profile image
Ian Sparkes •

really well done!

Read next

imsushant12 profile image

Deployment of a Node.js Application on AWS

Sushant Gaurav -

keploy profile image

How to run cypress run and cypress open at a time

keploy -

nenepo profile image

Optimizing React Re-Renders: Best Practices for Performance

Nenepo -

dotnetfullstackdev profile image

Experimental attribute in C# is a powerful tool : Let's explore

DotNet Full Stack Dev -