Lessons Learned from Implementing Mutual TLS (mTLS) in a Secure System

Introduction

In today’s security landscape, ensuring encrypted communication between services is crucial. One effective method to achieve this is through mutual Transport Layer Security (mTLS), which requires both the client and server to authenticate each other using certificates.

Through my experience implementing mTLS in a production environment, I encountered several challenges, learned valuable lessons, and optimized security while maintaining performance. In this article, I’ll share my experience, from setup to troubleshooting, along with best practices for implementing mTLS effectively.

What is mTLS?

Mutual TLS (mTLS) is an extension of TLS where both parties (client and server) authenticate each other’s identities using certificates. Unlike traditional TLS, which only verifies the server, mTLS ensures that only trusted clients can communicate with the server, adding an extra layer of security.

My Implementation Context

For my project, I was integrating mTLS in a Java-based microservices system. The key components involved were:

• Java Keystore (JKS) for storing certificates and keys
• Spring Boot and Tomcat as the application framework
• Dockerized environment running on OpenShift
• A customized WebClient that enforces mTLS for secure outbound requests.

The main goal was to establish secure communication with external services using custimized webclient to be used while calling this service.

What is mTLS flow?

Setting Up mTLS

Here’s how I implemented mTLS:

1. Combine JKS file that combine the client certiifcate and private key using KeyStore explore.
2. Implementing mTLS in WebClient:

@Bean
    public WebClient mTLSWebClient() throws Exception {
        KeyStore keyStore = KeyStore.getInstance("jks");
        try (FileInputStream certInputStream = new FileInputStream(certificate.path)) {
            keyStore.load(certInputStream, certificatePassword.toCharArray());
        }
        // Create a KeyManagerFactory
        KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        keyManagerFactory.init(keyStore, certificatePassword.toCharArray());
    // Load the server certificate into a separate truststore
    KeyStore trustStore = KeyStore.getInstance("jks");
    try (FileInputStream trustInputStream = new 
        FileInputStream(SERVER_TRUSTSTORE_PATH)) {
         trustStore.load(trustInputStream, 
         SERVER_TRUSTSTORE_PASSWORD.toCharArray());
      }

    // Create a TrustManagerFactory and initialize it with the server truststore
       TrustManagerFactory trustManagerFactory = 
 TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
       trustManagerFactory.init(trustStore);
// Create SslContext with the KeyManager and TrustManager
    SslContext sslContext = SslContextBuilder.forClient()
            .keyManager(keyManagerFactory)
            .trustManager(trustManagerFactory)  // use InsecureTrustManagerFactory.INSTANCE  if u need to trust any certificate from server.
            .build();
HttpClient httpClient = HttpClient.create()
                .secure(sslContextSpec -> sslContextSpec.sslContext(sslContext));
        return WebClient.builder()
                .clientConnector(new ReactorClientHttpConnector(httpClient));

}

Difference between trustStore and keyStore in Java

• The primary difference between TrustStore and KeyStore lies in their usage:

• TrustStore is used by TrustManager, which verifies whether a remote connection can be trusted.(where we define server certificate )

• KeyStore is used by KeyManager, which selects authentication credentials to send during the SSL handshake.(where we define clients certificate to be sent to server during handshake)

- How They Work in SSL/TLS

• SSL Server: Uses a private key from the KeyStore for key exchange and sends the corresponding certificate to the client.

• SSL Client (Java): Uses the TrustStore to validate the server’s certificate and establish trust.

• SSL certificates (typically .cer files) are imported into a KeyStore or TrustStore using utilities like keytool.

  Challenges Faced & Solutions

• Certificate Chain Issues: Initially, the client certificates were not being validated correctly.

• Solution: Ensured the full certificate chain (root CA + intermediate CAs) was included in the truststore.

- Interoperability with Other Services: Some third-party services did not support mTLS out of the box.

• Solution: Used an insecure truststore in development to allow any certificate while ensuring strict verification in production.

- WebClient SSL Configuration Issues: Debugging SSL/TLS handshake failures with the custom WebClient was challenging.

• Solution: Enabled debug logging (-Djavax.net.debug=all) to track SSL connection issues and validate certificates.

Lessons Learned

• During setup, share the full client certificate chain, not just the leaf, to ensure proper server validation.
• Always validate certificates properly and ensure proper certificate chaining.
• Use separate keystores and truststores to avoid accidental key exposure.
• Monitor TLS handshake failures using application logs and metrics.
• Debugging WebClient TLS issues requires enabling detailed SSL logs.

Conclusion

mTLS has significantly enhanced security by ensuring encrypted communication and preventing unauthorized access. While the setup posed challenges, proper planning and best practices streamlined the process. With effective certificate management and monitoring, mTLS serves as a strong security foundation for modern applications.