Deploying and Consuming Kafka event-driven messages in a Java-enabled Multi-Cloud environment

In an ever-increasingly complex world that we are living in, our expectation also rises exponentially in terms of how we demand information to be instantaneously ready for us as we are searching for them. In the computing world, there is an increasing need to handle the processing of huge volumes of data in the most efficient manner, and this is when the event-driven approach in messaging and sysem design has come to the forefront, when cloud and cloud-native computing have taken he world by storm. So can event messages become public "consumables" like what REST APIs have been since the last decade?

We'll take a look and step through a basic example of a multi- and cloud scenario, in which we'll leverage on the Kafka message broker running in a managed cloud platform - Confluent, and is responsible for producing messages to a topic, from which the Kafka consumer client as enabled by MicroProfile Reactive Messaging's Kafka Connector, and encapsulated in the Open Liberty runtime which gets powered underneath by the very efficient IBM Semeru Runtimes - the OpenJ9 JVM, and the client is launched using OpenShift onto the IBM Cloud. On the producer side, the Confluent Cloud can be launched on one of their support external cloud providers - Microsoft Azure, Amazon AWS, or the Google Cloud Platform.

Let's dive in:

Preliminary Steps (before any deployment work):

1. Message producing side is handled by the Confluent Cloud. The details will not be discussed here, as this article focuses primarily on the message consumer side. Please refer to the documentations on the Confluent website, such as How to build your first Apache KafkaProducer application. Another way, a quicker way, is to leverage on the Confluent Cloud console to create a topic.

2. Logon to the IBM Cloud, or sign up for a free-tier account if you have not done so yet.

3. Accessing your OpenShift cluster (please see one of the following ways accordingly):

   1. (FOR WORKSHOP PARTICIPANTS) If you are given a temporary OpenShift cluster at an IBM workshop: IBM Digital Developer Conference - Hybrid Cloud 2021 Please refer to the instructions that have been provided to you as a Bookmark URL in the #ddc-labs-courses Slack channel.
   2. (FOR CUSTOMERS/TRIAL CUSTOMERS) If you are ready to create a new OpenShift cluster, please refer to the IBM Cloud documentation on how to create your Red Hat OpenShift cluster.

Interact with your OpenShift cluster using one of the following ways:

     * OpenShift console (via the IBM Cloud console)
     * OpenShift CLI (`oc`)
     * IBM Cloud shell (via the IBM Cloud console)

1. Once you are logged on to your OpenShift cluster, you will need to install the Open Liberty operator. This will help us to deploy and manage our Open Liberty microservices application on the cluster. You can do so via the OpenShift console, or on the command-line.

2. Once the Open Liberty operator has been installed, you may want to verify that has succeeded.

    oc api-resources --api-group=openliberty.io
   

3. Add a private Docker credential:

Docker limits container image pull requests for free DockerHub subscriptions. For more information, see Understanding Docker Hub Rate Limiting. If you have a Docker account with a Pro or Team subscription, you can add a private credential to avoid any errors as a result of the limits.

To add a private credential, navigate to the OpenShift web console and select Workloads > Secrets from the sidebar menu. Ensure that the selected project is openshift-config. Search for pull-secret and click the three vertical dots menu; then Edit Secret. Scroll down and click Add credentials. Enter docker.io to the Registry Server Address field, your Docker user ID to the Username field, and your Docker password to the Password field. Click the Save button to save the credential.

Preparing to deploy the inventory microservice to OpenShift:

Use the example microservice (and currently we have named it as inventory) by cloning from the following repository:

git clone https://github.com/mgrygles-lab/guide-ddc.git

To deploy the inventory microservice, you must first package it, then create and run an OpenShift build to produce runnable container image of the packaged microservice.

1. Packaging the microservice
   Ensure that you are in the start directory and run the following commands to package the system and inventory microservices:

     mvn clean install
     mvn clean package
   

2. Building and pushing the images
   Require: a build template to configure how to build your container images. We have provided the build template for you.

Examine the build.yaml template file.

build.yaml
The build.yaml template includes two objects. The ImageStream object provides an abstraction from the image in the image registry. This allows you to reference and tag the image. The image registry used is the integrated internal OpenShift Container Registry.

The BuildConfig object defines a single build definition and any triggers that kickstart the build. The source spec defines the build input. In this case, the build inputs are your binary (local) files, which are streamed to OpenShift for the build. The uploaded files need to include the packaged WAR application binaries, which is why you needed to run the Maven commands. The template specifies a Docker strategy build, which invokes the docker build command, and creates a runnable container image of the microservice from the build input. The template is parameterized with the APP_NAME parameter so that you can use the same template to create the objects for the system and inventory microservices separately.

Run the following commands to create the objects for the system and inventory microservices:

       oc process -f build.yaml -p APP_NAME=inventory | oc create -f -

Next, run the following commands to view the newly created ImageStream object and the build configuration for the microservice:

       oc get all -l name=inventory

Look for the following resources similar to the following:

NAME	TYPE	FROM	LATEST
buildconfig.build.openshift.io/inventory-buildconfig	Docker	Binary	1
---	---	---	---
NAME	IMAGE	REPOSITORY-TAG	UPDATED
imagestream.image.openshift.io/inventory-imagestream	image-registry.openshift-image-registry.svc:5000/guide/inventory-imagestream	1.0-SNAPSHOT	2 days ago

Ensure that you are in the ddc directory and trigger the builds by running the following commands:

  oc start-build inventory-buildconfig --from-dir=inventory/.

The local inventory directory is uploaded to OpenShift to be built into the Docker image. Run the following command to list the builds and track their status:

   oc get builds

Look for the output that is similar to the following example:

NAME	TYPE	FROM	STATUS	STARTED
inventory-buildconfig-1	Docker	Binary@f24cb58	Running	13 seconds ago

You may need to wait some time until the build is complete. To check whether the build is complete, run the following commands to view the build logs until the Push successful message appears:

     oc logs build/inventory-buildconfig-1

Run the following command to view the ImageStream objects:

     oc get imagestreams

Run the following commands to get more details on the newly pushed images within the streams:

     oc describe imagestream/inventory-imagestream

The following example shows part of the inventory-imagestream output:

Name: inventory-imagestream
Namespace: guide
Created: 2 minutes ago
Labels: name=inventory
Annotations:
Image Repository: image-registry.openshift-image-registry.svc:5000/guide/inventory-imagestream
Image Lookup: local=false
Unique Images: 1
Tags: 1

...

Now you’re ready to deploy the images.

Deploying the images

You can configure the specifics of the Open Liberty Operator-controlled deployment with a YAML configuration file.

Examine the deploy.yaml configuration file.

deploy.yaml
The deploy.yaml file is configured to deploy the OpenLibertyApplication resource, named inventory, which is controlled by the Open Liberty Operator.

The applicationImage parameter defines what container image is deployed as part of the OpenLibertyApplication CRD. This parameter follows the /[:tag] format. The parameter can also point to an image hosted on an external registry, such as Docker Hub.

One of the benefits of using the ImageStream object is that the operator redeploys the application when it detects a new image is pushed. The env parameter is used to specify environment variables that are passed to the container at runtime. You need to specify the bootstrap address required to communicate with the Kafka cluster as an environment variable.

Update the [kafka-bootstrap-address] variable to the bootstrap address of the remote Kafka cluster that's hosting the producer, such as:

    pkc-epwny.eastus.azure.confluent.cloud:9092

Run the following command to deploy the inventory microservice with the previously explained configurations:

      oc apply -f deploy.yaml

Next, run the following command to view your newly created OpenLibertyApplications resource:

      oc get OpenLibertyApplications

You can also replace OpenLibertyApplications with the shortname olapps.

Look for output that is similar to the following example:

NAME	IMAGE	EXPOSED	RECONCILED	AGE
inventory	guide/inventory-imagestream:1.0-SNAPSHOT	true	True	10s

A RECONCILED state value of True indicates that the operator was able to successfully process the OpenLibertyApplications instance. Run the following commands to view details of your microservice:

       oc describe olapps/inventory

This example shows part of the olapps/inventory output:

Name: inventory
Namespace: guide
Labels: app.kubernetes.io/part-of=system
name=inventoryy
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"openliberty.io/v1beta1","kind":"OpenLibertyApplication","metadata":{"annotations":{},"labels":{"name":"inventory"},"name":"inv...
API Version: openliberty.io/v1beta1
Kind: OpenLibertyApplication

...

Now, we should be ready to perform the basic test. As messages are being produced from the Kafka producer, examine the log file on the Kafka consumer client side to see that the messages get delivered accordingly.