Publish-subscribe messaging with open web services

In this blog post we will show an example of pub/sub messaging using open web services available in Eyevinn Open Source Cloud. Publish-subscribe messaging, or pub/sub messaging, is an asynchronous communication model that is central for a scalable and distributed system of modules or services.

Based on the open web service Valkey in Eyevinn Open Source Cloud we can build an application using the pub/sub messaging model.

In this guide

1. Get an API Access Token and setup project
2. Develop a subscriber example application
3. Develop a publisher example application

Prerequisites

• An Eyevinn Open Source Cloud account.

Get an API Access Token and setup project

Navigate to Settings / API in the Eyevinn Open Source Cloud web console.

Copy this token and store in your shell's environment in the environment variable OSC_ACCESS_TOKEN.

% export OSC_ACCESS_TOKEN=<access-token-copied-above>

Setup a NodeJS project.

% mkdir pubsub
% cd pubsub
% npm init

Install the Javascript client SDK.

% npm install --save @osaas/client-core @osaas/client-services

Subscriber application

We will start by developing the subscriber application that will register (subscribe) to topics of interest.

Create a file called subscriber.js. Then add the following code to setup the Valkey instance that will be managing all topics and messages. Valkey is an open source, in-memory data store that is API compatible with a Redis client.

const { Context, getPortsForInstance } = require('@osaas/client-core');
const {
  createValkeyIoValkeyInstance,
  getValkeyIoValkeyInstance
} = require('@osaas/client-services');

async function setup() {
  const ctx = new Context();
  let valkey = await getValkeyIoValkeyInstance(ctx, 'example');
  if (!valkey) {
    valkey = await createValkeyIoValkeyInstance(ctx, {
      name: 'example',
      Password: 'secret'
    });
  }
  const token = await ctx.getServiceAccessToken('valkey-io-valkey');
  const ports = await getPortsForInstance(
    ctx,
    'valkey-io-valkey',
    'example',
    token
  );
  const port = ports.find((p) => p.internalPort === 6379);
  if (port) {
    return `redis://:${valkey.Password}@${port.externalIp}:${port.externalPort}`;
  }
  throw new Error('No redis port found');
}

The setup function will return a Redis URL to the Valkey instance.

Next step is to start waiting for messages and first we need to install a Redis client.

% npm install --save ioredis

We will add the following to the subscriber.js file to subscribe on a topic called "messages".

const Redis = require('ioredis');

async function main() {
  const redisUrl = await setup();
  const client = new Redis(redisUrl);
  try {
    await client.subscribe('messages');
    console.log('Waiting for messages...');
    client.on('message', (channel, message) => {
      console.log(`Received message: ${message} from ${channel}`);
    });
  } catch (err) {
    console.error('Error:', err);
  }
}

main();

Now running this application we will get.

% node subscriber.js
Waiting for messages...

Publisher application

Now open a new terminal and create a file called publisher.js. Copy the setup function we created in the subscriber application and in addition add the following to this file.

async function publishMessage(topic, message) {
  const redisUrl = await setup();
  const redisClient = new Redis(redisUrl);
  try {
    const receivedCount = await redisClient.publish(topic, message);
    console.log(
      `Message "${message}" published to channel "${topic}". Received by ${receivedCount} subscribers.`
    );
  } catch (err) {
    console.error('Error publishing message', err);
  } finally {
    await redisClient.quit();
  }
}

if (!process.argv[2]) {
  console.error('Please a message to publish');
  process.exit(1);
}
publishMessage('messages', process.argv[2]);

Now run the publisher application and publish a Hello world message.

% node publisher.js "Hello world"
Message "Hello world" published to channel "messages". Received by 1 subscribers.

And we will see that the subscriber application will output the message.

Waiting for messages...
Received message: Hello world from messages

Conclusion

You have now successfully created a pub/sub messaging application using the open source Valkey in-memory data store for the messaging. With Valkey as an open web service in Eyevinn Open Source Cloud you could immediately start building your pub/sub messaging application without having to host the Valkey instance yourself. You also avoided to be locked in with a specific cloud vendor as it is based on open source.