Decoding USP protobuf data sent by USP agents, using the recent AWS IoT Rules Engine native SQL decode function

Contents

1. Introduction
2. Context
3. Set-Up Steps
4. Exploring the payloads and testing with the AWS IoT Console MQTT Client
5. Conclusion
6. Author

Introduction

This post is a follow-up to:

Connect the Open Broadband USP Agent to AWS IoT Core using MQTT5, and Validate the Integration with AWS IoT Core Device Advisor

Alina Dima for IoT Builders ・ Dec 9 '22

#awsiot #tr369 #telco #mqtt5

, which showed how to connect a device running obuspa to AWS IoT Core, over MQTT 5, ingest data and validate the MQTT 5 connection using AWS IoT Core Device Advisor. It is recommended you read the preceding post, in order to get a better understanding of integrating a USP agent with AWS IoT.

In this post, you will see how to set up and use the native AWS IoT Rules Engine protobuf decoding SQL function, in order to decode USP records and messages sent by USP agents, on the fly, without the need for additional AWS Lambda functions with bespoke decoding logic.

Context

Protocol Buffers (protobuf) is an open-source data format used to serialize structured data in binary form. It is used for transmitting data over networks or storing it in files. Protobuf allows you to send data in small packet sizes and at a faster rate than other messaging formats. USP specifies Protocol Buffers Version 3 as a mechanism to serialize data to be sent over a Message Transfer Protocol, such as MQTT.

The protobuf decoding support in the AWS IoT Rules Engine was recently introduced (https://aws.amazon.com/about-aws/whats-new/2022/12/aws-iot-core-rules-engine-google-protocol-buffer-messaging-format/) to enable developers to decode protobuf payloads to JSON format directly in the rule, and route them to downstream services, by means of an AWS IoT Rules SQL function: decode(value, decodingScheme) (https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html#iot-sql-decode-base64).

This feature is interesting because it removes the need to write and maintain custom protobuf decoding logic, which was, prior to this feature, usually done using AWS Lambda functions, invoked either as Rule Actions, or as part of IoT Rule SQL expressions (see the diagram below - before and after the native protobuf support in AWS IoT Rules Engine).

!(https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ssc6t3oihgvwqpp24ggp.png)

Set-Up Steps

Prepare the required protobuf files

As part of the AWS IoT Rule decoding pre-requisites, the protobuf descriptor file needs to be created and uploaded to an S3 bucket.

1. Install the Protobuf Compiler (protoc) on your system. On MacOS, you could use Homebrew, as described here.
2. Download the 2 proto files record and message from the open source USP GitHub repository, corresponding to the USP version that you want to use in your project. This post will work with the latest version, USP 1.2.
3. Create the descriptor file required by the AWS IoT Rules Engine, as described in the docs

You can use the following command:

protoc —descriptor_set_out=usp-1-2.desc —proto_path=<PATH TO YOUR PROTO FILES> —include_imports usp-record-1-2.proto usp-msg-1-2.proto

Create the AWS Cloud resources

You will need to create the AWS IoT Rule, correct Roles and Policies, and S3 Bucket where the descriptor file will be uploded. You can use the AWS CloudFormation template below for this set-up. To create the AWS resources using this template, you can use the AWS CloudFormation CLI commands or the AWS SAM CLI commands.

For the purpose of this example, the Rule will decode the protobuf record, then decode the protobuf message (both using the decode () SQL function), and then republish the resulting JSON object, on a new topic, so we can examine correctness.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  USPProtobufDecodingRule:
    Type: AWS::IoT::TopicRule
    Properties:
      RuleName: USPProtobufDecodingRule
      TopicRulePayload:
        AwsIotSqlVersion: 2016-03-23
        RuleDisabled: False
        Sql: SELECT VALUE decode(decode(encode(*, 'base64'), 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-record-1-2.proto', 'Record').noSessionContext.payload, 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-msg-1-2.proto', 'Msg') FROM '/usp/controller'
        Actions:
          - Republish:
              RoleArn: !GetAtt RepublishDecodedRole.Arn
              Topic: '/usp/controller/decoded'

  RepublishDecodedRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
              - iot.amazonaws.com
      Policies:
        - PolicyName: allowRepublish
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: iot:Publish
                Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic//usp/controller/decoded" ] ]

  USPBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: usp-1-2-bucket

  USPBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref USPBucket
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action:
            - 's3:Get*'
            Effect: Allow
            Resource: !Join
              - ''
              - - 'arn:aws:s3:::'
                - !Ref USPBucket
                - /*
            Principal:
              Service:
                - iot.amazonaws.com

The AWS IoT Rule SQL expression for decoding could look like below:

SELECT VALUE decode(decode(encode(*, 'base64'), 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-record-1-2.proto', 'Record').noSessionContext.payload, 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-msg-1-2.proto', 'Msg') FROM '/usp/controller'

The SQL rule behaves the following way:

• First, the binary MQTT payload is transformed in a based64-encoded string,
• Then, the base64 encoded string is protobuf-decoded at USP record level, as specified by the descriptor,
• Finally, the USP message inside the record, located in the payload object, is protobuf decoded.

Because USP messages are encoded and wrapped inside encoded USP records, you will need to call the decode function twice.

Upload the descriptor file and start obuspa

1. Upload the descriptor file created in step 3 into your S3 bucket. You can do this using the AWS CLI or AWS Console.
2. You can now set up and start the obuspa, as described in the following sections: Integrate obuspa with AWS IoT Core over MQTT5 and Verify MQTT 5 Integration/Protobuf Data Publishing.

Once the agent is sending data, the set-up is complete.

Exploring the payloads and testing with the AWS IoT Console MQTT Client

As the protobuf records arrive on the topic, the IoT Rule will pick them up, decode them, and republish the JSON objects on the /usp/controller/decoded topic. If you subscribe with the test MQTT Client in the AWS IoT Console, you will be able to see the protobuf records on the /usp/controller topic, and the JSON payloads on the /usp/controller/decoded topic, as shown in the images below. The first image shows the original protobuf encoded USP record, arriving on the topic, and the second image the decoded USP message, in JSON format.

Conclusion

This post shows how to to decode protobuf records and messages sent by a USP agent, directly in the AWS IoT Rules Engine, using the decode function, as part of the rule expression. This is an AWS IoT Rules Engine feature released recently, making it easier to use protobuf directly in your IoT Rule, without the need to write and maintain custom decoding code in AWS Lambda functions.

Author

Alina DimaFollow

An engineer for about 20 years, love solving real world problems with code. I simplify complex problems to help developer communities build better and faster with AWS IoT.