DEV Community

Shivang Chauhan
Shivang Chauhan

Posted on • Originally published at devswisdom.com on

How I Built a Video Transcoder using AWS MediaConvert

#aws #serverless #node

In this post, you are going to see how I created a Serverless Video Transcoding Pipeline using AWS MediaConvert using Nodejs.

There will be two parts to this project this is the first part where I will be showing you how I built the backend for this using AWS Serverless.

Let’s get started by creating a blank folder with serverless.yml file which will be the core file to deploy our Serverless stack to AWS.

Creating Serverless.yml file 

service: video-transcoding-pipeline

provider:
  name: aws
  region: ${file(./env.yml):${opt:stage}.REGION}
  runtime: nodejs14.x
  versionFunctions: false
  tracing:
    lambda: true

functions:
  - ${file(./lambdaFunctions.yml)}

resources:
  - ${file(./permissions.yml)}
  - ${file(./db.yml)}
  - ${file(./s3.yml)}
Enter fullscreen mode Exit fullscreen mode

As you can see here that we are importing a bunch of yml files which we will be creating next, we are also setting the region which is imported from the env file of the project.

To know more about serverless.yml file check out “What is a serverless.yml file?” section here.

Creating S3 buckets 

Resources:
  MediaInputBucket:
    Type: AWS::S3::Bucket
    Properties:
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders: ["*"]
            AllowedMethods: [GET, PUT, POST]
            AllowedOrigins: ["*"]

  MediaOutputBucket:
    Type: AWS::S3::Bucket
    Properties:
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders: ["*"]
            AllowedMethods: [GET, PUT, POST]
            AllowedOrigins: ["*"]
      AccessControl: PublicRead
Enter fullscreen mode Exit fullscreen mode

Now we will be creating the s3.yml file which will be responsible for creating the S3 buckets, we are creating two buckets here.

The MediaInputBucket is the input bucket where the video file will be uploaded to get transcoded.

The MediaOutputBucket is the output bucket where the transcoded video will be saved by AWS MediaConvert.

  • CorsRules : This config is used to set the Cors for the buckets so we can interact with the buckets through the client side (these can be changed according to the need).
  • AccessContro l: This is giving the public access to the bucket so transcoded videos can be played publicly.

To check more configurations provided for S3 bucket creation, check out the official documentation.

Creating a DynamoDB Table 

Resources:
  VideoStatus:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      BillingMode: PAY_PER_REQUEST
Enter fullscreen mode Exit fullscreen mode

Here a DynamoDB table is being created, this table will be used to store the AWS MediaConvert Job status (more on this later).

Also, you can see that the table name is also being imported from the env file, so let’s create this file now.

Creating the env.yml file 

prod:
  MEDIA_INPUT_BUCKET: !Ref MediaInputBucket
  MEDIA_OUTPUT_BUCKET: !Ref MediaOutputBucket
  REGION: us-east-2
  VIDEO_STATUS_TABLE: VideoStatusTable
  ACCOUNT_ID: [REPLACE_THIS_WITH_YOUR_ACCOUNT_ID]
  MEDIA_ENDPOINT: [REPLACE_THIS_WITH_YOUR_ENDPOINT]
  MEDIA_CONVERT_ROLE: !GetAtt MediaConvertRole.Arn
Enter fullscreen mode Exit fullscreen mode

Here we are creating a bunch of env variables under the prod stage name.

  • MEDIA_ENDPOINT : This is the endpoint for MediaConvert which you can get from your AWS Console by going under the Account section in the MediaConvert dashboard.
  • MEDIA_CONVERT_ROLE : This is the IAM role for AWS MediaConvert.

Creating permissions.yml file

Now it’s time to create the permissions.yml file, there will be two roles created in this file, one will be used by all the Lambda functions and another one will be used by AWS MediaConvert.

Let’s break down this file as it is a bit long.

Creating policy for interacting with DynamoDB 

Resources:
  LambdaRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "LambdaRole-${opt:stage}"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: "LambdaRolePolicy-${opt:stage}"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - "dynamodb:PutItem"
                  - "dynamodb:UpdateItem"
                  - "mediaconvert:*"
                Effect: Allow
                Resource: "*"
Enter fullscreen mode Exit fullscreen mode

This policy will allow the lambda functions to interact with the DynamoDB table.

Creating policy for interacting with AWS MediaConvert 

Policies:
  - PolicyName: 'MediaConvertLambdaPolicy-${opt:stage}'
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Sid: PassRole
          Effect: Allow
          Action:
            - 'iam:PassRole'
          Resource: !GetAtt MediaConvertRole.Arn
        - Sid: MediaConvertService
          Effect: Allow
          Action:
            - 'mediaconvert:*'
          Resource:
            - '*'
        - Sid: MediaInputBucket
          Effect: Allow
          Action:
            - 's3:*'
          Resource:
            - '*'
Enter fullscreen mode Exit fullscreen mode

This policy will allow the Lambda functions to interact with AWS MediaConvert, to read more about how these permissions work check out this official documentation by AWS.

Creating policy to write CloudWatch Log streams 

Policies:
  - PolicyName: 'CloudWatchLogsPolicy-${opt:stage}'
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Action:
            - 'logs:CreateLogGroup'
            - 'logs:CreateLogStream'
            - 'logs:PutLogEvents'
          Effect: Allow
          Resource:
            - >-
              arn:aws:logs:${file(./env.yml):${opt:stage}.REGION}:${file(./env.yml):${opt:stage}.ACCOUNT_ID}:*
Enter fullscreen mode Exit fullscreen mode

This is straightforward as we are allowing the lambda log to be created in the same region and AWS account where we are deploying the stacks.

Now we will create the second role which will be attached to the MediaConvert.

Creating IAM role for AWS MediaConvert 

MediaConvertRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: "MediaConvertRole-${opt:stage}"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - "mediaconvert.amazonaws.com"
                - "mediaconvert.us-east-2.amazonaws.com"
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: "MediaConvertPolicy"
          PolicyDocument:
            Statement:
              - Effect: "Allow"
                Action:
                  - "s3:*"
                Resource:
                  - "*"
              - Effect: "Allow"
                Action:
                  - "cloudwatch:*"
                  - "logs:*"
                Resource:
                  - "*"
Enter fullscreen mode Exit fullscreen mode

This role will allow AWS MediaConvert to interact with S3 and also be able to write AWS CloudWatch logs to the AWS account.

That was a lot to take in but now you are done with creating core yml files, now there is only one yml file left that will create all the lambda functions which are needed, so let’s start with that.

Creating lambdaFunctions.yml file 

startJob:
  handler: resolvers/job/startJob.handler
  name: ${opt:stage}-startJob
  timeout: 600
  role: LambdaRole
  description: Lambda function to start the media convert job
  environment:
    VIDEO_STATUS_TABLE: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
    MEDIA_INPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
    MEDIA_OUTPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_OUTPUT_BUCKET}
    MEDIA_ENDPOINT: ${file(./env.yml):${opt:stage}.MEDIA_ENDPOINT}
    REGION: ${file(./env.yml):${opt:stage}.REGION}
    MEDIA_CONVERT_ROLE: ${file(./env.yml):${opt:stage}.MEDIA_CONVERT_ROLE}
  events:
      - s3:
          bucket: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
          event: s3:ObjectCreated:*
          existing: true

getSignedUrl:
  handler: resolvers/getSignedUrl.handler
  name: ${opt:stage}-getSignedUrl
  timeout: 600
  role: LambdaRole
  description: Lambda function to get the signed url to upload the video
  environment:
    MEDIA_INPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
  events:
      - http:
          path: getSignedUrl
          method: post
          cors: true

updateJobStatus:
  handler: resolvers/job/updateJobStatus.handler
  name: ${opt:stage}-updateJobStatus
  timeout: 600
  role: LambdaRole
  description: Lambda function to update the media convert job status in the DB
  environment:
    VIDEO_STATUS_TABLE: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
    REGION: ${file(./env.yml):${opt:stage}.REGION}
  events:
      - cloudwatchEvent:
          event:
            source:
              - 'aws.mediaconvert'
            detail-type:
              - 'MediaConvert Job State Change'
Enter fullscreen mode Exit fullscreen mode

There are three lambda functions that are being created here.

  • startJob : This lambda function will be responsible to start the AWS MediaConvert job and it will be called whenever any file will be uploaded to the input S3 bucket which you created earlier.
  • getSignedUrl : This lambda function will return the signed URL to upload the video file to the input bucket from the client side.
  • updateJobStatus : This lambda function will be updating the MediaConvert job status to the DynanmoDB table and it will be called whenever the job status gets changed in MediaConvert.

Now you are done with creating all the required yml files, let’s move on to creating the resolvers for the lambda functions.

getSignedUrl Lambda Resolver

This lambda function will be called first to get the signed URL back and then that signed URL will be used to upload the video file to S3 from the client side so we are uploading the video from the backend.

Adding validations 

const {
    fileName,
    metaData
} = JSON.parse(event.body)

if (!fileName || !fileName.trim()) {
    return sendResponse(400, {
        message: 'Bad Request'
    })
}
Enter fullscreen mode Exit fullscreen mode

Here you are getting the file name and the metadata from the client side and you are also checking that the file name must exist otherwise 400 Status code is being returned.

The sendResponse is a utility function that is just sending the response to the API request, you can find it in the source code.

Creating the signed URL 

const params = {
    Bucket: process.env.MEDIA_INPUT_BUCKET,
    Key: fileName,
    Expires: 3600,
    ContentType: 'video/*',
    Metadata: {
        ...metaData
    }
}

const response = s3.getSignedUrl('putObject', params)
Enter fullscreen mode Exit fullscreen mode

Here parameters are being created and getSignedUrl API call is made to get the signed URL, ContentType is set to video/* because only videos will be uploaded to the S3 bucket from the client side.

Now when the file will get uploaded to the S3 bucket by the client application using this signed URL, startJob lambda function will be triggered which will start the AWS MediaConvert job, let’s see what this lambda function looks like.

startJob Lambda Resolver

The first thing that I wanna show you is what are the imports that are added in this lambda resolver.

Imports 

const {
    sendResponse
} = require('../../utilities/index')
const AWS = require('aws-sdk')
AWS.config.mediaconvert = {
    endpoint: `https://${process.env.MEDIA_ENDPOINT}.mediaconvert.${process.env.REGION}.amazonaws.com`
}
const MediaConvert = new AWS.MediaConvert({
    apiVersion: '2017-08-29'
})
const s3 = new AWS.S3()
const params = require('./mediaParams.js')
const dbClient = new AWS.DynamoDB.DocumentClient()
Enter fullscreen mode Exit fullscreen mode

Notice here that I am updating the endpoint for the MediaConvert config, also there is a file named mediaParams.js which is being imported here.

This file will hold the configuration for starting the MediaConvert job, so we will now create this file first.

Creating mediaParams.js config file 

module.exports = {
  Settings: {
    TimecodeConfig: {
      Source: 'ZEROBASED'
    },
    OutputGroups: [
      {
        Name: 'Apple HLS',
        Outputs: [
          {
            ContainerSettings: {
              Container: 'M3U8',
              M3u8Settings: {}
            },
            VideoDescription: {
              Width: '',
              Height: '',
              CodecSettings: {
                Codec: 'H_264',
                H264Settings: {
                  MaxBitrate: '',
                  RateControlMode: 'QVBR',
                  SceneChangeDetect: 'TRANSITION_DETECTION'
                }
              }
            },
            AudioDescriptions: [
              {
                CodecSettings: {
                  Codec: 'AAC',
                  AacSettings: {
                    Bitrate: 96000,
                    CodingMode: 'CODING_MODE_2_0',
                    SampleRate: 48000
                  }
                }
              }
            ],
            OutputSettings: {
              HlsSettings: {}
            },
            NameModifier: 'hgh'
          }
        ],
        OutputGroupSettings: {
          Type: 'HLS_GROUP_SETTINGS',
          HlsGroupSettings: {
            SegmentLength: 10,
            MinSegmentLength: 0,
            DestinationSettings: {
              S3Settings: {
                AccessControl: {
                  CannedAcl: 'PUBLIC_READ'
                }
              }
            }
          }
        }
      },
      {
        CustomName: 'Thumbnail Creation Group',
        Name: 'File Group',
        Outputs: [
          {
            ContainerSettings: {
              Container: 'RAW'
            },
            VideoDescription: {
              Width: 1280,
              Height: 720,
              CodecSettings: {
                Codec: 'FRAME_CAPTURE',
                FrameCaptureSettings: {
                  FramerateNumerator: 1,
                  FramerateDenominator: 5,
                  MaxCaptures: 5,
                  Quality: 80
                }
              }
            }
          }
        ],
        OutputGroupSettings: {
          Type: 'FILE_GROUP_SETTINGS',
          FileGroupSettings: {
            DestinationSettings: {
              S3Settings: {
                AccessControl: {
                  CannedAcl: 'PUBLIC_READ'
                }
              }
            }
          }
        }
      }
    ],
    Inputs: [
      {
        AudioSelectors: {
          'Audio Selector 1': {
            DefaultSelection: 'DEFAULT'
          }
        },
        VideoSelector: {},
        TimecodeSource: 'ZEROBASED'
      }
    ]
  },
  AccelerationSettings: {
    Mode: 'DISABLED'
  },
  StatusUpdateInterval: 'SECONDS_60',
  Priority: 0
}
Enter fullscreen mode Exit fullscreen mode

As you can see that there are a lot of parameters added here but most of these values are static in this project, you will only be modifying transcoded video width/height and bitrate (many many more configurations can be made dynamic according to the requirement).

Fetching the metadata from the uploaded file 

const fileKey = event.Records[0].s3.object.key
const {
    metaData
} = await fetchMetaData(fileKey)
Enter fullscreen mode Exit fullscreen mode

Here you are getting the uploaded file key (which will be received in the lambda trigger attached to the S3 bucket) and calling fetchFromS3 function.

Creating fetchFromS3 function 

async function fetchMetaData (key) {
  try {
    const params = {
      Bucket: MEDIA_INPUT_BUCKET,
      Key: key
    }

    const response = await s3.headObject(params).promise()
    return { metaData: response.Metadata }
  } catch (err) {
    throw new Error(err)
  }
}
Enter fullscreen mode Exit fullscreen mode

Creating the params for starting the MediaConvert job 

const input = `s3://${MEDIA_INPUT_BUCKET}/${fileKey}`
const output = `s3://${MEDIA_OUTPUT_BUCKET}/`

params.Role = MEDIA_CONVERT_ROLE
params.Settings.OutputGroups[0].OutputGroupSettings.HlsGroupSettings.Destination = output
params.Settings.OutputGroups[1].OutputGroupSettings.FileGroupSettings.Destination = output
params.Settings.Inputs[0].FileInput = input
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.Width = metaData.videowidth || 1920
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.Height = metaData.videoheight || 1080
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.CodecSettings.H264Settings.MaxBitrate = metaData.videobitrate || 6000000

const response= await MediaConvert.createJob(params).promise()
Enter fullscreen mode Exit fullscreen mode

We are setting the IAM role for MediaConvert and other settings with metadata as we discussed previously.

Creating initial entry for the created job in the DB 

const vodObj = {
    TableName: VIDEO_STATUS_TABLE,
    Item: {
        id: response.Job.Id,
        createdAt: new Date().toISOString(),
        vodStatus: 'SUBMITTED'
    },
    ConditionExpression: 'attribute_not_exists(id)'
}

await dbClient.put(vodObj).promise()
Enter fullscreen mode Exit fullscreen mode

We are taking the created job id and making it a Sort key in the DynamoDB table and we are also setting the initial job status to SUBMITTED.

Now it is time to work on the last lambda function resolver.

updateJobStatus Lambda Resolver 

try {
    const { VIDEO_STATUS_TABLE, REGION } = process.env
    const { jobId, status, outputGroupDetails } = event.detail

    const params = {
      TableName: VIDEO_STATUS_TABLE,
      Key: {
        id: jobId
      },
      ExpressionAttributeValues: {
        ':vodStatus': status
      },
      UpdateExpression: 'SET vodStatus = :vodStatus',
      ReturnValues: 'ALL_NEW'
    }

    if (status !== 'INPUT_INFORMATION') {
      if (status === 'COMPLETE') {
        const splitOutput = outputGroupDetails[0].outputDetails[0].outputFilePaths[0].split('/')
        params.ExpressionAttributeValues[':outputPath'] = `https://${splitOutput[2]}.s3.${REGION}.amazonaws.com/${splitOutput[3]}`
        params.UpdateExpression += ', outputPath = :outputPath'
      }

      await dbClient.update(params).promise()
    }
  } catch (err) {
    return sendResponse(500, { message: 'Internal Server Error' })
  }
Enter fullscreen mode Exit fullscreen mode

This will be the final lambda function resolver you will need, this lambda will be called whenever the status of the MediaConvert job changes and it will update the new status to the DynamoDB table using the job id we stored earlier.

There are three main stages of a job progression –

  • SUBMITTED : This is the initial job status when it gets started and this is being stored by startJob lambda function.
  • PROGRESSING : This is the status when the job is going on and it will be set through this lambda function.
  • COMPLETE : This is the final status when the job gets successfully completed.

If you want to read more about the different stages of a job, you can check here.

AND we are done, pat yourself on the back if you have reached this far, there are a lot of improvements that can be done in this project.

Improvements

  • MediaConvert endpoint can be fetched using describeEndpoints API, read more here.
  • More configuration can be added to the AWS MediaConvert startJob parameters.
  • Multi-part upload can be implemented to upload larger video files.
  • Job status can be pushed to SNS topic to use in other places.
  • AWS CloudFront distribution can be used to distribute the transcoded video.

Conclusion

Today you saw how we can create a video transcoder using AWS MediaConvert with Serverless and Nodejs, you can play around with it and have fun adding new things, there will be a part 2 to this series where i will be showing how to make the Frontend for this

Find the full source code here.

Check more posts:

What is AWS Athena?

AWS Cognito Pricing

DynamoDB VS MongoDB: Detailed Comparison

AWS DynamoDB Pricing Model and Features Explained

The post How I Built a Video Transcoder using AWS MediaConvert appeared first on DevsWisdom.

Top comments (0)

Read next

alexandrammr profile image

Countdown to re:Invent is on: 5 days to go

Alex Radu -

shuvohoque profile image

Unlock CloudFront's New Logging Potential with Athena Partition Projection

MD. Zeaul Hoque Shuvo -

realacjoshua profile image

Setting Up Local Solana Environment

RealACJoshua -

franckpachot profile image

Aurora Limitless - SQL Limitations

Franck Pachot -