Game Highlights Processor with AWS, Docker & Terraform.

This project builds on our previous NCAA game highlight processing pipeline (link below), which used a deployment script. We're now implementing a full end-to-end deployment using Terraform. This Infrastructure-as-Code approach allows us to manage AWS resources consistently and scalably, automating video ingestion, processing, and delivery.

https://dev.to/gbenga700/streamlining-ncaa-game-highlights-with-rapidapi-docker-and-aws-mediaconvert-22li

Technical Diagram

RapidAPI is a robust API marketplace offering access to a variety of data sources, including NCAA game highlights. By integrating RapidAPI into our workflow, we can easily retrieve game highlight videos and relevant metadata.

AWS ECR provides a secure, scalable container registry for storing and managing Docker images while AWS ECS enables seamless deployment and orchestration of these containers.
Together, they ensure workflow portability and reproducibility across environments by encapsulating dependencies and application code within Docker images.

AWS MediaConvert is a powerful tool for converting media files into formats optimized for various use cases, such as streaming or archival. In this project, MediaConvert processes the raw video fetched from RapidAPI and converts it into desired output formats

Prerequisites
Before we get started, ensure you have the following:

1. RapidAPI Account: Sign up for a RapidAPI account and subscribe to an API that provides NCAA game highlights.
2. AWS Account: AWS access with the required permission to access the necessary services.
3. Docker Installed: Install Docker on your system to run the containerized workflow.
4. Terraform Installed: Ensure Terraform is installed for infrastructure deployment.
5. Basic CLI Knowledge: Familiarity with using command-line tools for API requests, AWS configurations, and Terraform commands.

Tech Stack

• Python
• AWS ECR, ECS & Elemental MediaConvert
• Docker
• Terraform

Step 1: Clone the project Repository

By cloning the project Repository, we'll have access to the application codes and terraform configurations needed for our deployment process.

Clone Repository: Use the

git clone

command to clone the Terraform code repository to your local machine. Ensure that you have Git installed and configured on your system.

https://github.com/OjoOluwagbenga700/game_highlight_processor_terraform.git

Navigate to Repository: cd game_highlight_processor_terraform

Step 2: Reviewing the content of the code

Now that we've cloned the project code repository, let's take a closer look at the file structure and its contents

app folder: This folder contains the main application configuration files. Each python script has a specific function to perform within the application.

config.py: Imports necessary environment variables and assigns them to Python variables, providing default values where appropriate.

fetch.py: fetches the highlights from the API and store them in an S3 bucket as a JSON file.

process_one_video.py: Connects to the S3 bucket and retrieves the JSON file. Extracts the first video URL from within the JSON file. Downloads the video file from the internet into the memory using the requests library. Saves the video as a new file in the S3 bucket under a different folder (videos/) and logs the status of each step.

mediaconvert_process.py: Uses MediaConvert to process a video file - configures the video codec, resolution and bitrate. Also configured the audio settings and stores the processed video back into S3 bucket.

run_all.py: Runs the scripts in a chronological order and provides buffer time for the tasks to be created.

Dockerfile: performs the following actions: Provides the step by step approach to build the docker image.

ecr.tf: This terraform code will create:
• private repository on Amazon ECR
• build Docker image from the app folder.
• push the image to the repository.

ecs.tf: This terraform code will create:
• an ecs cluster
• a task definition
• an ecs service
• an ecs cloudwatch log group

iam.tf: This terraform code will create the necessary permissions and policies for AWS ECR, ECS , S3 Bucket and AWS Elemental Mediaconvert.

s3.tf: This terraform code will create the s3 bucket for storing our fetched json data and the processed videos.

networking.tf: This terraform code will create:
• custom vpc, internet gateway, subnets
• security group for the ecs task

Step 3: Running Terraform Commands

1. Initialize Terraform: Run

terraform init

to initialize the Terraform working directory and download the necessary plugins.

1. Plan the Deployment: Run

terraform plan

to preview the resources that Terraform will create.

1. Apply the Configuration: Run

terraform apply –auto-approve

to deploy the infrastructure.

Step 4: Verify Application deployment by confirming resources deployed on AWS

S3 Bucket

AWS ECR

AWS ECS

AWS Elemental MediaConvert

AWS Cloudwatch Logs

Conclusion
Automating NCAA game highlights with RapidAPI, AWS ECS, AWS ECR, AWS MediaConvert, and Terraform provides a robust and scalable solution for processing sports footage efficiently. This approach eliminates manual overhead, ensures consistent deployments, and optimizes video delivery for various platforms.

Thanks for Reading!!!