Mastering Cost Optimisation with Shell Scripting: Automate Log Storage in S3 for Budget-Friendly Infrastructure

Learn how to leverage Shell scripting to streamline Jenkins log management and reduce cloud storage costs with AWS S3

💡 Introduction

Welcome to the world of DevOps! Today, we are diving into shell scripting by creating a script that will help reduce our infrastructure costs by storing Jenkins logs in AWS S3 rather than VMs.

Imagine a company like Google, which has thousands of microservices running, collecting logs, metrics, and traces stored in servers. Popular observability stacks used by companies are ELK or EFK. These tools collect different types of logs, such as:

• Application Logs: High-priority logs used to troubleshoot applications.
• Kubernetes Control-Plane Logs: High-priority logs for troubleshooting cluster issues.
• Infrastructure Logs: Logs from tools like Jenkins or Terraform.

Infrastructure logs are typically not as critical to store on servers. For example, if a Jenkins build fails, notifications are sent via email or Slack, enabling instant troubleshooting. However, retaining these logs for backup and restoration purposes is important.

To address this, we’ve created a shell script that uploads Jenkins build logs to S3, ensuring cost optimisation while maintaining access to logs when needed.

---

💡 Prerequisites

Before starting the project, ensure the following requirements are met:

• An AWS Account.
• Basic understanding of Shell Scripting.
• Basic understanding of AWS S3.
• Jenkins Installed on your system.

---

💡 Setting Up the Environment

Before we create the script, we need:

1. A Sample Jenkins Project:
   • Create a pipeline project in Jenkins named hello-world-project using a Hello World template.

• Run the pipeline 3–4 times to generate 3–4 log files.

1. An S3 Bucket:
   • Create an S3 bucket. For example: bucket-for-jenkins-logs (ensure the name is unique).

1. Configure AWS CLI:
   • Generate an access key from AWS IAM.
   • Use the command aws configure to set up your AWS CLI.

1. Locate the Jenkins Home Directory:
   • Go to Jenkins > Manage Jenkins > System. The Jenkins home directory path is displayed at the top.

---

💡 Writing the Shell Script

Now, let’s create the script that will automate the upload process:

The Script: s3upload.sh

#!/bin/bash

######################
## Author: Pravesh-Sudha
## Description: Shell script to upload Jenkins logs to S3 bucket
## Version: v1
######################

# Variables
JENKINS_HOME="/Users/praveshsudha/.jenkins"  # Replace with your Jenkins home directory
S3_BUCKET="s3://bucket-for-jenkins-logs"  # Replace with your S3 bucket name
DATE=$(date +%Y-%m-%d)  # Today's date

# Check if AWS CLI is installed
if ! command -v aws &> /dev/null; then
    echo "AWS CLI is not installed. Please install it to proceed."
    exit 1
fi

# Iterate through all job directories
for job_dir in "$JENKINS_HOME/jobs/"*/; do
    job_name=$(basename "$job_dir")

    # Iterate through build directories for the job
    for build_dir in "$job_dir/builds/"*/; do
        # Get build number and log file path
        build_number=$(basename "$build_dir")
        log_file="$build_dir/log"

        # Check if log file exists and was created today
        if [ -f "$log_file" ] && [ "$(date -r "$log_file" +%Y-%m-%d)" == "$DATE" ]; then
            # Upload log file to S3 with the build number as the filename
            aws s3 cp "$log_file" "$S3_BUCKET/$job_name-$build_number.log" --only-show-errors

            if [ $? -eq 0 ]; then
                echo "Uploaded: $job_name/$build_number to $S3_BUCKET/$job_name-$build_number.log"
            else
                echo "Failed to upload: $job_name/$build_number"
            fi
        fi
    done
done

Running the Script

1. Grant Execution Permissions:

   chmod 777 s3upload.sh

1. Execute the Script:

   ./s3upload.sh

The script will:

• Iterate through Jenkins job and build directories.
• Identify logs created on the current date.
• Upload logs to your specified S3 bucket with appropriate naming.

---

💡 Conclusion

Congratulations! 🎉 You’ve successfully created a shell script to automate the process of uploading Jenkins logs to S3, reducing costs and ensuring seamless log management.

But there’s more! AWS S3 supports Lifecycle Management, allowing you to define rules to automatically transition older or less important logs to cheaper storage classes like Glacier or Deep Archive. These options provide even greater cost savings for infrequently accessed logs.

This project demonstrates how a simple shell script, combined with cloud services, can solve real-world challenges in a DevOps workflow.

Keep experimenting, keep learning, and most importantly, keep scripting! 😊

🚀 For more informative blog, Follow me on Hashnode, X(Twitter) and LinkedIn.