Building an Automated S3 Event Notification System with AWS 🚀

Introduction

Hey there, fellow tech enthusiasts! 👋 Today, I'm excited to share a practical AWS automation project that demonstrates the power of integrating various AWS services. We'll build an automated notification system that alerts you via email whenever files are uploaded to an S3 bucket.

GitHub Repository: AWS S3 Event Notification System

Project Overview 🎯

This project creates an automated workflow that:

• Monitors an S3 bucket for new file uploads 📁
• Triggers a Lambda function automatically when new files are detected ⚡
• Sends email notifications through SNS 📧
• All orchestrated through a single shell script! 💪

Architecture

Our solution integrates three main AWS services:

S3 Bucket → Lambda Function → SNS Topic → Email Notification

Why This Project? 🤔

In real-world scenarios, you often need to know when files are uploaded to your S3 buckets. Use cases include:

• Content moderation systems
• Data processing pipelines
• Audit logging
• Team collaboration
• Backup verification

This solution provides real-time notifications without any manual monitoring needed.

AWS Services Used 🛠️

1. Amazon S3 (Simple Storage Service) 📦

• Primary storage service
• Triggers events on file uploads
• Highly scalable and reliable

2. AWS Lambda ⚡

• Serverless compute service
• Runs code in response to S3 events
• Auto-scales based on demand

3. Amazon SNS (Simple Notification Service) 📬

• Managed messaging service
• Handles email notifications
• Supports multiple subscription types

4. AWS IAM (Identity and Access Management) 🔐

• Manages security permissions
• Controls service access
• Implements least privilege principle

Implementation Challenges and Solutions 💡

Challenge 1: IAM Role Configuration

Issue: Initially faced permission issues between services
Solution: Created comprehensive IAM role with necessary policies:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Action": "sts:AssumeRole",
        "Effect": "Allow",
        "Principal": {
            "Service": [
                "lambda.amazonaws.com",
                "s3.amazonaws.com",
                "sns.amazonaws.com"
            ]
        }
    }]
}

Challenge 2: Lambda Function Handler

Issue: Lambda function wasn't processing S3 events correctly
Solution: Implemented robust Python handler:

def lambda_handler(event, context):
    try:
        sns = boto3.client('sns')
        # Get bucket name and object key from event
        bucket = event['Records'][0]['s3']['bucket']['name']
        key = event['Records'][0]['s3']['object']['key']
        # Process and send notification
        ...
    except Exception as e:
        logger.error(f"Error: {str(e)}")
        raise e

Challenge 3: S3 Event Configuration

Issue: Event notifications weren't triggering consistently
Solution: Added proper bucket notification configuration:

aws s3api put-bucket-notification-configuration \
    --bucket "$bucket_name" \
    --notification-configuration '{
        "LambdaFunctionConfigurations": [{
            "LambdaFunctionArn": "'"$LambdaFunctionArn"'",
            "Events": ["s3:ObjectCreated:*"]
        }]
    }'

Setup Instructions 📝

1. Clone the repository:

git clone https://github.com/Haripriya2408/shellscript-projects.git
cd shellscript-projects/s3-event-trigger

1. Install prerequisites:

sudo apt-get update
sudo apt-get install -y jq zip

1. Make the script executable:

chmod +x trigger.sh

1. Run the setup script:

./trigger.sh

1. Check your email and confirm the SNS subscription

Testing the Setup 🧪

1. Create a test file:

echo "Test content" > test_upload.txt

1. Upload to S3:

aws s3 cp test_upload.txt s3://your-bucket-name/

1. Check your email for the notification!

---

Happy Clouding! 👩‍💻👨‍💻