Building a Weather Data Collection System with AWS S3 and OpenWeather API

Project Overview

This project is a Weather Data Collection System that demonstrates core DevOps principles by combining:

• External API Integration (OpenWeather API)
• Cloud Storage (AWS S3)
• Infrastructure as Code
• Version Control (Git)
• Python Development
• Error Handling
• Environment Management

Features

• Fetches real-time weather data for multiple cities
• Displays temperature (°F), humidity, and weather conditions
• Automatically stores weather data in AWS S3
• Supports multiple cities tracking
• Timestamps all data for historical tracking

Technical Architecture

• Language: Python 3.x
• Cloud Provider: AWS (S3)
• External API: OpenWeather API
• Dependencies:
  • boto3 (AWS SDK)
  • python-dotenv
  • requests

Architectural Review

Project Structure

weather-dashboard/
  src/
    __init__.py
    weather_dashboard.py
  tests/
  data/
  .env
  .gitignore
  requirements.txt

Prerequisites

Ensure the following tools and credentials are set up on your local machine before proceeding:

• VS Code Editor you can see documentation here

  • AWS CLI you can see documentation here
  • AWS Account: Access credentials with sufficient permissions to interact with S3.
  • OpenWeather API Key: Generate an API key by signing up at OpenWeather API.

Setup Instructions

1. Clone the repository:

  git clone [e-mail]:adesegunasunmo/30-Days-DevOps-Challenge.git

1. Create a Virtual Environment

Before proceeding with the installation, it’s highly recommended to create a virtual environment for Python. This ensures that your package dependencies are isolated from the global environment. The installation might fail if the package is managed externally (e.g., through apt). By isolating the environment, you can safely install and manage the required dependencies without interference.

Then you have to activate the virtual environment

source envname/bin/activate

To do this:

Create a virtual environment

    python3 -m venv (your virtual environment name)

1. Install dependencies

   pip install -r requirements.txt

1. Configure environment variables (.env):

CopyOPENWEATHER_API_KEY=your_api_key
AWS_BUCKET_NAME=your_bucket_name

4.Configure AWS credentials:

aws configure

5.Run the Application Execute the Python script to fetch and upload weather data:

   python src/weather_dashboard.py

What We Learned

Working on this project helped me gain hands-on experience with:

• AWS S3: Creating and managing buckets.
• Environment Variable Management: Securing sensitive API keys.
• Python Best Practices: Writing clean, modular, and reusable code for API integration.
• Git Workflow: Using version control effectively for project development.
• Error Handling: Managing failures in distributed systems gracefully.
• Cloud Resource Management: Leveraging cloud services efficiently.

Future Enhancements

• Add weather forecasting functionality.
• Implement data visualization for weather trends.
• Expand the city tracking feature.
• Create automated testing for better reliability.
• Set up a CI/CD pipeline for streamlined deployments.

---

Congratulations!

We have successfully completed "Building a Weather Data Collection System with AWS S3 and OpenWeather API!" . Great job!

I know it took a lot of clicking and configurations to set everything up. While tool like Docker can automate this process, going through it manually is a great way to understand how all the components work together.

Important Reminder:
Don't forget to delete all the resources you created once you're done. Services like S3 e.t.c, can incur small costs, so make sure to clean up!
Thank you for reading, and I hope you found this guide helpful. Keep learning and building!