A weather data collection system using AWS S3 and OpenWeather API.

Introduction

Me:

Hi, there! I'm new here (this is my first post -- excitement). Instead of over analyzing every single word -- and ultimately feeling like I'll never find the perfect combination of words-- I am going to treat this as what it is, a blog. I'm here to document my learning, but the educator in me hopes this aids in your learning process in some way too.

I have experience in Full Stack development and have gotten my toes wet with some DevOps tools during my internship at Microsoft. To expound upon that and get even more familiar with cloud computing, I recently joined a 31-day DevOps coding challenge. Our Week 1 Day 1 challenge is to create a weather data collection system using AWS S3 and OpenWeather API. This blog will cover the steps I took to create this app.

Project Overview

The weather dashboard app is a python application that utilizes the OpenWeather API to fetch real-time weather data for multiple cities, displays that data in a readable format in the terminal, and automates storage of the data to an AWS S3 bucket. Storing our real-time weather insights in scalable cloud storage buckets makes the data more accessible and persistent.

Key Features

• Fetches real time weather data for specified locations using OpenWeather API.
• Displays temperature, humidity, and weather conditions.
• Automatically stores weather data in AWS S3.
• Uses environment variables to securely manage API keys.
• Timestamps all data for historical tracking.

OpenWeather API

In order to dynamically fetch data or populate a web page or web app, you need to make an API call to get the data you need from other systems or services.

Since my project needs to dynamically provide feedback about the weather, I used OpenWeather API which retrieves real-time weather data, forecasts, and other climate-related information for a specified location.

Here's what it looks like to make a request:

base_url = "https://api.openweathermap.org/data/2.5/weather"
        params = {
            "q": city,
            "appid": "your-api-key",
            "units": "imperial"
        }
        try:
            response = requests.get(base_url, params=params)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestExceptions as e:
            print(f'Error fetching weather data: {e}')
            return None

Note that my request is in a try-except block, this is for error handling. If for any reason my request fails, I'll get details about the reason why it's failing, rather than the program exiting or whatever else it would do upon failure.

Data Storage with AWS S3

I have some experience with Microsoft Azure and AzDO (Azure DevOps), but AWS is completely new to me. As such, I had to set up my AWS account using the free tier, install the AWS CLI and configure it to use SSO, learn what Amazon S3 is, and learn how to use boto3, a python library, to create and save files to an S3 bucket, in a python script. Whew.

Briefly, S3 is Amazons cloud storage, short for Simple Storage Service.. we can save data things (files, images, etc.) there. Cool.

To use boto3 to create and save files to an S3 bucket looks something like this:

import boto3
import json

s3_client = boto3.Session(
            profile_name='pro-name', region_name='your-region').client('s3')

s3_client.create_bucket(Bucket='bucket_name', CreateBucketConfiguration={
                    'LocationConstraint': 'your-region'})


s3_client.put_object(
                Bucket='bucket_name',
                Key=file_name,
            Body=json.dumps(weather_data),
                ContentType='application/json'
            )

Project Structure

weather-dashboard/
  src/
    __init__.py            # initiates python 
    weather_dashboard.py   # Defines functions/logic
  .env                     # Holds environment variables
  requirements.txt         # Dependencies

Set up Instructions

You can definitely build up this project from scratch. However, let's start with cloning my repository to streamline running the application.

1. Clone the Repository

git clone https://github.com/LilLoveBytes/weather-dashboard.git
cd weather-dashboard

2. Install Dependencies

pip install -r requirements.txt

3. Configure Environment Variable

OPENWEATHER_API_KEY=your_api_key
AWS_BUCKET_NAME=your_bucket_name

4. Configure AWS Credentials

Make sure you have AWS CLI installed and run:

aws configure

5. Run the application

python src/weather_dashboard.py

Conclusion

insert spongebob meme
What I learned today is...

• how to set up an AWS account using the free tier
• what amazon S3 is
• how to use boto3 to create and save to S3 buckets
• how to make API calls to OpenWeather API
• how to set up AWS CLI to use SSO

Further Enhancements

I am writing this after reaching the initial goal of this challenge: to create a simple app that fetches weather data, displays it to the terminal, and stores it to cloud storage.

If I have the capacity I will enhance this app by creating a webpage/ user interface for user interaction and data visualization. I will also use AWS Lambda to schedule automated data collection.

Let's Connect

You can view this project on my Github.
Also feel free to connect with me on LinkedIn.