A weather data collection system using AWS S3 and OpenWeather API.

Introduction

Me:

Hi, there! I'm new here (this is my first post, excitement). Instead of over analyzing every single word -- and ultimately feeling like I'll never find the perfect combination of words-- I am going to treat this as what it is, a blog. I'm here to document my learning, but the educator in me hopes this aids in your learning process in some way too.

I have experience in Full Stack development and have gotten my toes wet with some DevOps tools during my internship at Microsoft. To expound upon that and gain more familiarity with cloud computing, I recently joined a 30-Day DevOps coding challenge. Our Week 1 Day 1 challenge is to create a weather data collection system using AWS S3 and OpenWeather API. This blog will cover the steps I took to create this app.

Project Overview

The weather dashboard app is a python application that utilizes the OpenWeather API to fetch real-time weather data for multiple cities, displays that data in a readable format in the terminal, and automates storage of the data to an AWS S3 bucket. Storing our real-time weather insights in scalable cloud storage buckets makes the data more accessible and persistent.

Key Features

• Fetches real time weather data for specified locations using OpenWeather API.
• Displays temperature, humidity, and weather conditions.
• Automatically stores weather data in AWS S3.
• Uses environment variables to securely manage API keys.
• Timestamps all data for historical tracking.

OpenWeather API

In order to dynamically fetch data or populate a web page or web app, I need to make an API call to get the data I need from other systems or services.

Since my project needs to dynamically provide feedback about the weather, I used OpenWeather API which retrieves real-time weather data, forecasts, and other climate-related information for a specified location.

Here's what it looks like to make a request:

base_url = "https://api.openweathermap.org/data/2.5/weather"
        params = {
            "q": "city-name",
            "appid": "your-api-key",
            "units": "imperial"
        }
        try:
            response = requests.get(base_url, params=params)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestExceptions as e:
            print(f'Error fetching weather data: {e}')
            return None

I wrap my request is in a try-except block for error handling. If for any reason my request fails, rather than the program simply exiting (or doing some other not-detailed action), I'll get a response with details about the failure that will aid in debugging.

Data Storage with AWS S3

This project was my first time using AWS. So in order to get started I had to set up my AWS account (using the free tier), install the AWS CLI, configure the CLI to use SSO (single sign on which is a means of authentication), learn what Amazon S3 is, and learn how to use boto3, a python library, to create and save files to an S3 bucket, in a python script. Whew.

Briefly, S3 is Amazons cloud storage, short for Simple Storage Service.. we can save data things (files, images, etc.) there. SSO - Single Sign On - is a means of authentication for accessing AWS accounts. Boto3 is the python library we use to interact with AWS services and manage resources, like S3, using python code.

To use boto3 to create and save files to an S3 bucket looks something like this:

import boto3
import json

s3_client = boto3.Session(
            profile_name='pro-name', region_name='your-region').client('s3')

s3_client.create_bucket(Bucket='bucket_name', CreateBucketConfiguration={
                    'LocationConstraint': 'your-region'})


s3_client.put_object(
                Bucket='bucket_name',
                Key=file_name,
            Body=json.dumps(weather_data),
                ContentType='application/json'
            )

Project Structure

weather-dashboard/
  src/
    __init__.py            # initiates python 
    weather_dashboard.py   # Defines functions/logic
  .env                     # Holds environment variables
  requirements.txt         # Dependencies

Set up Instructions

These are the steps one could take to run my version of the app.

1. Clone the Repository

git clone https://github.com/LilLoveBytes/weather-dashboard.git
cd weather-dashboard

2. Install Dependencies

pip install -r requirements.txt

3. Configure Environment Variable

OPENWEATHER_API_KEY=your_api_key
AWS_BUCKET_NAME=your_bucket_name

4. Configure AWS Credentials

Make sure you have AWS CLI installed and run:

aws configure

5. Run the application

python src/weather_dashboard.py

Conclusion

insert spongebob meme

What I learned today is...

• how to set up an AWS account using the free tier
• what amazon S3 is
• how to use boto3/python to manage S3 resources
• how to make API calls to OpenWeather API
• how to set up AWS CLI to use SSO

Further Enhancements

I am writing this after reaching the initial goal of this challenge: to create a simple app that fetches weather data, displays it to the terminal, and stores it to cloud storage.

If I have the capacity I will enhance this app by creating a webpage/ user interface for user interaction and data visualization. I will also use AWS Lambda to schedule automated data collection.

Let's Connect

You can view this project on my Github.
Also feel free to connect with me on LinkedIn.