ArcticDB has been the modern database for Pandas' dataframe. It can handle billions of rows at scale, making it efficient for quantitative analysis. Therefore, I've decided to give it a spin in my data scrapping project.
Besides, Serverless Framework has been my top choice when it comes to developing Lambda function and its deployment to AWS. In this project, I wrote a data scrapping function that gets triggerred every min to scrap news and store them into ArcticDB.
First I created a S3 Bucket named devto-arctic, then connect locally with Jupyter Notebook to set up its library. I have opted to use AWS access key method to gain connection to the storage bucket.
# Jupyter Notebook
import pandas as pd
import arcticdb as adb
import os
import dotenv
dotenv.load_dotenv()
ac = adb.Arctic(f"s3://s3.us-east-2.amazonaws.com:devto-arctic?region=us-east-2&access={os.getenv('AWS_ACCESS_KEY_ID')}&secret={os.getenv('AWS_SECRET_ACCESS_KEY')}")
ac.create_library('intro')
ac.list_libraries() # output list of library in the db
df = pd.DataFrame()
ac.write('news_frame', df) # writing a empty df to a table
You will notice in your S3 bucket, a file *prefix*Intro will be created within.
Then, we should proceed to setting up Lambda Function with serverless framework. After npm install serverless, we can then initialize a Python project. Run serverless login to login to your serverless account before initialization. Next, execute serverless to choose a scheduled task python template as starter.
Once initialized, you'll get a Python project folder with all the necessary files. In the handler.py, it should be your function codes to make connection to ArcticDB for performing data read and write.
# handler.py
import datetime
import logging
import arcticdb as adb
import requests
import pandas as pd
import json
from dotenv import load_dotenv
import os
load_dotenv()
ac = adb.Arctic(f"s3://s3.us-east-2.amazonaws.com:devto-arctic?region=ap-southeast-1&access={os.environ['AWS_ACCESS_KEY_ENV']}&secret={os.environ['AWS_SECRET_ACCESS_KEY_ENV']}")
lib = ac.get_library('intro', create_if_missing=True)
ac.list_libraries()
ac.list_symbols() # symbols are equivalent to tables in a library
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
def fetch_news():
url = "https://news.endpoint.com/api?limit=500" # dummy endpoint
try:
response = requests.get(url)
response.raise_for_status() # Raise an exception for bad status codes
return response.json()
except requests.RequestException as e:
logger.error(f"Error fetching news: {str(e)}")
return None
def run(event, context):
symbol = 'news_frame'
current_time = datetime.datetime.now().timestamp() * 1000
logger.info("Your cron function ran at " + str(datetime.datetime.now().time()))
# Fetch news data
news_data = fetch_news()
if news_data is None:
return {
'statusCode': 500,
'body': json.dumps('Failed to fetch news data')
}
df = pd.DataFrame([{
'time': datetime.datetime.fromtimestamp(int(news['time'])/1000), # Convert ms to datetime
'title': str(news.get('title', '')),
'source': str(news.get('source', '')),
'news_id': str(news.get('news_id', '')),
'url': str(news.get('url', '')),
'icon': str(news.get('icon', '')),
'image': str(news.get('image', ''))
} for news in news_data])
try:
print(f"\nWriting DataFrame for {symbol}:")
lib.append(symbol, df) # use append so it doesn't overwrite old data
print(f"Successfully wrote {symbol} to ArcticDB")
except Exception as e:
print(f"Error writing {symbol} to ArcticDB: {str(e)}")
logger.info(f"Successfully processed news articles")
return {
'statusCode': 200,
'body': json.dumps({
'message': 'Successfully processed news data',
'time': str(current_time)
})
}
Now we can deploy the Lambda function, but first make sure requirements.txt has all the dependencies:
# requirement.txt
arcticdb; sys_platform != "darwin"
requests
pandas
numpy
python-dotenv
Note that we skip arcticdb from pip install because of binary support for Mac machine is not yet ready at the time of writing. Running pip install locally could fail without the sys_platform != "darwin" syntax. This is a work around so that Mac would skip installing arcticdb via pip. You don't need the syntax on Windows or Linux.
If you are on Mac and want to test the code locally, do activate a virtual python env and use conda install -c conda-forge arcticdb to install arcticdb, run serverless invoke local to execute the the function.
In the serverless' package.json, I have made sure plugin serverless-python-requirements is included so that during serverless deployment, python dependencies in the requirements.txt will be packaged as Layer for Lambda function to import the dependent modules from.
Next, if you are on Windows or Linux, you may deploy straight from local by running serverless deploy. Deploy from Mac machine could fail as arcticdb would spit error for not finding its binary distribution as mentioned.
The workaround will be using cloud CI/CD to package and deploy the Lambda.
The scripts install-plugin and deploy in package.json will be used on CI/CD. In this case, let's use Github Actions as deployment tool, deployment script as followed:
# deploy.yml
name: deploy serverless
on:
push:
branches:
- main
jobs:
deploy:
name: deploy
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
permissions:
contents: read
deployments: write
strategy:
matrix:
node-version: [18.x]
python-version: [3.9]
steps:
- uses: actions/checkout@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- run: npm ci --include=dev
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-2
- name: Install Plugin and Deploy
run: npm run install-plugin && npm run deploy
env:
SERVERLESS_ACCESS_KEY: ${{ secrets.SERVERLESS_ACCESS_KEY }}
The step to configure your AWS credentials allow serverless to deploy accordingly to your AWS environment. Please make sure the IAM for such access key is given admin permission to Lambda and S3.
Above Github Action will be triggered on push to main branch. You may configure to how you would like the deployment trigger.
After the deployment, you can see the Event Bridge is auto set up as the scheduler, and a Layer is uploaded and attached to the Lambda.
Hooray, here we go with the serverless approach to scrap data and save into ArcticDB! You may then use Jupyter Notebook to read data locally and analyze them while Lambda doing its thing in the background.
Top comments (0)