DEV Community

Derrick Sherrill for WayScript

Posted on

Send Text Messages from Scraped Website Data with Python on the Cloud

Introduction

Web scraping on the cloud has never been easier. Setting up an automated web scraping script on WayScript only takes a few minutes to do.

Prerequisites

No prerequisites but some content you might find helpful:

Automating a Script to Run Daily

Most things you create on WayScript can be activated daily by using a time trigger. When setting up the time trigger, we select our time that we want the script to run, and build the script below that tree in the workflow.

Scraping our content

We'll scrape our content in this example by using the python module. We'll drag this into our workflow and write some code that looks like this:

import requests
from bs4 import BeautifulSoup

ticker = 'AAPL'
url = 'https://finance.yahoo.com/quote/' + ticker

res = requests.get( url )
html = res.text

soup = BeautifulSoup( html, 'html.parser' )
market_cap_elem = soup.find( 'td', { 'data-test' : 'MARKET_CAP-value' } )
market_cap = market_cap_elem.text

print( ticker, 'Market Cap', market_cap )

variables[ 'MarketCap' ] = market_cap

With that code, we'll go and scrape information off another webste, and return it to our script as a variable using the variables dictionary. We'll use it to send ourselves a text message.

Questions, Concerns?

If there's any questions feel free to message us on discord. We're happy to help! If you want to see this full script template, you can find it here

Top comments (3)

Collapse
 
daedtech profile image
Erik Dietrich

Looks like WayScript gives you some pretty interesting capabilities. Do people mainly use it for rapid prototyping/quick scripts, or does it scale to support full-blown production applications?

Collapse
 
derricksherrill profile image
Derrick Sherrill

It's definitely great for getting quick scripts running in a few clicks, but our focus is making it a great application for teams, internal tools, and production apps. We still have some work to do in this regard, but that's where we want to be.

Collapse
 
daedtech profile image
Erik Dietrich

Got it, thanks. Sounds like if you use it for rapid prototyping, then, you can evolve the prototype rather than toss it. Always promising :)