DEV Community

Juan Carlos
Juan Carlos

Posted on

Faster Than Requests with Multi-Thread URL-to-SQLite Web Scraper

Alt Text

Faster Than Requests 0.9.6 🐍

βœ”οΈ 5 Web Scrapers, Built-in, One-Liner.
βœ”οΈ Multi-Threading mass Downloader.
βœ”οΈ GitHub Actions build from scratch on every Push.
βœ”οΈ GitHub Actions publish to PYPI on every Release.
βœ”οΈ Basic get(), post(), download(), etc.
βœ”οΈ Multi-Part data for all POST functions.
βœ”οΈ Complete documentation with examples.
βœ”οΈ Works on Alpine Linux and Dockers.
βœ”οΈ CI, PyPI, Docker, Tests, Fixes, etc.
βœ”οΈ 0 Dependency, Python 3.9 ready, native bare-metal C performance.

Multi-Threading Web Scraper

Alt Text

  • threads = True Multi-Threading, threads = False no Threads.

URL-to-SQLite Web Scraper, give it 1 URL, gives back 1 SQLite database

Alt Text

  • SQLite database can be visualized with any SQLite WYSIWYG, like https://sqlitebrowser.org
  • HTTP Headers are stored as Pretty-Printed JSON.
  • Date and Time are stored as Unix Timestamps.

Images and Photos Web Scraper

Alt Text

  • Can get the images from the new HTML5 <picture> element.

Gimme tha powah

https://github.com/juancarlospaco/faster-than-requests#extras-go-beyond-requests

  • Star faster_than_requests on GitHub!, awesome stuff incoming...

πŸ‘‘

Top comments (2)

Collapse
 
turningpro profile image
turningpro

Sounds amazing. What are the draw backs of that approach?

Collapse
 
prostomarkeloff profile image
prostomarkeloff

Nim is good! Thank you for this library, but i still wanna ask, will you implement moar in it? because it hasn't still been like awesome requests.