This is a submission for the Bright Data Web Scraping Challenge: Build a Web Scraper API to Solve Business Problems
What I Built
Tech Trend Tracker tracks technology trends from Reuters news articles, helping businesses effectively monitor industry movements through keyword rankings and semantic article search.
Features:
- Collect and process news articles using Bright Data's Web Scraper API
- Track technology companies, industry leaders, and products with keyword rankings
- Enable semantic article search using OpenAI embeddings and pgvector
- Display interactive dashboards for keyword trends and entity rankings
Demo
⚡️ Live demo: Tech Trend Tracker
🐙 Source code: GitHub
Screenshots:
The dashboard shows keyword rankings of tech companies, people, and services. Click any entity to see all related articles (keyword matching).
Beyond basic keyword matching, the semantic search bar uses OpenAI embeddings to find conceptually similar articles.
How I Used Bright Data
The project leverages Bright Data's Web Scraper API to collect Reuters technology articles efficiently and reliably. I implemented:
- Optimized batch collection with configurable time ranges
- Robust error handling and retry logic
- Structured data extraction for seamless processing
Additional Qualification:
This project might also qualify for "Prompt 3: Most Creative Use of Web Data for AI Models" as it leverages embeddings for semantic search and entity recognition.
This project brings together the tech stack I explored in 2024. I enjoyed working on the holiday project!
Top comments (0)