Hey, we're Apify , and we've been scraping the web for over 8 years. You can build, deploy, share, and monitor fast, reliable web scrapers on the Apify platform. Check us out .
Does Google even need an introduction? If our phones have become extensions of our hands, Google is one of the main reasons for that evolution. Google is a synonym for answers, speed, and accessibility - but also for billions of dollars, users, clicks, searches, and billions of terabytes of data. That data can be extracted automatically and effortlessly if you know the proper methods and have the right tools at hand.
In this brief how-to article, were going to show you exactly how to scrape the most extensive library in the world by using a ready-made tool on the Apify platform called Google Search Results Scraper. This is your step-by-step guide to how to scrape any information available from Google, including organic and paid results, ads, queries, People Also Ask boxes, prices, and reviews. Lets get started!
🕵 What are Google SERPs?
A Google SERP is a page containing the list of search results that Google displays to you when you type in your query and hit Enter. SERP, in this case, stands for Search Engine Results Page, and youll find SERPs not only on Google, which controls 90% of the search engine market, but also on other search engines, such as Bing, Yahoo, and others. We need to know this term to understand how to use web scraping on the Google Search Engine. You can consider the terms Google page, Google search page, and Google SERP to be equal and interchangeable, but well stick with Google SERP for the sake of being technically correct.
What do Google SERPs look like?
Google SERPs have changed a lot over the years, with the most prominent features being those infoboxes we all know too well - Knowledge Graphs, Carousels, Featured Snippets - so ubiquitous these days that we cant imagine the Google SERP interface looking any other way. Those now-classic Google SERP features were part of the Hummingbird algorithm release in 2013.
Its a far cry from the 2003 version of Google results. Does this prehistoric SERP interface ring a bell? Luckily, were not there anymore.
Google SERP interface in the beginning of the 2000s
How do you scrape Google SERP?
To scrape Google search results, we first need to understand how Google sees and prioritizes our searches. When you search for things on Google, what you see is not just an index of pages with URLs or so-called organic searches. While it used to be like that in the past (as seen above), the primary purpose and driving force of Google - or any search engine for that matter - has always been to have your queries answered as quickly and efficiently as possible, and in a way that will attract your attention but be easy on the eyes.
Thats why over time, Google search results have become much more multilayered, including the results of varying complexity and formats, like a giant layer cake. And that cake-like structure isnt going away any time soon, with voice command search, apps, and mobile search introducing their significant corrections into the way we google stuff. Today, Google search results consist of various levels, depending on the complexity and type of search, as you can see in this example of a James Webb Space Telescope query 🔭
Search results for James Webb Telescope
What are the elements of a Google search page?
So these days, Google SERP is packed with various content: featured snippets, so-called snap packs, ads, and organic results. Additional types may also show up: product ads, related searches, and multiple snap pack types (Wikipedia, Google Maps, YouTube videos, etc.)
The elements you receive will depend on the type of search query. For instance, as the James Webb Telescope is rather scientific and educational content, you won't see paid ads there or a carousel with products. But you will see those when googling everyday objects like shoes or headphones. Leveraging this complicated Google structure means giving access to an incredible amount of useful data, all of which you can scrape.
🔧 How to use data extracted from Google
Google is the main entry point to the internet for billions of people. This makes appearing in Google Search results a key factor for almost every business. And Google reviews and ratings have a massive impact on local businesses online profiles. Marketing agencies, especially those with a large number of clients from various industries, rely heavily on obtaining reliable SEO tools, including advanced AI tools. They are not only a means of effectively performing multiple tasks but also a means of successful management and analysis of results. You can look for things like how the top-ranking pages are writing their page titles, the keywords they're targeting, how they format their content, or take it a stage further and do some deeper link analysis.
Typical use cases for Google Search scraping are, among thousands of others:
Analyze Google algorithm and identify its main trends
Gain insights for Search engine optimization (SEO)monitor how your website performs in Google for specific queries over a period of time
Analyze ads ranking for a given set of keywords
Monitor competition in both organic and paid results
Build a URL list for specific keywords. This is useful if you, for example, need good relevant starting points when scraping web pages containing specific phrases.
Is it legal to scrape Google?
Google search results fall into the category of publicly available data, so scraping Google search results is legal. But there is still some data you should not be accumulating, such as personal information or copyrighted content. Learn more about regulations and laws connected to scraping at our legality article.
🤖 Can I use AI to scrape Google?
AI is currently unable to scrape websites directly, but it can help generate code for scraping Google if you prompt it with the target elements you want to scrape. Note that the code may not be functional, and website structure and design changes may impact the targeted elements and attributes.
🦾 Does Google search have an API?
Scraping Google search results is how you can create your own Google SERP API (read What is an API? for more info on APIs) to extract data from Google. But why? Technically, you can fish out some insights into the way Google works and displays results without the need to use any specific tools: just google your keyword and see what you get. Now google the same thing in incognito mode and see what you get again. But there are two problems with this approach.
First, the process is pretty time-consuming to do manually and at scale - an inefficient monkey job, essentially. Second, the results you get can't be considered objective , even if you're using incognito mode. At the beginning of the 2000s, when the Google SERPs were first introduced, they looked much the same to each user for the localized Google version per each country. Now Google algorithms have evolved to really zero in on the most relevant results, so they give out customized results tailored to each user. Google search results these days take into account many factors, such as:
Type of device 📱 If a user searches using their smartphone, the search results will look different since, starting in 2015, Google prefers showing mobile-optimized web pages.
Registration 🔒 If a Google user is logged into their account, what they see on SERPs will be aligned with their history and user behavior, provided that's allowed within their data-related settings.
Browser history 📖 If a user rarely empties their browser cache, Google will include that information concerning previous search queries with cookies and adjust the results.
Location 📍 If the geolocalization option is activated, Google aligns the SERPs with the user's location. That's why search results for the sushi takeaway query in Prague will be different from those in Los Angeles. If we're talking about local search, the results will be a combination of data from Google Search and Google Maps.
Just look at how differently Google shows the results for hot air balloon for users in Australia and Ukraine. Google's algorithm is highly adaptable to location, browser history, and device type.
What about the official Google Search API?
That's a funny question. Google doesn't provide its own SERP API for web search - so Google doesn't make it that easy to extract data from Google at scale. Moreover, only a limited subset of information available on any search results page can be provided to you via Google services such as Google Ads or Google Analytics. The two official methods suggested by Google for getting data are Google Custom Search API (deprecated in April 2018) and scraping by URLFetch method.
💡If you're interested in the Google APIs topic in more depth, see our quick guide through Google APIs: Top Google search APIs in 2022
How do you scrape a search engine?
So there's a lack of a working solution from Google, a demand to search results objectively, and lots of manual work ahead. The solution to these issues is an automated crawler that is simple enough to use and complex enough to scrape such a massive website as Google. In other words, an alternative SERP API - that's a lot of Caps, but essentially it's a program that will automatically collect data from Google SERP for you to analyze and use. This is precisely what our Google Search Result Scraper is created for.
Our SERP API supports the extraction of all data on:
organic and paid results 🔍
ads 🛍
queries
People Also Ask 🙋
prices 🏷
reviews
If you need additional attributes, you can also include a short snippet of JavaScript code to extract additional attributes from the HTML. But for now, you can try it out for free on our platform.
🔍 How to scrape Google search pages
Now that we've covered all the aspects and reasons for scraping Google let's get started with the tutorial. Promise it won't take long :), but if you would prefer a short video tutorial, we have one for you right here 📼:
How to scrape data from Google search pages
Step 1. Go to Google Search Results Scraper
Go to the scraper's page, and click the Try for free button. You will be redirected to Apify Console, which is your workspace to run tasks for your scrapers.
Google Search Results Scraper's page on Apify Store
If you are not signed in, you'll find yourself on the sign-up page. Sign up using your email account, Google, or GitHub - no credit card required. You will be redirected to the scraper's page on your Apify Console.
Log in or sign up with Google, GitHub, or your email address
Step 2. Insert the keyword you want to scrape
Now fill in the input fields. You can provide keywords or Google Search URLs as many as you want. Let's see what Google data we'll get for hot air balloon. We can use either a search URL or just the keyword; either will work.
Google Search Results Scraper on Apify Console - typing in the search query
Step 3. Choose the number of pages for extraction
Now set how many Google pages you want to scrape and how many results you want to see on each page. So if you are interested in the first 2 pages and 50 results on each, then you'll scrape 100 pages in total. But so is true if you will set the scraper for the first 5 pages and 20 results on each.
Choose the number of results and Google pages to scrape
Note that Google will always show you that it has found a gazillion results for almost any query. But in reality, the number of found pages rarely exceeds a few thousand 🤥 This is why it's unlikely that you will find a gazillion pages in your scraped run. You can check it yourself if you set Results per page to max on your Google account settings and see how the 000000 part of Google pages shrinks to just a few.
Step 4. Set up country domain and language of search
Last but not least - where are we searching? This is your time to specify the domain or country + the language. You can mix&match them as you like (for instance, find all French-speaking results in Canadian Google), but we're going to go with Czech-in-Czech.
Set up country and language of search. We're going to go for Czech Google in the Czech language 🇨🇿
Step 5. Collect your data extracted from Google search
Once you are all set, click the Start button. Notice that your task will change its status to Running, so wait for the scraper's run to finish. It will be just a minute before you see the status switch to Succeeded. In our case - 13 seconds, wow! 🚀
Note that we've only got 4 results which only means the scraper finished extracting data from the Google pages just as we've asked it. The actual number of results is 100 because we've decided to scrape 2 pages with 50 results each (see Step 3 ). You can view all results, only organic results or only paid results. You can also preview them as a table or in JSON.
After Google Scraper has finished extracting search data, you can preview the results: all, organic and paid.
You can also preview the data before downloading it by clicking the Preview in another tab 👁 button or viewing it in a new tab if the dataset is too large.
Step 6. View and download your Google data
Once you're ready to download Google data, move to the Export button to see the results of your scraping. Now you can download your scraped data in many formats, including HTML table, JSON, CSV, Excel, XML, and RSS feed.
You can choose to download only paid results, only organic results or all of them. Before downloading, you can also narrow down your dataset to a few specific fields that you want to keep or discard. Last option: getting your results via an API.
View and download your data in different formats
🛡 Do I need proxies for scraping Google SERPs?
Yes. Apify has proxies designed specifically for SERPs. Our SERP proxies will make your scraping much faster, and you'll be able to dynamically switch between countries so that you can get search information from any location. If you sign up for a free Apify account, you get a 30-day free trial of our SERP proxy service.
Now that you're all ready go ahead and start your first month with Apify by using our Google Search Scraper. If you need to scrape other parts of the Google giant, check out our other Google scrapers at Apify Store.
Top comments (0)