Intro
In this blog post, we'll go through the process of extracting product data from Walmart using Walmart Product API and the Python programming language.
In order to successfully extract Walmart Product results, you will need to pass the product_id parameter, this parameter is responsible for a specific product. You can extract this parameter from search results. Have a look at the Using Walmart Search Engine Results API from SerpApi blog post, in which I described in detail how to extract all the needed data.
You can look at the complete code in the online IDE (Replit).
If you prefer video format, we have a dedicated video that shows how to do that: #AskSerpApi: "How to search for 1 specific product by SKU/UPC across all Walmart Store IDs?".
What will be scraped
📌Note: By default, Walmart displays reviews at the bottom. In this case, the reviews are displayed on the right to make the image more compact.
Why using API?
There're a couple of reasons that may use API, ours in particular:
- No need to create a parser from scratch and maintain it.
- Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
- Pay for proxies, and CAPTCHA solvers.
- Don't need to use browser automation.
SerpApi handles everything on the backend with fast response times under ~2.5 seconds (~1.2 seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page.
Full Code
This code retrieves all the data with pagination:
from serpapi import GoogleSearch
import json
params = {
'api_key': '...', # https://serpapi.com/manage-api-key
'engine': 'walmart', # SerpApi search engine
'query': 'headphones', # the search query
}
search = GoogleSearch(params) # data extraction on the SerpApi backend
results = search.get_dict() # JSON -> Python dict
product_ids = [result['us_item_id'] for result in results['organic_results']]
walmart_products = []
for product_id in product_ids:
product_params = {
'api_key': '...', # https://serpapi.com/manage-api-key
'engine': 'walmart_product', # SerpApi search engine
'product_id': product_id, # Walmart ID of a product
}
product_search = GoogleSearch(product_params)
product_results = product_search.get_dict()
walmart_products.append({
'product_result': product_results['product_result'],
'reviews_results': product_results['reviews_results'],
})
print(json.dumps(walmart_products, indent=2, ensure_ascii=False))
Preparation
Install library:
pip install google-search-results
google-search-results is a SerpApi API package.
Code Explanation
Import libraries:
from serpapi import GoogleSearch
import json
| Library | Purpose |
|---|---|
GoogleSearch |
to scrape and parse Google results using SerpApi web scraping library. |
json |
to convert extracted data to a JSON object. |
At the beginning of the code, you need to make the request in order to get search results. Then the product_id will be extracted from them.
The parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the params dictionary:
params = {
'api_key': '...', # https://serpapi.com/manage-api-key
'engine': 'walmart', # SerpApi search engine
'query': 'headphones', # the search query
}
Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:
search = GoogleSearch(params) # data extraction on the SerpApi backend
results = search.get_dict() # JSON -> Python dict
At the moment, the first 50 search results from 1st page are stored in the results dictionary. If you are interested in all search results with pagination, then check out the Using Walmart Search Engine Results API from SerpApi blog post.
The product_ids list stores the IDs of the product which are extracted from each search result. You can pass upc, product_id and us_item_id. Better to use us_item_id because this id returns more reviews. These data will be needed later:
product_ids = [result['us_item_id'] for result in results['organic_results']]
Declaring the walmart_products list where the extracted data will be added:
walmart_products = []
Next, you need to access each product page separately by iterating the product_ids list:
for product_id in product_ids:
# data extraction will be here
These parameters are defined for generating the URL about the product. If you want to pass other parameters to the search, you can do so using the product_params dictionary:
product_params = {
'api_key': '...', # https://serpapi.com/manage-api-key
'engine': 'walmart_product', # SerpApi search engine
'product_id': product_id, # Walmart ID of a product
}
| Parameters | Explanation |
|---|---|
api_key |
Parameter defines the SerpApi private key to use. You can find it under your account -> API key |
engine |
Set parameter to walmart_product to use the Walmart Product API engine. |
product_id |
Parameter defines the product to get results for. |
Then, we create a product_search object where the data is retrieved from the SerpApi backend. In the product_results dictionary we get a new package of the data in JSON format:
product_search = GoogleSearch(product_params)
product_results = product_search.get_dict()
Adding information and reviews about the current product to the walmart_products list:
walmart_products.append({
'product_result': product_results['product_result'],
'reviews_results': product_results['reviews_results'],
})
# title = product_results['product_result']['title']
# rating = product_results['product_result']['rating']
# reviews = product_results['product_result']['reviews']
# price = product_results['product_result']['price_map']['price']
📌Note: In the comments above, I showed how to extract specific fields from the current product.
After the all data is retrieved, it is output in JSON format:
print(json.dumps(walmart_products, indent=2, ensure_ascii=False))
Output
[
{
"product_result": {
"us_item_id": "376188834",
"product_id": "74W4IGCNZME8",
"upc": "017817835015",
"title": "Bose QuietComfort 45 Headphones Noise Cancelling Over-Ear Wireless Bluetooth Earphones, Black",
"short_description_html": "The Bose QuietComfort 45 over-ear headphones are the perfect balance of quiet, comfort, and sound. With world-class noise cancelling performance, you can tune into your music, shows, and more without being distracted by noisy surroundings. They’re perfect for traveling, working, or relaxing at home. Simply slide the soft, plush cushions over your ears, flip the switch and experience every detail of your audio. Made with premium and durable materials, Bose QuietComfort 45 Bluetooth earphones were designed to last without sacrificing comfort. You can easily adjust the noise-cancelling setting by enabling Aware Mode. This helps you hear your music clearly while also being aware of your surroundings, such as when crossing a busy street or exercising outside. And the voice pickup of these noise cancelling headphones exceeds the competition. Tiny microphones in the earcups focus on your voice while also filtering out environmental sounds. With 24 hours of battery life per charge, these wireless headphones give you all-day power. Personalize your audio with adjustable EQ. With the Bose Music App you can adjust the bass, mid and treble to your liking or use one of the presets. The Bose Music app also guides you through the simple setup process, lets you personalize settings, and keeps your Bluetooth headphones’ software up to date. Bose QuietComfort 45, the ultimate noise cancelling headphones make every day sound amazing.",
"detailed_description_html": "<p><strong>Key Features</strong></p> <ul> <li>Zone in on your music, work, and favorite shows without any distractions with world-class noise-cancelling technology in these Bluetooth headphones. </li> <li>Personalize your audio with adjustable EQ. With the Bose Music App you can adjust the bass, mid and treble to your liking or use one of the presets.</li> <li>Choose from Quiet Mode for full noise cancelling, or Aware Mode to let in some of your surroundings.</li> <li>Get 24 hours of battery life on a single charge. A quick, 15-minute charge via a USB-C cable gives your wireless headphones up to 3 hours of play time.</li> <li>Enjoy premium comfort and durability with soft earcups and impact-resistant materials.</li> <li>Have crystal clear conversations with the advanced voice pickup of QC45 over-ear headphones. Tiny microphones in the earcups focus on your voice, while noise-rejecting technology filters out unwanted sounds.</li> <li>The Bose Music app walks you through a simple setup process, making it easy to get started. Plus, access all noise cancellation settings, manage your Bluetooth connections.</li> <li>Available in Black and White Smoke.<br /> </li> </ul>",
"categories": [
{
"name": "Electronics",
"url": "https://www.walmart.com/cp/3944"
},
{
"name": "Audio",
"url": "https://www.walmart.com/cp/133251"
},
{
"name": "Headphones",
"url": "https://www.walmart.com/cp/1095191"
},
{
"name": "Shop Headphones by Type",
"url": "https://www.walmart.com/cp/1230614"
},
{
"name": "Over-Ear and On-Ear Headphones",
"url": "https://www.walmart.com/cp/1230477"
}
],
"seller_id": "F55CDC31AB754BB68FE0B39041159D63",
"seller_name": "Walmart.com",
"specification_highlights": [
{
"key": "Features",
"value": "Bluetooth, Wireless, Active Noise Canceling",
"display_name": "Features"
},
{
"key": "Brand",
"value": "Bose",
"display_name": "Brand"
},
{
"key": "Color",
"value": "Black",
"display_name": "Color"
},
{
"key": "Manufacturer",
"value": "BOSE",
"display_name": "Manufacturer"
},
{
"key": "Manufacturer Part Number",
"value": "866724-0100",
"display_name": "Manufacturer Part Number"
},
{
"key": "Assembled Product Dimensions (L x W x H)",
"value": "6.34 x 7.25 x 3.25 Inches",
"display_name": "Assembled Product Dimensions (L x W x H)"
}
],
"manufacture_number": "866724-0100",
"product_type_id": "4159",
"product_type": "Headphones",
"manufacturer": "Bose",
"product_page_url": "https://www.walmart.com/ip/Bose-QuietComfort-45-Headphones-Noise-Cancelling-Over-Ear-Wireless-Bluetooth-Earphones-Black/376188834",
"price_map": {
"price": 249,
"was_price": {
"price": 329,
"priceString": "$329.00",
"variantPriceString": null,
"currencyUnit": "USD",
"bestValue": null
},
"currency": "USD"
},
"min_quantity": 1,
"max_quantity": 2,
"in_stock": true,
"images": [
"https://i5.walmartimages.com/asr/5b396d09-ff10-4c63-a10b-c042d8424c20.57346e60325e7fe6451be4c388aef066.jpeg",
"https://i5.walmartimages.com/asr/2c94a51f-2302-4f31-8bb1-dd19f4f29c51.a54f9b3a87c8a54d75e63f7d82a8baa8.jpeg",
"https://i5.walmartimages.com/asr/856b7189-1169-48cf-9801-9a768f7dbd66.33b6773c5e2a09f0493799d8d653a6a7.jpeg",
"https://i5.walmartimages.com/asr/3cb905d4-a3c5-448a-a83e-7f926b9d9823.20ec04a31a488e360006675f12523f48.jpeg",
"https://i5.walmartimages.com/asr/c6d3c05f-9714-4220-a3d1-9ad42802acf3.1eafa6ed59cc921fb4577ce3fc4c52a0.jpeg",
"https://i5.walmartimages.com/asr/b9debe8a-ea8f-4353-a916-88e74d1bc7ad.84bb47f740752c63cc1c1997df11c4d2.jpeg",
"https://i5.walmartimages.com/asr/dbd9a82d-f8d5-4d6a-b12e-e8fe59befaef.fc05bf60cd3ed060d014caf87aa29f0f.jpeg",
"https://i5.walmartimages.com/asr/3102eca2-e712-4109-9523-187b51b41273.9b985336a7ac3fd8b09e76cf31cc8b04.jpeg",
"https://i5.walmartimages.com/asr/2fa6515e-b758-4cbc-9360-9ec04c8aa409.bb195e619b3613fa376a9c897d1353ff.jpeg",
"https://i5.walmartimages.com/asr/a48cf165-fe22-40c1-bb9b-f1b362cc0655.c104b06991b8c5eea0bc4d8c18f96eee.jpeg",
"https://i5.walmartimages.com/asr/a36e97dd-e921-41bf-95b7-93eddefaaeba.91bd11212981850b24f53c0fd9bc6c05.jpeg"
],
"reviews": 869,
"rating": 4.5,
"offer_id": "F9F0C098DAB445A78BAE717A05E38015",
"offer_type": "ONLINE_AND_STORE",
"offers": [
{
"seller_id": "F55CDC31AB754BB68FE0B39041159D63",
"catalog_seller_id": 0,
"seller_name": "Walmart.com",
"seller_display_name": "Walmart.com",
"price": 249
}
],
"delivery_option": {
"ship_method": "TWO_DAY",
"arrival_date": 1670972340000,
"display_arrival_date": "2022-12-13T22:59:00.000Z"
},
"pickup_options": [
{
"availability": "AVAILABLE",
"pickup_method": "PICKUP_INSTORE"
},
{
"availability": "AVAILABLE",
"pickup_method": "PICKUP_CURBSIDE"
}
],
"variant_swatches": [
{
"name": "Actual Color",
"available_selections": [
{
"name": "Black",
"swatch_image_url": "https://i5.walmartimages.com/asr/a1c698b3-5ed6-4355-b8a8-b4b35e3ee638.9e679770605163dfb7923ad4574c005f.jpeg"
},
{
"name": "Eclipse Grey",
"swatch_image_url": "https://i5.walmartimages.com/asr/3c930d6e-e832-47fa-b473-70cdd061b9b3.f5a4086134e1c61b0ce8e83bdfa8e176.jpeg"
},
{
"name": "Midnight Blue",
"swatch_image_url": "https://i5.walmartimages.com/asr/2916eeb3-6bf3-4534-8c07-8f5961e57f66.86561d84b473f1cc8bf8975ace3223e2.jpeg"
},
{
"name": "White Smoke",
"swatch_image_url": "https://i5.walmartimages.com/asr/ac2d39f6-6907-46d6-9ccd-3fde33a59dbd.806fc44965831934f4a92835bf41dfe0.jpeg"
}
]
}
]
},
"reviews_results": {
"ratings": [
{
"stars": 1,
"count": 29
},
{
"stars": 2,
"count": 26
},
{
"stars": 3,
"count": 45
},
{
"stars": 4,
"count": 127
},
{
"stars": 5,
"count": 642
}
],
"reviews": {
"rating": 4.5,
"count": 869,
"top_positive": {
"title": "Bose knows headphones!",
"text": "Comfortable and wireless, I absolutely love them! The clarity is remarkable and the case is a big plus! ⭐⭐⭐⭐⭐",
"rating": 5,
"positive_feedback": 0,
"negative_feedback": 0,
"review_submission_time": "2/2/2022",
"user_nickname": "Lionssherr",
"customer_type": [
"VerifiedPurchaser"
]
},
"top_negative": {
"title": "Absolute poor product. Too many issues to list...",
"text": "I use this headset mostly for work. I'm using it with my windows laptop and iphone 11. Initial setup is easy. When it works, works great. But the headset is not reliable for following reasons.\r\n1. Bluetooth connectivity keeps losing while on calls. This happens multiple times a day. While on phone call, suddenly it disconnects. I need to turn off and back on to make it work.\r\n2. Microphone doesn't work reliably. Suddenly, the other party says the noise is faded as if I'm too far away. At least 2-3 times a day.\r\n3. Bose app doesn't remember the settings. In the app, I set the aware mode is on and remember the last mode. Next day when I turn on headset, it goes back to quiet mode.\r\n\r\nOverall, I'm really frustrated with this headset.",
"rating": 1,
"positive_feedback": 4,
"negative_feedback": 1,
"review_submission_time": "9/9/2022",
"user_nickname": "ram68"
},
"top_mentions": [
{
"score_percentage": 94,
"name": "Sound",
"count": 56
},
... other results
],
"customer_reviews": [
{
"title": "Bose knows headphones!",
"text": "Comfortable and wireless, I absolutely love them! The clarity is remarkable and the case is a big plus! ⭐⭐⭐⭐⭐",
"rating": 5,
"review_submission_time": "2/2/2022",
"user_nickname": "Lionssherr",
"customer_type": [
"VerifiedPurchaser"
]
},
... other results
]
}
}
}
]
📌Note: Head to the playground for a live and interactive demo.
Links
- Code in the online IDE
- Walmart Search Engine Results API
- Walmart Organic Results API
- Walmart Product API
Add a Feature Request💫 or a Bug🐞
Top comments (0)