How to Build a Product Scraper for Infinite Scroll Websites using ZenRows Web Scraper

In the realm of web scraping, accessing and extracting data from web pages that use infinite scrolling can be a challenge for developers. Many websites use this technique to load more content dynamically, making it hard to scrape all available data in one go. A good solution involves simulating user actions, like clicking a "Load More" button to reveal additional content.
This tutorial will delve into scraping product data from a page with infinite scroll, utilizing Zenrows open-source web scraper, you’ll build a scraper bot that will access contents from a web page, and you’ll use Zenrows to generate more products on the page by clicking the "Load More" button, to simulate an infinite scrolling.

Prerequisites

To follow this tutorial, you need to have the following:

• Python: You should have Python set up on your machine. If not, you can install it here.
• Web Scraping Foundation: You should have a solid grasp of how web scraping works.
• ZenRows SDK: You will be using the ZenRows service to bypass anti-scraping measures and simplify scraping dynamic content. You can sign up for a free ZenRows account here.

Getting Access to the Content

Once you have signed up for your Zenrows account and you have the prerequisites in place, the next step is to access the content from the web page; for this tutorial, you’ll be using this page https://www.scrapingcourse.com/button-click.

You'll also use ZenRows SDK to scrape the dynamic pages and handle various rendering and anti-bot measures. Let’s get you started:

Install the required Libraries:
Open the terminal of your preferred IDE and run the code to install the ZenRows Python SDK.

pip install zenrows python-dotenv

Setting Up your API

Head over to your dashboard and copy the API Key at the top right corner of your screen like in the image below.

Next, create the pages app.py and .env, then add the code below to your app.py file. And add your API key to the variable API_KEY in your .env file.

# Import ZenRows SDK
from zenrows import ZenRowsClient
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()

# Initialize ZenRows client with your API key
client = ZenRowsClient(os.getenv("API_KEY"))

# URL of the page you want to scrape
url = "https://www.scrapingcourse.com/button-click"

# Set up initial parameters for JavaScript rendering and interaction
base_params = {
    "js_render": "true",
    "json_response": "true",
    "premium_proxy": "true",
    "markdown_response": "true"
}

The code above initiates the Zenrow SDK using your API key. It sets up the URL variable for the webpage you’ll be scraping and establishes the base_params variable for the necessary parameters. You can execute the scraper using the command:

python app.py

This will provide you with the HTML representation of the page containing only the products on the current page.
.

You can always take this one step further.

Loading More Products

To enhance your scraper, you can implement additional parameters to interact with the "Load More" button at the bottom of the webpage and load more products.

Start by modifying your imports to include the necessary packages and adding a parse_products function that filters the product response:

import re
import json
import time

def parse_products(response_json):
    try:
        data = json.loads(response_json)
        md_content = data.get('md', '')
        pattern = r'\[!\[([^\]]+)\]\(([^\)]+)\)\\*\n([^\\\n]+)\\*\n\\*\n\$(\d+)\]\(([^\)]+)\)'
        matches = re.findall(pattern, md_content)

        products = []
        for match in matches:
            product = {
                'name': match[0],
                'image_link': match[1],
                'price': int(match[3]),
                'product_url': match[4]
            }
            products.append(product)

        return products
    except json.JSONDecodeError:
        print("Error: Unable to parse JSON response")
        print("Response content:", response_json[:500])
        return []
    except Exception as e:
        print(f"Error parsing products: {str(e)}")
        return []

# Zenrow SDK code here

Next, create a while loop to continuously scrape product information from multiple pages until a specified limit (max_products). Set the limit to 50 for this tutorial:

# Zenrow SDK code goes here

max_products = 50
all_products = []
page = 1

while len(all_products) < max_products:
    print(f"Scraping page {page}...")

    # Update parameters for each request
    params = base_params.copy()
    js_instructions = [{"click": "#load-more-btn"} for _ in range(page)]
    js_instructions.append({"wait": 5000})
    params["js_instructions"] = json.dumps(js_instructions)

    try:
        # Send the GET request to ZenRows
        response = client.get(url, params=params)

        # Parse the response JSON
        new_products = parse_products(response.text)

        if not new_products:
            print("No more products found. Stopping.")
            break

        all_products.extend(new_products)
        print(f"Found {len(new_products)} products on this page.")
        print(f"Total products so far: {len(all_products)}")

        page += 1

        # Add a delay to avoid overwhelming the server
        time.sleep(2)

    except Exception as e:
        print(f"Error occurred: {str(e)}")
        break

This loop will continue to scrap products by simulating the clicking of the "Load More" button until the specified limit is reached.
Parsing Product Information
Finally, you can parse the product information you scraped in the previous step. For each product, extract the product name, image link, price, and product page URL. You can also calculate the total price of all products and print the results as follows:

# Updated Params and while loop code goes here
# Calculate the total price of all products
total_sum = sum(product['price'] for product in all_products)

print("\nAll products:")
for product in all_products:
    print(product)

# Print the total sum of the product prices
print(f"\nTotal number of products: {len(all_products)}")
print(f"Total sum of product prices: ${total_sum}")

Parsing Into a CSV file

If you would prefer to parse your response into an exported csv file, in the next few steps, you'll take the product information you've scraped and learn how to export it to a CSV file.

Modifying the Script to Save Data

First, you need to use Python's built-in CSV module to save the product data. In this case, each product has four main attributes: name, image_link, price, and product_url.

You can use them as the headers for your CSV, loop through the list of scraped products, and then write each product as a row in the CSV file.

# Parse_product function goes here

# Saving to csv
def save_to_csv(products, filename='products.csv'):
    """Save product data to a CSV file."""
    with open(filename, mode='w', newline='') as file:
        writer = csv.DictWriter(file, fieldnames=['name', 'image_link', 'price', 'product_url'])
        writer.writeheader()
        for product in products:
            writer.writerow(product)
    print(f"Saved {len(products)} products to {filename}.")

Now, after scraping the data, just call the save_to_csv(all_products) function to store the data in a CSV file named products.csv.

Run the command to automatically save the data to a CSV file once the scraping process is complete.

Python app.py

Identifying the 5 Highest-Priced Products

Now that you have all the products in a structured format, you can go a step further and identify the 5 highest-priced products, and you’ll have to visit each product page to extract extra details like the product description and SKU code.

Sorting the Products by Price: Using Python's sorted() function, you can sort the product list by the price in descending order and retrieve the top 5 products.

You’ll need to visit each page using the requests.get() function to fetch the product data for each of them. From the response, you can extract the product description and SKU code.
You can also update the csv flie from the last step to include the additional details.

Here is the code to achieve that:

# save_to_csv code goes here

# This handles the extra product details to fetch and handles errors if there are none
def extract_product_details(product_url):
    """Extract product description and SKU from a product page."""
    try:
        response = requests.get(product_url)
        response.raise_for_status()  # Check for HTTP errors
        product_data = json.loads(response.text)

        description = product_data.get('description', 'No description found')
        sku = product_data.get('sku', 'No SKU found')

        return description, sku
    except Exception as e:
        print(f"Error fetching product details from {product_url}: {str(e)}")
        return None, None

Now, after scraping, you can now identify the highest-priced products:

# While loop and zenrows updated params code goes here


# Identify the 5 highest-priced products
highest_priced_products = sorted(all_products, key=lambda x: x['price'], reverse=True)[:5]

# Print and extract extra details for each product
for product in highest_priced_products:
    description, sku = extract_product_details(product['product_url'])
    print(f"Product: {product['name']}, Description: {description}, SKU: {sku}")

After retrieving the additional information, you can either modify the CSV file or create a new one with these details included.

The Complete Code

Here is how your complete app.py file should look like.

from zenrows import ZenRowsClient
import re
import json
import time
import csv
from dotenv import load_dotenv
import os
import requests


load_dotenv()


def parse_products(response_json):
    try:
        data = json.loads(response_json)
        md_content = data.get('md', '')
        pattern = r'\[!\[([^\]]+)\]\(([^\)]+)\)\\*\n([^\\\n]+)\\*\n\\*\n\$(\d+)\]\(([^\)]+)\)'
        matches = re.findall(pattern, md_content)

        products = []
        for match in matches:
            product = {
                'name': match[0],
                'image_link': match[1],
                'price': int(match[3]),
                'product_url': match[4]
            }
            products.append(product)

        return products
    except json.JSONDecodeError:
        print("Error: Unable to parse JSON response")
        print("Response content:", response_json[:500])
        return []
    except Exception as e:
        print(f"Error parsing products: {str(e)}")
        return []


def save_to_csv(products, filename='products.csv'):
    """Save product data to a CSV file."""
    with open(filename, mode='w', newline='') as file:
        writer = csv.DictWriter(file, fieldnames=['name', 'image_link', 'price', 'product_url'])
        writer.writeheader()
        for product in products:
            writer.writerow(product)
    print(f"Saved {len(products)} products to {filename}.")


def extract_product_details(product_url):
    """Extract product description and SKU from a product page."""
    try:
        response = requests.get(product_url)
        response.raise_for_status()  # Check for HTTP errors
        product_data = json.loads(response.text)


        # Example extraction logic for description and SKU; update according to the actual structure
        description = product_data.get('description', 'No description found')
        sku = product_data.get('sku', 'No SKU found')


        return description, sku
    except Exception as e:
        print(f"Error fetching product details from {product_url}: {str(e)}")
        return None, None


client = ZenRowsClient(os.getenv("API_KEY"))


# URL of the page you want to scrape
url = "https://www.scrapingcourse.com/button-click"


# Set up initial parameters for JavaScript rendering and interaction
base_params = {
    "js_render": "true",
    "json_response": "true",
    "premium_proxy": "true",
    "markdown_response": "true"
}


all_products = []
page = 1
max_products = 50  # Set this to the number of products you want to retrieve


while len(all_products) < max_products:
    print(f"Scraping page {page}...")

    # Update parameters for each request
    params = base_params.copy()
    js_instructions = [{"click": "#load-more-btn"} for _ in range(page)]
    js_instructions.append({"wait": 5000})
    params["js_instructions"] = json.dumps(js_instructions)

    try:
        # Send the GET request to ZenRows
        response = client.get(url, params=params)

        # Parse the response JSON
        new_products = parse_products(response.text)

        if not new_products:
            print("No more products found. Stopping.")
            break

        all_products.extend(new_products)
        print(f"Found {len(new_products)} products on this page.")
        print(f"Total products so far: {len(all_products)}")

        page += 1

        # Add a delay to avoid overwhelming the server
        time.sleep(2)

    except Exception as e:
        print(f"Error occurred: {str(e)}")
        break


# Save all products to a CSV file
save_to_csv(all_products)


# Identify the 5 highest-priced products
highest_priced_products = sorted(all_products, key=lambda x: x['price'], reverse=True)[:5]


print("\nFive highest-priced products:")
for product in highest_priced_products:
    print(product)


# Visit each product page and extract details
for product in highest_priced_products:
    description, sku = extract_product_details(product['product_url'])
    print(f"Product: {product['name']}, Description: {description}, SKU: {sku}")


# Calculate the total price of all products
total_sum = sum(product['price'] for product in all_products)


print("\nAll products:")
for product in all_products:
    print(product)


# Print the total sum of the product prices
print(f"\nTotal number of products: {len(all_products)}")
print(f"Total sum of product prices: ${total_sum}")

Here is what a successful response would look like.

$ python app.py
Scraping page 1...
Found 23 products on this page.
Total products so far: 23
Scraping page 2...
Found 34 products on this page.
Total products so far: 57
Saved 57 products to products.csv.

Five highest-priced products:
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
Error fetching product details from https://scrapingcourse.com/ecommerce/product/lando-gym-jacket: Expecting value: line 1 column 1 (char 0)
Product: Lando Gym Jacket, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/lando-gym-jacket: Expecting value: line 1 column 1 (char 0)
Product: Lando Gym Jacket, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/lando-gym-jacket: Expecting value: line 1 column 1 (char 0)
Product: Lando Gym Jacket, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie: Expecting value: line 1 column 1 (char 0)
Product: Marco Lightweight Active Hoodie, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie: Expecting value: line 1 column 1 (char 0)
Product: Marco Lightweight Active Hoodie, Description: None, SKU: None

All products:
{'name': 'Chaz Kangeroo Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie'}
{'name': 'Teton Pullover Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg', 'price': 70, 'product_url': 'https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie'}
{'name': 'Bruno Compete Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg', 'price': 63, 'product_url': 'https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie'}
{'name': 'Frankie  Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh04-green_main.jpg', 'price': 60, 'product_url': 'https://scrapingcourse.com/ecommerce/product/frankie--sweatshirt'}
{'name': 'Hollister Backyard Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh05-white_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hollister-backyard-sweatshirt'}
{'name': 'Stark Fundamental Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh06-blue_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/stark-fundamental-hoodie'}
{'name': 'Hero Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh07-gray_main.jpg', 'price': 54, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hero-hoodie'}
{'name': 'Oslo Trek Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh08-brown_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/oslo-trek-hoodie'}
{'name': 'Abominable Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh09-blue_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/abominable-hoodie'}
{'name': 'Mach Street Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh10-blue_main.jpg', 'price': 62, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mach-street-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Beaumont Summit Kit', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj01-yellow_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/beaumont-summit-kit'}
{'name': 'Hyperion Elements Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj02-green_main.jpg', 'price': 51, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hyperion-elements-jacket'}
{'name': 'Montana Wind Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj03-black_main.jpg', 'price': 49, 'product_url': 'https://scrapingcourse.com/ecommerce/product/montana-wind-jacket'}
{'name': 'Kenobi Trail Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj04-black_main.jpg', 'price': 47, 'product_url': 'https://scrapingcourse.com/ecommerce/product/kenobi-trail-jacket'}
{'name': 'Orion Two-Tone Fitted Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj07-red_main.jpg', 'price': 72, 'product_url': 'https://scrapingcourse.com/ecommerce/product/orion-two-tone-fitted-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Taurus Elements Shell', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg', 'price': 65, 'product_url': 'https://scrapingcourse.com/ecommerce/product/taurus-elements-shell'}
{'name': 'Mars HeatTech™ Pullover', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg', 'price': 66, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mars-heattech™-pullover'}
{'name': 'Chaz Kangeroo Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie'}
{'name': 'Teton Pullover Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg', 'price': 70, 'product_url': 'https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie'}
{'name': 'Bruno Compete Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg', 'price': 63, 'product_url': 'https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie'}
{'name': 'Frankie  Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh04-green_main.jpg', 'price': 60, 'product_url': 'https://scrapingcourse.com/ecommerce/product/frankie--sweatshirt'}
{'name': 'Hollister Backyard Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh05-white_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hollister-backyard-sweatshirt'}
{'name': 'Stark Fundamental Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh06-blue_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/stark-fundamental-hoodie'}
{'name': 'Hero Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh07-gray_main.jpg', 'price': 54, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hero-hoodie'}
{'name': 'Oslo Trek Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh08-brown_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/oslo-trek-hoodie'}
{'name': 'Abominable Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh09-blue_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/abominable-hoodie'}
{'name': 'Mach Street Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh10-blue_main.jpg', 'price': 62, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mach-street-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Beaumont Summit Kit', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj01-yellow_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/beaumont-summit-kit'}
{'name': 'Hyperion Elements Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj02-green_main.jpg', 'price': 51, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hyperion-elements-jacket'}
{'name': 'Montana Wind Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj03-black_main.jpg', 'price': 49, 'product_url': 'https://scrapingcourse.com/ecommerce/product/montana-wind-jacket'}
{'name': 'Kenobi Trail Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj04-black_main.jpg', 'price': 47, 'product_url': 'https://scrapingcourse.com/ecommerce/product/kenobi-trail-jacket'}
{'name': 'Orion Two-Tone Fitted Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj07-red_main.jpg', 'price': 72, 'product_url': 'https://scrapingcourse.com/ecommerce/product/orion-two-tone-fitted-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Taurus Elements Shell', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg', 'price': 65, 'product_url': 'https://scrapingcourse.com/ecommerce/product/taurus-elements-shell'}
{'name': 'Mars HeatTech™ Pullover', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg', 'price': 66, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mars-heattech™-pullover'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Beaumont Summit Kit', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj01-yellow_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/beaumont-summit-kit'}
{'name': 'Hyperion Elements Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj02-green_main.jpg', 'price': 51, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hyperion-elements-jacket'}
{'name': 'Montana Wind Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj03-black_main.jpg', 'price': 49, 'product_url': 'https://scrapingcourse.com/ecommerce/product/montana-wind-jacket'}
{'name': 'Kenobi Trail Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj04-black_main.jpg', 'price': 47, 'product_uourse.com/ecommerce/product/kenobi-trail-jacket'}
{'name': 'Orion Two-Tone Fitted Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj07-red_main.jpg', 'price': 72, 'product_url': 'https://scrapingcourse.com/ecommerce/product/orion-two-tone-fitted-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Taurus Elements Shell', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg', 'price': 65, 'product_url': 'https://scrapingcourse.com/ecommerce/product/taurus-elements-shell'}
{'name': 'Mars HeatTech™ Pullover', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg', 'price': 66, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mars-heattech™-pullover'}

Total number of products: 57
Total sum of product prices: $3492

$ python app.py
Scraping page 1...
Found 23 products on this page.
Total products so far: 23
Scraping page 2...
Found 34 products on this page.
Total products so far: 57
Saved 57 products to products.csv.

Five highest-priced products:
-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}   
Error fetching product details from https://scrapingcourse.com/ecommerce/product/lando-gym-jacket: Expecting value: line 1 column 1 (char 0)
Product: Lando Gym Jacket, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/lando-gym-jacket: Expecting value: line 1 column 1 (char 0)
Product: Lando Gym Jacket, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/lando-gym-jacket: Expecting value: line 1 column 1 (char 0)
Product: Lando Gym Jacket, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie: Expecting value: line 1 column 1 (char 0)
Product: Marco Lightweight Active Hoodie, Description: None, SKU: None
Error fetching product details from https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie: Expecting value: line 1 column 1 (char 0)
Product: Marco Lightweight Active Hoodie, Description: None, SKU: None

All products:
{'name': 'Chaz Kangeroo Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie'}
{'name': 'Teton Pullover Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg', 'price': 70, 'product_url': 'https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie'}
{'name': 'Bruno Compete Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg', 'price': 63, 'product_url': 'https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie'}
{'name': 'Frankie  Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh04-green_main.jpg', 'price': 60, 'product_url': 'https://scrapingcourse.com/ecommerce/product/frankie--sweatshirt'}
{'name': 'Hollister Backyard Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh05-white_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hollister-backyard-sweatshirt'}
{'name': 'Stark Fundamental Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh06-blue_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/stark-fundamental-hoodie'}
{'name': 'Hero Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh07-gray_main.jpg', 'price': 54, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hero-hoodie'}
{'name': 'Oslo Trek Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh08-brown_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/oslo-trek-hoodie'}
{'name': 'Abominable Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh09-blue_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/abominable-hoodie'}
{'name': 'Mach Street Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh10-blue_main.jpg', 'price': 62, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mach-street-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Beaumont Summit Kit', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj01-yellow_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/beaumont-summit-kit'}
{'name': 'Hyperion Elements Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj02-green_main.jpg', 'price': 51, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hyperion-elements-jacket'}
{'name': 'Montana Wind Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj03-black_main.jpg', 'price': 49, 'product_url': 'https://scrapingcourse.com/ecommerce/product/montana-wind-jacket'}
{'name': 'Kenobi Trail Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj04-black_main.jpg', 'price': 47, 'product_url': 'https://scrapingcourse.com/ecommerce/product/kenobi-trail-jacket'}
{'name': 'Orion Two-Tone Fitted Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj07-red_main.jpg', 'price': 72, 'product_url': 'https://scrapingcourse.com/ecommerce/product/orion-two-tone-fitted-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Taurus Elements Shell', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg', 'price': 65, 'product_url': 'https://scrapingcourse.com/ecommerce/product/taurus-elements-shell'}
{'name': 'Mars HeatTech™ Pullover', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg', 'price': 66, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mars-heattech™-pullover'}
{'name': 'Chaz Kangeroo Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh01-gray_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie'}
{'name': 'Teton Pullover Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh02-black_main.jpg', 'price': 70, 'product_url': 'https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie'}
{'name': 'Bruno Compete Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh03-black_main.jpg', 'price': 63, 'product_url': 'https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie'}
{'name': 'Frankie  Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh04-green_main.jpg', 'price': 60, 'product_url': 'https://scrapingcourse.com/ecommerce/product/frankie--sweatshirt'}
{'name': 'Hollister Backyard Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh05-white_main.jpg', 'price': 52, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hollister-backyard-sweatshirt'}
{'name': 'Stark Fundamental Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh06-blue_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/stark-fundamental-hoodie'}
{'name': 'Hero Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh07-gray_main.jpg', 'price': 54, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hero-hoodie'}
{'name': 'Oslo Trek Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh08-brown_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/oslo-trek-hoodie'}
{'name': 'Abominable Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh09-blue_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/abominable-hoodie'}
{'name': 'Mach Street Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh10-blue_main.jpg', 'price': 62, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mach-street-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Beaumont Summit Kit', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj01-yellow_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/beaumont-summit-kit'}
{'name': 'Hyperion Elements Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj02-green_main.jpg', 'price': 51, 'product_url': 'https://scrapingcourse.com/ecommerce/product/hyperion-elements-jacket'}
{'name': 'Montana Wind Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj03-black_main.jpg', 'price': 49, 'product_url': 'https://scrapingcourse.com/ecommerce/product/montana-wind-jacket'}
{'name': 'Kenobi Trail Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj04-black_main.jpg', 'price': 47, 'product_url': 'https://scrapingcourse.com/ecommerce/product/kenobi-trail-jacket'}
{'name': 'Orion Two-Tone Fitted Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj07-red_main.jpg', 'price': 72, 'product_url': 'https://scrapingcourse.com/ecommerce/product/orion-two-tone-fitted-jacket'}
{'name': 'Lando Gym Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Taurus Elements Shell', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg', 'price': 65, 'product_url': 'https://scrapingcourse.com/ecommerce/product/taurus-elements-shell'}
{'name': 'Mars HeatTech™ Pullover', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg', 'price': 66, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mars-heattech™-pullover'}
{'name': 'Grayson Crewneck Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh11-white_main.jpg', 'price': 64, 'product_url': 'https://scrapingcourse.com/ecommerce/product/grayson-crewneck-sweatshirt'}
{'name': 'Ajax Full-Zip Sweatshirt', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh12-green_main.jpg', 'price': 69, 'product_url': 'https://scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt'}
{'name': 'Marco Lightweight Active Hoodie', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh13-blue_main.jpg', 'price': 74, 'product_url': 'https://scrapingcourse.com/ecommerce/product/marco-lightweight-active-hoodie'}
{'name': 'Beaumont Summit Kit', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj01-yellow_main.jpg', 'price': 42, 'product_url': 'https://scrapingcourse.com/ecommerce/product/beaumont-summit-kit'}
{'name': 'Hyperion Elements Jacket', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj02-green_main.jpg', 'price': 51, 'prod://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj08-gray_main.jpg', 'price': 99, 'product_url': 'https://scrapingcourse.com/ecommerce/product/lando-gym-jacket'}
{'name': 'Taurus Elements Shell', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj09-yellow_main.jpg', 'price': 65, 'product_url': 'https://scrapingcourse.com/ecommerce/product/taurus-elements-shell'}
{'name': 'Mars HeatTech™ Pullover', 'image_link': 'https://scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mj10-red_main.jpg', 'price': 66, 'product_url': 'https://scrapingcourse.com/ecommerce/product/mars-heattech™-pullover'}

Total number of products: 57
Total sum of product prices: $3492

Check out the complete codebase on GitHub.

Conclusion

In this tutorial, you learned how to scrape products from a webpage with infinite scrolling using the "Load More" button. By following the outlined steps, you can extract valuable product information and enhance your scraping techniques using ZenRows.
To know more about how you can use Zenrow web scraping tools, check out the following articles on our blog.

• How to parse HTML with PHP
• How to use Hrequests for Web Scraping
• How to use playwright in Ruby Here’s a quick video on a no-code approach to using Zenrows Web scraping tools.