Understanding User-Agent in Puppeteer

Web automation is a continuous cycle. Websites work to block bots, and bots evolve to stay undetected. If you’re using Puppeteer without adjusting your user-agent, you’re making it easier for websites to identify you.
Each browser request includes a user-agent string, a digital identifier that reveals the browser, operating system, and sometimes the device you’re using. Websites use this data to optimize their layouts, serve specific content, and most importantly, detect automated behavior.
In this guide, we’ll explore the difference between random and custom user-agents, when to use each, and how to configure them in Puppeteer.

Random or Custom User-Agent: Which Best Fits Your Needs

Your choice depends on your goals.
Utilize a Random User-Agent When:
You’re scraping data and need to avoid detection.
You want each request to appear as if it’s coming from a different device.
You’re running high-volume automation and don’t want to get blocked.
Utilize a Custom User-Agent When:
You’re testing a web app and need consistent results.
You want to mimic a specific browser or device.
You’re running performance tests that require a stable environment.
Now, let's get hands-on and configure both options in Puppeteer.

Setting Up a Random User-Agent in Puppeteer

To rotate user-agents dynamically, install the user-agents package. Here’s how:

const puppeteer = require('puppeteer');  
const { UserAgent } = require('user-agents');  

(async () => {  
  const browser = await puppeteer.launch();  
  const page = await browser.newPage();  

  const userAgent = new UserAgent({ deviceCategory: 'desktop' }).toString();  
  await page.setUserAgent(userAgent);  

  await page.goto('https://example.com');  
  // Your automation tasks here.  

  await browser.close();  
})();

Step-by-Step Guide:
Install Puppeteer & Dependencies

npm install puppeteer user-agents  

Import Required Packages

const puppeteer = require('puppeteer');  
const { UserAgent } = require('user-agents');  

Generate and Use a Random User-Agent

const userAgent = new UserAgent({ deviceCategory: 'desktop' }).toString();  

Add it to Puppeteer

await page.setUserAgent(userAgent);  

Browse & Automate

await page.goto('https://example.com');  

Setting Up a Custom User-Agent in Puppeteer

Need full control? Set a fixed user-agent string:

const puppeteer = require('puppeteer');  

(async () => {  
  const browser = await puppeteer.launch();  
  const page = await browser.newPage();  

  await page.setUserAgent(  
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'  
  );  

  await page.goto('https://example.com');  
  // Your automation tasks here.  

  await browser.close();  
})();

Steps:
Install Puppeteer

npm install puppeteer  

Import Puppeteer

const puppeteer = require('puppeteer');  

Define a Custom User-Agent

await page.setUserAgent('Your_Custom_User_Agent');  

Avoid These Common Mistakes

Even with proper user-agent handling, issues can arise. Here’s how to fix them:
Getting Blocked by Websites
Websites monitor behavior beyond user-agent strings.
Fix: Rotate IPs using proxies, add delays, and mimic human behavior.
Incorrect User-Agent Format
Some sites reject improperly formatted user-agents.
Fix: Use real user-agent strings from trusted sources.
Rate Limiting & IP Bans
Even with a rotating user-agent, sending too many requests too quickly can get you flagged.
Fix: Space out requests using setTimeout() and respect site rate limits.
Custom User-Agent Not Working
Some sites require specific user-agents to function properly.
Fix: Use a widely recognized user-agent and update it regularly.
API Changes Breaking Your Setup
Libraries for user-agent rotation may become outdated.
Fix: Regularly update dependencies and check for changes.

Final Thoughts

User-agents are not just simple strings—they establish your browser's identity on the web. Rotating user-agents during scraping helps avoid detection, while properly setting them for test automation ensures stability.
Mastering user-agent manipulation in Puppeteer is crucial for smooth automation. Optimize your scripts to ensure reliable performance and prevent complications.