Tricking the Browser

Hello everyone, I've been working on a networking project for a few weeks now. It's going to include a built in proxy to be able to hide your activity; Only so that all your security professional tools can be used with some anonymity. Today I wanted to go over something that will accompany our proxy. When making requests to websites we go through steps associated with the HTTP protocol.

The steps included in this process are numerous and deserve a full post on their own. For now you can refer to this site for the steps. In the step after we've resolved the IP address to the hostname of the request and we are sending our first request to the remote server we send along detailed information about who we are. It includes our IP address, what DNS server we pinged and our user agent. Our user agent is simply a string sent to the server detailing the OS we are using, the version number, and the webkit being used. For example Mozilla for Firefox users or Apple Webkit for safari.

The goal for this script is to insert custom user agents into the header information as a way to fool the remote server about who we really are. This is a good way to mask your identity and can be useful in penetration testing or testing networking tools. Often times websites will employ server side software to track IP address and user agents so that any odd traffic from certain remote nodes (for example web crawling) can be banned.

This is proof of concept only and would need to be integrated into a tool.

Below is the code I created:

import random
import requests

user_agents = []
user_agents_file = open("user_agents.txt", "r") # Text file of user agents

for agents in user_agents_file:
    # cycles through the list adds them to the user_agents array

    user_agents.append(agents)
    random_agents = random.choice(user_agents)[1:-2] # Takes out parens and new line character, also randomizes array choice

url = "http://quotes.toscrape.com"
headers = {"user_agents" : random_agents}

r = requests.get(url, headers=headers)

print r.content

Lets break it down a little. We import our requests and random modules and we go to work. We create an array called user_agents. We have an associated file with this script which has plain text of over 900 different user agents that would come up in the HTTP header. The next for loop takes our empty array, populates it, goes through that array and chooses one at random to insert into our header, and remove the quotes from the string. Then we have our URL which we will send a request to. Using the Requests module we can actually insert info into our request header. So the line r = requests.get(url, headers=headers) gives our request and inserts the header information. It's that easy.