Scrapfly

Posted on Dec 3 • Originally published at scrapfly.io on Dec 3

What is HTTP 407 Status Code and How to Fix it

#http

Imagine trying to access a website, only to be stopped in your tracks by a frustrating error code. If you've encountered HTTP 407, you know how disruptive it can be, especially when working with proxies that demand seamless connectivity.

In this article, We’ll explore the HTTP 407 Proxy Authentication Required error, its causes, and how proxies trigger it. Learn about authentication methods, troubleshooting steps, and how Scrapfly's Proxy Saver simplifies proxy management.

What is HTTP Error 407?

The HTTP 407 error, also known as 407 Proxy Authentication Required, occurs when a proxy server requires authentication before granting access to the requested resource. Acting as a gatekeeper, the proxy server ensures that only authorized users can proceed, blocking any requests without valid credentials.

Key Characteristics of HTTP 407

HTTP Status Code : 407 — indicating that proxy authentication is required.
Response Header : Proxy-Authenticate — sent by the proxy to specify the authentication method it supports (e.g., Basic, Digest).
Request Header : Proxy-Authorization — sent by the client, containing the required credentials to authenticate with the proxy.

Example 407 Response :

HTTP/1.1 407 Proxy Authentication Required
Proxy-Authenticate: Basic realm="example"

How the Process Works

The HTTP 407 generally means that the request never actually reached the server and was stopped by proxy authentication. The process unfolds in three steps:

Initial request : The client sends a request to the server via the proxy.
Authentication demand : The proxy intercepts the request, responds with the HTTP 407 error, and specifies the required authentication method in the Proxy-Authenticate header.
Retry with correct credentials : The client resends the request, this time including the appropriate credentials in the Proxy-Authorization header.

So, 407 isn't a critical error just means that client configuration is not correct and need to be fixed.

What is a Proxy?

A proxy serves as a bridge between a client (like your browser or application) and the internet. Proxies are widely used for:

Privacy : They mask the client’s IP address, ensuring anonymity.
Performance : Proxies cache frequently accessed resources, improving load times.
Security and Compliance : Proxies filter and control internet usage, aligning with organizational policies.

For more on proxies see our proxy introduction guide.

What Are HTTP 407 Error Causes?

The HTTP error 407 is closely tied to issues with proxy authentication which means the client is misconfigured. To understand better let's take a look at the common causes of HTTP 407 errors and possible authentication methods.

Common Causes of HTTP 407 Errors

Several factors can lead to an HTTP 407 Proxy Authentication Required error. Here are the most common ones:

Cause	Description	Example
Missing Authentication	The `Proxy-Authorization` header is absent, meaning no credentials were provided for the request.	A client tries to connect to a proxy without supplying a username and password.
Invalid Credentials	Incorrect or expired credentials prevent successful authentication.	Providing the wrong username or password, or using an expired API key/token.
Unescaped Characters in Credentials	Special characters in usernames or passwords cause issues if not properly encoded.	A password like `pa$$word@123` fails because `@` and `$` are not escaped.
Misconfigured Proxy Settings	Incorrect proxy configuration, such as wrong URLs, ports, or protocols.	Configuring a proxy with `http://` instead of `https://` for secure requests.
Overloaded or Misconfigured Proxy Server	An overwhelmed or improperly configured proxy rejects even valid requests.	A proxy server handling too many requests simultaneously, leading to authentication failures.

The unescaped characters example is by far the most common cause of HTTP 407 errors and if you're unsure about the credentials you're using, it's always a good idea to double-check them.

To better understand how proxy authorization works take a look at this Python proxy request example that breaks down the most common proxy authentication method, Basic Authentication:

# manual way of sending proxy details for proxy
import httpx

# Define the proxy details
proxy_url = "http://proxy.example.com:8080"
proxy_username = "your_username"
proxy_password = "your_password"

# Create the basic authentication header
proxy_auth = f"{proxy_username}:{proxy_password}"
encoded_auth = httpx.utils.to_b64(proxy_auth).decode("utf-8")
proxy_headers = {
    "Proxy-Authorization": f"Basic {encoded_auth}"
}

client = httpx.Client(proxies={"http://": proxy_url, "https://": proxy_url})
response = client.get("https://httpbin.dev/headers", headers=proxy_headers)

# Print the response
print(response.json())

By identifying and addressing these causes, you can effectively resolve or prevent HTTP code 407 errors, ensuring smoother proxy operations.

Proxy Authentication Methods

To understand the HTTP 407 error better let's take a look at the most common proxy authentication methods and how they work.

Basic Authentication (most common)

The username and password are concatenated (username:password), then Base64 encoded.

For Example here's how basic proxy authentication works in Python and httpx

import httpx

proxy_url = "http://proxy.example.com:8080"
username = "your_username"
password = "your_password"

# Encode credentials to Base64
auth_header = f"{username}:{password}"
encoded_auth = httpx.utils.to_b64(auth_header).decode("utf-8")

# Headers for Proxy Authentication
proxy_headers = {"Proxy-Authorization": f"Basic {encoded_auth}"}

client = httpx.Client(proxies={"http://": proxy_url, "https://": proxy_url})

# Making a request with the Proxy-Authorization header
response = client.get("https://httpbin.org/get", headers=proxy_headers)
print(response.json())

Digest Authentication

Encrypts credentials using a cryptographic hash, offering better security than Basic. For example, here's how Digest authentication works in Python:

import httpx

proxy_url = "http://proxy.example.com:8080"
username = "your_username"
password = "your_password"

# Setting up Digest Authentication
auth = httpx.DigestAuth(username, password)

client = httpx.Client(proxies={"http://": proxy_url, "https://": proxy_url}, auth=auth)

# Making a request through the proxy
response = client.get("https://httpbin.org/get")
print(response.json())

NTLM/Kerberos Authentication

Used in Windows environments for single sign-on. For example, here's how NTLM authentication works in Python:

import httpx
from requests_ntlm import HttpNtlmAuth

proxy_url = "http://proxy.example.com:8080"
username = "DOMAIN\\your_username" # For NTLM, specify the domain
password = "your_password"

# NTLM Authentication
auth = HttpNtlmAuth(username, password)

# Wrap the NTLM session in httpx
with httpx.Client(proxies={"http://": proxy_url, "https://": proxy_url}, transport=httpx.HTTPTransport(auth=auth)) as client:
    response = client.get("https://httpbin.org/get")
    print(response.json())

OAuth Authentication

Token-based authentication for secure and scalable proxy access. For example, here's how OAuth authentication works in Python:

import httpx

proxy_url = "http://proxy.example.com:8080"
oauth_token = "your_oauth_token"

# Proxy-Authorization header with Bearer token
proxy_headers = {"Proxy-Authorization": f"Bearer {oauth_token}"}

client = httpx.Client(proxies={"http://": proxy_url, "https://": proxy_url})

# Making a request with OAuth Proxy-Authorization header
response = client.get("https://httpbin.org/get", headers=proxy_headers)
print(response.json())

The authentication method entirely depends on your proxy providers but understanding the authentication delivery method can help you find where authentication is failing.

Common Authentication Errors

To quickly summarize, these are the most common client side pitfalls that can cause HTTP 407 errors:

Improper Encoding : Special characters in credentials (like %, #, &) not encoded correctly.
Protocol Mismatch : The client uses HTTP, but the proxy expects HTTPS (or vice versa).
Timeouts : Delays in responding to authentication requests.
Unsupported Auth Methods : The proxy server supports one method (e.g., Basic), but the client attempts another (e.g., Digest).

Improper encoding being by far the most common cause of HTTP 407 errors and note that some characters like @ and $ need to be escaped before sending them to the proxy which is especially common in passwords!

Power-Up Your Proxies with Scrapfly

Proxies are powerful tools, but they come with challenges like 407 Proxy Authentication Required. Scrapfly’s Proxy Saver simplifies the process of managing proxies and resolving authentication errors.

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

You can learn more about how use proxies for web scraping in our dedicated article:

(https://scrapfly.io/blog/introduction-to-proxies-in-web-scraping/)

FAQ

To wrap up this guide, here are answers to some frequently asked questions about Error http 407 code.

What is the main difference between HTTP 401 and HTTP 407?

HTTP 401 means the server itself requires authentication from the client where's HTTP 407 means the proxy server requires authentication before forwarding the request to the server.

What causes the error message "Received HTTP code 407 from proxy after connect"?

This message occurs when a client tries to connect through a proxy but fails to provide valid credentials, or the provided credentials are incorrect, incomplete, or improperly formatted.

Can HTTP 407 errors be prevented?

Yes. Preventive steps include:

Using proxies with built-in authentication handling.
Validating credentials before requests.
Automating proxy management with tools like Scrapfly.

Summary

The HTTP 407 Proxy Authentication Required error stems from issues like missing credentials, misconfigured proxies, or improper authentication. Understanding its causes and using tools like Scrapfly’s Proxy Saver can help resolve and prevent these errors.

With the right tools and knowledge, you can manage proxies smoothly and eliminate HTTP 407 errors effortlessly!

DEV Community