How to use proxies in Python: requests, Scrapy and Selenium

requests: the basic case

A proxy is passed as a dict keyed by scheme. The same address goes under both http and https — this is the proxy's address, not the site's, so the scheme in the value describes how you connect to the proxy itself.

import requests

PROXY = "http://LOGIN:PASSWORD@GATEWAY:PORT"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://example.com", proxies=proxies, timeout=20)
print(r.status_code, r.text[:200])

Login, password, gateway address and port come from the dashboard once a plan is paid for. Always set a timeout: without one a hung request occupies a thread indefinitely, and in a multi-threaded parser that quietly eats your connection limit.

SOCKS5 and the DNS pitfall

If you need SOCKS5, install the extra dependency and watch the h in the scheme. This is the most common invisible mistake in proxy setup.

# SOCKS5 needs an extra dependency:
#   pip install "requests[socks]"

PROXY = "socks5h://LOGIN:PASSWORD@GATEWAY:PORT"
proxies = {"http": PROXY, "https": PROXY}

# socks5h, not socks5: the h makes DNS resolution go through the proxy.
# Without it your own machine resolves the hostname, so your ISP still
# sees which sites you open even though traffic goes through the proxy.

Scrapy: middleware and matching your plan

In Scrapy the proxy is set through request meta, and the mechanism itself is enabled by a built-in middleware. The second thing that matters is CONCURRENT_REQUESTS: it must not exceed your plan's thread count, or the surplus requests will simply queue.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}
CONCURRENT_REQUESTS = 50          # no higher than the plan's thread count
CONCURRENT_REQUESTS_PER_DOMAIN = 50
RETRY_TIMES = 3

# middlewares.py
class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = "http://LOGIN:PASSWORD@GATEWAY:PORT"

Setting CONCURRENT_REQUESTS well above the limit will not break the spider — it will just run slower than expected, and that cause is hard to spot in the logs later.

Selenium: why the password does not work

Browsers are a separate case. Chrome ignores credentials passed in --proxy-server and shows a native authentication dialog instead, which the driver will not fill in.

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=http://GATEWAY:PORT")
driver = webdriver.Chrome(options=options)

# Chrome ignores credentials in --proxy-server and pops a native auth
# dialog instead. For browsers, IP authentication is easier: add the
# machine's address to the whitelist in your dashboard.

There are two ways around it: an extension that supplies the credentials, or IP authentication. The second is simpler and more reliable — whitelist the machine's address and the browser connects without a password. Both authentication methods work at once here, so your scripts can keep using the login.

How to verify it worked

After setup, make two requests in a row and look at the address returned. With per-request rotation the addresses will differ — that is your proof traffic really goes through the pool.

import requests

PROXY = "http://LOGIN:PASSWORD@GATEWAY:PORT"
proxies = {"http": PROXY, "https": PROXY}

# Two consecutive requests should return different addresses
# when per-request rotation is enabled.
for _ in range(2):
    print(requests.get("https://op-proxy.com/api/check",
                       proxies=proxies, timeout=20).text)

• The address does not change — you are probably on timed rotation, not per-request.
• You see your home address — the proxy was not applied; check the scheme keys in the dict.
• Authentication error — check the plan has not expired and that the machine's IP matches the whitelist.

Timeouts: why a single number is a bad idea

The timeout in requests accepts not only a number but a pair of values: how long to wait for the connection and how long for the response. The difference is practical: a connection either establishes in a fraction of a second or not at all, while a response from a loaded page can take tens of seconds.

# Pass the timeout as a tuple: (connect, read).
# A single number applies the same limit to both phases, which is almost never
# what you want: connecting should be fast, while the response may be slow.
r = requests.get(url, proxies=proxies, timeout=(5, 30))

# Why the connect timeout must be small:
# requests and urllib3 do NOT implement Happy Eyeballs. If a host has both A and
# AAAA records, addresses are tried sequentially and the effective connect limit
# doubles — 5 seconds becomes 10 when IPv6 is unreachable.

One subtlety explains the oddly long hangs people see through IPv6 proxies. Libraries such as curl, and browsers, try IPv4 and IPv6 almost simultaneously and give IPv6 a small head start. Requests does not — addresses are tried in turn, so with IPv6 unreachable you wait the full timeout twice.

What to check when it does not work

The order matters: each step eliminates a whole class of causes, and skipping the first sends you hunting for a bug that is not there.

• The same URL with no proxy — if that fails too, the problem is your network or the site.
• The same proxy through curl with -v — this shows whether it admitted you at all, before any request to the site.
• A known-open endpoint through the proxy — separates a proxy refusal from a target refusal.
• The scheme in the dictionary: for an https URL the value describes the protocol to the proxy, not to the site.