Error 429 while scraping: how to size pauses and retries instead of guessing

Retry-After is the only honest signal

Along with a 429 the server may send a Retry-After header, either in seconds or as a date. When it is present there is nothing to guess — it states how long to wait. Ignoring it is pointless: retrying early almost always earns the same 429 and extends the penalty.

The good news is that the standard retry mechanism for requests respects Retry-After by default, and applies it to statuses 413, 429 and 503. There is nothing extra to configure.

What to retry and what is pointless

Only statuses that can change on their own are worth retrying: 408, 429, 500, 502, 503, 504, plus 522 and 524 for sites behind Cloudflare. Other 4xx codes are pointless — the same request returns the same answer.

• 400, 404, 405, 410, 422 — the request itself is wrong; a retry will not fix it.
• 401, 403, 407 — a question of credentials or address, not of pace.
• 429 — retry, but with a growing pause.

Exponential backoff and why jitter matters

The standard scheme doubles the pause with each attempt. One implementation detail surprises many people: the first retry has no pause at all, and the multiplier only starts from the second. With a factor of 0.5 the sequence is 0s, 0.5s, 1s, 2s, 4s.

Jitter is a random addition to the pause. Without it every thread that got a 429 at the same moment counts the same delay and strikes again simultaneously — as a burst. Jitter spreads them out and turns the burst into a stream.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=5,
    backoff_factor=0.5,          # 0s, 0.5, 1, 2, 4 ... the first retry has no pause
    backoff_jitter=0.3,          # requires urllib3 >= 2.0
    backoff_max=30,              # pause ceiling, 120s by default
    status_forcelist=(429, 500, 502, 503, 504),
    respect_retry_after_header=True,   # on by default
    raise_on_status=False,       # return the response instead of raising
)

session = requests.Session()
# pool_maxsize must be no lower than your thread count, or connections get rebuilt
session.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=32, pool_block=True))

r = session.get("https://example.com", timeout=(5, 30),
                proxies={"https": "http://LOGIN:PASSWORD@GATEWAY:PORT"})

Backoff works differently in Scrapy

This is easy to get wrong by expecting familiar behaviour. Scrapy's built-in retry mechanism is enabled by default, but it has no exponential backoff: a retried request simply returns to the scheduler and goes out in the general queue. Other settings set the pace.

# settings.py
# RetryMiddleware is enabled by default, but it has no exponential backoff:
# a retried request simply returns to the scheduler. These settings set the pace.
RETRY_TIMES = 4
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]

DOWNLOAD_DELAY = 1
RANDOMIZE_DOWNLOAD_DELAY = True   # 0.5-1.5 multiplier on the delay

AUTOTHROTTLE_ENABLED = True       # adaptive pause based on server response time
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

The practical takeaway: in Scrapy, do not look for an exponential pause setting — enable adaptive throttling instead, which adjusts the delay to the server's response time. That beats fixed values, because load on the target changes over the day.

What to change first

With persistent 429s the order is: pause first, then thread count, and only then the size of the address pool. The reason is that 429 is counted per address — adding threads without lowering the rate earns the same 429s, only faster.

And the reverse consideration, which matters for budgeting: on rotating plans traffic is not metered and billing is by concurrent connections. A retried request therefore costs nothing extra — asking again is cheaper than losing data and collecting it over.

What not to retry, even with a delay

Turn the previous section's list around: knowing what is pointless to retry matters more, because that is where time goes. The rule is simple — retry what can change on its own.

• An error in the request itself — a retry returns the same thing no matter how long you wait.
• A credentials or address question — fixed by the login pair or the whitelist, not by a pause.
• A missing page — it will not appear.

A note on request methods. By default the retry mechanism in requests does not repeat data submissions: repeating one could create a duplicate on the server. If you need those retried too, enable it deliberately and only where resubmission is safe.

Where to look at what is happening

Persistent refusals are easier to diagnose from the distribution of codes than from individual errors. It helps to log not just that a refusal happened but which attempt finally succeeded.

• The refusal rate grows with thread count — you hit the frequency limit; lower the pace.
• Refusals arrive in bursts at regular intervals — threads have synchronised; you need more jitter.
• Refusals do not track the pace at all — frequency is not the cause; check authentication and headers.