How many threads a scraper needs: the practical maths

What a thread is

A thread is one simultaneously open connection to the proxy. A 50-thread plan means no more than fifty connections are active at any moment. How many requests pass through them in a day is unlimited: fifty threads can deliver a thousand requests or a million.

The formula

The thread count you need follows from three numbers: how many requests to make, over what period, and how long one response takes on average.

threads = requests × response_time / deadline

Example: 10,000 cards, 0.7 s response, one hour to finish.
10,000 × 0.7 / 3600 ≈ 2 threads on average.

That looks surprisingly small, and it should: across an hour the load is spread thin. But the average is not what you should size against.

Why you add headroom

The average ignores three things that consume capacity in practice.

• Retries: some requests fail and go round again, taking extra connections.
• Response-time spread: slow pages hold a thread many times longer than the average.
• Burstiness: collection rarely flows evenly, with peaks at the start and end.

A practical multiplier is five to ten times the calculated average. For the example above that is 10–20 threads, and the 50-thread Starter-1 plan covers it with plenty to spare.

When you really do need many threads

The number grows from compressing the deadline, not from the catalogue. The same 10,000 cards in five minutes rather than an hour means about 25 threads on average and 150–250 with headroom. A twelvefold speed-up needs twelvefold the connections.

10,000 cards, 0.7 s response:
  in 1 hour  → ~2 average   → 50-thread plan
  in 10 min  → ~12 average  → 100–200 threads
  in 5 min   → ~25 average  → 300–500 threads

How to tell you have hit the ceiling

If raising the thread count in your software does not speed collection up, the bottleneck is not the plan. Check the target's response time and your retry rate — usually those, not the connection limit, set the pace.

The opposite sign is connections consistently waiting in queue while bandwidth use stays low. That is an exhausted thread limit, and the plan is worth raising.

Why your scraper's thread count is not your connection count

This is the main source of divergence between your settings and what the proxy sees. A connection with keep-alive enabled serves many requests in sequence, so ten worker threads may hold fewer than ten sockets. But the reverse also happens, and the second case is the unpleasant one.

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount("https://", HTTPAdapter(
    pool_maxsize=50,     # no lower than your worker count
    pool_block=True,     # extra threads wait instead of opening sockets past the pool
))

# What happens with pool_block=False (the default):
# connections beyond pool_maxsize are created on demand, but they are NOT
# retained after use. So 50 threads against a pool of 10 cause constant
# rebuilding — and the proxy sees not 10 sessions
# but a stream of new connections.

The point is that the connection pool and the thread count are independent settings, and when the pool is smaller, each extra connection is created fresh per request and closed afterwards. To the proxy that looks like many short sessions instead of a few long ones, and the limit drains faster than you planned.

• Pool smaller than thread count — constant connection rebuilding and wasted limit.
• HTTP/2 multiplexes many requests onto one socket: fewer connections than requests.
• Redirects and TLS handshakes add connections that your arithmetic does not include.
• A retry after a reset opens a new socket rather than reusing the old one.

Scrapy counts this differently

Check the settings before doing the arithmetic here. The overall ceiling is one value, but a per-domain limit also applies, and from some versions the new-project template writes fairly cautious defaults — one request per domain and a one-second delay.