Proxies for marketplace price monitoring: how many addresses a catalogue needs

What you are actually requesting

Price and availability usually arrive not in the page HTML but in a separate JSON response — the same one the storefront itself uses. That changes the sizing noticeably: those responses are lighter and faster, so each card costs less time and less traffic, and the achievable request rate at the same pace is higher.

Sizing the volume

The starting point is not catalogue size but the window you need to cover it in. Ten thousand items once an hour and the same ten thousand every five minutes differ twelvefold in requirements.

requests per hour = items × refreshes per hour

10,000 items, hourly        -> 10,000 req/hour  ≈ 2.8 per second
10,000 items, every 15 min  -> 40,000 req/hour  ≈ 11 per second
10,000 items, every 5 min   -> 120,000 req/hour ≈ 33 per second

Thread count follows from the rate: multiply requests per second by average response time. At 11 requests per second and a 0.7 second response you need roughly 8 concurrent connections on average, and with headroom for retries and slow responses, 50 to 100.

The ceiling is set by per-address frequency

This is the key difference from ordinary scraping. Limits are applied per address rather than per account, so raising thread count without growing the address pool merely earns refusals faster. A rotating pool solves this by spreading the same rate across many addresses, leaving each with a small share.

Which rotation mode to pick

• Walking a catalogue by SKU list — per-request rotation: the calls are independent, and wider distribution is better.
• A card served only inside a session — timed rotation: a cookie is issued first, then the content opens against it.
• Mixed collection — two plans with different modes cost less than one compromise.

Limit the rate, not just the threads

The typical mistake is setting 50 threads and letting go. The pool fires a burst: fifty requests leave at once, hit the rate limit, and everything after that is refusals. You need two limiters: how many connections at once, and how many requests per second.

import threading, requests
from concurrent.futures import ThreadPoolExecutor

THREADS = 50          # no higher than your plan's thread count
RATE_PER_SECOND = 3   # requests per second across the whole pool, tuned by trial

gate = threading.Semaphore(THREADS)
ticket = threading.Semaphore(RATE_PER_SECOND)

def fetch(sku, session):
    # Two limiters: how many connections at once, and how often at all.
    # Without the second, 50 threads fire a burst and earn 429s
    # even when the plan allows the concurrency.
    with gate, ticket:
        return session.get(f"https://example.com/api/card/{sku}", timeout=(5, 30))

session = requests.Session()
with ThreadPoolExecutor(max_workers=THREADS) as pool:
    for sku in range(1000):
        pool.submit(fetch, sku, session)

What to do about refusals

The order is counter-intuitive: with persistent refusals, lower the rate rather than adding threads. More threads at the same per-address frequency means more refusals in the same time. Once collection runs clean at a lower pace, you can carefully widen the address pool — that raises total throughput without raising the load on any single address.

On traffic specifically: rotating plans do not meter it, so catalogue size does not affect the price. You pay for concurrency rather than gigabytes, which suits price monitoring because the volume here is predictably large.

Recognising that the pace is already too high

Refusals are a late signal: by the time they start, some data is already lost. Earlier signs exist, and all of them are visible in your own logs without asking the platform.

• Average response time rises at the same thread count — you are hitting the ceiling before any refusals.
• The average number of attempts per success climbs, even though data keeps arriving.
• Empty or partial responses appear instead of errors — easy to mistake for an out-of-stock item.
• Refusals arrive in bursts at regular intervals — the threads have synchronised.

What counts as success

For price monitoring the metric that matters is data freshness, not traversal speed, and those are different things. A catalogue walked in ten minutes with a quarter of items missed is worse than one walked in an hour in full.

The practical consequence for sizing: plan the pace from the minimum that fits your refresh window, not the maximum the pool can sustain. The gap between those two numbers is your headroom for retries and slow responses.

• Measure the share of items successfully refreshed within the window, not requests per second.
• Track items that refreshed on no pass at all — those are what spoil reports.
• A slow complete pass beats a fast one with gaps.