Threads or gigabytes: working out which billing unit is cheaper for your job

Two units, and what each one limits

Thread billing limits how many requests can be in flight at once and says nothing about volume. Traffic billing limits volume and says nothing about concurrency. Which gives a simple consequence: the cheaper unit is whichever one your job consumes less of.

Here the count is concurrent connections, and traffic on rotating plans is not metered at all. Residential and mobile addresses on the market are almost always sold by the gigabyte. So the question reduces to your job's profile: many small requests, or few heavy ones.

The two numbers to measure

The first is average response size. There is no need to eyeball it — curl reports it directly. The second is peak concurrency, meaning how many requests are in flight at the same time. People routinely confuse it with requests per second, and they are not the same thing.

# Average response size across a dozen typical pages
for u in $(head -10 urls.txt); do
  curl -s -o /dev/null -w '%{size_download}\n' "$u"
done | awk '{s+=$1} END {print s/NR " bytes on average"}'

# And the average response time — you will need it for threads
for u in $(head -10 urls.txt); do
  curl -s -o /dev/null -w '%{time_total}\n' "$u"
done | awk '{s+=$1} END {print s/NR " s on average"}'

How that turns into threads

Concurrency is request rate multiplied by response time. Thirty requests a second against a response of just over a second means roughly three dozen requests open at once, and that is the number your plan has to accommodate. Counting from requests per day does not work: a million requests spread across a day needs a handful of threads, while the same million in an hour needs dozens.

If you already have logs from previous runs, they give concurrency more precisely than any formula: take the start and end timestamps of each request and find the largest number of intervals overlapping at any one point. The rate-times-duration formula gives an average, while a plan has to cover the peak — and with uneven load the peak can be twice the average.

# What to measure rather than guess
Average response size        4 KB      <- curl -w '%{size_download}'
Requests per day             500,000
                            ---------
Traffic per day              ~2 GB
Traffic per month            ~60 GB

# Peak concurrency — how many requests are in flight at once
Requests per second          30
Average response time        1.2 s
                            ---------
Threads needed               ~36       <- 30 x 1.2, Little's law
                            
# Plan: 50 threads covers 36 with headroom.
# Traffic on rotating plans is not metered, so those 60 GB do not count.

Where gigabytes win

Symmetrically and honestly: if responses are few but heavy, thread billing loses. Downloading files, pulling media, a dozen parallel transfers of a hundred megabytes each — concurrency there is tiny while volume is enormous, and paying for threads is poor value. Our product is not the best choice for that, and it is easier to say so up front than to work it out through a refund.

A rule of thumb for the estimate: a kilobyte across a million requests is roughly a gigabyte. If your profile is hundreds of thousands of responses a few kilobytes each, the volume is trivial and concurrency is the real constraint. If it is the other way round, count by volume.

How to tell you are paying for idle capacity

The inverse problem is just as common. The sign of idle capacity is a plan noticeably larger than your computed concurrency while the job does not speed up when you add threads in the program itself. That means the limit is not the proxy: the bottleneck is the target's latency, your parsing, or your own rate limiter.

A separate article covers where programs spend threads invisibly — A-Parser's built-in proxy checker, for instance, takes fifteen by default. Before shrinking a plan, check whether something auxiliary is eating them.

What this arithmetic does not include

• Competitors' prices: they change more often than an article gets edited, and quoting them would mislead within a month.
• Any promise that thread billing is always cheaper. It is cheaper at high concurrency with small responses — that is a condition, not a property.
• Retries. They spend both threads and traffic; with an aggressive retry policy both numbers need multiplying, and how to compute the pauses is covered in the article on 429.

The practical upshot: measure two numbers on your own job, compute concurrency and compare it against a plan. Ten minutes of measuring saves a month of paying for the wrong product — in either direction.