How to Fix HTTP 429 Too Many Requests While Scraping

HTTP 429 Too Many Requests means slow down, not go away. The server is telling you it still wants your traffic at a lower rate, and it usually tells you exactly how long to wait. So the fix is almost never a proxy or a new user agent. It is reading one header, sleeping properly, and capping how many requests you have in flight at once. This post covers all three, then shows you how to tell a real rate limit apart from a bot block wearing the same status code, because the two need opposite responses.

What you need

• Python 3.10 or newer, with the requests library, for the examples. The logic ports directly to any HTTP client.
• A terminal, so you can look at response headers before you write any retry code.
• CapSkip running for the last section only, either in Local mode on the loopback address or in Server mode on a machine your workers can reach. Both are covered under connection settings, so pick one before you start.

Step 1: read the response before you retry it

Most 429 handling is written blind, which is why it does not work. Look at the actual response first. The status code arrives with headers that tell you what the limit is and when it resets, and different services use different ones.

# No install needed. Dump headers, throw the body away.
curl -sS -o /dev/null -D - "https://example.com/api/items?page=2"

# Look for these, in this order of usefulness:
#   Retry-After: 30           seconds, or an HTTP date
#   RateLimit-Reset: 1724500000
#   X-RateLimit-Remaining: 0
#   RateLimit-Limit: 100

The one that matters is Retry-After. It is defined for exactly this situation and it comes in two forms: a number of seconds to wait, or an absolute HTTP date. Both are legal, both appear in the wild, and code that assumes the number will break on the sites that send the date. MDN documents both forms, and its reference page for the 429 status code is worth two minutes of your time as well.

Parse it defensively, honour it when it is there, and fall back to your own schedule when it is not:

# pip install requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_delay(response, fallback):
    """Seconds to wait, from Retry-After if the server sent one."""
    raw = response.headers.get("Retry-After")
    if not raw:
        return fallback
    try:
        return max(0.0, float(raw))          # the delay-seconds form
    except ValueError:
        pass
    try:
        when = parsedate_to_datetime(raw)    # the HTTP-date form
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return fallback

Cap whatever comes back. A server that asks for 3600 seconds is telling you to stop for the hour, and a worker that obediently sleeps that long inside a request handler will look like a hang to everything above it. Take the smaller of the header value and your own ceiling, then decide separately whether to shelve the job.

Step 2: back off exponentially, with jitter

When there is no Retry-After to follow, double the wait each time and add randomness. The doubling is what stops you hammering a service that is already struggling. The randomness is what stops twenty of your own workers, which all hit the limit in the same second, from retrying in the same second forever.

# pip install requests
import random, time, requests

def get_with_backoff(url, attempts=5, base=1.0, ceiling=60.0):
    for attempt in range(attempts):
        response = requests.get(url, timeout=30)
        if response.status_code != 429:
            return response
        # Full jitter: sleep somewhere in [0, base * 2 ** attempt].
        window = min(ceiling, base * (2 ** attempt))
        delay = retry_delay(response, random.uniform(0, window))
        time.sleep(min(delay, ceiling))
    raise RuntimeError(f"still rate limited after {attempts} attempts")

# Waits land near 0-1s, 0-2s, 0-4s, 0-8s, 0-16s unless the
# server named a delay, in which case that wins.

Full jitter, meaning a random value between zero and the window rather than the window plus a small wobble, is the variant that de-synchronises a fleet fastest. If you would rather not write it yourself, urllib3 has this built in, but it needs two arguments set explicitly to be useful:

# pip install requests
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=5,
    # status_forcelist defaults to none, so 429 is NOT retried
    # unless you list it here yourself. This is the usual bug.
    status_forcelist=[429, 500, 502, 503, 504],
    backoff_factor=1,        # 1 * 2 ** previous_retries seconds
    backoff_jitter=1.0,      # urllib3 2.x only
    allowed_methods=["GET", "HEAD"],
)

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))

Two details are worth knowing about that class. It honours Retry-After for you already, because 429 is one of the three status codes in its RETRY_AFTER_STATUS_CODES set alongside 413 and 503, and respect_retry_after_header defaults to true. But status_forcelist defaults to nothing at all, so a fresh Retry object does not retry a 429 until you name it. People assume the opposite and then wonder why the adapter did nothing.

Step 3: cap concurrency instead of retrying harder

Retry logic treats the symptom. If you are getting 429s steadily rather than in bursts, you are simply asking for more than you are allowed, and the fix is to send less. A semaphore plus a floor on the gap between requests fixes more rate limiting than any backoff curve.

# Standard library only.
import asyncio

# Six in flight is a sane starting point for an unknown API.
gate = asyncio.Semaphore(6)
MIN_GAP = 0.2          # seconds between starts, per worker

async def fetch(client, url):
    async with gate:
        response = await client.get(url)
        await asyncio.sleep(MIN_GAP)
        return response

# Tune down on the first 429, and stay there for a while.
# Tuning back up too eagerly just rediscovers the limit.

The instinct after a 429 is to spread the same load across more IPs. That works for some targets and it is a different decision with its own tradeoffs, which we went through separately in the guide to CAPTCHA proxy rotation. Do it as a capacity choice, not as a way to avoid reading a header.

A 429 is not a 403, and neither is a challenge

Here is where 429 handling goes wrong most expensively. Rate limiting and bot detection are different systems that sometimes share a status code, and they want opposite things from you. An HTTP 429 Too Many Requests from a rate limiter is a scheduling instruction, while the same code from an anti-bot edge is a refusal. Backing off politely against a bot block wastes an hour. Retrying hard against a real rate limit gets your IP banned.

That last row catches people out, because nothing failed. The request returned 200 and the body is a challenge page rather than your data, so a status-code-only retry loop will happily hammer it forever. Check for the marker you expect in the body, not just the status line. If a challenge is what you found, backing off is not the answer and solving it is.

Keep your solve loop off the same rake

The polling loop that waits for a CAPTCHA answer is itself a retry loop, and hand-rolled ones make exactly the mistakes above. Two things about CapSkip make this easier than it is against a metered service. It runs on your own hardware, so there is no per-solve quota to exhaust and no rate limit of its own to trip. And the SDKs already back off for you: polling starts at a quarter of a second and grows up to pollingInterval, which is a ceiling rather than a fixed gap.

# pip install capskip
from capskip import CapSkip, NetworkException, TimeoutException

# Local mode. In Server mode, host is the solver box's address.
solver = CapSkip(host="127.0.0.1", port=8080, pollingInterval=2)

try:
    result = solver.recaptcha(
        sitekey="YOUR_SITEKEY",
        url="https://example.com/page-with-recaptcha",
    )
    print(result["code"][:24])   # token, submit it with the form
except TimeoutException:
    # Polling ran past recaptchaTimeout, 300 seconds by default.
    print("gave up waiting, try again or lower the timeout")
except NetworkException:
    # The solver is not reachable on that host and port.
    print("check CapSkip is running and the mode you set")

Lowering pollingInterval makes an answer arrive sooner and costs you nothing, which is a choice you do not really have on a billed API. If you are polling the raw HTTP endpoints instead of using an SDK, the recommended delays per CAPTCHA type are listed in the API documentation, along with the pending response you will get while a solve is still in progress.

Running the solver on a server instead

Rate limiting is usually a problem for a fleet rather than one script, and a fleet does not share a loopback address. The connection settings cover both cases:

Point the SDK host at the solver machine and nothing else in your code changes, so ten workers can share one instance. A static public IP is recommended when the callers sit outside your own network. The details are under connection settings, and Server mode is still your hardware and still unmetered: it moves where the solver runs, not who owns it.

FAQ

The shortest version

HTTP 429 Too Many Requests is a scheduling problem, so treat it like one. Read Retry-After and obey it up to a ceiling you choose. Fall back to exponential backoff with full jitter when the header is absent. Then lower your concurrency, because steady 429s are a capacity problem that no retry curve solves. And check the body before you retry at all, since a challenge page arrives with a perfectly healthy status code and wants solving rather than waiting. For that part an unlimited captcha solver running locally is the piece that fits, and the notes on running a CAPTCHA solver for web scraping cover how it slots into a worker pool.