HTTP 429 Too Many Requests means slow down, not go away. The server is telling you it still wants your traffic at a lower rate, and it usually tells you exactly how long to wait. So the fix is almost never a proxy or a new user agent. It is reading one header, sleeping properly, and capping how many requests you have in flight at once. This post covers all three, then shows you how to tell a real rate limit apart from a bot block wearing the same status code, because the two need opposite responses.<\/p>\n
Most 429 handling is written blind, which is why it does not work. Look at the actual response first. The status code arrives with headers that tell you what the limit is and when it resets, and different services use different ones.<\/p>\n
# No install needed. Dump headers, throw the body away.\ncurl -sS -o \/dev\/null -D - "https:\/\/example.com\/api\/items?page=2"\n\n# Look for these, in this order of usefulness:\n# Retry-After: 30 seconds, or an HTTP date\n# RateLimit-Reset: 1724500000\n# X-RateLimit-Remaining: 0\n# RateLimit-Limit: 100<\/pre>\n<\/div>\nThe one that matters is Retry-After. It is defined for exactly this situation and it comes in two forms: a number of seconds to wait, or an absolute HTTP date. Both are legal, both appear in the wild, and code that assumes the number will break on the sites that send the date. MDN documents both forms<\/a>, and its reference page for the 429 status code<\/a> is worth two minutes of your time as well.<\/p>\n
Parse it defensively, honour it when it is there, and fall back to your own schedule when it is not:<\/p>\n
\n# pip install requests\nfrom email.utils import parsedate_to_datetime\nfrom datetime import datetime, timezone\n\ndef retry_delay(response, fallback):\n """Seconds to wait, from Retry-After if the server sent one."""\n raw = response.headers.get("Retry-After")\n if not raw:\n return fallback\n try:\n return max(0.0, float(raw)) # the delay-seconds form\n except ValueError:\n pass\n try:\n when = parsedate_to_datetime(raw) # the HTTP-date form\n return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())\n except (TypeError, ValueError):\n return fallback<\/pre>\n<\/div>\nCap whatever comes back. A server that asks for 3600 seconds is telling you to stop for the hour, and a worker that obediently sleeps that long inside a request handler will look like a hang to everything above it. Take the smaller of the header value and your own ceiling, then decide separately whether to shelve the job.<\/p>\n
Step 2: back off exponentially, with jitter<\/h2>\n
When there is no Retry-After to follow, double the wait each time and add randomness. The doubling is what stops you hammering a service that is already struggling. The randomness is what stops twenty of your own workers, which all hit the limit in the same second, from retrying in the same second forever.<\/p>\n
\n# pip install requests\nimport random, time, requests\n\ndef get_with_backoff(url, attempts=5, base=1.0, ceiling=60.0):\n for attempt in range(attempts):\n response = requests.get(url, timeout=30)\n if response.status_code != 429:\n return response\n # Full jitter: sleep somewhere in [0, base * 2 ** attempt].\n window = min(ceiling, base * (2 ** attempt))\n delay = retry_delay(response, random.uniform(0, window))\n time.sleep(min(delay, ceiling))\n raise RuntimeError(f"still rate limited after {attempts} attempts")\n\n# Waits land near 0-1s, 0-2s, 0-4s, 0-8s, 0-16s unless the\n# server named a delay, in which case that wins.<\/pre>\n<\/div>\nFull jitter, meaning a random value between zero and the window rather than the window plus a small wobble, is the variant that de-synchronises a fleet fastest. If you would rather not write it yourself, urllib3 has this built in, but it needs two arguments set explicitly to be useful:<\/p>\n
\n# pip install requests\nimport requests\nfrom requests.adapters import HTTPAdapter\nfrom urllib3.util import Retry\n\nretry = Retry(\n total=5,\n # status_forcelist defaults to none, so 429 is NOT retried\n # unless you list it here yourself. This is the usual bug.\n status_forcelist=[429, 500, 502, 503, 504],\n backoff_factor=1, # 1 * 2 ** previous_retries seconds\n backoff_jitter=1.0, # urllib3 2.x only\n allowed_methods=["GET", "HEAD"],\n)\n\nsession = requests.Session()\nsession.mount("https:\/\/", HTTPAdapter(max_retries=retry))<\/pre>\n<\/div>\nTwo details are worth knowing about that class. It honours Retry-After for you already, because 429 is one of the three status codes in its RETRY_AFTER_STATUS_CODES set alongside 413 and 503, and respect_retry_after_header defaults to true. But status_forcelist defaults to nothing at all, so a fresh Retry object does not retry a 429 until you name it. People assume the opposite and then wonder why the adapter did nothing.<\/p>\n
Step 3: cap concurrency instead of retrying harder<\/h2>\n
Retry logic treats the symptom. If you are getting 429s steadily rather than in bursts, you are simply asking for more than you are allowed, and the fix is to send less. A semaphore plus a floor on the gap between requests fixes more rate limiting than any backoff curve.<\/p>\n
\n# Standard library only.\nimport asyncio\n\n# Six in flight is a sane starting point for an unknown API.\ngate = asyncio.Semaphore(6)\nMIN_GAP = 0.2 # seconds between starts, per worker\n\nasync def fetch(client, url):\n async with gate:\n response = await client.get(url)\n await asyncio.sleep(MIN_GAP)\n return response\n\n# Tune down on the first 429, and stay there for a while.\n# Tuning back up too eagerly just rediscovers the limit.<\/pre>\n<\/div>\nThe instinct after a 429 is to spread the same load across more IPs. That works for some targets and it is a different decision with its own tradeoffs, which we went through separately in the guide to CAPTCHA proxy rotation<\/a>. Do it as a capacity choice, not as a way to avoid reading a header.<\/p>\n
A 429 is not a 403, and neither is a challenge<\/h2>\n
Here is where 429 handling goes wrong most expensively. Rate limiting and bot detection are different systems that sometimes share a status code, and they want opposite things from you. An HTTP 429 Too Many Requests from a rate limiter is a scheduling instruction, while the same code from an anti-bot edge is a refusal. Backing off politely against a bot block wastes an hour. Retrying hard against a real rate limit gets your IP banned.<\/p>\n