When a Python scraper gets a bad response, the first question is CAPTCHA or block, because the answer decides what to do next. A CAPTCHA can be solved. A rate limit needs a wait. A block page means your address or fingerprint is refused, and no token will change that. This guide builds a classifier of about 40 lines that reads the status, the headers and the markup and returns one of six labels, then a fetch loop that calls CapSkip only for the label where a solve can help. CapSkip runs on your own Windows PC with no per-solve fee, so skipping a solve that cannot work saves time, not money.<\/p>\n
How this was tested: the classifier and the fetch loop passed 17 checks on Python 3.14.0, each one a response served by a local fixture site. The solves went through the real capskip package to a mock CapSkip, not the app. Two public pages behind Cloudflare’s full-page check were also fetched on 29 October 2026, and both returned a 403 with a cf-mitigated header. No number below is a solve time.<\/p>\n
The two mistakes cost different things. Sending a block page to a solver wastes the solve and leaves you blocked. Rotating to a new proxy every time a CAPTCHA appears burns addresses you did not need to burn. A page that serves a CAPTCHA is usually asking you to prove something, and a solved token lets you through. A page that serves a block is refusing you outright. The HTTP 429 guide<\/a> covers the waiting side of this for one status code. This guide covers the decision that comes before it.<\/p>\n Six labels cover it: ok, rate_limited, captcha, cloudflare_challenge, blocked and unknown. The unknown label matters. A classifier forced to choose among the other five will sometimes choose wrong, and a loop that raises on unknown shows you the case you have not handled yet.<\/p>\n Create a virtual environment in an empty folder and install the two packages. The commands call the environment’s own Python by path, so you do not need to activate it.<\/p>\n Save this block and the next one as classify.py. The first block holds the patterns. The widget pattern looks for a tag whose class attribute names a widget, or an altcha-widget tag. A blog post that shows the markup inside escaped code, a link whose address contains a widget name, and a hidden g-recaptcha-response input all fail to match.<\/p>\n The last line holds phrases that identify a soft block, a page that returns a normal status but says you are refused. Be careful with it. Cloudflare’s error documentation lists Error 1020: Access denied, which is where the first two phrases come from. The other two are guesses from block pages in the wild, and your target will have its own wording. Replace them after you see a few real ones.<\/p>\n The order of the checks is the logic. Each rule only runs if the ones above it did not match.<\/p>\n Save this block, the next one and the one after that as policy.py. A 429 may carry a Retry-After header, and the header may hold seconds or an HTTP date. MDN lists both forms, and it says the header is optional on a 429. The helper handles both and falls back to a number you choose when the header is missing or unreadable.<\/p>\n This is the only place CapSkip is called. It reads the sitekey and the form action from the page, asks CapSkip for a token and posts it back in the g-recaptcha-response field. It handles reCAPTCHA v2 only, on purpose, and it posts that one field and nothing else, so a form with other required fields needs them added to the data. Every widget in the list gets the captcha label, so a Turnstile, hCaptcha, Friendly Captcha, CaptchaFox or ALTCHA page reaches this function too. It stops those with a RuntimeError that says to run the CAPTCHA type detector<\/a> first. Sending a Turnstile key to the reCAPTCHA method instead would end in ERROR_GOOGLEKEY: CapSkip refuses a key that does not look like a reCAPTCHA key, which I confirmed on a mock that copies the app’s key check, not on the app. For the reCAPTCHA call itself, see the reCAPTCHA v2 solver page<\/a>.<\/p>\n The loop acts on the CAPTCHA or block decision. It turns each label into one action and tries up to four times. A rate limit sleeps for the header value, with a floor of 1 second and a cap of 60. The cap means a Retry-After of a day is retried after a minute, which can extend the limit. A bare 429, one with no Retry-After, gets two waits of 5 seconds and then a rotation, and the count starts again for the new address. With four tries that is two waits, a rotation and one more wait. A 429 that carries Retry-After never rotates. A block calls your rotate function and never reaches the solver. Your rotate function should change what the next request looks like, for example by setting session.proxies, or raise to stop the run. A challenge page calls your on_challenge function, which should leave a valid cf_clearance cookie on session.cookies, because the values the Turnstile method needs for a full-page challenge are not in the HTML a plain request receives. After either function returns, the loop simply requests the URL again. On success the function returns the response to the form post, so if your target page is a different one, request it again. Each CAPTCHA try costs one solve and can wait up to five minutes, so four failing tries can use four solves. A NetworkException means CapSkip is not reachable and is not caught here. The cf_clearance guide<\/a> covers the cookie that comes after.<\/p>\n Save this as use.py next to the other two files. Replace the address, and put your own proxy logic in rotate. The sample rotate raises, so a blocked run stops instead of asking the same address four times. Run it with the environment’s own Python, as in .\\.venv\\Scripts\\python.exe use.py. The port 8080 is the default; use the port shown in CapSkip’s connection settings if yours differs. With the placeholder address the script ends with an unknown response error, which is the expected result.<\/p>\n To see what the classifier makes of real responses before you trust it, run this check on a few URLs of your own. In a run on 29 October 2026 it printed ok for a plain page, rate_limited for a 429 and blocked for a 403 from a test service.<\/p>\n It is a set of heuristics, and these cases are known.<\/p>\nStep 1: install<\/h2>\n
py -m venv .venv\n.\\.venv\\Scripts\\python.exe -m pip install capskip requests<\/pre>\n<\/div>\n
Step 2: the signals<\/h2>\n
import re\n\n# A tag whose class attribute names a widget, or an altcha-widget tag. A link, a hidden\n# input or a sentence that merely contains the name does not match.\nCLS = r"<[^>]*\\bclass\\s*=\\s*[\\"'][^\\"']*(?<![\\w-])%s(?![\\w-])"\nWIDGET = re.compile(CLS % "(?:g-recaptcha|h-captcha|cf-turnstile|frc-captcha|captchafox)"\n + r"|<altcha-widget\\b", re.I)\nTITLE = re.compile(r"<title[^>]*>(.*?)<\/title>", re.I | re.S)\nBLOCK = ("access denied", "error 1020", "unusual traffic", "you have been blocked")<\/pre>\n<\/div>\nStep 3: decide CAPTCHA or block<\/h2>\n
\n
def classify(resp, expect=None):\n """expect: a string only the real page contains, for pages that carry a widget anyway."""\n code, h = resp.status_code, resp.headers\n if h.get("cf-mitigated", "").lower() == "challenge":\n return "cloudflare_challenge" # Cloudflare docs\n if 300 <= code < 400: # allow_redirects=False callers\n return "unknown"\n html = "html" in h.get("content-type", "").lower()\n text = resp.text[:200000] if html else ""\n low = text.lower()\n if "_cf_chl_opt" in text or "<title>just a moment" in low:\n return "cloudflare_challenge" # header stripped by a proxy\n if expect and expect in text:\n return "ok"\n if WIDGET.search(text):\n return "captcha" # even on a 429 or a 403\n if code == 429 or (code == 503 and "retry-after" in h):\n return "rate_limited"\n if code == 403 and html and "csrf" not in low:\n return "blocked" # JSON 403s fall through\n m = TITLE.search(text)\n head = m.group(1).lower() if m else ""\n small = len(text) < 3000 # block pages are short\n if code < 400 and html and any(b in head or (small and b in low) for b in BLOCK):\n return "blocked" # soft block: 200 with a notice\n return "ok" if code < 400 else "unknown"<\/pre>\n<\/div>\nStep 4: read Retry-After<\/h2>\n
import re\nimport time\nfrom datetime import datetime, timezone\nfrom email.utils import parsedate_to_datetime\nfrom urllib.parse import urljoin\n\nfrom classify import classify\n\n\ndef retry_delay(resp, fallback):\n raw = resp.headers.get("Retry-After")\n if not raw:\n return fallback\n try:\n return max(1.0, float(raw)) # delay-seconds\n except ValueError:\n pass\n try:\n when = parsedate_to_datetime(raw) # HTTP-date\n return max(1.0, (when - datetime.now(timezone.utc)).total_seconds())\n except (TypeError, ValueError):\n return fallback<\/pre>\n<\/div>\nStep 5: solve the CAPTCHA<\/h2>\n
def solve_recaptcha_form(session, resp, solver):\n """reCAPTCHA v2 only. For other types, run the detector first."""\n key = re.search(r"data-sitekey=[\\"']([^\\"']+)", resp.text)\n form = re.search(r"<form[^>]*action=[\\"']([^\\"']*)", resp.text)\n if "g-recaptcha" not in resp.text or not (key and form):\n raise RuntimeError("not a plain reCAPTCHA v2 form: run the detector first")\n token = solver.recaptcha(sitekey=key.group(1), url=resp.url)["code"]\n return session.post(urljoin(resp.url, form.group(1)),\n data={"g-recaptcha-response": token}, timeout=30)<\/pre>\n<\/div>\nStep 6: the fetch loop<\/h2>\n
def fetch(session, url, solver, rotate, on_challenge,\n tries=4, sleep=time.sleep, expect=None):\n limited = 0\n for _ in range(tries):\n resp = session.get(url, timeout=30)\n kind = classify(resp, expect)\n if kind == "ok":\n return resp\n if kind == "rate_limited":\n limited += 1\n if limited > 2 and "Retry-After" not in resp.headers:\n rotate(session) # a bare 429 that outlasts two waits\n limited = 0 # the new address starts fresh\n else:\n sleep(min(retry_delay(resp, 5.0), 60.0))\n elif kind == "captcha":\n resp = solve_recaptcha_form(session, resp, solver)\n if classify(resp, expect) == "ok":\n return resp\n elif kind == "cloudflare_challenge":\n on_challenge(session, resp) # yours: a browser, or the Turnstile guide\n elif kind == "blocked":\n rotate(session) # never solve a block\n else:\n raise RuntimeError(f"unknown response {resp.status_code} from {url}")\n raise RuntimeError(f"gave up on {url} after {tries} tries")<\/pre>\n<\/div>\nRun it<\/h2>\n
import requests\nfrom capskip import CapSkip\nfrom policy import fetch\n\nsolver = CapSkip(host="127.0.0.1", port=8080)\n\n\ndef rotate(session):\n raise RuntimeError("blocked: change the proxy here, or stop the run")\n\n\ndef on_challenge(session, resp):\n raise RuntimeError("Cloudflare challenge page: needs a browser")\n\n\nsession = requests.Session()\npage = fetch(session, "https:\/\/example.com\/data", solver, rotate, on_challenge)\nprint(page.status_code, len(page.text))<\/pre>\n<\/div>\nimport requests\nfrom classify import classify\n\nURLS = [\n "https:\/\/example.com\/",\n "https:\/\/httpbin.org\/status\/429",\n "https:\/\/httpbin.org\/status\/403",\n]\n\nfor url in URLS:\n resp = requests.get(url, timeout=30)\n print(classify(resp), resp.status_code, url)<\/pre>\n<\/div>\n
Where the classifier is wrong<\/h2>\n