A Scrapy captcha belongs in a downloader middleware, not in your spider. The middleware sees every response, so it can spot the challenge once, solve it, and hand the real page back to the spider as if nothing happened. Your parse methods stay clean. This guide builds that middleware, keeps the solve off the event loop, and covers the settings that decide whether it works at scale or stalls your crawl.<\/p>\n
pip install capskip<\/code><\/li>\n<\/ul>\nEverything below assumes the solver is on 127.0.0.1:8080<\/code>. Nothing leaves your machine, which matters more than usual in a crawler: you are already sending a lot of requests, and a per-solve round trip to a third party adds latency to every one of them.<\/p>\nWhy Scrapy CAPTCHA handling belongs in a middleware<\/h2>\n
Handling the challenge in a callback means every callback needs the same branch. Miss one and that spider silently parses a block page as if it were data. A downloader middleware sits between the downloader and the spider, so it gets the response first and can replace it.<\/p>\n
That placement buys you three things. Detection lives in one function instead of being copy-pasted across spiders. The spider’s callbacks only ever receive real pages, so their selectors are allowed to assume the markup they expect. And when a site changes its challenge, you edit one file rather than auditing a project.<\/p>\n
Order matters, and it is the part people get wrong. Scrapy calls process_request<\/code> in increasing order and process_response<\/code> in decreasing<\/em> order. Registering at 585 means our middleware sees responses before RetryMiddleware<\/code> at 550 does.<\/p>\n# settings.py\nDOWNLOADER_MIDDLEWARES = {\n "myproject.middlewares.CaptchaMiddleware": 585,\n}\n\n# The async client needs Scrapy's asyncio reactor.\nTWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"<\/pre>\nStep 1: detect the challenge<\/h2>\n
Two signals cover most sites. The status code, and a marker in the body. Check both, because plenty of sites serve the challenge with a 200.<\/p>\n
# Markers for the two widgets you will hit most often.\nCAPTCHA_MARKERS = ("g-recaptcha", "cf-turnstile")\n\ndef looks_like_captcha(response):\n if response.status in (403, 429):\n return True\n # Only touch the body for HTML; binary responses have no text.\n ctype = response.headers.get("Content-Type", b"").decode()\n if "html" not in ctype:\n return False\n return any(m in response.text for m in CAPTCHA_MARKERS)<\/pre>\nKeep this function boring and cheap. It runs on every single response in the crawl.<\/p>\n
Step 2: pull the sitekey off the page<\/h2>\n
The sitekey is a public attribute on the widget element. Read it from the response you already have, never from a hardcoded constant, because sites rotate them.<\/p>\n
def extract_sitekey(response):\n # reCAPTCHA v2 and invisible both use data-sitekey.\n key = response.css(".g-recaptcha::attr(data-sitekey)").get()\n if key:\n return "recaptcha", key\n key = response.css(".cf-turnstile::attr(data-sitekey)").get()\n if key:\n return "turnstile", key\n return None, None<\/pre>\nIf the widget is injected by JavaScript, the sitekey will not be in the HTML Scrapy downloaded. That is a rendering problem rather than a solving one, and it usually means grabbing the key from the script tag with a regex instead.<\/p>\n
Step 3: solve without blocking the crawl<\/h2>\n
This is the step that quietly ruins throughput. Scrapy runs on a single-threaded event loop. A solve takes seconds, so calling the blocking client directly inside your middleware freezes every other request in flight for that whole time. With CONCURRENT_REQUESTS<\/code> at 16, you just serialised all 16.<\/p>\nTwo correct ways out. The Python SDK ships AsyncCapSkip<\/code>, which is a real async client rather than an alias, so under the asyncio reactor you can await it directly:<\/p>\n# pip install capskip\nfrom capskip import AsyncCapSkip\n\nsolver = AsyncCapSkip(host="127.0.0.1", port=8080)\n\n# Submit and poll both happen inside this await.\nresult = await solver.recaptcha(\n sitekey="YOUR_SITEKEY",\n url="https:\/\/example.com\/page-with-recaptcha",\n)\ntoken = result["code"]<\/pre>\nIf you are still on the classic Twisted reactor, push the blocking client into a thread and await the Deferred:<\/p>\n
from capskip import CapSkip\nfrom twisted.internet.threads import deferToThread\n\nsolver = CapSkip(host="127.0.0.1", port=8080)\n\n# deferToThread keeps the reactor free while the solve runs.\nresult = await deferToThread(\n solver.recaptcha,\n sitekey="YOUR_SITEKEY",\n url="https:\/\/example.com\/page-with-recaptcha",\n)<\/pre>\nBoth work because Scrapy lets any downloader middleware method be a coroutine function. Define it with async def<\/code> and Scrapy handles the rest.<\/p>\nStep 4: resubmit with the token<\/h2>\n
A token on its own does nothing. It has to go back to the site the way the site’s own front end would send it, which for a classic form means a field called g-recaptcha-response<\/code>.<\/p>\nReturn a new Request<\/code> from process_response<\/code> and Scrapy reschedules it. Two details keep this from going wrong: dont_filter=True<\/code>, because the URL has already been seen and the dupe filter would drop it, and a counter in meta<\/code> so a site that keeps challenging you cannot loop forever.<\/p>\nfrom scrapy import FormRequest\n\ndef resubmit(response, token, tries):\n return FormRequest.from_response(\n response,\n formdata={"g-recaptcha-response": token},\n dont_filter=True, # the dupe filter has seen this URL\n meta={"captcha_tries": tries + 1},\n )<\/pre>\nThe full middleware<\/h2>\n# myproject\/middlewares.py\nimport logging\n\nfrom capskip import (\n AsyncCapSkip, ApiException, NetworkException, TimeoutException)\nfrom scrapy import FormRequest\nfrom scrapy.exceptions import IgnoreRequest\n\nlogger = logging.getLogger(__name__)\nMAX_CAPTCHA_TRIES = 2\n\n\nclass CaptchaMiddleware:\n def __init__(self):\n self.solver = AsyncCapSkip(host="127.0.0.1", port=8080)\n\n async def process_response(self, request, response, spider):\n if not looks_like_captcha(response):\n return response\n\n tries = request.meta.get("captcha_tries", 0)\n if tries >= MAX_CAPTCHA_TRIES:\n raise IgnoreRequest("captcha not cleared: %s" % request.url)\n\n kind, sitekey = extract_sitekey(response)\n if not sitekey:\n return response # not a shape we handle\n\n try:\n if kind == "turnstile":\n result = await self.solver.turnstile(\n sitekey=sitekey, url=response.url)\n else:\n result = await self.solver.recaptcha(\n sitekey=sitekey, url=response.url)\n except (ApiException, NetworkException, TimeoutException) as e:\n logger.warning("solve failed for %s: %s", request.url, e)\n return response\n\n logger.info("solved %s captcha for %s", kind, request.url)\n\n return FormRequest.from_response(\n response,\n formdata={"g-recaptcha-response": result["code"]},\n dont_filter=True,\n meta={**request.meta, "captcha_tries": tries + 1},\n )<\/pre>\nThe SDK raises four exception types: ValidationException<\/code>, NetworkException<\/code>, ApiException<\/code> and TimeoutException<\/code>. The three above are the ones that happen at runtime; a ValidationException<\/code> means your parameters are wrong and should fail loudly during development rather than being swallowed. Returning the original response on failure lets the rest of your pipeline decide what to do, which beats crashing the spider.<\/p>\nSettings that actually matter<\/h2>\n