How to Handle a Scrapy CAPTCHA with Downloader Middleware

August 8, 2026 Comments Off

A Scrapy captcha belongs in a downloader middleware, not in your spider. The middleware sees every response, so it can spot the challenge once, solve it, and hand the real page back to the spider as if nothing happened. Your parse methods stay clean. This guide builds that middleware, keeps the solve off the event loop, and covers the settings that decide whether it works at scale or stalls your crawl.

What you need

Scrapy 2.x and Python 3.10 or newer
CapSkip running locally, with the API server on. See the setup guide
The Python SDK: pip install capskip

Everything below assumes the solver is on 127.0.0.1:8080. Nothing leaves your machine, which matters more than usual in a crawler: you are already sending a lot of requests, and a per-solve round trip to a third party adds latency to every one of them.

Why Scrapy CAPTCHA handling belongs in a middleware

Handling the challenge in a callback means every callback needs the same branch. Miss one and that spider silently parses a block page as if it were data. A downloader middleware sits between the downloader and the spider, so it gets the response first and can replace it.

That placement buys you three things. Detection lives in one function instead of being copy-pasted across spiders. The spider’s callbacks only ever receive real pages, so their selectors are allowed to assume the markup they expect. And when a site changes its challenge, you edit one file rather than auditing a project.

Order matters, and it is the part people get wrong. Scrapy calls process_request in increasing order and process_response in decreasing order. Registering at 585 means our middleware sees responses before RetryMiddleware at 550 does.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CaptchaMiddleware": 585,
}

# The async client needs Scrapy's asyncio reactor.
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Step 1: detect the challenge

Two signals cover most sites. The status code, and a marker in the body. Check both, because plenty of sites serve the challenge with a 200.

# Markers for the two widgets you will hit most often.
CAPTCHA_MARKERS = ("g-recaptcha", "cf-turnstile")

def looks_like_captcha(response):
    if response.status in (403, 429):
        return True
    # Only touch the body for HTML; binary responses have no text.
    ctype = response.headers.get("Content-Type", b"").decode()
    if "html" not in ctype:
        return False
    return any(m in response.text for m in CAPTCHA_MARKERS)

Keep this function boring and cheap. It runs on every single response in the crawl.

Step 2: pull the sitekey off the page

The sitekey is a public attribute on the widget element. Read it from the response you already have, never from a hardcoded constant, because sites rotate them.

def extract_sitekey(response):
    # reCAPTCHA v2 and invisible both use data-sitekey.
    key = response.css(".g-recaptcha::attr(data-sitekey)").get()
    if key:
        return "recaptcha", key
    key = response.css(".cf-turnstile::attr(data-sitekey)").get()
    if key:
        return "turnstile", key
    return None, None

If the widget is injected by JavaScript, the sitekey will not be in the HTML Scrapy downloaded. That is a rendering problem rather than a solving one, and it usually means grabbing the key from the script tag with a regex instead.

Step 3: solve without blocking the crawl

This is the step that quietly ruins throughput. Scrapy runs on a single-threaded event loop. A solve takes seconds, so calling the blocking client directly inside your middleware freezes every other request in flight for that whole time. With CONCURRENT_REQUESTS at 16, you just serialised all 16.

Two correct ways out. The Python SDK ships AsyncCapSkip, which is a real async client rather than an alias, so under the asyncio reactor you can await it directly:

# pip install capskip
from capskip import AsyncCapSkip

solver = AsyncCapSkip(host="127.0.0.1", port=8080)

# Submit and poll both happen inside this await.
result = await solver.recaptcha(
    sitekey="YOUR_SITEKEY",
    url="https://example.com/page-with-recaptcha",
)
token = result["code"]

If you are still on the classic Twisted reactor, push the blocking client into a thread and await the Deferred:

from capskip import CapSkip
from twisted.internet.threads import deferToThread

solver = CapSkip(host="127.0.0.1", port=8080)

# deferToThread keeps the reactor free while the solve runs.
result = await deferToThread(
    solver.recaptcha,
    sitekey="YOUR_SITEKEY",
    url="https://example.com/page-with-recaptcha",
)

Both work because Scrapy lets any downloader middleware method be a coroutine function. Define it with async def and Scrapy handles the rest.

Step 4: resubmit with the token

A token on its own does nothing. It has to go back to the site the way the site’s own front end would send it, which for a classic form means a field called g-recaptcha-response.

Return a new Request from process_response and Scrapy reschedules it. Two details keep this from going wrong: dont_filter=True, because the URL has already been seen and the dupe filter would drop it, and a counter in meta so a site that keeps challenging you cannot loop forever.

from scrapy import FormRequest

def resubmit(response, token, tries):
    return FormRequest.from_response(
        response,
        formdata={"g-recaptcha-response": token},
        dont_filter=True,          # the dupe filter has seen this URL
        meta={"captcha_tries": tries + 1},
    )

The full middleware

# myproject/middlewares.py
import logging

from capskip import (
    AsyncCapSkip, ApiException, NetworkException, TimeoutException)
from scrapy import FormRequest
from scrapy.exceptions import IgnoreRequest

logger = logging.getLogger(__name__)
MAX_CAPTCHA_TRIES = 2


class CaptchaMiddleware:
    def __init__(self):
        self.solver = AsyncCapSkip(host="127.0.0.1", port=8080)

    async def process_response(self, request, response, spider):
        if not looks_like_captcha(response):
            return response

        tries = request.meta.get("captcha_tries", 0)
        if tries >= MAX_CAPTCHA_TRIES:
            raise IgnoreRequest("captcha not cleared: %s" % request.url)

        kind, sitekey = extract_sitekey(response)
        if not sitekey:
            return response          # not a shape we handle

        try:
            if kind == "turnstile":
                result = await self.solver.turnstile(
                    sitekey=sitekey, url=response.url)
            else:
                result = await self.solver.recaptcha(
                    sitekey=sitekey, url=response.url)
        except (ApiException, NetworkException, TimeoutException) as e:
            logger.warning("solve failed for %s: %s", request.url, e)
            return response

        logger.info("solved %s captcha for %s", kind, request.url)

        return FormRequest.from_response(
            response,
            formdata={"g-recaptcha-response": result["code"]},
            dont_filter=True,
            meta={**request.meta, "captcha_tries": tries + 1},
        )

The SDK raises four exception types: ValidationException, NetworkException, ApiException and TimeoutException. The three above are the ones that happen at runtime; a ValidationException means your parameters are wrong and should fail loudly during development rather than being swallowed. Returning the original response on failure lets the rest of your pipeline decide what to do, which beats crashing the spider.

Settings that actually matter

Setting	Why
`CONCURRENT_REQUESTS_PER_DOMAIN`	Lower it. Hitting a CAPTCHA is usually a rate signal, and solving faster does not fix the reason you were challenged
`DOWNLOAD_DELAY` plus `AUTOTHROTTLE_ENABLED`	Cheaper than solving. Every challenge you avoid costs nothing
`COOKIES_ENABLED`	Must stay on. The clearance cookie from a solved challenge is what stops the next request being challenged
`RETRY_TIMES`	Independent of your CAPTCHA counter. Keep the two limits separate or they multiply

That third row is the one to internalise. If cookies are off, every request looks like a first visit and you will solve the same challenge forever. Most Scrapy captcha loops people report turn out to be exactly this.

Image CAPTCHAs inside a spider

Some sites use a plain distorted-text image on a login form. There is no sitekey involved, so it is simpler: download the image and pass the bytes as a data URI.

import base64
import scrapy
from capskip import CapSkip

solver = CapSkip(host="127.0.0.1", port=8080)

class LoginSpider(scrapy.Spider):
    def parse_captcha_image(self, response):
        # response.body is the raw image, fetched with session cookies.
        b64 = base64.b64encode(response.body).decode()
        result = solver.normal("data:image/png;base64," + b64)
        return result["code"]     # the text on the image

normal() also takes a file path or a remote URL. The data URI form is the one you want in Scrapy, because the image usually only renders correctly for the session that requested it. Note that proxies are not supported for image CAPTCHAs, only for reCAPTCHA, Turnstile and GeeTest.

Common Scrapy CAPTCHA errors

Symptom	Cause	Fix
Crawl throughput collapses	A blocking solve on the event loop	Use `AsyncCapSkip` or `deferToThread`
The resubmitted request never runs	The dupe filter dropped it	Add `dont_filter=True`
Same page challenges forever	Cookies disabled, or `meta` not carried forward	Enable cookies, merge `request.meta` into the new request
`ERROR_GOOGLEKEY`	The sitekey was stale or hardcoded	Read it from the live response every time
`NetworkException` on every solve	CapSkip is not running or the port differs	Start the app, check the port in Settings

Every error string the API can return is listed in the API documentation. Scrapy’s own downloader middleware reference covers the ordering rules in full.

Frequently asked questions

Does this work with Scrapy’s default reactor?

Yes, but only the deferToThread variant. AsyncCapSkip is an asyncio client, so it needs TWISTED_REACTOR set to the asyncio reactor. Both approaches keep the crawl moving.

Should I solve every Scrapy captcha I hit?

No. A sudden wall of challenges means your crawl pattern got flagged. Slow down first. Solving through it treats the symptom and usually earns you a harder block.

Can I route the solve through the same proxy as the request?

Yes, for reCAPTCHA, Turnstile and GeeTest. Pass proxy={"type": "HTTPS", "uri": "user:[email protected]:3128"} to the solve call so the token is generated from the same exit IP the page saw.

Where does the middleware go if I also use a proxy middleware?

Below it in process_response terms, which means a higher number. Proxy middlewares act on requests; ours acts on responses. At 585 it runs before the retry middleware sees the response.

Summary

A Scrapy captcha is a five-part problem: detect the challenge in a middleware, read the sitekey off the live response, solve it without blocking the reactor, resubmit with dont_filter=True, and cap the retries in meta. Get those right and spiders stay clean, because one file handles the whole thing.

For the wider picture, see how CapSkip fits into a crawler on the CAPTCHA solver for web scraping page, the Python integration guide, or the details of the widget itself on the reCAPTCHA v2 solver page. CapSkip is a captcha solver that runs on your own machine, so adding it to a crawl costs you no extra network hop.

CAPTCHA types

Developers

AI Integration

Alternatives

Getting Started

Developer Resources