How to Solve CAPTCHAs in Scrapling With page_action

A Scrapling captcha job splits cleanly in two, and knowing which half you are in saves an afternoon. Cloudflare has its own switch: pass solve_cloudflare to StealthyFetcher and the Turnstile or interstitial challenge is handled for you. Everything else is yours. reCAPTCHA v2, reCAPTCHA v3, GeeTest and image CAPTCHAs get solved outside the fetcher and injected into the page through the page_action hook. This walks through that second half, including the one detail that silently breaks it.

What you need

• Python 3.10 or newer. Scrapling requires it.
• Scrapling installed with the fetchers extra, plus its one-off browser download step.
• The URL of the protected page. You do not need the sitekey up front, because the first pass reads it.
• CapSkip running in Local mode if the scraper and the solver share a machine, or in Server mode if they do not. Both are described under connection settings.

# pip install capskip
pip install capskip
pip install "scrapling[fetchers]"

# Scrapling needs one more step. The bare package is the parser
# engine only, and importing from scrapling.fetchers without this
# raises ModuleNotFoundError.
scrapling install

What solve_cloudflare covers, and what it does not

Worth being precise about, because the name reads broader than the feature. StealthyFetcher’s solve_cloudflare argument works through Cloudflare’s Turnstile and interstitial challenges before handing you the response. That is a real convenience and you should use it. It is also the whole list.

So the shape of the work is the same every time. Get the challenge parameters out of the page, solve them somewhere else, then put the answer back into the DOM and let the site’s own form carry it. Scrapling gives you exactly one place to do that middle part, and it is page_action.

Step 1: read the sitekey without starting a browser

Scrapling has three fetchers and they are not interchangeable. Fetcher is plain HTTP. DynamicFetcher drives a Playwright browser. StealthyFetcher is the hardened one, and since version 0.3.13 it runs on Patchright rather than the Camoufox build it used before, which is worth knowing if you are following an older guide.

For a sitekey you almost never need a browser. The value sits in the markup as a data attribute, so the cheap fetcher can read it.

# pip install capskip
from scrapling.fetchers import Fetcher

PAGE_URL = "https://example.com/page-with-recaptcha"

# Plain HTTP. No browser starts, so this costs almost nothing
# and tells you which CAPTCHA you are actually facing.
page = Fetcher.get(PAGE_URL)

sitekey = page.css("[data-sitekey]::attr(data-sitekey)").get()
print(sitekey)

If that comes back empty, the widget is written in by JavaScript and is not in the served HTML. Fetch the same page once with DynamicFetcher and read it from the rendered DOM, or open it in a browser and hardcode the value. A sitekey is public and it is stable, so hardcoding it is not a shortcut you have to apologise for.

Step 2: solve it on your own hardware

One call. Every reCAPTCHA variant is the same method with different keyword arguments: invisible set to 1, enterprise set to 1, or version set to v3 with an action name. Turnstile and GeeTest have their own methods with the same shape, and the full parameter list is in the CapSkip API documentation.

# CapSkip listens on your own machine, so this is a loopback call.
from capskip import CapSkip

solver = CapSkip(host="127.0.0.1", port=8080)
token = solver.recaptcha(sitekey=sitekey, url=PAGE_URL)["code"]

That call blocks while it polls, and the poll is not on a flat interval: the SDK starts at 250 milliseconds and backs off towards pollingInterval, so it usually returns sooner than a hand-written loop against the raw endpoint. The ceiling is recaptchaTimeout, which defaults to 300 seconds.

Step 3: inject the token in page_action

page_action takes a function, receives the Playwright page object, and runs after navigation and after the network_idle wait, but before wait_selector. That ordering is the useful part: the widget has rendered by the time your function runs, and anything you wait for afterwards sees the result of what you did.

from playwright.sync_api import Page

def inject_token(page: Page):
    # The response textarea is hidden, so set the value directly
    # rather than trying to type into it.
    page.evaluate(
        "(t) => document.getElementById('g-recaptcha-response').value = t",
        token,
    )
    page.click("button[type=submit]")

The function does not need to return anything. Older versions of Scrapling wanted the page object back and the current documentation does not, so returning nothing is the safe spelling either way.

The isolated execution context, which is the part that bites

StealthyFetcher evaluates your JavaScript in Patchright’s isolated execution context by default. The DOM is shared, so reading and writing elements works exactly as you would expect. The page’s own JavaScript globals are not shared, so anything the site put on window is simply missing.

That distinction decides whether your injection works. Setting the value of the response textarea touches the DOM only, and it works fine in the isolated world. Calling the site’s reCAPTCHA callback does not, because that callback lives on a global the page’s own script created. Sites that use a callback rather than an ordinary form submit will look like they ignored a perfectly good token.

def inject_and_fire_callback(page: Page):
    # isolated_context=False drops into the page's own world,
    # where the site's grecaptcha object actually exists.
    page.evaluate(
        """(t) => {
            document.getElementById('g-recaptcha-response').value = t;
            window.onCaptchaSuccess(t);
        }""",
        token,
        isolated_context=False,
    )

Read the callback name off the widget before you write this. It is whatever the data-callback attribute on the reCAPTCHA element says, and it differs per site.

Full working example

Cheap HTTP pass to read the sitekey, one solve, then a single stealthy fetch that injects and submits. Note that the solve happens before the fetch, so the token is already in hand when page_action runs.

# pip install capskip
from capskip import CapSkip
from playwright.sync_api import Page
from scrapling.fetchers import Fetcher, StealthyFetcher

PAGE_URL = "https://example.com/page-with-recaptcha"

solver = CapSkip(host="127.0.0.1", port=8080)

sitekey = Fetcher.get(PAGE_URL).css("[data-sitekey]::attr(data-sitekey)").get()
token = solver.recaptcha(sitekey=sitekey, url=PAGE_URL)["code"]

def inject_token(page: Page):
    page.evaluate(
        "(t) => document.getElementById('g-recaptcha-response').value = t",
        token,
    )
    page.click("button[type=submit]")

page = StealthyFetcher.fetch(
    PAGE_URL,
    headless=True,
    network_idle=True,
    page_action=inject_token,
    wait_selector=".dashboard",
)

print(page.css(".dashboard h1::text").get())

Two arguments there are load bearing. network_idle makes the fetcher wait until there have been no network connections for 500 milliseconds, which is usually long enough for the widget to render. wait_selector is what proves the submit worked: point it at something that only exists on the far side of the form, and a failed submit becomes a timeout instead of a silent success.

Doing several pages with a session

Every class-level fetch starts a browser and throws it away. For more than a couple of pages, open a session instead and keep the browser between requests. Most fetch arguments can be set once on the session and overridden per request, including page_action, wait_selector and solve_cloudflare.

from scrapling.fetchers import StealthySession

# solve_cloudflare here handles the Cloudflare layer for every
# page in the session. Anything else still goes through
# page_action, per request.
with StealthySession(headless=True, solve_cloudflare=True) as session:
    for url in urls:
        page = session.fetch(url, page_action=inject_token)
        print(page.css("title::text").get())

Solve as late as you can. A reCAPTCHA token is good for about two minutes, so solving twenty of them up front and then walking a queue will expire most of them. Solve inside the loop, immediately before the fetch that uses it. What that lifetime means in practice is covered in the guide to reCAPTCHA token expiration.

Running the solver on another machine

Scrapers move. A VPS, a container or a scheduled worker is not the machine with the solver on it, and loopback there points at the worker itself, where nothing is listening.

CapSkip has two connection modes for exactly this. Local binds to 127.0.0.1 and answers that device only. Server binds to your network address or public IP, so a scraper anywhere can reach the same Windows machine over the API. A static public IP keeps the address stable. It is your hardware either way and it is unmetered either way, so a run that solves fifty thousand challenges costs the same as one that solves fifty.

# The SDK reads these three itself, so the same script runs
# whether the solver is on this box or on another one:
#   CAPSKIP_HOST=192.0.2.10
#   CAPSKIP_PORT=8080
#   CAPSKIP_API_KEY=your-key

solver = CapSkip()

Turn on key validation once the solver listens on a network address, and give each worker its own key so one can be revoked without disturbing the rest. Both modes are walked through in the CapSkip setup guide.

Proxies on both ends

If the site checks that the token came from the address that submits it, the solve and the fetch have to leave from the same exit. Scrapling takes a proxy argument on the fetchers, and CapSkip takes one per task for reCAPTCHA, Turnstile and GeeTest. Image CAPTCHAs do not accept a proxy and do not need one.

PROXY = "http://user:[email protected]:3128"

token = solver.recaptcha(
    sitekey=sitekey,
    url=PAGE_URL,
    proxy={"type": "HTTP", "uri": "user:[email protected]:3128"},
)["code"]

page = StealthyFetcher.fetch(PAGE_URL, proxy=PROXY, page_action=inject_token)

The two spellings are different on purpose: the solver takes a type and a URI as separate fields, the fetcher takes one URL string. Getting one of them wrong means the solve leaves from your real address while the fetch leaves from the proxy, which looks exactly like a bad token and is not one. Picking a proxy type is covered in the guide to rotating proxies while solving.

Common errors and what they mean

FAQ

The short version

Let Scrapling handle Cloudflare with solve_cloudflare, and handle everything else yourself in page_action. Read the sitekey with the plain HTTP fetcher, solve it, then set the response field from inside the hook. If the site fires a callback instead of submitting a form, evaluate with the isolated context turned off, or the token lands in a world the page cannot see. Point wait_selector at something that only exists after a successful submit, so a failure is loud.

The wider Python story lives on the Python CAPTCHA solver page, the specifics of the checkbox challenge on the reCAPTCHA v2 solver page, and the Cloudflare side on the Cloudflare Turnstile solver page. The same three calls exist in Node.js, PHP and C#, listed on the CAPTCHA solving SDK page.

One thing to weigh before you point this at a real crawl. CapSkip is an unlimited captcha solver that runs on hardware you already own, so the CAPTCHA line of a large scrape is a fixed cost rather than a per-solve one.