How to Solve CAPTCHA in Botasaurus Without an Extension

botasaurus captcha - How to Solve CAPTCHA in Botasaurus Without an Extension

A Botasaurus captcha is easiest to handle inside the task itself, not through a browser extension. The framework’s own documentation points at a paid cloud extension, which solves the challenge invisibly and bills you for every attempt. Calling a solver directly from your decorated function is less code, and it hands you the token as an ordinary Python value you can log, branch on and retry. This guide covers reading the sitekey, solving, injecting the answer with driver.run_js, and the two decorator options that quietly break the whole thing.

What you need

  • Python 3.10 or newer, which both packages target
  • Botasaurus installed, plus the solver client
  • CapSkip running and reachable. Local mode listens on 127.0.0.1 port 8080 when the scraper runs on the same machine, and Server mode listens on your network or public IP so a VPS or a container host can call it. See the connection settings for both
  • A page with a real widget on it. The sitekey has to come off the live DOM, because it is per site
# The scraper and the solver client, in one line.
python -m pip install --upgrade botasaurus capskip

Why not the extension route

Botasaurus supports Chrome extensions through the decorator, and its documentation shows a CapSolver extension wired in with an API key. That works. It also decides three things for you that are worth keeping.

You never see the token. The extension writes the answer into the page on its own schedule, so your code has no value to inspect. When a solve fails you get a page that still has an unsolved widget on it, which surfaces later as a confusing form error rather than an exception at the point of failure.

Every attempt is billed. A per solve cloud API turns retries into a line item. Botasaurus makes that sharper than usual, because max_retry re-runs the whole decorated function and the parallel option multiplies it by the number of browsers. Ten workers with three retries each is a plausible afternoon and a bill you did not plan. Running the solver on your own hardware removes the calculation entirely, and that argument is made in more detail on the CapSolver alternative page.

Your API key ships to the browser. An extension is loaded into the Chrome instance, key and all. A local solver takes a request from your Python process and nothing else.

Step 1: read the sitekey inside the task

Everything happens inside the decorated function, where you already have a driver.

# python -m pip install botasaurus
from botasaurus.browser import browser, Driver, Wait

@browser(block_images=True)
def solve_login(driver: Driver, data):
    driver.get("https://example.com/page-with-recaptcha")

    # The widget is injected by script, so wait for it
    # rather than reading the DOM immediately.
    widget = driver.select("[data-sitekey]", wait=Wait.LONG)
    sitekey = widget.get_attribute("data-sitekey")

    return {"sitekey": sitekey}   # written to output/solve_login.json

Two Botasaurus habits to pick up here. The select call returns the element or None rather than raising, so a missing widget reads as a NoneType error one line later unless you check for it. And whatever you return is written to a JSON file under the output directory, named after the function, which makes a half finished task easy to inspect while you are still building it.

Step 2: solve, then inject with driver.run_js

Botasaurus does not drive the browser through Selenium, so the execute_script recipes you find elsewhere do not apply here. The equivalent is driver.run_js, and it takes an arguments object as its second parameter.

# python -m pip install capskip
from capskip import CapSkip

solver = CapSkip(host="127.0.0.1", port=8080)

result = solver.recaptcha(
    sitekey=sitekey,
    url="https://example.com/page-with-recaptcha",
)

# args is how Botasaurus passes Python values into the page.
driver.run_js(
    "document.getElementById('g-recaptcha-response').value = args.token",
    {"token": result["code"]},
)

The same call covers the whole reCAPTCHA family. Pass invisible for an invisible widget, enterprise for Enterprise, and a version of v3 with an action for v3. Turnstile and GeeTest are separate methods with the same shape. For v3 you can also pass min_score: CapSkip accepts it so code ported from other services keeps working, but it does not change the token, because Google decides the score.

If the widget div carries a data-callback attribute, call that function with the token instead of submitting the form. Pages built that way never wire up a normal submit handler, so a click on the button does nothing at all.

The full working example

# python -m pip install botasaurus capskip
from botasaurus.browser import browser, Driver, Wait
from capskip import CapSkip, NetworkException, TimeoutException

solver = CapSkip(host="127.0.0.1", port=8080)
PAGE = "https://example.com/page-with-recaptcha"

@browser(block_images=True, max_retry=3, retry_wait=10)
def scrape_behind_recaptcha(driver: Driver, data):
    driver.get(PAGE)

    widget = driver.select("[data-sitekey]", wait=Wait.LONG)
    if widget is None:
        return {"error": "no widget on the page"}

    try:
        result = solver.recaptcha(
            sitekey=widget.get_attribute("data-sitekey"),
            url=driver.run_js("return location.href"),
        )
    except (NetworkException, TimeoutException) as exc:
        # Raising here lets max_retry take another run at it.
        raise RuntimeError(f"solve failed: {exc}")

    driver.run_js(
        "document.getElementById('g-recaptcha-response').value = args.token",
        {"token": result["code"]},
    )
    driver.click("button[type='submit']")

    return {"title": driver.run_js("return document.title")}

scrape_behind_recaptcha()

Note that the page URL comes out of the browser rather than the constant. If the site redirected on the way in, the URL the widget was rendered against is the one the solve needs, and reusing your own literal is the most common cause of a token the server then refuses.

Two decorator options that break the solve

Both are things Botasaurus does correctly, which is exactly why they are confusing when a CAPTCHA is involved.

Caching stores the returned value, keyed on the input. Turn it on, run the same input a second time, and the framework hands back the stored dictionary without opening a browser, so no solve happens and no request reaches the solver. That is a feature when you are re-running a big scrape. It looks like a broken integration when you are debugging one page, so clear the cache or drop the option while you work.

The proxy option on the decorator is the browser’s proxy, not the solver’s. They are separate settings and they need to agree. A token minted from your own IP while the browser sits behind a residential proxy is a mismatch the site can see, and it is why a solve that verified fine in isolation fails inside the real flow. Pass the same endpoint to the solve call:

PROXY = "user:[email protected]:3128"

@browser(proxy=f"http://{PROXY}")
def scrape(driver: Driver, data):
    result = solver.recaptcha(
        sitekey=sitekey,
        url=PAGE,
        proxy={"type": "HTTPS", "uri": PROXY},
    )

Proxies apply to reCAPTCHA, Turnstile, GeeTest and CaptchaFox. Image CAPTCHAs do not take one, because there is no request to a third party to route.

Running many browsers at once

This is where a local solver earns its place. Setting the parallel option on the decorator starts that many browsers, each running your function against its own input, and each hitting a challenge of its own.

@browser(parallel=8, reuse_driver=True, block_images=True)
def scrape_behind_recaptcha(driver: Driver, data):
    ...

scrape_behind_recaptcha([url1, url2, url3])

One client object is safe to share across all eight, because each call is an independent HTTP request. The SDK starts polling at 250 milliseconds and backs off to the pollingInterval ceiling, so a fast solve stays fast even with several in flight. Capacity is a hardware question rather than a billing one, which is what makes eight workers a reasonable default instead of a decision you have to justify. Wider scraping patterns are covered on the web scraping page.

Running CapSkip on a server instead

Botasaurus scrapers usually end up somewhere other than a laptop: a VPS, a Docker host, a scheduled box. Loopback does not follow them there.

ModeListens onUse it when
Local127.0.0.1, that device onlyThe scraper and the solver share a machine
ServerYour network or public IPThe scraper runs on a VPS, a container or another box

Only the constructor changes. The SDK does not read CAPSKIP_HOST or CAPSKIP_PORT by itself, so the constructor below reads them from the environment, and a deployed scraper then needs the two variables rather than a code edit.

import os
from capskip import CapSkip

# Same client, different address. The task never changes.
solver = CapSkip(
    host=os.getenv("CAPSKIP_HOST", "127.0.0.1"),
    port=int(os.getenv("CAPSKIP_PORT", "8080")),
)

A static public IP is recommended for the solver box so the address stays put, and the steps live in the connection settings. It is still your hardware and still unmetered. The only thing that changed is where the process listens.

Common errors and what they mean

SymptomCauseFix
AttributeError on NoneTypeThe select call found nothing and returned NonePass a wait argument, then check for None before reading the attribute
The task returns instantly and no browser opensCaching served a stored result for that inputClear the cache or drop the option while debugging
NetworkException on the first solveNothing is listening at that host and portStart CapSkip, or switch to Server mode if the scraper is remote
ERROR_GOOGLEKEYThe sitekey belongs to a Turnstile widget, or was cut shortPrint the attribute before solving; Turnstile has its own method
ERROR_PAGEURLA hardcoded URL that no longer matches the pageRead the address out of the browser instead
The form submits and the server rejects the tokenThe page uses a callback, so the hidden field alone is not enoughInvoke the data-callback function with the token
Works alone, fails behind a proxyThe browser is proxied and the solve is notGive both the same endpoint

Every parameter and every error string is listed in the API documentation.

Frequently asked questions

Does the request decorator work for this instead?

No, and the Botasaurus documentation is explicit about why: the request decorator makes browser-like HTTP calls but cannot run JavaScript, and a challenge widget is JavaScript. Use the browser decorator for the page that carries the widget. A request task is still the right tool for the pages behind it, once you hold a session.

What about the Cloudflare interstitial rather than a widget?

Those are two different things, and Botasaurus already handles the first one. The full page interstitial you get before a site loads is what the bypass_cloudflare argument on google_get is for. A Turnstile widget sitting inside a login form is a normal challenge on a normal page, and it needs a token like any other, so solve it and write the answer into the cf-turnstile-response field.

Can I run this headless on a server with no display?

Yes. Botasaurus takes a headless option and an X virtual framebuffer option for Linux boxes without a display, and neither one touches the solve, which is an HTTP call from your Python process. Point the client at a reachable address with Server mode and the same task runs unchanged on a VPS.

Should the solve go in a separate task decorator?

It can, and it is tidy when several scrapers share one solve routine. The task decorator runs any Python you like with the same caching and retry machinery behind it. Keep it in the browser function when the token has to go back into the page it came from, because splitting it means passing the driver around for no benefit.

The short version

Read the sitekey with a waited select, solve from inside the decorated function, and write the token into the page with driver.run_js and its args object. Skip the extension: holding the token as a value is what lets you retry deliberately, and it keeps the API key out of the browser.

The two options to watch are caching, which will hand back a stored result and never open a browser, and the proxy setting, which covers the browser and not the solve. Running the solver yourself is what makes a retry free, and that is the whole reason captcha bypass at eight workers is a capacity decision rather than a budget one. Client options are covered in the Python integration guide, and the framework’s own reference lives in the Botasaurus repository.