How to Solve CAPTCHAs in Locust Without Skewing Your Stats

A Locust captcha solve has to stay out of two places: your statistics, and the per-iteration path. Locust reports every request made through self.client, so solving through it drops the solver’s response times into the report you are trying to read. And a solve placed inside a task runs on every iteration of every user. Neither is hard to avoid once you know where the boundaries sit.

What you need

• Locust 2.x on Python 3.10 or newer, which is also what the CapSkip client needs.
• CapSkip running on a Windows machine, with the Python client installed alongside your locustfile.
• The sitekey and page URL of the protected form, passed in rather than rediscovered by every user.
• Server mode if Locust runs anywhere other than the solver’s own machine, which includes every container and every worker box. It is one setting under connection settings.

# pip install capskip
pip install -U locust capskip

Step 1: keep the solve off self.client

This is the part that quietly ruins a report. Locust’s documentation is explicit about what self.client is: an instance of HttpSession, which is a subclass and wrapper of requests.Session, and what it adds is the reporting of request results into Locust. Success and failure, response time, response length, name. Everything sent through it lands in the statistics table.

A solve takes seconds. Your application’s endpoints take milliseconds. Put one in the same table as the other and your ninety-fifth percentile becomes a measurement of how long a CAPTCHA took, which is not a number anybody asked for.

The good news is that the fix is doing nothing. The CapSkip client has its own HTTP transport and never touches self.client, so a solve is invisible to Locust’s statistics by default. Locust only logs what goes through its own session.

# pip install capskip
import os
from capskip import CapSkip

# CAPSKIP_HOST is the solver machine. 127.0.0.1 only works when
# Locust and CapSkip run on the same Windows box.
solver = CapSkip(
    host=os.environ.get("CAPSKIP_HOST", "127.0.0.1"),
    port=int(os.environ.get("CAPSKIP_PORT", "8080")),
    apiKey=os.environ.get("CAPSKIP_API_KEY", "capskip"),
)

def fresh_token(sitekey, page_url):
    result = solver.recaptcha(sitekey=sitekey, url=page_url)
    return result["code"]   # the token, and Locust never sees it

If you would rather hand-roll against the raw endpoints, the same rule holds: use a plain requests.Session, not self.client. Requests made with the requests library directly are not logged by Locust, which is exactly what you want here. Those two endpoints are documented in the CapSkip API documentation.

The one time to do the opposite is when you are deliberately load testing the solver itself. Then send it through self.client with a name argument, so every solve groups under a single row instead of one row per sitekey, and read that row separately from the rest.

Step 2: how often does your solve actually run?

Locust gives you four places to put it and they differ by orders of magnitude. Work out the multiplier before picking one.

The default answer for almost everybody is on_start. A user calls on_start when it starts running, so each simulated user gets its own token, holds it for its own session, and nothing has to be passed between greenlets. That is also the shape that matches what a real user does.

from locust import HttpUser, task, between

class Signup(HttpUser):
    wait_time = between(1, 3)

    def on_start(self):
        # One solve per simulated user, off the statistics.
        self.token = fresh_token(SITEKEY, PAGE_URL)

    @task
    def submit(self):
        self.client.post("/signup", data={
            "email": "[email protected]",
            "g-recaptcha-response": self.token,
        })

Five hundred solves sounds expensive because with a metered service it is. That is the reason most load testing guides contort themselves into solving once and sharing the result. With a solver running on your own hardware the number stops being a budget question and becomes a capacity question, which is a much easier one to answer: run the ramp and watch the machine.

Step 3: test_start fires on each node, not once per run

This is the Locust detail that surprises people, and it shows up as a solve count that is a neat multiple of something. The documentation says test_start is fired on each node when a new load test is started. Run with four worker processes and you have five nodes, so a solve in a plain test_start listener runs five times.

Guard it by checking the runner type. Locust ships MasterRunner and WorkerRunner for exactly this, and the pattern in its own distributed guide is to test which one you are on.

from locust import events
from locust.runners import WorkerRunner

@events.init.add_listener
def on_init(environment, **kwargs):
    # Workers listen. Registering here runs before the test starts.
    if isinstance(environment.runner, WorkerRunner):
        environment.runner.register_message("captcha_token", take_token)

@events.test_start.add_listener
def on_test_start(environment, **kwargs):
    # The master solves once and broadcasts. Workers skip this.
    if not isinstance(environment.runner, WorkerRunner):
        environment.runner.send_message(
            "captcha_token", fresh_token(SITEKEY, PAGE_URL)
        )

def take_token(environment, msg, **kwargs):
    environment.shared_token = msg.data

Two things about that block. The handler signature is fixed: it takes the environment, the message and keyword arguments, and the payload arrives as msg.data. And if a handler is going to take a while, register it with concurrent set to True so it runs in its own greenlet instead of blocking Locust’s heartbeat and other system messages. A handler that only stores a string does not need that. A handler that solves inside itself does.

Read the next section before building any of it, because one solve per run is usually the wrong target anyway.

Step 4: a token does not survive a long test

A reCAPTCHA token is good for about two minutes. A load test is usually longer than two minutes. So the tidy architecture, one solve at test start broadcast to every worker, produces a run where the first couple of minutes pass and everything after that fails on an expired token, with the failures attributed to your application.

That failure is worth recognising on sight, and it is the same one that catches people chaining queue jobs. It is worked through in the guide to reCAPTCHA token expiration.

So use the shared-token pattern only when the test is short, or when the token is needed once during ramp-up rather than on every iteration. Otherwise solve per user in on_start, and for a soak test that runs for hours, refresh on a schedule inside the user.

import time

class Signup(HttpUser):
    wait_time = between(1, 3)

    def on_start(self):
        self.token = fresh_token(SITEKEY, PAGE_URL)
        self.solved_at = time.monotonic()

    @task
    def submit(self):
        # Refresh before the token ages out, not after it fails.
        if time.monotonic() - self.solved_at > 90:
            self.token = fresh_token(SITEKEY, PAGE_URL)
            self.solved_at = time.monotonic()
        self.client.post("/signup", data={
            "g-recaptcha-response": self.token,
        })

Ninety seconds rather than a hundred and twenty, so the refresh happens while the token is still valid.

Step 5: Locust is gevent, so use the synchronous client

Locust runs every user inside its own greenlet and is event-based, using gevent. Its documentation makes the point that this is what lets you write your tests as normal blocking Python code rather than with callbacks. Blocking is the native style here, so the plain CapSkip client is the one to reach for.

Do not reach for AsyncCapSkip in a locustfile. In Python it is a genuine async implementation rather than an alias, which makes it the right client in an asyncio program, and Locust is not one. There is no event loop waiting for it, and starting one per user inside a greenlet is a lot of machinery to get back what gevent already does.

The polling behaviour helps here too. The client does not poll on a flat interval. It starts at a quarter of a second and backs off towards pollingInterval, so a fast solve returns fast instead of waiting out a fixed delay. Across five hundred greenlets ramping up together, that difference is most of your ramp. If you do have a batch of CAPTCHAs to solve at once in an asyncio program elsewhere, that case is covered in solving CAPTCHAs in parallel in Python.

Step 6: running the solver where the workers can reach it

Load generators rarely sit on the same machine as anything else. They get their own box, or several, or a pool of containers, precisely so the load is real. CapSkip runs on Windows, and your workers may not.

There are two connection modes. Local binds to 127.0.0.1 and answers that device only. Server binds to your network address or public IP, so another box, a container host or a hosted runner can reach the same Windows machine over the API. Both live under connection settings, and Server mode changes only which address the solver listens on. It is still your hardware and it is still unmetered.

Read the address from the environment rather than from the locustfile. The Python client does not read CAPSKIP_HOST, CAPSKIP_PORT or CAPSKIP_API_KEY by itself, which is why the solver above reads them and passes them in, so the same file works unchanged on your laptop and on a worker fleet.

Common errors and what they mean

That last one has a guide of its own, because the same response covers a key that is missing and a key that is merely wrong: how to fix ERROR_WRONG_USER_KEY.

FAQ

The short version

Solve with the CapSkip client so the request never reaches self.client and never reaches your statistics. Put the call in on_start for one token per simulated user, and refresh it inside the task if the run lasts longer than about ninety seconds. If you do solve in a test_start listener, guard it with a WorkerRunner check, because that event fires on every node. Stay on the synchronous client, because Locust’s concurrency is gevent. Run CapSkip in Server mode whenever the load generators are not on the solver’s own machine.

• The client surface and every CAPTCHA type it covers are on the Python CAPTCHA solver page.
• The checkbox challenge itself is explained on the reCAPTCHA v2 solver page.

One thing to settle before you size the ramp: CapSkip is an unlimited captcha solver running on hardware you already own, so a thousand simulated users each solving their own challenge costs exactly what ten of them cost.