How to Solve CAPTCHAs in a Celery Task (Python Queue)

A Celery captcha task has one rule that shapes everything else: it must solve from scratch on every attempt. Celery gives you at-least-once delivery, so a task can run twice, and a CapSkip result can be read once. Store a captcha id and resume from it after a retry and you get nothing back. Solve, use the token and finish, all inside one task body.

What you need

• Celery 5 with a broker, Redis or RabbitMQ. The broker choice changes one setting later on.
• CapSkip running on a Windows machine, with the Python client installed in the worker image.
• Server mode, in almost every real deployment. Workers usually run in Linux containers, and the solver does not.
• The sitekey and page URL, passed in as task arguments rather than baked into the task.

# pip install capskip
pip install -U celery[redis] capskip

Step 1: the task

The whole solve is one call. The SDK submits, polls and returns the token, so there is no id to carry between steps and nothing to persist.

# pip install capskip
import os
from celery import Celery
from capskip import CapSkip, NetworkException

app = Celery("solves", broker="redis://redis:6379/0")

# CAPSKIP_HOST is the solver machine. Loopback only works if the
# worker runs on the same Windows box as CapSkip.
solver = CapSkip(
    host=os.environ.get("CAPSKIP_HOST", "127.0.0.1"),
    port=int(os.environ.get("CAPSKIP_PORT", "8080")),
    apiKey=os.environ.get("CAPSKIP_API_KEY", "capskip"),
)

@app.task(
    bind=True,
    autoretry_for=(NetworkException,),
    retry_backoff=True,
    max_retries=3,
    soft_time_limit=330,
    time_limit=360,
)
def solve_and_submit(self, sitekey, page_url):
    result = solver.recaptcha(sitekey=sitekey, url=page_url)
    return submit_form(page_url, result["code"])   # token, used here

Note what is not in there. No captcha id in the return value, no second task to submit the token, no result stored for later. The token is used in the same task that made it. The reason is timing rather than tidiness, and it is covered under retries below.

That call is reCAPTCHA v2. The other types CapSkip supports are the same shape: pass invisible or enterprise set to 1, or version set to v3 with an action, or call turnstile or geetest instead. The full surface is on the Python CAPTCHA solver page.

Step 2: two time limits, and where to put them

Celery has no time limit by default, on either setting. That is the wrong default for a task that waits on a network service, because a stuck solve occupies a worker slot forever.

Set both, and set them above what the SDK itself allows. CapSkip gives a reCAPTCHA, Turnstile or GeeTest solve three hundred seconds and an image CAPTCHA one hundred and twenty, both configurable on the client. If the soft limit fires first, Celery raises SoftTimeLimitExceeded inside your task and you lose the SDK’s own TimeoutException, which is the more useful signal because it tells you the solver was reached and did not finish.

If you run image CAPTCHAs and reCAPTCHA through the same worker, give them separate tasks with separate limits rather than one task with the higher pair. A three hundred second ceiling on a job that normally finishes in a second hides real failures for five minutes.

Step 3: the retry rule that is specific to solving

Celery’s automatic retries are exactly right for a solver that is restarting and exactly wrong for a token you already hold. The difference is worth being precise about.

With retry_backoff set to True the first retry waits one second, then two, then four, then eight, and jitter is on by default so the real delay is a random value up to that maximum. The cap is retry_backoff_max, which defaults to six hundred seconds. Compare that with a reCAPTCHA token, which is good for about two minutes.

So a task that solved successfully, stored the token, failed on the submit and then retried can be resumed after a delay several times longer than the token’s life. It will fail on a token that was perfectly valid when it was made, and the log will blame the target site. That failure mode is covered in the guide to reCAPTCHA token expiration.

The fix is the task shape above: solve inside the retry, not before it.

Restrict autoretry_for to the exceptions that describe a transport problem. NetworkException means CapSkip was not reachable, which is worth retrying. ApiException and ValidationException mean the request was wrong and will be wrong again. TimeoutException is a judgement call, and usually worth one retry rather than three.

Step 4: acks_late, and why a result can only be read once

By default Celery acknowledges a message just before running it, so a worker that dies mid-task loses the job. Turning on task_acks_late moves the acknowledgement to after the task finishes, so a crashed worker’s job is redelivered and runs again. That is usually what you want for work that costs money elsewhere, and it is what makes the read-once rule matter.

A CapSkip result is readable once. If the first attempt submitted the challenge, read the token and then crashed before acknowledging, the redelivered attempt cannot re-read that id. It has to submit again. The task above does exactly that, because it holds no state between attempts, and the cost of resolving is a few seconds of your own hardware rather than a second charge.

# Redelivery is safe here because the task resolves rather
# than resuming. Pair it with reject_on_worker_lost so a
# killed worker requeues instead of dropping the job.
app.conf.task_acks_late = True
app.conf.task_reject_on_worker_lost = True

# Long tasks and a prefetch of 4 means idle workers sit on
# queued jobs. Drop it to 1 for solve queues.
app.conf.worker_prefetch_multiplier = 1

That last one is the quiet performance bug in most solve queues. The default prefetch multiplier is four, so each worker process reserves four messages up front. With tasks that take milliseconds that is a win. With tasks that wait on a solve, three of those four sit reserved behind a job that is doing nothing but polling, while another worker has nothing to do.

Step 5: the broker setting that duplicates solves

If your broker is Redis, there is one more number. Redis has no native acknowledgement, so Celery emulates it with a visibility timeout: the number of seconds it waits for a worker to acknowledge a task before redelivering the message to another worker. It defaults to one hour, and it lives inside broker_transport_options rather than being a setting of its own.

One hour is comfortably above a three hundred second solve, so the default is safe. The trouble starts when someone lowers it to make failed jobs recover faster, because Celery’s own documentation warns that a task whose execution time exceeds the visibility timeout gets executed again, and again, in a loop. Every slow solve then runs at least twice.

# Keep this above your hard time limit, not near it.
# 3600 is the default and it is fine. If you must lower it,
# stay well clear of the 360 second time_limit above.
app.conf.broker_transport_options = {"visibility_timeout": 3600}

RabbitMQ acknowledges natively and has no equivalent setting, which is one reason to prefer it for queues full of slow tasks.

Step 6: running the solver somewhere the workers can reach

Celery workers usually run in Linux containers on a cluster. CapSkip runs on Windows. So in practice the worker and the solver are on different machines, and loopback is not the answer.

CapSkip has two connection modes for this. Local binds to 127.0.0.1 and serves that device only. Server binds to your network address or public IP, so another box, a container host or a hosted platform can reach the same Windows machine over the API. Both live under connection settings, and Server mode changes only which address the solver listens on. It is still your hardware and it is still unmetered.

That last point is what makes a queue firing constantly reasonable to run at all.

Read the host and key from the environment rather than the code. The Python client does not read CAPSKIP_HOST, CAPSKIP_PORT or CAPSKIP_API_KEY by itself, which is why the task in Step 1 reads them and passes them in. A worker container then needs those variables and nothing else.

Solving many at once

Two ways, and they suit different shapes of work. One task per CAPTCHA, with worker concurrency doing the parallelism, is the normal answer and it is what the prefetch note above is about. For a batch that arrives together, Python’s client has a genuine async implementation, so one task can gather a batch inside itself.

import asyncio
import os
from capskip import AsyncCapSkip

# AsyncCapSkip in Python is a real async client, not an alias.
async def solve_batch(pairs):
    solver = AsyncCapSkip(
        host=os.environ.get("CAPSKIP_HOST", "127.0.0.1"), port=8080)
    return await asyncio.gather(*[
        solver.recaptcha(sitekey=k, url=u) for k, u in pairs
    ])

@app.task(soft_time_limit=330, time_limit=360)
def solve_many(pairs):
    return [r["code"] for r in asyncio.run(solve_batch(pairs))]

Worth knowing before you copy that into another language: the Node and .NET clients name a class AsyncCapSkip too, but there it is an alias rather than a second implementation. Python is the one where it means something. The full comparison is in the guide to solving CAPTCHAs in parallel in Python.

Common errors and what they mean

The key errors are worth reading about on their own, because the same response covers a key that is missing and a key that is simply wrong: how to fix ERROR_WRONG_USER_KEY.

FAQ

The short version

Put the solve and whatever uses the token in one task. Set a soft limit of 330 and a hard limit of 360 so the client’s own timeout reports first. Retry only on NetworkException, and let the retry resolve rather than resume, because a backoff can outlive a token by a long way and a result can be read once. Turn on late acknowledgement, drop the prefetch multiplier to 1, and keep the Redis visibility timeout far above the hard limit. Run CapSkip in Server mode whenever the workers are not on the solver’s own machine.

• The reCAPTCHA v2 checkbox itself is covered on the reCAPTCHA v2 solver page.
• The raw endpoints behind the client are documented in the CapSkip API documentation.

One last thing to weigh before you size the queue: CapSkip is a captcha solver that runs on hardware you already own, so a hundred workers hammering it and one worker trickling through the same jobs cost the same nothing.