Dramatiq CAPTCHA Tasks: Set Retries and Time Limits Right

dramatiq captcha - Dramatiq CAPTCHA Tasks: Set Retries and Time Limits Right

A dramatiq captcha actor goes wrong in three places, and all three come from how Dramatiq counts. Its time options are in milliseconds, its time limit cannot interrupt a socket that is waiting, and its retry rule counts every exception that leaves the actor. This guide puts the page fetch, the CapSkip solve and the form submission in one actor, then measures where the limit stops it and which failures deserve a retry. It runs on Windows with the StubBroker, so you need no Redis or RabbitMQ.

What you need

  • A Windows PC running CapSkip with the reCAPTCHA solver on. The sample solves reCAPTCHA v2; the same shape fits the other types on the Python CAPTCHA solver page.
  • Python 3.10 or later, the minimum that dramatiq 2.2.1 and capskip 1.7.0 declare. I ran everything on Python 3.11.7 and 3.14.0 with requests 2.34.2, and not on 3.10.
  • No message broker. The StubBroker keeps the queue inside one process. I used neither Redis nor RabbitMQ, so nothing here describes a real broker unless a sentence says it came from the source.
  • An address for CapSkip. On the same PC, the default 127.0.0.1 is enough. A worker on another PC needs Server mode, which makes CapSkip listen on your network or public IP, and you switch it on in the connection settings of the setup guide.

Testing: I wrote a mock CapSkip and a mock sign-up form, set the solve times of the mock myself, and used no real app or reCAPTCHA site. Every timing below is a mock timing. It shows how Dramatiq and the SDK behave around a solve, not how long a solve takes, and a failure the mock produces is my rule, not CapSkip behavior.

Install the three packages into a fresh virtual environment:

# Windows, in PowerShell or cmd
python -m venv .venv
.venv\Scripts\python -m pip install dramatiq capskip requests

Step 1: the broker, and an SDK that can time out

Put this block at the top of tasks.py. The StubBroker line is the only broker-specific part; a real queue would build a RedisBroker or a RabbitmqBroker there. The patch exists because capskip 1.7.0 calls requests.post and requests.get with no timeout; I read api.py in the installed package. If CapSkip accepts a connection and never answers, the call waits as long as the operating system allows, and the time limit of Dramatiq can’t help. The patch swaps the requests reference inside capskip.api, an internal part of the SDK, and the block fails loudly at import if that reference disappears. The SDK ignores CAPSKIP_HOST, CAPSKIP_PORT and CAPSKIP_API_KEY unless your code reads them, so the block reads them and hands them to the constructor.

# tasks.py, part 1: put this at the top of tasks.py
import functools
import os
import types

import capskip.api
import dramatiq
import requests
from capskip import ApiException, CapSkip, NetworkException, TimeoutException
from dramatiq.brokers.stub import StubBroker
from dramatiq.middleware import TimeLimitExceeded

# The StubBroker keeps the queue inside this process, so it needs no Redis
# or RabbitMQ. In production, build a RedisBroker or a RabbitmqBroker here
# and pass it to dramatiq.set_broker the same way.
broker = StubBroker()
dramatiq.set_broker(broker)

# capskip 1.7.0 calls requests.post and requests.get with no timeout, and
# time_limit cannot interrupt a socket that is waiting. Give every SDK call
# 5 seconds to connect and 30 seconds of silence on a read.
if not hasattr(capskip.api, "requests"):
    raise RuntimeError("capskip changed how it makes HTTP calls: recheck this patch")
capskip.api.requests = types.SimpleNamespace(
    RequestException=requests.RequestException,
    post=functools.partial(requests.post, timeout=(5, 30)),
    get=functools.partial(requests.get, timeout=(5, 30)),
)

# The SDK does not read these variables itself; this code passes them in.
solver = CapSkip(
    apiKey=os.environ.get("CAPSKIP_API_KEY", "capskip"),
    host=os.environ.get("CAPSKIP_HOST", "127.0.0.1"),
    port=int(os.environ.get("CAPSKIP_PORT", "8080")),
)

Step 2: the actor, with every time in milliseconds

The dramatiq captcha actor loads the page for its cookies, solves, and submits the form in one attempt. A CapSkip result is read once and a token goes stale after about two minutes (how long a reCAPTCHA token stays valid), so a retry must start from the page.

Every time option is in milliseconds: min_backoff here is 15 seconds and time_limit is 8 minutes. The 8 minutes is arithmetic: 300 seconds of SDK polling, a sleep of up to 5, and four HTTP calls that can each wait 5 seconds to connect and 30 seconds of silence on a read, about 445 seconds when CapSkip goes quiet.

worth_retrying runs after every failed attempt. A failed solve arrives as an ApiException containing ERROR_CAPTCHA_UNSOLVABLE (what that error means) and may work next time, as may an unreachable CapSkip, the timeout of the SDK, the limit of Dramatiq, and a 403, 408, 429 or 5xx answer from the form. A missing or malformed sitekey or page URL is refused when the task is submitted, with an ERROR_ code that does not contain UNSOLVABLE, and a wrong API key is refused the same way, so those end the message; so does a 4xx answer from the form other than the three above, which would come again, and so does a bug in your code. A well-formed but wrong sitekey may still arrive as ERROR_CAPTCHA_UNSOLVABLE and use all three retries; I could not test that against the real app.

# tasks.py, part 2: add this below part 1
MAX_RETRIES = 3


def worth_retrying(retries_so_far, exc):
    # Dramatiq calls this after every failed attempt. While retry_when is
    # set, max_retries is ignored, so the count lives here.
    if retries_so_far >= MAX_RETRIES:
        return False
    if isinstance(exc, ApiException):
        # CapSkip reports a solve that failed this way. A refused sitekey,
        # page URL or API key would be refused again.
        return "ERROR_CAPTCHA_UNSOLVABLE" in str(exc)
    if isinstance(exc, requests.HTTPError):
        # 403 and 429 can mean a refused or expired token, 5xx a site hiccup;
        # a 404 or 400 would be answered the same way next time.
        status = exc.response.status_code if exc.response is not None else 0
        return status in (403, 408, 429) or status >= 500
    transient = (NetworkException, TimeoutException, TimeLimitExceeded, requests.RequestException)
    return isinstance(exc, transient)


@dramatiq.actor(
    queue_name="captcha",
    retry_when=worth_retrying,
    min_backoff=15_000,  # milliseconds, not seconds
    max_backoff=120_000,
    time_limit=480_000,  # milliseconds as well: 8 minutes
)
def sign_up(sitekey, page_url, email):
    with requests.Session() as session:
        session.get(page_url, timeout=(5, 30))  # picks up the page's cookies
        # Solve inside the attempt, right before the form submission: a result is
        # read once, and a token ages.
        token = solver.recaptcha(sitekey=sitekey, url=page_url)["code"]
        reply = session.post(
            page_url,
            data={"email": email, "g-recaptcha-response": token},
            headers={"Referer": page_url},
            timeout=(5, 30),
        )
        reply.raise_for_status()

Step 3: run it with real worker threads and no broker

This script starts four Dramatiq worker threads in one process, sends a message and waits for the queue to drain. Save it as run_local.py next to tasks.py, replace YOUR_SITEKEY with the sitekey of a page you may automate, and pass the address of that page in PAGE_URL when you run it.

# run_local.py: real worker threads, no Redis or RabbitMQ
import os

from dramatiq import Worker

from tasks import broker, sign_up

# The dramatiq command emits this event for you. It starts the thread
# behind time_limit, so a script has to emit it itself.
broker.emit_after("process_boot")
worker = Worker(broker, worker_threads=4)
worker.start()
try:
    sign_up.send(
        "YOUR_SITEKEY",
        os.environ.get("PAGE_URL", "https://example.com/signup"),
        "[email protected]",
    )
    # fail_fast=False: do not re-raise an actor's exception out of join
    broker.join("captcha", fail_fast=False)
    worker.join()
finally:
    worker.stop()
print("dead-lettered messages:", len(broker.dead_letters))

From the folder that holds tasks.py, in PowerShell:

# PowerShell
$env:PAGE_URL = "https://your-page/signup"
.venv\Scripts\python run_local.py

In cmd, run set PAGE_URL=https://your-page/signup and then the same python command.

The emit_after line starts the watcher thread behind time_limit. The dramatiq command does that for you and a script must do it itself; without it a 2 second limit on a 6 second solve let the actor run 7.8 to 7.9 seconds. The join method of the broker in Dramatiq 2 also re-raises the exception of a dead-lettered message unless you pass fail_fast=False. Against the mock, one message ran, the form accepted one submission and the script printed 0 dead-lettered messages.

What time_limit can and cannot interrupt

The TimeLimit middleware wakes a watcher thread once a second. For a message past its limit, the watcher asks CPython to raise TimeLimitExceeded in the worker thread, and it arrives the next time that thread runs Python code. A sleeping thread, or one waiting on a socket, is running none. The Dramatiq user guide gives the default of 10 minutes, which already clears the arithmetic in Step 2.

The SDK sleeps between status polls, up to 5 seconds, so a limit that fires mid-sleep takes effect only when the sleep ends (first two rows below). A silent CapSkip is worse (rows three and four). With the SDK as shipped the actor ran on until the stuck call returned, and only the Step 1 patch bounded it, at the 30 second read timeout. So time_limit holds only if every blocking call has a timeout of its own.

TimeLimitExceeded derives from BaseException, so a handler for Exception does not catch it. A handler for BaseException does, and Dramatiq then counts a success (last row). Catch CapSkipError to log a failed solve.

Mock setupWhat happened
CapSkip never finishes, limit 3 sStopped at 3.8 s or at 7.8 s, depending on where the one second tick of the watcher fell against the SDK sleeps (five to one in the runs I did across two Pythons)
CapSkip never finishes, limit 8 sStopped at about 12.9 s on both Pythons
Status call silent for 25 s, no patch, limit 3 sLimit logged 3 to 4 s in, actor ran on to 25.0 s and never used the token in that reply
Status call silent for 45 s, Step 1 patch, limit 3 sStopped at 30.0 s
Limit written as 330 (seconds habit), 6 s solveEach attempt stopped within 2 s; four solves wasted, then a dead letter
Handler for BaseException around the solveMessage counted as a success, no retry, no dead letter

Which failures Dramatiq retries

The Retries middleware is on by default and its defaults are generous for a CAPTCHA. I read them from a running broker: 20 retries, a minimum backoff of 15 seconds, which makes the first retry wait 15 to 30 seconds, and a cap of 7 days. An actor with no options would retry a refused sitekey up to 20 times.

The max_retries option counts retries, not attempts, and every exception that leaves the actor counts, TimeLimitExceeded included. Passing retry_when takes over completely, which is why worth_retrying keeps its own count. The actor option "throws" skips retries by class and logs an info line, but ApiException covers both a failed solve and a refused key, so the sample uses retry_when. When worth_retrying says no, the log still reads "Retries exceeded for message"; read the exception line above it.

Each delay is min_backoff times 2 to the power of the retries so far, times a random factor from 1 to 2 and, if that passes max_backoff, drawn again between half and all of max_backoff. With 300 and 2000 the ranges are 300 to 600, 600 to 1200 and 1000 to 2000 milliseconds, and the delays I saw were 453, 896 and 1599 on Python 3.11 and 477, 1028 and 1219 on 3.14. Every retry is a fresh solve, and the abandoned attempt leaves a task in CapSkip that nobody reads: in the 330 row, four tasks were submitted and no token was read.

Mock setupWhat happened
Sample actor, every solve fails4 solves, 3 retry lines, then "Retries exceeded" and a dead letter
Sample actor, mock refuses the sitekey with an ERROR_ code (the mock’s rule)1 solve, no retry, dead-lettered
Sample actor, form answers 404, 405 or 400 to every submission1 solve each time, no retry, dead-lettered
Plain actor, max_retries of 3, raising the refused-key error4 runs of a request that could never work
Site answers 403 to the first form submission2 solves, second token accepted
Plain actor, throws set to ValidationException, raising it1 run, an info line, dead-lettered

Workers on Windows: threads, stops and Server mode

The dramatiq command loads the module you name, so the broker line in tasks.py decides what a worker consumes. With the StubBroker a started worker just idles, since each process has its own queue; I used it only to check the command. By default it starts one process per logical processor with 8 threads each. This PC has 24, and the default printed 24 lines saying "Worker process is ready for action": 192 threads that can each start a solve. CapSkip is set to run 10 reCAPTCHA solves at once by default (the Max. Threads setting; I did not run the app, so check your own value), and a task waiting inside CapSkip still uses up the time limit of the actor. Start with one process and ten threads:

# From the folder that holds tasks.py: one process, ten threads
.venv\Scripts\dramatiq tasks --processes 1 --threads 10

Plan for the stop. I started the command with a slow mock solve and sent Ctrl+Break to it. The command exited under 2 seconds later with code 21. Six stops on the two Pythons gave the same result, two of them after the solve had finished, and the four unfinished solves were never read. The main process forwards the signal to its workers with os.kill, and CPython documents that on Windows os.kill ends a process outright for any signal number except the two console events, using the number as the exit code. 21 is that number. I did not test Ctrl+C or a service stop, so treat every stop as a kill of the running solve.

Dramatiq acknowledges a message only after its actor finishes. In the source of the Redis broker, the unacknowledged messages of a dead worker go back onto the queue once its heartbeat is 60 seconds stale, the default. I read that and did not run it. Either way the message starts over from the page.

For workers on another PC, switch CapSkip to Server mode, then set CAPSKIP_HOST (and CAPSKIP_API_KEY if API key validation is on) in the environment of the dramatiq command. Turning on Server mode opens the CapSkip port to any address, so block every machine that is not one of your workers with a Windows Firewall block rule. The solves still happen on your own Windows machine and cost nothing per solve.

Common Dramatiq CAPTCHA errors

What you seeCauseFix
"Retries exceeded" after one attemptworth_retrying refused a malformed sitekey, page URL or API key, or a 4xx answer from the formRead the exception above it, fix the input
NetworkException on every attemptCapSkip is closed, or on 127.0.0.1 with the worker on another PCStart it, or use Server mode; see the API documentation

FAQ

Is the StubBroker enough to test a Dramatiq CAPTCHA actor?

For the logic of the actor itself, yes: retries, backoff, time limits and dead letters all ran here with real worker threads. It cannot show what a broker adds, such as redelivery after a worker dies.

Can workers on another machine reach CapSkip?

Yes, through Server mode. Run CapSkip on a Windows machine with a reachable address and set CAPSKIP_HOST in the environment of the workers. The actor is unchanged.

Does Dramatiq differ from Celery for this?

Dramatiq has one time limit, in milliseconds, whose error is a BaseException, and a retry rule that is a function you write. Late acknowledgement and prefetch are covered in the guide to solving CAPTCHAs in a Celery task.

The short version

For a dramatiq captcha actor, keep the fetch, the solve and the form submission in one attempt, write every time in milliseconds, and give the HTTP calls of the SDK a timeout so time_limit can reach the thread. Retry a failed solve, a silent CapSkip or a refused form, and expect a Windows stop to kill a running solve. With a local captcha solver on your own hardware, a retry or a killed solve costs time and nothing more.