How to Solve CAPTCHAs in RQ Jobs Past the 180-Second Limit

To solve CAPTCHAs in RQ, the Python job queue built on Redis, put the whole sequence in one job: fetch the page, solve the CAPTCHA with CapSkip, post the form. An RQ CAPTCHA job then needs one setting changed, its timeout. RQ gives every job 180 seconds unless you say otherwise. The CapSkip SDK waits up to 300 seconds for a reCAPTCHA token, so a solve that queues behind busy threads is killed in the middle of polling. RQ records a JobTimeoutException, and CapSkip finishes the solve for nobody. This guide covers the job, the timeout, the workers, retries and where the worker can run, Windows included.

What you need

• CapSkip running on a Windows machine. The samples solve reCAPTCHA v2, and the same job shape works for every type CapSkip supports.
• Python 3.10 or later with RQ 2.x, the capskip package and requests. The samples were run on RQ 2.12.
• A Redis server, version 5 or later, that the code enqueuing jobs and the workers can both reach. RQ also accepts Valkey 7.2 or later.
• A machine for the workers. RQ’s default worker relies on a Unix call that Windows does not have, so a section below covers the options on Windows, under WSL and on another machine.
• An address for the solver. In Local mode CapSkip answers on 127.0.0.1 for that device only. In Server mode it listens on your network address or public IP, so a worker on another machine can call it over the API. Both are set under connection settings.

pip install rq capskip requests

Step 1: write the job in its own module

RQ stores a job as the import path of a function plus its arguments, and the worker imports that function when it picks the job up. A function defined in the script that enqueues it lives in __main__, which the worker cannot import, so RQ refuses it at enqueue time with a ValueError. Put the job in a module of its own:

# tasks.py: the worker imports jobs by module path, so keep them here
# pip install rq capskip requests
import os
from html.parser import HTMLParser

import requests
from capskip import CapSkip

solver = CapSkip(
    apiKey=os.getenv("CAPSKIP_API_KEY", "capskip"),
    host=os.getenv("CAPSKIP_HOST", "127.0.0.1"),
    port=int(os.getenv("CAPSKIP_PORT", "8080")),
)


class FormPage(HTMLParser):
    """Collects the reCAPTCHA sitekey and the hidden fields of its form."""

    def __init__(self):
        super().__init__()
        self.sitekey, self.hidden, self._form = None, {}, None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "form":
            self._form = {}
        if a.get("data-sitekey"):
            self.sitekey = a["data-sitekey"]
            if self._form is not None:
                self.hidden = self._form
        hidden = (a.get("type") or "").lower() == "hidden"
        if tag == "input" and hidden and a.get("name") and self._form is not None:
            self._form[a["name"]] = a.get("value") or ""

    def handle_endtag(self, tag):
        if tag == "form":
            self._form = None


def sign_up(page_url, username):
    # One session per job, so the page's cookies go back with the form.
    session = requests.Session()
    page = FormPage()
    page.feed(session.get(page_url, timeout=30).text)

    # Solve inside the job, right before the post: tokens expire.
    token = solver.recaptcha(page.sitekey, page_url)["code"]

    resp = session.post(page_url, data={
        **page.hidden,
        "username": username,
        "g-recaptcha-response": token,
    }, headers={"Referer": page_url}, timeout=30)
    resp.raise_for_status()
    return resp.status_code

Each job opens its own requests.Session, so the session cookie the page sets goes back with the form. The parser keeps the hidden fields of the form that holds the widget, which include the CSRF token that cookie belongs to, and ignores a search or newsletter form elsewhere on the page. The sample posts back to the page URL; if the form’s action points elsewhere, post there. The Referer header is there because requests sends neither Referer nor Origin, and some frameworks, Django among them, refuse an HTTPS form post that has neither.

raise_for_status turns an HTTP error, such as a 403 from a failed CSRF check, into a failed job instead of a quiet success, which matters once retries are on. A site that rejects the token with a normal 200 page still counts as a success, so also check the response for what your site shows on success.

The solver client is created when the module is imported. With RQ’s default worker that happens in the fresh work horse of every job, and with SimpleWorker once per worker. Either is fine, because the client holds no connection open between calls. The SDK does not read environment variables by itself, which is why the code reads CAPSKIP_HOST, CAPSKIP_PORT and CAPSKIP_API_KEY and passes them to the constructor. Point a different worker at a different CapSkip machine by changing its environment, not its code.

Step 2: enqueue with a timeout that fits the solve

This is the setting that decides whether an RQ CAPTCHA job survives a slow solve. Pass job_timeout on every enqueue, or give the Queue object a default_timeout. That default lives in the Python object, not in Redis, so every script that enqueues has to build its Queue the same way:

# enqueue.py
from redis import Redis
from rq import Queue

from tasks import sign_up

# RQ's default is 180 seconds; a reCAPTCHA solve may take 300.
queue = Queue("captcha", connection=Redis(), default_timeout=420)

job = queue.enqueue(sign_up, "https://example.com/signup", "YOUR_USERNAME")
print(job.id)

The number comes from adding up the worst case. The SDK polls a reCAPTCHA solve for up to 300 seconds, its recaptchaTimeout, then raises TimeoutException. The page fetch and the post each pass timeout=30 to requests, which limits the connection and each wait for data rather than the whole request. Together that is about 360 seconds, and 420 leaves a minute of headroom for a slow page.

CapSkip keeps a clock of its own. In its reCAPTCHA settings a task may wait 250 seconds for one of its 10 threads, then has up to 250 seconds to load the challenge and 250 more to solve it. A task that runs out of any one of those is failed by CapSkip with an ApiException. One that waits a while for a thread and then solves slowly can pass the SDK’s 300 seconds first, which ends in TimeoutException. The SDK sets no timeout on its own HTTP calls to CapSkip, though. If CapSkip stops answering in the middle of a request, RQ’s default worker still ends the job at job_timeout; SimpleWorker, covered below, cannot.

Here is what happens with the default instead. At 180 seconds RQ raises JobTimeoutException inside the job, wherever it happens to be, which for a CAPTCHA job is nearly always the SDK’s poll loop. The job lands in the failed job registry with Task exceeded maximum timeout value (180 seconds) as its error. CapSkip is not told. It carries on, produces a token nobody collects, and the thread it used was spent for nothing. We reproduced exactly that against a solve that took 200 seconds.

Do not catch Exception around the solve

JobTimeoutException is an ordinary Exception subclass, raised into your code. A broad except Exception around the solve catches it, the job returns normally, and RQ files it as finished, with no trace of the timeout anywhere. In our test, a job written that way ran out its limit mid-solve and the worker logged Job OK. Catch the SDK’s own base class instead:

import logging

from capskip import CapSkipError

log = logging.getLogger(__name__)

try:
    token = solver.recaptcha(page.sitekey, page_url)["code"]
except CapSkipError as exc:
    # The SDK's errors only. RQ's JobTimeoutException is not one of
    # them, so a job that ran out of time still fails as it should.
    log.warning("solve failed for %s: %s", page_url, exc)
    raise

Step 3: run as many workers as CapSkip has threads

An RQ worker runs one job at a time, so for RQ CAPTCHA jobs the number of workers is the number of solves in flight. CapSkip solves 10 reCAPTCHAs at once by default, its Max. Threads setting, and makes any further task wait for a free thread. Workers beyond that number do not finish more work. Their tasks queue inside CapSkip instead, and the queueing eats into each job’s timeout. One worker per thread is the right size:

# Ten workers for CapSkip's ten reCAPTCHA threads
rq worker-pool captcha -n 10

worker-pool starts the workers and restarts any that die. Every worker it starts has the scheduler switched on, and one of them runs it for the queue, which the retries in the next step depend on. If you raise Max. Threads in CapSkip, raise the pool to match, and count every worker on every machine that calls the same CapSkip.

Step 4: retries, and why they need a scheduler

RQ retries nothing by default. A failed job goes to the failed job registry and stays there. To have RQ try again, pass a Retry when you enqueue:

from rq import Retry

job = queue.enqueue(
    sign_up, "https://example.com/signup", "YOUR_USERNAME",
    # Up to two more runs: 30 seconds, then 120 seconds after a failure.
    retry=Retry(max=2, interval=[30, 120]),
)

A retry with an interval is not put back on the queue. RQ schedules it, and only a worker running the scheduler ever moves it back. worker-pool always runs one. A single worker started with plain rq worker does not, and then every retried job waits in the scheduled registry forever without an error. We watched exactly that happen, and the same job ran within seconds once a worker with the scheduler started. Start single workers like this:

rq worker captcha --with-scheduler

A retry runs the whole job again: a fresh page, a fresh solve and a fresh post. That is what you want, because a reCAPTCHA token is accepted once, so there is nothing to resume. It is also why the job should end at the post. Anything that can fail after a successful post makes the retry post a second time.

Keep the solve and the post in the same job

It is tempting to split the work. One job solves and returns the token, and a second job, or the code that enqueued the first, posts it later. Do not. A reCAPTCHA token expires about two minutes after it is issued, and RQ makes no promise about when the next job starts. Under load it waits behind everything else on the queue. RQ also keeps a job’s return value for only 500 seconds by default. Solving inside the job, right before the post, keeps the token’s whole life inside one function call. How long a reCAPTCHA token stays valid covers the expiry rules in detail.

Where the RQ worker can run: Windows, WSL or another machine

RQ’s default worker forks a fresh process for every job. Windows has no os.fork, so there the worker stops with an AttributeError on its first job. RQ’s documentation says workers need fork and sends Windows users to WSL, although its page on worker classes offers SpawnWorker for Windows. On RQ 2.12 SpawnWorker fails too, because it waits for each job with os.wait4, which Windows does not have either. That leaves three routes.

On the CapSkip PC itself, use SimpleWorker. It runs each job inside the worker process instead of a child. On Windows with RQ 2.12 it ran our jobs and the scheduler, and worker-pool started several of them side by side. Treat that as tested here rather than supported upstream. It enforces job_timeout with a timer, but the timer only fires between Python statements. When we made a call to CapSkip stall for 40 seconds under an 8 second job_timeout, the job failed at 40 seconds, not 8. A call that never returns holds that worker until you restart it. CapSkip stays in Local mode, because the worker is on the same machine. Redis has no official Windows build, but a Redis running under WSL answers Windows programs on 127.0.0.1, which is how we tested this setup:

# On Windows: no fork, so run jobs inside the worker process
rq worker-pool captcha -n 10 -w rq.worker.SimpleWorker

Under WSL 2 on the same PC, treat the worker as another machine. In WSL 2’s default NAT networking, 127.0.0.1 inside Linux is the Linux VM, not Windows, so a worker there cannot reach CapSkip in Local mode. We checked, and the connection fails. Switch CapSkip to Server mode and point CAPSKIP_HOST at the Windows host’s address as Linux sees it. Microsoft’s guide to WSL networking shows how to find that address, and also describes mirrored networking on Windows 11 22H2 and later, where Linux can reach Windows servers on 127.0.0.1.

On a Linux server, a container host or a platform, use Server mode. CapSkip then listens on your network address or public IP, and every worker connects over the same API. Use a static public IP when the route crosses the internet and turn on API key validation. CapSkip opens its port to every address when Server mode starts, so to keep it to your workers, add a Windows Firewall block rule for the other addresses. The solver is still your own Windows machine, and solves are still unmetered, however many workers call it.

Full working example

Two files and one command. tasks.py holds the job, with the narrow exception handling from Step 2; enqueue.py sets the timeout and the retries.

# tasks.py
# pip install rq capskip requests
import logging
import os
from html.parser import HTMLParser

import requests
from capskip import CapSkip, CapSkipError

log = logging.getLogger(__name__)
solver = CapSkip(
    apiKey=os.getenv("CAPSKIP_API_KEY", "capskip"),
    host=os.getenv("CAPSKIP_HOST", "127.0.0.1"),
    port=int(os.getenv("CAPSKIP_PORT", "8080")),
)


class FormPage(HTMLParser):
    def __init__(self):
        super().__init__()
        self.sitekey, self.hidden, self._form = None, {}, None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "form":
            self._form = {}
        if a.get("data-sitekey"):
            self.sitekey = a["data-sitekey"]
            if self._form is not None:
                self.hidden = self._form
        hidden = (a.get("type") or "").lower() == "hidden"
        if tag == "input" and hidden and a.get("name") and self._form is not None:
            self._form[a["name"]] = a.get("value") or ""

    def handle_endtag(self, tag):
        if tag == "form":
            self._form = None


def sign_up(page_url, username):
    session = requests.Session()
    page = FormPage()
    page.feed(session.get(page_url, timeout=30).text)
    if not page.sitekey:
        raise ValueError(f"no data-sitekey on {page_url}")

    try:
        token = solver.recaptcha(page.sitekey, page_url)["code"]
    except CapSkipError as exc:
        # RQ's JobTimeoutException is not a CapSkipError, so it passes.
        log.warning("solve failed for %s: %s", page_url, exc)
        raise

    resp = session.post(page_url, data={
        **page.hidden,
        "username": username,
        "g-recaptcha-response": token,
    }, headers={"Referer": page_url}, timeout=30)
    resp.raise_for_status()
    return resp.status_code

# enqueue.py
from redis import Redis
from rq import Queue, Retry

from tasks import sign_up

queue = Queue("captcha", connection=Redis(), default_timeout=420)

for user in ["YOUR_USERNAME_1", "YOUR_USERNAME_2"]:
    job = queue.enqueue(
        sign_up, "https://example.com/signup", user,
        retry=Retry(max=2, interval=[30, 120]),
    )
    print(job.id)

# Linux or WSL; on Windows add -w rq.worker.SimpleWorker
rq worker-pool captcha -n 10

Run the pool from the folder that holds tasks.py, since that is where the workers import it from. rq info shows each queue’s waiting, executing, finished and failed counts, and the workers. It does not count the scheduled registry, where interval retries wait. A failed job keeps its traceback for a year by default: fetch the job with Job.fetch and read its exc_info.

Common errors and what they mean

FAQ

The short version

For an RQ CAPTCHA job, put the fetch, the solve and the post in one job function in its own module. Enqueue it with a job_timeout of 420 seconds, or give the queue that default, because RQ’s own 180 seconds is shorter than a reCAPTCHA solve can take. Catch CapSkipError around the solve, never Exception. Run as many workers as CapSkip has threads with worker-pool, which also runs the scheduler that interval retries need. On Windows use SimpleWorker; from WSL or another machine, switch CapSkip to Server mode.

• Every CAPTCHA type the Python package handles: the Python CAPTCHA solver page.
• How the reCAPTCHA v2 solve itself works: the reCAPTCHA v2 solver page.

Ten workers solving around the clock is exactly the load a metered service bills by the thousand. Running on your own machine, a captcha solver takes it without a per-solve bill.