How to Handle CAPTCHA in Crawlee with the Node.js SDK

Crawlee has no CAPTCHA hook, and it does not need one. A Crawlee captcha is solved inside your requestHandler, in the middle of the request you already have a browser page for. Three things make it work: detect the widget before you spend a solve on it, raise the handler timeout because the default is shorter than a reCAPTCHA solve, and throw on failure so Crawlee retries the request through its own queue instead of your loop. This guide shows all three against PlaywrightCrawler.

What you need

• Node.js 18 or newer, and a Crawlee project already crawling something
• CapSkip running and reachable. Local mode listens on 127.0.0.1 port 8080 for automation on the same machine, and Server mode listens on your network or public IP so a crawler on another box, a VPS or a container host can call it. Both are in the connection settings
• The three packages, installed together

# One install for the crawler, the browser and the solver client.
npm install crawlee playwright capskip

# Crawlee drives a real browser, so fetch one.
npx playwright install chromium

Samples here are CommonJS, which is the form the CapSkip README documents. Crawlee 3 ships both builds, so an ESM project can use import statements for the crawler instead.

Where the solve goes: inside requestHandler

Scrapy has downloader middleware and Selenium has whatever wrapper you built. Crawlee gives you the page object directly, so there is no interception layer to write. You detect the challenge, solve it, and carry on in the same function.

Detect first. Firing a solve at every page burns capacity on pages that were never challenged, and it hides the useful signal of how often you actually get blocked.

// npm install crawlee playwright capskip
const { PlaywrightCrawler } = require('crawlee');
const { CapSkip } = require('capskip');

// Local mode. Point host at a server IP to share one solver.
const solver = new CapSkip({ host: '127.0.0.1', port: 8080 });

async function solveIfChallenged(page, url, log) {
  const widget = page.locator('[data-sitekey]').first();
  if ((await widget.count()) === 0) return false;

  const sitekey = await widget.getAttribute('data-sitekey');
  log.info(`Solving sitekey ${sitekey}`);
  const result = await solver.recaptcha(sitekey, url);
  return result.code;   // the token
}

The data-sitekey attribute is on the widget div for reCAPTCHA v2 and on the Turnstile div too, which is why one selector covers both. reCAPTCHA v3 has no visible widget, so you read the key out of the script URL instead.

Inject the token, then submit

Solving gives you a token. The page still expects that token in the hidden field its own widget would have filled, so put it there and submit the form the way a browser would.

// The widget writes into a hidden textarea. Do the same.
await page.evaluate((token) => {
  const field = document.getElementById('g-recaptcha-response');
  field.value = token;
}, token);

// Then submit exactly as the page would, and wait for the result.
await Promise.all([
  page.waitForNavigation(),
  page.click('button[type=submit]'),
]);

Some pages call a JavaScript callback instead of posting a form. If the widget div carries a data-callback attribute, invoke that function with the token rather than clicking anything, because the click handler may never run.

Raise requestHandlerTimeoutSecs before anything else

This is the one that catches people, and it looks like a solver problem when it is not.

PlaywrightCrawler gives each request handler 60 seconds by default. A reCAPTCHA v2 job is not ready for the first 15 to 20 seconds, v3 takes 10 to 15, and that is before you have loaded the page, injected the token and waited for a navigation. The handler gets killed mid-solve, Crawlee logs a timeout, and the request goes back on the queue to do it all again.

// npm install crawlee playwright capskip
const crawler = new PlaywrightCrawler({
  // 60 is the default and it is shorter than a v2 solve plus a submit.
  requestHandlerTimeoutSecs: 180,

  // Three tries per URL, which is Crawlee's default and the right one.
  maxRequestRetries: 3,

  async requestHandler({ page, request, log }) {
    // your handler
  },
});

180 seconds is a sensible ceiling. It is about ten times a normal solve, and it sits deliberately below the SDK’s own 300 second reCAPTCHA polling limit, so Crawlee gives up on a genuinely stuck request instead of letting it hold a browser slot for a full five minutes. If you would rather the solver client be the one that gives up first, lower recaptchaTimeout to something under your handler timeout.

Full working example

One file, one crawler, one solve path. Drop your own start URL in the run call.

// npm install crawlee playwright capskip
const { PlaywrightCrawler, Dataset } = require('crawlee');
const { CapSkip } = require('capskip');

const solver = new CapSkip({ host: '127.0.0.1', port: 8080 });

const crawler = new PlaywrightCrawler({
  requestHandlerTimeoutSecs: 180,
  maxRequestRetries: 3,

  async requestHandler({ page, request, log }) {
    const widget = page.locator('[data-sitekey]').first();

    if ((await widget.count()) > 0) {
      const sitekey = await widget.getAttribute('data-sitekey');
      const result = await solver.recaptcha(sitekey, request.loadedUrl);

      await page.evaluate((token) => {
        document.getElementById('g-recaptcha-response').value = token;
      }, result.code);

      await Promise.all([
        page.waitForNavigation(),
        page.click('button[type=submit]'),
      ]);
      log.info(`Cleared the challenge on ${request.loadedUrl}`);
    }

    await Dataset.pushData({ url: request.loadedUrl, title: await page.title() });
  },
});

await crawler.run(['https://example.com/page-with-recaptcha']);

The solve runs on your own machine, so the retry budget above costs nothing but wall-clock time. That is the practical difference from a metered service, where three attempts per URL is a line item.

Let the queue retry, do not build your own loop

The instinct is to wrap the solve in a for loop. Do not. Crawlee already has a retry system that knows about the request queue, session pool and proxy configuration, and a hand-rolled loop inside the handler is invisible to all three.

Throw instead. A handler that throws sends the request back to the queue, and Crawlee retries it up to maxRequestRetries times with a fresh browser context.

// npm install capskip
const { ApiException, NetworkException, TimeoutException } = require('capskip');

const crawler = new PlaywrightCrawler({
  requestHandlerTimeoutSecs: 180,

  // Runs between retries, while attempts remain.
  errorHandler({ request, log }, error) {
    log.warning(`Retry ${request.retryCount} for ${request.url}: ${error.message}`);
  },

  // Runs once, after the last attempt fails.
  failedRequestHandler({ request, log }) {
    log.error(`Gave up on ${request.url}`);
  },
});

Which exception came out tells you what to change. A NetworkException means CapSkip was not reachable, so check the host and port before blaming the site. A TimeoutException means the polling window expired and the page is probably serving a harder challenge than you think. An ApiException carries a returned error code, and that is the one worth logging with the URL attached.

Running the crawler and the solver on different machines

Crawlee scales by running more of itself, and a crawl fleet on separate boxes cannot all talk to 127.0.0.1. Server mode is the answer: CapSkip listens on your network or public IP instead of loopback, and every worker points at the same address.

// npm install capskip
const { CapSkip } = require('capskip');

// Same client, different address. Nothing else in the code changes.
const solver = new CapSkip({
  host: process.env.CAPSKIP_HOST || '127.0.0.1',
  port: Number(process.env.CAPSKIP_PORT || 8080),
});

The SDK reads CAPSKIP_HOST and CAPSKIP_PORT from the environment on its own, so the fallback above is belt and braces for a container that starts without them. A static public IP is recommended for the solver box, and the setup steps are in the connection settings. It is still your hardware and still unmetered, so the only thing that changed is where the process runs.

Common errors and what they mean

Parameter names and the full error code list are in the API documentation.

FAQ

The short version

Detect the widget, solve in the request handler, raise the handler timeout to 180 seconds, and throw so the queue retries. Running the solver yourself is what makes retry-three-times a reasonable default rather than a cost decision, which is the same argument for using a local captcha bypass anywhere in a crawl. The Node.js integration guide covers the client setup, the Playwright guide has the browser-side details Crawlee inherits, and CAPTCHA solving for web scraping covers session handling across a whole crawl. For the same pattern in Python, see the Scrapy middleware post.