{"id":25286,"date":"2026-08-24T04:33:57","date_gmt":"2026-08-24T04:33:57","guid":{"rendered":"https:\/\/capskip.com\/?p=25286"},"modified":"2026-08-24T04:33:57","modified_gmt":"2026-08-24T04:33:57","slug":"crawlee-captcha","status":"publish","type":"post","link":"https:\/\/capskip.com\/zh\/crawlee-captcha\/","title":{"rendered":"\u5982\u4f55\u5728 Crawlee \u4e2d\u7528 Node.js SDK \u5904\u7406\u9a8c\u8bc1\u7801"},"content":{"rendered":"<p>Crawlee has no CAPTCHA hook, and it does not need one. A Crawlee captcha is solved inside your <code>requestHandler<\/code>, in the middle of the request you already have a browser page for. Three things make it work: detect the widget before you spend a solve on it, raise the handler timeout because the default is shorter than a reCAPTCHA solve, and throw on failure so Crawlee retries the request through its own queue instead of your loop. This guide shows all three against PlaywrightCrawler.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">What you need<\/h2>\n<ul>\n<li>Node.js 18 or newer, and a Crawlee project already crawling something<\/li>\n<li>CapSkip running and reachable. Local mode listens on 127.0.0.1 port 8080 for automation on the same machine, and Server mode listens on your network or public IP so a crawler on another box, a VPS or a container host can call it. Both are in the <a href=\"https:\/\/capskip.com\/setup-guide\/#connection-settings\">connection settings<\/a><\/li>\n<li>The three packages, installed together<\/li>\n<\/ul>\n<div data-no-translation>\n<pre data-enlighter-language=\"bash\" class=\"EnlighterJSRAW\"># One install for the crawler, the browser and the solver client.\nnpm install crawlee playwright capskip\n\n# Crawlee drives a real browser, so fetch one.\nnpx playwright install chromium<\/pre>\n<\/div>\n<p>Samples here are CommonJS, which is the form the CapSkip README documents. Crawlee 3 ships both builds, so an ESM project can use import statements for the crawler instead.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Where the solve goes: inside requestHandler<\/h2>\n<p>Scrapy has downloader middleware and Selenium has whatever wrapper you built. Crawlee gives you the page object directly, so there is no interception layer to write. You detect the challenge, solve it, and carry on in the same function.<\/p>\n<p>Detect first. Firing a solve at every page burns capacity on pages that were never challenged, and it hides the useful signal of how often you actually get blocked.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"js\" class=\"EnlighterJSRAW\">\/\/ npm install crawlee playwright capskip\nconst { PlaywrightCrawler } = require('crawlee');\nconst { CapSkip } = require('capskip');\n\n\/\/ Local mode. Point host at a server IP to share one solver.\nconst solver = new CapSkip({ host: '127.0.0.1', port: 8080 });\n\nasync function solveIfChallenged(page, url, log) {\n  const widget = page.locator('[data-sitekey]').first();\n  if ((await widget.count()) === 0) return false;\n\n  const sitekey = await widget.getAttribute('data-sitekey');\n  log.info(`Solving sitekey ${sitekey}`);\n  const result = await solver.recaptcha(sitekey, url);\n  return result.code;   \/\/ the token\n}<\/pre>\n<\/div>\n<p>The data-sitekey attribute is on the widget div for reCAPTCHA v2 and on the Turnstile div too, which is why one selector covers both. reCAPTCHA v3 has no visible widget, so you read the key out of the script URL instead.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Inject the token, then submit<\/h2>\n<p>Solving gives you a token. The page still expects that token in the hidden field its own widget would have filled, so put it there and submit the form the way a browser would.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"js\" class=\"EnlighterJSRAW\">\/\/ The widget writes into a hidden textarea. Do the same.\nawait page.evaluate((token) =&gt; {\n  const field = document.getElementById('g-recaptcha-response');\n  field.value = token;\n}, token);\n\n\/\/ Then submit exactly as the page would, and wait for the result.\nawait Promise.all([\n  page.waitForNavigation(),\n  page.click('button[type=submit]'),\n]);<\/pre>\n<\/div>\n<p>Some pages call a JavaScript callback instead of posting a form. If the widget div carries a <code>data-callback<\/code> attribute, invoke that function with the token rather than clicking anything, because the click handler may never run.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Raise requestHandlerTimeoutSecs before anything else<\/h2>\n<p>This is the one that catches people, and it looks like a solver problem when it is not.<\/p>\n<p>PlaywrightCrawler gives each request handler <strong>60 seconds<\/strong> by default. A reCAPTCHA v2 job is not ready for the first 15 to 20 seconds, v3 takes 10 to 15, and that is before you have loaded the page, injected the token and waited for a navigation. The handler gets killed mid-solve, Crawlee logs a timeout, and the request goes back on the queue to do it all again.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"js\" class=\"EnlighterJSRAW\">\/\/ npm install crawlee playwright capskip\nconst crawler = new PlaywrightCrawler({\n  \/\/ 60 is the default and it is shorter than a v2 solve plus a submit.\n  requestHandlerTimeoutSecs: 180,\n\n  \/\/ Three tries per URL, which is Crawlee's default and the right one.\n  maxRequestRetries: 3,\n\n  async requestHandler({ page, request, log }) {\n    \/\/ your handler\n  },\n});<\/pre>\n<\/div>\n<p>180 seconds is a sensible ceiling. It is about ten times a normal solve, and it sits deliberately below the SDK&#8217;s own 300 second reCAPTCHA polling limit, so Crawlee gives up on a genuinely stuck request instead of letting it hold a browser slot for a full five minutes. If you would rather the solver client be the one that gives up first, lower recaptchaTimeout to something under your handler timeout.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Full working example<\/h2>\n<p>One file, one crawler, one solve path. Drop your own start URL in the run call.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"js\" class=\"EnlighterJSRAW\">\/\/ npm install crawlee playwright capskip\nconst { PlaywrightCrawler, Dataset } = require('crawlee');\nconst { CapSkip } = require('capskip');\n\nconst solver = new CapSkip({ host: '127.0.0.1', port: 8080 });\n\nconst crawler = new PlaywrightCrawler({\n  requestHandlerTimeoutSecs: 180,\n  maxRequestRetries: 3,\n\n  async requestHandler({ page, request, log }) {\n    const widget = page.locator('[data-sitekey]').first();\n\n    if ((await widget.count()) &gt; 0) {\n      const sitekey = await widget.getAttribute('data-sitekey');\n      const result = await solver.recaptcha(sitekey, request.loadedUrl);\n\n      await page.evaluate((token) =&gt; {\n        document.getElementById('g-recaptcha-response').value = token;\n      }, result.code);\n\n      await Promise.all([\n        page.waitForNavigation(),\n        page.click('button[type=submit]'),\n      ]);\n      log.info(`Cleared the challenge on ${request.loadedUrl}`);\n    }\n\n    await Dataset.pushData({ url: request.loadedUrl, title: await page.title() });\n  },\n});\n\nawait crawler.run(['https:\/\/example.com\/page-with-recaptcha']);<\/pre>\n<\/div>\n<p>The solve runs on your own machine, so the retry budget above costs nothing but wall-clock time. That is the practical difference from a metered service, where three attempts per URL is a line item.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Let the queue retry, do not build your own loop<\/h2>\n<p>The instinct is to wrap the solve in a for loop. Do not. Crawlee already has a retry system that knows about the request queue, session pool and proxy configuration, and a hand-rolled loop inside the handler is invisible to all three.<\/p>\n<p>Throw instead. A handler that throws sends the request back to the queue, and Crawlee retries it up to <code>maxRequestRetries<\/code> times with a fresh browser context.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"js\" class=\"EnlighterJSRAW\">\/\/ npm install capskip\nconst { ApiException, NetworkException, TimeoutException } = require('capskip');\n\nconst crawler = new PlaywrightCrawler({\n  requestHandlerTimeoutSecs: 180,\n\n  \/\/ Runs between retries, while attempts remain.\n  errorHandler({ request, log }, error) {\n    log.warning(`Retry ${request.retryCount} for ${request.url}: ${error.message}`);\n  },\n\n  \/\/ Runs once, after the last attempt fails.\n  failedRequestHandler({ request, log }) {\n    log.error(`Gave up on ${request.url}`);\n  },\n});<\/pre>\n<\/div>\n<p>Which exception came out tells you what to change. A NetworkException means CapSkip was not reachable, so check the host and port before blaming the site. A TimeoutException means the polling window expired and the page is probably serving a harder challenge than you think. An ApiException carries a returned error code, and that is the one worth logging with the URL attached.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Running the crawler and the solver on different machines<\/h2>\n<p>Crawlee scales by running more of itself, and a crawl fleet on separate boxes cannot all talk to 127.0.0.1. Server mode is the answer: CapSkip listens on your network or public IP instead of loopback, and every worker points at the same address.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"js\" class=\"EnlighterJSRAW\">\/\/ npm install capskip\nconst { CapSkip } = require('capskip');\n\n\/\/ Same client, different address. Nothing else in the code changes.\nconst solver = new CapSkip({\n  host: process.env.CAPSKIP_HOST || '127.0.0.1',\n  port: Number(process.env.CAPSKIP_PORT || 8080),\n});<\/pre>\n<\/div>\n<p>The SDK reads CAPSKIP_HOST and CAPSKIP_PORT from the environment on its own, so the fallback above is belt and braces for a container that starts without them. A static public IP is recommended for the solver box, and the setup steps are in the <a href=\"https:\/\/capskip.com\/setup-guide\/#connection-settings\">connection settings<\/a>. It is still your hardware and still unmetered, so the only thing that changed is where the process runs.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Common errors and what they mean<\/h2>\n<table>\n<thead>\n<tr>\n<th>Symptom<\/th>\n<th>Cause<\/th>\n<th>Fix<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>requestHandler timed out after 60s<\/td>\n<td>The default handler timeout is shorter than a solve<\/td>\n<td>Set requestHandlerTimeoutSecs to 180<\/td>\n<\/tr>\n<tr>\n<td>Solve succeeds, page still blocks<\/td>\n<td>The token went in but the form was never submitted<\/td>\n<td>Check for a data-callback attribute and call it<\/td>\n<\/tr>\n<tr>\n<td><code>ERROR_GOOGLEKEY<\/code><\/td>\n<td>The sitekey attribute was empty or read from the wrong element<\/td>\n<td>Log the value before solving; v3 keys live in the script URL<\/td>\n<\/tr>\n<tr>\n<td><code>ERROR_PAGEURL<\/code><\/td>\n<td>The handler passed a relative or redirected URL<\/td>\n<td>Use request.loadedUrl, which is the URL after redirects<\/td>\n<\/tr>\n<tr>\n<td>NetworkException on every request<\/td>\n<td>The crawler cannot reach the solver<\/td>\n<td>Local mode is loopback only; switch to Server mode for a remote worker<\/td>\n<\/tr>\n<tr>\n<td>Every URL retried three times, then dropped<\/td>\n<td>The handler throws before the solve<\/td>\n<td>Read the errorHandler log; the first failure is the real one<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Parameter names and the full error code list are in the <a href=\"https:\/\/capskip.com\/api-docs\/\">API documentation<\/a>.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">FAQ<\/h2>\n<details style=\"border:1px solid #e2e5ee;border-radius:10px;padding:14px 18px;margin:0 0 12px;\">\n<summary style=\"cursor:pointer;\">\n<h3 style=\"font-size:1.15rem;line-height:1.4;display:inline;margin:0;\">Does this work with CheerioCrawler?<\/h3>\n<\/summary>\n<p style=\"margin:12px 0 0;\">Partly. CheerioCrawler has no browser, so there is no page object and no way to run the widget&#8217;s own JavaScript. You can still parse the sitekey out of the HTML, solve it, and post the token with the form body yourself. That is enough for a plain form submit and not enough for anything that expects a callback. Use PlaywrightCrawler when a challenge is likely.<\/p>\n<\/details>\n<details style=\"border:1px solid #e2e5ee;border-radius:10px;padding:14px 18px;margin:0 0 12px;\">\n<summary style=\"cursor:pointer;\">\n<h3 style=\"font-size:1.15rem;line-height:1.4;display:inline;margin:0;\">Should I solve in a preNavigationHook instead?<\/h3>\n<\/summary>\n<p style=\"margin:12px 0 0;\">No. Pre-navigation hooks run before the page loads, so there is nothing to detect yet. Post-navigation hooks are closer, but the request handler is where you already have the page, the URL after redirects and the logger. Keep the solve there and keep the hooks for cookies and headers.<\/p>\n<\/details>\n<details style=\"border:1px solid #e2e5ee;border-radius:10px;padding:14px 18px;margin:0 0 12px;\">\n<summary style=\"cursor:pointer;\">\n<h3 style=\"font-size:1.15rem;line-height:1.4;display:inline;margin:0;\">Can the crawler run on a hosted platform while the solver stays at home?<\/h3>\n<\/summary>\n<p style=\"margin:12px 0 0;\">Yes, with CapSkip in Server mode. The crawler needs a route to the solver&#8217;s address, so a home connection needs a static public IP and an open port, and a VPS is the simpler option. The client code is identical either way: only the host value changes.<\/p>\n<\/details>\n<details style=\"border:1px solid #e2e5ee;border-radius:10px;padding:14px 18px;margin:0 0 12px;\">\n<summary style=\"cursor:pointer;\">\n<h3 style=\"font-size:1.15rem;line-height:1.4;display:inline;margin:0;\">How many concurrent solves can a crawl push?<\/h3>\n<\/summary>\n<p style=\"margin:12px 0 0;\">Crawlee autoscales its own concurrency, and each handler awaits its own solve independently, so there is no queue to configure on the client. The SDK starts polling at 250 milliseconds and backs off to the pollingInterval ceiling, which keeps a fast solve fast even when several are in flight. Match your Crawlee concurrency to what the target site tolerates, not to the solver.<\/p>\n<\/details>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">The short version<\/h2>\n<p>Detect the widget, solve in the request handler, raise the handler timeout to 180 seconds, and throw so the queue retries. Running the solver yourself is what makes retry-three-times a reasonable default rather than a cost decision, which is the same argument for using a local <a href=\"https:\/\/capskip.com\/\">captcha bypass<\/a> anywhere in a crawl. The <a href=\"https:\/\/capskip.com\/nodejs-captcha-solver\/\">Node.js integration guide<\/a> covers the client setup, <a href=\"https:\/\/capskip.com\/playwright-captcha-solver\/\">the Playwright guide<\/a> has the browser-side details Crawlee inherits, and <a href=\"https:\/\/capskip.com\/captcha-solver-for-web-scraping\/\">CAPTCHA solving for web scraping<\/a> covers session handling across a whole crawl. For the same pattern in Python, see <a href=\"https:\/\/capskip.com\/scrapy-captcha-middleware\/\">the Scrapy middleware post<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Crawlee \u6ca1\u6709\u9a8c\u8bc1\u7801\u8bc6\u522b\u7684\u94a9\u5b50\uff0c\u6240\u4ee5\u8bc6\u522b\u903b\u8f91\u8981\u5199\u5728 requestHandler \u91cc\u9762\u3002\u672c\u6587\u8bf4\u660e\u5177\u4f53\u653e\u5728\u54ea\u4e2a\u4f4d\u7f6e\u3001\u5bb9\u6613\u8e29\u5751\u7684\u8d85\u65f6\u8bbe\u7f6e\uff0c\u4ee5\u53ca\u5982\u4f55\u901a\u8fc7\u961f\u5217\u91cd\u8bd5\u3002<\/p>","protected":false},"author":1,"featured_media":25285,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"Crawlee CAPTCHA: Solve It in the Request Handler | CapSkip","rank_math_description":"A Crawlee captcha stops the whole crawl, not one page. Solve it inside requestHandler, raise requestHandlerTimeoutSecs, and let the queue do the retries.","rank_math_focus_keyword":"crawlee captcha","footnotes":""},"categories":[70],"tags":[],"class_list":["post-25286","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-captcha"],"_links":{"self":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts\/25286","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/comments?post=25286"}],"version-history":[{"count":2,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts\/25286\/revisions"}],"predecessor-version":[{"id":25290,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts\/25286\/revisions\/25290"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/media\/25285"}],"wp:attachment":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/media?parent=25286"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/categories?post=25286"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/tags?post=25286"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}