{"id":27970,"date":"2026-11-14T03:34:54","date_gmt":"2026-11-14T03:34:54","guid":{"rendered":"https:\/\/capskip.com\/?p=27970"},"modified":"2026-11-14T03:34:54","modified_gmt":"2026-11-14T03:34:54","slug":"datadome-scrapy","status":"publish","type":"post","link":"https:\/\/capskip.com\/zh\/datadome-scrapy\/","title":{"rendered":"\u5982\u4f55\u7528\u4e2d\u95f4\u4ef6\u5904\u7406 Scrapy \u91cc\u7684 DataDome 403 \u62e6\u622a"},"content":{"rendered":"<p>To handle DataDome in Scrapy, put the work in one downloader middleware. It spots the 403 block page, gets a datadome cookie from CapSkip with the datadome method of the Python SDK, and retries with that cookie, its User-Agent and the same proxy. A DataDome Scrapy setup lives or dies on the priority number, the duplicate filter, a blocking SDK call and a cookie jar per proxy. This guide wires the <a href=\"https:\/\/capskip.com\/datadome-solver\/\">CapSkip DataDome solver<\/a> into a Scrapy project and reports what each choice did in my runs.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">What you need<\/h2>\n<ul>\n<li>A Windows PC with CapSkip 3.2.0 or later, which is the first release with DataDome. Follow the <a href=\"https:\/\/capskip.com\/setup-guide\/\">setup guide<\/a> to install CapSkip and start it.<\/li>\n<li>Python 3.11 or later, which Scrapy 2.20, w3lib and itemadapter all require.<\/li>\n<li>Scrapy 2.14 or later. On 2.13.4 the middleware fails with a TypeError, because that release still passes &quot;spider&quot; to every middleware method.<\/li>\n<li>capskip 1.5.0 or later, the first version with the datadome method.<\/li>\n<li>The address CapSkip answers on: 127.0.0.1 in Local mode, which reaches the same PC only, or your network or public IP in Server mode.<\/li>\n<li>One sticky proxy per crawl identity, as a full URL such as <code>http:\/\/user:pass@203.0.113.7:8080<\/code>. A proxy that changes its exit IP on every connection cannot produce a working cookie, say the CapSkip docs.<\/li>\n<\/ul>\n<p>In a PowerShell prompt, create the project that all blocks below go into. File names are relative to the folder that holds <code>scrapy.cfg<\/code>, and a block that starts a file replaces the generated one.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"bash\" class=\"EnlighterJSRAW\">pip install scrapy capskip\nscrapy startproject shop\ncd shop<\/pre>\n<\/div>\n<p>I tested against mock servers I wrote, not a real DataDome site and not the CapSkip app. One copies the in.php and res.php checks from the CapSkip API docs. The other answers 403 unless the request carries a cookie the first one issued, for the User-Agent and proxy it was issued for; those binding rules come from the CapSkip docs, not DataDome. I ran Python 3.11.7 and 3.14.0. Unless stated, counts come from Scrapy 2.20.0 and capskip 1.7.0, <code>DOWNLOAD_DELAY<\/code> at 0 and a mock that solves in 0.4 seconds, and matched on both Pythons. Scrapy 2.14.0 with capskip 1.5.0 and Scrapy 2.19.0 with capskip 1.6.0 gave the same counts in a few cases.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Build the DataDome Scrapy middleware<\/h2>\n<h3 style=\"font-size:1.3rem;line-height:1.4;\">Step 1: pick the priority number<\/h3>\n<p>Scrapy calls <code>process_request<\/code> in rising order of the numbers and <code>process_response<\/code> in falling order, so a bigger number sits closer to the downloader. HttpCompressionMiddleware at 590 unzips the body, so our number must be lower than 590 to read the block page. RetryMiddleware at 550 turns responses into retries, so ours must be higher to see them first. That leaves 551 to 589, and since 580 belongs to MetaRefreshMiddleware the settings use 570. The blocks below form one project, pasted in order, and none runs alone.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\"># shop\/settings.py: add these lines at the end\nDOWNLOADER_MIDDLEWARES = {\n    # process_response runs from high numbers to low, so 570 sees a response\n    # after HttpCompressionMiddleware (590) and before RetryMiddleware (550).\n    &quot;shop.middlewares.DataDomeMiddleware&quot;: 570,\n}\n\nDATADOME_CAPSKIP_HOST = &quot;127.0.0.1&quot;  # Server mode: your server's address\nDATADOME_CAPSKIP_PORT = 8080\nDATADOME_MAX_SOLVES = 3  # per request, and per domain and jar between good pages<\/pre>\n<\/div>\n<p>At 650 a gzipped block page reached the middleware as the bytes 1f 8b and the 403 went through; at 570 it was solved. With 403 in <code>RETRY_HTTP_CODES<\/code>, the mock site served 3 block pages before the solve at 540 and 1 at 570.<\/p>\n<h3 style=\"font-size:1.3rem;line-height:1.4;\">Step 2: spot the block page<\/h3>\n<p>A DataDome block arrives as status 401 or 403. When its page already holds the challenge iframe, the code takes that address as it is, as the CapSkip docs say. Otherwise it builds the <code>captcha_url<\/code> from the script object <code>dd<\/code>, where the value of <code>rt<\/code> picks the path: \/interstitial\/ for the device check and \/captcha\/ for the slider. The docs table lists the t parameter for \/captcha\/ only, but the docs&#8217; own JavaScript sets it unconditionally, and I followed the JavaScript. Anything unreadable returns None, so an ordinary 403 passes untouched. Nine hand-made responses covered each case, and a mock device check took 1 task for 2 pages. The layout comes from the CapSkip docs, not a live page.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\"># shop\/middlewares.py\nimport asyncio\nimport html\nimport json\nimport logging\nimport re\nfrom urllib.parse import urlencode\n\nfrom capskip import ApiException, CapSkip, CapSkipError\nfrom scrapy.http import TextResponse\nfrom scrapy.utils.httpobj import urlparse_cached\n\nlog = logging.getLogger(__name__)\nDD_OBJECT = re.compile(r&quot;var\\s+dd\\s*=\\s*(\\{[^}]*\\})&quot;)\nIFRAME = re.compile(r'src=&quot;(https:\/\/[^&quot;]*captcha-delivery\\.com\/(?:captcha|interstitial)\/\\?[^&quot;]*)&quot;')\n\n\ndef captcha_url_for(response):\n    &quot;&quot;&quot;The challenge address from a DataDome block page, or None.&quot;&quot;&quot;\n    if response.status not in (401, 403) or not isinstance(response, TextResponse):\n        return None\n    frame = IFRAME.search(response.text)\n    if frame:  # the page already carries the challenge address\n        return html.unescape(frame.group(1))\n    found = DD_OBJECT.search(response.text)\n    if not found:\n        return None\n    try:\n        dd = json.loads(found.group(1).replace(&quot;'&quot;, '&quot;'))\n        path = &quot;interstitial&quot; if dd[&quot;rt&quot;] == &quot;i&quot; else &quot;captcha&quot;\n        query = {&quot;initialCid&quot;: dd[&quot;cid&quot;], &quot;hash&quot;: dd[&quot;hsh&quot;], &quot;cid&quot;: dd[&quot;cookie&quot;],\n                 &quot;t&quot;: dd.get(&quot;t&quot;), &quot;referer&quot;: response.url, &quot;s&quot;: dd[&quot;s&quot;],\n                 &quot;e&quot;: dd.get(&quot;e&quot;), &quot;b&quot;: dd.get(&quot;b&quot;), &quot;dm&quot;: &quot;cd&quot;}\n    except (KeyError, ValueError):\n        return None  # a dd object this code does not understand\n    query = {k: v for k, v in query.items() if v}\n    host = dd.get(&quot;host&quot;, &quot;geo.captcha-delivery.com&quot;)\n    return f&quot;https:\/\/{host}\/{path}\/?{urlencode(query)}&quot;<\/pre>\n<\/div>\n<h3 style=\"font-size:1.3rem;line-height:1.4;\">Step 3: keep one identity per cookie jar<\/h3>\n<p>A cookie lives in a jar chosen by the <code>cookiejar<\/code> meta key, so the middleware keeps an identity per jar: the cookie, its User-Agent, a proxy when CapSkip chose one, and a generation number that grows with every solve. Without the User-Agent line the retry went out with the default Scrapy User-Agent, and four direct pages ended with 3 tasks and no page.<\/p>\n<p>The cookie goes in through the &quot;cookies&quot; argument, which CookiesMiddleware at 700 copies into the jar. It is added only until one request carrying it has come back, so a replacement cookie from the site is not overwritten. The mock sends one, and a page fetched afterwards carried it with no new task. A Cookie header does not work, because CookiesMiddleware rebuilds the header from the jar, and neither does turning cookies off. Both ended with 3 tasks and no page.<\/p>\n<p>The &quot;Domain&quot; and &quot;Path&quot; attributes come from the &quot;code&quot; field of the result, which holds the cookie as DataDome set it, so a cookie for a named site keeps its domain. The mock gave pages on 127.0.0.1 a host-only cookie.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\"># shop\/middlewares.py, continued\ndef cookie_attributes(code):\n    &quot;&quot;&quot;Domain, Path and the rest of a Set-Cookie string, names in lower case.&quot;&quot;&quot;\n    pairs = (part.partition(&quot;=&quot;) for part in code.split(&quot;;&quot;)[1:])\n    return {name.strip().lower(): value.strip() for name, _, value in pairs if value}\n\n\ndef cookies_without(cookies, name):\n    &quot;&quot;&quot;A request's cookies as a list, minus any with this name.&quot;&quot;&quot;\n    if isinstance(cookies, dict):\n        cookies = [{&quot;name&quot;: k, &quot;value&quot;: v} for k, v in cookies.items()]\n    return [c for c in cookies if c[&quot;name&quot;] != name]\n\n\nclass Identity:\n    &quot;&quot;&quot;What every request in one cookie jar has to keep sending.&quot;&quot;&quot;\n\n    def __init__(self):\n        self.user_agent = None\n        self.cookie = None  # datadome=VALUE\n        self.attrs = {}  # Domain and Path, from the cookie as DataDome set it\n        self.proxy = None  # set only when CapSkip solved through its own list\n        self.generation = 0  # grows with every solve\n        self.fresh = False  # True until a request has carried the cookie to the jar\n\n\nclass DataDomeMiddleware:\n    def __init__(self, crawler):\n        s = crawler.settings\n        self.stats = crawler.stats\n        self.max_solves = s.getint(&quot;DATADOME_MAX_SOLVES&quot;, 3)\n        self.solver = CapSkip(host=s.get(&quot;DATADOME_CAPSKIP_HOST&quot;, &quot;127.0.0.1&quot;),\n                              port=s.getint(&quot;DATADOME_CAPSKIP_PORT&quot;, 8080))\n        self.identities = {}  # cookiejar key -&gt; Identity\n        self.solving = {}  # cookiejar key -&gt; the solve that is running\n        self.spent = {}  # (domain, jar) -&gt; solves since the last page that was not a block\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        return cls(crawler)\n\n    def process_request(self, request):\n        # HttpProxyMiddleware (750) moves the proxy login into a header, so\n        # keep the proxy as the spider wrote it, login included.\n        request.meta.setdefault(&quot;dd_proxy&quot;, request.meta.get(&quot;proxy&quot;))\n        ident = self.identities.get(request.meta.get(&quot;cookiejar&quot;))\n        request.meta[&quot;dd_generation&quot;] = ident.generation if ident else 0\n        if not ident:\n            return\n        # The cookie only works with the User-Agent and address that earned it.\n        request.headers[&quot;User-Agent&quot;] = ident.user_agent\n        if ident.proxy:\n            request.meta[&quot;proxy&quot;] = ident.proxy\n        if ident.fresh:  # CookiesMiddleware (700) copies this into the jar\n            name, value = ident.cookie.split(&quot;=&quot;, 1)\n            cookie = {&quot;name&quot;: name, &quot;value&quot;: value,\n                      &quot;path&quot;: ident.attrs.get(&quot;path&quot;, &quot;\/&quot;)}\n            if &quot;domain&quot; in ident.attrs:\n                cookie[&quot;domain&quot;] = ident.attrs[&quot;domain&quot;]\n            request.cookies = cookies_without(request.cookies, name) + [cookie]\n            request.meta[&quot;dd_carried&quot;] = ident.generation<\/pre>\n<\/div>\n<h3 style=\"font-size:1.3rem;line-height:1.4;\">Step 4: solve in a thread, then retry<\/h3>\n<p>The datadome method of CapSkip blocks while it polls. Called straight from a coroutine it holds the reactor, so the middleware hands it to <code>asyncio.to_thread<\/code>. The SDK also ships <code>AsyncCapSkip<\/code>, whose datadome method you can await. The retried request sets <code>dont_filter<\/code>, because its URL has already passed the duplicate filter. Without it the dupefilter dropped both retries, and <code>dupefilter\/filtered<\/code> read 2.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\">    # shop\/middlewares.py, continued\n    async def solve(self, key, captcha_url, request):\n        proxy = request.meta.get(&quot;dd_proxy&quot;)\n        arg = None\n        if proxy:  # solve through the proxy the crawl uses\n            scheme, rest = proxy.split(&quot;:\/\/&quot;, 1)\n            arg = {&quot;type&quot;: scheme.upper(), &quot;uri&quot;: rest}\n        # The SDK blocks, so it runs in a thread and the reactor stays free.\n        result = await asyncio.to_thread(\n            self.solver.datadome, captcha_url, request.url, proxy=arg)\n        ident = self.identities.setdefault(key, Identity())\n        ident.user_agent = result[&quot;userAgent&quot;]\n        ident.cookie = result[&quot;cookie&quot;]\n        ident.attrs = cookie_attributes(result[&quot;code&quot;])\n        if result.get(&quot;proxy&quot;):\n            ident.proxy = &quot;http:\/\/&quot; + result[&quot;proxy&quot;]\n        ident.generation += 1\n        ident.fresh = True\n        self.stats.inc_value(&quot;datadome\/solves&quot;)\n        return ident\n\n    def retry(self, request):\n        meta = {**request.meta, &quot;dd_tries&quot;: request.meta.get(&quot;dd_tries&quot;, 0) + 1}\n        cookies = cookies_without(request.cookies, &quot;datadome&quot;)  # process_request adds the current one\n        # dont_filter: the dupefilter has already seen this URL\n        return request.replace(cookies=cookies, dont_filter=True, meta=meta)<\/pre>\n<\/div>\n<p>This extension ticks every 50 milliseconds and records the longest gap. The crawl had one blocked page, 30 open pages and 8 requests at once, the mock took 3 seconds, and only the solve call changed. Each row is the range of at least three runs on each Python.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\"># shop\/heartbeat.py (settings.py then needs EXTENSIONS = {&quot;shop.heartbeat.Heartbeat&quot;: 500})\nimport asyncio\nimport time\n\nfrom scrapy import signals\n\n\nclass Heartbeat:\n    &quot;&quot;&quot;Ticks every 50 ms and records the longest gap between two ticks.&quot;&quot;&quot;\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        ext = cls()\n        ext.stats = crawler.stats\n        crawler.signals.connect(ext.start, signals.spider_opened)\n        crawler.signals.connect(ext.stop, signals.spider_closed)\n        return ext\n\n    def start(self):\n        self.longest, self.last, self.running = 0.0, time.perf_counter(), True\n        self.tick()\n\n    def tick(self):\n        if self.running:\n            now = time.perf_counter()\n            self.longest = max(self.longest, now - self.last)\n            self.last = now\n            asyncio.get_running_loop().call_later(0.05, self.tick)\n\n    def stop(self):\n        self.running = False\n        self.stats.set_value(&quot;heartbeat\/longest_gap_ms&quot;, round(self.longest * 1000))<\/pre>\n<\/div>\n<table>\n<thead>\n<tr>\n<th>How solve calls the SDK<\/th>\n<th>Longest tick gap on one PC<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Direct call inside the coroutine<\/td>\n<td>about 3770 to 3840 ms<\/td>\n<\/tr>\n<tr>\n<td>asyncio.to_thread, the code above<\/td>\n<td>about 65 to 75 ms<\/td>\n<\/tr>\n<tr>\n<td>AsyncCapSkip awaited in the coroutine<\/td>\n<td>about 220 to 530 ms<\/td>\n<\/tr>\n<tr>\n<td>deferToThread with maybe_deferred_to_future<\/td>\n<td>about 65 to 90 ms<\/td>\n<\/tr>\n<tr>\n<td>deferToThread awaited with no conversion<\/td>\n<td>about 65 to 80 ms, request lost<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>AsyncCapSkip, which I ran six times on each Python, works but stalls longer: each call to CapSkip builds a new httpx client, and building one took about 130 to 190 ms on this PC. The last row looks right and is not: it raised RuntimeError, Task got bad yield, and the request was lost. The whole middleware also needs the asyncio reactor, the default in Scrapy 2.20.0. Under the classic select reactor it raised a RuntimeError and sent no task.<\/p>\n<h3 style=\"font-size:1.3rem;line-height:1.4;\">Step 5: decide what each blocked response does<\/h3>\n<p>Requests in one jar are blocked together, and each would start a solve. The first blocked response starts it and stores it in <code>self.solving<\/code>, and the others in the jar wait for it. Eight pages through one proxy produced 1 task with this code and 8 when each response solved for itself. Setting <code>CONCURRENT_REQUESTS_PER_DOMAIN<\/code> to 1 did not reduce the block pages: four pages sent without a proxy still collected four block pages before the first cookie arrived.<\/p>\n<p><code>DATADOME_MAX_SOLVES<\/code> caps two counts, and you need both. A request can be retried that many times (<code>dd_tries<\/code>). Each pair of domain and cookie jar can start that many solves between pages that got through (<code>self.spent<\/code>). Keyed by domain alone, five jars that each fetched one page at once (16 requests per domain allowed) got 3 tasks and lost 2 pages. Keyed by domain and jar they got 5 tasks and 5 pages. A URL that refuses every cookie, among 60 good pages, kept resetting the budget, and only the per-request count stopped it, at 3 tasks. Because CapSkip is an <a href=\"https:\/\/capskip.com\/\">unlimited captcha solver<\/a> on a flat license, the caps guard against a loop, not your wallet.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\">    # shop\/middlewares.py, continued\n    async def process_response(self, request, response):\n        key = request.meta.get(&quot;cookiejar&quot;)\n        ident = self.identities.get(key)\n        if ident and request.meta.get(&quot;dd_carried&quot;) == ident.generation:\n            ident.fresh = False  # the jar has seen the cookie by now\n        captcha_url = captcha_url_for(response)\n        budget = (urlparse_cached(request).hostname, key)\n        if captcha_url is None:\n            self.spent.pop(budget, None)  # a page got through: reset the budget\n            return response\n        if request.meta.get(&quot;dd_tries&quot;, 0) &gt;= self.max_solves:\n            return response\n        if ident and ident.generation &gt; request.meta.get(&quot;dd_generation&quot;, 0):\n            return self.retry(request)  # sent before the last solve\n        if self.spent.get(budget, 0) &gt;= self.max_solves:\n            self.stats.inc_value(&quot;datadome\/budget_used_up&quot;)\n            return response\n        task = self.solving.get(key)\n        if task is None:  # the first blocked request starts the solve\n            self.spent[budget] = self.spent.get(budget, 0) + 1\n            task = asyncio.ensure_future(self.solve(key, captcha_url, request))\n            self.solving[key] = task\n            task.add_done_callback(lambda _: self.solving.pop(key, None))\n        try:\n            await task  # the others in the same jar wait for it\n        except ApiException as err:\n            if &quot;ERROR_PROXY&quot; in str(err):  # a retry through that address fails again\n                log.error(&quot;DataDome solve refused for the address: %s&quot;, err)\n                return response\n            log.warning(&quot;DataDome solve refused: %s&quot;, err)\n            return self.retry(request)  # fetch the page again: a new challenge\n        except CapSkipError as err:\n            log.error(&quot;DataDome solve failed: %r&quot;, err)\n            return response\n        return self.retry(request)<\/pre>\n<\/div>\n<h3 style=\"font-size:1.3rem;line-height:1.4;\">Step 6: one cookie jar per proxy<\/h3>\n<p>A crawl through several proxies needs a jar per proxy, so every identity has one address, and the spider names each jar after its proxy. Scrapy does not carry either key to requests you build from a response: a link followed with <code>response.follow<\/code> and no meta went out directly through the default jar, and was blocked and solved again. Copy the proxy and cookiejar keys into every request you create.<\/p>\n<p>One jar shared by two proxies is the other mistake: eight pages split across them finished with 12 block pages instead of 8 in all three runs on each Python, at one request at a time with a delay of 0.2 seconds. Both layouts took 2 tasks.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"python\" class=\"EnlighterJSRAW\"># shop\/spiders\/shop.py\nimport scrapy\n\n# your own sticky proxies, with the login if they need one\nPROXIES = [&quot;http:\/\/user:pass@203.0.113.7:8080&quot;, &quot;http:\/\/user:pass@203.0.113.8:8080&quot;]\n# your own DataDome-protected pages\nURLS = [f&quot;https:\/\/www.example.com\/page\/{n}&quot; for n in range(1, 9)]\n\n\nclass ShopSpider(scrapy.Spider):\n    name = &quot;shop&quot;\n\n    async def start(self):\n        for n, url in enumerate(URLS):\n            proxy = PROXIES[n % len(PROXIES)]\n            # one cookie jar per proxy, named after it\n            yield scrapy.Request(url, meta={&quot;proxy&quot;: proxy, &quot;cookiejar&quot;: proxy})\n\n    def parse(self, response):\n        yield {&quot;url&quot;: response.url, &quot;status&quot;: response.status,\n               &quot;h1&quot;: response.css(&quot;h1::text&quot;).get()}<\/pre>\n<\/div>\n<p>Both lists hold placeholders; for my run I swapped in my mock proxies and pages and changed nothing else. With the template delay of 1 it took 2 tasks and loaded all 8 pages. Scrapy moves a proxy login into a header before the response comes back, so the middleware keeps the proxy as written; without that my mock received the task with no login.<\/p>\n<div data-no-translation>\n<pre data-enlighter-language=\"bash\" class=\"EnlighterJSRAW\"># run this in the folder that holds scrapy.cfg\nscrapy crawl shop -O out.jsonl<\/pre>\n<\/div>\n<p>When CapSkip solves through its own proxy list, the result names that proxy and the identity keeps it, since the cookie works only through it. The docs say it comes back without its login, so a listed proxy with a login is untested.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">DataDome 403 errors you may meet in Scrapy<\/h2>\n<p>I provoked every row in a run against my mock, except the missing <code>dd<\/code> object and an iframe-only page, which only hand-made responses covered. A refused task is logged and the page fetched again. A refusal that names the address, ERROR_PROXY_BANNED or ERROR_PROXY_CONNECT_REFUSED, is logged and passed on, because the docs say a retry through that address fails again. My mock returned both at result time, and I did not provoke them against the real app. Any other SDK error is logged and passed on.<\/p>\n<table>\n<thead>\n<tr>\n<th>What you see<\/th>\n<th>Cause<\/th>\n<th>Fix<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>TypeError about an unexpected keyword argument &quot;spider&quot;<\/td>\n<td>Scrapy 2.13 or older<\/td>\n<td>Upgrade to 2.14 or later<\/td>\n<\/tr>\n<tr>\n<td>RuntimeError: Task got bad yield<\/td>\n<td>A Deferred awaited directly in a coroutine<\/td>\n<td>Use asyncio.to_thread or maybe_deferred_to_future<\/td>\n<\/tr>\n<tr>\n<td>403 pages reach HttpErrorMiddleware and no task is sent<\/td>\n<td>The middleware number is above 590, or the block page has neither an iframe address nor a <code>dd<\/code> object<\/td>\n<td>Use 551 to 589 and look in the block page for either<\/td>\n<\/tr>\n<tr>\n<td>Pages stay blocked and a budget_used_up count appears in the stats<\/td>\n<td>Cookies off, a Cookie header or a changed User-Agent<\/td>\n<td>Compare the retry with the identity<\/td>\n<\/tr>\n<tr>\n<td>ERROR_PROXY_BANNED or ERROR_PROXY_CONNECT_REFUSED in the log<\/td>\n<td>CapSkip names the address as banned or unreachable<\/td>\n<td>Change the proxy for that jar<\/td>\n<\/tr>\n<tr>\n<td>ValidationException with <code>t=bv<\/code> in the log<\/td>\n<td>The block page says the address is banned<\/td>\n<td>Change the proxy for that jar<\/td>\n<\/tr>\n<tr>\n<td>NetworkException in the log<\/td>\n<td>Nothing answers at the host and port in settings.py<\/td>\n<td>Start CapSkip and check both<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Ordering rules live in the <a href=\"https:\/\/docs.scrapy.org\/en\/latest\/topics\/downloader-middleware.html\" rel=\"noopener nofollow\" target=\"_blank\">downloader middleware reference<\/a> of the Scrapy project. For the same idea applied to reCAPTCHA, see the <a href=\"https:\/\/capskip.com\/scrapy-captcha-middleware\/\">Scrapy CAPTCHA middleware guide<\/a>, and for crawling at scale see the <a href=\"https:\/\/capskip.com\/captcha-solver-for-web-scraping\/\">web scraping page<\/a>.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">Running CapSkip on a server<\/h2>\n<p>Server mode makes CapSkip answer on your network address or a public IP, so a crawler on another PC or a Windows VPS can reach it. Choose it in the <a href=\"https:\/\/capskip.com\/setup-guide\/#connection-settings\">connection settings<\/a> of CapSkip and put the address of that machine in <code>DATADOME_CAPSKIP_HOST<\/code>; nothing else changes. Starting Server mode leaves the CapSkip port open to every address, so add a Windows Firewall rule that blocks the machines which must not reach it. The solve runs on the server, so the cookie is earned from its address unless the request carries a proxy, which the middleware passes on. I ran none of this, only a mock on 127.0.0.1, and with API Key Validation on you would pass apiKey to the CapSkip constructor.<\/p>\n<h2 style=\"font-size:1.6rem;line-height:1.35;\">FAQ<\/h2>\n<details style=\"border:1px solid #e2e5ee;border-radius:10px;padding:14px 18px;margin:0 0 12px;\">\n<summary style=\"cursor:pointer;\">\n<h3 style=\"font-size:1.15rem;line-height:1.4;display:inline;margin:0;\">Can I await AsyncCapSkip instead of using a thread?<\/h3>\n<\/summary>\n<p style=\"margin:12px 0 0;\">Yes. Build AsyncCapSkip in place of CapSkip and await its datadome method in <code>self.solve<\/code>. It gave the same counts as the thread on both Pythons, with a longer stall of about 220 to 530 ms on my PC.<\/p>\n<\/details>\n<details style=\"border:1px solid #e2e5ee;border-radius:10px;padding:14px 18px;margin:0 0 12px;\">\n<summary style=\"cursor:pointer;\">\n<h3 style=\"font-size:1.15rem;line-height:1.4;display:inline;margin:0;\">How many solves will a real DataDome Scrapy crawl need?<\/h3>\n<\/summary>\n<p style=\"margin:12px 0 0;\">I can&#8217;t say. In the mock it was one per proxy, because the mock never expires a cookie. The stats key <code>datadome\/solves<\/code> counts the solves in your own crawl.<\/p>\n<\/details>\n","protected":false},"excerpt":{"rendered":"<p>\u4e00\u4e2a Scrapy \u4e0b\u8f7d\u5668\u4e2d\u95f4\u4ef6\uff1a\u53d1\u73b0 DataDome \u7684 403 \u9875\u9762\uff0c\u5728\u7ebf\u7a0b\u91cc\u5411 CapSkip \u53d6\u5f97 cookie\uff0c\u5e76\u7528\u540c\u4e00\u4e2a\u4ee3\u7406\u548c User-Agent \u91cd\u8bd5\u3002<\/p>","protected":false},"author":1,"featured_media":27969,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"DataDome Scrapy Middleware: Solve 403 Blocks | CapSkip","rank_math_description":"DataDome Scrapy middleware: spot the 403 block page, solve it in a thread, retry with the same cookie, proxy and User-Agent. Copy the middleware.","rank_math_focus_keyword":"datadome scrapy","footnotes":""},"categories":[70],"tags":[],"class_list":["post-27970","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-captcha"],"_links":{"self":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts\/27970","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/comments?post=27970"}],"version-history":[{"count":1,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts\/27970\/revisions"}],"predecessor-version":[{"id":28150,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/posts\/27970\/revisions\/28150"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/media\/27969"}],"wp:attachment":[{"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/media?parent=27970"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/categories?post=27970"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/capskip.com\/zh\/wp-json\/wp\/v2\/tags?post=27970"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}