How to Solve DataDome Scrapy 403 Blocks With a Middleware

To handle DataDome in Scrapy, put the work in one downloader middleware. It spots the 403 block page, gets a datadome cookie from CapSkip with the datadome method of the Python SDK, and retries with that cookie, its User-Agent and the same proxy. A DataDome Scrapy setup lives or dies on the priority number, the duplicate filter, a blocking SDK call and a cookie jar per proxy. This guide wires the CapSkip DataDome solver into a Scrapy project and reports what each choice did in my runs.
What you need
- A Windows PC with CapSkip 3.2.0 or later, which is the first release with DataDome. Follow the setup guide to install CapSkip and start it.
- Python 3.11 or later, which Scrapy 2.20, w3lib and itemadapter all require.
- Scrapy 2.14 or later. On 2.13.4 the middleware fails with a TypeError, because that release still passes "spider" to every middleware method.
- capskip 1.5.0 or later, the first version with the datadome method.
- The address CapSkip answers on: 127.0.0.1 in Local mode, which reaches the same PC only, or your network or public IP in Server mode.
- One sticky proxy per crawl identity, as a full URL such as
http://user:[email protected]:8080. A proxy that changes its exit IP on every connection cannot produce a working cookie, say the CapSkip docs.
In a PowerShell prompt, create the project that all blocks below go into. File names are relative to the folder that holds scrapy.cfg, and a block that starts a file replaces the generated one.
pip install scrapy capskip scrapy startproject shop cd shop
I tested against mock servers I wrote, not a real DataDome site and not the CapSkip app. One copies the in.php and res.php checks from the CapSkip API docs. The other answers 403 unless the request carries a cookie the first one issued, for the User-Agent and proxy it was issued for; those binding rules come from the CapSkip docs, not DataDome. I ran Python 3.11.7 and 3.14.0. Unless stated, counts come from Scrapy 2.20.0 and capskip 1.7.0, DOWNLOAD_DELAY at 0 and a mock that solves in 0.4 seconds, and matched on both Pythons. Scrapy 2.14.0 with capskip 1.5.0 and Scrapy 2.19.0 with capskip 1.6.0 gave the same counts in a few cases.
Build the DataDome Scrapy middleware
Step 1: pick the priority number
Scrapy calls process_request in rising order of the numbers and process_response in falling order, so a bigger number sits closer to the downloader. HttpCompressionMiddleware at 590 unzips the body, so our number must be lower than 590 to read the block page. RetryMiddleware at 550 turns responses into retries, so ours must be higher to see them first. That leaves 551 to 589, and since 580 belongs to MetaRefreshMiddleware the settings use 570. The blocks below form one project, pasted in order, and none runs alone.
# shop/settings.py: add these lines at the end
DOWNLOADER_MIDDLEWARES = {
# process_response runs from high numbers to low, so 570 sees a response
# after HttpCompressionMiddleware (590) and before RetryMiddleware (550).
"shop.middlewares.DataDomeMiddleware": 570,
}
DATADOME_CAPSKIP_HOST = "127.0.0.1" # Server mode: your server's address
DATADOME_CAPSKIP_PORT = 8080
DATADOME_MAX_SOLVES = 3 # per request, and per domain and jar between good pagesAt 650 a gzipped block page reached the middleware as the bytes 1f 8b and the 403 went through; at 570 it was solved. With 403 in RETRY_HTTP_CODES, the mock site served 3 block pages before the solve at 540 and 1 at 570.
Step 2: spot the block page
A DataDome block arrives as status 401 or 403. When its page already holds the challenge iframe, the code takes that address as it is, as the CapSkip docs say. Otherwise it builds the captcha_url from the script object dd, where the value of rt picks the path: /interstitial/ for the device check and /captcha/ for the slider. The docs table lists the t parameter for /captcha/ only, but the docs’ own JavaScript sets it unconditionally, and I followed the JavaScript. Anything unreadable returns None, so an ordinary 403 passes untouched. Nine hand-made responses covered each case, and a mock device check took 1 task for 2 pages. The layout comes from the CapSkip docs, not a live page.
# shop/middlewares.py
import asyncio
import html
import json
import logging
import re
from urllib.parse import urlencode
from capskip import ApiException, CapSkip, CapSkipError
from scrapy.http import TextResponse
from scrapy.utils.httpobj import urlparse_cached
log = logging.getLogger(__name__)
DD_OBJECT = re.compile(r"var\s+dd\s*=\s*(\{[^}]*\})")
IFRAME = re.compile(r'src="(https://[^"]*captcha-delivery\.com/(?:captcha|interstitial)/\?[^"]*)"')
def captcha_url_for(response):
"""The challenge address from a DataDome block page, or None."""
if response.status not in (401, 403) or not isinstance(response, TextResponse):
return None
frame = IFRAME.search(response.text)
if frame: # the page already carries the challenge address
return html.unescape(frame.group(1))
found = DD_OBJECT.search(response.text)
if not found:
return None
try:
dd = json.loads(found.group(1).replace("'", '"'))
path = "interstitial" if dd["rt"] == "i" else "captcha"
query = {"initialCid": dd["cid"], "hash": dd["hsh"], "cid": dd["cookie"],
"t": dd.get("t"), "referer": response.url, "s": dd["s"],
"e": dd.get("e"), "b": dd.get("b"), "dm": "cd"}
except (KeyError, ValueError):
return None # a dd object this code does not understand
query = {k: v for k, v in query.items() if v}
host = dd.get("host", "geo.captcha-delivery.com")
return f"https://{host}/{path}/?{urlencode(query)}"Step 3: keep one identity per cookie jar
A cookie lives in a jar chosen by the cookiejar meta key, so the middleware keeps an identity per jar: the cookie, its User-Agent, a proxy when CapSkip chose one, and a generation number that grows with every solve. Without the User-Agent line the retry went out with the default Scrapy User-Agent, and four direct pages ended with 3 tasks and no page.
The cookie goes in through the "cookies" argument, which CookiesMiddleware at 700 copies into the jar. It is added only until one request carrying it has come back, so a replacement cookie from the site is not overwritten. The mock sends one, and a page fetched afterwards carried it with no new task. A Cookie header does not work, because CookiesMiddleware rebuilds the header from the jar, and neither does turning cookies off. Both ended with 3 tasks and no page.
The "Domain" and "Path" attributes come from the "code" field of the result, which holds the cookie as DataDome set it, so a cookie for a named site keeps its domain. The mock gave pages on 127.0.0.1 a host-only cookie.
# shop/middlewares.py, continued
def cookie_attributes(code):
"""Domain, Path and the rest of a Set-Cookie string, names in lower case."""
pairs = (part.partition("=") for part in code.split(";")[1:])
return {name.strip().lower(): value.strip() for name, _, value in pairs if value}
def cookies_without(cookies, name):
"""A request's cookies as a list, minus any with this name."""
if isinstance(cookies, dict):
cookies = [{"name": k, "value": v} for k, v in cookies.items()]
return [c for c in cookies if c["name"] != name]
class Identity:
"""What every request in one cookie jar has to keep sending."""
def __init__(self):
self.user_agent = None
self.cookie = None # datadome=VALUE
self.attrs = {} # Domain and Path, from the cookie as DataDome set it
self.proxy = None # set only when CapSkip solved through its own list
self.generation = 0 # grows with every solve
self.fresh = False # True until a request has carried the cookie to the jar
class DataDomeMiddleware:
def __init__(self, crawler):
s = crawler.settings
self.stats = crawler.stats
self.max_solves = s.getint("DATADOME_MAX_SOLVES", 3)
self.solver = CapSkip(host=s.get("DATADOME_CAPSKIP_HOST", "127.0.0.1"),
port=s.getint("DATADOME_CAPSKIP_PORT", 8080))
self.identities = {} # cookiejar key -> Identity
self.solving = {} # cookiejar key -> the solve that is running
self.spent = {} # (domain, jar) -> solves since the last page that was not a block
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def process_request(self, request):
# HttpProxyMiddleware (750) moves the proxy login into a header, so
# keep the proxy as the spider wrote it, login included.
request.meta.setdefault("dd_proxy", request.meta.get("proxy"))
ident = self.identities.get(request.meta.get("cookiejar"))
request.meta["dd_generation"] = ident.generation if ident else 0
if not ident:
return
# The cookie only works with the User-Agent and address that earned it.
request.headers["User-Agent"] = ident.user_agent
if ident.proxy:
request.meta["proxy"] = ident.proxy
if ident.fresh: # CookiesMiddleware (700) copies this into the jar
name, value = ident.cookie.split("=", 1)
cookie = {"name": name, "value": value,
"path": ident.attrs.get("path", "/")}
if "domain" in ident.attrs:
cookie["domain"] = ident.attrs["domain"]
request.cookies = cookies_without(request.cookies, name) + [cookie]
request.meta["dd_carried"] = ident.generationStep 4: solve in a thread, then retry
The datadome method of CapSkip blocks while it polls. Called straight from a coroutine it holds the reactor, so the middleware hands it to asyncio.to_thread. The SDK also ships AsyncCapSkip, whose datadome method you can await. The retried request sets dont_filter, because its URL has already passed the duplicate filter. Without it the dupefilter dropped both retries, and dupefilter/filtered read 2.
# shop/middlewares.py, continued
async def solve(self, key, captcha_url, request):
proxy = request.meta.get("dd_proxy")
arg = None
if proxy: # solve through the proxy the crawl uses
scheme, rest = proxy.split("://", 1)
arg = {"type": scheme.upper(), "uri": rest}
# The SDK blocks, so it runs in a thread and the reactor stays free.
result = await asyncio.to_thread(
self.solver.datadome, captcha_url, request.url, proxy=arg)
ident = self.identities.setdefault(key, Identity())
ident.user_agent = result["userAgent"]
ident.cookie = result["cookie"]
ident.attrs = cookie_attributes(result["code"])
if result.get("proxy"):
ident.proxy = "http://" + result["proxy"]
ident.generation += 1
ident.fresh = True
self.stats.inc_value("datadome/solves")
return ident
def retry(self, request):
meta = {**request.meta, "dd_tries": request.meta.get("dd_tries", 0) + 1}
cookies = cookies_without(request.cookies, "datadome") # process_request adds the current one
# dont_filter: the dupefilter has already seen this URL
return request.replace(cookies=cookies, dont_filter=True, meta=meta)This extension ticks every 50 milliseconds and records the longest gap. The crawl had one blocked page, 30 open pages and 8 requests at once, the mock took 3 seconds, and only the solve call changed. Each row is the range of at least three runs on each Python.
# shop/heartbeat.py (settings.py then needs EXTENSIONS = {"shop.heartbeat.Heartbeat": 500})
import asyncio
import time
from scrapy import signals
class Heartbeat:
"""Ticks every 50 ms and records the longest gap between two ticks."""
@classmethod
def from_crawler(cls, crawler):
ext = cls()
ext.stats = crawler.stats
crawler.signals.connect(ext.start, signals.spider_opened)
crawler.signals.connect(ext.stop, signals.spider_closed)
return ext
def start(self):
self.longest, self.last, self.running = 0.0, time.perf_counter(), True
self.tick()
def tick(self):
if self.running:
now = time.perf_counter()
self.longest = max(self.longest, now - self.last)
self.last = now
asyncio.get_running_loop().call_later(0.05, self.tick)
def stop(self):
self.running = False
self.stats.set_value("heartbeat/longest_gap_ms", round(self.longest * 1000))| How solve calls the SDK | Longest tick gap on one PC |
|---|---|
| Direct call inside the coroutine | about 3770 to 3840 ms |
| asyncio.to_thread, the code above | about 65 to 75 ms |
| AsyncCapSkip awaited in the coroutine | about 220 to 530 ms |
| deferToThread with maybe_deferred_to_future | about 65 to 90 ms |
| deferToThread awaited with no conversion | about 65 to 80 ms, request lost |
AsyncCapSkip, which I ran six times on each Python, works but stalls longer: each call to CapSkip builds a new httpx client, and building one took about 130 to 190 ms on this PC. The last row looks right and is not: it raised RuntimeError, Task got bad yield, and the request was lost. The whole middleware also needs the asyncio reactor, the default in Scrapy 2.20.0. Under the classic select reactor it raised a RuntimeError and sent no task.
Step 5: decide what each blocked response does
Requests in one jar are blocked together, and each would start a solve. The first blocked response starts it and stores it in self.solving, and the others in the jar wait for it. Eight pages through one proxy produced 1 task with this code and 8 when each response solved for itself. Setting CONCURRENT_REQUESTS_PER_DOMAIN to 1 did not reduce the block pages: four pages sent without a proxy still collected four block pages before the first cookie arrived.
DATADOME_MAX_SOLVES caps two counts, and you need both. A request can be retried that many times (dd_tries). Each pair of domain and cookie jar can start that many solves between pages that got through (self.spent). Keyed by domain alone, five jars that each fetched one page at once (16 requests per domain allowed) got 3 tasks and lost 2 pages. Keyed by domain and jar they got 5 tasks and 5 pages. A URL that refuses every cookie, among 60 good pages, kept resetting the budget, and only the per-request count stopped it, at 3 tasks. Because CapSkip is an unlimited captcha solver on a flat license, the caps guard against a loop, not your wallet.
# shop/middlewares.py, continued
async def process_response(self, request, response):
key = request.meta.get("cookiejar")
ident = self.identities.get(key)
if ident and request.meta.get("dd_carried") == ident.generation:
ident.fresh = False # the jar has seen the cookie by now
captcha_url = captcha_url_for(response)
budget = (urlparse_cached(request).hostname, key)
if captcha_url is None:
self.spent.pop(budget, None) # a page got through: reset the budget
return response
if request.meta.get("dd_tries", 0) >= self.max_solves:
return response
if ident and ident.generation > request.meta.get("dd_generation", 0):
return self.retry(request) # sent before the last solve
if self.spent.get(budget, 0) >= self.max_solves:
self.stats.inc_value("datadome/budget_used_up")
return response
task = self.solving.get(key)
if task is None: # the first blocked request starts the solve
self.spent[budget] = self.spent.get(budget, 0) + 1
task = asyncio.ensure_future(self.solve(key, captcha_url, request))
self.solving[key] = task
task.add_done_callback(lambda _: self.solving.pop(key, None))
try:
await task # the others in the same jar wait for it
except ApiException as err:
if "ERROR_PROXY" in str(err): # a retry through that address fails again
log.error("DataDome solve refused for the address: %s", err)
return response
log.warning("DataDome solve refused: %s", err)
return self.retry(request) # fetch the page again: a new challenge
except CapSkipError as err:
log.error("DataDome solve failed: %r", err)
return response
return self.retry(request)Step 6: one cookie jar per proxy
A crawl through several proxies needs a jar per proxy, so every identity has one address, and the spider names each jar after its proxy. Scrapy does not carry either key to requests you build from a response: a link followed with response.follow and no meta went out directly through the default jar, and was blocked and solved again. Copy the proxy and cookiejar keys into every request you create.
One jar shared by two proxies is the other mistake: eight pages split across them finished with 12 block pages instead of 8 in all three runs on each Python, at one request at a time with a delay of 0.2 seconds. Both layouts took 2 tasks.
# shop/spiders/shop.py import scrapy # your own sticky proxies, with the login if they need one PROXIES = ["http://user:[email protected]:8080", "http://user:[email protected]:8080"] # your own DataDome-protected pages URLS = [f"https://www.example.com/page/{n}" for n in range(1, 9)] class ShopSpider(scrapy.Spider): name = "shop" async def start(self): for n, url in enumerate(URLS): proxy = PROXIES[n % len(PROXIES)] # one cookie jar per proxy, named after it yield scrapy.Request(url, meta={"proxy": proxy, "cookiejar": proxy}) def parse(self, response): yield {"url": response.url, "status": response.status, "h1": response.css("h1::text").get()}
Both lists hold placeholders; for my run I swapped in my mock proxies and pages and changed nothing else. With the template delay of 1 it took 2 tasks and loaded all 8 pages. Scrapy moves a proxy login into a header before the response comes back, so the middleware keeps the proxy as written; without that my mock received the task with no login.
# run this in the folder that holds scrapy.cfg scrapy crawl shop -O out.jsonl
When CapSkip solves through its own proxy list, the result names that proxy and the identity keeps it, since the cookie works only through it. The docs say it comes back without its login, so a listed proxy with a login is untested.
DataDome 403 errors you may meet in Scrapy
I provoked every row in a run against my mock, except the missing dd object and an iframe-only page, which only hand-made responses covered. A refused task is logged and the page fetched again. A refusal that names the address, ERROR_PROXY_BANNED or ERROR_PROXY_CONNECT_REFUSED, is logged and passed on, because the docs say a retry through that address fails again. My mock returned both at result time, and I did not provoke them against the real app. Any other SDK error is logged and passed on.
| What you see | Cause | Fix |
|---|---|---|
| TypeError about an unexpected keyword argument "spider" | Scrapy 2.13 or older | Upgrade to 2.14 or later |
| RuntimeError: Task got bad yield | A Deferred awaited directly in a coroutine | Use asyncio.to_thread or maybe_deferred_to_future |
| 403 pages reach HttpErrorMiddleware and no task is sent | The middleware number is above 590, or the block page has neither an iframe address nor a dd object | Use 551 to 589 and look in the block page for either |
| Pages stay blocked and a budget_used_up count appears in the stats | Cookies off, a Cookie header or a changed User-Agent | Compare the retry with the identity |
| ERROR_PROXY_BANNED or ERROR_PROXY_CONNECT_REFUSED in the log | CapSkip names the address as banned or unreachable | Change the proxy for that jar |
ValidationException with t=bv in the log | The block page says the address is banned | Change the proxy for that jar |
| NetworkException in the log | Nothing answers at the host and port in settings.py | Start CapSkip and check both |
Ordering rules live in the downloader middleware reference of the Scrapy project. For the same idea applied to reCAPTCHA, see the Scrapy CAPTCHA middleware guide, and for crawling at scale see the web scraping page.
Running CapSkip on a server
Server mode makes CapSkip answer on your network address or a public IP, so a crawler on another PC or a Windows VPS can reach it. Choose it in the connection settings of CapSkip and put the address of that machine in DATADOME_CAPSKIP_HOST; nothing else changes. Starting Server mode leaves the CapSkip port open to every address, so add a Windows Firewall rule that blocks the machines which must not reach it. The solve runs on the server, so the cookie is earned from its address unless the request carries a proxy, which the middleware passes on. I ran none of this, only a mock on 127.0.0.1, and with API Key Validation on you would pass apiKey to the CapSkip constructor.
FAQ
Can I await AsyncCapSkip instead of using a thread?
Yes. Build AsyncCapSkip in place of CapSkip and await its datadome method in self.solve. It gave the same counts as the thread on both Pythons, with a longer stall of about 220 to 530 ms on my PC.
How many solves will a real DataDome Scrapy crawl need?
I can’t say. In the mock it was one per proxy, because the mock never expires a cookie. The stats key datadome/solves counts the solves in your own crawl.
