How to Reuse cf_clearance Cookies in a Python Scraper

Solving the Cloudflare challenge is only half the job. What you get for solving it is a cf_clearance cookie, and that cookie is what actually keeps the next request from being challenged again. It expires quickly, it is tied to more than most people expect, and it stops working the moment your client drifts away from the one that earned it. This post covers what the cookie is, how to capture it, what silently invalidates it, and how to tell an expired one apart from a broken one.

What you need

• Python 3.10 or newer, with a session-capable HTTP client. The examples use a session so cookies persist across requests.
• A target that actually issues a challenge. A site that never challenges you will never set the cookie, so there is nothing to test against.
• CapSkip running, either in Local mode on the loopback address or in Server mode on a machine your workers can reach. Both are described under connection settings, so pick one before you start.

What the cf_clearance cookie actually is

It is a receipt. The Cloudflare reference describes it as the cookie that stores the proof of challenge passed, used so that a challenge is no longer issued when the cookie is present. It is also the cookie JavaScript detections are stored in, and it is set with SameSite None, Secure and Partitioned so that the state survives cross-site requests.

The lifetime is not yours to choose. It comes from the Challenge Passage setting on the site you are hitting, and Cloudflare documents the default plainly: the cf_clearance cookie has a lifetime of 30 minutes, with 15 to 45 minutes suggested as the sensible range. Some sites shorten it. Some lengthen it. You cannot read the value from outside, so treat every clearance as short-lived and build for the refresh rather than hoping it lasts.

Three practical consequences fall out of that. You should keep the cookie, because re-solving on every request is slow and wasteful. You should never assume it survives a restart. And you should have a code path that notices it has gone stale and quietly earns a new one.

Step 1: solve the challenge, then keep the whole client

The mistake worth avoiding first is treating the token as the prize. It is not. The token you get from solving a Turnstile challenge is what you exchange for clearance, and the cookie you get back is the thing with value afterwards. So the sequence is solve, submit, then hold on to the session that received the response.

# pip install capskip
from capskip import CapSkip

solver = CapSkip(host="127.0.0.1", port=8080)

# A challenge page needs two more values than a plain widget does.
result = solver.turnstile(
    sitekey="YOUR_SITEKEY",
    url="https://example.com/protected",
    data="YOUR_CDATA",
    pagedata="YOUR_CHLPAGEDATA",
)

token = result["code"]
agent = result["userAgent"]   # not optional, see the next section

Submit that token the way the page itself would, from a session you intend to keep. Once the response comes back, the clearance cookie is sitting in the cookie jar for that session and you can read it straight out.

# pip install requests
import requests

s = requests.Session()
s.headers["User-Agent"] = agent      # the exact agent the solver returned

# ... submit the token here, exactly as the challenge page does ...

clearance = s.cookies.get("cf_clearance")
print(bool(clearance))               # True once the challenge is cleared

Step 2: know what quietly invalidates it

Cloudflare does not publish the exact binding, so this table is observed behaviour rather than a documented contract. It is consistent enough to build on, and every row here has cost somebody an afternoon.

The first three rows are one rule wearing three hats: the clearance belongs to a client, not to you. So pin the identity that earned it. That means one user agent, one exit IP and one TLS profile for the whole life of that cookie. Rotating a proxy mid-session throws away clearance you already paid for, which is the single most common way a working scraper starts challenging again for no visible reason.

This is also why the solver hands back a user agent rather than only a token. Turnstile ties the token to the browser identity that produced it, so a token submitted under a different agent gets rejected even though the token itself is perfectly valid. Use the returned value verbatim, and keep using it for every request that carries the resulting cookie. The Turnstile challenge page walkthrough shows where the two extra input values come from, which is the other half people get wrong.

Step 3: persist it across runs

A clearance cookie that dies with your process is worth much less than one that survives a restart. Store the cookie jar alongside the identity that goes with it, because the cookie on its own is useless if you reload it under a different agent or a different proxy.

# pip install requests
import json, time

def save_clearance(session, agent, proxy, path="clearance.json"):
    """Store the cookie with the identity that earned it."""
    blob = {
        "cf_clearance": session.cookies.get("cf_clearance"),
        "user_agent": agent,
        "proxy": proxy,
        "stored_at": time.time(),
    }
    with open(path, "w") as fh:
        json.dump(blob, fh)

Reloading is the mirror image, with one extra check. Age the record out yourself instead of waiting to be challenged, because a proactive refresh costs one solve and a reactive one costs a failed request first.

# pip install requests
import json, time, requests

def load_clearance(path="clearance.json", max_age=900):
    """Return a ready session, or None if the record is too old."""
    with open(path) as fh:
        blob = json.load(fh)

    # 15 minutes, comfortably inside a 30 minute default.
    if time.time() - blob["stored_at"] > max_age:
        return None

    s = requests.Session()
    s.headers["User-Agent"] = blob["user_agent"]
    s.proxies = {"https": blob["proxy"]} if blob["proxy"] else {}
    s.cookies.set("cf_clearance", blob["cf_clearance"])
    return s

Fifteen minutes is a deliberately conservative ceiling. You cannot see the Challenge Passage value a site has configured, the default is 30 minutes, and refreshing halfway through costs one cheap solve rather than a broken batch. If the solver runs on your own hardware with no per-solve charge, being early is free.

Step 4: detect a stale clearance without guessing

A dead clearance does not announce itself with a clean error. You will usually get a normal-looking 403, or a 200 carrying an HTML interstitial rather than the JSON you asked for. Checking the status code alone will not catch the second case, and a retry loop that only watches status codes will happily loop on it forever.

# pip install requests
def needs_new_clearance(response):
    """True when this response is a challenge rather than content."""
    if response.status_code in (403, 503):
        return True
    body = response.text[:4000].lower()
    markers = ("cf-turnstile", "challenge-platform", "just a moment")
    return any(m in body for m in markers)

Wire that in front of your parser, not after it. When it returns true, discard the stored record, solve once, and retry with the fresh session. Do not retry with the old cookie and a longer sleep, because time is not what is wrong with it. The same distinction shows up with reCAPTCHA tokens, where the expiry window is much shorter still, and the post on how long a reCAPTCHA token stays valid covers that side of it.

Running the solver on a server instead

Clearance cookies are per identity, so a fleet of workers needs a fleet of clearances, and every one of them needs solving. That work does not have to happen on the worker. The connection settings cover both arrangements:

Point the SDK host at the solver machine and nothing else in your code changes, so twenty workers can share one instance while each keeps its own cookie jar. A static public IP is recommended when the callers sit outside your own network. The details live under connection settings, and Server mode is still your hardware and still unmetered: it moves where the solver runs, never who owns it.

FAQ

The shortest version

Treat the cf_clearance cookie as a short-lived receipt tied to one client. Capture it from the session that cleared the challenge, store it with the user agent and the proxy that earned it, and refresh on a timer of about fifteen minutes rather than waiting to be blocked. Check the body and not just the status code, because a challenge arrives with a perfectly healthy 200. When it does expire, one fresh solve is the whole fix, and a captcha solver running on your own hardware makes that cheap enough to do early. For the two input shapes a Turnstile solve can take, read the Cloudflare Turnstile solver page before you wire anything up. Worker-pool wiring is covered separately under CAPTCHA solver for web scraping, which is worth twenty minutes if you run more than one worker.