MechanicalSoup never runs the CAPTCHA widget, so the field the widget would fill in is usually missing from the form, and setting it raises LinkNotFoundError. The fix for a MechanicalSoup CAPTCHA takes three steps. Read the sitekey from the page MechanicalSoup already parsed, and solve it with CapSkip running on your own Windows machine. Then add the g-recaptcha-response field yourself with new_control before you call submit_selected. MechanicalSoup handles the rest: the session cookie, the hidden CSRF field and the Referer header a strict server checks for. This guide covers the reCAPTCHA checkbox, the invisible button, v3, Enterprise, the other widgets CapSkip solves, many logins at once and running the solver on another machine.<\/p>\n
# Python 3.10 or later; MechanicalSoup brings requests, bs4 and lxml\npip install capskip mechanicalsoup<\/pre>\n<\/div>\nStep 1: open the page and find the sitekey<\/h2>\n
A StatefulBrowser is a requests session with a parsed page attached. It keeps every cookie the site sets, follows redirects, and remembers the address it ended up on. Open the login page and look for the reCAPTCHA element:<\/p>\n
\n# pip install capskip mechanicalsoup\nimport mechanicalsoup\n\nPAGE_URL = "https:\/\/example.com\/login"\n\n# One StatefulBrowser per session: it keeps cookies, follows redirects\n# and sends the current page as Referer when it submits a form.\nbrowser = mechanicalsoup.StatefulBrowser(raise_on_404=True)\nbrowser.open(PAGE_URL)\n\n# The checkbox div, or the button on invisible reCAPTCHA, holds the key.\nwidget = browser.page.select_one(".g-recaptcha[data-sitekey]")\nif widget is None:\n raise RuntimeError("No reCAPTCHA widget in the HTML; see the v3 section.")\nsitekey = widget["data-sitekey"]\n\n# The form around the widget; a page often has a search form first.\nbrowser.select_form(widget.find_parent("form") or 'form[method="post" i]')\nprint(sitekey, browser.url)<\/pre>\n<\/div>\nThe checkbox widget is a div with the class g-recaptcha and a data-sitekey attribute. Invisible reCAPTCHA usually puts the same class and attribute on the submit button instead, so one selector finds both. A page that loads api.js with render=explicit builds the widget from its own script and passes the key to grecaptcha.render, so read it from that call in DevTools. Select the form around the widget rather than the first form on the page. Many sites put a search form in the header, and select_form picks that one by default. When the widget sits outside its form, the fallback takes the first form that posts, so pass the login form’s id instead if the page has two.<\/p>\n
browser.url is the address the page came from after any redirects. That is the URL to send to the solver, because a sitekey is registered for its site’s domains and a solve for the wrong page is wasted. raise_on_404 makes a missing page fail loudly at open rather than further down.<\/p>\n
Step 2: solve the reCAPTCHA with CapSkip<\/h2>\n
One call to the capskip package does the solving. CapSkip solves the CAPTCHA on your Windows machine and returns the token that the widget would have put in the form:<\/p>\n
\nfrom capskip import CapSkip\n\nsolver = CapSkip(host="127.0.0.1", port=8080)\n\n# enterprise.js in place of api.js means reCAPTCHA Enterprise.\nenterprise = int("recaptcha\/enterprise.js" in str(browser.page))\nif widget.name == "button" and widget.get("data-action"):\n # Google's v3 button binding: a g-recaptcha button with an action is v3.\n result = solver.recaptcha(sitekey=sitekey, url=browser.url, version="v3",\n action=widget["data-action"], enterprise=enterprise)\nelse:\n # A g-recaptcha button, or data-size="invisible", is the invisible widget.\n invisible = widget.name == "button" or widget.get("data-size") == "invisible"\n result = solver.recaptcha(sitekey=sitekey, url=browser.url,\n invisible=int(invisible), enterprise=enterprise)\n\ntoken = result["code"]\nprint(token[:40]) # what the widget would have written into the form<\/pre>\n<\/div>\nThe checkbox is the bare call, and invisible reCAPTCHA is the same call with invisible set to 1. A g-recaptcha button that also carries data-action is Google’s v3 button binding, so the script solves it as v3 with that action. Pages that load enterprise.js instead of api.js run reCAPTCHA Enterprise, and enterprise=1 tells CapSkip so. The token sits in result["code"]. It is valid for about two minutes and verifies only once, so solve right before you submit and solve again for every login.<\/p>\n
recaptcha polls for up to recaptchaTimeout, 300 seconds by default. Set a different value on the constructor, because a timeout passed per call raises TypeError: the method already passes one of its own. CapSkip keeps its own clock as well. A task waits up to 250 seconds (Wait Timeout) for a free reCAPTCHA thread, and the solve then has a limit of its own, Row Timeout, 250 seconds by default. When one of those runs out, CapSkip fails the task and the SDK raises ApiException with ERROR_CAPTCHA_UNSOLVABLE. The CapSkip client used here sets no timeout on its own HTTP calls, so a CapSkip that accepts the connection and never answers holds the call open.<\/p>\n
Step 3: add the field the widget would have written, then submit<\/h2>\n
This is the step that trips up every MechanicalSoup CAPTCHA script. In a real browser, Google’s script creates a hidden textarea named g-recaptcha-response and fills it when the CAPTCHA is solved. MechanicalSoup parses the HTML the server sent and runs no JavaScript, so on most pages that field does not exist. Assigning to it the usual way then raises LinkNotFoundError with the message No valid element named g-recaptcha-response. new_control adds the field to the selected form:<\/p>\n
\n# This line raises LinkNotFoundError on most pages: the widget script\n# creates the field in a real browser, so the parsed HTML lacks it.\n# browser["g-recaptcha-response"] = token\n\n# Add the field the widget would have written, replacing any copy.\nbrowser.new_control("hidden", "g-recaptcha-response", token)\nbrowser["email"] = "YOUR_EMAIL"\nbrowser["password"] = "YOUR_PASSWORD"\n\nresponse = browser.submit_selected()\nprint(response.status_code, browser.url)<\/pre>\n<\/div>\nnew_control removes any field of the same name before adding its own, which matters on pages that keep Google’s noscript fallback. That fallback includes a textarea with the same name, and MechanicalSoup’s parser reads it as part of the form. On those pages the plain assignment works too, but new_control covers both kinds of page with one line.<\/p>\n
submit_selected sends every field the form carries, the hidden CSRF token included, on the same session that loaded the page. It also sends the page’s address as the Referer header. Plain requests sends none, and neither does MechanicalSoup’s lower-level Browser.submit method. Some frameworks, Django among them, refuse an HTTPS form post without a Referer or Origin, however right the cookie and token are. So stay with submit_selected on the browser that opened the page. If the form has several submit buttons, pass the name of the one you want as btnName. And a 200 does not prove the login worked. Many sites answer a refused CAPTCHA with the same form and a 200, so check browser.page for something only a signed-in page shows.<\/p>\n
reCAPTCHA v3 has no widget to find<\/h3>\n
v3 asks the user nothing. The site loads Google’s script with its sitekey in the address, after render=, and its own script asks for a token with an action name such as login. Most v3 pages then write the token into a hidden input themselves. Read both values out of the HTML, note whether the script is api.js or enterprise.js, and pass them on with version set to v3. When a page binds v3 to a button instead, Step 2 already handles it.<\/p>\n
\nimport re\n\n# A v3 page has no widget element; the key sits in the script tags.\nhtml = str(browser.page)\nkey = re.search(r"recaptcha\/(api|enterprise)\\.js\\?[^\\"'<>\\s]*?\\brender=(?!explicit\\b)([\\w-]+)", html)\naction = re.search(r"execute\\(\\s*[^,()]+,\\s*\\{\\s*[\\"']?action[\\"']?\\s*:\\s*[\\"']([\\w\/]+)", html)\nif key is None:\n raise RuntimeError("No v3 render key in the HTML; check DevTools.")\n\nv3 = solver.recaptcha(sitekey=key.group(2), url=browser.url, version="v3",\n enterprise=int(key.group(1) == "enterprise"),\n **({"action": action.group(1)} if action else {}))\n\nbrowser.select_form('form[method="post" i]')\n# The field the page's own script fills in; read its name in DevTools.\nbrowser.new_control("hidden", "g-recaptcha-response", v3["code"])<\/pre>\n<\/div>\nOn a v3 page, run Step 1 only as far as browser.open, then the lines of Step 2 that create the solver, then this block, then Step 3’s credentials and submit_selected. The action matters because the site’s backend usually checks that the token was issued for the action it expected. When the pattern finds no action, the call goes without one and CapSkip uses verify, which a site expecting login rejects. If the execute call sits in an external script, read the action in DevTools and pass it yourself. The field name is whatever the page’s script fills in, often g-recaptcha-response but not always, so check it in DevTools by submitting once by hand. Then fill in the credentials and call submit_selected as above. Google’s reCAPTCHA v3 documentation<\/a> shows the render and execute calls that these patterns match. The guide to solving reCAPTCHA v3 in Python<\/a> covers the call itself in more depth.<\/p>\n
Other widgets work the same way<\/h3>\n
Every widget CapSkip solves follows the same pattern in MechanicalSoup: read the key, solve, add the field the widget would have created, submit. Only the selector, the field name and the method change:<\/p>\n
\n\n
\n \nWidget<\/th>\n Where the key is<\/th>\n Field to add<\/th>\n capskip method<\/th>\n<\/tr>\n<\/thead>\n \n reCAPTCHA v2 checkbox<\/td>\n div.g-recaptcha, data-sitekey<\/td>\n g-recaptcha-response<\/td>\n solver.recaptcha()<\/td>\n<\/tr>\n \n reCAPTCHA v2 invisible<\/td>\n the button with the g-recaptcha class, or a div with data-size=invisible<\/td>\n g-recaptcha-response<\/td>\n solver.recaptcha() with invisible=1<\/td>\n<\/tr>\n \n reCAPTCHA v3<\/td>\n api.js?render= or enterprise.js?render= in the script address, or a g-recaptcha button with data-action<\/td>\n the input the page’s script fills<\/td>\n solver.recaptcha() with version="v3" and the action<\/td>\n<\/tr>\n \n reCAPTCHA Enterprise<\/td>\n enterprise.js in place of api.js<\/td>\n as for v2 or v3<\/td>\n solver.recaptcha() with enterprise=1<\/td>\n<\/tr>\n \n Cloudflare Turnstile widget<\/td>\n div.cf-turnstile, data-sitekey<\/td>\n cf-turnstile-response<\/td>\n solver.turnstile()<\/td>\n<\/tr>\n \n Friendly Captcha<\/td>\n div.frc-captcha, data-sitekey<\/td>\n frc-captcha-response on v2, frc-captcha-solution on v1<\/td>\n solver.friendly_captcha() with version="v1" or "v2"<\/td>\n<\/tr>\n \n CaptchaFox<\/td>\n div.captchafox, data-sitekey<\/td>\n cf-captcha-response<\/td>\n solver.captchafox()<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n