fix(docker): replace headless-shell with real Chrome so kagane's challenge clears
chromedp/headless-shell cannot clear kagane.to's managed challenge. It is
a stripped Chrome build, and the tells are structural rather than a
header: navigator.webdriver is true, the plugin list is empty, and the
client hints are Chromium- rather than Chrome-branded. Overriding
webdriver through CDP was tried on its own and changed nothing.
Everything below was measured on 2026-08-08 from a single IP, against the
same kagane cover, so the comparisons are like for like:
chromedp/headless-shell:stable never cleared (90s)
zenika/alpine-chrome never cleared - ships Chrome 124, old
enough that Cloudflare refuses it and
old enough to break chromedp's CDP structs
google-chrome, default UA never cleared (60s) - --headless=new
advertises "HeadlessChrome"
google-chrome, stock UA, UTC never cleared (90s)
google-chrome, stock UA, TZ set cleared in ~4s
So both remaining tells are load-bearing, and each was tested in
isolation. chrome/ is a Debian image with google-chrome-stable, a UA
whose version is read back out of the binary at startup (a hardcoded one
would drift out of step with the Sec-CH-UA hints on the next Chrome
update and become a fresh tell), and no --enable-automation.
The timezone matters because Cloudflare scores a browser whose clock zone
disagrees with its egress IP's country as a proxy. Note that the usual
`-v /etc/localtime:/etc/localtime:ro` does not work here: Chrome resolves
the zone through ICU, which takes the name from that path's symlink
target and ignores the file's contents, so glibc reports the host zone
while Chrome still reports UTC. /etc/timezone carries the name and is
mounted instead; BROWSER_TZ overrides it for a host whose clock is UTC in
a country that is not.
Chrome also binds its DevTools port to loopback and silently ignores
--remote-debugging-address, which is why headless-shell fronted it with
socat. This image does the same, so it stays a drop-in: the compose
service keeps the headless-shell name and its pinned address, and
BROWSER_WS_URL is unchanged.
Deploying needs `docker compose build headless-shell`.
This commit is contained in:
@@ -20,6 +20,8 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
|
||||
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
||||
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
||||
- **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
|
||||
- **The CDP sidecar must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a clock zone matching the egress IP's country cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
|
||||
- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -37,7 +39,12 @@ Backend (`cd backend`):
|
||||
- Single test: `go test -run TestName ./...`
|
||||
- Build static binary: `CGO_ENABLED=0 go build`
|
||||
|
||||
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
|
||||
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). The `headless-shell` service keeps its name but now builds `chrome/` — real Google Chrome, for the reason in the hard constraints above.
|
||||
|
||||
Live CDP proof (needs a sidecar and network, skipped otherwise):
|
||||
`SMOKE_BROWSER_WS_URL=ws://<host>:<port> go test -run TestSmokeKagane ./internal/latest`
|
||||
— fetches a real kagane cover and chapter list. A red run means the challenge is
|
||||
not clearing from this IP, which is a live fact to re-check, not necessarily a defect.
|
||||
|
||||
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user