fix(docker): replace headless-shell with real Chrome so kagane's challenge clears

chromedp/headless-shell cannot clear kagane.to's managed challenge. It is a stripped Chrome build, and the tells are structural rather than a header: navigator.webdriver is true, the plugin list is empty, and the client hints are Chromium- rather than Chrome-branded. Overriding webdriver through CDP was tried on its own and changed nothing. Everything below was measured on 2026-08-08 from a single IP, against the same kagane cover, so the comparisons are like for like: chromedp/headless-shell:stable never cleared (90s) zenika/alpine-chrome never cleared - ships Chrome 124, old enough that Cloudflare refuses it and old enough to break chromedp's CDP structs google-chrome, default UA never cleared (60s) - --headless=new advertises "HeadlessChrome" google-chrome, stock UA, UTC never cleared (90s) google-chrome, stock UA, TZ set cleared in ~4s So both remaining tells are load-bearing, and each was tested in isolation. chrome/ is a Debian image with google-chrome-stable, a UA whose version is read back out of the binary at startup (a hardcoded one would drift out of step with the Sec-CH-UA hints on the next Chrome update and become a fresh tell), and no --enable-automation. The timezone matters because Cloudflare scores a browser whose clock zone disagrees with its egress IP's country as a proxy. Note that the usual `-v /etc/localtime:/etc/localtime:ro` does not work here: Chrome resolves the zone through ICU, which takes the name from that path's symlink target and ignores the file's contents, so glibc reports the host zone while Chrome still reports UTC. /etc/timezone carries the name and is mounted instead; BROWSER_TZ overrides it for a host whose clock is UTC in a country that is not. Chrome also binds its DevTools port to loopback and silently ignores --remote-debugging-address, which is why headless-shell fronted it with socat. This image does the same, so it stays a drop-in: the compose service keeps the headless-shell name and its pinned address, and BROWSER_WS_URL is unchanged. Deploying needs `docker compose build headless-shell`.
2026-08-08 23:03:41 +07:00
parent 05e93a4869
commit 5713e3d04a
4 changed files with 126 additions and 13 deletions
@@ -20,6 +20,8 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
 - Userscript run in **isolated world**, so embedded API token safe from site's JS.
 - Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
 - **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
+- **The CDP sidecar must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a clock zone matching the egress IP's country cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
+- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.

 ## Architecture

@@ -37,7 +39,12 @@ Backend (`cd backend`):
 - Single test: `go test -run TestName ./...`
 - Build static binary: `CGO_ENABLED=0 go build`

-Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
+Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). The `headless-shell` service keeps its name but now builds `chrome/` — real Google Chrome, for the reason in the hard constraints above.
+
+Live CDP proof (needs a sidecar and network, skipped otherwise):
+`SMOKE_BROWSER_WS_URL=ws://<host>:<port> go test -run TestSmokeKagane ./internal/latest`
+— fetches a real kagane cover and chapter list. A red run means the challenge is
+not clearing from this IP, which is a live fact to re-check, not necessarily a defect.

 Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.

@@ -0,0 +1,39 @@
+# syntax=docker/dockerfile:1
+
+# Real Google Chrome for the latest-chapter poller and the kagane cover proxy.
+#
+# Not chromedp/headless-shell, which this replaces. headless-shell is a stripped
+# Chrome build and Cloudflare's managed challenge on kagane.to never clears for
+# it: measured 2026-08-08, 60s of a held-open tab still served the interstitial,
+# while stock Chrome from the same IP cleared in ~4s. The tells are structural
+# rather than a header — navigator.webdriver true, an empty plugin list, and
+# Chromium- rather than Chrome-branded client hints. Overriding webdriver alone
+# was tried and did not move it, so the browser build itself is the fix.
+#
+# zenika/alpine-chrome was also tried: its Chrome is 124 (2024), old enough that
+# Cloudflare refuses it outright and old enough to break chromedp's CDP structs.
+FROM debian:trixie-slim
+
+# Chrome is deliberately unpinned, against the usual rule. A pinned build goes
+# stale, and a stale browser is exactly what Cloudflare turns away — the 124 in
+# alpine-chrome is the worked example. Rebuild is the upgrade path.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends ca-certificates wget gnupg \
+ && wget -qO- https://dl.google.com/linux/linux_signing_key.pub \
+      | gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg \
+ && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" \
+      > /etc/apt/sources.list.d/google-chrome.list \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends google-chrome-stable socat \
+ && rm -rf /var/lib/apt/lists/*
+
+# Unprivileged: Chrome refuses to run as root, and the CDP endpoint is a shell
+# on whatever user owns it.
+RUN useradd --create-home --shell /usr/sbin/nologin chrome
+USER chrome
+WORKDIR /home/chrome
+
+COPY entrypoint.sh /entrypoint.sh
+
+EXPOSE 9222
+ENTRYPOINT ["/entrypoint.sh"]
@@ -0,0 +1,60 @@
+#!/bin/sh
+set -eu
+
+# Cloudflare scores a browser whose clock zone disagrees with its egress IP's
+# country as a proxy, and kagane's challenge then never clears (measured
+# 2026-08-08 from a Thai IP: identical container, UTC never cleared in 90s,
+# Asia/Bangkok cleared in 4s). So the zone has to be right, and it has to be
+# right the way Chrome reads it.
+#
+# TZ must carry the zone *name*. Chrome resolves the zone through ICU, which
+# takes the name from /etc/localtime's symlink target and ignores the file's
+# contents; bind-mounting the host's /etc/localtime therefore lands on the
+# image's own symlink to Etc/UTC and leaves glibc reporting +07 while Chrome
+# still reports UTC. /etc/timezone, mounted by docker-compose.yml, is the name.
+[ -n "${TZ:-}" ] || TZ=$(cat /etc/timezone 2>/dev/null || echo UTC)
+export TZ
+
+# Chrome's own UA advertises "HeadlessChrome" under --headless=new, and that one
+# token is the difference between kagane.to's challenge clearing in ~4s and
+# never clearing at all (measured 2026-08-08, same host, same Chrome, only the
+# UA changed). Overriding it does not touch the Sec-CH-UA client hints, which
+# report the real version, so the version is read back out of the binary rather
+# than hardcoded: a hardcoded one would drift out of step with the hints on the
+# next Chrome update and become a fresh tell.
+major=$(google-chrome-stable --version | sed -E 's/[^0-9]*([0-9]+)\..*/\1/')
+ua="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${major}.0.0.0 Safari/537.36"
+
+# Chrome binds its DevTools port to loopback and silently ignores
+# --remote-debugging-address (verified 2026-08-08: Chrome 151 with
+# --remote-debugging-address=0.0.0.0 still listened on 127.0.0.1 only), so the
+# caller — another container — cannot reach it directly. socat fronting the
+# loopback port is how chromedp/headless-shell solved the same problem and is
+# why this image is a drop-in for it.
+#
+# Nothing publishes 9222; reachability is the `browser` network in
+# docker-compose.yml, and an exposed CDP endpoint is remote code execution.
+socat TCP-LISTEN:9222,fork,reuseaddr TCP:127.0.0.1:9223 &
+
+# Chrome stays in the foreground so that its death takes the container down and
+# compose's restart policy applies; a backgrounded browser behind a live socat
+# would leave the sidecar looking healthy while answering nothing.
+#
+# No --enable-automation: it sets navigator.webdriver, the first thing a bot
+# check reads.
+#
+# --no-sandbox because Chrome's zygote wants user namespaces, which Docker's
+# default profile does not hand out; the alternative is --cap-add=SYS_ADMIN,
+# which gives the container strictly more than it takes away. Containment here
+# is the unprivileged user, the isolated network, and the fact that this
+# browser only ever navigates to kagane.to and novelfull.com.
+exec google-chrome-stable \
+  --headless=new \
+  --no-sandbox \
+  --remote-debugging-port=9223 \
+  --user-agent="$ua" \
+  --user-data-dir=/home/chrome/profile \
+  --no-first-run \
+  --no-default-browser-check \
+  --disable-gpu \
+  about:blank
@@ -95,8 +95,24 @@ services:
      - db

  headless-shell:
-    image: chromedp/headless-shell:stable
+    # Real Google Chrome, not chromedp/headless-shell — see chrome/Dockerfile.
+    # The service name is kept so existing overrides and BROWSER_WS_URL stay put.
+    build: ./chrome
+    image: bookmarkmanager-chrome:latest
    restart: unless-stopped
+    environment:
+      # Cloudflare scores a browser whose clock zone disagrees with its egress
+      # IP's country as a proxy, and kagane's challenge then never clears
+      # (measured 2026-08-08: identical container, UTC never cleared in 90s,
+      # Asia/Bangkok cleared in 4s from a Thai IP). Unset falls back to the
+      # host's /etc/timezone below, which is right whenever the host clock is
+      # set to local time; set BROWSER_TZ when the host runs UTC somewhere that
+      # isn't, since it is the IP's country that has to match, not the clock's.
+      TZ: ${BROWSER_TZ:-}
+    volumes:
+      # The zone *name*, which is what Chrome's ICU needs — see chrome/entrypoint.sh.
+      # Absent on a non-Debian host, which the entrypoint handles by falling back to UTC.
+      - /etc/timezone:/etc/timezone:ro
    # Chrome allocates shared memory per tab and dies on Docker's 64MB default.
    shm_size: '1gb'
    # Reaps zombie renderer processes, which otherwise accumulate for the
@@ -104,17 +120,8 @@ services:
    init: true
    # Deliberately no `ports:` — an exposed CDP endpoint is remote code
    # execution. Only bookmark-api, via the `browser` network below, may reach it.
-    # Don't pass --remote-debugging-address/--remote-debugging-port here: the
-    # image's own entrypoint (/headless-shell/run.sh) already starts Chrome on
-    # 127.0.0.1:9223 and fronts it with a socat proxy listening on 0.0.0.0:9222.
-    # Redeclaring the port flag here overrides Chrome's, so it binds 9222
-    # directly (IPv6 loopback only) instead of 9223 — collides with socat's own
-    # bind on 9222 and leaves nothing listening on 9223, so every external
-    # connection to headless-shell:9222 fails with EOF. Only pass flags the
-    # entrypoint doesn't already set.
-    command:
-      - --disable-gpu
-      - --no-sandbox
+    # No `command:` either: every flag this browser needs is in its entrypoint,
+    # and the UA override there is load-bearing for the challenge.
    networks:
      browser:
        # Pinned so BROWSER_WS_URL can name an IP (required, see above) that