AGENTS.md

@@ -20,6 +20,8 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
 - Userscript run in **isolated world**, so embedded API token safe from site's JS.
 - Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
 - **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
+- **The CDP sidecar must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a clock zone matching the egress IP's country cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
+- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
 
 ## Architecture
 
@@ -37,7 +39,12 @@ Backend (`cd backend`):
 - Single test: `go test -run TestName ./...`
 - Build static binary: `CGO_ENABLED=0 go build`
 
-Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
+Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). The `headless-shell` service keeps its name but now builds `chrome/` — real Google Chrome, for the reason in the hard constraints above.
+
+Live CDP proof (needs a sidecar and network, skipped otherwise):
+`SMOKE_BROWSER_WS_URL=ws://<host>:<port> go test -run TestSmokeKagane ./internal/latest`
+— fetches a real kagane cover and chapter list. A red run means the challenge is
+not clearing from this IP, which is a live fact to re-check, not necessarily a defect.
 
 Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
 

chrome/Dockerfile

@@ -0,0 +1,39 @@
+# syntax=docker/dockerfile:1
+
+# Real Google Chrome for the latest-chapter poller and the kagane cover proxy.
+#
+# Not chromedp/headless-shell, which this replaces. headless-shell is a stripped
+# Chrome build and Cloudflare's managed challenge on kagane.to never clears for
+# it: measured 2026-08-08, 60s of a held-open tab still served the interstitial,
+# while stock Chrome from the same IP cleared in ~4s. The tells are structural
+# rather than a header — navigator.webdriver true, an empty plugin list, and
+# Chromium- rather than Chrome-branded client hints. Overriding webdriver alone
+# was tried and did not move it, so the browser build itself is the fix.
+#
+# zenika/alpine-chrome was also tried: its Chrome is 124 (2024), old enough that
+# Cloudflare refuses it outright and old enough to break chromedp's CDP structs.
+FROM debian:trixie-slim
+
+# Chrome is deliberately unpinned, against the usual rule. A pinned build goes
+# stale, and a stale browser is exactly what Cloudflare turns away — the 124 in
+# alpine-chrome is the worked example. Rebuild is the upgrade path.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends ca-certificates wget gnupg \
+ && wget -qO- https://dl.google.com/linux/linux_signing_key.pub \
+      | gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg \
+ && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" \
+      > /etc/apt/sources.list.d/google-chrome.list \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends google-chrome-stable socat \
+ && rm -rf /var/lib/apt/lists/*
+
+# Unprivileged: Chrome refuses to run as root, and the CDP endpoint is a shell
+# on whatever user owns it.
+RUN useradd --create-home --shell /usr/sbin/nologin chrome
+USER chrome
+WORKDIR /home/chrome
+
+COPY entrypoint.sh /entrypoint.sh
+
+EXPOSE 9222
+ENTRYPOINT ["/entrypoint.sh"]

chrome/entrypoint.sh

@@ -0,0 +1,60 @@
+#!/bin/sh
+set -eu
+
+# Cloudflare scores a browser whose clock zone disagrees with its egress IP's
+# country as a proxy, and kagane's challenge then never clears (measured
+# 2026-08-08 from a Thai IP: identical container, UTC never cleared in 90s,
+# Asia/Bangkok cleared in 4s). So the zone has to be right, and it has to be
+# right the way Chrome reads it.
+#
+# TZ must carry the zone *name*. Chrome resolves the zone through ICU, which
+# takes the name from /etc/localtime's symlink target and ignores the file's
+# contents; bind-mounting the host's /etc/localtime therefore lands on the
+# image's own symlink to Etc/UTC and leaves glibc reporting +07 while Chrome
+# still reports UTC. /etc/timezone, mounted by docker-compose.yml, is the name.
+[ -n "${TZ:-}" ] || TZ=$(cat /etc/timezone 2>/dev/null || echo UTC)
+export TZ
+
+# Chrome's own UA advertises "HeadlessChrome" under --headless=new, and that one
+# token is the difference between kagane.to's challenge clearing in ~4s and
+# never clearing at all (measured 2026-08-08, same host, same Chrome, only the
+# UA changed). Overriding it does not touch the Sec-CH-UA client hints, which
+# report the real version, so the version is read back out of the binary rather
+# than hardcoded: a hardcoded one would drift out of step with the hints on the
+# next Chrome update and become a fresh tell.
+major=$(google-chrome-stable --version | sed -E 's/[^0-9]*([0-9]+)\..*/\1/')
+ua="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${major}.0.0.0 Safari/537.36"
+
+# Chrome binds its DevTools port to loopback and silently ignores
+# --remote-debugging-address (verified 2026-08-08: Chrome 151 with
+# --remote-debugging-address=0.0.0.0 still listened on 127.0.0.1 only), so the
+# caller — another container — cannot reach it directly. socat fronting the
+# loopback port is how chromedp/headless-shell solved the same problem and is
+# why this image is a drop-in for it.
+#
+# Nothing publishes 9222; reachability is the `browser` network in
+# docker-compose.yml, and an exposed CDP endpoint is remote code execution.
+socat TCP-LISTEN:9222,fork,reuseaddr TCP:127.0.0.1:9223 &
+
+# Chrome stays in the foreground so that its death takes the container down and
+# compose's restart policy applies; a backgrounded browser behind a live socat
+# would leave the sidecar looking healthy while answering nothing.
+#
+# No --enable-automation: it sets navigator.webdriver, the first thing a bot
+# check reads.
+#
+# --no-sandbox because Chrome's zygote wants user namespaces, which Docker's
+# default profile does not hand out; the alternative is --cap-add=SYS_ADMIN,
+# which gives the container strictly more than it takes away. Containment here
+# is the unprivileged user, the isolated network, and the fact that this
+# browser only ever navigates to kagane.to and novelfull.com.
+exec google-chrome-stable \
+  --headless=new \
+  --no-sandbox \
+  --remote-debugging-port=9223 \
+  --user-agent="$ua" \
+  --user-data-dir=/home/chrome/profile \
+  --no-first-run \
+  --no-default-browser-check \
+  --disable-gpu \
+  about:blank

docker-compose.yml

@@ -95,8 +95,24 @@ services:
       - db
 
   headless-shell:
-    image: chromedp/headless-shell:stable
+    # Real Google Chrome, not chromedp/headless-shell — see chrome/Dockerfile.
+    # The service name is kept so existing overrides and BROWSER_WS_URL stay put.
+    build: ./chrome
+    image: bookmarkmanager-chrome:latest
     restart: unless-stopped
+    environment:
+      # Cloudflare scores a browser whose clock zone disagrees with its egress
+      # IP's country as a proxy, and kagane's challenge then never clears
+      # (measured 2026-08-08: identical container, UTC never cleared in 90s,
+      # Asia/Bangkok cleared in 4s from a Thai IP). Unset falls back to the
+      # host's /etc/timezone below, which is right whenever the host clock is
+      # set to local time; set BROWSER_TZ when the host runs UTC somewhere that
+      # isn't, since it is the IP's country that has to match, not the clock's.
+      TZ: ${BROWSER_TZ:-}
+    volumes:
+      # The zone *name*, which is what Chrome's ICU needs — see chrome/entrypoint.sh.
+      # Absent on a non-Debian host, which the entrypoint handles by falling back to UTC.
+      - /etc/timezone:/etc/timezone:ro
     # Chrome allocates shared memory per tab and dies on Docker's 64MB default.
     shm_size: '1gb'
     # Reaps zombie renderer processes, which otherwise accumulate for the
@@ -104,17 +120,8 @@ services:
     init: true
     # Deliberately no `ports:` — an exposed CDP endpoint is remote code
     # execution. Only bookmark-api, via the `browser` network below, may reach it.
-    # Don't pass --remote-debugging-address/--remote-debugging-port here: the
-    # image's own entrypoint (/headless-shell/run.sh) already starts Chrome on
-    # 127.0.0.1:9223 and fronts it with a socat proxy listening on 0.0.0.0:9222.
-    # Redeclaring the port flag here overrides Chrome's, so it binds 9222
-    # directly (IPv6 loopback only) instead of 9223 — collides with socat's own
-    # bind on 9222 and leaves nothing listening on 9223, so every external
-    # connection to headless-shell:9222 fails with EOF. Only pass flags the
-    # entrypoint doesn't already set.
-    command:
-      - --disable-gpu
-      - --no-sandbox
+    # No `command:` either: every flag this browser needs is in its entrypoint,
+    # and the UA override there is load-bearing for the challenge.
     networks:
       browser:
         # Pinned so BROWSER_WS_URL can name an IP (required, see above) that