diff --git a/AGENTS.md b/AGENTS.md index 2447414..dea9c68 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,8 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free - Userscript run in **isolated world**, so embedded API token safe from site's JS. - Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test. - **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS. +- **The CDP sidecar must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a clock zone matching the egress IP's country cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead. +- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives. ## Architecture @@ -37,7 +39,12 @@ Backend (`cd backend`): - Single test: `go test -run TestName ./...` - Build static binary: `CGO_ENABLED=0 go build` -Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). +Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). The `headless-shell` service keeps its name but now builds `chrome/` — real Google Chrome, for the reason in the hard constraints above. + +Live CDP proof (needs a sidecar and network, skipped otherwise): +`SMOKE_BROWSER_WS_URL=ws://: go test -run TestSmokeKagane ./internal/latest` +— fetches a real kagane cover and chapter list. A red run means the challenge is +not clearing from this IP, which is a live fact to re-check, not necessarily a defect. Smoke test: `curl` endpoints with `Authorization: Bearer `; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200. diff --git a/chrome/Dockerfile b/chrome/Dockerfile new file mode 100644 index 0000000..714d700 --- /dev/null +++ b/chrome/Dockerfile @@ -0,0 +1,39 @@ +# syntax=docker/dockerfile:1 + +# Real Google Chrome for the latest-chapter poller and the kagane cover proxy. +# +# Not chromedp/headless-shell, which this replaces. headless-shell is a stripped +# Chrome build and Cloudflare's managed challenge on kagane.to never clears for +# it: measured 2026-08-08, 60s of a held-open tab still served the interstitial, +# while stock Chrome from the same IP cleared in ~4s. The tells are structural +# rather than a header — navigator.webdriver true, an empty plugin list, and +# Chromium- rather than Chrome-branded client hints. Overriding webdriver alone +# was tried and did not move it, so the browser build itself is the fix. +# +# zenika/alpine-chrome was also tried: its Chrome is 124 (2024), old enough that +# Cloudflare refuses it outright and old enough to break chromedp's CDP structs. +FROM debian:trixie-slim + +# Chrome is deliberately unpinned, against the usual rule. A pinned build goes +# stale, and a stale browser is exactly what Cloudflare turns away — the 124 in +# alpine-chrome is the worked example. Rebuild is the upgrade path. +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates wget gnupg \ + && wget -qO- https://dl.google.com/linux/linux_signing_key.pub \ + | gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg \ + && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" \ + > /etc/apt/sources.list.d/google-chrome.list \ + && apt-get update \ + && apt-get install -y --no-install-recommends google-chrome-stable socat \ + && rm -rf /var/lib/apt/lists/* + +# Unprivileged: Chrome refuses to run as root, and the CDP endpoint is a shell +# on whatever user owns it. +RUN useradd --create-home --shell /usr/sbin/nologin chrome +USER chrome +WORKDIR /home/chrome + +COPY entrypoint.sh /entrypoint.sh + +EXPOSE 9222 +ENTRYPOINT ["/entrypoint.sh"] diff --git a/chrome/entrypoint.sh b/chrome/entrypoint.sh new file mode 100755 index 0000000..cc0dbad --- /dev/null +++ b/chrome/entrypoint.sh @@ -0,0 +1,60 @@ +#!/bin/sh +set -eu + +# Cloudflare scores a browser whose clock zone disagrees with its egress IP's +# country as a proxy, and kagane's challenge then never clears (measured +# 2026-08-08 from a Thai IP: identical container, UTC never cleared in 90s, +# Asia/Bangkok cleared in 4s). So the zone has to be right, and it has to be +# right the way Chrome reads it. +# +# TZ must carry the zone *name*. Chrome resolves the zone through ICU, which +# takes the name from /etc/localtime's symlink target and ignores the file's +# contents; bind-mounting the host's /etc/localtime therefore lands on the +# image's own symlink to Etc/UTC and leaves glibc reporting +07 while Chrome +# still reports UTC. /etc/timezone, mounted by docker-compose.yml, is the name. +[ -n "${TZ:-}" ] || TZ=$(cat /etc/timezone 2>/dev/null || echo UTC) +export TZ + +# Chrome's own UA advertises "HeadlessChrome" under --headless=new, and that one +# token is the difference between kagane.to's challenge clearing in ~4s and +# never clearing at all (measured 2026-08-08, same host, same Chrome, only the +# UA changed). Overriding it does not touch the Sec-CH-UA client hints, which +# report the real version, so the version is read back out of the binary rather +# than hardcoded: a hardcoded one would drift out of step with the hints on the +# next Chrome update and become a fresh tell. +major=$(google-chrome-stable --version | sed -E 's/[^0-9]*([0-9]+)\..*/\1/') +ua="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${major}.0.0.0 Safari/537.36" + +# Chrome binds its DevTools port to loopback and silently ignores +# --remote-debugging-address (verified 2026-08-08: Chrome 151 with +# --remote-debugging-address=0.0.0.0 still listened on 127.0.0.1 only), so the +# caller — another container — cannot reach it directly. socat fronting the +# loopback port is how chromedp/headless-shell solved the same problem and is +# why this image is a drop-in for it. +# +# Nothing publishes 9222; reachability is the `browser` network in +# docker-compose.yml, and an exposed CDP endpoint is remote code execution. +socat TCP-LISTEN:9222,fork,reuseaddr TCP:127.0.0.1:9223 & + +# Chrome stays in the foreground so that its death takes the container down and +# compose's restart policy applies; a backgrounded browser behind a live socat +# would leave the sidecar looking healthy while answering nothing. +# +# No --enable-automation: it sets navigator.webdriver, the first thing a bot +# check reads. +# +# --no-sandbox because Chrome's zygote wants user namespaces, which Docker's +# default profile does not hand out; the alternative is --cap-add=SYS_ADMIN, +# which gives the container strictly more than it takes away. Containment here +# is the unprivileged user, the isolated network, and the fact that this +# browser only ever navigates to kagane.to and novelfull.com. +exec google-chrome-stable \ + --headless=new \ + --no-sandbox \ + --remote-debugging-port=9223 \ + --user-agent="$ua" \ + --user-data-dir=/home/chrome/profile \ + --no-first-run \ + --no-default-browser-check \ + --disable-gpu \ + about:blank diff --git a/docker-compose.yml b/docker-compose.yml index 05cd2ad..78cc2e9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -95,8 +95,24 @@ services: - db headless-shell: - image: chromedp/headless-shell:stable + # Real Google Chrome, not chromedp/headless-shell — see chrome/Dockerfile. + # The service name is kept so existing overrides and BROWSER_WS_URL stay put. + build: ./chrome + image: bookmarkmanager-chrome:latest restart: unless-stopped + environment: + # Cloudflare scores a browser whose clock zone disagrees with its egress + # IP's country as a proxy, and kagane's challenge then never clears + # (measured 2026-08-08: identical container, UTC never cleared in 90s, + # Asia/Bangkok cleared in 4s from a Thai IP). Unset falls back to the + # host's /etc/timezone below, which is right whenever the host clock is + # set to local time; set BROWSER_TZ when the host runs UTC somewhere that + # isn't, since it is the IP's country that has to match, not the clock's. + TZ: ${BROWSER_TZ:-} + volumes: + # The zone *name*, which is what Chrome's ICU needs — see chrome/entrypoint.sh. + # Absent on a non-Debian host, which the entrypoint handles by falling back to UTC. + - /etc/timezone:/etc/timezone:ro # Chrome allocates shared memory per tab and dies on Docker's 64MB default. shm_size: '1gb' # Reaps zombie renderer processes, which otherwise accumulate for the @@ -104,17 +120,8 @@ services: init: true # Deliberately no `ports:` — an exposed CDP endpoint is remote code # execution. Only bookmark-api, via the `browser` network below, may reach it. - # Don't pass --remote-debugging-address/--remote-debugging-port here: the - # image's own entrypoint (/headless-shell/run.sh) already starts Chrome on - # 127.0.0.1:9223 and fronts it with a socat proxy listening on 0.0.0.0:9222. - # Redeclaring the port flag here overrides Chrome's, so it binds 9222 - # directly (IPv6 loopback only) instead of 9223 — collides with socat's own - # bind on 9222 and leaves nothing listening on 9223, so every external - # connection to headless-shell:9222 fails with EOF. Only pass flags the - # entrypoint doesn't already set. - command: - - --disable-gpu - - --no-sandbox + # No `command:` either: every flag this browser needs is in its entrypoint, + # and the UA override there is load-bearing for the challenge. networks: browser: # Pinned so BROWSER_WS_URL can name an IP (required, see above) that