sulthan
5713e3d04a
chromedp/headless-shell cannot clear kagane.to's managed challenge. It is
a stripped Chrome build, and the tells are structural rather than a
header: navigator.webdriver is true, the plugin list is empty, and the
client hints are Chromium- rather than Chrome-branded. Overriding
webdriver through CDP was tried on its own and changed nothing.
Everything below was measured on 2026-08-08 from a single IP, against the
same kagane cover, so the comparisons are like for like:
chromedp/headless-shell:stable never cleared (90s)
zenika/alpine-chrome never cleared - ships Chrome 124, old
enough that Cloudflare refuses it and
old enough to break chromedp's CDP structs
google-chrome, default UA never cleared (60s) - --headless=new
advertises "HeadlessChrome"
google-chrome, stock UA, UTC never cleared (90s)
google-chrome, stock UA, TZ set cleared in ~4s
So both remaining tells are load-bearing, and each was tested in
isolation. chrome/ is a Debian image with google-chrome-stable, a UA
whose version is read back out of the binary at startup (a hardcoded one
would drift out of step with the Sec-CH-UA hints on the next Chrome
update and become a fresh tell), and no --enable-automation.
The timezone matters because Cloudflare scores a browser whose clock zone
disagrees with its egress IP's country as a proxy. Note that the usual
`-v /etc/localtime:/etc/localtime:ro` does not work here: Chrome resolves
the zone through ICU, which takes the name from that path's symlink
target and ignores the file's contents, so glibc reports the host zone
while Chrome still reports UTC. /etc/timezone carries the name and is
mounted instead; BROWSER_TZ overrides it for a host whose clock is UTC in
a country that is not.
Chrome also binds its DevTools port to loopback and silently ignores
--remote-debugging-address, which is why headless-shell fronted it with
socat. This image does the same, so it stays a drop-in: the compose
service keeps the headless-shell name and its pinned address, and
BROWSER_WS_URL is unchanged.
Deploying needs `docker compose build headless-shell`.
172 lines
14 KiB
Markdown
172 lines
14 KiB
Markdown
# AGENTS.md
|
|
|
|
Guidance for OpenCode (and Claude Code) working in this repo.
|
|
|
|
## What this is
|
|
|
|
Read-progress tracker for two libraries — manga and novels — behind one self-hosted Go backend. Two separate Violentmonkey userscripts inject on-page UI (floating button + slide-in panel) and sync progress, so bookmarks unify across sites and devices:
|
|
|
|
- `manga-bookmark.user.js` — **asurascans.com** (current domain; asuracomic.net 301s here), **demonicscans.org**, **comix.to**, **kagane.to**.
|
|
- `novel-bookmark.user.js` — **novelfull.com**, **lightnovelworld.net**.
|
|
|
|
One backend, one `bookmarks` table: a `kind` column (`manga`|`novel`) splits the libraries and the web UI switches between them. Rows are keyed `<site>:<series_id>`.
|
|
|
|
## Hard constraints (drive design — don't violate)
|
|
|
|
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
|
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
|
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
|
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
|
|
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
|
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
|
- **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
|
|
- **The CDP sidecar must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a clock zone matching the egress IP's country cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
|
|
- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache)
|
|
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
|
```
|
|
|
|
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
|
|
|
|
## Commands
|
|
|
|
Backend (`cd backend`):
|
|
- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`).
|
|
- Single test: `go test -run TestName ./...`
|
|
- Build static binary: `CGO_ENABLED=0 go build`
|
|
|
|
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). The `headless-shell` service keeps its name but now builds `chrome/` — real Google Chrome, for the reason in the hard constraints above.
|
|
|
|
Live CDP proof (needs a sidecar and network, skipped otherwise):
|
|
`SMOKE_BROWSER_WS_URL=ws://<host>:<port> go test -run TestSmokeKagane ./internal/latest`
|
|
— fetches a real kagane cover and chapter list. A red run means the challenge is
|
|
not clearing from this IP, which is a live fact to re-check, not necessarily a defect.
|
|
|
|
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
|
|
|
## Forge: Gitea, not GitHub
|
|
|
|
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
|
|
|
|
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
|
|
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
|
|
- Issues: `tea issue create`, `tea issue list`
|
|
- Auth lives in `tea login`, not `GH_TOKEN` env var.
|
|
|
|
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
|
|
|
|
## Design system
|
|
|
|
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
|
|
— source of truth Claude Design project `BookmarkManager Web UI`
|
|
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
|
|
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
|
|
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
|
|
state (busy, error, destruction) may use `--ember`; destruction gets
|
|
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
|
|
only (never hardcode hex outside `:root`), both colour branches touched
|
|
together. Any move that pulls series out of list (archive/finish/remove)
|
|
must be confirm-gated via its own `.confirm-row`; only restore fires
|
|
instantly.
|
|
|
|
## Security invariants
|
|
|
|
Existing guarantees — don't regress:
|
|
|
|
- Auth on `/bookmarks*`: require `Authorization: Bearer <credential>` — the acting Reader's credential, matched by SHA-256 against `readers.token_sha256` — **constant-time compare** (via the hash, never the secret itself), 401 otherwise.
|
|
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
|
|
|
|
## Secure coding rules (code you write here)
|
|
|
|
Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example in-tree — match it, don't start a second convention. AI-written backends fail on exactly these: broken access control, injection, weak session/error handling, invented dependencies.
|
|
|
|
Go backend:
|
|
|
|
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
|
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
|
|
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
|
|
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
|
|
- Compare secrets with `hmac.Equal` / `subtle.ConstantTimeCompare`, never `==`. A credential is matched by the SHA-256 the `readers` table holds, which is already a fixed-width equality — a new secret comparison must not regress to `==`.
|
|
- Errors: generic text to the client (`http.Error(w, "internal error", 500)`), detail to `log.Printf`. Never log `TOKEN_KEY`, a Reader's credential, `DISCORD_CLIENT_SECRET`, a session id, or a whole `Authorization` header.
|
|
- Proxy headers are trusted only where they already are: `X-Forwarded-Proto` for the Secure cookie flag, **rightmost** `X-Forwarded-For` for client IP (leftmost is attacker-supplied). Don't read either anywhere else.
|
|
- Session cookies keep `HttpOnly`, `SameSite`, `Secure`-when-HTTPS; expiry is enforced by the `sessions` table lookup, not a signature.
|
|
- Stdlib crypto only. No hand-rolled hashing, no MD5/SHA-1 anywhere security-bearing.
|
|
- Validate at the handler boundary before storing: body capped by `http.MaxBytesReader` (64 KB), empty `key` and unknown `status`/`kind` rejected with `400`. A bad value that reaches the store becomes every later reader's problem.
|
|
|
|
Userscript:
|
|
|
|
- Site-derived and stored strings render via `el(..., {text})` / `textContent`. `{html}` and `innerHTML` are for author-written literal markup only (`TEMPLATE`, `CSS`) — never a title, chapter label, or API response field. The page DOM belongs to a third-party site; treat it as attacker-controlled.
|
|
- Isolated world protects the credential from the site's JS. It does not protect anything from an `innerHTML` sink you add yourself.
|
|
- The userscripts carry `__API_TOKEN__` placeholders, substituted at serve time with the requesting Reader's credential (`internal/userscript`). Never put a real credential in the repo, docs, commit messages, or issues. Rotation is a web-UI action (epoch bump, `internal/token`); `TOKEN_KEY` in backend env is what derives every credential — never log it.
|
|
- `fetch()` targets `API_BASE` only — no dynamic origin, no site-supplied URL. `authHeaders()` goes nowhere but the backend.
|
|
- `localStorage` is shared with the site's own JS: cache and queue live there, credentials never do.
|
|
- Wrap every `localStorage` read/write and `JSON.parse` in try/catch (quota, private mode, corrupt entry), as the existing helpers do.
|
|
|
|
Dependencies: stdlib first; a new module needs a stated reason. Confirm a package actually exists before adding it — a plausible name may be fiction (~20% of LLM-proposed packages don't resolve, which is how slopsquatting lands). Pin exact versions.
|
|
|
|
Review gate: auth, CORS, session, crypto, and the fetch gate are security-critical. Editing one is not a drive-by change — say which invariant you preserved and run `go test ./...` before calling it done.
|
|
|
|
## Comments
|
|
|
|
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
|
|
|
Write for:
|
|
- Why not what. Tradeoffs, non-obvious decisions.
|
|
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
|
|
- Non-local consequence, invisible from function alone.
|
|
- Wire format / encoding / interface contract — save callers re-deriving.
|
|
- Gotcha/workaround, with ref if exists.
|
|
- Domain/business rule not derivable from code.
|
|
|
|
Skip:
|
|
- Restating code (no `// increment i` above `i++`).
|
|
- Trivial getter/setter/pass-through.
|
|
- Banners, dividers, `// helpers`.
|
|
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
|
- Commented-out code — delete.
|
|
- TODO without concrete action.
|
|
|
|
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
|
|
|
|
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
|
|
|
|
## Relevant skills
|
|
|
|
`multi-stage-dockerfile` and `docker-compose-orchestration` for container work (referenced in plan).
|
|
|
|
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
|
|
|
## Agent skills
|
|
|
|
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
|
|
|
### Issue tracker
|
|
|
|
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
|
|
|
|
### Triage labels
|
|
|
|
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
|
|
|
### Domain docs
|
|
|
|
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
|
|
|
|
## graphify
|
|
|
|
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
|
|
|
Rules:
|
|
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. Return scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
|
|
- If graphify-out/wiki/index.md exists, use for broad navigation instead of raw source browsing.
|
|
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain don't surface enough context.
|
|
- After modifying code, run `graphify update .` to keep graph current (AST-only, no API cost).
|
|
|
|
## Notes
|
|
|
|
- Keep comms terse — drop articles, fluff, pleasantries. Code/commits/security written normally.
|