sulthan
27cf0955de
Closes #24. Child of #18; based on current main (includes Postgres, Reader table, Discord OAuth).
## What
Each Reader's userscript credential is derived from `TOKEN_KEY`, their Discord id and a token epoch (HMAC-SHA256, hex); only its SHA-256 sits in `readers.token_sha256` (new `token_epoch` column, migration 0006). One credential authenticates the script download path and the API bearer header.
- `internal/token`: derivation + hashing; the seed refreshes the owner's epoch-0 hash only before first rotation, so a restart can never resurrect a rotated-away credential
- `httpmw.Auth`/`ResolveReader`: acting Reader resolved from the credential hash, stashed in request context; the retired global `API_TOKEN` resolves to the owner until `API_TOKEN_GRACE_UNTIL` (enforced in code, logged per use) on both the bearer and script-download paths
- Userscript handler renders the bindmounted file with the resolved Reader's credential substituted for `__API_TOKEN__`; a legacy-path request during grace serves the derived credential, so installed devices self-migrate on their next update poll
- Web UI: "Userscripts" panel — session-gated install endpoints render the script directly (credential never in markup, address bar, or a redirect), confirm-gated rotation with an atomic epoch bump + hash rewrite and a reinstall warning
- Both userscripts carry `__API_TOKEN__` placeholders; the committed global-token literal is removed
## Design note
Credentials are derived rather than stored-random because the server must rebuild install URLs after restarts while the DB holds only hashes. HMAC output is high-entropy and unbrute-forceable; the AC's intent (unguessable, DB-leak-proof) is met.
## Deploy (also in DEPLOY.md)
1. Add `TOKEN_KEY` (`openssl rand -hex 32`) — required; changing it later invalidates every credential.
2. Keep `API_TOKEN` + set `API_TOKEN_GRACE_UNTIL` for the 14-day window.
3. After deploy, sign in → Userscripts → reinstall both scripts on every device. This also retires the old global credential for real — its literal survives in git history (present since 0ef5286), so rotation is what kills it.
## Verification
- Full Go suite green against real Postgres per test; userscript JS suite 45/45
- New router-level tests: per-Reader isolation (read/write/delete), grace expiry on bearer + script path, self-migrating legacy path, install serving, rotation (old cred 401/404, new cred works, install renders new credential), app page leaks no credential
- Store tests: hash lookup, token info, atomic rotation with stale-epoch rejection, rotation survives restart
- Live smoke of the built binary: grace acceptance logged, derived auth, substitution, restart resilience, stored hash = SHA-256 of derived credential
Reviewed-on: #32
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
165 lines
12 KiB
Markdown
165 lines
12 KiB
Markdown
# AGENTS.md
|
|
|
|
Guidance for OpenCode (and Claude Code) working in this repo.
|
|
|
|
## What this is
|
|
|
|
Read-progress tracker for two libraries — manga and novels — behind one self-hosted Go backend. Two separate Violentmonkey userscripts inject on-page UI (floating button + slide-in panel) and sync progress, so bookmarks unify across sites and devices:
|
|
|
|
- `manga-bookmark.user.js` — **asurascans.com** (current domain; asuracomic.net 301s here), **demonicscans.org**, **comix.to**, **kagane.to**.
|
|
- `novel-bookmark.user.js` — **novelfull.com**, **lightnovelworld.net**.
|
|
|
|
One backend, one `bookmarks` table: a `kind` column (`manga`|`novel`) splits the libraries and the web UI switches between them. Rows are keyed `<site>:<series_id>`.
|
|
|
|
## Hard constraints (drive design — don't violate)
|
|
|
|
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
|
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
|
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
|
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
|
|
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
|
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
|
- **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache)
|
|
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
|
```
|
|
|
|
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
|
|
|
|
## Commands
|
|
|
|
Backend (`cd backend`):
|
|
- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`).
|
|
- Single test: `go test -run TestName ./...`
|
|
- Build static binary: `CGO_ENABLED=0 go build`
|
|
|
|
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
|
|
|
|
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
|
|
|
## Forge: Gitea, not GitHub
|
|
|
|
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
|
|
|
|
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
|
|
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
|
|
- Issues: `tea issue create`, `tea issue list`
|
|
- Auth lives in `tea login`, not `GH_TOKEN` env var.
|
|
|
|
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
|
|
|
|
## Design system
|
|
|
|
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
|
|
— source of truth Claude Design project `BookmarkManager Web UI`
|
|
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
|
|
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
|
|
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
|
|
state (busy, error, destruction) may use `--ember`; destruction gets
|
|
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
|
|
only (never hardcode hex outside `:root`), both colour branches touched
|
|
together. Any move that pulls series out of list (archive/finish/remove)
|
|
must be confirm-gated via its own `.confirm-row`; only restore fires
|
|
instantly.
|
|
|
|
## Security invariants
|
|
|
|
Existing guarantees — don't regress:
|
|
|
|
- Auth on `/bookmarks*`: require `Authorization: Bearer <credential>` — the acting Reader's credential, matched by SHA-256 against `readers.token_sha256` — **constant-time compare** (via the hash, never the secret itself), 401 otherwise.
|
|
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
|
|
|
|
## Secure coding rules (code you write here)
|
|
|
|
Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example in-tree — match it, don't start a second convention. AI-written backends fail on exactly these: broken access control, injection, weak session/error handling, invented dependencies.
|
|
|
|
Go backend:
|
|
|
|
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
|
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
|
|
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
|
|
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
|
|
- Compare secrets with `hmac.Equal` / `subtle.ConstantTimeCompare`, never `==`. Covers the retired global token during its grace window.
|
|
- Errors: generic text to the client (`http.Error(w, "internal error", 500)`), detail to `log.Printf`. Never log `API_TOKEN`, `DISCORD_CLIENT_SECRET`, a session id, or a whole `Authorization` header.
|
|
- Proxy headers are trusted only where they already are: `X-Forwarded-Proto` for the Secure cookie flag, **rightmost** `X-Forwarded-For` for client IP (leftmost is attacker-supplied). Don't read either anywhere else.
|
|
- Session cookies keep `HttpOnly`, `SameSite`, `Secure`-when-HTTPS; expiry is enforced by the `sessions` table lookup, not a signature.
|
|
- Stdlib crypto only. No hand-rolled hashing, no MD5/SHA-1 anywhere security-bearing.
|
|
- Validate at the handler boundary before storing: body capped by `http.MaxBytesReader` (64 KB), empty `key` and unknown `status`/`kind` rejected with `400`. A bad value that reaches the store becomes every later reader's problem.
|
|
|
|
Userscript:
|
|
|
|
- Site-derived and stored strings render via `el(..., {text})` / `textContent`. `{html}` and `innerHTML` are for author-written literal markup only (`TEMPLATE`, `CSS`) — never a title, chapter label, or API response field. The page DOM belongs to a third-party site; treat it as attacker-controlled.
|
|
- Isolated world protects the credential from the site's JS. It does not protect anything from an `innerHTML` sink you add yourself.
|
|
- The userscripts carry `__API_TOKEN__` placeholders, substituted at serve time with the requesting Reader's credential (`internal/userscript`). Never put a real credential in the repo, docs, commit messages, or issues. Rotation is a web-UI action (epoch bump, `internal/token`); `TOKEN_KEY` in backend env is what derives every credential — never log it.
|
|
- `fetch()` targets `API_BASE` only — no dynamic origin, no site-supplied URL. `authHeaders()` goes nowhere but the backend.
|
|
- `localStorage` is shared with the site's own JS: cache and queue live there, credentials never do.
|
|
- Wrap every `localStorage` read/write and `JSON.parse` in try/catch (quota, private mode, corrupt entry), as the existing helpers do.
|
|
|
|
Dependencies: stdlib first; a new module needs a stated reason. Confirm a package actually exists before adding it — a plausible name may be fiction (~20% of LLM-proposed packages don't resolve, which is how slopsquatting lands). Pin exact versions.
|
|
|
|
Review gate: auth, CORS, session, crypto, and the fetch gate are security-critical. Editing one is not a drive-by change — say which invariant you preserved and run `go test ./...` before calling it done.
|
|
|
|
## Comments
|
|
|
|
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
|
|
|
Write for:
|
|
- Why not what. Tradeoffs, non-obvious decisions.
|
|
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
|
|
- Non-local consequence, invisible from function alone.
|
|
- Wire format / encoding / interface contract — save callers re-deriving.
|
|
- Gotcha/workaround, with ref if exists.
|
|
- Domain/business rule not derivable from code.
|
|
|
|
Skip:
|
|
- Restating code (no `// increment i` above `i++`).
|
|
- Trivial getter/setter/pass-through.
|
|
- Banners, dividers, `// helpers`.
|
|
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
|
- Commented-out code — delete.
|
|
- TODO without concrete action.
|
|
|
|
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
|
|
|
|
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
|
|
|
|
## Relevant skills
|
|
|
|
`multi-stage-dockerfile` and `docker-compose-orchestration` for container work (referenced in plan).
|
|
|
|
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
|
|
|
## Agent skills
|
|
|
|
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
|
|
|
### Issue tracker
|
|
|
|
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
|
|
|
|
### Triage labels
|
|
|
|
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
|
|
|
### Domain docs
|
|
|
|
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
|
|
|
|
## graphify
|
|
|
|
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
|
|
|
Rules:
|
|
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. Return scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
|
|
- If graphify-out/wiki/index.md exists, use for broad navigation instead of raw source browsing.
|
|
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain don't surface enough context.
|
|
- After modifying code, run `graphify update .` to keep graph current (AST-only, no API cost).
|
|
|
|
## Notes
|
|
|
|
- Keep comms terse — drop articles, fluff, pleasantries. Code/commits/security written normally.
|