sulthan
249aacab2e
Swap modernc.org/sqlite for jackc/pgx/v5 with no observable change: same endpoints, same wire format, same updated_at ordering rule. The schema now comes from numbered SQL embedded in the binary and applied on startup, one transaction each, recorded in schema_migrations. That replaces two pieces of SQLite-era machinery, both deleted rather than ported: the column probing (Postgres has ADD COLUMN IF NOT EXISTS, and there is no legacy database left to probe) and the Asura key rewrite, which has run clean on every start for months now that the userscripts strip build hashes before writing. Its regexp survives as latest.asuraBuildHash, where the poller still needs it to scope chapter links to a series whose slug carries a rotating hash. Types get real: favorite is a boolean, chapter numbers double precision, timestamps stay unix-ms bigint. SQLite's null-safe IS NOT becomes IS DISTINCT FROM, which is what implements the rule that only reading progress reorders a list. Inside COALESCE/NULLIF the status and kind parameters need an explicit ::text — there is no target column to infer from and Postgres refuses to guess. Tests lose their free t.TempDir() database, so Docker is now a hard prerequisite for `go test ./...`: internal/pgtest starts one postgres:17-alpine per test binary and hands each test a database of its own. Hand-rolled rather than testcontainers — it is one docker run, one docker port and a ping loop against a module list that is otherwise stdlib. BREAKING CHANGE: DB_PATH is retired for DATABASE_URL, which is required and has no default. Compose gains a postgres service on an internal network with its own volume; POSTGRES_PASSWORD joins .env. The old bookmarks-data volume is deliberately left undeclared so `docker compose down -v` cannot take the pre-migration database with it. Closes #20
165 lines
12 KiB
Markdown
165 lines
12 KiB
Markdown
# AGENTS.md
|
|
|
|
Guidance for OpenCode (and Claude Code) working in this repo.
|
|
|
|
## What this is
|
|
|
|
Read-progress tracker for two libraries — manga and novels — behind one self-hosted Go backend. Two separate Violentmonkey userscripts inject on-page UI (floating button + slide-in panel) and sync progress, so bookmarks unify across sites and devices:
|
|
|
|
- `manga-bookmark.user.js` — **asurascans.com** (current domain; asuracomic.net 301s here), **demonicscans.org**, **comix.to**, **kagane.to**.
|
|
- `novel-bookmark.user.js` — **novelfull.com**, **lightnovelworld.net**.
|
|
|
|
One backend, one `bookmarks` table: a `kind` column (`manga`|`novel`) splits the libraries and the web UI switches between them. Rows are keyed `<site>:<series_id>`.
|
|
|
|
## Hard constraints (drive design — don't violate)
|
|
|
|
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
|
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
|
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
|
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
|
|
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
|
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
|
- **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache)
|
|
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
|
```
|
|
|
|
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
|
|
|
|
## Commands
|
|
|
|
Backend (`cd backend`):
|
|
- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`).
|
|
- Single test: `go test -run TestName ./...`
|
|
- Build static binary: `CGO_ENABLED=0 go build`
|
|
|
|
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
|
|
|
|
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
|
|
|
## Forge: Gitea, not GitHub
|
|
|
|
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
|
|
|
|
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
|
|
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
|
|
- Issues: `tea issue create`, `tea issue list`
|
|
- Auth lives in `tea login`, not `GH_TOKEN` env var.
|
|
|
|
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
|
|
|
|
## Design system
|
|
|
|
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
|
|
— source of truth Claude Design project `BookmarkManager Web UI`
|
|
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
|
|
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
|
|
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
|
|
state (busy, error, destruction) may use `--ember`; destruction gets
|
|
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
|
|
only (never hardcode hex outside `:root`), both colour branches touched
|
|
together. Any move that pulls series out of list (archive/finish/remove)
|
|
must be confirm-gated via its own `.confirm-row`; only restore fires
|
|
instantly.
|
|
|
|
## Security invariants
|
|
|
|
Existing guarantees — don't regress:
|
|
|
|
- Auth on `/bookmarks*`: require `Authorization: Bearer <API_TOKEN>`, **constant-time compare**, 401 otherwise.
|
|
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
|
|
|
|
## Secure coding rules (code you write here)
|
|
|
|
Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example in-tree — match it, don't start a second convention. AI-written backends fail on exactly these: broken access control, injection, weak session/error handling, invented dependencies.
|
|
|
|
Go backend:
|
|
|
|
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
|
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
|
|
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
|
|
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
|
|
- Compare secrets with `hmac.Equal` / `subtle.ConstantTimeCompare`, never `==`. Covers API token, web password, session MAC.
|
|
- Errors: generic text to the client (`http.Error(w, "internal error", 500)`), detail to `log.Printf`. Never log `API_TOKEN`, `WEB_PASSWORD`, a session cookie value, or a whole `Authorization` header.
|
|
- Proxy headers are trusted only where they already are: `X-Forwarded-Proto` for the Secure cookie flag, **rightmost** `X-Forwarded-For` for client IP (leftmost is attacker-supplied). Don't read either anywhere else.
|
|
- Session cookies keep `HttpOnly`, `SameSite`, `Secure`-when-HTTPS, and expiry checked before signature.
|
|
- Stdlib crypto only. No hand-rolled hashing, no MD5/SHA-1 anywhere security-bearing.
|
|
- Validate at the handler boundary before storing: body capped by `http.MaxBytesReader` (64 KB), empty `key` and unknown `status`/`kind` rejected with `400`. A bad value that reaches the store becomes every later reader's problem.
|
|
|
|
Userscript:
|
|
|
|
- Site-derived and stored strings render via `el(..., {text})` / `textContent`. `{html}` and `innerHTML` are for author-written literal markup only (`TEMPLATE`, `CSS`) — never a title, chapter label, or API response field. The page DOM belongs to a third-party site; treat it as attacker-controlled.
|
|
- Isolated world protects the token from the site's JS. It does not protect anything from an `innerHTML` sink you add yourself.
|
|
- The `API_TOKEN` literal sits in both userscripts and must equal backend `API_TOKEN`. Never copy it into logs, docs, commit messages, issues, or a new file. Rotation touches three places: backend env plus both scripts.
|
|
- `fetch()` targets `API_BASE` only — no dynamic origin, no site-supplied URL. `authHeaders()` goes nowhere but the backend.
|
|
- `localStorage` is shared with the site's own JS: cache and queue live there, credentials never do.
|
|
- Wrap every `localStorage` read/write and `JSON.parse` in try/catch (quota, private mode, corrupt entry), as the existing helpers do.
|
|
|
|
Dependencies: stdlib first; a new module needs a stated reason. Confirm a package actually exists before adding it — a plausible name may be fiction (~20% of LLM-proposed packages don't resolve, which is how slopsquatting lands). Pin exact versions.
|
|
|
|
Review gate: auth, CORS, session, crypto, and the fetch gate are security-critical. Editing one is not a drive-by change — say which invariant you preserved and run `go test ./...` before calling it done.
|
|
|
|
## Comments
|
|
|
|
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
|
|
|
Write for:
|
|
- Why not what. Tradeoffs, non-obvious decisions.
|
|
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
|
|
- Non-local consequence, invisible from function alone.
|
|
- Wire format / encoding / interface contract — save callers re-deriving.
|
|
- Gotcha/workaround, with ref if exists.
|
|
- Domain/business rule not derivable from code.
|
|
|
|
Skip:
|
|
- Restating code (no `// increment i` above `i++`).
|
|
- Trivial getter/setter/pass-through.
|
|
- Banners, dividers, `// helpers`.
|
|
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
|
- Commented-out code — delete.
|
|
- TODO without concrete action.
|
|
|
|
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
|
|
|
|
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
|
|
|
|
## Relevant skills
|
|
|
|
`multi-stage-dockerfile` and `docker-compose-orchestration` for container work (referenced in plan).
|
|
|
|
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
|
|
|
## Agent skills
|
|
|
|
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
|
|
|
### Issue tracker
|
|
|
|
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
|
|
|
|
### Triage labels
|
|
|
|
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
|
|
|
### Domain docs
|
|
|
|
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
|
|
|
|
## graphify
|
|
|
|
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
|
|
|
Rules:
|
|
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. Return scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
|
|
- If graphify-out/wiki/index.md exists, use for broad navigation instead of raw source browsing.
|
|
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain don't surface enough context.
|
|
- After modifying code, run `graphify update .` to keep graph current (AST-only, no API cost).
|
|
|
|
## Notes
|
|
|
|
- Keep comms terse — drop articles, fluff, pleasantries. Code/commits/security written normally.
|