mangaBookmark/backend/AGENTS.md

Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.

- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
  Single binary, split into packages under `backend/internal/`: `store`
  (Bookmark type, Postgres persistence, migration runner), `latest` (background
  poller, site parsers, TLS fetcher), `session` (cookie signing, login
  rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
  bookmark handlers), `userscript` (userscript-serving handler), `web`
  (browser UI handler + `templates/` + `static/`, `go:embed`-ed).
  `backend/main.go` is the composition root — the only place that wires
  packages together into `newRouter`. Root-level `*_test.go` hold
  integration tests that exercise the full router; unit tests for a
  package live beside it under `internal/`.
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
  `go:embed`-ed and applied on every start by `store.migrate`: one numbered
  file per change, one transaction each, versions recorded in
  `schema_migrations`. Files are **append-only** — editing an applied one
  changes nothing on a database that already ran it. No column probing, no
  data-fixup migrations: both were SQLite-era machinery and are gone.
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
  container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
  its own database (`pgtest.URL(t)`). A package whose tests touch the store
  must have that `TestMain`.
- **Reader-owned store, four tables.** `readers` is keyed by Discord user ID
  and carries the SHA-256 of the Reader's userscript credential plus a
  `token_epoch` (issue #24). Credentials are derived, never stored: `token.Token(TOKEN_KEY, discord_id, epoch)` (HMAC, `internal/token`), and only its SHA-256 sits in `readers.token_sha256`, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the **owner** row at startup; its epoch-0 hash is refreshed on every start **only while the row has never been rotated**, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (`Store.EnsureReader`, idempotent on `discord_id`, and it never rewrites an existing row's hash). Rotation is `Store.RotateToken` (epoch bump + hash rewrite in one transaction), driven by the web UI.
  `series` keyed `(site, series_id)`
  (`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
  shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
  Latest Chapter, `latest_checked_at`, and the Sighting pair
  `latest_sighted_at`/`latest_raised_by` (issue #103) — and `bookmarks` holds only what
  differs between readers: progress, favourite, lifecycle bucket,
  `updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
  surrogate id; the wire `key` is derived as `site:series_id` on read — and
  every store read/write is scoped to the reader it names. Auth resolves the
  acting Reader from the presented credential (`httpmw.Auth`) and nothing
  else — there is no unauthenticated-by-Reader route and no global token; the
  reader id travels in the request context. Sync **last-write-wins**; the wire format
  stays flat (ADR-0004). `Store.Upsert` decomposes one flat body across two
  tables and enforces the ownership rule: client `title`/`series_url`/`cover`
  are written only when the series row is new (ADR-0003).
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
- **Web UI:** same binary serve the browser UI on a second
  hostname — `GET /` (list, or login page when no session),
  `GET /auth/discord` + `GET /auth/discord/callback` (Discord OAuth,
  ADR-0002), `POST /logout`, `GET /static/*`, htmx fragment endpoints
  under `/ui/*`. Templates + assets `go:embed`-ed under
  `backend/internal/web/`, so `backend/Dockerfile` must copy the whole
  `internal/` tree, not just `*.go`. Sessions are rows in the `sessions`
  table: the cookie carries only an opaque id, looked up (and expiry-
  checked) on every request, and deleting the row revokes the session.
  Guild membership *is* registration (issue #27): `discordCallback` gates on
  membership (and `DISCORD_REQUIRED_ROLE` when set) and then calls
  `Store.EnsureReader`, so a refusal creates nothing and a returning Reader
  reuses their row. The owner is the only Reader with administrative reach:
  `POST /readers/{id}/revoke` (404 for anyone else) drops that Reader's
  sessions, and the `readers` panel renders only on the owner's page.
  A Reader with no bookmarks at all sees `listView.Fresh`, whose empty state
  offers both install links instead of describing a filter.
  UI mutations read-modify-write
  through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
  place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
  **Design-tool caveat:** templates link `/static/style.css` root-absolutely
  (correct — served from `/`), but impeccable detector resolves
  stylesheet href with `path.resolve(fileDir, href)`, drops directory
  on leading `/` and silently skip file. Relative href don't help
  either: template's directory isn't its served path. So
  `detect.mjs backend/internal/web/templates` reports **false clean** —
  always pass `backend/internal/web/static` too. One finding there,
  `overused-font` on "Instrument Serif", deliberate identity choice, not debt.
- **Every action that moves series out of list is confirm-gated.**
  Archive, finish, remove each open own `.confirm-row` disclosure
  (`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
  `archive|finish|remove`); restore fire instantly since it's the reversal.
  Remove's row wear ember wash, two reversible ones wear `.calm` grey.
  `--ember` stay reserved for new-chapter signal: busy bar and inline
  error use `--mute`.
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100),
  each re-checking that Site's bookmarked series' newest published chapter from
  backend's own network access, so `latest_chapter` stays fresh when the user
  isn't browsing. Second, parallel signal — the userscript keeps its own
  `maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` schedule, and its
  `reportLatestChapter` PUTs every read, unchanged numbers included, because an
  unchanged read is exactly the Sighting worth deferring a Poll on (#103).
  Two independent clocks: per-series rest (`series.latest_checked_at`,
  enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and
  per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
  in the Site registry (`internal/latest/sites.go`), not config: the five env
  knobs that used to size a shared pace are gone.
  The poller walks **Series, not Bookmarks** — a series referenced by several
  bookmarks is fetched once per cycle, and the due queue orders
  `reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
  *before* fetch so broken series wait out the rest instead of retrying
  every tick; found chapter written straight to the series row via
  `Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
  order — is never touched.
  **Sightings** (issue #103, ADR-0011) let a Reader's own page read defer a
  Poll: `Store.RecordSighting` — called by the PUT handler *before* the Upsert,
  because the raise test needs the row as it stands — stamps
  `series.latest_sighted_at` and, when the report raises the stored number,
  names its Reader in `series.latest_raised_by`. The due query's HAVING clause
  is where deferral lives: a Series is skipped only while it has exactly one
  Bookmark, was sighted within one Rest, and is under the ceiling
  (`sightingCeilingRests`, six of that Site's rests) since its last Poll. So a
  shared Series is never deferred, and no Series goes six hours unpolled
  whatever arrives. `checkOne` judges the named Reader off the comparison it
  already makes: a lower number is a contradiction (logged with the Reader and
  both numbers), the same number an agreement, a higher number the Site
  publishing and neither — that last one clears the attribution instead, since
  the value the Poll then stores is its own and a later retraction is not the
  Reader's fault. Three contradictions
  (`store.SightingDisagreementLimit`) stop that Reader deferring — their
  reports still write the Latest Chapter — and twenty consecutive agreements
  (`store.SightingAgreementsToClear`) forgive them, as does the owner's
  clear-marks control. Deferral is recomputed from live facts every round, so
  nothing needs invalidating when a Series gains a second Bookmark; the one
  input read earlier is the Reader's marks, checked when the Sighting is
  recorded, so crossing the threshold or being cleared takes effect from that
  Reader's next Sighting and the standing already bought lasts out its rest.
  Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
  stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
  `errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
  that makes the other browser Lanes skip their passes for the same 15m, so a
  restarting Chrome doesn't stamp one Series per Lane per pass — after the
  window the flag decays and they probe again. Browser Lanes wake Chrome only
  when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
  and cover work (both healing a stored source URL and filling a blank from
  the series page) runs in the background so a slow CDN can't consume a
  Lane's gap.
  A refusal is only ever the challenge *page*: `isInterstitial` matches the
  orchestration path `/cdn-cgi/challenge-platform/h/`, never the bare prefix.
  Cloudflare injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into
  ordinary 200 pages once a zone turns JS detections on, which demonic did on
  2026-08-16 — the prefix match then read every real demonic page as a refusal
  and parked that Lane in 15m backoff while plain TLS was returning the full
  series page.
  Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
  against fingerprint-based blocking; any failure log and skip. kagane, comix
  and novelfull sit behind Cloudflare JavaScript challenges the TLS client
  can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and
  comix are simply not polled when that's unset, while novelfull falls back to
  a plain-TLS attempt — its challenge is a live time-varying fact, and its
  cover bytes never need the browser. comix's browser read is an in-tab
  `fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
  costs ~65 requests for the same server-rendered HTML one fetch returns
  (measured 2026-08-12, issue #98). See
  `docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
  The poller's series write is a single-column UPDATE
  (`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
  it cannot revert read progress or move `updated_at`, so the old
  stale-re-read race is gone with the Get+Upsert flow.
- **Covers are acquired at creation, then served from our own origin
  (ADR-0007):** the first Bookmark of a Series fires `Store.OnSeriesCreated`,
  which `latest.Acquirer` turns into one series-page fetch yielding both the
  Latest Chapter and the cover URL; the bytes then go through
  `latest.CoverBytesFetcher` into `Store.SetSeriesCover`. It runs in a
  goroutine — the Reader's PUT must neither block on a Site nor fail with one
  — and every failure is logged and dropped, leaving the Bookmark intact. The
  wire's `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
  bytes exist and `""` before, never an address that 404s. `GET /covers/{addr}`
  is public and uncredentialed: the userscript renders it on a Site's origin,
  where no cookie or token of ours travels. A client-sent `cover` is decoded
  and discarded, permanently (ADR-0004 compatibility).
  Browser-backed Sites join the same pipeline (issue #62, extended to comix by
  #98): kagane and comix pages *and* cover bytes go through the browser sidecar
  (nothing falls back to a plain fetch, which would only retrieve a challenge
  page), while novelfull needs the browser only for its HTML — the cover URL
  comes out of the browser-fetched page and the bytes go over plain TLS. With
  no browser configured, kagane and comix Covers are simply absent; novelfull
  still gets one — at creation and on the poll — when its page body happens to
  answer a plain request (the challenge is a live time-varying fact). comix
  cover bytes must arrive by direct navigation, not an in-page fetch: its
  Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
  page-context fetch of `static.comix.to`. The old kagane-only
  serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
  (issue #63): the one public route serves every Site.
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
  `finished`, orthogonal to `favorite`. Archived and finished appear only in
  own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
  checking archived series and skip finished ones. `finished` settable
  only from web UI; `PUT /bookmarks/{key}` reject it with 400.
  **Empty incoming status means "keep stored one"** — resolved on the
  `VALUES` side of `Store.Upsert`, not conflict clause, since
  `excluded.*` is post-evaluation row and default applied there would
  wipe bucket on every PUT from client that predates column. See
  `docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
- **Config via env:** `TOKEN_KEY` (derives every Reader's userscript credential;
  required), `OWNER_DISCORD_ID` (seeds the owner Reader — the administrator and
  the owner of every pre-registration bookmark; required),
  `ALLOWED_ORIGINS` (comma list),
  `DATABASE_URL` (Postgres connection URL, required — no default),
  `COVER_DIR` (required filesystem volume for content-addressed Cover bytes),
  `PUBLIC_BASE_URL` (required origin this deployment answers on, trailing
  slash trimmed; every Cover URL on the wire is built from it, absolute
  because the userscript renders on a Site's origin — ADR-0007),
  `PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
  `_REDIRECT_URI` (required; Discord OAuth for the browser UI),
  `DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
  `DISCORD_API_BASE` (default `https://discord.com/api/v10`),
  `LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill
  switch, default on). Pace is per Site in the registry (issue #100): every
  Site rests an hour and gaps ten seconds, a Site with more eligible Series
  than 360 tightens its own gap toward the 1s floor, and browser Lanes wake
  Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/
  `_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are
  gone. The 1h rest for browser Sites is safe on documented grounds: a
  challenged page costs seconds of a serialized single-tab browser, free-plan
  zones have no bot score and no published per-IP rate input, and
  `cf_clearance` expires in 30 minutes so every cadence at or above 1h
  re-solves anyway —
  `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
  `USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
  `/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
  defaults `/userscript/manga-bookmark.user.js` and
  `/userscript/novel-bookmark.user.js`, both supplied by bindmount; the
  `__API_TOKEN__` placeholder inside them is substituted with the requesting
  Reader's credential at serve time).
  `BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
  machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
  Used by the poller for kagane, comix and novelfull page fetches and by the
  cover pipeline for kagane's and comix's image bytes (the browser is the only
  route that clears the challenge those two serve their covers behind); unset —
  the default — disables browser polling and leaves kagane and comix Covers
  blank until stored bytes
  exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
  `/json/version` for any Host that isn't an IP or `localhost`).
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
  served by the one public `GET /covers/{addr}` route from content-addressed
  bytes. There is no proxy, no per-Site rewrite, no second place that decides
  a Cover's renderable address: the wire `cover` is it. The only place a Site
  name still appears in cover code is the extraction module (`latest`), where
  kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane
  answers a plain fetch with a challenge and
  `cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
  one with the same Cloudflare challenge its pages serve;
  every other Site's CDN answers plain TLS. Templates render `.Cover` — the
  wire value — never anything else.
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
  (renders the bindmounted script with the acting Reader's derived credential
  substituted in — the credential never appears in page markup, the address
  bar, or a redirect; `?download=1` adds `Content-Disposition: attachment` for
  mobile Violentmonkey, which ignores a `.user.js` navigation) and
  `POST /rotate-token` (atomic epoch bump + hash
  rewrite; invalidates every installed copy, so the panel warns to reinstall
  on all devices).
- **Owner-only admin page (`internal/web/admin.go`, issue #102):** `GET /admin`
  carries the Reader roster (sessions, Sighting counters, `POST
  /readers/{id}/revoke` and `POST /readers/{id}/clear-marks`) and Poll Lane
  status (`GET /ui/admin/lanes`, self-refreshing every 30s). Every route that
  reaches past the acting Reader is listed in `adminRoutes()` and wrapped in
  `requireOwner` at registration — add a route there, not a check inside a
  handler; `web.AdminPatterns()` is what the gate test walks. A non-owner gets
  404, never 403. Lane figures come from the running poller through the
  `web.LaneReporter` seam (`latest.Poller.LaneStatus`), never from a table: a
  nil reporter or a Lane that has not finished a pass renders "no data yet"
  rather than zeroes. `main.newRouter` takes the reporter as an interface and
  converts a nil `*Poller` to a nil interface — a typed nil would make the page
  claim a poller exists.
  The one owner comparison left outside `requireOwner` is in `index`
  (`view.Owner = readerID == h.store.OwnerID()`): it gates a link, not an
  endpoint, so it is a rendering decision a registration-time wrapper cannot
  express — do not "unify" it into the gate.
  A Lane pass that returns before computing its figures (refusal backoff,
  sidecar down) carries the previous pass's due count and gap forward rather
  than recording zeroes; a Lane that has never reached a pace renders no gap at
  all. `Checked` next to `Due` is what separates a stopped Lane from a quiet
  one, so neither figure may be dropped from the row.
  Due-without-Checked is *not* by itself a stall: a browser Lane under both
  wake thresholds sets `LaneState.Asleep` at the on-demand gate and renders
  "browser asleep" instead of "not checking", and never counts toward
  `Attention`. That is the commonest healthy state for kagane, comix and
  novelfull — one due Series, nothing checked — so spending the stall mark on
  it would train the owner to ignore the mark that matters.