sulthan
0b946ee2c4
- web_test.go walks web.AdminPatterns() rather than naming routes by hand, so a new administrative route that forgets requireOwner fails the gate test instead of shipping open. - The harnesses take a LaneReporter; a fake one keeps the page's tests free of a poller and a Site. - backend/AGENTS.md records the adminRoutes/requireOwner rule and the nil-poller trap; design-system.md records --patina and the admin page's shape.
227 lines
17 KiB
Markdown
227 lines
17 KiB
Markdown
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
|
|
|
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
|
Single binary, split into packages under `backend/internal/`: `store`
|
|
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
|
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
|
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
|
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
|
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
|
`backend/main.go` is the composition root — the only place that wires
|
|
packages together into `newRouter`. Root-level `*_test.go` hold
|
|
integration tests that exercise the full router; unit tests for a
|
|
package live beside it under `internal/`.
|
|
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
|
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
|
file per change, one transaction each, versions recorded in
|
|
`schema_migrations`. Files are **append-only** — editing an applied one
|
|
changes nothing on a database that already ran it. No column probing, no
|
|
data-fixup migrations: both were SQLite-era machinery and are gone.
|
|
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
|
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
|
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
|
must have that `TestMain`.
|
|
- **Reader-owned store, four tables.** `readers` is keyed by Discord user ID
|
|
and carries the SHA-256 of the Reader's userscript credential plus a
|
|
`token_epoch` (issue #24). Credentials are derived, never stored: `token.Token(TOKEN_KEY, discord_id, epoch)` (HMAC, `internal/token`), and only its SHA-256 sits in `readers.token_sha256`, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the **owner** row at startup; its epoch-0 hash is refreshed on every start **only while the row has never been rotated**, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (`Store.EnsureReader`, idempotent on `discord_id`, and it never rewrites an existing row's hash). Rotation is `Store.RotateToken` (epoch bump + hash rewrite in one transaction), driven by the web UI.
|
|
`series` keyed `(site, series_id)`
|
|
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
|
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
|
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
|
|
differs between readers: progress, favourite, lifecycle bucket,
|
|
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
|
|
surrogate id; the wire `key` is derived as `site:series_id` on read — and
|
|
every store read/write is scoped to the reader it names. Auth resolves the
|
|
acting Reader from the presented credential (`httpmw.Auth`) and nothing
|
|
else — there is no unauthenticated-by-Reader route and no global token; the
|
|
reader id travels in the request context. Sync **last-write-wins**; the wire format
|
|
stays flat (ADR-0004). `Store.Upsert` decomposes one flat body across two
|
|
tables and enforces the ownership rule: client `title`/`series_url`/`cover`
|
|
are written only when the series row is new (ADR-0003).
|
|
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
|
- **Web UI:** same binary serve the browser UI on a second
|
|
hostname — `GET /` (list, or login page when no session),
|
|
`GET /auth/discord` + `GET /auth/discord/callback` (Discord OAuth,
|
|
ADR-0002), `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
|
under `/ui/*`. Templates + assets `go:embed`-ed under
|
|
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
|
`internal/` tree, not just `*.go`. Sessions are rows in the `sessions`
|
|
table: the cookie carries only an opaque id, looked up (and expiry-
|
|
checked) on every request, and deleting the row revokes the session.
|
|
Guild membership *is* registration (issue #27): `discordCallback` gates on
|
|
membership (and `DISCORD_REQUIRED_ROLE` when set) and then calls
|
|
`Store.EnsureReader`, so a refusal creates nothing and a returning Reader
|
|
reuses their row. The owner is the only Reader with administrative reach:
|
|
`POST /readers/{id}/revoke` (404 for anyone else) drops that Reader's
|
|
sessions, and the `readers` panel renders only on the owner's page.
|
|
A Reader with no bookmarks at all sees `listView.Fresh`, whose empty state
|
|
offers both install links instead of describing a filter.
|
|
UI mutations read-modify-write
|
|
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
|
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
|
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
|
(correct — served from `/`), but impeccable detector resolves
|
|
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
|
on leading `/` and silently skip file. Relative href don't help
|
|
either: template's directory isn't its served path. So
|
|
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
|
always pass `backend/internal/web/static` too. One finding there,
|
|
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
|
- **Every action that moves series out of list is confirm-gated.**
|
|
Archive, finish, remove each open own `.confirm-row` disclosure
|
|
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
|
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
|
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
|
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
|
error use `--mute`.
|
|
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100),
|
|
each re-checking that Site's bookmarked series' newest published chapter from
|
|
backend's own network access, so `latest_chapter` stays fresh when the user
|
|
isn't browsing. Second, parallel signal — the userscript keeps its own
|
|
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
|
Two independent clocks: per-series rest (`series.latest_checked_at`,
|
|
enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and
|
|
per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
|
|
in the Site registry (`internal/latest/sites.go`), not config: the five env
|
|
knobs that used to size a shared pace are gone.
|
|
The poller walks **Series, not Bookmarks** — a series referenced by several
|
|
bookmarks is fetched once per cycle, and the due queue orders
|
|
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
|
*before* fetch so broken series wait out the rest instead of retrying
|
|
every tick; found chapter written straight to the series row via
|
|
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
|
order — is never touched.
|
|
Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
|
|
stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
|
|
`errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
|
|
that makes the other browser Lanes skip their passes for the same 15m, so a
|
|
restarting Chrome doesn't stamp one Series per Lane per pass — after the
|
|
window the flag decays and they probe again. Browser Lanes wake Chrome only
|
|
when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
|
|
and cover work (both healing a stored source URL and filling a blank from
|
|
the series page) runs in the background so a slow CDN can't consume a
|
|
Lane's gap.
|
|
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
|
against fingerprint-based blocking; any failure log and skip. kagane, comix
|
|
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
|
|
can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and
|
|
comix are simply not polled when that's unset, while novelfull falls back to
|
|
a plain-TLS attempt — its challenge is a live time-varying fact, and its
|
|
cover bytes never need the browser. comix's browser read is an in-tab
|
|
`fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
|
|
costs ~65 requests for the same server-rendered HTML one fetch returns
|
|
(measured 2026-08-12, issue #98). See
|
|
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
|
The poller's series write is a single-column UPDATE
|
|
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
|
it cannot revert read progress or move `updated_at`, so the old
|
|
stale-re-read race is gone with the Get+Upsert flow.
|
|
- **Covers are acquired at creation, then served from our own origin
|
|
(ADR-0007):** the first Bookmark of a Series fires `Store.OnSeriesCreated`,
|
|
which `latest.Acquirer` turns into one series-page fetch yielding both the
|
|
Latest Chapter and the cover URL; the bytes then go through
|
|
`latest.CoverBytesFetcher` into `Store.SetSeriesCover`. It runs in a
|
|
goroutine — the Reader's PUT must neither block on a Site nor fail with one
|
|
— and every failure is logged and dropped, leaving the Bookmark intact. The
|
|
wire's `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
|
|
bytes exist and `""` before, never an address that 404s. `GET /covers/{addr}`
|
|
is public and uncredentialed: the userscript renders it on a Site's origin,
|
|
where no cookie or token of ours travels. A client-sent `cover` is decoded
|
|
and discarded, permanently (ADR-0004 compatibility).
|
|
Browser-backed Sites join the same pipeline (issue #62, extended to comix by
|
|
#98): kagane and comix pages *and* cover bytes go through the browser sidecar
|
|
(nothing falls back to a plain fetch, which would only retrieve a challenge
|
|
page), while novelfull needs the browser only for its HTML — the cover URL
|
|
comes out of the browser-fetched page and the bytes go over plain TLS. With
|
|
no browser configured, kagane and comix Covers are simply absent; novelfull
|
|
still gets one — at creation and on the poll — when its page body happens to
|
|
answer a plain request (the challenge is a live time-varying fact). comix
|
|
cover bytes must arrive by direct navigation, not an in-page fetch: its
|
|
Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
|
|
page-context fetch of `static.comix.to`. The old kagane-only
|
|
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
|
|
(issue #63): the one public route serves every Site.
|
|
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
|
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
|
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
|
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
|
checking archived series and skip finished ones. `finished` settable
|
|
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
|
**Empty incoming status means "keep stored one"** — resolved on the
|
|
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
|
`excluded.*` is post-evaluation row and default applied there would
|
|
wipe bucket on every PUT from client that predates column. See
|
|
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
|
- **Config via env:** `TOKEN_KEY` (derives every Reader's userscript credential;
|
|
required), `OWNER_DISCORD_ID` (seeds the owner Reader — the administrator and
|
|
the owner of every pre-registration bookmark; required),
|
|
`ALLOWED_ORIGINS` (comma list),
|
|
`DATABASE_URL` (Postgres connection URL, required — no default),
|
|
`COVER_DIR` (required filesystem volume for content-addressed Cover bytes),
|
|
`PUBLIC_BASE_URL` (required origin this deployment answers on, trailing
|
|
slash trimmed; every Cover URL on the wire is built from it, absolute
|
|
because the userscript renders on a Site's origin — ADR-0007),
|
|
`PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
|
|
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
|
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
|
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
|
|
`LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill
|
|
switch, default on). Pace is per Site in the registry (issue #100): every
|
|
Site rests an hour and gaps ten seconds, a Site with more eligible Series
|
|
than 360 tightens its own gap toward the 1s floor, and browser Lanes wake
|
|
Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/
|
|
`_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are
|
|
gone. The 1h rest for browser Sites is safe on documented grounds: a
|
|
challenged page costs seconds of a serialized single-tab browser, free-plan
|
|
zones have no bot score and no published per-IP rate input, and
|
|
`cf_clearance` expires in 30 minutes so every cadence at or above 1h
|
|
re-solves anyway —
|
|
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
|
|
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
|
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
|
defaults `/userscript/manga-bookmark.user.js` and
|
|
`/userscript/novel-bookmark.user.js`, both supplied by bindmount; the
|
|
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
|
Reader's credential at serve time).
|
|
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
|
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
|
Used by the poller for kagane, comix and novelfull page fetches and by the
|
|
cover pipeline for kagane's and comix's image bytes (the browser is the only
|
|
route that clears the challenge those two serve their covers behind); unset —
|
|
the default — disables browser polling and leaves kagane and comix Covers
|
|
blank until stored bytes
|
|
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
|
`/json/version` for any Host that isn't an IP or `localhost`).
|
|
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
|
|
served by the one public `GET /covers/{addr}` route from content-addressed
|
|
bytes. There is no proxy, no per-Site rewrite, no second place that decides
|
|
a Cover's renderable address: the wire `cover` is it. The only place a Site
|
|
name still appears in cover code is the extraction module (`latest`), where
|
|
kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane
|
|
answers a plain fetch with a challenge and
|
|
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
|
|
one with the same Cloudflare challenge its pages serve;
|
|
every other Site's CDN answers plain TLS. Templates render `.Cover` — the
|
|
wire value — never anything else.
|
|
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
|
(renders the bindmounted script with the acting Reader's derived credential
|
|
substituted in — the credential never appears in page markup, the address
|
|
bar, or a redirect; `?download=1` adds `Content-Disposition: attachment` for
|
|
mobile Violentmonkey, which ignores a `.user.js` navigation) and
|
|
`POST /rotate-token` (atomic epoch bump + hash
|
|
rewrite; invalidates every installed copy, so the panel warns to reinstall
|
|
on all devices).
|
|
- **Owner-only admin page (`internal/web/admin.go`, issue #102):** `GET /admin`
|
|
carries the Reader roster (sessions, Sighting counters, `POST
|
|
/readers/{id}/revoke` and `POST /readers/{id}/clear-marks`) and Poll Lane
|
|
status (`GET /ui/admin/lanes`, self-refreshing every 30s). Every route that
|
|
reaches past the acting Reader is listed in `adminRoutes()` and wrapped in
|
|
`requireOwner` at registration — add a route there, not a check inside a
|
|
handler; `web.AdminPatterns()` is what the gate test walks. A non-owner gets
|
|
404, never 403. Lane figures come from the running poller through the
|
|
`web.LaneReporter` seam (`latest.Poller.LaneStatus`), never from a table: a
|
|
nil reporter or a Lane that has not finished a pass renders "no data yet"
|
|
rather than zeroes. `main.newRouter` takes the reporter as an interface and
|
|
converts a nil `*Poller` to a nil interface — a typed nil would make the page
|
|
claim a poller exists.
|