e4a313e626
The headless browser leaves the API stack. It becomes its own compose unit
(chrome/docker-compose.yml) deployed on the home machine and reached over the
tailnet, returning 471 MiB of working set to a 1974 MiB VPS that has no swap.
No fallback sidecar is left behind.
The backend needs no code change: BROWSER_WS_URL was already the only coupling,
so relocation is one environment variable. Its default is now empty rather than
a pinned Docker IP — an unreachable or unconfigured browser degrades exactly as
it always has, with plain-TLS libraries unaffected, kagane and novelfull logged
and skipped, and stored covers still served.
The browser unit publishes CDP on ${BROWSER_BIND_ADDR} with no default, because
CDP authenticates nothing and the home machine has a real LAN: an unset value
must fail the deploy rather than silently expose an endpoint that is remote code
execution for anything that reaches it. Resource limits are sized against the
measured 645 MiB untuned peak and the CI runner that already holds 1.2 GiB of
that box.
bookmark-api gains the default network. Dropping `browser` left it on `db`
alone, which is internal: true — that meant no published port and, worse, no
egress for the poller at all. Caught by bringing the stack up.
Docs: ADR-0006 for the topology, DEPLOY.md §7 for first-time setup of the
browser machine, REDEPLOY.md §8 for its independent update cadence, plus the
architecture diagrams, config tables and troubleshooting rows.
159 lines
12 KiB
Markdown
159 lines
12 KiB
Markdown
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
|
|
|
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
|
Single binary, split into packages under `backend/internal/`: `store`
|
|
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
|
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
|
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
|
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
|
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
|
`backend/main.go` is the composition root — the only place that wires
|
|
packages together into `newRouter`. Root-level `*_test.go` hold
|
|
integration tests that exercise the full router; unit tests for a
|
|
package live beside it under `internal/`.
|
|
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
|
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
|
file per change, one transaction each, versions recorded in
|
|
`schema_migrations`. Files are **append-only** — editing an applied one
|
|
changes nothing on a database that already ran it. No column probing, no
|
|
data-fixup migrations: both were SQLite-era machinery and are gone.
|
|
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
|
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
|
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
|
must have that `TestMain`.
|
|
- **Reader-owned store, four tables.** `readers` is keyed by Discord user ID
|
|
and carries the SHA-256 of the Reader's userscript credential plus a
|
|
`token_epoch` (issue #24). Credentials are derived, never stored: `token.Token(TOKEN_KEY, discord_id, epoch)` (HMAC, `internal/token`), and only its SHA-256 sits in `readers.token_sha256`, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the **owner** row at startup; its epoch-0 hash is refreshed on every start **only while the row has never been rotated**, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (`Store.EnsureReader`, idempotent on `discord_id`, and it never rewrites an existing row's hash). Rotation is `Store.RotateToken` (epoch bump + hash rewrite in one transaction), driven by the web UI.
|
|
`series` keyed `(site, series_id)`
|
|
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
|
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
|
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
|
|
differs between readers: progress, favourite, lifecycle bucket,
|
|
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
|
|
surrogate id; the wire `key` is derived as `site:series_id` on read — and
|
|
every store read/write is scoped to the reader it names. Auth resolves the
|
|
acting Reader from the presented credential (`httpmw.Auth`) and nothing
|
|
else — there is no unauthenticated-by-Reader route and no global token; the
|
|
reader id travels in the request context. Sync **last-write-wins**; the wire format
|
|
stays flat (ADR-0004). `Store.Upsert` decomposes one flat body across two
|
|
tables and enforces the ownership rule: client `title`/`series_url`/`cover`
|
|
are written only when the series row is new (ADR-0003).
|
|
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
|
- **Web UI:** same binary serve the browser UI on a second
|
|
hostname — `GET /` (list, or login page when no session),
|
|
`GET /auth/discord` + `GET /auth/discord/callback` (Discord OAuth,
|
|
ADR-0002), `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
|
under `/ui/*`. Templates + assets `go:embed`-ed under
|
|
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
|
`internal/` tree, not just `*.go`. Sessions are rows in the `sessions`
|
|
table: the cookie carries only an opaque id, looked up (and expiry-
|
|
checked) on every request, and deleting the row revokes the session.
|
|
Guild membership *is* registration (issue #27): `discordCallback` gates on
|
|
membership (and `DISCORD_REQUIRED_ROLE` when set) and then calls
|
|
`Store.EnsureReader`, so a refusal creates nothing and a returning Reader
|
|
reuses their row. The owner is the only Reader with administrative reach:
|
|
`POST /readers/{id}/revoke` (404 for anyone else) drops that Reader's
|
|
sessions, and the `readers` panel renders only on the owner's page.
|
|
A Reader with no bookmarks at all sees `listView.Fresh`, whose empty state
|
|
offers both install links instead of describing a filter.
|
|
UI mutations read-modify-write
|
|
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
|
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
|
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
|
(correct — served from `/`), but impeccable detector resolves
|
|
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
|
on leading `/` and silently skip file. Relative href don't help
|
|
either: template's directory isn't its served path. So
|
|
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
|
always pass `backend/internal/web/static` too. One finding there,
|
|
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
|
- **Every action that moves series out of list is confirm-gated.**
|
|
Archive, finish, remove each open own `.confirm-row` disclosure
|
|
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
|
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
|
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
|
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
|
error use `--mute`.
|
|
- **Latest-chapter poller:** ticker goroutine in same binary re-check
|
|
each bookmarked series' newest published chapter from backend's own
|
|
network access, so `latest_chapter` stay fresh when user not
|
|
browsing. Second, parallel signal — userscript keep own
|
|
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
|
Two independent clocks: per-series cooldown (`series.latest_checked_at`,
|
|
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
|
The poller walks **Series, not Bookmarks** — a series referenced by several
|
|
bookmarks is fetched once per cycle, and the due queue orders
|
|
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
|
*before* fetch so broken series wait out full cooldown instead of retrying
|
|
every tick; found chapter written straight to the series row via
|
|
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
|
order — is never touched.
|
|
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
|
against fingerprint-based blocking; any failure log and skip. kagane and
|
|
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
|
clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and
|
|
simply not polled when that's unset. See
|
|
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
|
The poller's series write is a single-column UPDATE
|
|
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
|
it cannot revert read progress or move `updated_at`, so the old
|
|
stale-re-read race is gone with the Get+Upsert flow.
|
|
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
|
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
|
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
|
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
|
checking archived series and skip finished ones. `finished` settable
|
|
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
|
**Empty incoming status means "keep stored one"** — resolved on the
|
|
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
|
`excluded.*` is post-evaluation row and default applied there would
|
|
wipe bucket on every PUT from client that predates column. See
|
|
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
|
- **Config via env:** `TOKEN_KEY` (derives every Reader's userscript credential;
|
|
required), `OWNER_DISCORD_ID` (seeds the owner Reader — the administrator and
|
|
the owner of every pre-registration bookmark; required),
|
|
`ALLOWED_ORIGINS` (comma list),
|
|
`DATABASE_URL` (Postgres connection URL, required — no default),
|
|
`PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
|
|
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
|
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
|
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
|
|
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/
|
|
`_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on,
|
|
`1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both
|
|
cooldowns have a `15m` floor).
|
|
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
|
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
|
defaults `/userscript/manga-bookmark.user.js` and
|
|
`/userscript/novel-bookmark.user.js`, both supplied by bindmount; the
|
|
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
|
Reader's credential at serve time).
|
|
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
|
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
|
Used by the poller for kagane and novelfull *and* by the web UI's kagane
|
|
cover proxy; unset — the default — disables browser polling and serves 404
|
|
for covers not already stored, leaving those sites to the userscript alone.
|
|
Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
|
`/json/version` for any Host that isn't an IP or `localhost`).
|
|
- **kagane covers are proxied, not hot-linked:** kagane serves cover images
|
|
behind the same challenge as its pages and with
|
|
`cross-origin-resource-policy: same-origin`, so no `<img>` on the web UI's
|
|
origin can load one — not even from a browser holding the clearance cookie
|
|
`og:image` to `/img/kagane/{id}`. `internal/web/cover.go` reads the persistent
|
|
`covers` table first, then fetches a miss through `latest.BrowserFetcher.Image`.
|
|
The templates render `.CoverURL`, never `.Cover`. The id is matched against a
|
|
UUID regex before it reaches the browser: the stored value is client-supplied,
|
|
so an unchecked one is an SSRF primitive pointed at the deployment's own
|
|
network.
|
|
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
|
(renders the bindmounted script with the acting Reader's derived credential
|
|
substituted in — the credential never appears in page markup, the address
|
|
bar, or a redirect; `?download=1` adds `Content-Disposition: attachment` for
|
|
mobile Violentmonkey, which ignores a `.user.js` navigation) and
|
|
`POST /rotate-token` (atomic epoch bump + hash
|
|
rewrite; invalidates every installed copy, so the panel warns to reinstall
|
|
on all devices).
|
|
Owner-only `POST /readers/{id}/revoke` (drops one Reader's session rows and
|
|
re-renders the `readers` panel; 404 for any non-owner) is the only route that
|
|
reaches across Readers.
|