sulthan
7c7d597019
Closes #63 Deletes the second way to reach a Cover. Since #62, every Site's cover bytes land in the content-addressed store at creation or on the poll, and the one public route serves them all — nothing needs the kagane proxy anymore. ## What went - **Template-level rewrite:** `Bookmark.CoverURL()` and both templates' use of it. Cards and chrome now render `.Cover` — the wire value — and nothing else. `Bookmark.CoverSource` was dead once `CoverURL` went, so it and its `bookmarkColumns` entry are gone too. - **Kagane-only cover route and its identifier validation:** `GET /img/kagane/{id}`, `web.CoverFetcher`, `coverIDRe`, and the whole `internal/web/cover.go`. - **The proxy's persistence:** `store.KaganeImageID`, `GetKaganeCover`, `PutKaganeCover`, `kaganeCoverSourceURL`, `kaganeCoverRe`. - **The kagane-shaped branch in the byte-fetch routing:** `fetchCoverBytes` no longer takes a `site` argument and no longer names a Site. The URL shape kagane's API publishes is claimed by the browser module itself — `kaganeImageURLRe` + `browserCoverURL` live in `latest/browser.go` with the rest of the per-Site knowledge — and `BrowserFetcher.Image` is now URL-driven (it validates the URL it will navigate to, same SSRF discipline as before). The no-plain-TLS-fallback rule for a claimed URL is preserved: a claimed address with no browser is an error, never a challenge-page fetch. ## What stayed (deliberately) - `BrowserFetcher.Image` and the browser-backed acquisition path: kagane genuinely serves cover bytes behind the challenge + `cross-origin-resource-policy: same-origin`, so the sidecar remains the only fetcher for them — it just routes by URL claim now instead of by Site name. - `fetcherFor`'s per-Site page routing (kagane/novelfull page fetches) — that is the page path, not a cover path. ## Acceptance criteria - [x] Template-level kagane cover rewrite gone - [x] Kagane-only cover route and its identifier validation gone - [x] Tests removed/rewritten against the general route, guarantees kept: unstored + traversal-shaped addresses serve nothing (`TestPublicCoverRejectsUnknownAddress`), non-image content types never echoed (`TestPublicCoverNeverEchoesNonImage` — new; the store-side gate was already pinned by `TestCoverStoreAcceptsAnySourceURL`). Store reopen-persistence and filesystem content-addressing tests rewritten against `PutCover`/`GetCover`, no guarantee lost. - [x] No Site name in a cover code path outside the acquisition module (`grep kagane backend`: store/web/templates/api are clean; remaining hits are `latest/browser.go` + `latest/sites.go`, tests, docs) - [x] Web UI and panel render Covers for all six Sites (templates render the wire address; panel renders `b.cover` — untouched, it never had a kagane path) - [x] `go test ./...` green ## Verification - `go vet ./...` clean - `go test ./...` — all packages pass (root 16.9s, latest 12.7s, store 12.7s, web 0.004s) - `CGO_ENABLED=0 go build` produces the static binary - Cover-path tests run verbosely: `TestPublicCoverServesStoredBytesUnauthenticated`, `TestPublicCoverRejectsUnknownAddress` (unknown/malformed/traversal/empty), `TestPublicCoverNeverEchoesNonImage`, `TestListRendersAcquiredCover`, `TestAcquireKaganeCoverThroughBrowser`, `TestRunOncePrefetchesKaganeCover`, `TestRunOnceRoutesNonKaganeCoverToPublicFetcher` all pass; the three `SMOKE_*` tests skip without the browser sidecar, as designed Live browser verification of the "web UI and panel render Covers for all six Sites" criterion is being run separately with Playwright against real Site pages and a locally mocked backend. Reviewed-on: #73 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
187 lines
14 KiB
Markdown
187 lines
14 KiB
Markdown
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
|
|
|
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
|
Single binary, split into packages under `backend/internal/`: `store`
|
|
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
|
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
|
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
|
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
|
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
|
`backend/main.go` is the composition root — the only place that wires
|
|
packages together into `newRouter`. Root-level `*_test.go` hold
|
|
integration tests that exercise the full router; unit tests for a
|
|
package live beside it under `internal/`.
|
|
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
|
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
|
file per change, one transaction each, versions recorded in
|
|
`schema_migrations`. Files are **append-only** — editing an applied one
|
|
changes nothing on a database that already ran it. No column probing, no
|
|
data-fixup migrations: both were SQLite-era machinery and are gone.
|
|
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
|
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
|
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
|
must have that `TestMain`.
|
|
- **Reader-owned store, four tables.** `readers` is keyed by Discord user ID
|
|
and carries the SHA-256 of the Reader's userscript credential plus a
|
|
`token_epoch` (issue #24). Credentials are derived, never stored: `token.Token(TOKEN_KEY, discord_id, epoch)` (HMAC, `internal/token`), and only its SHA-256 sits in `readers.token_sha256`, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the **owner** row at startup; its epoch-0 hash is refreshed on every start **only while the row has never been rotated**, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (`Store.EnsureReader`, idempotent on `discord_id`, and it never rewrites an existing row's hash). Rotation is `Store.RotateToken` (epoch bump + hash rewrite in one transaction), driven by the web UI.
|
|
`series` keyed `(site, series_id)`
|
|
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
|
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
|
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
|
|
differs between readers: progress, favourite, lifecycle bucket,
|
|
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
|
|
surrogate id; the wire `key` is derived as `site:series_id` on read — and
|
|
every store read/write is scoped to the reader it names. Auth resolves the
|
|
acting Reader from the presented credential (`httpmw.Auth`) and nothing
|
|
else — there is no unauthenticated-by-Reader route and no global token; the
|
|
reader id travels in the request context. Sync **last-write-wins**; the wire format
|
|
stays flat (ADR-0004). `Store.Upsert` decomposes one flat body across two
|
|
tables and enforces the ownership rule: client `title`/`series_url`/`cover`
|
|
are written only when the series row is new (ADR-0003).
|
|
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
|
- **Web UI:** same binary serve the browser UI on a second
|
|
hostname — `GET /` (list, or login page when no session),
|
|
`GET /auth/discord` + `GET /auth/discord/callback` (Discord OAuth,
|
|
ADR-0002), `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
|
under `/ui/*`. Templates + assets `go:embed`-ed under
|
|
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
|
`internal/` tree, not just `*.go`. Sessions are rows in the `sessions`
|
|
table: the cookie carries only an opaque id, looked up (and expiry-
|
|
checked) on every request, and deleting the row revokes the session.
|
|
Guild membership *is* registration (issue #27): `discordCallback` gates on
|
|
membership (and `DISCORD_REQUIRED_ROLE` when set) and then calls
|
|
`Store.EnsureReader`, so a refusal creates nothing and a returning Reader
|
|
reuses their row. The owner is the only Reader with administrative reach:
|
|
`POST /readers/{id}/revoke` (404 for anyone else) drops that Reader's
|
|
sessions, and the `readers` panel renders only on the owner's page.
|
|
A Reader with no bookmarks at all sees `listView.Fresh`, whose empty state
|
|
offers both install links instead of describing a filter.
|
|
UI mutations read-modify-write
|
|
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
|
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
|
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
|
(correct — served from `/`), but impeccable detector resolves
|
|
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
|
on leading `/` and silently skip file. Relative href don't help
|
|
either: template's directory isn't its served path. So
|
|
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
|
always pass `backend/internal/web/static` too. One finding there,
|
|
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
|
- **Every action that moves series out of list is confirm-gated.**
|
|
Archive, finish, remove each open own `.confirm-row` disclosure
|
|
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
|
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
|
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
|
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
|
error use `--mute`.
|
|
- **Latest-chapter poller:** ticker goroutine in same binary re-check
|
|
each bookmarked series' newest published chapter from backend's own
|
|
network access, so `latest_chapter` stay fresh when user not
|
|
browsing. Second, parallel signal — userscript keep own
|
|
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
|
Two independent clocks: per-series cooldown (`series.latest_checked_at`,
|
|
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
|
The poller walks **Series, not Bookmarks** — a series referenced by several
|
|
bookmarks is fetched once per cycle, and the due queue orders
|
|
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
|
*before* fetch so broken series wait out full cooldown instead of retrying
|
|
every tick; found chapter written straight to the series row via
|
|
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
|
order — is never touched.
|
|
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
|
against fingerprint-based blocking; any failure log and skip. kagane and
|
|
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
|
clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane is simply
|
|
not polled when that's unset, while novelfull falls back to a plain-TLS
|
|
attempt — its challenge is a live time-varying fact, and its cover bytes
|
|
never need the browser. See
|
|
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
|
The poller's series write is a single-column UPDATE
|
|
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
|
it cannot revert read progress or move `updated_at`, so the old
|
|
stale-re-read race is gone with the Get+Upsert flow.
|
|
- **Covers are acquired at creation, then served from our own origin
|
|
(ADR-0007):** the first Bookmark of a Series fires `Store.OnSeriesCreated`,
|
|
which `latest.Acquirer` turns into one series-page fetch yielding both the
|
|
Latest Chapter and the cover URL; the bytes then go through
|
|
`latest.CoverBytesFetcher` into `Store.SetSeriesCover`. It runs in a
|
|
goroutine — the Reader's PUT must neither block on a Site nor fail with one
|
|
— and every failure is logged and dropped, leaving the Bookmark intact. The
|
|
wire's `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
|
|
bytes exist and `""` before, never an address that 404s. `GET /covers/{addr}`
|
|
is public and uncredentialed: the userscript renders it on a Site's origin,
|
|
where no cookie or token of ours travels. A client-sent `cover` is decoded
|
|
and discarded, permanently (ADR-0004 compatibility).
|
|
Browser-backed Sites join the same pipeline (issue #62): kagane pages *and*
|
|
cover bytes go through the browser sidecar (nothing falls back to a plain
|
|
fetch, which would only retrieve a challenge page), while novelfull needs
|
|
the browser only for its HTML — the cover URL comes out of the
|
|
browser-fetched page and the bytes go over plain TLS. With no browser
|
|
configured, kagane Covers are simply absent; novelfull still gets one — at
|
|
creation and on the poll — when its page body happens to answer a plain
|
|
request (the challenge is a live time-varying fact). The old kagane-only
|
|
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
|
|
(issue #63): the one public route serves every Site.
|
|
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
|
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
|
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
|
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
|
checking archived series and skip finished ones. `finished` settable
|
|
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
|
**Empty incoming status means "keep stored one"** — resolved on the
|
|
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
|
`excluded.*` is post-evaluation row and default applied there would
|
|
wipe bucket on every PUT from client that predates column. See
|
|
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
|
- **Config via env:** `TOKEN_KEY` (derives every Reader's userscript credential;
|
|
required), `OWNER_DISCORD_ID` (seeds the owner Reader — the administrator and
|
|
the owner of every pre-registration bookmark; required),
|
|
`ALLOWED_ORIGINS` (comma list),
|
|
`DATABASE_URL` (Postgres connection URL, required — no default),
|
|
`COVER_DIR` (required filesystem volume for content-addressed Cover bytes),
|
|
`PUBLIC_BASE_URL` (required origin this deployment answers on, trailing
|
|
slash trimmed; every Cover URL on the wire is built from it, absolute
|
|
because the userscript renders on a Site's origin — ADR-0007),
|
|
`PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
|
|
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
|
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
|
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
|
|
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/
|
|
`_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on,
|
|
`1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both
|
|
cooldowns have a `15m` floor).
|
|
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
|
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
|
defaults `/userscript/manga-bookmark.user.js` and
|
|
`/userscript/novel-bookmark.user.js`, both supplied by bindmount; the
|
|
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
|
Reader's credential at serve time).
|
|
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
|
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
|
Used by the poller for kagane and novelfull page fetches and by the cover
|
|
pipeline for kagane's image bytes (the browser is the only route that clears
|
|
the challenge kagane serves its covers behind); unset — the default —
|
|
disables browser polling and leaves kagane Covers blank until stored bytes
|
|
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
|
`/json/version` for any Host that isn't an IP or `localhost`).
|
|
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
|
|
served by the one public `GET /covers/{addr}` route from content-addressed
|
|
bytes. There is no proxy, no per-Site rewrite, no second place that decides
|
|
a Cover's renderable address: the wire `cover` is it. The only place a Site
|
|
name still appears in cover code is the extraction module (`latest`), where
|
|
kagane's image URLs are claimed by `browserOnlyCoverURL` — they answer a
|
|
plain fetch with a challenge and `cross-origin-resource-policy: same-origin`;
|
|
every other Site's CDN answers plain TLS. Templates render `.Cover` — the
|
|
wire value — never anything else.
|
|
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
|
(renders the bindmounted script with the acting Reader's derived credential
|
|
substituted in — the credential never appears in page markup, the address
|
|
bar, or a redirect; `?download=1` adds `Content-Disposition: attachment` for
|
|
mobile Violentmonkey, which ignores a `.user.js` navigation) and
|
|
`POST /rotate-token` (atomic epoch bump + hash
|
|
rewrite; invalidates every installed copy, so the panel warns to reinstall
|
|
on all devices).
|
|
Owner-only `POST /readers/{id}/revoke` (drops one Reader's session rows and
|
|
re-renders the `readers` panel; 404 for any non-owner) is the only route that
|
|
reaches across Readers.
|