mangaBookmark/backend/AGENTS.md

Guidance for OpenCode (and Claude Code) working under backend/. See root AGENTS.md for the project-wide architecture diagram, hard constraints, and design system.

• Backend (backend/): stdlib net/http (handful routes, no framework) + Postgres over jackc/pgx/v5 (pure Go, CGO_ENABLED=0 -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain :8080. Single binary, split into packages under backend/internal/: store (Bookmark type, Postgres persistence, migration runner), latest (background poller, site parsers, TLS fetcher), session (cookie signing, login rate limiter), httpmw (Auth/Gzip/CORS middleware), api (JSON bookmark handlers), userscript (userscript-serving handler), web (browser UI handler + templates/ + static/, go:embed-ed). backend/main.go is the composition root — the only place that wires packages together into newRouter. Root-level *_test.go hold integration tests that exercise the full router; unit tests for a package live beside it under internal/.
• Schema is migration-owned. internal/store/migrations/*.sql is go:embed-ed and applied on every start by store.migrate: one numbered file per change, one transaction each, versions recorded in schema_migrations. Files are append-only — editing an applied one changes nothing on a database that already ran it. No column probing, no data-fixup migrations: both were SQLite-era machinery and are gone.
• Tests need Docker. internal/pgtest starts one postgres:17-alpine container per test binary (TestMain -> pgtest.Main) and hands each test its own database (pgtest.URL(t)). A package whose tests touch the store must have that TestMain.
• Reader-owned store, four tables. readers is keyed by Discord user ID and carries the SHA-256 of the Reader's userscript credential plus a token_epoch (issue #24). Credentials are derived, never stored: token.Token(TOKEN_KEY, discord_id, epoch) (HMAC, internal/token), and only its SHA-256 sits in readers.token_sha256, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the owner row at startup; its epoch-0 hash is refreshed on every start only while the row has never been rotated, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (Store.EnsureReader, idempotent on discord_id, and it never rewrites an existing row's hash). Rotation is Store.RotateToken (epoch bump + hash rewrite in one transaction), driven by the web UI. series keyed (site, series_id) (asura|demonic|comix|kagane|novelfull|lightnovelworld) owns the shared facts — title, cover, canonical URL, kind (manga|novel), Latest Chapter, latest_checked_at, and the Sighting pair latest_sighted_at/latest_raised_by (issue #103) — and bookmarks holds only what differs between readers: progress, favourite, lifecycle bucket, updated_at. A bookmark is keyed (reader_id, site, series_id) — no surrogate id; the wire key is derived as site:series_id on read — and every store read/write is scoped to the reader it names. Auth resolves the acting Reader from the presented credential (httpmw.Auth) and nothing else — there is no unauthenticated-by-Reader route and no global token; the reader id travels in the request context. Sync last-write-wins; the wire format stays flat (ADR-0004). Store.Upsert decomposes one flat body across two tables and enforces the ownership rule: client title/series_url/cover are written only when the series row is new (ADR-0003).
• Endpoints: GET /bookmarks, PUT /bookmarks/{key} (upsert; see updated_at rule below), DELETE /bookmarks/{key}, GET /healthz (no auth).
• Web UI: same binary serve the browser UI on a second hostname — GET / (list, or login page when no session), GET /auth/discord + GET /auth/discord/callback (Discord OAuth, ADR-0002), POST /logout, GET /static/*, htmx fragment endpoints under /ui/*. Templates + assets go:embed-ed under backend/internal/web/, so backend/Dockerfile must copy the whole internal/ tree, not just *.go. Sessions are rows in the sessions table: the cookie carries only an opaque id, looked up (and expiry- checked) on every request, and deleting the row revokes the session. Guild membership is registration (issue #27): discordCallback gates on membership (and DISCORD_REQUIRED_ROLE when set) and then calls Store.EnsureReader, so a refusal creates nothing and a returning Reader reuses their row. The owner is the only Reader with administrative reach: POST /readers/{id}/revoke (404 for anyone else) drops that Reader's sessions, and the readers panel renders only on the owner's page. A Reader with no bookmarks at all sees listView.Fresh, whose empty state offers both install links instead of describing a filter. UI mutations read-modify-write through Store.Get + Store.Upsert so updated_at rule stays one place. See docs/superpowers/specs/2026-07-25-web-ui-design.md. Design-tool caveat: templates link /static/style.css root-absolutely (correct — served from /), but impeccable detector resolves stylesheet href with path.resolve(fileDir, href), drops directory on leading / and silently skip file. Relative href don't help either: template's directory isn't its served path. So detect.mjs backend/internal/web/templates reports false clean — always pass backend/internal/web/static too. One finding there, overused-font on "Instrument Serif", deliberate identity choice, not debt.
• Every action that moves series out of list is confirm-gated. Archive, finish, remove each open own .confirm-row disclosure (toggleConfirmRow(key, kind) in filter.js, kind ∈ archive|finish|remove); restore fire instantly since it's the reversal. Remove's row wear ember wash, two reversible ones wear .calm grey. --ember stay reserved for new-chapter signal: busy bar and inline error use --mute.
• Latest-chapter poller: one goroutine per Site (a Poll Lane, issue #100), each re-checking that Site's bookmarked series' newest published chapter from backend's own network access, so latest_chapter stays fresh when the user isn't browsing. Second, parallel signal — the userscript keeps its own maybeCaptureLatestOnSeriesPage/backgroundRefreshLatest schedule, and its reportLatestChapter PUTs every read, unchanged numbers included, because an unchanged read is exactly the Sighting worth deferring a Poll on (#103). Two independent clocks: per-series rest (series.latest_checked_at, enforced by Store.DueForLatestCheck's WHERE clause — now - Rest) and per-Lane gap (the Lane sleeping between fetches, effectiveGap). Both live in the Site registry (internal/latest/sites.go), not config: the five env knobs that used to size a shared pace are gone. The poller walks Series, not Bookmarks — a series referenced by several bookmarks is fetched once per cycle, and the due queue orders reader_count DESC, latest_checked_at ASC (ADR-0003). Series row stamped before fetch so broken series wait out the rest instead of retrying every tick; found chapter written straight to the series row via Store.SetLatestChapter, so a bookmark's updated_at — and the list order — is never touched. Sightings (issue #103, ADR-0011) let a Reader's own page read defer a Poll: Store.RecordSighting — called by the PUT handler before the Upsert, because the raise test needs the row as it stands — stamps series.latest_sighted_at and, when the report raises the stored number, names its Reader in series.latest_raised_by. The due query's HAVING clause is where deferral lives: a Series is skipped only while it has exactly one Bookmark, was sighted within one Rest, and is under the ceiling (sightingCeilingRests, six of that Site's rests) since its last Poll. So a shared Series is never deferred, and no Series goes six hours unpolled whatever arrives. checkOne judges the named Reader off the comparison it already makes: a lower number is a contradiction (logged with the Reader and both numbers), the same number an agreement, a higher number the Site publishing and neither — that last one clears the attribution instead, since the value the Poll then stores is its own and a later retraction is not the Reader's fault. Three contradictions (store.SightingDisagreementLimit) stop that Reader deferring — their reports still write the Latest Chapter — and twenty consecutive agreements (store.SightingAgreementsToClear) forgive them, as does the owner's clear-marks control. Deferral is recomputed from live facts every round, so nothing needs invalidating when a Series gains a second Bookmark; the one input read earlier is the Reader's marks, checked when the Sighting is recorded, so crossing the threshold or being cleared takes effect from that Reader's next Sighting and the standing already bought lasts out its rest. Refusals and browser loss are Lane-local: two errChallengeHeld in one pass stop that Site for refuseBackoff (15m) while other Lanes continue; an errBrowserInterrupted (remote Chrome restart) sets a shared Poller flag that makes the other browser Lanes skip their passes for the same 15m, so a restarting Chrome doesn't stamp one Series per Lane per pass — after the window the flag decays and they probe again. Browser Lanes wake Chrome only when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser), and cover work (both healing a stored source URL and filling a blank from the series page) runs in the background so a slow CDN can't consume a Lane's gap. A refusal is only ever the challenge page: isInterstitial matches the orchestration path /cdn-cgi/challenge-platform/h/, never the bare prefix. Cloudflare injects /cdn-cgi/challenge-platform/scripts/jsd/main.js into ordinary 200 pages once a zone turns JS detections on, which demonic did on 2026-08-16 — the prefix match then read every real demonic page as a refusal and parked that Lane in 15m backoff while plain TLS was returning the full series page. Fetches use bogdanfinn/tls-client with Chrome profile as defence in depth against fingerprint-based blocking; any failure log and skip. kagane, comix and novelfull sit behind Cloudflare JavaScript challenges the TLS client can't clear, so they are fetched over CDP via BROWSER_WS_URL; kagane and comix are simply not polled when that's unset, while novelfull falls back to a plain-TLS attempt — its challenge is a live time-varying fact, and its cover bytes never need the browser. comix's browser read is an in-tab fetch() of the Series URL, not a DOM render: it is an SPA, so rendering costs ~65 requests for the same server-rendered HTML one fetch returns (measured 2026-08-12, issue #98). See docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md. The poller's series write is a single-column UPDATE (Store.SetLatestChapter), not a read-modify-write of the whole bookmark: it cannot revert read progress or move updated_at, so the old stale-re-read race is gone with the Get+Upsert flow.
• Covers are acquired at creation, then served from our own origin (ADR-0007): the first Bookmark of a Series fires Store.OnSeriesCreated, which latest.Acquirer turns into one series-page fetch yielding both the Latest Chapter and the cover URL; the bytes then go through latest.CoverBytesFetcher into Store.SetSeriesCover. It runs in a goroutine — the Reader's PUT must neither block on a Site nor fail with one — and every failure is logged and dropped, leaving the Bookmark intact. The wire's cover is the absolute PUBLIC_BASE_URL + /covers/{sha256} once bytes exist and "" before, never an address that 404s. GET /covers/{addr} is public and uncredentialed: the userscript renders it on a Site's origin, where no cookie or token of ours travels. A client-sent cover is decoded and discarded, permanently (ADR-0004 compatibility). Browser-backed Sites join the same pipeline (issue #62, extended to comix by #98): kagane and comix pages and cover bytes go through the browser sidecar (nothing falls back to a plain fetch, which would only retrieve a challenge page), while novelfull needs the browser only for its HTML — the cover URL comes out of the browser-fetched page and the bytes go over plain TLS. With no browser configured, kagane and comix Covers are simply absent; novelfull still gets one — at creation and on the poll — when its page body happens to answer a plain request (the challenge is a live time-varying fact). comix cover bytes must arrive by direct navigation, not an in-page fetch: its Series page sets cross-origin-embedder-policy: require-corp, which fails a page-context fetch of static.comix.to. The old kagane-only serving path (/img/kagane/{id}, template rewrite, CoverFetcher) is gone (issue #63): the one public route serves every Site.
• updated_at drives list order, so moves only on real reading progress: server apply its timestamp when row new or last_chapter_num changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. PUT therefore returns row as stored, clients must adopt that response rather than own payload. See plans/2026-07-25-bookmark-list-favorites-design.md §4.
• Lifecycle buckets: status on each bookmark is reading | archived | finished, orthogonal to favorite. Archived and finished appear only in own tab — not in All, Updated, Favourites, or recent strip. Poller keeps checking archived series and skip finished ones. finished settable only from web UI; PUT /bookmarks/{key} reject it with 400. Empty incoming status means "keep stored one" — resolved on the VALUES side of Store.Upsert, not conflict clause, since excluded.* is post-evaluation row and default applied there would wipe bucket on every PUT from client that predates column. See docs/superpowers/specs/2026-07-27-status-buckets-design.md.
• Config via env: TOKEN_KEY (derives every Reader's userscript credential; required), OWNER_DISCORD_ID (seeds the owner Reader — the administrator and the owner of every pre-registration bookmark; required), ALLOWED_ORIGINS (comma list), DATABASE_URL (Postgres connection URL, required — no default), COVER_DIR (required filesystem volume for content-addressed Cover bytes), PUBLIC_BASE_URL (required origin this deployment answers on, trailing slash trimmed; every Cover URL on the wire is built from it, absolute because the userscript renders on a Site's origin — ADR-0007), PORT (default 8080), DISCORD_CLIENT_ID/_CLIENT_SECRET/_GUILD_ID/ _REDIRECT_URI (required; Discord OAuth for the browser UI), DISCORD_REQUIRED_ROLE (optional role gate, empty by default), DISCORD_API_BASE (default https://discord.com/api/v10), LATEST_CHAPTER_POLL_ENABLED (background latest-chapter poller kill switch, default on). Pace is per Site in the registry (issue #100): every Site rests an hour and gaps ten seconds, a Site with more eligible Series than 360 tightens its own gap toward the 1s floor, and browser Lanes wake Chrome only on demand (ADR-0005). The _COOLDOWN/_BROWSER_COOLDOWN/ _INTERVAL/_BATCH/_STAGGER knobs that used to size a shared pace are gone. The 1h rest for browser Sites is safe on documented grounds: a challenged page costs seconds of a serialized single-tab browser, free-plan zones have no bot score and no published per-IP rate input, and cf_clearance expires in 30 minutes so every cadence at or above 1h re-solves anyway — docs/research/cloudflare-bot-scoring-and-poll-cadence.md. USERSCRIPT_PATH and NOVEL_USERSCRIPT_PATH (files served at /u/{token}/manga-bookmark.user.js and /u/{token}/novel-bookmark.user.js, defaults /userscript/manga-bookmark.user.js and /userscript/novel-bookmark.user.js, both supplied by bindmount; the __API_TOKEN__ placeholder inside them is substituted with the requesting Reader's credential at serve time). BROWSER_WS_URL (CDP endpoint of the browser, which runs on a separate machine and is reached over the tailnet — ADR-0006, chrome/docker-compose.yml. Used by the poller for kagane, comix and novelfull page fetches and by the cover pipeline for kagane's and comix's image bytes (the browser is the only route that clears the challenge those two serve their covers behind); unset — the default — disables browser polling and leaves kagane and comix Covers blank until stored bytes exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s /json/version for any Host that isn't an IP or localhost).
• No per-Site cover path (issue #63): every Cover — all six Sites — is served by the one public GET /covers/{addr} route from content-addressed bytes. There is no proxy, no per-Site rewrite, no second place that decides a Cover's renderable address: the wire cover is it. The only place a Site name still appears in cover code is the extraction module (latest), where kagane's and comix's image URLs are claimed by browserOnlyCoverURL — kagane answers a plain fetch with a challenge and cross-origin-resource-policy: same-origin, and static.comix.to answers one with the same Cloudflare challenge its pages serve; every other Site's CDN answers plain TLS. Templates render .Cover — the wire value — never anything else.
• Web UI also owns: session-gated GET /install/{manga,novel}-bookmark.user.js (renders the bindmounted script with the acting Reader's derived credential substituted in — the credential never appears in page markup, the address bar, or a redirect; ?download=1 adds Content-Disposition: attachment for mobile Violentmonkey, which ignores a .user.js navigation) and POST /rotate-token (atomic epoch bump + hash rewrite; invalidates every installed copy, so the panel warns to reinstall on all devices).
• Owner-only admin page (internal/web/admin.go, issue #102): GET /admin carries the Reader roster (sessions, Sighting counters, POST /readers/{id}/revoke and POST /readers/{id}/clear-marks) and Poll Lane status (GET /ui/admin/lanes, self-refreshing every 30s). Every route that reaches past the acting Reader is listed in adminRoutes() and wrapped in requireOwner at registration — add a route there, not a check inside a handler; web.AdminPatterns() is what the gate test walks. A non-owner gets 404, never 403. Lane figures come from the running poller through the web.LaneReporter seam (latest.Poller.LaneStatus), never from a table: a nil reporter or a Lane that has not finished a pass renders "no data yet" rather than zeroes. main.newRouter takes the reporter as an interface and converts a nil *Poller to a nil interface — a typed nil would make the page claim a poller exists. The one owner comparison left outside requireOwner is in index (view.Owner = readerID == h.store.OwnerID()): it gates a link, not an endpoint, so it is a rendering decision a registration-time wrapper cannot express — do not "unify" it into the gate. A Lane pass that returns before computing its figures (refusal backoff, sidecar down) carries the previous pass's due count and gap forward rather than recording zeroes; a Lane that has never reached a pace renders no gap at all. Checked next to Due is what separates a stopped Lane from a quiet one, so neither figure may be dropped from the row.