docs: make every AGENTS.md cite code, not docs or issues
A spec, ADR, plan file, or Gitea issue records what was true when it was written and then goes stale silently, so an agent that follows the pointer reads a decision that may already have been reversed. Code is the only source true at read time. Strip every non-code citation from the three AGENTS.md files (ADRs, spec and plan files, docs/research, docs/agents/*, DEPLOY/REDEPLOY, and issue numbers), restating inline any fact the linked doc actually carried: the tea command set and triage label strings move into the root Forge section. The Domain docs subsection goes entirely, as it pointed only at CONTEXT.md and docs/adr/, neither of which exists. Then rewrite the backend and userscript files around derivability, since prose that restates mechanism rots the same way a doc link does. Structure and mechanism now name a symbol and stop; rationale, rejected alternatives and dated measurements stay written out, because code cannot carry them. Record that split as a rule in the root file. Verified by extracting all 118 backticked identifiers and checking each against the Go, JS, SQL, HTML and CSS sources. That caught one claim that was already lying: the old cover text said CoverFetcher was gone, but NewCoverFetcher, TLSCoverFetcher and BrowserCoverFetcher are all live in internal/latest, so the sentence now names only the dead /img/kagane route. Also drop the "Guidance for OpenCode (and Claude Code)" openers, so the files read the same under any harness.
This commit is contained in:
+264
-270
@@ -1,274 +1,268 @@
|
||||
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
||||
Scope: `backend/`.
|
||||
|
||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
||||
Single binary, split into packages under `backend/internal/`: `store`
|
||||
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
||||
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
||||
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
||||
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
||||
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
||||
`backend/main.go` is the composition root — the only place that wires
|
||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
||||
integration tests that exercise the full router; unit tests for a
|
||||
package live beside it under `internal/`.
|
||||
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
||||
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
||||
file per change, one transaction each, versions recorded in
|
||||
`schema_migrations`. Files are **append-only** — editing an applied one
|
||||
changes nothing on a database that already ran it. No column probing, no
|
||||
data-fixup migrations: both were SQLite-era machinery and are gone.
|
||||
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
||||
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
||||
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
||||
must have that `TestMain`.
|
||||
- **Reader-owned store, four tables.** `readers` is keyed by Discord user ID
|
||||
and carries the SHA-256 of the Reader's userscript credential plus a
|
||||
`token_epoch` (issue #24). Credentials are derived, never stored: `token.Token(TOKEN_KEY, discord_id, epoch)` (HMAC, `internal/token`), and only its SHA-256 sits in `readers.token_sha256`, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the **owner** row at startup; its epoch-0 hash is refreshed on every start **only while the row has never been rotated**, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (`Store.EnsureReader`, idempotent on `discord_id`, and it never rewrites an existing row's hash). Rotation is `Store.RotateToken` (epoch bump + hash rewrite in one transaction), driven by the web UI.
|
||||
`series` keyed `(site, series_id)`
|
||||
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
||||
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
||||
Latest Chapter, `latest_checked_at`, and the Sighting pair
|
||||
`latest_sighted_at`/`latest_raised_by` (issue #103) — and `bookmarks` holds only what
|
||||
differs between readers: progress, favourite, lifecycle bucket,
|
||||
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
|
||||
surrogate id; the wire `key` is derived as `site:series_id` on read — and
|
||||
every store read/write is scoped to the reader it names. Auth resolves the
|
||||
acting Reader from the presented credential (`httpmw.Auth`) and nothing
|
||||
else — there is no unauthenticated-by-Reader route and no global token; the
|
||||
reader id travels in the request context. Sync **last-write-wins**; the wire format
|
||||
stays flat (ADR-0004). `Store.Upsert` decomposes one flat body across two
|
||||
tables and enforces the ownership rule: client `title`/`series_url`/`cover`
|
||||
are written only when the series row is new (ADR-0003).
|
||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
||||
- **Web UI:** same binary serve the browser UI on a second
|
||||
hostname — `GET /` (list, or login page when no session),
|
||||
`GET /auth/discord` + `GET /auth/discord/callback` (Discord OAuth,
|
||||
ADR-0002), `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
||||
under `/ui/*`. Templates + assets `go:embed`-ed under
|
||||
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
||||
`internal/` tree, not just `*.go`. Sessions are rows in the `sessions`
|
||||
table: the cookie carries only an opaque id, looked up (and expiry-
|
||||
checked) on every request, and deleting the row revokes the session.
|
||||
Guild membership *is* registration (issue #27): `discordCallback` gates on
|
||||
membership (and `DISCORD_REQUIRED_ROLE` when set) and then calls
|
||||
`Store.EnsureReader`, so a refusal creates nothing and a returning Reader
|
||||
reuses their row. The owner is the only Reader with administrative reach:
|
||||
`POST /readers/{id}/revoke` (404 for anyone else) drops that Reader's
|
||||
sessions, and the `readers` panel renders only on the owner's page.
|
||||
A Reader with no bookmarks at all sees `listView.Fresh`, whose empty state
|
||||
offers both install links instead of describing a filter.
|
||||
UI mutations read-modify-write
|
||||
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
||||
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
||||
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
||||
(correct — served from `/`), but impeccable detector resolves
|
||||
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
||||
on leading `/` and silently skip file. Relative href don't help
|
||||
either: template's directory isn't its served path. So
|
||||
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
||||
always pass `backend/internal/web/static` too. One finding there,
|
||||
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
||||
- **Every action that moves series out of list is confirm-gated.**
|
||||
Archive, finish, remove each open own `.confirm-row` disclosure
|
||||
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
||||
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
||||
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
||||
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
||||
error use `--mute`.
|
||||
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100),
|
||||
each re-checking that Site's bookmarked series' newest published chapter from
|
||||
backend's own network access, so `latest_chapter` stays fresh when the user
|
||||
isn't browsing. Second, parallel signal — the userscript keeps its own
|
||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` schedule, and its
|
||||
`reportLatestChapter` PUTs every read, unchanged numbers included, because an
|
||||
unchanged read is exactly the Sighting worth deferring a Poll on (#103).
|
||||
Two independent clocks: per-series rest (`series.latest_checked_at`,
|
||||
enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and
|
||||
per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
|
||||
in the Site registry (`internal/latest/sites.go`), not config: the five env
|
||||
knobs that used to size a shared pace are gone.
|
||||
The poller walks **Series, not Bookmarks** — a series referenced by several
|
||||
bookmarks is fetched once per cycle, and the due queue orders
|
||||
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
||||
*before* fetch so broken series wait out the rest instead of retrying
|
||||
every tick; found chapter written straight to the series row via
|
||||
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
||||
order — is never touched.
|
||||
**Sightings** (issue #103, ADR-0011) let a Reader's own page read defer a
|
||||
Poll: `Store.RecordSighting` — called by the PUT handler *before* the Upsert,
|
||||
because the raise test needs the row as it stands — stamps
|
||||
`series.latest_sighted_at` and, when the report raises the stored number,
|
||||
names its Reader in `series.latest_raised_by`. The due query's HAVING clause
|
||||
is where deferral lives: a Series is skipped only while it has exactly one
|
||||
Bookmark, was sighted within one Rest, and is under the ceiling
|
||||
(`sightingCeilingRests`, six of that Site's rests) since its last Poll. So a
|
||||
shared Series is never deferred, and no Series goes six hours unpolled
|
||||
whatever arrives. `checkOne` judges the named Reader off the comparison it
|
||||
already makes: a lower number is a contradiction (logged with the Reader and
|
||||
both numbers), the same number an agreement, a higher number the Site
|
||||
publishing and neither — that last one clears the attribution instead, since
|
||||
the value the Poll then stores is its own and a later retraction is not the
|
||||
Reader's fault. Three contradictions
|
||||
(`store.SightingDisagreementLimit`) stop that Reader deferring — their
|
||||
reports still write the Latest Chapter — and twenty consecutive agreements
|
||||
(`store.SightingAgreementsToClear`) forgive them, as does the owner's
|
||||
clear-marks control. Deferral is recomputed from live facts every round, so
|
||||
nothing needs invalidating when a Series gains a second Bookmark; the one
|
||||
input read earlier is the Reader's marks, checked when the Sighting is
|
||||
recorded, so crossing the threshold or being cleared takes effect from that
|
||||
Reader's next Sighting and the standing already bought lasts out its rest.
|
||||
Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
|
||||
stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
|
||||
`errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
|
||||
that makes the other browser Lanes skip their passes for the same 15m, so a
|
||||
restarting Chrome doesn't stamp one Series per Lane per pass — after the
|
||||
window the flag decays and they probe again. Browser Lanes wake Chrome only
|
||||
when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
|
||||
and cover work (both healing a stored source URL and filling a blank from
|
||||
the series page) runs in the background so a slow CDN can't consume a
|
||||
Lane's gap.
|
||||
A refusal is only ever the challenge *page*: `isInterstitial` matches the
|
||||
orchestration path `/cdn-cgi/challenge-platform/h/`, never the bare prefix.
|
||||
Cloudflare injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into
|
||||
ordinary 200 pages once a zone turns JS detections on, which demonic did on
|
||||
2026-08-16 — the prefix match then read every real demonic page as a refusal
|
||||
and parked that Lane in 15m backoff while plain TLS was returning the full
|
||||
series page.
|
||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||
against fingerprint-based blocking; any failure log and skip. kagane, comix
|
||||
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
|
||||
can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and
|
||||
comix are simply not polled when that's unset, while novelfull falls back to
|
||||
a plain-TLS attempt — its challenge is a live time-varying fact, and its
|
||||
cover bytes never need the browser. comix's browser read is an in-tab
|
||||
`fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
|
||||
costs ~65 requests for the same server-rendered HTML one fetch returns
|
||||
(measured 2026-08-12, issue #98). See
|
||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||
The poller's series write is a single-column UPDATE
|
||||
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
||||
it cannot revert read progress or move `updated_at`, so the old
|
||||
stale-re-read race is gone with the Get+Upsert flow.
|
||||
- **Covers are acquired at creation, then served from our own origin
|
||||
(ADR-0007):** the first Bookmark of a Series fires `Store.OnSeriesCreated`,
|
||||
which `latest.Acquirer` turns into one series-page fetch yielding both the
|
||||
Latest Chapter and the cover URL; the bytes then go through
|
||||
`latest.CoverBytesFetcher` into `Store.SetSeriesCover`. It runs in a
|
||||
goroutine — the Reader's PUT must neither block on a Site nor fail with one
|
||||
— and every failure is logged and dropped, leaving the Bookmark intact. The
|
||||
wire's `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
|
||||
bytes exist and `""` before, never an address that 404s. `GET /covers/{addr}`
|
||||
is public and uncredentialed: the userscript renders it on a Site's origin,
|
||||
where no cookie or token of ours travels. A client-sent `cover` is decoded
|
||||
and discarded, permanently (ADR-0004 compatibility).
|
||||
Browser-backed Sites join the same pipeline (issue #62, extended to comix by
|
||||
#98): kagane and comix pages *and* cover bytes go through the browser sidecar
|
||||
(nothing falls back to a plain fetch, which would only retrieve a challenge
|
||||
page), while novelfull needs the browser only for its HTML — the cover URL
|
||||
comes out of the browser-fetched page and the bytes go over plain TLS. With
|
||||
no browser configured, kagane and comix Covers are simply absent; novelfull
|
||||
still gets one — at creation and on the poll — when its page body happens to
|
||||
answer a plain request (the challenge is a live time-varying fact). comix
|
||||
cover bytes must arrive by direct navigation, not an in-page fetch: its
|
||||
Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
|
||||
page-context fetch of `static.comix.to`. The old kagane-only
|
||||
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
|
||||
(issue #63): the one public route serves every Site.
|
||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
||||
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
||||
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
||||
checking archived series and skip finished ones. `finished` settable
|
||||
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
||||
**Empty incoming status means "keep stored one"** — resolved on the
|
||||
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
||||
`excluded.*` is post-evaluation row and default applied there would
|
||||
wipe bucket on every PUT from client that predates column. See
|
||||
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
||||
- **Config via env:** `TOKEN_KEY` (derives every Reader's userscript credential;
|
||||
required), `OWNER_DISCORD_ID` (seeds the owner Reader — the administrator and
|
||||
the owner of every pre-registration bookmark; required),
|
||||
`ALLOWED_ORIGINS` (comma list),
|
||||
`DATABASE_URL` (Postgres connection URL, required — no default),
|
||||
`COVER_DIR` (required filesystem volume for content-addressed Cover bytes),
|
||||
`PUBLIC_BASE_URL` (required origin this deployment answers on, trailing
|
||||
slash trimmed; every Cover URL on the wire is built from it, absolute
|
||||
because the userscript renders on a Site's origin — ADR-0007),
|
||||
`PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
|
||||
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
||||
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
||||
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
|
||||
`LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill
|
||||
switch, default on). Pace is per Site in the registry (issue #100): every
|
||||
Site rests an hour and gaps ten seconds, a Site with more eligible Series
|
||||
than 360 tightens its own gap toward the 1s floor, and browser Lanes wake
|
||||
Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/
|
||||
`_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are
|
||||
gone. The 1h rest for browser Sites is safe on documented grounds: a
|
||||
challenged page costs seconds of a serialized single-tab browser, free-plan
|
||||
zones have no bot score and no published per-IP rate input, and
|
||||
`cf_clearance` expires in 30 minutes so every cadence at or above 1h
|
||||
re-solves anyway —
|
||||
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
|
||||
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
||||
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
||||
defaults `/userscript/manga-bookmark.user.js` and
|
||||
`/userscript/novel-bookmark.user.js`, both supplied by bindmount; the
|
||||
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
||||
Reader's credential at serve time).
|
||||
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
||||
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
||||
Used by the poller for kagane, comix and novelfull page fetches and by the
|
||||
cover pipeline for kagane's and comix's image bytes (the browser is the only
|
||||
route that clears the challenge those two serve their covers behind); unset —
|
||||
the default — disables browser polling and leaves kagane and comix Covers
|
||||
blank until stored bytes
|
||||
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
||||
`/json/version` for any Host that isn't an IP or `localhost`).
|
||||
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
|
||||
served by the one public `GET /covers/{addr}` route from content-addressed
|
||||
bytes. There is no proxy, no per-Site rewrite, no second place that decides
|
||||
a Cover's renderable address: the wire `cover` is it. The only place a Site
|
||||
name still appears in cover code is the extraction module (`latest`), where
|
||||
kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane
|
||||
answers a plain fetch with a challenge and
|
||||
Each entry names the code that holds the truth — read that for *what it does*.
|
||||
The prose here is only what code cannot tell you: rationale, rejected
|
||||
alternatives, dated measurements, and invariants a plausible refactor would
|
||||
silently break.
|
||||
|
||||
### Layout
|
||||
|
||||
`backend/main.go` → `newRouter` is the composition root, the only place
|
||||
packages are wired. Packages under `backend/internal/`: `store`, `latest`,
|
||||
`session`, `httpmw`, `api`, `userscript`, `web`, `token`, `pgtest`. Root-level
|
||||
`*_test.go` exercise the full router; unit tests live beside their package.
|
||||
|
||||
Not visible from any single file: stdlib `net/http` with no framework,
|
||||
Postgres over `jackc/pgx/v5`, `CGO_ENABLED=0` static binary into a distroless
|
||||
image, TLS terminated by the reverse proxy so the service listens plain `:8080`.
|
||||
|
||||
### Schema — `internal/store/migrations/*.sql`, run by `store.migrate`
|
||||
|
||||
- Migration files are **append-only**. Editing an applied one changes nothing
|
||||
on a database that already recorded its version in `schema_migrations`, so
|
||||
the fix silently applies to new deployments only.
|
||||
- No column probing, no data-fixup migrations. Both were SQLite-era machinery
|
||||
and were removed deliberately — don't reintroduce either.
|
||||
|
||||
### Tests need Docker — `internal/pgtest`
|
||||
|
||||
`pgtest.Main` from `TestMain` starts one `postgres:17-alpine` per test binary;
|
||||
`pgtest.URL` hands each test its own database. A package whose tests touch the
|
||||
store must have that `TestMain` or it has no database at all.
|
||||
|
||||
### Reader-owned store — `internal/store`, `internal/token`
|
||||
|
||||
Four tables; shape is in the migrations, behaviour in `Store`'s methods.
|
||||
|
||||
- **Credentials are derived, never stored.** `token.Token(TOKEN_KEY, discord_id, epoch)`
|
||||
is an HMAC; only its SHA-256 reaches `readers.token_sha256`. So install URLs
|
||||
can be rebuilt after any restart, and a database leak yields nothing usable.
|
||||
- **The owner's epoch-0 hash is refreshed at startup only while the row has
|
||||
never been rotated.** Drop that condition and a restart resurrects a
|
||||
rotated-away credential.
|
||||
- `Store.EnsureReader` never rewrites an existing row's hash — a returning
|
||||
Reader's login must not invalidate their installed scripts.
|
||||
- **Every read and write is scoped to the acting Reader**, resolved from the
|
||||
presented credential by `httpmw.Auth` and carried in the request context.
|
||||
There is no unauthenticated-by-Reader route and no global token.
|
||||
- **`series` holds what readers share, `bookmarks` only what differs.** A
|
||||
bookmark key is `(reader_id, site, series_id)` with no surrogate id; the wire
|
||||
`key` is derived as `site:series_id` on read.
|
||||
- `Store.Upsert` splits one flat body across both tables and enforces the
|
||||
ownership rule: client `title`/`series_url`/`cover` are written **only when
|
||||
the series row is new**, so one reader cannot retitle a shared series.
|
||||
- Sync is last-write-wins and the wire format stays flat — clients depend on
|
||||
both; neither is an implementation detail to tidy up.
|
||||
|
||||
### Web UI — `internal/web`
|
||||
|
||||
Routes, templates and assets are all in that package; `AdminPatterns()` and
|
||||
`adminRoutes()` enumerate the privileged ones.
|
||||
|
||||
- **`backend/Dockerfile` must copy the whole `internal/` tree**, not just
|
||||
`*.go`: templates and static assets are `go:embed`-ed from
|
||||
`internal/web/`.
|
||||
- **Guild membership *is* registration.** `discordCallback` gates on membership
|
||||
(plus `DISCORD_REQUIRED_ROLE` when set) and only then calls
|
||||
`Store.EnsureReader`, so a refusal creates nothing.
|
||||
- Sessions are rows, not signatures: the cookie carries an opaque id and
|
||||
expiry is checked on lookup, which is what makes deleting the row an instant
|
||||
revocation.
|
||||
- UI mutations go through `Store.Get` + `Store.Upsert` so the `updated_at` rule
|
||||
below stays in exactly one place.
|
||||
- `listView.Fresh` exists because a Reader with no bookmarks at all needs
|
||||
install links, not an empty-filter message.
|
||||
- **Design-tool caveat:** `detect.mjs backend/internal/web/templates` reports a
|
||||
**false clean**. Templates link `/static/style.css` root-absolutely (correct —
|
||||
it is served from `/`), but the detector resolves hrefs with
|
||||
`path.resolve(fileDir, href)`, which drops the directory on a leading `/` and
|
||||
skips the file silently; a relative href doesn't help either, since a
|
||||
template's directory isn't its served path. Always pass
|
||||
`backend/internal/web/static` too. The one finding there, `overused-font` on
|
||||
"Instrument Serif", is a deliberate identity choice, not debt.
|
||||
|
||||
### Confirm gating — `internal/web/static/filter.js`, `toggleConfirmRow(key, kind)`
|
||||
|
||||
Every action that pulls a series out of the list (`archive|finish|remove`) opens
|
||||
its own `.confirm-row`; restore fires instantly because it is the reversal.
|
||||
Remove wears the ember wash, the two reversible ones wear `.calm` grey.
|
||||
**`--ember` is reserved for the new-chapter signal** — the busy bar and inline
|
||||
errors must use `--mute`, or the one colour that means "something to read"
|
||||
stops meaning it.
|
||||
|
||||
### Latest-chapter poller — `internal/latest`, Site registry in `sites.go`
|
||||
|
||||
One goroutine per Site (a Poll Lane) re-checks that Site's bookmarked series
|
||||
from the backend's own network position, so `latest_chapter` stays fresh while
|
||||
nobody is browsing. The userscript's `reportLatestChapter` is a second,
|
||||
parallel signal — it PUTs every read, unchanged numbers included, because an
|
||||
unchanged read is exactly the Sighting worth deferring a Poll on.
|
||||
|
||||
- **Pace lives in the Site registry, not config.** Two clocks: per-series rest
|
||||
(`series.latest_checked_at`, enforced in `Store.DueForLatestCheck`'s WHERE)
|
||||
and per-Lane gap (`effectiveGap`). The five env knobs that used to size one
|
||||
shared pace are gone; don't add them back.
|
||||
- **The poller walks Series, not Bookmarks** — a series several readers hold is
|
||||
fetched once per cycle, and the due queue orders `reader_count DESC,
|
||||
latest_checked_at ASC` so the widely-read ones win contention.
|
||||
- **The series row is stamped *before* the fetch**, so a permanently broken
|
||||
series waits out its rest instead of being retried every tick.
|
||||
- `Store.SetLatestChapter` is a single-column UPDATE, deliberately not a
|
||||
read-modify-write of the bookmark: it therefore cannot revert read progress
|
||||
or move `updated_at`. The old stale-re-read race died with the Get+Upsert
|
||||
flow — don't restore one here.
|
||||
|
||||
**Sightings** (`Store.RecordSighting`, the due query's HAVING clause,
|
||||
`latest.checkOne`) let a Reader's own page read defer a Poll.
|
||||
|
||||
- Recorded by the PUT handler **before** the Upsert, because the raise test
|
||||
needs the row as it stands.
|
||||
- A Series is deferred only while it has exactly one Bookmark, was sighted
|
||||
within one Rest, and is under `sightingCeilingRests` since its last Poll — so
|
||||
a shared Series is never deferred and nothing goes six hours unpolled
|
||||
whatever arrives.
|
||||
- A *higher* report clears the attribution rather than crediting it: the value
|
||||
the Poll then stores is its own, so a later retraction isn't the Reader's
|
||||
fault.
|
||||
- `store.SightingDisagreementLimit` contradictions stop a Reader deferring —
|
||||
their reports still write the Latest Chapter — and
|
||||
`store.SightingAgreementsToClear` agreements forgive them, as does the
|
||||
owner's clear-marks control.
|
||||
- Deferral is recomputed from live facts each round, so nothing needs
|
||||
invalidating when a Series gains a second Bookmark. The one input read
|
||||
earlier is the Reader's marks, so crossing or clearing a threshold takes
|
||||
effect from their next Sighting and the standing already bought lasts out its
|
||||
rest.
|
||||
|
||||
**Refusals and browser loss are Lane-local.** Two `errChallengeHeld` in a pass
|
||||
stop that Site for `refuseBackoff` while other Lanes continue. An
|
||||
`errBrowserInterrupted` (remote Chrome restarted) sets a shared Poller flag so
|
||||
the *other* browser Lanes skip their passes for the same window — otherwise a
|
||||
restarting Chrome stamps one Series per Lane per pass, burning rests on
|
||||
failures. The flag decays and they probe again.
|
||||
|
||||
- **`isInterstitial` matches the orchestration path
|
||||
`/cdn-cgi/challenge-platform/h/`, never the bare prefix.** Cloudflare injects
|
||||
`/cdn-cgi/challenge-platform/scripts/jsd/main.js` into ordinary 200 pages
|
||||
once a zone turns JS detections on, which demonic did on 2026-08-16: the
|
||||
prefix match read every real demonic page as a refusal and parked the Lane in
|
||||
backoff while plain TLS was returning full series pages.
|
||||
- Fetches use `bogdanfinn/tls-client` with a Chrome profile as defence in depth
|
||||
against fingerprint blocking; any failure logs and skips.
|
||||
- kagane, comix and novelfull sit behind Cloudflare JS challenges the TLS
|
||||
client can't clear, so they go over CDP (`BROWSER_WS_URL`). kagane and comix
|
||||
are simply not polled when it's unset — a plain fetch would only retrieve a
|
||||
challenge page — while novelfull still attempts plain TLS, because its
|
||||
challenge is a live time-varying fact and its cover bytes never need a browser.
|
||||
- **comix's browser read is an in-tab `fetch()` of the Series URL, not a DOM
|
||||
render.** It is an SPA: rendering cost ~65 requests for the same
|
||||
server-rendered HTML one fetch returns (measured 2026-08-12).
|
||||
- Browser Lanes wake Chrome only when 5+ Series are due or one has waited 15m,
|
||||
and cover work runs in the background so a slow CDN can't eat a Lane's gap.
|
||||
|
||||
### Covers — `Store.OnSeriesCreated`, `latest.Acquirer`, `latest.CoverBytesFetcher`, `Store.SetSeriesCover`
|
||||
|
||||
Acquired once when the first Bookmark of a Series is created, then served from
|
||||
our own origin by the public `GET /covers/{addr}`.
|
||||
|
||||
- Acquisition runs in a goroutine: the Reader's PUT must neither block on a
|
||||
Site nor fail with one. Every failure is logged and dropped, leaving the
|
||||
Bookmark intact.
|
||||
- The wire `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
|
||||
bytes exist and `""` before — **never an address that 404s**. Absolute
|
||||
because the userscript renders it on a Site's origin.
|
||||
- `GET /covers/{addr}` is public and uncredentialed by design: no cookie or
|
||||
token of ours may travel to a Site's origin.
|
||||
- A client-sent `cover` is decoded and discarded, permanently — wire
|
||||
compatibility, not an oversight.
|
||||
- **One route serves all six Sites.** No proxy, no per-Site rewrite, no second
|
||||
place that decides a renderable address: the wire `cover` is it. Templates
|
||||
render `.Cover` and nothing else. The old kagane-only serving path
|
||||
(`/img/kagane/{id}` plus a template rewrite) is gone; don't reintroduce a
|
||||
per-Site route because one Site's CDN misbehaves.
|
||||
- The only Site names left in cover code are in `browserOnlyCoverURL`
|
||||
(`internal/latest`): kagane answers a plain fetch with a challenge *and*
|
||||
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
|
||||
one with the same Cloudflare challenge its pages serve;
|
||||
every other Site's CDN answers plain TLS. Templates render `.Cover` — the
|
||||
wire value — never anything else.
|
||||
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
||||
(renders the bindmounted script with the acting Reader's derived credential
|
||||
substituted in — the credential never appears in page markup, the address
|
||||
bar, or a redirect; `?download=1` adds `Content-Disposition: attachment` for
|
||||
mobile Violentmonkey, which ignores a `.user.js` navigation) and
|
||||
`POST /rotate-token` (atomic epoch bump + hash
|
||||
rewrite; invalidates every installed copy, so the panel warns to reinstall
|
||||
on all devices).
|
||||
- **Owner-only admin page (`internal/web/admin.go`, issue #102):** `GET /admin`
|
||||
carries the Reader roster (sessions, Sighting counters, `POST
|
||||
/readers/{id}/revoke` and `POST /readers/{id}/clear-marks`) and Poll Lane
|
||||
status (`GET /ui/admin/lanes`, self-refreshing every 30s). Every route that
|
||||
reaches past the acting Reader is listed in `adminRoutes()` and wrapped in
|
||||
`requireOwner` at registration — add a route there, not a check inside a
|
||||
handler; `web.AdminPatterns()` is what the gate test walks. A non-owner gets
|
||||
404, never 403. Lane figures come from the running poller through the
|
||||
`web.LaneReporter` seam (`latest.Poller.LaneStatus`), never from a table: a
|
||||
nil reporter or a Lane that has not finished a pass renders "no data yet"
|
||||
rather than zeroes. `main.newRouter` takes the reporter as an interface and
|
||||
converts a nil `*Poller` to a nil interface — a typed nil would make the page
|
||||
claim a poller exists.
|
||||
The one owner comparison left outside `requireOwner` is in `index`
|
||||
with the same challenge its pages serve. Every other Site's CDN answers plain
|
||||
TLS.
|
||||
- **comix cover bytes must arrive by direct navigation, not an in-page fetch:**
|
||||
its Series page sets `cross-origin-embedder-policy: require-corp`, which
|
||||
fails a page-context fetch of `static.comix.to`.
|
||||
- With no browser configured, kagane and comix Covers are simply absent;
|
||||
novelfull still gets one whenever its page answers a plain request.
|
||||
|
||||
### `updated_at` drives list order — `Store.Upsert`
|
||||
|
||||
The server applies its own timestamp only when the row is new or
|
||||
`last_chapter_num` changes, else it keeps the stored value. **Favouriting a
|
||||
series, or a newly published chapter arriving, must not reorder the list** —
|
||||
only real reading progress moves a row. Consequently `PUT` returns the row **as
|
||||
stored** and clients must adopt that response rather than their own payload.
|
||||
|
||||
### Lifecycle buckets — `status` on each bookmark
|
||||
|
||||
`reading` | `archived` | `finished`, orthogonal to `favorite`. Archived and
|
||||
finished appear only in their own tab, never in All, Updated, Favourites or the
|
||||
recent strip. The poller keeps checking archived series and skips finished ones.
|
||||
|
||||
- `finished` is settable only from the web UI; `PUT /bookmarks/{key}` rejects
|
||||
it with 400.
|
||||
- **An empty incoming status means "keep the stored one"**, and it is resolved
|
||||
on the `VALUES` side of `Store.Upsert`, not in the conflict clause:
|
||||
`excluded.*` is the post-evaluation row, so a default applied there would
|
||||
wipe the bucket on every PUT from a client predating the column.
|
||||
|
||||
### Config — `Config` / `loadConfig` / `loadLatestPoll` in `backend/main.go`
|
||||
|
||||
That function is the complete list of env vars, their defaults, and which are
|
||||
required. What it can't tell you:
|
||||
|
||||
- `PUBLIC_BASE_URL` must be an absolute origin because every Cover URL on the
|
||||
wire is built from it and the userscript renders on a Site's origin.
|
||||
- `BROWSER_WS_URL` **must be a tailnet IP, never a hostname** — Chrome's
|
||||
DevTools handler 500s `/json/version` for any Host that isn't an IP or
|
||||
`localhost`. Unset (the default) disables browser polling.
|
||||
- `USERSCRIPT_PATH` / `NOVEL_USERSCRIPT_PATH` are bindmounted files; the
|
||||
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
||||
Reader's credential at serve time.
|
||||
- Pace is per Site in the registry, not env. The
|
||||
`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER` knobs are
|
||||
gone on purpose.
|
||||
- The 1h rest for browser Sites is safe on documented grounds: a challenged
|
||||
page costs seconds of a serialized single-tab browser, free-plan zones carry
|
||||
no bot score and no published per-IP rate input, and `cf_clearance` expires
|
||||
in 30 minutes, so every cadence at or above 1h re-solves anyway.
|
||||
|
||||
### Userscript install & rotation — `internal/userscript`, `internal/token`
|
||||
|
||||
Session-gated `GET /install/{manga,novel}-bookmark.user.js` renders the
|
||||
bindmounted script with the acting Reader's derived credential substituted in,
|
||||
so the credential never appears in page markup, the address bar, or a redirect.
|
||||
`?download=1` adds `Content-Disposition: attachment` for mobile Violentmonkey,
|
||||
which ignores a `.user.js` navigation. `POST /rotate-token` is an atomic epoch
|
||||
bump plus hash rewrite and invalidates every installed copy — the panel must
|
||||
keep warning to reinstall on all devices.
|
||||
|
||||
### Owner-only admin — `internal/web/admin.go`
|
||||
|
||||
- **Every route reaching past the acting Reader is listed in `adminRoutes()`
|
||||
and wrapped in `requireOwner` at registration** — add it there, not as a
|
||||
check inside a handler; `web.AdminPatterns()` is what the gate test walks. A
|
||||
non-owner gets 404, never 403.
|
||||
- The one owner comparison left outside the gate is in `index`
|
||||
(`view.Owner = readerID == h.store.OwnerID()`): it gates a link, not an
|
||||
endpoint, so it is a rendering decision a registration-time wrapper cannot
|
||||
express — do not "unify" it into the gate.
|
||||
A Lane pass that returns before computing its figures (refusal backoff,
|
||||
sidecar down) carries the previous pass's due count and gap forward rather
|
||||
than recording zeroes; a Lane that has never reached a pace renders no gap at
|
||||
all. `Checked` next to `Due` is what separates a stopped Lane from a quiet
|
||||
one, so neither figure may be dropped from the row.
|
||||
Due-without-Checked is *not* by itself a stall: a browser Lane under both
|
||||
wake thresholds sets `LaneState.Asleep` at the on-demand gate and renders
|
||||
"browser asleep" instead of "not checking", and never counts toward
|
||||
`Attention`. That is the commonest healthy state for kagane, comix and
|
||||
novelfull — one due Series, nothing checked — so spending the stall mark on
|
||||
it would train the owner to ignore the mark that matters.
|
||||
express. Do not "unify" it into the gate.
|
||||
- Lane figures come through the `web.LaneReporter` seam
|
||||
(`latest.Poller.LaneStatus`), never a table. `main.newRouter` takes the
|
||||
reporter as an interface and converts a nil `*Poller` to a nil interface — a
|
||||
typed nil would make the page claim a poller exists.
|
||||
- A pass that returns before computing figures (refusal backoff, sidecar down)
|
||||
carries the previous pass's numbers forward rather than recording zeroes.
|
||||
- **`Checked` next to `Due` is what separates a stopped Lane from a quiet one**,
|
||||
so neither may be dropped from the row.
|
||||
- Due-without-Checked is **not** by itself a stall: a browser Lane under both
|
||||
wake thresholds sets `LaneState.Asleep` and renders "browser asleep", and
|
||||
never counts toward `Attention`. That is the commonest healthy state for
|
||||
kagane, comix and novelfull, so spending the stall mark on it would train the
|
||||
owner to ignore the mark that matters.
|
||||
|
||||
Reference in New Issue
Block a user