Compare commits

..

9 Commits

Author SHA1 Message Date
sulthan 2be9256e5e chore: refresh the graphify map after the #102 review fixes 2026-08-16 14:31:11 +07:00
sulthan 58014eb8dd review: verdigris patina, honest Lane figures, tests that can fail (#102)
Two-axis review found the shipped colour and three weak tests.

- --patina was a warm gold at the same hue family as --brass; the spec
  asked for a cool blue-green so an unhealthy Lane is unmistakably not
  ember. Now verdigris in both branches, with docs/design-system.md
  stating why the far side of the wheel is the point.
- A Lane pass that returns before computing its figures carries the
  previous pass's due count and gap forward instead of recording zeroes,
  and Checked rides beside Due so a stopped Lane is distinguishable from
  a quiet one.
- TestAdminPageWithoutAPollerSaysSo now separates the two causes it
  conflated, TestOwnerClearsReaderMarks seeds real counters so the
  clearing assertion can fail, and TestLaneStatus asserts Checked and
  the carry-forward.
- backend/AGENTS.md records the one deliberate owner comparison outside
  requireOwner and the carry-forward rule.
2026-08-16 14:31:00 +07:00
sulthan d878580df8 chore: refresh the graphify map for the admin page (#102) 2026-08-16 14:18:08 +07:00
sulthan 0b946ee2c4 test, docs: admin page gate tests and the page's written rules (#102)
- web_test.go walks web.AdminPatterns() rather than naming routes by hand, so a
  new administrative route that forgets requireOwner fails the gate test
  instead of shipping open.
- The harnesses take a LaneReporter; a fake one keeps the page's tests free of
  a poller and a Site.
- backend/AGENTS.md records the adminRoutes/requireOwner rule and the nil-poller
  trap; design-system.md records --patina and the admin page's shape.
2026-08-16 14:17:36 +07:00
sulthan e69da2a99b web: owner-only /admin with the Reader roster and Poll Lane status (#102)
The only operational surface was /healthz and a fold-out roster inside the
owner's own reading page. This gives the owner a page: two sections of facts on
the same measured sheet, no cards.

- admin.go holds every route that reaches past the acting Reader, listed once
  in adminRoutes() and wrapped in requireOwner at registration - a missing gate
  is visible in the route list rather than hidden inside a handler. A non-owner
  gets 404, the same answer revoke already gave.
- Lane figures arrive through the LaneReporter seam, so the page reads the
  running poller rather than a table. newRouter converts a nil *Poller to a nil
  interface: a typed nil would make the page claim a poller exists.
- The roster moves out of the reading page and gains the Sighting counters, the
  blocked verdict and Clear marks. Clearing restores a privilege, so it is a
  plain ghost button; --danger stays with revocation.
- --patina is the page's one accent, held at the weight of the other action
  accents. Neither --ember (new chapter) nor --danger (destruction) is borrowed
  for system health.
2026-08-16 14:17:30 +07:00
sulthan d555f54529 latest: report each Poll Lane's last pass to the owner's page (#102)
A Lane's pace, its refusal backoff and whether its Site needs the browser were
only ever visible in the log. The admin page (issue #102) has to state them, so
the Poller keeps one snapshot per Lane.

- LaneState/Status (status.go) is the page's view of a Lane: due, gap, clamped,
  refusing, browser.
- runOnce records a snapshot on every return path, including the pass that
  refuses, so a cooling Lane does not read as one that never ran.
- Refusing is derived at snapshot read time from the backoff, not stored: a
  Lane that cooled down between passes must report false without a new pass.
- LaneStatus reports only Sites that have completed a pass, so a fresh restart
  renders "no data yet" instead of confident zeroes.
2026-08-16 14:17:22 +07:00
sulthan cdd4eb7c25 store: Sighting counters and the owner's clear control (#102)
The owner's page (issue #102) shows each Reader's Sighting record and offers
one action to wipe it. The counters ship before the Sighting feature that
moves them (issue #103) on purpose: the remedy for a false mark must exist
before marks can be made, or the first broken adapter is fixed with SQL
against production.

- 0010 adds sighting_agreements / sighting_disagreements, both zero-defaulted,
  so every existing Reader reads as trusted.
- ReaderSummary carries both, plus Blocked() against SightingDisagreementLimit,
  so the page states the verdict rather than making the owner derive it.
- ClearReaderMarks zeroes one Reader's pair.
2026-08-16 14:17:10 +07:00
sulthan 3303a55b20 feat: one Poll Lane per Site, replacing the shared pace (#100) (#106)
Closes #100.

Each Site runs its own Poll Lane: an independent goroutine with its own rest
and pace from the registry (`backend/internal/latest/sites.go`), replacing the
shared cooldown/interval/stagger/batch configuration. Rest (1h, all six Sites
including the browser trio) is enforced by the due query's WHERE clause; the
Lane sleeps its effective gap between fetches — the registry 10s, or
rest/eligible when a Site holds enough Series, floored at 1s with a
Site-naming warning when the floor engages.

Lane-local failure handling:
- Two challenge-held results stop that Site's Lane for 15m; the probes keep
  their stamp, untried Series stay due.
- A lost browser sets a shared Poller flag: the other browser Lanes skip
  their passes for the same 15m (no stamp-per-pass-per-Lane on a dead tab),
  then decay and probe again.
- Browser wake gate preserved (5 due, or one waiting 15m, ADR-0005); one tab
  shared by the three browser Sites; "browser lane behind by X" logged every
  pass.
- Cover work (healing a stored source URL and filling a blank from the series
  page) runs in the background so a slow CDN cannot consume a Lane's gap.

Removed: `LATEST_CHAPTER_POLL_{COOLDOWN,BROWSER_COOLDOWN,INTERVAL,BATCH,STAGGER}`
and the 6h browser rest. Only `LATEST_CHAPTER_POLL_ENABLED` remains; DEPLOY.md
documents the exact `.env` edit. ADR-0010 records the decisions.

Reviewed-on: #106
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-16 13:16:19 +07:00
sulthan ddbd57070d Poll comix.to through the browser sidecar (#98) (#105)
Closes #98.

comix.to began answering plain-TLS fetches with a Cloudflare JavaScript
challenge on 2026-08-12, so every poll got a 403 interstitial. Its cover host
`static.comix.to` is gated the same way. comix therefore joins kagane and
novelfull as a browser-backed Site.

## What changed

- **Registry** (`internal/latest/sites.go`): comix gains a `Browser` entry —
  `comixRead`, `Done: body != "" && !isInterstitial(body)`, `Fallback: false`.
  Skip-when-no-browser falls out of the existing routing; no site-string compare
  was added anywhere.
- **Read shape** (`internal/latest/browser.go`): an in-tab `fetch()` of the
  Series URL, not a DOM render. comix is an SPA — rendering it costs ~65
  requests for the same server-rendered HTML one fetch returns (24.5 KB,
  ~480 ms measured). `comixSeriesPageURL` pins scheme + host + `/title/<slug>`
  and rebuilds the address, so a client-supplied `series_url` cannot aim the
  browser anywhere else.
- **Cover bytes**: `comixImageURLRe` pins `https://static.comix.to/<path>.<ext>`;
  `BrowserFetcher.Image` now gates on `browserOnlyCoverURL` rather than a
  kagane-only regex, so both Sites' image URLs route through the one path.
  Bytes come from direct navigation, not a page-context fetch — comix's Series
  page sets `cross-origin-embedder-policy: require-corp`, which fails one.
- **Parsers and stored Series identity: untouched.** The in-tab body is the same
  server-rendered HTML the existing fixtures were cut from.

## Verification

- `go test ./...` green (needs Docker).
- New seam tests: comix routes to the browser when one is configured, and is
  not fetched at all when none is (`TestComixUsesBrowserFetcher`,
  `TestComixSkippedWhenNoBrowserFetcher`); URL-pin and cover-gate table tests.
- Live proof against the real browser unit, `TestSmokeComix` (env-gated):
  page 24793 bytes in one in-tab fetch, chapter 53, cover accepted by the pin,
  26862 bytes of `image/jpg` retrieved.
- Two-axis review run; findings were stale comments on `BrowserFetcher`, `Get`
  and the `Fallback` field, fixed in f000cc7.

Docs updated: root `AGENTS.md` (constraint + smoke command, including the note
that this dev machine's ISP DNS-hijacks `comix.to`), `backend/AGENTS.md`
(poller, cover pipeline, `BROWSER_WS_URL`), `REDEPLOY.md` §8 degrade note.

Reviewed-on: #105
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-16 12:13:59 +07:00
36 changed files with 7060 additions and 3210 deletions
+5 -18
View File
@@ -73,26 +73,13 @@ DISCORD_REDIRECT_URI=
# The backend re-checks each bookmarked series' newest published chapter on its
# own schedule, so latest_chapter stays fresh even when you never open the manga
# sites. This runs in parallel with the userscript's own in-browser check.
# Set to 0 to turn it off entirely.
# Set to 0 to turn it off entirely. Pace is per Site (one Poll Lane per Site,
# issue #100) and lives in the backend registry, not here — there is nothing
# else to configure.
# LATEST_CHAPTER_POLL_ENABLED=1
#
# Two independent clocks. COOLDOWN is how long a plain-TLS series rests between
# checks; BROWSER_COOLDOWN is the longer rest for kagane and novelfull. INTERVAL
# is how often the poller wakes up and looks for series past their cooldowns.
# Shortening INTERVAL cannot shorten either cooldown.
LATEST_CHAPTER_POLL_COOLDOWN=1h # plain-TLS per series, floor 15m
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN=6h # browser-backed per series, floor 15m
LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
LATEST_CHAPTER_POLL_BATCH=14 # series per wake
LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
#
# Uses a ticker, not an immediate first run: the first poll happens one
# INTERVAL after startup, not at startup. A container restarting more often
# than INTERVAL never polls.
#
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
# Every Site rests an hour between checks and gaps ten seconds between fetches;
# a Site with many Series tightens its own gap. See backend/internal/latest/sites.go.
# CDP endpoint of the browser, used for the two sites behind a Cloudflare
# JavaScript challenge (kagane, novelfull) and by the web UI's kagane cover
+1 -1
View File
@@ -86,7 +86,7 @@ _Avoid_: client report, user poll, observation, claim
**Acquisition**:
The single read of a Series page made the moment the Series first exists, giving it
both its Latest Chapter and its Cover without waiting out the Poll queue. Distinct
both its Latest Chapter and its Cover without waiting for the Lane's pace. Distinct
from a Poll in the two ways that matter: a Reader is present — it is triggered by
their first Bookmark of that Series — and it is the only read that establishes a
Cover rather than refreshing facts. It happens once in a Series's life; every later
+9 -1
View File
@@ -93,6 +93,14 @@ the backend dials but not the password the database expects, and `bookmark-api`
crash-loops on `password authentication failed`. Set it before §2 and leave it
alone.
An `.env` written before issue #100 carries the old poll-pace names
(`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
`_STAGGER`). All five are dead configuration now — the pace lives in the Site
registry (`backend/internal/latest/sites.go`), so **delete those lines** and
keep only the kill switch `LATEST_CHAPTER_POLL_ENABLED`. Leaving them behind
is harmless (nothing reads them) but silently misleads the next person who
edits the file.
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
> names (check your Traefik static config — common alternatives: `https`,
> `myresolver`, `cloudflare`). Wrong names = no certificate issued.
@@ -505,7 +513,7 @@ picks up a restarted Chrome's new debugger UUID by itself.
| kagane rows never get a `latest_chapter`; log says `browser fetcher disabled` or nothing at all | `BROWSER_WS_URL` unset. Expected before §7 is done. |
| kagane polls all fail; log shows a 500 from `/json/version` | `BROWSER_WS_URL` names a MagicDNS hostname (or any name). Chrome's DevTools handler only accepts an IP or `localhost` — use the tailnet IP. |
| kagane polls fail with a connection error | Home machine off, off the tailnet, or the unit is down. `tailscale ping <machine>`, then `docker compose ps` in its `chrome/`. Costs freshness only; stored covers keep serving. |
| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series (up to `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, default 6h). |
| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series. |
| `compose` in `chrome/` errors `set BROWSER_BIND_ADDR to this machine's tailnet IP` | No `chrome/.env`, or the variable is empty. Deliberate — it has no default so an unset value cannot publish CDP to the LAN. |
| browser container restarts, or is OOM-killed | `docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'`. The 512 MiB cap is sized against a measured 645 MiB untuned peak; a real breach is a Chrome regression worth reading `docker logs` for, not a number to raise reflexively. |
+1 -6
View File
@@ -53,12 +53,7 @@ covers are stored, so the library renders in full with the browser switched off.
| `DISCORD_API_BASE` | `https://discord.com/api/v10` | Test seam — tests point it at a local stub so the real token exchange runs. |
| `USERSCRIPT_PATH` | `/userscript/manga-bookmark.user.js` | Bindmounted file served at `/u/{token}/manga-bookmark.user.js`. |
| `NOVEL_USERSCRIPT_PATH` | `/userscript/novel-bookmark.user.js` | Same, for the novel library. |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. |
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Rest between checks of one plain-TLS series; floor `15m`. |
| `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN` | `6h` | Rest between checks of one browser-backed series; floor `15m`. |
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often the poller wakes. Cannot shorten either cooldown. |
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake. Keep `BATCH × STAGGER` under `INTERVAL`. |
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch — this is the outbound request rate. |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. Pace is per Site in the registry — one Poll Lane per Site, each with its own rest and gap (issue #100) — so no other knobs exist. |
Compose reads a few more from the same `.env` that the backend never sees:
`POSTGRES_PASSWORD` (required — `DATABASE_URL` is built from it, and Postgres
+53 -19
View File
@@ -74,20 +74,33 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
`--ember` stay reserved for new-chapter signal: busy bar and inline
error use `--mute`.
- **Latest-chapter poller:** ticker goroutine in same binary re-check
each bookmarked series' newest published chapter from backend's own
network access, so `latest_chapter` stay fresh when user not
browsing. Second, parallel signal — userscript keep own
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100),
each re-checking that Site's bookmarked series' newest published chapter from
backend's own network access, so `latest_chapter` stays fresh when the user
isn't browsing. Second, parallel signal — the userscript keeps its own
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
Two independent clocks: per-series cooldown (`series.latest_checked_at`,
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
Two independent clocks: per-series rest (`series.latest_checked_at`,
enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and
per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
in the Site registry (`internal/latest/sites.go`), not config: the five env
knobs that used to size a shared pace are gone.
The poller walks **Series, not Bookmarks** — a series referenced by several
bookmarks is fetched once per cycle, and the due queue orders
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
*before* fetch so broken series wait out full cooldown instead of retrying
*before* fetch so broken series wait out the rest instead of retrying
every tick; found chapter written straight to the series row via
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
order — is never touched.
Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
`errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
that makes the other browser Lanes skip their passes for the same 15m, so a
restarting Chrome doesn't stamp one Series per Lane per pass — after the
window the flag decays and they probe again. Browser Lanes wake Chrome only
when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
and cover work (both healing a stored source URL and filling a blank from
the series page) runs in the background so a slow CDN can't consume a
Lane's gap.
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
against fingerprint-based blocking; any failure log and skip. kagane, comix
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
@@ -152,15 +165,17 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/
`_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on,
`1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both
cooldowns have a `15m` floor). The browser cooldown is longer for cost, not
for safety: a challenged page costs seconds of a serialized single-tab
browser, while a plain read costs one request. It buys no documented
reduction in challenge risk — free-plan zones have no bot score and no
published per-IP rate input, and `cf_clearance` expires in 30 minutes so
every cadence at or above 1h re-solves anyway —
`LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill
switch, default on). Pace is per Site in the registry (issue #100): every
Site rests an hour and gaps ten seconds, a Site with more eligible Series
than 360 tightens its own gap toward the 1s floor, and browser Lanes wake
Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/
`_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are
gone. The 1h rest for browser Sites is safe on documented grounds: a
challenged page costs seconds of a serialized single-tab browser, free-plan
zones have no bot score and no published per-IP rate input, and
`cf_clearance` expires in 30 minutes so every cadence at or above 1h
re-solves anyway —
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
@@ -196,6 +211,25 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
`POST /rotate-token` (atomic epoch bump + hash
rewrite; invalidates every installed copy, so the panel warns to reinstall
on all devices).
Owner-only `POST /readers/{id}/revoke` (drops one Reader's session rows and
re-renders the `readers` panel; 404 for any non-owner) is the only route that
reaches across Readers.
- **Owner-only admin page (`internal/web/admin.go`, issue #102):** `GET /admin`
carries the Reader roster (sessions, Sighting counters, `POST
/readers/{id}/revoke` and `POST /readers/{id}/clear-marks`) and Poll Lane
status (`GET /ui/admin/lanes`, self-refreshing every 30s). Every route that
reaches past the acting Reader is listed in `adminRoutes()` and wrapped in
`requireOwner` at registration — add a route there, not a check inside a
handler; `web.AdminPatterns()` is what the gate test walks. A non-owner gets
404, never 403. Lane figures come from the running poller through the
`web.LaneReporter` seam (`latest.Poller.LaneStatus`), never from a table: a
nil reporter or a Lane that has not finished a pass renders "no data yet"
rather than zeroes. `main.newRouter` takes the reporter as an interface and
converts a nil `*Poller` to a nil interface — a typed nil would make the page
claim a poller exists.
The one owner comparison left outside `requireOwner` is in `index`
(`view.Owner = readerID == h.store.OwnerID()`): it gates a link, not an
endpoint, so it is a rendering decision a registration-time wrapper cannot
express — do not "unify" it into the gate.
A Lane pass that returns before computing its figures (refusal backoff,
sidecar down) carries the previous pass's due count and gap forward rather
than recording zeroes; a Lane that has never reached a pace renders no gap at
all. `Checked` next to `Due` is what separates a stopped Lane from a quiet
one, so neither figure may be dropped from the row.
+4 -4
View File
@@ -48,7 +48,7 @@ func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
func newTestServer(t *testing.T) http.Handler {
t.Helper()
return newRouter(newTestStore(t), testConfig())
return newRouter(newTestStore(t), testConfig(), nil)
}
func newTestStore(t *testing.T) *store.Store {
@@ -599,7 +599,7 @@ func TestLoadConfigDiscord(t *testing.T) {
// cooldown and the poller would re-fetch that series on every single tick.
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
s := newTestStore(t)
srv := newRouter(s, testConfig())
srv := newRouter(s, testConfig(), nil)
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
@@ -637,7 +637,7 @@ func TestUserscriptServedWithWebUIDisabled(t *testing.T) {
rr := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/u/"+ownerCredential()+"/manga-bookmark.user.js", nil)
newRouter(s, cfg).ServeHTTP(rr, req)
newRouter(s, cfg, nil).ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rr.Code)
}
@@ -659,7 +659,7 @@ func TestNovelUserscriptServed(t *testing.T) {
cfg := testConfig()
cfg.NovelUserscriptPath = novelPath
srv := newRouter(s, cfg)
srv := newRouter(s, cfg, nil)
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, httptest.NewRequest(http.MethodGet,
+1 -1
View File
@@ -98,7 +98,7 @@ func TestPublicCoverNeverEchoesNonImage(t *testing.T) {
if _, err := db.Exec(`UPDATE covers SET content_type = 'text/html' WHERE address = $1`, address); err != nil {
t.Fatalf("poison row: %v", err)
}
rr := getCover(t, newRouter(st, testConfig()), "/covers/"+address, nil)
rr := getCover(t, newRouter(st, testConfig(), nil), "/covers/"+address, nil)
if rr.Code == http.StatusOK {
t.Fatalf("status = 200, want a refusal for a non-image row (body %q)", rr.Body.String())
}
+1 -1
View File
@@ -120,7 +120,7 @@ func (a *Acquirer) acquire(ctx context.Context, sr store.Series) {
// Stamped after success — the reverse of the poller, which stamps before
// the fetch: the Reader is here, watching the Series they just created, so
// a failed acquisition must leave the row due for a fast retry rather than
// consuming the cooldown. The stamp happens even when the page read
// consuming the rest. The stamp happens even when the page read
// succeeded but produced no facts to persist.
if err := a.Store.MarkLatestChecked(sr.Site, sr.SeriesID, time.Now().UnixMilli()); err != nil {
log.Printf("acquire %q: mark checked: %v", sr.Key(), err)
+2 -2
View File
@@ -18,7 +18,7 @@ import (
// challengeTimeout bounds one navigate-and-solve. A Cloudflare managed
// challenge clears in a few seconds when it clears at all; anything longer is a
// challenge that is not going to pass, and the caller's cooldown was already
// challenge that is not going to pass, and the caller's rest was already
// stamped before this ran.
const challengeTimeout = 45 * time.Second
@@ -213,7 +213,7 @@ func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, st
// errChallengeHeld reports that the budget ran out with the interstitial still
// up. Distinct from a transport failure: it means "this site said no", which
// the poller answers with a 403 and its ordinary cooldown.
// the poller answers with a refusal backoff for that Site's Lane (issue #100).
var errChallengeHeld = errors.New("challenge held")
// errBrowserInterrupted distinguishes a remote Chrome restart from the
+297 -72
View File
@@ -5,6 +5,8 @@ import (
"errors"
"log"
"net/url"
"sort"
"sync"
"time"
"bookmarkmanager/backend/internal/store"
@@ -28,15 +30,11 @@ type BrowserCoverFetcher interface {
// in parallel and report the same observable fact, so whichever writes last wins
// and neither needs to know about the other.
//
// Two clocks, deliberately independent:
//
// - Interval is how often this goroutine wakes up and looks.
// - Cooldowns are how long a series rests since its own last check. Browser-
// backed sites use the longer BrowserCooldown.
//
// Cooldowns are enforced by the WHERE clause in DueForLatestCheck rather than
// by any timer. Shortening Interval therefore cannot shorten anyone's cooldown;
// it only makes the poller wake up and find nothing due more often.
// Every Site gets its own Poll Lane: one independent stream of Polls with its
// own pace, running concurrently with every other Site's (issue #100). Rest
// time and gap live in the Site registry, not here — see sites.go. Rest is
// enforced by the WHERE clause in DueForLatestCheck rather than by any timer;
// the gap is enforced by the Lane sleeping between fetches.
type Poller struct {
Store *store.Store
Fetch Fetcher
@@ -50,11 +48,23 @@ type Poller struct {
// same failure-isolated prefetch path.
CoverBytesFetch CoverBytesFetcher
Now func() time.Time // injected so tests can freeze it
Cooldown time.Duration
BrowserCooldown time.Duration
Interval time.Duration
Stagger time.Duration
Batch int
// refuseUntil gates a Site's Lane after it refused twice in one run: no
// Series of that Site is attempted again before this time (issue #100).
// browserDownAt is when a browser Lane last lost the sidecar; the other
// browser Lanes skip their passes for the next refuseBackoff, so a
// restarting Chrome does not stamp one Series per pass per Lane (story 20).
mu sync.Mutex
refuseUntil map[string]time.Time
browserDownAt time.Time
// laneStates is the owner's page snapshot of each Lane's last pass
// (issue #102), keyed by Site. Guarded by mu; a Site appears only after
// its first pass, so a restart renders "no data yet" rather than zeroes.
laneStates map[string]LaneState
// coverWG tracks in-flight cover work. Covers heal in the background so a
// slow cover host cannot delay the next Series-page Poll; tests join it
// before asserting on cover fetches.
coverWG sync.WaitGroup
}
// fillBlankCover gives a Series its Cover when it has none. The blank state is
@@ -76,7 +86,14 @@ func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, cover stri
if cover == "" {
return
}
p.storeCover(ctx, sr, cover)
// Like healCover, the fill runs in the background: a large import of
// blanks would otherwise pay one og:image fetch per Series against the
// Lane's gap (issue #100, story 12).
p.coverWG.Add(1)
go func() {
defer p.coverWG.Done()
p.storeCover(ctx, sr, cover)
}()
}
// prefetchCover heals Series that already carry a third-party source URL but
@@ -143,72 +160,258 @@ func fetcherFor(site string, browser, tls Fetcher) Fetcher {
return nil
}
// Run polls until ctx is cancelled.
//
// runOnce is called synchronously, so a batch that overruns the tick delays the
// next one instead of stacking a second batch on top of it. That is the intended
// failure mode for a misconfigured batch x stagger: a slower cadence, never
// concurrent fetch storms.
// Run polls until ctx is cancelled: one goroutine per Site Lane, each pacing
// itself by the Site's effective gap. Lanes share nothing but the store and
// the browser fetcher's single tab (BrowserFetcher serializes itself), so one
// hostile Site burns only its own budget.
// laneNames returns every registry Site in the deterministic order both Run
// and runOnce iterate: sorted, so lane behaviour and its tests agree on who
// runs first.
func laneNames() []string {
names := make([]string, 0, len(sites))
for name := range sites {
names = append(names, name)
}
sort.Strings(names)
return names
}
func (p *Poller) Run(ctx context.Context) {
log.Printf("latest-chapter poller: interval=%s cooldown=%s browser-cooldown=%s batch=%d stagger=%s",
p.Interval, p.Cooldown, p.BrowserCooldown, p.Batch, p.Stagger)
t := time.NewTicker(p.Interval)
defer t.Stop()
names := laneNames()
log.Printf("latest-chapter poller: %d lanes, rest=%s gap=%s", len(names), defaultRest, defaultGap)
for _, name := range names {
go p.lane(ctx, name)
}
<-ctx.Done()
log.Println("latest-chapter poller: stopped")
}
// lane is one Site's Poll Lane: one pass, then sleep the pace the pass
// reported, then another pass, until ctx is cancelled. The sleep is the whole
// pace discipline — a pass that fetched nothing still reports its gap so the
// Lane wakes often enough to notice Series as they become due. The pass shares
// the Poller's browser-down state, so a sidecar loss is noticed once and the
// other browser Lanes skip passes until the backoff window decays.
func (p *Poller) lane(ctx context.Context, name string) {
for {
pace := p.runLanePass(ctx, name, true)
if ctx.Err() != nil {
return
}
select {
case <-ctx.Done():
log.Println("latest-chapter poller: stopped")
return
case <-t.C:
p.runOnce(ctx)
case <-time.After(pace):
}
}
}
// runOnce processes one batch of due series.
// runOnce processes one round: one pass of every Lane, back to back, no real
// time passing. This is the deterministic entry point the test suite drives a
// round at a time. The production Run loop does the same work paced by its own
// sleeps; pacing is the only difference.
func (p *Poller) runOnce(ctx context.Context) {
for _, name := range laneNames() {
p.runLanePass(ctx, name, false)
}
}
// runLanePass processes one pass of one Site's Lane: select the due Series,
// pace through them, and report how long the Lane should wait before its next
// pass. paced spaces consecutive fetches by the Site's effective gap — the
// production Lane's rate limit; the deterministic test entry runs back to back.
func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration {
now := p.Now()
cutoff := now.Add(-p.Cooldown).UnixMilli()
browserCutoff := now.Add(-p.BrowserCooldown).UnixMilli()
due, err := p.Store.DueForLatestCheck(cutoff, browserCutoff, browserBackedSites(), p.Batch)
if err != nil {
log.Printf("latest poll: due query: %v", err)
return
// Snapshot this pass for the owner's page (issue #102). Recorded on every
// return path, with the figures filled in where the pass computes them.
st := LaneState{Site: name, LastRun: now, Browser: isBrowserSite(name)}
defer func() { p.recordLaneState(st) }()
if until := p.refusalBackoff(name); now.Before(until) {
// Cooling down after a refusal: do not attempt this Site at all.
return until.Sub(now)
}
if isBrowserSite(name) {
if downFor, down := p.browserDownFor(now); down && downFor < refuseBackoff {
// A sibling browser Lane lost the sidecar within the backoff
// window: skip this pass, so a restarting Chrome does not stamp
// this Site's Series one pass at a time. After refuseBackoff the
// flag decays and the Lane probes again (issue #100, story 20).
log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor)
return refuseBackoff - downFor
}
}
s := sites[name]
f := fetcherFor(name, p.BrowserFetch, p.Fetch)
if f == nil {
// No fetcher at all right now (browser absent, no fallback): every
// Series stays unstamped and due, so a browser that appears after a
// restart finds its full queue waiting (issue #100).
st.Gap = defaultGap
return defaultGap
}
checked := 0
due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli())
if err != nil {
log.Printf("latest poll %s: due query: %v", name, err)
st.Gap = defaultGap
return defaultGap
}
st.Due = len(due)
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) {
// Below both thresholds Chrome stays asleep (ADR-0005 on-demand
// browser): waking it for a single Poll would cost a challenge solve
// per request. The Lane still paces at the default gap, which is what
// the owner's page must show rather than a zero.
st.Gap = defaultGap
return defaultGap
}
if s.Browser != nil {
// Browser Lanes share one tab, so their combined ceiling is about 360
// Polls an hour. When they cannot keep up, the wait past the rest time
// grows — log by how much, every pass, so the decision to give them
// more pages is made from a measurement rather than a guess.
if behind := maxSeriesWait(due, now, s.Rest) - s.Rest; behind > 0 {
log.Printf("latest poll %s: browser lane behind by %s (browser Sites cannot keep up with the hour)", name, behind)
}
}
eligible, err := p.Store.EligibleSeriesCount(name)
if err != nil {
log.Printf("latest poll %s: eligible count: %v", name, err)
st.Gap = defaultGap
return defaultGap
}
gap, clamped := effectiveGap(s, eligible)
st.Gap, st.Clamped = gap, clamped
if clamped {
log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible)
}
if eligible == 0 {
// Nothing to poll for the foreseeable future; sleep a full rest instead
// of re-querying every gap.
return s.Rest
}
refusals := 0
for i, sr := range due {
if ctx.Err() != nil {
break
}
// Staggered rather than fired together: a burst of simultaneous requests
// from one server IP is the traffic shape most likely to move that IP's
// bot score. This is the server-side analogue of the userscript's "one
// series per navigation ... indistinguishable from browsing" (L455-456).
stopped := false
if i > 0 && p.Stagger > 0 {
select {
case <-ctx.Done():
stopped = true
case <-time.After(p.Stagger):
}
}
if stopped {
if refusals >= 2 {
// This Site refused twice in a row: the remaining Series are left
// unstamped and due, and the Lane waits refuseBackoff before
// trying it again.
break
}
p.checkOne(ctx, sr)
checked++
if paced && i > 0 {
select {
case <-ctx.Done():
break
case <-time.After(gap):
}
if ctx.Err() != nil {
break
}
}
if err := p.checkOne(ctx, sr); err != nil {
switch {
case errors.Is(err, errChallengeHeld):
refusals++
case errors.Is(err, errBrowserInterrupted):
p.setBrowserDown(now)
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, refuseBackoff)
return gap
default:
refusals = 0
}
} else {
refusals = 0
}
st.Checked++
}
// due vs checked is how you tell which constraint is binding: ticks that
// report due=0 mean the cooldown is the limit, ticks that report due==batch
// every time mean throughput is.
log.Printf("latest poll: due=%d checked=%d", len(due), checked)
if st.Checked > 0 {
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), st.Checked)
}
if refusals >= 2 {
p.setRefusalBackoff(name, now.Add(refuseBackoff))
log.Printf("latest poll %s: refused twice this run, waiting %s", name, refuseBackoff)
return refuseBackoff
}
return gap
}
func (p *Poller) refusalBackoff(name string) time.Time {
p.mu.Lock()
defer p.mu.Unlock()
return p.refuseUntil[name]
}
func (p *Poller) setRefusalBackoff(name string, until time.Time) {
p.mu.Lock()
defer p.mu.Unlock()
if p.refuseUntil == nil {
p.refuseUntil = make(map[string]time.Time)
}
p.refuseUntil[name] = until
}
// setBrowserDown records when a browser Lane lost the sidecar. It is Poller
// state rather than pass state so the other browser Lanes see it too.
func (p *Poller) setBrowserDown(now time.Time) {
p.mu.Lock()
p.browserDownAt = now
p.mu.Unlock()
}
// browserDownFor reports how long the sidecar has been down and that it is
// down at all — the zero time means never down, which must not read as a
// zero-duration loss. The window decays: once refuseBackoff passes without a
// fresh loss, Lanes probe again.
func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) {
p.mu.Lock()
defer p.mu.Unlock()
if p.browserDownAt.IsZero() {
return 0, false
}
return now.Sub(p.browserDownAt), true
}
// isBrowserSite reports whether the registry routes this Site's page through
// the browser sidecar.
func isBrowserSite(name string) bool {
return sites[name].Browser != nil
}
// browserWakeDue reports whether a browser Lane may start a run: five or more
// of its Series are due, or any one of them has been due for browserWakeAge.
// Below both thresholds the Lane leaves Chrome asleep — Series Polled together
// become due together, so the group naturally stays clustered, and the age
// rule exists to stop a Series that drifted out of the group from starving.
func browserWakeDue(due []store.Series, now time.Time, rest time.Duration) bool {
if len(due) >= browserWakeCount {
return true
}
return maxSeriesWait(due, now, rest) >= browserWakeAge
}
// maxSeriesWait returns how long the most-overdue of the due Series has been
// waiting past its due moment (0 when due is empty).
func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration {
var oldest time.Duration
for _, sr := range due {
if w := now.Sub(time.UnixMilli(sr.LatestCheckedAt).Add(rest)); w > oldest {
oldest = w
}
}
return oldest
}
// checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a
// batch or take down the process.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
// Lane or take down the process. The returned error is the page read's
// classified outcome so the Lane can tell a refusal from a loss of the
// browser; non-classified failures still return nil-equivalent behaviour.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
defer func() {
if r := recover(); r != nil {
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
@@ -216,44 +419,46 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
}()
// Stamped before the fetch, not after, so an error, a timeout, or a shutdown
// mid-request still consumes the cooldown. Otherwise a renamed or deleted
// series would be retried on every single tick forever. The userscript
// stamps in the same order and for the same reason (L471-473).
// mid-request still consumes the rest. Otherwise a renamed or deleted
// series would be retried on every single pass forever. The userscript
// stamps in the same order and for the same reason (L471-473). A Series
// never reaches checkOne without a fetcher — runLanePass skips those — so
// the stamp means "attempted", and an untried Series stays due.
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
return
return nil
}
facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch)
if err != nil {
switch {
case errors.Is(err, errNotFetchable):
// The cooldown above is already consumed, so a row that never
// passes the gate is retried at cooldown pace rather than
// The rest above is already consumed, so a row that never
// passes the gate is retried at rest pace rather than
// hot-looping.
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
return
return err
case errors.Is(err, errNoFetcher):
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
return
return err
}
// A legacy cover heals independently of the page read: its source may
// answer — a CDN — while the origin does not, so a fetch failure does
// not skip the heal, matching the order the shared read replaced.
p.prefetchCover(ctx, sr)
p.healCover(ctx, sr)
log.Printf("latest poll %q: %v", sr.Key(), err)
return
return err
}
// A legacy cover source is healed independently of the page read.
p.prefetchCover(ctx, sr)
p.healCover(ctx, sr)
// Cover fill is independent of the chapter signal: a page that lost its
// chapter list may keep its og:image, and a blank Series heals either way.
p.fillBlankCover(ctx, sr, facts.Cover)
if !facts.HasLatest {
// Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a cooldown instead of hot-looping.
// already stamped, so this waits out a rest instead of hot-looping.
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen)
return
return nil
}
// Equality, not >, mirroring the userscript (L427): a site that retracts a
@@ -261,7 +466,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
// against the due-query snapshot; a concurrent write in between only costs
// one redundant UPDATE of the same absolute value, never a wrong one.
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num {
return
return nil
}
// Series-level write: the row is shared, so one update refreshes every
@@ -270,9 +475,29 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
// the list.
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil {
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
return
return nil
}
log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label)
return nil
}
// healCover runs prefetchCover in the background. Cover bytes come from a
// different host — often a CDN — and heal once in a Series's life, so they
// must not consume a Lane's gap: a large import with many blanks would
// otherwise make every Latest Chapter go stale behind a slow image host
// (issue #100).
func (p *Poller) healCover(ctx context.Context, sr store.Series) {
p.coverWG.Add(1)
go func() {
defer p.coverWG.Done()
p.prefetchCover(ctx, sr)
}()
}
// waitCovers blocks until every in-flight cover heal finishes. Tests call it
// after a round before asserting on cover fetches.
func (p *Poller) waitCovers() {
p.coverWG.Wait()
}
// fetchableSeriesURL reports whether site is a Site the registry knows and
+493 -114
View File
@@ -5,6 +5,7 @@ import (
"crypto/sha256"
"database/sql"
"errors"
"fmt"
"log"
"os"
"strings"
@@ -156,19 +157,16 @@ func (f *fakeBytesCoverFetcher) callCount() int {
return len(f.calls)
}
// newTestPoller wires a poller with a frozen clock and no stagger, so tests run
// instantly and deterministically.
// newTestPoller wires a poller with a frozen clock. Rest and gap come from the
// Site registry, so tests seed checked_at relative to the one-hour rest; the
// round entry point (runOnce) runs every Lane back to back with no real
// pacing, so tests stay instant and deterministic.
func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Poller {
t.Helper()
return &Poller{
Store: s,
Fetch: f,
Now: func() time.Time { return at },
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
Interval: 10 * time.Minute,
Stagger: 0,
Batch: 14,
Store: s,
Fetch: f,
Now: func() time.Time { return at },
}
}
@@ -208,9 +206,10 @@ func TestRunOncePrefetchesPublicCover(t *testing.T) {
covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverBytesFetch: covers,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if got := covers.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1", got)
@@ -241,9 +240,10 @@ func TestRunOnceDoesNotStoreNonImagePublicCover(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("challenge"), contentType: "text/html"},
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.GetCover(coverURL); err != nil || found {
t.Fatalf("non-image cover = found %v, err %v; want missing", found, err)
@@ -350,23 +350,6 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
}
}
func TestRunOnceRespectsBatchLimit(t *testing.T) {
s, _ := newTestStore(t)
for i := 0; i < 20; i++ {
key := "asura:s" + string(rune('a'+i))
seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0)
}
f := &fakeFetcher{body: "", status: 200}
p := newTestPoller(t, s, f, time.UnixMilli(5_000_000))
p.Batch = 5
p.runOnce(context.Background())
if got := f.callCount(); got != 5 {
t.Fatalf("fetched %d series, want 5 (batch limit)", got)
}
}
// The point of the split (ADR-0003): a series referenced by several bookmarks
// is fetched once per due cycle, not once per bookmark. Two bookmarks share a
// series when two readers track it (issue #22).
@@ -409,8 +392,8 @@ func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
}
}
// One unreachable series must not abandon the rest of the batch.
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
// One unreachable series must not abandon the rest of the Lane.
func TestRunOnceOneBadSeriesDoesNotStallLane(t *testing.T) {
s, _ := newTestStore(t)
keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"}
for _, k := range keys {
@@ -436,7 +419,9 @@ func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
}
}
func TestRunLogsCooldowns(t *testing.T) {
// Pace now lives in the registry, not config: Run logs the lane defaults so a
// deployment can see what the poller is doing without reading the source.
func TestRunLogsLaneDefaults(t *testing.T) {
var logs strings.Builder
previous := log.Writer()
log.SetOutput(&logs)
@@ -444,21 +429,19 @@ func TestRunLogsCooldowns(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
(&Poller{
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
Interval: time.Hour,
}).Run(ctx)
(&Poller{Now: func() time.Time { return time.Now() }}).Run(ctx)
if got := logs.String(); !strings.Contains(got, "cooldown=1h") ||
!strings.Contains(got, "browser-cooldown=6h") {
t.Fatalf("startup log = %q, want both cooldowns", got)
got := logs.String()
for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} {
if !strings.Contains(got, want) {
t.Fatalf("startup log = %q, want %q", got, want)
}
}
}
// The cooldown is enforced by the due query, so a second immediate pass must do
// nothing at all — this is what makes the tick interval independent of it.
func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
// The rest is enforced by the due query, so a second immediate pass must do
// nothing at all — this is what makes the Lane's sleep independent of it.
func TestRunOnceHonoursRestAcrossPasses(t *testing.T) {
s, _ := newTestStore(t)
const url = "https://asurascans.com/comics/x"
seedForCheck(t, s, "asura:x", url, 0)
@@ -471,56 +454,18 @@ func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
if got := f.callCount(); got != 1 {
t.Fatalf("first pass fetched %d, want 1", got)
}
// Same instant, and again 59 minutes later: both inside the 1h cooldown.
// Same instant, and again 59 minutes later: both inside the 1h rest.
p.runOnce(context.Background())
p.Now = func() time.Time { return now.Add(59 * time.Minute) }
p.runOnce(context.Background())
if got := f.callCount(); got != 1 {
t.Fatalf("fetched %d times inside the cooldown, want 1", got)
t.Fatalf("fetched %d times inside the rest, want 1", got)
}
// Past the cooldown, it is due again.
// Past the rest, it is due again.
p.Now = func() time.Time { return now.Add(61 * time.Minute) }
p.runOnce(context.Background())
if got := f.callCount(); got != 2 {
t.Fatalf("fetched %d times after the cooldown, want 2", got)
}
}
func TestRunOnceUsesBrowserCooldown(t *testing.T) {
s, _ := newTestStore(t)
const browserKey = "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
seedForCheck(t, s, "asura:plain", "https://asurascans.com/comics/plain", 0)
seedForCheck(t, s, browserKey, "https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", 0)
hour := time.Hour
now := time.Unix(2*int64(hour/time.Second), 0)
tls := &fakeFetcher{status: 200}
browser := &fakeFetcher{status: 200}
p := &Poller{
Store: s,
Fetch: tls,
BrowserFetch: browser,
Now: func() time.Time { return now },
Cooldown: hour,
BrowserCooldown: 6 * hour,
Batch: 10,
}
p.runOnce(context.Background())
if got := tls.callCount(); got != 1 {
t.Fatalf("plain-TLS fetches after 2h = %d, want 1", got)
}
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches after 2h = %d, want 0", got)
}
now = time.Unix(7*int64(hour/time.Second), 0)
p.runOnce(context.Background())
if got := tls.callCount(); got != 2 {
t.Fatalf("plain-TLS fetches after 7h = %d, want 2", got)
}
if got := browser.callCount(); got != 1 {
t.Fatalf("browser fetches after 7h = %d, want 1", got)
t.Fatalf("fetched %d times after the rest, want 2", got)
}
}
@@ -665,17 +610,18 @@ func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
f := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := &Poller{
Store: s,
Fetch: f,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
Store: s,
Fetch: f,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
if len(f.calls) != 0 {
t.Errorf("TLS fetcher was called for kagane: %v", f.calls)
}
if got := readLatestCheckedAt(t, s, "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"); got != 0 {
t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got)
}
}
// novelfull without a browser is not skipped outright: its challenge is a
@@ -699,11 +645,10 @@ func TestNovelfullUsesTLSWhenNoBrowserFetcher(t *testing.T) {
covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"}
p := &Poller{
Store: s, Fetch: tlsF, CoverBytesFetch: covers,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if len(tlsF.calls) != 1 {
t.Fatalf("TLS fetcher calls = %d, want 1", len(tlsF.calls))
@@ -741,9 +686,7 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
browserF := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := &Poller{
Store: s, Fetch: tlsF, BrowserFetch: browserF,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
@@ -789,11 +732,10 @@ func TestComixUsesBrowserFetcher(t *testing.T) {
p := &Poller{
Store: s, Fetch: tlsF, BrowserFetch: browserF,
CoverFetch: covers, CoverBytesFetch: tlsCovers,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if len(tlsF.calls) != 0 {
t.Errorf("TLS fetcher was called for comix: %v", tlsF.calls)
@@ -831,15 +773,16 @@ func TestComixSkippedWhenNoBrowserFetcher(t *testing.T) {
f := &fakeFetcher{body: comixSeriesFixture, status: 200}
p := &Poller{
Store: s, Fetch: f,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
if len(f.calls) != 0 {
t.Errorf("TLS fetcher was called for comix: %v", f.calls)
}
if got := readLatestCheckedAt(t, s, "comix:n8we-dungeons-and-crayons"); got != 0 {
t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got)
}
}
func TestRunOncePrefetchesKaganeCover(t *testing.T) {
@@ -860,10 +803,10 @@ func TestRunOncePrefetchesKaganeCover(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL))
if err != nil || !ok {
@@ -898,11 +841,13 @@ func TestRunOnceDoesNotRefetchKaganeCover(t *testing.T) {
covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"}
p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers,
Now: func() time.Time { return at }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
at = at.Add(2 * time.Hour)
p.runOnce(context.Background())
p.waitCovers()
if got := covers.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1 after two due cycles", got)
@@ -927,9 +872,10 @@ func TestRunOnceCoverFailureDoesNotBlockChapter(t *testing.T) {
p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
CoverFetch: &fakeCoverFetcher{err: errors.New("browser unavailable")},
Now: func() time.Time { return now }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
p.waitCovers()
got, found, err := s.Get(s.OwnerID(), key)
if err != nil || !found {
@@ -960,9 +906,10 @@ func TestRunOnceRejectsInvalidKaganeCover(t *testing.T) {
p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
CoverFetch: &fakeCoverFetcher{body: []byte("not an image"), contentType: "text/html"},
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
t.Fatalf("invalid cover persisted = %v, err %v; want missing", found, err)
@@ -985,9 +932,10 @@ func TestRunOnceWithoutCoverFetcherStillPollsKagane(t *testing.T) {
seedCoverSource(t, dbURL, "kagane", seriesID, coverURL)
p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
t.Fatalf("cover after nil CoverFetch = found %v, err %v; want missing", found, err)
@@ -1010,9 +958,10 @@ func TestRunOnceRoutesNonKaganeCoverToPublicFetcher(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverFetch: browserCovers,
CoverBytesFetch: publicCovers,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if got := publicCovers.callCount(); got != 1 {
t.Fatalf("public cover fetch calls = %d, want 1", got)
@@ -1128,10 +1077,10 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
p := &Poller{
Store: s, Fetch: page, BrowserFetch: page,
CoverBytesFetch: public, CoverFetch: browser,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
got := readBookmark(t, s, tc.key)
wantWire := testCoverBaseURL + "/covers/" + store.CoverAddress(tc.wantCover)
@@ -1181,7 +1130,7 @@ func TestRunOnceDoesNotReplaceExistingCover(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: public,
Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10,
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
at = at.Add(2 * time.Hour)
@@ -1216,9 +1165,10 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: public,
Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10,
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
if got := readBookmark(t, s, key); got.Cover != "" {
t.Fatalf("Cover after failed fetch = %q, want blank", got.Cover)
}
@@ -1231,6 +1181,7 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
public.contentType = "image/jpeg"
at = at.Add(2 * time.Hour)
p.runOnce(context.Background())
p.waitCovers()
if got := public.callCount(); got != 2 {
t.Fatalf("cover fetch calls after retry = %d, want 2", got)
@@ -1264,9 +1215,10 @@ func TestRunOnceBlankCoverFailureDoesNotBlockChapter(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{err: errors.New("cdn down")},
Now: func() time.Time { return now }, Cooldown: time.Hour, Batch: 10,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
p.waitCovers()
got := readBookmark(t, s, key)
if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 {
@@ -1289,3 +1241,430 @@ const kaganeAPIFixtureWithCover = `
{"book_id":"b","title":"Episode 41","chapter_no":"41","sort_no":41},
{"book_id":"c","title":"Episode 40.5","chapter_no":"40.5","sort_no":40}]}
`
func TestEffectiveGap(t *testing.T) {
asura := sites["asura"]
tests := []struct {
eligible int
want time.Duration
clamped bool
}{
{0, 10 * time.Second, false},
{5, 10 * time.Second, false},
{360, 10 * time.Second, false},
{720, 5 * time.Second, false},
{3600, time.Second, false},
{4000, time.Second, true},
}
for _, tt := range tests {
got, clamped := effectiveGap(asura, tt.eligible)
if got != tt.want || clamped != tt.clamped {
t.Errorf("effectiveGap(asura, %d) = (%s, %v), want (%s, %v)",
tt.eligible, got, clamped, tt.want, tt.clamped)
}
}
}
// The due query orders by sharedness first, then age: a series two readers
// track is polled before a single-reader series that has waited far longer
// (ADR-0003, issue #100).
func TestRunOnceOrdersBySharednessThenAge(t *testing.T) {
s, url := newTestStore(t)
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))}, t.TempDir(), testCoverBaseURL)
if err != nil {
t.Fatalf("Open second reader: %v", err)
}
t.Cleanup(func() { other.Close() })
now := time.UnixMilli(5_000_000)
// popular: two readers, due for 2 minutes. loner: one reader, due for 23
// minutes. Popularity must win — the loner waited far longer.
const (
popularKey = "asura:popular"
lonerKey = "asura:loner"
popularURL = "https://asurascans.com/comics/popular"
lonerURL = "https://asurascans.com/comics/loner"
)
seedForCheck(t, s, popularKey, popularURL, now.Add(-62*time.Minute).UnixMilli())
seedForCheck(t, s, lonerKey, lonerURL, 0)
if _, err := s.Upsert(other.OwnerID(), store.Bookmark{
Key: popularKey, Site: "asura", SeriesID: "popular", UpdatedAt: 2000,
}); err != nil {
t.Fatalf("seed second reader: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, now).runOnce(context.Background())
if len(f.calls) != 2 {
t.Fatalf("fetches = %d, want 2", len(f.calls))
}
if f.calls[0] != popularURL {
t.Fatalf("first fetch = %q, want the shared series %q", f.calls[0], popularURL)
}
}
// Two refusals in one pass stop the Lane: the remaining Series stay unstamped
// and due, and the Lane backs off for refuseBackoff before trying the Site
// again. One hostile Site burns only its own Lane's budget (issue #100).
func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
const n = 4
for i := 0; i < n; i++ {
key := fmt.Sprintf("kagane:s%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0)
}
browser := &fakeFetcher{status: 403}
tls := &fakeFetcher{status: 200}
// An asura Series sits on its own Lane: it must still be polled while
// kagane's Lane burns its budget on refusals (story 4).
seedForCheck(t, s, "asura:still-polls", "https://asurascans.com/series/still-polls", 0)
p := &Poller{
Store: s, Fetch: tls, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches after first pass = %d, want 2 (refused twice)", got)
}
if got := tls.callCount(); got != 1 {
t.Fatalf("asura fetches after first pass = %d, want 1 (its Lane is independent)", got)
}
stamped := 0
for i := 0; i < n; i++ {
if readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)) == now.UnixMilli() {
stamped++
}
}
if stamped != 2 {
t.Fatalf("stamped series = %d, want 2; the remaining two stay due", stamped)
}
// Inside the backoff window nothing is attempted.
p.Now = func() time.Time { return now.Add(14 * time.Minute) }
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches inside backoff = %d, want still 2", got)
}
// Past the backoff the Lane resumes and the two untried Series are polled.
p.Now = func() time.Time { return now.Add(16 * time.Minute) }
p.runOnce(context.Background())
if got := browser.callCount(); got != 4 {
t.Fatalf("browser fetches after backoff = %d, want 4", got)
}
for i := 0; i < n; i++ {
if got := readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)); got == 0 {
t.Fatalf("kagane:s%d still untried after backoff", i)
}
}
}
// Below five due Series with none waiting long, a browser Lane leaves Chrome
// asleep; five due, or one waiting browserWakeAge, wakes it (ADR-0005).
func TestBrowserLaneWakeThresholds(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
// Freshly due: checked two minutes before the rest elapses, so the wait
// is far below browserWakeAge.
seed := func(i int) {
key := fmt.Sprintf("kagane:w%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], now.Add(-62*time.Minute).UnixMilli())
}
for i := 0; i < 3; i++ {
seed(i)
}
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got)
}
// 5 due crosses the count threshold.
for i := 3; i < 5; i++ {
seed(i)
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 5 {
t.Fatalf("browser fetches with 5 due series = %d, want 5", got)
}
// A single long-neglected series wakes the browser by age alone.
seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0)
p.runOnce(context.Background())
if got := browser.callCount(); got != 6 {
t.Fatalf("browser fetches with one ancient series = %d, want 6", got)
}
}
// When one browser Lane loses the sidecar, the round's remaining browser
// Lanes are skipped: every fetch would fail anyway, and their Series must not
// burn their stamps on a dead Chrome (issue #100).
func TestRunOnceUnreachableBrowserStopsBrowserLanes(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
seedForCheck(t, s, "kagane:k", "https://kagane.to/series/k", 0)
seedForCheck(t, s, "novelfull:n", "https://novelfull.com/n.html", 0)
interrupted := fmt.Errorf("%w: %w", errBrowserInterrupted, errors.New("restart"))
browser := &fakeFetcher{status: 200, err: interrupted}
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
// Sorted lane order: comix, kagane, novelfull. Only comix attempted.
if got := browser.callCount(); got != 1 {
t.Fatalf("browser fetches = %d, want 1 (only the first browser lane)", got)
}
if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() {
t.Fatalf("comix stamp = %d, want %d", got, now.UnixMilli())
}
for _, key := range []string{"kagane:k", "novelfull:n"} {
if got := readLatestCheckedAt(t, s, key); got != 0 {
t.Fatalf("%s stamp = %d, want 0 (untried)", key, got)
}
}
// The shared flag decays after refuseBackoff: the next round probes
// again. comix's Series is resting (stamped last round), so the probe
// falls to kagane — the only Lane with something due — and its fresh
// loss re-gates the Lanes behind it.
p.Now = func() time.Time { return now.Add(16 * time.Minute) }
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches after backoff decay = %d, want 2 (kagane probes again)", got)
}
if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() {
t.Fatalf("comix stamp after decay = %d, want %d (resting, untouched)", got, now.UnixMilli())
}
if got := readLatestCheckedAt(t, s, "kagane:k"); got != now.Add(16*time.Minute).UnixMilli() {
t.Fatalf("kagane stamp after decay = %d, want %d (the probe)", got, now.Add(16*time.Minute).UnixMilli())
}
if got := readLatestCheckedAt(t, s, "novelfull:n"); got != 0 {
t.Fatalf("novelfull stamp after decay = %d, want 0 (gated by kagane's fresh loss)", got)
}
}
// A browser Lane that cannot keep up logs the backlog every pass, so the
// decision to give browser Sites more pages is measured, not guessed.
func TestRunOnceLogsBrowserLaneBehind(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
// Checked three rests ago: two rests' worth of wait past the due moment.
seedForCheck(t, s, "kagane:old", "https://kagane.to/series/old", now.Add(-3*time.Hour).UnixMilli())
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200},
BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := logs.String(); !strings.Contains(got, "latest poll kagane: browser lane behind by 1h0m0s") {
t.Fatalf("behind log = %q, want it to name kagane and the backlog", got)
}
}
// gatedCoverFetcher blocks every Fetch on a gate, so a test can hold a cover
// heal in flight and prove a Lane does not wait for it.
type gatedCoverFetcher struct {
inner *fakeBytesCoverFetcher
gate chan struct{}
started chan struct{}
once sync.Once
}
func (g *gatedCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte, string, error) {
g.once.Do(func() { g.started <- struct{}{} })
<-g.gate
return g.inner.Fetch(ctx, sourceURL)
}
// Cover heals run in the background: a heal stuck on a slow CDN must not
// delay the Lane's next Series-page Poll, or a large import with many blanks
// would make every Latest Chapter go stale (issue #100).
func TestRunOnceCoverFetchDoesNotDelayNextPoll(t *testing.T) {
s, dbURL := newTestStore(t)
// Two Series whose legacy cover sources still need healing.
for i := 0; i < 2; i++ {
key := fmt.Sprintf("asura:cover-%d", i)
seriesID := fmt.Sprintf("cover-%d", i)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID,
SeriesURL: "https://asurascans.com/comics/" + seriesID, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
seedCoverSource(t, dbURL, "asura", seriesID, fmt.Sprintf("https://cdn.example/covers/%d.jpg", i))
}
pages := &fakeFetcher{body: asuraSeriesFixture, status: 200}
gated := &gatedCoverFetcher{
inner: &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"},
gate: make(chan struct{}),
started: make(chan struct{}, 1),
}
p := &Poller{
Store: s, Fetch: pages, CoverBytesFetch: gated,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
done := make(chan struct{})
go func() {
p.runOnce(context.Background())
close(done)
}()
<-gated.started // the first cover heal is now stuck on its CDN
select {
case <-done:
// The Lane finished its page fetches without waiting for the cover.
case <-time.After(5 * time.Second):
t.Fatal("runOnce blocked on an in-flight cover fetch")
}
close(gated.gate)
p.waitCovers()
if got := pages.callCount(); got != 2 {
t.Fatalf("series page fetches = %d, want 2", got)
}
if got := gated.inner.callCount(); got != 2 {
t.Fatalf("cover fetches = %d, want 2", got)
}
}
// At 3601 eligible Series the effective gap falls below the one-second floor;
// the clamp warning must name the Site so the operator knows which Lane is
// outrunning its plan.
func TestRunOnceClampWarningNamesTheSite(t *testing.T) {
s, dbURL := newTestStore(t)
db, err := sql.Open("pgx", dbURL)
if err != nil {
t.Fatalf("open %s: %v", dbURL, err)
}
defer db.Close()
if _, err := db.Exec(`INSERT INTO series (site, series_id, series_url, latest_checked_at)
SELECT 'asura', 'bulk-' || g, 'https://asurascans.com/comics/bulk-' || g, 0
FROM generate_series(1, 3601) AS g`); err != nil {
t.Fatalf("bulk seed series: %v", err)
}
if _, err := db.Exec(`INSERT INTO bookmarks (reader_id, site, series_id, updated_at)
SELECT (SELECT id FROM readers ORDER BY id LIMIT 1), 'asura', 'bulk-' || g, 1000
FROM generate_series(1, 3601) AS g`); err != nil {
t.Fatalf("bulk seed bookmarks: %v", err)
}
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
newTestPoller(t, s, &fakeFetcher{body: "<html></html>", status: 200}, time.UnixMilli(5_000_000)).
runOnce(context.Background())
if got := logs.String(); !strings.Contains(got, "latest poll asura: gap clamped to 1s floor (eligible series=3601)") {
t.Fatalf("clamp warning = %q, want it to name asura and 3601", got)
}
}
// The owner's admin page (issue #102) reads Lane state out of the poller.
// Before any pass the snapshot is empty — a restart must render "no data
// yet", not zeroes — and each pass records what it saw: the frozen clock,
// the due count and the pace, with refusal backoff derived at snapshot time.
func TestLaneStatus(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
if st := p.LaneStatus(); len(st.Lanes) != 0 {
t.Fatalf("lanes before any pass = %d, want 0 (nothing has run)", len(st.Lanes))
} else if st.BrowserConfigured || st.BrowserReachable {
t.Fatalf("browser before any pass = configured=%v reachable=%v, want false without a browser fetcher", st.BrowserConfigured, st.BrowserReachable)
}
seedForCheck(t, s, "asura:chronicles", "https://asurascans.com/series/chronicles", 0)
// Two refusals put kagane's Lane into backoff; asura sits on its own Lane.
browser := &fakeFetcher{status: 403}
p.BrowserFetch = browser
for i := range 2 {
key := fmt.Sprintf("kagane:s%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0)
}
p.runOnce(context.Background())
st := p.LaneStatus()
var asura, kagane LaneState
for _, lane := range st.Lanes {
switch lane.Site {
case "asura":
asura = lane
case "kagane":
kagane = lane
}
}
if st.Lanes[0].Site != "asura" {
t.Fatalf("first lane = %q, want asura (snapshot sorted by Site)", st.Lanes[0].Site)
}
if asura.Site == "" {
t.Fatalf("asura missing from snapshot: %+v", st.Lanes)
}
if !asura.LastRun.Equal(now) {
t.Fatalf("asura LastRun = %s, want the frozen clock %s", asura.LastRun, now)
}
if asura.Due != 1 {
t.Fatalf("asura Due = %d, want 1", asura.Due)
}
if asura.Gap == 0 {
t.Fatal("asura Gap = 0, want the Lane's pace")
}
if asura.Browser || asura.Refusing {
t.Fatalf("asura = %+v, want a TLS Lane that is not refusing", asura)
}
if !kagane.Browser || !kagane.Refusing {
t.Fatalf("kagane = %+v, want a browser Lane in refusal backoff", kagane)
}
if !st.BrowserConfigured || !st.BrowserReachable {
t.Fatalf("browser after round = configured=%v reachable=%v, want true/true (sidecar never lost)", st.BrowserConfigured, st.BrowserReachable)
}
if asura.Checked != 1 {
t.Fatalf("asura Checked = %d, want the one Series it read", asura.Checked)
}
// A pass that declines to look (kagane is now in backoff) must not restate
// the figures it never gathered as zeroes: the last real pass's due count
// and pace stand until a pass replaces them.
before := kagane
if before.Due == 0 || before.Gap == 0 {
t.Fatalf("kagane after its refusing pass = %+v, want the figures that pass gathered", before)
}
p.runOnce(context.Background())
for _, lane := range p.LaneStatus().Lanes {
if lane.Site != "kagane" {
continue
}
if lane.Due != before.Due || lane.Gap != before.Gap {
t.Fatalf("kagane after a skipped pass = due %d gap %s, want the previous pass's %d / %s",
lane.Due, lane.Gap, before.Due, before.Gap)
}
}
// A lost sidecar reads as unreachable for the same window the Lanes skip.
p.setBrowserDown(now)
if st := p.LaneStatus(); !st.BrowserConfigured || st.BrowserReachable {
t.Fatalf("browser after loss = configured=%v reachable=%v, want true/false", st.BrowserConfigured, st.BrowserReachable)
}
}
+16 -1
View File
@@ -9,7 +9,7 @@ import (
// seriesRead carries the two facts the poll and the acquirer both extract
// from a series page. Persistence, stamps and scheduling stay with the
// callers, so the policies that keep the two flows distinct (stamp order,
// cooldowns) are not swallowed by the module.
// rests) are not swallowed by the module.
type seriesRead struct {
Latest latestChapter
HasLatest bool
@@ -22,6 +22,9 @@ type seriesRead struct {
// errNotFetchable and errNoFetcher separate the gate and the route from fetch
// failures so each caller keeps its own distinct log line for all three.
// errChallengeHeld (browser.go) is the outcome of a Site that answered with
// its interstitial — status 403 (cf-mitigated) or a challenge page body — and
// is how a Lane tells a refusal from an ordinary failure (issue #100).
var (
errNotFetchable = errors.New("series url not fetchable")
errNoFetcher = errors.New("no fetcher for site")
@@ -49,9 +52,21 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
if err != nil {
return seriesRead{}, fmt.Errorf("fetch %s: %w", seriesURL, err)
}
if status == 403 {
// Cloudflare's challenge response for these Sites (cf-mitigated). The
// browser fetcher returns exactly this on a held interstitial, and a
// plain-TLS 403 means the same: the Site is refusing.
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status)
}
if status != 200 {
return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status)
}
if isInterstitial(body) {
// A 200 that is the challenge page, not the payload: the TLS route can
// receive this where the browser would have kept re-reading. Same
// refusal as the 403.
return seriesRead{}, fmt.Errorf("%w: fetch %s: interstitial body", errChallengeHeld, seriesURL)
}
latest, hasLatest := latestChapterFrom(site, seriesURL, body)
cover, hasCover := coverFrom(site, seriesURL, body)
return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil
+65 -3
View File
@@ -9,6 +9,7 @@ import (
"sort"
"strconv"
"strings"
"time"
"github.com/chromedp/chromedp"
)
@@ -32,6 +33,11 @@ type site struct {
LatestChapter func(seriesURL, body string) (latestChapter, bool)
// Cover finds the Cover address in a fetched body.
Cover func(seriesURL, body string) (string, bool)
// Rest is how long a Series of this Site rests between Polls.
Rest time.Duration
// Gap is the Lane's strictest pace: at least one second must pass between
// two consecutive Series-page Polls of this Site (issue #100).
Gap time.Duration
// Browser reads this Site's payload from a cleared browser tab; nil
// means the page is fetched over plain TLS.
Browser *browserRead
@@ -380,6 +386,50 @@ func publishedCoverURL(value string) string {
return strings.ReplaceAll(value, " ", "%20")
}
// Poll Lane constants (issue #100). The per-Site structure is deliberately
// uniform at first — every Site rests an hour and gaps ten seconds — but it
// exists so a single Site can be slowed if it turns hostile, and the numbers
// stay in the registry so the structure has a place to differ.
const (
// defaultRest is how long every Series rests between Polls.
defaultRest = time.Hour
// defaultGap is the strictest pace of every Lane unless the eligible
// Series count forces it tighter.
defaultGap = 10 * time.Second
// minGap floors the effective gap. One request per second is already an
// order of magnitude past the strictest rate rule a free-plan Site can
// express (docs/research/cloudflare-bot-scoring-and-poll-cadence.md);
// below it the Lane is outrunning its own plan and says so loudly.
minGap = time.Second
// refuseBackoff is how long a Lane waits after its Site refused twice in
// one run before attempting it again.
refuseBackoff = 15 * time.Minute
// browserWakeCount and browserWakeAge gate a browser Lane's run: five or
// more due Series, or any one of them waiting this long, or Chrome stays
// asleep (ADR-0005 on-demand browser).
browserWakeCount = 5
browserWakeAge = 15 * time.Minute
)
// effectiveGap is a Site's pace: the registry gap, or one rest divided by the
// eligible Series count when that is smaller, never below one second. The
// denominator follows defaultRest rather than a literal hour so a Site whose
// rest is ever changed keeps its per-Series pace in step. The second return is
// true when the one-second floor engaged (and the Lane logs a warning naming
// the Site, every round it does).
func effectiveGap(s site, eligible int) (time.Duration, bool) {
gap := s.Gap
if eligible > 0 {
if perSeries := defaultRest / time.Duration(eligible); perSeries < gap {
gap = perSeries
}
}
if gap < minGap {
return minGap, true
}
return gap, false
}
// sites is the registry: one entry per Site, keyed by the stored site string.
// Adding a Site means adding an entry here and nowhere else — the dispatch
// functions above and the poller's route list are lookups into this map. An
@@ -390,16 +440,22 @@ var sites = map[string]site{
Host: "asurascans.com",
LatestChapter: asuraLatestChapter,
Cover: ogImageCover,
Rest: defaultRest,
Gap: defaultGap,
},
"demonic": {
Host: "demonicscans.org",
LatestChapter: demonicLatestChapter,
Cover: ogImageCover,
Rest: defaultRest,
Gap: defaultGap,
},
"comix": {
Host: "comix.to",
LatestChapter: comixLatestChapter,
Cover: comixCoverEntry,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
Read: comixRead,
// The interstitial is served in place of the page, so "arrived"
@@ -414,6 +470,8 @@ var sites = map[string]site{
Host: "kagane.to",
LatestChapter: kaganeLatestChapter,
Cover: kaganeCoverEntry,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
Read: kaganeRead,
Done: func(body string) bool { return body != "" },
@@ -426,6 +484,8 @@ var sites = map[string]site{
Host: "novelfull.com",
LatestChapter: novelfullLatestChapter,
Cover: novelfullCoverEntry,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
Read: novelfullRead,
// The interstitial has a DOM too, so "the payload arrived" has to
@@ -438,13 +498,15 @@ var sites = map[string]site{
Host: "lightnovelworld.net",
LatestChapter: lnwLatestChapter,
Cover: ogImageCover,
Rest: defaultRest,
Gap: defaultGap,
},
}
// browserBackedSites is derived from the registry: the Sites whose pages are
// read through the browser sidecar, which are also the ones granted the longer
// cooldown. Sorted so callers that range it (the due query, the browser
// fetcher's dispatch) see a stable order instead of map-iteration noise.
// read through the browser sidecar. Sorted so callers that range it (the
// browser fetcher's dispatch) see a stable order instead of map-iteration
// noise.
func browserBackedSites() []string {
out := make([]string, 0, len(sites))
for name, s := range sites {
+74
View File
@@ -0,0 +1,74 @@
package latest
import "time"
// LaneState is the administrative page's view of one Poll Lane (issue #102):
// what the Lane's last pass saw. Due, Gap and Checked are filled in as the
// pass computes them; a pass that returned before reaching a figure (refusal
// backoff, sidecar down) carries the previous pass's figures forward rather
// than overwriting them with zeroes the page would state as fact.
type LaneState struct {
Site string
Due int
LastRun time.Time
Gap time.Duration
// Checked is how many Series this pass actually read. A Lane with Series
// due and nothing checked has stopped working; one with nothing due is
// merely quiet, and the page must not draw the two the same (story 13).
Checked int
Clamped bool
Refusing bool
Browser bool
}
// Status is the owner's page snapshot of the whole poller (issue #102).
type Status struct {
Lanes []LaneState
BrowserConfigured bool
BrowserReachable bool
}
// LaneStatus returns a copy of the poller's Lane state for the owner's page.
// Only Sites that have completed a pass appear — a restart therefore renders
// "no data yet" instead of confident zeroes — in the same order Run iterates.
// Refusing is derived at snapshot time from the refusal backoff, not stored,
// so a Lane that cooled down between passes reports false without a new pass.
// BrowserReachable mirrors the Lanes' own gate: the sidecar is down only
// within the refuseBackoff window since its last loss.
func (p *Poller) LaneStatus() Status {
p.mu.Lock()
defer p.mu.Unlock()
lanes := make([]LaneState, 0, len(p.laneStates))
now := p.Now()
for _, name := range laneNames() {
st, ok := p.laneStates[name]
if !ok {
continue
}
st.Refusing = now.Before(p.refuseUntil[name])
lanes = append(lanes, st)
}
configured := p.BrowserFetch != nil
reachable := configured
if reachable && !p.browserDownAt.IsZero() && now.Sub(p.browserDownAt) < refuseBackoff {
reachable = false
}
return Status{Lanes: lanes, BrowserConfigured: configured, BrowserReachable: reachable}
}
// recordLaneState stores one Lane's last pass for LaneStatus. Called deferred
// from runLanePass so every return path records, even a pass that refused.
// A pass that never reached the pace (Gap zero) keeps the last pass's figures:
// the Lane's due count and gap did not become zero because this pass declined
// to look, and the row's own marks say why it declined.
func (p *Poller) recordLaneState(st LaneState) {
p.mu.Lock()
defer p.mu.Unlock()
if p.laneStates == nil {
p.laneStates = make(map[string]LaneState)
}
if prev, ok := p.laneStates[st.Site]; ok && st.Gap == 0 {
st.Due, st.Gap, st.Clamped, st.Checked = prev.Due, prev.Gap, prev.Clamped, prev.Checked
}
p.laneStates[st.Site] = st
}
@@ -0,0 +1,6 @@
-- The owner's administrative page (issue #102) renders these counters and
-- offers a control to clear them, deliberately shipped before the Sighting
-- feature (issue #103) that fills them, so a false mark never needs SQL
-- against production. Zero counters mean a trusted Reader.
ALTER TABLE readers ADD COLUMN sighting_agreements integer NOT NULL DEFAULT 0;
ALTER TABLE readers ADD COLUMN sighting_disagreements integer NOT NULL DEFAULT 0;
+74 -22
View File
@@ -16,7 +16,6 @@ import (
"strconv"
"strings"
"github.com/jackc/pgx/v5/pgtype"
_ "github.com/jackc/pgx/v5/stdlib"
)
@@ -39,7 +38,7 @@ type Bookmark struct {
// origin once the bytes exist, and "" until they do — never a third-party
// address and never an address that 404s (ADR-0007). A client may still
// send this field and it is discarded on the way in; see Upsert.
Cover string `json:"cover"`
Cover string `json:"cover"`
LastChapter string `json:"last_chapter"`
LastChapterNum float64 `json:"last_chapter_num"`
LastChapterURL string `json:"last_chapter_url"`
@@ -316,25 +315,42 @@ func (s *Store) EnsureReader(discordID string, epochZeroHash [32]byte) (int64, e
return id, nil
}
// SightingDisagreementLimit is the number of contradictions that stop that
// Reader's Sightings from deferring a Poll (issue #103). The counters exist
// before the mechanism that moves them, so the admin page (issue #102) can
// clear a false mark without waiting for the Sighting feature.
const SightingDisagreementLimit = 3
// Blocked reports whether this Reader's Sighting marks have reached the
// disagreement limit, which stops their Sightings from deferring a Poll.
func (r ReaderSummary) Blocked() bool {
return r.Disagreements >= SightingDisagreementLimit
}
// ReaderSummary is one Reader as the owner's administration panel sees them:
// who they are and how many live sessions they hold. No credential material,
// hashed or otherwise, is exposed.
// who they are, how many live sessions they hold, and their Sighting marks.
// No credential material, hashed or otherwise, is exposed.
type ReaderSummary struct {
ID int64
DiscordID string
// Sessions counts unexpired session rows — what the owner revokes.
Sessions int
// Agreements and Disagreements are the Sighting counters (issue #102);
// zero means a trusted Reader.
Agreements int
Disagreements int
}
// Readers lists every Reader with their live session count, oldest first, so
// the owner row (always the oldest) heads the list.
// Readers lists every Reader with their live session count and Sighting
// marks, oldest first, so the owner row (always the oldest) heads the list.
func (s *Store) Readers() ([]ReaderSummary, error) {
rows, err := s.db.Query(`
SELECT r.id, r.discord_id,
r.sighting_agreements, r.sighting_disagreements,
count(sess.id) FILTER (WHERE sess.expires_at > now()) AS sessions
FROM readers r
LEFT JOIN sessions sess ON sess.reader_id = r.id
GROUP BY r.id, r.discord_id
GROUP BY r.id, r.discord_id, r.sighting_agreements, r.sighting_disagreements
ORDER BY r.id`)
if err != nil {
return nil, fmt.Errorf("query readers: %w", err)
@@ -344,7 +360,7 @@ func (s *Store) Readers() ([]ReaderSummary, error) {
out := []ReaderSummary{}
for rows.Next() {
var r ReaderSummary
if err := rows.Scan(&r.ID, &r.DiscordID, &r.Sessions); err != nil {
if err := rows.Scan(&r.ID, &r.DiscordID, &r.Agreements, &r.Disagreements, &r.Sessions); err != nil {
return nil, fmt.Errorf("scan reader: %w", err)
}
out = append(out, r)
@@ -352,6 +368,18 @@ func (s *Store) Readers() ([]ReaderSummary, error) {
return out, rows.Err()
}
// ClearReaderMarks zeroes a Reader's Sighting counters. It is the owner's
// remedy for a mark produced by a broken Site adapter rather than a dishonest
// Reader: it restores a privilege, it is not destruction.
func (s *Store) ClearReaderMarks(readerID int64) error {
if _, err := s.db.Exec(`
UPDATE readers SET sighting_agreements = 0, sighting_disagreements = 0
WHERE id = $1`, readerID); err != nil {
return fmt.Errorf("clear reader marks for reader %d: %w", readerID, err)
}
return nil
}
// readersMigration is the version that creates the readers table. The owner
// seed runs between two migrate passes, so that the run-once migration which
// attaches existing bookmarks (0004) finds the owner row.
@@ -897,10 +925,13 @@ func (s *Store) Delete(readerID int64, key string) error {
return nil
}
// DueForLatestCheck returns series whose server-side latest-chapter check has
// aged past the appropriate cutoff, ordered by how many bookmarks reference
// them (descending) then least-recently-checked first, at most limit of them.
// Browser-backed sites use browserCutoffMs; every other site uses cutoffMs.
// DueForLatestCheck returns one Site's series whose server-side
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks
// reference them (descending) then least-recently-checked first. One Site per
// query, because each Poll Lane asks for its own list: the query carries one
// Site and one cut-off instead of parallel lists (issue #100). There is no
// limit — the Lane's own gap paces the fetches, and the batch size that used
// to cap this query is gone with the shared pace.
//
// The reader_count ordering is the point of the split (ADR-0003): a series
// shared by several readers is fetched once per due cycle, and the popular
@@ -916,20 +947,17 @@ func (s *Store) Delete(readerID int64, key string) error {
// burns requests. Archived bookmarks still count — knowing what a shelved
// series is up to is the whole reason for archiving instead of deleting.
// A series with no bookmarks at all never appears: the join excludes it.
func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites []string, limit int) ([]Series, error) {
func (s *Store) DueForLatestCheck(site string, cutoffMs int64) ([]Series, error) {
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
FROM series s
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
WHERE s.series_url <> ''
AND s.latest_checked_at <= CASE
WHEN s.site = ANY($3::text[]) THEN $2::bigint
ELSE $1::bigint
END
WHERE s.site = $1
AND s.series_url <> ''
AND s.latest_checked_at <= $2::bigint
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
ORDER BY reader_count DESC, s.latest_checked_at ASC
LIMIT $4`, cutoffMs, browserCutoffMs, pgtype.FlatArray[string](browserSites), limit)
ORDER BY reader_count DESC, s.latest_checked_at ASC`, site, cutoffMs)
if err != nil {
return nil, fmt.Errorf("query due series: %w", err)
}
@@ -946,6 +974,30 @@ func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites
return out, rows.Err()
}
// EligibleSeriesCount returns how many of a Site's Series still have at least
// one bookmark outside the finished bucket. It is the denominator of the
// Lane's pace (issue #100): the effective gap is the smaller of the registry
// gap and one hour divided by this count, so Series that will never be Polled
// do not make the Lane faster than it needs to be, and counting every eligible
// Series rather than only those currently due keeps the pace steady — the
// single worst moment to be fastest is startup, when everything is due at
// once.
func (s *Store) EligibleSeriesCount(site string) (int, error) {
var n int
err := s.db.QueryRow(`SELECT COUNT(*) FROM (
SELECT 1
FROM series s
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
WHERE s.site = $1
GROUP BY s.site, s.series_id
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
) e`, site).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count eligible series %s: %w", site, err)
}
return n, nil
}
// MarkLatestChecked records that the server looked at a series at ts, whatever
// the look turned up. Marking a missing series is not an error: the row may
// have been orphaned while a fetch was in flight.
@@ -954,7 +1006,7 @@ func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
// a whole Bookmark from the client and Upsert writes every series column it
// knows about, so a userscript PUT — which has no idea this field exists —
// would write a zero and reset the cooldown, making the poller re-fetch that
// would write a zero and reset the rest, making the poller re-fetch that
// series every tick for as long as the user kept reading it.
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
if _, err := s.db.Exec(
@@ -966,7 +1018,7 @@ func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
}
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
// tests outside this package (the poller's own tests assert on cooldown
// tests outside this package (the poller's own tests assert on rest
// bookkeeping) — see MarkLatestChecked for why the field stays off the
// client-visible row.
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
+127 -12
View File
@@ -298,7 +298,7 @@ func TestDueForLatestCheck(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:x", tt.seriesURL, tt.checkedAt)
due, err := s.DueForLatestCheck(now-hour, now-hour, nil, 10)
due, err := s.DueForLatestCheck("asura", now-hour)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
@@ -309,22 +309,25 @@ func TestDueForLatestCheck(t *testing.T) {
}
}
func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) {
func TestDueForLatestCheckOldestFirstAndScopedToSite(t *testing.T) {
s := newTestStore(t)
// Insert newest-checked first so a correct ORDER BY has to reverse it.
seedForCheck(t, s, "asura:c", "https://asurascans.com/comics/c", 300)
seedForCheck(t, s, "asura:b", "https://asurascans.com/comics/b", 200)
seedForCheck(t, s, "asura:a", "https://asurascans.com/comics/a", 100)
// A second Site's due series must not appear in asura's list: each Lane
// asks for one Site, and no Lane may see another's queue.
seedForCheck(t, s, "demonic:z", "https://demonicscans.org/manga/z", 0)
due, err := s.DueForLatestCheck(1000, 1000, nil, 2)
due, err := s.DueForLatestCheck("asura", 1000)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 2 {
t.Fatalf("got %d rows, want 2 (limit)", len(due))
if len(due) != 3 {
t.Fatalf("got %d rows, want 3 (all of asura's, none of demonic's)", len(due))
}
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" {
t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key())
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" || due[2].Key() != "asura:c" {
t.Fatalf("got %q,%q,%q; want asura:a,asura:b,asura:c (oldest first)", due[0].Key(), due[1].Key(), due[2].Key())
}
}
@@ -496,7 +499,7 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
}
}
due, err := store.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
due, err := store.DueForLatestCheck("asura", time.Now().UnixMilli())
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
@@ -512,6 +515,38 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
}
}
// The gap's denominator counts every Series the Lane will ever Poll: a
// finished Series must not make the Lane faster than it needs to be, and
// another Site's Series must not leak into this Site's count.
func TestEligibleSeriesCount(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:reading", "https://asurascans.com/comics/reading", 0)
seedForCheck(t, store, "asura:archived", "https://asurascans.com/comics/archived", 0)
if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:finished", Site: "asura", SeriesID: "finished",
SeriesURL: "https://asurascans.com/comics/finished",
Status: StatusFinished, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed finished: %v", err)
}
seedForCheck(t, store, "demonic:z", "https://demonicscans.org/manga/z", 0)
n, err := store.EligibleSeriesCount("asura")
if err != nil {
t.Fatalf("EligibleSeriesCount: %v", err)
}
if n != 2 {
t.Fatalf("eligible = %d, want 2 (finished excluded, demonic excluded)", n)
}
n, err = store.EligibleSeriesCount("demonic")
if err != nil {
t.Fatalf("EligibleSeriesCount(demonic): %v", err)
}
if n != 1 {
t.Fatalf("eligible(demonic) = %d, want 1", n)
}
}
func TestDisplayChapter(t *testing.T) {
cases := []struct {
name string
@@ -970,7 +1005,7 @@ func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) {
seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100)
due, err := s.DueForLatestCheck(1000, 1000, nil, 10)
due, err := s.DueForLatestCheck("asura", 1000)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
@@ -996,7 +1031,7 @@ func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
t.Fatalf("seed orphan series: %v", err)
}
due, err := s.DueForLatestCheck(1000, 1000, nil, 10)
due, err := s.DueForLatestCheck("asura", 1000)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
@@ -1299,6 +1334,86 @@ func TestReadersAndSessionRevocation(t *testing.T) {
}
}
// The Sighting counters ship before the mechanism that fills them (issue
// #102 before #103), so the admin page depends on their default: a fresh
// Reader reads back trusted. Marking one directly proves Readers() reports
// the counters and Blocked() flips at the limit, and that ClearReaderMarks —
// the owner's remedy for a false mark — zeroes them again.
func TestReaderSightingMarks(t *testing.T) {
s := newTestStore(t)
other := secondReader(t, s)
readers, err := s.Readers()
if err != nil {
t.Fatalf("Readers: %v", err)
}
if len(readers) != 2 {
t.Fatalf("readers = %d, want the owner and the second Reader", len(readers))
}
for _, r := range readers {
if r.Agreements != 0 || r.Disagreements != 0 || r.Blocked() {
t.Fatalf("fresh reader %d has marks: %+v", r.ID, r)
}
}
// The counters' default is the trusted state; writing them directly is
// the only way to exercise the read path until issue #103 moves them.
if _, err := s.db.Exec(`
UPDATE readers SET sighting_agreements = 5, sighting_disagreements = 2
WHERE id = $1`, other); err != nil {
t.Fatalf("mark reader: %v", err)
}
readers, err = s.Readers()
if err != nil {
t.Fatalf("Readers: %v", err)
}
var marked *ReaderSummary
for i := range readers {
if readers[i].ID == other {
marked = &readers[i]
}
}
if marked == nil || marked.Agreements != 5 || marked.Disagreements != 2 {
t.Fatalf("marked reader = %+v, want agreements 5, disagreements 2", marked)
}
if marked.Blocked() {
t.Fatalf("reader with 2 disagreements is blocked; limit is %d", SightingDisagreementLimit)
}
if _, err := s.db.Exec(`
UPDATE readers SET sighting_disagreements = 3 WHERE id = $1`, other); err != nil {
t.Fatalf("block reader: %v", err)
}
readers, err = s.Readers()
if err != nil {
t.Fatalf("Readers: %v", err)
}
for _, r := range readers {
if r.ID == other && !r.Blocked() {
t.Fatalf("reader at the disagreement limit is not blocked: %+v", r)
}
if r.ID == s.OwnerID() && r.Blocked() {
t.Fatalf("untouched owner became blocked: %+v", r)
}
}
if err := s.ClearReaderMarks(other); err != nil {
t.Fatalf("ClearReaderMarks: %v", err)
}
if err := s.ClearReaderMarks(other + 9999); err != nil {
t.Fatalf("ClearReaderMarks(unknown id): %v", err)
}
readers, err = s.Readers()
if err != nil {
t.Fatalf("Readers: %v", err)
}
for _, r := range readers {
if r.Agreements != 0 || r.Disagreements != 0 || r.Blocked() {
t.Fatalf("reader %d not cleared: %+v", r.ID, r)
}
}
}
// Two Readers on one Series: one series row, two independent progresses. The
// second Reader starts at zero however far the first has read, and the shared
// row is still due exactly once.
@@ -1334,7 +1449,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
t.Fatalf("series rows = %d, want 1 shared row for two bookmarks", series)
}
due, err := s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
due, err := s.DueForLatestCheck("asura", time.Now().UnixMilli())
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
@@ -1350,7 +1465,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
if b, ok, err := s.Get(s.OwnerID(), "asura:solo"); err != nil || !ok || b.LastChapterNum != 200 {
t.Fatalf("owner's bookmark after the other's delete = %+v ok=%v err=%v, want it intact", b, ok, err)
}
due, err = s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
due, err = s.DueForLatestCheck("asura", time.Now().UnixMilli())
if err != nil {
t.Fatalf("DueForLatestCheck after delete: %v", err)
}
+251
View File
@@ -0,0 +1,251 @@
package web
import (
"log"
"net/http"
"strconv"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store"
)
// LaneReporter is the administrative page's whole window onto the running
// poller: one snapshot of Poll Lane state, copied out of memory on request.
// The Poller satisfies it in production and a fake with fixed values satisfies
// it in tests, so the page's tests need neither a poller nor a Site.
type LaneReporter interface {
LaneStatus() latest.Status
}
// adminView is what the administrative page and the roster fragment receive.
type adminView struct {
Readers []store.ReaderSummary
// OwnerID travels with the roster so it can tell the owner's own row from
// the Readers they may act on.
OwnerID int64
Lanes lanesView
}
// lanesView is the Lane status block: one row per Site that has run, plus the
// browser fact, which is shared by the three browser Sites rather than held
// once per Site.
type lanesView struct {
Rows []laneRow
// PollerOff means no poller is running at all (disabled by config, or its
// client could not be built). The browser line must not answer "not
// configured" then: the sidecar is not the reason nothing is polled.
PollerOff bool
BrowserConfigured bool
BrowserReachable bool
}
// laneRow is one Lane formatted for reading rather than for arithmetic: the
// template renders strings and flags, and every judgement about what they mean
// is made here.
type laneRow struct {
Site string
Due int
Ran string
// Checked is how many Series the last pass read. Due without Checked is a
// Lane that has stopped working; the two figures side by side are what
// separate that from a Lane with nothing to do.
Checked int
// Gap is empty when no pass has reached the pace yet, so the row omits the
// figure instead of stating a zero.
Gap string
Clamped bool
Refusing bool
// BrowserLost marks a Lane whose pages can only be read through the
// sidecar while the sidecar is unreachable — including the case where none
// is configured, which stops those Series just as completely.
BrowserLost bool
// Stalled marks a Lane with Series waiting that its last pass did not read
// — the difference between a stopped Lane and a quiet one (story 13).
Stalled bool
// Attention is the one flag the template colours on, so an unhealthy Lane
// is found at a glance rather than read for.
Attention bool
}
// adminRoute pairs a route pattern with its handler so the route list and the
// gate cannot drift apart.
type adminRoute struct {
pattern string
handler http.HandlerFunc
}
// adminRoutes is every route that reaches past the acting Reader. Register
// wraps each one in requireOwner, so a new administrative route is gated by
// being listed here rather than by remembering to write a check inside it.
func (h *Handler) adminRoutes() []adminRoute {
return []adminRoute{
{"GET /admin", h.admin},
{"GET /ui/admin/lanes", h.uiLanes},
{"POST /readers/{id}/revoke", h.revokeReaderSessions},
{"POST /readers/{id}/clear-marks", h.clearReaderMarks},
}
}
// AdminPatterns names every administrative route, so one test can prove the
// owner gate covers all of them rather than one test per route. The receiver is
// nil because only the patterns are read; the bound handlers are never called.
func AdminPatterns() []string {
routes := (*Handler)(nil).adminRoutes()
out := make([]string, 0, len(routes))
for _, rt := range routes {
out = append(out, rt.pattern)
}
return out
}
// requireOwner is the owner test, in one place, layered on the session gate: no
// session is still 401, and a signed-in Reader who is not the owner gets 404
// rather than 403 — a refusal that confirms the address exists is a refusal
// that helps whoever is probing for it.
func (h *Handler) requireOwner(next http.HandlerFunc) http.HandlerFunc {
return h.requireSession(func(w http.ResponseWriter, r *http.Request) {
if readerOf(r) != h.store.OwnerID() {
http.NotFound(w, r)
return
}
next(w, r)
})
}
// admin renders the owner's page: the Reader roster and Poll Lane status.
func (h *Handler) admin(w http.ResponseWriter, r *http.Request) {
readers, err := h.store.Readers()
if err != nil {
log.Printf("admin: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
h.render(w, http.StatusOK, "admin", adminView{
Readers: readers,
OwnerID: h.store.OwnerID(),
Lanes: h.lanesView(),
})
}
// uiLanes answers the status block's own refresh. Only the block refreshes on a
// timer; the roster re-renders after an action, as it always has.
func (h *Handler) uiLanes(w http.ResponseWriter, r *http.Request) {
h.render(w, http.StatusOK, "lanes", h.lanesView())
}
// lanesView copies the poller's snapshot into display form. A nil reporter (no
// poller running) and a poller no Lane has reported to yet are the same thing
// to the page: no data, which it must say rather than draw as confident zeroes
// — an empty page a few seconds after a restart must not read as a stopped one.
func (h *Handler) lanesView() lanesView {
if h.lanes == nil {
return lanesView{PollerOff: true}
}
snap := h.lanes.LaneStatus()
v := lanesView{
Rows: make([]laneRow, 0, len(snap.Lanes)),
BrowserConfigured: snap.BrowserConfigured,
BrowserReachable: snap.BrowserReachable,
}
now := time.Now()
for _, l := range snap.Lanes {
lost := l.Browser && !snap.BrowserReachable
// Series waiting and none read is the shape of a Lane that has stopped
// working, as distinct from one that is quiet for want of work.
stalled := l.Due > 0 && l.Checked == 0
gap := ""
if l.Gap > 0 {
gap = l.Gap.Truncate(time.Second).String()
}
v.Rows = append(v.Rows, laneRow{
Site: l.Site,
Due: l.Due,
Ran: since(now, l.LastRun),
Checked: l.Checked,
Gap: gap,
Clamped: l.Clamped,
Refusing: l.Refusing,
BrowserLost: lost,
Stalled: stalled,
Attention: l.Clamped || l.Refusing || lost || stalled,
})
}
return v
}
// since formats how long ago a Lane last ran, at second resolution: the block
// refreshes every thirty seconds, so anything finer is noise the owner would
// have to ignore.
func since(now, then time.Time) string {
d := now.Sub(then).Truncate(time.Second)
if d < time.Second {
return "just now"
}
return d.String() + " ago"
}
// revokeReaderSessions logs one Reader out of every browser they are signed in
// on. The owner gate is the route's, not this handler's.
func (h *Handler) revokeReaderSessions(w http.ResponseWriter, r *http.Request) {
target, ok := readerPathID(w, r)
if !ok {
return
}
// The owner is not one of the Readers this endpoint reaches: revoking
// themselves would sign out the browser making the request, which is what
// logout is for. The roster hides the button; this refuses the hand-rolled
// POST behind it.
if target == h.store.OwnerID() {
http.NotFound(w, r)
return
}
if err := h.store.DeleteReaderSessions(target); err != nil {
log.Printf("revoke sessions: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
h.renderRoster(w, "revoke sessions")
}
// clearReaderMarks zeroes one Reader's Sighting counters. The guard those
// counters feed has one known false positive — a Site changing its page shape
// makes a correct adapter read a wrong high number and marks every honest
// Reader of that Site at once (issue #103) — and this is its remedy. It
// restores a privilege rather than destroying anything, so the control is
// confirmed but never wears the destruction accent.
func (h *Handler) clearReaderMarks(w http.ResponseWriter, r *http.Request) {
target, ok := readerPathID(w, r)
if !ok {
return
}
if err := h.store.ClearReaderMarks(target); err != nil {
log.Printf("clear marks: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
h.renderRoster(w, "clear marks")
}
// readerPathID reads the Reader a route names, answering the request itself
// when there is nobody to act on.
func readerPathID(w http.ResponseWriter, r *http.Request) (int64, bool) {
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
http.Error(w, "bad reader id", http.StatusBadRequest)
return 0, false
}
return id, true
}
// renderRoster answers an action with the whole roster, so the counts and marks
// it shows cannot describe the state before the tap.
func (h *Handler) renderRoster(w http.ResponseWriter, what string) {
readers, err := h.store.Readers()
if err != nil {
log.Printf("%s: %v", what, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
h.render(w, http.StatusOK, "readers", adminView{Readers: readers, OwnerID: h.store.OwnerID()})
}
+53 -4
View File
@@ -87,6 +87,11 @@
--moss: #7fae86; /* finished */
--clay: #b5906f; /* set chapter */
--trash: #977671; /* remove, resting — icons need 3:1, not 4.5:1 */
/* A Lane needing attention: the admin page's only accent. Verdigris — cool,
the far side of the wheel from ember's crimson, and clear of the archive
blue. Neither ember (new chapter) nor danger (destruction) may say
"system unhealthy". */
--patina: #5fb3a6;
/* Desktop cell borders for the two coloured action states. */
--play-hot-line: #3a1d18;
@@ -146,6 +151,7 @@
--moss: #3d6c46;
--clay: #7c5533;
--trash: #8c6558;
--patina: #1f6f66;
--play-hot-line: #f0cfc6;
--fav-line: #e3d3a4;
@@ -290,9 +296,21 @@ button { cursor: pointer; }
letter-spacing: .04em;
}
/* ---- reader roster (owner only): same hairline panel, one row per Reader ---- */
.readerlist { margin: 0; padding: 0; list-style: none; }
.readerlist li {
/* ---- admin page: two sections on the same measured sheet, no cards ----
The reading page is a list of series; this is a list of facts. Both are
sheets of hairline-separated rows, so the roster keeps the shape it had as
a fold-out and the Lane block copies it. */
.readers, .lanes { margin: 0 20px; padding: 12px 0 16px; border-bottom: 1px solid var(--rule); }
.readers h2, .lanes h2 {
margin: 0;
padding: 8px 0;
font: 500 10px/1 var(--font-mono);
letter-spacing: .2em;
text-transform: uppercase;
color: var(--mute-2);
}
.readerlist, .lanelist { margin: 0; padding: 0; list-style: none; }
.readerlist li, .lanelist li {
display: flex;
align-items: center;
flex-wrap: wrap;
@@ -300,7 +318,8 @@ button { cursor: pointer; }
min-height: 44px;
border-top: 1px solid var(--rule);
}
.readerlist form { margin: 0 0 0 auto; }
.reader-actions { display: flex; gap: 18px; margin-left: auto; }
.readerlist form { margin: 0; }
.reader-id {
font: 500 13px/1.4 var(--font-mono);
letter-spacing: .04em;
@@ -312,6 +331,30 @@ button { cursor: pointer; }
text-transform: uppercase;
color: var(--mute);
}
.reader-sightings, .lane-fact {
font: 500 10px/1 var(--font-mono);
letter-spacing: .14em;
text-transform: uppercase;
color: var(--mute-2);
}
/* Two states the owner is meant to find rather than read for: a Reader whose
reports no longer defer a Poll, and a Lane that is not keeping its promise.
Both wear --patina — never ember, which means one thing, and never danger,
which is destruction. */
.reader-blocked, .lane-mark {
font: 500 10px/1 var(--font-mono);
letter-spacing: .14em;
text-transform: uppercase;
color: var(--patina);
}
.lane-site {
font: 400 19px/1.2 var(--font-display);
color: var(--paper-dim);
}
/* The whole row leans patina when the Lane needs attention, so the scan is one
pass down the left edge rather than a read of every mark. */
.lanelist li.attention .lane-site { color: var(--patina); }
.lane-browser { padding: 12px 0 0; }
/* Revocation cuts someone off, so it wears --danger. Ember stays reserved for
the new-chapter signal. */
.ghost.danger { color: var(--danger); }
@@ -600,6 +643,9 @@ button { cursor: pointer; }
box-shadow: inset 0 -2px 0 var(--ember);
}
.topbar form { margin-left: 18px; }
/* The admin page's topbar has no switch to fill the middle, so its back link
keeps company with Log out at the right edge instead of floating centre. */
.topbar .back { margin-left: auto; }
/* At phone width brand + switch + Log out do not fit on one line, so the
switch takes its own row under the wordmark rather than pushing Log out
off-screen. */
@@ -609,6 +655,9 @@ button { cursor: pointer; }
.libswitch { order: 3; margin-left: 0; }
.libswitch a { flex: 1; text-align: center; padding: 8px 14px; }
.topbar form { margin-left: 12px; }
/* The admin page has no switch to take the second row, so its brand claims
the first outright and the back link keeps Log out company below. */
.topbar:has(.back) .brand { flex: 1 1 100%; }
}
/* ---- action strip: full-width on a phone, hairline-divided cells ---- */
+39
View File
@@ -0,0 +1,39 @@
{{/* The owner's administrative page: everything that reaches past one Reader,
at its own address so it can be bookmarked rather than hunted for inside
the reading page. Owner-only at route registration (requireOwner), which
is why nothing in here re-tests who is asking. */}}
{{define "admin"}}
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="color-scheme" content="dark light">
<title>BookmarkManager — Admin</title>
<link rel="icon" href="/static/logo.svg" type="image/svg+xml">
<link rel="stylesheet" href="/static/style.css">
<link rel="preload" href="/static/fonts/instrument-serif-400-latin.woff2" as="font" type="font/woff2" crossorigin>
<script src="/static/htmx.min.js" defer></script>
</head>
<body>
<div class="sheet">
<header class="topbar">
<h1 class="brand">{{template "mark" .}}<span>Bookmark<em>Manager</em></span></h1>
{{/* Back to the library, no switch: this page belongs to neither library,
and the ember-lit switch says which library you are reading. */}}
<a class="ghost back" href="/">Library</a>
<form method="post" action="/logout">
<button type="submit" class="ghost">Log out</button>
</form>
</header>
{{/* The live region wraps the swapped block rather than being it: the
refresh replaces the section wholesale, and a region recreated on every
update is never announced. */}}
<div aria-live="polite">{{template "lanes" .Lanes}}</div>
{{template "readers" .}}
</div>
</body>
</html>
{{end}}
+4 -2
View File
@@ -28,6 +28,10 @@
<a href="/?lib=novel&amp;tab=all" class="{{if eq .Lib "novel"}}active{{end}}"
{{if eq .Lib "novel"}}aria-current="page"{{end}}>Novels</a>
</nav>
{{/* The owner's only difference on this page: a link out to the
administrative one. It sits beside Log out rather than in the library
switch — that switch says which library, not which page. */}}
{{if .Owner}}<a class="ghost" href="/admin">Admin</a>{{end}}
<form method="post" action="/logout">
<button type="submit" class="ghost">Log out</button>
</form>
@@ -76,8 +80,6 @@
{{template "setup" .}}
{{if .Owner}}{{template "readers" .}}{{end}}
{{template "keyrow" .}}
{{template "recent" .}}
+41
View File
@@ -0,0 +1,41 @@
{{/* Poll Lane status: one row per Site, refreshing itself so a run can be
watched rather than sampled by reloading. The refresh is one attribute on
the fragment root and the endpoint answers with this same fragment, so the
swap replaces the element that asked for it.
Every figure here is read out of the running poller, never out of a table:
a Site absent from Rows has not completed a pass since the last restart,
which the empty state must say — zeroes would read as a stopped Lane. */}}
{{define "lanes"}}
<section class="lanes" id="lanes"
hx-get="/ui/admin/lanes" hx-trigger="every 30s" hx-swap="outerHTML">
<h2>Poll Lanes</h2>
{{if .Rows}}
<ul class="lanelist">
{{range .Rows}}
<li{{if .Attention}} class="attention"{{end}}>
<span class="lane-site">{{.Site}}</span>
<span class="lane-fact">{{.Due}} due</span>
<span class="lane-fact">{{.Checked}} checked</span>
<span class="lane-fact">ran {{.Ran}}</span>
{{if .Gap}}<span class="lane-fact">gap {{.Gap}}</span>{{end}}
{{if .Clamped}}<span class="lane-mark">gap at floor</span>{{end}}
{{if .Refusing}}<span class="lane-mark">refusing</span>{{end}}
{{if .BrowserLost}}<span class="lane-mark">no browser</span>{{end}}
{{if .Stalled}}<span class="lane-mark">not checking</span>{{end}}
</li>
{{end}}
</ul>
{{else}}
<p class="setup-copy">No data yet — no Lane has completed a pass since the
backend started.</p>
{{end}}
<p class="setup-copy lane-browser">
{{if .PollerOff}}Polling is switched off in this deployment: no Lane runs,
and Latest Chapter comes from the userscripts alone.
{{else}}Browser sidecar:
{{if not .BrowserConfigured}}not configured — comix, kagane and novelfull
pages are not fetched through it{{else if .BrowserReachable}}reachable
{{else}}unreachable{{end}}.{{end}}</p>
</section>
{{end}}
+36 -17
View File
@@ -1,29 +1,48 @@
{{/* The owner's Reader roster. Rendered only for the owner (listView.Owner),
and re-rendered whole as the response to a revocation so the session
counts cannot describe the state before the tap. Revocation is
confirm-gated: it signs someone out of every device at once. */}}
{{/* The Reader roster, on the owner's administrative page. Re-rendered whole
as the response to an action so the counts and marks it shows cannot
describe the state before the tap. Both actions are confirm-gated: one
signs a Reader out of every device at once, the other wipes a record.
Owner-only at route registration, so nothing here re-tests who is asking. */}}
{{define "readers"}}
<details class="setup" id="readers">
<summary>Readers</summary>
<section class="readers" id="readers">
<h2>Readers</h2>
<p class="setup-copy">Everyone who has signed in through Discord. Revoking
signs a Reader out of every device; their library and bookmarks are
untouched, and they can sign in again.</p>
untouched, and they can sign in again. The Sighting counters record how
often a later Poll confirmed or contradicted what that Reader's browser
reported; enough contradictions stop their reports deferring a Poll, and
clearing the marks gives that back.</p>
<ul class="readerlist">
{{range .Readers}}
<li>
<span class="reader-id">{{.DiscordID}}</span>
<span class="reader-sessions">{{.Sessions}} session{{if ne .Sessions 1}}s{{end}}</span>
{{/* The owner's own row never offers Revoke: it is the one row where the
button would sign the tapping browser out, and the endpoint refuses
it anyway. Logout is the deliberate way to do that. */}}
{{if and .Sessions (ne .ID $.OwnerID)}}
<form hx-post="/readers/{{.ID}}/revoke" hx-target="#readers" hx-swap="outerHTML"
hx-confirm="Revoking signs this Reader out on every device immediately. Revoke?">
<button type="submit" class="ghost danger">Revoke sessions</button>
</form>
{{end}}
<span class="reader-sightings">{{.Agreements}} confirmed / {{.Disagreements}} contradicted</span>
{{/* Blocked is spelled out rather than left to be worked out from two
numbers and a threshold. */}}
{{if .Blocked}}<span class="reader-blocked">deferral blocked</span>{{end}}
<span class="reader-actions">
{{/* Clearing restores a privilege, so it is a plain ghost button —
the destruction accent belongs to revocation alone. It is offered
on every row, including one reading zero: the remedy must be
findable before the counters climb, not after. */}}
<form hx-post="/readers/{{.ID}}/clear-marks" hx-target="#readers" hx-swap="outerHTML"
hx-confirm="Clearing wipes this Reader's whole Sighting record, confirmations included. Clear?">
<button type="submit" class="ghost">Clear marks</button>
</form>
{{/* The owner's own row never offers Revoke: it is the one row where the
button would sign the tapping browser out, and the endpoint refuses
it anyway. Logout is the deliberate way to do that. */}}
{{if and .Sessions (ne .ID $.OwnerID)}}
<form hx-post="/readers/{{.ID}}/revoke" hx-target="#readers" hx-swap="outerHTML"
hx-confirm="Revoking signs this Reader out on every device immediately. Revoke?">
<button type="submit" class="ghost danger">Revoke sessions</button>
</form>
{{end}}
</span>
</li>
{{end}}
</ul>
</details>
</section>
{{end}}
+18 -56
View File
@@ -50,6 +50,9 @@ type Handler struct {
// httpClient is the plain stdlib client that talks to Discord. It is not
// an injected interface: tests point APIBase at a stub server instead.
httpClient *http.Client
// lanes is the Poll Lane snapshot source the administrative page reads.
// Nil is a running deployment with no poller, not a bug.
lanes LaneReporter
}
// listView is what every list-rendering template receives.
@@ -77,14 +80,9 @@ type listView struct {
// It is not "newly registered": a Reader who deletes their last bookmark is
// in the same position and needs the same links.
EmptyLibrary bool
// Owner marks the acting Reader as the deployment's owner, which unlocks
// the Readers panel. Nothing else in the UI differs.
// Owner marks the acting Reader as the deployment's owner, which offers
// the link to the administrative page. Nothing else in the UI differs.
Owner bool
// Readers is the owner's roster, populated only for the owner's own page
// render and the revocation fragment. OwnerID travels with it so the roster
// can tell the owner's own row apart from the Readers they may revoke.
Readers []store.ReaderSummary
OwnerID int64
}
// PageURL and ListURL are the two link shapes every tab needs. Building them
@@ -111,7 +109,10 @@ type loginView struct {
// New parses every template up front so a broken one kills the process at
// startup rather than the first request that touches it.
func New(s *store.Store, discord DiscordConfig, tokenKey []byte, mangaPath, novelPath string) (*Handler, error) {
//
// lanes is the administrative page's window onto the running Poller; nil means
// nothing is polling, which the page reports rather than hides.
func New(s *store.Store, discord DiscordConfig, tokenKey []byte, mangaPath, novelPath string, lanes LaneReporter) (*Handler, error) {
tmpl, err := template.ParseFS(templateFS, "templates/*.html")
if err != nil {
return nil, err
@@ -126,6 +127,7 @@ func New(s *store.Store, discord DiscordConfig, tokenKey []byte, mangaPath, nove
states: newOAuthStates(),
limiter: session.NewLoginLimiter(),
httpClient: &http.Client{Timeout: discordTimeout},
lanes: lanes,
}, nil
}
@@ -149,9 +151,11 @@ func (h *Handler) Register(mux *http.ServeMux) {
mux.HandleFunc("GET /install/manga-bookmark.user.js", h.requireSession(h.installUserscript("manga-bookmark.user.js")))
mux.HandleFunc("GET /install/novel-bookmark.user.js", h.requireSession(h.installUserscript("novel-bookmark.user.js")))
mux.HandleFunc("POST /rotate-token", h.requireSession(h.rotateToken))
// Owner-only: the one place the UI crosses the Reader boundary.
mux.HandleFunc("POST /readers/{id}/revoke", h.requireSession(h.revokeReaderSessions))
// Owner-only: every route that reaches past the acting Reader is gated in
// one place, so a missing gate is visible in the route list.
for _, rt := range h.adminRoutes() {
mux.HandleFunc(rt.pattern, h.requireOwner(rt.handler))
}
}
// staticHandler serves the embedded assets. An hour, not longer: assets are
@@ -240,14 +244,9 @@ func (h *Handler) index(w http.ResponseWriter, r *http.Request) {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if readerID == h.store.OwnerID() {
view.Owner, view.OwnerID = true, readerID
if view.Readers, err = h.store.Readers(); err != nil {
log.Printf("index readers: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
}
// The owner's page differs only by the link to the administrative page:
// the roster lives there now, so the page read every day is only reading.
view.Owner = readerID == h.store.OwnerID()
h.render(w, http.StatusOK, "app", view)
}
@@ -618,40 +617,3 @@ func (h *Handler) rotateToken(w http.ResponseWriter, r *http.Request) {
view := listView{Lib: store.KindManga, Rotated: true}
h.render(w, http.StatusOK, "setup", view)
}
// revokeReaderSessions logs one Reader out of every browser they are signed
// in on. Owner-only: it reaches across the Reader boundary every other handler
// respects, so the guard is a comparison against the seeded owner rather than
// a role a Reader could acquire. A non-owner gets 404 — the panel does not
// exist for them, so neither should the endpoint.
func (h *Handler) revokeReaderSessions(w http.ResponseWriter, r *http.Request) {
if readerOf(r) != h.store.OwnerID() {
http.NotFound(w, r)
return
}
target, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
http.Error(w, "bad reader id", http.StatusBadRequest)
return
}
// The owner is not one of the Readers this endpoint reaches: revoking
// themselves would sign out the browser making the request, which is what
// logout is for. The roster hides the button; this refuses the hand-rolled
// POST behind it.
if target == h.store.OwnerID() {
http.NotFound(w, r)
return
}
if err := h.store.DeleteReaderSessions(target); err != nil {
log.Printf("revoke sessions: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
readers, err := h.store.Readers()
if err != nil {
log.Printf("revoke sessions: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
h.render(w, http.StatusOK, "readers", listView{Owner: true, Readers: readers, OwnerID: h.store.OwnerID()})
}
+31 -95
View File
@@ -7,7 +7,6 @@ import (
"net/http"
"os"
"os/signal"
"strconv"
"strings"
"syscall"
"time"
@@ -61,30 +60,15 @@ type Config struct {
// LatestPoll configures the background latest-chapter poller.
//
// Sizing: batch x (cooldown / interval) is how many series hold a true cooldown
// cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this
// deployment. Past that nothing breaks; the effective cadence stretches to
// N x interval / batch and the oldest-checked-first ordering keeps it uniform.
// Only the kill switch lives here. Pace is per Site — rest time and gap are
// registry properties (internal/latest/sites.go, issue #100), because each
// Lane has to be able to differ from the others. The five environment
// settings that used to size a shared pace (cooldown, browser cooldown,
// interval, stagger, batch) are gone with it: no deployed .env may carry them.
type LatestPoll struct {
Enabled bool
Cooldown time.Duration
BrowserCooldown time.Duration
Interval time.Duration
Stagger time.Duration
Batch int
Enabled bool
}
const (
defaultPollCooldown = time.Hour
defaultBrowserPollCooldown = 6 * time.Hour
defaultPollInterval = 10 * time.Minute
defaultPollStagger = 20 * time.Second
defaultPollBatch = 14
// minPollCooldown keeps a typo from turning a polite background check into
// a hammer against sites that are already bot-scoring us.
minPollCooldown = 15 * time.Minute
)
func envOr(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
@@ -107,66 +91,11 @@ func envBool(key string, def bool) bool {
}
}
// envDuration reads a duration env var. An unparseable or non-positive value
// falls back to def and logs rather than failing startup: the poller is an
// enhancement, and a typo in one of its knobs must not stop bookmark sync.
func envDuration(key string, def time.Duration) time.Duration {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
d, err := time.ParseDuration(raw)
if err != nil || d <= 0 {
log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def)
return def
}
return d
}
// envInt reads a positive integer env var, with the same fallback policy.
func envInt(key string, def int) int {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
n, err := strconv.Atoi(raw)
if err != nil || n <= 0 {
log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def)
return def
}
return n
}
func clampPollCooldown(name string, d time.Duration) time.Duration {
if d < minPollCooldown {
log.Printf("config: %s %s is below the %s floor, clamping", name, d, minPollCooldown)
return minPollCooldown
}
return d
}
// loadLatestPoll reads the poller's settings, clamping anything that would make
// it antisocial.
// loadLatestPoll reads the poller's settings. The pace knobs that used to be
// clamped here are registry properties now (issue #100), so there is nothing
// left to clamp.
func loadLatestPoll() LatestPoll {
p := LatestPoll{
Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true),
Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown),
BrowserCooldown: envDuration("LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", defaultBrowserPollCooldown),
Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval),
Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger),
Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch),
}
p.Cooldown = clampPollCooldown("cooldown", p.Cooldown)
p.BrowserCooldown = clampPollCooldown("browser cooldown", p.BrowserCooldown)
// batch x stagger has to fit inside one tick or a batch is still running
// when the next one is due. Run() serialises them, so this degrades to a
// slower cadence rather than to overlapping fetches — worth a warning, not
// a failure.
if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval {
log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick",
p.Batch, p.Stagger, span, p.Interval)
}
return p
return LatestPoll{Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true)}
}
func loadConfig() Config {
@@ -200,7 +129,10 @@ func loadConfig() Config {
// newRouter wires routes and middleware. CORS is the outermost layer so
// preflight OPTIONS short-circuits before auth; /bookmarks* is auth-protected,
// /healthz is public.
func newRouter(s *store.Store, cfg Config) http.Handler {
//
// lanes may be nil — polling disabled, or its client could not be built. The
// admin page reports that rather than pretending Lanes exist.
func newRouter(s *store.Store, cfg Config, lanes web.LaneReporter) http.Handler {
mux := http.NewServeMux()
h := &api.Handler{Store: s}
mux.HandleFunc("GET /healthz", api.Healthz)
@@ -231,7 +163,7 @@ func newRouter(s *store.Store, cfg Config) http.Handler {
// The browser UI is always registered; signing in is Discord OAuth, so
// there is no password to forget and no gate to leave unset.
wh, err := web.New(s, cfg.Discord, []byte(cfg.TokenKey),
cfg.UserscriptPath, cfg.NovelUserscriptPath)
cfg.UserscriptPath, cfg.NovelUserscriptPath, lanes)
if err != nil {
log.Fatalf("web handler: %v", err)
}
@@ -346,11 +278,17 @@ func main() {
}
s.OnSeriesCreated = acq.Acquire
}
startLatestPoller(pollCtx, s, cfg.LatestPoll, browser)
// A nil *Poller must not become a non-nil interface holding a nil pointer:
// the admin page tests the reporter for nil to decide whether anything is
// polling at all.
var lanes web.LaneReporter
if poller := startLatestPoller(pollCtx, s, cfg.LatestPoll, browser); poller != nil {
lanes = poller
}
srv := &http.Server{
Addr: ":" + cfg.Port,
Handler: newRouter(s, cfg),
Handler: newRouter(s, cfg, lanes),
ReadHeaderTimeout: 10 * time.Second,
}
@@ -377,7 +315,8 @@ func main() {
}
}
// newLatestPoller wires the configured cooldowns and fetchers into the poller.
// newLatestPoller wires the fetcher seams into the poller. Pace is registry
// property, not config (issue #100), so there are no knobs to pass through.
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller {
var covers latest.BrowserCoverFetcher
if f, ok := browser.(latest.BrowserCoverFetcher); ok {
@@ -390,27 +329,23 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
CoverFetch: covers,
CoverBytesFetch: latest.NewCoverFetcher(),
Now: time.Now,
Cooldown: cfg.Cooldown,
BrowserCooldown: cfg.BrowserCooldown,
Interval: cfg.Interval,
Stagger: cfg.Stagger,
Batch: cfg.Batch,
}
}
// startLatestPoller launches the background poller unless it is disabled or its
// HTTP client cannot be built. Any problem here is logged and skipped: this
// feature going missing degrades the service to userscript-only latest-chapter
// tracking, which is exactly how it behaved before.
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher) {
// tracking, which is exactly how it behaved before. It returns the running
// Poller, or nil when there is none — the admin page's Lane status reads it.
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher) *latest.Poller {
if !cfg.Enabled {
log.Println("latest-chapter poller: disabled by config")
return
return nil
}
f, err := latest.NewTLSFetcher()
if err != nil {
log.Printf("latest-chapter poller: disabled, cannot build client: %v", err)
return
return nil
}
// Nil browser: sites behind a JavaScript challenge are simply not polled,
// and their latest_chapter comes from the userscript alone — which is how
@@ -418,4 +353,5 @@ func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, brow
p := newLatestPoller(s, cfg, f, browser)
go p.Run(ctx)
return p
}
+21 -117
View File
@@ -9,31 +9,15 @@ import (
"net/http/httptest"
"strings"
"testing"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store"
)
func TestLoadLatestPollDefaults(t *testing.T) {
for _, k := range []string{
"LATEST_CHAPTER_POLL_ENABLED", "LATEST_CHAPTER_POLL_COOLDOWN",
"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", "LATEST_CHAPTER_POLL_INTERVAL",
"LATEST_CHAPTER_POLL_STAGGER", "LATEST_CHAPTER_POLL_BATCH",
} {
t.Setenv(k, "")
}
got := loadLatestPoll()
want := LatestPoll{
Enabled: true,
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
Interval: 10 * time.Minute,
Stagger: 20 * time.Second,
Batch: 14,
}
if got != want {
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, want)
t.Setenv("LATEST_CHAPTER_POLL_ENABLED", "")
if got := loadLatestPoll(); got != (LatestPoll{Enabled: true}) {
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, LatestPoll{Enabled: true})
}
}
@@ -63,105 +47,25 @@ func TestLoadLatestPollEnabledParsing(t *testing.T) {
}
}
func TestLoadLatestPollClampsAndFallsBack(t *testing.T) {
tests := []struct {
name string
env map[string]string
wantFrom func(LatestPoll) any
want any
}{
{
name: "cooldown below the floor is clamped up",
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "1m"},
wantFrom: func(p LatestPoll) any { return p.Cooldown },
want: 15 * time.Minute,
},
{
name: "cooldown at the floor is kept",
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "15m"},
wantFrom: func(p LatestPoll) any { return p.Cooldown },
want: 15 * time.Minute,
},
{
name: "browser cooldown below the floor is clamped up",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "1m"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 15 * time.Minute,
},
{
name: "browser cooldown at the floor is kept",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "15m"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 15 * time.Minute,
},
{
name: "browser cooldown override is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "8h"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 8 * time.Hour,
},
{
name: "browser cooldown unparseable value falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "six hours"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 6 * time.Hour,
},
{
name: "a valid override is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "5m"},
wantFrom: func(p LatestPoll) any { return p.Interval },
want: 5 * time.Minute,
},
{
name: "an unparseable duration falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "ten minutes"},
wantFrom: func(p LatestPoll) any { return p.Interval },
want: 10 * time.Minute,
},
{
name: "a zero duration falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_STAGGER": "0s"},
wantFrom: func(p LatestPoll) any { return p.Stagger },
want: 20 * time.Second,
},
{
name: "a valid batch is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "30"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 30,
},
{
name: "a negative batch falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "-5"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 14,
},
{
name: "a non-numeric batch falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "lots"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 14,
},
// newLatestPoller wires the fetcher seams; pace lives in the registry, so
// nothing here sizes a cooldown any more.
func TestNewLatestPollerWiresFetchers(t *testing.T) {
tls := &latest.TLSFetcher{}
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil)
if p.Fetch != tls {
t.Fatalf("Fetch not wired")
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
for k, v := range tt.env {
t.Setenv(k, v)
}
if got := tt.wantFrom(loadLatestPoll()); got != tt.want {
t.Fatalf("got %v, want %v", got, tt.want)
}
})
if p.BrowserFetch != nil {
t.Fatalf("BrowserFetch = %v, want nil for a browser-less deployment", p.BrowserFetch)
}
}
func TestNewLatestPollerWiresCooldowns(t *testing.T) {
p := newLatestPoller(nil, LatestPoll{
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
}, nil, nil)
if p.Cooldown != time.Hour || p.BrowserCooldown != 6*time.Hour {
t.Fatalf("poller cooldowns = %s/%s, want 1h/6h", p.Cooldown, p.BrowserCooldown)
if p.CoverFetch != nil {
t.Fatalf("CoverFetch = %v, want nil when the browser is absent", p.CoverFetch)
}
if p.CoverBytesFetch == nil {
t.Fatalf("CoverBytesFetch = nil, want the TLS cover fetcher")
}
if p.Now == nil {
t.Fatalf("Now = nil, want the live clock")
}
}
+3 -3
View File
@@ -49,7 +49,7 @@ func withBody(req *http.Request, body string) *http.Request {
// A refused credential is refused however plausible it looks: only a hash the
// readers table holds authenticates anything.
func TestUnknownCredentialRejected(t *testing.T) {
srv := newRouter(newTestStore(t), testConfig())
srv := newRouter(newTestStore(t), testConfig(), nil)
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, credRequest(http.MethodGet, "/bookmarks", readerCredential("never-registered")))
@@ -69,7 +69,7 @@ func TestUnknownCredentialRejected(t *testing.T) {
func TestPerReaderIsolation(t *testing.T) {
s := newTestStore(t)
registerReader(t, s, "other-reader")
srv := newRouter(s, testConfig())
srv := newRouter(s, testConfig(), nil)
ownerKey := "asura:solo"
putBookmark(t, srv, ownerKey, store.Bookmark{
@@ -267,7 +267,7 @@ func TestRotateCredentialViaWebUI(t *testing.T) {
}
cfg := testConfig()
cfg.UserscriptPath = path
srv := newRouter(s, cfg)
srv := newRouter(s, cfg, nil)
oldCred := ownerCredential()
rr := httptest.NewRecorder()
+246 -16
View File
@@ -1,6 +1,7 @@
package main
import (
"database/sql"
"encoding/json"
"fmt"
"io"
@@ -15,6 +16,7 @@ import (
"testing"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/session"
"bookmarkmanager/backend/internal/store"
"bookmarkmanager/backend/internal/web"
@@ -26,11 +28,17 @@ import (
const testOwnerID = "owner-snowflake"
// newWebTestServer returns the full router plus the store behind it, so tests
// can seed rows and assert on what the handlers wrote back.
func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) {
// can seed rows and assert on what the handlers wrote back. An optional lane
// reporter stands in for the running poller; omitted means none is running,
// which is what every test that is not about the admin page wants.
func newWebTestServer(t *testing.T, cfg Config, lanes ...web.LaneReporter) (http.Handler, *store.Store) {
t.Helper()
st := newTestStore(t)
return newRouter(st, cfg), st
var reporter web.LaneReporter
if len(lanes) > 0 {
reporter = lanes[0]
}
return newRouter(st, cfg, reporter), st
}
// sessionCookie mints a live session row for the owner and returns the cookie
@@ -141,12 +149,12 @@ func discordConfig(stubURL string) web.DiscordConfig {
// oauthWebTestServer returns the full router, its store, and a Discord stub
// wired as the configured API — the starting point for sign-in tests.
func oauthWebTestServer(t *testing.T) (http.Handler, *store.Store, *discordStub) {
func oauthWebTestServer(t *testing.T, lanes ...web.LaneReporter) (http.Handler, *store.Store, *discordStub) {
t.Helper()
stub, srv := newDiscordStub(t)
cfg := testConfig()
cfg.Discord = discordConfig(srv.URL)
router, st := newWebTestServer(t, cfg)
router, st := newWebTestServer(t, cfg, lanes...)
return router, st, stub
}
@@ -410,7 +418,7 @@ func TestDiscordLoginRefusesNonMember(t *testing.T) {
cfg.Discord = discordConfig(srv.URL)
cfg.Discord.RequiredRole = tc.require
st := newTestStore(t)
router := newRouter(st, cfg)
router := newRouter(st, cfg, nil)
rr := completeSignIn(t, router, startSignIn(t, router))
if rr.Code != http.StatusForbidden {
@@ -626,24 +634,52 @@ func TestOwnerRevokesAnotherReadersSessions(t *testing.T) {
}
}
// The owner's own page carries the roster; nobody else's does.
func TestOwnerSeesReadersPanel(t *testing.T) {
// fakeLanes is the admin page's poller stand-in: one fixed snapshot, so the
// page's tests need neither a poller nor a Site.
type fakeLanes struct{ status latest.Status }
func (f fakeLanes) LaneStatus() latest.Status { return f.status }
// The roster moved off the reading page onto its own address: the owner gets a
// link, everyone else gets nothing, and the page itself lists every Reader with
// the counters and the two controls.
func TestAdminPageCarriesRosterAndOwnerLink(t *testing.T) {
router, st, _ := oauthWebTestServer(t)
signInCookie(t, router)
theirCookie := signInCookie(t, router)
ownerCookie := sessionCookie(t, st)
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(sessionCookie(t, st))
req.AddCookie(ownerCookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
body := rr.Body.String()
if !strings.Contains(body, `id="readers"`) {
t.Fatal("the owner's page lacks the Readers panel")
if strings.Contains(body, `id="readers"`) {
t.Error("the reading page still carries the roster; it belongs on /admin")
}
if !strings.Contains(body, testOwnerID) {
t.Fatalf("the roster does not list the registered Reader:\n%s", body)
if !strings.Contains(body, `href="/admin"`) {
t.Error("the owner's reading page offers no link to the admin page")
}
if !strings.Contains(body, "Revoke sessions") {
t.Fatal("the roster offers no revocation control for a signed-in Reader")
req = httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(theirCookie)
rr = httptest.NewRecorder()
router.ServeHTTP(rr, req)
if strings.Contains(rr.Body.String(), `href="/admin"`) {
t.Error("a non-owner was offered the admin link")
}
req = httptest.NewRequest(http.MethodGet, "/admin", nil)
req.AddCookie(ownerCookie)
rr = httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("GET /admin status = %d, want 200", rr.Code)
}
body = rr.Body.String()
for _, want := range []string{`id="readers"`, testOwnerID, "Revoke sessions", "Clear marks", "confirmed"} {
if !strings.Contains(body, want) {
t.Errorf("admin page lacks %q:\n%s", want, body)
}
}
// Exactly one revocable row: the other Reader's. The owner's own row carries
// the same session count and no button.
@@ -652,6 +688,200 @@ func TestOwnerSeesReadersPanel(t *testing.T) {
}
}
// Every administrative route is gated the same way, so the test walks the list
// the router registers rather than naming routes by hand: no session is 401,
// a signed-in non-owner is 404, and the address is not confirmed to either.
func TestAdminRoutesAreOwnerOnly(t *testing.T) {
router, st, _ := oauthWebTestServer(t)
theirCookie := signInCookie(t, router)
ownerCookie := sessionCookie(t, st)
target := strconv.FormatInt(st.OwnerID(), 10)
patterns := web.AdminPatterns()
if len(patterns) == 0 {
t.Fatal("no administrative routes to test")
}
for _, pattern := range patterns {
method, path, ok := strings.Cut(pattern, " ")
if !ok {
t.Fatalf("route pattern %q has no method", pattern)
}
path = strings.Replace(path, "{id}", target, 1)
for _, tc := range []struct {
name string
cookie *http.Cookie
want int
}{
{"no session", nil, http.StatusUnauthorized},
{"non-owner", theirCookie, http.StatusNotFound},
} {
req := httptest.NewRequest(method, path, nil)
if tc.cookie != nil {
req.AddCookie(tc.cookie)
}
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != tc.want {
t.Errorf("%s %s as %s: status = %d, want %d", method, path, tc.name, rr.Code, tc.want)
}
}
req := httptest.NewRequest(method, path, nil)
req.AddCookie(ownerCookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code == http.StatusUnauthorized {
t.Errorf("%s %s as the owner: status = 401, the gate rejects the owner", method, path)
}
}
}
// The Lane block reports what the poller says, and marks the Lanes that need
// attention — a clamped gap, a refusal, a Site whose pages can only be read
// through a sidecar that is not there, and a Lane with Series waiting that its
// last pass did not read.
func TestAdminPageShowsLaneStatus(t *testing.T) {
lanes := fakeLanes{latest.Status{
Lanes: []latest.LaneState{
{Site: "asura", Due: 12, Checked: 12, LastRun: time.Now().Add(-90 * time.Second), Gap: 40 * time.Second},
{Site: "kagane", Due: 3, Checked: 3, LastRun: time.Now().Add(-time.Minute), Gap: time.Minute, Browser: true},
{Site: "demonic", Due: 400, Checked: 400, LastRun: time.Now(), Gap: 8 * time.Second, Clamped: true},
{Site: "comix", Due: 7, LastRun: time.Now(), Gap: time.Minute, Browser: true},
},
BrowserConfigured: true,
BrowserReachable: true,
}}
router, st, _ := oauthWebTestServer(t, lanes)
req := httptest.NewRequest(http.MethodGet, "/ui/admin/lanes", nil)
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("GET /ui/admin/lanes status = %d, want 200", rr.Code)
}
body := rr.Body.String()
for _, want := range []string{"asura", "kagane", "12 due", "12 checked", "gap 40s", "ran 1m30s ago", "gap at floor", "not checking", "reachable"} {
if !strings.Contains(body, want) {
t.Errorf("lane status lacks %q:\n%s", want, body)
}
}
// Nothing is refusing and the sidecar is up, so neither mark may appear:
// a mark the owner cannot act on is worse than none.
for _, unwanted := range []string{"refusing", "no browser"} {
if strings.Contains(body, unwanted) {
t.Errorf("lane status marks %q on a healthy run:\n%s", unwanted, body)
}
}
}
// A Lane whose pass never reached a figure must not have that figure drawn as
// a zero: a refusing Lane still reports the due count and gap its last real
// pass saw, and a Lane that has never reached one omits it entirely.
func TestLaneStatusOmitsUnknownGap(t *testing.T) {
lanes := fakeLanes{latest.Status{
Lanes: []latest.LaneState{{Site: "comix", LastRun: time.Now(), Refusing: true, Browser: true}},
BrowserConfigured: true,
BrowserReachable: true,
}}
router, st, _ := oauthWebTestServer(t, lanes)
req := httptest.NewRequest(http.MethodGet, "/ui/admin/lanes", nil)
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
body := rr.Body.String()
if strings.Contains(body, "gap 0s") {
t.Errorf("a Lane with no pace yet states a zero gap:\n%s", body)
}
if !strings.Contains(body, "refusing") {
t.Errorf("a refusing Lane is not marked as such:\n%s", body)
}
}
// No poller and a poller that has not finished a pass both render "no data
// yet" rather than zeroes that read as a stopped backend — but they are not
// the same fact, so the page must not blame the sidecar when nothing polls.
func TestAdminPageWithoutAPollerSaysSo(t *testing.T) {
for _, tc := range []struct {
name string
lanes []web.LaneReporter
want, unwant string
}{
{"no poller", nil, "Polling is switched off", "not configured"},
{"poller, no pass yet", []web.LaneReporter{fakeLanes{}}, "not configured", "Polling is switched off"},
} {
t.Run(tc.name, func(t *testing.T) {
router, st, _ := oauthWebTestServer(t, tc.lanes...)
req := httptest.NewRequest(http.MethodGet, "/admin", nil)
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
body := rr.Body.String()
if !strings.Contains(body, "No data yet") {
t.Errorf("admin page with no Lane data does not say so:\n%s", body)
}
if !strings.Contains(body, tc.want) {
t.Errorf("admin page lacks %q:\n%s", tc.want, body)
}
if strings.Contains(body, tc.unwant) {
t.Errorf("admin page states %q, which is not what is wrong:\n%s", tc.unwant, body)
}
})
}
}
// A Reader past the disagreement threshold is rendered as blocked, and
// clearing their marks both zeroes the counters and lifts the block in the
// roster the response carries back.
func TestOwnerClearsReaderMarks(t *testing.T) {
st, dsn := newTestStoreURL(t)
router := newRouter(st, testConfig(), nil)
cookie := sessionCookie(t, st)
// The counters are filled by issue #103; until it lands the only way to
// stand a marked Reader up is to write the columns directly.
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
if _, err := db.Exec(`UPDATE readers SET sighting_agreements = 4, sighting_disagreements = 3 WHERE id = $1`, st.OwnerID()); err != nil {
t.Fatalf("mark reader: %v", err)
}
req := httptest.NewRequest(http.MethodGet, "/admin", nil)
req.AddCookie(cookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
body := rr.Body.String()
if !strings.Contains(body, "4 confirmed / 3 contradicted") {
t.Errorf("roster does not report the Reader's marks:\n%s", body)
}
if !strings.Contains(body, "deferral blocked") {
t.Errorf("a Reader at the threshold is not rendered as blocked:\n%s", body)
}
req = httptest.NewRequest(http.MethodPost,
"/readers/"+strconv.FormatInt(st.OwnerID(), 10)+"/clear-marks", nil)
req.AddCookie(cookie)
rr = httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("clear marks: status = %d, want 200 (body %s)", rr.Code, rr.Body.String())
}
body = rr.Body.String()
if !strings.Contains(body, `id="readers"`) {
t.Fatalf("clear marks did not re-render the roster:\n%s", body)
}
if !strings.Contains(body, "0 confirmed / 0 contradicted") {
t.Errorf("roster does not report the cleared counters:\n%s", body)
}
if strings.Contains(body, "deferral blocked") {
t.Errorf("a cleared Reader is still marked blocked:\n%s", body)
}
}
func TestDiscordLoginTokenEndpointDown(t *testing.T) {
stub, srv := newDiscordStub(t)
stub.tokenStatus = http.StatusInternalServerError
+4 -7
View File
@@ -58,14 +58,11 @@ services:
# Second script from the same bindmount; the novel library is a separate
# Violentmonkey install.
NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js}
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill
# switch; it only takes effect because these are listed here.
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the
# kill switch; it only takes effect because it is listed here. Pace is
# per Site in the registry (one Poll Lane per Site, issue #100) — the
# cooldown/interval/stagger/batch knobs are gone with the shared pace.
LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1}
LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h}
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h}
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
# CDP endpoint for sites behind a JavaScript challenge (kagane,
# novelfull). The browser is not part of this stack — it runs on the home
# machine as its own unit (chrome/docker-compose.yml) and is reached over
+91
View File
@@ -0,0 +1,91 @@
# ADR-0010: Poll Lanes — one independent Poll stream per Site
Date: 2026-08-16
Status: accepted
## Decision
Replace the single shared polling pace with one **Poll Lane** per Site: an
independent goroutine that polls only that Site's Series, paced by that Site's
registry entry. Pace moves out of config and into the Site registry
(`internal/latest/sites.go`): every entry carries a `Rest` (how long a Series
rests between Polls) and a `Gap` (how long the Lane waits between fetches).
Rest is enforced by the due query's WHERE clause (`latest_checked_at <= now -
Rest`), never by a timer — the same mechanism that enforced the old cooldown.
The Lane enforces its own gap by sleeping between fetches. `effectiveGap` is
the registry gap, or one hour divided by the Site's eligible Series count when
that is smaller, never below one second.
The five environment settings that used to size the shared pace —
`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
`_STAGGER` — are deleted. Only the kill switch `LATEST_CHAPTER_POLL_ENABLED`
remains. No deployed `.env` may carry the deleted knobs.
## Why
The shared pace capped the whole backend at roughly 180 Polls an hour (one
20-second stagger across one queue). ~60 Series today, scaling to hundreds or
thousands, would stretch the hour beyond what the New Chapter signal can
tolerate. Worse, the queue mixed Sites with very different costs: kagane and
comix pay seconds of a serialized single-tab Chrome per Poll (a challenged
page, ADR-0005), and one hostile Site burning its challenge timeout made every
other Site's Series wait — "one hostile Site can eat most of an hour".
Lanes fix both at once:
- **Throughput scales per Site.** The six Lanes fetch concurrently; a Lane's
own gap paces it. The browser Lanes' combined ceiling stays about 360 Polls
an hour (one tab), and when they cannot keep up the wait past Rest grows and
is logged every pass — the "behind by X" measurement, so the decision to
give browser Sites more pages is made from data.
- **Hostility is contained.** A refusal (two challenge-held reads in one
pass) stops only that Site's Lane for `refuseBackoff` (15m); the rest of
that Lane's Series stay unstamped and due. A lost browser gates the other
browser Lanes' passes for the same window — the flag is shared Poller
state, so the loss is noticed once instead of once per Lane per pass, and
decays after 15m so the Lanes probe again. One Site can no longer tax the
others.
## Tradeoffs and rejections
- **Per-Site env knobs** (e.g. `KAGANE_POLL_GAP`) rejected: the registry is
the single place pace lives, testable and reviewable; config knobs would
recreate the shared-pace sprawl with six times the surface. All six entries
are deliberately uniform at first — rest an hour, gap ten seconds — so the
structure exists to differ without inventing numbers for Sites that have
not earned them.
- **Dynamic gap** (`rest / eligible`) is the one knob that stays automatic:
a Site with more Series than one per ten seconds would otherwise back up
behind its own gap, and the per-Series share of the hour is the natural
pace. The ten-second default is not arbitrary: one request per ten seconds
is the strictest rate rule a free-plan Site can even express (per-zone
rate limiting, as documented in
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`), so the
default pace is exactly what the most restrictive Site would demand of us.
The computed gap never goes below one second and logs loudly when the
floor engages.
- **Timer-based pacing** rejected: the old ticker made the poller's rate a
function of wall clock rather than of what was due. The due-query cutoff is
the only rate authority; the Lane sleep just prevents hammering.
- **Batch size** (the old `_BATCH` cap) is gone with the shared pace: a Lane
processes everything due, paced by its gap. There is no global queue left
to bound.
## Constraints preserved
- Stamp-before-fetch ("attempted" semantics): an untried Series stays due, so
a browser that appears after a restart finds its full queue waiting.
- Browser wake gate (ADR-0005): a browser Lane leaves Chrome asleep below
five due Series and 15 minutes of wait, per Lane.
- The browser is not in the API stack (ADR-0006): an unreachable browser
degrades a Lane exactly as an unset `BROWSER_WS_URL` — browser-only Sites
skipped, plain-TLS unaffected, stored covers still served.
- Cover heals moved to background goroutines (joined by the test suite via
`waitCovers`) so a slow cover CDN cannot consume a Lane's gap.
Supersedes the pace mechanics of ADR-0003's "raise throughput instead" note
(the stagger cut it rejected is what the per-Lane gap replaces) and the
6-hour browser cooldown introduced with the browser-backed Sites; the
1-hour browser rest was already cleared as safe by
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
+17 -4
View File
@@ -10,7 +10,7 @@ Implemented in:
| Surface | Files |
| --- | --- |
| Web UI (login, list, card, empty, errors) | `backend/internal/web/static/style.css`, `backend/internal/web/templates/{app,card,list,login,chrome,icons}.html`, `backend/internal/web/static/filter.js` |
| Web UI (login, list, card, empty, errors, admin) | `backend/internal/web/static/style.css`, `backend/internal/web/templates/{app,admin,lanes,readers,card,list,login,chrome,icons}.html`, `backend/internal/web/static/filter.js` |
| Userscript panel (Shadow DOM) | `userscript/manga-bookmark.user.js` — `TEMPLATE` and `CSS` at the bottom of the IIFE |
## 1. The one idea
@@ -73,6 +73,7 @@ Defined once in `backend/internal/web/static/style.css` `:root`, mirrored in the
| `--moss` | `#7fae86` | `#3d6c46` | finished accent |
| `--clay` | `#b5906f` | `#7c5533` | set-chapter accent |
| `--trash` | `#977671` | `#8c6558` | remove, at rest — icons need 3:1, not 4.5:1 |
| `--patina` | `#5fb3a6` | `#1f6f66` | admin page only — a Poll Lane needing attention, a Reader whose reports are blocked |
| `--play-hot-line` | `#3a1d18` | `#f0cfc6` | desktop cell border, play when `.is-new` |
| `--fav-line` | `#332b14` | `#e3d3a4` | desktop cell border, favourite when on |
| `--asura` | `#7d93a5` | `#4f6b80` | site tag |
@@ -81,9 +82,13 @@ Defined once in `backend/internal/web/static/style.css` `:root`, mirrored in the
| `--kagane` | `#9a8aa5` | `#6f5f7d` | site tag |
| `--hatch` / `--hatch-dim` | 135° 5px stripe | paper stripe | missing-cover slot |
`--slate`/`--moss`/`--clay`/`--brass` are held at the same weight deliberately:
one accent per action, so a press says which lane it belongs to, with none of
them competing with ember. Dark is the default (`color-scheme: dark light`);
`--slate`/`--moss`/`--clay`/`--brass`/`--patina` are held at the same weight
deliberately: one accent per meaning, so a press says which lane it belongs to,
with none of them competing with ember. `--patina` is the admin page's only
colour — a cool verdigris, the far side of the wheel from ember's crimson and
clear of the archive blue: system health is neither a new chapter nor
destruction, so it borrows neither `--ember` nor `--danger`.
Dark is the default (`color-scheme: dark light`);
light is a `@media (prefers-color-scheme: light)` override of the same names.
**Any new colour must be added in both branches** — light is not a filter over
dark, the hues are re-tuned.
@@ -140,6 +145,14 @@ Recurring specs (copy these rather than inventing sizes):
main#list article.card … | .empty
```
The owner's admin page (`admin.html`) is the same sheet with two sections in
place of the list — `.lanes` (Poll Lane rows) and `.readers` (the roster) —
and no library switch: it belongs to neither library, so its topbar carries a
plain `.ghost.back` link home. Both sections are eyebrow + hairline-separated
rows, the shape the roster already had as a fold-out. `.lanes` refreshes itself
every 30s via `hx-get="/ui/admin/lanes"` with `hx-swap="outerHTML"`; the roster
re-renders only in answer to an action.
**Brand mark**: an inline `<svg class="mark">` (`viewBox="0 0 200 172"`),
defined once in `chrome.html`'s `mark` template and reused by `app.html` and
`login.html` so it takes the page's `--ink`/`currentColor`/`--ember` rather
+5 -7
View File
@@ -30,7 +30,8 @@
"28": "Allocation Patterns",
"29": "Observability & Alerting",
"30": "AGENTS.md",
"32": "Repo Hard Constraints",
"31": "Store",
"32": "Open",
"33": "Go Testing Guide",
"34": "Session Store",
"35": "Web UI Filter Logic",
@@ -39,7 +40,7 @@
"38": "novel-logic.test.js",
"39": "UI Critique 2026-07-26A",
"40": "UI Critique 2026-07-26B",
"41": "sessions_test.go",
"42": "pgtest.go",
"43": "Issue Tracker & Triage",
"44": "Ticket Workflow",
"45": "Go Perf Alert Rules",
@@ -122,6 +123,7 @@
"122": "Identity comes from Discord OAuth; we store no passwords and send no email",
"123": "The wire format stays flat and deliberately does not mirror the schema",
"124": "ADR-0005: On-demand browser sidecar",
"125": "sessions_test.go",
"126": "Bookmark Manager",
"127": "triage-labels.md",
"128": "Cross-Ticket Contract",
@@ -149,7 +151,7 @@
"150": "Reviewer Subagent (opencode)",
"151": "Finding Severity Rubric",
"152": "Spec Compliance Review",
"158": "T",
"156": "ResponseWriter",
"161": "Why Use samber/oops",
"162": "singleflight Cache Stampede Prevention",
"163": "Struct Field Alignment",
@@ -204,10 +206,6 @@
"245": "wayfinder map/ticket mechanism",
"246": "Triage labels: canonical roles to tracker labels",
"247": "Canonical triage role labels (needs-triage ... wontfix)",
"248": "Cinder (BookmarkManager Web UI design system)",
"249": "Heat is typographic: ember reserved for unread chapters",
"250": "Design tokens (dark + light branches, no hardcoded hex)",
"251": "Three type roles: display serif / mono small-caps / sans",
"252": "a[aria-label='All Chapter'] priority pointer",
"253": "Research: lightnovelworld chapter slug vs series slug",
"254": "Gitea issue #77 (chapter vs series slug)",
+73 -69
View File
@@ -1,16 +1,16 @@
# Graph Report - mangaBookmark (2026-08-16)
## Corpus Check
- 112 files · ~274,851 words
- 116 files · ~284,419 words
- Verdict: corpus is large enough that graph structure adds value.
## Summary
- 1640 nodes · 3294 edges · 221 communities (67 shown, 154 thin omitted)
- Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 313 edges (avg confidence: 0.77)
- 1701 nodes · 3463 edges · 219 communities (69 shown, 150 thin omitted)
- Extraction: 91% EXTRACTED · 9% INFERRED · 0% AMBIGUOUS · INFERRED: 326 edges (avg confidence: 0.77)
- Token cost: 0 input · 0 output
## Graph Freshness
- Built from commit: `f000cc7e`
- Built from commit: `58014eb8`
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
- Run `graphify update .` after code changes (no API cost).
@@ -45,7 +45,8 @@
- [[_COMMUNITY_Find Skills Guide|Find Skills Guide]]
- [[_COMMUNITY_Allocation Patterns|Allocation Patterns]]
- [[_COMMUNITY_Observability & Alerting|Observability & Alerting]]
- [[_COMMUNITY_Repo Hard Constraints|Repo Hard Constraints]]
- [[_COMMUNITY_AGENTS|AGENTS.md]]
- [[_COMMUNITY_Store|Store]]
- [[_COMMUNITY_Go Testing Guide|Go Testing Guide]]
- [[_COMMUNITY_Session Store|Session Store]]
- [[_COMMUNITY_Web UI Filter Logic|Web UI Filter Logic]]
@@ -54,7 +55,7 @@
- [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]]
- [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]]
- [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]]
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
- [[_COMMUNITY_pgtest.go|pgtest.go]]
- [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]]
- [[_COMMUNITY_Ticket Workflow|Ticket Workflow]]
- [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]]
@@ -137,6 +138,7 @@
- [[_COMMUNITY_Identity comes from Discord OAuth; we store no passwords and send no email|Identity comes from Discord OAuth; we store no passwords and send no email]]
- [[_COMMUNITY_The wire format stays flat and deliberately does not mirror the schema|The wire format stays flat and deliberately does not mirror the schema]]
- [[_COMMUNITY_ADR-0005 On-demand browser sidecar|ADR-0005: On-demand browser sidecar]]
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
- [[_COMMUNITY_Bookmark Manager|Bookmark Manager]]
- [[_COMMUNITY_triage-labels|triage-labels.md]]
- [[_COMMUNITY_Cross-Ticket Contract|Cross-Ticket Contract]]
@@ -164,7 +166,7 @@
- [[_COMMUNITY_Reviewer Subagent (opencode)|Reviewer Subagent (opencode)]]
- [[_COMMUNITY_Finding Severity Rubric|Finding Severity Rubric]]
- [[_COMMUNITY_Spec Compliance Review|Spec Compliance Review]]
- [[_COMMUNITY_T|T]]
- [[_COMMUNITY_ResponseWriter|ResponseWriter]]
- [[_COMMUNITY_Why Use samberoops|Why Use samber/oops]]
- [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]]
- [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]]
@@ -219,10 +221,6 @@
- [[_COMMUNITY_wayfinder mapticket mechanism|wayfinder map/ticket mechanism]]
- [[_COMMUNITY_Triage labels canonical roles to tracker labels|Triage labels: canonical roles to tracker labels]]
- [[_COMMUNITY_Canonical triage role labels (needs-triage ... wontfix)|Canonical triage role labels (needs-triage ... wontfix)]]
- [[_COMMUNITY_Cinder (BookmarkManager Web UI design system)|Cinder (BookmarkManager Web UI design system)]]
- [[_COMMUNITY_Heat is typographic ember reserved for unread chapters|Heat is typographic: ember reserved for unread chapters]]
- [[_COMMUNITY_Design tokens (dark + light branches, no hardcoded hex)|Design tokens (dark + light branches, no hardcoded hex)]]
- [[_COMMUNITY_Three type roles display serif mono small-caps sans|Three type roles: display serif / mono small-caps / sans]]
- [[_COMMUNITY_aaria-label='All Chapter' priority pointer|a[aria-label='All Chapter'] priority pointer]]
- [[_COMMUNITY_Research lightnovelworld chapter slug vs series slug|Research: lightnovelworld chapter slug vs series slug]]
- [[_COMMUNITY_Gitea issue 77 (chapter vs series slug)|Gitea issue #77 (chapter vs series slug)]]
@@ -237,16 +235,16 @@
- [[_COMMUNITY_Userscript CLAUDE guidance|Userscript CLAUDE guidance]]
## God Nodes (most connected - your core abstractions)
1. `testConfig()` - 53 edges
2. `newWebTestServer()` - 49 edges
3. `newTestStore()` - 43 edges
4. `newTestStore()` - 42 edges
1. `testConfig()` - 54 edges
2. `newTestStore()` - 49 edges
3. `newWebTestServer()` - 49 edges
4. `newTestStore()` - 44 edges
5. `e()` - 33 edges
6. `Handler` - 29 edges
7. `ne()` - 28 edges
8. `De()` - 28 edges
9. `Open()` - 27 edges
10. `se()` - 27 edges
6. `Open()` - 30 edges
7. `Handler` - 29 edges
8. `Store` - 28 edges
9. `ne()` - 28 edges
10. `De()` - 28 edges
## Surprising Connections (you probably didn't know these)
- `el()` --indirect_call--> `c()` [INFERRED]
@@ -266,48 +264,46 @@
## Hyperedges (group relationships)
- **Batch Ticket Implementation Pipeline** — _claude_skills_implement_tickets_skill_implement_tickets, _omp_agents_ticket_implementer_ticket_implementer, _omp_agents_ticket_implementer_cr_spec, _omp_agents_ticket_implementer_cr_standards [INFERRED 0.85]
- **Subagent-Driven Development Pipeline** — _opencode_agent_implementer_implementer, _opencode_agent_reviewer_reviewer, _opencode_agent_implementer_subagent_driven_development [INFERRED 0.85]
- **Go HTML Template Family** — backend_internal_web_templates_app_doc, backend_internal_web_templates_card_doc, backend_internal_web_templates_list_doc, backend_internal_web_templates_chrome_doc, backend_internal_web_templates_login_doc, backend_internal_web_templates_setup_doc, backend_internal_web_templates_readers_doc, backend_internal_web_templates_icons_doc [INFERRED 0.95]
- **htmx Fragment Swap Flow** — backend_internal_web_templates_app_doc, backend_internal_web_templates_card_doc, backend_internal_web_templates_chrome_doc, backend_internal_web_templates_setup_doc, backend_internal_web_templates_readers_doc [INFERRED 0.95]
- **Backend owns the truth (single-writer ownership of shared facts)** — docs_adr_0003_series_shared_and_poll_owned_poll_owned_writes, docs_adr_0004_wire_format_does_not_mirror_the_schema_flat_wire_contract, docs_adr_0007_backend_hosts_cover_bytes_server_side_covers [INFERRED 0.85]
- **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85]
- **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85]
## Communities (221 total, 154 thin omitted)
## Communities (219 total, 150 thin omitted)
### Community 0 - "HTMX Library Internals"
Cohesion: 0.08
Nodes (101): A(), ae(), an(), at(), B(), be(), bn(), bt() (+93 more)
### Community 1 - "Cover Fetch Test Helpers"
Cohesion: 0.10
Nodes (84): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecorder, T, TestListRendersAcquiredCover() (+76 more)
Cohesion: 0.09
Nodes (89): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecorder, T, TestListRendersAcquiredCover() (+81 more)
### Community 2 - "Manga Userscript Adapters"
Cohesion: 0.06
Nodes (77): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+69 more)
Nodes (76): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+68 more)
### Community 3 - "Novel Userscript Adapters"
Cohesion: 0.06
Nodes (78): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+70 more)
Nodes (79): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+71 more)
### Community 4 - "Series Acquisition Tests"
Cohesion: 0.09
Nodes (69): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+61 more)
Cohesion: 0.08
Nodes (78): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+70 more)
### Community 5 - "Bookmarks API Tests"
Cohesion: 0.08
Nodes (67): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+59 more)
Cohesion: 0.07
Nodes (70): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+62 more)
### Community 7 - "Cover & Acquire Internals"
Cohesion: 0.10
Nodes (30): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+22 more)
Cohesion: 0.07
Nodes (43): Addr, fakeLanes, Context, Store, WaitGroup, defaultCoverResolver(), fetchCoverBytes(), Client (+35 more)
### Community 8 - "System Architecture Concepts"
Cohesion: 0.13
Nodes (20): app.html — App Shell Template, Manga/Novel Library Switch, Bookmark Bucket Tabs, Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key (+12 more)
Cohesion: 0.15
Nodes (15): Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key, Brand Mark SVG, Continue Reading Strip, icons.html — Icon Sprite Template (+7 more)
### Community 9 - "Session Middleware"
Cohesion: 0.07
Cohesion: 0.08
Nodes (35): ClearCookie(), ClientIP(), Duration, Mutex, Request, ResponseWriter, Time, isHTTPS() (+27 more)
### Community 10 - "Go Test Helpers"
@@ -315,12 +311,12 @@ Cohesion: 0.05
Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more)
### Community 11 - "Store Tests"
Cohesion: 0.15
Nodes (43): Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck(), seedSecondReader() (+35 more)
Cohesion: 0.14
Nodes (45): Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck(), seedSecondReader() (+37 more)
### Community 12 - "Bookmarks API Handler"
Cohesion: 0.11
Nodes (24): Auth(), compressible(), CORS(), Handler, ResponseWriter, Store, Gzip(), ResolveReader() (+16 more)
Cohesion: 0.08
Nodes (32): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+24 more)
### Community 13 - "Web UI Handlers"
Cohesion: 0.14
@@ -331,8 +327,8 @@ Cohesion: 0.06
Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more)
### Community 15 - "CDP Browser Client"
Cohesion: 0.07
Nodes (36): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+28 more)
Cohesion: 0.20
Nodes (17): applyMigration(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen(), TestMigration0002BackfillsExistingBookmarks() (+9 more)
### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented"
Cohesion: 0.06
@@ -343,8 +339,8 @@ Cohesion: 0.08
Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more)
### Community 19 - "Store"
Cohesion: 0.09
Nodes (8): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, scanSeries(), TestDisplayChapter(), Bookmark, ReaderSummary
Cohesion: 0.33
Nodes (5): ADR-0010: Poll Lanes — one independent Poll stream per Site, Constraints preserved, Decision, Tradeoffs and rejections, Why
### Community 20 - "I/O Performance Patterns"
Cohesion: 0.11
@@ -371,24 +367,28 @@ Cohesion: 0.33
Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more)
### Community 26 - "Open"
Cohesion: 0.20
Nodes (17): applyMigration(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen(), TestMigration0002BackfillsExistingBookmarks() (+9 more)
Cohesion: 0.05
Nodes (62): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+54 more)
### Community 27 - "Find Skills Guide"
Cohesion: 0.14
Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more)
### Community 28 - "Allocation Patterns"
Cohesion: 0.11
Nodes (19): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map of pointers for large, frequently updated structs, Map size hints (+11 more)
Cohesion: 0.14
Nodes (14): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map size hints, Memory Optimization (+6 more)
### Community 29 - "Observability & Alerting"
Cohesion: 0.22
Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more)
### Community 32 - "Repo Hard Constraints"
Cohesion: 0.18
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
### Community 30 - "AGENTS.md"
Cohesion: 0.40
Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct
### Community 31 - "Store"
Cohesion: 0.08
Nodes (8): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, scanSeries(), TestDisplayChapter(), Bookmark, ReaderSummary
### Community 33 - "Go Testing Guide"
Cohesion: 0.20
@@ -418,9 +418,9 @@ Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations,
Cohesion: 0.29
Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider
### Community 41 - "sessions_test.go"
Cohesion: 0.48
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
### Community 42 - "pgtest.go"
Cohesion: 0.18
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
### Community 45 - "Go Perf Alert Rules"
Cohesion: 0.50
@@ -530,9 +530,13 @@ Nodes (3): Consequence, The wire format stays flat and deliberately does not mir
Cohesion: 0.50
Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision
### Community 158 - "T"
Cohesion: 0.08
Nodes (38): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Request, ReaderID() (+30 more)
### Community 125 - "sessions_test.go"
Cohesion: 0.48
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
### Community 156 - "ResponseWriter"
Cohesion: 0.18
Nodes (13): AdminPatterns(), HandlerFunc, Request, ResponseWriter, Time, Handler, readerPathID(), since() (+5 more)
### Community 273 - "AGENTS.md"
Cohesion: 0.50
@@ -541,22 +545,22 @@ Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against
## Knowledge Gaps
- **520 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+515 more)
These have ≤1 connection - possible missing edges or undocumented components.
- **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
- **150 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
## Suggested Questions
_Questions this graph is uniquely positioned to answer:_
- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`, `Open`?**
_High betweenness centrality (0.051) - this node is a cross-community bridge._
- **Why does `Open()` connect `Open` to `Repo Hard Constraints`, `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `Store Tests`, `Store`?**
_High betweenness centrality (0.038) - this node is a cross-community bridge._
- **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?**
_High betweenness centrality (0.030) - this node is a cross-community bridge._
- **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?**
_`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._
- **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?**
_`newWebTestServer()` has 8 INFERRED edges - model-reasoned connections that need verification._
- **Why does `Open()` connect `CDP Browser Client` to `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `pgtest.go`, `Store Tests`, `Store`?**
_High betweenness centrality (0.050) - this node is a cross-community bridge._
- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`, `CDP Browser Client`, `ResponseWriter`?**
_High betweenness centrality (0.048) - this node is a cross-community bridge._
- **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`, `ResponseWriter`?**
_High betweenness centrality (0.031) - this node is a cross-community bridge._
- **Are the 48 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?**
_`testConfig()` has 48 INFERRED edges - model-reasoned connections that need verification._
- **Are the 12 inferred relationships involving `newTestStore()` (e.g. with `TestAcquireDoesNotBlockTheWrite()` and `TestAcquireFailureLeavesTheBookmarkIntact()`) actually correct?**
_`newTestStore()` has 12 INFERRED edges - model-reasoned connections that need verification._
- **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?**
_`newWebTestServer()` has 8 INFERRED edges - model-reasoned connections that need verification._
- **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?**
_553 weakly-connected nodes found - possible documentation gaps or missing edges._
_552 weakly-connected nodes found - possible documentation gaps or missing edges._
File diff suppressed because one or more lines are too long
+4795 -2463
View File
File diff suppressed because it is too large Load Diff
+98 -68
View File
@@ -25,9 +25,9 @@
"semantic_hash": "dac242903b0e98c3e4395159d609e08e"
},
"backend/main.go": {
"mtime": 1786501521.228955,
"ast_hash": "6e98a3ae91aaa132df251e43c4dfca6d",
"semantic_hash": "6e98a3ae91aaa132df251e43c4dfca6d"
"mtime": 1786863963.3450089,
"ast_hash": "d5a50b03438d7c120079d40cc578462b",
"semantic_hash": ""
},
"skills-lock.json": {
"mtime": 1784884678.6842625,
@@ -140,18 +140,18 @@
"semantic_hash": "3a08979e4603aae5c32a58d5b6c39765"
},
"CLAUDE.md": {
"mtime": 1786856633.445473,
"ast_hash": "fab288c23be960d9c6afbfe3f21ff41c",
"mtime": 1786857336.6414917,
"ast_hash": "c79e49f912d7852f9832565a5f9a1c39",
"semantic_hash": ""
},
"DEPLOY.md": {
"mtime": 1786501942.7367291,
"ast_hash": "3c7b785c44badb6dda6234d2b39a9290",
"mtime": 1786861012.026504,
"ast_hash": "b7c2f813aded562ee9291c9baed795ad",
"semantic_hash": ""
},
"README.md": {
"mtime": 1786499529.7651505,
"ast_hash": "9d6be8aa8a2946c23ad48d8f2864b5ca",
"mtime": 1786861012.026504,
"ast_hash": "81af12a0a2d43e791efd030b5f4d6cbc",
"semantic_hash": ""
},
"docker-compose.prod.yml": {
@@ -160,8 +160,8 @@
"semantic_hash": "0751998a532297b8ac507a01ec48dc31"
},
"docker-compose.yml": {
"mtime": 1786499529.7725692,
"ast_hash": "124fd581bf0a662ff15012abfdb40a92",
"mtime": 1786861012.026504,
"ast_hash": "d3b53a8f42a8e0acb4fc4306ea42c093",
"semantic_hash": ""
},
".claude/settings.json": {
@@ -170,14 +170,14 @@
"semantic_hash": "e51077b6a7f1f67afc748f1a32a1557d"
},
"backend/web_test.go": {
"mtime": 1786216141.700644,
"ast_hash": "8f1b093b59eb1ed81bc7fc0c22495c50",
"semantic_hash": "8f1b093b59eb1ed81bc7fc0c22495c50"
"mtime": 1786865248.7717106,
"ast_hash": "7c298e39c63e4502cf6272d1e206e9ef",
"semantic_hash": ""
},
"backend/main_test.go": {
"mtime": 1786501521.228955,
"ast_hash": "8a165955cf28ad47481fec5ea7afb3d6",
"semantic_hash": "8a165955cf28ad47481fec5ea7afb3d6"
"mtime": 1786863966.7091317,
"ast_hash": "0a5d4dbdc770b40c329ccccc899b59f4",
"semantic_hash": ""
},
".claude/settings.local.json": {
"mtime": 1785697645.350201,
@@ -200,8 +200,8 @@
"semantic_hash": "e69a8340a371579ca3ea689660f7d7bd"
},
"AGENTS.md": {
"mtime": 1786856633.445473,
"ast_hash": "fab288c23be960d9c6afbfe3f21ff41c",
"mtime": 1786857336.6414917,
"ast_hash": "c79e49f912d7852f9832565a5f9a1c39",
"semantic_hash": ""
},
"userscript/test/logic.test.js": {
@@ -215,24 +215,24 @@
"semantic_hash": "8f3c0132eb4787a2c8736eb99f7689af"
},
"REDEPLOY.md": {
"mtime": 1786856643.0770833,
"mtime": 1786857336.6414917,
"ast_hash": "e5e910f0244a040e2ccdc669b17eb4db",
"semantic_hash": ""
},
"docs/design-system.md": {
"mtime": 1786022513.9623306,
"ast_hash": "421cd7e57f02d4b467f120ca6ddd7b6a",
"semantic_hash": "421cd7e57f02d4b467f120ca6ddd7b6a"
"mtime": 1786865139.9022946,
"ast_hash": "3bd39932997c8e245508874194db4e49",
"semantic_hash": ""
},
"backend/api_test.go": {
"mtime": 1786499529.7651505,
"ast_hash": "8e4b9293bc2e45ee3f42027315594fd5",
"mtime": 1786863966.7056258,
"ast_hash": "0d443b4fe3c04ab057505d73b8a88d07",
"semantic_hash": ""
},
"backend/cover_test.go": {
"mtime": 1786363889.552731,
"ast_hash": "c7e313d6c92eb28e6d370e5e89035984",
"semantic_hash": "c7e313d6c92eb28e6d370e5e89035984"
"mtime": 1786863966.7084978,
"ast_hash": "fd6b3f6ef46bd17b0241d104b6e5bdf6",
"semantic_hash": ""
},
"backend/internal/api/handlers.go": {
"mtime": 1786363889.552731,
@@ -245,12 +245,12 @@
"semantic_hash": "385b36f58488b7e6d93eb6d6034e9ee3"
},
"backend/internal/latest/browser.go": {
"mtime": 1786856879.3397639,
"ast_hash": "fbdba4bb56d8c804087bcb6e141e52ad",
"mtime": 1786861012.026504,
"ast_hash": "75f34a974568d122681939dd297944a2",
"semantic_hash": ""
},
"backend/internal/latest/browser_test.go": {
"mtime": 1786856254.36163,
"mtime": 1786857336.6414917,
"ast_hash": "800fa6aa471ada054a3c48943ad17ab7",
"semantic_hash": ""
},
@@ -260,22 +260,22 @@
"semantic_hash": ""
},
"backend/internal/latest/poller.go": {
"mtime": 1786856670.129715,
"ast_hash": "6b7060ff52994729832d9fcc3cc97dad",
"mtime": 1786865046.096795,
"ast_hash": "2bfb51177e3df474b80ea1b05523377e",
"semantic_hash": ""
},
"backend/internal/latest/poller_test.go": {
"mtime": 1786856311.8535168,
"ast_hash": "b6837fab377c94fa1d2f20a931975a4a",
"mtime": 1786865291.3758123,
"ast_hash": "1da669ac748a8d1a0289e557392fa3fd",
"semantic_hash": ""
},
"backend/internal/latest/sites.go": {
"mtime": 1786856865.4529688,
"ast_hash": "2f00456ac9d1d8148ba1bc6cbceb24f6",
"mtime": 1786861012.026504,
"ast_hash": "9b90f9b710ccbd0d0a0a6a0dd721ff3c",
"semantic_hash": ""
},
"backend/internal/latest/sites_test.go": {
"mtime": 1786856890.1231265,
"mtime": 1786857336.6414917,
"ast_hash": "0db2028a6073f342fee61ad15c2e5f0f",
"semantic_hash": ""
},
@@ -340,14 +340,14 @@
"semantic_hash": "0b6764a0ee20f5cb7748eecd31a1d220"
},
"backend/internal/store/store.go": {
"mtime": 1786363889.5602942,
"ast_hash": "54367a8ab043983e2491b2eb2650961c",
"semantic_hash": "54367a8ab043983e2491b2eb2650961c"
"mtime": 1786863704.5112119,
"ast_hash": "989629af18232e35b12f1eef2a8cb422",
"semantic_hash": ""
},
"backend/internal/store/store_test.go": {
"mtime": 1786363889.5602942,
"ast_hash": "dc823fd77bcce2268114e31d759b20a5",
"semantic_hash": "dc823fd77bcce2268114e31d759b20a5"
"mtime": 1786863712.075631,
"ast_hash": "e48f21e34238ba46874cc88cf96e022f",
"semantic_hash": ""
},
"backend/internal/userscript/userscript.go": {
"mtime": 1786216141.692644,
@@ -380,14 +380,14 @@
"semantic_hash": "19a573773be4ca22570ca2f8543120c5"
},
"backend/internal/web/web.go": {
"mtime": 1786363889.5678573,
"ast_hash": "8308949c658d3a08ce1c3cdbea6a907c",
"semantic_hash": "8308949c658d3a08ce1c3cdbea6a907c"
"mtime": 1786863744.1453943,
"ast_hash": "b051f2de214c4b253ee09fbbdb8ebb60",
"semantic_hash": ""
},
"backend/reader_credential_test.go": {
"mtime": 1786216141.700644,
"ast_hash": "2a751ea6d9c06635aa3179db0aef1b2b",
"semantic_hash": "2a751ea6d9c06635aa3179db0aef1b2b"
"mtime": 1786863966.7092986,
"ast_hash": "0c93de387af595901c43b6bdb3bed8cc",
"semantic_hash": ""
},
"chrome/entrypoint.sh": {
"mtime": 1786363889.5678573,
@@ -415,8 +415,8 @@
"semantic_hash": "e44a2f6f624db044e19508bc5ab05592"
},
"CONTEXT.md": {
"mtime": 1786855457.9327722,
"ast_hash": "24548f60414b4c5ff58538acaada5345",
"mtime": 1786861012.022683,
"ast_hash": "4aafbce0b046e6e34734fb414df818ce",
"semantic_hash": ""
},
"CUTOVER.md": {
@@ -425,19 +425,19 @@
"semantic_hash": "6c6f3e4c4c2f57867894280bce728c50"
},
"backend/AGENTS.md": {
"mtime": 1786856656.6826186,
"ast_hash": "d5610ba24076b57e17b9d76241acaa65",
"mtime": 1786865319.314815,
"ast_hash": "5bbabcd7dd425302cb0d989d8897baf6",
"semantic_hash": ""
},
"backend/CLAUDE.md": {
"mtime": 1786856656.6826186,
"ast_hash": "d5610ba24076b57e17b9d76241acaa65",
"mtime": 1786865319.314815,
"ast_hash": "5bbabcd7dd425302cb0d989d8897baf6",
"semantic_hash": ""
},
"backend/internal/web/templates/app.html": {
"mtime": 1786216141.692644,
"ast_hash": "3965e20e204afb71ba2a3aa86cb7c61c",
"semantic_hash": "3965e20e204afb71ba2a3aa86cb7c61c"
"mtime": 1786864445.1584811,
"ast_hash": "99d3e1139042d0eb61627155dddb3843",
"semantic_hash": ""
},
"backend/internal/web/templates/card.html": {
"mtime": 1786363889.5678573,
@@ -465,9 +465,9 @@
"semantic_hash": "bcc3101498a66cf8b79f9d97c6c9cd6b"
},
"backend/internal/web/templates/readers.html": {
"mtime": 1786216141.696644,
"ast_hash": "c5034e76bd20a705d2799cb5ecb328f0",
"semantic_hash": "c5034e76bd20a705d2799cb5ecb328f0"
"mtime": 1786864296.770521,
"ast_hash": "2b1cbb24d6185e48561966db1af04ba4",
"semantic_hash": ""
},
"backend/internal/web/templates/setup.html": {
"mtime": 1786216141.696644,
@@ -560,7 +560,7 @@
"semantic_hash": "46cf7822d4f667e3cab36b547abe5e97"
},
"backend/internal/latest/cover.go": {
"mtime": 1786856664.2957466,
"mtime": 1786857336.6414917,
"ast_hash": "d5f2248c3d11de74bf5a3977651b17c2",
"semantic_hash": ""
},
@@ -570,8 +570,8 @@
"semantic_hash": "60d9eb7c59a3751baf4f31c7655217e7"
},
"backend/internal/latest/acquire.go": {
"mtime": 1786499529.7688599,
"ast_hash": "6c1ad34bbe9f5b49d0fd1eae9093f55d",
"mtime": 1786861012.026504,
"ast_hash": "d605e3c94a62fc7787efbc139696b9d2",
"semantic_hash": ""
},
"backend/internal/latest/acquire_test.go": {
@@ -615,8 +615,8 @@
"semantic_hash": ""
},
"backend/internal/latest/read.go": {
"mtime": 1786499529.7688599,
"ast_hash": "3cf29046ddaef39fafb1df70b9f9ae8c",
"mtime": 1786861012.026504,
"ast_hash": "9f039cc3ad74f803d7621f7ef4157bf2",
"semantic_hash": ""
},
"docs/research/cloudflare-bot-scoring-and-poll-cadence.md": {
@@ -625,8 +625,38 @@
"semantic_hash": ""
},
"backend/internal/latest/smoke_comix_test.go": {
"mtime": 1786856356.3432186,
"mtime": 1786857336.6414917,
"ast_hash": "111fdbb75fc68ac2ab1013bc916063cf",
"semantic_hash": ""
},
"docs/adr/0010-poll-lanes-per-site-pace.md": {
"mtime": 1786861012.026504,
"ast_hash": "dc19d75f034ca920d93b9c71e6ca28e6",
"semantic_hash": ""
},
"backend/internal/latest/status.go": {
"mtime": 1786865033.5648637,
"ast_hash": "8141514efa5a7a66e77b9a62d4982d52",
"semantic_hash": ""
},
"backend/internal/store/migrations/0010_reader_sightings.sql": {
"mtime": 1786863695.6957178,
"ast_hash": "a72c08c63fad530df3c793d16edfa385",
"semantic_hash": ""
},
"backend/internal/web/admin.go": {
"mtime": 1786865087.473256,
"ast_hash": "b1c0f23a102a9bc7f84adf306d743f56",
"semantic_hash": ""
},
"backend/internal/web/templates/admin.html": {
"mtime": 1786865108.4095602,
"ast_hash": "f04ea1cb01369d53053eac489e796faa",
"semantic_hash": ""
},
"backend/internal/web/templates/lanes.html": {
"mtime": 1786865113.5362902,
"ast_hash": "8b9f9c7a56e0ec3bf950aa70acd95e92",
"semantic_hash": ""
}
}