From 90ce14aec6cb3edf6be8be7fadd74ea96249bf81 Mon Sep 17 00:00:00 2001 From: Sulthan Zaki Date: Sun, 16 Aug 2026 12:51:01 +0700 Subject: [PATCH] feat: one Poll Lane per Site, replacing the shared pace (#100) Each Site runs its own poll goroutine, paced by Rest and Gap from the registry (sites.go) instead of the shared cooldown/interval/stagger/batch config. Rest is enforced by the due query's WHERE clause; the Lane sleeps its effective gap between fetches (hour/eligible, floored at 1s). Lane-local failures: two refusals stop that Site for 15m, a lost browser stops only the round's remaining browser Lanes, and cover heals moved to background goroutines so a slow CDN cannot consume a Lane's gap. The five LATEST_CHAPTER_POLL_* pace env vars are gone; only the kill switch remains. --- .env.example | 23 +- CONTEXT.md | 2 +- DEPLOY.md | 10 +- README.md | 7 +- backend/AGENTS.md | 47 +- backend/internal/latest/acquire.go | 2 +- backend/internal/latest/browser.go | 4 +- backend/internal/latest/poller.go | 351 +- backend/internal/latest/poller_test.go | 519 ++- backend/internal/latest/read.go | 17 +- backend/internal/latest/sites.go | 68 +- backend/internal/store/store.go | 55 +- backend/internal/store/store_test.go | 59 +- backend/main.go | 99 +- backend/main_test.go | 138 +- docker-compose.yml | 11 +- docs/adr/0010-poll-lanes-per-site-pace.md | 91 + graphify-out/.graphify_labels.json | 7 +- graphify-out/GRAPH_REPORT.md | 109 +- graphify-out/graph.html | 8 +- graphify-out/graph.json | 3807 ++++++++++++++------- graphify-out/manifest.json | 95 +- 22 files changed, 3645 insertions(+), 1884 deletions(-) create mode 100644 docs/adr/0010-poll-lanes-per-site-pace.md diff --git a/.env.example b/.env.example index 487548b..123ac7c 100644 --- a/.env.example +++ b/.env.example @@ -73,26 +73,13 @@ DISCORD_REDIRECT_URI= # The backend re-checks each bookmarked series' newest published chapter on its # own schedule, so latest_chapter stays fresh even when you never open the manga # sites. This runs in parallel with the userscript's own in-browser check. -# Set to 0 to turn it off entirely. +# Set to 0 to turn it off entirely. Pace is per Site (one Poll Lane per Site, +# issue #100) and lives in the backend registry, not here — there is nothing +# else to configure. # LATEST_CHAPTER_POLL_ENABLED=1 # -# Two independent clocks. COOLDOWN is how long a plain-TLS series rests between -# checks; BROWSER_COOLDOWN is the longer rest for kagane and novelfull. INTERVAL -# is how often the poller wakes up and looks for series past their cooldowns. -# Shortening INTERVAL cannot shorten either cooldown. -LATEST_CHAPTER_POLL_COOLDOWN=1h # plain-TLS per series, floor 15m -LATEST_CHAPTER_POLL_BROWSER_COOLDOWN=6h # browser-backed per series, floor 15m -LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake -LATEST_CHAPTER_POLL_BATCH=14 # series per wake -LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch -# -# Uses a ticker, not an immediate first run: the first poll happens one -# INTERVAL after startup, not at startup. A container restarting more often -# than INTERVAL never polls. -# -# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these -# defaults. Beyond that the cadence stretches uniformly rather than breaking; -# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL. +# Every Site rests an hour between checks and gaps ten seconds between fetches; +# a Site with many Series tightens its own gap. See backend/internal/latest/sites.go. # CDP endpoint of the browser, used for the two sites behind a Cloudflare # JavaScript challenge (kagane, novelfull) and by the web UI's kagane cover diff --git a/CONTEXT.md b/CONTEXT.md index 6a98ae7..1f53110 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -86,7 +86,7 @@ _Avoid_: client report, user poll, observation, claim **Acquisition**: The single read of a Series page made the moment the Series first exists, giving it -both its Latest Chapter and its Cover without waiting out the Poll queue. Distinct +both its Latest Chapter and its Cover without waiting for the Lane's pace. Distinct from a Poll in the two ways that matter: a Reader is present — it is triggered by their first Bookmark of that Series — and it is the only read that establishes a Cover rather than refreshing facts. It happens once in a Series's life; every later diff --git a/DEPLOY.md b/DEPLOY.md index 9fb6d9f..13741f4 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -93,6 +93,14 @@ the backend dials but not the password the database expects, and `bookmark-api` crash-loops on `password authentication failed`. Set it before §2 and leave it alone. +An `.env` written before issue #100 carries the old poll-pace names +(`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`, +`_STAGGER`). All five are dead configuration now — the pace lives in the Site +registry (`backend/internal/latest/sites.go`), so **delete those lines** and +keep only the kill switch `LATEST_CHAPTER_POLL_ENABLED`. Leaving them behind +is harmless (nothing reads them) but silently misleads the next person who +edits the file. + > Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual > names (check your Traefik static config — common alternatives: `https`, > `myresolver`, `cloudflare`). Wrong names = no certificate issued. @@ -505,7 +513,7 @@ picks up a restarted Chrome's new debugger UUID by itself. | kagane rows never get a `latest_chapter`; log says `browser fetcher disabled` or nothing at all | `BROWSER_WS_URL` unset. Expected before §7 is done. | | kagane polls all fail; log shows a 500 from `/json/version` | `BROWSER_WS_URL` names a MagicDNS hostname (or any name). Chrome's DevTools handler only accepts an IP or `localhost` — use the tailnet IP. | | kagane polls fail with a connection error | Home machine off, off the tailnet, or the unit is down. `tailscale ping `, then `docker compose ps` in its `chrome/`. Costs freshness only; stored covers keep serving. | -| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series (up to `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, default 6h). | +| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series. | | `compose` in `chrome/` errors `set BROWSER_BIND_ADDR to this machine's tailnet IP` | No `chrome/.env`, or the variable is empty. Deliberate — it has no default so an unset value cannot publish CDP to the LAN. | | browser container restarts, or is OOM-killed | `docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'`. The 512 MiB cap is sized against a measured 645 MiB untuned peak; a real breach is a Chrome regression worth reading `docker logs` for, not a number to raise reflexively. | diff --git a/README.md b/README.md index 5dc0b35..11e096b 100644 --- a/README.md +++ b/README.md @@ -53,12 +53,7 @@ covers are stored, so the library renders in full with the browser switched off. | `DISCORD_API_BASE` | `https://discord.com/api/v10` | Test seam — tests point it at a local stub so the real token exchange runs. | | `USERSCRIPT_PATH` | `/userscript/manga-bookmark.user.js` | Bindmounted file served at `/u/{token}/manga-bookmark.user.js`. | | `NOVEL_USERSCRIPT_PATH` | `/userscript/novel-bookmark.user.js` | Same, for the novel library. | -| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. | -| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Rest between checks of one plain-TLS series; floor `15m`. | -| `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN` | `6h` | Rest between checks of one browser-backed series; floor `15m`. | -| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often the poller wakes. Cannot shorten either cooldown. | -| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake. Keep `BATCH × STAGGER` under `INTERVAL`. | -| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch — this is the outbound request rate. | +| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. Pace is per Site in the registry — one Poll Lane per Site, each with its own rest and gap (issue #100) — so no other knobs exist. | Compose reads a few more from the same `.env` that the backend never sees: `POSTGRES_PASSWORD` (required — `DATABASE_URL` is built from it, and Postgres diff --git a/backend/AGENTS.md b/backend/AGENTS.md index cfc73a8..f365f31 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -74,20 +74,33 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN Remove's row wear ember wash, two reversible ones wear `.calm` grey. `--ember` stay reserved for new-chapter signal: busy bar and inline error use `--mute`. -- **Latest-chapter poller:** ticker goroutine in same binary re-check - each bookmarked series' newest published chapter from backend's own - network access, so `latest_chapter` stay fresh when user not - browsing. Second, parallel signal — userscript keep own +- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100), + each re-checking that Site's bookmarked series' newest published chapter from + backend's own network access, so `latest_chapter` stays fresh when the user + isn't browsing. Second, parallel signal — the userscript keeps its own `maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged. - Two independent clocks: per-series cooldown (`series.latest_checked_at`, - enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval. + Two independent clocks: per-series rest (`series.latest_checked_at`, + enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and + per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live + in the Site registry (`internal/latest/sites.go`), not config: the five env + knobs that used to size a shared pace are gone. The poller walks **Series, not Bookmarks** — a series referenced by several bookmarks is fetched once per cycle, and the due queue orders `reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped - *before* fetch so broken series wait out full cooldown instead of retrying + *before* fetch so broken series wait out the rest instead of retrying every tick; found chapter written straight to the series row via `Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list order — is never touched. + Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass + stop that Site for `refuseBackoff` (15m) while other Lanes continue; an + `errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag + that makes the other browser Lanes skip their passes for the same 15m, so a + restarting Chrome doesn't stamp one Series per Lane per pass — after the + window the flag decays and they probe again. Browser Lanes wake Chrome only + when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser), + and cover work (both healing a stored source URL and filling a blank from + the series page) runs in the background so a slow CDN can't consume a + Lane's gap. Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth against fingerprint-based blocking; any failure log and skip. kagane, comix and novelfull sit behind Cloudflare JavaScript challenges the TLS client @@ -152,15 +165,17 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN `_REDIRECT_URI` (required; Discord OAuth for the browser UI), `DISCORD_REQUIRED_ROLE` (optional role gate, empty by default), `DISCORD_API_BASE` (default `https://discord.com/api/v10`), - `LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/ - `_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on, - `1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both - cooldowns have a `15m` floor). The browser cooldown is longer for cost, not - for safety: a challenged page costs seconds of a serialized single-tab - browser, while a plain read costs one request. It buys no documented - reduction in challenge risk — free-plan zones have no bot score and no - published per-IP rate input, and `cf_clearance` expires in 30 minutes so - every cadence at or above 1h re-solves anyway — + `LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill + switch, default on). Pace is per Site in the registry (issue #100): every + Site rests an hour and gaps ten seconds, a Site with more eligible Series + than 360 tightens its own gap toward the 1s floor, and browser Lanes wake + Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/ + `_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are + gone. The 1h rest for browser Sites is safe on documented grounds: a + challenged page costs seconds of a serialized single-tab browser, free-plan + zones have no bot score and no published per-IP rate input, and + `cf_clearance` expires in 30 minutes so every cadence at or above 1h + re-solves anyway — `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. `USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at `/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`, diff --git a/backend/internal/latest/acquire.go b/backend/internal/latest/acquire.go index 929ffa7..2a8bac4 100644 --- a/backend/internal/latest/acquire.go +++ b/backend/internal/latest/acquire.go @@ -120,7 +120,7 @@ func (a *Acquirer) acquire(ctx context.Context, sr store.Series) { // Stamped after success — the reverse of the poller, which stamps before // the fetch: the Reader is here, watching the Series they just created, so // a failed acquisition must leave the row due for a fast retry rather than - // consuming the cooldown. The stamp happens even when the page read + // consuming the rest. The stamp happens even when the page read // succeeded but produced no facts to persist. if err := a.Store.MarkLatestChecked(sr.Site, sr.SeriesID, time.Now().UnixMilli()); err != nil { log.Printf("acquire %q: mark checked: %v", sr.Key(), err) diff --git a/backend/internal/latest/browser.go b/backend/internal/latest/browser.go index ba01bd0..d8c7a20 100644 --- a/backend/internal/latest/browser.go +++ b/backend/internal/latest/browser.go @@ -18,7 +18,7 @@ import ( // challengeTimeout bounds one navigate-and-solve. A Cloudflare managed // challenge clears in a few seconds when it clears at all; anything longer is a -// challenge that is not going to pass, and the caller's cooldown was already +// challenge that is not going to pass, and the caller's rest was already // stamped before this ran. const challengeTimeout = 45 * time.Second @@ -213,7 +213,7 @@ func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, st // errChallengeHeld reports that the budget ran out with the interstitial still // up. Distinct from a transport failure: it means "this site said no", which -// the poller answers with a 403 and its ordinary cooldown. +// the poller answers with a refusal backoff for that Site's Lane (issue #100). var errChallengeHeld = errors.New("challenge held") // errBrowserInterrupted distinguishes a remote Chrome restart from the diff --git a/backend/internal/latest/poller.go b/backend/internal/latest/poller.go index 057a2bd..433b6f8 100644 --- a/backend/internal/latest/poller.go +++ b/backend/internal/latest/poller.go @@ -5,6 +5,8 @@ import ( "errors" "log" "net/url" + "sort" + "sync" "time" "bookmarkmanager/backend/internal/store" @@ -28,15 +30,11 @@ type BrowserCoverFetcher interface { // in parallel and report the same observable fact, so whichever writes last wins // and neither needs to know about the other. // -// Two clocks, deliberately independent: -// -// - Interval is how often this goroutine wakes up and looks. -// - Cooldowns are how long a series rests since its own last check. Browser- -// backed sites use the longer BrowserCooldown. -// -// Cooldowns are enforced by the WHERE clause in DueForLatestCheck rather than -// by any timer. Shortening Interval therefore cannot shorten anyone's cooldown; -// it only makes the poller wake up and find nothing due more often. +// Every Site gets its own Poll Lane: one independent stream of Polls with its +// own pace, running concurrently with every other Site's (issue #100). Rest +// time and gap live in the Site registry, not here — see sites.go. Rest is +// enforced by the WHERE clause in DueForLatestCheck rather than by any timer; +// the gap is enforced by the Lane sleeping between fetches. type Poller struct { Store *store.Store Fetch Fetcher @@ -50,11 +48,19 @@ type Poller struct { // same failure-isolated prefetch path. CoverBytesFetch CoverBytesFetcher Now func() time.Time // injected so tests can freeze it - Cooldown time.Duration - BrowserCooldown time.Duration - Interval time.Duration - Stagger time.Duration - Batch int + + // refuseUntil gates a Site's Lane after it refused twice in one run: no + // Series of that Site is attempted again before this time (issue #100). + // browserDownAt is when a browser Lane last lost the sidecar; the other + // browser Lanes skip their passes for the next refuseBackoff, so a + // restarting Chrome does not stamp one Series per pass per Lane (story 20). + mu sync.Mutex + refuseUntil map[string]time.Time + browserDownAt time.Time + // coverWG tracks in-flight cover work. Covers heal in the background so a + // slow cover host cannot delay the next Series-page Poll; tests join it + // before asserting on cover fetches. + coverWG sync.WaitGroup } // fillBlankCover gives a Series its Cover when it has none. The blank state is @@ -76,7 +82,14 @@ func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, cover stri if cover == "" { return } - p.storeCover(ctx, sr, cover) + // Like healCover, the fill runs in the background: a large import of + // blanks would otherwise pay one og:image fetch per Series against the + // Lane's gap (issue #100, story 12). + p.coverWG.Add(1) + go func() { + defer p.coverWG.Done() + p.storeCover(ctx, sr, cover) + }() } // prefetchCover heals Series that already carry a third-party source URL but @@ -143,72 +156,248 @@ func fetcherFor(site string, browser, tls Fetcher) Fetcher { return nil } -// Run polls until ctx is cancelled. -// -// runOnce is called synchronously, so a batch that overruns the tick delays the -// next one instead of stacking a second batch on top of it. That is the intended -// failure mode for a misconfigured batch x stagger: a slower cadence, never -// concurrent fetch storms. +// Run polls until ctx is cancelled: one goroutine per Site Lane, each pacing +// itself by the Site's effective gap. Lanes share nothing but the store and +// the browser fetcher's single tab (BrowserFetcher serializes itself), so one +// hostile Site burns only its own budget. +// laneNames returns every registry Site in the deterministic order both Run +// and runOnce iterate: sorted, so lane behaviour and its tests agree on who +// runs first. +func laneNames() []string { + names := make([]string, 0, len(sites)) + for name := range sites { + names = append(names, name) + } + sort.Strings(names) + return names +} + func (p *Poller) Run(ctx context.Context) { - log.Printf("latest-chapter poller: interval=%s cooldown=%s browser-cooldown=%s batch=%d stagger=%s", - p.Interval, p.Cooldown, p.BrowserCooldown, p.Batch, p.Stagger) - t := time.NewTicker(p.Interval) - defer t.Stop() + names := laneNames() + log.Printf("latest-chapter poller: %d lanes, rest=%s gap=%s", len(names), defaultRest, defaultGap) + for _, name := range names { + go p.lane(ctx, name) + } + <-ctx.Done() + log.Println("latest-chapter poller: stopped") +} + +// lane is one Site's Poll Lane: one pass, then sleep the pace the pass +// reported, then another pass, until ctx is cancelled. The sleep is the whole +// pace discipline — a pass that fetched nothing still reports its gap so the +// Lane wakes often enough to notice Series as they become due. The pass shares +// the Poller's browser-down state, so a sidecar loss is noticed once and the +// other browser Lanes skip passes until the backoff window decays. +func (p *Poller) lane(ctx context.Context, name string) { for { + pace := p.runLanePass(ctx, name, true) + if ctx.Err() != nil { + return + } select { case <-ctx.Done(): - log.Println("latest-chapter poller: stopped") return - case <-t.C: - p.runOnce(ctx) + case <-time.After(pace): } } } -// runOnce processes one batch of due series. +// runOnce processes one round: one pass of every Lane, back to back, no real +// time passing. This is the deterministic entry point the test suite drives a +// round at a time. The production Run loop does the same work paced by its own +// sleeps; pacing is the only difference. func (p *Poller) runOnce(ctx context.Context) { + for _, name := range laneNames() { + p.runLanePass(ctx, name, false) + } +} + +// runLanePass processes one pass of one Site's Lane: select the due Series, +// pace through them, and report how long the Lane should wait before its next +// pass. paced spaces consecutive fetches by the Site's effective gap — the +// production Lane's rate limit; the deterministic test entry runs back to back. +func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration { now := p.Now() - cutoff := now.Add(-p.Cooldown).UnixMilli() - browserCutoff := now.Add(-p.BrowserCooldown).UnixMilli() - due, err := p.Store.DueForLatestCheck(cutoff, browserCutoff, browserBackedSites(), p.Batch) - if err != nil { - log.Printf("latest poll: due query: %v", err) - return + if until := p.refusalBackoff(name); now.Before(until) { + // Cooling down after a refusal: do not attempt this Site at all. + return until.Sub(now) + } + if isBrowserSite(name) { + if downFor, down := p.browserDownFor(now); down && downFor < refuseBackoff { + // A sibling browser Lane lost the sidecar within the backoff + // window: skip this pass, so a restarting Chrome does not stamp + // this Site's Series one pass at a time. After refuseBackoff the + // flag decays and the Lane probes again (issue #100, story 20). + log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor) + return refuseBackoff - downFor + } + } + s := sites[name] + f := fetcherFor(name, p.BrowserFetch, p.Fetch) + if f == nil { + // No fetcher at all right now (browser absent, no fallback): every + // Series stays unstamped and due, so a browser that appears after a + // restart finds its full queue waiting (issue #100). + return defaultGap } + due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli()) + if err != nil { + log.Printf("latest poll %s: due query: %v", name, err) + return defaultGap + } + if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) { + // Below both thresholds Chrome stays asleep (ADR-0005 on-demand + // browser): waking it for a single Poll would cost a challenge solve + // per request. + return defaultGap + } + if s.Browser != nil { + // Browser Lanes share one tab, so their combined ceiling is about 360 + // Polls an hour. When they cannot keep up, the wait past the rest time + // grows — log by how much, every pass, so the decision to give them + // more pages is made from a measurement rather than a guess. + if behind := maxSeriesWait(due, now, s.Rest) - s.Rest; behind > 0 { + log.Printf("latest poll %s: browser lane behind by %s (browser Sites cannot keep up with the hour)", name, behind) + } + } + + eligible, err := p.Store.EligibleSeriesCount(name) + if err != nil { + log.Printf("latest poll %s: eligible count: %v", name, err) + return defaultGap + } + gap, clamped := effectiveGap(s, eligible) + if clamped { + log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible) + } + if eligible == 0 { + // Nothing to poll for the foreseeable future; sleep a full rest instead + // of re-querying every gap. + return s.Rest + } + + refusals := 0 checked := 0 for i, sr := range due { if ctx.Err() != nil { break } - // Staggered rather than fired together: a burst of simultaneous requests - // from one server IP is the traffic shape most likely to move that IP's - // bot score. This is the server-side analogue of the userscript's "one - // series per navigation ... indistinguishable from browsing" (L455-456). - stopped := false - if i > 0 && p.Stagger > 0 { - select { - case <-ctx.Done(): - stopped = true - case <-time.After(p.Stagger): - } - } - if stopped { + if refusals >= 2 { + // This Site refused twice in a row: the remaining Series are left + // unstamped and due, and the Lane waits refuseBackoff before + // trying it again. break } - p.checkOne(ctx, sr) + if paced && i > 0 { + select { + case <-ctx.Done(): + break + case <-time.After(gap): + } + if ctx.Err() != nil { + break + } + } + if err := p.checkOne(ctx, sr); err != nil { + switch { + case errors.Is(err, errChallengeHeld): + refusals++ + case errors.Is(err, errBrowserInterrupted): + p.setBrowserDown(now) + log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, refuseBackoff) + return gap + default: + refusals = 0 + } + } else { + refusals = 0 + } checked++ } - // due vs checked is how you tell which constraint is binding: ticks that - // report due=0 mean the cooldown is the limit, ticks that report due==batch - // every time mean throughput is. - log.Printf("latest poll: due=%d checked=%d", len(due), checked) + if checked > 0 { + log.Printf("latest poll %s: due=%d checked=%d", name, len(due), checked) + } + if refusals >= 2 { + p.setRefusalBackoff(name, now.Add(refuseBackoff)) + log.Printf("latest poll %s: refused twice this run, waiting %s", name, refuseBackoff) + return refuseBackoff + } + return gap +} + +func (p *Poller) refusalBackoff(name string) time.Time { + p.mu.Lock() + defer p.mu.Unlock() + return p.refuseUntil[name] +} + +func (p *Poller) setRefusalBackoff(name string, until time.Time) { + p.mu.Lock() + defer p.mu.Unlock() + if p.refuseUntil == nil { + p.refuseUntil = make(map[string]time.Time) + } + p.refuseUntil[name] = until +} + +// setBrowserDown records when a browser Lane lost the sidecar. It is Poller +// state rather than pass state so the other browser Lanes see it too. +func (p *Poller) setBrowserDown(now time.Time) { + p.mu.Lock() + p.browserDownAt = now + p.mu.Unlock() +} + +// browserDownFor reports how long the sidecar has been down and that it is +// down at all — the zero time means never down, which must not read as a +// zero-duration loss. The window decays: once refuseBackoff passes without a +// fresh loss, Lanes probe again. +func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) { + p.mu.Lock() + defer p.mu.Unlock() + if p.browserDownAt.IsZero() { + return 0, false + } + return now.Sub(p.browserDownAt), true +} + +// isBrowserSite reports whether the registry routes this Site's page through +// the browser sidecar. +func isBrowserSite(name string) bool { + return sites[name].Browser != nil +} + +// browserWakeDue reports whether a browser Lane may start a run: five or more +// of its Series are due, or any one of them has been due for browserWakeAge. +// Below both thresholds the Lane leaves Chrome asleep — Series Polled together +// become due together, so the group naturally stays clustered, and the age +// rule exists to stop a Series that drifted out of the group from starving. +func browserWakeDue(due []store.Series, now time.Time, rest time.Duration) bool { + if len(due) >= browserWakeCount { + return true + } + return maxSeriesWait(due, now, rest) >= browserWakeAge +} + +// maxSeriesWait returns how long the most-overdue of the due Series has been +// waiting past its due moment (0 when due is empty). +func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration { + var oldest time.Duration + for _, sr := range due { + if w := now.Sub(time.UnixMilli(sr.LatestCheckedAt).Add(rest)); w > oldest { + oldest = w + } + } + return oldest } // checkOne re-checks one series. Every failure path here is "log and move on": // the poller is a best-effort enhancement, and no single bad series may stall a -// batch or take down the process. -func (p *Poller) checkOne(ctx context.Context, sr store.Series) { +// Lane or take down the process. The returned error is the page read's +// classified outcome so the Lane can tell a refusal from a loss of the +// browser; non-classified failures still return nil-equivalent behaviour. +func (p *Poller) checkOne(ctx context.Context, sr store.Series) error { defer func() { if r := recover(); r != nil { log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r) @@ -216,44 +405,46 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) { }() // Stamped before the fetch, not after, so an error, a timeout, or a shutdown - // mid-request still consumes the cooldown. Otherwise a renamed or deleted - // series would be retried on every single tick forever. The userscript - // stamps in the same order and for the same reason (L471-473). + // mid-request still consumes the rest. Otherwise a renamed or deleted + // series would be retried on every single pass forever. The userscript + // stamps in the same order and for the same reason (L471-473). A Series + // never reaches checkOne without a fetcher — runLanePass skips those — so + // the stamp means "attempted", and an untried Series stays due. if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil { log.Printf("latest poll %q: mark checked: %v", sr.Key(), err) - return + return nil } facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch) if err != nil { switch { case errors.Is(err, errNotFetchable): - // The cooldown above is already consumed, so a row that never - // passes the gate is retried at cooldown pace rather than + // The rest above is already consumed, so a row that never + // passes the gate is retried at rest pace rather than // hot-looping. log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL) - return + return err case errors.Is(err, errNoFetcher): log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site) - return + return err } // A legacy cover heals independently of the page read: its source may // answer — a CDN — while the origin does not, so a fetch failure does // not skip the heal, matching the order the shared read replaced. - p.prefetchCover(ctx, sr) + p.healCover(ctx, sr) log.Printf("latest poll %q: %v", sr.Key(), err) - return + return err } // A legacy cover source is healed independently of the page read. - p.prefetchCover(ctx, sr) + p.healCover(ctx, sr) // Cover fill is independent of the chapter signal: a page that lost its // chapter list may keep its og:image, and a blank Series heals either way. p.fillBlankCover(ctx, sr, facts.Cover) if !facts.HasLatest { // Most likely a challenge page or a layout change. Either way the row is - // already stamped, so this waits out a cooldown instead of hot-looping. + // already stamped, so this waits out a rest instead of hot-looping. log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen) - return + return nil } // Equality, not >, mirroring the userscript (L427): a site that retracts a @@ -261,7 +452,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) { // against the due-query snapshot; a concurrent write in between only costs // one redundant UPDATE of the same absolute value, never a wrong one. if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num { - return + return nil } // Series-level write: the row is shared, so one update refreshes every @@ -270,9 +461,29 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) { // the list. if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil { log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err) - return + return nil } log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label) + return nil +} + +// healCover runs prefetchCover in the background. Cover bytes come from a +// different host — often a CDN — and heal once in a Series's life, so they +// must not consume a Lane's gap: a large import with many blanks would +// otherwise make every Latest Chapter go stale behind a slow image host +// (issue #100). +func (p *Poller) healCover(ctx context.Context, sr store.Series) { + p.coverWG.Add(1) + go func() { + defer p.coverWG.Done() + p.prefetchCover(ctx, sr) + }() +} + +// waitCovers blocks until every in-flight cover heal finishes. Tests call it +// after a round before asserting on cover fetches. +func (p *Poller) waitCovers() { + p.coverWG.Wait() } // fetchableSeriesURL reports whether site is a Site the registry knows and diff --git a/backend/internal/latest/poller_test.go b/backend/internal/latest/poller_test.go index b1e0376..4f9d8ce 100644 --- a/backend/internal/latest/poller_test.go +++ b/backend/internal/latest/poller_test.go @@ -5,6 +5,7 @@ import ( "crypto/sha256" "database/sql" "errors" + "fmt" "log" "os" "strings" @@ -156,19 +157,16 @@ func (f *fakeBytesCoverFetcher) callCount() int { return len(f.calls) } -// newTestPoller wires a poller with a frozen clock and no stagger, so tests run -// instantly and deterministically. +// newTestPoller wires a poller with a frozen clock. Rest and gap come from the +// Site registry, so tests seed checked_at relative to the one-hour rest; the +// round entry point (runOnce) runs every Lane back to back with no real +// pacing, so tests stay instant and deterministic. func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Poller { t.Helper() return &Poller{ - Store: s, - Fetch: f, - Now: func() time.Time { return at }, - Cooldown: time.Hour, - BrowserCooldown: 6 * time.Hour, - Interval: 10 * time.Minute, - Stagger: 0, - Batch: 14, + Store: s, + Fetch: f, + Now: func() time.Time { return at }, } } @@ -208,9 +206,10 @@ func TestRunOncePrefetchesPublicCover(t *testing.T) { covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"} p := &Poller{ Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverBytesFetch: covers, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if got := covers.callCount(); got != 1 { t.Fatalf("cover fetch calls = %d, want 1", got) @@ -241,9 +240,10 @@ func TestRunOnceDoesNotStoreNonImagePublicCover(t *testing.T) { p := &Poller{ Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("challenge"), contentType: "text/html"}, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if _, _, found, err := s.GetCover(coverURL); err != nil || found { t.Fatalf("non-image cover = found %v, err %v; want missing", found, err) @@ -350,23 +350,6 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) { } } -func TestRunOnceRespectsBatchLimit(t *testing.T) { - s, _ := newTestStore(t) - for i := 0; i < 20; i++ { - key := "asura:s" + string(rune('a'+i)) - seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0) - } - - f := &fakeFetcher{body: "", status: 200} - p := newTestPoller(t, s, f, time.UnixMilli(5_000_000)) - p.Batch = 5 - p.runOnce(context.Background()) - - if got := f.callCount(); got != 5 { - t.Fatalf("fetched %d series, want 5 (batch limit)", got) - } -} - // The point of the split (ADR-0003): a series referenced by several bookmarks // is fetched once per due cycle, not once per bookmark. Two bookmarks share a // series when two readers track it (issue #22). @@ -409,8 +392,8 @@ func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) { } } -// One unreachable series must not abandon the rest of the batch. -func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) { +// One unreachable series must not abandon the rest of the Lane. +func TestRunOnceOneBadSeriesDoesNotStallLane(t *testing.T) { s, _ := newTestStore(t) keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"} for _, k := range keys { @@ -436,7 +419,9 @@ func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) { } } -func TestRunLogsCooldowns(t *testing.T) { +// Pace now lives in the registry, not config: Run logs the lane defaults so a +// deployment can see what the poller is doing without reading the source. +func TestRunLogsLaneDefaults(t *testing.T) { var logs strings.Builder previous := log.Writer() log.SetOutput(&logs) @@ -444,21 +429,19 @@ func TestRunLogsCooldowns(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel() - (&Poller{ - Cooldown: time.Hour, - BrowserCooldown: 6 * time.Hour, - Interval: time.Hour, - }).Run(ctx) + (&Poller{Now: func() time.Time { return time.Now() }}).Run(ctx) - if got := logs.String(); !strings.Contains(got, "cooldown=1h") || - !strings.Contains(got, "browser-cooldown=6h") { - t.Fatalf("startup log = %q, want both cooldowns", got) + got := logs.String() + for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} { + if !strings.Contains(got, want) { + t.Fatalf("startup log = %q, want %q", got, want) + } } } -// The cooldown is enforced by the due query, so a second immediate pass must do -// nothing at all — this is what makes the tick interval independent of it. -func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) { +// The rest is enforced by the due query, so a second immediate pass must do +// nothing at all — this is what makes the Lane's sleep independent of it. +func TestRunOnceHonoursRestAcrossPasses(t *testing.T) { s, _ := newTestStore(t) const url = "https://asurascans.com/comics/x" seedForCheck(t, s, "asura:x", url, 0) @@ -471,56 +454,18 @@ func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) { if got := f.callCount(); got != 1 { t.Fatalf("first pass fetched %d, want 1", got) } - // Same instant, and again 59 minutes later: both inside the 1h cooldown. + // Same instant, and again 59 minutes later: both inside the 1h rest. p.runOnce(context.Background()) p.Now = func() time.Time { return now.Add(59 * time.Minute) } p.runOnce(context.Background()) if got := f.callCount(); got != 1 { - t.Fatalf("fetched %d times inside the cooldown, want 1", got) + t.Fatalf("fetched %d times inside the rest, want 1", got) } - // Past the cooldown, it is due again. + // Past the rest, it is due again. p.Now = func() time.Time { return now.Add(61 * time.Minute) } p.runOnce(context.Background()) if got := f.callCount(); got != 2 { - t.Fatalf("fetched %d times after the cooldown, want 2", got) - } -} - -func TestRunOnceUsesBrowserCooldown(t *testing.T) { - s, _ := newTestStore(t) - const browserKey = "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b" - seedForCheck(t, s, "asura:plain", "https://asurascans.com/comics/plain", 0) - seedForCheck(t, s, browserKey, "https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", 0) - - hour := time.Hour - now := time.Unix(2*int64(hour/time.Second), 0) - tls := &fakeFetcher{status: 200} - browser := &fakeFetcher{status: 200} - p := &Poller{ - Store: s, - Fetch: tls, - BrowserFetch: browser, - Now: func() time.Time { return now }, - Cooldown: hour, - BrowserCooldown: 6 * hour, - Batch: 10, - } - - p.runOnce(context.Background()) - if got := tls.callCount(); got != 1 { - t.Fatalf("plain-TLS fetches after 2h = %d, want 1", got) - } - if got := browser.callCount(); got != 0 { - t.Fatalf("browser fetches after 2h = %d, want 0", got) - } - - now = time.Unix(7*int64(hour/time.Second), 0) - p.runOnce(context.Background()) - if got := tls.callCount(); got != 2 { - t.Fatalf("plain-TLS fetches after 7h = %d, want 2", got) - } - if got := browser.callCount(); got != 1 { - t.Fatalf("browser fetches after 7h = %d, want 1", got) + t.Fatalf("fetched %d times after the rest, want 2", got) } } @@ -665,17 +610,18 @@ func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) { f := &fakeFetcher{body: kaganeAPIFixture, status: 200} p := &Poller{ - Store: s, - Fetch: f, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, - Interval: time.Hour, Batch: 10, + Store: s, + Fetch: f, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) if len(f.calls) != 0 { t.Errorf("TLS fetcher was called for kagane: %v", f.calls) } + if got := readLatestCheckedAt(t, s, "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"); got != 0 { + t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got) + } } // novelfull without a browser is not skipped outright: its challenge is a @@ -699,11 +645,10 @@ func TestNovelfullUsesTLSWhenNoBrowserFetcher(t *testing.T) { covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"} p := &Poller{ Store: s, Fetch: tlsF, CoverBytesFetch: covers, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, - Interval: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if len(tlsF.calls) != 1 { t.Fatalf("TLS fetcher calls = %d, want 1", len(tlsF.calls)) @@ -741,9 +686,7 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) { browserF := &fakeFetcher{body: kaganeAPIFixture, status: 200} p := &Poller{ Store: s, Fetch: tlsF, BrowserFetch: browserF, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, - Interval: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) @@ -789,11 +732,10 @@ func TestComixUsesBrowserFetcher(t *testing.T) { p := &Poller{ Store: s, Fetch: tlsF, BrowserFetch: browserF, CoverFetch: covers, CoverBytesFetch: tlsCovers, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, - Interval: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if len(tlsF.calls) != 0 { t.Errorf("TLS fetcher was called for comix: %v", tlsF.calls) @@ -831,15 +773,16 @@ func TestComixSkippedWhenNoBrowserFetcher(t *testing.T) { f := &fakeFetcher{body: comixSeriesFixture, status: 200} p := &Poller{ Store: s, Fetch: f, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, - Interval: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) if len(f.calls) != 0 { t.Errorf("TLS fetcher was called for comix: %v", f.calls) } + if got := readLatestCheckedAt(t, s, "comix:n8we-dungeons-and-crayons"); got != 0 { + t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got) + } } func TestRunOncePrefetchesKaganeCover(t *testing.T) { @@ -860,10 +803,10 @@ func TestRunOncePrefetchesKaganeCover(t *testing.T) { p := &Poller{ Store: s, Fetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL)) if err != nil || !ok { @@ -898,11 +841,13 @@ func TestRunOnceDoesNotRefetchKaganeCover(t *testing.T) { covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"} p := &Poller{ Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers, - Now: func() time.Time { return at }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return at }, } p.runOnce(context.Background()) + p.waitCovers() at = at.Add(2 * time.Hour) p.runOnce(context.Background()) + p.waitCovers() if got := covers.callCount(); got != 1 { t.Fatalf("cover fetch calls = %d, want 1 after two due cycles", got) @@ -927,9 +872,10 @@ func TestRunOnceCoverFailureDoesNotBlockChapter(t *testing.T) { p := &Poller{ Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: &fakeCoverFetcher{err: errors.New("browser unavailable")}, - Now: func() time.Time { return now }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return now }, } p.runOnce(context.Background()) + p.waitCovers() got, found, err := s.Get(s.OwnerID(), key) if err != nil || !found { @@ -960,9 +906,10 @@ func TestRunOnceRejectsInvalidKaganeCover(t *testing.T) { p := &Poller{ Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: &fakeCoverFetcher{body: []byte("not an image"), contentType: "text/html"}, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found { t.Fatalf("invalid cover persisted = %v, err %v; want missing", found, err) @@ -985,9 +932,10 @@ func TestRunOnceWithoutCoverFetcherStillPollsKagane(t *testing.T) { seedCoverSource(t, dbURL, "kagane", seriesID, coverURL) p := &Poller{ Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found { t.Fatalf("cover after nil CoverFetch = found %v, err %v; want missing", found, err) @@ -1010,9 +958,10 @@ func TestRunOnceRoutesNonKaganeCoverToPublicFetcher(t *testing.T) { p := &Poller{ Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverFetch: browserCovers, CoverBytesFetch: publicCovers, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() if got := publicCovers.callCount(); got != 1 { t.Fatalf("public cover fetch calls = %d, want 1", got) @@ -1128,10 +1077,10 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) { p := &Poller{ Store: s, Fetch: page, BrowserFetch: page, CoverBytesFetch: public, CoverFetch: browser, - Now: func() time.Time { return time.UnixMilli(5_000_000) }, - Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, } p.runOnce(context.Background()) + p.waitCovers() got := readBookmark(t, s, tc.key) wantWire := testCoverBaseURL + "/covers/" + store.CoverAddress(tc.wantCover) @@ -1181,7 +1130,7 @@ func TestRunOnceDoesNotReplaceExistingCover(t *testing.T) { p := &Poller{ Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200}, CoverBytesFetch: public, - Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10, + Now: func() time.Time { return at }, } p.runOnce(context.Background()) at = at.Add(2 * time.Hour) @@ -1216,9 +1165,10 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) { p := &Poller{ Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200}, CoverBytesFetch: public, - Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10, + Now: func() time.Time { return at }, } p.runOnce(context.Background()) + p.waitCovers() if got := readBookmark(t, s, key); got.Cover != "" { t.Fatalf("Cover after failed fetch = %q, want blank", got.Cover) } @@ -1231,6 +1181,7 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) { public.contentType = "image/jpeg" at = at.Add(2 * time.Hour) p.runOnce(context.Background()) + p.waitCovers() if got := public.callCount(); got != 2 { t.Fatalf("cover fetch calls after retry = %d, want 2", got) @@ -1264,9 +1215,10 @@ func TestRunOnceBlankCoverFailureDoesNotBlockChapter(t *testing.T) { p := &Poller{ Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200}, CoverBytesFetch: &fakeBytesCoverFetcher{err: errors.New("cdn down")}, - Now: func() time.Time { return now }, Cooldown: time.Hour, Batch: 10, + Now: func() time.Time { return now }, } p.runOnce(context.Background()) + p.waitCovers() got := readBookmark(t, s, key) if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 { @@ -1289,3 +1241,342 @@ const kaganeAPIFixtureWithCover = ` {"book_id":"b","title":"Episode 41","chapter_no":"41","sort_no":41}, {"book_id":"c","title":"Episode 40.5","chapter_no":"40.5","sort_no":40}]} ` + +func TestEffectiveGap(t *testing.T) { + asura := sites["asura"] + tests := []struct { + eligible int + want time.Duration + clamped bool + }{ + {0, 10 * time.Second, false}, + {5, 10 * time.Second, false}, + {360, 10 * time.Second, false}, + {720, 5 * time.Second, false}, + {3600, time.Second, false}, + {4000, time.Second, true}, + } + for _, tt := range tests { + got, clamped := effectiveGap(asura, tt.eligible) + if got != tt.want || clamped != tt.clamped { + t.Errorf("effectiveGap(asura, %d) = (%s, %v), want (%s, %v)", + tt.eligible, got, clamped, tt.want, tt.clamped) + } + } +} + +// The due query orders by sharedness first, then age: a series two readers +// track is polled before a single-reader series that has waited far longer +// (ADR-0003, issue #100). +func TestRunOnceOrdersBySharednessThenAge(t *testing.T) { + s, url := newTestStore(t) + other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))}, t.TempDir(), testCoverBaseURL) + if err != nil { + t.Fatalf("Open second reader: %v", err) + } + t.Cleanup(func() { other.Close() }) + + now := time.UnixMilli(5_000_000) + // popular: two readers, due for 2 minutes. loner: one reader, due for 23 + // minutes. Popularity must win — the loner waited far longer. + const ( + popularKey = "asura:popular" + lonerKey = "asura:loner" + popularURL = "https://asurascans.com/comics/popular" + lonerURL = "https://asurascans.com/comics/loner" + ) + seedForCheck(t, s, popularKey, popularURL, now.Add(-62*time.Minute).UnixMilli()) + seedForCheck(t, s, lonerKey, lonerURL, 0) + if _, err := s.Upsert(other.OwnerID(), store.Bookmark{ + Key: popularKey, Site: "asura", SeriesID: "popular", UpdatedAt: 2000, + }); err != nil { + t.Fatalf("seed second reader: %v", err) + } + + f := &fakeFetcher{body: asuraSeriesFixture, status: 200} + newTestPoller(t, s, f, now).runOnce(context.Background()) + + if len(f.calls) != 2 { + t.Fatalf("fetches = %d, want 2", len(f.calls)) + } + if f.calls[0] != popularURL { + t.Fatalf("first fetch = %q, want the shared series %q", f.calls[0], popularURL) + } +} + +// Two refusals in one pass stop the Lane: the remaining Series stay unstamped +// and due, and the Lane backs off for refuseBackoff before trying the Site +// again. One hostile Site burns only its own Lane's budget (issue #100). +func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) { + s, _ := newTestStore(t) + now := time.UnixMilli(5_000_000) + const n = 4 + for i := 0; i < n; i++ { + key := fmt.Sprintf("kagane:s%d", i) + seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0) + } + + browser := &fakeFetcher{status: 403} + tls := &fakeFetcher{status: 200} + // An asura Series sits on its own Lane: it must still be polled while + // kagane's Lane burns its budget on refusals (story 4). + seedForCheck(t, s, "asura:still-polls", "https://asurascans.com/series/still-polls", 0) + p := &Poller{ + Store: s, Fetch: tls, BrowserFetch: browser, + Now: func() time.Time { return now }, + } + + p.runOnce(context.Background()) + if got := browser.callCount(); got != 2 { + t.Fatalf("browser fetches after first pass = %d, want 2 (refused twice)", got) + } + if got := tls.callCount(); got != 1 { + t.Fatalf("asura fetches after first pass = %d, want 1 (its Lane is independent)", got) + } + stamped := 0 + for i := 0; i < n; i++ { + if readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)) == now.UnixMilli() { + stamped++ + } + } + if stamped != 2 { + t.Fatalf("stamped series = %d, want 2; the remaining two stay due", stamped) + } + + // Inside the backoff window nothing is attempted. + p.Now = func() time.Time { return now.Add(14 * time.Minute) } + p.runOnce(context.Background()) + if got := browser.callCount(); got != 2 { + t.Fatalf("browser fetches inside backoff = %d, want still 2", got) + } + + // Past the backoff the Lane resumes and the two untried Series are polled. + p.Now = func() time.Time { return now.Add(16 * time.Minute) } + p.runOnce(context.Background()) + if got := browser.callCount(); got != 4 { + t.Fatalf("browser fetches after backoff = %d, want 4", got) + } + for i := 0; i < n; i++ { + if got := readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)); got == 0 { + t.Fatalf("kagane:s%d still untried after backoff", i) + } + } +} + +// Below five due Series with none waiting long, a browser Lane leaves Chrome +// asleep; five due, or one waiting browserWakeAge, wakes it (ADR-0005). +func TestBrowserLaneWakeThresholds(t *testing.T) { + s, _ := newTestStore(t) + now := time.UnixMilli(5_000_000) + // Freshly due: checked two minutes before the rest elapses, so the wait + // is far below browserWakeAge. + seed := func(i int) { + key := fmt.Sprintf("kagane:w%d", i) + seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], now.Add(-62*time.Minute).UnixMilli()) + } + for i := 0; i < 3; i++ { + seed(i) + } + + browser := &fakeFetcher{body: kaganeAPIFixture, status: 200} + p := &Poller{ + Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser, + Now: func() time.Time { return now }, + } + + p.runOnce(context.Background()) + if got := browser.callCount(); got != 0 { + t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got) + } + // 5 due crosses the count threshold. + for i := 3; i < 5; i++ { + seed(i) + } + p.runOnce(context.Background()) + if got := browser.callCount(); got != 5 { + t.Fatalf("browser fetches with 5 due series = %d, want 5", got) + } + // A single long-neglected series wakes the browser by age alone. + seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0) + p.runOnce(context.Background()) + if got := browser.callCount(); got != 6 { + t.Fatalf("browser fetches with one ancient series = %d, want 6", got) + } +} + +// When one browser Lane loses the sidecar, the round's remaining browser +// Lanes are skipped: every fetch would fail anyway, and their Series must not +// burn their stamps on a dead Chrome (issue #100). +func TestRunOnceUnreachableBrowserStopsBrowserLanes(t *testing.T) { + s, _ := newTestStore(t) + now := time.UnixMilli(5_000_000) + seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0) + seedForCheck(t, s, "kagane:k", "https://kagane.to/series/k", 0) + seedForCheck(t, s, "novelfull:n", "https://novelfull.com/n.html", 0) + + interrupted := fmt.Errorf("%w: %w", errBrowserInterrupted, errors.New("restart")) + browser := &fakeFetcher{status: 200, err: interrupted} + p := &Poller{ + Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser, + Now: func() time.Time { return now }, + } + p.runOnce(context.Background()) + + // Sorted lane order: comix, kagane, novelfull. Only comix attempted. + if got := browser.callCount(); got != 1 { + t.Fatalf("browser fetches = %d, want 1 (only the first browser lane)", got) + } + if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() { + t.Fatalf("comix stamp = %d, want %d", got, now.UnixMilli()) + } + for _, key := range []string{"kagane:k", "novelfull:n"} { + if got := readLatestCheckedAt(t, s, key); got != 0 { + t.Fatalf("%s stamp = %d, want 0 (untried)", key, got) + } + } + + // The shared flag decays after refuseBackoff: the next round probes + // again. comix's Series is resting (stamped last round), so the probe + // falls to kagane — the only Lane with something due — and its fresh + // loss re-gates the Lanes behind it. + p.Now = func() time.Time { return now.Add(16 * time.Minute) } + p.runOnce(context.Background()) + if got := browser.callCount(); got != 2 { + t.Fatalf("browser fetches after backoff decay = %d, want 2 (kagane probes again)", got) + } + if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() { + t.Fatalf("comix stamp after decay = %d, want %d (resting, untouched)", got, now.UnixMilli()) + } + if got := readLatestCheckedAt(t, s, "kagane:k"); got != now.Add(16*time.Minute).UnixMilli() { + t.Fatalf("kagane stamp after decay = %d, want %d (the probe)", got, now.Add(16*time.Minute).UnixMilli()) + } + if got := readLatestCheckedAt(t, s, "novelfull:n"); got != 0 { + t.Fatalf("novelfull stamp after decay = %d, want 0 (gated by kagane's fresh loss)", got) + } +} + +// A browser Lane that cannot keep up logs the backlog every pass, so the +// decision to give browser Sites more pages is measured, not guessed. +func TestRunOnceLogsBrowserLaneBehind(t *testing.T) { + s, _ := newTestStore(t) + now := time.UnixMilli(5_000_000) + // Checked three rests ago: two rests' worth of wait past the due moment. + seedForCheck(t, s, "kagane:old", "https://kagane.to/series/old", now.Add(-3*time.Hour).UnixMilli()) + + var logs strings.Builder + prev := log.Writer() + log.SetOutput(&logs) + t.Cleanup(func() { log.SetOutput(prev) }) + + p := &Poller{ + Store: s, Fetch: &fakeFetcher{status: 200}, + BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, + Now: func() time.Time { return now }, + } + p.runOnce(context.Background()) + + if got := logs.String(); !strings.Contains(got, "latest poll kagane: browser lane behind by 1h0m0s") { + t.Fatalf("behind log = %q, want it to name kagane and the backlog", got) + } +} + +// gatedCoverFetcher blocks every Fetch on a gate, so a test can hold a cover +// heal in flight and prove a Lane does not wait for it. +type gatedCoverFetcher struct { + inner *fakeBytesCoverFetcher + gate chan struct{} + started chan struct{} + once sync.Once +} + +func (g *gatedCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte, string, error) { + g.once.Do(func() { g.started <- struct{}{} }) + <-g.gate + return g.inner.Fetch(ctx, sourceURL) +} + +// Cover heals run in the background: a heal stuck on a slow CDN must not +// delay the Lane's next Series-page Poll, or a large import with many blanks +// would make every Latest Chapter go stale (issue #100). +func TestRunOnceCoverFetchDoesNotDelayNextPoll(t *testing.T) { + s, dbURL := newTestStore(t) + // Two Series whose legacy cover sources still need healing. + for i := 0; i < 2; i++ { + key := fmt.Sprintf("asura:cover-%d", i) + seriesID := fmt.Sprintf("cover-%d", i) + if _, err := s.Upsert(s.OwnerID(), store.Bookmark{ + Key: key, Site: "asura", SeriesID: seriesID, + SeriesURL: "https://asurascans.com/comics/" + seriesID, UpdatedAt: 1000, + }); err != nil { + t.Fatalf("seed: %v", err) + } + seedCoverSource(t, dbURL, "asura", seriesID, fmt.Sprintf("https://cdn.example/covers/%d.jpg", i)) + } + + pages := &fakeFetcher{body: asuraSeriesFixture, status: 200} + gated := &gatedCoverFetcher{ + inner: &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}, + gate: make(chan struct{}), + started: make(chan struct{}, 1), + } + p := &Poller{ + Store: s, Fetch: pages, CoverBytesFetch: gated, + Now: func() time.Time { return time.UnixMilli(5_000_000) }, + } + + done := make(chan struct{}) + go func() { + p.runOnce(context.Background()) + close(done) + }() + <-gated.started // the first cover heal is now stuck on its CDN + select { + case <-done: + // The Lane finished its page fetches without waiting for the cover. + case <-time.After(5 * time.Second): + t.Fatal("runOnce blocked on an in-flight cover fetch") + } + close(gated.gate) + p.waitCovers() + + if got := pages.callCount(); got != 2 { + t.Fatalf("series page fetches = %d, want 2", got) + } + if got := gated.inner.callCount(); got != 2 { + t.Fatalf("cover fetches = %d, want 2", got) + } +} + +// At 3601 eligible Series the effective gap falls below the one-second floor; +// the clamp warning must name the Site so the operator knows which Lane is +// outrunning its plan. +func TestRunOnceClampWarningNamesTheSite(t *testing.T) { + s, dbURL := newTestStore(t) + db, err := sql.Open("pgx", dbURL) + if err != nil { + t.Fatalf("open %s: %v", dbURL, err) + } + defer db.Close() + if _, err := db.Exec(`INSERT INTO series (site, series_id, series_url, latest_checked_at) + SELECT 'asura', 'bulk-' || g, 'https://asurascans.com/comics/bulk-' || g, 0 + FROM generate_series(1, 3601) AS g`); err != nil { + t.Fatalf("bulk seed series: %v", err) + } + if _, err := db.Exec(`INSERT INTO bookmarks (reader_id, site, series_id, updated_at) + SELECT (SELECT id FROM readers ORDER BY id LIMIT 1), 'asura', 'bulk-' || g, 1000 + FROM generate_series(1, 3601) AS g`); err != nil { + t.Fatalf("bulk seed bookmarks: %v", err) + } + + var logs strings.Builder + prev := log.Writer() + log.SetOutput(&logs) + t.Cleanup(func() { log.SetOutput(prev) }) + + newTestPoller(t, s, &fakeFetcher{body: "", status: 200}, time.UnixMilli(5_000_000)). + runOnce(context.Background()) + + if got := logs.String(); !strings.Contains(got, "latest poll asura: gap clamped to 1s floor (eligible series=3601)") { + t.Fatalf("clamp warning = %q, want it to name asura and 3601", got) + } +} diff --git a/backend/internal/latest/read.go b/backend/internal/latest/read.go index f830d8b..dcf8d4b 100644 --- a/backend/internal/latest/read.go +++ b/backend/internal/latest/read.go @@ -9,7 +9,7 @@ import ( // seriesRead carries the two facts the poll and the acquirer both extract // from a series page. Persistence, stamps and scheduling stay with the // callers, so the policies that keep the two flows distinct (stamp order, -// cooldowns) are not swallowed by the module. +// rests) are not swallowed by the module. type seriesRead struct { Latest latestChapter HasLatest bool @@ -22,6 +22,9 @@ type seriesRead struct { // errNotFetchable and errNoFetcher separate the gate and the route from fetch // failures so each caller keeps its own distinct log line for all three. +// errChallengeHeld (browser.go) is the outcome of a Site that answered with +// its interstitial — status 403 (cf-mitigated) or a challenge page body — and +// is how a Lane tells a refusal from an ordinary failure (issue #100). var ( errNotFetchable = errors.New("series url not fetchable") errNoFetcher = errors.New("no fetcher for site") @@ -49,9 +52,21 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe if err != nil { return seriesRead{}, fmt.Errorf("fetch %s: %w", seriesURL, err) } + if status == 403 { + // Cloudflare's challenge response for these Sites (cf-mitigated). The + // browser fetcher returns exactly this on a held interstitial, and a + // plain-TLS 403 means the same: the Site is refusing. + return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status) + } if status != 200 { return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status) } + if isInterstitial(body) { + // A 200 that is the challenge page, not the payload: the TLS route can + // receive this where the browser would have kept re-reading. Same + // refusal as the 403. + return seriesRead{}, fmt.Errorf("%w: fetch %s: interstitial body", errChallengeHeld, seriesURL) + } latest, hasLatest := latestChapterFrom(site, seriesURL, body) cover, hasCover := coverFrom(site, seriesURL, body) return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil diff --git a/backend/internal/latest/sites.go b/backend/internal/latest/sites.go index 12009b1..0b7182d 100644 --- a/backend/internal/latest/sites.go +++ b/backend/internal/latest/sites.go @@ -9,6 +9,7 @@ import ( "sort" "strconv" "strings" + "time" "github.com/chromedp/chromedp" ) @@ -32,6 +33,11 @@ type site struct { LatestChapter func(seriesURL, body string) (latestChapter, bool) // Cover finds the Cover address in a fetched body. Cover func(seriesURL, body string) (string, bool) + // Rest is how long a Series of this Site rests between Polls. + Rest time.Duration + // Gap is the Lane's strictest pace: at least one second must pass between + // two consecutive Series-page Polls of this Site (issue #100). + Gap time.Duration // Browser reads this Site's payload from a cleared browser tab; nil // means the page is fetched over plain TLS. Browser *browserRead @@ -380,6 +386,50 @@ func publishedCoverURL(value string) string { return strings.ReplaceAll(value, " ", "%20") } +// Poll Lane constants (issue #100). The per-Site structure is deliberately +// uniform at first — every Site rests an hour and gaps ten seconds — but it +// exists so a single Site can be slowed if it turns hostile, and the numbers +// stay in the registry so the structure has a place to differ. +const ( + // defaultRest is how long every Series rests between Polls. + defaultRest = time.Hour + // defaultGap is the strictest pace of every Lane unless the eligible + // Series count forces it tighter. + defaultGap = 10 * time.Second + // minGap floors the effective gap. One request per second is already an + // order of magnitude past the strictest rate rule a free-plan Site can + // express (docs/research/cloudflare-bot-scoring-and-poll-cadence.md); + // below it the Lane is outrunning its own plan and says so loudly. + minGap = time.Second + // refuseBackoff is how long a Lane waits after its Site refused twice in + // one run before attempting it again. + refuseBackoff = 15 * time.Minute + // browserWakeCount and browserWakeAge gate a browser Lane's run: five or + // more due Series, or any one of them waiting this long, or Chrome stays + // asleep (ADR-0005 on-demand browser). + browserWakeCount = 5 + browserWakeAge = 15 * time.Minute +) + +// effectiveGap is a Site's pace: the registry gap, or one rest divided by the +// eligible Series count when that is smaller, never below one second. The +// denominator follows defaultRest rather than a literal hour so a Site whose +// rest is ever changed keeps its per-Series pace in step. The second return is +// true when the one-second floor engaged (and the Lane logs a warning naming +// the Site, every round it does). +func effectiveGap(s site, eligible int) (time.Duration, bool) { + gap := s.Gap + if eligible > 0 { + if perSeries := defaultRest / time.Duration(eligible); perSeries < gap { + gap = perSeries + } + } + if gap < minGap { + return minGap, true + } + return gap, false +} + // sites is the registry: one entry per Site, keyed by the stored site string. // Adding a Site means adding an entry here and nowhere else — the dispatch // functions above and the poller's route list are lookups into this map. An @@ -390,16 +440,22 @@ var sites = map[string]site{ Host: "asurascans.com", LatestChapter: asuraLatestChapter, Cover: ogImageCover, + Rest: defaultRest, + Gap: defaultGap, }, "demonic": { Host: "demonicscans.org", LatestChapter: demonicLatestChapter, Cover: ogImageCover, + Rest: defaultRest, + Gap: defaultGap, }, "comix": { Host: "comix.to", LatestChapter: comixLatestChapter, Cover: comixCoverEntry, + Rest: defaultRest, + Gap: defaultGap, Browser: &browserRead{ Read: comixRead, // The interstitial is served in place of the page, so "arrived" @@ -414,6 +470,8 @@ var sites = map[string]site{ Host: "kagane.to", LatestChapter: kaganeLatestChapter, Cover: kaganeCoverEntry, + Rest: defaultRest, + Gap: defaultGap, Browser: &browserRead{ Read: kaganeRead, Done: func(body string) bool { return body != "" }, @@ -426,6 +484,8 @@ var sites = map[string]site{ Host: "novelfull.com", LatestChapter: novelfullLatestChapter, Cover: novelfullCoverEntry, + Rest: defaultRest, + Gap: defaultGap, Browser: &browserRead{ Read: novelfullRead, // The interstitial has a DOM too, so "the payload arrived" has to @@ -438,13 +498,15 @@ var sites = map[string]site{ Host: "lightnovelworld.net", LatestChapter: lnwLatestChapter, Cover: ogImageCover, + Rest: defaultRest, + Gap: defaultGap, }, } // browserBackedSites is derived from the registry: the Sites whose pages are -// read through the browser sidecar, which are also the ones granted the longer -// cooldown. Sorted so callers that range it (the due query, the browser -// fetcher's dispatch) see a stable order instead of map-iteration noise. +// read through the browser sidecar. Sorted so callers that range it (the +// browser fetcher's dispatch) see a stable order instead of map-iteration +// noise. func browserBackedSites() []string { out := make([]string, 0, len(sites)) for name, s := range sites { diff --git a/backend/internal/store/store.go b/backend/internal/store/store.go index 8cb1264..f9f8d5d 100644 --- a/backend/internal/store/store.go +++ b/backend/internal/store/store.go @@ -16,7 +16,6 @@ import ( "strconv" "strings" - "github.com/jackc/pgx/v5/pgtype" _ "github.com/jackc/pgx/v5/stdlib" ) @@ -39,7 +38,7 @@ type Bookmark struct { // origin once the bytes exist, and "" until they do — never a third-party // address and never an address that 404s (ADR-0007). A client may still // send this field and it is discarded on the way in; see Upsert. - Cover string `json:"cover"` + Cover string `json:"cover"` LastChapter string `json:"last_chapter"` LastChapterNum float64 `json:"last_chapter_num"` LastChapterURL string `json:"last_chapter_url"` @@ -897,10 +896,13 @@ func (s *Store) Delete(readerID int64, key string) error { return nil } -// DueForLatestCheck returns series whose server-side latest-chapter check has -// aged past the appropriate cutoff, ordered by how many bookmarks reference -// them (descending) then least-recently-checked first, at most limit of them. -// Browser-backed sites use browserCutoffMs; every other site uses cutoffMs. +// DueForLatestCheck returns one Site's series whose server-side +// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks +// reference them (descending) then least-recently-checked first. One Site per +// query, because each Poll Lane asks for its own list: the query carries one +// Site and one cut-off instead of parallel lists (issue #100). There is no +// limit — the Lane's own gap paces the fetches, and the batch size that used +// to cap this query is gone with the shared pace. // // The reader_count ordering is the point of the split (ADR-0003): a series // shared by several readers is fetched once per due cycle, and the popular @@ -916,20 +918,17 @@ func (s *Store) Delete(readerID int64, key string) error { // burns requests. Archived bookmarks still count — knowing what a shelved // series is up to is the whole reason for archiving instead of deleting. // A series with no bookmarks at all never appears: the join excludes it. -func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites []string, limit int) ([]Series, error) { +func (s *Store) DueForLatestCheck(site string, cutoffMs int64) ([]Series, error) { rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count FROM series s JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id - WHERE s.series_url <> '' - AND s.latest_checked_at <= CASE - WHEN s.site = ANY($3::text[]) THEN $2::bigint - ELSE $1::bigint - END + WHERE s.site = $1 + AND s.series_url <> '' + AND s.latest_checked_at <= $2::bigint GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover, s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0 - ORDER BY reader_count DESC, s.latest_checked_at ASC - LIMIT $4`, cutoffMs, browserCutoffMs, pgtype.FlatArray[string](browserSites), limit) + ORDER BY reader_count DESC, s.latest_checked_at ASC`, site, cutoffMs) if err != nil { return nil, fmt.Errorf("query due series: %w", err) } @@ -946,6 +945,30 @@ func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites return out, rows.Err() } +// EligibleSeriesCount returns how many of a Site's Series still have at least +// one bookmark outside the finished bucket. It is the denominator of the +// Lane's pace (issue #100): the effective gap is the smaller of the registry +// gap and one hour divided by this count, so Series that will never be Polled +// do not make the Lane faster than it needs to be, and counting every eligible +// Series rather than only those currently due keeps the pace steady — the +// single worst moment to be fastest is startup, when everything is due at +// once. +func (s *Store) EligibleSeriesCount(site string) (int, error) { + var n int + err := s.db.QueryRow(`SELECT COUNT(*) FROM ( + SELECT 1 + FROM series s + JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id + WHERE s.site = $1 + GROUP BY s.site, s.series_id + HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0 + ) e`, site).Scan(&n) + if err != nil { + return 0, fmt.Errorf("count eligible series %s: %w", site, err) + } + return n, nil +} + // MarkLatestChecked records that the server looked at a series at ts, whatever // the look turned up. Marking a missing series is not an error: the row may // have been orphaned while a fetch was in flight. @@ -954,7 +977,7 @@ func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites // out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes // a whole Bookmark from the client and Upsert writes every series column it // knows about, so a userscript PUT — which has no idea this field exists — -// would write a zero and reset the cooldown, making the poller re-fetch that +// would write a zero and reset the rest, making the poller re-fetch that // series every tick for as long as the user kept reading it. func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error { if _, err := s.db.Exec( @@ -966,7 +989,7 @@ func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error { } // LatestCheckedAt reads the column MarkLatestChecked writes. It exists for -// tests outside this package (the poller's own tests assert on cooldown +// tests outside this package (the poller's own tests assert on rest // bookkeeping) — see MarkLatestChecked for why the field stays off the // client-visible row. func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) { diff --git a/backend/internal/store/store_test.go b/backend/internal/store/store_test.go index e273288..93d93ab 100644 --- a/backend/internal/store/store_test.go +++ b/backend/internal/store/store_test.go @@ -298,7 +298,7 @@ func TestDueForLatestCheck(t *testing.T) { s := newTestStore(t) seedForCheck(t, s, "asura:x", tt.seriesURL, tt.checkedAt) - due, err := s.DueForLatestCheck(now-hour, now-hour, nil, 10) + due, err := s.DueForLatestCheck("asura", now-hour) if err != nil { t.Fatalf("DueForLatestCheck: %v", err) } @@ -309,22 +309,25 @@ func TestDueForLatestCheck(t *testing.T) { } } -func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) { +func TestDueForLatestCheckOldestFirstAndScopedToSite(t *testing.T) { s := newTestStore(t) // Insert newest-checked first so a correct ORDER BY has to reverse it. seedForCheck(t, s, "asura:c", "https://asurascans.com/comics/c", 300) seedForCheck(t, s, "asura:b", "https://asurascans.com/comics/b", 200) seedForCheck(t, s, "asura:a", "https://asurascans.com/comics/a", 100) + // A second Site's due series must not appear in asura's list: each Lane + // asks for one Site, and no Lane may see another's queue. + seedForCheck(t, s, "demonic:z", "https://demonicscans.org/manga/z", 0) - due, err := s.DueForLatestCheck(1000, 1000, nil, 2) + due, err := s.DueForLatestCheck("asura", 1000) if err != nil { t.Fatalf("DueForLatestCheck: %v", err) } - if len(due) != 2 { - t.Fatalf("got %d rows, want 2 (limit)", len(due)) + if len(due) != 3 { + t.Fatalf("got %d rows, want 3 (all of asura's, none of demonic's)", len(due)) } - if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" { - t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key()) + if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" || due[2].Key() != "asura:c" { + t.Fatalf("got %q,%q,%q; want asura:a,asura:b,asura:c (oldest first)", due[0].Key(), due[1].Key(), due[2].Key()) } } @@ -496,7 +499,7 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) { } } - due, err := store.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10) + due, err := store.DueForLatestCheck("asura", time.Now().UnixMilli()) if err != nil { t.Fatalf("DueForLatestCheck: %v", err) } @@ -512,6 +515,38 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) { } } +// The gap's denominator counts every Series the Lane will ever Poll: a +// finished Series must not make the Lane faster than it needs to be, and +// another Site's Series must not leak into this Site's count. +func TestEligibleSeriesCount(t *testing.T) { + store := newTestStore(t) + seedForCheck(t, store, "asura:reading", "https://asurascans.com/comics/reading", 0) + seedForCheck(t, store, "asura:archived", "https://asurascans.com/comics/archived", 0) + if _, err := store.Upsert(store.OwnerID(), Bookmark{ + Key: "asura:finished", Site: "asura", SeriesID: "finished", + SeriesURL: "https://asurascans.com/comics/finished", + Status: StatusFinished, UpdatedAt: 1000, + }); err != nil { + t.Fatalf("seed finished: %v", err) + } + seedForCheck(t, store, "demonic:z", "https://demonicscans.org/manga/z", 0) + + n, err := store.EligibleSeriesCount("asura") + if err != nil { + t.Fatalf("EligibleSeriesCount: %v", err) + } + if n != 2 { + t.Fatalf("eligible = %d, want 2 (finished excluded, demonic excluded)", n) + } + n, err = store.EligibleSeriesCount("demonic") + if err != nil { + t.Fatalf("EligibleSeriesCount(demonic): %v", err) + } + if n != 1 { + t.Fatalf("eligible(demonic) = %d, want 1", n) + } +} + func TestDisplayChapter(t *testing.T) { cases := []struct { name string @@ -970,7 +1005,7 @@ func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) { seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001) seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100) - due, err := s.DueForLatestCheck(1000, 1000, nil, 10) + due, err := s.DueForLatestCheck("asura", 1000) if err != nil { t.Fatalf("DueForLatestCheck: %v", err) } @@ -996,7 +1031,7 @@ func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) { t.Fatalf("seed orphan series: %v", err) } - due, err := s.DueForLatestCheck(1000, 1000, nil, 10) + due, err := s.DueForLatestCheck("asura", 1000) if err != nil { t.Fatalf("DueForLatestCheck: %v", err) } @@ -1334,7 +1369,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) { t.Fatalf("series rows = %d, want 1 shared row for two bookmarks", series) } - due, err := s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10) + due, err := s.DueForLatestCheck("asura", time.Now().UnixMilli()) if err != nil { t.Fatalf("DueForLatestCheck: %v", err) } @@ -1350,7 +1385,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) { if b, ok, err := s.Get(s.OwnerID(), "asura:solo"); err != nil || !ok || b.LastChapterNum != 200 { t.Fatalf("owner's bookmark after the other's delete = %+v ok=%v err=%v, want it intact", b, ok, err) } - due, err = s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10) + due, err = s.DueForLatestCheck("asura", time.Now().UnixMilli()) if err != nil { t.Fatalf("DueForLatestCheck after delete: %v", err) } diff --git a/backend/main.go b/backend/main.go index 2a8f3c7..74f16cd 100644 --- a/backend/main.go +++ b/backend/main.go @@ -7,7 +7,6 @@ import ( "net/http" "os" "os/signal" - "strconv" "strings" "syscall" "time" @@ -61,30 +60,15 @@ type Config struct { // LatestPoll configures the background latest-chapter poller. // -// Sizing: batch x (cooldown / interval) is how many series hold a true cooldown -// cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this -// deployment. Past that nothing breaks; the effective cadence stretches to -// N x interval / batch and the oldest-checked-first ordering keeps it uniform. +// Only the kill switch lives here. Pace is per Site — rest time and gap are +// registry properties (internal/latest/sites.go, issue #100), because each +// Lane has to be able to differ from the others. The five environment +// settings that used to size a shared pace (cooldown, browser cooldown, +// interval, stagger, batch) are gone with it: no deployed .env may carry them. type LatestPoll struct { - Enabled bool - Cooldown time.Duration - BrowserCooldown time.Duration - Interval time.Duration - Stagger time.Duration - Batch int + Enabled bool } -const ( - defaultPollCooldown = time.Hour - defaultBrowserPollCooldown = 6 * time.Hour - defaultPollInterval = 10 * time.Minute - defaultPollStagger = 20 * time.Second - defaultPollBatch = 14 - // minPollCooldown keeps a typo from turning a polite background check into - // a hammer against sites that are already bot-scoring us. - minPollCooldown = 15 * time.Minute -) - func envOr(key, def string) string { if v := os.Getenv(key); v != "" { return v @@ -107,66 +91,11 @@ func envBool(key string, def bool) bool { } } -// envDuration reads a duration env var. An unparseable or non-positive value -// falls back to def and logs rather than failing startup: the poller is an -// enhancement, and a typo in one of its knobs must not stop bookmark sync. -func envDuration(key string, def time.Duration) time.Duration { - raw := strings.TrimSpace(os.Getenv(key)) - if raw == "" { - return def - } - d, err := time.ParseDuration(raw) - if err != nil || d <= 0 { - log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def) - return def - } - return d -} - -// envInt reads a positive integer env var, with the same fallback policy. -func envInt(key string, def int) int { - raw := strings.TrimSpace(os.Getenv(key)) - if raw == "" { - return def - } - n, err := strconv.Atoi(raw) - if err != nil || n <= 0 { - log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def) - return def - } - return n -} - -func clampPollCooldown(name string, d time.Duration) time.Duration { - if d < minPollCooldown { - log.Printf("config: %s %s is below the %s floor, clamping", name, d, minPollCooldown) - return minPollCooldown - } - return d -} - -// loadLatestPoll reads the poller's settings, clamping anything that would make -// it antisocial. +// loadLatestPoll reads the poller's settings. The pace knobs that used to be +// clamped here are registry properties now (issue #100), so there is nothing +// left to clamp. func loadLatestPoll() LatestPoll { - p := LatestPoll{ - Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true), - Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown), - BrowserCooldown: envDuration("LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", defaultBrowserPollCooldown), - Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval), - Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger), - Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch), - } - p.Cooldown = clampPollCooldown("cooldown", p.Cooldown) - p.BrowserCooldown = clampPollCooldown("browser cooldown", p.BrowserCooldown) - // batch x stagger has to fit inside one tick or a batch is still running - // when the next one is due. Run() serialises them, so this degrades to a - // slower cadence rather than to overlapping fetches — worth a warning, not - // a failure. - if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval { - log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick", - p.Batch, p.Stagger, span, p.Interval) - } - return p + return LatestPoll{Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true)} } func loadConfig() Config { @@ -377,7 +306,8 @@ func main() { } } -// newLatestPoller wires the configured cooldowns and fetchers into the poller. +// newLatestPoller wires the fetcher seams into the poller. Pace is registry +// property, not config (issue #100), so there are no knobs to pass through. func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller { var covers latest.BrowserCoverFetcher if f, ok := browser.(latest.BrowserCoverFetcher); ok { @@ -390,11 +320,6 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch CoverFetch: covers, CoverBytesFetch: latest.NewCoverFetcher(), Now: time.Now, - Cooldown: cfg.Cooldown, - BrowserCooldown: cfg.BrowserCooldown, - Interval: cfg.Interval, - Stagger: cfg.Stagger, - Batch: cfg.Batch, } } diff --git a/backend/main_test.go b/backend/main_test.go index 52f2fbd..5960b48 100644 --- a/backend/main_test.go +++ b/backend/main_test.go @@ -9,31 +9,15 @@ import ( "net/http/httptest" "strings" "testing" - "time" + "bookmarkmanager/backend/internal/latest" "bookmarkmanager/backend/internal/store" ) func TestLoadLatestPollDefaults(t *testing.T) { - for _, k := range []string{ - "LATEST_CHAPTER_POLL_ENABLED", "LATEST_CHAPTER_POLL_COOLDOWN", - "LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", "LATEST_CHAPTER_POLL_INTERVAL", - "LATEST_CHAPTER_POLL_STAGGER", "LATEST_CHAPTER_POLL_BATCH", - } { - t.Setenv(k, "") - } - - got := loadLatestPoll() - want := LatestPoll{ - Enabled: true, - Cooldown: time.Hour, - BrowserCooldown: 6 * time.Hour, - Interval: 10 * time.Minute, - Stagger: 20 * time.Second, - Batch: 14, - } - if got != want { - t.Fatalf("loadLatestPoll() = %+v, want %+v", got, want) + t.Setenv("LATEST_CHAPTER_POLL_ENABLED", "") + if got := loadLatestPoll(); got != (LatestPoll{Enabled: true}) { + t.Fatalf("loadLatestPoll() = %+v, want %+v", got, LatestPoll{Enabled: true}) } } @@ -63,105 +47,25 @@ func TestLoadLatestPollEnabledParsing(t *testing.T) { } } -func TestLoadLatestPollClampsAndFallsBack(t *testing.T) { - tests := []struct { - name string - env map[string]string - wantFrom func(LatestPoll) any - want any - }{ - { - name: "cooldown below the floor is clamped up", - env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "1m"}, - wantFrom: func(p LatestPoll) any { return p.Cooldown }, - want: 15 * time.Minute, - }, - { - name: "cooldown at the floor is kept", - env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "15m"}, - wantFrom: func(p LatestPoll) any { return p.Cooldown }, - want: 15 * time.Minute, - }, - { - name: "browser cooldown below the floor is clamped up", - env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "1m"}, - wantFrom: func(p LatestPoll) any { return p.BrowserCooldown }, - want: 15 * time.Minute, - }, - { - name: "browser cooldown at the floor is kept", - env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "15m"}, - wantFrom: func(p LatestPoll) any { return p.BrowserCooldown }, - want: 15 * time.Minute, - }, - { - name: "browser cooldown override is honoured", - env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "8h"}, - wantFrom: func(p LatestPoll) any { return p.BrowserCooldown }, - want: 8 * time.Hour, - }, - { - name: "browser cooldown unparseable value falls back", - env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "six hours"}, - wantFrom: func(p LatestPoll) any { return p.BrowserCooldown }, - want: 6 * time.Hour, - }, - { - name: "a valid override is honoured", - env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "5m"}, - wantFrom: func(p LatestPoll) any { return p.Interval }, - want: 5 * time.Minute, - }, - { - name: "an unparseable duration falls back", - env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "ten minutes"}, - wantFrom: func(p LatestPoll) any { return p.Interval }, - want: 10 * time.Minute, - }, - { - name: "a zero duration falls back", - env: map[string]string{"LATEST_CHAPTER_POLL_STAGGER": "0s"}, - wantFrom: func(p LatestPoll) any { return p.Stagger }, - want: 20 * time.Second, - }, - { - name: "a valid batch is honoured", - env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "30"}, - wantFrom: func(p LatestPoll) any { return p.Batch }, - want: 30, - }, - { - name: "a negative batch falls back", - env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "-5"}, - wantFrom: func(p LatestPoll) any { return p.Batch }, - want: 14, - }, - { - name: "a non-numeric batch falls back", - env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "lots"}, - wantFrom: func(p LatestPoll) any { return p.Batch }, - want: 14, - }, +// newLatestPoller wires the fetcher seams; pace lives in the registry, so +// nothing here sizes a cooldown any more. +func TestNewLatestPollerWiresFetchers(t *testing.T) { + tls := &latest.TLSFetcher{} + p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil) + if p.Fetch != tls { + t.Fatalf("Fetch not wired") } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - for k, v := range tt.env { - t.Setenv(k, v) - } - if got := tt.wantFrom(loadLatestPoll()); got != tt.want { - t.Fatalf("got %v, want %v", got, tt.want) - } - }) + if p.BrowserFetch != nil { + t.Fatalf("BrowserFetch = %v, want nil for a browser-less deployment", p.BrowserFetch) } -} - -func TestNewLatestPollerWiresCooldowns(t *testing.T) { - p := newLatestPoller(nil, LatestPoll{ - Cooldown: time.Hour, - BrowserCooldown: 6 * time.Hour, - }, nil, nil) - if p.Cooldown != time.Hour || p.BrowserCooldown != 6*time.Hour { - t.Fatalf("poller cooldowns = %s/%s, want 1h/6h", p.Cooldown, p.BrowserCooldown) + if p.CoverFetch != nil { + t.Fatalf("CoverFetch = %v, want nil when the browser is absent", p.CoverFetch) + } + if p.CoverBytesFetch == nil { + t.Fatalf("CoverBytesFetch = nil, want the TLS cover fetcher") + } + if p.Now == nil { + t.Fatalf("Now = nil, want the live clock") } } diff --git a/docker-compose.yml b/docker-compose.yml index 3e7ef6d..de80e68 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -58,14 +58,11 @@ services: # Second script from the same bindmount; the novel library is a separate # Violentmonkey install. NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js} - # Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill - # switch; it only takes effect because these are listed here. + # Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the + # kill switch; it only takes effect because it is listed here. Pace is + # per Site in the registry (one Poll Lane per Site, issue #100) — the + # cooldown/interval/stagger/batch knobs are gone with the shared pace. LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1} - LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h} - LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h} - LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m} - LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14} - LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s} # CDP endpoint for sites behind a JavaScript challenge (kagane, # novelfull). The browser is not part of this stack — it runs on the home # machine as its own unit (chrome/docker-compose.yml) and is reached over diff --git a/docs/adr/0010-poll-lanes-per-site-pace.md b/docs/adr/0010-poll-lanes-per-site-pace.md new file mode 100644 index 0000000..2305f4a --- /dev/null +++ b/docs/adr/0010-poll-lanes-per-site-pace.md @@ -0,0 +1,91 @@ +# ADR-0010: Poll Lanes — one independent Poll stream per Site + +Date: 2026-08-16 +Status: accepted + +## Decision + +Replace the single shared polling pace with one **Poll Lane** per Site: an +independent goroutine that polls only that Site's Series, paced by that Site's +registry entry. Pace moves out of config and into the Site registry +(`internal/latest/sites.go`): every entry carries a `Rest` (how long a Series +rests between Polls) and a `Gap` (how long the Lane waits between fetches). + +Rest is enforced by the due query's WHERE clause (`latest_checked_at <= now - +Rest`), never by a timer — the same mechanism that enforced the old cooldown. +The Lane enforces its own gap by sleeping between fetches. `effectiveGap` is +the registry gap, or one hour divided by the Site's eligible Series count when +that is smaller, never below one second. + +The five environment settings that used to size the shared pace — +`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`, +`_STAGGER` — are deleted. Only the kill switch `LATEST_CHAPTER_POLL_ENABLED` +remains. No deployed `.env` may carry the deleted knobs. + +## Why + +The shared pace capped the whole backend at roughly 180 Polls an hour (one +20-second stagger across one queue). ~60 Series today, scaling to hundreds or +thousands, would stretch the hour beyond what the New Chapter signal can +tolerate. Worse, the queue mixed Sites with very different costs: kagane and +comix pay seconds of a serialized single-tab Chrome per Poll (a challenged +page, ADR-0005), and one hostile Site burning its challenge timeout made every +other Site's Series wait — "one hostile Site can eat most of an hour". + +Lanes fix both at once: + +- **Throughput scales per Site.** The six Lanes fetch concurrently; a Lane's + own gap paces it. The browser Lanes' combined ceiling stays about 360 Polls + an hour (one tab), and when they cannot keep up the wait past Rest grows and + is logged every pass — the "behind by X" measurement, so the decision to + give browser Sites more pages is made from data. +- **Hostility is contained.** A refusal (two challenge-held reads in one + pass) stops only that Site's Lane for `refuseBackoff` (15m); the rest of + that Lane's Series stay unstamped and due. A lost browser gates the other + browser Lanes' passes for the same window — the flag is shared Poller + state, so the loss is noticed once instead of once per Lane per pass, and + decays after 15m so the Lanes probe again. One Site can no longer tax the + others. + +## Tradeoffs and rejections + +- **Per-Site env knobs** (e.g. `KAGANE_POLL_GAP`) rejected: the registry is + the single place pace lives, testable and reviewable; config knobs would + recreate the shared-pace sprawl with six times the surface. All six entries + are deliberately uniform at first — rest an hour, gap ten seconds — so the + structure exists to differ without inventing numbers for Sites that have + not earned them. +- **Dynamic gap** (`rest / eligible`) is the one knob that stays automatic: + a Site with more Series than one per ten seconds would otherwise back up + behind its own gap, and the per-Series share of the hour is the natural + pace. The ten-second default is not arbitrary: one request per ten seconds + is the strictest rate rule a free-plan Site can even express (per-zone + rate limiting, as documented in + `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`), so the + default pace is exactly what the most restrictive Site would demand of us. + The computed gap never goes below one second and logs loudly when the + floor engages. +- **Timer-based pacing** rejected: the old ticker made the poller's rate a + function of wall clock rather than of what was due. The due-query cutoff is + the only rate authority; the Lane sleep just prevents hammering. +- **Batch size** (the old `_BATCH` cap) is gone with the shared pace: a Lane + processes everything due, paced by its gap. There is no global queue left + to bound. + +## Constraints preserved + +- Stamp-before-fetch ("attempted" semantics): an untried Series stays due, so + a browser that appears after a restart finds its full queue waiting. +- Browser wake gate (ADR-0005): a browser Lane leaves Chrome asleep below + five due Series and 15 minutes of wait, per Lane. +- The browser is not in the API stack (ADR-0006): an unreachable browser + degrades a Lane exactly as an unset `BROWSER_WS_URL` — browser-only Sites + skipped, plain-TLS unaffected, stored covers still served. +- Cover heals moved to background goroutines (joined by the test suite via + `waitCovers`) so a slow cover CDN cannot consume a Lane's gap. + +Supersedes the pace mechanics of ADR-0003's "raise throughput instead" note +(the stagger cut it rejected is what the per-Lane gap replaces) and the +6-hour browser cooldown introduced with the browser-backed Sites; the +1-hour browser rest was already cleared as safe by +`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. diff --git a/graphify-out/.graphify_labels.json b/graphify-out/.graphify_labels.json index 0f36568..eb7606b 100644 --- a/graphify-out/.graphify_labels.json +++ b/graphify-out/.graphify_labels.json @@ -30,7 +30,8 @@ "28": "Allocation Patterns", "29": "Observability & Alerting", "30": "AGENTS.md", - "32": "Repo Hard Constraints", + "31": "Store", + "32": "Open", "33": "Go Testing Guide", "34": "Session Store", "35": "Web UI Filter Logic", @@ -39,7 +40,7 @@ "38": "novel-logic.test.js", "39": "UI Critique 2026-07-26A", "40": "UI Critique 2026-07-26B", - "41": "sessions_test.go", + "42": "pgtest.go", "43": "Issue Tracker & Triage", "44": "Ticket Workflow", "45": "Go Perf Alert Rules", @@ -122,6 +123,7 @@ "122": "Identity comes from Discord OAuth; we store no passwords and send no email", "123": "The wire format stays flat and deliberately does not mirror the schema", "124": "ADR-0005: On-demand browser sidecar", + "125": "sessions_test.go", "126": "Bookmark Manager", "127": "triage-labels.md", "128": "Cross-Ticket Contract", @@ -149,7 +151,6 @@ "150": "Reviewer Subagent (opencode)", "151": "Finding Severity Rubric", "152": "Spec Compliance Review", - "158": "T", "161": "Why Use samber/oops", "162": "singleflight Cache Stampede Prevention", "163": "Struct Field Alignment", diff --git a/graphify-out/GRAPH_REPORT.md b/graphify-out/GRAPH_REPORT.md index 38e5ce3..edb1344 100644 --- a/graphify-out/GRAPH_REPORT.md +++ b/graphify-out/GRAPH_REPORT.md @@ -1,16 +1,16 @@ # Graph Report - mangaBookmark (2026-08-16) ## Corpus Check -- 112 files · ~274,851 words +- 113 files · ~279,161 words - Verdict: corpus is large enough that graph structure adds value. ## Summary -- 1640 nodes · 3294 edges · 221 communities (67 shown, 154 thin omitted) -- Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 313 edges (avg confidence: 0.77) +- 1668 nodes · 3379 edges · 222 communities (68 shown, 154 thin omitted) +- Extraction: 91% EXTRACTED · 9% INFERRED · 0% AMBIGUOUS · INFERRED: 320 edges (avg confidence: 0.77) - Token cost: 0 input · 0 output ## Graph Freshness -- Built from commit: `f000cc7e` +- Built from commit: `4f1cbcfd` - Run `git rev-parse HEAD` and compare to check if the graph is stale. - Run `graphify update .` after code changes (no API cost). @@ -45,7 +45,8 @@ - [[_COMMUNITY_Find Skills Guide|Find Skills Guide]] - [[_COMMUNITY_Allocation Patterns|Allocation Patterns]] - [[_COMMUNITY_Observability & Alerting|Observability & Alerting]] -- [[_COMMUNITY_Repo Hard Constraints|Repo Hard Constraints]] +- [[_COMMUNITY_Store|Store]] +- [[_COMMUNITY_Open|Open]] - [[_COMMUNITY_Go Testing Guide|Go Testing Guide]] - [[_COMMUNITY_Session Store|Session Store]] - [[_COMMUNITY_Web UI Filter Logic|Web UI Filter Logic]] @@ -54,7 +55,7 @@ - [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]] - [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]] - [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]] -- [[_COMMUNITY_sessions_test.go|sessions_test.go]] +- [[_COMMUNITY_pgtest.go|pgtest.go]] - [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]] - [[_COMMUNITY_Ticket Workflow|Ticket Workflow]] - [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]] @@ -137,6 +138,7 @@ - [[_COMMUNITY_Identity comes from Discord OAuth; we store no passwords and send no email|Identity comes from Discord OAuth; we store no passwords and send no email]] - [[_COMMUNITY_The wire format stays flat and deliberately does not mirror the schema|The wire format stays flat and deliberately does not mirror the schema]] - [[_COMMUNITY_ADR-0005 On-demand browser sidecar|ADR-0005: On-demand browser sidecar]] +- [[_COMMUNITY_sessions_test.go|sessions_test.go]] - [[_COMMUNITY_Bookmark Manager|Bookmark Manager]] - [[_COMMUNITY_triage-labels|triage-labels.md]] - [[_COMMUNITY_Cross-Ticket Contract|Cross-Ticket Contract]] @@ -164,7 +166,6 @@ - [[_COMMUNITY_Reviewer Subagent (opencode)|Reviewer Subagent (opencode)]] - [[_COMMUNITY_Finding Severity Rubric|Finding Severity Rubric]] - [[_COMMUNITY_Spec Compliance Review|Spec Compliance Review]] -- [[_COMMUNITY_T|T]] - [[_COMMUNITY_Why Use samberoops|Why Use samber/oops]] - [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]] - [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]] @@ -239,14 +240,14 @@ ## God Nodes (most connected - your core abstractions) 1. `testConfig()` - 53 edges 2. `newWebTestServer()` - 49 edges -3. `newTestStore()` - 43 edges -4. `newTestStore()` - 42 edges +3. `newTestStore()` - 48 edges +4. `newTestStore()` - 43 edges 5. `e()` - 33 edges -6. `Handler` - 29 edges -7. `ne()` - 28 edges -8. `De()` - 28 edges -9. `Open()` - 27 edges -10. `se()` - 27 edges +6. `Open()` - 29 edges +7. `Handler` - 29 edges +8. `ne()` - 28 edges +9. `De()` - 28 edges +10. `Store` - 27 edges ## Surprising Connections (you probably didn't know these) - `el()` --indirect_call--> `c()` [INFERRED] @@ -272,7 +273,7 @@ - **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85] - **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85] -## Communities (221 total, 154 thin omitted) +## Communities (222 total, 154 thin omitted) ### Community 0 - "HTMX Library Internals" Cohesion: 0.08 @@ -284,30 +285,30 @@ Nodes (84): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecor ### Community 2 - "Manga Userscript Adapters" Cohesion: 0.06 -Nodes (77): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+69 more) +Nodes (76): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+68 more) ### Community 3 - "Novel Userscript Adapters" Cohesion: 0.06 -Nodes (78): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+70 more) +Nodes (79): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+71 more) ### Community 4 - "Series Acquisition Tests" -Cohesion: 0.09 -Nodes (69): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+61 more) +Cohesion: 0.08 +Nodes (78): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+70 more) ### Community 5 - "Bookmarks API Tests" -Cohesion: 0.08 -Nodes (67): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+59 more) +Cohesion: 0.06 +Nodes (74): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+66 more) ### Community 7 - "Cover & Acquire Internals" -Cohesion: 0.10 -Nodes (30): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+22 more) +Cohesion: 0.08 +Nodes (37): Addr, Context, Store, WaitGroup, isInterstitial(), defaultCoverResolver(), fetchCoverBytes(), Client (+29 more) ### Community 8 - "System Architecture Concepts" Cohesion: 0.13 Nodes (20): app.html — App Shell Template, Manga/Novel Library Switch, Bookmark Bucket Tabs, Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key (+12 more) ### Community 9 - "Session Middleware" -Cohesion: 0.07 +Cohesion: 0.08 Nodes (35): ClearCookie(), ClientIP(), Duration, Mutex, Request, ResponseWriter, Time, isHTTPS() (+27 more) ### Community 10 - "Go Test Helpers" @@ -315,12 +316,12 @@ Cohesion: 0.05 Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more) ### Community 11 - "Store Tests" -Cohesion: 0.15 -Nodes (43): Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck(), seedSecondReader() (+35 more) +Cohesion: 0.14 +Nodes (45): scanSeries(), Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck() (+37 more) ### Community 12 - "Bookmarks API Handler" -Cohesion: 0.11 -Nodes (24): Auth(), compressible(), CORS(), Handler, ResponseWriter, Store, Gzip(), ResolveReader() (+16 more) +Cohesion: 0.08 +Nodes (32): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+24 more) ### Community 13 - "Web UI Handlers" Cohesion: 0.14 @@ -331,8 +332,8 @@ Cohesion: 0.06 Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more) ### Community 15 - "CDP Browser Client" -Cohesion: 0.07 -Nodes (36): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+28 more) +Cohesion: 0.06 +Nodes (56): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+48 more) ### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented" Cohesion: 0.06 @@ -343,8 +344,8 @@ Cohesion: 0.08 Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more) ### Community 19 - "Store" -Cohesion: 0.09 -Nodes (8): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, scanSeries(), TestDisplayChapter(), Bookmark, ReaderSummary +Cohesion: 0.33 +Nodes (5): ADR-0010: Poll Lanes — one independent Poll stream per Site, Constraints preserved, Decision, Tradeoffs and rejections, Why ### Community 20 - "I/O Performance Patterns" Cohesion: 0.11 @@ -371,24 +372,28 @@ Cohesion: 0.33 Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more) ### Community 26 - "Open" -Cohesion: 0.20 -Nodes (17): applyMigration(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen(), TestMigration0002BackfillsExistingBookmarks() (+9 more) +Cohesion: 0.40 +Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct ### Community 27 - "Find Skills Guide" Cohesion: 0.14 Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more) ### Community 28 - "Allocation Patterns" -Cohesion: 0.11 -Nodes (19): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map of pointers for large, frequently updated structs, Map size hints (+11 more) +Cohesion: 0.14 +Nodes (14): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map size hints, Memory Optimization (+6 more) ### Community 29 - "Observability & Alerting" Cohesion: 0.22 Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more) -### Community 32 - "Repo Hard Constraints" -Cohesion: 0.18 -Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more) +### Community 31 - "Store" +Cohesion: 0.12 +Nodes (3): coverRelativePath(), coverSourceAddress(), Store + +### Community 32 - "Open" +Cohesion: 0.14 +Nodes (20): applyMigration(), displayChapter(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen() (+12 more) ### Community 33 - "Go Testing Guide" Cohesion: 0.20 @@ -418,9 +423,9 @@ Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Cohesion: 0.29 Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider -### Community 41 - "sessions_test.go" -Cohesion: 0.48 -Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID() +### Community 42 - "pgtest.go" +Cohesion: 0.18 +Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more) ### Community 45 - "Go Perf Alert Rules" Cohesion: 0.50 @@ -530,28 +535,28 @@ Nodes (3): Consequence, The wire format stays flat and deliberately does not mir Cohesion: 0.50 Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision -### Community 158 - "T" -Cohesion: 0.08 -Nodes (38): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Request, ReaderID() (+30 more) +### Community 125 - "sessions_test.go" +Cohesion: 0.48 +Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID() ### Community 273 - "AGENTS.md" Cohesion: 0.50 Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust), Second script: `novel-bookmark.user.js`, Userscript structure (single IIFE, `manga-bookmark.user.js`) ## Knowledge Gaps -- **520 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+515 more) +- **524 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+519 more) These have ≤1 connection - possible missing edges or undocumented components. - **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. ## Suggested Questions _Questions this graph is uniquely positioned to answer:_ -- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`, `Open`?** - _High betweenness centrality (0.051) - this node is a cross-community bridge._ -- **Why does `Open()` connect `Open` to `Repo Hard Constraints`, `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `Store Tests`, `Store`?** - _High betweenness centrality (0.038) - this node is a cross-community bridge._ +- **Why does `New()` connect `Series Acquisition Tests` to `Open`, `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`?** + _High betweenness centrality (0.045) - this node is a cross-community bridge._ +- **Why does `Open()` connect `Open` to `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `pgtest.go`, `Store Tests`, `Store`?** + _High betweenness centrality (0.033) - this node is a cross-community bridge._ - **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?** - _High betweenness centrality (0.030) - this node is a cross-community bridge._ + _High betweenness centrality (0.027) - this node is a cross-community bridge._ - **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?** _`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._ - **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?** @@ -559,4 +564,4 @@ _Questions this graph is uniquely positioned to answer:_ - **Are the 12 inferred relationships involving `newTestStore()` (e.g. with `TestAcquireDoesNotBlockTheWrite()` and `TestAcquireFailureLeavesTheBookmarkIntact()`) actually correct?** _`newTestStore()` has 12 INFERRED edges - model-reasoned connections that need verification._ - **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?** - _553 weakly-connected nodes found - possible documentation gaps or missing edges._ \ No newline at end of file + _557 weakly-connected nodes found - possible documentation gaps or missing edges._ \ No newline at end of file diff --git a/graphify-out/graph.html b/graphify-out/graph.html index 417d4d4..544976f 100644 --- a/graphify-out/graph.html +++ b/graphify-out/graph.html @@ -63,12 +63,12 @@
-
1640 nodes · 3294 edges · 221 communities
+
1668 nodes · 3379 edges · 222 communities