mangaBookmark/docs/adr/0010-poll-lanes-per-site-pace.md

ADR-0010: Poll Lanes — one independent Poll stream per Site

Date: 2026-08-16 Status: accepted

Decision

Replace the single shared polling pace with one Poll Lane per Site: an independent goroutine that polls only that Site's Series, paced by that Site's registry entry. Pace moves out of config and into the Site registry (internal/latest/sites.go): every entry carries a Rest (how long a Series rests between Polls) and a Gap (how long the Lane waits between fetches).

Rest is enforced by the due query's WHERE clause (latest_checked_at <= now - Rest), never by a timer — the same mechanism that enforced the old cooldown. The Lane enforces its own gap by sleeping between fetches. effectiveGap is the registry gap, or one hour divided by the Site's eligible Series count when that is smaller, never below one second.

The five environment settings that used to size the shared pace — LATEST_CHAPTER_POLL_COOLDOWN, _BROWSER_COOLDOWN, _INTERVAL, _BATCH, _STAGGER — are deleted. Only the kill switch LATEST_CHAPTER_POLL_ENABLED remains. No deployed .env may carry the deleted knobs.

Why

The shared pace capped the whole backend at roughly 180 Polls an hour (one 20-second stagger across one queue). ~60 Series today, scaling to hundreds or thousands, would stretch the hour beyond what the New Chapter signal can tolerate. Worse, the queue mixed Sites with very different costs: kagane and comix pay seconds of a serialized single-tab Chrome per Poll (a challenged page, ADR-0005), and one hostile Site burning its challenge timeout made every other Site's Series wait — "one hostile Site can eat most of an hour".

Lanes fix both at once:

• Throughput scales per Site. The six Lanes fetch concurrently; a Lane's own gap paces it. The browser Lanes' combined ceiling stays about 360 Polls an hour (one tab), and when they cannot keep up the wait past Rest grows and is logged every pass — the "behind by X" measurement, so the decision to give browser Sites more pages is made from data.
• Hostility is contained. A refusal (two challenge-held reads in one pass) stops only that Site's Lane for refuseBackoff (15m); the rest of that Lane's Series stay unstamped and due. A lost browser gates the other browser Lanes' passes for the same window — the flag is shared Poller state, so the loss is noticed once instead of once per Lane per pass, and decays after 15m so the Lanes probe again. One Site can no longer tax the others.

Tradeoffs and rejections

• Per-Site env knobs (e.g. KAGANE_POLL_GAP) rejected: the registry is the single place pace lives, testable and reviewable; config knobs would recreate the shared-pace sprawl with six times the surface. All six entries are deliberately uniform at first — rest an hour, gap ten seconds — so the structure exists to differ without inventing numbers for Sites that have not earned them.
• Dynamic gap (rest / eligible) is the one knob that stays automatic: a Site with more Series than one per ten seconds would otherwise back up behind its own gap, and the per-Series share of the hour is the natural pace. The ten-second default is not arbitrary: one request per ten seconds is the strictest rate rule a free-plan Site can even express (per-zone rate limiting, as documented in docs/research/cloudflare-bot-scoring-and-poll-cadence.md), so the default pace is exactly what the most restrictive Site would demand of us. The computed gap never goes below one second and logs loudly when the floor engages.
• Timer-based pacing rejected: the old ticker made the poller's rate a function of wall clock rather than of what was due. The due-query cutoff is the only rate authority; the Lane sleep just prevents hammering.
• Batch size (the old _BATCH cap) is gone with the shared pace: a Lane processes everything due, paced by its gap. There is no global queue left to bound.

Constraints preserved

• Stamp-before-fetch ("attempted" semantics): an untried Series stays due, so a browser that appears after a restart finds its full queue waiting.
• Browser wake gate (ADR-0005): a browser Lane leaves Chrome asleep below five due Series and 15 minutes of wait, per Lane.
• The browser is not in the API stack (ADR-0006): an unreachable browser degrades a Lane exactly as an unset BROWSER_WS_URL — browser-only Sites skipped, plain-TLS unaffected, stored covers still served.
• Cover heals moved to background goroutines (joined by the test suite via waitCovers) so a slow cover CDN cannot consume a Lane's gap.

Supersedes the pace mechanics of ADR-0003's "raise throughput instead" note (the stagger cut it rejected is what the per-Lane gap replaces) and the 6-hour browser cooldown introduced with the browser-backed Sites; the 1-hour browser rest was already cleared as safe by docs/research/cloudflare-bot-scoring-and-poll-cadence.md.