Poll Lanes: one independent Poll stream per Site, replacing the shared pace #101
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Blocked by: #98
Problem Statement
As a Reader growing my Library, my Series stop being checked every hour long before I notice it. The backend promises each Series a Poll once an hour, but the machinery underneath cannot deliver that past about 84 Series, and it fails silently — nothing logs, nothing warns, the Latest Chapter just quietly gets older.
The cause is a single shared pace. Every Poll of every Site waits behind one 20-second gap, so the whole backend can issue at most 180 Polls an hour no matter how the batch size or wake interval are set. I have roughly 60 Series today and expect several hundred, possibly a thousand. At 250 Series each one is Polled about every three hours; at 1000, about every twelve.
Two more things make it worse. Browser Sites rest for six hours rather than one, so a third of my Library is deliberately stale. And when a Site refuses — which comix, kagane and novelfull all do behind a Cloudflare challenge — each refusal burns the full challenge timeout, so one hostile Site can eat most of an hour of the shared budget and push every other Site's Polls out of the way.
Solution
Give every Site its own Poll Lane: an independent stream of Polls with its own pace and its own rest time, running at the same time as every other Site's.
The pace is per Site, so a slow or refusing Site cannot hold up the other five, and the ceiling grows with the number of Sites rather than being one shared number. Each Lane's gap defaults to ten seconds — the strictest rate rule a free-plan Site can even express is one request per ten seconds — and shortens automatically when a Site holds enough Series that ten seconds cannot fit them all in the hour. It never goes below one second, and reaching that floor logs a warning naming the Site, so the hour becomes a promise the backend keeps loudly or breaks loudly.
Every Site's rest time becomes one hour, including the three browser Sites. The separate six-hour browser rest time disappears, along with the global wake interval, batch size and shared stagger, which no longer mean anything.
User Stories
Implementation Decisions
Poll Lane. The unit of scheduling becomes the Poll Lane, now a glossary term: one Site's own stream of Polls, with its own pace and rest time, unable to slow, block or borrow from another's, and never influenced by a Reader. Every Site has exactly one. All Lanes run concurrently.
Rest time moves to the registry, one hour everywhere. Each Site's registry entry carries its rest time. All six start at one hour. The separate browser rest time and its environment variable are removed, not defaulted. The per-Site structure is deliberately uniform at first: it exists so a single Site can be slowed if it turns hostile, and that intent should be recorded as a comment, since a reader will otherwise ask why the structure exists when every value matches.
Pace. Each Site's registry entry carries a gap, defaulting to ten seconds. The effective gap is the smaller of that and one hour divided by the Site's eligible Series count, with a floor of one second. Eligible means every Series of that Site whose Bookmarks are not all in the finished Lifecycle bucket. Counting all eligible Series rather than only those currently due keeps the pace steady, and specifically avoids computing the fastest possible pace at startup, when everything is due at once — the single worst moment to be fastest.
Reaching the floor is loud. When the computed gap is clamped to one second, log a warning naming the Site, every round. At one request per second against one Site from one address, all day, the backend is ten times over the strictest rule a free-plan Site can express; that is a state the operator must never be in without knowing.
Ordering within a Lane is unchanged. Most-shared Series first, then longest-unchecked. No Reader-derived priority of any kind: shortage is absorbed by the long tail, not by an unlucky Reader. This is already how the due query orders, and it is already the fair answer, because a Poll is work done on behalf of the Series.
Removed settings. The global wake interval, batch size and shared stagger are deleted from the code, the example environment file, the README and the Compose file. They are not kept as ignored values. Their tests go with them. The deploy notes must list the exact environment edit this requires, because a deployed environment file will still carry the old names.
Browser Sites are not selected when the browser is absent. Today they are selected, stamped as checked, and then fail for want of a fetcher, which ages every browser Series by an hour on every tick of a browserless deployment. The fix moves the stamp to after a non-challenge result, so an untried Series keeps its place in the queue. This is a change to the invariant that a Series is stamped before it is fetched, and it needs its own test.
A Site refuses. After two consecutive challenge-held results in one run, that Lane skips its remaining Series for the rest of the run. The two probe Series keep their stamp; the untried ones were never stamped and stay due. The Lane then waits fifteen minutes before attempting that Site again. Counting refusals per Site rather than globally means an unreachable Chrome — which is about all three browser Sites at once — costs two probes per Site before everything stops, which is a small and acceptable extra cost for keeping one counter rule instead of two.
Chrome unreachable. All three browser Lanes stop for that run, not only the Lane that noticed.
Browser wake. A browser Lane starts a run when five or more of its Series are due, or when any one of its Series has been due for fifteen minutes. Below both thresholds it leaves Chrome asleep. Series Polled together become due together, so they naturally stay clustered; these two numbers exist to stop a Series that was added late, or that failed once, from drifting out of the group and waking Chrome on its own.
Browser throughput. One page at a time, shared by all three browser Sites, unchanged. Their combined ceiling is therefore about 360 Polls an hour rather than 1080. That covers the three browser Sites unless they hold more than 360 Series between them. When the shared page cannot keep up, the browser Lanes stretch past the hour and log by how much. Giving each browser Site its own page stays on the shelf until Chrome's memory with three concurrent pages has been measured on the machine that hosts it.
Covers do not consume a Lane's gap. A Series still missing its Cover costs a second request, often to a different host with its own budget. Counting Cover requests against the Series-page gap would mean a large import makes every Latest Chapter go stale. The double cost happens once in a Series's life.
Capacity, for the record. Six Sites at one request per ten seconds is 2160 Polls an hour, against a target of 1000. The hour holds as long as no single Site holds more than 360 Series; past that, the auto-shortening takes over, and past 3600 Series on one Site the floor engages and the hour is formally broken with a warning.
Testing Decisions
A good test here asserts observable scheduling behaviour: which Series were fetched, from which Site, in what order, how far apart, and what was left unstamped. It must not assert how Lanes are structured internally, how many goroutines exist, or what the snapshot type looks like.
Everything lands at the existing Poller seam. The Poller is already constructed in tests with an injected page fetcher, an injected browser fetcher, an injected clock and a real Postgres, and driven a round at a time. Roughly thirty tests already sit there. That seam carries this entire spec, and no new seam is needed. It does require one deterministic entry point equivalent to today's single-round call, so that a test can run exactly one round of every Lane without real time passing.
The behaviours to cover, all at that seam:
Prior art for every one of these is in the existing poller tests: the cooldown-across-passes test, the tests that assert a shared Series is fetched once, the tests that assert routing to the browser fetcher, and the table test over the fetchable-Series-URL gate.
Configuration tests follow the existing pattern of asserting that a named environment variable produces a given field value; the tests for the removed settings are deleted rather than adapted.
Out of Scope
Further Notes
The ten-second default is not arbitrary and should not be tuned away without reading the reasoning. The supporting research established that no Site in this set publishes a crawl delay applicable to a general client — the only delays published name AI crawlers — so the ceiling is the backend's to choose. Ten seconds is the strictest rate rule a free-plan Site can even express, counted per address, which makes it the strongest limit any of these Sites could hold the backend to.
The same research established that a Cloudflare clearance cookie defaults to a thirty-minute lifetime, so any cadence at or above an hour re-solves the challenge on every Poll regardless. Moving the browser Sites from six hours to one therefore does not change whether a challenge is solved, only how many times a day. For comix specifically, combined with the in-tab read shape, the faster cadence is a tenfold traffic reduction rather than an increase.
This spec needs a decision record covering Poll Lanes, the per-Site gap, the automatic shortening and why the floor sits at one second. The floor in particular deserves the reasoning attached: it is an emergency stop rather than a rate, and a Site whose gap is genuinely clamped there is one the operator should be talking to rather than polling.
Duplicate of #100 — the body is byte-identical (only the issue number and state differ). #100 is closed and shipped on main in
3303a55("feat: one Poll Lane per Site, replacing the shared pace", PR #106): per-Site Lanes, registry Rest/Gap, effectiveGap = min(gap, rest/eligible) floored at 1s with a Site-naming warning, refusal backoff, browser wake gate and shared tab, background cover work, removed poll knobs, ADR-0010, DEPLOY.md env edit. Closing as duplicate; reopen if the spec was meant to differ.