sulthan
d998f8725c
Restore two review findings against spec claims 'the Poll keeps its own Cover policy' and 'pinning is the only behavioural change': prefetchCover now also runs on ticks where the page fetch itself fails (the heal is independent of the page read, and its source may answer while the origin does not), and the no-chapter log surfaces the fetched body length again via a BodyLen fact on seriesRead. Browser dispatch iterates the sorted browser site list so its outcome cannot depend on map order.
5.0 KiB
ADR-0009: A Site answers fixed questions; an unusual Site owns its own fetch
Date: 2026-08-11 Status: accepted
Decision
Per-Site knowledge lives in one registry in backend/internal/latest/sites.go,
keyed by the stored site string. An entry answers a fixed set of questions: the
hostname a series_url must carry, how to find the Latest Chapter in a body,
how to find the Cover address in a body, and — for a Site behind a JavaScript
challenge — how to read its payload from a cleared tab, how to tell that the
payload arrived, and whether the plain-TLS fetcher may take over when no
browser is configured (Fallback; kagane never falls back, novelfull does,
each on measured evidence).
The question set does not grow to accommodate one Site. When a Site needs something the set cannot express, that Site gets an optional override and performs its own fetch, leaving the other entries untouched. The override is a per-Site escape hatch, not a stage every Site passes through, and it is added to the registry type when the first Site needs it rather than in advance.
Two things stay outside the registry. Cover bytes are routed by address shape
in fetchCoverBytes, never by Site name, so the Poll and the Acquisition cannot
drift apart. And the host pin inside a browser entry's read is kept even though
fetchableSeriesURL has already pinned the same host: a headless browser is a
strong SSRF primitive and series_url arrives in a client-supplied PUT body, so
the second check is deliberate and must not be deduplicated.
Why
Before the registry, the site string was compared in six places across three
files: the Latest Chapter switch (sites.go:102), the Cover switch
(sites.go:263), the browser-backed list and the fetcher choice
(poller.go:60, poller.go:132), the host pins (poller.go:314-325), and the
payload read (browser.go:96-119). Nothing tied them together, so adding a
seventh Site meant finding all six unaided, and a Site added to five of them
failed at the sixth in production rather than at compile time.
The Sites are not alike and the registry does not ask them to be. asura strips a rotating build hash from its slug before scoping a regex; comix reads a JSON blob embedded in server-rendered HTML; kagane's chapter list exists only in its JSON API, which must be called from inside the page so the request carries the clearance cookie; lightnovelworld must truncate the body at the comment thread first. What they have in common is not behaviour, it is the questions they answer. Arbitrary behaviour behind one entry is the point.
Making a browser Site contribute a read and a completion test, rather than
letting it drive the browser, was chosen because the tab lifecycle in
BrowserFetcher.run is load-bearing and shared. It holds one tab open across
re-reads, because a Cloudflare interstitial needs several seconds of live page
to solve itself and write clearance into the shared cookie jar; reading once and
closing the tab, which is what this did before 2026-08-08, never clears
anything. It also serialises the browser, binds the caller's deadline to the
tab, distinguishes a lost browser from a retryable read, and paces re-reads.
Spreading that across per-Site adapters would put one subtle, measured loop
behind six doors.
This costs the adapters little, because chromedp.Run takes an Action and
chromedp.Tasks is an Action. A Site that must click, wait on a selector, and
then evaluate expresses all of it as its read. Only a Site needing something
outside the per-tab loop — its own cadence, two tabs, a tab held between calls,
cookies set before navigation — falls outside, and that Site takes the override.
Considered options
Widen the shared interface whenever a Site needs something new. Rejected: one Site's requirement becomes a field on all seven entries, and the entries that ignore it still have to be read and understood by anyone adding the eighth.
Give every Site the whole fetch. Rejected: it makes the browser lifecycle
above a per-Site concern, and pulls chromedp into adapters for five Sites that
never open a browser.
A Go interface with a method set instead of a registry of records.
Rejected: most Sites differ in one or two answers, and three share a single
Cover implementation, so a method set produces near-empty types. A missing
answer is a nil value caught at dispatch, which is where an unknown Site is
already handled.
Consequences
Adding a Site is one registry entry. The existing dispatch functions —
latestChapterFrom, coverFrom, fetchableSeriesURL — become registry
lookups, so the table tests that drive them by site string are unchanged.
A future architecture review will see an override that only one Site uses and read it as an inconsistency to collapse. It is not. Collapsing it means either widening the question set for every Site or moving the shared tab lifecycle into the adapters, and both were rejected here on the evidence above.
An unknown site string resolves to the zero entry and fails the existing not-fetchable and no-fetcher paths, which log and skip. That is unchanged.