92 lines
4.9 KiB
Markdown
92 lines
4.9 KiB
Markdown
# ADR-0009: A Site answers fixed questions; an unusual Site owns its own fetch
|
|
|
|
Date: 2026-08-11
|
|
Status: accepted
|
|
|
|
## Decision
|
|
|
|
Per-Site knowledge lives in one registry in `backend/internal/latest/sites.go`,
|
|
keyed by the stored site string. An entry answers a fixed set of questions: the
|
|
hostname a `series_url` must carry, how to find the Latest Chapter in a body,
|
|
how to find the Cover address in a body, and — for a Site behind a JavaScript
|
|
challenge — how to read its payload from a cleared tab and how to tell that the
|
|
payload arrived.
|
|
|
|
The question set does not grow to accommodate one Site. When a Site needs
|
|
something the set cannot express, that Site gets an optional override and
|
|
performs its own fetch, leaving the other entries untouched. The override is a
|
|
per-Site escape hatch, not a stage every Site passes through, and it is added to
|
|
the registry type when the first Site needs it rather than in advance.
|
|
|
|
Two things stay outside the registry. Cover *bytes* are routed by address shape
|
|
in `fetchCoverBytes`, never by Site name, so the Poll and the Acquisition cannot
|
|
drift apart. And the host pin inside a browser entry's read is kept even though
|
|
`fetchableSeriesURL` has already pinned the same host: a headless browser is a
|
|
strong SSRF primitive and `series_url` arrives in a client-supplied PUT body, so
|
|
the second check is deliberate and must not be deduplicated.
|
|
|
|
## Why
|
|
|
|
Before the registry, the site string was compared in six places across three
|
|
files: the Latest Chapter switch (`sites.go:102`), the Cover switch
|
|
(`sites.go:263`), the browser-backed list and the fetcher choice
|
|
(`poller.go:60`, `poller.go:132`), the host pins (`poller.go:314-325`), and the
|
|
payload read (`browser.go:96-119`). Nothing tied them together, so adding a
|
|
seventh Site meant finding all six unaided, and a Site added to five of them
|
|
failed at the sixth in production rather than at compile time.
|
|
|
|
The Sites are not alike and the registry does not ask them to be. asura strips a
|
|
rotating build hash from its slug before scoping a regex; comix reads a JSON
|
|
blob embedded in server-rendered HTML; kagane's chapter list exists only in its
|
|
JSON API, which must be called from inside the page so the request carries the
|
|
clearance cookie; lightnovelworld must truncate the body at the comment thread
|
|
first. What they have in common is not behaviour, it is the questions they
|
|
answer. Arbitrary behaviour behind one entry is the point.
|
|
|
|
Making a browser Site contribute a read and a completion test, rather than
|
|
letting it drive the browser, was chosen because the tab lifecycle in
|
|
`BrowserFetcher.run` is load-bearing and shared. It holds one tab open across
|
|
re-reads, because a Cloudflare interstitial needs several seconds of live page
|
|
to solve itself and write clearance into the shared cookie jar; reading once and
|
|
closing the tab, which is what this did before 2026-08-08, never clears
|
|
anything. It also serialises the browser, binds the caller's deadline to the
|
|
tab, distinguishes a lost browser from a retryable read, and paces re-reads.
|
|
Spreading that across per-Site adapters would put one subtle, measured loop
|
|
behind six doors.
|
|
|
|
This costs the adapters little, because `chromedp.Run` takes an Action and
|
|
`chromedp.Tasks` is an Action. A Site that must click, wait on a selector, and
|
|
then evaluate expresses all of it as its read. Only a Site needing something
|
|
outside the per-tab loop — its own cadence, two tabs, a tab held between calls,
|
|
cookies set before navigation — falls outside, and that Site takes the override.
|
|
|
|
## Considered options
|
|
|
|
**Widen the shared interface whenever a Site needs something new.** Rejected:
|
|
one Site's requirement becomes a field on all seven entries, and the entries
|
|
that ignore it still have to be read and understood by anyone adding the eighth.
|
|
|
|
**Give every Site the whole fetch.** Rejected: it makes the browser lifecycle
|
|
above a per-Site concern, and pulls `chromedp` into adapters for five Sites that
|
|
never open a browser.
|
|
|
|
**A Go `interface` with a method set instead of a registry of records.**
|
|
Rejected: most Sites differ in one or two answers, and three share a single
|
|
Cover implementation, so a method set produces near-empty types. A missing
|
|
answer is a nil value caught at dispatch, which is where an unknown Site is
|
|
already handled.
|
|
|
|
## Consequences
|
|
|
|
Adding a Site is one registry entry. The existing dispatch functions —
|
|
`latestChapterFrom`, `coverFrom`, `fetchableSeriesURL` — become registry
|
|
lookups, so the table tests that drive them by site string are unchanged.
|
|
|
|
A future architecture review will see an override that only one Site uses and
|
|
read it as an inconsistency to collapse. It is not. Collapsing it means either
|
|
widening the question set for every Site or moving the shared tab lifecycle into
|
|
the adapters, and both were rejected here on the evidence above.
|
|
|
|
An unknown site string resolves to the zero entry and fails the existing
|
|
not-fetchable and no-fetcher paths, which log and skip. That is unchanged.
|