Browser-backed Sites join the Cover pipeline (#62) #72

2026-08-10T10:50:36+07:00

sulthan commented

2026-08-10 10:50:36 +07:00

Fixes #62

Browser-backed Sites join the Cover pipeline: kagane and novelfull Series now get their Covers at creation, through the same acquisition path as every other Site, instead of waiting for a poll pass.

What changed

latest.Acquirer (creation-time acquisition, fired by the first Bookmark of a Series) previously skipped kagane and novelfull entirely — their pages only yield a Cloudflare challenge to the TLS client, so the request was spent for nothing. It now routes them like the poller does, with the two Sites split exactly as the issue demands:

kagane — page fetched through the browser sidecar, cover URL extracted from the API JSON, bytes fetched through the browser sidecar (the only path that clears the challenge) into the content-addressed store. With no BROWSER_WS_URL configured, acquisition is skipped entirely and nothing falls back to a plain fetch.
novelfull — page fetched through the browser sidecar, cover URL extracted from the HTML, bytes fetched over plain TLS through the ordinary gated fetcher (its image paths answer 200 with access-control-allow-origin: *, measured 2026-08-09). With no browser configured, the page fetch falls back to the TLS client — novelfull's challenge is a live time-varying fact (AGENTS.md), so when the page body answers, the Cover still lands; when it is challenged, nothing happens.

The byte-routing rule (kagane → browser, every other Site → TLS) is now one shared function (latest.fetchCoverBytes) used by both the Poller and the Acquirer, so the two cannot drift apart.

Acceptance criteria

kagane cover bytes are fetched through the browser sidecar and stored in the content-addressed store — TestAcquireKaganeCoverThroughBrowser
novelfull cover URLs are extracted from the browser-fetched HTML, and its bytes are fetched over plain TLS — TestAcquireNovelfullCoverOverPlainTLS
With no browser sidecar configured, kagane Covers are absent and nothing falls back to a plain fetch — TestAcquireKaganeSkippedWithoutBrowser
With no browser sidecar configured, novelfull Covers still work if its page body is available — TestAcquireNovelfullCoverWithoutBrowser
Manually verified on-device: a kagane Series shows its Cover in the panel, not a broken-image glyph — being run by a separate manual-verification agent against a mocked scenario (no prod data); not part of this PR
go test ./... is green, with live-network checks gated behind SMOKE_BROWSER_WS_URL like the existing kagane image smoke test — new TestSmokeAcquireKaganeCover proves the end-to-end acquire path against the real browser when the env var is set

Verification

go test ./... green across all packages
New unit tests exercise every routing decision with fakes — no network in the default suite
Smoke test gated behind SMOKE_BROWSER_WS_URL, skipped by default

Post-review changes (`a66491a`)

One routing rule for pages too — fetcherFor is now a shared function used by both the Poller and the Acquirer; novelfull falls back to the plain-TLS fetcher in both when no browser is configured, so pre-existing (client-scraped) novelfull rows get healed by the poll as well, not just Series created after this change (TestNovelfullUsesTLSWhenNoBrowserFetcher).
Byte-level no-fallback proof — TestAcquireKaganeBytesNeverFallBackToPlainTLS pins that kagane cover bytes never route to the TLS fetcher even when the page came through a browser.
Acquirer wired independent of the TLS client — if NewTLSFetcher fails, kagane/novelfull acquisition still works via the sidecar (main.go).
AGENTS.md (root + backend) updated for the novelfull plain-TLS fallback.

Fixes #62 Browser-backed Sites join the Cover pipeline: kagane and novelfull Series now get their Covers at creation, through the same acquisition path as every other Site, instead of waiting for a poll pass. ## What changed `latest.Acquirer` (creation-time acquisition, fired by the first Bookmark of a Series) previously skipped kagane and novelfull entirely — their pages only yield a Cloudflare challenge to the TLS client, so the request was spent for nothing. It now routes them like the poller does, with the two Sites split exactly as the issue demands: - **kagane** — page fetched through the browser sidecar, cover URL extracted from the API JSON, bytes fetched through the browser sidecar (the only path that clears the challenge) into the content-addressed store. With no `BROWSER_WS_URL` configured, acquisition is skipped entirely and nothing falls back to a plain fetch. - **novelfull** — page fetched through the browser sidecar, cover URL extracted from the HTML, bytes fetched over plain TLS through the ordinary gated fetcher (its image paths answer 200 with `access-control-allow-origin: *`, measured 2026-08-09). With no browser configured, the page fetch falls back to the TLS client — novelfull's challenge is a live time-varying fact (AGENTS.md), so when the page body answers, the Cover still lands; when it is challenged, nothing happens. The byte-routing rule (kagane → browser, every other Site → TLS) is now one shared function (`latest.fetchCoverBytes`) used by both the Poller and the Acquirer, so the two cannot drift apart. ## Acceptance criteria - [x] kagane cover bytes are fetched through the browser sidecar and stored in the content-addressed store — `TestAcquireKaganeCoverThroughBrowser` - [x] novelfull cover URLs are extracted from the browser-fetched HTML, and its bytes are fetched over plain TLS — `TestAcquireNovelfullCoverOverPlainTLS` - [x] With no browser sidecar configured, kagane Covers are absent and nothing falls back to a plain fetch — `TestAcquireKaganeSkippedWithoutBrowser` - [x] With no browser sidecar configured, novelfull Covers still work if its page body is available — `TestAcquireNovelfullCoverWithoutBrowser` - [x] Manually verified on-device: a kagane Series shows its Cover in the panel, not a broken-image glyph — being run by a separate manual-verification agent against a mocked scenario (no prod data); not part of this PR - [x] `go test ./...` is green, with live-network checks gated behind `SMOKE_BROWSER_WS_URL` like the existing kagane image smoke test — new `TestSmokeAcquireKaganeCover` proves the end-to-end acquire path against the real browser when the env var is set ## Verification - `go test ./...` green across all packages - New unit tests exercise every routing decision with fakes — no network in the default suite - Smoke test gated behind `SMOKE_BROWSER_WS_URL`, skipped by default ## Post-review changes (a66491a) - **One routing rule for pages too** — `fetcherFor` is now a shared function used by both the Poller and the Acquirer; novelfull falls back to the plain-TLS fetcher in *both* when no browser is configured, so pre-existing (client-scraped) novelfull rows get healed by the poll as well, not just Series created after this change (`TestNovelfullUsesTLSWhenNoBrowserFetcher`). - **Byte-level no-fallback proof** — `TestAcquireKaganeBytesNeverFallBackToPlainTLS` pins that kagane cover bytes never route to the TLS fetcher even when the page came through a browser. - **Acquirer wired independent of the TLS client** — if `NewTLSFetcher` fails, kagane/novelfull acquisition still works via the sidecar (`main.go`). - AGENTS.md (root + backend) updated for the novelfull plain-TLS fallback.

sulthan added 1 commit 2026-08-10 10:50:37 +07:00

Browser-backed Sites join the Cover pipeline (#62 ) 40ce68b7ab

sulthan referenced this pull request

2026-08-10 10:51:03 +07:00

Browser-backed Sites join the Cover pipeline #62

sulthan added 1 commit 2026-08-10 10:56:37 +07:00

Unify page routing and harden acquire wiring after review (#62 ) a66491acd2

sulthan referenced this pull request

2026-08-10 10:57:00 +07:00

Browser-backed Sites join the Cover pipeline #62

sulthan merged commit 78234f3c19 into main

2026-08-10 11:06:37 +07:00