Browser-backed Sites join the Cover pipeline #62

Closed
opened 2026-08-09 23:12:13 +07:00 by sulthan · 4 comments
Owner

#62 Browser-backed Sites join the Cover pipeline

Parent

Spec: #55. Originating bug: #47. Architecture and rejected alternatives:
docs/adr/0007-backend-hosts-cover-bytes.md. Domain vocabulary: CONTEXT.md.

Do not close #47 or #55 from this ticket.

What to build

kagane and novelfull Series get their Covers through the same pipeline as
everything else, so that the panel shows a kagane Cover — the second symptom
reported in #47 — and novelfull Covers stop depending on a client scrape
that no longer exists.

The two Sites need the browser for different reasons, and conflating them
wastes the scarcest resource in the stack:

• kagane needs the browser for its bytes. It serves cover images
with cross-origin-resource-policy: same-origin behind a JavaScript challenge,
which is why no on another origin can load one and why the bytes must
come through the sidecar that already clears that challenge.
• novelfull needs the browser only for its HTML. Its pages answer
cf-mitigated: challenge to a plain fetch, but its image paths do not — a
plain fetch of a novelfull cover answers 200 with access-control-allow-
origin: * (measured 2026-08-09). Its bytes therefore go over plain TLS
through the ordinary gated fetcher. Routing them through the browser would
be needless work on a resource that exists for seconds at a time.

When the browser sidecar is unconfigured, kagane Covers are simply
unavailable — consistent with how kagane chapter polling already degrades.
There is no fallback to a plain fetch, which would only ever retrieve a
challenge page.

Acceptance criteria

[x] kagane cover bytes are fetched through the browser sidecar and stored in
the content-addressed store
[x] novelfull cover URLs are extracted from the browser-fetched HTML, and
its bytes are fetched over plain TLS
[x] With no browser sidecar configured, kagane Covers are absent and nothing
falls back to a plain fetch
[x] With no browser sidecar configured, novelfull Covers still work if its
page body is available
[x] Manually verified on-device: a kagane Series shows its Cover in the
panel, not a broken-image glyph
[x] go test ./... is green, with live-network checks gated behind an
environment variable as the existing kagane image smoke test is

Blocked by

• #59 — it reuses that ticket's acquisition and wire path

# #62 Browser-backed Sites join the Cover pipeline ## Parent Spec: #55. Originating bug: #47. Architecture and rejected alternatives: docs/adr/0007-backend-hosts-cover-bytes.md. Domain vocabulary: CONTEXT.md. Do not close #47 or #55 from this ticket. ## What to build kagane and novelfull Series get their Covers through the same pipeline as everything else, so that the panel shows a kagane Cover — the second symptom reported in #47 — and novelfull Covers stop depending on a client scrape that no longer exists. The two Sites need the browser for different reasons, and conflating them wastes the scarcest resource in the stack: • **kagane** needs the browser for its **bytes**. It serves cover images with cross-origin-resource-policy: same-origin behind a JavaScript challenge, which is why no <img> on another origin can load one and why the bytes must come through the sidecar that already clears that challenge. • **novelfull** needs the browser only for its **HTML**. Its pages answer cf-mitigated: challenge to a plain fetch, but its image paths do not — a plain fetch of a novelfull cover answers 200 with access-control-allow- origin: * (measured 2026-08-09). Its bytes therefore go over plain TLS through the ordinary gated fetcher. Routing them through the browser would be needless work on a resource that exists for seconds at a time. When the browser sidecar is unconfigured, kagane Covers are simply unavailable — consistent with how kagane chapter polling already degrades. There is no fallback to a plain fetch, which would only ever retrieve a challenge page. ## Acceptance criteria [x] kagane cover bytes are fetched through the browser sidecar and stored in the content-addressed store [x] novelfull cover URLs are extracted from the browser-fetched HTML, and its bytes are fetched over plain TLS [x] With no browser sidecar configured, kagane Covers are absent and nothing falls back to a plain fetch [x] With no browser sidecar configured, novelfull Covers still work if its page body is available [x] Manually verified on-device: a kagane Series shows its Cover in the panel, not a broken-image glyph [x] go test ./... is green, with live-network checks gated behind an environment variable as the existing kagane image smoke test is ## Blocked by • #59 — it reuses that ticket's acquisition and wire path
sulthan added the ready-for-agent label 2026-08-09 23:12:13 +07:00
Author
Owner

Implemented in PR #72 (branch feat/62-browser-sites-join-cover-pipeline).

Done and verified by automated tests (all green, go test ./...):

  • kagane cover bytes through the browser sidecar into the content-addressed store (TestAcquireKaganeCoverThroughBrowser)
  • novelfull cover URL from browser-fetched HTML, bytes over plain TLS (TestAcquireNovelfullCoverOverPlainTLS)
  • no browser sidecar: kagane covers absent, no plain-fetch fallback (TestAcquireKaganeSkippedWithoutBrowser)
  • no browser sidecar: novelfull covers still work when the page body is available (TestAcquireNovelfullCoverWithoutBrowser)
  • go test ./... green; live checks gated behind SMOKE_BROWSER_WS_URL (new TestSmokeAcquireKaganeCover)

Not done here: the on-device manual verification (kagane Cover renders in the panel) — a separate agent is running it against a mocked scenario, no prod data. Will report back separately.

PR references Fixes #62, so this issue auto-closes when the PR merges.

Implemented in PR #72 (branch feat/62-browser-sites-join-cover-pipeline). Done and verified by automated tests (all green, go test ./...): - kagane cover bytes through the browser sidecar into the content-addressed store (TestAcquireKaganeCoverThroughBrowser) - novelfull cover URL from browser-fetched HTML, bytes over plain TLS (TestAcquireNovelfullCoverOverPlainTLS) - no browser sidecar: kagane covers absent, no plain-fetch fallback (TestAcquireKaganeSkippedWithoutBrowser) - no browser sidecar: novelfull covers still work when the page body is available (TestAcquireNovelfullCoverWithoutBrowser) - go test ./... green; live checks gated behind SMOKE_BROWSER_WS_URL (new TestSmokeAcquireKaganeCover) Not done here: the on-device manual verification (kagane Cover renders in the panel) — a separate agent is running it against a mocked scenario, no prod data. Will report back separately. PR references Fixes #62, so this issue auto-closes when the PR merges.
Author
Owner

Update: code review (standards + spec axes) found one hard issue — the Poller and Acquirer encoded opposite novelfull policies in duplicated fetcherFor methods. Fixed in a66491a: one shared fetcherFor for both paths (novelfull now also falls back to plain TLS on the poll, so pre-existing client-scraped rows get healed, not just Series created after this change), a byte-level no-fallback test for kagane (TestAcquireKaganeBytesNeverFallBackToPlainTLS), and the Acquirer is now wired even when the TLS client fails (kagane needs only the sidecar). Full go test ./... green. PR #72 updated.

Update: code review (standards + spec axes) found one hard issue — the Poller and Acquirer encoded opposite novelfull policies in duplicated fetcherFor methods. Fixed in a66491a: one shared fetcherFor for both paths (novelfull now also falls back to plain TLS on the poll, so pre-existing client-scraped rows get healed, not just Series created after this change), a byte-level no-fallback test for kagane (TestAcquireKaganeBytesNeverFallBackToPlainTLS), and the Acquirer is now wired even when the TLS client fails (kagane needs only the sidecar). Full go test ./... green. PR #72 updated.
Author
Owner

Manual verification of criterion 5 — mocked, on-device

Verdict: PASS (mocked scenario; no real kagane.to traffic, no prod stack, no prod data).

Code under test: branch feat/62-browser-sites-join-cover-pipeline, commit 40ce68b. Evidence saved under /tmp/manual-verify-62/ on the dev box.

What was mocked (and what was real)

Real Mocked
Backend built from this working tree (docker compose up -d --build), Postgres 17, cover volume .env — random TOKEN_KEY, OWNER_DISCORD_ID=1046923170000000000, PUBLIC_BASE_URL=http://localhost:8080, placeholder DISCORD_*
The userscript actually served by GET /u/{token}/manga-bookmark.user.js its API_BASE constant patched from the prod origin to http://localhost:8080 (one line; nothing else touched)
store.SetSeriesCover — real repo code, real DB rows, real content-addressed file the cover bytes: a 60×90 PNG generated locally by a throwaway Go program in /tmp, and a fake source URL https://kagane.to/api/v2/image/<uuid>/compressed (never fetched)
The panel: real renderItem cover block running in a real Chromium page on origin https://kagane.to the kagane.to page itself + the <script> — fulfilled by Playwright route interception; every other network egress aborted

BROWSER_WS_URL was empty for the whole run (docker inspect bookmark-api → BROWSER_WS_URL=), i.e. no sidecar browser.

1. Stack up

$ curl -s -o - -w "healthz HTTP %{http_code}\n" http://localhost:8080/healthz
ok
healthz HTTP 200

2. No browser configured → kagane cover is absent, nothing falls back to a plain fetch

$ curl -s -X PUT http://localhost:8080/bookmarks/kagane:3f1c9a52-... -H "Authorization: Bearer <token>" -d '{... "site":"kagane", "series_url":"https://kagane.to/series/3f1c9a52-..." ...}'
{"key":"kagane:3f1c9a52-7b64-4e0a-9d21-8c5ab0e17f42","site":"kagane",...,"cover":"",...}
PUT HTTP 200

cover is "" — verified. Backend log line at the same instant:

bookmark-api | acquire "kagane:3f1c9a52-...": no fetcher for site "kagane"

That is the designed no-browser behaviour: the cover is skipped, not fetched over plain TLS.

3. Mocked stored cover → wire URL, bytes, and a real <img>

Throwaway Go program (module bookmarkmanager/backend/tmpseed, replace onto backend/, built CGO_ENABLED=0, run in a scratch container as uid 65532 on the compose db network with the cover-data volume mounted) calling the repo's own store:

stored 234 bytes for kagane/3f1c9a52-7b64-4e0a-9d21-8c5ab0e17f42 from https://kagane.to/api/v2/image/9c2f18d0-.../compressed
$ curl -s http://localhost:8080/bookmarks -H "Authorization: Bearer <token>" | jq -r '.[] | [.title,.cover] | @tsv'
Mock Kagane Unloadable-Cover  http://localhost:8080/covers/b2f60790f2bba1fad03623f2bc3b988d70a9a01f64f2e59cad930774c75e9391
Mock Kagane No-Cover
Mock Kagane Series            http://localhost:8080/covers/9c6c7108d8134a51aad1c1bd00639e3541d4b6d1ce7bbf1ef191961765c6667e

The address is exactly sha256("https://kagane.to/api/v2/image/9c2f18d0-.../compressed") — checked with printf %s <url> | sha256sum → 9c6c7108d813.... Absolute, on PUBLIC_BASE_URL, as ADR-0007 requires.

$ curl -s -o served-cover.png -D - http://localhost:8080/covers/9c6c7108...
HTTP/1.1 200 OK
Cache-Control: public, max-age=604800, immutable
Content-Type: image/png
Content-Length: 234

$ file served-cover.png
served-cover.png: PNG image data, 60 x 90, 8-bit/color RGB, non-interlaced
$ sha256sum served-cover.png cover.png
4c8fdd56df91afc6e14d966166673a3d0aff9da72615b4ab15011cabaafc6356  served-cover.png
4c8fdd56df91afc6e14d966166673a3d0aff9da72615b4ab15011cabaafc6356  cover.png   # locally generated original

Byte-identical round trip.

4. The panel, end to end in a browser

Playwright: every request intercepted. https://kagane.to/series/3f1c9a52-... fulfilled with a local mock HTML page, https://kagane.to/__mock__/manga-bookmark.user.js fulfilled with the script the backend served (so the page origin is genuinely https://kagane.to and the real @match origin semantics hold); http://localhost:8080/** proxied through Playwright's request context with CORS headers (Chrome will not let an https:// page reach http://localhost directly — a mock artifact, prod is https→https). One cover URL was deliberately route.abort('failed')ed to exercise the onerror path.

The userscript booted with zero page errors, built its shadow-root panel, and GET /bookmarks returned 3 items. After clicking #fab:

[
 {"title":"Mock Kagane Unloadable-Cover","kind":"placeholder","rendered":{"w":52,"h":70}},
 {"title":"Mock Kagane No-Cover","kind":"placeholder","rendered":{"w":52,"h":70}},
 {"title":"Mock Kagane Series","kind":"img",
  "src":"http://localhost:8080/covers/9c6c7108...","naturalWidth":60,"naturalHeight":90,
  "complete":true,"rendered":{"w":52,"h":70}}
]
  • kagane Series with a stored Cover → <img class="cover">, naturalWidth=60, naturalHeight=90, complete=true, painted at 52×70. The screenshot shows the actual checkerboard image in the panel.
  • kagane Series with no Cover → div.cover.ph placeholder, never an <img>, so no glyph is possible.
  • kagane Series whose Cover request fails → the onerror handler (#47) had already replaced the <img> with div.cover.ph — the DOM contains no <img> at all afterwards.

Console for the whole run: 3 errors, all of them net::ERR_FAILED for the URL I deliberately aborted; the rest are Chrome mixed-content warnings caused by the http PUBLIC_BASE_URL in the mock.

Screenshot: /tmp/manual-verify-62/panel-kagane.png (panel open, one real cover + two placeholders).

Verified vs inferred

  • Verified by running: every HTTP status, the cover:"" on a browser-less kagane PUT, the absolute wire URL, the 200/image/png/234-byte cover serve, the byte-identical hash, the rendered naturalWidth/naturalHeight, the placeholder substitution in both cover-less and failed-cover cases, the screenshot.
  • Inferred, not run: that a real kagane cover arrives through the browser sidecar — that path needs BROWSER_WS_URL and live kagane, which was explicitly out of scope here. This run proves everything downstream of SetSeriesCover: once bytes are stored under the kagane series, the wire URL, the cover endpoint and the panel all behave. The sidecar-side fetch remains covered by the unit + SMOKE_* tests (not re-run here; the implementing agent reported go test ./... green).
  • Mock artifact, not a product finding: my seeding program called store.Open with a zero TokenHash, which re-seeded the owner row's token_sha256; I restored it with a psql UPDATE. That is my program's bug, not the backend's.

Cleanup

Throwaway Go program deleted (/tmp/seedcover), local static server stopped. The local compose stack and its mock .env are left up for inspection; no tracked repo file was modified.

## Manual verification of criterion 5 — mocked, on-device **Verdict: PASS** (mocked scenario; no real kagane.to traffic, no prod stack, no prod data). Code under test: branch `feat/62-browser-sites-join-cover-pipeline`, commit `40ce68b`. Evidence saved under `/tmp/manual-verify-62/` on the dev box. ### What was mocked (and what was real) | Real | Mocked | |---|---| | Backend built from this working tree (`docker compose up -d --build`), Postgres 17, cover volume | `.env` — random `TOKEN_KEY`, `OWNER_DISCORD_ID=1046923170000000000`, `PUBLIC_BASE_URL=http://localhost:8080`, placeholder `DISCORD_*` | | The userscript actually served by `GET /u/{token}/manga-bookmark.user.js` | its `API_BASE` constant patched from the prod origin to `http://localhost:8080` (one line; nothing else touched) | | `store.SetSeriesCover` — real repo code, real DB rows, real content-addressed file | the cover bytes: a 60×90 PNG generated locally by a throwaway Go program in `/tmp`, and a fake source URL `https://kagane.to/api/v2/image/<uuid>/compressed` (never fetched) | | The panel: real `renderItem` cover block running in a real Chromium page on origin `https://kagane.to` | the kagane.to page itself + the `<script>` — fulfilled by Playwright route interception; every other network egress aborted | `BROWSER_WS_URL` was empty for the whole run (`docker inspect bookmark-api` → `BROWSER_WS_URL=`), i.e. no sidecar browser. ### 1. Stack up ``` $ curl -s -o - -w "healthz HTTP %{http_code}\n" http://localhost:8080/healthz ok healthz HTTP 200 ``` ### 2. No browser configured → kagane cover is absent, nothing falls back to a plain fetch ``` $ curl -s -X PUT http://localhost:8080/bookmarks/kagane:3f1c9a52-... -H "Authorization: Bearer <token>" -d '{... "site":"kagane", "series_url":"https://kagane.to/series/3f1c9a52-..." ...}' {"key":"kagane:3f1c9a52-7b64-4e0a-9d21-8c5ab0e17f42","site":"kagane",...,"cover":"",...} PUT HTTP 200 ``` `cover` is `""` — verified. Backend log line at the same instant: ``` bookmark-api | acquire "kagane:3f1c9a52-...": no fetcher for site "kagane" ``` That is the designed no-browser behaviour: the cover is skipped, not fetched over plain TLS. ### 3. Mocked stored cover → wire URL, bytes, and a real `<img>` Throwaway Go program (module `bookmarkmanager/backend/tmpseed`, `replace` onto `backend/`, built `CGO_ENABLED=0`, run in a scratch container as uid 65532 on the compose `db` network with the `cover-data` volume mounted) calling the repo's own store: ``` stored 234 bytes for kagane/3f1c9a52-7b64-4e0a-9d21-8c5ab0e17f42 from https://kagane.to/api/v2/image/9c2f18d0-.../compressed ``` ``` $ curl -s http://localhost:8080/bookmarks -H "Authorization: Bearer <token>" | jq -r '.[] | [.title,.cover] | @tsv' Mock Kagane Unloadable-Cover http://localhost:8080/covers/b2f60790f2bba1fad03623f2bc3b988d70a9a01f64f2e59cad930774c75e9391 Mock Kagane No-Cover Mock Kagane Series http://localhost:8080/covers/9c6c7108d8134a51aad1c1bd00639e3541d4b6d1ce7bbf1ef191961765c6667e ``` The address is exactly `sha256("https://kagane.to/api/v2/image/9c2f18d0-.../compressed")` — checked with `printf %s <url> | sha256sum` → `9c6c7108d813...`. Absolute, on `PUBLIC_BASE_URL`, as ADR-0007 requires. ``` $ curl -s -o served-cover.png -D - http://localhost:8080/covers/9c6c7108... HTTP/1.1 200 OK Cache-Control: public, max-age=604800, immutable Content-Type: image/png Content-Length: 234 $ file served-cover.png served-cover.png: PNG image data, 60 x 90, 8-bit/color RGB, non-interlaced $ sha256sum served-cover.png cover.png 4c8fdd56df91afc6e14d966166673a3d0aff9da72615b4ab15011cabaafc6356 served-cover.png 4c8fdd56df91afc6e14d966166673a3d0aff9da72615b4ab15011cabaafc6356 cover.png # locally generated original ``` Byte-identical round trip. ### 4. The panel, end to end in a browser Playwright: every request intercepted. `https://kagane.to/series/3f1c9a52-...` fulfilled with a local mock HTML page, `https://kagane.to/__mock__/manga-bookmark.user.js` fulfilled with the script the backend served (so the page origin is genuinely `https://kagane.to` and the real `@match` origin semantics hold); `http://localhost:8080/**` proxied through Playwright's request context with CORS headers (Chrome will not let an `https://` page reach `http://localhost` directly — a mock artifact, prod is https→https). One cover URL was deliberately `route.abort('failed')`ed to exercise the `onerror` path. The userscript booted with **zero page errors**, built its shadow-root panel, and `GET /bookmarks` returned 3 items. After clicking `#fab`: ```json [ {"title":"Mock Kagane Unloadable-Cover","kind":"placeholder","rendered":{"w":52,"h":70}}, {"title":"Mock Kagane No-Cover","kind":"placeholder","rendered":{"w":52,"h":70}}, {"title":"Mock Kagane Series","kind":"img", "src":"http://localhost:8080/covers/9c6c7108...","naturalWidth":60,"naturalHeight":90, "complete":true,"rendered":{"w":52,"h":70}} ] ``` - **kagane Series with a stored Cover → `<img class="cover">`, `naturalWidth=60`, `naturalHeight=90`, `complete=true`, painted at 52×70.** The screenshot shows the actual checkerboard image in the panel. - kagane Series with no Cover → `div.cover.ph` placeholder, never an `<img>`, so no glyph is possible. - kagane Series whose Cover request fails → the `onerror` handler (#47) had already replaced the `<img>` with `div.cover.ph` — the DOM contains no `<img>` at all afterwards. Console for the whole run: 3 errors, all of them `net::ERR_FAILED` for the URL I deliberately aborted; the rest are Chrome mixed-content warnings caused by the http `PUBLIC_BASE_URL` in the mock. Screenshot: `/tmp/manual-verify-62/panel-kagane.png` (panel open, one real cover + two placeholders). ### Verified vs inferred - **Verified by running:** every HTTP status, the `cover:""` on a browser-less kagane PUT, the absolute wire URL, the 200/`image/png`/234-byte cover serve, the byte-identical hash, the rendered `naturalWidth`/`naturalHeight`, the placeholder substitution in both cover-less and failed-cover cases, the screenshot. - **Inferred, not run:** that a *real* kagane cover arrives through the browser sidecar — that path needs `BROWSER_WS_URL` and live kagane, which was explicitly out of scope here. This run proves everything downstream of `SetSeriesCover`: once bytes are stored under the kagane series, the wire URL, the cover endpoint and the panel all behave. The sidecar-side fetch remains covered by the unit + `SMOKE_*` tests (not re-run here; the implementing agent reported `go test ./...` green). - **Mock artifact, not a product finding:** my seeding program called `store.Open` with a zero `TokenHash`, which re-seeded the owner row's `token_sha256`; I restored it with a `psql` UPDATE. That is my program's bug, not the backend's. ### Cleanup Throwaway Go program deleted (`/tmp/seedcover`), local static server stopped. The local compose stack and its mock `.env` are left up for inspection; no tracked repo file was modified.
Author
Owner

Manual verification (criterion 5) — PASS, run by a separate verification agent against a fully mocked scenario (no prod stack, no prod data, no real kagane.to traffic):

  • no-browser behavior on the local stack: PUT kagane bookmark -> cover "" ; backend logged 'no fetcher for site kagane' (no plain-fetch fallback)
  • mocked stored cover: real store.SetSeriesCover with a locally generated 60x90 PNG -> wire cover URL = PUBLIC_BASE_URL/covers/; GET /covers/ returned 200 image/png, served bytes sha256-identical to the generated file
  • real served userscript (API_BASE only patched to localhost) booted on a route-mocked https://kagane.to series page: covered series rendered naturalWidth=60 complete=true; cover-less series rendered the placeholder div; an aborted cover request hit the onerror fallback -> placeholder. No broken-image glyph in any of the three cases.

Evidence: /tmp/manual-verify-62/ (screenshots, curl outputs, seed logs). Full report posted by the verification agent (prior comment).

Manual verification (criterion 5) — PASS, run by a separate verification agent against a fully mocked scenario (no prod stack, no prod data, no real kagane.to traffic): - no-browser behavior on the local stack: PUT kagane bookmark -> cover "" ; backend logged 'no fetcher for site kagane' (no plain-fetch fallback) - mocked stored cover: real store.SetSeriesCover with a locally generated 60x90 PNG -> wire cover URL = PUBLIC_BASE_URL/covers/<sha256 of sourceURL>; GET /covers/<sha> returned 200 image/png, served bytes sha256-identical to the generated file - real served userscript (API_BASE only patched to localhost) booted on a route-mocked https://kagane.to series page: covered series rendered <img class=cover> naturalWidth=60 complete=true; cover-less series rendered the placeholder div; an aborted cover request hit the onerror fallback -> placeholder. No broken-image glyph in any of the three cases. Evidence: /tmp/manual-verify-62/ (screenshots, curl outputs, seed logs). Full report posted by the verification agent (prior comment).
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sulthan/mangaBookmark#62