Spec: the backend acquires, stores and serves every Series Cover (fixes #47) #55

New Issue

2026-08-09T23:03:01+07:00

sulthan commented

2026-08-09 23:03:01 +07:00

Spec for #47. Design settled in the grilling session of 2026-08-09; the architectural decision and its rejected alternatives are recorded in docs/adr/0007-backend-hosts-cover-bytes.md, and the term Cover is now defined in CONTEXT.md. Deferred follow-up: #54.

Problem Statement

A Reader who bookmarks a Series expects to recognise it in a list by its artwork. Today that fails in two different ways, and both are visible right now.

The Cover is often never captured at all. A Reader bookmarks a Series from the chapter they are reading — that is the whole point of the userscript, it captures Progress where Progress happens. But three of the six Sites put no cover anywhere on a chapter page: comix and lightnovelworld expose neither an image nor a metadata tag, and the userscript's scrape returns an empty string. Because a Series records the facts that are true regardless of who is reading, and those facts are written once when the Series is first created, the empty Cover is not a temporary gap. It is permanent. No later Poll, no later bookmark by another Reader, and no amount of revisiting the series page will ever fill it. The Reader sees the monogram placeholder in the web UI and in the panel, forever, and there is nothing they can do about it. This is what the reporter observed on comix with Full-Time Awakening, and the identical defect exists unreported on the novel side with lightnovelworld.

When the Cover is captured, it may still be unrenderable. kagane serves its cover images with cross-origin-resource-policy: same-origin behind a Cloudflare JavaScript challenge, so no <img> on any other origin can load one. The web UI escapes this because its templates quietly rewrite kagane covers to a backend proxy on the deployment's own origin. The JSON API does no such rewrite: it hands out the raw kagane URL, the userscript panel puts it straight into an <img>, and the Reader gets the browser's broken-image glyph wedged in the corner of the cover box. Two clients, one stored value, two different outcomes — and the client that got it wrong is the one the Reader uses while actually reading.

Underneath both symptoms is a single unstated assumption: that a Cover is an address on a third-party Site, and that rendering it is each client's problem. It isn't. Whether an address is loadable depends on the Site's CORP header, its bot scoring, its CDN configuration, and which origin the client happens to be running on — none of which any client can see, and all of which can change without notice. Every client that renders a Cover has to independently know which Sites are special, and the moment one of them doesn't, a Reader gets a broken image.

Solution

Make the Cover the backend's responsibility end to end, so that no client ever handles a third-party address and every client renders every Cover the same way.

The backend acquires the Cover itself, from the same series-page fetch that already discovers the Latest Chapter. It fetches the image bytes, stores them, and serves them from its own origin at a stable address. What the API hands a client is always an address the client can load — or nothing at all, when there is genuinely no Cover yet. Clients lose their cover-scraping code entirely; they gain a fallback so that an image which fails to load shows the designed placeholder rather than a broken glyph.

For the Reader, three things change:

A Series bookmarked mid-chapter on any Site gets its Cover within seconds, on every device, not just the one that happened to be on the right page — and not only after the Poll queue eventually reaches it.
kagane Series show their Cover in the userscript panel, the same as they already do in the web UI.
A Cover that cannot be produced looks like a deliberate placeholder instead of a rendering failure.

User Stories

As a Reader, I want the Cover to appear for a Series I bookmarked from the middle of a chapter, so that I can recognise it in my list without having to visit its series page first.
As a Reader, I want the Cover to appear for a comix Series, so that my comix Bookmarks stop being an indistinguishable column of placeholders.
As a Reader, I want the Cover to appear for a lightnovelworld Series, so that the novel Library is as recognisable as the manga one.
As a Reader, I want kagane Covers to render inside the userscript panel, so that the panel matches what the web UI already shows me.
As a Reader, I want a Cover that fails to load to fall back to the placeholder, so that a failure looks intentional rather than broken.
As a Reader, I want the Cover to appear within seconds of bookmarking a new Series, so that the list is useful immediately rather than after a background cycle I cannot observe.
As a Reader, I want a Series I bookmarked on my phone to show its Cover when I open the web UI on my laptop, so that the library looks the same everywhere.
As a Reader, I want Covers to render the same way in the web UI and in the panel, so that I never have to wonder whether a missing image means a missing Bookmark.
As a Reader, I want a Cover to keep rendering after the Site changes its hotlink or CORP policy, so that my library does not silently degrade because of a decision made by a third party.
As a Reader, I want Covers to load quickly on repeat views, so that scrolling my library does not re-download the same images.
As a Reader, I want a Series with no Cover yet to show the placeholder rather than a broken image, so that "not yet" and "failed" are visually distinct.
As a Reader browsing on a slow connection, I want the Cover request to be lazily loaded and cacheable, so that opening the panel does not stall on images.
As a Reader, I want my Progress and Latest Chapter to be unaffected by any Cover problem, so that a cosmetic failure never costs me the feature I actually depend on.
As a Reader, I want the panel to keep working when the deployment has no browser sidecar configured, so that a partial deployment degrades to missing kagane Covers rather than to a broken panel.
As a Reader of the novel Library, I want everything above to apply to novels too, so that the two halves of the collection behave identically.
As a Reader, I want a Cover fetch that fails once to be retried later, so that a transient network failure does not permanently blank a Series.
As a Reader, I want the Site's own artwork rather than a generic image, so that Covers remain a reliable way to identify a Series at a glance.
As the Owner of the deployment, I want cover images served from my own origin, so that no manga or novel Site can see my Readers' IP addresses or referrers when they browse their library.
As the Owner, I want the backend never to be usable as a fetcher against my own internal network, so that a hostile page cannot turn a cover fetch into a probe of services only my server can reach.
As the Owner, I want a cover fetch to be bounded in size, so that a hostile or misconfigured host cannot exhaust my server's memory or disk.
As the Owner, I want cover storage to live on a volume I can size and back up separately, so that image bytes do not inflate my database dumps.
As the Owner, I want the disk cost of Covers to be predictable, so that I can provision the volume without guessing.
As the Owner, I want the cover route to be safe to expose publicly, so that userscript panels on third-party pages can load Covers without me putting a credential into a page's DOM.
As the Owner, I want a Cover that has already been fetched to survive a restart and a redeploy, so that a rebuild does not re-hammer the Sites.
As the Owner, I want Covers fetched once per Series rather than once per Reader, so that the fetch cost does not scale with how popular a Series is.
As the Owner, I want a failing Cover fetch to be visible in the logs with the Series it belongs to, so that I can tell a broken extractor from an unreachable Site.
As the Owner, I want the poll's chapter checking to continue unaffected when a Cover fetch fails, so that a cosmetic subsystem cannot degrade the product's core signal.
As the Owner, I want cover fetching to respect the same politeness constraints as the existing Poll, so that adding Covers does not raise my request rate against Sites that already bot-score my single IP.
As a developer, I want exactly one place that decides a Cover's renderable address, so that adding a client does not mean rediscovering which Sites are special.
As a developer, I want per-Site cover extraction to sit beside per-Site chapter extraction, so that adding a Site means editing one module rather than hunting for a second convention.
As a developer, I want cover extraction covered by fixtures trimmed from live pages, so that a Site changing its markup produces a failing test rather than a silent blank.
As a developer, I want the destination gate tested directly, so that a refactor cannot quietly remove the control that stops the server from fetching its own network.
As a developer, I want installed userscripts to keep working when the change ships, so that I am not forced to coordinate a client rollout with a server deploy.
As a developer reading the API later, I want the inert cover request field to explain itself, so that I do not "clean up" a field that exists for backwards compatibility.
As a developer, I want the userscripts to lose their cover-scraping code, so that there is no per-Site DOM scraping left to maintain in two scripts.
As an operator, I want the new configuration to have sane behaviour when unset, so that an incomplete deployment fails loudly at startup rather than serving broken image addresses.

Implementation Decisions

Cover ownership moves to the backend

A Cover is a fact about a Series, in the same category as its title and Latest Chapter, and ADR-0003 already establishes that only the backend's own fetch may write Series facts — the reason being that values scraped from third-party pages are attacker-controlled, and a Series row is shared by every Reader who bookmarks it. Cover was the one Series fact that never made that move; it stayed client-supplied-at-creation. Finishing the move is what fixes the class of bug rather than one instance of it.

Acquisition happens on Series creation, from the fetch that already exists

When a Bookmark creates a Series that nobody has bookmarked before, the backend fetches that Series' page and extracts both the Latest Chapter and the Cover from the same response. This runs at creation rather than waiting for the Poll queue.

Two reasons this shape and not another. First, a newly created Series is at the back of a queue ordered by how many Readers hold it, so a Series with one Reader waits behind every popular one; the Reader who just bookmarked it would see neither a Cover nor a Latest Chapter until the queue reached them. Second, it is why no client-supplied hint is worth accepting: the backend must fetch that page anyway for the chapter signal, so a hint saves no request, and it would be absent precisely in the case that matters, because comix and lightnovelworld expose no cover on chapter pages.

The creation-time fetch is asynchronous with respect to the write. A Reader's bookmark action must not block on a third-party Site's latency, and it must not fail because that Site is down.

Per-Site cover extraction joins per-Site chapter extraction

Extraction rules live beside the existing latest-chapter extraction, in the same module, driven by the same fetched body. Verified against live pages on 2026-08-09:

asura — og:image in the server-rendered HTML.
demonic — og:image. The value contains a raw unencoded space and must be percent-encoded before it is fetched.
comix — no og:image exists. The cover is in the same server-rendered state blob that latestChapterUrl is read from, under poster, which carries both a medium (roughly 36 KB) and a large (full size) absolute URL.
lightnovelworld — og:image.
novelfull — meta[name=image]. Its HTML answers cf-mitigated: challenge to a plain fetch, so extraction rides the browser-backed fetch the Poll already uses for this Site.
kagane — the cover URL is in the series API JSON the browser-backed fetch already retrieves.

Where a Site offers several renditions, prefer the smaller one that is deterministically named: for comix that is poster.medium. Otherwise take the metadata tag as published. The largest thing a Cover is ever rendered into is a card, so storing a full-resolution image costs disk and phone bandwidth for pixels nobody sees. Do not attempt to synthesise a thumbnail URL by string surgery on a Site that does not publish one — asura publishes a -400 variant that the page uses but the metadata tag does not, and deriving it by editing a URL is exactly the kind of guess that breaks silently.

Note for whoever writes the asura branch: its .webp cover URL answers Content-Type: image/jpeg. Trust the response header, never the extension.

Byte fetching, and which Sites need the browser

Five of the six Sites serve their cover bytes over plain TLS with no challenge and no CORP restriction. Only kagane requires the CDP browser, for the same reason it already does elsewhere. This is worth stating explicitly because it is counter-intuitive for novelfull: its HTML is challenge-gated but its image paths are not — a plain fetch of a novelfull cover answers 200 with access-control-allow-origin: * (measured 2026-08-09). Routing novelfull's bytes through the browser would be needless work on a scarce resource.

When the browser sidecar is unconfigured, kagane Covers are simply unavailable, consistent with how kagane chapter polling already degrades. No fallback to a plain fetch: that would only ever retrieve a challenge page.

The outbound fetch is gated by destination class, not by a host allowlist

Every cover URL originates in a third-party page, so fetching one points our server at an address an attacker may control. The control is:

https only.
Resolve the host first and refuse the connection when the resolved address is loopback, private, link-local, unique-local, or CGNAT. Refusing by literal IP is not sufficient — a hostile page need only publish a DNS name that resolves to 127.0.0.1. This resolve-then-classify step is the load-bearing part of the whole gate.
Re-apply the same check on every redirect hop, since the first hop's address says nothing about the second's.
Cap the response body, and accept only image content types.

This deliberately differs from the host allowlist that gates series_url, and the ADR records why: cover hosts are CDNs that move independently of their Site, as demonicscans demonstrates by serving its covers from readermc.org. An allowlist would stop producing Covers the day a Site switched CDN, and the failure would present as this very bug. The trade accepted in exchange is that the server may fetch from any public host; that is a bandwidth and reputation concern, not a breach, and the URL still only ever comes from a Series page we had already decided to poll.

Reuse the existing body cap and the existing image content-type check rather than introducing parallel ones.

Storage is a filesystem volume; the database holds the path

Cover bytes are stored on a filesystem volume, content-addressed by the SHA-256 of the source URL and sharded two levels deep so that no single directory accumulates thousands of entries. The database row records the content address, the path, and the content type.

Content addressing is chosen over a Series-keyed path because it makes the stored object immutable: a new image is a new address, which means the long-lived cache headers are always correct and no invalidation logic is needed anywhere — including for the admin refetch deferred to #54.

The existing kagane cover table stores bytes as a column; that column goes away. Its existing rows are dropped rather than migrated, because that path already re-fetches on demand when a cover is missing, so a migration would carefully preserve a cache that rebuilds itself for free.

The accepted cost, recorded in the ADR because it cuts against ADR-0001: durability is now two things to back up rather than one.

Serving: one public route, immutable caching

Covers are served from a single route on the deployment's own origin, with no session and no credential.

The reason is concrete rather than philosophical. An <img> cannot send an Authorization header, and it cannot be given a token in its URL either: the userscript panel's shadow root is open, so the host page's own JavaScript can read any src attribute we set, and a credential in an image URL is a credential handed to a third-party site. Since the route serves public artwork from public Sites and its address reveals nothing about which Reader holds what, public is the correct answer rather than a concession. This does relax the current kagane proxy's session gate, which the ADR records as a knowing decision.

Responses keep a long-lived immutable cache directive, which content addressing makes safe.

Wire contract

GET /bookmarks returns cover as an absolute URL on the deployment's public origin, built from a newly required configuration value. Absolute rather than relative because the userscript renders on third-party origins, where a relative path resolves against the Site rather than against us; and per ADR-0004 the server owns the wire shape rather than exporting a join to every client.

cover is the empty string until the bytes exist. The alternative — always returning an address that 404s until the fetch lands — was rejected because it makes "no Cover yet" and "Cover is broken" indistinguishable in both the UI and the logs, and it turns the client-side fallback into a routine occurrence instead of a genuine last resort.

PUT /bookmarks/{key} continues to accept a cover field and ignores it. Rejecting it would break every installed userscript the moment the server deployed, and ADR-0004's compatibility argument depends on old scripts continuing to work. This extends ADR-0003's "ignored after creation" to "ignored always". The field is permanently inert rather than pending removal, and the decode site must say so — a bare TODO would be a promise nobody intends to keep, which the repo's comment rules forbid.

The flat shape of the wire is unchanged in every other respect.

The Poll fills blanks, never overwrites

The Poll's cover prefetch, currently guarded to a single Site, applies to every Site and both Libraries. Nothing about Covers is conditioned on a Bookmark's kind; the shared card template already renders both Libraries identically, so this is a guard to remove rather than a branch to add.

A Poll fills a Cover only when the stored one is blank. It never replaces a Cover that already exists — a Cover changing under the Reader for no visible reason is noise, and refetching on every Poll would add a request per Series per cycle against Sites that already bot-score the deployment's single IP. This is also what repairs the Series rows that are blank today: the Poll walks every Series anyway, so no migration or backfill script is needed.

A failed cover fetch never fails the chapter poll, never blocks it, and never consumes anything the chapter poll depends on. It is retried the next time that Series is polled, with no separate retry queue.

Clients

Both userscripts lose their cover-scraping code: the DOM scan, the metadata reads, and the cover field in the request body. A scraped address has no rendering path left, and keeping it would reintroduce third-party URLs into exactly the place this bug came from.

Both userscripts gain an error handler on the cover image that swaps in the existing placeholder element on a failed load. This is the only part of the change that is worth doing even in isolation: it is the difference between a Reader seeing a designed empty state and a Reader seeing a broken glyph.

The web UI's template-level rewrite disappears, because the address it was rewriting no longer reaches a template.

Configuration

Two new values: the deployment's public base URL, used to build absolute cover addresses, and the cover storage directory. Both are required. A public base URL that is unset cannot be guessed and would produce addresses no client can load, so startup must fail loudly rather than serve them; this matches how the existing configuration treats values it cannot default safely. The storage directory needs a Compose volume alongside the database's.

Testing Decisions

A good test here pins an externally observable promise: what a client receives, what reaches the network, what survives a restart. It does not assert on internal call sequences, private helpers, or the order in which the implementation happens to do things. Every test must be able to fail for a real reason — if a plausible bug in the code under test would still leave it green, it is not earning its place. Tests stay deterministic and touch no live network; the suite already holds that line and this change must not be the exception.

Prefer the seams that already exist. There are three, and only one of them is new.

Seam 1 — the composed HTTP router (package main). This is the primary seam and most cases belong here. These tests build the whole router over a throwaway Postgres and drive it with recorded requests, with fetchers injected through the config struct exactly as the existing kagane cover tests inject a fake browser. Prior art: the current cover proxy tests, which already cover persistence, restart survival, a missing fetcher, and rejected input; and the existing API and web handler tests. Cases to cover here:

Creating a Bookmark for a previously unknown Series causes exactly one series-page fetch, and the Cover and the Latest Chapter both land from it.
Creating a Bookmark for a Series that already exists causes no fetch and does not disturb the stored Cover.
A request body containing a cover value is accepted and the value is ignored, both at creation and afterwards.
GET /bookmarks returns an absolute cover address once bytes exist, and the empty string before they do.
The cover route serves stored bytes with no session and no credential.
The cover route answers not-found for an address that was never stored, and never invents one.
Stored bytes survive a restart of the process against the same volume and database, without a second fetch.
A malformed or traversal-shaped address gets no bytes.
A fetch that returns a non-image content type stores nothing and serves nothing.
A Cover fetch that fails leaves the Bookmark, its Progress and its Latest Chapter intact and the cover field empty.
Both Libraries behave identically: the same assertions hold for a novel Bookmark.

Seam 2 — per-Site extraction (package latest). Pure table tests over HTML and JSON fixtures trimmed from live pages, in the same file and the same style as the existing latest-chapter extraction tests, with each fixture carrying a comment naming the URL and the date it was taken. This is where each Site's quirks get pinned: comix's cover living in the state blob rather than a metadata tag, demonic's raw unencoded space, asura's mismatch between file extension and content type, novelfull's and kagane's browser-fetched bodies, and a Site page that carries no cover at all yielding empty rather than a wrong value. Add the Cloudflare challenge body as a case too — the existing fixtures already include one — and assert it produces no Cover instead of garbage.

Seam 3 — the destination gate (new, and deliberately small). The classification step needs a seam because the interesting cases cannot be produced through the router: proving that a hostname resolving into private space is refused requires control of resolution. Inject the name lookup as a function on the fetcher, defaulting to the real resolver, in the same spirit as the Fetcher interface the Poller already injects for the same reason. Cover: a plain-HTTP URL refused; a literal loopback, private, link-local and CGNAT address refused; a public name allowed; a hostname resolving into private space refused; a redirect from a public host into private space refused mid-chain; an oversized body truncated or rejected rather than buffered. Assert on the observable outcome — no connection made, no bytes stored — rather than on which predicate returned what.

Do not add a fourth seam for the Poll's blank-fill rule. The existing poller tests already run a real store with a fake fetcher and assert prefetch behaviour, including that a second poll does not refetch; extend those rather than starting a parallel harness.

Userscripts. The existing Node harness covers pure logic only — adapters, parsers, helpers — under a hand-written browser stub, and must not grow a DOM harness. The cover work here is a deletion on that side, so the corresponding cover-scrape cases are deleted with it, and the exported symbol lists shrink accordingly. The error-handler fallback is DOM behaviour and is therefore not testable in that harness by design; it is verified on-device, along with the panel actually rendering a kagane Cover, and the report should say so plainly rather than inventing coverage. Run the parse check and the logic tests for both scripts before calling the client side done.

Verification beyond the suite. The change is not done on the strength of green tests alone. Exercise it against a running backend: bookmark a comix Series from a chapter page and confirm the Cover appears in both the web UI and the panel; do the same for a lightnovelworld Series; confirm a kagane Cover renders in the panel. A live smoke path already exists for kagane images and is gated behind an environment variable so it never runs in CI — extend that pattern rather than putting live fetches in the default suite.

Out of Scope

Changing or overriding a Cover. Refetching a Series' Cover after a Site re-arts it is #54, deliberately deferred until there is an admin surface to hang it off. Until then a stored Cover is not updateable by anyone. Reader-supplied URLs and uploads are rejected outright, not deferred: they would put attacker-controlled input into a row every Reader sees, which ADR-0003 closed on purpose.
Per-Reader Cover overrides. Out of scope permanently, consistent with the existing rejection of per-Reader title overrides. A Cover is a fact about the Series.
Image processing. No resizing, re-encoding, format conversion, or thumbnail generation. Store what the Site serves, choosing the smaller published rendition where one exists.
Eviction, quotas or garbage collection of stored covers. The arithmetic does not justify machinery: a few thousand Series at tens of kilobytes each is a volume measured in tens of megabytes. Revisit if measurement ever contradicts that.
Covers for Series nobody has bookmarked. Acquisition is driven by Bookmarks, as polling already is.
Changing the flat wire format in any way beyond the meaning of the cover value.
Redesigning the placeholder. The monogram and hatch placeholders already exist and are unchanged; this spec only decides when they are shown.
The Cinder design system. No new visual states, no new tokens, no colour changes. A Cover is either an image or the existing placeholder.
New Sites. Six Sites, unchanged.

Further Notes

The two symptoms in #47 look like one bug and are worth keeping separate while implementing, because they fail independently: acquisition (nothing was ever stored) and rendering (something was stored that the client cannot load). A change that fixes only acquisition still leaves kagane broken in the panel; a change that fixes only rendering still leaves comix and lightnovelworld blank forever. Both halves are required for either symptom to be gone.

Sequencing that keeps the deployment coherent at every step: backend acquisition and storage first, since it is testable behind the existing harness and changes nothing a client sees; then the wire change, which is when Covers begin appearing in the web UI; then the two userscripts last, so that no installed script is ever given an address the server cannot yet serve. The reverse order would ship a client expecting a route that does not exist.

Two live-fact caveats for whoever picks this up. Cloudflare's bot scoring is time-varying, and the repo's guidance is explicit that "does a plain fetch work right now" is a fact to re-check rather than a property of a machine — the measurements in this spec are dated 2026-08-09 and should be re-verified rather than trusted if something behaves unexpectedly. And the extraction anchors are third-party markup: they are pinned by fixtures precisely so that a Site changing its page produces a failing test rather than a library quietly filling with placeholders.

Finally, the reason to prefer uniformity over per-Site exceptions is empirical rather than aesthetic. The previous fix for kagane was the tidy minimal one — proxy only the Site that needs proxying, in the layer that needed it — and it produced this issue, because the second client did not know about the exception. The property being bought is that there is no exception left for a future client to be unaware of.

Spec for #47. Design settled in the grilling session of 2026-08-09; the architectural decision and its rejected alternatives are recorded in `docs/adr/0007-backend-hosts-cover-bytes.md`, and the term **Cover** is now defined in `CONTEXT.md`. Deferred follow-up: #54. ## Problem Statement A Reader who bookmarks a Series expects to recognise it in a list by its artwork. Today that fails in two different ways, and both are visible right now. **The Cover is often never captured at all.** A Reader bookmarks a Series from the chapter they are reading — that is the whole point of the userscript, it captures Progress where Progress happens. But three of the six Sites put no cover anywhere on a chapter page: comix and lightnovelworld expose neither an image nor a metadata tag, and the userscript's scrape returns an empty string. Because a Series records the facts that are true regardless of who is reading, and those facts are written once when the Series is first created, the empty Cover is not a temporary gap. It is permanent. No later Poll, no later bookmark by another Reader, and no amount of revisiting the series page will ever fill it. The Reader sees the monogram placeholder in the web UI and in the panel, forever, and there is nothing they can do about it. This is what the reporter observed on comix with *Full-Time Awakening*, and the identical defect exists unreported on the novel side with lightnovelworld. **When the Cover is captured, it may still be unrenderable.** kagane serves its cover images with `cross-origin-resource-policy: same-origin` behind a Cloudflare JavaScript challenge, so no `<img>` on any other origin can load one. The web UI escapes this because its templates quietly rewrite kagane covers to a backend proxy on the deployment's own origin. The JSON API does no such rewrite: it hands out the raw kagane URL, the userscript panel puts it straight into an `<img>`, and the Reader gets the browser's broken-image glyph wedged in the corner of the cover box. Two clients, one stored value, two different outcomes — and the client that got it wrong is the one the Reader uses while actually reading. Underneath both symptoms is a single unstated assumption: that a Cover is *an address on a third-party Site*, and that rendering it is each client's problem. It isn't. Whether an address is loadable depends on the Site's CORP header, its bot scoring, its CDN configuration, and which origin the client happens to be running on — none of which any client can see, and all of which can change without notice. Every client that renders a Cover has to independently know which Sites are special, and the moment one of them doesn't, a Reader gets a broken image. ## Solution Make the Cover the backend's responsibility end to end, so that no client ever handles a third-party address and every client renders every Cover the same way. The backend acquires the Cover itself, from the same series-page fetch that already discovers the Latest Chapter. It fetches the image bytes, stores them, and serves them from its own origin at a stable address. What the API hands a client is always an address the client can load — or nothing at all, when there is genuinely no Cover yet. Clients lose their cover-scraping code entirely; they gain a fallback so that an image which fails to load shows the designed placeholder rather than a broken glyph. For the Reader, three things change: - A Series bookmarked mid-chapter on any Site gets its Cover within seconds, on every device, not just the one that happened to be on the right page — and not only after the Poll queue eventually reaches it. - kagane Series show their Cover in the userscript panel, the same as they already do in the web UI. - A Cover that cannot be produced looks like a deliberate placeholder instead of a rendering failure. ## User Stories 1. As a Reader, I want the Cover to appear for a Series I bookmarked from the middle of a chapter, so that I can recognise it in my list without having to visit its series page first. 2. As a Reader, I want the Cover to appear for a comix Series, so that my comix Bookmarks stop being an indistinguishable column of placeholders. 3. As a Reader, I want the Cover to appear for a lightnovelworld Series, so that the novel Library is as recognisable as the manga one. 4. As a Reader, I want kagane Covers to render inside the userscript panel, so that the panel matches what the web UI already shows me. 5. As a Reader, I want a Cover that fails to load to fall back to the placeholder, so that a failure looks intentional rather than broken. 6. As a Reader, I want the Cover to appear within seconds of bookmarking a new Series, so that the list is useful immediately rather than after a background cycle I cannot observe. 7. As a Reader, I want a Series I bookmarked on my phone to show its Cover when I open the web UI on my laptop, so that the library looks the same everywhere. 8. As a Reader, I want Covers to render the same way in the web UI and in the panel, so that I never have to wonder whether a missing image means a missing Bookmark. 9. As a Reader, I want a Cover to keep rendering after the Site changes its hotlink or CORP policy, so that my library does not silently degrade because of a decision made by a third party. 10. As a Reader, I want Covers to load quickly on repeat views, so that scrolling my library does not re-download the same images. 11. As a Reader, I want a Series with no Cover yet to show the placeholder rather than a broken image, so that "not yet" and "failed" are visually distinct. 12. As a Reader browsing on a slow connection, I want the Cover request to be lazily loaded and cacheable, so that opening the panel does not stall on images. 13. As a Reader, I want my Progress and Latest Chapter to be unaffected by any Cover problem, so that a cosmetic failure never costs me the feature I actually depend on. 14. As a Reader, I want the panel to keep working when the deployment has no browser sidecar configured, so that a partial deployment degrades to missing kagane Covers rather than to a broken panel. 15. As a Reader of the novel Library, I want everything above to apply to novels too, so that the two halves of the collection behave identically. 16. As a Reader, I want a Cover fetch that fails once to be retried later, so that a transient network failure does not permanently blank a Series. 17. As a Reader, I want the Site's own artwork rather than a generic image, so that Covers remain a reliable way to identify a Series at a glance. 18. As the Owner of the deployment, I want cover images served from my own origin, so that no manga or novel Site can see my Readers' IP addresses or referrers when they browse their library. 19. As the Owner, I want the backend never to be usable as a fetcher against my own internal network, so that a hostile page cannot turn a cover fetch into a probe of services only my server can reach. 20. As the Owner, I want a cover fetch to be bounded in size, so that a hostile or misconfigured host cannot exhaust my server's memory or disk. 21. As the Owner, I want cover storage to live on a volume I can size and back up separately, so that image bytes do not inflate my database dumps. 22. As the Owner, I want the disk cost of Covers to be predictable, so that I can provision the volume without guessing. 23. As the Owner, I want the cover route to be safe to expose publicly, so that userscript panels on third-party pages can load Covers without me putting a credential into a page's DOM. 24. As the Owner, I want a Cover that has already been fetched to survive a restart and a redeploy, so that a rebuild does not re-hammer the Sites. 25. As the Owner, I want Covers fetched once per Series rather than once per Reader, so that the fetch cost does not scale with how popular a Series is. 26. As the Owner, I want a failing Cover fetch to be visible in the logs with the Series it belongs to, so that I can tell a broken extractor from an unreachable Site. 27. As the Owner, I want the poll's chapter checking to continue unaffected when a Cover fetch fails, so that a cosmetic subsystem cannot degrade the product's core signal. 28. As the Owner, I want cover fetching to respect the same politeness constraints as the existing Poll, so that adding Covers does not raise my request rate against Sites that already bot-score my single IP. 29. As a developer, I want exactly one place that decides a Cover's renderable address, so that adding a client does not mean rediscovering which Sites are special. 30. As a developer, I want per-Site cover extraction to sit beside per-Site chapter extraction, so that adding a Site means editing one module rather than hunting for a second convention. 31. As a developer, I want cover extraction covered by fixtures trimmed from live pages, so that a Site changing its markup produces a failing test rather than a silent blank. 32. As a developer, I want the destination gate tested directly, so that a refactor cannot quietly remove the control that stops the server from fetching its own network. 33. As a developer, I want installed userscripts to keep working when the change ships, so that I am not forced to coordinate a client rollout with a server deploy. 34. As a developer reading the API later, I want the inert `cover` request field to explain itself, so that I do not "clean up" a field that exists for backwards compatibility. 35. As a developer, I want the userscripts to lose their cover-scraping code, so that there is no per-Site DOM scraping left to maintain in two scripts. 36. As an operator, I want the new configuration to have sane behaviour when unset, so that an incomplete deployment fails loudly at startup rather than serving broken image addresses. ## Implementation Decisions ### Cover ownership moves to the backend A Cover is a fact about a Series, in the same category as its title and Latest Chapter, and ADR-0003 already establishes that only the backend's own fetch may write Series facts — the reason being that values scraped from third-party pages are attacker-controlled, and a Series row is shared by every Reader who bookmarks it. Cover was the one Series fact that never made that move; it stayed client-supplied-at-creation. Finishing the move is what fixes the class of bug rather than one instance of it. ### Acquisition happens on Series creation, from the fetch that already exists When a Bookmark creates a Series that nobody has bookmarked before, the backend fetches that Series' page and extracts both the Latest Chapter and the Cover from the same response. This runs at creation rather than waiting for the Poll queue. Two reasons this shape and not another. First, a newly created Series is at the back of a queue ordered by how many Readers hold it, so a Series with one Reader waits behind every popular one; the Reader who just bookmarked it would see neither a Cover nor a Latest Chapter until the queue reached them. Second, it is why no client-supplied hint is worth accepting: the backend must fetch that page anyway for the chapter signal, so a hint saves no request, and it would be absent precisely in the case that matters, because comix and lightnovelworld expose no cover on chapter pages. The creation-time fetch is asynchronous with respect to the write. A Reader's bookmark action must not block on a third-party Site's latency, and it must not fail because that Site is down. ### Per-Site cover extraction joins per-Site chapter extraction Extraction rules live beside the existing latest-chapter extraction, in the same module, driven by the same fetched body. Verified against live pages on 2026-08-09: - **asura** — `og:image` in the server-rendered HTML. - **demonic** — `og:image`. The value contains a raw unencoded space and must be percent-encoded before it is fetched. - **comix** — no `og:image` exists. The cover is in the same server-rendered state blob that `latestChapterUrl` is read from, under `poster`, which carries both a `medium` (roughly 36 KB) and a `large` (full size) absolute URL. - **lightnovelworld** — `og:image`. - **novelfull** — `meta[name=image]`. Its HTML answers `cf-mitigated: challenge` to a plain fetch, so extraction rides the browser-backed fetch the Poll already uses for this Site. - **kagane** — the cover URL is in the series API JSON the browser-backed fetch already retrieves. Where a Site offers several renditions, prefer the smaller one that is deterministically named: for comix that is `poster.medium`. Otherwise take the metadata tag as published. The largest thing a Cover is ever rendered into is a card, so storing a full-resolution image costs disk and phone bandwidth for pixels nobody sees. Do not attempt to synthesise a thumbnail URL by string surgery on a Site that does not publish one — asura publishes a `-400` variant that the page uses but the metadata tag does not, and deriving it by editing a URL is exactly the kind of guess that breaks silently. Note for whoever writes the asura branch: its `.webp` cover URL answers `Content-Type: image/jpeg`. Trust the response header, never the extension. ### Byte fetching, and which Sites need the browser Five of the six Sites serve their cover bytes over plain TLS with no challenge and no CORP restriction. Only kagane requires the CDP browser, for the same reason it already does elsewhere. This is worth stating explicitly because it is counter-intuitive for novelfull: its HTML is challenge-gated but its image paths are not — a plain fetch of a novelfull cover answers 200 with `access-control-allow-origin: *` (measured 2026-08-09). Routing novelfull's *bytes* through the browser would be needless work on a scarce resource. When the browser sidecar is unconfigured, kagane Covers are simply unavailable, consistent with how kagane chapter polling already degrades. No fallback to a plain fetch: that would only ever retrieve a challenge page. ### The outbound fetch is gated by destination class, not by a host allowlist Every cover URL originates in a third-party page, so fetching one points our server at an address an attacker may control. The control is: - `https` only. - Resolve the host first and refuse the connection when the resolved address is loopback, private, link-local, unique-local, or CGNAT. Refusing by literal IP is not sufficient — a hostile page need only publish a DNS name that resolves to `127.0.0.1`. This resolve-then-classify step is the load-bearing part of the whole gate. - Re-apply the same check on every redirect hop, since the first hop's address says nothing about the second's. - Cap the response body, and accept only image content types. This deliberately differs from the host allowlist that gates `series_url`, and the ADR records why: cover hosts are CDNs that move independently of their Site, as demonicscans demonstrates by serving its covers from `readermc.org`. An allowlist would stop producing Covers the day a Site switched CDN, and the failure would present as this very bug. The trade accepted in exchange is that the server may fetch from any *public* host; that is a bandwidth and reputation concern, not a breach, and the URL still only ever comes from a Series page we had already decided to poll. Reuse the existing body cap and the existing image content-type check rather than introducing parallel ones. ### Storage is a filesystem volume; the database holds the path Cover bytes are stored on a filesystem volume, content-addressed by the SHA-256 of the source URL and sharded two levels deep so that no single directory accumulates thousands of entries. The database row records the content address, the path, and the content type. Content addressing is chosen over a Series-keyed path because it makes the stored object immutable: a new image is a new address, which means the long-lived cache headers are always correct and no invalidation logic is needed anywhere — including for the admin refetch deferred to #54. The existing kagane cover table stores bytes as a column; that column goes away. Its existing rows are dropped rather than migrated, because that path already re-fetches on demand when a cover is missing, so a migration would carefully preserve a cache that rebuilds itself for free. The accepted cost, recorded in the ADR because it cuts against ADR-0001: durability is now two things to back up rather than one. ### Serving: one public route, immutable caching Covers are served from a single route on the deployment's own origin, with no session and no credential. The reason is concrete rather than philosophical. An `<img>` cannot send an `Authorization` header, and it cannot be given a token in its URL either: the userscript panel's shadow root is open, so the host page's own JavaScript can read any `src` attribute we set, and a credential in an image URL is a credential handed to a third-party site. Since the route serves public artwork from public Sites and its address reveals nothing about which Reader holds what, public is the correct answer rather than a concession. This does relax the current kagane proxy's session gate, which the ADR records as a knowing decision. Responses keep a long-lived immutable cache directive, which content addressing makes safe. ### Wire contract `GET /bookmarks` returns `cover` as an **absolute** URL on the deployment's public origin, built from a newly required configuration value. Absolute rather than relative because the userscript renders on third-party origins, where a relative path resolves against the Site rather than against us; and per ADR-0004 the server owns the wire shape rather than exporting a join to every client. `cover` is the empty string until the bytes exist. The alternative — always returning an address that 404s until the fetch lands — was rejected because it makes "no Cover yet" and "Cover is broken" indistinguishable in both the UI and the logs, and it turns the client-side fallback into a routine occurrence instead of a genuine last resort. `PUT /bookmarks/{key}` continues to accept a `cover` field and ignores it. Rejecting it would break every installed userscript the moment the server deployed, and ADR-0004's compatibility argument depends on old scripts continuing to work. This extends ADR-0003's "ignored after creation" to "ignored always". The field is permanently inert rather than pending removal, and the decode site must say so — a bare TODO would be a promise nobody intends to keep, which the repo's comment rules forbid. The flat shape of the wire is unchanged in every other respect. ### The Poll fills blanks, never overwrites The Poll's cover prefetch, currently guarded to a single Site, applies to every Site and both Libraries. Nothing about Covers is conditioned on a Bookmark's `kind`; the shared card template already renders both Libraries identically, so this is a guard to remove rather than a branch to add. A Poll fills a Cover only when the stored one is blank. It never replaces a Cover that already exists — a Cover changing under the Reader for no visible reason is noise, and refetching on every Poll would add a request per Series per cycle against Sites that already bot-score the deployment's single IP. This is also what repairs the Series rows that are blank today: the Poll walks every Series anyway, so no migration or backfill script is needed. A failed cover fetch never fails the chapter poll, never blocks it, and never consumes anything the chapter poll depends on. It is retried the next time that Series is polled, with no separate retry queue. ### Clients Both userscripts lose their cover-scraping code: the DOM scan, the metadata reads, and the `cover` field in the request body. A scraped address has no rendering path left, and keeping it would reintroduce third-party URLs into exactly the place this bug came from. Both userscripts gain an error handler on the cover image that swaps in the existing placeholder element on a failed load. This is the only part of the change that is worth doing even in isolation: it is the difference between a Reader seeing a designed empty state and a Reader seeing a broken glyph. The web UI's template-level rewrite disappears, because the address it was rewriting no longer reaches a template. ### Configuration Two new values: the deployment's public base URL, used to build absolute cover addresses, and the cover storage directory. Both are required. A public base URL that is unset cannot be guessed and would produce addresses no client can load, so startup must fail loudly rather than serve them; this matches how the existing configuration treats values it cannot default safely. The storage directory needs a Compose volume alongside the database's. ## Testing Decisions A good test here pins an externally observable promise: what a client receives, what reaches the network, what survives a restart. It does not assert on internal call sequences, private helpers, or the order in which the implementation happens to do things. Every test must be able to fail for a real reason — if a plausible bug in the code under test would still leave it green, it is not earning its place. Tests stay deterministic and touch no live network; the suite already holds that line and this change must not be the exception. Prefer the seams that already exist. There are three, and only one of them is new. **Seam 1 — the composed HTTP router (package `main`). This is the primary seam and most cases belong here.** These tests build the whole router over a throwaway Postgres and drive it with recorded requests, with fetchers injected through the config struct exactly as the existing kagane cover tests inject a fake browser. Prior art: the current cover proxy tests, which already cover persistence, restart survival, a missing fetcher, and rejected input; and the existing API and web handler tests. Cases to cover here: - Creating a Bookmark for a previously unknown Series causes exactly one series-page fetch, and the Cover and the Latest Chapter both land from it. - Creating a Bookmark for a Series that already exists causes no fetch and does not disturb the stored Cover. - A request body containing a `cover` value is accepted and the value is ignored, both at creation and afterwards. - `GET /bookmarks` returns an absolute cover address once bytes exist, and the empty string before they do. - The cover route serves stored bytes with no session and no credential. - The cover route answers not-found for an address that was never stored, and never invents one. - Stored bytes survive a restart of the process against the same volume and database, without a second fetch. - A malformed or traversal-shaped address gets no bytes. - A fetch that returns a non-image content type stores nothing and serves nothing. - A Cover fetch that fails leaves the Bookmark, its Progress and its Latest Chapter intact and the cover field empty. - Both Libraries behave identically: the same assertions hold for a novel Bookmark. **Seam 2 — per-Site extraction (package `latest`).** Pure table tests over HTML and JSON fixtures trimmed from live pages, in the same file and the same style as the existing latest-chapter extraction tests, with each fixture carrying a comment naming the URL and the date it was taken. This is where each Site's quirks get pinned: comix's cover living in the state blob rather than a metadata tag, demonic's raw unencoded space, asura's mismatch between file extension and content type, novelfull's and kagane's browser-fetched bodies, and a Site page that carries no cover at all yielding empty rather than a wrong value. Add the Cloudflare challenge body as a case too — the existing fixtures already include one — and assert it produces no Cover instead of garbage. **Seam 3 — the destination gate (new, and deliberately small).** The classification step needs a seam because the interesting cases cannot be produced through the router: proving that a hostname resolving into private space is refused requires control of resolution. Inject the name lookup as a function on the fetcher, defaulting to the real resolver, in the same spirit as the `Fetcher` interface the Poller already injects for the same reason. Cover: a plain-HTTP URL refused; a literal loopback, private, link-local and CGNAT address refused; a public name allowed; a hostname resolving into private space refused; a redirect from a public host into private space refused mid-chain; an oversized body truncated or rejected rather than buffered. Assert on the observable outcome — no connection made, no bytes stored — rather than on which predicate returned what. Do not add a fourth seam for the Poll's blank-fill rule. The existing poller tests already run a real store with a fake fetcher and assert prefetch behaviour, including that a second poll does not refetch; extend those rather than starting a parallel harness. **Userscripts.** The existing Node harness covers pure logic only — adapters, parsers, helpers — under a hand-written browser stub, and must not grow a DOM harness. The cover work here is a deletion on that side, so the corresponding cover-scrape cases are deleted with it, and the exported symbol lists shrink accordingly. The error-handler fallback is DOM behaviour and is therefore not testable in that harness by design; it is verified on-device, along with the panel actually rendering a kagane Cover, and the report should say so plainly rather than inventing coverage. Run the parse check and the logic tests for both scripts before calling the client side done. **Verification beyond the suite.** The change is not done on the strength of green tests alone. Exercise it against a running backend: bookmark a comix Series from a chapter page and confirm the Cover appears in both the web UI and the panel; do the same for a lightnovelworld Series; confirm a kagane Cover renders in the panel. A live smoke path already exists for kagane images and is gated behind an environment variable so it never runs in CI — extend that pattern rather than putting live fetches in the default suite. ## Out of Scope - **Changing or overriding a Cover.** Refetching a Series' Cover after a Site re-arts it is #54, deliberately deferred until there is an admin surface to hang it off. Until then a stored Cover is not updateable by anyone. Reader-supplied URLs and uploads are rejected outright, not deferred: they would put attacker-controlled input into a row every Reader sees, which ADR-0003 closed on purpose. - **Per-Reader Cover overrides.** Out of scope permanently, consistent with the existing rejection of per-Reader title overrides. A Cover is a fact about the Series. - **Image processing.** No resizing, re-encoding, format conversion, or thumbnail generation. Store what the Site serves, choosing the smaller published rendition where one exists. - **Eviction, quotas or garbage collection** of stored covers. The arithmetic does not justify machinery: a few thousand Series at tens of kilobytes each is a volume measured in tens of megabytes. Revisit if measurement ever contradicts that. - **Covers for Series nobody has bookmarked.** Acquisition is driven by Bookmarks, as polling already is. - **Changing the flat wire format** in any way beyond the meaning of the `cover` value. - **Redesigning the placeholder.** The monogram and hatch placeholders already exist and are unchanged; this spec only decides when they are shown. - **The Cinder design system.** No new visual states, no new tokens, no colour changes. A Cover is either an image or the existing placeholder. - **New Sites.** Six Sites, unchanged. ## Further Notes The two symptoms in #47 look like one bug and are worth keeping separate while implementing, because they fail independently: acquisition (nothing was ever stored) and rendering (something was stored that the client cannot load). A change that fixes only acquisition still leaves kagane broken in the panel; a change that fixes only rendering still leaves comix and lightnovelworld blank forever. Both halves are required for either symptom to be gone. Sequencing that keeps the deployment coherent at every step: backend acquisition and storage first, since it is testable behind the existing harness and changes nothing a client sees; then the wire change, which is when Covers begin appearing in the web UI; then the two userscripts last, so that no installed script is ever given an address the server cannot yet serve. The reverse order would ship a client expecting a route that does not exist. Two live-fact caveats for whoever picks this up. Cloudflare's bot scoring is time-varying, and the repo's guidance is explicit that "does a plain fetch work right now" is a fact to re-check rather than a property of a machine — the measurements in this spec are dated 2026-08-09 and should be re-verified rather than trusted if something behaves unexpectedly. And the extraction anchors are third-party markup: they are pinned by fixtures precisely so that a Site changing its page produces a failing test rather than a library quietly filling with placeholders. Finally, the reason to prefer uniformity over per-Site exceptions is empirical rather than aesthetic. The previous fix for kagane *was* the tidy minimal one — proxy only the Site that needs proxying, in the layer that needed it — and it produced this issue, because the second client did not know about the exception. The property being bought is that there is no exception left for a future client to be unaware of.

sulthan added the ready-for-agent label 2026-08-09 23:03:01 +07:00

~~sulthan referenced this issue 2026-08-09 23:12:11 +07:00~~

Cover bytes move to a content-addressed filesystem store #56

~~sulthan referenced this issue 2026-08-09 23:12:11 +07:00~~