feat(latest): poll comix through the browser sidecar (#98)
comix.to began answering plain-TLS fetches with a Cloudflare JavaScript challenge on 2026-08-12, so every poll got a 403 interstitial and its cover host static.comix.to is gated the same way. comix joins kagane and novelfull as a browser Site: one registry entry, no plain-TLS fallback, and cover bytes routed through the browser's image path behind a fully pinned URL pattern. The read is an in-tab fetch of the Series URL, not a DOM render: comix is an SPA, so rendering costs ~65 requests for the same server-rendered HTML one fetch returns (24.5 KB, ~480 ms). Parsers and stored Series identity are untouched. Verified live against the real browser unit: page 24793 bytes in one fetch, chapter 53, cover accepted by the pin and 26862 image bytes retrieved by direct navigation (comix's Series page sets cross-origin-embedder-policy: require-corp, so an in-page fetch of the cover host cannot work).
This commit is contained in:
+29
-20
@@ -89,12 +89,15 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
||||
order — is never touched.
|
||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||
against fingerprint-based blocking; any failure log and skip. kagane and
|
||||
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
||||
clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane is simply
|
||||
not polled when that's unset, while novelfull falls back to a plain-TLS
|
||||
attempt — its challenge is a live time-varying fact, and its cover bytes
|
||||
never need the browser. See
|
||||
against fingerprint-based blocking; any failure log and skip. kagane, comix
|
||||
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
|
||||
can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and
|
||||
comix are simply not polled when that's unset, while novelfull falls back to
|
||||
a plain-TLS attempt — its challenge is a live time-varying fact, and its
|
||||
cover bytes never need the browser. comix's browser read is an in-tab
|
||||
`fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
|
||||
costs ~65 requests for the same server-rendered HTML one fetch returns
|
||||
(measured 2026-08-12, issue #98). See
|
||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||
The poller's series write is a single-column UPDATE
|
||||
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
||||
@@ -112,14 +115,17 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
is public and uncredentialed: the userscript renders it on a Site's origin,
|
||||
where no cookie or token of ours travels. A client-sent `cover` is decoded
|
||||
and discarded, permanently (ADR-0004 compatibility).
|
||||
Browser-backed Sites join the same pipeline (issue #62): kagane pages *and*
|
||||
cover bytes go through the browser sidecar (nothing falls back to a plain
|
||||
fetch, which would only retrieve a challenge page), while novelfull needs
|
||||
the browser only for its HTML — the cover URL comes out of the
|
||||
browser-fetched page and the bytes go over plain TLS. With no browser
|
||||
configured, kagane Covers are simply absent; novelfull still gets one — at
|
||||
creation and on the poll — when its page body happens to answer a plain
|
||||
request (the challenge is a live time-varying fact). The old kagane-only
|
||||
Browser-backed Sites join the same pipeline (issue #62, extended to comix by
|
||||
#98): kagane and comix pages *and* cover bytes go through the browser sidecar
|
||||
(nothing falls back to a plain fetch, which would only retrieve a challenge
|
||||
page), while novelfull needs the browser only for its HTML — the cover URL
|
||||
comes out of the browser-fetched page and the bytes go over plain TLS. With
|
||||
no browser configured, kagane and comix Covers are simply absent; novelfull
|
||||
still gets one — at creation and on the poll — when its page body happens to
|
||||
answer a plain request (the challenge is a live time-varying fact). comix
|
||||
cover bytes must arrive by direct navigation, not an in-page fetch: its
|
||||
Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
|
||||
page-context fetch of `static.comix.to`. The old kagane-only
|
||||
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
|
||||
(issue #63): the one public route serves every Site.
|
||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||
@@ -164,10 +170,11 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
Reader's credential at serve time).
|
||||
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
||||
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
||||
Used by the poller for kagane and novelfull page fetches and by the cover
|
||||
pipeline for kagane's image bytes (the browser is the only route that clears
|
||||
the challenge kagane serves its covers behind); unset — the default —
|
||||
disables browser polling and leaves kagane Covers blank until stored bytes
|
||||
Used by the poller for kagane, comix and novelfull page fetches and by the
|
||||
cover pipeline for kagane's and comix's image bytes (the browser is the only
|
||||
route that clears the challenge those two serve their covers behind); unset —
|
||||
the default — disables browser polling and leaves kagane and comix Covers
|
||||
blank until stored bytes
|
||||
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
||||
`/json/version` for any Host that isn't an IP or `localhost`).
|
||||
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
|
||||
@@ -175,8 +182,10 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
bytes. There is no proxy, no per-Site rewrite, no second place that decides
|
||||
a Cover's renderable address: the wire `cover` is it. The only place a Site
|
||||
name still appears in cover code is the extraction module (`latest`), where
|
||||
kagane's image URLs are claimed by `browserOnlyCoverURL` — they answer a
|
||||
plain fetch with a challenge and `cross-origin-resource-policy: same-origin`;
|
||||
kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane
|
||||
answers a plain fetch with a challenge and
|
||||
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
|
||||
one with the same Cloudflare challenge its pages serve;
|
||||
every other Site's CDN answers plain TLS. Templates render `.Cover` — the
|
||||
wire value — never anything else.
|
||||
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
||||
|
||||
Reference in New Issue
Block a user