comix.to now behind a Cloudflare JS challenge: poller gets 403, needs the browser sidecar #98
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
comix.to now sits behind a Cloudflare JavaScript challenge, so the plain-TLS poller only ever gets a 403 interstitial:
Verified 2026-08-12 (dev machine, CGNAT residential IP)
Plain
curlwith a stock Chrome UA:https://comix.to/title/m12d-classmatehttps://comix.to/https://static.comix.to/<cover>.jpghttps://asurascans.com/https://demonicscans.org/The 403 carries
cf-mitigated: challenge,server: cloudflare, a 5650-byteJust a moment...body withchallenges.cloudflare.comin the CSP. So this isthe same gate kagane.to and novelfull.com sit behind, not an IP-reputation
block: asura and demonic answer fine from the same IP at the same minute. The
cover host
static.comix.tois gated too, so cover bytes are lost along withthe chapter poll.
Real browser (Playwright, same IP, stock Chrome, non-UTC clock):
https://comix.to/title/m12d-classmate— interstitial served, cleared in under 10s, title becomesClassmate.script#initial-data(22.5 KB) for the cover, and"latestChapterUrl":"/title/m12d-classmate/11187196-chapter-53"for the chapter.comixLatestChapter/comixCoverURLneed no changes.fetch()ofstatic.comix.tofails (the comix page iscross-origin-embedder-policy: require-corp), but<img>loads render and direct navigation to the cover URL renders the image — the same routeBrowserFetcher.Imagealready uses for kagane.Fix shape
Registry-only, matching the existing kagane/novelfull entries in
backend/internal/latest/sites.go:"comix"aBrowser: &browserRead{...}entry —Readreading the rendered DOM,Doneexcluding the interstitial (isInterstitial, as novelfull does),Fallback: false(a plain fetch retrieves only a challenge page, measured above). That alone routes the poll through the sidecar, moves comix ontoBrowserCooldown, and makes it skip cleanly whenBROWSER_WS_URLis unset.browserOnlyCoverURLcurrently matches onlykaganeImageURLRe. Add a pinnedstatic.comix.topattern and teachBrowserFetcher.Imageto accept it, so covers go through the browser too. Pin the full URL (scheme + host + path shape) — the value can be client-supplied and the headless browser is an SSRF primitive.Consequences
BROWSER_WS_URL, comix joins kagane as logged-and-skipped; localdocker compose upwill no longer poll it.sites_test.gofixtures were cut from served HTML; the browser path returns rendered DOM. The anchors are present in both, but new fixtures should come from the rendered page.Cadence research done, and it changes the recommendation in the body.
Research note:
docs/research/cloudflare-bot-scoring-and-poll-cadence.md(22 primary Cloudflare doc pages + RFC 9309, all read 2026-08-12).Findings that matter here:
cf_clearancedefaults to a 30-minute lifetime (Challenge Passage, site-configurable). So any cadence at or above 1 hour re-solves the challenge on every Poll — 1h and 6h differ only in how many solves per day, not in whether one happens.browser.go.So the 6-hour browser cooldown was buying nothing documented. Default lowered to
1hinmain.go(defaultBrowserPollCooldown); the knob stays as the valve to pull if a Site turns hostile.Second measurement, on the comix read shape (Playwright, real Chrome, 2026-08-12):
fetch()of the Series URL returns the server-rendered HTML in one request, 24.5 KB, 480 ms, and it still carries both parser anchors (latestChapterUrlandscript#initial-data).So comix's registry
Readshould be an in-tab fetch, the shapekaganeReadalready uses — not a DOM render likenovelfullRead. At 1h with the in-tab read, comix sees ~24 requests/day/Series. Today's 6h full-render Poll is ~240/day/Series. The faster cadence is a 10x traffic reduction, not an increase.Also measured:
comix.to/api/v1/manga/<id>/chaptersreturns JSON but rejects a call without the SPA's signed_=token ({"message":"Missing token."}), so the API is not a shortcut. The served HTML is.robots.txton comix.to isUser-agent: * / Allow: /withContent-Signal: search=yes,ai-train=no,use=reference; the Disallow list names AI crawlers only. Nothing there forbids this Poll.Not doing, and why: per-Site cooldowns (nothing needs a different number yet) and one-tab-per-wave batching (the profile's cookie jar already shares clearance across a wave, and a 14-Series wave at 20s stagger is 280s, well inside the 30-minute clearance TTL, so consecutive Polls in one wave already skip the challenge).
Problem Statement
As a Reader with comix Series in my manga Library, my Latest Chapter stopped advancing on 2026-08-12 and new comix Series get no Cover at all. From where I sit the Library looks frozen: no ember, no Updated count, and a blank cover slot on anything I add. comix began answering the backend with a Cloudflare JavaScript challenge, so every Poll receives a 403 interstitial instead of a Series page, and its Cover host is gated the same way.
Solution
Make comix a browser Site, joining kagane and novelfull. Its Series pages are read inside a tab that already holds Cloudflare clearance, and its Cover bytes are retrieved by the browser too, because the Cover host answers a plain fetch with the same challenge.
Nothing a Reader touches changes: the userscripts, the wire format, the web UI and the comix adapters all stay as they are. When the browser is absent or unreachable, comix degrades exactly as kagane does — logged, skipped, and already-stored Covers still served.
User Stories
Implementation Decisions
Registry. comix gains a browser read in the Site registry, shaped like the existing kagane and novelfull entries, with no plain-TLS fallback. A plain fetch of comix returns only a challenge page, measured, so a fallback would spend a request to retrieve something unparseable. This single entry also makes comix skip cleanly when the browser is unconfigured, because that behaviour already hangs off the registry.
Read shape: in-tab fetch, not a DOM render. A full navigation of a comix Series page costs about 65 requests, because comix is a single-page application. Once the tab holds clearance, a same-origin fetch of the Series URL returns the server-rendered HTML in one request, 24.5 KB, in roughly 480 ms, and that HTML still carries both parser anchors. This is the shape the kagane read already uses. The novelfull-style DOM render is the wrong choice here and would cost 65× the traffic for the same two facts.
Parsers unchanged. The chapter anchor and the initial-data script that the comix parsers key on are present in both the served HTML and the post-clearance DOM. No parser work is in scope.
Done condition. The read's completion test must exclude the interstitial, the way novelfull's does, so a tab still solving the challenge is not mistaken for a finished page.
Cover route. The browser-only Cover URL gate currently recognises only the kagane image pattern. It gains a pinned pattern for the comix static host, and the browser's image path accepts that pattern. The pin covers scheme, host and path shape together. This is a security boundary, not a tidiness rule: the Cover address can originate in a client-supplied body, and a headless browser is a capable SSRF primitive.
Cover retrieval method. Cover bytes must come from direct navigation to the Cover URL, not from a fetch issued inside the Series page. The comix page sets a cross-origin embedder policy that makes the in-page fetch fail; direct navigation renders the image. This is the same route the browser already uses for kagane Covers.
Rejected: the chapters API. comix exposes a JSON chapters endpoint, but it refuses any call lacking the SPA's signed token. It is not a shortcut. The served HTML is the cheapest complete source.
Cadence. The browser cooldown default was already lowered to one hour when the supporting research landed. This spec makes no further cadence change; per-Site rest times and pacing belong to the Poll Lanes spec that follows this one.
Testing Decisions
A good test here asserts what the backend observably does — which fetcher a Site's Poll is routed to, what a parse yields, and what happens when the browser is absent — never how the routing decision is structured internally.
Site registry fixtures (existing seam). Parse a saved comix Series page and assert the Latest Chapter and the Cover address that come out. The fixture must be cut from what the browser returns, not from a plain-TLS response; the existing comix fixtures predate the challenge and were cut from served HTML. Prior art: the existing per-Site fixture tests that cover asura, demonic, kagane and the novel Sites.
Poller seam (existing seam). With a fake page fetcher and a frozen clock against a real Postgres, assert three things: a comix Series is fetched through the browser fetcher when one is present; it is not fetched at all when the browser is absent; and it never reaches the plain-TLS fetcher in either case. Prior art: the existing tests that assert kagane routes to the browser and that a missing browser skips it.
Cover URL gate. A table test asserting the comix static host is accepted and that near misses are rejected — a different host, a plain-HTTP scheme, and a wrong path shape. Prior art: the existing table test over the fetchable-Series-URL gate, which covers exactly this class of pin.
Live smoke check. A test skipped unless a browser address is supplied by environment variable, fetching a real comix Series page and Cover, mirroring the existing kagane smoke test. A red run means the challenge is not clearing from that address right now, which is a live fact to re-check rather than automatically a defect.
Out of Scope
Further Notes
comix's robots.txt is
User-agent: * / Allow: /with a content signal permitting reference use; its disallow list names AI training crawlers only. Nothing there forbids this Poll.With comix included, three of the six Sites need the browser, so the sidecar is no longer optional in practice for the manga Library. That does not change this spec, but it does raise the cost of the browser being down, which the dashboard spec addresses by making that state visible.
Verification evidence for every measurement quoted above is in the issue body and its research comment, together with the supporting research note on Cloudflare protection and Poll cadence.