Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 55ecb29b81 | |||
| 82a11f255b | |||
| f000cc7eaa | |||
| 86160c164a | |||
| 3f53c79cf4 | |||
| 17ee0bd3f8 |
@@ -18,11 +18,11 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
|
||||
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
||||
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
|
||||
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
||||
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
||||
- **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`). When that's unset, kagane is skipped entirely (a plain fetch would only retrieve a challenge page) while novelfull pages are still attempted over plain TLS — its challenge is a live time-varying fact and its cover bytes never need the browser. The four other sites poll fine over plain TLS.
|
||||
- Cloudflare's block on manga sites is **per-zone configuration plus request fingerprint, not IP reputation — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — a Site can turn its protection on overnight, which is exactly what comix.to did on 2026-08-12. An earlier version of this line blamed "Cloudflare's bot scoring"; that was wrong. The 1-99 bot score is Enterprise Bot Management only and does not exist for a free-plan zone, and no per-IP request rate is documented as an input to challenge issuance — `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
||||
- **kagane.to, comix.to and novelfull.com are the exception to the above** — all three sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`). When that's unset, kagane and comix are skipped entirely (a plain fetch would only retrieve a challenge page) while novelfull pages are still attempted over plain TLS — its challenge is a live time-varying fact and its cover bytes never need the browser. comix turned hostile on 2026-08-12 (#98): its cover host `static.comix.to` is gated too, so its cover bytes go through the browser as well, and its page is read as an in-tab `fetch()` of the series URL rather than a rendered DOM — comix is an SPA, and rendering costs ~65 requests for the same server-rendered HTML one fetch returns. The three other sites poll fine over plain TLS.
|
||||
- **The CDP browser must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a non-UTC clock zone cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, ignoring the file's contents, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
|
||||
- **The browser is not in the API stack and must not be put back.** It's its own compose unit (`chrome/docker-compose.yml`) on a second machine, reached over the tailnet — it held 471 MiB on a 1974 MiB swapless VPS, and a residential egress scores better with Cloudflare anyway (ADR-0006). Consequences that constrain code: `BROWSER_WS_URL` must be a tailnet **IP** (a MagicDNS name 500s at `/json/version`, same trap as the old Docker service name); the CDP port binds to the tailnet address only, since CDP authenticates nothing and that host has a real LAN; and the browser is on-demand (ADR-0005), so an unreachable or asleep one must degrade exactly as an unset `BROWSER_WS_URL` — plain-TLS libraries unaffected, kagane/novelfull logged and skipped, stored covers still served. Never add `chromedp.NoModifyURL`: discovery per fetch is what makes a restarted Chrome invisible.
|
||||
- **UTC is the tell, not a country mismatch.** A UTC clock is the datacenter default, so Cloudflare scores it as one; any real zone clears. Measured 2026-08-08, identical container, one Indonesian egress IP: UTC never cleared in 60s (twice), while `Asia/Jakarta` **and** `America/New_York` both cleared in 4s. An earlier note here claimed the zone had to match the egress IP's country — that was wrong, inferred from the host clock (`Asia/Bangkok`) rather than the measured egress. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one.
|
||||
- **The browser is not in the API stack and must not be put back.** It's its own compose unit (`chrome/docker-compose.yml`) on a second machine, reached over the tailnet — it held 471 MiB on a 1974 MiB swapless VPS, and a residential egress avoids the cloud-hosting-IP signature Bot Fight Mode documentedly challenges (ADR-0006; not a better "score" — free-plan zones have no score). Consequences that constrain code: `BROWSER_WS_URL` must be a tailnet **IP** (a MagicDNS name 500s at `/json/version`, same trap as the old Docker service name); the CDP port binds to the tailnet address only, since CDP authenticates nothing and that host has a real LAN; and the browser is on-demand (ADR-0005), so an unreachable or asleep one must degrade exactly as an unset `BROWSER_WS_URL` — plain-TLS libraries unaffected, kagane/comix logged and skipped, stored covers still served. Never add `chromedp.NoModifyURL`: discovery per fetch is what makes a restarted Chrome invisible.
|
||||
- **UTC is the tell, not a country mismatch.** A UTC clock is the datacenter default, and the challenge refuses it; any real zone clears. Measured 2026-08-08, identical container, one Indonesian egress IP: UTC never cleared in 60s (twice), while `Asia/Jakarta` **and** `America/New_York` both cleared in 4s. An earlier note here claimed the zone had to match the egress IP's country — that was wrong, inferred from the host clock (`Asia/Bangkok`) rather than the measured egress. A second earlier claim, that Cloudflare "scores" a UTC clock, was also wrong: the measurement is real but the mechanism is not documented anywhere — Cloudflare publishes no timezone signal, and free-plan zones carry no score at all. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one.
|
||||
- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
|
||||
|
||||
## Architecture
|
||||
@@ -45,12 +45,20 @@ Backend (`cd backend`):
|
||||
- Single test: `go test -run TestName ./...`
|
||||
- Build static binary: `CGO_ENABLED=0 go build`
|
||||
|
||||
Local stack: `docker compose up` (bookmark-api + postgres only; `postgres-data` named volume, `restart: unless-stopped`). No browser — without `BROWSER_WS_URL` the poller logs and skips kagane and novelfull. To run one: `cd chrome && BROWSER_BIND_ADDR=172.17.0.1 docker compose up -d --build`, then `BROWSER_WS_URL=ws://172.17.0.1:9222` in the root `.env` (bridge gateway, so the API container can name it by IP).
|
||||
Local stack: `docker compose up` (bookmark-api + postgres only; `postgres-data` named volume, `restart: unless-stopped`). No browser — without `BROWSER_WS_URL` the poller logs and skips kagane and comix. To run one: `cd chrome && BROWSER_BIND_ADDR=172.17.0.1 docker compose up -d --build`, then `BROWSER_WS_URL=ws://172.17.0.1:9222` in the root `.env` (bridge gateway, so the API container can name it by IP).
|
||||
|
||||
Live CDP proof (needs that browser and network, skipped otherwise):
|
||||
`SMOKE_BROWSER_WS_URL=ws://<ip>:<port> go test -run TestSmokeKagane ./internal/latest`
|
||||
— fetches a real kagane cover and chapter list. A red run means the challenge is
|
||||
`SMOKE_BROWSER_WS_URL=ws://<ip>:<port> go test -run 'TestSmokeKagane|TestSmokeComix' ./internal/latest`
|
||||
— fetches a real kagane and comix cover and chapter list. A red run means the challenge is
|
||||
not clearing from this IP, which is a live fact to re-check, not necessarily a defect.
|
||||
A red `TestSmokeComix` reporting `ERR_CERT_COMMON_NAME_INVALID` is not the
|
||||
challenge: it means the resolver the browser container uses hijacks `comix.to`.
|
||||
Observed 2026-08-16 on one Indonesian ISP, which CNAMEs it to a block page
|
||||
(`aduankonten.id`). Check with `docker exec <browser> getent hosts comix.to`,
|
||||
and if it is hijacked, run the container with
|
||||
`--add-host comix.to:<ip> --add-host static.comix.to:<ip>` from a DoH lookup
|
||||
(`curl -H 'accept: application/dns-json' 'https://1.1.1.1/dns-query?name=comix.to&type=A'`).
|
||||
Machine-local, so don't put those hosts in `chrome/docker-compose.yml`.
|
||||
|
||||
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
||||
|
||||
|
||||
+20
-6
@@ -56,12 +56,12 @@ is real activity, so only Progress reorders the list.
|
||||
_Avoid_: position, bookmark (the noun is taken), last read
|
||||
|
||||
**Latest Chapter**:
|
||||
The highest-numbered chapter a Site has published for a Series, discovered without the
|
||||
reader present. The number is what ranks it, never a date and never the Site's own
|
||||
"newest chapter" banner — where a Site disagrees with itself, its list of chapters is
|
||||
the record and its summary of that list is not. Distinct from Progress in every way
|
||||
that matters: it is a fact about the Site, not about the reader, and it must never
|
||||
reorder the list.
|
||||
The highest-numbered chapter a Site has published for a Series. The number is what
|
||||
ranks it, never a date and never the Site's own "newest chapter" banner — where a Site
|
||||
disagrees with itself, its list of chapters is the record and its summary of that list
|
||||
is not. Established by a Poll and, between Polls, by a Sighting. Distinct from Progress
|
||||
in every way that matters: it is a fact about the Site, not about the reader, and it
|
||||
must never reorder the list.
|
||||
_Avoid_: newest, current chapter, update
|
||||
|
||||
**Poll**:
|
||||
@@ -70,6 +70,20 @@ Reader present. Performed once per Series no matter how many Readers bookmarked
|
||||
a Poll is work done on behalf of the Series, never on behalf of a Reader.
|
||||
_Avoid_: scrape, refresh, check, sync
|
||||
|
||||
**Poll Lane**:
|
||||
One Site's own stream of Polls, carrying the pace at which that Site is willing to be
|
||||
asked. Every Site has exactly one and no Lane can slow, block or borrow from another's;
|
||||
a Reader never has one and never influences one.
|
||||
_Avoid_: worker, queue, scheduler, batch, wave
|
||||
|
||||
**Sighting**:
|
||||
What a Reader's browser happened to see of a Series's Latest Chapter while that Reader
|
||||
was on the page. It reports the same fact as a Poll but carries none of its authority:
|
||||
a Poll always overrules it, and only a Sighting on a Series no Reader else holds may
|
||||
defer one. A Sighting a later Poll contradicts downwards is a false Sighting, and
|
||||
enough of those cost the Reader the right to defer at all.
|
||||
_Avoid_: client report, user poll, observation, claim
|
||||
|
||||
**Acquisition**:
|
||||
The single read of a Series page made the moment the Series first exists, giving it
|
||||
both its Latest Chapter and its Cover without waiting out the Poll queue. Distinct
|
||||
|
||||
@@ -275,7 +275,8 @@ copy immediately — reinstall on all devices, or they silently stop syncing.
|
||||
Kagane and novelfull sit behind a Cloudflare JavaScript challenge no TLS
|
||||
fingerprint clears, so the poller reaches them through a real Chrome over CDP.
|
||||
That browser does **not** run on the VPS: it held 471 MiB of a 1974 MiB box
|
||||
with no swap, and it scores better from a residential IP anyway (ADR-0006). It
|
||||
with no swap, and a residential IP avoids the cloud-hosting-IP signature Bot
|
||||
Fight Mode challenges anyway (ADR-0006). It
|
||||
is its own compose unit, deployed and updated independently of everything
|
||||
above.
|
||||
|
||||
@@ -454,7 +455,8 @@ SMOKE_BROWSER_WS_URL=ws://100.x.y.z:9222 go test -run TestSmokeKagane ./internal
|
||||
```
|
||||
|
||||
A red run means "not clearing from this address right now", which is a live
|
||||
fact to re-check before it is a defect — Cloudflare's scoring moves. Then, from
|
||||
fact to re-check before it is a defect — a Site's Cloudflare settings, and the
|
||||
fingerprint this Chrome presents after an update, both move. Then, from
|
||||
the web UI, open a bookmarked kagane series and confirm the cover renders. Once
|
||||
a cover is stored it is served from Postgres forever after, so the browser being
|
||||
asleep, unreachable, or mid-power-outage costs chapter freshness and nothing
|
||||
|
||||
+2
-2
@@ -436,8 +436,8 @@ free -m # the Gitea runner should still have its headroom
|
||||
|
||||
Nothing here needs doing during an API redeploy. The API stack does not
|
||||
`depends_on` the browser, and an unreachable one degrades exactly as an unset
|
||||
`BROWSER_WS_URL`: plain-TLS libraries unaffected, kagane and novelfull logged
|
||||
and skipped, stored covers still served.
|
||||
`BROWSER_WS_URL`: plain-TLS libraries unaffected, kagane and comix logged
|
||||
and skipped, novelfull attempted over plain TLS, stored covers still served.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+36
-21
@@ -89,12 +89,15 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
||||
order — is never touched.
|
||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||
against fingerprint-based blocking; any failure log and skip. kagane and
|
||||
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
||||
clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane is simply
|
||||
not polled when that's unset, while novelfull falls back to a plain-TLS
|
||||
attempt — its challenge is a live time-varying fact, and its cover bytes
|
||||
never need the browser. See
|
||||
against fingerprint-based blocking; any failure log and skip. kagane, comix
|
||||
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
|
||||
can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and
|
||||
comix are simply not polled when that's unset, while novelfull falls back to
|
||||
a plain-TLS attempt — its challenge is a live time-varying fact, and its
|
||||
cover bytes never need the browser. comix's browser read is an in-tab
|
||||
`fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
|
||||
costs ~65 requests for the same server-rendered HTML one fetch returns
|
||||
(measured 2026-08-12, issue #98). See
|
||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||
The poller's series write is a single-column UPDATE
|
||||
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
||||
@@ -112,14 +115,17 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
is public and uncredentialed: the userscript renders it on a Site's origin,
|
||||
where no cookie or token of ours travels. A client-sent `cover` is decoded
|
||||
and discarded, permanently (ADR-0004 compatibility).
|
||||
Browser-backed Sites join the same pipeline (issue #62): kagane pages *and*
|
||||
cover bytes go through the browser sidecar (nothing falls back to a plain
|
||||
fetch, which would only retrieve a challenge page), while novelfull needs
|
||||
the browser only for its HTML — the cover URL comes out of the
|
||||
browser-fetched page and the bytes go over plain TLS. With no browser
|
||||
configured, kagane Covers are simply absent; novelfull still gets one — at
|
||||
creation and on the poll — when its page body happens to answer a plain
|
||||
request (the challenge is a live time-varying fact). The old kagane-only
|
||||
Browser-backed Sites join the same pipeline (issue #62, extended to comix by
|
||||
#98): kagane and comix pages *and* cover bytes go through the browser sidecar
|
||||
(nothing falls back to a plain fetch, which would only retrieve a challenge
|
||||
page), while novelfull needs the browser only for its HTML — the cover URL
|
||||
comes out of the browser-fetched page and the bytes go over plain TLS. With
|
||||
no browser configured, kagane and comix Covers are simply absent; novelfull
|
||||
still gets one — at creation and on the poll — when its page body happens to
|
||||
answer a plain request (the challenge is a live time-varying fact). comix
|
||||
cover bytes must arrive by direct navigation, not an in-page fetch: its
|
||||
Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
|
||||
page-context fetch of `static.comix.to`. The old kagane-only
|
||||
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
|
||||
(issue #63): the one public route serves every Site.
|
||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||
@@ -149,7 +155,13 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/
|
||||
`_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on,
|
||||
`1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both
|
||||
cooldowns have a `15m` floor).
|
||||
cooldowns have a `15m` floor). The browser cooldown is longer for cost, not
|
||||
for safety: a challenged page costs seconds of a serialized single-tab
|
||||
browser, while a plain read costs one request. It buys no documented
|
||||
reduction in challenge risk — free-plan zones have no bot score and no
|
||||
published per-IP rate input, and `cf_clearance` expires in 30 minutes so
|
||||
every cadence at or above 1h re-solves anyway —
|
||||
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
|
||||
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
||||
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
||||
defaults `/userscript/manga-bookmark.user.js` and
|
||||
@@ -158,10 +170,11 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
Reader's credential at serve time).
|
||||
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
||||
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
||||
Used by the poller for kagane and novelfull page fetches and by the cover
|
||||
pipeline for kagane's image bytes (the browser is the only route that clears
|
||||
the challenge kagane serves its covers behind); unset — the default —
|
||||
disables browser polling and leaves kagane Covers blank until stored bytes
|
||||
Used by the poller for kagane, comix and novelfull page fetches and by the
|
||||
cover pipeline for kagane's and comix's image bytes (the browser is the only
|
||||
route that clears the challenge those two serve their covers behind); unset —
|
||||
the default — disables browser polling and leaves kagane and comix Covers
|
||||
blank until stored bytes
|
||||
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
||||
`/json/version` for any Host that isn't an IP or `localhost`).
|
||||
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
|
||||
@@ -169,8 +182,10 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
bytes. There is no proxy, no per-Site rewrite, no second place that decides
|
||||
a Cover's renderable address: the wire `cover` is it. The only place a Site
|
||||
name still appears in cover code is the extraction module (`latest`), where
|
||||
kagane's image URLs are claimed by `browserOnlyCoverURL` — they answer a
|
||||
plain fetch with a challenge and `cross-origin-resource-policy: same-origin`;
|
||||
kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane
|
||||
answers a plain fetch with a challenge and
|
||||
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
|
||||
one with the same Cloudflare challenge its pages serve;
|
||||
every other Site's CDN answers plain TLS. Templates render `.Cover` — the
|
||||
wire value — never anything else.
|
||||
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
||||
|
||||
@@ -24,12 +24,16 @@ const challengeTimeout = 45 * time.Second
|
||||
|
||||
var kaganeSeriesRe = regexp.MustCompile(`^/series/([0-9a-f-]{36})/?$`)
|
||||
|
||||
// comixSeriesPathRe matches the one path shape comixRead will open: a Series
|
||||
// page, "/title/<id>-<slug>". Verified live 2026-08-12.
|
||||
var comixSeriesPathRe = regexp.MustCompile(`^/title/[^/?#]+/?$`)
|
||||
|
||||
// BrowserFetcher retrieves pages through a remote headless Chrome over the
|
||||
// DevTools Protocol.
|
||||
//
|
||||
// It exists for one reason: kagane.to and novelfull.com sit behind a
|
||||
// Cloudflare JavaScript challenge. Verified 2026-08-03 (kagane) and 2026-08-05
|
||||
// (novelfull) from the deployment host, plain HTTP and bogdanfinn/tls-client
|
||||
// It exists for one reason: kagane.to, novelfull.com and comix.to sit behind a
|
||||
// Cloudflare JavaScript challenge. Verified 2026-08-03 (kagane), 2026-08-05
|
||||
// (novelfull) and 2026-08-12 (comix), plain HTTP and bogdanfinn/tls-client
|
||||
// with a Chrome_133 profile both get 403 with cf-mitigated: challenge on every
|
||||
// path, including the API, robots.txt and images. Clearing it requires
|
||||
// executing the challenge script, which only a real browser does.
|
||||
@@ -40,10 +44,11 @@ var kaganeSeriesRe = regexp.MustCompile(`^/series/([0-9a-f-]{36})/?$`)
|
||||
// sync that break silently and separately. The browser's own cookie jar
|
||||
// persists across polls, so the challenge is solved once every few hours.
|
||||
//
|
||||
// The two sites differ in how the chapter list is read: kagane serves it from
|
||||
// a JSON API that must be called from inside the page (so the request carries
|
||||
// the clearance cookie), while novelfull renders it into the HTML so the
|
||||
// cleared DOM is the payload.
|
||||
// The three sites differ in what a cleared tab is asked for: kagane fetches a
|
||||
// JSON API from inside the page (the list exists nowhere else), comix fetches
|
||||
// its own Series URL from inside the page (the served HTML carries the facts,
|
||||
// and rendering the SPA costs ~65 requests instead of one), and novelfull
|
||||
// renders its list into the HTML so the cleared DOM is the payload.
|
||||
type BrowserFetcher struct {
|
||||
allocCtx context.Context
|
||||
cancel context.CancelFunc
|
||||
@@ -87,10 +92,9 @@ func (f *BrowserFetcher) Close() {
|
||||
}
|
||||
|
||||
// Get navigates to seriesURL, lets any challenge resolve, then reads the
|
||||
// payload the Site's registry entry describes — kagane's chapter-list API from
|
||||
// inside the page so the request carries the clearance cookie, novelfull's
|
||||
// served HTML. The returned body is whatever the Site's chapter list lives in,
|
||||
// which is what the entry's LatestChapter parse expects.
|
||||
// payload the Site's registry entry describes (the shapes are listed on
|
||||
// BrowserFetcher). The returned body is whatever the Site's chapter list lives
|
||||
// in, which is what the entry's LatestChapter parse expects.
|
||||
func (f *BrowserFetcher) Get(ctx context.Context, seriesURL string) (string, int, error) {
|
||||
var body string
|
||||
// Sorted order (browserBackedSites sorts) makes dispatch deterministic:
|
||||
@@ -141,29 +145,47 @@ func novelfullRead(seriesURL string, out *string) (chromedp.Action, bool) {
|
||||
return chromedp.OuterHTML("html", out, chromedp.ByQuery), true
|
||||
}
|
||||
|
||||
// comixRead fetches the Series page from inside the cleared tab. comix is an
|
||||
// SPA: rendering the page costs ~65 requests, while one same-origin fetch of
|
||||
// the same address returns the server-rendered HTML — 24.5 KB, ~480 ms,
|
||||
// carrying both parser anchors (measured 2026-08-12, issue #98). So this is
|
||||
// kaganeRead's shape, not novelfullRead's, even though the payload is HTML.
|
||||
// Refusing any other address is the per-Site half of the SSRF gate.
|
||||
func comixRead(seriesURL string, out *string) (chromedp.Action, bool) {
|
||||
pageURL, ok := comixSeriesPageURL(seriesURL)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
return chromedp.Evaluate(
|
||||
`fetch(`+jsString(pageURL)+`).then(r => r.ok ? r.text() : "")`,
|
||||
out, awaitPromise), true
|
||||
}
|
||||
|
||||
// Image retrieves one cover's bytes through the browser sidecar, and its
|
||||
// content type.
|
||||
//
|
||||
// It exists because kagane serves covers behind the same challenge as its
|
||||
// pages *and* with `cross-origin-resource-policy: same-origin`, so the bytes
|
||||
// are only reachable from inside a browser that already holds the clearance
|
||||
// cookie (verified 2026-08-08). Acquisition through the sidecar is the only
|
||||
// route.
|
||||
// It exists because kagane and comix serve covers behind the same challenge as
|
||||
// their pages — kagane additionally with
|
||||
// `cross-origin-resource-policy: same-origin` — so the bytes are only
|
||||
// reachable from inside a browser that already holds the clearance cookie
|
||||
// (verified 2026-08-08 for kagane, 2026-08-12 for comix). Acquisition through
|
||||
// the sidecar is the only route.
|
||||
//
|
||||
// The image URL is navigated to rather than fetched from some other kagane
|
||||
// page: the challenge only runs on a top-level navigation, and once it clears
|
||||
// The image URL is navigated to rather than fetched from another page of the
|
||||
// Site: the challenge only runs on a top-level navigation, and once it clears
|
||||
// the document *is* the image, so a same-origin fetch of location.href reads
|
||||
// it straight back out of the cache.
|
||||
// it straight back out of the cache. For comix the navigation is also the only
|
||||
// route that works at all — its Series page sets
|
||||
// `cross-origin-embedder-policy: require-corp`, which fails a page-context
|
||||
// fetch of the cover host.
|
||||
//
|
||||
// The challenge is not solved by the first read: WaitReady("body") is satisfied
|
||||
// by the interstitial too. run holds the tab open until the in-page fetch
|
||||
// succeeds, which is what gives the challenge script the seconds it needs.
|
||||
func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, string, error) {
|
||||
m := kaganeImageURLRe.FindStringSubmatch(imageURL)
|
||||
if m == nil {
|
||||
if !browserOnlyCoverURL(imageURL) {
|
||||
return nil, "", fmt.Errorf("not a browser-fetchable cover url: %q", imageURL)
|
||||
}
|
||||
imageID := m[1]
|
||||
var dataURL string
|
||||
err := f.run(ctx, imageURL,
|
||||
chromedp.Evaluate(`fetch(location.href).then(r => r.ok
|
||||
@@ -175,16 +197,16 @@ func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, st
|
||||
: "")`, &dataURL, awaitPromise),
|
||||
func() bool { return dataURL != "" })
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("browser image %s: %w", imageID, err)
|
||||
return nil, "", fmt.Errorf("browser image %s: %w", imageURL, err)
|
||||
}
|
||||
// "data:image/webp;base64,<payload>".
|
||||
head, payload, ok := strings.Cut(dataURL, ";base64,")
|
||||
if !ok {
|
||||
return nil, "", fmt.Errorf("browser image %s: not a data url", imageID)
|
||||
return nil, "", fmt.Errorf("browser image %s: not a data url", imageURL)
|
||||
}
|
||||
raw, err := base64.StdEncoding.DecodeString(payload)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("browser image %s: %w", imageID, err)
|
||||
return nil, "", fmt.Errorf("browser image %s: %w", imageURL, err)
|
||||
}
|
||||
return raw, strings.TrimPrefix(head, "data:"), nil
|
||||
}
|
||||
@@ -323,6 +345,19 @@ func novelfullSeriesURL(seriesURL string) bool {
|
||||
strings.HasSuffix(u.Path, ".html")
|
||||
}
|
||||
|
||||
// comixSeriesPageURL returns the address comixRead fetches inside the tab: the
|
||||
// Series page itself, rebuilt from the pinned host and path so nothing else
|
||||
// travels. Host-pinned here for the same reason kagane's is — series_url is
|
||||
// client-supplied and a headless browser is a strong SSRF primitive.
|
||||
func comixSeriesPageURL(seriesURL string) (string, bool) {
|
||||
u, err := url.Parse(seriesURL)
|
||||
if err != nil || u.Scheme != "https" || u.Hostname() != "comix.to" ||
|
||||
!comixSeriesPathRe.MatchString(u.Path) {
|
||||
return "", false
|
||||
}
|
||||
return "https://comix.to" + u.Path, true
|
||||
}
|
||||
|
||||
// awaitPromise makes Evaluate resolve the promise rather than returning a
|
||||
// serialised Promise object.
|
||||
func awaitPromise(p *runtime.EvaluateParams) *runtime.EvaluateParams {
|
||||
|
||||
@@ -61,6 +61,62 @@ func TestNovelfullSeriesURL(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestComixSeriesPageURL(t *testing.T) {
|
||||
const series = "https://comix.to/title/n8we-dungeons-and-crayons"
|
||||
cases := []struct {
|
||||
name string
|
||||
url string
|
||||
want string
|
||||
}{
|
||||
{"series page", series, series},
|
||||
{"trailing slash kept", series + "/", series + "/"},
|
||||
// Query and fragment are dropped: only the pinned path travels.
|
||||
{"query dropped", series + "?tab=chapters", series},
|
||||
{"foreign host", "https://evil.example/title/x", ""},
|
||||
{"lookalike host", "https://comix.to.evil.example/title/x", ""},
|
||||
{"not https", "http://comix.to/title/x", ""},
|
||||
{"not a series path", "https://comix.to/search", ""},
|
||||
{"chapter page", series + "/11139891-chapter-80", ""},
|
||||
{"garbage", "://nope", ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got, ok := comixSeriesPageURL(tc.url)
|
||||
if ok != (tc.want != "") || got != tc.want {
|
||||
t.Fatalf("comixSeriesPageURL(%q) = %q, %v; want %q", tc.url, got, ok, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The browser is an SSRF primitive and a cover address can originate in a
|
||||
// client-supplied PUT body, so this gate decides what it may navigate to.
|
||||
func TestBrowserOnlyCoverURL(t *testing.T) {
|
||||
cases := []struct {
|
||||
url string
|
||||
want bool
|
||||
}{
|
||||
{"https://static.comix.to/039d/i/1/34/6a6742bf15736@280.jpg", true},
|
||||
{"https://kagane.to/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed", true},
|
||||
// Every other Site's CDN answers plain TLS.
|
||||
{"https://gg.asuracomic.net/covers/x.webp", false},
|
||||
{"http://static.comix.to/039d/x.jpg", false},
|
||||
{"https://static.comix.to.evil.example/039d/x.jpg", false},
|
||||
{"https://evil.example/static.comix.to/x.jpg", false},
|
||||
{"https://static.comix.to/039d/x.jpg?next=http://169.254.169.254/", false},
|
||||
{"https://static.comix.to/039d/x.svg", false},
|
||||
{"https://static.comix.to/../etc/passwd.jpg", false},
|
||||
{"https://static.comix.to/", false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.url, func(t *testing.T) {
|
||||
if got := browserOnlyCoverURL(tc.url); got != tc.want {
|
||||
t.Fatalf("browserOnlyCoverURL(%q) = %v, want %v", tc.url, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
func TestClassifyBrowserInterruption(t *testing.T) {
|
||||
if err := classifyBrowserError(context.Background(), true, context.Canceled); !errors.Is(err, errBrowserInterrupted) {
|
||||
t.Fatalf("classifyBrowserError(context.Canceled) = %v, want browser interruption", err)
|
||||
|
||||
@@ -25,7 +25,8 @@ type CoverBytesFetcher interface {
|
||||
// fetchCoverBytes routes a cover's byte retrieval by URL shape, not by Site
|
||||
// name: the browser fetcher's module claims the addresses only it can fetch
|
||||
// (kagane's image route answers a plain fetch with a challenge and
|
||||
// `cross-origin-resource-policy: same-origin`), and everything else goes over
|
||||
// `cross-origin-resource-policy: same-origin`, static.comix.to answers one with
|
||||
// the same challenge its pages serve), and everything else goes over
|
||||
// plain TLS. Missing fetchers degrade to an error the caller logs, never a
|
||||
// fallback onto a path that cannot succeed. One routing rule for the poll and
|
||||
// the acquirer, so the two cannot drift apart.
|
||||
|
||||
@@ -17,8 +17,8 @@ type Fetcher interface {
|
||||
}
|
||||
|
||||
// BrowserCoverFetcher retrieves one cover's bytes through the browser-backed
|
||||
// path — the only route that clears the challenge kagane's image URLs answer
|
||||
// a plain fetch with. Satisfied by BrowserFetcher.
|
||||
// path — the only route that clears the challenge kagane's and comix's image
|
||||
// URLs answer a plain fetch with. Satisfied by BrowserFetcher.
|
||||
type BrowserCoverFetcher interface {
|
||||
Image(ctx context.Context, imageURL string) (body []byte, contentType string, err error)
|
||||
}
|
||||
@@ -119,9 +119,9 @@ func (p *Poller) storeCover(ctx context.Context, sr store.Series, sourceURL stri
|
||||
|
||||
// fetcherFor returns the fetcher a site's page needs, or nil when the site
|
||||
// cannot be fetched at all right now. A Site whose registry entry carries a
|
||||
// Browser read — kagane and novelfull, both behind a Cloudflare JavaScript
|
||||
// challenge no TLS fingerprint clears — prefers the browser; when it is
|
||||
// absent, the entry's Fallback decides whether plain TLS may take over. One
|
||||
// Browser read — kagane, comix and novelfull, all behind a Cloudflare
|
||||
// JavaScript challenge no TLS fingerprint clears — prefers the browser; when it
|
||||
// is absent, the entry's Fallback decides whether plain TLS may take over. One
|
||||
// routing rule for the poll and the acquirer, so the two cannot drift apart.
|
||||
func fetcherFor(site string, browser, tls Fetcher) Fetcher {
|
||||
s, known := sites[site]
|
||||
|
||||
@@ -762,6 +762,86 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// comix joined kagane behind the challenge on 2026-08-12 (#98): its page goes
|
||||
// to the browser, its Cover bytes go through the browser's image route because
|
||||
// static.comix.to is gated the same way, and the TLS fetcher is never asked
|
||||
// for either.
|
||||
func TestComixUsesBrowserFetcher(t *testing.T) {
|
||||
s, dbURL := newTestStore(t)
|
||||
const (
|
||||
key = "comix:n8we-dungeons-and-crayons"
|
||||
seriesID = "n8we-dungeons-and-crayons"
|
||||
seriesURL = "https://comix.to/title/n8we-dungeons-and-crayons"
|
||||
coverURL = "https://static.comix.to/039d/i/1/34/6a6742bf15736@280.jpg"
|
||||
)
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key, Site: "comix", SeriesID: seriesID, SeriesURL: seriesURL,
|
||||
UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
seedCoverSource(t, dbURL, "comix", seriesID, coverURL)
|
||||
|
||||
tlsF := &fakeFetcher{body: "", status: 200}
|
||||
browserF := &fakeFetcher{body: comixSeriesFixture, status: 200}
|
||||
covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}
|
||||
tlsCovers := &fakeBytesCoverFetcher{body: []byte("tls-bytes"), contentType: "image/jpeg"}
|
||||
p := &Poller{
|
||||
Store: s, Fetch: tlsF, BrowserFetch: browserF,
|
||||
CoverFetch: covers, CoverBytesFetch: tlsCovers,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
if len(tlsF.calls) != 0 {
|
||||
t.Errorf("TLS fetcher was called for comix: %v", tlsF.calls)
|
||||
}
|
||||
if len(browserF.calls) != 1 {
|
||||
t.Fatalf("browser fetcher calls = %v, want 1", browserF.calls)
|
||||
}
|
||||
if got := tlsCovers.callCount(); got != 0 {
|
||||
t.Errorf("TLS cover fetches = %d, want 0: static.comix.to answers a challenge", got)
|
||||
}
|
||||
if got := covers.callCount(); got != 1 {
|
||||
t.Fatalf("browser cover fetches = %d, want 1", got)
|
||||
}
|
||||
got, found, err := s.Get(s.OwnerID(), key)
|
||||
if err != nil || !found {
|
||||
t.Fatalf("Get: %v found=%v", err, found)
|
||||
}
|
||||
if got.LatestChapterNum == nil || *got.LatestChapterNum != 80 {
|
||||
t.Errorf("LatestChapterNum = %v, want 80", got.LatestChapterNum)
|
||||
}
|
||||
}
|
||||
|
||||
// Without a browser, comix is skipped outright rather than handed to plain
|
||||
// TLS: a plain fetch retrieves only a challenge page (measured 2026-08-12).
|
||||
func TestComixSkippedWhenNoBrowserFetcher(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: "comix:n8we-dungeons-and-crayons", Site: "comix",
|
||||
SeriesID: "n8we-dungeons-and-crayons",
|
||||
SeriesURL: "https://comix.to/title/n8we-dungeons-and-crayons",
|
||||
UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
f := &fakeFetcher{body: comixSeriesFixture, status: 200}
|
||||
p := &Poller{
|
||||
Store: s, Fetch: f,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
if len(f.calls) != 0 {
|
||||
t.Errorf("TLS fetcher was called for comix: %v", f.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunOncePrefetchesKaganeCover(t *testing.T) {
|
||||
s, dbURL := newTestStore(t)
|
||||
const (
|
||||
|
||||
@@ -47,9 +47,9 @@ type browserRead struct {
|
||||
// Done reports whether the payload arrived.
|
||||
Done func(body string) bool
|
||||
// Fallback allows the plain-TLS fetcher when no browser is configured.
|
||||
// False skips the Site instead. kagane is false — a plain fetch would
|
||||
// only ever retrieve a challenge page — and novelfull is true, because
|
||||
// its challenge is a live time-varying fact (AGENTS.md).
|
||||
// False skips the Site instead. kagane and comix are false — a plain fetch
|
||||
// would only ever retrieve a challenge page — and novelfull is true,
|
||||
// because its challenge is a live time-varying fact (AGENTS.md).
|
||||
Fallback bool
|
||||
}
|
||||
|
||||
@@ -248,14 +248,25 @@ var comixInitialDataRe = regexp.MustCompile(`(?is)<script\b[^>]*\bid\s*=\s*["']i
|
||||
// supplied, and a headless browser is a strong SSRF primitive.
|
||||
var kaganeImageURLRe = regexp.MustCompile(`^https://kagane\.to/api/v2/image/([0-9a-f-]{36})/compressed$`)
|
||||
|
||||
// comixImageURLRe matches comix's cover host and path shape. Pinned in full
|
||||
// (scheme, host, path characters, image extension) for the same reason
|
||||
// kaganeImageURLRe is: the address reaches a headless browser, and it can
|
||||
// originate in a client-supplied PUT body. No dot is allowed inside the path,
|
||||
// so no traversal or second extension can hide in it. Shape from a live page,
|
||||
// 2026-08-10: /039d/i/1/34/6a6742bf15736@280.jpg.
|
||||
var comixImageURLRe = regexp.MustCompile(`^https://static\.comix\.to/[A-Za-z0-9@/_-]+\.(?:jpg|jpeg|png|webp)$`)
|
||||
|
||||
// browserOnlyCoverURL reports whether the browser sidecar is the only fetcher
|
||||
// for cover bytes at imageURL. kagane's image route answers a plain fetch with
|
||||
// a challenge and `cross-origin-resource-policy: same-origin`, so a TLS fetch
|
||||
// would only ever retrieve a challenge page and must not be attempted
|
||||
// (ADR-0007). This is the byte-fetch router's per-Site knowledge; it lives in
|
||||
// the extraction module, which owns kagane's URL shapes.
|
||||
// a challenge and `cross-origin-resource-policy: same-origin`, and
|
||||
// static.comix.to answers one with the same Cloudflare challenge its pages
|
||||
// serve (measured 2026-08-12, issue #98), so a TLS fetch would only ever
|
||||
// retrieve a challenge page and must not be attempted (ADR-0007). This is the
|
||||
// byte-fetch router's per-Site knowledge; it lives in the extraction module,
|
||||
// which owns those URL shapes.
|
||||
func browserOnlyCoverURL(imageURL string) bool {
|
||||
return kaganeImageURLRe.MatchString(imageURL)
|
||||
return kaganeImageURLRe.MatchString(imageURL) ||
|
||||
comixImageURLRe.MatchString(imageURL)
|
||||
}
|
||||
|
||||
// kagane's browser-fetched series response publishes cover image IDs under
|
||||
@@ -389,6 +400,15 @@ var sites = map[string]site{
|
||||
Host: "comix.to",
|
||||
LatestChapter: comixLatestChapter,
|
||||
Cover: comixCoverEntry,
|
||||
Browser: &browserRead{
|
||||
Read: comixRead,
|
||||
// The interstitial is served in place of the page, so "arrived"
|
||||
// has to exclude it explicitly, as novelfull's does.
|
||||
Done: func(body string) bool { return body != "" && !isInterstitial(body) },
|
||||
// Never falls back: a plain fetch of a comix page or cover
|
||||
// retrieves only a challenge page (measured 2026-08-12).
|
||||
Fallback: false,
|
||||
},
|
||||
},
|
||||
"kagane": {
|
||||
Host: "kagane.to",
|
||||
|
||||
@@ -41,6 +41,13 @@ const challengeFixture = `<!DOCTYPE html><html><head><title>Just a moment...</ti
|
||||
// https://comix.to/title/n8we-dungeons-and-crayons fetched 2026-08-03. comix is
|
||||
// an SPA: the page ships a JSON state blob rather than a list of chapter
|
||||
// anchors, and latestChapterUrl is where the newest chapter actually lives.
|
||||
//
|
||||
// Still the right fixture after comix moved behind the challenge (#98): the
|
||||
// browser read is an in-tab fetch of the Series URL, so the body a poll parses
|
||||
// is this same server-rendered HTML, not a rendered DOM. Confirmed against a
|
||||
// live cleared tab 2026-08-16 (TestSmokeComix): the in-tab fetch returned
|
||||
// 24793 bytes of server-rendered HTML that these same parses read a chapter
|
||||
// and a cover out of.
|
||||
const comixSeriesFixture = `
|
||||
{"firstChapterUrl":"/title/n8we-dungeons-and-crayons/5038739-chapter-1","latestChapterUrl":"/title/n8we-dungeons-and-crayons/11139891-chapter-80"},
|
||||
{""manga","recommended","n8we",1]":{"items":[{"latestChapterUrl":"/title/qqwrm-full-time-awakening/99999999-chapter-999"}]}
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package latest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"bookmarkmanager/backend/internal/store"
|
||||
)
|
||||
|
||||
// TestSmokeComix answers "is comix's challenge clearing from this browser right
|
||||
// now" — a live, time-varying fact, so a red run is something to re-check
|
||||
// before it is a defect. Needs the real browser unit with outbound network:
|
||||
//
|
||||
// cd chrome && BROWSER_BIND_ADDR=127.0.0.1 docker compose up -d --build
|
||||
// SMOKE_BROWSER_WS_URL=ws://127.0.0.1:9222 go test -run TestSmokeComix ./internal/latest
|
||||
//
|
||||
// It walks the whole read: the in-tab page fetch, both parses, and the Cover
|
||||
// bytes by direct navigation to static.comix.to. The Cover address comes out of
|
||||
// the page rather than being pinned in the test, because a stored one rots.
|
||||
func TestSmokeComix(t *testing.T) {
|
||||
ws := os.Getenv("SMOKE_BROWSER_WS_URL")
|
||||
if ws == "" {
|
||||
t.Skip("SMOKE_BROWSER_WS_URL unset")
|
||||
}
|
||||
const seriesURL = "https://comix.to/title/m12d-classmate"
|
||||
|
||||
f, err := NewBrowserFetcher(ws)
|
||||
if err != nil {
|
||||
t.Fatalf("NewBrowserFetcher: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
||||
defer cancel()
|
||||
|
||||
body, status, err := f.Get(ctx, seriesURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Get: %v", err)
|
||||
}
|
||||
t.Logf("status=%d bytes=%d", status, len(body))
|
||||
if status != 200 {
|
||||
t.Fatalf("status = %d, want 200 — the sidecar is not clearing the challenge", status)
|
||||
}
|
||||
chapter, ok := latestChapterFrom("comix", seriesURL, body)
|
||||
if !ok {
|
||||
t.Fatalf("no latest chapter in %d bytes — page shape changed", len(body))
|
||||
}
|
||||
t.Logf("latest chapter: %v %q", chapter.Num, chapter.Label)
|
||||
|
||||
cover, ok := coverFrom("comix", seriesURL, body)
|
||||
if !ok {
|
||||
t.Fatalf("no cover address in %d bytes — page shape changed", len(body))
|
||||
}
|
||||
t.Logf("cover: %s", cover)
|
||||
if !browserOnlyCoverURL(cover) {
|
||||
t.Fatalf("cover %q is not claimed by the browser gate: the pin and the live URL shape disagree", cover)
|
||||
}
|
||||
|
||||
bytes, contentType, err := f.Image(ctx, cover)
|
||||
if err != nil {
|
||||
t.Fatalf("Image: %v", err)
|
||||
}
|
||||
if len(bytes) < 1000 {
|
||||
t.Fatalf("cover is %d bytes, want a real image", len(bytes))
|
||||
}
|
||||
t.Logf("fetched %d bytes of %s", len(bytes), contentType)
|
||||
if _, ok := store.CoverContentType(contentType); !ok {
|
||||
t.Fatalf("content type %q is not storable", contentType)
|
||||
}
|
||||
}
|
||||
@@ -19,8 +19,15 @@ and because the Series row now knows how many Readers hold it, the poll queue is
|
||||
absorbs the shortfall. That ordering is only expressible because the split happened.
|
||||
|
||||
Raising throughput instead was rejected: sweeping 400 Series hourly needs the stagger
|
||||
cut from 20s to ~9s, doubling request rate against sites that already bot-score the
|
||||
single VPS IP.
|
||||
cut from 20s to ~9s, doubling request rate against sites already fronted by Cloudflare
|
||||
from the single VPS IP.
|
||||
|
||||
Corrected 2026-08-12: the original wording said those sites "bot-score" the VPS IP.
|
||||
They do not — the 1-99 bot score is Enterprise Bot Management only, and no per-IP
|
||||
request rate is documented as an input to challenge issuance
|
||||
(`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`). The decision stands on
|
||||
its first argument, sweep depth versus the 1-hour cooldown; the rate-limit fear was
|
||||
never evidenced.
|
||||
|
||||
## Only the Poll writes Series fields
|
||||
|
||||
|
||||
@@ -23,8 +23,15 @@ requests: the poller's due query joins bookmarks, production held four kagane
|
||||
series and no bookmarks on any of them, and with no kagane bookmark the web UI
|
||||
never rendered a kagane cover either.
|
||||
|
||||
The home machine has 5.9 GiB of swap and a residential egress, which Cloudflare
|
||||
scores better than a datacenter IP. Both machines were already on the tailnet.
|
||||
The home machine has 5.9 GiB of swap and a residential egress, which avoids the
|
||||
cloud-hosting-IP signature Cloudflare's Bot Fight Mode documentedly challenges. Both
|
||||
machines were already on the tailnet.
|
||||
|
||||
Corrected 2026-08-12: the original wording said Cloudflare "scores" a residential
|
||||
egress better than a datacenter IP. There is no score on a free-plan zone; what is
|
||||
documented is signature matching, and hosting-provider IP space is one of the
|
||||
signatures (`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`). Memory was
|
||||
the load-bearing reason regardless.
|
||||
|
||||
This move is only safe because covers are persisted (ADR-0005's sibling work,
|
||||
issue #43/#45) and the browser is on-demand (ADR-0005). Without stored covers a
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
# Cloudflare bot scoring and poll cadence — what is actually documented
|
||||
|
||||
Research note for the browser-backed poller cadence decision (kagane.to, novelfull.com, comix.to). All pages were fetched live from **developers.cloudflare.com / blog.cloudflare.com on 2026-08-12**. Primary sources only: Cloudflare's own documentation, Cloudflare blog posts, and RFCs/standards where noted. Where Cloudflare does not publicly document something, this note says **`Not publicly documented`** instead of guessing. Repo-measured facts from the existing poller work are reused without re-derivation and marked as such.
|
||||
|
||||
The site configuration of the three challenged sites (which plan, which bot product, Challenge Passage setting, whether Precursor is enabled) is **not observable from outside** — Cloudflare does not expose a zone's security configuration to anonymous clients. Anything in this note that depends on those unknowns is flagged `[INFERENCE]`.
|
||||
|
||||
---
|
||||
|
||||
## Short answer
|
||||
|
||||
**No — polling once per hour per series, from one residential IP through one real Chrome holding a valid `cf_clearance`, carries no documented challenge risk beyond polling every six hours.** Challenge issuance on Free/Pro-grade protection (Bot Fight Mode, WAF rules) is signature- and fingerprint-driven (headless browsers, cloud-hosted IPs, browser signals); the only rate-aware detector — the per-request bot score — exists solely on Enterprise Bot Management, and free-plan Rate Limiting Rules count per-IP over 10-second windows, which 20–60 requests/hour cannot trip. Both cadences re-solve the challenge every visit anyway: `cf_clearance` expires after **30 minutes by default** (site-configurable), so a 1-hour gap always finds it expired. The documented lever that matters — already verified in this repo — is **fingerprint quality**: real Chrome + real timezone clears in ~4 s; headless variants never do. Residual, undocumented risk is site-specific: Challenge Passage, Precursor (behavior-bound re-challenge), and custom WAF rules are zone settings not observable from outside.
|
||||
|
||||
---
|
||||
|
||||
## Summary answer table
|
||||
|
||||
| Question | Answer | Section |
|
||||
|---|---|---|
|
||||
| What is a bot score? | Integer 1–99 = Cloudflare's certainty a request is automated; **Enterprise Bot Management only**; everyone else gets coarse "bot groupings" (Pro+ analytics) or nothing. | §1 |
|
||||
| Is request frequency documented as a bot-score input? | Partially: ML inputs are "headers, session characteristics, and browser signals"; the `__cf_bm` cookie "measures a single user's request pattern". **No numeric rate threshold is documented.** Volume policing is Rate Limiting, a separate product. | §1, §5 |
|
||||
| What can a free-plan site deploy? | Bot Fight Mode only: challenges *signatures* (headless browsers, cloud-hosting IPs) with a computational challenge; JavaScript Detections forced on; no scores, no analytics, cannot be skipped/customized. | §2 |
|
||||
| Does the free tier score continuously? | **No.** No score exists on Free at all — granular scores need Enterprise Bot Management, groupings need Pro+. BFM just challenges signature matches. | §2 |
|
||||
| What does `cf-mitigated: challenge` mean? | The response was a Cloudflare Challenge Page (any type); `challenge` is the only value; body is always `text/html`. | §3 |
|
||||
| How is a successful solve remembered? | `cf_clearance` cookie, issued with `SameSite=None; Secure; Partitioned`; suppresses challenges while valid. | §3, §4 |
|
||||
| `cf_clearance` lifetime? | **30 minutes by default**, configurable by the site via Challenge Passage (15–45 min recommended); +skew minutes; +1 h for XHR. | §4 |
|
||||
| Is `cf_clearance` bound to IP / device? | Documented: "securely tied to the specific visitor and device it was issued to"; the *solve request* must come from the same IP that received the challenge (different IP → invalid solve → challenge loop). Replay from another machine/IP is therefore **not** valid. | §4 |
|
||||
| What invalidates clearance early? | Precursor (if enabled): suspicious session → clearance reduced/invalidated, re-challenge even before expiry. Zone-level toggle; unobservable from outside. | §4 |
|
||||
| Does polling more often raise challenge risk? | **No documented mechanism at 20–60 req/hour.** Free-plan rate limiting is 10 s/IP-only; DDoS thresholds are ~1,000 errors/sec. Scores (the only rate-aware thing) are Enterprise-only. | §5 |
|
||||
| Is there a documented "legitimate poller" path? | Yes, but it requires **self-identification** (Web Bot Auth signature or published IP list + stable UA) via the verified-bots application — not anonymity. robots.txt is voluntary; nothing exempts anonymous scrapers. | §6 |
|
||||
|
||||
---
|
||||
|
||||
## 1. What a bot score is and what feeds it
|
||||
|
||||
### 1.1 The score itself
|
||||
|
||||
Cloudflare documents the bot score as "a score from _1_ to _99_ that indicates how likely that request came from a bot" — 1 = quite certain automated, 99 = quite certain human. Source: [Bot scores — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot-score/), read 2026-08-12.
|
||||
|
||||
Two access tiers, both gated:
|
||||
|
||||
- **Granular 1–99 scores are only available to Enterprise customers who purchased Bot Management.** "All other customers can only access this information through bot groupings in Bot Analytics" (categories: `Not computed` = 0, `Automated` = 1, `Likely automated` = 2–29, `Likely human` = 30–99, `Verified bot`). Bot groupings themselves require "a Pro plan or higher". Source: [Bot scores — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot-score/), read 2026-08-12.
|
||||
- A score of 0 means "Bot Management did not evaluate the request" (redirected, handled by another feature) — "does not indicate the request is safe or human". Same source.
|
||||
|
||||
So: **on a Free-plan site there is no bot score at all, for anyone.** `[INFERENCE]` the three challenged sites are almost certainly not Enterprise Bot Management customers, but this is not externally verifiable.
|
||||
|
||||
### 1.2 The detection engines (Enterprise Bot Management)
|
||||
|
||||
Cloudflare documents four engines, all stated to apply to Enterprise Bot Management (the bot-score page: "The following detection engines only apply to Enterprise Bot Management"). Sources: [Bot scores — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot-score/) and [Bot detection engines — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot-detection-engines/), both read 2026-08-12.
|
||||
|
||||
| Engine | Documented behavior | Score it produces |
|
||||
|---|---|---|
|
||||
| **Heuristics** | "Processes all requests"; pattern matching against "a growing database of malicious fingerprints". | 1 for high-confidence deterministic detections; occasionally 29 "where Cloudflare has identified automated traffic and is still assessing traffic overlap" |
|
||||
| **Machine learning** | Supervised model, trained on "billions of daily requests". Input variables: "headers, session characteristics, and browser signals". Output: "predicted probability that a client is human (such as the probability of successfully solving a Challenge)". | Most scores 2–99 |
|
||||
| **Anomaly detection** | Unsupervised; learns a per-domain baseline, flags outlier requests; **deprecated, not onboarding new customers**. | 1 |
|
||||
| **JavaScript detections** | "Identifies headless browsers and other automation tools" via "a lightweight, invisible JavaScript injection"; runs client-side; "blocks, challenges, or passes requests to other engines". Enabled by default (but optional) in Bot Management. | Pass/fail (`cf.bot_management.js_detection.passed`), not a score |
|
||||
|
||||
Crucially, the ML engine's documented inputs are *headers, session characteristics, and browser signals* — **no rate or per-IP volume parameter is listed.** The only place request patterns appear is the `__cf_bm` cookie note: "Cloudflare uses the `__cf_bm` cookie to smooth out the bot score and reduce false positives… The Bot Management cookie measures a single user's request pattern and applies it to the machine learning data to generate a reliable bot score for all of that user's requests." ([Bot scores — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot-score/), read 2026-08-12). So frequency is *a* signal inside Enterprise Bot Management via the session cookie — **but no numeric threshold, window, or per-IP rate is published anywhere.** `Not publicly documented`: any specific requests-per-hour / requests-per-IP value that raises or lowers a bot score.
|
||||
|
||||
### 1.3 Rate limiting is a separate product
|
||||
|
||||
Volume enforcement is not part of bot scoring at all. Rate Limiting Rules are a distinct WAF product with their own evaluation phase (`http_ratelimit`, running after custom rules and before SBFM). Sources: [Rate limiting rules — Cloudflare docs](https://developers.cloudflare.com/waf/rate-limiting-rules/) and [Security features interoperability — Cloudflare docs](https://developers.cloudflare.com/waf/feature-interoperability/), read 2026-08-12. See §5 for what the Free plan's version of that product can actually do.
|
||||
|
||||
---
|
||||
|
||||
## 2. The free-plan reality
|
||||
|
||||
### 2.1 What each plan gets
|
||||
|
||||
Cloudflare's plan table ([Plans — Cloudflare docs](https://developers.cloudflare.com/bots/plans/), read 2026-08-12):
|
||||
|
||||
| Plan | Bot product | Documented detections | Action | Control |
|
||||
|---|---|---|---|---|
|
||||
| **Free** | **Bot Fight Mode** (BFM) | "Simple bots from cloud hosting providers and headless browsers" | "Cloudflare issues a computationally expensive challenge" | Applied to all traffic across the domain; no exceptions possible |
|
||||
| **Pro / Business / Enterprise (no BM)** | **Super Bot Fight Mode** (SBFM) | Configurable actions per bot category (Definitely automated / Likely automated / Verified bots) | Challenge or block | Runs on Ruleset Engine; **can** be skipped via custom rules |
|
||||
| **Enterprise + Bot Management** | Bot Management | "Simple and sophisticated bots, headless browsers, and domain-specific anomalies" | Customer-chosen (block, challenges) | Per-path / per-IP rules; access to bot score, JA3/JA4, bot tags, detection IDs |
|
||||
|
||||
Sources: [Plans — Free](https://developers.cloudflare.com/bots/plans/free/), [Plans — Bot Management for Enterprise](https://developers.cloudflare.com/bots/plans/bm-subscription/), [Bot Fight Mode — Cloudflare docs](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/), [Super Bot Fight Mode — Cloudflare docs](https://developers.cloudflare.com/bots/get-started/super-bot-fight-mode/) — all read 2026-08-12.
|
||||
|
||||
### 2.2 Bot Fight Mode specifics (the Free-plan product)
|
||||
|
||||
- Identifies "traffic matching patterns of known bots" and "issues computationally expensive challenges that force the requesting client to perform CPU-intensive calculations". ([Bot Fight Mode — Cloudflare docs](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/), read 2026-08-12.)
|
||||
- It "does not run on the Ruleset Engine — it operates in a separate evaluation pipeline where _Skip_, _Bypass_, and _Allow_ actions have no effect"; **you cannot bypass or skip BFM** with custom rules or Page Rules. ([Bot Fight Mode — Cloudflare docs](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/) and [Security features interoperability — Cloudflare docs](https://developers.cloudflare.com/waf/feature-interoperability/), read 2026-08-12.)
|
||||
- **JavaScript Detections is automatically enabled for BFM customers and cannot be disabled.** ([Bot Fight Mode — Cloudflare docs](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/), read 2026-08-12.) This is the documented hook that explains the repo's `headless-shell` / `HeadlessChrome` failures: JSD "identifies headless browsers" ([JavaScript detections — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/javascript-detections/), read 2026-08-12).
|
||||
- False positives on *legitimate automated* traffic are acknowledged as expected behavior: "false positives can occur where legitimate human or automated traffic is incorrectly challenged or blocked", and the only remedies are disabling BFM or upgrading to Bot Management. ([Handle False Positives — Cloudflare docs](https://developers.cloudflare.com/bots/troubleshooting/false-positives/), read 2026-08-12.)
|
||||
|
||||
### 2.3 Does the free tier "score" continuously?
|
||||
|
||||
**No.** The Free plan exposes no score, no bot analytics (groupings need Pro+), and no per-request decision data. BFM is a static on/off toggle that challenges signature matches; there is no continuous per-request score on Free. ([Plans — Free](https://developers.cloudflare.com/bots/plans/free/) and [Bot scores — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot-score/), read 2026-08-12.) JSD *runs* on every HTML request even on Free (forced by BFM), but the only documented way to act on its result — the `cf.bot_management.js_detection.passed` field — is gated behind an Enterprise Bot Management subscription ("Prerequisites: You must have an Enterprise Bot Management subscription"). ([JavaScript detections — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/javascript-detections/), read 2026-08-12.) On Free, JSD output feeds BFM's internal challenge decision; `Not publicly documented` exactly how.
|
||||
|
||||
`[INFERENCE]` The observed behavior on the three sites (real Chrome + real timezone passes in ~4 s; headless variants never do) is consistent with either BFM or a WAF custom rule using a challenge action — Cloudflare does not expose which product a site runs, and the failure signature (HeadlessChrome UA / headless-shell never passing) matches JSD's documented headless-browser detection either way.
|
||||
|
||||
---
|
||||
|
||||
## 3. Managed Challenge / JS challenge mechanics and `cf-mitigated`
|
||||
|
||||
### 3.1 What the observed response is
|
||||
|
||||
A `403` with `cf-mitigated: challenge`, `server: cloudflare`, and a "Just a moment…" body is a **Cloudflare Challenge Page**. Cloudflare documents: "the Challenge Page response (regardless of the Challenge Page type) will have the `cf-mitigated` header present and set to `challenge`… `challenge` is the only valid value. The header is set for all Challenge Page types", and "the content-type of a challenge will be `text/html`". ([Detect a Challenge Page response — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/), read 2026-08-12.)
|
||||
|
||||
### 3.2 What a Challenge Page does
|
||||
|
||||
"An interstitial Challenge Page… acts as a gate between the visitor and your website… The Challenge Page intercepts the visitor… by holding the request and evaluating the browser environment for automated signals, and serving a challenge. The visitor cannot reach their destination without passing the challenge." ([Interstitial Challenge Pages — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/), read 2026-08-12.)
|
||||
|
||||
Three variants, in increasing severity ([Interstitial Challenge Pages — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/), read 2026-08-12):
|
||||
|
||||
- **Non-Interactive**: Cloudflare judges automation from browser signals gathered by injected JS; the page needs no human interaction, typically < 5 s of JS processing.
|
||||
- **Managed Challenge**: "Cloudflare dynamically chooses the appropriate type of challenge… based on the characteristics of a request from the signals indicated by their browser. Most human visitors are automatically verified and the Challenge Page will display **Successful**. However, if Cloudflare detects non-human attributes… they may be required to interact." Cloudflare's stated recommendation for WAF rules.
|
||||
- **Interactive**: requires explicit human interaction (CAPTCHA-style). Cloudflare's "End the CAPTCHA era" position is that Managed Challenges should make this rare.
|
||||
|
||||
Cloudflare's own framing matches the repo's ~4 s real-Chrome solve: a normal browser passes with no interaction (Managed Challenge auto-verify or Non-Interactive JS processing).
|
||||
|
||||
### 3.3 Which product issues which challenge
|
||||
|
||||
Documented mapping ([How Challenges work — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/concepts/how-challenges-work/), read 2026-08-12):
|
||||
|
||||
| Trigger | Challenge type |
|
||||
|---|---|
|
||||
| WAF custom rules, **rate limiting rules**, IP Access rules | Interstitial Challenge Page |
|
||||
| Bot Management | JavaScript Detections (invisible, per-request) |
|
||||
| **Bot Fight Mode / Super Bot Fight Mode** | Interstitial Challenge Page |
|
||||
| Under Attack Mode | Managed Challenge |
|
||||
|
||||
### 3.4 How a successful solve is remembered
|
||||
|
||||
Solving issues the **`cf_clearance`** cookie: "Clearance Cookie stores the proof of challenge passed. It is used to no longer issue a challenge if present. It is required to reach an origin server." ([Cloudflare Cookies — Cloudflare docs](https://developers.cloudflare.com/fundamentals/reference/policies-compliances/cloudflare-cookies/), read 2026-08-12.) "When that visitor tries to access other parts of your website, Cloudflare evaluates the cookie before presenting another challenge. If the cookie is still valid, no challenges will be shown." ([Challenge Passage — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/challenge-passage/), read 2026-08-12.)
|
||||
|
||||
`cf_clearance` is set with `SameSite=None; Secure; Partitioned`; because of the `Partitioned` (CHIPS) attribute, "a clearance obtained in one top-level context is not reused in a different top-level context" — so a clearance from kagane.to does not carry to comix.to even on the same browser. ([SameSite cookie interaction — Cloudflare docs](https://developers.cloudflare.com/waf/troubleshooting/samesite-cookie-interaction/), read 2026-08-12.)
|
||||
|
||||
---
|
||||
|
||||
## 4. `cf_clearance` — lifetime, binding, invalidation (the key question)
|
||||
|
||||
### 4.1 Lifetime
|
||||
|
||||
- **Default: 30 minutes.** "By default, the `cf_clearance` cookie has a lifetime of 30 minutes. Cloudflare recommends a setting between 15 and 45 minutes." The site owner can change it via the **Challenge Passage** setting. ([Challenge Passage — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/challenge-passage/), read 2026-08-12; also [SameSite cookie interaction — Cloudflare docs](https://developers.cloudflare.com/waf/troubleshooting/samesite-cookie-interaction/), read 2026-08-12.)
|
||||
- Validation grace: "a few extra minutes are included to account for clock skew. For XmlHTTP requests, an extra hour is added to the validation time." ([Challenge Passage — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/challenge-passage/), read 2026-08-12.)
|
||||
- "The Challenge Passage does not apply to rate limiting rules." Same source.
|
||||
- `Not publicly documented`: whether the three target sites have changed the Challenge Passage from the 30-minute default, and whether there is any maximum value Cloudflare enforces.
|
||||
|
||||
**Consequence for cadence:** with the default 30-minute TTL, *any* poll cadence ≥ 1 hour finds the cookie expired and re-solves the challenge on every visit. A 1-hour and a 6-hour cadence therefore differ only in *how many times per day* the browser re-solves (~4× for the same series), not in whether a re-solve happens. This repo already measured the re-solve cost: ~4 s with real Chrome + real timezone. `[INFERENCE]` a site could raise the Challenge Passage to hours/days, which would make a 1-hour cadence *cheaper* (cookie still valid, no re-solve) — but that setting is unobservable and unlikely to be long on free manga sites.
|
||||
|
||||
### 4.2 What it is bound to — can clearance be replayed from another IP?
|
||||
|
||||
Documented statements, both from Cloudflare's own docs:
|
||||
|
||||
1. **Device/visitor binding:** "The cookie is securely tied to the specific visitor and device it was issued to, preventing reuse across machines." ([Clearance — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/concepts/clearance/), read 2026-08-12.)
|
||||
2. **IP binding of the solve:** under Challenge limitations, Cloudflare lists "Client software where the solve request of a Managed Challenge comes from a different IP than the original IP a Challenge request was issued to. For example, if you receive the Challenge from one IP and solve it using another IP, the solve is not valid and you may encounter a Challenge loop." ([How Challenges work — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/concepts/how-challenges-work/), read 2026-08-12.)
|
||||
3. **Top-level-site binding** via CHIPS partitioning: clearance is not reused across embedding contexts. ([SameSite cookie interaction — Cloudflare docs](https://developers.cloudflare.com/waf/troubleshooting/samesite-cookie-interaction/), read 2026-08-12.)
|
||||
|
||||
So the repo's belief is **verified by primary sources**: a `cf_clearance` obtained on one machine cannot be replayed from a different IP — the solve is IP-bound and the cookie is device-bound. `Not publicly documented`: whether the cookie value is also cryptographically bound to the User-Agent or TLS/JA3 fingerprint. The only UA-adjacent documented statement is the reverse direction: challenge *solving* breaks when a browser extension modifies the User-Agent or Canvas/WebGL APIs ("Cloudflare Challenges cannot support… Browser extensions that modify the browser's User-Agent value or Web APIs such as Canvas and WebGL") — i.e., tampering with browser signals is documented to *fail* challenges ([How Challenges work — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/concepts/how-challenges-work/), read 2026-08-12).
|
||||
|
||||
### 4.3 Two-tier clearance, and the behavior-bound invalidation (Precursor)
|
||||
|
||||
`cf_clearance` now carries two kinds of clearance ([Clearance — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/concepts/clearance/), read 2026-08-12):
|
||||
|
||||
- **Challenge clearance** — granted by solving a challenge; level-gated (Interactive > Managed > Non-Interactive; higher clears bypass lower challenges); "remains valid for the duration configured by the customer (Challenge Passage), **unless Precursor determines the session is suspicious**".
|
||||
- **Precursor clearance** — "continuously updated based on session behavior"; an ongoing client-side process that periodically reassesses behavior. "If Precursor determines that a session is suspicious: the visitor's effective Challenge clearance may be **reduced or invalidated**; the visitor may be **re-challenged, even if the cookie has not expired**."
|
||||
|
||||
Precursor is documented as "client-side, session-based verification that continuously evaluates visitor behavior to identify automation… to detect automation that appears legitimate in individual requests but exhibits non-human patterns across a session", writing session state back into `cf_clearance`. It is a **zone-level toggle** (Security → Settings → Precursor; modes: Minimize Friction default, Maximize Security recommended), and "Precursor supersedes JavaScript Detections (JSD)". ([Precursor — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/precursor/), read 2026-08-12.)
|
||||
|
||||
**Implication:** the only documented mechanism by which *behavior over time* (as opposed to a single request's fingerprint) can revoke a valid clearance is Precursor — and it is opt-in per zone. `[INFERENCE]` It is unlikely to be enabled on free manga sites, but this is not observable from outside. With Precursor off (the default posture `[INFERENCE]`), a valid `cf_clearance` is honored purely on TTL + device/IP binding.
|
||||
|
||||
---
|
||||
|
||||
## 5. Does polling more often raise challenge risk?
|
||||
|
||||
### 5.1 Is per-IP request rate an input to challenge issuance? (documented answer: no such lever on non-Enterprise protection)
|
||||
|
||||
- **Bot scores** (the only per-request automated-detection output) are Enterprise-Bot-Management-only (§1.1); the ML engine's documented inputs are headers/session/browser signals, with request *pattern* entering only via `__cf_bm` — and no numeric rate is published (§1.2). `Not publicly documented`: any requests-per-hour value that changes a bot score or challenge probability.
|
||||
- **BFM/SBFM** match "patterns of known bots" — signatures, not volumes ([Bot Fight Mode — Cloudflare docs](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/), read 2026-08-12).
|
||||
- **Rate limiting** is the product that polices volume, and it is opt-in per zone with explicit per-plan constraints (§5.2). Cloudflare's only documented link between "one valid clearance + high volume" is a *recommendation to site owners*: "Cloudflare recommends that customers add a rate limiting rule based on the `cf_clearance` cookie value. This helps ensure that a single, valid cookie cannot be abused by one machine to send an excessive volume of requests." ([Clearance — Cloudflare docs](https://developers.cloudflare.com/cloudflare-challenges/concepts/clearance/), read 2026-08-12.) Note: counting by cookie value is only available on Enterprise (see table below); a Free/Pro site cannot even build that rule.
|
||||
|
||||
### 5.2 What Rate Limiting Rules would do to 20–60 requests/hour
|
||||
|
||||
Rate limiting rules are opt-in; nothing runs them unless the site creates a rule. The documented per-plan capabilities ([Rate limiting rules — Cloudflare docs](https://developers.cloudflare.com/waf/rate-limiting-rules/), read 2026-08-12):
|
||||
|
||||
| Capability | Free | Pro | Business |
|
||||
|---|---|---|---|
|
||||
| Number of rules | **1** | 2 | 5 |
|
||||
| Counting characteristics | **IP only** | IP only | IP, IP w/ NAT |
|
||||
| Counting period | **10 s only** | ≤ 1 min | ≤ 10 min |
|
||||
| Mitigation timeout | **10 s** | ≤ 1 h | ≤ 1 day |
|
||||
| Fields in expression | **Path, Verified Bot** | + Host, URI, Full URI, Query | + Method, Source IP, User Agent |
|
||||
|
||||
Even the most aggressive free-plan rule (1 request per 10 s = 360/hour) is 6–18× above our 20–60/hour volume; the counting window is 10 s, so a per-hour burst is invisible to it. At 1 request per minute worst-case, **20–60 requests/hour from one residential IP cannot trip any rate limiting rule the Free plan can express.** Also documented: rate limiting is approximate, not precise — "there may be a delay of up to a few seconds between detecting a request and updating rate counters… excess requests could still reach the origin", and counters are per-data-center. ([Rate limiting rules — Cloudflare docs](https://developers.cloudflare.com/waf/rate-limiting-rules/), read 2026-08-12.) `[INFERENCE]` a site could also challenge on rate via WAF custom rules, but that requires Pro+ (custom rules are not on Free) and its own configuration.
|
||||
|
||||
### 5.3 DDoS protection (always on, all plans)
|
||||
|
||||
HTTP DDoS Attack Protection "is always enabled" and can only be tuned, not disabled. The only *published numeric* thresholds are error-rate-based: origin-error floods mitigate at the default "High" sensitivity of **1,000 errors per second** (Pro+ also requires 5× normal origin traffic). Per-IP volumetric thresholds are adaptive and `Not publicly documented` in the managed ruleset docs. 20–60 requests/hour is ~9 orders of magnitude below the published figure. ([HTTP DDoS Attack Protection — Cloudflare docs](https://developers.cloudflare.com/ddos-protection/managed-rulesets/http/), read 2026-08-12.)
|
||||
|
||||
### 5.4 Execution order (which product fires first)
|
||||
|
||||
Documented phase order: `ddos_l7` → custom rules → `http_ratelimit` (rate limiting) → managed rules → `http_request_sbfm` (SBFM); BFM runs outside this pipeline and cannot be skipped; a terminating action (block/challenge) stops later phases. ([Security features interoperability — Cloudflare docs](https://developers.cloudflare.com/waf/feature-interoperability/), read 2026-08-12.) Practical reading: on the three sites, the challenge we see could come from any of these stages; none of them documents a volume input at our scale (§5.1–5.3).
|
||||
|
||||
---
|
||||
|
||||
## 6. The documented legitimate side
|
||||
|
||||
### 6.1 Verified bots — the only "treated well" path, and it requires self-identification
|
||||
|
||||
Cloudflare documents a Verified bot as one meeting two bars ([Verified bots — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-08-12):
|
||||
|
||||
1. **Honest self-identification** — "through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS".
|
||||
2. **Non-abusive behavior** — "it obeys `robots.txt` and crawl directives, **maintains reasonable request rates**, and has not been observed evading website owner preferences or attacking sites".
|
||||
|
||||
Relevant verified-bot *behavior classes* exist for exactly this kind of client: "**Feed Fetching** — RSS readers, podcast aggregators, and news feed bots" and "**Monitoring & Operations** — Uptime monitoring, webhooks, and health checks". Becoming verified requires an application via the dashboard and validation via Web Bot Auth or IP validation; breach of the policy (e.g. "An AI Crawler that does not respect the crawl-delay directive") removes the bot from the allowlist. ([Verified bots — Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), read 2026-08-12.)
|
||||
|
||||
"Historically, Verified bots have been excluded in default bot configurations across all plans" (same source) — i.e., verified bots are *default-allowed* under SBFM/Bot Management. **But** this path is the opposite of what a scraper wants: it requires the poller to publicly identify itself (stable, published IPs or cryptographic signatures) and to have its identity vetted by Cloudflare — and the *site* still decides via verified-bot policy whether to allow the category. There is **no documented mechanism for an anonymous low-volume automated client to be treated well.** `[INFERENCE]` a manga-site scraper would never qualify (it would be classified as Data Collection / scraping behavior, which is not a default-allowed class).
|
||||
|
||||
### 6.2 robots.txt and crawl control
|
||||
|
||||
- `robots.txt` **compliance is voluntary** — "The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level." Enforcement requires Cloudflare's AI Crawl Control. ([robots.txt setting — Cloudflare docs](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/), read 2026-08-12.)
|
||||
- The managed `robots.txt` feature (all plans) is aimed at AI crawlers; it prepends `Disallow` rules for AI bots and a Content Signals Policy. It does not create any allowance for generic scrapers. (Same source.)
|
||||
- RFC-side: the robots exclusion standard is an unauthenticated convention; nothing in it grants access rights. ([RFC 9309 "Robots Exclusion Protocol"](https://www.rfc-editor.org/rfc/rfc9309.html) — read 2026-08-12.) The standard defines crawl-delay etc. as voluntary directives; Cloudflare's docs are the operative statement for CF-protected sites.
|
||||
|
||||
### 6.3 The documented takeaway for "slowing down vs. fingerprint quality"
|
||||
|
||||
Cloudflare's own documentation repeatedly points at **browser/device signals** as the decision input on non-Enterprise protection (JSD detecting headless browsers; Managed Challenge choosing based on "signals indicated by their browser"; challenges failing when UA/Canvas/WebGL are modified — §3.2, §4.2), and at **identity** (verified bots) as the only legitimacy signal for automation (§6.1). Request *rate* appears only as: (a) an unnamed component of Enterprise-ML "session characteristics", (b) a voluntary verified-bot behavioral bar, and (c) the separate, opt-in, Free-plan-impotent Rate Limiting product (§5). Nothing documented says "slow down and you'll be challenged less" for a free-plan site — **fingerprint quality is the lever that the documentation actually describes**, which matches this repo's measurements (real Chrome + real timezone passes; every headless variant fails regardless of rate).
|
||||
|
||||
---
|
||||
|
||||
## 7. Implications for cadence design
|
||||
|
||||
### Documented facts (with sources above)
|
||||
|
||||
1. **1-hour vs 6-hour cadence is not a documented risk lever.** Challenge issuance on Free/Pro-grade protection is signature-based; the rate-aware scoring only exists on Enterprise Bot Management; free-plan rate limiting cannot express a limit our volume could trip (§1, §2, §5).
|
||||
2. **Every poll ≥ 1 hour re-solves the challenge anyway.** `cf_clearance` defaults to 30 minutes; Challenge Passage is site-configurable and unobservable. The re-solve cost is what this repo measured (~4 s, real Chrome + real timezone) (§4.1, repo measurements).
|
||||
3. **The documented failure modes are fingerprint, not rate:** headless browsers (JSD), cloud-hosting IPs (BFM heuristics), modified UA/Canvas/WebGL (challenge solve failure) (§2.2, §3, §4.2).
|
||||
4. **Clearance is not portable:** device-bound + solve-IP-bound + CHIPS-partitioned; replaying a cookie from another IP is documented invalid (§4.2).
|
||||
5. **Anonymity has no documented "good citizen" path:** the only legitimate-automation route (verified bots) requires self-identification and site-side allowance (§6).
|
||||
6. **The one behavior-bound revocation mechanism (Precursor) is opt-in per zone**, not a default documented behavior (§4.3).
|
||||
|
||||
### Inferences (not documented)
|
||||
|
||||
- `[INFERENCE]` The three sites run Free/Pro-grade protection (BFM, SBFM, or WAF challenge rules), not Enterprise Bot Management; therefore no continuous per-request bot score exists for our traffic.
|
||||
- `[INFERENCE]` The sites have not changed Challenge Passage to hours/days (free manga sites default to the 30-minute default); if they had, hourly polling would get *cheaper* (valid cookie, no re-solve).
|
||||
- `[INFERENCE]` Precursor is not enabled on these sites; if it were, hourly re-visits from an automated Chrome could accumulate session-behavior signals and trigger re-challenge even with a valid cookie — the only documented scenario in which polling *frequency* (via session behavior) could matter.
|
||||
- `[INFERENCE]` 6-hour cooldowns buy nothing documented beyond raw request-count reduction (fewer challenge solves per day, less origin load); the risk profile at 1 request/hour/series is not documented to differ from 6 request/hour/series.
|
||||
- `[INFERENCE]` If the owner wants belt-and-braces, the engineering levers that match the documentation are: keep the real-Chrome fingerprint (no UA spoofing, no headless-shell, real timezone — already done), keep a persistent user-data profile so `cf_clearance`/`__cf_bm` persist across visits, and treat any change of exit IP (e.g. home connection rebooting to a new IP) as a guaranteed re-solve, since clearance does not travel with the IP.
|
||||
|
||||
### Bottom line
|
||||
|
||||
Moving browser-backed sites from 6-hour to 1-hour per-series cooldown is **not contradicted by any documented Cloudflare mechanism** at 20–60 requests/hour from one residential IP through one real Chrome. The documented risk is carried by fingerprint quality (already solved in this repo) and by unobservable site configuration (Challenge Passage, Precursor, possible custom WAF rules). The residual, non-documented risk is that these sites sit behind Cloudflare's *proprietary* detection, and Cloudflare publishes neither its per-IP thresholds nor the ML feature set — so "no documented lever" is not "no lever".
|
||||
@@ -15,19 +15,21 @@
|
||||
"13": "Web UI Handlers",
|
||||
"14": "Go Error Handling",
|
||||
"15": "CDP Browser Client",
|
||||
"16": "Cloudflare bot scoring and poll cadence — what is actually documented",
|
||||
"17": "Go Code Style Guide",
|
||||
"18": "Agent Skills",
|
||||
"19": "Store",
|
||||
"20": "I/O Performance Patterns",
|
||||
"21": "CPU Optimization",
|
||||
"22": "Caching Patterns",
|
||||
"23": "Browser Entrypoint",
|
||||
"24": "Memory Allocation & GC",
|
||||
"25": "Cover Fetcher Tests",
|
||||
"26": "readSeries",
|
||||
"26": "Open",
|
||||
"27": "Find Skills Guide",
|
||||
"28": "Allocation Patterns",
|
||||
"29": "Observability & Alerting",
|
||||
"31": "Memory Layout",
|
||||
"30": "AGENTS.md",
|
||||
"32": "Repo Hard Constraints",
|
||||
"33": "Go Testing Guide",
|
||||
"34": "Session Store",
|
||||
@@ -37,6 +39,7 @@
|
||||
"38": "novel-logic.test.js",
|
||||
"39": "UI Critique 2026-07-26A",
|
||||
"40": "UI Critique 2026-07-26B",
|
||||
"41": "sessions_test.go",
|
||||
"43": "Issue Tracker & Triage",
|
||||
"44": "Ticket Workflow",
|
||||
"45": "Go Perf Alert Rules",
|
||||
@@ -146,22 +149,11 @@
|
||||
"150": "Reviewer Subagent (opencode)",
|
||||
"151": "Finding Severity Rubric",
|
||||
"152": "Spec Compliance Review",
|
||||
"158": "T",
|
||||
"161": "Why Use samber/oops",
|
||||
"162": "singleflight Cache Stampede Prevention",
|
||||
"163": "Struct Field Alignment",
|
||||
"164": "testing/synctest Deterministic Goroutine Testing",
|
||||
"165": "API Package (Bookmark JSON Handlers)",
|
||||
"166": "Cover Acquisition & Serving Pipeline",
|
||||
"167": "Backend AGENTS.md Guidance",
|
||||
"168": "HTTP Middleware (Auth/Gzip/CORS)",
|
||||
"169": "Latest Package (Site Parsers & Poller)",
|
||||
"170": "Latest-Chapter Poller",
|
||||
"171": "Main Composition Root",
|
||||
"172": "Migration-Owned Schema",
|
||||
"173": "Session Package (Cookie Signing & Rate Limit)",
|
||||
"174": "updated_at List-Order Rule",
|
||||
"175": "Userscript Package (Serving Handler)",
|
||||
"176": "AGENTS.md",
|
||||
"177": "Backend CLAUDE.md Guidance",
|
||||
"178": "Graphify Knowledge Graph (graphify-out/)",
|
||||
"179": "CLAUDE.md (Symlink to AGENTS.md)",
|
||||
@@ -226,10 +218,6 @@
|
||||
"259": "Dark-First Design Constraint",
|
||||
"260": "Discord Guild Membership",
|
||||
"261": "Reader Isolation Invariant",
|
||||
"268": "Browser Unit Redeploy",
|
||||
"269": "pg_dump Hot Backup",
|
||||
"270": "Redeploy Runbook",
|
||||
"271": "Rollback Strategy",
|
||||
"273": "AGENTS.md",
|
||||
"279": "Userscript CLAUDE guidance"
|
||||
}
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
# Graph Report - mangaBookmark (2026-08-12)
|
||||
# Graph Report - mangaBookmark (2026-08-16)
|
||||
|
||||
## Corpus Check
|
||||
- 110 files · ~267,791 words
|
||||
- 112 files · ~274,851 words
|
||||
- Verdict: corpus is large enough that graph structure adds value.
|
||||
|
||||
## Summary
|
||||
- 1621 nodes · 3242 edges · 233 communities (63 shown, 170 thin omitted)
|
||||
- Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 312 edges (avg confidence: 0.77)
|
||||
- 1640 nodes · 3294 edges · 221 communities (67 shown, 154 thin omitted)
|
||||
- Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 313 edges (avg confidence: 0.77)
|
||||
- Token cost: 0 input · 0 output
|
||||
|
||||
## Graph Freshness
|
||||
- Built from commit: `8ae98816`
|
||||
- Built from commit: `f000cc7e`
|
||||
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
|
||||
- Run `graphify update .` after code changes (no API cost).
|
||||
|
||||
@@ -31,19 +31,20 @@
|
||||
- [[_COMMUNITY_Web UI Handlers|Web UI Handlers]]
|
||||
- [[_COMMUNITY_Go Error Handling|Go Error Handling]]
|
||||
- [[_COMMUNITY_CDP Browser Client|CDP Browser Client]]
|
||||
- [[_COMMUNITY_Cloudflare bot scoring and poll cadence — what is actually documented|Cloudflare bot scoring and poll cadence — what is actually documented]]
|
||||
- [[_COMMUNITY_Go Code Style Guide|Go Code Style Guide]]
|
||||
- [[_COMMUNITY_Agent Skills|Agent Skills]]
|
||||
- [[_COMMUNITY_Store|Store]]
|
||||
- [[_COMMUNITY_IO Performance Patterns|I/O Performance Patterns]]
|
||||
- [[_COMMUNITY_CPU Optimization|CPU Optimization]]
|
||||
- [[_COMMUNITY_Caching Patterns|Caching Patterns]]
|
||||
- [[_COMMUNITY_Browser Entrypoint|Browser Entrypoint]]
|
||||
- [[_COMMUNITY_Memory Allocation & GC|Memory Allocation & GC]]
|
||||
- [[_COMMUNITY_Cover Fetcher Tests|Cover Fetcher Tests]]
|
||||
- [[_COMMUNITY_readSeries|readSeries]]
|
||||
- [[_COMMUNITY_Open|Open]]
|
||||
- [[_COMMUNITY_Find Skills Guide|Find Skills Guide]]
|
||||
- [[_COMMUNITY_Allocation Patterns|Allocation Patterns]]
|
||||
- [[_COMMUNITY_Observability & Alerting|Observability & Alerting]]
|
||||
- [[_COMMUNITY_Memory Layout|Memory Layout]]
|
||||
- [[_COMMUNITY_Repo Hard Constraints|Repo Hard Constraints]]
|
||||
- [[_COMMUNITY_Go Testing Guide|Go Testing Guide]]
|
||||
- [[_COMMUNITY_Session Store|Session Store]]
|
||||
@@ -53,6 +54,7 @@
|
||||
- [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]]
|
||||
- [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]]
|
||||
- [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]]
|
||||
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
|
||||
- [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]]
|
||||
- [[_COMMUNITY_Ticket Workflow|Ticket Workflow]]
|
||||
- [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]]
|
||||
@@ -162,21 +164,11 @@
|
||||
- [[_COMMUNITY_Reviewer Subagent (opencode)|Reviewer Subagent (opencode)]]
|
||||
- [[_COMMUNITY_Finding Severity Rubric|Finding Severity Rubric]]
|
||||
- [[_COMMUNITY_Spec Compliance Review|Spec Compliance Review]]
|
||||
- [[_COMMUNITY_T|T]]
|
||||
- [[_COMMUNITY_Why Use samberoops|Why Use samber/oops]]
|
||||
- [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]]
|
||||
- [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]]
|
||||
- [[_COMMUNITY_testingsynctest Deterministic Goroutine Testing|testing/synctest Deterministic Goroutine Testing]]
|
||||
- [[_COMMUNITY_API Package (Bookmark JSON Handlers)|API Package (Bookmark JSON Handlers)]]
|
||||
- [[_COMMUNITY_Cover Acquisition & Serving Pipeline|Cover Acquisition & Serving Pipeline]]
|
||||
- [[_COMMUNITY_Backend AGENTS.md Guidance|Backend AGENTS.md Guidance]]
|
||||
- [[_COMMUNITY_HTTP Middleware (AuthGzipCORS)|HTTP Middleware (Auth/Gzip/CORS)]]
|
||||
- [[_COMMUNITY_Latest Package (Site Parsers & Poller)|Latest Package (Site Parsers & Poller)]]
|
||||
- [[_COMMUNITY_Latest-Chapter Poller|Latest-Chapter Poller]]
|
||||
- [[_COMMUNITY_Main Composition Root|Main Composition Root]]
|
||||
- [[_COMMUNITY_Migration-Owned Schema|Migration-Owned Schema]]
|
||||
- [[_COMMUNITY_Session Package (Cookie Signing & Rate Limit)|Session Package (Cookie Signing & Rate Limit)]]
|
||||
- [[_COMMUNITY_updated_at List-Order Rule|updated_at List-Order Rule]]
|
||||
- [[_COMMUNITY_Userscript Package (Serving Handler)|Userscript Package (Serving Handler)]]
|
||||
- [[_COMMUNITY_Backend CLAUDE.md Guidance|Backend CLAUDE.md Guidance]]
|
||||
- [[_COMMUNITY_Graphify Knowledge Graph (graphify-out)|Graphify Knowledge Graph (graphify-out/)]]
|
||||
- [[_COMMUNITY_CLAUDE.md (Symlink to AGENTS.md)|CLAUDE.md (Symlink to AGENTS.md)]]
|
||||
@@ -241,18 +233,14 @@
|
||||
- [[_COMMUNITY_Dark-First Design Constraint|Dark-First Design Constraint]]
|
||||
- [[_COMMUNITY_Discord Guild Membership|Discord Guild Membership]]
|
||||
- [[_COMMUNITY_Reader Isolation Invariant|Reader Isolation Invariant]]
|
||||
- [[_COMMUNITY_Browser Unit Redeploy|Browser Unit Redeploy]]
|
||||
- [[_COMMUNITY_pg_dump Hot Backup|pg_dump Hot Backup]]
|
||||
- [[_COMMUNITY_Redeploy Runbook|Redeploy Runbook]]
|
||||
- [[_COMMUNITY_Rollback Strategy|Rollback Strategy]]
|
||||
- [[_COMMUNITY_AGENTS|AGENTS.md]]
|
||||
- [[_COMMUNITY_Userscript CLAUDE guidance|Userscript CLAUDE guidance]]
|
||||
|
||||
## God Nodes (most connected - your core abstractions)
|
||||
1. `testConfig()` - 53 edges
|
||||
2. `newWebTestServer()` - 49 edges
|
||||
3. `newTestStore()` - 42 edges
|
||||
4. `newTestStore()` - 41 edges
|
||||
3. `newTestStore()` - 43 edges
|
||||
4. `newTestStore()` - 42 edges
|
||||
5. `e()` - 33 edges
|
||||
6. `Handler` - 29 edges
|
||||
7. `ne()` - 28 edges
|
||||
@@ -261,8 +249,6 @@
|
||||
10. `se()` - 27 edges
|
||||
|
||||
## Surprising Connections (you probably didn't know these)
|
||||
- `Browser Sidecar Service` --semantically_similar_to--> `Browser Sidecar (BROWSER_WS_URL)` [INFERRED] [semantically similar]
|
||||
chrome/docker-compose.yml → backend/AGENTS.md
|
||||
- `el()` --indirect_call--> `c()` [INFERRED]
|
||||
userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js
|
||||
- `el()` --indirect_call--> `c()` [INFERRED]
|
||||
@@ -271,6 +257,8 @@
|
||||
userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js
|
||||
- `el()` --indirect_call--> `k()` [INFERRED]
|
||||
userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js
|
||||
- `el()` --indirect_call--> `k()` [INFERRED]
|
||||
userscript/novel-bookmark.user.js → backend/internal/web/static/htmx.min.js
|
||||
|
||||
## Import Cycles
|
||||
- None detected.
|
||||
@@ -284,7 +272,7 @@
|
||||
- **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85]
|
||||
- **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85]
|
||||
|
||||
## Communities (233 total, 170 thin omitted)
|
||||
## Communities (221 total, 154 thin omitted)
|
||||
|
||||
### Community 0 - "HTMX Library Internals"
|
||||
Cohesion: 0.08
|
||||
@@ -296,27 +284,27 @@ Nodes (84): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecor
|
||||
|
||||
### Community 2 - "Manga Userscript Adapters"
|
||||
Cohesion: 0.06
|
||||
Nodes (76): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+68 more)
|
||||
Nodes (77): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+69 more)
|
||||
|
||||
### Community 3 - "Novel Userscript Adapters"
|
||||
Cohesion: 0.06
|
||||
Nodes (79): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+71 more)
|
||||
Nodes (78): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+70 more)
|
||||
|
||||
### Community 4 - "Series Acquisition Tests"
|
||||
Cohesion: 0.10
|
||||
Nodes (67): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+59 more)
|
||||
Cohesion: 0.09
|
||||
Nodes (69): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+61 more)
|
||||
|
||||
### Community 5 - "Bookmarks API Tests"
|
||||
Cohesion: 0.08
|
||||
Nodes (67): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+59 more)
|
||||
|
||||
### Community 7 - "Cover & Acquire Internals"
|
||||
Cohesion: 0.09
|
||||
Nodes (31): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+23 more)
|
||||
Cohesion: 0.10
|
||||
Nodes (30): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+22 more)
|
||||
|
||||
### Community 8 - "System Architecture Concepts"
|
||||
Cohesion: 0.10
|
||||
Nodes (26): Confirm-Gated Destructive Actions, Discord OAuth & Guild-Membership Gate, Lifecycle Buckets (reading/archived/finished), Reader-Owned Store, HMAC-Derived Reader Credentials, Web Package (Browser UI + Templates), app.html — App Shell Template, Manga/Novel Library Switch (+18 more)
|
||||
Cohesion: 0.13
|
||||
Nodes (20): app.html — App Shell Template, Manga/Novel Library Switch, Bookmark Bucket Tabs, Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key (+12 more)
|
||||
|
||||
### Community 9 - "Session Middleware"
|
||||
Cohesion: 0.07
|
||||
@@ -327,29 +315,37 @@ Cohesion: 0.05
|
||||
Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more)
|
||||
|
||||
### Community 11 - "Store Tests"
|
||||
Cohesion: 0.07
|
||||
Nodes (79): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+71 more)
|
||||
Cohesion: 0.15
|
||||
Nodes (43): Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck(), seedSecondReader() (+35 more)
|
||||
|
||||
### Community 12 - "Bookmarks API Handler"
|
||||
Cohesion: 0.08
|
||||
Nodes (33): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+25 more)
|
||||
Cohesion: 0.11
|
||||
Nodes (24): Auth(), compressible(), CORS(), Handler, ResponseWriter, Store, Gzip(), ResolveReader() (+16 more)
|
||||
|
||||
### Community 13 - "Web UI Handlers"
|
||||
Cohesion: 0.06
|
||||
Nodes (24): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, currentLib(), currentTab(), filterBookmarks(), Client (+16 more)
|
||||
Cohesion: 0.14
|
||||
Nodes (18): currentLib(), currentTab(), filterBookmarks(), Client, HandlerFunc, Request, ResponseWriter, Store (+10 more)
|
||||
|
||||
### Community 14 - "Go Error Handling"
|
||||
Cohesion: 0.06
|
||||
Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more)
|
||||
|
||||
### Community 15 - "CDP Browser Client"
|
||||
Cohesion: 0.08
|
||||
Nodes (31): awaitPromise(), browserConnectionLost(), classifyBrowserError(), Action, Context, Mutex, jsString(), kaganeAPIURL() (+23 more)
|
||||
Cohesion: 0.07
|
||||
Nodes (36): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+28 more)
|
||||
|
||||
### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented"
|
||||
Cohesion: 0.06
|
||||
Nodes (33): 1.1 The score itself, 1.2 The detection engines (Enterprise Bot Management), 1.3 Rate limiting is a separate product, 1. What a bot score is and what feeds it, 2.1 What each plan gets, 2.2 Bot Fight Mode specifics (the Free-plan product), 2.3 Does the free tier "score" continuously?, 2. The free-plan reality (+25 more)
|
||||
|
||||
### Community 17 - "Go Code Style Guide"
|
||||
Cohesion: 0.08
|
||||
Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more)
|
||||
|
||||
### Community 19 - "Store"
|
||||
Cohesion: 0.09
|
||||
Nodes (8): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, scanSeries(), TestDisplayChapter(), Bookmark, ReaderSummary
|
||||
|
||||
### Community 20 - "I/O Performance Patterns"
|
||||
Cohesion: 0.11
|
||||
Nodes (18): Avoid io.ReadAll for large payloads, Batch Operations, Buffered I/O, Cgo Overhead, Channel: batch processing from a stream, Concurrent Multi-Stage Pipelines, Connection pooling, Database: batch inserts over row-by-row (+10 more)
|
||||
@@ -374,25 +370,25 @@ Nodes (15): Allocation Rate Reduction, Ballast pattern (pre-Go 1.19), Garbage Co
|
||||
Cohesion: 0.33
|
||||
Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more)
|
||||
|
||||
### Community 26 - "readSeries"
|
||||
Cohesion: 0.13
|
||||
Nodes (28): asuraLatestChapter(), browserOnlyCoverURL(), comixCoverEntry(), comixCoverURL(), comixLatestChapter(), comixSeriesID(), coverFrom(), demonicLatestChapter() (+20 more)
|
||||
### Community 26 - "Open"
|
||||
Cohesion: 0.20
|
||||
Nodes (17): applyMigration(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen(), TestMigration0002BackfillsExistingBookmarks() (+9 more)
|
||||
|
||||
### Community 27 - "Find Skills Guide"
|
||||
Cohesion: 0.14
|
||||
Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more)
|
||||
|
||||
### Community 28 - "Allocation Patterns"
|
||||
Cohesion: 0.14
|
||||
Nodes (14): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map size hints, Memory Optimization (+6 more)
|
||||
Cohesion: 0.11
|
||||
Nodes (19): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map of pointers for large, frequently updated structs, Map size hints (+11 more)
|
||||
|
||||
### Community 29 - "Observability & Alerting"
|
||||
Cohesion: 0.22
|
||||
Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more)
|
||||
|
||||
### Community 31 - "Memory Layout"
|
||||
Cohesion: 0.40
|
||||
Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct
|
||||
### Community 32 - "Repo Hard Constraints"
|
||||
Cohesion: 0.18
|
||||
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
|
||||
|
||||
### Community 33 - "Go Testing Guide"
|
||||
Cohesion: 0.20
|
||||
@@ -422,6 +418,10 @@ Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations,
|
||||
Cohesion: 0.29
|
||||
Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider
|
||||
|
||||
### Community 41 - "sessions_test.go"
|
||||
Cohesion: 0.48
|
||||
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
|
||||
|
||||
### Community 45 - "Go Perf Alert Rules"
|
||||
Cohesion: 0.50
|
||||
Nodes (4): Prometheus Alerting Rules (Go Performance), GoroutineLeak Alert, HighGCPauseTime Alert, MemoryNearLimit Alert
|
||||
@@ -443,8 +443,8 @@ Cohesion: 1.00
|
||||
Nodes (3): Mirrored Double Bookmark Mark, Ember Flame Accent, BookmarkManager Logo
|
||||
|
||||
### Community 103 - "bookmark-api Service"
|
||||
Cohesion: 0.24
|
||||
Nodes (10): Backend Go Service (stdlib net/http), Browser Sidecar (BROWSER_WS_URL), Browser Sidecar Service, CDP Endpoint (Tailnet-Bound :9222), Persistent Chrome Profile Volume, chrome/docker-compose.yml — Browser Deployable Unit, bookmark-api Prod Override, CDP Never on Shared Proxy Network (+2 more)
|
||||
Cohesion: 0.32
|
||||
Nodes (8): Browser Sidecar Service, CDP Endpoint (Tailnet-Bound :9222), Persistent Chrome Profile Volume, chrome/docker-compose.yml — Browser Deployable Unit, bookmark-api Prod Override, CDP Never on Shared Proxy Network, docker-compose.prod.yml — Production Override, Traefik Reverse Proxy Labels
|
||||
|
||||
### Community 104 - "AGENTS.md"
|
||||
Cohesion: 0.12
|
||||
@@ -530,29 +530,33 @@ Nodes (3): Consequence, The wire format stays flat and deliberately does not mir
|
||||
Cohesion: 0.50
|
||||
Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision
|
||||
|
||||
### Community 158 - "T"
|
||||
Cohesion: 0.08
|
||||
Nodes (38): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Request, ReaderID() (+30 more)
|
||||
|
||||
### Community 273 - "AGENTS.md"
|
||||
Cohesion: 0.50
|
||||
Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust), Second script: `novel-bookmark.user.js`, Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
||||
|
||||
## Knowledge Gaps
|
||||
- **508 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+503 more)
|
||||
- **520 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+515 more)
|
||||
These have ≤1 connection - possible missing edges or undocumented components.
|
||||
- **170 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||
- **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||
|
||||
## Suggested Questions
|
||||
_Questions this graph is uniquely positioned to answer:_
|
||||
|
||||
- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Store Tests`, `Web UI Handlers`?**
|
||||
_High betweenness centrality (0.052) - this node is a cross-community bridge._
|
||||
- **Why does `Open()` connect `Store Tests` to `Cover Fetch Test Helpers`, `Web UI Handlers`, `Series Acquisition Tests`, `Bookmarks API Tests`?**
|
||||
_High betweenness centrality (0.034) - this node is a cross-community bridge._
|
||||
- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`, `Open`?**
|
||||
_High betweenness centrality (0.051) - this node is a cross-community bridge._
|
||||
- **Why does `Open()` connect `Open` to `Repo Hard Constraints`, `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `Store Tests`, `Store`?**
|
||||
_High betweenness centrality (0.038) - this node is a cross-community bridge._
|
||||
- **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?**
|
||||
_High betweenness centrality (0.027) - this node is a cross-community bridge._
|
||||
_High betweenness centrality (0.030) - this node is a cross-community bridge._
|
||||
- **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?**
|
||||
_`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?**
|
||||
_`newWebTestServer()` has 8 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **Are the 6 inferred relationships involving `newTestStore()` (e.g. with `TestCreateAndGetSession()` and `TestDeleteSessionIsPerReader()`) actually correct?**
|
||||
_`newTestStore()` has 6 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **Are the 12 inferred relationships involving `newTestStore()` (e.g. with `TestAcquireDoesNotBlockTheWrite()` and `TestAcquireFailureLeavesTheBookmarkIntact()`) actually correct?**
|
||||
_`newTestStore()` has 12 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?**
|
||||
_548 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
_553 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
File diff suppressed because one or more lines are too long
+1451
-807
File diff suppressed because it is too large
Load Diff
+65
-55
@@ -25,7 +25,7 @@
|
||||
"semantic_hash": "dac242903b0e98c3e4395159d609e08e"
|
||||
},
|
||||
"backend/main.go": {
|
||||
"mtime": 1786363889.5678573,
|
||||
"mtime": 1786501521.228955,
|
||||
"ast_hash": "6e98a3ae91aaa132df251e43c4dfca6d",
|
||||
"semantic_hash": "6e98a3ae91aaa132df251e43c4dfca6d"
|
||||
},
|
||||
@@ -35,7 +35,7 @@
|
||||
"semantic_hash": "4a94ac85bad6bce330d085bcc0ae3ffd"
|
||||
},
|
||||
"userscript/manga-bookmark.user.js": {
|
||||
"mtime": 1786488438.9080842,
|
||||
"mtime": 1786499529.779988,
|
||||
"ast_hash": "1f8bcddd3632d709f058a8401af8f127",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -140,17 +140,17 @@
|
||||
"semantic_hash": "3a08979e4603aae5c32a58d5b6c39765"
|
||||
},
|
||||
"CLAUDE.md": {
|
||||
"mtime": 1786488493.5922732,
|
||||
"ast_hash": "7362a25f37333a2a6a55a5aeccf9b0cb",
|
||||
"mtime": 1786856633.445473,
|
||||
"ast_hash": "fab288c23be960d9c6afbfe3f21ff41c",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"DEPLOY.md": {
|
||||
"mtime": 1786488464.5532806,
|
||||
"ast_hash": "2b7b537aa1c0954400c19acc4b634029",
|
||||
"mtime": 1786501942.7367291,
|
||||
"ast_hash": "3c7b785c44badb6dda6234d2b39a9290",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"README.md": {
|
||||
"mtime": 1786488487.8305292,
|
||||
"mtime": 1786499529.7651505,
|
||||
"ast_hash": "9d6be8aa8a2946c23ad48d8f2864b5ca",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -160,7 +160,7 @@
|
||||
"semantic_hash": "0751998a532297b8ac507a01ec48dc31"
|
||||
},
|
||||
"docker-compose.yml": {
|
||||
"mtime": 1786488450.3508112,
|
||||
"mtime": 1786499529.7725692,
|
||||
"ast_hash": "124fd581bf0a662ff15012abfdb40a92",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -175,7 +175,7 @@
|
||||
"semantic_hash": "8f1b093b59eb1ed81bc7fc0c22495c50"
|
||||
},
|
||||
"backend/main_test.go": {
|
||||
"mtime": 1786363889.5678573,
|
||||
"mtime": 1786501521.228955,
|
||||
"ast_hash": "8a165955cf28ad47481fec5ea7afb3d6",
|
||||
"semantic_hash": "8a165955cf28ad47481fec5ea7afb3d6"
|
||||
},
|
||||
@@ -200,12 +200,12 @@
|
||||
"semantic_hash": "e69a8340a371579ca3ea689660f7d7bd"
|
||||
},
|
||||
"AGENTS.md": {
|
||||
"mtime": 1786488493.5922732,
|
||||
"ast_hash": "7362a25f37333a2a6a55a5aeccf9b0cb",
|
||||
"mtime": 1786856633.445473,
|
||||
"ast_hash": "fab288c23be960d9c6afbfe3f21ff41c",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"userscript/test/logic.test.js": {
|
||||
"mtime": 1786488563.581765,
|
||||
"mtime": 1786499529.779988,
|
||||
"ast_hash": "80d512bfe6fa8b847f3ba6169c321a74",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -215,9 +215,9 @@
|
||||
"semantic_hash": "8f3c0132eb4787a2c8736eb99f7689af"
|
||||
},
|
||||
"REDEPLOY.md": {
|
||||
"mtime": 1786363889.552731,
|
||||
"ast_hash": "d0baf08b95e7b5986234a9f36759c12e",
|
||||
"semantic_hash": "d0baf08b95e7b5986234a9f36759c12e"
|
||||
"mtime": 1786856643.0770833,
|
||||
"ast_hash": "e5e910f0244a040e2ccdc669b17eb4db",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/design-system.md": {
|
||||
"mtime": 1786022513.9623306,
|
||||
@@ -225,7 +225,7 @@
|
||||
"semantic_hash": "421cd7e57f02d4b467f120ca6ddd7b6a"
|
||||
},
|
||||
"backend/api_test.go": {
|
||||
"mtime": 1786488480.637822,
|
||||
"mtime": 1786499529.7651505,
|
||||
"ast_hash": "8e4b9293bc2e45ee3f42027315594fd5",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -245,38 +245,38 @@
|
||||
"semantic_hash": "385b36f58488b7e6d93eb6d6034e9ee3"
|
||||
},
|
||||
"backend/internal/latest/browser.go": {
|
||||
"mtime": 1786469354.923288,
|
||||
"ast_hash": "ed129a7f00601ea90c877ff29fa21220",
|
||||
"mtime": 1786856879.3397639,
|
||||
"ast_hash": "fbdba4bb56d8c804087bcb6e141e52ad",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/browser_test.go": {
|
||||
"mtime": 1786262323.6964688,
|
||||
"ast_hash": "e900f92971486f47d7ef76e9a95217fe",
|
||||
"semantic_hash": "e900f92971486f47d7ef76e9a95217fe"
|
||||
"mtime": 1786856254.36163,
|
||||
"ast_hash": "800fa6aa471ada054a3c48943ad17ab7",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/fetch.go": {
|
||||
"mtime": 1786446006.9219902,
|
||||
"mtime": 1786499529.7688599,
|
||||
"ast_hash": "3e20ad86aa46783e9aa95c2b746551ee",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/poller.go": {
|
||||
"mtime": 1786469641.8342345,
|
||||
"ast_hash": "44fef6074ac2eaffc8233f46aad5236b",
|
||||
"mtime": 1786856670.129715,
|
||||
"ast_hash": "6b7060ff52994729832d9fcc3cc97dad",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/poller_test.go": {
|
||||
"mtime": 1786469644.9306462,
|
||||
"ast_hash": "64bc838c822f1bf33bbf9e291215454b",
|
||||
"mtime": 1786856311.8535168,
|
||||
"ast_hash": "b6837fab377c94fa1d2f20a931975a4a",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/sites.go": {
|
||||
"mtime": 1786469348.5394242,
|
||||
"ast_hash": "b744cc685363317a526cc3bebceea39e",
|
||||
"mtime": 1786856865.4529688,
|
||||
"ast_hash": "2f00456ac9d1d8148ba1bc6cbceb24f6",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/sites_test.go": {
|
||||
"mtime": 1786446006.9219902,
|
||||
"ast_hash": "eabca9014a306e3c71d238b0ae499f61",
|
||||
"mtime": 1786856890.1231265,
|
||||
"ast_hash": "0db2028a6073f342fee61ad15c2e5f0f",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/smoke_image_test.go": {
|
||||
@@ -395,12 +395,12 @@
|
||||
"semantic_hash": "8008a187690764436540fab47ba0cfcc"
|
||||
},
|
||||
"userscript/novel-bookmark.user.js": {
|
||||
"mtime": 1786454823.798836,
|
||||
"mtime": 1786499529.779988,
|
||||
"ast_hash": "834effb0821f8d6c9f57f6554a5db462",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"userscript/test/novel-logic.test.js": {
|
||||
"mtime": 1786454566.070801,
|
||||
"mtime": 1786499529.779988,
|
||||
"ast_hash": "b25a7377af210dd0aff8284501fd0252",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -415,8 +415,8 @@
|
||||
"semantic_hash": "e44a2f6f624db044e19508bc5ab05592"
|
||||
},
|
||||
"CONTEXT.md": {
|
||||
"mtime": 1786466512.6719532,
|
||||
"ast_hash": "544b7d93f1d5cb9d347cb0b92fc3a709",
|
||||
"mtime": 1786855457.9327722,
|
||||
"ast_hash": "24548f60414b4c5ff58538acaada5345",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"CUTOVER.md": {
|
||||
@@ -425,14 +425,14 @@
|
||||
"semantic_hash": "6c6f3e4c4c2f57867894280bce728c50"
|
||||
},
|
||||
"backend/AGENTS.md": {
|
||||
"mtime": 1786363889.552731,
|
||||
"ast_hash": "6356ee58447f299e5fa7aa83486bbbe0",
|
||||
"semantic_hash": "6356ee58447f299e5fa7aa83486bbbe0"
|
||||
"mtime": 1786856656.6826186,
|
||||
"ast_hash": "d5610ba24076b57e17b9d76241acaa65",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/CLAUDE.md": {
|
||||
"mtime": 1786363889.552731,
|
||||
"ast_hash": "6356ee58447f299e5fa7aa83486bbbe0",
|
||||
"semantic_hash": "6356ee58447f299e5fa7aa83486bbbe0"
|
||||
"mtime": 1786856656.6826186,
|
||||
"ast_hash": "d5610ba24076b57e17b9d76241acaa65",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/web/templates/app.html": {
|
||||
"mtime": 1786216141.692644,
|
||||
@@ -485,9 +485,9 @@
|
||||
"semantic_hash": "852a04d86659385085da6ffc8b489933"
|
||||
},
|
||||
"docs/adr/0003-series-shared-and-poll-owned.md": {
|
||||
"mtime": 1786216141.7071996,
|
||||
"ast_hash": "58bb4d24e20f6f7adc1a8b9cf7967c3f",
|
||||
"semantic_hash": "58bb4d24e20f6f7adc1a8b9cf7967c3f"
|
||||
"mtime": 1786501942.7367291,
|
||||
"ast_hash": "6138b113340693e0cc667d1ecbb75f72",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/adr/0004-wire-format-does-not-mirror-the-schema.md": {
|
||||
"mtime": 1786216141.7071996,
|
||||
@@ -510,12 +510,12 @@
|
||||
"semantic_hash": "69114d07ed792d6bb1d13758ba5435e1"
|
||||
},
|
||||
"userscript/AGENTS.md": {
|
||||
"mtime": 1786454605.6727684,
|
||||
"mtime": 1786499529.7762787,
|
||||
"ast_hash": "e276ffe9a6e7b55fd3235466a1995c22",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"userscript/CLAUDE.md": {
|
||||
"mtime": 1786454605.6727684,
|
||||
"mtime": 1786499529.7762787,
|
||||
"ast_hash": "e276ffe9a6e7b55fd3235466a1995c22",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -545,9 +545,9 @@
|
||||
"semantic_hash": "5605599395a3f085904e78a2bfec1e58"
|
||||
},
|
||||
"docs/adr/0006-browser-on-the-home-machine.md": {
|
||||
"mtime": 1786292465.8305523,
|
||||
"ast_hash": "dfd6bbc045d23315f2942ee8d98eb7db",
|
||||
"semantic_hash": "dfd6bbc045d23315f2942ee8d98eb7db"
|
||||
"mtime": 1786501942.7367291,
|
||||
"ast_hash": "bfcaf39b6c7e96610a7caa00eeb36533",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/adr/0007-backend-hosts-cover-bytes.md": {
|
||||
"mtime": 1786292465.8305523,
|
||||
@@ -560,9 +560,9 @@
|
||||
"semantic_hash": "46cf7822d4f667e3cab36b547abe5e97"
|
||||
},
|
||||
"backend/internal/latest/cover.go": {
|
||||
"mtime": 1786363889.5565126,
|
||||
"ast_hash": "e6749cfe3cd7c2e71d4392dde84f55f9",
|
||||
"semantic_hash": "e6749cfe3cd7c2e71d4392dde84f55f9"
|
||||
"mtime": 1786856664.2957466,
|
||||
"ast_hash": "d5f2248c3d11de74bf5a3977651b17c2",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/cover_fetch_test.go": {
|
||||
"mtime": 1786363889.5565126,
|
||||
@@ -570,7 +570,7 @@
|
||||
"semantic_hash": "60d9eb7c59a3751baf4f31c7655217e7"
|
||||
},
|
||||
"backend/internal/latest/acquire.go": {
|
||||
"mtime": 1786468950.9432797,
|
||||
"mtime": 1786499529.7688599,
|
||||
"ast_hash": "6c1ad34bbe9f5b49d0fd1eae9093f55d",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -605,18 +605,28 @@
|
||||
"semantic_hash": "74a4e538875f0a7e8ca3d5dc48c0bb53"
|
||||
},
|
||||
"backend/internal/latest/smoke_lnw_test.go": {
|
||||
"mtime": 1786446006.9219902,
|
||||
"mtime": 1786499529.7725692,
|
||||
"ast_hash": "2d65da8a081759172918fdf159760f45",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/adr/0009-a-site-answers-questions-its-own-way.md": {
|
||||
"mtime": 1786469367.6875648,
|
||||
"mtime": 1786499529.7725692,
|
||||
"ast_hash": "8039012a5b6de2359ff1a47079f51b66",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/read.go": {
|
||||
"mtime": 1786469329.7445614,
|
||||
"mtime": 1786499529.7688599,
|
||||
"ast_hash": "3cf29046ddaef39fafb1df70b9f9ae8c",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/research/cloudflare-bot-scoring-and-poll-cadence.md": {
|
||||
"mtime": 1786501942.7367291,
|
||||
"ast_hash": "1aa17575ab20f2f36583602999a6a60f",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/smoke_comix_test.go": {
|
||||
"mtime": 1786856356.3432186,
|
||||
"ast_hash": "111fdbb75fc68ac2ab1013bc916063cf",
|
||||
"semantic_hash": ""
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user