Compare commits

..

2 Commits

Author SHA1 Message Date
sulthan 3e097a3cdf chore: refresh the graphify knowledge graph 2026-08-16 11:32:06 +07:00
sulthan dc26993847 docs: add Poll Lane and Sighting to the shared vocabulary
Latest Chapter no longer claims to be discovered without the reader
present: a Sighting establishes it between Polls. Specs for the work
these terms describe are issues #101, #102 and #103.
2026-08-16 11:31:31 +07:00
28 changed files with 2553 additions and 4573 deletions
+18 -5
View File
@@ -73,13 +73,26 @@ DISCORD_REDIRECT_URI=
# The backend re-checks each bookmarked series' newest published chapter on its # The backend re-checks each bookmarked series' newest published chapter on its
# own schedule, so latest_chapter stays fresh even when you never open the manga # own schedule, so latest_chapter stays fresh even when you never open the manga
# sites. This runs in parallel with the userscript's own in-browser check. # sites. This runs in parallel with the userscript's own in-browser check.
# Set to 0 to turn it off entirely. Pace is per Site (one Poll Lane per Site, # Set to 0 to turn it off entirely.
# issue #100) and lives in the backend registry, not here — there is nothing
# else to configure.
# LATEST_CHAPTER_POLL_ENABLED=1 # LATEST_CHAPTER_POLL_ENABLED=1
# #
# Every Site rests an hour between checks and gaps ten seconds between fetches; # Two independent clocks. COOLDOWN is how long a plain-TLS series rests between
# a Site with many Series tightens its own gap. See backend/internal/latest/sites.go. # checks; BROWSER_COOLDOWN is the longer rest for kagane and novelfull. INTERVAL
# is how often the poller wakes up and looks for series past their cooldowns.
# Shortening INTERVAL cannot shorten either cooldown.
LATEST_CHAPTER_POLL_COOLDOWN=1h # plain-TLS per series, floor 15m
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN=6h # browser-backed per series, floor 15m
LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
LATEST_CHAPTER_POLL_BATCH=14 # series per wake
LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
#
# Uses a ticker, not an immediate first run: the first poll happens one
# INTERVAL after startup, not at startup. A container restarting more often
# than INTERVAL never polls.
#
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
# CDP endpoint of the browser, used for the two sites behind a Cloudflare # CDP endpoint of the browser, used for the two sites behind a Cloudflare
# JavaScript challenge (kagane, novelfull) and by the web UI's kagane cover # JavaScript challenge (kagane, novelfull) and by the web UI's kagane cover
+5 -13
View File
@@ -19,9 +19,9 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional. - Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
- Userscript run in **isolated world**, so embedded API token safe from site's JS. - Userscript run in **isolated world**, so embedded API token safe from site's JS.
- Cloudflare's block on manga sites is **per-zone configuration plus request fingerprint, not IP reputation — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — a Site can turn its protection on overnight, which is exactly what comix.to did on 2026-08-12. An earlier version of this line blamed "Cloudflare's bot scoring"; that was wrong. The 1-99 bot score is Enterprise Bot Management only and does not exist for a free-plan zone, and no per-IP request rate is documented as an input to challenge issuance — `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test. - Cloudflare's block on manga sites is **per-zone configuration plus request fingerprint, not IP reputation — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — a Site can turn its protection on overnight, which is exactly what comix.to did on 2026-08-12. An earlier version of this line blamed "Cloudflare's bot scoring"; that was wrong. The 1-99 bot score is Enterprise Bot Management only and does not exist for a free-plan zone, and no per-IP request rate is documented as an input to challenge issuance — `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
- **kagane.to, comix.to and novelfull.com are the exception to the above** — all three sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`). When that's unset, kagane and comix are skipped entirely (a plain fetch would only retrieve a challenge page) while novelfull pages are still attempted over plain TLS — its challenge is a live time-varying fact and its cover bytes never need the browser. comix turned hostile on 2026-08-12 (#98): its cover host `static.comix.to` is gated too, so its cover bytes go through the browser as well, and its page is read as an in-tab `fetch()` of the series URL rather than a rendered DOM — comix is an SPA, and rendering costs ~65 requests for the same server-rendered HTML one fetch returns. The three other sites poll fine over plain TLS. - **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`). When that's unset, kagane is skipped entirely (a plain fetch would only retrieve a challenge page) while novelfull pages are still attempted over plain TLS — its challenge is a live time-varying fact and its cover bytes never need the browser. The four other sites poll fine over plain TLS.
- **The CDP browser must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a non-UTC clock zone cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, ignoring the file's contents, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead. - **The CDP browser must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a non-UTC clock zone cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, ignoring the file's contents, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
- **The browser is not in the API stack and must not be put back.** It's its own compose unit (`chrome/docker-compose.yml`) on a second machine, reached over the tailnet — it held 471 MiB on a 1974 MiB swapless VPS, and a residential egress avoids the cloud-hosting-IP signature Bot Fight Mode documentedly challenges (ADR-0006; not a better "score" — free-plan zones have no score). Consequences that constrain code: `BROWSER_WS_URL` must be a tailnet **IP** (a MagicDNS name 500s at `/json/version`, same trap as the old Docker service name); the CDP port binds to the tailnet address only, since CDP authenticates nothing and that host has a real LAN; and the browser is on-demand (ADR-0005), so an unreachable or asleep one must degrade exactly as an unset `BROWSER_WS_URL` — plain-TLS libraries unaffected, kagane/comix logged and skipped, stored covers still served. Never add `chromedp.NoModifyURL`: discovery per fetch is what makes a restarted Chrome invisible. - **The browser is not in the API stack and must not be put back.** It's its own compose unit (`chrome/docker-compose.yml`) on a second machine, reached over the tailnet — it held 471 MiB on a 1974 MiB swapless VPS, and a residential egress avoids the cloud-hosting-IP signature Bot Fight Mode documentedly challenges (ADR-0006; not a better "score" — free-plan zones have no score). Consequences that constrain code: `BROWSER_WS_URL` must be a tailnet **IP** (a MagicDNS name 500s at `/json/version`, same trap as the old Docker service name); the CDP port binds to the tailnet address only, since CDP authenticates nothing and that host has a real LAN; and the browser is on-demand (ADR-0005), so an unreachable or asleep one must degrade exactly as an unset `BROWSER_WS_URL` — plain-TLS libraries unaffected, kagane/novelfull logged and skipped, stored covers still served. Never add `chromedp.NoModifyURL`: discovery per fetch is what makes a restarted Chrome invisible.
- **UTC is the tell, not a country mismatch.** A UTC clock is the datacenter default, and the challenge refuses it; any real zone clears. Measured 2026-08-08, identical container, one Indonesian egress IP: UTC never cleared in 60s (twice), while `Asia/Jakarta` **and** `America/New_York` both cleared in 4s. An earlier note here claimed the zone had to match the egress IP's country — that was wrong, inferred from the host clock (`Asia/Bangkok`) rather than the measured egress. A second earlier claim, that Cloudflare "scores" a UTC clock, was also wrong: the measurement is real but the mechanism is not documented anywhere — Cloudflare publishes no timezone signal, and free-plan zones carry no score at all. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one. - **UTC is the tell, not a country mismatch.** A UTC clock is the datacenter default, and the challenge refuses it; any real zone clears. Measured 2026-08-08, identical container, one Indonesian egress IP: UTC never cleared in 60s (twice), while `Asia/Jakarta` **and** `America/New_York` both cleared in 4s. An earlier note here claimed the zone had to match the egress IP's country — that was wrong, inferred from the host clock (`Asia/Bangkok`) rather than the measured egress. A second earlier claim, that Cloudflare "scores" a UTC clock, was also wrong: the measurement is real but the mechanism is not documented anywhere — Cloudflare publishes no timezone signal, and free-plan zones carry no score at all. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one.
- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives. - **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
@@ -45,20 +45,12 @@ Backend (`cd backend`):
- Single test: `go test -run TestName ./...` - Single test: `go test -run TestName ./...`
- Build static binary: `CGO_ENABLED=0 go build` - Build static binary: `CGO_ENABLED=0 go build`
Local stack: `docker compose up` (bookmark-api + postgres only; `postgres-data` named volume, `restart: unless-stopped`). No browser — without `BROWSER_WS_URL` the poller logs and skips kagane and comix. To run one: `cd chrome && BROWSER_BIND_ADDR=172.17.0.1 docker compose up -d --build`, then `BROWSER_WS_URL=ws://172.17.0.1:9222` in the root `.env` (bridge gateway, so the API container can name it by IP). Local stack: `docker compose up` (bookmark-api + postgres only; `postgres-data` named volume, `restart: unless-stopped`). No browser — without `BROWSER_WS_URL` the poller logs and skips kagane and novelfull. To run one: `cd chrome && BROWSER_BIND_ADDR=172.17.0.1 docker compose up -d --build`, then `BROWSER_WS_URL=ws://172.17.0.1:9222` in the root `.env` (bridge gateway, so the API container can name it by IP).
Live CDP proof (needs that browser and network, skipped otherwise): Live CDP proof (needs that browser and network, skipped otherwise):
`SMOKE_BROWSER_WS_URL=ws://<ip>:<port> go test -run 'TestSmokeKagane|TestSmokeComix' ./internal/latest` `SMOKE_BROWSER_WS_URL=ws://<ip>:<port> go test -run TestSmokeKagane ./internal/latest`
— fetches a real kagane and comix cover and chapter list. A red run means the challenge is — fetches a real kagane cover and chapter list. A red run means the challenge is
not clearing from this IP, which is a live fact to re-check, not necessarily a defect. not clearing from this IP, which is a live fact to re-check, not necessarily a defect.
A red `TestSmokeComix` reporting `ERR_CERT_COMMON_NAME_INVALID` is not the
challenge: it means the resolver the browser container uses hijacks `comix.to`.
Observed 2026-08-16 on one Indonesian ISP, which CNAMEs it to a block page
(`aduankonten.id`). Check with `docker exec <browser> getent hosts comix.to`,
and if it is hijacked, run the container with
`--add-host comix.to:<ip> --add-host static.comix.to:<ip>` from a DoH lookup
(`curl -H 'accept: application/dns-json' 'https://1.1.1.1/dns-query?name=comix.to&type=A'`).
Machine-local, so don't put those hosts in `chrome/docker-compose.yml`.
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200. Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
+1 -1
View File
@@ -86,7 +86,7 @@ _Avoid_: client report, user poll, observation, claim
**Acquisition**: **Acquisition**:
The single read of a Series page made the moment the Series first exists, giving it The single read of a Series page made the moment the Series first exists, giving it
both its Latest Chapter and its Cover without waiting for the Lane's pace. Distinct both its Latest Chapter and its Cover without waiting out the Poll queue. Distinct
from a Poll in the two ways that matter: a Reader is present — it is triggered by from a Poll in the two ways that matter: a Reader is present — it is triggered by
their first Bookmark of that Series — and it is the only read that establishes a their first Bookmark of that Series — and it is the only read that establishes a
Cover rather than refreshing facts. It happens once in a Series's life; every later Cover rather than refreshing facts. It happens once in a Series's life; every later
+1 -9
View File
@@ -93,14 +93,6 @@ the backend dials but not the password the database expects, and `bookmark-api`
crash-loops on `password authentication failed`. Set it before §2 and leave it crash-loops on `password authentication failed`. Set it before §2 and leave it
alone. alone.
An `.env` written before issue #100 carries the old poll-pace names
(`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
`_STAGGER`). All five are dead configuration now — the pace lives in the Site
registry (`backend/internal/latest/sites.go`), so **delete those lines** and
keep only the kill switch `LATEST_CHAPTER_POLL_ENABLED`. Leaving them behind
is harmless (nothing reads them) but silently misleads the next person who
edits the file.
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual > Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
> names (check your Traefik static config — common alternatives: `https`, > names (check your Traefik static config — common alternatives: `https`,
> `myresolver`, `cloudflare`). Wrong names = no certificate issued. > `myresolver`, `cloudflare`). Wrong names = no certificate issued.
@@ -513,7 +505,7 @@ picks up a restarted Chrome's new debugger UUID by itself.
| kagane rows never get a `latest_chapter`; log says `browser fetcher disabled` or nothing at all | `BROWSER_WS_URL` unset. Expected before §7 is done. | | kagane rows never get a `latest_chapter`; log says `browser fetcher disabled` or nothing at all | `BROWSER_WS_URL` unset. Expected before §7 is done. |
| kagane polls all fail; log shows a 500 from `/json/version` | `BROWSER_WS_URL` names a MagicDNS hostname (or any name). Chrome's DevTools handler only accepts an IP or `localhost` — use the tailnet IP. | | kagane polls all fail; log shows a 500 from `/json/version` | `BROWSER_WS_URL` names a MagicDNS hostname (or any name). Chrome's DevTools handler only accepts an IP or `localhost` — use the tailnet IP. |
| kagane polls fail with a connection error | Home machine off, off the tailnet, or the unit is down. `tailscale ping <machine>`, then `docker compose ps` in its `chrome/`. Costs freshness only; stored covers keep serving. | | kagane polls fail with a connection error | Home machine off, off the tailnet, or the unit is down. `tailscale ping <machine>`, then `docker compose ps` in its `chrome/`. Costs freshness only; stored covers keep serving. |
| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series. | | kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series (up to `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, default 6h). |
| `compose` in `chrome/` errors `set BROWSER_BIND_ADDR to this machine's tailnet IP` | No `chrome/.env`, or the variable is empty. Deliberate — it has no default so an unset value cannot publish CDP to the LAN. | | `compose` in `chrome/` errors `set BROWSER_BIND_ADDR to this machine's tailnet IP` | No `chrome/.env`, or the variable is empty. Deliberate — it has no default so an unset value cannot publish CDP to the LAN. |
| browser container restarts, or is OOM-killed | `docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'`. The 512 MiB cap is sized against a measured 645 MiB untuned peak; a real breach is a Chrome regression worth reading `docker logs` for, not a number to raise reflexively. | | browser container restarts, or is OOM-killed | `docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'`. The 512 MiB cap is sized against a measured 645 MiB untuned peak; a real breach is a Chrome regression worth reading `docker logs` for, not a number to raise reflexively. |
+6 -1
View File
@@ -53,7 +53,12 @@ covers are stored, so the library renders in full with the browser switched off.
| `DISCORD_API_BASE` | `https://discord.com/api/v10` | Test seam — tests point it at a local stub so the real token exchange runs. | | `DISCORD_API_BASE` | `https://discord.com/api/v10` | Test seam — tests point it at a local stub so the real token exchange runs. |
| `USERSCRIPT_PATH` | `/userscript/manga-bookmark.user.js` | Bindmounted file served at `/u/{token}/manga-bookmark.user.js`. | | `USERSCRIPT_PATH` | `/userscript/manga-bookmark.user.js` | Bindmounted file served at `/u/{token}/manga-bookmark.user.js`. |
| `NOVEL_USERSCRIPT_PATH` | `/userscript/novel-bookmark.user.js` | Same, for the novel library. | | `NOVEL_USERSCRIPT_PATH` | `/userscript/novel-bookmark.user.js` | Same, for the novel library. |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. Pace is per Site in the registry — one Poll Lane per Site, each with its own rest and gap (issue #100) — so no other knobs exist. | | `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. |
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Rest between checks of one plain-TLS series; floor `15m`. |
| `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN` | `6h` | Rest between checks of one browser-backed series; floor `15m`. |
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often the poller wakes. Cannot shorten either cooldown. |
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake. Keep `BATCH × STAGGER` under `INTERVAL`. |
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch — this is the outbound request rate. |
Compose reads a few more from the same `.env` that the backend never sees: Compose reads a few more from the same `.env` that the backend never sees:
`POSTGRES_PASSWORD` (required — `DATABASE_URL` is built from it, and Postgres `POSTGRES_PASSWORD` (required — `DATABASE_URL` is built from it, and Postgres
+2 -2
View File
@@ -436,8 +436,8 @@ free -m # the Gitea runner should still have its headroom
Nothing here needs doing during an API redeploy. The API stack does not Nothing here needs doing during an API redeploy. The API stack does not
`depends_on` the browser, and an unreachable one degrades exactly as an unset `depends_on` the browser, and an unreachable one degrades exactly as an unset
`BROWSER_WS_URL`: plain-TLS libraries unaffected, kagane and comix logged `BROWSER_WS_URL`: plain-TLS libraries unaffected, kagane and novelfull logged
and skipped, novelfull attempted over plain TLS, stored covers still served. and skipped, stored covers still served.
--- ---
+36 -60
View File
@@ -74,43 +74,27 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
Remove's row wear ember wash, two reversible ones wear `.calm` grey. Remove's row wear ember wash, two reversible ones wear `.calm` grey.
`--ember` stay reserved for new-chapter signal: busy bar and inline `--ember` stay reserved for new-chapter signal: busy bar and inline
error use `--mute`. error use `--mute`.
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100), - **Latest-chapter poller:** ticker goroutine in same binary re-check
each re-checking that Site's bookmarked series' newest published chapter from each bookmarked series' newest published chapter from backend's own
backend's own network access, so `latest_chapter` stays fresh when the user network access, so `latest_chapter` stay fresh when user not
isn't browsing. Second, parallel signal — the userscript keeps its own browsing. Second, parallel signal — userscript keep own
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged. `maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
Two independent clocks: per-series rest (`series.latest_checked_at`, Two independent clocks: per-series cooldown (`series.latest_checked_at`,
enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
in the Site registry (`internal/latest/sites.go`), not config: the five env
knobs that used to size a shared pace are gone.
The poller walks **Series, not Bookmarks** — a series referenced by several The poller walks **Series, not Bookmarks** — a series referenced by several
bookmarks is fetched once per cycle, and the due queue orders bookmarks is fetched once per cycle, and the due queue orders
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped `reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
*before* fetch so broken series wait out the rest instead of retrying *before* fetch so broken series wait out full cooldown instead of retrying
every tick; found chapter written straight to the series row via every tick; found chapter written straight to the series row via
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list `Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
order — is never touched. order — is never touched.
Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
`errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
that makes the other browser Lanes skip their passes for the same 15m, so a
restarting Chrome doesn't stamp one Series per Lane per pass — after the
window the flag decays and they probe again. Browser Lanes wake Chrome only
when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
and cover work (both healing a stored source URL and filling a blank from
the series page) runs in the background so a slow CDN can't consume a
Lane's gap.
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
against fingerprint-based blocking; any failure log and skip. kagane, comix against fingerprint-based blocking; any failure log and skip. kagane and
and novelfull sit behind Cloudflare JavaScript challenges the TLS client novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane is simply
comix are simply not polled when that's unset, while novelfull falls back to not polled when that's unset, while novelfull falls back to a plain-TLS
a plain-TLS attempt — its challenge is a live time-varying fact, and its attempt — its challenge is a live time-varying fact, and its cover bytes
cover bytes never need the browser. comix's browser read is an in-tab never need the browser. See
`fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
costs ~65 requests for the same server-rendered HTML one fetch returns
(measured 2026-08-12, issue #98). See
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`. `docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
The poller's series write is a single-column UPDATE The poller's series write is a single-column UPDATE
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark: (`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
@@ -128,17 +112,14 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
is public and uncredentialed: the userscript renders it on a Site's origin, is public and uncredentialed: the userscript renders it on a Site's origin,
where no cookie or token of ours travels. A client-sent `cover` is decoded where no cookie or token of ours travels. A client-sent `cover` is decoded
and discarded, permanently (ADR-0004 compatibility). and discarded, permanently (ADR-0004 compatibility).
Browser-backed Sites join the same pipeline (issue #62, extended to comix by Browser-backed Sites join the same pipeline (issue #62): kagane pages *and*
#98): kagane and comix pages *and* cover bytes go through the browser sidecar cover bytes go through the browser sidecar (nothing falls back to a plain
(nothing falls back to a plain fetch, which would only retrieve a challenge fetch, which would only retrieve a challenge page), while novelfull needs
page), while novelfull needs the browser only for its HTML — the cover URL the browser only for its HTML — the cover URL comes out of the
comes out of the browser-fetched page and the bytes go over plain TLS. With browser-fetched page and the bytes go over plain TLS. With no browser
no browser configured, kagane and comix Covers are simply absent; novelfull configured, kagane Covers are simply absent; novelfull still gets one — at
still gets one — at creation and on the poll — when its page body happens to creation and on the poll — when its page body happens to answer a plain
answer a plain request (the challenge is a live time-varying fact). comix request (the challenge is a live time-varying fact). The old kagane-only
cover bytes must arrive by direct navigation, not an in-page fetch: its
Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
page-context fetch of `static.comix.to`. The old kagane-only
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
(issue #63): the one public route serves every Site. (issue #63): the one public route serves every Site.
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4. - **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
@@ -165,17 +146,15 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
`_REDIRECT_URI` (required; Discord OAuth for the browser UI), `_REDIRECT_URI` (required; Discord OAuth for the browser UI),
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default), `DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
`DISCORD_API_BASE` (default `https://discord.com/api/v10`), `DISCORD_API_BASE` (default `https://discord.com/api/v10`),
`LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill `LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/
switch, default on). Pace is per Site in the registry (issue #100): every `_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on,
Site rests an hour and gaps ten seconds, a Site with more eligible Series `1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both
than 360 tightens its own gap toward the 1s floor, and browser Lanes wake cooldowns have a `15m` floor). The browser cooldown is longer for cost, not
Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/ for safety: a challenged page costs seconds of a serialized single-tab
`_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are browser, while a plain read costs one request. It buys no documented
gone. The 1h rest for browser Sites is safe on documented grounds: a reduction in challenge risk — free-plan zones have no bot score and no
challenged page costs seconds of a serialized single-tab browser, free-plan published per-IP rate input, and `cf_clearance` expires in 30 minutes so
zones have no bot score and no published per-IP rate input, and every cadence at or above 1h re-solves anyway —
`cf_clearance` expires in 30 minutes so every cadence at or above 1h
re-solves anyway —
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at `USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`, `/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
@@ -185,11 +164,10 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
Reader's credential at serve time). Reader's credential at serve time).
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate `BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`. machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
Used by the poller for kagane, comix and novelfull page fetches and by the Used by the poller for kagane and novelfull page fetches and by the cover
cover pipeline for kagane's and comix's image bytes (the browser is the only pipeline for kagane's image bytes (the browser is the only route that clears
route that clears the challenge those two serve their covers behind); unset — the challenge kagane serves its covers behind); unset — the default —
the default — disables browser polling and leaves kagane and comix Covers disables browser polling and leaves kagane Covers blank until stored bytes
blank until stored bytes
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
`/json/version` for any Host that isn't an IP or `localhost`). `/json/version` for any Host that isn't an IP or `localhost`).
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is - **No per-Site cover path (issue #63):** every Cover — all six Sites — is
@@ -197,10 +175,8 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
bytes. There is no proxy, no per-Site rewrite, no second place that decides bytes. There is no proxy, no per-Site rewrite, no second place that decides
a Cover's renderable address: the wire `cover` is it. The only place a Site a Cover's renderable address: the wire `cover` is it. The only place a Site
name still appears in cover code is the extraction module (`latest`), where name still appears in cover code is the extraction module (`latest`), where
kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane kagane's image URLs are claimed by `browserOnlyCoverURL` — they answer a
answers a plain fetch with a challenge and plain fetch with a challenge and `cross-origin-resource-policy: same-origin`;
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
one with the same Cloudflare challenge its pages serve;
every other Site's CDN answers plain TLS. Templates render `.Cover` — the every other Site's CDN answers plain TLS. Templates render `.Cover` — the
wire value — never anything else. wire value — never anything else.
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js` - **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
+1 -1
View File
@@ -120,7 +120,7 @@ func (a *Acquirer) acquire(ctx context.Context, sr store.Series) {
// Stamped after success — the reverse of the poller, which stamps before // Stamped after success — the reverse of the poller, which stamps before
// the fetch: the Reader is here, watching the Series they just created, so // the fetch: the Reader is here, watching the Series they just created, so
// a failed acquisition must leave the row due for a fast retry rather than // a failed acquisition must leave the row due for a fast retry rather than
// consuming the rest. The stamp happens even when the page read // consuming the cooldown. The stamp happens even when the page read
// succeeded but produced no facts to persist. // succeeded but produced no facts to persist.
if err := a.Store.MarkLatestChecked(sr.Site, sr.SeriesID, time.Now().UnixMilli()); err != nil { if err := a.Store.MarkLatestChecked(sr.Site, sr.SeriesID, time.Now().UnixMilli()); err != nil {
log.Printf("acquire %q: mark checked: %v", sr.Key(), err) log.Printf("acquire %q: mark checked: %v", sr.Key(), err)
+27 -62
View File
@@ -18,22 +18,18 @@ import (
// challengeTimeout bounds one navigate-and-solve. A Cloudflare managed // challengeTimeout bounds one navigate-and-solve. A Cloudflare managed
// challenge clears in a few seconds when it clears at all; anything longer is a // challenge clears in a few seconds when it clears at all; anything longer is a
// challenge that is not going to pass, and the caller's rest was already // challenge that is not going to pass, and the caller's cooldown was already
// stamped before this ran. // stamped before this ran.
const challengeTimeout = 45 * time.Second const challengeTimeout = 45 * time.Second
var kaganeSeriesRe = regexp.MustCompile(`^/series/([0-9a-f-]{36})/?$`) var kaganeSeriesRe = regexp.MustCompile(`^/series/([0-9a-f-]{36})/?$`)
// comixSeriesPathRe matches the one path shape comixRead will open: a Series
// page, "/title/<id>-<slug>". Verified live 2026-08-12.
var comixSeriesPathRe = regexp.MustCompile(`^/title/[^/?#]+/?$`)
// BrowserFetcher retrieves pages through a remote headless Chrome over the // BrowserFetcher retrieves pages through a remote headless Chrome over the
// DevTools Protocol. // DevTools Protocol.
// //
// It exists for one reason: kagane.to, novelfull.com and comix.to sit behind a // It exists for one reason: kagane.to and novelfull.com sit behind a
// Cloudflare JavaScript challenge. Verified 2026-08-03 (kagane), 2026-08-05 // Cloudflare JavaScript challenge. Verified 2026-08-03 (kagane) and 2026-08-05
// (novelfull) and 2026-08-12 (comix), plain HTTP and bogdanfinn/tls-client // (novelfull) from the deployment host, plain HTTP and bogdanfinn/tls-client
// with a Chrome_133 profile both get 403 with cf-mitigated: challenge on every // with a Chrome_133 profile both get 403 with cf-mitigated: challenge on every
// path, including the API, robots.txt and images. Clearing it requires // path, including the API, robots.txt and images. Clearing it requires
// executing the challenge script, which only a real browser does. // executing the challenge script, which only a real browser does.
@@ -44,11 +40,10 @@ var comixSeriesPathRe = regexp.MustCompile(`^/title/[^/?#]+/?$`)
// sync that break silently and separately. The browser's own cookie jar // sync that break silently and separately. The browser's own cookie jar
// persists across polls, so the challenge is solved once every few hours. // persists across polls, so the challenge is solved once every few hours.
// //
// The three sites differ in what a cleared tab is asked for: kagane fetches a // The two sites differ in how the chapter list is read: kagane serves it from
// JSON API from inside the page (the list exists nowhere else), comix fetches // a JSON API that must be called from inside the page (so the request carries
// its own Series URL from inside the page (the served HTML carries the facts, // the clearance cookie), while novelfull renders it into the HTML so the
// and rendering the SPA costs ~65 requests instead of one), and novelfull // cleared DOM is the payload.
// renders its list into the HTML so the cleared DOM is the payload.
type BrowserFetcher struct { type BrowserFetcher struct {
allocCtx context.Context allocCtx context.Context
cancel context.CancelFunc cancel context.CancelFunc
@@ -92,9 +87,10 @@ func (f *BrowserFetcher) Close() {
} }
// Get navigates to seriesURL, lets any challenge resolve, then reads the // Get navigates to seriesURL, lets any challenge resolve, then reads the
// payload the Site's registry entry describes (the shapes are listed on // payload the Site's registry entry describes — kagane's chapter-list API from
// BrowserFetcher). The returned body is whatever the Site's chapter list lives // inside the page so the request carries the clearance cookie, novelfull's
// in, which is what the entry's LatestChapter parse expects. // served HTML. The returned body is whatever the Site's chapter list lives in,
// which is what the entry's LatestChapter parse expects.
func (f *BrowserFetcher) Get(ctx context.Context, seriesURL string) (string, int, error) { func (f *BrowserFetcher) Get(ctx context.Context, seriesURL string) (string, int, error) {
var body string var body string
// Sorted order (browserBackedSites sorts) makes dispatch deterministic: // Sorted order (browserBackedSites sorts) makes dispatch deterministic:
@@ -145,47 +141,29 @@ func novelfullRead(seriesURL string, out *string) (chromedp.Action, bool) {
return chromedp.OuterHTML("html", out, chromedp.ByQuery), true return chromedp.OuterHTML("html", out, chromedp.ByQuery), true
} }
// comixRead fetches the Series page from inside the cleared tab. comix is an
// SPA: rendering the page costs ~65 requests, while one same-origin fetch of
// the same address returns the server-rendered HTML — 24.5 KB, ~480 ms,
// carrying both parser anchors (measured 2026-08-12, issue #98). So this is
// kaganeRead's shape, not novelfullRead's, even though the payload is HTML.
// Refusing any other address is the per-Site half of the SSRF gate.
func comixRead(seriesURL string, out *string) (chromedp.Action, bool) {
pageURL, ok := comixSeriesPageURL(seriesURL)
if !ok {
return nil, false
}
return chromedp.Evaluate(
`fetch(`+jsString(pageURL)+`).then(r => r.ok ? r.text() : "")`,
out, awaitPromise), true
}
// Image retrieves one cover's bytes through the browser sidecar, and its // Image retrieves one cover's bytes through the browser sidecar, and its
// content type. // content type.
// //
// It exists because kagane and comix serve covers behind the same challenge as // It exists because kagane serves covers behind the same challenge as its
// their pages — kagane additionally with // pages *and* with `cross-origin-resource-policy: same-origin`, so the bytes
// `cross-origin-resource-policy: same-origin` — so the bytes are only // are only reachable from inside a browser that already holds the clearance
// reachable from inside a browser that already holds the clearance cookie // cookie (verified 2026-08-08). Acquisition through the sidecar is the only
// (verified 2026-08-08 for kagane, 2026-08-12 for comix). Acquisition through // route.
// the sidecar is the only route.
// //
// The image URL is navigated to rather than fetched from another page of the // The image URL is navigated to rather than fetched from some other kagane
// Site: the challenge only runs on a top-level navigation, and once it clears // page: the challenge only runs on a top-level navigation, and once it clears
// the document *is* the image, so a same-origin fetch of location.href reads // the document *is* the image, so a same-origin fetch of location.href reads
// it straight back out of the cache. For comix the navigation is also the only // it straight back out of the cache.
// route that works at all — its Series page sets
// `cross-origin-embedder-policy: require-corp`, which fails a page-context
// fetch of the cover host.
// //
// The challenge is not solved by the first read: WaitReady("body") is satisfied // The challenge is not solved by the first read: WaitReady("body") is satisfied
// by the interstitial too. run holds the tab open until the in-page fetch // by the interstitial too. run holds the tab open until the in-page fetch
// succeeds, which is what gives the challenge script the seconds it needs. // succeeds, which is what gives the challenge script the seconds it needs.
func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, string, error) { func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, string, error) {
if !browserOnlyCoverURL(imageURL) { m := kaganeImageURLRe.FindStringSubmatch(imageURL)
if m == nil {
return nil, "", fmt.Errorf("not a browser-fetchable cover url: %q", imageURL) return nil, "", fmt.Errorf("not a browser-fetchable cover url: %q", imageURL)
} }
imageID := m[1]
var dataURL string var dataURL string
err := f.run(ctx, imageURL, err := f.run(ctx, imageURL,
chromedp.Evaluate(`fetch(location.href).then(r => r.ok chromedp.Evaluate(`fetch(location.href).then(r => r.ok
@@ -197,23 +175,23 @@ func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, st
: "")`, &dataURL, awaitPromise), : "")`, &dataURL, awaitPromise),
func() bool { return dataURL != "" }) func() bool { return dataURL != "" })
if err != nil { if err != nil {
return nil, "", fmt.Errorf("browser image %s: %w", imageURL, err) return nil, "", fmt.Errorf("browser image %s: %w", imageID, err)
} }
// "data:image/webp;base64,<payload>". // "data:image/webp;base64,<payload>".
head, payload, ok := strings.Cut(dataURL, ";base64,") head, payload, ok := strings.Cut(dataURL, ";base64,")
if !ok { if !ok {
return nil, "", fmt.Errorf("browser image %s: not a data url", imageURL) return nil, "", fmt.Errorf("browser image %s: not a data url", imageID)
} }
raw, err := base64.StdEncoding.DecodeString(payload) raw, err := base64.StdEncoding.DecodeString(payload)
if err != nil { if err != nil {
return nil, "", fmt.Errorf("browser image %s: %w", imageURL, err) return nil, "", fmt.Errorf("browser image %s: %w", imageID, err)
} }
return raw, strings.TrimPrefix(head, "data:"), nil return raw, strings.TrimPrefix(head, "data:"), nil
} }
// errChallengeHeld reports that the budget ran out with the interstitial still // errChallengeHeld reports that the budget ran out with the interstitial still
// up. Distinct from a transport failure: it means "this site said no", which // up. Distinct from a transport failure: it means "this site said no", which
// the poller answers with a refusal backoff for that Site's Lane (issue #100). // the poller answers with a 403 and its ordinary cooldown.
var errChallengeHeld = errors.New("challenge held") var errChallengeHeld = errors.New("challenge held")
// errBrowserInterrupted distinguishes a remote Chrome restart from the // errBrowserInterrupted distinguishes a remote Chrome restart from the
@@ -345,19 +323,6 @@ func novelfullSeriesURL(seriesURL string) bool {
strings.HasSuffix(u.Path, ".html") strings.HasSuffix(u.Path, ".html")
} }
// comixSeriesPageURL returns the address comixRead fetches inside the tab: the
// Series page itself, rebuilt from the pinned host and path so nothing else
// travels. Host-pinned here for the same reason kagane's is — series_url is
// client-supplied and a headless browser is a strong SSRF primitive.
func comixSeriesPageURL(seriesURL string) (string, bool) {
u, err := url.Parse(seriesURL)
if err != nil || u.Scheme != "https" || u.Hostname() != "comix.to" ||
!comixSeriesPathRe.MatchString(u.Path) {
return "", false
}
return "https://comix.to" + u.Path, true
}
// awaitPromise makes Evaluate resolve the promise rather than returning a // awaitPromise makes Evaluate resolve the promise rather than returning a
// serialised Promise object. // serialised Promise object.
func awaitPromise(p *runtime.EvaluateParams) *runtime.EvaluateParams { func awaitPromise(p *runtime.EvaluateParams) *runtime.EvaluateParams {
-56
View File
@@ -61,62 +61,6 @@ func TestNovelfullSeriesURL(t *testing.T) {
}) })
} }
} }
func TestComixSeriesPageURL(t *testing.T) {
const series = "https://comix.to/title/n8we-dungeons-and-crayons"
cases := []struct {
name string
url string
want string
}{
{"series page", series, series},
{"trailing slash kept", series + "/", series + "/"},
// Query and fragment are dropped: only the pinned path travels.
{"query dropped", series + "?tab=chapters", series},
{"foreign host", "https://evil.example/title/x", ""},
{"lookalike host", "https://comix.to.evil.example/title/x", ""},
{"not https", "http://comix.to/title/x", ""},
{"not a series path", "https://comix.to/search", ""},
{"chapter page", series + "/11139891-chapter-80", ""},
{"garbage", "://nope", ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, ok := comixSeriesPageURL(tc.url)
if ok != (tc.want != "") || got != tc.want {
t.Fatalf("comixSeriesPageURL(%q) = %q, %v; want %q", tc.url, got, ok, tc.want)
}
})
}
}
// The browser is an SSRF primitive and a cover address can originate in a
// client-supplied PUT body, so this gate decides what it may navigate to.
func TestBrowserOnlyCoverURL(t *testing.T) {
cases := []struct {
url string
want bool
}{
{"https://static.comix.to/039d/i/1/34/6a6742bf15736@280.jpg", true},
{"https://kagane.to/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed", true},
// Every other Site's CDN answers plain TLS.
{"https://gg.asuracomic.net/covers/x.webp", false},
{"http://static.comix.to/039d/x.jpg", false},
{"https://static.comix.to.evil.example/039d/x.jpg", false},
{"https://evil.example/static.comix.to/x.jpg", false},
{"https://static.comix.to/039d/x.jpg?next=http://169.254.169.254/", false},
{"https://static.comix.to/039d/x.svg", false},
{"https://static.comix.to/../etc/passwd.jpg", false},
{"https://static.comix.to/", false},
}
for _, tc := range cases {
t.Run(tc.url, func(t *testing.T) {
if got := browserOnlyCoverURL(tc.url); got != tc.want {
t.Fatalf("browserOnlyCoverURL(%q) = %v, want %v", tc.url, got, tc.want)
}
})
}
}
func TestClassifyBrowserInterruption(t *testing.T) { func TestClassifyBrowserInterruption(t *testing.T) {
if err := classifyBrowserError(context.Background(), true, context.Canceled); !errors.Is(err, errBrowserInterrupted) { if err := classifyBrowserError(context.Background(), true, context.Canceled); !errors.Is(err, errBrowserInterrupted) {
t.Fatalf("classifyBrowserError(context.Canceled) = %v, want browser interruption", err) t.Fatalf("classifyBrowserError(context.Canceled) = %v, want browser interruption", err)
+1 -2
View File
@@ -25,8 +25,7 @@ type CoverBytesFetcher interface {
// fetchCoverBytes routes a cover's byte retrieval by URL shape, not by Site // fetchCoverBytes routes a cover's byte retrieval by URL shape, not by Site
// name: the browser fetcher's module claims the addresses only it can fetch // name: the browser fetcher's module claims the addresses only it can fetch
// (kagane's image route answers a plain fetch with a challenge and // (kagane's image route answers a plain fetch with a challenge and
// `cross-origin-resource-policy: same-origin`, static.comix.to answers one with // `cross-origin-resource-policy: same-origin`), and everything else goes over
// the same challenge its pages serve), and everything else goes over
// plain TLS. Missing fetchers degrade to an error the caller logs, never a // plain TLS. Missing fetchers degrade to an error the caller logs, never a
// fallback onto a path that cannot succeed. One routing rule for the poll and // fallback onto a path that cannot succeed. One routing rule for the poll and
// the acquirer, so the two cannot drift apart. // the acquirer, so the two cannot drift apart.
+70 -281
View File
@@ -5,8 +5,6 @@ import (
"errors" "errors"
"log" "log"
"net/url" "net/url"
"sort"
"sync"
"time" "time"
"bookmarkmanager/backend/internal/store" "bookmarkmanager/backend/internal/store"
@@ -19,8 +17,8 @@ type Fetcher interface {
} }
// BrowserCoverFetcher retrieves one cover's bytes through the browser-backed // BrowserCoverFetcher retrieves one cover's bytes through the browser-backed
// path — the only route that clears the challenge kagane's and comix's image // path — the only route that clears the challenge kagane's image URLs answer
// URLs answer a plain fetch with. Satisfied by BrowserFetcher. // a plain fetch with. Satisfied by BrowserFetcher.
type BrowserCoverFetcher interface { type BrowserCoverFetcher interface {
Image(ctx context.Context, imageURL string) (body []byte, contentType string, err error) Image(ctx context.Context, imageURL string) (body []byte, contentType string, err error)
} }
@@ -30,11 +28,15 @@ type BrowserCoverFetcher interface {
// in parallel and report the same observable fact, so whichever writes last wins // in parallel and report the same observable fact, so whichever writes last wins
// and neither needs to know about the other. // and neither needs to know about the other.
// //
// Every Site gets its own Poll Lane: one independent stream of Polls with its // Two clocks, deliberately independent:
// own pace, running concurrently with every other Site's (issue #100). Rest //
// time and gap live in the Site registry, not here — see sites.go. Rest is // - Interval is how often this goroutine wakes up and looks.
// enforced by the WHERE clause in DueForLatestCheck rather than by any timer; // - Cooldowns are how long a series rests since its own last check. Browser-
// the gap is enforced by the Lane sleeping between fetches. // backed sites use the longer BrowserCooldown.
//
// Cooldowns are enforced by the WHERE clause in DueForLatestCheck rather than
// by any timer. Shortening Interval therefore cannot shorten anyone's cooldown;
// it only makes the poller wake up and find nothing due more often.
type Poller struct { type Poller struct {
Store *store.Store Store *store.Store
Fetch Fetcher Fetch Fetcher
@@ -48,19 +50,11 @@ type Poller struct {
// same failure-isolated prefetch path. // same failure-isolated prefetch path.
CoverBytesFetch CoverBytesFetcher CoverBytesFetch CoverBytesFetcher
Now func() time.Time // injected so tests can freeze it Now func() time.Time // injected so tests can freeze it
Cooldown time.Duration
// refuseUntil gates a Site's Lane after it refused twice in one run: no BrowserCooldown time.Duration
// Series of that Site is attempted again before this time (issue #100). Interval time.Duration
// browserDownAt is when a browser Lane last lost the sidecar; the other Stagger time.Duration
// browser Lanes skip their passes for the next refuseBackoff, so a Batch int
// restarting Chrome does not stamp one Series per pass per Lane (story 20).
mu sync.Mutex
refuseUntil map[string]time.Time
browserDownAt time.Time
// coverWG tracks in-flight cover work. Covers heal in the background so a
// slow cover host cannot delay the next Series-page Poll; tests join it
// before asserting on cover fetches.
coverWG sync.WaitGroup
} }
// fillBlankCover gives a Series its Cover when it has none. The blank state is // fillBlankCover gives a Series its Cover when it has none. The blank state is
@@ -82,14 +76,7 @@ func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, cover stri
if cover == "" { if cover == "" {
return return
} }
// Like healCover, the fill runs in the background: a large import of
// blanks would otherwise pay one og:image fetch per Series against the
// Lane's gap (issue #100, story 12).
p.coverWG.Add(1)
go func() {
defer p.coverWG.Done()
p.storeCover(ctx, sr, cover) p.storeCover(ctx, sr, cover)
}()
} }
// prefetchCover heals Series that already carry a third-party source URL but // prefetchCover heals Series that already carry a third-party source URL but
@@ -132,9 +119,9 @@ func (p *Poller) storeCover(ctx context.Context, sr store.Series, sourceURL stri
// fetcherFor returns the fetcher a site's page needs, or nil when the site // fetcherFor returns the fetcher a site's page needs, or nil when the site
// cannot be fetched at all right now. A Site whose registry entry carries a // cannot be fetched at all right now. A Site whose registry entry carries a
// Browser read — kagane, comix and novelfull, all behind a Cloudflare // Browser read — kagane and novelfull, both behind a Cloudflare JavaScript
// JavaScript challenge no TLS fingerprint clears — prefers the browser; when it // challenge no TLS fingerprint clears — prefers the browser; when it is
// is absent, the entry's Fallback decides whether plain TLS may take over. One // absent, the entry's Fallback decides whether plain TLS may take over. One
// routing rule for the poll and the acquirer, so the two cannot drift apart. // routing rule for the poll and the acquirer, so the two cannot drift apart.
func fetcherFor(site string, browser, tls Fetcher) Fetcher { func fetcherFor(site string, browser, tls Fetcher) Fetcher {
s, known := sites[site] s, known := sites[site]
@@ -156,248 +143,72 @@ func fetcherFor(site string, browser, tls Fetcher) Fetcher {
return nil return nil
} }
// Run polls until ctx is cancelled: one goroutine per Site Lane, each pacing // Run polls until ctx is cancelled.
// itself by the Site's effective gap. Lanes share nothing but the store and //
// the browser fetcher's single tab (BrowserFetcher serializes itself), so one // runOnce is called synchronously, so a batch that overruns the tick delays the
// hostile Site burns only its own budget. // next one instead of stacking a second batch on top of it. That is the intended
// laneNames returns every registry Site in the deterministic order both Run // failure mode for a misconfigured batch x stagger: a slower cadence, never
// and runOnce iterate: sorted, so lane behaviour and its tests agree on who // concurrent fetch storms.
// runs first.
func laneNames() []string {
names := make([]string, 0, len(sites))
for name := range sites {
names = append(names, name)
}
sort.Strings(names)
return names
}
func (p *Poller) Run(ctx context.Context) { func (p *Poller) Run(ctx context.Context) {
names := laneNames() log.Printf("latest-chapter poller: interval=%s cooldown=%s browser-cooldown=%s batch=%d stagger=%s",
log.Printf("latest-chapter poller: %d lanes, rest=%s gap=%s", len(names), defaultRest, defaultGap) p.Interval, p.Cooldown, p.BrowserCooldown, p.Batch, p.Stagger)
for _, name := range names { t := time.NewTicker(p.Interval)
go p.lane(ctx, name) defer t.Stop()
}
<-ctx.Done()
log.Println("latest-chapter poller: stopped")
}
// lane is one Site's Poll Lane: one pass, then sleep the pace the pass
// reported, then another pass, until ctx is cancelled. The sleep is the whole
// pace discipline — a pass that fetched nothing still reports its gap so the
// Lane wakes often enough to notice Series as they become due. The pass shares
// the Poller's browser-down state, so a sidecar loss is noticed once and the
// other browser Lanes skip passes until the backoff window decays.
func (p *Poller) lane(ctx context.Context, name string) {
for { for {
pace := p.runLanePass(ctx, name, true)
if ctx.Err() != nil {
return
}
select { select {
case <-ctx.Done(): case <-ctx.Done():
log.Println("latest-chapter poller: stopped")
return return
case <-time.After(pace): case <-t.C:
p.runOnce(ctx)
} }
} }
} }
// runOnce processes one round: one pass of every Lane, back to back, no real // runOnce processes one batch of due series.
// time passing. This is the deterministic entry point the test suite drives a
// round at a time. The production Run loop does the same work paced by its own
// sleeps; pacing is the only difference.
func (p *Poller) runOnce(ctx context.Context) { func (p *Poller) runOnce(ctx context.Context) {
for _, name := range laneNames() {
p.runLanePass(ctx, name, false)
}
}
// runLanePass processes one pass of one Site's Lane: select the due Series,
// pace through them, and report how long the Lane should wait before its next
// pass. paced spaces consecutive fetches by the Site's effective gap — the
// production Lane's rate limit; the deterministic test entry runs back to back.
func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration {
now := p.Now() now := p.Now()
if until := p.refusalBackoff(name); now.Before(until) { cutoff := now.Add(-p.Cooldown).UnixMilli()
// Cooling down after a refusal: do not attempt this Site at all. browserCutoff := now.Add(-p.BrowserCooldown).UnixMilli()
return until.Sub(now) due, err := p.Store.DueForLatestCheck(cutoff, browserCutoff, browserBackedSites(), p.Batch)
}
if isBrowserSite(name) {
if downFor, down := p.browserDownFor(now); down && downFor < refuseBackoff {
// A sibling browser Lane lost the sidecar within the backoff
// window: skip this pass, so a restarting Chrome does not stamp
// this Site's Series one pass at a time. After refuseBackoff the
// flag decays and the Lane probes again (issue #100, story 20).
log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor)
return refuseBackoff - downFor
}
}
s := sites[name]
f := fetcherFor(name, p.BrowserFetch, p.Fetch)
if f == nil {
// No fetcher at all right now (browser absent, no fallback): every
// Series stays unstamped and due, so a browser that appears after a
// restart finds its full queue waiting (issue #100).
return defaultGap
}
due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli())
if err != nil { if err != nil {
log.Printf("latest poll %s: due query: %v", name, err) log.Printf("latest poll: due query: %v", err)
return defaultGap return
}
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) {
// Below both thresholds Chrome stays asleep (ADR-0005 on-demand
// browser): waking it for a single Poll would cost a challenge solve
// per request.
return defaultGap
}
if s.Browser != nil {
// Browser Lanes share one tab, so their combined ceiling is about 360
// Polls an hour. When they cannot keep up, the wait past the rest time
// grows — log by how much, every pass, so the decision to give them
// more pages is made from a measurement rather than a guess.
if behind := maxSeriesWait(due, now, s.Rest) - s.Rest; behind > 0 {
log.Printf("latest poll %s: browser lane behind by %s (browser Sites cannot keep up with the hour)", name, behind)
}
} }
eligible, err := p.Store.EligibleSeriesCount(name)
if err != nil {
log.Printf("latest poll %s: eligible count: %v", name, err)
return defaultGap
}
gap, clamped := effectiveGap(s, eligible)
if clamped {
log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible)
}
if eligible == 0 {
// Nothing to poll for the foreseeable future; sleep a full rest instead
// of re-querying every gap.
return s.Rest
}
refusals := 0
checked := 0 checked := 0
for i, sr := range due { for i, sr := range due {
if ctx.Err() != nil { if ctx.Err() != nil {
break break
} }
if refusals >= 2 { // Staggered rather than fired together: a burst of simultaneous requests
// This Site refused twice in a row: the remaining Series are left // from one server IP is the traffic shape most likely to move that IP's
// unstamped and due, and the Lane waits refuseBackoff before // bot score. This is the server-side analogue of the userscript's "one
// trying it again. // series per navigation ... indistinguishable from browsing" (L455-456).
break stopped := false
} if i > 0 && p.Stagger > 0 {
if paced && i > 0 {
select { select {
case <-ctx.Done(): case <-ctx.Done():
break stopped = true
case <-time.After(gap): case <-time.After(p.Stagger):
} }
if ctx.Err() != nil { }
if stopped {
break break
} }
} p.checkOne(ctx, sr)
if err := p.checkOne(ctx, sr); err != nil {
switch {
case errors.Is(err, errChallengeHeld):
refusals++
case errors.Is(err, errBrowserInterrupted):
p.setBrowserDown(now)
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, refuseBackoff)
return gap
default:
refusals = 0
}
} else {
refusals = 0
}
checked++ checked++
} }
if checked > 0 { // due vs checked is how you tell which constraint is binding: ticks that
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), checked) // report due=0 mean the cooldown is the limit, ticks that report due==batch
} // every time mean throughput is.
if refusals >= 2 { log.Printf("latest poll: due=%d checked=%d", len(due), checked)
p.setRefusalBackoff(name, now.Add(refuseBackoff))
log.Printf("latest poll %s: refused twice this run, waiting %s", name, refuseBackoff)
return refuseBackoff
}
return gap
}
func (p *Poller) refusalBackoff(name string) time.Time {
p.mu.Lock()
defer p.mu.Unlock()
return p.refuseUntil[name]
}
func (p *Poller) setRefusalBackoff(name string, until time.Time) {
p.mu.Lock()
defer p.mu.Unlock()
if p.refuseUntil == nil {
p.refuseUntil = make(map[string]time.Time)
}
p.refuseUntil[name] = until
}
// setBrowserDown records when a browser Lane lost the sidecar. It is Poller
// state rather than pass state so the other browser Lanes see it too.
func (p *Poller) setBrowserDown(now time.Time) {
p.mu.Lock()
p.browserDownAt = now
p.mu.Unlock()
}
// browserDownFor reports how long the sidecar has been down and that it is
// down at all — the zero time means never down, which must not read as a
// zero-duration loss. The window decays: once refuseBackoff passes without a
// fresh loss, Lanes probe again.
func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) {
p.mu.Lock()
defer p.mu.Unlock()
if p.browserDownAt.IsZero() {
return 0, false
}
return now.Sub(p.browserDownAt), true
}
// isBrowserSite reports whether the registry routes this Site's page through
// the browser sidecar.
func isBrowserSite(name string) bool {
return sites[name].Browser != nil
}
// browserWakeDue reports whether a browser Lane may start a run: five or more
// of its Series are due, or any one of them has been due for browserWakeAge.
// Below both thresholds the Lane leaves Chrome asleep — Series Polled together
// become due together, so the group naturally stays clustered, and the age
// rule exists to stop a Series that drifted out of the group from starving.
func browserWakeDue(due []store.Series, now time.Time, rest time.Duration) bool {
if len(due) >= browserWakeCount {
return true
}
return maxSeriesWait(due, now, rest) >= browserWakeAge
}
// maxSeriesWait returns how long the most-overdue of the due Series has been
// waiting past its due moment (0 when due is empty).
func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration {
var oldest time.Duration
for _, sr := range due {
if w := now.Sub(time.UnixMilli(sr.LatestCheckedAt).Add(rest)); w > oldest {
oldest = w
}
}
return oldest
} }
// checkOne re-checks one series. Every failure path here is "log and move on": // checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a // the poller is a best-effort enhancement, and no single bad series may stall a
// Lane or take down the process. The returned error is the page read's // batch or take down the process.
// classified outcome so the Lane can tell a refusal from a loss of the func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
// browser; non-classified failures still return nil-equivalent behaviour.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
defer func() { defer func() {
if r := recover(); r != nil { if r := recover(); r != nil {
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r) log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
@@ -405,46 +216,44 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
}() }()
// Stamped before the fetch, not after, so an error, a timeout, or a shutdown // Stamped before the fetch, not after, so an error, a timeout, or a shutdown
// mid-request still consumes the rest. Otherwise a renamed or deleted // mid-request still consumes the cooldown. Otherwise a renamed or deleted
// series would be retried on every single pass forever. The userscript // series would be retried on every single tick forever. The userscript
// stamps in the same order and for the same reason (L471-473). A Series // stamps in the same order and for the same reason (L471-473).
// never reaches checkOne without a fetcher — runLanePass skips those — so
// the stamp means "attempted", and an untried Series stays due.
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil { if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err) log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
return nil return
} }
facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch) facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch)
if err != nil { if err != nil {
switch { switch {
case errors.Is(err, errNotFetchable): case errors.Is(err, errNotFetchable):
// The rest above is already consumed, so a row that never // The cooldown above is already consumed, so a row that never
// passes the gate is retried at rest pace rather than // passes the gate is retried at cooldown pace rather than
// hot-looping. // hot-looping.
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL) log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
return err return
case errors.Is(err, errNoFetcher): case errors.Is(err, errNoFetcher):
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site) log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
return err return
} }
// A legacy cover heals independently of the page read: its source may // A legacy cover heals independently of the page read: its source may
// answer — a CDN — while the origin does not, so a fetch failure does // answer — a CDN — while the origin does not, so a fetch failure does
// not skip the heal, matching the order the shared read replaced. // not skip the heal, matching the order the shared read replaced.
p.healCover(ctx, sr) p.prefetchCover(ctx, sr)
log.Printf("latest poll %q: %v", sr.Key(), err) log.Printf("latest poll %q: %v", sr.Key(), err)
return err return
} }
// A legacy cover source is healed independently of the page read. // A legacy cover source is healed independently of the page read.
p.healCover(ctx, sr) p.prefetchCover(ctx, sr)
// Cover fill is independent of the chapter signal: a page that lost its // Cover fill is independent of the chapter signal: a page that lost its
// chapter list may keep its og:image, and a blank Series heals either way. // chapter list may keep its og:image, and a blank Series heals either way.
p.fillBlankCover(ctx, sr, facts.Cover) p.fillBlankCover(ctx, sr, facts.Cover)
if !facts.HasLatest { if !facts.HasLatest {
// Most likely a challenge page or a layout change. Either way the row is // Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a rest instead of hot-looping. // already stamped, so this waits out a cooldown instead of hot-looping.
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen) log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen)
return nil return
} }
// Equality, not >, mirroring the userscript (L427): a site that retracts a // Equality, not >, mirroring the userscript (L427): a site that retracts a
@@ -452,7 +261,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// against the due-query snapshot; a concurrent write in between only costs // against the due-query snapshot; a concurrent write in between only costs
// one redundant UPDATE of the same absolute value, never a wrong one. // one redundant UPDATE of the same absolute value, never a wrong one.
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num { if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num {
return nil return
} }
// Series-level write: the row is shared, so one update refreshes every // Series-level write: the row is shared, so one update refreshes every
@@ -461,29 +270,9 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// the list. // the list.
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil { if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil {
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err) log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
return nil return
} }
log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label) log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label)
return nil
}
// healCover runs prefetchCover in the background. Cover bytes come from a
// different host — often a CDN — and heal once in a Series's life, so they
// must not consume a Lane's gap: a large import with many blanks would
// otherwise make every Latest Chapter go stale behind a slow image host
// (issue #100).
func (p *Poller) healCover(ctx context.Context, sr store.Series) {
p.coverWG.Add(1)
go func() {
defer p.coverWG.Done()
p.prefetchCover(ctx, sr)
}()
}
// waitCovers blocks until every in-flight cover heal finishes. Tests call it
// after a round before asserting on cover fetches.
func (p *Poller) waitCovers() {
p.coverWG.Wait()
} }
// fetchableSeriesURL reports whether site is a Site the registry knows and // fetchableSeriesURL reports whether site is a Site the registry knows and
+98 -469
View File
@@ -5,7 +5,6 @@ import (
"crypto/sha256" "crypto/sha256"
"database/sql" "database/sql"
"errors" "errors"
"fmt"
"log" "log"
"os" "os"
"strings" "strings"
@@ -157,16 +156,19 @@ func (f *fakeBytesCoverFetcher) callCount() int {
return len(f.calls) return len(f.calls)
} }
// newTestPoller wires a poller with a frozen clock. Rest and gap come from the // newTestPoller wires a poller with a frozen clock and no stagger, so tests run
// Site registry, so tests seed checked_at relative to the one-hour rest; the // instantly and deterministically.
// round entry point (runOnce) runs every Lane back to back with no real
// pacing, so tests stay instant and deterministic.
func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Poller { func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Poller {
t.Helper() t.Helper()
return &Poller{ return &Poller{
Store: s, Store: s,
Fetch: f, Fetch: f,
Now: func() time.Time { return at }, Now: func() time.Time { return at },
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
Interval: 10 * time.Minute,
Stagger: 0,
Batch: 14,
} }
} }
@@ -206,10 +208,9 @@ func TestRunOncePrefetchesPublicCover(t *testing.T) {
covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"} covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}
p := &Poller{ p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverBytesFetch: covers, Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverBytesFetch: covers,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if got := covers.callCount(); got != 1 { if got := covers.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1", got) t.Fatalf("cover fetch calls = %d, want 1", got)
@@ -240,10 +241,9 @@ func TestRunOnceDoesNotStoreNonImagePublicCover(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("challenge"), contentType: "text/html"}, CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("challenge"), contentType: "text/html"},
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.GetCover(coverURL); err != nil || found { if _, _, found, err := s.GetCover(coverURL); err != nil || found {
t.Fatalf("non-image cover = found %v, err %v; want missing", found, err) t.Fatalf("non-image cover = found %v, err %v; want missing", found, err)
@@ -350,6 +350,23 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
} }
} }
func TestRunOnceRespectsBatchLimit(t *testing.T) {
s, _ := newTestStore(t)
for i := 0; i < 20; i++ {
key := "asura:s" + string(rune('a'+i))
seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0)
}
f := &fakeFetcher{body: "", status: 200}
p := newTestPoller(t, s, f, time.UnixMilli(5_000_000))
p.Batch = 5
p.runOnce(context.Background())
if got := f.callCount(); got != 5 {
t.Fatalf("fetched %d series, want 5 (batch limit)", got)
}
}
// The point of the split (ADR-0003): a series referenced by several bookmarks // The point of the split (ADR-0003): a series referenced by several bookmarks
// is fetched once per due cycle, not once per bookmark. Two bookmarks share a // is fetched once per due cycle, not once per bookmark. Two bookmarks share a
// series when two readers track it (issue #22). // series when two readers track it (issue #22).
@@ -392,8 +409,8 @@ func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
} }
} }
// One unreachable series must not abandon the rest of the Lane. // One unreachable series must not abandon the rest of the batch.
func TestRunOnceOneBadSeriesDoesNotStallLane(t *testing.T) { func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
s, _ := newTestStore(t) s, _ := newTestStore(t)
keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"} keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"}
for _, k := range keys { for _, k := range keys {
@@ -419,9 +436,7 @@ func TestRunOnceOneBadSeriesDoesNotStallLane(t *testing.T) {
} }
} }
// Pace now lives in the registry, not config: Run logs the lane defaults so a func TestRunLogsCooldowns(t *testing.T) {
// deployment can see what the poller is doing without reading the source.
func TestRunLogsLaneDefaults(t *testing.T) {
var logs strings.Builder var logs strings.Builder
previous := log.Writer() previous := log.Writer()
log.SetOutput(&logs) log.SetOutput(&logs)
@@ -429,19 +444,21 @@ func TestRunLogsLaneDefaults(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
(&Poller{Now: func() time.Time { return time.Now() }}).Run(ctx) (&Poller{
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
Interval: time.Hour,
}).Run(ctx)
got := logs.String() if got := logs.String(); !strings.Contains(got, "cooldown=1h") ||
for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} { !strings.Contains(got, "browser-cooldown=6h") {
if !strings.Contains(got, want) { t.Fatalf("startup log = %q, want both cooldowns", got)
t.Fatalf("startup log = %q, want %q", got, want)
}
} }
} }
// The rest is enforced by the due query, so a second immediate pass must do // The cooldown is enforced by the due query, so a second immediate pass must do
// nothing at all — this is what makes the Lane's sleep independent of it. // nothing at all — this is what makes the tick interval independent of it.
func TestRunOnceHonoursRestAcrossPasses(t *testing.T) { func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
s, _ := newTestStore(t) s, _ := newTestStore(t)
const url = "https://asurascans.com/comics/x" const url = "https://asurascans.com/comics/x"
seedForCheck(t, s, "asura:x", url, 0) seedForCheck(t, s, "asura:x", url, 0)
@@ -454,18 +471,56 @@ func TestRunOnceHonoursRestAcrossPasses(t *testing.T) {
if got := f.callCount(); got != 1 { if got := f.callCount(); got != 1 {
t.Fatalf("first pass fetched %d, want 1", got) t.Fatalf("first pass fetched %d, want 1", got)
} }
// Same instant, and again 59 minutes later: both inside the 1h rest. // Same instant, and again 59 minutes later: both inside the 1h cooldown.
p.runOnce(context.Background()) p.runOnce(context.Background())
p.Now = func() time.Time { return now.Add(59 * time.Minute) } p.Now = func() time.Time { return now.Add(59 * time.Minute) }
p.runOnce(context.Background()) p.runOnce(context.Background())
if got := f.callCount(); got != 1 { if got := f.callCount(); got != 1 {
t.Fatalf("fetched %d times inside the rest, want 1", got) t.Fatalf("fetched %d times inside the cooldown, want 1", got)
} }
// Past the rest, it is due again. // Past the cooldown, it is due again.
p.Now = func() time.Time { return now.Add(61 * time.Minute) } p.Now = func() time.Time { return now.Add(61 * time.Minute) }
p.runOnce(context.Background()) p.runOnce(context.Background())
if got := f.callCount(); got != 2 { if got := f.callCount(); got != 2 {
t.Fatalf("fetched %d times after the rest, want 2", got) t.Fatalf("fetched %d times after the cooldown, want 2", got)
}
}
func TestRunOnceUsesBrowserCooldown(t *testing.T) {
s, _ := newTestStore(t)
const browserKey = "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
seedForCheck(t, s, "asura:plain", "https://asurascans.com/comics/plain", 0)
seedForCheck(t, s, browserKey, "https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", 0)
hour := time.Hour
now := time.Unix(2*int64(hour/time.Second), 0)
tls := &fakeFetcher{status: 200}
browser := &fakeFetcher{status: 200}
p := &Poller{
Store: s,
Fetch: tls,
BrowserFetch: browser,
Now: func() time.Time { return now },
Cooldown: hour,
BrowserCooldown: 6 * hour,
Batch: 10,
}
p.runOnce(context.Background())
if got := tls.callCount(); got != 1 {
t.Fatalf("plain-TLS fetches after 2h = %d, want 1", got)
}
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches after 2h = %d, want 0", got)
}
now = time.Unix(7*int64(hour/time.Second), 0)
p.runOnce(context.Background())
if got := tls.callCount(); got != 2 {
t.Fatalf("plain-TLS fetches after 7h = %d, want 2", got)
}
if got := browser.callCount(); got != 1 {
t.Fatalf("browser fetches after 7h = %d, want 1", got)
} }
} }
@@ -613,15 +668,14 @@ func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
Store: s, Store: s,
Fetch: f, Fetch: f,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
if len(f.calls) != 0 { if len(f.calls) != 0 {
t.Errorf("TLS fetcher was called for kagane: %v", f.calls) t.Errorf("TLS fetcher was called for kagane: %v", f.calls)
} }
if got := readLatestCheckedAt(t, s, "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"); got != 0 {
t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got)
}
} }
// novelfull without a browser is not skipped outright: its challenge is a // novelfull without a browser is not skipped outright: its challenge is a
@@ -646,9 +700,10 @@ func TestNovelfullUsesTLSWhenNoBrowserFetcher(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: tlsF, CoverBytesFetch: covers, Store: s, Fetch: tlsF, CoverBytesFetch: covers,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if len(tlsF.calls) != 1 { if len(tlsF.calls) != 1 {
t.Fatalf("TLS fetcher calls = %d, want 1", len(tlsF.calls)) t.Fatalf("TLS fetcher calls = %d, want 1", len(tlsF.calls))
@@ -687,6 +742,8 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: tlsF, BrowserFetch: browserF, Store: s, Fetch: tlsF, BrowserFetch: browserF,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour,
Interval: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
@@ -705,86 +762,6 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
} }
} }
// comix joined kagane behind the challenge on 2026-08-12 (#98): its page goes
// to the browser, its Cover bytes go through the browser's image route because
// static.comix.to is gated the same way, and the TLS fetcher is never asked
// for either.
func TestComixUsesBrowserFetcher(t *testing.T) {
s, dbURL := newTestStore(t)
const (
key = "comix:n8we-dungeons-and-crayons"
seriesID = "n8we-dungeons-and-crayons"
seriesURL = "https://comix.to/title/n8we-dungeons-and-crayons"
coverURL = "https://static.comix.to/039d/i/1/34/6a6742bf15736@280.jpg"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "comix", SeriesID: seriesID, SeriesURL: seriesURL,
UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
seedCoverSource(t, dbURL, "comix", seriesID, coverURL)
tlsF := &fakeFetcher{body: "", status: 200}
browserF := &fakeFetcher{body: comixSeriesFixture, status: 200}
covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}
tlsCovers := &fakeBytesCoverFetcher{body: []byte("tls-bytes"), contentType: "image/jpeg"}
p := &Poller{
Store: s, Fetch: tlsF, BrowserFetch: browserF,
CoverFetch: covers, CoverBytesFetch: tlsCovers,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
p.waitCovers()
if len(tlsF.calls) != 0 {
t.Errorf("TLS fetcher was called for comix: %v", tlsF.calls)
}
if len(browserF.calls) != 1 {
t.Fatalf("browser fetcher calls = %v, want 1", browserF.calls)
}
if got := tlsCovers.callCount(); got != 0 {
t.Errorf("TLS cover fetches = %d, want 0: static.comix.to answers a challenge", got)
}
if got := covers.callCount(); got != 1 {
t.Fatalf("browser cover fetches = %d, want 1", got)
}
got, found, err := s.Get(s.OwnerID(), key)
if err != nil || !found {
t.Fatalf("Get: %v found=%v", err, found)
}
if got.LatestChapterNum == nil || *got.LatestChapterNum != 80 {
t.Errorf("LatestChapterNum = %v, want 80", got.LatestChapterNum)
}
}
// Without a browser, comix is skipped outright rather than handed to plain
// TLS: a plain fetch retrieves only a challenge page (measured 2026-08-12).
func TestComixSkippedWhenNoBrowserFetcher(t *testing.T) {
s, _ := newTestStore(t)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: "comix:n8we-dungeons-and-crayons", Site: "comix",
SeriesID: "n8we-dungeons-and-crayons",
SeriesURL: "https://comix.to/title/n8we-dungeons-and-crayons",
UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
f := &fakeFetcher{body: comixSeriesFixture, status: 200}
p := &Poller{
Store: s, Fetch: f,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
p.runOnce(context.Background())
if len(f.calls) != 0 {
t.Errorf("TLS fetcher was called for comix: %v", f.calls)
}
if got := readLatestCheckedAt(t, s, "comix:n8we-dungeons-and-crayons"); got != 0 {
t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got)
}
}
func TestRunOncePrefetchesKaganeCover(t *testing.T) { func TestRunOncePrefetchesKaganeCover(t *testing.T) {
s, dbURL := newTestStore(t) s, dbURL := newTestStore(t)
const ( const (
@@ -804,9 +781,9 @@ func TestRunOncePrefetchesKaganeCover(t *testing.T) {
Store: s, Fetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, Store: s, Fetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL)) body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL))
if err != nil || !ok { if err != nil || !ok {
@@ -841,13 +818,11 @@ func TestRunOnceDoesNotRefetchKaganeCover(t *testing.T) {
covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"} covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"}
p := &Poller{ p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers, Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers,
Now: func() time.Time { return at }, Now: func() time.Time { return at }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
at = at.Add(2 * time.Hour) at = at.Add(2 * time.Hour)
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if got := covers.callCount(); got != 1 { if got := covers.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1 after two due cycles", got) t.Fatalf("cover fetch calls = %d, want 1 after two due cycles", got)
@@ -872,10 +847,9 @@ func TestRunOnceCoverFailureDoesNotBlockChapter(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
CoverFetch: &fakeCoverFetcher{err: errors.New("browser unavailable")}, CoverFetch: &fakeCoverFetcher{err: errors.New("browser unavailable")},
Now: func() time.Time { return now }, Now: func() time.Time { return now }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
got, found, err := s.Get(s.OwnerID(), key) got, found, err := s.Get(s.OwnerID(), key)
if err != nil || !found { if err != nil || !found {
@@ -906,10 +880,9 @@ func TestRunOnceRejectsInvalidKaganeCover(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
CoverFetch: &fakeCoverFetcher{body: []byte("not an image"), contentType: "text/html"}, CoverFetch: &fakeCoverFetcher{body: []byte("not an image"), contentType: "text/html"},
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found { if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
t.Fatalf("invalid cover persisted = %v, err %v; want missing", found, err) t.Fatalf("invalid cover persisted = %v, err %v; want missing", found, err)
@@ -932,10 +905,9 @@ func TestRunOnceWithoutCoverFetcherStillPollsKagane(t *testing.T) {
seedCoverSource(t, dbURL, "kagane", seriesID, coverURL) seedCoverSource(t, dbURL, "kagane", seriesID, coverURL)
p := &Poller{ p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found { if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
t.Fatalf("cover after nil CoverFetch = found %v, err %v; want missing", found, err) t.Fatalf("cover after nil CoverFetch = found %v, err %v; want missing", found, err)
@@ -958,10 +930,9 @@ func TestRunOnceRoutesNonKaganeCoverToPublicFetcher(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverFetch: browserCovers, Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverFetch: browserCovers,
CoverBytesFetch: publicCovers, CoverBytesFetch: publicCovers,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if got := publicCovers.callCount(); got != 1 { if got := publicCovers.callCount(); got != 1 {
t.Fatalf("public cover fetch calls = %d, want 1", got) t.Fatalf("public cover fetch calls = %d, want 1", got)
@@ -1078,9 +1049,9 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
Store: s, Fetch: page, BrowserFetch: page, Store: s, Fetch: page, BrowserFetch: page,
CoverBytesFetch: public, CoverFetch: browser, CoverBytesFetch: public, CoverFetch: browser,
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Now: func() time.Time { return time.UnixMilli(5_000_000) },
Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
got := readBookmark(t, s, tc.key) got := readBookmark(t, s, tc.key)
wantWire := testCoverBaseURL + "/covers/" + store.CoverAddress(tc.wantCover) wantWire := testCoverBaseURL + "/covers/" + store.CoverAddress(tc.wantCover)
@@ -1130,7 +1101,7 @@ func TestRunOnceDoesNotReplaceExistingCover(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200}, Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: public, CoverBytesFetch: public,
Now: func() time.Time { return at }, Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
at = at.Add(2 * time.Hour) at = at.Add(2 * time.Hour)
@@ -1165,10 +1136,9 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200}, Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: public, CoverBytesFetch: public,
Now: func() time.Time { return at }, Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if got := readBookmark(t, s, key); got.Cover != "" { if got := readBookmark(t, s, key); got.Cover != "" {
t.Fatalf("Cover after failed fetch = %q, want blank", got.Cover) t.Fatalf("Cover after failed fetch = %q, want blank", got.Cover)
} }
@@ -1181,7 +1151,6 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
public.contentType = "image/jpeg" public.contentType = "image/jpeg"
at = at.Add(2 * time.Hour) at = at.Add(2 * time.Hour)
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
if got := public.callCount(); got != 2 { if got := public.callCount(); got != 2 {
t.Fatalf("cover fetch calls after retry = %d, want 2", got) t.Fatalf("cover fetch calls after retry = %d, want 2", got)
@@ -1215,10 +1184,9 @@ func TestRunOnceBlankCoverFailureDoesNotBlockChapter(t *testing.T) {
p := &Poller{ p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200}, Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{err: errors.New("cdn down")}, CoverBytesFetch: &fakeBytesCoverFetcher{err: errors.New("cdn down")},
Now: func() time.Time { return now }, Now: func() time.Time { return now }, Cooldown: time.Hour, Batch: 10,
} }
p.runOnce(context.Background()) p.runOnce(context.Background())
p.waitCovers()
got := readBookmark(t, s, key) got := readBookmark(t, s, key)
if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 { if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 {
@@ -1241,342 +1209,3 @@ const kaganeAPIFixtureWithCover = `
{"book_id":"b","title":"Episode 41","chapter_no":"41","sort_no":41}, {"book_id":"b","title":"Episode 41","chapter_no":"41","sort_no":41},
{"book_id":"c","title":"Episode 40.5","chapter_no":"40.5","sort_no":40}]} {"book_id":"c","title":"Episode 40.5","chapter_no":"40.5","sort_no":40}]}
` `
func TestEffectiveGap(t *testing.T) {
asura := sites["asura"]
tests := []struct {
eligible int
want time.Duration
clamped bool
}{
{0, 10 * time.Second, false},
{5, 10 * time.Second, false},
{360, 10 * time.Second, false},
{720, 5 * time.Second, false},
{3600, time.Second, false},
{4000, time.Second, true},
}
for _, tt := range tests {
got, clamped := effectiveGap(asura, tt.eligible)
if got != tt.want || clamped != tt.clamped {
t.Errorf("effectiveGap(asura, %d) = (%s, %v), want (%s, %v)",
tt.eligible, got, clamped, tt.want, tt.clamped)
}
}
}
// The due query orders by sharedness first, then age: a series two readers
// track is polled before a single-reader series that has waited far longer
// (ADR-0003, issue #100).
func TestRunOnceOrdersBySharednessThenAge(t *testing.T) {
s, url := newTestStore(t)
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))}, t.TempDir(), testCoverBaseURL)
if err != nil {
t.Fatalf("Open second reader: %v", err)
}
t.Cleanup(func() { other.Close() })
now := time.UnixMilli(5_000_000)
// popular: two readers, due for 2 minutes. loner: one reader, due for 23
// minutes. Popularity must win — the loner waited far longer.
const (
popularKey = "asura:popular"
lonerKey = "asura:loner"
popularURL = "https://asurascans.com/comics/popular"
lonerURL = "https://asurascans.com/comics/loner"
)
seedForCheck(t, s, popularKey, popularURL, now.Add(-62*time.Minute).UnixMilli())
seedForCheck(t, s, lonerKey, lonerURL, 0)
if _, err := s.Upsert(other.OwnerID(), store.Bookmark{
Key: popularKey, Site: "asura", SeriesID: "popular", UpdatedAt: 2000,
}); err != nil {
t.Fatalf("seed second reader: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, now).runOnce(context.Background())
if len(f.calls) != 2 {
t.Fatalf("fetches = %d, want 2", len(f.calls))
}
if f.calls[0] != popularURL {
t.Fatalf("first fetch = %q, want the shared series %q", f.calls[0], popularURL)
}
}
// Two refusals in one pass stop the Lane: the remaining Series stay unstamped
// and due, and the Lane backs off for refuseBackoff before trying the Site
// again. One hostile Site burns only its own Lane's budget (issue #100).
func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
const n = 4
for i := 0; i < n; i++ {
key := fmt.Sprintf("kagane:s%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0)
}
browser := &fakeFetcher{status: 403}
tls := &fakeFetcher{status: 200}
// An asura Series sits on its own Lane: it must still be polled while
// kagane's Lane burns its budget on refusals (story 4).
seedForCheck(t, s, "asura:still-polls", "https://asurascans.com/series/still-polls", 0)
p := &Poller{
Store: s, Fetch: tls, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches after first pass = %d, want 2 (refused twice)", got)
}
if got := tls.callCount(); got != 1 {
t.Fatalf("asura fetches after first pass = %d, want 1 (its Lane is independent)", got)
}
stamped := 0
for i := 0; i < n; i++ {
if readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)) == now.UnixMilli() {
stamped++
}
}
if stamped != 2 {
t.Fatalf("stamped series = %d, want 2; the remaining two stay due", stamped)
}
// Inside the backoff window nothing is attempted.
p.Now = func() time.Time { return now.Add(14 * time.Minute) }
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches inside backoff = %d, want still 2", got)
}
// Past the backoff the Lane resumes and the two untried Series are polled.
p.Now = func() time.Time { return now.Add(16 * time.Minute) }
p.runOnce(context.Background())
if got := browser.callCount(); got != 4 {
t.Fatalf("browser fetches after backoff = %d, want 4", got)
}
for i := 0; i < n; i++ {
if got := readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)); got == 0 {
t.Fatalf("kagane:s%d still untried after backoff", i)
}
}
}
// Below five due Series with none waiting long, a browser Lane leaves Chrome
// asleep; five due, or one waiting browserWakeAge, wakes it (ADR-0005).
func TestBrowserLaneWakeThresholds(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
// Freshly due: checked two minutes before the rest elapses, so the wait
// is far below browserWakeAge.
seed := func(i int) {
key := fmt.Sprintf("kagane:w%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], now.Add(-62*time.Minute).UnixMilli())
}
for i := 0; i < 3; i++ {
seed(i)
}
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got)
}
// 5 due crosses the count threshold.
for i := 3; i < 5; i++ {
seed(i)
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 5 {
t.Fatalf("browser fetches with 5 due series = %d, want 5", got)
}
// A single long-neglected series wakes the browser by age alone.
seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0)
p.runOnce(context.Background())
if got := browser.callCount(); got != 6 {
t.Fatalf("browser fetches with one ancient series = %d, want 6", got)
}
}
// When one browser Lane loses the sidecar, the round's remaining browser
// Lanes are skipped: every fetch would fail anyway, and their Series must not
// burn their stamps on a dead Chrome (issue #100).
func TestRunOnceUnreachableBrowserStopsBrowserLanes(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
seedForCheck(t, s, "kagane:k", "https://kagane.to/series/k", 0)
seedForCheck(t, s, "novelfull:n", "https://novelfull.com/n.html", 0)
interrupted := fmt.Errorf("%w: %w", errBrowserInterrupted, errors.New("restart"))
browser := &fakeFetcher{status: 200, err: interrupted}
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
// Sorted lane order: comix, kagane, novelfull. Only comix attempted.
if got := browser.callCount(); got != 1 {
t.Fatalf("browser fetches = %d, want 1 (only the first browser lane)", got)
}
if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() {
t.Fatalf("comix stamp = %d, want %d", got, now.UnixMilli())
}
for _, key := range []string{"kagane:k", "novelfull:n"} {
if got := readLatestCheckedAt(t, s, key); got != 0 {
t.Fatalf("%s stamp = %d, want 0 (untried)", key, got)
}
}
// The shared flag decays after refuseBackoff: the next round probes
// again. comix's Series is resting (stamped last round), so the probe
// falls to kagane — the only Lane with something due — and its fresh
// loss re-gates the Lanes behind it.
p.Now = func() time.Time { return now.Add(16 * time.Minute) }
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches after backoff decay = %d, want 2 (kagane probes again)", got)
}
if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() {
t.Fatalf("comix stamp after decay = %d, want %d (resting, untouched)", got, now.UnixMilli())
}
if got := readLatestCheckedAt(t, s, "kagane:k"); got != now.Add(16*time.Minute).UnixMilli() {
t.Fatalf("kagane stamp after decay = %d, want %d (the probe)", got, now.Add(16*time.Minute).UnixMilli())
}
if got := readLatestCheckedAt(t, s, "novelfull:n"); got != 0 {
t.Fatalf("novelfull stamp after decay = %d, want 0 (gated by kagane's fresh loss)", got)
}
}
// A browser Lane that cannot keep up logs the backlog every pass, so the
// decision to give browser Sites more pages is measured, not guessed.
func TestRunOnceLogsBrowserLaneBehind(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
// Checked three rests ago: two rests' worth of wait past the due moment.
seedForCheck(t, s, "kagane:old", "https://kagane.to/series/old", now.Add(-3*time.Hour).UnixMilli())
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200},
BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := logs.String(); !strings.Contains(got, "latest poll kagane: browser lane behind by 1h0m0s") {
t.Fatalf("behind log = %q, want it to name kagane and the backlog", got)
}
}
// gatedCoverFetcher blocks every Fetch on a gate, so a test can hold a cover
// heal in flight and prove a Lane does not wait for it.
type gatedCoverFetcher struct {
inner *fakeBytesCoverFetcher
gate chan struct{}
started chan struct{}
once sync.Once
}
func (g *gatedCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte, string, error) {
g.once.Do(func() { g.started <- struct{}{} })
<-g.gate
return g.inner.Fetch(ctx, sourceURL)
}
// Cover heals run in the background: a heal stuck on a slow CDN must not
// delay the Lane's next Series-page Poll, or a large import with many blanks
// would make every Latest Chapter go stale (issue #100).
func TestRunOnceCoverFetchDoesNotDelayNextPoll(t *testing.T) {
s, dbURL := newTestStore(t)
// Two Series whose legacy cover sources still need healing.
for i := 0; i < 2; i++ {
key := fmt.Sprintf("asura:cover-%d", i)
seriesID := fmt.Sprintf("cover-%d", i)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID,
SeriesURL: "https://asurascans.com/comics/" + seriesID, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
seedCoverSource(t, dbURL, "asura", seriesID, fmt.Sprintf("https://cdn.example/covers/%d.jpg", i))
}
pages := &fakeFetcher{body: asuraSeriesFixture, status: 200}
gated := &gatedCoverFetcher{
inner: &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"},
gate: make(chan struct{}),
started: make(chan struct{}, 1),
}
p := &Poller{
Store: s, Fetch: pages, CoverBytesFetch: gated,
Now: func() time.Time { return time.UnixMilli(5_000_000) },
}
done := make(chan struct{})
go func() {
p.runOnce(context.Background())
close(done)
}()
<-gated.started // the first cover heal is now stuck on its CDN
select {
case <-done:
// The Lane finished its page fetches without waiting for the cover.
case <-time.After(5 * time.Second):
t.Fatal("runOnce blocked on an in-flight cover fetch")
}
close(gated.gate)
p.waitCovers()
if got := pages.callCount(); got != 2 {
t.Fatalf("series page fetches = %d, want 2", got)
}
if got := gated.inner.callCount(); got != 2 {
t.Fatalf("cover fetches = %d, want 2", got)
}
}
// At 3601 eligible Series the effective gap falls below the one-second floor;
// the clamp warning must name the Site so the operator knows which Lane is
// outrunning its plan.
func TestRunOnceClampWarningNamesTheSite(t *testing.T) {
s, dbURL := newTestStore(t)
db, err := sql.Open("pgx", dbURL)
if err != nil {
t.Fatalf("open %s: %v", dbURL, err)
}
defer db.Close()
if _, err := db.Exec(`INSERT INTO series (site, series_id, series_url, latest_checked_at)
SELECT 'asura', 'bulk-' || g, 'https://asurascans.com/comics/bulk-' || g, 0
FROM generate_series(1, 3601) AS g`); err != nil {
t.Fatalf("bulk seed series: %v", err)
}
if _, err := db.Exec(`INSERT INTO bookmarks (reader_id, site, series_id, updated_at)
SELECT (SELECT id FROM readers ORDER BY id LIMIT 1), 'asura', 'bulk-' || g, 1000
FROM generate_series(1, 3601) AS g`); err != nil {
t.Fatalf("bulk seed bookmarks: %v", err)
}
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
newTestPoller(t, s, &fakeFetcher{body: "<html></html>", status: 200}, time.UnixMilli(5_000_000)).
runOnce(context.Background())
if got := logs.String(); !strings.Contains(got, "latest poll asura: gap clamped to 1s floor (eligible series=3601)") {
t.Fatalf("clamp warning = %q, want it to name asura and 3601", got)
}
}
+1 -16
View File
@@ -9,7 +9,7 @@ import (
// seriesRead carries the two facts the poll and the acquirer both extract // seriesRead carries the two facts the poll and the acquirer both extract
// from a series page. Persistence, stamps and scheduling stay with the // from a series page. Persistence, stamps and scheduling stay with the
// callers, so the policies that keep the two flows distinct (stamp order, // callers, so the policies that keep the two flows distinct (stamp order,
// rests) are not swallowed by the module. // cooldowns) are not swallowed by the module.
type seriesRead struct { type seriesRead struct {
Latest latestChapter Latest latestChapter
HasLatest bool HasLatest bool
@@ -22,9 +22,6 @@ type seriesRead struct {
// errNotFetchable and errNoFetcher separate the gate and the route from fetch // errNotFetchable and errNoFetcher separate the gate and the route from fetch
// failures so each caller keeps its own distinct log line for all three. // failures so each caller keeps its own distinct log line for all three.
// errChallengeHeld (browser.go) is the outcome of a Site that answered with
// its interstitial — status 403 (cf-mitigated) or a challenge page body — and
// is how a Lane tells a refusal from an ordinary failure (issue #100).
var ( var (
errNotFetchable = errors.New("series url not fetchable") errNotFetchable = errors.New("series url not fetchable")
errNoFetcher = errors.New("no fetcher for site") errNoFetcher = errors.New("no fetcher for site")
@@ -52,21 +49,9 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
if err != nil { if err != nil {
return seriesRead{}, fmt.Errorf("fetch %s: %w", seriesURL, err) return seriesRead{}, fmt.Errorf("fetch %s: %w", seriesURL, err)
} }
if status == 403 {
// Cloudflare's challenge response for these Sites (cf-mitigated). The
// browser fetcher returns exactly this on a held interstitial, and a
// plain-TLS 403 means the same: the Site is refusing.
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status)
}
if status != 200 { if status != 200 {
return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status) return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status)
} }
if isInterstitial(body) {
// A 200 that is the challenge page, not the payload: the TLS route can
// receive this where the browser would have kept re-reading. Same
// refusal as the 403.
return seriesRead{}, fmt.Errorf("%w: fetch %s: interstitial body", errChallengeHeld, seriesURL)
}
latest, hasLatest := latestChapterFrom(site, seriesURL, body) latest, hasLatest := latestChapterFrom(site, seriesURL, body)
cover, hasCover := coverFrom(site, seriesURL, body) cover, hasCover := coverFrom(site, seriesURL, body)
return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil
+11 -93
View File
@@ -9,7 +9,6 @@ import (
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
"time"
"github.com/chromedp/chromedp" "github.com/chromedp/chromedp"
) )
@@ -33,11 +32,6 @@ type site struct {
LatestChapter func(seriesURL, body string) (latestChapter, bool) LatestChapter func(seriesURL, body string) (latestChapter, bool)
// Cover finds the Cover address in a fetched body. // Cover finds the Cover address in a fetched body.
Cover func(seriesURL, body string) (string, bool) Cover func(seriesURL, body string) (string, bool)
// Rest is how long a Series of this Site rests between Polls.
Rest time.Duration
// Gap is the Lane's strictest pace: at least one second must pass between
// two consecutive Series-page Polls of this Site (issue #100).
Gap time.Duration
// Browser reads this Site's payload from a cleared browser tab; nil // Browser reads this Site's payload from a cleared browser tab; nil
// means the page is fetched over plain TLS. // means the page is fetched over plain TLS.
Browser *browserRead Browser *browserRead
@@ -53,9 +47,9 @@ type browserRead struct {
// Done reports whether the payload arrived. // Done reports whether the payload arrived.
Done func(body string) bool Done func(body string) bool
// Fallback allows the plain-TLS fetcher when no browser is configured. // Fallback allows the plain-TLS fetcher when no browser is configured.
// False skips the Site instead. kagane and comix are false — a plain fetch // False skips the Site instead. kagane is false — a plain fetch would
// would only ever retrieve a challenge page — and novelfull is true, // only ever retrieve a challenge page — and novelfull is true, because
// because its challenge is a live time-varying fact (AGENTS.md). // its challenge is a live time-varying fact (AGENTS.md).
Fallback bool Fallback bool
} }
@@ -254,25 +248,14 @@ var comixInitialDataRe = regexp.MustCompile(`(?is)<script\b[^>]*\bid\s*=\s*["']i
// supplied, and a headless browser is a strong SSRF primitive. // supplied, and a headless browser is a strong SSRF primitive.
var kaganeImageURLRe = regexp.MustCompile(`^https://kagane\.to/api/v2/image/([0-9a-f-]{36})/compressed$`) var kaganeImageURLRe = regexp.MustCompile(`^https://kagane\.to/api/v2/image/([0-9a-f-]{36})/compressed$`)
// comixImageURLRe matches comix's cover host and path shape. Pinned in full
// (scheme, host, path characters, image extension) for the same reason
// kaganeImageURLRe is: the address reaches a headless browser, and it can
// originate in a client-supplied PUT body. No dot is allowed inside the path,
// so no traversal or second extension can hide in it. Shape from a live page,
// 2026-08-10: /039d/i/1/34/6a6742bf15736@280.jpg.
var comixImageURLRe = regexp.MustCompile(`^https://static\.comix\.to/[A-Za-z0-9@/_-]+\.(?:jpg|jpeg|png|webp)$`)
// browserOnlyCoverURL reports whether the browser sidecar is the only fetcher // browserOnlyCoverURL reports whether the browser sidecar is the only fetcher
// for cover bytes at imageURL. kagane's image route answers a plain fetch with // for cover bytes at imageURL. kagane's image route answers a plain fetch with
// a challenge and `cross-origin-resource-policy: same-origin`, and // a challenge and `cross-origin-resource-policy: same-origin`, so a TLS fetch
// static.comix.to answers one with the same Cloudflare challenge its pages // would only ever retrieve a challenge page and must not be attempted
// serve (measured 2026-08-12, issue #98), so a TLS fetch would only ever // (ADR-0007). This is the byte-fetch router's per-Site knowledge; it lives in
// retrieve a challenge page and must not be attempted (ADR-0007). This is the // the extraction module, which owns kagane's URL shapes.
// byte-fetch router's per-Site knowledge; it lives in the extraction module,
// which owns those URL shapes.
func browserOnlyCoverURL(imageURL string) bool { func browserOnlyCoverURL(imageURL string) bool {
return kaganeImageURLRe.MatchString(imageURL) || return kaganeImageURLRe.MatchString(imageURL)
comixImageURLRe.MatchString(imageURL)
} }
// kagane's browser-fetched series response publishes cover image IDs under // kagane's browser-fetched series response publishes cover image IDs under
@@ -386,50 +369,6 @@ func publishedCoverURL(value string) string {
return strings.ReplaceAll(value, " ", "%20") return strings.ReplaceAll(value, " ", "%20")
} }
// Poll Lane constants (issue #100). The per-Site structure is deliberately
// uniform at first — every Site rests an hour and gaps ten seconds — but it
// exists so a single Site can be slowed if it turns hostile, and the numbers
// stay in the registry so the structure has a place to differ.
const (
// defaultRest is how long every Series rests between Polls.
defaultRest = time.Hour
// defaultGap is the strictest pace of every Lane unless the eligible
// Series count forces it tighter.
defaultGap = 10 * time.Second
// minGap floors the effective gap. One request per second is already an
// order of magnitude past the strictest rate rule a free-plan Site can
// express (docs/research/cloudflare-bot-scoring-and-poll-cadence.md);
// below it the Lane is outrunning its own plan and says so loudly.
minGap = time.Second
// refuseBackoff is how long a Lane waits after its Site refused twice in
// one run before attempting it again.
refuseBackoff = 15 * time.Minute
// browserWakeCount and browserWakeAge gate a browser Lane's run: five or
// more due Series, or any one of them waiting this long, or Chrome stays
// asleep (ADR-0005 on-demand browser).
browserWakeCount = 5
browserWakeAge = 15 * time.Minute
)
// effectiveGap is a Site's pace: the registry gap, or one rest divided by the
// eligible Series count when that is smaller, never below one second. The
// denominator follows defaultRest rather than a literal hour so a Site whose
// rest is ever changed keeps its per-Series pace in step. The second return is
// true when the one-second floor engaged (and the Lane logs a warning naming
// the Site, every round it does).
func effectiveGap(s site, eligible int) (time.Duration, bool) {
gap := s.Gap
if eligible > 0 {
if perSeries := defaultRest / time.Duration(eligible); perSeries < gap {
gap = perSeries
}
}
if gap < minGap {
return minGap, true
}
return gap, false
}
// sites is the registry: one entry per Site, keyed by the stored site string. // sites is the registry: one entry per Site, keyed by the stored site string.
// Adding a Site means adding an entry here and nowhere else — the dispatch // Adding a Site means adding an entry here and nowhere else — the dispatch
// functions above and the poller's route list are lookups into this map. An // functions above and the poller's route list are lookups into this map. An
@@ -440,38 +379,21 @@ var sites = map[string]site{
Host: "asurascans.com", Host: "asurascans.com",
LatestChapter: asuraLatestChapter, LatestChapter: asuraLatestChapter,
Cover: ogImageCover, Cover: ogImageCover,
Rest: defaultRest,
Gap: defaultGap,
}, },
"demonic": { "demonic": {
Host: "demonicscans.org", Host: "demonicscans.org",
LatestChapter: demonicLatestChapter, LatestChapter: demonicLatestChapter,
Cover: ogImageCover, Cover: ogImageCover,
Rest: defaultRest,
Gap: defaultGap,
}, },
"comix": { "comix": {
Host: "comix.to", Host: "comix.to",
LatestChapter: comixLatestChapter, LatestChapter: comixLatestChapter,
Cover: comixCoverEntry, Cover: comixCoverEntry,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
Read: comixRead,
// The interstitial is served in place of the page, so "arrived"
// has to exclude it explicitly, as novelfull's does.
Done: func(body string) bool { return body != "" && !isInterstitial(body) },
// Never falls back: a plain fetch of a comix page or cover
// retrieves only a challenge page (measured 2026-08-12).
Fallback: false,
},
}, },
"kagane": { "kagane": {
Host: "kagane.to", Host: "kagane.to",
LatestChapter: kaganeLatestChapter, LatestChapter: kaganeLatestChapter,
Cover: kaganeCoverEntry, Cover: kaganeCoverEntry,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{ Browser: &browserRead{
Read: kaganeRead, Read: kaganeRead,
Done: func(body string) bool { return body != "" }, Done: func(body string) bool { return body != "" },
@@ -484,8 +406,6 @@ var sites = map[string]site{
Host: "novelfull.com", Host: "novelfull.com",
LatestChapter: novelfullLatestChapter, LatestChapter: novelfullLatestChapter,
Cover: novelfullCoverEntry, Cover: novelfullCoverEntry,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{ Browser: &browserRead{
Read: novelfullRead, Read: novelfullRead,
// The interstitial has a DOM too, so "the payload arrived" has to // The interstitial has a DOM too, so "the payload arrived" has to
@@ -498,15 +418,13 @@ var sites = map[string]site{
Host: "lightnovelworld.net", Host: "lightnovelworld.net",
LatestChapter: lnwLatestChapter, LatestChapter: lnwLatestChapter,
Cover: ogImageCover, Cover: ogImageCover,
Rest: defaultRest,
Gap: defaultGap,
}, },
} }
// browserBackedSites is derived from the registry: the Sites whose pages are // browserBackedSites is derived from the registry: the Sites whose pages are
// read through the browser sidecar. Sorted so callers that range it (the // read through the browser sidecar, which are also the ones granted the longer
// browser fetcher's dispatch) see a stable order instead of map-iteration // cooldown. Sorted so callers that range it (the due query, the browser
// noise. // fetcher's dispatch) see a stable order instead of map-iteration noise.
func browserBackedSites() []string { func browserBackedSites() []string {
out := make([]string, 0, len(sites)) out := make([]string, 0, len(sites))
for name, s := range sites { for name, s := range sites {
-7
View File
@@ -41,13 +41,6 @@ const challengeFixture = `<!DOCTYPE html><html><head><title>Just a moment...</ti
// https://comix.to/title/n8we-dungeons-and-crayons fetched 2026-08-03. comix is // https://comix.to/title/n8we-dungeons-and-crayons fetched 2026-08-03. comix is
// an SPA: the page ships a JSON state blob rather than a list of chapter // an SPA: the page ships a JSON state blob rather than a list of chapter
// anchors, and latestChapterUrl is where the newest chapter actually lives. // anchors, and latestChapterUrl is where the newest chapter actually lives.
//
// Still the right fixture after comix moved behind the challenge (#98): the
// browser read is an in-tab fetch of the Series URL, so the body a poll parses
// is this same server-rendered HTML, not a rendered DOM. Confirmed against a
// live cleared tab 2026-08-16 (TestSmokeComix): the in-tab fetch returned
// 24793 bytes of server-rendered HTML that these same parses read a chapter
// and a cover out of.
const comixSeriesFixture = ` const comixSeriesFixture = `
{"firstChapterUrl":"/title/n8we-dungeons-and-crayons/5038739-chapter-1","latestChapterUrl":"/title/n8we-dungeons-and-crayons/11139891-chapter-80"}, {"firstChapterUrl":"/title/n8we-dungeons-and-crayons/5038739-chapter-1","latestChapterUrl":"/title/n8we-dungeons-and-crayons/11139891-chapter-80"},
{""manga","recommended","n8we",1]":{"items":[{"latestChapterUrl":"/title/qqwrm-full-time-awakening/99999999-chapter-999"}]} {""manga","recommended","n8we",1]":{"items":[{"latestChapterUrl":"/title/qqwrm-full-time-awakening/99999999-chapter-999"}]}
@@ -1,72 +0,0 @@
package latest
import (
"context"
"os"
"testing"
"time"
"bookmarkmanager/backend/internal/store"
)
// TestSmokeComix answers "is comix's challenge clearing from this browser right
// now" — a live, time-varying fact, so a red run is something to re-check
// before it is a defect. Needs the real browser unit with outbound network:
//
// cd chrome && BROWSER_BIND_ADDR=127.0.0.1 docker compose up -d --build
// SMOKE_BROWSER_WS_URL=ws://127.0.0.1:9222 go test -run TestSmokeComix ./internal/latest
//
// It walks the whole read: the in-tab page fetch, both parses, and the Cover
// bytes by direct navigation to static.comix.to. The Cover address comes out of
// the page rather than being pinned in the test, because a stored one rots.
func TestSmokeComix(t *testing.T) {
ws := os.Getenv("SMOKE_BROWSER_WS_URL")
if ws == "" {
t.Skip("SMOKE_BROWSER_WS_URL unset")
}
const seriesURL = "https://comix.to/title/m12d-classmate"
f, err := NewBrowserFetcher(ws)
if err != nil {
t.Fatalf("NewBrowserFetcher: %v", err)
}
defer f.Close()
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
defer cancel()
body, status, err := f.Get(ctx, seriesURL)
if err != nil {
t.Fatalf("Get: %v", err)
}
t.Logf("status=%d bytes=%d", status, len(body))
if status != 200 {
t.Fatalf("status = %d, want 200 — the sidecar is not clearing the challenge", status)
}
chapter, ok := latestChapterFrom("comix", seriesURL, body)
if !ok {
t.Fatalf("no latest chapter in %d bytes — page shape changed", len(body))
}
t.Logf("latest chapter: %v %q", chapter.Num, chapter.Label)
cover, ok := coverFrom("comix", seriesURL, body)
if !ok {
t.Fatalf("no cover address in %d bytes — page shape changed", len(body))
}
t.Logf("cover: %s", cover)
if !browserOnlyCoverURL(cover) {
t.Fatalf("cover %q is not claimed by the browser gate: the pin and the live URL shape disagree", cover)
}
bytes, contentType, err := f.Image(ctx, cover)
if err != nil {
t.Fatalf("Image: %v", err)
}
if len(bytes) < 1000 {
t.Fatalf("cover is %d bytes, want a real image", len(bytes))
}
t.Logf("fetched %d bytes of %s", len(bytes), contentType)
if _, ok := store.CoverContentType(contentType); !ok {
t.Fatalf("content type %q is not storable", contentType)
}
}
+15 -38
View File
@@ -16,6 +16,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"github.com/jackc/pgx/v5/pgtype"
_ "github.com/jackc/pgx/v5/stdlib" _ "github.com/jackc/pgx/v5/stdlib"
) )
@@ -896,13 +897,10 @@ func (s *Store) Delete(readerID int64, key string) error {
return nil return nil
} }
// DueForLatestCheck returns one Site's series whose server-side // DueForLatestCheck returns series whose server-side latest-chapter check has
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks // aged past the appropriate cutoff, ordered by how many bookmarks reference
// reference them (descending) then least-recently-checked first. One Site per // them (descending) then least-recently-checked first, at most limit of them.
// query, because each Poll Lane asks for its own list: the query carries one // Browser-backed sites use browserCutoffMs; every other site uses cutoffMs.
// Site and one cut-off instead of parallel lists (issue #100). There is no
// limit — the Lane's own gap paces the fetches, and the batch size that used
// to cap this query is gone with the shared pace.
// //
// The reader_count ordering is the point of the split (ADR-0003): a series // The reader_count ordering is the point of the split (ADR-0003): a series
// shared by several readers is fetched once per due cycle, and the popular // shared by several readers is fetched once per due cycle, and the popular
@@ -918,17 +916,20 @@ func (s *Store) Delete(readerID int64, key string) error {
// burns requests. Archived bookmarks still count — knowing what a shelved // burns requests. Archived bookmarks still count — knowing what a shelved
// series is up to is the whole reason for archiving instead of deleting. // series is up to is the whole reason for archiving instead of deleting.
// A series with no bookmarks at all never appears: the join excludes it. // A series with no bookmarks at all never appears: the join excludes it.
func (s *Store) DueForLatestCheck(site string, cutoffMs int64) ([]Series, error) { func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites []string, limit int) ([]Series, error) {
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
FROM series s FROM series s
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
WHERE s.site = $1 WHERE s.series_url <> ''
AND s.series_url <> '' AND s.latest_checked_at <= CASE
AND s.latest_checked_at <= $2::bigint WHEN s.site = ANY($3::text[]) THEN $2::bigint
ELSE $1::bigint
END
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover, GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0 HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
ORDER BY reader_count DESC, s.latest_checked_at ASC`, site, cutoffMs) ORDER BY reader_count DESC, s.latest_checked_at ASC
LIMIT $4`, cutoffMs, browserCutoffMs, pgtype.FlatArray[string](browserSites), limit)
if err != nil { if err != nil {
return nil, fmt.Errorf("query due series: %w", err) return nil, fmt.Errorf("query due series: %w", err)
} }
@@ -945,30 +946,6 @@ func (s *Store) DueForLatestCheck(site string, cutoffMs int64) ([]Series, error)
return out, rows.Err() return out, rows.Err()
} }
// EligibleSeriesCount returns how many of a Site's Series still have at least
// one bookmark outside the finished bucket. It is the denominator of the
// Lane's pace (issue #100): the effective gap is the smaller of the registry
// gap and one hour divided by this count, so Series that will never be Polled
// do not make the Lane faster than it needs to be, and counting every eligible
// Series rather than only those currently due keeps the pace steady — the
// single worst moment to be fastest is startup, when everything is due at
// once.
func (s *Store) EligibleSeriesCount(site string) (int, error) {
var n int
err := s.db.QueryRow(`SELECT COUNT(*) FROM (
SELECT 1
FROM series s
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
WHERE s.site = $1
GROUP BY s.site, s.series_id
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
) e`, site).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count eligible series %s: %w", site, err)
}
return n, nil
}
// MarkLatestChecked records that the server looked at a series at ts, whatever // MarkLatestChecked records that the server looked at a series at ts, whatever
// the look turned up. Marking a missing series is not an error: the row may // the look turned up. Marking a missing series is not an error: the row may
// have been orphaned while a fetch was in flight. // have been orphaned while a fetch was in flight.
@@ -977,7 +954,7 @@ func (s *Store) EligibleSeriesCount(site string) (int, error) {
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes // out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
// a whole Bookmark from the client and Upsert writes every series column it // a whole Bookmark from the client and Upsert writes every series column it
// knows about, so a userscript PUT — which has no idea this field exists — // knows about, so a userscript PUT — which has no idea this field exists —
// would write a zero and reset the rest, making the poller re-fetch that // would write a zero and reset the cooldown, making the poller re-fetch that
// series every tick for as long as the user kept reading it. // series every tick for as long as the user kept reading it.
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error { func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
if _, err := s.db.Exec( if _, err := s.db.Exec(
@@ -989,7 +966,7 @@ func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
} }
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for // LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
// tests outside this package (the poller's own tests assert on rest // tests outside this package (the poller's own tests assert on cooldown
// bookkeeping) — see MarkLatestChecked for why the field stays off the // bookkeeping) — see MarkLatestChecked for why the field stays off the
// client-visible row. // client-visible row.
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) { func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
+12 -47
View File
@@ -298,7 +298,7 @@ func TestDueForLatestCheck(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
seedForCheck(t, s, "asura:x", tt.seriesURL, tt.checkedAt) seedForCheck(t, s, "asura:x", tt.seriesURL, tt.checkedAt)
due, err := s.DueForLatestCheck("asura", now-hour) due, err := s.DueForLatestCheck(now-hour, now-hour, nil, 10)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
@@ -309,25 +309,22 @@ func TestDueForLatestCheck(t *testing.T) {
} }
} }
func TestDueForLatestCheckOldestFirstAndScopedToSite(t *testing.T) { func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
// Insert newest-checked first so a correct ORDER BY has to reverse it. // Insert newest-checked first so a correct ORDER BY has to reverse it.
seedForCheck(t, s, "asura:c", "https://asurascans.com/comics/c", 300) seedForCheck(t, s, "asura:c", "https://asurascans.com/comics/c", 300)
seedForCheck(t, s, "asura:b", "https://asurascans.com/comics/b", 200) seedForCheck(t, s, "asura:b", "https://asurascans.com/comics/b", 200)
seedForCheck(t, s, "asura:a", "https://asurascans.com/comics/a", 100) seedForCheck(t, s, "asura:a", "https://asurascans.com/comics/a", 100)
// A second Site's due series must not appear in asura's list: each Lane
// asks for one Site, and no Lane may see another's queue.
seedForCheck(t, s, "demonic:z", "https://demonicscans.org/manga/z", 0)
due, err := s.DueForLatestCheck("asura", 1000) due, err := s.DueForLatestCheck(1000, 1000, nil, 2)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
if len(due) != 3 { if len(due) != 2 {
t.Fatalf("got %d rows, want 3 (all of asura's, none of demonic's)", len(due)) t.Fatalf("got %d rows, want 2 (limit)", len(due))
} }
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" || due[2].Key() != "asura:c" { if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" {
t.Fatalf("got %q,%q,%q; want asura:a,asura:b,asura:c (oldest first)", due[0].Key(), due[1].Key(), due[2].Key()) t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key())
} }
} }
@@ -499,7 +496,7 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
} }
} }
due, err := store.DueForLatestCheck("asura", time.Now().UnixMilli()) due, err := store.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
@@ -515,38 +512,6 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
} }
} }
// The gap's denominator counts every Series the Lane will ever Poll: a
// finished Series must not make the Lane faster than it needs to be, and
// another Site's Series must not leak into this Site's count.
func TestEligibleSeriesCount(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:reading", "https://asurascans.com/comics/reading", 0)
seedForCheck(t, store, "asura:archived", "https://asurascans.com/comics/archived", 0)
if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:finished", Site: "asura", SeriesID: "finished",
SeriesURL: "https://asurascans.com/comics/finished",
Status: StatusFinished, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed finished: %v", err)
}
seedForCheck(t, store, "demonic:z", "https://demonicscans.org/manga/z", 0)
n, err := store.EligibleSeriesCount("asura")
if err != nil {
t.Fatalf("EligibleSeriesCount: %v", err)
}
if n != 2 {
t.Fatalf("eligible = %d, want 2 (finished excluded, demonic excluded)", n)
}
n, err = store.EligibleSeriesCount("demonic")
if err != nil {
t.Fatalf("EligibleSeriesCount(demonic): %v", err)
}
if n != 1 {
t.Fatalf("eligible(demonic) = %d, want 1", n)
}
}
func TestDisplayChapter(t *testing.T) { func TestDisplayChapter(t *testing.T) {
cases := []struct { cases := []struct {
name string name string
@@ -1005,7 +970,7 @@ func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) {
seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001) seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100) seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100)
due, err := s.DueForLatestCheck("asura", 1000) due, err := s.DueForLatestCheck(1000, 1000, nil, 10)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
@@ -1031,7 +996,7 @@ func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
t.Fatalf("seed orphan series: %v", err) t.Fatalf("seed orphan series: %v", err)
} }
due, err := s.DueForLatestCheck("asura", 1000) due, err := s.DueForLatestCheck(1000, 1000, nil, 10)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
@@ -1369,7 +1334,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
t.Fatalf("series rows = %d, want 1 shared row for two bookmarks", series) t.Fatalf("series rows = %d, want 1 shared row for two bookmarks", series)
} }
due, err := s.DueForLatestCheck("asura", time.Now().UnixMilli()) due, err := s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
@@ -1385,7 +1350,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
if b, ok, err := s.Get(s.OwnerID(), "asura:solo"); err != nil || !ok || b.LastChapterNum != 200 { if b, ok, err := s.Get(s.OwnerID(), "asura:solo"); err != nil || !ok || b.LastChapterNum != 200 {
t.Fatalf("owner's bookmark after the other's delete = %+v ok=%v err=%v, want it intact", b, ok, err) t.Fatalf("owner's bookmark after the other's delete = %+v ok=%v err=%v, want it intact", b, ok, err)
} }
due, err = s.DueForLatestCheck("asura", time.Now().UnixMilli()) due, err = s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
if err != nil { if err != nil {
t.Fatalf("DueForLatestCheck after delete: %v", err) t.Fatalf("DueForLatestCheck after delete: %v", err)
} }
+86 -11
View File
@@ -7,6 +7,7 @@ import (
"net/http" "net/http"
"os" "os"
"os/signal" "os/signal"
"strconv"
"strings" "strings"
"syscall" "syscall"
"time" "time"
@@ -60,15 +61,30 @@ type Config struct {
// LatestPoll configures the background latest-chapter poller. // LatestPoll configures the background latest-chapter poller.
// //
// Only the kill switch lives here. Pace is per Site — rest time and gap are // Sizing: batch x (cooldown / interval) is how many series hold a true cooldown
// registry properties (internal/latest/sites.go, issue #100), because each // cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this
// Lane has to be able to differ from the others. The five environment // deployment. Past that nothing breaks; the effective cadence stretches to
// settings that used to size a shared pace (cooldown, browser cooldown, // N x interval / batch and the oldest-checked-first ordering keeps it uniform.
// interval, stagger, batch) are gone with it: no deployed .env may carry them.
type LatestPoll struct { type LatestPoll struct {
Enabled bool Enabled bool
Cooldown time.Duration
BrowserCooldown time.Duration
Interval time.Duration
Stagger time.Duration
Batch int
} }
const (
defaultPollCooldown = time.Hour
defaultBrowserPollCooldown = 6 * time.Hour
defaultPollInterval = 10 * time.Minute
defaultPollStagger = 20 * time.Second
defaultPollBatch = 14
// minPollCooldown keeps a typo from turning a polite background check into
// a hammer against sites that are already bot-scoring us.
minPollCooldown = 15 * time.Minute
)
func envOr(key, def string) string { func envOr(key, def string) string {
if v := os.Getenv(key); v != "" { if v := os.Getenv(key); v != "" {
return v return v
@@ -91,11 +107,66 @@ func envBool(key string, def bool) bool {
} }
} }
// loadLatestPoll reads the poller's settings. The pace knobs that used to be // envDuration reads a duration env var. An unparseable or non-positive value
// clamped here are registry properties now (issue #100), so there is nothing // falls back to def and logs rather than failing startup: the poller is an
// left to clamp. // enhancement, and a typo in one of its knobs must not stop bookmark sync.
func envDuration(key string, def time.Duration) time.Duration {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
d, err := time.ParseDuration(raw)
if err != nil || d <= 0 {
log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def)
return def
}
return d
}
// envInt reads a positive integer env var, with the same fallback policy.
func envInt(key string, def int) int {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
n, err := strconv.Atoi(raw)
if err != nil || n <= 0 {
log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def)
return def
}
return n
}
func clampPollCooldown(name string, d time.Duration) time.Duration {
if d < minPollCooldown {
log.Printf("config: %s %s is below the %s floor, clamping", name, d, minPollCooldown)
return minPollCooldown
}
return d
}
// loadLatestPoll reads the poller's settings, clamping anything that would make
// it antisocial.
func loadLatestPoll() LatestPoll { func loadLatestPoll() LatestPoll {
return LatestPoll{Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true)} p := LatestPoll{
Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true),
Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown),
BrowserCooldown: envDuration("LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", defaultBrowserPollCooldown),
Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval),
Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger),
Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch),
}
p.Cooldown = clampPollCooldown("cooldown", p.Cooldown)
p.BrowserCooldown = clampPollCooldown("browser cooldown", p.BrowserCooldown)
// batch x stagger has to fit inside one tick or a batch is still running
// when the next one is due. Run() serialises them, so this degrades to a
// slower cadence rather than to overlapping fetches — worth a warning, not
// a failure.
if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval {
log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick",
p.Batch, p.Stagger, span, p.Interval)
}
return p
} }
func loadConfig() Config { func loadConfig() Config {
@@ -306,8 +377,7 @@ func main() {
} }
} }
// newLatestPoller wires the fetcher seams into the poller. Pace is registry // newLatestPoller wires the configured cooldowns and fetchers into the poller.
// property, not config (issue #100), so there are no knobs to pass through.
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller { func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller {
var covers latest.BrowserCoverFetcher var covers latest.BrowserCoverFetcher
if f, ok := browser.(latest.BrowserCoverFetcher); ok { if f, ok := browser.(latest.BrowserCoverFetcher); ok {
@@ -320,6 +390,11 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
CoverFetch: covers, CoverFetch: covers,
CoverBytesFetch: latest.NewCoverFetcher(), CoverBytesFetch: latest.NewCoverFetcher(),
Now: time.Now, Now: time.Now,
Cooldown: cfg.Cooldown,
BrowserCooldown: cfg.BrowserCooldown,
Interval: cfg.Interval,
Stagger: cfg.Stagger,
Batch: cfg.Batch,
} }
} }
+115 -19
View File
@@ -9,15 +9,31 @@ import (
"net/http/httptest" "net/http/httptest"
"strings" "strings"
"testing" "testing"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store" "bookmarkmanager/backend/internal/store"
) )
func TestLoadLatestPollDefaults(t *testing.T) { func TestLoadLatestPollDefaults(t *testing.T) {
t.Setenv("LATEST_CHAPTER_POLL_ENABLED", "") for _, k := range []string{
if got := loadLatestPoll(); got != (LatestPoll{Enabled: true}) { "LATEST_CHAPTER_POLL_ENABLED", "LATEST_CHAPTER_POLL_COOLDOWN",
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, LatestPoll{Enabled: true}) "LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", "LATEST_CHAPTER_POLL_INTERVAL",
"LATEST_CHAPTER_POLL_STAGGER", "LATEST_CHAPTER_POLL_BATCH",
} {
t.Setenv(k, "")
}
got := loadLatestPoll()
want := LatestPoll{
Enabled: true,
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
Interval: 10 * time.Minute,
Stagger: 20 * time.Second,
Batch: 14,
}
if got != want {
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, want)
} }
} }
@@ -47,25 +63,105 @@ func TestLoadLatestPollEnabledParsing(t *testing.T) {
} }
} }
// newLatestPoller wires the fetcher seams; pace lives in the registry, so func TestLoadLatestPollClampsAndFallsBack(t *testing.T) {
// nothing here sizes a cooldown any more. tests := []struct {
func TestNewLatestPollerWiresFetchers(t *testing.T) { name string
tls := &latest.TLSFetcher{} env map[string]string
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil) wantFrom func(LatestPoll) any
if p.Fetch != tls { want any
t.Fatalf("Fetch not wired") }{
{
name: "cooldown below the floor is clamped up",
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "1m"},
wantFrom: func(p LatestPoll) any { return p.Cooldown },
want: 15 * time.Minute,
},
{
name: "cooldown at the floor is kept",
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "15m"},
wantFrom: func(p LatestPoll) any { return p.Cooldown },
want: 15 * time.Minute,
},
{
name: "browser cooldown below the floor is clamped up",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "1m"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 15 * time.Minute,
},
{
name: "browser cooldown at the floor is kept",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "15m"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 15 * time.Minute,
},
{
name: "browser cooldown override is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "8h"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 8 * time.Hour,
},
{
name: "browser cooldown unparseable value falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "six hours"},
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
want: 6 * time.Hour,
},
{
name: "a valid override is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "5m"},
wantFrom: func(p LatestPoll) any { return p.Interval },
want: 5 * time.Minute,
},
{
name: "an unparseable duration falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "ten minutes"},
wantFrom: func(p LatestPoll) any { return p.Interval },
want: 10 * time.Minute,
},
{
name: "a zero duration falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_STAGGER": "0s"},
wantFrom: func(p LatestPoll) any { return p.Stagger },
want: 20 * time.Second,
},
{
name: "a valid batch is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "30"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 30,
},
{
name: "a negative batch falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "-5"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 14,
},
{
name: "a non-numeric batch falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "lots"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 14,
},
} }
if p.BrowserFetch != nil { for _, tt := range tests {
t.Fatalf("BrowserFetch = %v, want nil for a browser-less deployment", p.BrowserFetch) t.Run(tt.name, func(t *testing.T) {
for k, v := range tt.env {
t.Setenv(k, v)
} }
if p.CoverFetch != nil { if got := tt.wantFrom(loadLatestPoll()); got != tt.want {
t.Fatalf("CoverFetch = %v, want nil when the browser is absent", p.CoverFetch) t.Fatalf("got %v, want %v", got, tt.want)
} }
if p.CoverBytesFetch == nil { })
t.Fatalf("CoverBytesFetch = nil, want the TLS cover fetcher")
} }
if p.Now == nil { }
t.Fatalf("Now = nil, want the live clock")
func TestNewLatestPollerWiresCooldowns(t *testing.T) {
p := newLatestPoller(nil, LatestPoll{
Cooldown: time.Hour,
BrowserCooldown: 6 * time.Hour,
}, nil, nil)
if p.Cooldown != time.Hour || p.BrowserCooldown != 6*time.Hour {
t.Fatalf("poller cooldowns = %s/%s, want 1h/6h", p.Cooldown, p.BrowserCooldown)
} }
} }
+7 -4
View File
@@ -58,11 +58,14 @@ services:
# Second script from the same bindmount; the novel library is a separate # Second script from the same bindmount; the novel library is a separate
# Violentmonkey install. # Violentmonkey install.
NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js} NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js}
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the # Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill
# kill switch; it only takes effect because it is listed here. Pace is # switch; it only takes effect because these are listed here.
# per Site in the registry (one Poll Lane per Site, issue #100) — the
# cooldown/interval/stagger/batch knobs are gone with the shared pace.
LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1} LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1}
LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h}
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h}
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
# CDP endpoint for sites behind a JavaScript challenge (kagane, # CDP endpoint for sites behind a JavaScript challenge (kagane,
# novelfull). The browser is not part of this stack — it runs on the home # novelfull). The browser is not part of this stack — it runs on the home
# machine as its own unit (chrome/docker-compose.yml) and is reached over # machine as its own unit (chrome/docker-compose.yml) and is reached over
-91
View File
@@ -1,91 +0,0 @@
# ADR-0010: Poll Lanes — one independent Poll stream per Site
Date: 2026-08-16
Status: accepted
## Decision
Replace the single shared polling pace with one **Poll Lane** per Site: an
independent goroutine that polls only that Site's Series, paced by that Site's
registry entry. Pace moves out of config and into the Site registry
(`internal/latest/sites.go`): every entry carries a `Rest` (how long a Series
rests between Polls) and a `Gap` (how long the Lane waits between fetches).
Rest is enforced by the due query's WHERE clause (`latest_checked_at <= now -
Rest`), never by a timer — the same mechanism that enforced the old cooldown.
The Lane enforces its own gap by sleeping between fetches. `effectiveGap` is
the registry gap, or one hour divided by the Site's eligible Series count when
that is smaller, never below one second.
The five environment settings that used to size the shared pace —
`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
`_STAGGER` — are deleted. Only the kill switch `LATEST_CHAPTER_POLL_ENABLED`
remains. No deployed `.env` may carry the deleted knobs.
## Why
The shared pace capped the whole backend at roughly 180 Polls an hour (one
20-second stagger across one queue). ~60 Series today, scaling to hundreds or
thousands, would stretch the hour beyond what the New Chapter signal can
tolerate. Worse, the queue mixed Sites with very different costs: kagane and
comix pay seconds of a serialized single-tab Chrome per Poll (a challenged
page, ADR-0005), and one hostile Site burning its challenge timeout made every
other Site's Series wait — "one hostile Site can eat most of an hour".
Lanes fix both at once:
- **Throughput scales per Site.** The six Lanes fetch concurrently; a Lane's
own gap paces it. The browser Lanes' combined ceiling stays about 360 Polls
an hour (one tab), and when they cannot keep up the wait past Rest grows and
is logged every pass — the "behind by X" measurement, so the decision to
give browser Sites more pages is made from data.
- **Hostility is contained.** A refusal (two challenge-held reads in one
pass) stops only that Site's Lane for `refuseBackoff` (15m); the rest of
that Lane's Series stay unstamped and due. A lost browser gates the other
browser Lanes' passes for the same window — the flag is shared Poller
state, so the loss is noticed once instead of once per Lane per pass, and
decays after 15m so the Lanes probe again. One Site can no longer tax the
others.
## Tradeoffs and rejections
- **Per-Site env knobs** (e.g. `KAGANE_POLL_GAP`) rejected: the registry is
the single place pace lives, testable and reviewable; config knobs would
recreate the shared-pace sprawl with six times the surface. All six entries
are deliberately uniform at first — rest an hour, gap ten seconds — so the
structure exists to differ without inventing numbers for Sites that have
not earned them.
- **Dynamic gap** (`rest / eligible`) is the one knob that stays automatic:
a Site with more Series than one per ten seconds would otherwise back up
behind its own gap, and the per-Series share of the hour is the natural
pace. The ten-second default is not arbitrary: one request per ten seconds
is the strictest rate rule a free-plan Site can even express (per-zone
rate limiting, as documented in
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`), so the
default pace is exactly what the most restrictive Site would demand of us.
The computed gap never goes below one second and logs loudly when the
floor engages.
- **Timer-based pacing** rejected: the old ticker made the poller's rate a
function of wall clock rather than of what was due. The due-query cutoff is
the only rate authority; the Lane sleep just prevents hammering.
- **Batch size** (the old `_BATCH` cap) is gone with the shared pace: a Lane
processes everything due, paced by its gap. There is no global queue left
to bound.
## Constraints preserved
- Stamp-before-fetch ("attempted" semantics): an untried Series stays due, so
a browser that appears after a restart finds its full queue waiting.
- Browser wake gate (ADR-0005): a browser Lane leaves Chrome asleep below
five due Series and 15 minutes of wait, per Lane.
- The browser is not in the API stack (ADR-0006): an unreachable browser
degrades a Lane exactly as an unset `BROWSER_WS_URL` — browser-only Sites
skipped, plain-TLS unaffected, stored covers still served.
- Cover heals moved to background goroutines (joined by the test suite via
`waitCovers`) so a slow cover CDN cannot consume a Lane's gap.
Supersedes the pace mechanics of ADR-0003's "raise throughput instead" note
(the stagger cut it rejected is what the per-Lane gap replaces) and the
6-hour browser cooldown introduced with the browser-backed Sites; the
1-hour browser rest was already cleared as safe by
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
+18 -7
View File
@@ -18,20 +18,17 @@
"16": "Cloudflare bot scoring and poll cadence — what is actually documented", "16": "Cloudflare bot scoring and poll cadence — what is actually documented",
"17": "Go Code Style Guide", "17": "Go Code Style Guide",
"18": "Agent Skills", "18": "Agent Skills",
"19": "Store",
"20": "I/O Performance Patterns", "20": "I/O Performance Patterns",
"21": "CPU Optimization", "21": "CPU Optimization",
"22": "Caching Patterns", "22": "Caching Patterns",
"23": "Browser Entrypoint", "23": "Browser Entrypoint",
"24": "Memory Allocation & GC", "24": "Memory Allocation & GC",
"25": "Cover Fetcher Tests", "25": "Cover Fetcher Tests",
"26": "Open",
"27": "Find Skills Guide", "27": "Find Skills Guide",
"28": "Allocation Patterns", "28": "Allocation Patterns",
"29": "Observability & Alerting", "29": "Observability & Alerting",
"30": "AGENTS.md", "31": "Memory Layout",
"31": "Store", "32": "Repo Hard Constraints",
"32": "Open",
"33": "Go Testing Guide", "33": "Go Testing Guide",
"34": "Session Store", "34": "Session Store",
"35": "Web UI Filter Logic", "35": "Web UI Filter Logic",
@@ -40,7 +37,6 @@
"38": "novel-logic.test.js", "38": "novel-logic.test.js",
"39": "UI Critique 2026-07-26A", "39": "UI Critique 2026-07-26A",
"40": "UI Critique 2026-07-26B", "40": "UI Critique 2026-07-26B",
"42": "pgtest.go",
"43": "Issue Tracker & Triage", "43": "Issue Tracker & Triage",
"44": "Ticket Workflow", "44": "Ticket Workflow",
"45": "Go Perf Alert Rules", "45": "Go Perf Alert Rules",
@@ -123,7 +119,6 @@
"122": "Identity comes from Discord OAuth; we store no passwords and send no email", "122": "Identity comes from Discord OAuth; we store no passwords and send no email",
"123": "The wire format stays flat and deliberately does not mirror the schema", "123": "The wire format stays flat and deliberately does not mirror the schema",
"124": "ADR-0005: On-demand browser sidecar", "124": "ADR-0005: On-demand browser sidecar",
"125": "sessions_test.go",
"126": "Bookmark Manager", "126": "Bookmark Manager",
"127": "triage-labels.md", "127": "triage-labels.md",
"128": "Cross-Ticket Contract", "128": "Cross-Ticket Contract",
@@ -155,6 +150,18 @@
"162": "singleflight Cache Stampede Prevention", "162": "singleflight Cache Stampede Prevention",
"163": "Struct Field Alignment", "163": "Struct Field Alignment",
"164": "testing/synctest Deterministic Goroutine Testing", "164": "testing/synctest Deterministic Goroutine Testing",
"165": "API Package (Bookmark JSON Handlers)",
"166": "Cover Acquisition & Serving Pipeline",
"167": "Backend AGENTS.md Guidance",
"168": "HTTP Middleware (Auth/Gzip/CORS)",
"169": "Latest Package (Site Parsers & Poller)",
"170": "Latest-Chapter Poller",
"171": "Main Composition Root",
"172": "Migration-Owned Schema",
"173": "Session Package (Cookie Signing & Rate Limit)",
"174": "updated_at List-Order Rule",
"175": "Userscript Package (Serving Handler)",
"176": "AGENTS.md",
"177": "Backend CLAUDE.md Guidance", "177": "Backend CLAUDE.md Guidance",
"178": "Graphify Knowledge Graph (graphify-out/)", "178": "Graphify Knowledge Graph (graphify-out/)",
"179": "CLAUDE.md (Symlink to AGENTS.md)", "179": "CLAUDE.md (Symlink to AGENTS.md)",
@@ -219,6 +226,10 @@
"259": "Dark-First Design Constraint", "259": "Dark-First Design Constraint",
"260": "Discord Guild Membership", "260": "Discord Guild Membership",
"261": "Reader Isolation Invariant", "261": "Reader Isolation Invariant",
"268": "Browser Unit Redeploy",
"269": "pg_dump Hot Backup",
"270": "Redeploy Runbook",
"271": "Rollback Strategy",
"273": "AGENTS.md", "273": "AGENTS.md",
"279": "Userscript CLAUDE guidance" "279": "Userscript CLAUDE guidance"
} }
+67 -76
View File
@@ -1,16 +1,16 @@
# Graph Report - mangaBookmark (2026-08-16) # Graph Report - mangaBookmark (2026-08-16)
## Corpus Check ## Corpus Check
- 113 files · ~279,161 words - 111 files · ~273,061 words
- Verdict: corpus is large enough that graph structure adds value. - Verdict: corpus is large enough that graph structure adds value.
## Summary ## Summary
- 1668 nodes · 3379 edges · 222 communities (68 shown, 154 thin omitted) - 1655 nodes · 3275 edges · 233 communities (63 shown, 170 thin omitted)
- Extraction: 91% EXTRACTED · 9% INFERRED · 0% AMBIGUOUS · INFERRED: 320 edges (avg confidence: 0.77) - Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 312 edges (avg confidence: 0.77)
- Token cost: 0 input · 0 output - Token cost: 0 input · 0 output
## Graph Freshness ## Graph Freshness
- Built from commit: `4f1cbcfd` - Built from commit: `dc269938`
- Run `git rev-parse HEAD` and compare to check if the graph is stale. - Run `git rev-parse HEAD` and compare to check if the graph is stale.
- Run `graphify update .` after code changes (no API cost). - Run `graphify update .` after code changes (no API cost).
@@ -34,19 +34,17 @@
- [[_COMMUNITY_Cloudflare bot scoring and poll cadence — what is actually documented|Cloudflare bot scoring and poll cadence — what is actually documented]] - [[_COMMUNITY_Cloudflare bot scoring and poll cadence — what is actually documented|Cloudflare bot scoring and poll cadence — what is actually documented]]
- [[_COMMUNITY_Go Code Style Guide|Go Code Style Guide]] - [[_COMMUNITY_Go Code Style Guide|Go Code Style Guide]]
- [[_COMMUNITY_Agent Skills|Agent Skills]] - [[_COMMUNITY_Agent Skills|Agent Skills]]
- [[_COMMUNITY_Store|Store]]
- [[_COMMUNITY_IO Performance Patterns|I/O Performance Patterns]] - [[_COMMUNITY_IO Performance Patterns|I/O Performance Patterns]]
- [[_COMMUNITY_CPU Optimization|CPU Optimization]] - [[_COMMUNITY_CPU Optimization|CPU Optimization]]
- [[_COMMUNITY_Caching Patterns|Caching Patterns]] - [[_COMMUNITY_Caching Patterns|Caching Patterns]]
- [[_COMMUNITY_Browser Entrypoint|Browser Entrypoint]] - [[_COMMUNITY_Browser Entrypoint|Browser Entrypoint]]
- [[_COMMUNITY_Memory Allocation & GC|Memory Allocation & GC]] - [[_COMMUNITY_Memory Allocation & GC|Memory Allocation & GC]]
- [[_COMMUNITY_Cover Fetcher Tests|Cover Fetcher Tests]] - [[_COMMUNITY_Cover Fetcher Tests|Cover Fetcher Tests]]
- [[_COMMUNITY_Open|Open]]
- [[_COMMUNITY_Find Skills Guide|Find Skills Guide]] - [[_COMMUNITY_Find Skills Guide|Find Skills Guide]]
- [[_COMMUNITY_Allocation Patterns|Allocation Patterns]] - [[_COMMUNITY_Allocation Patterns|Allocation Patterns]]
- [[_COMMUNITY_Observability & Alerting|Observability & Alerting]] - [[_COMMUNITY_Observability & Alerting|Observability & Alerting]]
- [[_COMMUNITY_Store|Store]] - [[_COMMUNITY_Memory Layout|Memory Layout]]
- [[_COMMUNITY_Open|Open]] - [[_COMMUNITY_Repo Hard Constraints|Repo Hard Constraints]]
- [[_COMMUNITY_Go Testing Guide|Go Testing Guide]] - [[_COMMUNITY_Go Testing Guide|Go Testing Guide]]
- [[_COMMUNITY_Session Store|Session Store]] - [[_COMMUNITY_Session Store|Session Store]]
- [[_COMMUNITY_Web UI Filter Logic|Web UI Filter Logic]] - [[_COMMUNITY_Web UI Filter Logic|Web UI Filter Logic]]
@@ -55,7 +53,6 @@
- [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]] - [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]]
- [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]] - [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]]
- [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]] - [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]]
- [[_COMMUNITY_pgtest.go|pgtest.go]]
- [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]] - [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]]
- [[_COMMUNITY_Ticket Workflow|Ticket Workflow]] - [[_COMMUNITY_Ticket Workflow|Ticket Workflow]]
- [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]] - [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]]
@@ -138,7 +135,6 @@
- [[_COMMUNITY_Identity comes from Discord OAuth; we store no passwords and send no email|Identity comes from Discord OAuth; we store no passwords and send no email]] - [[_COMMUNITY_Identity comes from Discord OAuth; we store no passwords and send no email|Identity comes from Discord OAuth; we store no passwords and send no email]]
- [[_COMMUNITY_The wire format stays flat and deliberately does not mirror the schema|The wire format stays flat and deliberately does not mirror the schema]] - [[_COMMUNITY_The wire format stays flat and deliberately does not mirror the schema|The wire format stays flat and deliberately does not mirror the schema]]
- [[_COMMUNITY_ADR-0005 On-demand browser sidecar|ADR-0005: On-demand browser sidecar]] - [[_COMMUNITY_ADR-0005 On-demand browser sidecar|ADR-0005: On-demand browser sidecar]]
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
- [[_COMMUNITY_Bookmark Manager|Bookmark Manager]] - [[_COMMUNITY_Bookmark Manager|Bookmark Manager]]
- [[_COMMUNITY_triage-labels|triage-labels.md]] - [[_COMMUNITY_triage-labels|triage-labels.md]]
- [[_COMMUNITY_Cross-Ticket Contract|Cross-Ticket Contract]] - [[_COMMUNITY_Cross-Ticket Contract|Cross-Ticket Contract]]
@@ -170,6 +166,17 @@
- [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]] - [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]]
- [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]] - [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]]
- [[_COMMUNITY_testingsynctest Deterministic Goroutine Testing|testing/synctest Deterministic Goroutine Testing]] - [[_COMMUNITY_testingsynctest Deterministic Goroutine Testing|testing/synctest Deterministic Goroutine Testing]]
- [[_COMMUNITY_API Package (Bookmark JSON Handlers)|API Package (Bookmark JSON Handlers)]]
- [[_COMMUNITY_Cover Acquisition & Serving Pipeline|Cover Acquisition & Serving Pipeline]]
- [[_COMMUNITY_Backend AGENTS.md Guidance|Backend AGENTS.md Guidance]]
- [[_COMMUNITY_HTTP Middleware (AuthGzipCORS)|HTTP Middleware (Auth/Gzip/CORS)]]
- [[_COMMUNITY_Latest Package (Site Parsers & Poller)|Latest Package (Site Parsers & Poller)]]
- [[_COMMUNITY_Latest-Chapter Poller|Latest-Chapter Poller]]
- [[_COMMUNITY_Main Composition Root|Main Composition Root]]
- [[_COMMUNITY_Migration-Owned Schema|Migration-Owned Schema]]
- [[_COMMUNITY_Session Package (Cookie Signing & Rate Limit)|Session Package (Cookie Signing & Rate Limit)]]
- [[_COMMUNITY_updated_at List-Order Rule|updated_at List-Order Rule]]
- [[_COMMUNITY_Userscript Package (Serving Handler)|Userscript Package (Serving Handler)]]
- [[_COMMUNITY_Backend CLAUDE.md Guidance|Backend CLAUDE.md Guidance]] - [[_COMMUNITY_Backend CLAUDE.md Guidance|Backend CLAUDE.md Guidance]]
- [[_COMMUNITY_Graphify Knowledge Graph (graphify-out)|Graphify Knowledge Graph (graphify-out/)]] - [[_COMMUNITY_Graphify Knowledge Graph (graphify-out)|Graphify Knowledge Graph (graphify-out/)]]
- [[_COMMUNITY_CLAUDE.md (Symlink to AGENTS.md)|CLAUDE.md (Symlink to AGENTS.md)]] - [[_COMMUNITY_CLAUDE.md (Symlink to AGENTS.md)|CLAUDE.md (Symlink to AGENTS.md)]]
@@ -234,22 +241,28 @@
- [[_COMMUNITY_Dark-First Design Constraint|Dark-First Design Constraint]] - [[_COMMUNITY_Dark-First Design Constraint|Dark-First Design Constraint]]
- [[_COMMUNITY_Discord Guild Membership|Discord Guild Membership]] - [[_COMMUNITY_Discord Guild Membership|Discord Guild Membership]]
- [[_COMMUNITY_Reader Isolation Invariant|Reader Isolation Invariant]] - [[_COMMUNITY_Reader Isolation Invariant|Reader Isolation Invariant]]
- [[_COMMUNITY_Browser Unit Redeploy|Browser Unit Redeploy]]
- [[_COMMUNITY_pg_dump Hot Backup|pg_dump Hot Backup]]
- [[_COMMUNITY_Redeploy Runbook|Redeploy Runbook]]
- [[_COMMUNITY_Rollback Strategy|Rollback Strategy]]
- [[_COMMUNITY_AGENTS|AGENTS.md]] - [[_COMMUNITY_AGENTS|AGENTS.md]]
- [[_COMMUNITY_Userscript CLAUDE guidance|Userscript CLAUDE guidance]] - [[_COMMUNITY_Userscript CLAUDE guidance|Userscript CLAUDE guidance]]
## God Nodes (most connected - your core abstractions) ## God Nodes (most connected - your core abstractions)
1. `testConfig()` - 53 edges 1. `testConfig()` - 53 edges
2. `newWebTestServer()` - 49 edges 2. `newWebTestServer()` - 49 edges
3. `newTestStore()` - 48 edges 3. `newTestStore()` - 42 edges
4. `newTestStore()` - 43 edges 4. `newTestStore()` - 41 edges
5. `e()` - 33 edges 5. `e()` - 33 edges
6. `Open()` - 29 edges 6. `Handler` - 29 edges
7. `Handler` - 29 edges 7. `ne()` - 28 edges
8. `ne()` - 28 edges 8. `De()` - 28 edges
9. `De()` - 28 edges 9. `Open()` - 27 edges
10. `Store` - 27 edges 10. `se()` - 27 edges
## Surprising Connections (you probably didn't know these) ## Surprising Connections (you probably didn't know these)
- `Browser Sidecar Service` --semantically_similar_to--> `Browser Sidecar (BROWSER_WS_URL)` [INFERRED] [semantically similar]
chrome/docker-compose.yml → backend/AGENTS.md
- `el()` --indirect_call--> `c()` [INFERRED] - `el()` --indirect_call--> `c()` [INFERRED]
userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js
- `el()` --indirect_call--> `c()` [INFERRED] - `el()` --indirect_call--> `c()` [INFERRED]
@@ -258,8 +271,6 @@
userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js
- `el()` --indirect_call--> `k()` [INFERRED] - `el()` --indirect_call--> `k()` [INFERRED]
userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js userscript/manga-bookmark.user.js → backend/internal/web/static/htmx.min.js
- `el()` --indirect_call--> `k()` [INFERRED]
userscript/novel-bookmark.user.js → backend/internal/web/static/htmx.min.js
## Import Cycles ## Import Cycles
- None detected. - None detected.
@@ -273,67 +284,67 @@
- **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85] - **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85]
- **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85] - **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85]
## Communities (222 total, 154 thin omitted) ## Communities (233 total, 170 thin omitted)
### Community 0 - "HTMX Library Internals" ### Community 0 - "HTMX Library Internals"
Cohesion: 0.08 Cohesion: 0.08
Nodes (101): A(), ae(), an(), at(), B(), be(), bn(), bt() (+93 more) Nodes (101): A(), ae(), an(), at(), B(), be(), bn(), bt() (+93 more)
### Community 1 - "Cover Fetch Test Helpers" ### Community 1 - "Cover Fetch Test Helpers"
Cohesion: 0.10 Cohesion: 0.09
Nodes (84): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecorder, T, TestListRendersAcquiredCover() (+76 more) Nodes (86): floatPtr(), testConfig(), Config, getCover(), Cookie, Handler, ResponseRecorder, T (+78 more)
### Community 2 - "Manga Userscript Adapters" ### Community 2 - "Manga Userscript Adapters"
Cohesion: 0.06 Cohesion: 0.06
Nodes (76): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+68 more) Nodes (77): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+69 more)
### Community 3 - "Novel Userscript Adapters" ### Community 3 - "Novel Userscript Adapters"
Cohesion: 0.06 Cohesion: 0.06
Nodes (79): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+71 more) Nodes (78): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+70 more)
### Community 4 - "Series Acquisition Tests" ### Community 4 - "Series Acquisition Tests"
Cohesion: 0.08 Cohesion: 0.10
Nodes (78): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+70 more) Nodes (67): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+59 more)
### Community 5 - "Bookmarks API Tests" ### Community 5 - "Bookmarks API Tests"
Cohesion: 0.06 Cohesion: 0.08
Nodes (74): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+66 more) Nodes (66): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+58 more)
### Community 7 - "Cover & Acquire Internals" ### Community 7 - "Cover & Acquire Internals"
Cohesion: 0.08 Cohesion: 0.05
Nodes (37): Addr, Context, Store, WaitGroup, isInterstitial(), defaultCoverResolver(), fetchCoverBytes(), Client (+29 more) Nodes (60): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+52 more)
### Community 8 - "System Architecture Concepts" ### Community 8 - "System Architecture Concepts"
Cohesion: 0.13 Cohesion: 0.10
Nodes (20): app.html — App Shell Template, Manga/Novel Library Switch, Bookmark Bucket Tabs, Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key (+12 more) Nodes (26): Confirm-Gated Destructive Actions, Discord OAuth & Guild-Membership Gate, Lifecycle Buckets (reading/archived/finished), Reader-Owned Store, HMAC-Derived Reader Credentials, Web Package (Browser UI + Templates), app.html — App Shell Template, Manga/Novel Library Switch (+18 more)
### Community 9 - "Session Middleware" ### Community 9 - "Session Middleware"
Cohesion: 0.08 Cohesion: 0.08
Nodes (35): ClearCookie(), ClientIP(), Duration, Mutex, Request, ResponseWriter, Time, isHTTPS() (+27 more) Nodes (34): ClearCookie(), ClientIP(), Duration, Mutex, Request, ResponseWriter, Time, isHTTPS() (+26 more)
### Community 10 - "Go Test Helpers" ### Community 10 - "Go Test Helpers"
Cohesion: 0.05 Cohesion: 0.05
Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more) Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more)
### Community 11 - "Store Tests" ### Community 11 - "Store Tests"
Cohesion: 0.14 Cohesion: 0.07
Nodes (45): scanSeries(), Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck() (+37 more) Nodes (79): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+71 more)
### Community 12 - "Bookmarks API Handler" ### Community 12 - "Bookmarks API Handler"
Cohesion: 0.08 Cohesion: 0.08
Nodes (32): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+24 more) Nodes (33): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+25 more)
### Community 13 - "Web UI Handlers" ### Community 13 - "Web UI Handlers"
Cohesion: 0.14 Cohesion: 0.06
Nodes (18): currentLib(), currentTab(), filterBookmarks(), Client, HandlerFunc, Request, ResponseWriter, Store (+10 more) Nodes (24): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, currentLib(), currentTab(), filterBookmarks(), Client (+16 more)
### Community 14 - "Go Error Handling" ### Community 14 - "Go Error Handling"
Cohesion: 0.06 Cohesion: 0.06
Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more) Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more)
### Community 15 - "CDP Browser Client" ### Community 15 - "CDP Browser Client"
Cohesion: 0.06 Cohesion: 0.08
Nodes (56): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+48 more) Nodes (30): awaitPromise(), browserConnectionLost(), classifyBrowserError(), Action, Context, Mutex, jsString(), kaganeAPIURL() (+22 more)
### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented" ### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented"
Cohesion: 0.06 Cohesion: 0.06
@@ -343,10 +354,6 @@ Nodes (33): 1.1 The score itself, 1.2 The detection engines (Enterprise Bot Mana
Cohesion: 0.08 Cohesion: 0.08
Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more) Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more)
### Community 19 - "Store"
Cohesion: 0.33
Nodes (5): ADR-0010: Poll Lanes — one independent Poll stream per Site, Constraints preserved, Decision, Tradeoffs and rejections, Why
### Community 20 - "I/O Performance Patterns" ### Community 20 - "I/O Performance Patterns"
Cohesion: 0.11 Cohesion: 0.11
Nodes (18): Avoid io.ReadAll for large payloads, Batch Operations, Buffered I/O, Cgo Overhead, Channel: batch processing from a stream, Concurrent Multi-Stage Pipelines, Connection pooling, Database: batch inserts over row-by-row (+10 more) Nodes (18): Avoid io.ReadAll for large payloads, Batch Operations, Buffered I/O, Cgo Overhead, Channel: batch processing from a stream, Concurrent Multi-Stage Pipelines, Connection pooling, Database: batch inserts over row-by-row (+10 more)
@@ -371,10 +378,6 @@ Nodes (15): Allocation Rate Reduction, Ballast pattern (pre-Go 1.19), Garbage Co
Cohesion: 0.33 Cohesion: 0.33
Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more) Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more)
### Community 26 - "Open"
Cohesion: 0.40
Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct
### Community 27 - "Find Skills Guide" ### Community 27 - "Find Skills Guide"
Cohesion: 0.14 Cohesion: 0.14
Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more) Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more)
@@ -387,13 +390,9 @@ Nodes (14): Allocation Patterns, Backing Array Leaks, Direct indexing vs append,
Cohesion: 0.22 Cohesion: 0.22
Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more) Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more)
### Community 31 - "Store" ### Community 31 - "Memory Layout"
Cohesion: 0.12 Cohesion: 0.40
Nodes (3): coverRelativePath(), coverSourceAddress(), Store Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct
### Community 32 - "Open"
Cohesion: 0.14
Nodes (20): applyMigration(), displayChapter(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen() (+12 more)
### Community 33 - "Go Testing Guide" ### Community 33 - "Go Testing Guide"
Cohesion: 0.20 Cohesion: 0.20
@@ -423,10 +422,6 @@ Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations,
Cohesion: 0.29 Cohesion: 0.29
Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider
### Community 42 - "pgtest.go"
Cohesion: 0.18
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
### Community 45 - "Go Perf Alert Rules" ### Community 45 - "Go Perf Alert Rules"
Cohesion: 0.50 Cohesion: 0.50
Nodes (4): Prometheus Alerting Rules (Go Performance), GoroutineLeak Alert, HighGCPauseTime Alert, MemoryNearLimit Alert Nodes (4): Prometheus Alerting Rules (Go Performance), GoroutineLeak Alert, HighGCPauseTime Alert, MemoryNearLimit Alert
@@ -448,12 +443,12 @@ Cohesion: 1.00
Nodes (3): Mirrored Double Bookmark Mark, Ember Flame Accent, BookmarkManager Logo Nodes (3): Mirrored Double Bookmark Mark, Ember Flame Accent, BookmarkManager Logo
### Community 103 - "bookmark-api Service" ### Community 103 - "bookmark-api Service"
Cohesion: 0.32 Cohesion: 0.24
Nodes (8): Browser Sidecar Service, CDP Endpoint (Tailnet-Bound :9222), Persistent Chrome Profile Volume, chrome/docker-compose.yml — Browser Deployable Unit, bookmark-api Prod Override, CDP Never on Shared Proxy Network, docker-compose.prod.yml — Production Override, Traefik Reverse Proxy Labels Nodes (10): Backend Go Service (stdlib net/http), Browser Sidecar (BROWSER_WS_URL), Browser Sidecar Service, CDP Endpoint (Tailnet-Bound :9222), Persistent Chrome Profile Volume, chrome/docker-compose.yml — Browser Deployable Unit, bookmark-api Prod Override, CDP Never on Shared Proxy Network (+2 more)
### Community 104 - "AGENTS.md" ### Community 104 - "AGENTS.md"
Cohesion: 0.12 Cohesion: 0.12
Nodes (14): Agent skills, Architecture, Commands, Comments, Design system, Domain docs, Forge: Gitea, not GitHub, graphify (+6 more) Nodes (15): Agent skills, AGENTS.md, Architecture, Commands, Comments, Design system, Domain docs, Forge: Gitea, not GitHub (+7 more)
### Community 105 - "reviewer.md" ### Community 105 - "reviewer.md"
Cohesion: 0.12 Cohesion: 0.12
@@ -535,33 +530,29 @@ Nodes (3): Consequence, The wire format stays flat and deliberately does not mir
Cohesion: 0.50 Cohesion: 0.50
Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision
### Community 125 - "sessions_test.go"
Cohesion: 0.48
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
### Community 273 - "AGENTS.md" ### Community 273 - "AGENTS.md"
Cohesion: 0.50 Cohesion: 0.50
Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust), Second script: `novel-bookmark.user.js`, Userscript structure (single IIFE, `manga-bookmark.user.js`) Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust), Second script: `novel-bookmark.user.js`, Userscript structure (single IIFE, `manga-bookmark.user.js`)
## Knowledge Gaps ## Knowledge Gaps
- **524 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+519 more) - **533 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+528 more)
These have ≤1 connection - possible missing edges or undocumented components. These have ≤1 connection - possible missing edges or undocumented components.
- **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. - **170 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
## Suggested Questions ## Suggested Questions
_Questions this graph is uniquely positioned to answer:_ _Questions this graph is uniquely positioned to answer:_
- **Why does `New()` connect `Series Acquisition Tests` to `Open`, `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`?** - **Why does `New()` connect `Series Acquisition Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Store Tests`, `Web UI Handlers`?**
_High betweenness centrality (0.045) - this node is a cross-community bridge._ _High betweenness centrality (0.047) - this node is a cross-community bridge._
- **Why does `Open()` connect `Open` to `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `pgtest.go`, `Store Tests`, `Store`?** - **Why does `Open()` connect `Store Tests` to `Cover Fetch Test Helpers`, `Web UI Handlers`, `Series Acquisition Tests`, `Bookmarks API Tests`?**
_High betweenness centrality (0.033) - this node is a cross-community bridge._ _High betweenness centrality (0.032) - this node is a cross-community bridge._
- **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?** - **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?**
_High betweenness centrality (0.027) - this node is a cross-community bridge._ _High betweenness centrality (0.025) - this node is a cross-community bridge._
- **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?** - **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?**
_`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._ _`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._
- **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?** - **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?**
_`newWebTestServer()` has 8 INFERRED edges - model-reasoned connections that need verification._ _`newWebTestServer()` has 8 INFERRED edges - model-reasoned connections that need verification._
- **Are the 12 inferred relationships involving `newTestStore()` (e.g. with `TestAcquireDoesNotBlockTheWrite()` and `TestAcquireFailureLeavesTheBookmarkIntact()`) actually correct?** - **Are the 6 inferred relationships involving `newTestStore()` (e.g. with `TestCreateAndGetSession()` and `TestDeleteSessionIsPerReader()`) actually correct?**
_`newTestStore()` has 12 INFERRED edges - model-reasoned connections that need verification._ _`newTestStore()` has 6 INFERRED edges - model-reasoned connections that need verification._
- **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?** - **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?**
_557 weakly-connected nodes found - possible documentation gaps or missing edges._ _573 weakly-connected nodes found - possible documentation gaps or missing edges._
File diff suppressed because one or more lines are too long
+1885 -3050
View File
File diff suppressed because it is too large Load Diff
+51 -61
View File
@@ -25,9 +25,9 @@
"semantic_hash": "dac242903b0e98c3e4395159d609e08e" "semantic_hash": "dac242903b0e98c3e4395159d609e08e"
}, },
"backend/main.go": { "backend/main.go": {
"mtime": 1786859325.126449, "mtime": 1786501521.228955,
"ast_hash": "4ad77c286522f97d14b6009ff37118e5", "ast_hash": "6e98a3ae91aaa132df251e43c4dfca6d",
"semantic_hash": "" "semantic_hash": "6e98a3ae91aaa132df251e43c4dfca6d"
}, },
"skills-lock.json": { "skills-lock.json": {
"mtime": 1784884678.6842625, "mtime": 1784884678.6842625,
@@ -140,18 +140,18 @@
"semantic_hash": "3a08979e4603aae5c32a58d5b6c39765" "semantic_hash": "3a08979e4603aae5c32a58d5b6c39765"
}, },
"CLAUDE.md": { "CLAUDE.md": {
"mtime": 1786857336.6414917, "mtime": 1786501942.7367291,
"ast_hash": "c79e49f912d7852f9832565a5f9a1c39", "ast_hash": "16b34d73e066d56d8f04523e6b3f6bd9",
"semantic_hash": "" "semantic_hash": ""
}, },
"DEPLOY.md": { "DEPLOY.md": {
"mtime": 1786859620.9977603, "mtime": 1786501942.7367291,
"ast_hash": "b7c2f813aded562ee9291c9baed795ad", "ast_hash": "3c7b785c44badb6dda6234d2b39a9290",
"semantic_hash": "" "semantic_hash": ""
}, },
"README.md": { "README.md": {
"mtime": 1786859273.3238454, "mtime": 1786499529.7651505,
"ast_hash": "81af12a0a2d43e791efd030b5f4d6cbc", "ast_hash": "9d6be8aa8a2946c23ad48d8f2864b5ca",
"semantic_hash": "" "semantic_hash": ""
}, },
"docker-compose.prod.yml": { "docker-compose.prod.yml": {
@@ -160,8 +160,8 @@
"semantic_hash": "0751998a532297b8ac507a01ec48dc31" "semantic_hash": "0751998a532297b8ac507a01ec48dc31"
}, },
"docker-compose.yml": { "docker-compose.yml": {
"mtime": 1786859258.6255276, "mtime": 1786499529.7725692,
"ast_hash": "d3b53a8f42a8e0acb4fc4306ea42c093", "ast_hash": "124fd581bf0a662ff15012abfdb40a92",
"semantic_hash": "" "semantic_hash": ""
}, },
".claude/settings.json": { ".claude/settings.json": {
@@ -175,9 +175,9 @@
"semantic_hash": "8f1b093b59eb1ed81bc7fc0c22495c50" "semantic_hash": "8f1b093b59eb1ed81bc7fc0c22495c50"
}, },
"backend/main_test.go": { "backend/main_test.go": {
"mtime": 1786859172.6449091, "mtime": 1786501521.228955,
"ast_hash": "0a5d4dbdc770b40c329ccccc899b59f4", "ast_hash": "8a165955cf28ad47481fec5ea7afb3d6",
"semantic_hash": "" "semantic_hash": "8a165955cf28ad47481fec5ea7afb3d6"
}, },
".claude/settings.local.json": { ".claude/settings.local.json": {
"mtime": 1785697645.350201, "mtime": 1785697645.350201,
@@ -200,8 +200,8 @@
"semantic_hash": "e69a8340a371579ca3ea689660f7d7bd" "semantic_hash": "e69a8340a371579ca3ea689660f7d7bd"
}, },
"AGENTS.md": { "AGENTS.md": {
"mtime": 1786857336.6414917, "mtime": 1786501942.7367291,
"ast_hash": "c79e49f912d7852f9832565a5f9a1c39", "ast_hash": "16b34d73e066d56d8f04523e6b3f6bd9",
"semantic_hash": "" "semantic_hash": ""
}, },
"userscript/test/logic.test.js": { "userscript/test/logic.test.js": {
@@ -215,9 +215,9 @@
"semantic_hash": "8f3c0132eb4787a2c8736eb99f7689af" "semantic_hash": "8f3c0132eb4787a2c8736eb99f7689af"
}, },
"REDEPLOY.md": { "REDEPLOY.md": {
"mtime": 1786857336.6414917, "mtime": 1786363889.552731,
"ast_hash": "e5e910f0244a040e2ccdc669b17eb4db", "ast_hash": "d0baf08b95e7b5986234a9f36759c12e",
"semantic_hash": "" "semantic_hash": "d0baf08b95e7b5986234a9f36759c12e"
}, },
"docs/design-system.md": { "docs/design-system.md": {
"mtime": 1786022513.9623306, "mtime": 1786022513.9623306,
@@ -245,14 +245,14 @@
"semantic_hash": "385b36f58488b7e6d93eb6d6034e9ee3" "semantic_hash": "385b36f58488b7e6d93eb6d6034e9ee3"
}, },
"backend/internal/latest/browser.go": { "backend/internal/latest/browser.go": {
"mtime": 1786859355.8677974, "mtime": 1786499529.7688599,
"ast_hash": "75f34a974568d122681939dd297944a2", "ast_hash": "ed129a7f00601ea90c877ff29fa21220",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/browser_test.go": { "backend/internal/latest/browser_test.go": {
"mtime": 1786857336.6414917, "mtime": 1786262323.6964688,
"ast_hash": "800fa6aa471ada054a3c48943ad17ab7", "ast_hash": "e900f92971486f47d7ef76e9a95217fe",
"semantic_hash": "" "semantic_hash": "e900f92971486f47d7ef76e9a95217fe"
}, },
"backend/internal/latest/fetch.go": { "backend/internal/latest/fetch.go": {
"mtime": 1786499529.7688599, "mtime": 1786499529.7688599,
@@ -260,23 +260,23 @@
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/poller.go": { "backend/internal/latest/poller.go": {
"mtime": 1786860083.6713927, "mtime": 1786499529.7688599,
"ast_hash": "be728405cde3df49baf72bd7837f2971", "ast_hash": "44fef6074ac2eaffc8233f46aad5236b",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/poller_test.go": { "backend/internal/latest/poller_test.go": {
"mtime": 1786860168.103984, "mtime": 1786499529.7688599,
"ast_hash": "5dba515d28c0523500c1ceb581a54e58", "ast_hash": "64bc838c822f1bf33bbf9e291215454b",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/sites.go": { "backend/internal/latest/sites.go": {
"mtime": 1786860007.8224697, "mtime": 1786499529.7688599,
"ast_hash": "9b90f9b710ccbd0d0a0a6a0dd721ff3c", "ast_hash": "b744cc685363317a526cc3bebceea39e",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/sites_test.go": { "backend/internal/latest/sites_test.go": {
"mtime": 1786857336.6414917, "mtime": 1786499529.7725692,
"ast_hash": "0db2028a6073f342fee61ad15c2e5f0f", "ast_hash": "eabca9014a306e3c71d238b0ae499f61",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/smoke_image_test.go": { "backend/internal/latest/smoke_image_test.go": {
@@ -340,14 +340,14 @@
"semantic_hash": "0b6764a0ee20f5cb7748eecd31a1d220" "semantic_hash": "0b6764a0ee20f5cb7748eecd31a1d220"
}, },
"backend/internal/store/store.go": { "backend/internal/store/store.go": {
"mtime": 1786860012.3431706, "mtime": 1786363889.5602942,
"ast_hash": "c62e73142386c58ab833d404d0c24dc4", "ast_hash": "54367a8ab043983e2491b2eb2650961c",
"semantic_hash": "" "semantic_hash": "54367a8ab043983e2491b2eb2650961c"
}, },
"backend/internal/store/store_test.go": { "backend/internal/store/store_test.go": {
"mtime": 1786858248.9426548, "mtime": 1786363889.5602942,
"ast_hash": "039729554517960d1ae98f8409baf4ba", "ast_hash": "dc823fd77bcce2268114e31d759b20a5",
"semantic_hash": "" "semantic_hash": "dc823fd77bcce2268114e31d759b20a5"
}, },
"backend/internal/userscript/userscript.go": { "backend/internal/userscript/userscript.go": {
"mtime": 1786216141.692644, "mtime": 1786216141.692644,
@@ -415,8 +415,8 @@
"semantic_hash": "e44a2f6f624db044e19508bc5ab05592" "semantic_hash": "e44a2f6f624db044e19508bc5ab05592"
}, },
"CONTEXT.md": { "CONTEXT.md": {
"mtime": 1786859395.4497814, "mtime": 1786850406.6525955,
"ast_hash": "4aafbce0b046e6e34734fb414df818ce", "ast_hash": "24548f60414b4c5ff58538acaada5345",
"semantic_hash": "" "semantic_hash": ""
}, },
"CUTOVER.md": { "CUTOVER.md": {
@@ -425,13 +425,13 @@
"semantic_hash": "6c6f3e4c4c2f57867894280bce728c50" "semantic_hash": "6c6f3e4c4c2f57867894280bce728c50"
}, },
"backend/AGENTS.md": { "backend/AGENTS.md": {
"mtime": 1786860230.7890837, "mtime": 1786501942.7367291,
"ast_hash": "7ceec7c4576f6e91eaf02c200a0dd1b3", "ast_hash": "23d8dbbcb9796c679d25a74bb3847f88",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/CLAUDE.md": { "backend/CLAUDE.md": {
"mtime": 1786860230.7890837, "mtime": 1786501942.7367291,
"ast_hash": "7ceec7c4576f6e91eaf02c200a0dd1b3", "ast_hash": "23d8dbbcb9796c679d25a74bb3847f88",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/web/templates/app.html": { "backend/internal/web/templates/app.html": {
@@ -560,9 +560,9 @@
"semantic_hash": "46cf7822d4f667e3cab36b547abe5e97" "semantic_hash": "46cf7822d4f667e3cab36b547abe5e97"
}, },
"backend/internal/latest/cover.go": { "backend/internal/latest/cover.go": {
"mtime": 1786857336.6414917, "mtime": 1786363889.5565126,
"ast_hash": "d5f2248c3d11de74bf5a3977651b17c2", "ast_hash": "e6749cfe3cd7c2e71d4392dde84f55f9",
"semantic_hash": "" "semantic_hash": "e6749cfe3cd7c2e71d4392dde84f55f9"
}, },
"backend/internal/latest/cover_fetch_test.go": { "backend/internal/latest/cover_fetch_test.go": {
"mtime": 1786363889.5565126, "mtime": 1786363889.5565126,
@@ -570,8 +570,8 @@
"semantic_hash": "60d9eb7c59a3751baf4f31c7655217e7" "semantic_hash": "60d9eb7c59a3751baf4f31c7655217e7"
}, },
"backend/internal/latest/acquire.go": { "backend/internal/latest/acquire.go": {
"mtime": 1786859346.3297389, "mtime": 1786499529.7688599,
"ast_hash": "d605e3c94a62fc7787efbc139696b9d2", "ast_hash": "6c1ad34bbe9f5b49d0fd1eae9093f55d",
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/acquire_test.go": { "backend/internal/latest/acquire_test.go": {
@@ -615,23 +615,13 @@
"semantic_hash": "" "semantic_hash": ""
}, },
"backend/internal/latest/read.go": { "backend/internal/latest/read.go": {
"mtime": 1786859355.8679621, "mtime": 1786499529.7688599,
"ast_hash": "9f039cc3ad74f803d7621f7ef4157bf2", "ast_hash": "3cf29046ddaef39fafb1df70b9f9ae8c",
"semantic_hash": "" "semantic_hash": ""
}, },
"docs/research/cloudflare-bot-scoring-and-poll-cadence.md": { "docs/research/cloudflare-bot-scoring-and-poll-cadence.md": {
"mtime": 1786501942.7367291, "mtime": 1786501942.7367291,
"ast_hash": "1aa17575ab20f2f36583602999a6a60f", "ast_hash": "1aa17575ab20f2f36583602999a6a60f",
"semantic_hash": "" "semantic_hash": ""
},
"backend/internal/latest/smoke_comix_test.go": {
"mtime": 1786857336.6414917,
"ast_hash": "111fdbb75fc68ac2ab1013bc916063cf",
"semantic_hash": ""
},
"docs/adr/0010-poll-lanes-per-site-pace.md": {
"mtime": 1786860214.4637265,
"ast_hash": "dc19d75f034ca920d93b9c71e6ca28e6",
"semantic_hash": ""
} }
} }