Compare commits
28 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 264839e798 | |||
| cbe0a28921 | |||
| e45445cb20 | |||
| 4a95657425 | |||
| 5b7adf5f2c | |||
| 00fa237151 | |||
| fe2cd12049 | |||
| c432a3be30 | |||
| 6f9109c885 | |||
| f5b52e48c8 | |||
| bf08d6e65c | |||
| d72c48295d | |||
| fc8a40cfc6 | |||
| cb2b104e63 | |||
| 134c9307b1 | |||
| 09a094ca67 | |||
| e8a3c5f826 | |||
| 14990bde21 | |||
| 503fb49d0a | |||
| e0b9063d9e | |||
| fd1131d11d | |||
| 030ffdc26e | |||
| 0a245a0dde | |||
| 249f11e1fe | |||
| f5d3fe58ec | |||
| 766aa8f00d | |||
| 550b258c59 | |||
| 3ac865cd08 |
@@ -0,0 +1,75 @@
|
|||||||
|
---
|
||||||
|
name: gitea
|
||||||
|
description: Use for every forge operation in this repo — read, create, comment on, label, close, or search an issue; create, review, merge, or check out a PR; and whenever `gh`, `issue://`, or `pr://` fails or a ticket number is ambiguous. This repo's forge is self-hosted Gitea driven by `tea`, not GitHub.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Gitea, not GitHub
|
||||||
|
|
||||||
|
`origin` is the self-hosted Gitea instance `gitea.violetcrown.my.id`, repo
|
||||||
|
`sulthan/mangaBookmark`. Everything past plain git goes through
|
||||||
|
[`tea`](https://gitea.com/gitea/tea) (0.14.2 on this machine).
|
||||||
|
|
||||||
|
**`gh` is not installed**, so the harness's `issue://<n>` and `pr://<n>` URIs
|
||||||
|
error out (`GitHub CLI (gh) is not installed`, measured 2026-08-17). There is no
|
||||||
|
fallback to add — read tickets with `tea`.
|
||||||
|
|
||||||
|
`tea` infers the repo from `origin`; auth lives in `tea login`, never a
|
||||||
|
`GH_TOKEN`. Your Gitea username comes from `tea login list` — `tea` has no `@me`.
|
||||||
|
|
||||||
|
Flags are the environment's job: run `tea <command> --help` rather than trusting
|
||||||
|
a remembered flag. This file carries only what `--help` will not tell you.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
| Job | Command |
|
||||||
|
|---|---|
|
||||||
|
| Read | `tea issue <n> --comments` / `tea pr <n> --comments` — `--comments` is not optional |
|
||||||
|
| List | `tea issue list --state open\|closed\|all -o json --fields index,title,body,labels,state,author` |
|
||||||
|
| Search | `tea issue list -k "<keyword>" -L "<label>" -A "<author>"` (`-K all` also searches PRs) |
|
||||||
|
| Create | `tea issue create -t "..." -d "..."` (`-L`, `-a` optional) |
|
||||||
|
| Comment | `tea comment <n> "..."` |
|
||||||
|
| Label | `tea issue edit <n> --add-labels "..."` / `--remove-labels "..."` |
|
||||||
|
| Close | `tea issue close <n>` / `tea pr close <n>` |
|
||||||
|
| PR | `tea pr create --head <branch> --base main -t "..." -d "..."`, `tea pr checkout <n>`, `tea pr review <n>`, `tea pr merge <n>` |
|
||||||
|
|
||||||
|
## Traps
|
||||||
|
|
||||||
|
- **A bare read is a truncated read.** Without `--comments`, `tea issue <n>` and
|
||||||
|
`tea pr <n>` print the opening body and drop every comment silently — no
|
||||||
|
prompt, no marker, no hint that more exists (measured 2026-08-17: issue #123
|
||||||
|
prints 40 lines bare, 132 with `--comments`). The comments are where the
|
||||||
|
decisions live and the body is usually the stalest part of the ticket, so
|
||||||
|
**every read that exists to understand an issue or PR passes `--comments`**,
|
||||||
|
and understanding means body plus all comments plus whatever ticket they point
|
||||||
|
at. Comment count is `tea issue list --fields index,comments`, so a read that
|
||||||
|
shows fewer than that is incomplete. A PR's review comments are a second
|
||||||
|
stream: `tea pr review-comments <n>`.
|
||||||
|
- **One index space for issues and PRs.** A bare `#42` may be either: try
|
||||||
|
`tea pr 42`, fall back to `tea issue 42`. Say which one you found.
|
||||||
|
- **Output is rendered boxes**, not plain text. Anything you parse needs
|
||||||
|
`-o json`, plus `--fields` to keep the payload small. `tea pr create` prints
|
||||||
|
the PR URL on its last line.
|
||||||
|
- **`close` takes no `--comment`.** Comment with `tea comment <n>`, then close.
|
||||||
|
- **Gitea will not auto-create a label.** `tea labels list` first; missing one
|
||||||
|
gets `tea labels create --name "..." --color "#rrggbb"` before the `edit`.
|
||||||
|
- **Multi-line bodies go through a heredoc**, never inline escapes:
|
||||||
|
```bash
|
||||||
|
tea issue create -t "Title" -d "$(cat <<'EOF'
|
||||||
|
body line one
|
||||||
|
|
||||||
|
- acceptance criterion
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
```
|
||||||
|
- **No sub-issue and no dependency command.** Gitea's API has issue
|
||||||
|
dependencies, `tea` does not expose them, so parentage and blocking live as
|
||||||
|
body lines — the shapes are in `docs/agents/issue-tracker.md`.
|
||||||
|
|
||||||
|
## Conventions this repo layers on top
|
||||||
|
|
||||||
|
Ticket bodies, wayfinding issues, and the PR-as-request-surface flag:
|
||||||
|
`docs/agents/issue-tracker.md`. Triage label strings: `docs/agents/triage-labels.md`.
|
||||||
|
A label named there still has to exist in the tracker before `--add-labels`.
|
||||||
|
|
||||||
|
Finish a forge action by stating the number you touched and its state after —
|
||||||
|
"commented and closed #71" — so the write is checkable without a second query.
|
||||||
@@ -11,7 +11,7 @@ to the tracker. You do not write the implementation — every line of ticket cod
|
|||||||
is written by a `ticket-implementer` subagent in its own git worktree. Reach for
|
is written by a `ticket-implementer` subagent in its own git worktree. Reach for
|
||||||
the editor yourself only for a merge conflict resolution.
|
the editor yourself only for a merge conflict resolution.
|
||||||
|
|
||||||
Ticket source and `tea` usage: `docs/agents/issue-tracker.md`.
|
Ticket source and tracker conventions: `docs/agents/issue-tracker.md`. `tea` usage: skill `gitea`.
|
||||||
|
|
||||||
## 1. Collect the tickets
|
## 1. Collect the tickets
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# AGENTS.md
|
# AGENTS.md
|
||||||
|
|
||||||
Guidance for OpenCode (and Claude Code) working in this repo.
|
Repo-wide guidance for coding agents.
|
||||||
|
|
||||||
## What this is
|
## What this is
|
||||||
|
|
||||||
@@ -13,15 +13,18 @@ One backend, one `bookmarks` table: a `kind` column (`manga`|`novel`) splits the
|
|||||||
|
|
||||||
## Hard constraints (drive design — don't violate)
|
## Hard constraints (drive design — don't violate)
|
||||||
|
|
||||||
|
Nothing below is derivable from reading the code — it is why the code looks the
|
||||||
|
way it does, plus dated measurements against services we don't control.
|
||||||
|
|
||||||
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
||||||
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
||||||
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
||||||
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
|
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional.
|
||||||
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
||||||
- Cloudflare's block on manga sites is **per-zone configuration plus request fingerprint, not IP reputation — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — a Site can turn its protection on overnight, which is exactly what comix.to did on 2026-08-12. An earlier version of this line blamed "Cloudflare's bot scoring"; that was wrong. The 1-99 bot score is Enterprise Bot Management only and does not exist for a free-plan zone, and no per-IP request rate is documented as an input to challenge issuance — `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
- Cloudflare's block on manga sites is **per-zone configuration plus request fingerprint, not IP reputation — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — a Site can turn its protection on overnight, which is exactly what comix.to did on 2026-08-12. An earlier version of this line blamed "Cloudflare's bot scoring"; that was wrong. The 1-99 bot score is Enterprise Bot Management only and does not exist for a free-plan zone, and no per-IP request rate is documented as an input to challenge issuance. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
|
||||||
- **kagane.to, comix.to and novelfull.com are the exception to the above** — all three sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`). When that's unset, kagane and comix are skipped entirely (a plain fetch would only retrieve a challenge page) while novelfull pages are still attempted over plain TLS — its challenge is a live time-varying fact and its cover bytes never need the browser. comix turned hostile on 2026-08-12 (#98): its cover host `static.comix.to` is gated too, so its cover bytes go through the browser as well, and its page is read as an in-tab `fetch()` of the series URL rather than a rendered DOM — comix is an SPA, and rendering costs ~65 requests for the same server-rendered HTML one fetch returns. The three other sites poll fine over plain TLS.
|
- **kagane.to, comix.to and novelfull.com are the exception to the above** — all three sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`). When that's unset, kagane and comix are skipped entirely (a plain fetch would only retrieve a challenge page) while novelfull pages are still attempted over plain TLS — its challenge is a live time-varying fact and its cover bytes never need the browser. comix turned hostile on 2026-08-12: its cover host `static.comix.to` is gated too, so its cover bytes go through the browser as well, and its page is read as an in-tab `fetch()` of the series URL rather than a rendered DOM — comix is an SPA, and rendering costs ~65 requests for the same server-rendered HTML one fetch returns. The three other sites poll fine over plain TLS.
|
||||||
- **The CDP browser must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a non-UTC clock zone cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, ignoring the file's contents, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
|
- **The CDP browser must look like a real browser, and stock headless images don't.** Measured 2026-08-08 against kagane.to, all from the same IP: `chromedp/headless-shell:stable` never cleared the challenge in 90s (`navigator.webdriver` true, empty plugin list, Chromium-branded client hints — suppressing `webdriver` alone changed nothing); `zenika/alpine-chrome` ships Chrome 124, refused outright; real Chrome with the default `--headless=new` UA never cleared, because the UA says `HeadlessChrome`; real Chrome with a stock UA **and** a non-UTC clock zone cleared in ~4s. Hence `chrome/` — a Debian image with `google-chrome-stable`, a version-derived UA, and `TZ`/`BROWSER_TZ`. Chrome reads the zone *name* through ICU from `/etc/localtime`'s symlink target, ignoring the file's contents, so mounting the host's `/etc/localtime` does **not** work; `/etc/timezone` is mounted instead.
|
||||||
- **The browser is not in the API stack and must not be put back.** It's its own compose unit (`chrome/docker-compose.yml`) on a second machine, reached over the tailnet — it held 471 MiB on a 1974 MiB swapless VPS, and a residential egress avoids the cloud-hosting-IP signature Bot Fight Mode documentedly challenges (ADR-0006; not a better "score" — free-plan zones have no score). Consequences that constrain code: `BROWSER_WS_URL` must be a tailnet **IP** (a MagicDNS name 500s at `/json/version`, same trap as the old Docker service name); the CDP port binds to the tailnet address only, since CDP authenticates nothing and that host has a real LAN; and the browser is on-demand (ADR-0005), so an unreachable or asleep one must degrade exactly as an unset `BROWSER_WS_URL` — plain-TLS libraries unaffected, kagane/comix logged and skipped, stored covers still served. Never add `chromedp.NoModifyURL`: discovery per fetch is what makes a restarted Chrome invisible.
|
- **The browser is not in the API stack and must not be put back.** It's its own compose unit (`chrome/docker-compose.yml`) on a second machine, reached over the tailnet — it held 471 MiB on a 1974 MiB swapless VPS, and a residential egress avoids the cloud-hosting-IP signature Bot Fight Mode documentedly challenges (not a better "score" — free-plan zones have no score). Consequences that constrain code: `BROWSER_WS_URL` must be a tailnet **IP** (a MagicDNS name 500s at `/json/version`, same trap as the old Docker service name); the CDP port binds to the tailnet address only, since CDP authenticates nothing and that host has a real LAN; and the browser is on-demand, so an unreachable or asleep one must degrade exactly as an unset `BROWSER_WS_URL` — plain-TLS libraries unaffected, kagane/comix logged and skipped, stored covers still served. Never add `chromedp.NoModifyURL`: discovery per fetch is what makes a restarted Chrome invisible.
|
||||||
- **UTC is the tell, not a country mismatch.** A UTC clock is the datacenter default, and the challenge refuses it; any real zone clears. Measured 2026-08-08, identical container, one Indonesian egress IP: UTC never cleared in 60s (twice), while `Asia/Jakarta` **and** `America/New_York` both cleared in 4s. An earlier note here claimed the zone had to match the egress IP's country — that was wrong, inferred from the host clock (`Asia/Bangkok`) rather than the measured egress. A second earlier claim, that Cloudflare "scores" a UTC clock, was also wrong: the measurement is real but the mechanism is not documented anywhere — Cloudflare publishes no timezone signal, and free-plan zones carry no score at all. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one.
|
- **UTC is the tell, not a country mismatch.** A UTC clock is the datacenter default, and the challenge refuses it; any real zone clears. Measured 2026-08-08, identical container, one Indonesian egress IP: UTC never cleared in 60s (twice), while `Asia/Jakarta` **and** `America/New_York` both cleared in 4s. An earlier note here claimed the zone had to match the egress IP's country — that was wrong, inferred from the host clock (`Asia/Bangkok`) rather than the measured egress. A second earlier claim, that Cloudflare "scores" a UTC clock, was also wrong: the measurement is real but the mechanism is not documented anywhere — Cloudflare publishes no timezone signal, and free-plan zones carry no score at all. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one.
|
||||||
- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
|
- **A challenged page needs the tab kept open.** The interstitial takes seconds to solve and only then writes clearance into the browser's shared cookie jar. Navigate-read-close never clears anything; `BrowserFetcher.run` holds one tab and re-reads until the payload arrives.
|
||||||
|
|
||||||
@@ -36,7 +39,7 @@ Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage c
|
|||||||
on-demand Chrome, separate machine (chrome/)
|
on-demand Chrome, separate machine (chrome/)
|
||||||
```
|
```
|
||||||
|
|
||||||
Two deployable units on two machines: the API stack (`docker-compose.yml` + `docker-compose.prod.yml`, on the VPS) and the browser (`chrome/docker-compose.yml`, on the home machine). They share nothing but `BROWSER_WS_URL` and update independently. Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`. Deploy order `DEPLOY.md` (§7 for the browser), redeploy `REDEPLOY.md` (§8 for the browser).
|
Two deployable units on two machines: the API stack (`docker-compose.yml` + `docker-compose.prod.yml`, on the VPS) and the browser (`chrome/docker-compose.yml`, on the home machine). They share nothing but `BROWSER_WS_URL` and update independently. Backend-specific detail lives in `backend/AGENTS.md`, userscript-specific detail in `userscript/AGENTS.md`.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
@@ -64,28 +67,22 @@ Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTI
|
|||||||
|
|
||||||
## Forge: Gitea, not GitHub
|
## Forge: Gitea, not GitHub
|
||||||
|
|
||||||
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
|
`origin` is self-hosted Gitea (`gitea.violetcrown.my.id`, repo `sulthan/mangaBookmark`), so **`gh` don't work here and the `issue://`/`pr://` URIs error out — drive the forge with `tea`.** How to run it — commands, traps, JSON output: skill `gitea`. Tracker conventions (ticket bodies, wayfinding, PR-as-request-surface flag): `docs/agents/issue-tracker.md`. Triage label strings: `docs/agents/triage-labels.md`.
|
||||||
|
|
||||||
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
|
|
||||||
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
|
|
||||||
- Issues: `tea issue create`, `tea issue list`
|
|
||||||
- Auth lives in `tea login`, not `GH_TOKEN` env var.
|
|
||||||
|
|
||||||
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
|
|
||||||
|
|
||||||
## Design system
|
## Design system
|
||||||
|
|
||||||
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
|
Web UI + userscript panel follow **Cinder**. Tokens are the `:root` block in
|
||||||
— source of truth Claude Design project `BookmarkManager Web UI`
|
`backend/internal/web/static/style.css`; that file, `backend/internal/web/templates/*`,
|
||||||
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
|
and the userscript `TEMPLATE`/`CSS` are the only places it is expressed.
|
||||||
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
|
Source of truth for the visual language is the Claude Design project
|
||||||
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
|
`BookmarkManager Web UI` (`969ac210-fe02-4c01-ae1b-9a271dcc779a`).
|
||||||
state (busy, error, destruction) may use `--ember`; destruction gets
|
|
||||||
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
|
Core law: **ember means new chapter only** — no other state (busy, error,
|
||||||
only (never hardcode hex outside `:root`), both colour branches touched
|
destruction) may use `--ember`; destruction gets `--danger`. No
|
||||||
together. Any move that pulls series out of list (archive/finish/remove)
|
cards/corners/shadows, one `--measure: 760px` column, tokens only (never
|
||||||
must be confirm-gated via its own `.confirm-row`; only restore fires
|
hardcode hex outside `:root`), both colour branches touched together. Any move
|
||||||
instantly.
|
that pulls a series out of the list (archive/finish/remove) must be
|
||||||
|
confirm-gated via its own `.confirm-row`; only restore fires instantly.
|
||||||
|
|
||||||
## Security invariants
|
## Security invariants
|
||||||
|
|
||||||
@@ -127,13 +124,19 @@ Review gate: auth, CORS, session, crypto, and the fetch gate are security-critic
|
|||||||
## Comments
|
## Comments
|
||||||
|
|
||||||
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
||||||
|
Wrong comment worse than none: it misleads readers and measurably degrades
|
||||||
|
LLM performance on the file. Missing comment costs little. Bias to fewer.
|
||||||
|
|
||||||
Write for:
|
Docstring on public/exported surface — exception, near-always worth it.
|
||||||
- Why not what. Tradeoffs, non-obvious decisions.
|
Contract only: what it takes, returns, throws, mutates; units; pre/post
|
||||||
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
|
conditions. Not a restatement of the body. Skip on private/obvious.
|
||||||
|
|
||||||
|
Inline — write for:
|
||||||
|
- Why not what. Tradeoffs, non-obvious decisions, rejected alternatives.
|
||||||
|
- heavy detail looking incidental — say so if "simplify" breaks it.
|
||||||
- Non-local consequence, invisible from function alone.
|
- Non-local consequence, invisible from function alone.
|
||||||
- Wire format / encoding / interface contract — save callers re-deriving.
|
- Wire format / encoding / ordering / invariant — save callers re-deriving.
|
||||||
- Gotcha/workaround, with ref if exists.
|
- Gotcha/workaround, with ref (issue, RFC, vendor bug) if exists.
|
||||||
- Domain/business rule not derivable from code.
|
- Domain/business rule not derivable from code.
|
||||||
|
|
||||||
Skip:
|
Skip:
|
||||||
@@ -142,24 +145,49 @@ Skip:
|
|||||||
- Banners, dividers, `// helpers`.
|
- Banners, dividers, `// helpers`.
|
||||||
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
||||||
- Commented-out code — delete.
|
- Commented-out code — delete.
|
||||||
- TODO without concrete action.
|
- TODO without concrete action + owner.
|
||||||
|
- Narrating the plan you just reasoned through. Plan in prose or in your head;
|
||||||
|
ship the code, not the transcript.
|
||||||
|
- Anything restating a name that could be fixed by renaming instead.
|
||||||
|
|
||||||
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
|
Staleness filter: if the comment describes something likely to change
|
||||||
|
independently of this line, it will rot and start lying. Either anchor it to
|
||||||
|
something stable, assert it in a test, or leave it out.
|
||||||
|
|
||||||
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
|
Style: one dense comment over a function beats one per line inside. Tight; no
|
||||||
|
worked example unless the bug is subtle. On edit, update or delete stale
|
||||||
|
comments in the code you touch — silence beats a lie.
|
||||||
|
|
||||||
## Agent skills
|
Test: "competent reader get this from code in a few sec?" Yes → skip.
|
||||||
|
Needs detour through another file/spec/git-blame/external doc → write it.
|
||||||
|
|
||||||
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
## Writing an AGENTS.md
|
||||||
|
|
||||||
### Issue tracker
|
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md`
|
||||||
|
in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
||||||
|
|
||||||
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
|
**Cite code, never docs, issues, or plans.** A spec, ADR, plan file, or Gitea
|
||||||
|
issue records what was true when it was written and then goes stale silently;
|
||||||
|
an agent that follows the pointer reads a decision that may already have been
|
||||||
|
reversed. Code is the only source true at read time — cite a package, file,
|
||||||
|
symbol, env var, or route. The sole non-code exception is a sibling
|
||||||
|
`AGENTS.md`. If a doc holds a fact an agent needs, restate the fact here rather
|
||||||
|
than linking to it.
|
||||||
|
|
||||||
### Triage labels
|
**State a fact in prose only if the code cannot answer it.** Split by
|
||||||
|
derivability:
|
||||||
|
|
||||||
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
- *Structure* — packages, routes, env vars, columns, struct fields. Rots fast,
|
||||||
|
cheap to re-read. **Name the symbol, write nothing else.**
|
||||||
|
- *Mechanism* — what a function does, how a flow proceeds. **Name the symbol
|
||||||
|
plus at most one line of orientation.**
|
||||||
|
- *Rationale* — why it is this way, what a "simplify" would break, what was
|
||||||
|
tried and rejected. Not in the code and cannot be re-derived. **Write it out.**
|
||||||
|
- *Measurement* — an observation against something we don't control. **Write it
|
||||||
|
out with the date**; a dated fact is honest, an undated one pretends to be
|
||||||
|
permanent.
|
||||||
|
|
||||||
### Domain docs
|
Restating mechanism in prose is how these files rot: the code changes, the
|
||||||
|
paragraph doesn't, and the next agent trusts the paragraph. A pointer degrades
|
||||||
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
|
more honestly — and every symbol you name must actually exist, since a dead
|
||||||
|
pointer is a bug, not a stale sentence.
|
||||||
|
|||||||
+53
-3
@@ -43,6 +43,13 @@ Readers: Progress, Favourite, Lifecycle bucket. Facts about the Series itself be
|
|||||||
to the Series, not here.
|
to the Series, not here.
|
||||||
_Avoid_: entry, item, record, subscription
|
_Avoid_: entry, item, record, subscription
|
||||||
|
|
||||||
|
**Orphan Series**:
|
||||||
|
A Series no Reader bookmarks. Removing a Bookmark never removes the Series, so the row
|
||||||
|
outlives every relationship to it: nothing reads it, no Poll visits it, and it still owns
|
||||||
|
a Cover. A state of the Series, not a Lifecycle bucket — it says how many Readers hold it,
|
||||||
|
never anything about a Reader.
|
||||||
|
_Avoid_: dangling, unused, dead series, stale
|
||||||
|
|
||||||
**Library**:
|
**Library**:
|
||||||
One of the two halves of the collection — manga or novel — selected by a Bookmark's
|
One of the two halves of the collection — manga or novel — selected by a Bookmark's
|
||||||
`kind`. The web UI and the userscripts each address exactly one Library at a time.
|
`kind`. The web UI and the userscripts each address exactly one Library at a time.
|
||||||
@@ -76,6 +83,41 @@ asked. Every Site has exactly one and no Lane can slow, block or borrow from ano
|
|||||||
a Reader never has one and never influences one.
|
a Reader never has one and never influences one.
|
||||||
_Avoid_: worker, queue, scheduler, batch, wave
|
_Avoid_: worker, queue, scheduler, batch, wave
|
||||||
|
|
||||||
|
**Lane Pass**:
|
||||||
|
One sweep of a Poll Lane over the Series due on its Site: what it found waiting, how many it
|
||||||
|
read, and whether it declined to work at all. A fact about the Lane rather than about any
|
||||||
|
Series — a pass that read nothing is still a pass, and one that declined carries the reason it
|
||||||
|
declined, since a Lane resting and a Lane stuck look identical from a count alone. Its record
|
||||||
|
outlives the process that made it: "the poller has done nothing for six hours" is only
|
||||||
|
answerable by something written down.
|
||||||
|
_Avoid_: run, cycle, tick, batch, poll history
|
||||||
|
|
||||||
|
**Forced Poll**:
|
||||||
|
A Poll the owner asks for by hand instead of waiting for the Series's turn. It jumps its
|
||||||
|
Lane's queue and ignores every waiting rule — the rest between Polls, a Sighting standing
|
||||||
|
in for a check, a Series only finished Readers hold — but never overrules a Site that is
|
||||||
|
refusing us, the Lane's spacing between fetches, or a Series with no page to fetch. Asked
|
||||||
|
for by marking the Series, never by commanding the poller, so it happens on the Lane's
|
||||||
|
next pass rather than at the moment of asking.
|
||||||
|
It also takes whatever Cover the Site publishes today: asking for one is asking to accept the
|
||||||
|
page as it now stands, so it is the only read after Acquisition that can replace a Cover.
|
||||||
|
_Avoid_: manual poll, refresh, retry, force refresh
|
||||||
|
|
||||||
|
**Paused Lane**:
|
||||||
|
A Poll Lane the owner has stopped for a bounded time. It makes no Polls until the pause
|
||||||
|
expires, so its Series stay due and unstamped exactly as they do when a Site cannot be
|
||||||
|
reached. Every pause carries an expiry — a Lane cannot be stopped indefinitely — and it
|
||||||
|
outlives a restart, being a fact about the Site rather than about the running process.
|
||||||
|
_Avoid_: disabled, off, stopped, suspended, kill switch (that is the deploy-time switch)
|
||||||
|
|
||||||
|
**Stall**:
|
||||||
|
A Poll Lane that owed Polls, made none, and has nothing to say for it. Distinct from the
|
||||||
|
two conditions it resembles: a Site that refuses is exercising the pace it is entitled to,
|
||||||
|
and a Lane the owner paused was told to stop — a Stall is neither asked for nor explained.
|
||||||
|
It is the one fault no Reader surface can show: every Bookmark still opens, Progress still
|
||||||
|
syncs, and Latest Chapter is quietly wrong for as long as it lasts.
|
||||||
|
_Avoid_: outage, downtime, failure, backlog, lag
|
||||||
|
|
||||||
**Sighting**:
|
**Sighting**:
|
||||||
What a Reader's browser happened to see of a Series's Latest Chapter while that Reader
|
What a Reader's browser happened to see of a Series's Latest Chapter while that Reader
|
||||||
was on the page. It reports the same fact as a Poll but carries none of its authority:
|
was on the page. It reports the same fact as a Poll but carries none of its authority:
|
||||||
@@ -88,11 +130,19 @@ _Avoid_: client report, user poll, observation, claim
|
|||||||
The single read of a Series page made the moment the Series first exists, giving it
|
The single read of a Series page made the moment the Series first exists, giving it
|
||||||
both its Latest Chapter and its Cover without waiting for the Lane's pace. Distinct
|
both its Latest Chapter and its Cover without waiting for the Lane's pace. Distinct
|
||||||
from a Poll in the two ways that matter: a Reader is present — it is triggered by
|
from a Poll in the two ways that matter: a Reader is present — it is triggered by
|
||||||
their first Bookmark of that Series — and it is the only read that establishes a
|
their first Bookmark of that Series — and it establishes a Cover rather than refreshing
|
||||||
Cover rather than refreshing facts. It happens once in a Series's life; every later
|
facts, which no Poll does unless the owner forces one. It happens once in a Series's
|
||||||
read of the same page is a Poll.
|
life; every later read of the same page is a Poll.
|
||||||
_Avoid_: initial poll, first fetch, prefetch, warm-up
|
_Avoid_: initial poll, first fetch, prefetch, warm-up
|
||||||
|
|
||||||
|
**Correction**:
|
||||||
|
A Latest Chapter the owner sets by hand, on a Series no Poll can read. It reports the
|
||||||
|
same fact as a Poll and carries even less authority than a Sighting: the next Poll
|
||||||
|
overwrites it, so does any Reader's Sighting, and it is never a floor or a pin. It
|
||||||
|
exists only because the Site page is unreadable — where a Poll can read the page, the
|
||||||
|
Poll is the answer and a Correction is not wanted.
|
||||||
|
_Avoid_: override, pin, manual value, fix
|
||||||
|
|
||||||
**New Chapter**:
|
**New Chapter**:
|
||||||
The state where Latest Chapter is ahead of Progress. The single condition the ember
|
The state where Latest Chapter is ahead of Progress. The single condition the ember
|
||||||
accent is permitted to signal.
|
accent is permitted to signal.
|
||||||
|
|||||||
+265
-270
@@ -1,274 +1,269 @@
|
|||||||
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
Scope: `backend/`.
|
||||||
|
|
||||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
Each entry names the code that holds the truth — read that for *what it does*.
|
||||||
Single binary, split into packages under `backend/internal/`: `store`
|
The prose here is only what code cannot tell you: rationale, rejected
|
||||||
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
alternatives, dated measurements, and invariants a plausible refactor would
|
||||||
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
silently break.
|
||||||
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
|
||||||
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
### Layout
|
||||||
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
|
||||||
`backend/main.go` is the composition root — the only place that wires
|
`backend/main.go` → `newRouter` is the composition root, the only place
|
||||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
packages are wired. Packages under `backend/internal/`: `store`, `latest`,
|
||||||
integration tests that exercise the full router; unit tests for a
|
`session`, `httpmw`, `api`, `userscript`, `web`, `token`, `pgtest`. Root-level
|
||||||
package live beside it under `internal/`.
|
`*_test.go` exercise the full router; unit tests live beside their package.
|
||||||
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
|
||||||
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
Not visible from any single file: stdlib `net/http` with no framework,
|
||||||
file per change, one transaction each, versions recorded in
|
Postgres over `jackc/pgx/v5`, `CGO_ENABLED=0` static binary into a distroless
|
||||||
`schema_migrations`. Files are **append-only** — editing an applied one
|
image, TLS terminated by the reverse proxy so the service listens plain `:8080`.
|
||||||
changes nothing on a database that already ran it. No column probing, no
|
|
||||||
data-fixup migrations: both were SQLite-era machinery and are gone.
|
### Schema — `internal/store/migrations/*.sql`, run by `store.migrate`
|
||||||
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
|
||||||
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
- Migration files are **append-only**. Editing an applied one changes nothing
|
||||||
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
on a database that already recorded its version in `schema_migrations`, so
|
||||||
must have that `TestMain`.
|
the fix silently applies to new deployments only.
|
||||||
- **Reader-owned store, four tables.** `readers` is keyed by Discord user ID
|
- No column probing, no data-fixup migrations. Both were SQLite-era machinery
|
||||||
and carries the SHA-256 of the Reader's userscript credential plus a
|
and were removed deliberately — don't reintroduce either.
|
||||||
`token_epoch` (issue #24). Credentials are derived, never stored: `token.Token(TOKEN_KEY, discord_id, epoch)` (HMAC, `internal/token`), and only its SHA-256 sits in `readers.token_sha256`, so install URLs can be rebuilt after any restart while a database leak yields nothing but hashes. The seed creates the **owner** row at startup; its epoch-0 hash is refreshed on every start **only while the row has never been rotated**, so a restart can never resurrect a rotated-away credential. Every other row is created by that Reader's own first login (`Store.EnsureReader`, idempotent on `discord_id`, and it never rewrites an existing row's hash). Rotation is `Store.RotateToken` (epoch bump + hash rewrite in one transaction), driven by the web UI.
|
|
||||||
`series` keyed `(site, series_id)`
|
### Tests need Docker — `internal/pgtest`
|
||||||
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
|
||||||
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
`pgtest.Main` from `TestMain` starts one `postgres:17-alpine` per test binary;
|
||||||
Latest Chapter, `latest_checked_at`, and the Sighting pair
|
`pgtest.URL` hands each test its own database. A package whose tests touch the
|
||||||
`latest_sighted_at`/`latest_raised_by` (issue #103) — and `bookmarks` holds only what
|
store must have that `TestMain` or it has no database at all.
|
||||||
differs between readers: progress, favourite, lifecycle bucket,
|
|
||||||
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
|
### Reader-owned store — `internal/store`, `internal/token`
|
||||||
surrogate id; the wire `key` is derived as `site:series_id` on read — and
|
- The Reader-owned tables are `readers`, `bookmarks`, `series`, and `sessions`; auxiliary `covers`, `poll_lanes`, and `poll_passes` are also defined in the migrations.
|
||||||
every store read/write is scoped to the reader it names. Auth resolves the
|
|
||||||
acting Reader from the presented credential (`httpmw.Auth`) and nothing
|
- **Credentials are derived, never stored.** `token.Token(TOKEN_KEY, discord_id, epoch)`
|
||||||
else — there is no unauthenticated-by-Reader route and no global token; the
|
is an HMAC; only its SHA-256 reaches `readers.token_sha256`. So install URLs
|
||||||
reader id travels in the request context. Sync **last-write-wins**; the wire format
|
can be rebuilt after any restart, and a database leak yields nothing usable.
|
||||||
stays flat (ADR-0004). `Store.Upsert` decomposes one flat body across two
|
- **The owner's epoch-0 hash is refreshed at startup only while the row has
|
||||||
tables and enforces the ownership rule: client `title`/`series_url`/`cover`
|
never been rotated.** Drop that condition and a restart resurrects a
|
||||||
are written only when the series row is new (ADR-0003).
|
rotated-away credential.
|
||||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
- `Store.EnsureReader` never rewrites an existing row's hash — a returning
|
||||||
- **Web UI:** same binary serve the browser UI on a second
|
Reader's login must not invalidate their installed scripts.
|
||||||
hostname — `GET /` (list, or login page when no session),
|
- **Every read and write is scoped to the acting Reader**, resolved from the
|
||||||
`GET /auth/discord` + `GET /auth/discord/callback` (Discord OAuth,
|
presented credential by `httpmw.Auth` and carried in the request context.
|
||||||
ADR-0002), `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
There is no unauthenticated-by-Reader route and no global token.
|
||||||
under `/ui/*`. Templates + assets `go:embed`-ed under
|
- **`series` holds what readers share, `bookmarks` only what differs.** A
|
||||||
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
bookmark key is `(reader_id, site, series_id)` with no surrogate id; the wire
|
||||||
`internal/` tree, not just `*.go`. Sessions are rows in the `sessions`
|
`key` is derived as `site:series_id` on read.
|
||||||
table: the cookie carries only an opaque id, looked up (and expiry-
|
- `Store.Upsert` splits one flat body across both tables and enforces the
|
||||||
checked) on every request, and deleting the row revokes the session.
|
ownership rule: client `title`/`series_url`/`cover` are written **only when
|
||||||
Guild membership *is* registration (issue #27): `discordCallback` gates on
|
the series row is new**, so one reader cannot retitle a shared series.
|
||||||
membership (and `DISCORD_REQUIRED_ROLE` when set) and then calls
|
- Sync is last-write-wins and the wire format stays flat — clients depend on
|
||||||
`Store.EnsureReader`, so a refusal creates nothing and a returning Reader
|
both; neither is an implementation detail to tidy up.
|
||||||
reuses their row. The owner is the only Reader with administrative reach:
|
|
||||||
`POST /readers/{id}/revoke` (404 for anyone else) drops that Reader's
|
### Web UI — `internal/web`
|
||||||
sessions, and the `readers` panel renders only on the owner's page.
|
|
||||||
A Reader with no bookmarks at all sees `listView.Fresh`, whose empty state
|
Routes, templates and assets are all in that package; `AdminPatterns()` and
|
||||||
offers both install links instead of describing a filter.
|
`adminRoutes()` enumerate the privileged ones.
|
||||||
UI mutations read-modify-write
|
|
||||||
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
- **`backend/Dockerfile` must copy the whole `internal/` tree**, not just
|
||||||
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
`*.go`: templates and static assets are `go:embed`-ed from
|
||||||
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
`internal/web/`.
|
||||||
(correct — served from `/`), but impeccable detector resolves
|
- **Guild membership *is* registration.** `discordCallback` gates on membership
|
||||||
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
(plus `DISCORD_REQUIRED_ROLE` when set) and only then calls
|
||||||
on leading `/` and silently skip file. Relative href don't help
|
`Store.EnsureReader`, so a refusal creates nothing.
|
||||||
either: template's directory isn't its served path. So
|
- Sessions are rows, not signatures: the cookie carries an opaque id and
|
||||||
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
expiry is checked on lookup, which is what makes deleting the row an instant
|
||||||
always pass `backend/internal/web/static` too. One finding there,
|
revocation.
|
||||||
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
- UI mutations go through `Store.Get` + `Store.Upsert` so the `updated_at` rule
|
||||||
- **Every action that moves series out of list is confirm-gated.**
|
below stays in exactly one place.
|
||||||
Archive, finish, remove each open own `.confirm-row` disclosure
|
- `listView.Fresh` exists because a Reader with no bookmarks at all needs
|
||||||
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
install links, not an empty-filter message.
|
||||||
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
- **Design-tool caveat:** `detect.mjs backend/internal/web/templates` reports a
|
||||||
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
**false clean**. Templates link `/static/style.css` root-absolutely (correct —
|
||||||
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
it is served from `/`), but the detector resolves hrefs with
|
||||||
error use `--mute`.
|
`path.resolve(fileDir, href)`, which drops the directory on a leading `/` and
|
||||||
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100),
|
skips the file silently; a relative href doesn't help either, since a
|
||||||
each re-checking that Site's bookmarked series' newest published chapter from
|
template's directory isn't its served path. Always pass
|
||||||
backend's own network access, so `latest_chapter` stays fresh when the user
|
`backend/internal/web/static` too. The one finding there, `overused-font` on
|
||||||
isn't browsing. Second, parallel signal — the userscript keeps its own
|
"Instrument Serif", is a deliberate identity choice, not debt.
|
||||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` schedule, and its
|
|
||||||
`reportLatestChapter` PUTs every read, unchanged numbers included, because an
|
### Confirm gating — `internal/web/static/filter.js`, `toggleConfirmRow(key, kind)`
|
||||||
unchanged read is exactly the Sighting worth deferring a Poll on (#103).
|
|
||||||
Two independent clocks: per-series rest (`series.latest_checked_at`,
|
Every action that pulls a series out of the list (`archive|finish|remove`) opens
|
||||||
enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and
|
its own `.confirm-row`; restore fires instantly because it is the reversal.
|
||||||
per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
|
Remove wears the ember wash, the two reversible ones wear `.calm` grey.
|
||||||
in the Site registry (`internal/latest/sites.go`), not config: the five env
|
**`--ember` is reserved for the new-chapter signal** — the busy bar and inline
|
||||||
knobs that used to size a shared pace are gone.
|
errors must use `--mute`, or the one colour that means "something to read"
|
||||||
The poller walks **Series, not Bookmarks** — a series referenced by several
|
stops meaning it.
|
||||||
bookmarks is fetched once per cycle, and the due queue orders
|
|
||||||
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
### Latest-chapter poller — `internal/latest`, Site registry in `sites.go`
|
||||||
*before* fetch so broken series wait out the rest instead of retrying
|
|
||||||
every tick; found chapter written straight to the series row via
|
One goroutine per Site (a Poll Lane) re-checks that Site's bookmarked series
|
||||||
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
from the backend's own network position, so `latest_chapter` stays fresh while
|
||||||
order — is never touched.
|
nobody is browsing. The userscript's `reportLatestChapter` is a second,
|
||||||
**Sightings** (issue #103, ADR-0011) let a Reader's own page read defer a
|
parallel signal — it PUTs every read, unchanged numbers included, because an
|
||||||
Poll: `Store.RecordSighting` — called by the PUT handler *before* the Upsert,
|
unchanged read is exactly the Sighting worth deferring a Poll on.
|
||||||
because the raise test needs the row as it stands — stamps
|
|
||||||
`series.latest_sighted_at` and, when the report raises the stored number,
|
- **Pace lives in the Site registry, not config.** Two clocks: per-series rest
|
||||||
names its Reader in `series.latest_raised_by`. The due query's HAVING clause
|
(`series.latest_checked_at`, enforced in `Store.DueForLatestCheck`'s WHERE)
|
||||||
is where deferral lives: a Series is skipped only while it has exactly one
|
and per-Lane gap (`effectiveGap`). The five env knobs that used to size one
|
||||||
Bookmark, was sighted within one Rest, and is under the ceiling
|
shared pace are gone; don't add them back.
|
||||||
(`sightingCeilingRests`, six of that Site's rests) since its last Poll. So a
|
- **The poller walks Series, not Bookmarks** — a series several readers hold is
|
||||||
shared Series is never deferred, and no Series goes six hours unpolled
|
fetched once per cycle, and the due queue orders `reader_count DESC,
|
||||||
whatever arrives. `checkOne` judges the named Reader off the comparison it
|
latest_checked_at ASC` so the widely-read ones win contention.
|
||||||
already makes: a lower number is a contradiction (logged with the Reader and
|
- **The series row is stamped *before* the fetch**, so a permanently broken
|
||||||
both numbers), the same number an agreement, a higher number the Site
|
series waits out its rest instead of being retried every tick.
|
||||||
publishing and neither — that last one clears the attribution instead, since
|
- `Store.SetLatestChapter` is a single-column UPDATE, deliberately not a
|
||||||
the value the Poll then stores is its own and a later retraction is not the
|
read-modify-write of the bookmark: it therefore cannot revert read progress
|
||||||
Reader's fault. Three contradictions
|
or move `updated_at`. The old stale-re-read race died with the Get+Upsert
|
||||||
(`store.SightingDisagreementLimit`) stop that Reader deferring — their
|
flow — don't restore one here.
|
||||||
reports still write the Latest Chapter — and twenty consecutive agreements
|
|
||||||
(`store.SightingAgreementsToClear`) forgive them, as does the owner's
|
**Sightings** (`Store.RecordSighting`, the due query's HAVING clause,
|
||||||
clear-marks control. Deferral is recomputed from live facts every round, so
|
`latest.checkOne`) let a Reader's own page read defer a Poll.
|
||||||
nothing needs invalidating when a Series gains a second Bookmark; the one
|
|
||||||
input read earlier is the Reader's marks, checked when the Sighting is
|
- Recorded by the PUT handler **before** the Upsert, because the raise test
|
||||||
recorded, so crossing the threshold or being cleared takes effect from that
|
needs the row as it stands.
|
||||||
Reader's next Sighting and the standing already bought lasts out its rest.
|
- A Series is deferred only while it has exactly one Bookmark, was sighted
|
||||||
Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
|
within one Rest, and is under `sightingCeilingRests` since its last Poll — so
|
||||||
stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
|
a shared Series is never deferred and nothing goes six hours unpolled
|
||||||
`errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
|
whatever arrives.
|
||||||
that makes the other browser Lanes skip their passes for the same 15m, so a
|
- A *higher* report clears the attribution rather than crediting it: the value
|
||||||
restarting Chrome doesn't stamp one Series per Lane per pass — after the
|
the Poll then stores is its own, so a later retraction isn't the Reader's
|
||||||
window the flag decays and they probe again. Browser Lanes wake Chrome only
|
fault.
|
||||||
when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
|
- `store.SightingDisagreementLimit` contradictions stop a Reader deferring —
|
||||||
and cover work (both healing a stored source URL and filling a blank from
|
their reports still write the Latest Chapter — and
|
||||||
the series page) runs in the background so a slow CDN can't consume a
|
`store.SightingAgreementsToClear` agreements forgive them, as does the
|
||||||
Lane's gap.
|
owner's clear-marks control.
|
||||||
A refusal is only ever the challenge *page*: `isInterstitial` matches the
|
- Deferral is recomputed from live facts each round, so nothing needs
|
||||||
orchestration path `/cdn-cgi/challenge-platform/h/`, never the bare prefix.
|
invalidating when a Series gains a second Bookmark. The one input read
|
||||||
Cloudflare injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into
|
earlier is the Reader's marks, so crossing or clearing a threshold takes
|
||||||
ordinary 200 pages once a zone turns JS detections on, which demonic did on
|
effect from their next Sighting and the standing already bought lasts out its
|
||||||
2026-08-16 — the prefix match then read every real demonic page as a refusal
|
rest.
|
||||||
and parked that Lane in 15m backoff while plain TLS was returning the full
|
|
||||||
series page.
|
**Refusals and browser loss are Lane-local.** Two `errChallengeHeld` in a pass
|
||||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
stop that Site for `RefuseBackoff` while other Lanes continue. An
|
||||||
against fingerprint-based blocking; any failure log and skip. kagane, comix
|
`errBrowserInterrupted` (remote Chrome restarted) sets a shared Poller flag so
|
||||||
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
|
the *other* browser Lanes skip their passes for the same window — otherwise a
|
||||||
can't clear, so they are fetched over CDP via `BROWSER_WS_URL`; kagane and
|
restarting Chrome stamps one Series per Lane per pass, burning rests on
|
||||||
comix are simply not polled when that's unset, while novelfull falls back to
|
failures. The flag decays and they probe again.
|
||||||
a plain-TLS attempt — its challenge is a live time-varying fact, and its
|
|
||||||
cover bytes never need the browser. comix's browser read is an in-tab
|
- **`isInterstitial` matches the orchestration path
|
||||||
`fetch()` of the Series URL, not a DOM render: it is an SPA, so rendering
|
`/cdn-cgi/challenge-platform/h/`, never the bare prefix.** Cloudflare injects
|
||||||
costs ~65 requests for the same server-rendered HTML one fetch returns
|
`/cdn-cgi/challenge-platform/scripts/jsd/main.js` into ordinary 200 pages
|
||||||
(measured 2026-08-12, issue #98). See
|
once a zone turns JS detections on, which demonic did on 2026-08-16: the
|
||||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
prefix match read every real demonic page as a refusal and parked the Lane in
|
||||||
The poller's series write is a single-column UPDATE
|
backoff while plain TLS was returning full series pages.
|
||||||
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
- Fetches use `bogdanfinn/tls-client` with a Chrome profile as defence in depth
|
||||||
it cannot revert read progress or move `updated_at`, so the old
|
against fingerprint blocking; any failure logs and skips.
|
||||||
stale-re-read race is gone with the Get+Upsert flow.
|
- kagane, comix and novelfull sit behind Cloudflare JS challenges the TLS
|
||||||
- **Covers are acquired at creation, then served from our own origin
|
client can't clear, so they go over CDP (`BROWSER_WS_URL`). kagane and comix
|
||||||
(ADR-0007):** the first Bookmark of a Series fires `Store.OnSeriesCreated`,
|
are simply not polled when it's unset — a plain fetch would only retrieve a
|
||||||
which `latest.Acquirer` turns into one series-page fetch yielding both the
|
challenge page — while novelfull still attempts plain TLS, because its
|
||||||
Latest Chapter and the cover URL; the bytes then go through
|
challenge is a live time-varying fact and its cover bytes never need a browser.
|
||||||
`latest.CoverBytesFetcher` into `Store.SetSeriesCover`. It runs in a
|
- **comix's browser read is an in-tab `fetch()` of the Series URL, not a DOM
|
||||||
goroutine — the Reader's PUT must neither block on a Site nor fail with one
|
render.** It is an SPA: rendering cost ~65 requests for the same
|
||||||
— and every failure is logged and dropped, leaving the Bookmark intact. The
|
server-rendered HTML one fetch returns (measured 2026-08-12).
|
||||||
wire's `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
|
- Browser Lanes wake Chrome only when 5+ Series are due or one has waited 15m,
|
||||||
bytes exist and `""` before, never an address that 404s. `GET /covers/{addr}`
|
and cover work runs in the background so a slow CDN can't eat a Lane's gap.
|
||||||
is public and uncredentialed: the userscript renders it on a Site's origin,
|
|
||||||
where no cookie or token of ours travels. A client-sent `cover` is decoded
|
### Covers — `Store.OnSeriesCreated`, `latest.Acquirer`, `latest.CoverBytesFetcher`, `Store.SetSeriesCover`
|
||||||
and discarded, permanently (ADR-0004 compatibility).
|
|
||||||
Browser-backed Sites join the same pipeline (issue #62, extended to comix by
|
Acquired once when the first Bookmark of a Series is created, then served from
|
||||||
#98): kagane and comix pages *and* cover bytes go through the browser sidecar
|
our own origin by the public `GET /covers/{addr}`.
|
||||||
(nothing falls back to a plain fetch, which would only retrieve a challenge
|
|
||||||
page), while novelfull needs the browser only for its HTML — the cover URL
|
- Acquisition runs in a goroutine: the Reader's PUT must neither block on a
|
||||||
comes out of the browser-fetched page and the bytes go over plain TLS. With
|
Site nor fail with one. Every failure is logged and dropped, leaving the
|
||||||
no browser configured, kagane and comix Covers are simply absent; novelfull
|
Bookmark intact.
|
||||||
still gets one — at creation and on the poll — when its page body happens to
|
- The wire `cover` is the absolute `PUBLIC_BASE_URL + /covers/{sha256}` once
|
||||||
answer a plain request (the challenge is a live time-varying fact). comix
|
bytes exist and `""` before — **never an address that 404s**. Absolute
|
||||||
cover bytes must arrive by direct navigation, not an in-page fetch: its
|
because the userscript renders it on a Site's origin.
|
||||||
Series page sets `cross-origin-embedder-policy: require-corp`, which fails a
|
- `GET /covers/{addr}` is public and uncredentialed by design: no cookie or
|
||||||
page-context fetch of `static.comix.to`. The old kagane-only
|
token of ours may travel to a Site's origin.
|
||||||
serving path (`/img/kagane/{id}`, template rewrite, `CoverFetcher`) is gone
|
- A client-sent `cover` is decoded and discarded, permanently — wire
|
||||||
(issue #63): the one public route serves every Site.
|
compatibility, not an oversight.
|
||||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
- **One route serves all six Sites.** No proxy, no per-Site rewrite, no second
|
||||||
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
place that decides a renderable address: the wire `cover` is it. Templates
|
||||||
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
render `.Cover` and nothing else. The old kagane-only serving path
|
||||||
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
(`/img/kagane/{id}` plus a template rewrite) is gone; don't reintroduce a
|
||||||
checking archived series and skip finished ones. `finished` settable
|
per-Site route because one Site's CDN misbehaves.
|
||||||
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
- The only Site names left in cover code are in `browserOnlyCoverURL`
|
||||||
**Empty incoming status means "keep stored one"** — resolved on the
|
(`internal/latest`): kagane answers a plain fetch with a challenge *and*
|
||||||
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
|
||||||
`excluded.*` is post-evaluation row and default applied there would
|
|
||||||
wipe bucket on every PUT from client that predates column. See
|
|
||||||
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
|
||||||
- **Config via env:** `TOKEN_KEY` (derives every Reader's userscript credential;
|
|
||||||
required), `OWNER_DISCORD_ID` (seeds the owner Reader — the administrator and
|
|
||||||
the owner of every pre-registration bookmark; required),
|
|
||||||
`ALLOWED_ORIGINS` (comma list),
|
|
||||||
`DATABASE_URL` (Postgres connection URL, required — no default),
|
|
||||||
`COVER_DIR` (required filesystem volume for content-addressed Cover bytes),
|
|
||||||
`PUBLIC_BASE_URL` (required origin this deployment answers on, trailing
|
|
||||||
slash trimmed; every Cover URL on the wire is built from it, absolute
|
|
||||||
because the userscript renders on a Site's origin — ADR-0007),
|
|
||||||
`PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
|
|
||||||
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
|
||||||
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
|
||||||
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
|
|
||||||
`LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill
|
|
||||||
switch, default on). Pace is per Site in the registry (issue #100): every
|
|
||||||
Site rests an hour and gaps ten seconds, a Site with more eligible Series
|
|
||||||
than 360 tightens its own gap toward the 1s floor, and browser Lanes wake
|
|
||||||
Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/
|
|
||||||
`_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are
|
|
||||||
gone. The 1h rest for browser Sites is safe on documented grounds: a
|
|
||||||
challenged page costs seconds of a serialized single-tab browser, free-plan
|
|
||||||
zones have no bot score and no published per-IP rate input, and
|
|
||||||
`cf_clearance` expires in 30 minutes so every cadence at or above 1h
|
|
||||||
re-solves anyway —
|
|
||||||
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
|
|
||||||
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
|
||||||
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
|
||||||
defaults `/userscript/manga-bookmark.user.js` and
|
|
||||||
`/userscript/novel-bookmark.user.js`, both supplied by bindmount; the
|
|
||||||
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
|
||||||
Reader's credential at serve time).
|
|
||||||
`BROWSER_WS_URL` (CDP endpoint of the browser, which runs on a **separate
|
|
||||||
machine** and is reached over the tailnet — ADR-0006, `chrome/docker-compose.yml`.
|
|
||||||
Used by the poller for kagane, comix and novelfull page fetches and by the
|
|
||||||
cover pipeline for kagane's and comix's image bytes (the browser is the only
|
|
||||||
route that clears the challenge those two serve their covers behind); unset —
|
|
||||||
the default — disables browser polling and leaves kagane and comix Covers
|
|
||||||
blank until stored bytes
|
|
||||||
exist. Must be a tailnet IP, never a hostname: Chrome's DevTools handler 500s
|
|
||||||
`/json/version` for any Host that isn't an IP or `localhost`).
|
|
||||||
- **No per-Site cover path (issue #63):** every Cover — all six Sites — is
|
|
||||||
served by the one public `GET /covers/{addr}` route from content-addressed
|
|
||||||
bytes. There is no proxy, no per-Site rewrite, no second place that decides
|
|
||||||
a Cover's renderable address: the wire `cover` is it. The only place a Site
|
|
||||||
name still appears in cover code is the extraction module (`latest`), where
|
|
||||||
kagane's and comix's image URLs are claimed by `browserOnlyCoverURL` — kagane
|
|
||||||
answers a plain fetch with a challenge and
|
|
||||||
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
|
`cross-origin-resource-policy: same-origin`, and `static.comix.to` answers
|
||||||
one with the same Cloudflare challenge its pages serve;
|
with the same challenge its pages serve. Every other Site's CDN answers plain
|
||||||
every other Site's CDN answers plain TLS. Templates render `.Cover` — the
|
TLS.
|
||||||
wire value — never anything else.
|
- **comix cover bytes must arrive by direct navigation, not an in-page fetch:**
|
||||||
- **Web UI also owns:** session-gated `GET /install/{manga,novel}-bookmark.user.js`
|
its Series page sets `cross-origin-embedder-policy: require-corp`, which
|
||||||
(renders the bindmounted script with the acting Reader's derived credential
|
fails a page-context fetch of `static.comix.to`.
|
||||||
substituted in — the credential never appears in page markup, the address
|
- With no browser configured, kagane and comix Covers are simply absent;
|
||||||
bar, or a redirect; `?download=1` adds `Content-Disposition: attachment` for
|
novelfull still gets one whenever its page answers a plain request.
|
||||||
mobile Violentmonkey, which ignores a `.user.js` navigation) and
|
|
||||||
`POST /rotate-token` (atomic epoch bump + hash
|
### `updated_at` drives list order — `Store.Upsert`
|
||||||
rewrite; invalidates every installed copy, so the panel warns to reinstall
|
|
||||||
on all devices).
|
The server applies its own timestamp only when the row is new or
|
||||||
- **Owner-only admin page (`internal/web/admin.go`, issue #102):** `GET /admin`
|
`last_chapter_num` changes, else it keeps the stored value. **Favouriting a
|
||||||
carries the Reader roster (sessions, Sighting counters, `POST
|
series, or a newly published chapter arriving, must not reorder the list** —
|
||||||
/readers/{id}/revoke` and `POST /readers/{id}/clear-marks`) and Poll Lane
|
only real reading progress moves a row. Consequently `PUT` returns the row **as
|
||||||
status (`GET /ui/admin/lanes`, self-refreshing every 30s). Every route that
|
stored** and clients must adopt that response rather than their own payload.
|
||||||
reaches past the acting Reader is listed in `adminRoutes()` and wrapped in
|
|
||||||
`requireOwner` at registration — add a route there, not a check inside a
|
### Lifecycle buckets — `status` on each bookmark
|
||||||
handler; `web.AdminPatterns()` is what the gate test walks. A non-owner gets
|
|
||||||
404, never 403. Lane figures come from the running poller through the
|
`reading` | `archived` | `finished`, orthogonal to `favorite`. Archived and
|
||||||
`web.LaneReporter` seam (`latest.Poller.LaneStatus`), never from a table: a
|
finished appear only in their own tab, never in All, Updated, Favourites or the
|
||||||
nil reporter or a Lane that has not finished a pass renders "no data yet"
|
recent strip. The poller keeps checking archived series and skips finished ones.
|
||||||
rather than zeroes. `main.newRouter` takes the reporter as an interface and
|
|
||||||
converts a nil `*Poller` to a nil interface — a typed nil would make the page
|
- `finished` is settable only from the web UI; `PUT /bookmarks/{key}` rejects
|
||||||
claim a poller exists.
|
it with 400.
|
||||||
The one owner comparison left outside `requireOwner` is in `index`
|
- **An empty incoming status means "keep the stored one"**, and it is resolved
|
||||||
|
on the `VALUES` side of `Store.Upsert`, not in the conflict clause:
|
||||||
|
`excluded.*` is the post-evaluation row, so a default applied there would
|
||||||
|
wipe the bucket on every PUT from a client predating the column.
|
||||||
|
|
||||||
|
### Config — `Config` / `loadConfig` / `loadLatestPoll` in `backend/main.go`
|
||||||
|
|
||||||
|
That function is the complete list of env vars, their defaults, and which are
|
||||||
|
required. What it can't tell you:
|
||||||
|
|
||||||
|
- `PUBLIC_BASE_URL` must be an absolute origin because every Cover URL on the
|
||||||
|
wire is built from it and the userscript renders on a Site's origin.
|
||||||
|
- `BROWSER_WS_URL` **must be a tailnet IP, never a hostname** — Chrome's
|
||||||
|
DevTools handler 500s `/json/version` for any Host that isn't an IP or
|
||||||
|
`localhost`. Unset (the default) disables browser polling.
|
||||||
|
- `USERSCRIPT_PATH` / `NOVEL_USERSCRIPT_PATH` are bindmounted files; the
|
||||||
|
`__API_TOKEN__` placeholder inside them is substituted with the requesting
|
||||||
|
Reader's credential at serve time.
|
||||||
|
- Pace is per Site in the registry, not env. The
|
||||||
|
`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER` knobs are
|
||||||
|
gone on purpose.
|
||||||
|
- The 1h rest for browser Sites is safe on documented grounds: a challenged
|
||||||
|
page costs seconds of a serialized single-tab browser, free-plan zones carry
|
||||||
|
no bot score and no published per-IP rate input, and `cf_clearance` expires
|
||||||
|
in 30 minutes, so every cadence at or above 1h re-solves anyway.
|
||||||
|
|
||||||
|
### Userscript install & rotation — `internal/userscript`, `internal/token`
|
||||||
|
|
||||||
|
Session-gated `GET /install/{manga,novel}-bookmark.user.js` renders the
|
||||||
|
bindmounted script with the acting Reader's derived credential substituted in,
|
||||||
|
so the credential never appears in page markup, the address bar, or a redirect.
|
||||||
|
`?download=1` adds `Content-Disposition: attachment` for mobile Violentmonkey,
|
||||||
|
which ignores a `.user.js` navigation. `POST /rotate-token` is an atomic epoch
|
||||||
|
bump plus hash rewrite and invalidates every installed copy — the panel must
|
||||||
|
keep warning to reinstall on all devices.
|
||||||
|
|
||||||
|
### Owner-only admin — `internal/web/admin.go`
|
||||||
|
|
||||||
|
- **Every route reaching past the acting Reader is listed in `adminRoutes()`
|
||||||
|
and wrapped in `requireOwner` at registration** — add it there, not as a
|
||||||
|
check inside a handler; `web.AdminPatterns()` is what the gate test walks. A
|
||||||
|
non-owner gets 404, never 403.
|
||||||
|
- The one owner comparison left outside the gate is in `index`
|
||||||
(`view.Owner = readerID == h.store.OwnerID()`): it gates a link, not an
|
(`view.Owner = readerID == h.store.OwnerID()`): it gates a link, not an
|
||||||
endpoint, so it is a rendering decision a registration-time wrapper cannot
|
endpoint, so it is a rendering decision a registration-time wrapper cannot
|
||||||
express — do not "unify" it into the gate.
|
express. Do not "unify" it into the gate.
|
||||||
A Lane pass that returns before computing its figures (refusal backoff,
|
- The Lanes page reads the pass log, never a running poller: `lanesView()` in
|
||||||
sidecar down) carries the previous pass's due count and gap forward rather
|
`admin_lanes.go` projects `store.LatestLanePasses()` and
|
||||||
than recording zeroes; a Lane that has never reached a pace renders no gap at
|
`store.LanePassOutcomes()` (ADR-0012), so a restart answers the instant the
|
||||||
all. `Checked` next to `Due` is what separates a stopped Lane from a quiet
|
database is up. Browser configuration is a config fact and reachability is
|
||||||
one, so neither figure may be dropped from the row.
|
derived from recent browser-Site passes inside `latest.RefuseBackoff` — no
|
||||||
Due-without-Checked is *not* by itself a stall: a browser Lane under both
|
reporter interface exists to fake.
|
||||||
wake thresholds sets `LaneState.Asleep` at the on-demand gate and renders
|
- A pass that returns before computing figures (refusal backoff, sidecar down)
|
||||||
"browser asleep" instead of "not checking", and never counts toward
|
carries the previous pass's numbers forward rather than recording zeroes.
|
||||||
`Attention`. That is the commonest healthy state for kagane, comix and
|
- **`Checked` next to `Due` is what separates a stopped Lane from a quiet one**,
|
||||||
novelfull — one due Series, nothing checked — so spending the stall mark on
|
so neither may be dropped from the row.
|
||||||
it would train the owner to ignore the mark that matters.
|
- Due-without-Checked is **not** by itself a stall: a browser Lane under both
|
||||||
|
wake thresholds records its pass with the `SkipAsleep` skip and renders
|
||||||
|
"browser asleep", and that never counts toward `Attention`. It is the
|
||||||
|
commonest healthy state for kagane, comix and novelfull, so spending the
|
||||||
|
stall mark on it would train the owner to ignore the mark that matters.
|
||||||
|
|||||||
+5
-5
@@ -48,7 +48,7 @@ func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
|||||||
|
|
||||||
func newTestServer(t *testing.T) http.Handler {
|
func newTestServer(t *testing.T) http.Handler {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return newRouter(newTestStore(t), testConfig(), nil)
|
return newRouter(newTestStore(t), testConfig())
|
||||||
}
|
}
|
||||||
|
|
||||||
func newTestStore(t *testing.T) *store.Store {
|
func newTestStore(t *testing.T) *store.Store {
|
||||||
@@ -599,7 +599,7 @@ func TestLoadConfigDiscord(t *testing.T) {
|
|||||||
// cooldown and the poller would re-fetch that series on every single tick.
|
// cooldown and the poller would re-fetch that series on every single tick.
|
||||||
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
|
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
|
||||||
s := newTestStore(t)
|
s := newTestStore(t)
|
||||||
srv := newRouter(s, testConfig(), nil)
|
srv := newRouter(s, testConfig())
|
||||||
|
|
||||||
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
|
||||||
|
|
||||||
@@ -627,7 +627,7 @@ func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
|
|||||||
// series stops being due the moment the PUT lands.
|
// series stops being due the moment the PUT lands.
|
||||||
func TestPutRecordsASighting(t *testing.T) {
|
func TestPutRecordsASighting(t *testing.T) {
|
||||||
s := newTestStore(t)
|
s := newTestStore(t)
|
||||||
srv := newRouter(s, testConfig(), nil)
|
srv := newRouter(s, testConfig())
|
||||||
|
|
||||||
now := time.Now().UnixMilli()
|
now := time.Now().UnixMilli()
|
||||||
hour := time.Hour.Milliseconds()
|
hour := time.Hour.Milliseconds()
|
||||||
@@ -677,7 +677,7 @@ func TestUserscriptServedWithWebUIDisabled(t *testing.T) {
|
|||||||
|
|
||||||
rr := httptest.NewRecorder()
|
rr := httptest.NewRecorder()
|
||||||
req := httptest.NewRequest(http.MethodGet, "/u/"+ownerCredential()+"/manga-bookmark.user.js", nil)
|
req := httptest.NewRequest(http.MethodGet, "/u/"+ownerCredential()+"/manga-bookmark.user.js", nil)
|
||||||
newRouter(s, cfg, nil).ServeHTTP(rr, req)
|
newRouter(s, cfg).ServeHTTP(rr, req)
|
||||||
if rr.Code != http.StatusOK {
|
if rr.Code != http.StatusOK {
|
||||||
t.Fatalf("status = %d, want 200", rr.Code)
|
t.Fatalf("status = %d, want 200", rr.Code)
|
||||||
}
|
}
|
||||||
@@ -699,7 +699,7 @@ func TestNovelUserscriptServed(t *testing.T) {
|
|||||||
|
|
||||||
cfg := testConfig()
|
cfg := testConfig()
|
||||||
cfg.NovelUserscriptPath = novelPath
|
cfg.NovelUserscriptPath = novelPath
|
||||||
srv := newRouter(s, cfg, nil)
|
srv := newRouter(s, cfg)
|
||||||
|
|
||||||
rr := httptest.NewRecorder()
|
rr := httptest.NewRecorder()
|
||||||
srv.ServeHTTP(rr, httptest.NewRequest(http.MethodGet,
|
srv.ServeHTTP(rr, httptest.NewRequest(http.MethodGet,
|
||||||
|
|||||||
@@ -98,7 +98,7 @@ func TestPublicCoverNeverEchoesNonImage(t *testing.T) {
|
|||||||
if _, err := db.Exec(`UPDATE covers SET content_type = 'text/html' WHERE address = $1`, address); err != nil {
|
if _, err := db.Exec(`UPDATE covers SET content_type = 'text/html' WHERE address = $1`, address); err != nil {
|
||||||
t.Fatalf("poison row: %v", err)
|
t.Fatalf("poison row: %v", err)
|
||||||
}
|
}
|
||||||
rr := getCover(t, newRouter(st, testConfig(), nil), "/covers/"+address, nil)
|
rr := getCover(t, newRouter(st, testConfig()), "/covers/"+address, nil)
|
||||||
if rr.Code == http.StatusOK {
|
if rr.Code == http.StatusOK {
|
||||||
t.Fatalf("status = 200, want a refusal for a non-image row (body %q)", rr.Body.String())
|
t.Fatalf("status = 200, want a refusal for a non-image row (body %q)", rr.Body.String())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -135,6 +135,33 @@ func TestAcquireFillsChapterAndCoverFromOneFetch(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A pause governs the Lane only: a Reader's first bookmark of a Series on a
|
||||||
|
// paused Site still reads the page, because acquisition is the creation-time
|
||||||
|
// fetch, not the poll queue (issue #147).
|
||||||
|
func TestAcquireIgnoresLanePause(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
if err := s.PauseLane("asura", time.Now().Add(6*time.Hour).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
page := &fakeFetcher{body: asuraSeriesAndCoverFixture, status: 200}
|
||||||
|
covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}
|
||||||
|
acq := newAcquirer(s, page, covers)
|
||||||
|
|
||||||
|
bookmarkNewSeries(t, s, acquireSeriesURL)
|
||||||
|
acq.Wait()
|
||||||
|
|
||||||
|
if got := page.callCount(); got != 1 {
|
||||||
|
t.Fatalf("series page fetches on a paused Site = %d, want 1", got)
|
||||||
|
}
|
||||||
|
if got := covers.callCount(); got != 1 {
|
||||||
|
t.Fatalf("cover fetches = %d, want 1", got)
|
||||||
|
}
|
||||||
|
got := readBookmark(t, s, acquireKey)
|
||||||
|
if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 {
|
||||||
|
t.Fatalf("LatestChapterNum = %v, want 181", got.LatestChapterNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// A Series that already exists is not re-acquired: no fetch, and the Cover it
|
// A Series that already exists is not re-acquired: no fetch, and the Cover it
|
||||||
// already has is left alone.
|
// already has is left alone.
|
||||||
func TestAcquireSkipsAnExistingSeries(t *testing.T) {
|
func TestAcquireSkipsAnExistingSeries(t *testing.T) {
|
||||||
|
|||||||
@@ -47,6 +47,15 @@ func fetchCoverBytes(ctx context.Context, cover string, browser BrowserCoverFetc
|
|||||||
// inject it to exercise hostile DNS results without touching the live network.
|
// inject it to exercise hostile DNS results without touching the live network.
|
||||||
type CoverResolver func(context.Context, string) ([]netip.Addr, error)
|
type CoverResolver func(context.Context, string) ([]netip.Addr, error)
|
||||||
|
|
||||||
|
// maxCoverBytes caps one cover, separately from the series-page maxBodyBytes:
|
||||||
|
// a cover is a bounded binary asset, not a text page, and 4 MiB rejected 12%
|
||||||
|
// of asurascans covers measured 2026-08-17 (p90 4.52 MB, max 8.57 MB — two of
|
||||||
|
// the three over-cap files were JPEGs, not the animated GIF of issue #71).
|
||||||
|
// 10 MiB is ~18% headroom over that worst case and matches the GitHub and
|
||||||
|
// Discord image limits; see docs/research/gif-maximum-byte-size.md. GIF itself
|
||||||
|
// has no maximum size, so this number is policy, not format.
|
||||||
|
const maxCoverBytes = 10 << 20
|
||||||
|
|
||||||
// TLSCoverFetcher retrieves image bytes with the standard HTTPS client. Unlike
|
// TLSCoverFetcher retrieves image bytes with the standard HTTPS client. Unlike
|
||||||
// TLSFetcher, it does not need a browser fingerprint: cover hosts are public
|
// TLSFetcher, it does not need a browser fingerprint: cover hosts are public
|
||||||
// CDNs and the response is accepted only after the destination gate passes.
|
// CDNs and the response is accepted only after the destination gate passes.
|
||||||
@@ -152,15 +161,15 @@ func (f *TLSCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte,
|
|||||||
if !ok {
|
if !ok {
|
||||||
return nil, "", fmt.Errorf("fetch cover: unsupported content type %q", raw)
|
return nil, "", fmt.Errorf("fetch cover: unsupported content type %q", raw)
|
||||||
}
|
}
|
||||||
if resp.ContentLength > maxBodyBytes {
|
if resp.ContentLength > maxCoverBytes {
|
||||||
return nil, "", fmt.Errorf("fetch cover: response exceeds %d bytes", maxBodyBytes)
|
return nil, "", fmt.Errorf("fetch cover: response exceeds %d bytes", maxCoverBytes)
|
||||||
}
|
}
|
||||||
body, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes+1))
|
body, err := io.ReadAll(io.LimitReader(resp.Body, maxCoverBytes+1))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, "", fmt.Errorf("read cover: %w", err)
|
return nil, "", fmt.Errorf("read cover: %w", err)
|
||||||
}
|
}
|
||||||
if len(body) > maxBodyBytes {
|
if len(body) > maxCoverBytes {
|
||||||
return nil, "", fmt.Errorf("fetch cover: response exceeds %d bytes", maxBodyBytes)
|
return nil, "", fmt.Errorf("fetch cover: response exceeds %d bytes", maxCoverBytes)
|
||||||
}
|
}
|
||||||
return body, contentType, nil
|
return body, contentType, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -171,7 +171,7 @@ func TestCoverFetcherRejectsOversizedBody(t *testing.T) {
|
|||||||
var calls int
|
var calls int
|
||||||
client := &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
client := &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
||||||
calls++
|
calls++
|
||||||
response := coverResponse(http.StatusOK, "image/webp", "", bytes.Repeat([]byte("x"), maxBodyBytes+1))
|
response := coverResponse(http.StatusOK, "image/webp", "", bytes.Repeat([]byte("x"), maxCoverBytes+1))
|
||||||
response.ContentLength = -1
|
response.ContentLength = -1
|
||||||
return response, nil
|
return response, nil
|
||||||
})}
|
})}
|
||||||
@@ -187,6 +187,29 @@ func TestCoverFetcherRejectsOversizedBody(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Covers between the series-page cap and the cover cap must be accepted: the
|
||||||
|
// 4 MiB page cap rejected 12% of asurascans covers (issue #71).
|
||||||
|
func TestCoverFetcherAcceptsCoverOverPageCap(t *testing.T) {
|
||||||
|
body := bytes.Repeat([]byte("x"), maxBodyBytes+1)
|
||||||
|
client := &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
||||||
|
return coverResponse(http.StatusOK, "image/gif", "", body), nil
|
||||||
|
})}
|
||||||
|
fetcher := newCoverFetcher(client, func(context.Context, string) ([]netip.Addr, error) {
|
||||||
|
return []netip.Addr{netip.MustParseAddr("198.51.100.10")}, nil
|
||||||
|
})
|
||||||
|
|
||||||
|
got, contentType, err := fetcher.Fetch(context.Background(), "https://cdn.example/big.gif")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Fetch rejected a %d-byte cover: %v", len(body), err)
|
||||||
|
}
|
||||||
|
if len(got) != len(body) {
|
||||||
|
t.Fatalf("body = %d bytes, want %d", len(got), len(body))
|
||||||
|
}
|
||||||
|
if contentType != "image/gif" {
|
||||||
|
t.Fatalf("content type = %q, want image/gif", contentType)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestCoverFetcherRejectsNonImage(t *testing.T) {
|
func TestCoverFetcherRejectsNonImage(t *testing.T) {
|
||||||
var calls int
|
var calls int
|
||||||
client := &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
client := &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"log"
|
"log"
|
||||||
"net/url"
|
"net/url"
|
||||||
"sort"
|
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -48,19 +47,22 @@ type Poller struct {
|
|||||||
// same failure-isolated prefetch path.
|
// same failure-isolated prefetch path.
|
||||||
CoverBytesFetch CoverBytesFetcher
|
CoverBytesFetch CoverBytesFetcher
|
||||||
Now func() time.Time // injected so tests can freeze it
|
Now func() time.Time // injected so tests can freeze it
|
||||||
|
// eligibleCount reports how many of a Site's Series are eligible for
|
||||||
|
// polling, defaulting to Store.EligibleSeriesCount. Injected so tests can
|
||||||
|
// fail the count alone: the eligible query shares the due query's tables,
|
||||||
|
// so no real store failure can reach this path without breaking the due
|
||||||
|
// query first (issue #141).
|
||||||
|
eligibleCount func(site string) (int, error)
|
||||||
|
|
||||||
// refuseUntil gates a Site's Lane after it refused twice in one run: no
|
// refuseUntil gates a Site's Lane after it refused twice in one run: no
|
||||||
// Series of that Site is attempted again before this time (issue #100).
|
// Series of that Site is attempted again before this time (issue #100).
|
||||||
|
// The stamp is durable — the pass gate reads it from the store, so a
|
||||||
|
// restart does not forget the refusal; nothing of it lives in memory.
|
||||||
// browserDownAt is when a browser Lane last lost the sidecar; the other
|
// browserDownAt is when a browser Lane last lost the sidecar; the other
|
||||||
// browser Lanes skip their passes for the next refuseBackoff, so a
|
// browser Lanes skip their passes for the next RefuseBackoff, so a
|
||||||
// restarting Chrome does not stamp one Series per pass per Lane (story 20).
|
// restarting Chrome does not stamp one Series per pass per Lane (story 20).
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
refuseUntil map[string]time.Time
|
|
||||||
browserDownAt time.Time
|
browserDownAt time.Time
|
||||||
// laneStates is the owner's page snapshot of each Lane's last pass
|
|
||||||
// (issue #102), keyed by Site. Guarded by mu; a Site appears only after
|
|
||||||
// its first pass, so a restart renders "no data yet" rather than zeroes.
|
|
||||||
laneStates map[string]LaneState
|
|
||||||
// coverWG tracks in-flight cover work. Covers heal in the background so a
|
// coverWG tracks in-flight cover work. Covers heal in the background so a
|
||||||
// slow cover host cannot delay the next Series-page Poll; tests join it
|
// slow cover host cannot delay the next Series-page Poll; tests join it
|
||||||
// before asserting on cover fetches.
|
// before asserting on cover fetches.
|
||||||
@@ -167,14 +169,7 @@ func fetcherFor(site string, browser, tls Fetcher) Fetcher {
|
|||||||
// laneNames returns every registry Site in the deterministic order both Run
|
// laneNames returns every registry Site in the deterministic order both Run
|
||||||
// and runOnce iterate: sorted, so lane behaviour and its tests agree on who
|
// and runOnce iterate: sorted, so lane behaviour and its tests agree on who
|
||||||
// runs first.
|
// runs first.
|
||||||
func laneNames() []string {
|
func laneNames() []string { return SiteNames() }
|
||||||
names := make([]string, 0, len(sites))
|
|
||||||
for name := range sites {
|
|
||||||
names = append(names, name)
|
|
||||||
}
|
|
||||||
sort.Strings(names)
|
|
||||||
return names
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *Poller) Run(ctx context.Context) {
|
func (p *Poller) Run(ctx context.Context) {
|
||||||
names := laneNames()
|
names := laneNames()
|
||||||
@@ -216,28 +211,120 @@ func (p *Poller) runOnce(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// One skip value per way a Lane Pass can return before its loop (issue #141);
|
||||||
|
// empty means the pass reached the loop. The values are wire strings — stored
|
||||||
|
// in poll_passes and read by the Lanes page — so they are stable, not prose.
|
||||||
|
const (
|
||||||
|
// Exported so the web layer renders a skip's reason without retyping the
|
||||||
|
// wire string (issue #145); the values are storage and page-stable.
|
||||||
|
SkipPaused = "paused" // the pause row was read at the top
|
||||||
|
SkipRefusing = "refusing" // refusal backoff
|
||||||
|
SkipSidecarDown = "sidecar-down" // a sibling browser Lane lost Chrome
|
||||||
|
SkipNoFetcher = "no-fetcher" // browser Site, no browser configured, no fallback
|
||||||
|
SkipDueQuery = "due-query" // the due query failed
|
||||||
|
SkipAsleep = "asleep" // under both browser wake thresholds
|
||||||
|
SkipEligibleCount = "eligible-count" // the eligible count failed
|
||||||
|
SkipNothingEligible = "nothing-eligible" // nothing eligible; sleeps a full rest
|
||||||
|
)
|
||||||
|
|
||||||
|
// readOutcome classifies one Series read for the pass row's outcome counts
|
||||||
|
// (issue #141). The classification the read already makes is counted, never a
|
||||||
|
// second taxonomy: refused is the Site holding a challenge, unreachable the
|
||||||
|
// browser interrupting, noChapter a 200 with real HTML but no chapter links,
|
||||||
|
// unfetchable the host pin or a missing fetcher, and errors everything else.
|
||||||
|
type readOutcome int
|
||||||
|
|
||||||
|
const (
|
||||||
|
outcomeSuccess readOutcome = iota
|
||||||
|
outcomeRefused
|
||||||
|
outcomeUnreachable
|
||||||
|
outcomeNoChapter
|
||||||
|
outcomeUnfetchable
|
||||||
|
outcomeError
|
||||||
|
)
|
||||||
|
|
||||||
|
// outcomeCounts are the five named outcome counts of one pass. A success
|
||||||
|
// count is derived, never stored: checked minus the four, with unreachable
|
||||||
|
// excluded because the sidecar-loss path returns before the checked counter
|
||||||
|
// increments (issue #141).
|
||||||
|
type outcomeCounts struct {
|
||||||
|
refused, unreachable, noChapter, unfetchable, errors int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *outcomeCounts) add(o readOutcome) {
|
||||||
|
switch o {
|
||||||
|
case outcomeRefused:
|
||||||
|
c.refused++
|
||||||
|
case outcomeUnreachable:
|
||||||
|
c.unreachable++
|
||||||
|
case outcomeNoChapter:
|
||||||
|
c.noChapter++
|
||||||
|
case outcomeUnfetchable:
|
||||||
|
c.unfetchable++
|
||||||
|
case outcomeError:
|
||||||
|
c.errors++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// passRecord is what one pass's durable row will be: the skip value and
|
||||||
|
// outcome counts filled in along the pass's return path. recordPass assembles
|
||||||
|
// the row, so every exit records exactly once.
|
||||||
|
type passRecord struct {
|
||||||
|
site string
|
||||||
|
ranAt int64
|
||||||
|
skip string
|
||||||
|
counts outcomeCounts
|
||||||
|
}
|
||||||
|
|
||||||
|
// lanePassRetention is how far back a Lane's pass log is kept. It is not the
|
||||||
|
// display window: retention is how far back a question can reach, and the
|
||||||
|
// window is what the owner is shown (issue #139).
|
||||||
|
const lanePassRetention = 14 * 24 * time.Hour
|
||||||
|
|
||||||
// runLanePass processes one pass of one Site's Lane: select the due Series,
|
// runLanePass processes one pass of one Site's Lane: select the due Series,
|
||||||
// pace through them, and report how long the Lane should wait before its next
|
// pace through them, and report how long the Lane should wait before its next
|
||||||
// pass. paced spaces consecutive fetches by the Site's effective gap — the
|
// pass. paced spaces consecutive fetches by the Site's effective gap — the
|
||||||
// production Lane's rate limit; the deterministic test entry runs back to back.
|
// production Lane's rate limit; the deterministic test entry runs back to back.
|
||||||
func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration {
|
func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration {
|
||||||
now := p.Now()
|
now := p.Now()
|
||||||
// Snapshot this pass for the owner's page (issue #102). Recorded on every
|
// Durable pass log (issue #141): one row per exit. The figures are filled
|
||||||
// return path, with the figures filled in where the pass computes them.
|
// in as the pass measures them; a pass that returns before measuring
|
||||||
st := LaneState{Site: name, LastRun: now, Browser: isBrowserSite(name)}
|
// carries the previous pass's forward inside recordPass.
|
||||||
defer func() { p.recordLaneState(st) }()
|
fig := passFigures{}
|
||||||
if until := p.refusalBackoff(name); now.Before(until) {
|
rec := passRecord{site: name, ranAt: now.UnixMilli()}
|
||||||
|
defer func() { p.recordPass(rec, fig) }()
|
||||||
|
|
||||||
|
// One Lane row read at the top of a pass, serving two gates (issue #139).
|
||||||
|
// Both stamps outlive our process, so the gates read the durable row
|
||||||
|
// rather than memory: a refusal is the Site's mood and a pause the
|
||||||
|
// owner's order, and neither is lost to a restart.
|
||||||
|
pausedUntil, refuseUntil, err := p.Store.LaneGates(name)
|
||||||
|
if err != nil {
|
||||||
|
// Fail open: a store that cannot answer the gate cannot record the
|
||||||
|
// pass either, and one Lane must not stall on its own gate read.
|
||||||
|
log.Printf("latest poll %s: lane gates: %v", name, err)
|
||||||
|
}
|
||||||
|
if pausedUntil > now.UnixMilli() {
|
||||||
|
// Paused ahead of the refusal check: no Series is touched, so the
|
||||||
|
// queue stays intact for when the pause lifts (issue #141, #147).
|
||||||
|
rec.skip = SkipPaused
|
||||||
|
log.Printf("latest poll %s: paused until %s, skipping pass", name, time.UnixMilli(pausedUntil).Format(time.RFC3339))
|
||||||
|
return time.Duration(pausedUntil-now.UnixMilli()) * time.Millisecond
|
||||||
|
}
|
||||||
|
if refuseUntil > now.UnixMilli() {
|
||||||
// Cooling down after a refusal: do not attempt this Site at all.
|
// Cooling down after a refusal: do not attempt this Site at all.
|
||||||
return until.Sub(now)
|
rec.skip = SkipRefusing
|
||||||
|
return time.Duration(refuseUntil-now.UnixMilli()) * time.Millisecond
|
||||||
}
|
}
|
||||||
if isBrowserSite(name) {
|
if isBrowserSite(name) {
|
||||||
if downFor, down := p.browserDownFor(now); down && downFor < refuseBackoff {
|
if downFor, down := p.browserDownFor(now); down && downFor < RefuseBackoff {
|
||||||
// A sibling browser Lane lost the sidecar within the backoff
|
// A sibling browser Lane lost the sidecar within the backoff
|
||||||
// window: skip this pass, so a restarting Chrome does not stamp
|
// window: skip this pass, so a restarting Chrome does not stamp
|
||||||
// this Site's Series one pass at a time. After refuseBackoff the
|
// this Site's Series one pass at a time. After RefuseBackoff the
|
||||||
// flag decays and the Lane probes again (issue #100, story 20).
|
// flag decays and the Lane probes again (issue #100, story 20).
|
||||||
|
rec.skip = SkipSidecarDown
|
||||||
log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor)
|
log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor)
|
||||||
return refuseBackoff - downFor
|
return RefuseBackoff - downFor
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
s := sites[name]
|
s := sites[name]
|
||||||
@@ -246,24 +333,29 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
|
|||||||
// No fetcher at all right now (browser absent, no fallback): every
|
// No fetcher at all right now (browser absent, no fallback): every
|
||||||
// Series stays unstamped and due, so a browser that appears after a
|
// Series stays unstamped and due, so a browser that appears after a
|
||||||
// restart finds its full queue waiting (issue #100).
|
// restart finds its full queue waiting (issue #100).
|
||||||
st.Gap = defaultGap
|
rec.skip = SkipNoFetcher
|
||||||
|
fig.Gap = defaultGap
|
||||||
return defaultGap
|
return defaultGap
|
||||||
}
|
}
|
||||||
|
|
||||||
due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli(),
|
due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli(),
|
||||||
now.Add(-sightingCeilingRests*s.Rest).UnixMilli())
|
now.Add(-sightingCeilingRests*s.Rest).UnixMilli())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
rec.skip = SkipDueQuery
|
||||||
log.Printf("latest poll %s: due query: %v", name, err)
|
log.Printf("latest poll %s: due query: %v", name, err)
|
||||||
st.Gap = defaultGap
|
fig.Gap = defaultGap
|
||||||
return defaultGap
|
return defaultGap
|
||||||
}
|
}
|
||||||
st.Due = len(due)
|
fig.Due = len(due)
|
||||||
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) {
|
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) && !anyForced(due) {
|
||||||
// Below both thresholds Chrome stays asleep (ADR-0005 on-demand
|
// Below both thresholds Chrome stays asleep (ADR-0005 on-demand
|
||||||
// browser): waking it for a single Poll would cost a challenge solve
|
// browser): waking it for a single Poll would cost a challenge solve
|
||||||
// per request. The Lane still paces at the default gap, which is what
|
// per request. A forced Series is the one exception — a human asking
|
||||||
// the owner's page must show rather than a zero.
|
// is not the machine waking itself (issue #146). The Lane still paces
|
||||||
st.Gap, st.Asleep = defaultGap, true
|
// at the default gap, which is what the owner's page must show rather
|
||||||
|
// than a zero.
|
||||||
|
rec.skip = SkipAsleep
|
||||||
|
fig.Gap = defaultGap
|
||||||
return defaultGap
|
return defaultGap
|
||||||
}
|
}
|
||||||
if s.Browser != nil {
|
if s.Browser != nil {
|
||||||
@@ -276,20 +368,22 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
eligible, err := p.Store.EligibleSeriesCount(name)
|
eligible, err := p.countEligible(name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
rec.skip = SkipEligibleCount
|
||||||
log.Printf("latest poll %s: eligible count: %v", name, err)
|
log.Printf("latest poll %s: eligible count: %v", name, err)
|
||||||
st.Gap = defaultGap
|
fig.Gap = defaultGap
|
||||||
return defaultGap
|
return defaultGap
|
||||||
}
|
}
|
||||||
gap, clamped := effectiveGap(s, eligible)
|
gap, clamped := effectiveGap(s, eligible)
|
||||||
st.Gap, st.Clamped = gap, clamped
|
fig.Gap, fig.Clamped = gap, clamped
|
||||||
if clamped {
|
if clamped {
|
||||||
log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible)
|
log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible)
|
||||||
}
|
}
|
||||||
if eligible == 0 {
|
if eligible == 0 {
|
||||||
// Nothing to poll for the foreseeable future; sleep a full rest instead
|
// Nothing to poll for the foreseeable future; sleep a full rest instead
|
||||||
// of re-querying every gap.
|
// of re-querying every gap.
|
||||||
|
rec.skip = SkipNothingEligible
|
||||||
return s.Rest
|
return s.Rest
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -300,7 +394,7 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
|
|||||||
}
|
}
|
||||||
if refusals >= 2 {
|
if refusals >= 2 {
|
||||||
// This Site refused twice in a row: the remaining Series are left
|
// This Site refused twice in a row: the remaining Series are left
|
||||||
// unstamped and due, and the Lane waits refuseBackoff before
|
// unstamped and due, and the Lane waits RefuseBackoff before
|
||||||
// trying it again.
|
// trying it again.
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -314,46 +408,98 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := p.checkOne(ctx, sr); err != nil {
|
outcome := p.checkOne(ctx, sr)
|
||||||
switch {
|
if outcome == outcomeUnreachable {
|
||||||
case errors.Is(err, errChallengeHeld):
|
// The mid-loop browser loss writes an empty skip on purpose: the
|
||||||
refusals++
|
// pass returns before the checked counter increments, so its row
|
||||||
case errors.Is(err, errBrowserInterrupted):
|
// is stall-shaped (due > 0, checked 0, skip ''), and a stall is
|
||||||
p.setBrowserDown(now)
|
// the exact signal this exit produces. A tenth skip value would
|
||||||
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, refuseBackoff)
|
// make it legible but is deliberately not invented here.
|
||||||
return gap
|
rec.counts.add(outcome)
|
||||||
default:
|
p.setBrowserDown(now)
|
||||||
refusals = 0
|
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, RefuseBackoff)
|
||||||
}
|
return gap
|
||||||
|
}
|
||||||
|
if outcome == outcomeRefused {
|
||||||
|
refusals++
|
||||||
} else {
|
} else {
|
||||||
refusals = 0
|
refusals = 0
|
||||||
}
|
}
|
||||||
st.Checked++
|
rec.counts.add(outcome)
|
||||||
|
fig.Checked++
|
||||||
}
|
}
|
||||||
if st.Checked > 0 {
|
if fig.Checked > 0 {
|
||||||
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), st.Checked)
|
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), fig.Checked)
|
||||||
}
|
}
|
||||||
if refusals >= 2 {
|
if refusals >= 2 {
|
||||||
p.setRefusalBackoff(name, now.Add(refuseBackoff))
|
// The refusal outlives the process: the durable stamp gates a restart,
|
||||||
log.Printf("latest poll %s: refused twice this run, waiting %s", name, refuseBackoff)
|
// so a Site that just told us to back off is not re-probed.
|
||||||
return refuseBackoff
|
if err := p.Store.SetLaneRefusal(name, now.Add(RefuseBackoff).UnixMilli()); err != nil {
|
||||||
|
log.Printf("latest poll %s: persist refusal: %v", name, err)
|
||||||
|
}
|
||||||
|
log.Printf("latest poll %s: refused twice this run, waiting %s", name, RefuseBackoff)
|
||||||
|
return RefuseBackoff
|
||||||
}
|
}
|
||||||
return gap
|
return gap
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Poller) refusalBackoff(name string) time.Time {
|
// passFigures are the numbers one pass measured for its durable row (issue
|
||||||
p.mu.Lock()
|
// #141): due and checked as the pass saw them, the pace it chose, and whether
|
||||||
defer p.mu.Unlock()
|
// the gap sat on the floor. A pass that returned before measuring keeps the
|
||||||
return p.refuseUntil[name]
|
// previous pass's figures via carry-forward in recordPass; the in-memory
|
||||||
|
// snapshot those once mirrored into is gone — the page reads the durable row
|
||||||
|
// now (issue #145).
|
||||||
|
type passFigures struct {
|
||||||
|
Due, Checked int
|
||||||
|
Gap time.Duration
|
||||||
|
Clamped bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Poller) setRefusalBackoff(name string, until time.Time) {
|
// recordPass writes the durable row for one pass (issue #141). Called deferred
|
||||||
p.mu.Lock()
|
// from runLanePass so every return path records exactly one row. A pass that
|
||||||
defer p.mu.Unlock()
|
// never computed its own figures — its gap is zero — carries the previous
|
||||||
if p.refuseUntil == nil {
|
// pass's due, gap, clamped and checked forward rather than stating zeroes it
|
||||||
p.refuseUntil = make(map[string]time.Time)
|
// did not measure; the skip column says why it declined, so the zeroes that
|
||||||
|
// remain (due-query, no-fetcher) read as explanations rather than
|
||||||
|
// measurements.
|
||||||
|
func (p *Poller) recordPass(rec passRecord, fig passFigures) {
|
||||||
|
row := store.LanePass{
|
||||||
|
Site: rec.site,
|
||||||
|
RanAt: rec.ranAt,
|
||||||
|
Skip: rec.skip,
|
||||||
|
Due: fig.Due,
|
||||||
|
Checked: fig.Checked,
|
||||||
|
GapMS: fig.Gap.Milliseconds(),
|
||||||
|
Clamped: fig.Clamped,
|
||||||
|
Refused: rec.counts.refused,
|
||||||
|
Unreachable: rec.counts.unreachable,
|
||||||
|
NoChapter: rec.counts.noChapter,
|
||||||
|
Unfetchable: rec.counts.unfetchable,
|
||||||
|
Errors: rec.counts.errors,
|
||||||
}
|
}
|
||||||
p.refuseUntil[name] = until
|
if row.GapMS == 0 {
|
||||||
|
// The pass never computed a gap, so it has no figures of its own:
|
||||||
|
// carry the previous pass's, in one latest-per-Site read — the
|
||||||
|
// recorder needs one Site, not six (issue #139).
|
||||||
|
if prev, ok, err := p.Store.LatestLanePass(rec.site); err != nil {
|
||||||
|
log.Printf("latest poll %s: previous pass: %v", rec.site, err)
|
||||||
|
} else if ok {
|
||||||
|
row.Due, row.Checked = prev.Due, prev.Checked
|
||||||
|
row.GapMS, row.Clamped = prev.GapMS, prev.Clamped
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := p.Store.RecordLanePass(row, rec.ranAt-lanePassRetention.Milliseconds()); err != nil {
|
||||||
|
log.Printf("latest poll %s: record lane pass: %v", rec.site, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// countEligible routes the eligible count through the test seam when one is
|
||||||
|
// set, else the store.
|
||||||
|
func (p *Poller) countEligible(site string) (int, error) {
|
||||||
|
if p.eligibleCount != nil {
|
||||||
|
return p.eligibleCount(site)
|
||||||
|
}
|
||||||
|
return p.Store.EligibleSeriesCount(site)
|
||||||
}
|
}
|
||||||
|
|
||||||
// setBrowserDown records when a browser Lane lost the sidecar. It is Poller
|
// setBrowserDown records when a browser Lane lost the sidecar. It is Poller
|
||||||
@@ -366,7 +512,7 @@ func (p *Poller) setBrowserDown(now time.Time) {
|
|||||||
|
|
||||||
// browserDownFor reports how long the sidecar has been down and that it is
|
// browserDownFor reports how long the sidecar has been down and that it is
|
||||||
// down at all — the zero time means never down, which must not read as a
|
// down at all — the zero time means never down, which must not read as a
|
||||||
// zero-duration loss. The window decays: once refuseBackoff passes without a
|
// zero-duration loss. The window decays: once RefuseBackoff passes without a
|
||||||
// fresh loss, Lanes probe again.
|
// fresh loss, Lanes probe again.
|
||||||
func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) {
|
func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) {
|
||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
@@ -395,6 +541,19 @@ func browserWakeDue(due []store.Series, now time.Time, rest time.Duration) bool
|
|||||||
return maxSeriesWait(due, now, rest) >= browserWakeAge
|
return maxSeriesWait(due, now, rest) >= browserWakeAge
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// anyForced reports whether the due list holds a forced Series: one whose
|
||||||
|
// owner check-now request (issue #146) has not been answered yet. A human
|
||||||
|
// asking wakes a sleeping Chrome even below the wake thresholds; the request
|
||||||
|
// itself still ages visibly if the home machine is off.
|
||||||
|
func anyForced(due []store.Series) bool {
|
||||||
|
for _, sr := range due {
|
||||||
|
if sr.Forced {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// maxSeriesWait returns how long the most-overdue of the due Series has been
|
// maxSeriesWait returns how long the most-overdue of the due Series has been
|
||||||
// waiting past its due moment (0 when due is empty).
|
// waiting past its due moment (0 when due is empty).
|
||||||
func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration {
|
func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration {
|
||||||
@@ -409,13 +568,15 @@ func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.D
|
|||||||
|
|
||||||
// checkOne re-checks one series. Every failure path here is "log and move on":
|
// checkOne re-checks one series. Every failure path here is "log and move on":
|
||||||
// the poller is a best-effort enhancement, and no single bad series may stall a
|
// the poller is a best-effort enhancement, and no single bad series may stall a
|
||||||
// Lane or take down the process. The returned error is the page read's
|
// Lane or take down the process. The returned outcome classifies the read for
|
||||||
// classified outcome so the Lane can tell a refusal from a loss of the
|
// the pass row (issue #141), so the Lane can count a refusal, a lost browser,
|
||||||
// browser; non-classified failures still return nil-equivalent behaviour.
|
// a chapter-less page, an unfetchable address or a transport error without
|
||||||
func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
// re-deriving the taxonomy.
|
||||||
|
func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOutcome) {
|
||||||
defer func() {
|
defer func() {
|
||||||
if r := recover(); r != nil {
|
if r := recover(); r != nil {
|
||||||
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
|
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
|
||||||
|
outcome = outcomeError
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
@@ -427,7 +588,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
|||||||
// the stamp means "attempted", and an untried Series stays due.
|
// the stamp means "attempted", and an untried Series stays due.
|
||||||
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
|
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
|
||||||
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
|
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
|
||||||
return nil
|
return outcomeError
|
||||||
}
|
}
|
||||||
|
|
||||||
facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch)
|
facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch)
|
||||||
@@ -438,17 +599,23 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
|||||||
// passes the gate is retried at rest pace rather than
|
// passes the gate is retried at rest pace rather than
|
||||||
// hot-looping.
|
// hot-looping.
|
||||||
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
|
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
|
||||||
return err
|
return outcomeUnfetchable
|
||||||
case errors.Is(err, errNoFetcher):
|
case errors.Is(err, errNoFetcher):
|
||||||
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
|
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
|
||||||
return err
|
return outcomeUnfetchable
|
||||||
}
|
}
|
||||||
// A legacy cover heals independently of the page read: its source may
|
// A legacy cover heals independently of the page read: its source may
|
||||||
// answer — a CDN — while the origin does not, so a fetch failure does
|
// answer — a CDN — while the origin does not, so a fetch failure does
|
||||||
// not skip the heal, matching the order the shared read replaced.
|
// not skip the heal, matching the order the shared read replaced.
|
||||||
p.healCover(ctx, sr)
|
p.healCover(ctx, sr)
|
||||||
log.Printf("latest poll %q: %v", sr.Key(), err)
|
log.Printf("latest poll %q: %v", sr.Key(), err)
|
||||||
return err
|
if errors.Is(err, errChallengeHeld) {
|
||||||
|
return outcomeRefused
|
||||||
|
}
|
||||||
|
if errors.Is(err, errBrowserInterrupted) {
|
||||||
|
return outcomeUnreachable
|
||||||
|
}
|
||||||
|
return outcomeError
|
||||||
}
|
}
|
||||||
// A legacy cover source is healed independently of the page read.
|
// A legacy cover source is healed independently of the page read.
|
||||||
p.healCover(ctx, sr)
|
p.healCover(ctx, sr)
|
||||||
@@ -459,7 +626,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
|||||||
// Most likely a challenge page or a layout change. Either way the row is
|
// Most likely a challenge page or a layout change. Either way the row is
|
||||||
// already stamped, so this waits out a rest instead of hot-looping.
|
// already stamped, so this waits out a rest instead of hot-looping.
|
||||||
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen)
|
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen)
|
||||||
return nil
|
return outcomeNoChapter
|
||||||
}
|
}
|
||||||
|
|
||||||
// The Poll is the oracle for whatever Sighting last raised this Series
|
// The Poll is the oracle for whatever Sighting last raised this Series
|
||||||
@@ -472,7 +639,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
|||||||
// against the due-query snapshot; a concurrent write in between only costs
|
// against the due-query snapshot; a concurrent write in between only costs
|
||||||
// one redundant UPDATE of the same absolute value, never a wrong one.
|
// one redundant UPDATE of the same absolute value, never a wrong one.
|
||||||
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num {
|
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num {
|
||||||
return nil
|
return outcomeSuccess
|
||||||
}
|
}
|
||||||
|
|
||||||
// Series-level write: the row is shared, so one update refreshes every
|
// Series-level write: the row is shared, so one update refreshes every
|
||||||
@@ -481,10 +648,10 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
|||||||
// the list.
|
// the list.
|
||||||
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil {
|
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil {
|
||||||
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
|
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
|
||||||
return nil
|
return outcomeError
|
||||||
}
|
}
|
||||||
log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label)
|
log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label)
|
||||||
return nil
|
return outcomeSuccess
|
||||||
}
|
}
|
||||||
|
|
||||||
// judgeSighting settles the Sighting the Series' stored Latest Chapter is owed
|
// judgeSighting settles the Sighting the Series' stored Latest Chapter is owed
|
||||||
|
|||||||
@@ -427,9 +427,12 @@ func TestRunLogsLaneDefaults(t *testing.T) {
|
|||||||
log.SetOutput(&logs)
|
log.SetOutput(&logs)
|
||||||
t.Cleanup(func() { log.SetOutput(previous) })
|
t.Cleanup(func() { log.SetOutput(previous) })
|
||||||
|
|
||||||
|
// The pass log writes through the store, so Run needs a real one — a
|
||||||
|
// poller without a Store is not a poller.
|
||||||
|
s, _ := newTestStore(t)
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
cancel()
|
cancel()
|
||||||
(&Poller{Now: func() time.Time { return time.Now() }}).Run(ctx)
|
(&Poller{Store: s, Now: func() time.Time { return time.Now() }}).Run(ctx)
|
||||||
|
|
||||||
got := logs.String()
|
got := logs.String()
|
||||||
for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} {
|
for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} {
|
||||||
@@ -1305,7 +1308,7 @@ func TestRunOnceOrdersBySharednessThenAge(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Two refusals in one pass stop the Lane: the remaining Series stay unstamped
|
// Two refusals in one pass stop the Lane: the remaining Series stay unstamped
|
||||||
// and due, and the Lane backs off for refuseBackoff before trying the Site
|
// and due, and the Lane backs off for RefuseBackoff before trying the Site
|
||||||
// again. One hostile Site burns only its own Lane's budget (issue #100).
|
// again. One hostile Site burns only its own Lane's budget (issue #100).
|
||||||
func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) {
|
func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) {
|
||||||
s, _ := newTestStore(t)
|
s, _ := newTestStore(t)
|
||||||
@@ -1388,11 +1391,9 @@ func TestBrowserLaneWakeThresholds(t *testing.T) {
|
|||||||
if got := browser.callCount(); got != 0 {
|
if got := browser.callCount(); got != 0 {
|
||||||
t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got)
|
t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got)
|
||||||
}
|
}
|
||||||
// The owner's page reads this state off the snapshot, and Due-without-
|
// The skipped pass still records its row, carrying the due count it never
|
||||||
// Checked has to be distinguishable there from a Lane that has stopped.
|
// read; the Lanes page (issue #145) reads that row — see
|
||||||
if lane := laneByName(t, p, "kagane"); !lane.Asleep || lane.Due != 3 || lane.Checked != 0 {
|
// TestRunLanePassRecordsEveryExit/"asleep".
|
||||||
t.Fatalf("asleep kagane lane = %+v, want Asleep with 3 due and 0 checked", lane)
|
|
||||||
}
|
|
||||||
// 5 due crosses the count threshold.
|
// 5 due crosses the count threshold.
|
||||||
for i := 3; i < 5; i++ {
|
for i := 3; i < 5; i++ {
|
||||||
seed(i)
|
seed(i)
|
||||||
@@ -1401,9 +1402,6 @@ func TestBrowserLaneWakeThresholds(t *testing.T) {
|
|||||||
if got := browser.callCount(); got != 5 {
|
if got := browser.callCount(); got != 5 {
|
||||||
t.Fatalf("browser fetches with 5 due series = %d, want 5", got)
|
t.Fatalf("browser fetches with 5 due series = %d, want 5", got)
|
||||||
}
|
}
|
||||||
if lane := laneByName(t, p, "kagane"); lane.Asleep {
|
|
||||||
t.Fatalf("woken kagane lane still reports Asleep: %+v", lane)
|
|
||||||
}
|
|
||||||
// A single long-neglected series wakes the browser by age alone.
|
// A single long-neglected series wakes the browser by age alone.
|
||||||
seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0)
|
seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0)
|
||||||
p.runOnce(context.Background())
|
p.runOnce(context.Background())
|
||||||
@@ -1412,19 +1410,6 @@ func TestBrowserLaneWakeThresholds(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// laneByName pulls one Lane out of the poller's snapshot, failing rather than
|
|
||||||
// returning a zero LaneState a caller would assert against by accident.
|
|
||||||
func laneByName(t *testing.T, p *Poller, site string) LaneState {
|
|
||||||
t.Helper()
|
|
||||||
for _, lane := range p.LaneStatus().Lanes {
|
|
||||||
if lane.Site == site {
|
|
||||||
return lane
|
|
||||||
}
|
|
||||||
}
|
|
||||||
t.Fatalf("no %q lane in the snapshot", site)
|
|
||||||
return LaneState{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// When one browser Lane loses the sidecar, the round's remaining browser
|
// When one browser Lane loses the sidecar, the round's remaining browser
|
||||||
// Lanes are skipped: every fetch would fail anyway, and their Series must not
|
// Lanes are skipped: every fetch would fail anyway, and their Series must not
|
||||||
// burn their stamps on a dead Chrome (issue #100).
|
// burn their stamps on a dead Chrome (issue #100).
|
||||||
@@ -1456,7 +1441,7 @@ func TestRunOnceUnreachableBrowserStopsBrowserLanes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The shared flag decays after refuseBackoff: the next round probes
|
// The shared flag decays after RefuseBackoff: the next round probes
|
||||||
// again. comix's Series is resting (stamped last round), so the probe
|
// again. comix's Series is resting (stamped last round), so the probe
|
||||||
// falls to kagane — the only Lane with something due — and its fresh
|
// falls to kagane — the only Lane with something due — and its fresh
|
||||||
// loss re-gates the Lanes behind it.
|
// loss re-gates the Lanes behind it.
|
||||||
@@ -1602,19 +1587,21 @@ func TestRunOnceClampWarningNamesTheSite(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The owner's admin page (issue #102) reads Lane state out of the poller.
|
// The Lanes page (issue #145) reads the durable pass log, so the poller's
|
||||||
// Before any pass the snapshot is empty — a restart must render "no data
|
// only duty to it is that every return path writes its row; the snapshot it
|
||||||
// yet", not zeroes — and each pass records what it saw: the frozen clock,
|
// used to mirror into memory is gone. A refusing pass still records why it
|
||||||
// the due count and the pace, with refusal backoff derived at snapshot time.
|
// declined, and the latest row per Site is what the page renders — nothing
|
||||||
func TestLaneStatus(t *testing.T) {
|
// else is left to assert against here, because the page's seam moved into the
|
||||||
|
// web layer (web_test.go, seeded-row tests).
|
||||||
|
func TestPassLogIsTheLanesPagesOnlyWindow(t *testing.T) {
|
||||||
s, _ := newTestStore(t)
|
s, _ := newTestStore(t)
|
||||||
now := time.UnixMilli(5_000_000)
|
now := time.UnixMilli(5_000_000)
|
||||||
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
|
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
|
||||||
|
|
||||||
if st := p.LaneStatus(); len(st.Lanes) != 0 {
|
// No pass has run: the log is empty, which the page renders as "none
|
||||||
t.Fatalf("lanes before any pass = %d, want 0 (nothing has run)", len(st.Lanes))
|
// observed" rather than confident zeroes.
|
||||||
} else if st.BrowserConfigured || st.BrowserReachable {
|
if _, ok, err := s.LatestLanePass("asura"); err != nil || ok {
|
||||||
t.Fatalf("browser before any pass = configured=%v reachable=%v, want false without a browser fetcher", st.BrowserConfigured, st.BrowserReachable)
|
t.Fatalf("LatestLanePass before any pass = ok %v err %v, want no row", ok, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
seedForCheck(t, s, "asura:chronicles", "https://asurascans.com/series/chronicles", 0)
|
seedForCheck(t, s, "asura:chronicles", "https://asurascans.com/series/chronicles", 0)
|
||||||
@@ -1627,65 +1614,726 @@ func TestLaneStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
p.runOnce(context.Background())
|
p.runOnce(context.Background())
|
||||||
|
|
||||||
st := p.LaneStatus()
|
// Both lanes wrote their rows: asura the pass it read; kagane the
|
||||||
var asura, kagane LaneState
|
// mid-loop double-refusal exit, which records an empty skip by design
|
||||||
for _, lane := range st.Lanes {
|
// with its two refused counts — the durable row is the whole record, and
|
||||||
switch lane.Site {
|
// the page reads it rather than a snapshot.
|
||||||
case "asura":
|
asura := latestPassFor(t, s, "asura")
|
||||||
asura = lane
|
if asura.Due != 1 || asura.Checked != 1 || asura.GapMS == 0 {
|
||||||
case "kagane":
|
t.Fatalf("asura row = %+v, want due 1 checked 1 with the Lane's pace", asura)
|
||||||
kagane = lane
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if st.Lanes[0].Site != "asura" {
|
kagane := latestPassFor(t, s, "kagane")
|
||||||
t.Fatalf("first lane = %q, want asura (snapshot sorted by Site)", st.Lanes[0].Site)
|
if kagane.Skip != "" || kagane.Refused != 2 || kagane.Due != 2 || kagane.GapMS == 0 {
|
||||||
}
|
t.Fatalf("kagane row = %+v, want an empty-skip double-refusal pass with 2 refused", kagane)
|
||||||
if asura.Site == "" {
|
|
||||||
t.Fatalf("asura missing from snapshot: %+v", st.Lanes)
|
|
||||||
}
|
|
||||||
if !asura.LastRun.Equal(now) {
|
|
||||||
t.Fatalf("asura LastRun = %s, want the frozen clock %s", asura.LastRun, now)
|
|
||||||
}
|
|
||||||
if asura.Due != 1 {
|
|
||||||
t.Fatalf("asura Due = %d, want 1", asura.Due)
|
|
||||||
}
|
|
||||||
if asura.Gap == 0 {
|
|
||||||
t.Fatal("asura Gap = 0, want the Lane's pace")
|
|
||||||
}
|
|
||||||
if asura.Browser || asura.Refusing {
|
|
||||||
t.Fatalf("asura = %+v, want a TLS Lane that is not refusing", asura)
|
|
||||||
}
|
|
||||||
if !kagane.Browser || !kagane.Refusing {
|
|
||||||
t.Fatalf("kagane = %+v, want a browser Lane in refusal backoff", kagane)
|
|
||||||
}
|
|
||||||
if !st.BrowserConfigured || !st.BrowserReachable {
|
|
||||||
t.Fatalf("browser after round = configured=%v reachable=%v, want true/true (sidecar never lost)", st.BrowserConfigured, st.BrowserReachable)
|
|
||||||
}
|
|
||||||
if asura.Checked != 1 {
|
|
||||||
t.Fatalf("asura Checked = %d, want the one Series it read", asura.Checked)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// A pass that declines to look (kagane is now in backoff) must not restate
|
// With the refusal now durable, the next pass declines ahead of the loop:
|
||||||
// the figures it never gathered as zeroes: the last real pass's due count
|
// it records the refusing skip and carries the previous pass's figures
|
||||||
// and pace stand until a pass replaces them.
|
// forward rather than restating zeroes it never gathered (the carry logic
|
||||||
|
// itself is driven in TestRunLanePassCarryForwardOnlyWhenGapZero).
|
||||||
before := kagane
|
before := kagane
|
||||||
if before.Due == 0 || before.Gap == 0 {
|
p.Now = func() time.Time { return now.Add(time.Minute) }
|
||||||
t.Fatalf("kagane after its refusing pass = %+v, want the figures that pass gathered", before)
|
|
||||||
}
|
|
||||||
p.runOnce(context.Background())
|
p.runOnce(context.Background())
|
||||||
for _, lane := range p.LaneStatus().Lanes {
|
again := latestPassFor(t, s, "kagane")
|
||||||
if lane.Site != "kagane" {
|
if again.Skip != SkipRefusing {
|
||||||
continue
|
t.Fatalf("kagane refusing pass skip = %q, want %q", again.Skip, SkipRefusing)
|
||||||
}
|
|
||||||
if lane.Due != before.Due || lane.Gap != before.Gap {
|
|
||||||
t.Fatalf("kagane after a skipped pass = due %d gap %s, want the previous pass's %d / %s",
|
|
||||||
lane.Due, lane.Gap, before.Due, before.Gap)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
if again.Due != before.Due || again.GapMS != before.GapMS || again.Checked != before.Checked {
|
||||||
// A lost sidecar reads as unreachable for the same window the Lanes skip.
|
t.Fatalf("kagane after a skipped pass = due %d gap %d, want the previous pass's %d / %d",
|
||||||
p.setBrowserDown(now)
|
again.Due, again.GapMS, before.Due, before.GapMS)
|
||||||
if st := p.LaneStatus(); !st.BrowserConfigured || st.BrowserReachable {
|
}
|
||||||
t.Fatalf("browser after loss = configured=%v reachable=%v, want true/false", st.BrowserConfigured, st.BrowserReachable)
|
}
|
||||||
|
|
||||||
|
// latestPassFor reads a Site's newest durable pass row, failing rather than
|
||||||
|
// returning a zero LanePass a caller would assert against by accident.
|
||||||
|
func latestPassFor(t *testing.T, s *store.Store, site string) store.LanePass {
|
||||||
|
t.Helper()
|
||||||
|
pass, ok, err := s.LatestLanePass(site)
|
||||||
|
if err != nil || !ok {
|
||||||
|
t.Fatalf("LatestLanePass(%s): ok=%v err=%v", site, ok, err)
|
||||||
|
}
|
||||||
|
return pass
|
||||||
|
}
|
||||||
|
|
||||||
|
// countPassRows counts a Site's durable pass rows, for asserting that a pass
|
||||||
|
// records exactly one.
|
||||||
|
func countPassRows(t *testing.T, dbURL, site string) int {
|
||||||
|
t.Helper()
|
||||||
|
db, err := sql.Open("pgx", dbURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("open %s: %v", dbURL, err)
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
var n int
|
||||||
|
if err := db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1`, site).Scan(&n); err != nil {
|
||||||
|
t.Fatalf("count passes for %s: %v", site, err)
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
// One durable pass row per exit, with the skip value naming the exit. The
|
||||||
|
// mid-loop browser-unreachable return also writes one row — but with an empty
|
||||||
|
// skip, so the row is stall-shaped (due > 0, checked 0, skip ”), matching the
|
||||||
|
// deliberate absence of a tenth skip value.
|
||||||
|
func TestRunLanePassRecordsEveryExit(t *testing.T) {
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
|
||||||
|
t.Run("paused", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
|
||||||
|
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||||
|
p := newTestPoller(t, s, f, now)
|
||||||
|
if pace := p.runLanePass(context.Background(), "asura", false); pace != 30*time.Minute {
|
||||||
|
t.Fatalf("paused pace = %s, want 30m (sleep until the expiry)", pace)
|
||||||
|
}
|
||||||
|
if f.callCount() != 0 {
|
||||||
|
t.Fatalf("fetches while paused = %d, want 0", f.callCount())
|
||||||
|
}
|
||||||
|
if got := readLatestCheckedAt(t, s, "asura:x"); got != 0 {
|
||||||
|
t.Fatalf("stamp while paused = %d, want 0 (Series stay due and unstamped)", got)
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipPaused {
|
||||||
|
t.Fatalf("skip = %q, want %q", pass.Skip, SkipPaused)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "asura"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("refusing", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "kagane:x", "https://kagane.to/series/x", 0)
|
||||||
|
if err := s.SetLaneRefusal("kagane", now.Add(10*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
browser := &fakeFetcher{status: 200}
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = browser
|
||||||
|
if pace := p.runLanePass(context.Background(), "kagane", false); pace != 10*time.Minute {
|
||||||
|
t.Fatalf("refusing pace = %s, want 10m", pace)
|
||||||
|
}
|
||||||
|
if browser.callCount() != 0 {
|
||||||
|
t.Fatalf("fetches while refusing = %d, want 0", browser.callCount())
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "kagane"); pass.Skip != SkipRefusing {
|
||||||
|
t.Fatalf("skip = %q, want %q", pass.Skip, SkipRefusing)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "kagane"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("sidecar-down", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "kagane:x", "https://kagane.to/series/x", 0)
|
||||||
|
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = browser
|
||||||
|
p.setBrowserDown(now) // a sibling Lane lost Chrome within the backoff window
|
||||||
|
if pace := p.runLanePass(context.Background(), "kagane", false); pace != RefuseBackoff {
|
||||||
|
t.Fatalf("sidecar-down pace = %s, want %s", pace, RefuseBackoff)
|
||||||
|
}
|
||||||
|
if browser.callCount() != 0 {
|
||||||
|
t.Fatalf("fetches with the sidecar down = %d, want 0", browser.callCount())
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "kagane"); pass.Skip != SkipSidecarDown {
|
||||||
|
t.Fatalf("skip = %q, want %q", pass.Skip, SkipSidecarDown)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "kagane"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("no-fetcher", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
if pace := p.runLanePass(context.Background(), "comix", false); pace != defaultGap {
|
||||||
|
t.Fatalf("no-fetcher pace = %s, want %s", pace, defaultGap)
|
||||||
|
}
|
||||||
|
pass := latestPassFor(t, s, "comix")
|
||||||
|
if pass.Skip != SkipNoFetcher || pass.GapMS != defaultGap.Milliseconds() {
|
||||||
|
t.Fatalf("no-fetcher pass = %+v, want skip %q with its own gap %s", pass, SkipNoFetcher, defaultGap)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "comix"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("due-query", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
|
||||||
|
// The due query is the pass's first store read after the gates; making
|
||||||
|
// it fail without touching poll_passes takes its tables away.
|
||||||
|
db, err := sql.Open("pgx", dbURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("open %s: %v", dbURL, err)
|
||||||
|
}
|
||||||
|
if _, err := db.Exec(`DROP TABLE bookmarks`); err != nil {
|
||||||
|
t.Fatalf("drop bookmarks: %v", err)
|
||||||
|
}
|
||||||
|
db.Close()
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
|
||||||
|
if pace := p.runLanePass(context.Background(), "asura", false); pace != defaultGap {
|
||||||
|
t.Fatalf("due-query pace = %s, want %s", pace, defaultGap)
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipDueQuery {
|
||||||
|
t.Fatalf("skip = %q, want %q", pass.Skip, SkipDueQuery)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "asura"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("asleep", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
key := fmt.Sprintf("kagane:w%d", i)
|
||||||
|
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], now.Add(-62*time.Minute).UnixMilli())
|
||||||
|
}
|
||||||
|
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = browser
|
||||||
|
if pace := p.runLanePass(context.Background(), "kagane", false); pace != defaultGap {
|
||||||
|
t.Fatalf("asleep pace = %s, want %s", pace, defaultGap)
|
||||||
|
}
|
||||||
|
if browser.callCount() != 0 {
|
||||||
|
t.Fatalf("fetches while Chrome is asleep = %d, want 0", browser.callCount())
|
||||||
|
}
|
||||||
|
pass := latestPassFor(t, s, "kagane")
|
||||||
|
if pass.Skip != SkipAsleep || pass.Due != 3 || pass.GapMS != defaultGap.Milliseconds() {
|
||||||
|
t.Fatalf("asleep pass = %+v, want skip %q with 3 due and the default gap", pass, SkipAsleep)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "kagane"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("eligible-count", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
// The eligible query shares the due query's tables, so no real store
|
||||||
|
// failure reaches it after a successful due read; the seam is how the
|
||||||
|
// path is driven at all (see Poller.eligibleCount).
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.eligibleCount = func(string) (int, error) { return 0, errors.New("count failed") }
|
||||||
|
if pace := p.runLanePass(context.Background(), "asura", false); pace != defaultGap {
|
||||||
|
t.Fatalf("eligible-count pace = %s, want %s", pace, defaultGap)
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipEligibleCount {
|
||||||
|
t.Fatalf("skip = %q, want %q", pass.Skip, SkipEligibleCount)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "asura"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("nothing-eligible", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
if pace := p.runLanePass(context.Background(), "asura", false); pace != sites["asura"].Rest {
|
||||||
|
t.Fatalf("nothing-eligible pace = %s, want a full rest %s", pace, sites["asura"].Rest)
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipNothingEligible {
|
||||||
|
t.Fatalf("skip = %q, want %q", pass.Skip, SkipNothingEligible)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "asura"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("reached the loop", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
|
||||||
|
seedForCheck(t, s, "asura:chronicles-of-the-demon-faction-f886a8af", url, 0)
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
|
||||||
|
if pace := p.runLanePass(context.Background(), "asura", false); pace != defaultGap {
|
||||||
|
t.Fatalf("loop pace = %s, want %s", pace, defaultGap)
|
||||||
|
}
|
||||||
|
pass := latestPassFor(t, s, "asura")
|
||||||
|
if pass.Skip != "" || pass.Due != 1 || pass.Checked != 1 {
|
||||||
|
t.Fatalf("loop pass = %+v, want an empty skip with 1 due and 1 checked", pass)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "asura"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("browser-unreachable mid-loop", func(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
|
||||||
|
interrupted := fmt.Errorf("%w: %w", errBrowserInterrupted, errors.New("restart"))
|
||||||
|
browser := &fakeFetcher{status: 200, err: interrupted}
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = browser
|
||||||
|
if pace := p.runLanePass(context.Background(), "comix", false); pace != defaultGap {
|
||||||
|
t.Fatalf("mid-loop pace = %s, want %s", pace, defaultGap)
|
||||||
|
}
|
||||||
|
// One row was still written, and it is stall-shaped by construction:
|
||||||
|
// empty skip with due > 0 and checked 0 because the return precedes
|
||||||
|
// the checked counter. The assertion below pins that shape precisely.
|
||||||
|
pass := latestPassFor(t, s, "comix")
|
||||||
|
if pass.Skip != "" || pass.Unreachable != 1 || pass.Checked != 0 || pass.Due != 1 {
|
||||||
|
t.Fatalf("mid-loop pass = %+v, want empty skip, unreachable 1, checked 0, due 1", pass)
|
||||||
|
}
|
||||||
|
if got := countPassRows(t, dbURL, "comix"); got != 1 {
|
||||||
|
t.Fatalf("pass rows = %d, want exactly 1", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Carry-forward moves verbatim from the in-memory snapshot (issue #141): a
|
||||||
|
// pass that never computed its own gap carries the previous pass's due, gap,
|
||||||
|
// clamped and checked forward; a pass with a gap of its own records its own
|
||||||
|
// figures, zeroes included, beside the skip reason that explains them.
|
||||||
|
func TestRunLanePassCarryForwardOnlyWhenGapZero(t *testing.T) {
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
|
||||||
|
t.Run("no gap of its own carries the previous pass's figures", func(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "kagane:x", "https://kagane.to/series/x", 0)
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = &fakeFetcher{body: kaganeAPIFixture, status: 200}
|
||||||
|
p.runLanePass(context.Background(), "kagane", false)
|
||||||
|
first := latestPassFor(t, s, "kagane")
|
||||||
|
if first.Due != 1 || first.Checked != 1 || first.GapMS == 0 {
|
||||||
|
t.Fatalf("first pass = %+v, want a measured pass", first)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.SetLaneRefusal("kagane", now.Add(10*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
// The pass row is keyed (site, ran_at), so the second pass needs its
|
||||||
|
// own timestamp: a minute later is still inside the refusal.
|
||||||
|
p.Now = func() time.Time { return now.Add(time.Minute) }
|
||||||
|
p.runLanePass(context.Background(), "kagane", false)
|
||||||
|
second := latestPassFor(t, s, "kagane")
|
||||||
|
if second.Skip != SkipRefusing {
|
||||||
|
t.Fatalf("second pass skip = %q, want %q", second.Skip, SkipRefusing)
|
||||||
|
}
|
||||||
|
if second.Due != first.Due || second.Checked != first.Checked ||
|
||||||
|
second.GapMS != first.GapMS || second.Clamped != first.Clamped {
|
||||||
|
t.Fatalf("carried pass = %+v, want the first pass's figures %+v", second, first)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("a gap of its own records its own figures", func(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = &fakeFetcher{body: comixSeriesFixture, status: 200}
|
||||||
|
p.runLanePass(context.Background(), "comix", false)
|
||||||
|
first := latestPassFor(t, s, "comix")
|
||||||
|
if first.Due != 1 || first.GapMS == 0 {
|
||||||
|
t.Fatalf("first pass = %+v, want a measured pass", first)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The no-fetcher exit sets its own gap, so no carry-forward: the due
|
||||||
|
// count it never gathered records as zero beside its skip reason. A
|
||||||
|
// minute later gives the second pass its own (site, ran_at) key.
|
||||||
|
p.BrowserFetch = nil
|
||||||
|
p.Now = func() time.Time { return now.Add(time.Minute) }
|
||||||
|
p.runLanePass(context.Background(), "comix", false)
|
||||||
|
second := latestPassFor(t, s, "comix")
|
||||||
|
if second.Skip != SkipNoFetcher {
|
||||||
|
t.Fatalf("second pass skip = %q, want %q", second.Skip, SkipNoFetcher)
|
||||||
|
}
|
||||||
|
if second.GapMS != defaultGap.Milliseconds() {
|
||||||
|
t.Fatalf("second pass gap = %d, want its own %d (not carried)", second.GapMS, defaultGap.Milliseconds())
|
||||||
|
}
|
||||||
|
if second.Due != 0 || second.Checked != 0 {
|
||||||
|
t.Fatalf("second pass = %+v, want due 0 checked 0 (its own, not the previous pass's)", second)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// The pass row's five outcome counts are the classification the Series read
|
||||||
|
// already makes — never a second taxonomy (issue #141). Success is derived,
|
||||||
|
// never stored: checked minus the four named counts, unreachable excluded
|
||||||
|
// because its exit returns before the checked counter increments.
|
||||||
|
func TestRunLanePassCountsOutcomes(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
const (
|
||||||
|
successKey = "asura:chronicles-of-the-demon-faction-f886a8af"
|
||||||
|
successURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
|
||||||
|
)
|
||||||
|
seeds := map[string]string{
|
||||||
|
"asura:refused": "https://asurascans.com/series/refused",
|
||||||
|
"asura:no-chapter": "https://asurascans.com/comics/no-chapter",
|
||||||
|
"asura:unfetchable": "https://evil.example/x",
|
||||||
|
"asura:transport": "https://asurascans.com/series/transport",
|
||||||
|
successKey: successURL,
|
||||||
|
}
|
||||||
|
for key, url := range seeds {
|
||||||
|
seedForCheck(t, s, key, url, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
f := &fakeFetcher{perURL: map[string]fakeResponse{
|
||||||
|
seeds["asura:refused"]: {status: 403},
|
||||||
|
seeds["asura:no-chapter"]: {body: "<html></html>", status: 200},
|
||||||
|
seeds["asura:transport"]: {err: errors.New("dial tcp: refused")},
|
||||||
|
successURL: {body: asuraSeriesFixture, status: 200},
|
||||||
|
}}
|
||||||
|
newTestPoller(t, s, f, now).runLanePass(context.Background(), "asura", false)
|
||||||
|
|
||||||
|
pass := latestPassFor(t, s, "asura")
|
||||||
|
if pass.Skip != "" || pass.Due != 5 || pass.Checked != 5 {
|
||||||
|
t.Fatalf("pass = %+v, want a full pass over 5 due Series", pass)
|
||||||
|
}
|
||||||
|
if pass.Refused != 1 || pass.Unreachable != 0 || pass.NoChapter != 1 ||
|
||||||
|
pass.Unfetchable != 1 || pass.Errors != 1 {
|
||||||
|
t.Fatalf("outcome counts = refused %d unreachable %d no_chapter %d unfetchable %d errors %d, want 1 0 1 1 1",
|
||||||
|
pass.Refused, pass.Unreachable, pass.NoChapter, pass.Unfetchable, pass.Errors)
|
||||||
|
}
|
||||||
|
if success := pass.Checked - (pass.Refused + pass.NoChapter + pass.Unfetchable + pass.Errors); success != 1 {
|
||||||
|
t.Fatalf("derived success = %d, want 1", success)
|
||||||
|
}
|
||||||
|
// The one genuine read went through: the success Series carries the
|
||||||
|
// fixture's newest chapter, and none of the four failures do.
|
||||||
|
b, ok, err := s.Get(s.OwnerID(), successKey)
|
||||||
|
if err != nil || !ok {
|
||||||
|
t.Fatalf("Get: %v ok=%v", err, ok)
|
||||||
|
}
|
||||||
|
if b.LatestChapterNum == nil || *b.LatestChapterNum != 181 {
|
||||||
|
t.Fatalf("success Series latest = %v, want 181 (the fixture's newest)", b.LatestChapterNum)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"asura:refused", "asura:no-chapter", "asura:unfetchable", "asura:transport"} {
|
||||||
|
b, _, err := s.Get(s.OwnerID(), key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Get %s: %v", key, err)
|
||||||
|
}
|
||||||
|
if b.LatestChapterNum != nil {
|
||||||
|
t.Fatalf("%s latest = %v, want nil (no chapter survived a failed read)", key, *b.LatestChapterNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A refusal is the Site's mood and outlives our process: the durable stamp a
|
||||||
|
// pass writes is honoured by a freshly constructed poller, which must not
|
||||||
|
// re-probe the Site inside its backoff (issue #141).
|
||||||
|
func TestDurableRefusalSurvivesFreshPoller(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
// Four due Series: the Lane refuses twice, stamps those two, and leaves
|
||||||
|
// the remaining two untried and due — the queue the fresh poller must
|
||||||
|
// find intact once the durable backoff lifts.
|
||||||
|
for i := 0; i < 4; i++ {
|
||||||
|
key := fmt.Sprintf("kagane:s%d", i)
|
||||||
|
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
|
||||||
|
p.BrowserFetch = &fakeFetcher{status: 403}
|
||||||
|
p.runLanePass(context.Background(), "kagane", false)
|
||||||
|
if got := p.BrowserFetch.(*fakeFetcher).callCount(); got != 2 {
|
||||||
|
t.Fatalf("fetches on the refusing pass = %d, want 2 (refused twice)", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A restart: a brand-new poller, no in-memory refusal, same store. The
|
||||||
|
// durable stamp gates the pass.
|
||||||
|
fresh := newTestPoller(t, s, &fakeFetcher{status: 200}, now.Add(14*time.Minute))
|
||||||
|
fresh.BrowserFetch = &fakeFetcher{status: 403}
|
||||||
|
fresh.runLanePass(context.Background(), "kagane", false)
|
||||||
|
if got := fresh.BrowserFetch.(*fakeFetcher).callCount(); got != 0 {
|
||||||
|
t.Fatalf("fetches by a fresh poller inside the backoff = %d, want 0", got)
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "kagane"); pass.Skip != SkipRefusing {
|
||||||
|
t.Fatalf("fresh poller's pass skip = %q, want %q", pass.Skip, SkipRefusing)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Past the backoff the fresh poller probes again — the two Series the
|
||||||
|
// original pass never reached, still due with their stamps untouched.
|
||||||
|
fresh.Now = func() time.Time { return now.Add(16 * time.Minute) }
|
||||||
|
fresh.runLanePass(context.Background(), "kagane", false)
|
||||||
|
if got := fresh.BrowserFetch.(*fakeFetcher).callCount(); got != 2 {
|
||||||
|
t.Fatalf("fetches after the backoff = %d, want 2", got)
|
||||||
|
}
|
||||||
|
for i := 2; i < 4; i++ {
|
||||||
|
if got := readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)); got == 0 {
|
||||||
|
t.Fatalf("kagane:s%d still untried after the backoff", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The pause is read ahead of the refusal check: a Lane that is both paused
|
||||||
|
// and inside a refusal backoff records the paused skip value, not the
|
||||||
|
// refusing one. The pause is the owner's order and outranks the Site's mood
|
||||||
|
// (issue #147).
|
||||||
|
func TestPauseGatePrecedesRefusalGate(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
|
||||||
|
if err := s.SetLaneRefusal("asura", now.Add(10*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||||
|
p := newTestPoller(t, s, f, now)
|
||||||
|
if pace := p.runLanePass(context.Background(), "asura", false); pace != 30*time.Minute {
|
||||||
|
t.Fatalf("paused-while-refusing pace = %s, want 30m (the pause's expiry)", pace)
|
||||||
|
}
|
||||||
|
if f.callCount() != 0 {
|
||||||
|
t.Fatalf("fetches while paused and refusing = %d, want 0", f.callCount())
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipPaused {
|
||||||
|
t.Fatalf("skip = %q, want %q (the pause outranks the refusal)", pass.Skip, SkipPaused)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A pause is a fact about the Site, not about the process: a freshly
|
||||||
|
// constructed poller against a store holding a pause row stays paused until
|
||||||
|
// the expiry, then runs the Lane normally. The restart criterion is the
|
||||||
|
// whole point of writing a row instead of commanding a poller (issue #147).
|
||||||
|
func TestDurablePauseSurvivesFreshPoller(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
|
||||||
|
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A restart: a brand-new poller, no in-memory state, same store.
|
||||||
|
fresh := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
|
||||||
|
if pace := fresh.runLanePass(context.Background(), "asura", false); pace != 30*time.Minute {
|
||||||
|
t.Fatalf("fresh poller's paused pace = %s, want 30m", pace)
|
||||||
|
}
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipPaused {
|
||||||
|
t.Fatalf("fresh poller's pass skip = %q, want %q", pass.Skip, SkipPaused)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Past the expiry the same fresh poller runs the Lane normally.
|
||||||
|
fresh.Now = func() time.Time { return now.Add(31 * time.Minute) }
|
||||||
|
fresh.runLanePass(context.Background(), "asura", false)
|
||||||
|
if pass := latestPassFor(t, s, "asura"); pass.Skip != "" {
|
||||||
|
t.Fatalf("pass after the expiry skip = %q, want the loop reached", pass.Skip)
|
||||||
|
}
|
||||||
|
if got := readLatestCheckedAt(t, s, "asura:x"); got == 0 {
|
||||||
|
t.Fatal("the Series was not checked after the pause lifted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResumeLane zeroes the pause and the Lane's next pass finds its full queue
|
||||||
|
// waiting: a pause delays work rather than discarding it, so the due Series
|
||||||
|
// sit unstamped while paused and are all fetched once the pause lifts
|
||||||
|
// (issue #147).
|
||||||
|
func TestResumeLaneRestoresTheQueue(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
seedForCheck(t, s, fmt.Sprintf("asura:s%d", i), "https://asurascans.com/series/x", 0)
|
||||||
|
}
|
||||||
|
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||||
|
p := newTestPoller(t, s, f, now)
|
||||||
|
p.runLanePass(context.Background(), "asura", false)
|
||||||
|
if f.callCount() != 0 {
|
||||||
|
t.Fatalf("fetches while paused = %d, want 0", f.callCount())
|
||||||
|
}
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
if got := readLatestCheckedAt(t, s, fmt.Sprintf("asura:s%d", i)); got != 0 {
|
||||||
|
t.Fatalf("asura:s%d stamp while paused = %d, want 0 (due and unstamped)", i, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.ResumeLane("asura"); err != nil {
|
||||||
|
t.Fatalf("ResumeLane: %v", err)
|
||||||
|
}
|
||||||
|
before := f.callCount()
|
||||||
|
p.runLanePass(context.Background(), "asura", false)
|
||||||
|
if got := f.callCount() - before; got != 3 {
|
||||||
|
t.Fatalf("fetches after resume = %d, want 3 (the full due queue)", got)
|
||||||
|
}
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
if got := readLatestCheckedAt(t, s, fmt.Sprintf("asura:s%d", i)); got == 0 {
|
||||||
|
t.Fatalf("asura:s%d still untried after resume", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RecordLanePass prunes in the same call that inserts, so the retention
|
||||||
|
// cutoff the recorder passes is observable in what survives: a row just inside
|
||||||
|
// 14 days behind the poller's clock is kept, one just outside is pruned
|
||||||
|
// (issue #139, #141).
|
||||||
|
func TestPassRetentionCutoffIsFourteenDays(t *testing.T) {
|
||||||
|
s, dbURL := newTestStore(t)
|
||||||
|
// A real-world clock: the seeded rows sit 14 days back, so they must be
|
||||||
|
// positive timestamps or the seed's own prune (ran_at < 0) removes them.
|
||||||
|
now := time.UnixMilli(1_800_000_000_000)
|
||||||
|
kept := now.Add(-14*24*time.Hour + time.Minute).UnixMilli()
|
||||||
|
pruned := now.Add(-14*24*time.Hour - time.Minute).UnixMilli()
|
||||||
|
for _, ranAt := range []int64{kept, pruned} {
|
||||||
|
if err := s.RecordLanePass(store.LanePass{Site: "asura", RanAt: ranAt}, 0); err != nil {
|
||||||
|
t.Fatalf("seed pass at %d: %v", ranAt, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
|
||||||
|
newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now).
|
||||||
|
runLanePass(context.Background(), "asura", false)
|
||||||
|
|
||||||
|
db, err := sql.Open("pgx", dbURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("open %s: %v", dbURL, err)
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
var n int
|
||||||
|
if err := db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1 AND ran_at = $2`, "asura", pruned).Scan(&n); err != nil {
|
||||||
|
t.Fatalf("count pruned row: %v", err)
|
||||||
|
}
|
||||||
|
if n != 0 {
|
||||||
|
t.Fatalf("row at %d survived, want it pruned (older than 14 days)", pruned)
|
||||||
|
}
|
||||||
|
if err := db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1`, "asura").Scan(&n); err != nil {
|
||||||
|
t.Fatalf("count passes: %v", err)
|
||||||
|
}
|
||||||
|
if n != 2 {
|
||||||
|
t.Fatalf("passes = %d, want 2 (this pass plus the kept row)", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// stampRecordingFetcher is a fakeFetcher that records the Series' check stamp
|
||||||
|
// at call time — the seam that proves the check stamp is written before the
|
||||||
|
// fetch (issue #146). If the order were swapped, the recorded stamp would be
|
||||||
|
// the pre-pass value and the ordering assertion would fail.
|
||||||
|
type stampRecordingFetcher struct {
|
||||||
|
fakeFetcher
|
||||||
|
store *store.Store
|
||||||
|
site string
|
||||||
|
seriesID string
|
||||||
|
stampAtCall int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *stampRecordingFetcher) Get(ctx context.Context, url string) (string, int, error) {
|
||||||
|
ts, err := f.store.LatestCheckedAt(f.site, f.seriesID)
|
||||||
|
if err != nil {
|
||||||
|
return "", 0, err
|
||||||
|
}
|
||||||
|
f.stampAtCall = ts
|
||||||
|
return f.fakeFetcher.Get(ctx, url)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The check stamp is written before the fetch is attempted: the fake fetcher
|
||||||
|
// records the stamp it sees at call time, and it must already be the pass's
|
||||||
|
// own stamp. The order is load-bearing — a forced request is pending while
|
||||||
|
// force_poll_at > latest_checked_at, so stamping after the fetch would make a
|
||||||
|
// failed forced request sticky — and the assertion fails if it is swapped.
|
||||||
|
func TestCheckStampIsWrittenBeforeFetch(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
const (
|
||||||
|
key = "asura:chronicles-of-the-demon-faction-f886a8af"
|
||||||
|
seriesURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
|
||||||
|
)
|
||||||
|
seedForCheck(t, s, key, seriesURL, 0)
|
||||||
|
now := time.UnixMilli(7_000_000)
|
||||||
|
rec := &stampRecordingFetcher{
|
||||||
|
fakeFetcher: fakeFetcher{body: asuraSeriesFixture, status: 200},
|
||||||
|
store: s, site: "asura", seriesID: "chronicles-of-the-demon-faction-f886a8af",
|
||||||
|
}
|
||||||
|
newTestPoller(t, s, rec, now).runOnce(context.Background())
|
||||||
|
if rec.stampAtCall != now.UnixMilli() {
|
||||||
|
t.Fatalf("check stamp at fetch time = %d, want %d (the stamp must be written before the fetch)", rec.stampAtCall, now.UnixMilli())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A forced Series whose attempt fails is no longer pending: the check stamp
|
||||||
|
// is written before the fetch, so the first attempt ends the pending state
|
||||||
|
// whatever it returns. A naive implementation (stamp only on success, or
|
||||||
|
// after the fetch) leaves the request sticky and the row due next pass.
|
||||||
|
func TestForcedSeriesSelfClearsOnFailedAttempt(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(7_000_000)
|
||||||
|
const url = "https://asurascans.com/comics/x"
|
||||||
|
// Freshly checked, so only the force flag makes it due.
|
||||||
|
seedForCheck(t, s, "asura:x", url, now.Add(-30*time.Minute).UnixMilli())
|
||||||
|
if err := s.ForceSeriesPoll("asura", "x", now.UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck("asura", now.Add(-time.Hour).UnixMilli(), -1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 1 || due[0].Key() != "asura:x" || !due[0].Forced {
|
||||||
|
t.Fatalf("forced row not due and flagged before the attempt: %+v", due)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The attempt fails, but the attempt still happened: the row is stamped
|
||||||
|
// and no longer pending.
|
||||||
|
f := &fakeFetcher{err: errors.New("dial tcp: refused")}
|
||||||
|
newTestPoller(t, s, f, now).runOnce(context.Background())
|
||||||
|
if got := readLatestCheckedAt(t, s, "asura:x"); got != now.UnixMilli() {
|
||||||
|
t.Fatalf("latest_checked_at = %d, want %d", got, now.UnixMilli())
|
||||||
|
}
|
||||||
|
due, err = s.DueForLatestCheck("asura", now.Add(-time.Hour).UnixMilli(), -1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 0 {
|
||||||
|
t.Fatalf("failed attempt left the forced row due: %v", due)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The refusal backoff is a Lane gate, not a Series gate: a forced Series does
|
||||||
|
// not override a Site that is actively refusing, because hand-forcing a
|
||||||
|
// request into a refusal only makes it worse (issue #146).
|
||||||
|
func TestForcedSeriesDoesNotOverrideRefusalBackoff(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
seedForCheck(t, s, "kagane:w", "https://kagane.to/series/w", 0)
|
||||||
|
if err := s.ForceSeriesPoll("kagane", "w", now.UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.SetLaneRefusal("kagane", now.Add(RefuseBackoff).UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
|
||||||
|
p := &Poller{
|
||||||
|
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
|
||||||
|
Now: func() time.Time { return now },
|
||||||
|
}
|
||||||
|
p.runOnce(context.Background())
|
||||||
|
if got := browser.callCount(); got != 0 {
|
||||||
|
t.Fatalf("browser fetches through a refusal backoff = %d, want 0", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A forced Series wakes a sleeping browser Lane: the wake thresholds exist to
|
||||||
|
// stop the machine waking itself for one unattended check, and a human asking
|
||||||
|
// is not that (issue #146). Below both thresholds the Lane still sleeps when
|
||||||
|
// nothing is forced.
|
||||||
|
func TestForcedSeriesWakesSleepingBrowser(t *testing.T) {
|
||||||
|
s, _ := newTestStore(t)
|
||||||
|
now := time.UnixMilli(5_000_000)
|
||||||
|
// Freshly due: checked two minutes before the rest elapses, so the wait
|
||||||
|
// is far below browserWakeAge and the count is under browserWakeCount.
|
||||||
|
seedForCheck(t, s, "kagane:w1", "https://kagane.to/series/w1", now.Add(-62*time.Minute).UnixMilli())
|
||||||
|
seedForCheck(t, s, "kagane:w2", "https://kagane.to/series/w2", now.Add(-62*time.Minute).UnixMilli())
|
||||||
|
|
||||||
|
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
|
||||||
|
p := &Poller{
|
||||||
|
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
|
||||||
|
Now: func() time.Time { return now },
|
||||||
|
}
|
||||||
|
p.runOnce(context.Background())
|
||||||
|
if got := browser.callCount(); got != 0 {
|
||||||
|
t.Fatalf("browser fetches without a forced series = %d, want 0 (Chrome stays asleep)", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.ForceSeriesPoll("kagane", "w1", now.UnixMilli()); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
p.runOnce(context.Background())
|
||||||
|
if got := browser.callCount(); got != 2 {
|
||||||
|
t.Fatalf("browser fetches with a forced series = %d, want 2 (the lane wakes)", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -401,9 +401,10 @@ const (
|
|||||||
// express (docs/research/cloudflare-bot-scoring-and-poll-cadence.md);
|
// express (docs/research/cloudflare-bot-scoring-and-poll-cadence.md);
|
||||||
// below it the Lane is outrunning its own plan and says so loudly.
|
// below it the Lane is outrunning its own plan and says so loudly.
|
||||||
minGap = time.Second
|
minGap = time.Second
|
||||||
// refuseBackoff is how long a Lane waits after its Site refused twice in
|
// RefuseBackoff is how long a Lane waits after its Site refused twice in
|
||||||
// one run before attempting it again.
|
// one run before attempting it again. Exported so the web layer can derive
|
||||||
refuseBackoff = 15 * time.Minute
|
// browser reachability from the pass log over the same window (issue #145).
|
||||||
|
RefuseBackoff = 15 * time.Minute
|
||||||
// browserWakeCount and browserWakeAge gate a browser Lane's run: five or
|
// browserWakeCount and browserWakeAge gate a browser Lane's run: five or
|
||||||
// more due Series, or any one of them waiting this long, or Chrome stays
|
// more due Series, or any one of them waiting this long, or Chrome stays
|
||||||
// asleep (ADR-0005 on-demand browser).
|
// asleep (ADR-0005 on-demand browser).
|
||||||
@@ -511,6 +512,18 @@ var sites = map[string]site{
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SiteNames returns every registry Site, sorted. The admin Series list's Site
|
||||||
|
// select needs the full registry, not just the Sites that have rows, and
|
||||||
|
// laneNames() is the poller's copy of the same list — both read this.
|
||||||
|
func SiteNames() []string {
|
||||||
|
names := make([]string, 0, len(sites))
|
||||||
|
for name := range sites {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
return names
|
||||||
|
}
|
||||||
|
|
||||||
// browserBackedSites is derived from the registry: the Sites whose pages are
|
// browserBackedSites is derived from the registry: the Sites whose pages are
|
||||||
// read through the browser sidecar. Sorted so callers that range it (the
|
// read through the browser sidecar. Sorted so callers that range it (the
|
||||||
// browser fetcher's dispatch) see a stable order instead of map-iteration
|
// browser fetcher's dispatch) see a stable order instead of map-iteration
|
||||||
|
|||||||
@@ -1,79 +0,0 @@
|
|||||||
package latest
|
|
||||||
|
|
||||||
import "time"
|
|
||||||
|
|
||||||
// LaneState is the administrative page's view of one Poll Lane (issue #102):
|
|
||||||
// what the Lane's last pass saw. Due, Gap and Checked are filled in as the
|
|
||||||
// pass computes them; a pass that returned before reaching a figure (refusal
|
|
||||||
// backoff, sidecar down) carries the previous pass's figures forward rather
|
|
||||||
// than overwriting them with zeroes the page would state as fact.
|
|
||||||
type LaneState struct {
|
|
||||||
Site string
|
|
||||||
Due int
|
|
||||||
LastRun time.Time
|
|
||||||
Gap time.Duration
|
|
||||||
// Checked is how many Series this pass actually read. A Lane with Series
|
|
||||||
// due and nothing checked has stopped working; one with nothing due is
|
|
||||||
// merely quiet, and the page must not draw the two the same (story 13).
|
|
||||||
Checked int
|
|
||||||
Clamped bool
|
|
||||||
Refusing bool
|
|
||||||
Browser bool
|
|
||||||
// Asleep marks a browser Lane whose last pass declined to wake Chrome
|
|
||||||
// because it was under both wake thresholds (ADR-0005). Due without
|
|
||||||
// Checked then means "waiting for the group to gather", not "stopped", and
|
|
||||||
// the page must not draw it as a stall.
|
|
||||||
Asleep bool
|
|
||||||
}
|
|
||||||
|
|
||||||
// Status is the owner's page snapshot of the whole poller (issue #102).
|
|
||||||
type Status struct {
|
|
||||||
Lanes []LaneState
|
|
||||||
BrowserConfigured bool
|
|
||||||
BrowserReachable bool
|
|
||||||
}
|
|
||||||
|
|
||||||
// LaneStatus returns a copy of the poller's Lane state for the owner's page.
|
|
||||||
// Only Sites that have completed a pass appear — a restart therefore renders
|
|
||||||
// "no data yet" instead of confident zeroes — in the same order Run iterates.
|
|
||||||
// Refusing is derived at snapshot time from the refusal backoff, not stored,
|
|
||||||
// so a Lane that cooled down between passes reports false without a new pass.
|
|
||||||
// BrowserReachable mirrors the Lanes' own gate: the sidecar is down only
|
|
||||||
// within the refuseBackoff window since its last loss.
|
|
||||||
func (p *Poller) LaneStatus() Status {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
lanes := make([]LaneState, 0, len(p.laneStates))
|
|
||||||
now := p.Now()
|
|
||||||
for _, name := range laneNames() {
|
|
||||||
st, ok := p.laneStates[name]
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
st.Refusing = now.Before(p.refuseUntil[name])
|
|
||||||
lanes = append(lanes, st)
|
|
||||||
}
|
|
||||||
configured := p.BrowserFetch != nil
|
|
||||||
reachable := configured
|
|
||||||
if reachable && !p.browserDownAt.IsZero() && now.Sub(p.browserDownAt) < refuseBackoff {
|
|
||||||
reachable = false
|
|
||||||
}
|
|
||||||
return Status{Lanes: lanes, BrowserConfigured: configured, BrowserReachable: reachable}
|
|
||||||
}
|
|
||||||
|
|
||||||
// recordLaneState stores one Lane's last pass for LaneStatus. Called deferred
|
|
||||||
// from runLanePass so every return path records, even a pass that refused.
|
|
||||||
// A pass that never reached the pace (Gap zero) keeps the last pass's figures:
|
|
||||||
// the Lane's due count and gap did not become zero because this pass declined
|
|
||||||
// to look, and the row's own marks say why it declined.
|
|
||||||
func (p *Poller) recordLaneState(st LaneState) {
|
|
||||||
p.mu.Lock()
|
|
||||||
defer p.mu.Unlock()
|
|
||||||
if p.laneStates == nil {
|
|
||||||
p.laneStates = make(map[string]LaneState)
|
|
||||||
}
|
|
||||||
if prev, ok := p.laneStates[st.Site]; ok && st.Gap == 0 {
|
|
||||||
st.Due, st.Gap, st.Clamped, st.Checked = prev.Due, prev.Gap, prev.Clamped, prev.Checked
|
|
||||||
}
|
|
||||||
p.laneStates[st.Site] = st
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,276 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Series filter names (issue #140), ordered permanent-then-fixable — the
|
||||||
|
// repairs nothing will ever undo first, the ones a Poll can make right after.
|
||||||
|
// A name is the repair a row needs, not the SQL that finds it; the values are
|
||||||
|
// the wire form the Series list URL carries (#142). "all" is the absent and
|
||||||
|
// unknown case: every Series.
|
||||||
|
const (
|
||||||
|
SeriesFilterAll = "all"
|
||||||
|
SeriesFilterNoURL = "no_series_url"
|
||||||
|
SeriesFilterNoChapter = "never_read_a_chapter"
|
||||||
|
SeriesFilterNoReaders = "no_readers"
|
||||||
|
SeriesFilterNeverChecked = "never_checked"
|
||||||
|
SeriesFilterStale = "stale"
|
||||||
|
SeriesFilterNoCover = "no_cover"
|
||||||
|
SeriesFilterReaderReport = "reader_report"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SeriesFilter is one named hygiene predicate over the whole library. Site
|
||||||
|
// and Kind narrow the row read; Name picks the predicate; Cutoff is the
|
||||||
|
// staleness boundary the "stale" filter compares against, supplied by the
|
||||||
|
// caller's clock — the store has no clock; Page is 1-based.
|
||||||
|
type SeriesFilter struct {
|
||||||
|
Site string // "" = every Site
|
||||||
|
Kind string // "" = both library buckets' series
|
||||||
|
Name string // one of the SeriesFilter* constants; "" = SeriesFilterAll
|
||||||
|
Cutoff int64 // unix ms; "stale" reads it, the store never does
|
||||||
|
Page int // 1-based page of the row read; default 1
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminSeriesColumns is the owner's library-wide Series projection in
|
||||||
|
// scanAdminSeries order. It is the privacy boundary: a Series' row carries
|
||||||
|
// the Reader id that raised its Latest Chapter (latest_raised_by), and that id
|
||||||
|
// must never leave the store package — so the projection does not select it,
|
||||||
|
// and only the anonymous boolean in raisedByReaderAnswer crosses it.
|
||||||
|
const adminSeriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover_address,
|
||||||
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.force_poll_at`
|
||||||
|
|
||||||
|
// raisedByReaderAnswer answers "did a Reader's report set this number" without
|
||||||
|
// naming which Reader. Kept apart from adminSeriesColumns so the column list —
|
||||||
|
// the shape scanAdminSeries is fed — stays free of the Sighting-raiser
|
||||||
|
// identity, and the owner learns which rows to act on and nothing about the
|
||||||
|
// Reader behind them.
|
||||||
|
const raisedByReaderAnswer = `(s.latest_raised_by IS NOT NULL) AS raised_by_reader`
|
||||||
|
|
||||||
|
// seriesPageSize is the row read's page length. The tie-break in the query's
|
||||||
|
// ORDER BY is what makes this a stable page boundary — see SeriesPage.
|
||||||
|
const seriesPageSize = 50
|
||||||
|
|
||||||
|
// AdminSeries is one Series as the owner's library-wide view sees it: a
|
||||||
|
// Series-level fact plus an anonymous Reader count. ReaderCount being zero is
|
||||||
|
// the orphan marker. RaisedByReader is the only trace of the Sighting
|
||||||
|
// mechanism here; the Reader id behind it never reaches this type.
|
||||||
|
type AdminSeries struct {
|
||||||
|
Site string
|
||||||
|
SeriesID string
|
||||||
|
Title string
|
||||||
|
SeriesURL string
|
||||||
|
CoverAddress string // "" = no Cover yet
|
||||||
|
Kind string
|
||||||
|
LatestChapter string
|
||||||
|
LatestChapterNum *float64 // nil until first captured
|
||||||
|
LatestCheckedAt int64
|
||||||
|
// ForcePollAt is the owner's "check now" request stamp (issue #146), zero
|
||||||
|
// meaning never asked. Pending is derived, never stored: a request is
|
||||||
|
// pending while ForcePollAt is newer than LatestCheckedAt.
|
||||||
|
ForcePollAt int64
|
||||||
|
ReaderCount int
|
||||||
|
RaisedByReader bool // a Reader's report set LatestChapterNum
|
||||||
|
}
|
||||||
|
|
||||||
|
// SeriesPage is one page of the owner's filtered Series list plus the count
|
||||||
|
// of every Series matching the same filter — a window number, not the page's
|
||||||
|
// len, so the landing page's figure and the list heading come from one query.
|
||||||
|
type SeriesPage struct {
|
||||||
|
Rows []AdminSeries
|
||||||
|
Total int
|
||||||
|
}
|
||||||
|
|
||||||
|
// SiteSeriesShape is one Site's share of the Series matching a filter: how
|
||||||
|
// many, and the manga/novel split. One grouped pass, then library-wide totals
|
||||||
|
// are summed in Go over the rows — the landing page's per-Site table reads
|
||||||
|
// this and never pays for the rows the list discards.
|
||||||
|
type SiteSeriesShape struct {
|
||||||
|
Site string
|
||||||
|
Total int
|
||||||
|
Manga int
|
||||||
|
Novel int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>").
|
||||||
|
func (a AdminSeries) Key() string { return a.Site + ":" + a.SeriesID }
|
||||||
|
|
||||||
|
// adminFilter maps a filter's named predicate to its compile-time WHERE and
|
||||||
|
// HAVING clauses and their bound parameters — the name never reaches query
|
||||||
|
// text, and Site and Kind bind as parameters. Shared by the row read and the
|
||||||
|
// per-Site aggregate so the two cannot disagree on what a filter means.
|
||||||
|
//
|
||||||
|
// The WHERE set is: no URL (an empty URL only — the host-failing-the-fetch-
|
||||||
|
// gate case is invisible to SQL, needs the Site registry in Go, and belongs to
|
||||||
|
// a later repair), never-read-a-chapter and never-checked as disjoint halves
|
||||||
|
// (non-zero versus zero check stamp), stale, no cover, and Reader-report.
|
||||||
|
// no_readers is the one HAVING predicate: it is the orphan test, an aggregate
|
||||||
|
// over the LEFT JOIN, where a bare WHERE has no row to test.
|
||||||
|
//
|
||||||
|
// stale is the checked-but-old half of the stamp partition — because the
|
||||||
|
// verdict line wants "not checked in twelve hours" as one figure, and a never
|
||||||
|
// checked Series is already counted on its own "waiting"/never-checked
|
||||||
|
// filter, folding it in would double-report it. The landing page computes the
|
||||||
|
// inclusive number as stale + never-checked.
|
||||||
|
func adminFilter(f SeriesFilter) (where, having string, args []any, err error) {
|
||||||
|
var clauses []string
|
||||||
|
if f.Kind != "" {
|
||||||
|
args = append(args, f.Kind)
|
||||||
|
clauses = append(clauses, "s.kind = $"+strconv.Itoa(len(args)))
|
||||||
|
}
|
||||||
|
switch f.Name {
|
||||||
|
case "", SeriesFilterAll:
|
||||||
|
case SeriesFilterNoURL:
|
||||||
|
clauses = append(clauses, `s.series_url = ''`)
|
||||||
|
case SeriesFilterNoChapter:
|
||||||
|
clauses = append(clauses, `s.latest_checked_at <> 0 AND s.latest_chapter_num IS NULL`)
|
||||||
|
case SeriesFilterNeverChecked:
|
||||||
|
clauses = append(clauses, `s.latest_checked_at = 0`)
|
||||||
|
case SeriesFilterStale:
|
||||||
|
clauses = append(clauses, `s.latest_checked_at > 0 AND s.latest_checked_at < $`+strconv.Itoa(len(args)+1))
|
||||||
|
args = append(args, f.Cutoff)
|
||||||
|
case SeriesFilterNoCover:
|
||||||
|
clauses = append(clauses, `s.cover_address = ''`)
|
||||||
|
case SeriesFilterReaderReport:
|
||||||
|
clauses = append(clauses, `s.latest_raised_by IS NOT NULL`)
|
||||||
|
case SeriesFilterNoReaders:
|
||||||
|
having = `HAVING COUNT(b.reader_id) = 0`
|
||||||
|
default:
|
||||||
|
return "", "", nil, fmt.Errorf("unknown series filter %q", f.Name)
|
||||||
|
}
|
||||||
|
if len(clauses) > 0 {
|
||||||
|
where = "WHERE " + strings.Join(clauses, " AND ")
|
||||||
|
}
|
||||||
|
return where, having, args, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SeriesPage returns one page of the Series matching the filter, least
|
||||||
|
// recently checked first. The LEFT JOIN to Bookmarks is what surfaces the
|
||||||
|
// orphans that hygiene has to find — an inner join would hide them, exactly
|
||||||
|
// as the Lane's join does. ReaderCount is a plain count of every Bookmark on
|
||||||
|
// the Series, which knowingly disagrees with the two Lane queries for as long
|
||||||
|
// as the finished lifecycle bucket exists (#140).
|
||||||
|
//
|
||||||
|
// The tie-break is mandatory, not decorative: every unpollable Series shares a
|
||||||
|
// zero check stamp, so ordering on that column alone gives no stable page
|
||||||
|
// boundary and rows would repeat or vanish across pages. (site, series_id) is
|
||||||
|
// the primary key, hence total. The filtered total is a window count in the
|
||||||
|
// same query — window functions run after grouping and before the limit, so
|
||||||
|
// one where-clause cannot disagree with a second copy of itself.
|
||||||
|
func (s *Store) SeriesPage(f SeriesFilter) (SeriesPage, error) {
|
||||||
|
where, having, args, err := adminFilter(f)
|
||||||
|
if err != nil {
|
||||||
|
return SeriesPage{}, err
|
||||||
|
}
|
||||||
|
if f.Page < 1 {
|
||||||
|
f.Page = 1
|
||||||
|
}
|
||||||
|
// Site narrowing is the row read's own; the aggregate must see every Site.
|
||||||
|
if f.Site != "" {
|
||||||
|
args = append(args, f.Site)
|
||||||
|
clause := "s.site = $" + strconv.Itoa(len(args))
|
||||||
|
if where == "" {
|
||||||
|
where = "WHERE " + clause
|
||||||
|
} else {
|
||||||
|
where += " AND " + clause
|
||||||
|
}
|
||||||
|
}
|
||||||
|
base := len(args)
|
||||||
|
args = append(args, seriesPageSize, seriesPageSize*(f.Page-1))
|
||||||
|
rows, err := s.db.Query(`
|
||||||
|
SELECT `+adminSeriesColumns+`, `+raisedByReaderAnswer+`,
|
||||||
|
COUNT(b.reader_id) AS reader_count,
|
||||||
|
COUNT(*) OVER () AS filtered_total
|
||||||
|
FROM series s
|
||||||
|
LEFT JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||||
|
`+where+`
|
||||||
|
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover_address,
|
||||||
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
|
||||||
|
s.force_poll_at, s.latest_raised_by
|
||||||
|
`+having+`
|
||||||
|
ORDER BY s.latest_checked_at, s.site, s.series_id
|
||||||
|
LIMIT $`+strconv.Itoa(base+1)+` OFFSET $`+strconv.Itoa(base+2), args...)
|
||||||
|
if err != nil {
|
||||||
|
return SeriesPage{}, fmt.Errorf("query series page: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := SeriesPage{}
|
||||||
|
for rows.Next() {
|
||||||
|
a, total, err := scanAdminSeries(rows.Scan)
|
||||||
|
if err != nil {
|
||||||
|
return SeriesPage{}, fmt.Errorf("scan series page: %w", err)
|
||||||
|
}
|
||||||
|
out.Rows = append(out.Rows, a)
|
||||||
|
out.Total = total
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// SeriesShapes returns each Site's share of the Series matching the filter,
|
||||||
|
// one grouped pass. Site and Page are row-read concerns and are ignored; the
|
||||||
|
// Landing page reads this per Site and sums the totals in Go for the
|
||||||
|
// library-wide figure.
|
||||||
|
func (s *Store) SeriesShapes(f SeriesFilter) ([]SiteSeriesShape, error) {
|
||||||
|
where, having, args, err := adminFilter(f)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
rows, err := s.db.Query(`
|
||||||
|
SELECT site,
|
||||||
|
COUNT(*) AS total,
|
||||||
|
COUNT(*) FILTER (WHERE kind = 'manga') AS manga,
|
||||||
|
COUNT(*) FILTER (WHERE kind = 'novel') AS novel
|
||||||
|
FROM (
|
||||||
|
SELECT s.site, s.kind
|
||||||
|
FROM series s
|
||||||
|
LEFT JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||||
|
`+where+`
|
||||||
|
GROUP BY s.site, s.series_id, s.kind
|
||||||
|
`+having+`
|
||||||
|
) shape
|
||||||
|
GROUP BY site
|
||||||
|
ORDER BY site`, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("query series shapes: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := []SiteSeriesShape{}
|
||||||
|
for rows.Next() {
|
||||||
|
var sh SiteSeriesShape
|
||||||
|
if err := rows.Scan(&sh.Site, &sh.Total, &sh.Manga, &sh.Novel); err != nil {
|
||||||
|
return nil, fmt.Errorf("scan series shape: %w", err)
|
||||||
|
}
|
||||||
|
out = append(out, sh)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanAdminSeries reads one row in adminSeriesColumns + raisedByReaderAnswer
|
||||||
|
// order, plus the query's reader_count and filtered_total columns, and returns
|
||||||
|
// the window total alongside the row. latest_chapter_num is NULL until first
|
||||||
|
// captured — the "never read a chapter" state. The Sighting-raiser column is
|
||||||
|
// never among the scanned columns.
|
||||||
|
func scanAdminSeries(scan func(...any) error) (AdminSeries, int, error) {
|
||||||
|
var (
|
||||||
|
a AdminSeries
|
||||||
|
latestChapterNum sql.NullFloat64
|
||||||
|
total int
|
||||||
|
)
|
||||||
|
if err := scan(
|
||||||
|
&a.Site, &a.SeriesID, &a.Title, &a.SeriesURL, &a.CoverAddress,
|
||||||
|
&a.Kind, &a.LatestChapter, &latestChapterNum, &a.LatestCheckedAt,
|
||||||
|
&a.ForcePollAt,
|
||||||
|
&a.RaisedByReader, &a.ReaderCount, &total,
|
||||||
|
); err != nil {
|
||||||
|
return AdminSeries{}, 0, err
|
||||||
|
}
|
||||||
|
if latestChapterNum.Valid {
|
||||||
|
a.LatestChapterNum = &latestChapterNum.Float64
|
||||||
|
}
|
||||||
|
return a, total, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,396 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// seriesSeed describes one Series (and optionally its bookmarks) to stand up
|
||||||
|
// for an admin filter test. Direct SQL, because the filters separate rows the
|
||||||
|
// Upsert path could not produce together: an orphan has no bookmark, and a
|
||||||
|
// Reader-raised Latest Chapter needs a Sighting the store does not create.
|
||||||
|
type seriesSeed struct {
|
||||||
|
key string
|
||||||
|
kind string
|
||||||
|
url string
|
||||||
|
cover string // cover_address
|
||||||
|
checkedAt int64
|
||||||
|
latestNum *float64
|
||||||
|
bookmarks int // readers that hold it; 0 = orphan
|
||||||
|
raisedBy bool // a Reader's report is attributed as the raiser
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedAdminSeries inserts one series row and its bookmarks (owner first, then
|
||||||
|
// fresh readers), with the exact admin-relevant facts a test needs.
|
||||||
|
func seedAdminSeries(t *testing.T, s *Store, seed seriesSeed) {
|
||||||
|
t.Helper()
|
||||||
|
site, seriesID, ok := strings.Cut(seed.key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", seed.key)
|
||||||
|
}
|
||||||
|
if seed.kind == "" {
|
||||||
|
seed.kind = "manga"
|
||||||
|
}
|
||||||
|
var latestChapter any = ""
|
||||||
|
if seed.latestNum != nil {
|
||||||
|
latestChapter = "Chapter " + strconv.FormatFloat(*seed.latestNum, 'f', -1, 64)
|
||||||
|
}
|
||||||
|
if _, err := s.db.Exec(`
|
||||||
|
INSERT INTO series (site, series_id, title, kind, series_url, cover_address,
|
||||||
|
latest_checked_at, latest_chapter, latest_chapter_num)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
|
||||||
|
site, seriesID, "Title of "+seed.key, seed.kind, seed.url, seed.cover,
|
||||||
|
seed.checkedAt, latestChapter, seed.latestNum); err != nil {
|
||||||
|
t.Fatalf("seed series %q: %v", seed.key, err)
|
||||||
|
}
|
||||||
|
for i := range seed.bookmarks {
|
||||||
|
var readerID int64 = s.OwnerID()
|
||||||
|
if i > 0 {
|
||||||
|
readerID = secondReader(t, s)
|
||||||
|
}
|
||||||
|
if _, err := s.db.Exec(`
|
||||||
|
INSERT INTO bookmarks (reader_id, site, series_id,
|
||||||
|
last_chapter, last_chapter_num, last_chapter_url,
|
||||||
|
favorite, status, updated_at)
|
||||||
|
VALUES ($1, $2, $3, '', 0, '', false, 'reading', $4)`,
|
||||||
|
readerID, site, seriesID, seed.checkedAt); err != nil {
|
||||||
|
t.Fatalf("seed bookmark %q: %v", seed.key, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if seed.raisedBy {
|
||||||
|
if _, err := s.db.Exec(
|
||||||
|
`UPDATE series SET latest_raised_by = $1 WHERE site = $2 AND series_id = $3`,
|
||||||
|
s.OwnerID(), site, seriesID); err != nil {
|
||||||
|
t.Fatalf("seed raised-by %q: %v", seed.key, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func pageKeys(t *testing.T, s *Store, f SeriesFilter) map[string]bool {
|
||||||
|
t.Helper()
|
||||||
|
page, err := s.SeriesPage(f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage(%+v): %v", f, err)
|
||||||
|
}
|
||||||
|
keys := map[string]bool{}
|
||||||
|
for _, a := range page.Rows {
|
||||||
|
keys[a.Key()] = true
|
||||||
|
}
|
||||||
|
return keys
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each filter must return the rows it names and no others, over one shared
|
||||||
|
// seeded mix where every healthy neighbour is present to be wrongly returned.
|
||||||
|
// The stale cutoff is 5000: a Series checked at 9000 is current, at 2000 stale.
|
||||||
|
func TestAdminSeriesFilters(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:healthy", url: "https://asurascans.com/comics/healthy", cover: "aaa", checkedAt: 9000, latestNum: new(10.0), bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:nourl", url: "", cover: "bbb", checkedAt: 9000, latestNum: new(5.0), bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:nochapter", url: "https://asurascans.com/comics/nochapter", cover: "ccc", checkedAt: 9000, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:neverchecked", url: "https://asurascans.com/comics/neverchecked", cover: "ddd", checkedAt: 0, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:orphan", url: "https://asurascans.com/comics/orphan", cover: "eee", checkedAt: 9000, latestNum: new(7.0), bookmarks: 0})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:stale", url: "https://asurascans.com/comics/stale", cover: "fff", checkedAt: 2000, latestNum: new(4.0), bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:nocover", url: "https://asurascans.com/comics/nocover", checkedAt: 9000, latestNum: new(9.0), bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:report", url: "https://asurascans.com/comics/report", cover: "ggg", checkedAt: 9000, latestNum: new(8.0), bookmarks: 1, raisedBy: true})
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
f SeriesFilter
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{"all", SeriesFilter{}, []string{"asura:healthy", "asura:nourl", "asura:nochapter", "asura:neverchecked", "asura:orphan", "asura:stale", "asura:nocover", "asura:report"}},
|
||||||
|
{"no series url", SeriesFilter{Name: SeriesFilterNoURL}, []string{"asura:nourl"}},
|
||||||
|
{"never read a chapter", SeriesFilter{Name: SeriesFilterNoChapter}, []string{"asura:nochapter"}},
|
||||||
|
{"never checked", SeriesFilter{Name: SeriesFilterNeverChecked}, []string{"asura:neverchecked"}},
|
||||||
|
{"no readers", SeriesFilter{Name: SeriesFilterNoReaders}, []string{"asura:orphan"}},
|
||||||
|
{"stale", SeriesFilter{Name: SeriesFilterStale, Cutoff: 5000}, []string{"asura:stale"}},
|
||||||
|
{"no cover", SeriesFilter{Name: SeriesFilterNoCover}, []string{"asura:nocover"}},
|
||||||
|
{"reader report", SeriesFilter{Name: SeriesFilterReaderReport}, []string{"asura:report"}},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
got := pageKeys(t, s, tc.f)
|
||||||
|
want := map[string]bool{}
|
||||||
|
for _, k := range tc.want {
|
||||||
|
want[k] = true
|
||||||
|
}
|
||||||
|
if len(got) != len(want) {
|
||||||
|
t.Fatalf("%+v returned %v, want exactly %v", tc.f, got, want)
|
||||||
|
}
|
||||||
|
for k := range want {
|
||||||
|
if !got[k] {
|
||||||
|
t.Fatalf("%+v dropped %q (got %v)", tc.f, k, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// "Never read a chapter" and "never checked" are disjoint by construction:
|
||||||
|
// the first requires a non-zero check stamp, the second a zero one. Over a
|
||||||
|
// mix that should satisfy both, no row may be counted twice.
|
||||||
|
func TestAdminNeverChapterAndNeverCheckedAreDisjoint(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:nochapter", url: "u", checkedAt: 9000, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:neverchecked", url: "u", checkedAt: 0, bookmarks: 1})
|
||||||
|
// A zero-stamp, no-chapter row is never-checked only: if never-read-a-
|
||||||
|
// chapter ever lost its non-zero-stamp guard, it would claim this row too
|
||||||
|
// and the two counts would double-report it.
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:both", url: "u", checkedAt: 0, bookmarks: 1})
|
||||||
|
noChapter := pageKeys(t, s, SeriesFilter{Name: SeriesFilterNoChapter})
|
||||||
|
neverChecked := pageKeys(t, s, SeriesFilter{Name: SeriesFilterNeverChecked})
|
||||||
|
for k := range noChapter {
|
||||||
|
if neverChecked[k] {
|
||||||
|
t.Fatalf("row %q matches both never-read-a-chapter and never-checked", k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !noChapter["asura:nochapter"] || !neverChecked["asura:neverchecked"] {
|
||||||
|
t.Fatalf("disjoint split lost its own rows: no-chapter=%v never-checked=%v", noChapter, neverChecked)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Several rows share a zero check stamp, so ordering on latest_checked_at
|
||||||
|
// alone gives no stable page boundary. The (site, series_id) tie-break must
|
||||||
|
// make page 2 a strict continuation of page 1: no repeat, no vanishing row.
|
||||||
|
func TestAdminSeriesPageTieBreakIsStable(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const total = 53 // > one page, < two (page size 50)
|
||||||
|
for i := range total {
|
||||||
|
id := "tie" + strconv.Itoa(i)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:" + id, url: "u", checkedAt: 0, bookmarks: 1})
|
||||||
|
}
|
||||||
|
// A second Site's zero-stamp row is part of the same all-filter list, and
|
||||||
|
// must land on a valid page boundary rather than duplicating or dropping
|
||||||
|
// one of asura's rows: the tie-break is global (site, series_id).
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "demonic:z", url: "u", checkedAt: 0, bookmarks: 1})
|
||||||
|
wantTotal := total + 1
|
||||||
|
|
||||||
|
p1, err := s.SeriesPage(SeriesFilter{Name: SeriesFilterAll})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage page 1: %v", err)
|
||||||
|
}
|
||||||
|
p2, err := s.SeriesPage(SeriesFilter{Name: SeriesFilterAll, Page: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage page 2: %v", err)
|
||||||
|
}
|
||||||
|
if len(p1.Rows) != seriesPageSize {
|
||||||
|
t.Fatalf("page 1 has %d rows, want %d", len(p1.Rows), seriesPageSize)
|
||||||
|
}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, a := range append(append([]AdminSeries{}, p1.Rows...), p2.Rows...) {
|
||||||
|
if seen[a.Key()] {
|
||||||
|
t.Fatalf("row %q repeats across pages", a.Key())
|
||||||
|
}
|
||||||
|
seen[a.Key()] = true
|
||||||
|
}
|
||||||
|
if len(seen) != wantTotal {
|
||||||
|
t.Fatalf("%d distinct rows across pages, want %d (a row vanished)", len(seen), wantTotal)
|
||||||
|
}
|
||||||
|
if p1.Total != wantTotal {
|
||||||
|
t.Fatalf("page total = %d, want %d (the window count must span pages)", p1.Total, wantTotal)
|
||||||
|
}
|
||||||
|
// A page beyond the end is empty, not an error (the list re-reads page 1).
|
||||||
|
// The window count runs over the rows present in the result set, so an
|
||||||
|
// overflow page has no rows and therefore no total — the caller must not
|
||||||
|
// render it, which is exactly why the list re-reads page 1.
|
||||||
|
pFinal, err := s.SeriesPage(SeriesFilter{Name: SeriesFilterAll, Page: 99})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage beyond end: %v", err)
|
||||||
|
}
|
||||||
|
if len(pFinal.Rows) != 0 {
|
||||||
|
t.Fatalf("page beyond end = %d rows, want 0", len(pFinal.Rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The filtered total is the window number over the same filter the rows use,
|
||||||
|
// and the per-Site aggregate sums to the same figure — so the landing page's
|
||||||
|
// count and the list's heading can never disagree, whoever computes them.
|
||||||
|
func TestAdminTotalAgreesWithRowsAndShapes(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:a", url: "u", cover: "a", checkedAt: 9000, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:b", url: "u", checkedAt: 9000, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:c", url: "u", checkedAt: 9000, bookmarks: 1, latestNum: new(2.0), raisedBy: true})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "demonic:d", url: "u", kind: "novel", checkedAt: 9000, bookmarks: 1})
|
||||||
|
|
||||||
|
filters := []SeriesFilter{
|
||||||
|
{},
|
||||||
|
{Name: SeriesFilterNoCover},
|
||||||
|
{Name: SeriesFilterReaderReport},
|
||||||
|
{Name: SeriesFilterNoChapter},
|
||||||
|
}
|
||||||
|
for _, f := range filters {
|
||||||
|
page, err := s.SeriesPage(f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage(%+v): %v", f, err)
|
||||||
|
}
|
||||||
|
want := len(page.Rows)
|
||||||
|
if f.Page == 0 && want == seriesPageSize {
|
||||||
|
t.Fatalf("seed produced a full page; bump the seed or drop page size in the test")
|
||||||
|
}
|
||||||
|
if page.Total != want {
|
||||||
|
t.Fatalf("%+v total = %d, want %d (window count disagrees with row count)", f, page.Total, want)
|
||||||
|
}
|
||||||
|
shapes, err := s.SeriesShapes(f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesShapes(%+v): %v", f, err)
|
||||||
|
}
|
||||||
|
sum := 0
|
||||||
|
for _, sh := range shapes {
|
||||||
|
sum += sh.Total
|
||||||
|
}
|
||||||
|
if sum != want {
|
||||||
|
t.Fatalf("%+v aggregate sum = %d, want %d (aggregate disagrees with row query)", f, sum, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The default filter's aggregate carries the library shape: per-Site
|
||||||
|
// totals and the manga/novel split, summed in Go for library wide.
|
||||||
|
shapes, err := s.SeriesShapes(SeriesFilter{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesShapes default: %v", err)
|
||||||
|
}
|
||||||
|
if len(shapes) != 2 || shapes[0].Site != "asura" || shapes[1].Site != "demonic" {
|
||||||
|
t.Fatalf("shapes = %+v, want asura then demonic", shapes)
|
||||||
|
}
|
||||||
|
if shapes[0].Total != 3 || shapes[0].Manga != 3 || shapes[0].Novel != 0 {
|
||||||
|
t.Fatalf("asura shape = %+v, want 3 manga, 0 novel", shapes[0])
|
||||||
|
}
|
||||||
|
if shapes[1].Total != 1 || shapes[1].Manga != 0 || shapes[1].Novel != 1 {
|
||||||
|
t.Fatalf("demonic shape = %+v, want 1 novel", shapes[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Site and Library narrowing stack on a named filter without changing what
|
||||||
|
// the filter means.
|
||||||
|
func TestAdminFilterSiteAndKindNarrow(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:aa", url: "u", checkedAt: 9000, bookmarks: 1, latestNum: new(1.0)})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:ab", url: "", checkedAt: 9000, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "demonic:aa", url: "u", kind: "novel", checkedAt: 9000, bookmarks: 1})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "demonic:ab", url: "", kind: "novel", checkedAt: 9000, bookmarks: 1})
|
||||||
|
|
||||||
|
got := pageKeys(t, s, SeriesFilter{Name: SeriesFilterNoURL, Site: "asura"})
|
||||||
|
if len(got) != 1 || !got["asura:ab"] {
|
||||||
|
t.Fatalf("site+nourl = %v, want only asura:ab", got)
|
||||||
|
}
|
||||||
|
got = pageKeys(t, s, SeriesFilter{Name: SeriesFilterNoURL, Kind: "novel"})
|
||||||
|
if len(got) != 1 || !got["demonic:ab"] {
|
||||||
|
t.Fatalf("kind+nourl = %v, want only demonic:ab", got)
|
||||||
|
}
|
||||||
|
got = pageKeys(t, s, SeriesFilter{Name: SeriesFilterAll, Site: "demonic", Kind: "novel"})
|
||||||
|
if len(got) != 2 || !got["demonic:aa"] || !got["demonic:ab"] {
|
||||||
|
t.Fatalf("site+kind+all = %v, want both demonic rows", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The aggregate ignores the Site narrowing (it is per-Site by shape), but
|
||||||
|
// honours the Library narrowing: asura's missing-URL row is manga, so the
|
||||||
|
// novel no-URL list is demonic alone.
|
||||||
|
shapes, err := s.SeriesShapes(SeriesFilter{Name: SeriesFilterNoURL, Kind: "novel"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesShapes: %v", err)
|
||||||
|
}
|
||||||
|
if len(shapes) != 1 || shapes[0].Site != "demonic" ||
|
||||||
|
shapes[0].Total != 1 || shapes[0].Novel != 1 {
|
||||||
|
t.Fatalf("novel no-URL aggregate = %+v, want demonic {Total:1 Novel:1}", shapes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The projection is the privacy boundary: a Series whose Latest Chapter was
|
||||||
|
// raised by a Reader's report reads back with the anonymous boolean set, not
|
||||||
|
// with the Reader's id, and no Reader id travels in any returned row.
|
||||||
|
func TestAdminSeriesReportsAnonymously(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:raised", url: "u", checkedAt: 9000, latestNum: new(9.0), bookmarks: 1, raisedBy: true})
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:polled", url: "u", checkedAt: 9000, latestNum: new(8.0), bookmarks: 1})
|
||||||
|
|
||||||
|
page, err := s.SeriesPage(SeriesFilter{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage: %v", err)
|
||||||
|
}
|
||||||
|
byKey := map[string]AdminSeries{}
|
||||||
|
for _, a := range page.Rows {
|
||||||
|
byKey[a.Key()] = a
|
||||||
|
}
|
||||||
|
if !byKey["asura:raised"].RaisedByReader {
|
||||||
|
t.Fatal("Reader-raised Series read back RaisedByReader=false")
|
||||||
|
}
|
||||||
|
if byKey["asura:polled"].RaisedByReader {
|
||||||
|
t.Fatal("Poll-raised Series read back RaisedByReader=true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The privacy test that cannot rot into a template-only guarantee: assert the
|
||||||
|
// admin column constant does not carry the Sighting-raiser column and that the
|
||||||
|
// admin row type has no field for it, modelled on the guard on the Bookmark
|
||||||
|
// column list.
|
||||||
|
func TestAdminProjectionHidesSightingRaiser(t *testing.T) {
|
||||||
|
if strings.Contains(adminSeriesColumns, "latest_raised_by") {
|
||||||
|
t.Fatal("admin column list carries latest_raised_by: the Sighting-raiser id would reach the owner")
|
||||||
|
}
|
||||||
|
if _, ok := reflect.TypeOf(AdminSeries{}).FieldByName("LatestRaisedBy"); ok {
|
||||||
|
t.Fatal("AdminSeries carries a field for the Sighting-raiser id")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An unknown filter name is rejected rather than silently meaning "all" —
|
||||||
|
// otherwise a mistyped URL would present an empty page as the whole library.
|
||||||
|
func TestAdminFilterUnknownNameRejected(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:a", url: "u", checkedAt: 9000, bookmarks: 1})
|
||||||
|
for name, call := range map[string]func() error{
|
||||||
|
"page": func() error { _, err := s.SeriesPage(SeriesFilter{Name: "bogus"}); return err },
|
||||||
|
"shape": func() error { _, err := s.SeriesShapes(SeriesFilter{Name: "bogus"}); return err },
|
||||||
|
} {
|
||||||
|
if err := call(); err == nil || !strings.Contains(err.Error(), "unknown series filter") {
|
||||||
|
t.Fatalf("%s with bogus filter = %v, want unknown-filter error", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ForceSeriesPoll is the idempotent stamp write: a second press overwrites
|
||||||
|
// the request time, and touching a missing series is not an error.
|
||||||
|
func TestForceSeriesPollStampsIdempotently(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:x", url: "u", checkedAt: 9000, bookmarks: 1})
|
||||||
|
|
||||||
|
if err := s.ForceSeriesPoll("asura", "x", 42); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.ForceSeriesPoll("asura", "x", 99); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll re-stamp: %v", err)
|
||||||
|
}
|
||||||
|
// Touching a missing series is not an error: the row may have been
|
||||||
|
// orphaned, and the caller's read decides what exists.
|
||||||
|
if err := s.ForceSeriesPoll("asura", "ghost", 99); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll missing: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var got int64
|
||||||
|
if err := s.db.QueryRow(
|
||||||
|
`SELECT force_poll_at FROM series WHERE site = 'asura' AND series_id = 'x'`).Scan(&got); err != nil {
|
||||||
|
t.Fatalf("read force_poll_at: %v", err)
|
||||||
|
}
|
||||||
|
if got != 99 {
|
||||||
|
t.Fatalf("force_poll_at = %d, want 99 (the later press wins)", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The admin projection carries the force stamp so the web layer can derive
|
||||||
|
// the pending flag without a second read.
|
||||||
|
func TestAdminSeriesCarriesForcePollAt(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedAdminSeries(t, s, seriesSeed{key: "asura:x", url: "u", checkedAt: 1000, bookmarks: 1})
|
||||||
|
if err := s.ForceSeriesPoll("asura", "x", 5000); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
page, err := s.SeriesPage(SeriesFilter{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SeriesPage: %v", err)
|
||||||
|
}
|
||||||
|
if len(page.Rows) != 1 || page.Rows[0].ForcePollAt != 5000 {
|
||||||
|
t.Fatalf("row = %+v, want ForcePollAt 5000", page.Rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
-- One durable state row per Poll Lane. Zero means no pause or refusal is set.
|
||||||
|
CREATE TABLE poll_lanes (
|
||||||
|
site text NOT NULL PRIMARY KEY,
|
||||||
|
paused_until bigint NOT NULL DEFAULT 0,
|
||||||
|
refuse_until bigint NOT NULL DEFAULT 0
|
||||||
|
);
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
-- Append-only Lane Pass log. Timestamps are unix milliseconds from the poller's clock.
|
||||||
|
CREATE TABLE poll_passes (
|
||||||
|
site text NOT NULL,
|
||||||
|
ran_at bigint NOT NULL,
|
||||||
|
skip text NOT NULL,
|
||||||
|
due integer NOT NULL,
|
||||||
|
checked integer NOT NULL,
|
||||||
|
gap_ms bigint NOT NULL,
|
||||||
|
clamped boolean NOT NULL,
|
||||||
|
refused integer NOT NULL,
|
||||||
|
unreachable integer NOT NULL,
|
||||||
|
no_chapter integer NOT NULL,
|
||||||
|
unfetchable integer NOT NULL,
|
||||||
|
errors integer NOT NULL,
|
||||||
|
PRIMARY KEY (site, ran_at)
|
||||||
|
);
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
-- Admin read-model foundation (#140). The Series list's default order is
|
||||||
|
-- least-recently-checked first, so the table — which has only its primary key
|
||||||
|
-- today — gets an index that can serve it. A grouped query over a join may
|
||||||
|
-- ignore the index, so this is a judgement, not a measurement: re-time on real
|
||||||
|
-- data before adding a second.
|
||||||
|
CREATE INDEX series_latest_checked_at_idx ON series (latest_checked_at);
|
||||||
|
|
||||||
|
-- force_poll_at is the "ask for one Series to be checked now" stamp (#146).
|
||||||
|
-- Zero means never forced; nothing reads the column before that ticket wires
|
||||||
|
-- it, so it lands here unused.
|
||||||
|
ALTER TABLE series ADD COLUMN force_poll_at bigint NOT NULL DEFAULT 0;
|
||||||
+273
-12
@@ -88,6 +88,39 @@ type Series struct {
|
|||||||
// readerCount is the number of bookmarks referencing this series, filled
|
// readerCount is the number of bookmarks referencing this series, filled
|
||||||
// only by the due-queue query that orders on it.
|
// only by the due-queue query that orders on it.
|
||||||
readerCount int
|
readerCount int
|
||||||
|
// Forced is whether the owner asked for a check now (issue #146): the
|
||||||
|
// request stamp is newer than the check stamp. Derived in the due query,
|
||||||
|
// never stored, and the flag that jumps the queue and opens the browser
|
||||||
|
// wake gate.
|
||||||
|
Forced bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// LanePass is one Poll Lane's durable pass snapshot. Pause and refusal stamps
|
||||||
|
// are joined from poll_lanes on read; they are not pass facts.
|
||||||
|
type LanePass struct {
|
||||||
|
Site string
|
||||||
|
RanAt int64
|
||||||
|
Skip string
|
||||||
|
Due, Checked int
|
||||||
|
GapMS int64
|
||||||
|
Clamped bool
|
||||||
|
Refused, Unreachable, NoChapter int
|
||||||
|
Unfetchable, Errors int
|
||||||
|
PausedUntil, RefuseUntil int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// SiteOutcomes is one Site's summed Lane Pass outcomes over a caller-supplied
|
||||||
|
// window.
|
||||||
|
type SiteOutcomes struct {
|
||||||
|
Site string
|
||||||
|
Refused, Unreachable, NoChapter int
|
||||||
|
Unfetchable, Errors int
|
||||||
|
}
|
||||||
|
|
||||||
|
// LanePause is one persisted Lane pause stamp.
|
||||||
|
type LanePause struct {
|
||||||
|
Site string
|
||||||
|
PausedUntil int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
|
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
|
||||||
@@ -188,9 +221,9 @@ const (
|
|||||||
//go:embed migrations/*.sql
|
//go:embed migrations/*.sql
|
||||||
var migrations embed.FS
|
var migrations embed.FS
|
||||||
|
|
||||||
// bookmarkColumns is the only value ever concatenated into query text. It is a
|
// These column lists are the only values ever concatenated into query text.
|
||||||
// compile-time constant; every request value is bound as a parameter. The
|
// They are compile-time constants; every request value is bound as a parameter.
|
||||||
// series-owned fields are joined in from the series table, in scanBookmark
|
// The series-owned fields are joined in from the series table, in scanBookmark
|
||||||
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
|
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
|
||||||
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover_address,
|
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover_address,
|
||||||
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
|
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
|
||||||
@@ -202,6 +235,10 @@ const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover_add
|
|||||||
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover, s.cover_address,
|
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover, s.cover_address,
|
||||||
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.latest_raised_by`
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.latest_raised_by`
|
||||||
|
|
||||||
|
const lanePassColumns = `p.site, p.ran_at, p.skip, p.due, p.checked, p.gap_ms, p.clamped,
|
||||||
|
p.refused, p.unreachable, p.no_chapter, p.unfetchable, p.errors,
|
||||||
|
COALESCE(l.paused_until, 0), COALESCE(l.refuse_until, 0)`
|
||||||
|
|
||||||
// Owner is the person running the service: the first Reader, seeded at startup
|
// Owner is the person running the service: the first Reader, seeded at startup
|
||||||
// so a fresh deployment has a library before anyone logs in. The seed makes
|
// so a fresh deployment has a library before anyone logs in. The seed makes
|
||||||
// sure exactly one readers row matches their Discord ID, carrying the SHA-256
|
// sure exactly one readers row matches their Discord ID, carrying the SHA-256
|
||||||
@@ -589,8 +626,9 @@ func (s *Store) scanBookmark(scan func(...any) error) (Bookmark, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// scanSeries reads one row in seriesColumns order, plus the due query's
|
// scanSeries reads one row in seriesColumns order, plus the due query's
|
||||||
// reader_count column. latest_chapter_num and latest_raised_by are both
|
// forced flag and reader_count columns. latest_chapter_num and
|
||||||
// nullable, same as latest_chapter_num on the bookmark read path.
|
// latest_raised_by are both nullable, same as latest_chapter_num on the
|
||||||
|
// bookmark read path.
|
||||||
func scanSeries(scan func(...any) error) (Series, error) {
|
func scanSeries(scan func(...any) error) (Series, error) {
|
||||||
var (
|
var (
|
||||||
sr Series
|
sr Series
|
||||||
@@ -600,7 +638,7 @@ func scanSeries(scan func(...any) error) (Series, error) {
|
|||||||
if err := scan(
|
if err := scan(
|
||||||
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover, &sr.CoverAddress,
|
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover, &sr.CoverAddress,
|
||||||
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt, &latestRaisedBy,
|
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt, &latestRaisedBy,
|
||||||
&sr.readerCount,
|
&sr.Forced, &sr.readerCount,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return Series{}, err
|
return Series{}, err
|
||||||
}
|
}
|
||||||
@@ -613,6 +651,18 @@ func scanSeries(scan func(...any) error) (Series, error) {
|
|||||||
return sr, nil
|
return sr, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func scanLanePass(scan func(...any) error) (LanePass, error) {
|
||||||
|
var p LanePass
|
||||||
|
if err := scan(
|
||||||
|
&p.Site, &p.RanAt, &p.Skip, &p.Due, &p.Checked, &p.GapMS, &p.Clamped,
|
||||||
|
&p.Refused, &p.Unreachable, &p.NoChapter, &p.Unfetchable, &p.Errors,
|
||||||
|
&p.PausedUntil, &p.RefuseUntil,
|
||||||
|
); err != nil {
|
||||||
|
return LanePass{}, err
|
||||||
|
}
|
||||||
|
return p, nil
|
||||||
|
}
|
||||||
|
|
||||||
// Close releases the underlying database handle.
|
// Close releases the underlying database handle.
|
||||||
func (s *Store) Close() error { return s.db.Close() }
|
func (s *Store) Close() error { return s.db.Close() }
|
||||||
|
|
||||||
@@ -934,6 +984,184 @@ func (s *Store) Delete(readerID int64, key string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RecordLanePass appends one pass and prunes every older row in the same
|
||||||
|
// transaction. retainBefore is supplied by the poller's clock.
|
||||||
|
func (s *Store) RecordLanePass(p LanePass, retainBefore int64) error {
|
||||||
|
tx, err := s.db.Begin()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("begin lane pass %s: %w", p.Site, err)
|
||||||
|
}
|
||||||
|
defer tx.Rollback()
|
||||||
|
if _, err := tx.Exec(`
|
||||||
|
INSERT INTO poll_passes
|
||||||
|
(site, ran_at, skip, due, checked, gap_ms, clamped,
|
||||||
|
refused, unreachable, no_chapter, unfetchable, errors)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)`,
|
||||||
|
p.Site, p.RanAt, p.Skip, p.Due, p.Checked, p.GapMS, p.Clamped,
|
||||||
|
p.Refused, p.Unreachable, p.NoChapter, p.Unfetchable, p.Errors); err != nil {
|
||||||
|
return fmt.Errorf("insert lane pass %s at %d: %w", p.Site, p.RanAt, err)
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(`DELETE FROM poll_passes WHERE ran_at < $1`, retainBefore); err != nil {
|
||||||
|
return fmt.Errorf("prune lane passes before %d: %w", retainBefore, err)
|
||||||
|
}
|
||||||
|
if err := tx.Commit(); err != nil {
|
||||||
|
return fmt.Errorf("commit lane pass %s at %d: %w", p.Site, p.RanAt, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LatestLanePass returns the newest pass for one Site, with its current Lane
|
||||||
|
// state joined on. A Site without a pass has no durable snapshot yet.
|
||||||
|
func (s *Store) LatestLanePass(site string) (LanePass, bool, error) {
|
||||||
|
p, err := scanLanePass(s.db.QueryRow(`SELECT `+lanePassColumns+`
|
||||||
|
FROM poll_passes p
|
||||||
|
LEFT JOIN poll_lanes l ON l.site = p.site
|
||||||
|
WHERE p.site = $1
|
||||||
|
ORDER BY p.ran_at DESC
|
||||||
|
LIMIT 1`, site).Scan)
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return LanePass{}, false, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return LanePass{}, false, fmt.Errorf("latest lane pass %s: %w", site, err)
|
||||||
|
}
|
||||||
|
return p, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LatestLanePasses returns the newest pass for each Site, with current Lane
|
||||||
|
// state joined on. Sites without a pass have no row yet.
|
||||||
|
func (s *Store) LatestLanePasses() ([]LanePass, error) {
|
||||||
|
rows, err := s.db.Query(`SELECT ` + lanePassColumns + `
|
||||||
|
FROM (
|
||||||
|
SELECT DISTINCT ON (site)
|
||||||
|
site, ran_at, skip, due, checked, gap_ms, clamped,
|
||||||
|
refused, unreachable, no_chapter, unfetchable, errors
|
||||||
|
FROM poll_passes
|
||||||
|
ORDER BY site, ran_at DESC
|
||||||
|
) p
|
||||||
|
LEFT JOIN poll_lanes l ON l.site = p.site
|
||||||
|
ORDER BY p.site`)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("query latest lane passes: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := []LanePass{}
|
||||||
|
for rows.Next() {
|
||||||
|
p, err := scanLanePass(rows.Scan)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("scan latest lane pass: %w", err)
|
||||||
|
}
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// LanePassOutcomes sums the named outcomes for each Site at or after since.
|
||||||
|
// The window boundary is supplied by the caller; the store has no clock.
|
||||||
|
func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
|
||||||
|
rows, err := s.db.Query(`
|
||||||
|
SELECT site, SUM(refused), SUM(unreachable), SUM(no_chapter),
|
||||||
|
SUM(unfetchable), SUM(errors)
|
||||||
|
FROM poll_passes
|
||||||
|
WHERE ran_at >= $1
|
||||||
|
GROUP BY site
|
||||||
|
ORDER BY site`, since)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("query lane pass outcomes: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := []SiteOutcomes{}
|
||||||
|
for rows.Next() {
|
||||||
|
var outcomes SiteOutcomes
|
||||||
|
if err := rows.Scan(
|
||||||
|
&outcomes.Site, &outcomes.Refused, &outcomes.Unreachable,
|
||||||
|
&outcomes.NoChapter, &outcomes.Unfetchable, &outcomes.Errors,
|
||||||
|
); err != nil {
|
||||||
|
return nil, fmt.Errorf("scan lane pass outcomes: %w", err)
|
||||||
|
}
|
||||||
|
out = append(out, outcomes)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetLaneRefusal persists a Site's refusal backoff stamp without touching its
|
||||||
|
// pause. until is supplied by the caller's clock.
|
||||||
|
func (s *Store) SetLaneRefusal(site string, until int64) error {
|
||||||
|
if _, err := s.db.Exec(`
|
||||||
|
INSERT INTO poll_lanes (site, refuse_until) VALUES ($1, $2)
|
||||||
|
ON CONFLICT (site) DO UPDATE SET refuse_until = EXCLUDED.refuse_until`, site, until); err != nil {
|
||||||
|
return fmt.Errorf("set lane refusal %s: %w", site, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PauseLane persists a bounded pause. The caller must ensure until is after
|
||||||
|
// its current timestamp; the store has no clock and rejects only the invalid
|
||||||
|
// zero and negative sentinels.
|
||||||
|
func (s *Store) PauseLane(site string, until int64) error {
|
||||||
|
if until <= 0 {
|
||||||
|
return fmt.Errorf("pause lane %s: expiry must be positive", site)
|
||||||
|
}
|
||||||
|
if _, err := s.db.Exec(`
|
||||||
|
INSERT INTO poll_lanes (site, paused_until) VALUES ($1, $2)
|
||||||
|
ON CONFLICT (site) DO UPDATE SET paused_until = EXCLUDED.paused_until`, site, until); err != nil {
|
||||||
|
return fmt.Errorf("pause lane %s: %w", site, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResumeLane clears only the pause stamp and keeps the Lane state row, along
|
||||||
|
// with any refusal stamp already persisted on it.
|
||||||
|
func (s *Store) ResumeLane(site string) error {
|
||||||
|
if _, err := s.db.Exec(
|
||||||
|
`UPDATE poll_lanes SET paused_until = 0 WHERE site = $1`, site); err != nil {
|
||||||
|
return fmt.Errorf("resume lane %s: %w", site, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PausedLanes returns Lane rows with a nonzero pause stamp. Expiry comparison
|
||||||
|
// stays with the caller because the store is deliberately clockless.
|
||||||
|
func (s *Store) PausedLanes() ([]LanePause, error) {
|
||||||
|
rows, err := s.db.Query(`
|
||||||
|
SELECT site, paused_until
|
||||||
|
FROM poll_lanes
|
||||||
|
WHERE paused_until > 0
|
||||||
|
ORDER BY site`)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("query paused lanes: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := []LanePause{}
|
||||||
|
for rows.Next() {
|
||||||
|
var pause LanePause
|
||||||
|
if err := rows.Scan(&pause.Site, &pause.PausedUntil); err != nil {
|
||||||
|
return nil, fmt.Errorf("scan paused lane: %w", err)
|
||||||
|
}
|
||||||
|
out = append(out, pause)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// LaneGates reads a Site's pause and refusal stamps in one row read — the
|
||||||
|
// top-of-pass gate the poller uses (issue #141). A missing state row is the
|
||||||
|
// default: unpaused and not refusing.
|
||||||
|
func (s *Store) LaneGates(site string) (pausedUntil, refuseUntil int64, err error) {
|
||||||
|
err = s.db.QueryRow(
|
||||||
|
`SELECT paused_until, refuse_until FROM poll_lanes WHERE site = $1`, site).
|
||||||
|
Scan(&pausedUntil, &refuseUntil)
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return 0, 0, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0, fmt.Errorf("lane gates %s: %w", site, err)
|
||||||
|
}
|
||||||
|
return pausedUntil, refuseUntil, nil
|
||||||
|
}
|
||||||
|
|
||||||
// DueForLatestCheck returns one Site's series whose server-side
|
// DueForLatestCheck returns one Site's series whose server-side
|
||||||
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks
|
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks
|
||||||
// reference them (descending) then least-recently-checked first. One Site per
|
// reference them (descending) then least-recently-checked first. One Site per
|
||||||
@@ -942,6 +1170,13 @@ func (s *Store) Delete(readerID int64, key string) error {
|
|||||||
// limit — the Lane's own gap paces the fetches, and the batch size that used
|
// limit — the Lane's own gap paces the fetches, and the batch size that used
|
||||||
// to cap this query is gone with the shared pace.
|
// to cap this query is gone with the shared pace.
|
||||||
//
|
//
|
||||||
|
// A forced Series (force_poll_at newer than latest_checked_at, issue #146)
|
||||||
|
// overrides exactly three gates: the rest cutoff, the Sighting-deferral
|
||||||
|
// clause and the finished-only bucket. It never overrides an empty
|
||||||
|
// series_url or the Bookmarks join — nothing to fetch, and no consumer for
|
||||||
|
// the result — so those stay unconditional. Forced rows sort to the front of
|
||||||
|
// the queue; the reader-count-then-age ordering among the rest is ADR-0003.
|
||||||
|
//
|
||||||
// The reader_count ordering is the point of the split (ADR-0003): a series
|
// The reader_count ordering is the point of the split (ADR-0003): a series
|
||||||
// shared by several readers is fetched once per due cycle, and the popular
|
// shared by several readers is fetched once per due cycle, and the popular
|
||||||
// ones stay freshest while the long tail absorbs any shortfall. Within one
|
// ones stay freshest while the long tail absorbs any shortfall. Within one
|
||||||
@@ -970,19 +1205,26 @@ func (s *Store) Delete(readerID int64, key string) error {
|
|||||||
// allowed to defer at all was settled when the Sighting was recorded — see
|
// allowed to defer at all was settled when the Sighting was recorded — see
|
||||||
// RecordSighting.
|
// RecordSighting.
|
||||||
func (s *Store) DueForLatestCheck(site string, cutoffMs, ceilingMs int64) ([]Series, error) {
|
func (s *Store) DueForLatestCheck(site string, cutoffMs, ceilingMs int64) ([]Series, error) {
|
||||||
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
|
rows, err := s.db.Query(`SELECT `+seriesColumns+`,
|
||||||
|
(s.force_poll_at > s.latest_checked_at) AS forced,
|
||||||
|
COUNT(*) AS reader_count
|
||||||
FROM series s
|
FROM series s
|
||||||
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||||
WHERE s.site = $1
|
WHERE s.site = $1
|
||||||
AND s.series_url <> ''
|
AND s.series_url <> ''
|
||||||
AND s.latest_checked_at <= $2::bigint
|
AND (s.latest_checked_at <= $2::bigint
|
||||||
|
OR s.force_poll_at > s.latest_checked_at)
|
||||||
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
|
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||||
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
|
||||||
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
|
s.force_poll_at
|
||||||
|
HAVING (COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
|
||||||
|
OR s.force_poll_at > s.latest_checked_at)
|
||||||
AND (COUNT(*) > 1
|
AND (COUNT(*) > 1
|
||||||
OR s.latest_sighted_at <= $2::bigint
|
OR s.latest_sighted_at <= $2::bigint
|
||||||
OR s.latest_checked_at <= $3::bigint)
|
OR s.latest_checked_at <= $3::bigint
|
||||||
ORDER BY reader_count DESC, s.latest_checked_at ASC`, site, cutoffMs, ceilingMs)
|
OR s.force_poll_at > s.latest_checked_at)
|
||||||
|
ORDER BY (s.force_poll_at > s.latest_checked_at) DESC,
|
||||||
|
reader_count DESC, s.latest_checked_at ASC`, site, cutoffMs, ceilingMs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("query due series: %w", err)
|
return nil, fmt.Errorf("query due series: %w", err)
|
||||||
}
|
}
|
||||||
@@ -1042,6 +1284,25 @@ func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ForceSeriesPoll stamps a Series with the owner's "check now" request
|
||||||
|
// (issue #146): a fact about the Series the Lane's next pass reads through
|
||||||
|
// DueForLatestCheck, never a command to the poller — so the request survives
|
||||||
|
// a restart. Writing again overwrites the request time; the write is
|
||||||
|
// idempotent. Touching a missing series is not an error: the row may have
|
||||||
|
// been orphaned, and the caller's read decides what exists. The stamp never
|
||||||
|
// expires by itself — an unanswered request keeps ageing — and pending is
|
||||||
|
// derived as force_poll_at > latest_checked_at, which is why the poller's
|
||||||
|
// check stamp is written before the fetch: the first attempt ends the
|
||||||
|
// pending state whatever it returns.
|
||||||
|
func (s *Store) ForceSeriesPoll(site, seriesID string, at int64) error {
|
||||||
|
if _, err := s.db.Exec(
|
||||||
|
`UPDATE series SET force_poll_at = $1 WHERE site = $2 AND series_id = $3`,
|
||||||
|
at, site, seriesID); err != nil {
|
||||||
|
return fmt.Errorf("force poll %s:%s: %w", site, seriesID, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
||||||
// tests outside this package (the poller's own tests assert on rest
|
// tests outside this package (the poller's own tests assert on rest
|
||||||
// bookkeeping) — see MarkLatestChecked for why the field stays off the
|
// bookkeeping) — see MarkLatestChecked for why the field stays off the
|
||||||
|
|||||||
@@ -758,7 +758,7 @@ func TestMigration0008DropsLegacyCoverRows(t *testing.T) {
|
|||||||
func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
|
func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
sr, err := scanSeries(s.db.QueryRow(
|
sr, err := scanSeries(s.db.QueryRow(
|
||||||
`SELECT `+seriesColumns+`, 0 AS reader_count FROM series s
|
`SELECT `+seriesColumns+`, false AS forced, 0 AS reader_count FROM series s
|
||||||
WHERE s.site = $1 AND s.series_id = $2`, site, seriesID).Scan)
|
WHERE s.site = $1 AND s.series_id = $2`, site, seriesID).Scan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("read series %s:%s: %v", site, seriesID, err)
|
t.Fatalf("read series %s:%s: %v", site, seriesID, err)
|
||||||
@@ -1584,3 +1584,318 @@ func TestCoverStoreAcceptsAnySourceURL(t *testing.T) {
|
|||||||
t.Fatalf("rejected cover = found %v, err %v; want missing", ok, err)
|
t.Fatalf("rejected cover = found %v, err %v; want missing", ok, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
func TestRecordLanePassPrunesBeforeInsertCutoff(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
for _, pass := range []LanePass{
|
||||||
|
{Site: "asura", RanAt: 99},
|
||||||
|
{Site: "asura", RanAt: 100},
|
||||||
|
} {
|
||||||
|
if err := s.RecordLanePass(pass, 100); err != nil {
|
||||||
|
t.Fatalf("RecordLanePass(%d): %v", pass.RanAt, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.RecordLanePass(LanePass{Site: "asura", RanAt: 200}, 100); err != nil {
|
||||||
|
t.Fatalf("RecordLanePass(200): %v", err)
|
||||||
|
}
|
||||||
|
var count int
|
||||||
|
if err := s.db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1`, "asura").Scan(&count); err != nil {
|
||||||
|
t.Fatalf("count passes: %v", err)
|
||||||
|
}
|
||||||
|
if count != 2 {
|
||||||
|
t.Fatalf("retained passes = %d, want 2", count)
|
||||||
|
}
|
||||||
|
if _, ok, err := s.LatestLanePass("asura"); err != nil || !ok {
|
||||||
|
t.Fatalf("LatestLanePass = ok %v, err %v; want latest row", ok, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLatestLanePassesKeepsNewestPerSiteAndJoinsState(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
for _, pass := range []LanePass{
|
||||||
|
{Site: "asura", RanAt: 100, Due: 1},
|
||||||
|
{Site: "asura", RanAt: 200, Skip: "due-query", Due: 2, Checked: 3, GapMS: 4000, Clamped: true},
|
||||||
|
{Site: "demonic", RanAt: 150, Due: 4},
|
||||||
|
} {
|
||||||
|
if err := s.RecordLanePass(pass, -1); err != nil {
|
||||||
|
t.Fatalf("RecordLanePass(%s/%d): %v", pass.Site, pass.RanAt, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.PauseLane("asura", 1234); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.SetLaneRefusal("asura", 5678); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := s.LatestLanePasses()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LatestLanePasses: %v", err)
|
||||||
|
}
|
||||||
|
if len(got) != 2 {
|
||||||
|
t.Fatalf("latest passes = %d, want one per Site", len(got))
|
||||||
|
}
|
||||||
|
bySite := map[string]LanePass{}
|
||||||
|
for _, pass := range got {
|
||||||
|
bySite[pass.Site] = pass
|
||||||
|
}
|
||||||
|
asura := bySite["asura"]
|
||||||
|
if asura.RanAt != 200 || asura.Skip != "due-query" || asura.Due != 2 || asura.Checked != 3 || asura.GapMS != 4000 || !asura.Clamped ||
|
||||||
|
asura.PausedUntil != 1234 || asura.RefuseUntil != 5678 {
|
||||||
|
t.Fatalf("asura latest pass = %+v, want newest pass and joined state", asura)
|
||||||
|
}
|
||||||
|
if demonic := bySite["demonic"]; demonic.RanAt != 150 || demonic.Due != 4 {
|
||||||
|
t.Fatalf("demonic latest pass = %+v, want its only pass", demonic)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLanePassOutcomesSumsWindow(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
for _, pass := range []LanePass{
|
||||||
|
{Site: "asura", RanAt: 99, Refused: 1, Unreachable: 2, NoChapter: 3, Unfetchable: 4, Errors: 5},
|
||||||
|
{Site: "asura", RanAt: 100, Refused: 2, Unreachable: 3, NoChapter: 4, Unfetchable: 5, Errors: 6},
|
||||||
|
{Site: "asura", RanAt: 200, Refused: 3, Unreachable: 4, NoChapter: 5, Unfetchable: 6, Errors: 7},
|
||||||
|
{Site: "demonic", RanAt: 150, Refused: 8, Unreachable: 9, NoChapter: 10, Unfetchable: 11, Errors: 12},
|
||||||
|
} {
|
||||||
|
if err := s.RecordLanePass(pass, -1); err != nil {
|
||||||
|
t.Fatalf("RecordLanePass(%s/%d): %v", pass.Site, pass.RanAt, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := s.LanePassOutcomes(100)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LanePassOutcomes: %v", err)
|
||||||
|
}
|
||||||
|
if len(got) != 2 {
|
||||||
|
t.Fatalf("outcome Sites = %d, want 2", len(got))
|
||||||
|
}
|
||||||
|
bySite := map[string]SiteOutcomes{}
|
||||||
|
for _, outcomes := range got {
|
||||||
|
bySite[outcomes.Site] = outcomes
|
||||||
|
}
|
||||||
|
if want := (SiteOutcomes{Site: "asura", Refused: 5, Unreachable: 7, NoChapter: 9, Unfetchable: 11, Errors: 13}); bySite["asura"] != want {
|
||||||
|
t.Fatalf("asura outcomes = %+v, want %+v", bySite["asura"], want)
|
||||||
|
}
|
||||||
|
if want := (SiteOutcomes{Site: "demonic", Refused: 8, Unreachable: 9, NoChapter: 10, Unfetchable: 11, Errors: 12}); bySite["demonic"] != want {
|
||||||
|
t.Fatalf("demonic outcomes = %+v, want %+v", bySite["demonic"], want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLaneGatesReadsOneRow(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
paused, refused, err := s.LaneGates("asura")
|
||||||
|
if err != nil || paused != 0 || refused != 0 {
|
||||||
|
t.Fatalf("LaneGates on a missing state row = (%d, %d, %v), want (0, 0, nil)", paused, refused, err)
|
||||||
|
}
|
||||||
|
if err := s.PauseLane("asura", 2000); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.SetLaneRefusal("asura", 3000); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
paused, refused, err = s.LaneGates("asura")
|
||||||
|
if err != nil || paused != 2000 || refused != 3000 {
|
||||||
|
t.Fatalf("LaneGates = (%d, %d, %v), want (2000, 3000, nil)", paused, refused, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLaneStatePauseResumeAndRefusal(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
for _, until := range []int64{0, -1} {
|
||||||
|
if err := s.PauseLane("asura", until); err == nil {
|
||||||
|
t.Fatalf("PauseLane(%d) accepted a non-future expiry", until)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.PauseLane("asura", 2000); err != nil {
|
||||||
|
t.Fatalf("PauseLane: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.SetLaneRefusal("asura", 3000); err != nil {
|
||||||
|
t.Fatalf("SetLaneRefusal: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.RecordLanePass(LanePass{Site: "asura", RanAt: 1}, -1); err != nil {
|
||||||
|
t.Fatalf("RecordLanePass: %v", err)
|
||||||
|
}
|
||||||
|
pausedUntil, refuseUntil, err := s.LaneGates("asura")
|
||||||
|
if err != nil || pausedUntil != 2000 || refuseUntil != 3000 {
|
||||||
|
t.Fatalf("LaneGates = %d/%d, %v; want 2000/3000", pausedUntil, refuseUntil, err)
|
||||||
|
}
|
||||||
|
paused, err := s.PausedLanes()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PausedLanes: %v", err)
|
||||||
|
}
|
||||||
|
if len(paused) != 1 || paused[0] != (LanePause{Site: "asura", PausedUntil: 2000}) {
|
||||||
|
t.Fatalf("PausedLanes = %+v, want asura/2000", paused)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.ResumeLane("asura"); err != nil {
|
||||||
|
t.Fatalf("ResumeLane: %v", err)
|
||||||
|
}
|
||||||
|
latest, ok, err := s.LatestLanePass("asura")
|
||||||
|
if err != nil || !ok || latest.RefuseUntil != 3000 {
|
||||||
|
t.Fatalf("latest refusal after resume = %+v, ok=%v, err=%v; want 3000 preserved", latest, ok, err)
|
||||||
|
}
|
||||||
|
if got, got2, err := s.LaneGates("asura"); err != nil || got != 0 || got2 != 3000 {
|
||||||
|
t.Fatalf("LaneGates after resume = %d/%d, %v; want 0/3000", got, got2, err)
|
||||||
|
}
|
||||||
|
if paused, err := s.PausedLanes(); err != nil || len(paused) != 0 {
|
||||||
|
t.Fatalf("PausedLanes after resume = %+v, %v; want empty", paused, err)
|
||||||
|
}
|
||||||
|
var rows int
|
||||||
|
if err := s.db.QueryRow(`SELECT count(*) FROM poll_lanes WHERE site = $1`, "asura").Scan(&rows); err != nil {
|
||||||
|
t.Fatalf("count lane state: %v", err)
|
||||||
|
}
|
||||||
|
if rows != 1 {
|
||||||
|
t.Fatalf("lane state rows after resume = %d, want 1", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A forced Series is due ahead of the rest cutoff: the request overrides the
|
||||||
|
// rest gate so the Lane's next pass picks it up however recently it was
|
||||||
|
// checked. An unforced series under the rest stays out.
|
||||||
|
func TestDueForLatestCheckForcedOverridesRestCutoff(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const now = int64(10 * 3600_000)
|
||||||
|
seedForCheck(t, s, "asura:forced", "https://asurascans.com/comics/forced", now-30*60_000)
|
||||||
|
seedForCheck(t, s, "asura:fresh", "https://asurascans.com/comics/fresh", now-30*60_000)
|
||||||
|
if err := s.ForceSeriesPoll("asura", "forced", now); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck("asura", now-3600_000, noCeiling)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 1 || due[0].Key() != "asura:forced" {
|
||||||
|
t.Fatalf("due = %v, want only the forced series", due)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The Sighting-deferral clause holds a solitary series back after a recent
|
||||||
|
// Reader report; a forced request overrides it and puts the series back on
|
||||||
|
// the Lane's list.
|
||||||
|
func TestDueForLatestCheckForcedOverridesSightingDeferral(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const now = int64(10 * 3600_000)
|
||||||
|
// One bookmark (so deferral can apply), sighted and checked 10m ago:
|
||||||
|
// inside the deferral window and under the ceiling.
|
||||||
|
seedForCheck(t, s, "asura:deferred", "https://asurascans.com/comics/deferred", now-10*60_000)
|
||||||
|
if _, err := s.db.Exec(
|
||||||
|
`UPDATE series SET latest_sighted_at = $1 WHERE site = 'asura' AND series_id = 'deferred'`,
|
||||||
|
now-10*60_000); err != nil {
|
||||||
|
t.Fatalf("seed sighting: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Unforced: deferred, and under the rest anyway.
|
||||||
|
due, err := s.DueForLatestCheck("asura", now-3600_000, now-3*3600_000)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 0 {
|
||||||
|
t.Fatalf("unforced deferred series is due: %v", due)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Forced: the request overrides the deferral.
|
||||||
|
if err := s.ForceSeriesPoll("asura", "deferred", now); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
due, err = s.DueForLatestCheck("asura", now-3600_000, now-3*3600_000)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 1 || due[0].Key() != "asura:deferred" {
|
||||||
|
t.Fatalf("forced deferred series not due: %v", due)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The finished-only bucket excludes a series whose only bookmarks are
|
||||||
|
// finished; a forced request overrides it — the owner asked, so the Lane
|
||||||
|
// looks.
|
||||||
|
func TestDueForLatestCheckForcedOverridesFinishedBucket(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "asura:reading", "https://asurascans.com/comics/reading", 0)
|
||||||
|
if _, err := s.Upsert(s.OwnerID(), Bookmark{
|
||||||
|
Key: "asura:finished", Site: "asura", SeriesID: "finished",
|
||||||
|
SeriesURL: "https://asurascans.com/comics/finished",
|
||||||
|
Status: StatusFinished, UpdatedAt: 1000,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("seed finished: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck("asura", 1000, noCeiling)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
for _, sr := range due {
|
||||||
|
if sr.Key() == "asura:finished" {
|
||||||
|
t.Fatalf("unforced finished series is due: %v", due)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.ForceSeriesPoll("asura", "finished", 5000); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
due, err = s.DueForLatestCheck("asura", 1000, noCeiling)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
got := map[string]bool{}
|
||||||
|
for _, sr := range due {
|
||||||
|
got[sr.Key()] = true
|
||||||
|
}
|
||||||
|
if !got["asura:reading"] || !got["asura:finished"] {
|
||||||
|
t.Fatalf("forced finished series not due: %v", due)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A forced Series jumps the queue: it sorts ahead of a more-read series that
|
||||||
|
// is due by rest, without disturbing the reader-count-then-age tie-break
|
||||||
|
// among the unforced rows (ADR-0003).
|
||||||
|
func TestDueForLatestCheckForcedSortsFirst(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
// "popular" has two readers and is long overdue; "forced" has one reader
|
||||||
|
// and a fresh check stamp. The forced row must come first.
|
||||||
|
seedForCheck(t, s, "asura:popular", "https://asurascans.com/comics/popular", 100)
|
||||||
|
seedSecondReader(t, s, "asura:popular:2", "asura", "popular", 1001)
|
||||||
|
seedForCheck(t, s, "asura:forced", "https://asurascans.com/comics/forced", 900)
|
||||||
|
if err := s.ForceSeriesPoll("asura", "forced", 5000); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck("asura", 1000, noCeiling)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 2 {
|
||||||
|
t.Fatalf("due = %d rows, want 2", len(due))
|
||||||
|
}
|
||||||
|
if due[0].Key() != "asura:forced" || due[1].Key() != "asura:popular" {
|
||||||
|
t.Fatalf("due order = %q, %q; want forced first, then popular", due[0].Key(), due[1].Key())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A forced Series with no series URL is still not fetched — nothing to fetch —
|
||||||
|
// and one with no Bookmarks is still excluded by the join. The force flag
|
||||||
|
// opens the three gates it is allowed to, not the whole query.
|
||||||
|
func TestDueForLatestCheckForcedDoesNotOverrideURLOrJoin(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "asura:nourl", "", 0)
|
||||||
|
if err := s.ForceSeriesPoll("asura", "nourl", 5000); err != nil {
|
||||||
|
t.Fatalf("ForceSeriesPoll: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.db.Exec(`
|
||||||
|
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||||
|
latest_chapter, latest_chapter_num, latest_checked_at,
|
||||||
|
force_poll_at)
|
||||||
|
VALUES ('asura', 'orphan', 'Orphan', 'https://asurascans.com/comics/orphan',
|
||||||
|
'', 'manga', '', NULL, 0, 5000)`); err != nil {
|
||||||
|
t.Fatalf("seed orphan: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck("asura", 1000, noCeiling)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 0 {
|
||||||
|
t.Fatalf("due = %v, want neither the URL-less nor the orphan series", due)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+37
-118
@@ -6,69 +6,27 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"bookmarkmanager/backend/internal/latest"
|
|
||||||
"bookmarkmanager/backend/internal/store"
|
"bookmarkmanager/backend/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
// LaneReporter is the administrative page's whole window onto the running
|
// ownerWindow is the staleness boundary the Series list's "not checked in
|
||||||
// poller: one snapshot of Poll Lane state, copied out of memory on request.
|
// 12h" filter compares against. Declared once; later admin tickets read it.
|
||||||
// The Poller satisfies it in production and a fake with fixed values satisfies
|
const ownerWindow = 12 * time.Hour
|
||||||
// it in tests, so the page's tests need neither a poller nor a Site.
|
|
||||||
type LaneReporter interface {
|
|
||||||
LaneStatus() latest.Status
|
|
||||||
}
|
|
||||||
|
|
||||||
// adminView is what the administrative page and the roster fragment receive.
|
// adminView is the shared shell data for an administrative page and the roster
|
||||||
|
// fragment returned after a Reader action.
|
||||||
type adminView struct {
|
type adminView struct {
|
||||||
|
Page string
|
||||||
Readers []store.ReaderSummary
|
Readers []store.ReaderSummary
|
||||||
// OwnerID travels with the roster so it can tell the owner's own row from
|
// OwnerID travels with the roster so it can tell the owner's own row from
|
||||||
// the Readers they may act on.
|
// the Readers they may act on.
|
||||||
OwnerID int64
|
OwnerID int64
|
||||||
Lanes lanesView
|
Lanes lanesView
|
||||||
}
|
SeriesList seriesListView
|
||||||
|
// Detail is the per-Series page data; zero on every other page.
|
||||||
// lanesView is the Lane status block: one row per Site that has run, plus the
|
Detail seriesDetailView
|
||||||
// browser fact, which is shared by the three browser Sites rather than held
|
// Overview is the landing page data; zero on every other page.
|
||||||
// once per Site.
|
Overview overviewView
|
||||||
type lanesView struct {
|
|
||||||
Rows []laneRow
|
|
||||||
// PollerOff means no poller is running at all (disabled by config, or its
|
|
||||||
// client could not be built). The browser line must not answer "not
|
|
||||||
// configured" then: the sidecar is not the reason nothing is polled.
|
|
||||||
PollerOff bool
|
|
||||||
BrowserConfigured bool
|
|
||||||
BrowserReachable bool
|
|
||||||
}
|
|
||||||
|
|
||||||
// laneRow is one Lane formatted for reading rather than for arithmetic: the
|
|
||||||
// template renders strings and flags, and every judgement about what they mean
|
|
||||||
// is made here.
|
|
||||||
type laneRow struct {
|
|
||||||
Site string
|
|
||||||
Due int
|
|
||||||
Ran string
|
|
||||||
// Checked is how many Series the last pass read. Due without Checked is a
|
|
||||||
// Lane that has stopped working; the two figures side by side are what
|
|
||||||
// separate that from a Lane with nothing to do.
|
|
||||||
Checked int
|
|
||||||
// Gap is empty when no pass has reached the pace yet, so the row omits the
|
|
||||||
// figure instead of stating a zero.
|
|
||||||
Gap string
|
|
||||||
Clamped bool
|
|
||||||
Refusing bool
|
|
||||||
// BrowserLost marks a Lane whose pages can only be read through the
|
|
||||||
// sidecar while the sidecar is unreachable — including the case where none
|
|
||||||
// is configured, which stops those Series just as completely.
|
|
||||||
BrowserLost bool
|
|
||||||
// Stalled marks a Lane with Series waiting that its last pass did not read
|
|
||||||
// — the difference between a stopped Lane and a quiet one (story 13). A
|
|
||||||
// browser Lane holding Chrome asleep under the wake thresholds is neither,
|
|
||||||
// so it carries Asleep instead and never Stalled.
|
|
||||||
Stalled bool
|
|
||||||
Asleep bool
|
|
||||||
// Attention is the one flag the template colours on, so an unhealthy Lane
|
|
||||||
// is found at a glance rather than read for.
|
|
||||||
Attention bool
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// adminRoute pairs a route pattern with its handler so the route list and the
|
// adminRoute pairs a route pattern with its handler so the route list and the
|
||||||
@@ -84,6 +42,13 @@ type adminRoute struct {
|
|||||||
func (h *Handler) adminRoutes() []adminRoute {
|
func (h *Handler) adminRoutes() []adminRoute {
|
||||||
return []adminRoute{
|
return []adminRoute{
|
||||||
{"GET /admin", h.admin},
|
{"GET /admin", h.admin},
|
||||||
|
{"GET /admin/lanes", h.adminLanes},
|
||||||
|
{"GET /admin/readers", h.adminReaders},
|
||||||
|
{"GET /admin/series", h.adminSeries},
|
||||||
|
{"GET /admin/series/{key}", h.adminSeriesDetail},
|
||||||
|
{"POST /admin/series/{key}/poll", h.adminSeriesPoll},
|
||||||
|
{"POST /admin/lanes/{site}/pause", h.adminLanePause},
|
||||||
|
{"POST /admin/lanes/{site}/resume", h.adminLaneResume},
|
||||||
{"GET /ui/admin/lanes", h.uiLanes},
|
{"GET /ui/admin/lanes", h.uiLanes},
|
||||||
{"POST /readers/{id}/revoke", h.revokeReaderSessions},
|
{"POST /readers/{id}/revoke", h.revokeReaderSessions},
|
||||||
{"POST /readers/{id}/clear-marks", h.clearReaderMarks},
|
{"POST /readers/{id}/clear-marks", h.clearReaderMarks},
|
||||||
@@ -116,78 +81,32 @@ func (h *Handler) requireOwner(next http.HandlerFunc) http.HandlerFunc {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// admin renders the owner's page: the Reader roster and Poll Lane status.
|
// admin renders the Overview landing page: a verdict line, a stats block
|
||||||
|
// where every figure is a door into the list it counts, and the per-Site
|
||||||
|
// library shape table — all read from the database, never from a poller.
|
||||||
func (h *Handler) admin(w http.ResponseWriter, r *http.Request) {
|
func (h *Handler) admin(w http.ResponseWriter, r *http.Request) {
|
||||||
readers, err := h.store.Readers()
|
view, err := h.overviewView()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("admin: %v", err)
|
log.Printf("admin overview: %v", err)
|
||||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
h.render(w, http.StatusOK, "admin", adminView{
|
h.renderAdmin(w, adminView{Page: "overview", Overview: view})
|
||||||
Readers: readers,
|
|
||||||
OwnerID: h.store.OwnerID(),
|
|
||||||
Lanes: h.lanesView(),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// uiLanes answers the status block's own refresh. Only the block refreshes on a
|
// adminReaders renders the Reader roster on its own bookmarkable page.
|
||||||
// timer; the roster re-renders after an action, as it always has.
|
func (h *Handler) adminReaders(w http.ResponseWriter, r *http.Request) {
|
||||||
func (h *Handler) uiLanes(w http.ResponseWriter, r *http.Request) {
|
readers, err := h.store.Readers()
|
||||||
h.render(w, http.StatusOK, "lanes", h.lanesView())
|
if err != nil {
|
||||||
|
log.Printf("admin readers: %v", err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.renderAdmin(w, adminView{Page: "readers", Readers: readers, OwnerID: h.store.OwnerID()})
|
||||||
}
|
}
|
||||||
|
|
||||||
// lanesView copies the poller's snapshot into display form. A nil reporter (no
|
func (h *Handler) renderAdmin(w http.ResponseWriter, view adminView) {
|
||||||
// poller running) and a poller no Lane has reported to yet are the same thing
|
h.render(w, http.StatusOK, "admin", view)
|
||||||
// to the page: no data, which it must say rather than draw as confident zeroes
|
|
||||||
// — an empty page a few seconds after a restart must not read as a stopped one.
|
|
||||||
func (h *Handler) lanesView() lanesView {
|
|
||||||
if h.lanes == nil {
|
|
||||||
return lanesView{PollerOff: true}
|
|
||||||
}
|
|
||||||
snap := h.lanes.LaneStatus()
|
|
||||||
v := lanesView{
|
|
||||||
Rows: make([]laneRow, 0, len(snap.Lanes)),
|
|
||||||
BrowserConfigured: snap.BrowserConfigured,
|
|
||||||
BrowserReachable: snap.BrowserReachable,
|
|
||||||
}
|
|
||||||
now := time.Now()
|
|
||||||
for _, l := range snap.Lanes {
|
|
||||||
lost := l.Browser && !snap.BrowserReachable
|
|
||||||
// Series waiting and none read is the shape of a Lane that has stopped
|
|
||||||
// working, as distinct from one that is quiet for want of work — or one
|
|
||||||
// deliberately leaving Chrome asleep until its group gathers.
|
|
||||||
stalled := l.Due > 0 && l.Checked == 0 && !l.Asleep
|
|
||||||
gap := ""
|
|
||||||
if l.Gap > 0 {
|
|
||||||
gap = l.Gap.Truncate(time.Second).String()
|
|
||||||
}
|
|
||||||
v.Rows = append(v.Rows, laneRow{
|
|
||||||
Site: l.Site,
|
|
||||||
Due: l.Due,
|
|
||||||
Ran: since(now, l.LastRun),
|
|
||||||
Checked: l.Checked,
|
|
||||||
Gap: gap,
|
|
||||||
Clamped: l.Clamped,
|
|
||||||
Refusing: l.Refusing,
|
|
||||||
BrowserLost: lost,
|
|
||||||
Stalled: stalled,
|
|
||||||
Asleep: l.Asleep,
|
|
||||||
Attention: l.Clamped || l.Refusing || lost || stalled,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
// since formats how long ago a Lane last ran, at second resolution: the block
|
|
||||||
// refreshes every thirty seconds, so anything finer is noise the owner would
|
|
||||||
// have to ignore.
|
|
||||||
func since(now, then time.Time) string {
|
|
||||||
d := now.Sub(then).Truncate(time.Second)
|
|
||||||
if d < time.Second {
|
|
||||||
return "just now"
|
|
||||||
}
|
|
||||||
return d.String() + " ago"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// revokeReaderSessions logs one Reader out of every browser they are signed in
|
// revokeReaderSessions logs one Reader out of every browser they are signed in
|
||||||
|
|||||||
@@ -0,0 +1,338 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"slices"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/latest"
|
||||||
|
"bookmarkmanager/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// lanesView is the Lane status block: one row per Site's latest durable pass,
|
||||||
|
// plus the browser fact derived from that same log. No poller is consulted —
|
||||||
|
// the page answers from the database, so it is complete thirty seconds after
|
||||||
|
// a deploy (issue #145).
|
||||||
|
type lanesView struct {
|
||||||
|
Rows []laneRow
|
||||||
|
// PollerOff means latest-chapter polling is switched off in this
|
||||||
|
// deployment (LATEST_CHAPTER_POLL_ENABLED). It is a config fact, not a
|
||||||
|
// poller answering "absent": the browser line must not blame the sidecar
|
||||||
|
// when nothing polls.
|
||||||
|
PollerOff bool
|
||||||
|
BrowserConfigured bool
|
||||||
|
BrowserReachable bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// laneRow is one Lane formatted for reading rather than for arithmetic: the
|
||||||
|
// template renders strings and flags, and every judgement about what they
|
||||||
|
// mean is made here.
|
||||||
|
type laneRow struct {
|
||||||
|
Site string
|
||||||
|
Due int
|
||||||
|
Checked int
|
||||||
|
// Gap is the last pass's pace, or "—" when no pass has reached one yet —
|
||||||
|
// a refused Lane still reports the pace its last real pass chose, so a
|
||||||
|
// zero here would be a figure the row never measured.
|
||||||
|
Gap string
|
||||||
|
Ran string
|
||||||
|
// Chips are the named outcome counts over the owner's window, in the
|
||||||
|
// taxonomy's fixed order. Empty writes "none observed".
|
||||||
|
Chips []chip
|
||||||
|
HasChips bool
|
||||||
|
// StatePhrase is the reason this Lane declined to work: a skipped pass's
|
||||||
|
// own sentence, or the one true stall. Empty means the pass reached its
|
||||||
|
// loop and read normally. StateGood marks a healthy way to do nothing
|
||||||
|
// (paused, browser asleep, nothing eligible) rather than a fault.
|
||||||
|
StatePhrase string
|
||||||
|
StateGood bool
|
||||||
|
// Attention is the one flag the template colours on, so a Lane that
|
||||||
|
// needs the owner is found at a glance rather than read for.
|
||||||
|
Attention bool
|
||||||
|
// Paused is the live pause state — the poll_lanes stamp the pass row
|
||||||
|
// joins on, still in the future — not the pass's skip: the control must
|
||||||
|
// offer Resume from the moment the owner presses Pause, with no pass
|
||||||
|
// having run to record it (issue #147).
|
||||||
|
Paused bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// chip is one named outcome count over the owner's window.
|
||||||
|
type chip struct {
|
||||||
|
Name string
|
||||||
|
Count int
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminLanes renders the page that hosts the live Lane fragment.
|
||||||
|
func (h *Handler) adminLanes(w http.ResponseWriter, r *http.Request) {
|
||||||
|
h.renderAdmin(w, adminView{Page: "lanes", Lanes: h.lanesView()})
|
||||||
|
}
|
||||||
|
|
||||||
|
// uiLanes answers the status block's own refresh. Only the block refreshes on
|
||||||
|
// a timer; the roster re-renders after an action, as it always has.
|
||||||
|
func (h *Handler) uiLanes(w http.ResponseWriter, r *http.Request) {
|
||||||
|
h.render(w, http.StatusOK, "lanes", h.lanesView())
|
||||||
|
}
|
||||||
|
|
||||||
|
// pauseDurations are the offered pause lengths, by their wire value. A fixed
|
||||||
|
// allow-list rather than time.ParseDuration: the unoffered value must be
|
||||||
|
// refused, and a permissive parser turns the offered set into "anything Go
|
||||||
|
// can read" (issue #147).
|
||||||
|
var pauseDurations = map[string]time.Duration{
|
||||||
|
"1h": time.Hour,
|
||||||
|
"6h": 6 * time.Hour,
|
||||||
|
"24h": 24 * time.Hour,
|
||||||
|
}
|
||||||
|
|
||||||
|
// laneSite reads the Site a lane route names, answering the request itself
|
||||||
|
// when it is not a registry Site. The path value is client-supplied, so it
|
||||||
|
// is checked against the registry before it reaches the store.
|
||||||
|
func laneSite(w http.ResponseWriter, r *http.Request) (string, bool) {
|
||||||
|
site := r.PathValue("site")
|
||||||
|
if !slices.Contains(latest.SiteNames(), site) {
|
||||||
|
http.Error(w, "unknown site", http.StatusBadRequest)
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return site, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminLanePause writes a bounded pause for one Site and answers with the
|
||||||
|
// freshly rendered Lanes block, so the figures describe the state after the
|
||||||
|
// press. The pause is a fact about the Site — the Lane's next pass reads it
|
||||||
|
// from the durable row, never from this process — so it survives a restart.
|
||||||
|
// The owner gate is the route's, not this handler's; the body is capped like
|
||||||
|
// the API path caps its bodies; the Site and the duration are validated
|
||||||
|
// here, before the store sees them (issue #147).
|
||||||
|
func (h *Handler) adminLanePause(w http.ResponseWriter, r *http.Request) {
|
||||||
|
site, ok := laneSite(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
|
||||||
|
if err := r.ParseForm(); err != nil {
|
||||||
|
http.Error(w, "invalid form", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
d, ok := pauseDurations[r.PostFormValue("duration")]
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "unknown pause duration", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.store.PauseLane(site, time.Now().Add(d).UnixMilli()); err != nil {
|
||||||
|
log.Printf("pause lane %s: %v", site, err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.render(w, http.StatusOK, "lanes", h.lanesView())
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminLaneResume zeroes one Site's pause and answers with the freshly
|
||||||
|
// rendered Lanes block. Resume is the reversal of a bounded pause, so it
|
||||||
|
// fires instantly with no confirm row (issue #147).
|
||||||
|
func (h *Handler) adminLaneResume(w http.ResponseWriter, r *http.Request) {
|
||||||
|
site, ok := laneSite(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
|
||||||
|
if err := r.ParseForm(); err != nil {
|
||||||
|
http.Error(w, "invalid form", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.store.ResumeLane(site); err != nil {
|
||||||
|
log.Printf("resume lane %s: %v", site, err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.render(w, http.StatusOK, "lanes", h.lanesView())
|
||||||
|
}
|
||||||
|
|
||||||
|
// lanesView builds the Lane status block from the durable pass log. Both
|
||||||
|
// reads are the store's latest-per-Site projection, so the page's seam is a
|
||||||
|
// seeded row rather than a fake poller; errors degrade to the empty state and
|
||||||
|
// are logged, never shown to the owner in detail.
|
||||||
|
func (h *Handler) lanesView() lanesView {
|
||||||
|
v := lanesView{
|
||||||
|
PollerOff: !h.pollerEnabled,
|
||||||
|
BrowserConfigured: h.browserConfigured,
|
||||||
|
}
|
||||||
|
passes, err := h.store.LatestLanePasses()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("admin lanes: latest passes: %v", err)
|
||||||
|
// No evidence of a lost sidecar reads as reachable, per the same rule
|
||||||
|
// browserReachable applies: a store failure must not condemn the
|
||||||
|
// browser. The empty table already says no Lane has recorded a pass.
|
||||||
|
v.BrowserReachable = true
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
outcomes, err := h.store.LanePassOutcomes(now.Add(-ownerWindow).UnixMilli())
|
||||||
|
if err != nil {
|
||||||
|
// The rows are complete without the chips, so a failed outcome sum
|
||||||
|
// must not blank the table into "no data yet" — that is the confident
|
||||||
|
// wrong statement the page exists to avoid. Every row renders "none
|
||||||
|
// observed" instead, which is honest.
|
||||||
|
log.Printf("admin lanes: outcomes: %v", err)
|
||||||
|
outcomes = nil
|
||||||
|
}
|
||||||
|
bySite := make(map[string]store.SiteOutcomes, len(outcomes))
|
||||||
|
for _, o := range outcomes {
|
||||||
|
bySite[o.Site] = o
|
||||||
|
}
|
||||||
|
v.Rows = make([]laneRow, 0, len(passes))
|
||||||
|
v.BrowserReachable = browserReachable(passes, now)
|
||||||
|
for _, p := range passes {
|
||||||
|
v.Rows = append(v.Rows, buildLaneRow(p, bySite[p.Site], now))
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// browserReachable derives the sidecar's reachability from the pass log: a
|
||||||
|
// browser Site is down when its latest pass inside the refusal backoff is a
|
||||||
|
// sidecar loss, a missing fetcher, or an interrupted read. Only browser Sites
|
||||||
|
// ever produce those signals, so no Site registry leaks into the web layer.
|
||||||
|
// A configured browser with no such evidence reads as reachable; an unset
|
||||||
|
// BROWSER_WS_URL degrades identically to a browser that is down.
|
||||||
|
func browserReachable(passes []store.LanePass, now time.Time) bool {
|
||||||
|
backoff := latest.RefuseBackoff
|
||||||
|
for _, p := range passes {
|
||||||
|
ran := time.UnixMilli(p.RanAt)
|
||||||
|
if now.Sub(ran) >= backoff || ran.After(now) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if p.Skip == latest.SkipSidecarDown || p.Skip == latest.SkipNoFetcher || p.Unreachable > 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildLaneRow turns one Site's latest pass and window outcome sums into the
|
||||||
|
// row the template prints. The skip column is the authority on why a pass did
|
||||||
|
// nothing; the outcomes render named and unlinked, because the pass row holds
|
||||||
|
// counts and never identities.
|
||||||
|
func buildLaneRow(p store.LanePass, o store.SiteOutcomes, now time.Time) laneRow {
|
||||||
|
row := laneRow{
|
||||||
|
Site: p.Site,
|
||||||
|
Due: p.Due,
|
||||||
|
Checked: p.Checked,
|
||||||
|
Gap: "—",
|
||||||
|
Ran: since(now, time.UnixMilli(p.RanAt)),
|
||||||
|
}
|
||||||
|
if p.GapMS > 0 {
|
||||||
|
row.Gap = (time.Duration(p.GapMS) * time.Millisecond).Truncate(time.Second).String()
|
||||||
|
}
|
||||||
|
row.Chips = outcomeChips(o)
|
||||||
|
row.HasChips = len(row.Chips) > 0
|
||||||
|
row.StatePhrase, row.StateGood, row.Attention = laneState(p, now)
|
||||||
|
row.Paused = time.UnixMilli(p.PausedUntil).After(now)
|
||||||
|
return row
|
||||||
|
}
|
||||||
|
|
||||||
|
// outcomeChips lists a Site's nonzero window sums in the taxonomy's fixed
|
||||||
|
// order, so the chips never reorder as the window changes. None observed is
|
||||||
|
// written by the template, not drawn as a confident zero count.
|
||||||
|
func outcomeChips(o store.SiteOutcomes) []chip {
|
||||||
|
fixed := []struct {
|
||||||
|
name string
|
||||||
|
count int
|
||||||
|
}{
|
||||||
|
{"refused", o.Refused},
|
||||||
|
{"unreachable", o.Unreachable},
|
||||||
|
{"no chapter", o.NoChapter},
|
||||||
|
{"unfetchable", o.Unfetchable},
|
||||||
|
{"errors", o.Errors},
|
||||||
|
}
|
||||||
|
var out []chip
|
||||||
|
for _, f := range fixed {
|
||||||
|
if f.count > 0 {
|
||||||
|
out = append(out, chip{Name: f.name, Count: f.count})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// laneState renders the reason a Lane's last pass did nothing, in one sentence
|
||||||
|
// per skip value with the one true stall kept apart from every Lane that
|
||||||
|
// declined and said why. Good states — a pause, a sleeping browser, nothing
|
||||||
|
// eligible — carry no Attention: the mark must stay spendable on the faults
|
||||||
|
// that actually need the owner.
|
||||||
|
func laneState(p store.LanePass, now time.Time) (phrase string, good, attention bool) {
|
||||||
|
// The pause phrase reads the live poll_lanes stamp the pass row joins
|
||||||
|
// on, not the pass's skip: the owner's press must render as paused on
|
||||||
|
// the very answer it gets, with no pass having run to record it. The
|
||||||
|
// pause is a fact about the Site, and the join delivers it (issue #147).
|
||||||
|
if pausedUntil := time.UnixMilli(p.PausedUntil); pausedUntil.After(now) {
|
||||||
|
phrase = "paused · resumes in " + humanDuration(pausedUntil.Sub(now))
|
||||||
|
good = true
|
||||||
|
return phrase, good, attention
|
||||||
|
}
|
||||||
|
switch p.Skip {
|
||||||
|
case latest.SkipPaused:
|
||||||
|
// A paused pass whose stamp has since lapsed: the Lane still
|
||||||
|
// declined with a reason, so it is never the one true stall.
|
||||||
|
phrase = "paused · resumes in " + humanDuration(time.UnixMilli(p.PausedUntil).Sub(now))
|
||||||
|
good = true
|
||||||
|
case latest.SkipRefusing:
|
||||||
|
phrase = "refusing"
|
||||||
|
if until := time.UnixMilli(p.RefuseUntil); until.After(now) {
|
||||||
|
phrase += " · backs off until " + until.Format("15:04")
|
||||||
|
}
|
||||||
|
attention = true
|
||||||
|
case latest.SkipSidecarDown, latest.SkipNoFetcher:
|
||||||
|
// Known false positive shipped per spec: a sibling Lane's Chrome loss
|
||||||
|
// stamps this Site too, and the enum deliberately has no tenth value
|
||||||
|
// to separate it (issue #141). Render it as written.
|
||||||
|
phrase = "no browser"
|
||||||
|
attention = true
|
||||||
|
case latest.SkipAsleep:
|
||||||
|
phrase = "browser asleep"
|
||||||
|
good = true
|
||||||
|
case latest.SkipDueQuery:
|
||||||
|
phrase = "due query failed"
|
||||||
|
attention = true
|
||||||
|
case latest.SkipEligibleCount:
|
||||||
|
phrase = "eligible count failed"
|
||||||
|
attention = true
|
||||||
|
case latest.SkipNothingEligible:
|
||||||
|
phrase = "nothing eligible"
|
||||||
|
good = true
|
||||||
|
}
|
||||||
|
if phrase == "" && p.Due > 0 && p.Checked == 0 {
|
||||||
|
// The one true stall: the pass reached its loop, Series were waiting,
|
||||||
|
// and none were read. Every skip above is a Lane that said why.
|
||||||
|
phrase = "not checking"
|
||||||
|
attention = true
|
||||||
|
}
|
||||||
|
return phrase, good, attention
|
||||||
|
}
|
||||||
|
|
||||||
|
// humanDuration renders a positive duration compactly for a "resumes in" clue
|
||||||
|
// at the pause and refusal scales — minutes under an hour, then h and h+m.
|
||||||
|
func humanDuration(d time.Duration) string {
|
||||||
|
d = d.Round(time.Minute)
|
||||||
|
if d <= 0 {
|
||||||
|
return "soon"
|
||||||
|
}
|
||||||
|
if d < time.Hour {
|
||||||
|
return fmt.Sprintf("%dm", int(d/time.Minute))
|
||||||
|
}
|
||||||
|
h := int(d / time.Hour)
|
||||||
|
if m := int(d%time.Hour) / int(time.Minute); m == 0 {
|
||||||
|
return fmt.Sprintf("%dh", h)
|
||||||
|
} else {
|
||||||
|
return fmt.Sprintf("%dh%dm", h, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// since formats how long ago a Lane last ran, at second resolution: the block
|
||||||
|
// refreshes every thirty seconds, so anything finer is noise the owner would
|
||||||
|
// have to ignore.
|
||||||
|
func since(now, then time.Time) string {
|
||||||
|
d := now.Sub(then).Truncate(time.Second)
|
||||||
|
if d < time.Second {
|
||||||
|
return "just now"
|
||||||
|
}
|
||||||
|
return d.String() + " ago"
|
||||||
|
}
|
||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// overviewView is the Overview landing page's data: one verdict line, the
|
||||||
|
// hygiene and library stats blocks, and the per-Site library shape table.
|
||||||
|
// Every judgement — the verdict state, which figures link, what a Lane's
|
||||||
|
// state means — is made here; the template only prints.
|
||||||
|
type overviewView struct {
|
||||||
|
// Verdict is the attention phrase that leads the page.
|
||||||
|
Verdict string
|
||||||
|
// HasCounts is false on a virgin pass log: the waiting figure would be a
|
||||||
|
// confident zero, and "nothing has happened" must not render as health.
|
||||||
|
HasCounts bool
|
||||||
|
// Waiting is the sum of Due over the latest pass per Site.
|
||||||
|
Waiting int
|
||||||
|
// Unchecked is the number of Series not checked in the window, computed
|
||||||
|
// as stale + never_checked: a never-checked Series is already counted on
|
||||||
|
// its own filter, and the verdict wants the inclusive number.
|
||||||
|
Unchecked int
|
||||||
|
// Hygiene is the seven problem filters in the Series list's own render
|
||||||
|
// order; Library is the library split plus the roster. Every figure is a
|
||||||
|
// door into the list that counts it, except a zero.
|
||||||
|
Hygiene []fig
|
||||||
|
Library []fig
|
||||||
|
// Sites is the per-Site library shape table, one row per Site with any
|
||||||
|
// Series, in the store's Site order.
|
||||||
|
Sites []siteRow
|
||||||
|
}
|
||||||
|
|
||||||
|
// fig is one stats figure: its label, the list it counts, and the count
|
||||||
|
// itself. Href empty means the count is zero: a measured zero is a real
|
||||||
|
// figure that stays on the page, but it is not a door, because following it
|
||||||
|
// lands on an empty list.
|
||||||
|
type fig struct {
|
||||||
|
Label string
|
||||||
|
Href string
|
||||||
|
Count int
|
||||||
|
}
|
||||||
|
|
||||||
|
// siteRow is one Site's share of the library: the Series total and the three
|
||||||
|
// hygiene counts the per-Site table carries, each a door to the list narrowed
|
||||||
|
// to that Site, plus the Lane state phrase derived from its latest pass. The
|
||||||
|
// table is library shape only — the Poll outcome sums live on the Lanes page.
|
||||||
|
type siteRow struct {
|
||||||
|
Site string
|
||||||
|
SiteHref string
|
||||||
|
Figs []fig
|
||||||
|
// State is the Lane's own sentence; "" means the last pass read normally.
|
||||||
|
// StateGood / StateBad pick the ok / bad second class.
|
||||||
|
State string
|
||||||
|
StateGood bool
|
||||||
|
StateBad bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// overviewView assembles the landing page from the store's read model: one
|
||||||
|
// SeriesShapes pass per filter summed in Go (the shipped surface offers eight
|
||||||
|
// grouped passes, not a stats query — #140), the pass log's latest pass per
|
||||||
|
// Site, and the roster. A failure in any read is a 500 with a logged reason,
|
||||||
|
// never a page of silent zeroes.
|
||||||
|
func (h *Handler) overviewView() (overviewView, error) {
|
||||||
|
now := time.Now()
|
||||||
|
cutoff := now.Add(-ownerWindow).UnixMilli()
|
||||||
|
|
||||||
|
shapes := make(map[string][]store.SiteSeriesShape, len(seriesFilterOrder))
|
||||||
|
totals := make(map[string]int, len(seriesFilterOrder))
|
||||||
|
for _, name := range seriesFilterOrder {
|
||||||
|
rows, err := h.store.SeriesShapes(store.SeriesFilter{Name: name, Cutoff: cutoff})
|
||||||
|
if err != nil {
|
||||||
|
return overviewView{}, err
|
||||||
|
}
|
||||||
|
shapes[name] = rows
|
||||||
|
for _, sh := range rows {
|
||||||
|
totals[name] += sh.Total
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
passes, err := h.store.LatestLanePasses()
|
||||||
|
if err != nil {
|
||||||
|
return overviewView{}, err
|
||||||
|
}
|
||||||
|
readers, err := h.store.Readers()
|
||||||
|
if err != nil {
|
||||||
|
return overviewView{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
view := overviewView{Waiting: waiting(passes)}
|
||||||
|
view.Unchecked = totals[store.SeriesFilterStale] + totals[store.SeriesFilterNeverChecked]
|
||||||
|
view.Verdict, view.HasCounts = overviewVerdict(passes, now)
|
||||||
|
|
||||||
|
// The seven problem filters, in seriesFilterOrder's permanent-then-fixable
|
||||||
|
// order; the All filter's count belongs to the Library block, not to a
|
||||||
|
// "hygiene" figure.
|
||||||
|
hygiene := make([]fig, 0, len(seriesFilterOrder)-1)
|
||||||
|
for _, name := range seriesFilterOrder[1:] {
|
||||||
|
hygiene = append(hygiene, door(seriesFilterLabels[name], totals[name], seriesListHref(name, "", "", 0)))
|
||||||
|
}
|
||||||
|
view.Hygiene = hygiene
|
||||||
|
|
||||||
|
var manga, novel int
|
||||||
|
for _, sh := range shapes[store.SeriesFilterAll] {
|
||||||
|
manga += sh.Manga
|
||||||
|
novel += sh.Novel
|
||||||
|
}
|
||||||
|
view.Library = []fig{
|
||||||
|
door("Series", totals[store.SeriesFilterAll], seriesListHref("", "", "", 0)),
|
||||||
|
door("Manga", manga, seriesListHref("", "", store.KindManga, 0)),
|
||||||
|
door("Novels", novel, seriesListHref("", "", store.KindNovel, 0)),
|
||||||
|
door("Readers", len(readers), "/admin/readers"),
|
||||||
|
}
|
||||||
|
|
||||||
|
// One row per Site with any Series, from the All shapes; the hygiene
|
||||||
|
// counts come from the same per-Site projection so the table cannot
|
||||||
|
// disagree with the library-wide figures above it.
|
||||||
|
siteCounts := make(map[string]map[string]int, len(shapes))
|
||||||
|
for name, rows := range shapes {
|
||||||
|
m := make(map[string]int, len(rows))
|
||||||
|
for _, sh := range rows {
|
||||||
|
m[sh.Site] = sh.Total
|
||||||
|
}
|
||||||
|
siteCounts[name] = m
|
||||||
|
}
|
||||||
|
passBySite := make(map[string]store.LanePass, len(passes))
|
||||||
|
for _, p := range passes {
|
||||||
|
passBySite[p.Site] = p
|
||||||
|
}
|
||||||
|
view.Sites = make([]siteRow, 0, len(shapes[store.SeriesFilterAll]))
|
||||||
|
for _, sh := range shapes[store.SeriesFilterAll] {
|
||||||
|
row := siteRow{
|
||||||
|
Site: sh.Site,
|
||||||
|
SiteHref: seriesListHref("", sh.Site, "", 0),
|
||||||
|
Figs: []fig{
|
||||||
|
door("", sh.Total, seriesListHref("", sh.Site, "", 0)),
|
||||||
|
door("", siteCounts[store.SeriesFilterNoCover][sh.Site], seriesListHref(store.SeriesFilterNoCover, sh.Site, "", 0)),
|
||||||
|
door("", siteCounts[store.SeriesFilterNeverChecked][sh.Site], seriesListHref(store.SeriesFilterNeverChecked, sh.Site, "", 0)),
|
||||||
|
door("", siteCounts[store.SeriesFilterStale][sh.Site], seriesListHref(store.SeriesFilterStale, sh.Site, "", 0)),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if p, ok := passBySite[sh.Site]; ok {
|
||||||
|
row.State, row.StateGood, row.StateBad = laneState(p, now)
|
||||||
|
} else {
|
||||||
|
row.State = "no pass yet"
|
||||||
|
}
|
||||||
|
view.Sites = append(view.Sites, row)
|
||||||
|
}
|
||||||
|
return view, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// door is one figure with its door: the list that counts it. A measured zero
|
||||||
|
// is still a real figure, but the door closes — following it would land on an
|
||||||
|
// empty list. The count is written once so the figure and what it links to
|
||||||
|
// cannot drift apart.
|
||||||
|
func door(label string, count int, href string) fig {
|
||||||
|
if count == 0 {
|
||||||
|
href = ""
|
||||||
|
}
|
||||||
|
return fig{Label: label, Href: href, Count: count}
|
||||||
|
}
|
||||||
|
|
||||||
|
// overviewVerdict decides the landing page's one line from the latest pass
|
||||||
|
// per Site: no passes at all is "no Lane has reported yet" — never confident
|
||||||
|
// zeroes; otherwise the count of Lanes whose last pass needs the owner, or
|
||||||
|
// "all lanes healthy". The count comes from the same laneState judgement the
|
||||||
|
// Lanes page colours on, so the two pages cannot disagree on what a fault is.
|
||||||
|
func overviewVerdict(passes []store.LanePass, now time.Time) (phrase string, counts bool) {
|
||||||
|
if len(passes) == 0 {
|
||||||
|
return "no Lane has reported yet", false
|
||||||
|
}
|
||||||
|
attention := 0
|
||||||
|
for _, p := range passes {
|
||||||
|
if _, _, attn := laneState(p, now); attn {
|
||||||
|
attention++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if attention == 0 {
|
||||||
|
return "all lanes healthy", true
|
||||||
|
}
|
||||||
|
if attention == 1 {
|
||||||
|
return "1 lane needs a look", true
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%d lanes need a look", attention), true
|
||||||
|
}
|
||||||
|
|
||||||
|
// waiting sums Due over the latest pass per Site: how many Series the Lanes
|
||||||
|
// found waiting, from the durable log rather than a running poller.
|
||||||
|
func waiting(passes []store.LanePass) int {
|
||||||
|
n := 0
|
||||||
|
for _, p := range passes {
|
||||||
|
n += p.Due
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
@@ -0,0 +1,398 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/latest"
|
||||||
|
"bookmarkmanager/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// seriesPageSize matches the store's row-read page length: the pager's range
|
||||||
|
// must agree with the LIMIT the store applies or the "of N" figure describes
|
||||||
|
// the wrong page. The store does not export it (#140).
|
||||||
|
const seriesPageSize = 50
|
||||||
|
|
||||||
|
// seriesFilterLabels names every hygiene filter for the Series list select,
|
||||||
|
// keyed by the wire constant the URL carries. The render order is
|
||||||
|
// seriesFilterOrder; the labels are read by later admin tickets too, so the
|
||||||
|
// map and the constants cannot drift apart.
|
||||||
|
var seriesFilterLabels = map[string]string{
|
||||||
|
store.SeriesFilterAll: "All series",
|
||||||
|
store.SeriesFilterNoURL: "No series URL",
|
||||||
|
store.SeriesFilterNoChapter: "Never read a chapter",
|
||||||
|
store.SeriesFilterNoReaders: "No Readers",
|
||||||
|
store.SeriesFilterNeverChecked: "Never checked",
|
||||||
|
store.SeriesFilterStale: "Not checked in 12h",
|
||||||
|
store.SeriesFilterNoCover: "No cover",
|
||||||
|
store.SeriesFilterReaderReport: "Latest from a Reader",
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesFilterOrder is the select's render order: All first, then the
|
||||||
|
// permanent repairs, then the fixable ones (issue #140).
|
||||||
|
var seriesFilterOrder = []string{
|
||||||
|
store.SeriesFilterAll,
|
||||||
|
store.SeriesFilterNoURL,
|
||||||
|
store.SeriesFilterNoChapter,
|
||||||
|
store.SeriesFilterNoReaders,
|
||||||
|
store.SeriesFilterNeverChecked,
|
||||||
|
store.SeriesFilterStale,
|
||||||
|
store.SeriesFilterNoCover,
|
||||||
|
store.SeriesFilterReaderReport,
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesListView is the Series list page's data. The template renders strings
|
||||||
|
// and flags, and every judgement about what a value means is made here.
|
||||||
|
type seriesListView struct {
|
||||||
|
Filters []seriesFilterOption
|
||||||
|
Sites []string
|
||||||
|
Site string // "" = every Site
|
||||||
|
Kind string // "" = both libraries
|
||||||
|
FilterLabel string
|
||||||
|
Rows []seriesRowView
|
||||||
|
Total int
|
||||||
|
// KindBoth / KindManga / KindNovel are the Library segment links, and
|
||||||
|
// PrevHref / NextHref the pager's, all carrying the active filter, Site
|
||||||
|
// and Kind so narrowing never drops state.
|
||||||
|
KindBoth string
|
||||||
|
KindManga string
|
||||||
|
KindNovel string
|
||||||
|
PrevHref string
|
||||||
|
NextHref string
|
||||||
|
Range string
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesFilterOption is one entry of the Show select: its wire value, its
|
||||||
|
// rendered label with the library-wide count, and whether it is the active
|
||||||
|
// filter.
|
||||||
|
type seriesFilterOption struct {
|
||||||
|
Name string
|
||||||
|
Label string
|
||||||
|
Count int
|
||||||
|
Selected bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesRowView is one Series row formatted for the template. Band carries
|
||||||
|
// the alternating row tint by class rather than nth-of-type, so the confirm
|
||||||
|
// rows later tickets add are row siblings without breaking the alternation.
|
||||||
|
// Attention tints the title patina: a row with any hygiene chip needs one.
|
||||||
|
//
|
||||||
|
// CanPoll is the Check now control's visibility: absent on a Series with no
|
||||||
|
// page to fetch and on an orphan, so the owner is never offered a button that
|
||||||
|
// can never do anything. Pending is derived — the request stamp is newer than
|
||||||
|
// the check stamp — and Requested is its ageing label.
|
||||||
|
type seriesRowView struct {
|
||||||
|
Key string
|
||||||
|
Title string
|
||||||
|
Site string
|
||||||
|
Ch string // chapter number; "—" until first captured
|
||||||
|
Age string // checked age; "never" until first check
|
||||||
|
Readers int
|
||||||
|
Notes []string // chips, capped at two
|
||||||
|
More int // chips past the cap, rendered as a +N tail
|
||||||
|
Band bool
|
||||||
|
Attention bool
|
||||||
|
CanPoll bool
|
||||||
|
Pending bool
|
||||||
|
Requested string // "requested 3m ago", rendered only while pending
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminSeries renders the filterable, bookmarkable Series list: filter, Site,
|
||||||
|
// Library and page all live in the query string, so the list's state is an
|
||||||
|
// address rather than a click path.
|
||||||
|
func (h *Handler) adminSeries(w http.ResponseWriter, r *http.Request) {
|
||||||
|
view, err := h.seriesListView(r)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("admin series: %v", err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.renderAdmin(w, adminView{Page: "series", SeriesList: view})
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminSeriesPoll is the Check now action: it stamps the Series' force_poll_at
|
||||||
|
// and answers with the freshly rendered row, so the figures describe the
|
||||||
|
// state after the press. The control never commands the poller — the request
|
||||||
|
// is a fact about the Series, and the Lane's next pass reads it through
|
||||||
|
// DueForLatestCheck (ADR-0013). The owner gate is the route's, not this
|
||||||
|
// handler's; the body is capped like the API path caps its bodies; the key is
|
||||||
|
// validated here — a malformed key is a 400 and an unknown one a 404.
|
||||||
|
func (h *Handler) adminSeriesPoll(w http.ResponseWriter, r *http.Request) {
|
||||||
|
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
|
||||||
|
if !ok || site == "" || seriesID == "" {
|
||||||
|
http.Error(w, "bad series key", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
|
||||||
|
if err := r.ParseForm(); err != nil {
|
||||||
|
http.Error(w, "invalid form", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
|
||||||
|
log.Printf("series poll %s: %v", site+":"+seriesID, err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
} else if !found {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.store.ForceSeriesPoll(site, seriesID, time.Now().UnixMilli()); err != nil {
|
||||||
|
log.Printf("series poll %s: %v", site+":"+seriesID, err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Re-read after the stamp: the answer must describe the state after the
|
||||||
|
// press. The detail page's control swaps its meta in place and the list
|
||||||
|
// row's swaps the row; htmx names an id target in HX-Target, so the
|
||||||
|
// response matches the surface it came from. The row's band parity travels
|
||||||
|
// with the press (hx-vals), so the swap keeps the zebra alternation.
|
||||||
|
a, found, err := h.adminSeriesByKey(site, seriesID)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("series poll %s: %v", site+":"+seriesID, err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if r.Header.Get("HX-Target") == "detail-meta" {
|
||||||
|
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
band := 0
|
||||||
|
if r.PostFormValue("band") == "1" {
|
||||||
|
band = 1
|
||||||
|
}
|
||||||
|
h.render(w, http.StatusOK, "series-row", seriesRow(a, band, time.Now()))
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesListView assembles one Series list view from the request's query
|
||||||
|
// string. An unknown filter value is the absent All case, never an error: the
|
||||||
|
// select's options are not the only way this URL can be reached.
|
||||||
|
func (h *Handler) seriesListView(r *http.Request) (seriesListView, error) {
|
||||||
|
q := r.URL.Query()
|
||||||
|
filter := q.Get("filter")
|
||||||
|
if _, ok := seriesFilterLabels[filter]; !ok {
|
||||||
|
filter = store.SeriesFilterAll
|
||||||
|
}
|
||||||
|
site := q.Get("site")
|
||||||
|
kind := q.Get("kind")
|
||||||
|
if kind != store.KindManga && kind != store.KindNovel {
|
||||||
|
kind = ""
|
||||||
|
}
|
||||||
|
page := 1
|
||||||
|
if p, err := strconv.Atoi(q.Get("page")); err == nil && p > 1 {
|
||||||
|
page = p
|
||||||
|
}
|
||||||
|
|
||||||
|
sf := store.SeriesFilter{
|
||||||
|
Site: site,
|
||||||
|
Kind: kind,
|
||||||
|
Name: filter,
|
||||||
|
Cutoff: time.Now().Add(-ownerWindow).UnixMilli(),
|
||||||
|
Page: page,
|
||||||
|
}
|
||||||
|
data, err := h.store.SeriesPage(sf)
|
||||||
|
if err != nil {
|
||||||
|
return seriesListView{}, err
|
||||||
|
}
|
||||||
|
// A page past the end is not an empty list: the store's window count runs
|
||||||
|
// over the rows the result set carries, so an overflow page reports zero
|
||||||
|
// rows and zero total, and the list re-reads at page 1 to know the truth.
|
||||||
|
if len(data.Rows) == 0 && page > 1 {
|
||||||
|
page = 1
|
||||||
|
sf.Page = 1
|
||||||
|
data, err = h.store.SeriesPage(sf)
|
||||||
|
if err != nil {
|
||||||
|
return seriesListView{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
view := seriesListView{
|
||||||
|
Site: site,
|
||||||
|
Kind: kind,
|
||||||
|
FilterLabel: seriesFilterLabels[filter],
|
||||||
|
Rows: make([]seriesRowView, 0, len(data.Rows)),
|
||||||
|
Total: data.Total,
|
||||||
|
Sites: latest.SiteNames(),
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
for i, a := range data.Rows {
|
||||||
|
view.Rows = append(view.Rows, seriesRow(a, i, now))
|
||||||
|
}
|
||||||
|
view.Filters, err = h.seriesFilterOptions(filter, sf.Cutoff)
|
||||||
|
if err != nil {
|
||||||
|
return seriesListView{}, err
|
||||||
|
}
|
||||||
|
view.KindBoth = seriesListHref(filter, site, "", 0)
|
||||||
|
view.KindManga = seriesListHref(filter, site, store.KindManga, 0)
|
||||||
|
view.KindNovel = seriesListHref(filter, site, store.KindNovel, 0)
|
||||||
|
if page > 1 {
|
||||||
|
view.PrevHref = seriesListHref(filter, site, kind, page-1)
|
||||||
|
}
|
||||||
|
if last := (data.Total + seriesPageSize - 1) / seriesPageSize; page < last {
|
||||||
|
view.NextHref = seriesListHref(filter, site, kind, page+1)
|
||||||
|
}
|
||||||
|
view.Range = pagerRange(data.Total, len(data.Rows), page)
|
||||||
|
return view, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesFilterOptions renders every hygiene filter with its library-wide
|
||||||
|
// count, one SeriesShapes pass per filter summed in Go — the shipped surface
|
||||||
|
// offers eight grouped passes, not a single stats query (#140). The counts
|
||||||
|
// are library-wide because the select sits next to the Site narrowing and
|
||||||
|
// must not shift as the owner narrows the list itself. Cutoff travels with
|
||||||
|
// the stale filter, or its count would always be zero.
|
||||||
|
func (h *Handler) seriesFilterOptions(selected string, cutoff int64) ([]seriesFilterOption, error) {
|
||||||
|
out := make([]seriesFilterOption, 0, len(seriesFilterOrder))
|
||||||
|
for _, name := range seriesFilterOrder {
|
||||||
|
shapes, err := h.store.SeriesShapes(store.SeriesFilter{Name: name, Cutoff: cutoff})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
count := 0
|
||||||
|
for _, sh := range shapes {
|
||||||
|
count += sh.Total
|
||||||
|
}
|
||||||
|
out = append(out, seriesFilterOption{
|
||||||
|
Name: name,
|
||||||
|
Label: seriesFilterLabels[name],
|
||||||
|
Count: count,
|
||||||
|
Selected: name == selected,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesRow shapes one store row for the template, capping its chips at two
|
||||||
|
// plus a +N tail; attention marks a row that carries any.
|
||||||
|
// pollState derives the Check now control and the pending marker (issue
|
||||||
|
// #146), shared by the list row and the detail page: CanPoll is false on a
|
||||||
|
// Series with no page to fetch and on an orphan, so the owner is never
|
||||||
|
// offered a button that can never do anything. Pending is derived — the
|
||||||
|
// request stamp is newer than the check stamp — and requested is its ageing
|
||||||
|
// label, which never expires.
|
||||||
|
func pollState(a store.AdminSeries, now time.Time) (canPoll, pending bool, requested string) {
|
||||||
|
canPoll = a.SeriesURL != "" && a.ReaderCount > 0
|
||||||
|
if a.ForcePollAt > a.LatestCheckedAt {
|
||||||
|
pending = true
|
||||||
|
requested = requestedAge(now, a.ForcePollAt)
|
||||||
|
}
|
||||||
|
return canPoll, pending, requested
|
||||||
|
}
|
||||||
|
|
||||||
|
func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
|
||||||
|
canPoll, pending, requested := pollState(a, now)
|
||||||
|
row := seriesRowView{
|
||||||
|
Key: a.Key(),
|
||||||
|
Site: a.Site,
|
||||||
|
Title: a.Title,
|
||||||
|
Readers: a.ReaderCount,
|
||||||
|
Band: i%2 == 1,
|
||||||
|
CanPoll: canPoll,
|
||||||
|
Pending: pending,
|
||||||
|
Requested: requested,
|
||||||
|
}
|
||||||
|
if a.LatestChapterNum != nil {
|
||||||
|
row.Ch = strconv.FormatFloat(*a.LatestChapterNum, 'f', -1, 64)
|
||||||
|
} else {
|
||||||
|
row.Ch = "—"
|
||||||
|
}
|
||||||
|
row.Age = checkedAge(now, a.LatestCheckedAt)
|
||||||
|
notes := seriesNotes(a, now)
|
||||||
|
if n := len(notes); n > 2 {
|
||||||
|
row.Notes, row.More = notes[:2], n-2
|
||||||
|
} else {
|
||||||
|
row.Notes = notes
|
||||||
|
}
|
||||||
|
row.Attention = len(notes) > 0
|
||||||
|
return row
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesNotes are a row's hygiene chips in the design's order: no URL, no
|
||||||
|
// cover, orphan, stale, reader sighting.
|
||||||
|
func seriesNotes(a store.AdminSeries, now time.Time) []string {
|
||||||
|
notes := []string{}
|
||||||
|
if a.SeriesURL == "" {
|
||||||
|
notes = append(notes, "no URL")
|
||||||
|
}
|
||||||
|
if a.CoverAddress == "" {
|
||||||
|
notes = append(notes, "no cover")
|
||||||
|
}
|
||||||
|
if a.ReaderCount == 0 {
|
||||||
|
notes = append(notes, "orphan")
|
||||||
|
}
|
||||||
|
if a.LatestCheckedAt > 0 && a.LatestCheckedAt < now.Add(-ownerWindow).UnixMilli() {
|
||||||
|
notes = append(notes, "stale")
|
||||||
|
}
|
||||||
|
if a.RaisedByReader {
|
||||||
|
notes = append(notes, "reader sighting")
|
||||||
|
}
|
||||||
|
return notes
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkedAge formats how long ago a Series was last checked, at the
|
||||||
|
// granularity the list reads at — minutes, hours, days. Zero means never.
|
||||||
|
func checkedAge(now time.Time, ts int64) string {
|
||||||
|
if ts == 0 {
|
||||||
|
return "never"
|
||||||
|
}
|
||||||
|
d := now.Sub(time.UnixMilli(ts))
|
||||||
|
switch {
|
||||||
|
case d < time.Hour:
|
||||||
|
m := int(d / time.Minute)
|
||||||
|
if m < 1 {
|
||||||
|
m = 1
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%dm ago", m)
|
||||||
|
case d < 24*time.Hour:
|
||||||
|
return fmt.Sprintf("%dh ago", int(d/time.Hour))
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("%dd ago", int(d/(24*time.Hour)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestedAge is the pending marker's text: how long ago the owner asked,
|
||||||
|
// and nothing about when the request will run — the page does not know when a
|
||||||
|
// sleeping browser will wake (issue #146). An unanswered request ages forever;
|
||||||
|
// there is no expiry.
|
||||||
|
func requestedAge(now time.Time, ts int64) string {
|
||||||
|
return "requested " + checkedAge(now, ts)
|
||||||
|
}
|
||||||
|
|
||||||
|
// pagerRange is the pager's "1–50 of 120" line. The template renders the
|
||||||
|
// pager only over rows (the empty state replaces it), so it is never asked
|
||||||
|
// to describe an empty list.
|
||||||
|
func pagerRange(total, rows, page int) string {
|
||||||
|
from := (page-1)*seriesPageSize + 1
|
||||||
|
return fmt.Sprintf("%d–%d of %d", from, from+rows-1, total)
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesListHref is one Series list address carrying the filter, Site, Kind
|
||||||
|
// and page. The All filter and page 1 are the absent cases and stay out of
|
||||||
|
// the URL, so the default address is the shortest one.
|
||||||
|
func seriesListHref(filter, site, kind string, page int) string {
|
||||||
|
q := url.Values{}
|
||||||
|
if filter != "" && filter != store.SeriesFilterAll {
|
||||||
|
q.Set("filter", filter)
|
||||||
|
}
|
||||||
|
if site != "" {
|
||||||
|
q.Set("site", site)
|
||||||
|
}
|
||||||
|
if kind != "" {
|
||||||
|
q.Set("kind", kind)
|
||||||
|
}
|
||||||
|
if page > 1 {
|
||||||
|
q.Set("page", strconv.Itoa(page))
|
||||||
|
}
|
||||||
|
if len(q) == 0 {
|
||||||
|
return "/admin/series"
|
||||||
|
}
|
||||||
|
return "/admin/series?" + q.Encode()
|
||||||
|
}
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// seriesDetailView is one Series' page as the owner sees it: strings and
|
||||||
|
// flags, every judgement made here, the template left to print. ReaderCount
|
||||||
|
// is the only figure that crosses the privacy boundary — the owner learns how
|
||||||
|
// many Readers hold the Series, never which Reader reads what.
|
||||||
|
type seriesDetailView struct {
|
||||||
|
Key string // "<site>:<series_id>", the page's address and the Series' identity
|
||||||
|
Site string
|
||||||
|
Kind string
|
||||||
|
// Title, Cover and Chapter come from the shared Series row; the Cover is
|
||||||
|
// the wire URL of the stored bytes, "" before any exist.
|
||||||
|
Title string
|
||||||
|
Cover string
|
||||||
|
Chapter string // Latest Chapter number, or "—" before the first capture
|
||||||
|
Checked string // how long ago the poller last checked, or "never"
|
||||||
|
Readers int
|
||||||
|
|
||||||
|
// Marks, one per hygiene fact, rendered only while it holds.
|
||||||
|
Unpollable bool // no SeriesURL to fetch
|
||||||
|
NoCover bool
|
||||||
|
Orphan bool // no Reader holds the Series
|
||||||
|
SightingRaised bool // a Reader's Sighting set the Latest Chapter
|
||||||
|
|
||||||
|
// Poll is the Check now control and the pending marker (issue #146): the
|
||||||
|
// same derivation and visibility as the list row. CanPoll is false on a
|
||||||
|
// Series with no page to fetch and on an orphan; Pending is derived —
|
||||||
|
// the request stamp is newer than the check stamp — and Requested is its
|
||||||
|
// ageing label.
|
||||||
|
CanPoll bool
|
||||||
|
Pending bool
|
||||||
|
Requested string
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminSeriesDetail renders one Series' page, keyed by the composite
|
||||||
|
// "<site>:<series_id>" the list row already shows. The row is read through
|
||||||
|
// the list's own SeriesPage read narrowed to the key's Site: the admin
|
||||||
|
// projection is the privacy boundary, and a dedicated single-row read would
|
||||||
|
// be a second definition of it.
|
||||||
|
func (h *Handler) adminSeriesDetail(w http.ResponseWriter, r *http.Request) {
|
||||||
|
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
|
||||||
|
if !ok || site == "" || seriesID == "" {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a, found, err := h.adminSeriesByKey(site, seriesID)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("series detail %s: %v", site+":"+seriesID, err)
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.renderAdmin(w, adminView{Page: "series-detail", Detail: h.seriesDetailView(a)})
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminSeriesByKey reads one Series through the list's own SeriesPage read
|
||||||
|
// narrowed to the key's Site: the admin projection is the privacy boundary,
|
||||||
|
// and a dedicated single-row read would be a second definition of it. Absence
|
||||||
|
// is reported with found=false, never an error.
|
||||||
|
// ponytail: a page scan per keyed read, one query per page of the Site's rows
|
||||||
|
// up to the window total; a keyed read alongside SeriesPage when the library
|
||||||
|
// outgrows the page size.
|
||||||
|
func (h *Handler) adminSeriesByKey(site, seriesID string) (store.AdminSeries, bool, error) {
|
||||||
|
seen := 0
|
||||||
|
for page := 1; ; page++ {
|
||||||
|
p, err := h.store.SeriesPage(store.SeriesFilter{Site: site, Page: page})
|
||||||
|
if err != nil {
|
||||||
|
return store.AdminSeries{}, false, err
|
||||||
|
}
|
||||||
|
seen += len(p.Rows)
|
||||||
|
for i := range p.Rows {
|
||||||
|
if p.Rows[i].SeriesID == seriesID {
|
||||||
|
return p.Rows[i], true, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if seen >= p.Total {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return store.AdminSeries{}, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// seriesDetailView shapes one AdminSeries row for display: every judgement in
|
||||||
|
// Go, the template left to print strings and flags.
|
||||||
|
func (h *Handler) seriesDetailView(a store.AdminSeries) seriesDetailView {
|
||||||
|
canPoll, pending, requested := pollState(a, time.Now())
|
||||||
|
v := seriesDetailView{
|
||||||
|
Key: a.Key(),
|
||||||
|
Site: a.Site,
|
||||||
|
Kind: a.Kind,
|
||||||
|
Title: a.Title,
|
||||||
|
Cover: h.store.CoverWireURL(a.CoverAddress),
|
||||||
|
Readers: a.ReaderCount,
|
||||||
|
Unpollable: a.SeriesURL == "",
|
||||||
|
NoCover: a.CoverAddress == "",
|
||||||
|
Orphan: a.ReaderCount == 0,
|
||||||
|
SightingRaised: a.RaisedByReader,
|
||||||
|
CanPoll: canPoll,
|
||||||
|
Pending: pending,
|
||||||
|
Requested: requested,
|
||||||
|
}
|
||||||
|
if a.LatestChapterNum == nil {
|
||||||
|
v.Chapter = "—"
|
||||||
|
} else {
|
||||||
|
v.Chapter = strconv.FormatFloat(*a.LatestChapterNum, 'f', -1, 64)
|
||||||
|
}
|
||||||
|
if a.LatestCheckedAt == 0 {
|
||||||
|
v.Checked = "never"
|
||||||
|
} else {
|
||||||
|
v.Checked = since(time.Now(), time.UnixMilli(a.LatestCheckedAt))
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
@@ -0,0 +1,830 @@
|
|||||||
|
:root {
|
||||||
|
--measure-wide: 1080px;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-color-scheme: light) {
|
||||||
|
:root {
|
||||||
|
--measure-wide: 1080px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet {
|
||||||
|
max-width: var(--measure-wide);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .brand em {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .brand .mark > g > g:last-child {
|
||||||
|
stroke: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.topbar-actions {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 18px;
|
||||||
|
margin-left: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.navrow {
|
||||||
|
display: flex;
|
||||||
|
gap: 18px;
|
||||||
|
padding: 2px 20px 0;
|
||||||
|
overflow-x: auto;
|
||||||
|
overflow-y: hidden;
|
||||||
|
scrollbar-width: none;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
.navrow::-webkit-scrollbar {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.navrow a {
|
||||||
|
flex: none;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
min-width: 44px;
|
||||||
|
padding: 8px 0 12px;
|
||||||
|
color: var(--mute);
|
||||||
|
font: 400 17px var(--font-display);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.navrow a:hover {
|
||||||
|
color: var(--paper-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.navrow a.active {
|
||||||
|
color: var(--paper);
|
||||||
|
border-bottom: 2px solid var(--paper);
|
||||||
|
margin-bottom: -1px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-page {
|
||||||
|
padding: 0 20px 40px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-page > .sec,
|
||||||
|
.admin-sheet .readers h2,
|
||||||
|
.admin-sheet .lanes h2 {
|
||||||
|
position: relative;
|
||||||
|
margin: 0;
|
||||||
|
padding: 26px 0 8px;
|
||||||
|
font: 500 11px/1 var(--font-mono);
|
||||||
|
letter-spacing: .2em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
color: var(--mute-2);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-page > .sec::before,
|
||||||
|
.admin-sheet .readers h2::before,
|
||||||
|
.admin-sheet .lanes h2::before {
|
||||||
|
content: "";
|
||||||
|
position: absolute;
|
||||||
|
top: 14px;
|
||||||
|
left: 0;
|
||||||
|
width: 34px;
|
||||||
|
height: 2px;
|
||||||
|
background: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .readers,
|
||||||
|
.admin-sheet .lanes {
|
||||||
|
margin: 0;
|
||||||
|
padding: 0 0 16px;
|
||||||
|
border-bottom: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .readerlist,
|
||||||
|
.admin-sheet .lanelist {
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
list-style: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .readerlist li,
|
||||||
|
.admin-sheet .lanelist li {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 4px 16px;
|
||||||
|
min-height: 48px;
|
||||||
|
padding: 10px 0;
|
||||||
|
border-top: 1px solid var(--rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-actions {
|
||||||
|
display: flex;
|
||||||
|
flex: 0 0 auto;
|
||||||
|
gap: 18px;
|
||||||
|
margin-left: auto;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-actions .ghost,
|
||||||
|
.admin-sheet .c-act .ghost {
|
||||||
|
font-size: 12px;
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-actions .ghost.danger,
|
||||||
|
.admin-sheet .c-act .ghost.danger {
|
||||||
|
color: var(--danger);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .readerlist form {
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .lane-browser {
|
||||||
|
padding: 12px 0 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .ghost.danger {
|
||||||
|
color: var(--danger);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .ghost.danger:hover {
|
||||||
|
color: var(--danger);
|
||||||
|
border-bottom-color: var(--danger);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-id {
|
||||||
|
font: 500 15px/1.5 var(--font-mono);
|
||||||
|
letter-spacing: .01em;
|
||||||
|
color: var(--paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-sessions,
|
||||||
|
.admin-sheet .reader-sightings,
|
||||||
|
.admin-sheet .reader-blocked,
|
||||||
|
.admin-sheet .lane-fact,
|
||||||
|
.admin-sheet .lane-mark {
|
||||||
|
font: 500 13px/1.4 var(--font-mono);
|
||||||
|
letter-spacing: .06em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-sessions {
|
||||||
|
color: var(--paper-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-sightings,
|
||||||
|
.admin-sheet .lane-fact {
|
||||||
|
color: var(--mute);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .reader-blocked,
|
||||||
|
.admin-sheet .lane-mark {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .lane-site {
|
||||||
|
font: 400 19px/1.2 var(--font-display);
|
||||||
|
color: var(--paper-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .lanelist li.attention .lane-site {
|
||||||
|
color: var(--danger);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* A single grid keeps row rules continuous; cell padding supplies gutters. */
|
||||||
|
.admin-sheet .sechead {
|
||||||
|
display: flex;
|
||||||
|
align-items: baseline;
|
||||||
|
justify-content: space-between;
|
||||||
|
gap: 20px;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .sechead .statusline {
|
||||||
|
padding: 0 0 8px;
|
||||||
|
font-size: 11px;
|
||||||
|
letter-spacing: .14em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .statusline {
|
||||||
|
margin: 0;
|
||||||
|
padding: 0 0 10px;
|
||||||
|
font: 500 12px/1 var(--font-mono);
|
||||||
|
letter-spacing: .04em;
|
||||||
|
color: var(--mute-2);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The verdict line is set in the data face, not the display face: it is
|
||||||
|
three counts, not a page title. The judgement is the only bright thing. */
|
||||||
|
.admin-sheet .verdict {
|
||||||
|
padding: 16px 0 12px;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
font: 500 15px/1.6 var(--font-mono);
|
||||||
|
letter-spacing: .04em;
|
||||||
|
color: var(--mute);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .verdict .attn {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .verdict .counts b {
|
||||||
|
color: var(--paper);
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: minmax(240px, 1fr) 156px 92px 110px 76px minmax(150px, 220px) 140px;
|
||||||
|
column-gap: 0;
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .thead {
|
||||||
|
display: contents;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .thead > * {
|
||||||
|
padding: 10px 14px 8px 0;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
font: 500 12px/1 var(--font-mono);
|
||||||
|
letter-spacing: .12em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
color: var(--mute-2);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow {
|
||||||
|
display: contents;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow > * {
|
||||||
|
padding: 11px 14px 11px 0;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
font: 500 15px/1.5 var(--font-mono);
|
||||||
|
letter-spacing: .01em;
|
||||||
|
color: var(--paper-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .thead > *:last-child,
|
||||||
|
.admin-sheet .tbl .trow > *:last-child {
|
||||||
|
padding-right: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-title,
|
||||||
|
.admin-sheet .tbl .c-site {
|
||||||
|
font: 400 18px/1.35 var(--font-display);
|
||||||
|
letter-spacing: 0;
|
||||||
|
color: var(--paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-title a:hover,
|
||||||
|
.admin-sheet .tbl .c-act .ghost:hover {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow.attention .c-title {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-ch,
|
||||||
|
.admin-sheet .tbl .c-rd {
|
||||||
|
text-align: right;
|
||||||
|
padding-right: 26px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow .c-note .mark {
|
||||||
|
margin-right: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-act {
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-act .ghost + .ghost {
|
||||||
|
margin-left: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow > .confirm-row {
|
||||||
|
grid-column: 1 / -1;
|
||||||
|
padding: 10px 12px;
|
||||||
|
border-bottom: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stats {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(232px, 1fr));
|
||||||
|
margin: 4px 0 0;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stat {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 16px;
|
||||||
|
padding: 11px 24px 11px 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stat .lbl {
|
||||||
|
font: 500 13px/1.3 var(--font-mono);
|
||||||
|
letter-spacing: .1em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
color: var(--mute-2);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stat .fig {
|
||||||
|
font: 500 19px/1 var(--font-mono);
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
color: var(--paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stat .fig.zero {
|
||||||
|
color: var(--mute);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stat a.fig {
|
||||||
|
/* The hover rule below sets the border on :hover only, which shifts the
|
||||||
|
row a pixel on hover; the transparent baseline holds the layout. */
|
||||||
|
border-bottom: 1px solid transparent;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .stat a.fig:hover {
|
||||||
|
color: var(--patina);
|
||||||
|
border-bottom: 1px solid var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .mark,
|
||||||
|
.admin-sheet .mark-faint {
|
||||||
|
font: 500 13px/1 var(--font-mono);
|
||||||
|
letter-spacing: .06em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .mark {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .mark-faint {
|
||||||
|
color: var(--mute-2);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .mark.mark-strong {
|
||||||
|
font-size: 13px;
|
||||||
|
letter-spacing: .14em;
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .mark.mark-strong::before {
|
||||||
|
content: "";
|
||||||
|
display: inline-block;
|
||||||
|
width: 7px;
|
||||||
|
height: 7px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: var(--patina);
|
||||||
|
margin-right: 8px;
|
||||||
|
vertical-align: .08em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .mark.bad {
|
||||||
|
color: var(--danger);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.sites {
|
||||||
|
grid-template-columns: 170px repeat(4, 106px) minmax(180px, 1fr);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.sites .c-site,
|
||||||
|
.admin-sheet .tbl.lanes .c-site {
|
||||||
|
font: 400 18px/1.35 var(--font-display);
|
||||||
|
letter-spacing: 0;
|
||||||
|
color: var(--paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.sites .c-state {
|
||||||
|
color: var(--patina);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.sites .c-state.bad,
|
||||||
|
.admin-sheet .tbl.lanes .c-skip .bad,
|
||||||
|
.admin-sheet .tbl.lanes .trow.attention .c-site {
|
||||||
|
color: var(--danger);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .c-skip .ok {
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.sites .thead > *:nth-child(n+2):nth-child(-n+5),
|
||||||
|
.admin-sheet .tbl.sites .trow > *:nth-child(n+2):nth-child(-n+5) {
|
||||||
|
padding-right: 0;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes {
|
||||||
|
grid-template-columns: 150px 62px 88px 66px 118px minmax(0, 1fr) 168px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .c-skip {
|
||||||
|
white-space: normal;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .c-skip > * {
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .c-ctrl {
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .thead > *:nth-child(n+2):nth-child(-n+4),
|
||||||
|
.admin-sheet .tbl.lanes .trow > *:nth-child(n+2):nth-child(-n+4) {
|
||||||
|
padding-right: 26px;
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series {
|
||||||
|
grid-template-columns: 150px 84px 104px 76px minmax(120px, 1fr) 212px;
|
||||||
|
row-gap: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .thead > *:first-child {
|
||||||
|
padding-left: 20px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .thead > *:last-child {
|
||||||
|
padding-right: 20px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The site cell's colour is a class, never an inline style: site is
|
||||||
|
client-supplied and unvalidated, and a hostile value reaching a CSS
|
||||||
|
context would render ZgotmplZ, while an unknown class degrades to the
|
||||||
|
unstyled cell. */
|
||||||
|
.admin-sheet .tbl .c-site.site-asura {
|
||||||
|
color: var(--asura);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-site.site-demonic {
|
||||||
|
color: var(--demonic);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-site.site-comix {
|
||||||
|
color: var(--comix);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-site.site-kagane {
|
||||||
|
color: var(--kagane);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-site.site-novelfull {
|
||||||
|
color: var(--novelfull);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-site.site-lightnovelworld {
|
||||||
|
color: var(--lightnovelworld);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .trow {
|
||||||
|
display: grid;
|
||||||
|
grid-column: 1 / -1;
|
||||||
|
grid-template-columns: subgrid;
|
||||||
|
padding: 13px 20px 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .trow.band {
|
||||||
|
background: var(--hover);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .trow > * {
|
||||||
|
padding: 0 14px 0 0;
|
||||||
|
border-bottom: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .c-title {
|
||||||
|
display: flex;
|
||||||
|
grid-column: 1 / -1;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 16px;
|
||||||
|
padding: 0 0 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .c-title .mark {
|
||||||
|
margin-left: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .filterbar {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
align-items: center;
|
||||||
|
gap: 14px 22px;
|
||||||
|
padding: 16px 0 14px;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .fsel {
|
||||||
|
display: flex;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .fsel > span {
|
||||||
|
font: 500 11px/1 var(--font-mono);
|
||||||
|
letter-spacing: .14em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
color: var(--mute-2);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .fsel select,
|
||||||
|
.admin-sheet .pausebar select {
|
||||||
|
color-scheme: dark;
|
||||||
|
background: var(--ink);
|
||||||
|
border: none;
|
||||||
|
border-bottom: 1px solid var(--field-line);
|
||||||
|
color: var(--paper);
|
||||||
|
font: 400 16px/1.4 var(--font-display);
|
||||||
|
padding: 4px 4px 5px 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .fsel select:hover,
|
||||||
|
.admin-sheet .pausebar select:hover {
|
||||||
|
border-bottom-color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .fsel select option,
|
||||||
|
.admin-sheet .pausebar select option {
|
||||||
|
background: var(--ink);
|
||||||
|
color: var(--paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .segrow {
|
||||||
|
display: inline-flex;
|
||||||
|
gap: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .segrow a {
|
||||||
|
padding: 6px 11px 7px;
|
||||||
|
border-bottom: 2px solid transparent;
|
||||||
|
color: var(--mute);
|
||||||
|
font: 500 14px/1 var(--font-mono);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .segrow a.active {
|
||||||
|
border-bottom-color: var(--patina);
|
||||||
|
color: var(--patina);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .listhead {
|
||||||
|
display: flex;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 16px 0 4px;
|
||||||
|
color: var(--paper);
|
||||||
|
font: 400 18px/1.2 var(--font-display);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .listhead .lbl {
|
||||||
|
color: var(--mute);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .listhead .lbl em {
|
||||||
|
color: var(--patina);
|
||||||
|
font-style: normal;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .pager {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 14px;
|
||||||
|
padding: 14px 0 0;
|
||||||
|
color: var(--mute-2);
|
||||||
|
font: 500 11px/1 var(--font-mono);
|
||||||
|
letter-spacing: .14em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .pager .pg {
|
||||||
|
color: var(--paper-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .pager .pg:hover {
|
||||||
|
color: var(--paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .pager .pg.disabled {
|
||||||
|
color: var(--faint);
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .empty {
|
||||||
|
padding: 28px 0;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .empty strong {
|
||||||
|
color: var(--paper);
|
||||||
|
font: 400 20px var(--font-display);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .empty p {
|
||||||
|
margin: 6px 0 0;
|
||||||
|
font: 14px/1.5 var(--font-body);
|
||||||
|
color: var(--mute);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .confirm-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
padding: 10px 12px;
|
||||||
|
background: var(--danger-wash);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .confirm-row span {
|
||||||
|
flex: 1 1 16ch;
|
||||||
|
color: var(--danger-soft);
|
||||||
|
font: 400 15px/1.3 var(--font-display);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .confirm-row div {
|
||||||
|
display: flex;
|
||||||
|
flex: none;
|
||||||
|
gap: 12px;
|
||||||
|
margin-left: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .detail-back {
|
||||||
|
display: inline-block;
|
||||||
|
margin: 18px 0 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .detail-title {
|
||||||
|
margin: 10px 0 2px;
|
||||||
|
color: var(--paper);
|
||||||
|
font: 400 28px/1.25 var(--font-display);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .detail-key {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--mute-2);
|
||||||
|
font: 500 11px/1.4 var(--font-mono);
|
||||||
|
letter-spacing: .08em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .detail-meta {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 4px 14px;
|
||||||
|
margin: 10px 0 0;
|
||||||
|
color: var(--mute-2);
|
||||||
|
font: 500 12px/1.5 var(--font-mono);
|
||||||
|
letter-spacing: .08em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .cover {
|
||||||
|
width: 160px;
|
||||||
|
aspect-ratio: 3 / 4;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
margin: 18px 0 4px;
|
||||||
|
background: var(--hatch);
|
||||||
|
color: var(--mute-2);
|
||||||
|
font: 500 10px/1 var(--font-mono);
|
||||||
|
letter-spacing: .2em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .detail-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr 1fr;
|
||||||
|
gap: 0 28px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .dform {
|
||||||
|
padding: 14px 0 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .dform h3 {
|
||||||
|
margin: 0 0 4px;
|
||||||
|
color: var(--mute-2);
|
||||||
|
font: 500 10px/1 var(--font-mono);
|
||||||
|
letter-spacing: .2em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .dform .field {
|
||||||
|
display: flex;
|
||||||
|
gap: 10px;
|
||||||
|
margin-top: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .pausebar {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: flex-end;
|
||||||
|
gap: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 719px) {
|
||||||
|
.admin-sheet .topbar {
|
||||||
|
flex-wrap: wrap;
|
||||||
|
row-gap: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .brand {
|
||||||
|
flex: 1 1 100%;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .topbar-actions {
|
||||||
|
margin-left: auto;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 1019px) {
|
||||||
|
.admin-sheet .tbl.lanes {
|
||||||
|
display: block;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .thead {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .trow {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 4px 16px;
|
||||||
|
padding: 12px 0;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .trow > * {
|
||||||
|
padding: 0;
|
||||||
|
border-bottom: none;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .c-site {
|
||||||
|
width: 100%;
|
||||||
|
padding-bottom: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.lanes .c-ctrl {
|
||||||
|
margin-left: auto;
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 899px) {
|
||||||
|
.admin-sheet .tbl,
|
||||||
|
.admin-sheet .tbl.lanes {
|
||||||
|
display: block;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .thead {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 2px 10px;
|
||||||
|
padding: 11px 0;
|
||||||
|
border-bottom: 1px solid var(--rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .trow > * {
|
||||||
|
padding: 0;
|
||||||
|
border-bottom: none;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-ch,
|
||||||
|
.admin-sheet .tbl .c-rd {
|
||||||
|
padding-right: 0;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-title {
|
||||||
|
width: 100%;
|
||||||
|
padding-bottom: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .trow {
|
||||||
|
display: flex;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl.series .trow > * {
|
||||||
|
padding: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .detail-grid {
|
||||||
|
grid-template-columns: 1fr;
|
||||||
|
}
|
||||||
|
|
||||||
|
.admin-sheet .tbl .c-act {
|
||||||
|
margin-left: auto;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -296,69 +296,6 @@ button { cursor: pointer; }
|
|||||||
letter-spacing: .04em;
|
letter-spacing: .04em;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ---- admin page: two sections on the same measured sheet, no cards ----
|
|
||||||
The reading page is a list of series; this is a list of facts. Both are
|
|
||||||
sheets of hairline-separated rows, so the roster keeps the shape it had as
|
|
||||||
a fold-out and the Lane block copies it. */
|
|
||||||
.readers, .lanes { margin: 0 20px; padding: 12px 0 16px; border-bottom: 1px solid var(--rule); }
|
|
||||||
.readers h2, .lanes h2 {
|
|
||||||
margin: 0;
|
|
||||||
padding: 8px 0;
|
|
||||||
font: 500 10px/1 var(--font-mono);
|
|
||||||
letter-spacing: .2em;
|
|
||||||
text-transform: uppercase;
|
|
||||||
color: var(--mute-2);
|
|
||||||
}
|
|
||||||
.readerlist, .lanelist { margin: 0; padding: 0; list-style: none; }
|
|
||||||
.readerlist li, .lanelist li {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
gap: 4px 16px;
|
|
||||||
min-height: 44px;
|
|
||||||
border-top: 1px solid var(--rule);
|
|
||||||
}
|
|
||||||
.reader-actions { display: flex; gap: 18px; margin-left: auto; }
|
|
||||||
.readerlist form { margin: 0; }
|
|
||||||
.reader-id {
|
|
||||||
font: 500 13px/1.4 var(--font-mono);
|
|
||||||
letter-spacing: .04em;
|
|
||||||
color: var(--paper);
|
|
||||||
}
|
|
||||||
.reader-sessions {
|
|
||||||
font: 500 10px/1 var(--font-mono);
|
|
||||||
letter-spacing: .14em;
|
|
||||||
text-transform: uppercase;
|
|
||||||
color: var(--mute);
|
|
||||||
}
|
|
||||||
.reader-sightings, .lane-fact {
|
|
||||||
font: 500 10px/1 var(--font-mono);
|
|
||||||
letter-spacing: .14em;
|
|
||||||
text-transform: uppercase;
|
|
||||||
color: var(--mute-2);
|
|
||||||
}
|
|
||||||
/* Two states the owner is meant to find rather than read for: a Reader whose
|
|
||||||
reports no longer defer a Poll, and a Lane that is not keeping its promise.
|
|
||||||
Both wear --patina — never ember, which means one thing, and never danger,
|
|
||||||
which is destruction. */
|
|
||||||
.reader-blocked, .lane-mark {
|
|
||||||
font: 500 10px/1 var(--font-mono);
|
|
||||||
letter-spacing: .14em;
|
|
||||||
text-transform: uppercase;
|
|
||||||
color: var(--patina);
|
|
||||||
}
|
|
||||||
.lane-site {
|
|
||||||
font: 400 19px/1.2 var(--font-display);
|
|
||||||
color: var(--paper-dim);
|
|
||||||
}
|
|
||||||
/* The whole row leans patina when the Lane needs attention, so the scan is one
|
|
||||||
pass down the left edge rather than a read of every mark. */
|
|
||||||
.lanelist li.attention .lane-site { color: var(--patina); }
|
|
||||||
.lane-browser { padding: 12px 0 0; }
|
|
||||||
/* Revocation cuts someone off, so it wears --danger. Ember stays reserved for
|
|
||||||
the new-chapter signal. */
|
|
||||||
.ghost.danger { color: var(--danger); }
|
|
||||||
.ghost.danger:hover { color: var(--danger); border-bottom-color: var(--danger); }
|
|
||||||
|
|
||||||
.chrome { display: flex; flex-direction: column; }
|
.chrome { display: flex; flex-direction: column; }
|
||||||
|
|
||||||
@@ -643,9 +580,6 @@ button { cursor: pointer; }
|
|||||||
box-shadow: inset 0 -2px 0 var(--ember);
|
box-shadow: inset 0 -2px 0 var(--ember);
|
||||||
}
|
}
|
||||||
.topbar form { margin-left: 18px; }
|
.topbar form { margin-left: 18px; }
|
||||||
/* The admin page's topbar has no switch to fill the middle, so its back link
|
|
||||||
keeps company with Log out at the right edge instead of floating centre. */
|
|
||||||
.topbar .back { margin-left: auto; }
|
|
||||||
/* At phone width brand + switch + Log out do not fit on one line, so the
|
/* At phone width brand + switch + Log out do not fit on one line, so the
|
||||||
switch takes its own row under the wordmark rather than pushing Log out
|
switch takes its own row under the wordmark rather than pushing Log out
|
||||||
off-screen. */
|
off-screen. */
|
||||||
@@ -655,9 +589,6 @@ button { cursor: pointer; }
|
|||||||
.libswitch { order: 3; margin-left: 0; }
|
.libswitch { order: 3; margin-left: 0; }
|
||||||
.libswitch a { flex: 1; text-align: center; padding: 8px 14px; }
|
.libswitch a { flex: 1; text-align: center; padding: 8px 14px; }
|
||||||
.topbar form { margin-left: 12px; }
|
.topbar form { margin-left: 12px; }
|
||||||
/* The admin page has no switch to take the second row, so its brand claims
|
|
||||||
the first outright and the back link keeps Log out company below. */
|
|
||||||
.topbar:has(.back) .brand { flex: 1 1 100%; }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ---- action strip: full-width on a phone, hairline-divided cells ---- */
|
/* ---- action strip: full-width on a phone, hairline-divided cells ---- */
|
||||||
|
|||||||
@@ -1,7 +1,5 @@
|
|||||||
{{/* The owner's administrative page: everything that reaches past one Reader,
|
{{/* Every owner-only address shares this shell; page content stays behind its
|
||||||
at its own address so it can be bookmarked rather than hunted for inside
|
bookmarkable route so the active tab survives a reload. */}}
|
||||||
the reading page. Owner-only at route registration (requireOwner), which
|
|
||||||
is why nothing in here re-tests who is asking. */}}
|
|
||||||
{{define "admin"}}
|
{{define "admin"}}
|
||||||
<!doctype html>
|
<!doctype html>
|
||||||
<html lang="en">
|
<html lang="en">
|
||||||
@@ -12,27 +10,40 @@
|
|||||||
<title>BookmarkManager — Admin</title>
|
<title>BookmarkManager — Admin</title>
|
||||||
<link rel="icon" href="/static/logo.svg" type="image/svg+xml">
|
<link rel="icon" href="/static/logo.svg" type="image/svg+xml">
|
||||||
<link rel="stylesheet" href="/static/style.css">
|
<link rel="stylesheet" href="/static/style.css">
|
||||||
|
<link rel="stylesheet" href="/static/admin.css">
|
||||||
<link rel="preload" href="/static/fonts/instrument-serif-400-latin.woff2" as="font" type="font/woff2" crossorigin>
|
<link rel="preload" href="/static/fonts/instrument-serif-400-latin.woff2" as="font" type="font/woff2" crossorigin>
|
||||||
<script src="/static/htmx.min.js" defer></script>
|
<script src="/static/htmx.min.js" defer></script>
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
<div class="sheet">
|
<div class="sheet admin-sheet">
|
||||||
<header class="topbar">
|
<header class="topbar">
|
||||||
<h1 class="brand">{{template "mark" .}}<span>Bookmark<em>Manager</em></span></h1>
|
<h1 class="brand">{{template "mark" .}}<span>Bookmark<em>Manager</em></span></h1>
|
||||||
{{/* Back to the library, no switch: this page belongs to neither library,
|
<span class="topbar-actions">
|
||||||
and the ember-lit switch says which library you are reading. */}}
|
<a class="ghost" href="/">Library</a>
|
||||||
<a class="ghost back" href="/">Library</a>
|
<form method="post" action="/logout">
|
||||||
<form method="post" action="/logout">
|
<button type="submit" class="ghost">Log out</button>
|
||||||
<button type="submit" class="ghost">Log out</button>
|
</form>
|
||||||
</form>
|
</span>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
{{/* The live region wraps the swapped block rather than being it: the
|
<nav class="navrow" aria-label="Admin pages">
|
||||||
refresh replaces the section wholesale, and a region recreated on every
|
<a href="/admin" class="{{if eq .Page "overview"}}active{{end}}" {{if eq .Page "overview"}}aria-current="page"{{end}}>Overview</a>
|
||||||
update is never announced. */}}
|
<a href="/admin/lanes" class="{{if eq .Page "lanes"}}active{{end}}" {{if eq .Page "lanes"}}aria-current="page"{{end}}>Lanes</a>
|
||||||
<div aria-live="polite">{{template "lanes" .Lanes}}</div>
|
<a href="/admin/readers" class="{{if eq .Page "readers"}}active{{end}}" {{if eq .Page "readers"}}aria-current="page"{{end}}>Readers</a>
|
||||||
|
<a href="/admin/series" class="{{if or (eq .Page "series") (eq .Page "series-detail")}}active{{end}}" {{if or (eq .Page "series") (eq .Page "series-detail")}}aria-current="page"{{end}}>Series</a>
|
||||||
|
</nav>
|
||||||
|
|
||||||
{{template "readers" .}}
|
<main class="page admin-page">
|
||||||
|
{{if eq .Page "lanes"}}
|
||||||
|
<div aria-live="polite">{{template "lanes" .Lanes}}</div>
|
||||||
|
{{else if eq .Page "readers"}}
|
||||||
|
{{template "readers" .}}
|
||||||
|
{{else if eq .Page "series"}}
|
||||||
|
{{template "series-list" .SeriesList}}
|
||||||
|
{{else if eq .Page "series-detail"}}
|
||||||
|
{{template "series-detail" .Detail}}
|
||||||
|
{{else}}{{template "overview" .Overview}}{{end}}
|
||||||
|
</main>
|
||||||
</div>
|
</div>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -1,42 +1,60 @@
|
|||||||
{{/* Poll Lane status: one row per Site, refreshing itself so a run can be
|
{{/* Poll Lane status: one row per Site's latest durable pass, refreshing
|
||||||
watched rather than sampled by reloading. The refresh is one attribute on
|
itself so a run can be watched rather than sampled by reloading. The
|
||||||
the fragment root and the endpoint answers with this same fragment, so the
|
refresh is one attribute on the fragment root and the endpoint answers
|
||||||
swap replaces the element that asked for it.
|
with this same fragment, so the swap replaces the element that asked.
|
||||||
|
|
||||||
Every figure here is read out of the running poller, never out of a table:
|
Every figure is read from poll_passes, never from a running poller: a
|
||||||
a Site absent from Rows has not completed a pass since the last restart,
|
restart answers from the database the moment it is up (issue #145). The
|
||||||
which the empty state must say — zeroes would read as a stopped Lane. */}}
|
browser fact is a deployment-config fact plus a reachability derived from
|
||||||
|
the pass log; the cause chips and the state phrase are decided in Go,
|
||||||
|
this template only prints them. */}}
|
||||||
{{define "lanes"}}
|
{{define "lanes"}}
|
||||||
<section class="lanes" id="lanes"
|
<section class="lanes" id="lanes"
|
||||||
hx-get="/ui/admin/lanes" hx-trigger="every 30s" hx-swap="outerHTML">
|
hx-get="/ui/admin/lanes" hx-trigger="every 30s" hx-swap="outerHTML">
|
||||||
<h2>Poll Lanes</h2>
|
<div class="sechead">
|
||||||
|
<h2 class="sec">Poll Lanes</h2>
|
||||||
|
<p class="statusline">
|
||||||
|
{{if .PollerOff}}Polling: <span class="mark-faint">off</span>
|
||||||
|
{{else}}Browser: {{if not .BrowserConfigured}}<span class="mark-faint">not configured</span>{{else if .BrowserReachable}}<span class="mark mark-strong">reachable</span>{{else}}<span class="mark bad">unreachable</span>{{end}}{{end}}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
{{if .Rows}}
|
{{if .Rows}}
|
||||||
<ul class="lanelist">
|
<div class="tbl lanes">
|
||||||
|
<div class="thead">
|
||||||
|
<span>Site</span><span>Due</span><span>Checked</span><span>Gap</span>
|
||||||
|
<span>Last pass</span><span>Outcomes · state</span><span></span>
|
||||||
|
</div>
|
||||||
{{range .Rows}}
|
{{range .Rows}}
|
||||||
<li{{if .Attention}} class="attention"{{end}}>
|
<div class="trow{{if .Attention}} attention{{end}}">
|
||||||
<span class="lane-site">{{.Site}}</span>
|
<span class="c-site">{{.Site}}</span>
|
||||||
<span class="lane-fact">{{.Due}} due</span>
|
<span>{{.Due}}</span>
|
||||||
<span class="lane-fact">{{.Checked}} checked</span>
|
<span>{{.Checked}}</span>
|
||||||
<span class="lane-fact">ran {{.Ran}}</span>
|
<span>{{.Gap}}</span>
|
||||||
{{if .Gap}}<span class="lane-fact">gap {{.Gap}}</span>{{end}}
|
<span>ran {{.Ran}}</span>
|
||||||
{{if .Clamped}}<span class="lane-mark">gap at floor</span>{{end}}
|
<span class="c-skip">{{if .HasChips}}{{range $i, $c := .Chips}}{{if $i}}<span class="mark-faint"> · </span>{{end}}<span class="mark">{{$c.Name}} {{$c.Count}}</span>{{end}}{{else}}<span class="mark-faint">none observed</span>{{end}}{{if .StatePhrase}} · <span class="{{if .StateGood}}ok{{else}}bad{{end}}">{{.StatePhrase}}</span>{{end}}</span>
|
||||||
{{if .Refusing}}<span class="lane-mark">refusing</span>{{end}}
|
{{/* The pause control lives in the one slot the design leaves for it:
|
||||||
{{if .BrowserLost}}<span class="lane-mark">no browser</span>{{end}}
|
a running Lane offers the three durations and Pause; a paused Lane
|
||||||
{{if .Stalled}}<span class="lane-mark">not checking</span>{{end}}
|
offers Resume in the same place. Pause is not destruction — it
|
||||||
{{if .Asleep}}<span class="lane-mark">browser asleep</span>{{end}}
|
takes nothing away and reverses in one press — so neither wears a
|
||||||
</li>
|
confirm row or the danger accent. The form wraps the select so the
|
||||||
|
offered duration travels with the press. */}}
|
||||||
|
<span class="c-ctrl">{{if .Paused}}<span class="pausebar">
|
||||||
|
<form hx-post="/admin/lanes/{{.Site}}/resume" hx-target="#lanes" hx-swap="outerHTML">
|
||||||
|
<button type="submit" class="ghost">Resume</button>
|
||||||
|
</form>
|
||||||
|
</span>{{else}}<span class="pausebar">
|
||||||
|
<form hx-post="/admin/lanes/{{.Site}}/pause" hx-target="#lanes" hx-swap="outerHTML">
|
||||||
|
<select name="duration" aria-label="Pause duration">
|
||||||
|
<option>1h</option><option selected>6h</option><option>24h</option>
|
||||||
|
</select>
|
||||||
|
<button type="submit" class="ghost">Pause</button>
|
||||||
|
</form>
|
||||||
|
</span>{{end}}</span>
|
||||||
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
</ul>
|
</div>
|
||||||
{{else}}
|
{{else}}
|
||||||
<p class="setup-copy">No data yet — no Lane has completed a pass since the
|
<p class="empty">No data yet — no Lane has recorded a pass.</p>
|
||||||
backend started.</p>
|
|
||||||
{{end}}
|
{{end}}
|
||||||
<p class="setup-copy lane-browser">
|
|
||||||
{{if .PollerOff}}Polling is switched off in this deployment: no Lane runs,
|
|
||||||
and Latest Chapter comes from the userscripts alone.
|
|
||||||
{{else}}Browser sidecar:
|
|
||||||
{{if not .BrowserConfigured}}not configured — comix, kagane and novelfull
|
|
||||||
pages are not fetched through it{{else if .BrowserReachable}}reachable
|
|
||||||
{{else}}unreachable{{end}}.{{end}}</p>
|
|
||||||
</section>
|
</section>
|
||||||
{{end}}
|
{{end}}
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
{{/* The Overview landing page: one verdict line leading, then a stats block
|
||||||
|
where every figure is a door into the list that counts it, and the
|
||||||
|
per-Site library shape table. Every judgement — the verdict state, which
|
||||||
|
figures link, what a Lane's state means — is made in Go; this template
|
||||||
|
only prints. */}}
|
||||||
|
{{define "overview"}}
|
||||||
|
<p class="verdict"><span class="attn">{{.Verdict}}</span> {{if .HasCounts}}<span class="counts">· <b>{{.Waiting}}</b> series waiting · <b>{{.Unchecked}}</b> unchecked over 12h</span>{{end}}</p>
|
||||||
|
<h2 class="sec">Hygiene</h2>
|
||||||
|
<div class="stats">{{range .Hygiene}}<div class="stat"><span class="lbl">{{.Label}}</span>{{if .Href}}<a class="fig" href="{{.Href}}">{{.Count}}</a>{{else}}<span class="fig zero">{{.Count}}</span>{{end}}</div>{{end}}</div>
|
||||||
|
<h2 class="sec">Library</h2>
|
||||||
|
<div class="stats">{{range .Library}}<div class="stat"><span class="lbl">{{.Label}}</span>{{if .Href}}<a class="fig" href="{{.Href}}">{{.Count}}</a>{{else}}<span class="fig zero">{{.Count}}</span>{{end}}</div>{{end}}</div>
|
||||||
|
<h2 class="sec">Sites · last 12h</h2>
|
||||||
|
<div class="tbl sites"><div class="thead"><span>Site</span><span>Series</span><span>No cover</span><span>Never chk</span><span>Stale</span><span>State</span></div>{{range .Sites}}<div class="trow"><a class="c-site site-{{.Site}}" href="{{.SiteHref}}">{{.Site}}</a>{{range .Figs}}{{if .Href}}<a class="fig" href="{{.Href}}">{{.Count}}</a>{{else}}<span class="fig zero">{{.Count}}</span>{{end}}{{end}}<span class="c-state{{if .StateGood}} ok{{end}}{{if .StateBad}} bad{{end}}">{{.State}}</span></div>{{end}}</div>
|
||||||
|
{{end}}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
{{/* Per-Series page: one address per Series, keyed "<site>:<series_id>" so the
|
||||||
|
list row is one hop from it. Everything here is a Series-level fact plus
|
||||||
|
the anonymous Reader count. The Check now control lands in its own .dform
|
||||||
|
below the (empty) .detail-grid; the pending marker rides the meta line
|
||||||
|
with the other marks. */}}
|
||||||
|
{{define "series-detail"}}
|
||||||
|
<a class="ghost detail-back" href="/admin/series">← Series</a>
|
||||||
|
<h1 class="detail-title">{{.Title}}</h1>
|
||||||
|
<p class="detail-key">{{.Key}} · {{.Site}} · {{.Kind}}</p>
|
||||||
|
{{if .Cover}}<div class="cover"><img src="{{.Cover}}" alt="" loading="lazy"></div>
|
||||||
|
{{else}}<div class="cover"></div>{{end}}
|
||||||
|
{{template "series-detail-meta" .}}
|
||||||
|
<div class="detail-grid"></div>
|
||||||
|
{{if .CanPoll}}
|
||||||
|
<div class="dform">
|
||||||
|
<div class="field"><a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="#detail-meta" hx-swap="outerHTML" href="#">Check now</a></div>
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
|
{{end}}
|
||||||
|
|
||||||
|
{{/* series-detail-meta is the meta line, and the answer a Check now press on
|
||||||
|
the detail page swaps into its place: the same marks, re-rendered after
|
||||||
|
the stamp so the pending marker shows. */}}
|
||||||
|
{{define "series-detail-meta"}}
|
||||||
|
<div class="detail-meta" id="detail-meta">
|
||||||
|
<span>ch {{.Chapter}}</span>
|
||||||
|
<span>checked {{.Checked}}</span>
|
||||||
|
<span>{{.Readers}} readers</span>
|
||||||
|
{{if .Pending}}<span class="mark">{{.Requested}}</span>{{end}}
|
||||||
|
{{if .Unpollable}}<span class="mark">unpollable</span>{{end}}
|
||||||
|
{{if .NoCover}}<span class="mark">no cover</span>{{end}}
|
||||||
|
{{if .Orphan}}<span class="mark">orphan</span>{{end}}
|
||||||
|
{{if .SightingRaised}}<span class="mark">sighting-raised</span>{{end}}
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
{{/* The Series list: every Series across every Reader's library, filtered by
|
||||||
|
one hygiene rule and narrowed by Site and Library. Filter, Site, Library
|
||||||
|
and page all live in the query string, so the list's state is an address
|
||||||
|
that can be bookmarked: the two selects submit the GET form, and the
|
||||||
|
Library segment links and the pager preserve the filter and Site. */}}
|
||||||
|
{{define "series-list"}}
|
||||||
|
<form class="filterbar" method="get" action="/admin/series">
|
||||||
|
<input type="hidden" name="kind" value="{{.Kind}}">
|
||||||
|
<label class="fsel"><span>Show</span><select name="filter" onchange="this.form.submit()">
|
||||||
|
{{range .Filters}}<option value="{{.Name}}"{{if .Selected}} selected{{end}}>{{.Label}} ({{.Count}})</option>{{end}}
|
||||||
|
</select></label>
|
||||||
|
<label class="fsel"><span>Site</span><select name="site" onchange="this.form.submit()">
|
||||||
|
<option value=""{{if not .Site}} selected{{end}}>All sites</option>
|
||||||
|
{{range .Sites}}<option value="{{.}}"{{if eq $.Site .}} selected{{end}}>{{.}}</option>{{end}}
|
||||||
|
</select></label>
|
||||||
|
<span class="segrow">
|
||||||
|
<a href="{{.KindBoth}}"{{if not .Kind}} class="active"{{end}}>both</a>
|
||||||
|
<a href="{{.KindManga}}"{{if eq .Kind "manga"}} class="active"{{end}}>manga</a>
|
||||||
|
<a href="{{.KindNovel}}"{{if eq .Kind "novel"}} class="active"{{end}}>novels</a>
|
||||||
|
</span>
|
||||||
|
</form>
|
||||||
|
<div class="listhead">{{.Total}} series <span class="lbl">· <em>{{.FilterLabel}}</em></span></div>
|
||||||
|
{{if .Rows}}
|
||||||
|
<div class="tbl series">
|
||||||
|
<div class="thead"><span>Site</span><span class="c-ch">Ch</span><span>Checked</span><span class="c-rd">Readers</span><span>Notes</span><span></span></div>
|
||||||
|
{{range .Rows}}{{template "series-row" .}}{{end}}
|
||||||
|
</div>
|
||||||
|
<div class="pager">
|
||||||
|
{{if .PrevHref}}<a class="pg" href="{{.PrevHref}}">‹ prev</a>{{else}}<span class="pg disabled">‹ prev</span>{{end}}
|
||||||
|
<span>{{.Range}}</span>
|
||||||
|
{{if .NextHref}}<a class="pg" href="{{.NextHref}}">next ›</a>{{else}}<span class="pg disabled">next ›</span>{{end}}
|
||||||
|
</div>
|
||||||
|
{{else}}
|
||||||
|
<div class="empty"><strong>No series</strong><p>Nothing matches <em>{{.FilterLabel}}</em>.</p></div>
|
||||||
|
{{end}}
|
||||||
|
{{end}}
|
||||||
|
|
||||||
|
{{/* series-row is one Series list row, and the answer a Check now press swaps
|
||||||
|
into the row's place (hx-target="closest .trow"): it must render the
|
||||||
|
pending marker the press created. The control is absent on a Series with
|
||||||
|
no page to fetch and on an orphan, so the owner is never offered a button
|
||||||
|
that can never do anything. */}}
|
||||||
|
{{define "series-row"}}
|
||||||
|
<div class="trow{{if .Attention}} attention{{end}}{{if .Band}} band{{end}}">
|
||||||
|
<span class="c-title"><a href="/admin/series/{{.Key}}">{{.Title}}</a>{{if .Pending}}<span class="mark">{{.Requested}}</span>{{end}}</span>
|
||||||
|
<span class="c-site site-{{.Site}}">{{.Site}}</span>
|
||||||
|
<span class="c-ch">{{.Ch}}</span>
|
||||||
|
<span>{{.Age}}</span>
|
||||||
|
<span class="c-rd">{{.Readers}}</span>
|
||||||
|
<span class="c-note">{{range .Notes}}<span class="mark">{{.}}</span>{{end}}{{if .More}}<span class="mark mark-faint">+{{.More}}</span>{{end}}</span>
|
||||||
|
<span class="c-act">{{if .CanPoll}}<a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="closest .trow" hx-swap="outerHTML" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' href="#">Check now</a>{{end}}</span>
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
@@ -50,9 +50,14 @@ type Handler struct {
|
|||||||
// httpClient is the plain stdlib client that talks to Discord. It is not
|
// httpClient is the plain stdlib client that talks to Discord. It is not
|
||||||
// an injected interface: tests point APIBase at a stub server instead.
|
// an injected interface: tests point APIBase at a stub server instead.
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
// lanes is the Poll Lane snapshot source the administrative page reads.
|
// pollerEnabled reports whether latest-chapter polling is switched on in
|
||||||
// Nil is a running deployment with no poller, not a bug.
|
// this deployment (LATEST_CHAPTER_POLL_ENABLED) and browserConfigured
|
||||||
lanes LaneReporter
|
// whether a browser sidecar is configured (BROWSER_WS_URL set). Both are
|
||||||
|
// deployment facts resolved by the composition root; the Lanes page (issue
|
||||||
|
// #145) reports them from config and derives reachability from the pass
|
||||||
|
// log rather than from whether a poller goroutine happened to start.
|
||||||
|
pollerEnabled bool
|
||||||
|
browserConfigured bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// listView is what every list-rendering template receives.
|
// listView is what every list-rendering template receives.
|
||||||
@@ -110,9 +115,11 @@ type loginView struct {
|
|||||||
// New parses every template up front so a broken one kills the process at
|
// New parses every template up front so a broken one kills the process at
|
||||||
// startup rather than the first request that touches it.
|
// startup rather than the first request that touches it.
|
||||||
//
|
//
|
||||||
// lanes is the administrative page's window onto the running Poller; nil means
|
// pollerEnabled and browserConfigured are deployment facts the composition
|
||||||
// nothing is polling, which the page reports rather than hides.
|
// root resolves from LATEST_CHAPTER_POLL_ENABLED and BROWSER_WS_URL: the Lanes
|
||||||
func New(s *store.Store, discord DiscordConfig, tokenKey []byte, mangaPath, novelPath string, lanes LaneReporter) (*Handler, error) {
|
// page (issue #145) reports them and derives browser reachability from the
|
||||||
|
// pass log, so no running poller is wired through here at all.
|
||||||
|
func New(s *store.Store, discord DiscordConfig, tokenKey []byte, mangaPath, novelPath string, pollerEnabled, browserConfigured bool) (*Handler, error) {
|
||||||
tmpl, err := template.ParseFS(templateFS, "templates/*.html")
|
tmpl, err := template.ParseFS(templateFS, "templates/*.html")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -127,7 +134,8 @@ func New(s *store.Store, discord DiscordConfig, tokenKey []byte, mangaPath, nove
|
|||||||
states: newOAuthStates(),
|
states: newOAuthStates(),
|
||||||
limiter: session.NewLoginLimiter(),
|
limiter: session.NewLoginLimiter(),
|
||||||
httpClient: &http.Client{Timeout: discordTimeout},
|
httpClient: &http.Client{Timeout: discordTimeout},
|
||||||
lanes: lanes,
|
pollerEnabled: pollerEnabled,
|
||||||
|
browserConfigured: browserConfigured,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+23
-15
@@ -54,6 +54,11 @@ type Config struct {
|
|||||||
// /u/{token}/novel-bookmark.user.js. Same bindmount, second script: the
|
// /u/{token}/novel-bookmark.user.js. Same bindmount, second script: the
|
||||||
// two libraries are separate installs.
|
// two libraries are separate installs.
|
||||||
NovelUserscriptPath string
|
NovelUserscriptPath string
|
||||||
|
// BrowserWSURL is the CDP websocket the poller's browser Sites read
|
||||||
|
// through. Set means a browser sidecar is configured in this deployment —
|
||||||
|
// the Lanes page reports the fact and derives reachability from the pass
|
||||||
|
// log rather than asking the poller (issue #145).
|
||||||
|
BrowserWSURL string
|
||||||
// LatestPoll configures the background latest-chapter fetcher.
|
// LatestPoll configures the background latest-chapter fetcher.
|
||||||
LatestPoll LatestPoll
|
LatestPoll LatestPoll
|
||||||
}
|
}
|
||||||
@@ -108,6 +113,7 @@ func loadConfig() Config {
|
|||||||
OwnerDiscordID: os.Getenv("OWNER_DISCORD_ID"),
|
OwnerDiscordID: os.Getenv("OWNER_DISCORD_ID"),
|
||||||
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
|
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
|
||||||
NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"),
|
NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"),
|
||||||
|
BrowserWSURL: os.Getenv("BROWSER_WS_URL"),
|
||||||
LatestPoll: loadLatestPoll(),
|
LatestPoll: loadLatestPoll(),
|
||||||
}
|
}
|
||||||
c.Discord = web.DiscordConfig{
|
c.Discord = web.DiscordConfig{
|
||||||
@@ -130,9 +136,10 @@ func loadConfig() Config {
|
|||||||
// preflight OPTIONS short-circuits before auth; /bookmarks* is auth-protected,
|
// preflight OPTIONS short-circuits before auth; /bookmarks* is auth-protected,
|
||||||
// /healthz is public.
|
// /healthz is public.
|
||||||
//
|
//
|
||||||
// lanes may be nil — polling disabled, or its client could not be built. The
|
// The web layer learns the deployment's poller and browser config from cfg —
|
||||||
// admin page reports that rather than pretending Lanes exist.
|
// nothing of the running poller is wired through here; the Lanes page reads
|
||||||
func newRouter(s *store.Store, cfg Config, lanes web.LaneReporter) http.Handler {
|
// the database (issue #145).
|
||||||
|
func newRouter(s *store.Store, cfg Config) http.Handler {
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
h := &api.Handler{Store: s}
|
h := &api.Handler{Store: s}
|
||||||
mux.HandleFunc("GET /healthz", api.Healthz)
|
mux.HandleFunc("GET /healthz", api.Healthz)
|
||||||
@@ -161,9 +168,12 @@ func newRouter(s *store.Store, cfg Config, lanes web.LaneReporter) http.Handler
|
|||||||
mux.Handle("/bookmarks/", auth)
|
mux.Handle("/bookmarks/", auth)
|
||||||
|
|
||||||
// The browser UI is always registered; signing in is Discord OAuth, so
|
// The browser UI is always registered; signing in is Discord OAuth, so
|
||||||
// there is no password to forget and no gate to leave unset.
|
// there is no password to forget and no gate to leave unset. The poller
|
||||||
|
// and browser facts are config, not the poller's: the Lanes page reads
|
||||||
|
// the pass log and reports the deployment as configured.
|
||||||
wh, err := web.New(s, cfg.Discord, []byte(cfg.TokenKey),
|
wh, err := web.New(s, cfg.Discord, []byte(cfg.TokenKey),
|
||||||
cfg.UserscriptPath, cfg.NovelUserscriptPath, lanes)
|
cfg.UserscriptPath, cfg.NovelUserscriptPath,
|
||||||
|
cfg.LatestPoll.Enabled, strings.TrimSpace(cfg.BrowserWSURL) != "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Fatalf("web handler: %v", err)
|
log.Fatalf("web handler: %v", err)
|
||||||
}
|
}
|
||||||
@@ -241,7 +251,7 @@ func main() {
|
|||||||
var browser latest.Fetcher
|
var browser latest.Fetcher
|
||||||
pollCtx, stopPoll := context.WithCancel(context.Background())
|
pollCtx, stopPoll := context.WithCancel(context.Background())
|
||||||
defer stopPoll()
|
defer stopPoll()
|
||||||
if ws := strings.TrimSpace(os.Getenv("BROWSER_WS_URL")); ws != "" {
|
if ws := strings.TrimSpace(cfg.BrowserWSURL); ws != "" {
|
||||||
bf, err := latest.NewBrowserFetcher(ws)
|
bf, err := latest.NewBrowserFetcher(ws)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("browser fetcher disabled: %v", err)
|
log.Printf("browser fetcher disabled: %v", err)
|
||||||
@@ -278,17 +288,14 @@ func main() {
|
|||||||
}
|
}
|
||||||
s.OnSeriesCreated = acq.Acquire
|
s.OnSeriesCreated = acq.Acquire
|
||||||
}
|
}
|
||||||
// A nil *Poller must not become a non-nil interface holding a nil pointer:
|
// The poller's only connection to the web layer is the database now: it is
|
||||||
// the admin page tests the reporter for nil to decide whether anything is
|
// started for its own sake, and the Lanes page reads the pass rows it
|
||||||
// polling at all.
|
// records (issue #145).
|
||||||
var lanes web.LaneReporter
|
startLatestPoller(pollCtx, s, cfg.LatestPoll, browser)
|
||||||
if poller := startLatestPoller(pollCtx, s, cfg.LatestPoll, browser); poller != nil {
|
|
||||||
lanes = poller
|
|
||||||
}
|
|
||||||
|
|
||||||
srv := &http.Server{
|
srv := &http.Server{
|
||||||
Addr: ":" + cfg.Port,
|
Addr: ":" + cfg.Port,
|
||||||
Handler: newRouter(s, cfg, lanes),
|
Handler: newRouter(s, cfg),
|
||||||
ReadHeaderTimeout: 10 * time.Second,
|
ReadHeaderTimeout: 10 * time.Second,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -336,7 +343,8 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
|
|||||||
// HTTP client cannot be built. Any problem here is logged and skipped: this
|
// HTTP client cannot be built. Any problem here is logged and skipped: this
|
||||||
// feature going missing degrades the service to userscript-only latest-chapter
|
// feature going missing degrades the service to userscript-only latest-chapter
|
||||||
// tracking, which is exactly how it behaved before. It returns the running
|
// tracking, which is exactly how it behaved before. It returns the running
|
||||||
// Poller, or nil when there is none — the admin page's Lane status reads it.
|
// Poller, or nil when there is none; the caller starts it for its own sake —
|
||||||
|
// the Lanes page reads the pass log, so no return value is wired anywhere.
|
||||||
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher) *latest.Poller {
|
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher) *latest.Poller {
|
||||||
if !cfg.Enabled {
|
if !cfg.Enabled {
|
||||||
log.Println("latest-chapter poller: disabled by config")
|
log.Println("latest-chapter poller: disabled by config")
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ func withBody(req *http.Request, body string) *http.Request {
|
|||||||
// A refused credential is refused however plausible it looks: only a hash the
|
// A refused credential is refused however plausible it looks: only a hash the
|
||||||
// readers table holds authenticates anything.
|
// readers table holds authenticates anything.
|
||||||
func TestUnknownCredentialRejected(t *testing.T) {
|
func TestUnknownCredentialRejected(t *testing.T) {
|
||||||
srv := newRouter(newTestStore(t), testConfig(), nil)
|
srv := newRouter(newTestStore(t), testConfig())
|
||||||
|
|
||||||
rr := httptest.NewRecorder()
|
rr := httptest.NewRecorder()
|
||||||
srv.ServeHTTP(rr, credRequest(http.MethodGet, "/bookmarks", readerCredential("never-registered")))
|
srv.ServeHTTP(rr, credRequest(http.MethodGet, "/bookmarks", readerCredential("never-registered")))
|
||||||
@@ -69,7 +69,7 @@ func TestUnknownCredentialRejected(t *testing.T) {
|
|||||||
func TestPerReaderIsolation(t *testing.T) {
|
func TestPerReaderIsolation(t *testing.T) {
|
||||||
s := newTestStore(t)
|
s := newTestStore(t)
|
||||||
registerReader(t, s, "other-reader")
|
registerReader(t, s, "other-reader")
|
||||||
srv := newRouter(s, testConfig(), nil)
|
srv := newRouter(s, testConfig())
|
||||||
|
|
||||||
ownerKey := "asura:solo"
|
ownerKey := "asura:solo"
|
||||||
putBookmark(t, srv, ownerKey, store.Bookmark{
|
putBookmark(t, srv, ownerKey, store.Bookmark{
|
||||||
@@ -267,7 +267,7 @@ func TestRotateCredentialViaWebUI(t *testing.T) {
|
|||||||
}
|
}
|
||||||
cfg := testConfig()
|
cfg := testConfig()
|
||||||
cfg.UserscriptPath = path
|
cfg.UserscriptPath = path
|
||||||
srv := newRouter(s, cfg, nil)
|
srv := newRouter(s, cfg)
|
||||||
|
|
||||||
oldCred := ownerCredential()
|
oldCred := ownerCredential()
|
||||||
rr := httptest.NewRecorder()
|
rr := httptest.NewRecorder()
|
||||||
|
|||||||
+1530
-100
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,44 @@
|
|||||||
|
# ADR-0012: Persisted lane state
|
||||||
|
|
||||||
|
Date: 2026-08-21
|
||||||
|
Status: accepted
|
||||||
|
|
||||||
|
Supersedes the in-memory lane snapshot carried by `latest`'s `LaneState`/`Status`
|
||||||
|
and the `web.LaneReporter` seam (ADR-0010 wrote the durable rows this page now
|
||||||
|
reads).
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The admin Lanes page stops reading the poller's in-memory Lane state and
|
||||||
|
becomes a read of `poll_passes`/`poll_lanes` in Postgres. There is no
|
||||||
|
`LaneReporter` interface: `web/admin_lanes.go` walks `store.LatestLanePasses()`
|
||||||
|
into one row per Site and adds the window's outcome sums from
|
||||||
|
`store.LanePassOutcomes()`. The `latest` package's `LaneState`/`Status` snapshot
|
||||||
|
and its `web.LaneReporter` seam are deleted.
|
||||||
|
|
||||||
|
The browser is a deployment configuration fact plus a reachability derived
|
||||||
|
from the pass log: `BROWSER_WS_URL` set means "configured", and the browser is
|
||||||
|
"reachable" unless a recent browser-Site pass inside `latest.RefuseBackoff` is
|
||||||
|
a sidecar loss, a missing fetcher, or an interrupted read. A skip reason is
|
||||||
|
the whole difference between a Lane resting and a Lane stuck: a skipped pass
|
||||||
|
prints its sentence, and only an empty skip with Series due and none read
|
||||||
|
draws the true-stall fault. Sleep skips never count toward `Attention`.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
The old page lived on a poller snapshot. Because that state was in memory, a
|
||||||
|
deploy erased it: the page read zeroes until a fresh pass ran, and browser
|
||||||
|
reachability came through a reporter interface only a live poller could
|
||||||
|
serve. Making the page answer from the database means a restart is complete
|
||||||
|
the instant the store is up, the browser fact survives a poller restart, and
|
||||||
|
a Lane that has not yet gathered figures shows a placeholder rather than a
|
||||||
|
confident zero.
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
The poller still owns the writes: each pass exit records one row (ADR-0010),
|
||||||
|
and a pass that returns before gathering figures carries the previous pass's
|
||||||
|
numbers forward instead of recording zeroes. A skip is a stable wire string;
|
||||||
|
`asleep` never counts toward `Attention`. When polling is switched off
|
||||||
|
(`LATEST_CHAPTER_POLL_ENABLED` unset) the page must say so, and the browser
|
||||||
|
statusline appears only when polling is switched on.
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# ADR-0013: Commands through the database
|
||||||
|
|
||||||
|
Date: 2026-08-22
|
||||||
|
Status: accepted
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Owner interventions are **facts about rows, never commands to the poller**.
|
||||||
|
*Check now* (`POST /admin/series/{key}/poll`) writes one stamp —
|
||||||
|
`series.force_poll_at`, unix ms, zero meaning never asked (the column landed
|
||||||
|
in migration 0014) — and the poller's next pass reads it through
|
||||||
|
`Store.DueForLatestCheck`. The control never signals the running process, so
|
||||||
|
a request survives a restart, and the whole surface is testable with no
|
||||||
|
poller running at all.
|
||||||
|
|
||||||
|
**Pending is derived, never stored**: a Series is pending while
|
||||||
|
`force_poll_at > latest_checked_at`. It self-clears with no second write and
|
||||||
|
no sweeper because the check stamp is written *before* the fetch (the same
|
||||||
|
"attempted" discipline as ADR-0010) — the first attempt ends the pending
|
||||||
|
state whatever the attempt returns. There is no expiry: a request the Lane
|
||||||
|
never reaches keeps ageing in the UI, and an old pending marker is itself the
|
||||||
|
evidence that a Lane is stuck. Writing again re-stamps the request time; the
|
||||||
|
write is idempotent.
|
||||||
|
|
||||||
|
**Queue-jump rules.** A forced Series overrides exactly three gates in the
|
||||||
|
due query: the rest cutoff, the Sighting-deferral clause and the finished-only
|
||||||
|
bucket, and it sorts to the front of the queue
|
||||||
|
(`ORDER BY forced DESC, reader_count DESC, latest_checked_at ASC`). It never
|
||||||
|
overrides an empty `series_url` (nothing to fetch), the Bookmarks join (a
|
||||||
|
Series no Reader holds has no consumer for the result), the Lane's refusal
|
||||||
|
backoff, the sidecar-down skip, or the Lane's gap — the last three are
|
||||||
|
poller-side gates the query cannot see and must not. The one pass-level gate
|
||||||
|
a forced Series does open is the browser wake threshold: a human asking wakes
|
||||||
|
a sleeping Chrome, where the thresholds exist to stop the machine waking
|
||||||
|
itself for one unattended check. If the home machine is off, nothing happens
|
||||||
|
and the request ages visibly, which is correct.
|
||||||
|
|
||||||
|
Rejected: zeroing the check stamp as the force signal. It would corrupt the
|
||||||
|
never-checked and stale counts the landing page exists to show, and make a
|
||||||
|
pending marker impossible.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
A stuck-looking Series previously waited for its turn in the Lane's hour, and
|
||||||
|
there was no way to ask for one check sooner. A direct poller command would
|
||||||
|
have been lost on every restart and untestable without a running poller; a
|
||||||
|
row the poller already reads is neither. Deriving pending from the two stamps
|
||||||
|
keeps the flag honest across restarts and makes the mechanism two column
|
||||||
|
writes and three query clauses instead of a state machine.
|
||||||
|
|
||||||
|
## Constraints
|
||||||
|
|
||||||
|
- The finished-status clause the force flag overrides is today's Lifecycle
|
||||||
|
test; a later spec in this series deletes it wholesale rather than amending
|
||||||
|
it, so the clause stays as it stands.
|
||||||
|
- The control is unconfirmed (it takes nothing away) and renders no
|
||||||
|
`.confirm-row`; it is hidden on a Series with no `series_url` and on an
|
||||||
|
orphan — the same pair the due query refuses to override.
|
||||||
|
- The answer to a press is the freshly rendered row, so the figures describe
|
||||||
|
the state after the press.
|
||||||
@@ -0,0 +1,344 @@
|
|||||||
|
# GIF — maximum byte size of a file
|
||||||
|
|
||||||
|
Research note for Gitea issue #71 (backend `maxBodyBytes` = 4 MiB rejects the
|
||||||
|
8,571,192-byte animated cover GIF at
|
||||||
|
`https://cdn.asurascans.com/asura-images/covers/a-dragonslayers-peerless-regression.gif`).
|
||||||
|
|
||||||
|
All facts fetched live on **2026-08-17**: the GIF89a spec at
|
||||||
|
`https://www.w3.org/Graphics/GIF/spec-gif89a.txt`, Go stdlib `image/gif`
|
||||||
|
sources at `/usr/local/go/src/image/gif/reader.go` (Go 1.26.5), Chromium
|
||||||
|
`blink/renderer/platform/image-decoders/` sources via
|
||||||
|
`chromium.googlesource.com`, Firefox `image/decoders/nsGIFDecoder2.cpp` via
|
||||||
|
`hg.mozilla.org`, and cover bytes probed with plain `curl` (desktop Chrome UA;
|
||||||
|
`HEAD`/ranged `GET`). **No Cloudflare challenge was encountered on any CDN
|
||||||
|
probe** — every request returned real headers, consistent with the AGENTS.md
|
||||||
|
note of 2026-07-26 that plain `curl` works against both scan sites from the
|
||||||
|
dev machine and the VPS.
|
||||||
|
|
||||||
|
Every claim carries the URL it came from, or a reproducible command.
|
||||||
|
Interpretation rather than observation is marked `[INFERENCE]`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Summary answer table
|
||||||
|
|
||||||
|
| Question | Answer | Evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| Does the GIF89a spec define a maximum file size? | **No.** There is no file-size field anywhere in the format; the only numeric ceilings are per-field (16-bit screen/image dimensions, 255-byte sub-blocks, 12-bit LZW codes). | §2 |
|
||||||
|
| Maximum logical screen | 65535 × 65535 pixels (unsigned 16-bit width/height). | §2.1 |
|
||||||
|
| Number of frames / image descriptors | Unbounded — "An unlimited number of images may be present per Data Stream." | §2.2 |
|
||||||
|
| Formal max byte size of any single GIF | None. Single-frame worst case ≈ **6.44 GB** (12-bit LZW, max canvas); animated GIFs are **unbounded** because frames are unbounded. | §3 |
|
||||||
|
| Does the backend's decoder (Go `image/gif`) bound size? | **No.** It reads 16-bit dimensions and allocates `width×height` bytes per frame; a 65535² frame forces a ~4 GiB allocation. No total-size or dimension guard. | §4.1 |
|
||||||
|
| Do browsers bound on-disk GIF size? | Chromium and Firefox: no on-wire size cap in their GIF readers; Chromium caps *decoded* memory at min(4 B × pixels, platform budget). | §4.3, §4.4 |
|
||||||
|
| Real cover sizes (asurascans, n=25) | min 190,410 B · median 1,275,082 B · p90 4,524,788 B · max 8,571,192 B · **3/25 > 4 MiB** (two JPEGs and the animated GIF) | §5 |
|
||||||
|
| Real cover sizes (demonicscans/readermc, n=78) | min 13,298 B · median 63,061 B · max 801,200 B · 0/78 > 4 MiB | §5 |
|
||||||
|
| Comparable service caps | GitHub: 10 MB for images/GIFs. Discord API: default 10 MiB per file. Wikimedia: 100 MiB upload / 5 GiB host. | §6 |
|
||||||
|
| Recommended cover cap for #71 | **10 MiB** (separate from the 4 MiB series-page cap). Covers 100% of the 103 observed covers; matches GitHub/Discord calibration; ≤ 20 MiB worst-case transient per concurrent fetch+serve on a 1974 MiB swapless VPS. | §7 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. What the GIF89a specification actually bounds
|
||||||
|
|
||||||
|
Source: `https://www.w3.org/Graphics/GIF/spec-gif89a.txt` (fetched 2026-08-17).
|
||||||
|
|
||||||
|
### 2.1 Fixed-width fields — the only hard ceilings
|
||||||
|
|
||||||
|
The format is a stream of fixed-width blocks; the numeric fields that *do*
|
||||||
|
have a ceiling are all 16-bit unsigned, little-endian ("multi-byte numeric
|
||||||
|
fields are ordered Least Significant Byte first", §4 of the spec):
|
||||||
|
|
||||||
|
- **Logical Screen Width / Height** — "Unsigned" 2-byte fields (§18, Logical
|
||||||
|
Screen Descriptor) → maximum **65535 × 65535** pixels.
|
||||||
|
- **Image Left / Top Position, Image Width / Height** — "Unsigned" 2-byte
|
||||||
|
fields (§20, Image Descriptor). Each image "must fit within the boundaries
|
||||||
|
of the Logical Screen" (§20a), so an image cannot exceed the 65535² canvas
|
||||||
|
even though its own fields would allow it.
|
||||||
|
- **Data sub-blocks** — "A data sub-block may contain from 0 to 255 data
|
||||||
|
bytes" (§15); each sub-block is preceded by a 1-byte size field and the
|
||||||
|
stream is terminated by a 0x00 Block Terminator (§16). This bounds a
|
||||||
|
*chunk*, not the stream.
|
||||||
|
- **Global/Local Color Tables** — optional, "3 x 2^(Size of Global Color
|
||||||
|
Table+1)" bytes with a 3-bit size field → at most 3 × 2⁸ = **768 bytes**
|
||||||
|
each (§19, §21).
|
||||||
|
- **LZW codes** — "The output codes are of variable length, starting at
|
||||||
|
<code size>+1 bits per code, **up to 12 bits per code**. This defines a
|
||||||
|
maximum code value of 4095 (0xFFF)" (Appendix F, COMPRESSION, rule 4).
|
||||||
|
- **Trailer** — a single byte, fixed value 0x3B, "indicating the end of the
|
||||||
|
GIF Data Stream" (§27).
|
||||||
|
|
||||||
|
### 2.2 What is unbounded
|
||||||
|
|
||||||
|
- **Number of images (frames).** §20a, verbatim: "This block is REQUIRED for
|
||||||
|
an image. Exactly one Image Descriptor must be present per image in the
|
||||||
|
Data Stream. **An unlimited number of images may be present per Data
|
||||||
|
Stream.**"
|
||||||
|
- **The Data Stream itself.** The grammar in Appendix B is
|
||||||
|
`<GIF Data Stream> ::= Header <Logical Screen> <Data>* Trailer`, and the
|
||||||
|
spec states "the entity Data … may be repeated any number of times,
|
||||||
|
including 0 times." There is **no field anywhere that carries a file size,
|
||||||
|
byte count, frame count, or total-length value**. §13 (Block Sizes) only
|
||||||
|
defines sizes *within* blocks.
|
||||||
|
|
||||||
|
### 2.3 Verdict
|
||||||
|
|
||||||
|
**The GIF89a specification defines no maximum file size.** The only hard
|
||||||
|
bounds are per-field: 65535×65535 pixels per screen/image, 255 bytes per
|
||||||
|
sub-block, 12 bits per LZW code, and one trailer byte. A compliant decoder
|
||||||
|
must process whatever stream the blocks describe. Any byte ceiling a
|
||||||
|
particular GIF actually hits is therefore *implicit* — 16-bit dimensions,
|
||||||
|
LZW code width, decoder memory, or an external policy — never something the
|
||||||
|
format itself enforces. `[INFERENCE]` This is why real-world GIFs cap out at
|
||||||
|
"a few GB at most" and every service that wants a bound has to impose one
|
||||||
|
itself (see §6; Wikimedia explicitly documents that a 4 GiB host limit was a
|
||||||
|
storage-representation artifact of 32-bit integers, `phab:T191805`, not a
|
||||||
|
format limit).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Theoretical worst case
|
||||||
|
|
||||||
|
### 3.1 Single frame, maximal canvas, 8-bit pixels
|
||||||
|
|
||||||
|
| Quantity | Value | Derivation |
|
||||||
|
|---|---|---|
|
||||||
|
| Max pixels | 4,294,836,225 | 65535 × 65535 |
|
||||||
|
| Raw 8-bit palette-index raster | 4,294,836,225 B ≈ **4.29 GB / 4.00 GiB** | 1 byte per pixel (Table Based Image Data, §22; Go's `image.Paletted` uses exactly 1 byte/pixel) |
|
||||||
|
| LZW worst case | ≈ **6.44 GB / 6.00 GiB** | codes ≤ 12 bits each (Appendix F), at most ~1 code per pixel for incompressible data → ≤ 12 bits/px = 1.5 B/px → 4,294,836,225 × 1.5 B |
|
||||||
|
| Sub-block overhead | ≈ +25.3 MB | every ≤255-byte chunk carries a 1-byte size field (§15): ⌈6,442,254,338 / 255⌉ ≈ 25,263,743 size bytes, + 1 block terminator |
|
||||||
|
| Fixed overhead | ≈ +1.6 KB | header 6 B (§17) + logical screen descriptor 7 B (§18) + global color table ≤ 768 B (§19) + image descriptor 10 B (§20) + local color table ≤ 768 B (§21) + LZW minimum code size 1 B (§22) |
|
||||||
|
|
||||||
|
So a **single maximal-frame GIF cannot exceed ≈ 6.47 GB on the wire**
|
||||||
|
(12-bit LZW bound), and LZW being lossless means the real byte count depends
|
||||||
|
entirely on image content — the same canvas can be a few KB (flat color) or
|
||||||
|
~6 GB (noise).
|
||||||
|
|
||||||
|
Two caveats, both marked `[INFERENCE]`:
|
||||||
|
|
||||||
|
- The "1.5 B/px" figure assumes ~one emitted code per pixel. An encoder is
|
||||||
|
permitted to emit a Clear code at any point (Appendix F: "The Clear code
|
||||||
|
can appear at any point in the image data stream"), so a
|
||||||
|
pathological-but-compliant encoder emitting clear+pixel per pixel reaches
|
||||||
|
~24 bits/px ≈ 12.9 GB for the max canvas. Real encoders do not do this;
|
||||||
|
12-bit/px is the practical bound.
|
||||||
|
- The spec's deferred-clear note (cover sheet) explicitly allows an encoder
|
||||||
|
to keep using a full table at 12-bit codes without clearing, so the 12-bit
|
||||||
|
cap holds for the whole stream, it cannot "grow" past 12 bits.
|
||||||
|
|
||||||
|
### 3.2 Animated GIFs: unbounded
|
||||||
|
|
||||||
|
Every frame is one Image Descriptor, each bounded by the 65535² canvas, but
|
||||||
|
the *count* of frames is unbounded (§2.2). Total bytes = sum over frames —
|
||||||
|
therefore **there is no finite maximum byte size for an animated GIF** in
|
||||||
|
the format. The only thing that stops a real one is decoder memory, a
|
||||||
|
service cap, or disk space. `[INFERENCE]` This is the category the issue #71
|
||||||
|
cover falls into: it is an animated GIF (NETSCAPE2.0 loop extension found at
|
||||||
|
offset 0x310 of the file, verified 2026-08-17 by a ranged GET), and its
|
||||||
|
8,571,192 bytes are ~2.04× the current 4 MiB backend cap.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Decoder-side real limits
|
||||||
|
|
||||||
|
### 4.1 Go `image/gif` (the backend's decoder path, stdlib)
|
||||||
|
|
||||||
|
Source: `/usr/local/go/src/image/gif/reader.go`, Go 1.26.5.
|
||||||
|
|
||||||
|
- Dimensions are read as little-endian uint16 — `left/top/width/height :=
|
||||||
|
int(d.tmp[N]) + int(d.tmp[N+1])<<8` (reader.go:490-493) — so the format
|
||||||
|
ceiling 65535 applies, and nothing smaller is enforced.
|
||||||
|
- The only geometric check is that each frame fits inside the logical
|
||||||
|
screen: `if left+width > d.width || top+height > d.height` →
|
||||||
|
`errors.New("gif: frame bounds larger than image bounds")` (reader.go:512-513).
|
||||||
|
- **There is no file-size, byte-count, frame-count, or pixel-count guard.**
|
||||||
|
Each frame allocates `image.NewPaletted(...)` (reader.go:515) — a
|
||||||
|
`[]byte` of width×height — so decoding one legal 65535² frame attempts a
|
||||||
|
**~4.29 GB allocation**. `DecodeAll` (reader.go:603-605) additionally
|
||||||
|
retains every frame's `Pix` slice for the lifetime of the returned `*GIF`.
|
||||||
|
- `[INFERENCE]` On the 1974 MiB swapless VPS (root AGENTS.md), decoding such
|
||||||
|
a file would OOM rather than error cleanly; nothing in stdlib protects
|
||||||
|
the process. This matters for §7: the backend stores cover bytes without
|
||||||
|
decoding them (see §5.3), so the fetch path never triggers this — but any
|
||||||
|
future "validate/re-encode server-side" scheme would.
|
||||||
|
- Grep for `MaxInt|limit|too large|bounds` in reader.go: the only hits are
|
||||||
|
the frame-bounds check above and the `tmp [1024]byte` scratch buffer
|
||||||
|
(reader.go:109); no size caps exist.
|
||||||
|
|
||||||
|
### 4.2 giflib / libgif
|
||||||
|
|
||||||
|
**Not verified from source.** On 2026-08-17 the giflib sources were not
|
||||||
|
reachable from this network: `github.com/giflib/giflib` returns 404 (repo
|
||||||
|
gone/moved), `gitlab.com/giflib/giflib/-/raw/...` answers a Cloudflare
|
||||||
|
"Just a moment…" challenge, and the SourceForge project download path
|
||||||
|
404s. No limit claim about giflib is made here. `[INFERENCE]` giflib is
|
||||||
|
widely known to be allocation-driven with no dimension cap, but that is not
|
||||||
|
checked against source and is not needed for issue #71 (the backend uses Go
|
||||||
|
stdlib, not giflib).
|
||||||
|
|
||||||
|
### 4.3 Chromium (browser behaviour, first-party source)
|
||||||
|
|
||||||
|
- `third_party/blink/renderer/platform/image-decoders/gif/gif_image_reader.cc`
|
||||||
|
(via `chromium.googlesource.com/chromium/src/+/main/...`, fetched
|
||||||
|
2026-08-17): **no GIF byte-size or dimension cap found** — grep for
|
||||||
|
`max|limit|too large|dimension|65535|overflow` matches only license text.
|
||||||
|
- The base `ImageDecoder` caps *decoded memory*, not transfer size:
|
||||||
|
`CalculateMaxDecodedBytes` computes `min(4 * num_pixels, platform_max_decoded_bytes)`
|
||||||
|
(8 bytes/pixel for high-bit-depth), and the header comment says "Ignoring
|
||||||
|
this limit can cause excessive memory use or even crashes on low-memory
|
||||||
|
devices"
|
||||||
|
(`image_decoder.cc:94-117`, `image_decoder.h:545-549`). The GIF reader
|
||||||
|
itself is untouched by this — it is a decoded-buffer budget.
|
||||||
|
- Practical consequence `[INFERENCE]`: a browser will happily download and
|
||||||
|
store a multi-GB GIF from its own cache perspective; Chromium only limits
|
||||||
|
what it *decodes* into pixels.
|
||||||
|
|
||||||
|
### 4.4 Firefox
|
||||||
|
|
||||||
|
`image/decoders/nsGIFDecoder2.cpp` (via `hg.mozilla.org/mozilla-central/
|
||||||
|
raw-file/tip/...`, fetched 2026-08-17): **no dimension or size limit**; the
|
||||||
|
only guards are on LZW code width (`MAX_BITS` = 12, "maximum codeword size
|
||||||
|
of 12 bits") and the decode stack. Nothing bounds the on-disk byte size.
|
||||||
|
|
||||||
|
### 4.5 Summary
|
||||||
|
|
||||||
|
No mainstream decoder enforces a byte-size ceiling; they stop at the 16-bit
|
||||||
|
dimension ceiling (Go, by construction) or at decoded-memory budgets
|
||||||
|
(Chromium) or nowhere (Firefox). A GIF's byte size is policed only by
|
||||||
|
*storage* policies — which is what §6 calibrates and §7 sets.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Practical distribution — what real manga covers weigh
|
||||||
|
|
||||||
|
Probed **2026-08-17** with `curl -sI` (HEAD) and ranged GETs, desktop Chrome
|
||||||
|
UA. No Cloudflare block on any request. Sample = covers *as the backend
|
||||||
|
would fetch them* (the `og:image`/page-listed cover URL), not thumbnails we
|
||||||
|
chose by hand.
|
||||||
|
|
||||||
|
### 5.1 Exact commands
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# asurascans.com — harvest cover URLs from the homepage, then HEAD each
|
||||||
|
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0" https://asurascans.com/ -o home.html
|
||||||
|
grep -oE 'https://cdn\.asurascans\.com/asura-images/covers/[^"&\\< ]+\.(webp|gif|jpg|jpeg|png)' home.html \
|
||||||
|
| sort -u | grep -v '\-400\.' | head -25 > sample.txt # one full-res cover per series, no -400 thumbs
|
||||||
|
while read -r u; do curl -s -A "…Chrome/126.0" -I "$u" | tr -d '\r' \
|
||||||
|
| grep -iE '^content-length:'; done < sample.txt
|
||||||
|
|
||||||
|
# demonicscans.org — covers live on readermc.org (ADR-0007), URLs contain spaces/UTF-8
|
||||||
|
curl -s -A "…Chrome/126.0" https://demonicscans.org/ -o demonic.html
|
||||||
|
grep -oE 'src="https://readermc\.org/images/thumbnails/[^"]+"' demonic.html | tr -d 'src="' > demonic.txt
|
||||||
|
# …plus og:image from 5 manga pages (Catastrophic-Necromancer, Magic-Emperor, …)
|
||||||
|
# each URL percent-encoded per path segment (urllib.parse.quote, safe=':/') before HEAD
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 asurascans — 25 full-res covers (mixed formats)
|
||||||
|
|
||||||
|
Homepage fetched 200 (664,700 B). All 25 returned `200` with a real
|
||||||
|
`Content-Length`. Distribution:
|
||||||
|
|
||||||
|
| Statistic | Bytes |
|
||||||
|
|---|---|
|
||||||
|
| n | 25 |
|
||||||
|
| min | 190,410 |
|
||||||
|
| median | 1,275,082 |
|
||||||
|
| p90 | 4,524,788 |
|
||||||
|
| max | 8,571,192 |
|
||||||
|
| mean | 1,943,651 |
|
||||||
|
| **> 4 MiB (4,194,304)** | **3 (12%)** — `a-dragonslayers-peerless-regression.gif` 8,571,192 (the issue #71 cover, animated: NETSCAPE2.0 at 0x310, 550×733, 256 colors); `bad-born-blood.3008f6.webp` 4,524,788 `image/jpeg`; `ending-maker.cfbf53.webp` 4,619,303 `image/jpeg` |
|
||||||
|
|
||||||
|
Notes: the CDN serves `Content-Type` by stored bytes, not by URL extension
|
||||||
|
(the `.webp` URLs return `image/png`, `image/jpeg`, or `image/webp` — the
|
||||||
|
sample spans all four of `png/jpeg/webp/gif`). Two of the three over-cap
|
||||||
|
files are **not GIFs**, so the current 4 MiB cap already silently drops 12%
|
||||||
|
of asura covers of any format. p90 itself (4.52 MB) exceeds the cap.
|
||||||
|
|
||||||
|
### 5.3 demonicscans — 78 covers on readermc.org
|
||||||
|
|
||||||
|
78 unique cover URLs (73 from the homepage's `/images/thumbnails/` plus 5
|
||||||
|
`og:image` values from manga pages — demonicscans publishes the thumbnail
|
||||||
|
file as the full cover, so that is exactly what the backend would fetch).
|
||||||
|
**78/78 returned 200 with a real Content-Length** (spaces and UTF-8 in the
|
||||||
|
filenames were percent-encoded per path segment; the homepage's raw HTML
|
||||||
|
carries `’`-style mojibake for curly quotes, which was repaired by
|
||||||
|
latin-1→utf-8 re-encoding before probing).
|
||||||
|
|
||||||
|
| Statistic | Bytes |
|
||||||
|
|---|---|
|
||||||
|
| n | 78 |
|
||||||
|
| min | 13,298 |
|
||||||
|
| median | 63,061 |
|
||||||
|
| p90 | 206,994 |
|
||||||
|
| max | 801,200 |
|
||||||
|
| mean | 110,038 |
|
||||||
|
| > 4 MiB | 0 |
|
||||||
|
|
||||||
|
### 5.4 Reading
|
||||||
|
|
||||||
|
`[INFERENCE]` asurascans covers are the heavy tail (median 1.3 MB, top
|
||||||
|
decile > 4 MiB, occasional ~5–9 MB), demonicscans covers are tiny (all
|
||||||
|
< 0.8 MB). A cover cap must be chosen against the *asura* distribution —
|
||||||
|
the 8.57 MB animated GIF is not a freak one-off outlier; the 90th
|
||||||
|
percentile already crosses 4 MiB and two JPEGs sit between 4.5–4.7 MB.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Comparable documented byte caps (first-party docs only)
|
||||||
|
|
||||||
|
| Service | Cap | Source (fetched 2026-08-17) |
|
||||||
|
|---|---|---|
|
||||||
|
| GitHub (issues/PR comments) | **10 MB for images and gifs**; 25 MB other files; 10/100 MB video | `https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/attaching-files` — "The maximum file size is: 10MB for images and gifs … 25MB for all other files" |
|
||||||
|
| Discord (API uploads) | default **10 MiB per file**, higher with Nitro / boost tier | `https://discord.com/developers/docs/reference#uploading-files` — "The file upload size limit applies to each file in a request. The default limit is `10 MiB` for all users" (help-center article `support.discord.com/hc/en-us/articles/115002935588` exists but answered 403 from this network on the probe date, so its figures were not verified here) |
|
||||||
|
| Wikimedia Commons | **100 MiB** upload limit; hosting up to **5 GiB**; GIF thumbnails limited to **100 megapixels**; prior 4 GiB host cap was a 32-bit storage artifact (phab:T191805) | `https://commons.wikimedia.org/wiki/Commons:Maximum_file_size` |
|
||||||
|
| MDN | nothing — MDN documents no byte-size limit for images; browsers impose none (see §4.3–4.4) | `[INFERENCE]` from absence in the platform docs read in §4 |
|
||||||
|
|
||||||
|
Calibration takeaway: two major platforms independently land on **~10 MB**
|
||||||
|
as the ceiling for an uploadable image/GIF (GitHub exactly 10 MB, Discord
|
||||||
|
exactly 10 MiB), with Wikimedia the outlier at 100 MiB/5 GiB because it is a
|
||||||
|
media *archive*. A 10 MiB cover cap is therefore squarely inside industry
|
||||||
|
normal.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Recommendation for issue #71
|
||||||
|
|
||||||
|
**Raise the cover cap to 10 MiB (10,485,760 B) — as a separate constant, not
|
||||||
|
by moving the shared one.**
|
||||||
|
|
||||||
|
Why:
|
||||||
|
|
||||||
|
- **Fits the measured reality.** The largest observed cover is 8,571,192 B
|
||||||
|
(the issue's animated GIF) = 82% of 10 MiB; 10 MiB covers **100% of the
|
||||||
|
103 sampled covers** and the *entire* asura distribution, including its
|
||||||
|
heavy tail. 4 MiB rejects 12% of asura covers (two of them plain JPEGs).
|
||||||
|
- **Matches industry calibration** (§6): GitHub 10 MB images/GIFs, Discord
|
||||||
|
10 MiB default. A 10 MiB cap is a number every engineer recognizes, and
|
||||||
|
it leaves ~18% headroom over the current worst observed file.
|
||||||
|
- **Costs little on the target hardware.** The backend buffers cover bytes
|
||||||
|
whole during fetch (`backend/internal/latest/cover.go`: `ContentLength >
|
||||||
|
maxBodyBytes` rejection at :155, then `io.ReadAll(io.LimitReader(…,
|
||||||
|
maxBodyBytes+1))` at :158) and loads the full body per `GET /covers/…`
|
||||||
|
(`backend/internal/api/handlers.go`, `Cover` → `w.Write(body)`). Worst
|
||||||
|
case per concurrent fetch **+** serve is therefore 2 × cap = 20 MiB; even
|
||||||
|
ten of each concurrently is ~200 MiB of a 1974 MiB swapless VPS (~10%),
|
||||||
|
and the browser unit (471 MiB, root AGENTS.md) is no longer on that box.
|
||||||
|
The 4 MiB series-page cap is *not* the issue — measured pages run
|
||||||
|
100 KB–1.2 MB (`backend/internal/latest/fetch.go` comment) — so keep it.
|
||||||
|
- **The cap is a separate knob.** Today one `const maxBodyBytes = 4 << 20`
|
||||||
|
(`backend/internal/latest/fetch.go:17`) gates *both* series pages and
|
||||||
|
covers (`cover.go` references it). Raising it wholesale would loosen the
|
||||||
|
page-side memory guard for no benefit; a cover-specific constant (e.g.
|
||||||
|
`maxCoverBytes = 10 << 20`) keeps the two policies independent. The fetch
|
||||||
|
already double-checks `ContentLength` and the post-`LimitReader` length,
|
||||||
|
so a larger constant changes nothing else.
|
||||||
|
|
||||||
|
Alternatives and their costs:
|
||||||
|
|
||||||
|
| Option | Cost |
|
||||||
|
|---|---|
|
||||||
|
| Keep 4 MiB | 12% of asura covers (incl. non-GIF JPEGs) never stored — current bug, silent missing covers. |
|
||||||
|
| 16 MiB cap | 2× headroom over the observed max for future GIFs; +60% worst-case transient memory vs 10 MiB; diverges from the GitHub/Discord 10 MB calibration. |
|
||||||
|
| Server-side re-encode / downscale covers | Requires decoding → Go `image/gif` allocates width×height per frame with **no guard** (§4.1); a legal 65535² GIF forces a ~4.29 GB allocation on a 1974 MiB swapless box — OOM, not an error. Also mutates bytes, which the store treats as immutable/content-addressed (ADR-0007). Highest risk, no upside at this scale. |
|
||||||
|
| No cap | Unbounded transient memory and disk; rejected outright. |
|
||||||
|
|
||||||
|
Decision is the user's; on the evidence, **10 MiB for covers, 4 MiB for
|
||||||
|
pages** is the defensible middle.
|
||||||
+94
-84
@@ -1,93 +1,103 @@
|
|||||||
Guidance for OpenCode (and Claude Code) working under `userscript/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
Scope: `userscript/`.
|
||||||
|
|
||||||
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
Each entry names the code that holds the truth. The prose is only what the code
|
||||||
|
cannot tell you: rationale, invariants a refactor would break, and dated
|
||||||
|
observations about sites we don't control.
|
||||||
|
|
||||||
1. **Site adapters** — one per host, `detect(location, document)` return page `type` + IDs. Identify type/IDs from **URL regex** (most stable); pull `title` from **`og:title`** (or the page heading where a site ships no og: tags), not CSS classes. **No adapter reads a cover**: the backend acquires, stores and serves every Cover from its own origin (ADR-0007), the wire's `cover` is already an address on our origin, and `apiPut` strips any `cover` off an outgoing body.
|
### Structure — single IIFE, `manga-bookmark.user.js`
|
||||||
2. **API client** — `apiGet/apiPut/apiDelete` with bearer header; `localStorage` key `bmgr:manga:cache` for instant render + offline fallback.
|
|
||||||
3. **Progress logic** — auto-upsert `last_chapter` only when `chapterNum >= stored last_chapter_num` (re-reading old chapters must not regress progress; unparseable -> set current). Manual panel override forces any value.
|
|
||||||
4. **Retry queue** — every write go through `pushBookmark`/`pushDelete`, so
|
|
||||||
failed mutation park in `localStorage` (`bmgr:manga:queue`) and replayed on
|
|
||||||
next navigation, reconnect, or `refresh()`. Entries are markers
|
|
||||||
(`{key, op, sendStatus, attempts}`), never payloads — body read from
|
|
||||||
cache at send time, so one entry per key give ordering and coalescing for
|
|
||||||
free. `sendStatus` is **sticky**: while archive pending, later writes to
|
|
||||||
that key keep carrying bucket, which stop successful
|
|
||||||
in-between write from silently un-archiving series. `refresh()` drains
|
|
||||||
before it fetches and overlays anything still pending, so list never
|
|
||||||
flaps. 400 drops entry, 401 abort pass and keep queue, and
|
|
||||||
transient failures retry to cap of 10. Latest-chapter writes deliberately
|
|
||||||
stay out of queue. See
|
|
||||||
`docs/superpowers/specs/2026-07-27-offline-retry-queue-design.md`.
|
|
||||||
5. **UI** — rendered inside **Shadow DOM** root to isolate from site CSS
|
|
||||||
(critical on mobile). Three tabs (All / Favourites / Archived) and row of
|
|
||||||
link chips to web UI and both manga sites; `WEB_BASE` sits in CONFIG
|
|
||||||
block next to `API_BASE`. FAB is `7 × 44` edge tab whose *hit* area
|
|
||||||
widened to `28 × 72` by invisible `#hit` child; `#fab` must keep
|
|
||||||
`touch-action: none` and must **not** regain `overflow: hidden`. Since
|
|
||||||
`touch-action` resolved at gesture start, strip can't be both
|
|
||||||
browser-scrolled and script-dragged, so `makeDraggable` splits by intent: swipe
|
|
||||||
from `#hit` scrolls via `window.scrollBy`, hold of `ARM_MS` arms
|
|
||||||
reposition drag, visible sliver drags with no hold. See
|
|
||||||
`docs/superpowers/specs/2026-07-28-edge-tab-hitbox-design.md`.
|
|
||||||
6. **SPA navigation** — Asura is Astro, client-routed on comic/chapter pages: patch `history.pushState`/`replaceState` + listen `popstate`, re-run `detect()` on URL change so auto-update fire without reload. Demonic uses classic reloads (initial `document-idle` run suffice).
|
|
||||||
|
|
||||||
### Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust)
|
Six parts, in file order: site adapters, API client, progress logic, retry
|
||||||
|
queue, UI, SPA navigation.
|
||||||
|
|
||||||
- **asurascans.com**: series `/comics/<slug>` (slug carries trailing
|
**Site adapters** — one per host, `detect(location, document)` returning page
|
||||||
site-wide build-hash suffix, e.g. `-059befe1`, that **rotates on every
|
`type` + IDs.
|
||||||
redeploy**), chapter `/comics/<slug>/chapter/<n>`. `seriesId` must strip
|
|
||||||
hash (`/-[0-9a-f]{8}$/`, `stripBuildHash` in userscript,
|
- Identify type and IDs from **URL regex**, which is the most stable surface a
|
||||||
`asuraBuildHash` in backend); URLs keep full slug — stale-hash
|
site exposes; take `title` from **`og:title`** (or the page heading where a
|
||||||
URLs 302 to current ones. Astro-rendered; chapter links present in raw
|
site ships no og: tags), never CSS classes.
|
||||||
server HTML.
|
- **No adapter reads a cover.** The backend acquires, stores and serves every
|
||||||
- **demonicscans.org**: series `/manga/<slug>` (slug may URL-encode punctuation, e.g. `%2527` for `'`), chapter `/title/<slug>/chapter/<n>/<page>` (older `chaptered.php?manga=<id>&chapter=<n>` form still exists as redirect, what series-page chapter-list anchors link through).
|
Cover from its own origin, the wire `cover` is already an address there, and
|
||||||
Encodings (incl. triple-encoded punctuation like `%25252D`) identical
|
`apiPut` strips any `cover` off an outgoing body.
|
||||||
on /manga/ and /title/ pages, so decode-once seriesIds match — verified
|
|
||||||
2026-07-28.
|
**Progress logic** — auto-upsert `last_chapter` only when
|
||||||
- **comix.to**: series `/title/<id>-<slug>`, chapter
|
`chapterNum >= stored last_chapter_num`; unparseable sets the current value.
|
||||||
`/title/<id>-<slug>/<uploadId>-chapter-<n>`. Only the leading `<id>` is
|
Re-reading an old chapter must not regress progress. A manual panel override
|
||||||
identity — the slug re-renders when a series is renamed (`comixSeriesId`).
|
forces any value.
|
||||||
An SPA that **never rewrites `og:title`**: the server-rendered head keeps
|
|
||||||
whatever document loaded first, so on a cold load `og:title` is the homepage's
|
**Retry queue** — every write goes through `pushBookmark`/`pushDelete`.
|
||||||
"Comix — Read Comics online for free" and after an in-page hop it is the
|
|
||||||
*previous* series' name. `document.title` is the one thing client routing does
|
- Entries are markers (`{key, op, sendStatus, attempts}`), **never payloads**:
|
||||||
update, so titles come from there, with the chapter page's `" · Ch.<n>"` tail
|
the body is read from cache at send time, so one entry per key gives ordering
|
||||||
stripped. It publishes no `og:image` either, which is one of the reasons cover
|
and coalescing for free.
|
||||||
|
- `sendStatus` is **sticky** — while an archive is pending, later writes to that
|
||||||
|
key keep carrying the bucket. Without it a successful in-between write
|
||||||
|
silently un-archives the series.
|
||||||
|
- `refresh()` drains before it fetches and overlays anything still pending, so
|
||||||
|
the list never flaps.
|
||||||
|
- 400 drops the entry, 401 aborts the pass and keeps the queue, transient
|
||||||
|
failures retry to a cap. Latest-chapter writes deliberately stay out of the
|
||||||
|
queue.
|
||||||
|
|
||||||
|
**UI** — rendered inside a **Shadow DOM** root to isolate it from site CSS,
|
||||||
|
which is critical on mobile.
|
||||||
|
|
||||||
|
- The FAB's *hit* area is widened by an invisible `#hit` child. `#fab` must keep
|
||||||
|
`touch-action: none` and must **not** regain `overflow: hidden`.
|
||||||
|
- `touch-action` is resolved at gesture start, so the strip cannot be both
|
||||||
|
browser-scrolled and script-dragged. `makeDraggable` therefore splits by
|
||||||
|
intent: a swipe from `#hit` scrolls via `window.scrollBy`, a hold of `ARM_MS`
|
||||||
|
arms a reposition drag, and the visible sliver drags with no hold.
|
||||||
|
|
||||||
|
**SPA navigation** — Asura is Astro and client-routes on comic/chapter pages, so
|
||||||
|
`history.pushState`/`replaceState` are patched and `popstate` listened to, and
|
||||||
|
`detect()` re-runs on URL change. Demonic uses classic reloads, where the
|
||||||
|
initial `document-idle` run suffices.
|
||||||
|
|
||||||
|
### Live URL shapes
|
||||||
|
|
||||||
|
Encoded in the adapters; the notes below are the parts a reader of the regex
|
||||||
|
would get wrong. **Verified 2026-07-26 unless dated otherwise — sites drift, so
|
||||||
|
re-check against a live page before trusting any of it.**
|
||||||
|
|
||||||
|
- **asurascans.com** — the series slug carries a site-wide build-hash suffix
|
||||||
|
(e.g. `-059befe1`) that **rotates on every redeploy**, so `seriesId` must
|
||||||
|
strip it (`stripBuildHash` here, `asuraBuildHash` in the backend) while URLs
|
||||||
|
keep the full slug — stale-hash URLs 302 to current ones.
|
||||||
|
- **demonicscans.org** — slugs may URL-encode punctuation, and the older
|
||||||
|
`chaptered.php?manga=<id>&chapter=<n>` form still exists as a redirect, which
|
||||||
|
is what series-page chapter-list anchors link through. Encodings (including
|
||||||
|
triple-encoded punctuation like `%25252D`) are identical on `/manga/` and
|
||||||
|
`/title/` pages, so decode-once seriesIds match (verified 2026-07-28).
|
||||||
|
- **comix.to** — only the leading `<id>` is identity; the slug re-renders when a
|
||||||
|
series is renamed (`comixSeriesId`). It is an SPA that **never rewrites
|
||||||
|
`og:title`**: the server-rendered head keeps whatever document loaded first,
|
||||||
|
so on a cold load `og:title` is the homepage's name and after an in-page hop
|
||||||
|
it is the *previous* series'. `document.title` is the one thing client routing
|
||||||
|
updates, hence titles come from there with the chapter page's `" · Ch.<n>"`
|
||||||
|
tail stripped. It publishes no `og:image` either, one of the reasons cover
|
||||||
acquisition moved to the backend.
|
acquisition moved to the backend.
|
||||||
- **kagane.to**: series `/series/<uuid>`, reader
|
- **kagane.to** — reader URLs carry no chapter number, so the number comes out
|
||||||
`/series/<uuid>/reader/<bookUuid>`. Reader URLs carry no chapter number, so
|
of `og:title`. Two shapes exist, `"<Series> - Chapter <n>[ - Episode <n>]"`
|
||||||
the number comes out of `og:title`. Two shapes exist: `"<Series> - Chapter
|
and `"<Series> - Volume <v> Chapter <n>"`; both must yield a bare series
|
||||||
<n>[ - Episode <n>]"` and, for volume-numbered series, `"<Series> - Volume <v>
|
title, or the volume tail lands in the bookmark's title. Its covers are
|
||||||
Chapter <n>"` with no episode name — both must yield a bare series title, or
|
challenge- and CORP-protected, so nothing outside kagane.to can load one —
|
||||||
the volume tail lands in the bookmark's title.
|
the panel renders the backend's cover address like every other Site.
|
||||||
Its covers are challenge- and CORP-protected, so nothing outside kagane.to can
|
- **novelfull.com** (novel script) — no `og:*` tags at all, so the title comes
|
||||||
load one directly; the panel renders the backend's own cover address like every
|
from `h3.title` (series) or `a.truyen-title` (chapter).
|
||||||
other Site. Behind a Cloudflare JS challenge, so the backend polls it
|
- **lightnovelworld.net** (novel script) — chapter paths are flat at the site
|
||||||
through the headless browser.
|
root and their slug is a **Chapter Slug, not an identity**: a Series may
|
||||||
- **novelfull.com** (novel script): series `/<slug>.html`, chapter
|
publish under several. The Series address is read off the page's
|
||||||
`/<slug>/chapter-<n>[-<title-slug>].html`. No `og:*` tags at all — title from
|
|
||||||
`h3.title` (series) or `a.truyen-title` (chapter); the script reads no cover.
|
|
||||||
Behind a Cloudflare JS challenge no TLS fingerprint
|
|
||||||
clears, so the backend polls it through the headless browser.
|
|
||||||
- **lightnovelworld.net** (novel script): series `/novel/<slug>/`, chapter
|
|
||||||
`/<slug>-chapter-<n>/` — flat, at the site root. The chapter path's slug is a
|
|
||||||
Chapter Slug, not an identity: the Series address is read off the page's
|
|
||||||
`a[aria-label='All Chapter']` (fallback: the BreadcrumbList's second crumb),
|
`a[aria-label='All Chapter']` (fallback: the BreadcrumbList's second crumb),
|
||||||
and a Series may publish under several Chapter Slugs. A chapter page with no
|
and a chapter page with no pointer resolves to `other` so no Bookmark is
|
||||||
pointer resolves to `other`, so no Bookmark is offered. `h1.entry-title` is
|
offered. The client runs **no latest-chapter scan** for this Site —
|
||||||
the clean title on a series page and `<Title> Chapter <n>` on a chapter page.
|
`computeLatestChapter` yields null and `backgroundRefreshLatest` skips it
|
||||||
Its series page lists every chapter with an
|
before any fetch — because the backend Poll's one-hour cooldown dominates the
|
||||||
absolute href, so the backend polls it with the plain TLS client.
|
client's four-hour throttle, so a scan would add no freshness while having to
|
||||||
The client performs no latest-chapter scan for this Site: the Poll's
|
truncate at the page's wpdiscuz thread, a public write surface.
|
||||||
one-hour cooldown dominates the client's four-hour throttle, so a scan
|
|
||||||
would add no freshness, and the page's wpdiscuz thread is a public write
|
|
||||||
surface a scan would have to truncate at. `computeLatestChapter` yields
|
|
||||||
null here and `backgroundRefreshLatest` skips the Site before any fetch.
|
|
||||||
|
|
||||||
### Second script: `novel-bookmark.user.js`
|
### Second script — `novel-bookmark.user.js`
|
||||||
|
|
||||||
A copy of the manga script with two adapters, `LIBRARY = "novel"` and
|
A copy of the manga script with two adapters, `LIBRARY = "novel"` and
|
||||||
`STORE_PREFIX = "bmgr:novel:"`. No migration loop (this script has no previous
|
`STORE_PREFIX = "bmgr:novel:"`. No migration loop, because this script has no
|
||||||
installation to carry keys over from). Installed alongside the manga script;
|
previous installation to carry keys over from. Installed alongside the manga
|
||||||
both write to the same backend with the same `LIBRARY` column discriminating
|
script; both write to the same backend, discriminated by `LIBRARY`.
|
||||||
them.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user