Compare commits

..

14 Commits

Author SHA1 Message Date
sulthan c545b1b92e chore: address final review must-fix items 2026-08-06 03:34:41 +07:00
sulthan 50e4d6a6e2 feat: serve novel-bookmark.user.js and allow the novel sites' origins 2026-08-06 03:29:24 +07:00
sulthan c9a2fd4614 feat(userscript): add novel-bookmark.user.js for novelfull and lightnovelworld 2026-08-06 03:26:03 +07:00
sulthan 51fcd8732b feat(userscript): stamp and filter by kind in the manga script 2026-08-06 03:22:28 +07:00
sulthan d465028443 feat(web): split the UI into manga and novel libraries 2026-08-06 03:19:49 +07:00
sulthan f24924031e feat(web): apply design snapshot — libswitch, novel site colours, login art 2026-08-06 03:15:36 +07:00
sulthan 92f1fbf6ec feat(latest): poll novelfull via headless browser, lightnovelworld via TLS 2026-08-06 03:10:50 +07:00
sulthan c81e50c7f1 feat(latest): parse newest chapter for novelfull and lightnovelworld 2026-08-06 03:07:22 +07:00
sulthan d9d7a1c00d feat(api): validate kind on PUT /bookmarks 2026-08-06 03:05:14 +07:00
sulthan dcec12ae72 feat(store): add kind column splitting manga and novel libraries 2026-08-06 03:03:43 +07:00
sulthan 44a8df43e1 refactor: rebrand compose, env and docs to BookmarkManager 2026-08-06 02:55:46 +07:00
sulthan d41a1d2c0b refactor: rebrand userscript storage keys, hosts and wordmark 2026-08-06 02:47:33 +07:00
sulthan f30d4cc7cb refactor: rename Go module to bookmarkmanager 2026-08-06 02:44:43 +07:00
sulthan 3149dc0c26 chore: pre-flight baseline for novel-support plan
- .gitignore: drop .superpowers/ and local go.work
- AGENTS.md, docs/design-system.md, backend/AGENTS.md, userscript/AGENTS.md
- login-art.png: 2.8MB design asset, picked up by //go:embed
2026-08-06 02:43:29 +07:00
33 changed files with 1096 additions and 1729 deletions
-8
View File
@@ -9,14 +9,6 @@ API_TOKEN=changeme-generate-a-long-random-token
# sites' hostnames change. # sites' hostnames change.
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net
# Password for the bundled Postgres container, and therefore half of the
# DATABASE_URL compose builds for the backend. Generate one:
# openssl rand -hex 24
POSTGRES_PASSWORD=changeme-generate-a-long-random-password
# Override only to point the backend at a Postgres compose does not run.
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
# --- Prod override (Traefik) only --- # --- Prod override (Traefik) only ---
# Subdomain Traefik routes to this service (required by the prod override). # Subdomain Traefik routes to this service (required by the prod override).
# BOOKMARK_API_HOST=bookmark-api.example.com # BOOKMARK_API_HOST=bookmark-api.example.com
+6 -60
View File
@@ -4,28 +4,22 @@ Guidance for OpenCode (and Claude Code) working in this repo.
## What this is ## What this is
Read-progress tracker for two libraries — manga and novels — behind one self-hosted Go backend. Two separate Violentmonkey userscripts inject on-page UI (floating button + slide-in panel) and sync progress, so bookmarks unify across sites and devices: Manga read-progress tracker, user read on **asurascans.com** (current domain; asuracomic.net 301s here) and **demonicscans.org** via **Violentmonkey**. Userscript inject on-page UI (floating button + slide-in panel), sync progress to self-hosted Go backend so bookmarks unify across both sites and devices.
- `manga-bookmark.user.js` — **asurascans.com** (current domain; asuracomic.net 301s here), **demonicscans.org**, **comix.to**, **kagane.to**.
- `novel-bookmark.user.js` — **novelfull.com**, **lightnovelworld.net**.
One backend, one `bookmarks` table: a `kind` column (`manga`|`novel`) splits the libraries and the web UI switches between them. Rows are keyed `<site>:<series_id>`.
## Hard constraints (drive design — don't violate) ## Hard constraints (drive design — don't violate)
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines: Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin. - **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block). - Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
- Every site is its **own origin with its own `localStorage`** — a shared remote store is the only way to unify bookmarks. Cloud sync required, not optional. - Asura and Demonic are **separate origins with separate `localStorage`** — shared remote store only way to unify bookmarks. Cloud sync required, not optional.
- Userscript run in **isolated world**, so embedded API token safe from site's JS. - Userscript run in **isolated world**, so embedded API token safe from site's JS.
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test. - Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, direct probe) before finalizing, not assumed from single earlier test.
- **kagane.to and novelfull.com are the exception to the above** — both sit behind a Cloudflare JavaScript challenge no TLS fingerprint clears, so the backend polls them over CDP (`BROWSER_WS_URL`) and skips them entirely when that's unset. The four other sites poll fine over plain TLS.
## Architecture ## Architecture
``` ```
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache) Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume) -- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
``` ```
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`. Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
@@ -33,11 +27,11 @@ Backend-specific architecture (packages, endpoints, poller, config env vars) liv
## Commands ## Commands
Backend (`cd backend`): Backend (`cd backend`):
- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`). - Test all: `go test ./...`
- Single test: `go test -run TestName ./...` - Single test: `go test -run TestName ./...`
- Build static binary: `CGO_ENABLED=0 go build` - Build static binary: `CGO_ENABLED=0 go build`
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`).
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200. Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
@@ -68,41 +62,9 @@ instantly.
## Security invariants ## Security invariants
Existing guarantees — don't regress:
- Auth on `/bookmarks*`: require `Authorization: Bearer <API_TOKEN>`, **constant-time compare**, 401 otherwise. - Auth on `/bookmarks*`: require `Authorization: Bearer <API_TOKEN>`, **constant-time compare**, 401 otherwise.
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`. - CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
## Secure coding rules (code you write here)
Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example in-tree — match it, don't start a second convention. AI-written backends fail on exactly these: broken access control, injection, weak session/error handling, invented dependencies.
Go backend:
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
- Compare secrets with `hmac.Equal` / `subtle.ConstantTimeCompare`, never `==`. Covers API token, web password, session MAC.
- Errors: generic text to the client (`http.Error(w, "internal error", 500)`), detail to `log.Printf`. Never log `API_TOKEN`, `WEB_PASSWORD`, a session cookie value, or a whole `Authorization` header.
- Proxy headers are trusted only where they already are: `X-Forwarded-Proto` for the Secure cookie flag, **rightmost** `X-Forwarded-For` for client IP (leftmost is attacker-supplied). Don't read either anywhere else.
- Session cookies keep `HttpOnly`, `SameSite`, `Secure`-when-HTTPS, and expiry checked before signature.
- Stdlib crypto only. No hand-rolled hashing, no MD5/SHA-1 anywhere security-bearing.
- Validate at the handler boundary before storing: body capped by `http.MaxBytesReader` (64 KB), empty `key` and unknown `status`/`kind` rejected with `400`. A bad value that reaches the store becomes every later reader's problem.
Userscript:
- Site-derived and stored strings render via `el(..., {text})` / `textContent`. `{html}` and `innerHTML` are for author-written literal markup only (`TEMPLATE`, `CSS`) — never a title, chapter label, or API response field. The page DOM belongs to a third-party site; treat it as attacker-controlled.
- Isolated world protects the token from the site's JS. It does not protect anything from an `innerHTML` sink you add yourself.
- The `API_TOKEN` literal sits in both userscripts and must equal backend `API_TOKEN`. Never copy it into logs, docs, commit messages, issues, or a new file. Rotation touches three places: backend env plus both scripts.
- `fetch()` targets `API_BASE` only — no dynamic origin, no site-supplied URL. `authHeaders()` goes nowhere but the backend.
- `localStorage` is shared with the site's own JS: cache and queue live there, credentials never do.
- Wrap every `localStorage` read/write and `JSON.parse` in try/catch (quota, private mode, corrupt entry), as the existing helpers do.
Dependencies: stdlib first; a new module needs a stated reason. Confirm a package actually exists before adding it — a plausible name may be fiction (~20% of LLM-proposed packages don't resolve, which is how slopsquatting lands). Pin exact versions.
Review gate: auth, CORS, session, crypto, and the fetch gate are security-critical. Editing one is not a drive-by change — say which invariant you preserved and run `go test ./...` before calling it done.
## Comments ## Comments
Comment only if code alone can't carry info. Cost per read — must earn spot. Comment only if code alone can't carry info. Cost per read — must earn spot.
@@ -133,22 +95,6 @@ Test: "competent reader get this from code in few sec?" Yes → skip. Needs deto
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work. `golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
## Agent skills
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
### Issue tracker
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
## graphify ## graphify
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships. Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
-1
View File
@@ -1 +0,0 @@
AGENTS.md
+106
View File
@@ -0,0 +1,106 @@
# CLAUDE.md
Guidance for Claude Code (claude.ai/code) working in this repo.
## What this is
Manga read-progress tracker, user read on **asurascans.com** (current domain; asuracomic.net 301s here) and **demonicscans.org** via **Violentmonkey**. Userscript inject on-page UI (floating button + slide-in panel), sync progress to self-hosted Go backend so bookmarks unify across both sites and devices.
## Hard constraints (drive design — don't violate)
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
- Asura and Demonic are **separate origins with separate `localStorage`** — shared remote store only way to unify bookmarks. Cloud sync required, not optional.
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, or direct probe) before finalize, not assumed from single earlier test.
## Architecture
```
Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
```
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/CLAUDE.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/CLAUDE.md`.
## Commands
Backend (`cd backend`):
- Test all: `go test ./...`
- Single test: `go test -run TestName ./...`
- Build static binary: `CGO_ENABLED=0 go build`
Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`).
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
## Forge: Gitea, not GitHub
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
- Issues: `tea issue create`, `tea issue list`
- Auth lives in `tea login`, not `GH_TOKEN` env var.
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
## Design system
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
— source of truth Claude Design project `BookmarkManager Web UI`
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
state (busy, error, destruction) may use `--ember`; destruction gets
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
only (never hardcode hex outside `:root`), both colour branches touched
together. Any move that pulls series out of list (archive/finish/remove)
must be confirm-gated via its own `.confirm-row`; only restore fires
instantly.
## Security invariants
- Auth on `/bookmarks*`: require `Authorization: Bearer <API_TOKEN>`, **constant-time compare**, 401 otherwise.
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
## Comments
Comment only if code alone can't carry info. Cost per read — must earn spot.
Write for:
- Why not what. Tradeoffs, non-obvious decisions.
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
- Non-local consequence, invisible from function alone.
- Wire format / encoding / interface contract — save callers re-deriving.
- Gotcha/workaround, with ref if exists.
- Domain/business rule not derivable from code.
Skip:
- Restating code (no `// increment i` above `i++`).
- Trivial getter/setter/pass-through.
- Banners, dividers, `// helpers`.
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
- Commented-out code — delete.
- TODO without concrete action.
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
## Relevant skills
`multi-stage-dockerfile` and `docker-compose-orchestration` for container work (referenced in plan).
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
## graphify
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
Rules:
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. Return scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
- If graphify-out/wiki/index.md exists, use for broad navigation instead of raw source browsing.
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain don't surface enough context.
- After modifying code, run `graphify update .` to keep graph current (AST-only, no API cost).
-69
View File
@@ -1,69 +0,0 @@
# Bookmark Manager
Read-progress tracker for serialised fiction. A reader browses third-party manga and
novel sites; userscripts capture where they got to and sync it to a self-hosted backend,
so progress survives across sites and devices.
## Language
**Series**:
One ongoing work — a manga or a novel — as published by a Site. Identified by its
stable slug on that Site, never by its title. A Series exists once and is shared by
every Reader who bookmarks it; it owns the facts that are true regardless of who is
reading — title, cover, Latest Chapter. A Reader cannot change them; they describe the
Series, not anyone's relationship to it.
_Avoid_: manga, title, book, comic
**Site**:
One third-party source a Series is published on. A Series on two Sites is two Series.
_Avoid_: source, host, provider, domain
**Reader**:
A person with their own Progress. Exactly one per set of credentials, so there is no
separate "account" concept to model — the credential belongs to the Reader.
_Avoid_: user, account, member, subscriber
**Bookmark**:
One Reader's tracked relationship with one Series, holding only what differs between
Readers: Progress, Favourite, Lifecycle bucket. Facts about the Series itself belong
to the Series, not here.
_Avoid_: entry, item, record, subscription
**Library**:
One of the two halves of the collection — manga or novel — selected by a Bookmark's
`kind`. The web UI and the userscripts each address exactly one Library at a time.
Not a per-person concept: "everything one person has bookmarked" is a different idea
and must not be called a Library.
_Avoid_: section, tab, category
**Progress**:
The furthest chapter a reader has actually read in a Series. Only a change in Progress
is real activity, so only Progress reorders the list.
_Avoid_: position, bookmark (the noun is taken), last read
**Latest Chapter**:
The newest chapter a Site has published for a Series, discovered without the reader
present. Distinct from Progress in every way that matters: it is a fact about the Site,
not about the reader, and it must never reorder the list.
_Avoid_: newest, current chapter, update
**Poll**:
The backend's own check of a Site for a Series's Latest Chapter, made without the
Reader present. Performed once per Series no matter how many Readers bookmarked it —
a Poll is work done on behalf of the Series, never on behalf of a Reader.
_Avoid_: scrape, refresh, check, sync
**New Chapter**:
The state where Latest Chapter is ahead of Progress. The single condition the ember
accent is permitted to signal.
_Avoid_: unread, update available
**Lifecycle bucket**:
Which of three mutually exclusive states a Bookmark sits in — reading, archived, or
finished. A Bookmark is in exactly one. Orthogonal to being a favourite.
_Avoid_: state, status (as a domain word), list
**Favourite**:
A reader's manual pin on a Bookmark. Orthogonal to the Lifecycle bucket, and never a
reason to reorder the list.
_Avoid_: starred, pinned, priority
+8 -33
View File
@@ -38,14 +38,6 @@ API_TOKEN=<paste output of: openssl rand -hex 32>
# CORS allowlist — leave as-is unless a site changes hostname. # CORS allowlist — leave as-is unless a site changes hostname.
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to
# Required — password for the bundled Postgres container. Compose builds the
# backend's DATABASE_URL out of it and has no fallback for either.
POSTGRES_PASSWORD=<paste output of: openssl rand -hex 24>
# Leave unset. Only set this to point the backend at a Postgres compose does
# not run; it then replaces the URL built from POSTGRES_PASSWORD above.
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
# Required for the Traefik override. Both have no fallback — compose refuses # Required for the Traefik override. Both have no fallback — compose refuses
# to start without them. BOOKMARK_WEB_HOST is required even if you never set # to start without them. BOOKMARK_WEB_HOST is required even if you never set
# WEB_PASSWORD; see 1b. # WEB_PASSWORD; see 1b.
@@ -58,20 +50,13 @@ BOOKMARK_WEB_HOST=bookmark.violetcrown.my.id
# TRAEFIK_CERTRESOLVER=le # TRAEFIK_CERTRESOLVER=le
``` ```
Generate + insert the two secrets in three lines: Generate + insert the token in one line:
```bash ```bash
sed -i "s|^API_TOKEN=.*|API_TOKEN=$(openssl rand -hex 32)|" .env sed -i "s|^API_TOKEN=.*|API_TOKEN=$(openssl rand -hex 32)|" .env
sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 24)|" .env
grep -E '^API_TOKEN=' .env # copy this — the userscript needs the same value grep -E '^API_TOKEN=' .env # copy this — the userscript needs the same value
``` ```
`POSTGRES_PASSWORD` is read **only while the `postgres-data` volume is empty**,
which in practice means at first boot. Changing it afterwards changes the URL
the backend dials but not the password the database expects, and `bookmark-api`
crash-loops on `password authentication failed`. Set it before §2 and leave it
alone.
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual > Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
> names (check your Traefik static config — common alternatives: `https`, > names (check your Traefik static config — common alternatives: `https`,
> `myresolver`, `cloudflare`). Wrong names = no certificate issued. > `myresolver`, `cloudflare`). Wrong names = no certificate issued.
@@ -131,21 +116,16 @@ This merges the base file (build/image/env/volume) with the prod override
(no host port, Traefik network + router labels). Always pass **both** `-f` (no host port, Traefik network + router labels). Always pass **both** `-f`
flags — the prod file is not standalone. flags — the prod file is not standalone.
Three services come up: `bookmark-api` (the backend), `postgres` (its database, Two services come up: `bookmark-api` (the backend) and `headless-shell`, a CDP
`postgres:17-alpine`), and `headless-shell`, a CDP sidecar the poller uses to sidecar the poller uses to fetch kagane (behind a Cloudflare JS challenge).
fetch kagane (behind a Cloudflare JS challenge). Neither of the latter two It has no published port — only `bookmark-api` can reach it, over
publishes a port: `postgres` sits alone with `bookmark-api` on an `BROWSER_WS_URL`. Missing or unreachable, the poller just skips kagane and
`internal: true` network, and `headless-shell` is reachable only over logs it; nothing else is affected.
`BROWSER_WS_URL`. A missing headless-shell just makes the poller skip kagane and
log it. A missing Postgres stops everything — `bookmark-api` waits for
`pg_isready` to pass, then applies its embedded migrations, and only then
listens. The schema is created that way; there is nothing to import by hand.
Check it's up and healthy: Check it's up and healthy:
```bash ```bash
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
# bookmark-api Up; postgres Up (healthy)
docker logs bookmark-api --tail 20 # expect: "listening on :8080 ..." docker logs bookmark-api --tail 20 # expect: "listening on :8080 ..."
``` ```
@@ -234,10 +214,7 @@ Pull new code, then rebuild:
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
``` ```
Data persists in the named volume `postgres-data` across rebuilds. (If this SQLite data persists in the named volume `bookmarks-data` across rebuilds.
server predates the Postgres migration, the old SQLite volume `bookmarks-data`
is still on disk and deliberately undeclared in compose so `down -v` cannot take
it; see `REDEPLOY.md` §1 for when to remove it.)
--- ---
@@ -250,9 +227,7 @@ it; see `REDEPLOY.md` §1 for when to remove it.)
| `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). | | `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). |
| 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. | | 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. |
| Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. | | Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. |
| `compose ... config` errors about `API_TOKEN` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. Both are required and neither has a fallback. | | `compose ... config` errors about `API_TOKEN` | Run compose from the dir with `.env`, or export the vars. |
| `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). |
| `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. |
Backend config reference and endpoint list: see `README.md`. Backend config reference and endpoint list: see `README.md`.
+4 -10
View File
@@ -7,14 +7,14 @@ all four sites and all devices.
Two parts: Two parts:
- **`backend/`** — tiny Go (`net/http` + Postgres via pure-Go `pgx`) sync service. 4 routes, - **`backend/`** — tiny Go (`net/http` + pure-Go SQLite) sync service. 4 routes,
static binary, distroless container. static binary, distroless container.
- **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript - **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript
(no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`. (no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`.
``` ```
Bromite userscript (isolated world, Shadow DOM UI, localStorage cache) Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume) -- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
``` ```
--- ---
@@ -27,7 +27,7 @@ Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|-----|---------|-------| |-----|---------|-------|
| `API_TOKEN` | *(required)* | Bearer token shared with the userscript. | | `API_TOKEN` | *(required)* | Bearer token shared with the userscript. |
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. | | `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. | | `DB_PATH` | `/data/bookmarks.db` | SQLite file location. |
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. | | `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
| `BROWSER_WS_URL` | `ws://172.28.0.10:9222` | Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header. | | `BROWSER_WS_URL` | `ws://172.28.0.10:9222` | Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header. |
@@ -60,17 +60,11 @@ go test ./... # unit + handler tests
CGO_ENABLED=0 go build # static binary CGO_ENABLED=0 go build # static binary
``` ```
**`go test ./...` requires Docker.** The store talks to a real Postgres, so
each test package starts a throwaway `postgres:17-alpine` container and gives
every test its own database inside it (`internal/pgtest`). Nothing is stubbed
and nothing reaches the network beyond the local Docker daemon.
### Run the stack ### Run the stack
```bash ```bash
cp .env.example .env cp .env.example .env
# edit .env: set API_TOKEN (openssl rand -hex 32) and # edit .env: set API_TOKEN (openssl rand -hex 32)
# POSTGRES_PASSWORD (openssl rand -hex 24)
docker compose up -d --build # binds 127.0.0.1:8080 docker compose up -d --build # binds 127.0.0.1:8080
``` ```
+80 -136
View File
@@ -18,7 +18,7 @@ the checkout cannot take the backups with them.
``` ```
/opt/ /opt/
├── bookmarkmanager/ <- the checkout (this repo) ├── bookmarkmanager/ <- the checkout (this repo)
└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.dump └── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.db
``` ```
--- ---
@@ -53,98 +53,78 @@ echo "$BACKUP_DIR" # -> /opt/bookmarkmanager-backups
## 1. Back up the database ## 1. Back up the database
The database is Postgres, running as the `postgres` service on the named volume The database is a single SQLite file in the named Docker volume, at
`postgres-data`. It has **no published port** — nothing outside the internal `db` `/data/bookmarks.db` inside the container. Find the volume's real name — Compose
network can reach it — so every command below goes in through the container: prefixes it with the project directory:
```bash ```bash
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt' docker volume ls --filter name=bookmarks-data
# -> bookmarks, schema_migrations, series # -> local bookmarkmanager_bookmarks-data
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
``` ```
Inside the container that connects over the local socket as the `bookmarks` ### Preferred: hot backup, no downtime
superuser, so no password is needed anywhere in this section. `-T` is not
optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and
silently corrupts any binary stream flowing back out — see the dump below.
### Preferred: hot dump, no downtime The store runs in **WAL mode**, so recent writes may still be sitting in
`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can
`pg_dump` runs in a single repeatable-read transaction, so it writes one therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and
point-in-time-consistent snapshot while the API keeps serving. No stopping, no writes one consistent file, safe to run against a live database:
WAL to worry about — that is the server's problem, not yours.
```bash ```bash
STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \ docker run --rm \
> "$BACKUP_DIR/bookmarks-$STAMP.dump" -v "$VOL":/data \
-v "$BACKUP_DIR":/backup \
alpine sh -c "apk add -q sqlite &&
sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\""
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db
``` ```
`-Fc` is the custom archive format rather than plain SQL: it is compressed, and `$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the
`pg_restore` can inspect and replay it selectively — list its table of contents, files sort in real order and never collide across a DST shift.
restore one table, restore schema without data, reorder. A plain `.sql` dump can
only be piped into `psql` whole, and gives you no way to check what is in it
short of reading it.
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.dump`. UTC, so Note the source volume is mounted **read-write**, which looks wrong for a backup
the files sort in real order and never collide across a DST shift. and is not. Opening a WAL database requires creating the `-shm` shared-memory
file; with `:ro` the command fails with `unable to open database file` and no
backup is produced. `VACUUM INTO` never writes to the source itself.
Verify it before you trust it. An unreadable backup is worse than none, because Verify it before you trust it. An unreadable backup is worse than none, because
you will act as though you have one: you will act as though you have one:
```bash ```bash
# 1. The dump parses and contains the tables. Uses the same image compose docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite &&
# already pulls, so nothing new to install. sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' &&
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \ sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'"
pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA' # -> ok
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
# -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks
# -> 1236; 0 0 TABLE DATA public series series
# 2. Sanity-check the live row count you just captured.
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
-c 'select count(*) from bookmarks'
# -> 37 # -> 37
``` ```
A custom-format archive stores row counts nowhere, so step 1 proves the file is The count should match what the web UI shows. Zero rows on a server you know has
a readable archive with the right tables in it, not that the rows are there; bookmarks means you backed up the wrong volume.
step 2 is the number those rows should be. It should match what the web UI
shows. Zero on a server you know has bookmarks means the API and your `psql`
are looking at different databases — check `DATABASE_URL`.
### Fallback: cold volume archive ### Fallback: cold copy (no network for `apk add sqlite`)
Use this when you want the whole data directory rather than a logical dump — a Stop the service first, then copy the database **and its sidecars** — the `-wal`
like-for-like restore of the same Postgres major version onto the same host. is not optional, it is where the newest writes are:
**The stack must be stopped first.** A running Postgres has dirty pages in
shared buffers and WAL that has not been replayed into the data files, and `tar`
walks the directory over several seconds while the server keeps writing to it.
The archive you get is torn: files from different instants, possibly a
half-written page. It may restore, start, and be quietly wrong. Online
filesystem-level backup is `pg_basebackup`'s job, not `tar`'s; with the
container stopped the shutdown checkpoint has already flushed everything and a
plain archive of the volume is consistent.
```bash ```bash
VOL=$(docker volume ls --filter name=postgres-data -q | head -1)
echo "$VOL" # -> bookmarkmanager_postgres-data
$COMPOSE stop $COMPOSE stop
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \ docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c "
tar czf "/to/postgres-data-$STAMP.tgz" -C /from . cp /data/bookmarks.db /backup/bookmarks-$STAMP.db
[ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal
[ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm
ls -1 /backup"
$COMPOSE start $COMPOSE start
ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz
``` ```
Costs ~15 seconds of downtime. Read-only on the source is safe here precisely Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away,
because nothing is running against it. Restoring this variant means untarring it so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for
back into an *empty* `postgres-data` volume with the stack down — it is a whole the case where it did not. Restoring this variant means putting whichever files
data directory, not a file you can drop next to the live one, and it will only you got back together, under their original names.
start under `postgres:17`.
Read-only is safe here precisely because nothing opens the database: it is a file
copy, not a SQLite connection.
### Retention ### Retention
@@ -152,19 +132,7 @@ Keep a month, drop the rest — a bookmark database this small compresses the
decision to "disk is free, but not infinite": decision to "disk is free, but not infinite":
```bash ```bash
ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v
```
### A note on the old `bookmarks-data` volume
`bookmarks-data` is the **pre-migration SQLite volume**. It is deliberately not
declared in `docker-compose.yml` any more, which is what keeps `docker compose
down -v` from taking it with the rest of the stack. It is not the live database
and nothing reads it. Once the Postgres data has been trusted for a while,
remove it by hand — nothing else will:
```bash
docker volume rm bookmarkmanager_bookmarks-data
``` ```
--- ---
@@ -203,15 +171,11 @@ rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
bindmounted read-only and read fresh per request. bindmounted read-only and read fresh per request.
```bash ```bash
$COMPOSE ps # bookmark-api Up; postgres Up (healthy) $COMPOSE ps # Up, and recently (re)created
docker logs bookmark-api --tail 20 # -> "listening on :8080 ..." docker logs bookmark-api --tail 20 # -> "listening on :8080 ..."
``` ```
Nothing in the log about the database, the migrations or the poller failing. Nothing in the log about the database or the poller failing. The image is tagged
`bookmark-api` waits on `postgres` reporting healthy before it starts and the
binary applies any pending migration before it listens, so an API that never
says "listening" is usually the database, not the code — `$COMPOSE logs
postgres` first. The image is tagged
`bookmarkmanager-backend:latest`, so the previous image is still on disk untagged — `bookmarkmanager-backend:latest`, so the previous image is still on disk untagged —
that is what makes the rollback in §6 quick. that is what makes the rollback in §6 quick.
@@ -235,16 +199,9 @@ curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
$API/bookmarks/x | grep -i access-control # -> allow-origin echoed $API/bookmarks/x | grep -i access-control # -> allow-origin echoed
``` ```
`[]` from the third call is the alarm that matters: you are talking to an empty `[]` from the third call is the alarm that matters: the volume is not attached
database, which means the API found a *different* Postgres than the one holding and you are looking at an empty database. Stop and check `$COMPOSE config
your data — a renamed project directory, a fresh `postgres-data`, or a --volumes` before touching anything else.
`DATABASE_URL` override in `.env` pointing elsewhere. Stop and check, before
touching anything else:
```bash
$COMPOSE config --volumes # -> postgres-data
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks'
```
Web UI and its assets: Web UI and its assets:
@@ -310,41 +267,33 @@ git checkout <previous-hash>
$COMPOSE up -d --build $COMPOSE up -d --build
``` ```
**Database damaged** — restore the dump from §1. Stop **only the API**, not the **Database damaged** — restore the backup from §1. Stop first: the running
whole stack: `pg_restore` needs the server up to restore into, and it needs process holds the WAL, and dropping a file under a live SQLite connection
`bookmark-api`'s connection pool gone, because `--clean` cannot drop a table corrupts what you were trying to save.
other sessions are holding open.
```bash ```bash
$COMPOSE stop bookmark-api $COMPOSE stop
$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \ docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c '
< "$BACKUP_DIR/bookmarks-<STAMP>.dump" rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm &&
cp /backup/bookmarks-<STAMP>.db /data/bookmarks.db &&
chown 65532:65532 /data/bookmarks.db &&
ls -l /data'
$COMPOSE start bookmark-api $COMPOSE start
docker logs bookmark-api --tail 20 docker logs bookmark-api --tail 20
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200 curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
``` ```
Three things here are easy to skip and all three bite: Two steps here are easy to skip and both bite:
- **`--clean --if-exists`.** Without `--clean` the dump's rows land *on top of* - **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database
what is already there and you get primary-key collisions half way through, a mixes two different histories; SQLite will either refuse to open it or quietly
partially restored database, and a non-zero exit you may not notice. reapply writes you meant to discard.
`--if-exists` only suppresses the "does not exist" noise when the target is - **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as
already empty; it is not the part doing the work. that uid, while the helper container above writes as root. A root-owned
- **`-T` again.** Feeding a custom-format archive into a TTY-allocated `exec` database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then
corrupts it in flight and `pg_restore` fails with a garbled-header error on a every write fails. That is the worst possible failure mode — it looks restored.
file that is perfectly fine on disk.
- **Stop the API, not Postgres.** `$COMPOSE stop` (everything) leaves you with
nothing to restore into; leaving `bookmark-api` running leaves connections
that block the drops *and* lets the poller write into a half-restored table.
No ownership fixing is needed any more — the Postgres image owns `postgres-data`
itself and `pg_restore` writes through the server, not the filesystem.
`schema_migrations` is inside the dump, so the database comes back at whatever
schema version the backup was taken at; the migration runner applies anything
newer the next time `bookmark-api` starts.
--- ---
@@ -356,24 +305,21 @@ For a routine redeploy where nothing needs deciding:
cd /opt/bookmarkmanager cd /opt/bookmarkmanager
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml" COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR" BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
STAMP=$(date -u +%Y%m%d-%H%M%S) STAMP=$(date -u +%Y%m%d-%H%M%S)
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \ docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \
> "$BACKUP_DIR/bookmarks-$STAMP.dump" && "apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" &&
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \ sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" &&
pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null &&
git pull --ff-only && git pull --ff-only &&
$COMPOSE up -d --build && $COMPOSE up -d --build &&
sleep 5 && sleep 5 &&
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok" curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"
``` ```
The `&&` chain is deliberate: if the dump or its `pg_restore --list` check The `&&` chain is deliberate: if the backup or its integrity check fails,
fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell
short or empty `.dump` behind — the shell creates the file before `pg_dump` command can tell you the panel works on the phone.
runs — so delete it rather than letting it sit in the backup directory looking
like a backup. Then still do §5 by hand — no shell command can tell you the
panel works on the phone.
--- ---
@@ -381,18 +327,16 @@ panel works on the phone.
| Symptom | Cause / fix | | Symptom | Cause / fix |
|---|---| |---|---|
| `/bookmarks` returns `[]` after redeploy | You are on an empty Postgres. Check `$COMPOSE config --volumes` lists `postgres-data`, that you passed both `-f` files, and that `.env` has no stray `DATABASE_URL` override. Do **not** re-bookmark; the data is still in the volume. | | `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. |
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. | | UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. | | CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. | | Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. | | Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
| `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. | | `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. | | Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
| `bookmark-api` crash-loops, log says `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` in `.env` no longer matches the one burned into `postgres-data` at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: `$COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks'` (prompts, so nothing lands in shell history) and then match `.env` to it. | | `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. |
| `compose` errors `set POSTGRES_PASSWORD in .env` | Unset. Compose builds the backend's `DATABASE_URL` out of it, so it is required even though you never write that URL yourself. Run from the directory holding `.env`. | | Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). |
| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. | | Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). |
| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. |
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`. Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
UI conventions: `docs/design-system.md`. UI conventions: `docs/design-system.md`.
+16 -40
View File
@@ -1,8 +1,8 @@
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system. Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`. - **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
Single binary, split into packages under `backend/internal/`: `store` Single binary, split into packages under `backend/internal/`: `store`
(Bookmark type, Postgres persistence, migration runner), `latest` (background (Bookmark type, SQLite persistence, migrations), `latest` (background
poller, site parsers, TLS fetcher), `session` (cookie signing, login poller, site parsers, TLS fetcher), `session` (cookie signing, login
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
bookmark handlers), `userscript` (userscript-serving handler), `web` bookmark handlers), `userscript` (userscript-serving handler), `web`
@@ -11,25 +11,7 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
packages together into `newRouter`. Root-level `*_test.go` hold packages together into `newRouter`. Root-level `*_test.go` hold
integration tests that exercise the full router; unit tests for a integration tests that exercise the full router; unit tests for a
package live beside it under `internal/`. package live beside it under `internal/`.
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is - **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`). Sync **last-write-wins**. Schema and endpoint list in plan.
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
file per change, one transaction each, versions recorded in
`schema_migrations`. Files are **append-only** — editing an applied one
changes nothing on a database that already ran it. No column probing, no
data-fixup migrations: both were SQLite-era machinery and are gone.
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
its own database (`pgtest.URL(t)`). A package whose tests touch the store
must have that `TestMain`.
- **Single-user store, two tables.** `series` keyed `(site, series_id)`
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
differs between readers: progress, favourite, lifecycle bucket,
`updated_at`. Sync **last-write-wins**; the wire format stays flat
(ADR-0004). `Store.Upsert` decomposes one flat body across both tables and
enforces the ownership rule: client `title`/`series_url`/`cover` are
written only when the series row is new (ADR-0003).
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth). - **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
- **Web UI:** same binary serve password-gated browser UI on second - **Web UI:** same binary serve password-gated browser UI on second
hostname — `GET /` (list, or login page when no session), hostname — `GET /` (list, or login page when no session),
@@ -61,25 +43,22 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
network access, so `latest_chapter` stay fresh when user not network access, so `latest_chapter` stay fresh when user not
browsing. Second, parallel signal — userscript keep own browsing. Second, parallel signal — userscript keep own
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged. `maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
Two independent clocks: per-series cooldown (`series.latest_checked_at`, Two independent clocks: per-bookmark cooldown (`latest_checked_at` column,
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval. enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
The poller walks **Series, not Bookmarks** — a series referenced by several Row stamped *before* fetch so broken series wait out full
bookmarks is fetched once per cycle, and the due queue orders cooldown instead of retrying every tick, and writes go through
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped `Store.Get` + `Store.Upsert` so new chapter never reorders list.
*before* fetch so broken series wait out full cooldown instead of retrying
every tick; found chapter written straight to the series row via
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
order — is never touched.
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
against fingerprint-based blocking; any failure log and skip. kagane and against fingerprint-based blocking; any failure log and skip. kagane and
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and
simply not polled when that's unset. See simply not polled when that's unset. See
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`. `docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
The poller's series write is a single-column UPDATE Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark: userscript `PUT` that commits between the two can get overwritten by
it cannot revert read progress or move `updated_at`, so the old poller's stale re-read — reverting that read progress and, since stored
stale-re-read race is gone with the Get+Upsert flow. value now differs, moving `updated_at` and reordering list. Known,
accepted limitation for single-user deployment, not bug to fix.
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4. - **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` | - **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
`finished`, orthogonal to `favorite`. Archived and finished appear only in `finished`, orthogonal to `favorite`. Archived and finished appear only in
@@ -91,16 +70,13 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
`excluded.*` is post-evaluation row and default applied there would `excluded.*` is post-evaluation row and default applied there would
wipe bucket on every PUT from client that predates column. See wipe bucket on every PUT from client that predates column. See
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`. `docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), - **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
`DATABASE_URL` (Postgres connection URL, required — no default), (default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
`PORT` (default `8080`), `WEB_PASSWORD`
(gates browser UI; unset disable it), (gates browser UI; unset disable it),
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER` `LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`). (background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at `USERSCRIPT_PATH` (file served at `/u/{token}/manga-bookmark.user.js`,
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`, default `/userscript/manga-bookmark.user.js`, supplied by bindmount).
defaults `/userscript/manga-bookmark.user.js` and
`/userscript/novel-bookmark.user.js`, both supplied by bindmount).
`BROWSER_WS_URL` (headless-shell CDP endpoint for kagane and novelfull; `BROWSER_WS_URL` (headless-shell CDP endpoint for kagane and novelfull;
unset disables browser polling and leaves those sites to the userscript unset disables browser polling and leaves those sites to the userscript
alone). alone).
-1
View File
@@ -1 +0,0 @@
AGENTS.md
+81
View File
@@ -0,0 +1,81 @@
Guidance for Claude Code working under `backend/`. See root `CLAUDE.md` for the project-wide architecture diagram, hard constraints, and design system.
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
Single binary, split into packages under `backend/internal/`: `store`
(Bookmark type, SQLite persistence, migrations), `latest` (background
poller, site parsers, TLS fetcher), `session` (cookie signing, login
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
bookmark handlers), `userscript` (userscript-serving handler), `web`
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
`backend/main.go` is the composition root — the only place that wires
packages together into `newRouter`. Root-level `*_test.go` hold
integration tests that exercise the full router; unit tests for a
package live beside it under `internal/`.
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`). Sync **last-write-wins**. Schema and endpoint list in plan.
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
- **Web UI:** same binary serve password-gated browser UI on second
hostname — `GET /` (list, or login page when no session),
`POST /login`, `POST /logout`, `GET /static/*`, htmx fragment endpoints
under `/ui/*`. Templates + assets `go:embed`-ed under
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
`internal/` tree, not just `*.go`. Sessions stateless
HMAC cookies keyed off `API_TOKEN`; `WEB_PASSWORD` gates them, and when empty,
web routes not registered at all. UI mutations read-modify-write
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
(correct — served from `/`), but impeccable detector resolves
stylesheet href with `path.resolve(fileDir, href)`, drops directory
on leading `/` and silently skip file. Relative href don't help
either: template's directory isn't its served path. So
`detect.mjs backend/internal/web/templates` reports **false clean** —
always pass `backend/internal/web/static` too. One finding there,
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
- **Every action that moves series out of list is confirm-gated.**
Archive, finish, remove each open own `.confirm-row` disclosure
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
`archive|finish|remove`); restore fire instantly since it's the reversal.
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
`--ember` stay reserved for new-chapter signal: busy bar and inline
error use `--mute`.
- **Latest-chapter poller:** ticker goroutine in same binary re-check
each bookmarked series' newest published chapter from backend's own
network access, so `latest_chapter` stay fresh when user not
browsing. Second, parallel signal — userscript keep own
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
Two independent clocks: per-bookmark cooldown (`latest_checked_at` column,
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
Row stamped *before* fetch so broken series wait out full
cooldown instead of retrying every tick, and writes go through
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
against fingerprint-based blocking; any failure log and skip. kagane sits
behind a Cloudflare JavaScript challenge the TLS client can't clear, so it is
browser-only: fetched over CDP via `BROWSER_WS_URL`, and simply not polled
when that's unset. See
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
userscript `PUT` that commits between the two can get overwritten by
poller's stale re-read — reverting that read progress and, since stored
value now differs, moving `updated_at` and reordering list. Known,
accepted limitation for single-user deployment, not bug to fix.
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
`finished`, orthogonal to `favorite`. Archived and finished appear only in
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
checking archived series and skip finished ones. `finished` settable
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
**Empty incoming status means "keep stored one"** — resolved on the
`VALUES` side of `Store.Upsert`, not conflict clause, since
`excluded.*` is post-evaluation row and default applied there would
wipe bucket on every PUT from client that predates column. See
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
(gates browser UI; unset disable it),
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
`USERSCRIPT_PATH` (file served at `/u/{token}/manga-bookmark.user.js`,
default `/userscript/manga-bookmark.user.js`, supplied by bindmount).
`BROWSER_WS_URL` (headless-shell CDP endpoint for kagane; unset disables
browser polling and leaves that site to the userscript alone).
+7 -3
View File
@@ -14,17 +14,21 @@ RUN go mod download
COPY *.go ./ COPY *.go ./
COPY internal/ ./internal/ COPY internal/ ./internal/
# Static binary: the Postgres driver (jackc/pgx) is pure Go, so CGO_ENABLED=0 # Static binary: pure-Go sqlite means CGO_ENABLED=0 -> no libc dependency.
# leaves no libc dependency.
# -trimpath + -ldflags strip paths and debug info for a smaller image. # -trimpath + -ldflags strip paths and debug info for a smaller image.
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server . RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server .
# Data dir with the runtime user's ownership so the mounted volume inherits it.
RUN mkdir -p /out/data
# --- runtime stage: distroless static, non-root --- # --- runtime stage: distroless static, non-root ---
FROM gcr.io/distroless/static:nonroot FROM gcr.io/distroless/static:nonroot
WORKDIR / WORKDIR /
COPY --from=build /out/server /server COPY --from=build /out/server /server
COPY --from=build --chown=65532:65532 /out/data /data
VOLUME ["/data"]
EXPOSE 8080 EXPOSE 8080
USER nonroot:nonroot USER nonroot:nonroot
ENV PORT=8080 ENV DB_PATH=/data/bookmarks.db PORT=8080
ENTRYPOINT ["/server"] ENTRYPOINT ["/server"]
+25 -146
View File
@@ -12,7 +12,6 @@ import (
"testing" "testing"
"time" "time"
"bookmarkmanager/backend/internal/pgtest"
"bookmarkmanager/backend/internal/store" "bookmarkmanager/backend/internal/store"
) )
@@ -26,21 +25,15 @@ func testConfig() Config {
} }
} }
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
func newTestServer(t *testing.T) http.Handler { func newTestServer(t *testing.T) http.Handler {
t.Helper() t.Helper()
return newRouter(newTestStore(t), testConfig()) dbPath := filepath.Join(t.TempDir(), "test.db")
} s, err := store.Open(dbPath)
func newTestStore(t *testing.T) *store.Store {
t.Helper()
s, err := store.Open(pgtest.URL(t))
if err != nil { if err != nil {
t.Fatalf("store.Open: %v", err) t.Fatalf("store.Open: %v", err)
} }
t.Cleanup(func() { s.Close() }) t.Cleanup(func() { s.Close() })
return s return newRouter(s, testConfig())
} }
func auth(req *http.Request) *http.Request { func auth(req *http.Request) *http.Request {
@@ -50,35 +43,26 @@ func auth(req *http.Request) *http.Request {
func floatPtr(f float64) *float64 { return &f } func floatPtr(f float64) *float64 { return &f }
// seedForCheck inserts a bookmark (and with it its series) and forces the // seedForCheck inserts a bookmark and forces its latest_checked_at.
// series' latest_checked_at.
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) { func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(store.Bookmark{
Key: key, Key: key,
Site: site, Site: "asura",
SeriesID: seriesID, SeriesID: key,
SeriesURL: seriesURL, SeriesURL: seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed %q: %v", key, err) t.Fatalf("seed %q: %v", key, err)
} }
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil { if err := s.MarkLatestChecked(key, checkedAt); err != nil {
t.Fatalf("seed mark %q: %v", key, err) t.Fatalf("seed mark %q: %v", key, err)
} }
} }
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 { func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":") ts, err := s.LatestCheckedAt(key)
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
ts, err := s.LatestCheckedAt(site, seriesID)
if err != nil { if err != nil {
t.Fatalf("LatestCheckedAt %q: %v", key, err) t.Fatalf("LatestCheckedAt %q: %v", key, err)
} }
@@ -238,125 +222,6 @@ func TestBookmarkRoundTrip(t *testing.T) {
} }
} }
// The wire contract (ADR-0004): GET and PUT speak exactly the flat field set
// they always did, with the series-owned fields as siblings of the bookmark
// fields, not nested. Asserted as a key set, not by inspection.
func TestFlatWireFieldSet(t *testing.T) {
srv := newTestServer(t)
key := "comix:some-title"
in := store.Bookmark{
Key: key,
Site: "comix",
SeriesID: "some-title",
Title: "Some Title",
SeriesURL: "https://comix.to/title/some-title",
Cover: "https://comix.to/covers/some-title.jpg",
LastChapter: "Chapter 7",
LastChapterNum: 7,
LastChapterURL: "https://comix.to/title/some-title/ch/7",
Favorite: true,
LatestChapter: "Chapter 8",
LatestChapterNum: floatPtr(8),
Status: store.StatusArchived,
Kind: store.KindManga,
}
body, _ := json.Marshal(in)
wantKeys := map[string]bool{
"key": true, "site": true, "series_id": true, "title": true,
"series_url": true, "cover": true, "last_chapter": true,
"last_chapter_num": true, "last_chapter_url": true, "favorite": true,
"latest_chapter": true, "latest_chapter_num": true, "updated_at": true,
"status": true, "kind": true,
}
checkFlat := func(t *testing.T, payload []byte) map[string]json.RawMessage {
t.Helper()
var obj map[string]json.RawMessage
if err := json.Unmarshal(payload, &obj); err != nil {
t.Fatalf("decode: %v", err)
}
if len(obj) != len(wantKeys) {
t.Fatalf("field count = %d, want %d (%s)", len(obj), len(wantKeys), payload)
}
for k := range obj {
if !wantKeys[k] {
t.Fatalf("unexpected field %q", k)
}
}
return obj
}
// PUT
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, auth(httptest.NewRequest(http.MethodPut, "/bookmarks/"+key, bytes.NewReader(body))))
if rr.Code != http.StatusOK {
t.Fatalf("PUT status = %d, want 200", rr.Code)
}
checkFlat(t, rr.Body.Bytes())
// Every field round-trips with its value, and updated_at is server-stamped.
var stored store.Bookmark
if err := json.Unmarshal(rr.Body.Bytes(), &stored); err != nil {
t.Fatalf("decode PUT response: %v", err)
}
latestNum := floatPtr(8)
want := store.Bookmark{
Key: key, Site: "comix", SeriesID: "some-title",
Title: in.Title, SeriesURL: in.SeriesURL, Cover: in.Cover,
LastChapter: in.LastChapter, LastChapterNum: in.LastChapterNum,
LastChapterURL: in.LastChapterURL, Favorite: true,
LatestChapter: in.LatestChapter, LatestChapterNum: latestNum,
Status: store.StatusArchived, Kind: store.KindManga,
}
if stored.Title != want.Title || stored.SeriesURL != want.SeriesURL || stored.Cover != want.Cover ||
stored.LastChapter != want.LastChapter || stored.LastChapterNum != want.LastChapterNum ||
stored.LastChapterURL != want.LastChapterURL || stored.Favorite != want.Favorite ||
stored.LatestChapter != want.LatestChapter ||
stored.LatestChapterNum == nil || *stored.LatestChapterNum != *want.LatestChapterNum ||
stored.Status != want.Status || stored.Kind != want.Kind {
t.Fatalf("PUT response = %+v, want %+v", stored, want)
}
if stored.UpdatedAt == 0 {
t.Fatal("updated_at not server-stamped")
}
// GET reports the same flat shape.
list := getBookmarks(t, srv)
if len(list) != 1 {
t.Fatalf("list = %d items, want 1", len(list))
}
body2, _ := json.Marshal(list[0])
checkFlat(t, body2)
}
// A PUT naming an existing series must ignore client-supplied title, cover and
// URL — the security boundary from ADR-0003, where a hostile site's scraped
// values could otherwise land on a shared row — while progress still lands.
func TestPutExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
srv := newTestServer(t)
key := "asura:solo"
first := putBookmark(t, srv, key, store.Bookmark{
Title: "Solo Leveling",
SeriesURL: "https://asurascans.com/comics/solo",
Cover: "https://asurascans.com/covers/solo.jpg",
LastChapterNum: 10,
})
second := putBookmark(t, srv, key, store.Bookmark{
Title: "Scraped Rename",
SeriesURL: "https://evil.example/solo",
Cover: "https://evil.example/solo.jpg",
LastChapterNum: 11,
})
if second.Title != first.Title || second.SeriesURL != first.SeriesURL || second.Cover != first.Cover {
t.Fatalf("stored = %+v, want original title/url/cover kept", second)
}
if second.LastChapterNum != 11 {
t.Fatalf("LastChapterNum = %v, want 11 — progress must still land", second.LastChapterNum)
}
}
// putBookmark PUTs b at key and returns the bookmark the server echoes back, // putBookmark PUTs b at key and returns the bookmark the server echoes back,
// which is the row as actually stored (not the request payload). // which is the row as actually stored (not the request payload).
func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark { func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark {
@@ -552,7 +417,12 @@ func TestLoadConfigWebPassword(t *testing.T) {
// moved into bookmarkColumns, this test catches it: the PUT would reset the // moved into bookmarkColumns, this test catches it: the PUT would reset the
// cooldown and the poller would re-fetch that series on every single tick. // cooldown and the poller would re-fetch that series on every single tick.
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) { func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
s := newTestStore(t) dbPath := filepath.Join(t.TempDir(), "test.db")
s, err := store.Open(dbPath)
if err != nil {
t.Fatalf("store.Open: %v", err)
}
t.Cleanup(func() { s.Close() })
srv := newRouter(s, testConfig()) srv := newRouter(s, testConfig())
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777) seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
@@ -584,7 +454,12 @@ func TestUserscriptServedWithWebUIDisabled(t *testing.T) {
t.Fatalf("write script: %v", err) t.Fatalf("write script: %v", err)
} }
s := newTestStore(t) dbPath := filepath.Join(t.TempDir(), "nopass.db")
s, err := store.Open(dbPath)
if err != nil {
t.Fatalf("store.Open: %v", err)
}
t.Cleanup(func() { s.Close() })
cfg := testConfig() // WebPassword empty cfg := testConfig() // WebPassword empty
cfg.UserscriptPath = path cfg.UserscriptPath = path
@@ -605,7 +480,11 @@ func TestNovelUserscriptServed(t *testing.T) {
t.Fatalf("write script: %v", err) t.Fatalf("write script: %v", err)
} }
s := newTestStore(t) s, err := store.Open(filepath.Join(dir, "test.db"))
if err != nil {
t.Fatalf("store.Open: %v", err)
}
t.Cleanup(func() { s.Close() })
cfg := testConfig() cfg := testConfig()
cfg.NovelUserscriptPath = novelPath cfg.NovelUserscriptPath = novelPath
+13 -5
View File
@@ -7,7 +7,7 @@ require (
github.com/bogdanfinn/tls-client v1.15.1 github.com/bogdanfinn/tls-client v1.15.1
github.com/chromedp/cdproto v0.0.0-20260714215040-dc233986426f github.com/chromedp/cdproto v0.0.0-20260714215040-dc233986426f
github.com/chromedp/chromedp v0.16.0 github.com/chromedp/chromedp v0.16.0
github.com/jackc/pgx/v5 v5.10.0 modernc.org/sqlite v1.34.4
) )
require ( require (
@@ -18,19 +18,27 @@ require (
github.com/bogdanfinn/utls v1.7.7-barnius // indirect github.com/bogdanfinn/utls v1.7.7-barnius // indirect
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
github.com/chromedp/sysutil v1.1.0 // indirect github.com/chromedp/sysutil v1.1.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 // indirect github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 // indirect
github.com/gobwas/httphead v0.1.0 // indirect github.com/gobwas/httphead v0.1.0 // indirect
github.com/gobwas/pool v0.2.1 // indirect github.com/gobwas/pool v0.2.1 // indirect
github.com/gobwas/ws v1.4.0 // indirect github.com/gobwas/ws v1.4.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect github.com/google/uuid v1.6.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.18.2 // indirect github.com/klauspost/compress v1.18.2 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/quic-go/qpack v0.6.0 // indirect github.com/quic-go/qpack v0.6.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
golang.org/x/crypto v0.46.0 // indirect golang.org/x/crypto v0.46.0 // indirect
golang.org/x/net v0.48.0 // indirect golang.org/x/net v0.48.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.47.0 // indirect golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.32.0 // indirect golang.org/x/text v0.32.0 // indirect
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
modernc.org/libc v1.55.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
modernc.org/strutil v1.2.0 // indirect
modernc.org/token v1.1.0 // indirect
) )
+44 -14
View File
@@ -20,9 +20,10 @@ github.com/chromedp/chromedp v0.16.0 h1:rOO4deOm4CbZgBCa8mD9g2rDyIoNs0BkgvNrlbp5
github.com/chromedp/chromedp v0.16.0/go.mod h1:rbuGKFT1vMcFcFqKfPIO1GpX/N+2s8onm2qMxZLbU5U= github.com/chromedp/chromedp v0.16.0/go.mod h1:rbuGKFT1vMcFcFqKfPIO1GpX/N+2s8onm2qMxZLbU5U=
github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM= github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM=
github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8= github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 h1:KZaTBSyshWX3MP5jukJcNSuXDQTO+rNpt0J564dX/eg= github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 h1:KZaTBSyshWX3MP5jukJcNSuXDQTO+rNpt0J564dX/eg=
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg= github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg=
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU= github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
@@ -31,27 +32,28 @@ github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw= github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs= github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs=
github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc= github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk= github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4= github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo= github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo=
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs= github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw= github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw=
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0= github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8= github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII= github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc= github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
@@ -62,6 +64,8 @@ go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o= go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU= golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0= golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU= golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY= golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
@@ -77,7 +81,33 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU= golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY= golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 h1:5D53IMaUuA5InSeMu9eJtlQXS2NxAhyWQvkKEgXZhHI=
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6/go.mod h1:Qz0X07sNOR1jWYCrJMEnbW/X55x206Q7Vt4mz6/wHp4=
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
modernc.org/sqlite v1.34.4 h1:sjdARozcL5KJBvYQvLlZEmctRgW9xqIZc2ncN7PU0P8=
modernc.org/sqlite v1.34.4/go.mod h1:3QQFCG2SEMtc2nv+Wq4cQCH7Hjcg+p/RMlS1XK+zwbk=
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
+46 -29
View File
@@ -23,9 +23,9 @@ type Fetcher interface {
// Two clocks, deliberately independent: // Two clocks, deliberately independent:
// //
// - Interval is how often this goroutine wakes up and looks. // - Interval is how often this goroutine wakes up and looks.
// - Cooldown is how long one series rests since its own last check. // - Cooldown is how long one bookmark rests since its own last check.
// //
// Only the cooldown is per series, and it is enforced by the WHERE clause in // Only the cooldown is per bookmark, and it is enforced by the WHERE clause in
// DueForLatestCheck rather than by any timer. Shortening Interval therefore // DueForLatestCheck rather than by any timer. Shortening Interval therefore
// cannot shorten anyone's cooldown; it only makes the poller wake up and find // cannot shorten anyone's cooldown; it only makes the poller wake up and find
// nothing due more often. // nothing due more often.
@@ -78,7 +78,7 @@ func (p *Poller) Run(ctx context.Context) {
} }
} }
// runOnce processes one batch of due series. // runOnce processes one batch of due bookmarks.
func (p *Poller) runOnce(ctx context.Context) { func (p *Poller) runOnce(ctx context.Context) {
cutoff := p.Now().Add(-p.Cooldown).UnixMilli() cutoff := p.Now().Add(-p.Cooldown).UnixMilli()
due, err := p.Store.DueForLatestCheck(cutoff, p.Batch) due, err := p.Store.DueForLatestCheck(cutoff, p.Batch)
@@ -88,7 +88,7 @@ func (p *Poller) runOnce(ctx context.Context) {
} }
checked := 0 checked := 0
for i, sr := range due { for i, b := range due {
if ctx.Err() != nil { if ctx.Err() != nil {
break break
} }
@@ -107,7 +107,7 @@ func (p *Poller) runOnce(ctx context.Context) {
if stopped { if stopped {
break break
} }
p.checkOne(ctx, sr) p.checkOne(ctx, b)
checked++ checked++
} }
// due vs checked is how you tell which constraint is binding: ticks that // due vs checked is how you tell which constraint is binding: ticks that
@@ -119,10 +119,10 @@ func (p *Poller) runOnce(ctx context.Context) {
// checkOne re-checks one series. Every failure path here is "log and move on": // checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a // the poller is a best-effort enhancement, and no single bad series may stall a
// batch or take down the process. // batch or take down the process.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) { func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
defer func() { defer func() {
if r := recover(); r != nil { if r := recover(); r != nil {
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r) log.Printf("latest poll %q: recovered from panic: %v", b.Key, r)
} }
}() }()
@@ -130,8 +130,8 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
// mid-request still consumes the cooldown. Otherwise a renamed or deleted // mid-request still consumes the cooldown. Otherwise a renamed or deleted
// series would be retried on every single tick forever. The userscript // series would be retried on every single tick forever. The userscript
// stamps in the same order and for the same reason (L471-473). // stamps in the same order and for the same reason (L471-473).
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil { if err := p.Store.MarkLatestChecked(b.Key, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err) log.Printf("latest poll %q: mark checked: %v", b.Key, err)
return return
} }
@@ -142,52 +142,69 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
// link-local/internal addresses or non-https schemes. The cooldown above // link-local/internal addresses or non-https schemes. The cooldown above
// is already consumed, so a row that never passes this check is retried at // is already consumed, so a row that never passes this check is retried at
// cooldown pace rather than hot-looping. // cooldown pace rather than hot-looping.
if !fetchableSeriesURL(sr.Site, sr.SeriesURL) { if !fetchableSeriesURL(b.Site, b.SeriesURL) {
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL) log.Printf("latest poll %q: not fetchable: site=%q url=%q", b.Key, b.Site, b.SeriesURL)
return return
} }
f := p.fetcherFor(sr.Site) f := p.fetcherFor(b.Site)
if f == nil { if f == nil {
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site) log.Printf("latest poll %q: no fetcher for site %q", b.Key, b.Site)
return return
} }
body, status, err := f.Get(ctx, sr.SeriesURL) body, status, err := f.Get(ctx, b.SeriesURL)
if err != nil { if err != nil {
log.Printf("latest poll %q: fetch %s: %v", sr.Key(), sr.SeriesURL, err) log.Printf("latest poll %q: fetch %s: %v", b.Key, b.SeriesURL, err)
return return
} }
if status != 200 { if status != 200 {
log.Printf("latest poll %q: fetch %s: status %d", sr.Key(), sr.SeriesURL, status) log.Printf("latest poll %q: fetch %s: status %d", b.Key, b.SeriesURL, status)
return return
} }
latest, ok := latestChapterFrom(sr.Site, sr.SeriesURL, body) latest, ok := latestChapterFrom(b.Site, b.SeriesURL, body)
if !ok { if !ok {
// Most likely a challenge page or a layout change. Either way the row is // Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a cooldown instead of hot-looping. // already stamped, so this waits out a cooldown instead of hot-looping.
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), len(body)) log.Printf("latest poll %q: no chapter links in %d bytes", b.Key, len(body))
return return
} }
// Re-read: the row may have been updated or deleted while the fetch was in
// flight, and writing b back wholesale would undo that.
//
// ponytail: non-transactional read-modify-write, wrap Get+Upsert in a tx if
// this ever runs for more than one user. A client PUT that commits between
// these two statements is lost to the stale re-read — reverting read
// progress or a status change, and moving updated_at because the stored
// value now differs. Accepted for a single-user deployment: the window is
// milliseconds and the loser is one poll cycle.
cur, found, err := p.Store.Get(b.Key)
if err != nil {
log.Printf("latest poll %q: reread: %v", b.Key, err)
return
}
if !found {
return
}
// Equality, not >, mirroring the userscript (L427): a site that retracts a // Equality, not >, mirroring the userscript (L427): a site that retracts a
// chapter should correct the stored number downward. The comparison is // chapter should correct the stored number downward.
// against the due-query snapshot; a concurrent write in between only costs if cur.LatestChapterNum != nil && *cur.LatestChapterNum == latest.Num {
// one redundant UPDATE of the same absolute value, never a wrong one.
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == latest.Num {
return return
} }
// Series-level write: the row is shared, so one update refreshes every num := latest.Num
// bookmark joining to it, and the bookmark's updated_at is never touched — cur.LatestChapter = latest.Label
// a newly published chapter is not reading progress and must not reorder cur.LatestChapterNum = &num
// the list. // A candidate only. last_chapter_num is untouched, so the CASE in Upsert
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, latest.Label, latest.Num); err != nil { // keeps the stored updated_at and the bookmark list does not reorder.
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err) cur.UpdatedAt = p.Now().UnixMilli()
if _, err := p.Store.Upsert(cur); err != nil {
log.Printf("latest poll %q: upsert: %v", b.Key, err)
return return
} }
log.Printf("latest poll %q: latest is now %s", sr.Key(), latest.Label) log.Printf("latest poll %q: latest is now %s", b.Key, latest.Label)
} }
// fetchableSeriesURL reports whether site is a site latestChapterFrom knows how // fetchableSeriesURL reports whether site is a site latestChapterFrom knows how
+10 -58
View File
@@ -3,22 +3,18 @@ package latest
import ( import (
"context" "context"
"errors" "errors"
"os" "path/filepath"
"strings"
"sync" "sync"
"testing" "testing"
"time" "time"
"bookmarkmanager/backend/internal/pgtest"
"bookmarkmanager/backend/internal/store" "bookmarkmanager/backend/internal/store"
) )
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } // newTestStore opens a fresh SQLite store in a temp dir.
// newTestStore opens a store on a Postgres database of this test's own.
func newTestStore(t *testing.T) *store.Store { func newTestStore(t *testing.T) *store.Store {
t.Helper() t.Helper()
s, err := store.Open(pgtest.URL(t)) s, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil { if err != nil {
t.Fatalf("Open: %v", err) t.Fatalf("Open: %v", err)
} }
@@ -26,35 +22,26 @@ func newTestStore(t *testing.T) *store.Store {
return s return s
} }
// seedForCheck inserts a bookmark (and with it its series) and forces the // seedForCheck inserts a bookmark and forces its latest_checked_at.
// series' latest_checked_at.
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) { func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(store.Bookmark{
Key: key, Key: key,
Site: site, Site: "asura",
SeriesID: seriesID, SeriesID: key,
SeriesURL: seriesURL, SeriesURL: seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed %q: %v", key, err) t.Fatalf("seed %q: %v", key, err)
} }
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil { if err := s.MarkLatestChecked(key, checkedAt); err != nil {
t.Fatalf("seed mark %q: %v", key, err) t.Fatalf("seed mark %q: %v", key, err)
} }
} }
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 { func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":") ts, err := s.LatestCheckedAt(key)
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
ts, err := s.LatestCheckedAt(site, seriesID)
if err != nil { if err != nil {
t.Fatalf("LatestCheckedAt %q: %v", key, err) t.Fatalf("LatestCheckedAt %q: %v", key, err)
} }
@@ -227,41 +214,6 @@ func TestRunOnceRespectsBatchLimit(t *testing.T) {
} }
} }
// The point of the split (ADR-0003): a series referenced by several bookmarks
// is fetched once per due cycle, not once per bookmark. Today the bookmark key
// is <site>:<series_id>, so the second bookmark only exists once keys stop
// being derived from the series identity (issue #22).
func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
s := newTestStore(t)
// The slug must match the fixture's own anchors: asura's parser scopes
// chapter links to the stored slug.
const slug = "chronicles-of-the-demon-faction-f886a8af"
const url = "https://asurascans.com/comics/" + slug
seedForCheck(t, s, "asura:"+slug, url, 0)
if _, err := s.Upsert(store.Bookmark{
Key: "asura:" + slug + ":2", Site: "asura", SeriesID: slug, UpdatedAt: 2000,
}); err != nil {
t.Fatalf("seed second reader: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background())
if got := f.callCount(); got != 1 {
t.Fatalf("fetched shared series %d times, want 1", got)
}
// Both bookmarks join to the same updated series row.
for _, key := range []string{"asura:" + slug, "asura:" + slug + ":2"} {
b, ok, err := s.Get(key)
if err != nil || !ok {
t.Fatalf("Get %s: %v ok=%v", key, err, ok)
}
if b.LatestChapterNum == nil || *b.LatestChapterNum != 181 {
t.Fatalf("%s LatestChapterNum = %v, want 181", key, b.LatestChapterNum)
}
}
}
// One unreachable series must not abandon the rest of the batch. // One unreachable series must not abandon the rest of the batch.
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) { func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
@@ -375,8 +327,8 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
now := time.UnixMilli(4_000_000) now := time.UnixMilli(4_000_000)
f := &fakeFetcher{body: asuraSeriesFixture, status: 200} f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, now).checkOne(context.Background(), store.Series{ newTestPoller(t, s, f, now).checkOne(context.Background(), store.Bookmark{
Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL, Key: key, Site: tt.site, SeriesURL: tt.seriesURL,
}) })
if got := f.callCount(); got != tt.wantCalls { if got := f.callCount(); got != tt.wantCalls {
+5 -9
View File
@@ -5,6 +5,8 @@ import (
"regexp" "regexp"
"strconv" "strconv"
"strings" "strings"
"bookmarkmanager/backend/internal/store"
) )
// latestChapter is the newest chapter a series page advertises. // latestChapter is the newest chapter a series page advertises.
@@ -20,12 +22,6 @@ type latestChapter struct {
// before using the slug to scope anything. // before using the slug to scope anything.
var asuraSlugRe = regexp.MustCompile(`/comics/([^/?#]+)`) var asuraSlugRe = regexp.MustCompile(`/comics/([^/?#]+)`)
// asuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura
// appends to every series slug. It rotates on each site redeploy, so it is
// never part of a stable series_id. Must stay in sync with stripBuildHash in
// userscript/manga-bookmark.user.js.
var asuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`)
// demonicChapterRe matches the pre-redirect anchors demonic series pages link // demonicChapterRe matches the pre-redirect anchors demonic series pages link
// through. Both the raw "&" and the HTML-escaped "&amp;" forms occur. // through. Both the raw "&" and the HTML-escaped "&amp;" forms occur.
var demonicChapterRe = regexp.MustCompile(`chaptered\.php\?manga=\d+&(?:amp;)?chapter=([0-9.]+)`) var demonicChapterRe = regexp.MustCompile(`chaptered\.php\?manga=\d+&(?:amp;)?chapter=([0-9.]+)`)
@@ -78,9 +74,9 @@ func latestChapterFrom(site, seriesURL, body string) (latestChapter, bool) {
} }
// Stored URLs predating a redeploy may carry a stale build hash; // Stored URLs predating a redeploy may carry a stale build hash;
// chapter hrefs in the fetched body carry the current one. Strip to // chapter hrefs in the fetched body carry the current one. Strip to
// the stable ID and make the hash optional in the pattern, so scoping // the stable ID (same rule as migrateAsuraKeys) and make the hash
// survives rotations. // optional in the pattern, so scoping survives rotations.
slug := asuraBuildHash.ReplaceAllString(m[1], "") slug := store.AsuraBuildHash.ReplaceAllString(m[1], "")
// Compiled per call rather than cached: this runs once per fetch, which // Compiled per call rather than cached: this runs once per fetch, which
// is at most a few times a minute, and the slug varies per series. // is at most a few times a minute, and the slug varies per series.
re = regexp.MustCompile(`/comics/` + regexp.QuoteMeta(slug) + `(?:-[0-9a-f]{8})?/chapter/([0-9.]+)`) re = regexp.MustCompile(`/comics/` + regexp.QuoteMeta(slug) + `(?:-[0-9a-f]{8})?/chapter/([0-9.]+)`)
-120
View File
@@ -1,120 +0,0 @@
// Package pgtest runs the Postgres the test suite needs: one throwaway
// container per test binary, one fresh database per test. Docker is therefore
// a hard prerequisite for `go test ./...`.
//
// Rolled by hand rather than pulled in as a dependency — it is one `docker
// run`, one `docker port` and a ping loop, against a module list that is
// otherwise stdlib plus what the poller genuinely needs.
package pgtest
import (
"database/sql"
"fmt"
"os/exec"
"strconv"
"strings"
"sync/atomic"
"testing"
"time"
_ "github.com/jackc/pgx/v5/stdlib"
)
const (
image = "postgres:17-alpine"
readyLimit = 60 * time.Second
)
var (
adminURL string
dbSeq atomic.Int64
)
// Main starts the container, runs the package's tests and tears the container
// down. Every test package that touches the store calls it from TestMain:
//
// func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
func Main(m *testing.M) int {
id, url, err := start()
if err != nil {
fmt.Println("pgtest:", err)
return 1
}
defer exec.Command("docker", "rm", "-f", id).Run()
adminURL = url
return m.Run()
}
// URL creates a database of its own for t and returns a connection URL for it.
// Nothing drops it again: the container goes away wholesale when Main returns.
func URL(t testing.TB) string {
t.Helper()
if adminURL == "" {
t.Fatal("pgtest: no container; this package needs TestMain to call pgtest.Main")
}
// Generated, never derived from the test name, so it needs no quoting and
// cannot collide when tests run in parallel.
name := "test_" + strconv.FormatInt(dbSeq.Add(1), 10)
admin, err := sql.Open("pgx", adminURL)
if err != nil {
t.Fatalf("pgtest: open admin connection: %v", err)
}
defer admin.Close()
if _, err := admin.Exec(`CREATE DATABASE ` + name); err != nil {
t.Fatalf("pgtest: create database %s: %v", name, err)
}
return strings.Replace(adminURL, "/postgres?", "/"+name+"?", 1)
}
// start launches the container and waits for it to accept queries, returning
// its id and a connection URL for the default database.
func start() (id, url string, err error) {
out, err := exec.Command("docker", "run", "-d", "--rm",
"-e", "POSTGRES_PASSWORD=pgtest",
"-P", image,
// Durability buys nothing for a database that dies with the test
// binary, and turning it off is most of the container's start-up cost.
"-c", "fsync=off", "-c", "full_page_writes=off",
).Output()
if err != nil {
return "", "", fmt.Errorf("docker run %s: %w", image, err)
}
id = strings.TrimSpace(string(out))
port, err := exec.Command("docker", "port", id, "5432/tcp").Output()
if err != nil {
exec.Command("docker", "rm", "-f", id).Run()
return "", "", fmt.Errorf("docker port: %w", err)
}
// "0.0.0.0:32768" (and possibly a second, IPv6 line); the port is all we want.
first, _, _ := strings.Cut(strings.TrimSpace(string(port)), "\n")
url = fmt.Sprintf("postgres://postgres:pgtest@127.0.0.1:%s/postgres?sslmode=disable",
first[strings.LastIndex(first, ":")+1:])
if err := waitReady(url); err != nil {
exec.Command("docker", "rm", "-f", id).Run()
return "", "", err
}
return id, url, nil
}
func waitReady(url string) error {
db, err := sql.Open("pgx", url)
if err != nil {
return err
}
defer db.Close()
deadline := time.Now().Add(readyLimit)
for {
if err = db.Ping(); err == nil {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("postgres not ready after %s: %w", readyLimit, err)
}
time.Sleep(200 * time.Millisecond)
}
}
@@ -1,28 +0,0 @@
-- One row per tracked series, keyed "<site>:<series_id>".
--
-- Everything is NOT NULL with a default except latest_chapter_num, where NULL
-- is a distinct state: nothing has been captured yet, which is not the same as
-- chapter zero.
--
-- Timestamps are unix milliseconds as bigint, not timestamptz: the userscripts
-- send Date.now() over the wire and the ordering rule compares them directly.
CREATE TABLE bookmarks (
key text PRIMARY KEY,
site text NOT NULL,
series_id text NOT NULL,
title text NOT NULL DEFAULT '',
series_url text NOT NULL DEFAULT '',
cover text NOT NULL DEFAULT '',
last_chapter text NOT NULL DEFAULT '',
last_chapter_num double precision NOT NULL DEFAULT 0,
last_chapter_url text NOT NULL DEFAULT '',
favorite boolean NOT NULL DEFAULT false,
latest_chapter text NOT NULL DEFAULT '',
latest_chapter_num double precision,
-- When the server last polled this series, unix ms; 0 means never, and sorts
-- first so a new bookmark is picked up on the next tick with no special case.
latest_checked_at bigint NOT NULL DEFAULT 0,
status text NOT NULL DEFAULT 'reading',
kind text NOT NULL DEFAULT 'manga',
updated_at bigint NOT NULL
);
@@ -1,45 +0,0 @@
-- One row per distinct work, shared by every bookmark that tracks it
-- (ADR-0003). Keyed (site, series_id), the pair a bookmark key decomposes
-- into. title/series_url/cover are written once, at creation, and never
-- again: client-supplied values are ignored once the row exists and the
-- poller is the only party that may change them. kind and the latest-chapter
-- fields are last-write-wins like the bookmark's own fields.
CREATE TABLE series (
site text NOT NULL,
series_id text NOT NULL,
title text NOT NULL DEFAULT '',
series_url text NOT NULL DEFAULT '',
cover text NOT NULL DEFAULT '',
kind text NOT NULL DEFAULT 'manga',
latest_chapter text NOT NULL DEFAULT '',
latest_chapter_num double precision,
-- When the server last polled this series, unix ms; 0 means never, and sorts
-- first so a new bookmark is picked up on the next tick with no special case.
latest_checked_at bigint NOT NULL DEFAULT 0,
PRIMARY KEY (site, series_id)
);
-- Backfill from today's rows. The bookmark key's uniqueness makes
-- (site, series_id) unique in practice; DISTINCT is belt and braces.
INSERT INTO series (site, series_id, title, series_url, cover, kind,
latest_chapter, latest_chapter_num, latest_checked_at)
SELECT DISTINCT site, series_id, title, series_url, cover, kind,
latest_chapter, latest_chapter_num, latest_checked_at
FROM bookmarks;
-- The bookmark keeps only what differs between readers (ADR-0003): progress,
-- favourite, lifecycle bucket. The dropped columns now live on series.
ALTER TABLE bookmarks
DROP COLUMN title,
DROP COLUMN series_url,
DROP COLUMN cover,
DROP COLUMN kind,
DROP COLUMN latest_chapter,
DROP COLUMN latest_chapter_num,
DROP COLUMN latest_checked_at;
-- A bookmark may not point at a series that does not exist. No cascade: a
-- series outlives its last bookmark, and deleting one is not a store operation.
ALTER TABLE bookmarks
ADD CONSTRAINT bookmarks_series_fk
FOREIGN KEY (site, series_id) REFERENCES series (site, series_id);
+250 -265
View File
@@ -2,28 +2,19 @@ package store
import ( import (
"database/sql" "database/sql"
"embed"
"errors" "errors"
"fmt" "fmt"
"io/fs"
"path"
"regexp" "regexp"
"slices"
"strconv" "strconv"
"strings" "strings"
_ "github.com/jackc/pgx/v5/stdlib" _ "modernc.org/sqlite"
) )
// Bookmark is one tracked series, keyed "<site>:<series_id>" across both sites. // Bookmark is one tracked series, keyed "<site>:<series_id>" across both sites.
// //
// LastChapter* is the user's read progress; LatestChapter* is the newest // LastChapter* is the user's read progress; LatestChapter* is the newest
// chapter the site has published, captured opportunistically by the userscript. // chapter the site has published, captured opportunistically by the userscript.
//
// Title, SeriesURL, Cover, Kind and LatestChapter* live on the shared Series
// row (ADR-0003) and are joined in on read; Bookmark carries only what differs
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
// format stays flat regardless — see ADR-0004.
type Bookmark struct { type Bookmark struct {
Key string `json:"key"` Key string `json:"key"`
Site string `json:"site"` Site string `json:"site"`
@@ -47,35 +38,6 @@ type Bookmark struct {
Kind string `json:"kind"` Kind string `json:"kind"`
} }
// Series is one distinct work, shared by every bookmark that tracks it. It is
// keyed (site, series_id) — the pair a bookmark key decomposes into — and
// exists once no matter how many bookmarks point at it (ADR-0003).
//
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
// existing Series has them ignored, and only the backend's own Poll may change
// them. Kind and the latest-chapter fields are last-write-wins like the
// bookmark's own fields. Never serialized: the wire format is the flat
// Bookmark (ADR-0004).
type Series struct {
Site string
SeriesID string
Title string
SeriesURL string
Cover string
Kind string
LatestChapter string
LatestChapterNum *float64 // nil until first captured
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
// readerCount is the number of bookmarks referencing this series, filled
// only by the due-queue query that orders on it.
readerCount int
}
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
// used by the poller's logs and by tests asserting on the due queue.
func (s Series) Key() string { return s.Site + ":" + s.SeriesID }
// HasNewChapter reports whether the site has published past the read point. // HasNewChapter reports whether the site has published past the read point.
// A nil LatestChapterNum means nothing has been captured yet, which is not the // A nil LatestChapterNum means nothing has been captured yet, which is not the
// same as "nothing new". // same as "nothing new".
@@ -152,164 +114,236 @@ const (
StatusFinished = "finished" StatusFinished = "finished"
) )
//go:embed migrations/*.sql const schema = `
var migrations embed.FS CREATE TABLE IF NOT EXISTS bookmarks (
key TEXT PRIMARY KEY,
site TEXT NOT NULL,
series_id TEXT NOT NULL,
title TEXT,
series_url TEXT,
cover TEXT,
last_chapter TEXT,
last_chapter_num REAL,
last_chapter_url TEXT,
favorite INTEGER NOT NULL DEFAULT 0,
latest_chapter TEXT NOT NULL DEFAULT '',
latest_chapter_num REAL,
latest_checked_at INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'reading',
kind TEXT NOT NULL DEFAULT 'manga',
updated_at INTEGER NOT NULL
);`
// bookmarkColumns is the only value ever concatenated into query text. It is a // The columns above that databases created before them will be missing.
// compile-time constant; every request value is bound as a parameter. The // SQLite has no ADD COLUMN IF NOT EXISTS, so each is added only when absent.
// series-owned fields are joined in from the series table, in scanBookmark var addedColumns = []struct{ name, ddl string }{
// order, so the flat Bookmark reads back whole despite the split (ADR-0004). {"favorite", `ALTER TABLE bookmarks ADD COLUMN favorite INTEGER NOT NULL DEFAULT 0`},
const bookmarkColumns = `b.key, b.site, b.series_id, s.title, s.series_url, s.cover, {"latest_chapter", `ALTER TABLE bookmarks ADD COLUMN latest_chapter TEXT NOT NULL DEFAULT ''`},
b.last_chapter, b.last_chapter_num, b.last_chapter_url, {"latest_chapter_num", `ALTER TABLE bookmarks ADD COLUMN latest_chapter_num REAL`},
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind` // When the server last looked at this series, unix ms; 0 means never, and
// sorts first so a new bookmark is picked up on the next tick with no
// special case. Deliberately NOT in bookmarkColumns — see MarkLatestChecked.
{"latest_checked_at", `ALTER TABLE bookmarks ADD COLUMN latest_checked_at INTEGER NOT NULL DEFAULT 0`},
// Lifecycle bucket. The DEFAULT backfills every pre-existing row as
// 'reading', so there is no separate migration step.
{"status", `ALTER TABLE bookmarks ADD COLUMN status TEXT NOT NULL DEFAULT 'reading'`},
// Library bucket. The DEFAULT backfills every pre-existing row as 'manga',
// which is what every row written before novels existed actually is.
{"kind", `ALTER TABLE bookmarks ADD COLUMN kind TEXT NOT NULL DEFAULT 'manga'`},
}
// seriesColumns is the series row in scanSeries order, used by the poller's const bookmarkColumns = `key, site, series_id, title, series_url, cover,
// due query. latest_checked_at lives only on series — see MarkLatestChecked last_chapter, last_chapter_num, last_chapter_url,
// for why it stays off every client-visible write. favorite, latest_chapter, latest_chapter_num, updated_at, status, kind`
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
// Store is the Postgres-backed bookmark store. // Store is the SQLite-backed bookmark store.
type Store struct { type Store struct {
db *sql.DB db *sql.DB
} }
// Open connects to Postgres at url — a libpq connection URL such as // OpenStore opens (or creates) the SQLite database at path and applies the schema.
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — and brings its func Open(path string) (*Store, error) {
// schema up to date. // busy_timeout guards against SQLITE_BUSY under the reverse proxy's
func Open(url string) (*Store, error) { // concurrent requests; a single writer connection keeps writes serialized.
db, err := sql.Open("pgx", url) dsn := path
if err != nil { if !strings.Contains(dsn, "?") {
return nil, fmt.Errorf("open postgres: %w", err) dsn += "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
} }
if err := migrate(db); err != nil { db, err := sql.Open("sqlite", dsn)
if err != nil {
return nil, fmt.Errorf("open sqlite %q: %w", path, err)
}
db.SetMaxOpenConns(1)
if _, err := db.Exec(schema); err != nil {
db.Close() db.Close()
return nil, fmt.Errorf("migrate: %w", err) return nil, fmt.Errorf("apply schema: %w", err)
}
if err := migrateColumns(db); err != nil {
db.Close()
return nil, fmt.Errorf("migrate schema: %w", err)
}
if err := migrateAsuraKeys(db); err != nil {
db.Close()
return nil, fmt.Errorf("migrate asura keys: %w", err)
} }
return &Store{db: db}, nil return &Store{db: db}, nil
} }
// migrate applies every embedded migration this database has not recorded, in // migrateColumns brings a pre-existing bookmarks table up to the current
// filename order, each in its own transaction. Files are named // schema. Safe to run on every start: columns already present are skipped.
// "<version>_<name>.sql" and are append-only: editing an applied file changes func migrateColumns(db *sql.DB) error {
// nothing, because schema_migrations is how a database remembers what it ran. have, err := existingColumns(db, "bookmarks")
// Runs on every start and is a no-op once current.
func migrate(db *sql.DB) error {
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
version bigint PRIMARY KEY,
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
return fmt.Errorf("create version table: %w", err)
}
names, err := fs.Glob(migrations, "migrations/*.sql")
if err != nil { if err != nil {
return err return err
} }
slices.Sort(names) for _, c := range addedColumns {
if _, ok := have[c.name]; ok {
for _, name := range names { continue
version, err := strconv.ParseInt(strings.SplitN(path.Base(name), "_", 2)[0], 10, 64)
if err != nil {
return fmt.Errorf("migration %q: filename must start with a version number", name)
} }
body, err := migrations.ReadFile(name) if _, err := db.Exec(c.ddl); err != nil {
if err != nil { return fmt.Errorf("add column %q: %w", c.name, err)
return err
}
if err := applyMigration(db, version, string(body)); err != nil {
return fmt.Errorf("migration %q: %w", name, err)
} }
} }
return nil return nil
} }
// applyMigration runs one migration and records its version in the same // AsuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura
// transaction, so an interrupted start leaves neither half behind. // appends to every series slug. It rotates on each site redeploy, so it
func applyMigration(db *sql.DB, version int64, body string) error { // must not be part of series_id. Must stay in sync with stripBuildHash in
tx, err := db.Begin() // userscript/manga-bookmark.user.js.
if err != nil { var AsuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`)
return err
}
defer tx.Rollback()
var applied bool // migrateAsuraKeys rewrites asura bookmarks whose series_id still carries
if err := tx.QueryRow( // the build hash to the stable, hashless ID. Rows keyed with a hash are
`SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1)`, // orphaned on every Asura redeploy (old-hash URLs 302 to new-hash ones, so
version).Scan(&applied); err != nil { // detection yields a key that never matches). When two hash-generations of
// one series collide, the row with the newest updated_at wins and the rest
// are deleted. Idempotent: hashless IDs never match the regex.
func migrateAsuraKeys(db *sql.DB) error {
rows, err := db.Query(`SELECT key, series_id, updated_at FROM bookmarks WHERE site = 'asura'`)
if err != nil {
return fmt.Errorf("list asura rows: %w", err)
}
type row struct {
key, id string
updated int64
}
var all []row
for rows.Next() {
var r row
if err := rows.Scan(&r.key, &r.id, &r.updated); err != nil {
rows.Close()
return fmt.Errorf("scan asura row: %w", err)
}
all = append(all, r)
}
if err := rows.Close(); err != nil {
return err return err
} }
if applied {
return nil groups := map[string][]row{}
for _, r := range all {
stripped := AsuraBuildHash.ReplaceAllString(r.id, "")
groups[stripped] = append(groups[stripped], r)
} }
// No parameters, so this goes over the simple protocol and a migration may for stripped, g := range groups {
// hold more than one statement. winner := 0
if _, err := tx.Exec(body); err != nil { for i := range g {
return err if g[i].updated > g[winner].updated {
winner = i
}
}
// Losers go first: rewriting the winner to the stripped key while a
// pre-existing hashless row still holds it is a primary-key collision.
for i, r := range g {
if i == winner {
continue
}
if _, err := db.Exec(`DELETE FROM bookmarks WHERE key = ?`, r.key); err != nil {
return fmt.Errorf("drop duplicate %q: %w", r.key, err)
}
}
if r := g[winner]; r.id != stripped {
if _, err := db.Exec(
`UPDATE bookmarks SET key = ?, series_id = ? WHERE key = ?`,
"asura:"+stripped, stripped, r.key); err != nil {
return fmt.Errorf("rewrite key %q: %w", r.key, err)
}
}
} }
if _, err := tx.Exec(`INSERT INTO schema_migrations (version) VALUES ($1)`, version); err != nil { return nil
return err
}
return tx.Commit()
} }
// scanBookmark reads one row in bookmarkColumns order. Every column is NOT func existingColumns(db *sql.DB, table string) (map[string]struct{}, error) {
// NULL except latest_chapter_num, where NULL means "never captured" — a rows, err := db.Query(`SELECT name FROM pragma_table_info(?)`, table)
// distinct state from chapter zero, and the reason for the pointer. if err != nil {
return nil, fmt.Errorf("read %s columns: %w", table, err)
}
defer rows.Close()
out := map[string]struct{}{}
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
return nil, fmt.Errorf("scan column name: %w", err)
}
out[name] = struct{}{}
}
return out, rows.Err()
}
// scanBookmark reads one row in bookmarkColumns order, translating SQLite's
// integer bool and nullable latest_chapter_num into Go types.
//
// The optional columns are read through Null* types because rows predating
// this code (or written by hand) may hold NULL where the app only ever writes
// zero values. Only latest_chapter_num distinguishes the two: everywhere else
// NULL and the zero value mean the same thing to clients.
func scanBookmark(scan func(...any) error) (Bookmark, error) { func scanBookmark(scan func(...any) error) (Bookmark, error) {
var ( var (
b Bookmark b Bookmark
latestChapterNum sql.NullFloat64 title, seriesURL, cover sql.NullString
lastChapter, lastChapterURL, latestChapter sql.NullString
status sql.NullString
lastChapterNum, latestChapterNum sql.NullFloat64
favorite sql.NullInt64
) )
if err := scan( if err := scan(
&b.Key, &b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover, &b.Key, &b.Site, &b.SeriesID, &title, &seriesURL, &cover,
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL, &lastChapter, &lastChapterNum, &lastChapterURL,
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind, &favorite, &latestChapter, &latestChapterNum, &b.UpdatedAt, &status, &b.Kind,
); err != nil { ); err != nil {
return Bookmark{}, err return Bookmark{}, err
} }
b.Title = title.String
b.SeriesURL = seriesURL.String
b.Cover = cover.String
b.LastChapter = lastChapter.String
b.LastChapterNum = lastChapterNum.Float64
b.LastChapterURL = lastChapterURL.String
b.Favorite = favorite.Int64 != 0
b.LatestChapter = latestChapter.String
if latestChapterNum.Valid { if latestChapterNum.Valid {
b.LatestChapterNum = &latestChapterNum.Float64 b.LatestChapterNum = &latestChapterNum.Float64
} }
// An unrecognised bucket (a hand-edited row) would leave the row in no list // A NULL, empty, or unrecognised bucket (e.g. a hand-edited row) would
// at all, so anything outside the three known buckets reads as the default // leave the row in no list at all, so anything outside the three known
// rather than being passed through. // buckets reads as the default rather than being passed through.
b.Status = status.String
if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished { if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished {
b.Status = StatusReading b.Status = StatusReading
} }
return b, nil return b, nil
} }
// scanSeries reads one row in seriesColumns order, plus the due query's
// reader_count column. latest_chapter_num is NULL until the first capture,
// same as on the bookmark read path.
func scanSeries(scan func(...any) error) (Series, error) {
var (
sr Series
latestChapterNum sql.NullFloat64
)
if err := scan(
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
&sr.readerCount,
); err != nil {
return Series{}, err
}
if latestChapterNum.Valid {
sr.LatestChapterNum = &latestChapterNum.Float64
}
return sr, nil
}
// Close releases the underlying database handle. // Close releases the underlying database handle.
func (s *Store) Close() error { return s.db.Close() } func (s *Store) Close() error { return s.db.Close() }
// List returns every bookmark, newest activity first. Series-owned fields are // List returns every bookmark, newest activity first.
// joined in, so each Bookmark reads back whole and flat (ADR-0004).
func (s *Store) List() ([]Bookmark, error) { func (s *Store) List() ([]Bookmark, error) {
rows, err := s.db.Query(`SELECT ` + bookmarkColumns + ` rows, err := s.db.Query(`SELECT ` + bookmarkColumns + `
FROM bookmarks b FROM bookmarks
JOIN series s ON s.site = b.site AND s.series_id = b.series_id ORDER BY updated_at DESC`)
ORDER BY b.updated_at DESC`)
if err != nil { if err != nil {
return nil, fmt.Errorf("query bookmarks: %w", err) return nil, fmt.Errorf("query bookmarks: %w", err)
} }
@@ -331,9 +365,7 @@ func (s *Store) List() ([]Bookmark, error) {
// the fields they do not touch. // the fields they do not touch.
func (s *Store) Get(key string) (Bookmark, bool, error) { func (s *Store) Get(key string) (Bookmark, bool, error) {
b, err := scanBookmark(s.db.QueryRow( b, err := scanBookmark(s.db.QueryRow(
`SELECT `+bookmarkColumns+` FROM bookmarks b `SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, key).Scan)
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.key = $1`, key).Scan)
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
return Bookmark{}, false, nil return Bookmark{}, false, nil
} }
@@ -344,15 +376,7 @@ func (s *Store) Get(key string) (Bookmark, bool, error) {
} }
// Upsert inserts or replaces a bookmark by key (last-write-wins) and returns // Upsert inserts or replaces a bookmark by key (last-write-wins) and returns
// the row as actually stored — one flat object with the series-owned fields // the row as actually stored.
// joined in, exactly as GET reports it (ADR-0004).
//
// The flat body is decomposed across two tables in one transaction. The series
// row is written first (the bookmarks FK requires it to exist), then the
// bookmark row. On the series side, title/series_url/cover are applied only
// when the row is brand new: once a series exists, client-supplied values are
// ignored, because the row is shared and the values are scraped page content —
// see ADR-0003. Kind and the latest-chapter fields are last-write-wins.
// //
// b.UpdatedAt is only a candidate: it is applied when the row is new or when // b.UpdatedAt is only a candidate: it is applied when the row is new or when
// last_chapter_num changes, and otherwise the stored value is kept. Clients // last_chapter_num changes, and otherwise the stored value is kept. Clients
@@ -371,65 +395,49 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
latestNum = *b.LatestChapterNum latestNum = *b.LatestChapterNum
} }
// The kind column resolves on the VALUES side, not in the conflict clause: // IS NOT is SQLite's null-safe comparison. Within DO UPDATE, a bare column
// excluded.* is the row *after* these expressions are evaluated, so a // is the stored row and excluded.* is the incoming one; a brand-new key
// default applied there would look identical to a real 'manga' and would // never reaches this clause, so it keeps the fresh timestamp from VALUES.
// overwrite a novel series on every PUT from a client that knows nothing
// about the column. Resolved once here, an empty incoming kind means "keep
// what is stored", and only a brand-new row falls through to the literal
// default. The subquery runs inside this transaction, so it sees the row
// this statement is about to conflict with. Same pattern as the status
// COALESCE on the bookmark insert below.
// //
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no // The status and kind columns resolve on the VALUES side, not in the
// target column to infer the parameter type from, and Postgres rejects the // conflict clause: excluded.* is the row *after* these expressions are
// statement rather than guessing. // evaluated, so a default applied there would look identical to a real
// 'reading' / 'manga' and would overwrite an archived or novel row on
// every PUT from a client that knows nothing about the column. Resolved
// once here, an empty incoming status or kind means "keep what is
// stored", and only a brand-new row falls through to the literal
// default. The subquery runs inside this transaction, so it sees the
// row this statement is about to conflict with.
if _, err := tx.Exec(` if _, err := tx.Exec(`
INSERT INTO series (site, series_id, title, series_url, cover, kind, INSERT INTO bookmarks (`+bookmarkColumns+`)
latest_chapter, latest_chapter_num) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
VALUES ($1, $2, $3, $4, $5, COALESCE(NULLIF(?, ''), (SELECT status FROM bookmarks WHERE key = ?), 'reading'),
COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'), COALESCE(NULLIF(?, ''), (SELECT kind FROM bookmarks WHERE key = ?), 'manga'))
$7, $8) ON CONFLICT(key) DO UPDATE SET
ON CONFLICT (site, series_id) DO UPDATE SET site=excluded.site, series_id=excluded.series_id, title=excluded.title,
kind=excluded.kind, series_url=excluded.series_url, cover=excluded.cover,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num`,
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
b.LatestChapter, latestNum); err != nil {
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
}
// IS DISTINCT FROM is Postgres's null-safe comparison, and it is what
// implements the ordering rule. Within DO UPDATE, a bare column is the
// stored row and excluded.* is the incoming one; a brand-new key never
// reaches this clause, so it keeps the fresh timestamp from VALUES.
if _, err := tx.Exec(`
INSERT INTO bookmarks (key, site, series_id, last_chapter, last_chapter_num,
last_chapter_url, favorite, status, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7,
COALESCE(NULLIF($8::text, ''), (SELECT status FROM bookmarks WHERE key = $1), 'reading'),
$9)
ON CONFLICT (key) DO UPDATE SET
site=excluded.site, series_id=excluded.series_id,
last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num, last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num,
last_chapter_url=excluded.last_chapter_url, last_chapter_url=excluded.last_chapter_url,
favorite=excluded.favorite, favorite=excluded.favorite,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num,
status=excluded.status, status=excluded.status,
kind=excluded.kind,
updated_at=CASE updated_at=CASE
WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num WHEN bookmarks.last_chapter_num IS NOT excluded.last_chapter_num
THEN excluded.updated_at THEN excluded.updated_at
ELSE bookmarks.updated_at ELSE bookmarks.updated_at
END`, END`,
b.Key, b.Site, b.SeriesID, b.Key, b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover,
b.LastChapter, b.LastChapterNum, b.LastChapterURL, b.LastChapter, b.LastChapterNum, b.LastChapterURL,
b.Favorite, b.Status, b.UpdatedAt); err != nil { b.Favorite, b.LatestChapter, latestNum, b.UpdatedAt,
b.Status, b.Key,
b.Kind, b.Key); err != nil {
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err) return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
} }
stored, err := scanBookmark(tx.QueryRow( stored, err := scanBookmark(tx.QueryRow(
`SELECT `+bookmarkColumns+` FROM bookmarks b `SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, b.Key).Scan)
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.key = $1`, b.Key).Scan)
if err != nil { if err != nil {
return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err) return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err)
} }
@@ -441,100 +449,77 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
// Delete removes a bookmark by key. Deleting a missing key is not an error. // Delete removes a bookmark by key. Deleting a missing key is not an error.
func (s *Store) Delete(key string) error { func (s *Store) Delete(key string) error {
if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = $1`, key); err != nil { if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = ?`, key); err != nil {
return fmt.Errorf("delete %q: %w", key, err) return fmt.Errorf("delete %q: %w", key, err)
} }
return nil return nil
} }
// DueForLatestCheck returns series whose server-side latest-chapter check has // DueForLatestCheck returns bookmarks whose server-side latest-chapter check has
// aged past cutoffMs, ordered by how many bookmarks reference them (descending) // aged past cutoffMs, least-recently-checked first, at most limit of them.
// then least-recently-checked first, at most limit of them.
// //
// The reader_count ordering is the point of the split (ADR-0003): a series // Oldest-first is what keeps the poller fair when the backlog outgrows its
// shared by several readers is fetched once per due cycle, and the popular
// ones stay freshest while the long tail absorbs any shortfall. Within one
// reader count, oldest-first keeps the poll fair when the backlog outgrows its
// throughput: the most neglected series is always next, so a large collection // throughput: the most neglected series is always next, so a large collection
// refreshes uniformly slower rather than leaving a tail that never refreshes at // refreshes uniformly slower rather than leaving a tail that never refreshes at
// all. The userscript sorts its own queue the same way (L453). // all. The userscript sorts its own queue the same way (L453).
// //
// Series with no series_url are skipped — there is nothing to fetch, which is // Bookmarks with no series_url are skipped — there is nothing to fetch, which
// the same filter the userscript applies at L452. Series whose only bookmarks // is the same filter the userscript applies at L452.
// are finished are skipped too: nothing more is coming, so fetching them only //
// burns requests. Archived bookmarks still count — knowing what a shelved // Finished series are excluded: nothing more is coming, so fetching them only
// series is up to is the whole reason for archiving instead of deleting. // burns requests. Archived ones are deliberately still polled — knowing what a
// A series with no bookmarks at all never appears: the join excludes it. // shelved series is up to is the whole reason for archiving instead of deleting.
func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Series, error) { func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Bookmark, error) {
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(b.key) AS reader_count rows, err := s.db.Query(`SELECT `+bookmarkColumns+`
FROM series s FROM bookmarks
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id WHERE series_url IS NOT NULL AND series_url <> ''
WHERE s.series_url <> '' AND status IS NOT 'finished'
AND s.latest_checked_at <= $1 AND latest_checked_at <= ?
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover, ORDER BY latest_checked_at ASC
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at LIMIT ?`, cutoffMs, limit)
HAVING COUNT(b.key) FILTER (WHERE b.status <> 'finished') > 0
ORDER BY reader_count DESC, s.latest_checked_at ASC
LIMIT $2`, cutoffMs, limit)
if err != nil { if err != nil {
return nil, fmt.Errorf("query due series: %w", err) return nil, fmt.Errorf("query due bookmarks: %w", err)
} }
defer rows.Close() defer rows.Close()
out := []Series{} out := []Bookmark{}
for rows.Next() { for rows.Next() {
sr, err := scanSeries(rows.Scan) b, err := scanBookmark(rows.Scan)
if err != nil { if err != nil {
return nil, fmt.Errorf("scan due series: %w", err) return nil, fmt.Errorf("scan due bookmark: %w", err)
} }
out = append(out, sr) out = append(out, b)
} }
return out, rows.Err() return out, rows.Err()
} }
// MarkLatestChecked records that the server looked at a series at ts, whatever // MarkLatestChecked records that the server looked at key at ts, whatever the
// the look turned up. Marking a missing series is not an error: the row may // look turned up. Marking a missing key is not an error: the row may have been
// have been orphaned while a fetch was in flight. // deleted while a fetch was in flight.
// //
// This is the one write that does not go through Upsert, and the column is kept // This is the one write that does not go through Upsert, and the column is kept
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes // out of bookmarkColumns on purpose. PUT /bookmarks/{key} decodes a whole
// a whole Bookmark from the client and Upsert writes every series column it // Bookmark from the client and Upsert writes every column it knows about, so a
// knows about, so a userscript PUT — which has no idea this field exists — // userscript PUT — which has no idea this field exists — would write a zero and
// would write a zero and reset the cooldown, making the poller re-fetch that // reset the cooldown, making the poller re-fetch that series every tick for as
// series every tick for as long as the user kept reading it. // long as the user kept reading it.
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error { func (s *Store) MarkLatestChecked(key string, ts int64) error {
if _, err := s.db.Exec( if _, err := s.db.Exec(
`UPDATE series SET latest_checked_at = $1 WHERE site = $2 AND series_id = $3`, `UPDATE bookmarks SET latest_checked_at = ? WHERE key = ?`, ts, key); err != nil {
ts, site, seriesID); err != nil { return fmt.Errorf("mark checked %q: %w", key, err)
return fmt.Errorf("mark checked %s:%s: %w", site, seriesID, err)
} }
return nil return nil
} }
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for // LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
// tests outside this package (the poller's own tests assert on cooldown // tests outside this package (the poller's own tests assert on cooldown
// bookkeeping) — see MarkLatestChecked for why the field stays off the // bookkeeping) — see MarkLatestChecked for why the field itself stays off
// client-visible row. // Bookmark.
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) { func (s *Store) LatestCheckedAt(key string) (int64, error) {
var ts int64 var ts int64
if err := s.db.QueryRow( if err := s.db.QueryRow(
`SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`, `SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil {
site, seriesID).Scan(&ts); err != nil { return 0, fmt.Errorf("latest checked at %q: %w", key, err)
return 0, fmt.Errorf("latest checked at %s:%s: %w", site, seriesID, err)
} }
return ts, nil return ts, nil
} }
// SetLatestChapter records the newest chapter the poll found on a series page.
// The poller walks Series rather than Bookmarks, so this is a series-level
// write: the row is shared, and updating it once refreshes every bookmark that
// joins to it. Touching a missing series is not an error.
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
if _, err := s.db.Exec(
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
WHERE site = $1 AND series_id = $2`,
site, seriesID, label, num); err != nil {
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
}
return nil
}
+312 -302
View File
@@ -2,55 +2,79 @@ package store
import ( import (
"database/sql" "database/sql"
"os" "path/filepath"
"strings"
"testing" "testing"
"time" "time"
"bookmarkmanager/backend/internal/pgtest"
) )
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
func newTestStore(t *testing.T) *Store { func newTestStore(t *testing.T) *Store {
t.Helper() t.Helper()
store, err := Open(pgtest.URL(t)) store, err := Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil { if err != nil {
t.Fatalf("Open: %v", err) t.Fatalf("OpenStore: %v", err)
} }
t.Cleanup(func() { store.Close() }) t.Cleanup(func() { store.Close() })
return store return store
} }
// The migration runner runs on every start, so a second Open against a func TestOpenStoreMigratesLegacySchema(t *testing.T) {
// database it already built must be a no-op rather than a duplicate-table dbPath := filepath.Join(t.TempDir(), "legacy.db")
// error, and must leave the rows alone.
func TestOpenIsIdempotent(t *testing.T) {
url := pgtest.URL(t)
first, err := Open(url)
if err != nil {
t.Fatalf("Open: %v", err)
}
if _, err := first.Upsert(Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
first.Close()
second, err := Open(url) legacy, err := sql.Open("sqlite", dbPath)
if err != nil { if err != nil {
t.Fatalf("reopen: %v", err) t.Fatalf("open legacy db: %v", err)
}
if _, err := legacy.Exec(`
CREATE TABLE bookmarks (
key TEXT PRIMARY KEY,
site TEXT NOT NULL,
series_id TEXT NOT NULL,
title TEXT,
series_url TEXT,
cover TEXT,
last_chapter TEXT,
last_chapter_num REAL,
last_chapter_url TEXT,
updated_at INTEGER NOT NULL
)`); err != nil {
t.Fatalf("create legacy schema: %v", err)
}
if _, err := legacy.Exec(`
INSERT INTO bookmarks (key, site, series_id, title, last_chapter, last_chapter_num, updated_at)
VALUES ('asura:legacy', 'asura', 'legacy', 'Legacy Series', 'Chapter 7', 7, 123)`); err != nil {
t.Fatalf("seed legacy row: %v", err)
}
if err := legacy.Close(); err != nil {
t.Fatalf("close legacy db: %v", err)
} }
t.Cleanup(func() { second.Close() })
list, err := second.List() store, err := Open(dbPath)
if err != nil {
t.Fatalf("OpenStore on legacy db: %v", err)
}
t.Cleanup(func() { store.Close() })
list, err := store.List()
if err != nil { if err != nil {
t.Fatalf("List: %v", err) t.Fatalf("List: %v", err)
} }
if len(list) != 1 || list[0].Key != "asura:solo" { if len(list) != 1 || list[0].Key != "asura:legacy" {
t.Fatalf("rows after reopen = %+v, want only the seeded one", list) t.Fatalf("legacy row lost: %+v", list)
} }
got := list[0]
if got.Title != "Legacy Series" || got.LastChapterNum != 7 || got.UpdatedAt != 123 {
t.Fatalf("legacy data mangled: %+v", got)
}
if got.Favorite || got.LatestChapter != "" || got.LatestChapterNum != nil {
t.Fatalf("new columns should default empty, got %+v", got)
}
// Reopening an already-migrated database must be a no-op, not an error.
store2, err := Open(dbPath)
if err != nil {
t.Fatalf("OpenStore is not idempotent: %v", err)
}
store2.Close()
} }
func TestStoreGet(t *testing.T) { func TestStoreGet(t *testing.T) {
@@ -126,41 +150,30 @@ func TestBookmarkContinueURL(t *testing.T) {
} }
// readLatestCheckedAt reads the column directly. It is deliberately absent from // readLatestCheckedAt reads the column directly. It is deliberately absent from
// Series (see Store.MarkLatestChecked), so tests cannot assert on it any other // Bookmark (see Store.Upsert), so tests cannot assert on it any other way.
// way. The key splits the same way the API handler derives site/series_id.
func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 { func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
var ts int64 var ts int64
if err := s.db.QueryRow( if err := s.db.QueryRow(
`SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`, `SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil {
site, seriesID).Scan(&ts); err != nil {
t.Fatalf("read latest_checked_at %q: %v", key, err) t.Fatalf("read latest_checked_at %q: %v", key, err)
} }
return ts return ts
} }
// seedForCheck inserts a bookmark (and with it its series) and forces the // seedForCheck inserts a bookmark and forces its latest_checked_at.
// series' latest_checked_at.
func seedForCheck(t *testing.T, s *Store, key, seriesURL string, checkedAt int64) { func seedForCheck(t *testing.T, s *Store, key, seriesURL string, checkedAt int64) {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := s.Upsert(Bookmark{ if _, err := s.Upsert(Bookmark{
Key: key, Key: key,
Site: site, Site: "asura",
SeriesID: seriesID, SeriesID: key,
SeriesURL: seriesURL, SeriesURL: seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed %q: %v", key, err) t.Fatalf("seed %q: %v", key, err)
} }
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil { if err := s.MarkLatestChecked(key, checkedAt); err != nil {
t.Fatalf("seed mark %q: %v", key, err) t.Fatalf("seed mark %q: %v", key, err)
} }
} }
@@ -211,8 +224,8 @@ func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) {
if len(due) != 2 { if len(due) != 2 {
t.Fatalf("got %d rows, want 2 (limit)", len(due)) t.Fatalf("got %d rows, want 2 (limit)", len(due))
} }
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" { if due[0].Key != "asura:a" || due[1].Key != "asura:b" {
t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key()) t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key, due[1].Key)
} }
} }
@@ -220,21 +233,20 @@ func TestMarkLatestChecked(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 0) seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 0)
if err := s.MarkLatestChecked("asura", "x", 4242); err != nil { if err := s.MarkLatestChecked("asura:x", 4242); err != nil {
t.Fatalf("MarkLatestChecked: %v", err) t.Fatalf("MarkLatestChecked: %v", err)
} }
if got := readLatestCheckedAt(t, s, "asura:x"); got != 4242 { if got := readLatestCheckedAt(t, s, "asura:x"); got != 4242 {
t.Fatalf("latest_checked_at = %d, want 4242", got) t.Fatalf("latest_checked_at = %d, want 4242", got)
} }
// A missing series is not an error: its bookmarks may have been deleted // A missing key is not an error: the row may have been deleted mid-fetch.
// mid-fetch. if err := s.MarkLatestChecked("asura:gone", 1); err != nil {
if err := s.MarkLatestChecked("asura", "gone", 1); err != nil { t.Fatalf("MarkLatestChecked on missing key: %v", err)
t.Fatalf("MarkLatestChecked on missing series: %v", err)
} }
} }
// Upsert must not touch latest_checked_at. If the column ever migrates into // Upsert must not touch latest_checked_at. If the column ever migrates into
// the client-visible write path, this fails and the cooldown is silently dead. // bookmarkColumns, this fails and the cooldown is silently dead.
func TestUpsertPreservesLatestCheckedAt(t *testing.T) { func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 999) seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 999)
@@ -252,6 +264,53 @@ func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
} }
} }
// migrateColumns must be able to bring a database created before this column up
// to date, not just create it fresh.
func TestMigrateAddsLatestCheckedAt(t *testing.T) {
path := filepath.Join(t.TempDir(), "old.db")
old, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("open: %v", err)
}
// A pre-latest_checked_at table, matching the schema as it shipped before.
if _, err := old.Exec(`CREATE TABLE bookmarks (
key TEXT PRIMARY KEY, site TEXT NOT NULL, series_id TEXT NOT NULL,
title TEXT, series_url TEXT, cover TEXT,
last_chapter TEXT, last_chapter_num REAL, last_chapter_url TEXT,
favorite INTEGER NOT NULL DEFAULT 0,
latest_chapter TEXT NOT NULL DEFAULT '', latest_chapter_num REAL,
updated_at INTEGER NOT NULL)`); err != nil {
t.Fatalf("create old table: %v", err)
}
if _, err := old.Exec(
`INSERT INTO bookmarks (key, site, series_id, series_url, updated_at)
VALUES ('asura:x', 'asura', 'x', 'https://asurascans.com/comics/x', 5)`); err != nil {
t.Fatalf("seed old row: %v", err)
}
if err := old.Close(); err != nil {
t.Fatalf("close: %v", err)
}
s, err := Open(path)
if err != nil {
t.Fatalf("OpenStore on pre-existing db: %v", err)
}
t.Cleanup(func() { s.Close() })
// The migrated row must default to 0 (never checked) and so be due.
if got := readLatestCheckedAt(t, s, "asura:x"); got != 0 {
t.Fatalf("migrated latest_checked_at = %d, want 0", got)
}
due, err := s.DueForLatestCheck(1000, 10)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 1 {
t.Fatalf("got %d due rows after migration, want 1", len(due))
}
}
func TestUpsertDefaultsStatusToReading(t *testing.T) { func TestUpsertDefaultsStatusToReading(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
stored, err := store.Upsert(Bookmark{ stored, err := store.Upsert(Bookmark{
@@ -366,6 +425,41 @@ func TestUpsertStatusChangeKeepsUpdatedAt(t *testing.T) {
} }
} }
// A database written before the column existed must gain it, with every
// pre-existing row landing in the reading bucket.
func TestMigrationAddsStatusToLegacyDatabase(t *testing.T) {
path := filepath.Join(t.TempDir(), "legacy.db")
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("open: %v", err)
}
if _, err := db.Exec(`
CREATE TABLE bookmarks (
key TEXT PRIMARY KEY, site TEXT NOT NULL, series_id TEXT NOT NULL,
title TEXT, series_url TEXT, cover TEXT,
last_chapter TEXT, last_chapter_num REAL, last_chapter_url TEXT,
updated_at INTEGER NOT NULL);
INSERT INTO bookmarks (key, site, series_id, updated_at)
VALUES ('asura:old', 'asura', 'old', 1)`); err != nil {
t.Fatalf("seed legacy: %v", err)
}
db.Close()
store, err := Open(path)
if err != nil {
t.Fatalf("OpenStore: %v", err)
}
defer store.Close()
b, ok, err := store.Get("asura:old")
if err != nil || !ok {
t.Fatalf("Get: ok=%v err=%v", ok, err)
}
if b.Status != StatusReading {
t.Fatalf("Status = %q, want %q", b.Status, StatusReading)
}
}
// Archiving is the reason to keep polling — the point is to come back to a // Archiving is the reason to keep polling — the point is to come back to a
// series that has moved on. A finished series has nothing left to publish. // series that has moved on. A finished series has nothing left to publish.
func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) { func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
@@ -376,7 +470,7 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
{"asura:finished", StatusFinished}, {"asura:finished", StatusFinished},
} { } {
if _, err := store.Upsert(Bookmark{ if _, err := store.Upsert(Bookmark{
Key: tc.key, Site: "asura", SeriesID: strings.TrimPrefix(tc.key, "asura:"), Key: tc.key, Site: "asura", SeriesID: tc.key,
SeriesURL: "https://asurascans.com/comics/" + tc.key, SeriesURL: "https://asurascans.com/comics/" + tc.key,
Status: tc.status, UpdatedAt: time.Now().UnixMilli(), Status: tc.status, UpdatedAt: time.Now().UnixMilli(),
}); err != nil { }); err != nil {
@@ -389,8 +483,8 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
got := map[string]bool{} got := map[string]bool{}
for _, sr := range due { for _, b := range due {
got[sr.Key()] = true got[b.Key] = true
} }
if !got["asura:reading"] || !got["asura:archived"] { if !got["asura:reading"] || !got["asura:archived"] {
t.Fatalf("due = %v, want reading and archived present", got) t.Fatalf("due = %v, want reading and archived present", got)
@@ -400,6 +494,135 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
} }
} }
// Asura slugs used to include the site build hash; rows keyed with it must
// be rewritten to the stable ID on open, merging hash-generations of the
// same series into the newest row.
func TestOpenStoreMigratesAsuraBuildHashKeys(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "hash.db")
store, err := Open(dbPath)
if err != nil {
t.Fatalf("open: %v", err)
}
seed := []Bookmark{
{Key: "asura:swordmasters-youngest-son-f886a8af", Site: "asura",
SeriesID: "swordmasters-youngest-son-f886a8af", Title: "Old gen",
LastChapterNum: 50, UpdatedAt: 100},
{Key: "asura:swordmasters-youngest-son-059befe1", Site: "asura",
SeriesID: "swordmasters-youngest-son-059befe1", Title: "Re-bookmarked",
LastChapterNum: 60, UpdatedAt: 200},
{Key: "asura:overgeared-059befe1", Site: "asura",
SeriesID: "overgeared-059befe1", Title: "Single gen", UpdatedAt: 150},
// Hash-like suffix on another site must be left alone.
{Key: "demonic:x-deadbeef", Site: "demonic",
SeriesID: "x-deadbeef", Title: "Not asura", UpdatedAt: 300},
}
for _, b := range seed {
if _, err := store.Upsert(b); err != nil {
t.Fatalf("seed %s: %v", b.Key, err)
}
}
if err := store.Close(); err != nil {
t.Fatalf("close: %v", err)
}
reopened, err := Open(dbPath)
if err != nil {
t.Fatalf("reopen: %v", err)
}
t.Cleanup(func() { reopened.Close() })
list, err := reopened.List()
if err != nil {
t.Fatalf("List: %v", err)
}
byKey := map[string]Bookmark{}
for _, b := range list {
byKey[b.Key] = b
}
if len(list) != 3 {
t.Fatalf("want 3 rows after merge, got %d: %+v", len(list), list)
}
merged, ok := byKey["asura:swordmasters-youngest-son"]
if !ok {
t.Fatalf("merged key missing: %+v", byKey)
}
// Newest row wins the merge.
if merged.Title != "Re-bookmarked" || merged.LastChapterNum != 60 || merged.UpdatedAt != 200 {
t.Fatalf("merge kept wrong row: %+v", merged)
}
if merged.SeriesID != "swordmasters-youngest-son" {
t.Fatalf("series_id not stripped: %q", merged.SeriesID)
}
if _, ok := byKey["asura:overgeared-059befe1"]; ok {
t.Fatal("single-generation hashed key not rewritten")
}
if _, ok := byKey["asura:overgeared"]; !ok {
t.Fatal("single-generation row missing under stripped key")
}
if _, ok := byKey["demonic:x-deadbeef"]; !ok {
t.Fatal("non-asura row touched")
}
// Idempotent: a third open changes nothing.
third, err := Open(dbPath)
if err != nil {
t.Fatalf("third open: %v", err)
}
third.Close()
}
// A hashed row and a pre-existing hashless row of the same series collide on
// the stripped key. The winner rewrite must happen only after the loser is
// gone, or the UPDATE hits a primary-key collision and OpenStore fails.
func TestOpenStoreMigratesAsuraHashlessCollision(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "collision.db")
store, err := Open(dbPath)
if err != nil {
t.Fatalf("open: %v", err)
}
seed := []Bookmark{
{Key: "asura:overgeared", Site: "asura", SeriesID: "overgeared",
Title: "Hashless", LastChapterNum: 10, UpdatedAt: 100},
{Key: "asura:overgeared-059befe1", Site: "asura",
SeriesID: "overgeared-059befe1", Title: "Hashed newer",
LastChapterNum: 20, UpdatedAt: 200},
}
for _, b := range seed {
if _, err := store.Upsert(b); err != nil {
t.Fatalf("seed %s: %v", b.Key, err)
}
}
if err := store.Close(); err != nil {
t.Fatalf("close: %v", err)
}
reopened, err := Open(dbPath)
if err != nil {
t.Fatalf("reopen: %v", err)
}
t.Cleanup(func() { reopened.Close() })
list, err := reopened.List()
if err != nil {
t.Fatalf("List: %v", err)
}
if len(list) != 1 {
t.Fatalf("want 1 row after merge, got %d: %+v", len(list), list)
}
merged := list[0]
if merged.Key != "asura:overgeared" {
t.Fatalf("merged key = %q, want asura:overgeared", merged.Key)
}
if merged.SeriesID != "overgeared" {
t.Fatalf("series_id = %q, want overgeared", merged.SeriesID)
}
if merged.Title != "Hashed newer" || merged.LastChapterNum != 20 || merged.UpdatedAt != 200 {
t.Fatalf("merge kept wrong row: %+v", merged)
}
}
func TestDisplayChapter(t *testing.T) { func TestDisplayChapter(t *testing.T) {
cases := []struct { cases := []struct {
name string name string
@@ -491,263 +714,50 @@ func TestUpsertEmptyKindKeepsStoredValue(t *testing.T) {
} }
} }
// The 0002 backfill must survive a database that already ran 0001 with real // A database created before this column exists must gain it, backfilled as
// rows: one series row per distinct (site, series_id) carrying the moved // manga, without losing anything.
// columns, and the bookmark keeping the rest. That is the upgrade path for func TestLegacyDatabaseGainsKindAsManga(t *testing.T) {
// every deployed database, so it is exercised rather than trusted. dbPath := filepath.Join(t.TempDir(), "legacy.db")
func TestMigration0002BackfillsExistingBookmarks(t *testing.T) {
url := pgtest.URL(t)
db, err := sql.Open("pgx", url)
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { db.Close() })
// Run only 0001, as a database created before this change would have. legacy, err := sql.Open("sqlite", dbPath)
// migrate() normally creates the version table first; do the same here.
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
version bigint PRIMARY KEY,
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
t.Fatalf("create version table: %v", err)
}
body, err := migrations.ReadFile("migrations/0001_bookmarks.sql")
if err != nil { if err != nil {
t.Fatalf("read 0001: %v", err) t.Fatalf("open legacy db: %v", err)
} }
if err := applyMigration(db, 1, string(body)); err != nil { if _, err := legacy.Exec(`
t.Fatalf("apply 0001: %v", err) CREATE TABLE bookmarks (
key TEXT PRIMARY KEY,
site TEXT NOT NULL,
series_id TEXT NOT NULL,
title TEXT,
series_url TEXT,
cover TEXT,
last_chapter TEXT,
last_chapter_num REAL,
last_chapter_url TEXT,
updated_at INTEGER NOT NULL
)`); err != nil {
t.Fatalf("create legacy schema: %v", err)
} }
if _, err := db.Exec(` if _, err := legacy.Exec(`
INSERT INTO bookmarks (key, site, series_id, title, series_url, cover, INSERT INTO bookmarks (key, site, series_id, title, updated_at)
last_chapter, last_chapter_num, last_chapter_url, favorite, latest_chapter, VALUES ('asura:legacy', 'asura', 'legacy', 'Legacy Series', 123)`); err != nil {
latest_chapter_num, latest_checked_at, status, kind, updated_at)
VALUES ('asura:solo', 'asura', 'solo', 'Solo Leveling',
'https://asurascans.com/comics/solo', 'https://asurascans.com/covers/solo.jpg',
'Chapter 10', 10, 'https://asurascans.com/comics/solo/ch/10', true,
'Chapter 11', 11, 123456, 'reading', 'manga', 1000)`); err != nil {
t.Fatalf("seed legacy row: %v", err) t.Fatalf("seed legacy row: %v", err)
} }
if err := legacy.Close(); err != nil {
// Bring it current: 0002 must backfill the series row, not lose data. t.Fatalf("close legacy db: %v", err)
if err := migrate(db); err != nil {
t.Fatalf("migrate: %v", err)
}
var (
title string
checked int64
fav bool
lastNum float64
)
if err := db.QueryRow(`SELECT title, latest_checked_at FROM series
WHERE site = 'asura' AND series_id = 'solo'`).Scan(&title, &checked); err != nil {
t.Fatalf("series row missing after migration: %v", err)
}
if title != "Solo Leveling" || checked != 123456 {
t.Fatalf("series = (%q, %d), want backfilled title and latest_checked_at", title, checked)
}
if err := db.QueryRow(`SELECT favorite, last_chapter_num FROM bookmarks
WHERE key = 'asura:solo'`).Scan(&fav, &lastNum); err != nil {
t.Fatalf("bookmark row missing after migration: %v", err)
}
if !fav || lastNum != 10 {
t.Fatalf("bookmark = (%v, %v), want favorite and progress kept", fav, lastNum)
} }
// The migrated database opens as a normal store. store, err := Open(dbPath)
st, err := Open(url)
if err != nil { if err != nil {
t.Fatalf("Open after migrate: %v", err) t.Fatalf("Open on legacy db: %v", err)
} }
defer st.Close() t.Cleanup(func() { store.Close() })
}
// readSeries reads the series row directly, for asserting on what Upsert list, err := store.List()
// actually stored rather than what the joined Bookmark reports.
func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
t.Helper()
sr, err := scanSeries(s.db.QueryRow(
`SELECT `+seriesColumns+`, 0 AS reader_count FROM series s
WHERE s.site = $1 AND s.series_id = $2`, site, seriesID).Scan)
if err != nil { if err != nil {
t.Fatalf("read series %s:%s: %v", site, seriesID, err) t.Fatalf("List: %v", err)
} }
return sr if len(list) != 1 || list[0].Kind != KindManga {
} t.Fatalf("legacy row should backfill as manga, got %+v", list)
// The first PUT for a series creates its row from the client's title, cover
// and URL — there is no other source for them (ADR-0003).
func TestUpsertCreatesSeriesFromClient(t *testing.T) {
store := newTestStore(t)
if _, err := store.Upsert(Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
Cover: "https://asurascans.com/covers/solo.jpg", Kind: KindManga,
UpdatedAt: 1000,
}); err != nil {
t.Fatalf("Upsert: %v", err)
}
sr := readSeries(t, store, "asura", "solo")
if sr.Title != "Solo Leveling" || sr.SeriesURL != "https://asurascans.com/comics/solo" ||
sr.Cover != "https://asurascans.com/covers/solo.jpg" {
t.Fatalf("series = %+v, want client title/url/cover stored", sr)
}
}
// A PUT naming an existing series must not overwrite its title, cover or URL:
// the row is shared, and those values are scraped page content (ADR-0003).
func TestUpsertExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
store := newTestStore(t)
base := Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
Cover: "https://asurascans.com/covers/solo.jpg", LastChapterNum: 10,
UpdatedAt: 1000,
}
if _, err := store.Upsert(base); err != nil {
t.Fatalf("seed: %v", err)
}
// Same series, hostile/compromised values, real progress advance.
base.Title = "Scraped Rename"
base.SeriesURL = "https://evil.example/solo"
base.Cover = "https://evil.example/solo.jpg"
base.LastChapterNum = 11
got, err := store.Upsert(base)
if err != nil {
t.Fatalf("Upsert: %v", err)
}
if got.Title != "Solo Leveling" || got.SeriesURL != "https://asurascans.com/comics/solo" ||
got.Cover != "https://asurascans.com/covers/solo.jpg" {
t.Fatalf("stored = %+v, want original title/url/cover kept", got)
}
if got.LastChapterNum != 11 {
t.Fatalf("LastChapterNum = %v, want 11 — progress in the same request must still land", got.LastChapterNum)
}
}
// Kind and latest-chapter are last-write-wins even on an existing series: the
// poller and the userscript both report the latest chapter, and kind is only
// known to whichever client created the row.
func TestUpsertExistingSeriesAcceptsKindAndLatest(t *testing.T) {
store := newTestStore(t)
base := Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", Kind: KindManga,
UpdatedAt: 1000,
}
if _, err := store.Upsert(base); err != nil {
t.Fatalf("seed: %v", err)
}
num := 12.0
base.Kind = KindNovel
base.LatestChapter = "Chapter 12"
base.LatestChapterNum = &num
got, err := store.Upsert(base)
if err != nil {
t.Fatalf("Upsert: %v", err)
}
if got.Kind != KindNovel || got.LatestChapter != "Chapter 12" ||
got.LatestChapterNum == nil || *got.LatestChapterNum != 12 {
t.Fatalf("stored = %+v, want kind and latest chapter updated", got)
}
}
// Deleting the last bookmark must leave the series row behind, so a later
// re-bookmark shows title and cover immediately instead of waiting for a poll.
func TestDeleteKeepsSeriesRow(t *testing.T) {
store := newTestStore(t)
if _, err := store.Upsert(Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", Cover: "https://asurascans.com/covers/solo.jpg",
UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := store.Delete("asura:solo"); err != nil {
t.Fatalf("Delete: %v", err)
}
sr := readSeries(t, store, "asura", "solo")
if sr.Title != "Solo Leveling" {
t.Fatalf("series = %+v, want it kept after the last bookmark is deleted", sr)
}
// Re-bookmark with nothing but progress: the stored title/cover come back.
stored, err := store.Upsert(Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
LastChapterNum: 5, UpdatedAt: 2000,
})
if err != nil {
t.Fatalf("re-upsert: %v", err)
}
if stored.Title != "Solo Leveling" || stored.Cover != "https://asurascans.com/covers/solo.jpg" {
t.Fatalf("re-bookmark = %+v, want title/cover from the surviving series row", stored)
}
}
// seedSecondReader inserts an extra bookmark on an existing series. Today the
// bookmark key is <site>:<series_id>, so two bookmarks can share a series only
// once keys stop being derived from the series identity (issue #22); the due
// queue's reader-count ordering must already be right for that world.
func seedSecondReader(t *testing.T, s *Store, key, site, seriesID string, updatedAt int64) {
t.Helper()
if _, err := s.Upsert(Bookmark{
Key: key, Site: site, SeriesID: seriesID, UpdatedAt: updatedAt,
}); err != nil {
t.Fatalf("seed second reader %q: %v", key, err)
}
}
// The whole point of the split: a shared series is due once, ordered ahead of
// single-reader series by how many bookmarks reference it.
func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) {
s := newTestStore(t)
// "pop" has two bookmarks but was checked most recently; "solo" has one and
// was checked long ago. Reader count must win over age.
seedForCheck(t, s, "asura:pop", "https://asurascans.com/comics/pop", 900)
seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100)
due, err := s.DueForLatestCheck(1000, 10)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 2 {
t.Fatalf("due = %d rows, want 2", len(due))
}
if due[0].Key() != "asura:pop" || due[1].Key() != "asura:solo" {
t.Fatalf("due order = %q, %q; want asura:pop (2 readers), asura:solo (1)",
due[0].Key(), due[1].Key())
}
}
// A series with no bookmarks must never appear in the due queue, and nothing
// in the store ever deletes it (see TestDeleteKeepsSeriesRow).
func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:kept", "https://asurascans.com/comics/kept", 0)
if _, err := s.db.Exec(`
INSERT INTO series (site, series_id, title, series_url, cover, kind,
latest_chapter, latest_chapter_num, latest_checked_at)
VALUES ('asura', 'orphan', 'Orphan', 'https://asurascans.com/comics/orphan',
'', 'manga', '', NULL, 0)`); err != nil {
t.Fatalf("seed orphan series: %v", err)
}
due, err := s.DueForLatestCheck(1000, 10)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 1 || due[0].Key() != "asura:kept" {
t.Fatalf("due = %+v, want only the bookmarked series", due)
}
var n int
if err := s.db.QueryRow(
`SELECT count(*) FROM series WHERE site = 'asura' AND series_id = 'orphan'`).Scan(&n); err != nil {
t.Fatalf("count orphan series: %v", err)
}
if n != 1 {
t.Fatalf("orphan series count = %d, want 1 (never deleted)", n)
} }
} }
+5 -11
View File
@@ -24,10 +24,8 @@ import (
type Config struct { type Config struct {
Token string Token string
AllowedOrigins []string AllowedOrigins []string
// DatabaseURL is the Postgres connection URL; required, no default, DBPath string
// because a wrong guess would silently start on an empty database. Port string
DatabaseURL string
Port string
// WebPassword gates the browser UI. Empty disables the web routes entirely. // WebPassword gates the browser UI. Empty disables the web routes entirely.
WebPassword string WebPassword string
// UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js. // UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js.
@@ -145,7 +143,7 @@ func loadLatestPoll() LatestPoll {
func loadConfig() Config { func loadConfig() Config {
c := Config{ c := Config{
Token: os.Getenv("API_TOKEN"), Token: os.Getenv("API_TOKEN"),
DatabaseURL: os.Getenv("DATABASE_URL"), DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
Port: envOr("PORT", "8080"), Port: envOr("PORT", "8080"),
WebPassword: os.Getenv("WEB_PASSWORD"), WebPassword: os.Getenv("WEB_PASSWORD"),
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"), UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
@@ -217,11 +215,8 @@ func main() {
if cfg.Token == "" { if cfg.Token == "" {
log.Fatal("API_TOKEN is required") log.Fatal("API_TOKEN is required")
} }
if cfg.DatabaseURL == "" {
log.Fatal("DATABASE_URL is required")
}
s, err := store.Open(cfg.DatabaseURL) s, err := store.Open(cfg.DBPath)
if err != nil { if err != nil {
log.Fatalf("open store: %v", err) log.Fatalf("open store: %v", err)
} }
@@ -241,8 +236,7 @@ func main() {
} }
go func() { go func() {
// The connection URL carries a password, so it stays out of the log. log.Printf("listening on :%s (db=%s, origins=%v)", cfg.Port, cfg.DBPath, cfg.AllowedOrigins)
log.Printf("listening on :%s (origins=%v)", cfg.Port, cfg.AllowedOrigins)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("serve: %v", err) log.Fatalf("serve: %v", err)
} }
+6 -1
View File
@@ -5,6 +5,7 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
"path/filepath"
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
@@ -27,7 +28,11 @@ func webConfig() Config {
// can seed rows and assert on what the handlers wrote back. // can seed rows and assert on what the handlers wrote back.
func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) { func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) {
t.Helper() t.Helper()
st := newTestStore(t) st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("store.Open: %v", err)
}
t.Cleanup(func() { st.Close() })
return newRouter(st, cfg), st return newRouter(st, cfg), st
} }
+4 -9
View File
@@ -23,18 +23,13 @@ services:
# isn't an IP or "localhost". # isn't an IP or "localhost".
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222} BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
depends_on: depends_on:
headless-shell: - headless-shell
condition: service_started # `networks:` here replaces the base file's list entirely, so both must be
postgres: # named: `proxy` for Traefik routing, `browser` (defined in the base file)
condition: service_healthy # to keep reaching headless-shell without putting it on `proxy` too.
# `networks:` here replaces the base file's list entirely, so all three must
# be named: `proxy` for Traefik routing, and `browser` / `db` (defined in
# the base file) to keep reaching headless-shell and Postgres without
# putting either on `proxy`.
networks: networks:
- proxy - proxy
- browser - browser
- db
labels: labels:
- "traefik.enable=true" - "traefik.enable=true"
- "traefik.docker.network=${PROXY_NETWORK:-proxy}" - "traefik.docker.network=${PROXY_NETWORK:-proxy}"
+4 -38
View File
@@ -16,9 +16,7 @@ services:
# API_TOKEN is required — compose refuses to start without it. # API_TOKEN is required — compose refuses to start without it.
API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env} API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env}
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net} ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
# The bookmarks database. Host is the compose service name; the password DB_PATH: /data/bookmarks.db
# comes from .env so it is never committed.
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
PORT: "8080" PORT: "8080"
# Gates the browser UI. Unset means the web routes are not served at all. # Gates the browser UI. Unset means the web routes are not served at all.
WEB_PASSWORD: ${WEB_PASSWORD:-} WEB_PASSWORD: ${WEB_PASSWORD:-}
@@ -44,13 +42,9 @@ services:
# this URL survives container recreation. # this URL survives container recreation.
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222} BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
depends_on: depends_on:
headless-shell: - headless-shell
condition: service_started
# The migration runner is the first thing the binary does, so a Postgres
# that is still initialising means a crash-loop until it is not.
postgres:
condition: service_healthy
volumes: volumes:
- bookmarks-data:/data
# The userscript is served from here, read fresh on every request. Editing # The userscript is served from here, read fresh on every request. Editing
# the file in this checkout takes effect on the next Violentmonkey poll — # the file in this checkout takes effect on the next Violentmonkey poll —
# no rebuild, no restart. `git pull` restores the committed version, which # no rebuild, no restart. `git pull` restores the committed version, which
@@ -62,26 +56,6 @@ services:
- "127.0.0.1:8080:8080" - "127.0.0.1:8080:8080"
networks: networks:
- browser - browser
- db
postgres:
image: postgres:17-alpine
restart: unless-stopped
environment:
POSTGRES_DB: bookmarks
POSTGRES_USER: bookmarks
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"]
interval: 5s
timeout: 3s
retries: 10
volumes:
- postgres-data:/var/lib/postgresql/data
# Deliberately no `ports:` — only bookmark-api, over the `db` network,
# reaches it. Use `docker compose exec postgres psql` for a shell.
networks:
- db
headless-shell: headless-shell:
image: chromedp/headless-shell:stable image: chromedp/headless-shell:stable
@@ -111,11 +85,7 @@ services:
ipv4_address: 172.28.0.10 ipv4_address: 172.28.0.10
volumes: volumes:
postgres-data: bookmarks-data:
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
# declared here: undeclared means `docker compose down -v` cannot take it
# with the rest, so the old database survives the cutover until someone
# removes it by hand.
networks: networks:
# Not `internal: true`: headless Chrome still needs outbound access to reach # Not `internal: true`: headless Chrome still needs outbound access to reach
@@ -125,7 +95,3 @@ networks:
ipam: ipam:
config: config:
- subnet: 172.28.0.0/24 - subnet: 172.28.0.0/24
# Postgres needs no egress and nothing outside bookmark-api needs to reach
# it, so this one really can be cut off from the outside world.
db:
internal: true
-40
View File
@@ -1,40 +0,0 @@
# Postgres replaces SQLite as the primary datastore
Status: accepted
The project is moving from a single-reader tracker to a service published to a community,
so we replaced `modernc.org/sqlite` with Postgres (`jackc/pgx/v5`, still pure Go, so
`CGO_ENABLED=0` and the distroless image are unaffected). The deciding reason is future
supportability — managed hosting, a datastore that survives the app outgrowing one box —
**not** concurrency, which was measured and found to be a non-issue.
## Considered options
**Stay on SQLite.** Benchmarked against the real store at 1,500 rows (≈50 readers × 30
series): ~9,700 upserts/sec single-writer, plateauing at ~780/sec under 8–50 concurrent
writers, with `List()` holding 90–103 calls/sec under continuous write load. Projected
real load at 50 readers is ~0.02 writes/sec — roughly four and a half orders of magnitude
of headroom. `SetMaxOpenConns(1)` serialises writes but was shown not to starve reads;
an apparent read collapse traced to row count and per-row scanning, not lock contention.
SQLite would have worked. It was rejected for where the project is going, not for what
it does today.
**Postgres.** Chosen. Migrating is cheapest now — 29 rows in one table — and gets
materially harder once there are live readers and a multi-tenant schema.
## Consequences
- Every statement in `internal/store` is rewritten: `?` → `$N`, `IS NOT` →
`IS DISTINCT FROM` (this one is load-bearing; it implements the `updated_at`
ordering rule), `pragma_table_info` → `information_schema.columns`,
`INTEGER`/`REAL` → `bigint`/`double precision`, `favorite` int-as-bool → `boolean`.
- ~75 tests currently get a free isolated database from `t.TempDir()`. They now need a
live server, which makes Docker a hard prerequisite for `go test ./...`. This is the
permanent cost of the decision and the main reason it was close.
- Backups get worse, not better: `VACUUM INTO` produced one self-contained file;
restoring now means `pg_dump`/`pg_restore`, a role, and a password.
- A second stateful container joins the VPS alongside the existing headless-shell.
- **Postgres does not address the real scaling limit.** At batch 14 per 10-minute tick
the poller checks at most 84 series/hour; 50 readers × 30 series is 1,500 bookmarks,
an 18-hour sweep against a configured 1-hour cooldown. That ceiling is an outbound
fetch budget and is fixed by deduplicating polls per Series, not by the datastore.
@@ -1,44 +0,0 @@
# Identity comes from Discord OAuth; we store no passwords and send no email
Status: accepted
The service is being published to a community that already lives on Discord, and we have
no transactional email infrastructure. Rather than build email verification and password
reset to get accounts, Readers sign in with Discord OAuth2 (authorization code grant,
`identify` + `guilds.members.read`), and guild membership replaces both the invite gate
and the email-verification step. No password is ever stored and no mail is ever sent.
## Considered options
**Email + password with invite codes, no verification.** Viable and dependency-free:
an invite code proves community membership, which is what email verification was
standing in for anyway. Rejected because it still requires password hashing, a manual
admin-driven reset path, and a credential store — all of which Discord removes.
**Email + password with a transactional provider** (Resend, Brevo). Rejected as
premature: it builds verification and self-serve reset before anyone has asked for them,
and adds deliverability as an operational concern.
**Discord OAuth.** Chosen. It is less code than either alternative — no hashing, no
reset flow, no invite table — and the authorization question ("is this person in my
community?") is answered by the same call that answers the authentication question.
## Consequences
- **Availability is now coupled to Discord.** If Discord's OAuth endpoint is down,
nobody can start a new session. Existing sessions are unaffected, which bounds the
blast radius.
- **Identity is a Discord snowflake.** Migrating off Discord later means re-identifying
every Reader, because we hold no other credential for them. This is the lock-in the
decision buys, and it is the reason this ADR exists.
- **`guilds.members.read` is checked at login, not continuously.** Someone who leaves
the guild keeps their session until it expires. Acceptable; revocation is a session
delete, not an architectural change.
- **The userscripts cannot use OAuth.** They run in an isolated world on third-party
pages with no redirect surface, so they keep a bearer token — now issued per Reader by
the backend rather than a single shared `API_TOKEN` literal. OAuth gates the web UI;
the web UI is where a Reader obtains their personal userscript.
- `WEB_PASSWORD` disappears, and with it the session HMAC key derivation
(`sha256(API_TOKEN | WEB_PASSWORD | …)`), which needs a replacement secret.
- Seeding the first Reader during migration requires knowing the owner's Discord user
ID up front — a stable snowflake, copied from the Discord client.
@@ -1,44 +0,0 @@
# Series is a shared entity, and only the Poll may update it
Status: accepted
Facts about a Series that are true regardless of who is reading — title, cover, canonical
URL, Latest Chapter — moved off the Bookmark onto a shared `series` row keyed
`(site, series_id)`. A Bookmark now holds only what differs between Readers: Progress,
Favourite, Lifecycle bucket. Fifty Readers tracking one Series produce fifty Bookmarks
and one Series, so the Series is polled once rather than fifty times.
## Why
The poller checks at most 84 series/hour (batch 14 per 10-minute tick). With ~50 Readers
holding ~30 Series each, polling per Bookmark means a 1,500-item sweep — roughly 18 hours
against a configured 1-hour cooldown, quietly breaking the New Chapter signal that is the
product's reason to exist. Deduplicating to distinct Series cuts the sweep several-fold,
and because the Series row now knows how many Readers hold it, the poll queue is ordered
`reader_count DESC, latest_checked_at ASC` — popular Series stay fresh and the long tail
absorbs the shortfall. That ordering is only expressible because the split happened.
Raising throughput instead was rejected: sweeping 400 Series hourly needs the stagger
cut from 20s to ~9s, doubling request rate against sites that already bot-score the
single VPS IP.
## Only the Poll writes Series fields
A client may supply `title`, `cover` and `series_url` only when creating a Series nobody
has bookmarked yet. After that, client-supplied values are ignored; only the backend's
own fetch updates them.
This is a security boundary, not tidiness. Those values are scraped from third-party
pages, which `AGENTS.md` requires be treated as attacker-controlled. Before the split, a
hostile or compromised site could corrupt exactly one Reader's row. After it, the same
write lands on a row every Reader sees — one Reader's browser becomes a write path into
everyone else's UI, and a cover URL can point anywhere. The backend's own fetch is the
higher-trust source: its network, its parser, no third-party JavaScript in the path.
## Consequences
- `Store.Upsert` decomposes one incoming flat body across two tables and enforces the
ownership rule at that seam.
- The `updated_at` ordering rule stays on the Bookmark, where Progress lives. Unchanged.
- Per-Reader title overrides are deliberately not supported; they would reintroduce the
duplication this removes.
@@ -1,32 +0,0 @@
# The wire format stays flat and deliberately does not mirror the schema
Status: accepted
Storage splits a tracked series across two tables (ADR-0003), but `GET /bookmarks` and
`PUT /bookmarks/{key}` keep emitting and accepting one **flat** JSON object with `title`,
`cover`, `last_chapter` and `latest_chapter` as siblings — exactly the shape they had
when there was one table. The server joins on the way out and decomposes on the way in.
## Why a future reader will find this surprising
The obvious move after splitting a table is to nest the JSON to match. Don't "fix" this.
**A nested payload would have broken every installed userscript instantly.** Scripts read
`b.title` directly; moving it to `b.series.title` yields `undefined` — no error, just
blank rows and a New Chapter signal that silently reports nothing forever. Because
Violentmonkey updates roughly once a day per device, the migration relies on old scripts
continuing to work during a 14-day grace window. A nested format and that grace window
are mutually exclusive.
**It is also the better contract independently of compatibility.** A client rendering one
row needs the title and the reading position together; nesting exports the re-stitching
to every browser to mirror a decision about disk layout it should not know about. Keeping
them separate lets storage change again later without a client release — which is the
whole reason this ADR is worth the paragraph.
## Consequence
The flat shape is a contract, not an implementation detail. Changing the storage schema
must not change it. It follows the rule already in force for `updated_at`: the server
owns the truth and returns the row **as stored**, and clients adopt the response rather
than their own payload.
-47
View File
@@ -1,47 +0,0 @@
# Domain Docs
How the engineering skills should consume this repo's domain documentation when exploring the
codebase. Layout: **single-context** — one `CONTEXT.md` plus `docs/adr/` at the repo root.
## Before exploring, read these
- **`CONTEXT.md`** at the repo root — the glossary / ubiquitous language.
- **`docs/adr/`** — read ADRs that touch the area you're about to work in.
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest
creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and
`/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
Neither exists yet in this repo. The existing `AGENTS.md` / `CLAUDE.md` and `docs/design-system.md`
carry the current architecture and design law — read those regardless.
## File structure
```
/
├── CONTEXT.md
├── docs/adr/
│ ├── 0001-....md
│ └── 0002-....md
├── backend/
└── userscript/
```
If this repo ever splits into genuinely separate contexts, add a root `CONTEXT-MAP.md` pointing at
one `CONTEXT.md` per context and update this file.
## Use the glossary's vocabulary
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a
test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary
explicitly avoids.
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing
language the project doesn't use (reconsider) or there's a real gap (note it for
`/domain-modeling`).
## Flag ADR conflicts
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
> _Contradicts ADR-0002 (…) — but worth reopening because…_
-60
View File
@@ -1,60 +0,0 @@
# Issue tracker: Gitea (`tea` CLI)
Issues and specs for this repo live as issues on the self-hosted Gitea instance
`gitea.violetcrown.my.id` (repo `sulthan/mangaBookmark`). **`gh` does not work here** — use
[`tea`](https://gitea.com/gitea/tea) for everything past plain git. Auth lives in `tea login`,
not a `GH_TOKEN` env var. `tea` infers the repo from the local clone's `origin`.
`tea` prints rendered boxes rather than plain text; pass `--output json` (or `-o json`) when a
skill needs to parse the result.
## Conventions
- **Create an issue**: `tea issue create --title "..." --description "..."` (`--labels`,
`--assignees` optional). Multi-line bodies: pass the body through a shell variable or heredoc.
- **Read an issue**: `tea issue <number> --comments` (add `-o json` for machine-readable output).
- **List issues**: `tea issue list --state open -o json --fields index,title,body,labels,state,author`;
filter with `--labels "..."`, `--state open|closed|all`, `--assignee`, `--keyword`.
- **Comment**: `tea comment <number> "..."` (alias of `tea comments add`).
- **Apply / remove labels**: `tea issue edit <number> --add-labels "..."` / `--remove-labels "..."`.
Labels must exist first — see `tea labels list` / `tea labels create --name "..." --color "#rrggbb"`.
- **Close**: `tea issue close <number>` (comment separately with `tea comment`; `close` takes no
`--comment` flag).
## Pull requests as a triage surface
**PRs as a request surface: no.** _(Set to `yes` if this repo treats external PRs as feature
requests; `/triage` reads this flag.)_
When set to `yes`, PRs run through the same labels and states as issues, using the `tea pr`
equivalents: `tea pr <number> --comments`, `tea pr list --state open -o json`,
`tea pr create --head <branch> --base main --title "..." --description "..."`, `tea comment`,
`tea pr close`. Gitea shares one index space across issues and PRs, so a bare `#42` may be either
— resolve with `tea pr 42` and fall back to `tea issue 42`.
## When a skill says "publish to the issue tracker"
Create a Gitea issue with `tea issue create`.
## When a skill says "fetch the relevant ticket"
Run `tea issue <number> --comments`.
## Wayfinding operations
Used by `/wayfinder`. The **map** is a single issue; **tickets** are child issues.
- **Map**: one issue labelled `wayfinder:map` holding the Notes / Decisions-so-far / Fog body.
`tea issue create --labels wayfinder:map --title "..." --description "..."`.
- **Child ticket**: an issue labelled `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`)
with `Part of #<map>` as the first body line, and a task-list entry in the map body. `tea` has no
sub-issue command, so the task list plus the `Part of` line is the canonical link.
- **Blocking**: a `Blocked by: #<n>, #<n>` line at the top of the child body. Gitea's native issue
dependencies exist in the API but `tea` does not expose them; the body line is the source of
truth. A ticket is unblocked when every listed blocker is closed.
- **Frontier query**: `tea issue list --state open -o json` scoped to the map's task list; drop any
ticket with an open blocker or an assignee; first in map order wins.
- **Claim**: `tea issue edit <n> --add-assignees <your-username>` — the session's first write.
(`tea` has no `@me` shorthand; use the Gitea username from `tea login list`.)
- **Resolve**: `tea comment <n> "<answer>"`, then `tea issue close <n>`, then append a context
pointer to the map's Decisions-so-far via `tea issue edit <map> --description "..."`.
-20
View File
@@ -1,20 +0,0 @@
# Triage Labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual
label strings used in this repo's issue tracker (Gitea — see `docs/agents/issue-tracker.md`).
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label
string from this table.
Gitea will not auto-create labels on `tea issue edit --add-labels`; create a missing one first with
`tea labels create --name "<label>" --color "#rrggbb"`.
Edit the right-hand column to match whatever vocabulary you actually use.
-1
View File
@@ -1 +0,0 @@
AGENTS.md
+64
View File
@@ -0,0 +1,64 @@
Guidance for Claude Code working under `userscript/`. See root `CLAUDE.md` for the project-wide architecture diagram, hard constraints, and design system.
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
1. **Site adapters** — one per host, `detect(location, document)` return page `type` + IDs. Identify type/IDs from **URL regex** (most stable); pull `title`/`cover` from **`og:title`/`og:image` meta tags**, not CSS classes.
2. **API client** — `apiGet/apiPut/apiDelete` with bearer header; `localStorage` key `bmgr:manga:cache` for instant render + offline fallback.
3. **Progress logic** — auto-upsert `last_chapter` only when `chapterNum >= stored last_chapter_num` (re-reading old chapters must not regress progress; unparseable -> set current). Manual panel override forces any value.
4. **Retry queue** — every write go through `pushBookmark`/`pushDelete`, so
failed mutation park in `localStorage` (`bmgr:manga:queue`) and replayed on
next navigation, reconnect, or `refresh()`. Entries are markers
(`{key, op, sendStatus, attempts}`), never payloads — body read from
cache at send time, so one entry per key give ordering and coalescing for
free. `sendStatus` is **sticky**: while archive pending, later writes to
that key keep carrying bucket, which stop successful
in-between write from silently un-archiving series. `refresh()` drains
before it fetches and overlays anything still pending, so list never
flaps. 400 drops entry, 401 abort pass and keep queue, and
transient failures retry to cap of 10. Latest-chapter writes deliberately
stay out of queue. See
`docs/superpowers/specs/2026-07-27-offline-retry-queue-design.md`.
5. **UI** — rendered inside **Shadow DOM** root to isolate from site CSS
(critical on mobile). Three tabs (All / Favourites / Archived) and row of
link chips to web UI and both manga sites; `WEB_BASE` sits in CONFIG
block next to `API_BASE`. FAB is `7 × 44` edge tab whose *hit* area
widened to `28 × 72` by invisible `#hit` child; `#fab` must keep
`touch-action: none` and must **not** regain `overflow: hidden`. Since
`touch-action` resolved at gesture start, strip can't be both
browser-scrolled and script-dragged, so `makeDraggable` splits by intent: swipe
from `#hit` scrolls via `window.scrollBy`, hold of `ARM_MS` arms
reposition drag, visible sliver drags with no hold. See
`docs/superpowers/specs/2026-07-28-edge-tab-hitbox-design.md`.
6. **SPA navigation** — Asura is Astro, client-routed on comic/chapter pages: patch `history.pushState`/`replaceState` + listen `popstate`, re-run `detect()` on URL change so auto-update fire without reload. Demonic uses classic reloads (initial `document-idle` run suffice).
### Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust)
- **asurascans.com**: series `/comics/<slug>` (slug carries trailing
site-wide build-hash suffix, e.g. `-059befe1`, that **rotates on every
redeploy**), chapter `/comics/<slug>/chapter/<n>`. `seriesId` must strip
hash (`/-[0-9a-f]{8}$/`, `stripBuildHash` in userscript,
`asuraBuildHash` in backend); URLs keep full slug — stale-hash
URLs 302 to current ones. Astro-rendered; chapter links present in raw
server HTML.
- **demonicscans.org**: series `/manga/<slug>` (slug may URL-encode punctuation, e.g. `%2527` for `'`), chapter `/title/<slug>/chapter/<n>/<page>` (older `chaptered.php?manga=<id>&chapter=<n>` form still exists as redirect, what series-page chapter-list anchors link through).
Encodings (incl. triple-encoded punctuation like `%25252D`) identical
on /manga/ and /title/ pages, so decode-once seriesIds match — verified
2026-07-28.
- **novelfull.com** (novel script): series `/<slug>.html`, chapter
`/<slug>/chapter-<n>[-<title-slug>].html`. No `og:*` tags at all — title from
`h3.title` (series) or `a.truyen-title` (chapter), cover from
`meta[name="image"]`. Behind a Cloudflare JS challenge no TLS fingerprint
clears, so the backend polls it through the headless browser.
- **lightnovelworld.net** (novel script): series `/novel/<slug>/`, chapter
`/<slug>-chapter-<n>/` — flat, at the site root. `h1.entry-title` is the clean
title on a series page and `<Title> Chapter <n>` on a chapter page. Chapter
pages carry no `og:image`. Its series page lists every chapter with an
absolute href, so the backend polls it with the plain TLS client.
### Second script: `novel-bookmark.user.js`
A copy of the manga script with two adapters, `LIBRARY = "novel"` and
`STORE_PREFIX = "bmgr:novel:"`. No migration loop (this script has no previous
installation to carry keys over from). Installed alongside the manga script;
both write to the same backend with the same `LIBRARY` column discriminating
them.