Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a48aba67b8 | |||
| 08749df050 |
@@ -9,6 +9,14 @@ API_TOKEN=changeme-generate-a-long-random-token
|
|||||||
# sites' hostnames change.
|
# sites' hostnames change.
|
||||||
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net
|
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net
|
||||||
|
|
||||||
|
# Password for the bundled Postgres container, and therefore half of the
|
||||||
|
# DATABASE_URL compose builds for the backend. Generate one:
|
||||||
|
# openssl rand -hex 24
|
||||||
|
POSTGRES_PASSWORD=changeme-generate-a-long-random-password
|
||||||
|
|
||||||
|
# Override only to point the backend at a Postgres compose does not run.
|
||||||
|
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
||||||
|
|
||||||
# --- Prod override (Traefik) only ---
|
# --- Prod override (Traefik) only ---
|
||||||
# Subdomain Traefik routes to this service (required by the prod override).
|
# Subdomain Traefik routes to this service (required by the prod override).
|
||||||
# BOOKMARK_API_HOST=bookmark-api.example.com
|
# BOOKMARK_API_HOST=bookmark-api.example.com
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
|
|||||||
|
|
||||||
```
|
```
|
||||||
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache)
|
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache)
|
||||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
|
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
||||||
```
|
```
|
||||||
|
|
||||||
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
|
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
|
||||||
@@ -33,11 +33,11 @@ Backend-specific architecture (packages, endpoints, poller, config env vars) liv
|
|||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
Backend (`cd backend`):
|
Backend (`cd backend`):
|
||||||
- Test all: `go test ./...`
|
- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`).
|
||||||
- Single test: `go test -run TestName ./...`
|
- Single test: `go test -run TestName ./...`
|
||||||
- Build static binary: `CGO_ENABLED=0 go build`
|
- Build static binary: `CGO_ENABLED=0 go build`
|
||||||
|
|
||||||
Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`).
|
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
|
||||||
|
|
||||||
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
||||||
|
|
||||||
@@ -79,7 +79,7 @@ Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example
|
|||||||
|
|
||||||
Go backend:
|
Go backend:
|
||||||
|
|
||||||
- SQL always parameterized (`?`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
||||||
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
|
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
|
||||||
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
|
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
|
||||||
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
|
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
|
||||||
@@ -133,6 +133,22 @@ Test: "competent reader get this from code in few sec?" Yes → skip. Needs deto
|
|||||||
|
|
||||||
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
||||||
|
|
||||||
|
## Agent skills
|
||||||
|
|
||||||
|
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
||||||
|
|
||||||
|
### Issue tracker
|
||||||
|
|
||||||
|
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
|
||||||
|
|
||||||
|
### Triage labels
|
||||||
|
|
||||||
|
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
||||||
|
|
||||||
|
### Domain docs
|
||||||
|
|
||||||
|
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
|
||||||
|
|
||||||
## graphify
|
## graphify
|
||||||
|
|
||||||
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
||||||
|
|||||||
@@ -1,106 +0,0 @@
|
|||||||
# CLAUDE.md
|
|
||||||
|
|
||||||
Guidance for Claude Code (claude.ai/code) working in this repo.
|
|
||||||
|
|
||||||
## What this is
|
|
||||||
|
|
||||||
Manga read-progress tracker, user read on **asurascans.com** (current domain; asuracomic.net 301s here) and **demonicscans.org** via **Violentmonkey**. Userscript inject on-page UI (floating button + slide-in panel), sync progress to self-hosted Go backend so bookmarks unify across both sites and devices.
|
|
||||||
|
|
||||||
## Hard constraints (drive design — don't violate)
|
|
||||||
|
|
||||||
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
|
||||||
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
|
||||||
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
|
||||||
- Asura and Demonic are **separate origins with separate `localStorage`** — shared remote store only way to unify bookmarks. Cloud sync required, not optional.
|
|
||||||
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
|
||||||
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, or direct probe) before finalize, not assumed from single earlier test.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
|
|
||||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
|
|
||||||
```
|
|
||||||
|
|
||||||
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/CLAUDE.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/CLAUDE.md`.
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
Backend (`cd backend`):
|
|
||||||
- Test all: `go test ./...`
|
|
||||||
- Single test: `go test -run TestName ./...`
|
|
||||||
- Build static binary: `CGO_ENABLED=0 go build`
|
|
||||||
|
|
||||||
Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`).
|
|
||||||
|
|
||||||
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
|
||||||
|
|
||||||
## Forge: Gitea, not GitHub
|
|
||||||
|
|
||||||
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
|
|
||||||
|
|
||||||
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
|
|
||||||
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
|
|
||||||
- Issues: `tea issue create`, `tea issue list`
|
|
||||||
- Auth lives in `tea login`, not `GH_TOKEN` env var.
|
|
||||||
|
|
||||||
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
|
|
||||||
|
|
||||||
## Design system
|
|
||||||
|
|
||||||
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
|
|
||||||
— source of truth Claude Design project `BookmarkManager Web UI`
|
|
||||||
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
|
|
||||||
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
|
|
||||||
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
|
|
||||||
state (busy, error, destruction) may use `--ember`; destruction gets
|
|
||||||
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
|
|
||||||
only (never hardcode hex outside `:root`), both colour branches touched
|
|
||||||
together. Any move that pulls series out of list (archive/finish/remove)
|
|
||||||
must be confirm-gated via its own `.confirm-row`; only restore fires
|
|
||||||
instantly.
|
|
||||||
|
|
||||||
## Security invariants
|
|
||||||
|
|
||||||
- Auth on `/bookmarks*`: require `Authorization: Bearer <API_TOKEN>`, **constant-time compare**, 401 otherwise.
|
|
||||||
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
|
|
||||||
|
|
||||||
## Comments
|
|
||||||
|
|
||||||
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
|
||||||
|
|
||||||
Write for:
|
|
||||||
- Why not what. Tradeoffs, non-obvious decisions.
|
|
||||||
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
|
|
||||||
- Non-local consequence, invisible from function alone.
|
|
||||||
- Wire format / encoding / interface contract — save callers re-deriving.
|
|
||||||
- Gotcha/workaround, with ref if exists.
|
|
||||||
- Domain/business rule not derivable from code.
|
|
||||||
|
|
||||||
Skip:
|
|
||||||
- Restating code (no `// increment i` above `i++`).
|
|
||||||
- Trivial getter/setter/pass-through.
|
|
||||||
- Banners, dividers, `// helpers`.
|
|
||||||
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
|
||||||
- Commented-out code — delete.
|
|
||||||
- TODO without concrete action.
|
|
||||||
|
|
||||||
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
|
|
||||||
|
|
||||||
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
|
|
||||||
|
|
||||||
## Relevant skills
|
|
||||||
|
|
||||||
`multi-stage-dockerfile` and `docker-compose-orchestration` for container work (referenced in plan).
|
|
||||||
|
|
||||||
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
|
||||||
|
|
||||||
## graphify
|
|
||||||
|
|
||||||
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. Return scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
|
|
||||||
- If graphify-out/wiki/index.md exists, use for broad navigation instead of raw source browsing.
|
|
||||||
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain don't surface enough context.
|
|
||||||
- After modifying code, run `graphify update .` to keep graph current (AST-only, no API cost).
|
|
||||||
+69
@@ -0,0 +1,69 @@
|
|||||||
|
# Bookmark Manager
|
||||||
|
|
||||||
|
Read-progress tracker for serialised fiction. A reader browses third-party manga and
|
||||||
|
novel sites; userscripts capture where they got to and sync it to a self-hosted backend,
|
||||||
|
so progress survives across sites and devices.
|
||||||
|
|
||||||
|
## Language
|
||||||
|
|
||||||
|
**Series**:
|
||||||
|
One ongoing work — a manga or a novel — as published by a Site. Identified by its
|
||||||
|
stable slug on that Site, never by its title. A Series exists once and is shared by
|
||||||
|
every Reader who bookmarks it; it owns the facts that are true regardless of who is
|
||||||
|
reading — title, cover, Latest Chapter. A Reader cannot change them; they describe the
|
||||||
|
Series, not anyone's relationship to it.
|
||||||
|
_Avoid_: manga, title, book, comic
|
||||||
|
|
||||||
|
**Site**:
|
||||||
|
One third-party source a Series is published on. A Series on two Sites is two Series.
|
||||||
|
_Avoid_: source, host, provider, domain
|
||||||
|
|
||||||
|
**Reader**:
|
||||||
|
A person with their own Progress. Exactly one per set of credentials, so there is no
|
||||||
|
separate "account" concept to model — the credential belongs to the Reader.
|
||||||
|
_Avoid_: user, account, member, subscriber
|
||||||
|
|
||||||
|
**Bookmark**:
|
||||||
|
One Reader's tracked relationship with one Series, holding only what differs between
|
||||||
|
Readers: Progress, Favourite, Lifecycle bucket. Facts about the Series itself belong
|
||||||
|
to the Series, not here.
|
||||||
|
_Avoid_: entry, item, record, subscription
|
||||||
|
|
||||||
|
**Library**:
|
||||||
|
One of the two halves of the collection — manga or novel — selected by a Bookmark's
|
||||||
|
`kind`. The web UI and the userscripts each address exactly one Library at a time.
|
||||||
|
Not a per-person concept: "everything one person has bookmarked" is a different idea
|
||||||
|
and must not be called a Library.
|
||||||
|
_Avoid_: section, tab, category
|
||||||
|
|
||||||
|
**Progress**:
|
||||||
|
The furthest chapter a reader has actually read in a Series. Only a change in Progress
|
||||||
|
is real activity, so only Progress reorders the list.
|
||||||
|
_Avoid_: position, bookmark (the noun is taken), last read
|
||||||
|
|
||||||
|
**Latest Chapter**:
|
||||||
|
The newest chapter a Site has published for a Series, discovered without the reader
|
||||||
|
present. Distinct from Progress in every way that matters: it is a fact about the Site,
|
||||||
|
not about the reader, and it must never reorder the list.
|
||||||
|
_Avoid_: newest, current chapter, update
|
||||||
|
|
||||||
|
**Poll**:
|
||||||
|
The backend's own check of a Site for a Series's Latest Chapter, made without the
|
||||||
|
Reader present. Performed once per Series no matter how many Readers bookmarked it —
|
||||||
|
a Poll is work done on behalf of the Series, never on behalf of a Reader.
|
||||||
|
_Avoid_: scrape, refresh, check, sync
|
||||||
|
|
||||||
|
**New Chapter**:
|
||||||
|
The state where Latest Chapter is ahead of Progress. The single condition the ember
|
||||||
|
accent is permitted to signal.
|
||||||
|
_Avoid_: unread, update available
|
||||||
|
|
||||||
|
**Lifecycle bucket**:
|
||||||
|
Which of three mutually exclusive states a Bookmark sits in — reading, archived, or
|
||||||
|
finished. A Bookmark is in exactly one. Orthogonal to being a favourite.
|
||||||
|
_Avoid_: state, status (as a domain word), list
|
||||||
|
|
||||||
|
**Favourite**:
|
||||||
|
A reader's manual pin on a Bookmark. Orthogonal to the Lifecycle bucket, and never a
|
||||||
|
reason to reorder the list.
|
||||||
|
_Avoid_: starred, pinned, priority
|
||||||
@@ -38,6 +38,14 @@ API_TOKEN=<paste output of: openssl rand -hex 32>
|
|||||||
# CORS allowlist — leave as-is unless a site changes hostname.
|
# CORS allowlist — leave as-is unless a site changes hostname.
|
||||||
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to
|
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to
|
||||||
|
|
||||||
|
# Required — password for the bundled Postgres container. Compose builds the
|
||||||
|
# backend's DATABASE_URL out of it and has no fallback for either.
|
||||||
|
POSTGRES_PASSWORD=<paste output of: openssl rand -hex 24>
|
||||||
|
|
||||||
|
# Leave unset. Only set this to point the backend at a Postgres compose does
|
||||||
|
# not run; it then replaces the URL built from POSTGRES_PASSWORD above.
|
||||||
|
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
||||||
|
|
||||||
# Required for the Traefik override. Both have no fallback — compose refuses
|
# Required for the Traefik override. Both have no fallback — compose refuses
|
||||||
# to start without them. BOOKMARK_WEB_HOST is required even if you never set
|
# to start without them. BOOKMARK_WEB_HOST is required even if you never set
|
||||||
# WEB_PASSWORD; see 1b.
|
# WEB_PASSWORD; see 1b.
|
||||||
@@ -50,13 +58,20 @@ BOOKMARK_WEB_HOST=bookmark.violetcrown.my.id
|
|||||||
# TRAEFIK_CERTRESOLVER=le
|
# TRAEFIK_CERTRESOLVER=le
|
||||||
```
|
```
|
||||||
|
|
||||||
Generate + insert the token in one line:
|
Generate + insert the two secrets in three lines:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sed -i "s|^API_TOKEN=.*|API_TOKEN=$(openssl rand -hex 32)|" .env
|
sed -i "s|^API_TOKEN=.*|API_TOKEN=$(openssl rand -hex 32)|" .env
|
||||||
|
sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 24)|" .env
|
||||||
grep -E '^API_TOKEN=' .env # copy this — the userscript needs the same value
|
grep -E '^API_TOKEN=' .env # copy this — the userscript needs the same value
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`POSTGRES_PASSWORD` is read **only while the `postgres-data` volume is empty**,
|
||||||
|
which in practice means at first boot. Changing it afterwards changes the URL
|
||||||
|
the backend dials but not the password the database expects, and `bookmark-api`
|
||||||
|
crash-loops on `password authentication failed`. Set it before §2 and leave it
|
||||||
|
alone.
|
||||||
|
|
||||||
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
|
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
|
||||||
> names (check your Traefik static config — common alternatives: `https`,
|
> names (check your Traefik static config — common alternatives: `https`,
|
||||||
> `myresolver`, `cloudflare`). Wrong names = no certificate issued.
|
> `myresolver`, `cloudflare`). Wrong names = no certificate issued.
|
||||||
@@ -116,16 +131,21 @@ This merges the base file (build/image/env/volume) with the prod override
|
|||||||
(no host port, Traefik network + router labels). Always pass **both** `-f`
|
(no host port, Traefik network + router labels). Always pass **both** `-f`
|
||||||
flags — the prod file is not standalone.
|
flags — the prod file is not standalone.
|
||||||
|
|
||||||
Two services come up: `bookmark-api` (the backend) and `headless-shell`, a CDP
|
Three services come up: `bookmark-api` (the backend), `postgres` (its database,
|
||||||
sidecar the poller uses to fetch kagane (behind a Cloudflare JS challenge).
|
`postgres:17-alpine`), and `headless-shell`, a CDP sidecar the poller uses to
|
||||||
It has no published port — only `bookmark-api` can reach it, over
|
fetch kagane (behind a Cloudflare JS challenge). Neither of the latter two
|
||||||
`BROWSER_WS_URL`. Missing or unreachable, the poller just skips kagane and
|
publishes a port: `postgres` sits alone with `bookmark-api` on an
|
||||||
logs it; nothing else is affected.
|
`internal: true` network, and `headless-shell` is reachable only over
|
||||||
|
`BROWSER_WS_URL`. A missing headless-shell just makes the poller skip kagane and
|
||||||
|
log it. A missing Postgres stops everything — `bookmark-api` waits for
|
||||||
|
`pg_isready` to pass, then applies its embedded migrations, and only then
|
||||||
|
listens. The schema is created that way; there is nothing to import by hand.
|
||||||
|
|
||||||
Check it's up and healthy:
|
Check it's up and healthy:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
|
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
|
||||||
|
# bookmark-api Up; postgres Up (healthy)
|
||||||
docker logs bookmark-api --tail 20 # expect: "listening on :8080 ..."
|
docker logs bookmark-api --tail 20 # expect: "listening on :8080 ..."
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -214,7 +234,10 @@ Pull new code, then rebuild:
|
|||||||
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
|
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
SQLite data persists in the named volume `bookmarks-data` across rebuilds.
|
Data persists in the named volume `postgres-data` across rebuilds. (If this
|
||||||
|
server predates the Postgres migration, the old SQLite volume `bookmarks-data`
|
||||||
|
is still on disk and deliberately undeclared in compose so `down -v` cannot take
|
||||||
|
it; see `REDEPLOY.md` §1 for when to remove it.)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -227,7 +250,9 @@ SQLite data persists in the named volume `bookmarks-data` across rebuilds.
|
|||||||
| `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). |
|
| `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). |
|
||||||
| 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. |
|
| 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. |
|
||||||
| Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. |
|
| Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. |
|
||||||
| `compose ... config` errors about `API_TOKEN` | Run compose from the dir with `.env`, or export the vars. |
|
| `compose ... config` errors about `API_TOKEN` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. Both are required and neither has a fallback. |
|
||||||
|
| `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). |
|
||||||
|
| `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. |
|
||||||
|
|
||||||
Backend config reference and endpoint list: see `README.md`.
|
Backend config reference and endpoint list: see `README.md`.
|
||||||
|
|
||||||
|
|||||||
@@ -7,14 +7,14 @@ all four sites and all devices.
|
|||||||
|
|
||||||
Two parts:
|
Two parts:
|
||||||
|
|
||||||
- **`backend/`** — tiny Go (`net/http` + pure-Go SQLite) sync service. 4 routes,
|
- **`backend/`** — tiny Go (`net/http` + Postgres via pure-Go `pgx`) sync service. 4 routes,
|
||||||
static binary, distroless container.
|
static binary, distroless container.
|
||||||
- **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript
|
- **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript
|
||||||
(no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`.
|
(no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`.
|
||||||
|
|
||||||
```
|
```
|
||||||
Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|
Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|
||||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
|
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -27,7 +27,7 @@ Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|
|||||||
|-----|---------|-------|
|
|-----|---------|-------|
|
||||||
| `API_TOKEN` | *(required)* | Bearer token shared with the userscript. |
|
| `API_TOKEN` | *(required)* | Bearer token shared with the userscript. |
|
||||||
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
|
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
|
||||||
| `DB_PATH` | `/data/bookmarks.db` | SQLite file location. |
|
| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. |
|
||||||
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
|
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
|
||||||
| `BROWSER_WS_URL` | `ws://172.28.0.10:9222` | Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header. |
|
| `BROWSER_WS_URL` | `ws://172.28.0.10:9222` | Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header. |
|
||||||
|
|
||||||
@@ -60,11 +60,17 @@ go test ./... # unit + handler tests
|
|||||||
CGO_ENABLED=0 go build # static binary
|
CGO_ENABLED=0 go build # static binary
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**`go test ./...` requires Docker.** The store talks to a real Postgres, so
|
||||||
|
each test package starts a throwaway `postgres:17-alpine` container and gives
|
||||||
|
every test its own database inside it (`internal/pgtest`). Nothing is stubbed
|
||||||
|
and nothing reaches the network beyond the local Docker daemon.
|
||||||
|
|
||||||
### Run the stack
|
### Run the stack
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
# edit .env: set API_TOKEN (openssl rand -hex 32)
|
# edit .env: set API_TOKEN (openssl rand -hex 32) and
|
||||||
|
# POSTGRES_PASSWORD (openssl rand -hex 24)
|
||||||
|
|
||||||
docker compose up -d --build # binds 127.0.0.1:8080
|
docker compose up -d --build # binds 127.0.0.1:8080
|
||||||
```
|
```
|
||||||
|
|||||||
+136
-80
@@ -18,7 +18,7 @@ the checkout cannot take the backups with them.
|
|||||||
```
|
```
|
||||||
/opt/
|
/opt/
|
||||||
├── bookmarkmanager/ <- the checkout (this repo)
|
├── bookmarkmanager/ <- the checkout (this repo)
|
||||||
└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.db
|
└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.dump
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -53,78 +53,98 @@ echo "$BACKUP_DIR" # -> /opt/bookmarkmanager-backups
|
|||||||
|
|
||||||
## 1. Back up the database
|
## 1. Back up the database
|
||||||
|
|
||||||
The database is a single SQLite file in the named Docker volume, at
|
The database is Postgres, running as the `postgres` service on the named volume
|
||||||
`/data/bookmarks.db` inside the container. Find the volume's real name — Compose
|
`postgres-data`. It has **no published port** — nothing outside the internal `db`
|
||||||
prefixes it with the project directory:
|
network can reach it — so every command below goes in through the container:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker volume ls --filter name=bookmarks-data
|
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
|
||||||
# -> local bookmarkmanager_bookmarks-data
|
# -> bookmarks, schema_migrations, series
|
||||||
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Preferred: hot backup, no downtime
|
Inside the container that connects over the local socket as the `bookmarks`
|
||||||
|
superuser, so no password is needed anywhere in this section. `-T` is not
|
||||||
|
optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and
|
||||||
|
silently corrupts any binary stream flowing back out — see the dump below.
|
||||||
|
|
||||||
The store runs in **WAL mode**, so recent writes may still be sitting in
|
### Preferred: hot dump, no downtime
|
||||||
`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can
|
|
||||||
therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and
|
`pg_dump` runs in a single repeatable-read transaction, so it writes one
|
||||||
writes one consistent file, safe to run against a live database:
|
point-in-time-consistent snapshot while the API keeps serving. No stopping, no
|
||||||
|
WAL to worry about — that is the server's problem, not yours.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text
|
STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text
|
||||||
docker run --rm \
|
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
|
||||||
-v "$VOL":/data \
|
> "$BACKUP_DIR/bookmarks-$STAMP.dump"
|
||||||
-v "$BACKUP_DIR":/backup \
|
|
||||||
alpine sh -c "apk add -q sqlite &&
|
|
||||||
sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\""
|
|
||||||
|
|
||||||
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db
|
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump
|
||||||
```
|
```
|
||||||
|
|
||||||
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the
|
`-Fc` is the custom archive format rather than plain SQL: it is compressed, and
|
||||||
files sort in real order and never collide across a DST shift.
|
`pg_restore` can inspect and replay it selectively — list its table of contents,
|
||||||
|
restore one table, restore schema without data, reorder. A plain `.sql` dump can
|
||||||
|
only be piped into `psql` whole, and gives you no way to check what is in it
|
||||||
|
short of reading it.
|
||||||
|
|
||||||
Note the source volume is mounted **read-write**, which looks wrong for a backup
|
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.dump`. UTC, so
|
||||||
and is not. Opening a WAL database requires creating the `-shm` shared-memory
|
the files sort in real order and never collide across a DST shift.
|
||||||
file; with `:ro` the command fails with `unable to open database file` and no
|
|
||||||
backup is produced. `VACUUM INTO` never writes to the source itself.
|
|
||||||
|
|
||||||
Verify it before you trust it. An unreadable backup is worse than none, because
|
Verify it before you trust it. An unreadable backup is worse than none, because
|
||||||
you will act as though you have one:
|
you will act as though you have one:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite &&
|
# 1. The dump parses and contains the tables. Uses the same image compose
|
||||||
sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' &&
|
# already pulls, so nothing new to install.
|
||||||
sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'"
|
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
|
||||||
# -> ok
|
pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
|
||||||
|
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
|
||||||
|
# -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks
|
||||||
|
# -> 1236; 0 0 TABLE DATA public series series
|
||||||
|
|
||||||
|
# 2. Sanity-check the live row count you just captured.
|
||||||
|
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
|
||||||
|
-c 'select count(*) from bookmarks'
|
||||||
# -> 37
|
# -> 37
|
||||||
```
|
```
|
||||||
|
|
||||||
The count should match what the web UI shows. Zero rows on a server you know has
|
A custom-format archive stores row counts nowhere, so step 1 proves the file is
|
||||||
bookmarks means you backed up the wrong volume.
|
a readable archive with the right tables in it, not that the rows are there;
|
||||||
|
step 2 is the number those rows should be. It should match what the web UI
|
||||||
|
shows. Zero on a server you know has bookmarks means the API and your `psql`
|
||||||
|
are looking at different databases — check `DATABASE_URL`.
|
||||||
|
|
||||||
### Fallback: cold copy (no network for `apk add sqlite`)
|
### Fallback: cold volume archive
|
||||||
|
|
||||||
Stop the service first, then copy the database **and its sidecars** — the `-wal`
|
Use this when you want the whole data directory rather than a logical dump — a
|
||||||
is not optional, it is where the newest writes are:
|
like-for-like restore of the same Postgres major version onto the same host.
|
||||||
|
|
||||||
|
**The stack must be stopped first.** A running Postgres has dirty pages in
|
||||||
|
shared buffers and WAL that has not been replayed into the data files, and `tar`
|
||||||
|
walks the directory over several seconds while the server keeps writing to it.
|
||||||
|
The archive you get is torn: files from different instants, possibly a
|
||||||
|
half-written page. It may restore, start, and be quietly wrong. Online
|
||||||
|
filesystem-level backup is `pg_basebackup`'s job, not `tar`'s; with the
|
||||||
|
container stopped the shutdown checkpoint has already flushed everything and a
|
||||||
|
plain archive of the volume is consistent.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
VOL=$(docker volume ls --filter name=postgres-data -q | head -1)
|
||||||
|
echo "$VOL" # -> bookmarkmanager_postgres-data
|
||||||
|
|
||||||
$COMPOSE stop
|
$COMPOSE stop
|
||||||
docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c "
|
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \
|
||||||
cp /data/bookmarks.db /backup/bookmarks-$STAMP.db
|
tar czf "/to/postgres-data-$STAMP.tgz" -C /from .
|
||||||
[ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal
|
|
||||||
[ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm
|
|
||||||
ls -1 /backup"
|
|
||||||
$COMPOSE start
|
$COMPOSE start
|
||||||
|
|
||||||
|
ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz
|
||||||
```
|
```
|
||||||
|
|
||||||
Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away,
|
Costs ~15 seconds of downtime. Read-only on the source is safe here precisely
|
||||||
so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for
|
because nothing is running against it. Restoring this variant means untarring it
|
||||||
the case where it did not. Restoring this variant means putting whichever files
|
back into an *empty* `postgres-data` volume with the stack down — it is a whole
|
||||||
you got back together, under their original names.
|
data directory, not a file you can drop next to the live one, and it will only
|
||||||
|
start under `postgres:17`.
|
||||||
Read-only is safe here precisely because nothing opens the database: it is a file
|
|
||||||
copy, not a SQLite connection.
|
|
||||||
|
|
||||||
### Retention
|
### Retention
|
||||||
|
|
||||||
@@ -132,7 +152,19 @@ Keep a month, drop the rest — a bookmark database this small compresses the
|
|||||||
decision to "disk is free, but not infinite":
|
decision to "disk is free, but not infinite":
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v
|
ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v
|
||||||
|
```
|
||||||
|
|
||||||
|
### A note on the old `bookmarks-data` volume
|
||||||
|
|
||||||
|
`bookmarks-data` is the **pre-migration SQLite volume**. It is deliberately not
|
||||||
|
declared in `docker-compose.yml` any more, which is what keeps `docker compose
|
||||||
|
down -v` from taking it with the rest of the stack. It is not the live database
|
||||||
|
and nothing reads it. Once the Postgres data has been trusted for a while,
|
||||||
|
remove it by hand — nothing else will:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker volume rm bookmarkmanager_bookmarks-data
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -171,11 +203,15 @@ rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
|
|||||||
bindmounted read-only and read fresh per request.
|
bindmounted read-only and read fresh per request.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$COMPOSE ps # Up, and recently (re)created
|
$COMPOSE ps # bookmark-api Up; postgres Up (healthy)
|
||||||
docker logs bookmark-api --tail 20 # -> "listening on :8080 ..."
|
docker logs bookmark-api --tail 20 # -> "listening on :8080 ..."
|
||||||
```
|
```
|
||||||
|
|
||||||
Nothing in the log about the database or the poller failing. The image is tagged
|
Nothing in the log about the database, the migrations or the poller failing.
|
||||||
|
`bookmark-api` waits on `postgres` reporting healthy before it starts and the
|
||||||
|
binary applies any pending migration before it listens, so an API that never
|
||||||
|
says "listening" is usually the database, not the code — `$COMPOSE logs
|
||||||
|
postgres` first. The image is tagged
|
||||||
`bookmarkmanager-backend:latest`, so the previous image is still on disk untagged —
|
`bookmarkmanager-backend:latest`, so the previous image is still on disk untagged —
|
||||||
that is what makes the rollback in §6 quick.
|
that is what makes the rollback in §6 quick.
|
||||||
|
|
||||||
@@ -199,9 +235,16 @@ curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
|
|||||||
$API/bookmarks/x | grep -i access-control # -> allow-origin echoed
|
$API/bookmarks/x | grep -i access-control # -> allow-origin echoed
|
||||||
```
|
```
|
||||||
|
|
||||||
`[]` from the third call is the alarm that matters: the volume is not attached
|
`[]` from the third call is the alarm that matters: you are talking to an empty
|
||||||
and you are looking at an empty database. Stop and check `$COMPOSE config
|
database, which means the API found a *different* Postgres than the one holding
|
||||||
--volumes` before touching anything else.
|
your data — a renamed project directory, a fresh `postgres-data`, or a
|
||||||
|
`DATABASE_URL` override in `.env` pointing elsewhere. Stop and check, before
|
||||||
|
touching anything else:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$COMPOSE config --volumes # -> postgres-data
|
||||||
|
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks'
|
||||||
|
```
|
||||||
|
|
||||||
Web UI and its assets:
|
Web UI and its assets:
|
||||||
|
|
||||||
@@ -267,33 +310,41 @@ git checkout <previous-hash>
|
|||||||
$COMPOSE up -d --build
|
$COMPOSE up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
**Database damaged** — restore the backup from §1. Stop first: the running
|
**Database damaged** — restore the dump from §1. Stop **only the API**, not the
|
||||||
process holds the WAL, and dropping a file under a live SQLite connection
|
whole stack: `pg_restore` needs the server up to restore into, and it needs
|
||||||
corrupts what you were trying to save.
|
`bookmark-api`'s connection pool gone, because `--clean` cannot drop a table
|
||||||
|
other sessions are holding open.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$COMPOSE stop
|
$COMPOSE stop bookmark-api
|
||||||
|
|
||||||
docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c '
|
$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \
|
||||||
rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm &&
|
< "$BACKUP_DIR/bookmarks-<STAMP>.dump"
|
||||||
cp /backup/bookmarks-<STAMP>.db /data/bookmarks.db &&
|
|
||||||
chown 65532:65532 /data/bookmarks.db &&
|
|
||||||
ls -l /data'
|
|
||||||
|
|
||||||
$COMPOSE start
|
$COMPOSE start bookmark-api
|
||||||
docker logs bookmark-api --tail 20
|
docker logs bookmark-api --tail 20
|
||||||
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
|
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
|
||||||
```
|
```
|
||||||
|
|
||||||
Two steps here are easy to skip and both bite:
|
Three things here are easy to skip and all three bite:
|
||||||
|
|
||||||
- **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database
|
- **`--clean --if-exists`.** Without `--clean` the dump's rows land *on top of*
|
||||||
mixes two different histories; SQLite will either refuse to open it or quietly
|
what is already there and you get primary-key collisions half way through, a
|
||||||
reapply writes you meant to discard.
|
partially restored database, and a non-zero exit you may not notice.
|
||||||
- **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as
|
`--if-exists` only suppresses the "does not exist" noise when the target is
|
||||||
that uid, while the helper container above writes as root. A root-owned
|
already empty; it is not the part doing the work.
|
||||||
database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then
|
- **`-T` again.** Feeding a custom-format archive into a TTY-allocated `exec`
|
||||||
every write fails. That is the worst possible failure mode — it looks restored.
|
corrupts it in flight and `pg_restore` fails with a garbled-header error on a
|
||||||
|
file that is perfectly fine on disk.
|
||||||
|
- **Stop the API, not Postgres.** `$COMPOSE stop` (everything) leaves you with
|
||||||
|
nothing to restore into; leaving `bookmark-api` running leaves connections
|
||||||
|
that block the drops *and* lets the poller write into a half-restored table.
|
||||||
|
|
||||||
|
No ownership fixing is needed any more — the Postgres image owns `postgres-data`
|
||||||
|
itself and `pg_restore` writes through the server, not the filesystem.
|
||||||
|
`schema_migrations` is inside the dump, so the database comes back at whatever
|
||||||
|
schema version the backup was taken at; the migration runner applies anything
|
||||||
|
newer the next time `bookmark-api` starts.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -305,21 +356,24 @@ For a routine redeploy where nothing needs deciding:
|
|||||||
cd /opt/bookmarkmanager
|
cd /opt/bookmarkmanager
|
||||||
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
|
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
|
||||||
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
|
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
|
||||||
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
|
|
||||||
STAMP=$(date -u +%Y%m%d-%H%M%S)
|
STAMP=$(date -u +%Y%m%d-%H%M%S)
|
||||||
|
|
||||||
docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \
|
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
|
||||||
"apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" &&
|
> "$BACKUP_DIR/bookmarks-$STAMP.dump" &&
|
||||||
sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" &&
|
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
|
||||||
|
pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null &&
|
||||||
git pull --ff-only &&
|
git pull --ff-only &&
|
||||||
$COMPOSE up -d --build &&
|
$COMPOSE up -d --build &&
|
||||||
sleep 5 &&
|
sleep 5 &&
|
||||||
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"
|
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"
|
||||||
```
|
```
|
||||||
|
|
||||||
The `&&` chain is deliberate: if the backup or its integrity check fails,
|
The `&&` chain is deliberate: if the dump or its `pg_restore --list` check
|
||||||
nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell
|
fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a
|
||||||
command can tell you the panel works on the phone.
|
short or empty `.dump` behind — the shell creates the file before `pg_dump`
|
||||||
|
runs — so delete it rather than letting it sit in the backup directory looking
|
||||||
|
like a backup. Then still do §5 by hand — no shell command can tell you the
|
||||||
|
panel works on the phone.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -327,16 +381,18 @@ command can tell you the panel works on the phone.
|
|||||||
|
|
||||||
| Symptom | Cause / fix |
|
| Symptom | Cause / fix |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. |
|
| `/bookmarks` returns `[]` after redeploy | You are on an empty Postgres. Check `$COMPOSE config --volumes` lists `postgres-data`, that you passed both `-f` files, and that `.env` has no stray `DATABASE_URL` override. Do **not** re-bookmark; the data is still in the volume. |
|
||||||
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
|
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
|
||||||
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
|
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
|
||||||
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
|
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
|
||||||
| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
|
| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
|
||||||
| `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
|
| `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
|
||||||
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
|
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
|
||||||
| `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. |
|
| `bookmark-api` crash-loops, log says `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` in `.env` no longer matches the one burned into `postgres-data` at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: `$COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks'` (prompts, so nothing lands in shell history) and then match `.env` to it. |
|
||||||
| Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). |
|
| `compose` errors `set POSTGRES_PASSWORD in .env` | Unset. Compose builds the backend's `DATABASE_URL` out of it, so it is required even though you never write that URL yourself. Run from the directory holding `.env`. |
|
||||||
| Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). |
|
| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. |
|
||||||
|
| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. |
|
||||||
|
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
|
||||||
|
|
||||||
Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
|
Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
|
||||||
UI conventions: `docs/design-system.md`.
|
UI conventions: `docs/design-system.md`.
|
||||||
|
|||||||
+36
-14
@@ -1,8 +1,8 @@
|
|||||||
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
||||||
|
|
||||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
||||||
Single binary, split into packages under `backend/internal/`: `store`
|
Single binary, split into packages under `backend/internal/`: `store`
|
||||||
(Bookmark type, SQLite persistence, migrations), `latest` (background
|
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
||||||
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
||||||
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
||||||
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
||||||
@@ -11,7 +11,25 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
|||||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
packages together into `newRouter`. Root-level `*_test.go` hold
|
||||||
integration tests that exercise the full router; unit tests for a
|
integration tests that exercise the full router; unit tests for a
|
||||||
package live beside it under `internal/`.
|
package live beside it under `internal/`.
|
||||||
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`), with a `kind` column (`manga`|`novel`) splitting the two libraries. Sync **last-write-wins**. Schema and endpoint list in plan.
|
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
||||||
|
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
||||||
|
file per change, one transaction each, versions recorded in
|
||||||
|
`schema_migrations`. Files are **append-only** — editing an applied one
|
||||||
|
changes nothing on a database that already ran it. No column probing, no
|
||||||
|
data-fixup migrations: both were SQLite-era machinery and are gone.
|
||||||
|
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
||||||
|
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
||||||
|
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
||||||
|
must have that `TestMain`.
|
||||||
|
- **Single-user store, two tables.** `series` keyed `(site, series_id)`
|
||||||
|
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
||||||
|
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
||||||
|
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
|
||||||
|
differs between readers: progress, favourite, lifecycle bucket,
|
||||||
|
`updated_at`. Sync **last-write-wins**; the wire format stays flat
|
||||||
|
(ADR-0004). `Store.Upsert` decomposes one flat body across both tables and
|
||||||
|
enforces the ownership rule: client `title`/`series_url`/`cover` are
|
||||||
|
written only when the series row is new (ADR-0003).
|
||||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
||||||
- **Web UI:** same binary serve password-gated browser UI on second
|
- **Web UI:** same binary serve password-gated browser UI on second
|
||||||
hostname — `GET /` (list, or login page when no session),
|
hostname — `GET /` (list, or login page when no session),
|
||||||
@@ -43,22 +61,25 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
|||||||
network access, so `latest_chapter` stay fresh when user not
|
network access, so `latest_chapter` stay fresh when user not
|
||||||
browsing. Second, parallel signal — userscript keep own
|
browsing. Second, parallel signal — userscript keep own
|
||||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
||||||
Two independent clocks: per-bookmark cooldown (`latest_checked_at` column,
|
Two independent clocks: per-series cooldown (`series.latest_checked_at`,
|
||||||
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
||||||
Row stamped *before* fetch so broken series wait out full
|
The poller walks **Series, not Bookmarks** — a series referenced by several
|
||||||
cooldown instead of retrying every tick, and writes go through
|
bookmarks is fetched once per cycle, and the due queue orders
|
||||||
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
|
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
||||||
|
*before* fetch so broken series wait out full cooldown instead of retrying
|
||||||
|
every tick; found chapter written straight to the series row via
|
||||||
|
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
||||||
|
order — is never touched.
|
||||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||||
against fingerprint-based blocking; any failure log and skip. kagane and
|
against fingerprint-based blocking; any failure log and skip. kagane and
|
||||||
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
||||||
clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and
|
clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and
|
||||||
simply not polled when that's unset. See
|
simply not polled when that's unset. See
|
||||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||||
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
|
The poller's series write is a single-column UPDATE
|
||||||
userscript `PUT` that commits between the two can get overwritten by
|
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
||||||
poller's stale re-read — reverting that read progress and, since stored
|
it cannot revert read progress or move `updated_at`, so the old
|
||||||
value now differs, moving `updated_at` and reordering list. Known,
|
stale-re-read race is gone with the Get+Upsert flow.
|
||||||
accepted limitation for single-user deployment, not bug to fix.
|
|
||||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||||
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
||||||
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
||||||
@@ -70,8 +91,9 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
|||||||
`excluded.*` is post-evaluation row and default applied there would
|
`excluded.*` is post-evaluation row and default applied there would
|
||||||
wipe bucket on every PUT from client that predates column. See
|
wipe bucket on every PUT from client that predates column. See
|
||||||
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
||||||
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
|
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list),
|
||||||
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
|
`DATABASE_URL` (Postgres connection URL, required — no default),
|
||||||
|
`PORT` (default `8080`), `WEB_PASSWORD`
|
||||||
(gates browser UI; unset disable it),
|
(gates browser UI; unset disable it),
|
||||||
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
|
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
|
||||||
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
||||||
|
|||||||
@@ -1,81 +0,0 @@
|
|||||||
Guidance for Claude Code working under `backend/`. See root `CLAUDE.md` for the project-wide architecture diagram, hard constraints, and design system.
|
|
||||||
|
|
||||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
|
||||||
Single binary, split into packages under `backend/internal/`: `store`
|
|
||||||
(Bookmark type, SQLite persistence, migrations), `latest` (background
|
|
||||||
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
|
||||||
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
|
||||||
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
|
||||||
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
|
||||||
`backend/main.go` is the composition root — the only place that wires
|
|
||||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
|
||||||
integration tests that exercise the full router; unit tests for a
|
|
||||||
package live beside it under `internal/`.
|
|
||||||
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`). Sync **last-write-wins**. Schema and endpoint list in plan.
|
|
||||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
|
||||||
- **Web UI:** same binary serve password-gated browser UI on second
|
|
||||||
hostname — `GET /` (list, or login page when no session),
|
|
||||||
`POST /login`, `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
|
||||||
under `/ui/*`. Templates + assets `go:embed`-ed under
|
|
||||||
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
|
||||||
`internal/` tree, not just `*.go`. Sessions stateless
|
|
||||||
HMAC cookies keyed off `API_TOKEN`; `WEB_PASSWORD` gates them, and when empty,
|
|
||||||
web routes not registered at all. UI mutations read-modify-write
|
|
||||||
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
|
||||||
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
|
||||||
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
|
||||||
(correct — served from `/`), but impeccable detector resolves
|
|
||||||
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
|
||||||
on leading `/` and silently skip file. Relative href don't help
|
|
||||||
either: template's directory isn't its served path. So
|
|
||||||
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
|
||||||
always pass `backend/internal/web/static` too. One finding there,
|
|
||||||
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
|
||||||
- **Every action that moves series out of list is confirm-gated.**
|
|
||||||
Archive, finish, remove each open own `.confirm-row` disclosure
|
|
||||||
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
|
||||||
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
|
||||||
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
|
||||||
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
|
||||||
error use `--mute`.
|
|
||||||
- **Latest-chapter poller:** ticker goroutine in same binary re-check
|
|
||||||
each bookmarked series' newest published chapter from backend's own
|
|
||||||
network access, so `latest_chapter` stay fresh when user not
|
|
||||||
browsing. Second, parallel signal — userscript keep own
|
|
||||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
|
||||||
Two independent clocks: per-bookmark cooldown (`latest_checked_at` column,
|
|
||||||
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
|
||||||
Row stamped *before* fetch so broken series wait out full
|
|
||||||
cooldown instead of retrying every tick, and writes go through
|
|
||||||
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
|
|
||||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
|
||||||
against fingerprint-based blocking; any failure log and skip. kagane sits
|
|
||||||
behind a Cloudflare JavaScript challenge the TLS client can't clear, so it is
|
|
||||||
browser-only: fetched over CDP via `BROWSER_WS_URL`, and simply not polled
|
|
||||||
when that's unset. See
|
|
||||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
|
||||||
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
|
|
||||||
userscript `PUT` that commits between the two can get overwritten by
|
|
||||||
poller's stale re-read — reverting that read progress and, since stored
|
|
||||||
value now differs, moving `updated_at` and reordering list. Known,
|
|
||||||
accepted limitation for single-user deployment, not bug to fix.
|
|
||||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
|
||||||
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
|
||||||
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
|
||||||
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
|
||||||
checking archived series and skip finished ones. `finished` settable
|
|
||||||
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
|
||||||
**Empty incoming status means "keep stored one"** — resolved on the
|
|
||||||
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
|
||||||
`excluded.*` is post-evaluation row and default applied there would
|
|
||||||
wipe bucket on every PUT from client that predates column. See
|
|
||||||
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
|
||||||
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
|
|
||||||
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
|
|
||||||
(gates browser UI; unset disable it),
|
|
||||||
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
|
|
||||||
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
|
||||||
`USERSCRIPT_PATH` (file served at `/u/{token}/manga-bookmark.user.js`,
|
|
||||||
default `/userscript/manga-bookmark.user.js`, supplied by bindmount).
|
|
||||||
`BROWSER_WS_URL` (headless-shell CDP endpoint for kagane; unset disables
|
|
||||||
browser polling and leaves that site to the userscript alone).
|
|
||||||
Symlink
+1
@@ -0,0 +1 @@
|
|||||||
|
AGENTS.md
|
||||||
+3
-7
@@ -14,21 +14,17 @@ RUN go mod download
|
|||||||
COPY *.go ./
|
COPY *.go ./
|
||||||
COPY internal/ ./internal/
|
COPY internal/ ./internal/
|
||||||
|
|
||||||
# Static binary: pure-Go sqlite means CGO_ENABLED=0 -> no libc dependency.
|
# Static binary: the Postgres driver (jackc/pgx) is pure Go, so CGO_ENABLED=0
|
||||||
|
# leaves no libc dependency.
|
||||||
# -trimpath + -ldflags strip paths and debug info for a smaller image.
|
# -trimpath + -ldflags strip paths and debug info for a smaller image.
|
||||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server .
|
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server .
|
||||||
|
|
||||||
# Data dir with the runtime user's ownership so the mounted volume inherits it.
|
|
||||||
RUN mkdir -p /out/data
|
|
||||||
|
|
||||||
# --- runtime stage: distroless static, non-root ---
|
# --- runtime stage: distroless static, non-root ---
|
||||||
FROM gcr.io/distroless/static:nonroot
|
FROM gcr.io/distroless/static:nonroot
|
||||||
WORKDIR /
|
WORKDIR /
|
||||||
COPY --from=build /out/server /server
|
COPY --from=build /out/server /server
|
||||||
COPY --from=build --chown=65532:65532 /out/data /data
|
|
||||||
|
|
||||||
VOLUME ["/data"]
|
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
USER nonroot:nonroot
|
USER nonroot:nonroot
|
||||||
ENV DB_PATH=/data/bookmarks.db PORT=8080
|
ENV PORT=8080
|
||||||
ENTRYPOINT ["/server"]
|
ENTRYPOINT ["/server"]
|
||||||
|
|||||||
+146
-25
@@ -12,6 +12,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/pgtest"
|
||||||
"bookmarkmanager/backend/internal/store"
|
"bookmarkmanager/backend/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -25,15 +26,21 @@ func testConfig() Config {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||||
|
|
||||||
func newTestServer(t *testing.T) http.Handler {
|
func newTestServer(t *testing.T) http.Handler {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
dbPath := filepath.Join(t.TempDir(), "test.db")
|
return newRouter(newTestStore(t), testConfig())
|
||||||
s, err := store.Open(dbPath)
|
}
|
||||||
|
|
||||||
|
func newTestStore(t *testing.T) *store.Store {
|
||||||
|
t.Helper()
|
||||||
|
s, err := store.Open(pgtest.URL(t))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("store.Open: %v", err)
|
t.Fatalf("store.Open: %v", err)
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { s.Close() })
|
t.Cleanup(func() { s.Close() })
|
||||||
return newRouter(s, testConfig())
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
func auth(req *http.Request) *http.Request {
|
func auth(req *http.Request) *http.Request {
|
||||||
@@ -43,26 +50,35 @@ func auth(req *http.Request) *http.Request {
|
|||||||
|
|
||||||
func floatPtr(f float64) *float64 { return &f }
|
func floatPtr(f float64) *float64 { return &f }
|
||||||
|
|
||||||
// seedForCheck inserts a bookmark and forces its latest_checked_at.
|
// seedForCheck inserts a bookmark (and with it its series) and forces the
|
||||||
|
// series' latest_checked_at.
|
||||||
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
|
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
site, seriesID, ok := strings.Cut(key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", key)
|
||||||
|
}
|
||||||
if _, err := s.Upsert(store.Bookmark{
|
if _, err := s.Upsert(store.Bookmark{
|
||||||
Key: key,
|
Key: key,
|
||||||
Site: "asura",
|
Site: site,
|
||||||
SeriesID: key,
|
SeriesID: seriesID,
|
||||||
SeriesURL: seriesURL,
|
SeriesURL: seriesURL,
|
||||||
UpdatedAt: 1000,
|
UpdatedAt: 1000,
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("seed %q: %v", key, err)
|
t.Fatalf("seed %q: %v", key, err)
|
||||||
}
|
}
|
||||||
if err := s.MarkLatestChecked(key, checkedAt); err != nil {
|
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
|
||||||
t.Fatalf("seed mark %q: %v", key, err)
|
t.Fatalf("seed mark %q: %v", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
|
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
ts, err := s.LatestCheckedAt(key)
|
site, seriesID, ok := strings.Cut(key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", key)
|
||||||
|
}
|
||||||
|
ts, err := s.LatestCheckedAt(site, seriesID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LatestCheckedAt %q: %v", key, err)
|
t.Fatalf("LatestCheckedAt %q: %v", key, err)
|
||||||
}
|
}
|
||||||
@@ -222,6 +238,125 @@ func TestBookmarkRoundTrip(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The wire contract (ADR-0004): GET and PUT speak exactly the flat field set
|
||||||
|
// they always did, with the series-owned fields as siblings of the bookmark
|
||||||
|
// fields, not nested. Asserted as a key set, not by inspection.
|
||||||
|
func TestFlatWireFieldSet(t *testing.T) {
|
||||||
|
srv := newTestServer(t)
|
||||||
|
key := "comix:some-title"
|
||||||
|
in := store.Bookmark{
|
||||||
|
Key: key,
|
||||||
|
Site: "comix",
|
||||||
|
SeriesID: "some-title",
|
||||||
|
Title: "Some Title",
|
||||||
|
SeriesURL: "https://comix.to/title/some-title",
|
||||||
|
Cover: "https://comix.to/covers/some-title.jpg",
|
||||||
|
LastChapter: "Chapter 7",
|
||||||
|
LastChapterNum: 7,
|
||||||
|
LastChapterURL: "https://comix.to/title/some-title/ch/7",
|
||||||
|
Favorite: true,
|
||||||
|
LatestChapter: "Chapter 8",
|
||||||
|
LatestChapterNum: floatPtr(8),
|
||||||
|
Status: store.StatusArchived,
|
||||||
|
Kind: store.KindManga,
|
||||||
|
}
|
||||||
|
body, _ := json.Marshal(in)
|
||||||
|
|
||||||
|
wantKeys := map[string]bool{
|
||||||
|
"key": true, "site": true, "series_id": true, "title": true,
|
||||||
|
"series_url": true, "cover": true, "last_chapter": true,
|
||||||
|
"last_chapter_num": true, "last_chapter_url": true, "favorite": true,
|
||||||
|
"latest_chapter": true, "latest_chapter_num": true, "updated_at": true,
|
||||||
|
"status": true, "kind": true,
|
||||||
|
}
|
||||||
|
checkFlat := func(t *testing.T, payload []byte) map[string]json.RawMessage {
|
||||||
|
t.Helper()
|
||||||
|
var obj map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(payload, &obj); err != nil {
|
||||||
|
t.Fatalf("decode: %v", err)
|
||||||
|
}
|
||||||
|
if len(obj) != len(wantKeys) {
|
||||||
|
t.Fatalf("field count = %d, want %d (%s)", len(obj), len(wantKeys), payload)
|
||||||
|
}
|
||||||
|
for k := range obj {
|
||||||
|
if !wantKeys[k] {
|
||||||
|
t.Fatalf("unexpected field %q", k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return obj
|
||||||
|
}
|
||||||
|
|
||||||
|
// PUT
|
||||||
|
rr := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rr, auth(httptest.NewRequest(http.MethodPut, "/bookmarks/"+key, bytes.NewReader(body))))
|
||||||
|
if rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("PUT status = %d, want 200", rr.Code)
|
||||||
|
}
|
||||||
|
checkFlat(t, rr.Body.Bytes())
|
||||||
|
|
||||||
|
// Every field round-trips with its value, and updated_at is server-stamped.
|
||||||
|
var stored store.Bookmark
|
||||||
|
if err := json.Unmarshal(rr.Body.Bytes(), &stored); err != nil {
|
||||||
|
t.Fatalf("decode PUT response: %v", err)
|
||||||
|
}
|
||||||
|
latestNum := floatPtr(8)
|
||||||
|
want := store.Bookmark{
|
||||||
|
Key: key, Site: "comix", SeriesID: "some-title",
|
||||||
|
Title: in.Title, SeriesURL: in.SeriesURL, Cover: in.Cover,
|
||||||
|
LastChapter: in.LastChapter, LastChapterNum: in.LastChapterNum,
|
||||||
|
LastChapterURL: in.LastChapterURL, Favorite: true,
|
||||||
|
LatestChapter: in.LatestChapter, LatestChapterNum: latestNum,
|
||||||
|
Status: store.StatusArchived, Kind: store.KindManga,
|
||||||
|
}
|
||||||
|
if stored.Title != want.Title || stored.SeriesURL != want.SeriesURL || stored.Cover != want.Cover ||
|
||||||
|
stored.LastChapter != want.LastChapter || stored.LastChapterNum != want.LastChapterNum ||
|
||||||
|
stored.LastChapterURL != want.LastChapterURL || stored.Favorite != want.Favorite ||
|
||||||
|
stored.LatestChapter != want.LatestChapter ||
|
||||||
|
stored.LatestChapterNum == nil || *stored.LatestChapterNum != *want.LatestChapterNum ||
|
||||||
|
stored.Status != want.Status || stored.Kind != want.Kind {
|
||||||
|
t.Fatalf("PUT response = %+v, want %+v", stored, want)
|
||||||
|
}
|
||||||
|
if stored.UpdatedAt == 0 {
|
||||||
|
t.Fatal("updated_at not server-stamped")
|
||||||
|
}
|
||||||
|
|
||||||
|
// GET reports the same flat shape.
|
||||||
|
list := getBookmarks(t, srv)
|
||||||
|
if len(list) != 1 {
|
||||||
|
t.Fatalf("list = %d items, want 1", len(list))
|
||||||
|
}
|
||||||
|
body2, _ := json.Marshal(list[0])
|
||||||
|
checkFlat(t, body2)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A PUT naming an existing series must ignore client-supplied title, cover and
|
||||||
|
// URL — the security boundary from ADR-0003, where a hostile site's scraped
|
||||||
|
// values could otherwise land on a shared row — while progress still lands.
|
||||||
|
func TestPutExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
|
||||||
|
srv := newTestServer(t)
|
||||||
|
key := "asura:solo"
|
||||||
|
|
||||||
|
first := putBookmark(t, srv, key, store.Bookmark{
|
||||||
|
Title: "Solo Leveling",
|
||||||
|
SeriesURL: "https://asurascans.com/comics/solo",
|
||||||
|
Cover: "https://asurascans.com/covers/solo.jpg",
|
||||||
|
LastChapterNum: 10,
|
||||||
|
})
|
||||||
|
|
||||||
|
second := putBookmark(t, srv, key, store.Bookmark{
|
||||||
|
Title: "Scraped Rename",
|
||||||
|
SeriesURL: "https://evil.example/solo",
|
||||||
|
Cover: "https://evil.example/solo.jpg",
|
||||||
|
LastChapterNum: 11,
|
||||||
|
})
|
||||||
|
if second.Title != first.Title || second.SeriesURL != first.SeriesURL || second.Cover != first.Cover {
|
||||||
|
t.Fatalf("stored = %+v, want original title/url/cover kept", second)
|
||||||
|
}
|
||||||
|
if second.LastChapterNum != 11 {
|
||||||
|
t.Fatalf("LastChapterNum = %v, want 11 — progress must still land", second.LastChapterNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// putBookmark PUTs b at key and returns the bookmark the server echoes back,
|
// putBookmark PUTs b at key and returns the bookmark the server echoes back,
|
||||||
// which is the row as actually stored (not the request payload).
|
// which is the row as actually stored (not the request payload).
|
||||||
func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark {
|
func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark {
|
||||||
@@ -417,12 +552,7 @@ func TestLoadConfigWebPassword(t *testing.T) {
|
|||||||
// moved into bookmarkColumns, this test catches it: the PUT would reset the
|
// moved into bookmarkColumns, this test catches it: the PUT would reset the
|
||||||
// cooldown and the poller would re-fetch that series on every single tick.
|
// cooldown and the poller would re-fetch that series on every single tick.
|
||||||
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
|
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
|
||||||
dbPath := filepath.Join(t.TempDir(), "test.db")
|
s := newTestStore(t)
|
||||||
s, err := store.Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("store.Open: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { s.Close() })
|
|
||||||
srv := newRouter(s, testConfig())
|
srv := newRouter(s, testConfig())
|
||||||
|
|
||||||
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
|
||||||
@@ -454,12 +584,7 @@ func TestUserscriptServedWithWebUIDisabled(t *testing.T) {
|
|||||||
t.Fatalf("write script: %v", err)
|
t.Fatalf("write script: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
dbPath := filepath.Join(t.TempDir(), "nopass.db")
|
s := newTestStore(t)
|
||||||
s, err := store.Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("store.Open: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { s.Close() })
|
|
||||||
cfg := testConfig() // WebPassword empty
|
cfg := testConfig() // WebPassword empty
|
||||||
cfg.UserscriptPath = path
|
cfg.UserscriptPath = path
|
||||||
|
|
||||||
@@ -480,11 +605,7 @@ func TestNovelUserscriptServed(t *testing.T) {
|
|||||||
t.Fatalf("write script: %v", err)
|
t.Fatalf("write script: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
s, err := store.Open(filepath.Join(dir, "test.db"))
|
s := newTestStore(t)
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("store.Open: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { s.Close() })
|
|
||||||
|
|
||||||
cfg := testConfig()
|
cfg := testConfig()
|
||||||
cfg.NovelUserscriptPath = novelPath
|
cfg.NovelUserscriptPath = novelPath
|
||||||
|
|||||||
+5
-13
@@ -7,7 +7,7 @@ require (
|
|||||||
github.com/bogdanfinn/tls-client v1.15.1
|
github.com/bogdanfinn/tls-client v1.15.1
|
||||||
github.com/chromedp/cdproto v0.0.0-20260714215040-dc233986426f
|
github.com/chromedp/cdproto v0.0.0-20260714215040-dc233986426f
|
||||||
github.com/chromedp/chromedp v0.16.0
|
github.com/chromedp/chromedp v0.16.0
|
||||||
modernc.org/sqlite v1.34.4
|
github.com/jackc/pgx/v5 v5.10.0
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
@@ -18,27 +18,19 @@ require (
|
|||||||
github.com/bogdanfinn/utls v1.7.7-barnius // indirect
|
github.com/bogdanfinn/utls v1.7.7-barnius // indirect
|
||||||
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
|
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
|
||||||
github.com/chromedp/sysutil v1.1.0 // indirect
|
github.com/chromedp/sysutil v1.1.0 // indirect
|
||||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
|
||||||
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 // indirect
|
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 // indirect
|
||||||
github.com/gobwas/httphead v0.1.0 // indirect
|
github.com/gobwas/httphead v0.1.0 // indirect
|
||||||
github.com/gobwas/pool v0.2.1 // indirect
|
github.com/gobwas/pool v0.2.1 // indirect
|
||||||
github.com/gobwas/ws v1.4.0 // indirect
|
github.com/gobwas/ws v1.4.0 // indirect
|
||||||
github.com/google/uuid v1.6.0 // indirect
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||||
github.com/klauspost/compress v1.18.2 // indirect
|
github.com/klauspost/compress v1.18.2 // indirect
|
||||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
|
||||||
github.com/ncruces/go-strftime v0.1.9 // indirect
|
|
||||||
github.com/quic-go/qpack v0.6.0 // indirect
|
github.com/quic-go/qpack v0.6.0 // indirect
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
|
||||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
|
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
|
||||||
golang.org/x/crypto v0.46.0 // indirect
|
golang.org/x/crypto v0.46.0 // indirect
|
||||||
golang.org/x/net v0.48.0 // indirect
|
golang.org/x/net v0.48.0 // indirect
|
||||||
|
golang.org/x/sync v0.19.0 // indirect
|
||||||
golang.org/x/sys v0.47.0 // indirect
|
golang.org/x/sys v0.47.0 // indirect
|
||||||
golang.org/x/text v0.32.0 // indirect
|
golang.org/x/text v0.32.0 // indirect
|
||||||
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
|
|
||||||
modernc.org/libc v1.55.3 // indirect
|
|
||||||
modernc.org/mathutil v1.6.0 // indirect
|
|
||||||
modernc.org/memory v1.8.0 // indirect
|
|
||||||
modernc.org/strutil v1.2.0 // indirect
|
|
||||||
modernc.org/token v1.1.0 // indirect
|
|
||||||
)
|
)
|
||||||
|
|||||||
+14
-44
@@ -20,10 +20,9 @@ github.com/chromedp/chromedp v0.16.0 h1:rOO4deOm4CbZgBCa8mD9g2rDyIoNs0BkgvNrlbp5
|
|||||||
github.com/chromedp/chromedp v0.16.0/go.mod h1:rbuGKFT1vMcFcFqKfPIO1GpX/N+2s8onm2qMxZLbU5U=
|
github.com/chromedp/chromedp v0.16.0/go.mod h1:rbuGKFT1vMcFcFqKfPIO1GpX/N+2s8onm2qMxZLbU5U=
|
||||||
github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM=
|
github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM=
|
||||||
github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8=
|
github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8=
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
|
||||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
|
||||||
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 h1:KZaTBSyshWX3MP5jukJcNSuXDQTO+rNpt0J564dX/eg=
|
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 h1:KZaTBSyshWX3MP5jukJcNSuXDQTO+rNpt0J564dX/eg=
|
||||||
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg=
|
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg=
|
||||||
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
|
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
|
||||||
@@ -32,28 +31,27 @@ github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
|
|||||||
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
|
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
|
||||||
github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs=
|
github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs=
|
||||||
github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc=
|
github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc=
|
||||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
|
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
|
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
|
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
|
||||||
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
|
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
|
||||||
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo=
|
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo=
|
||||||
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
|
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
|
||||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
|
||||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
|
||||||
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
|
|
||||||
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
|
||||||
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw=
|
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw=
|
||||||
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
|
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
|
||||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
|
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
|
||||||
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
|
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
|
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
|
||||||
@@ -64,8 +62,6 @@ go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
|
|||||||
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
|
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
|
||||||
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
|
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
|
||||||
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
||||||
golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
|
|
||||||
golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
|
|
||||||
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||||
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
||||||
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
|
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
|
||||||
@@ -81,33 +77,7 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
|||||||
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
||||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
|
|
||||||
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
|
|
||||||
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
|
|
||||||
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
|
|
||||||
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
|
|
||||||
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
|
|
||||||
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
|
|
||||||
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
|
|
||||||
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 h1:5D53IMaUuA5InSeMu9eJtlQXS2NxAhyWQvkKEgXZhHI=
|
|
||||||
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6/go.mod h1:Qz0X07sNOR1jWYCrJMEnbW/X55x206Q7Vt4mz6/wHp4=
|
|
||||||
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
|
|
||||||
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
|
|
||||||
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
|
|
||||||
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
|
|
||||||
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
|
|
||||||
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
|
|
||||||
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
|
|
||||||
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
|
|
||||||
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
|
|
||||||
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
|
|
||||||
modernc.org/sqlite v1.34.4 h1:sjdARozcL5KJBvYQvLlZEmctRgW9xqIZc2ncN7PU0P8=
|
|
||||||
modernc.org/sqlite v1.34.4/go.mod h1:3QQFCG2SEMtc2nv+Wq4cQCH7Hjcg+p/RMlS1XK+zwbk=
|
|
||||||
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
|
|
||||||
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
|
|
||||||
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
|
|
||||||
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
|
||||||
|
|||||||
@@ -23,9 +23,9 @@ type Fetcher interface {
|
|||||||
// Two clocks, deliberately independent:
|
// Two clocks, deliberately independent:
|
||||||
//
|
//
|
||||||
// - Interval is how often this goroutine wakes up and looks.
|
// - Interval is how often this goroutine wakes up and looks.
|
||||||
// - Cooldown is how long one bookmark rests since its own last check.
|
// - Cooldown is how long one series rests since its own last check.
|
||||||
//
|
//
|
||||||
// Only the cooldown is per bookmark, and it is enforced by the WHERE clause in
|
// Only the cooldown is per series, and it is enforced by the WHERE clause in
|
||||||
// DueForLatestCheck rather than by any timer. Shortening Interval therefore
|
// DueForLatestCheck rather than by any timer. Shortening Interval therefore
|
||||||
// cannot shorten anyone's cooldown; it only makes the poller wake up and find
|
// cannot shorten anyone's cooldown; it only makes the poller wake up and find
|
||||||
// nothing due more often.
|
// nothing due more often.
|
||||||
@@ -78,7 +78,7 @@ func (p *Poller) Run(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// runOnce processes one batch of due bookmarks.
|
// runOnce processes one batch of due series.
|
||||||
func (p *Poller) runOnce(ctx context.Context) {
|
func (p *Poller) runOnce(ctx context.Context) {
|
||||||
cutoff := p.Now().Add(-p.Cooldown).UnixMilli()
|
cutoff := p.Now().Add(-p.Cooldown).UnixMilli()
|
||||||
due, err := p.Store.DueForLatestCheck(cutoff, p.Batch)
|
due, err := p.Store.DueForLatestCheck(cutoff, p.Batch)
|
||||||
@@ -88,7 +88,7 @@ func (p *Poller) runOnce(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
checked := 0
|
checked := 0
|
||||||
for i, b := range due {
|
for i, sr := range due {
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -107,7 +107,7 @@ func (p *Poller) runOnce(ctx context.Context) {
|
|||||||
if stopped {
|
if stopped {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
p.checkOne(ctx, b)
|
p.checkOne(ctx, sr)
|
||||||
checked++
|
checked++
|
||||||
}
|
}
|
||||||
// due vs checked is how you tell which constraint is binding: ticks that
|
// due vs checked is how you tell which constraint is binding: ticks that
|
||||||
@@ -119,10 +119,10 @@ func (p *Poller) runOnce(ctx context.Context) {
|
|||||||
// checkOne re-checks one series. Every failure path here is "log and move on":
|
// checkOne re-checks one series. Every failure path here is "log and move on":
|
||||||
// the poller is a best-effort enhancement, and no single bad series may stall a
|
// the poller is a best-effort enhancement, and no single bad series may stall a
|
||||||
// batch or take down the process.
|
// batch or take down the process.
|
||||||
func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
|
func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
|
||||||
defer func() {
|
defer func() {
|
||||||
if r := recover(); r != nil {
|
if r := recover(); r != nil {
|
||||||
log.Printf("latest poll %q: recovered from panic: %v", b.Key, r)
|
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
@@ -130,8 +130,8 @@ func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
|
|||||||
// mid-request still consumes the cooldown. Otherwise a renamed or deleted
|
// mid-request still consumes the cooldown. Otherwise a renamed or deleted
|
||||||
// series would be retried on every single tick forever. The userscript
|
// series would be retried on every single tick forever. The userscript
|
||||||
// stamps in the same order and for the same reason (L471-473).
|
// stamps in the same order and for the same reason (L471-473).
|
||||||
if err := p.Store.MarkLatestChecked(b.Key, p.Now().UnixMilli()); err != nil {
|
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
|
||||||
log.Printf("latest poll %q: mark checked: %v", b.Key, err)
|
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,69 +142,52 @@ func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
|
|||||||
// link-local/internal addresses or non-https schemes. The cooldown above
|
// link-local/internal addresses or non-https schemes. The cooldown above
|
||||||
// is already consumed, so a row that never passes this check is retried at
|
// is already consumed, so a row that never passes this check is retried at
|
||||||
// cooldown pace rather than hot-looping.
|
// cooldown pace rather than hot-looping.
|
||||||
if !fetchableSeriesURL(b.Site, b.SeriesURL) {
|
if !fetchableSeriesURL(sr.Site, sr.SeriesURL) {
|
||||||
log.Printf("latest poll %q: not fetchable: site=%q url=%q", b.Key, b.Site, b.SeriesURL)
|
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
f := p.fetcherFor(b.Site)
|
f := p.fetcherFor(sr.Site)
|
||||||
if f == nil {
|
if f == nil {
|
||||||
log.Printf("latest poll %q: no fetcher for site %q", b.Key, b.Site)
|
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
body, status, err := f.Get(ctx, b.SeriesURL)
|
body, status, err := f.Get(ctx, sr.SeriesURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("latest poll %q: fetch %s: %v", b.Key, b.SeriesURL, err)
|
log.Printf("latest poll %q: fetch %s: %v", sr.Key(), sr.SeriesURL, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if status != 200 {
|
if status != 200 {
|
||||||
log.Printf("latest poll %q: fetch %s: status %d", b.Key, b.SeriesURL, status)
|
log.Printf("latest poll %q: fetch %s: status %d", sr.Key(), sr.SeriesURL, status)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
latest, ok := latestChapterFrom(b.Site, b.SeriesURL, body)
|
latest, ok := latestChapterFrom(sr.Site, sr.SeriesURL, body)
|
||||||
if !ok {
|
if !ok {
|
||||||
// Most likely a challenge page or a layout change. Either way the row is
|
// Most likely a challenge page or a layout change. Either way the row is
|
||||||
// already stamped, so this waits out a cooldown instead of hot-looping.
|
// already stamped, so this waits out a cooldown instead of hot-looping.
|
||||||
log.Printf("latest poll %q: no chapter links in %d bytes", b.Key, len(body))
|
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), len(body))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Re-read: the row may have been updated or deleted while the fetch was in
|
|
||||||
// flight, and writing b back wholesale would undo that.
|
|
||||||
//
|
|
||||||
// ponytail: non-transactional read-modify-write, wrap Get+Upsert in a tx if
|
|
||||||
// this ever runs for more than one user. A client PUT that commits between
|
|
||||||
// these two statements is lost to the stale re-read — reverting read
|
|
||||||
// progress or a status change, and moving updated_at because the stored
|
|
||||||
// value now differs. Accepted for a single-user deployment: the window is
|
|
||||||
// milliseconds and the loser is one poll cycle.
|
|
||||||
cur, found, err := p.Store.Get(b.Key)
|
|
||||||
if err != nil {
|
|
||||||
log.Printf("latest poll %q: reread: %v", b.Key, err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if !found {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
// Equality, not >, mirroring the userscript (L427): a site that retracts a
|
// Equality, not >, mirroring the userscript (L427): a site that retracts a
|
||||||
// chapter should correct the stored number downward.
|
// chapter should correct the stored number downward. The comparison is
|
||||||
if cur.LatestChapterNum != nil && *cur.LatestChapterNum == latest.Num {
|
// against the due-query snapshot; a concurrent write in between only costs
|
||||||
|
// one redundant UPDATE of the same absolute value, never a wrong one.
|
||||||
|
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == latest.Num {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
num := latest.Num
|
// Series-level write: the row is shared, so one update refreshes every
|
||||||
cur.LatestChapter = latest.Label
|
// bookmark joining to it, and the bookmark's updated_at is never touched —
|
||||||
cur.LatestChapterNum = &num
|
// a newly published chapter is not reading progress and must not reorder
|
||||||
// A candidate only. last_chapter_num is untouched, so the CASE in Upsert
|
// the list.
|
||||||
// keeps the stored updated_at and the bookmark list does not reorder.
|
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, latest.Label, latest.Num); err != nil {
|
||||||
cur.UpdatedAt = p.Now().UnixMilli()
|
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
|
||||||
if _, err := p.Store.Upsert(cur); err != nil {
|
|
||||||
log.Printf("latest poll %q: upsert: %v", b.Key, err)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
log.Printf("latest poll %q: latest is now %s", b.Key, latest.Label)
|
log.Printf("latest poll %q: latest is now %s", sr.Key(), latest.Label)
|
||||||
}
|
}
|
||||||
|
|
||||||
// fetchableSeriesURL reports whether site is a site latestChapterFrom knows how
|
// fetchableSeriesURL reports whether site is a site latestChapterFrom knows how
|
||||||
|
|||||||
@@ -3,18 +3,22 @@ package latest
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"path/filepath"
|
"os"
|
||||||
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/pgtest"
|
||||||
"bookmarkmanager/backend/internal/store"
|
"bookmarkmanager/backend/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
// newTestStore opens a fresh SQLite store in a temp dir.
|
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||||
|
|
||||||
|
// newTestStore opens a store on a Postgres database of this test's own.
|
||||||
func newTestStore(t *testing.T) *store.Store {
|
func newTestStore(t *testing.T) *store.Store {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
s, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
s, err := store.Open(pgtest.URL(t))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Open: %v", err)
|
t.Fatalf("Open: %v", err)
|
||||||
}
|
}
|
||||||
@@ -22,26 +26,35 @@ func newTestStore(t *testing.T) *store.Store {
|
|||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
// seedForCheck inserts a bookmark and forces its latest_checked_at.
|
// seedForCheck inserts a bookmark (and with it its series) and forces the
|
||||||
|
// series' latest_checked_at.
|
||||||
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
|
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
site, seriesID, ok := strings.Cut(key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", key)
|
||||||
|
}
|
||||||
if _, err := s.Upsert(store.Bookmark{
|
if _, err := s.Upsert(store.Bookmark{
|
||||||
Key: key,
|
Key: key,
|
||||||
Site: "asura",
|
Site: site,
|
||||||
SeriesID: key,
|
SeriesID: seriesID,
|
||||||
SeriesURL: seriesURL,
|
SeriesURL: seriesURL,
|
||||||
UpdatedAt: 1000,
|
UpdatedAt: 1000,
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("seed %q: %v", key, err)
|
t.Fatalf("seed %q: %v", key, err)
|
||||||
}
|
}
|
||||||
if err := s.MarkLatestChecked(key, checkedAt); err != nil {
|
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
|
||||||
t.Fatalf("seed mark %q: %v", key, err)
|
t.Fatalf("seed mark %q: %v", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
|
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
ts, err := s.LatestCheckedAt(key)
|
site, seriesID, ok := strings.Cut(key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", key)
|
||||||
|
}
|
||||||
|
ts, err := s.LatestCheckedAt(site, seriesID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LatestCheckedAt %q: %v", key, err)
|
t.Fatalf("LatestCheckedAt %q: %v", key, err)
|
||||||
}
|
}
|
||||||
@@ -214,6 +227,41 @@ func TestRunOnceRespectsBatchLimit(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The point of the split (ADR-0003): a series referenced by several bookmarks
|
||||||
|
// is fetched once per due cycle, not once per bookmark. Today the bookmark key
|
||||||
|
// is <site>:<series_id>, so the second bookmark only exists once keys stop
|
||||||
|
// being derived from the series identity (issue #22).
|
||||||
|
func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
// The slug must match the fixture's own anchors: asura's parser scopes
|
||||||
|
// chapter links to the stored slug.
|
||||||
|
const slug = "chronicles-of-the-demon-faction-f886a8af"
|
||||||
|
const url = "https://asurascans.com/comics/" + slug
|
||||||
|
seedForCheck(t, s, "asura:"+slug, url, 0)
|
||||||
|
if _, err := s.Upsert(store.Bookmark{
|
||||||
|
Key: "asura:" + slug + ":2", Site: "asura", SeriesID: slug, UpdatedAt: 2000,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("seed second reader: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||||
|
newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background())
|
||||||
|
|
||||||
|
if got := f.callCount(); got != 1 {
|
||||||
|
t.Fatalf("fetched shared series %d times, want 1", got)
|
||||||
|
}
|
||||||
|
// Both bookmarks join to the same updated series row.
|
||||||
|
for _, key := range []string{"asura:" + slug, "asura:" + slug + ":2"} {
|
||||||
|
b, ok, err := s.Get(key)
|
||||||
|
if err != nil || !ok {
|
||||||
|
t.Fatalf("Get %s: %v ok=%v", key, err, ok)
|
||||||
|
}
|
||||||
|
if b.LatestChapterNum == nil || *b.LatestChapterNum != 181 {
|
||||||
|
t.Fatalf("%s LatestChapterNum = %v, want 181", key, b.LatestChapterNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// One unreachable series must not abandon the rest of the batch.
|
// One unreachable series must not abandon the rest of the batch.
|
||||||
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
|
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
|
||||||
s := newTestStore(t)
|
s := newTestStore(t)
|
||||||
@@ -327,8 +375,8 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
|
|||||||
|
|
||||||
now := time.UnixMilli(4_000_000)
|
now := time.UnixMilli(4_000_000)
|
||||||
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||||
newTestPoller(t, s, f, now).checkOne(context.Background(), store.Bookmark{
|
newTestPoller(t, s, f, now).checkOne(context.Background(), store.Series{
|
||||||
Key: key, Site: tt.site, SeriesURL: tt.seriesURL,
|
Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL,
|
||||||
})
|
})
|
||||||
|
|
||||||
if got := f.callCount(); got != tt.wantCalls {
|
if got := f.callCount(); got != tt.wantCalls {
|
||||||
|
|||||||
@@ -5,8 +5,6 @@ import (
|
|||||||
"regexp"
|
"regexp"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"bookmarkmanager/backend/internal/store"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// latestChapter is the newest chapter a series page advertises.
|
// latestChapter is the newest chapter a series page advertises.
|
||||||
@@ -22,6 +20,12 @@ type latestChapter struct {
|
|||||||
// before using the slug to scope anything.
|
// before using the slug to scope anything.
|
||||||
var asuraSlugRe = regexp.MustCompile(`/comics/([^/?#]+)`)
|
var asuraSlugRe = regexp.MustCompile(`/comics/([^/?#]+)`)
|
||||||
|
|
||||||
|
// asuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura
|
||||||
|
// appends to every series slug. It rotates on each site redeploy, so it is
|
||||||
|
// never part of a stable series_id. Must stay in sync with stripBuildHash in
|
||||||
|
// userscript/manga-bookmark.user.js.
|
||||||
|
var asuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`)
|
||||||
|
|
||||||
// demonicChapterRe matches the pre-redirect anchors demonic series pages link
|
// demonicChapterRe matches the pre-redirect anchors demonic series pages link
|
||||||
// through. Both the raw "&" and the HTML-escaped "&" forms occur.
|
// through. Both the raw "&" and the HTML-escaped "&" forms occur.
|
||||||
var demonicChapterRe = regexp.MustCompile(`chaptered\.php\?manga=\d+&(?:amp;)?chapter=([0-9.]+)`)
|
var demonicChapterRe = regexp.MustCompile(`chaptered\.php\?manga=\d+&(?:amp;)?chapter=([0-9.]+)`)
|
||||||
@@ -74,9 +78,9 @@ func latestChapterFrom(site, seriesURL, body string) (latestChapter, bool) {
|
|||||||
}
|
}
|
||||||
// Stored URLs predating a redeploy may carry a stale build hash;
|
// Stored URLs predating a redeploy may carry a stale build hash;
|
||||||
// chapter hrefs in the fetched body carry the current one. Strip to
|
// chapter hrefs in the fetched body carry the current one. Strip to
|
||||||
// the stable ID (same rule as migrateAsuraKeys) and make the hash
|
// the stable ID and make the hash optional in the pattern, so scoping
|
||||||
// optional in the pattern, so scoping survives rotations.
|
// survives rotations.
|
||||||
slug := store.AsuraBuildHash.ReplaceAllString(m[1], "")
|
slug := asuraBuildHash.ReplaceAllString(m[1], "")
|
||||||
// Compiled per call rather than cached: this runs once per fetch, which
|
// Compiled per call rather than cached: this runs once per fetch, which
|
||||||
// is at most a few times a minute, and the slug varies per series.
|
// is at most a few times a minute, and the slug varies per series.
|
||||||
re = regexp.MustCompile(`/comics/` + regexp.QuoteMeta(slug) + `(?:-[0-9a-f]{8})?/chapter/([0-9.]+)`)
|
re = regexp.MustCompile(`/comics/` + regexp.QuoteMeta(slug) + `(?:-[0-9a-f]{8})?/chapter/([0-9.]+)`)
|
||||||
|
|||||||
@@ -0,0 +1,120 @@
|
|||||||
|
// Package pgtest runs the Postgres the test suite needs: one throwaway
|
||||||
|
// container per test binary, one fresh database per test. Docker is therefore
|
||||||
|
// a hard prerequisite for `go test ./...`.
|
||||||
|
//
|
||||||
|
// Rolled by hand rather than pulled in as a dependency — it is one `docker
|
||||||
|
// run`, one `docker port` and a ping loop, against a module list that is
|
||||||
|
// otherwise stdlib plus what the poller genuinely needs.
|
||||||
|
package pgtest
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
"os/exec"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
_ "github.com/jackc/pgx/v5/stdlib"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
image = "postgres:17-alpine"
|
||||||
|
readyLimit = 60 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
adminURL string
|
||||||
|
dbSeq atomic.Int64
|
||||||
|
)
|
||||||
|
|
||||||
|
// Main starts the container, runs the package's tests and tears the container
|
||||||
|
// down. Every test package that touches the store calls it from TestMain:
|
||||||
|
//
|
||||||
|
// func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||||
|
func Main(m *testing.M) int {
|
||||||
|
id, url, err := start()
|
||||||
|
if err != nil {
|
||||||
|
fmt.Println("pgtest:", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
defer exec.Command("docker", "rm", "-f", id).Run()
|
||||||
|
|
||||||
|
adminURL = url
|
||||||
|
return m.Run()
|
||||||
|
}
|
||||||
|
|
||||||
|
// URL creates a database of its own for t and returns a connection URL for it.
|
||||||
|
// Nothing drops it again: the container goes away wholesale when Main returns.
|
||||||
|
func URL(t testing.TB) string {
|
||||||
|
t.Helper()
|
||||||
|
if adminURL == "" {
|
||||||
|
t.Fatal("pgtest: no container; this package needs TestMain to call pgtest.Main")
|
||||||
|
}
|
||||||
|
// Generated, never derived from the test name, so it needs no quoting and
|
||||||
|
// cannot collide when tests run in parallel.
|
||||||
|
name := "test_" + strconv.FormatInt(dbSeq.Add(1), 10)
|
||||||
|
|
||||||
|
admin, err := sql.Open("pgx", adminURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("pgtest: open admin connection: %v", err)
|
||||||
|
}
|
||||||
|
defer admin.Close()
|
||||||
|
if _, err := admin.Exec(`CREATE DATABASE ` + name); err != nil {
|
||||||
|
t.Fatalf("pgtest: create database %s: %v", name, err)
|
||||||
|
}
|
||||||
|
return strings.Replace(adminURL, "/postgres?", "/"+name+"?", 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// start launches the container and waits for it to accept queries, returning
|
||||||
|
// its id and a connection URL for the default database.
|
||||||
|
func start() (id, url string, err error) {
|
||||||
|
out, err := exec.Command("docker", "run", "-d", "--rm",
|
||||||
|
"-e", "POSTGRES_PASSWORD=pgtest",
|
||||||
|
"-P", image,
|
||||||
|
// Durability buys nothing for a database that dies with the test
|
||||||
|
// binary, and turning it off is most of the container's start-up cost.
|
||||||
|
"-c", "fsync=off", "-c", "full_page_writes=off",
|
||||||
|
).Output()
|
||||||
|
if err != nil {
|
||||||
|
return "", "", fmt.Errorf("docker run %s: %w", image, err)
|
||||||
|
}
|
||||||
|
id = strings.TrimSpace(string(out))
|
||||||
|
|
||||||
|
port, err := exec.Command("docker", "port", id, "5432/tcp").Output()
|
||||||
|
if err != nil {
|
||||||
|
exec.Command("docker", "rm", "-f", id).Run()
|
||||||
|
return "", "", fmt.Errorf("docker port: %w", err)
|
||||||
|
}
|
||||||
|
// "0.0.0.0:32768" (and possibly a second, IPv6 line); the port is all we want.
|
||||||
|
first, _, _ := strings.Cut(strings.TrimSpace(string(port)), "\n")
|
||||||
|
url = fmt.Sprintf("postgres://postgres:pgtest@127.0.0.1:%s/postgres?sslmode=disable",
|
||||||
|
first[strings.LastIndex(first, ":")+1:])
|
||||||
|
|
||||||
|
if err := waitReady(url); err != nil {
|
||||||
|
exec.Command("docker", "rm", "-f", id).Run()
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
return id, url, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitReady(url string) error {
|
||||||
|
db, err := sql.Open("pgx", url)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(readyLimit)
|
||||||
|
for {
|
||||||
|
if err = db.Ping(); err == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
return fmt.Errorf("postgres not ready after %s: %w", readyLimit, err)
|
||||||
|
}
|
||||||
|
time.Sleep(200 * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
-- One row per tracked series, keyed "<site>:<series_id>".
|
||||||
|
--
|
||||||
|
-- Everything is NOT NULL with a default except latest_chapter_num, where NULL
|
||||||
|
-- is a distinct state: nothing has been captured yet, which is not the same as
|
||||||
|
-- chapter zero.
|
||||||
|
--
|
||||||
|
-- Timestamps are unix milliseconds as bigint, not timestamptz: the userscripts
|
||||||
|
-- send Date.now() over the wire and the ordering rule compares them directly.
|
||||||
|
CREATE TABLE bookmarks (
|
||||||
|
key text PRIMARY KEY,
|
||||||
|
site text NOT NULL,
|
||||||
|
series_id text NOT NULL,
|
||||||
|
title text NOT NULL DEFAULT '',
|
||||||
|
series_url text NOT NULL DEFAULT '',
|
||||||
|
cover text NOT NULL DEFAULT '',
|
||||||
|
last_chapter text NOT NULL DEFAULT '',
|
||||||
|
last_chapter_num double precision NOT NULL DEFAULT 0,
|
||||||
|
last_chapter_url text NOT NULL DEFAULT '',
|
||||||
|
favorite boolean NOT NULL DEFAULT false,
|
||||||
|
latest_chapter text NOT NULL DEFAULT '',
|
||||||
|
latest_chapter_num double precision,
|
||||||
|
-- When the server last polled this series, unix ms; 0 means never, and sorts
|
||||||
|
-- first so a new bookmark is picked up on the next tick with no special case.
|
||||||
|
latest_checked_at bigint NOT NULL DEFAULT 0,
|
||||||
|
status text NOT NULL DEFAULT 'reading',
|
||||||
|
kind text NOT NULL DEFAULT 'manga',
|
||||||
|
updated_at bigint NOT NULL
|
||||||
|
);
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
-- One row per distinct work, shared by every bookmark that tracks it
|
||||||
|
-- (ADR-0003). Keyed (site, series_id), the pair a bookmark key decomposes
|
||||||
|
-- into. title/series_url/cover are written once, at creation, and never
|
||||||
|
-- again: client-supplied values are ignored once the row exists and the
|
||||||
|
-- poller is the only party that may change them. kind and the latest-chapter
|
||||||
|
-- fields are last-write-wins like the bookmark's own fields.
|
||||||
|
CREATE TABLE series (
|
||||||
|
site text NOT NULL,
|
||||||
|
series_id text NOT NULL,
|
||||||
|
title text NOT NULL DEFAULT '',
|
||||||
|
series_url text NOT NULL DEFAULT '',
|
||||||
|
cover text NOT NULL DEFAULT '',
|
||||||
|
kind text NOT NULL DEFAULT 'manga',
|
||||||
|
latest_chapter text NOT NULL DEFAULT '',
|
||||||
|
latest_chapter_num double precision,
|
||||||
|
-- When the server last polled this series, unix ms; 0 means never, and sorts
|
||||||
|
-- first so a new bookmark is picked up on the next tick with no special case.
|
||||||
|
latest_checked_at bigint NOT NULL DEFAULT 0,
|
||||||
|
PRIMARY KEY (site, series_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Backfill from today's rows. The bookmark key's uniqueness makes
|
||||||
|
-- (site, series_id) unique in practice; DISTINCT is belt and braces.
|
||||||
|
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||||
|
latest_chapter, latest_chapter_num, latest_checked_at)
|
||||||
|
SELECT DISTINCT site, series_id, title, series_url, cover, kind,
|
||||||
|
latest_chapter, latest_chapter_num, latest_checked_at
|
||||||
|
FROM bookmarks;
|
||||||
|
|
||||||
|
-- The bookmark keeps only what differs between readers (ADR-0003): progress,
|
||||||
|
-- favourite, lifecycle bucket. The dropped columns now live on series.
|
||||||
|
ALTER TABLE bookmarks
|
||||||
|
DROP COLUMN title,
|
||||||
|
DROP COLUMN series_url,
|
||||||
|
DROP COLUMN cover,
|
||||||
|
DROP COLUMN kind,
|
||||||
|
DROP COLUMN latest_chapter,
|
||||||
|
DROP COLUMN latest_chapter_num,
|
||||||
|
DROP COLUMN latest_checked_at;
|
||||||
|
|
||||||
|
-- A bookmark may not point at a series that does not exist. No cascade: a
|
||||||
|
-- series outlives its last bookmark, and deleting one is not a store operation.
|
||||||
|
ALTER TABLE bookmarks
|
||||||
|
ADD CONSTRAINT bookmarks_series_fk
|
||||||
|
FOREIGN KEY (site, series_id) REFERENCES series (site, series_id);
|
||||||
+263
-248
@@ -2,19 +2,28 @@ package store
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"database/sql"
|
"database/sql"
|
||||||
|
"embed"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"path"
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
_ "modernc.org/sqlite"
|
_ "github.com/jackc/pgx/v5/stdlib"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Bookmark is one tracked series, keyed "<site>:<series_id>" across both sites.
|
// Bookmark is one tracked series, keyed "<site>:<series_id>" across both sites.
|
||||||
//
|
//
|
||||||
// LastChapter* is the user's read progress; LatestChapter* is the newest
|
// LastChapter* is the user's read progress; LatestChapter* is the newest
|
||||||
// chapter the site has published, captured opportunistically by the userscript.
|
// chapter the site has published, captured opportunistically by the userscript.
|
||||||
|
//
|
||||||
|
// Title, SeriesURL, Cover, Kind and LatestChapter* live on the shared Series
|
||||||
|
// row (ADR-0003) and are joined in on read; Bookmark carries only what differs
|
||||||
|
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
|
||||||
|
// format stays flat regardless — see ADR-0004.
|
||||||
type Bookmark struct {
|
type Bookmark struct {
|
||||||
Key string `json:"key"`
|
Key string `json:"key"`
|
||||||
Site string `json:"site"`
|
Site string `json:"site"`
|
||||||
@@ -38,6 +47,35 @@ type Bookmark struct {
|
|||||||
Kind string `json:"kind"`
|
Kind string `json:"kind"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Series is one distinct work, shared by every bookmark that tracks it. It is
|
||||||
|
// keyed (site, series_id) — the pair a bookmark key decomposes into — and
|
||||||
|
// exists once no matter how many bookmarks point at it (ADR-0003).
|
||||||
|
//
|
||||||
|
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
|
||||||
|
// existing Series has them ignored, and only the backend's own Poll may change
|
||||||
|
// them. Kind and the latest-chapter fields are last-write-wins like the
|
||||||
|
// bookmark's own fields. Never serialized: the wire format is the flat
|
||||||
|
// Bookmark (ADR-0004).
|
||||||
|
type Series struct {
|
||||||
|
Site string
|
||||||
|
SeriesID string
|
||||||
|
Title string
|
||||||
|
SeriesURL string
|
||||||
|
Cover string
|
||||||
|
Kind string
|
||||||
|
LatestChapter string
|
||||||
|
LatestChapterNum *float64 // nil until first captured
|
||||||
|
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
|
||||||
|
|
||||||
|
// readerCount is the number of bookmarks referencing this series, filled
|
||||||
|
// only by the due-queue query that orders on it.
|
||||||
|
readerCount int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
|
||||||
|
// used by the poller's logs and by tests asserting on the due queue.
|
||||||
|
func (s Series) Key() string { return s.Site + ":" + s.SeriesID }
|
||||||
|
|
||||||
// HasNewChapter reports whether the site has published past the read point.
|
// HasNewChapter reports whether the site has published past the read point.
|
||||||
// A nil LatestChapterNum means nothing has been captured yet, which is not the
|
// A nil LatestChapterNum means nothing has been captured yet, which is not the
|
||||||
// same as "nothing new".
|
// same as "nothing new".
|
||||||
@@ -114,236 +152,164 @@ const (
|
|||||||
StatusFinished = "finished"
|
StatusFinished = "finished"
|
||||||
)
|
)
|
||||||
|
|
||||||
const schema = `
|
//go:embed migrations/*.sql
|
||||||
CREATE TABLE IF NOT EXISTS bookmarks (
|
var migrations embed.FS
|
||||||
key TEXT PRIMARY KEY,
|
|
||||||
site TEXT NOT NULL,
|
|
||||||
series_id TEXT NOT NULL,
|
|
||||||
title TEXT,
|
|
||||||
series_url TEXT,
|
|
||||||
cover TEXT,
|
|
||||||
last_chapter TEXT,
|
|
||||||
last_chapter_num REAL,
|
|
||||||
last_chapter_url TEXT,
|
|
||||||
favorite INTEGER NOT NULL DEFAULT 0,
|
|
||||||
latest_chapter TEXT NOT NULL DEFAULT '',
|
|
||||||
latest_chapter_num REAL,
|
|
||||||
latest_checked_at INTEGER NOT NULL DEFAULT 0,
|
|
||||||
status TEXT NOT NULL DEFAULT 'reading',
|
|
||||||
kind TEXT NOT NULL DEFAULT 'manga',
|
|
||||||
updated_at INTEGER NOT NULL
|
|
||||||
);`
|
|
||||||
|
|
||||||
// The columns above that databases created before them will be missing.
|
// bookmarkColumns is the only value ever concatenated into query text. It is a
|
||||||
// SQLite has no ADD COLUMN IF NOT EXISTS, so each is added only when absent.
|
// compile-time constant; every request value is bound as a parameter. The
|
||||||
var addedColumns = []struct{ name, ddl string }{
|
// series-owned fields are joined in from the series table, in scanBookmark
|
||||||
{"favorite", `ALTER TABLE bookmarks ADD COLUMN favorite INTEGER NOT NULL DEFAULT 0`},
|
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
|
||||||
{"latest_chapter", `ALTER TABLE bookmarks ADD COLUMN latest_chapter TEXT NOT NULL DEFAULT ''`},
|
const bookmarkColumns = `b.key, b.site, b.series_id, s.title, s.series_url, s.cover,
|
||||||
{"latest_chapter_num", `ALTER TABLE bookmarks ADD COLUMN latest_chapter_num REAL`},
|
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
|
||||||
// When the server last looked at this series, unix ms; 0 means never, and
|
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
|
||||||
// sorts first so a new bookmark is picked up on the next tick with no
|
|
||||||
// special case. Deliberately NOT in bookmarkColumns — see MarkLatestChecked.
|
|
||||||
{"latest_checked_at", `ALTER TABLE bookmarks ADD COLUMN latest_checked_at INTEGER NOT NULL DEFAULT 0`},
|
|
||||||
// Lifecycle bucket. The DEFAULT backfills every pre-existing row as
|
|
||||||
// 'reading', so there is no separate migration step.
|
|
||||||
{"status", `ALTER TABLE bookmarks ADD COLUMN status TEXT NOT NULL DEFAULT 'reading'`},
|
|
||||||
// Library bucket. The DEFAULT backfills every pre-existing row as 'manga',
|
|
||||||
// which is what every row written before novels existed actually is.
|
|
||||||
{"kind", `ALTER TABLE bookmarks ADD COLUMN kind TEXT NOT NULL DEFAULT 'manga'`},
|
|
||||||
}
|
|
||||||
|
|
||||||
const bookmarkColumns = `key, site, series_id, title, series_url, cover,
|
// seriesColumns is the series row in scanSeries order, used by the poller's
|
||||||
last_chapter, last_chapter_num, last_chapter_url,
|
// due query. latest_checked_at lives only on series — see MarkLatestChecked
|
||||||
favorite, latest_chapter, latest_chapter_num, updated_at, status, kind`
|
// for why it stays off every client-visible write.
|
||||||
|
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||||
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
|
||||||
|
|
||||||
// Store is the SQLite-backed bookmark store.
|
// Store is the Postgres-backed bookmark store.
|
||||||
type Store struct {
|
type Store struct {
|
||||||
db *sql.DB
|
db *sql.DB
|
||||||
}
|
}
|
||||||
|
|
||||||
// OpenStore opens (or creates) the SQLite database at path and applies the schema.
|
// Open connects to Postgres at url — a libpq connection URL such as
|
||||||
func Open(path string) (*Store, error) {
|
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — and brings its
|
||||||
// busy_timeout guards against SQLITE_BUSY under the reverse proxy's
|
// schema up to date.
|
||||||
// concurrent requests; a single writer connection keeps writes serialized.
|
func Open(url string) (*Store, error) {
|
||||||
dsn := path
|
db, err := sql.Open("pgx", url)
|
||||||
if !strings.Contains(dsn, "?") {
|
|
||||||
dsn += "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
|
|
||||||
}
|
|
||||||
db, err := sql.Open("sqlite", dsn)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("open sqlite %q: %w", path, err)
|
return nil, fmt.Errorf("open postgres: %w", err)
|
||||||
}
|
}
|
||||||
db.SetMaxOpenConns(1)
|
if err := migrate(db); err != nil {
|
||||||
if _, err := db.Exec(schema); err != nil {
|
|
||||||
db.Close()
|
db.Close()
|
||||||
return nil, fmt.Errorf("apply schema: %w", err)
|
return nil, fmt.Errorf("migrate: %w", err)
|
||||||
}
|
|
||||||
if err := migrateColumns(db); err != nil {
|
|
||||||
db.Close()
|
|
||||||
return nil, fmt.Errorf("migrate schema: %w", err)
|
|
||||||
}
|
|
||||||
if err := migrateAsuraKeys(db); err != nil {
|
|
||||||
db.Close()
|
|
||||||
return nil, fmt.Errorf("migrate asura keys: %w", err)
|
|
||||||
}
|
}
|
||||||
return &Store{db: db}, nil
|
return &Store{db: db}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// migrateColumns brings a pre-existing bookmarks table up to the current
|
// migrate applies every embedded migration this database has not recorded, in
|
||||||
// schema. Safe to run on every start: columns already present are skipped.
|
// filename order, each in its own transaction. Files are named
|
||||||
func migrateColumns(db *sql.DB) error {
|
// "<version>_<name>.sql" and are append-only: editing an applied file changes
|
||||||
have, err := existingColumns(db, "bookmarks")
|
// nothing, because schema_migrations is how a database remembers what it ran.
|
||||||
|
// Runs on every start and is a no-op once current.
|
||||||
|
func migrate(db *sql.DB) error {
|
||||||
|
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
|
version bigint PRIMARY KEY,
|
||||||
|
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
|
||||||
|
return fmt.Errorf("create version table: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
names, err := fs.Glob(migrations, "migrations/*.sql")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
for _, c := range addedColumns {
|
slices.Sort(names)
|
||||||
if _, ok := have[c.name]; ok {
|
|
||||||
continue
|
for _, name := range names {
|
||||||
|
version, err := strconv.ParseInt(strings.SplitN(path.Base(name), "_", 2)[0], 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("migration %q: filename must start with a version number", name)
|
||||||
}
|
}
|
||||||
if _, err := db.Exec(c.ddl); err != nil {
|
body, err := migrations.ReadFile(name)
|
||||||
return fmt.Errorf("add column %q: %w", c.name, err)
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := applyMigration(db, version, string(body)); err != nil {
|
||||||
|
return fmt.Errorf("migration %q: %w", name, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// AsuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura
|
// applyMigration runs one migration and records its version in the same
|
||||||
// appends to every series slug. It rotates on each site redeploy, so it
|
// transaction, so an interrupted start leaves neither half behind.
|
||||||
// must not be part of series_id. Must stay in sync with stripBuildHash in
|
func applyMigration(db *sql.DB, version int64, body string) error {
|
||||||
// userscript/manga-bookmark.user.js.
|
tx, err := db.Begin()
|
||||||
var AsuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`)
|
|
||||||
|
|
||||||
// migrateAsuraKeys rewrites asura bookmarks whose series_id still carries
|
|
||||||
// the build hash to the stable, hashless ID. Rows keyed with a hash are
|
|
||||||
// orphaned on every Asura redeploy (old-hash URLs 302 to new-hash ones, so
|
|
||||||
// detection yields a key that never matches). When two hash-generations of
|
|
||||||
// one series collide, the row with the newest updated_at wins and the rest
|
|
||||||
// are deleted. Idempotent: hashless IDs never match the regex.
|
|
||||||
func migrateAsuraKeys(db *sql.DB) error {
|
|
||||||
rows, err := db.Query(`SELECT key, series_id, updated_at FROM bookmarks WHERE site = 'asura'`)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("list asura rows: %w", err)
|
|
||||||
}
|
|
||||||
type row struct {
|
|
||||||
key, id string
|
|
||||||
updated int64
|
|
||||||
}
|
|
||||||
var all []row
|
|
||||||
for rows.Next() {
|
|
||||||
var r row
|
|
||||||
if err := rows.Scan(&r.key, &r.id, &r.updated); err != nil {
|
|
||||||
rows.Close()
|
|
||||||
return fmt.Errorf("scan asura row: %w", err)
|
|
||||||
}
|
|
||||||
all = append(all, r)
|
|
||||||
}
|
|
||||||
if err := rows.Close(); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
defer tx.Rollback()
|
||||||
|
|
||||||
groups := map[string][]row{}
|
var applied bool
|
||||||
for _, r := range all {
|
if err := tx.QueryRow(
|
||||||
stripped := AsuraBuildHash.ReplaceAllString(r.id, "")
|
`SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1)`,
|
||||||
groups[stripped] = append(groups[stripped], r)
|
version).Scan(&applied); err != nil {
|
||||||
|
return err
|
||||||
}
|
}
|
||||||
for stripped, g := range groups {
|
if applied {
|
||||||
winner := 0
|
return nil
|
||||||
for i := range g {
|
|
||||||
if g[i].updated > g[winner].updated {
|
|
||||||
winner = i
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Losers go first: rewriting the winner to the stripped key while a
|
|
||||||
// pre-existing hashless row still holds it is a primary-key collision.
|
|
||||||
for i, r := range g {
|
|
||||||
if i == winner {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if _, err := db.Exec(`DELETE FROM bookmarks WHERE key = ?`, r.key); err != nil {
|
|
||||||
return fmt.Errorf("drop duplicate %q: %w", r.key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if r := g[winner]; r.id != stripped {
|
|
||||||
if _, err := db.Exec(
|
|
||||||
`UPDATE bookmarks SET key = ?, series_id = ? WHERE key = ?`,
|
|
||||||
"asura:"+stripped, stripped, r.key); err != nil {
|
|
||||||
return fmt.Errorf("rewrite key %q: %w", r.key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
return nil
|
// No parameters, so this goes over the simple protocol and a migration may
|
||||||
|
// hold more than one statement.
|
||||||
|
if _, err := tx.Exec(body); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(`INSERT INTO schema_migrations (version) VALUES ($1)`, version); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return tx.Commit()
|
||||||
}
|
}
|
||||||
|
|
||||||
func existingColumns(db *sql.DB, table string) (map[string]struct{}, error) {
|
// scanBookmark reads one row in bookmarkColumns order. Every column is NOT
|
||||||
rows, err := db.Query(`SELECT name FROM pragma_table_info(?)`, table)
|
// NULL except latest_chapter_num, where NULL means "never captured" — a
|
||||||
if err != nil {
|
// distinct state from chapter zero, and the reason for the pointer.
|
||||||
return nil, fmt.Errorf("read %s columns: %w", table, err)
|
|
||||||
}
|
|
||||||
defer rows.Close()
|
|
||||||
|
|
||||||
out := map[string]struct{}{}
|
|
||||||
for rows.Next() {
|
|
||||||
var name string
|
|
||||||
if err := rows.Scan(&name); err != nil {
|
|
||||||
return nil, fmt.Errorf("scan column name: %w", err)
|
|
||||||
}
|
|
||||||
out[name] = struct{}{}
|
|
||||||
}
|
|
||||||
return out, rows.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
// scanBookmark reads one row in bookmarkColumns order, translating SQLite's
|
|
||||||
// integer bool and nullable latest_chapter_num into Go types.
|
|
||||||
//
|
|
||||||
// The optional columns are read through Null* types because rows predating
|
|
||||||
// this code (or written by hand) may hold NULL where the app only ever writes
|
|
||||||
// zero values. Only latest_chapter_num distinguishes the two: everywhere else
|
|
||||||
// NULL and the zero value mean the same thing to clients.
|
|
||||||
func scanBookmark(scan func(...any) error) (Bookmark, error) {
|
func scanBookmark(scan func(...any) error) (Bookmark, error) {
|
||||||
var (
|
var (
|
||||||
b Bookmark
|
b Bookmark
|
||||||
title, seriesURL, cover sql.NullString
|
latestChapterNum sql.NullFloat64
|
||||||
lastChapter, lastChapterURL, latestChapter sql.NullString
|
|
||||||
status sql.NullString
|
|
||||||
lastChapterNum, latestChapterNum sql.NullFloat64
|
|
||||||
favorite sql.NullInt64
|
|
||||||
)
|
)
|
||||||
if err := scan(
|
if err := scan(
|
||||||
&b.Key, &b.Site, &b.SeriesID, &title, &seriesURL, &cover,
|
&b.Key, &b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover,
|
||||||
&lastChapter, &lastChapterNum, &lastChapterURL,
|
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
|
||||||
&favorite, &latestChapter, &latestChapterNum, &b.UpdatedAt, &status, &b.Kind,
|
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return Bookmark{}, err
|
return Bookmark{}, err
|
||||||
}
|
}
|
||||||
b.Title = title.String
|
|
||||||
b.SeriesURL = seriesURL.String
|
|
||||||
b.Cover = cover.String
|
|
||||||
b.LastChapter = lastChapter.String
|
|
||||||
b.LastChapterNum = lastChapterNum.Float64
|
|
||||||
b.LastChapterURL = lastChapterURL.String
|
|
||||||
b.Favorite = favorite.Int64 != 0
|
|
||||||
b.LatestChapter = latestChapter.String
|
|
||||||
if latestChapterNum.Valid {
|
if latestChapterNum.Valid {
|
||||||
b.LatestChapterNum = &latestChapterNum.Float64
|
b.LatestChapterNum = &latestChapterNum.Float64
|
||||||
}
|
}
|
||||||
// A NULL, empty, or unrecognised bucket (e.g. a hand-edited row) would
|
// An unrecognised bucket (a hand-edited row) would leave the row in no list
|
||||||
// leave the row in no list at all, so anything outside the three known
|
// at all, so anything outside the three known buckets reads as the default
|
||||||
// buckets reads as the default rather than being passed through.
|
// rather than being passed through.
|
||||||
b.Status = status.String
|
|
||||||
if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished {
|
if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished {
|
||||||
b.Status = StatusReading
|
b.Status = StatusReading
|
||||||
}
|
}
|
||||||
return b, nil
|
return b, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// scanSeries reads one row in seriesColumns order, plus the due query's
|
||||||
|
// reader_count column. latest_chapter_num is NULL until the first capture,
|
||||||
|
// same as on the bookmark read path.
|
||||||
|
func scanSeries(scan func(...any) error) (Series, error) {
|
||||||
|
var (
|
||||||
|
sr Series
|
||||||
|
latestChapterNum sql.NullFloat64
|
||||||
|
)
|
||||||
|
if err := scan(
|
||||||
|
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
|
||||||
|
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
|
||||||
|
&sr.readerCount,
|
||||||
|
); err != nil {
|
||||||
|
return Series{}, err
|
||||||
|
}
|
||||||
|
if latestChapterNum.Valid {
|
||||||
|
sr.LatestChapterNum = &latestChapterNum.Float64
|
||||||
|
}
|
||||||
|
return sr, nil
|
||||||
|
}
|
||||||
|
|
||||||
// Close releases the underlying database handle.
|
// Close releases the underlying database handle.
|
||||||
func (s *Store) Close() error { return s.db.Close() }
|
func (s *Store) Close() error { return s.db.Close() }
|
||||||
|
|
||||||
// List returns every bookmark, newest activity first.
|
// List returns every bookmark, newest activity first. Series-owned fields are
|
||||||
|
// joined in, so each Bookmark reads back whole and flat (ADR-0004).
|
||||||
func (s *Store) List() ([]Bookmark, error) {
|
func (s *Store) List() ([]Bookmark, error) {
|
||||||
rows, err := s.db.Query(`SELECT ` + bookmarkColumns + `
|
rows, err := s.db.Query(`SELECT ` + bookmarkColumns + `
|
||||||
FROM bookmarks
|
FROM bookmarks b
|
||||||
ORDER BY updated_at DESC`)
|
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||||
|
ORDER BY b.updated_at DESC`)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("query bookmarks: %w", err)
|
return nil, fmt.Errorf("query bookmarks: %w", err)
|
||||||
}
|
}
|
||||||
@@ -365,7 +331,9 @@ func (s *Store) List() ([]Bookmark, error) {
|
|||||||
// the fields they do not touch.
|
// the fields they do not touch.
|
||||||
func (s *Store) Get(key string) (Bookmark, bool, error) {
|
func (s *Store) Get(key string) (Bookmark, bool, error) {
|
||||||
b, err := scanBookmark(s.db.QueryRow(
|
b, err := scanBookmark(s.db.QueryRow(
|
||||||
`SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, key).Scan)
|
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
||||||
|
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||||
|
WHERE b.key = $1`, key).Scan)
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
return Bookmark{}, false, nil
|
return Bookmark{}, false, nil
|
||||||
}
|
}
|
||||||
@@ -376,7 +344,15 @@ func (s *Store) Get(key string) (Bookmark, bool, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Upsert inserts or replaces a bookmark by key (last-write-wins) and returns
|
// Upsert inserts or replaces a bookmark by key (last-write-wins) and returns
|
||||||
// the row as actually stored.
|
// the row as actually stored — one flat object with the series-owned fields
|
||||||
|
// joined in, exactly as GET reports it (ADR-0004).
|
||||||
|
//
|
||||||
|
// The flat body is decomposed across two tables in one transaction. The series
|
||||||
|
// row is written first (the bookmarks FK requires it to exist), then the
|
||||||
|
// bookmark row. On the series side, title/series_url/cover are applied only
|
||||||
|
// when the row is brand new: once a series exists, client-supplied values are
|
||||||
|
// ignored, because the row is shared and the values are scraped page content —
|
||||||
|
// see ADR-0003. Kind and the latest-chapter fields are last-write-wins.
|
||||||
//
|
//
|
||||||
// b.UpdatedAt is only a candidate: it is applied when the row is new or when
|
// b.UpdatedAt is only a candidate: it is applied when the row is new or when
|
||||||
// last_chapter_num changes, and otherwise the stored value is kept. Clients
|
// last_chapter_num changes, and otherwise the stored value is kept. Clients
|
||||||
@@ -395,49 +371,65 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
|
|||||||
latestNum = *b.LatestChapterNum
|
latestNum = *b.LatestChapterNum
|
||||||
}
|
}
|
||||||
|
|
||||||
// IS NOT is SQLite's null-safe comparison. Within DO UPDATE, a bare column
|
// The kind column resolves on the VALUES side, not in the conflict clause:
|
||||||
// is the stored row and excluded.* is the incoming one; a brand-new key
|
// excluded.* is the row *after* these expressions are evaluated, so a
|
||||||
// never reaches this clause, so it keeps the fresh timestamp from VALUES.
|
// default applied there would look identical to a real 'manga' and would
|
||||||
|
// overwrite a novel series on every PUT from a client that knows nothing
|
||||||
|
// about the column. Resolved once here, an empty incoming kind means "keep
|
||||||
|
// what is stored", and only a brand-new row falls through to the literal
|
||||||
|
// default. The subquery runs inside this transaction, so it sees the row
|
||||||
|
// this statement is about to conflict with. Same pattern as the status
|
||||||
|
// COALESCE on the bookmark insert below.
|
||||||
//
|
//
|
||||||
// The status and kind columns resolve on the VALUES side, not in the
|
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no
|
||||||
// conflict clause: excluded.* is the row *after* these expressions are
|
// target column to infer the parameter type from, and Postgres rejects the
|
||||||
// evaluated, so a default applied there would look identical to a real
|
// statement rather than guessing.
|
||||||
// 'reading' / 'manga' and would overwrite an archived or novel row on
|
|
||||||
// every PUT from a client that knows nothing about the column. Resolved
|
|
||||||
// once here, an empty incoming status or kind means "keep what is
|
|
||||||
// stored", and only a brand-new row falls through to the literal
|
|
||||||
// default. The subquery runs inside this transaction, so it sees the
|
|
||||||
// row this statement is about to conflict with.
|
|
||||||
if _, err := tx.Exec(`
|
if _, err := tx.Exec(`
|
||||||
INSERT INTO bookmarks (`+bookmarkColumns+`)
|
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
|
latest_chapter, latest_chapter_num)
|
||||||
COALESCE(NULLIF(?, ''), (SELECT status FROM bookmarks WHERE key = ?), 'reading'),
|
VALUES ($1, $2, $3, $4, $5,
|
||||||
COALESCE(NULLIF(?, ''), (SELECT kind FROM bookmarks WHERE key = ?), 'manga'))
|
COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
|
||||||
ON CONFLICT(key) DO UPDATE SET
|
$7, $8)
|
||||||
site=excluded.site, series_id=excluded.series_id, title=excluded.title,
|
ON CONFLICT (site, series_id) DO UPDATE SET
|
||||||
series_url=excluded.series_url, cover=excluded.cover,
|
kind=excluded.kind,
|
||||||
|
latest_chapter=excluded.latest_chapter,
|
||||||
|
latest_chapter_num=excluded.latest_chapter_num`,
|
||||||
|
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
|
||||||
|
b.LatestChapter, latestNum); err != nil {
|
||||||
|
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// IS DISTINCT FROM is Postgres's null-safe comparison, and it is what
|
||||||
|
// implements the ordering rule. Within DO UPDATE, a bare column is the
|
||||||
|
// stored row and excluded.* is the incoming one; a brand-new key never
|
||||||
|
// reaches this clause, so it keeps the fresh timestamp from VALUES.
|
||||||
|
if _, err := tx.Exec(`
|
||||||
|
INSERT INTO bookmarks (key, site, series_id, last_chapter, last_chapter_num,
|
||||||
|
last_chapter_url, favorite, status, updated_at)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7,
|
||||||
|
COALESCE(NULLIF($8::text, ''), (SELECT status FROM bookmarks WHERE key = $1), 'reading'),
|
||||||
|
$9)
|
||||||
|
ON CONFLICT (key) DO UPDATE SET
|
||||||
|
site=excluded.site, series_id=excluded.series_id,
|
||||||
last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num,
|
last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num,
|
||||||
last_chapter_url=excluded.last_chapter_url,
|
last_chapter_url=excluded.last_chapter_url,
|
||||||
favorite=excluded.favorite,
|
favorite=excluded.favorite,
|
||||||
latest_chapter=excluded.latest_chapter,
|
|
||||||
latest_chapter_num=excluded.latest_chapter_num,
|
|
||||||
status=excluded.status,
|
status=excluded.status,
|
||||||
kind=excluded.kind,
|
|
||||||
updated_at=CASE
|
updated_at=CASE
|
||||||
WHEN bookmarks.last_chapter_num IS NOT excluded.last_chapter_num
|
WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num
|
||||||
THEN excluded.updated_at
|
THEN excluded.updated_at
|
||||||
ELSE bookmarks.updated_at
|
ELSE bookmarks.updated_at
|
||||||
END`,
|
END`,
|
||||||
b.Key, b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover,
|
b.Key, b.Site, b.SeriesID,
|
||||||
b.LastChapter, b.LastChapterNum, b.LastChapterURL,
|
b.LastChapter, b.LastChapterNum, b.LastChapterURL,
|
||||||
b.Favorite, b.LatestChapter, latestNum, b.UpdatedAt,
|
b.Favorite, b.Status, b.UpdatedAt); err != nil {
|
||||||
b.Status, b.Key,
|
|
||||||
b.Kind, b.Key); err != nil {
|
|
||||||
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
|
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
stored, err := scanBookmark(tx.QueryRow(
|
stored, err := scanBookmark(tx.QueryRow(
|
||||||
`SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, b.Key).Scan)
|
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
||||||
|
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||||
|
WHERE b.key = $1`, b.Key).Scan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err)
|
return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err)
|
||||||
}
|
}
|
||||||
@@ -449,77 +441,100 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
|
|||||||
|
|
||||||
// Delete removes a bookmark by key. Deleting a missing key is not an error.
|
// Delete removes a bookmark by key. Deleting a missing key is not an error.
|
||||||
func (s *Store) Delete(key string) error {
|
func (s *Store) Delete(key string) error {
|
||||||
if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = ?`, key); err != nil {
|
if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = $1`, key); err != nil {
|
||||||
return fmt.Errorf("delete %q: %w", key, err)
|
return fmt.Errorf("delete %q: %w", key, err)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// DueForLatestCheck returns bookmarks whose server-side latest-chapter check has
|
// DueForLatestCheck returns series whose server-side latest-chapter check has
|
||||||
// aged past cutoffMs, least-recently-checked first, at most limit of them.
|
// aged past cutoffMs, ordered by how many bookmarks reference them (descending)
|
||||||
|
// then least-recently-checked first, at most limit of them.
|
||||||
//
|
//
|
||||||
// Oldest-first is what keeps the poller fair when the backlog outgrows its
|
// The reader_count ordering is the point of the split (ADR-0003): a series
|
||||||
|
// shared by several readers is fetched once per due cycle, and the popular
|
||||||
|
// ones stay freshest while the long tail absorbs any shortfall. Within one
|
||||||
|
// reader count, oldest-first keeps the poll fair when the backlog outgrows its
|
||||||
// throughput: the most neglected series is always next, so a large collection
|
// throughput: the most neglected series is always next, so a large collection
|
||||||
// refreshes uniformly slower rather than leaving a tail that never refreshes at
|
// refreshes uniformly slower rather than leaving a tail that never refreshes at
|
||||||
// all. The userscript sorts its own queue the same way (L453).
|
// all. The userscript sorts its own queue the same way (L453).
|
||||||
//
|
//
|
||||||
// Bookmarks with no series_url are skipped — there is nothing to fetch, which
|
// Series with no series_url are skipped — there is nothing to fetch, which is
|
||||||
// is the same filter the userscript applies at L452.
|
// the same filter the userscript applies at L452. Series whose only bookmarks
|
||||||
//
|
// are finished are skipped too: nothing more is coming, so fetching them only
|
||||||
// Finished series are excluded: nothing more is coming, so fetching them only
|
// burns requests. Archived bookmarks still count — knowing what a shelved
|
||||||
// burns requests. Archived ones are deliberately still polled — knowing what a
|
// series is up to is the whole reason for archiving instead of deleting.
|
||||||
// shelved series is up to is the whole reason for archiving instead of deleting.
|
// A series with no bookmarks at all never appears: the join excludes it.
|
||||||
func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Bookmark, error) {
|
func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Series, error) {
|
||||||
rows, err := s.db.Query(`SELECT `+bookmarkColumns+`
|
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(b.key) AS reader_count
|
||||||
FROM bookmarks
|
FROM series s
|
||||||
WHERE series_url IS NOT NULL AND series_url <> ''
|
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||||
AND status IS NOT 'finished'
|
WHERE s.series_url <> ''
|
||||||
AND latest_checked_at <= ?
|
AND s.latest_checked_at <= $1
|
||||||
ORDER BY latest_checked_at ASC
|
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||||
LIMIT ?`, cutoffMs, limit)
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
|
||||||
|
HAVING COUNT(b.key) FILTER (WHERE b.status <> 'finished') > 0
|
||||||
|
ORDER BY reader_count DESC, s.latest_checked_at ASC
|
||||||
|
LIMIT $2`, cutoffMs, limit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("query due bookmarks: %w", err)
|
return nil, fmt.Errorf("query due series: %w", err)
|
||||||
}
|
}
|
||||||
defer rows.Close()
|
defer rows.Close()
|
||||||
|
|
||||||
out := []Bookmark{}
|
out := []Series{}
|
||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
b, err := scanBookmark(rows.Scan)
|
sr, err := scanSeries(rows.Scan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("scan due bookmark: %w", err)
|
return nil, fmt.Errorf("scan due series: %w", err)
|
||||||
}
|
}
|
||||||
out = append(out, b)
|
out = append(out, sr)
|
||||||
}
|
}
|
||||||
return out, rows.Err()
|
return out, rows.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
// MarkLatestChecked records that the server looked at key at ts, whatever the
|
// MarkLatestChecked records that the server looked at a series at ts, whatever
|
||||||
// look turned up. Marking a missing key is not an error: the row may have been
|
// the look turned up. Marking a missing series is not an error: the row may
|
||||||
// deleted while a fetch was in flight.
|
// have been orphaned while a fetch was in flight.
|
||||||
//
|
//
|
||||||
// This is the one write that does not go through Upsert, and the column is kept
|
// This is the one write that does not go through Upsert, and the column is kept
|
||||||
// out of bookmarkColumns on purpose. PUT /bookmarks/{key} decodes a whole
|
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
|
||||||
// Bookmark from the client and Upsert writes every column it knows about, so a
|
// a whole Bookmark from the client and Upsert writes every series column it
|
||||||
// userscript PUT — which has no idea this field exists — would write a zero and
|
// knows about, so a userscript PUT — which has no idea this field exists —
|
||||||
// reset the cooldown, making the poller re-fetch that series every tick for as
|
// would write a zero and reset the cooldown, making the poller re-fetch that
|
||||||
// long as the user kept reading it.
|
// series every tick for as long as the user kept reading it.
|
||||||
func (s *Store) MarkLatestChecked(key string, ts int64) error {
|
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
|
||||||
if _, err := s.db.Exec(
|
if _, err := s.db.Exec(
|
||||||
`UPDATE bookmarks SET latest_checked_at = ? WHERE key = ?`, ts, key); err != nil {
|
`UPDATE series SET latest_checked_at = $1 WHERE site = $2 AND series_id = $3`,
|
||||||
return fmt.Errorf("mark checked %q: %w", key, err)
|
ts, site, seriesID); err != nil {
|
||||||
|
return fmt.Errorf("mark checked %s:%s: %w", site, seriesID, err)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
||||||
// tests outside this package (the poller's own tests assert on cooldown
|
// tests outside this package (the poller's own tests assert on cooldown
|
||||||
// bookkeeping) — see MarkLatestChecked for why the field itself stays off
|
// bookkeeping) — see MarkLatestChecked for why the field stays off the
|
||||||
// Bookmark.
|
// client-visible row.
|
||||||
func (s *Store) LatestCheckedAt(key string) (int64, error) {
|
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
|
||||||
var ts int64
|
var ts int64
|
||||||
if err := s.db.QueryRow(
|
if err := s.db.QueryRow(
|
||||||
`SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil {
|
`SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`,
|
||||||
return 0, fmt.Errorf("latest checked at %q: %w", key, err)
|
site, seriesID).Scan(&ts); err != nil {
|
||||||
|
return 0, fmt.Errorf("latest checked at %s:%s: %w", site, seriesID, err)
|
||||||
}
|
}
|
||||||
return ts, nil
|
return ts, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetLatestChapter records the newest chapter the poll found on a series page.
|
||||||
|
// The poller walks Series rather than Bookmarks, so this is a series-level
|
||||||
|
// write: the row is shared, and updating it once refreshes every bookmark that
|
||||||
|
// joins to it. Touching a missing series is not an error.
|
||||||
|
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
|
||||||
|
if _, err := s.db.Exec(
|
||||||
|
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
|
||||||
|
WHERE site = $1 AND series_id = $2`,
|
||||||
|
site, seriesID, label, num); err != nil {
|
||||||
|
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,79 +2,55 @@ package store
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"database/sql"
|
"database/sql"
|
||||||
"path/filepath"
|
"os"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"bookmarkmanager/backend/internal/pgtest"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||||
|
|
||||||
func newTestStore(t *testing.T) *Store {
|
func newTestStore(t *testing.T) *Store {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
store, err := Open(filepath.Join(t.TempDir(), "test.db"))
|
store, err := Open(pgtest.URL(t))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("OpenStore: %v", err)
|
t.Fatalf("Open: %v", err)
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { store.Close() })
|
t.Cleanup(func() { store.Close() })
|
||||||
return store
|
return store
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestOpenStoreMigratesLegacySchema(t *testing.T) {
|
// The migration runner runs on every start, so a second Open against a
|
||||||
dbPath := filepath.Join(t.TempDir(), "legacy.db")
|
// database it already built must be a no-op rather than a duplicate-table
|
||||||
|
// error, and must leave the rows alone.
|
||||||
legacy, err := sql.Open("sqlite", dbPath)
|
func TestOpenIsIdempotent(t *testing.T) {
|
||||||
|
url := pgtest.URL(t)
|
||||||
|
first, err := Open(url)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("open legacy db: %v", err)
|
t.Fatalf("Open: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := legacy.Exec(`
|
if _, err := first.Upsert(Bookmark{
|
||||||
CREATE TABLE bookmarks (
|
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
|
||||||
key TEXT PRIMARY KEY,
|
}); err != nil {
|
||||||
site TEXT NOT NULL,
|
t.Fatalf("seed: %v", err)
|
||||||
series_id TEXT NOT NULL,
|
|
||||||
title TEXT,
|
|
||||||
series_url TEXT,
|
|
||||||
cover TEXT,
|
|
||||||
last_chapter TEXT,
|
|
||||||
last_chapter_num REAL,
|
|
||||||
last_chapter_url TEXT,
|
|
||||||
updated_at INTEGER NOT NULL
|
|
||||||
)`); err != nil {
|
|
||||||
t.Fatalf("create legacy schema: %v", err)
|
|
||||||
}
|
|
||||||
if _, err := legacy.Exec(`
|
|
||||||
INSERT INTO bookmarks (key, site, series_id, title, last_chapter, last_chapter_num, updated_at)
|
|
||||||
VALUES ('asura:legacy', 'asura', 'legacy', 'Legacy Series', 'Chapter 7', 7, 123)`); err != nil {
|
|
||||||
t.Fatalf("seed legacy row: %v", err)
|
|
||||||
}
|
|
||||||
if err := legacy.Close(); err != nil {
|
|
||||||
t.Fatalf("close legacy db: %v", err)
|
|
||||||
}
|
}
|
||||||
|
first.Close()
|
||||||
|
|
||||||
store, err := Open(dbPath)
|
second, err := Open(url)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("OpenStore on legacy db: %v", err)
|
t.Fatalf("reopen: %v", err)
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { store.Close() })
|
t.Cleanup(func() { second.Close() })
|
||||||
|
|
||||||
list, err := store.List()
|
list, err := second.List()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("List: %v", err)
|
t.Fatalf("List: %v", err)
|
||||||
}
|
}
|
||||||
if len(list) != 1 || list[0].Key != "asura:legacy" {
|
if len(list) != 1 || list[0].Key != "asura:solo" {
|
||||||
t.Fatalf("legacy row lost: %+v", list)
|
t.Fatalf("rows after reopen = %+v, want only the seeded one", list)
|
||||||
}
|
}
|
||||||
got := list[0]
|
|
||||||
if got.Title != "Legacy Series" || got.LastChapterNum != 7 || got.UpdatedAt != 123 {
|
|
||||||
t.Fatalf("legacy data mangled: %+v", got)
|
|
||||||
}
|
|
||||||
if got.Favorite || got.LatestChapter != "" || got.LatestChapterNum != nil {
|
|
||||||
t.Fatalf("new columns should default empty, got %+v", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Reopening an already-migrated database must be a no-op, not an error.
|
|
||||||
store2, err := Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("OpenStore is not idempotent: %v", err)
|
|
||||||
}
|
|
||||||
store2.Close()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestStoreGet(t *testing.T) {
|
func TestStoreGet(t *testing.T) {
|
||||||
@@ -150,30 +126,41 @@ func TestBookmarkContinueURL(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// readLatestCheckedAt reads the column directly. It is deliberately absent from
|
// readLatestCheckedAt reads the column directly. It is deliberately absent from
|
||||||
// Bookmark (see Store.Upsert), so tests cannot assert on it any other way.
|
// Series (see Store.MarkLatestChecked), so tests cannot assert on it any other
|
||||||
|
// way. The key splits the same way the API handler derives site/series_id.
|
||||||
func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 {
|
func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
site, seriesID, ok := strings.Cut(key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", key)
|
||||||
|
}
|
||||||
var ts int64
|
var ts int64
|
||||||
if err := s.db.QueryRow(
|
if err := s.db.QueryRow(
|
||||||
`SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil {
|
`SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`,
|
||||||
|
site, seriesID).Scan(&ts); err != nil {
|
||||||
t.Fatalf("read latest_checked_at %q: %v", key, err)
|
t.Fatalf("read latest_checked_at %q: %v", key, err)
|
||||||
}
|
}
|
||||||
return ts
|
return ts
|
||||||
}
|
}
|
||||||
|
|
||||||
// seedForCheck inserts a bookmark and forces its latest_checked_at.
|
// seedForCheck inserts a bookmark (and with it its series) and forces the
|
||||||
|
// series' latest_checked_at.
|
||||||
func seedForCheck(t *testing.T, s *Store, key, seriesURL string, checkedAt int64) {
|
func seedForCheck(t *testing.T, s *Store, key, seriesURL string, checkedAt int64) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
site, seriesID, ok := strings.Cut(key, ":")
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("key %q: no ':' separator", key)
|
||||||
|
}
|
||||||
if _, err := s.Upsert(Bookmark{
|
if _, err := s.Upsert(Bookmark{
|
||||||
Key: key,
|
Key: key,
|
||||||
Site: "asura",
|
Site: site,
|
||||||
SeriesID: key,
|
SeriesID: seriesID,
|
||||||
SeriesURL: seriesURL,
|
SeriesURL: seriesURL,
|
||||||
UpdatedAt: 1000,
|
UpdatedAt: 1000,
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
t.Fatalf("seed %q: %v", key, err)
|
t.Fatalf("seed %q: %v", key, err)
|
||||||
}
|
}
|
||||||
if err := s.MarkLatestChecked(key, checkedAt); err != nil {
|
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
|
||||||
t.Fatalf("seed mark %q: %v", key, err)
|
t.Fatalf("seed mark %q: %v", key, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -224,8 +211,8 @@ func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) {
|
|||||||
if len(due) != 2 {
|
if len(due) != 2 {
|
||||||
t.Fatalf("got %d rows, want 2 (limit)", len(due))
|
t.Fatalf("got %d rows, want 2 (limit)", len(due))
|
||||||
}
|
}
|
||||||
if due[0].Key != "asura:a" || due[1].Key != "asura:b" {
|
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" {
|
||||||
t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key, due[1].Key)
|
t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -233,20 +220,21 @@ func TestMarkLatestChecked(t *testing.T) {
|
|||||||
s := newTestStore(t)
|
s := newTestStore(t)
|
||||||
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 0)
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 0)
|
||||||
|
|
||||||
if err := s.MarkLatestChecked("asura:x", 4242); err != nil {
|
if err := s.MarkLatestChecked("asura", "x", 4242); err != nil {
|
||||||
t.Fatalf("MarkLatestChecked: %v", err)
|
t.Fatalf("MarkLatestChecked: %v", err)
|
||||||
}
|
}
|
||||||
if got := readLatestCheckedAt(t, s, "asura:x"); got != 4242 {
|
if got := readLatestCheckedAt(t, s, "asura:x"); got != 4242 {
|
||||||
t.Fatalf("latest_checked_at = %d, want 4242", got)
|
t.Fatalf("latest_checked_at = %d, want 4242", got)
|
||||||
}
|
}
|
||||||
// A missing key is not an error: the row may have been deleted mid-fetch.
|
// A missing series is not an error: its bookmarks may have been deleted
|
||||||
if err := s.MarkLatestChecked("asura:gone", 1); err != nil {
|
// mid-fetch.
|
||||||
t.Fatalf("MarkLatestChecked on missing key: %v", err)
|
if err := s.MarkLatestChecked("asura", "gone", 1); err != nil {
|
||||||
|
t.Fatalf("MarkLatestChecked on missing series: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Upsert must not touch latest_checked_at. If the column ever migrates into
|
// Upsert must not touch latest_checked_at. If the column ever migrates into
|
||||||
// bookmarkColumns, this fails and the cooldown is silently dead.
|
// the client-visible write path, this fails and the cooldown is silently dead.
|
||||||
func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
|
func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
|
||||||
s := newTestStore(t)
|
s := newTestStore(t)
|
||||||
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 999)
|
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 999)
|
||||||
@@ -264,53 +252,6 @@ func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// migrateColumns must be able to bring a database created before this column up
|
|
||||||
// to date, not just create it fresh.
|
|
||||||
func TestMigrateAddsLatestCheckedAt(t *testing.T) {
|
|
||||||
path := filepath.Join(t.TempDir(), "old.db")
|
|
||||||
|
|
||||||
old, err := sql.Open("sqlite", path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("open: %v", err)
|
|
||||||
}
|
|
||||||
// A pre-latest_checked_at table, matching the schema as it shipped before.
|
|
||||||
if _, err := old.Exec(`CREATE TABLE bookmarks (
|
|
||||||
key TEXT PRIMARY KEY, site TEXT NOT NULL, series_id TEXT NOT NULL,
|
|
||||||
title TEXT, series_url TEXT, cover TEXT,
|
|
||||||
last_chapter TEXT, last_chapter_num REAL, last_chapter_url TEXT,
|
|
||||||
favorite INTEGER NOT NULL DEFAULT 0,
|
|
||||||
latest_chapter TEXT NOT NULL DEFAULT '', latest_chapter_num REAL,
|
|
||||||
updated_at INTEGER NOT NULL)`); err != nil {
|
|
||||||
t.Fatalf("create old table: %v", err)
|
|
||||||
}
|
|
||||||
if _, err := old.Exec(
|
|
||||||
`INSERT INTO bookmarks (key, site, series_id, series_url, updated_at)
|
|
||||||
VALUES ('asura:x', 'asura', 'x', 'https://asurascans.com/comics/x', 5)`); err != nil {
|
|
||||||
t.Fatalf("seed old row: %v", err)
|
|
||||||
}
|
|
||||||
if err := old.Close(); err != nil {
|
|
||||||
t.Fatalf("close: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
s, err := Open(path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("OpenStore on pre-existing db: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { s.Close() })
|
|
||||||
|
|
||||||
// The migrated row must default to 0 (never checked) and so be due.
|
|
||||||
if got := readLatestCheckedAt(t, s, "asura:x"); got != 0 {
|
|
||||||
t.Fatalf("migrated latest_checked_at = %d, want 0", got)
|
|
||||||
}
|
|
||||||
due, err := s.DueForLatestCheck(1000, 10)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("DueForLatestCheck: %v", err)
|
|
||||||
}
|
|
||||||
if len(due) != 1 {
|
|
||||||
t.Fatalf("got %d due rows after migration, want 1", len(due))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUpsertDefaultsStatusToReading(t *testing.T) {
|
func TestUpsertDefaultsStatusToReading(t *testing.T) {
|
||||||
store := newTestStore(t)
|
store := newTestStore(t)
|
||||||
stored, err := store.Upsert(Bookmark{
|
stored, err := store.Upsert(Bookmark{
|
||||||
@@ -425,41 +366,6 @@ func TestUpsertStatusChangeKeepsUpdatedAt(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A database written before the column existed must gain it, with every
|
|
||||||
// pre-existing row landing in the reading bucket.
|
|
||||||
func TestMigrationAddsStatusToLegacyDatabase(t *testing.T) {
|
|
||||||
path := filepath.Join(t.TempDir(), "legacy.db")
|
|
||||||
db, err := sql.Open("sqlite", path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("open: %v", err)
|
|
||||||
}
|
|
||||||
if _, err := db.Exec(`
|
|
||||||
CREATE TABLE bookmarks (
|
|
||||||
key TEXT PRIMARY KEY, site TEXT NOT NULL, series_id TEXT NOT NULL,
|
|
||||||
title TEXT, series_url TEXT, cover TEXT,
|
|
||||||
last_chapter TEXT, last_chapter_num REAL, last_chapter_url TEXT,
|
|
||||||
updated_at INTEGER NOT NULL);
|
|
||||||
INSERT INTO bookmarks (key, site, series_id, updated_at)
|
|
||||||
VALUES ('asura:old', 'asura', 'old', 1)`); err != nil {
|
|
||||||
t.Fatalf("seed legacy: %v", err)
|
|
||||||
}
|
|
||||||
db.Close()
|
|
||||||
|
|
||||||
store, err := Open(path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("OpenStore: %v", err)
|
|
||||||
}
|
|
||||||
defer store.Close()
|
|
||||||
|
|
||||||
b, ok, err := store.Get("asura:old")
|
|
||||||
if err != nil || !ok {
|
|
||||||
t.Fatalf("Get: ok=%v err=%v", ok, err)
|
|
||||||
}
|
|
||||||
if b.Status != StatusReading {
|
|
||||||
t.Fatalf("Status = %q, want %q", b.Status, StatusReading)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Archiving is the reason to keep polling — the point is to come back to a
|
// Archiving is the reason to keep polling — the point is to come back to a
|
||||||
// series that has moved on. A finished series has nothing left to publish.
|
// series that has moved on. A finished series has nothing left to publish.
|
||||||
func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
||||||
@@ -470,7 +376,7 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
|||||||
{"asura:finished", StatusFinished},
|
{"asura:finished", StatusFinished},
|
||||||
} {
|
} {
|
||||||
if _, err := store.Upsert(Bookmark{
|
if _, err := store.Upsert(Bookmark{
|
||||||
Key: tc.key, Site: "asura", SeriesID: tc.key,
|
Key: tc.key, Site: "asura", SeriesID: strings.TrimPrefix(tc.key, "asura:"),
|
||||||
SeriesURL: "https://asurascans.com/comics/" + tc.key,
|
SeriesURL: "https://asurascans.com/comics/" + tc.key,
|
||||||
Status: tc.status, UpdatedAt: time.Now().UnixMilli(),
|
Status: tc.status, UpdatedAt: time.Now().UnixMilli(),
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
@@ -483,8 +389,8 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
|||||||
t.Fatalf("DueForLatestCheck: %v", err)
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
}
|
}
|
||||||
got := map[string]bool{}
|
got := map[string]bool{}
|
||||||
for _, b := range due {
|
for _, sr := range due {
|
||||||
got[b.Key] = true
|
got[sr.Key()] = true
|
||||||
}
|
}
|
||||||
if !got["asura:reading"] || !got["asura:archived"] {
|
if !got["asura:reading"] || !got["asura:archived"] {
|
||||||
t.Fatalf("due = %v, want reading and archived present", got)
|
t.Fatalf("due = %v, want reading and archived present", got)
|
||||||
@@ -494,135 +400,6 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Asura slugs used to include the site build hash; rows keyed with it must
|
|
||||||
// be rewritten to the stable ID on open, merging hash-generations of the
|
|
||||||
// same series into the newest row.
|
|
||||||
func TestOpenStoreMigratesAsuraBuildHashKeys(t *testing.T) {
|
|
||||||
dbPath := filepath.Join(t.TempDir(), "hash.db")
|
|
||||||
|
|
||||||
store, err := Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("open: %v", err)
|
|
||||||
}
|
|
||||||
seed := []Bookmark{
|
|
||||||
{Key: "asura:swordmasters-youngest-son-f886a8af", Site: "asura",
|
|
||||||
SeriesID: "swordmasters-youngest-son-f886a8af", Title: "Old gen",
|
|
||||||
LastChapterNum: 50, UpdatedAt: 100},
|
|
||||||
{Key: "asura:swordmasters-youngest-son-059befe1", Site: "asura",
|
|
||||||
SeriesID: "swordmasters-youngest-son-059befe1", Title: "Re-bookmarked",
|
|
||||||
LastChapterNum: 60, UpdatedAt: 200},
|
|
||||||
{Key: "asura:overgeared-059befe1", Site: "asura",
|
|
||||||
SeriesID: "overgeared-059befe1", Title: "Single gen", UpdatedAt: 150},
|
|
||||||
// Hash-like suffix on another site must be left alone.
|
|
||||||
{Key: "demonic:x-deadbeef", Site: "demonic",
|
|
||||||
SeriesID: "x-deadbeef", Title: "Not asura", UpdatedAt: 300},
|
|
||||||
}
|
|
||||||
for _, b := range seed {
|
|
||||||
if _, err := store.Upsert(b); err != nil {
|
|
||||||
t.Fatalf("seed %s: %v", b.Key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if err := store.Close(); err != nil {
|
|
||||||
t.Fatalf("close: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
reopened, err := Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reopen: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { reopened.Close() })
|
|
||||||
|
|
||||||
list, err := reopened.List()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("List: %v", err)
|
|
||||||
}
|
|
||||||
byKey := map[string]Bookmark{}
|
|
||||||
for _, b := range list {
|
|
||||||
byKey[b.Key] = b
|
|
||||||
}
|
|
||||||
if len(list) != 3 {
|
|
||||||
t.Fatalf("want 3 rows after merge, got %d: %+v", len(list), list)
|
|
||||||
}
|
|
||||||
merged, ok := byKey["asura:swordmasters-youngest-son"]
|
|
||||||
if !ok {
|
|
||||||
t.Fatalf("merged key missing: %+v", byKey)
|
|
||||||
}
|
|
||||||
// Newest row wins the merge.
|
|
||||||
if merged.Title != "Re-bookmarked" || merged.LastChapterNum != 60 || merged.UpdatedAt != 200 {
|
|
||||||
t.Fatalf("merge kept wrong row: %+v", merged)
|
|
||||||
}
|
|
||||||
if merged.SeriesID != "swordmasters-youngest-son" {
|
|
||||||
t.Fatalf("series_id not stripped: %q", merged.SeriesID)
|
|
||||||
}
|
|
||||||
if _, ok := byKey["asura:overgeared-059befe1"]; ok {
|
|
||||||
t.Fatal("single-generation hashed key not rewritten")
|
|
||||||
}
|
|
||||||
if _, ok := byKey["asura:overgeared"]; !ok {
|
|
||||||
t.Fatal("single-generation row missing under stripped key")
|
|
||||||
}
|
|
||||||
if _, ok := byKey["demonic:x-deadbeef"]; !ok {
|
|
||||||
t.Fatal("non-asura row touched")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Idempotent: a third open changes nothing.
|
|
||||||
third, err := Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("third open: %v", err)
|
|
||||||
}
|
|
||||||
third.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
// A hashed row and a pre-existing hashless row of the same series collide on
|
|
||||||
// the stripped key. The winner rewrite must happen only after the loser is
|
|
||||||
// gone, or the UPDATE hits a primary-key collision and OpenStore fails.
|
|
||||||
func TestOpenStoreMigratesAsuraHashlessCollision(t *testing.T) {
|
|
||||||
dbPath := filepath.Join(t.TempDir(), "collision.db")
|
|
||||||
|
|
||||||
store, err := Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("open: %v", err)
|
|
||||||
}
|
|
||||||
seed := []Bookmark{
|
|
||||||
{Key: "asura:overgeared", Site: "asura", SeriesID: "overgeared",
|
|
||||||
Title: "Hashless", LastChapterNum: 10, UpdatedAt: 100},
|
|
||||||
{Key: "asura:overgeared-059befe1", Site: "asura",
|
|
||||||
SeriesID: "overgeared-059befe1", Title: "Hashed newer",
|
|
||||||
LastChapterNum: 20, UpdatedAt: 200},
|
|
||||||
}
|
|
||||||
for _, b := range seed {
|
|
||||||
if _, err := store.Upsert(b); err != nil {
|
|
||||||
t.Fatalf("seed %s: %v", b.Key, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if err := store.Close(); err != nil {
|
|
||||||
t.Fatalf("close: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
reopened, err := Open(dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reopen: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { reopened.Close() })
|
|
||||||
|
|
||||||
list, err := reopened.List()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("List: %v", err)
|
|
||||||
}
|
|
||||||
if len(list) != 1 {
|
|
||||||
t.Fatalf("want 1 row after merge, got %d: %+v", len(list), list)
|
|
||||||
}
|
|
||||||
merged := list[0]
|
|
||||||
if merged.Key != "asura:overgeared" {
|
|
||||||
t.Fatalf("merged key = %q, want asura:overgeared", merged.Key)
|
|
||||||
}
|
|
||||||
if merged.SeriesID != "overgeared" {
|
|
||||||
t.Fatalf("series_id = %q, want overgeared", merged.SeriesID)
|
|
||||||
}
|
|
||||||
if merged.Title != "Hashed newer" || merged.LastChapterNum != 20 || merged.UpdatedAt != 200 {
|
|
||||||
t.Fatalf("merge kept wrong row: %+v", merged)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDisplayChapter(t *testing.T) {
|
func TestDisplayChapter(t *testing.T) {
|
||||||
cases := []struct {
|
cases := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -714,50 +491,263 @@ func TestUpsertEmptyKindKeepsStoredValue(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A database created before this column exists must gain it, backfilled as
|
// The 0002 backfill must survive a database that already ran 0001 with real
|
||||||
// manga, without losing anything.
|
// rows: one series row per distinct (site, series_id) carrying the moved
|
||||||
func TestLegacyDatabaseGainsKindAsManga(t *testing.T) {
|
// columns, and the bookmark keeping the rest. That is the upgrade path for
|
||||||
dbPath := filepath.Join(t.TempDir(), "legacy.db")
|
// every deployed database, so it is exercised rather than trusted.
|
||||||
|
func TestMigration0002BackfillsExistingBookmarks(t *testing.T) {
|
||||||
legacy, err := sql.Open("sqlite", dbPath)
|
url := pgtest.URL(t)
|
||||||
|
db, err := sql.Open("pgx", url)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("open legacy db: %v", err)
|
t.Fatalf("open: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := legacy.Exec(`
|
t.Cleanup(func() { db.Close() })
|
||||||
CREATE TABLE bookmarks (
|
|
||||||
key TEXT PRIMARY KEY,
|
// Run only 0001, as a database created before this change would have.
|
||||||
site TEXT NOT NULL,
|
// migrate() normally creates the version table first; do the same here.
|
||||||
series_id TEXT NOT NULL,
|
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
title TEXT,
|
version bigint PRIMARY KEY,
|
||||||
series_url TEXT,
|
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
|
||||||
cover TEXT,
|
t.Fatalf("create version table: %v", err)
|
||||||
last_chapter TEXT,
|
|
||||||
last_chapter_num REAL,
|
|
||||||
last_chapter_url TEXT,
|
|
||||||
updated_at INTEGER NOT NULL
|
|
||||||
)`); err != nil {
|
|
||||||
t.Fatalf("create legacy schema: %v", err)
|
|
||||||
}
|
}
|
||||||
if _, err := legacy.Exec(`
|
body, err := migrations.ReadFile("migrations/0001_bookmarks.sql")
|
||||||
INSERT INTO bookmarks (key, site, series_id, title, updated_at)
|
if err != nil {
|
||||||
VALUES ('asura:legacy', 'asura', 'legacy', 'Legacy Series', 123)`); err != nil {
|
t.Fatalf("read 0001: %v", err)
|
||||||
|
}
|
||||||
|
if err := applyMigration(db, 1, string(body)); err != nil {
|
||||||
|
t.Fatalf("apply 0001: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := db.Exec(`
|
||||||
|
INSERT INTO bookmarks (key, site, series_id, title, series_url, cover,
|
||||||
|
last_chapter, last_chapter_num, last_chapter_url, favorite, latest_chapter,
|
||||||
|
latest_chapter_num, latest_checked_at, status, kind, updated_at)
|
||||||
|
VALUES ('asura:solo', 'asura', 'solo', 'Solo Leveling',
|
||||||
|
'https://asurascans.com/comics/solo', 'https://asurascans.com/covers/solo.jpg',
|
||||||
|
'Chapter 10', 10, 'https://asurascans.com/comics/solo/ch/10', true,
|
||||||
|
'Chapter 11', 11, 123456, 'reading', 'manga', 1000)`); err != nil {
|
||||||
t.Fatalf("seed legacy row: %v", err)
|
t.Fatalf("seed legacy row: %v", err)
|
||||||
}
|
}
|
||||||
if err := legacy.Close(); err != nil {
|
|
||||||
t.Fatalf("close legacy db: %v", err)
|
// Bring it current: 0002 must backfill the series row, not lose data.
|
||||||
|
if err := migrate(db); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
var (
|
||||||
|
title string
|
||||||
|
checked int64
|
||||||
|
fav bool
|
||||||
|
lastNum float64
|
||||||
|
)
|
||||||
|
if err := db.QueryRow(`SELECT title, latest_checked_at FROM series
|
||||||
|
WHERE site = 'asura' AND series_id = 'solo'`).Scan(&title, &checked); err != nil {
|
||||||
|
t.Fatalf("series row missing after migration: %v", err)
|
||||||
|
}
|
||||||
|
if title != "Solo Leveling" || checked != 123456 {
|
||||||
|
t.Fatalf("series = (%q, %d), want backfilled title and latest_checked_at", title, checked)
|
||||||
|
}
|
||||||
|
if err := db.QueryRow(`SELECT favorite, last_chapter_num FROM bookmarks
|
||||||
|
WHERE key = 'asura:solo'`).Scan(&fav, &lastNum); err != nil {
|
||||||
|
t.Fatalf("bookmark row missing after migration: %v", err)
|
||||||
|
}
|
||||||
|
if !fav || lastNum != 10 {
|
||||||
|
t.Fatalf("bookmark = (%v, %v), want favorite and progress kept", fav, lastNum)
|
||||||
}
|
}
|
||||||
|
|
||||||
store, err := Open(dbPath)
|
// The migrated database opens as a normal store.
|
||||||
|
st, err := Open(url)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Open on legacy db: %v", err)
|
t.Fatalf("Open after migrate: %v", err)
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { store.Close() })
|
defer st.Close()
|
||||||
|
}
|
||||||
|
|
||||||
list, err := store.List()
|
// readSeries reads the series row directly, for asserting on what Upsert
|
||||||
|
// actually stored rather than what the joined Bookmark reports.
|
||||||
|
func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
|
||||||
|
t.Helper()
|
||||||
|
sr, err := scanSeries(s.db.QueryRow(
|
||||||
|
`SELECT `+seriesColumns+`, 0 AS reader_count FROM series s
|
||||||
|
WHERE s.site = $1 AND s.series_id = $2`, site, seriesID).Scan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("List: %v", err)
|
t.Fatalf("read series %s:%s: %v", site, seriesID, err)
|
||||||
}
|
}
|
||||||
if len(list) != 1 || list[0].Kind != KindManga {
|
return sr
|
||||||
t.Fatalf("legacy row should backfill as manga, got %+v", list)
|
}
|
||||||
|
|
||||||
|
// The first PUT for a series creates its row from the client's title, cover
|
||||||
|
// and URL — there is no other source for them (ADR-0003).
|
||||||
|
func TestUpsertCreatesSeriesFromClient(t *testing.T) {
|
||||||
|
store := newTestStore(t)
|
||||||
|
if _, err := store.Upsert(Bookmark{
|
||||||
|
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||||
|
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
|
||||||
|
Cover: "https://asurascans.com/covers/solo.jpg", Kind: KindManga,
|
||||||
|
UpdatedAt: 1000,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("Upsert: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sr := readSeries(t, store, "asura", "solo")
|
||||||
|
if sr.Title != "Solo Leveling" || sr.SeriesURL != "https://asurascans.com/comics/solo" ||
|
||||||
|
sr.Cover != "https://asurascans.com/covers/solo.jpg" {
|
||||||
|
t.Fatalf("series = %+v, want client title/url/cover stored", sr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A PUT naming an existing series must not overwrite its title, cover or URL:
|
||||||
|
// the row is shared, and those values are scraped page content (ADR-0003).
|
||||||
|
func TestUpsertExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
|
||||||
|
store := newTestStore(t)
|
||||||
|
base := Bookmark{
|
||||||
|
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||||
|
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
|
||||||
|
Cover: "https://asurascans.com/covers/solo.jpg", LastChapterNum: 10,
|
||||||
|
UpdatedAt: 1000,
|
||||||
|
}
|
||||||
|
if _, err := store.Upsert(base); err != nil {
|
||||||
|
t.Fatalf("seed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Same series, hostile/compromised values, real progress advance.
|
||||||
|
base.Title = "Scraped Rename"
|
||||||
|
base.SeriesURL = "https://evil.example/solo"
|
||||||
|
base.Cover = "https://evil.example/solo.jpg"
|
||||||
|
base.LastChapterNum = 11
|
||||||
|
got, err := store.Upsert(base)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Upsert: %v", err)
|
||||||
|
}
|
||||||
|
if got.Title != "Solo Leveling" || got.SeriesURL != "https://asurascans.com/comics/solo" ||
|
||||||
|
got.Cover != "https://asurascans.com/covers/solo.jpg" {
|
||||||
|
t.Fatalf("stored = %+v, want original title/url/cover kept", got)
|
||||||
|
}
|
||||||
|
if got.LastChapterNum != 11 {
|
||||||
|
t.Fatalf("LastChapterNum = %v, want 11 — progress in the same request must still land", got.LastChapterNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Kind and latest-chapter are last-write-wins even on an existing series: the
|
||||||
|
// poller and the userscript both report the latest chapter, and kind is only
|
||||||
|
// known to whichever client created the row.
|
||||||
|
func TestUpsertExistingSeriesAcceptsKindAndLatest(t *testing.T) {
|
||||||
|
store := newTestStore(t)
|
||||||
|
base := Bookmark{
|
||||||
|
Key: "asura:solo", Site: "asura", SeriesID: "solo", Kind: KindManga,
|
||||||
|
UpdatedAt: 1000,
|
||||||
|
}
|
||||||
|
if _, err := store.Upsert(base); err != nil {
|
||||||
|
t.Fatalf("seed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
num := 12.0
|
||||||
|
base.Kind = KindNovel
|
||||||
|
base.LatestChapter = "Chapter 12"
|
||||||
|
base.LatestChapterNum = &num
|
||||||
|
got, err := store.Upsert(base)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Upsert: %v", err)
|
||||||
|
}
|
||||||
|
if got.Kind != KindNovel || got.LatestChapter != "Chapter 12" ||
|
||||||
|
got.LatestChapterNum == nil || *got.LatestChapterNum != 12 {
|
||||||
|
t.Fatalf("stored = %+v, want kind and latest chapter updated", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deleting the last bookmark must leave the series row behind, so a later
|
||||||
|
// re-bookmark shows title and cover immediately instead of waiting for a poll.
|
||||||
|
func TestDeleteKeepsSeriesRow(t *testing.T) {
|
||||||
|
store := newTestStore(t)
|
||||||
|
if _, err := store.Upsert(Bookmark{
|
||||||
|
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||||
|
Title: "Solo Leveling", Cover: "https://asurascans.com/covers/solo.jpg",
|
||||||
|
UpdatedAt: 1000,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("seed: %v", err)
|
||||||
|
}
|
||||||
|
if err := store.Delete("asura:solo"); err != nil {
|
||||||
|
t.Fatalf("Delete: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sr := readSeries(t, store, "asura", "solo")
|
||||||
|
if sr.Title != "Solo Leveling" {
|
||||||
|
t.Fatalf("series = %+v, want it kept after the last bookmark is deleted", sr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-bookmark with nothing but progress: the stored title/cover come back.
|
||||||
|
stored, err := store.Upsert(Bookmark{
|
||||||
|
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||||
|
LastChapterNum: 5, UpdatedAt: 2000,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("re-upsert: %v", err)
|
||||||
|
}
|
||||||
|
if stored.Title != "Solo Leveling" || stored.Cover != "https://asurascans.com/covers/solo.jpg" {
|
||||||
|
t.Fatalf("re-bookmark = %+v, want title/cover from the surviving series row", stored)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedSecondReader inserts an extra bookmark on an existing series. Today the
|
||||||
|
// bookmark key is <site>:<series_id>, so two bookmarks can share a series only
|
||||||
|
// once keys stop being derived from the series identity (issue #22); the due
|
||||||
|
// queue's reader-count ordering must already be right for that world.
|
||||||
|
func seedSecondReader(t *testing.T, s *Store, key, site, seriesID string, updatedAt int64) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := s.Upsert(Bookmark{
|
||||||
|
Key: key, Site: site, SeriesID: seriesID, UpdatedAt: updatedAt,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("seed second reader %q: %v", key, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The whole point of the split: a shared series is due once, ordered ahead of
|
||||||
|
// single-reader series by how many bookmarks reference it.
|
||||||
|
func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
// "pop" has two bookmarks but was checked most recently; "solo" has one and
|
||||||
|
// was checked long ago. Reader count must win over age.
|
||||||
|
seedForCheck(t, s, "asura:pop", "https://asurascans.com/comics/pop", 900)
|
||||||
|
seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001)
|
||||||
|
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100)
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck(1000, 10)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 2 {
|
||||||
|
t.Fatalf("due = %d rows, want 2", len(due))
|
||||||
|
}
|
||||||
|
if due[0].Key() != "asura:pop" || due[1].Key() != "asura:solo" {
|
||||||
|
t.Fatalf("due order = %q, %q; want asura:pop (2 readers), asura:solo (1)",
|
||||||
|
due[0].Key(), due[1].Key())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A series with no bookmarks must never appear in the due queue, and nothing
|
||||||
|
// in the store ever deletes it (see TestDeleteKeepsSeriesRow).
|
||||||
|
func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
seedForCheck(t, s, "asura:kept", "https://asurascans.com/comics/kept", 0)
|
||||||
|
if _, err := s.db.Exec(`
|
||||||
|
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||||
|
latest_chapter, latest_chapter_num, latest_checked_at)
|
||||||
|
VALUES ('asura', 'orphan', 'Orphan', 'https://asurascans.com/comics/orphan',
|
||||||
|
'', 'manga', '', NULL, 0)`); err != nil {
|
||||||
|
t.Fatalf("seed orphan series: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
due, err := s.DueForLatestCheck(1000, 10)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DueForLatestCheck: %v", err)
|
||||||
|
}
|
||||||
|
if len(due) != 1 || due[0].Key() != "asura:kept" {
|
||||||
|
t.Fatalf("due = %+v, want only the bookmarked series", due)
|
||||||
|
}
|
||||||
|
var n int
|
||||||
|
if err := s.db.QueryRow(
|
||||||
|
`SELECT count(*) FROM series WHERE site = 'asura' AND series_id = 'orphan'`).Scan(&n); err != nil {
|
||||||
|
t.Fatalf("count orphan series: %v", err)
|
||||||
|
}
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("orphan series count = %d, want 1 (never deleted)", n)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+11
-5
@@ -24,8 +24,10 @@ import (
|
|||||||
type Config struct {
|
type Config struct {
|
||||||
Token string
|
Token string
|
||||||
AllowedOrigins []string
|
AllowedOrigins []string
|
||||||
DBPath string
|
// DatabaseURL is the Postgres connection URL; required, no default,
|
||||||
Port string
|
// because a wrong guess would silently start on an empty database.
|
||||||
|
DatabaseURL string
|
||||||
|
Port string
|
||||||
// WebPassword gates the browser UI. Empty disables the web routes entirely.
|
// WebPassword gates the browser UI. Empty disables the web routes entirely.
|
||||||
WebPassword string
|
WebPassword string
|
||||||
// UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js.
|
// UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js.
|
||||||
@@ -143,7 +145,7 @@ func loadLatestPoll() LatestPoll {
|
|||||||
func loadConfig() Config {
|
func loadConfig() Config {
|
||||||
c := Config{
|
c := Config{
|
||||||
Token: os.Getenv("API_TOKEN"),
|
Token: os.Getenv("API_TOKEN"),
|
||||||
DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
|
DatabaseURL: os.Getenv("DATABASE_URL"),
|
||||||
Port: envOr("PORT", "8080"),
|
Port: envOr("PORT", "8080"),
|
||||||
WebPassword: os.Getenv("WEB_PASSWORD"),
|
WebPassword: os.Getenv("WEB_PASSWORD"),
|
||||||
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
|
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
|
||||||
@@ -215,8 +217,11 @@ func main() {
|
|||||||
if cfg.Token == "" {
|
if cfg.Token == "" {
|
||||||
log.Fatal("API_TOKEN is required")
|
log.Fatal("API_TOKEN is required")
|
||||||
}
|
}
|
||||||
|
if cfg.DatabaseURL == "" {
|
||||||
|
log.Fatal("DATABASE_URL is required")
|
||||||
|
}
|
||||||
|
|
||||||
s, err := store.Open(cfg.DBPath)
|
s, err := store.Open(cfg.DatabaseURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Fatalf("open store: %v", err)
|
log.Fatalf("open store: %v", err)
|
||||||
}
|
}
|
||||||
@@ -236,7 +241,8 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
log.Printf("listening on :%s (db=%s, origins=%v)", cfg.Port, cfg.DBPath, cfg.AllowedOrigins)
|
// The connection URL carries a password, so it stays out of the log.
|
||||||
|
log.Printf("listening on :%s (origins=%v)", cfg.Port, cfg.AllowedOrigins)
|
||||||
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||||
log.Fatalf("serve: %v", err)
|
log.Fatalf("serve: %v", err)
|
||||||
}
|
}
|
||||||
|
|||||||
+1
-6
@@ -5,7 +5,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
"path/filepath"
|
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -28,11 +27,7 @@ func webConfig() Config {
|
|||||||
// can seed rows and assert on what the handlers wrote back.
|
// can seed rows and assert on what the handlers wrote back.
|
||||||
func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) {
|
func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
st := newTestStore(t)
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("store.Open: %v", err)
|
|
||||||
}
|
|
||||||
t.Cleanup(func() { st.Close() })
|
|
||||||
return newRouter(st, cfg), st
|
return newRouter(st, cfg), st
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -23,13 +23,18 @@ services:
|
|||||||
# isn't an IP or "localhost".
|
# isn't an IP or "localhost".
|
||||||
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
||||||
depends_on:
|
depends_on:
|
||||||
- headless-shell
|
headless-shell:
|
||||||
# `networks:` here replaces the base file's list entirely, so both must be
|
condition: service_started
|
||||||
# named: `proxy` for Traefik routing, `browser` (defined in the base file)
|
postgres:
|
||||||
# to keep reaching headless-shell without putting it on `proxy` too.
|
condition: service_healthy
|
||||||
|
# `networks:` here replaces the base file's list entirely, so all three must
|
||||||
|
# be named: `proxy` for Traefik routing, and `browser` / `db` (defined in
|
||||||
|
# the base file) to keep reaching headless-shell and Postgres without
|
||||||
|
# putting either on `proxy`.
|
||||||
networks:
|
networks:
|
||||||
- proxy
|
- proxy
|
||||||
- browser
|
- browser
|
||||||
|
- db
|
||||||
labels:
|
labels:
|
||||||
- "traefik.enable=true"
|
- "traefik.enable=true"
|
||||||
- "traefik.docker.network=${PROXY_NETWORK:-proxy}"
|
- "traefik.docker.network=${PROXY_NETWORK:-proxy}"
|
||||||
|
|||||||
+38
-4
@@ -16,7 +16,9 @@ services:
|
|||||||
# API_TOKEN is required — compose refuses to start without it.
|
# API_TOKEN is required — compose refuses to start without it.
|
||||||
API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env}
|
API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env}
|
||||||
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
|
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
|
||||||
DB_PATH: /data/bookmarks.db
|
# The bookmarks database. Host is the compose service name; the password
|
||||||
|
# comes from .env so it is never committed.
|
||||||
|
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
|
||||||
PORT: "8080"
|
PORT: "8080"
|
||||||
# Gates the browser UI. Unset means the web routes are not served at all.
|
# Gates the browser UI. Unset means the web routes are not served at all.
|
||||||
WEB_PASSWORD: ${WEB_PASSWORD:-}
|
WEB_PASSWORD: ${WEB_PASSWORD:-}
|
||||||
@@ -42,9 +44,13 @@ services:
|
|||||||
# this URL survives container recreation.
|
# this URL survives container recreation.
|
||||||
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
||||||
depends_on:
|
depends_on:
|
||||||
- headless-shell
|
headless-shell:
|
||||||
|
condition: service_started
|
||||||
|
# The migration runner is the first thing the binary does, so a Postgres
|
||||||
|
# that is still initialising means a crash-loop until it is not.
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
volumes:
|
volumes:
|
||||||
- bookmarks-data:/data
|
|
||||||
# The userscript is served from here, read fresh on every request. Editing
|
# The userscript is served from here, read fresh on every request. Editing
|
||||||
# the file in this checkout takes effect on the next Violentmonkey poll —
|
# the file in this checkout takes effect on the next Violentmonkey poll —
|
||||||
# no rebuild, no restart. `git pull` restores the committed version, which
|
# no rebuild, no restart. `git pull` restores the committed version, which
|
||||||
@@ -56,6 +62,26 @@ services:
|
|||||||
- "127.0.0.1:8080:8080"
|
- "127.0.0.1:8080:8080"
|
||||||
networks:
|
networks:
|
||||||
- browser
|
- browser
|
||||||
|
- db
|
||||||
|
|
||||||
|
postgres:
|
||||||
|
image: postgres:17-alpine
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: bookmarks
|
||||||
|
POSTGRES_USER: bookmarks
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 10
|
||||||
|
volumes:
|
||||||
|
- postgres-data:/var/lib/postgresql/data
|
||||||
|
# Deliberately no `ports:` — only bookmark-api, over the `db` network,
|
||||||
|
# reaches it. Use `docker compose exec postgres psql` for a shell.
|
||||||
|
networks:
|
||||||
|
- db
|
||||||
|
|
||||||
headless-shell:
|
headless-shell:
|
||||||
image: chromedp/headless-shell:stable
|
image: chromedp/headless-shell:stable
|
||||||
@@ -85,7 +111,11 @@ services:
|
|||||||
ipv4_address: 172.28.0.10
|
ipv4_address: 172.28.0.10
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
bookmarks-data:
|
postgres-data:
|
||||||
|
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
|
||||||
|
# declared here: undeclared means `docker compose down -v` cannot take it
|
||||||
|
# with the rest, so the old database survives the cutover until someone
|
||||||
|
# removes it by hand.
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
# Not `internal: true`: headless Chrome still needs outbound access to reach
|
# Not `internal: true`: headless Chrome still needs outbound access to reach
|
||||||
@@ -95,3 +125,7 @@ networks:
|
|||||||
ipam:
|
ipam:
|
||||||
config:
|
config:
|
||||||
- subnet: 172.28.0.0/24
|
- subnet: 172.28.0.0/24
|
||||||
|
# Postgres needs no egress and nothing outside bookmark-api needs to reach
|
||||||
|
# it, so this one really can be cut off from the outside world.
|
||||||
|
db:
|
||||||
|
internal: true
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Postgres replaces SQLite as the primary datastore
|
||||||
|
|
||||||
|
Status: accepted
|
||||||
|
|
||||||
|
The project is moving from a single-reader tracker to a service published to a community,
|
||||||
|
so we replaced `modernc.org/sqlite` with Postgres (`jackc/pgx/v5`, still pure Go, so
|
||||||
|
`CGO_ENABLED=0` and the distroless image are unaffected). The deciding reason is future
|
||||||
|
supportability — managed hosting, a datastore that survives the app outgrowing one box —
|
||||||
|
**not** concurrency, which was measured and found to be a non-issue.
|
||||||
|
|
||||||
|
## Considered options
|
||||||
|
|
||||||
|
**Stay on SQLite.** Benchmarked against the real store at 1,500 rows (≈50 readers × 30
|
||||||
|
series): ~9,700 upserts/sec single-writer, plateauing at ~780/sec under 8–50 concurrent
|
||||||
|
writers, with `List()` holding 90–103 calls/sec under continuous write load. Projected
|
||||||
|
real load at 50 readers is ~0.02 writes/sec — roughly four and a half orders of magnitude
|
||||||
|
of headroom. `SetMaxOpenConns(1)` serialises writes but was shown not to starve reads;
|
||||||
|
an apparent read collapse traced to row count and per-row scanning, not lock contention.
|
||||||
|
SQLite would have worked. It was rejected for where the project is going, not for what
|
||||||
|
it does today.
|
||||||
|
|
||||||
|
**Postgres.** Chosen. Migrating is cheapest now — 29 rows in one table — and gets
|
||||||
|
materially harder once there are live readers and a multi-tenant schema.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Every statement in `internal/store` is rewritten: `?` → `$N`, `IS NOT` →
|
||||||
|
`IS DISTINCT FROM` (this one is load-bearing; it implements the `updated_at`
|
||||||
|
ordering rule), `pragma_table_info` → `information_schema.columns`,
|
||||||
|
`INTEGER`/`REAL` → `bigint`/`double precision`, `favorite` int-as-bool → `boolean`.
|
||||||
|
- ~75 tests currently get a free isolated database from `t.TempDir()`. They now need a
|
||||||
|
live server, which makes Docker a hard prerequisite for `go test ./...`. This is the
|
||||||
|
permanent cost of the decision and the main reason it was close.
|
||||||
|
- Backups get worse, not better: `VACUUM INTO` produced one self-contained file;
|
||||||
|
restoring now means `pg_dump`/`pg_restore`, a role, and a password.
|
||||||
|
- A second stateful container joins the VPS alongside the existing headless-shell.
|
||||||
|
- **Postgres does not address the real scaling limit.** At batch 14 per 10-minute tick
|
||||||
|
the poller checks at most 84 series/hour; 50 readers × 30 series is 1,500 bookmarks,
|
||||||
|
an 18-hour sweep against a configured 1-hour cooldown. That ceiling is an outbound
|
||||||
|
fetch budget and is fixed by deduplicating polls per Series, not by the datastore.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Identity comes from Discord OAuth; we store no passwords and send no email
|
||||||
|
|
||||||
|
Status: accepted
|
||||||
|
|
||||||
|
The service is being published to a community that already lives on Discord, and we have
|
||||||
|
no transactional email infrastructure. Rather than build email verification and password
|
||||||
|
reset to get accounts, Readers sign in with Discord OAuth2 (authorization code grant,
|
||||||
|
`identify` + `guilds.members.read`), and guild membership replaces both the invite gate
|
||||||
|
and the email-verification step. No password is ever stored and no mail is ever sent.
|
||||||
|
|
||||||
|
## Considered options
|
||||||
|
|
||||||
|
**Email + password with invite codes, no verification.** Viable and dependency-free:
|
||||||
|
an invite code proves community membership, which is what email verification was
|
||||||
|
standing in for anyway. Rejected because it still requires password hashing, a manual
|
||||||
|
admin-driven reset path, and a credential store — all of which Discord removes.
|
||||||
|
|
||||||
|
**Email + password with a transactional provider** (Resend, Brevo). Rejected as
|
||||||
|
premature: it builds verification and self-serve reset before anyone has asked for them,
|
||||||
|
and adds deliverability as an operational concern.
|
||||||
|
|
||||||
|
**Discord OAuth.** Chosen. It is less code than either alternative — no hashing, no
|
||||||
|
reset flow, no invite table — and the authorization question ("is this person in my
|
||||||
|
community?") is answered by the same call that answers the authentication question.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- **Availability is now coupled to Discord.** If Discord's OAuth endpoint is down,
|
||||||
|
nobody can start a new session. Existing sessions are unaffected, which bounds the
|
||||||
|
blast radius.
|
||||||
|
- **Identity is a Discord snowflake.** Migrating off Discord later means re-identifying
|
||||||
|
every Reader, because we hold no other credential for them. This is the lock-in the
|
||||||
|
decision buys, and it is the reason this ADR exists.
|
||||||
|
- **`guilds.members.read` is checked at login, not continuously.** Someone who leaves
|
||||||
|
the guild keeps their session until it expires. Acceptable; revocation is a session
|
||||||
|
delete, not an architectural change.
|
||||||
|
- **The userscripts cannot use OAuth.** They run in an isolated world on third-party
|
||||||
|
pages with no redirect surface, so they keep a bearer token — now issued per Reader by
|
||||||
|
the backend rather than a single shared `API_TOKEN` literal. OAuth gates the web UI;
|
||||||
|
the web UI is where a Reader obtains their personal userscript.
|
||||||
|
- `WEB_PASSWORD` disappears, and with it the session HMAC key derivation
|
||||||
|
(`sha256(API_TOKEN | WEB_PASSWORD | …)`), which needs a replacement secret.
|
||||||
|
- Seeding the first Reader during migration requires knowing the owner's Discord user
|
||||||
|
ID up front — a stable snowflake, copied from the Discord client.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Series is a shared entity, and only the Poll may update it
|
||||||
|
|
||||||
|
Status: accepted
|
||||||
|
|
||||||
|
Facts about a Series that are true regardless of who is reading — title, cover, canonical
|
||||||
|
URL, Latest Chapter — moved off the Bookmark onto a shared `series` row keyed
|
||||||
|
`(site, series_id)`. A Bookmark now holds only what differs between Readers: Progress,
|
||||||
|
Favourite, Lifecycle bucket. Fifty Readers tracking one Series produce fifty Bookmarks
|
||||||
|
and one Series, so the Series is polled once rather than fifty times.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
The poller checks at most 84 series/hour (batch 14 per 10-minute tick). With ~50 Readers
|
||||||
|
holding ~30 Series each, polling per Bookmark means a 1,500-item sweep — roughly 18 hours
|
||||||
|
against a configured 1-hour cooldown, quietly breaking the New Chapter signal that is the
|
||||||
|
product's reason to exist. Deduplicating to distinct Series cuts the sweep several-fold,
|
||||||
|
and because the Series row now knows how many Readers hold it, the poll queue is ordered
|
||||||
|
`reader_count DESC, latest_checked_at ASC` — popular Series stay fresh and the long tail
|
||||||
|
absorbs the shortfall. That ordering is only expressible because the split happened.
|
||||||
|
|
||||||
|
Raising throughput instead was rejected: sweeping 400 Series hourly needs the stagger
|
||||||
|
cut from 20s to ~9s, doubling request rate against sites that already bot-score the
|
||||||
|
single VPS IP.
|
||||||
|
|
||||||
|
## Only the Poll writes Series fields
|
||||||
|
|
||||||
|
A client may supply `title`, `cover` and `series_url` only when creating a Series nobody
|
||||||
|
has bookmarked yet. After that, client-supplied values are ignored; only the backend's
|
||||||
|
own fetch updates them.
|
||||||
|
|
||||||
|
This is a security boundary, not tidiness. Those values are scraped from third-party
|
||||||
|
pages, which `AGENTS.md` requires be treated as attacker-controlled. Before the split, a
|
||||||
|
hostile or compromised site could corrupt exactly one Reader's row. After it, the same
|
||||||
|
write lands on a row every Reader sees — one Reader's browser becomes a write path into
|
||||||
|
everyone else's UI, and a cover URL can point anywhere. The backend's own fetch is the
|
||||||
|
higher-trust source: its network, its parser, no third-party JavaScript in the path.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- `Store.Upsert` decomposes one incoming flat body across two tables and enforces the
|
||||||
|
ownership rule at that seam.
|
||||||
|
- The `updated_at` ordering rule stays on the Bookmark, where Progress lives. Unchanged.
|
||||||
|
- Per-Reader title overrides are deliberately not supported; they would reintroduce the
|
||||||
|
duplication this removes.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# The wire format stays flat and deliberately does not mirror the schema
|
||||||
|
|
||||||
|
Status: accepted
|
||||||
|
|
||||||
|
Storage splits a tracked series across two tables (ADR-0003), but `GET /bookmarks` and
|
||||||
|
`PUT /bookmarks/{key}` keep emitting and accepting one **flat** JSON object with `title`,
|
||||||
|
`cover`, `last_chapter` and `latest_chapter` as siblings — exactly the shape they had
|
||||||
|
when there was one table. The server joins on the way out and decomposes on the way in.
|
||||||
|
|
||||||
|
## Why a future reader will find this surprising
|
||||||
|
|
||||||
|
The obvious move after splitting a table is to nest the JSON to match. Don't "fix" this.
|
||||||
|
|
||||||
|
**A nested payload would have broken every installed userscript instantly.** Scripts read
|
||||||
|
`b.title` directly; moving it to `b.series.title` yields `undefined` — no error, just
|
||||||
|
blank rows and a New Chapter signal that silently reports nothing forever. Because
|
||||||
|
Violentmonkey updates roughly once a day per device, the migration relies on old scripts
|
||||||
|
continuing to work during a 14-day grace window. A nested format and that grace window
|
||||||
|
are mutually exclusive.
|
||||||
|
|
||||||
|
**It is also the better contract independently of compatibility.** A client rendering one
|
||||||
|
row needs the title and the reading position together; nesting exports the re-stitching
|
||||||
|
to every browser to mirror a decision about disk layout it should not know about. Keeping
|
||||||
|
them separate lets storage change again later without a client release — which is the
|
||||||
|
whole reason this ADR is worth the paragraph.
|
||||||
|
|
||||||
|
## Consequence
|
||||||
|
|
||||||
|
The flat shape is a contract, not an implementation detail. Changing the storage schema
|
||||||
|
must not change it. It follows the rule already in force for `updated_at`: the server
|
||||||
|
owns the truth and returns the row **as stored**, and clients adopt the response rather
|
||||||
|
than their own payload.
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Domain Docs
|
||||||
|
|
||||||
|
How the engineering skills should consume this repo's domain documentation when exploring the
|
||||||
|
codebase. Layout: **single-context** — one `CONTEXT.md` plus `docs/adr/` at the repo root.
|
||||||
|
|
||||||
|
## Before exploring, read these
|
||||||
|
|
||||||
|
- **`CONTEXT.md`** at the repo root — the glossary / ubiquitous language.
|
||||||
|
- **`docs/adr/`** — read ADRs that touch the area you're about to work in.
|
||||||
|
|
||||||
|
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest
|
||||||
|
creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and
|
||||||
|
`/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
|
||||||
|
|
||||||
|
Neither exists yet in this repo. The existing `AGENTS.md` / `CLAUDE.md` and `docs/design-system.md`
|
||||||
|
carry the current architecture and design law — read those regardless.
|
||||||
|
|
||||||
|
## File structure
|
||||||
|
|
||||||
|
```
|
||||||
|
/
|
||||||
|
├── CONTEXT.md
|
||||||
|
├── docs/adr/
|
||||||
|
│ ├── 0001-....md
|
||||||
|
│ └── 0002-....md
|
||||||
|
├── backend/
|
||||||
|
└── userscript/
|
||||||
|
```
|
||||||
|
|
||||||
|
If this repo ever splits into genuinely separate contexts, add a root `CONTEXT-MAP.md` pointing at
|
||||||
|
one `CONTEXT.md` per context and update this file.
|
||||||
|
|
||||||
|
## Use the glossary's vocabulary
|
||||||
|
|
||||||
|
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a
|
||||||
|
test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary
|
||||||
|
explicitly avoids.
|
||||||
|
|
||||||
|
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing
|
||||||
|
language the project doesn't use (reconsider) or there's a real gap (note it for
|
||||||
|
`/domain-modeling`).
|
||||||
|
|
||||||
|
## Flag ADR conflicts
|
||||||
|
|
||||||
|
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
|
||||||
|
|
||||||
|
> _Contradicts ADR-0002 (…) — but worth reopening because…_
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Issue tracker: Gitea (`tea` CLI)
|
||||||
|
|
||||||
|
Issues and specs for this repo live as issues on the self-hosted Gitea instance
|
||||||
|
`gitea.violetcrown.my.id` (repo `sulthan/mangaBookmark`). **`gh` does not work here** — use
|
||||||
|
[`tea`](https://gitea.com/gitea/tea) for everything past plain git. Auth lives in `tea login`,
|
||||||
|
not a `GH_TOKEN` env var. `tea` infers the repo from the local clone's `origin`.
|
||||||
|
|
||||||
|
`tea` prints rendered boxes rather than plain text; pass `--output json` (or `-o json`) when a
|
||||||
|
skill needs to parse the result.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **Create an issue**: `tea issue create --title "..." --description "..."` (`--labels`,
|
||||||
|
`--assignees` optional). Multi-line bodies: pass the body through a shell variable or heredoc.
|
||||||
|
- **Read an issue**: `tea issue <number> --comments` (add `-o json` for machine-readable output).
|
||||||
|
- **List issues**: `tea issue list --state open -o json --fields index,title,body,labels,state,author`;
|
||||||
|
filter with `--labels "..."`, `--state open|closed|all`, `--assignee`, `--keyword`.
|
||||||
|
- **Comment**: `tea comment <number> "..."` (alias of `tea comments add`).
|
||||||
|
- **Apply / remove labels**: `tea issue edit <number> --add-labels "..."` / `--remove-labels "..."`.
|
||||||
|
Labels must exist first — see `tea labels list` / `tea labels create --name "..." --color "#rrggbb"`.
|
||||||
|
- **Close**: `tea issue close <number>` (comment separately with `tea comment`; `close` takes no
|
||||||
|
`--comment` flag).
|
||||||
|
|
||||||
|
## Pull requests as a triage surface
|
||||||
|
|
||||||
|
**PRs as a request surface: no.** _(Set to `yes` if this repo treats external PRs as feature
|
||||||
|
requests; `/triage` reads this flag.)_
|
||||||
|
|
||||||
|
When set to `yes`, PRs run through the same labels and states as issues, using the `tea pr`
|
||||||
|
equivalents: `tea pr <number> --comments`, `tea pr list --state open -o json`,
|
||||||
|
`tea pr create --head <branch> --base main --title "..." --description "..."`, `tea comment`,
|
||||||
|
`tea pr close`. Gitea shares one index space across issues and PRs, so a bare `#42` may be either
|
||||||
|
— resolve with `tea pr 42` and fall back to `tea issue 42`.
|
||||||
|
|
||||||
|
## When a skill says "publish to the issue tracker"
|
||||||
|
|
||||||
|
Create a Gitea issue with `tea issue create`.
|
||||||
|
|
||||||
|
## When a skill says "fetch the relevant ticket"
|
||||||
|
|
||||||
|
Run `tea issue <number> --comments`.
|
||||||
|
|
||||||
|
## Wayfinding operations
|
||||||
|
|
||||||
|
Used by `/wayfinder`. The **map** is a single issue; **tickets** are child issues.
|
||||||
|
|
||||||
|
- **Map**: one issue labelled `wayfinder:map` holding the Notes / Decisions-so-far / Fog body.
|
||||||
|
`tea issue create --labels wayfinder:map --title "..." --description "..."`.
|
||||||
|
- **Child ticket**: an issue labelled `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`)
|
||||||
|
with `Part of #<map>` as the first body line, and a task-list entry in the map body. `tea` has no
|
||||||
|
sub-issue command, so the task list plus the `Part of` line is the canonical link.
|
||||||
|
- **Blocking**: a `Blocked by: #<n>, #<n>` line at the top of the child body. Gitea's native issue
|
||||||
|
dependencies exist in the API but `tea` does not expose them; the body line is the source of
|
||||||
|
truth. A ticket is unblocked when every listed blocker is closed.
|
||||||
|
- **Frontier query**: `tea issue list --state open -o json` scoped to the map's task list; drop any
|
||||||
|
ticket with an open blocker or an assignee; first in map order wins.
|
||||||
|
- **Claim**: `tea issue edit <n> --add-assignees <your-username>` — the session's first write.
|
||||||
|
(`tea` has no `@me` shorthand; use the Gitea username from `tea login list`.)
|
||||||
|
- **Resolve**: `tea comment <n> "<answer>"`, then `tea issue close <n>`, then append a context
|
||||||
|
pointer to the map's Decisions-so-far via `tea issue edit <map> --description "..."`.
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Triage Labels
|
||||||
|
|
||||||
|
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual
|
||||||
|
label strings used in this repo's issue tracker (Gitea — see `docs/agents/issue-tracker.md`).
|
||||||
|
|
||||||
|
| Label in mattpocock/skills | Label in our tracker | Meaning |
|
||||||
|
| -------------------------- | -------------------- | ---------------------------------------- |
|
||||||
|
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
||||||
|
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
||||||
|
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
|
||||||
|
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
||||||
|
| `wontfix` | `wontfix` | Will not be actioned |
|
||||||
|
|
||||||
|
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label
|
||||||
|
string from this table.
|
||||||
|
|
||||||
|
Gitea will not auto-create labels on `tea issue edit --add-labels`; create a missing one first with
|
||||||
|
`tea labels create --name "<label>" --color "#rrggbb"`.
|
||||||
|
|
||||||
|
Edit the right-hand column to match whatever vocabulary you actually use.
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
Guidance for Claude Code working under `userscript/`. See root `CLAUDE.md` for the project-wide architecture diagram, hard constraints, and design system.
|
|
||||||
|
|
||||||
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
|
||||||
|
|
||||||
1. **Site adapters** — one per host, `detect(location, document)` return page `type` + IDs. Identify type/IDs from **URL regex** (most stable); pull `title`/`cover` from **`og:title`/`og:image` meta tags**, not CSS classes.
|
|
||||||
2. **API client** — `apiGet/apiPut/apiDelete` with bearer header; `localStorage` key `bmgr:manga:cache` for instant render + offline fallback.
|
|
||||||
3. **Progress logic** — auto-upsert `last_chapter` only when `chapterNum >= stored last_chapter_num` (re-reading old chapters must not regress progress; unparseable -> set current). Manual panel override forces any value.
|
|
||||||
4. **Retry queue** — every write go through `pushBookmark`/`pushDelete`, so
|
|
||||||
failed mutation park in `localStorage` (`bmgr:manga:queue`) and replayed on
|
|
||||||
next navigation, reconnect, or `refresh()`. Entries are markers
|
|
||||||
(`{key, op, sendStatus, attempts}`), never payloads — body read from
|
|
||||||
cache at send time, so one entry per key give ordering and coalescing for
|
|
||||||
free. `sendStatus` is **sticky**: while archive pending, later writes to
|
|
||||||
that key keep carrying bucket, which stop successful
|
|
||||||
in-between write from silently un-archiving series. `refresh()` drains
|
|
||||||
before it fetches and overlays anything still pending, so list never
|
|
||||||
flaps. 400 drops entry, 401 abort pass and keep queue, and
|
|
||||||
transient failures retry to cap of 10. Latest-chapter writes deliberately
|
|
||||||
stay out of queue. See
|
|
||||||
`docs/superpowers/specs/2026-07-27-offline-retry-queue-design.md`.
|
|
||||||
5. **UI** — rendered inside **Shadow DOM** root to isolate from site CSS
|
|
||||||
(critical on mobile). Three tabs (All / Favourites / Archived) and row of
|
|
||||||
link chips to web UI and both manga sites; `WEB_BASE` sits in CONFIG
|
|
||||||
block next to `API_BASE`. FAB is `7 × 44` edge tab whose *hit* area
|
|
||||||
widened to `28 × 72` by invisible `#hit` child; `#fab` must keep
|
|
||||||
`touch-action: none` and must **not** regain `overflow: hidden`. Since
|
|
||||||
`touch-action` resolved at gesture start, strip can't be both
|
|
||||||
browser-scrolled and script-dragged, so `makeDraggable` splits by intent: swipe
|
|
||||||
from `#hit` scrolls via `window.scrollBy`, hold of `ARM_MS` arms
|
|
||||||
reposition drag, visible sliver drags with no hold. See
|
|
||||||
`docs/superpowers/specs/2026-07-28-edge-tab-hitbox-design.md`.
|
|
||||||
6. **SPA navigation** — Asura is Astro, client-routed on comic/chapter pages: patch `history.pushState`/`replaceState` + listen `popstate`, re-run `detect()` on URL change so auto-update fire without reload. Demonic uses classic reloads (initial `document-idle` run suffice).
|
|
||||||
|
|
||||||
### Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust)
|
|
||||||
|
|
||||||
- **asurascans.com**: series `/comics/<slug>` (slug carries trailing
|
|
||||||
site-wide build-hash suffix, e.g. `-059befe1`, that **rotates on every
|
|
||||||
redeploy**), chapter `/comics/<slug>/chapter/<n>`. `seriesId` must strip
|
|
||||||
hash (`/-[0-9a-f]{8}$/`, `stripBuildHash` in userscript,
|
|
||||||
`asuraBuildHash` in backend); URLs keep full slug — stale-hash
|
|
||||||
URLs 302 to current ones. Astro-rendered; chapter links present in raw
|
|
||||||
server HTML.
|
|
||||||
- **demonicscans.org**: series `/manga/<slug>` (slug may URL-encode punctuation, e.g. `%2527` for `'`), chapter `/title/<slug>/chapter/<n>/<page>` (older `chaptered.php?manga=<id>&chapter=<n>` form still exists as redirect, what series-page chapter-list anchors link through).
|
|
||||||
Encodings (incl. triple-encoded punctuation like `%25252D`) identical
|
|
||||||
on /manga/ and /title/ pages, so decode-once seriesIds match — verified
|
|
||||||
2026-07-28.
|
|
||||||
- **novelfull.com** (novel script): series `/<slug>.html`, chapter
|
|
||||||
`/<slug>/chapter-<n>[-<title-slug>].html`. No `og:*` tags at all — title from
|
|
||||||
`h3.title` (series) or `a.truyen-title` (chapter), cover from
|
|
||||||
`meta[name="image"]`. Behind a Cloudflare JS challenge no TLS fingerprint
|
|
||||||
clears, so the backend polls it through the headless browser.
|
|
||||||
- **lightnovelworld.net** (novel script): series `/novel/<slug>/`, chapter
|
|
||||||
`/<slug>-chapter-<n>/` — flat, at the site root. `h1.entry-title` is the clean
|
|
||||||
title on a series page and `<Title> Chapter <n>` on a chapter page. Chapter
|
|
||||||
pages carry no `og:image`. Its series page lists every chapter with an
|
|
||||||
absolute href, so the backend polls it with the plain TLS client.
|
|
||||||
|
|
||||||
### Second script: `novel-bookmark.user.js`
|
|
||||||
|
|
||||||
A copy of the manga script with two adapters, `LIBRARY = "novel"` and
|
|
||||||
`STORE_PREFIX = "bmgr:novel:"`. No migration loop (this script has no previous
|
|
||||||
installation to carry keys over from). Installed alongside the manga script;
|
|
||||||
both write to the same backend with the same `LIBRARY` column discriminating
|
|
||||||
them.
|
|
||||||
Symlink
+1
@@ -0,0 +1 @@
|
|||||||
|
AGENTS.md
|
||||||
Reference in New Issue
Block a user