Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3f7664ef9b | |||
| 984965ed9f | |||
| 08749df050 | |||
| b9f9aea82c |
@@ -4,11 +4,23 @@
|
||||
# openssl rand -hex 32
|
||||
API_TOKEN=changeme-generate-a-long-random-token
|
||||
|
||||
# The owner's Discord user ID — the one Reader every bookmark belongs to
|
||||
# (seeded at startup). Discord snowflake, e.g. 1046923170000000000.
|
||||
OWNER_DISCORD_ID=changeme-your-discord-user-id
|
||||
|
||||
# Comma-separated origins allowed to call the API (CORS). Both Asura domains
|
||||
# plus Demonic, Comix, Kagane, and the two novel sites. Add/remove as the
|
||||
# sites' hostnames change.
|
||||
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net
|
||||
|
||||
# Password for the bundled Postgres container, and therefore half of the
|
||||
# DATABASE_URL compose builds for the backend. Generate one:
|
||||
# openssl rand -hex 24
|
||||
POSTGRES_PASSWORD=changeme-generate-a-long-random-password
|
||||
|
||||
# Override only to point the backend at a Postgres compose does not run.
|
||||
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
||||
|
||||
# --- Prod override (Traefik) only ---
|
||||
# Subdomain Traefik routes to this service (required by the prod override).
|
||||
# BOOKMARK_API_HOST=bookmark-api.example.com
|
||||
|
||||
@@ -25,7 +25,7 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free
|
||||
|
||||
```
|
||||
Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache)
|
||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
|
||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
||||
```
|
||||
|
||||
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`.
|
||||
@@ -33,11 +33,11 @@ Backend-specific architecture (packages, endpoints, poller, config env vars) liv
|
||||
## Commands
|
||||
|
||||
Backend (`cd backend`):
|
||||
- Test all: `go test ./...`
|
||||
- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`).
|
||||
- Single test: `go test -run TestName ./...`
|
||||
- Build static binary: `CGO_ENABLED=0 go build`
|
||||
|
||||
Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`).
|
||||
Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`).
|
||||
|
||||
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
||||
|
||||
@@ -79,7 +79,7 @@ Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example
|
||||
|
||||
Go backend:
|
||||
|
||||
- SQL always parameterized (`?`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
||||
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
|
||||
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
|
||||
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
|
||||
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
|
||||
@@ -133,6 +133,22 @@ Test: "competent reader get this from code in few sec?" Yes → skip. Needs deto
|
||||
|
||||
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
||||
|
||||
## Agent skills
|
||||
|
||||
`AGENTS.md` is the single source of truth for agent guidance; every `CLAUDE.md` in this repo is a symlink to the `AGENTS.md` beside it. Edit `AGENTS.md`.
|
||||
|
||||
### Issue tracker
|
||||
|
||||
Issues live as Gitea issues on `gitea.violetcrown.my.id` (`sulthan/mangaBookmark`), driven by the `tea` CLI — not `gh`. See `docs/agents/issue-tracker.md`.
|
||||
|
||||
### Triage labels
|
||||
|
||||
Default five-role vocabulary, label strings unchanged (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
|
||||
|
||||
### Domain docs
|
||||
|
||||
Single-context: one root `CONTEXT.md` plus `docs/adr/`, both created lazily. See `docs/agents/domain.md`.
|
||||
|
||||
## graphify
|
||||
|
||||
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Guidance for Claude Code (claude.ai/code) working in this repo.
|
||||
|
||||
## What this is
|
||||
|
||||
Manga read-progress tracker, user read on **asurascans.com** (current domain; asuracomic.net 301s here) and **demonicscans.org** via **Violentmonkey**. Userscript inject on-page UI (floating button + slide-in panel), sync progress to self-hosted Go backend so bookmarks unify across both sites and devices.
|
||||
|
||||
## Hard constraints (drive design — don't violate)
|
||||
|
||||
Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free where plain web APIs suffice — keeps portability across engines:
|
||||
- **Avoid `GM_*` unless needed.** Prefer page `localStorage` over `GM_setValue`/`GM_getValue`, on-page UI over `GM_registerMenuCommand`, plain `fetch()` over `GM_xmlhttpRequest` for cross-origin.
|
||||
- Cross-origin `fetch()` work **only** against CORS-enabled backend. Manga sites `https://`, so backend **must be HTTPS** (else mixed-content block).
|
||||
- Asura and Demonic are **separate origins with separate `localStorage`** — shared remote store only way to unify bookmarks. Cloud sync required, not optional.
|
||||
- Userscript run in **isolated world**, so embedded API token safe from site's JS.
|
||||
- Cloudflare's block on manga sites **IP-reputation-based, not universal — and not reliably reproducible.** Verified 2026-07-26: plain `curl` from both CGNAT dev machine *and* deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. Contradicts earlier untested assumption CGNAT dev IP blocked; wasn't, at least this date. Treat "does curl work right now" as live, time-varying fact to re-check, not fixed property of machine — Cloudflare's bot scoring can flip previously-clean IP without notice. Backend fetcher still needs graceful-degrade path for when challenged, and adapters should be **verified against live pages** (Playwright MCP, on-device devtools, or direct probe) before finalize, not assumed from single earlier test.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
|
||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
|
||||
```
|
||||
|
||||
Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/CLAUDE.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/CLAUDE.md`.
|
||||
|
||||
## Commands
|
||||
|
||||
Backend (`cd backend`):
|
||||
- Test all: `go test ./...`
|
||||
- Single test: `go test -run TestName ./...`
|
||||
- Build static binary: `CGO_ENABLED=0 go build`
|
||||
|
||||
Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`).
|
||||
|
||||
Smoke test: `curl` endpoints with `Authorization: Bearer <token>`; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200.
|
||||
|
||||
## Forge: Gitea, not GitHub
|
||||
|
||||
`origin` is self-hosted Gitea instance (`gitea.violetcrown.my.id`), so **`gh` don't work here — use `tea` (Gitea CLI) for anything past plain git.** Common ones:
|
||||
|
||||
- Open PR: `tea pr create --head <branch> --base main --title "..." --description "..."`
|
||||
- List / view / check out: `tea pr list`, `tea pr <n>`, `tea pr checkout <n>`
|
||||
- Issues: `tea issue create`, `tea issue list`
|
||||
- Auth lives in `tea login`, not `GH_TOKEN` env var.
|
||||
|
||||
`tea` print output as rendered boxes rather than plain text; PR URL lands on last line.
|
||||
|
||||
## Design system
|
||||
|
||||
Web UI + userscript panel follow **Cinder**, rules in `docs/design-system.md`
|
||||
— source of truth Claude Design project `BookmarkManager Web UI`
|
||||
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`). Read it before touching
|
||||
`backend/internal/web/static/style.css`, `backend/internal/web/templates/*`, or userscript
|
||||
`TEMPLATE`/`CSS`. Core law: **ember means new chapter only** — no other
|
||||
state (busy, error, destruction) may use `--ember`; destruction gets
|
||||
`--danger`. No cards/corners/shadows, one `--measure: 760px` column, tokens
|
||||
only (never hardcode hex outside `:root`), both colour branches touched
|
||||
together. Any move that pulls series out of list (archive/finish/remove)
|
||||
must be confirm-gated via its own `.confirm-row`; only restore fires
|
||||
instantly.
|
||||
|
||||
## Security invariants
|
||||
|
||||
- Auth on `/bookmarks*`: require `Authorization: Bearer <API_TOKEN>`, **constant-time compare**, 401 otherwise.
|
||||
- CORS: reflect `Origin` only when in `ALLOWED_ORIGINS`; allow `GET,PUT,DELETE,OPTIONS` + headers `Authorization,Content-Type`; answer preflight `OPTIONS` with `204`.
|
||||
|
||||
## Comments
|
||||
|
||||
Comment only if code alone can't carry info. Cost per read — must earn spot.
|
||||
|
||||
Write for:
|
||||
- Why not what. Tradeoffs, non-obvious decisions.
|
||||
- Load-bearing detail looking incidental — say so if "simplify" breaks it.
|
||||
- Non-local consequence, invisible from function alone.
|
||||
- Wire format / encoding / interface contract — save callers re-deriving.
|
||||
- Gotcha/workaround, with ref if exists.
|
||||
- Domain/business rule not derivable from code.
|
||||
|
||||
Skip:
|
||||
- Restating code (no `// increment i` above `i++`).
|
||||
- Trivial getter/setter/pass-through.
|
||||
- Banners, dividers, `// helpers`.
|
||||
- Change narration (`// fix bug`, `// as requested`, `// new impl`) — git's job.
|
||||
- Commented-out code — delete.
|
||||
- TODO without concrete action.
|
||||
|
||||
Style: one dense comment over function beats one per line inside. Tight, no worked example unless bug subtle. Wrong comment worse than none — update/delete on change. Default fewer — sparse+high-signal beats comprehensive.
|
||||
|
||||
Test: "competent reader get this from code in few sec?" Yes → skip. Needs detour through another file/spec/git-blame → write it.
|
||||
|
||||
## Relevant skills
|
||||
|
||||
`multi-stage-dockerfile` and `docker-compose-orchestration` for container work (referenced in plan).
|
||||
|
||||
`golang-code-style`, `golang-error-handling`, `golang-performance`, `golang-testing` for backend Go work.
|
||||
|
||||
## graphify
|
||||
|
||||
Project has knowledge graph at graphify-out/ with god nodes, community structure, cross-file relationships.
|
||||
|
||||
Rules:
|
||||
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. Return scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
|
||||
- If graphify-out/wiki/index.md exists, use for broad navigation instead of raw source browsing.
|
||||
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain don't surface enough context.
|
||||
- After modifying code, run `graphify update .` to keep graph current (AST-only, no API cost).
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
# Bookmark Manager
|
||||
|
||||
Read-progress tracker for serialised fiction. A reader browses third-party manga and
|
||||
novel sites; userscripts capture where they got to and sync it to a self-hosted backend,
|
||||
so progress survives across sites and devices.
|
||||
|
||||
## Language
|
||||
|
||||
**Series**:
|
||||
One ongoing work — a manga or a novel — as published by a Site. Identified by its
|
||||
stable slug on that Site, never by its title. A Series exists once and is shared by
|
||||
every Reader who bookmarks it; it owns the facts that are true regardless of who is
|
||||
reading — title, cover, Latest Chapter. A Reader cannot change them; they describe the
|
||||
Series, not anyone's relationship to it.
|
||||
_Avoid_: manga, title, book, comic
|
||||
|
||||
**Site**:
|
||||
One third-party source a Series is published on. A Series on two Sites is two Series.
|
||||
_Avoid_: source, host, provider, domain
|
||||
|
||||
**Reader**:
|
||||
A person with their own Progress. Exactly one per set of credentials, so there is no
|
||||
separate "account" concept to model — the credential belongs to the Reader.
|
||||
_Avoid_: user, account, member, subscriber
|
||||
|
||||
**Bookmark**:
|
||||
One Reader's tracked relationship with one Series, holding only what differs between
|
||||
Readers: Progress, Favourite, Lifecycle bucket. Facts about the Series itself belong
|
||||
to the Series, not here.
|
||||
_Avoid_: entry, item, record, subscription
|
||||
|
||||
**Library**:
|
||||
One of the two halves of the collection — manga or novel — selected by a Bookmark's
|
||||
`kind`. The web UI and the userscripts each address exactly one Library at a time.
|
||||
Not a per-person concept: "everything one person has bookmarked" is a different idea
|
||||
and must not be called a Library.
|
||||
_Avoid_: section, tab, category
|
||||
|
||||
**Progress**:
|
||||
The furthest chapter a reader has actually read in a Series. Only a change in Progress
|
||||
is real activity, so only Progress reorders the list.
|
||||
_Avoid_: position, bookmark (the noun is taken), last read
|
||||
|
||||
**Latest Chapter**:
|
||||
The newest chapter a Site has published for a Series, discovered without the reader
|
||||
present. Distinct from Progress in every way that matters: it is a fact about the Site,
|
||||
not about the reader, and it must never reorder the list.
|
||||
_Avoid_: newest, current chapter, update
|
||||
|
||||
**Poll**:
|
||||
The backend's own check of a Site for a Series's Latest Chapter, made without the
|
||||
Reader present. Performed once per Series no matter how many Readers bookmarked it —
|
||||
a Poll is work done on behalf of the Series, never on behalf of a Reader.
|
||||
_Avoid_: scrape, refresh, check, sync
|
||||
|
||||
**New Chapter**:
|
||||
The state where Latest Chapter is ahead of Progress. The single condition the ember
|
||||
accent is permitted to signal.
|
||||
_Avoid_: unread, update available
|
||||
|
||||
**Lifecycle bucket**:
|
||||
Which of three mutually exclusive states a Bookmark sits in — reading, archived, or
|
||||
finished. A Bookmark is in exactly one. Orthogonal to being a favourite.
|
||||
_Avoid_: state, status (as a domain word), list
|
||||
|
||||
**Favourite**:
|
||||
A reader's manual pin on a Bookmark. Orthogonal to the Lifecycle bucket, and never a
|
||||
reason to reorder the list.
|
||||
_Avoid_: starred, pinned, priority
|
||||
@@ -35,9 +35,22 @@ Edit `.env`:
|
||||
# Required — long random secret, also goes in the userscript.
|
||||
API_TOKEN=<paste output of: openssl rand -hex 32>
|
||||
|
||||
# Required — the owner's Discord user ID. Seeds the one Reader every bookmark
|
||||
# belongs to; the value is the snowflake in your Discord profile (Settings →
|
||||
# Advanced → Developer Mode → right-click your name → Copy User ID).
|
||||
OWNER_DISCORD_ID=<discord user id>
|
||||
|
||||
# CORS allowlist — leave as-is unless a site changes hostname.
|
||||
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to
|
||||
|
||||
# Required — password for the bundled Postgres container. Compose builds the
|
||||
# backend's DATABASE_URL out of it and has no fallback for either.
|
||||
POSTGRES_PASSWORD=<paste output of: openssl rand -hex 24>
|
||||
|
||||
# Leave unset. Only set this to point the backend at a Postgres compose does
|
||||
# not run; it then replaces the URL built from POSTGRES_PASSWORD above.
|
||||
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
||||
|
||||
# Required for the Traefik override. Both have no fallback — compose refuses
|
||||
# to start without them. BOOKMARK_WEB_HOST is required even if you never set
|
||||
# WEB_PASSWORD; see 1b.
|
||||
@@ -50,13 +63,20 @@ BOOKMARK_WEB_HOST=bookmark.violetcrown.my.id
|
||||
# TRAEFIK_CERTRESOLVER=le
|
||||
```
|
||||
|
||||
Generate + insert the token in one line:
|
||||
Generate + insert the two secrets in three lines:
|
||||
|
||||
```bash
|
||||
sed -i "s|^API_TOKEN=.*|API_TOKEN=$(openssl rand -hex 32)|" .env
|
||||
sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 24)|" .env
|
||||
grep -E '^API_TOKEN=' .env # copy this — the userscript needs the same value
|
||||
```
|
||||
|
||||
`POSTGRES_PASSWORD` is read **only while the `postgres-data` volume is empty**,
|
||||
which in practice means at first boot. Changing it afterwards changes the URL
|
||||
the backend dials but not the password the database expects, and `bookmark-api`
|
||||
crash-loops on `password authentication failed`. Set it before §2 and leave it
|
||||
alone.
|
||||
|
||||
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
|
||||
> names (check your Traefik static config — common alternatives: `https`,
|
||||
> `myresolver`, `cloudflare`). Wrong names = no certificate issued.
|
||||
@@ -116,16 +136,21 @@ This merges the base file (build/image/env/volume) with the prod override
|
||||
(no host port, Traefik network + router labels). Always pass **both** `-f`
|
||||
flags — the prod file is not standalone.
|
||||
|
||||
Two services come up: `bookmark-api` (the backend) and `headless-shell`, a CDP
|
||||
sidecar the poller uses to fetch kagane (behind a Cloudflare JS challenge).
|
||||
It has no published port — only `bookmark-api` can reach it, over
|
||||
`BROWSER_WS_URL`. Missing or unreachable, the poller just skips kagane and
|
||||
logs it; nothing else is affected.
|
||||
Three services come up: `bookmark-api` (the backend), `postgres` (its database,
|
||||
`postgres:17-alpine`), and `headless-shell`, a CDP sidecar the poller uses to
|
||||
fetch kagane (behind a Cloudflare JS challenge). Neither of the latter two
|
||||
publishes a port: `postgres` sits alone with `bookmark-api` on an
|
||||
`internal: true` network, and `headless-shell` is reachable only over
|
||||
`BROWSER_WS_URL`. A missing headless-shell just makes the poller skip kagane and
|
||||
log it. A missing Postgres stops everything — `bookmark-api` waits for
|
||||
`pg_isready` to pass, then applies its embedded migrations, and only then
|
||||
listens. The schema is created that way; there is nothing to import by hand.
|
||||
|
||||
Check it's up and healthy:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
|
||||
# bookmark-api Up; postgres Up (healthy)
|
||||
docker logs bookmark-api --tail 20 # expect: "listening on :8080 ..."
|
||||
```
|
||||
|
||||
@@ -214,7 +239,10 @@ Pull new code, then rebuild:
|
||||
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
|
||||
```
|
||||
|
||||
SQLite data persists in the named volume `bookmarks-data` across rebuilds.
|
||||
Data persists in the named volume `postgres-data` across rebuilds. (If this
|
||||
server predates the Postgres migration, the old SQLite volume `bookmarks-data`
|
||||
is still on disk and deliberately undeclared in compose so `down -v` cannot take
|
||||
it; see `REDEPLOY.md` §1 for when to remove it.)
|
||||
|
||||
---
|
||||
|
||||
@@ -227,7 +255,9 @@ SQLite data persists in the named volume `bookmarks-data` across rebuilds.
|
||||
| `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). |
|
||||
| 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. |
|
||||
| Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. |
|
||||
| `compose ... config` errors about `API_TOKEN` | Run compose from the dir with `.env`, or export the vars. |
|
||||
| `compose ... config` errors about `API_TOKEN`, `OWNER_DISCORD_ID` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. All three are required and none has a fallback. |
|
||||
| `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). |
|
||||
| `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. |
|
||||
|
||||
Backend config reference and endpoint list: see `README.md`.
|
||||
|
||||
|
||||
@@ -7,14 +7,14 @@ all four sites and all devices.
|
||||
|
||||
Two parts:
|
||||
|
||||
- **`backend/`** — tiny Go (`net/http` + pure-Go SQLite) sync service. 4 routes,
|
||||
- **`backend/`** — tiny Go (`net/http` + Postgres via pure-Go `pgx`) sync service. 4 routes,
|
||||
static binary, distroless container.
|
||||
- **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript
|
||||
(no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`.
|
||||
|
||||
```
|
||||
Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|
||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume)
|
||||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -26,8 +26,9 @@ Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|
||||
| Var | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `API_TOKEN` | *(required)* | Bearer token shared with the userscript. |
|
||||
| `OWNER_DISCORD_ID` | *(required)* | Discord user ID of the owner; seeds the one Reader all bookmarks belong to. |
|
||||
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
|
||||
| `DB_PATH` | `/data/bookmarks.db` | SQLite file location. |
|
||||
| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. |
|
||||
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
|
||||
| `BROWSER_WS_URL` | `ws://172.28.0.10:9222` | Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header. |
|
||||
|
||||
@@ -60,11 +61,17 @@ go test ./... # unit + handler tests
|
||||
CGO_ENABLED=0 go build # static binary
|
||||
```
|
||||
|
||||
**`go test ./...` requires Docker.** The store talks to a real Postgres, so
|
||||
each test package starts a throwaway `postgres:17-alpine` container and gives
|
||||
every test its own database inside it (`internal/pgtest`). Nothing is stubbed
|
||||
and nothing reaches the network beyond the local Docker daemon.
|
||||
|
||||
### Run the stack
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# edit .env: set API_TOKEN (openssl rand -hex 32)
|
||||
# edit .env: set API_TOKEN (openssl rand -hex 32) and
|
||||
# POSTGRES_PASSWORD (openssl rand -hex 24)
|
||||
|
||||
docker compose up -d --build # binds 127.0.0.1:8080
|
||||
```
|
||||
|
||||
+136
-80
@@ -18,7 +18,7 @@ the checkout cannot take the backups with them.
|
||||
```
|
||||
/opt/
|
||||
├── bookmarkmanager/ <- the checkout (this repo)
|
||||
└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.db
|
||||
└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.dump
|
||||
```
|
||||
|
||||
---
|
||||
@@ -53,78 +53,98 @@ echo "$BACKUP_DIR" # -> /opt/bookmarkmanager-backups
|
||||
|
||||
## 1. Back up the database
|
||||
|
||||
The database is a single SQLite file in the named Docker volume, at
|
||||
`/data/bookmarks.db` inside the container. Find the volume's real name — Compose
|
||||
prefixes it with the project directory:
|
||||
The database is Postgres, running as the `postgres` service on the named volume
|
||||
`postgres-data`. It has **no published port** — nothing outside the internal `db`
|
||||
network can reach it — so every command below goes in through the container:
|
||||
|
||||
```bash
|
||||
docker volume ls --filter name=bookmarks-data
|
||||
# -> local bookmarkmanager_bookmarks-data
|
||||
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
|
||||
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
|
||||
# -> bookmarks, schema_migrations, series
|
||||
```
|
||||
|
||||
### Preferred: hot backup, no downtime
|
||||
Inside the container that connects over the local socket as the `bookmarks`
|
||||
superuser, so no password is needed anywhere in this section. `-T` is not
|
||||
optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and
|
||||
silently corrupts any binary stream flowing back out — see the dump below.
|
||||
|
||||
The store runs in **WAL mode**, so recent writes may still be sitting in
|
||||
`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can
|
||||
therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and
|
||||
writes one consistent file, safe to run against a live database:
|
||||
### Preferred: hot dump, no downtime
|
||||
|
||||
`pg_dump` runs in a single repeatable-read transaction, so it writes one
|
||||
point-in-time-consistent snapshot while the API keeps serving. No stopping, no
|
||||
WAL to worry about — that is the server's problem, not yours.
|
||||
|
||||
```bash
|
||||
STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text
|
||||
docker run --rm \
|
||||
-v "$VOL":/data \
|
||||
-v "$BACKUP_DIR":/backup \
|
||||
alpine sh -c "apk add -q sqlite &&
|
||||
sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\""
|
||||
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
|
||||
> "$BACKUP_DIR/bookmarks-$STAMP.dump"
|
||||
|
||||
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db
|
||||
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump
|
||||
```
|
||||
|
||||
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the
|
||||
files sort in real order and never collide across a DST shift.
|
||||
`-Fc` is the custom archive format rather than plain SQL: it is compressed, and
|
||||
`pg_restore` can inspect and replay it selectively — list its table of contents,
|
||||
restore one table, restore schema without data, reorder. A plain `.sql` dump can
|
||||
only be piped into `psql` whole, and gives you no way to check what is in it
|
||||
short of reading it.
|
||||
|
||||
Note the source volume is mounted **read-write**, which looks wrong for a backup
|
||||
and is not. Opening a WAL database requires creating the `-shm` shared-memory
|
||||
file; with `:ro` the command fails with `unable to open database file` and no
|
||||
backup is produced. `VACUUM INTO` never writes to the source itself.
|
||||
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.dump`. UTC, so
|
||||
the files sort in real order and never collide across a DST shift.
|
||||
|
||||
Verify it before you trust it. An unreadable backup is worse than none, because
|
||||
you will act as though you have one:
|
||||
|
||||
```bash
|
||||
docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite &&
|
||||
sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' &&
|
||||
sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'"
|
||||
# -> ok
|
||||
# 1. The dump parses and contains the tables. Uses the same image compose
|
||||
# already pulls, so nothing new to install.
|
||||
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
|
||||
pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
|
||||
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
|
||||
# -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks
|
||||
# -> 1236; 0 0 TABLE DATA public series series
|
||||
|
||||
# 2. Sanity-check the live row count you just captured.
|
||||
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
|
||||
-c 'select count(*) from bookmarks'
|
||||
# -> 37
|
||||
```
|
||||
|
||||
The count should match what the web UI shows. Zero rows on a server you know has
|
||||
bookmarks means you backed up the wrong volume.
|
||||
A custom-format archive stores row counts nowhere, so step 1 proves the file is
|
||||
a readable archive with the right tables in it, not that the rows are there;
|
||||
step 2 is the number those rows should be. It should match what the web UI
|
||||
shows. Zero on a server you know has bookmarks means the API and your `psql`
|
||||
are looking at different databases — check `DATABASE_URL`.
|
||||
|
||||
### Fallback: cold copy (no network for `apk add sqlite`)
|
||||
### Fallback: cold volume archive
|
||||
|
||||
Stop the service first, then copy the database **and its sidecars** — the `-wal`
|
||||
is not optional, it is where the newest writes are:
|
||||
Use this when you want the whole data directory rather than a logical dump — a
|
||||
like-for-like restore of the same Postgres major version onto the same host.
|
||||
|
||||
**The stack must be stopped first.** A running Postgres has dirty pages in
|
||||
shared buffers and WAL that has not been replayed into the data files, and `tar`
|
||||
walks the directory over several seconds while the server keeps writing to it.
|
||||
The archive you get is torn: files from different instants, possibly a
|
||||
half-written page. It may restore, start, and be quietly wrong. Online
|
||||
filesystem-level backup is `pg_basebackup`'s job, not `tar`'s; with the
|
||||
container stopped the shutdown checkpoint has already flushed everything and a
|
||||
plain archive of the volume is consistent.
|
||||
|
||||
```bash
|
||||
VOL=$(docker volume ls --filter name=postgres-data -q | head -1)
|
||||
echo "$VOL" # -> bookmarkmanager_postgres-data
|
||||
|
||||
$COMPOSE stop
|
||||
docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c "
|
||||
cp /data/bookmarks.db /backup/bookmarks-$STAMP.db
|
||||
[ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal
|
||||
[ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm
|
||||
ls -1 /backup"
|
||||
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \
|
||||
tar czf "/to/postgres-data-$STAMP.tgz" -C /from .
|
||||
$COMPOSE start
|
||||
|
||||
ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz
|
||||
```
|
||||
|
||||
Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away,
|
||||
so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for
|
||||
the case where it did not. Restoring this variant means putting whichever files
|
||||
you got back together, under their original names.
|
||||
|
||||
Read-only is safe here precisely because nothing opens the database: it is a file
|
||||
copy, not a SQLite connection.
|
||||
Costs ~15 seconds of downtime. Read-only on the source is safe here precisely
|
||||
because nothing is running against it. Restoring this variant means untarring it
|
||||
back into an *empty* `postgres-data` volume with the stack down — it is a whole
|
||||
data directory, not a file you can drop next to the live one, and it will only
|
||||
start under `postgres:17`.
|
||||
|
||||
### Retention
|
||||
|
||||
@@ -132,7 +152,19 @@ Keep a month, drop the rest — a bookmark database this small compresses the
|
||||
decision to "disk is free, but not infinite":
|
||||
|
||||
```bash
|
||||
ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v
|
||||
ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v
|
||||
```
|
||||
|
||||
### A note on the old `bookmarks-data` volume
|
||||
|
||||
`bookmarks-data` is the **pre-migration SQLite volume**. It is deliberately not
|
||||
declared in `docker-compose.yml` any more, which is what keeps `docker compose
|
||||
down -v` from taking it with the rest of the stack. It is not the live database
|
||||
and nothing reads it. Once the Postgres data has been trusted for a while,
|
||||
remove it by hand — nothing else will:
|
||||
|
||||
```bash
|
||||
docker volume rm bookmarkmanager_bookmarks-data
|
||||
```
|
||||
|
||||
---
|
||||
@@ -171,11 +203,15 @@ rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
|
||||
bindmounted read-only and read fresh per request.
|
||||
|
||||
```bash
|
||||
$COMPOSE ps # Up, and recently (re)created
|
||||
$COMPOSE ps # bookmark-api Up; postgres Up (healthy)
|
||||
docker logs bookmark-api --tail 20 # -> "listening on :8080 ..."
|
||||
```
|
||||
|
||||
Nothing in the log about the database or the poller failing. The image is tagged
|
||||
Nothing in the log about the database, the migrations or the poller failing.
|
||||
`bookmark-api` waits on `postgres` reporting healthy before it starts and the
|
||||
binary applies any pending migration before it listens, so an API that never
|
||||
says "listening" is usually the database, not the code — `$COMPOSE logs
|
||||
postgres` first. The image is tagged
|
||||
`bookmarkmanager-backend:latest`, so the previous image is still on disk untagged —
|
||||
that is what makes the rollback in §6 quick.
|
||||
|
||||
@@ -199,9 +235,16 @@ curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
|
||||
$API/bookmarks/x | grep -i access-control # -> allow-origin echoed
|
||||
```
|
||||
|
||||
`[]` from the third call is the alarm that matters: the volume is not attached
|
||||
and you are looking at an empty database. Stop and check `$COMPOSE config
|
||||
--volumes` before touching anything else.
|
||||
`[]` from the third call is the alarm that matters: you are talking to an empty
|
||||
database, which means the API found a *different* Postgres than the one holding
|
||||
your data — a renamed project directory, a fresh `postgres-data`, or a
|
||||
`DATABASE_URL` override in `.env` pointing elsewhere. Stop and check, before
|
||||
touching anything else:
|
||||
|
||||
```bash
|
||||
$COMPOSE config --volumes # -> postgres-data
|
||||
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks'
|
||||
```
|
||||
|
||||
Web UI and its assets:
|
||||
|
||||
@@ -267,33 +310,41 @@ git checkout <previous-hash>
|
||||
$COMPOSE up -d --build
|
||||
```
|
||||
|
||||
**Database damaged** — restore the backup from §1. Stop first: the running
|
||||
process holds the WAL, and dropping a file under a live SQLite connection
|
||||
corrupts what you were trying to save.
|
||||
**Database damaged** — restore the dump from §1. Stop **only the API**, not the
|
||||
whole stack: `pg_restore` needs the server up to restore into, and it needs
|
||||
`bookmark-api`'s connection pool gone, because `--clean` cannot drop a table
|
||||
other sessions are holding open.
|
||||
|
||||
```bash
|
||||
$COMPOSE stop
|
||||
$COMPOSE stop bookmark-api
|
||||
|
||||
docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c '
|
||||
rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm &&
|
||||
cp /backup/bookmarks-<STAMP>.db /data/bookmarks.db &&
|
||||
chown 65532:65532 /data/bookmarks.db &&
|
||||
ls -l /data'
|
||||
$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \
|
||||
< "$BACKUP_DIR/bookmarks-<STAMP>.dump"
|
||||
|
||||
$COMPOSE start
|
||||
$COMPOSE start bookmark-api
|
||||
docker logs bookmark-api --tail 20
|
||||
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
|
||||
```
|
||||
|
||||
Two steps here are easy to skip and both bite:
|
||||
Three things here are easy to skip and all three bite:
|
||||
|
||||
- **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database
|
||||
mixes two different histories; SQLite will either refuse to open it or quietly
|
||||
reapply writes you meant to discard.
|
||||
- **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as
|
||||
that uid, while the helper container above writes as root. A root-owned
|
||||
database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then
|
||||
every write fails. That is the worst possible failure mode — it looks restored.
|
||||
- **`--clean --if-exists`.** Without `--clean` the dump's rows land *on top of*
|
||||
what is already there and you get primary-key collisions half way through, a
|
||||
partially restored database, and a non-zero exit you may not notice.
|
||||
`--if-exists` only suppresses the "does not exist" noise when the target is
|
||||
already empty; it is not the part doing the work.
|
||||
- **`-T` again.** Feeding a custom-format archive into a TTY-allocated `exec`
|
||||
corrupts it in flight and `pg_restore` fails with a garbled-header error on a
|
||||
file that is perfectly fine on disk.
|
||||
- **Stop the API, not Postgres.** `$COMPOSE stop` (everything) leaves you with
|
||||
nothing to restore into; leaving `bookmark-api` running leaves connections
|
||||
that block the drops *and* lets the poller write into a half-restored table.
|
||||
|
||||
No ownership fixing is needed any more — the Postgres image owns `postgres-data`
|
||||
itself and `pg_restore` writes through the server, not the filesystem.
|
||||
`schema_migrations` is inside the dump, so the database comes back at whatever
|
||||
schema version the backup was taken at; the migration runner applies anything
|
||||
newer the next time `bookmark-api` starts.
|
||||
|
||||
---
|
||||
|
||||
@@ -305,21 +356,24 @@ For a routine redeploy where nothing needs deciding:
|
||||
cd /opt/bookmarkmanager
|
||||
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
|
||||
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
|
||||
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
|
||||
STAMP=$(date -u +%Y%m%d-%H%M%S)
|
||||
|
||||
docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \
|
||||
"apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" &&
|
||||
sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" &&
|
||||
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
|
||||
> "$BACKUP_DIR/bookmarks-$STAMP.dump" &&
|
||||
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
|
||||
pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null &&
|
||||
git pull --ff-only &&
|
||||
$COMPOSE up -d --build &&
|
||||
sleep 5 &&
|
||||
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"
|
||||
```
|
||||
|
||||
The `&&` chain is deliberate: if the backup or its integrity check fails,
|
||||
nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell
|
||||
command can tell you the panel works on the phone.
|
||||
The `&&` chain is deliberate: if the dump or its `pg_restore --list` check
|
||||
fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a
|
||||
short or empty `.dump` behind — the shell creates the file before `pg_dump`
|
||||
runs — so delete it rather than letting it sit in the backup directory looking
|
||||
like a backup. Then still do §5 by hand — no shell command can tell you the
|
||||
panel works on the phone.
|
||||
|
||||
---
|
||||
|
||||
@@ -327,16 +381,18 @@ command can tell you the panel works on the phone.
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
|---|---|
|
||||
| `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. |
|
||||
| `/bookmarks` returns `[]` after redeploy | You are on an empty Postgres. Check `$COMPOSE config --volumes` lists `postgres-data`, that you passed both `-f` files, and that `.env` has no stray `DATABASE_URL` override. Do **not** re-bookmark; the data is still in the volume. |
|
||||
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
|
||||
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
|
||||
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
|
||||
| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
|
||||
| `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
|
||||
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
|
||||
| `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. |
|
||||
| Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). |
|
||||
| Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). |
|
||||
| `bookmark-api` crash-loops, log says `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` in `.env` no longer matches the one burned into `postgres-data` at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: `$COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks'` (prompts, so nothing lands in shell history) and then match `.env` to it. |
|
||||
| `compose` errors `set POSTGRES_PASSWORD in .env` | Unset. Compose builds the backend's `DATABASE_URL` out of it, so it is required even though you never write that URL yourself. Run from the directory holding `.env`. |
|
||||
| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. |
|
||||
| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. |
|
||||
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
|
||||
|
||||
Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
|
||||
UI conventions: `docs/design-system.md`.
|
||||
|
||||
+44
-14
@@ -1,8 +1,8 @@
|
||||
Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system.
|
||||
|
||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
||||
Single binary, split into packages under `backend/internal/`: `store`
|
||||
(Bookmark type, SQLite persistence, migrations), `latest` (background
|
||||
(Bookmark type, Postgres persistence, migration runner), `latest` (background
|
||||
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
||||
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
||||
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
||||
@@ -11,7 +11,32 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
||||
integration tests that exercise the full router; unit tests for a
|
||||
package live beside it under `internal/`.
|
||||
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`), with a `kind` column (`manga`|`novel`) splitting the two libraries. Sync **last-write-wins**. Schema and endpoint list in plan.
|
||||
- **Schema is migration-owned.** `internal/store/migrations/*.sql` is
|
||||
`go:embed`-ed and applied on every start by `store.migrate`: one numbered
|
||||
file per change, one transaction each, versions recorded in
|
||||
`schema_migrations`. Files are **append-only** — editing an applied one
|
||||
changes nothing on a database that already ran it. No column probing, no
|
||||
data-fixup migrations: both were SQLite-era machinery and are gone.
|
||||
- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine`
|
||||
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
|
||||
its own database (`pgtest.URL(t)`). A package whose tests touch the store
|
||||
must have that `TestMain`.
|
||||
- **Single-owner store, three tables.** `readers` is keyed by Discord user ID
|
||||
and carries the SHA-256 of the owner's userscript token (the global
|
||||
`API_TOKEN` today; issue #22). The seed creates exactly one row at startup.
|
||||
`series` keyed `(site, series_id)`
|
||||
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
|
||||
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
|
||||
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
|
||||
differs between readers: progress, favourite, lifecycle bucket,
|
||||
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
|
||||
surrogate id; the wire `key` is derived as `site:series_id` on read — and
|
||||
every store read/write is scoped to the reader it names. `Store.OwnerID()`
|
||||
is the seeded owner, which every handler passes while the global token is
|
||||
still the only credential. Sync **last-write-wins**; the wire format stays
|
||||
flat (ADR-0004). `Store.Upsert` decomposes one flat body across two tables
|
||||
and enforces the ownership rule: client `title`/`series_url`/`cover` are
|
||||
written only when the series row is new (ADR-0003).
|
||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
||||
- **Web UI:** same binary serve password-gated browser UI on second
|
||||
hostname — `GET /` (list, or login page when no session),
|
||||
@@ -43,22 +68,25 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
network access, so `latest_chapter` stay fresh when user not
|
||||
browsing. Second, parallel signal — userscript keep own
|
||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
||||
Two independent clocks: per-bookmark cooldown (`latest_checked_at` column,
|
||||
Two independent clocks: per-series cooldown (`series.latest_checked_at`,
|
||||
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
||||
Row stamped *before* fetch so broken series wait out full
|
||||
cooldown instead of retrying every tick, and writes go through
|
||||
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
|
||||
The poller walks **Series, not Bookmarks** — a series referenced by several
|
||||
bookmarks is fetched once per cycle, and the due queue orders
|
||||
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
||||
*before* fetch so broken series wait out full cooldown instead of retrying
|
||||
every tick; found chapter written straight to the series row via
|
||||
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
||||
order — is never touched.
|
||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||
against fingerprint-based blocking; any failure log and skip. kagane and
|
||||
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
|
||||
clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and
|
||||
simply not polled when that's unset. See
|
||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
|
||||
userscript `PUT` that commits between the two can get overwritten by
|
||||
poller's stale re-read — reverting that read progress and, since stored
|
||||
value now differs, moving `updated_at` and reordering list. Known,
|
||||
accepted limitation for single-user deployment, not bug to fix.
|
||||
The poller's series write is a single-column UPDATE
|
||||
(`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
|
||||
it cannot revert read progress or move `updated_at`, so the old
|
||||
stale-re-read race is gone with the Get+Upsert flow.
|
||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
||||
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
||||
@@ -70,8 +98,10 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
`excluded.*` is post-evaluation row and default applied there would
|
||||
wipe bucket on every PUT from client that predates column. See
|
||||
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
||||
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
|
||||
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
|
||||
- **Config via env:** `API_TOKEN`, `OWNER_DISCORD_ID` (seeds the owner Reader;
|
||||
required), `ALLOWED_ORIGINS` (comma list),
|
||||
`DATABASE_URL` (Postgres connection URL, required — no default),
|
||||
`PORT` (default `8080`), `WEB_PASSWORD`
|
||||
(gates browser UI; unset disable it),
|
||||
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
|
||||
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
Guidance for Claude Code working under `backend/`. See root `CLAUDE.md` for the project-wide architecture diagram, hard constraints, and design system.
|
||||
|
||||
- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`.
|
||||
Single binary, split into packages under `backend/internal/`: `store`
|
||||
(Bookmark type, SQLite persistence, migrations), `latest` (background
|
||||
poller, site parsers, TLS fetcher), `session` (cookie signing, login
|
||||
rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON
|
||||
bookmark handlers), `userscript` (userscript-serving handler), `web`
|
||||
(browser UI handler + `templates/` + `static/`, `go:embed`-ed).
|
||||
`backend/main.go` is the composition root — the only place that wires
|
||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
||||
integration tests that exercise the full router; unit tests for a
|
||||
package live beside it under `internal/`.
|
||||
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`). Sync **last-write-wins**. Schema and endpoint list in plan.
|
||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
||||
- **Web UI:** same binary serve password-gated browser UI on second
|
||||
hostname — `GET /` (list, or login page when no session),
|
||||
`POST /login`, `POST /logout`, `GET /static/*`, htmx fragment endpoints
|
||||
under `/ui/*`. Templates + assets `go:embed`-ed under
|
||||
`backend/internal/web/`, so `backend/Dockerfile` must copy the whole
|
||||
`internal/` tree, not just `*.go`. Sessions stateless
|
||||
HMAC cookies keyed off `API_TOKEN`; `WEB_PASSWORD` gates them, and when empty,
|
||||
web routes not registered at all. UI mutations read-modify-write
|
||||
through `Store.Get` + `Store.Upsert` so `updated_at` rule stays one
|
||||
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
||||
**Design-tool caveat:** templates link `/static/style.css` root-absolutely
|
||||
(correct — served from `/`), but impeccable detector resolves
|
||||
stylesheet href with `path.resolve(fileDir, href)`, drops directory
|
||||
on leading `/` and silently skip file. Relative href don't help
|
||||
either: template's directory isn't its served path. So
|
||||
`detect.mjs backend/internal/web/templates` reports **false clean** —
|
||||
always pass `backend/internal/web/static` too. One finding there,
|
||||
`overused-font` on "Instrument Serif", deliberate identity choice, not debt.
|
||||
- **Every action that moves series out of list is confirm-gated.**
|
||||
Archive, finish, remove each open own `.confirm-row` disclosure
|
||||
(`toggleConfirmRow(key, kind)` in `filter.js`, `kind` ∈
|
||||
`archive|finish|remove`); restore fire instantly since it's the reversal.
|
||||
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
||||
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
||||
error use `--mute`.
|
||||
- **Latest-chapter poller:** ticker goroutine in same binary re-check
|
||||
each bookmarked series' newest published chapter from backend's own
|
||||
network access, so `latest_chapter` stay fresh when user not
|
||||
browsing. Second, parallel signal — userscript keep own
|
||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
||||
Two independent clocks: per-bookmark cooldown (`latest_checked_at` column,
|
||||
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
||||
Row stamped *before* fetch so broken series wait out full
|
||||
cooldown instead of retrying every tick, and writes go through
|
||||
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
|
||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||
against fingerprint-based blocking; any failure log and skip. kagane sits
|
||||
behind a Cloudflare JavaScript challenge the TLS client can't clear, so it is
|
||||
browser-only: fetched over CDP via `BROWSER_WS_URL`, and simply not polled
|
||||
when that's unset. See
|
||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
|
||||
userscript `PUT` that commits between the two can get overwritten by
|
||||
poller's stale re-read — reverting that read progress and, since stored
|
||||
value now differs, moving `updated_at` and reordering list. Known,
|
||||
accepted limitation for single-user deployment, not bug to fix.
|
||||
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
|
||||
`finished`, orthogonal to `favorite`. Archived and finished appear only in
|
||||
own tab — not in All, Updated, Favourites, or recent strip. Poller keeps
|
||||
checking archived series and skip finished ones. `finished` settable
|
||||
only from web UI; `PUT /bookmarks/{key}` reject it with 400.
|
||||
**Empty incoming status means "keep stored one"** — resolved on the
|
||||
`VALUES` side of `Store.Upsert`, not conflict clause, since
|
||||
`excluded.*` is post-evaluation row and default applied there would
|
||||
wipe bucket on every PUT from client that predates column. See
|
||||
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
|
||||
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
|
||||
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
|
||||
(gates browser UI; unset disable it),
|
||||
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
|
||||
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
||||
`USERSCRIPT_PATH` (file served at `/u/{token}/manga-bookmark.user.js`,
|
||||
default `/userscript/manga-bookmark.user.js`, supplied by bindmount).
|
||||
`BROWSER_WS_URL` (headless-shell CDP endpoint for kagane; unset disables
|
||||
browser polling and leaves that site to the userscript alone).
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
+3
-7
@@ -14,21 +14,17 @@ RUN go mod download
|
||||
COPY *.go ./
|
||||
COPY internal/ ./internal/
|
||||
|
||||
# Static binary: pure-Go sqlite means CGO_ENABLED=0 -> no libc dependency.
|
||||
# Static binary: the Postgres driver (jackc/pgx) is pure Go, so CGO_ENABLED=0
|
||||
# leaves no libc dependency.
|
||||
# -trimpath + -ldflags strip paths and debug info for a smaller image.
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server .
|
||||
|
||||
# Data dir with the runtime user's ownership so the mounted volume inherits it.
|
||||
RUN mkdir -p /out/data
|
||||
|
||||
# --- runtime stage: distroless static, non-root ---
|
||||
FROM gcr.io/distroless/static:nonroot
|
||||
WORKDIR /
|
||||
COPY --from=build /out/server /server
|
||||
COPY --from=build --chown=65532:65532 /out/data /data
|
||||
|
||||
VOLUME ["/data"]
|
||||
EXPOSE 8080
|
||||
USER nonroot:nonroot
|
||||
ENV DB_PATH=/data/bookmarks.db PORT=8080
|
||||
ENV PORT=8080
|
||||
ENTRYPOINT ["/server"]
|
||||
|
||||
+150
-26
@@ -2,6 +2,7 @@ package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
@@ -12,6 +13,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"bookmarkmanager/backend/internal/pgtest"
|
||||
"bookmarkmanager/backend/internal/store"
|
||||
)
|
||||
|
||||
@@ -25,15 +27,23 @@ func testConfig() Config {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||
|
||||
func newTestServer(t *testing.T) http.Handler {
|
||||
t.Helper()
|
||||
dbPath := filepath.Join(t.TempDir(), "test.db")
|
||||
s, err := store.Open(dbPath)
|
||||
return newRouter(newTestStore(t), testConfig())
|
||||
}
|
||||
|
||||
func newTestStore(t *testing.T) *store.Store {
|
||||
t.Helper()
|
||||
s, err := store.Open(pgtest.URL(t), store.Owner{
|
||||
DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("owner-token-hash")),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("store.Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
return newRouter(s, testConfig())
|
||||
return s
|
||||
}
|
||||
|
||||
func auth(req *http.Request) *http.Request {
|
||||
@@ -43,26 +53,35 @@ func auth(req *http.Request) *http.Request {
|
||||
|
||||
func floatPtr(f float64) *float64 { return &f }
|
||||
|
||||
// seedForCheck inserts a bookmark and forces its latest_checked_at.
|
||||
// seedForCheck inserts a bookmark (and with it its series) and forces the
|
||||
// series' latest_checked_at.
|
||||
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
|
||||
t.Helper()
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
site, seriesID, ok := strings.Cut(key, ":")
|
||||
if !ok {
|
||||
t.Fatalf("key %q: no ':' separator", key)
|
||||
}
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key,
|
||||
Site: "asura",
|
||||
SeriesID: key,
|
||||
Site: site,
|
||||
SeriesID: seriesID,
|
||||
SeriesURL: seriesURL,
|
||||
UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed %q: %v", key, err)
|
||||
}
|
||||
if err := s.MarkLatestChecked(key, checkedAt); err != nil {
|
||||
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
|
||||
t.Fatalf("seed mark %q: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
|
||||
t.Helper()
|
||||
ts, err := s.LatestCheckedAt(key)
|
||||
site, seriesID, ok := strings.Cut(key, ":")
|
||||
if !ok {
|
||||
t.Fatalf("key %q: no ':' separator", key)
|
||||
}
|
||||
ts, err := s.LatestCheckedAt(site, seriesID)
|
||||
if err != nil {
|
||||
t.Fatalf("LatestCheckedAt %q: %v", key, err)
|
||||
}
|
||||
@@ -222,6 +241,125 @@ func TestBookmarkRoundTrip(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The wire contract (ADR-0004): GET and PUT speak exactly the flat field set
|
||||
// they always did, with the series-owned fields as siblings of the bookmark
|
||||
// fields, not nested. Asserted as a key set, not by inspection.
|
||||
func TestFlatWireFieldSet(t *testing.T) {
|
||||
srv := newTestServer(t)
|
||||
key := "comix:some-title"
|
||||
in := store.Bookmark{
|
||||
Key: key,
|
||||
Site: "comix",
|
||||
SeriesID: "some-title",
|
||||
Title: "Some Title",
|
||||
SeriesURL: "https://comix.to/title/some-title",
|
||||
Cover: "https://comix.to/covers/some-title.jpg",
|
||||
LastChapter: "Chapter 7",
|
||||
LastChapterNum: 7,
|
||||
LastChapterURL: "https://comix.to/title/some-title/ch/7",
|
||||
Favorite: true,
|
||||
LatestChapter: "Chapter 8",
|
||||
LatestChapterNum: floatPtr(8),
|
||||
Status: store.StatusArchived,
|
||||
Kind: store.KindManga,
|
||||
}
|
||||
body, _ := json.Marshal(in)
|
||||
|
||||
wantKeys := map[string]bool{
|
||||
"key": true, "site": true, "series_id": true, "title": true,
|
||||
"series_url": true, "cover": true, "last_chapter": true,
|
||||
"last_chapter_num": true, "last_chapter_url": true, "favorite": true,
|
||||
"latest_chapter": true, "latest_chapter_num": true, "updated_at": true,
|
||||
"status": true, "kind": true,
|
||||
}
|
||||
checkFlat := func(t *testing.T, payload []byte) map[string]json.RawMessage {
|
||||
t.Helper()
|
||||
var obj map[string]json.RawMessage
|
||||
if err := json.Unmarshal(payload, &obj); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if len(obj) != len(wantKeys) {
|
||||
t.Fatalf("field count = %d, want %d (%s)", len(obj), len(wantKeys), payload)
|
||||
}
|
||||
for k := range obj {
|
||||
if !wantKeys[k] {
|
||||
t.Fatalf("unexpected field %q", k)
|
||||
}
|
||||
}
|
||||
return obj
|
||||
}
|
||||
|
||||
// PUT
|
||||
rr := httptest.NewRecorder()
|
||||
srv.ServeHTTP(rr, auth(httptest.NewRequest(http.MethodPut, "/bookmarks/"+key, bytes.NewReader(body))))
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("PUT status = %d, want 200", rr.Code)
|
||||
}
|
||||
checkFlat(t, rr.Body.Bytes())
|
||||
|
||||
// Every field round-trips with its value, and updated_at is server-stamped.
|
||||
var stored store.Bookmark
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &stored); err != nil {
|
||||
t.Fatalf("decode PUT response: %v", err)
|
||||
}
|
||||
latestNum := floatPtr(8)
|
||||
want := store.Bookmark{
|
||||
Key: key, Site: "comix", SeriesID: "some-title",
|
||||
Title: in.Title, SeriesURL: in.SeriesURL, Cover: in.Cover,
|
||||
LastChapter: in.LastChapter, LastChapterNum: in.LastChapterNum,
|
||||
LastChapterURL: in.LastChapterURL, Favorite: true,
|
||||
LatestChapter: in.LatestChapter, LatestChapterNum: latestNum,
|
||||
Status: store.StatusArchived, Kind: store.KindManga,
|
||||
}
|
||||
if stored.Title != want.Title || stored.SeriesURL != want.SeriesURL || stored.Cover != want.Cover ||
|
||||
stored.LastChapter != want.LastChapter || stored.LastChapterNum != want.LastChapterNum ||
|
||||
stored.LastChapterURL != want.LastChapterURL || stored.Favorite != want.Favorite ||
|
||||
stored.LatestChapter != want.LatestChapter ||
|
||||
stored.LatestChapterNum == nil || *stored.LatestChapterNum != *want.LatestChapterNum ||
|
||||
stored.Status != want.Status || stored.Kind != want.Kind {
|
||||
t.Fatalf("PUT response = %+v, want %+v", stored, want)
|
||||
}
|
||||
if stored.UpdatedAt == 0 {
|
||||
t.Fatal("updated_at not server-stamped")
|
||||
}
|
||||
|
||||
// GET reports the same flat shape.
|
||||
list := getBookmarks(t, srv)
|
||||
if len(list) != 1 {
|
||||
t.Fatalf("list = %d items, want 1", len(list))
|
||||
}
|
||||
body2, _ := json.Marshal(list[0])
|
||||
checkFlat(t, body2)
|
||||
}
|
||||
|
||||
// A PUT naming an existing series must ignore client-supplied title, cover and
|
||||
// URL — the security boundary from ADR-0003, where a hostile site's scraped
|
||||
// values could otherwise land on a shared row — while progress still lands.
|
||||
func TestPutExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
|
||||
srv := newTestServer(t)
|
||||
key := "asura:solo"
|
||||
|
||||
first := putBookmark(t, srv, key, store.Bookmark{
|
||||
Title: "Solo Leveling",
|
||||
SeriesURL: "https://asurascans.com/comics/solo",
|
||||
Cover: "https://asurascans.com/covers/solo.jpg",
|
||||
LastChapterNum: 10,
|
||||
})
|
||||
|
||||
second := putBookmark(t, srv, key, store.Bookmark{
|
||||
Title: "Scraped Rename",
|
||||
SeriesURL: "https://evil.example/solo",
|
||||
Cover: "https://evil.example/solo.jpg",
|
||||
LastChapterNum: 11,
|
||||
})
|
||||
if second.Title != first.Title || second.SeriesURL != first.SeriesURL || second.Cover != first.Cover {
|
||||
t.Fatalf("stored = %+v, want original title/url/cover kept", second)
|
||||
}
|
||||
if second.LastChapterNum != 11 {
|
||||
t.Fatalf("LastChapterNum = %v, want 11 — progress must still land", second.LastChapterNum)
|
||||
}
|
||||
}
|
||||
|
||||
// putBookmark PUTs b at key and returns the bookmark the server echoes back,
|
||||
// which is the row as actually stored (not the request payload).
|
||||
func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark {
|
||||
@@ -417,12 +555,7 @@ func TestLoadConfigWebPassword(t *testing.T) {
|
||||
// moved into bookmarkColumns, this test catches it: the PUT would reset the
|
||||
// cooldown and the poller would re-fetch that series on every single tick.
|
||||
func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) {
|
||||
dbPath := filepath.Join(t.TempDir(), "test.db")
|
||||
s, err := store.Open(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("store.Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
s := newTestStore(t)
|
||||
srv := newRouter(s, testConfig())
|
||||
|
||||
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777)
|
||||
@@ -454,12 +587,7 @@ func TestUserscriptServedWithWebUIDisabled(t *testing.T) {
|
||||
t.Fatalf("write script: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(t.TempDir(), "nopass.db")
|
||||
s, err := store.Open(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("store.Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
s := newTestStore(t)
|
||||
cfg := testConfig() // WebPassword empty
|
||||
cfg.UserscriptPath = path
|
||||
|
||||
@@ -480,11 +608,7 @@ func TestNovelUserscriptServed(t *testing.T) {
|
||||
t.Fatalf("write script: %v", err)
|
||||
}
|
||||
|
||||
s, err := store.Open(filepath.Join(dir, "test.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("store.Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
s := newTestStore(t)
|
||||
|
||||
cfg := testConfig()
|
||||
cfg.NovelUserscriptPath = novelPath
|
||||
|
||||
+5
-13
@@ -7,7 +7,7 @@ require (
|
||||
github.com/bogdanfinn/tls-client v1.15.1
|
||||
github.com/chromedp/cdproto v0.0.0-20260714215040-dc233986426f
|
||||
github.com/chromedp/chromedp v0.16.0
|
||||
modernc.org/sqlite v1.34.4
|
||||
github.com/jackc/pgx/v5 v5.10.0
|
||||
)
|
||||
|
||||
require (
|
||||
@@ -18,27 +18,19 @@ require (
|
||||
github.com/bogdanfinn/utls v1.7.7-barnius // indirect
|
||||
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
|
||||
github.com/chromedp/sysutil v1.1.0 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 // indirect
|
||||
github.com/gobwas/httphead v0.1.0 // indirect
|
||||
github.com/gobwas/pool v0.2.1 // indirect
|
||||
github.com/gobwas/ws v1.4.0 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.18.2 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/ncruces/go-strftime v0.1.9 // indirect
|
||||
github.com/quic-go/qpack v0.6.0 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
|
||||
golang.org/x/crypto v0.46.0 // indirect
|
||||
golang.org/x/net v0.48.0 // indirect
|
||||
golang.org/x/sync v0.19.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.32.0 // indirect
|
||||
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
|
||||
modernc.org/libc v1.55.3 // indirect
|
||||
modernc.org/mathutil v1.6.0 // indirect
|
||||
modernc.org/memory v1.8.0 // indirect
|
||||
modernc.org/strutil v1.2.0 // indirect
|
||||
modernc.org/token v1.1.0 // indirect
|
||||
)
|
||||
|
||||
+14
-44
@@ -20,10 +20,9 @@ github.com/chromedp/chromedp v0.16.0 h1:rOO4deOm4CbZgBCa8mD9g2rDyIoNs0BkgvNrlbp5
|
||||
github.com/chromedp/chromedp v0.16.0/go.mod h1:rbuGKFT1vMcFcFqKfPIO1GpX/N+2s8onm2qMxZLbU5U=
|
||||
github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM=
|
||||
github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 h1:KZaTBSyshWX3MP5jukJcNSuXDQTO+rNpt0J564dX/eg=
|
||||
github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg=
|
||||
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
|
||||
@@ -32,28 +31,27 @@ github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
|
||||
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
|
||||
github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs=
|
||||
github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc=
|
||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
|
||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
|
||||
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
|
||||
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
|
||||
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo=
|
||||
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
|
||||
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw=
|
||||
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
|
||||
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
|
||||
@@ -64,8 +62,6 @@ go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
|
||||
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
|
||||
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
|
||||
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
||||
golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
|
||||
golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
|
||||
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
||||
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
|
||||
@@ -81,33 +77,7 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
|
||||
golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
|
||||
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
|
||||
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
|
||||
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
|
||||
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
|
||||
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
|
||||
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
|
||||
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
|
||||
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 h1:5D53IMaUuA5InSeMu9eJtlQXS2NxAhyWQvkKEgXZhHI=
|
||||
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6/go.mod h1:Qz0X07sNOR1jWYCrJMEnbW/X55x206Q7Vt4mz6/wHp4=
|
||||
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
|
||||
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
|
||||
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
|
||||
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
|
||||
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
|
||||
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
|
||||
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
|
||||
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
|
||||
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
|
||||
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
|
||||
modernc.org/sqlite v1.34.4 h1:sjdARozcL5KJBvYQvLlZEmctRgW9xqIZc2ncN7PU0P8=
|
||||
modernc.org/sqlite v1.34.4/go.mod h1:3QQFCG2SEMtc2nv+Wq4cQCH7Hjcg+p/RMlS1XK+zwbk=
|
||||
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
|
||||
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
|
||||
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
|
||||
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
||||
|
||||
@@ -13,6 +13,9 @@ import (
|
||||
// Handler serves the userscript-facing JSON bookmark API.
|
||||
type Handler struct {
|
||||
Store *store.Store
|
||||
// ReaderID is the Reader this request acts as. Authentication is still the
|
||||
// single global token, so that is always the seeded owner (issue #22).
|
||||
ReaderID int64
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||
@@ -25,9 +28,9 @@ func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||
}
|
||||
}
|
||||
|
||||
// List returns all bookmarks. GET /bookmarks
|
||||
// List returns all bookmarks of the acting Reader. GET /bookmarks
|
||||
func (h *Handler) List(w http.ResponseWriter, r *http.Request) {
|
||||
items, err := h.Store.List()
|
||||
items, err := h.Store.List(h.ReaderID)
|
||||
if err != nil {
|
||||
log.Printf("list: %v", err)
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
@@ -91,7 +94,7 @@ func (h *Handler) Put(w http.ResponseWriter, r *http.Request) {
|
||||
// reading progress actually moved. Any client value is ignored.
|
||||
b.UpdatedAt = time.Now().UnixMilli()
|
||||
|
||||
stored, err := h.Store.Upsert(b)
|
||||
stored, err := h.Store.Upsert(h.ReaderID, b)
|
||||
if err != nil {
|
||||
log.Printf("upsert: %v", err)
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
@@ -109,7 +112,7 @@ func (h *Handler) Delete(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "missing key", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if err := h.Store.Delete(key); err != nil {
|
||||
if err := h.Store.Delete(h.ReaderID, key); err != nil {
|
||||
log.Printf("delete: %v", err)
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
return
|
||||
|
||||
@@ -23,9 +23,9 @@ type Fetcher interface {
|
||||
// Two clocks, deliberately independent:
|
||||
//
|
||||
// - Interval is how often this goroutine wakes up and looks.
|
||||
// - Cooldown is how long one bookmark rests since its own last check.
|
||||
// - Cooldown is how long one series rests since its own last check.
|
||||
//
|
||||
// Only the cooldown is per bookmark, and it is enforced by the WHERE clause in
|
||||
// Only the cooldown is per series, and it is enforced by the WHERE clause in
|
||||
// DueForLatestCheck rather than by any timer. Shortening Interval therefore
|
||||
// cannot shorten anyone's cooldown; it only makes the poller wake up and find
|
||||
// nothing due more often.
|
||||
@@ -78,7 +78,7 @@ func (p *Poller) Run(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// runOnce processes one batch of due bookmarks.
|
||||
// runOnce processes one batch of due series.
|
||||
func (p *Poller) runOnce(ctx context.Context) {
|
||||
cutoff := p.Now().Add(-p.Cooldown).UnixMilli()
|
||||
due, err := p.Store.DueForLatestCheck(cutoff, p.Batch)
|
||||
@@ -88,7 +88,7 @@ func (p *Poller) runOnce(ctx context.Context) {
|
||||
}
|
||||
|
||||
checked := 0
|
||||
for i, b := range due {
|
||||
for i, sr := range due {
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
@@ -107,7 +107,7 @@ func (p *Poller) runOnce(ctx context.Context) {
|
||||
if stopped {
|
||||
break
|
||||
}
|
||||
p.checkOne(ctx, b)
|
||||
p.checkOne(ctx, sr)
|
||||
checked++
|
||||
}
|
||||
// due vs checked is how you tell which constraint is binding: ticks that
|
||||
@@ -119,10 +119,10 @@ func (p *Poller) runOnce(ctx context.Context) {
|
||||
// checkOne re-checks one series. Every failure path here is "log and move on":
|
||||
// the poller is a best-effort enhancement, and no single bad series may stall a
|
||||
// batch or take down the process.
|
||||
func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
|
||||
func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
log.Printf("latest poll %q: recovered from panic: %v", b.Key, r)
|
||||
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -130,8 +130,8 @@ func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
|
||||
// mid-request still consumes the cooldown. Otherwise a renamed or deleted
|
||||
// series would be retried on every single tick forever. The userscript
|
||||
// stamps in the same order and for the same reason (L471-473).
|
||||
if err := p.Store.MarkLatestChecked(b.Key, p.Now().UnixMilli()); err != nil {
|
||||
log.Printf("latest poll %q: mark checked: %v", b.Key, err)
|
||||
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
|
||||
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -142,69 +142,52 @@ func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
|
||||
// link-local/internal addresses or non-https schemes. The cooldown above
|
||||
// is already consumed, so a row that never passes this check is retried at
|
||||
// cooldown pace rather than hot-looping.
|
||||
if !fetchableSeriesURL(b.Site, b.SeriesURL) {
|
||||
log.Printf("latest poll %q: not fetchable: site=%q url=%q", b.Key, b.Site, b.SeriesURL)
|
||||
if !fetchableSeriesURL(sr.Site, sr.SeriesURL) {
|
||||
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
|
||||
return
|
||||
}
|
||||
|
||||
f := p.fetcherFor(b.Site)
|
||||
f := p.fetcherFor(sr.Site)
|
||||
if f == nil {
|
||||
log.Printf("latest poll %q: no fetcher for site %q", b.Key, b.Site)
|
||||
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
|
||||
return
|
||||
}
|
||||
|
||||
body, status, err := f.Get(ctx, b.SeriesURL)
|
||||
body, status, err := f.Get(ctx, sr.SeriesURL)
|
||||
if err != nil {
|
||||
log.Printf("latest poll %q: fetch %s: %v", b.Key, b.SeriesURL, err)
|
||||
log.Printf("latest poll %q: fetch %s: %v", sr.Key(), sr.SeriesURL, err)
|
||||
return
|
||||
}
|
||||
if status != 200 {
|
||||
log.Printf("latest poll %q: fetch %s: status %d", b.Key, b.SeriesURL, status)
|
||||
log.Printf("latest poll %q: fetch %s: status %d", sr.Key(), sr.SeriesURL, status)
|
||||
return
|
||||
}
|
||||
|
||||
latest, ok := latestChapterFrom(b.Site, b.SeriesURL, body)
|
||||
latest, ok := latestChapterFrom(sr.Site, sr.SeriesURL, body)
|
||||
if !ok {
|
||||
// Most likely a challenge page or a layout change. Either way the row is
|
||||
// already stamped, so this waits out a cooldown instead of hot-looping.
|
||||
log.Printf("latest poll %q: no chapter links in %d bytes", b.Key, len(body))
|
||||
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), len(body))
|
||||
return
|
||||
}
|
||||
|
||||
// Re-read: the row may have been updated or deleted while the fetch was in
|
||||
// flight, and writing b back wholesale would undo that.
|
||||
//
|
||||
// ponytail: non-transactional read-modify-write, wrap Get+Upsert in a tx if
|
||||
// this ever runs for more than one user. A client PUT that commits between
|
||||
// these two statements is lost to the stale re-read — reverting read
|
||||
// progress or a status change, and moving updated_at because the stored
|
||||
// value now differs. Accepted for a single-user deployment: the window is
|
||||
// milliseconds and the loser is one poll cycle.
|
||||
cur, found, err := p.Store.Get(b.Key)
|
||||
if err != nil {
|
||||
log.Printf("latest poll %q: reread: %v", b.Key, err)
|
||||
return
|
||||
}
|
||||
if !found {
|
||||
return
|
||||
}
|
||||
// Equality, not >, mirroring the userscript (L427): a site that retracts a
|
||||
// chapter should correct the stored number downward.
|
||||
if cur.LatestChapterNum != nil && *cur.LatestChapterNum == latest.Num {
|
||||
// chapter should correct the stored number downward. The comparison is
|
||||
// against the due-query snapshot; a concurrent write in between only costs
|
||||
// one redundant UPDATE of the same absolute value, never a wrong one.
|
||||
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == latest.Num {
|
||||
return
|
||||
}
|
||||
|
||||
num := latest.Num
|
||||
cur.LatestChapter = latest.Label
|
||||
cur.LatestChapterNum = &num
|
||||
// A candidate only. last_chapter_num is untouched, so the CASE in Upsert
|
||||
// keeps the stored updated_at and the bookmark list does not reorder.
|
||||
cur.UpdatedAt = p.Now().UnixMilli()
|
||||
if _, err := p.Store.Upsert(cur); err != nil {
|
||||
log.Printf("latest poll %q: upsert: %v", b.Key, err)
|
||||
// Series-level write: the row is shared, so one update refreshes every
|
||||
// bookmark joining to it, and the bookmark's updated_at is never touched —
|
||||
// a newly published chapter is not reading progress and must not reorder
|
||||
// the list.
|
||||
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, latest.Label, latest.Num); err != nil {
|
||||
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
|
||||
return
|
||||
}
|
||||
log.Printf("latest poll %q: latest is now %s", b.Key, latest.Label)
|
||||
log.Printf("latest poll %q: latest is now %s", sr.Key(), latest.Label)
|
||||
}
|
||||
|
||||
// fetchableSeriesURL reports whether site is a site latestChapterFrom knows how
|
||||
|
||||
@@ -2,46 +2,67 @@ package latest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"bookmarkmanager/backend/internal/pgtest"
|
||||
"bookmarkmanager/backend/internal/store"
|
||||
)
|
||||
|
||||
// newTestStore opens a fresh SQLite store in a temp dir.
|
||||
func newTestStore(t *testing.T) *store.Store {
|
||||
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||
|
||||
// testOwner is the owner every test store seeds. A second reader, where a
|
||||
// test needs one, is created by opening the same database as a second owner.
|
||||
var testOwner = store.Owner{DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("owner-token-hash"))}
|
||||
|
||||
// newTestStore opens a store on a Postgres database of this test's own and
|
||||
// returns the URL, for helpers that need a second connection to the same
|
||||
// database (see TestRunOnceFetchesSharedSeriesOnce).
|
||||
func newTestStore(t *testing.T) (*store.Store, string) {
|
||||
t.Helper()
|
||||
s, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
||||
url := pgtest.URL(t)
|
||||
s, err := store.Open(url, testOwner)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
return s
|
||||
return s, url
|
||||
}
|
||||
|
||||
// seedForCheck inserts a bookmark and forces its latest_checked_at.
|
||||
// seedForCheck inserts a bookmark (and with it its series) and forces the
|
||||
// series' latest_checked_at.
|
||||
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
|
||||
t.Helper()
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
site, seriesID, ok := strings.Cut(key, ":")
|
||||
if !ok {
|
||||
t.Fatalf("key %q: no ':' separator", key)
|
||||
}
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key,
|
||||
Site: "asura",
|
||||
SeriesID: key,
|
||||
Site: site,
|
||||
SeriesID: seriesID,
|
||||
SeriesURL: seriesURL,
|
||||
UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed %q: %v", key, err)
|
||||
}
|
||||
if err := s.MarkLatestChecked(key, checkedAt); err != nil {
|
||||
if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
|
||||
t.Fatalf("seed mark %q: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
|
||||
t.Helper()
|
||||
ts, err := s.LatestCheckedAt(key)
|
||||
site, seriesID, ok := strings.Cut(key, ":")
|
||||
if !ok {
|
||||
t.Fatalf("key %q: no ':' separator", key)
|
||||
}
|
||||
ts, err := s.LatestCheckedAt(site, seriesID)
|
||||
if err != nil {
|
||||
t.Fatalf("LatestCheckedAt %q: %v", key, err)
|
||||
}
|
||||
@@ -98,7 +119,7 @@ func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Polle
|
||||
}
|
||||
|
||||
func TestRunOnceRecordsLatestChapter(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
|
||||
seedForCheck(t, s, "asura:chronicles-of-the-demon-faction-f886a8af", url, 0)
|
||||
|
||||
@@ -106,7 +127,7 @@ func TestRunOnceRecordsLatestChapter(t *testing.T) {
|
||||
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||
newTestPoller(t, s, f, now).runOnce(context.Background())
|
||||
|
||||
b, ok, err := s.Get("asura:chronicles-of-the-demon-faction-f886a8af")
|
||||
b, ok, err := s.Get(s.OwnerID(), "asura:chronicles-of-the-demon-faction-f886a8af")
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get: %v ok=%v", err, ok)
|
||||
}
|
||||
@@ -124,19 +145,19 @@ func TestRunOnceRecordsLatestChapter(t *testing.T) {
|
||||
// The whole point of the updated_at CASE in Upsert: a newly published chapter is
|
||||
// not reading progress and must not move the series up the list.
|
||||
func TestRunOnceDoesNotReorderList(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
|
||||
const key = "asura:chronicles-of-the-demon-faction-f886a8af"
|
||||
|
||||
// "other" is the most recently read, so it must stay at the top of List().
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: "asura:other", Site: "asura", SeriesID: "other",
|
||||
SeriesURL: "https://asurascans.com/comics/other", UpdatedAt: 9_000_000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed other: %v", err)
|
||||
}
|
||||
seedForCheck(t, s, key, url, 0)
|
||||
before, _, err := s.Get(key)
|
||||
before, _, err := s.Get(s.OwnerID(), key)
|
||||
if err != nil {
|
||||
t.Fatalf("Get before: %v", err)
|
||||
}
|
||||
@@ -144,7 +165,7 @@ func TestRunOnceDoesNotReorderList(t *testing.T) {
|
||||
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||
newTestPoller(t, s, f, time.UnixMilli(9_999_999)).runOnce(context.Background())
|
||||
|
||||
after, _, err := s.Get(key)
|
||||
after, _, err := s.Get(s.OwnerID(), key)
|
||||
if err != nil {
|
||||
t.Fatalf("Get after: %v", err)
|
||||
}
|
||||
@@ -152,7 +173,7 @@ func TestRunOnceDoesNotReorderList(t *testing.T) {
|
||||
t.Fatalf("updated_at moved from %d to %d on a latest-chapter bump",
|
||||
before.UpdatedAt, after.UpdatedAt)
|
||||
}
|
||||
list, err := s.List()
|
||||
list, err := s.List(s.OwnerID())
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
@@ -175,7 +196,7 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
const url = "https://asurascans.com/comics/x"
|
||||
seedForCheck(t, s, "asura:x", url, 0)
|
||||
|
||||
@@ -186,7 +207,7 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
|
||||
if got := readLatestCheckedAt(t, s, "asura:x"); got != now.UnixMilli() {
|
||||
t.Fatalf("latest_checked_at = %d, want %d", got, now.UnixMilli())
|
||||
}
|
||||
b, _, err := s.Get("asura:x")
|
||||
b, _, err := s.Get(s.OwnerID(), "asura:x")
|
||||
if err != nil {
|
||||
t.Fatalf("Get: %v", err)
|
||||
}
|
||||
@@ -198,7 +219,7 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunOnceRespectsBatchLimit(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
for i := 0; i < 20; i++ {
|
||||
key := "asura:s" + string(rune('a'+i))
|
||||
seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0)
|
||||
@@ -214,9 +235,51 @@ func TestRunOnceRespectsBatchLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the split (ADR-0003): a series referenced by several bookmarks
|
||||
// is fetched once per due cycle, not once per bookmark. Two bookmarks share a
|
||||
// series when two readers track it (issue #22).
|
||||
func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
|
||||
s, url := newTestStore(t)
|
||||
// The slug must match the fixture's own anchors: asura's parser scopes
|
||||
// chapter links to the stored slug.
|
||||
const slug = "chronicles-of-the-demon-faction-f886a8af"
|
||||
const seriesURL = "https://asurascans.com/comics/" + slug
|
||||
seedForCheck(t, s, "asura:"+slug, seriesURL, 0)
|
||||
// A second reader tracks the same series. The seed is the only
|
||||
// reader-creation path, so a second Open as a different owner is how a
|
||||
// test gets a second reader on the same database.
|
||||
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))})
|
||||
if err != nil {
|
||||
t.Fatalf("Open second reader: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { other.Close() })
|
||||
if _, err := s.Upsert(other.OwnerID(), store.Bookmark{
|
||||
Key: "asura:" + slug, Site: "asura", SeriesID: slug, UpdatedAt: 2000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed second reader: %v", err)
|
||||
}
|
||||
|
||||
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||
newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background())
|
||||
|
||||
if got := f.callCount(); got != 1 {
|
||||
t.Fatalf("fetched shared series %d times, want 1", got)
|
||||
}
|
||||
// Both bookmarks join to the same updated series row.
|
||||
for _, st := range []*store.Store{s, other} {
|
||||
b, ok, err := st.Get(st.OwnerID(), "asura:"+slug)
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get: %v ok=%v", err, ok)
|
||||
}
|
||||
if b.LatestChapterNum == nil || *b.LatestChapterNum != 181 {
|
||||
t.Fatalf("LatestChapterNum = %v, want 181", b.LatestChapterNum)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// One unreachable series must not abandon the rest of the batch.
|
||||
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"}
|
||||
for _, k := range keys {
|
||||
seedForCheck(t, s, k, "https://asurascans.com/comics/"+k, 0)
|
||||
@@ -244,7 +307,7 @@ func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
|
||||
// The cooldown is enforced by the due query, so a second immediate pass must do
|
||||
// nothing at all — this is what makes the tick interval independent of it.
|
||||
func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
const url = "https://asurascans.com/comics/x"
|
||||
seedForCheck(t, s, "asura:x", url, 0)
|
||||
|
||||
@@ -274,12 +337,12 @@ func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
|
||||
// A site that retracts a chapter should correct the stored number downward,
|
||||
// mirroring the userscript's equality check (L427) rather than a >.
|
||||
func TestRunOnceCorrectsDownward(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
const url = "https://demonicscans.org/manga/Catastrophic-Necromancer"
|
||||
const key = "demonic:Catastrophic-Necromancer"
|
||||
|
||||
high := 400.0
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key, Site: "demonic", SeriesID: "Catastrophic-Necromancer",
|
||||
SeriesURL: url, LatestChapter: "Chapter 400", LatestChapterNum: &high,
|
||||
UpdatedAt: 1000,
|
||||
@@ -290,7 +353,7 @@ func TestRunOnceCorrectsDownward(t *testing.T) {
|
||||
f := &fakeFetcher{body: demonicSeriesFixture, status: 200}
|
||||
newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background())
|
||||
|
||||
b, _, err := s.Get(key)
|
||||
b, _, err := s.Get(s.OwnerID(), key)
|
||||
if err != nil {
|
||||
t.Fatalf("Get: %v", err)
|
||||
}
|
||||
@@ -316,9 +379,9 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
key := tt.site + ":x"
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key, Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL,
|
||||
UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
@@ -327,8 +390,8 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
|
||||
|
||||
now := time.UnixMilli(4_000_000)
|
||||
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||
newTestPoller(t, s, f, now).checkOne(context.Background(), store.Bookmark{
|
||||
Key: key, Site: tt.site, SeriesURL: tt.seriesURL,
|
||||
newTestPoller(t, s, f, now).checkOne(context.Background(), store.Series{
|
||||
Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL,
|
||||
})
|
||||
|
||||
if got := f.callCount(); got != tt.wantCalls {
|
||||
@@ -343,7 +406,7 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
|
||||
|
||||
// A cancelled context must abandon the batch rather than run it to completion.
|
||||
func TestRunOnceStopsOnCancelledContext(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
for _, k := range []string{"asura:a", "asura:b", "asura:c"} {
|
||||
seedForCheck(t, s, k, "https://asurascans.com/comics/"+k, 0)
|
||||
}
|
||||
@@ -392,8 +455,8 @@ func TestFetchableSeriesURL(t *testing.T) {
|
||||
// receive a challenge page, and the browser fetcher is the whole reason kagane
|
||||
// is pollable at all.
|
||||
func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
s, _ := newTestStore(t)
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b",
|
||||
Site: "kagane",
|
||||
SeriesID: "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b",
|
||||
@@ -418,9 +481,9 @@ func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
|
||||
|
||||
// With a browser fetcher wired up, kagane goes to it and not to the TLS one.
|
||||
func TestKaganeUsesBrowserFetcher(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
s, _ := newTestStore(t)
|
||||
key := "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
|
||||
if _, err := s.Upsert(store.Bookmark{
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key,
|
||||
Site: "kagane",
|
||||
SeriesID: "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b",
|
||||
@@ -445,7 +508,7 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
|
||||
if len(browserF.calls) != 1 {
|
||||
t.Fatalf("browser fetcher calls = %v, want 1", browserF.calls)
|
||||
}
|
||||
got, found, err := s.Get(key)
|
||||
got, found, err := s.Get(s.OwnerID(), key)
|
||||
if err != nil || !found {
|
||||
t.Fatalf("Get: %v found=%v", err, found)
|
||||
}
|
||||
|
||||
@@ -5,8 +5,6 @@ import (
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"bookmarkmanager/backend/internal/store"
|
||||
)
|
||||
|
||||
// latestChapter is the newest chapter a series page advertises.
|
||||
@@ -22,6 +20,12 @@ type latestChapter struct {
|
||||
// before using the slug to scope anything.
|
||||
var asuraSlugRe = regexp.MustCompile(`/comics/([^/?#]+)`)
|
||||
|
||||
// asuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura
|
||||
// appends to every series slug. It rotates on each site redeploy, so it is
|
||||
// never part of a stable series_id. Must stay in sync with stripBuildHash in
|
||||
// userscript/manga-bookmark.user.js.
|
||||
var asuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`)
|
||||
|
||||
// demonicChapterRe matches the pre-redirect anchors demonic series pages link
|
||||
// through. Both the raw "&" and the HTML-escaped "&" forms occur.
|
||||
var demonicChapterRe = regexp.MustCompile(`chaptered\.php\?manga=\d+&(?:amp;)?chapter=([0-9.]+)`)
|
||||
@@ -74,9 +78,9 @@ func latestChapterFrom(site, seriesURL, body string) (latestChapter, bool) {
|
||||
}
|
||||
// Stored URLs predating a redeploy may carry a stale build hash;
|
||||
// chapter hrefs in the fetched body carry the current one. Strip to
|
||||
// the stable ID (same rule as migrateAsuraKeys) and make the hash
|
||||
// optional in the pattern, so scoping survives rotations.
|
||||
slug := store.AsuraBuildHash.ReplaceAllString(m[1], "")
|
||||
// the stable ID and make the hash optional in the pattern, so scoping
|
||||
// survives rotations.
|
||||
slug := asuraBuildHash.ReplaceAllString(m[1], "")
|
||||
// Compiled per call rather than cached: this runs once per fetch, which
|
||||
// is at most a few times a minute, and the slug varies per series.
|
||||
re = regexp.MustCompile(`/comics/` + regexp.QuoteMeta(slug) + `(?:-[0-9a-f]{8})?/chapter/([0-9.]+)`)
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
// Package pgtest runs the Postgres the test suite needs: one throwaway
|
||||
// container per test binary, one fresh database per test. Docker is therefore
|
||||
// a hard prerequisite for `go test ./...`.
|
||||
//
|
||||
// Rolled by hand rather than pulled in as a dependency — it is one `docker
|
||||
// run`, one `docker port` and a ping loop, against a module list that is
|
||||
// otherwise stdlib plus what the poller genuinely needs.
|
||||
package pgtest
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
const (
|
||||
image = "postgres:17-alpine"
|
||||
readyLimit = 60 * time.Second
|
||||
)
|
||||
|
||||
var (
|
||||
adminURL string
|
||||
dbSeq atomic.Int64
|
||||
)
|
||||
|
||||
// Main starts the container, runs the package's tests and tears the container
|
||||
// down. Every test package that touches the store calls it from TestMain:
|
||||
//
|
||||
// func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||
func Main(m *testing.M) int {
|
||||
id, url, err := start()
|
||||
if err != nil {
|
||||
fmt.Println("pgtest:", err)
|
||||
return 1
|
||||
}
|
||||
defer exec.Command("docker", "rm", "-f", id).Run()
|
||||
|
||||
adminURL = url
|
||||
return m.Run()
|
||||
}
|
||||
|
||||
// URL creates a database of its own for t and returns a connection URL for it.
|
||||
// Nothing drops it again: the container goes away wholesale when Main returns.
|
||||
func URL(t testing.TB) string {
|
||||
t.Helper()
|
||||
if adminURL == "" {
|
||||
t.Fatal("pgtest: no container; this package needs TestMain to call pgtest.Main")
|
||||
}
|
||||
// Generated, never derived from the test name, so it needs no quoting and
|
||||
// cannot collide when tests run in parallel.
|
||||
name := "test_" + strconv.FormatInt(dbSeq.Add(1), 10)
|
||||
|
||||
admin, err := sql.Open("pgx", adminURL)
|
||||
if err != nil {
|
||||
t.Fatalf("pgtest: open admin connection: %v", err)
|
||||
}
|
||||
defer admin.Close()
|
||||
if _, err := admin.Exec(`CREATE DATABASE ` + name); err != nil {
|
||||
t.Fatalf("pgtest: create database %s: %v", name, err)
|
||||
}
|
||||
return strings.Replace(adminURL, "/postgres?", "/"+name+"?", 1)
|
||||
}
|
||||
|
||||
// start launches the container and waits for it to accept queries, returning
|
||||
// its id and a connection URL for the default database.
|
||||
func start() (id, url string, err error) {
|
||||
out, err := exec.Command("docker", "run", "-d", "--rm",
|
||||
"-e", "POSTGRES_PASSWORD=pgtest",
|
||||
"-P", image,
|
||||
// Durability buys nothing for a database that dies with the test
|
||||
// binary, and turning it off is most of the container's start-up cost.
|
||||
"-c", "fsync=off", "-c", "full_page_writes=off",
|
||||
).Output()
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("docker run %s: %w", image, err)
|
||||
}
|
||||
id = strings.TrimSpace(string(out))
|
||||
|
||||
port, err := exec.Command("docker", "port", id, "5432/tcp").Output()
|
||||
if err != nil {
|
||||
exec.Command("docker", "rm", "-f", id).Run()
|
||||
return "", "", fmt.Errorf("docker port: %w", err)
|
||||
}
|
||||
// "0.0.0.0:32768" (and possibly a second, IPv6 line); the port is all we want.
|
||||
first, _, _ := strings.Cut(strings.TrimSpace(string(port)), "\n")
|
||||
url = fmt.Sprintf("postgres://postgres:pgtest@127.0.0.1:%s/postgres?sslmode=disable",
|
||||
first[strings.LastIndex(first, ":")+1:])
|
||||
|
||||
if err := waitReady(url); err != nil {
|
||||
exec.Command("docker", "rm", "-f", id).Run()
|
||||
return "", "", err
|
||||
}
|
||||
return id, url, nil
|
||||
}
|
||||
|
||||
func waitReady(url string) error {
|
||||
db, err := sql.Open("pgx", url)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
deadline := time.Now().Add(readyLimit)
|
||||
for {
|
||||
if err = db.Ping(); err == nil {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("postgres not ready after %s: %w", readyLimit, err)
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
-- One row per tracked series, keyed "<site>:<series_id>".
|
||||
--
|
||||
-- Everything is NOT NULL with a default except latest_chapter_num, where NULL
|
||||
-- is a distinct state: nothing has been captured yet, which is not the same as
|
||||
-- chapter zero.
|
||||
--
|
||||
-- Timestamps are unix milliseconds as bigint, not timestamptz: the userscripts
|
||||
-- send Date.now() over the wire and the ordering rule compares them directly.
|
||||
CREATE TABLE bookmarks (
|
||||
key text PRIMARY KEY,
|
||||
site text NOT NULL,
|
||||
series_id text NOT NULL,
|
||||
title text NOT NULL DEFAULT '',
|
||||
series_url text NOT NULL DEFAULT '',
|
||||
cover text NOT NULL DEFAULT '',
|
||||
last_chapter text NOT NULL DEFAULT '',
|
||||
last_chapter_num double precision NOT NULL DEFAULT 0,
|
||||
last_chapter_url text NOT NULL DEFAULT '',
|
||||
favorite boolean NOT NULL DEFAULT false,
|
||||
latest_chapter text NOT NULL DEFAULT '',
|
||||
latest_chapter_num double precision,
|
||||
-- When the server last polled this series, unix ms; 0 means never, and sorts
|
||||
-- first so a new bookmark is picked up on the next tick with no special case.
|
||||
latest_checked_at bigint NOT NULL DEFAULT 0,
|
||||
status text NOT NULL DEFAULT 'reading',
|
||||
kind text NOT NULL DEFAULT 'manga',
|
||||
updated_at bigint NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,45 @@
|
||||
-- One row per distinct work, shared by every bookmark that tracks it
|
||||
-- (ADR-0003). Keyed (site, series_id), the pair a bookmark key decomposes
|
||||
-- into. title/series_url/cover are written once, at creation, and never
|
||||
-- again: client-supplied values are ignored once the row exists and the
|
||||
-- poller is the only party that may change them. kind and the latest-chapter
|
||||
-- fields are last-write-wins like the bookmark's own fields.
|
||||
CREATE TABLE series (
|
||||
site text NOT NULL,
|
||||
series_id text NOT NULL,
|
||||
title text NOT NULL DEFAULT '',
|
||||
series_url text NOT NULL DEFAULT '',
|
||||
cover text NOT NULL DEFAULT '',
|
||||
kind text NOT NULL DEFAULT 'manga',
|
||||
latest_chapter text NOT NULL DEFAULT '',
|
||||
latest_chapter_num double precision,
|
||||
-- When the server last polled this series, unix ms; 0 means never, and sorts
|
||||
-- first so a new bookmark is picked up on the next tick with no special case.
|
||||
latest_checked_at bigint NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (site, series_id)
|
||||
);
|
||||
|
||||
-- Backfill from today's rows. The bookmark key's uniqueness makes
|
||||
-- (site, series_id) unique in practice; DISTINCT is belt and braces.
|
||||
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||
latest_chapter, latest_chapter_num, latest_checked_at)
|
||||
SELECT DISTINCT site, series_id, title, series_url, cover, kind,
|
||||
latest_chapter, latest_chapter_num, latest_checked_at
|
||||
FROM bookmarks;
|
||||
|
||||
-- The bookmark keeps only what differs between readers (ADR-0003): progress,
|
||||
-- favourite, lifecycle bucket. The dropped columns now live on series.
|
||||
ALTER TABLE bookmarks
|
||||
DROP COLUMN title,
|
||||
DROP COLUMN series_url,
|
||||
DROP COLUMN cover,
|
||||
DROP COLUMN kind,
|
||||
DROP COLUMN latest_chapter,
|
||||
DROP COLUMN latest_chapter_num,
|
||||
DROP COLUMN latest_checked_at;
|
||||
|
||||
-- A bookmark may not point at a series that does not exist. No cascade: a
|
||||
-- series outlives its last bookmark, and deleting one is not a store operation.
|
||||
ALTER TABLE bookmarks
|
||||
ADD CONSTRAINT bookmarks_series_fk
|
||||
FOREIGN KEY (site, series_id) REFERENCES series (site, series_id);
|
||||
@@ -0,0 +1,17 @@
|
||||
-- One row per person. Keyed by their Discord user ID; carries the SHA-256 of
|
||||
-- their userscript token and when they were created. Hashed because a token
|
||||
-- in the database is a token anyone with the database can replay; SHA-256 is
|
||||
-- enough because the tokens are high-entropy random values with nothing to
|
||||
-- brute-force. No one can register yet, so this table holds exactly the one
|
||||
-- owner row the seed creates at startup (see Store.Open).
|
||||
CREATE TABLE readers (
|
||||
id bigserial PRIMARY KEY,
|
||||
discord_id text NOT NULL UNIQUE,
|
||||
token_sha256 bytea NOT NULL UNIQUE,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Every bookmark now belongs to a reader. Added nullable: rows created before
|
||||
-- this migration have no owner yet — 0004 attaches them to the seeded owner
|
||||
-- before NOT NULL and the composite key land.
|
||||
ALTER TABLE bookmarks ADD COLUMN reader_id bigint;
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Attach every pre-existing bookmark to the owner reader, seeded between the
|
||||
-- two migrate passes (Store.Open). The oldest reader is the owner by
|
||||
-- construction: only the seed creates readers, and it runs once per database.
|
||||
-- Run-once via the version table, like every migration.
|
||||
UPDATE bookmarks SET reader_id = (SELECT id FROM readers ORDER BY id LIMIT 1);
|
||||
|
||||
-- Ownership lands structurally: reader_id becomes part of the key, so a
|
||||
-- bookmark is one Reader's progress on one Series and a duplicate for the
|
||||
-- same pair is impossible at the database level. Deleting a Reader takes
|
||||
-- their bookmarks with them. The old text key is gone — the wire "key" is
|
||||
-- derived as site:series_id on read, and nothing references the column.
|
||||
-- Dropping it drops the primary key it carried; the composite key replaces
|
||||
-- it, and the FK index the series constraint needs is created automatically.
|
||||
ALTER TABLE bookmarks
|
||||
ALTER COLUMN reader_id SET NOT NULL,
|
||||
DROP COLUMN key,
|
||||
ADD PRIMARY KEY (reader_id, site, series_id),
|
||||
ADD CONSTRAINT bookmarks_reader_fk
|
||||
FOREIGN KEY (reader_id) REFERENCES readers (id) ON DELETE CASCADE;
|
||||
+351
-255
@@ -2,19 +2,28 @@ package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"embed"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
// Bookmark is one tracked series, keyed "<site>:<series_id>" across both sites.
|
||||
//
|
||||
// LastChapter* is the user's read progress; LatestChapter* is the newest
|
||||
// chapter the site has published, captured opportunistically by the userscript.
|
||||
//
|
||||
// Title, SeriesURL, Cover, Kind and LatestChapter* live on the shared Series
|
||||
// row (ADR-0003) and are joined in on read; Bookmark carries only what differs
|
||||
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
|
||||
// format stays flat regardless — see ADR-0004.
|
||||
type Bookmark struct {
|
||||
Key string `json:"key"`
|
||||
Site string `json:"site"`
|
||||
@@ -38,6 +47,35 @@ type Bookmark struct {
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
|
||||
// Series is one distinct work, shared by every bookmark that tracks it. It is
|
||||
// keyed (site, series_id) — the pair a bookmark key decomposes into — and
|
||||
// exists once no matter how many bookmarks point at it (ADR-0003).
|
||||
//
|
||||
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
|
||||
// existing Series has them ignored, and only the backend's own Poll may change
|
||||
// them. Kind and the latest-chapter fields are last-write-wins like the
|
||||
// bookmark's own fields. Never serialized: the wire format is the flat
|
||||
// Bookmark (ADR-0004).
|
||||
type Series struct {
|
||||
Site string
|
||||
SeriesID string
|
||||
Title string
|
||||
SeriesURL string
|
||||
Cover string
|
||||
Kind string
|
||||
LatestChapter string
|
||||
LatestChapterNum *float64 // nil until first captured
|
||||
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
|
||||
|
||||
// readerCount is the number of bookmarks referencing this series, filled
|
||||
// only by the due-queue query that orders on it.
|
||||
readerCount int
|
||||
}
|
||||
|
||||
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
|
||||
// used by the poller's logs and by tests asserting on the due queue.
|
||||
func (s Series) Key() string { return s.Site + ":" + s.SeriesID }
|
||||
|
||||
// HasNewChapter reports whether the site has published past the read point.
|
||||
// A nil LatestChapterNum means nothing has been captured yet, which is not the
|
||||
// same as "nothing new".
|
||||
@@ -114,236 +152,231 @@ const (
|
||||
StatusFinished = "finished"
|
||||
)
|
||||
|
||||
const schema = `
|
||||
CREATE TABLE IF NOT EXISTS bookmarks (
|
||||
key TEXT PRIMARY KEY,
|
||||
site TEXT NOT NULL,
|
||||
series_id TEXT NOT NULL,
|
||||
title TEXT,
|
||||
series_url TEXT,
|
||||
cover TEXT,
|
||||
last_chapter TEXT,
|
||||
last_chapter_num REAL,
|
||||
last_chapter_url TEXT,
|
||||
favorite INTEGER NOT NULL DEFAULT 0,
|
||||
latest_chapter TEXT NOT NULL DEFAULT '',
|
||||
latest_chapter_num REAL,
|
||||
latest_checked_at INTEGER NOT NULL DEFAULT 0,
|
||||
status TEXT NOT NULL DEFAULT 'reading',
|
||||
kind TEXT NOT NULL DEFAULT 'manga',
|
||||
updated_at INTEGER NOT NULL
|
||||
);`
|
||||
//go:embed migrations/*.sql
|
||||
var migrations embed.FS
|
||||
|
||||
// The columns above that databases created before them will be missing.
|
||||
// SQLite has no ADD COLUMN IF NOT EXISTS, so each is added only when absent.
|
||||
var addedColumns = []struct{ name, ddl string }{
|
||||
{"favorite", `ALTER TABLE bookmarks ADD COLUMN favorite INTEGER NOT NULL DEFAULT 0`},
|
||||
{"latest_chapter", `ALTER TABLE bookmarks ADD COLUMN latest_chapter TEXT NOT NULL DEFAULT ''`},
|
||||
{"latest_chapter_num", `ALTER TABLE bookmarks ADD COLUMN latest_chapter_num REAL`},
|
||||
// When the server last looked at this series, unix ms; 0 means never, and
|
||||
// sorts first so a new bookmark is picked up on the next tick with no
|
||||
// special case. Deliberately NOT in bookmarkColumns — see MarkLatestChecked.
|
||||
{"latest_checked_at", `ALTER TABLE bookmarks ADD COLUMN latest_checked_at INTEGER NOT NULL DEFAULT 0`},
|
||||
// Lifecycle bucket. The DEFAULT backfills every pre-existing row as
|
||||
// 'reading', so there is no separate migration step.
|
||||
{"status", `ALTER TABLE bookmarks ADD COLUMN status TEXT NOT NULL DEFAULT 'reading'`},
|
||||
// Library bucket. The DEFAULT backfills every pre-existing row as 'manga',
|
||||
// which is what every row written before novels existed actually is.
|
||||
{"kind", `ALTER TABLE bookmarks ADD COLUMN kind TEXT NOT NULL DEFAULT 'manga'`},
|
||||
// bookmarkColumns is the only value ever concatenated into query text. It is a
|
||||
// compile-time constant; every request value is bound as a parameter. The
|
||||
// series-owned fields are joined in from the series table, in scanBookmark
|
||||
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
|
||||
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover,
|
||||
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
|
||||
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
|
||||
|
||||
// seriesColumns is the series row in scanSeries order, used by the poller's
|
||||
// due query. latest_checked_at lives only on series — see MarkLatestChecked
|
||||
// for why it stays off every client-visible write.
|
||||
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
|
||||
|
||||
// Owner is the person running the service: the first Reader, and the only one
|
||||
// until registration exists. The seed makes sure exactly one readers row
|
||||
// matches their Discord ID, carrying the SHA-256 of their userscript token —
|
||||
// which today is the global API token.
|
||||
type Owner struct {
|
||||
DiscordID string
|
||||
// TokenHash is the SHA-256 of the userscript token; the array shape makes
|
||||
// it a compile error to store anything that is not a hash.
|
||||
TokenHash [32]byte
|
||||
}
|
||||
|
||||
const bookmarkColumns = `key, site, series_id, title, series_url, cover,
|
||||
last_chapter, last_chapter_num, last_chapter_url,
|
||||
favorite, latest_chapter, latest_chapter_num, updated_at, status, kind`
|
||||
|
||||
// Store is the SQLite-backed bookmark store.
|
||||
// Store is the Postgres-backed bookmark store.
|
||||
type Store struct {
|
||||
db *sql.DB
|
||||
// ownerID is the seeded owner Reader (issue #22). Authentication is still
|
||||
// the single global token, so every request acts as this Reader; the store
|
||||
// methods take the id explicitly so the scoping survives per-Reader auth.
|
||||
ownerID int64
|
||||
}
|
||||
|
||||
// OpenStore opens (or creates) the SQLite database at path and applies the schema.
|
||||
func Open(path string) (*Store, error) {
|
||||
// busy_timeout guards against SQLITE_BUSY under the reverse proxy's
|
||||
// concurrent requests; a single writer connection keeps writes serialized.
|
||||
dsn := path
|
||||
if !strings.Contains(dsn, "?") {
|
||||
dsn += "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
|
||||
}
|
||||
db, err := sql.Open("sqlite", dsn)
|
||||
// OwnerID returns the seeded owner Reader's id — the Reader every request
|
||||
// acts as while the global token is still the only credential.
|
||||
func (s *Store) OwnerID() int64 { return s.ownerID }
|
||||
|
||||
// readersMigration is the version that creates the readers table. The owner
|
||||
// seed runs between two migrate passes, so that the run-once migration which
|
||||
// attaches existing bookmarks (0004) finds the owner row.
|
||||
const readersMigration = 3
|
||||
|
||||
// allMigrations is the migrate() cap that applies every pending version.
|
||||
const allMigrations = 0
|
||||
|
||||
// Open connects to Postgres at url — a libpq connection URL such as
|
||||
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — brings its
|
||||
// schema up to date, and seeds the owner Reader.
|
||||
func Open(url string, owner Owner) (*Store, error) {
|
||||
db, err := sql.Open("pgx", url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open sqlite %q: %w", path, err)
|
||||
return nil, fmt.Errorf("open postgres: %w", err)
|
||||
}
|
||||
db.SetMaxOpenConns(1)
|
||||
if _, err := db.Exec(schema); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("apply schema: %w", err)
|
||||
}
|
||||
if err := migrateColumns(db); err != nil {
|
||||
// Schema runs in two passes with the seed between: 0003 creates the
|
||||
// readers table, the owner row must exist before 0004 attaches the
|
||||
// existing bookmarks to it. Anything past 0004 is applied by the second
|
||||
// pass.
|
||||
if err := migrate(db, readersMigration); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("migrate schema: %w", err)
|
||||
}
|
||||
if err := migrateAsuraKeys(db); err != nil {
|
||||
if err := seedOwner(db, owner); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("migrate asura keys: %w", err)
|
||||
return nil, fmt.Errorf("seed owner: %w", err)
|
||||
}
|
||||
return &Store{db: db}, nil
|
||||
if err := migrate(db, allMigrations); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("migrate: %w", err)
|
||||
}
|
||||
var ownerID int64
|
||||
if err := db.QueryRow(
|
||||
`SELECT id FROM readers WHERE discord_id = $1`, owner.DiscordID).Scan(&ownerID); err != nil {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("resolve owner: %w", err)
|
||||
}
|
||||
return &Store{db: db, ownerID: ownerID}, nil
|
||||
}
|
||||
|
||||
// migrateColumns brings a pre-existing bookmarks table up to the current
|
||||
// schema. Safe to run on every start: columns already present are skipped.
|
||||
func migrateColumns(db *sql.DB) error {
|
||||
have, err := existingColumns(db, "bookmarks")
|
||||
// seedOwner makes sure the configured owner exists as exactly one readers row,
|
||||
// and keeps its token hash current on every start: rotating the userscript
|
||||
// token must refresh the hash, or the stored credential goes stale.
|
||||
func seedOwner(db *sql.DB, o Owner) error {
|
||||
if _, err := db.Exec(`
|
||||
INSERT INTO readers (discord_id, token_sha256) VALUES ($1, $2)
|
||||
ON CONFLICT (discord_id) DO UPDATE SET token_sha256 = EXCLUDED.token_sha256`,
|
||||
o.DiscordID, o.TokenHash[:]); err != nil {
|
||||
return fmt.Errorf("seed owner: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// migrate applies every embedded migration this database has not recorded, in
|
||||
// filename order, each in its own transaction. upto caps the highest version
|
||||
// applied; 0 means all. Files are named "<version>_<name>.sql" and are
|
||||
// append-only: editing an applied file changes nothing, because
|
||||
// schema_migrations is how a database remembers what it ran. Runs on every
|
||||
// start and is a no-op once current.
|
||||
func migrate(db *sql.DB, upto int64) error {
|
||||
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version bigint PRIMARY KEY,
|
||||
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
|
||||
return fmt.Errorf("create version table: %w", err)
|
||||
}
|
||||
|
||||
names, err := fs.Glob(migrations, "migrations/*.sql")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, c := range addedColumns {
|
||||
if _, ok := have[c.name]; ok {
|
||||
slices.Sort(names)
|
||||
|
||||
for _, name := range names {
|
||||
version, err := strconv.ParseInt(strings.SplitN(path.Base(name), "_", 2)[0], 10, 64)
|
||||
if err != nil {
|
||||
return fmt.Errorf("migration %q: filename must start with a version number", name)
|
||||
}
|
||||
if upto > 0 && version > upto {
|
||||
continue
|
||||
}
|
||||
if _, err := db.Exec(c.ddl); err != nil {
|
||||
return fmt.Errorf("add column %q: %w", c.name, err)
|
||||
body, err := migrations.ReadFile(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := applyMigration(db, version, string(body)); err != nil {
|
||||
return fmt.Errorf("migration %q: %w", name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AsuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura
|
||||
// appends to every series slug. It rotates on each site redeploy, so it
|
||||
// must not be part of series_id. Must stay in sync with stripBuildHash in
|
||||
// userscript/manga-bookmark.user.js.
|
||||
var AsuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`)
|
||||
|
||||
// migrateAsuraKeys rewrites asura bookmarks whose series_id still carries
|
||||
// the build hash to the stable, hashless ID. Rows keyed with a hash are
|
||||
// orphaned on every Asura redeploy (old-hash URLs 302 to new-hash ones, so
|
||||
// detection yields a key that never matches). When two hash-generations of
|
||||
// one series collide, the row with the newest updated_at wins and the rest
|
||||
// are deleted. Idempotent: hashless IDs never match the regex.
|
||||
func migrateAsuraKeys(db *sql.DB) error {
|
||||
rows, err := db.Query(`SELECT key, series_id, updated_at FROM bookmarks WHERE site = 'asura'`)
|
||||
// applyMigration runs one migration and records its version in the same
|
||||
// transaction, so an interrupted start leaves neither half behind.
|
||||
func applyMigration(db *sql.DB, version int64, body string) error {
|
||||
tx, err := db.Begin()
|
||||
if err != nil {
|
||||
return fmt.Errorf("list asura rows: %w", err)
|
||||
}
|
||||
type row struct {
|
||||
key, id string
|
||||
updated int64
|
||||
}
|
||||
var all []row
|
||||
for rows.Next() {
|
||||
var r row
|
||||
if err := rows.Scan(&r.key, &r.id, &r.updated); err != nil {
|
||||
rows.Close()
|
||||
return fmt.Errorf("scan asura row: %w", err)
|
||||
}
|
||||
all = append(all, r)
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
groups := map[string][]row{}
|
||||
for _, r := range all {
|
||||
stripped := AsuraBuildHash.ReplaceAllString(r.id, "")
|
||||
groups[stripped] = append(groups[stripped], r)
|
||||
var applied bool
|
||||
if err := tx.QueryRow(
|
||||
`SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1)`,
|
||||
version).Scan(&applied); err != nil {
|
||||
return err
|
||||
}
|
||||
for stripped, g := range groups {
|
||||
winner := 0
|
||||
for i := range g {
|
||||
if g[i].updated > g[winner].updated {
|
||||
winner = i
|
||||
}
|
||||
}
|
||||
// Losers go first: rewriting the winner to the stripped key while a
|
||||
// pre-existing hashless row still holds it is a primary-key collision.
|
||||
for i, r := range g {
|
||||
if i == winner {
|
||||
continue
|
||||
}
|
||||
if _, err := db.Exec(`DELETE FROM bookmarks WHERE key = ?`, r.key); err != nil {
|
||||
return fmt.Errorf("drop duplicate %q: %w", r.key, err)
|
||||
}
|
||||
}
|
||||
if r := g[winner]; r.id != stripped {
|
||||
if _, err := db.Exec(
|
||||
`UPDATE bookmarks SET key = ?, series_id = ? WHERE key = ?`,
|
||||
"asura:"+stripped, stripped, r.key); err != nil {
|
||||
return fmt.Errorf("rewrite key %q: %w", r.key, err)
|
||||
}
|
||||
}
|
||||
if applied {
|
||||
return nil
|
||||
}
|
||||
return nil
|
||||
// No parameters, so this goes over the simple protocol and a migration may
|
||||
// hold more than one statement.
|
||||
if _, err := tx.Exec(body); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(`INSERT INTO schema_migrations (version) VALUES ($1)`, version); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func existingColumns(db *sql.DB, table string) (map[string]struct{}, error) {
|
||||
rows, err := db.Query(`SELECT name FROM pragma_table_info(?)`, table)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read %s columns: %w", table, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
out := map[string]struct{}{}
|
||||
for rows.Next() {
|
||||
var name string
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
return nil, fmt.Errorf("scan column name: %w", err)
|
||||
}
|
||||
out[name] = struct{}{}
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// scanBookmark reads one row in bookmarkColumns order, translating SQLite's
|
||||
// integer bool and nullable latest_chapter_num into Go types.
|
||||
//
|
||||
// The optional columns are read through Null* types because rows predating
|
||||
// this code (or written by hand) may hold NULL where the app only ever writes
|
||||
// zero values. Only latest_chapter_num distinguishes the two: everywhere else
|
||||
// NULL and the zero value mean the same thing to clients.
|
||||
// scanBookmark reads one row in bookmarkColumns order. Every column is NOT
|
||||
// NULL except latest_chapter_num, where NULL means "never captured" — a
|
||||
// distinct state from chapter zero, and the reason for the pointer.
|
||||
func scanBookmark(scan func(...any) error) (Bookmark, error) {
|
||||
var (
|
||||
b Bookmark
|
||||
title, seriesURL, cover sql.NullString
|
||||
lastChapter, lastChapterURL, latestChapter sql.NullString
|
||||
status sql.NullString
|
||||
lastChapterNum, latestChapterNum sql.NullFloat64
|
||||
favorite sql.NullInt64
|
||||
b Bookmark
|
||||
latestChapterNum sql.NullFloat64
|
||||
)
|
||||
if err := scan(
|
||||
&b.Key, &b.Site, &b.SeriesID, &title, &seriesURL, &cover,
|
||||
&lastChapter, &lastChapterNum, &lastChapterURL,
|
||||
&favorite, &latestChapter, &latestChapterNum, &b.UpdatedAt, &status, &b.Kind,
|
||||
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover,
|
||||
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
|
||||
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
|
||||
); err != nil {
|
||||
return Bookmark{}, err
|
||||
}
|
||||
b.Title = title.String
|
||||
b.SeriesURL = seriesURL.String
|
||||
b.Cover = cover.String
|
||||
b.LastChapter = lastChapter.String
|
||||
b.LastChapterNum = lastChapterNum.Float64
|
||||
b.LastChapterURL = lastChapterURL.String
|
||||
b.Favorite = favorite.Int64 != 0
|
||||
b.LatestChapter = latestChapter.String
|
||||
if latestChapterNum.Valid {
|
||||
b.LatestChapterNum = &latestChapterNum.Float64
|
||||
}
|
||||
// A NULL, empty, or unrecognised bucket (e.g. a hand-edited row) would
|
||||
// leave the row in no list at all, so anything outside the three known
|
||||
// buckets reads as the default rather than being passed through.
|
||||
b.Status = status.String
|
||||
// The wire identity is derived: there is no stored key column, the
|
||||
// bookmark is keyed (reader_id, site, series_id) (issue #22).
|
||||
b.Key = b.Site + ":" + b.SeriesID
|
||||
// An unrecognised bucket (a hand-edited row) would leave the row in no list
|
||||
// at all, so anything outside the three known buckets reads as the default
|
||||
// rather than being passed through.
|
||||
if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished {
|
||||
b.Status = StatusReading
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
// scanSeries reads one row in seriesColumns order, plus the due query's
|
||||
// reader_count column. latest_chapter_num is NULL until the first capture,
|
||||
// same as on the bookmark read path.
|
||||
func scanSeries(scan func(...any) error) (Series, error) {
|
||||
var (
|
||||
sr Series
|
||||
latestChapterNum sql.NullFloat64
|
||||
)
|
||||
if err := scan(
|
||||
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
|
||||
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
|
||||
&sr.readerCount,
|
||||
); err != nil {
|
||||
return Series{}, err
|
||||
}
|
||||
if latestChapterNum.Valid {
|
||||
sr.LatestChapterNum = &latestChapterNum.Float64
|
||||
}
|
||||
return sr, nil
|
||||
}
|
||||
|
||||
// Close releases the underlying database handle.
|
||||
func (s *Store) Close() error { return s.db.Close() }
|
||||
|
||||
// List returns every bookmark, newest activity first.
|
||||
func (s *Store) List() ([]Bookmark, error) {
|
||||
rows, err := s.db.Query(`SELECT ` + bookmarkColumns + `
|
||||
FROM bookmarks
|
||||
ORDER BY updated_at DESC`)
|
||||
// List returns every bookmark of one reader, newest activity first.
|
||||
// Series-owned fields are joined in, so each Bookmark reads back whole and
|
||||
// flat (ADR-0004).
|
||||
func (s *Store) List(readerID int64) ([]Bookmark, error) {
|
||||
rows, err := s.db.Query(`SELECT `+bookmarkColumns+`
|
||||
FROM bookmarks b
|
||||
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||
WHERE b.reader_id = $1
|
||||
ORDER BY b.updated_at DESC`, readerID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query bookmarks: %w", err)
|
||||
}
|
||||
@@ -360,12 +393,19 @@ func (s *Store) List() ([]Bookmark, error) {
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// Get returns one bookmark by key. A missing key is not an error: ok is false
|
||||
// and err is nil. UI mutations read-modify-write through this so they preserve
|
||||
// the fields they do not touch.
|
||||
func (s *Store) Get(key string) (Bookmark, bool, error) {
|
||||
// Get returns one bookmark of one reader by key. A missing key is not an
|
||||
// error: ok is false and err is nil. UI mutations read-modify-write through
|
||||
// this so they preserve the fields they do not touch.
|
||||
func (s *Store) Get(readerID int64, key string) (Bookmark, bool, error) {
|
||||
site, seriesID, ok := strings.Cut(key, ":")
|
||||
if !ok {
|
||||
return Bookmark{}, false, nil
|
||||
}
|
||||
b, err := scanBookmark(s.db.QueryRow(
|
||||
`SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, key).Scan)
|
||||
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
||||
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
|
||||
readerID, site, seriesID).Scan)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Bookmark{}, false, nil
|
||||
}
|
||||
@@ -375,15 +415,25 @@ func (s *Store) Get(key string) (Bookmark, bool, error) {
|
||||
return b, true, nil
|
||||
}
|
||||
|
||||
// Upsert inserts or replaces a bookmark by key (last-write-wins) and returns
|
||||
// the row as actually stored.
|
||||
// Upsert inserts or replaces one reader's bookmark by key (last-write-wins)
|
||||
// and returns the row as actually stored — one flat object with the
|
||||
// series-owned fields joined in, exactly as GET reports it (ADR-0004). A
|
||||
// bookmark is keyed (reader_id, site, series_id), so the same key upserts two
|
||||
// independent rows for two readers.
|
||||
//
|
||||
// The flat body is decomposed across two tables in one transaction. The series
|
||||
// row is written first (the bookmarks FK requires it to exist), then the
|
||||
// bookmark row. On the series side, title/series_url/cover are applied only
|
||||
// when the row is brand new: once a series exists, client-supplied values are
|
||||
// ignored, because the row is shared and the values are scraped page content —
|
||||
// see ADR-0003. Kind and the latest-chapter fields are last-write-wins.
|
||||
//
|
||||
// b.UpdatedAt is only a candidate: it is applied when the row is new or when
|
||||
// last_chapter_num changes, and otherwise the stored value is kept. Clients
|
||||
// order their list by updated_at, so favoriting a series or recording a newly
|
||||
// published chapter must not disturb that order — only real reading progress
|
||||
// does. Callers must therefore use the returned bookmark, not the argument.
|
||||
func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
|
||||
func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
return Bookmark{}, fmt.Errorf("begin %q: %w", b.Key, err)
|
||||
@@ -395,49 +445,65 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
|
||||
latestNum = *b.LatestChapterNum
|
||||
}
|
||||
|
||||
// IS NOT is SQLite's null-safe comparison. Within DO UPDATE, a bare column
|
||||
// is the stored row and excluded.* is the incoming one; a brand-new key
|
||||
// never reaches this clause, so it keeps the fresh timestamp from VALUES.
|
||||
// The kind column resolves on the VALUES side, not in the conflict clause:
|
||||
// excluded.* is the row *after* these expressions are evaluated, so a
|
||||
// default applied there would look identical to a real 'manga' and would
|
||||
// overwrite a novel series on every PUT from a client that knows nothing
|
||||
// about the column. Resolved once here, an empty incoming kind means "keep
|
||||
// what is stored", and only a brand-new row falls through to the literal
|
||||
// default. The subquery runs inside this transaction, so it sees the row
|
||||
// this statement is about to conflict with. Same pattern as the status
|
||||
// COALESCE on the bookmark insert below.
|
||||
//
|
||||
// The status and kind columns resolve on the VALUES side, not in the
|
||||
// conflict clause: excluded.* is the row *after* these expressions are
|
||||
// evaluated, so a default applied there would look identical to a real
|
||||
// 'reading' / 'manga' and would overwrite an archived or novel row on
|
||||
// every PUT from a client that knows nothing about the column. Resolved
|
||||
// once here, an empty incoming status or kind means "keep what is
|
||||
// stored", and only a brand-new row falls through to the literal
|
||||
// default. The subquery runs inside this transaction, so it sees the
|
||||
// row this statement is about to conflict with.
|
||||
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no
|
||||
// target column to infer the parameter type from, and Postgres rejects the
|
||||
// statement rather than guessing.
|
||||
if _, err := tx.Exec(`
|
||||
INSERT INTO bookmarks (`+bookmarkColumns+`)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
|
||||
COALESCE(NULLIF(?, ''), (SELECT status FROM bookmarks WHERE key = ?), 'reading'),
|
||||
COALESCE(NULLIF(?, ''), (SELECT kind FROM bookmarks WHERE key = ?), 'manga'))
|
||||
ON CONFLICT(key) DO UPDATE SET
|
||||
site=excluded.site, series_id=excluded.series_id, title=excluded.title,
|
||||
series_url=excluded.series_url, cover=excluded.cover,
|
||||
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||
latest_chapter, latest_chapter_num)
|
||||
VALUES ($1, $2, $3, $4, $5,
|
||||
COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
|
||||
$7, $8)
|
||||
ON CONFLICT (site, series_id) DO UPDATE SET
|
||||
kind=excluded.kind,
|
||||
latest_chapter=excluded.latest_chapter,
|
||||
latest_chapter_num=excluded.latest_chapter_num`,
|
||||
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
|
||||
b.LatestChapter, latestNum); err != nil {
|
||||
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
|
||||
}
|
||||
|
||||
// IS DISTINCT FROM is Postgres's null-safe comparison, and it is what
|
||||
// implements the ordering rule. Within DO UPDATE, a bare column is the
|
||||
// stored row and excluded.* is the incoming one; a brand-new key never
|
||||
// reaches this clause, so it keeps the fresh timestamp from VALUES.
|
||||
if _, err := tx.Exec(`
|
||||
INSERT INTO bookmarks (reader_id, site, series_id, last_chapter, last_chapter_num,
|
||||
last_chapter_url, favorite, status, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7,
|
||||
COALESCE(NULLIF($8::text, ''), (SELECT status FROM bookmarks WHERE reader_id = $1 AND site = $2 AND series_id = $3), 'reading'),
|
||||
$9)
|
||||
ON CONFLICT (reader_id, site, series_id) DO UPDATE SET
|
||||
last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num,
|
||||
last_chapter_url=excluded.last_chapter_url,
|
||||
favorite=excluded.favorite,
|
||||
latest_chapter=excluded.latest_chapter,
|
||||
latest_chapter_num=excluded.latest_chapter_num,
|
||||
status=excluded.status,
|
||||
kind=excluded.kind,
|
||||
updated_at=CASE
|
||||
WHEN bookmarks.last_chapter_num IS NOT excluded.last_chapter_num
|
||||
WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num
|
||||
THEN excluded.updated_at
|
||||
ELSE bookmarks.updated_at
|
||||
END`,
|
||||
b.Key, b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover,
|
||||
readerID, b.Site, b.SeriesID,
|
||||
b.LastChapter, b.LastChapterNum, b.LastChapterURL,
|
||||
b.Favorite, b.LatestChapter, latestNum, b.UpdatedAt,
|
||||
b.Status, b.Key,
|
||||
b.Kind, b.Key); err != nil {
|
||||
b.Favorite, b.Status, b.UpdatedAt); err != nil {
|
||||
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
|
||||
}
|
||||
|
||||
stored, err := scanBookmark(tx.QueryRow(
|
||||
`SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, b.Key).Scan)
|
||||
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
||||
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
|
||||
readerID, b.Site, b.SeriesID).Scan)
|
||||
if err != nil {
|
||||
return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err)
|
||||
}
|
||||
@@ -447,79 +513,109 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
|
||||
return stored, nil
|
||||
}
|
||||
|
||||
// Delete removes a bookmark by key. Deleting a missing key is not an error.
|
||||
func (s *Store) Delete(key string) error {
|
||||
if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = ?`, key); err != nil {
|
||||
// Delete removes one reader's bookmark by key. Deleting a missing key is not
|
||||
// an error.
|
||||
func (s *Store) Delete(readerID int64, key string) error {
|
||||
site, seriesID, ok := strings.Cut(key, ":")
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
if _, err := s.db.Exec(
|
||||
`DELETE FROM bookmarks WHERE reader_id = $1 AND site = $2 AND series_id = $3`,
|
||||
readerID, site, seriesID); err != nil {
|
||||
return fmt.Errorf("delete %q: %w", key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DueForLatestCheck returns bookmarks whose server-side latest-chapter check has
|
||||
// aged past cutoffMs, least-recently-checked first, at most limit of them.
|
||||
// DueForLatestCheck returns series whose server-side latest-chapter check has
|
||||
// aged past cutoffMs, ordered by how many bookmarks reference them (descending)
|
||||
// then least-recently-checked first, at most limit of them.
|
||||
//
|
||||
// Oldest-first is what keeps the poller fair when the backlog outgrows its
|
||||
// The reader_count ordering is the point of the split (ADR-0003): a series
|
||||
// shared by several readers is fetched once per due cycle, and the popular
|
||||
// ones stay freshest while the long tail absorbs any shortfall. Within one
|
||||
// reader count, oldest-first keeps the poll fair when the backlog outgrows its
|
||||
// throughput: the most neglected series is always next, so a large collection
|
||||
// refreshes uniformly slower rather than leaving a tail that never refreshes at
|
||||
// all. The userscript sorts its own queue the same way (L453).
|
||||
//
|
||||
// Bookmarks with no series_url are skipped — there is nothing to fetch, which
|
||||
// is the same filter the userscript applies at L452.
|
||||
//
|
||||
// Finished series are excluded: nothing more is coming, so fetching them only
|
||||
// burns requests. Archived ones are deliberately still polled — knowing what a
|
||||
// shelved series is up to is the whole reason for archiving instead of deleting.
|
||||
func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Bookmark, error) {
|
||||
rows, err := s.db.Query(`SELECT `+bookmarkColumns+`
|
||||
FROM bookmarks
|
||||
WHERE series_url IS NOT NULL AND series_url <> ''
|
||||
AND status IS NOT 'finished'
|
||||
AND latest_checked_at <= ?
|
||||
ORDER BY latest_checked_at ASC
|
||||
LIMIT ?`, cutoffMs, limit)
|
||||
// Series with no series_url are skipped — there is nothing to fetch, which is
|
||||
// the same filter the userscript applies at L452. Series whose only bookmarks
|
||||
// are finished are skipped too: nothing more is coming, so fetching them only
|
||||
// burns requests. Archived bookmarks still count — knowing what a shelved
|
||||
// series is up to is the whole reason for archiving instead of deleting.
|
||||
// A series with no bookmarks at all never appears: the join excludes it.
|
||||
func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Series, error) {
|
||||
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
|
||||
FROM series s
|
||||
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||
WHERE s.series_url <> ''
|
||||
AND s.latest_checked_at <= $1
|
||||
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
|
||||
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
|
||||
ORDER BY reader_count DESC, s.latest_checked_at ASC
|
||||
LIMIT $2`, cutoffMs, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query due bookmarks: %w", err)
|
||||
return nil, fmt.Errorf("query due series: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
out := []Bookmark{}
|
||||
out := []Series{}
|
||||
for rows.Next() {
|
||||
b, err := scanBookmark(rows.Scan)
|
||||
sr, err := scanSeries(rows.Scan)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("scan due bookmark: %w", err)
|
||||
return nil, fmt.Errorf("scan due series: %w", err)
|
||||
}
|
||||
out = append(out, b)
|
||||
out = append(out, sr)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// MarkLatestChecked records that the server looked at key at ts, whatever the
|
||||
// look turned up. Marking a missing key is not an error: the row may have been
|
||||
// deleted while a fetch was in flight.
|
||||
// MarkLatestChecked records that the server looked at a series at ts, whatever
|
||||
// the look turned up. Marking a missing series is not an error: the row may
|
||||
// have been orphaned while a fetch was in flight.
|
||||
//
|
||||
// This is the one write that does not go through Upsert, and the column is kept
|
||||
// out of bookmarkColumns on purpose. PUT /bookmarks/{key} decodes a whole
|
||||
// Bookmark from the client and Upsert writes every column it knows about, so a
|
||||
// userscript PUT — which has no idea this field exists — would write a zero and
|
||||
// reset the cooldown, making the poller re-fetch that series every tick for as
|
||||
// long as the user kept reading it.
|
||||
func (s *Store) MarkLatestChecked(key string, ts int64) error {
|
||||
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
|
||||
// a whole Bookmark from the client and Upsert writes every series column it
|
||||
// knows about, so a userscript PUT — which has no idea this field exists —
|
||||
// would write a zero and reset the cooldown, making the poller re-fetch that
|
||||
// series every tick for as long as the user kept reading it.
|
||||
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
|
||||
if _, err := s.db.Exec(
|
||||
`UPDATE bookmarks SET latest_checked_at = ? WHERE key = ?`, ts, key); err != nil {
|
||||
return fmt.Errorf("mark checked %q: %w", key, err)
|
||||
`UPDATE series SET latest_checked_at = $1 WHERE site = $2 AND series_id = $3`,
|
||||
ts, site, seriesID); err != nil {
|
||||
return fmt.Errorf("mark checked %s:%s: %w", site, seriesID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
||||
// tests outside this package (the poller's own tests assert on cooldown
|
||||
// bookkeeping) — see MarkLatestChecked for why the field itself stays off
|
||||
// Bookmark.
|
||||
func (s *Store) LatestCheckedAt(key string) (int64, error) {
|
||||
// bookkeeping) — see MarkLatestChecked for why the field stays off the
|
||||
// client-visible row.
|
||||
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
|
||||
var ts int64
|
||||
if err := s.db.QueryRow(
|
||||
`SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil {
|
||||
return 0, fmt.Errorf("latest checked at %q: %w", key, err)
|
||||
`SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`,
|
||||
site, seriesID).Scan(&ts); err != nil {
|
||||
return 0, fmt.Errorf("latest checked at %s:%s: %w", site, seriesID, err)
|
||||
}
|
||||
return ts, nil
|
||||
}
|
||||
|
||||
// SetLatestChapter records the newest chapter the poll found on a series page.
|
||||
// The poller walks Series rather than Bookmarks, so this is a series-level
|
||||
// write: the row is shared, and updating it once refreshes every bookmark that
|
||||
// joins to it. Touching a missing series is not an error.
|
||||
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
|
||||
if _, err := s.db.Exec(
|
||||
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
|
||||
WHERE site = $1 AND series_id = $2`,
|
||||
site, seriesID, label, num); err != nil {
|
||||
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -32,6 +32,9 @@ const RecentCount = 5
|
||||
// representations (HTML versus JSON) to different clients under different auth.
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
// readerID is the Reader this UI acts as — the seeded owner, while the web
|
||||
// password is still the only credential (issue #22).
|
||||
readerID int64
|
||||
tmpl *template.Template
|
||||
key []byte
|
||||
password string
|
||||
@@ -81,13 +84,14 @@ type loginView struct {
|
||||
|
||||
// New parses every template up front so a broken one kills the process at
|
||||
// startup rather than the first request that touches it.
|
||||
func New(s *store.Store, apiToken, webPassword string) (*Handler, error) {
|
||||
func New(s *store.Store, readerID int64, apiToken, webPassword string) (*Handler, error) {
|
||||
tmpl, err := template.ParseFS(templateFS, "templates/*.html")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Handler{
|
||||
store: s,
|
||||
readerID: readerID,
|
||||
tmpl: tmpl,
|
||||
key: session.Key(apiToken, webPassword),
|
||||
password: webPassword,
|
||||
@@ -216,7 +220,7 @@ func libOf(q string) string {
|
||||
// archived favourite therefore shows only under Archived: Favourites means
|
||||
// "favourites I am currently reading".
|
||||
func (h *Handler) buildListView(lib, tab string) (listView, error) {
|
||||
all, err := h.store.List() // already ordered updated_at DESC
|
||||
all, err := h.store.List(h.readerID) // already ordered updated_at DESC
|
||||
if err != nil {
|
||||
return listView{}, err
|
||||
}
|
||||
@@ -374,7 +378,7 @@ func (h *Handler) loadForMutation(w http.ResponseWriter, r *http.Request) (store
|
||||
http.Error(w, "missing key", http.StatusBadRequest)
|
||||
return store.Bookmark{}, false
|
||||
}
|
||||
b, ok, err := h.store.Get(key)
|
||||
b, ok, err := h.store.Get(h.readerID, key)
|
||||
if err != nil {
|
||||
log.Printf("ui get %q: %v", key, err)
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
@@ -397,7 +401,7 @@ func (h *Handler) loadForMutation(w http.ResponseWriter, r *http.Request) (store
|
||||
// describe the whole library, so they are rebuilt out of band on every
|
||||
// mutation, at the cost of one extra list read per toggle.
|
||||
func (h *Handler) saveAndRenderCard(w http.ResponseWriter, r *http.Request, b store.Bookmark) {
|
||||
stored, err := h.store.Upsert(b)
|
||||
stored, err := h.store.Upsert(h.readerID, b)
|
||||
if err != nil {
|
||||
log.Printf("ui upsert %q: %v", b.Key, err)
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
@@ -489,7 +493,7 @@ func (h *Handler) uiDelete(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "missing key", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if err := h.store.Delete(key); err != nil {
|
||||
if err := h.store.Delete(h.readerID, key); err != nil {
|
||||
log.Printf("ui delete %q: %v", key, err)
|
||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||
return
|
||||
|
||||
+25
-7
@@ -2,6 +2,7 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"log"
|
||||
"net/http"
|
||||
@@ -24,10 +25,15 @@ import (
|
||||
type Config struct {
|
||||
Token string
|
||||
AllowedOrigins []string
|
||||
DBPath string
|
||||
Port string
|
||||
// DatabaseURL is the Postgres connection URL; required, no default,
|
||||
// because a wrong guess would silently start on an empty database.
|
||||
DatabaseURL string
|
||||
Port string
|
||||
// WebPassword gates the browser UI. Empty disables the web routes entirely.
|
||||
WebPassword string
|
||||
// OwnerDiscordID identifies the seeded owner Reader (issue #22). Required:
|
||||
// bookmarks are scoped to a Reader, and without an owner there is none.
|
||||
OwnerDiscordID string
|
||||
// UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js.
|
||||
// Supplied by a bindmount so the script can be edited without a rebuild.
|
||||
UserscriptPath string
|
||||
@@ -143,9 +149,10 @@ func loadLatestPoll() LatestPoll {
|
||||
func loadConfig() Config {
|
||||
c := Config{
|
||||
Token: os.Getenv("API_TOKEN"),
|
||||
DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
|
||||
DatabaseURL: os.Getenv("DATABASE_URL"),
|
||||
Port: envOr("PORT", "8080"),
|
||||
WebPassword: os.Getenv("WEB_PASSWORD"),
|
||||
OwnerDiscordID: os.Getenv("OWNER_DISCORD_ID"),
|
||||
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
|
||||
NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"),
|
||||
LatestPoll: loadLatestPoll(),
|
||||
@@ -171,7 +178,7 @@ func newRouter(s *store.Store, cfg Config) http.Handler {
|
||||
mux.HandleFunc("GET /u/{token}/manga-bookmark.user.js", userscript.Handler(cfg.Token, cfg.UserscriptPath))
|
||||
mux.HandleFunc("GET /u/{token}/novel-bookmark.user.js", userscript.Handler(cfg.Token, cfg.NovelUserscriptPath))
|
||||
|
||||
h := &api.Handler{Store: s}
|
||||
h := &api.Handler{Store: s, ReaderID: s.OwnerID()}
|
||||
protected := http.NewServeMux()
|
||||
protected.HandleFunc("GET /bookmarks", h.List)
|
||||
protected.HandleFunc("PUT /bookmarks/{key}", h.Put)
|
||||
@@ -185,7 +192,7 @@ func newRouter(s *store.Store, cfg Config) http.Handler {
|
||||
// deployment that forgets WEB_PASSWORD exposes nothing rather than
|
||||
// exposing an unprotected list.
|
||||
if cfg.WebPassword != "" {
|
||||
wh, err := web.New(s, cfg.Token, cfg.WebPassword)
|
||||
wh, err := web.New(s, s.OwnerID(), cfg.Token, cfg.WebPassword)
|
||||
if err != nil {
|
||||
log.Fatalf("web handler: %v", err)
|
||||
}
|
||||
@@ -215,8 +222,18 @@ func main() {
|
||||
if cfg.Token == "" {
|
||||
log.Fatal("API_TOKEN is required")
|
||||
}
|
||||
if cfg.OwnerDiscordID == "" {
|
||||
log.Fatal("OWNER_DISCORD_ID is required")
|
||||
}
|
||||
if cfg.DatabaseURL == "" {
|
||||
log.Fatal("DATABASE_URL is required")
|
||||
}
|
||||
|
||||
s, err := store.Open(cfg.DBPath)
|
||||
// The owner's userscript token is the global API token today (issue #22);
|
||||
// the readers row carries its SHA-256, not the token itself.
|
||||
owner := store.Owner{DiscordID: cfg.OwnerDiscordID, TokenHash: sha256.Sum256([]byte(cfg.Token))}
|
||||
|
||||
s, err := store.Open(cfg.DatabaseURL, owner)
|
||||
if err != nil {
|
||||
log.Fatalf("open store: %v", err)
|
||||
}
|
||||
@@ -236,7 +253,8 @@ func main() {
|
||||
}
|
||||
|
||||
go func() {
|
||||
log.Printf("listening on :%s (db=%s, origins=%v)", cfg.Port, cfg.DBPath, cfg.AllowedOrigins)
|
||||
// The connection URL carries a password, so it stays out of the log.
|
||||
log.Printf("listening on :%s (origins=%v)", cfg.Port, cfg.AllowedOrigins)
|
||||
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
log.Fatalf("serve: %v", err)
|
||||
}
|
||||
|
||||
+18
-23
@@ -5,7 +5,6 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -28,11 +27,7 @@ func webConfig() Config {
|
||||
// can seed rows and assert on what the handlers wrote back.
|
||||
func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) {
|
||||
t.Helper()
|
||||
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("store.Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
st := newTestStore(t)
|
||||
return newRouter(st, cfg), st
|
||||
}
|
||||
|
||||
@@ -61,7 +56,7 @@ func TestIndexWithoutSessionShowsLogin(t *testing.T) {
|
||||
func TestIndexWithSessionShowsList(t *testing.T) {
|
||||
cfg := webConfig()
|
||||
srv, st := newWebTestServer(t, cfg)
|
||||
if _, err := st.Upsert(store.Bookmark{
|
||||
if _, err := st.Upsert(st.OwnerID(), store.Bookmark{
|
||||
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||
Title: "Solo Leveling", LastChapter: "45", LastChapterNum: 45,
|
||||
UpdatedAt: time.Now().UnixMilli(),
|
||||
@@ -208,7 +203,7 @@ func TestStaticAssetsServed(t *testing.T) {
|
||||
// seed inserts one bookmark and returns it as stored.
|
||||
func seed(t *testing.T, st *store.Store, b store.Bookmark) store.Bookmark {
|
||||
t.Helper()
|
||||
stored, err := st.Upsert(b)
|
||||
stored, err := st.Upsert(st.OwnerID(), b)
|
||||
if err != nil {
|
||||
t.Fatalf("Upsert: %v", err)
|
||||
}
|
||||
@@ -262,7 +257,7 @@ func TestFavoriteTogglesWithoutReordering(t *testing.T) {
|
||||
t.Fatalf("favorite status = %d, want 200", rr.Code)
|
||||
}
|
||||
|
||||
after, ok, err := st.Get("asura:solo")
|
||||
after, ok, err := st.Get(st.OwnerID(), "asura:solo")
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get after favorite: %v ok=%v", err, ok)
|
||||
}
|
||||
@@ -280,7 +275,7 @@ func TestFavoriteTogglesWithoutReordering(t *testing.T) {
|
||||
// Toggling again turns it back off.
|
||||
rr = httptest.NewRecorder()
|
||||
srv.ServeHTTP(rr, uiRequest(t, cfg, http.MethodPost, "/ui/bookmarks/asura:solo/favorite", nil))
|
||||
back, _, _ := st.Get("asura:solo")
|
||||
back, _, _ := st.Get(st.OwnerID(), "asura:solo")
|
||||
if back.Favorite {
|
||||
t.Fatal("Favorite = true after a second toggle, want false")
|
||||
}
|
||||
@@ -339,7 +334,7 @@ func TestChapterOverrideMovesUpdatedAt(t *testing.T) {
|
||||
t.Fatalf("chapter override status = %d, want 200", rr.Code)
|
||||
}
|
||||
|
||||
after, ok, err := st.Get("asura:solo")
|
||||
after, ok, err := st.Get(st.OwnerID(), "asura:solo")
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get after override: %v ok=%v", err, ok)
|
||||
}
|
||||
@@ -379,7 +374,7 @@ func TestChapterOverrideNoOpPreservesURLAndUpdatedAt(t *testing.T) {
|
||||
t.Fatalf("chapter no-op status = %d, want 200", rr.Code)
|
||||
}
|
||||
|
||||
after, ok, err := st.Get("asura:solo")
|
||||
after, ok, err := st.Get(st.OwnerID(), "asura:solo")
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get after no-op override: %v ok=%v", err, ok)
|
||||
}
|
||||
@@ -413,7 +408,7 @@ func TestChapterOverrideRejectsBadInput(t *testing.T) {
|
||||
if rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rr.Code)
|
||||
}
|
||||
after, _, _ := st.Get("asura:solo")
|
||||
after, _, _ := st.Get(st.OwnerID(), "asura:solo")
|
||||
if after.LastChapterNum != 45 {
|
||||
t.Fatalf("chapter changed to %v on invalid input", after.LastChapterNum)
|
||||
}
|
||||
@@ -464,7 +459,7 @@ func TestUIDeleteRemovesRow(t *testing.T) {
|
||||
if !strings.Contains(body, `id="new-count" hx-swap-oob="true"`) {
|
||||
t.Fatalf("delete body = %q, want the out-of-band badge", body)
|
||||
}
|
||||
if _, ok, _ := st.Get("asura:solo"); ok {
|
||||
if _, ok, _ := st.Get(st.OwnerID(), "asura:solo"); ok {
|
||||
t.Fatal("row still present after delete")
|
||||
}
|
||||
}
|
||||
@@ -543,7 +538,7 @@ func seedStatusRows(t *testing.T, st *store.Store) {
|
||||
}
|
||||
for _, b := range rows {
|
||||
b.UpdatedAt = time.Now().UnixMilli()
|
||||
if _, err := st.Upsert(b); err != nil {
|
||||
if _, err := st.Upsert(st.OwnerID(), b); err != nil {
|
||||
t.Fatalf("seed %s: %v", b.Key, err)
|
||||
}
|
||||
}
|
||||
@@ -617,7 +612,7 @@ func TestRecentStripCarriesUnreadOnlyAndOnlyOnAll(t *testing.T) {
|
||||
Status: store.StatusReading, LastChapterNum: 40, LatestChapter: "40",
|
||||
LatestChapterNum: floatPtr(40), UpdatedAt: time.Now().UnixMilli(),
|
||||
}
|
||||
if _, err := st.Upsert(caught); err != nil {
|
||||
if _, err := st.Upsert(st.OwnerID(), caught); err != nil {
|
||||
t.Fatalf("seed %s: %v", caught.Key, err)
|
||||
}
|
||||
|
||||
@@ -637,12 +632,12 @@ func TestRecentStripCarriesUnreadOnlyAndOnlyOnAll(t *testing.T) {
|
||||
}
|
||||
|
||||
// Nothing new anywhere: the strip has nothing to say and does not render.
|
||||
reading, _, err := st.Get("asura:reading")
|
||||
reading, _, err := st.Get(st.OwnerID(), "asura:reading")
|
||||
if err != nil {
|
||||
t.Fatalf("Get: %v", err)
|
||||
}
|
||||
reading.LatestChapterNum = floatPtr(reading.LastChapterNum)
|
||||
if _, err := st.Upsert(reading); err != nil {
|
||||
if _, err := st.Upsert(st.OwnerID(), reading); err != nil {
|
||||
t.Fatalf("Upsert: %v", err)
|
||||
}
|
||||
// The section still ships (an out-of-band swap needs the id to exist) but
|
||||
@@ -667,7 +662,7 @@ func TestRecentStripCapped(t *testing.T) {
|
||||
Status: store.StatusReading, LastChapterNum: 1, LatestChapter: "2",
|
||||
LatestChapterNum: floatPtr(2), UpdatedAt: time.Now().UnixMilli() + int64(i),
|
||||
}
|
||||
if _, err := st.Upsert(b); err != nil {
|
||||
if _, err := st.Upsert(st.OwnerID(), b); err != nil {
|
||||
t.Fatalf("seed %s: %v", b.Key, err)
|
||||
}
|
||||
}
|
||||
@@ -697,7 +692,7 @@ func TestUIStatusSetsBucket(t *testing.T) {
|
||||
if rr := postStatus(t, srv, cfg, "asura:reading", want); rr.Code != http.StatusOK {
|
||||
t.Fatalf("set %s: status = %d, body %s", want, rr.Code, rr.Body.String())
|
||||
}
|
||||
b, ok, err := st.Get("asura:reading")
|
||||
b, ok, err := st.Get(st.OwnerID(), "asura:reading")
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get: ok=%v err=%v", ok, err)
|
||||
}
|
||||
@@ -715,7 +710,7 @@ func TestUIStatusRejectsUnknownValue(t *testing.T) {
|
||||
if rr := postStatus(t, srv, cfg, "asura:reading", "dropped"); rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rr.Code)
|
||||
}
|
||||
b, _, _ := st.Get("asura:reading")
|
||||
b, _, _ := st.Get(st.OwnerID(), "asura:reading")
|
||||
if b.Status != store.StatusReading {
|
||||
t.Fatalf("stored status = %q, want it untouched", b.Status)
|
||||
}
|
||||
@@ -741,12 +736,12 @@ func TestUIStatusDoesNotReorderList(t *testing.T) {
|
||||
srv, st := newWebTestServer(t, cfg)
|
||||
seedStatusRows(t, st)
|
||||
|
||||
before, _, _ := st.Get("asura:reading")
|
||||
before, _, _ := st.Get(st.OwnerID(), "asura:reading")
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
if rr := postStatus(t, srv, cfg, "asura:reading", store.StatusArchived); rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d", rr.Code)
|
||||
}
|
||||
after, _, _ := st.Get("asura:reading")
|
||||
after, _, _ := st.Get(st.OwnerID(), "asura:reading")
|
||||
if after.UpdatedAt != before.UpdatedAt {
|
||||
t.Fatalf("UpdatedAt moved %d -> %d", before.UpdatedAt, after.UpdatedAt)
|
||||
}
|
||||
|
||||
@@ -23,13 +23,18 @@ services:
|
||||
# isn't an IP or "localhost".
|
||||
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
||||
depends_on:
|
||||
- headless-shell
|
||||
# `networks:` here replaces the base file's list entirely, so both must be
|
||||
# named: `proxy` for Traefik routing, `browser` (defined in the base file)
|
||||
# to keep reaching headless-shell without putting it on `proxy` too.
|
||||
headless-shell:
|
||||
condition: service_started
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
# `networks:` here replaces the base file's list entirely, so all three must
|
||||
# be named: `proxy` for Traefik routing, and `browser` / `db` (defined in
|
||||
# the base file) to keep reaching headless-shell and Postgres without
|
||||
# putting either on `proxy`.
|
||||
networks:
|
||||
- proxy
|
||||
- browser
|
||||
- db
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.docker.network=${PROXY_NETWORK:-proxy}"
|
||||
|
||||
+40
-4
@@ -15,8 +15,12 @@ services:
|
||||
environment:
|
||||
# API_TOKEN is required — compose refuses to start without it.
|
||||
API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env}
|
||||
# Owner's Discord user ID — required, seeds the one Reader row.
|
||||
OWNER_DISCORD_ID: ${OWNER_DISCORD_ID:?set OWNER_DISCORD_ID in .env}
|
||||
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
|
||||
DB_PATH: /data/bookmarks.db
|
||||
# The bookmarks database. Host is the compose service name; the password
|
||||
# comes from .env so it is never committed.
|
||||
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
|
||||
PORT: "8080"
|
||||
# Gates the browser UI. Unset means the web routes are not served at all.
|
||||
WEB_PASSWORD: ${WEB_PASSWORD:-}
|
||||
@@ -42,9 +46,13 @@ services:
|
||||
# this URL survives container recreation.
|
||||
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
||||
depends_on:
|
||||
- headless-shell
|
||||
headless-shell:
|
||||
condition: service_started
|
||||
# The migration runner is the first thing the binary does, so a Postgres
|
||||
# that is still initialising means a crash-loop until it is not.
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- bookmarks-data:/data
|
||||
# The userscript is served from here, read fresh on every request. Editing
|
||||
# the file in this checkout takes effect on the next Violentmonkey poll —
|
||||
# no rebuild, no restart. `git pull` restores the committed version, which
|
||||
@@ -56,6 +64,26 @@ services:
|
||||
- "127.0.0.1:8080:8080"
|
||||
networks:
|
||||
- browser
|
||||
- db
|
||||
|
||||
postgres:
|
||||
image: postgres:17-alpine
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: bookmarks
|
||||
POSTGRES_USER: bookmarks
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
# Deliberately no `ports:` — only bookmark-api, over the `db` network,
|
||||
# reaches it. Use `docker compose exec postgres psql` for a shell.
|
||||
networks:
|
||||
- db
|
||||
|
||||
headless-shell:
|
||||
image: chromedp/headless-shell:stable
|
||||
@@ -85,7 +113,11 @@ services:
|
||||
ipv4_address: 172.28.0.10
|
||||
|
||||
volumes:
|
||||
bookmarks-data:
|
||||
postgres-data:
|
||||
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
|
||||
# declared here: undeclared means `docker compose down -v` cannot take it
|
||||
# with the rest, so the old database survives the cutover until someone
|
||||
# removes it by hand.
|
||||
|
||||
networks:
|
||||
# Not `internal: true`: headless Chrome still needs outbound access to reach
|
||||
@@ -95,3 +127,7 @@ networks:
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.28.0.0/24
|
||||
# Postgres needs no egress and nothing outside bookmark-api needs to reach
|
||||
# it, so this one really can be cut off from the outside world.
|
||||
db:
|
||||
internal: true
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Postgres replaces SQLite as the primary datastore
|
||||
|
||||
Status: accepted
|
||||
|
||||
The project is moving from a single-reader tracker to a service published to a community,
|
||||
so we replaced `modernc.org/sqlite` with Postgres (`jackc/pgx/v5`, still pure Go, so
|
||||
`CGO_ENABLED=0` and the distroless image are unaffected). The deciding reason is future
|
||||
supportability — managed hosting, a datastore that survives the app outgrowing one box —
|
||||
**not** concurrency, which was measured and found to be a non-issue.
|
||||
|
||||
## Considered options
|
||||
|
||||
**Stay on SQLite.** Benchmarked against the real store at 1,500 rows (≈50 readers × 30
|
||||
series): ~9,700 upserts/sec single-writer, plateauing at ~780/sec under 8–50 concurrent
|
||||
writers, with `List()` holding 90–103 calls/sec under continuous write load. Projected
|
||||
real load at 50 readers is ~0.02 writes/sec — roughly four and a half orders of magnitude
|
||||
of headroom. `SetMaxOpenConns(1)` serialises writes but was shown not to starve reads;
|
||||
an apparent read collapse traced to row count and per-row scanning, not lock contention.
|
||||
SQLite would have worked. It was rejected for where the project is going, not for what
|
||||
it does today.
|
||||
|
||||
**Postgres.** Chosen. Migrating is cheapest now — 29 rows in one table — and gets
|
||||
materially harder once there are live readers and a multi-tenant schema.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every statement in `internal/store` is rewritten: `?` → `$N`, `IS NOT` →
|
||||
`IS DISTINCT FROM` (this one is load-bearing; it implements the `updated_at`
|
||||
ordering rule), `pragma_table_info` → `information_schema.columns`,
|
||||
`INTEGER`/`REAL` → `bigint`/`double precision`, `favorite` int-as-bool → `boolean`.
|
||||
- ~75 tests currently get a free isolated database from `t.TempDir()`. They now need a
|
||||
live server, which makes Docker a hard prerequisite for `go test ./...`. This is the
|
||||
permanent cost of the decision and the main reason it was close.
|
||||
- Backups get worse, not better: `VACUUM INTO` produced one self-contained file;
|
||||
restoring now means `pg_dump`/`pg_restore`, a role, and a password.
|
||||
- A second stateful container joins the VPS alongside the existing headless-shell.
|
||||
- **Postgres does not address the real scaling limit.** At batch 14 per 10-minute tick
|
||||
the poller checks at most 84 series/hour; 50 readers × 30 series is 1,500 bookmarks,
|
||||
an 18-hour sweep against a configured 1-hour cooldown. That ceiling is an outbound
|
||||
fetch budget and is fixed by deduplicating polls per Series, not by the datastore.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Identity comes from Discord OAuth; we store no passwords and send no email
|
||||
|
||||
Status: accepted
|
||||
|
||||
The service is being published to a community that already lives on Discord, and we have
|
||||
no transactional email infrastructure. Rather than build email verification and password
|
||||
reset to get accounts, Readers sign in with Discord OAuth2 (authorization code grant,
|
||||
`identify` + `guilds.members.read`), and guild membership replaces both the invite gate
|
||||
and the email-verification step. No password is ever stored and no mail is ever sent.
|
||||
|
||||
## Considered options
|
||||
|
||||
**Email + password with invite codes, no verification.** Viable and dependency-free:
|
||||
an invite code proves community membership, which is what email verification was
|
||||
standing in for anyway. Rejected because it still requires password hashing, a manual
|
||||
admin-driven reset path, and a credential store — all of which Discord removes.
|
||||
|
||||
**Email + password with a transactional provider** (Resend, Brevo). Rejected as
|
||||
premature: it builds verification and self-serve reset before anyone has asked for them,
|
||||
and adds deliverability as an operational concern.
|
||||
|
||||
**Discord OAuth.** Chosen. It is less code than either alternative — no hashing, no
|
||||
reset flow, no invite table — and the authorization question ("is this person in my
|
||||
community?") is answered by the same call that answers the authentication question.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Availability is now coupled to Discord.** If Discord's OAuth endpoint is down,
|
||||
nobody can start a new session. Existing sessions are unaffected, which bounds the
|
||||
blast radius.
|
||||
- **Identity is a Discord snowflake.** Migrating off Discord later means re-identifying
|
||||
every Reader, because we hold no other credential for them. This is the lock-in the
|
||||
decision buys, and it is the reason this ADR exists.
|
||||
- **`guilds.members.read` is checked at login, not continuously.** Someone who leaves
|
||||
the guild keeps their session until it expires. Acceptable; revocation is a session
|
||||
delete, not an architectural change.
|
||||
- **The userscripts cannot use OAuth.** They run in an isolated world on third-party
|
||||
pages with no redirect surface, so they keep a bearer token — now issued per Reader by
|
||||
the backend rather than a single shared `API_TOKEN` literal. OAuth gates the web UI;
|
||||
the web UI is where a Reader obtains their personal userscript.
|
||||
- `WEB_PASSWORD` disappears, and with it the session HMAC key derivation
|
||||
(`sha256(API_TOKEN | WEB_PASSWORD | …)`), which needs a replacement secret.
|
||||
- Seeding the first Reader during migration requires knowing the owner's Discord user
|
||||
ID up front — a stable snowflake, copied from the Discord client.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Series is a shared entity, and only the Poll may update it
|
||||
|
||||
Status: accepted
|
||||
|
||||
Facts about a Series that are true regardless of who is reading — title, cover, canonical
|
||||
URL, Latest Chapter — moved off the Bookmark onto a shared `series` row keyed
|
||||
`(site, series_id)`. A Bookmark now holds only what differs between Readers: Progress,
|
||||
Favourite, Lifecycle bucket. Fifty Readers tracking one Series produce fifty Bookmarks
|
||||
and one Series, so the Series is polled once rather than fifty times.
|
||||
|
||||
## Why
|
||||
|
||||
The poller checks at most 84 series/hour (batch 14 per 10-minute tick). With ~50 Readers
|
||||
holding ~30 Series each, polling per Bookmark means a 1,500-item sweep — roughly 18 hours
|
||||
against a configured 1-hour cooldown, quietly breaking the New Chapter signal that is the
|
||||
product's reason to exist. Deduplicating to distinct Series cuts the sweep several-fold,
|
||||
and because the Series row now knows how many Readers hold it, the poll queue is ordered
|
||||
`reader_count DESC, latest_checked_at ASC` — popular Series stay fresh and the long tail
|
||||
absorbs the shortfall. That ordering is only expressible because the split happened.
|
||||
|
||||
Raising throughput instead was rejected: sweeping 400 Series hourly needs the stagger
|
||||
cut from 20s to ~9s, doubling request rate against sites that already bot-score the
|
||||
single VPS IP.
|
||||
|
||||
## Only the Poll writes Series fields
|
||||
|
||||
A client may supply `title`, `cover` and `series_url` only when creating a Series nobody
|
||||
has bookmarked yet. After that, client-supplied values are ignored; only the backend's
|
||||
own fetch updates them.
|
||||
|
||||
This is a security boundary, not tidiness. Those values are scraped from third-party
|
||||
pages, which `AGENTS.md` requires be treated as attacker-controlled. Before the split, a
|
||||
hostile or compromised site could corrupt exactly one Reader's row. After it, the same
|
||||
write lands on a row every Reader sees — one Reader's browser becomes a write path into
|
||||
everyone else's UI, and a cover URL can point anywhere. The backend's own fetch is the
|
||||
higher-trust source: its network, its parser, no third-party JavaScript in the path.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `Store.Upsert` decomposes one incoming flat body across two tables and enforces the
|
||||
ownership rule at that seam.
|
||||
- The `updated_at` ordering rule stays on the Bookmark, where Progress lives. Unchanged.
|
||||
- Per-Reader title overrides are deliberately not supported; they would reintroduce the
|
||||
duplication this removes.
|
||||
@@ -0,0 +1,32 @@
|
||||
# The wire format stays flat and deliberately does not mirror the schema
|
||||
|
||||
Status: accepted
|
||||
|
||||
Storage splits a tracked series across two tables (ADR-0003), but `GET /bookmarks` and
|
||||
`PUT /bookmarks/{key}` keep emitting and accepting one **flat** JSON object with `title`,
|
||||
`cover`, `last_chapter` and `latest_chapter` as siblings — exactly the shape they had
|
||||
when there was one table. The server joins on the way out and decomposes on the way in.
|
||||
|
||||
## Why a future reader will find this surprising
|
||||
|
||||
The obvious move after splitting a table is to nest the JSON to match. Don't "fix" this.
|
||||
|
||||
**A nested payload would have broken every installed userscript instantly.** Scripts read
|
||||
`b.title` directly; moving it to `b.series.title` yields `undefined` — no error, just
|
||||
blank rows and a New Chapter signal that silently reports nothing forever. Because
|
||||
Violentmonkey updates roughly once a day per device, the migration relies on old scripts
|
||||
continuing to work during a 14-day grace window. A nested format and that grace window
|
||||
are mutually exclusive.
|
||||
|
||||
**It is also the better contract independently of compatibility.** A client rendering one
|
||||
row needs the title and the reading position together; nesting exports the re-stitching
|
||||
to every browser to mirror a decision about disk layout it should not know about. Keeping
|
||||
them separate lets storage change again later without a client release — which is the
|
||||
whole reason this ADR is worth the paragraph.
|
||||
|
||||
## Consequence
|
||||
|
||||
The flat shape is a contract, not an implementation detail. Changing the storage schema
|
||||
must not change it. It follows the rule already in force for `updated_at`: the server
|
||||
owns the truth and returns the row **as stored**, and clients adopt the response rather
|
||||
than their own payload.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Domain Docs
|
||||
|
||||
How the engineering skills should consume this repo's domain documentation when exploring the
|
||||
codebase. Layout: **single-context** — one `CONTEXT.md` plus `docs/adr/` at the repo root.
|
||||
|
||||
## Before exploring, read these
|
||||
|
||||
- **`CONTEXT.md`** at the repo root — the glossary / ubiquitous language.
|
||||
- **`docs/adr/`** — read ADRs that touch the area you're about to work in.
|
||||
|
||||
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest
|
||||
creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and
|
||||
`/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
|
||||
|
||||
Neither exists yet in this repo. The existing `AGENTS.md` / `CLAUDE.md` and `docs/design-system.md`
|
||||
carry the current architecture and design law — read those regardless.
|
||||
|
||||
## File structure
|
||||
|
||||
```
|
||||
/
|
||||
├── CONTEXT.md
|
||||
├── docs/adr/
|
||||
│ ├── 0001-....md
|
||||
│ └── 0002-....md
|
||||
├── backend/
|
||||
└── userscript/
|
||||
```
|
||||
|
||||
If this repo ever splits into genuinely separate contexts, add a root `CONTEXT-MAP.md` pointing at
|
||||
one `CONTEXT.md` per context and update this file.
|
||||
|
||||
## Use the glossary's vocabulary
|
||||
|
||||
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a
|
||||
test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary
|
||||
explicitly avoids.
|
||||
|
||||
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing
|
||||
language the project doesn't use (reconsider) or there's a real gap (note it for
|
||||
`/domain-modeling`).
|
||||
|
||||
## Flag ADR conflicts
|
||||
|
||||
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
|
||||
|
||||
> _Contradicts ADR-0002 (…) — but worth reopening because…_
|
||||
@@ -0,0 +1,60 @@
|
||||
# Issue tracker: Gitea (`tea` CLI)
|
||||
|
||||
Issues and specs for this repo live as issues on the self-hosted Gitea instance
|
||||
`gitea.violetcrown.my.id` (repo `sulthan/mangaBookmark`). **`gh` does not work here** — use
|
||||
[`tea`](https://gitea.com/gitea/tea) for everything past plain git. Auth lives in `tea login`,
|
||||
not a `GH_TOKEN` env var. `tea` infers the repo from the local clone's `origin`.
|
||||
|
||||
`tea` prints rendered boxes rather than plain text; pass `--output json` (or `-o json`) when a
|
||||
skill needs to parse the result.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Create an issue**: `tea issue create --title "..." --description "..."` (`--labels`,
|
||||
`--assignees` optional). Multi-line bodies: pass the body through a shell variable or heredoc.
|
||||
- **Read an issue**: `tea issue <number> --comments` (add `-o json` for machine-readable output).
|
||||
- **List issues**: `tea issue list --state open -o json --fields index,title,body,labels,state,author`;
|
||||
filter with `--labels "..."`, `--state open|closed|all`, `--assignee`, `--keyword`.
|
||||
- **Comment**: `tea comment <number> "..."` (alias of `tea comments add`).
|
||||
- **Apply / remove labels**: `tea issue edit <number> --add-labels "..."` / `--remove-labels "..."`.
|
||||
Labels must exist first — see `tea labels list` / `tea labels create --name "..." --color "#rrggbb"`.
|
||||
- **Close**: `tea issue close <number>` (comment separately with `tea comment`; `close` takes no
|
||||
`--comment` flag).
|
||||
|
||||
## Pull requests as a triage surface
|
||||
|
||||
**PRs as a request surface: no.** _(Set to `yes` if this repo treats external PRs as feature
|
||||
requests; `/triage` reads this flag.)_
|
||||
|
||||
When set to `yes`, PRs run through the same labels and states as issues, using the `tea pr`
|
||||
equivalents: `tea pr <number> --comments`, `tea pr list --state open -o json`,
|
||||
`tea pr create --head <branch> --base main --title "..." --description "..."`, `tea comment`,
|
||||
`tea pr close`. Gitea shares one index space across issues and PRs, so a bare `#42` may be either
|
||||
— resolve with `tea pr 42` and fall back to `tea issue 42`.
|
||||
|
||||
## When a skill says "publish to the issue tracker"
|
||||
|
||||
Create a Gitea issue with `tea issue create`.
|
||||
|
||||
## When a skill says "fetch the relevant ticket"
|
||||
|
||||
Run `tea issue <number> --comments`.
|
||||
|
||||
## Wayfinding operations
|
||||
|
||||
Used by `/wayfinder`. The **map** is a single issue; **tickets** are child issues.
|
||||
|
||||
- **Map**: one issue labelled `wayfinder:map` holding the Notes / Decisions-so-far / Fog body.
|
||||
`tea issue create --labels wayfinder:map --title "..." --description "..."`.
|
||||
- **Child ticket**: an issue labelled `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`)
|
||||
with `Part of #<map>` as the first body line, and a task-list entry in the map body. `tea` has no
|
||||
sub-issue command, so the task list plus the `Part of` line is the canonical link.
|
||||
- **Blocking**: a `Blocked by: #<n>, #<n>` line at the top of the child body. Gitea's native issue
|
||||
dependencies exist in the API but `tea` does not expose them; the body line is the source of
|
||||
truth. A ticket is unblocked when every listed blocker is closed.
|
||||
- **Frontier query**: `tea issue list --state open -o json` scoped to the map's task list; drop any
|
||||
ticket with an open blocker or an assignee; first in map order wins.
|
||||
- **Claim**: `tea issue edit <n> --add-assignees <your-username>` — the session's first write.
|
||||
(`tea` has no `@me` shorthand; use the Gitea username from `tea login list`.)
|
||||
- **Resolve**: `tea comment <n> "<answer>"`, then `tea issue close <n>`, then append a context
|
||||
pointer to the map's Decisions-so-far via `tea issue edit <map> --description "..."`.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Triage Labels
|
||||
|
||||
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual
|
||||
label strings used in this repo's issue tracker (Gitea — see `docs/agents/issue-tracker.md`).
|
||||
|
||||
| Label in mattpocock/skills | Label in our tracker | Meaning |
|
||||
| -------------------------- | -------------------- | ---------------------------------------- |
|
||||
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
||||
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
||||
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
|
||||
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
||||
| `wontfix` | `wontfix` | Will not be actioned |
|
||||
|
||||
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label
|
||||
string from this table.
|
||||
|
||||
Gitea will not auto-create labels on `tea issue edit --add-labels`; create a missing one first with
|
||||
`tea labels create --name "<label>" --color "#rrggbb"`.
|
||||
|
||||
Edit the right-hand column to match whatever vocabulary you actually use.
|
||||
@@ -1,64 +0,0 @@
|
||||
Guidance for Claude Code working under `userscript/`. See root `CLAUDE.md` for the project-wide architecture diagram, hard constraints, and design system.
|
||||
|
||||
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
||||
|
||||
1. **Site adapters** — one per host, `detect(location, document)` return page `type` + IDs. Identify type/IDs from **URL regex** (most stable); pull `title`/`cover` from **`og:title`/`og:image` meta tags**, not CSS classes.
|
||||
2. **API client** — `apiGet/apiPut/apiDelete` with bearer header; `localStorage` key `bmgr:manga:cache` for instant render + offline fallback.
|
||||
3. **Progress logic** — auto-upsert `last_chapter` only when `chapterNum >= stored last_chapter_num` (re-reading old chapters must not regress progress; unparseable -> set current). Manual panel override forces any value.
|
||||
4. **Retry queue** — every write go through `pushBookmark`/`pushDelete`, so
|
||||
failed mutation park in `localStorage` (`bmgr:manga:queue`) and replayed on
|
||||
next navigation, reconnect, or `refresh()`. Entries are markers
|
||||
(`{key, op, sendStatus, attempts}`), never payloads — body read from
|
||||
cache at send time, so one entry per key give ordering and coalescing for
|
||||
free. `sendStatus` is **sticky**: while archive pending, later writes to
|
||||
that key keep carrying bucket, which stop successful
|
||||
in-between write from silently un-archiving series. `refresh()` drains
|
||||
before it fetches and overlays anything still pending, so list never
|
||||
flaps. 400 drops entry, 401 abort pass and keep queue, and
|
||||
transient failures retry to cap of 10. Latest-chapter writes deliberately
|
||||
stay out of queue. See
|
||||
`docs/superpowers/specs/2026-07-27-offline-retry-queue-design.md`.
|
||||
5. **UI** — rendered inside **Shadow DOM** root to isolate from site CSS
|
||||
(critical on mobile). Three tabs (All / Favourites / Archived) and row of
|
||||
link chips to web UI and both manga sites; `WEB_BASE` sits in CONFIG
|
||||
block next to `API_BASE`. FAB is `7 × 44` edge tab whose *hit* area
|
||||
widened to `28 × 72` by invisible `#hit` child; `#fab` must keep
|
||||
`touch-action: none` and must **not** regain `overflow: hidden`. Since
|
||||
`touch-action` resolved at gesture start, strip can't be both
|
||||
browser-scrolled and script-dragged, so `makeDraggable` splits by intent: swipe
|
||||
from `#hit` scrolls via `window.scrollBy`, hold of `ARM_MS` arms
|
||||
reposition drag, visible sliver drags with no hold. See
|
||||
`docs/superpowers/specs/2026-07-28-edge-tab-hitbox-design.md`.
|
||||
6. **SPA navigation** — Asura is Astro, client-routed on comic/chapter pages: patch `history.pushState`/`replaceState` + listen `popstate`, re-run `detect()` on URL change so auto-update fire without reload. Demonic uses classic reloads (initial `document-idle` run suffice).
|
||||
|
||||
### Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust)
|
||||
|
||||
- **asurascans.com**: series `/comics/<slug>` (slug carries trailing
|
||||
site-wide build-hash suffix, e.g. `-059befe1`, that **rotates on every
|
||||
redeploy**), chapter `/comics/<slug>/chapter/<n>`. `seriesId` must strip
|
||||
hash (`/-[0-9a-f]{8}$/`, `stripBuildHash` in userscript,
|
||||
`asuraBuildHash` in backend); URLs keep full slug — stale-hash
|
||||
URLs 302 to current ones. Astro-rendered; chapter links present in raw
|
||||
server HTML.
|
||||
- **demonicscans.org**: series `/manga/<slug>` (slug may URL-encode punctuation, e.g. `%2527` for `'`), chapter `/title/<slug>/chapter/<n>/<page>` (older `chaptered.php?manga=<id>&chapter=<n>` form still exists as redirect, what series-page chapter-list anchors link through).
|
||||
Encodings (incl. triple-encoded punctuation like `%25252D`) identical
|
||||
on /manga/ and /title/ pages, so decode-once seriesIds match — verified
|
||||
2026-07-28.
|
||||
- **novelfull.com** (novel script): series `/<slug>.html`, chapter
|
||||
`/<slug>/chapter-<n>[-<title-slug>].html`. No `og:*` tags at all — title from
|
||||
`h3.title` (series) or `a.truyen-title` (chapter), cover from
|
||||
`meta[name="image"]`. Behind a Cloudflare JS challenge no TLS fingerprint
|
||||
clears, so the backend polls it through the headless browser.
|
||||
- **lightnovelworld.net** (novel script): series `/novel/<slug>/`, chapter
|
||||
`/<slug>-chapter-<n>/` — flat, at the site root. `h1.entry-title` is the clean
|
||||
title on a series page and `<Title> Chapter <n>` on a chapter page. Chapter
|
||||
pages carry no `og:image`. Its series page lists every chapter with an
|
||||
absolute href, so the backend polls it with the plain TLS client.
|
||||
|
||||
### Second script: `novel-bookmark.user.js`
|
||||
|
||||
A copy of the manga script with two adapters, `LIBRARY = "novel"` and
|
||||
`STORE_PREFIX = "bmgr:novel:"`. No migration loop (this script has no previous
|
||||
installation to carry keys over from). Installed alongside the manga script;
|
||||
both write to the same backend with the same `LIBRARY` column discriminating
|
||||
them.
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
Reference in New Issue
Block a user