Compare commits

...

2 Commits

Author SHA1 Message Date
sulthan 3f7664ef9b feat(backend): give every Bookmark an owner (Reader table) (#22)
A readers table appears, keyed by Discord user ID and carrying the SHA-256
of the owner's userscript token (the global API token today). Startup seeds
exactly one Reader from OWNER_DISCORD_ID, idempotently, and a run-once
migration (0004, version-table-gated) attaches existing bookmarks to it
before reshaping: the surrogate key column is dropped and bookmarks are
keyed (reader_id, site, series_id) with an FK to readers ON DELETE CASCADE,
so a duplicate bookmark for one Reader and Series is impossible at the
database level.

Every store read and write is now scoped to the reader it names; handlers
act as the seeded owner while the global token remains the only credential.
Authentication and the wire format are untouched: the flat JSON still
carries key/site/series_id, with key derived on read.

OWNER_DISCORD_ID is a new required env var (compose + docs updated).
2026-08-08 07:59:40 +07:00
sulthan 984965ed9f Split Series from Bookmark, keeping the wire format flat (#21) (#29)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-08 07:19:54 +07:00
18 changed files with 1255 additions and 268 deletions
+4
View File
@@ -4,6 +4,10 @@
# openssl rand -hex 32 # openssl rand -hex 32
API_TOKEN=changeme-generate-a-long-random-token API_TOKEN=changeme-generate-a-long-random-token
# The owner's Discord user ID — the one Reader every bookmark belongs to
# (seeded at startup). Discord snowflake, e.g. 1046923170000000000.
OWNER_DISCORD_ID=changeme-your-discord-user-id
# Comma-separated origins allowed to call the API (CORS). Both Asura domains # Comma-separated origins allowed to call the API (CORS). Both Asura domains
# plus Demonic, Comix, Kagane, and the two novel sites. Add/remove as the # plus Demonic, Comix, Kagane, and the two novel sites. Add/remove as the
# sites' hostnames change. # sites' hostnames change.
+6 -1
View File
@@ -35,6 +35,11 @@ Edit `.env`:
# Required — long random secret, also goes in the userscript. # Required — long random secret, also goes in the userscript.
API_TOKEN=<paste output of: openssl rand -hex 32> API_TOKEN=<paste output of: openssl rand -hex 32>
# Required — the owner's Discord user ID. Seeds the one Reader every bookmark
# belongs to; the value is the snowflake in your Discord profile (Settings →
# Advanced → Developer Mode → right-click your name → Copy User ID).
OWNER_DISCORD_ID=<discord user id>
# CORS allowlist — leave as-is unless a site changes hostname. # CORS allowlist — leave as-is unless a site changes hostname.
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to
@@ -250,7 +255,7 @@ it; see `REDEPLOY.md` §1 for when to remove it.)
| `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). | | `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). |
| 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. | | 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. |
| Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. | | Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. |
| `compose ... config` errors about `API_TOKEN` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. Both are required and neither has a fallback. | | `compose ... config` errors about `API_TOKEN`, `OWNER_DISCORD_ID` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. All three are required and none has a fallback. |
| `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). | | `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). |
| `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. | | `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. |
+1
View File
@@ -26,6 +26,7 @@ Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
| Var | Default | Notes | | Var | Default | Notes |
|-----|---------|-------| |-----|---------|-------|
| `API_TOKEN` | *(required)* | Bearer token shared with the userscript. | | `API_TOKEN` | *(required)* | Bearer token shared with the userscript. |
| `OWNER_DISCORD_ID` | *(required)* | Discord user ID of the owner; seeds the one Reader all bookmarks belong to. |
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. | | `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. | | `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. |
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. | | `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
+2 -1
View File
@@ -59,7 +59,7 @@ network can reach it — so every command below goes in through the container:
```bash ```bash
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt' $COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
# -> bookmarks, schema_migrations # -> bookmarks, schema_migrations, series
``` ```
Inside the container that connects over the local socket as the `bookmarks` Inside the container that connects over the local socket as the `bookmarks`
@@ -100,6 +100,7 @@ docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA' pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks # -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
# -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks # -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks
# -> 1236; 0 0 TABLE DATA public series series
# 2. Sanity-check the live row count you just captured. # 2. Sanity-check the live row count you just captured.
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \ $COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
+30 -11
View File
@@ -21,7 +21,22 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
container per test binary (`TestMain` -> `pgtest.Main`) and hands each test container per test binary (`TestMain` -> `pgtest.Main`) and hands each test
its own database (`pgtest.URL(t)`). A package whose tests touch the store its own database (`pgtest.URL(t)`). A package whose tests touch the store
must have that `TestMain`. must have that `TestMain`.
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`), with a `kind` column (`manga`|`novel`) splitting the two libraries. Sync **last-write-wins**. Schema and endpoint list in plan. - **Single-owner store, three tables.** `readers` is keyed by Discord user ID
and carries the SHA-256 of the owner's userscript token (the global
`API_TOKEN` today; issue #22). The seed creates exactly one row at startup.
`series` keyed `(site, series_id)`
(`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`) owns the
shared facts — title, cover, canonical URL, `kind` (`manga`|`novel`),
Latest Chapter, `latest_checked_at` — and `bookmarks` holds only what
differs between readers: progress, favourite, lifecycle bucket,
`updated_at`. A bookmark is keyed `(reader_id, site, series_id)` — no
surrogate id; the wire `key` is derived as `site:series_id` on read — and
every store read/write is scoped to the reader it names. `Store.OwnerID()`
is the seeded owner, which every handler passes while the global token is
still the only credential. Sync **last-write-wins**; the wire format stays
flat (ADR-0004). `Store.Upsert` decomposes one flat body across two tables
and enforces the ownership rule: client `title`/`series_url`/`cover` are
written only when the series row is new (ADR-0003).
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth). - **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
- **Web UI:** same binary serve password-gated browser UI on second - **Web UI:** same binary serve password-gated browser UI on second
hostname — `GET /` (list, or login page when no session), hostname — `GET /` (list, or login page when no session),
@@ -53,22 +68,25 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
network access, so `latest_chapter` stay fresh when user not network access, so `latest_chapter` stay fresh when user not
browsing. Second, parallel signal — userscript keep own browsing. Second, parallel signal — userscript keep own
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged. `maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
Two independent clocks: per-bookmark cooldown (`latest_checked_at` column, Two independent clocks: per-series cooldown (`series.latest_checked_at`,
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval. enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
Row stamped *before* fetch so broken series wait out full The poller walks **Series, not Bookmarks** — a series referenced by several
cooldown instead of retrying every tick, and writes go through bookmarks is fetched once per cycle, and the due queue orders
`Store.Get` + `Store.Upsert` so new chapter never reorders list. `reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
*before* fetch so broken series wait out full cooldown instead of retrying
every tick; found chapter written straight to the series row via
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
order — is never touched.
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
against fingerprint-based blocking; any failure log and skip. kagane and against fingerprint-based blocking; any failure log and skip. kagane and
novelfull sit behind Cloudflare JavaScript challenges the TLS client can't novelfull sit behind Cloudflare JavaScript challenges the TLS client can't
clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and clear, so they are browser-only: fetched over CDP via `BROWSER_WS_URL`, and
simply not polled when that's unset. See simply not polled when that's unset. See
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`. `docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so The poller's series write is a single-column UPDATE
userscript `PUT` that commits between the two can get overwritten by (`Store.SetLatestChapter`), not a read-modify-write of the whole bookmark:
poller's stale re-read — reverting that read progress and, since stored it cannot revert read progress or move `updated_at`, so the old
value now differs, moving `updated_at` and reordering list. Known, stale-re-read race is gone with the Get+Upsert flow.
accepted limitation for single-user deployment, not bug to fix.
- **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4. - **`updated_at` drives list order, so moves only on real reading progress:** server apply its timestamp when row new or `last_chapter_num` changes, else keep stored value — favouriting series or recording newly published chapter must not reorder list. `PUT` therefore returns row **as stored**, clients must adopt that response rather than own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
- **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` | - **Lifecycle buckets:** `status` on each bookmark is `reading` | `archived` |
`finished`, orthogonal to `favorite`. Archived and finished appear only in `finished`, orthogonal to `favorite`. Archived and finished appear only in
@@ -80,7 +98,8 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
`excluded.*` is post-evaluation row and default applied there would `excluded.*` is post-evaluation row and default applied there would
wipe bucket on every PUT from client that predates column. See wipe bucket on every PUT from client that predates column. See
`docs/superpowers/specs/2026-07-27-status-buckets-design.md`. `docs/superpowers/specs/2026-07-27-status-buckets-design.md`.
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), - **Config via env:** `API_TOKEN`, `OWNER_DISCORD_ID` (seeds the owner Reader;
required), `ALLOWED_ORIGINS` (comma list),
`DATABASE_URL` (Postgres connection URL, required — no default), `DATABASE_URL` (Postgres connection URL, required — no default),
`PORT` (default `8080`), `WEB_PASSWORD` `PORT` (default `8080`), `WEB_PASSWORD`
(gates browser UI; unset disable it), (gates browser UI; unset disable it),
+138 -7
View File
@@ -2,6 +2,7 @@ package main
import ( import (
"bytes" "bytes"
"crypto/sha256"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
@@ -35,7 +36,9 @@ func newTestServer(t *testing.T) http.Handler {
func newTestStore(t *testing.T) *store.Store { func newTestStore(t *testing.T) *store.Store {
t.Helper() t.Helper()
s, err := store.Open(pgtest.URL(t)) s, err := store.Open(pgtest.URL(t), store.Owner{
DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("owner-token-hash")),
})
if err != nil { if err != nil {
t.Fatalf("store.Open: %v", err) t.Fatalf("store.Open: %v", err)
} }
@@ -50,26 +53,35 @@ func auth(req *http.Request) *http.Request {
func floatPtr(f float64) *float64 { return &f } func floatPtr(f float64) *float64 { return &f }
// seedForCheck inserts a bookmark and forces its latest_checked_at. // seedForCheck inserts a bookmark (and with it its series) and forces the
// series' latest_checked_at.
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) { func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
t.Helper() t.Helper()
if _, err := s.Upsert(store.Bookmark{ site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Key: key,
Site: "asura", Site: site,
SeriesID: key, SeriesID: seriesID,
SeriesURL: seriesURL, SeriesURL: seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed %q: %v", key, err) t.Fatalf("seed %q: %v", key, err)
} }
if err := s.MarkLatestChecked(key, checkedAt); err != nil { if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
t.Fatalf("seed mark %q: %v", key, err) t.Fatalf("seed mark %q: %v", key, err)
} }
} }
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 { func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
t.Helper() t.Helper()
ts, err := s.LatestCheckedAt(key) site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
ts, err := s.LatestCheckedAt(site, seriesID)
if err != nil { if err != nil {
t.Fatalf("LatestCheckedAt %q: %v", key, err) t.Fatalf("LatestCheckedAt %q: %v", key, err)
} }
@@ -229,6 +241,125 @@ func TestBookmarkRoundTrip(t *testing.T) {
} }
} }
// The wire contract (ADR-0004): GET and PUT speak exactly the flat field set
// they always did, with the series-owned fields as siblings of the bookmark
// fields, not nested. Asserted as a key set, not by inspection.
func TestFlatWireFieldSet(t *testing.T) {
srv := newTestServer(t)
key := "comix:some-title"
in := store.Bookmark{
Key: key,
Site: "comix",
SeriesID: "some-title",
Title: "Some Title",
SeriesURL: "https://comix.to/title/some-title",
Cover: "https://comix.to/covers/some-title.jpg",
LastChapter: "Chapter 7",
LastChapterNum: 7,
LastChapterURL: "https://comix.to/title/some-title/ch/7",
Favorite: true,
LatestChapter: "Chapter 8",
LatestChapterNum: floatPtr(8),
Status: store.StatusArchived,
Kind: store.KindManga,
}
body, _ := json.Marshal(in)
wantKeys := map[string]bool{
"key": true, "site": true, "series_id": true, "title": true,
"series_url": true, "cover": true, "last_chapter": true,
"last_chapter_num": true, "last_chapter_url": true, "favorite": true,
"latest_chapter": true, "latest_chapter_num": true, "updated_at": true,
"status": true, "kind": true,
}
checkFlat := func(t *testing.T, payload []byte) map[string]json.RawMessage {
t.Helper()
var obj map[string]json.RawMessage
if err := json.Unmarshal(payload, &obj); err != nil {
t.Fatalf("decode: %v", err)
}
if len(obj) != len(wantKeys) {
t.Fatalf("field count = %d, want %d (%s)", len(obj), len(wantKeys), payload)
}
for k := range obj {
if !wantKeys[k] {
t.Fatalf("unexpected field %q", k)
}
}
return obj
}
// PUT
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, auth(httptest.NewRequest(http.MethodPut, "/bookmarks/"+key, bytes.NewReader(body))))
if rr.Code != http.StatusOK {
t.Fatalf("PUT status = %d, want 200", rr.Code)
}
checkFlat(t, rr.Body.Bytes())
// Every field round-trips with its value, and updated_at is server-stamped.
var stored store.Bookmark
if err := json.Unmarshal(rr.Body.Bytes(), &stored); err != nil {
t.Fatalf("decode PUT response: %v", err)
}
latestNum := floatPtr(8)
want := store.Bookmark{
Key: key, Site: "comix", SeriesID: "some-title",
Title: in.Title, SeriesURL: in.SeriesURL, Cover: in.Cover,
LastChapter: in.LastChapter, LastChapterNum: in.LastChapterNum,
LastChapterURL: in.LastChapterURL, Favorite: true,
LatestChapter: in.LatestChapter, LatestChapterNum: latestNum,
Status: store.StatusArchived, Kind: store.KindManga,
}
if stored.Title != want.Title || stored.SeriesURL != want.SeriesURL || stored.Cover != want.Cover ||
stored.LastChapter != want.LastChapter || stored.LastChapterNum != want.LastChapterNum ||
stored.LastChapterURL != want.LastChapterURL || stored.Favorite != want.Favorite ||
stored.LatestChapter != want.LatestChapter ||
stored.LatestChapterNum == nil || *stored.LatestChapterNum != *want.LatestChapterNum ||
stored.Status != want.Status || stored.Kind != want.Kind {
t.Fatalf("PUT response = %+v, want %+v", stored, want)
}
if stored.UpdatedAt == 0 {
t.Fatal("updated_at not server-stamped")
}
// GET reports the same flat shape.
list := getBookmarks(t, srv)
if len(list) != 1 {
t.Fatalf("list = %d items, want 1", len(list))
}
body2, _ := json.Marshal(list[0])
checkFlat(t, body2)
}
// A PUT naming an existing series must ignore client-supplied title, cover and
// URL — the security boundary from ADR-0003, where a hostile site's scraped
// values could otherwise land on a shared row — while progress still lands.
func TestPutExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
srv := newTestServer(t)
key := "asura:solo"
first := putBookmark(t, srv, key, store.Bookmark{
Title: "Solo Leveling",
SeriesURL: "https://asurascans.com/comics/solo",
Cover: "https://asurascans.com/covers/solo.jpg",
LastChapterNum: 10,
})
second := putBookmark(t, srv, key, store.Bookmark{
Title: "Scraped Rename",
SeriesURL: "https://evil.example/solo",
Cover: "https://evil.example/solo.jpg",
LastChapterNum: 11,
})
if second.Title != first.Title || second.SeriesURL != first.SeriesURL || second.Cover != first.Cover {
t.Fatalf("stored = %+v, want original title/url/cover kept", second)
}
if second.LastChapterNum != 11 {
t.Fatalf("LastChapterNum = %v, want 11 — progress must still land", second.LastChapterNum)
}
}
// putBookmark PUTs b at key and returns the bookmark the server echoes back, // putBookmark PUTs b at key and returns the bookmark the server echoes back,
// which is the row as actually stored (not the request payload). // which is the row as actually stored (not the request payload).
func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark { func putBookmark(t *testing.T, srv http.Handler, key string, b store.Bookmark) store.Bookmark {
+7 -4
View File
@@ -13,6 +13,9 @@ import (
// Handler serves the userscript-facing JSON bookmark API. // Handler serves the userscript-facing JSON bookmark API.
type Handler struct { type Handler struct {
Store *store.Store Store *store.Store
// ReaderID is the Reader this request acts as. Authentication is still the
// single global token, so that is always the seeded owner (issue #22).
ReaderID int64
} }
func writeJSON(w http.ResponseWriter, status int, v any) { func writeJSON(w http.ResponseWriter, status int, v any) {
@@ -25,9 +28,9 @@ func writeJSON(w http.ResponseWriter, status int, v any) {
} }
} }
// List returns all bookmarks. GET /bookmarks // List returns all bookmarks of the acting Reader. GET /bookmarks
func (h *Handler) List(w http.ResponseWriter, r *http.Request) { func (h *Handler) List(w http.ResponseWriter, r *http.Request) {
items, err := h.Store.List() items, err := h.Store.List(h.ReaderID)
if err != nil { if err != nil {
log.Printf("list: %v", err) log.Printf("list: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
@@ -91,7 +94,7 @@ func (h *Handler) Put(w http.ResponseWriter, r *http.Request) {
// reading progress actually moved. Any client value is ignored. // reading progress actually moved. Any client value is ignored.
b.UpdatedAt = time.Now().UnixMilli() b.UpdatedAt = time.Now().UnixMilli()
stored, err := h.Store.Upsert(b) stored, err := h.Store.Upsert(h.ReaderID, b)
if err != nil { if err != nil {
log.Printf("upsert: %v", err) log.Printf("upsert: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
@@ -109,7 +112,7 @@ func (h *Handler) Delete(w http.ResponseWriter, r *http.Request) {
http.Error(w, "missing key", http.StatusBadRequest) http.Error(w, "missing key", http.StatusBadRequest)
return return
} }
if err := h.Store.Delete(key); err != nil { if err := h.Store.Delete(h.ReaderID, key); err != nil {
log.Printf("delete: %v", err) log.Printf("delete: %v", err)
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
return return
+29 -46
View File
@@ -23,9 +23,9 @@ type Fetcher interface {
// Two clocks, deliberately independent: // Two clocks, deliberately independent:
// //
// - Interval is how often this goroutine wakes up and looks. // - Interval is how often this goroutine wakes up and looks.
// - Cooldown is how long one bookmark rests since its own last check. // - Cooldown is how long one series rests since its own last check.
// //
// Only the cooldown is per bookmark, and it is enforced by the WHERE clause in // Only the cooldown is per series, and it is enforced by the WHERE clause in
// DueForLatestCheck rather than by any timer. Shortening Interval therefore // DueForLatestCheck rather than by any timer. Shortening Interval therefore
// cannot shorten anyone's cooldown; it only makes the poller wake up and find // cannot shorten anyone's cooldown; it only makes the poller wake up and find
// nothing due more often. // nothing due more often.
@@ -78,7 +78,7 @@ func (p *Poller) Run(ctx context.Context) {
} }
} }
// runOnce processes one batch of due bookmarks. // runOnce processes one batch of due series.
func (p *Poller) runOnce(ctx context.Context) { func (p *Poller) runOnce(ctx context.Context) {
cutoff := p.Now().Add(-p.Cooldown).UnixMilli() cutoff := p.Now().Add(-p.Cooldown).UnixMilli()
due, err := p.Store.DueForLatestCheck(cutoff, p.Batch) due, err := p.Store.DueForLatestCheck(cutoff, p.Batch)
@@ -88,7 +88,7 @@ func (p *Poller) runOnce(ctx context.Context) {
} }
checked := 0 checked := 0
for i, b := range due { for i, sr := range due {
if ctx.Err() != nil { if ctx.Err() != nil {
break break
} }
@@ -107,7 +107,7 @@ func (p *Poller) runOnce(ctx context.Context) {
if stopped { if stopped {
break break
} }
p.checkOne(ctx, b) p.checkOne(ctx, sr)
checked++ checked++
} }
// due vs checked is how you tell which constraint is binding: ticks that // due vs checked is how you tell which constraint is binding: ticks that
@@ -119,10 +119,10 @@ func (p *Poller) runOnce(ctx context.Context) {
// checkOne re-checks one series. Every failure path here is "log and move on": // checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a // the poller is a best-effort enhancement, and no single bad series may stall a
// batch or take down the process. // batch or take down the process.
func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) { func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
defer func() { defer func() {
if r := recover(); r != nil { if r := recover(); r != nil {
log.Printf("latest poll %q: recovered from panic: %v", b.Key, r) log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
} }
}() }()
@@ -130,8 +130,8 @@ func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
// mid-request still consumes the cooldown. Otherwise a renamed or deleted // mid-request still consumes the cooldown. Otherwise a renamed or deleted
// series would be retried on every single tick forever. The userscript // series would be retried on every single tick forever. The userscript
// stamps in the same order and for the same reason (L471-473). // stamps in the same order and for the same reason (L471-473).
if err := p.Store.MarkLatestChecked(b.Key, p.Now().UnixMilli()); err != nil { if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: mark checked: %v", b.Key, err) log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
return return
} }
@@ -142,69 +142,52 @@ func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
// link-local/internal addresses or non-https schemes. The cooldown above // link-local/internal addresses or non-https schemes. The cooldown above
// is already consumed, so a row that never passes this check is retried at // is already consumed, so a row that never passes this check is retried at
// cooldown pace rather than hot-looping. // cooldown pace rather than hot-looping.
if !fetchableSeriesURL(b.Site, b.SeriesURL) { if !fetchableSeriesURL(sr.Site, sr.SeriesURL) {
log.Printf("latest poll %q: not fetchable: site=%q url=%q", b.Key, b.Site, b.SeriesURL) log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
return return
} }
f := p.fetcherFor(b.Site) f := p.fetcherFor(sr.Site)
if f == nil { if f == nil {
log.Printf("latest poll %q: no fetcher for site %q", b.Key, b.Site) log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
return return
} }
body, status, err := f.Get(ctx, b.SeriesURL) body, status, err := f.Get(ctx, sr.SeriesURL)
if err != nil { if err != nil {
log.Printf("latest poll %q: fetch %s: %v", b.Key, b.SeriesURL, err) log.Printf("latest poll %q: fetch %s: %v", sr.Key(), sr.SeriesURL, err)
return return
} }
if status != 200 { if status != 200 {
log.Printf("latest poll %q: fetch %s: status %d", b.Key, b.SeriesURL, status) log.Printf("latest poll %q: fetch %s: status %d", sr.Key(), sr.SeriesURL, status)
return return
} }
latest, ok := latestChapterFrom(b.Site, b.SeriesURL, body) latest, ok := latestChapterFrom(sr.Site, sr.SeriesURL, body)
if !ok { if !ok {
// Most likely a challenge page or a layout change. Either way the row is // Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a cooldown instead of hot-looping. // already stamped, so this waits out a cooldown instead of hot-looping.
log.Printf("latest poll %q: no chapter links in %d bytes", b.Key, len(body)) log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), len(body))
return return
} }
// Re-read: the row may have been updated or deleted while the fetch was in
// flight, and writing b back wholesale would undo that.
//
// ponytail: non-transactional read-modify-write, wrap Get+Upsert in a tx if
// this ever runs for more than one user. A client PUT that commits between
// these two statements is lost to the stale re-read — reverting read
// progress or a status change, and moving updated_at because the stored
// value now differs. Accepted for a single-user deployment: the window is
// milliseconds and the loser is one poll cycle.
cur, found, err := p.Store.Get(b.Key)
if err != nil {
log.Printf("latest poll %q: reread: %v", b.Key, err)
return
}
if !found {
return
}
// Equality, not >, mirroring the userscript (L427): a site that retracts a // Equality, not >, mirroring the userscript (L427): a site that retracts a
// chapter should correct the stored number downward. // chapter should correct the stored number downward. The comparison is
if cur.LatestChapterNum != nil && *cur.LatestChapterNum == latest.Num { // against the due-query snapshot; a concurrent write in between only costs
// one redundant UPDATE of the same absolute value, never a wrong one.
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == latest.Num {
return return
} }
num := latest.Num // Series-level write: the row is shared, so one update refreshes every
cur.LatestChapter = latest.Label // bookmark joining to it, and the bookmark's updated_at is never touched —
cur.LatestChapterNum = &num // a newly published chapter is not reading progress and must not reorder
// A candidate only. last_chapter_num is untouched, so the CASE in Upsert // the list.
// keeps the stored updated_at and the bookmark list does not reorder. if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, latest.Label, latest.Num); err != nil {
cur.UpdatedAt = p.Now().UnixMilli() log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
if _, err := p.Store.Upsert(cur); err != nil {
log.Printf("latest poll %q: upsert: %v", b.Key, err)
return return
} }
log.Printf("latest poll %q: latest is now %s", b.Key, latest.Label) log.Printf("latest poll %q: latest is now %s", sr.Key(), latest.Label)
} }
// fetchableSeriesURL reports whether site is a site latestChapterFrom knows how // fetchableSeriesURL reports whether site is a site latestChapterFrom knows how
+95 -35
View File
@@ -2,8 +2,10 @@ package latest
import ( import (
"context" "context"
"crypto/sha256"
"errors" "errors"
"os" "os"
"strings"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -14,37 +16,53 @@ import (
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
// newTestStore opens a store on a Postgres database of this test's own. // testOwner is the owner every test store seeds. A second reader, where a
func newTestStore(t *testing.T) *store.Store { // test needs one, is created by opening the same database as a second owner.
var testOwner = store.Owner{DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("owner-token-hash"))}
// newTestStore opens a store on a Postgres database of this test's own and
// returns the URL, for helpers that need a second connection to the same
// database (see TestRunOnceFetchesSharedSeriesOnce).
func newTestStore(t *testing.T) (*store.Store, string) {
t.Helper() t.Helper()
s, err := store.Open(pgtest.URL(t)) url := pgtest.URL(t)
s, err := store.Open(url, testOwner)
if err != nil { if err != nil {
t.Fatalf("Open: %v", err) t.Fatalf("Open: %v", err)
} }
t.Cleanup(func() { s.Close() }) t.Cleanup(func() { s.Close() })
return s return s, url
} }
// seedForCheck inserts a bookmark and forces its latest_checked_at. // seedForCheck inserts a bookmark (and with it its series) and forces the
// series' latest_checked_at.
func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) { func seedForCheck(t *testing.T, s *store.Store, key, seriesURL string, checkedAt int64) {
t.Helper() t.Helper()
if _, err := s.Upsert(store.Bookmark{ site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Key: key,
Site: "asura", Site: site,
SeriesID: key, SeriesID: seriesID,
SeriesURL: seriesURL, SeriesURL: seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed %q: %v", key, err) t.Fatalf("seed %q: %v", key, err)
} }
if err := s.MarkLatestChecked(key, checkedAt); err != nil { if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
t.Fatalf("seed mark %q: %v", key, err) t.Fatalf("seed mark %q: %v", key, err)
} }
} }
func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 { func readLatestCheckedAt(t *testing.T, s *store.Store, key string) int64 {
t.Helper() t.Helper()
ts, err := s.LatestCheckedAt(key) site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
ts, err := s.LatestCheckedAt(site, seriesID)
if err != nil { if err != nil {
t.Fatalf("LatestCheckedAt %q: %v", key, err) t.Fatalf("LatestCheckedAt %q: %v", key, err)
} }
@@ -101,7 +119,7 @@ func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Polle
} }
func TestRunOnceRecordsLatestChapter(t *testing.T) { func TestRunOnceRecordsLatestChapter(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af" const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
seedForCheck(t, s, "asura:chronicles-of-the-demon-faction-f886a8af", url, 0) seedForCheck(t, s, "asura:chronicles-of-the-demon-faction-f886a8af", url, 0)
@@ -109,7 +127,7 @@ func TestRunOnceRecordsLatestChapter(t *testing.T) {
f := &fakeFetcher{body: asuraSeriesFixture, status: 200} f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, now).runOnce(context.Background()) newTestPoller(t, s, f, now).runOnce(context.Background())
b, ok, err := s.Get("asura:chronicles-of-the-demon-faction-f886a8af") b, ok, err := s.Get(s.OwnerID(), "asura:chronicles-of-the-demon-faction-f886a8af")
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("Get: %v ok=%v", err, ok) t.Fatalf("Get: %v ok=%v", err, ok)
} }
@@ -127,19 +145,19 @@ func TestRunOnceRecordsLatestChapter(t *testing.T) {
// The whole point of the updated_at CASE in Upsert: a newly published chapter is // The whole point of the updated_at CASE in Upsert: a newly published chapter is
// not reading progress and must not move the series up the list. // not reading progress and must not move the series up the list.
func TestRunOnceDoesNotReorderList(t *testing.T) { func TestRunOnceDoesNotReorderList(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af" const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
const key = "asura:chronicles-of-the-demon-faction-f886a8af" const key = "asura:chronicles-of-the-demon-faction-f886a8af"
// "other" is the most recently read, so it must stay at the top of List(). // "other" is the most recently read, so it must stay at the top of List().
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: "asura:other", Site: "asura", SeriesID: "other", Key: "asura:other", Site: "asura", SeriesID: "other",
SeriesURL: "https://asurascans.com/comics/other", UpdatedAt: 9_000_000, SeriesURL: "https://asurascans.com/comics/other", UpdatedAt: 9_000_000,
}); err != nil { }); err != nil {
t.Fatalf("seed other: %v", err) t.Fatalf("seed other: %v", err)
} }
seedForCheck(t, s, key, url, 0) seedForCheck(t, s, key, url, 0)
before, _, err := s.Get(key) before, _, err := s.Get(s.OwnerID(), key)
if err != nil { if err != nil {
t.Fatalf("Get before: %v", err) t.Fatalf("Get before: %v", err)
} }
@@ -147,7 +165,7 @@ func TestRunOnceDoesNotReorderList(t *testing.T) {
f := &fakeFetcher{body: asuraSeriesFixture, status: 200} f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, time.UnixMilli(9_999_999)).runOnce(context.Background()) newTestPoller(t, s, f, time.UnixMilli(9_999_999)).runOnce(context.Background())
after, _, err := s.Get(key) after, _, err := s.Get(s.OwnerID(), key)
if err != nil { if err != nil {
t.Fatalf("Get after: %v", err) t.Fatalf("Get after: %v", err)
} }
@@ -155,7 +173,7 @@ func TestRunOnceDoesNotReorderList(t *testing.T) {
t.Fatalf("updated_at moved from %d to %d on a latest-chapter bump", t.Fatalf("updated_at moved from %d to %d on a latest-chapter bump",
before.UpdatedAt, after.UpdatedAt) before.UpdatedAt, after.UpdatedAt)
} }
list, err := s.List() list, err := s.List(s.OwnerID())
if err != nil { if err != nil {
t.Fatalf("List: %v", err) t.Fatalf("List: %v", err)
} }
@@ -178,7 +196,7 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
const url = "https://asurascans.com/comics/x" const url = "https://asurascans.com/comics/x"
seedForCheck(t, s, "asura:x", url, 0) seedForCheck(t, s, "asura:x", url, 0)
@@ -189,7 +207,7 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
if got := readLatestCheckedAt(t, s, "asura:x"); got != now.UnixMilli() { if got := readLatestCheckedAt(t, s, "asura:x"); got != now.UnixMilli() {
t.Fatalf("latest_checked_at = %d, want %d", got, now.UnixMilli()) t.Fatalf("latest_checked_at = %d, want %d", got, now.UnixMilli())
} }
b, _, err := s.Get("asura:x") b, _, err := s.Get(s.OwnerID(), "asura:x")
if err != nil { if err != nil {
t.Fatalf("Get: %v", err) t.Fatalf("Get: %v", err)
} }
@@ -201,7 +219,7 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
} }
func TestRunOnceRespectsBatchLimit(t *testing.T) { func TestRunOnceRespectsBatchLimit(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
for i := 0; i < 20; i++ { for i := 0; i < 20; i++ {
key := "asura:s" + string(rune('a'+i)) key := "asura:s" + string(rune('a'+i))
seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0) seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0)
@@ -217,9 +235,51 @@ func TestRunOnceRespectsBatchLimit(t *testing.T) {
} }
} }
// The point of the split (ADR-0003): a series referenced by several bookmarks
// is fetched once per due cycle, not once per bookmark. Two bookmarks share a
// series when two readers track it (issue #22).
func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
s, url := newTestStore(t)
// The slug must match the fixture's own anchors: asura's parser scopes
// chapter links to the stored slug.
const slug = "chronicles-of-the-demon-faction-f886a8af"
const seriesURL = "https://asurascans.com/comics/" + slug
seedForCheck(t, s, "asura:"+slug, seriesURL, 0)
// A second reader tracks the same series. The seed is the only
// reader-creation path, so a second Open as a different owner is how a
// test gets a second reader on the same database.
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))})
if err != nil {
t.Fatalf("Open second reader: %v", err)
}
t.Cleanup(func() { other.Close() })
if _, err := s.Upsert(other.OwnerID(), store.Bookmark{
Key: "asura:" + slug, Site: "asura", SeriesID: slug, UpdatedAt: 2000,
}); err != nil {
t.Fatalf("seed second reader: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background())
if got := f.callCount(); got != 1 {
t.Fatalf("fetched shared series %d times, want 1", got)
}
// Both bookmarks join to the same updated series row.
for _, st := range []*store.Store{s, other} {
b, ok, err := st.Get(st.OwnerID(), "asura:"+slug)
if err != nil || !ok {
t.Fatalf("Get: %v ok=%v", err, ok)
}
if b.LatestChapterNum == nil || *b.LatestChapterNum != 181 {
t.Fatalf("LatestChapterNum = %v, want 181", b.LatestChapterNum)
}
}
}
// One unreachable series must not abandon the rest of the batch. // One unreachable series must not abandon the rest of the batch.
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) { func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"} keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"}
for _, k := range keys { for _, k := range keys {
seedForCheck(t, s, k, "https://asurascans.com/comics/"+k, 0) seedForCheck(t, s, k, "https://asurascans.com/comics/"+k, 0)
@@ -247,7 +307,7 @@ func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
// The cooldown is enforced by the due query, so a second immediate pass must do // The cooldown is enforced by the due query, so a second immediate pass must do
// nothing at all — this is what makes the tick interval independent of it. // nothing at all — this is what makes the tick interval independent of it.
func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) { func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
const url = "https://asurascans.com/comics/x" const url = "https://asurascans.com/comics/x"
seedForCheck(t, s, "asura:x", url, 0) seedForCheck(t, s, "asura:x", url, 0)
@@ -277,12 +337,12 @@ func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
// A site that retracts a chapter should correct the stored number downward, // A site that retracts a chapter should correct the stored number downward,
// mirroring the userscript's equality check (L427) rather than a >. // mirroring the userscript's equality check (L427) rather than a >.
func TestRunOnceCorrectsDownward(t *testing.T) { func TestRunOnceCorrectsDownward(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
const url = "https://demonicscans.org/manga/Catastrophic-Necromancer" const url = "https://demonicscans.org/manga/Catastrophic-Necromancer"
const key = "demonic:Catastrophic-Necromancer" const key = "demonic:Catastrophic-Necromancer"
high := 400.0 high := 400.0
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "demonic", SeriesID: "Catastrophic-Necromancer", Key: key, Site: "demonic", SeriesID: "Catastrophic-Necromancer",
SeriesURL: url, LatestChapter: "Chapter 400", LatestChapterNum: &high, SeriesURL: url, LatestChapter: "Chapter 400", LatestChapterNum: &high,
UpdatedAt: 1000, UpdatedAt: 1000,
@@ -293,7 +353,7 @@ func TestRunOnceCorrectsDownward(t *testing.T) {
f := &fakeFetcher{body: demonicSeriesFixture, status: 200} f := &fakeFetcher{body: demonicSeriesFixture, status: 200}
newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background()) newTestPoller(t, s, f, time.UnixMilli(5_000_000)).runOnce(context.Background())
b, _, err := s.Get(key) b, _, err := s.Get(s.OwnerID(), key)
if err != nil { if err != nil {
t.Fatalf("Get: %v", err) t.Fatalf("Get: %v", err)
} }
@@ -319,9 +379,9 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
key := tt.site + ":x" key := tt.site + ":x"
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL, Key: key, Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
@@ -330,8 +390,8 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
now := time.UnixMilli(4_000_000) now := time.UnixMilli(4_000_000)
f := &fakeFetcher{body: asuraSeriesFixture, status: 200} f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
newTestPoller(t, s, f, now).checkOne(context.Background(), store.Bookmark{ newTestPoller(t, s, f, now).checkOne(context.Background(), store.Series{
Key: key, Site: tt.site, SeriesURL: tt.seriesURL, Site: tt.site, SeriesID: "x", SeriesURL: tt.seriesURL,
}) })
if got := f.callCount(); got != tt.wantCalls { if got := f.callCount(); got != tt.wantCalls {
@@ -346,7 +406,7 @@ func TestCheckOneValidatesSeriesURLBeforeFetching(t *testing.T) {
// A cancelled context must abandon the batch rather than run it to completion. // A cancelled context must abandon the batch rather than run it to completion.
func TestRunOnceStopsOnCancelledContext(t *testing.T) { func TestRunOnceStopsOnCancelledContext(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
for _, k := range []string{"asura:a", "asura:b", "asura:c"} { for _, k := range []string{"asura:a", "asura:b", "asura:c"} {
seedForCheck(t, s, k, "https://asurascans.com/comics/"+k, 0) seedForCheck(t, s, k, "https://asurascans.com/comics/"+k, 0)
} }
@@ -395,8 +455,8 @@ func TestFetchableSeriesURL(t *testing.T) {
// receive a challenge page, and the browser fetcher is the whole reason kagane // receive a challenge page, and the browser fetcher is the whole reason kagane
// is pollable at all. // is pollable at all.
func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) { func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", Key: "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b",
Site: "kagane", Site: "kagane",
SeriesID: "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", SeriesID: "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b",
@@ -421,9 +481,9 @@ func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
// With a browser fetcher wired up, kagane goes to it and not to the TLS one. // With a browser fetcher wired up, kagane goes to it and not to the TLS one.
func TestKaganeUsesBrowserFetcher(t *testing.T) { func TestKaganeUsesBrowserFetcher(t *testing.T) {
s := newTestStore(t) s, _ := newTestStore(t)
key := "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b" key := "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
if _, err := s.Upsert(store.Bookmark{ if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Key: key,
Site: "kagane", Site: "kagane",
SeriesID: "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", SeriesID: "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b",
@@ -448,7 +508,7 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
if len(browserF.calls) != 1 { if len(browserF.calls) != 1 {
t.Fatalf("browser fetcher calls = %v, want 1", browserF.calls) t.Fatalf("browser fetcher calls = %v, want 1", browserF.calls)
} }
got, found, err := s.Get(key) got, found, err := s.Get(s.OwnerID(), key)
if err != nil || !found { if err != nil || !found {
t.Fatalf("Get: %v found=%v", err, found) t.Fatalf("Get: %v found=%v", err, found)
} }
@@ -0,0 +1,45 @@
-- One row per distinct work, shared by every bookmark that tracks it
-- (ADR-0003). Keyed (site, series_id), the pair a bookmark key decomposes
-- into. title/series_url/cover are written once, at creation, and never
-- again: client-supplied values are ignored once the row exists and the
-- poller is the only party that may change them. kind and the latest-chapter
-- fields are last-write-wins like the bookmark's own fields.
CREATE TABLE series (
site text NOT NULL,
series_id text NOT NULL,
title text NOT NULL DEFAULT '',
series_url text NOT NULL DEFAULT '',
cover text NOT NULL DEFAULT '',
kind text NOT NULL DEFAULT 'manga',
latest_chapter text NOT NULL DEFAULT '',
latest_chapter_num double precision,
-- When the server last polled this series, unix ms; 0 means never, and sorts
-- first so a new bookmark is picked up on the next tick with no special case.
latest_checked_at bigint NOT NULL DEFAULT 0,
PRIMARY KEY (site, series_id)
);
-- Backfill from today's rows. The bookmark key's uniqueness makes
-- (site, series_id) unique in practice; DISTINCT is belt and braces.
INSERT INTO series (site, series_id, title, series_url, cover, kind,
latest_chapter, latest_chapter_num, latest_checked_at)
SELECT DISTINCT site, series_id, title, series_url, cover, kind,
latest_chapter, latest_chapter_num, latest_checked_at
FROM bookmarks;
-- The bookmark keeps only what differs between readers (ADR-0003): progress,
-- favourite, lifecycle bucket. The dropped columns now live on series.
ALTER TABLE bookmarks
DROP COLUMN title,
DROP COLUMN series_url,
DROP COLUMN cover,
DROP COLUMN kind,
DROP COLUMN latest_chapter,
DROP COLUMN latest_chapter_num,
DROP COLUMN latest_checked_at;
-- A bookmark may not point at a series that does not exist. No cascade: a
-- series outlives its last bookmark, and deleting one is not a store operation.
ALTER TABLE bookmarks
ADD CONSTRAINT bookmarks_series_fk
FOREIGN KEY (site, series_id) REFERENCES series (site, series_id);
@@ -0,0 +1,17 @@
-- One row per person. Keyed by their Discord user ID; carries the SHA-256 of
-- their userscript token and when they were created. Hashed because a token
-- in the database is a token anyone with the database can replay; SHA-256 is
-- enough because the tokens are high-entropy random values with nothing to
-- brute-force. No one can register yet, so this table holds exactly the one
-- owner row the seed creates at startup (see Store.Open).
CREATE TABLE readers (
id bigserial PRIMARY KEY,
discord_id text NOT NULL UNIQUE,
token_sha256 bytea NOT NULL UNIQUE,
created_at timestamptz NOT NULL DEFAULT now()
);
-- Every bookmark now belongs to a reader. Added nullable: rows created before
-- this migration have no owner yet — 0004 attaches them to the seeded owner
-- before NOT NULL and the composite key land.
ALTER TABLE bookmarks ADD COLUMN reader_id bigint;
@@ -0,0 +1,19 @@
-- Attach every pre-existing bookmark to the owner reader, seeded between the
-- two migrate passes (Store.Open). The oldest reader is the owner by
-- construction: only the seed creates readers, and it runs once per database.
-- Run-once via the version table, like every migration.
UPDATE bookmarks SET reader_id = (SELECT id FROM readers ORDER BY id LIMIT 1);
-- Ownership lands structurally: reader_id becomes part of the key, so a
-- bookmark is one Reader's progress on one Series and a duplicate for the
-- same pair is impossible at the database level. Deleting a Reader takes
-- their bookmarks with them. The old text key is gone — the wire "key" is
-- derived as site:series_id on read, and nothing references the column.
-- Dropping it drops the primary key it carried; the composite key replaces
-- it, and the FK index the series constraint needs is created automatically.
ALTER TABLE bookmarks
ALTER COLUMN reader_id SET NOT NULL,
DROP COLUMN key,
ADD PRIMARY KEY (reader_id, site, series_id),
ADD CONSTRAINT bookmarks_reader_fk
FOREIGN KEY (reader_id) REFERENCES readers (id) ON DELETE CASCADE;
+287 -96
View File
@@ -19,6 +19,11 @@ import (
// //
// LastChapter* is the user's read progress; LatestChapter* is the newest // LastChapter* is the user's read progress; LatestChapter* is the newest
// chapter the site has published, captured opportunistically by the userscript. // chapter the site has published, captured opportunistically by the userscript.
//
// Title, SeriesURL, Cover, Kind and LatestChapter* live on the shared Series
// row (ADR-0003) and are joined in on read; Bookmark carries only what differs
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
// format stays flat regardless — see ADR-0004.
type Bookmark struct { type Bookmark struct {
Key string `json:"key"` Key string `json:"key"`
Site string `json:"site"` Site string `json:"site"`
@@ -42,6 +47,35 @@ type Bookmark struct {
Kind string `json:"kind"` Kind string `json:"kind"`
} }
// Series is one distinct work, shared by every bookmark that tracks it. It is
// keyed (site, series_id) — the pair a bookmark key decomposes into — and
// exists once no matter how many bookmarks point at it (ADR-0003).
//
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
// existing Series has them ignored, and only the backend's own Poll may change
// them. Kind and the latest-chapter fields are last-write-wins like the
// bookmark's own fields. Never serialized: the wire format is the flat
// Bookmark (ADR-0004).
type Series struct {
Site string
SeriesID string
Title string
SeriesURL string
Cover string
Kind string
LatestChapter string
LatestChapterNum *float64 // nil until first captured
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
// readerCount is the number of bookmarks referencing this series, filled
// only by the due-queue query that orders on it.
readerCount int
}
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
// used by the poller's logs and by tests asserting on the due queue.
func (s Series) Key() string { return s.Site + ":" + s.SeriesID }
// HasNewChapter reports whether the site has published past the read point. // HasNewChapter reports whether the site has published past the read point.
// A nil LatestChapterNum means nothing has been captured yet, which is not the // A nil LatestChapterNum means nothing has been captured yet, which is not the
// same as "nothing new". // same as "nothing new".
@@ -122,37 +156,104 @@ const (
var migrations embed.FS var migrations embed.FS
// bookmarkColumns is the only value ever concatenated into query text. It is a // bookmarkColumns is the only value ever concatenated into query text. It is a
// compile-time constant; every request value is bound as a parameter. // compile-time constant; every request value is bound as a parameter. The
const bookmarkColumns = `key, site, series_id, title, series_url, cover, // series-owned fields are joined in from the series table, in scanBookmark
last_chapter, last_chapter_num, last_chapter_url, // order, so the flat Bookmark reads back whole despite the split (ADR-0004).
favorite, latest_chapter, latest_chapter_num, updated_at, status, kind` const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover,
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
// seriesColumns is the series row in scanSeries order, used by the poller's
// due query. latest_checked_at lives only on series — see MarkLatestChecked
// for why it stays off every client-visible write.
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
// Owner is the person running the service: the first Reader, and the only one
// until registration exists. The seed makes sure exactly one readers row
// matches their Discord ID, carrying the SHA-256 of their userscript token —
// which today is the global API token.
type Owner struct {
DiscordID string
// TokenHash is the SHA-256 of the userscript token; the array shape makes
// it a compile error to store anything that is not a hash.
TokenHash [32]byte
}
// Store is the Postgres-backed bookmark store. // Store is the Postgres-backed bookmark store.
type Store struct { type Store struct {
db *sql.DB db *sql.DB
// ownerID is the seeded owner Reader (issue #22). Authentication is still
// the single global token, so every request acts as this Reader; the store
// methods take the id explicitly so the scoping survives per-Reader auth.
ownerID int64
} }
// OwnerID returns the seeded owner Reader's id — the Reader every request
// acts as while the global token is still the only credential.
func (s *Store) OwnerID() int64 { return s.ownerID }
// readersMigration is the version that creates the readers table. The owner
// seed runs between two migrate passes, so that the run-once migration which
// attaches existing bookmarks (0004) finds the owner row.
const readersMigration = 3
// allMigrations is the migrate() cap that applies every pending version.
const allMigrations = 0
// Open connects to Postgres at url — a libpq connection URL such as // Open connects to Postgres at url — a libpq connection URL such as
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — and brings its // "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — brings its
// schema up to date. // schema up to date, and seeds the owner Reader.
func Open(url string) (*Store, error) { func Open(url string, owner Owner) (*Store, error) {
db, err := sql.Open("pgx", url) db, err := sql.Open("pgx", url)
if err != nil { if err != nil {
return nil, fmt.Errorf("open postgres: %w", err) return nil, fmt.Errorf("open postgres: %w", err)
} }
if err := migrate(db); err != nil { // Schema runs in two passes with the seed between: 0003 creates the
// readers table, the owner row must exist before 0004 attaches the
// existing bookmarks to it. Anything past 0004 is applied by the second
// pass.
if err := migrate(db, readersMigration); err != nil {
db.Close()
return nil, fmt.Errorf("migrate schema: %w", err)
}
if err := seedOwner(db, owner); err != nil {
db.Close()
return nil, fmt.Errorf("seed owner: %w", err)
}
if err := migrate(db, allMigrations); err != nil {
db.Close() db.Close()
return nil, fmt.Errorf("migrate: %w", err) return nil, fmt.Errorf("migrate: %w", err)
} }
return &Store{db: db}, nil var ownerID int64
if err := db.QueryRow(
`SELECT id FROM readers WHERE discord_id = $1`, owner.DiscordID).Scan(&ownerID); err != nil {
db.Close()
return nil, fmt.Errorf("resolve owner: %w", err)
}
return &Store{db: db, ownerID: ownerID}, nil
}
// seedOwner makes sure the configured owner exists as exactly one readers row,
// and keeps its token hash current on every start: rotating the userscript
// token must refresh the hash, or the stored credential goes stale.
func seedOwner(db *sql.DB, o Owner) error {
if _, err := db.Exec(`
INSERT INTO readers (discord_id, token_sha256) VALUES ($1, $2)
ON CONFLICT (discord_id) DO UPDATE SET token_sha256 = EXCLUDED.token_sha256`,
o.DiscordID, o.TokenHash[:]); err != nil {
return fmt.Errorf("seed owner: %w", err)
}
return nil
} }
// migrate applies every embedded migration this database has not recorded, in // migrate applies every embedded migration this database has not recorded, in
// filename order, each in its own transaction. Files are named // filename order, each in its own transaction. upto caps the highest version
// "<version>_<name>.sql" and are append-only: editing an applied file changes // applied; 0 means all. Files are named "<version>_<name>.sql" and are
// nothing, because schema_migrations is how a database remembers what it ran. // append-only: editing an applied file changes nothing, because
// Runs on every start and is a no-op once current. // schema_migrations is how a database remembers what it ran. Runs on every
func migrate(db *sql.DB) error { // start and is a no-op once current.
func migrate(db *sql.DB, upto int64) error {
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations ( if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
version bigint PRIMARY KEY, version bigint PRIMARY KEY,
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil { applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
@@ -170,6 +271,9 @@ func migrate(db *sql.DB) error {
if err != nil { if err != nil {
return fmt.Errorf("migration %q: filename must start with a version number", name) return fmt.Errorf("migration %q: filename must start with a version number", name)
} }
if upto > 0 && version > upto {
continue
}
body, err := migrations.ReadFile(name) body, err := migrations.ReadFile(name)
if err != nil { if err != nil {
return err return err
@@ -219,7 +323,7 @@ func scanBookmark(scan func(...any) error) (Bookmark, error) {
latestChapterNum sql.NullFloat64 latestChapterNum sql.NullFloat64
) )
if err := scan( if err := scan(
&b.Key, &b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover, &b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover,
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL, &b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind, &b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
); err != nil { ); err != nil {
@@ -228,6 +332,9 @@ func scanBookmark(scan func(...any) error) (Bookmark, error) {
if latestChapterNum.Valid { if latestChapterNum.Valid {
b.LatestChapterNum = &latestChapterNum.Float64 b.LatestChapterNum = &latestChapterNum.Float64
} }
// The wire identity is derived: there is no stored key column, the
// bookmark is keyed (reader_id, site, series_id) (issue #22).
b.Key = b.Site + ":" + b.SeriesID
// An unrecognised bucket (a hand-edited row) would leave the row in no list // An unrecognised bucket (a hand-edited row) would leave the row in no list
// at all, so anything outside the three known buckets reads as the default // at all, so anything outside the three known buckets reads as the default
// rather than being passed through. // rather than being passed through.
@@ -237,14 +344,39 @@ func scanBookmark(scan func(...any) error) (Bookmark, error) {
return b, nil return b, nil
} }
// scanSeries reads one row in seriesColumns order, plus the due query's
// reader_count column. latest_chapter_num is NULL until the first capture,
// same as on the bookmark read path.
func scanSeries(scan func(...any) error) (Series, error) {
var (
sr Series
latestChapterNum sql.NullFloat64
)
if err := scan(
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
&sr.readerCount,
); err != nil {
return Series{}, err
}
if latestChapterNum.Valid {
sr.LatestChapterNum = &latestChapterNum.Float64
}
return sr, nil
}
// Close releases the underlying database handle. // Close releases the underlying database handle.
func (s *Store) Close() error { return s.db.Close() } func (s *Store) Close() error { return s.db.Close() }
// List returns every bookmark, newest activity first. // List returns every bookmark of one reader, newest activity first.
func (s *Store) List() ([]Bookmark, error) { // Series-owned fields are joined in, so each Bookmark reads back whole and
rows, err := s.db.Query(`SELECT ` + bookmarkColumns + ` // flat (ADR-0004).
FROM bookmarks func (s *Store) List(readerID int64) ([]Bookmark, error) {
ORDER BY updated_at DESC`) rows, err := s.db.Query(`SELECT `+bookmarkColumns+`
FROM bookmarks b
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.reader_id = $1
ORDER BY b.updated_at DESC`, readerID)
if err != nil { if err != nil {
return nil, fmt.Errorf("query bookmarks: %w", err) return nil, fmt.Errorf("query bookmarks: %w", err)
} }
@@ -261,12 +393,19 @@ func (s *Store) List() ([]Bookmark, error) {
return out, rows.Err() return out, rows.Err()
} }
// Get returns one bookmark by key. A missing key is not an error: ok is false // Get returns one bookmark of one reader by key. A missing key is not an
// and err is nil. UI mutations read-modify-write through this so they preserve // error: ok is false and err is nil. UI mutations read-modify-write through
// the fields they do not touch. // this so they preserve the fields they do not touch.
func (s *Store) Get(key string) (Bookmark, bool, error) { func (s *Store) Get(readerID int64, key string) (Bookmark, bool, error) {
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
return Bookmark{}, false, nil
}
b, err := scanBookmark(s.db.QueryRow( b, err := scanBookmark(s.db.QueryRow(
`SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = $1`, key).Scan) `SELECT `+bookmarkColumns+` FROM bookmarks b
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
readerID, site, seriesID).Scan)
if errors.Is(err, sql.ErrNoRows) { if errors.Is(err, sql.ErrNoRows) {
return Bookmark{}, false, nil return Bookmark{}, false, nil
} }
@@ -276,15 +415,25 @@ func (s *Store) Get(key string) (Bookmark, bool, error) {
return b, true, nil return b, true, nil
} }
// Upsert inserts or replaces a bookmark by key (last-write-wins) and returns // Upsert inserts or replaces one reader's bookmark by key (last-write-wins)
// the row as actually stored. // and returns the row as actually stored — one flat object with the
// series-owned fields joined in, exactly as GET reports it (ADR-0004). A
// bookmark is keyed (reader_id, site, series_id), so the same key upserts two
// independent rows for two readers.
//
// The flat body is decomposed across two tables in one transaction. The series
// row is written first (the bookmarks FK requires it to exist), then the
// bookmark row. On the series side, title/series_url/cover are applied only
// when the row is brand new: once a series exists, client-supplied values are
// ignored, because the row is shared and the values are scraped page content —
// see ADR-0003. Kind and the latest-chapter fields are last-write-wins.
// //
// b.UpdatedAt is only a candidate: it is applied when the row is new or when // b.UpdatedAt is only a candidate: it is applied when the row is new or when
// last_chapter_num changes, and otherwise the stored value is kept. Clients // last_chapter_num changes, and otherwise the stored value is kept. Clients
// order their list by updated_at, so favoriting a series or recording a newly // order their list by updated_at, so favoriting a series or recording a newly
// published chapter must not disturb that order — only real reading progress // published chapter must not disturb that order — only real reading progress
// does. Callers must therefore use the returned bookmark, not the argument. // does. Callers must therefore use the returned bookmark, not the argument.
func (s *Store) Upsert(b Bookmark) (Bookmark, error) { func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
tx, err := s.db.Begin() tx, err := s.db.Begin()
if err != nil { if err != nil {
return Bookmark{}, fmt.Errorf("begin %q: %w", b.Key, err) return Bookmark{}, fmt.Errorf("begin %q: %w", b.Key, err)
@@ -296,53 +445,65 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
latestNum = *b.LatestChapterNum latestNum = *b.LatestChapterNum
} }
// IS DISTINCT FROM is Postgres's null-safe comparison, and it is what // The kind column resolves on the VALUES side, not in the conflict clause:
// implements the ordering rule. Within DO UPDATE, a bare column is the // excluded.* is the row *after* these expressions are evaluated, so a
// stored row and excluded.* is the incoming one; a brand-new key never // default applied there would look identical to a real 'manga' and would
// reaches this clause, so it keeps the fresh timestamp from VALUES. // overwrite a novel series on every PUT from a client that knows nothing
// // about the column. Resolved once here, an empty incoming kind means "keep
// The status and kind columns resolve on the VALUES side, not in the // what is stored", and only a brand-new row falls through to the literal
// conflict clause: excluded.* is the row *after* these expressions are // default. The subquery runs inside this transaction, so it sees the row
// evaluated, so a default applied there would look identical to a real // this statement is about to conflict with. Same pattern as the status
// 'reading' / 'manga' and would overwrite an archived or novel row on // COALESCE on the bookmark insert below.
// every PUT from a client that knows nothing about the column. Resolved
// once here, an empty incoming status or kind means "keep what is
// stored", and only a brand-new row falls through to the literal
// default. The subquery runs inside this transaction, so it sees the
// row this statement is about to conflict with.
// //
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no // The ::text casts are load-bearing: inside COALESCE/NULLIF there is no
// target column to infer the parameter type from, and Postgres rejects the // target column to infer the parameter type from, and Postgres rejects the
// statement rather than guessing. // statement rather than guessing.
if _, err := tx.Exec(` if _, err := tx.Exec(`
INSERT INTO bookmarks (`+bookmarkColumns+`) INSERT INTO series (site, series_id, title, series_url, cover, kind,
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, latest_chapter, latest_chapter_num)
COALESCE(NULLIF($14::text, ''), (SELECT status FROM bookmarks WHERE key = $1), 'reading'), VALUES ($1, $2, $3, $4, $5,
COALESCE(NULLIF($15::text, ''), (SELECT kind FROM bookmarks WHERE key = $1), 'manga')) COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
ON CONFLICT (key) DO UPDATE SET $7, $8)
site=excluded.site, series_id=excluded.series_id, title=excluded.title, ON CONFLICT (site, series_id) DO UPDATE SET
series_url=excluded.series_url, cover=excluded.cover, kind=excluded.kind,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num`,
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
b.LatestChapter, latestNum); err != nil {
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
}
// IS DISTINCT FROM is Postgres's null-safe comparison, and it is what
// implements the ordering rule. Within DO UPDATE, a bare column is the
// stored row and excluded.* is the incoming one; a brand-new key never
// reaches this clause, so it keeps the fresh timestamp from VALUES.
if _, err := tx.Exec(`
INSERT INTO bookmarks (reader_id, site, series_id, last_chapter, last_chapter_num,
last_chapter_url, favorite, status, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7,
COALESCE(NULLIF($8::text, ''), (SELECT status FROM bookmarks WHERE reader_id = $1 AND site = $2 AND series_id = $3), 'reading'),
$9)
ON CONFLICT (reader_id, site, series_id) DO UPDATE SET
last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num, last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num,
last_chapter_url=excluded.last_chapter_url, last_chapter_url=excluded.last_chapter_url,
favorite=excluded.favorite, favorite=excluded.favorite,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num,
status=excluded.status, status=excluded.status,
kind=excluded.kind,
updated_at=CASE updated_at=CASE
WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num
THEN excluded.updated_at THEN excluded.updated_at
ELSE bookmarks.updated_at ELSE bookmarks.updated_at
END`, END`,
b.Key, b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, readerID, b.Site, b.SeriesID,
b.LastChapter, b.LastChapterNum, b.LastChapterURL, b.LastChapter, b.LastChapterNum, b.LastChapterURL,
b.Favorite, b.LatestChapter, latestNum, b.UpdatedAt, b.Favorite, b.Status, b.UpdatedAt); err != nil {
b.Status, b.Kind); err != nil {
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err) return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
} }
stored, err := scanBookmark(tx.QueryRow( stored, err := scanBookmark(tx.QueryRow(
`SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = $1`, b.Key).Scan) `SELECT `+bookmarkColumns+` FROM bookmarks b
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
readerID, b.Site, b.SeriesID).Scan)
if err != nil { if err != nil {
return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err) return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err)
} }
@@ -352,79 +513,109 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) {
return stored, nil return stored, nil
} }
// Delete removes a bookmark by key. Deleting a missing key is not an error. // Delete removes one reader's bookmark by key. Deleting a missing key is not
func (s *Store) Delete(key string) error { // an error.
if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = $1`, key); err != nil { func (s *Store) Delete(readerID int64, key string) error {
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
return nil
}
if _, err := s.db.Exec(
`DELETE FROM bookmarks WHERE reader_id = $1 AND site = $2 AND series_id = $3`,
readerID, site, seriesID); err != nil {
return fmt.Errorf("delete %q: %w", key, err) return fmt.Errorf("delete %q: %w", key, err)
} }
return nil return nil
} }
// DueForLatestCheck returns bookmarks whose server-side latest-chapter check has // DueForLatestCheck returns series whose server-side latest-chapter check has
// aged past cutoffMs, least-recently-checked first, at most limit of them. // aged past cutoffMs, ordered by how many bookmarks reference them (descending)
// then least-recently-checked first, at most limit of them.
// //
// Oldest-first is what keeps the poller fair when the backlog outgrows its // The reader_count ordering is the point of the split (ADR-0003): a series
// shared by several readers is fetched once per due cycle, and the popular
// ones stay freshest while the long tail absorbs any shortfall. Within one
// reader count, oldest-first keeps the poll fair when the backlog outgrows its
// throughput: the most neglected series is always next, so a large collection // throughput: the most neglected series is always next, so a large collection
// refreshes uniformly slower rather than leaving a tail that never refreshes at // refreshes uniformly slower rather than leaving a tail that never refreshes at
// all. The userscript sorts its own queue the same way (L453). // all. The userscript sorts its own queue the same way (L453).
// //
// Bookmarks with no series_url are skipped — there is nothing to fetch, which // Series with no series_url are skipped — there is nothing to fetch, which is
// is the same filter the userscript applies at L452. // the same filter the userscript applies at L452. Series whose only bookmarks
// // are finished are skipped too: nothing more is coming, so fetching them only
// Finished series are excluded: nothing more is coming, so fetching them only // burns requests. Archived bookmarks still count — knowing what a shelved
// burns requests. Archived ones are deliberately still polled — knowing what a // series is up to is the whole reason for archiving instead of deleting.
// shelved series is up to is the whole reason for archiving instead of deleting. // A series with no bookmarks at all never appears: the join excludes it.
func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Bookmark, error) { func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Series, error) {
rows, err := s.db.Query(`SELECT `+bookmarkColumns+` rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
FROM bookmarks FROM series s
WHERE series_url <> '' JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
AND status IS DISTINCT FROM 'finished' WHERE s.series_url <> ''
AND latest_checked_at <= $1 AND s.latest_checked_at <= $1
ORDER BY latest_checked_at ASC GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
ORDER BY reader_count DESC, s.latest_checked_at ASC
LIMIT $2`, cutoffMs, limit) LIMIT $2`, cutoffMs, limit)
if err != nil { if err != nil {
return nil, fmt.Errorf("query due bookmarks: %w", err) return nil, fmt.Errorf("query due series: %w", err)
} }
defer rows.Close() defer rows.Close()
out := []Bookmark{} out := []Series{}
for rows.Next() { for rows.Next() {
b, err := scanBookmark(rows.Scan) sr, err := scanSeries(rows.Scan)
if err != nil { if err != nil {
return nil, fmt.Errorf("scan due bookmark: %w", err) return nil, fmt.Errorf("scan due series: %w", err)
} }
out = append(out, b) out = append(out, sr)
} }
return out, rows.Err() return out, rows.Err()
} }
// MarkLatestChecked records that the server looked at key at ts, whatever the // MarkLatestChecked records that the server looked at a series at ts, whatever
// look turned up. Marking a missing key is not an error: the row may have been // the look turned up. Marking a missing series is not an error: the row may
// deleted while a fetch was in flight. // have been orphaned while a fetch was in flight.
// //
// This is the one write that does not go through Upsert, and the column is kept // This is the one write that does not go through Upsert, and the column is kept
// out of bookmarkColumns on purpose. PUT /bookmarks/{key} decodes a whole // out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
// Bookmark from the client and Upsert writes every column it knows about, so a // a whole Bookmark from the client and Upsert writes every series column it
// userscript PUT — which has no idea this field exists — would write a zero and // knows about, so a userscript PUT — which has no idea this field exists —
// reset the cooldown, making the poller re-fetch that series every tick for as // would write a zero and reset the cooldown, making the poller re-fetch that
// long as the user kept reading it. // series every tick for as long as the user kept reading it.
func (s *Store) MarkLatestChecked(key string, ts int64) error { func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
if _, err := s.db.Exec( if _, err := s.db.Exec(
`UPDATE bookmarks SET latest_checked_at = $1 WHERE key = $2`, ts, key); err != nil { `UPDATE series SET latest_checked_at = $1 WHERE site = $2 AND series_id = $3`,
return fmt.Errorf("mark checked %q: %w", key, err) ts, site, seriesID); err != nil {
return fmt.Errorf("mark checked %s:%s: %w", site, seriesID, err)
} }
return nil return nil
} }
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for // LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
// tests outside this package (the poller's own tests assert on cooldown // tests outside this package (the poller's own tests assert on cooldown
// bookkeeping) — see MarkLatestChecked for why the field itself stays off // bookkeeping) — see MarkLatestChecked for why the field stays off the
// Bookmark. // client-visible row.
func (s *Store) LatestCheckedAt(key string) (int64, error) { func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
var ts int64 var ts int64
if err := s.db.QueryRow( if err := s.db.QueryRow(
`SELECT latest_checked_at FROM bookmarks WHERE key = $1`, key).Scan(&ts); err != nil { `SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`,
return 0, fmt.Errorf("latest checked at %q: %w", key, err) site, seriesID).Scan(&ts); err != nil {
return 0, fmt.Errorf("latest checked at %s:%s: %w", site, seriesID, err)
} }
return ts, nil return ts, nil
} }
// SetLatestChapter records the newest chapter the poll found on a series page.
// The poller walks Series rather than Bookmarks, so this is a series-level
// write: the row is shared, and updating it once refreshes every bookmark that
// joins to it. Touching a missing series is not an error.
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
if _, err := s.db.Exec(
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
WHERE site = $1 AND series_id = $2`,
site, seriesID, label, num); err != nil {
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
}
return nil
}
+532 -42
View File
@@ -1,7 +1,12 @@
package store package store
import ( import (
"bytes"
"crypto/sha256"
"database/sql"
"os" "os"
"strconv"
"strings"
"testing" "testing"
"time" "time"
@@ -10,9 +15,13 @@ import (
func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
// testOwner is the owner every test store seeds. Tests that need a second
// reader insert one directly (see secondReader).
var testOwner = Owner{DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("owner-token-hash"))}
func newTestStore(t *testing.T) *Store { func newTestStore(t *testing.T) *Store {
t.Helper() t.Helper()
store, err := Open(pgtest.URL(t)) store, err := Open(pgtest.URL(t), testOwner)
if err != nil { if err != nil {
t.Fatalf("Open: %v", err) t.Fatalf("Open: %v", err)
} }
@@ -20,29 +29,44 @@ func newTestStore(t *testing.T) *Store {
return store return store
} }
// secondReader inserts an extra reader row and returns its id. The store API
// has no reader-creation path yet — the seed is the only one — so tests that
// need reader isolation insert directly.
func secondReader(t *testing.T, s *Store) int64 {
t.Helper()
hash := sha256.Sum256([]byte("second-token-hash"))
var id int64
if err := s.db.QueryRow(
`INSERT INTO readers (discord_id, token_sha256) VALUES ($1, $2) RETURNING id`,
"second-"+strconv.FormatInt(time.Now().UnixNano(), 10), hash[:]).Scan(&id); err != nil {
t.Fatalf("seed second reader: %v", err)
}
return id
}
// The migration runner runs on every start, so a second Open against a // The migration runner runs on every start, so a second Open against a
// database it already built must be a no-op rather than a duplicate-table // database it already built must be a no-op rather than a duplicate-table
// error, and must leave the rows alone. // error, and must leave the rows alone.
func TestOpenIsIdempotent(t *testing.T) { func TestOpenIsIdempotent(t *testing.T) {
url := pgtest.URL(t) url := pgtest.URL(t)
first, err := Open(url) first, err := Open(url, testOwner)
if err != nil { if err != nil {
t.Fatalf("Open: %v", err) t.Fatalf("Open: %v", err)
} }
if _, err := first.Upsert(Bookmark{ if _, err := first.Upsert(first.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000, Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed: %v", err) t.Fatalf("seed: %v", err)
} }
first.Close() first.Close()
second, err := Open(url) second, err := Open(url, testOwner)
if err != nil { if err != nil {
t.Fatalf("reopen: %v", err) t.Fatalf("reopen: %v", err)
} }
t.Cleanup(func() { second.Close() }) t.Cleanup(func() { second.Close() })
list, err := second.List() list, err := second.List(second.OwnerID())
if err != nil { if err != nil {
t.Fatalf("List: %v", err) t.Fatalf("List: %v", err)
} }
@@ -53,14 +77,14 @@ func TestOpenIsIdempotent(t *testing.T) {
func TestStoreGet(t *testing.T) { func TestStoreGet(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
if _, err := store.Upsert(Bookmark{ if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", LastChapterNum: 45, UpdatedAt: 1000, Title: "Solo Leveling", LastChapterNum: 45, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
got, ok, err := store.Get("asura:solo") got, ok, err := store.Get(store.OwnerID(), "asura:solo")
if err != nil { if err != nil {
t.Fatalf("Get: %v", err) t.Fatalf("Get: %v", err)
} }
@@ -74,7 +98,7 @@ func TestStoreGet(t *testing.T) {
func TestStoreGetMissing(t *testing.T) { func TestStoreGetMissing(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
_, ok, err := store.Get("asura:nope") _, ok, err := store.Get(store.OwnerID(), "asura:nope")
if err != nil { if err != nil {
t.Fatalf("Get missing returned error %v, want nil", err) t.Fatalf("Get missing returned error %v, want nil", err)
} }
@@ -124,30 +148,41 @@ func TestBookmarkContinueURL(t *testing.T) {
} }
// readLatestCheckedAt reads the column directly. It is deliberately absent from // readLatestCheckedAt reads the column directly. It is deliberately absent from
// Bookmark (see Store.Upsert), so tests cannot assert on it any other way. // Series (see Store.MarkLatestChecked), so tests cannot assert on it any other
// way. The key splits the same way the API handler derives site/series_id.
func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 { func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 {
t.Helper() t.Helper()
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
var ts int64 var ts int64
if err := s.db.QueryRow( if err := s.db.QueryRow(
`SELECT latest_checked_at FROM bookmarks WHERE key = $1`, key).Scan(&ts); err != nil { `SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`,
site, seriesID).Scan(&ts); err != nil {
t.Fatalf("read latest_checked_at %q: %v", key, err) t.Fatalf("read latest_checked_at %q: %v", key, err)
} }
return ts return ts
} }
// seedForCheck inserts a bookmark and forces its latest_checked_at. // seedForCheck inserts a bookmark (and with it its series) and forces the
// series' latest_checked_at.
func seedForCheck(t *testing.T, s *Store, key, seriesURL string, checkedAt int64) { func seedForCheck(t *testing.T, s *Store, key, seriesURL string, checkedAt int64) {
t.Helper() t.Helper()
if _, err := s.Upsert(Bookmark{ site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := s.Upsert(s.OwnerID(), Bookmark{
Key: key, Key: key,
Site: "asura", Site: site,
SeriesID: key, SeriesID: seriesID,
SeriesURL: seriesURL, SeriesURL: seriesURL,
UpdatedAt: 1000, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed %q: %v", key, err) t.Fatalf("seed %q: %v", key, err)
} }
if err := s.MarkLatestChecked(key, checkedAt); err != nil { if err := s.MarkLatestChecked(site, seriesID, checkedAt); err != nil {
t.Fatalf("seed mark %q: %v", key, err) t.Fatalf("seed mark %q: %v", key, err)
} }
} }
@@ -198,8 +233,8 @@ func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) {
if len(due) != 2 { if len(due) != 2 {
t.Fatalf("got %d rows, want 2 (limit)", len(due)) t.Fatalf("got %d rows, want 2 (limit)", len(due))
} }
if due[0].Key != "asura:a" || due[1].Key != "asura:b" { if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" {
t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key, due[1].Key) t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key())
} }
} }
@@ -207,30 +242,31 @@ func TestMarkLatestChecked(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 0) seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 0)
if err := s.MarkLatestChecked("asura:x", 4242); err != nil { if err := s.MarkLatestChecked("asura", "x", 4242); err != nil {
t.Fatalf("MarkLatestChecked: %v", err) t.Fatalf("MarkLatestChecked: %v", err)
} }
if got := readLatestCheckedAt(t, s, "asura:x"); got != 4242 { if got := readLatestCheckedAt(t, s, "asura:x"); got != 4242 {
t.Fatalf("latest_checked_at = %d, want 4242", got) t.Fatalf("latest_checked_at = %d, want 4242", got)
} }
// A missing key is not an error: the row may have been deleted mid-fetch. // A missing series is not an error: its bookmarks may have been deleted
if err := s.MarkLatestChecked("asura:gone", 1); err != nil { // mid-fetch.
t.Fatalf("MarkLatestChecked on missing key: %v", err) if err := s.MarkLatestChecked("asura", "gone", 1); err != nil {
t.Fatalf("MarkLatestChecked on missing series: %v", err)
} }
} }
// Upsert must not touch latest_checked_at. If the column ever migrates into // Upsert must not touch latest_checked_at. If the column ever migrates into
// bookmarkColumns, this fails and the cooldown is silently dead. // the client-visible write path, this fails and the cooldown is silently dead.
func TestUpsertPreservesLatestCheckedAt(t *testing.T) { func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 999) seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 999)
b, ok, err := s.Get("asura:x") b, ok, err := s.Get(s.OwnerID(), "asura:x")
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("Get: %v ok=%v", err, ok) t.Fatalf("Get: %v ok=%v", err, ok)
} }
b.Title = "changed" b.Title = "changed"
if _, err := s.Upsert(b); err != nil { if _, err := s.Upsert(s.OwnerID(), b); err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
if got := readLatestCheckedAt(t, s, "asura:x"); got != 999 { if got := readLatestCheckedAt(t, s, "asura:x"); got != 999 {
@@ -240,7 +276,7 @@ func TestUpsertPreservesLatestCheckedAt(t *testing.T) {
func TestUpsertDefaultsStatusToReading(t *testing.T) { func TestUpsertDefaultsStatusToReading(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
stored, err := store.Upsert(Bookmark{ stored, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", Key: "asura:solo", Site: "asura", SeriesID: "solo",
UpdatedAt: time.Now().UnixMilli(), UpdatedAt: time.Now().UnixMilli(),
}) })
@@ -260,13 +296,13 @@ func TestUpsertEmptyStatusPreservesStored(t *testing.T) {
Key: "asura:solo", Site: "asura", SeriesID: "solo", Key: "asura:solo", Site: "asura", SeriesID: "solo",
Status: StatusArchived, UpdatedAt: time.Now().UnixMilli(), Status: StatusArchived, UpdatedAt: time.Now().UnixMilli(),
} }
if _, err := store.Upsert(base); err != nil { if _, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err) t.Fatalf("seed: %v", err)
} }
base.Status = "" base.Status = ""
base.LastChapterNum = 12 base.LastChapterNum = 12
stored, err := store.Upsert(base) stored, err := store.Upsert(store.OwnerID(), base)
if err != nil { if err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
@@ -285,11 +321,11 @@ func TestLatestPollRoundTripPreservesArchived(t *testing.T) {
Key: "asura:solo", Site: "asura", SeriesID: "solo", Key: "asura:solo", Site: "asura", SeriesID: "solo",
Status: StatusArchived, UpdatedAt: time.Now().UnixMilli(), Status: StatusArchived, UpdatedAt: time.Now().UnixMilli(),
} }
if _, err := store.Upsert(base); err != nil { if _, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err) t.Fatalf("seed: %v", err)
} }
cur, found, err := store.Get(base.Key) cur, found, err := store.Get(store.OwnerID(), base.Key)
if err != nil || !found { if err != nil || !found {
t.Fatalf("Get: found=%v err=%v", found, err) t.Fatalf("Get: found=%v err=%v", found, err)
} }
@@ -299,7 +335,7 @@ func TestLatestPollRoundTripPreservesArchived(t *testing.T) {
cur.LatestChapterNum = &num cur.LatestChapterNum = &num
cur.UpdatedAt = time.Now().UnixMilli() cur.UpdatedAt = time.Now().UnixMilli()
stored, err := store.Upsert(cur) stored, err := store.Upsert(store.OwnerID(), cur)
if err != nil { if err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
@@ -314,12 +350,12 @@ func TestUpsertReplacesStatusWhenGiven(t *testing.T) {
Key: "asura:solo", Site: "asura", SeriesID: "solo", Key: "asura:solo", Site: "asura", SeriesID: "solo",
Status: StatusArchived, UpdatedAt: time.Now().UnixMilli(), Status: StatusArchived, UpdatedAt: time.Now().UnixMilli(),
} }
if _, err := store.Upsert(base); err != nil { if _, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err) t.Fatalf("seed: %v", err)
} }
base.Status = StatusReading base.Status = StatusReading
stored, err := store.Upsert(base) stored, err := store.Upsert(store.OwnerID(), base)
if err != nil { if err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
@@ -336,14 +372,14 @@ func TestUpsertStatusChangeKeepsUpdatedAt(t *testing.T) {
LastChapter: "45", LastChapterNum: 45, LastChapter: "45", LastChapterNum: 45,
UpdatedAt: time.Now().UnixMilli(), UpdatedAt: time.Now().UnixMilli(),
} }
first, err := store.Upsert(base) first, err := store.Upsert(store.OwnerID(), base)
if err != nil { if err != nil {
t.Fatalf("seed: %v", err) t.Fatalf("seed: %v", err)
} }
base.Status = StatusArchived base.Status = StatusArchived
base.UpdatedAt = first.UpdatedAt + 60_000 base.UpdatedAt = first.UpdatedAt + 60_000
stored, err := store.Upsert(base) stored, err := store.Upsert(store.OwnerID(), base)
if err != nil { if err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
@@ -361,8 +397,8 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
{"asura:archived", StatusArchived}, {"asura:archived", StatusArchived},
{"asura:finished", StatusFinished}, {"asura:finished", StatusFinished},
} { } {
if _, err := store.Upsert(Bookmark{ if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: tc.key, Site: "asura", SeriesID: tc.key, Key: tc.key, Site: "asura", SeriesID: strings.TrimPrefix(tc.key, "asura:"),
SeriesURL: "https://asurascans.com/comics/" + tc.key, SeriesURL: "https://asurascans.com/comics/" + tc.key,
Status: tc.status, UpdatedAt: time.Now().UnixMilli(), Status: tc.status, UpdatedAt: time.Now().UnixMilli(),
}); err != nil { }); err != nil {
@@ -375,8 +411,8 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
t.Fatalf("DueForLatestCheck: %v", err) t.Fatalf("DueForLatestCheck: %v", err)
} }
got := map[string]bool{} got := map[string]bool{}
for _, b := range due { for _, sr := range due {
got[b.Key] = true got[sr.Key()] = true
} }
if !got["asura:reading"] || !got["asura:archived"] { if !got["asura:reading"] || !got["asura:archived"] {
t.Fatalf("due = %v, want reading and archived present", got) t.Fatalf("due = %v, want reading and archived present", got)
@@ -426,7 +462,7 @@ func TestDisplayChapter(t *testing.T) {
func TestUpsertKindDefaultsToManga(t *testing.T) { func TestUpsertKindDefaultsToManga(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
got, err := store.Upsert(Bookmark{ got, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000, Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
}) })
if err != nil { if err != nil {
@@ -439,7 +475,7 @@ func TestUpsertKindDefaultsToManga(t *testing.T) {
func TestUpsertKindRoundTrips(t *testing.T) { func TestUpsertKindRoundTrips(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
got, err := store.Upsert(Bookmark{ got, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "lightnovelworld:a-will-eternal", Site: "lightnovelworld", Key: "lightnovelworld:a-will-eternal", Site: "lightnovelworld",
SeriesID: "a-will-eternal", Kind: KindNovel, UpdatedAt: 1000, SeriesID: "a-will-eternal", Kind: KindNovel, UpdatedAt: 1000,
}) })
@@ -455,14 +491,14 @@ func TestUpsertKindRoundTrips(t *testing.T) {
// must keep the stored library, not silently demote a novel to manga. // must keep the stored library, not silently demote a novel to manga.
func TestUpsertEmptyKindKeepsStoredValue(t *testing.T) { func TestUpsertEmptyKindKeepsStoredValue(t *testing.T) {
store := newTestStore(t) store := newTestStore(t)
if _, err := store.Upsert(Bookmark{ if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "lightnovelworld:a-will-eternal", Site: "lightnovelworld", Key: "lightnovelworld:a-will-eternal", Site: "lightnovelworld",
SeriesID: "a-will-eternal", Kind: KindNovel, LastChapterNum: 10, UpdatedAt: 1000, SeriesID: "a-will-eternal", Kind: KindNovel, LastChapterNum: 10, UpdatedAt: 1000,
}); err != nil { }); err != nil {
t.Fatalf("seed: %v", err) t.Fatalf("seed: %v", err)
} }
got, err := store.Upsert(Bookmark{ got, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "lightnovelworld:a-will-eternal", Site: "lightnovelworld", Key: "lightnovelworld:a-will-eternal", Site: "lightnovelworld",
SeriesID: "a-will-eternal", Kind: "", LastChapterNum: 11, UpdatedAt: 2000, SeriesID: "a-will-eternal", Kind: "", LastChapterNum: 11, UpdatedAt: 2000,
}) })
@@ -476,3 +512,457 @@ func TestUpsertEmptyKindKeepsStoredValue(t *testing.T) {
t.Fatalf("LastChapterNum = %v, want 11 — progress in the same request must still land", got.LastChapterNum) t.Fatalf("LastChapterNum = %v, want 11 — progress in the same request must still land", got.LastChapterNum)
} }
} }
// The 0002 backfill must survive a database that already ran 0001 with real
// rows: one series row per distinct (site, series_id) carrying the moved
// columns, and the bookmark keeping the rest. That is the upgrade path for
// every deployed database, so it is exercised rather than trusted.
func TestMigration0002BackfillsExistingBookmarks(t *testing.T) {
url := pgtest.URL(t)
db, err := sql.Open("pgx", url)
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { db.Close() })
// Run only 0001, as a database created before this change would have.
// migrate() normally creates the version table first; do the same here.
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
version bigint PRIMARY KEY,
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
t.Fatalf("create version table: %v", err)
}
body, err := migrations.ReadFile("migrations/0001_bookmarks.sql")
if err != nil {
t.Fatalf("read 0001: %v", err)
}
if err := applyMigration(db, 1, string(body)); err != nil {
t.Fatalf("apply 0001: %v", err)
}
if _, err := db.Exec(`
INSERT INTO bookmarks (key, site, series_id, title, series_url, cover,
last_chapter, last_chapter_num, last_chapter_url, favorite, latest_chapter,
latest_chapter_num, latest_checked_at, status, kind, updated_at)
VALUES ('asura:solo', 'asura', 'solo', 'Solo Leveling',
'https://asurascans.com/comics/solo', 'https://asurascans.com/covers/solo.jpg',
'Chapter 10', 10, 'https://asurascans.com/comics/solo/ch/10', true,
'Chapter 11', 11, 123456, 'reading', 'manga', 1000)`); err != nil {
t.Fatalf("seed legacy row: %v", err)
}
// Bring it current through the production path: Open runs the schema to
// 0003, seeds the owner, then applies 0004 which attaches this row. 0002
// must have backfilled the series row, not lost data.
st, err := Open(url, testOwner)
if err != nil {
t.Fatalf("Open after migrate: %v", err)
}
defer st.Close()
var (
title string
checked int64
fav bool
lastNum float64
)
if err := db.QueryRow(`SELECT title, latest_checked_at FROM series
WHERE site = 'asura' AND series_id = 'solo'`).Scan(&title, &checked); err != nil {
t.Fatalf("series row missing after migration: %v", err)
}
if title != "Solo Leveling" || checked != 123456 {
t.Fatalf("series = (%q, %d), want backfilled title and latest_checked_at", title, checked)
}
// The key column is gone; the bookmark is read by its composite key.
if err := db.QueryRow(`SELECT favorite, last_chapter_num FROM bookmarks
WHERE reader_id = $1 AND site = 'asura' AND series_id = 'solo'`,
st.OwnerID()).Scan(&fav, &lastNum); err != nil {
t.Fatalf("bookmark row missing after migration: %v", err)
}
if !fav || lastNum != 10 {
t.Fatalf("bookmark = (%v, %v), want favorite and progress kept", fav, lastNum)
}
}
// readSeries reads the series row directly, for asserting on what Upsert
// actually stored rather than what the joined Bookmark reports.
func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
t.Helper()
sr, err := scanSeries(s.db.QueryRow(
`SELECT `+seriesColumns+`, 0 AS reader_count FROM series s
WHERE s.site = $1 AND s.series_id = $2`, site, seriesID).Scan)
if err != nil {
t.Fatalf("read series %s:%s: %v", site, seriesID, err)
}
return sr
}
// The first PUT for a series creates its row from the client's title, cover
// and URL — there is no other source for them (ADR-0003).
func TestUpsertCreatesSeriesFromClient(t *testing.T) {
store := newTestStore(t)
if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
Cover: "https://asurascans.com/covers/solo.jpg", Kind: KindManga,
UpdatedAt: 1000,
}); err != nil {
t.Fatalf("Upsert: %v", err)
}
sr := readSeries(t, store, "asura", "solo")
if sr.Title != "Solo Leveling" || sr.SeriesURL != "https://asurascans.com/comics/solo" ||
sr.Cover != "https://asurascans.com/covers/solo.jpg" {
t.Fatalf("series = %+v, want client title/url/cover stored", sr)
}
}
// A PUT naming an existing series must not overwrite its title, cover or URL:
// the row is shared, and those values are scraped page content (ADR-0003).
func TestUpsertExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
store := newTestStore(t)
base := Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
Cover: "https://asurascans.com/covers/solo.jpg", LastChapterNum: 10,
UpdatedAt: 1000,
}
if _, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err)
}
// Same series, hostile/compromised values, real progress advance.
base.Title = "Scraped Rename"
base.SeriesURL = "https://evil.example/solo"
base.Cover = "https://evil.example/solo.jpg"
base.LastChapterNum = 11
got, err := store.Upsert(store.OwnerID(), base)
if err != nil {
t.Fatalf("Upsert: %v", err)
}
if got.Title != "Solo Leveling" || got.SeriesURL != "https://asurascans.com/comics/solo" ||
got.Cover != "https://asurascans.com/covers/solo.jpg" {
t.Fatalf("stored = %+v, want original title/url/cover kept", got)
}
if got.LastChapterNum != 11 {
t.Fatalf("LastChapterNum = %v, want 11 — progress in the same request must still land", got.LastChapterNum)
}
}
// Kind and latest-chapter are last-write-wins even on an existing series: the
// poller and the userscript both report the latest chapter, and kind is only
// known to whichever client created the row.
func TestUpsertExistingSeriesAcceptsKindAndLatest(t *testing.T) {
store := newTestStore(t)
base := Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", Kind: KindManga,
UpdatedAt: 1000,
}
if _, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err)
}
num := 12.0
base.Kind = KindNovel
base.LatestChapter = "Chapter 12"
base.LatestChapterNum = &num
got, err := store.Upsert(store.OwnerID(), base)
if err != nil {
t.Fatalf("Upsert: %v", err)
}
if got.Kind != KindNovel || got.LatestChapter != "Chapter 12" ||
got.LatestChapterNum == nil || *got.LatestChapterNum != 12 {
t.Fatalf("stored = %+v, want kind and latest chapter updated", got)
}
}
// Deleting the last bookmark must leave the series row behind, so a later
// re-bookmark shows title and cover immediately instead of waiting for a poll.
func TestDeleteKeepsSeriesRow(t *testing.T) {
store := newTestStore(t)
if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", Cover: "https://asurascans.com/covers/solo.jpg",
UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := store.Delete(store.OwnerID(), "asura:solo"); err != nil {
t.Fatalf("Delete: %v", err)
}
sr := readSeries(t, store, "asura", "solo")
if sr.Title != "Solo Leveling" {
t.Fatalf("series = %+v, want it kept after the last bookmark is deleted", sr)
}
// Re-bookmark with nothing but progress: the stored title/cover come back.
stored, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
LastChapterNum: 5, UpdatedAt: 2000,
})
if err != nil {
t.Fatalf("re-upsert: %v", err)
}
if stored.Title != "Solo Leveling" || stored.Cover != "https://asurascans.com/covers/solo.jpg" {
t.Fatalf("re-bookmark = %+v, want title/cover from the surviving series row", stored)
}
}
// seedSecondReader inserts an extra bookmark on an existing series, owned by a
// second reader. Two bookmarks can share a series only across readers now
// (issue #22); the due queue's reader-count ordering counts them all.
func seedSecondReader(t *testing.T, s *Store, key, site, seriesID string, updatedAt int64) {
t.Helper()
if _, err := s.Upsert(secondReader(t, s), Bookmark{
Key: key, Site: site, SeriesID: seriesID, UpdatedAt: updatedAt,
}); err != nil {
t.Fatalf("seed second reader %q: %v", key, err)
}
}
// The whole point of the split: a shared series is due once, ordered ahead of
// single-reader series by how many bookmarks reference it.
func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) {
s := newTestStore(t)
// "pop" has two bookmarks but was checked most recently; "solo" has one and
// was checked long ago. Reader count must win over age.
seedForCheck(t, s, "asura:pop", "https://asurascans.com/comics/pop", 900)
seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100)
due, err := s.DueForLatestCheck(1000, 10)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 2 {
t.Fatalf("due = %d rows, want 2", len(due))
}
if due[0].Key() != "asura:pop" || due[1].Key() != "asura:solo" {
t.Fatalf("due order = %q, %q; want asura:pop (2 readers), asura:solo (1)",
due[0].Key(), due[1].Key())
}
}
// A series with no bookmarks must never appear in the due queue, and nothing
// in the store ever deletes it (see TestDeleteKeepsSeriesRow).
func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:kept", "https://asurascans.com/comics/kept", 0)
if _, err := s.db.Exec(`
INSERT INTO series (site, series_id, title, series_url, cover, kind,
latest_chapter, latest_chapter_num, latest_checked_at)
VALUES ('asura', 'orphan', 'Orphan', 'https://asurascans.com/comics/orphan',
'', 'manga', '', NULL, 0)`); err != nil {
t.Fatalf("seed orphan series: %v", err)
}
due, err := s.DueForLatestCheck(1000, 10)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 1 || due[0].Key() != "asura:kept" {
t.Fatalf("due = %+v, want only the bookmarked series", due)
}
var n int
if err := s.db.QueryRow(
`SELECT count(*) FROM series WHERE site = 'asura' AND series_id = 'orphan'`).Scan(&n); err != nil {
t.Fatalf("count orphan series: %v", err)
}
if n != 1 {
t.Fatalf("orphan series count = %d, want 1 (never deleted)", n)
}
}
// The seed must never multiply the owner row: reopening the same database with
// a different token hash refreshes the stored hash, not the row. That is what
// keeps the readers table at exactly one row across restarts and token
// rotations.
func TestSeedOwnerIdempotentAndRefreshesTokenHash(t *testing.T) {
url := pgtest.URL(t)
first, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v1"))})
if err != nil {
t.Fatalf("Open: %v", err)
}
ownerID := first.OwnerID()
first.Close()
second, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v2"))})
if err != nil {
t.Fatalf("reopen: %v", err)
}
defer second.Close()
if second.OwnerID() != ownerID {
t.Fatalf("owner id = %d after reopen, want %d (same row)", second.OwnerID(), ownerID)
}
var (
n int
hash []byte
)
if err := second.db.QueryRow(`SELECT count(*), (SELECT token_sha256 FROM readers LIMIT 1) FROM readers`).Scan(&n, &hash); err != nil {
t.Fatalf("read readers: %v", err)
}
if n != 1 {
t.Fatalf("readers count = %d, want 1", n)
}
want := sha256.Sum256([]byte("hash-v2"))
if !bytes.Equal(hash, want[:]) {
t.Fatalf("token hash = %x, want the refreshed sha256", hash)
}
}
// The upgrade path for a deployed database: bookmarks created before readers
// existed must all land on the seeded owner, the key column must be gone, and
// the same database must be able to hold two readers' bookmarks for one series.
func TestMigration0004AttachesBookmarksToOwner(t *testing.T) {
url := pgtest.URL(t)
db, err := sql.Open("pgx", url)
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { db.Close() })
// A database at the state before #21 shipped: 0001 applied, bookmarks
// keyed by <site>:<series_id>, no series table. Rows land before 0002, the
// way a real deployment's data did.
if err := migrate(db, 1); err != nil {
t.Fatalf("migrate to 0001: %v", err)
}
for _, key := range []string{"asura:solo", "demonic:catastrophic-necromancer"} {
site, seriesID, ok := strings.Cut(key, ":")
if !ok {
t.Fatalf("key %q: no ':' separator", key)
}
if _, err := db.Exec(`
INSERT INTO bookmarks (key, site, series_id, updated_at)
VALUES ($1, $2, $3, 1000)`, key, site, seriesID); err != nil {
t.Fatalf("seed legacy row %q: %v", key, err)
}
}
// 0002 backfills the series rows, as it did in the real upgrade.
if err := migrate(db, 2); err != nil {
t.Fatalf("migrate to 0002: %v", err)
}
st, err := Open(url, testOwner)
if err != nil {
t.Fatalf("Open: %v", err)
}
defer st.Close()
var (
attached int
readers int
)
if err := st.db.QueryRow(
`SELECT count(*) FROM bookmarks WHERE reader_id = $1`, st.OwnerID()).Scan(&attached); err != nil {
t.Fatalf("count attached bookmarks: %v", err)
}
if attached != 2 {
t.Fatalf("bookmarks attached to owner = %d, want all 2", attached)
}
if err := st.db.QueryRow(`SELECT count(*) FROM readers`).Scan(&readers); err != nil {
t.Fatalf("count readers: %v", err)
}
if readers != 1 {
t.Fatalf("readers = %d, want 1", readers)
}
// The surrogate key column is gone; only the composite key remains.
if _, err := st.db.Query(`SELECT key FROM bookmarks`); err == nil {
t.Fatal("bookmarks.key still exists after the migration")
}
}
// One Reader and Series pair must admit at most one bookmark, enforced by the
// primary key itself — a raw INSERT that skips the upsert must fail.
func TestBookmarkDuplicateImpossibleAtDatabaseLevel(t *testing.T) {
st := newTestStore(t)
// A series row on its own, no bookmark: the raw inserts below must only
// ever collide on the bookmark primary key.
if _, err := st.db.Exec(
`INSERT INTO series (site, series_id) VALUES ('asura', 'solo')`); err != nil {
t.Fatalf("seed series: %v", err)
}
insert := func() error {
_, err := st.db.Exec(`
INSERT INTO bookmarks (reader_id, site, series_id, updated_at)
VALUES ($1, 'asura', 'solo', 1000)`, st.OwnerID())
return err
}
if err := insert(); err != nil {
t.Fatalf("first insert: %v", err)
}
if err := insert(); err == nil {
t.Fatal("duplicate bookmark for the same reader and series was accepted")
}
}
// Every read and write is scoped to the reader it names: a second reader sees
// an empty list, cannot read or delete the owner's row, and a delete by the
// wrong reader leaves the row alone.
func TestStoreScopesBookmarksToReader(t *testing.T) {
st := newTestStore(t)
other := secondReader(t, st)
if _, err := st.Upsert(st.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed owner bookmark: %v", err)
}
otherList, err := st.List(other)
if err != nil {
t.Fatalf("List(other): %v", err)
}
if len(otherList) != 0 {
t.Fatalf("other reader's list = %+v, want empty", otherList)
}
if _, ok, err := st.Get(other, "asura:solo"); err != nil || ok {
t.Fatalf("Get(other, asura:solo) = ok:%v err:%v, want not found", ok, err)
}
if err := st.Delete(other, "asura:solo"); err != nil {
t.Fatalf("Delete(other): %v", err)
}
ownerList, err := st.List(st.OwnerID())
if err != nil {
t.Fatalf("List(owner): %v", err)
}
if len(ownerList) != 1 || ownerList[0].Key != "asura:solo" {
t.Fatalf("owner's list after other's delete = %+v, want the row intact", ownerList)
}
// The same key under a second reader is an independent bookmark.
if _, err := st.Upsert(other, Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 2000,
}); err != nil {
t.Fatalf("upsert other's bookmark: %v", err)
}
if got, err := st.List(other); err != nil || len(got) != 1 {
t.Fatalf("other's list after own upsert = %+v err:%v, want 1 row", got, err)
}
}
// Deleting a reader must take their bookmarks with them (ON DELETE CASCADE)
// while leaving the shared series row behind.
func TestDeleteReaderCascadesToBookmarks(t *testing.T) {
st := newTestStore(t)
other := secondReader(t, st)
if _, err := st.Upsert(other, Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed other's bookmark: %v", err)
}
if _, err := st.db.Exec(`DELETE FROM readers WHERE id = $1`, other); err != nil {
t.Fatalf("delete reader: %v", err)
}
var n int
if err := st.db.QueryRow(`SELECT count(*) FROM bookmarks`).Scan(&n); err != nil {
t.Fatalf("count bookmarks: %v", err)
}
if n != 0 {
t.Fatalf("bookmarks after reader delete = %d, want 0 (cascade)", n)
}
sr := readSeries(t, st, "asura", "solo")
if sr.Title != "Solo Leveling" {
t.Fatalf("series = %+v, want it kept after its only reader was deleted", sr)
}
}
+9 -5
View File
@@ -32,6 +32,9 @@ const RecentCount = 5
// representations (HTML versus JSON) to different clients under different auth. // representations (HTML versus JSON) to different clients under different auth.
type Handler struct { type Handler struct {
store *store.Store store *store.Store
// readerID is the Reader this UI acts as — the seeded owner, while the web
// password is still the only credential (issue #22).
readerID int64
tmpl *template.Template tmpl *template.Template
key []byte key []byte
password string password string
@@ -81,13 +84,14 @@ type loginView struct {
// New parses every template up front so a broken one kills the process at // New parses every template up front so a broken one kills the process at
// startup rather than the first request that touches it. // startup rather than the first request that touches it.
func New(s *store.Store, apiToken, webPassword string) (*Handler, error) { func New(s *store.Store, readerID int64, apiToken, webPassword string) (*Handler, error) {
tmpl, err := template.ParseFS(templateFS, "templates/*.html") tmpl, err := template.ParseFS(templateFS, "templates/*.html")
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &Handler{ return &Handler{
store: s, store: s,
readerID: readerID,
tmpl: tmpl, tmpl: tmpl,
key: session.Key(apiToken, webPassword), key: session.Key(apiToken, webPassword),
password: webPassword, password: webPassword,
@@ -216,7 +220,7 @@ func libOf(q string) string {
// archived favourite therefore shows only under Archived: Favourites means // archived favourite therefore shows only under Archived: Favourites means
// "favourites I am currently reading". // "favourites I am currently reading".
func (h *Handler) buildListView(lib, tab string) (listView, error) { func (h *Handler) buildListView(lib, tab string) (listView, error) {
all, err := h.store.List() // already ordered updated_at DESC all, err := h.store.List(h.readerID) // already ordered updated_at DESC
if err != nil { if err != nil {
return listView{}, err return listView{}, err
} }
@@ -374,7 +378,7 @@ func (h *Handler) loadForMutation(w http.ResponseWriter, r *http.Request) (store
http.Error(w, "missing key", http.StatusBadRequest) http.Error(w, "missing key", http.StatusBadRequest)
return store.Bookmark{}, false return store.Bookmark{}, false
} }
b, ok, err := h.store.Get(key) b, ok, err := h.store.Get(h.readerID, key)
if err != nil { if err != nil {
log.Printf("ui get %q: %v", key, err) log.Printf("ui get %q: %v", key, err)
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
@@ -397,7 +401,7 @@ func (h *Handler) loadForMutation(w http.ResponseWriter, r *http.Request) (store
// describe the whole library, so they are rebuilt out of band on every // describe the whole library, so they are rebuilt out of band on every
// mutation, at the cost of one extra list read per toggle. // mutation, at the cost of one extra list read per toggle.
func (h *Handler) saveAndRenderCard(w http.ResponseWriter, r *http.Request, b store.Bookmark) { func (h *Handler) saveAndRenderCard(w http.ResponseWriter, r *http.Request, b store.Bookmark) {
stored, err := h.store.Upsert(b) stored, err := h.store.Upsert(h.readerID, b)
if err != nil { if err != nil {
log.Printf("ui upsert %q: %v", b.Key, err) log.Printf("ui upsert %q: %v", b.Key, err)
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
@@ -489,7 +493,7 @@ func (h *Handler) uiDelete(w http.ResponseWriter, r *http.Request) {
http.Error(w, "missing key", http.StatusBadRequest) http.Error(w, "missing key", http.StatusBadRequest)
return return
} }
if err := h.store.Delete(key); err != nil { if err := h.store.Delete(h.readerID, key); err != nil {
log.Printf("ui delete %q: %v", key, err) log.Printf("ui delete %q: %v", key, err)
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
return return
+15 -3
View File
@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"crypto/sha256"
"errors" "errors"
"log" "log"
"net/http" "net/http"
@@ -30,6 +31,9 @@ type Config struct {
Port string Port string
// WebPassword gates the browser UI. Empty disables the web routes entirely. // WebPassword gates the browser UI. Empty disables the web routes entirely.
WebPassword string WebPassword string
// OwnerDiscordID identifies the seeded owner Reader (issue #22). Required:
// bookmarks are scoped to a Reader, and without an owner there is none.
OwnerDiscordID string
// UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js. // UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js.
// Supplied by a bindmount so the script can be edited without a rebuild. // Supplied by a bindmount so the script can be edited without a rebuild.
UserscriptPath string UserscriptPath string
@@ -148,6 +152,7 @@ func loadConfig() Config {
DatabaseURL: os.Getenv("DATABASE_URL"), DatabaseURL: os.Getenv("DATABASE_URL"),
Port: envOr("PORT", "8080"), Port: envOr("PORT", "8080"),
WebPassword: os.Getenv("WEB_PASSWORD"), WebPassword: os.Getenv("WEB_PASSWORD"),
OwnerDiscordID: os.Getenv("OWNER_DISCORD_ID"),
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"), UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"), NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"),
LatestPoll: loadLatestPoll(), LatestPoll: loadLatestPoll(),
@@ -173,7 +178,7 @@ func newRouter(s *store.Store, cfg Config) http.Handler {
mux.HandleFunc("GET /u/{token}/manga-bookmark.user.js", userscript.Handler(cfg.Token, cfg.UserscriptPath)) mux.HandleFunc("GET /u/{token}/manga-bookmark.user.js", userscript.Handler(cfg.Token, cfg.UserscriptPath))
mux.HandleFunc("GET /u/{token}/novel-bookmark.user.js", userscript.Handler(cfg.Token, cfg.NovelUserscriptPath)) mux.HandleFunc("GET /u/{token}/novel-bookmark.user.js", userscript.Handler(cfg.Token, cfg.NovelUserscriptPath))
h := &api.Handler{Store: s} h := &api.Handler{Store: s, ReaderID: s.OwnerID()}
protected := http.NewServeMux() protected := http.NewServeMux()
protected.HandleFunc("GET /bookmarks", h.List) protected.HandleFunc("GET /bookmarks", h.List)
protected.HandleFunc("PUT /bookmarks/{key}", h.Put) protected.HandleFunc("PUT /bookmarks/{key}", h.Put)
@@ -187,7 +192,7 @@ func newRouter(s *store.Store, cfg Config) http.Handler {
// deployment that forgets WEB_PASSWORD exposes nothing rather than // deployment that forgets WEB_PASSWORD exposes nothing rather than
// exposing an unprotected list. // exposing an unprotected list.
if cfg.WebPassword != "" { if cfg.WebPassword != "" {
wh, err := web.New(s, cfg.Token, cfg.WebPassword) wh, err := web.New(s, s.OwnerID(), cfg.Token, cfg.WebPassword)
if err != nil { if err != nil {
log.Fatalf("web handler: %v", err) log.Fatalf("web handler: %v", err)
} }
@@ -217,11 +222,18 @@ func main() {
if cfg.Token == "" { if cfg.Token == "" {
log.Fatal("API_TOKEN is required") log.Fatal("API_TOKEN is required")
} }
if cfg.OwnerDiscordID == "" {
log.Fatal("OWNER_DISCORD_ID is required")
}
if cfg.DatabaseURL == "" { if cfg.DatabaseURL == "" {
log.Fatal("DATABASE_URL is required") log.Fatal("DATABASE_URL is required")
} }
s, err := store.Open(cfg.DatabaseURL) // The owner's userscript token is the global API token today (issue #22);
// the readers row carries its SHA-256, not the token itself.
owner := store.Owner{DiscordID: cfg.OwnerDiscordID, TokenHash: sha256.Sum256([]byte(cfg.Token))}
s, err := store.Open(cfg.DatabaseURL, owner)
if err != nil { if err != nil {
log.Fatalf("open store: %v", err) log.Fatalf("open store: %v", err)
} }
+17 -17
View File
@@ -56,7 +56,7 @@ func TestIndexWithoutSessionShowsLogin(t *testing.T) {
func TestIndexWithSessionShowsList(t *testing.T) { func TestIndexWithSessionShowsList(t *testing.T) {
cfg := webConfig() cfg := webConfig()
srv, st := newWebTestServer(t, cfg) srv, st := newWebTestServer(t, cfg)
if _, err := st.Upsert(store.Bookmark{ if _, err := st.Upsert(st.OwnerID(), store.Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", LastChapter: "45", LastChapterNum: 45, Title: "Solo Leveling", LastChapter: "45", LastChapterNum: 45,
UpdatedAt: time.Now().UnixMilli(), UpdatedAt: time.Now().UnixMilli(),
@@ -203,7 +203,7 @@ func TestStaticAssetsServed(t *testing.T) {
// seed inserts one bookmark and returns it as stored. // seed inserts one bookmark and returns it as stored.
func seed(t *testing.T, st *store.Store, b store.Bookmark) store.Bookmark { func seed(t *testing.T, st *store.Store, b store.Bookmark) store.Bookmark {
t.Helper() t.Helper()
stored, err := st.Upsert(b) stored, err := st.Upsert(st.OwnerID(), b)
if err != nil { if err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
@@ -257,7 +257,7 @@ func TestFavoriteTogglesWithoutReordering(t *testing.T) {
t.Fatalf("favorite status = %d, want 200", rr.Code) t.Fatalf("favorite status = %d, want 200", rr.Code)
} }
after, ok, err := st.Get("asura:solo") after, ok, err := st.Get(st.OwnerID(), "asura:solo")
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("Get after favorite: %v ok=%v", err, ok) t.Fatalf("Get after favorite: %v ok=%v", err, ok)
} }
@@ -275,7 +275,7 @@ func TestFavoriteTogglesWithoutReordering(t *testing.T) {
// Toggling again turns it back off. // Toggling again turns it back off.
rr = httptest.NewRecorder() rr = httptest.NewRecorder()
srv.ServeHTTP(rr, uiRequest(t, cfg, http.MethodPost, "/ui/bookmarks/asura:solo/favorite", nil)) srv.ServeHTTP(rr, uiRequest(t, cfg, http.MethodPost, "/ui/bookmarks/asura:solo/favorite", nil))
back, _, _ := st.Get("asura:solo") back, _, _ := st.Get(st.OwnerID(), "asura:solo")
if back.Favorite { if back.Favorite {
t.Fatal("Favorite = true after a second toggle, want false") t.Fatal("Favorite = true after a second toggle, want false")
} }
@@ -334,7 +334,7 @@ func TestChapterOverrideMovesUpdatedAt(t *testing.T) {
t.Fatalf("chapter override status = %d, want 200", rr.Code) t.Fatalf("chapter override status = %d, want 200", rr.Code)
} }
after, ok, err := st.Get("asura:solo") after, ok, err := st.Get(st.OwnerID(), "asura:solo")
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("Get after override: %v ok=%v", err, ok) t.Fatalf("Get after override: %v ok=%v", err, ok)
} }
@@ -374,7 +374,7 @@ func TestChapterOverrideNoOpPreservesURLAndUpdatedAt(t *testing.T) {
t.Fatalf("chapter no-op status = %d, want 200", rr.Code) t.Fatalf("chapter no-op status = %d, want 200", rr.Code)
} }
after, ok, err := st.Get("asura:solo") after, ok, err := st.Get(st.OwnerID(), "asura:solo")
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("Get after no-op override: %v ok=%v", err, ok) t.Fatalf("Get after no-op override: %v ok=%v", err, ok)
} }
@@ -408,7 +408,7 @@ func TestChapterOverrideRejectsBadInput(t *testing.T) {
if rr.Code != http.StatusBadRequest { if rr.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", rr.Code) t.Fatalf("status = %d, want 400", rr.Code)
} }
after, _, _ := st.Get("asura:solo") after, _, _ := st.Get(st.OwnerID(), "asura:solo")
if after.LastChapterNum != 45 { if after.LastChapterNum != 45 {
t.Fatalf("chapter changed to %v on invalid input", after.LastChapterNum) t.Fatalf("chapter changed to %v on invalid input", after.LastChapterNum)
} }
@@ -459,7 +459,7 @@ func TestUIDeleteRemovesRow(t *testing.T) {
if !strings.Contains(body, `id="new-count" hx-swap-oob="true"`) { if !strings.Contains(body, `id="new-count" hx-swap-oob="true"`) {
t.Fatalf("delete body = %q, want the out-of-band badge", body) t.Fatalf("delete body = %q, want the out-of-band badge", body)
} }
if _, ok, _ := st.Get("asura:solo"); ok { if _, ok, _ := st.Get(st.OwnerID(), "asura:solo"); ok {
t.Fatal("row still present after delete") t.Fatal("row still present after delete")
} }
} }
@@ -538,7 +538,7 @@ func seedStatusRows(t *testing.T, st *store.Store) {
} }
for _, b := range rows { for _, b := range rows {
b.UpdatedAt = time.Now().UnixMilli() b.UpdatedAt = time.Now().UnixMilli()
if _, err := st.Upsert(b); err != nil { if _, err := st.Upsert(st.OwnerID(), b); err != nil {
t.Fatalf("seed %s: %v", b.Key, err) t.Fatalf("seed %s: %v", b.Key, err)
} }
} }
@@ -612,7 +612,7 @@ func TestRecentStripCarriesUnreadOnlyAndOnlyOnAll(t *testing.T) {
Status: store.StatusReading, LastChapterNum: 40, LatestChapter: "40", Status: store.StatusReading, LastChapterNum: 40, LatestChapter: "40",
LatestChapterNum: floatPtr(40), UpdatedAt: time.Now().UnixMilli(), LatestChapterNum: floatPtr(40), UpdatedAt: time.Now().UnixMilli(),
} }
if _, err := st.Upsert(caught); err != nil { if _, err := st.Upsert(st.OwnerID(), caught); err != nil {
t.Fatalf("seed %s: %v", caught.Key, err) t.Fatalf("seed %s: %v", caught.Key, err)
} }
@@ -632,12 +632,12 @@ func TestRecentStripCarriesUnreadOnlyAndOnlyOnAll(t *testing.T) {
} }
// Nothing new anywhere: the strip has nothing to say and does not render. // Nothing new anywhere: the strip has nothing to say and does not render.
reading, _, err := st.Get("asura:reading") reading, _, err := st.Get(st.OwnerID(), "asura:reading")
if err != nil { if err != nil {
t.Fatalf("Get: %v", err) t.Fatalf("Get: %v", err)
} }
reading.LatestChapterNum = floatPtr(reading.LastChapterNum) reading.LatestChapterNum = floatPtr(reading.LastChapterNum)
if _, err := st.Upsert(reading); err != nil { if _, err := st.Upsert(st.OwnerID(), reading); err != nil {
t.Fatalf("Upsert: %v", err) t.Fatalf("Upsert: %v", err)
} }
// The section still ships (an out-of-band swap needs the id to exist) but // The section still ships (an out-of-band swap needs the id to exist) but
@@ -662,7 +662,7 @@ func TestRecentStripCapped(t *testing.T) {
Status: store.StatusReading, LastChapterNum: 1, LatestChapter: "2", Status: store.StatusReading, LastChapterNum: 1, LatestChapter: "2",
LatestChapterNum: floatPtr(2), UpdatedAt: time.Now().UnixMilli() + int64(i), LatestChapterNum: floatPtr(2), UpdatedAt: time.Now().UnixMilli() + int64(i),
} }
if _, err := st.Upsert(b); err != nil { if _, err := st.Upsert(st.OwnerID(), b); err != nil {
t.Fatalf("seed %s: %v", b.Key, err) t.Fatalf("seed %s: %v", b.Key, err)
} }
} }
@@ -692,7 +692,7 @@ func TestUIStatusSetsBucket(t *testing.T) {
if rr := postStatus(t, srv, cfg, "asura:reading", want); rr.Code != http.StatusOK { if rr := postStatus(t, srv, cfg, "asura:reading", want); rr.Code != http.StatusOK {
t.Fatalf("set %s: status = %d, body %s", want, rr.Code, rr.Body.String()) t.Fatalf("set %s: status = %d, body %s", want, rr.Code, rr.Body.String())
} }
b, ok, err := st.Get("asura:reading") b, ok, err := st.Get(st.OwnerID(), "asura:reading")
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("Get: ok=%v err=%v", ok, err) t.Fatalf("Get: ok=%v err=%v", ok, err)
} }
@@ -710,7 +710,7 @@ func TestUIStatusRejectsUnknownValue(t *testing.T) {
if rr := postStatus(t, srv, cfg, "asura:reading", "dropped"); rr.Code != http.StatusBadRequest { if rr := postStatus(t, srv, cfg, "asura:reading", "dropped"); rr.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", rr.Code) t.Fatalf("status = %d, want 400", rr.Code)
} }
b, _, _ := st.Get("asura:reading") b, _, _ := st.Get(st.OwnerID(), "asura:reading")
if b.Status != store.StatusReading { if b.Status != store.StatusReading {
t.Fatalf("stored status = %q, want it untouched", b.Status) t.Fatalf("stored status = %q, want it untouched", b.Status)
} }
@@ -736,12 +736,12 @@ func TestUIStatusDoesNotReorderList(t *testing.T) {
srv, st := newWebTestServer(t, cfg) srv, st := newWebTestServer(t, cfg)
seedStatusRows(t, st) seedStatusRows(t, st)
before, _, _ := st.Get("asura:reading") before, _, _ := st.Get(st.OwnerID(), "asura:reading")
time.Sleep(2 * time.Millisecond) time.Sleep(2 * time.Millisecond)
if rr := postStatus(t, srv, cfg, "asura:reading", store.StatusArchived); rr.Code != http.StatusOK { if rr := postStatus(t, srv, cfg, "asura:reading", store.StatusArchived); rr.Code != http.StatusOK {
t.Fatalf("status = %d", rr.Code) t.Fatalf("status = %d", rr.Code)
} }
after, _, _ := st.Get("asura:reading") after, _, _ := st.Get(st.OwnerID(), "asura:reading")
if after.UpdatedAt != before.UpdatedAt { if after.UpdatedAt != before.UpdatedAt {
t.Fatalf("UpdatedAt moved %d -> %d", before.UpdatedAt, after.UpdatedAt) t.Fatalf("UpdatedAt moved %d -> %d", before.UpdatedAt, after.UpdatedAt)
} }
+2
View File
@@ -15,6 +15,8 @@ services:
environment: environment:
# API_TOKEN is required — compose refuses to start without it. # API_TOKEN is required — compose refuses to start without it.
API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env} API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env}
# Owner's Discord user ID — required, seeds the one Reader row.
OWNER_DISCORD_ID: ${OWNER_DISCORD_ID:?set OWNER_DISCORD_ID in .env}
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net} ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
# The bookmarks database. Host is the compose service name; the password # The bookmarks database. Host is the compose service name; the password
# comes from .env so it is never committed. # comes from .env so it is never committed.