From 249aacab2e5a0854612620dda659988808fdbd81 Mon Sep 17 00:00:00 2001 From: Sulthan Zaki Date: Sat, 8 Aug 2026 06:43:53 +0700 Subject: [PATCH] feat(backend)!: run on Postgres with a migration-owned schema MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swap modernc.org/sqlite for jackc/pgx/v5 with no observable change: same endpoints, same wire format, same updated_at ordering rule. The schema now comes from numbered SQL embedded in the binary and applied on startup, one transaction each, recorded in schema_migrations. That replaces two pieces of SQLite-era machinery, both deleted rather than ported: the column probing (Postgres has ADD COLUMN IF NOT EXISTS, and there is no legacy database left to probe) and the Asura key rewrite, which has run clean on every start for months now that the userscripts strip build hashes before writing. Its regexp survives as latest.asuraBuildHash, where the poller still needs it to scope chapter links to a series whose slug carries a rotating hash. Types get real: favorite is a boolean, chapter numbers double precision, timestamps stay unix-ms bigint. SQLite's null-safe IS NOT becomes IS DISTINCT FROM, which is what implements the rule that only reading progress reorders a list. Inside COALESCE/NULLIF the status and kind parameters need an explicit ::text — there is no target column to infer from and Postgres refuses to guess. Tests lose their free t.TempDir() database, so Docker is now a hard prerequisite for `go test ./...`: internal/pgtest starts one postgres:17-alpine per test binary and hands each test a database of its own. Hand-rolled rather than testcontainers — it is one docker run, one docker port and a ping loop against a module list that is otherwise stdlib. BREAKING CHANGE: DB_PATH is retired for DATABASE_URL, which is required and has no default. Compose gains a postgres service on an internal network with its own volume; POSTGRES_PASSWORD joins .env. The old bookmarks-data volume is deliberately left undeclared so `docker compose down -v` cannot take the pre-migration database with it. Closes #20 --- .env.example | 8 + AGENTS.md | 8 +- DEPLOY.md | 41 ++- README.md | 14 +- REDEPLOY.md | 215 ++++++----- backend/AGENTS.md | 19 +- backend/Dockerfile | 10 +- backend/api_test.go | 33 +- backend/go.mod | 18 +- backend/go.sum | 58 +-- backend/internal/latest/poller_test.go | 9 +- backend/internal/latest/sites.go | 14 +- backend/internal/pgtest/pgtest.go | 120 +++++++ .../store/migrations/0001_bookmarks.sql | 28 ++ backend/internal/store/store.go | 291 +++++---------- backend/internal/store/store_test.go | 337 ++---------------- backend/main.go | 16 +- backend/web_test.go | 7 +- docker-compose.prod.yml | 13 +- docker-compose.yml | 42 ++- 20 files changed, 586 insertions(+), 715 deletions(-) create mode 100644 backend/internal/pgtest/pgtest.go create mode 100644 backend/internal/store/migrations/0001_bookmarks.sql diff --git a/.env.example b/.env.example index 61b3f85..e3577f2 100644 --- a/.env.example +++ b/.env.example @@ -9,6 +9,14 @@ API_TOKEN=changeme-generate-a-long-random-token # sites' hostnames change. ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net +# Password for the bundled Postgres container, and therefore half of the +# DATABASE_URL compose builds for the backend. Generate one: +# openssl rand -hex 24 +POSTGRES_PASSWORD=changeme-generate-a-long-random-password + +# Override only to point the backend at a Postgres compose does not run. +# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require + # --- Prod override (Traefik) only --- # Subdomain Traefik routes to this service (required by the prod override). # BOOKMARK_API_HOST=bookmark-api.example.com diff --git a/AGENTS.md b/AGENTS.md index 1730d07..7e882d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ Userscript targets **Violentmonkey**, so `GM_*` APIs available, but stay GM-free ``` Two Violentmonkey userscripts (isolated world, per-site adapters, localStorage cache) - -- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume) + -- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume) ``` Backend-specific architecture (packages, endpoints, poller, config env vars) lives in `backend/AGENTS.md`. Userscript-specific structure (adapters, retry queue, UI, live URL shapes) lives in `userscript/AGENTS.md`. @@ -33,11 +33,11 @@ Backend-specific architecture (packages, endpoints, poller, config env vars) liv ## Commands Backend (`cd backend`): -- Test all: `go test ./...` +- Test all: `go test ./...` — **needs Docker.** Each test package starts a throwaway `postgres:17-alpine` container (`internal/pgtest`). - Single test: `go test -run TestName ./...` - Build static binary: `CGO_ENABLED=0 go build` -Local stack: `docker compose up` (named volume mounted at `/data`, `restart: unless-stopped`). +Local stack: `docker compose up` (bookmark-api + postgres + headless-shell; `postgres-data` named volume, `restart: unless-stopped`). Smoke test: `curl` endpoints with `Authorization: Bearer `; confirm `OPTIONS` preflight return CORS headers and `/healthz` return 200. @@ -79,7 +79,7 @@ Anchored to OWASP Top 10 / ASVS. Every rule below already has a working example Go backend: -- SQL always parameterized (`?`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one. +- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one. - `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on. - Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one. - Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end. diff --git a/DEPLOY.md b/DEPLOY.md index b5911e9..762bcb4 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -38,6 +38,14 @@ API_TOKEN= # CORS allowlist — leave as-is unless a site changes hostname. ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to +# Required — password for the bundled Postgres container. Compose builds the +# backend's DATABASE_URL out of it and has no fallback for either. +POSTGRES_PASSWORD= + +# Leave unset. Only set this to point the backend at a Postgres compose does +# not run; it then replaces the URL built from POSTGRES_PASSWORD above. +# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require + # Required for the Traefik override. Both have no fallback — compose refuses # to start without them. BOOKMARK_WEB_HOST is required even if you never set # WEB_PASSWORD; see 1b. @@ -50,13 +58,20 @@ BOOKMARK_WEB_HOST=bookmark.violetcrown.my.id # TRAEFIK_CERTRESOLVER=le ``` -Generate + insert the token in one line: +Generate + insert the two secrets in three lines: ```bash sed -i "s|^API_TOKEN=.*|API_TOKEN=$(openssl rand -hex 32)|" .env +sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 24)|" .env grep -E '^API_TOKEN=' .env # copy this — the userscript needs the same value ``` +`POSTGRES_PASSWORD` is read **only while the `postgres-data` volume is empty**, +which in practice means at first boot. Changing it afterwards changes the URL +the backend dials but not the password the database expects, and `bookmark-api` +crash-loops on `password authentication failed`. Set it before §2 and leave it +alone. + > Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual > names (check your Traefik static config — common alternatives: `https`, > `myresolver`, `cloudflare`). Wrong names = no certificate issued. @@ -116,16 +131,21 @@ This merges the base file (build/image/env/volume) with the prod override (no host port, Traefik network + router labels). Always pass **both** `-f` flags — the prod file is not standalone. -Two services come up: `bookmark-api` (the backend) and `headless-shell`, a CDP -sidecar the poller uses to fetch kagane (behind a Cloudflare JS challenge). -It has no published port — only `bookmark-api` can reach it, over -`BROWSER_WS_URL`. Missing or unreachable, the poller just skips kagane and -logs it; nothing else is affected. +Three services come up: `bookmark-api` (the backend), `postgres` (its database, +`postgres:17-alpine`), and `headless-shell`, a CDP sidecar the poller uses to +fetch kagane (behind a Cloudflare JS challenge). Neither of the latter two +publishes a port: `postgres` sits alone with `bookmark-api` on an +`internal: true` network, and `headless-shell` is reachable only over +`BROWSER_WS_URL`. A missing headless-shell just makes the poller skip kagane and +log it. A missing Postgres stops everything — `bookmark-api` waits for +`pg_isready` to pass, then applies its embedded migrations, and only then +listens. The schema is created that way; there is nothing to import by hand. Check it's up and healthy: ```bash docker compose -f docker-compose.yml -f docker-compose.prod.yml ps +# bookmark-api Up; postgres Up (healthy) docker logs bookmark-api --tail 20 # expect: "listening on :8080 ..." ``` @@ -214,7 +234,10 @@ Pull new code, then rebuild: docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build ``` -SQLite data persists in the named volume `bookmarks-data` across rebuilds. +Data persists in the named volume `postgres-data` across rebuilds. (If this +server predates the Postgres migration, the old SQLite volume `bookmarks-data` +is still on disk and deliberately undeclared in compose so `down -v` cannot take +it; see `REDEPLOY.md` §1 for when to remove it.) --- @@ -227,7 +250,9 @@ SQLite data persists in the named volume `bookmarks-data` across rebuilds. | `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). | | 401 with the right token | Trailing space/newline in `API_TOKEN`; regenerate and restart. | | Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. | -| `compose ... config` errors about `API_TOKEN` | Run compose from the dir with `.env`, or export the vars. | +| `compose ... config` errors about `API_TOKEN` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. Both are required and neither has a fallback. | +| `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). | +| `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. | Backend config reference and endpoint list: see `README.md`. diff --git a/README.md b/README.md index 2174af3..fa2c67c 100644 --- a/README.md +++ b/README.md @@ -7,14 +7,14 @@ all four sites and all devices. Two parts: -- **`backend/`** — tiny Go (`net/http` + pure-Go SQLite) sync service. 4 routes, +- **`backend/`** — tiny Go (`net/http` + Postgres via pure-Go `pgx`) sync service. 4 routes, static binary, distroless container. - **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript (no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`. ``` Bromite userscript (isolated world, Shadow DOM UI, localStorage cache) - -- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> SQLite (volume) + -- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume) ``` --- @@ -27,7 +27,7 @@ Bromite userscript (isolated world, Shadow DOM UI, localStorage cache) |-----|---------|-------| | `API_TOKEN` | *(required)* | Bearer token shared with the userscript. | | `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. | -| `DB_PATH` | `/data/bookmarks.db` | SQLite file location. | +| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. | | `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. | | `BROWSER_WS_URL` | `ws://172.28.0.10:9222` | Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header. | @@ -60,11 +60,17 @@ go test ./... # unit + handler tests CGO_ENABLED=0 go build # static binary ``` +**`go test ./...` requires Docker.** The store talks to a real Postgres, so +each test package starts a throwaway `postgres:17-alpine` container and gives +every test its own database inside it (`internal/pgtest`). Nothing is stubbed +and nothing reaches the network beyond the local Docker daemon. + ### Run the stack ```bash cp .env.example .env -# edit .env: set API_TOKEN (openssl rand -hex 32) +# edit .env: set API_TOKEN (openssl rand -hex 32) and +# POSTGRES_PASSWORD (openssl rand -hex 24) docker compose up -d --build # binds 127.0.0.1:8080 ``` diff --git a/REDEPLOY.md b/REDEPLOY.md index 9a714c1..5118915 100644 --- a/REDEPLOY.md +++ b/REDEPLOY.md @@ -18,7 +18,7 @@ the checkout cannot take the backups with them. ``` /opt/ ├── bookmarkmanager/ <- the checkout (this repo) -└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.db +└── bookmarkmanager-backups/ <- bookmarks-YYYYmmdd-HHMMSS.dump ``` --- @@ -53,78 +53,97 @@ echo "$BACKUP_DIR" # -> /opt/bookmarkmanager-backups ## 1. Back up the database -The database is a single SQLite file in the named Docker volume, at -`/data/bookmarks.db` inside the container. Find the volume's real name — Compose -prefixes it with the project directory: +The database is Postgres, running as the `postgres` service on the named volume +`postgres-data`. It has **no published port** — nothing outside the internal `db` +network can reach it — so every command below goes in through the container: ```bash -docker volume ls --filter name=bookmarks-data -# -> local bookmarkmanager_bookmarks-data -VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1) +$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt' +# -> bookmarks, schema_migrations ``` -### Preferred: hot backup, no downtime +Inside the container that connects over the local socket as the `bookmarks` +superuser, so no password is needed anywhere in this section. `-T` is not +optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and +silently corrupts any binary stream flowing back out — see the dump below. -The store runs in **WAL mode**, so recent writes may still be sitting in -`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can -therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and -writes one consistent file, safe to run against a live database: +### Preferred: hot dump, no downtime + +`pg_dump` runs in a single repeatable-read transaction, so it writes one +point-in-time-consistent snapshot while the API keeps serving. No stopping, no +WAL to worry about — that is the server's problem, not yours. ```bash STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text -docker run --rm \ - -v "$VOL":/data \ - -v "$BACKUP_DIR":/backup \ - alpine sh -c "apk add -q sqlite && - sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\"" +$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \ + > "$BACKUP_DIR/bookmarks-$STAMP.dump" -ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db +ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump ``` -`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the -files sort in real order and never collide across a DST shift. +`-Fc` is the custom archive format rather than plain SQL: it is compressed, and +`pg_restore` can inspect and replay it selectively — list its table of contents, +restore one table, restore schema without data, reorder. A plain `.sql` dump can +only be piped into `psql` whole, and gives you no way to check what is in it +short of reading it. -Note the source volume is mounted **read-write**, which looks wrong for a backup -and is not. Opening a WAL database requires creating the `-shm` shared-memory -file; with `:ro` the command fails with `unable to open database file` and no -backup is produced. `VACUUM INTO` never writes to the source itself. +`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.dump`. UTC, so +the files sort in real order and never collide across a DST shift. Verify it before you trust it. An unreadable backup is worse than none, because you will act as though you have one: ```bash -docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite && - sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' && - sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'" -# -> ok +# 1. The dump parses and contains the tables. Uses the same image compose +# already pulls, so nothing new to install. +docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \ + pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA' +# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks +# -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks + +# 2. Sanity-check the live row count you just captured. +$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \ + -c 'select count(*) from bookmarks' # -> 37 ``` -The count should match what the web UI shows. Zero rows on a server you know has -bookmarks means you backed up the wrong volume. +A custom-format archive stores row counts nowhere, so step 1 proves the file is +a readable archive with the right tables in it, not that the rows are there; +step 2 is the number those rows should be. It should match what the web UI +shows. Zero on a server you know has bookmarks means the API and your `psql` +are looking at different databases — check `DATABASE_URL`. -### Fallback: cold copy (no network for `apk add sqlite`) +### Fallback: cold volume archive -Stop the service first, then copy the database **and its sidecars** — the `-wal` -is not optional, it is where the newest writes are: +Use this when you want the whole data directory rather than a logical dump — a +like-for-like restore of the same Postgres major version onto the same host. + +**The stack must be stopped first.** A running Postgres has dirty pages in +shared buffers and WAL that has not been replayed into the data files, and `tar` +walks the directory over several seconds while the server keeps writing to it. +The archive you get is torn: files from different instants, possibly a +half-written page. It may restore, start, and be quietly wrong. Online +filesystem-level backup is `pg_basebackup`'s job, not `tar`'s; with the +container stopped the shutdown checkpoint has already flushed everything and a +plain archive of the volume is consistent. ```bash +VOL=$(docker volume ls --filter name=postgres-data -q | head -1) +echo "$VOL" # -> bookmarkmanager_postgres-data + $COMPOSE stop -docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c " - cp /data/bookmarks.db /backup/bookmarks-$STAMP.db - [ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal - [ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm - ls -1 /backup" +docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \ + tar czf "/to/postgres-data-$STAMP.tgz" -C /from . $COMPOSE start + +ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz ``` -Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away, -so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for -the case where it did not. Restoring this variant means putting whichever files -you got back together, under their original names. - -Read-only is safe here precisely because nothing opens the database: it is a file -copy, not a SQLite connection. +Costs ~15 seconds of downtime. Read-only on the source is safe here precisely +because nothing is running against it. Restoring this variant means untarring it +back into an *empty* `postgres-data` volume with the stack down — it is a whole +data directory, not a file you can drop next to the live one, and it will only +start under `postgres:17`. ### Retention @@ -132,7 +151,19 @@ Keep a month, drop the rest — a bookmark database this small compresses the decision to "disk is free, but not infinite": ```bash -ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v +ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v +``` + +### A note on the old `bookmarks-data` volume + +`bookmarks-data` is the **pre-migration SQLite volume**. It is deliberately not +declared in `docker-compose.yml` any more, which is what keeps `docker compose +down -v` from taking it with the rest of the stack. It is not the live database +and nothing reads it. Once the Postgres data has been trusted for a while, +remove it by hand — nothing else will: + +```bash +docker volume rm bookmarkmanager_bookmarks-data ``` --- @@ -171,11 +202,15 @@ rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is bindmounted read-only and read fresh per request. ```bash -$COMPOSE ps # Up, and recently (re)created +$COMPOSE ps # bookmark-api Up; postgres Up (healthy) docker logs bookmark-api --tail 20 # -> "listening on :8080 ..." ``` -Nothing in the log about the database or the poller failing. The image is tagged +Nothing in the log about the database, the migrations or the poller failing. +`bookmark-api` waits on `postgres` reporting healthy before it starts and the +binary applies any pending migration before it listens, so an API that never +says "listening" is usually the database, not the code — `$COMPOSE logs +postgres` first. The image is tagged `bookmarkmanager-backend:latest`, so the previous image is still on disk untagged — that is what makes the rollback in §6 quick. @@ -199,9 +234,16 @@ curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \ $API/bookmarks/x | grep -i access-control # -> allow-origin echoed ``` -`[]` from the third call is the alarm that matters: the volume is not attached -and you are looking at an empty database. Stop and check `$COMPOSE config ---volumes` before touching anything else. +`[]` from the third call is the alarm that matters: you are talking to an empty +database, which means the API found a *different* Postgres than the one holding +your data — a renamed project directory, a fresh `postgres-data`, or a +`DATABASE_URL` override in `.env` pointing elsewhere. Stop and check, before +touching anything else: + +```bash +$COMPOSE config --volumes # -> postgres-data +$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks' +``` Web UI and its assets: @@ -267,33 +309,41 @@ git checkout $COMPOSE up -d --build ``` -**Database damaged** — restore the backup from §1. Stop first: the running -process holds the WAL, and dropping a file under a live SQLite connection -corrupts what you were trying to save. +**Database damaged** — restore the dump from §1. Stop **only the API**, not the +whole stack: `pg_restore` needs the server up to restore into, and it needs +`bookmark-api`'s connection pool gone, because `--clean` cannot drop a table +other sessions are holding open. ```bash -$COMPOSE stop +$COMPOSE stop bookmark-api -docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c ' - rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm && - cp /backup/bookmarks-.db /data/bookmarks.db && - chown 65532:65532 /data/bookmarks.db && - ls -l /data' +$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \ + < "$BACKUP_DIR/bookmarks-.dump" -$COMPOSE start +$COMPOSE start bookmark-api docker logs bookmark-api --tail 20 curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200 ``` -Two steps here are easy to skip and both bite: +Three things here are easy to skip and all three bite: -- **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database - mixes two different histories; SQLite will either refuse to open it or quietly - reapply writes you meant to discard. -- **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as - that uid, while the helper container above writes as root. A root-owned - database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then - every write fails. That is the worst possible failure mode — it looks restored. +- **`--clean --if-exists`.** Without `--clean` the dump's rows land *on top of* + what is already there and you get primary-key collisions half way through, a + partially restored database, and a non-zero exit you may not notice. + `--if-exists` only suppresses the "does not exist" noise when the target is + already empty; it is not the part doing the work. +- **`-T` again.** Feeding a custom-format archive into a TTY-allocated `exec` + corrupts it in flight and `pg_restore` fails with a garbled-header error on a + file that is perfectly fine on disk. +- **Stop the API, not Postgres.** `$COMPOSE stop` (everything) leaves you with + nothing to restore into; leaving `bookmark-api` running leaves connections + that block the drops *and* lets the poller write into a half-restored table. + +No ownership fixing is needed any more — the Postgres image owns `postgres-data` +itself and `pg_restore` writes through the server, not the filesystem. +`schema_migrations` is inside the dump, so the database comes back at whatever +schema version the backup was taken at; the migration runner applies anything +newer the next time `bookmark-api` starts. --- @@ -305,21 +355,24 @@ For a routine redeploy where nothing needs deciding: cd /opt/bookmarkmanager COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml" BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR" -VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1) STAMP=$(date -u +%Y%m%d-%H%M%S) -docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \ - "apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" && - sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" && +$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \ + > "$BACKUP_DIR/bookmarks-$STAMP.dump" && +docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \ + pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null && git pull --ff-only && $COMPOSE up -d --build && sleep 5 && curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok" ``` -The `&&` chain is deliberate: if the backup or its integrity check fails, -nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell -command can tell you the panel works on the phone. +The `&&` chain is deliberate: if the dump or its `pg_restore --list` check +fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a +short or empty `.dump` behind — the shell creates the file before `pg_dump` +runs — so delete it rather than letting it sit in the backup directory looking +like a backup. Then still do §5 by hand — no shell command can tell you the +panel works on the phone. --- @@ -327,16 +380,18 @@ command can tell you the panel works on the phone. | Symptom | Cause / fix | |---|---| -| `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. | +| `/bookmarks` returns `[]` after redeploy | You are on an empty Postgres. Check `$COMPOSE config --volumes` lists `postgres-data`, that you passed both `-f` files, and that `.env` has no stray `DATABASE_URL` override. Do **not** re-bookmark; the data is still in the volume. | | UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. | | CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. | | Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. | | Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. | | `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. | | Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. | -| `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. | -| Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). | -| Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). | +| `bookmark-api` crash-loops, log says `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` in `.env` no longer matches the one burned into `postgres-data` at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: `$COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks'` (prompts, so nothing lands in shell history) and then match `.env` to it. | +| `compose` errors `set POSTGRES_PASSWORD in .env` | Unset. Compose builds the backend's `DATABASE_URL` out of it, so it is required even though you never write that URL yourself. Run from the directory holding `.env`. | +| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. | +| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. | +| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). | Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`. UI conventions: `docs/design-system.md`. diff --git a/backend/AGENTS.md b/backend/AGENTS.md index c29bdb4..d682c1c 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -1,8 +1,8 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGENTS.md` for the project-wide architecture diagram, hard constraints, and design system. -- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + `modernc.org/sqlite` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`. +- **Backend** (`backend/`): stdlib `net/http` (handful routes, no framework) + Postgres over `jackc/pgx/v5` (pure Go, `CGO_ENABLED=0` -> static binary -> distroless/scratch image). Reverse proxy terminates TLS; Go service listens plain `:8080`. Single binary, split into packages under `backend/internal/`: `store` - (Bookmark type, SQLite persistence, migrations), `latest` (background + (Bookmark type, Postgres persistence, migration runner), `latest` (background poller, site parsers, TLS fetcher), `session` (cookie signing, login rate limiter), `httpmw` (Auth/Gzip/CORS middleware), `api` (JSON bookmark handlers), `userscript` (userscript-serving handler), `web` @@ -11,6 +11,16 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN packages together into `newRouter`. Root-level `*_test.go` hold integration tests that exercise the full router; unit tests for a package live beside it under `internal/`. +- **Schema is migration-owned.** `internal/store/migrations/*.sql` is + `go:embed`-ed and applied on every start by `store.migrate`: one numbered + file per change, one transaction each, versions recorded in + `schema_migrations`. Files are **append-only** — editing an applied one + changes nothing on a database that already ran it. No column probing, no + data-fixup migrations: both were SQLite-era machinery and are gone. +- **Tests need Docker.** `internal/pgtest` starts one `postgres:17-alpine` + container per test binary (`TestMain` -> `pgtest.Main`) and hands each test + its own database (`pgtest.URL(t)`). A package whose tests touch the store + must have that `TestMain`. - **Single-user store.** One `bookmarks` table keyed `:` (`asura`|`demonic`|`comix`|`kagane`|`novelfull`|`lightnovelworld`), with a `kind` column (`manga`|`novel`) splitting the two libraries. Sync **last-write-wins**. Schema and endpoint list in plan. - **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth). - **Web UI:** same binary serve password-gated browser UI on second @@ -70,8 +80,9 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN `excluded.*` is post-evaluation row and default applied there would wipe bucket on every PUT from client that predates column. See `docs/superpowers/specs/2026-07-27-status-buckets-design.md`. -- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH` - (default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD` +- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), + `DATABASE_URL` (Postgres connection URL, required — no default), + `PORT` (default `8080`), `WEB_PASSWORD` (gates browser UI; unset disable it), `LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`). diff --git a/backend/Dockerfile b/backend/Dockerfile index 15afbd4..b40f8b3 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -14,21 +14,17 @@ RUN go mod download COPY *.go ./ COPY internal/ ./internal/ -# Static binary: pure-Go sqlite means CGO_ENABLED=0 -> no libc dependency. +# Static binary: the Postgres driver (jackc/pgx) is pure Go, so CGO_ENABLED=0 +# leaves no libc dependency. # -trimpath + -ldflags strip paths and debug info for a smaller image. RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server . -# Data dir with the runtime user's ownership so the mounted volume inherits it. -RUN mkdir -p /out/data - # --- runtime stage: distroless static, non-root --- FROM gcr.io/distroless/static:nonroot WORKDIR / COPY --from=build /out/server /server -COPY --from=build --chown=65532:65532 /out/data /data -VOLUME ["/data"] EXPOSE 8080 USER nonroot:nonroot -ENV DB_PATH=/data/bookmarks.db PORT=8080 +ENV PORT=8080 ENTRYPOINT ["/server"] diff --git a/backend/api_test.go b/backend/api_test.go index fafc1fa..e4282c8 100644 --- a/backend/api_test.go +++ b/backend/api_test.go @@ -12,6 +12,7 @@ import ( "testing" "time" + "bookmarkmanager/backend/internal/pgtest" "bookmarkmanager/backend/internal/store" ) @@ -25,15 +26,21 @@ func testConfig() Config { } } +func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } + func newTestServer(t *testing.T) http.Handler { t.Helper() - dbPath := filepath.Join(t.TempDir(), "test.db") - s, err := store.Open(dbPath) + return newRouter(newTestStore(t), testConfig()) +} + +func newTestStore(t *testing.T) *store.Store { + t.Helper() + s, err := store.Open(pgtest.URL(t)) if err != nil { t.Fatalf("store.Open: %v", err) } t.Cleanup(func() { s.Close() }) - return newRouter(s, testConfig()) + return s } func auth(req *http.Request) *http.Request { @@ -417,12 +424,7 @@ func TestLoadConfigWebPassword(t *testing.T) { // moved into bookmarkColumns, this test catches it: the PUT would reset the // cooldown and the poller would re-fetch that series on every single tick. func TestPutDoesNotClobberLatestCheckedAt(t *testing.T) { - dbPath := filepath.Join(t.TempDir(), "test.db") - s, err := store.Open(dbPath) - if err != nil { - t.Fatalf("store.Open: %v", err) - } - t.Cleanup(func() { s.Close() }) + s := newTestStore(t) srv := newRouter(s, testConfig()) seedForCheck(t, s, "asura:x", "https://asurascans.com/comics/x", 777) @@ -454,12 +456,7 @@ func TestUserscriptServedWithWebUIDisabled(t *testing.T) { t.Fatalf("write script: %v", err) } - dbPath := filepath.Join(t.TempDir(), "nopass.db") - s, err := store.Open(dbPath) - if err != nil { - t.Fatalf("store.Open: %v", err) - } - t.Cleanup(func() { s.Close() }) + s := newTestStore(t) cfg := testConfig() // WebPassword empty cfg.UserscriptPath = path @@ -480,11 +477,7 @@ func TestNovelUserscriptServed(t *testing.T) { t.Fatalf("write script: %v", err) } - s, err := store.Open(filepath.Join(dir, "test.db")) - if err != nil { - t.Fatalf("store.Open: %v", err) - } - t.Cleanup(func() { s.Close() }) + s := newTestStore(t) cfg := testConfig() cfg.NovelUserscriptPath = novelPath diff --git a/backend/go.mod b/backend/go.mod index 43db272..a9848d3 100644 --- a/backend/go.mod +++ b/backend/go.mod @@ -7,7 +7,7 @@ require ( github.com/bogdanfinn/tls-client v1.15.1 github.com/chromedp/cdproto v0.0.0-20260714215040-dc233986426f github.com/chromedp/chromedp v0.16.0 - modernc.org/sqlite v1.34.4 + github.com/jackc/pgx/v5 v5.10.0 ) require ( @@ -18,27 +18,19 @@ require ( github.com/bogdanfinn/utls v1.7.7-barnius // indirect github.com/bogdanfinn/websocket v1.5.5-barnius // indirect github.com/chromedp/sysutil v1.1.0 // indirect - github.com/dustin/go-humanize v1.0.1 // indirect github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 // indirect github.com/gobwas/httphead v0.1.0 // indirect github.com/gobwas/pool v0.2.1 // indirect github.com/gobwas/ws v1.4.0 // indirect - github.com/google/uuid v1.6.0 // indirect - github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/klauspost/compress v1.18.2 // indirect - github.com/mattn/go-isatty v0.0.20 // indirect - github.com/ncruces/go-strftime v0.1.9 // indirect github.com/quic-go/qpack v0.6.0 // indirect - github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect golang.org/x/crypto v0.46.0 // indirect golang.org/x/net v0.48.0 // indirect + golang.org/x/sync v0.19.0 // indirect golang.org/x/sys v0.47.0 // indirect golang.org/x/text v0.32.0 // indirect - modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect - modernc.org/libc v1.55.3 // indirect - modernc.org/mathutil v1.6.0 // indirect - modernc.org/memory v1.8.0 // indirect - modernc.org/strutil v1.2.0 // indirect - modernc.org/token v1.1.0 // indirect ) diff --git a/backend/go.sum b/backend/go.sum index ef779f6..898d9aa 100644 --- a/backend/go.sum +++ b/backend/go.sum @@ -20,10 +20,9 @@ github.com/chromedp/chromedp v0.16.0 h1:rOO4deOm4CbZgBCa8mD9g2rDyIoNs0BkgvNrlbp5 github.com/chromedp/chromedp v0.16.0/go.mod h1:rbuGKFT1vMcFcFqKfPIO1GpX/N+2s8onm2qMxZLbU5U= github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM= github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= -github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68 h1:KZaTBSyshWX3MP5jukJcNSuXDQTO+rNpt0J564dX/eg= github.com/go-json-experiment/json v0.0.0-20260623181947-01eb4420fa68/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg= github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU= @@ -32,28 +31,27 @@ github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og= github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw= github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs= github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc= -github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo= -github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw= -github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= -github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= -github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0= +github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk= github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4= github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo= github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs= -github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= -github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= -github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4= -github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw= github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8= github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII= -github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= -github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc= @@ -64,8 +62,6 @@ go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko= go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o= golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU= golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0= -golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk= -golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc= golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU= golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY= @@ -81,33 +77,7 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU= golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ= -golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ= -modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ= -modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y= -modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s= -modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE= -modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ= -modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw= -modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU= -modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 h1:5D53IMaUuA5InSeMu9eJtlQXS2NxAhyWQvkKEgXZhHI= -modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6/go.mod h1:Qz0X07sNOR1jWYCrJMEnbW/X55x206Q7Vt4mz6/wHp4= -modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U= -modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w= -modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4= -modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo= -modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E= -modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU= -modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4= -modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0= -modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc= -modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss= -modernc.org/sqlite v1.34.4 h1:sjdARozcL5KJBvYQvLlZEmctRgW9xqIZc2ncN7PU0P8= -modernc.org/sqlite v1.34.4/go.mod h1:3QQFCG2SEMtc2nv+Wq4cQCH7Hjcg+p/RMlS1XK+zwbk= -modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA= -modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0= -modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y= -modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM= diff --git a/backend/internal/latest/poller_test.go b/backend/internal/latest/poller_test.go index fdb9985..a9fa3d1 100644 --- a/backend/internal/latest/poller_test.go +++ b/backend/internal/latest/poller_test.go @@ -3,18 +3,21 @@ package latest import ( "context" "errors" - "path/filepath" + "os" "sync" "testing" "time" + "bookmarkmanager/backend/internal/pgtest" "bookmarkmanager/backend/internal/store" ) -// newTestStore opens a fresh SQLite store in a temp dir. +func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } + +// newTestStore opens a store on a Postgres database of this test's own. func newTestStore(t *testing.T) *store.Store { t.Helper() - s, err := store.Open(filepath.Join(t.TempDir(), "test.db")) + s, err := store.Open(pgtest.URL(t)) if err != nil { t.Fatalf("Open: %v", err) } diff --git a/backend/internal/latest/sites.go b/backend/internal/latest/sites.go index 78d14e0..52ebfa1 100644 --- a/backend/internal/latest/sites.go +++ b/backend/internal/latest/sites.go @@ -5,8 +5,6 @@ import ( "regexp" "strconv" "strings" - - "bookmarkmanager/backend/internal/store" ) // latestChapter is the newest chapter a series page advertises. @@ -22,6 +20,12 @@ type latestChapter struct { // before using the slug to scope anything. var asuraSlugRe = regexp.MustCompile(`/comics/([^/?#]+)`) +// asuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura +// appends to every series slug. It rotates on each site redeploy, so it is +// never part of a stable series_id. Must stay in sync with stripBuildHash in +// userscript/manga-bookmark.user.js. +var asuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`) + // demonicChapterRe matches the pre-redirect anchors demonic series pages link // through. Both the raw "&" and the HTML-escaped "&" forms occur. var demonicChapterRe = regexp.MustCompile(`chaptered\.php\?manga=\d+&(?:amp;)?chapter=([0-9.]+)`) @@ -74,9 +78,9 @@ func latestChapterFrom(site, seriesURL, body string) (latestChapter, bool) { } // Stored URLs predating a redeploy may carry a stale build hash; // chapter hrefs in the fetched body carry the current one. Strip to - // the stable ID (same rule as migrateAsuraKeys) and make the hash - // optional in the pattern, so scoping survives rotations. - slug := store.AsuraBuildHash.ReplaceAllString(m[1], "") + // the stable ID and make the hash optional in the pattern, so scoping + // survives rotations. + slug := asuraBuildHash.ReplaceAllString(m[1], "") // Compiled per call rather than cached: this runs once per fetch, which // is at most a few times a minute, and the slug varies per series. re = regexp.MustCompile(`/comics/` + regexp.QuoteMeta(slug) + `(?:-[0-9a-f]{8})?/chapter/([0-9.]+)`) diff --git a/backend/internal/pgtest/pgtest.go b/backend/internal/pgtest/pgtest.go new file mode 100644 index 0000000..b4271a7 --- /dev/null +++ b/backend/internal/pgtest/pgtest.go @@ -0,0 +1,120 @@ +// Package pgtest runs the Postgres the test suite needs: one throwaway +// container per test binary, one fresh database per test. Docker is therefore +// a hard prerequisite for `go test ./...`. +// +// Rolled by hand rather than pulled in as a dependency — it is one `docker +// run`, one `docker port` and a ping loop, against a module list that is +// otherwise stdlib plus what the poller genuinely needs. +package pgtest + +import ( + "database/sql" + "fmt" + "os/exec" + "strconv" + "strings" + "sync/atomic" + "testing" + "time" + + _ "github.com/jackc/pgx/v5/stdlib" +) + +const ( + image = "postgres:17-alpine" + readyLimit = 60 * time.Second +) + +var ( + adminURL string + dbSeq atomic.Int64 +) + +// Main starts the container, runs the package's tests and tears the container +// down. Every test package that touches the store calls it from TestMain: +// +// func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } +func Main(m *testing.M) int { + id, url, err := start() + if err != nil { + fmt.Println("pgtest:", err) + return 1 + } + defer exec.Command("docker", "rm", "-f", id).Run() + + adminURL = url + return m.Run() +} + +// URL creates a database of its own for t and returns a connection URL for it. +// Nothing drops it again: the container goes away wholesale when Main returns. +func URL(t testing.TB) string { + t.Helper() + if adminURL == "" { + t.Fatal("pgtest: no container; this package needs TestMain to call pgtest.Main") + } + // Generated, never derived from the test name, so it needs no quoting and + // cannot collide when tests run in parallel. + name := "test_" + strconv.FormatInt(dbSeq.Add(1), 10) + + admin, err := sql.Open("pgx", adminURL) + if err != nil { + t.Fatalf("pgtest: open admin connection: %v", err) + } + defer admin.Close() + if _, err := admin.Exec(`CREATE DATABASE ` + name); err != nil { + t.Fatalf("pgtest: create database %s: %v", name, err) + } + return strings.Replace(adminURL, "/postgres?", "/"+name+"?", 1) +} + +// start launches the container and waits for it to accept queries, returning +// its id and a connection URL for the default database. +func start() (id, url string, err error) { + out, err := exec.Command("docker", "run", "-d", "--rm", + "-e", "POSTGRES_PASSWORD=pgtest", + "-P", image, + // Durability buys nothing for a database that dies with the test + // binary, and turning it off is most of the container's start-up cost. + "-c", "fsync=off", "-c", "full_page_writes=off", + ).Output() + if err != nil { + return "", "", fmt.Errorf("docker run %s: %w", image, err) + } + id = strings.TrimSpace(string(out)) + + port, err := exec.Command("docker", "port", id, "5432/tcp").Output() + if err != nil { + exec.Command("docker", "rm", "-f", id).Run() + return "", "", fmt.Errorf("docker port: %w", err) + } + // "0.0.0.0:32768" (and possibly a second, IPv6 line); the port is all we want. + first, _, _ := strings.Cut(strings.TrimSpace(string(port)), "\n") + url = fmt.Sprintf("postgres://postgres:pgtest@127.0.0.1:%s/postgres?sslmode=disable", + first[strings.LastIndex(first, ":")+1:]) + + if err := waitReady(url); err != nil { + exec.Command("docker", "rm", "-f", id).Run() + return "", "", err + } + return id, url, nil +} + +func waitReady(url string) error { + db, err := sql.Open("pgx", url) + if err != nil { + return err + } + defer db.Close() + + deadline := time.Now().Add(readyLimit) + for { + if err = db.Ping(); err == nil { + return nil + } + if time.Now().After(deadline) { + return fmt.Errorf("postgres not ready after %s: %w", readyLimit, err) + } + time.Sleep(200 * time.Millisecond) + } +} diff --git a/backend/internal/store/migrations/0001_bookmarks.sql b/backend/internal/store/migrations/0001_bookmarks.sql new file mode 100644 index 0000000..6d817d4 --- /dev/null +++ b/backend/internal/store/migrations/0001_bookmarks.sql @@ -0,0 +1,28 @@ +-- One row per tracked series, keyed ":". +-- +-- Everything is NOT NULL with a default except latest_chapter_num, where NULL +-- is a distinct state: nothing has been captured yet, which is not the same as +-- chapter zero. +-- +-- Timestamps are unix milliseconds as bigint, not timestamptz: the userscripts +-- send Date.now() over the wire and the ordering rule compares them directly. +CREATE TABLE bookmarks ( + key text PRIMARY KEY, + site text NOT NULL, + series_id text NOT NULL, + title text NOT NULL DEFAULT '', + series_url text NOT NULL DEFAULT '', + cover text NOT NULL DEFAULT '', + last_chapter text NOT NULL DEFAULT '', + last_chapter_num double precision NOT NULL DEFAULT 0, + last_chapter_url text NOT NULL DEFAULT '', + favorite boolean NOT NULL DEFAULT false, + latest_chapter text NOT NULL DEFAULT '', + latest_chapter_num double precision, + -- When the server last polled this series, unix ms; 0 means never, and sorts + -- first so a new bookmark is picked up on the next tick with no special case. + latest_checked_at bigint NOT NULL DEFAULT 0, + status text NOT NULL DEFAULT 'reading', + kind text NOT NULL DEFAULT 'manga', + updated_at bigint NOT NULL +); diff --git a/backend/internal/store/store.go b/backend/internal/store/store.go index 903e66a..956009a 100644 --- a/backend/internal/store/store.go +++ b/backend/internal/store/store.go @@ -2,13 +2,17 @@ package store import ( "database/sql" + "embed" "errors" "fmt" + "io/fs" + "path" "regexp" + "slices" "strconv" "strings" - _ "modernc.org/sqlite" + _ "github.com/jackc/pgx/v5/stdlib" ) // Bookmark is one tracked series, keyed ":" across both sites. @@ -114,222 +118,119 @@ const ( StatusFinished = "finished" ) -const schema = ` -CREATE TABLE IF NOT EXISTS bookmarks ( - key TEXT PRIMARY KEY, - site TEXT NOT NULL, - series_id TEXT NOT NULL, - title TEXT, - series_url TEXT, - cover TEXT, - last_chapter TEXT, - last_chapter_num REAL, - last_chapter_url TEXT, - favorite INTEGER NOT NULL DEFAULT 0, - latest_chapter TEXT NOT NULL DEFAULT '', - latest_chapter_num REAL, - latest_checked_at INTEGER NOT NULL DEFAULT 0, - status TEXT NOT NULL DEFAULT 'reading', - kind TEXT NOT NULL DEFAULT 'manga', - updated_at INTEGER NOT NULL -);` - -// The columns above that databases created before them will be missing. -// SQLite has no ADD COLUMN IF NOT EXISTS, so each is added only when absent. -var addedColumns = []struct{ name, ddl string }{ - {"favorite", `ALTER TABLE bookmarks ADD COLUMN favorite INTEGER NOT NULL DEFAULT 0`}, - {"latest_chapter", `ALTER TABLE bookmarks ADD COLUMN latest_chapter TEXT NOT NULL DEFAULT ''`}, - {"latest_chapter_num", `ALTER TABLE bookmarks ADD COLUMN latest_chapter_num REAL`}, - // When the server last looked at this series, unix ms; 0 means never, and - // sorts first so a new bookmark is picked up on the next tick with no - // special case. Deliberately NOT in bookmarkColumns — see MarkLatestChecked. - {"latest_checked_at", `ALTER TABLE bookmarks ADD COLUMN latest_checked_at INTEGER NOT NULL DEFAULT 0`}, - // Lifecycle bucket. The DEFAULT backfills every pre-existing row as - // 'reading', so there is no separate migration step. - {"status", `ALTER TABLE bookmarks ADD COLUMN status TEXT NOT NULL DEFAULT 'reading'`}, - // Library bucket. The DEFAULT backfills every pre-existing row as 'manga', - // which is what every row written before novels existed actually is. - {"kind", `ALTER TABLE bookmarks ADD COLUMN kind TEXT NOT NULL DEFAULT 'manga'`}, -} +//go:embed migrations/*.sql +var migrations embed.FS +// bookmarkColumns is the only value ever concatenated into query text. It is a +// compile-time constant; every request value is bound as a parameter. const bookmarkColumns = `key, site, series_id, title, series_url, cover, last_chapter, last_chapter_num, last_chapter_url, favorite, latest_chapter, latest_chapter_num, updated_at, status, kind` -// Store is the SQLite-backed bookmark store. +// Store is the Postgres-backed bookmark store. type Store struct { db *sql.DB } -// OpenStore opens (or creates) the SQLite database at path and applies the schema. -func Open(path string) (*Store, error) { - // busy_timeout guards against SQLITE_BUSY under the reverse proxy's - // concurrent requests; a single writer connection keeps writes serialized. - dsn := path - if !strings.Contains(dsn, "?") { - dsn += "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)" - } - db, err := sql.Open("sqlite", dsn) +// Open connects to Postgres at url — a libpq connection URL such as +// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — and brings its +// schema up to date. +func Open(url string) (*Store, error) { + db, err := sql.Open("pgx", url) if err != nil { - return nil, fmt.Errorf("open sqlite %q: %w", path, err) + return nil, fmt.Errorf("open postgres: %w", err) } - db.SetMaxOpenConns(1) - if _, err := db.Exec(schema); err != nil { + if err := migrate(db); err != nil { db.Close() - return nil, fmt.Errorf("apply schema: %w", err) - } - if err := migrateColumns(db); err != nil { - db.Close() - return nil, fmt.Errorf("migrate schema: %w", err) - } - if err := migrateAsuraKeys(db); err != nil { - db.Close() - return nil, fmt.Errorf("migrate asura keys: %w", err) + return nil, fmt.Errorf("migrate: %w", err) } return &Store{db: db}, nil } -// migrateColumns brings a pre-existing bookmarks table up to the current -// schema. Safe to run on every start: columns already present are skipped. -func migrateColumns(db *sql.DB) error { - have, err := existingColumns(db, "bookmarks") +// migrate applies every embedded migration this database has not recorded, in +// filename order, each in its own transaction. Files are named +// "_.sql" and are append-only: editing an applied file changes +// nothing, because schema_migrations is how a database remembers what it ran. +// Runs on every start and is a no-op once current. +func migrate(db *sql.DB) error { + if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations ( + version bigint PRIMARY KEY, + applied_at timestamptz NOT NULL DEFAULT now())`); err != nil { + return fmt.Errorf("create version table: %w", err) + } + + names, err := fs.Glob(migrations, "migrations/*.sql") if err != nil { return err } - for _, c := range addedColumns { - if _, ok := have[c.name]; ok { - continue + slices.Sort(names) + + for _, name := range names { + version, err := strconv.ParseInt(strings.SplitN(path.Base(name), "_", 2)[0], 10, 64) + if err != nil { + return fmt.Errorf("migration %q: filename must start with a version number", name) } - if _, err := db.Exec(c.ddl); err != nil { - return fmt.Errorf("add column %q: %w", c.name, err) + body, err := migrations.ReadFile(name) + if err != nil { + return err + } + if err := applyMigration(db, version, string(body)); err != nil { + return fmt.Errorf("migration %q: %w", name, err) } } return nil } -// AsuraBuildHash matches the trailing "-xxxxxxxx" site-wide build ID Asura -// appends to every series slug. It rotates on each site redeploy, so it -// must not be part of series_id. Must stay in sync with stripBuildHash in -// userscript/manga-bookmark.user.js. -var AsuraBuildHash = regexp.MustCompile(`-[0-9a-f]{8}$`) - -// migrateAsuraKeys rewrites asura bookmarks whose series_id still carries -// the build hash to the stable, hashless ID. Rows keyed with a hash are -// orphaned on every Asura redeploy (old-hash URLs 302 to new-hash ones, so -// detection yields a key that never matches). When two hash-generations of -// one series collide, the row with the newest updated_at wins and the rest -// are deleted. Idempotent: hashless IDs never match the regex. -func migrateAsuraKeys(db *sql.DB) error { - rows, err := db.Query(`SELECT key, series_id, updated_at FROM bookmarks WHERE site = 'asura'`) +// applyMigration runs one migration and records its version in the same +// transaction, so an interrupted start leaves neither half behind. +func applyMigration(db *sql.DB, version int64, body string) error { + tx, err := db.Begin() if err != nil { - return fmt.Errorf("list asura rows: %w", err) - } - type row struct { - key, id string - updated int64 - } - var all []row - for rows.Next() { - var r row - if err := rows.Scan(&r.key, &r.id, &r.updated); err != nil { - rows.Close() - return fmt.Errorf("scan asura row: %w", err) - } - all = append(all, r) - } - if err := rows.Close(); err != nil { return err } + defer tx.Rollback() - groups := map[string][]row{} - for _, r := range all { - stripped := AsuraBuildHash.ReplaceAllString(r.id, "") - groups[stripped] = append(groups[stripped], r) + var applied bool + if err := tx.QueryRow( + `SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1)`, + version).Scan(&applied); err != nil { + return err } - for stripped, g := range groups { - winner := 0 - for i := range g { - if g[i].updated > g[winner].updated { - winner = i - } - } - // Losers go first: rewriting the winner to the stripped key while a - // pre-existing hashless row still holds it is a primary-key collision. - for i, r := range g { - if i == winner { - continue - } - if _, err := db.Exec(`DELETE FROM bookmarks WHERE key = ?`, r.key); err != nil { - return fmt.Errorf("drop duplicate %q: %w", r.key, err) - } - } - if r := g[winner]; r.id != stripped { - if _, err := db.Exec( - `UPDATE bookmarks SET key = ?, series_id = ? WHERE key = ?`, - "asura:"+stripped, stripped, r.key); err != nil { - return fmt.Errorf("rewrite key %q: %w", r.key, err) - } - } + if applied { + return nil } - return nil + // No parameters, so this goes over the simple protocol and a migration may + // hold more than one statement. + if _, err := tx.Exec(body); err != nil { + return err + } + if _, err := tx.Exec(`INSERT INTO schema_migrations (version) VALUES ($1)`, version); err != nil { + return err + } + return tx.Commit() } -func existingColumns(db *sql.DB, table string) (map[string]struct{}, error) { - rows, err := db.Query(`SELECT name FROM pragma_table_info(?)`, table) - if err != nil { - return nil, fmt.Errorf("read %s columns: %w", table, err) - } - defer rows.Close() - - out := map[string]struct{}{} - for rows.Next() { - var name string - if err := rows.Scan(&name); err != nil { - return nil, fmt.Errorf("scan column name: %w", err) - } - out[name] = struct{}{} - } - return out, rows.Err() -} - -// scanBookmark reads one row in bookmarkColumns order, translating SQLite's -// integer bool and nullable latest_chapter_num into Go types. -// -// The optional columns are read through Null* types because rows predating -// this code (or written by hand) may hold NULL where the app only ever writes -// zero values. Only latest_chapter_num distinguishes the two: everywhere else -// NULL and the zero value mean the same thing to clients. +// scanBookmark reads one row in bookmarkColumns order. Every column is NOT +// NULL except latest_chapter_num, where NULL means "never captured" — a +// distinct state from chapter zero, and the reason for the pointer. func scanBookmark(scan func(...any) error) (Bookmark, error) { var ( - b Bookmark - title, seriesURL, cover sql.NullString - lastChapter, lastChapterURL, latestChapter sql.NullString - status sql.NullString - lastChapterNum, latestChapterNum sql.NullFloat64 - favorite sql.NullInt64 + b Bookmark + latestChapterNum sql.NullFloat64 ) if err := scan( - &b.Key, &b.Site, &b.SeriesID, &title, &seriesURL, &cover, - &lastChapter, &lastChapterNum, &lastChapterURL, - &favorite, &latestChapter, &latestChapterNum, &b.UpdatedAt, &status, &b.Kind, + &b.Key, &b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover, + &b.LastChapter, &b.LastChapterNum, &b.LastChapterURL, + &b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind, ); err != nil { return Bookmark{}, err } - b.Title = title.String - b.SeriesURL = seriesURL.String - b.Cover = cover.String - b.LastChapter = lastChapter.String - b.LastChapterNum = lastChapterNum.Float64 - b.LastChapterURL = lastChapterURL.String - b.Favorite = favorite.Int64 != 0 - b.LatestChapter = latestChapter.String if latestChapterNum.Valid { b.LatestChapterNum = &latestChapterNum.Float64 } - // A NULL, empty, or unrecognised bucket (e.g. a hand-edited row) would - // leave the row in no list at all, so anything outside the three known - // buckets reads as the default rather than being passed through. - b.Status = status.String + // An unrecognised bucket (a hand-edited row) would leave the row in no list + // at all, so anything outside the three known buckets reads as the default + // rather than being passed through. if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished { b.Status = StatusReading } @@ -365,7 +266,7 @@ func (s *Store) List() ([]Bookmark, error) { // the fields they do not touch. func (s *Store) Get(key string) (Bookmark, bool, error) { b, err := scanBookmark(s.db.QueryRow( - `SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, key).Scan) + `SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = $1`, key).Scan) if errors.Is(err, sql.ErrNoRows) { return Bookmark{}, false, nil } @@ -395,9 +296,10 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) { latestNum = *b.LatestChapterNum } - // IS NOT is SQLite's null-safe comparison. Within DO UPDATE, a bare column - // is the stored row and excluded.* is the incoming one; a brand-new key - // never reaches this clause, so it keeps the fresh timestamp from VALUES. + // IS DISTINCT FROM is Postgres's null-safe comparison, and it is what + // implements the ordering rule. Within DO UPDATE, a bare column is the + // stored row and excluded.* is the incoming one; a brand-new key never + // reaches this clause, so it keeps the fresh timestamp from VALUES. // // The status and kind columns resolve on the VALUES side, not in the // conflict clause: excluded.* is the row *after* these expressions are @@ -408,12 +310,16 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) { // stored", and only a brand-new row falls through to the literal // default. The subquery runs inside this transaction, so it sees the // row this statement is about to conflict with. + // + // The ::text casts are load-bearing: inside COALESCE/NULLIF there is no + // target column to infer the parameter type from, and Postgres rejects the + // statement rather than guessing. if _, err := tx.Exec(` INSERT INTO bookmarks (`+bookmarkColumns+`) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, - COALESCE(NULLIF(?, ''), (SELECT status FROM bookmarks WHERE key = ?), 'reading'), - COALESCE(NULLIF(?, ''), (SELECT kind FROM bookmarks WHERE key = ?), 'manga')) - ON CONFLICT(key) DO UPDATE SET + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, + COALESCE(NULLIF($14::text, ''), (SELECT status FROM bookmarks WHERE key = $1), 'reading'), + COALESCE(NULLIF($15::text, ''), (SELECT kind FROM bookmarks WHERE key = $1), 'manga')) + ON CONFLICT (key) DO UPDATE SET site=excluded.site, series_id=excluded.series_id, title=excluded.title, series_url=excluded.series_url, cover=excluded.cover, last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num, @@ -424,20 +330,19 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) { status=excluded.status, kind=excluded.kind, updated_at=CASE - WHEN bookmarks.last_chapter_num IS NOT excluded.last_chapter_num + WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num THEN excluded.updated_at ELSE bookmarks.updated_at END`, b.Key, b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.LastChapter, b.LastChapterNum, b.LastChapterURL, b.Favorite, b.LatestChapter, latestNum, b.UpdatedAt, - b.Status, b.Key, - b.Kind, b.Key); err != nil { + b.Status, b.Kind); err != nil { return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err) } stored, err := scanBookmark(tx.QueryRow( - `SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = ?`, b.Key).Scan) + `SELECT `+bookmarkColumns+` FROM bookmarks WHERE key = $1`, b.Key).Scan) if err != nil { return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err) } @@ -449,7 +354,7 @@ func (s *Store) Upsert(b Bookmark) (Bookmark, error) { // Delete removes a bookmark by key. Deleting a missing key is not an error. func (s *Store) Delete(key string) error { - if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = ?`, key); err != nil { + if _, err := s.db.Exec(`DELETE FROM bookmarks WHERE key = $1`, key); err != nil { return fmt.Errorf("delete %q: %w", key, err) } return nil @@ -472,11 +377,11 @@ func (s *Store) Delete(key string) error { func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Bookmark, error) { rows, err := s.db.Query(`SELECT `+bookmarkColumns+` FROM bookmarks - WHERE series_url IS NOT NULL AND series_url <> '' - AND status IS NOT 'finished' - AND latest_checked_at <= ? + WHERE series_url <> '' + AND status IS DISTINCT FROM 'finished' + AND latest_checked_at <= $1 ORDER BY latest_checked_at ASC - LIMIT ?`, cutoffMs, limit) + LIMIT $2`, cutoffMs, limit) if err != nil { return nil, fmt.Errorf("query due bookmarks: %w", err) } @@ -505,7 +410,7 @@ func (s *Store) DueForLatestCheck(cutoffMs int64, limit int) ([]Bookmark, error) // long as the user kept reading it. func (s *Store) MarkLatestChecked(key string, ts int64) error { if _, err := s.db.Exec( - `UPDATE bookmarks SET latest_checked_at = ? WHERE key = ?`, ts, key); err != nil { + `UPDATE bookmarks SET latest_checked_at = $1 WHERE key = $2`, ts, key); err != nil { return fmt.Errorf("mark checked %q: %w", key, err) } return nil @@ -518,7 +423,7 @@ func (s *Store) MarkLatestChecked(key string, ts int64) error { func (s *Store) LatestCheckedAt(key string) (int64, error) { var ts int64 if err := s.db.QueryRow( - `SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil { + `SELECT latest_checked_at FROM bookmarks WHERE key = $1`, key).Scan(&ts); err != nil { return 0, fmt.Errorf("latest checked at %q: %w", key, err) } return ts, nil diff --git a/backend/internal/store/store_test.go b/backend/internal/store/store_test.go index 7a29188..c444f02 100644 --- a/backend/internal/store/store_test.go +++ b/backend/internal/store/store_test.go @@ -1,80 +1,54 @@ package store import ( - "database/sql" - "path/filepath" + "os" "testing" "time" + + "bookmarkmanager/backend/internal/pgtest" ) +func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) } + func newTestStore(t *testing.T) *Store { t.Helper() - store, err := Open(filepath.Join(t.TempDir(), "test.db")) + store, err := Open(pgtest.URL(t)) if err != nil { - t.Fatalf("OpenStore: %v", err) + t.Fatalf("Open: %v", err) } t.Cleanup(func() { store.Close() }) return store } -func TestOpenStoreMigratesLegacySchema(t *testing.T) { - dbPath := filepath.Join(t.TempDir(), "legacy.db") - - legacy, err := sql.Open("sqlite", dbPath) +// The migration runner runs on every start, so a second Open against a +// database it already built must be a no-op rather than a duplicate-table +// error, and must leave the rows alone. +func TestOpenIsIdempotent(t *testing.T) { + url := pgtest.URL(t) + first, err := Open(url) if err != nil { - t.Fatalf("open legacy db: %v", err) + t.Fatalf("Open: %v", err) } - if _, err := legacy.Exec(` - CREATE TABLE bookmarks ( - key TEXT PRIMARY KEY, - site TEXT NOT NULL, - series_id TEXT NOT NULL, - title TEXT, - series_url TEXT, - cover TEXT, - last_chapter TEXT, - last_chapter_num REAL, - last_chapter_url TEXT, - updated_at INTEGER NOT NULL - )`); err != nil { - t.Fatalf("create legacy schema: %v", err) - } - if _, err := legacy.Exec(` - INSERT INTO bookmarks (key, site, series_id, title, last_chapter, last_chapter_num, updated_at) - VALUES ('asura:legacy', 'asura', 'legacy', 'Legacy Series', 'Chapter 7', 7, 123)`); err != nil { - t.Fatalf("seed legacy row: %v", err) - } - if err := legacy.Close(); err != nil { - t.Fatalf("close legacy db: %v", err) + if _, err := first.Upsert(Bookmark{ + Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000, + }); err != nil { + t.Fatalf("seed: %v", err) } + first.Close() - store, err := Open(dbPath) + second, err := Open(url) if err != nil { - t.Fatalf("OpenStore on legacy db: %v", err) + t.Fatalf("reopen: %v", err) } - t.Cleanup(func() { store.Close() }) + t.Cleanup(func() { second.Close() }) - list, err := store.List() + list, err := second.List() if err != nil { t.Fatalf("List: %v", err) } - if len(list) != 1 || list[0].Key != "asura:legacy" { - t.Fatalf("legacy row lost: %+v", list) + if len(list) != 1 || list[0].Key != "asura:solo" { + t.Fatalf("rows after reopen = %+v, want only the seeded one", list) } - got := list[0] - if got.Title != "Legacy Series" || got.LastChapterNum != 7 || got.UpdatedAt != 123 { - t.Fatalf("legacy data mangled: %+v", got) - } - if got.Favorite || got.LatestChapter != "" || got.LatestChapterNum != nil { - t.Fatalf("new columns should default empty, got %+v", got) - } - - // Reopening an already-migrated database must be a no-op, not an error. - store2, err := Open(dbPath) - if err != nil { - t.Fatalf("OpenStore is not idempotent: %v", err) - } - store2.Close() } func TestStoreGet(t *testing.T) { @@ -155,7 +129,7 @@ func readLatestCheckedAt(t *testing.T, s *Store, key string) int64 { t.Helper() var ts int64 if err := s.db.QueryRow( - `SELECT latest_checked_at FROM bookmarks WHERE key = ?`, key).Scan(&ts); err != nil { + `SELECT latest_checked_at FROM bookmarks WHERE key = $1`, key).Scan(&ts); err != nil { t.Fatalf("read latest_checked_at %q: %v", key, err) } return ts @@ -264,53 +238,6 @@ func TestUpsertPreservesLatestCheckedAt(t *testing.T) { } } -// migrateColumns must be able to bring a database created before this column up -// to date, not just create it fresh. -func TestMigrateAddsLatestCheckedAt(t *testing.T) { - path := filepath.Join(t.TempDir(), "old.db") - - old, err := sql.Open("sqlite", path) - if err != nil { - t.Fatalf("open: %v", err) - } - // A pre-latest_checked_at table, matching the schema as it shipped before. - if _, err := old.Exec(`CREATE TABLE bookmarks ( - key TEXT PRIMARY KEY, site TEXT NOT NULL, series_id TEXT NOT NULL, - title TEXT, series_url TEXT, cover TEXT, - last_chapter TEXT, last_chapter_num REAL, last_chapter_url TEXT, - favorite INTEGER NOT NULL DEFAULT 0, - latest_chapter TEXT NOT NULL DEFAULT '', latest_chapter_num REAL, - updated_at INTEGER NOT NULL)`); err != nil { - t.Fatalf("create old table: %v", err) - } - if _, err := old.Exec( - `INSERT INTO bookmarks (key, site, series_id, series_url, updated_at) - VALUES ('asura:x', 'asura', 'x', 'https://asurascans.com/comics/x', 5)`); err != nil { - t.Fatalf("seed old row: %v", err) - } - if err := old.Close(); err != nil { - t.Fatalf("close: %v", err) - } - - s, err := Open(path) - if err != nil { - t.Fatalf("OpenStore on pre-existing db: %v", err) - } - t.Cleanup(func() { s.Close() }) - - // The migrated row must default to 0 (never checked) and so be due. - if got := readLatestCheckedAt(t, s, "asura:x"); got != 0 { - t.Fatalf("migrated latest_checked_at = %d, want 0", got) - } - due, err := s.DueForLatestCheck(1000, 10) - if err != nil { - t.Fatalf("DueForLatestCheck: %v", err) - } - if len(due) != 1 { - t.Fatalf("got %d due rows after migration, want 1", len(due)) - } -} - func TestUpsertDefaultsStatusToReading(t *testing.T) { store := newTestStore(t) stored, err := store.Upsert(Bookmark{ @@ -425,41 +352,6 @@ func TestUpsertStatusChangeKeepsUpdatedAt(t *testing.T) { } } -// A database written before the column existed must gain it, with every -// pre-existing row landing in the reading bucket. -func TestMigrationAddsStatusToLegacyDatabase(t *testing.T) { - path := filepath.Join(t.TempDir(), "legacy.db") - db, err := sql.Open("sqlite", path) - if err != nil { - t.Fatalf("open: %v", err) - } - if _, err := db.Exec(` - CREATE TABLE bookmarks ( - key TEXT PRIMARY KEY, site TEXT NOT NULL, series_id TEXT NOT NULL, - title TEXT, series_url TEXT, cover TEXT, - last_chapter TEXT, last_chapter_num REAL, last_chapter_url TEXT, - updated_at INTEGER NOT NULL); - INSERT INTO bookmarks (key, site, series_id, updated_at) - VALUES ('asura:old', 'asura', 'old', 1)`); err != nil { - t.Fatalf("seed legacy: %v", err) - } - db.Close() - - store, err := Open(path) - if err != nil { - t.Fatalf("OpenStore: %v", err) - } - defer store.Close() - - b, ok, err := store.Get("asura:old") - if err != nil || !ok { - t.Fatalf("Get: ok=%v err=%v", ok, err) - } - if b.Status != StatusReading { - t.Fatalf("Status = %q, want %q", b.Status, StatusReading) - } -} - // Archiving is the reason to keep polling — the point is to come back to a // series that has moved on. A finished series has nothing left to publish. func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) { @@ -494,135 +386,6 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) { } } -// Asura slugs used to include the site build hash; rows keyed with it must -// be rewritten to the stable ID on open, merging hash-generations of the -// same series into the newest row. -func TestOpenStoreMigratesAsuraBuildHashKeys(t *testing.T) { - dbPath := filepath.Join(t.TempDir(), "hash.db") - - store, err := Open(dbPath) - if err != nil { - t.Fatalf("open: %v", err) - } - seed := []Bookmark{ - {Key: "asura:swordmasters-youngest-son-f886a8af", Site: "asura", - SeriesID: "swordmasters-youngest-son-f886a8af", Title: "Old gen", - LastChapterNum: 50, UpdatedAt: 100}, - {Key: "asura:swordmasters-youngest-son-059befe1", Site: "asura", - SeriesID: "swordmasters-youngest-son-059befe1", Title: "Re-bookmarked", - LastChapterNum: 60, UpdatedAt: 200}, - {Key: "asura:overgeared-059befe1", Site: "asura", - SeriesID: "overgeared-059befe1", Title: "Single gen", UpdatedAt: 150}, - // Hash-like suffix on another site must be left alone. - {Key: "demonic:x-deadbeef", Site: "demonic", - SeriesID: "x-deadbeef", Title: "Not asura", UpdatedAt: 300}, - } - for _, b := range seed { - if _, err := store.Upsert(b); err != nil { - t.Fatalf("seed %s: %v", b.Key, err) - } - } - if err := store.Close(); err != nil { - t.Fatalf("close: %v", err) - } - - reopened, err := Open(dbPath) - if err != nil { - t.Fatalf("reopen: %v", err) - } - t.Cleanup(func() { reopened.Close() }) - - list, err := reopened.List() - if err != nil { - t.Fatalf("List: %v", err) - } - byKey := map[string]Bookmark{} - for _, b := range list { - byKey[b.Key] = b - } - if len(list) != 3 { - t.Fatalf("want 3 rows after merge, got %d: %+v", len(list), list) - } - merged, ok := byKey["asura:swordmasters-youngest-son"] - if !ok { - t.Fatalf("merged key missing: %+v", byKey) - } - // Newest row wins the merge. - if merged.Title != "Re-bookmarked" || merged.LastChapterNum != 60 || merged.UpdatedAt != 200 { - t.Fatalf("merge kept wrong row: %+v", merged) - } - if merged.SeriesID != "swordmasters-youngest-son" { - t.Fatalf("series_id not stripped: %q", merged.SeriesID) - } - if _, ok := byKey["asura:overgeared-059befe1"]; ok { - t.Fatal("single-generation hashed key not rewritten") - } - if _, ok := byKey["asura:overgeared"]; !ok { - t.Fatal("single-generation row missing under stripped key") - } - if _, ok := byKey["demonic:x-deadbeef"]; !ok { - t.Fatal("non-asura row touched") - } - - // Idempotent: a third open changes nothing. - third, err := Open(dbPath) - if err != nil { - t.Fatalf("third open: %v", err) - } - third.Close() -} - -// A hashed row and a pre-existing hashless row of the same series collide on -// the stripped key. The winner rewrite must happen only after the loser is -// gone, or the UPDATE hits a primary-key collision and OpenStore fails. -func TestOpenStoreMigratesAsuraHashlessCollision(t *testing.T) { - dbPath := filepath.Join(t.TempDir(), "collision.db") - - store, err := Open(dbPath) - if err != nil { - t.Fatalf("open: %v", err) - } - seed := []Bookmark{ - {Key: "asura:overgeared", Site: "asura", SeriesID: "overgeared", - Title: "Hashless", LastChapterNum: 10, UpdatedAt: 100}, - {Key: "asura:overgeared-059befe1", Site: "asura", - SeriesID: "overgeared-059befe1", Title: "Hashed newer", - LastChapterNum: 20, UpdatedAt: 200}, - } - for _, b := range seed { - if _, err := store.Upsert(b); err != nil { - t.Fatalf("seed %s: %v", b.Key, err) - } - } - if err := store.Close(); err != nil { - t.Fatalf("close: %v", err) - } - - reopened, err := Open(dbPath) - if err != nil { - t.Fatalf("reopen: %v", err) - } - t.Cleanup(func() { reopened.Close() }) - - list, err := reopened.List() - if err != nil { - t.Fatalf("List: %v", err) - } - if len(list) != 1 { - t.Fatalf("want 1 row after merge, got %d: %+v", len(list), list) - } - merged := list[0] - if merged.Key != "asura:overgeared" { - t.Fatalf("merged key = %q, want asura:overgeared", merged.Key) - } - if merged.SeriesID != "overgeared" { - t.Fatalf("series_id = %q, want overgeared", merged.SeriesID) - } - if merged.Title != "Hashed newer" || merged.LastChapterNum != 20 || merged.UpdatedAt != 200 { - t.Fatalf("merge kept wrong row: %+v", merged) - } -} - func TestDisplayChapter(t *testing.T) { cases := []struct { name string @@ -713,51 +476,3 @@ func TestUpsertEmptyKindKeepsStoredValue(t *testing.T) { t.Fatalf("LastChapterNum = %v, want 11 — progress in the same request must still land", got.LastChapterNum) } } - -// A database created before this column exists must gain it, backfilled as -// manga, without losing anything. -func TestLegacyDatabaseGainsKindAsManga(t *testing.T) { - dbPath := filepath.Join(t.TempDir(), "legacy.db") - - legacy, err := sql.Open("sqlite", dbPath) - if err != nil { - t.Fatalf("open legacy db: %v", err) - } - if _, err := legacy.Exec(` - CREATE TABLE bookmarks ( - key TEXT PRIMARY KEY, - site TEXT NOT NULL, - series_id TEXT NOT NULL, - title TEXT, - series_url TEXT, - cover TEXT, - last_chapter TEXT, - last_chapter_num REAL, - last_chapter_url TEXT, - updated_at INTEGER NOT NULL - )`); err != nil { - t.Fatalf("create legacy schema: %v", err) - } - if _, err := legacy.Exec(` - INSERT INTO bookmarks (key, site, series_id, title, updated_at) - VALUES ('asura:legacy', 'asura', 'legacy', 'Legacy Series', 123)`); err != nil { - t.Fatalf("seed legacy row: %v", err) - } - if err := legacy.Close(); err != nil { - t.Fatalf("close legacy db: %v", err) - } - - store, err := Open(dbPath) - if err != nil { - t.Fatalf("Open on legacy db: %v", err) - } - t.Cleanup(func() { store.Close() }) - - list, err := store.List() - if err != nil { - t.Fatalf("List: %v", err) - } - if len(list) != 1 || list[0].Kind != KindManga { - t.Fatalf("legacy row should backfill as manga, got %+v", list) - } -} diff --git a/backend/main.go b/backend/main.go index 275f5b8..b27ee15 100644 --- a/backend/main.go +++ b/backend/main.go @@ -24,8 +24,10 @@ import ( type Config struct { Token string AllowedOrigins []string - DBPath string - Port string + // DatabaseURL is the Postgres connection URL; required, no default, + // because a wrong guess would silently start on an empty database. + DatabaseURL string + Port string // WebPassword gates the browser UI. Empty disables the web routes entirely. WebPassword string // UserscriptPath is the file served at /u/{token}/manga-bookmark.user.js. @@ -143,7 +145,7 @@ func loadLatestPoll() LatestPoll { func loadConfig() Config { c := Config{ Token: os.Getenv("API_TOKEN"), - DBPath: envOr("DB_PATH", "/data/bookmarks.db"), + DatabaseURL: os.Getenv("DATABASE_URL"), Port: envOr("PORT", "8080"), WebPassword: os.Getenv("WEB_PASSWORD"), UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"), @@ -215,8 +217,11 @@ func main() { if cfg.Token == "" { log.Fatal("API_TOKEN is required") } + if cfg.DatabaseURL == "" { + log.Fatal("DATABASE_URL is required") + } - s, err := store.Open(cfg.DBPath) + s, err := store.Open(cfg.DatabaseURL) if err != nil { log.Fatalf("open store: %v", err) } @@ -236,7 +241,8 @@ func main() { } go func() { - log.Printf("listening on :%s (db=%s, origins=%v)", cfg.Port, cfg.DBPath, cfg.AllowedOrigins) + // The connection URL carries a password, so it stays out of the log. + log.Printf("listening on :%s (origins=%v)", cfg.Port, cfg.AllowedOrigins) if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("serve: %v", err) } diff --git a/backend/web_test.go b/backend/web_test.go index 0dd662c..4c3f514 100644 --- a/backend/web_test.go +++ b/backend/web_test.go @@ -5,7 +5,6 @@ import ( "net/http" "net/http/httptest" "net/url" - "path/filepath" "strconv" "strings" "testing" @@ -28,11 +27,7 @@ func webConfig() Config { // can seed rows and assert on what the handlers wrote back. func newWebTestServer(t *testing.T, cfg Config) (http.Handler, *store.Store) { t.Helper() - st, err := store.Open(filepath.Join(t.TempDir(), "test.db")) - if err != nil { - t.Fatalf("store.Open: %v", err) - } - t.Cleanup(func() { st.Close() }) + st := newTestStore(t) return newRouter(st, cfg), st } diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 4851023..031e574 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -23,13 +23,18 @@ services: # isn't an IP or "localhost". BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222} depends_on: - - headless-shell - # `networks:` here replaces the base file's list entirely, so both must be - # named: `proxy` for Traefik routing, `browser` (defined in the base file) - # to keep reaching headless-shell without putting it on `proxy` too. + headless-shell: + condition: service_started + postgres: + condition: service_healthy + # `networks:` here replaces the base file's list entirely, so all three must + # be named: `proxy` for Traefik routing, and `browser` / `db` (defined in + # the base file) to keep reaching headless-shell and Postgres without + # putting either on `proxy`. networks: - proxy - browser + - db labels: - "traefik.enable=true" - "traefik.docker.network=${PROXY_NETWORK:-proxy}" diff --git a/docker-compose.yml b/docker-compose.yml index bf3f0fb..7247598 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -16,7 +16,9 @@ services: # API_TOKEN is required — compose refuses to start without it. API_TOKEN: ${API_TOKEN:?set API_TOKEN in .env} ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net} - DB_PATH: /data/bookmarks.db + # The bookmarks database. Host is the compose service name; the password + # comes from .env so it is never committed. + DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable} PORT: "8080" # Gates the browser UI. Unset means the web routes are not served at all. WEB_PASSWORD: ${WEB_PASSWORD:-} @@ -42,9 +44,13 @@ services: # this URL survives container recreation. BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222} depends_on: - - headless-shell + headless-shell: + condition: service_started + # The migration runner is the first thing the binary does, so a Postgres + # that is still initialising means a crash-loop until it is not. + postgres: + condition: service_healthy volumes: - - bookmarks-data:/data # The userscript is served from here, read fresh on every request. Editing # the file in this checkout takes effect on the next Violentmonkey poll — # no rebuild, no restart. `git pull` restores the committed version, which @@ -56,6 +62,26 @@ services: - "127.0.0.1:8080:8080" networks: - browser + - db + + postgres: + image: postgres:17-alpine + restart: unless-stopped + environment: + POSTGRES_DB: bookmarks + POSTGRES_USER: bookmarks + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env} + healthcheck: + test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"] + interval: 5s + timeout: 3s + retries: 10 + volumes: + - postgres-data:/var/lib/postgresql/data + # Deliberately no `ports:` — only bookmark-api, over the `db` network, + # reaches it. Use `docker compose exec postgres psql` for a shell. + networks: + - db headless-shell: image: chromedp/headless-shell:stable @@ -85,7 +111,11 @@ services: ipv4_address: 172.28.0.10 volumes: - bookmarks-data: + postgres-data: + # The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer + # declared here: undeclared means `docker compose down -v` cannot take it + # with the rest, so the old database survives the cutover until someone + # removes it by hand. networks: # Not `internal: true`: headless Chrome still needs outbound access to reach @@ -95,3 +125,7 @@ networks: ipam: config: - subnet: 172.28.0.0/24 + # Postgres needs no egress and nothing outside bookmark-api needs to reach + # it, so this one really can be cut off from the outside world. + db: + internal: true