sulthan
08749df050
Swap modernc.org/sqlite for jackc/pgx/v5 with no observable change: same endpoints, same wire format, same updated_at ordering rule. The schema now comes from numbered SQL embedded in the binary and applied on startup, one transaction each, recorded in schema_migrations. That replaces two pieces of SQLite-era machinery, both deleted rather than ported: the column probing (Postgres has ADD COLUMN IF NOT EXISTS, and there is no legacy database left to probe) and the Asura key rewrite, which has run clean on every start for months now that the userscripts strip build hashes before writing. Its regexp survives as latest.asuraBuildHash, where the poller still needs it to scope chapter links to a series whose slug carries a rotating hash. Types get real: favorite is a boolean, chapter numbers double precision, timestamps stay unix-ms bigint. SQLite's null-safe IS NOT becomes IS DISTINCT FROM, which is what implements the rule that only reading progress reorders a list. Inside COALESCE/NULLIF the status and kind parameters need an explicit ::text -- there is no target column to infer from and Postgres refuses to guess. Tests lose their free t.TempDir() database, so Docker is now a hard prerequisite for `go test ./...`: internal/pgtest starts one postgres:17-alpine per test binary and hands each test a database of its own. Also lands CONTEXT.md and the four ADRs written while scoping #18. BREAKING CHANGE: DB_PATH is retired for DATABASE_URL, which is required and has no default. Compose gains a postgres service on an internal network with its own volume; POSTGRES_PASSWORD joins .env. The old bookmarks-data volume is deliberately left undeclared so `docker compose down -v` cannot take the pre-migration database with it. main is not deployable until #25 and #26 land. Closes #20 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
41 lines
2.4 KiB
Markdown
41 lines
2.4 KiB
Markdown
# Postgres replaces SQLite as the primary datastore
|
||
|
||
Status: accepted
|
||
|
||
The project is moving from a single-reader tracker to a service published to a community,
|
||
so we replaced `modernc.org/sqlite` with Postgres (`jackc/pgx/v5`, still pure Go, so
|
||
`CGO_ENABLED=0` and the distroless image are unaffected). The deciding reason is future
|
||
supportability — managed hosting, a datastore that survives the app outgrowing one box —
|
||
**not** concurrency, which was measured and found to be a non-issue.
|
||
|
||
## Considered options
|
||
|
||
**Stay on SQLite.** Benchmarked against the real store at 1,500 rows (≈50 readers × 30
|
||
series): ~9,700 upserts/sec single-writer, plateauing at ~780/sec under 8–50 concurrent
|
||
writers, with `List()` holding 90–103 calls/sec under continuous write load. Projected
|
||
real load at 50 readers is ~0.02 writes/sec — roughly four and a half orders of magnitude
|
||
of headroom. `SetMaxOpenConns(1)` serialises writes but was shown not to starve reads;
|
||
an apparent read collapse traced to row count and per-row scanning, not lock contention.
|
||
SQLite would have worked. It was rejected for where the project is going, not for what
|
||
it does today.
|
||
|
||
**Postgres.** Chosen. Migrating is cheapest now — 29 rows in one table — and gets
|
||
materially harder once there are live readers and a multi-tenant schema.
|
||
|
||
## Consequences
|
||
|
||
- Every statement in `internal/store` is rewritten: `?` → `$N`, `IS NOT` →
|
||
`IS DISTINCT FROM` (this one is load-bearing; it implements the `updated_at`
|
||
ordering rule), `pragma_table_info` → `information_schema.columns`,
|
||
`INTEGER`/`REAL` → `bigint`/`double precision`, `favorite` int-as-bool → `boolean`.
|
||
- ~75 tests currently get a free isolated database from `t.TempDir()`. They now need a
|
||
live server, which makes Docker a hard prerequisite for `go test ./...`. This is the
|
||
permanent cost of the decision and the main reason it was close.
|
||
- Backups get worse, not better: `VACUUM INTO` produced one self-contained file;
|
||
restoring now means `pg_dump`/`pg_restore`, a role, and a password.
|
||
- A second stateful container joins the VPS alongside the existing headless-shell.
|
||
- **Postgres does not address the real scaling limit.** At batch 14 per 10-minute tick
|
||
the poller checks at most 84 series/hour; 50 readers × 30 series is 1,500 bookmarks,
|
||
an 18-hour sweep against a configured 1-hour cooldown. That ceiling is an outbound
|
||
fetch budget and is fixed by deduplicating polls per Series, not by the datastore.
|