mangaBookmark/CUTOVER.md

# SQLite → Postgres cutover runbook

One-way, one-time. Moves the owner's reading history out of the retired SQLite
volume (`<compose project>_bookmarks-data`, holding `/data/bookmarks.db`) and into
the Postgres schema the migration runner builds. There is no dual-write period:
the old database is read once, at cutover, from a **fresh export** — anything
written to SQLite after the export is lost, so the old API must already be down.

Routine deploys are `REDEPLOY.md`; first-time setup is `DEPLOY.md`. This file is
run once and then only ever read for reference.

Proven end to end on 2026-08-08 against a copy of `bookmarks-20260807-213515.db`
into a scratch Postgres: 29 Bookmarks (18 reading, 11 archived, 7 favourites),
29 Series, all owned by the seeded Reader, and every field of every row matching
the source exactly. Production was not touched.

---

## 0. The generator is throwaway

It is written at cutover, run once, and deleted. It is deliberately **not** in
this repository and never will be:

- Its output is the owner's personal reading history. That does not enter
  version control.
- It reads SQLite. The backend module dropped `modernc.org/sqlite` (ADR-0001);
  a committed generator would drag the dependency back in through the side door.

So §3 specifies the transformation rather than shipping a script. It is a
twenty-line program against a sixteen-column table (fifteen after `key`, which
is dropped) — writing it from the spec below costs less than maintaining it
would.

Beyond `DEPLOY.md`'s prerequisites (Docker and Compose), this runbook needs
`python3`: its stdlib `sqlite3` module is the whole SQLite dependency, and §5's
read-path check uses it in place of `jq`, which the server does not have. It
does not have to run on the server — §3 only reads the snapshot copy, so it can
run on a laptop and the resulting `import.sql` be copied over.

---

## 1. Stop the old API and take a fresh export

**Order matters.** Export after the API stops, or you migrate a snapshot that is
already stale.

```bash
cd ~/mangaBookmark                 # wherever the checkout lives
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/$(basename "$PWD")-backups"; mkdir -p "$BACKUP_DIR"
STAMP=$(date -u +%Y%m%d-%H%M%S)

# The volume is <compose project>_bookmarks-data, and the project name defaults
# to the lowercased *directory* name, not the repo name — on this host the
# checkout is ~/mangaBookmark, so the volume is mangabookmark_bookmarks-data.
# Derive it exactly rather than with a `--filter name=` substring match, which
# would return every volume whose name merely contains the string.
VOL="$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_bookmarks-data"
docker volume inspect "$VOL" >/dev/null && echo "$VOL"

$COMPOSE stop bookmark-api

# A clean SIGTERM closes the store, which checkpoints and unlinks the -wal, so
# bookmarks.db alone is then the whole database. But `compose stop` SIGKILLs
# after 10s, and a surviving -wal holds writes the main file does not — assert
# it is gone rather than assuming the shutdown was clean.
docker run --rm -v "$VOL":/d:ro alpine ls -l /d      # -> bookmarks.db, alone

docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to \
  alpine cp /from/bookmarks.db "/to/bookmarks-$STAMP.db"

ls -lh "$BACKUP_DIR/bookmarks-$STAMP.db"
```

If `-wal` and `-shm` are still there, the container was killed mid-write. Copy
all three under the same basename and let SQLite replay the log when §3 opens
it — copying only `bookmarks.db` silently drops whatever the log still holds.

Work on a **copy** of that file for the rest of this runbook. The export is the
last line of retreat; nothing below should be able to write to it.

```bash
mkdir -p /tmp/cutover && cp "$BACKUP_DIR/bookmarks-$STAMP.db" /tmp/cutover/snapshot.db
chmod 444 /tmp/cutover/snapshot.db
```

---

## 2. Bring up Postgres with the schema and the owner Reader

The new stack builds its own schema and seeds exactly one Reader from
`OWNER_DISCORD_ID` — do not hand-write either. Pull the Postgres-era commit
first: on a server that has only ever run the SQLite build, `--build` without a
pull silently rebuilds the old image and the checks below fail with
"relation readers does not exist".

```bash
git pull --ff-only
git log --oneline -1

# .env needs the new required vars (DATABASE_URL is built from
# POSTGRES_PASSWORD; TOKEN_KEY, OWNER_DISCORD_ID and the DISCORD_* set are
# required). Compose fails at start for a missing one.
git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml

$COMPOSE up -d --build
docker logs bookmark-api --tail 20        # -> "listening on :8080"

$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
# -> bookmarks, readers, schema_migrations, series, sessions
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
  -c 'select id, discord_id from readers'
# -> exactly one row, and discord_id is the owner's
```

Two rows in `readers`, or zero, means `OWNER_DISCORD_ID` is wrong or the seed
failed. Stop here — the import attaches history to "the oldest reader row", and
that is only unambiguous while there is one.

`bookmarks` and `series` are empty at this point. That is what makes the import
a plain sequence of `INSERT`s with no conflict handling.

---

## 3. Generate the import SQL

Read `/tmp/cutover/snapshot.db` and emit plain SQL on stdout. The old table is
flat and its columns map one-for-one onto the split schema — no transformation
beyond the split itself:

| SQLite `bookmarks` column | lands in | notes |
|---|---|---|
| `site`, `series_id` | both tables | the Series key; the wire `key` column is dropped, it is re-derived as `site:series_id` on read |
| `title`, `series_url`, `cover`, `kind` | `series` | shared facts (ADR-0003) |
| `latest_chapter`, `latest_chapter_num`, `latest_checked_at` | `series` | `latest_chapter_num` is nullable on **both** sides and `NULL` is meaningful — never coerce it to `0` |
| `last_chapter`, `last_chapter_num`, `last_chapter_url` | `bookmarks` | Progress |
| `favorite`, `status`, `updated_at` | `bookmarks` | `favorite` is `0`/`1` in SQLite and a real `boolean` in Postgres — emit `true`/`false` |
| — | `bookmarks.reader_id` | the seeded owner |

**`latest_chapter_num` is the only column where `NULL` survives.** The SQLite
table declares `title`, `series_url`, `cover`, `last_chapter`,
`last_chapter_url` as bare `TEXT` and `last_chapter_num` as bare `REAL` — all
six nullable — while their Postgres targets are `NOT NULL DEFAULT ''` /
`NOT NULL DEFAULT 0`. One `NULL` in any of them aborts the whole import on a
not-null violation. Coalesce them in the `SELECT` (`ifnull(title,'')`,
`ifnull(last_chapter_num,0)`, …) rather than discovering it at §5. The
2026-08-07 export happened to have none; a fresh export is not promised the
same.

Rules the generator must follow:

- **Series first, Bookmarks second.** `bookmarks` has a foreign key onto
  `series (site, series_id)`; the reverse order fails on the first row.
- **`SELECT DISTINCT` the Series.** The old key's uniqueness already makes
  `(site, series_id)` unique, so this is belt and braces — but if it ever
  collapses two rows, the count check in §5 catches it.
- **Never hardcode the reader id.** Emit
  `INSERT INTO bookmarks (reader_id, …) SELECT id, … FROM owner`, where `owner`
  is a temp table built once at the top:
  `CREATE TEMP TABLE owner ON COMMIT DROP AS SELECT id FROM readers ORDER BY id LIMIT 1;`
  A literal id is a number nobody verifies; this one cannot be wrong.
- **Wrap the whole file in `BEGIN; … COMMIT;`, temp table included.** Postgres
  has transactional DDL and DML: a failure half way leaves an empty database
  rather than half a library. The ordering is load-bearing —
  `ON COMMIT DROP` outside the transaction means the temp table drops itself
  the instant it is created (psql autocommits) and every
  `SELECT … FROM owner` then fails.
- **Quote strings by doubling `'`.** Titles contain apostrophes and the URLs
  contain `%5C%27` escapes. Emit standard SQL literals only — no `E''` strings,
  no backslash escaping (`standard_conforming_strings` is on, so a backslash is
  a literal backslash and the URLs survive verbatim).

```bash
python3 gen_import.py /tmp/cutover/snapshot.db > /tmp/cutover/import.sql
wc -l /tmp/cutover/import.sql       # -> 2 header + 29 series + 29 bookmarks + framing
```

---

## 4. Review it by eye

29 rows is small enough to actually read, and this is the last point at which a
mistake is free:

```bash
less /tmp/cutover/import.sql
grep -c '^INSERT INTO series'    /tmp/cutover/import.sql   # -> 29
grep -c '^INSERT INTO bookmarks' /tmp/cutover/import.sql   # -> 29
```

Look for: a title whose apostrophe is not doubled, a `favorite` that is still
`0`/`1`, a `latest_chapter_num` that turned into `0`, and any `reader_id`
written as a bare number.

---

## 5. Apply it

```bash
docker cp /tmp/cutover/import.sql "$($COMPOSE ps -q postgres)":/tmp/import.sql
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -v ON_ERROR_STOP=1 \
  -f /tmp/import.sql
```

`ON_ERROR_STOP=1` is not optional: without it `psql` reports the error, keeps
going, and exits `0` on a half-imported database.

Then the checklist. Every number here is asserted, not eyeballed:

```bash
OWNER=$(grep -E '^OWNER_DISCORD_ID=' .env | cut -d= -f2)

$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -x -c "
SELECT (SELECT count(*) FROM bookmarks)                        AS bookmarks_total,
       (SELECT count(*) FROM bookmarks WHERE status='reading') AS reading,
       (SELECT count(*) FROM bookmarks WHERE status='archived')AS archived,
       (SELECT count(*) FROM series)                           AS series_total,
       (SELECT count(*) FROM readers)                          AS readers_total,
       (SELECT count(*) FROM bookmarks
          WHERE reader_id <> (SELECT id FROM readers WHERE discord_id='$OWNER'))
                                                               AS not_owned_by_owner;"
```

`not_owned_by_owner` resolves the Reader by **Discord id**, not by
`ORDER BY id LIMIT 1`. The second form is the expression §3 tells the generator
to import with, so comparing against it is true by construction and could never
fail; resolving by Discord id is an independent check that the rows landed on
the identity the owner will actually log in as. If that subquery returns NULL
the whole count comes back `0` for the wrong reason — hence `readers_total`
beside it.

Expected, for the 2026-08-07 export: `29`, `18`, `11`, `29`, `1`, `0`. Against a
different export, the invariants rather than the literals are what hold:

- `bookmarks_total` equals the SQLite row count.
- `reading + archived` equals `bookmarks_total` (nothing was `finished`).
- `series_total` equals `SELECT count(*) FROM (SELECT DISTINCT site, series_id FROM bookmarks)`
  in the source.
- `readers_total` is `1` and `not_owned_by_owner` is `0`.

Then spot-check the values themselves against the source — read position,
favourite flag and latest chapter. Take the sample from each bucket explicitly:
`ORDER BY updated_at DESC LIMIT 5` alone returns the most recently *progressed*
rows, which are the ones least likely to be archived.

```bash
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c "
SELECT s.title, b.last_chapter, b.last_chapter_num, b.favorite,
       s.latest_chapter, b.status
FROM bookmarks b JOIN series s USING (site, series_id)
WHERE b.status='reading'  ORDER BY b.updated_at DESC LIMIT 3;"

$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c "
SELECT s.title, b.last_chapter, b.last_chapter_num, b.favorite,
       s.latest_chapter, b.status
FROM bookmarks b JOIN series s USING (site, series_id)
WHERE b.status='archived' ORDER BY b.updated_at DESC LIMIT 2;"

$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c "
SELECT s.title, b.last_chapter, b.last_chapter_num, b.favorite,
       s.latest_chapter, b.status
FROM bookmarks b JOIN series s USING (site, series_id)
WHERE b.favorite ORDER BY b.updated_at DESC LIMIT 2;"
```

Compare each against the same row in the snapshot — the generator's own source
is the reference, so read it back with the same `python3` you used in §3.

Finally, prove the **read path**, not just the tables — this is the check that
would catch a correct import behind a broken join:

```bash
API=https://bookmark-api.violetcrown.my.id
# Your own Reader credential: sign in to the web UI and take it from the
# Userscripts panel's install link, or read the API_TOKEN constant out of an
# already-installed script. There is no credential in .env to grep.
TOKEN=<your Reader credential>
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks |
  python3 -c 'import json,sys; print(len(json.load(sys.stdin)))'   # -> 29
```

---

## 6. Afterwards

- **Keep the old SQLite volume for a month.** It is already undeclared in
  compose, so `docker compose down -v` cannot take it. Remove it by hand once
  the Postgres data has been trusted for a while. That happens in a shell where
  `$VOL` from §1 is long gone, so re-derive it:
  `docker volume rm "$(basename ~/mangaBookmark | tr '[:upper:]' '[:lower:]')_bookmarks-data"`
  (see `REDEPLOY.md` §1).
- **Delete the generator and the working copies:** `rm -rf /tmp/cutover`. The
  timestamped export in `$BACKUP_DIR` is the copy that is kept.
- **Take the first Postgres dump immediately** — `REDEPLOY.md` §1. Until that
  exists, the only backup of the migrated data is the SQLite file it came from.

If the import is wrong, there is nothing to unpick: drop the rows and start
again from §3 — `TRUNCATE bookmarks, series;` leaves the seeded Reader and the
schema in place.