docs: correct cutover and redeploy runbooks against the real deployment (#26)

Dry-running CUTOVER.md against production surfaced four things that would
have failed mid-cutover:

- The volume is named after the compose project, which is the lowercased
  directory name (mangabookmark), not the repo name. Both runbooks hardcoded
  bookmarkmanager_bookmarks-data. Derive it from docker instead.
- §1 asserted a clean stop leaves no -wal. compose stop SIGKILLs after 10s,
  and a surviving -wal holds writes bookmarks.db alone does not, so the
  export would silently lose them. Check rather than assume.
- jq is not installed on the server; §5's read-path check now uses python3,
  which the runbook already requires.
- REDEPLOY §1 still listed the pre-split table set, omitting readers and
  sessions from both the \dt output and the pg_restore contents.

Also records the git-pull failure the redeploy hits on a checkout whose
remote is the HTTPS clone URL.
This commit is contained in:
2026-08-08 15:46:05 +07:00
parent 2cc1e69f5d
commit 01de8903b4
2 changed files with 36 additions and 17 deletions
+26 -12
View File
@@ -1,7 +1,7 @@
# SQLite → Postgres cutover runbook # SQLite → Postgres cutover runbook
One-way, one-time. Moves the owner's reading history out of the retired SQLite One-way, one-time. Moves the owner's reading history out of the retired SQLite
volume (`bookmarkmanager_bookmarks-data`, holding `/data/bookmarks.db`) and into volume (`<compose project>_bookmarks-data`, holding `/data/bookmarks.db`) and into
the Postgres schema the migration runner builds. There is no dual-write period: the Postgres schema the migration runner builds. There is no dual-write period:
the old database is read once, at cutover, from a **fresh export** — anything the old database is read once, at cutover, from a **fresh export** — anything
written to SQLite after the export is lost, so the old API must already be down. written to SQLite after the export is lost, so the old API must already be down.
@@ -32,10 +32,10 @@ is dropped) — writing it from the spec below costs less than maintaining it
would. would.
Beyond `DEPLOY.md`'s prerequisites (Docker and Compose), this runbook needs Beyond `DEPLOY.md`'s prerequisites (Docker and Compose), this runbook needs
`python3` on the machine running §3 — its stdlib `sqlite3` module is the whole `python3` — its stdlib `sqlite3` module is the whole SQLite dependency, and it
SQLite dependency — and `jq` for the one read-path check in §5. Neither has to stands in for `jq` in §5, which is not installed on the server. It does not have
be the server: §3 only reads the snapshot copy, so it can run on a laptop and to run on the server: §3 only reads the snapshot copy, so it can run on a laptop
the resulting `import.sql` be copied over. and the resulting `import.sql` be copied over.
--- ---
@@ -45,21 +45,34 @@ the resulting `import.sql` be copied over.
already stale. already stale.
```bash ```bash
cd /opt/bookmarkmanager cd ~/mangaBookmark # wherever the checkout lives
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml" COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR" BACKUP_DIR="$(cd .. && pwd)/$(basename "$PWD")-backups"; mkdir -p "$BACKUP_DIR"
STAMP=$(date -u +%Y%m%d-%H%M%S) STAMP=$(date -u +%Y%m%d-%H%M%S)
# The volume is <compose project>_bookmarks-data, and the project name defaults
# to the lowercased *directory* name, not the repo name — here that makes it
# mangabookmark_bookmarks-data. Ask Docker instead of typing it out.
VOL=$(docker volume ls -q --filter name=_bookmarks-data); echo "$VOL"
$COMPOSE stop bookmark-api $COMPOSE stop bookmark-api
# Copy the file straight out of the retired volume. Nothing is writing to it, # A clean SIGTERM closes the store, which checkpoints and unlinks the -wal, so
# so a plain copy is consistent — no -wal to worry about after a clean stop. # bookmarks.db alone is then the whole database. But `compose stop` SIGKILLs
docker run --rm -v bookmarkmanager_bookmarks-data:/from:ro -v "$BACKUP_DIR":/to \ # after 10s, and a surviving -wal holds writes the main file does not — assert
# it is gone rather than assuming the shutdown was clean.
docker run --rm -v "$VOL":/d:ro alpine ls -l /d # -> bookmarks.db, alone
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to \
alpine cp /from/bookmarks.db "/to/bookmarks-$STAMP.db" alpine cp /from/bookmarks.db "/to/bookmarks-$STAMP.db"
ls -lh "$BACKUP_DIR/bookmarks-$STAMP.db" ls -lh "$BACKUP_DIR/bookmarks-$STAMP.db"
``` ```
If `-wal` and `-shm` are still there, the container was killed mid-write. Copy
all three under the same basename and let SQLite replay the log when §3 opens
it — copying only `bookmarks.db` silently drops whatever the log still holds.
Work on a **copy** of that file for the rest of this runbook. The export is the Work on a **copy** of that file for the rest of this runbook. The export is the
last line of retreat; nothing below should be able to write to it. last line of retreat; nothing below should be able to write to it.
@@ -256,7 +269,8 @@ would catch a correct import behind a broken join:
```bash ```bash
API=https://bookmark-api.violetcrown.my.id API=https://bookmark-api.violetcrown.my.id
TOKEN=$(grep -E '^API_TOKEN=' .env | cut -d= -f2) # grace-window credential TOKEN=$(grep -E '^API_TOKEN=' .env | cut -d= -f2) # grace-window credential
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | jq 'length' # -> 29 curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks |
python3 -c 'import json,sys; print(len(json.load(sys.stdin)))' # -> 29
``` ```
--- ---
@@ -266,7 +280,7 @@ curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | jq 'length' # -> 29
- **Keep the old SQLite volume for a month.** It is already undeclared in - **Keep the old SQLite volume for a month.** It is already undeclared in
compose, so `docker compose down -v` cannot take it. Remove it by hand once compose, so `docker compose down -v` cannot take it. Remove it by hand once
the Postgres data has been trusted for a while: the Postgres data has been trusted for a while:
`docker volume rm bookmarkmanager_bookmarks-data` (see `REDEPLOY.md` §1). `docker volume rm "$VOL"` (see `REDEPLOY.md` §1).
- **Delete the generator and the working copies:** `rm -rf /tmp/cutover`. The - **Delete the generator and the working copies:** `rm -rf /tmp/cutover`. The
timestamped export in `$BACKUP_DIR` is the copy that is kept. timestamped export in `$BACKUP_DIR` is the copy that is kept.
- **Take the first Postgres dump immediately** — `REDEPLOY.md` §1. Until that - **Take the first Postgres dump immediately** — `REDEPLOY.md` §1. Until that
+10 -5
View File
@@ -59,7 +59,7 @@ network can reach it — so every command below goes in through the container:
```bash ```bash
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt' $COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
# -> bookmarks, schema_migrations, series # -> bookmarks, readers, schema_migrations, series, sessions
``` ```
Inside the container that connects over the local socket as the `bookmarks` Inside the container that connects over the local socket as the `bookmarks`
@@ -99,8 +99,10 @@ you will act as though you have one:
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \ docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA' pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks # -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
# -> 1235; 0 0 TABLE DATA public schema_migrations bookmarks # -> 1235; 0 0 TABLE DATA public readers bookmarks
# -> 1236; 0 0 TABLE DATA public series series # -> 1236; 0 0 TABLE DATA public schema_migrations bookmarks
# -> 1237; 0 0 TABLE DATA public series bookmarks
# -> 1238; 0 0 TABLE DATA public sessions bookmarks
# 2. Sanity-check the live row count you just captured. # 2. Sanity-check the live row count you just captured.
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \ $COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
@@ -161,10 +163,12 @@ ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v
declared in `docker-compose.yml` any more, which is what keeps `docker compose declared in `docker-compose.yml` any more, which is what keeps `docker compose
down -v` from taking it with the rest of the stack. It is not the live database down -v` from taking it with the rest of the stack. It is not the live database
and nothing reads it — the one-way move out of it is `CUTOVER.md`. Once the and nothing reads it — the one-way move out of it is `CUTOVER.md`. Once the
Postgres data has been trusted for a while, remove it by hand — nothing else will: Postgres data has been trusted for a while, remove it by hand — nothing else will.
Its full name is `<compose project>_bookmarks-data`, and the project name is the
lowercased directory name of the checkout, so ask Docker rather than typing it:
```bash ```bash
docker volume rm bookmarkmanager_bookmarks-data docker volume rm "$(docker volume ls -q --filter name=_bookmarks-data)"
``` ```
--- ---
@@ -395,6 +399,7 @@ panel works on the phone.
| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. | | `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. |
| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. | | `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. |
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). | | Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
| `git pull`: `could not read Username for 'https://…'` | The checkout's remote is the HTTPS clone URL and the server has no credential helper, so the pull prompts into a closed stdin. Switch it to SSH once — `git remote set-url origin ssh://git@gitea.violetcrown.my.id:2222/sulthan/mangaBookmark.git`. Gitea's SSH listens on **2222**, not 22; port 22 is the host's own sshd and answers `Permission denied (publickey)` no matter which key is registered. |
Full first-time setup: `DEPLOY.md`. The one-off SQLite→Postgres move: Full first-time setup: `DEPLOY.md`. The one-off SQLite→Postgres move:
`CUTOVER.md`. Config reference and endpoints: `README.md`. `CUTOVER.md`. Config reference and endpoints: `README.md`.