2026-07-30 09:47:09 +07:00
1 changed files with 342 additions and 0 deletions
@@ -0,0 +1,342 @@
+# Redeploy runbook
+
+Shipping new code to a server that is already running. First-time setup (DNS,
+`.env`, Traefik, installing the userscript) is `DEPLOY.md` — this file assumes
+all of that exists and picks up at "there is a running stack and I want it to
+run the new commit."
+
+Whole thing is ~5 minutes, most of it waiting on `docker build`. Order matters:
+**back up before you pull.** A backup taken after a bad migration is a backup of
+the damage.
+
+Paths below assume the checkout is at `/opt/mangabm`; substitute your own. The
+one absolute rule about paths: **backups live in `../mangabm-backups/`**, a
+sibling of the project directory (`/opt/mangabm-backups`), never inside it. It
+sits outside the repo so `git pull`, `git clean -fd` and a bad `rm -rf` inside
+the checkout cannot take the backups with them.
+
+```
+/opt/
+├── mangabm/            <- the checkout (this repo)
+└── mangabm-backups/    <- bookmarks-YYYYmmdd-HHMMSS.db
+```
+
+---
+
+## 0. Preflight
+
+```bash
+cd /opt/mangabm
+
+# Both -f flags, every time. The prod override is not standalone.
+COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
+
+$COMPOSE ps                        # manga-api should be Up
+git status --short                 # expect empty
+git log --oneline -1               # note this hash — it is your rollback target
+df -h /var/lib/docker | tail -1    # a build needs room
+```
+
+If `git status` is dirty, someone edited files on the server. Decide before you
+pull: `git stash` to keep it, `git checkout -- .` to discard. A `git pull` onto a
+dirty tree fails halfway and leaves you in a worse spot than either.
+
+Create the backup directory once, and make sure it is a sibling, not a child:
+
+```bash
+mkdir -p ../mangabm-backups
+BACKUP_DIR="$(cd .. && pwd)/mangabm-backups"   # absolute — Docker needs it
+echo "$BACKUP_DIR"                             # -> /opt/mangabm-backups
+```
+
+---
+
+## 1. Back up the database
+
+The database is a single SQLite file in the named Docker volume, at
+`/data/bookmarks.db` inside the container. Find the volume's real name — Compose
+prefixes it with the project directory:
+
+```bash
+docker volume ls --filter name=bookmarks-data
+# -> local   mangabm_bookmarks-data
+VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
+```
+
+### Preferred: hot backup, no downtime
+
+The store runs in **WAL mode**, so recent writes may still be sitting in
+`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can
+therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and
+writes one consistent file, safe to run against a live database:
+
+```bash
+STAMP=$(date -u +%Y%m%d-%H%M%S)      # UTC, sorts chronologically as text
+docker run --rm \
+  -v "$VOL":/data \
+  -v "$BACKUP_DIR":/backup \
+  alpine sh -c "apk add -q sqlite &&
+    sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\""
+
+ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db
+```
+
+`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the
+files sort in real order and never collide across a DST shift.
+
+Note the source volume is mounted **read-write**, which looks wrong for a backup
+and is not. Opening a WAL database requires creating the `-shm` shared-memory
+file; with `:ro` the command fails with `unable to open database file` and no
+backup is produced. `VACUUM INTO` never writes to the source itself.
+
+Verify it before you trust it. An unreadable backup is worse than none, because
+you will act as though you have one:
+
+```bash
+docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite &&
+  sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' &&
+  sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'"
+# -> ok
+# -> 37
+```
+
+The count should match what the web UI shows. Zero rows on a server you know has
+bookmarks means you backed up the wrong volume.
+
+### Fallback: cold copy (no network for `apk add sqlite`)
+
+Stop the service first, then copy the database **and its sidecars** — the `-wal`
+is not optional, it is where the newest writes are:
+
+```bash
+$COMPOSE stop
+docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c "
+  cp /data/bookmarks.db /backup/bookmarks-$STAMP.db
+  [ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal
+  [ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm
+  ls -1 /backup"
+$COMPOSE start
+```
+
+Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away,
+so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for
+the case where it did not. Restoring this variant means putting whichever files
+you got back together, under their original names.
+
+Read-only is safe here precisely because nothing opens the database: it is a file
+copy, not a SQLite connection.
+
+### Retention
+
+Keep a month, drop the rest — a bookmark database this small compresses the
+decision to "disk is free, but not infinite":
+
+```bash
+ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v
+```
+
+---
+
+## 2. Pull the new code
+
+```bash
+git pull --ff-only
+git log --oneline -3
+```
+
+`--ff-only` so a diverged history fails loudly instead of opening a merge you
+did not plan on the production box.
+
+Check whether `.env` needs anything new. New config lands in `.env.example`, and
+compose fails at start for a missing required var — better to find out now:
+
+```bash
+git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml
+```
+
+If a variable was added there, add it to `.env` before continuing.
+
+---
+
+## 3. Rebuild and restart
+
+```bash
+$COMPOSE up -d --build
+```
+
+**A rebuild is mandatory for any UI change.** The HTML templates, CSS,
+JavaScript and fonts are compiled into the binary by `//go:embed`, so editing
+them on the server — or pulling them — changes nothing until the image is
+rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
+bindmounted read-only and read fresh per request.
+
+```bash
+$COMPOSE ps                        # Up, and recently (re)created
+docker logs manga-api --tail 20    # -> "listening on :8080 ..."
+```
+
+Nothing in the log about the database or the poller failing. The image is tagged
+`mangabm-backend:latest`, so the previous image is still on disk untagged —
+that is what makes the rollback in §6 quick.
+
+---
+
+## 4. Verify the deploy
+
+Same four API checks as `DEPLOY.md` §3, plus the web UI. Set the host names once:
+
+```bash
+API=https://manga-api.violetcrown.my.id
+WEB=https://manga.violetcrown.my.id
+TOKEN=$(grep -E '^API_TOKEN=' .env | cut -d= -f2)
+
+curl -s $API/healthz                                            # -> ok
+curl -s -o /dev/null -w '%{http_code}\n' $API/bookmarks         # -> 401
+curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
+                                                                # -> your data, not []
+curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
+  -H 'Access-Control-Request-Method: PUT' \
+  $API/bookmarks/x | grep -i access-control                     # -> allow-origin echoed
+```
+
+`[]` from the third call is the alarm that matters: the volume is not attached
+and you are looking at an empty database. Stop and check `$COMPOSE config
+--volumes` before touching anything else.
+
+Web UI and its assets:
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' $WEB/                  # -> 200 (login page)
+curl -s -o /dev/null -w '%{http_code}\n' $WEB/static/style.css  # -> 200
+
+# The fonts are self-hosted; this is what tells you they shipped and are typed.
+curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
+  $WEB/static/fonts/instrument-serif-400-latin.woff2
+# -> 200 font/woff2
+
+# The served userscript, whose @version is its mtime — a pull bumps it.
+curl -s $API/u/$TOKEN/manga-bookmark.user.js | grep '@version'
+```
+
+`404` on the font means `static/fonts/` did not make it into the image; the UI
+will still render, in Georgia, which is easy to miss on a phone. `application/
+octet-stream` instead of `font/woff2` means an older binary is running.
+
+Then open `$WEB` in a browser and confirm, in one glance:
+
+- Serif brand and serif row titles — not the system sans fallback.
+- A series with an unread chapter has a **crimson title on an ember underline**;
+  everything else is cool grey. That single detail exercises the whole design
+  path (template class, CSS, and the poller's `latest_chapter`).
+- Tapping the pencil opens the chapter form; Save closes it and the row keeps
+  its place in the list.
+
+---
+
+## 5. Smoke-test the full loop
+
+The API answering is not the same as the product working. Do this on the phone,
+against the real sites — it is the only check that covers the userscript, CORS
+and the shared store together:
+
+1. Open a series on **asurascans.com**, open the panel, **+ Bookmark this**.
+2. On the server: `curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks`
+   → the series is in the JSON.
+3. Open a chapter of it, reopen the panel → last-read shows that chapter, and
+   opening an *older* chapter does not move it backwards.
+4. Open **demonicscans.org**, open the panel → the Asura bookmark is listed
+   there too. Different origin, one store — this is the whole point of the
+   backend, and the check that fails first when CORS or TLS regressed.
+5. Open `$WEB` → the same series appears, with the same chapter.
+6. Turn airplane mode on, tap ★ on a row, turn it off, reopen the panel → the
+   star stuck. That exercises the retry queue.
+
+If 1–5 pass, the deploy is good.
+
+---
+
+## 6. Rollback
+
+Two independent things can be wrong, so undo only what broke.
+
+**Bad code, database fine** — go back to the previous commit and rebuild:
+
+```bash
+git log --oneline -5
+git checkout <previous-hash>
+$COMPOSE up -d --build
+```
+
+**Database damaged** — restore the backup from §1. Stop first: the running
+process holds the WAL, and dropping a file under a live SQLite connection
+corrupts what you were trying to save.
+
+```bash
+$COMPOSE stop
+
+docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c '
+  rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm &&
+  cp /backup/bookmarks-<STAMP>.db /data/bookmarks.db &&
+  chown 65532:65532 /data/bookmarks.db &&
+  ls -l /data'
+
+$COMPOSE start
+docker logs manga-api --tail 20
+curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
+```
+
+Two steps here are easy to skip and both bite:
+
+- **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database
+  mixes two different histories; SQLite will either refuse to open it or quietly
+  reapply writes you meant to discard.
+- **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as
+  that uid, while the helper container above writes as root. A root-owned
+  database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then
+  every write fails. That is the worst possible failure mode — it looks restored.
+
+---
+
+## 7. The whole thing, as one block
+
+For a routine redeploy where nothing needs deciding:
+
+```bash
+cd /opt/mangabm
+COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
+BACKUP_DIR="$(cd .. && pwd)/mangabm-backups"; mkdir -p "$BACKUP_DIR"
+VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
+STAMP=$(date -u +%Y%m%d-%H%M%S)
+
+docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \
+  "apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" &&
+   sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" &&
+git pull --ff-only &&
+$COMPOSE up -d --build &&
+sleep 5 &&
+curl -sf https://manga-api.violetcrown.my.id/healthz && echo " deploy ok"
+```
+
+The `&&` chain is deliberate: if the backup or its integrity check fails,
+nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell
+command can tell you the panel works on the phone.
+
+---
+
+## Troubleshooting
+
+| Symptom | Cause / fix |
+|---|---|
+| `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. |
+| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
+| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
+| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
+| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
+| `compose` errors about `MANGA_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
+| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
+| `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. |
+| Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). |
+| Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). |
+
+Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
+UI conventions: `docs/design-system.md`.