sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Add a redeploy runbook

Browse Source 
DEPLOY.md covers standing the stack up from nothing and ends with a two-line
"Updating" section, which is the operation actually performed every time and
the one where the irreversible mistakes live. Redeploying has a required
order — back up before pulling, because a backup taken after a bad deploy is
a backup of the damage — and three traps that are invisible until they cost
data:

- The store runs in WAL mode, so copying bookmarks.db alone while the
  container is up can silently drop the newest bookmarks. VACUUM INTO folds
  the WAL in; the cold-copy fallback has to take the sidecar files.
- That backup needs the source volume mounted read-write, which looks wrong.
  Opening a WAL database creates the -shm file, so :ro fails outright.
- A restored file lands root-owned while the container runs as uid 65532.
  Reads succeed and writes do not, so the restore looks like it worked.

Backups go to ../mangabm-backups/, a sibling of the checkout rather than a
directory inside it, so no git operation or careless rm in the project dir
can take the backups along with it. Names carry a UTC timestamp so they sort
chronologically as plain text and cannot collide across a DST shift.

Also documents that a rebuild is mandatory for any UI change now that the
templates, CSS and fonts are //go:embed-ed, and ends at the phone smoke test:
no curl can tell you the panel works on the device.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Sulthan Zaki
2026-07-30 09:42:27 +07:00
parent a547cc9769
commit b20ecf1928
1 changed files with 342 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REDEPLOY.md
+342
View File
@@ -0,0 +1,342 @@
# Redeploy runbook
Shipping new code to a server that is already running. First-time setup (DNS,
`.env`, Traefik, installing the userscript) is `DEPLOY.md` — this file assumes
all of that exists and picks up at "there is a running stack and I want it to
run the new commit."
Whole thing is ~5 minutes, most of it waiting on `docker build`. Order matters:
**back up before you pull.** A backup taken after a bad migration is a backup of
the damage.
Paths below assume the checkout is at `/opt/mangabm`; substitute your own. The
one absolute rule about paths: **backups live in `../mangabm-backups/`**, a
sibling of the project directory (`/opt/mangabm-backups`), never inside it. It
sits outside the repo so `git pull`, `git clean -fd` and a bad `rm -rf` inside
the checkout cannot take the backups with them.
```
/opt/
├── mangabm/ <- the checkout (this repo)
└── mangabm-backups/ <- bookmarks-YYYYmmdd-HHMMSS.db
```
---
## 0. Preflight
```bash
cd /opt/mangabm
# Both -f flags, every time. The prod override is not standalone.
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
$COMPOSE ps # manga-api should be Up
git status --short # expect empty
git log --oneline -1 # note this hash — it is your rollback target
df -h /var/lib/docker | tail -1 # a build needs room
```
If `git status` is dirty, someone edited files on the server. Decide before you
pull: `git stash` to keep it, `git checkout -- .` to discard. A `git pull` onto a
dirty tree fails halfway and leaves you in a worse spot than either.
Create the backup directory once, and make sure it is a sibling, not a child:
```bash
mkdir -p ../mangabm-backups
BACKUP_DIR="$(cd .. && pwd)/mangabm-backups" # absolute — Docker needs it
echo "$BACKUP_DIR" # -> /opt/mangabm-backups
```
---
## 1. Back up the database
The database is a single SQLite file in the named Docker volume, at
`/data/bookmarks.db` inside the container. Find the volume's real name — Compose
prefixes it with the project directory:
```bash
docker volume ls --filter name=bookmarks-data
# -> local mangabm_bookmarks-data
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
```
### Preferred: hot backup, no downtime
The store runs in **WAL mode**, so recent writes may still be sitting in
`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can
therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and
writes one consistent file, safe to run against a live database:
```bash
STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text
docker run --rm \
-v "$VOL":/data \
-v "$BACKUP_DIR":/backup \
alpine sh -c "apk add -q sqlite &&
sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\""
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db
```
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the
files sort in real order and never collide across a DST shift.
Note the source volume is mounted **read-write**, which looks wrong for a backup
and is not. Opening a WAL database requires creating the `-shm` shared-memory
file; with `:ro` the command fails with `unable to open database file` and no
backup is produced. `VACUUM INTO` never writes to the source itself.
Verify it before you trust it. An unreadable backup is worse than none, because
you will act as though you have one:
```bash
docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite &&
sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' &&
sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'"
# -> ok
# -> 37
```
The count should match what the web UI shows. Zero rows on a server you know has
bookmarks means you backed up the wrong volume.
### Fallback: cold copy (no network for `apk add sqlite`)
Stop the service first, then copy the database **and its sidecars** — the `-wal`
is not optional, it is where the newest writes are:
```bash
$COMPOSE stop
docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c "
cp /data/bookmarks.db /backup/bookmarks-$STAMP.db
[ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal
[ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm
ls -1 /backup"
$COMPOSE start
```
Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away,
so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for
the case where it did not. Restoring this variant means putting whichever files
you got back together, under their original names.
Read-only is safe here precisely because nothing opens the database: it is a file
copy, not a SQLite connection.
### Retention
Keep a month, drop the rest — a bookmark database this small compresses the
decision to "disk is free, but not infinite":
```bash
ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v
```
---
## 2. Pull the new code
```bash
git pull --ff-only
git log --oneline -3
```
`--ff-only` so a diverged history fails loudly instead of opening a merge you
did not plan on the production box.
Check whether `.env` needs anything new. New config lands in `.env.example`, and
compose fails at start for a missing required var — better to find out now:
```bash
git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml
```
If a variable was added there, add it to `.env` before continuing.
---
## 3. Rebuild and restart
```bash
$COMPOSE up -d --build
```
**A rebuild is mandatory for any UI change.** The HTML templates, CSS,
JavaScript and fonts are compiled into the binary by `//go:embed`, so editing
them on the server — or pulling them — changes nothing until the image is
rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
bindmounted read-only and read fresh per request.
```bash
$COMPOSE ps # Up, and recently (re)created
docker logs manga-api --tail 20 # -> "listening on :8080 ..."
```
Nothing in the log about the database or the poller failing. The image is tagged
`mangabm-backend:latest`, so the previous image is still on disk untagged —
that is what makes the rollback in §6 quick.
---
## 4. Verify the deploy
Same four API checks as `DEPLOY.md` §3, plus the web UI. Set the host names once:
```bash
API=https://manga-api.violetcrown.my.id
WEB=https://manga.violetcrown.my.id
TOKEN=$(grep -E '^API_TOKEN=' .env | cut -d= -f2)
curl -s $API/healthz # -> ok
curl -s -o /dev/null -w '%{http_code}\n' $API/bookmarks # -> 401
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
# -> your data, not []
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
-H 'Access-Control-Request-Method: PUT' \
$API/bookmarks/x | grep -i access-control # -> allow-origin echoed
```
`[]` from the third call is the alarm that matters: the volume is not attached
and you are looking at an empty database. Stop and check `$COMPOSE config
--volumes` before touching anything else.
Web UI and its assets:
```bash
curl -s -o /dev/null -w '%{http_code}\n' $WEB/ # -> 200 (login page)
curl -s -o /dev/null -w '%{http_code}\n' $WEB/static/style.css # -> 200
# The fonts are self-hosted; this is what tells you they shipped and are typed.
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
$WEB/static/fonts/instrument-serif-400-latin.woff2
# -> 200 font/woff2
# The served userscript, whose @version is its mtime — a pull bumps it.
curl -s $API/u/$TOKEN/manga-bookmark.user.js | grep '@version'
```
`404` on the font means `static/fonts/` did not make it into the image; the UI
will still render, in Georgia, which is easy to miss on a phone. `application/
octet-stream` instead of `font/woff2` means an older binary is running.
Then open `$WEB` in a browser and confirm, in one glance:
- Serif brand and serif row titles — not the system sans fallback.
- A series with an unread chapter has a **crimson title on an ember underline**;
everything else is cool grey. That single detail exercises the whole design
path (template class, CSS, and the poller's `latest_chapter`).
- Tapping the pencil opens the chapter form; Save closes it and the row keeps
its place in the list.
---
## 5. Smoke-test the full loop
The API answering is not the same as the product working. Do this on the phone,
against the real sites — it is the only check that covers the userscript, CORS
and the shared store together:
1. Open a series on **asurascans.com**, open the panel, **+ Bookmark this**.
2. On the server: `curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks`
→ the series is in the JSON.
3. Open a chapter of it, reopen the panel → last-read shows that chapter, and
opening an *older* chapter does not move it backwards.
4. Open **demonicscans.org**, open the panel → the Asura bookmark is listed
there too. Different origin, one store — this is the whole point of the
backend, and the check that fails first when CORS or TLS regressed.
5. Open `$WEB` → the same series appears, with the same chapter.
6. Turn airplane mode on, tap ★ on a row, turn it off, reopen the panel → the
star stuck. That exercises the retry queue.
If 1–5 pass, the deploy is good.
---
## 6. Rollback
Two independent things can be wrong, so undo only what broke.
**Bad code, database fine** — go back to the previous commit and rebuild:
```bash
git log --oneline -5
git checkout <previous-hash>
$COMPOSE up -d --build
```
**Database damaged** — restore the backup from §1. Stop first: the running
process holds the WAL, and dropping a file under a live SQLite connection
corrupts what you were trying to save.
```bash
$COMPOSE stop
docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c '
rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm &&
cp /backup/bookmarks-<STAMP>.db /data/bookmarks.db &&
chown 65532:65532 /data/bookmarks.db &&
ls -l /data'
$COMPOSE start
docker logs manga-api --tail 20
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
```
Two steps here are easy to skip and both bite:
- **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database
mixes two different histories; SQLite will either refuse to open it or quietly
reapply writes you meant to discard.
- **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as
that uid, while the helper container above writes as root. A root-owned
database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then
every write fails. That is the worst possible failure mode — it looks restored.
---
## 7. The whole thing, as one block
For a routine redeploy where nothing needs deciding:
```bash
cd /opt/mangabm
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/mangabm-backups"; mkdir -p "$BACKUP_DIR"
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
STAMP=$(date -u +%Y%m%d-%H%M%S)
docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \
"apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" &&
sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" &&
git pull --ff-only &&
$COMPOSE up -d --build &&
sleep 5 &&
curl -sf https://manga-api.violetcrown.my.id/healthz && echo " deploy ok"
```
The `&&` chain is deliberate: if the backup or its integrity check fails,
nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell
command can tell you the panel works on the phone.
---
## Troubleshooting
| Symptom | Cause / fix |
|---|---|
| `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. |
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
| `compose` errors about `MANGA_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
| `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. |
| Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). |
| Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). |
Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
UI conventions: `docs/design-system.md`.