# Redeploy runbook

Shipping new code to a server that is already running. First-time setup (DNS,
`.env`, Traefik, installing the userscript) is `DEPLOY.md` — this file assumes
all of that exists and picks up at "there is a running stack and I want it to
run the new commit."

Whole thing is ~5 minutes, most of it waiting on `docker build`. Order matters:
**back up before you pull.** A backup taken after a bad migration is a backup of
the damage.

Paths below assume the checkout is at `/opt/mangabm`; substitute your own. The
one absolute rule about paths: **backups live in `../mangabm-backups/`**, a
sibling of the project directory (`/opt/mangabm-backups`), never inside it. It
sits outside the repo so `git pull`, `git clean -fd` and a bad `rm -rf` inside
the checkout cannot take the backups with them.

```
/opt/
├── mangabm/            <- the checkout (this repo)
└── mangabm-backups/    <- bookmarks-YYYYmmdd-HHMMSS.db
```

---

## 0. Preflight

```bash
cd /opt/mangabm

# Both -f flags, every time. The prod override is not standalone.
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"

$COMPOSE ps                        # manga-api should be Up
git status --short                 # expect empty
git log --oneline -1               # note this hash — it is your rollback target
df -h /var/lib/docker | tail -1    # a build needs room
```

If `git status` is dirty, someone edited files on the server. Decide before you
pull: `git stash` to keep it, `git checkout -- .` to discard. A `git pull` onto a
dirty tree fails halfway and leaves you in a worse spot than either.

Create the backup directory once, and make sure it is a sibling, not a child:

```bash
mkdir -p ../mangabm-backups
BACKUP_DIR="$(cd .. && pwd)/mangabm-backups"   # absolute — Docker needs it
echo "$BACKUP_DIR"                             # -> /opt/mangabm-backups
```

---

## 1. Back up the database

The database is a single SQLite file in the named Docker volume, at
`/data/bookmarks.db` inside the container. Find the volume's real name — Compose
prefixes it with the project directory:

```bash
docker volume ls --filter name=bookmarks-data
# -> local   mangabm_bookmarks-data
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
```

### Preferred: hot backup, no downtime

The store runs in **WAL mode**, so recent writes may still be sitting in
`bookmarks.db-wal`. Copying `bookmarks.db` alone while the container runs can
therefore silently drop the newest bookmarks. `VACUUM INTO` folds the WAL in and
writes one consistent file, safe to run against a live database:

```bash
STAMP=$(date -u +%Y%m%d-%H%M%S)      # UTC, sorts chronologically as text
docker run --rm \
  -v "$VOL":/data \
  -v "$BACKUP_DIR":/backup \
  alpine sh -c "apk add -q sqlite &&
    sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\""

ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.db
```

`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.db`. UTC, so the
files sort in real order and never collide across a DST shift.

Note the source volume is mounted **read-write**, which looks wrong for a backup
and is not. Opening a WAL database requires creating the `-shm` shared-memory
file; with `:ro` the command fails with `unable to open database file` and no
backup is produced. `VACUUM INTO` never writes to the source itself.

Verify it before you trust it. An unreadable backup is worse than none, because
you will act as though you have one:

```bash
docker run --rm -v "$BACKUP_DIR":/backup alpine sh -c "apk add -q sqlite &&
  sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;' &&
  sqlite3 /backup/bookmarks-$STAMP.db 'SELECT count(*) FROM bookmarks;'"
# -> ok
# -> 37
```

The count should match what the web UI shows. Zero rows on a server you know has
bookmarks means you backed up the wrong volume.

### Fallback: cold copy (no network for `apk add sqlite`)

Stop the service first, then copy the database **and its sidecars** — the `-wal`
is not optional, it is where the newest writes are:

```bash
$COMPOSE stop
docker run --rm -v "$VOL":/data:ro -v "$BACKUP_DIR":/backup alpine sh -c "
  cp /data/bookmarks.db /backup/bookmarks-$STAMP.db
  [ -f /data/bookmarks.db-wal ] && cp /data/bookmarks.db-wal /backup/bookmarks-$STAMP.db-wal
  [ -f /data/bookmarks.db-shm ] && cp /data/bookmarks.db-shm /backup/bookmarks-$STAMP.db-shm
  ls -1 /backup"
$COMPOSE start
```

Costs ~10 seconds of downtime. A clean shutdown usually checkpoints the WAL away,
so seeing only the `.db` file is normal and fine — the `[ -f ]` guards exist for
the case where it did not. Restoring this variant means putting whichever files
you got back together, under their original names.

Read-only is safe here precisely because nothing opens the database: it is a file
copy, not a SQLite connection.

### Retention

Keep a month, drop the rest — a bookmark database this small compresses the
decision to "disk is free, but not infinite":

```bash
ls -1t "$BACKUP_DIR"/bookmarks-*.db | tail -n +31 | xargs -r rm -v
```

---

## 2. Pull the new code

```bash
git pull --ff-only
git log --oneline -3
```

`--ff-only` so a diverged history fails loudly instead of opening a merge you
did not plan on the production box.

Check whether `.env` needs anything new. New config lands in `.env.example`, and
compose fails at start for a missing required var — better to find out now:

```bash
git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml
```

If a variable was added there, add it to `.env` before continuing.

---

## 3. Rebuild and restart

```bash
$COMPOSE up -d --build
```

**A rebuild is mandatory for any UI change.** The HTML templates, CSS,
JavaScript and fonts are compiled into the binary by `//go:embed`, so editing
them on the server — or pulling them — changes nothing until the image is
rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
bindmounted read-only and read fresh per request.

```bash
$COMPOSE ps                        # Up, and recently (re)created
docker logs manga-api --tail 20    # -> "listening on :8080 ..."
```

Nothing in the log about the database or the poller failing. The image is tagged
`mangabm-backend:latest`, so the previous image is still on disk untagged —
that is what makes the rollback in §6 quick.

---

## 4. Verify the deploy

Same four API checks as `DEPLOY.md` §3, plus the web UI. Set the host names once:

```bash
API=https://manga-api.violetcrown.my.id
WEB=https://manga.violetcrown.my.id
TOKEN=$(grep -E '^API_TOKEN=' .env | cut -d= -f2)

curl -s $API/healthz                                            # -> ok
curl -s -o /dev/null -w '%{http_code}\n' $API/bookmarks         # -> 401
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
                                                                # -> your data, not []
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
  -H 'Access-Control-Request-Method: PUT' \
  $API/bookmarks/x | grep -i access-control                     # -> allow-origin echoed
```

`[]` from the third call is the alarm that matters: the volume is not attached
and you are looking at an empty database. Stop and check `$COMPOSE config
--volumes` before touching anything else.

Web UI and its assets:

```bash
curl -s -o /dev/null -w '%{http_code}\n' $WEB/                  # -> 200 (login page)
curl -s -o /dev/null -w '%{http_code}\n' $WEB/static/style.css  # -> 200

# The fonts are self-hosted; this is what tells you they shipped and are typed.
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
  $WEB/static/fonts/instrument-serif-400-latin.woff2
# -> 200 font/woff2

# The served userscript, whose @version is its mtime — a pull bumps it.
curl -s $API/u/$TOKEN/manga-bookmark.user.js | grep '@version'
```

`404` on the font means `static/fonts/` did not make it into the image; the UI
will still render, in Georgia, which is easy to miss on a phone. `application/
octet-stream` instead of `font/woff2` means an older binary is running.

Then open `$WEB` in a browser and confirm, in one glance:

- Serif brand and serif row titles — not the system sans fallback.
- A series with an unread chapter has a **crimson title on an ember underline**;
  everything else is cool grey. That single detail exercises the whole design
  path (template class, CSS, and the poller's `latest_chapter`).
- Tapping the pencil opens the chapter form; Save closes it and the row keeps
  its place in the list.

---

## 5. Smoke-test the full loop

The API answering is not the same as the product working. Do this on the phone,
against the real sites — it is the only check that covers the userscript, CORS
and the shared store together:

1. Open a series on **asurascans.com**, open the panel, **+ Bookmark this**.
2. On the server: `curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks`
   → the series is in the JSON.
3. Open a chapter of it, reopen the panel → last-read shows that chapter, and
   opening an *older* chapter does not move it backwards.
4. Open **demonicscans.org**, open the panel → the Asura bookmark is listed
   there too. Different origin, one store — this is the whole point of the
   backend, and the check that fails first when CORS or TLS regressed.
5. Open `$WEB` → the same series appears, with the same chapter.
6. Turn airplane mode on, tap ★ on a row, turn it off, reopen the panel → the
   star stuck. That exercises the retry queue.

If 1–5 pass, the deploy is good.

---

## 6. Rollback

Two independent things can be wrong, so undo only what broke.

**Bad code, database fine** — go back to the previous commit and rebuild:

```bash
git log --oneline -5
git checkout <previous-hash>
$COMPOSE up -d --build
```

**Database damaged** — restore the backup from §1. Stop first: the running
process holds the WAL, and dropping a file under a live SQLite connection
corrupts what you were trying to save.

```bash
$COMPOSE stop

docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c '
  rm -f /data/bookmarks.db /data/bookmarks.db-wal /data/bookmarks.db-shm &&
  cp /backup/bookmarks-<STAMP>.db /data/bookmarks.db &&
  chown 65532:65532 /data/bookmarks.db &&
  ls -l /data'

$COMPOSE start
docker logs manga-api --tail 20
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
```

Two steps here are easy to skip and both bite:

- **Delete the stale `-wal` and `-shm`.** Leaving them beside a restored database
  mixes two different histories; SQLite will either refuse to open it or quietly
  reapply writes you meant to discard.
- **`chown 65532:65532`.** The image is `distroless/static:nonroot` and runs as
  that uid, while the helper container above writes as root. A root-owned
  database opens read-only-ish: reads work, so `/bookmarks` looks fine, and then
  every write fails. That is the worst possible failure mode — it looks restored.

---

## 7. The whole thing, as one block

For a routine redeploy where nothing needs deciding:

```bash
cd /opt/mangabm
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/mangabm-backups"; mkdir -p "$BACKUP_DIR"
VOL=$(docker volume ls --filter name=bookmarks-data -q | head -1)
STAMP=$(date -u +%Y%m%d-%H%M%S)

docker run --rm -v "$VOL":/data -v "$BACKUP_DIR":/backup alpine sh -c \
  "apk add -q sqlite && sqlite3 /data/bookmarks.db \"VACUUM INTO '/backup/bookmarks-$STAMP.db'\" &&
   sqlite3 /backup/bookmarks-$STAMP.db 'PRAGMA integrity_check;'" &&
git pull --ff-only &&
$COMPOSE up -d --build &&
sleep 5 &&
curl -sf https://manga-api.violetcrown.my.id/healthz && echo " deploy ok"
```

The `&&` chain is deliberate: if the backup or its integrity check fails,
nothing is pulled and nothing is rebuilt. Then still do §5 by hand — no shell
command can tell you the panel works on the phone.

---

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `/bookmarks` returns `[]` after redeploy | Volume not attached — check `$COMPOSE config --volumes` and that you passed both `-f` files. Do **not** re-bookmark; the data is still in the volume. |
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
| Everyone logged out of the web UI | `API_TOKEN` or `WEB_PASSWORD` changed; sessions are derived from both. Expected, just log in again. |
| `compose` errors about `MANGA_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
| `apk add sqlite` fails (no network) | Use the cold-copy fallback in §1 — and copy `bookmarks.db-wal` too. |
| Reads work but every write fails after a restore | Restored file is root-owned; the container is uid 65532. `chown 65532:65532` it (§6). |
| Backup command: `unable to open database file` | Source volume mounted `:ro`. WAL needs to create `-shm`; mount it read-write (§1). |

Full first-time setup: `DEPLOY.md`. Config reference and endpoints: `README.md`.
UI conventions: `docs/design-system.md`.