# Redeploy runbook

Shipping new code to a server that is already running. First-time setup (DNS,
`.env`, Traefik, installing the userscript) is `DEPLOY.md` — this file assumes
all of that exists and picks up at "there is a running stack and I want it to
run the new commit."

Whole thing is ~5 minutes, most of it waiting on `docker build`. Order matters:
**back up before you pull.** A backup taken after a bad migration is a backup of
the damage.

Paths below assume the checkout is at `~/mangaBookmark`, which is where it lives
on this deployment; substitute your own. The one absolute rule about paths:
**backups live in a `-backups` sibling of the checkout**, never inside it. It
sits outside the repo so `git pull`, `git clean -fd` and a bad `rm -rf` inside
the checkout cannot take the backups with them.

```
~/
├── mangaBookmark/            <- the checkout (this repo)
└── mangaBookmark-backups/    <- bookmarks-YYYYmmdd-HHMMSS.dump
```

---

## 0. Preflight

```bash
cd ~/mangaBookmark

# Both -f flags, every time. The prod override is not standalone.
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"

$COMPOSE ps                        # bookmark-api should be Up
git status --short                 # expect empty
git log --oneline -1               # note this hash — it is your rollback target
df -h /var/lib/docker | tail -1    # a build needs room
```

If `git status` is dirty, someone edited files on the server. Decide before you
pull: `git stash` to keep it, `git checkout -- .` to discard. A `git pull` onto a
dirty tree fails halfway and leaves you in a worse spot than either.

Create the backup directory once, and make sure it is a sibling, not a child:

```bash
BACKUP_DIR="$(cd .. && pwd)/$(basename "$PWD")-backups"   # absolute — Docker needs it
mkdir -p "$BACKUP_DIR"
echo "$BACKUP_DIR"                        # -> /home/sulthan/mangaBookmark-backups
```

---

## 1. Back up the database

The database is Postgres, running as the `postgres` service on the named volume
`postgres-data`. It has **no published port** — nothing outside the internal `db`
network can reach it — so every command below goes in through the container:

```bash
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
# -> bookmarks, readers, schema_migrations, series, sessions
```

Inside the container that connects over the local socket as the `bookmarks`
superuser, so no password is needed anywhere in this section. `-T` is not
optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and
silently corrupts any binary stream flowing back out — see the dump below.

### Preferred: hot dump, no downtime

`pg_dump` runs in a single repeatable-read transaction, so it writes one
point-in-time-consistent snapshot while the API keeps serving. No stopping, no
WAL to worry about — that is the server's problem, not yours.

```bash
STAMP=$(date -u +%Y%m%d-%H%M%S)      # UTC, sorts chronologically as text
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
  > "$BACKUP_DIR/bookmarks-$STAMP.dump"

ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump
```

`-Fc` is the custom archive format rather than plain SQL: it is compressed, and
`pg_restore` can inspect and replay it selectively — list its table of contents,
restore one table, restore schema without data, reorder. A plain `.sql` dump can
only be piped into `psql` whole, and gives you no way to check what is in it
short of reading it.

`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.dump`. UTC, so
the files sort in real order and never collide across a DST shift.

Verify it before you trust it. An unreadable backup is worse than none, because
you will act as though you have one:

```bash
# 1. The dump parses and contains the tables. Uses the same image compose
#    already pulls, so nothing new to install.
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
  pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
# -> 1235; 0 0 TABLE DATA public readers bookmarks
# -> 1236; 0 0 TABLE DATA public schema_migrations bookmarks
# -> 1237; 0 0 TABLE DATA public series bookmarks
# -> 1238; 0 0 TABLE DATA public sessions bookmarks

# 2. Sanity-check the live row count you just captured.
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
  -c 'select count(*) from bookmarks'
# -> 37
```

A custom-format archive stores row counts nowhere, so step 1 proves the file is
a readable archive with the right tables in it, not that the rows are there;
step 2 is the number those rows should be. It should match what the web UI
shows. Zero on a server you know has bookmarks means the API and your `psql`
are looking at different databases — check `DATABASE_URL`.

### Fallback: cold volume archive

Use this when you want the whole data directory rather than a logical dump — a
like-for-like restore of the same Postgres major version onto the same host.

**The stack must be stopped first.** A running Postgres has dirty pages in
shared buffers and WAL that has not been replayed into the data files, and `tar`
walks the directory over several seconds while the server keeps writing to it.
The archive you get is torn: files from different instants, possibly a
half-written page. It may restore, start, and be quietly wrong. Online
filesystem-level backup is `pg_basebackup`'s job, not `tar`'s; with the
container stopped the shutdown checkpoint has already flushed everything and a
plain archive of the volume is consistent.

```bash
# Derived exactly, not with a `--filter name=` substring match plus `head -1`:
# that quietly picks the first of however many volumes happen to contain the
# string, and archiving the wrong data directory is not a visible failure.
VOL="$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_postgres-data"
docker volume inspect "$VOL" >/dev/null && echo "$VOL"   # -> mangabookmark_postgres-data

$COMPOSE stop
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \
  tar czf "/to/postgres-data-$STAMP.tgz" -C /from .
$COMPOSE start

ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz
```

Costs ~15 seconds of downtime. Read-only on the source is safe here precisely
because nothing is running against it. Restoring this variant means untarring it
back into an *empty* `postgres-data` volume with the stack down — it is a whole
data directory, not a file you can drop next to the live one, and it will only
start under `postgres:17`.

### Retention

Keep a month, drop the rest — a bookmark database this small compresses the
decision to "disk is free, but not infinite":

```bash
ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v
```

### A note on the old `bookmarks-data` volume

`bookmarks-data` is the **pre-migration SQLite volume**. It is deliberately not
declared in `docker-compose.yml` any more, which is what keeps `docker compose
down -v` from taking it with the rest of the stack. It is not the live database
and nothing reads it — the one-way move out of it is `CUTOVER.md`. Once the
Postgres data has been trusted for a while, remove it by hand — nothing else will.
Its full name is `<compose project>_bookmarks-data`, and the project name is the
lowercased directory name of the checkout:

```bash
docker volume rm "$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_bookmarks-data"
```

---

## 2. Pull the new code

```bash
git pull --ff-only
git log --oneline -3
```

`--ff-only` so a diverged history fails loudly instead of opening a merge you
did not plan on the production box.

Check whether `.env` needs anything new. New config lands in `.env.example`, and
compose fails at start for a missing required var — better to find out now:

```bash
git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml
```

If a variable was added there, add it to `.env` before continuing.

---

## 3. Rebuild and restart

```bash
$COMPOSE up -d --build
```

**A rebuild is mandatory for any UI change.** The HTML templates, CSS,
JavaScript and fonts are compiled into the binary by `//go:embed`, so editing
them on the server — or pulling them — changes nothing until the image is
rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
bindmounted read-only and read fresh per request.

```bash
$COMPOSE ps                        # bookmark-api Up; postgres Up (healthy)
docker logs bookmark-api --tail 20    # -> "listening on :8080 ..."
```

Nothing in the log about the database, the migrations or the poller failing.
`bookmark-api` waits on `postgres` reporting healthy before it starts and the
binary applies any pending migration before it listens, so an API that never
says "listening" is usually the database, not the code — `$COMPOSE logs
postgres` first. The image is tagged
`bookmarkmanager-backend:latest`, so the previous image is still on disk untagged —
that is what makes the rollback in §6 quick.

---

## 4. Verify the deploy

Same four API checks as `DEPLOY.md` §3, plus the web UI. Set the host names once:

```bash
API=https://bookmark-api.violetcrown.my.id
WEB=https://bookmark.violetcrown.my.id
# During the grace window the retired global credential still resolves to the
# owner; afterwards it is 401 like any other wrong credential.
TOKEN=$(grep -E '^API_TOKEN=' .env | cut -d= -f2)

curl -s $API/healthz                                            # -> ok
curl -s -o /dev/null -w '%{http_code}\n' $API/bookmarks         # -> 401
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
                                                                # -> your data, not []
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
  -H 'Access-Control-Request-Method: PUT' \
  $API/bookmarks/x | grep -i access-control                     # -> allow-origin echoed
```

`[]` from the third call is the alarm that matters: you are talking to an empty
database, which means the API found a *different* Postgres than the one holding
your data — a renamed project directory, a fresh `postgres-data`, or a
`DATABASE_URL` override in `.env` pointing elsewhere. Stop and check, before
touching anything else:

```bash
$COMPOSE config --volumes                                       # -> postgres-data
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks'
```

Web UI and its assets:

```bash
curl -s -o /dev/null -w '%{http_code}\n' $WEB/                  # -> 200 (login page)
curl -s -o /dev/null -w '%{http_code}\n' $WEB/static/style.css  # -> 200

# The fonts are self-hosted; this is what tells you they shipped and are typed.
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
  $WEB/static/fonts/instrument-serif-400-latin.woff2
# -> 200 font/woff2

# The served userscript, whose @version is its mtime — a pull bumps it.
curl -s $API/u/$TOKEN/manga-bookmark.user.js | grep '@version'
```

`404` on the font means `static/fonts/` did not make it into the image; the UI
will still render, in Georgia, which is easy to miss on a phone. `application/
octet-stream` instead of `font/woff2` means an older binary is running.

Then open `$WEB` in a browser and confirm, in one glance:

- Serif brand and serif row titles — not the system sans fallback.
- A series with an unread chapter has a **crimson title on an ember underline**;
  everything else is cool grey. That single detail exercises the whole design
  path (template class, CSS, and the poller's `latest_chapter`).
- Tapping the pencil opens the chapter form; Save closes it and the row keeps
  its place in the list.

---

## 5. Smoke-test the full loop

The API answering is not the same as the product working. Do this on the phone,
against the real sites — it is the only check that covers the userscript, CORS
and the shared store together:

1. Open a series on **asurascans.com**, open the panel, **+ Bookmark this**.
2. On the server: `curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks`
   → the series is in the JSON.
3. Open a chapter of it, reopen the panel → last-read shows that chapter, and
   opening an *older* chapter does not move it backwards.
4. Open **demonicscans.org**, open the panel → the Asura bookmark is listed
   there too. Different origin, one store — this is the whole point of the
   backend, and the check that fails first when CORS or TLS regressed.
5. Open `$WEB` → the same series appears, with the same chapter.
6. Turn airplane mode on, tap ★ on a row, turn it off, reopen the panel → the
   star stuck. That exercises the retry queue.

If 1–5 pass, the deploy is good.

---

## 6. Rollback

Two independent things can be wrong, so undo only what broke.

**Bad code, database fine** — go back to the previous commit and rebuild:

```bash
git log --oneline -5
git checkout <previous-hash>
$COMPOSE up -d --build
```

**Database damaged** — restore the dump from §1. Stop **only the API**, not the
whole stack: `pg_restore` needs the server up to restore into, and it needs
`bookmark-api`'s connection pool gone, because `--clean` cannot drop a table
other sessions are holding open.

```bash
$COMPOSE stop bookmark-api

$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \
  < "$BACKUP_DIR/bookmarks-<STAMP>.dump"

$COMPOSE start bookmark-api
docker logs bookmark-api --tail 20
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
```

Three things here are easy to skip and all three bite:

- **`--clean --if-exists`.** Without `--clean` the dump's rows land *on top of*
  what is already there and you get primary-key collisions half way through, a
  partially restored database, and a non-zero exit you may not notice.
  `--if-exists` only suppresses the "does not exist" noise when the target is
  already empty; it is not the part doing the work.
- **`-T` again.** Feeding a custom-format archive into a TTY-allocated `exec`
  corrupts it in flight and `pg_restore` fails with a garbled-header error on a
  file that is perfectly fine on disk.
- **Stop the API, not Postgres.** `$COMPOSE stop` (everything) leaves you with
  nothing to restore into; leaving `bookmark-api` running leaves connections
  that block the drops *and* lets the poller write into a half-restored table.

No ownership fixing is needed any more — the Postgres image owns `postgres-data`
itself and `pg_restore` writes through the server, not the filesystem.
`schema_migrations` is inside the dump, so the database comes back at whatever
schema version the backup was taken at; the migration runner applies anything
newer the next time `bookmark-api` starts.

---

## 7. The whole thing, as one block

For a routine redeploy where nothing needs deciding:

```bash
cd /opt/bookmarkmanager
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
STAMP=$(date -u +%Y%m%d-%H%M%S)

$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
  > "$BACKUP_DIR/bookmarks-$STAMP.dump" &&
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
  pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null &&
git pull --ff-only &&
$COMPOSE up -d --build &&
sleep 5 &&
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"
```

The `&&` chain is deliberate: if the dump or its `pg_restore --list` check
fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a
short or empty `.dump` behind — the shell creates the file before `pg_dump`
runs — so delete it rather than letting it sit in the backup directory looking
like a backup. Then still do §5 by hand — no shell command can tell you the
panel works on the phone.

---

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `/bookmarks` returns `[]` after redeploy | You are on an empty Postgres. Check `$COMPOSE config --volumes` lists `postgres-data`, that you passed both `-f` files, and that `.env` has no stray `DATABASE_URL` override. Do **not** re-bookmark; the data is still in the volume. |
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
| Everyone logged out of the web UI | The `sessions` table was wiped; sessions are database rows, not signed cookies. Expected after a deliberate revoke. |
| `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
| `bookmark-api` crash-loops, log says `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` in `.env` no longer matches the one burned into `postgres-data` at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: `$COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks'` (prompts, so nothing lands in shell history) and then match `.env` to it. |
| `compose` errors `set POSTGRES_PASSWORD in .env` | Unset. Compose builds the backend's `DATABASE_URL` out of it, so it is required even though you never write that URL yourself. Run from the directory holding `.env`. |
| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. |
| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. |
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
| `git pull`: `could not read Username for 'https://…'` | The checkout's remote is the HTTPS clone URL and the server has no credential helper, so the pull prompts into a closed stdin. Switch it to SSH once — `git remote set-url origin ssh://git@gitea.violetcrown.my.id:2222/sulthan/mangaBookmark.git`. Gitea's SSH listens on **2222**, not 22; port 22 is the host's own sshd and answers `Permission denied (publickey)` no matter which key is registered. |

Full first-time setup: `DEPLOY.md`. The one-off SQLite→Postgres move:
`CUTOVER.md`. Config reference and endpoints: `README.md`.
UI conventions: `docs/design-system.md`.