Files
mangaBookmark/REDEPLOY.md
T
sulthan e4a313e626 Move the browser off the VPS to its own unit (#46)
The headless browser leaves the API stack. It becomes its own compose unit
(chrome/docker-compose.yml) deployed on the home machine and reached over the
tailnet, returning 471 MiB of working set to a 1974 MiB VPS that has no swap.
No fallback sidecar is left behind.

The backend needs no code change: BROWSER_WS_URL was already the only coupling,
so relocation is one environment variable. Its default is now empty rather than
a pinned Docker IP — an unreachable or unconfigured browser degrades exactly as
it always has, with plain-TLS libraries unaffected, kagane and novelfull logged
and skipped, and stored covers still served.

The browser unit publishes CDP on ${BROWSER_BIND_ADDR} with no default, because
CDP authenticates nothing and the home machine has a real LAN: an unset value
must fail the deploy rather than silently expose an endpoint that is remote code
execution for anything that reaches it. Resource limits are sized against the
measured 645 MiB untuned peak and the CI runner that already holds 1.2 GiB of
that box.

bookmark-api gains the default network. Dropping `browser` left it on `db`
alone, which is internal: true — that meant no published port and, worse, no
egress for the poller at all. Caught by bringing the stack up.

Docs: ADR-0006 for the topology, DEPLOY.md §7 for first-time setup of the
browser machine, REDEPLOY.md §8 for its independent update cadence, plus the
architecture diagrams, config tables and troubleshooting rows.
2026-08-09 15:19:54 +07:00

465 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Redeploy runbook
Shipping new code to a server that is already running. First-time setup (DNS,
`.env`, Traefik, installing the userscript) is `DEPLOY.md` — this file assumes
all of that exists and picks up at "there is a running stack and I want it to
run the new commit."
Whole thing is ~5 minutes, most of it waiting on `docker build`. Order matters:
**back up before you pull.** A backup taken after a bad migration is a backup of
the damage.
Paths below assume the checkout is at `~/mangaBookmark`, which is where it lives
on this deployment; substitute your own. The one absolute rule about paths:
**backups live in a `-backups` sibling of the checkout**, never inside it. It
sits outside the repo so `git pull`, `git clean -fd` and a bad `rm -rf` inside
the checkout cannot take the backups with them.
```
~/
├── mangaBookmark/ <- the checkout (this repo)
└── mangaBookmark-backups/ <- bookmarks-YYYYmmdd-HHMMSS.dump
```
---
## 0. Preflight
```bash
cd ~/mangaBookmark
# Both -f flags, every time. The prod override is not standalone.
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
$COMPOSE ps # bookmark-api should be Up
git status --short # expect empty
git log --oneline -1 # note this hash — it is your rollback target
df -h /var/lib/docker | tail -1 # a build needs room
```
If `git status` is dirty, someone edited files on the server. Decide before you
pull: `git stash` to keep it, `git checkout -- .` to discard. A `git pull` onto a
dirty tree fails halfway and leaves you in a worse spot than either.
Create the backup directory once, and make sure it is a sibling, not a child:
```bash
BACKUP_DIR="$(cd .. && pwd)/$(basename "$PWD")-backups" # absolute — Docker needs it
mkdir -p "$BACKUP_DIR"
echo "$BACKUP_DIR" # -> /home/sulthan/mangaBookmark-backups
```
---
## 1. Back up the database
The database is Postgres, running as the `postgres` service on the named volume
`postgres-data`. It has **no published port** — nothing outside the internal `db`
network can reach it — so every command below goes in through the container:
```bash
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
# -> bookmarks, covers, readers, schema_migrations, series, sessions
```
Inside the container that connects over the local socket as the `bookmarks`
superuser, so no password is needed anywhere in this section. `-T` is not
optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and
silently corrupts any binary stream flowing back out — see the dump below.
### Preferred: hot dump, no downtime
`pg_dump` runs in a single repeatable-read transaction, so it writes one
point-in-time-consistent snapshot while the API keeps serving. No stopping, no
WAL to worry about — that is the server's problem, not yours.
```bash
STAMP=$(date -u +%Y%m%d-%H%M%S) # UTC, sorts chronologically as text
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
> "$BACKUP_DIR/bookmarks-$STAMP.dump"
ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump
```
`-Fc` is the custom archive format rather than plain SQL: it is compressed, and
`pg_restore` can inspect and replay it selectively — list its table of contents,
restore one table, restore schema without data, reorder. A plain `.sql` dump can
only be piped into `psql` whole, and gives you no way to check what is in it
short of reading it.
`$STAMP` is the "time in the name" — `bookmarks-20260730-014233.dump`. UTC, so
the files sort in real order and never collide across a DST shift.
Verify it before you trust it. An unreadable backup is worse than none, because
you will act as though you have one:
```bash
# 1. The dump parses and contains the tables. Uses the same image compose
# already pulls, so nothing new to install.
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
# -> 1235; 0 0 TABLE DATA public covers bookmarks
# -> 1236; 0 0 TABLE DATA public readers bookmarks
# -> 1237; 0 0 TABLE DATA public schema_migrations bookmarks
# -> 1238; 0 0 TABLE DATA public series bookmarks
# -> 1239; 0 0 TABLE DATA public sessions bookmarks
# 2. Sanity-check the live row count you just captured.
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
-c 'select count(*) from bookmarks'
# -> 37
```
A custom-format archive stores row counts nowhere, so step 1 proves the file is
a readable archive with the right tables in it, not that the rows are there;
step 2 is the number those rows should be. It should match what the web UI
shows. Zero on a server you know has bookmarks means the API and your `psql`
are looking at different databases — check `DATABASE_URL`.
### Fallback: cold volume archive
Use this when you want the whole data directory rather than a logical dump — a
like-for-like restore of the same Postgres major version onto the same host.
**The stack must be stopped first.** A running Postgres has dirty pages in
shared buffers and WAL that has not been replayed into the data files, and `tar`
walks the directory over several seconds while the server keeps writing to it.
The archive you get is torn: files from different instants, possibly a
half-written page. It may restore, start, and be quietly wrong. Online
filesystem-level backup is `pg_basebackup`'s job, not `tar`'s; with the
container stopped the shutdown checkpoint has already flushed everything and a
plain archive of the volume is consistent.
```bash
# Derived exactly, not with a `--filter name=` substring match plus `head -1`:
# that quietly picks the first of however many volumes happen to contain the
# string, and archiving the wrong data directory is not a visible failure.
VOL="$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_postgres-data"
docker volume inspect "$VOL" >/dev/null && echo "$VOL" # -> mangabookmark_postgres-data
$COMPOSE stop
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \
tar czf "/to/postgres-data-$STAMP.tgz" -C /from .
$COMPOSE start
ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz
```
Costs ~15 seconds of downtime. Read-only on the source is safe here precisely
because nothing is running against it. Restoring this variant means untarring it
back into an *empty* `postgres-data` volume with the stack down — it is a whole
data directory, not a file you can drop next to the live one, and it will only
start under `postgres:17`.
### Retention
Keep a month, drop the rest — a bookmark database this small compresses the
decision to "disk is free, but not infinite":
```bash
ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v
```
### A note on the old `bookmarks-data` volume
`bookmarks-data` is the **pre-migration SQLite volume**. It is deliberately not
declared in `docker-compose.yml` any more, which is what keeps `docker compose
down -v` from taking it with the rest of the stack. It is not the live database
and nothing reads it — the one-way move out of it is `CUTOVER.md`. Once the
Postgres data has been trusted for a while, remove it by hand — nothing else will.
Its full name is `<compose project>_bookmarks-data`, and the project name is the
lowercased directory name of the checkout:
```bash
docker volume rm "$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_bookmarks-data"
```
---
## 2. Pull the new code
```bash
git pull --ff-only
git log --oneline -3
```
`--ff-only` so a diverged history fails loudly instead of opening a merge you
did not plan on the production box.
Check whether `.env` needs anything new. New config lands in `.env.example`, and
compose fails at start for a missing required var — better to find out now:
```bash
git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml
```
If a variable was added there, add it to `.env` before continuing.
---
## 3. Rebuild and restart
```bash
$COMPOSE up -d --build
```
**A rebuild is mandatory for any UI change.** The HTML templates, CSS,
JavaScript and fonts are compiled into the binary by `//go:embed`, so editing
them on the server — or pulling them — changes nothing until the image is
rebuilt. The one exception is `userscript/manga-bookmark.user.js`, which is
bindmounted read-only and read fresh per request.
```bash
$COMPOSE ps # bookmark-api Up; postgres Up (healthy)
docker logs bookmark-api --tail 20 # -> "listening on :8080 ..."
```
Nothing in the log about the database, the migrations or the poller failing.
`bookmark-api` waits on `postgres` reporting healthy before it starts and the
binary applies any pending migration before it listens, so an API that never
says "listening" is usually the database, not the code — `$COMPOSE logs
postgres` first. The image is tagged
`bookmarkmanager-backend:latest`, so the previous image is still on disk untagged —
that is what makes the rollback in §6 quick.
---
## 4. Verify the deploy
Same four API checks as `DEPLOY.md` §3, plus the web UI. Set the host names once:
```bash
API=https://bookmark-api.violetcrown.my.id
WEB=https://bookmark.violetcrown.my.id
# Your own Reader credential - derived, never stored in .env. Take it from the
# Userscripts panel's install link after signing in, or from an installed
# script's API_TOKEN constant.
TOKEN=<your Reader credential>
curl -s $API/healthz # -> ok
curl -s -o /dev/null -w '%{http_code}\n' $API/bookmarks # -> 401
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
# -> your data, not []
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
-H 'Access-Control-Request-Method: PUT' \
$API/bookmarks/x | grep -i access-control # -> allow-origin echoed
```
`[]` from the third call is the alarm that matters: you are talking to an empty
database, which means the API found a *different* Postgres than the one holding
your data — a renamed project directory, a fresh `postgres-data`, or a
`DATABASE_URL` override in `.env` pointing elsewhere. Stop and check, before
touching anything else:
```bash
$COMPOSE config --volumes # -> postgres-data
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks'
```
Web UI and its assets:
```bash
curl -s -o /dev/null -w '%{http_code}\n' $WEB/ # -> 200 (login page)
curl -s -o /dev/null -w '%{http_code}\n' $WEB/static/style.css # -> 200
# The fonts are self-hosted; this is what tells you they shipped and are typed.
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
$WEB/static/fonts/instrument-serif-400-latin.woff2
# -> 200 font/woff2
# The served userscript, whose @version is its mtime — a pull bumps it.
curl -s $API/u/$TOKEN/manga-bookmark.user.js | grep '@version'
```
`404` on the font means `static/fonts/` did not make it into the image; the UI
will still render, in Georgia, which is easy to miss on a phone. `application/
octet-stream` instead of `font/woff2` means an older binary is running.
Then open `$WEB` in a browser and confirm, in one glance:
- Serif brand and serif row titles — not the system sans fallback.
- A series with an unread chapter has a **crimson title on an ember underline**;
everything else is cool grey. That single detail exercises the whole design
path (template class, CSS, and the poller's `latest_chapter`).
- Tapping the pencil opens the chapter form; Save closes it and the row keeps
its place in the list.
---
## 5. Smoke-test the full loop
The API answering is not the same as the product working. Do this on the phone,
against the real sites — it is the only check that covers the userscript, CORS
and the shared store together:
1. Open a series on **asurascans.com**, open the panel, **+ Bookmark this**.
2. On the server: `curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks`
→ the series is in the JSON.
3. Open a chapter of it, reopen the panel → last-read shows that chapter, and
opening an *older* chapter does not move it backwards.
4. Open **demonicscans.org**, open the panel → the Asura bookmark is listed
there too. Different origin, one store — this is the whole point of the
backend, and the check that fails first when CORS or TLS regressed.
5. Open `$WEB` → the same series appears, with the same chapter.
6. Turn airplane mode on, tap ★ on a row, turn it off, reopen the panel → the
star stuck. That exercises the retry queue.
If 1–5 pass, the deploy is good.
---
## 6. Rollback
Two independent things can be wrong, so undo only what broke.
**Bad code, database fine** — go back to the previous commit and rebuild:
```bash
git log --oneline -5
git checkout <previous-hash>
$COMPOSE up -d --build
```
**Database damaged** — restore the dump from §1. Stop **only the API**, not the
whole stack: `pg_restore` needs the server up to restore into, and it needs
`bookmark-api`'s connection pool gone, because `--clean` cannot drop a table
other sessions are holding open.
```bash
$COMPOSE stop bookmark-api
$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \
< "$BACKUP_DIR/bookmarks-<STAMP>.dump"
$COMPOSE start bookmark-api
docker logs bookmark-api --tail 20
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
```
Three things here are easy to skip and all three bite:
- **`--clean --if-exists`.** Without `--clean` the dump's rows land *on top of*
what is already there and you get primary-key collisions half way through, a
partially restored database, and a non-zero exit you may not notice.
`--if-exists` only suppresses the "does not exist" noise when the target is
already empty; it is not the part doing the work.
- **`-T` again.** Feeding a custom-format archive into a TTY-allocated `exec`
corrupts it in flight and `pg_restore` fails with a garbled-header error on a
file that is perfectly fine on disk.
- **Stop the API, not Postgres.** `$COMPOSE stop` (everything) leaves you with
nothing to restore into; leaving `bookmark-api` running leaves connections
that block the drops *and* lets the poller write into a half-restored table.
No ownership fixing is needed any more — the Postgres image owns `postgres-data`
itself and `pg_restore` writes through the server, not the filesystem.
`schema_migrations` is inside the dump, so the database comes back at whatever
schema version the backup was taken at; the migration runner applies anything
newer the next time `bookmark-api` starts.
---
## 7. The whole thing, as one block
For a routine redeploy where nothing needs deciding:
```bash
cd /opt/bookmarkmanager
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
STAMP=$(date -u +%Y%m%d-%H%M%S)
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
> "$BACKUP_DIR/bookmarks-$STAMP.dump" &&
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null &&
git pull --ff-only &&
$COMPOSE up -d --build &&
sleep 5 &&
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"
```
The `&&` chain is deliberate: if the dump or its `pg_restore --list` check
fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a
short or empty `.dump` behind — the shell creates the file before `pg_dump`
runs — so delete it rather than letting it sit in the backup directory looking
like a backup. Then still do §5 by hand — no shell command can tell you the
panel works on the phone.
---
## 8. The browser unit (separate machine, separate cadence)
Everything above is the API stack on the VPS. The headless browser is its own
compose unit on the home machine (ADR-0006, `DEPLOY.md` §7) and is redeployed
on its own schedule — it holds no data you can lose, so there is nothing to
back up and no ordering constraint against the API.
```bash
cd ~/mangaBookmark/chrome
git pull --ff-only
docker compose up -d --build
```
Then confirm it answers, and that a stopped-and-restarted Chrome is invisible
to the API:
```bash
curl -s -m 15 http://$(tailscale ip -4):9222/json/version | head -c 120
# -> {"Browser":"Chrome/1xx...","webSocketDebuggerUrl":"ws://...<new uuid>"}
```
The first call takes a few seconds: Chrome is not running until something
connects, and it is reaped again after five idle minutes. The debugger UUID
changes on every start and the API does not care — chromedp re-runs
`/json/version` discovery per fetch, which is exactly why `chromedp.NoModifyURL`
must never be added to `browser.go`.
**Rebuild is the Chrome upgrade path.** The image installs
`google-chrome-stable` unpinned on purpose: a stale browser is what Cloudflare
turns away, and the pinned Chrome 124 in `zenika/alpine-chrome` is the worked
example. The `chrome-profile` volume survives `--build`, so clearance cookies
are reused rather than re-solved.
Two things worth a glance after several days, both from the acceptance criteria
of the move:
```bash
docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'
# -> 0 false
free -m # the Gitea runner should still have its headroom
```
Nothing here needs doing during an API redeploy. The API stack does not
`depends_on` the browser, and an unreachable one degrades exactly as an unset
`BROWSER_WS_URL`: plain-TLS libraries unaffected, kagane and novelfull logged
and skipped, stored covers still served.
---
## Troubleshooting
| Symptom | Cause / fix |
|---|---|
| `/bookmarks` returns `[]` after redeploy | You are on an empty Postgres. Check `$COMPOSE config --volumes` lists `postgres-data`, that you passed both `-f` files, and that `.env` has no stray `DATABASE_URL` override. Do **not** re-bookmark; the data is still in the volume. |
| UI looks like plain Georgia / system sans | `static/fonts/` missing from the image, or the browser cached an old `style.css`. `/static/*` is served `max-age=3600`, so hard-reload or wait an hour. |
| CSS or template change did not appear | You restarted without `--build`. Assets are `//go:embed`ed. |
| Font answers `application/octet-stream` | Old binary — the `.woff2` MIME registration is in `web.go`. Rebuild. |
| Everyone logged out of the web UI | The `sessions` table was wiped; sessions are database rows, not signed cookies. Expected after a deliberate revoke. |
| `compose` errors about `BOOKMARK_WEB_HOST` | Run from the directory holding `.env`. Both host vars are required even when the web UI is unused. |
| Userscript did not update on the phone | Violentmonkey polls on its own schedule; force a check. `@version` comes from the file's mtime, so confirm the pull actually touched it. |
| `bookmark-api` crash-loops, log says `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` in `.env` no longer matches the one burned into `postgres-data` at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: `$COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks'` (prompts, so nothing lands in shell history) and then match `.env` to it. |
| `compose` errors `set POSTGRES_PASSWORD in .env` | Unset. Compose builds the backend's `DATABASE_URL` out of it, so it is required even though you never write that URL yourself. Run from the directory holding `.env`. |
| `postgres` never leaves `starting`; `bookmark-api` never starts either | The healthcheck (`pg_isready`) is failing and `bookmark-api` waits on it. `$COMPOSE logs postgres` — usually `postgres-data` was initialised by a different major version ("database files are incompatible with server"), or the disk is full. |
| `pg_restore`: `cannot drop … other objects depend on it` / `being accessed by other users` | Live connections block `--clean`. `$COMPOSE stop bookmark-api` first (§6). If they persist: `$COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()"`. |
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
| `git pull`: `could not read Username for 'https://…'` | The checkout's remote is the HTTPS clone URL and the server has no credential helper, so the pull prompts into a closed stdin. Switch it to SSH once — `git remote set-url origin ssh://git@gitea.violetcrown.my.id:2222/sulthan/mangaBookmark.git`. Gitea's SSH listens on **2222**, not 22; port 22 is the host's own sshd and answers `Permission denied (publickey)` no matter which key is registered. |
| kagane rows stopped updating after a redeploy | Check `BROWSER_WS_URL` survived the `.env` edit and still names the home machine's tailnet **IP**. A hostname 500s at `/json/version`; an empty value disables the browser silently. Plain-TLS sites keep working either way, which is why this is easy to miss. |
| kagane covers went blank in the web UI | They should not — covers are rows in `covers`, not an in-process cache. `$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from covers'`. Zero after a restore means the dump predates the covers table; they refill on the next poll of each series. |
| Browser unit will not start: `set BROWSER_BIND_ADDR to this machine's tailnet IP` | `chrome/.env` is missing or the variable is empty. It has no default on purpose — an unset value must fail the deploy rather than publish an unauthenticated CDP port to the LAN. |
| `bookmark-browser` shows `OOMKilled true` | The 512 MiB cap did its job. Read `docker logs bookmark-browser` before raising it: the cap is sized against a measured 645 MiB untuned peak and exists so the kernel never takes the Gitea runner instead. |
Full first-time setup: `DEPLOY.md`. The one-off SQLite→Postgres move:
`CUTOVER.md`. Config reference and endpoints: `README.md`.
UI conventions: `docs/design-system.md`.