mangaBookmark/REDEPLOY.md

Redeploy runbook

Shipping new code to a server that is already running. First-time setup (DNS, .env, Traefik, installing the userscript) is DEPLOY.md — this file assumes all of that exists and picks up at "there is a running stack and I want it to run the new commit."

Whole thing is ~5 minutes, most of it waiting on docker build. Order matters: back up before you pull. A backup taken after a bad migration is a backup of the damage.

Paths below assume the checkout is at ~/mangaBookmark, which is where it lives on this deployment; substitute your own. The one absolute rule about paths: backups live in a -backups sibling of the checkout, never inside it. It sits outside the repo so git pull, git clean -fd and a bad rm -rf inside the checkout cannot take the backups with them.

~/
├── mangaBookmark/            <- the checkout (this repo)
└── mangaBookmark-backups/    <- bookmarks-YYYYmmdd-HHMMSS.dump

---

0. Preflight

cd ~/mangaBookmark

# Both -f flags, every time. The prod override is not standalone.
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"

$COMPOSE ps                        # bookmark-api should be Up
git status --short                 # expect empty
git log --oneline -1               # note this hash — it is your rollback target
df -h /var/lib/docker | tail -1    # a build needs room

If git status is dirty, someone edited files on the server. Decide before you pull: git stash to keep it, git checkout -- . to discard. A git pull onto a dirty tree fails halfway and leaves you in a worse spot than either.

Create the backup directory once, and make sure it is a sibling, not a child:

BACKUP_DIR="$(cd .. && pwd)/$(basename "$PWD")-backups"   # absolute — Docker needs it
mkdir -p "$BACKUP_DIR"
echo "$BACKUP_DIR"                        # -> /home/sulthan/mangaBookmark-backups

---

1. Back up the database

The database is Postgres, running as the postgres service on the named volume postgres-data. It has no published port — nothing outside the internal db network can reach it — so every command below goes in through the container:

$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
# -> bookmarks, covers, readers, schema_migrations, series, sessions

Inside the container that connects over the local socket as the bookmarks superuser, so no password is needed anywhere in this section. -T is not optional: without it Compose allocates a TTY, which rewrites \n to \r\n and silently corrupts any binary stream flowing back out — see the dump below.

Preferred: hot dump, no downtime

pg_dump runs in a single repeatable-read transaction, so it writes one point-in-time-consistent snapshot while the API keeps serving. No stopping, no WAL to worry about — that is the server's problem, not yours.

STAMP=$(date -u +%Y%m%d-%H%M%S)      # UTC, sorts chronologically as text
$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
  > "$BACKUP_DIR/bookmarks-$STAMP.dump"

ls -lh "$BACKUP_DIR"/bookmarks-$STAMP.dump

-Fc is the custom archive format rather than plain SQL: it is compressed, and pg_restore can inspect and replay it selectively — list its table of contents, restore one table, restore schema without data, reorder. A plain .sql dump can only be piped into psql whole, and gives you no way to check what is in it short of reading it.

$STAMP is the "time in the name" — bookmarks-20260730-014233.dump. UTC, so the files sort in real order and never collide across a DST shift.

Verify it before you trust it. An unreadable backup is worse than none, because you will act as though you have one:

# 1. The dump parses and contains the tables. Uses the same image compose
#    already pulls, so nothing new to install.
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
  pg_restore --list "/backup/bookmarks-$STAMP.dump" | grep 'TABLE DATA'
# -> 1234; 0 0 TABLE DATA public bookmarks bookmarks
# -> 1235; 0 0 TABLE DATA public covers bookmarks
# -> 1236; 0 0 TABLE DATA public readers bookmarks
# -> 1237; 0 0 TABLE DATA public schema_migrations bookmarks
# -> 1238; 0 0 TABLE DATA public series bookmarks
# -> 1239; 0 0 TABLE DATA public sessions bookmarks

# 2. Sanity-check the live row count you just captured.
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks \
  -c 'select count(*) from bookmarks'
# -> 37

A custom-format archive stores row counts nowhere, so step 1 proves the file is a readable archive with the right tables in it, not that the rows are there; step 2 is the number those rows should be. It should match what the web UI shows. Zero on a server you know has bookmarks means the API and your psql are looking at different databases — check DATABASE_URL.

Fallback: cold volume archive

Use this when you want the whole data directory rather than a logical dump — a like-for-like restore of the same Postgres major version onto the same host.

The stack must be stopped first. A running Postgres has dirty pages in shared buffers and WAL that has not been replayed into the data files, and tar walks the directory over several seconds while the server keeps writing to it. The archive you get is torn: files from different instants, possibly a half-written page. It may restore, start, and be quietly wrong. Online filesystem-level backup is pg_basebackup's job, not tar's; with the container stopped the shutdown checkpoint has already flushed everything and a plain archive of the volume is consistent.

# Derived exactly, not with a `--filter name=` substring match plus `head -1`:
# that quietly picks the first of however many volumes happen to contain the
# string, and archiving the wrong data directory is not a visible failure.
VOL="$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_postgres-data"
docker volume inspect "$VOL" >/dev/null && echo "$VOL"   # -> mangabookmark_postgres-data

$COMPOSE stop
docker run --rm -v "$VOL":/from:ro -v "$BACKUP_DIR":/to alpine \
  tar czf "/to/postgres-data-$STAMP.tgz" -C /from .
$COMPOSE start

ls -lh "$BACKUP_DIR"/postgres-data-$STAMP.tgz

Costs ~15 seconds of downtime. Read-only on the source is safe here precisely because nothing is running against it. Restoring this variant means untarring it back into an empty postgres-data volume with the stack down — it is a whole data directory, not a file you can drop next to the live one, and it will only start under postgres:17.

Retention

Keep a month, drop the rest — a bookmark database this small compresses the decision to "disk is free, but not infinite":

ls -1t "$BACKUP_DIR"/bookmarks-*.dump | tail -n +31 | xargs -r rm -v

A note on the old bookmarks-data volume

bookmarks-data is the pre-migration SQLite volume. It is deliberately not declared in docker-compose.yml any more, which is what keeps docker compose down -v from taking it with the rest of the stack. It is not the live database and nothing reads it — the one-way move out of it is CUTOVER.md. Once the Postgres data has been trusted for a while, remove it by hand — nothing else will. Its full name is <compose project>_bookmarks-data, and the project name is the lowercased directory name of the checkout:

docker volume rm "$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_bookmarks-data"

---

2. Pull the new code

git pull --ff-only
git log --oneline -3

--ff-only so a diverged history fails loudly instead of opening a merge you did not plan on the production box.

Check whether .env needs anything new. New config lands in .env.example, and compose fails at start for a missing required var — better to find out now:

git diff HEAD@{1} HEAD -- .env.example docker-compose.yml docker-compose.prod.yml

If a variable was added there, add it to .env before continuing.

---

3. Rebuild and restart

$COMPOSE up -d --build

A rebuild is mandatory for any UI change. The HTML templates, CSS, JavaScript and fonts are compiled into the binary by //go:embed, so editing them on the server — or pulling them — changes nothing until the image is rebuilt. The one exception is userscript/manga-bookmark.user.js, which is bindmounted read-only and read fresh per request.

$COMPOSE ps                        # bookmark-api Up; postgres Up (healthy)
docker logs bookmark-api --tail 20    # -> "listening on :8080 ..."

Nothing in the log about the database, the migrations or the poller failing. bookmark-api waits on postgres reporting healthy before it starts and the binary applies any pending migration before it listens, so an API that never says "listening" is usually the database, not the code — $COMPOSE logs postgres first. The image is tagged bookmarkmanager-backend:latest, so the previous image is still on disk untagged — that is what makes the rollback in §6 quick.

---

4. Verify the deploy

Same four API checks as DEPLOY.md §3, plus the web UI. Set the host names once:

API=https://bookmark-api.violetcrown.my.id
WEB=https://bookmark.violetcrown.my.id
# Your own Reader credential - derived, never stored in .env. Take it from the
# Userscripts panel's install link after signing in, or from an installed
# script's API_TOKEN constant.
TOKEN=<your Reader credential>

curl -s $API/healthz                                            # -> ok
curl -s -o /dev/null -w '%{http_code}\n' $API/bookmarks         # -> 401
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200
                                                                # -> your data, not []
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
  -H 'Access-Control-Request-Method: PUT' \
  $API/bookmarks/x | grep -i access-control                     # -> allow-origin echoed

[] from the third call is the alarm that matters: you are talking to an empty database, which means the API found a different Postgres than the one holding your data — a renamed project directory, a fresh postgres-data, or a DATABASE_URL override in .env pointing elsewhere. Stop and check, before touching anything else:

$COMPOSE config --volumes                                       # -> postgres-data
$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from bookmarks'

Web UI and its assets:

curl -s -o /dev/null -w '%{http_code}\n' $WEB/                  # -> 200 (login page)
curl -s -o /dev/null -w '%{http_code}\n' $WEB/static/style.css  # -> 200

# The fonts are self-hosted; this is what tells you they shipped and are typed.
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
  $WEB/static/fonts/instrument-serif-400-latin.woff2
# -> 200 font/woff2

# The served userscript, whose @version is its mtime — a pull bumps it.
curl -s $API/u/$TOKEN/manga-bookmark.user.js | grep '@version'

404 on the font means static/fonts/ did not make it into the image; the UI will still render, in Georgia, which is easy to miss on a phone. application/ octet-stream instead of font/woff2 means an older binary is running.

Then open $WEB in a browser and confirm, in one glance:

• Serif brand and serif row titles — not the system sans fallback.
• A series with an unread chapter has a crimson title on an ember underline; everything else is cool grey. That single detail exercises the whole design path (template class, CSS, and the poller's latest_chapter).
• Tapping the pencil opens the chapter form; Save closes it and the row keeps its place in the list.

---

5. Smoke-test the full loop

The API answering is not the same as the product working. Do this on the phone, against the real sites — it is the only check that covers the userscript, CORS and the shared store together:

1. Open a series on asurascans.com, open the panel, + Bookmark this.
2. On the server: curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks → the series is in the JSON.
3. Open a chapter of it, reopen the panel → last-read shows that chapter, and opening an older chapter does not move it backwards.
4. Open demonicscans.org, open the panel → the Asura bookmark is listed there too. Different origin, one store — this is the whole point of the backend, and the check that fails first when CORS or TLS regressed.
5. Open $WEB → the same series appears, with the same chapter.
6. Turn airplane mode on, tap ★ on a row, turn it off, reopen the panel → the star stuck. That exercises the retry queue.

If 1–5 pass, the deploy is good.

---

6. Rollback

Two independent things can be wrong, so undo only what broke.

Bad code, database fine — go back to the previous commit and rebuild:

git log --oneline -5
git checkout <previous-hash>
$COMPOSE up -d --build

Database damaged — restore the dump from §1. Stop only the API, not the whole stack: pg_restore needs the server up to restore into, and it needs bookmark-api's connection pool gone, because --clean cannot drop a table other sessions are holding open.

$COMPOSE stop bookmark-api

$COMPOSE exec -T postgres pg_restore -U bookmarks -d bookmarks --clean --if-exists \
  < "$BACKUP_DIR/bookmarks-<STAMP>.dump"

$COMPOSE start bookmark-api
docker logs bookmark-api --tail 20
curl -s -H "Authorization: Bearer $TOKEN" $API/bookmarks | head -c 200

Three things here are easy to skip and all three bite:

• --clean --if-exists. Without --clean the dump's rows land on top of what is already there and you get primary-key collisions half way through, a partially restored database, and a non-zero exit you may not notice. --if-exists only suppresses the "does not exist" noise when the target is already empty; it is not the part doing the work.
• -T again. Feeding a custom-format archive into a TTY-allocated exec corrupts it in flight and pg_restore fails with a garbled-header error on a file that is perfectly fine on disk.
• Stop the API, not Postgres. $COMPOSE stop (everything) leaves you with nothing to restore into; leaving bookmark-api running leaves connections that block the drops and lets the poller write into a half-restored table.

No ownership fixing is needed any more — the Postgres image owns postgres-data itself and pg_restore writes through the server, not the filesystem. schema_migrations is inside the dump, so the database comes back at whatever schema version the backup was taken at; the migration runner applies anything newer the next time bookmark-api starts.

---

7. The whole thing, as one block

For a routine redeploy where nothing needs deciding:

cd /opt/bookmarkmanager
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
BACKUP_DIR="$(cd .. && pwd)/bookmarkmanager-backups"; mkdir -p "$BACKUP_DIR"
STAMP=$(date -u +%Y%m%d-%H%M%S)

$COMPOSE exec -T postgres pg_dump -U bookmarks -d bookmarks -Fc \
  > "$BACKUP_DIR/bookmarks-$STAMP.dump" &&
docker run --rm -v "$BACKUP_DIR":/backup postgres:17-alpine \
  pg_restore --list "/backup/bookmarks-$STAMP.dump" > /dev/null &&
git pull --ff-only &&
$COMPOSE up -d --build &&
sleep 5 &&
curl -sf https://bookmark-api.violetcrown.my.id/healthz && echo " deploy ok"

The && chain is deliberate: if the dump or its pg_restore --list check fails, nothing is pulled and nothing is rebuilt. A failed dump still leaves a short or empty .dump behind — the shell creates the file before pg_dump runs — so delete it rather than letting it sit in the backup directory looking like a backup. Then still do §5 by hand — no shell command can tell you the panel works on the phone.

---

8. The browser unit (separate machine, separate cadence)

Everything above is the API stack on the VPS. The headless browser is its own compose unit on the home machine (ADR-0006, DEPLOY.md §7) and is redeployed on its own schedule — it holds no data you can lose, so there is nothing to back up and no ordering constraint against the API.

cd ~/mangaBookmark/chrome
git pull --ff-only
docker compose up -d --build

Then confirm it answers, and that a stopped-and-restarted Chrome is invisible to the API:

curl -s -m 15 http://$(tailscale ip -4):9222/json/version | head -c 120
# -> {"Browser":"Chrome/1xx...","webSocketDebuggerUrl":"ws://...<new uuid>"}

The first call takes a few seconds: Chrome is not running until something connects, and it is reaped again after five idle minutes. The debugger UUID changes on every start and the API does not care — chromedp re-runs /json/version discovery per fetch, which is exactly why chromedp.NoModifyURL must never be added to browser.go.

Rebuild is the Chrome upgrade path. The image installs google-chrome-stable unpinned on purpose: a stale browser is what Cloudflare turns away, and the pinned Chrome 124 in zenika/alpine-chrome is the worked example. The chrome-profile volume survives --build, so clearance cookies are reused rather than re-solved.

Two things worth a glance after several days, both from the acceptance criteria of the move:

docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'
# -> 0 false
free -m                     # the Gitea runner should still have its headroom

Nothing here needs doing during an API redeploy. The API stack does not depends_on the browser, and an unreachable one degrades exactly as an unset BROWSER_WS_URL: plain-TLS libraries unaffected, kagane and novelfull logged and skipped, stored covers still served.

---

Troubleshooting

Symptom	Cause / fix
/bookmarks returns [] after redeploy	You are on an empty Postgres. Check $COMPOSE config --volumes lists postgres-data, that you passed both -f files, and that .env has no stray DATABASE_URL override. Do not re-bookmark; the data is still in the volume.
UI looks like plain Georgia / system sans	static/fonts/ missing from the image, or the browser cached an old style.css. /static/* is served max-age=3600, so hard-reload or wait an hour.
CSS or template change did not appear	You restarted without --build. Assets are //go:embeded.
Font answers application/octet-stream	Old binary — the .woff2 MIME registration is in web.go. Rebuild.
Everyone logged out of the web UI	The sessions table was wiped; sessions are database rows, not signed cookies. Expected after a deliberate revoke.
compose errors about BOOKMARK_WEB_HOST	Run from the directory holding .env. Both host vars are required even when the web UI is unused.
Userscript did not update on the phone	Violentmonkey polls on its own schedule; force a check. @version comes from the file's mtime, so confirm the pull actually touched it.
bookmark-api crash-loops, log says password authentication failed for user "bookmarks"	POSTGRES_PASSWORD in .env no longer matches the one burned into postgres-data at first init — Postgres reads that variable only when initialising an empty volume. Put the old value back, or reset the role: $COMPOSE exec postgres psql -U bookmarks -d bookmarks -c '\password bookmarks' (prompts, so nothing lands in shell history) and then match .env to it.
compose errors set POSTGRES_PASSWORD in .env	Unset. Compose builds the backend's DATABASE_URL out of it, so it is required even though you never write that URL yourself. Run from the directory holding .env.
postgres never leaves starting; bookmark-api never starts either	The healthcheck (pg_isready) is failing and bookmark-api waits on it. $COMPOSE logs postgres — usually postgres-data was initialised by a different major version ("database files are incompatible with server"), or the disk is full.
pg_restore: cannot drop … other objects depend on it / being accessed by other users	Live connections block --clean. $COMPOSE stop bookmark-api first (§6). If they persist: $COMPOSE exec -T postgres psql -U bookmarks -d postgres -c "select pg_terminate_backend(pid) from pg_stat_activity where datname='bookmarks' and pid <> pg_backend_pid()".
Dump is 0 bytes, or pg_restore: did not find magic string in file header	You ran exec without -T. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1).
git pull: could not read Username for 'https://…'	The checkout's remote is the HTTPS clone URL and the server has no credential helper, so the pull prompts into a closed stdin. Switch it to SSH once — git remote set-url origin ssh://git@gitea.violetcrown.my.id:2222/sulthan/mangaBookmark.git. Gitea's SSH listens on 2222, not 22; port 22 is the host's own sshd and answers Permission denied (publickey) no matter which key is registered.
kagane rows stopped updating after a redeploy	Check BROWSER_WS_URL survived the .env edit and still names the home machine's tailnet IP. A hostname 500s at /json/version; an empty value disables the browser silently. Plain-TLS sites keep working either way, which is why this is easy to miss.
kagane covers went blank in the web UI	They should not — covers are rows in covers, not an in-process cache. $COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from covers'. Zero after a restore means the dump predates the covers table; they refill on the next poll of each series.
Browser unit will not start: set BROWSER_BIND_ADDR to this machine's tailnet IP	chrome/.env is missing or the variable is empty. It has no default on purpose — an unset value must fail the deploy rather than publish an unauthenticated CDP port to the LAN.
bookmark-browser shows OOMKilled true	The cap did its job. Read docker logs bookmark-browser before raising it — the sizing and what the cap protects are in ADR-0006.

Full first-time setup: DEPLOY.md. The one-off SQLite→Postgres move: CUTOVER.md. Config reference and endpoints: README.md. UI conventions: docs/design-system.md.