mangaBookmark/DEPLOY.md

# Deployment

Step-by-step for the backend (Docker + Traefik) and the Bromite userscript.
Assumes you already run Traefik in Docker with a working HTTPS entrypoint and an
ACME/cert resolver, and control a domain.

---

## 0. Prerequisites

- Docker + Docker Compose on the server.
- A Traefik instance watching a Docker network (default name assumed: `proxy`).
- DNS: an `A`/`AAAA` record for `bookmark-api.<yourdomain>` pointing at the server.
- The repo copied to the server, e.g. `~/mangaBookmark/` (needs `backend/`,
  `docker-compose.yml`, `docker-compose.prod.yml`, `.env.example`).

Confirm the Traefik network exists (create if not):

```bash
docker network ls | grep proxy || docker network create proxy
```

---

## 1. Configure `.env`

```bash
cd ~/mangaBookmark
cp .env.example .env
```

Edit `.env`:

```ini
# Required — secret every Reader's userscript credential is derived from.
# Only SHA-256 hashes of credentials are stored.
TOKEN_KEY=<paste output of: openssl rand -hex 32>

# Required — the owner's Discord user ID. Seeds the first Reader: the
# administrator, and the owner of every bookmark that predates registration.
# The value is the snowflake in your Discord profile (Settings →
# Advanced → Developer Mode → right-click your name → Copy User ID).
OWNER_DISCORD_ID=<discord user id>

# CORS allowlist — leave as-is unless a site changes hostname.
ALLOWED_ORIGINS=https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net

# Required — password for the bundled Postgres container. Compose builds the
# backend's DATABASE_URL out of it and has no fallback for either.
POSTGRES_PASSWORD=<paste output of: openssl rand -hex 24>

# Leave unset. Only set this to point the backend at a Postgres compose does
# not run; it then replaces the URL built from POSTGRES_PASSWORD above.
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require

# Required path inside bookmark-api. Compose builds the image and mounts the
# named cover-data volume at this path.
COVER_DIR=/covers

# Required — the origin this deployment answers on, no trailing slash. Cover
# URLs on the wire are absolute, because the userscript renders them on a
# Site's own origin (ADR-0007). Same host as BOOKMARK_API_HOST below.
PUBLIC_BASE_URL=https://bookmark-api.violetcrown.my.id

# Required for the Traefik override. Both have no fallback — compose refuses
# to start without them. BOOKMARK_WEB_HOST is required even if the web UI
# were unused; see 1b.
BOOKMARK_API_HOST=bookmark-api.violetcrown.my.id
BOOKMARK_WEB_HOST=bookmark.violetcrown.my.id

# Only if your Traefik setup differs from these defaults:
# PROXY_NETWORK=proxy
# TRAEFIK_ENTRYPOINT=websecure
# TRAEFIK_CERTRESOLVER=le
```

Generate + insert the two secrets in three lines:

```bash
sed -i "s|^TOKEN_KEY=.*|TOKEN_KEY=$(openssl rand -hex 32)|" .env
sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$(openssl rand -hex 24)|" .env
grep -E '^TOKEN_KEY=' .env
```

`TOKEN_KEY` derives every Reader's userscript credential (issue #24); only
SHA-256 hashes of the credentials are stored, so this secret is what a
database leak alone cannot recover. Changing it invalidates every installed
script at once.

`POSTGRES_PASSWORD` is read **only while the `postgres-data` volume is empty**,
which in practice means at first boot. Changing it afterwards changes the URL
the backend dials but not the password the database expects, and `bookmark-api`
crash-loops on `password authentication failed`. Set it before §2 and leave it
alone.

An `.env` written before issue #100 carries the old poll-pace names
(`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
`_STAGGER`). All five are dead configuration now — the pace lives in the Site
registry (`backend/internal/latest/sites.go`), so **delete those lines** and
keep only the kill switch `LATEST_CHAPTER_POLL_ENABLED`. Leaving them behind
is harmless (nothing reads them) but silently misleads the next person who
edits the file.

> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
> names (check your Traefik static config — common alternatives: `https`,
> `myresolver`, `cloudflare`). Wrong names = no certificate issued.

---

## 1b. Web UI

The browser UI is served by the same container on a second hostname. Sign-in
is a Discord authorization code grant (ADR-0002): the owner's Discord account,
gated by membership in one configured guild.

1. Add a DNS `A`/`AAAA` record for `bookmark.<yourdomain>` pointing at the
   server — the same address as `bookmark-api.<yourdomain>`.

2. Create the Discord application at <https://discord.com/developers/applications>:
   - **OAuth2 → Redirects:** add the exact callback URL
     `https://bookmark.violetcrown.my.id/auth/discord/callback`. Discord
     matches it verbatim — a trailing slash or different hostname breaks
     sign-in.
   - **OAuth2 → General:** note the Client ID, and generate a Client Secret.
   - No scopes or bot setup are needed in the dashboard; the service requests
     `identify` and `guilds.members.read` itself, and checks the *user's*
     membership of the guild, not the application's.

3. Set the variables in `.env`:

   ```ini
   BOOKMARK_WEB_HOST=bookmark.violetcrown.my.id
   DISCORD_CLIENT_ID=<client id>
   DISCORD_CLIENT_SECRET=<client secret>
   DISCORD_GUILD_ID=<guild snowflake>
   DISCORD_REDIRECT_URI=https://bookmark.violetcrown.my.id/auth/discord/callback
   # Optional: only members holding this role may sign in.
   # DISCORD_REQUIRED_ROLE=<role snowflake>
   ```

   The guild id is in Discord's client with Developer Mode on: right-click the
   server name → Copy Server ID. The four uncommented variables are required —
   the backend refuses to start without them. Guild membership *is*
   registration: any member of `DISCORD_GUILD_ID` becomes a Reader with their
   own library on their first sign-in. `OWNER_DISCORD_ID` from §1 is only the
   administrator — the Reader who can revoke another Reader's sessions.

4. Redeploy and check:

   ```bash
   docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
   curl -s -o /dev/null -w '%{http_code}\n' https://bookmark.violetcrown.my.id/
   ```

   Expected `200`, serving the login page with the Discord button. Signing in
   lands on the library; an account outside the guild is refused with a message
   that names neither the guild nor its id.

Sessions are rows in the database: the cookie carries only an opaque id, and
every request looks the row up and checks its expiry. Deleting a session row —
or the whole `sessions` table — logs the browser out immediately; nothing is
signed, so rotating a credential does not affect browser sessions. Sessions
last 60 days.

---

## 2. Build + start

```bash
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
```

This merges the base file (build/image/env/volume) with the prod override
(no host port, Traefik network + router labels). Always pass **both** `-f`
flags — the prod file is not standalone.

Two services come up: `bookmark-api` (the backend) and `postgres` (its
database, `postgres:17-alpine`). Postgres publishes no port — it sits alone
with `bookmark-api` on an `internal: true` network — and stops everything if it
is missing: `bookmark-api` waits for `pg_isready` to pass, then applies its
embedded migrations, and only then listens. The schema is created that way;
there is nothing to import by hand.

There is deliberately no browser here. Kagane and novelfull need one, and it
runs on a **separate machine** over the tailnet — §7. Until you do that step,
`BROWSER_WS_URL` is unset, the poller logs and skips those two sites, and
everything else works normally.

Check it's up and healthy:

```bash
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
# bookmark-api Up; postgres Up (healthy)
docker logs bookmark-api --tail 20   # expect: "listening on :8080 ..."
```

---

## 3. Verify over HTTPS

Give Traefik a few seconds to issue the cert, then:

```bash
# Health (no auth) — must be valid TLS, no cert warning.
curl -s https://bookmark-api.violetcrown.my.id/healthz          # -> ok

# Auth enforced.
curl -s -o /dev/null -w '%{http_code}\n' \
  https://bookmark-api.violetcrown.my.id/bookmarks              # -> 401

# A Reader's own credential. It is derived, never stored in .env — take it from
# the Userscripts panel's install link after signing in, or from an installed
# script's API_TOKEN constant.
TOKEN=<your Reader credential>
curl -s -H "Authorization: Bearer $TOKEN" \
  https://bookmark-api.violetcrown.my.id/bookmarks              # -> []

# CORS preflight from a real site origin.
curl -s -i -X OPTIONS \
  -H 'Origin: https://asurascans.com' \
  -H 'Access-Control-Request-Method: PUT' \
  https://bookmark-api.violetcrown.my.id/bookmarks/x | grep -i access-control
# -> Access-Control-Allow-Origin: https://asurascans.com  (+ Methods/Headers)
```

All four must pass. Valid TLS is non-negotiable — the manga sites are HTTPS, so
a bad cert makes the browser block the userscript's `fetch()` (mixed content).

---

## 4. Configure the userscript

The bindmounted `userscript/*.user.js` files carry `__API_TOKEN__` placeholders
and the deployment's `@downloadURL`/`@updateURL` lines. Check the metadata
block — it ships hardcoded to this deployment's domain, so a deployer who
copies the repo to another domain must edit the two lines or the script
auto-updates from someone else's backend:

```js
// @downloadURL  https://bookmark-api.yourdomain.com/u/__API_TOKEN__/manga-bookmark.user.js
// @updateURL    https://bookmark-api.yourdomain.com/u/__API_TOKEN__/manga-bookmark.user.js
```

The backend substitutes `__API_TOKEN__` with the requesting Reader's derived
credential at serve time (issue #24), so no real credential ever sits in the
file. Only the `API_BASE` constant and the metadata hostname are deployer
edits; do not put a credential in this file.

---

## 5. Install on Bromite

1. Bromite → **Settings → User scripts** → enable (accept the permission prompt).
2. Sign in to the web UI, open the **Userscripts** panel, and open the install
   link — Bromite detects `.user.js` and offers to install. The script already
   carries your credential; you never see or type one.
3. Confirm install — the `@match` list covers both sites.
4. Open a series on asurascans.com or demonicscans.org → a 📑 button appears
   bottom-right → tap → **+ Bookmark this**.

Optional desktop test: the script is `GM_*`-free, so the same file installs in
Tampermonkey/Violentmonkey for quick checks before going mobile.

Rotating the credential in the same web-UI panel invalidates every installed
copy immediately — reinstall on all devices, or they silently stop syncing.

---

## 6. Smoke-test the full loop

1. Bookmark a series on Asura.
2. `curl -s -H "Authorization: Bearer $TOKEN" https://bookmark-api.yourdomain.com/bookmarks`
   on the server — the series should appear.
3. Open a chapter of that series — reopen the panel; last-read updates to that
   chapter (auto, never regresses on older chapters).
4. Open Demonic, open the panel — the Asura bookmark shows there too (shared
   store, cross-site unified list).

---

## 7. The browser, on the home machine

Kagane and novelfull sit behind a Cloudflare JavaScript challenge no TLS
fingerprint clears, so the poller reaches them through a real Chrome over CDP.
That browser does **not** run on the VPS: it held 471 MiB of a 1974 MiB box
with no swap, and a residential IP avoids the cloud-hosting-IP signature Bot
Fight Mode challenges anyway (ADR-0006). It
is its own compose unit, deployed and updated independently of everything
above.

Do this after §2, on the second machine. Both machines must already be on the
same tailnet.

First, on the VPS, record what you are reclaiming — this is the whole point of
the move and there is no way to measure it afterwards:

```bash
free -m | awk '/^Mem:/ {print "available before:", $NF, "MiB"}'
```

Take it again after §7 is finished and the old sidecar is gone. Expect roughly
the sidecar's former footprint back (measured at 471 MiB working set, 595 MiB
cgroup).

**On the home machine:**

```bash
git clone <this repo> ~/mangaBookmark && cd ~/mangaBookmark/chrome

tailscale ip -4                    # -> 100.x.y.z, this machine's tailnet IP
cp .env.example .env
echo "BROWSER_BIND_ADDR=$(tailscale ip -4)" >> .env

docker compose up -d --build
```

The clone is only for `chrome/`; nothing else on this machine reads the rest of
the repo. The unit is its own compose project (`bookmark-browser`), so it shares
no volume, network or lifecycle with an API stack that happens to sit beside it.

`BROWSER_BIND_ADDR` has no default on purpose. CDP authenticates nothing —
whatever reaches port 9222 drives the browser and, through it, this host — so
the bind address *is* the access control, backed by Tailscale device identity.
On the VPS that job was done by Docker network membership; this machine has a
real LAN, so `0.0.0.0` would be a hole punched into your home network. Compose
refuses to start rather than guess.

**Narrow it to the one device that needs it.** The bind address keeps CDP off
your LAN; it still leaves port 9222 open to every device on the tailnet, and
CDP has no login — a compromised phone is enough to drive this host. A new
tailnet's policy is allow-all, so this is the step that makes "Tailscale
identity is the access control" true rather than aspirational.

Tailscale has no `deny`, so a restriction is expressed by removing the blanket
grant and enumerating what is left. That only works if the browser machine can
be *excluded* from a selector that still covers your own devices — which is
what tagging buys: a tagged device has no user, so `autogroup:member` and
`autogroup:self` stop matching it. Tagging is the mechanism, not decoration.

In the admin console, under **Access controls**, the shipped policy grants
`{"src": ["*"], "dst": ["*"], "ip": ["*"]}`. Replace it:

```jsonc
{
  "tagOwners": {
    // Empty list: implicitly owned by the tailnet Owner/Admins, which is you.
    "tag:bookmark-api":     [],
    "tag:bookmark-browser": [],
  },

  "grants": [
    // The only thing on the tailnet that may drive the browser.
    {
      "src": ["tag:bookmark-api"],
      "dst": ["tag:bookmark-browser"],
      "ip":  ["tcp:9222"],
    },
    // Your own devices reach your own devices, and the VPS, in full.
    {
      "src": ["autogroup:member"],
      "dst": ["autogroup:self", "tag:bookmark-api"],
      "ip":  ["*"],
    },
    // On the browser machine you get SSH and nothing else. Widen this to `*`
    // and the restriction above is void; delete it and you are locked out.
    {
      "src": ["autogroup:member"],
      "dst": ["tag:bookmark-browser"],
      "ip":  ["tcp:22"],
    },
    // Uncomment if you route traffic through an exit node — dropping the
    // blanket grant takes exit-node access with it.
    // {"src": ["autogroup:member"], "dst": ["autogroup:internet"], "ip": ["*"]},
  ],

  // Tagged devices left `autogroup:self`, so Tailscale SSH needs them named.
  // Irrelevant if you reach these boxes with ordinary sshd over the tailnet —
  // that is the `tcp:22` grant above.
  "ssh": [
    {
      "action": "check",
      "src":    ["autogroup:member"],
      "dst":    ["autogroup:self", "tag:bookmark-api", "tag:bookmark-browser"],
      "users":  ["autogroup:nonroot", "root"],
    },
  ],

  // Run on every save, so a later edit that reopens 9222 is rejected outright.
  "tests": [
    { "src": "tag:bookmark-api", "accept": ["tag:bookmark-browser:9222"] },
    {
      "src":    "you@example.com",
      "accept": ["tag:bookmark-browser:22"],
      "deny":   ["tag:bookmark-browser:9222"],
    },
  ],
}
```

Then apply the tags — on the VPS and the home machine respectively:

```bash
sudo tailscale up --advertise-tags=tag:bookmark-api
sudo tailscale up --advertise-tags=tag:bookmark-browser
```

Each re-authenticates in a browser and issues a new node key; the tailnet IP is
unchanged, so `BROWSER_WS_URL` and `BROWSER_BIND_ADDR` still hold. Key expiry is
disabled once a device is tagged, which is what you want for a server — an
expired key would otherwise take the poller down every few months.

**Tagging replaces the device's user identity**, so do this only to machines
that exist to run these services. If your "home machine" is also your daily
driver, tag it anyway and reach it through the `:22` rule above, or skip the
tag and accept that any device of yours can reach CDP.

Enforcement is by the destination's packet filter, so the check below is real,
not advisory.

Prove the bind is tight, from the home machine itself:

```bash
curl -s -m 3 http://$(tailscale ip -4):9222/json/version   # -> JSON
curl -s -m 3 http://<this machine's LAN IP>:9222/json/version
# -> curl: (7) Failed to connect ... Connection refused
```

The first call is also what wakes Chrome: it is not running until something
connects, and it is reaped again after five idle minutes. A cold first response
takes a few seconds; that is the browser starting, not a fault.

That check proves the *bind*, not the ACL — traffic that starts on the node is
not filtered. Prove the ACL from somewhere else: on your laptop or phone the
same URL must now time out, and from the VPS it must answer.

```bash
# on any other device of yours -> hangs until timeout
curl -s -m 5 http://<home machine tailnet IP>:9222/json/version
# on the VPS -> JSON
curl -s -m 20 http://<home machine tailnet IP>:9222/json/version
```

**On the VPS:**

```bash
cd ~/mangaBookmark
echo 'BROWSER_WS_URL=ws://100.x.y.z:9222' >> .env      # the home machine's tailnet IP
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```

It must be the tailnet **IP**. A MagicDNS hostname fails: Chrome's DevTools HTTP
handler answers `/json/version` with a 500 for any `Host` header that is not an
IP or `localhost`, and the failure looks like a broken site rather than a broken
hostname.

**Prove it end to end.** This is the only check that says the challenge actually
clears from that machine's egress — it fetches a real kagane cover and a real
chapter list:

```bash
cd backend
SMOKE_BROWSER_WS_URL=ws://100.x.y.z:9222 go test -run TestSmokeKagane ./internal/latest
```

A red run means "not clearing from this address right now", which is a live
fact to re-check before it is a defect — a Site's Cloudflare settings, and the
fingerprint this Chrome presents after an update, both move. Then, from
the web UI, open a bookmarked kagane series and confirm the cover renders. Once
a cover is stored it is served from Postgres forever after, so the browser being
asleep, unreachable, or mid-power-outage costs chapter freshness and nothing
visible.

Finally, take the VPS `free -m` reading again and compare it against the one
from the top of this section.

**Updating the browser** is independent of the API stack and has its own
runbook — `REDEPLOY.md` §8.

---

## Updating

Pull new code, then rebuild:

```bash
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
```

Data persists in the named volume `postgres-data` across rebuilds. (If this
server predates the Postgres migration, the old SQLite volume `bookmarks-data`
is still on disk and deliberately undeclared in compose so `down -v` cannot take
it; see `REDEPLOY.md` §1 for when to remove it.)

The browser is a separate unit on a separate machine with its own update
command — §7. Nothing above touches it, and it needs no coordination: the API
picks up a restarted Chrome's new debugger UUID by itself.

---

## Troubleshooting

| Symptom | Likely cause / fix |
|---------|--------------------|
| No cert / TLS error at the domain | `TRAEFIK_ENTRYPOINT` or `TRAEFIK_CERTRESOLVER` name wrong; or DNS not resolving yet. Check `docker logs <traefik>`. |
| 404 from Traefik | Service not on the `proxy` network, or `BOOKMARK_API_HOST` mismatch. Confirm `docker network inspect proxy` lists `bookmark-api`. |
| `fetch` fails in the userscript, `curl` works | Origin missing from `ALLOWED_ORIGINS`, or mixed content (backend not HTTPS). |
| 401 with the right credential | The script's credential no longer matches the stored hash — most likely a rotation happened and the device was not reinstalled. Reinstall from the web UI. |
| 401 after rotation, even right after reinstalling | `TOKEN_KEY` changed between the rotation and the reinstall; credentials are derived from it, so changing it invalidates every credential. Keep it stable. |
| Panel button absent | URL didn't match an adapter, or user scripts disabled in Bromite. |
| `compose ... config` errors about `TOKEN_KEY`, `OWNER_DISCORD_ID` or `POSTGRES_PASSWORD` | Run compose from the dir with `.env`, or export the vars. All three are required and none has a fallback. |
| `bookmark-api` restarts in a loop, `password authentication failed for user "bookmarks"` | `POSTGRES_PASSWORD` was changed after first boot; Postgres only applies it to an empty `postgres-data`. Restore the old value, or reset the role (`REDEPLOY.md` troubleshooting). |
| `bookmark-api` never logs `listening on :8080` | It is blocked on `postgres` passing `pg_isready`, or a migration failed. `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs postgres`. |
| kagane rows never get a `latest_chapter`; log says `browser fetcher disabled` or nothing at all | `BROWSER_WS_URL` unset. Expected before §7 is done. |
| kagane polls all fail; log shows a 500 from `/json/version` | `BROWSER_WS_URL` names a MagicDNS hostname (or any name). Chrome's DevTools handler only accepts an IP or `localhost` — use the tailnet IP. |
| kagane polls fail with a connection error | Home machine off, off the tailnet, or the unit is down. `tailscale ping <machine>`, then `docker compose ps` in its `chrome/`. Costs freshness only; stored covers keep serving. |
| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series. |
| `compose` in `chrome/` errors `set BROWSER_BIND_ADDR to this machine's tailnet IP` | No `chrome/.env`, or the variable is empty. Deliberate — it has no default so an unset value cannot publish CDP to the LAN. |
| browser container restarts, or is OOM-killed | `docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'`. The 512 MiB cap is sized against a measured 645 MiB untuned peak; a real breach is a Chrome regression worth reading `docker logs` for, not a number to raise reflexively. |

Backend config reference and endpoint list: see `README.md`.

---

## Installing / updating the userscript

The backend serves the script itself, so Violentmonkey can auto-update it.
Complements §4 above — the `@downloadURL`/`@updateURL` lines point at the
credential-bearing path, so auto-updates come from the same place as the
install.

Install once, on the phone (Cromite + Violentmonkey): sign in to the web UI,
open the **Userscripts** panel, and open the install link for the library —
the script is served with your credential already inside it. Its
`@downloadURL`/`@updateURL` point at the same credential-bearing path for
updates:

```
https://bookmark-api.<your-domain>/u/<your credential>/manga-bookmark.user.js
```

Violentmonkey offers to install it. The credential is in the path because
Violentmonkey's update poll sends no `Authorization` header, and the script
embeds the credential in plain text — an open URL would leak it. A wrong
credential answers 404. The credential is derived from `TOKEN_KEY` and never
appears anywhere but this URL and the rendered script.

Updating, without a redeploy:

```bash
vi userscript/manga-bookmark.user.js   # on the VPS, in this checkout
```

`./userscript` is bindmounted read-only into the container and read fresh on
every request, so the edit is live immediately. The served `@version` is derived
from the file's mtime (`YYYY.MM.DD.HHMM`, UTC), not from the `@version` in the
file, so any edit outranks the installed copy and Violentmonkey pulls it on its
next check. The `@version` in the repo is a human marker only.

Updating via redeploy: `git pull` overwrites the file with the committed
version, which is the intended behaviour — a deploy always ships the repo's
script. Note that `git pull` sets mtime to checkout time, so even a rollback
serves a *higher* version and is adopted.

If the mount is missing, the endpoint answers 404 and logs it; bookmark sync is
unaffected.