92eba07da7
Closes #59. Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55. ## What changed A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s. ### Store - Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on. - `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there. - `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address. - The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path. - `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load. ### Acquisition - `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position). - Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them. - Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid. - Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later. - Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62. ### Wire and route - `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk. - `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep. - `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading. ### Config `PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`. ## Acceptance criteria All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence. ## Verification - `go test ./...` green (Docker-backed Postgres suite). - Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type. - Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red. ## Reviewed Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail. ## Known sequencing A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly. Reviewed-on: #68 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
138 lines
7.2 KiB
YAML
138 lines
7.2 KiB
YAML
# Base stack — works standalone for local smoke testing (`docker compose up`).
|
|
# The service binds 127.0.0.1:8080; a host reverse proxy (nginx/Caddy/Traefik)
|
|
# terminates TLS for bookmark-api.<domain> and forwards to it.
|
|
#
|
|
# If your proxy runs in Docker on its own network, use the prod override which
|
|
# attaches to that network instead of publishing a port:
|
|
# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
|
|
#
|
|
# The browser is not here. It is its own unit on the home machine —
|
|
# chrome/docker-compose.yml — reached over the tailnet via BROWSER_WS_URL.
|
|
|
|
services:
|
|
bookmark-api:
|
|
build:
|
|
context: ./backend
|
|
args:
|
|
COVER_DIR: ${COVER_DIR:?set COVER_DIR in .env}
|
|
image: bookmarkmanager-backend:latest
|
|
container_name: bookmark-api
|
|
restart: unless-stopped
|
|
environment:
|
|
# TOKEN_KEY derives every Reader's userscript credential (issue #24) —
|
|
# compose refuses to start without it.
|
|
TOKEN_KEY: ${TOKEN_KEY:?set TOKEN_KEY in .env}
|
|
# Owner's Discord user ID — required. Seeds the owner Reader (the
|
|
# administrator); every other Reader registers on their first login.
|
|
OWNER_DISCORD_ID: ${OWNER_DISCORD_ID:?set OWNER_DISCORD_ID in .env}
|
|
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
|
|
# The bookmarks database. Host is the compose service name; the password
|
|
# comes from .env so it is never committed.
|
|
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
|
|
# Required path inside the API. The build seeds ownership at this path
|
|
# and the named volume below mounts there.
|
|
COVER_DIR: ${COVER_DIR:?set COVER_DIR in .env}
|
|
# Public origin of this deployment, no trailing slash. Required: the
|
|
# Cover URLs on the wire are absolute, since the userscript renders them
|
|
# on a Site's origin rather than ours (ADR-0007).
|
|
PUBLIC_BASE_URL: ${PUBLIC_BASE_URL:?set PUBLIC_BASE_URL in .env}
|
|
PORT: "8080"
|
|
# Log timestamps only. Go's `log` stamps lines in local time, and this
|
|
# service has no other use for a zone: bookmark timestamps are unix ms
|
|
# and the two real time columns are timestamptz, both absolute instants.
|
|
# Purely so these lines read on the same clock as the browser's. Named
|
|
# API_TZ rather than TZ so an operator's exported shell TZ cannot leak
|
|
# in; distroless already carries tzdata, so the name just resolves.
|
|
TZ: ${API_TZ:-Asia/Jakarta}
|
|
# Discord OAuth for the browser UI (ADR-0002). The first four are
|
|
# required; DISCORD_REQUIRED_ROLE is optional and empty by default.
|
|
# Guild membership is the whole gate: any member becomes a Reader.
|
|
DISCORD_CLIENT_ID: ${DISCORD_CLIENT_ID:?set DISCORD_CLIENT_ID in .env}
|
|
DISCORD_CLIENT_SECRET: ${DISCORD_CLIENT_SECRET:?set DISCORD_CLIENT_SECRET in .env}
|
|
DISCORD_GUILD_ID: ${DISCORD_GUILD_ID:?set DISCORD_GUILD_ID in .env}
|
|
DISCORD_REQUIRED_ROLE: ${DISCORD_REQUIRED_ROLE:-}
|
|
DISCORD_API_BASE: ${DISCORD_API_BASE:-https://discord.com/api/v10}
|
|
DISCORD_REDIRECT_URI: ${DISCORD_REDIRECT_URI:?set DISCORD_REDIRECT_URI in .env}
|
|
# Path inside the container; matches the bindmount above.
|
|
USERSCRIPT_PATH: ${USERSCRIPT_PATH:-/userscript/manga-bookmark.user.js}
|
|
# Second script from the same bindmount; the novel library is a separate
|
|
# Violentmonkey install.
|
|
NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js}
|
|
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill
|
|
# switch; it only takes effect because these are listed here.
|
|
LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1}
|
|
LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h}
|
|
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h}
|
|
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
|
|
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
|
|
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
|
|
# CDP endpoint for sites behind a JavaScript challenge (kagane,
|
|
# novelfull). The browser is not part of this stack — it runs on the home
|
|
# machine as its own unit (chrome/docker-compose.yml) and is reached over
|
|
# the tailnet. Unset disables browser polling for those sites and serves
|
|
# 404 from the cover proxy for covers not already stored; the userscript
|
|
# still covers them. Set it in .env to ws://<home machine tailnet IP>:9222.
|
|
#
|
|
# Must be an IP, not a MagicDNS hostname: Chrome's DevTools HTTP handler
|
|
# rejects the discovery request (GET /json/version) with a 500 unless the
|
|
# Host header is an IP address or "localhost" — confirmed 2026-08-03,
|
|
# independent of chromedp's own dial logic. The same trap that used to
|
|
# force a pinned Docker IP now forbids the tailnet name.
|
|
BROWSER_WS_URL: ${BROWSER_WS_URL:-}
|
|
depends_on:
|
|
# The migration runner is the first thing the binary does, so a Postgres
|
|
# that is still initialising means a crash-loop until it is not.
|
|
postgres:
|
|
condition: service_healthy
|
|
volumes:
|
|
# The userscript is served from here, read fresh on every request. Editing
|
|
# the file in this checkout takes effect on the next Violentmonkey poll —
|
|
# no rebuild, no restart. `git pull` restores the committed version, which
|
|
# is why a redeploy always ships the repo's script.
|
|
- ./userscript:/userscript:ro
|
|
# Content-addressed cover bytes survive API restarts and redeploys.
|
|
- cover-data:${COVER_DIR:?set COVER_DIR in .env}
|
|
# Bound to loopback only: the proxy (or curl during smoke test) reaches it,
|
|
# the public internet does not.
|
|
ports:
|
|
- "127.0.0.1:8080:8080"
|
|
# `default` is not decoration: `db` is `internal: true`, and a container on
|
|
# nothing but an internal network gets neither a published port nor egress
|
|
# — which would silently kill every poller fetch.
|
|
networks:
|
|
- default
|
|
- db
|
|
|
|
postgres:
|
|
image: postgres:17-alpine
|
|
restart: unless-stopped
|
|
environment:
|
|
POSTGRES_DB: bookmarks
|
|
POSTGRES_USER: bookmarks
|
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"]
|
|
interval: 5s
|
|
timeout: 3s
|
|
retries: 10
|
|
volumes:
|
|
- postgres-data:/var/lib/postgresql/data
|
|
# Deliberately no `ports:` — only bookmark-api, over the `db` network,
|
|
# reaches it. Use `docker compose exec postgres psql` for a shell.
|
|
networks:
|
|
- db
|
|
|
|
volumes:
|
|
postgres-data:
|
|
cover-data:
|
|
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
|
|
# declared here: undeclared means `docker compose down -v` cannot take it
|
|
# with the rest, so the old database survives the cutover until someone
|
|
# removes it by hand.
|
|
|
|
networks:
|
|
# Postgres needs no egress and nothing outside bookmark-api needs to reach
|
|
# it, so this one really can be cut off from the outside world.
|
|
db:
|
|
internal: true
|