sulthan
92eba07da7
Closes #59. Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55. ## What changed A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s. ### Store - Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on. - `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there. - `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address. - The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path. - `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load. ### Acquisition - `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position). - Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them. - Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid. - Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later. - Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62. ### Wire and route - `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk. - `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep. - `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading. ### Config `PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`. ## Acceptance criteria All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence. ## Verification - `go test ./...` green (Docker-backed Postgres suite). - Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type. - Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red. ## Reviewed Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail. ## Known sequencing A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly. Reviewed-on: #68 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
122 lines
6.1 KiB
Bash
122 lines
6.1 KiB
Bash
# Copy to .env and fill in. Never commit the real .env.
|
|
|
|
# Secret every Reader's userscript credential is derived from (issue #24):
|
|
# the backend rebuilds install URLs from it, and only SHA-256 hashes of the
|
|
# credentials ever touch the database. Generate one:
|
|
# openssl rand -hex 32
|
|
TOKEN_KEY=changeme-generate-a-long-random-token
|
|
|
|
# The owner's Discord user ID — seeded at startup as the first Reader, the
|
|
# administrator (the only one who can revoke another Reader's sessions), and
|
|
# the owner of every bookmark that predates registration. Discord snowflake,
|
|
# e.g. 1046923170000000000.
|
|
OWNER_DISCORD_ID=changeme-your-discord-user-id
|
|
|
|
# Comma-separated origins allowed to call the API (CORS). Both Asura domains
|
|
# plus Demonic, Comix, Kagane, and the two novel sites. Add/remove as the
|
|
# sites' hostnames change.
|
|
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net
|
|
|
|
# Password for the bundled Postgres container, and therefore half of the
|
|
# DATABASE_URL compose builds for the backend. Generate one:
|
|
# openssl rand -hex 24
|
|
POSTGRES_PASSWORD=changeme-generate-a-long-random-password
|
|
|
|
# Override only to point the backend at a Postgres compose does not run.
|
|
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
|
|
|
# Directory inside bookmark-api for immutable, content-addressed Cover bytes.
|
|
# Compose builds the image and mounts its named volume at this path.
|
|
COVER_DIR=/covers
|
|
|
|
# Public origin this deployment answers on, no trailing slash. Required: Cover
|
|
# URLs go out absolute, because the userscript renders them on a Site's own
|
|
# origin where a relative path would resolve against the Site (ADR-0007).
|
|
PUBLIC_BASE_URL=https://bookmark-api.example.com
|
|
|
|
# --- Prod override (Traefik) only ---
|
|
# Subdomain Traefik routes to this service (required by the prod override).
|
|
# BOOKMARK_API_HOST=bookmark-api.example.com
|
|
# Traefik's docker network name, if not "proxy".
|
|
# PROXY_NETWORK=proxy
|
|
# Traefik HTTPS entrypoint + cert resolver names, if yours differ from these.
|
|
# TRAEFIK_ENTRYPOINT=websecure
|
|
# TRAEFIK_CERTRESOLVER=le
|
|
|
|
# --- Web UI (Discord OAuth) ---
|
|
# Sign-in is a Discord authorization code grant (ADR-0002), and it is also
|
|
# registration: any member of the configured guild becomes a Reader on their
|
|
# first successful login, with their own empty library. Create the application
|
|
# at https://discord.com/developers/applications and register the exact
|
|
# callback URL ($BOOKMARK_WEB_HOST/auth/discord/callback) as an OAuth2
|
|
# redirect.
|
|
DISCORD_CLIENT_ID=
|
|
DISCORD_CLIENT_SECRET=
|
|
# The guild whose membership gates sign-in (Developer Mode -> right-click the
|
|
# server -> Copy Server ID).
|
|
DISCORD_GUILD_ID=
|
|
# Exact callback URL, e.g. https://bookmark.example.com/auth/discord/callback.
|
|
# Discord matches it verbatim, so it must equal the registered redirect.
|
|
DISCORD_REDIRECT_URI=
|
|
# Optional: a role snowflake members must hold on top of guild membership.
|
|
# Empty (the default) means membership alone suffices.
|
|
# DISCORD_REQUIRED_ROLE=
|
|
|
|
# Subdomain Traefik routes to the browser UI (required by the prod override).
|
|
# Left commented on purpose: an example value here would be a silent
|
|
# wrong-hostname fallback, and Traefik would publish the UI router on a domain
|
|
# you do not own. The same container also answers on BOOKMARK_API_HOST for the
|
|
# userscript's API.
|
|
# BOOKMARK_WEB_HOST=bookmark.example.com
|
|
|
|
# --- Latest-chapter poller ---
|
|
# The backend re-checks each bookmarked series' newest published chapter on its
|
|
# own schedule, so latest_chapter stays fresh even when you never open the manga
|
|
# sites. This runs in parallel with the userscript's own in-browser check.
|
|
# Set to 0 to turn it off entirely.
|
|
# LATEST_CHAPTER_POLL_ENABLED=1
|
|
#
|
|
# Two independent clocks. COOLDOWN is how long a plain-TLS series rests between
|
|
# checks; BROWSER_COOLDOWN is the longer rest for kagane and novelfull. INTERVAL
|
|
# is how often the poller wakes up and looks for series past their cooldowns.
|
|
# Shortening INTERVAL cannot shorten either cooldown.
|
|
LATEST_CHAPTER_POLL_COOLDOWN=1h # plain-TLS per series, floor 15m
|
|
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN=6h # browser-backed per series, floor 15m
|
|
LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
|
|
LATEST_CHAPTER_POLL_BATCH=14 # series per wake
|
|
LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
|
|
#
|
|
# Uses a ticker, not an immediate first run: the first poll happens one
|
|
# INTERVAL after startup, not at startup. A container restarting more often
|
|
# than INTERVAL never polls.
|
|
#
|
|
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
|
|
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
|
|
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
|
|
|
|
# CDP endpoint of the browser, used for the two sites behind a Cloudflare
|
|
# JavaScript challenge (kagane, novelfull) and by the web UI's kagane cover
|
|
# proxy. Unset disables browser polling and serves 404 for covers not already
|
|
# stored; those sites then rely on the userscript alone. That is also exactly
|
|
# how an unreachable browser degrades, so a home machine that is off costs
|
|
# chapter freshness and nothing else.
|
|
#
|
|
# The browser does NOT run in this stack. It is its own compose unit on the
|
|
# home machine (chrome/docker-compose.yml, chrome/.env.example) and is reached
|
|
# over the tailnet, so set this to that machine's tailnet address:
|
|
#
|
|
# BROWSER_WS_URL=ws://100.x.y.z:9222
|
|
#
|
|
# It must be the tailnet **IP**, never a MagicDNS hostname and never the old
|
|
# Docker service name: Chrome's DevTools HTTP handler 500s any /json/version
|
|
# request whose Host header isn't an IP or "localhost", which silently breaks
|
|
# every kagane poll. Left unset here on purpose — a wrong default would poll a
|
|
# stranger's address, and "no browser" is a safe, self-announcing state.
|
|
# BROWSER_WS_URL=ws://100.x.y.z:9222
|
|
|
|
# Zone the backend stamps its log lines in. Cosmetic only. Nothing else in
|
|
# the service has a zone: bookmark timestamps are unix ms, and the two real
|
|
# time columns are timestamptz. Defaults to Asia/Jakarta; set to UTC for the
|
|
# conventional server default.
|
|
# API_TZ=Asia/Jakarta
|