Files
mangaBookmark/docker-compose.yml
T
sulthan cd3a7e3d01 feat(latest): split browser poll cooldown (#48)
## Summary

Split latest-chapter polling cooldowns by fetch cost. Browser-backed kagane and novelfull series now rest longer without changing the cadence of plain-TLS sites.

## Behavior

- Plain-TLS series keep the 1h default cooldown.
- Browser-backed series use `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, defaulting to 6h.
- Both cooldowns share the existing 15m minimum floor; invalid values retain the existing fallback behavior.
- The poller still selects both classes in one due query per cycle.
- Existing ordering and exclusions remain unchanged: reader-count precedence, least-recently-checked ordering, finished exclusion, archived polling, and orphan exclusion.

## Implementation

- Added the browser cooldown to backend configuration and passed it through production poller construction.
- Added the browser-site list as the single routing source used for both due-query cutoff selection and fetcher choice.
- Kept all query values parameterized; the site list is passed as a bound PostgreSQL array parameter.
- Updated startup logging to report interval, plain cooldown, browser cooldown, batch, and stagger.
- Documented the variable, default, and floor in `README.md`, `.env.example`, `backend/AGENTS.md`, and `docker-compose.yml`.

## Review findings addressed

The first review found that configuration parsing was correct but `startLatestPoller` did not pass `BrowserCooldown` into `latest.Poller`; every browser-backed row would therefore have been due immediately. Production construction now goes through `newLatestPoller`, with a regression test covering both cooldown fields.

The review also identified duplicated browser-site knowledge in fetch routing. `slices.Contains(browserBackedSites, site)` now reuses the same list already supplied to the store query.

## Verification

- Focused backend tests pass: `go test ./internal/latest ./internal/store .`.
- Full suite passes: `go test ./...`.
- `graphify update .` completed.
- Issue #42 was updated and closed.

Reviewed-on: #48
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-09 06:25:48 +07:00

159 lines
8.1 KiB
YAML

# Base stack — works standalone for local smoke testing (`docker compose up`).
# The service binds 127.0.0.1:8080; a host reverse proxy (nginx/Caddy/Traefik)
# terminates TLS for bookmark-api.<domain> and forwards to it.
#
# If your proxy runs in Docker on its own network, use the prod override which
# attaches to that network instead of publishing a port:
# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
services:
bookmark-api:
build: ./backend
image: bookmarkmanager-backend:latest
container_name: bookmark-api
restart: unless-stopped
environment:
# TOKEN_KEY derives every Reader's userscript credential (issue #24) —
# compose refuses to start without it.
TOKEN_KEY: ${TOKEN_KEY:?set TOKEN_KEY in .env}
# Owner's Discord user ID — required. Seeds the owner Reader (the
# administrator); every other Reader registers on their first login.
OWNER_DISCORD_ID: ${OWNER_DISCORD_ID:?set OWNER_DISCORD_ID in .env}
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
# The bookmarks database. Host is the compose service name; the password
# comes from .env so it is never committed.
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
PORT: "8080"
# Log timestamps only. Go's `log` stamps lines in local time, and this
# service has no other use for a zone: bookmark timestamps are unix ms
# and the two real time columns are timestamptz, both absolute instants.
# Purely so these lines read on the same clock as the sidecar's. Named
# API_TZ rather than TZ so an operator's exported shell TZ cannot leak
# in; distroless already carries tzdata, so the name just resolves.
TZ: ${API_TZ:-Asia/Jakarta}
# Discord OAuth for the browser UI (ADR-0002). The first four are
# required; DISCORD_REQUIRED_ROLE is optional and empty by default.
# Guild membership is the whole gate: any member becomes a Reader.
DISCORD_CLIENT_ID: ${DISCORD_CLIENT_ID:?set DISCORD_CLIENT_ID in .env}
DISCORD_CLIENT_SECRET: ${DISCORD_CLIENT_SECRET:?set DISCORD_CLIENT_SECRET in .env}
DISCORD_GUILD_ID: ${DISCORD_GUILD_ID:?set DISCORD_GUILD_ID in .env}
DISCORD_REQUIRED_ROLE: ${DISCORD_REQUIRED_ROLE:-}
DISCORD_API_BASE: ${DISCORD_API_BASE:-https://discord.com/api/v10}
DISCORD_REDIRECT_URI: ${DISCORD_REDIRECT_URI:?set DISCORD_REDIRECT_URI in .env}
# Path inside the container; matches the bindmount above.
USERSCRIPT_PATH: ${USERSCRIPT_PATH:-/userscript/manga-bookmark.user.js}
# Second script from the same bindmount; the novel library is a separate
# Violentmonkey install.
NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js}
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill
# switch; it only takes effect because these are listed here.
LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1}
LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h}
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h}
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
# CDP endpoint for sites behind a JavaScript challenge (kagane). Unset
# disables browser polling for those sites; the userscript still covers them.
# Must be an IP, not the "headless-shell" DNS name: Chrome's DevTools HTTP
# handler rejects the discovery request (GET /json/version) with a 500
# unless the Host header is an IP address or "localhost" — confirmed
# 2026-08-03 against chromedp/headless-shell:stable, independent of
# chromedp's own dial logic. The sidecar's static address below exists so
# this URL survives container recreation.
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
depends_on:
headless-shell:
condition: service_started
# The migration runner is the first thing the binary does, so a Postgres
# that is still initialising means a crash-loop until it is not.
postgres:
condition: service_healthy
volumes:
# The userscript is served from here, read fresh on every request. Editing
# the file in this checkout takes effect on the next Violentmonkey poll —
# no rebuild, no restart. `git pull` restores the committed version, which
# is why a redeploy always ships the repo's script.
- ./userscript:/userscript:ro
# Bound to loopback only: the proxy (or curl during smoke test) reaches it,
# the public internet does not.
ports:
- "127.0.0.1:8080:8080"
networks:
- browser
- db
postgres:
image: postgres:17-alpine
restart: unless-stopped
environment:
POSTGRES_DB: bookmarks
POSTGRES_USER: bookmarks
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"]
interval: 5s
timeout: 3s
retries: 10
volumes:
- postgres-data:/var/lib/postgresql/data
# Deliberately no `ports:` — only bookmark-api, over the `db` network,
# reaches it. Use `docker compose exec postgres psql` for a shell.
networks:
- db
headless-shell:
# Real Google Chrome, not chromedp/headless-shell — see chrome/Dockerfile.
# The service name is kept so existing overrides and BROWSER_WS_URL stay put.
build: ./chrome
image: bookmarkmanager-chrome:latest
restart: unless-stopped
environment:
# A UTC clock is itself the bot signal: Cloudflare treats it as the
# datacenter default, and kagane's challenge then never clears. Measured
# 2026-08-08, identical container, one Indonesian egress IP: UTC never
# cleared in 60s (twice); Asia/Jakarta and America/New_York both cleared
# in 4s. So any real zone works and it need not match the IP's country —
# only UTC fails. Unset falls back to the host's /etc/timezone below,
# which is a real zone whenever the host clock is set to local time; set
# BROWSER_TZ when the host runs UTC.
TZ: ${BROWSER_TZ:-}
volumes:
# The zone *name*, which is what Chrome's ICU needs — see chrome/entrypoint.sh.
# Absent on a non-Debian host, which the entrypoint handles by falling back to UTC.
- /etc/timezone:/etc/timezone:ro
# Chrome allocates shared memory per tab and dies on Docker's 64MB default.
shm_size: '1gb'
# Reaps zombie renderer processes, which otherwise accumulate for the
# container's lifetime.
init: true
# Deliberately no `ports:` — an exposed CDP endpoint is remote code
# execution. Only bookmark-api, via the `browser` network below, may reach it.
# No `command:` either: every flag this browser needs is in its entrypoint,
# and the UA override there is load-bearing for the challenge.
networks:
browser:
# Pinned so BROWSER_WS_URL can name an IP (required, see above) that
# survives `docker compose up` recreating this container.
ipv4_address: 172.28.0.10
volumes:
postgres-data:
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
# declared here: undeclared means `docker compose down -v` cannot take it
# with the rest, so the old database survives the cutover until someone
# removes it by hand.
networks:
# Not `internal: true`: headless Chrome still needs outbound access to reach
# kagane.to. Isolation here comes from membership (only bookmark-api and
# headless-shell join it), not from cutting egress.
browser:
ipam:
config:
- subnet: 172.28.0.0/24
# Postgres needs no egress and nothing outside bookmark-api needs to reach
# it, so this one really can be cut off from the outside world.
db:
internal: true