cd3a7e3d01
## Summary Split latest-chapter polling cooldowns by fetch cost. Browser-backed kagane and novelfull series now rest longer without changing the cadence of plain-TLS sites. ## Behavior - Plain-TLS series keep the 1h default cooldown. - Browser-backed series use `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, defaulting to 6h. - Both cooldowns share the existing 15m minimum floor; invalid values retain the existing fallback behavior. - The poller still selects both classes in one due query per cycle. - Existing ordering and exclusions remain unchanged: reader-count precedence, least-recently-checked ordering, finished exclusion, archived polling, and orphan exclusion. ## Implementation - Added the browser cooldown to backend configuration and passed it through production poller construction. - Added the browser-site list as the single routing source used for both due-query cutoff selection and fetcher choice. - Kept all query values parameterized; the site list is passed as a bound PostgreSQL array parameter. - Updated startup logging to report interval, plain cooldown, browser cooldown, batch, and stagger. - Documented the variable, default, and floor in `README.md`, `.env.example`, `backend/AGENTS.md`, and `docker-compose.yml`. ## Review findings addressed The first review found that configuration parsing was correct but `startLatestPoller` did not pass `BrowserCooldown` into `latest.Poller`; every browser-backed row would therefore have been due immediately. Production construction now goes through `newLatestPoller`, with a regression test covering both cooldown fields. The review also identified duplicated browser-site knowledge in fetch routing. `slices.Contains(browserBackedSites, site)` now reuses the same list already supplied to the store query. ## Verification - Focused backend tests pass: `go test ./internal/latest ./internal/store .`. - Full suite passes: `go test ./...`. - `graphify update .` completed. - Issue #42 was updated and closed. Reviewed-on: #48 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
159 lines
8.1 KiB
YAML
159 lines
8.1 KiB
YAML
# Base stack — works standalone for local smoke testing (`docker compose up`).
|
|
# The service binds 127.0.0.1:8080; a host reverse proxy (nginx/Caddy/Traefik)
|
|
# terminates TLS for bookmark-api.<domain> and forwards to it.
|
|
#
|
|
# If your proxy runs in Docker on its own network, use the prod override which
|
|
# attaches to that network instead of publishing a port:
|
|
# docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
|
|
|
|
services:
|
|
bookmark-api:
|
|
build: ./backend
|
|
image: bookmarkmanager-backend:latest
|
|
container_name: bookmark-api
|
|
restart: unless-stopped
|
|
environment:
|
|
# TOKEN_KEY derives every Reader's userscript credential (issue #24) —
|
|
# compose refuses to start without it.
|
|
TOKEN_KEY: ${TOKEN_KEY:?set TOKEN_KEY in .env}
|
|
# Owner's Discord user ID — required. Seeds the owner Reader (the
|
|
# administrator); every other Reader registers on their first login.
|
|
OWNER_DISCORD_ID: ${OWNER_DISCORD_ID:?set OWNER_DISCORD_ID in .env}
|
|
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net}
|
|
# The bookmarks database. Host is the compose service name; the password
|
|
# comes from .env so it is never committed.
|
|
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
|
|
PORT: "8080"
|
|
# Log timestamps only. Go's `log` stamps lines in local time, and this
|
|
# service has no other use for a zone: bookmark timestamps are unix ms
|
|
# and the two real time columns are timestamptz, both absolute instants.
|
|
# Purely so these lines read on the same clock as the sidecar's. Named
|
|
# API_TZ rather than TZ so an operator's exported shell TZ cannot leak
|
|
# in; distroless already carries tzdata, so the name just resolves.
|
|
TZ: ${API_TZ:-Asia/Jakarta}
|
|
# Discord OAuth for the browser UI (ADR-0002). The first four are
|
|
# required; DISCORD_REQUIRED_ROLE is optional and empty by default.
|
|
# Guild membership is the whole gate: any member becomes a Reader.
|
|
DISCORD_CLIENT_ID: ${DISCORD_CLIENT_ID:?set DISCORD_CLIENT_ID in .env}
|
|
DISCORD_CLIENT_SECRET: ${DISCORD_CLIENT_SECRET:?set DISCORD_CLIENT_SECRET in .env}
|
|
DISCORD_GUILD_ID: ${DISCORD_GUILD_ID:?set DISCORD_GUILD_ID in .env}
|
|
DISCORD_REQUIRED_ROLE: ${DISCORD_REQUIRED_ROLE:-}
|
|
DISCORD_API_BASE: ${DISCORD_API_BASE:-https://discord.com/api/v10}
|
|
DISCORD_REDIRECT_URI: ${DISCORD_REDIRECT_URI:?set DISCORD_REDIRECT_URI in .env}
|
|
# Path inside the container; matches the bindmount above.
|
|
USERSCRIPT_PATH: ${USERSCRIPT_PATH:-/userscript/manga-bookmark.user.js}
|
|
# Second script from the same bindmount; the novel library is a separate
|
|
# Violentmonkey install.
|
|
NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js}
|
|
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill
|
|
# switch; it only takes effect because these are listed here.
|
|
LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1}
|
|
LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h}
|
|
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h}
|
|
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
|
|
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
|
|
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
|
|
# CDP endpoint for sites behind a JavaScript challenge (kagane). Unset
|
|
# disables browser polling for those sites; the userscript still covers them.
|
|
# Must be an IP, not the "headless-shell" DNS name: Chrome's DevTools HTTP
|
|
# handler rejects the discovery request (GET /json/version) with a 500
|
|
# unless the Host header is an IP address or "localhost" — confirmed
|
|
# 2026-08-03 against chromedp/headless-shell:stable, independent of
|
|
# chromedp's own dial logic. The sidecar's static address below exists so
|
|
# this URL survives container recreation.
|
|
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://172.28.0.10:9222}
|
|
depends_on:
|
|
headless-shell:
|
|
condition: service_started
|
|
# The migration runner is the first thing the binary does, so a Postgres
|
|
# that is still initialising means a crash-loop until it is not.
|
|
postgres:
|
|
condition: service_healthy
|
|
volumes:
|
|
# The userscript is served from here, read fresh on every request. Editing
|
|
# the file in this checkout takes effect on the next Violentmonkey poll —
|
|
# no rebuild, no restart. `git pull` restores the committed version, which
|
|
# is why a redeploy always ships the repo's script.
|
|
- ./userscript:/userscript:ro
|
|
# Bound to loopback only: the proxy (or curl during smoke test) reaches it,
|
|
# the public internet does not.
|
|
ports:
|
|
- "127.0.0.1:8080:8080"
|
|
networks:
|
|
- browser
|
|
- db
|
|
|
|
postgres:
|
|
image: postgres:17-alpine
|
|
restart: unless-stopped
|
|
environment:
|
|
POSTGRES_DB: bookmarks
|
|
POSTGRES_USER: bookmarks
|
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U bookmarks -d bookmarks"]
|
|
interval: 5s
|
|
timeout: 3s
|
|
retries: 10
|
|
volumes:
|
|
- postgres-data:/var/lib/postgresql/data
|
|
# Deliberately no `ports:` — only bookmark-api, over the `db` network,
|
|
# reaches it. Use `docker compose exec postgres psql` for a shell.
|
|
networks:
|
|
- db
|
|
|
|
headless-shell:
|
|
# Real Google Chrome, not chromedp/headless-shell — see chrome/Dockerfile.
|
|
# The service name is kept so existing overrides and BROWSER_WS_URL stay put.
|
|
build: ./chrome
|
|
image: bookmarkmanager-chrome:latest
|
|
restart: unless-stopped
|
|
environment:
|
|
# A UTC clock is itself the bot signal: Cloudflare treats it as the
|
|
# datacenter default, and kagane's challenge then never clears. Measured
|
|
# 2026-08-08, identical container, one Indonesian egress IP: UTC never
|
|
# cleared in 60s (twice); Asia/Jakarta and America/New_York both cleared
|
|
# in 4s. So any real zone works and it need not match the IP's country —
|
|
# only UTC fails. Unset falls back to the host's /etc/timezone below,
|
|
# which is a real zone whenever the host clock is set to local time; set
|
|
# BROWSER_TZ when the host runs UTC.
|
|
TZ: ${BROWSER_TZ:-}
|
|
volumes:
|
|
# The zone *name*, which is what Chrome's ICU needs — see chrome/entrypoint.sh.
|
|
# Absent on a non-Debian host, which the entrypoint handles by falling back to UTC.
|
|
- /etc/timezone:/etc/timezone:ro
|
|
# Chrome allocates shared memory per tab and dies on Docker's 64MB default.
|
|
shm_size: '1gb'
|
|
# Reaps zombie renderer processes, which otherwise accumulate for the
|
|
# container's lifetime.
|
|
init: true
|
|
# Deliberately no `ports:` — an exposed CDP endpoint is remote code
|
|
# execution. Only bookmark-api, via the `browser` network below, may reach it.
|
|
# No `command:` either: every flag this browser needs is in its entrypoint,
|
|
# and the UA override there is load-bearing for the challenge.
|
|
networks:
|
|
browser:
|
|
# Pinned so BROWSER_WS_URL can name an IP (required, see above) that
|
|
# survives `docker compose up` recreating this container.
|
|
ipv4_address: 172.28.0.10
|
|
|
|
volumes:
|
|
postgres-data:
|
|
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
|
|
# declared here: undeclared means `docker compose down -v` cannot take it
|
|
# with the rest, so the old database survives the cutover until someone
|
|
# removes it by hand.
|
|
|
|
networks:
|
|
# Not `internal: true`: headless Chrome still needs outbound access to reach
|
|
# kagane.to. Isolation here comes from membership (only bookmark-api and
|
|
# headless-shell join it), not from cutting egress.
|
|
browser:
|
|
ipam:
|
|
config:
|
|
- subnet: 172.28.0.0/24
|
|
# Postgres needs no egress and nothing outside bookmark-api needs to reach
|
|
# it, so this one really can be cut off from the outside world.
|
|
db:
|
|
internal: true
|