sulthan
cd3a7e3d01
## Summary Split latest-chapter polling cooldowns by fetch cost. Browser-backed kagane and novelfull series now rest longer without changing the cadence of plain-TLS sites. ## Behavior - Plain-TLS series keep the 1h default cooldown. - Browser-backed series use `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, defaulting to 6h. - Both cooldowns share the existing 15m minimum floor; invalid values retain the existing fallback behavior. - The poller still selects both classes in one due query per cycle. - Existing ordering and exclusions remain unchanged: reader-count precedence, least-recently-checked ordering, finished exclusion, archived polling, and orphan exclusion. ## Implementation - Added the browser cooldown to backend configuration and passed it through production poller construction. - Added the browser-site list as the single routing source used for both due-query cutoff selection and fetcher choice. - Kept all query values parameterized; the site list is passed as a bound PostgreSQL array parameter. - Updated startup logging to report interval, plain cooldown, browser cooldown, batch, and stagger. - Documented the variable, default, and floor in `README.md`, `.env.example`, `backend/AGENTS.md`, and `docker-compose.yml`. ## Review findings addressed The first review found that configuration parsing was correct but `startLatestPoller` did not pass `BrowserCooldown` into `latest.Poller`; every browser-backed row would therefore have been due immediately. Production construction now goes through `newLatestPoller`, with a regression test covering both cooldown fields. The review also identified duplicated browser-site knowledge in fetch routing. `slices.Contains(browserBackedSites, site)` now reuses the same list already supplied to the store query. ## Verification - Focused backend tests pass: `go test ./internal/latest ./internal/store .`. - Full suite passes: `go test ./...`. - `graphify update .` completed. - Issue #42 was updated and closed. Reviewed-on: #48 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
115 lines
5.9 KiB
Bash
115 lines
5.9 KiB
Bash
# Copy to .env and fill in. Never commit the real .env.
|
|
|
|
# Secret every Reader's userscript credential is derived from (issue #24):
|
|
# the backend rebuilds install URLs from it, and only SHA-256 hashes of the
|
|
# credentials ever touch the database. Generate one:
|
|
# openssl rand -hex 32
|
|
TOKEN_KEY=changeme-generate-a-long-random-token
|
|
|
|
# The owner's Discord user ID — seeded at startup as the first Reader, the
|
|
# administrator (the only one who can revoke another Reader's sessions), and
|
|
# the owner of every bookmark that predates registration. Discord snowflake,
|
|
# e.g. 1046923170000000000.
|
|
OWNER_DISCORD_ID=changeme-your-discord-user-id
|
|
|
|
# Comma-separated origins allowed to call the API (CORS). Both Asura domains
|
|
# plus Demonic, Comix, Kagane, and the two novel sites. Add/remove as the
|
|
# sites' hostnames change.
|
|
ALLOWED_ORIGINS=https://asuracomic.net,https://asurascans.com,https://demonicscans.org,https://comix.to,https://kagane.to,https://novelfull.com,https://lightnovelworld.net
|
|
|
|
# Password for the bundled Postgres container, and therefore half of the
|
|
# DATABASE_URL compose builds for the backend. Generate one:
|
|
# openssl rand -hex 24
|
|
POSTGRES_PASSWORD=changeme-generate-a-long-random-password
|
|
|
|
# Override only to point the backend at a Postgres compose does not run.
|
|
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
|
|
|
# --- Prod override (Traefik) only ---
|
|
# Subdomain Traefik routes to this service (required by the prod override).
|
|
# BOOKMARK_API_HOST=bookmark-api.example.com
|
|
# Traefik's docker network name, if not "proxy".
|
|
# PROXY_NETWORK=proxy
|
|
# Traefik HTTPS entrypoint + cert resolver names, if yours differ from these.
|
|
# TRAEFIK_ENTRYPOINT=websecure
|
|
# TRAEFIK_CERTRESOLVER=le
|
|
|
|
# --- Web UI (Discord OAuth) ---
|
|
# Sign-in is a Discord authorization code grant (ADR-0002), and it is also
|
|
# registration: any member of the configured guild becomes a Reader on their
|
|
# first successful login, with their own empty library. Create the application
|
|
# at https://discord.com/developers/applications and register the exact
|
|
# callback URL ($BOOKMARK_WEB_HOST/auth/discord/callback) as an OAuth2
|
|
# redirect.
|
|
DISCORD_CLIENT_ID=
|
|
DISCORD_CLIENT_SECRET=
|
|
# The guild whose membership gates sign-in (Developer Mode -> right-click the
|
|
# server -> Copy Server ID).
|
|
DISCORD_GUILD_ID=
|
|
# Exact callback URL, e.g. https://bookmark.example.com/auth/discord/callback.
|
|
# Discord matches it verbatim, so it must equal the registered redirect.
|
|
DISCORD_REDIRECT_URI=
|
|
# Optional: a role snowflake members must hold on top of guild membership.
|
|
# Empty (the default) means membership alone suffices.
|
|
# DISCORD_REQUIRED_ROLE=
|
|
|
|
# Subdomain Traefik routes to the browser UI (required by the prod override).
|
|
# Left commented on purpose: an example value here would be a silent
|
|
# wrong-hostname fallback, and Traefik would publish the UI router on a domain
|
|
# you do not own. The same container also answers on BOOKMARK_API_HOST for the
|
|
# userscript's API.
|
|
# BOOKMARK_WEB_HOST=bookmark.example.com
|
|
|
|
# --- Latest-chapter poller ---
|
|
# The backend re-checks each bookmarked series' newest published chapter on its
|
|
# own schedule, so latest_chapter stays fresh even when you never open the manga
|
|
# sites. This runs in parallel with the userscript's own in-browser check.
|
|
# Set to 0 to turn it off entirely.
|
|
# LATEST_CHAPTER_POLL_ENABLED=1
|
|
#
|
|
# Two independent clocks. COOLDOWN is how long a plain-TLS series rests between
|
|
# checks; BROWSER_COOLDOWN is the longer rest for kagane and novelfull. INTERVAL
|
|
# is how often the poller wakes up and looks for series past their cooldowns.
|
|
# Shortening INTERVAL cannot shorten either cooldown.
|
|
LATEST_CHAPTER_POLL_COOLDOWN=1h # plain-TLS per series, floor 15m
|
|
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN=6h # browser-backed per series, floor 15m
|
|
LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
|
|
LATEST_CHAPTER_POLL_BATCH=14 # series per wake
|
|
LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
|
|
#
|
|
# Uses a ticker, not an immediate first run: the first poll happens one
|
|
# INTERVAL after startup, not at startup. A container restarting more often
|
|
# than INTERVAL never polls.
|
|
#
|
|
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
|
|
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
|
|
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
|
|
|
|
# Headless-shell CDP endpoint for sites behind a JavaScript challenge (kagane).
|
|
# Unset disables browser polling; those sites then rely on the userscript alone.
|
|
# Leave commented — the compose files' own default (ws://172.28.0.10:9222) is
|
|
# correct. Do NOT set this to the "headless-shell" DNS name: Chrome's DevTools
|
|
# HTTP handler 500s any /json/version request whose Host header isn't an IP or
|
|
# "localhost", which silently breaks every kagane poll.
|
|
# BROWSER_WS_URL=ws://172.28.0.10:9222
|
|
|
|
# Clock zone the headless browser reports. A UTC clock is itself the bot
|
|
# signal — Cloudflare treats it as the datacenter default — and kagane's
|
|
# challenge then never clears. Measured 2026-08-08, identical container, one
|
|
# Indonesian egress IP: UTC never cleared in 60s (twice); Asia/Jakarta and
|
|
# America/New_York both cleared in 4s. So any real zone works; it does not
|
|
# have to match the IP's country, it just must not be UTC.
|
|
#
|
|
# Unset falls back to the host's /etc/timezone, which is a real zone whenever
|
|
# the host clock is set to local time. Set this when the host runs UTC — a UTC
|
|
# server is exactly the case that fails. Only the browser sidecar reads it —
|
|
# the backend's own zone is API_TZ below, and is cosmetic.
|
|
# BROWSER_TZ=Asia/Jakarta
|
|
|
|
# Zone the backend stamps its log lines in. Cosmetic only — it exists so the
|
|
# API's logs read on the same clock as the browser sidecar's. Nothing else in
|
|
# the service has a zone: bookmark timestamps are unix ms, and the two real
|
|
# time columns are timestamptz. Defaults to Asia/Jakarta; set to UTC for the
|
|
# conventional server default.
|
|
# API_TZ=Asia/Jakarta
|