feat: wire the latest-chapter poller into startup

Five env vars, all with defaults sized for this deployment (84 series
at a 1h cadence). Bad values log and fall back rather than failing
startup, and the cooldown is floored at 15m. Fetches go through
tls-client with a Chrome profile as defence in depth.
This commit is contained in:
2026-07-26 15:51:20 +07:00
parent 3211b5b42f
commit 28f2d32e45
7 changed files with 383 additions and 10 deletions
+19
View File
@@ -29,3 +29,22 @@ WEB_PASSWORD=
# publish the UI router on a domain you do not own. The same container also # publish the UI router on a domain you do not own. The same container also
# answers on MANGA_API_HOST for the userscript's API. # answers on MANGA_API_HOST for the userscript's API.
# MANGA_WEB_HOST=manga.example.com # MANGA_WEB_HOST=manga.example.com
# --- Latest-chapter poller ---
# The backend re-checks each bookmarked series' newest published chapter on its
# own schedule, so latest_chapter stays fresh even when you never open the manga
# sites. This runs in parallel with the userscript's own in-browser check.
# Set to 0 to turn it off entirely.
# LATEST_CHAPTER_POLL_ENABLED=1
#
# Two independent clocks. COOLDOWN is how long one series rests between checks;
# INTERVAL is how often the poller wakes up and looks for series past that
# cooldown. Shortening INTERVAL cannot shorten a COOLDOWN.
# LATEST_CHAPTER_POLL_COOLDOWN=1h # per series, floor 15m
# LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
# LATEST_CHAPTER_POLL_BATCH=14 # series per wake
# LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
#
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
+18 -1
View File
@@ -38,8 +38,25 @@ Bromite userscript (isolated world, per-site adapters, localStorage cache)
the web routes are not registered at all. UI mutations read-modify-write the web routes are not registered at all. UI mutations read-modify-write
through `Store.Get` + `Store.Upsert` so the `updated_at` rule stays in one through `Store.Get` + `Store.Upsert` so the `updated_at` rule stays in one
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`. place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
- **Latest-chapter poller:** a ticker goroutine in the same binary re-checks
each bookmarked series' newest published chapter from the backend's own
network access, so `latest_chapter` stays fresh when the user is not
browsing. It is a *second, parallel* signal — the userscript keeps its own
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
Two independent clocks: a per-bookmark cooldown (`latest_checked_at` column,
enforced by `Store.DueForLatestCheck`'s WHERE clause) and a wake interval.
The row is stamped *before* the fetch so a broken series waits out a full
cooldown instead of retrying every tick, and writes go through
`Store.Get` + `Store.Upsert` so a new chapter never reorders the list.
Fetches use `bogdanfinn/tls-client` with a Chrome profile as defence in depth
against fingerprint-based blocking; any failure logs and skips. See
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
- **`updated_at` drives list order, so it moves only on real reading progress:** the server applies its timestamp when the row is new or `last_chapter_num` changes, and otherwise keeps the stored value — favouriting a series or recording a newly published chapter must not reorder the list. `PUT` therefore returns the row **as stored**, and clients must adopt that response rather than their own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4. - **`updated_at` drives list order, so it moves only on real reading progress:** the server applies its timestamp when the row is new or `last_chapter_num` changes, and otherwise keeps the stored value — favouriting a series or recording a newly published chapter must not reorder the list. `PUT` therefore returns the row **as stored**, and clients must adopt that response rather than their own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH` (default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD` (gates the browser UI; unset disables it). - **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
(gates the browser UI; unset disables it),
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
### Userscript structure (single IIFE, `manga-bookmark.user.js`) ### Userscript structure (single IIFE, `manga-bookmark.user.js`)
+5 -3
View File
@@ -2,15 +2,17 @@ module mangabm/backend
go 1.24.1 go 1.24.1
require modernc.org/sqlite v1.34.4 require (
github.com/bogdanfinn/fhttp v0.6.8
github.com/bogdanfinn/tls-client v1.15.1
modernc.org/sqlite v1.34.4
)
require ( require (
github.com/andybalholm/brotli v1.2.0 // indirect github.com/andybalholm/brotli v1.2.0 // indirect
github.com/bdandy/go-errors v1.2.2 // indirect github.com/bdandy/go-errors v1.2.2 // indirect
github.com/bdandy/go-socks4 v1.2.3 // indirect github.com/bdandy/go-socks4 v1.2.3 // indirect
github.com/bogdanfinn/fhttp v0.6.8 // indirect
github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
github.com/bogdanfinn/tls-client v1.15.1 // indirect
github.com/bogdanfinn/utls v1.7.7-barnius // indirect github.com/bogdanfinn/utls v1.7.7-barnius // indirect
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
github.com/dustin/go-humanize v1.0.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect
+16 -6
View File
@@ -14,6 +14,8 @@ github.com/bogdanfinn/utls v1.7.7-barnius h1:OuJ497cc7F3yKNVHRsYPQdGggmk5x6+V5Zl
github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg= github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI= github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI= github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo= github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
@@ -34,20 +36,26 @@ github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII= github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc= github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng= github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU= golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0= golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic= golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU= golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY= golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk= golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -55,8 +63,10 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU= golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY= golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw= golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc= golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ= modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ= modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y= modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
+76
View File
@@ -0,0 +1,76 @@
package main
import (
"context"
"fmt"
"io"
fhttp "github.com/bogdanfinn/fhttp"
tls_client "github.com/bogdanfinn/tls-client"
"github.com/bogdanfinn/tls-client/profiles"
)
// maxBodyBytes caps what a single series page can cost in memory. Real pages
// measured 100-400 KB on 2026-07-26, so this is roughly 10x headroom and mostly
// guards against a proxy handing back something enormous.
const maxBodyBytes = 4 << 20
// chromeUA matches the client profile below. A Chrome fingerprint paired with a
// non-Chrome user agent is itself a signal.
const chromeUA = "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 " +
"(KHTML, like Gecko) Chrome/133.0.0.0 Mobile Safari/537.36"
// tlsFetcher fetches series pages with a Chrome TLS fingerprint.
//
// Plain net/http was verified working against both sites on 2026-07-26, so this
// is not fixing an observed block — it is deliberate defence-in-depth against a
// future fingerprint-based one, chosen up front rather than reacted to later.
// The library is pure Go, so CGO_ENABLED=0, the static binary, and the
// distroless image are all unaffected.
type tlsFetcher struct {
client tls_client.HttpClient
}
var _ fetcher = (*tlsFetcher)(nil)
func newTLSFetcher() (*tlsFetcher, error) {
c, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(),
tls_client.WithTimeoutSeconds(30),
tls_client.WithClientProfile(profiles.Chrome_133),
)
if err != nil {
return nil, fmt.Errorf("new tls client: %w", err)
}
return &tlsFetcher{client: c}, nil
}
// Get fetches url and returns the body and status. Redirects are followed: the
// demonic chapter anchors are a redirect form, and asura has moved domains
// before.
func (f *tlsFetcher) Get(ctx context.Context, url string) (string, int, error) {
req, err := fhttp.NewRequest(fhttp.MethodGet, url, nil)
if err != nil {
return "", 0, fmt.Errorf("build request %q: %w", url, err)
}
req = req.WithContext(ctx)
// Header order is part of what is being fingerprinted, so it is stated
// explicitly instead of left to Go's map iteration order.
req.Header = fhttp.Header{
"user-agent": {chromeUA},
"accept": {"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
"accept-language": {"en-US,en;q=0.9"},
fhttp.HeaderOrderKey: {"user-agent", "accept", "accept-language"},
}
resp, err := f.client.Do(req)
if err != nil {
return "", 0, fmt.Errorf("get %q: %w", url, err)
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes))
if err != nil {
return "", resp.StatusCode, fmt.Errorf("read %q: %w", url, err)
}
return string(body), resp.StatusCode, nil
}
+134
View File
@@ -7,6 +7,7 @@ import (
"net/http" "net/http"
"os" "os"
"os/signal" "os/signal"
"strconv"
"strings" "strings"
"syscall" "syscall"
"time" "time"
@@ -20,8 +21,34 @@ type Config struct {
Port string Port string
// WebPassword gates the browser UI. Empty disables the web routes entirely. // WebPassword gates the browser UI. Empty disables the web routes entirely.
WebPassword string WebPassword string
// LatestPoll configures the background latest-chapter fetcher.
LatestPoll LatestPoll
} }
// LatestPoll configures the background latest-chapter poller.
//
// Sizing: batch x (cooldown / interval) is how many series hold a true cooldown
// cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this
// deployment. Past that nothing breaks; the effective cadence stretches to
// N x interval / batch and the oldest-checked-first ordering keeps it uniform.
type LatestPoll struct {
Enabled bool
Cooldown time.Duration
Interval time.Duration
Stagger time.Duration
Batch int
}
const (
defaultPollCooldown = time.Hour
defaultPollInterval = 10 * time.Minute
defaultPollStagger = 20 * time.Second
defaultPollBatch = 14
// minPollCooldown keeps a typo from turning a polite background check into
// a hammer against sites that are already bot-scoring us.
minPollCooldown = 15 * time.Minute
)
func envOr(key, def string) string { func envOr(key, def string) string {
if v := os.Getenv(key); v != "" { if v := os.Getenv(key); v != "" {
return v return v
@@ -29,12 +56,83 @@ func envOr(key, def string) string {
return def return def
} }
// envBool reads a boolean env var. Anything unrecognised falls back to def.
func envBool(key string, def bool) bool {
switch v := strings.ToLower(strings.TrimSpace(os.Getenv(key))); v {
case "":
return def
case "0", "false", "no", "off":
return false
case "1", "true", "yes", "on":
return true
default:
log.Printf("config: %s=%q is not a boolean, using %v", key, v, def)
return def
}
}
// envDuration reads a duration env var. An unparseable or non-positive value
// falls back to def and logs rather than failing startup: the poller is an
// enhancement, and a typo in one of its knobs must not stop bookmark sync.
func envDuration(key string, def time.Duration) time.Duration {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
d, err := time.ParseDuration(raw)
if err != nil || d <= 0 {
log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def)
return def
}
return d
}
// envInt reads a positive integer env var, with the same fallback policy.
func envInt(key string, def int) int {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
n, err := strconv.Atoi(raw)
if err != nil || n <= 0 {
log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def)
return def
}
return n
}
// loadLatestPoll reads the poller's settings, clamping anything that would make
// it antisocial.
func loadLatestPoll() LatestPoll {
p := LatestPoll{
Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true),
Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown),
Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval),
Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger),
Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch),
}
if p.Cooldown < minPollCooldown {
log.Printf("config: cooldown %s is below the %s floor, clamping", p.Cooldown, minPollCooldown)
p.Cooldown = minPollCooldown
}
// batch x stagger has to fit inside one tick or a batch is still running
// when the next one is due. Run() serialises them, so this degrades to a
// slower cadence rather than to overlapping fetches — worth a warning, not
// a failure.
if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval {
log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick",
p.Batch, p.Stagger, span, p.Interval)
}
return p
}
func loadConfig() Config { func loadConfig() Config {
c := Config{ c := Config{
Token: os.Getenv("API_TOKEN"), Token: os.Getenv("API_TOKEN"),
DBPath: envOr("DB_PATH", "/data/bookmarks.db"), DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
Port: envOr("PORT", "8080"), Port: envOr("PORT", "8080"),
WebPassword: os.Getenv("WEB_PASSWORD"), WebPassword: os.Getenv("WEB_PASSWORD"),
LatestPoll: loadLatestPoll(),
} }
for _, o := range strings.Split(os.Getenv("ALLOWED_ORIGINS"), ",") { for _, o := range strings.Split(os.Getenv("ALLOWED_ORIGINS"), ",") {
if o = strings.TrimSpace(o); o != "" { if o = strings.TrimSpace(o); o != "" {
@@ -87,6 +185,13 @@ func main() {
} }
defer store.Close() defer store.Close()
// The poller is off the request path entirely: if it cannot start, the
// service still serves bookmarks and the userscript still captures latest
// chapters on its own.
pollCtx, stopPoll := context.WithCancel(context.Background())
defer stopPoll()
startLatestPoller(pollCtx, store, cfg.LatestPoll)
srv := &http.Server{ srv := &http.Server{
Addr: ":" + cfg.Port, Addr: ":" + cfg.Port,
Handler: newRouter(store, cfg), Handler: newRouter(store, cfg),
@@ -105,9 +210,38 @@ func main() {
<-stop <-stop
log.Println("shutting down") log.Println("shutting down")
// Stop polling before draining requests, so an in-flight series fetch does
// not hold the process open past the shutdown deadline.
stopPoll()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel() defer cancel()
if err := srv.Shutdown(ctx); err != nil { if err := srv.Shutdown(ctx); err != nil {
log.Printf("shutdown: %v", err) log.Printf("shutdown: %v", err)
} }
} }
// startLatestPoller launches the background poller unless it is disabled or its
// HTTP client cannot be built. Any problem here is logged and skipped: this
// feature going missing degrades the service to userscript-only latest-chapter
// tracking, which is exactly how it behaved before.
func startLatestPoller(ctx context.Context, store *Store, cfg LatestPoll) {
if !cfg.Enabled {
log.Println("latest-chapter poller: disabled by config")
return
}
f, err := newTLSFetcher()
if err != nil {
log.Printf("latest-chapter poller: disabled, cannot build client: %v", err)
return
}
p := &latestPoller{
store: store,
fetch: f,
now: time.Now,
cooldown: cfg.Cooldown,
interval: cfg.Interval,
stagger: cfg.Stagger,
batch: cfg.Batch,
}
go p.Run(ctx)
}
+115
View File
@@ -0,0 +1,115 @@
package main
import (
"testing"
"time"
)
func TestLoadLatestPollDefaults(t *testing.T) {
for _, k := range []string{
"LATEST_CHAPTER_POLL_ENABLED", "LATEST_CHAPTER_POLL_COOLDOWN",
"LATEST_CHAPTER_POLL_INTERVAL", "LATEST_CHAPTER_POLL_STAGGER",
"LATEST_CHAPTER_POLL_BATCH",
} {
t.Setenv(k, "")
}
got := loadLatestPoll()
want := LatestPoll{
Enabled: true,
Cooldown: time.Hour,
Interval: 10 * time.Minute,
Stagger: 20 * time.Second,
Batch: 14,
}
if got != want {
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, want)
}
}
func TestLoadLatestPollEnabledParsing(t *testing.T) {
tests := []struct {
raw string
want bool
}{
{"", true}, {"1", true}, {"true", true}, {"TRUE", true}, {"yes", true}, {"on", true},
{"0", false}, {"false", false}, {"False", false}, {"no", false}, {"off", false},
{"maybe", true}, // unparseable falls back to the default
}
for _, tt := range tests {
t.Run("raw="+tt.raw, func(t *testing.T) {
t.Setenv("LATEST_CHAPTER_POLL_ENABLED", tt.raw)
if got := loadLatestPoll().Enabled; got != tt.want {
t.Fatalf("Enabled = %v, want %v", got, tt.want)
}
})
}
}
func TestLoadLatestPollClampsAndFallsBack(t *testing.T) {
tests := []struct {
name string
env map[string]string
wantFrom func(LatestPoll) any
want any
}{
{
name: "cooldown below the floor is clamped up",
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "1m"},
wantFrom: func(p LatestPoll) any { return p.Cooldown },
want: 15 * time.Minute,
},
{
name: "cooldown at the floor is kept",
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "15m"},
wantFrom: func(p LatestPoll) any { return p.Cooldown },
want: 15 * time.Minute,
},
{
name: "a valid override is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "5m"},
wantFrom: func(p LatestPoll) any { return p.Interval },
want: 5 * time.Minute,
},
{
name: "an unparseable duration falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "ten minutes"},
wantFrom: func(p LatestPoll) any { return p.Interval },
want: 10 * time.Minute,
},
{
name: "a zero duration falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_STAGGER": "0s"},
wantFrom: func(p LatestPoll) any { return p.Stagger },
want: 20 * time.Second,
},
{
name: "a valid batch is honoured",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "30"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 30,
},
{
name: "a negative batch falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "-5"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 14,
},
{
name: "a non-numeric batch falls back",
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "lots"},
wantFrom: func(p LatestPoll) any { return p.Batch },
want: 14,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
for k, v := range tt.env {
t.Setenv(k, v)
}
if got := tt.wantFrom(loadLatestPoll()); got != tt.want {
t.Fatalf("got %v, want %v", got, tt.want)
}
})
}
}