feat: wire the latest-chapter poller into startup
Five env vars, all with defaults sized for this deployment (84 series at a 1h cadence). Bad values log and fall back rather than failing startup, and the cooldown is floored at 15m. Fetches go through tls-client with a Chrome profile as defence in depth.
This commit is contained in:
@@ -29,3 +29,22 @@ WEB_PASSWORD=
|
|||||||
# publish the UI router on a domain you do not own. The same container also
|
# publish the UI router on a domain you do not own. The same container also
|
||||||
# answers on MANGA_API_HOST for the userscript's API.
|
# answers on MANGA_API_HOST for the userscript's API.
|
||||||
# MANGA_WEB_HOST=manga.example.com
|
# MANGA_WEB_HOST=manga.example.com
|
||||||
|
|
||||||
|
# --- Latest-chapter poller ---
|
||||||
|
# The backend re-checks each bookmarked series' newest published chapter on its
|
||||||
|
# own schedule, so latest_chapter stays fresh even when you never open the manga
|
||||||
|
# sites. This runs in parallel with the userscript's own in-browser check.
|
||||||
|
# Set to 0 to turn it off entirely.
|
||||||
|
# LATEST_CHAPTER_POLL_ENABLED=1
|
||||||
|
#
|
||||||
|
# Two independent clocks. COOLDOWN is how long one series rests between checks;
|
||||||
|
# INTERVAL is how often the poller wakes up and looks for series past that
|
||||||
|
# cooldown. Shortening INTERVAL cannot shorten a COOLDOWN.
|
||||||
|
# LATEST_CHAPTER_POLL_COOLDOWN=1h # per series, floor 15m
|
||||||
|
# LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
|
||||||
|
# LATEST_CHAPTER_POLL_BATCH=14 # series per wake
|
||||||
|
# LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
|
||||||
|
#
|
||||||
|
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
|
||||||
|
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
|
||||||
|
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
|
||||||
|
|||||||
@@ -38,8 +38,25 @@ Bromite userscript (isolated world, per-site adapters, localStorage cache)
|
|||||||
the web routes are not registered at all. UI mutations read-modify-write
|
the web routes are not registered at all. UI mutations read-modify-write
|
||||||
through `Store.Get` + `Store.Upsert` so the `updated_at` rule stays in one
|
through `Store.Get` + `Store.Upsert` so the `updated_at` rule stays in one
|
||||||
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
place. See `docs/superpowers/specs/2026-07-25-web-ui-design.md`.
|
||||||
|
- **Latest-chapter poller:** a ticker goroutine in the same binary re-checks
|
||||||
|
each bookmarked series' newest published chapter from the backend's own
|
||||||
|
network access, so `latest_chapter` stays fresh when the user is not
|
||||||
|
browsing. It is a *second, parallel* signal — the userscript keeps its own
|
||||||
|
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
||||||
|
Two independent clocks: a per-bookmark cooldown (`latest_checked_at` column,
|
||||||
|
enforced by `Store.DueForLatestCheck`'s WHERE clause) and a wake interval.
|
||||||
|
The row is stamped *before* the fetch so a broken series waits out a full
|
||||||
|
cooldown instead of retrying every tick, and writes go through
|
||||||
|
`Store.Get` + `Store.Upsert` so a new chapter never reorders the list.
|
||||||
|
Fetches use `bogdanfinn/tls-client` with a Chrome profile as defence in depth
|
||||||
|
against fingerprint-based blocking; any failure logs and skips. See
|
||||||
|
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||||
- **`updated_at` drives list order, so it moves only on real reading progress:** the server applies its timestamp when the row is new or `last_chapter_num` changes, and otherwise keeps the stored value — favouriting a series or recording a newly published chapter must not reorder the list. `PUT` therefore returns the row **as stored**, and clients must adopt that response rather than their own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
- **`updated_at` drives list order, so it moves only on real reading progress:** the server applies its timestamp when the row is new or `last_chapter_num` changes, and otherwise keeps the stored value — favouriting a series or recording a newly published chapter must not reorder the list. `PUT` therefore returns the row **as stored**, and clients must adopt that response rather than their own payload. See `plans/2026-07-25-bookmark-list-favorites-design.md` §4.
|
||||||
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH` (default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD` (gates the browser UI; unset disables it).
|
- **Config via env:** `API_TOKEN`, `ALLOWED_ORIGINS` (comma list), `DB_PATH`
|
||||||
|
(default `/data/bookmarks.db`), `PORT` (default `8080`), `WEB_PASSWORD`
|
||||||
|
(gates the browser UI; unset disables it),
|
||||||
|
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_INTERVAL`/`_BATCH`/`_STAGGER`
|
||||||
|
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
||||||
|
|
||||||
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
||||||
|
|
||||||
|
|||||||
+5
-3
@@ -2,15 +2,17 @@ module mangabm/backend
|
|||||||
|
|
||||||
go 1.24.1
|
go 1.24.1
|
||||||
|
|
||||||
require modernc.org/sqlite v1.34.4
|
require (
|
||||||
|
github.com/bogdanfinn/fhttp v0.6.8
|
||||||
|
github.com/bogdanfinn/tls-client v1.15.1
|
||||||
|
modernc.org/sqlite v1.34.4
|
||||||
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/andybalholm/brotli v1.2.0 // indirect
|
github.com/andybalholm/brotli v1.2.0 // indirect
|
||||||
github.com/bdandy/go-errors v1.2.2 // indirect
|
github.com/bdandy/go-errors v1.2.2 // indirect
|
||||||
github.com/bdandy/go-socks4 v1.2.3 // indirect
|
github.com/bdandy/go-socks4 v1.2.3 // indirect
|
||||||
github.com/bogdanfinn/fhttp v0.6.8 // indirect
|
|
||||||
github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
|
github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
|
||||||
github.com/bogdanfinn/tls-client v1.15.1 // indirect
|
|
||||||
github.com/bogdanfinn/utls v1.7.7-barnius // indirect
|
github.com/bogdanfinn/utls v1.7.7-barnius // indirect
|
||||||
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
|
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
|
||||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||||
|
|||||||
+16
-6
@@ -14,6 +14,8 @@ github.com/bogdanfinn/utls v1.7.7-barnius h1:OuJ497cc7F3yKNVHRsYPQdGggmk5x6+V5Zl
|
|||||||
github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
|
github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
|
||||||
github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
|
github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
|
||||||
github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
|
github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||||
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
|
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
|
||||||
@@ -34,20 +36,26 @@ github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
|
|||||||
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
|
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||||
|
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||||
|
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
|
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
|
||||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
|
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
|
||||||
|
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||||
|
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||||
|
go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
|
||||||
|
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
|
||||||
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
|
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
|
||||||
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
||||||
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
|
golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
|
||||||
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
|
||||||
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||||
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
||||||
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
|
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
|
||||||
|
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
|
||||||
|
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
|
|
||||||
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
|
||||||
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
|
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
|
||||||
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||||
@@ -55,8 +63,10 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
|||||||
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
||||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
|
golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
|
||||||
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
|
golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
|
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
|
||||||
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
|
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
|
||||||
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
|
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
|
||||||
|
|||||||
@@ -0,0 +1,76 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
|
||||||
|
fhttp "github.com/bogdanfinn/fhttp"
|
||||||
|
tls_client "github.com/bogdanfinn/tls-client"
|
||||||
|
"github.com/bogdanfinn/tls-client/profiles"
|
||||||
|
)
|
||||||
|
|
||||||
|
// maxBodyBytes caps what a single series page can cost in memory. Real pages
|
||||||
|
// measured 100-400 KB on 2026-07-26, so this is roughly 10x headroom and mostly
|
||||||
|
// guards against a proxy handing back something enormous.
|
||||||
|
const maxBodyBytes = 4 << 20
|
||||||
|
|
||||||
|
// chromeUA matches the client profile below. A Chrome fingerprint paired with a
|
||||||
|
// non-Chrome user agent is itself a signal.
|
||||||
|
const chromeUA = "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 " +
|
||||||
|
"(KHTML, like Gecko) Chrome/133.0.0.0 Mobile Safari/537.36"
|
||||||
|
|
||||||
|
// tlsFetcher fetches series pages with a Chrome TLS fingerprint.
|
||||||
|
//
|
||||||
|
// Plain net/http was verified working against both sites on 2026-07-26, so this
|
||||||
|
// is not fixing an observed block — it is deliberate defence-in-depth against a
|
||||||
|
// future fingerprint-based one, chosen up front rather than reacted to later.
|
||||||
|
// The library is pure Go, so CGO_ENABLED=0, the static binary, and the
|
||||||
|
// distroless image are all unaffected.
|
||||||
|
type tlsFetcher struct {
|
||||||
|
client tls_client.HttpClient
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ fetcher = (*tlsFetcher)(nil)
|
||||||
|
|
||||||
|
func newTLSFetcher() (*tlsFetcher, error) {
|
||||||
|
c, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(),
|
||||||
|
tls_client.WithTimeoutSeconds(30),
|
||||||
|
tls_client.WithClientProfile(profiles.Chrome_133),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("new tls client: %w", err)
|
||||||
|
}
|
||||||
|
return &tlsFetcher{client: c}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get fetches url and returns the body and status. Redirects are followed: the
|
||||||
|
// demonic chapter anchors are a redirect form, and asura has moved domains
|
||||||
|
// before.
|
||||||
|
func (f *tlsFetcher) Get(ctx context.Context, url string) (string, int, error) {
|
||||||
|
req, err := fhttp.NewRequest(fhttp.MethodGet, url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return "", 0, fmt.Errorf("build request %q: %w", url, err)
|
||||||
|
}
|
||||||
|
req = req.WithContext(ctx)
|
||||||
|
// Header order is part of what is being fingerprinted, so it is stated
|
||||||
|
// explicitly instead of left to Go's map iteration order.
|
||||||
|
req.Header = fhttp.Header{
|
||||||
|
"user-agent": {chromeUA},
|
||||||
|
"accept": {"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
|
||||||
|
"accept-language": {"en-US,en;q=0.9"},
|
||||||
|
fhttp.HeaderOrderKey: {"user-agent", "accept", "accept-language"},
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := f.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return "", 0, fmt.Errorf("get %q: %w", url, err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
body, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes))
|
||||||
|
if err != nil {
|
||||||
|
return "", resp.StatusCode, fmt.Errorf("read %q: %w", url, err)
|
||||||
|
}
|
||||||
|
return string(body), resp.StatusCode, nil
|
||||||
|
}
|
||||||
+134
@@ -7,6 +7,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
@@ -20,8 +21,34 @@ type Config struct {
|
|||||||
Port string
|
Port string
|
||||||
// WebPassword gates the browser UI. Empty disables the web routes entirely.
|
// WebPassword gates the browser UI. Empty disables the web routes entirely.
|
||||||
WebPassword string
|
WebPassword string
|
||||||
|
// LatestPoll configures the background latest-chapter fetcher.
|
||||||
|
LatestPoll LatestPoll
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LatestPoll configures the background latest-chapter poller.
|
||||||
|
//
|
||||||
|
// Sizing: batch x (cooldown / interval) is how many series hold a true cooldown
|
||||||
|
// cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this
|
||||||
|
// deployment. Past that nothing breaks; the effective cadence stretches to
|
||||||
|
// N x interval / batch and the oldest-checked-first ordering keeps it uniform.
|
||||||
|
type LatestPoll struct {
|
||||||
|
Enabled bool
|
||||||
|
Cooldown time.Duration
|
||||||
|
Interval time.Duration
|
||||||
|
Stagger time.Duration
|
||||||
|
Batch int
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultPollCooldown = time.Hour
|
||||||
|
defaultPollInterval = 10 * time.Minute
|
||||||
|
defaultPollStagger = 20 * time.Second
|
||||||
|
defaultPollBatch = 14
|
||||||
|
// minPollCooldown keeps a typo from turning a polite background check into
|
||||||
|
// a hammer against sites that are already bot-scoring us.
|
||||||
|
minPollCooldown = 15 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
func envOr(key, def string) string {
|
func envOr(key, def string) string {
|
||||||
if v := os.Getenv(key); v != "" {
|
if v := os.Getenv(key); v != "" {
|
||||||
return v
|
return v
|
||||||
@@ -29,12 +56,83 @@ func envOr(key, def string) string {
|
|||||||
return def
|
return def
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// envBool reads a boolean env var. Anything unrecognised falls back to def.
|
||||||
|
func envBool(key string, def bool) bool {
|
||||||
|
switch v := strings.ToLower(strings.TrimSpace(os.Getenv(key))); v {
|
||||||
|
case "":
|
||||||
|
return def
|
||||||
|
case "0", "false", "no", "off":
|
||||||
|
return false
|
||||||
|
case "1", "true", "yes", "on":
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
log.Printf("config: %s=%q is not a boolean, using %v", key, v, def)
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// envDuration reads a duration env var. An unparseable or non-positive value
|
||||||
|
// falls back to def and logs rather than failing startup: the poller is an
|
||||||
|
// enhancement, and a typo in one of its knobs must not stop bookmark sync.
|
||||||
|
func envDuration(key string, def time.Duration) time.Duration {
|
||||||
|
raw := strings.TrimSpace(os.Getenv(key))
|
||||||
|
if raw == "" {
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
d, err := time.ParseDuration(raw)
|
||||||
|
if err != nil || d <= 0 {
|
||||||
|
log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def)
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
|
||||||
|
// envInt reads a positive integer env var, with the same fallback policy.
|
||||||
|
func envInt(key string, def int) int {
|
||||||
|
raw := strings.TrimSpace(os.Getenv(key))
|
||||||
|
if raw == "" {
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
n, err := strconv.Atoi(raw)
|
||||||
|
if err != nil || n <= 0 {
|
||||||
|
log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def)
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadLatestPoll reads the poller's settings, clamping anything that would make
|
||||||
|
// it antisocial.
|
||||||
|
func loadLatestPoll() LatestPoll {
|
||||||
|
p := LatestPoll{
|
||||||
|
Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true),
|
||||||
|
Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown),
|
||||||
|
Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval),
|
||||||
|
Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger),
|
||||||
|
Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch),
|
||||||
|
}
|
||||||
|
if p.Cooldown < minPollCooldown {
|
||||||
|
log.Printf("config: cooldown %s is below the %s floor, clamping", p.Cooldown, minPollCooldown)
|
||||||
|
p.Cooldown = minPollCooldown
|
||||||
|
}
|
||||||
|
// batch x stagger has to fit inside one tick or a batch is still running
|
||||||
|
// when the next one is due. Run() serialises them, so this degrades to a
|
||||||
|
// slower cadence rather than to overlapping fetches — worth a warning, not
|
||||||
|
// a failure.
|
||||||
|
if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval {
|
||||||
|
log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick",
|
||||||
|
p.Batch, p.Stagger, span, p.Interval)
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
func loadConfig() Config {
|
func loadConfig() Config {
|
||||||
c := Config{
|
c := Config{
|
||||||
Token: os.Getenv("API_TOKEN"),
|
Token: os.Getenv("API_TOKEN"),
|
||||||
DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
|
DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
|
||||||
Port: envOr("PORT", "8080"),
|
Port: envOr("PORT", "8080"),
|
||||||
WebPassword: os.Getenv("WEB_PASSWORD"),
|
WebPassword: os.Getenv("WEB_PASSWORD"),
|
||||||
|
LatestPoll: loadLatestPoll(),
|
||||||
}
|
}
|
||||||
for _, o := range strings.Split(os.Getenv("ALLOWED_ORIGINS"), ",") {
|
for _, o := range strings.Split(os.Getenv("ALLOWED_ORIGINS"), ",") {
|
||||||
if o = strings.TrimSpace(o); o != "" {
|
if o = strings.TrimSpace(o); o != "" {
|
||||||
@@ -87,6 +185,13 @@ func main() {
|
|||||||
}
|
}
|
||||||
defer store.Close()
|
defer store.Close()
|
||||||
|
|
||||||
|
// The poller is off the request path entirely: if it cannot start, the
|
||||||
|
// service still serves bookmarks and the userscript still captures latest
|
||||||
|
// chapters on its own.
|
||||||
|
pollCtx, stopPoll := context.WithCancel(context.Background())
|
||||||
|
defer stopPoll()
|
||||||
|
startLatestPoller(pollCtx, store, cfg.LatestPoll)
|
||||||
|
|
||||||
srv := &http.Server{
|
srv := &http.Server{
|
||||||
Addr: ":" + cfg.Port,
|
Addr: ":" + cfg.Port,
|
||||||
Handler: newRouter(store, cfg),
|
Handler: newRouter(store, cfg),
|
||||||
@@ -105,9 +210,38 @@ func main() {
|
|||||||
<-stop
|
<-stop
|
||||||
|
|
||||||
log.Println("shutting down")
|
log.Println("shutting down")
|
||||||
|
// Stop polling before draining requests, so an in-flight series fetch does
|
||||||
|
// not hold the process open past the shutdown deadline.
|
||||||
|
stopPoll()
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
if err := srv.Shutdown(ctx); err != nil {
|
if err := srv.Shutdown(ctx); err != nil {
|
||||||
log.Printf("shutdown: %v", err)
|
log.Printf("shutdown: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// startLatestPoller launches the background poller unless it is disabled or its
|
||||||
|
// HTTP client cannot be built. Any problem here is logged and skipped: this
|
||||||
|
// feature going missing degrades the service to userscript-only latest-chapter
|
||||||
|
// tracking, which is exactly how it behaved before.
|
||||||
|
func startLatestPoller(ctx context.Context, store *Store, cfg LatestPoll) {
|
||||||
|
if !cfg.Enabled {
|
||||||
|
log.Println("latest-chapter poller: disabled by config")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f, err := newTLSFetcher()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("latest-chapter poller: disabled, cannot build client: %v", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p := &latestPoller{
|
||||||
|
store: store,
|
||||||
|
fetch: f,
|
||||||
|
now: time.Now,
|
||||||
|
cooldown: cfg.Cooldown,
|
||||||
|
interval: cfg.Interval,
|
||||||
|
stagger: cfg.Stagger,
|
||||||
|
batch: cfg.Batch,
|
||||||
|
}
|
||||||
|
go p.Run(ctx)
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,115 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLoadLatestPollDefaults(t *testing.T) {
|
||||||
|
for _, k := range []string{
|
||||||
|
"LATEST_CHAPTER_POLL_ENABLED", "LATEST_CHAPTER_POLL_COOLDOWN",
|
||||||
|
"LATEST_CHAPTER_POLL_INTERVAL", "LATEST_CHAPTER_POLL_STAGGER",
|
||||||
|
"LATEST_CHAPTER_POLL_BATCH",
|
||||||
|
} {
|
||||||
|
t.Setenv(k, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
got := loadLatestPoll()
|
||||||
|
want := LatestPoll{
|
||||||
|
Enabled: true,
|
||||||
|
Cooldown: time.Hour,
|
||||||
|
Interval: 10 * time.Minute,
|
||||||
|
Stagger: 20 * time.Second,
|
||||||
|
Batch: 14,
|
||||||
|
}
|
||||||
|
if got != want {
|
||||||
|
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadLatestPollEnabledParsing(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
raw string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"", true}, {"1", true}, {"true", true}, {"TRUE", true}, {"yes", true}, {"on", true},
|
||||||
|
{"0", false}, {"false", false}, {"False", false}, {"no", false}, {"off", false},
|
||||||
|
{"maybe", true}, // unparseable falls back to the default
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run("raw="+tt.raw, func(t *testing.T) {
|
||||||
|
t.Setenv("LATEST_CHAPTER_POLL_ENABLED", tt.raw)
|
||||||
|
if got := loadLatestPoll().Enabled; got != tt.want {
|
||||||
|
t.Fatalf("Enabled = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadLatestPollClampsAndFallsBack(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
env map[string]string
|
||||||
|
wantFrom func(LatestPoll) any
|
||||||
|
want any
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "cooldown below the floor is clamped up",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "1m"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Cooldown },
|
||||||
|
want: 15 * time.Minute,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "cooldown at the floor is kept",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "15m"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Cooldown },
|
||||||
|
want: 15 * time.Minute,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a valid override is honoured",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "5m"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Interval },
|
||||||
|
want: 5 * time.Minute,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "an unparseable duration falls back",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "ten minutes"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Interval },
|
||||||
|
want: 10 * time.Minute,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a zero duration falls back",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_STAGGER": "0s"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Stagger },
|
||||||
|
want: 20 * time.Second,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a valid batch is honoured",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "30"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Batch },
|
||||||
|
want: 30,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a negative batch falls back",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "-5"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Batch },
|
||||||
|
want: 14,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "a non-numeric batch falls back",
|
||||||
|
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "lots"},
|
||||||
|
wantFrom: func(p LatestPoll) any { return p.Batch },
|
||||||
|
want: 14,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
for k, v := range tt.env {
|
||||||
|
t.Setenv(k, v)
|
||||||
|
}
|
||||||
|
if got := tt.wantFrom(loadLatestPoll()); got != tt.want {
|
||||||
|
t.Fatalf("got %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user