62772e1eaa
Adds a background goroutine to the backend that re-checks each bookmarked series' newest published chapter on its own schedule, so `latest_chapter` stays fresh even when the manga sites are never opened in a browser.
This is a *second, parallel* signal, not a replacement: the userscript keeps its own `maybeCaptureLatestOnSeriesPage` / `backgroundRefreshLatest` logic, unchanged. `userscript/manga-bookmark.user.js` is byte-identical to `main`.
## How it works
One ticker goroutine in the same binary. Each wake it asks SQLite for bookmarks whose `latest_checked_at` has aged past a per-bookmark cooldown, fetches those series pages through a Chrome-fingerprinted HTTP client, extracts the max chapter number with a per-site regex, and writes it back through `Store.Get` + `Store.Upsert`. Every failure path logs and moves on.
Two independent clocks:
- **cooldown** — how long one bookmark rests between checks, enforced by the `WHERE` clause in `Store.DueForLatestCheck`, not by a timer.
- **interval** — how often the goroutine wakes and looks.
Shortening the interval therefore cannot shorten anyone's cooldown; it only makes the poller wake and find nothing due more often.
The row is stamped **before** the fetch, so an error, a timeout, or a shutdown mid-request still consumes the cooldown — a renamed or challenged series waits out a full cooldown instead of being retried every tick.
## Design decisions worth reviewing
**`latest_checked_at` is deliberately absent from the `Bookmark` struct and from `bookmarkColumns`.** `PUT /bookmarks/{key}` decodes a whole `Bookmark` and `Upsert` writes every column it knows about, so a userscript PUT — which has no idea this field exists — would write a zero and reset the cooldown, making the poller re-fetch that series on every tick for as long as the user kept reading it. Two tests guard this: `TestUpsertPreservesLatestCheckedAt` and `TestPutDoesNotClobberLatestCheckedAt`, the latter driving a real router PUT with a userscript-shaped body.
**`updated_at` never moves on a latest-chapter bump.** All chapter writes go through `Store.Get` + `Store.Upsert`, so the existing `CASE` keeps the stored timestamp when only `latest_chapter_num` changes and the bookmark list does not reorder. `TestRunOnceDoesNotReorderList` asserts both the timestamp and the `List()` head position.
**Fetches use `bogdanfinn/tls-client` with a Chrome profile.** Plain `net/http` was verified working against both sites on 2026-07-26, so this is not fixing an observed block — it is deliberate defence-in-depth against a future fingerprint-based one. The library is pure Go, so `CGO_ENABLED=0`, the static binary, and the distroless image are all unaffected. It does require the Go floor to move 1.23 → 1.24.
**`checkOne` validates before spending a request.** `series_url` is entirely client-supplied through `PUT /bookmarks/{key}`, so without a guard the poller would issue GETs from the server's own network position to any URL a token holder writes. The check requires a known site and an `https` URL with a non-empty host, and sits *after* the cooldown stamp so an unfetchable row is retried at cooldown pace rather than hot-looping.
## Config
Five new env vars, all with defaults sized for this deployment, all wired through `docker-compose.yml`:
| Variable | Default | Meaning |
| --- | --- | --- |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | Kill switch |
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Per series, floored at `15m` |
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often to wake |
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake |
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch |
`batch × (cooldown / interval)` = 84 series hold a true cooldown cadence at these defaults. Past that nothing breaks: the cadence stretches uniformly and the oldest-checked-first ordering keeps it fair. Bad values log and fall back rather than failing startup — the poller is an enhancement, and a typo in one of its knobs must not stop bookmark sync.
## Known limitation (accepted, documented)
The poller's `Store.Get` + `Store.Upsert` is not wrapped in a single transaction. If a userscript `PUT` commits in the sub-millisecond window between the two, the poller writes back its stale re-read — reverting that progress and, since the stored `last_chapter_num` now differs, tripping the `updated_at` `CASE` and reordering the list.
Accepted rather than fixed for a single-user deployment: the window is one SELECT wide, the poller only writes when a chapter number actually changed, and the next read self-heals it. The alternative — a transactional read-modify-write — means moving or duplicating the `updated_at` `CASE` that four tests and the whole list-ordering invariant depend on. Recorded in `CLAUDE.md` next to the poller's architecture bullet so it is not a silent trap.
## Testing
- Full suite green, including `-race`; `go vet` clean; `CGO_ENABLED=0` static build and `docker compose build` both pass on the bumped `golang:1.24-alpine`.
- No test touches the network: the `fetcher` interface exists so tests inject a fake, and no test imports `tls-client` or reaches either manga site.
- Extraction is fixture-driven against markup trimmed from real pages (2026-07-26), including a Cloudflare challenge page, cross-series chapter links, decimal chapters, and both raw `&` and `&` forms.
- Poller tests cover the no-reorder invariant, cooldown enforcement across passes, batch limiting, one bad series not stalling a batch, downward correction on a retracted chapter, cancelled contexts, and all four failure shapes still consuming the cooldown.
- Migration from a pre-column database has its own test — `newTestStore` takes the `CREATE TABLE` path, so the `ALTER TABLE` path would otherwise be untested.
- **Live smoke test:** real server, real fetch of asurascans.com. Log showed `latest is now Chapter 181` and `due=1 checked=1`; `GET /bookmarks` returned `latest_chapter_num: 181` with `updated_at` byte-identical to the PUT that created the row — the no-reorder invariant confirmed against a live site, not just a fake.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #2
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
248 lines
7.3 KiB
Go
248 lines
7.3 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"log"
|
|
"net/http"
|
|
"os"
|
|
"os/signal"
|
|
"strconv"
|
|
"strings"
|
|
"syscall"
|
|
"time"
|
|
)
|
|
|
|
// Config holds all runtime settings, sourced from environment variables.
|
|
type Config struct {
|
|
Token string
|
|
AllowedOrigins []string
|
|
DBPath string
|
|
Port string
|
|
// WebPassword gates the browser UI. Empty disables the web routes entirely.
|
|
WebPassword string
|
|
// LatestPoll configures the background latest-chapter fetcher.
|
|
LatestPoll LatestPoll
|
|
}
|
|
|
|
// LatestPoll configures the background latest-chapter poller.
|
|
//
|
|
// Sizing: batch x (cooldown / interval) is how many series hold a true cooldown
|
|
// cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this
|
|
// deployment. Past that nothing breaks; the effective cadence stretches to
|
|
// N x interval / batch and the oldest-checked-first ordering keeps it uniform.
|
|
type LatestPoll struct {
|
|
Enabled bool
|
|
Cooldown time.Duration
|
|
Interval time.Duration
|
|
Stagger time.Duration
|
|
Batch int
|
|
}
|
|
|
|
const (
|
|
defaultPollCooldown = time.Hour
|
|
defaultPollInterval = 10 * time.Minute
|
|
defaultPollStagger = 20 * time.Second
|
|
defaultPollBatch = 14
|
|
// minPollCooldown keeps a typo from turning a polite background check into
|
|
// a hammer against sites that are already bot-scoring us.
|
|
minPollCooldown = 15 * time.Minute
|
|
)
|
|
|
|
func envOr(key, def string) string {
|
|
if v := os.Getenv(key); v != "" {
|
|
return v
|
|
}
|
|
return def
|
|
}
|
|
|
|
// envBool reads a boolean env var. Anything unrecognised falls back to def.
|
|
func envBool(key string, def bool) bool {
|
|
switch v := strings.ToLower(strings.TrimSpace(os.Getenv(key))); v {
|
|
case "":
|
|
return def
|
|
case "0", "false", "no", "off":
|
|
return false
|
|
case "1", "true", "yes", "on":
|
|
return true
|
|
default:
|
|
log.Printf("config: %s=%q is not a boolean, using %v", key, v, def)
|
|
return def
|
|
}
|
|
}
|
|
|
|
// envDuration reads a duration env var. An unparseable or non-positive value
|
|
// falls back to def and logs rather than failing startup: the poller is an
|
|
// enhancement, and a typo in one of its knobs must not stop bookmark sync.
|
|
func envDuration(key string, def time.Duration) time.Duration {
|
|
raw := strings.TrimSpace(os.Getenv(key))
|
|
if raw == "" {
|
|
return def
|
|
}
|
|
d, err := time.ParseDuration(raw)
|
|
if err != nil || d <= 0 {
|
|
log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def)
|
|
return def
|
|
}
|
|
return d
|
|
}
|
|
|
|
// envInt reads a positive integer env var, with the same fallback policy.
|
|
func envInt(key string, def int) int {
|
|
raw := strings.TrimSpace(os.Getenv(key))
|
|
if raw == "" {
|
|
return def
|
|
}
|
|
n, err := strconv.Atoi(raw)
|
|
if err != nil || n <= 0 {
|
|
log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def)
|
|
return def
|
|
}
|
|
return n
|
|
}
|
|
|
|
// loadLatestPoll reads the poller's settings, clamping anything that would make
|
|
// it antisocial.
|
|
func loadLatestPoll() LatestPoll {
|
|
p := LatestPoll{
|
|
Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true),
|
|
Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown),
|
|
Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval),
|
|
Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger),
|
|
Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch),
|
|
}
|
|
if p.Cooldown < minPollCooldown {
|
|
log.Printf("config: cooldown %s is below the %s floor, clamping", p.Cooldown, minPollCooldown)
|
|
p.Cooldown = minPollCooldown
|
|
}
|
|
// batch x stagger has to fit inside one tick or a batch is still running
|
|
// when the next one is due. Run() serialises them, so this degrades to a
|
|
// slower cadence rather than to overlapping fetches — worth a warning, not
|
|
// a failure.
|
|
if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval {
|
|
log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick",
|
|
p.Batch, p.Stagger, span, p.Interval)
|
|
}
|
|
return p
|
|
}
|
|
|
|
func loadConfig() Config {
|
|
c := Config{
|
|
Token: os.Getenv("API_TOKEN"),
|
|
DBPath: envOr("DB_PATH", "/data/bookmarks.db"),
|
|
Port: envOr("PORT", "8080"),
|
|
WebPassword: os.Getenv("WEB_PASSWORD"),
|
|
LatestPoll: loadLatestPoll(),
|
|
}
|
|
for _, o := range strings.Split(os.Getenv("ALLOWED_ORIGINS"), ",") {
|
|
if o = strings.TrimSpace(o); o != "" {
|
|
c.AllowedOrigins = append(c.AllowedOrigins, o)
|
|
}
|
|
}
|
|
return c
|
|
}
|
|
|
|
// newRouter wires routes and middleware. CORS is the outermost layer so
|
|
// preflight OPTIONS short-circuits before auth; /bookmarks* is auth-protected,
|
|
// /healthz is public.
|
|
func newRouter(store *Store, cfg Config) http.Handler {
|
|
mux := http.NewServeMux()
|
|
mux.HandleFunc("GET /healthz", healthz)
|
|
|
|
h := &bookmarkHandler{store: store}
|
|
protected := http.NewServeMux()
|
|
protected.HandleFunc("GET /bookmarks", h.list)
|
|
protected.HandleFunc("PUT /bookmarks/{key}", h.put)
|
|
protected.HandleFunc("DELETE /bookmarks/{key}", h.delete)
|
|
|
|
auth := withAuth(cfg.Token, protected)
|
|
mux.Handle("/bookmarks", auth)
|
|
mux.Handle("/bookmarks/", auth)
|
|
|
|
// The browser UI is registered only when a password is configured, so a
|
|
// deployment that forgets WEB_PASSWORD exposes nothing rather than
|
|
// exposing an unprotected list.
|
|
if cfg.WebPassword != "" {
|
|
web, err := newWebHandler(store, cfg)
|
|
if err != nil {
|
|
log.Fatalf("web handler: %v", err)
|
|
}
|
|
web.register(mux)
|
|
}
|
|
|
|
return withCORS(cfg.AllowedOrigins, mux)
|
|
}
|
|
|
|
func main() {
|
|
cfg := loadConfig()
|
|
if cfg.Token == "" {
|
|
log.Fatal("API_TOKEN is required")
|
|
}
|
|
|
|
store, err := OpenStore(cfg.DBPath)
|
|
if err != nil {
|
|
log.Fatalf("open store: %v", err)
|
|
}
|
|
defer store.Close()
|
|
|
|
// The poller is off the request path entirely: if it cannot start, the
|
|
// service still serves bookmarks and the userscript still captures latest
|
|
// chapters on its own.
|
|
pollCtx, stopPoll := context.WithCancel(context.Background())
|
|
defer stopPoll()
|
|
startLatestPoller(pollCtx, store, cfg.LatestPoll)
|
|
|
|
srv := &http.Server{
|
|
Addr: ":" + cfg.Port,
|
|
Handler: newRouter(store, cfg),
|
|
ReadHeaderTimeout: 10 * time.Second,
|
|
}
|
|
|
|
go func() {
|
|
log.Printf("listening on :%s (db=%s, origins=%v)", cfg.Port, cfg.DBPath, cfg.AllowedOrigins)
|
|
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
|
log.Fatalf("serve: %v", err)
|
|
}
|
|
}()
|
|
|
|
stop := make(chan os.Signal, 1)
|
|
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
|
|
<-stop
|
|
|
|
log.Println("shutting down")
|
|
// Stop polling before draining requests, so an in-flight series fetch does
|
|
// not hold the process open past the shutdown deadline.
|
|
stopPoll()
|
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
|
defer cancel()
|
|
if err := srv.Shutdown(ctx); err != nil {
|
|
log.Printf("shutdown: %v", err)
|
|
}
|
|
}
|
|
|
|
// startLatestPoller launches the background poller unless it is disabled or its
|
|
// HTTP client cannot be built. Any problem here is logged and skipped: this
|
|
// feature going missing degrades the service to userscript-only latest-chapter
|
|
// tracking, which is exactly how it behaved before.
|
|
func startLatestPoller(ctx context.Context, store *Store, cfg LatestPoll) {
|
|
if !cfg.Enabled {
|
|
log.Println("latest-chapter poller: disabled by config")
|
|
return
|
|
}
|
|
f, err := newTLSFetcher()
|
|
if err != nil {
|
|
log.Printf("latest-chapter poller: disabled, cannot build client: %v", err)
|
|
return
|
|
}
|
|
p := &latestPoller{
|
|
store: store,
|
|
fetch: f,
|
|
now: time.Now,
|
|
cooldown: cfg.Cooldown,
|
|
interval: cfg.Interval,
|
|
stagger: cfg.Stagger,
|
|
batch: cfg.Batch,
|
|
}
|
|
go p.Run(ctx)
|
|
}
|