feat: one Poll Lane per Site, replacing the shared pace (#100) #106
+5
-18
@@ -73,26 +73,13 @@ DISCORD_REDIRECT_URI=
|
||||
# The backend re-checks each bookmarked series' newest published chapter on its
|
||||
# own schedule, so latest_chapter stays fresh even when you never open the manga
|
||||
# sites. This runs in parallel with the userscript's own in-browser check.
|
||||
# Set to 0 to turn it off entirely.
|
||||
# Set to 0 to turn it off entirely. Pace is per Site (one Poll Lane per Site,
|
||||
# issue #100) and lives in the backend registry, not here — there is nothing
|
||||
# else to configure.
|
||||
# LATEST_CHAPTER_POLL_ENABLED=1
|
||||
#
|
||||
# Two independent clocks. COOLDOWN is how long a plain-TLS series rests between
|
||||
# checks; BROWSER_COOLDOWN is the longer rest for kagane and novelfull. INTERVAL
|
||||
# is how often the poller wakes up and looks for series past their cooldowns.
|
||||
# Shortening INTERVAL cannot shorten either cooldown.
|
||||
LATEST_CHAPTER_POLL_COOLDOWN=1h # plain-TLS per series, floor 15m
|
||||
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN=6h # browser-backed per series, floor 15m
|
||||
LATEST_CHAPTER_POLL_INTERVAL=10m # how often to wake
|
||||
LATEST_CHAPTER_POLL_BATCH=14 # series per wake
|
||||
LATEST_CHAPTER_POLL_STAGGER=20s # delay between fetches in a batch
|
||||
#
|
||||
# Uses a ticker, not an immediate first run: the first poll happens one
|
||||
# INTERVAL after startup, not at startup. A container restarting more often
|
||||
# than INTERVAL never polls.
|
||||
#
|
||||
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
|
||||
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
|
||||
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
|
||||
# Every Site rests an hour between checks and gaps ten seconds between fetches;
|
||||
# a Site with many Series tightens its own gap. See backend/internal/latest/sites.go.
|
||||
|
||||
# CDP endpoint of the browser, used for the two sites behind a Cloudflare
|
||||
# JavaScript challenge (kagane, novelfull) and by the web UI's kagane cover
|
||||
|
||||
+1
-1
@@ -86,7 +86,7 @@ _Avoid_: client report, user poll, observation, claim
|
||||
|
||||
**Acquisition**:
|
||||
The single read of a Series page made the moment the Series first exists, giving it
|
||||
both its Latest Chapter and its Cover without waiting out the Poll queue. Distinct
|
||||
both its Latest Chapter and its Cover without waiting for the Lane's pace. Distinct
|
||||
from a Poll in the two ways that matter: a Reader is present — it is triggered by
|
||||
their first Bookmark of that Series — and it is the only read that establishes a
|
||||
Cover rather than refreshing facts. It happens once in a Series's life; every later
|
||||
|
||||
@@ -93,6 +93,14 @@ the backend dials but not the password the database expects, and `bookmark-api`
|
||||
crash-loops on `password authentication failed`. Set it before §2 and leave it
|
||||
alone.
|
||||
|
||||
An `.env` written before issue #100 carries the old poll-pace names
|
||||
(`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
|
||||
`_STAGGER`). All five are dead configuration now — the pace lives in the Site
|
||||
registry (`backend/internal/latest/sites.go`), so **delete those lines** and
|
||||
keep only the kill switch `LATEST_CHAPTER_POLL_ENABLED`. Leaving them behind
|
||||
is harmless (nothing reads them) but silently misleads the next person who
|
||||
edits the file.
|
||||
|
||||
> Match `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER` to your Traefik's actual
|
||||
> names (check your Traefik static config — common alternatives: `https`,
|
||||
> `myresolver`, `cloudflare`). Wrong names = no certificate issued.
|
||||
@@ -505,7 +513,7 @@ picks up a restarted Chrome's new debugger UUID by itself.
|
||||
| kagane rows never get a `latest_chapter`; log says `browser fetcher disabled` or nothing at all | `BROWSER_WS_URL` unset. Expected before §7 is done. |
|
||||
| kagane polls all fail; log shows a 500 from `/json/version` | `BROWSER_WS_URL` names a MagicDNS hostname (or any name). Chrome's DevTools handler only accepts an IP or `localhost` — use the tailnet IP. |
|
||||
| kagane polls fail with a connection error | Home machine off, off the tailnet, or the unit is down. `tailscale ping <machine>`, then `docker compose ps` in its `chrome/`. Costs freshness only; stored covers keep serving. |
|
||||
| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series (up to `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, default 6h). |
|
||||
| kagane cover is a placeholder for a newly bookmarked series | Its cover has never been fetched and the browser is unreachable. It fills in on the next successful poll of that series. |
|
||||
| `compose` in `chrome/` errors `set BROWSER_BIND_ADDR to this machine's tailnet IP` | No `chrome/.env`, or the variable is empty. Deliberate — it has no default so an unset value cannot publish CDP to the LAN. |
|
||||
| browser container restarts, or is OOM-killed | `docker inspect bookmark-browser --format '{{.RestartCount}} {{.State.OOMKilled}}'`. The 512 MiB cap is sized against a measured 645 MiB untuned peak; a real breach is a Chrome regression worth reading `docker logs` for, not a number to raise reflexively. |
|
||||
|
||||
|
||||
@@ -53,12 +53,7 @@ covers are stored, so the library renders in full with the browser switched off.
|
||||
| `DISCORD_API_BASE` | `https://discord.com/api/v10` | Test seam — tests point it at a local stub so the real token exchange runs. |
|
||||
| `USERSCRIPT_PATH` | `/userscript/manga-bookmark.user.js` | Bindmounted file served at `/u/{token}/manga-bookmark.user.js`. |
|
||||
| `NOVEL_USERSCRIPT_PATH` | `/userscript/novel-bookmark.user.js` | Same, for the novel library. |
|
||||
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. |
|
||||
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Rest between checks of one plain-TLS series; floor `15m`. |
|
||||
| `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN` | `6h` | Rest between checks of one browser-backed series; floor `15m`. |
|
||||
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often the poller wakes. Cannot shorten either cooldown. |
|
||||
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake. Keep `BATCH × STAGGER` under `INTERVAL`. |
|
||||
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch — this is the outbound request rate. |
|
||||
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. Pace is per Site in the registry — one Poll Lane per Site, each with its own rest and gap (issue #100) — so no other knobs exist. |
|
||||
|
||||
Compose reads a few more from the same `.env` that the backend never sees:
|
||||
`POSTGRES_PASSWORD` (required — `DATABASE_URL` is built from it, and Postgres
|
||||
|
||||
+31
-16
@@ -74,20 +74,33 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
Remove's row wear ember wash, two reversible ones wear `.calm` grey.
|
||||
`--ember` stay reserved for new-chapter signal: busy bar and inline
|
||||
error use `--mute`.
|
||||
- **Latest-chapter poller:** ticker goroutine in same binary re-check
|
||||
each bookmarked series' newest published chapter from backend's own
|
||||
network access, so `latest_chapter` stay fresh when user not
|
||||
browsing. Second, parallel signal — userscript keep own
|
||||
- **Latest-chapter poller:** one goroutine per Site (a Poll Lane, issue #100),
|
||||
each re-checking that Site's bookmarked series' newest published chapter from
|
||||
backend's own network access, so `latest_chapter` stays fresh when the user
|
||||
isn't browsing. Second, parallel signal — the userscript keeps its own
|
||||
`maybeCaptureLatestOnSeriesPage`/`backgroundRefreshLatest` logic unchanged.
|
||||
Two independent clocks: per-series cooldown (`series.latest_checked_at`,
|
||||
enforced by `Store.DueForLatestCheck`'s WHERE clause) and wake interval.
|
||||
Two independent clocks: per-series rest (`series.latest_checked_at`,
|
||||
enforced by `Store.DueForLatestCheck`'s WHERE clause — `now - Rest`) and
|
||||
per-Lane gap (the Lane sleeping between fetches, `effectiveGap`). Both live
|
||||
in the Site registry (`internal/latest/sites.go`), not config: the five env
|
||||
knobs that used to size a shared pace are gone.
|
||||
The poller walks **Series, not Bookmarks** — a series referenced by several
|
||||
bookmarks is fetched once per cycle, and the due queue orders
|
||||
`reader_count DESC, latest_checked_at ASC` (ADR-0003). Series row stamped
|
||||
*before* fetch so broken series wait out full cooldown instead of retrying
|
||||
*before* fetch so broken series wait out the rest instead of retrying
|
||||
every tick; found chapter written straight to the series row via
|
||||
`Store.SetLatestChapter`, so a bookmark's `updated_at` — and the list
|
||||
order — is never touched.
|
||||
Refusals and browser loss are Lane-local: two `errChallengeHeld` in one pass
|
||||
stop that Site for `refuseBackoff` (15m) while other Lanes continue; an
|
||||
`errBrowserInterrupted` (remote Chrome restart) sets a shared Poller flag
|
||||
that makes the other browser Lanes skip their passes for the same 15m, so a
|
||||
restarting Chrome doesn't stamp one Series per Lane per pass — after the
|
||||
window the flag decays and they probe again. Browser Lanes wake Chrome only
|
||||
when 5+ Series are due or one has waited 15m (ADR-0005 on-demand browser),
|
||||
and cover work (both healing a stored source URL and filling a blank from
|
||||
the series page) runs in the background so a slow CDN can't consume a
|
||||
Lane's gap.
|
||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||
against fingerprint-based blocking; any failure log and skip. kagane, comix
|
||||
and novelfull sit behind Cloudflare JavaScript challenges the TLS client
|
||||
@@ -152,15 +165,17 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
||||
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
||||
`DISCORD_API_BASE` (default `https://discord.com/api/v10`),
|
||||
`LATEST_CHAPTER_POLL_ENABLED`/`_COOLDOWN`/`_BROWSER_COOLDOWN`/`_INTERVAL`/
|
||||
`_BATCH`/`_STAGGER` (background latest-chapter poller; defaults on,
|
||||
`1h` plain-TLS cooldown, `6h` browser cooldown, `10m`/`14`/`20s`; both
|
||||
cooldowns have a `15m` floor). The browser cooldown is longer for cost, not
|
||||
for safety: a challenged page costs seconds of a serialized single-tab
|
||||
browser, while a plain read costs one request. It buys no documented
|
||||
reduction in challenge risk — free-plan zones have no bot score and no
|
||||
published per-IP rate input, and `cf_clearance` expires in 30 minutes so
|
||||
every cadence at or above 1h re-solves anyway —
|
||||
`LATEST_CHAPTER_POLL_ENABLED` (background latest-chapter poller kill
|
||||
switch, default on). Pace is per Site in the registry (issue #100): every
|
||||
Site rests an hour and gaps ten seconds, a Site with more eligible Series
|
||||
than 360 tightens its own gap toward the 1s floor, and browser Lanes wake
|
||||
Chrome only on demand (ADR-0005). The `_COOLDOWN`/`_BROWSER_COOLDOWN`/
|
||||
`_INTERVAL`/`_BATCH`/`_STAGGER` knobs that used to size a shared pace are
|
||||
gone. The 1h rest for browser Sites is safe on documented grounds: a
|
||||
challenged page costs seconds of a serialized single-tab browser, free-plan
|
||||
zones have no bot score and no published per-IP rate input, and
|
||||
`cf_clearance` expires in 30 minutes so every cadence at or above 1h
|
||||
re-solves anyway —
|
||||
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
|
||||
`USERSCRIPT_PATH` and `NOVEL_USERSCRIPT_PATH` (files served at
|
||||
`/u/{token}/manga-bookmark.user.js` and `/u/{token}/novel-bookmark.user.js`,
|
||||
|
||||
@@ -120,7 +120,7 @@ func (a *Acquirer) acquire(ctx context.Context, sr store.Series) {
|
||||
// Stamped after success — the reverse of the poller, which stamps before
|
||||
// the fetch: the Reader is here, watching the Series they just created, so
|
||||
// a failed acquisition must leave the row due for a fast retry rather than
|
||||
// consuming the cooldown. The stamp happens even when the page read
|
||||
// consuming the rest. The stamp happens even when the page read
|
||||
// succeeded but produced no facts to persist.
|
||||
if err := a.Store.MarkLatestChecked(sr.Site, sr.SeriesID, time.Now().UnixMilli()); err != nil {
|
||||
log.Printf("acquire %q: mark checked: %v", sr.Key(), err)
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
|
||||
// challengeTimeout bounds one navigate-and-solve. A Cloudflare managed
|
||||
// challenge clears in a few seconds when it clears at all; anything longer is a
|
||||
// challenge that is not going to pass, and the caller's cooldown was already
|
||||
// challenge that is not going to pass, and the caller's rest was already
|
||||
// stamped before this ran.
|
||||
const challengeTimeout = 45 * time.Second
|
||||
|
||||
@@ -213,7 +213,7 @@ func (f *BrowserFetcher) Image(ctx context.Context, imageURL string) ([]byte, st
|
||||
|
||||
// errChallengeHeld reports that the budget ran out with the interstitial still
|
||||
// up. Distinct from a transport failure: it means "this site said no", which
|
||||
// the poller answers with a 403 and its ordinary cooldown.
|
||||
// the poller answers with a refusal backoff for that Site's Lane (issue #100).
|
||||
var errChallengeHeld = errors.New("challenge held")
|
||||
|
||||
// errBrowserInterrupted distinguishes a remote Chrome restart from the
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"errors"
|
||||
"log"
|
||||
"net/url"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"bookmarkmanager/backend/internal/store"
|
||||
@@ -28,15 +30,11 @@ type BrowserCoverFetcher interface {
|
||||
// in parallel and report the same observable fact, so whichever writes last wins
|
||||
// and neither needs to know about the other.
|
||||
//
|
||||
// Two clocks, deliberately independent:
|
||||
//
|
||||
// - Interval is how often this goroutine wakes up and looks.
|
||||
// - Cooldowns are how long a series rests since its own last check. Browser-
|
||||
// backed sites use the longer BrowserCooldown.
|
||||
//
|
||||
// Cooldowns are enforced by the WHERE clause in DueForLatestCheck rather than
|
||||
// by any timer. Shortening Interval therefore cannot shorten anyone's cooldown;
|
||||
// it only makes the poller wake up and find nothing due more often.
|
||||
// Every Site gets its own Poll Lane: one independent stream of Polls with its
|
||||
// own pace, running concurrently with every other Site's (issue #100). Rest
|
||||
// time and gap live in the Site registry, not here — see sites.go. Rest is
|
||||
// enforced by the WHERE clause in DueForLatestCheck rather than by any timer;
|
||||
// the gap is enforced by the Lane sleeping between fetches.
|
||||
type Poller struct {
|
||||
Store *store.Store
|
||||
Fetch Fetcher
|
||||
@@ -50,11 +48,19 @@ type Poller struct {
|
||||
// same failure-isolated prefetch path.
|
||||
CoverBytesFetch CoverBytesFetcher
|
||||
Now func() time.Time // injected so tests can freeze it
|
||||
Cooldown time.Duration
|
||||
BrowserCooldown time.Duration
|
||||
Interval time.Duration
|
||||
Stagger time.Duration
|
||||
Batch int
|
||||
|
||||
// refuseUntil gates a Site's Lane after it refused twice in one run: no
|
||||
// Series of that Site is attempted again before this time (issue #100).
|
||||
// browserDownAt is when a browser Lane last lost the sidecar; the other
|
||||
// browser Lanes skip their passes for the next refuseBackoff, so a
|
||||
// restarting Chrome does not stamp one Series per pass per Lane (story 20).
|
||||
mu sync.Mutex
|
||||
refuseUntil map[string]time.Time
|
||||
browserDownAt time.Time
|
||||
// coverWG tracks in-flight cover work. Covers heal in the background so a
|
||||
// slow cover host cannot delay the next Series-page Poll; tests join it
|
||||
// before asserting on cover fetches.
|
||||
coverWG sync.WaitGroup
|
||||
}
|
||||
|
||||
// fillBlankCover gives a Series its Cover when it has none. The blank state is
|
||||
@@ -76,7 +82,14 @@ func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, cover stri
|
||||
if cover == "" {
|
||||
return
|
||||
}
|
||||
// Like healCover, the fill runs in the background: a large import of
|
||||
// blanks would otherwise pay one og:image fetch per Series against the
|
||||
// Lane's gap (issue #100, story 12).
|
||||
p.coverWG.Add(1)
|
||||
go func() {
|
||||
defer p.coverWG.Done()
|
||||
p.storeCover(ctx, sr, cover)
|
||||
}()
|
||||
}
|
||||
|
||||
// prefetchCover heals Series that already carry a third-party source URL but
|
||||
@@ -143,72 +156,248 @@ func fetcherFor(site string, browser, tls Fetcher) Fetcher {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Run polls until ctx is cancelled.
|
||||
//
|
||||
// runOnce is called synchronously, so a batch that overruns the tick delays the
|
||||
// next one instead of stacking a second batch on top of it. That is the intended
|
||||
// failure mode for a misconfigured batch x stagger: a slower cadence, never
|
||||
// concurrent fetch storms.
|
||||
// Run polls until ctx is cancelled: one goroutine per Site Lane, each pacing
|
||||
// itself by the Site's effective gap. Lanes share nothing but the store and
|
||||
// the browser fetcher's single tab (BrowserFetcher serializes itself), so one
|
||||
// hostile Site burns only its own budget.
|
||||
// laneNames returns every registry Site in the deterministic order both Run
|
||||
// and runOnce iterate: sorted, so lane behaviour and its tests agree on who
|
||||
// runs first.
|
||||
func laneNames() []string {
|
||||
names := make([]string, 0, len(sites))
|
||||
for name := range sites {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
func (p *Poller) Run(ctx context.Context) {
|
||||
log.Printf("latest-chapter poller: interval=%s cooldown=%s browser-cooldown=%s batch=%d stagger=%s",
|
||||
p.Interval, p.Cooldown, p.BrowserCooldown, p.Batch, p.Stagger)
|
||||
t := time.NewTicker(p.Interval)
|
||||
defer t.Stop()
|
||||
names := laneNames()
|
||||
log.Printf("latest-chapter poller: %d lanes, rest=%s gap=%s", len(names), defaultRest, defaultGap)
|
||||
for _, name := range names {
|
||||
go p.lane(ctx, name)
|
||||
}
|
||||
<-ctx.Done()
|
||||
log.Println("latest-chapter poller: stopped")
|
||||
}
|
||||
|
||||
// lane is one Site's Poll Lane: one pass, then sleep the pace the pass
|
||||
// reported, then another pass, until ctx is cancelled. The sleep is the whole
|
||||
// pace discipline — a pass that fetched nothing still reports its gap so the
|
||||
// Lane wakes often enough to notice Series as they become due. The pass shares
|
||||
// the Poller's browser-down state, so a sidecar loss is noticed once and the
|
||||
// other browser Lanes skip passes until the backoff window decays.
|
||||
func (p *Poller) lane(ctx context.Context, name string) {
|
||||
for {
|
||||
pace := p.runLanePass(ctx, name, true)
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Println("latest-chapter poller: stopped")
|
||||
return
|
||||
case <-t.C:
|
||||
p.runOnce(ctx)
|
||||
case <-time.After(pace):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// runOnce processes one batch of due series.
|
||||
// runOnce processes one round: one pass of every Lane, back to back, no real
|
||||
// time passing. This is the deterministic entry point the test suite drives a
|
||||
// round at a time. The production Run loop does the same work paced by its own
|
||||
// sleeps; pacing is the only difference.
|
||||
func (p *Poller) runOnce(ctx context.Context) {
|
||||
now := p.Now()
|
||||
cutoff := now.Add(-p.Cooldown).UnixMilli()
|
||||
browserCutoff := now.Add(-p.BrowserCooldown).UnixMilli()
|
||||
due, err := p.Store.DueForLatestCheck(cutoff, browserCutoff, browserBackedSites(), p.Batch)
|
||||
if err != nil {
|
||||
log.Printf("latest poll: due query: %v", err)
|
||||
return
|
||||
for _, name := range laneNames() {
|
||||
p.runLanePass(ctx, name, false)
|
||||
}
|
||||
}
|
||||
|
||||
// runLanePass processes one pass of one Site's Lane: select the due Series,
|
||||
// pace through them, and report how long the Lane should wait before its next
|
||||
// pass. paced spaces consecutive fetches by the Site's effective gap — the
|
||||
// production Lane's rate limit; the deterministic test entry runs back to back.
|
||||
func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration {
|
||||
now := p.Now()
|
||||
if until := p.refusalBackoff(name); now.Before(until) {
|
||||
// Cooling down after a refusal: do not attempt this Site at all.
|
||||
return until.Sub(now)
|
||||
}
|
||||
if isBrowserSite(name) {
|
||||
if downFor, down := p.browserDownFor(now); down && downFor < refuseBackoff {
|
||||
// A sibling browser Lane lost the sidecar within the backoff
|
||||
// window: skip this pass, so a restarting Chrome does not stamp
|
||||
// this Site's Series one pass at a time. After refuseBackoff the
|
||||
// flag decays and the Lane probes again (issue #100, story 20).
|
||||
log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor)
|
||||
return refuseBackoff - downFor
|
||||
}
|
||||
}
|
||||
s := sites[name]
|
||||
f := fetcherFor(name, p.BrowserFetch, p.Fetch)
|
||||
if f == nil {
|
||||
// No fetcher at all right now (browser absent, no fallback): every
|
||||
// Series stays unstamped and due, so a browser that appears after a
|
||||
// restart finds its full queue waiting (issue #100).
|
||||
return defaultGap
|
||||
}
|
||||
|
||||
due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli())
|
||||
if err != nil {
|
||||
log.Printf("latest poll %s: due query: %v", name, err)
|
||||
return defaultGap
|
||||
}
|
||||
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) {
|
||||
// Below both thresholds Chrome stays asleep (ADR-0005 on-demand
|
||||
// browser): waking it for a single Poll would cost a challenge solve
|
||||
// per request.
|
||||
return defaultGap
|
||||
}
|
||||
if s.Browser != nil {
|
||||
// Browser Lanes share one tab, so their combined ceiling is about 360
|
||||
// Polls an hour. When they cannot keep up, the wait past the rest time
|
||||
// grows — log by how much, every pass, so the decision to give them
|
||||
// more pages is made from a measurement rather than a guess.
|
||||
if behind := maxSeriesWait(due, now, s.Rest) - s.Rest; behind > 0 {
|
||||
log.Printf("latest poll %s: browser lane behind by %s (browser Sites cannot keep up with the hour)", name, behind)
|
||||
}
|
||||
}
|
||||
|
||||
eligible, err := p.Store.EligibleSeriesCount(name)
|
||||
if err != nil {
|
||||
log.Printf("latest poll %s: eligible count: %v", name, err)
|
||||
return defaultGap
|
||||
}
|
||||
gap, clamped := effectiveGap(s, eligible)
|
||||
if clamped {
|
||||
log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible)
|
||||
}
|
||||
if eligible == 0 {
|
||||
// Nothing to poll for the foreseeable future; sleep a full rest instead
|
||||
// of re-querying every gap.
|
||||
return s.Rest
|
||||
}
|
||||
|
||||
refusals := 0
|
||||
checked := 0
|
||||
for i, sr := range due {
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
// Staggered rather than fired together: a burst of simultaneous requests
|
||||
// from one server IP is the traffic shape most likely to move that IP's
|
||||
// bot score. This is the server-side analogue of the userscript's "one
|
||||
// series per navigation ... indistinguishable from browsing" (L455-456).
|
||||
stopped := false
|
||||
if i > 0 && p.Stagger > 0 {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
stopped = true
|
||||
case <-time.After(p.Stagger):
|
||||
}
|
||||
}
|
||||
if stopped {
|
||||
if refusals >= 2 {
|
||||
// This Site refused twice in a row: the remaining Series are left
|
||||
// unstamped and due, and the Lane waits refuseBackoff before
|
||||
// trying it again.
|
||||
break
|
||||
}
|
||||
p.checkOne(ctx, sr)
|
||||
if paced && i > 0 {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
break
|
||||
case <-time.After(gap):
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if err := p.checkOne(ctx, sr); err != nil {
|
||||
switch {
|
||||
case errors.Is(err, errChallengeHeld):
|
||||
refusals++
|
||||
case errors.Is(err, errBrowserInterrupted):
|
||||
p.setBrowserDown(now)
|
||||
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, refuseBackoff)
|
||||
return gap
|
||||
default:
|
||||
refusals = 0
|
||||
}
|
||||
} else {
|
||||
refusals = 0
|
||||
}
|
||||
checked++
|
||||
}
|
||||
// due vs checked is how you tell which constraint is binding: ticks that
|
||||
// report due=0 mean the cooldown is the limit, ticks that report due==batch
|
||||
// every time mean throughput is.
|
||||
log.Printf("latest poll: due=%d checked=%d", len(due), checked)
|
||||
if checked > 0 {
|
||||
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), checked)
|
||||
}
|
||||
if refusals >= 2 {
|
||||
p.setRefusalBackoff(name, now.Add(refuseBackoff))
|
||||
log.Printf("latest poll %s: refused twice this run, waiting %s", name, refuseBackoff)
|
||||
return refuseBackoff
|
||||
}
|
||||
return gap
|
||||
}
|
||||
|
||||
func (p *Poller) refusalBackoff(name string) time.Time {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
return p.refuseUntil[name]
|
||||
}
|
||||
|
||||
func (p *Poller) setRefusalBackoff(name string, until time.Time) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.refuseUntil == nil {
|
||||
p.refuseUntil = make(map[string]time.Time)
|
||||
}
|
||||
p.refuseUntil[name] = until
|
||||
}
|
||||
|
||||
// setBrowserDown records when a browser Lane lost the sidecar. It is Poller
|
||||
// state rather than pass state so the other browser Lanes see it too.
|
||||
func (p *Poller) setBrowserDown(now time.Time) {
|
||||
p.mu.Lock()
|
||||
p.browserDownAt = now
|
||||
p.mu.Unlock()
|
||||
}
|
||||
|
||||
// browserDownFor reports how long the sidecar has been down and that it is
|
||||
// down at all — the zero time means never down, which must not read as a
|
||||
// zero-duration loss. The window decays: once refuseBackoff passes without a
|
||||
// fresh loss, Lanes probe again.
|
||||
func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.browserDownAt.IsZero() {
|
||||
return 0, false
|
||||
}
|
||||
return now.Sub(p.browserDownAt), true
|
||||
}
|
||||
|
||||
// isBrowserSite reports whether the registry routes this Site's page through
|
||||
// the browser sidecar.
|
||||
func isBrowserSite(name string) bool {
|
||||
return sites[name].Browser != nil
|
||||
}
|
||||
|
||||
// browserWakeDue reports whether a browser Lane may start a run: five or more
|
||||
// of its Series are due, or any one of them has been due for browserWakeAge.
|
||||
// Below both thresholds the Lane leaves Chrome asleep — Series Polled together
|
||||
// become due together, so the group naturally stays clustered, and the age
|
||||
// rule exists to stop a Series that drifted out of the group from starving.
|
||||
func browserWakeDue(due []store.Series, now time.Time, rest time.Duration) bool {
|
||||
if len(due) >= browserWakeCount {
|
||||
return true
|
||||
}
|
||||
return maxSeriesWait(due, now, rest) >= browserWakeAge
|
||||
}
|
||||
|
||||
// maxSeriesWait returns how long the most-overdue of the due Series has been
|
||||
// waiting past its due moment (0 when due is empty).
|
||||
func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration {
|
||||
var oldest time.Duration
|
||||
for _, sr := range due {
|
||||
if w := now.Sub(time.UnixMilli(sr.LatestCheckedAt).Add(rest)); w > oldest {
|
||||
oldest = w
|
||||
}
|
||||
}
|
||||
return oldest
|
||||
}
|
||||
|
||||
// checkOne re-checks one series. Every failure path here is "log and move on":
|
||||
// the poller is a best-effort enhancement, and no single bad series may stall a
|
||||
// batch or take down the process.
|
||||
func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
|
||||
// Lane or take down the process. The returned error is the page read's
|
||||
// classified outcome so the Lane can tell a refusal from a loss of the
|
||||
// browser; non-classified failures still return nil-equivalent behaviour.
|
||||
func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
|
||||
@@ -216,44 +405,46 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
|
||||
}()
|
||||
|
||||
// Stamped before the fetch, not after, so an error, a timeout, or a shutdown
|
||||
// mid-request still consumes the cooldown. Otherwise a renamed or deleted
|
||||
// series would be retried on every single tick forever. The userscript
|
||||
// stamps in the same order and for the same reason (L471-473).
|
||||
// mid-request still consumes the rest. Otherwise a renamed or deleted
|
||||
// series would be retried on every single pass forever. The userscript
|
||||
// stamps in the same order and for the same reason (L471-473). A Series
|
||||
// never reaches checkOne without a fetcher — runLanePass skips those — so
|
||||
// the stamp means "attempted", and an untried Series stays due.
|
||||
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
|
||||
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
|
||||
return
|
||||
return nil
|
||||
}
|
||||
|
||||
facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, errNotFetchable):
|
||||
// The cooldown above is already consumed, so a row that never
|
||||
// passes the gate is retried at cooldown pace rather than
|
||||
// The rest above is already consumed, so a row that never
|
||||
// passes the gate is retried at rest pace rather than
|
||||
// hot-looping.
|
||||
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
|
||||
return
|
||||
return err
|
||||
case errors.Is(err, errNoFetcher):
|
||||
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
|
||||
return
|
||||
return err
|
||||
}
|
||||
// A legacy cover heals independently of the page read: its source may
|
||||
// answer — a CDN — while the origin does not, so a fetch failure does
|
||||
// not skip the heal, matching the order the shared read replaced.
|
||||
p.prefetchCover(ctx, sr)
|
||||
p.healCover(ctx, sr)
|
||||
log.Printf("latest poll %q: %v", sr.Key(), err)
|
||||
return
|
||||
return err
|
||||
}
|
||||
// A legacy cover source is healed independently of the page read.
|
||||
p.prefetchCover(ctx, sr)
|
||||
p.healCover(ctx, sr)
|
||||
// Cover fill is independent of the chapter signal: a page that lost its
|
||||
// chapter list may keep its og:image, and a blank Series heals either way.
|
||||
p.fillBlankCover(ctx, sr, facts.Cover)
|
||||
if !facts.HasLatest {
|
||||
// Most likely a challenge page or a layout change. Either way the row is
|
||||
// already stamped, so this waits out a cooldown instead of hot-looping.
|
||||
// already stamped, so this waits out a rest instead of hot-looping.
|
||||
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen)
|
||||
return
|
||||
return nil
|
||||
}
|
||||
|
||||
// Equality, not >, mirroring the userscript (L427): a site that retracts a
|
||||
@@ -261,7 +452,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
|
||||
// against the due-query snapshot; a concurrent write in between only costs
|
||||
// one redundant UPDATE of the same absolute value, never a wrong one.
|
||||
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
|
||||
// Series-level write: the row is shared, so one update refreshes every
|
||||
@@ -270,9 +461,29 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
|
||||
// the list.
|
||||
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil {
|
||||
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
|
||||
return
|
||||
return nil
|
||||
}
|
||||
log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label)
|
||||
return nil
|
||||
}
|
||||
|
||||
// healCover runs prefetchCover in the background. Cover bytes come from a
|
||||
// different host — often a CDN — and heal once in a Series's life, so they
|
||||
// must not consume a Lane's gap: a large import with many blanks would
|
||||
// otherwise make every Latest Chapter go stale behind a slow image host
|
||||
// (issue #100).
|
||||
func (p *Poller) healCover(ctx context.Context, sr store.Series) {
|
||||
p.coverWG.Add(1)
|
||||
go func() {
|
||||
defer p.coverWG.Done()
|
||||
p.prefetchCover(ctx, sr)
|
||||
}()
|
||||
}
|
||||
|
||||
// waitCovers blocks until every in-flight cover heal finishes. Tests call it
|
||||
// after a round before asserting on cover fetches.
|
||||
func (p *Poller) waitCovers() {
|
||||
p.coverWG.Wait()
|
||||
}
|
||||
|
||||
// fetchableSeriesURL reports whether site is a Site the registry knows and
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
@@ -156,19 +157,16 @@ func (f *fakeBytesCoverFetcher) callCount() int {
|
||||
return len(f.calls)
|
||||
}
|
||||
|
||||
// newTestPoller wires a poller with a frozen clock and no stagger, so tests run
|
||||
// instantly and deterministically.
|
||||
// newTestPoller wires a poller with a frozen clock. Rest and gap come from the
|
||||
// Site registry, so tests seed checked_at relative to the one-hour rest; the
|
||||
// round entry point (runOnce) runs every Lane back to back with no real
|
||||
// pacing, so tests stay instant and deterministic.
|
||||
func newTestPoller(t *testing.T, s *store.Store, f Fetcher, at time.Time) *Poller {
|
||||
t.Helper()
|
||||
return &Poller{
|
||||
Store: s,
|
||||
Fetch: f,
|
||||
Now: func() time.Time { return at },
|
||||
Cooldown: time.Hour,
|
||||
BrowserCooldown: 6 * time.Hour,
|
||||
Interval: 10 * time.Minute,
|
||||
Stagger: 0,
|
||||
Batch: 14,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -208,9 +206,10 @@ func TestRunOncePrefetchesPublicCover(t *testing.T) {
|
||||
covers := &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"}
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverBytesFetch: covers,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if got := covers.callCount(); got != 1 {
|
||||
t.Fatalf("cover fetch calls = %d, want 1", got)
|
||||
@@ -241,9 +240,10 @@ func TestRunOnceDoesNotStoreNonImagePublicCover(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200},
|
||||
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("challenge"), contentType: "text/html"},
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if _, _, found, err := s.GetCover(coverURL); err != nil || found {
|
||||
t.Fatalf("non-image cover = found %v, err %v; want missing", found, err)
|
||||
@@ -350,23 +350,6 @@ func TestRunOnceMarksCheckedOnFailure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunOnceRespectsBatchLimit(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
for i := 0; i < 20; i++ {
|
||||
key := "asura:s" + string(rune('a'+i))
|
||||
seedForCheck(t, s, key, "https://asurascans.com/comics/"+key, 0)
|
||||
}
|
||||
|
||||
f := &fakeFetcher{body: "", status: 200}
|
||||
p := newTestPoller(t, s, f, time.UnixMilli(5_000_000))
|
||||
p.Batch = 5
|
||||
p.runOnce(context.Background())
|
||||
|
||||
if got := f.callCount(); got != 5 {
|
||||
t.Fatalf("fetched %d series, want 5 (batch limit)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the split (ADR-0003): a series referenced by several bookmarks
|
||||
// is fetched once per due cycle, not once per bookmark. Two bookmarks share a
|
||||
// series when two readers track it (issue #22).
|
||||
@@ -409,8 +392,8 @@ func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// One unreachable series must not abandon the rest of the batch.
|
||||
func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
|
||||
// One unreachable series must not abandon the rest of the Lane.
|
||||
func TestRunOnceOneBadSeriesDoesNotStallLane(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
keys := []string{"asura:a", "asura:b", "asura:c", "asura:d", "asura:e"}
|
||||
for _, k := range keys {
|
||||
@@ -436,7 +419,9 @@ func TestRunOnceOneBadSeriesDoesNotStallBatch(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunLogsCooldowns(t *testing.T) {
|
||||
// Pace now lives in the registry, not config: Run logs the lane defaults so a
|
||||
// deployment can see what the poller is doing without reading the source.
|
||||
func TestRunLogsLaneDefaults(t *testing.T) {
|
||||
var logs strings.Builder
|
||||
previous := log.Writer()
|
||||
log.SetOutput(&logs)
|
||||
@@ -444,21 +429,19 @@ func TestRunLogsCooldowns(t *testing.T) {
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
(&Poller{
|
||||
Cooldown: time.Hour,
|
||||
BrowserCooldown: 6 * time.Hour,
|
||||
Interval: time.Hour,
|
||||
}).Run(ctx)
|
||||
(&Poller{Now: func() time.Time { return time.Now() }}).Run(ctx)
|
||||
|
||||
if got := logs.String(); !strings.Contains(got, "cooldown=1h") ||
|
||||
!strings.Contains(got, "browser-cooldown=6h") {
|
||||
t.Fatalf("startup log = %q, want both cooldowns", got)
|
||||
got := logs.String()
|
||||
for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Fatalf("startup log = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The cooldown is enforced by the due query, so a second immediate pass must do
|
||||
// nothing at all — this is what makes the tick interval independent of it.
|
||||
func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
|
||||
// The rest is enforced by the due query, so a second immediate pass must do
|
||||
// nothing at all — this is what makes the Lane's sleep independent of it.
|
||||
func TestRunOnceHonoursRestAcrossPasses(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
const url = "https://asurascans.com/comics/x"
|
||||
seedForCheck(t, s, "asura:x", url, 0)
|
||||
@@ -471,56 +454,18 @@ func TestRunOnceHonoursCooldownAcrossPasses(t *testing.T) {
|
||||
if got := f.callCount(); got != 1 {
|
||||
t.Fatalf("first pass fetched %d, want 1", got)
|
||||
}
|
||||
// Same instant, and again 59 minutes later: both inside the 1h cooldown.
|
||||
// Same instant, and again 59 minutes later: both inside the 1h rest.
|
||||
p.runOnce(context.Background())
|
||||
p.Now = func() time.Time { return now.Add(59 * time.Minute) }
|
||||
p.runOnce(context.Background())
|
||||
if got := f.callCount(); got != 1 {
|
||||
t.Fatalf("fetched %d times inside the cooldown, want 1", got)
|
||||
t.Fatalf("fetched %d times inside the rest, want 1", got)
|
||||
}
|
||||
// Past the cooldown, it is due again.
|
||||
// Past the rest, it is due again.
|
||||
p.Now = func() time.Time { return now.Add(61 * time.Minute) }
|
||||
p.runOnce(context.Background())
|
||||
if got := f.callCount(); got != 2 {
|
||||
t.Fatalf("fetched %d times after the cooldown, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunOnceUsesBrowserCooldown(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
const browserKey = "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
|
||||
seedForCheck(t, s, "asura:plain", "https://asurascans.com/comics/plain", 0)
|
||||
seedForCheck(t, s, browserKey, "https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b", 0)
|
||||
|
||||
hour := time.Hour
|
||||
now := time.Unix(2*int64(hour/time.Second), 0)
|
||||
tls := &fakeFetcher{status: 200}
|
||||
browser := &fakeFetcher{status: 200}
|
||||
p := &Poller{
|
||||
Store: s,
|
||||
Fetch: tls,
|
||||
BrowserFetch: browser,
|
||||
Now: func() time.Time { return now },
|
||||
Cooldown: hour,
|
||||
BrowserCooldown: 6 * hour,
|
||||
Batch: 10,
|
||||
}
|
||||
|
||||
p.runOnce(context.Background())
|
||||
if got := tls.callCount(); got != 1 {
|
||||
t.Fatalf("plain-TLS fetches after 2h = %d, want 1", got)
|
||||
}
|
||||
if got := browser.callCount(); got != 0 {
|
||||
t.Fatalf("browser fetches after 2h = %d, want 0", got)
|
||||
}
|
||||
|
||||
now = time.Unix(7*int64(hour/time.Second), 0)
|
||||
p.runOnce(context.Background())
|
||||
if got := tls.callCount(); got != 2 {
|
||||
t.Fatalf("plain-TLS fetches after 7h = %d, want 2", got)
|
||||
}
|
||||
if got := browser.callCount(); got != 1 {
|
||||
t.Fatalf("browser fetches after 7h = %d, want 1", got)
|
||||
t.Fatalf("fetched %d times after the rest, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -668,14 +613,15 @@ func TestKaganeSkippedWhenNoBrowserFetcher(t *testing.T) {
|
||||
Store: s,
|
||||
Fetch: f,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
if len(f.calls) != 0 {
|
||||
t.Errorf("TLS fetcher was called for kagane: %v", f.calls)
|
||||
}
|
||||
if got := readLatestCheckedAt(t, s, "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"); got != 0 {
|
||||
t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// novelfull without a browser is not skipped outright: its challenge is a
|
||||
@@ -700,10 +646,9 @@ func TestNovelfullUsesTLSWhenNoBrowserFetcher(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: tlsF, CoverBytesFetch: covers,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if len(tlsF.calls) != 1 {
|
||||
t.Fatalf("TLS fetcher calls = %d, want 1", len(tlsF.calls))
|
||||
@@ -742,8 +687,6 @@ func TestKaganeUsesBrowserFetcher(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: tlsF, BrowserFetch: browserF,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
@@ -790,10 +733,9 @@ func TestComixUsesBrowserFetcher(t *testing.T) {
|
||||
Store: s, Fetch: tlsF, BrowserFetch: browserF,
|
||||
CoverFetch: covers, CoverBytesFetch: tlsCovers,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if len(tlsF.calls) != 0 {
|
||||
t.Errorf("TLS fetcher was called for comix: %v", tlsF.calls)
|
||||
@@ -832,14 +774,15 @@ func TestComixSkippedWhenNoBrowserFetcher(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: f,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour,
|
||||
Interval: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
if len(f.calls) != 0 {
|
||||
t.Errorf("TLS fetcher was called for comix: %v", f.calls)
|
||||
}
|
||||
if got := readLatestCheckedAt(t, s, "comix:n8we-dungeons-and-crayons"); got != 0 {
|
||||
t.Errorf("latest_checked_at = %d, want 0 (untried stays due until a browser appears)", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunOncePrefetchesKaganeCover(t *testing.T) {
|
||||
@@ -861,9 +804,9 @@ func TestRunOncePrefetchesKaganeCover(t *testing.T) {
|
||||
Store: s, Fetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
|
||||
BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL))
|
||||
if err != nil || !ok {
|
||||
@@ -898,11 +841,13 @@ func TestRunOnceDoesNotRefetchKaganeCover(t *testing.T) {
|
||||
covers := &fakeCoverFetcher{body: []byte("cover-bytes"), contentType: "image/webp"}
|
||||
p := &Poller{
|
||||
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200}, CoverFetch: covers,
|
||||
Now: func() time.Time { return at }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return at },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
at = at.Add(2 * time.Hour)
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if got := covers.callCount(); got != 1 {
|
||||
t.Fatalf("cover fetch calls = %d, want 1 after two due cycles", got)
|
||||
@@ -927,9 +872,10 @@ func TestRunOnceCoverFailureDoesNotBlockChapter(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
|
||||
CoverFetch: &fakeCoverFetcher{err: errors.New("browser unavailable")},
|
||||
Now: func() time.Time { return now }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return now },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
got, found, err := s.Get(s.OwnerID(), key)
|
||||
if err != nil || !found {
|
||||
@@ -960,9 +906,10 @@ func TestRunOnceRejectsInvalidKaganeCover(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
|
||||
CoverFetch: &fakeCoverFetcher{body: []byte("not an image"), contentType: "text/html"},
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
|
||||
t.Fatalf("invalid cover persisted = %v, err %v; want missing", found, err)
|
||||
@@ -985,9 +932,10 @@ func TestRunOnceWithoutCoverFetcherStillPollsKagane(t *testing.T) {
|
||||
seedCoverSource(t, dbURL, "kagane", seriesID, coverURL)
|
||||
p := &Poller{
|
||||
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
|
||||
t.Fatalf("cover after nil CoverFetch = found %v, err %v; want missing", found, err)
|
||||
@@ -1010,9 +958,10 @@ func TestRunOnceRoutesNonKaganeCoverToPublicFetcher(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture, status: 200}, CoverFetch: browserCovers,
|
||||
CoverBytesFetch: publicCovers,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) }, Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if got := publicCovers.callCount(); got != 1 {
|
||||
t.Fatalf("public cover fetch calls = %d, want 1", got)
|
||||
@@ -1129,9 +1078,9 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
|
||||
Store: s, Fetch: page, BrowserFetch: page,
|
||||
CoverBytesFetch: public, CoverFetch: browser,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
Cooldown: time.Hour, BrowserCooldown: time.Hour, Batch: 10,
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
got := readBookmark(t, s, tc.key)
|
||||
wantWire := testCoverBaseURL + "/covers/" + store.CoverAddress(tc.wantCover)
|
||||
@@ -1181,7 +1130,7 @@ func TestRunOnceDoesNotReplaceExistingCover(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
|
||||
CoverBytesFetch: public,
|
||||
Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return at },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
at = at.Add(2 * time.Hour)
|
||||
@@ -1216,9 +1165,10 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
|
||||
CoverBytesFetch: public,
|
||||
Now: func() time.Time { return at }, Cooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return at },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
if got := readBookmark(t, s, key); got.Cover != "" {
|
||||
t.Fatalf("Cover after failed fetch = %q, want blank", got.Cover)
|
||||
}
|
||||
@@ -1231,6 +1181,7 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
|
||||
public.contentType = "image/jpeg"
|
||||
at = at.Add(2 * time.Hour)
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
if got := public.callCount(); got != 2 {
|
||||
t.Fatalf("cover fetch calls after retry = %d, want 2", got)
|
||||
@@ -1264,9 +1215,10 @@ func TestRunOnceBlankCoverFailureDoesNotBlockChapter(t *testing.T) {
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
|
||||
CoverBytesFetch: &fakeBytesCoverFetcher{err: errors.New("cdn down")},
|
||||
Now: func() time.Time { return now }, Cooldown: time.Hour, Batch: 10,
|
||||
Now: func() time.Time { return now },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
p.waitCovers()
|
||||
|
||||
got := readBookmark(t, s, key)
|
||||
if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 {
|
||||
@@ -1289,3 +1241,342 @@ const kaganeAPIFixtureWithCover = `
|
||||
{"book_id":"b","title":"Episode 41","chapter_no":"41","sort_no":41},
|
||||
{"book_id":"c","title":"Episode 40.5","chapter_no":"40.5","sort_no":40}]}
|
||||
`
|
||||
|
||||
func TestEffectiveGap(t *testing.T) {
|
||||
asura := sites["asura"]
|
||||
tests := []struct {
|
||||
eligible int
|
||||
want time.Duration
|
||||
clamped bool
|
||||
}{
|
||||
{0, 10 * time.Second, false},
|
||||
{5, 10 * time.Second, false},
|
||||
{360, 10 * time.Second, false},
|
||||
{720, 5 * time.Second, false},
|
||||
{3600, time.Second, false},
|
||||
{4000, time.Second, true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
got, clamped := effectiveGap(asura, tt.eligible)
|
||||
if got != tt.want || clamped != tt.clamped {
|
||||
t.Errorf("effectiveGap(asura, %d) = (%s, %v), want (%s, %v)",
|
||||
tt.eligible, got, clamped, tt.want, tt.clamped)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The due query orders by sharedness first, then age: a series two readers
|
||||
// track is polled before a single-reader series that has waited far longer
|
||||
// (ADR-0003, issue #100).
|
||||
func TestRunOnceOrdersBySharednessThenAge(t *testing.T) {
|
||||
s, url := newTestStore(t)
|
||||
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))}, t.TempDir(), testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open second reader: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { other.Close() })
|
||||
|
||||
now := time.UnixMilli(5_000_000)
|
||||
// popular: two readers, due for 2 minutes. loner: one reader, due for 23
|
||||
// minutes. Popularity must win — the loner waited far longer.
|
||||
const (
|
||||
popularKey = "asura:popular"
|
||||
lonerKey = "asura:loner"
|
||||
popularURL = "https://asurascans.com/comics/popular"
|
||||
lonerURL = "https://asurascans.com/comics/loner"
|
||||
)
|
||||
seedForCheck(t, s, popularKey, popularURL, now.Add(-62*time.Minute).UnixMilli())
|
||||
seedForCheck(t, s, lonerKey, lonerURL, 0)
|
||||
if _, err := s.Upsert(other.OwnerID(), store.Bookmark{
|
||||
Key: popularKey, Site: "asura", SeriesID: "popular", UpdatedAt: 2000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed second reader: %v", err)
|
||||
}
|
||||
|
||||
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||
newTestPoller(t, s, f, now).runOnce(context.Background())
|
||||
|
||||
if len(f.calls) != 2 {
|
||||
t.Fatalf("fetches = %d, want 2", len(f.calls))
|
||||
}
|
||||
if f.calls[0] != popularURL {
|
||||
t.Fatalf("first fetch = %q, want the shared series %q", f.calls[0], popularURL)
|
||||
}
|
||||
}
|
||||
|
||||
// Two refusals in one pass stop the Lane: the remaining Series stay unstamped
|
||||
// and due, and the Lane backs off for refuseBackoff before trying the Site
|
||||
// again. One hostile Site burns only its own Lane's budget (issue #100).
|
||||
func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
now := time.UnixMilli(5_000_000)
|
||||
const n = 4
|
||||
for i := 0; i < n; i++ {
|
||||
key := fmt.Sprintf("kagane:s%d", i)
|
||||
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0)
|
||||
}
|
||||
|
||||
browser := &fakeFetcher{status: 403}
|
||||
tls := &fakeFetcher{status: 200}
|
||||
// An asura Series sits on its own Lane: it must still be polled while
|
||||
// kagane's Lane burns its budget on refusals (story 4).
|
||||
seedForCheck(t, s, "asura:still-polls", "https://asurascans.com/series/still-polls", 0)
|
||||
p := &Poller{
|
||||
Store: s, Fetch: tls, BrowserFetch: browser,
|
||||
Now: func() time.Time { return now },
|
||||
}
|
||||
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 2 {
|
||||
t.Fatalf("browser fetches after first pass = %d, want 2 (refused twice)", got)
|
||||
}
|
||||
if got := tls.callCount(); got != 1 {
|
||||
t.Fatalf("asura fetches after first pass = %d, want 1 (its Lane is independent)", got)
|
||||
}
|
||||
stamped := 0
|
||||
for i := 0; i < n; i++ {
|
||||
if readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)) == now.UnixMilli() {
|
||||
stamped++
|
||||
}
|
||||
}
|
||||
if stamped != 2 {
|
||||
t.Fatalf("stamped series = %d, want 2; the remaining two stay due", stamped)
|
||||
}
|
||||
|
||||
// Inside the backoff window nothing is attempted.
|
||||
p.Now = func() time.Time { return now.Add(14 * time.Minute) }
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 2 {
|
||||
t.Fatalf("browser fetches inside backoff = %d, want still 2", got)
|
||||
}
|
||||
|
||||
// Past the backoff the Lane resumes and the two untried Series are polled.
|
||||
p.Now = func() time.Time { return now.Add(16 * time.Minute) }
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 4 {
|
||||
t.Fatalf("browser fetches after backoff = %d, want 4", got)
|
||||
}
|
||||
for i := 0; i < n; i++ {
|
||||
if got := readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)); got == 0 {
|
||||
t.Fatalf("kagane:s%d still untried after backoff", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Below five due Series with none waiting long, a browser Lane leaves Chrome
|
||||
// asleep; five due, or one waiting browserWakeAge, wakes it (ADR-0005).
|
||||
func TestBrowserLaneWakeThresholds(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
now := time.UnixMilli(5_000_000)
|
||||
// Freshly due: checked two minutes before the rest elapses, so the wait
|
||||
// is far below browserWakeAge.
|
||||
seed := func(i int) {
|
||||
key := fmt.Sprintf("kagane:w%d", i)
|
||||
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], now.Add(-62*time.Minute).UnixMilli())
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
seed(i)
|
||||
}
|
||||
|
||||
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
|
||||
Now: func() time.Time { return now },
|
||||
}
|
||||
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 0 {
|
||||
t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got)
|
||||
}
|
||||
// 5 due crosses the count threshold.
|
||||
for i := 3; i < 5; i++ {
|
||||
seed(i)
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 5 {
|
||||
t.Fatalf("browser fetches with 5 due series = %d, want 5", got)
|
||||
}
|
||||
// A single long-neglected series wakes the browser by age alone.
|
||||
seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0)
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 6 {
|
||||
t.Fatalf("browser fetches with one ancient series = %d, want 6", got)
|
||||
}
|
||||
}
|
||||
|
||||
// When one browser Lane loses the sidecar, the round's remaining browser
|
||||
// Lanes are skipped: every fetch would fail anyway, and their Series must not
|
||||
// burn their stamps on a dead Chrome (issue #100).
|
||||
func TestRunOnceUnreachableBrowserStopsBrowserLanes(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
now := time.UnixMilli(5_000_000)
|
||||
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
|
||||
seedForCheck(t, s, "kagane:k", "https://kagane.to/series/k", 0)
|
||||
seedForCheck(t, s, "novelfull:n", "https://novelfull.com/n.html", 0)
|
||||
|
||||
interrupted := fmt.Errorf("%w: %w", errBrowserInterrupted, errors.New("restart"))
|
||||
browser := &fakeFetcher{status: 200, err: interrupted}
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
|
||||
Now: func() time.Time { return now },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
// Sorted lane order: comix, kagane, novelfull. Only comix attempted.
|
||||
if got := browser.callCount(); got != 1 {
|
||||
t.Fatalf("browser fetches = %d, want 1 (only the first browser lane)", got)
|
||||
}
|
||||
if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() {
|
||||
t.Fatalf("comix stamp = %d, want %d", got, now.UnixMilli())
|
||||
}
|
||||
for _, key := range []string{"kagane:k", "novelfull:n"} {
|
||||
if got := readLatestCheckedAt(t, s, key); got != 0 {
|
||||
t.Fatalf("%s stamp = %d, want 0 (untried)", key, got)
|
||||
}
|
||||
}
|
||||
|
||||
// The shared flag decays after refuseBackoff: the next round probes
|
||||
// again. comix's Series is resting (stamped last round), so the probe
|
||||
// falls to kagane — the only Lane with something due — and its fresh
|
||||
// loss re-gates the Lanes behind it.
|
||||
p.Now = func() time.Time { return now.Add(16 * time.Minute) }
|
||||
p.runOnce(context.Background())
|
||||
if got := browser.callCount(); got != 2 {
|
||||
t.Fatalf("browser fetches after backoff decay = %d, want 2 (kagane probes again)", got)
|
||||
}
|
||||
if got := readLatestCheckedAt(t, s, "comix:c"); got != now.UnixMilli() {
|
||||
t.Fatalf("comix stamp after decay = %d, want %d (resting, untouched)", got, now.UnixMilli())
|
||||
}
|
||||
if got := readLatestCheckedAt(t, s, "kagane:k"); got != now.Add(16*time.Minute).UnixMilli() {
|
||||
t.Fatalf("kagane stamp after decay = %d, want %d (the probe)", got, now.Add(16*time.Minute).UnixMilli())
|
||||
}
|
||||
if got := readLatestCheckedAt(t, s, "novelfull:n"); got != 0 {
|
||||
t.Fatalf("novelfull stamp after decay = %d, want 0 (gated by kagane's fresh loss)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A browser Lane that cannot keep up logs the backlog every pass, so the
|
||||
// decision to give browser Sites more pages is measured, not guessed.
|
||||
func TestRunOnceLogsBrowserLaneBehind(t *testing.T) {
|
||||
s, _ := newTestStore(t)
|
||||
now := time.UnixMilli(5_000_000)
|
||||
// Checked three rests ago: two rests' worth of wait past the due moment.
|
||||
seedForCheck(t, s, "kagane:old", "https://kagane.to/series/old", now.Add(-3*time.Hour).UnixMilli())
|
||||
|
||||
var logs strings.Builder
|
||||
prev := log.Writer()
|
||||
log.SetOutput(&logs)
|
||||
t.Cleanup(func() { log.SetOutput(prev) })
|
||||
|
||||
p := &Poller{
|
||||
Store: s, Fetch: &fakeFetcher{status: 200},
|
||||
BrowserFetch: &fakeFetcher{body: kaganeAPIFixture, status: 200},
|
||||
Now: func() time.Time { return now },
|
||||
}
|
||||
p.runOnce(context.Background())
|
||||
|
||||
if got := logs.String(); !strings.Contains(got, "latest poll kagane: browser lane behind by 1h0m0s") {
|
||||
t.Fatalf("behind log = %q, want it to name kagane and the backlog", got)
|
||||
}
|
||||
}
|
||||
|
||||
// gatedCoverFetcher blocks every Fetch on a gate, so a test can hold a cover
|
||||
// heal in flight and prove a Lane does not wait for it.
|
||||
type gatedCoverFetcher struct {
|
||||
inner *fakeBytesCoverFetcher
|
||||
gate chan struct{}
|
||||
started chan struct{}
|
||||
once sync.Once
|
||||
}
|
||||
|
||||
func (g *gatedCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte, string, error) {
|
||||
g.once.Do(func() { g.started <- struct{}{} })
|
||||
<-g.gate
|
||||
return g.inner.Fetch(ctx, sourceURL)
|
||||
}
|
||||
|
||||
// Cover heals run in the background: a heal stuck on a slow CDN must not
|
||||
// delay the Lane's next Series-page Poll, or a large import with many blanks
|
||||
// would make every Latest Chapter go stale (issue #100).
|
||||
func TestRunOnceCoverFetchDoesNotDelayNextPoll(t *testing.T) {
|
||||
s, dbURL := newTestStore(t)
|
||||
// Two Series whose legacy cover sources still need healing.
|
||||
for i := 0; i < 2; i++ {
|
||||
key := fmt.Sprintf("asura:cover-%d", i)
|
||||
seriesID := fmt.Sprintf("cover-%d", i)
|
||||
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
|
||||
Key: key, Site: "asura", SeriesID: seriesID,
|
||||
SeriesURL: "https://asurascans.com/comics/" + seriesID, UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
seedCoverSource(t, dbURL, "asura", seriesID, fmt.Sprintf("https://cdn.example/covers/%d.jpg", i))
|
||||
}
|
||||
|
||||
pages := &fakeFetcher{body: asuraSeriesFixture, status: 200}
|
||||
gated := &gatedCoverFetcher{
|
||||
inner: &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"},
|
||||
gate: make(chan struct{}),
|
||||
started: make(chan struct{}, 1),
|
||||
}
|
||||
p := &Poller{
|
||||
Store: s, Fetch: pages, CoverBytesFetch: gated,
|
||||
Now: func() time.Time { return time.UnixMilli(5_000_000) },
|
||||
}
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
p.runOnce(context.Background())
|
||||
close(done)
|
||||
}()
|
||||
<-gated.started // the first cover heal is now stuck on its CDN
|
||||
select {
|
||||
case <-done:
|
||||
// The Lane finished its page fetches without waiting for the cover.
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("runOnce blocked on an in-flight cover fetch")
|
||||
}
|
||||
close(gated.gate)
|
||||
p.waitCovers()
|
||||
|
||||
if got := pages.callCount(); got != 2 {
|
||||
t.Fatalf("series page fetches = %d, want 2", got)
|
||||
}
|
||||
if got := gated.inner.callCount(); got != 2 {
|
||||
t.Fatalf("cover fetches = %d, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// At 3601 eligible Series the effective gap falls below the one-second floor;
|
||||
// the clamp warning must name the Site so the operator knows which Lane is
|
||||
// outrunning its plan.
|
||||
func TestRunOnceClampWarningNamesTheSite(t *testing.T) {
|
||||
s, dbURL := newTestStore(t)
|
||||
db, err := sql.Open("pgx", dbURL)
|
||||
if err != nil {
|
||||
t.Fatalf("open %s: %v", dbURL, err)
|
||||
}
|
||||
defer db.Close()
|
||||
if _, err := db.Exec(`INSERT INTO series (site, series_id, series_url, latest_checked_at)
|
||||
SELECT 'asura', 'bulk-' || g, 'https://asurascans.com/comics/bulk-' || g, 0
|
||||
FROM generate_series(1, 3601) AS g`); err != nil {
|
||||
t.Fatalf("bulk seed series: %v", err)
|
||||
}
|
||||
if _, err := db.Exec(`INSERT INTO bookmarks (reader_id, site, series_id, updated_at)
|
||||
SELECT (SELECT id FROM readers ORDER BY id LIMIT 1), 'asura', 'bulk-' || g, 1000
|
||||
FROM generate_series(1, 3601) AS g`); err != nil {
|
||||
t.Fatalf("bulk seed bookmarks: %v", err)
|
||||
}
|
||||
|
||||
var logs strings.Builder
|
||||
prev := log.Writer()
|
||||
log.SetOutput(&logs)
|
||||
t.Cleanup(func() { log.SetOutput(prev) })
|
||||
|
||||
newTestPoller(t, s, &fakeFetcher{body: "<html></html>", status: 200}, time.UnixMilli(5_000_000)).
|
||||
runOnce(context.Background())
|
||||
|
||||
if got := logs.String(); !strings.Contains(got, "latest poll asura: gap clamped to 1s floor (eligible series=3601)") {
|
||||
t.Fatalf("clamp warning = %q, want it to name asura and 3601", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
// seriesRead carries the two facts the poll and the acquirer both extract
|
||||
// from a series page. Persistence, stamps and scheduling stay with the
|
||||
// callers, so the policies that keep the two flows distinct (stamp order,
|
||||
// cooldowns) are not swallowed by the module.
|
||||
// rests) are not swallowed by the module.
|
||||
type seriesRead struct {
|
||||
Latest latestChapter
|
||||
HasLatest bool
|
||||
@@ -22,6 +22,9 @@ type seriesRead struct {
|
||||
|
||||
// errNotFetchable and errNoFetcher separate the gate and the route from fetch
|
||||
// failures so each caller keeps its own distinct log line for all three.
|
||||
// errChallengeHeld (browser.go) is the outcome of a Site that answered with
|
||||
// its interstitial — status 403 (cf-mitigated) or a challenge page body — and
|
||||
// is how a Lane tells a refusal from an ordinary failure (issue #100).
|
||||
var (
|
||||
errNotFetchable = errors.New("series url not fetchable")
|
||||
errNoFetcher = errors.New("no fetcher for site")
|
||||
@@ -49,9 +52,21 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
|
||||
if err != nil {
|
||||
return seriesRead{}, fmt.Errorf("fetch %s: %w", seriesURL, err)
|
||||
}
|
||||
if status == 403 {
|
||||
// Cloudflare's challenge response for these Sites (cf-mitigated). The
|
||||
// browser fetcher returns exactly this on a held interstitial, and a
|
||||
// plain-TLS 403 means the same: the Site is refusing.
|
||||
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status)
|
||||
}
|
||||
if status != 200 {
|
||||
return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status)
|
||||
}
|
||||
if isInterstitial(body) {
|
||||
// A 200 that is the challenge page, not the payload: the TLS route can
|
||||
// receive this where the browser would have kept re-reading. Same
|
||||
// refusal as the 403.
|
||||
return seriesRead{}, fmt.Errorf("%w: fetch %s: interstitial body", errChallengeHeld, seriesURL)
|
||||
}
|
||||
latest, hasLatest := latestChapterFrom(site, seriesURL, body)
|
||||
cover, hasCover := coverFrom(site, seriesURL, body)
|
||||
return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/chromedp/chromedp"
|
||||
)
|
||||
@@ -32,6 +33,11 @@ type site struct {
|
||||
LatestChapter func(seriesURL, body string) (latestChapter, bool)
|
||||
// Cover finds the Cover address in a fetched body.
|
||||
Cover func(seriesURL, body string) (string, bool)
|
||||
// Rest is how long a Series of this Site rests between Polls.
|
||||
Rest time.Duration
|
||||
// Gap is the Lane's strictest pace: at least one second must pass between
|
||||
// two consecutive Series-page Polls of this Site (issue #100).
|
||||
Gap time.Duration
|
||||
// Browser reads this Site's payload from a cleared browser tab; nil
|
||||
// means the page is fetched over plain TLS.
|
||||
Browser *browserRead
|
||||
@@ -380,6 +386,50 @@ func publishedCoverURL(value string) string {
|
||||
return strings.ReplaceAll(value, " ", "%20")
|
||||
}
|
||||
|
||||
// Poll Lane constants (issue #100). The per-Site structure is deliberately
|
||||
// uniform at first — every Site rests an hour and gaps ten seconds — but it
|
||||
// exists so a single Site can be slowed if it turns hostile, and the numbers
|
||||
// stay in the registry so the structure has a place to differ.
|
||||
const (
|
||||
// defaultRest is how long every Series rests between Polls.
|
||||
defaultRest = time.Hour
|
||||
// defaultGap is the strictest pace of every Lane unless the eligible
|
||||
// Series count forces it tighter.
|
||||
defaultGap = 10 * time.Second
|
||||
// minGap floors the effective gap. One request per second is already an
|
||||
// order of magnitude past the strictest rate rule a free-plan Site can
|
||||
// express (docs/research/cloudflare-bot-scoring-and-poll-cadence.md);
|
||||
// below it the Lane is outrunning its own plan and says so loudly.
|
||||
minGap = time.Second
|
||||
// refuseBackoff is how long a Lane waits after its Site refused twice in
|
||||
// one run before attempting it again.
|
||||
refuseBackoff = 15 * time.Minute
|
||||
// browserWakeCount and browserWakeAge gate a browser Lane's run: five or
|
||||
// more due Series, or any one of them waiting this long, or Chrome stays
|
||||
// asleep (ADR-0005 on-demand browser).
|
||||
browserWakeCount = 5
|
||||
browserWakeAge = 15 * time.Minute
|
||||
)
|
||||
|
||||
// effectiveGap is a Site's pace: the registry gap, or one rest divided by the
|
||||
// eligible Series count when that is smaller, never below one second. The
|
||||
// denominator follows defaultRest rather than a literal hour so a Site whose
|
||||
// rest is ever changed keeps its per-Series pace in step. The second return is
|
||||
// true when the one-second floor engaged (and the Lane logs a warning naming
|
||||
// the Site, every round it does).
|
||||
func effectiveGap(s site, eligible int) (time.Duration, bool) {
|
||||
gap := s.Gap
|
||||
if eligible > 0 {
|
||||
if perSeries := defaultRest / time.Duration(eligible); perSeries < gap {
|
||||
gap = perSeries
|
||||
}
|
||||
}
|
||||
if gap < minGap {
|
||||
return minGap, true
|
||||
}
|
||||
return gap, false
|
||||
}
|
||||
|
||||
// sites is the registry: one entry per Site, keyed by the stored site string.
|
||||
// Adding a Site means adding an entry here and nowhere else — the dispatch
|
||||
// functions above and the poller's route list are lookups into this map. An
|
||||
@@ -390,16 +440,22 @@ var sites = map[string]site{
|
||||
Host: "asurascans.com",
|
||||
LatestChapter: asuraLatestChapter,
|
||||
Cover: ogImageCover,
|
||||
Rest: defaultRest,
|
||||
Gap: defaultGap,
|
||||
},
|
||||
"demonic": {
|
||||
Host: "demonicscans.org",
|
||||
LatestChapter: demonicLatestChapter,
|
||||
Cover: ogImageCover,
|
||||
Rest: defaultRest,
|
||||
Gap: defaultGap,
|
||||
},
|
||||
"comix": {
|
||||
Host: "comix.to",
|
||||
LatestChapter: comixLatestChapter,
|
||||
Cover: comixCoverEntry,
|
||||
Rest: defaultRest,
|
||||
Gap: defaultGap,
|
||||
Browser: &browserRead{
|
||||
Read: comixRead,
|
||||
// The interstitial is served in place of the page, so "arrived"
|
||||
@@ -414,6 +470,8 @@ var sites = map[string]site{
|
||||
Host: "kagane.to",
|
||||
LatestChapter: kaganeLatestChapter,
|
||||
Cover: kaganeCoverEntry,
|
||||
Rest: defaultRest,
|
||||
Gap: defaultGap,
|
||||
Browser: &browserRead{
|
||||
Read: kaganeRead,
|
||||
Done: func(body string) bool { return body != "" },
|
||||
@@ -426,6 +484,8 @@ var sites = map[string]site{
|
||||
Host: "novelfull.com",
|
||||
LatestChapter: novelfullLatestChapter,
|
||||
Cover: novelfullCoverEntry,
|
||||
Rest: defaultRest,
|
||||
Gap: defaultGap,
|
||||
Browser: &browserRead{
|
||||
Read: novelfullRead,
|
||||
// The interstitial has a DOM too, so "the payload arrived" has to
|
||||
@@ -438,13 +498,15 @@ var sites = map[string]site{
|
||||
Host: "lightnovelworld.net",
|
||||
LatestChapter: lnwLatestChapter,
|
||||
Cover: ogImageCover,
|
||||
Rest: defaultRest,
|
||||
Gap: defaultGap,
|
||||
},
|
||||
}
|
||||
|
||||
// browserBackedSites is derived from the registry: the Sites whose pages are
|
||||
// read through the browser sidecar, which are also the ones granted the longer
|
||||
// cooldown. Sorted so callers that range it (the due query, the browser
|
||||
// fetcher's dispatch) see a stable order instead of map-iteration noise.
|
||||
// read through the browser sidecar. Sorted so callers that range it (the
|
||||
// browser fetcher's dispatch) see a stable order instead of map-iteration
|
||||
// noise.
|
||||
func browserBackedSites() []string {
|
||||
out := make([]string, 0, len(sites))
|
||||
for name, s := range sites {
|
||||
|
||||
@@ -16,7 +16,6 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgtype"
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
@@ -897,10 +896,13 @@ func (s *Store) Delete(readerID int64, key string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// DueForLatestCheck returns series whose server-side latest-chapter check has
|
||||
// aged past the appropriate cutoff, ordered by how many bookmarks reference
|
||||
// them (descending) then least-recently-checked first, at most limit of them.
|
||||
// Browser-backed sites use browserCutoffMs; every other site uses cutoffMs.
|
||||
// DueForLatestCheck returns one Site's series whose server-side
|
||||
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks
|
||||
// reference them (descending) then least-recently-checked first. One Site per
|
||||
// query, because each Poll Lane asks for its own list: the query carries one
|
||||
// Site and one cut-off instead of parallel lists (issue #100). There is no
|
||||
// limit — the Lane's own gap paces the fetches, and the batch size that used
|
||||
// to cap this query is gone with the shared pace.
|
||||
//
|
||||
// The reader_count ordering is the point of the split (ADR-0003): a series
|
||||
// shared by several readers is fetched once per due cycle, and the popular
|
||||
@@ -916,20 +918,17 @@ func (s *Store) Delete(readerID int64, key string) error {
|
||||
// burns requests. Archived bookmarks still count — knowing what a shelved
|
||||
// series is up to is the whole reason for archiving instead of deleting.
|
||||
// A series with no bookmarks at all never appears: the join excludes it.
|
||||
func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites []string, limit int) ([]Series, error) {
|
||||
func (s *Store) DueForLatestCheck(site string, cutoffMs int64) ([]Series, error) {
|
||||
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
|
||||
FROM series s
|
||||
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||
WHERE s.series_url <> ''
|
||||
AND s.latest_checked_at <= CASE
|
||||
WHEN s.site = ANY($3::text[]) THEN $2::bigint
|
||||
ELSE $1::bigint
|
||||
END
|
||||
WHERE s.site = $1
|
||||
AND s.series_url <> ''
|
||||
AND s.latest_checked_at <= $2::bigint
|
||||
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
|
||||
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
|
||||
ORDER BY reader_count DESC, s.latest_checked_at ASC
|
||||
LIMIT $4`, cutoffMs, browserCutoffMs, pgtype.FlatArray[string](browserSites), limit)
|
||||
ORDER BY reader_count DESC, s.latest_checked_at ASC`, site, cutoffMs)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query due series: %w", err)
|
||||
}
|
||||
@@ -946,6 +945,30 @@ func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// EligibleSeriesCount returns how many of a Site's Series still have at least
|
||||
// one bookmark outside the finished bucket. It is the denominator of the
|
||||
// Lane's pace (issue #100): the effective gap is the smaller of the registry
|
||||
// gap and one hour divided by this count, so Series that will never be Polled
|
||||
// do not make the Lane faster than it needs to be, and counting every eligible
|
||||
// Series rather than only those currently due keeps the pace steady — the
|
||||
// single worst moment to be fastest is startup, when everything is due at
|
||||
// once.
|
||||
func (s *Store) EligibleSeriesCount(site string) (int, error) {
|
||||
var n int
|
||||
err := s.db.QueryRow(`SELECT COUNT(*) FROM (
|
||||
SELECT 1
|
||||
FROM series s
|
||||
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
||||
WHERE s.site = $1
|
||||
GROUP BY s.site, s.series_id
|
||||
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
|
||||
) e`, site).Scan(&n)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count eligible series %s: %w", site, err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// MarkLatestChecked records that the server looked at a series at ts, whatever
|
||||
// the look turned up. Marking a missing series is not an error: the row may
|
||||
// have been orphaned while a fetch was in flight.
|
||||
@@ -954,7 +977,7 @@ func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites
|
||||
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
|
||||
// a whole Bookmark from the client and Upsert writes every series column it
|
||||
// knows about, so a userscript PUT — which has no idea this field exists —
|
||||
// would write a zero and reset the cooldown, making the poller re-fetch that
|
||||
// would write a zero and reset the rest, making the poller re-fetch that
|
||||
// series every tick for as long as the user kept reading it.
|
||||
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
|
||||
if _, err := s.db.Exec(
|
||||
@@ -966,7 +989,7 @@ func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
|
||||
}
|
||||
|
||||
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
||||
// tests outside this package (the poller's own tests assert on cooldown
|
||||
// tests outside this package (the poller's own tests assert on rest
|
||||
// bookkeeping) — see MarkLatestChecked for why the field stays off the
|
||||
// client-visible row.
|
||||
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
|
||||
|
||||
@@ -298,7 +298,7 @@ func TestDueForLatestCheck(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
seedForCheck(t, s, "asura:x", tt.seriesURL, tt.checkedAt)
|
||||
|
||||
due, err := s.DueForLatestCheck(now-hour, now-hour, nil, 10)
|
||||
due, err := s.DueForLatestCheck("asura", now-hour)
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck: %v", err)
|
||||
}
|
||||
@@ -309,22 +309,25 @@ func TestDueForLatestCheck(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDueForLatestCheckOldestFirstAndLimited(t *testing.T) {
|
||||
func TestDueForLatestCheckOldestFirstAndScopedToSite(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
// Insert newest-checked first so a correct ORDER BY has to reverse it.
|
||||
seedForCheck(t, s, "asura:c", "https://asurascans.com/comics/c", 300)
|
||||
seedForCheck(t, s, "asura:b", "https://asurascans.com/comics/b", 200)
|
||||
seedForCheck(t, s, "asura:a", "https://asurascans.com/comics/a", 100)
|
||||
// A second Site's due series must not appear in asura's list: each Lane
|
||||
// asks for one Site, and no Lane may see another's queue.
|
||||
seedForCheck(t, s, "demonic:z", "https://demonicscans.org/manga/z", 0)
|
||||
|
||||
due, err := s.DueForLatestCheck(1000, 1000, nil, 2)
|
||||
due, err := s.DueForLatestCheck("asura", 1000)
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck: %v", err)
|
||||
}
|
||||
if len(due) != 2 {
|
||||
t.Fatalf("got %d rows, want 2 (limit)", len(due))
|
||||
if len(due) != 3 {
|
||||
t.Fatalf("got %d rows, want 3 (all of asura's, none of demonic's)", len(due))
|
||||
}
|
||||
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" {
|
||||
t.Fatalf("got %q,%q; want asura:a,asura:b (oldest first)", due[0].Key(), due[1].Key())
|
||||
if due[0].Key() != "asura:a" || due[1].Key() != "asura:b" || due[2].Key() != "asura:c" {
|
||||
t.Fatalf("got %q,%q,%q; want asura:a,asura:b,asura:c (oldest first)", due[0].Key(), due[1].Key(), due[2].Key())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -496,7 +499,7 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
due, err := store.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
|
||||
due, err := store.DueForLatestCheck("asura", time.Now().UnixMilli())
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck: %v", err)
|
||||
}
|
||||
@@ -512,6 +515,38 @@ func TestDueForLatestCheckSkipsFinishedKeepsArchived(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The gap's denominator counts every Series the Lane will ever Poll: a
|
||||
// finished Series must not make the Lane faster than it needs to be, and
|
||||
// another Site's Series must not leak into this Site's count.
|
||||
func TestEligibleSeriesCount(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
seedForCheck(t, store, "asura:reading", "https://asurascans.com/comics/reading", 0)
|
||||
seedForCheck(t, store, "asura:archived", "https://asurascans.com/comics/archived", 0)
|
||||
if _, err := store.Upsert(store.OwnerID(), Bookmark{
|
||||
Key: "asura:finished", Site: "asura", SeriesID: "finished",
|
||||
SeriesURL: "https://asurascans.com/comics/finished",
|
||||
Status: StatusFinished, UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed finished: %v", err)
|
||||
}
|
||||
seedForCheck(t, store, "demonic:z", "https://demonicscans.org/manga/z", 0)
|
||||
|
||||
n, err := store.EligibleSeriesCount("asura")
|
||||
if err != nil {
|
||||
t.Fatalf("EligibleSeriesCount: %v", err)
|
||||
}
|
||||
if n != 2 {
|
||||
t.Fatalf("eligible = %d, want 2 (finished excluded, demonic excluded)", n)
|
||||
}
|
||||
n, err = store.EligibleSeriesCount("demonic")
|
||||
if err != nil {
|
||||
t.Fatalf("EligibleSeriesCount(demonic): %v", err)
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("eligible(demonic) = %d, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDisplayChapter(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -970,7 +1005,7 @@ func TestDueForLatestCheckOrdersByReaderCountThenAge(t *testing.T) {
|
||||
seedSecondReader(t, s, "asura:pop:2", "asura", "pop", 1001)
|
||||
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 100)
|
||||
|
||||
due, err := s.DueForLatestCheck(1000, 1000, nil, 10)
|
||||
due, err := s.DueForLatestCheck("asura", 1000)
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck: %v", err)
|
||||
}
|
||||
@@ -996,7 +1031,7 @@ func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
|
||||
t.Fatalf("seed orphan series: %v", err)
|
||||
}
|
||||
|
||||
due, err := s.DueForLatestCheck(1000, 1000, nil, 10)
|
||||
due, err := s.DueForLatestCheck("asura", 1000)
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck: %v", err)
|
||||
}
|
||||
@@ -1334,7 +1369,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
|
||||
t.Fatalf("series rows = %d, want 1 shared row for two bookmarks", series)
|
||||
}
|
||||
|
||||
due, err := s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
|
||||
due, err := s.DueForLatestCheck("asura", time.Now().UnixMilli())
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck: %v", err)
|
||||
}
|
||||
@@ -1350,7 +1385,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
|
||||
if b, ok, err := s.Get(s.OwnerID(), "asura:solo"); err != nil || !ok || b.LastChapterNum != 200 {
|
||||
t.Fatalf("owner's bookmark after the other's delete = %+v ok=%v err=%v, want it intact", b, ok, err)
|
||||
}
|
||||
due, err = s.DueForLatestCheck(time.Now().UnixMilli(), time.Now().UnixMilli(), nil, 10)
|
||||
due, err = s.DueForLatestCheck("asura", time.Now().UnixMilli())
|
||||
if err != nil {
|
||||
t.Fatalf("DueForLatestCheck after delete: %v", err)
|
||||
}
|
||||
|
||||
+11
-86
@@ -7,7 +7,6 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
@@ -61,30 +60,15 @@ type Config struct {
|
||||
|
||||
// LatestPoll configures the background latest-chapter poller.
|
||||
//
|
||||
// Sizing: batch x (cooldown / interval) is how many series hold a true cooldown
|
||||
// cadence — 14 x (1h / 10m) = 84 with these defaults, which covers this
|
||||
// deployment. Past that nothing breaks; the effective cadence stretches to
|
||||
// N x interval / batch and the oldest-checked-first ordering keeps it uniform.
|
||||
// Only the kill switch lives here. Pace is per Site — rest time and gap are
|
||||
// registry properties (internal/latest/sites.go, issue #100), because each
|
||||
// Lane has to be able to differ from the others. The five environment
|
||||
// settings that used to size a shared pace (cooldown, browser cooldown,
|
||||
// interval, stagger, batch) are gone with it: no deployed .env may carry them.
|
||||
type LatestPoll struct {
|
||||
Enabled bool
|
||||
Cooldown time.Duration
|
||||
BrowserCooldown time.Duration
|
||||
Interval time.Duration
|
||||
Stagger time.Duration
|
||||
Batch int
|
||||
}
|
||||
|
||||
const (
|
||||
defaultPollCooldown = time.Hour
|
||||
defaultBrowserPollCooldown = 6 * time.Hour
|
||||
defaultPollInterval = 10 * time.Minute
|
||||
defaultPollStagger = 20 * time.Second
|
||||
defaultPollBatch = 14
|
||||
// minPollCooldown keeps a typo from turning a polite background check into
|
||||
// a hammer against sites that are already bot-scoring us.
|
||||
minPollCooldown = 15 * time.Minute
|
||||
)
|
||||
|
||||
func envOr(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
@@ -107,66 +91,11 @@ func envBool(key string, def bool) bool {
|
||||
}
|
||||
}
|
||||
|
||||
// envDuration reads a duration env var. An unparseable or non-positive value
|
||||
// falls back to def and logs rather than failing startup: the poller is an
|
||||
// enhancement, and a typo in one of its knobs must not stop bookmark sync.
|
||||
func envDuration(key string, def time.Duration) time.Duration {
|
||||
raw := strings.TrimSpace(os.Getenv(key))
|
||||
if raw == "" {
|
||||
return def
|
||||
}
|
||||
d, err := time.ParseDuration(raw)
|
||||
if err != nil || d <= 0 {
|
||||
log.Printf("config: %s=%q is not a positive duration, using %s", key, raw, def)
|
||||
return def
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// envInt reads a positive integer env var, with the same fallback policy.
|
||||
func envInt(key string, def int) int {
|
||||
raw := strings.TrimSpace(os.Getenv(key))
|
||||
if raw == "" {
|
||||
return def
|
||||
}
|
||||
n, err := strconv.Atoi(raw)
|
||||
if err != nil || n <= 0 {
|
||||
log.Printf("config: %s=%q is not a positive integer, using %d", key, raw, def)
|
||||
return def
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func clampPollCooldown(name string, d time.Duration) time.Duration {
|
||||
if d < minPollCooldown {
|
||||
log.Printf("config: %s %s is below the %s floor, clamping", name, d, minPollCooldown)
|
||||
return minPollCooldown
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// loadLatestPoll reads the poller's settings, clamping anything that would make
|
||||
// it antisocial.
|
||||
// loadLatestPoll reads the poller's settings. The pace knobs that used to be
|
||||
// clamped here are registry properties now (issue #100), so there is nothing
|
||||
// left to clamp.
|
||||
func loadLatestPoll() LatestPoll {
|
||||
p := LatestPoll{
|
||||
Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true),
|
||||
Cooldown: envDuration("LATEST_CHAPTER_POLL_COOLDOWN", defaultPollCooldown),
|
||||
BrowserCooldown: envDuration("LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", defaultBrowserPollCooldown),
|
||||
Interval: envDuration("LATEST_CHAPTER_POLL_INTERVAL", defaultPollInterval),
|
||||
Stagger: envDuration("LATEST_CHAPTER_POLL_STAGGER", defaultPollStagger),
|
||||
Batch: envInt("LATEST_CHAPTER_POLL_BATCH", defaultPollBatch),
|
||||
}
|
||||
p.Cooldown = clampPollCooldown("cooldown", p.Cooldown)
|
||||
p.BrowserCooldown = clampPollCooldown("browser cooldown", p.BrowserCooldown)
|
||||
// batch x stagger has to fit inside one tick or a batch is still running
|
||||
// when the next one is due. Run() serialises them, so this degrades to a
|
||||
// slower cadence rather than to overlapping fetches — worth a warning, not
|
||||
// a failure.
|
||||
if span := time.Duration(p.Batch) * p.Stagger; span > p.Interval {
|
||||
log.Printf("config: batch(%d) x stagger(%s) = %s exceeds interval %s; batches will overrun their tick",
|
||||
p.Batch, p.Stagger, span, p.Interval)
|
||||
}
|
||||
return p
|
||||
return LatestPoll{Enabled: envBool("LATEST_CHAPTER_POLL_ENABLED", true)}
|
||||
}
|
||||
|
||||
func loadConfig() Config {
|
||||
@@ -377,7 +306,8 @@ func main() {
|
||||
}
|
||||
}
|
||||
|
||||
// newLatestPoller wires the configured cooldowns and fetchers into the poller.
|
||||
// newLatestPoller wires the fetcher seams into the poller. Pace is registry
|
||||
// property, not config (issue #100), so there are no knobs to pass through.
|
||||
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller {
|
||||
var covers latest.BrowserCoverFetcher
|
||||
if f, ok := browser.(latest.BrowserCoverFetcher); ok {
|
||||
@@ -390,11 +320,6 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
|
||||
CoverFetch: covers,
|
||||
CoverBytesFetch: latest.NewCoverFetcher(),
|
||||
Now: time.Now,
|
||||
Cooldown: cfg.Cooldown,
|
||||
BrowserCooldown: cfg.BrowserCooldown,
|
||||
Interval: cfg.Interval,
|
||||
Stagger: cfg.Stagger,
|
||||
Batch: cfg.Batch,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+19
-115
@@ -9,31 +9,15 @@ import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"bookmarkmanager/backend/internal/latest"
|
||||
"bookmarkmanager/backend/internal/store"
|
||||
)
|
||||
|
||||
func TestLoadLatestPollDefaults(t *testing.T) {
|
||||
for _, k := range []string{
|
||||
"LATEST_CHAPTER_POLL_ENABLED", "LATEST_CHAPTER_POLL_COOLDOWN",
|
||||
"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN", "LATEST_CHAPTER_POLL_INTERVAL",
|
||||
"LATEST_CHAPTER_POLL_STAGGER", "LATEST_CHAPTER_POLL_BATCH",
|
||||
} {
|
||||
t.Setenv(k, "")
|
||||
}
|
||||
|
||||
got := loadLatestPoll()
|
||||
want := LatestPoll{
|
||||
Enabled: true,
|
||||
Cooldown: time.Hour,
|
||||
BrowserCooldown: 6 * time.Hour,
|
||||
Interval: 10 * time.Minute,
|
||||
Stagger: 20 * time.Second,
|
||||
Batch: 14,
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, want)
|
||||
t.Setenv("LATEST_CHAPTER_POLL_ENABLED", "")
|
||||
if got := loadLatestPoll(); got != (LatestPoll{Enabled: true}) {
|
||||
t.Fatalf("loadLatestPoll() = %+v, want %+v", got, LatestPoll{Enabled: true})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,105 +47,25 @@ func TestLoadLatestPollEnabledParsing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadLatestPollClampsAndFallsBack(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
env map[string]string
|
||||
wantFrom func(LatestPoll) any
|
||||
want any
|
||||
}{
|
||||
{
|
||||
name: "cooldown below the floor is clamped up",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "1m"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Cooldown },
|
||||
want: 15 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "cooldown at the floor is kept",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_COOLDOWN": "15m"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Cooldown },
|
||||
want: 15 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "browser cooldown below the floor is clamped up",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "1m"},
|
||||
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
|
||||
want: 15 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "browser cooldown at the floor is kept",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "15m"},
|
||||
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
|
||||
want: 15 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "browser cooldown override is honoured",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "8h"},
|
||||
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
|
||||
want: 8 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "browser cooldown unparseable value falls back",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BROWSER_COOLDOWN": "six hours"},
|
||||
wantFrom: func(p LatestPoll) any { return p.BrowserCooldown },
|
||||
want: 6 * time.Hour,
|
||||
},
|
||||
{
|
||||
name: "a valid override is honoured",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "5m"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Interval },
|
||||
want: 5 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "an unparseable duration falls back",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_INTERVAL": "ten minutes"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Interval },
|
||||
want: 10 * time.Minute,
|
||||
},
|
||||
{
|
||||
name: "a zero duration falls back",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_STAGGER": "0s"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Stagger },
|
||||
want: 20 * time.Second,
|
||||
},
|
||||
{
|
||||
name: "a valid batch is honoured",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "30"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Batch },
|
||||
want: 30,
|
||||
},
|
||||
{
|
||||
name: "a negative batch falls back",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "-5"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Batch },
|
||||
want: 14,
|
||||
},
|
||||
{
|
||||
name: "a non-numeric batch falls back",
|
||||
env: map[string]string{"LATEST_CHAPTER_POLL_BATCH": "lots"},
|
||||
wantFrom: func(p LatestPoll) any { return p.Batch },
|
||||
want: 14,
|
||||
},
|
||||
// newLatestPoller wires the fetcher seams; pace lives in the registry, so
|
||||
// nothing here sizes a cooldown any more.
|
||||
func TestNewLatestPollerWiresFetchers(t *testing.T) {
|
||||
tls := &latest.TLSFetcher{}
|
||||
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil)
|
||||
if p.Fetch != tls {
|
||||
t.Fatalf("Fetch not wired")
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
for k, v := range tt.env {
|
||||
t.Setenv(k, v)
|
||||
if p.BrowserFetch != nil {
|
||||
t.Fatalf("BrowserFetch = %v, want nil for a browser-less deployment", p.BrowserFetch)
|
||||
}
|
||||
if got := tt.wantFrom(loadLatestPoll()); got != tt.want {
|
||||
t.Fatalf("got %v, want %v", got, tt.want)
|
||||
if p.CoverFetch != nil {
|
||||
t.Fatalf("CoverFetch = %v, want nil when the browser is absent", p.CoverFetch)
|
||||
}
|
||||
})
|
||||
if p.CoverBytesFetch == nil {
|
||||
t.Fatalf("CoverBytesFetch = nil, want the TLS cover fetcher")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewLatestPollerWiresCooldowns(t *testing.T) {
|
||||
p := newLatestPoller(nil, LatestPoll{
|
||||
Cooldown: time.Hour,
|
||||
BrowserCooldown: 6 * time.Hour,
|
||||
}, nil, nil)
|
||||
if p.Cooldown != time.Hour || p.BrowserCooldown != 6*time.Hour {
|
||||
t.Fatalf("poller cooldowns = %s/%s, want 1h/6h", p.Cooldown, p.BrowserCooldown)
|
||||
if p.Now == nil {
|
||||
t.Fatalf("Now = nil, want the live clock")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+4
-7
@@ -58,14 +58,11 @@ services:
|
||||
# Second script from the same bindmount; the novel library is a separate
|
||||
# Violentmonkey install.
|
||||
NOVEL_USERSCRIPT_PATH: ${NOVEL_USERSCRIPT_PATH:-/userscript/novel-bookmark.user.js}
|
||||
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the kill
|
||||
# switch; it only takes effect because these are listed here.
|
||||
# Latest-chapter poller. LATEST_CHAPTER_POLL_ENABLED=0 in .env is the
|
||||
# kill switch; it only takes effect because it is listed here. Pace is
|
||||
# per Site in the registry (one Poll Lane per Site, issue #100) — the
|
||||
# cooldown/interval/stagger/batch knobs are gone with the shared pace.
|
||||
LATEST_CHAPTER_POLL_ENABLED: ${LATEST_CHAPTER_POLL_ENABLED:-1}
|
||||
LATEST_CHAPTER_POLL_COOLDOWN: ${LATEST_CHAPTER_POLL_COOLDOWN:-1h}
|
||||
LATEST_CHAPTER_POLL_BROWSER_COOLDOWN: ${LATEST_CHAPTER_POLL_BROWSER_COOLDOWN:-6h}
|
||||
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
|
||||
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
|
||||
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
|
||||
# CDP endpoint for sites behind a JavaScript challenge (kagane,
|
||||
# novelfull). The browser is not part of this stack — it runs on the home
|
||||
# machine as its own unit (chrome/docker-compose.yml) and is reached over
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# ADR-0010: Poll Lanes — one independent Poll stream per Site
|
||||
|
||||
Date: 2026-08-16
|
||||
Status: accepted
|
||||
|
||||
## Decision
|
||||
|
||||
Replace the single shared polling pace with one **Poll Lane** per Site: an
|
||||
independent goroutine that polls only that Site's Series, paced by that Site's
|
||||
registry entry. Pace moves out of config and into the Site registry
|
||||
(`internal/latest/sites.go`): every entry carries a `Rest` (how long a Series
|
||||
rests between Polls) and a `Gap` (how long the Lane waits between fetches).
|
||||
|
||||
Rest is enforced by the due query's WHERE clause (`latest_checked_at <= now -
|
||||
Rest`), never by a timer — the same mechanism that enforced the old cooldown.
|
||||
The Lane enforces its own gap by sleeping between fetches. `effectiveGap` is
|
||||
the registry gap, or one hour divided by the Site's eligible Series count when
|
||||
that is smaller, never below one second.
|
||||
|
||||
The five environment settings that used to size the shared pace —
|
||||
`LATEST_CHAPTER_POLL_COOLDOWN`, `_BROWSER_COOLDOWN`, `_INTERVAL`, `_BATCH`,
|
||||
`_STAGGER` — are deleted. Only the kill switch `LATEST_CHAPTER_POLL_ENABLED`
|
||||
remains. No deployed `.env` may carry the deleted knobs.
|
||||
|
||||
## Why
|
||||
|
||||
The shared pace capped the whole backend at roughly 180 Polls an hour (one
|
||||
20-second stagger across one queue). ~60 Series today, scaling to hundreds or
|
||||
thousands, would stretch the hour beyond what the New Chapter signal can
|
||||
tolerate. Worse, the queue mixed Sites with very different costs: kagane and
|
||||
comix pay seconds of a serialized single-tab Chrome per Poll (a challenged
|
||||
page, ADR-0005), and one hostile Site burning its challenge timeout made every
|
||||
other Site's Series wait — "one hostile Site can eat most of an hour".
|
||||
|
||||
Lanes fix both at once:
|
||||
|
||||
- **Throughput scales per Site.** The six Lanes fetch concurrently; a Lane's
|
||||
own gap paces it. The browser Lanes' combined ceiling stays about 360 Polls
|
||||
an hour (one tab), and when they cannot keep up the wait past Rest grows and
|
||||
is logged every pass — the "behind by X" measurement, so the decision to
|
||||
give browser Sites more pages is made from data.
|
||||
- **Hostility is contained.** A refusal (two challenge-held reads in one
|
||||
pass) stops only that Site's Lane for `refuseBackoff` (15m); the rest of
|
||||
that Lane's Series stay unstamped and due. A lost browser gates the other
|
||||
browser Lanes' passes for the same window — the flag is shared Poller
|
||||
state, so the loss is noticed once instead of once per Lane per pass, and
|
||||
decays after 15m so the Lanes probe again. One Site can no longer tax the
|
||||
others.
|
||||
|
||||
## Tradeoffs and rejections
|
||||
|
||||
- **Per-Site env knobs** (e.g. `KAGANE_POLL_GAP`) rejected: the registry is
|
||||
the single place pace lives, testable and reviewable; config knobs would
|
||||
recreate the shared-pace sprawl with six times the surface. All six entries
|
||||
are deliberately uniform at first — rest an hour, gap ten seconds — so the
|
||||
structure exists to differ without inventing numbers for Sites that have
|
||||
not earned them.
|
||||
- **Dynamic gap** (`rest / eligible`) is the one knob that stays automatic:
|
||||
a Site with more Series than one per ten seconds would otherwise back up
|
||||
behind its own gap, and the per-Series share of the hour is the natural
|
||||
pace. The ten-second default is not arbitrary: one request per ten seconds
|
||||
is the strictest rate rule a free-plan Site can even express (per-zone
|
||||
rate limiting, as documented in
|
||||
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`), so the
|
||||
default pace is exactly what the most restrictive Site would demand of us.
|
||||
The computed gap never goes below one second and logs loudly when the
|
||||
floor engages.
|
||||
- **Timer-based pacing** rejected: the old ticker made the poller's rate a
|
||||
function of wall clock rather than of what was due. The due-query cutoff is
|
||||
the only rate authority; the Lane sleep just prevents hammering.
|
||||
- **Batch size** (the old `_BATCH` cap) is gone with the shared pace: a Lane
|
||||
processes everything due, paced by its gap. There is no global queue left
|
||||
to bound.
|
||||
|
||||
## Constraints preserved
|
||||
|
||||
- Stamp-before-fetch ("attempted" semantics): an untried Series stays due, so
|
||||
a browser that appears after a restart finds its full queue waiting.
|
||||
- Browser wake gate (ADR-0005): a browser Lane leaves Chrome asleep below
|
||||
five due Series and 15 minutes of wait, per Lane.
|
||||
- The browser is not in the API stack (ADR-0006): an unreachable browser
|
||||
degrades a Lane exactly as an unset `BROWSER_WS_URL` — browser-only Sites
|
||||
skipped, plain-TLS unaffected, stored covers still served.
|
||||
- Cover heals moved to background goroutines (joined by the test suite via
|
||||
`waitCovers`) so a slow cover CDN cannot consume a Lane's gap.
|
||||
|
||||
Supersedes the pace mechanics of ADR-0003's "raise throughput instead" note
|
||||
(the stagger cut it rejected is what the per-Lane gap replaces) and the
|
||||
6-hour browser cooldown introduced with the browser-backed Sites; the
|
||||
1-hour browser rest was already cleared as safe by
|
||||
`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.
|
||||
@@ -30,7 +30,8 @@
|
||||
"28": "Allocation Patterns",
|
||||
"29": "Observability & Alerting",
|
||||
"30": "AGENTS.md",
|
||||
"32": "Repo Hard Constraints",
|
||||
"31": "Store",
|
||||
"32": "Open",
|
||||
"33": "Go Testing Guide",
|
||||
"34": "Session Store",
|
||||
"35": "Web UI Filter Logic",
|
||||
@@ -39,7 +40,7 @@
|
||||
"38": "novel-logic.test.js",
|
||||
"39": "UI Critique 2026-07-26A",
|
||||
"40": "UI Critique 2026-07-26B",
|
||||
"41": "sessions_test.go",
|
||||
"42": "pgtest.go",
|
||||
"43": "Issue Tracker & Triage",
|
||||
"44": "Ticket Workflow",
|
||||
"45": "Go Perf Alert Rules",
|
||||
@@ -122,6 +123,7 @@
|
||||
"122": "Identity comes from Discord OAuth; we store no passwords and send no email",
|
||||
"123": "The wire format stays flat and deliberately does not mirror the schema",
|
||||
"124": "ADR-0005: On-demand browser sidecar",
|
||||
"125": "sessions_test.go",
|
||||
"126": "Bookmark Manager",
|
||||
"127": "triage-labels.md",
|
||||
"128": "Cross-Ticket Contract",
|
||||
@@ -149,7 +151,6 @@
|
||||
"150": "Reviewer Subagent (opencode)",
|
||||
"151": "Finding Severity Rubric",
|
||||
"152": "Spec Compliance Review",
|
||||
"158": "T",
|
||||
"161": "Why Use samber/oops",
|
||||
"162": "singleflight Cache Stampede Prevention",
|
||||
"163": "Struct Field Alignment",
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
# Graph Report - mangaBookmark (2026-08-16)
|
||||
|
||||
## Corpus Check
|
||||
- 112 files · ~274,851 words
|
||||
- 113 files · ~279,161 words
|
||||
- Verdict: corpus is large enough that graph structure adds value.
|
||||
|
||||
## Summary
|
||||
- 1640 nodes · 3294 edges · 221 communities (67 shown, 154 thin omitted)
|
||||
- Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 313 edges (avg confidence: 0.77)
|
||||
- 1668 nodes · 3379 edges · 222 communities (68 shown, 154 thin omitted)
|
||||
- Extraction: 91% EXTRACTED · 9% INFERRED · 0% AMBIGUOUS · INFERRED: 320 edges (avg confidence: 0.77)
|
||||
- Token cost: 0 input · 0 output
|
||||
|
||||
## Graph Freshness
|
||||
- Built from commit: `f000cc7e`
|
||||
- Built from commit: `4f1cbcfd`
|
||||
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
|
||||
- Run `graphify update .` after code changes (no API cost).
|
||||
|
||||
@@ -45,7 +45,8 @@
|
||||
- [[_COMMUNITY_Find Skills Guide|Find Skills Guide]]
|
||||
- [[_COMMUNITY_Allocation Patterns|Allocation Patterns]]
|
||||
- [[_COMMUNITY_Observability & Alerting|Observability & Alerting]]
|
||||
- [[_COMMUNITY_Repo Hard Constraints|Repo Hard Constraints]]
|
||||
- [[_COMMUNITY_Store|Store]]
|
||||
- [[_COMMUNITY_Open|Open]]
|
||||
- [[_COMMUNITY_Go Testing Guide|Go Testing Guide]]
|
||||
- [[_COMMUNITY_Session Store|Session Store]]
|
||||
- [[_COMMUNITY_Web UI Filter Logic|Web UI Filter Logic]]
|
||||
@@ -54,7 +55,7 @@
|
||||
- [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]]
|
||||
- [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]]
|
||||
- [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]]
|
||||
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
|
||||
- [[_COMMUNITY_pgtest.go|pgtest.go]]
|
||||
- [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]]
|
||||
- [[_COMMUNITY_Ticket Workflow|Ticket Workflow]]
|
||||
- [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]]
|
||||
@@ -137,6 +138,7 @@
|
||||
- [[_COMMUNITY_Identity comes from Discord OAuth; we store no passwords and send no email|Identity comes from Discord OAuth; we store no passwords and send no email]]
|
||||
- [[_COMMUNITY_The wire format stays flat and deliberately does not mirror the schema|The wire format stays flat and deliberately does not mirror the schema]]
|
||||
- [[_COMMUNITY_ADR-0005 On-demand browser sidecar|ADR-0005: On-demand browser sidecar]]
|
||||
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
|
||||
- [[_COMMUNITY_Bookmark Manager|Bookmark Manager]]
|
||||
- [[_COMMUNITY_triage-labels|triage-labels.md]]
|
||||
- [[_COMMUNITY_Cross-Ticket Contract|Cross-Ticket Contract]]
|
||||
@@ -164,7 +166,6 @@
|
||||
- [[_COMMUNITY_Reviewer Subagent (opencode)|Reviewer Subagent (opencode)]]
|
||||
- [[_COMMUNITY_Finding Severity Rubric|Finding Severity Rubric]]
|
||||
- [[_COMMUNITY_Spec Compliance Review|Spec Compliance Review]]
|
||||
- [[_COMMUNITY_T|T]]
|
||||
- [[_COMMUNITY_Why Use samberoops|Why Use samber/oops]]
|
||||
- [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]]
|
||||
- [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]]
|
||||
@@ -239,14 +240,14 @@
|
||||
## God Nodes (most connected - your core abstractions)
|
||||
1. `testConfig()` - 53 edges
|
||||
2. `newWebTestServer()` - 49 edges
|
||||
3. `newTestStore()` - 43 edges
|
||||
4. `newTestStore()` - 42 edges
|
||||
3. `newTestStore()` - 48 edges
|
||||
4. `newTestStore()` - 43 edges
|
||||
5. `e()` - 33 edges
|
||||
6. `Handler` - 29 edges
|
||||
7. `ne()` - 28 edges
|
||||
8. `De()` - 28 edges
|
||||
9. `Open()` - 27 edges
|
||||
10. `se()` - 27 edges
|
||||
6. `Open()` - 29 edges
|
||||
7. `Handler` - 29 edges
|
||||
8. `ne()` - 28 edges
|
||||
9. `De()` - 28 edges
|
||||
10. `Store` - 27 edges
|
||||
|
||||
## Surprising Connections (you probably didn't know these)
|
||||
- `el()` --indirect_call--> `c()` [INFERRED]
|
||||
@@ -272,7 +273,7 @@
|
||||
- **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85]
|
||||
- **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85]
|
||||
|
||||
## Communities (221 total, 154 thin omitted)
|
||||
## Communities (222 total, 154 thin omitted)
|
||||
|
||||
### Community 0 - "HTMX Library Internals"
|
||||
Cohesion: 0.08
|
||||
@@ -284,30 +285,30 @@ Nodes (84): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecor
|
||||
|
||||
### Community 2 - "Manga Userscript Adapters"
|
||||
Cohesion: 0.06
|
||||
Nodes (77): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+69 more)
|
||||
Nodes (76): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+68 more)
|
||||
|
||||
### Community 3 - "Novel Userscript Adapters"
|
||||
Cohesion: 0.06
|
||||
Nodes (78): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+70 more)
|
||||
Nodes (79): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+71 more)
|
||||
|
||||
### Community 4 - "Series Acquisition Tests"
|
||||
Cohesion: 0.09
|
||||
Nodes (69): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+61 more)
|
||||
Cohesion: 0.08
|
||||
Nodes (78): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+70 more)
|
||||
|
||||
### Community 5 - "Bookmarks API Tests"
|
||||
Cohesion: 0.08
|
||||
Nodes (67): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+59 more)
|
||||
Cohesion: 0.06
|
||||
Nodes (74): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+66 more)
|
||||
|
||||
### Community 7 - "Cover & Acquire Internals"
|
||||
Cohesion: 0.10
|
||||
Nodes (30): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+22 more)
|
||||
Cohesion: 0.08
|
||||
Nodes (37): Addr, Context, Store, WaitGroup, isInterstitial(), defaultCoverResolver(), fetchCoverBytes(), Client (+29 more)
|
||||
|
||||
### Community 8 - "System Architecture Concepts"
|
||||
Cohesion: 0.13
|
||||
Nodes (20): app.html — App Shell Template, Manga/Novel Library Switch, Bookmark Bucket Tabs, Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key (+12 more)
|
||||
|
||||
### Community 9 - "Session Middleware"
|
||||
Cohesion: 0.07
|
||||
Cohesion: 0.08
|
||||
Nodes (35): ClearCookie(), ClientIP(), Duration, Mutex, Request, ResponseWriter, Time, isHTTPS() (+27 more)
|
||||
|
||||
### Community 10 - "Go Test Helpers"
|
||||
@@ -315,12 +316,12 @@ Cohesion: 0.05
|
||||
Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more)
|
||||
|
||||
### Community 11 - "Store Tests"
|
||||
Cohesion: 0.15
|
||||
Nodes (43): Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck(), seedSecondReader() (+35 more)
|
||||
Cohesion: 0.14
|
||||
Nodes (45): scanSeries(), Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck() (+37 more)
|
||||
|
||||
### Community 12 - "Bookmarks API Handler"
|
||||
Cohesion: 0.11
|
||||
Nodes (24): Auth(), compressible(), CORS(), Handler, ResponseWriter, Store, Gzip(), ResolveReader() (+16 more)
|
||||
Cohesion: 0.08
|
||||
Nodes (32): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+24 more)
|
||||
|
||||
### Community 13 - "Web UI Handlers"
|
||||
Cohesion: 0.14
|
||||
@@ -331,8 +332,8 @@ Cohesion: 0.06
|
||||
Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more)
|
||||
|
||||
### Community 15 - "CDP Browser Client"
|
||||
Cohesion: 0.07
|
||||
Nodes (36): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+28 more)
|
||||
Cohesion: 0.06
|
||||
Nodes (56): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+48 more)
|
||||
|
||||
### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented"
|
||||
Cohesion: 0.06
|
||||
@@ -343,8 +344,8 @@ Cohesion: 0.08
|
||||
Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more)
|
||||
|
||||
### Community 19 - "Store"
|
||||
Cohesion: 0.09
|
||||
Nodes (8): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, scanSeries(), TestDisplayChapter(), Bookmark, ReaderSummary
|
||||
Cohesion: 0.33
|
||||
Nodes (5): ADR-0010: Poll Lanes — one independent Poll stream per Site, Constraints preserved, Decision, Tradeoffs and rejections, Why
|
||||
|
||||
### Community 20 - "I/O Performance Patterns"
|
||||
Cohesion: 0.11
|
||||
@@ -371,24 +372,28 @@ Cohesion: 0.33
|
||||
Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more)
|
||||
|
||||
### Community 26 - "Open"
|
||||
Cohesion: 0.20
|
||||
Nodes (17): applyMigration(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen(), TestMigration0002BackfillsExistingBookmarks() (+9 more)
|
||||
Cohesion: 0.40
|
||||
Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct
|
||||
|
||||
### Community 27 - "Find Skills Guide"
|
||||
Cohesion: 0.14
|
||||
Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more)
|
||||
|
||||
### Community 28 - "Allocation Patterns"
|
||||
Cohesion: 0.11
|
||||
Nodes (19): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map of pointers for large, frequently updated structs, Map size hints (+11 more)
|
||||
Cohesion: 0.14
|
||||
Nodes (14): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map size hints, Memory Optimization (+6 more)
|
||||
|
||||
### Community 29 - "Observability & Alerting"
|
||||
Cohesion: 0.22
|
||||
Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more)
|
||||
|
||||
### Community 32 - "Repo Hard Constraints"
|
||||
Cohesion: 0.18
|
||||
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
|
||||
### Community 31 - "Store"
|
||||
Cohesion: 0.12
|
||||
Nodes (3): coverRelativePath(), coverSourceAddress(), Store
|
||||
|
||||
### Community 32 - "Open"
|
||||
Cohesion: 0.14
|
||||
Nodes (20): applyMigration(), displayChapter(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen() (+12 more)
|
||||
|
||||
### Community 33 - "Go Testing Guide"
|
||||
Cohesion: 0.20
|
||||
@@ -418,9 +423,9 @@ Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations,
|
||||
Cohesion: 0.29
|
||||
Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider
|
||||
|
||||
### Community 41 - "sessions_test.go"
|
||||
Cohesion: 0.48
|
||||
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
|
||||
### Community 42 - "pgtest.go"
|
||||
Cohesion: 0.18
|
||||
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
|
||||
|
||||
### Community 45 - "Go Perf Alert Rules"
|
||||
Cohesion: 0.50
|
||||
@@ -530,28 +535,28 @@ Nodes (3): Consequence, The wire format stays flat and deliberately does not mir
|
||||
Cohesion: 0.50
|
||||
Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision
|
||||
|
||||
### Community 158 - "T"
|
||||
Cohesion: 0.08
|
||||
Nodes (38): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Request, ReaderID() (+30 more)
|
||||
### Community 125 - "sessions_test.go"
|
||||
Cohesion: 0.48
|
||||
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
|
||||
|
||||
### Community 273 - "AGENTS.md"
|
||||
Cohesion: 0.50
|
||||
Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust), Second script: `novel-bookmark.user.js`, Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
||||
|
||||
## Knowledge Gaps
|
||||
- **520 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+515 more)
|
||||
- **524 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+519 more)
|
||||
These have ≤1 connection - possible missing edges or undocumented components.
|
||||
- **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||
|
||||
## Suggested Questions
|
||||
_Questions this graph is uniquely positioned to answer:_
|
||||
|
||||
- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`, `Open`?**
|
||||
_High betweenness centrality (0.051) - this node is a cross-community bridge._
|
||||
- **Why does `Open()` connect `Open` to `Repo Hard Constraints`, `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `Store Tests`, `Store`?**
|
||||
_High betweenness centrality (0.038) - this node is a cross-community bridge._
|
||||
- **Why does `New()` connect `Series Acquisition Tests` to `Open`, `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`?**
|
||||
_High betweenness centrality (0.045) - this node is a cross-community bridge._
|
||||
- **Why does `Open()` connect `Open` to `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `pgtest.go`, `Store Tests`, `Store`?**
|
||||
_High betweenness centrality (0.033) - this node is a cross-community bridge._
|
||||
- **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?**
|
||||
_High betweenness centrality (0.030) - this node is a cross-community bridge._
|
||||
_High betweenness centrality (0.027) - this node is a cross-community bridge._
|
||||
- **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?**
|
||||
_`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?**
|
||||
@@ -559,4 +564,4 @@ _Questions this graph is uniquely positioned to answer:_
|
||||
- **Are the 12 inferred relationships involving `newTestStore()` (e.g. with `TestAcquireDoesNotBlockTheWrite()` and `TestAcquireFailureLeavesTheBookmarkIntact()`) actually correct?**
|
||||
_`newTestStore()` has 12 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?**
|
||||
_553 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
_557 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
File diff suppressed because one or more lines are too long
+2499
-1308
File diff suppressed because it is too large
Load Diff
+50
-45
@@ -25,9 +25,9 @@
|
||||
"semantic_hash": "dac242903b0e98c3e4395159d609e08e"
|
||||
},
|
||||
"backend/main.go": {
|
||||
"mtime": 1786501521.228955,
|
||||
"ast_hash": "6e98a3ae91aaa132df251e43c4dfca6d",
|
||||
"semantic_hash": "6e98a3ae91aaa132df251e43c4dfca6d"
|
||||
"mtime": 1786859325.126449,
|
||||
"ast_hash": "4ad77c286522f97d14b6009ff37118e5",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"skills-lock.json": {
|
||||
"mtime": 1784884678.6842625,
|
||||
@@ -140,18 +140,18 @@
|
||||
"semantic_hash": "3a08979e4603aae5c32a58d5b6c39765"
|
||||
},
|
||||
"CLAUDE.md": {
|
||||
"mtime": 1786856633.445473,
|
||||
"ast_hash": "fab288c23be960d9c6afbfe3f21ff41c",
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "c79e49f912d7852f9832565a5f9a1c39",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"DEPLOY.md": {
|
||||
"mtime": 1786501942.7367291,
|
||||
"ast_hash": "3c7b785c44badb6dda6234d2b39a9290",
|
||||
"mtime": 1786859620.9977603,
|
||||
"ast_hash": "b7c2f813aded562ee9291c9baed795ad",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"README.md": {
|
||||
"mtime": 1786499529.7651505,
|
||||
"ast_hash": "9d6be8aa8a2946c23ad48d8f2864b5ca",
|
||||
"mtime": 1786859273.3238454,
|
||||
"ast_hash": "81af12a0a2d43e791efd030b5f4d6cbc",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docker-compose.prod.yml": {
|
||||
@@ -160,8 +160,8 @@
|
||||
"semantic_hash": "0751998a532297b8ac507a01ec48dc31"
|
||||
},
|
||||
"docker-compose.yml": {
|
||||
"mtime": 1786499529.7725692,
|
||||
"ast_hash": "124fd581bf0a662ff15012abfdb40a92",
|
||||
"mtime": 1786859258.6255276,
|
||||
"ast_hash": "d3b53a8f42a8e0acb4fc4306ea42c093",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
".claude/settings.json": {
|
||||
@@ -175,9 +175,9 @@
|
||||
"semantic_hash": "8f1b093b59eb1ed81bc7fc0c22495c50"
|
||||
},
|
||||
"backend/main_test.go": {
|
||||
"mtime": 1786501521.228955,
|
||||
"ast_hash": "8a165955cf28ad47481fec5ea7afb3d6",
|
||||
"semantic_hash": "8a165955cf28ad47481fec5ea7afb3d6"
|
||||
"mtime": 1786859172.6449091,
|
||||
"ast_hash": "0a5d4dbdc770b40c329ccccc899b59f4",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
".claude/settings.local.json": {
|
||||
"mtime": 1785697645.350201,
|
||||
@@ -200,8 +200,8 @@
|
||||
"semantic_hash": "e69a8340a371579ca3ea689660f7d7bd"
|
||||
},
|
||||
"AGENTS.md": {
|
||||
"mtime": 1786856633.445473,
|
||||
"ast_hash": "fab288c23be960d9c6afbfe3f21ff41c",
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "c79e49f912d7852f9832565a5f9a1c39",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"userscript/test/logic.test.js": {
|
||||
@@ -215,7 +215,7 @@
|
||||
"semantic_hash": "8f3c0132eb4787a2c8736eb99f7689af"
|
||||
},
|
||||
"REDEPLOY.md": {
|
||||
"mtime": 1786856643.0770833,
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "e5e910f0244a040e2ccdc669b17eb4db",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -245,12 +245,12 @@
|
||||
"semantic_hash": "385b36f58488b7e6d93eb6d6034e9ee3"
|
||||
},
|
||||
"backend/internal/latest/browser.go": {
|
||||
"mtime": 1786856879.3397639,
|
||||
"ast_hash": "fbdba4bb56d8c804087bcb6e141e52ad",
|
||||
"mtime": 1786859355.8677974,
|
||||
"ast_hash": "75f34a974568d122681939dd297944a2",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/browser_test.go": {
|
||||
"mtime": 1786856254.36163,
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "800fa6aa471ada054a3c48943ad17ab7",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -260,22 +260,22 @@
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/poller.go": {
|
||||
"mtime": 1786856670.129715,
|
||||
"ast_hash": "6b7060ff52994729832d9fcc3cc97dad",
|
||||
"mtime": 1786860083.6713927,
|
||||
"ast_hash": "be728405cde3df49baf72bd7837f2971",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/poller_test.go": {
|
||||
"mtime": 1786856311.8535168,
|
||||
"ast_hash": "b6837fab377c94fa1d2f20a931975a4a",
|
||||
"mtime": 1786860168.103984,
|
||||
"ast_hash": "5dba515d28c0523500c1ceb581a54e58",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/sites.go": {
|
||||
"mtime": 1786856865.4529688,
|
||||
"ast_hash": "2f00456ac9d1d8148ba1bc6cbceb24f6",
|
||||
"mtime": 1786860007.8224697,
|
||||
"ast_hash": "9b90f9b710ccbd0d0a0a6a0dd721ff3c",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/sites_test.go": {
|
||||
"mtime": 1786856890.1231265,
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "0db2028a6073f342fee61ad15c2e5f0f",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -340,14 +340,14 @@
|
||||
"semantic_hash": "0b6764a0ee20f5cb7748eecd31a1d220"
|
||||
},
|
||||
"backend/internal/store/store.go": {
|
||||
"mtime": 1786363889.5602942,
|
||||
"ast_hash": "54367a8ab043983e2491b2eb2650961c",
|
||||
"semantic_hash": "54367a8ab043983e2491b2eb2650961c"
|
||||
"mtime": 1786860012.3431706,
|
||||
"ast_hash": "c62e73142386c58ab833d404d0c24dc4",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/store/store_test.go": {
|
||||
"mtime": 1786363889.5602942,
|
||||
"ast_hash": "dc823fd77bcce2268114e31d759b20a5",
|
||||
"semantic_hash": "dc823fd77bcce2268114e31d759b20a5"
|
||||
"mtime": 1786858248.9426548,
|
||||
"ast_hash": "039729554517960d1ae98f8409baf4ba",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/userscript/userscript.go": {
|
||||
"mtime": 1786216141.692644,
|
||||
@@ -415,8 +415,8 @@
|
||||
"semantic_hash": "e44a2f6f624db044e19508bc5ab05592"
|
||||
},
|
||||
"CONTEXT.md": {
|
||||
"mtime": 1786855457.9327722,
|
||||
"ast_hash": "24548f60414b4c5ff58538acaada5345",
|
||||
"mtime": 1786859395.4497814,
|
||||
"ast_hash": "4aafbce0b046e6e34734fb414df818ce",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"CUTOVER.md": {
|
||||
@@ -425,13 +425,13 @@
|
||||
"semantic_hash": "6c6f3e4c4c2f57867894280bce728c50"
|
||||
},
|
||||
"backend/AGENTS.md": {
|
||||
"mtime": 1786856656.6826186,
|
||||
"ast_hash": "d5610ba24076b57e17b9d76241acaa65",
|
||||
"mtime": 1786860230.7890837,
|
||||
"ast_hash": "7ceec7c4576f6e91eaf02c200a0dd1b3",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/CLAUDE.md": {
|
||||
"mtime": 1786856656.6826186,
|
||||
"ast_hash": "d5610ba24076b57e17b9d76241acaa65",
|
||||
"mtime": 1786860230.7890837,
|
||||
"ast_hash": "7ceec7c4576f6e91eaf02c200a0dd1b3",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/web/templates/app.html": {
|
||||
@@ -560,7 +560,7 @@
|
||||
"semantic_hash": "46cf7822d4f667e3cab36b547abe5e97"
|
||||
},
|
||||
"backend/internal/latest/cover.go": {
|
||||
"mtime": 1786856664.2957466,
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "d5f2248c3d11de74bf5a3977651b17c2",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
@@ -570,8 +570,8 @@
|
||||
"semantic_hash": "60d9eb7c59a3751baf4f31c7655217e7"
|
||||
},
|
||||
"backend/internal/latest/acquire.go": {
|
||||
"mtime": 1786499529.7688599,
|
||||
"ast_hash": "6c1ad34bbe9f5b49d0fd1eae9093f55d",
|
||||
"mtime": 1786859346.3297389,
|
||||
"ast_hash": "d605e3c94a62fc7787efbc139696b9d2",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/acquire_test.go": {
|
||||
@@ -615,8 +615,8 @@
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/read.go": {
|
||||
"mtime": 1786499529.7688599,
|
||||
"ast_hash": "3cf29046ddaef39fafb1df70b9f9ae8c",
|
||||
"mtime": 1786859355.8679621,
|
||||
"ast_hash": "9f039cc3ad74f803d7621f7ef4157bf2",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/research/cloudflare-bot-scoring-and-poll-cadence.md": {
|
||||
@@ -625,8 +625,13 @@
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"backend/internal/latest/smoke_comix_test.go": {
|
||||
"mtime": 1786856356.3432186,
|
||||
"mtime": 1786857336.6414917,
|
||||
"ast_hash": "111fdbb75fc68ac2ab1013bc916063cf",
|
||||
"semantic_hash": ""
|
||||
},
|
||||
"docs/adr/0010-poll-lanes-per-site-pace.md": {
|
||||
"mtime": 1786860214.4637265,
|
||||
"ast_hash": "dc19d75f034ca920d93b9c71e6ca28e6",
|
||||
"semantic_hash": ""
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user