feat: one Poll Lane per Site, replacing the shared pace (#100)
Each Site runs its own poll goroutine, paced by Rest and Gap from the registry (sites.go) instead of the shared cooldown/interval/stagger/batch config. Rest is enforced by the due query's WHERE clause; the Lane sleeps its effective gap between fetches (hour/eligible, floored at 1s). Lane-local failures: two refusals stop that Site for 15m, a lost browser stops only the round's remaining browser Lanes, and cover heals moved to background goroutines so a slow CDN cannot consume a Lane's gap. The five LATEST_CHAPTER_POLL_* pace env vars are gone; only the kill switch remains.
This commit is contained in:
@@ -120,7 +120,7 @@ func (a *Acquirer) acquire(ctx context.Context, sr store.Series) {
|
||||
// Stamped after success — the reverse of the poller, which stamps before
|
||||
// the fetch: the Reader is here, watching the Series they just created, so
|
||||
// a failed acquisition must leave the row due for a fast retry rather than
|
||||
// consuming the cooldown. The stamp happens even when the page read
|
||||
// consuming the rest. The stamp happens even when the page read
|
||||
// succeeded but produced no facts to persist.
|
||||
if err := a.Store.MarkLatestChecked(sr.Site, sr.SeriesID, time.Now().UnixMilli()); err != nil {
|
||||
log.Printf("acquire %q: mark checked: %v", sr.Key(), err)
|
||||
|
||||
Reference in New Issue
Block a user