Admin dashboard: pages, Lane observability, poll pass log, per-Series intervention (#134) (#148)

Spec #134, all ten tickets. Closes #134.

## What ships

The admin surface becomes four bookmarkable addresses behind one nav row, and Lane observability stops dying with the process.

- **#138** `/admin` splits into Overview, Lanes, Readers, Series, each a real route with the active tab underlined.
- **#139** `poll_passes` and `poll_lanes` land as durable tables with their store surface.
- **#140** cross-Series admin read model, with the privacy boundary in the projection: the Reader id that raised a Latest Chapter never leaves the store package.
- **#141** the poller records exactly one pass row per exit, with a skip reason and outcome counts.
- **#142** Series list: eight hygiene filters, Site and Library narrowing, paging — all of it in the query string, so a filtered list is a bookmark.
- **#143** Overview: a three-state verdict line and a stats block where every non-zero figure links to the list that counts it.
- **#144** per-Series detail page, keyed by the `site:series_id` composite the rest of the system already uses.
- **#145** the Lanes page reads the database; the in-memory Lane state, `web.LaneReporter` and `latest.Status` are deleted.
- **#146** Forced Poll: *Check now* stamps `series.force_poll_at` and never commands the poller.
- **#147** pause and resume one Site's Lane, with a mandatory 1h/6h/24h expiry.

## Shape of the design

Two decisions carry the rest. **Commands go through the database, never at the poller**: both *Check now* and a Lane pause write a row the next pass reads, so they survive a restart and the whole surface stays testable with no poller running. And **pending is derived, never stored** — the request stamp being newer than the check stamp — which self-clears on the check stamp with no second write and no sweeper, because the check stamp is written before the fetch.

ADRs: `docs/adr/0012-persisted-lane-state.md`, `docs/adr/0013-commands-through-the-database.md`.

## Verification

`go test ./...` green on the merged base (`264839e`), all packages, Docker-backed. `gofmt -l` and `go vet` clean.

Every ticket was reviewed on both axes (`cr-spec` + `cr-standards`) before merge.

## Known, non-blocking

- **#143** the verdict ignores never-reported Lanes when other Lanes have reported, and the per-Site table lists Sites that have Series rather than the whole registry. The ticket prose asks for eight hygiene figures per Site; the design mock and the landed `.tbl.sites` grid both say six columns, and the mock won.
- **#146** two `SeriesPage` scans per press instead of a keyed read — `ponytail:`-commented in-tree with the upgrade path.
- **#147** a paused Site with no pass row yet renders no row and so no control, since the Lanes page lists Sites that have passed.
- **#141** a sibling browser Lane declining at the top of a pass records as `sidecar-down`. Specified deliberately; the later spec in this series settles it.

Reviewed-on: #148
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
This commit was merged in pull request #148.
This commit is contained in:
2026-08-22 08:27:19 +07:00
committed by sulthan
parent 0a245a0dde
commit 889f0f3f38
33 changed files with 6112 additions and 634 deletions
+242 -75
View File
@@ -5,7 +5,6 @@ import (
"errors"
"log"
"net/url"
"sort"
"sync"
"time"
@@ -48,19 +47,22 @@ type Poller struct {
// same failure-isolated prefetch path.
CoverBytesFetch CoverBytesFetcher
Now func() time.Time // injected so tests can freeze it
// eligibleCount reports how many of a Site's Series are eligible for
// polling, defaulting to Store.EligibleSeriesCount. Injected so tests can
// fail the count alone: the eligible query shares the due query's tables,
// so no real store failure can reach this path without breaking the due
// query first (issue #141).
eligibleCount func(site string) (int, error)
// refuseUntil gates a Site's Lane after it refused twice in one run: no
// Series of that Site is attempted again before this time (issue #100).
// The stamp is durable — the pass gate reads it from the store, so a
// restart does not forget the refusal; nothing of it lives in memory.
// browserDownAt is when a browser Lane last lost the sidecar; the other
// browser Lanes skip their passes for the next refuseBackoff, so a
// browser Lanes skip their passes for the next RefuseBackoff, so a
// restarting Chrome does not stamp one Series per pass per Lane (story 20).
mu sync.Mutex
refuseUntil map[string]time.Time
browserDownAt time.Time
// laneStates is the owner's page snapshot of each Lane's last pass
// (issue #102), keyed by Site. Guarded by mu; a Site appears only after
// its first pass, so a restart renders "no data yet" rather than zeroes.
laneStates map[string]LaneState
// coverWG tracks in-flight cover work. Covers heal in the background so a
// slow cover host cannot delay the next Series-page Poll; tests join it
// before asserting on cover fetches.
@@ -167,14 +169,7 @@ func fetcherFor(site string, browser, tls Fetcher) Fetcher {
// laneNames returns every registry Site in the deterministic order both Run
// and runOnce iterate: sorted, so lane behaviour and its tests agree on who
// runs first.
func laneNames() []string {
names := make([]string, 0, len(sites))
for name := range sites {
names = append(names, name)
}
sort.Strings(names)
return names
}
func laneNames() []string { return SiteNames() }
func (p *Poller) Run(ctx context.Context) {
names := laneNames()
@@ -216,28 +211,120 @@ func (p *Poller) runOnce(ctx context.Context) {
}
}
// One skip value per way a Lane Pass can return before its loop (issue #141);
// empty means the pass reached the loop. The values are wire strings — stored
// in poll_passes and read by the Lanes page — so they are stable, not prose.
const (
// Exported so the web layer renders a skip's reason without retyping the
// wire string (issue #145); the values are storage and page-stable.
SkipPaused = "paused" // the pause row was read at the top
SkipRefusing = "refusing" // refusal backoff
SkipSidecarDown = "sidecar-down" // a sibling browser Lane lost Chrome
SkipNoFetcher = "no-fetcher" // browser Site, no browser configured, no fallback
SkipDueQuery = "due-query" // the due query failed
SkipAsleep = "asleep" // under both browser wake thresholds
SkipEligibleCount = "eligible-count" // the eligible count failed
SkipNothingEligible = "nothing-eligible" // nothing eligible; sleeps a full rest
)
// readOutcome classifies one Series read for the pass row's outcome counts
// (issue #141). The classification the read already makes is counted, never a
// second taxonomy: refused is the Site holding a challenge, unreachable the
// browser interrupting, noChapter a 200 with real HTML but no chapter links,
// unfetchable the host pin or a missing fetcher, and errors everything else.
type readOutcome int
const (
outcomeSuccess readOutcome = iota
outcomeRefused
outcomeUnreachable
outcomeNoChapter
outcomeUnfetchable
outcomeError
)
// outcomeCounts are the five named outcome counts of one pass. A success
// count is derived, never stored: checked minus the four, with unreachable
// excluded because the sidecar-loss path returns before the checked counter
// increments (issue #141).
type outcomeCounts struct {
refused, unreachable, noChapter, unfetchable, errors int
}
func (c *outcomeCounts) add(o readOutcome) {
switch o {
case outcomeRefused:
c.refused++
case outcomeUnreachable:
c.unreachable++
case outcomeNoChapter:
c.noChapter++
case outcomeUnfetchable:
c.unfetchable++
case outcomeError:
c.errors++
}
}
// passRecord is what one pass's durable row will be: the skip value and
// outcome counts filled in along the pass's return path. recordPass assembles
// the row, so every exit records exactly once.
type passRecord struct {
site string
ranAt int64
skip string
counts outcomeCounts
}
// lanePassRetention is how far back a Lane's pass log is kept. It is not the
// display window: retention is how far back a question can reach, and the
// window is what the owner is shown (issue #139).
const lanePassRetention = 14 * 24 * time.Hour
// runLanePass processes one pass of one Site's Lane: select the due Series,
// pace through them, and report how long the Lane should wait before its next
// pass. paced spaces consecutive fetches by the Site's effective gap — the
// production Lane's rate limit; the deterministic test entry runs back to back.
func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.Duration {
now := p.Now()
// Snapshot this pass for the owner's page (issue #102). Recorded on every
// return path, with the figures filled in where the pass computes them.
st := LaneState{Site: name, LastRun: now, Browser: isBrowserSite(name)}
defer func() { p.recordLaneState(st) }()
if until := p.refusalBackoff(name); now.Before(until) {
// Durable pass log (issue #141): one row per exit. The figures are filled
// in as the pass measures them; a pass that returns before measuring
// carries the previous pass's forward inside recordPass.
fig := passFigures{}
rec := passRecord{site: name, ranAt: now.UnixMilli()}
defer func() { p.recordPass(rec, fig) }()
// One Lane row read at the top of a pass, serving two gates (issue #139).
// Both stamps outlive our process, so the gates read the durable row
// rather than memory: a refusal is the Site's mood and a pause the
// owner's order, and neither is lost to a restart.
pausedUntil, refuseUntil, err := p.Store.LaneGates(name)
if err != nil {
// Fail open: a store that cannot answer the gate cannot record the
// pass either, and one Lane must not stall on its own gate read.
log.Printf("latest poll %s: lane gates: %v", name, err)
}
if pausedUntil > now.UnixMilli() {
// Paused ahead of the refusal check: no Series is touched, so the
// queue stays intact for when the pause lifts (issue #141, #147).
rec.skip = SkipPaused
log.Printf("latest poll %s: paused until %s, skipping pass", name, time.UnixMilli(pausedUntil).Format(time.RFC3339))
return time.Duration(pausedUntil-now.UnixMilli()) * time.Millisecond
}
if refuseUntil > now.UnixMilli() {
// Cooling down after a refusal: do not attempt this Site at all.
return until.Sub(now)
rec.skip = SkipRefusing
return time.Duration(refuseUntil-now.UnixMilli()) * time.Millisecond
}
if isBrowserSite(name) {
if downFor, down := p.browserDownFor(now); down && downFor < refuseBackoff {
if downFor, down := p.browserDownFor(now); down && downFor < RefuseBackoff {
// A sibling browser Lane lost the sidecar within the backoff
// window: skip this pass, so a restarting Chrome does not stamp
// this Site's Series one pass at a time. After refuseBackoff the
// this Site's Series one pass at a time. After RefuseBackoff the
// flag decays and the Lane probes again (issue #100, story 20).
rec.skip = SkipSidecarDown
log.Printf("latest poll %s: browser lane skipping pass (sidecar down %s ago)", name, downFor)
return refuseBackoff - downFor
return RefuseBackoff - downFor
}
}
s := sites[name]
@@ -246,24 +333,29 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
// No fetcher at all right now (browser absent, no fallback): every
// Series stays unstamped and due, so a browser that appears after a
// restart finds its full queue waiting (issue #100).
st.Gap = defaultGap
rec.skip = SkipNoFetcher
fig.Gap = defaultGap
return defaultGap
}
due, err := p.Store.DueForLatestCheck(name, now.Add(-s.Rest).UnixMilli(),
now.Add(-sightingCeilingRests*s.Rest).UnixMilli())
if err != nil {
rec.skip = SkipDueQuery
log.Printf("latest poll %s: due query: %v", name, err)
st.Gap = defaultGap
fig.Gap = defaultGap
return defaultGap
}
st.Due = len(due)
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) {
fig.Due = len(due)
if s.Browser != nil && f == p.BrowserFetch && !browserWakeDue(due, now, s.Rest) && !anyForced(due) {
// Below both thresholds Chrome stays asleep (ADR-0005 on-demand
// browser): waking it for a single Poll would cost a challenge solve
// per request. The Lane still paces at the default gap, which is what
// the owner's page must show rather than a zero.
st.Gap, st.Asleep = defaultGap, true
// per request. A forced Series is the one exception — a human asking
// is not the machine waking itself (issue #146). The Lane still paces
// at the default gap, which is what the owner's page must show rather
// than a zero.
rec.skip = SkipAsleep
fig.Gap = defaultGap
return defaultGap
}
if s.Browser != nil {
@@ -276,20 +368,22 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
}
}
eligible, err := p.Store.EligibleSeriesCount(name)
eligible, err := p.countEligible(name)
if err != nil {
rec.skip = SkipEligibleCount
log.Printf("latest poll %s: eligible count: %v", name, err)
st.Gap = defaultGap
fig.Gap = defaultGap
return defaultGap
}
gap, clamped := effectiveGap(s, eligible)
st.Gap, st.Clamped = gap, clamped
fig.Gap, fig.Clamped = gap, clamped
if clamped {
log.Printf("latest poll %s: gap clamped to %s floor (eligible series=%d)", name, minGap, eligible)
}
if eligible == 0 {
// Nothing to poll for the foreseeable future; sleep a full rest instead
// of re-querying every gap.
rec.skip = SkipNothingEligible
return s.Rest
}
@@ -300,7 +394,7 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
}
if refusals >= 2 {
// This Site refused twice in a row: the remaining Series are left
// unstamped and due, and the Lane waits refuseBackoff before
// unstamped and due, and the Lane waits RefuseBackoff before
// trying it again.
break
}
@@ -314,46 +408,98 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
break
}
}
if err := p.checkOne(ctx, sr); err != nil {
switch {
case errors.Is(err, errChallengeHeld):
refusals++
case errors.Is(err, errBrowserInterrupted):
p.setBrowserDown(now)
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, refuseBackoff)
return gap
default:
refusals = 0
}
outcome := p.checkOne(ctx, sr)
if outcome == outcomeUnreachable {
// The mid-loop browser loss writes an empty skip on purpose: the
// pass returns before the checked counter increments, so its row
// is stall-shaped (due > 0, checked 0, skip ''), and a stall is
// the exact signal this exit produces. A tenth skip value would
// make it legible but is deliberately not invented here.
rec.counts.add(outcome)
p.setBrowserDown(now)
log.Printf("latest poll %s: browser unreachable, browser lanes skipping passes for %s", name, RefuseBackoff)
return gap
}
if outcome == outcomeRefused {
refusals++
} else {
refusals = 0
}
st.Checked++
rec.counts.add(outcome)
fig.Checked++
}
if st.Checked > 0 {
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), st.Checked)
if fig.Checked > 0 {
log.Printf("latest poll %s: due=%d checked=%d", name, len(due), fig.Checked)
}
if refusals >= 2 {
p.setRefusalBackoff(name, now.Add(refuseBackoff))
log.Printf("latest poll %s: refused twice this run, waiting %s", name, refuseBackoff)
return refuseBackoff
// The refusal outlives the process: the durable stamp gates a restart,
// so a Site that just told us to back off is not re-probed.
if err := p.Store.SetLaneRefusal(name, now.Add(RefuseBackoff).UnixMilli()); err != nil {
log.Printf("latest poll %s: persist refusal: %v", name, err)
}
log.Printf("latest poll %s: refused twice this run, waiting %s", name, RefuseBackoff)
return RefuseBackoff
}
return gap
}
func (p *Poller) refusalBackoff(name string) time.Time {
p.mu.Lock()
defer p.mu.Unlock()
return p.refuseUntil[name]
// passFigures are the numbers one pass measured for its durable row (issue
// #141): due and checked as the pass saw them, the pace it chose, and whether
// the gap sat on the floor. A pass that returned before measuring keeps the
// previous pass's figures via carry-forward in recordPass; the in-memory
// snapshot those once mirrored into is gone — the page reads the durable row
// now (issue #145).
type passFigures struct {
Due, Checked int
Gap time.Duration
Clamped bool
}
func (p *Poller) setRefusalBackoff(name string, until time.Time) {
p.mu.Lock()
defer p.mu.Unlock()
if p.refuseUntil == nil {
p.refuseUntil = make(map[string]time.Time)
// recordPass writes the durable row for one pass (issue #141). Called deferred
// from runLanePass so every return path records exactly one row. A pass that
// never computed its own figures — its gap is zero — carries the previous
// pass's due, gap, clamped and checked forward rather than stating zeroes it
// did not measure; the skip column says why it declined, so the zeroes that
// remain (due-query, no-fetcher) read as explanations rather than
// measurements.
func (p *Poller) recordPass(rec passRecord, fig passFigures) {
row := store.LanePass{
Site: rec.site,
RanAt: rec.ranAt,
Skip: rec.skip,
Due: fig.Due,
Checked: fig.Checked,
GapMS: fig.Gap.Milliseconds(),
Clamped: fig.Clamped,
Refused: rec.counts.refused,
Unreachable: rec.counts.unreachable,
NoChapter: rec.counts.noChapter,
Unfetchable: rec.counts.unfetchable,
Errors: rec.counts.errors,
}
p.refuseUntil[name] = until
if row.GapMS == 0 {
// The pass never computed a gap, so it has no figures of its own:
// carry the previous pass's, in one latest-per-Site read — the
// recorder needs one Site, not six (issue #139).
if prev, ok, err := p.Store.LatestLanePass(rec.site); err != nil {
log.Printf("latest poll %s: previous pass: %v", rec.site, err)
} else if ok {
row.Due, row.Checked = prev.Due, prev.Checked
row.GapMS, row.Clamped = prev.GapMS, prev.Clamped
}
}
if err := p.Store.RecordLanePass(row, rec.ranAt-lanePassRetention.Milliseconds()); err != nil {
log.Printf("latest poll %s: record lane pass: %v", rec.site, err)
}
}
// countEligible routes the eligible count through the test seam when one is
// set, else the store.
func (p *Poller) countEligible(site string) (int, error) {
if p.eligibleCount != nil {
return p.eligibleCount(site)
}
return p.Store.EligibleSeriesCount(site)
}
// setBrowserDown records when a browser Lane lost the sidecar. It is Poller
@@ -366,7 +512,7 @@ func (p *Poller) setBrowserDown(now time.Time) {
// browserDownFor reports how long the sidecar has been down and that it is
// down at all — the zero time means never down, which must not read as a
// zero-duration loss. The window decays: once refuseBackoff passes without a
// zero-duration loss. The window decays: once RefuseBackoff passes without a
// fresh loss, Lanes probe again.
func (p *Poller) browserDownFor(now time.Time) (time.Duration, bool) {
p.mu.Lock()
@@ -395,6 +541,19 @@ func browserWakeDue(due []store.Series, now time.Time, rest time.Duration) bool
return maxSeriesWait(due, now, rest) >= browserWakeAge
}
// anyForced reports whether the due list holds a forced Series: one whose
// owner check-now request (issue #146) has not been answered yet. A human
// asking wakes a sleeping Chrome even below the wake thresholds; the request
// itself still ages visibly if the home machine is off.
func anyForced(due []store.Series) bool {
for _, sr := range due {
if sr.Forced {
return true
}
}
return false
}
// maxSeriesWait returns how long the most-overdue of the due Series has been
// waiting past its due moment (0 when due is empty).
func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.Duration {
@@ -409,13 +568,15 @@ func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.D
// checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a
// Lane or take down the process. The returned error is the page read's
// classified outcome so the Lane can tell a refusal from a loss of the
// browser; non-classified failures still return nil-equivalent behaviour.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// Lane or take down the process. The returned outcome classifies the read for
// the pass row (issue #141), so the Lane can count a refusal, a lost browser,
// a chapter-less page, an unfetchable address or a transport error without
// re-deriving the taxonomy.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOutcome) {
defer func() {
if r := recover(); r != nil {
log.Printf("latest poll %q: recovered from panic: %v", sr.Key(), r)
outcome = outcomeError
}
}()
@@ -427,7 +588,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// the stamp means "attempted", and an untried Series stays due.
if err := p.Store.MarkLatestChecked(sr.Site, sr.SeriesID, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: mark checked: %v", sr.Key(), err)
return nil
return outcomeError
}
facts, err := readSeriesPage(ctx, sr.Site, sr.SeriesURL, p.BrowserFetch, p.Fetch)
@@ -438,17 +599,23 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// passes the gate is retried at rest pace rather than
// hot-looping.
log.Printf("latest poll %q: not fetchable: site=%q url=%q", sr.Key(), sr.Site, sr.SeriesURL)
return err
return outcomeUnfetchable
case errors.Is(err, errNoFetcher):
log.Printf("latest poll %q: no fetcher for site %q", sr.Key(), sr.Site)
return err
return outcomeUnfetchable
}
// A legacy cover heals independently of the page read: its source may
// answer — a CDN — while the origin does not, so a fetch failure does
// not skip the heal, matching the order the shared read replaced.
p.healCover(ctx, sr)
log.Printf("latest poll %q: %v", sr.Key(), err)
return err
if errors.Is(err, errChallengeHeld) {
return outcomeRefused
}
if errors.Is(err, errBrowserInterrupted) {
return outcomeUnreachable
}
return outcomeError
}
// A legacy cover source is healed independently of the page read.
p.healCover(ctx, sr)
@@ -459,7 +626,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a rest instead of hot-looping.
log.Printf("latest poll %q: no chapter links in %d bytes", sr.Key(), facts.BodyLen)
return nil
return outcomeNoChapter
}
// The Poll is the oracle for whatever Sighting last raised this Series
@@ -472,7 +639,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// against the due-query snapshot; a concurrent write in between only costs
// one redundant UPDATE of the same absolute value, never a wrong one.
if sr.LatestChapterNum != nil && *sr.LatestChapterNum == facts.Latest.Num {
return nil
return outcomeSuccess
}
// Series-level write: the row is shared, so one update refreshes every
@@ -481,10 +648,10 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) error {
// the list.
if err := p.Store.SetLatestChapter(sr.Site, sr.SeriesID, facts.Latest.Label, facts.Latest.Num); err != nil {
log.Printf("latest poll %q: set latest chapter: %v", sr.Key(), err)
return nil
return outcomeError
}
log.Printf("latest poll %q: latest is now %s", sr.Key(), facts.Latest.Label)
return nil
return outcomeSuccess
}
// judgeSighting settles the Sighting the Series' stored Latest Chapter is owed