sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Admin dashboard: pages, Lane observability, poll pass log, per-Series intervention (#134) (#148)

Browse Source 
Spec #134, all ten tickets. Closes #134.

## What ships

The admin surface becomes four bookmarkable addresses behind one nav row, and Lane observability stops dying with the process.

- **#138** `/admin` splits into Overview, Lanes, Readers, Series, each a real route with the active tab underlined.
- **#139** `poll_passes` and `poll_lanes` land as durable tables with their store surface.
- **#140** cross-Series admin read model, with the privacy boundary in the projection: the Reader id that raised a Latest Chapter never leaves the store package.
- **#141** the poller records exactly one pass row per exit, with a skip reason and outcome counts.
- **#142** Series list: eight hygiene filters, Site and Library narrowing, paging — all of it in the query string, so a filtered list is a bookmark.
- **#143** Overview: a three-state verdict line and a stats block where every non-zero figure links to the list that counts it.
- **#144** per-Series detail page, keyed by the `site:series_id` composite the rest of the system already uses.
- **#145** the Lanes page reads the database; the in-memory Lane state, `web.LaneReporter` and `latest.Status` are deleted.
- **#146** Forced Poll: *Check now* stamps `series.force_poll_at` and never commands the poller.
- **#147** pause and resume one Site's Lane, with a mandatory 1h/6h/24h expiry.

## Shape of the design

Two decisions carry the rest. **Commands go through the database, never at the poller**: both *Check now* and a Lane pause write a row the next pass reads, so they survive a restart and the whole surface stays testable with no poller running. And **pending is derived, never stored** — the request stamp being newer than the check stamp — which self-clears on the check stamp with no second write and no sweeper, because the check stamp is written before the fetch.

ADRs: `docs/adr/0012-persisted-lane-state.md`, `docs/adr/0013-commands-through-the-database.md`.

## Verification

`go test ./...` green on the merged base (`264839e`), all packages, Docker-backed. `gofmt -l` and `go vet` clean.

Every ticket was reviewed on both axes (`cr-spec` + `cr-standards`) before merge.

## Known, non-blocking

- **#143** the verdict ignores never-reported Lanes when other Lanes have reported, and the per-Site table lists Sites that have Series rather than the whole registry. The ticket prose asks for eight hygiene figures per Site; the design mock and the landed `.tbl.sites` grid both say six columns, and the mock won.
- **#146** two `SeriesPage` scans per press instead of a keyed read — `ponytail:`-commented in-tree with the upgrade path.
- **#147** a paused Site with no pass row yet renders no row and so no control, since the Lanes page lists Sites that have passed.
- **#141** a sibling browser Lane declining at the top of a pass records as `sidecar-down`. Specified deliberately; the later spec in this series settles it.

Reviewed-on: #148
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
This commit was merged in pull request #148.
This commit is contained in:
Sulthan Zaki
2026-08-22 08:27:19 +07:00
committed by sulthan
parent 0a245a0dde
commit 889f0f3f38
33 changed files with 6112 additions and 634 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/internal/latest/poller_test.go
+735 -87
View File
@@ -427,9 +427,12 @@ func TestRunLogsLaneDefaults(t *testing.T) {
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(previous) })
// The pass log writes through the store, so Run needs a real one — a
// poller without a Store is not a poller.
s, _ := newTestStore(t)
ctx, cancel := context.WithCancel(context.Background())
cancel()
(&Poller{Now: func() time.Time { return time.Now() }}).Run(ctx)
(&Poller{Store: s, Now: func() time.Time { return time.Now() }}).Run(ctx)
got := logs.String()
for _, want := range []string{"6 lanes", "rest=1h0m0s", "gap=10s"} {
@@ -1305,7 +1308,7 @@ func TestRunOnceOrdersBySharednessThenAge(t *testing.T) {
}
// Two refusals in one pass stop the Lane: the remaining Series stay unstamped
// and due, and the Lane backs off for refuseBackoff before trying the Site
// and due, and the Lane backs off for RefuseBackoff before trying the Site
// again. One hostile Site burns only its own Lane's budget (issue #100).
func TestRunOnceSiteRefusalSkipsRestOfLaneAndBacksOff(t *testing.T) {
s, _ := newTestStore(t)
@@ -1388,11 +1391,9 @@ func TestBrowserLaneWakeThresholds(t *testing.T) {
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches with 3 freshly-due series = %d, want 0 (Chrome stays asleep)", got)
}
// The owner's page reads this state off the snapshot, and Due-without-
// Checked has to be distinguishable there from a Lane that has stopped.
if lane := laneByName(t, p, "kagane"); !lane.Asleep || lane.Due != 3 || lane.Checked != 0 {
t.Fatalf("asleep kagane lane = %+v, want Asleep with 3 due and 0 checked", lane)
}
// The skipped pass still records its row, carrying the due count it never
// read; the Lanes page (issue #145) reads that row — see
// TestRunLanePassRecordsEveryExit/"asleep".
// 5 due crosses the count threshold.
for i := 3; i < 5; i++ {
seed(i)
@@ -1401,9 +1402,6 @@ func TestBrowserLaneWakeThresholds(t *testing.T) {
if got := browser.callCount(); got != 5 {
t.Fatalf("browser fetches with 5 due series = %d, want 5", got)
}
if lane := laneByName(t, p, "kagane"); lane.Asleep {
t.Fatalf("woken kagane lane still reports Asleep: %+v", lane)
}
// A single long-neglected series wakes the browser by age alone.
seedForCheck(t, s, "kagane:ancient", "https://kagane.to/series/ancient", 0)
p.runOnce(context.Background())
@@ -1412,19 +1410,6 @@ func TestBrowserLaneWakeThresholds(t *testing.T) {
}
}
// laneByName pulls one Lane out of the poller's snapshot, failing rather than
// returning a zero LaneState a caller would assert against by accident.
func laneByName(t *testing.T, p *Poller, site string) LaneState {
t.Helper()
for _, lane := range p.LaneStatus().Lanes {
if lane.Site == site {
return lane
}
}
t.Fatalf("no %q lane in the snapshot", site)
return LaneState{}
}
// When one browser Lane loses the sidecar, the round's remaining browser
// Lanes are skipped: every fetch would fail anyway, and their Series must not
// burn their stamps on a dead Chrome (issue #100).
@@ -1456,7 +1441,7 @@ func TestRunOnceUnreachableBrowserStopsBrowserLanes(t *testing.T) {
}
}
// The shared flag decays after refuseBackoff: the next round probes
// The shared flag decays after RefuseBackoff: the next round probes
// again. comix's Series is resting (stamped last round), so the probe
// falls to kagane — the only Lane with something due — and its fresh
// loss re-gates the Lanes behind it.
@@ -1602,19 +1587,21 @@ func TestRunOnceClampWarningNamesTheSite(t *testing.T) {
}
}
// The owner's admin page (issue #102) reads Lane state out of the poller.
// Before any pass the snapshot is empty — a restart must render "no data
// yet", not zeroes — and each pass records what it saw: the frozen clock,
// the due count and the pace, with refusal backoff derived at snapshot time.
func TestLaneStatus(t *testing.T) {
// The Lanes page (issue #145) reads the durable pass log, so the poller's
// only duty to it is that every return path writes its row; the snapshot it
// used to mirror into memory is gone. A refusing pass still records why it
// declined, and the latest row per Site is what the page renders — nothing
// else is left to assert against here, because the page's seam moved into the
// web layer (web_test.go, seeded-row tests).
func TestPassLogIsTheLanesPagesOnlyWindow(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
if st := p.LaneStatus(); len(st.Lanes) != 0 {
t.Fatalf("lanes before any pass = %d, want 0 (nothing has run)", len(st.Lanes))
} else if st.BrowserConfigured || st.BrowserReachable {
t.Fatalf("browser before any pass = configured=%v reachable=%v, want false without a browser fetcher", st.BrowserConfigured, st.BrowserReachable)
// No pass has run: the log is empty, which the page renders as "none
// observed" rather than confident zeroes.
if _, ok, err := s.LatestLanePass("asura"); err != nil || ok {
t.Fatalf("LatestLanePass before any pass = ok %v err %v, want no row", ok, err)
}
seedForCheck(t, s, "asura:chronicles", "https://asurascans.com/series/chronicles", 0)
@@ -1627,65 +1614,726 @@ func TestLaneStatus(t *testing.T) {
}
p.runOnce(context.Background())
st := p.LaneStatus()
var asura, kagane LaneState
for _, lane := range st.Lanes {
switch lane.Site {
case "asura":
asura = lane
case "kagane":
kagane = lane
}
// Both lanes wrote their rows: asura the pass it read; kagane the
// mid-loop double-refusal exit, which records an empty skip by design
// with its two refused counts — the durable row is the whole record, and
// the page reads it rather than a snapshot.
asura := latestPassFor(t, s, "asura")
if asura.Due != 1 || asura.Checked != 1 || asura.GapMS == 0 {
t.Fatalf("asura row = %+v, want due 1 checked 1 with the Lane's pace", asura)
}
if st.Lanes[0].Site != "asura" {
t.Fatalf("first lane = %q, want asura (snapshot sorted by Site)", st.Lanes[0].Site)
}
if asura.Site == "" {
t.Fatalf("asura missing from snapshot: %+v", st.Lanes)
}
if !asura.LastRun.Equal(now) {
t.Fatalf("asura LastRun = %s, want the frozen clock %s", asura.LastRun, now)
}
if asura.Due != 1 {
t.Fatalf("asura Due = %d, want 1", asura.Due)
}
if asura.Gap == 0 {
t.Fatal("asura Gap = 0, want the Lane's pace")
}
if asura.Browser || asura.Refusing {
t.Fatalf("asura = %+v, want a TLS Lane that is not refusing", asura)
}
if !kagane.Browser || !kagane.Refusing {
t.Fatalf("kagane = %+v, want a browser Lane in refusal backoff", kagane)
}
if !st.BrowserConfigured || !st.BrowserReachable {
t.Fatalf("browser after round = configured=%v reachable=%v, want true/true (sidecar never lost)", st.BrowserConfigured, st.BrowserReachable)
}
if asura.Checked != 1 {
t.Fatalf("asura Checked = %d, want the one Series it read", asura.Checked)
kagane := latestPassFor(t, s, "kagane")
if kagane.Skip != "" || kagane.Refused != 2 || kagane.Due != 2 || kagane.GapMS == 0 {
t.Fatalf("kagane row = %+v, want an empty-skip double-refusal pass with 2 refused", kagane)
}
// A pass that declines to look (kagane is now in backoff) must not restate
// the figures it never gathered as zeroes: the last real pass's due count
// and pace stand until a pass replaces them.
// With the refusal now durable, the next pass declines ahead of the loop:
// it records the refusing skip and carries the previous pass's figures
// forward rather than restating zeroes it never gathered (the carry logic
// itself is driven in TestRunLanePassCarryForwardOnlyWhenGapZero).
before := kagane
if before.Due == 0 || before.Gap == 0 {
t.Fatalf("kagane after its refusing pass = %+v, want the figures that pass gathered", before)
}
p.Now = func() time.Time { return now.Add(time.Minute) }
p.runOnce(context.Background())
for _, lane := range p.LaneStatus().Lanes {
if lane.Site != "kagane" {
continue
}
if lane.Due != before.Due || lane.Gap != before.Gap {
t.Fatalf("kagane after a skipped pass = due %d gap %s, want the previous pass's %d / %s",
lane.Due, lane.Gap, before.Due, before.Gap)
}
again := latestPassFor(t, s, "kagane")
if again.Skip != SkipRefusing {
t.Fatalf("kagane refusing pass skip = %q, want %q", again.Skip, SkipRefusing)
}
// A lost sidecar reads as unreachable for the same window the Lanes skip.
p.setBrowserDown(now)
if st := p.LaneStatus(); !st.BrowserConfigured || st.BrowserReachable {
t.Fatalf("browser after loss = configured=%v reachable=%v, want true/false", st.BrowserConfigured, st.BrowserReachable)
if again.Due != before.Due || again.GapMS != before.GapMS || again.Checked != before.Checked {
t.Fatalf("kagane after a skipped pass = due %d gap %d, want the previous pass's %d / %d",
again.Due, again.GapMS, before.Due, before.GapMS)
}
}
// latestPassFor reads a Site's newest durable pass row, failing rather than
// returning a zero LanePass a caller would assert against by accident.
func latestPassFor(t *testing.T, s *store.Store, site string) store.LanePass {
t.Helper()
pass, ok, err := s.LatestLanePass(site)
if err != nil || !ok {
t.Fatalf("LatestLanePass(%s): ok=%v err=%v", site, ok, err)
}
return pass
}
// countPassRows counts a Site's durable pass rows, for asserting that a pass
// records exactly one.
func countPassRows(t *testing.T, dbURL, site string) int {
t.Helper()
db, err := sql.Open("pgx", dbURL)
if err != nil {
t.Fatalf("open %s: %v", dbURL, err)
}
defer db.Close()
var n int
if err := db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1`, site).Scan(&n); err != nil {
t.Fatalf("count passes for %s: %v", site, err)
}
return n
}
// One durable pass row per exit, with the skip value naming the exit. The
// mid-loop browser-unreachable return also writes one row — but with an empty
// skip, so the row is stall-shaped (due > 0, checked 0, skip ”), matching the
// deliberate absence of a tenth skip value.
func TestRunLanePassRecordsEveryExit(t *testing.T) {
now := time.UnixMilli(5_000_000)
t.Run("paused", func(t *testing.T) {
s, dbURL := newTestStore(t)
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
t.Fatalf("PauseLane: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
p := newTestPoller(t, s, f, now)
if pace := p.runLanePass(context.Background(), "asura", false); pace != 30*time.Minute {
t.Fatalf("paused pace = %s, want 30m (sleep until the expiry)", pace)
}
if f.callCount() != 0 {
t.Fatalf("fetches while paused = %d, want 0", f.callCount())
}
if got := readLatestCheckedAt(t, s, "asura:x"); got != 0 {
t.Fatalf("stamp while paused = %d, want 0 (Series stay due and unstamped)", got)
}
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipPaused {
t.Fatalf("skip = %q, want %q", pass.Skip, SkipPaused)
}
if got := countPassRows(t, dbURL, "asura"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("refusing", func(t *testing.T) {
s, dbURL := newTestStore(t)
seedForCheck(t, s, "kagane:x", "https://kagane.to/series/x", 0)
if err := s.SetLaneRefusal("kagane", now.Add(10*time.Minute).UnixMilli()); err != nil {
t.Fatalf("SetLaneRefusal: %v", err)
}
browser := &fakeFetcher{status: 200}
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = browser
if pace := p.runLanePass(context.Background(), "kagane", false); pace != 10*time.Minute {
t.Fatalf("refusing pace = %s, want 10m", pace)
}
if browser.callCount() != 0 {
t.Fatalf("fetches while refusing = %d, want 0", browser.callCount())
}
if pass := latestPassFor(t, s, "kagane"); pass.Skip != SkipRefusing {
t.Fatalf("skip = %q, want %q", pass.Skip, SkipRefusing)
}
if got := countPassRows(t, dbURL, "kagane"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("sidecar-down", func(t *testing.T) {
s, dbURL := newTestStore(t)
seedForCheck(t, s, "kagane:x", "https://kagane.to/series/x", 0)
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = browser
p.setBrowserDown(now) // a sibling Lane lost Chrome within the backoff window
if pace := p.runLanePass(context.Background(), "kagane", false); pace != RefuseBackoff {
t.Fatalf("sidecar-down pace = %s, want %s", pace, RefuseBackoff)
}
if browser.callCount() != 0 {
t.Fatalf("fetches with the sidecar down = %d, want 0", browser.callCount())
}
if pass := latestPassFor(t, s, "kagane"); pass.Skip != SkipSidecarDown {
t.Fatalf("skip = %q, want %q", pass.Skip, SkipSidecarDown)
}
if got := countPassRows(t, dbURL, "kagane"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("no-fetcher", func(t *testing.T) {
s, dbURL := newTestStore(t)
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
if pace := p.runLanePass(context.Background(), "comix", false); pace != defaultGap {
t.Fatalf("no-fetcher pace = %s, want %s", pace, defaultGap)
}
pass := latestPassFor(t, s, "comix")
if pass.Skip != SkipNoFetcher || pass.GapMS != defaultGap.Milliseconds() {
t.Fatalf("no-fetcher pass = %+v, want skip %q with its own gap %s", pass, SkipNoFetcher, defaultGap)
}
if got := countPassRows(t, dbURL, "comix"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("due-query", func(t *testing.T) {
s, dbURL := newTestStore(t)
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
// The due query is the pass's first store read after the gates; making
// it fail without touching poll_passes takes its tables away.
db, err := sql.Open("pgx", dbURL)
if err != nil {
t.Fatalf("open %s: %v", dbURL, err)
}
if _, err := db.Exec(`DROP TABLE bookmarks`); err != nil {
t.Fatalf("drop bookmarks: %v", err)
}
db.Close()
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
if pace := p.runLanePass(context.Background(), "asura", false); pace != defaultGap {
t.Fatalf("due-query pace = %s, want %s", pace, defaultGap)
}
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipDueQuery {
t.Fatalf("skip = %q, want %q", pass.Skip, SkipDueQuery)
}
if got := countPassRows(t, dbURL, "asura"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("asleep", func(t *testing.T) {
s, dbURL := newTestStore(t)
for i := 0; i < 3; i++ {
key := fmt.Sprintf("kagane:w%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], now.Add(-62*time.Minute).UnixMilli())
}
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = browser
if pace := p.runLanePass(context.Background(), "kagane", false); pace != defaultGap {
t.Fatalf("asleep pace = %s, want %s", pace, defaultGap)
}
if browser.callCount() != 0 {
t.Fatalf("fetches while Chrome is asleep = %d, want 0", browser.callCount())
}
pass := latestPassFor(t, s, "kagane")
if pass.Skip != SkipAsleep || pass.Due != 3 || pass.GapMS != defaultGap.Milliseconds() {
t.Fatalf("asleep pass = %+v, want skip %q with 3 due and the default gap", pass, SkipAsleep)
}
if got := countPassRows(t, dbURL, "kagane"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("eligible-count", func(t *testing.T) {
s, dbURL := newTestStore(t)
// The eligible query shares the due query's tables, so no real store
// failure reaches it after a successful due read; the seam is how the
// path is driven at all (see Poller.eligibleCount).
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.eligibleCount = func(string) (int, error) { return 0, errors.New("count failed") }
if pace := p.runLanePass(context.Background(), "asura", false); pace != defaultGap {
t.Fatalf("eligible-count pace = %s, want %s", pace, defaultGap)
}
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipEligibleCount {
t.Fatalf("skip = %q, want %q", pass.Skip, SkipEligibleCount)
}
if got := countPassRows(t, dbURL, "asura"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("nothing-eligible", func(t *testing.T) {
s, dbURL := newTestStore(t)
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
if pace := p.runLanePass(context.Background(), "asura", false); pace != sites["asura"].Rest {
t.Fatalf("nothing-eligible pace = %s, want a full rest %s", pace, sites["asura"].Rest)
}
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipNothingEligible {
t.Fatalf("skip = %q, want %q", pass.Skip, SkipNothingEligible)
}
if got := countPassRows(t, dbURL, "asura"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("reached the loop", func(t *testing.T) {
s, dbURL := newTestStore(t)
const url = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
seedForCheck(t, s, "asura:chronicles-of-the-demon-faction-f886a8af", url, 0)
p := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
if pace := p.runLanePass(context.Background(), "asura", false); pace != defaultGap {
t.Fatalf("loop pace = %s, want %s", pace, defaultGap)
}
pass := latestPassFor(t, s, "asura")
if pass.Skip != "" || pass.Due != 1 || pass.Checked != 1 {
t.Fatalf("loop pass = %+v, want an empty skip with 1 due and 1 checked", pass)
}
if got := countPassRows(t, dbURL, "asura"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
t.Run("browser-unreachable mid-loop", func(t *testing.T) {
s, dbURL := newTestStore(t)
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
interrupted := fmt.Errorf("%w: %w", errBrowserInterrupted, errors.New("restart"))
browser := &fakeFetcher{status: 200, err: interrupted}
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = browser
if pace := p.runLanePass(context.Background(), "comix", false); pace != defaultGap {
t.Fatalf("mid-loop pace = %s, want %s", pace, defaultGap)
}
// One row was still written, and it is stall-shaped by construction:
// empty skip with due > 0 and checked 0 because the return precedes
// the checked counter. The assertion below pins that shape precisely.
pass := latestPassFor(t, s, "comix")
if pass.Skip != "" || pass.Unreachable != 1 || pass.Checked != 0 || pass.Due != 1 {
t.Fatalf("mid-loop pass = %+v, want empty skip, unreachable 1, checked 0, due 1", pass)
}
if got := countPassRows(t, dbURL, "comix"); got != 1 {
t.Fatalf("pass rows = %d, want exactly 1", got)
}
})
}
// Carry-forward moves verbatim from the in-memory snapshot (issue #141): a
// pass that never computed its own gap carries the previous pass's due, gap,
// clamped and checked forward; a pass with a gap of its own records its own
// figures, zeroes included, beside the skip reason that explains them.
func TestRunLanePassCarryForwardOnlyWhenGapZero(t *testing.T) {
now := time.UnixMilli(5_000_000)
t.Run("no gap of its own carries the previous pass's figures", func(t *testing.T) {
s, _ := newTestStore(t)
seedForCheck(t, s, "kagane:x", "https://kagane.to/series/x", 0)
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = &fakeFetcher{body: kaganeAPIFixture, status: 200}
p.runLanePass(context.Background(), "kagane", false)
first := latestPassFor(t, s, "kagane")
if first.Due != 1 || first.Checked != 1 || first.GapMS == 0 {
t.Fatalf("first pass = %+v, want a measured pass", first)
}
if err := s.SetLaneRefusal("kagane", now.Add(10*time.Minute).UnixMilli()); err != nil {
t.Fatalf("SetLaneRefusal: %v", err)
}
// The pass row is keyed (site, ran_at), so the second pass needs its
// own timestamp: a minute later is still inside the refusal.
p.Now = func() time.Time { return now.Add(time.Minute) }
p.runLanePass(context.Background(), "kagane", false)
second := latestPassFor(t, s, "kagane")
if second.Skip != SkipRefusing {
t.Fatalf("second pass skip = %q, want %q", second.Skip, SkipRefusing)
}
if second.Due != first.Due || second.Checked != first.Checked ||
second.GapMS != first.GapMS || second.Clamped != first.Clamped {
t.Fatalf("carried pass = %+v, want the first pass's figures %+v", second, first)
}
})
t.Run("a gap of its own records its own figures", func(t *testing.T) {
s, _ := newTestStore(t)
seedForCheck(t, s, "comix:c", "https://comix.to/title/c", 0)
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = &fakeFetcher{body: comixSeriesFixture, status: 200}
p.runLanePass(context.Background(), "comix", false)
first := latestPassFor(t, s, "comix")
if first.Due != 1 || first.GapMS == 0 {
t.Fatalf("first pass = %+v, want a measured pass", first)
}
// The no-fetcher exit sets its own gap, so no carry-forward: the due
// count it never gathered records as zero beside its skip reason. A
// minute later gives the second pass its own (site, ran_at) key.
p.BrowserFetch = nil
p.Now = func() time.Time { return now.Add(time.Minute) }
p.runLanePass(context.Background(), "comix", false)
second := latestPassFor(t, s, "comix")
if second.Skip != SkipNoFetcher {
t.Fatalf("second pass skip = %q, want %q", second.Skip, SkipNoFetcher)
}
if second.GapMS != defaultGap.Milliseconds() {
t.Fatalf("second pass gap = %d, want its own %d (not carried)", second.GapMS, defaultGap.Milliseconds())
}
if second.Due != 0 || second.Checked != 0 {
t.Fatalf("second pass = %+v, want due 0 checked 0 (its own, not the previous pass's)", second)
}
})
}
// The pass row's five outcome counts are the classification the Series read
// already makes — never a second taxonomy (issue #141). Success is derived,
// never stored: checked minus the four named counts, unreachable excluded
// because its exit returns before the checked counter increments.
func TestRunLanePassCountsOutcomes(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
const (
successKey = "asura:chronicles-of-the-demon-faction-f886a8af"
successURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
)
seeds := map[string]string{
"asura:refused": "https://asurascans.com/series/refused",
"asura:no-chapter": "https://asurascans.com/comics/no-chapter",
"asura:unfetchable": "https://evil.example/x",
"asura:transport": "https://asurascans.com/series/transport",
successKey: successURL,
}
for key, url := range seeds {
seedForCheck(t, s, key, url, 0)
}
f := &fakeFetcher{perURL: map[string]fakeResponse{
seeds["asura:refused"]: {status: 403},
seeds["asura:no-chapter"]: {body: "<html></html>", status: 200},
seeds["asura:transport"]: {err: errors.New("dial tcp: refused")},
successURL: {body: asuraSeriesFixture, status: 200},
}}
newTestPoller(t, s, f, now).runLanePass(context.Background(), "asura", false)
pass := latestPassFor(t, s, "asura")
if pass.Skip != "" || pass.Due != 5 || pass.Checked != 5 {
t.Fatalf("pass = %+v, want a full pass over 5 due Series", pass)
}
if pass.Refused != 1 || pass.Unreachable != 0 || pass.NoChapter != 1 ||
pass.Unfetchable != 1 || pass.Errors != 1 {
t.Fatalf("outcome counts = refused %d unreachable %d no_chapter %d unfetchable %d errors %d, want 1 0 1 1 1",
pass.Refused, pass.Unreachable, pass.NoChapter, pass.Unfetchable, pass.Errors)
}
if success := pass.Checked - (pass.Refused + pass.NoChapter + pass.Unfetchable + pass.Errors); success != 1 {
t.Fatalf("derived success = %d, want 1", success)
}
// The one genuine read went through: the success Series carries the
// fixture's newest chapter, and none of the four failures do.
b, ok, err := s.Get(s.OwnerID(), successKey)
if err != nil || !ok {
t.Fatalf("Get: %v ok=%v", err, ok)
}
if b.LatestChapterNum == nil || *b.LatestChapterNum != 181 {
t.Fatalf("success Series latest = %v, want 181 (the fixture's newest)", b.LatestChapterNum)
}
for _, key := range []string{"asura:refused", "asura:no-chapter", "asura:unfetchable", "asura:transport"} {
b, _, err := s.Get(s.OwnerID(), key)
if err != nil {
t.Fatalf("Get %s: %v", key, err)
}
if b.LatestChapterNum != nil {
t.Fatalf("%s latest = %v, want nil (no chapter survived a failed read)", key, *b.LatestChapterNum)
}
}
}
// A refusal is the Site's mood and outlives our process: the durable stamp a
// pass writes is honoured by a freshly constructed poller, which must not
// re-probe the Site inside its backoff (issue #141).
func TestDurableRefusalSurvivesFreshPoller(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
// Four due Series: the Lane refuses twice, stamps those two, and leaves
// the remaining two untried and due — the queue the fresh poller must
// find intact once the durable backoff lifts.
for i := 0; i < 4; i++ {
key := fmt.Sprintf("kagane:s%d", i)
seedForCheck(t, s, key, "https://kagane.to/series/"+key[7:], 0)
}
p := newTestPoller(t, s, &fakeFetcher{status: 200}, now)
p.BrowserFetch = &fakeFetcher{status: 403}
p.runLanePass(context.Background(), "kagane", false)
if got := p.BrowserFetch.(*fakeFetcher).callCount(); got != 2 {
t.Fatalf("fetches on the refusing pass = %d, want 2 (refused twice)", got)
}
// A restart: a brand-new poller, no in-memory refusal, same store. The
// durable stamp gates the pass.
fresh := newTestPoller(t, s, &fakeFetcher{status: 200}, now.Add(14*time.Minute))
fresh.BrowserFetch = &fakeFetcher{status: 403}
fresh.runLanePass(context.Background(), "kagane", false)
if got := fresh.BrowserFetch.(*fakeFetcher).callCount(); got != 0 {
t.Fatalf("fetches by a fresh poller inside the backoff = %d, want 0", got)
}
if pass := latestPassFor(t, s, "kagane"); pass.Skip != SkipRefusing {
t.Fatalf("fresh poller's pass skip = %q, want %q", pass.Skip, SkipRefusing)
}
// Past the backoff the fresh poller probes again — the two Series the
// original pass never reached, still due with their stamps untouched.
fresh.Now = func() time.Time { return now.Add(16 * time.Minute) }
fresh.runLanePass(context.Background(), "kagane", false)
if got := fresh.BrowserFetch.(*fakeFetcher).callCount(); got != 2 {
t.Fatalf("fetches after the backoff = %d, want 2", got)
}
for i := 2; i < 4; i++ {
if got := readLatestCheckedAt(t, s, fmt.Sprintf("kagane:s%d", i)); got == 0 {
t.Fatalf("kagane:s%d still untried after the backoff", i)
}
}
}
// The pause is read ahead of the refusal check: a Lane that is both paused
// and inside a refusal backoff records the paused skip value, not the
// refusing one. The pause is the owner's order and outranks the Site's mood
// (issue #147).
func TestPauseGatePrecedesRefusalGate(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
if err := s.SetLaneRefusal("asura", now.Add(10*time.Minute).UnixMilli()); err != nil {
t.Fatalf("SetLaneRefusal: %v", err)
}
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
t.Fatalf("PauseLane: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
p := newTestPoller(t, s, f, now)
if pace := p.runLanePass(context.Background(), "asura", false); pace != 30*time.Minute {
t.Fatalf("paused-while-refusing pace = %s, want 30m (the pause's expiry)", pace)
}
if f.callCount() != 0 {
t.Fatalf("fetches while paused and refusing = %d, want 0", f.callCount())
}
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipPaused {
t.Fatalf("skip = %q, want %q (the pause outranks the refusal)", pass.Skip, SkipPaused)
}
}
// A pause is a fact about the Site, not about the process: a freshly
// constructed poller against a store holding a pause row stays paused until
// the expiry, then runs the Lane normally. The restart criterion is the
// whole point of writing a row instead of commanding a poller (issue #147).
func TestDurablePauseSurvivesFreshPoller(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
t.Fatalf("PauseLane: %v", err)
}
// A restart: a brand-new poller, no in-memory state, same store.
fresh := newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now)
if pace := fresh.runLanePass(context.Background(), "asura", false); pace != 30*time.Minute {
t.Fatalf("fresh poller's paused pace = %s, want 30m", pace)
}
if pass := latestPassFor(t, s, "asura"); pass.Skip != SkipPaused {
t.Fatalf("fresh poller's pass skip = %q, want %q", pass.Skip, SkipPaused)
}
// Past the expiry the same fresh poller runs the Lane normally.
fresh.Now = func() time.Time { return now.Add(31 * time.Minute) }
fresh.runLanePass(context.Background(), "asura", false)
if pass := latestPassFor(t, s, "asura"); pass.Skip != "" {
t.Fatalf("pass after the expiry skip = %q, want the loop reached", pass.Skip)
}
if got := readLatestCheckedAt(t, s, "asura:x"); got == 0 {
t.Fatal("the Series was not checked after the pause lifted")
}
}
// ResumeLane zeroes the pause and the Lane's next pass finds its full queue
// waiting: a pause delays work rather than discarding it, so the due Series
// sit unstamped while paused and are all fetched once the pause lifts
// (issue #147).
func TestResumeLaneRestoresTheQueue(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
for i := 0; i < 3; i++ {
seedForCheck(t, s, fmt.Sprintf("asura:s%d", i), "https://asurascans.com/series/x", 0)
}
if err := s.PauseLane("asura", now.Add(30*time.Minute).UnixMilli()); err != nil {
t.Fatalf("PauseLane: %v", err)
}
f := &fakeFetcher{body: asuraSeriesFixture, status: 200}
p := newTestPoller(t, s, f, now)
p.runLanePass(context.Background(), "asura", false)
if f.callCount() != 0 {
t.Fatalf("fetches while paused = %d, want 0", f.callCount())
}
for i := 0; i < 3; i++ {
if got := readLatestCheckedAt(t, s, fmt.Sprintf("asura:s%d", i)); got != 0 {
t.Fatalf("asura:s%d stamp while paused = %d, want 0 (due and unstamped)", i, got)
}
}
if err := s.ResumeLane("asura"); err != nil {
t.Fatalf("ResumeLane: %v", err)
}
before := f.callCount()
p.runLanePass(context.Background(), "asura", false)
if got := f.callCount() - before; got != 3 {
t.Fatalf("fetches after resume = %d, want 3 (the full due queue)", got)
}
for i := 0; i < 3; i++ {
if got := readLatestCheckedAt(t, s, fmt.Sprintf("asura:s%d", i)); got == 0 {
t.Fatalf("asura:s%d still untried after resume", i)
}
}
}
// RecordLanePass prunes in the same call that inserts, so the retention
// cutoff the recorder passes is observable in what survives: a row just inside
// 14 days behind the poller's clock is kept, one just outside is pruned
// (issue #139, #141).
func TestPassRetentionCutoffIsFourteenDays(t *testing.T) {
s, dbURL := newTestStore(t)
// A real-world clock: the seeded rows sit 14 days back, so they must be
// positive timestamps or the seed's own prune (ran_at < 0) removes them.
now := time.UnixMilli(1_800_000_000_000)
kept := now.Add(-14*24*time.Hour + time.Minute).UnixMilli()
pruned := now.Add(-14*24*time.Hour - time.Minute).UnixMilli()
for _, ranAt := range []int64{kept, pruned} {
if err := s.RecordLanePass(store.LanePass{Site: "asura", RanAt: ranAt}, 0); err != nil {
t.Fatalf("seed pass at %d: %v", ranAt, err)
}
}
seedForCheck(t, s, "asura:x", "https://asurascans.com/series/x", 0)
newTestPoller(t, s, &fakeFetcher{body: asuraSeriesFixture, status: 200}, now).
runLanePass(context.Background(), "asura", false)
db, err := sql.Open("pgx", dbURL)
if err != nil {
t.Fatalf("open %s: %v", dbURL, err)
}
defer db.Close()
var n int
if err := db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1 AND ran_at = $2`, "asura", pruned).Scan(&n); err != nil {
t.Fatalf("count pruned row: %v", err)
}
if n != 0 {
t.Fatalf("row at %d survived, want it pruned (older than 14 days)", pruned)
}
if err := db.QueryRow(`SELECT count(*) FROM poll_passes WHERE site = $1`, "asura").Scan(&n); err != nil {
t.Fatalf("count passes: %v", err)
}
if n != 2 {
t.Fatalf("passes = %d, want 2 (this pass plus the kept row)", n)
}
}
// stampRecordingFetcher is a fakeFetcher that records the Series' check stamp
// at call time — the seam that proves the check stamp is written before the
// fetch (issue #146). If the order were swapped, the recorded stamp would be
// the pre-pass value and the ordering assertion would fail.
type stampRecordingFetcher struct {
fakeFetcher
store *store.Store
site string
seriesID string
stampAtCall int64
}
func (f *stampRecordingFetcher) Get(ctx context.Context, url string) (string, int, error) {
ts, err := f.store.LatestCheckedAt(f.site, f.seriesID)
if err != nil {
return "", 0, err
}
f.stampAtCall = ts
return f.fakeFetcher.Get(ctx, url)
}
// The check stamp is written before the fetch is attempted: the fake fetcher
// records the stamp it sees at call time, and it must already be the pass's
// own stamp. The order is load-bearing — a forced request is pending while
// force_poll_at > latest_checked_at, so stamping after the fetch would make a
// failed forced request sticky — and the assertion fails if it is swapped.
func TestCheckStampIsWrittenBeforeFetch(t *testing.T) {
s, _ := newTestStore(t)
const (
key = "asura:chronicles-of-the-demon-faction-f886a8af"
seriesURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
)
seedForCheck(t, s, key, seriesURL, 0)
now := time.UnixMilli(7_000_000)
rec := &stampRecordingFetcher{
fakeFetcher: fakeFetcher{body: asuraSeriesFixture, status: 200},
store: s, site: "asura", seriesID: "chronicles-of-the-demon-faction-f886a8af",
}
newTestPoller(t, s, rec, now).runOnce(context.Background())
if rec.stampAtCall != now.UnixMilli() {
t.Fatalf("check stamp at fetch time = %d, want %d (the stamp must be written before the fetch)", rec.stampAtCall, now.UnixMilli())
}
}
// A forced Series whose attempt fails is no longer pending: the check stamp
// is written before the fetch, so the first attempt ends the pending state
// whatever it returns. A naive implementation (stamp only on success, or
// after the fetch) leaves the request sticky and the row due next pass.
func TestForcedSeriesSelfClearsOnFailedAttempt(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(7_000_000)
const url = "https://asurascans.com/comics/x"
// Freshly checked, so only the force flag makes it due.
seedForCheck(t, s, "asura:x", url, now.Add(-30*time.Minute).UnixMilli())
if err := s.ForceSeriesPoll("asura", "x", now.UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
due, err := s.DueForLatestCheck("asura", now.Add(-time.Hour).UnixMilli(), -1)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 1 || due[0].Key() != "asura:x" || !due[0].Forced {
t.Fatalf("forced row not due and flagged before the attempt: %+v", due)
}
// The attempt fails, but the attempt still happened: the row is stamped
// and no longer pending.
f := &fakeFetcher{err: errors.New("dial tcp: refused")}
newTestPoller(t, s, f, now).runOnce(context.Background())
if got := readLatestCheckedAt(t, s, "asura:x"); got != now.UnixMilli() {
t.Fatalf("latest_checked_at = %d, want %d", got, now.UnixMilli())
}
due, err = s.DueForLatestCheck("asura", now.Add(-time.Hour).UnixMilli(), -1)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
if len(due) != 0 {
t.Fatalf("failed attempt left the forced row due: %v", due)
}
}
// The refusal backoff is a Lane gate, not a Series gate: a forced Series does
// not override a Site that is actively refusing, because hand-forcing a
// request into a refusal only makes it worse (issue #146).
func TestForcedSeriesDoesNotOverrideRefusalBackoff(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
seedForCheck(t, s, "kagane:w", "https://kagane.to/series/w", 0)
if err := s.ForceSeriesPoll("kagane", "w", now.UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
if err := s.SetLaneRefusal("kagane", now.Add(RefuseBackoff).UnixMilli()); err != nil {
t.Fatalf("SetLaneRefusal: %v", err)
}
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches through a refusal backoff = %d, want 0", got)
}
}
// A forced Series wakes a sleeping browser Lane: the wake thresholds exist to
// stop the machine waking itself for one unattended check, and a human asking
// is not that (issue #146). Below both thresholds the Lane still sleeps when
// nothing is forced.
func TestForcedSeriesWakesSleepingBrowser(t *testing.T) {
s, _ := newTestStore(t)
now := time.UnixMilli(5_000_000)
// Freshly due: checked two minutes before the rest elapses, so the wait
// is far below browserWakeAge and the count is under browserWakeCount.
seedForCheck(t, s, "kagane:w1", "https://kagane.to/series/w1", now.Add(-62*time.Minute).UnixMilli())
seedForCheck(t, s, "kagane:w2", "https://kagane.to/series/w2", now.Add(-62*time.Minute).UnixMilli())
browser := &fakeFetcher{body: kaganeAPIFixture, status: 200}
p := &Poller{
Store: s, Fetch: &fakeFetcher{status: 200}, BrowserFetch: browser,
Now: func() time.Time { return now },
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 0 {
t.Fatalf("browser fetches without a forced series = %d, want 0 (Chrome stays asleep)", got)
}
if err := s.ForceSeriesPoll("kagane", "w1", now.UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
p.runOnce(context.Background())
if got := browser.callCount(); got != 2 {
t.Fatalf("browser fetches with a forced series = %d, want 2 (the lane wakes)", got)
}
}