Spec #137: per-Series poll failure state, completion hint, and outbound owner notification #174

Merged
sulthan merged 30 commits from spec-137 into main 2026-08-23 11:31:11 +07:00
31 changed files with 3745 additions and 176 deletions
+7 -1
View File
@@ -100,7 +100,13 @@ DISCORD_REDIRECT_URI=
# every kagane poll. Left unset here on purpose — a wrong default would poll a
# stranger's address, and "no browser" is a safe, self-announcing state.
# BROWSER_WS_URL=ws://100.x.y.z:9222
#
# Discord webhook for owner notices (outbound alerting when a poll Lane
# stalls). Unset means the whole path is off — a local stack needs no webhook,
# exactly as the browser URL behaves. The address is a secret in the class of
# TOKEN_KEY: never commit it, never paste it anywhere public.
# DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
#
# Zone the backend stamps its log lines in. Cosmetic only. Nothing else in
# the service has a zone: bookmark timestamps are unix ms, and the two real
# time columns are timestamptz. Defaults to Asia/Jakarta; set to UTC for the
+9
View File
@@ -159,6 +159,15 @@ failures. The flag decays and they probe again.
- Browser Lanes wake Chrome only when 5+ Series are due or one has waited 15m,
and cover work runs in the background so a slow CDN can't eat a Lane's gap.
### Owner notices — `internal/notify`, `latest.Fault`, `latest.Notifier`, `latest.FaultsFrom`
The poller's outbound owner-notice path (issue #171): one condition today
(the stall), judged from the durable pass log alone so the poller and any
future reader of the same judgement cannot disagree. The webhook address is a
secret in the class of TOKEN_KEY — never logged, never rendered, never
carried in an error. The `owner_notices` suppression table (one row per
condition + site) is the only state; every threshold is derived, not stored.
### Covers — `Store.OnSeriesCreated`, `latest.Acquirer`, `latest.CoverBytesFetcher`, `Store.SetSeriesCover`
Acquired once when the first Bookmark of a Series is created, then served from
+210
View File
@@ -0,0 +1,210 @@
package latest
import (
"context"
"fmt"
"time"
"bookmarkmanager/backend/internal/store"
)
// ConditionStall is the owner-notice machine word for a Lane that owed Polls,
// made none, and has nothing to say for it. The word is the message's footer
// and its suppression key; it is wire-stable. #172 declares the other three
// words (no-browser-route, sidecar-down, adapter-broken); this ticket
// declares only the stall.
const ConditionStall = "stall"
// ConditionNoBrowserRoute is the owner-notice machine word for a Site whose
// challenge refuses for longer than the owner window with no browser route
// to clear it — the 403-with-interstitial the reader maps to
// errChallengeHeld (read.go), which plain TLS cannot clear. The word is the
// message's footer and its suppression key; it is wire-stable.
const ConditionNoBrowserRoute = "no-browser-route"
// ConditionSidecarDown is the owner-notice machine word for a browser
// sidecar no Lane has reached for longer than the owner window: every
// browser-backed Lane's latest pass is a sidecar skip. The word is the
// message's footer and its suppression key; it is wire-stable, and its
// suppression row holds the empty Site (AC4).
const ConditionSidecarDown = "sidecar-down"
// ConditionAdapterBroken is the owner-notice machine word for a Site whose
// adapter stopped finding chapters: more than half of its Series hold an
// old no-chapter failure row. The word is the message's footer and its
// suppression key; it is wire-stable.
const ConditionAdapterBroken = "adapter-broken"
// OwnerWindow is the class-level staleness boundary every owner-notice
// condition measures against — the same twelve hours the Lanes page's
// "not checked in 12h" filter uses (internal/web/admin.go). Declared here
// once so #172's three conditions and the admin filters share one figure.
const OwnerWindow = 12 * time.Hour
// Fault is one condition the owner is told about, judged from durable rows
// alone. Site is "" for a fault that is not one Site's.
type Fault struct {
Condition string // one of the Condition* words
Site string
Since int64 // unix ms the episode began; the message's age
}
// FaultInput is everything the judgement reads. A struct so #172's three
// conditions can add inputs without changing either caller.
type FaultInput struct {
Passes []store.LanePass
// RefusingSince is, per Site, the unix ms when that Site's current
// unbroken run of refusing passes began, or absent when its latest pass
// did not refuse. Its zero value is an empty map, which contributes no
// fault.
RefusingSince map[string]int64
// SidecarOK is, per browser-backed Site, the unix ms of that Site's most
// recent pass that actually reached the sidecar. Zero when the pass log
// holds none — an asleep Lane never reached it and never counts as
// evidence either way. Its zero value is an empty map.
SidecarOK map[string]int64
// NoChapterShare is, per Site, the share of that Site's Series holding a
// no-chapter failure row older than the owner window. Its zero value is
// an empty map, which contributes no fault.
NoChapterShare map[string]float64
}
// FaultsFrom judges the owner-notice conditions from durable rows alone, so
// the poller and the landing page cannot disagree about what a fault is.
//
// A Lane stalls when its latest pass shows due > 0, none checked, no skip
// value and no refusal — exactly the row the mid-loop browser loss writes
// (see the comment at the outcomeUnreachable return in runLanePass), so a
// Lane that owed Polls, made none, and has nothing to say for it is a fault.
// The three #172 conditions share the same OwnerWindow boundary and the same
// fail-open shape: an input's absence contributes no fault, never a false
// one.
//
// - no-browser-route: a Site whose refusing run began before the window
// and that has no browser route to clear the challenge (AC2). Both
// refusal shapes count — the gate's skip='refusing' rows and the loop's
// refused>0 rows (the twice-refused break writes skip="") — so the run
// stays unbroken across them, and one healthy pass ends it.
// - sidecar-down: every browser-backed Site's latest pass is a sidecar
// skip — SkipSidecarDown or SkipNoFetcher, the two early returns that
// record a skip value — and each Site's most recent sidecar-reaching
// pass is older than the window (AC4). The skip clause is what keeps
// SkipAsleep out: a Lane under both wake thresholds is the commonest
// healthy state, never reached the sidecar, and would otherwise age into
// a false alarm. Emitted once, with Site "" (AC4's one row per episode).
// - adapter-broken: more than half of one Site's Series hold a no-chapter
// failure row older than the window (AC6). Strictly above half: the
// filter is the tool, and one Site change makes hundreds of rows, so a
// single failing Series never fires (AC7). The episode's age is the
// window — the old rows prove the episode is at least that old.
func FaultsFrom(in FaultInput, now time.Time) []Fault {
var faults []Fault
for _, p := range in.Passes {
if p.Due > 0 && p.Checked == 0 && p.Skip == "" && p.Refused == 0 {
faults = append(faults, Fault{Condition: ConditionStall, Site: p.Site, Since: p.RanAt})
}
}
cutoff := now.Add(-OwnerWindow).UnixMilli()
for site, since := range in.RefusingSince {
if since < cutoff && !isBrowserSite(site) {
faults = append(faults, Fault{Condition: ConditionNoBrowserRoute, Site: site, Since: since})
}
}
if since, down := sidecarDownSince(in.Passes, in.SidecarOK, cutoff); down {
faults = append(faults, Fault{Condition: ConditionSidecarDown, Site: "", Since: since})
}
for site, share := range in.NoChapterShare {
if share > 0.5 {
faults = append(faults, Fault{Condition: ConditionAdapterBroken, Site: site, Since: now.Add(-OwnerWindow).UnixMilli()})
}
}
return faults
}
// sidecarDownSince reports whether no browser Lane has reached the sidecar
// for longer than the owner window and, when it has, the last moment any
// Lane reached it. The skip clause — every browser-backed Site's latest pass
// must be SkipSidecarDown or SkipNoFetcher — keeps SkipAsleep out (see
// FaultsFrom). A Site with no pass row at all is not judged down either: a
// fresh database is not a dead sidecar.
func sidecarDownSince(passes []store.LanePass, ok map[string]int64, cutoff int64) (since int64, down bool) {
latest := make(map[string]store.LanePass, len(passes))
for _, p := range passes {
latest[p.Site] = p
}
for _, site := range browserBackedSites() {
p, found := latest[site]
if !found || (p.Skip != SkipSidecarDown && p.Skip != SkipNoFetcher) {
return 0, false
}
reached := ok[site]
if reached > 0 && reached >= cutoff {
return 0, false
}
if reached > since {
since = reached
}
}
// No Lane's retained pass log shows a sidecar reach: the honest age is
// the window itself — "at least twelve hours" — not the epoch, which
// humanAge would render as tens of thousands of days.
if since == 0 {
since = cutoff
}
return since, true
}
// Notifier delivers one owner notice. The poller neither retries nor queues:
// an error is logged and the suppression row left unwritten, so the next pass
// tries again while the condition holds.
type Notifier interface {
Notify(ctx context.Context, f Fault, sentence, href string) error
}
// ownerNoticeConditions is every condition this package judges, for the
// clear loop in recordPass: a condition absent from a pass's fault list
// forgets its episode, so the next occurrence sends again. #172 extends the
// list when it adds its conditions.
var ownerNoticeConditions = []string{ConditionStall, ConditionNoBrowserRoute, ConditionSidecarDown, ConditionAdapterBroken}
// noticeFor renders one fault's message: the description sentence — condition,
// age, repair — and the deep link the embed's title points at. Each condition
// provides its own wording; the stall is the only one today (issue #171).
func noticeFor(f Fault, row store.LanePass, now time.Time) (sentence, href string) {
switch f.Condition {
case ConditionStall:
return fmt.Sprintf(
"%s owed %d Polls and made none — %s; check the Lane's browser sidecar and the Site's challenge state",
f.Site, row.Due, humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
case ConditionNoBrowserRoute:
return fmt.Sprintf(
"%s has refused for %s with no browser route — the challenge does not clear on plain TLS; redeploy or add a browser route",
f.Site, humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
case ConditionSidecarDown:
return fmt.Sprintf(
"no browser Lane has reached the sidecar for %s — the browser sidecar is down; start or repair the browser machine",
humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
case ConditionAdapterBroken:
return fmt.Sprintf(
"more than half of %s's Series have failed no-chapter reads for at least %s — the Site's layout changed and the adapter is broken",
f.Site, humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
}
return "", ""
}
// humanAge renders a duration the way an owner reads it in a message: minutes
// under an hour, then hours, then days; a stall that was just born reads
// "just now".
func humanAge(d time.Duration) string {
switch {
case d < time.Minute:
return "just now"
case d < time.Hour:
return fmt.Sprintf("%dm", int(d.Minutes()))
case d < 24*time.Hour:
return fmt.Sprintf("%dh", int(d.Hours()))
default:
return fmt.Sprintf("%dd", int(d.Hours()/24))
}
}
+189 -12
View File
@@ -46,7 +46,10 @@ type Poller struct {
// CoverBytesFetch is optional; it handles plain-TLS sources through the
// same failure-isolated prefetch path.
CoverBytesFetch CoverBytesFetcher
Now func() time.Time // injected so tests can freeze it
// Notify delivers owner notices. Nil disables the whole path (issue #171):
// the poller is not the place a missing webhook becomes an error.
Notify Notifier
Now func() time.Time // injected so tests can freeze it
// eligibleCount reports how many of a Site's Series are eligible for
// polling, defaulting to Store.EligibleSeriesCount. Injected so tests can
// fail the count alone: the eligible query shares the due query's tables,
@@ -273,7 +276,9 @@ const (
// (issue #141). The classification the read already makes is counted, never a
// second taxonomy: refused is the Site holding a challenge, unreachable the
// browser interrupting, noChapter a 200 with real HTML but no chapter links,
// unfetchable the host pin or a missing fetcher, and errors everything else.
// unfetchable the host pin or a missing fetcher, notFound a 4xx other than
// the 403 refusal — the Site answered with a client status — and errors
// everything else.
type readOutcome int
const (
@@ -283,14 +288,33 @@ const (
outcomeNoChapter
outcomeUnfetchable
outcomeError
outcomeNotFound
)
// word returns the wire spelling this outcome stores in poll_failures — the
// same strings the pass log's columns use (C1, issue #164). Success, refusal
// and browser loss return "" so recordFailure's "no statement" case is one
// return: a challenge or a lost sidecar is no evidence about any particular
// Series (ADR-0016).
func (o readOutcome) word() string {
switch o {
case outcomeNotFound:
return "not_found"
case outcomeNoChapter:
return "no_chapter"
case outcomeUnfetchable:
return "unfetchable"
case outcomeError:
return "errors"
}
return ""
}
// outcomeCounts are the five named outcome counts of one pass. A success
// count is derived, never stored: checked minus the four, with unreachable
// excluded because the sidecar-loss path returns before the checked counter
// increments (issue #141).
// outcomeCounts are the six named outcome counts of one pass. A success
// count is derived, never stored: checked minus the five named failures,
// with unreachable excluded because the sidecar-loss path returns before the
// checked counter increments (issue #141).
type outcomeCounts struct {
refused, unreachable, noChapter, unfetchable, errors int
refused, unreachable, noChapter, unfetchable, errors, notFound int
}
func (c *outcomeCounts) add(o readOutcome) {
@@ -303,6 +327,8 @@ func (c *outcomeCounts) add(o readOutcome) {
c.noChapter++
case outcomeUnfetchable:
c.unfetchable++
case outcomeNotFound:
c.notFound++
case outcomeError:
c.errors++
}
@@ -334,7 +360,7 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
// carries the previous pass's forward inside recordPass.
fig := passFigures{}
rec := passRecord{site: name, ranAt: now.UnixMilli()}
defer func() { p.recordPass(rec, fig) }()
defer func() { p.recordPass(ctx, rec, fig) }()
// One Lane row read at the top of a pass, serving two gates (issue #139).
// Both stamps outlive our process, so the gates read the durable row
@@ -451,6 +477,7 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
}
}
outcome := p.checkOne(ctx, sr)
p.recordFailure(sr, outcome)
if outcome == outcomeUnreachable {
// The mid-loop browser loss writes an empty skip on purpose: the
// pass returns before the checked counter increments, so its row
@@ -503,8 +530,9 @@ type passFigures struct {
// pass's due, gap, clamped and checked forward rather than stating zeroes it
// did not measure; the skip column says why it declined, so the zeroes that
// remain (due-query, no-fetcher) read as explanations rather than
// measurements.
func (p *Poller) recordPass(rec passRecord, fig passFigures) {
// measurements. Once the row is durable, the owner-notice judgement runs
// beside it (issue #171).
func (p *Poller) recordPass(ctx context.Context, rec passRecord, fig passFigures) {
row := store.LanePass{
Site: rec.site,
RanAt: rec.ranAt,
@@ -517,6 +545,7 @@ func (p *Poller) recordPass(rec passRecord, fig passFigures) {
Unreachable: rec.counts.unreachable,
NoChapter: rec.counts.noChapter,
Unfetchable: rec.counts.unfetchable,
NotFound: rec.counts.notFound,
Errors: rec.counts.errors,
}
if row.GapMS == 0 {
@@ -532,7 +561,111 @@ func (p *Poller) recordPass(rec passRecord, fig passFigures) {
}
if err := p.Store.RecordLanePass(row, rec.ranAt-lanePassRetention.Milliseconds()); err != nil {
log.Printf("latest poll %s: record lane pass: %v", rec.site, err)
return
}
p.ownerNotices(ctx, row)
}
// ownerNotices judges the owner-notice conditions for the pass just recorded
// and fires (issue #171). It sits in recordPass because that deferred call is
// the one place every return path passes through: three of the four
// conditions occur on early returns and the success path can never see them.
// The judgement reads the whole pass log plus three derived reads — the
// other Lanes' latest passes, each Site's refusing-run start, its last
// sidecar-reaching pass, and its no-chapter share — so one pass judges every
// condition (issue #172). Per fault: NoticeSent → send → MarkNoticeSent, so
// a fault lasting a month sends one message, not one per pass; a condition
// absent from this pass's fault list forgets its episode, so the next
// occurrence sends again. Everything here is best-effort: a failed send, a
// failed store read and a failed notice write are all logged and never
// change the pass's outcome counts or its return value. The clear runs even
// when Notify is nil, so a deployment that turns the webhook off does not
// leave stale rows that suppress the first real notice after it is turned
// back on.
func (p *Poller) ownerNotices(ctx context.Context, row store.LanePass) {
now := p.Now()
in := FaultInput{Passes: []store.LanePass{row}}
// Each read fails independently: a failure logs and contributes no fault,
// never a false one.
if passes, err := p.Store.LatestLanePasses(); err != nil {
log.Printf("latest poll %s: latest lane passes: %v", row.Site, err)
} else {
in.Passes = passes
}
if since, err := p.Store.RefusingSince(now.UnixMilli()); err != nil {
log.Printf("latest poll %s: refusing since: %v", row.Site, err)
} else {
in.RefusingSince = since
}
if ok, err := p.Store.SidecarOK(browserBackedSites()); err != nil {
log.Printf("latest poll %s: sidecar ok: %v", row.Site, err)
} else {
in.SidecarOK = ok
}
if share, err := p.Store.NoChapterShare(now.Add(-OwnerWindow).UnixMilli()); err != nil {
log.Printf("latest poll %s: no-chapter share: %v", row.Site, err)
} else {
in.NoChapterShare = share
}
faults := FaultsFrom(in, now)
bySite := make(map[string]store.LanePass, len(in.Passes))
for _, pass := range in.Passes {
bySite[pass.Site] = pass
}
for _, f := range faults {
if p.Notify == nil {
continue
}
sent, err := p.Store.NoticeSent(f.Condition, f.Site)
if err != nil {
log.Printf("latest poll %s: notice sent: %v", row.Site, err)
continue
}
if sent {
continue
}
// The fault's own Site's pass renders its sentence — a stall judged
// from another Lane's pass must not quote this pass's figures.
pass, ok := bySite[f.Site]
if !ok {
pass = row
}
sentence, href := noticeFor(f, pass, now)
if err := p.Notify.Notify(ctx, f, sentence, href); err != nil {
// The stamp stays unset: no queue, no backoff — the condition is
// durable, so the next pass tries again while it holds.
log.Printf("latest poll %s: owner notice %s: %v", row.Site, f.Condition, err)
continue
}
if err := p.Store.MarkNoticeSent(f.Condition, f.Site, now.UnixMilli()); err != nil {
log.Printf("latest poll %s: mark notice sent: %v", row.Site, err)
}
}
for _, cond := range ownerNoticeConditions {
if !hasFault(faults, cond, row.Site) {
if err := p.Store.ClearNotice(cond, row.Site); err != nil {
log.Printf("latest poll %s: clear owner notice: %v", row.Site, err)
}
}
}
// sidecar-down suppresses under the empty Site — one row across all
// browser Lanes (AC4) — so clear that row when it is absent from this
// pass's fault list, and a lifted sidecar fires again when it returns.
if !hasFault(faults, ConditionSidecarDown, "") {
if err := p.Store.ClearNotice(ConditionSidecarDown, ""); err != nil {
log.Printf("latest poll %s: clear owner notice: %v", row.Site, err)
}
}
}
// hasFault reports whether faults hold the given condition for the site.
func hasFault(faults []Fault, condition, site string) bool {
for _, f := range faults {
if f.Condition == condition && f.Site == site {
return true
}
}
return false
}
// countEligible routes the eligible count through the test seam when one is
@@ -607,13 +740,35 @@ func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.D
}
return oldest
}
// recordFailure keeps one Series' failure row in step with its read
// (ADR-0016): a failure word is upserted, a successful read deletes the row,
// and refused or unreachable issue no statement at all. Called for every
// outcome from the pass loop, so the four failure words and the success path
// share one write point, and a forced Poll that reads the page clears through
// the ordinary success path — no branch of its own. Log a store failure and
// carry on: this is best-effort, and no single bad Series may stall a Lane.
func (p *Poller) recordFailure(sr store.Series, outcome readOutcome) {
if outcome == outcomeSuccess {
if err := p.Store.ClearSeriesFailure(sr.Site, sr.SeriesID); err != nil {
log.Printf("latest poll %q: clear failure: %v", sr.Key(), err)
}
return
}
word := outcome.word()
if word == "" {
return
}
if err := p.Store.RecordSeriesFailure(sr.Site, sr.SeriesID, word, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: record failure: %v", sr.Key(), err)
}
}
// checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a
// Lane or take down the process. The returned outcome classifies the read for
// the pass row (issue #141), so the Lane can count a refusal, a lost browser,
// a chapter-less page, an unfetchable address or a transport error without
// re-deriving the taxonomy.
// a chapter-less page, an unfetchable address, a missing page or a transport
// error without re-deriving the taxonomy.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOutcome) {
defer func() {
if r := recover(); r != nil {
@@ -657,6 +812,9 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOut
if errors.Is(err, errBrowserInterrupted) {
return outcomeUnreachable
}
if errors.Is(err, errNotFound) {
return outcomeNotFound
}
return outcomeError
}
// A legacy cover source is healed independently of the page read.
@@ -669,6 +827,25 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOut
} else {
p.fillBlankCover(ctx, sr, facts.Cover)
}
// Learned from the successful read: the write sits after the error switch
// (a refused, unreachable or errored read reaches nothing) and before the
// returns below — a completed page whose chapter number did not change
// still has to write. The transition is zero-versus-nonzero, not the
// stamp's value: a Series still completed keeps its original stamp, so the
// age #170 prints is "since the Site first said so"; one that stopped
// being completed is zeroed.
stamp := int64(0)
if facts.SiteCompleted {
stamp = p.Now().UnixMilli()
}
if (sr.SiteCompletedAt == 0) != (stamp == 0) {
if err := p.Store.SetSiteCompletedAt(sr.Site, sr.SeriesID, stamp); err != nil {
// Best-effort, like every poller write: never change the outcome
// word the pass counts.
log.Printf("latest poll %q: set site completed: %v", sr.Key(), err)
}
}
if !facts.HasLatest {
// Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a rest instead of hot-looping.
File diff suppressed because it is too large Load Diff
+24 -7
View File
@@ -6,15 +6,19 @@ import (
"fmt"
)
// seriesRead carries the two facts the poll and the acquirer both extract
// from a series page. Persistence, stamps and scheduling stay with the
// callers, so the policies that keep the two flows distinct (stamp order,
// rests) are not swallowed by the module.
// seriesRead carries the facts the poll and the acquirer both extract from a
// series page. Persistence, stamps and scheduling stay with the callers, so
// the policies that keep the two flows distinct (stamp order, rests) are not
// swallowed by the module.
type seriesRead struct {
Latest latestChapter
HasLatest bool
Cover string
HasCover bool
// SiteCompleted is whether the Site's own completed value was on the page.
// A challenge body and a redesign both read false — an absent hint, never
// a claim (issue #168).
SiteCompleted bool
// BodyLen is the fetched body's length, surfaced because the no-chapter
// log uses it to tell a markup change from a body the size cap cut short.
BodyLen int
@@ -25,15 +29,18 @@ type seriesRead struct {
// errChallengeHeld (browser.go) is the outcome of a Site that answered with
// its interstitial — status 403 (cf-mitigated) or a challenge page body — and
// is how a Lane tells a refusal from an ordinary failure (issue #100).
// errNotFound marks any other 4xx: the page is gone — a fact the owner can act
// on — as distinct from a Site or database that is merely unwell (issue #164).
var (
errNotFetchable = errors.New("series url not fetchable")
errNoFetcher = errors.New("no fetcher for site")
errNotFound = errors.New("series page not found")
)
// readSeriesPage performs the series-page read the poll and the acquirer have
// in common: gate the address, choose the route, fetch the page, extract the
// Latest Chapter and the Cover address. It persists nothing and stamps
// nothing.
// Latest Chapter, the Cover address and the Site's completed value. It
// persists nothing and stamps nothing.
//
// series_url arrives in a client-supplied PUT body (PUT /bookmarks/{key}
// accepts any string), so the gate is not an optimisation against burning a
@@ -59,6 +66,9 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status)
}
if status != 200 {
if status >= 400 && status < 500 {
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errNotFound, seriesURL, status)
}
return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status)
}
if isInterstitial(body) {
@@ -69,5 +79,12 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
}
latest, hasLatest := latestChapterFrom(site, seriesURL, body)
cover, hasCover := coverFrom(site, seriesURL, body)
return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil
return seriesRead{
Latest: latest,
HasLatest: hasLatest,
Cover: cover,
HasCover: hasCover,
SiteCompleted: siteCompletedFrom(site, seriesURL, body),
BodyLen: len(body),
}, nil
}
+119 -14
View File
@@ -22,10 +22,10 @@ type latestChapter struct {
// site answers the fixed questions every series-page read asks of its Site
// (ADR-0009): the host its addresses must carry, how to find the Latest
// Chapter and the Cover address in a body, and — for a Site behind a
// JavaScript challenge — how to read its payload from a cleared tab. One
// entry describes everything about one Site, and nowhere else gets to compare
// the site string.
// Chapter and the Cover address in a body, whether the Site calls the work
// completed, and — for a Site behind a JavaScript challenge — how to read its
// payload from a cleared tab. One entry describes everything about one Site,
// and nowhere else gets to compare the site string.
type site struct {
// Host is the exact hostname a series_url for this Site must carry.
Host string
@@ -33,6 +33,10 @@ type site struct {
LatestChapter func(seriesURL, body string) (latestChapter, bool)
// Cover finds the Cover address in a fetched body.
Cover func(seriesURL, body string) (string, bool)
// Completed reports whether this body carries the Site's own completed
// value. False for a body that carries any other value, and false for a
// failed extraction — never an error and never a third state.
Completed func(seriesURL, body string) bool
// Rest is how long a Series of this Site rests between Polls.
Rest time.Duration
// Gap is the Lane's strictest pace: at least one second must pass between
@@ -299,34 +303,42 @@ func kaganeCoverURL(body string) string {
return ""
}
func comixCoverURL(seriesURL, body string) string {
// comixDetailQuery returns the ["manga","detail","<id>"] query entry of
// comix's initial-data JSON, or nil. The cover and completed reads share the
// scoped lookup so a "recommended" strip entry can never contribute either
// answer.
func comixDetailQuery(seriesURL, body string) json.RawMessage {
id, ok := comixSeriesID(seriesURL)
if !ok {
return ""
return nil
}
data := comixInitialDataRe.FindStringSubmatch(body)
if data == nil {
return ""
return nil
}
var state struct {
Queries map[string]json.RawMessage `json:"queries"`
}
if err := json.Unmarshal([]byte(data[1]), &state); err != nil {
return nil
}
return state.Queries[`["manga","detail","`+id+`"]`]
}
func comixCoverURL(seriesURL, body string) string {
detail := comixDetailQuery(seriesURL, body)
if len(detail) == 0 {
return ""
}
raw := state.Queries[`["manga","detail","`+id+`"]`]
if len(raw) == 0 {
return ""
}
var detail struct {
var entry struct {
Poster struct {
Medium string `json:"medium"`
} `json:"poster"`
}
if err := json.Unmarshal(raw, &detail); err != nil {
if err := json.Unmarshal(detail, &entry); err != nil {
return ""
}
return publishedCoverURL(detail.Poster.Medium)
return publishedCoverURL(entry.Poster.Medium)
}
// ogImageCover reads the og:image metadata shared by asura, demonic and
@@ -360,6 +372,87 @@ func coverFrom(site, seriesURL, body string) (string, bool) {
return "", false
}
// asuraStatusRe matches the status value inside the escaped astro-island
// props blob, in both the &quot; form the served document carries and the "
// form a decoded copy would. "completed" is the only true value: "dropped"
// is scanlation editorial (the work itself continues elsewhere) and hiatus is
// its own value.
var asuraStatusRe = regexp.MustCompile(`(?:&quot;|")status(?:&quot;|"):\[0,(?:&quot;|")completed(?:&quot;|")\]`)
func asuraCompleted(_, body string) bool {
return asuraStatusRe.MatchString(body)
}
// demonicStatusRe matches the info block's status pair: a Status label <li>
// immediately followed by the value <li>. The site's whole status vocabulary
// is {Ongoing, Completed} (its advanced-search status filter), so the literal
// Completed value is the entire signal.
var demonicStatusRe = regexp.MustCompile(`<li[^>]*>\s*Status\s*</li>\s*<li[^>]*>\s*Completed\s*</li>`)
func demonicCompleted(_, body string) bool {
return demonicStatusRe.MatchString(body)
}
// comixCompleted reads "status" from the scoped detail entry only;
// "finished" is the completed value, and on_hiatus and discontinued are
// distinct values.
func comixCompleted(seriesURL, body string) bool {
detail := comixDetailQuery(seriesURL, body)
if len(detail) == 0 {
return false
}
var entry struct {
Status string `json:"status"`
}
if err := json.Unmarshal(detail, &entry); err != nil {
return false
}
return entry.Status == "finished"
}
// kaganeCompleted reads publication_status only: upload_status is the
// release's state, and the two provably diverge ('Cause Calypso Can,
// 2026-08-19: publication Ongoing, upload Hiatus), so a Completed upload
// must never read as a Completed work.
func kaganeCompleted(_, body string) bool {
var series struct {
PublicationStatus string `json:"publication_status"`
}
if err := json.Unmarshal([]byte(body), &series); err != nil {
return false
}
return series.PublicationStatus == "Completed"
}
// novelfullStatusRe matches the info panel's status link. The page's whole
// status vocabulary is {Ongoing, Completed} (the "OnGoing" spelling aliases
// "Ongoing" on the taxonomy), so the Completed href is the signal.
var novelfullStatusRe = regexp.MustCompile(`href="/status/Completed"`)
func novelfullCompleted(_, body string) bool {
return novelfullStatusRe.MatchString(body)
}
// lnwCompleted matches creativeWorkStatus in the head's JSON-LD block. The
// whole body is scanned and the comment marker is not required, unlike
// lnwLatestChapter: the status block sits ahead of the visitor-writable
// thread, and hiatus maps to a distinct PotentialActionStatus value.
var lnwStatusRe = regexp.MustCompile(`"creativeWorkStatus"\s*:\s*"https://schema\.org/CompletedActionStatus"`)
func lnwCompleted(_, body string) bool {
return lnwStatusRe.MatchString(body)
}
// siteCompletedFrom reports whether the Site calls this work completed, via
// the Site's registry entry. False for an unknown site, a challenge body and
// a redesign alike: an absent hint, never a claim.
func siteCompletedFrom(site, seriesURL, body string) bool {
if fn := sites[site].Completed; fn != nil {
return fn(seriesURL, body)
}
return false
}
// metaContent returns the content of the first <meta> whose attrName is
// attrValue. It keeps scanning after an empty match so a later published cover
// is not hidden by an empty tag.
@@ -449,6 +542,7 @@ var sites = map[string]site{
Host: "asurascans.com",
LatestChapter: asuraLatestChapter,
Cover: ogImageCover,
Completed: asuraCompleted,
Rest: defaultRest,
Gap: defaultGap,
},
@@ -456,6 +550,7 @@ var sites = map[string]site{
Host: "demonicscans.org",
LatestChapter: demonicLatestChapter,
Cover: ogImageCover,
Completed: demonicCompleted,
Rest: defaultRest,
Gap: defaultGap,
},
@@ -463,6 +558,7 @@ var sites = map[string]site{
Host: "comix.to",
LatestChapter: comixLatestChapter,
Cover: comixCoverEntry,
Completed: comixCompleted,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
@@ -479,6 +575,7 @@ var sites = map[string]site{
Host: "kagane.to",
LatestChapter: kaganeLatestChapter,
Cover: kaganeCoverEntry,
Completed: kaganeCompleted,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
@@ -493,6 +590,7 @@ var sites = map[string]site{
Host: "novelfull.com",
LatestChapter: novelfullLatestChapter,
Cover: novelfullCoverEntry,
Completed: novelfullCompleted,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
@@ -507,6 +605,7 @@ var sites = map[string]site{
Host: "lightnovelworld.net",
LatestChapter: lnwLatestChapter,
Cover: ogImageCover,
Completed: lnwCompleted,
Rest: defaultRest,
Gap: defaultGap,
},
@@ -524,6 +623,12 @@ func SiteNames() []string {
return names
}
// BrowserBackedSites is derived from the registry: the Sites whose pages are
// read through the browser sidecar. Sorted so callers that range it (the
// browser fetcher's dispatch) see a stable order instead of map-iteration
// noise. Exported so the web layer shares the same set the poller does.
func BrowserBackedSites() []string { return browserBackedSites() }
// browserBackedSites is derived from the registry: the Sites whose pages are
// read through the browser sidecar. Sorted so callers that range it (the
// browser fetcher's dispatch) see a stable order instead of map-iteration
+273
View File
@@ -188,6 +188,104 @@ const lnwSeriesFixture = `
</div>
`
// Completed-marker fixtures, trimmed from the live pages fetched 2026-08-22
// for the verification step below. Selector-removed rows delete the marker
// from the consts and must answer false.
// Trimmed from https://asurascans.com/comics/solo-leveling (301 to
// /comics/solo-leveling-b60d532c) fetched 2026-08-22 by a curl probe from
// this machine. The astro-island props are HTML-escaped in the served
// document, so the quotes arrive as &quot;.
const asuraCompletedFixture = `
&quot;bookmarkCount&quot;:[0,39128],&quot;status&quot;:[0,&quot;completed&quot;],&quot;type&quot;:[0,&quot;manhwa&quot;]
`
// Trimmed from https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af
// fetched 2026-08-22 by a curl probe.
const asuraOngoingFixture = `
&quot;bookmarkCount&quot;:[0,54027],&quot;status&quot;:[0,&quot;ongoing&quot;],&quot;type&quot;:[0,&quot;manhwa&quot;]
`
// Trimmed from https://demonicscans.org/manga/Solo-Leveling fetched
// 2026-08-22 by a curl probe. The info block is sloppy: bare <li>s inside a
// <div>, label and value as a sibling pair.
const demonicCompletedFixture = `
<div class="flex flex-row">
<li style="width:150px;color:#b2b2b2;">Status</li>
<li>Completed</li>
</div>
`
// Trimmed from https://demonicscans.org/manga/Catastrophic-Necromancer
// fetched 2026-08-22 by a curl probe. Same block, ongoing value.
const demonicOngoingFixture = `
<div class="flex flex-row">
<li style="width:150px;color:#b2b2b2;">Status</li>
<li>Ongoing</li>
</div>
`
// Trimmed from https://comix.to/title/q77m-countach fetched 2026-08-22 by a
// cleared Chrome tab (the CDP sidecar: the Series URL is fetched in-tab, the
// same server-rendered payload the poll reads). The served page escapes the
// query map keys as \u0022; the fixture carries the decoded form, which
// parses to the same key. The recommended strip on this very page carries a
// finished entry; only the ["manga","detail","q77m"] entry counts.
const comixCompletedFixture = `<script type="application/json" id="initial-data">{"queries":{"[\"manga\",\"recommended\",\"q77m\",1]":{"items":[{"hid":"pz65","title":"Wangan Midnight: C1 Runner","status":"finished"}]},"[\"manga\",\"detail\",\"q77m\"]":{"id":13211,"hid":"q77m","title":"Countach","status":"finished","originalLanguage":"ja"}}}</script>`
// Trimmed from https://comix.to/title/n8we-dungeons-and-crayons fetched
// 2026-08-22 the same way (keys in their decoded form, as above). The
// recommended strip includes a finished entry; the target detail is
// releasing, and only the detail read counts.
const comixOngoingFixture = `<script type="application/json" id="initial-data">{"queries":{"[\"manga\",\"recommended\",\"n8we\",1]":{"items":[{"hid":"g2rk","title":"On the Way to Meet Mom","status":"finished"}]},"[\"manga\",\"detail\",\"n8we\"]":{"id":1429,"hid":"n8we","title":"Dungeons and Crayons","status":"releasing","originalLanguage":"ko"}}}</script>`
// Trimmed from GET https://kagane.to/api/v2/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b
// fetched 2026-08-22 in a cleared Chrome tab (plain TLS serves the Cloudflare
// challenge).
const kaganeOngoingFixture = `{"series_id":"019f84bc-9ba0-7ed9-86f5-8b905ec7c28b","title":"Infinite Decryption: The Strongest Level 0","publication_status":"Ongoing","upload_status":"Ongoing"}`
// Trimmed from GET https://kagane.to/api/v2/series/019c29c3-5abd-70e6-9efc-1f1f22d839f6
// fetched 2026-08-22 the same way.
const kaganeCompletedFixture = `{"series_id":"019c29c3-5abd-70e6-9efc-1f1f22d839f6","title":"Real Account 1-17","publication_status":"Completed","upload_status":"Completed"}`
// Composed, not a single live trim: upload Completed alongside a
// non-Completed publication status is the research note's inferred inverse
// case and did not turn up in the ~1900-series live scan of 2026-08-22 (both
// field vocabularies are live-verified; the note's live divergence is 'Cause
// Calypso Can, publication Ongoing + upload Hiatus). The predicate must read
// publication_status only.
const kaganeDivergentFixture = `{"series_id":"019f84bc-9ba0-7ed9-86f5-8b905ec7c28b","title":"Infinite Decryption: The Strongest Level 0","publication_status":"Ongoing","upload_status":"Completed"}`
// Trimmed from https://novelfull.com/reverend-insanity.html fetched
// 2026-08-22 in a cleared Chrome tab after the plain-TLS probe was served the
// challenge (time-varying; plain curl answered 403 the same day).
const novelfullCompletedFixture = `<div><h3>Status:</h3><a href="/status/Completed">Completed</a></div>`
// Trimmed from https://novelfull.com/a-cunning-pervert-in-the-cultivation-world.html
// fetched 2026-08-22 the same way.
const novelfullOngoingFixture = `<div><h3>Status:</h3><a href="/status/Ongoing">Ongoing</a></div>`
// Trimmed from the JSON-LD block in the head of
// https://lightnovelworld.net/novel/a-will-eternal/ fetched 2026-08-22. The
// block sits ahead of the wpdiscuz thread, so the predicate reads the whole
// body and needs no comment marker, unlike lnwLatestChapter.
const lnwCompletedFixture = `<script type="application/ld+json">{
"@context": "https://schema.org",
"@type": "Book",
"name": "A Will Eternal",
"creativeWorkStatus": "https://schema.org/CompletedActionStatus"
}</script>`
// Trimmed from
// https://lightnovelworld.net/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/
// fetched 2026-08-22. Same block, ongoing value.
const lnwOngoingFixture = `<script type="application/ld+json">{
"@context": "https://schema.org",
"@type": "Book",
"name": "All Jobs and Classes I Just Wanted One Skill Not Them All",
"creativeWorkStatus": "https://schema.org/ActiveActionStatus"
}</script>`
// Trimmed from https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af
// (redirected to ...-00dcbf97) on 2026-08-10.
const asuraCoverFixture = `<meta property="og:image" content="https://cdn.asurascans.com/asura-images/covers/chronicles-of-the-demon-faction.d4dcb8.webp">`
@@ -473,3 +571,178 @@ func TestLatestChapterFrom(t *testing.T) {
})
}
}
func TestSiteCompletedFrom(t *testing.T) {
const asuraURL = "https://asurascans.com/comics/solo-leveling-b60d532c"
const demonicURL = "https://demonicscans.org/manga/Solo-Leveling"
const comixURL = "https://comix.to/title/q77m-countach"
const kaganeURL = "https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
const novelfullURL = "https://novelfull.com/reverend-insanity.html"
const lnwURL = "https://lightnovelworld.net/novel/a-will-eternal/"
tests := []struct {
name string
site string
seriesURL string
body string
want bool
}{
{
name: "asura completed via escaped props status",
site: "asura", seriesURL: asuraURL, body: asuraCompletedFixture,
want: true,
},
{
name: "asura ongoing value is not completed",
site: "asura",
seriesURL: "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af",
body: asuraOngoingFixture,
want: false,
},
{
name: "asura with the status key removed",
site: "asura", seriesURL: asuraURL,
body: strings.ReplaceAll(asuraCompletedFixture, `&quot;status&quot;:[0,&quot;completed&quot;]`, ""),
want: false,
},
{
name: "asura challenge page",
site: "asura", seriesURL: asuraURL, body: challengeFixture,
want: false,
},
{
name: "demonic completed info-block pair",
site: "demonic", seriesURL: demonicURL, body: demonicCompletedFixture,
want: true,
},
{
name: "demonic ongoing value is not completed",
site: "demonic",
seriesURL: "https://demonicscans.org/manga/Catastrophic-Necromancer",
body: demonicOngoingFixture,
want: false,
},
{
name: "demonic with the value li removed",
site: "demonic", seriesURL: demonicURL,
body: strings.ReplaceAll(demonicCompletedFixture, "<li>Completed</li>", ""),
want: false,
},
{
name: "comix recommended finished does not count",
site: "comix",
seriesURL: "https://comix.to/title/n8we-dungeons-and-crayons",
body: comixOngoingFixture,
want: false,
},
{
name: "comix detail finished wins over a finished recommended strip",
site: "comix",
seriesURL: comixURL,
body: comixCompletedFixture,
want: true,
},
{
name: "comix with the detail status removed",
site: "comix", seriesURL: comixURL,
body: strings.ReplaceAll(comixCompletedFixture, `"status":"finished",`, ""),
want: false,
},
{
name: "comix challenge page",
site: "comix", seriesURL: comixURL, body: challengeFixture,
want: false,
},
{
name: "kagane publication Completed",
site: "kagane",
seriesURL: "https://kagane.to/series/019c29c3-5abd-70e6-9efc-1f1f22d839f6",
body: kaganeCompletedFixture,
want: true,
},
{
name: "kagane upload Completed does not count",
site: "kagane", seriesURL: kaganeURL, body: kaganeDivergentFixture,
want: false,
},
{
name: "kagane ongoing values are not completed",
site: "kagane", seriesURL: kaganeURL, body: kaganeOngoingFixture,
want: false,
},
{
name: "kagane with publication_status removed",
site: "kagane", seriesURL: kaganeURL,
body: strings.ReplaceAll(kaganeCompletedFixture, `"publication_status":"Completed",`, ""),
want: false,
},
{
name: "kagane challenge page",
site: "kagane", seriesURL: kaganeURL, body: challengeFixture,
want: false,
},
{
name: "novelfull status link Completed",
site: "novelfull",
seriesURL: novelfullURL,
body: novelfullCompletedFixture,
want: true,
},
{
name: "novelfull ongoing link is not completed",
site: "novelfull",
seriesURL: "https://novelfull.com/a-cunning-pervert-in-the-cultivation-world.html",
body: novelfullOngoingFixture,
want: false,
},
{
name: "novelfull with the status link removed",
site: "novelfull", seriesURL: novelfullURL,
body: strings.ReplaceAll(novelfullCompletedFixture, `<a href="/status/Completed">Completed</a>`, ""),
want: false,
},
{
name: "novelfull challenge page",
site: "novelfull", seriesURL: novelfullURL, body: challengeFixture,
want: false,
},
{
name: "lightnovelworld JSON-LD CompletedActionStatus",
site: "lightnovelworld",
seriesURL: lnwURL,
body: lnwCompletedFixture,
want: true,
},
{
name: "lightnovelworld ActiveActionStatus is not completed",
site: "lightnovelworld",
seriesURL: "https://lightnovelworld.net/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/",
body: lnwOngoingFixture,
want: false,
},
{
name: "lightnovelworld with creativeWorkStatus removed",
site: "lightnovelworld", seriesURL: lnwURL,
body: strings.ReplaceAll(lnwCompletedFixture, `"creativeWorkStatus": "https://schema.org/CompletedActionStatus"`, ""),
want: false,
},
{
name: "lightnovelworld challenge page",
site: "lightnovelworld", seriesURL: lnwURL, body: challengeFixture,
want: false,
},
{
name: "unknown site",
site: "mangadex", seriesURL: "https://mangadex.org/title/x", body: asuraCompletedFixture,
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := siteCompletedFrom(tt.site, tt.seriesURL, tt.body); got != tt.want {
t.Errorf("siteCompletedFrom(%q, %q, body) = %v, want %v", tt.site, tt.seriesURL, got, tt.want)
}
})
}
}
+122
View File
@@ -0,0 +1,122 @@
// Package notify posts owner-notice embeds to a Discord webhook. It is
// deliberately small and stdlib-only: one POST of a JSON body needs no
// Discord library, and the path imports nothing of this repo's session or
// OAuth packages — that independence is why a webhook was chosen over a bot
// (issue #171, AC9).
package notify
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
"bookmarkmanager/backend/internal/latest"
)
// dangerColor is the dark design branch's --danger, #cf5c4d = 13589581. The
// integer is unreadable, so a future edit will otherwise "fix" it — do not:
// --ember means new chapter only, and a fault wearing ember would tell the
// owner a stall is a release. This is the one colour a fault wears.
const dangerColor = 13589581
// Client posts owner notices to one Discord webhook. The webhook address is a
// secret in the class of TOKEN_KEY: it is never logged, never rendered, and
// never carried in a returned error, which the poller logs.
type Client struct {
webhookURL string
baseURL string // the deployment's public origin; embed URLs resolve against it
http *http.Client
}
// New returns a Client posting to webhookURL. baseURL is the deployment's
// public origin (Config.PublicBaseURL); the embed's deep-linked title is
// built from it.
func New(webhookURL, baseURL string) *Client {
return &Client{
webhookURL: webhookURL,
baseURL: strings.TrimSuffix(baseURL, "/"),
http: &http.Client{Timeout: 10 * time.Second},
}
}
// Notify posts one owner notice as a Discord embed: the danger colour, the
// subject as a deep-linked title, the sentence as the description, the pass
// time as the timestamp and the machine word in the footer. The send is
// wrapped in a deadline so a hanging Discord cannot hold a poller Lane. On
// failure the error carries no part of the webhook address (the poller logs
// it), and the caller leaves the suppression row unwritten so the next pass
// retries while the condition holds.
func (c *Client) Notify(ctx context.Context, f latest.Fault, sentence, href string) error {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
body, err := json.Marshal(c.payload(f, sentence, href))
if err != nil {
return fmt.Errorf("owner notice: marshal: %w", err)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.webhookURL, strings.NewReader(string(body)))
if err != nil {
return errors.New("owner notice: build request")
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
if err != nil {
// The transport error embeds the webhook address; the address is a
// secret in the class of TOKEN_KEY and the poller logs this error.
return errors.New("owner notice: send failed")
}
defer resp.Body.Close()
// Cap the read: a Discord error page is enough, and draining the body
// lets the connection be reused.
io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return fmt.Errorf("owner notice: webhook status %d", resp.StatusCode)
}
return nil
}
// payload is the webhook body: one embed and nothing else. No fields, no
// thumbnail, no author block — the wire shape is what Discord reads.
func (c *Client) payload(f latest.Fault, sentence, href string) webhookPayload {
return webhookPayload{Embeds: []embed{{
Color: dangerColor,
Title: subject(f),
URL: c.baseURL + href,
Description: sentence,
Timestamp: time.UnixMilli(f.Since).UTC().Format(time.RFC3339),
Footer: embedFooter{Text: f.Condition},
}}}
}
// subject renders the embed's title: the Site the fault is about, with the
// machine word so the title needs no per-condition wording here — #172's
// conditions pass different sentences, not a different builder. For a fault
// that is not one Site's, the machine word stands alone.
func subject(f latest.Fault) string {
if f.Site == "" {
return f.Condition
}
return f.Site + ": " + f.Condition
}
type webhookPayload struct {
Embeds []embed `json:"embeds"`
}
type embed struct {
Color int `json:"color"`
Title string `json:"title"`
URL string `json:"url"`
Description string `json:"description"`
Timestamp string `json:"timestamp"`
Footer embedFooter `json:"footer"`
}
type embedFooter struct {
Text string `json:"text"`
}
+100
View File
@@ -0,0 +1,100 @@
package notify_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/notify"
)
// TestNotifyEmbedShape pins the wire shape Discord actually reads: one embed
// in the danger colour with the subject as a deep-linked title built from the
// base URL, the sentence as the description, the pass time as an RFC3339
// timestamp, the machine word in the footer, and no fields grid (nor
// thumbnail, nor author block) at all. The webhook is a local server, so no
// test can reach Discord.
func TestNotifyEmbedShape(t *testing.T) {
var body map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if ct := r.Header.Get("Content-Type"); ct != "application/json" {
t.Errorf("Content-Type = %q, want application/json", ct)
}
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Errorf("decode request body: %v", err)
}
w.WriteHeader(http.StatusNoContent)
}))
defer srv.Close()
passTime := time.UnixMilli(5_000_000).UTC()
c := notify.New(srv.URL, "https://bookmarks.test/")
err := c.Notify(context.Background(),
latest.Fault{Condition: latest.ConditionStall, Site: "comix", Since: passTime.UnixMilli()},
"sentence", "/admin/lanes")
if err != nil {
t.Fatalf("Notify: %v", err)
}
embeds, ok := body["embeds"].([]any)
if !ok || len(embeds) != 1 {
t.Fatalf("embeds = %#v, want exactly one embed", body["embeds"])
}
embed, ok := embeds[0].(map[string]any)
if !ok {
t.Fatalf("embed = %#v, want an object", embeds[0])
}
if color, ok := embed["color"].(float64); !ok || int(color) != 13589581 {
t.Fatalf("color = %#v, want 13589581 (--danger #cf5c4d)", embed["color"])
}
if url := embed["url"]; url != "https://bookmarks.test/admin/lanes" {
t.Fatalf("url = %#v, want the title deep-linked off the base URL", url)
}
if desc := embed["description"]; desc != "sentence" {
t.Fatalf("description = %#v, want the sentence", desc)
}
footer, ok := embed["footer"].(map[string]any)
if !ok || footer["text"] != latest.ConditionStall {
t.Fatalf("footer = %#v, want the machine word in the footer", embed["footer"])
}
ts, ok := embed["timestamp"].(string)
if !ok {
t.Fatalf("timestamp = %#v, want an RFC3339 string", embed["timestamp"])
}
parsed, err := time.Parse(time.RFC3339, ts)
if err != nil || !parsed.Equal(passTime) {
t.Fatalf("timestamp = %q, want %s (the pass time, RFC3339)", ts, passTime.Format(time.RFC3339))
}
for _, banned := range []string{"fields", "thumbnail", "author"} {
if _, ok := embed[banned]; ok {
t.Fatalf("embed has %q, want it absent (no field grid, no thumbnail, no author block)", banned)
}
}
}
// A non-2xx answer is an error the caller logs, and the error never carries
// the webhook address — a secret in the class of TOKEN_KEY, and the poller
// logs every notify error.
func TestNotifyNon2xxIsErrorWithoutTheAddress(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer srv.Close()
c := notify.New(srv.URL, "https://bookmarks.test")
err := c.Notify(context.Background(),
latest.Fault{Condition: latest.ConditionStall, Site: "comix", Since: 5_000_000},
"sentence", "/admin/lanes")
if err == nil {
t.Fatal("Notify = nil, want an error for a 500")
}
if strings.Contains(err.Error(), srv.URL) {
t.Fatalf("error %q leaks the webhook address", err)
}
}
+75 -32
View File
@@ -9,32 +9,38 @@ import (
// Series filter names (issue #140): the seven repair filters are ordered
// permanent-then-fixable — the repairs nothing will ever undo first, the
// ones a Poll can make right after. SeriesFilterFinished is not part of
// that ordering: a finished Series is a deliberate state, not a repair, so
// it sits last, informational. A name is the repair a row needs, not the
// SQL that finds it; the values are the wire form the Series list URL
// carries (#142). "all" is the absent and unknown case: every Series.
// ones a Poll can make right after. SeriesFilterFinished and
// SeriesFilterSiteCompleted are not part of that ordering: a finished
// Series is a deliberate state and a Site-completed one is the Site's own
// marker, not repairs, so the pair rides the tail, informational. A name is
// the repair a row needs, not the SQL that finds it; the values are the
// wire form the Series list URL carries (#142). "all" is the absent and
// unknown case: every Series.
const (
SeriesFilterAll = "all"
SeriesFilterNoURL = "no_series_url"
SeriesFilterNoChapter = "never_read_a_chapter"
SeriesFilterNoReaders = "no_readers"
SeriesFilterNeverChecked = "never_checked"
SeriesFilterStale = "stale"
SeriesFilterNoCover = "no_cover"
SeriesFilterReaderReport = "reader_report"
SeriesFilterFinished = "finished"
SeriesFilterAll = "all"
SeriesFilterNoURL = "no_series_url"
SeriesFilterNoChapter = "never_read_a_chapter"
SeriesFilterNoReaders = "no_readers"
SeriesFilterNeverChecked = "never_checked"
SeriesFilterStale = "stale"
SeriesFilterNoCover = "no_cover"
SeriesFilterReaderReport = "reader_report"
SeriesFilterFinished = "finished"
SeriesFilterSiteCompleted = "site_completed"
SeriesFilterFailing = "failing"
SeriesFilterUnverified = "unverified"
)
// SeriesFilter is one named filter predicate over the whole library. Site
// and Kind narrow the row read; Name picks the predicate; Cutoff is the
// staleness boundary the "stale" filter compares against, supplied by the
// caller's clock — the store has no clock; Page is 1-based.
// staleness boundary the age-based filters — "stale" and the failing pair —
// compare against, supplied by the caller's clock — the store has no clock;
// Page is 1-based.
type SeriesFilter struct {
Site string // "" = every Site
Kind string // "" = both library buckets' series
Name string // one of the SeriesFilter* constants; "" = SeriesFilterAll
Cutoff int64 // unix ms; "stale" reads it, the store never does
Cutoff int64 // unix ms; the age-based filters read it, the store never does
Page int // 1-based page of the row read; default 1
}
@@ -45,7 +51,7 @@ type SeriesFilter struct {
// and only the anonymous boolean in raisedByReaderAnswer crosses it.
const adminSeriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.force_poll_at,
s.latest_corrected_at, s.finished_at`
s.latest_corrected_at, s.finished_at, s.site_completed_at`
// raisedByReaderAnswer answers "did a Reader's report set this number" without
// naming which Reader. Kept apart from adminSeriesColumns so the column list —
@@ -54,6 +60,12 @@ const adminSeriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover_
// Reader behind them.
const raisedByReaderAnswer = `(s.latest_raised_by IS NOT NULL) AS raised_by_reader`
// failureAnswer is the poll-failures row's answer (issue #165), kept apart
// from adminSeriesColumns like raisedByReaderAnswer: outcome and failing_since
// are not Series columns, and the COALESCE keeps the row scannable when the
// LEFT JOIN finds no failure row.
const failureAnswer = `COALESCE(f.outcome, ''), COALESCE(f.failing_since, 0)`
// seriesPageSize is the row read's page length. The tie-break in the query's
// ORDER BY is what makes this a stable page boundary — see SeriesPage.
const seriesPageSize = 50
@@ -83,9 +95,19 @@ type AdminSeries struct {
LatestCorrectedAt int64
ReaderCount int
RaisedByReader bool // a Reader's report set LatestChapterNum
// FailureOutcome is the outcome word of the Series' standing failure, ""
// when no failure row stands. FailingSince is when the run of failures
// began, zero with no row. Both come from the LEFT JOIN, not the Series
// row (issue #165: the row's existence is the state).
FailureOutcome string
FailingSince int64
// FinishedAt is the owner's finish stamp: unix ms, zero while the Series is
// not finished — the same shape as the Correction stamp, and its own undo.
FinishedAt int64
// SiteCompletedAt is when the last successful Poll read saw the Site's own
// completed value, zero meaning it did not (issue #168). The Site's marker,
// never a Lifecycle decision: the owner's Finish is the only retirement.
SiteCompletedAt int64
}
// SeriesPage is one page of the owner's filtered Series list plus the count
@@ -119,17 +141,26 @@ func (a AdminSeries) Key() string { return a.Site + ":" + a.SeriesID }
// gate case is invisible to SQL, needs the Site registry in Go, and belongs to
// a later repair), never-read-a-chapter and never-checked as disjoint halves
// (non-zero versus zero check stamp), stale, no cover, finished (the
// retirement stamp, read directly), and Reader-report.
// retirement stamp, read directly), site completed (the Site's marker, read
// directly), Reader-report, and the failing pair — failing (the failure row
// exists, a chapter exists, and failing_since is past the cutoff) and
// unverified (the Reader-attributed subset of failing).
// failing's chapter IS NOT NULL is the exact complement of never-read-a-
// chapter's IS NULL half, so the two are disjoint by construction.
// no_readers is the one HAVING predicate: it is the orphan test, an aggregate
// over the LEFT JOIN, where a bare WHERE has no row to test.
//
// The clock-versus-outcome split decides which predicates exclude finished
// Series (`s.finished_at = 0` in each of the four): the clock-driven one —
// Series (`s.finished_at = 0` in each of the five): the clock-driven ones —
// never-checked, stale, no-chapter, no-cover — keep ticking after the last
// Poll, so they would report a retired row as a problem no Poll is coming to
// fix; the three outcome-driven ones — no-URL, no-readers, Reader-report —
// read stored facts that simply stop arriving, so a finished Series needing a
// genuine repair still shows up under them.
// fix; site-completed's stamp is the Site's, and it keeps standing after the
// owner retires the row, so a finished Series would be reported as work
// nobody is going to do. The outcome-driven ones — no-URL, no-readers,
// Reader-report, failing, unverified — read stored facts that simply stop
// arriving, so a finished Series needing a genuine repair still shows up
// under them. The failing pair carries no finished guard for exactly that
// contrast: a failure row is a stored outcome, not a ticking clock.
//
// stale is the checked-but-old half of the stamp partition — because the
// verdict line wants "not checked in twelve hours" as one figure, and a never
@@ -157,8 +188,16 @@ func adminFilter(f SeriesFilter) (where, having string, args []any, err error) {
clauses = append(clauses, `s.cover_address = '' AND s.finished_at = 0`)
case SeriesFilterReaderReport:
clauses = append(clauses, `s.latest_raised_by IS NOT NULL`)
case SeriesFilterFailing:
clauses = append(clauses, `f.site IS NOT NULL AND s.latest_chapter_num IS NOT NULL AND f.failing_since < $`+strconv.Itoa(len(args)+1))
args = append(args, f.Cutoff)
case SeriesFilterUnverified:
clauses = append(clauses, `f.site IS NOT NULL AND s.latest_chapter_num IS NOT NULL AND f.failing_since < $`+strconv.Itoa(len(args)+1)+` AND s.latest_raised_by IS NOT NULL`)
args = append(args, f.Cutoff)
case SeriesFilterFinished:
clauses = append(clauses, `s.finished_at > 0`)
case SeriesFilterSiteCompleted:
clauses = append(clauses, `s.site_completed_at > 0 AND s.finished_at = 0`)
case SeriesFilterNoReaders:
having = `HAVING COUNT(b.reader_id) = 0`
default:
@@ -204,15 +243,18 @@ func (s *Store) SeriesPage(f SeriesFilter) (SeriesPage, error) {
base := len(args)
args = append(args, seriesPageSize, seriesPageSize*(f.Page-1))
rows, err := s.db.Query(`
SELECT `+adminSeriesColumns+`, `+raisedByReaderAnswer+`,
SELECT `+adminSeriesColumns+`, `+raisedByReaderAnswer+`, `+failureAnswer+`,
COUNT(b.reader_id) AS reader_count,
COUNT(*) OVER () AS filtered_total
FROM series s
LEFT JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
LEFT JOIN poll_failures f ON f.site = s.site AND f.series_id = s.series_id
`+where+`
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
s.force_poll_at, s.latest_corrected_at, s.finished_at, s.latest_raised_by
s.force_poll_at, s.latest_corrected_at, s.finished_at, s.site_completed_at,
s.latest_raised_by,
f.outcome, f.failing_since
`+having+`
ORDER BY s.latest_checked_at, s.site, s.series_id
LIMIT $`+strconv.Itoa(base+1)+` OFFSET $`+strconv.Itoa(base+2), args...)
@@ -251,6 +293,7 @@ func (s *Store) SeriesShapes(f SeriesFilter) ([]SiteSeriesShape, error) {
SELECT s.site, s.kind
FROM series s
LEFT JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
LEFT JOIN poll_failures f ON f.site = s.site AND f.series_id = s.series_id
`+where+`
GROUP BY s.site, s.series_id, s.kind
`+having+`
@@ -273,11 +316,11 @@ func (s *Store) SeriesShapes(f SeriesFilter) ([]SiteSeriesShape, error) {
return out, rows.Err()
}
// scanAdminSeries reads one row in adminSeriesColumns + raisedByReaderAnswer
// order, plus the query's reader_count and filtered_total columns, and returns
// the window total alongside the row. latest_chapter_num is NULL until first
// captured — the "never read a chapter" state. The Sighting-raiser column is
// never among the scanned columns.
// scanAdminSeries reads one row in adminSeriesColumns + raisedByReaderAnswer +
// failureAnswer order, plus the query's reader_count and filtered_total
// columns, and returns the window total alongside the row. latest_chapter_num
// is NULL until first captured — the "never read a chapter" state. The
// Sighting-raiser column is never among the scanned columns.
func scanAdminSeries(scan func(...any) error) (AdminSeries, int, error) {
var (
a AdminSeries
@@ -287,8 +330,8 @@ func scanAdminSeries(scan func(...any) error) (AdminSeries, int, error) {
if err := scan(
&a.Site, &a.SeriesID, &a.Title, &a.SeriesURL, &a.CoverAddress,
&a.Kind, &a.LatestChapter, &latestChapterNum, &a.LatestCheckedAt,
&a.ForcePollAt, &a.LatestCorrectedAt, &a.FinishedAt,
&a.RaisedByReader, &a.ReaderCount, &total,
&a.ForcePollAt, &a.LatestCorrectedAt, &a.FinishedAt, &a.SiteCompletedAt,
&a.RaisedByReader, &a.FailureOutcome, &a.FailingSince, &a.ReaderCount, &total,
); err != nil {
return AdminSeries{}, 0, err
}
+218 -11
View File
@@ -12,14 +12,15 @@ import (
// Upsert path could not produce together: an orphan has no bookmark, and a
// Reader-raised Latest Chapter needs a Sighting the store does not create.
type seriesSeed struct {
key string
kind string
url string
cover string // cover_address
checkedAt int64
latestNum *float64
bookmarks int // readers that hold it; 0 = orphan
raisedBy bool // a Reader's report is attributed as the raiser
key string
kind string
url string
cover string // cover_address
checkedAt int64
latestNum *float64
bookmarks int // readers that hold it; 0 = orphan
raisedBy bool // a Reader's report is attributed as the raiser
siteCompletedAt int64 // the Site's own marker (issue #168), 0 = not set
}
// seedAdminSeries inserts one series row and its bookmarks (owner first, then
@@ -39,10 +40,10 @@ func seedAdminSeries(t *testing.T, s *Store, seed seriesSeed) {
}
if _, err := s.db.Exec(`
INSERT INTO series (site, series_id, title, kind, series_url, cover_address,
latest_checked_at, latest_chapter, latest_chapter_num)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
latest_checked_at, latest_chapter, latest_chapter_num, site_completed_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
site, seriesID, "Title of "+seed.key, seed.kind, seed.url, seed.cover,
seed.checkedAt, latestChapter, seed.latestNum); err != nil {
seed.checkedAt, latestChapter, seed.latestNum, seed.siteCompletedAt); err != nil {
t.Fatalf("seed series %q: %v", seed.key, err)
}
for i := range seed.bookmarks {
@@ -527,3 +528,209 @@ func TestAdminSeriesCarriesFinishedAt(t *testing.T) {
t.Fatalf("row = %+v, want FinishedAt 5000", page.Rows)
}
}
// The failing pair reads the failure row's age, not the Series row (issue
// #165): failing requires the joined row, a Latest Chapter, and failing_since
// past the cutoff — the same constant stale reads; unverified is the
// Reader-attributed subset of the same test. A failure inside the window is
// in neither — one bad fetch is not a fault to correct — and a Series that
// never captured a chapter is never failing, the exact complement of
// never-read-a-chapter's IS NULL half, so the two filters are disjoint by
// construction. A finished failing Series still appears: this pair reads
// stored outcomes, which simply stop arriving, and carries no finished guard.
func TestAdminFailingAndUnverifiedFilters(t *testing.T) {
s := newTestStore(t)
const cutoff = 5000
seedAdminSeries(t, s, seriesSeed{key: "asura:failing", url: "u", cover: "c", checkedAt: 9000, latestNum: new(10.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:recent", url: "u", cover: "c", checkedAt: 9000, latestNum: new(9.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:nochapter", url: "u", cover: "c", checkedAt: 9000, bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:reader", url: "u", cover: "c", checkedAt: 9000, latestNum: new(8.0), bookmarks: 1, raisedBy: true})
seedAdminSeries(t, s, seriesSeed{key: "asura:polled", url: "u", cover: "c", checkedAt: 9000, latestNum: new(7.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-failing", url: "u", cover: "c", checkedAt: 9000, latestNum: new(6.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:healthy", url: "u", cover: "c", checkedAt: 9000, latestNum: new(5.0), bookmarks: 1})
// Failure rows seed the run's start stamp; the cutoff decides the age.
for _, f := range []struct {
key, word string
since int64
}{
{"asura:failing", "not_found", 2000},
{"asura:recent", "not_found", 9000},
{"asura:nochapter", "not_found", 2000},
{"asura:reader", "not_found", 2000},
{"asura:polled", "errors", 2000},
{"asura:fin-failing", "not_found", 2000},
} {
site, id, _ := strings.Cut(f.key, ":")
if err := s.RecordSeriesFailure(site, id, f.word, f.since); err != nil {
t.Fatalf("seed failure %s: %v", f.key, err)
}
}
if err := s.SetSeriesFinished("asura", "fin-failing", 1000); err != nil {
t.Fatalf("finish asura:fin-failing: %v", err)
}
cases := []struct {
name string
f SeriesFilter
want []string
}{
{"failing", SeriesFilter{Name: SeriesFilterFailing, Cutoff: cutoff}, []string{"asura:failing", "asura:reader", "asura:polled", "asura:fin-failing"}},
{"unverified", SeriesFilter{Name: SeriesFilterUnverified, Cutoff: cutoff}, []string{"asura:reader"}},
{"never read a chapter", SeriesFilter{Name: SeriesFilterNoChapter}, []string{"asura:nochapter"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := pageKeys(t, s, tc.f)
want := map[string]bool{}
for _, k := range tc.want {
want[k] = true
}
if len(got) != len(want) {
t.Fatalf("%+v returned %v, want exactly %v", tc.f, got, want)
}
for k := range want {
if !got[k] {
t.Fatalf("%+v dropped %q (got %v)", tc.f, k, got)
}
}
})
}
// Disjointness: the failing pair and never-read-a-chapter are disjoint by
// the chapter column — IS NOT NULL here, IS NULL there — so no row may be
// counted under both.
failing := pageKeys(t, s, SeriesFilter{Name: SeriesFilterFailing, Cutoff: cutoff})
noChapter := pageKeys(t, s, SeriesFilter{Name: SeriesFilterNoChapter})
for k := range failing {
if noChapter[k] {
t.Fatalf("row %q matches both failing and never-read-a-chapter", k)
}
}
if failing["asura:nochapter"] {
t.Fatal("a Series with no chapter ever captured appears in failing")
}
if !noChapter["asura:nochapter"] {
t.Fatal("the no-chapter Series vanished from never-read-a-chapter")
}
// The aggregates count the same rows the lists do: the landing page's
// figures and the select's options come from these two passes.
for _, name := range []string{SeriesFilterFailing, SeriesFilterUnverified} {
shapes, err := s.SeriesShapes(SeriesFilter{Name: name, Cutoff: cutoff})
if err != nil {
t.Fatalf("SeriesShapes(%s): %v", name, err)
}
sum := 0
for _, sh := range shapes {
sum += sh.Total
}
want := len(pageKeys(t, s, SeriesFilter{Name: name, Cutoff: cutoff}))
if sum != want {
t.Fatalf("%s aggregate sum = %d, want %d (aggregate disagrees with row query)", name, sum, want)
}
}
}
// The projection carries the failure facts from the LEFT JOIN, not the
// Series row: a failing Series reads back its outcome word and the stamp its
// run began at; a Series with no failure row reads empty and zero. Nothing
// new is projected from the Series row itself.
func TestAdminFailureProjection(t *testing.T) {
s := newTestStore(t)
seedAdminSeries(t, s, seriesSeed{key: "asura:broken", url: "u", cover: "c", checkedAt: 9000, latestNum: new(9.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fine", url: "u", cover: "c", checkedAt: 9000, latestNum: new(8.0), bookmarks: 1})
if err := s.RecordSeriesFailure("asura", "broken", "not_found", 2000); err != nil {
t.Fatalf("seed failure row: %v", err)
}
page, err := s.SeriesPage(SeriesFilter{})
if err != nil {
t.Fatalf("SeriesPage: %v", err)
}
byKey := map[string]AdminSeries{}
for _, a := range page.Rows {
byKey[a.Key()] = a
}
if byKey["asura:broken"].FailureOutcome != "not_found" || byKey["asura:broken"].FailingSince != 2000 {
t.Fatalf("broken row = %+v, want FailureOutcome not_found, FailingSince 2000", byKey["asura:broken"])
}
if byKey["asura:fine"].FailureOutcome != "" || byKey["asura:fine"].FailingSince != 0 {
t.Fatalf("fine row = %+v, want empty outcome and zero stamp", byKey["asura:fine"])
}
}
// The site-completed filter (issue #170) lists the Site's own marker as work
// to work through, carrying the same finished guard the clock-driven
// predicates gained: the Site's stamp keeps standing after the owner retires
// the row, so a finished Series would be reported as work nobody is going to
// do. A zero stamp never appears. Un-finishing puts the Series back in the
// Poll query — the finished_at gate is #157's own, asserted through
// DueForLatestCheck rather than re-derived here.
func TestAdminSiteCompletedFilter(t *testing.T) {
s := newTestStore(t)
seedAdminSeries(t, s, seriesSeed{key: "asura:done", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1, siteCompletedAt: 5000})
seedAdminSeries(t, s, seriesSeed{key: "asura:never", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:retired", url: "u", cover: "c", checkedAt: 0, latestNum: new(4.0), bookmarks: 1, siteCompletedAt: 5000})
seedAdminSeries(t, s, seriesSeed{key: "asura:healthy", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
if err := s.SetSeriesFinished("asura", "retired", 1000); err != nil {
t.Fatalf("finish asura:retired: %v", err)
}
got := pageKeys(t, s, SeriesFilter{Name: SeriesFilterSiteCompleted})
if len(got) != 1 || !got["asura:done"] {
t.Fatalf("site-completed returned %v, want only asura:done", got)
}
// The projection carries the stamp so the detail page can age it.
page, err := s.SeriesPage(SeriesFilter{Name: SeriesFilterSiteCompleted})
if err != nil {
t.Fatalf("SeriesPage(site_completed): %v", err)
}
if len(page.Rows) != 1 || page.Rows[0].SiteCompletedAt != 5000 {
t.Fatalf("row = %+v, want SiteCompletedAt 5000", page.Rows)
}
// The aggregate counts the same row: the landing figure and the select's
// option come from this pass, so they cannot disagree with the list.
shapes, err := s.SeriesShapes(SeriesFilter{Name: SeriesFilterSiteCompleted})
if err != nil {
t.Fatalf("SeriesShapes(site_completed): %v", err)
}
sum := 0
for _, sh := range shapes {
sum += sh.Total
}
if sum != 1 {
t.Fatalf("site-completed aggregate = %d, want 1", sum)
}
// Un-finishing puts the Series back in the Poll query: the due read's
// finished_at gate (issue #157) admits it again — asserted through the
// Lane's own read, not re-derived here.
due, err := s.DueForLatestCheck("asura", 1000, noCeiling)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
for _, sr := range due {
if sr.Key() == "asura:retired" {
t.Fatalf("a finished Series is still due for a Poll:\n%+v", due)
}
}
if err := s.SetSeriesFinished("asura", "retired", 0); err != nil {
t.Fatalf("un-finish asura:retired: %v", err)
}
due, err = s.DueForLatestCheck("asura", 1000, noCeiling)
if err != nil {
t.Fatalf("DueForLatestCheck after un-finish: %v", err)
}
found := false
for _, sr := range due {
if sr.Key() == "asura:retired" {
found = true
}
}
if !found {
t.Fatalf("un-finished Series is not due for a Poll:\n%+v", due)
}
}
@@ -0,0 +1,4 @@
-- A 4xx other than the 403 refusal is the page being gone, not the Site being
-- unwell; the pass log counts it separately so the Lanes page can say "not
-- found" (#164). DEFAULT 0 keeps pre-existing rows readable.
ALTER TABLE poll_passes ADD COLUMN not_found integer NOT NULL DEFAULT 0;
@@ -0,0 +1,13 @@
-- One row per Series that is failing right now (ADR-0016): the row's
-- existence is the failure state, failing_since ages the run of failures,
-- and a correct read deletes the row. Keyed by the same (site, series_id)
-- composite the rest of the system uses, with the cascade so deleting a
-- Series takes its failure row and orphan removal stays a single statement.
CREATE TABLE poll_failures (
site text NOT NULL,
series_id text NOT NULL,
outcome text NOT NULL,
failing_since bigint NOT NULL,
PRIMARY KEY (site, series_id),
FOREIGN KEY (site, series_id) REFERENCES series (site, series_id) ON DELETE CASCADE
);
@@ -0,0 +1,4 @@
-- When the last successful Poll read saw the Site's own completed value
-- (#168): epoch-ms, zero meaning it did not. DEFAULT 0 keeps pre-existing
-- rows readable.
ALTER TABLE series ADD COLUMN site_completed_at bigint NOT NULL DEFAULT 0;
@@ -0,0 +1,12 @@
-- Owner notices (issue #171): one row per episode, remembered only as "the
-- owner was told". The row's presence is the whole state — the poller checks
-- it before sending, writes it after a successful send, and clears it when
-- the condition no longer holds. site is '' for a fault that is not one
-- Site's; the composite primary key is what makes the row a lock against a
-- second message for the same episode.
CREATE TABLE owner_notices (
condition text NOT NULL,
site text NOT NULL,
notified_at bigint NOT NULL,
PRIMARY KEY (condition, site)
);
+195 -13
View File
@@ -84,6 +84,10 @@ type Series struct {
LatestChapter string
LatestChapterNum *float64 // nil until first captured
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
// SiteCompletedAt is when the last successful Poll read saw the Site's
// own completed value, zero meaning it did not (issue #168). A poller
// fact, not reading progress: Upsert never touches it.
SiteCompletedAt int64
// LatestRaisedBy is the Reader whose Sighting last raised LatestChapter,
// and nil when the stored value is a Poll's own finding. It is what lets a
// Poll that contradicts the value downwards name a Reader instead of
@@ -110,7 +114,7 @@ type LanePass struct {
GapMS int64
Clamped bool
Refused, Unreachable, NoChapter int
Unfetchable, Errors int
Unfetchable, Errors, NotFound int
PausedUntil, RefuseUntil int64
}
@@ -119,7 +123,7 @@ type LanePass struct {
type SiteOutcomes struct {
Site string
Refused, Unreachable, NoChapter int
Unfetchable, Errors int
Unfetchable, Errors, NotFound int
}
// LanePause is one persisted Lane pause stamp.
@@ -240,10 +244,11 @@ const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover_add
// due query. latest_checked_at lives only on series — see MarkLatestChecked
// for why it stays off every client-visible write.
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.latest_raised_by`
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.latest_raised_by,
s.site_completed_at`
const lanePassColumns = `p.site, p.ran_at, p.skip, p.due, p.checked, p.gap_ms, p.clamped,
p.refused, p.unreachable, p.no_chapter, p.unfetchable, p.errors,
p.refused, p.unreachable, p.no_chapter, p.unfetchable, p.errors, p.not_found,
COALESCE(l.paused_until, 0), COALESCE(l.refuse_until, 0)`
// Owner is the person running the service: the first Reader, seeded at startup
@@ -646,7 +651,7 @@ func scanSeries(scan func(...any) error) (Series, error) {
if err := scan(
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover, &sr.CoverAddress,
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt, &latestRaisedBy,
&sr.Forced, &sr.readerCount,
&sr.SiteCompletedAt, &sr.Forced, &sr.readerCount,
); err != nil {
return Series{}, err
}
@@ -663,7 +668,7 @@ func scanLanePass(scan func(...any) error) (LanePass, error) {
var p LanePass
if err := scan(
&p.Site, &p.RanAt, &p.Skip, &p.Due, &p.Checked, &p.GapMS, &p.Clamped,
&p.Refused, &p.Unreachable, &p.NoChapter, &p.Unfetchable, &p.Errors,
&p.Refused, &p.Unreachable, &p.NoChapter, &p.Unfetchable, &p.Errors, &p.NotFound,
&p.PausedUntil, &p.RefuseUntil,
); err != nil {
return LanePass{}, err
@@ -1131,10 +1136,10 @@ func (s *Store) RecordLanePass(p LanePass, retainBefore int64) error {
if _, err := tx.Exec(`
INSERT INTO poll_passes
(site, ran_at, skip, due, checked, gap_ms, clamped,
refused, unreachable, no_chapter, unfetchable, errors)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)`,
refused, unreachable, no_chapter, unfetchable, errors, not_found)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)`,
p.Site, p.RanAt, p.Skip, p.Due, p.Checked, p.GapMS, p.Clamped,
p.Refused, p.Unreachable, p.NoChapter, p.Unfetchable, p.Errors); err != nil {
p.Refused, p.Unreachable, p.NoChapter, p.Unfetchable, p.Errors, p.NotFound); err != nil {
return fmt.Errorf("insert lane pass %s at %d: %w", p.Site, p.RanAt, err)
}
if _, err := tx.Exec(`DELETE FROM poll_passes WHERE ran_at < $1`, retainBefore); err != nil {
@@ -1171,7 +1176,7 @@ func (s *Store) LatestLanePasses() ([]LanePass, error) {
FROM (
SELECT DISTINCT ON (site)
site, ran_at, skip, due, checked, gap_ms, clamped,
refused, unreachable, no_chapter, unfetchable, errors
refused, unreachable, no_chapter, unfetchable, errors, not_found
FROM poll_passes
ORDER BY site, ran_at DESC
) p
@@ -1198,7 +1203,7 @@ func (s *Store) LatestLanePasses() ([]LanePass, error) {
func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
rows, err := s.db.Query(`
SELECT site, SUM(refused), SUM(unreachable), SUM(no_chapter),
SUM(unfetchable), SUM(errors)
SUM(unfetchable), SUM(errors), SUM(not_found)
FROM poll_passes
WHERE ran_at >= $1
GROUP BY site
@@ -1213,7 +1218,7 @@ func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
var outcomes SiteOutcomes
if err := rows.Scan(
&outcomes.Site, &outcomes.Refused, &outcomes.Unreachable,
&outcomes.NoChapter, &outcomes.Unfetchable, &outcomes.Errors,
&outcomes.NoChapter, &outcomes.Unfetchable, &outcomes.Errors, &outcomes.NotFound,
); err != nil {
return nil, fmt.Errorf("scan lane pass outcomes: %w", err)
}
@@ -1222,6 +1227,101 @@ func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
return out, rows.Err()
}
// RefusingSince reports, per Site, when that Site's current unbroken run of
// refusing passes began: a pass refuses when the Lane gate returned early
// (skip = 'refusing') or the pass loop counted refusals (refused > 0), so a
// Site parked in refusal backoff keeps its run unbroken. A Site whose
// latest pass did not refuse is absent. The run is bounded by the pass
// log's retention — a run older than the log answers with the oldest row
// present — and nothing after now counts: the cutoff is the caller's clock.
func (s *Store) RefusingSince(now int64) (map[string]int64, error) {
rows, err := s.db.Query(`
SELECT site, MIN(ran_at)
FROM poll_passes p
WHERE ran_at <= $1
AND (refused > 0 OR skip = 'refusing')
AND ran_at > COALESCE((
SELECT MAX(q.ran_at) FROM poll_passes q
WHERE q.site = p.site AND q.ran_at <= $1
AND NOT (q.refused > 0 OR q.skip = 'refusing')
), 0)
GROUP BY site`, now)
if err != nil {
return nil, fmt.Errorf("query refusing since: %w", err)
}
defer rows.Close()
out := map[string]int64{}
for rows.Next() {
var site string
var since int64
if err := rows.Scan(&site, &since); err != nil {
return nil, fmt.Errorf("scan refusing since: %w", err)
}
out[site] = since
}
return out, rows.Err()
}
// SidecarOK reports, per Site in sites, the unix ms of that Site's most
// recent pass that actually reached the sidecar: the pass ran its loop
// (skip = '') and no Series read lost Chrome (unreachable = 0) — a
// challenge answer still reached the sidecar, a lost sidecar did not. A
// Site with no such pass is absent: an asleep Lane never reached it, so it
// is absent too and cannot age into a sidecar-down alarm by itself.
func (s *Store) SidecarOK(sites []string) (map[string]int64, error) {
rows, err := s.db.Query(`
SELECT DISTINCT ON (site) site, ran_at
FROM poll_passes
WHERE site = ANY($1) AND skip = '' AND unreachable = 0
ORDER BY site, ran_at DESC`, sites)
if err != nil {
return nil, fmt.Errorf("query sidecar ok: %w", err)
}
defer rows.Close()
out := map[string]int64{}
for rows.Next() {
var site string
var ranAt int64
if err := rows.Scan(&site, &ranAt); err != nil {
return nil, fmt.Errorf("scan sidecar ok: %w", err)
}
out[site] = ranAt
}
return out, rows.Err()
}
// NoChapterShare reports, per Site, the share of that Site's Series holding
// a no-chapter failure row older than cutoff: old rows over the Site's
// whole Series count, so a four-Series Site and a 200-Series Site are
// judged on the same scale. A Site with no Series has no share and is
// absent.
func (s *Store) NoChapterShare(cutoff int64) (map[string]float64, error) {
rows, err := s.db.Query(`
SELECT s.site,
(COUNT(*) FILTER (WHERE f.outcome = 'no_chapter' AND f.failing_since < $1))::float8
/ (COUNT(*)::float8)
FROM series s
LEFT JOIN poll_failures f ON f.site = s.site AND f.series_id = s.series_id
GROUP BY s.site
ORDER BY s.site`, cutoff)
if err != nil {
return nil, fmt.Errorf("query no-chapter share: %w", err)
}
defer rows.Close()
out := map[string]float64{}
for rows.Next() {
var site string
var share float64
if err := rows.Scan(&site, &share); err != nil {
return nil, fmt.Errorf("scan no-chapter share: %w", err)
}
out[site] = share
}
return out, rows.Err()
}
// SetLaneRefusal persists a Site's refusal backoff stamp without touching its
// pause. until is supplied by the caller's clock.
func (s *Store) SetLaneRefusal(site string, until int64) error {
@@ -1298,6 +1398,72 @@ func (s *Store) LaneGates(site string) (pausedUntil, refuseUntil int64, err erro
return pausedUntil, refuseUntil, nil
}
// RecordSeriesFailure upserts one Series' failure row: the outcome word is
// updated when it changes, failing_since is never overwritten, and an
// unchanged word writes nothing. One statement, so the identical-word no-op
// and the keep-the-stamp rule are the same guarantee: the SET arm fires only
// when the word differs, and failing_since is simply absent from it
// (ADR-0016).
func (s *Store) RecordSeriesFailure(site, seriesID, outcome string, now int64) error {
if _, err := s.db.Exec(`
INSERT INTO poll_failures (site, series_id, outcome, failing_since) VALUES ($1, $2, $3, $4)
ON CONFLICT (site, series_id) DO UPDATE SET outcome = EXCLUDED.outcome
WHERE poll_failures.outcome <> EXCLUDED.outcome`, site, seriesID, outcome, now); err != nil {
return fmt.Errorf("record series failure %s:%s: %w", site, seriesID, err)
}
return nil
}
// ClearSeriesFailure deletes one Series' failure row. A row that is not there
// is not an error.
func (s *Store) ClearSeriesFailure(site, seriesID string) error {
if _, err := s.db.Exec(
`DELETE FROM poll_failures WHERE site = $1 AND series_id = $2`, site, seriesID); err != nil {
return fmt.Errorf("clear series failure %s:%s: %w", site, seriesID, err)
}
return nil
}
// NoticeSent reports whether the owner has already been told about this
// episode (issue #171). The row's presence is the whole state, so a missing
// row reads false without error.
func (s *Store) NoticeSent(condition, site string) (bool, error) {
var at int64
err := s.db.QueryRow(
`SELECT notified_at FROM owner_notices WHERE condition = $1 AND site = $2`,
condition, site).Scan(&at)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("notice sent %s/%s: %w", condition, site, err)
}
return true, nil
}
// MarkNoticeSent records that the owner was told. Called only after a
// successful send. Re-marking the same episode just moves the stamp: the row
// is a lock against a second message, not a log.
func (s *Store) MarkNoticeSent(condition, site string, at int64) error {
if _, err := s.db.Exec(`
INSERT INTO owner_notices (condition, site, notified_at) VALUES ($1, $2, $3)
ON CONFLICT (condition, site) DO UPDATE SET notified_at = EXCLUDED.notified_at`,
condition, site, at); err != nil {
return fmt.Errorf("mark notice sent %s/%s: %w", condition, site, err)
}
return nil
}
// ClearNotice forgets an episode, so the condition's next occurrence sends
// again. A row that is not there is not an error.
func (s *Store) ClearNotice(condition, site string) error {
if _, err := s.db.Exec(
`DELETE FROM owner_notices WHERE condition = $1 AND site = $2`, condition, site); err != nil {
return fmt.Errorf("clear notice %s/%s: %w", condition, site, err)
}
return nil
}
// DueForLatestCheck returns one Site's series whose server-side
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks
// reference them (descending) then least-recently-checked first. One Site per
@@ -1356,7 +1522,7 @@ func (s *Store) DueForLatestCheck(site string, cutoffMs, ceilingMs int64) ([]Ser
AND (s.finished_at = 0 OR s.force_poll_at > s.latest_checked_at)
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
s.force_poll_at, s.finished_at
s.site_completed_at, s.force_poll_at, s.finished_at
HAVING (COUNT(*) > 1
OR s.latest_sighted_at <= $2::bigint
OR s.latest_checked_at <= $3::bigint
@@ -1491,6 +1657,22 @@ func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) erro
return nil
}
// SetSiteCompletedAt stores when the last successful read saw the Site's
// completed value, zero meaning it did not. Series-level, like the Latest
// Chapter: it is a fact about the work, not about one Reader's bookmark, and
// the bookmark's updated_at is never touched — this is not reading progress
// and must not reorder any Reader's list, the same rule SetLatestChapter's
// comment states. Touching a missing series is not an error: the row may
// have been orphaned, and the caller's read decides what exists.
func (s *Store) SetSiteCompletedAt(site, seriesID string, at int64) error {
if _, err := s.db.Exec(
`UPDATE series SET site_completed_at = $1 WHERE site = $2 AND series_id = $3`,
at, site, seriesID); err != nil {
return fmt.Errorf("set site completed %s:%s: %w", site, seriesID, err)
}
return nil
}
// SetSeriesURL stores the owner's repair for a Series' source address
// (issue #151): the one write that lifts the write-once rule documented on
// Series.SeriesURL. It is a store, not a verification — the caller has
+286
View File
@@ -7,6 +7,7 @@ import (
"encoding/hex"
"errors"
"io/fs"
"fmt"
"os"
"path/filepath"
"strconv"
@@ -2611,3 +2612,288 @@ func TestRemoveSeriesMissingKeyIsCleanNoOp(t *testing.T) {
t.Fatalf("RemoveSeries on a missing key = %v, want nil", err)
}
}
// The failure row's existence is the failure state (ADR-0016):
// RecordSeriesFailure upserts the outcome word, keeps the original
// failing_since through a word change, and an unchanged word writes nothing
// at all — a broken Series costs the same as a healthy one every hour.
func TestRecordSeriesFailureUpsertKeepsFailingSince(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := s.RecordSeriesFailure("asura", "solo", "not_found", 1000); err != nil {
t.Fatalf("record: %v", err)
}
word, since, found := failureRow(t, s, "asura", "solo")
if !found || word != "not_found" || since != 1000 {
t.Fatalf("row after create = (%q, %d, %v), want (not_found, 1000, true)", word, since, found)
}
// A changed word updates the outcome and never touches failing_since.
if err := s.RecordSeriesFailure("asura", "solo", "errors", 2000); err != nil {
t.Fatalf("record changed word: %v", err)
}
word, since, found = failureRow(t, s, "asura", "solo")
if !found || word != "errors" || since != 1000 {
t.Fatalf("row after word change = (%q, %d, %v), want (errors, 1000, true)", word, since, found)
}
// An identical word writes nothing: the stamp survives a later now.
if err := s.RecordSeriesFailure("asura", "solo", "errors", 3000); err != nil {
t.Fatalf("record identical word: %v", err)
}
word, since, found = failureRow(t, s, "asura", "solo")
if !found || word != "errors" || since != 1000 {
t.Fatalf("row after identical word = (%q, %d, %v), want the earlier (errors, 1000, true)", word, since, found)
}
}
// ClearSeriesFailure deletes the row, and a row that is not there is not an
// error — the poller clears on every successful read, so most clears find
// nothing.
func TestClearSeriesFailure(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := s.ClearSeriesFailure("asura", "solo"); err != nil {
t.Fatalf("clear a missing row: %v, want nil", err)
}
if err := s.RecordSeriesFailure("asura", "solo", "not_found", 1000); err != nil {
t.Fatalf("record: %v", err)
}
if err := s.ClearSeriesFailure("asura", "solo"); err != nil {
t.Fatalf("clear: %v", err)
}
if _, _, found := failureRow(t, s, "asura", "solo"); found {
t.Fatal("row after clear: still present")
}
}
// Deleting a Series takes its failure row with it (the composite FK's
// cascade), so orphan removal stays a single statement.
func TestRemoveSeriesCascadesToFailureRow(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := s.RecordSeriesFailure("asura", "solo", "not_found", 1000); err != nil {
t.Fatalf("record: %v", err)
}
if err := s.Delete(s.OwnerID(), "asura:solo"); err != nil {
t.Fatalf("delete bookmark: %v", err)
}
if err := s.RemoveSeries("asura", "solo"); err != nil {
t.Fatalf("RemoveSeries: %v", err)
}
if _, _, found := failureRow(t, s, "asura", "solo"); found {
t.Fatal("failure row after series delete: still present (no cascade)")
}
}
// failureRow reads one Series' failure row as stored.
func failureRow(t *testing.T, s *Store, site, seriesID string) (word string, since int64, found bool) {
t.Helper()
err := s.db.QueryRow(
`SELECT outcome, failing_since FROM poll_failures WHERE site = $1 AND series_id = $2`,
site, seriesID).Scan(&word, &since)
if errors.Is(err, sql.ErrNoRows) {
return "", 0, false
}
if err != nil {
t.Fatalf("read failure row %s:%s: %v", site, seriesID, err)
}
return word, since, true
}
// Owner notices are one row per episode: the row's presence is the whole
// state. MarkNoticeSent stamps it, NoticeSent reads it, and ClearNotice
// forgets it — including a row that was never there, which is not an error.
func TestOwnerNoticeRows(t *testing.T) {
s := newTestStore(t)
if sent, err := s.NoticeSent("stall", "comix"); err != nil || sent {
t.Fatalf("NoticeSent on a fresh table = %v, %v; want false, nil", sent, err)
}
if err := s.MarkNoticeSent("stall", "comix", 4242); err != nil {
t.Fatalf("MarkNoticeSent: %v", err)
}
if sent, err := s.NoticeSent("stall", "comix"); err != nil || !sent {
t.Fatalf("NoticeSent after mark = %v, %v; want true, nil", sent, err)
}
// Re-marking the same episode just moves the stamp: the row is a lock,
// not a log. A different site is its own episode — the key is
// (condition, site).
if err := s.MarkNoticeSent("stall", "comix", 4343); err != nil {
t.Fatalf("re-mark: %v", err)
}
if sent, err := s.NoticeSent("stall", "kagane"); err != nil || sent {
t.Fatalf("NoticeSent on another site = %v, %v; want false, nil", sent, err)
}
if err := s.ClearNotice("stall", "comix"); err != nil {
t.Fatalf("ClearNotice: %v", err)
}
if sent, err := s.NoticeSent("stall", "comix"); err != nil || sent {
t.Fatalf("NoticeSent after clear = %v, %v; want false, nil", sent, err)
}
if err := s.ClearNotice("stall", "comix"); err != nil {
t.Fatalf("ClearNotice on a missing row: %v", err)
}
}
// RefusingSince reads one Site's current unbroken run of refusing passes
// (issue #172). A refusing pass is one the Lane gate returned early from
// (skip 'refusing') or one whose loop counted refusals (refused > 0) — the
// two shapes a persistently-challenged Site alternates between, so the run
// must not break when the gate row follows the refusal row. A Site whose
// latest pass did not refuse is absent, nothing after the caller's clock
// counts, and a run older than the log answers with the oldest row present.
func TestRefusingSince(t *testing.T) {
s := newTestStore(t)
seed := func(site string, ranAt int64, skip string, refused int) {
t.Helper()
if err := s.RecordLanePass(LanePass{Site: site, RanAt: ranAt, Skip: skip, Refused: refused}, -1); err != nil {
t.Fatalf("RecordLanePass(%s/%d): %v", site, ranAt, err)
}
}
// asura: a refusing run broken by a healthy pass, then resumed; the
// current run began at the pass after the break.
seed("asura", 100, "", 2)
seed("asura", 200, "", 0)
seed("asura", 300, "refusing", 0)
seed("asura", 400, "", 2)
// demonic: the latest pass is healthy — no run, absent.
seed("demonic", 100, "", 2)
seed("demonic", 200, "", 0)
// novelfull: a healthy pass after now must not break the run — the run
// is judged as of the caller's clock.
seed("novelfull", 100, "", 2)
seed("novelfull", 600, "", 0)
// comix: an unbroken run reaches back to the oldest row present.
seed("comix", 100, "", 2)
seed("comix", 200, "refusing", 0)
got, err := s.RefusingSince(450)
if err != nil {
t.Fatalf("RefusingSince: %v", err)
}
want := map[string]int64{"asura": 300, "novelfull": 100, "comix": 100}
if len(got) != len(want) {
t.Fatalf("RefusingSince = %v, want %v", got, want)
}
for site, since := range want {
if got[site] != since {
t.Fatalf("RefusingSince[%s] = %d, want %d (got %v)", site, got[site], since, got)
}
}
if _, ok := got["demonic"]; ok {
t.Fatalf("RefusingSince names demonic, whose latest pass did not refuse: %v", got)
}
}
// SidecarOK reads, per Site, the most recent pass that actually reached the
// sidecar (issue #172): the pass ran its loop (skip '') and no read lost
// Chrome (unreachable 0). A challenge answer still reached the sidecar, a
// lost sidecar and every skip value did not, and a Site with no qualifying
// pass — an asleep Lane included — is absent.
func TestSidecarOK(t *testing.T) {
s := newTestStore(t)
seed := func(site string, ranAt int64, skip string, unreachable int) {
t.Helper()
if err := s.RecordLanePass(LanePass{Site: site, RanAt: ranAt, Skip: skip, Unreachable: unreachable}, -1); err != nil {
t.Fatalf("RecordLanePass(%s/%d): %v", site, ranAt, err)
}
}
// comix: only the skip='' unreachable=0 pass reached the sidecar; the
// mid-loop browser-loss row (skip '', unreachable > 0) and the skip
// values never did.
seed("comix", 100, "", 1)
seed("comix", 200, "sidecar-down", 0)
seed("comix", 300, "", 0)
seed("comix", 400, "refusing", 0)
// kagane: two passes reached the sidecar; the newest wins.
seed("kagane", 150, "", 0)
seed("kagane", 250, "", 0)
// novelfull: an asleep Lane never reached it.
seed("novelfull", 120, "asleep", 0)
got, err := s.SidecarOK([]string{"comix", "kagane", "novelfull", "lightnovelworld"})
if err != nil {
t.Fatalf("SidecarOK: %v", err)
}
want := map[string]int64{"comix": 300, "kagane": 250}
if len(got) != len(want) {
t.Fatalf("SidecarOK = %v, want %v", got, want)
}
for site, ranAt := range want {
if got[site] != ranAt {
t.Fatalf("SidecarOK[%s] = %d, want %d (got %v)", site, got[site], ranAt, got)
}
}
for _, site := range []string{"novelfull", "lightnovelworld"} {
if _, ok := got[site]; ok {
t.Fatalf("SidecarOK names %s, which never reached the sidecar: %v", site, got)
}
}
}
// NoChapterShare reads, per Site, the share of its Series holding a
// no-chapter failure row older than the cutoff (issue #172): old rows over
// the Site's whole Series count, so a four-Series Site and a 200-Series
// Site are judged on the same scale. A fresh no-chapter row and a row of
// any other outcome do not count.
func TestNoChapterShare(t *testing.T) {
s := newTestStore(t)
for i := 1; i <= 4; i++ {
seedForCheck(t, s, fmt.Sprintf("asura:a%d", i), fmt.Sprintf("https://asurascans.com/comics/a%d", i), 0)
}
seedForCheck(t, s, "comix:c1", "https://comix.to/title/c1", 0)
seedForCheck(t, s, "comix:c2", "https://comix.to/title/c2", 0)
seedForCheck(t, s, "demonic:d1", "https://demonicscans.org/series/d1", 0)
for i := 1; i <= 200; i++ {
seedForCheck(t, s, fmt.Sprintf("novelfull:n%d", i), fmt.Sprintf("https://novelfull.com/n%d.html", i), 0)
}
const cutoff = 1000
oldNoChapter := func(site, seriesID string) {
t.Helper()
if err := s.RecordSeriesFailure(site, seriesID, "no_chapter", 100); err != nil {
t.Fatalf("seed no-chapter failure %s:%s: %v", site, seriesID, err)
}
}
// asura: three of four hold an old no-chapter row; the fourth's
// no-chapter row is fresh — it is not old, so it does not count.
oldNoChapter("asura", "a1")
oldNoChapter("asura", "a2")
oldNoChapter("asura", "a3")
if err := s.RecordSeriesFailure("asura", "a4", "no_chapter", 2000); err != nil {
t.Fatalf("seed fresh no-chapter failure: %v", err)
}
// comix: an old no-chapter row and an old errors row — the errors row
// is not a no-chapter row, so the share is 1/2, the exact boundary.
oldNoChapter("comix", "c1")
if err := s.RecordSeriesFailure("comix", "c2", "errors", 100); err != nil {
t.Fatalf("seed errors failure: %v", err)
}
// demonic and novelfull: one old no-chapter row each, against sites of
// one and two hundred Series.
oldNoChapter("demonic", "d1")
oldNoChapter("novelfull", "n1")
got, err := s.NoChapterShare(cutoff)
if err != nil {
t.Fatalf("NoChapterShare: %v", err)
}
checks := []struct {
site string
want float64
}{
{"asura", 3.0 / 4.0},
{"comix", 1.0 / 2.0},
{"demonic", 1.0 / 1.0},
{"novelfull", 1.0 / 200.0},
}
for _, c := range checks {
if got[c.site] != c.want {
t.Fatalf("NoChapterShare[%s] = %v, want %v", c.site, got[c.site], c.want)
}
}
}
+5 -3
View File
@@ -4,14 +4,16 @@ import (
"log"
"net/http"
"strconv"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store"
)
// ownerWindow is the staleness boundary the Series list's "not checked in
// 12h" filter compares against. Declared once; later admin tickets read it.
const ownerWindow = 12 * time.Hour
// 12h" filter compares against. It reads latest.OwnerWindow — the one place
// the class-level twelve hours lives, shared with the owner-notice
// conditions (issue #171).
const ownerWindow = latest.OwnerWindow
// adminView is the shared shell data for an administrative page and the roster
// fragment returned after a Reader action.
+18 -6
View File
@@ -56,6 +56,10 @@ type laneRow struct {
// offer Resume from the moment the owner presses Pause, with no pass
// having run to record it (issue #147).
Paused bool
// FailingHref is the one navigation the row offers: the Site's name
// links to that Site's failing Series. The chips beside it stay
// unlinked; the withdrawn promise lives on outcomeChips (issue #167).
FailingHref string
}
// chip is one named outcome count over the owner's window.
@@ -223,6 +227,7 @@ func buildLaneRow(p store.LanePass, o store.SiteOutcomes, now time.Time) laneRow
if p.GapMS > 0 {
row.Gap = (time.Duration(p.GapMS) * time.Millisecond).Truncate(time.Second).String()
}
row.FailingHref = seriesListHref(store.SeriesFilterFailing, p.Site, "", 0)
row.Chips = outcomeChips(o)
row.HasChips = len(row.Chips) > 0
row.StatePhrase, row.StateGood, row.Attention = laneState(p, now)
@@ -232,17 +237,24 @@ func buildLaneRow(p store.LanePass, o store.SiteOutcomes, now time.Time) laneRow
// outcomeChips lists a Site's nonzero window sums in the taxonomy's fixed
// order, so the chips never reorder as the window changes. None observed is
// written by the template, not drawn as a confident zero count.
// written by the template, not drawn as a confident zero count. The names are
// spelled through outcomeWord, the one vocabulary the failure surface shares
// with the Series list's fact line (issue #167). The counts stay unlinked
// permanently — a withdrawn promise: two of the six words write no per-Series
// state, and the other four count attempts inside the owner window while the
// failing filter lists Series failing now, so neither set contains the other
// and no chip can be a door to its list.
func outcomeChips(o store.SiteOutcomes) []chip {
fixed := []struct {
name string
count int
}{
{"refused", o.Refused},
{"unreachable", o.Unreachable},
{"no chapter", o.NoChapter},
{"unfetchable", o.Unfetchable},
{"errors", o.Errors},
{outcomeWord("refused"), o.Refused},
{outcomeWord("unreachable"), o.Unreachable},
{outcomeWord("no_chapter"), o.NoChapter},
{outcomeWord("unfetchable"), o.Unfetchable},
{outcomeWord("not_found"), o.NotFound},
{outcomeWord("errors"), o.Errors},
}
var out []chip
for _, f := range fixed {
+54 -27
View File
@@ -2,8 +2,10 @@ package web
import (
"fmt"
"log"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store"
)
@@ -23,10 +25,10 @@ type overviewView struct {
// as stale + never_checked: a never-checked Series is already counted on
// its own filter, and the verdict wants the inclusive number.
Unchecked int
// Hygiene is the seven problem filters in the Series list's own render
// order plus the finished figure riding last (informational); Library is
// the library split plus the roster. Every figure is a door into the list
// that counts it, except a zero.
// Hygiene is the problem filters in the Series list's own render order
// plus the informational tail — finished, then site completed — riding
// last; Library is the library split plus the roster. Every figure is a
// door into the list that counts it, except a zero.
Hygiene []fig
Library []fig
// Sites is the per-Site library shape table, one row per Site with any
@@ -60,10 +62,13 @@ type siteRow struct {
}
// overviewView assembles the landing page from the store's read model: one
// SeriesShapes pass per filter summed in Go (the shipped surface offers nine
// SeriesShapes pass per filter summed in Go (the shipped surface offers ten
// grouped passes, not a stats query — #140), the pass log's latest pass per
// Site, and the roster. A failure in any read is a 500 with a logged reason,
// never a page of silent zeroes.
// Site, and the roster. A failure in the SeriesShapes, pass or roster reads
// is a 500 with a logged reason, never a page of silent zeroes. The three
// fault-input reads (RefusingSince, SidecarOK, NoChapterShare) fail open:
// a failing read logs and contributes no fault, so the landing page still
// renders — the same fail-open the poller uses for owner notices.
func (h *Handler) overviewView() (overviewView, error) {
now := time.Now()
cutoff := now.Add(-ownerWindow).UnixMilli()
@@ -92,12 +97,36 @@ func (h *Handler) overviewView() (overviewView, error) {
view := overviewView{Waiting: waiting(passes)}
view.Unchecked = totals[store.SeriesFilterStale] + totals[store.SeriesFilterNeverChecked]
view.Verdict, view.HasCounts = overviewVerdict(passes, now)
var refusingSince map[string]int64
if m, err := h.store.RefusingSince(now.UnixMilli()); err != nil {
log.Printf("admin overview: refusing since: %v", err)
} else {
refusingSince = m
}
var sidecarOK map[string]int64
if m, err := h.store.SidecarOK(latest.BrowserBackedSites()); err != nil {
log.Printf("admin overview: sidecar ok: %v", err)
} else {
sidecarOK = m
}
var noChapterShare map[string]float64
if m, err := h.store.NoChapterShare(cutoff); err != nil {
log.Printf("admin overview: no-chapter share: %v", err)
} else {
noChapterShare = m
}
faults := latest.FaultsFrom(latest.FaultInput{
Passes: passes,
RefusingSince: refusingSince,
SidecarOK: sidecarOK,
NoChapterShare: noChapterShare,
}, now)
view.Verdict, view.HasCounts = overviewVerdict(passes, faults)
// The hygiene figures, in seriesFilterOrder's tail: the seven problem
// filters in permanent-then-fixable order, then the finished figure last —
// informational, not a problem, and last because seriesFilterOrder appends
// it there. The All filter's count belongs to the Library block, not to a
// The hygiene figures, in seriesFilterOrder's tail: the problem filters
// in permanent-then-fixable order, then the informational tail — finished
// and site completed — riding last because seriesFilterOrder appends them
// there. The All filter's count belongs to the Library block, not to a
// "hygiene" figure.
hygiene := make([]fig, 0, len(seriesFilterOrder)-1)
for _, name := range seriesFilterOrder[1:] {
@@ -165,28 +194,26 @@ func door(label string, count int, href string) fig {
return fig{Label: label, Href: href, Count: count}
}
// overviewVerdict decides the landing page's one line from the latest pass
// per Site: no passes at all is "no Lane has reported yet" — never confident
// zeroes; otherwise the count of Lanes whose last pass needs the owner, or
// "all lanes healthy". The count comes from the same laneState judgement the
// Lanes page colours on, so the two pages cannot disagree on what a fault is.
func overviewVerdict(passes []store.LanePass, now time.Time) (phrase string, counts bool) {
// overviewVerdict decides the landing page's one line: no passes at all is
// "no Lane has reported yet" — never confident zeroes; otherwise the count of
// faults from the shared FaultsFrom judgement, so the page and the push
// cannot disagree. Zero faults is "all lanes healthy". A sidecar-down fault
// carries Site == "" and is still one fault. The Lanes page's per-row
// laneState is a different question (is this Lane's last pass healthy) from
// the notice class, and its known false positives live there deliberately, so
// the two now differ.
func overviewVerdict(passes []store.LanePass, faults []latest.Fault) (phrase string, counts bool) {
if len(passes) == 0 {
return "no Lane has reported yet", false
}
attention := 0
for _, p := range passes {
if _, _, attn := laneState(p, now); attn {
attention++
}
}
if attention == 0 {
n := len(faults)
if n == 0 {
return "all lanes healthy", true
}
if attention == 1 {
if n == 1 {
return "1 lane needs a look", true
}
return fmt.Sprintf("%d lanes need a look", attention), true
return fmt.Sprintf("%d lanes need a look", n), true
}
// waiting sums Due over the latest pass per Site: how many Series the Lanes
+41 -12
View File
@@ -25,21 +25,25 @@ const seriesPageSize = 50
// the labels are read by later admin tickets too, so the map and the
// constants cannot drift apart.
var seriesFilterLabels = map[string]string{
store.SeriesFilterAll: "All series",
store.SeriesFilterNoURL: "No series URL",
store.SeriesFilterNoChapter: "Never read a chapter",
store.SeriesFilterNoReaders: "No Readers",
store.SeriesFilterNeverChecked: "Never checked",
store.SeriesFilterStale: "Not checked in 12h",
store.SeriesFilterNoCover: "No cover",
store.SeriesFilterReaderReport: "Latest from a Reader",
store.SeriesFilterFinished: "Finished",
store.SeriesFilterAll: "All series",
store.SeriesFilterNoURL: "No series URL",
store.SeriesFilterNoChapter: "Never read a chapter",
store.SeriesFilterNoReaders: "No Readers",
store.SeriesFilterNeverChecked: "Never checked",
store.SeriesFilterStale: "Not checked in 12h",
store.SeriesFilterNoCover: "No cover",
store.SeriesFilterReaderReport: "Latest from a Reader",
store.SeriesFilterFailing: "Failing over 12h",
store.SeriesFilterUnverified: "Unverified Reader number",
store.SeriesFilterFinished: "Finished",
store.SeriesFilterSiteCompleted: "Site says completed",
}
// seriesFilterOrder is the select's render order: All first, then the
// permanent repairs, then the fixable ones (issue #140). Finished rides the
// tail, last — deliberate, not a repair — and the Overview's stats block
// renders the same tail, which is what sits the finished figure last there.
// permanent repairs, then the fixable ones (issue #140). Finished and the
// site-completed hint ride the tail — deliberate, not repairs — and the
// Overview's stats block renders the same tail, which is what sits the
// finished and site-completed figures last there.
var seriesFilterOrder = []string{
store.SeriesFilterAll,
store.SeriesFilterNoURL,
@@ -49,7 +53,10 @@ var seriesFilterOrder = []string{
store.SeriesFilterStale,
store.SeriesFilterNoCover,
store.SeriesFilterReaderReport,
store.SeriesFilterFailing,
store.SeriesFilterUnverified,
store.SeriesFilterFinished,
store.SeriesFilterSiteCompleted,
}
// seriesListView is the Series list page's data. The template renders strings
@@ -119,6 +126,11 @@ type seriesRowView struct {
// page, where a press that retires a Series from the Lane is on purpose
// and confirm-gated (issue #158).
Finished bool
// Failure names the standing failure and how long it has stood — "not
// found · 3d ago", "" while no failure row stands. Its own field, never a
// Notes chip: the chips cap at two plus a tail, so the one fact that
// names the failure would be the most likely to be truncated away.
Failure string
}
// adminSeries renders the filterable, bookmarkable Series list: filter, Site,
@@ -679,6 +691,9 @@ func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
row.Ch = "—"
}
row.Age = checkedAge(now, a.LatestCheckedAt)
if a.FailureOutcome != "" {
row.Failure = outcomeWord(a.FailureOutcome) + " · " + checkedAge(now, a.FailingSince)
}
notes := seriesNotes(a, now)
if n := len(notes); n > 2 {
row.Notes, row.More = notes[:2], n-2
@@ -689,6 +704,20 @@ func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
return row
}
// outcomeWord spells the wire failure word as the Lanes chips spell it — a
// space, not the underscore: not_found reads "not found", no_chapter "no
// chapter". The remaining words are their own spelling, so an unknown word
// degrades to itself rather than vanishing from the page.
func outcomeWord(wire string) string {
switch wire {
case "not_found":
return "not found"
case "no_chapter":
return "no chapter"
}
return wire
}
// seriesNotes are a row's hygiene chips in the design's order: no URL, no
// cover, orphan, stale, reader sighting.
func seriesNotes(a store.AdminSeries, now time.Time) []string {
@@ -64,6 +64,15 @@ type seriesDetailView struct {
// stands. It rides the meta fragment both presses swap, so the answer
// itself shows how long the Series has been finished.
FinishedSince string
// Unverified is the sentence beside the Latest Chapter correction
// control while a Reader's number stands behind a failure past the
// owner window: the value is unconfirmed and the owner should not trust
// it. It never names the Reader. "" otherwise.
Unverified string
// SiteCompleted is the hint's line, "" while the Site has said nothing or
// the owner has finished the Series: "the site says this work is
// completed (since 3d ago)". A hint, never a control (issue #170).
SiteCompleted string
}
// adminSeriesDetail renders one Series' page, keyed by the composite
@@ -151,6 +160,17 @@ func (h *Handler) seriesDetailView(a store.AdminSeries) seriesDetailView {
v.Corrected = correctedAge(time.Now(), a.LatestCorrectedAt)
v.Finished = a.FinishedAt != 0
v.FinishedSince = finishedAge(time.Now(), a.FinishedAt)
// Unverified: a Reader's number standing behind a failure that has outlived
// the owner window is a number nobody has re-checked since — say so next to
// the correction control, without naming the Reader. A machine read or a
// failure still inside the window carries no sentence; the failure itself
// has not yet outlived the twelve hours' worth of trust.
if a.RaisedByReader && a.FailureOutcome != "" && a.FailingSince < time.Now().Add(-ownerWindow).UnixMilli() {
v.Unverified = "Unverified Reader number: the page has been failing for over 12h, so this Reader-reported chapter is unconfirmed."
}
if a.SiteCompletedAt != 0 && a.FinishedAt == 0 {
v.SiteCompleted = "the site says this work is completed (since " + checkedAge(time.Now(), a.SiteCompletedAt) + ")"
}
// Provenance: the actor class behind the current number, evaluated in the
// order the classes outrank one another — the owner's stamp, which a
+1 -1
View File
@@ -26,7 +26,7 @@
</div>
{{range .Rows}}
<div class="trow{{if .Attention}} attention{{end}}">
<span class="c-site">{{.Site}}</span>
<a class="c-site" href="{{.FailingHref}}">{{.Site}}</a>
<span>{{.Due}}</span>
<span>{{.Checked}}</span>
<span>{{.Gap}}</span>
@@ -19,6 +19,7 @@
<input type="number" name="chapter" step="any" placeholder="{{.Chapter}}" required>
<button type="submit" class="ghost">Set</button>
</div>
{{if .Unverified}}<p class="hint">{{.Unverified}}</p>{{end}}
</form>
<form class="dform" hx-post="/admin/series/{{.Key}}/series-url" hx-target="#detail-meta" hx-swap="outerHTML">
<h3>Repair series URL</h3>
@@ -46,6 +47,7 @@
{{else}}
<div class="dform">
<div class="field"><button type="button" class="ghost" onclick="document.getElementById('confirm-finish').hidden = false">Finish</button></div>
{{if .SiteCompleted}}<p class="hint">{{.SiteCompleted}}</p>{{end}}
<div class="confirm-row calm" id="confirm-finish" role="group" aria-live="polite" hidden>
<span>Mark this Series finished?</span>
<div>
@@ -47,7 +47,7 @@
<span class="c-ch">{{.Ch}}</span>
<span>{{.Age}}</span>
<span class="c-rd">{{.Readers}}</span>
<span class="c-note">{{range .Notes}}<span class="mark">{{.}}</span>{{end}}{{if .More}}<span class="mark mark-faint">+{{.More}}</span>{{end}}{{if .Finished}}<span class="mark mark-faint">finished</span>{{end}}</span>
<span class="c-note">{{if .Failure}}<span class="mark">{{.Failure}}</span>{{end}}{{range .Notes}}<span class="mark">{{.}}</span>{{end}}{{if .More}}<span class="mark mark-faint">+{{.More}}</span>{{end}}{{if .Finished}}<span class="mark mark-faint">finished</span>{{end}}</span>
<span class="c-act">{{if .CanPoll}}<a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="closest .trow" hx-swap="outerHTML" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' href="#">Check now</a>{{end}}{{if .CanRemove}}<button class="ghost danger" hx-post="/admin/series/{{.Key}}/remove" hx-target="closest .trow" hx-swap="outerHTML" hx-include="#filterbar" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' hx-confirm="Removes this series and its stored cover. No Reader has it bookmarked; one re-bookmarking it recreates the row.">Remove</button>{{end}}</span>
{{if .RemovalRefused}}<span class="row-msg">a Reader has bookmarked this Series again</span>{{end}}
</div>
+28 -4
View File
@@ -14,6 +14,7 @@ import (
"bookmarkmanager/backend/internal/api"
"bookmarkmanager/backend/internal/httpmw"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/notify"
"bookmarkmanager/backend/internal/store"
"bookmarkmanager/backend/internal/token"
"bookmarkmanager/backend/internal/userscript"
@@ -59,6 +60,12 @@ type Config struct {
// the Lanes page reports the fact and derives reachability from the pass
// log rather than asking the poller (issue #145).
BrowserWSURL string
// DiscordWebhookURL is the webhook owner notices post to (issue #171).
// Unset means the whole path is off — a local stack needs no webhook,
// exactly as the browser URL behaves. The address is a secret in the
// class of TOKEN_KEY: never logged, and it must not reach any line that
// prints configuration.
DiscordWebhookURL string
// LatestPoll configures the background latest-chapter fetcher.
LatestPoll LatestPoll
}
@@ -114,6 +121,7 @@ func loadConfig() Config {
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"),
BrowserWSURL: os.Getenv("BROWSER_WS_URL"),
DiscordWebhookURL: os.Getenv("DISCORD_WEBHOOK_URL"),
LatestPoll: loadLatestPoll(),
}
c.Discord = web.DiscordConfig{
@@ -288,10 +296,22 @@ func main() {
}
s.OnSeriesCreated = acq.Acquire
}
// Owner notices (issue #171): a configured webhook makes the poller tell
// the owner about stalled Lanes. Unset means the whole path is off — a
// local stack needs no webhook, exactly as the browser URL behaves. Only
// the presence is logged; the address itself is a secret in the class of
// TOKEN_KEY.
var notifier latest.Notifier
if u := strings.TrimSpace(cfg.DiscordWebhookURL); u != "" {
notifier = notify.New(u, cfg.PublicBaseURL)
log.Println("owner notices: enabled")
} else {
log.Println("owner notices: disabled (DISCORD_WEBHOOK_URL unset)")
}
// The poller's only connection to the web layer is the database now: it is
// started for its own sake, and the Lanes page reads the pass rows it
// records (issue #145).
startLatestPoller(pollCtx, s, cfg.LatestPoll, browser)
startLatestPoller(pollCtx, s, cfg.LatestPoll, browser, notifier)
srv := &http.Server{
Addr: ":" + cfg.Port,
@@ -324,7 +344,9 @@ func main() {
// newLatestPoller wires the fetcher seams into the poller. Pace is registry
// property, not config (issue #100), so there are no knobs to pass through.
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller {
// notifier is nil when no webhook is configured: a missing webhook is a
// silent off switch, not an error (issue #171).
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher, notifier latest.Notifier) *latest.Poller {
var covers latest.BrowserCoverFetcher
if f, ok := browser.(latest.BrowserCoverFetcher); ok {
covers = f
@@ -335,6 +357,7 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
BrowserFetch: browser,
CoverFetch: covers,
CoverBytesFetch: latest.NewCoverFetcher(),
Notify: notifier,
Now: time.Now,
}
}
@@ -345,7 +368,8 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
// tracking, which is exactly how it behaved before. It returns the running
// Poller, or nil when there is none; the caller starts it for its own sake —
// the Lanes page reads the pass log, so no return value is wired anywhere.
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher) *latest.Poller {
// notifier is nil when DISCORD_WEBHOOK_URL is unset (issue #171).
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher, notifier latest.Notifier) *latest.Poller {
if !cfg.Enabled {
log.Println("latest-chapter poller: disabled by config")
return nil
@@ -358,7 +382,7 @@ func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, brow
// Nil browser: sites behind a JavaScript challenge are simply not polled,
// and their latest_chapter comes from the userscript alone — which is how
// the service behaved before the sidecar existed.
p := newLatestPoller(s, cfg, f, browser)
p := newLatestPoller(s, cfg, f, browser, notifier)
go p.Run(ctx)
return p
+23 -1
View File
@@ -28,6 +28,20 @@ func TestLoadConfigReadsCoverDirectory(t *testing.T) {
}
}
func TestLoadConfigReadsDiscordWebhook(t *testing.T) {
// The address is read, never defaulted: unset stays empty (the whole
// path is off), set flows into the Config for the poller's notifier.
const url = "https://discord.com/api/webhooks/000000/secret"
t.Setenv("DISCORD_WEBHOOK_URL", url)
if got := loadConfig().DiscordWebhookURL; got != url {
t.Fatalf("DiscordWebhookURL = %q, want %q", got, url)
}
t.Setenv("DISCORD_WEBHOOK_URL", "")
if got := loadConfig().DiscordWebhookURL; got != "" {
t.Fatalf("DiscordWebhookURL = %q, want empty when unset", got)
}
}
func TestLoadLatestPollEnabledParsing(t *testing.T) {
tests := []struct {
raw string
@@ -51,7 +65,8 @@ func TestLoadLatestPollEnabledParsing(t *testing.T) {
// nothing here sizes a cooldown any more.
func TestNewLatestPollerWiresFetchers(t *testing.T) {
tls := &latest.TLSFetcher{}
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil)
notifier := &stubNotifier{}
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil, notifier)
if p.Fetch != tls {
t.Fatalf("Fetch not wired")
}
@@ -67,8 +82,15 @@ func TestNewLatestPollerWiresFetchers(t *testing.T) {
if p.Now == nil {
t.Fatalf("Now = nil, want the live clock")
}
if p.Notify != notifier {
t.Fatalf("Notify = %v, want the configured notifier", p.Notify)
}
}
// stubNotifier satisfies latest.Notifier so newLatestPoller's wiring can be
// asserted; it is never called.
type stubNotifier struct{ latest.Notifier }
func TestPutStatusValidation(t *testing.T) {
cases := []struct {
name string
+469 -27
View File
@@ -918,7 +918,7 @@ func TestLanesRenderFromSeededRowsAfterRestart(t *testing.T) {
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.Add(-time.Minute).UnixMilli(), Due: 5, Checked: 5, GapMS: 20_000})
body := lanesBody(t, router, st)
if !strings.Contains(body, `class="c-site">asura</span>`) {
if !strings.Contains(body, `<a class="c-site" href="/admin/series?filter=failing&amp;site=asura">asura</a>`) {
t.Fatalf("asura row missing from a restart-read database:\n%s", body)
}
// The fragment carries its own single timer and answers the swap it asked
@@ -953,22 +953,32 @@ func TestSkipReasonIsNotAStall(t *testing.T) {
}
}
// The five outcome counts render named — the page never prints the word
// "failures" — and a Site with none observed says so rather than drawing a
// blank cell.
// The six outcome counts render named — the page never prints the word
// "failures" — in the taxonomy's fixed order, not found beside the word it
// split out of. A Site with none observed says so rather than drawing a
// blank cell, and a zero not_found renders no chip of its own.
func TestNamedOutcomeChips(t *testing.T) {
router, st := newWebTestServer(t, lanesConfig())
now := time.Now()
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.UnixMilli(), Refused: 5, NoChapter: 2, Errors: 1})
seedPass(t, st, store.LanePass{Site: "demonic", RanAt: now.UnixMilli(), Unreachable: 3, Unfetchable: 4})
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.UnixMilli(), Refused: 5, Unreachable: 3, NoChapter: 2, Unfetchable: 4, NotFound: 6, Errors: 1})
seedPass(t, st, store.LanePass{Site: "demonic", RanAt: now.UnixMilli(), Unreachable: 7, Unfetchable: 8})
seedPass(t, st, store.LanePass{Site: "comix", RanAt: now.UnixMilli()})
body := lanesBody(t, router, st)
for _, want := range []string{"refused 5", "no chapter 2", "errors 1", "unreachable 3", "unfetchable 4"} {
for _, want := range []string{"refused 5", "unreachable 3", "no chapter 2", "unfetchable 4", "not found 6", "errors 1"} {
if !strings.Contains(body, want) {
t.Errorf("lane chips lack %q:\n%s", want, body)
}
}
wantSeq := `<span class="mark">refused 5</span><span class="mark-faint"> · </span><span class="mark">unreachable 3</span><span class="mark-faint"> · </span><span class="mark">no chapter 2</span><span class="mark-faint"> · </span><span class="mark">unfetchable 4</span><span class="mark-faint"> · </span><span class="mark">not found 6</span><span class="mark-faint"> · </span><span class="mark">errors 1</span>`
if !strings.Contains(body, wantSeq) {
t.Errorf("chips do not render in the fixed order (… unfetchable, not found, errors …):\n%s", body)
}
// demonic has outcomes but no not_found: exactly one "not found" in the
// whole fragment — asura's — so a zero count renders none.
if got := strings.Count(body, "not found"); got != 1 {
t.Errorf("the word \"not found\" appears %d times, want exactly the one seeded chip", got)
}
if strings.Contains(body, "failures") {
t.Errorf("the page prints the forbidden word \"failures\":\n%s", body)
}
@@ -979,6 +989,31 @@ func TestNamedOutcomeChips(t *testing.T) {
}
}
// Each Site row on the Lanes page carries one plain navigation link to that
// Site's failing Series — a navigation, not a number: the outcome chips
// beside it stay unlinked permanently, so the row offers one click and one
// answer. The clean Site links too, into an empty list (issue #167).
func TestLaneSiteLinksToFailingSeriesAndChipsStayUnlinked(t *testing.T) {
router, st := newWebTestServer(t, lanesConfig())
now := time.Now()
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.UnixMilli(), NotFound: 6})
seedPass(t, st, store.LanePass{Site: "demonic", RanAt: now.UnixMilli()})
body := lanesBody(t, router, st)
if !strings.Contains(body, `<a class="c-site" href="/admin/series?filter=failing&amp;site=asura">asura</a>`) {
t.Errorf("the failing Site's name is not a link to its failing Series:\n%s", body)
}
if !strings.Contains(body, `<a class="c-site" href="/admin/series?filter=failing&amp;site=demonic">demonic</a>`) {
t.Errorf("the clean Site's name is not a link to its (empty) failing Series:\n%s", body)
}
if !strings.Contains(body, `<span class="mark">not found 6</span>`) {
t.Errorf("the outcome chip no longer renders beside the link:\n%s", body)
}
if got := strings.Count(body, `<a class="c-site"`); got != 2 {
t.Errorf("Site anchor count = %d, want exactly 2 (only the two Site names navigate):\n%s", got, body)
}
}
// Browser configuration is a deployment fact and reachability is derived from
// the latest browser-Site passes inside the refusal backoff — no poller in any
// of the three.
@@ -2177,14 +2212,15 @@ func TestNovelNewTabFallsBackToAll(t *testing.T) {
// cannot produce an orphan series or a Reader-raised Latest Chapter — the
// same reason the store's admin tests seed this way.
type seriesRowSeed struct {
key string
kind string
url string
cover string // cover_address
checkedAt int64
latestNum *float64
bookmarks int // readers that hold it; 0 = orphan
raisedBy bool // a Reader's report is attributed as the raiser
key string
kind string
url string
cover string // cover_address
checkedAt int64
latestNum *float64
bookmarks int // readers that hold it; 0 = orphan
raisedBy bool // a Reader's report is attributed as the raiser
siteCompletedAt int64 // the Site's own marker (issue #168), 0 = not set
}
// seedSeriesRow inserts one series row and its bookmarks (owner first, then
@@ -2204,10 +2240,10 @@ func seedSeriesRow(t *testing.T, st *store.Store, db *sql.DB, seed seriesRowSeed
}
if _, err := db.Exec(`
INSERT INTO series (site, series_id, title, kind, series_url, cover_address,
latest_checked_at, latest_chapter, latest_chapter_num)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
latest_checked_at, latest_chapter, latest_chapter_num, site_completed_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
site, seriesID, "Title of "+seed.key, seed.kind, seed.url, seed.cover,
seed.checkedAt, latestChapter, seed.latestNum); err != nil {
seed.checkedAt, latestChapter, seed.latestNum, seed.siteCompletedAt); err != nil {
t.Fatalf("seed series %q: %v", seed.key, err)
}
for i := range seed.bookmarks {
@@ -2343,6 +2379,46 @@ func TestSeriesListFinishedFilterOption(t *testing.T) {
}
}
// The site-completed filter is in the select's vocabulary like any other,
// and the list it opens shows exactly the hinted Series — the Site's marker
// with no owner's Finish, because a retired Series is not work a Poll is
// coming to do (issue #170).
func TestSeriesListSiteCompletedFilterOption(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
hinted := time.Now().Add(-72 * time.Hour).UnixMilli()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:hinted", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(10), bookmarks: 1, siteCompletedAt: hinted})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:plain", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:retired", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(2), bookmarks: 1, siteCompletedAt: hinted})
if err := st.SetSeriesFinished("asura", "retired", time.Now().UnixMilli()); err != nil {
t.Fatalf("finish asura:retired: %v", err)
}
srv := newRouter(st, testConfig())
body := adminSeriesPage(t, srv, st, "?filter=site_completed")
if !strings.Contains(body, "Title of asura:hinted") {
t.Errorf("site-completed list misses its row:\n%s", body)
}
for _, banned := range []string{"Title of asura:plain", "Title of asura:retired"} {
if strings.Contains(body, banned) {
t.Errorf("site-completed list renders %q:\n%s", banned, body)
}
}
if !strings.Contains(body, "1 series") {
t.Errorf("site-completed heading lacks the count:\n%s", body)
}
if !strings.Contains(body, `Site says completed (1)`) {
t.Errorf("the site-completed option lacks its count:\n%s", body)
}
if !strings.Contains(body, `<option value="site_completed" selected>`) {
t.Errorf("the site-completed option is not selected:\n%s", body)
}
}
// ?filter=stale&site=kagane&kind=manga narrows on all three at once: only the
// kagane manga stale row renders, and every link the page emits carries the
// filter and Site so the narrowing survives in the URL.
@@ -2619,6 +2695,44 @@ func TestSeriesListRowShape(t *testing.T) {
}
}
// The row's fact line names the standing failure and how long it has stood,
// through one format for every age: a one-hour-old failure and a
// three-month-old one both render as "<word> · <checkedAge>", and a Series
// with no failure row renders no marker at all. The marker is its own field,
// never a Notes chip — the chips cap at two plus a tail, so the failure
// would be the fact most likely to be truncated away (issue #167).
func TestSeriesListFailureMarkerAges(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
now := time.Now()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:hour", url: "u", cover: "c", checkedAt: now.UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:months", url: "u", cover: "c", checkedAt: now.UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:fine", url: "u", cover: "c", checkedAt: now.UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
if err := st.RecordSeriesFailure("asura", "hour", "not_found", now.Add(-time.Hour).UnixMilli()); err != nil {
t.Fatalf("seed failure row on hour: %v", err)
}
if err := st.RecordSeriesFailure("asura", "months", "no_chapter", now.Add(-90*24*time.Hour).UnixMilli()); err != nil {
t.Fatalf("seed failure row on months: %v", err)
}
body := adminSeriesPage(t, newRouter(st, testConfig()), st, "")
if !strings.Contains(body, `<span class="mark">not found · 1h ago</span>`) {
t.Errorf("the one-hour failure lacks its marker:\n%s", body)
}
if !strings.Contains(body, `<span class="mark">no chapter · 90d ago</span>`) {
t.Errorf("the three-month failure lacks its marker:\n%s", body)
}
// The clean row carries none, and neither word leaks anywhere else on
// the page: exactly one marker per failing row.
if strings.Count(body, "not found") != 1 || strings.Count(body, "no chapter") != 1 {
t.Errorf("failure words appear more than once, or on a clean row:\n%s", body)
}
}
// The per-Series page renders every Series-level fact the admin read model
// holds for the key the list row already shows: title, the composite key with
// Site and kind, the Latest Chapter, the check age and the anonymous Reader
@@ -2758,6 +2872,57 @@ func TestAdminSeriesDetailRendersMarks(t *testing.T) {
}
}
// The detail page adds one sentence beside the Latest Chapter correction
// control only while the number came from a Reader and the failure row has
// stood past the owner window: a Reader-attributed failing Series carries it,
// a machine-read failing Series does not, and neither does a
// Reader-attributed Series whose failure is still inside the window. The
// sentence names no Reader (issue #167).
func TestAdminSeriesDetailUnverifiedSentence(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
now := time.Now()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:reader-fail", url: "u", cover: "c", checkedAt: now.UnixMilli(), latestNum: floatPtr(2), bookmarks: 1, raisedBy: true})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:machine-fail", url: "u", cover: "c", checkedAt: now.UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:reader-recent", url: "u", cover: "c", checkedAt: now.UnixMilli(), latestNum: floatPtr(3), bookmarks: 1, raisedBy: true})
if err := st.RecordSeriesFailure("asura", "reader-fail", "not_found", now.Add(-24*time.Hour).UnixMilli()); err != nil {
t.Fatalf("seed failure row on reader-fail: %v", err)
}
if err := st.RecordSeriesFailure("asura", "machine-fail", "not_found", now.Add(-24*time.Hour).UnixMilli()); err != nil {
t.Fatalf("seed failure row on machine-fail: %v", err)
}
if err := st.RecordSeriesFailure("asura", "reader-recent", "not_found", now.Add(-time.Hour).UnixMilli()); err != nil {
t.Fatalf("seed failure row on reader-recent: %v", err)
}
srv := newRouter(st, testConfig())
want := "Unverified Reader number: the page has been failing for over 12h, so this Reader-reported chapter is unconfirmed."
fetch := func(key string) string {
t.Helper()
req := httptest.NewRequest(http.MethodGet, "/admin/series/"+key, nil)
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("GET /admin/series/%s status = %d, want 200", key, rr.Code)
}
return rr.Body.String()
}
if body := fetch("asura:reader-fail"); !strings.Contains(body, want) {
t.Errorf("a Reader-attributed failing Series lacks the unverified sentence:\n%s", body)
}
if body := fetch("asura:machine-fail"); strings.Contains(body, want) {
t.Errorf("a machine-read failing Series carries the unverified sentence:\n%s", body)
}
if body := fetch("asura:reader-recent"); strings.Contains(body, want) {
t.Errorf("a Reader-attributed Series failing inside the window carries the sentence:\n%s", body)
}
}
// The provenance line beside the chapter names the actor class behind the
// value — "machine read" for a checked Series, "correction" for the owner's
// stamp, "sighting" for a Reader-raised one — and appears nowhere in the
@@ -2934,7 +3099,7 @@ func TestOverviewVerdictThreeStates(t *testing.T) {
})
t.Run("lanes need a look", func(t *testing.T) {
seedPass(t, st, store.LanePass{Site: "demonic", RanAt: now.UnixMilli(), Skip: latest.SkipRefusing})
seedPass(t, st, store.LanePass{Site: "demonic", RanAt: now.UnixMilli(), Due: 1, Checked: 0, Skip: "", Refused: 0})
body := overviewBody(t, router, st)
if !strings.Contains(body, "1 lane needs a look") {
t.Errorf("attention verdict missing:\n%s", body)
@@ -2953,6 +3118,96 @@ func TestOverviewVerdictThreeStates(t *testing.T) {
})
}
// TestOverviewVerdictFromFaults mirrors the four fault inputs the notifier judges
// (stall, no-browser-route, sidecar-down, adapter-broken); plus healthy and
// virgin. Each seeds the durable rows the page reads, with no poller running.
func TestOverviewVerdictFromFaults(t *testing.T) {
t.Run("stall is a fault", func(t *testing.T) {
router, st := newWebTestServer(t, testConfig())
now := time.Now()
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.UnixMilli(), Due: 3, Checked: 0, Skip: "", Refused: 0})
body := overviewBody(t, router, st)
if !strings.Contains(body, "1 lane needs a look") {
t.Errorf("stall fault verdict missing:\n%s", body)
}
})
t.Run("no-browser-route is a fault on plain TLS site", func(t *testing.T) {
router, st := newWebTestServer(t, testConfig())
old := time.Now().Add(-13 * time.Hour).UnixMilli()
seedPass(t, st, store.LanePass{Site: "asura", RanAt: old, Skip: latest.SkipRefusing})
body := overviewBody(t, router, st)
if !strings.Contains(body, "1 lane needs a look") {
t.Errorf("no-browser-route fault verdict missing:\n%s", body)
}
})
t.Run("sidecar-down is one fault even though every browser lane is down", func(t *testing.T) {
router, st := newWebTestServer(t, testConfig())
now := time.Now().UnixMilli()
for _, site := range []string{"comix", "kagane", "novelfull"} {
seedPass(t, st, store.LanePass{Site: site, RanAt: now, Skip: latest.SkipSidecarDown})
}
body := overviewBody(t, router, st)
if !strings.Contains(body, "1 lane needs a look") {
t.Errorf("sidecar-down fault should count as one, want %q:\n%s", "1 lane needs a look", body)
}
if strings.Contains(body, "3 lanes need a look") {
t.Errorf("sidecar-down fault must not count lanes, got 3:\n%s", body)
}
})
t.Run("adapter-broken is a fault over half no_chapter", func(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
router := newRouter(st, testConfig())
// Four series of the same site; three hold an old no_chapter row (>12h).
now := time.Now()
old := now.Add(-13 * time.Hour).UnixMilli()
for i := 1; i <= 4; i++ {
key := fmt.Sprintf("asura:a%d", i)
url := fmt.Sprintf("https://asurascans.com/comics/a%d", i)
seedSeriesRow(t, st, db, seriesRowSeed{key: key, url: url, cover: "aaa", checkedAt: now.UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
}
for i := 1; i <= 3; i++ {
if err := st.RecordSeriesFailure("asura", fmt.Sprintf("a%d", i), "no_chapter", old); err != nil {
t.Fatalf("seed no_chapter failure: %v", err)
}
}
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.UnixMilli(), Due: 1, Checked: 1})
body := overviewBody(t, router, st)
if !strings.Contains(body, "1 lane needs a look") {
t.Errorf("adapter-broken fault verdict missing:\n%s", body)
}
})
t.Run("healthy when no fault holds", func(t *testing.T) {
router, st := newWebTestServer(t, testConfig())
now := time.Now()
seedPass(t, st, store.LanePass{Site: "asura", RanAt: now.UnixMilli(), Due: 1, Checked: 1})
seedPass(t, st, store.LanePass{Site: "kagane", RanAt: now.UnixMilli(), Skip: latest.SkipAsleep, Due: 1})
body := overviewBody(t, router, st)
if !strings.Contains(body, "all lanes healthy") {
t.Errorf("healthy verdict missing:\n%s", body)
}
})
t.Run("virgin still draws no figures", func(t *testing.T) {
virgin, fresh := newWebTestServer(t, testConfig())
body := overviewBody(t, virgin, fresh)
if !strings.Contains(body, "no Lane has reported yet") {
t.Errorf("virgin verdict missing:\n%s", body)
}
if strings.Contains(body, "all lanes healthy") || strings.Contains(body, "series waiting") {
t.Errorf("virgin verdict draws confident zeroes:\n%s", body)
}
})
}
// The Overview never refreshes itself: the Lane rest is an hour, so a timer
// would re-run a cross-Series join to redraw identical rows. Only the Lanes
// block carries a refresh attribute.
@@ -3027,10 +3282,11 @@ func TestOverviewZeroIsAnUnlinkedDigit(t *testing.T) {
}
}
// The finished figure rides last in the Hygiene block: seriesFilterOrder
// appends it at the tail, the block renders order[1:], and the count comes
// from the same aggregate the select is numbered from. Asserting on the body
// order catches a render that merely shows the figure somewhere.
// The finished figure and the site-completed hint ride last in the Hygiene
// block: seriesFilterOrder appends the pair at the tail, the block renders
// order[1:], and each count comes from the same aggregate the select is
// numbered from. Asserting on the body order catches a render that merely
// shows a figure somewhere.
func TestOverviewFinishedFigureRidesLast(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
@@ -3055,17 +3311,69 @@ func TestOverviewFinishedFigureRidesLast(t *testing.T) {
}
labels = append(labels, m[1])
}
if len(labels) != 8 {
t.Fatalf("hygiene block renders %d figures, want 8 (seven problems + finished):\n%s", len(labels), body)
if len(labels) != 11 {
t.Fatalf("hygiene block renders %d figures, want 11 (nine problems + finished + site completed):\n%s", len(labels), body)
}
if labels[len(labels)-1] != "Finished" {
t.Errorf("finished figure does not ride last; hygiene order = %v:\n%s", labels, body)
if labels[len(labels)-2] != "Finished" || labels[len(labels)-1] != "Site says completed" {
t.Errorf("the informational tail does not ride last; hygiene order = %v:\n%s", labels, body)
}
if !strings.Contains(body, `href="/admin/series?filter=finished">1</a>`) {
t.Errorf("the finished figure lacks its count and door:\n%s", body)
}
}
// The site-completed figure is a door into the list that counts it, and the
// figure agrees with the list's own heading: both come from the same
// aggregate, so they cannot drift apart.
func TestOverviewSiteCompletedFigureAgreesWithListHeading(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
hinted := time.Now().Add(-72 * time.Hour).UnixMilli()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:hinted", url: "https://asurascans.com/comics/hinted", cover: "aaa", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(10), bookmarks: 1, siteCompletedAt: hinted})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:plain", url: "https://asurascans.com/comics/plain", cover: "bbb", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
srv := newRouter(st, testConfig())
body := overviewBody(t, srv, st)
fig := regexp.MustCompile(`href="/admin/series\?filter=site_completed">(\d+)</a>`).FindStringSubmatch(body)
if fig == nil {
t.Fatalf("overview has no site-completed figure:\n%s", body)
}
list := adminSeriesPage(t, srv, st, "?filter=site_completed")
heading := regexp.MustCompile(`(\d+) series <span`).FindStringSubmatch(list)
if heading == nil {
t.Fatalf("site-completed list has no heading count:\n%s", list)
}
if fig[1] != heading[1] {
t.Errorf("the overview says %s, the list it links to says %s", fig[1], heading[1])
}
}
// A measured zero site-completed count stays on the page as a muted digit,
// never a link: following it would land on an empty list, exactly like every
// other zero figure.
func TestOverviewSiteCompletedZeroIsAnUnlinkedDigit(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:one", url: "https://asurascans.com/comics/one", cover: "aaa", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
srv := newRouter(st, testConfig())
body := overviewBody(t, srv, st)
if !strings.Contains(body, `Site says completed</span><span class="fig zero">0</span>`) {
t.Errorf("a zero site-completed figure does not render as a muted digit:\n%s", body)
}
if strings.Contains(body, `href="/admin/series?filter=site_completed"`) {
t.Errorf("a zero site-completed figure is still a link:\n%s", body)
}
}
// A measured zero finished count stays on the page as a muted digit, never a
// link: following it would land on an empty list, exactly like every other
// zero figure.
@@ -3574,6 +3882,94 @@ func TestAdminSeriesDetailFinishControl(t *testing.T) {
}
}
// The hint line rides the Finish control: present for a Series the Site
// marked completed, with its age, absent while the Site has said nothing
// (zero stamp) and gone once the owner finishes — the Site's stamp keeps
// standing after retirement, so the hint is suppressed by the same
// finished_at guard the clock-driven filters carry (issue #170).
func TestAdminSeriesDetailSiteCompletedHint(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
hinted := time.Now().Add(-72 * time.Hour).UnixMilli()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:hinted", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1, siteCompletedAt: hinted})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:plain", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:retired", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1, siteCompletedAt: hinted})
if err := st.SetSeriesFinished("asura", "retired", time.Now().UnixMilli()); err != nil {
t.Fatalf("finish asura:retired: %v", err)
}
srv := newRouter(st, testConfig())
body := seriesDetailPage(t, srv, st, "asura:hinted")
if !strings.Contains(body, "the site says this work is completed (since 3d ago)") {
t.Errorf("hinted Series lacks the hint with its age:\n%s", body)
}
// The hint sits between the Finish opener and the confirm row — a line
// beside the control, not the control itself.
opener := strings.Index(body, "document.getElementById('confirm-finish').hidden = false")
hintAt := strings.Index(body, "the site says this work is completed")
confirmAt := strings.Index(body, `id="confirm-finish"`)
if opener < 0 || hintAt < 0 || confirmAt < 0 || hintAt < opener || confirmAt < hintAt {
t.Errorf("the hint is not a line beside the Finish control (opener %d, hint %d, row %d):\n%s", opener, hintAt, confirmAt, body)
}
if strings.Contains(seriesDetailPage(t, srv, st, "asura:plain"), "the site says this work is completed") {
t.Errorf("an unmarked Series renders the hint:\n%s", body)
}
if strings.Contains(seriesDetailPage(t, srv, st, "asura:retired"), "the site says this work is completed") {
t.Errorf("a finished Series renders the hint:\n%s", body)
}
}
// The hint is a line of text, never a control: with and without a hint, the
// Finish control's markup is byte-identical (AC4). Two stores seed the same
// key so the hx-post URLs match byte for byte; only the hint line differs,
// and it sits outside the confirm row.
func TestAdminSeriesDetailFinishControlUnchangedWithHint(t *testing.T) {
hinted := time.Now().Add(-72 * time.Hour).UnixMilli()
var bodies [2]string
for i, sc := range []int64{hinted, 0} {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:solo", url: "u", cover: "c", checkedAt: time.Now().UnixMilli(), latestNum: floatPtr(1), bookmarks: 1, siteCompletedAt: sc})
db.Close()
srv := newRouter(st, testConfig())
bodies[i] = seriesDetailPage(t, srv, st, "asura:solo")
}
withHint, without := bodies[0], bodies[1]
if !strings.Contains(withHint, "the site says this work is completed") {
t.Fatalf("the hinted store did not render a hint:\n%s", withHint)
}
if strings.Contains(without, "the site says this work is completed") {
t.Fatalf("the plain store rendered a hint:\n%s", without)
}
if got, want := confirmRowMarkup(t, withHint), confirmRowMarkup(t, without); got != want {
t.Errorf("the Finish control's markup changed with a hint present:\nwith hint: %s\nwithout: %s", got, want)
}
}
// confirmRowMarkup returns the confirm-finish row's HTML, byte for byte —
// the Finish control AC4 says must not change with a hint present. The row
// is the outer div (the span plus the inner button pair), with no nested
// divs beyond that, so the first two </div> closes end it.
func confirmRowMarkup(t *testing.T, body string) string {
t.Helper()
start := strings.Index(body, `<div class="confirm-row calm" id="confirm-finish"`)
if start < 0 {
t.Fatalf("no confirm-finish row in:\n%s", body)
}
open := strings.Index(body[start:], ">") + start + 1
inner := strings.Index(body[open:], "</div>")
outer := strings.Index(body[open+inner+len("</div>"):], "</div>")
end := open + inner + len("</div>") + outer + len("</div>")
return body[start:end]
}
// The finish route is the owner's one writer: a finish press stamps
// finished_at and answers with the swapped detail-meta fragment carrying the
// "finished <age> ago" mark, and an un-finish press writes zero and answers
@@ -4035,3 +4431,49 @@ func TestRemoveRejectsBadKeysAndCapsBody(t *testing.T) {
t.Errorf("an oversized body still removed the row:\n%s", list)
}
}
// The failing pair render in the stats block from the same aggregate the
// select is numbered from: failing over 12h counts every Series with a
// failure row older than the cutoff, unverified the Reader-attributed
// subset, and a measured zero stays a muted digit — never a door.
func TestOverviewFailingAndUnverifiedFigures(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
now := time.Now().UnixMilli()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:broken", url: "u", cover: "c", checkedAt: now, latestNum: floatPtr(1), bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:reader", url: "u", cover: "c", checkedAt: now, latestNum: floatPtr(2), bookmarks: 1, raisedBy: true})
if err := st.RecordSeriesFailure("asura", "broken", "not_found", now-24*3600*1000); err != nil {
t.Fatalf("seed failure row on broken: %v", err)
}
if err := st.RecordSeriesFailure("asura", "reader", "not_found", now-24*3600*1000); err != nil {
t.Fatalf("seed failure row on reader: %v", err)
}
srv := newRouter(st, testConfig())
body := overviewBody(t, srv, st)
if !strings.Contains(body, `href="/admin/series?filter=failing">2</a>`) {
t.Errorf("the failing figure lacks its count and door:\n%s", body)
}
if !strings.Contains(body, `href="/admin/series?filter=unverified">1</a>`) {
t.Errorf("the unverified figure lacks its count and door:\n%s", body)
}
empty, emptySt := newWebTestServer(t, testConfig())
body = overviewBody(t, empty, emptySt)
if !strings.Contains(body, `Failing over 12h</span><span class="fig zero">0</span>`) {
t.Errorf("a zero failing figure does not render as a muted digit:\n%s", body)
}
if !strings.Contains(body, `Unverified Reader number</span><span class="fig zero">0</span>`) {
t.Errorf("a zero unverified figure does not render as a muted digit:\n%s", body)
}
if strings.Contains(body, `href="/admin/series?filter=failing"`) {
t.Errorf("a zero failing figure is still a link:\n%s", body)
}
if strings.Contains(body, `href="/admin/series?filter=unverified"`) {
t.Errorf("a zero unverified figure is still a link:\n%s", body)
}
}
+5 -1
View File
@@ -75,7 +75,11 @@ services:
# Host header is an IP address or "localhost" — confirmed 2026-08-03,
# independent of chromedp's own dial logic. The same trap that used to
# force a pinned Docker IP now forbids the tailnet name.
BROWSER_WS_URL: ${BROWSER_WS_URL:-}
# Owner-notice webhook (issue #171). Empty default, never a
# required-guard: unset means the whole path is off, so a local stack
# runs exactly as it does today. An env var not listed here never
# reaches the container.
DISCORD_WEBHOOK_URL: ${DISCORD_WEBHOOK_URL:-}
depends_on:
# The migration runner is the first thing the binary does, so a Postgres
# that is still initialising means a crash-loop until it is not.
@@ -0,0 +1,147 @@
# ADR-0016: The failure row is the state
Date: 2026-08-22
Status: accepted
## Decision
A Series that used to work and has stopped is recorded in one table,
`poll_failures`, keyed by the same `(site, series_id)` composite the rest of
the system uses. One row per failing Series, carrying the outcome word and a
`failing_since` stamp. The row's **existence** is the failure state: there is
no success sentinel, no counter, no history, and nothing added to the Series
row. A correct read deletes the row, and a repeated identical failure writes
nothing — the stamp ages the run of failures, not the current word.
The write is one upsert, from the poll pass loop:
```sql
INSERT INTO poll_failures (site, series_id, outcome, failing_since)
VALUES ($1, $2, $3, $4)
ON CONFLICT (site, series_id) DO UPDATE SET outcome = EXCLUDED.outcome
WHERE poll_failures.outcome <> EXCLUDED.outcome
```
`failing_since` is simply absent from the `SET` arm, so it is never
overwritten, and the `WHERE` makes an unchanged word write nothing at all —
one statement, no read-then-write race. The clear is a plain `DELETE`; a row
that is not there is not an error, because the poller clears on every
successful read and most clears find nothing. The composite foreign key
cascades: deleting a Series takes its failure row, so orphan removal stays a
single statement.
## Why a future reader will find this surprising
The failure state lives in a table of its own, not on the Series row and not
in the pass log. A flag on `series` would be the familiar shape, and the
pass log already records outcomes per pass — why a third place?
Because the failure must outlive the pass that observed it and mean
something the pass row cannot say. The pass log is a per-Lane stream: it
records that a Site returned N `errors` and M `not_found` on a given run,
but "this exact Series has been failing for three months" is not a question
any single pass row answers — it is a question across rows, and the Lanes
page is a live view of the last 14 days, not a Series index. A Series-level
fact needs Series-level storage, and a flag on the Series row is the wrong
shape too: the failure is *transient by definition* (a correct read ends it)
and *repeatable* (the same Series can fail again next year), so the honest
record of "failing since" is a stamp that moves, not a column that flips.
A row that is created and deleted by the read's outcome is that stamp, and
nothing else: the moment a read succeeds the row is gone, so "is this Series
failing?" is answered by one indexed existence check — no success sentinel
to keep consistent with the failure, no counter to reset, no history to
prune. The state cannot drift out of step with the reads that maintain it,
because the reads *are* the maintenance.
The two no-write outcomes are the second surprise. `refused` (the Site is
holding a challenge) and `unreachable` (the browser sidecar was lost
mid-loop) issue **no statement at all** — neither an upsert nor a delete.
The direction matters, and both directions are wrong to touch:
- **Writing would condemn a whole library for one Site's bad day.** A
refusal is the Site's mood, not a fact about any particular Series: when
Cloudflare turns a zone's JS detection on, every Series on that Site reads
refused in the same pass. Writing those rows would stamp every Series in
the library as failing on the evidence of one Site's configuration, and
#166's worklist would present a Site-wide outage as thousands of broken
Series.
- **Deleting would claim a recovery nothing read.** A lost sidecar tells us
nothing about the page — the read never happened. Deleting the row would
report the Series healthy, and worse, it would reset `failing_since`: the
age that makes a three-month failure findable would start over on a
browser restart, erasing evidence no page read contradicted.
So a refusal or an unreachable pass leaves the table exactly as it found
it — the pass neither adds rows that would condemn nor deletes rows that
would claim recovery. The four outcomes that are real evidence about the
page (`not_found`, `no_chapter`, `unfetchable`, `errors`) write; the two
that are not write nothing; success deletes. There is no third case.
## Considered options
**A `failing_since` column on the Series row, alongside the failure word.**
Rejected: it is two more columns on a table every due query and every
bookmark join already touches, for a state that is transient and repeatable.
The column would need its own "clear on success" writer anyway — the same
maintenance the row has — while permanently widening the hottest table in
the system for a value that is usually absent. And the worklist's join
would have to distinguish "never failed" from "currently healthy", which
is exactly the sentinel problem the row avoids: an empty column means both.
**A counter or last-outcome timestamp instead of (or beside) the row.**
Rejected: nothing in the system consumes a failure *count* or a
last-outcome time — the worklist needs "failing and since when". A counter
invites "three failures = something" thresholds that the ticket explicitly
keeps out of scope, and a last-outcome timestamp conflates "the word
changed" with "the run restarted", which the age is deliberately defined
against. The stamp is the only number that means anything, and the row
carries exactly that.
**Write into `poll_passes` and derive the failure state from the pass
stream.**
Rejected: a pass row is per-Lane and per-run; deriving "this Series is
failing since" means scanning 14 days of outcome counts per Series and
guessing at continuity across retention boundaries. The pass log answers
"what did this Site's lane do recently"; the failure table answers "which
Series are broken right now". Two questions, two tables — the pass log is
already pruned on a fixed cutoff, which would silently reset every
failure's age the moment the evidence aged out.
**Refused/unreachable write nothing, but the delete happens anyway (or the
upsert happens, with no delete).**
Rejected in both directions, above: writing condemns a library for one
Site's mood; deleting claims a recovery nothing read and resets the age
that makes a long failure findable. The asymmetry is the point — the two
outcomes are evidence about the *environment*, never about a page.
## Consequences
- The poller writes from the pass loop, one call after `checkOne`, and
clears through the ordinary success path: a Forced Poll that reads the
page deletes the row with no forced branch of its own, and a Forced Poll
*request* — which only stamps the request — clears nothing.
- `poll_failures` is a poll-write table like `poll_lanes` and
`poll_passes`: the store's two methods live next to the Lane-pass
writers, and neither runs on a request path. A store failure is logged
and the poll continues — a failure row is best-effort, and no single bad
Series may stall a Lane.
- The due query does not join the table. A failing Series is polled at the
same pace as any other, because the moment the query started pacing by
failure state, the age would stop meaning what the worklist reads it as.
- Orphan removal stays a single statement: the composite foreign key's
`ON DELETE CASCADE` is what takes the failure row with the Series.
- The table starts empty at deploy, so nothing is findable for the first
twelve hours after deploy and a pre-existing breakage reads as new.
Accepted; the alternative is inventing history.
## Cost of reversing
The failure state is derived, not stored: a rollback drops the table and
every in-progress failure age with it, leaving the pass log's outcome
counts as the only trace — which is exactly the "log line nobody reads"
the ticket set out to replace. Recreating the table later starts the ages
over, so a reversal that is later reversed loses the evidence of the
intervening failures. The failure state itself, however, is the one thing
that is *not* lost by reversing: it is re-derived from the next pass, in
the same direction the original design derives it — the row is
recreated by the next failing read and deleted by the next good one.