sulthan
3303a55b20
Closes #100. Each Site runs its own Poll Lane: an independent goroutine with its own rest and pace from the registry (`backend/internal/latest/sites.go`), replacing the shared cooldown/interval/stagger/batch configuration. Rest (1h, all six Sites including the browser trio) is enforced by the due query's WHERE clause; the Lane sleeps its effective gap between fetches — the registry 10s, or rest/eligible when a Site holds enough Series, floored at 1s with a Site-naming warning when the floor engages. Lane-local failure handling: - Two challenge-held results stop that Site's Lane for 15m; the probes keep their stamp, untried Series stay due. - A lost browser sets a shared Poller flag: the other browser Lanes skip their passes for the same 15m (no stamp-per-pass-per-Lane on a dead tab), then decay and probe again. - Browser wake gate preserved (5 due, or one waiting 15m, ADR-0005); one tab shared by the three browser Sites; "browser lane behind by X" logged every pass. - Cover work (healing a stored source URL and filling a blank from the series page) runs in the background so a slow CDN cannot consume a Lane's gap. Removed: `LATEST_CHAPTER_POLL_{COOLDOWN,BROWSER_COOLDOWN,INTERVAL,BATCH,STAGGER}` and the 6h browser rest. Only `LATEST_CHAPTER_POLL_ENABLED` remains; DEPLOY.md documents the exact `.env` edit. ADR-0010 records the decisions. Reviewed-on: #106 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
74 lines
3.1 KiB
Go
74 lines
3.1 KiB
Go
package latest
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
)
|
|
|
|
// seriesRead carries the two facts the poll and the acquirer both extract
|
|
// from a series page. Persistence, stamps and scheduling stay with the
|
|
// callers, so the policies that keep the two flows distinct (stamp order,
|
|
// rests) are not swallowed by the module.
|
|
type seriesRead struct {
|
|
Latest latestChapter
|
|
HasLatest bool
|
|
Cover string
|
|
HasCover bool
|
|
// BodyLen is the fetched body's length, surfaced because the no-chapter
|
|
// log uses it to tell a markup change from a body the size cap cut short.
|
|
BodyLen int
|
|
}
|
|
|
|
// errNotFetchable and errNoFetcher separate the gate and the route from fetch
|
|
// failures so each caller keeps its own distinct log line for all three.
|
|
// errChallengeHeld (browser.go) is the outcome of a Site that answered with
|
|
// its interstitial — status 403 (cf-mitigated) or a challenge page body — and
|
|
// is how a Lane tells a refusal from an ordinary failure (issue #100).
|
|
var (
|
|
errNotFetchable = errors.New("series url not fetchable")
|
|
errNoFetcher = errors.New("no fetcher for site")
|
|
)
|
|
|
|
// readSeriesPage performs the series-page read the poll and the acquirer have
|
|
// in common: gate the address, choose the route, fetch the page, extract the
|
|
// Latest Chapter and the Cover address. It persists nothing and stamps
|
|
// nothing.
|
|
//
|
|
// series_url arrives in a client-supplied PUT body (PUT /bookmarks/{key}
|
|
// accepts any string), so the gate is not an optimisation against burning a
|
|
// request on an unknown site: without it, the server would issue a GET from
|
|
// its own network position to whatever URL a token-holder writes, including
|
|
// link-local/internal addresses or non-https schemes.
|
|
func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fetcher) (seriesRead, error) {
|
|
if !fetchableSeriesURL(site, seriesURL) {
|
|
return seriesRead{}, fmt.Errorf("%w: site=%q url=%q", errNotFetchable, site, seriesURL)
|
|
}
|
|
f := fetcherFor(site, browser, tls)
|
|
if f == nil {
|
|
return seriesRead{}, fmt.Errorf("%w: site %q", errNoFetcher, site)
|
|
}
|
|
body, status, err := f.Get(ctx, seriesURL)
|
|
if err != nil {
|
|
return seriesRead{}, fmt.Errorf("fetch %s: %w", seriesURL, err)
|
|
}
|
|
if status == 403 {
|
|
// Cloudflare's challenge response for these Sites (cf-mitigated). The
|
|
// browser fetcher returns exactly this on a held interstitial, and a
|
|
// plain-TLS 403 means the same: the Site is refusing.
|
|
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status)
|
|
}
|
|
if status != 200 {
|
|
return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status)
|
|
}
|
|
if isInterstitial(body) {
|
|
// A 200 that is the challenge page, not the payload: the TLS route can
|
|
// receive this where the browser would have kept re-reading. Same
|
|
// refusal as the 403.
|
|
return seriesRead{}, fmt.Errorf("%w: fetch %s: interstitial body", errChallengeHeld, seriesURL)
|
|
}
|
|
latest, hasLatest := latestChapterFrom(site, seriesURL, body)
|
|
cover, hasCover := coverFrom(site, seriesURL, body)
|
|
return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil
|
|
}
|