sulthan
62772e1eaa
Adds a background goroutine to the backend that re-checks each bookmarked series' newest published chapter on its own schedule, so `latest_chapter` stays fresh even when the manga sites are never opened in a browser.
This is a *second, parallel* signal, not a replacement: the userscript keeps its own `maybeCaptureLatestOnSeriesPage` / `backgroundRefreshLatest` logic, unchanged. `userscript/manga-bookmark.user.js` is byte-identical to `main`.
## How it works
One ticker goroutine in the same binary. Each wake it asks SQLite for bookmarks whose `latest_checked_at` has aged past a per-bookmark cooldown, fetches those series pages through a Chrome-fingerprinted HTTP client, extracts the max chapter number with a per-site regex, and writes it back through `Store.Get` + `Store.Upsert`. Every failure path logs and moves on.
Two independent clocks:
- **cooldown** — how long one bookmark rests between checks, enforced by the `WHERE` clause in `Store.DueForLatestCheck`, not by a timer.
- **interval** — how often the goroutine wakes and looks.
Shortening the interval therefore cannot shorten anyone's cooldown; it only makes the poller wake and find nothing due more often.
The row is stamped **before** the fetch, so an error, a timeout, or a shutdown mid-request still consumes the cooldown — a renamed or challenged series waits out a full cooldown instead of being retried every tick.
## Design decisions worth reviewing
**`latest_checked_at` is deliberately absent from the `Bookmark` struct and from `bookmarkColumns`.** `PUT /bookmarks/{key}` decodes a whole `Bookmark` and `Upsert` writes every column it knows about, so a userscript PUT — which has no idea this field exists — would write a zero and reset the cooldown, making the poller re-fetch that series on every tick for as long as the user kept reading it. Two tests guard this: `TestUpsertPreservesLatestCheckedAt` and `TestPutDoesNotClobberLatestCheckedAt`, the latter driving a real router PUT with a userscript-shaped body.
**`updated_at` never moves on a latest-chapter bump.** All chapter writes go through `Store.Get` + `Store.Upsert`, so the existing `CASE` keeps the stored timestamp when only `latest_chapter_num` changes and the bookmark list does not reorder. `TestRunOnceDoesNotReorderList` asserts both the timestamp and the `List()` head position.
**Fetches use `bogdanfinn/tls-client` with a Chrome profile.** Plain `net/http` was verified working against both sites on 2026-07-26, so this is not fixing an observed block — it is deliberate defence-in-depth against a future fingerprint-based one. The library is pure Go, so `CGO_ENABLED=0`, the static binary, and the distroless image are all unaffected. It does require the Go floor to move 1.23 → 1.24.
**`checkOne` validates before spending a request.** `series_url` is entirely client-supplied through `PUT /bookmarks/{key}`, so without a guard the poller would issue GETs from the server's own network position to any URL a token holder writes. The check requires a known site and an `https` URL with a non-empty host, and sits *after* the cooldown stamp so an unfetchable row is retried at cooldown pace rather than hot-looping.
## Config
Five new env vars, all with defaults sized for this deployment, all wired through `docker-compose.yml`:
| Variable | Default | Meaning |
| --- | --- | --- |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | Kill switch |
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Per series, floored at `15m` |
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often to wake |
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake |
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch |
`batch × (cooldown / interval)` = 84 series hold a true cooldown cadence at these defaults. Past that nothing breaks: the cadence stretches uniformly and the oldest-checked-first ordering keeps it fair. Bad values log and fall back rather than failing startup — the poller is an enhancement, and a typo in one of its knobs must not stop bookmark sync.
## Known limitation (accepted, documented)
The poller's `Store.Get` + `Store.Upsert` is not wrapped in a single transaction. If a userscript `PUT` commits in the sub-millisecond window between the two, the poller writes back its stale re-read — reverting that progress and, since the stored `last_chapter_num` now differs, tripping the `updated_at` `CASE` and reordering the list.
Accepted rather than fixed for a single-user deployment: the window is one SELECT wide, the poller only writes when a chapter number actually changed, and the next read self-heals it. The alternative — a transactional read-modify-write — means moving or duplicating the `updated_at` `CASE` that four tests and the whole list-ordering invariant depend on. Recorded in `CLAUDE.md` next to the poller's architecture bullet so it is not a silent trap.
## Testing
- Full suite green, including `-race`; `go vet` clean; `CGO_ENABLED=0` static build and `docker compose build` both pass on the bumped `golang:1.24-alpine`.
- No test touches the network: the `fetcher` interface exists so tests inject a fake, and no test imports `tls-client` or reaches either manga site.
- Extraction is fixture-driven against markup trimmed from real pages (2026-07-26), including a Cloudflare challenge page, cross-series chapter links, decimal chapters, and both raw `&` and `&` forms.
- Poller tests cover the no-reorder invariant, cooldown enforcement across passes, batch limiting, one bad series not stalling a batch, downward correction on a retracted chapter, cancelled contexts, and all four failure shapes still consuming the cooldown.
- Migration from a pre-column database has its own test — `newTestStore` takes the `CREATE TABLE` path, so the `ALTER TABLE` path would otherwise be untested.
- **Live smoke test:** real server, real fetch of asurascans.com. Log showed `latest is now Chapter 181` and `due=1 checked=1`; `GET /bookmarks` returned `latest_chapter_num: 181` with `updated_at` byte-identical to the PUT that created the row — the no-reorder invariant confirmed against a live site, not just a fake.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #2
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
116 lines
4.6 KiB
Go
116 lines
4.6 KiB
Go
package main
|
|
|
|
import "testing"
|
|
|
|
// Trimmed from https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af
|
|
// fetched 2026-07-26. The first anchor is the "First Chapter" shortcut: it is a
|
|
// real chapter link with no "Chapter N" text, and it must not be mistaken for
|
|
// the latest just because it parses.
|
|
const asuraSeriesFixture = `
|
|
<a href="/comics/chronicles-of-the-demon-faction-f886a8af/chapter/1" class="py-3 rounded-md bg-[#E8E8E8]"><svg class="w-4 h-4"></svg>First Chapter</a>
|
|
<a href="/comics/chronicles-of-the-demon-faction-f886a8af/chapter/179" data-astro-prefetch="hover" class="group flex"><span class="font-medium">Chapter 179</span></a>
|
|
<a href="/comics/chronicles-of-the-demon-faction-f886a8af/chapter/181" data-astro-prefetch="hover" class="group flex"><span class="font-medium">Chapter 181</span></a>
|
|
<a href="/comics/chronicles-of-the-demon-faction-f886a8af/chapter/180" data-astro-prefetch="hover" class="group flex"><span class="font-medium">Chapter 180</span></a>
|
|
`
|
|
|
|
// A chapter link belonging to a different series, of the kind a "you might also
|
|
// like" strip would introduce. Slug scoping must exclude it.
|
|
const asuraCrossSeriesFixture = asuraSeriesFixture + `
|
|
<a href="/comics/some-other-series-aabbccdd/chapter/999" class="group flex"><span>Chapter 999</span></a>
|
|
`
|
|
|
|
// Trimmed from https://demonicscans.org/manga/Catastrophic-Necromancer fetched
|
|
// 2026-07-26. Note the raw "&", the doubled space after <a, and the decimal
|
|
// chapters, all as they appear live.
|
|
const demonicSeriesFixture = `
|
|
<a href="/chaptered.php?manga=11799&chapter=0.5" class="chplinks" title="Catastrophic Necromancer 0.5">Chapter 0.5</a>
|
|
<a href="/chaptered.php?manga=11799&chapter=294" class="chplinks" title="Catastrophic Necromancer 294">Chapter 294</a>
|
|
<a href="/chaptered.php?manga=11799&chapter=296" class="chplinks" title="Catastrophic Necromancer 296">Chapter 296</a>
|
|
<a href="/chaptered.php?manga=11799&chapter=295" class="chplinks" title="Catastrophic Necromancer 295">Chapter 295</a>
|
|
`
|
|
|
|
// What Cloudflare serves instead of the page when an IP's bot score flips.
|
|
const challengeFixture = `<!DOCTYPE html><html><head><title>Just a moment...</title>
|
|
<script src="/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1"></script></head>
|
|
<body><div id="challenge-running">Checking your browser</div></body></html>`
|
|
|
|
func TestLatestChapterFrom(t *testing.T) {
|
|
const asuraURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
|
|
const demonicURL = "https://demonicscans.org/manga/Catastrophic-Necromancer"
|
|
|
|
tests := []struct {
|
|
name string
|
|
site string
|
|
seriesURL string
|
|
body string
|
|
wantOK bool
|
|
wantNum float64
|
|
wantLabel string
|
|
}{
|
|
{
|
|
name: "asura takes the max, not the last listed",
|
|
site: "asura", seriesURL: asuraURL, body: asuraSeriesFixture,
|
|
wantOK: true, wantNum: 181, wantLabel: "Chapter 181",
|
|
},
|
|
{
|
|
name: "asura ignores another series' chapter links",
|
|
site: "asura", seriesURL: asuraURL, body: asuraCrossSeriesFixture,
|
|
wantOK: true, wantNum: 181, wantLabel: "Chapter 181",
|
|
},
|
|
{
|
|
name: "asura with an unparseable series url",
|
|
site: "asura", seriesURL: "https://asurascans.com/", body: asuraSeriesFixture,
|
|
wantOK: false,
|
|
},
|
|
{
|
|
name: "demonic takes the max across raw and escaped ampersands",
|
|
site: "demonic", seriesURL: demonicURL, body: demonicSeriesFixture,
|
|
wantOK: true, wantNum: 296, wantLabel: "Chapter 296",
|
|
},
|
|
{
|
|
name: "demonic keeps decimal chapters parseable",
|
|
site: "demonic", seriesURL: demonicURL,
|
|
body: `<a href="/chaptered.php?manga=11799&chapter=0.5">Chapter 0.5</a>`,
|
|
wantOK: true, wantNum: 0.5, wantLabel: "Chapter 0.5",
|
|
},
|
|
{
|
|
name: "empty body",
|
|
site: "asura", seriesURL: asuraURL, body: "",
|
|
wantOK: false,
|
|
},
|
|
{
|
|
name: "cloudflare challenge page",
|
|
site: "asura", seriesURL: asuraURL, body: challengeFixture,
|
|
wantOK: false,
|
|
},
|
|
{
|
|
name: "demonic markup handed to the asura rule",
|
|
site: "asura", seriesURL: asuraURL, body: demonicSeriesFixture,
|
|
wantOK: false,
|
|
},
|
|
{
|
|
name: "unknown site",
|
|
site: "mangadex", seriesURL: "https://example.com/x", body: asuraSeriesFixture,
|
|
wantOK: false,
|
|
},
|
|
}
|
|
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
got, ok := latestChapterFrom(tt.site, tt.seriesURL, tt.body)
|
|
if ok != tt.wantOK {
|
|
t.Fatalf("ok = %v, want %v (got %+v)", ok, tt.wantOK, got)
|
|
}
|
|
if !tt.wantOK {
|
|
return
|
|
}
|
|
if got.Num != tt.wantNum {
|
|
t.Errorf("Num = %v, want %v", got.Num, tt.wantNum)
|
|
}
|
|
if got.Label != tt.wantLabel {
|
|
t.Errorf("Label = %q, want %q", got.Label, tt.wantLabel)
|
|
}
|
|
})
|
|
}
|
|
}
|