feat: server-side latest-chapter polling #2

2026-07-26T18:52:48+07:00

sulthan commented

2026-07-26 18:52:48 +07:00

Adds a background goroutine to the backend that re-checks each bookmarked series' newest published chapter on its own schedule, so latest_chapter stays fresh even when the manga sites are never opened in a browser.

This is a second, parallel signal, not a replacement: the userscript keeps its own maybeCaptureLatestOnSeriesPage / backgroundRefreshLatest logic, unchanged. userscript/manga-bookmark.user.js is byte-identical to main.

How it works

One ticker goroutine in the same binary. Each wake it asks SQLite for bookmarks whose latest_checked_at has aged past a per-bookmark cooldown, fetches those series pages through a Chrome-fingerprinted HTTP client, extracts the max chapter number with a per-site regex, and writes it back through Store.Get + Store.Upsert. Every failure path logs and moves on.

Two independent clocks:

cooldown — how long one bookmark rests between checks, enforced by the WHERE clause in Store.DueForLatestCheck, not by a timer.
interval — how often the goroutine wakes and looks.

Shortening the interval therefore cannot shorten anyone's cooldown; it only makes the poller wake and find nothing due more often.

The row is stamped before the fetch, so an error, a timeout, or a shutdown mid-request still consumes the cooldown — a renamed or challenged series waits out a full cooldown instead of being retried every tick.

Design decisions worth reviewing

latest_checked_at is deliberately absent from the Bookmark struct and from bookmarkColumns. PUT /bookmarks/{key} decodes a whole Bookmark and Upsert writes every column it knows about, so a userscript PUT — which has no idea this field exists — would write a zero and reset the cooldown, making the poller re-fetch that series on every tick for as long as the user kept reading it. Two tests guard this: TestUpsertPreservesLatestCheckedAt and TestPutDoesNotClobberLatestCheckedAt, the latter driving a real router PUT with a userscript-shaped body.

updated_at never moves on a latest-chapter bump. All chapter writes go through Store.Get + Store.Upsert, so the existing CASE keeps the stored timestamp when only latest_chapter_num changes and the bookmark list does not reorder. TestRunOnceDoesNotReorderList asserts both the timestamp and the List() head position.

Fetches use bogdanfinn/tls-client with a Chrome profile. Plain net/http was verified working against both sites on 2026-07-26, so this is not fixing an observed block — it is deliberate defence-in-depth against a future fingerprint-based one. The library is pure Go, so CGO_ENABLED=0, the static binary, and the distroless image are all unaffected. It does require the Go floor to move 1.23 → 1.24.

checkOne validates before spending a request. series_url is entirely client-supplied through PUT /bookmarks/{key}, so without a guard the poller would issue GETs from the server's own network position to any URL a token holder writes. The check requires a known site and an https URL with a non-empty host, and sits after the cooldown stamp so an unfetchable row is retried at cooldown pace rather than hot-looping.

Config

Five new env vars, all with defaults sized for this deployment, all wired through docker-compose.yml:

Variable	Default	Meaning
`LATEST_CHAPTER_POLL_ENABLED`	`1`	Kill switch
`LATEST_CHAPTER_POLL_COOLDOWN`	`1h`	Per series, floored at `15m`
`LATEST_CHAPTER_POLL_INTERVAL`	`10m`	How often to wake
`LATEST_CHAPTER_POLL_BATCH`	`14`	Series per wake
`LATEST_CHAPTER_POLL_STAGGER`	`20s`	Delay between fetches in a batch

batch × (cooldown / interval) = 84 series hold a true cooldown cadence at these defaults. Past that nothing breaks: the cadence stretches uniformly and the oldest-checked-first ordering keeps it fair. Bad values log and fall back rather than failing startup — the poller is an enhancement, and a typo in one of its knobs must not stop bookmark sync.

Known limitation (accepted, documented)

The poller's Store.Get + Store.Upsert is not wrapped in a single transaction. If a userscript PUT commits in the sub-millisecond window between the two, the poller writes back its stale re-read — reverting that progress and, since the stored last_chapter_num now differs, tripping the updated_at CASE and reordering the list.

Accepted rather than fixed for a single-user deployment: the window is one SELECT wide, the poller only writes when a chapter number actually changed, and the next read self-heals it. The alternative — a transactional read-modify-write — means moving or duplicating the updated_at CASE that four tests and the whole list-ordering invariant depend on. Recorded in CLAUDE.md next to the poller's architecture bullet so it is not a silent trap.

Testing

Full suite green, including -race; go vet clean; CGO_ENABLED=0 static build and docker compose build both pass on the bumped golang:1.24-alpine.
No test touches the network: the fetcher interface exists so tests inject a fake, and no test imports tls-client or reaches either manga site.
Extraction is fixture-driven against markup trimmed from real pages (2026-07-26), including a Cloudflare challenge page, cross-series chapter links, decimal chapters, and both raw & and & forms.
Poller tests cover the no-reorder invariant, cooldown enforcement across passes, batch limiting, one bad series not stalling a batch, downward correction on a retracted chapter, cancelled contexts, and all four failure shapes still consuming the cooldown.
Migration from a pre-column database has its own test — newTestStore takes the CREATE TABLE path, so the ALTER TABLE path would otherwise be untested.
Live smoke test: real server, real fetch of asurascans.com. Log showed latest is now Chapter 181 and due=1 checked=1; GET /bookmarks returned latest_chapter_num: 181 with updated_at byte-identical to the PUT that created the row — the no-reorder invariant confirmed against a live site, not just a fake.

🤖 Generated with Claude Code

Adds a background goroutine to the backend that re-checks each bookmarked series' newest published chapter on its own schedule, so `latest_chapter` stays fresh even when the manga sites are never opened in a browser. This is a *second, parallel* signal, not a replacement: the userscript keeps its own `maybeCaptureLatestOnSeriesPage` / `backgroundRefreshLatest` logic, unchanged. `userscript/manga-bookmark.user.js` is byte-identical to `main`. ## How it works One ticker goroutine in the same binary. Each wake it asks SQLite for bookmarks whose `latest_checked_at` has aged past a per-bookmark cooldown, fetches those series pages through a Chrome-fingerprinted HTTP client, extracts the max chapter number with a per-site regex, and writes it back through `Store.Get` + `Store.Upsert`. Every failure path logs and moves on. Two independent clocks: - **cooldown** — how long one bookmark rests between checks, enforced by the `WHERE` clause in `Store.DueForLatestCheck`, not by a timer. - **interval** — how often the goroutine wakes and looks. Shortening the interval therefore cannot shorten anyone's cooldown; it only makes the poller wake and find nothing due more often. The row is stamped **before** the fetch, so an error, a timeout, or a shutdown mid-request still consumes the cooldown — a renamed or challenged series waits out a full cooldown instead of being retried every tick. ## Design decisions worth reviewing **`latest_checked_at` is deliberately absent from the `Bookmark` struct and from `bookmarkColumns`.** `PUT /bookmarks/{key}` decodes a whole `Bookmark` and `Upsert` writes every column it knows about, so a userscript PUT — which has no idea this field exists — would write a zero and reset the cooldown, making the poller re-fetch that series on every tick for as long as the user kept reading it. Two tests guard this: `TestUpsertPreservesLatestCheckedAt` and `TestPutDoesNotClobberLatestCheckedAt`, the latter driving a real router PUT with a userscript-shaped body. **`updated_at` never moves on a latest-chapter bump.** All chapter writes go through `Store.Get` + `Store.Upsert`, so the existing `CASE` keeps the stored timestamp when only `latest_chapter_num` changes and the bookmark list does not reorder. `TestRunOnceDoesNotReorderList` asserts both the timestamp and the `List()` head position. **Fetches use `bogdanfinn/tls-client` with a Chrome profile.** Plain `net/http` was verified working against both sites on 2026-07-26, so this is not fixing an observed block — it is deliberate defence-in-depth against a future fingerprint-based one. The library is pure Go, so `CGO_ENABLED=0`, the static binary, and the distroless image are all unaffected. It does require the Go floor to move 1.23 → 1.24. **`checkOne` validates before spending a request.** `series_url` is entirely client-supplied through `PUT /bookmarks/{key}`, so without a guard the poller would issue GETs from the server's own network position to any URL a token holder writes. The check requires a known site and an `https` URL with a non-empty host, and sits *after* the cooldown stamp so an unfetchable row is retried at cooldown pace rather than hot-looping. ## Config Five new env vars, all with defaults sized for this deployment, all wired through `docker-compose.yml`: | Variable | Default | Meaning | | --- | --- | --- | | `LATEST_CHAPTER_POLL_ENABLED` | `1` | Kill switch | | `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Per series, floored at `15m` | | `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often to wake | | `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake | | `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch | `batch × (cooldown / interval)` = 84 series hold a true cooldown cadence at these defaults. Past that nothing breaks: the cadence stretches uniformly and the oldest-checked-first ordering keeps it fair. Bad values log and fall back rather than failing startup — the poller is an enhancement, and a typo in one of its knobs must not stop bookmark sync. ## Known limitation (accepted, documented) The poller's `Store.Get` + `Store.Upsert` is not wrapped in a single transaction. If a userscript `PUT` commits in the sub-millisecond window between the two, the poller writes back its stale re-read — reverting that progress and, since the stored `last_chapter_num` now differs, tripping the `updated_at` `CASE` and reordering the list. Accepted rather than fixed for a single-user deployment: the window is one SELECT wide, the poller only writes when a chapter number actually changed, and the next read self-heals it. The alternative — a transactional read-modify-write — means moving or duplicating the `updated_at` `CASE` that four tests and the whole list-ordering invariant depend on. Recorded in `CLAUDE.md` next to the poller's architecture bullet so it is not a silent trap. ## Testing - Full suite green, including `-race`; `go vet` clean; `CGO_ENABLED=0` static build and `docker compose build` both pass on the bumped `golang:1.24-alpine`. - No test touches the network: the `fetcher` interface exists so tests inject a fake, and no test imports `tls-client` or reaches either manga site. - Extraction is fixture-driven against markup trimmed from real pages (2026-07-26), including a Cloudflare challenge page, cross-series chapter links, decimal chapters, and both raw `&` and `&` forms. - Poller tests cover the no-reorder invariant, cooldown enforcement across passes, batch limiting, one bad series not stalling a batch, downward correction on a retracted chapter, cancelled contexts, and all four failure shapes still consuming the cooldown. - Migration from a pre-column database has its own test — `newTestStore` takes the `CREATE TABLE` path, so the `ALTER TABLE` path would otherwise be untested. - **Live smoke test:** real server, real fetch of asurascans.com. Log showed `latest is now Chapter 181` and `due=1 checked=1`; `GET /bookmarks` returned `latest_chapter_num: 181` with `updated_at` byte-identical to the PUT that created the row — the no-reorder invariant confirmed against a live site, not just a fake. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

sulthan added 7 commits 2026-07-26 18:52:49 +07:00

docs: correct manga-site adapter details and Cloudflare assumptions b35a89d6a1

asurascans.com moved to Astro with /comics/<slug> paths (was documented
as Next.js /series/<id>). Also replace the untested "CGNAT gets
challenge-paged" claim with live-verified results: curl passes clean
from both the dev machine and VPS as of 2026-07-26, so Cloudflare's
block is IP-reputation-based and time-varying, not a fixed property of
either machine.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

build: bump to Go 1.24 and add bogdanfinn/tls-client c55a1de354

tls-client v1.15.1 declares go 1.24.1; every release back to v1.13.0
does the same, so the 1.23 floor cannot stay. Pure Go, so the
CGO_ENABLED=0 static build and distroless image are unchanged.

feat(store): track when the server last checked a series 0a9042dc42

Adds latest_checked_at plus DueForLatestCheck and MarkLatestChecked.
The column is kept out of bookmarkColumns on purpose: PUT /bookmarks
decodes a whole Bookmark and Upsert writes every column it knows, so a
client PUT would zero the field and defeat the cooldown.

feat: per-site latest-chapter extraction from series HTML c38ed247df

Ports latestChapterFromAnchors from the userscript for asura and
demonic. Asura's pattern is scoped to the series' own slug, which
subsumes the userscript's anchor-text check and also excludes chapter
links belonging to other series. Fixtures are trimmed from real pages.

feat: background poller for latest published chapter 3211b5b42f

Ticker goroutine reads bookmarks past their per-bookmark cooldown,
fetches the series page, and writes latest_chapter through Get+Upsert
so updated_at never moves and the list never reorders. The row is
stamped before the fetch so a broken series waits out a cooldown
instead of retrying every tick.

feat: wire the latest-chapter poller into startup 28f2d32e45

Five env vars, all with defaults sized for this deployment (84 series
at a 1h cadence). Bad values log and fall back rather than failing
startup, and the cooldown is floored at 15m. Fetches go through
tls-client with a Chrome profile as defence in depth.

fix: address final-review findings on the latest-chapter poller e966182e6f

Wires the five poll env vars into docker-compose (the documented kill
switch was inert), skips fetching unknown sites and non-https URLs
before spending a request, and makes runOnce's summary log fire on
empty and cancelled ticks. Records the accepted non-atomic Get+Upsert
window and the one-interval startup delay in the docs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

sulthan merged commit 62772e1eaa into main

2026-07-26 18:54:43 +07:00

sulthan referenced this pull request

2026-07-27 00:30:02 +07:00

Web UI: Updated tab, inline errors, mobile card fixes #3

sulthan referenced this issue from a commit

2026-07-27 00:42:22 +07:00

Web UI: Updated tab, inline errors, mobile card fixes (#3)

sulthan deleted branch explore/tls-fingerprint-fetch

2026-07-27 12:00:33 +07:00

sulthan referenced this pull request

2026-08-10 19:36:24 +07:00