Files

T

sulthan 62772e1eaa feat: server-side latest-chapter polling (#2 )

Adds a background goroutine to the backend that re-checks each bookmarked series' newest published chapter on its own schedule, so `latest_chapter` stays fresh even when the manga sites are never opened in a browser.

This is a *second, parallel* signal, not a replacement: the userscript keeps its own `maybeCaptureLatestOnSeriesPage` / `backgroundRefreshLatest` logic, unchanged. `userscript/manga-bookmark.user.js` is byte-identical to `main`.

## How it works

One ticker goroutine in the same binary. Each wake it asks SQLite for bookmarks whose `latest_checked_at` has aged past a per-bookmark cooldown, fetches those series pages through a Chrome-fingerprinted HTTP client, extracts the max chapter number with a per-site regex, and writes it back through `Store.Get` + `Store.Upsert`. Every failure path logs and moves on.

Two independent clocks:

- **cooldown** — how long one bookmark rests between checks, enforced by the `WHERE` clause in `Store.DueForLatestCheck`, not by a timer.
- **interval** — how often the goroutine wakes and looks.

Shortening the interval therefore cannot shorten anyone's cooldown; it only makes the poller wake and find nothing due more often.

The row is stamped **before** the fetch, so an error, a timeout, or a shutdown mid-request still consumes the cooldown — a renamed or challenged series waits out a full cooldown instead of being retried every tick.

## Design decisions worth reviewing

**`latest_checked_at` is deliberately absent from the `Bookmark` struct and from `bookmarkColumns`.** `PUT /bookmarks/{key}` decodes a whole `Bookmark` and `Upsert` writes every column it knows about, so a userscript PUT — which has no idea this field exists — would write a zero and reset the cooldown, making the poller re-fetch that series on every tick for as long as the user kept reading it. Two tests guard this: `TestUpsertPreservesLatestCheckedAt` and `TestPutDoesNotClobberLatestCheckedAt`, the latter driving a real router PUT with a userscript-shaped body.

**`updated_at` never moves on a latest-chapter bump.** All chapter writes go through `Store.Get` + `Store.Upsert`, so the existing `CASE` keeps the stored timestamp when only `latest_chapter_num` changes and the bookmark list does not reorder. `TestRunOnceDoesNotReorderList` asserts both the timestamp and the `List()` head position.

**Fetches use `bogdanfinn/tls-client` with a Chrome profile.** Plain `net/http` was verified working against both sites on 2026-07-26, so this is not fixing an observed block — it is deliberate defence-in-depth against a future fingerprint-based one. The library is pure Go, so `CGO_ENABLED=0`, the static binary, and the distroless image are all unaffected. It does require the Go floor to move 1.23 → 1.24.

**`checkOne` validates before spending a request.** `series_url` is entirely client-supplied through `PUT /bookmarks/{key}`, so without a guard the poller would issue GETs from the server's own network position to any URL a token holder writes. The check requires a known site and an `https` URL with a non-empty host, and sits *after* the cooldown stamp so an unfetchable row is retried at cooldown pace rather than hot-looping.

## Config

Five new env vars, all with defaults sized for this deployment, all wired through `docker-compose.yml`:

| Variable | Default | Meaning |
| --- | --- | --- |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | Kill switch |
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Per series, floored at `15m` |
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often to wake |
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake |
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch |

`batch × (cooldown / interval)` = 84 series hold a true cooldown cadence at these defaults. Past that nothing breaks: the cadence stretches uniformly and the oldest-checked-first ordering keeps it fair. Bad values log and fall back rather than failing startup — the poller is an enhancement, and a typo in one of its knobs must not stop bookmark sync.

## Known limitation (accepted, documented)

The poller's `Store.Get` + `Store.Upsert` is not wrapped in a single transaction. If a userscript `PUT` commits in the sub-millisecond window between the two, the poller writes back its stale re-read — reverting that progress and, since the stored `last_chapter_num` now differs, tripping the `updated_at` `CASE` and reordering the list.

Accepted rather than fixed for a single-user deployment: the window is one SELECT wide, the poller only writes when a chapter number actually changed, and the next read self-heals it. The alternative — a transactional read-modify-write — means moving or duplicating the `updated_at` `CASE` that four tests and the whole list-ordering invariant depend on. Recorded in `CLAUDE.md` next to the poller's architecture bullet so it is not a silent trap.

## Testing

- Full suite green, including `-race`; `go vet` clean; `CGO_ENABLED=0` static build and `docker compose build` both pass on the bumped `golang:1.24-alpine`.
- No test touches the network: the `fetcher` interface exists so tests inject a fake, and no test imports `tls-client` or reaches either manga site.
- Extraction is fixture-driven against markup trimmed from real pages (2026-07-26), including a Cloudflare challenge page, cross-series chapter links, decimal chapters, and both raw `&` and `&amp;` forms.
- Poller tests cover the no-reorder invariant, cooldown enforcement across passes, batch limiting, one bad series not stalling a batch, downward correction on a retracted chapter, cancelled contexts, and all four failure shapes still consuming the cooldown.
- Migration from a pre-column database has its own test — `newTestStore` takes the `CREATE TABLE` path, so the `ALTER TABLE` path would otherwise be untested.
- **Live smoke test:** real server, real fetch of asurascans.com. Log showed `latest is now Chapter 181` and `due=1 checked=1`; `GET /bookmarks` returned `latest_chapter_num: 181` with `updated_at` byte-identical to the PUT that created the row — the no-reorder invariant confirmed against a live site, not just a fake.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #2
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>

2026-07-26 18:54:42 +07:00

9.3 KiB

Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Status

Greenfield. Only plans/mangaBookmark.md exists — no code yet. That plan is the spec; read it before building. Two deliverables: a Go sync backend and a single Bromite-compatible userscript.

What this is

A manga read-progress tracker for a user reading on asurascans.com (the current domain; asuracomic.net 301s here) and demonicscans.org from Bromite (mobile Chromium). A userscript injects on-page UI (floating button + slide-in panel) and syncs progress to a self-hosted Go backend so bookmarks unify across both sites and across devices.

Hard constraints (these drive the design — do not violate)

Bromite uses Chromium's native userscript engine, not Tampermonkey:

No GM_* APIs anywhere. No GM_setValue/GM_getValue (use page localStorage), no GM_registerMenuCommand (inject on-page UI), no GM_xmlhttpRequest for cross-origin (use plain fetch()). Keeping the script GM-free also lets it run in desktop Tampermonkey/Violentmonkey for faster iteration.
Cross-origin fetch() works only against a CORS-enabled backend. Manga sites are https://, so backend must be HTTPS (mixed-content block otherwise).
Asura and Demonic are separate origins with separate localStorage — a shared remote store is the only way to unify bookmarks. Cloud sync is required, not optional.
Userscript runs in an isolated world, so the embedded API token is safe from the site's JS.
Cloudflare's block on fetching the manga sites is IP-reputation-based, not universal — and not reliably reproducible. Verified 2026-07-26: plain curl from both the CGNAT dev machine and the deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. This contradicts an earlier, untested assumption that the CGNAT dev IP would be blocked; it was not, at least on this date. Treat "does curl work right now" as a live, time-varying fact to re-check, not a fixed property of a given machine — Cloudflare's bot scoring can flip a previously-clean IP without notice. Any backend fetcher still needs a graceful-degrade path for when it does get challenged, and adapters should be verified against live pages (Playwright MCP, on-device devtools, or a direct probe) before finalizing, not assumed from a single earlier test.

Architecture

Bromite userscript (isolated world, per-site adapters, localStorage cache)
   -- fetch() HTTPS -->  reverse proxy (TLS + CORS)  -->  Go net/http  -->  SQLite (volume)

Backend (backend/): stdlib net/http (a handful of routes, no framework) + modernc.org/sqlite (pure Go, CGO_ENABLED=0 -> static binary -> distroless/scratch image). The reverse proxy terminates TLS; the Go service listens plain :8080.
Single-user store. One bookmarks table keyed <site>:<series_id> (asura|demonic). Sync is last-write-wins. Schema and endpoint list are in the plan.
Endpoints: GET /bookmarks, PUT /bookmarks/{key} (upsert; see updated_at rule below), DELETE /bookmarks/{key}, GET /healthz (no auth).
Web UI: the same binary serves a password-gated browser UI on a second hostname — GET / (list, or login page when there is no session), POST /login, POST /logout, GET /static/*, and htmx fragment endpoints under /ui/*. Templates and assets are go:embed-ed, so backend/Dockerfile must copy templates/ and static/ as well as *.go. Sessions are stateless HMAC cookies keyed off API_TOKEN; WEB_PASSWORD gates them and, when empty, the web routes are not registered at all. UI mutations read-modify-write through Store.Get + Store.Upsert so the updated_at rule stays in one place. See docs/superpowers/specs/2026-07-25-web-ui-design.md.
Latest-chapter poller: a ticker goroutine in the same binary re-checks each bookmarked series' newest published chapter from the backend's own network access, so latest_chapter stays fresh when the user is not browsing. It is a second, parallel signal — the userscript keeps its own maybeCaptureLatestOnSeriesPage/backgroundRefreshLatest logic unchanged. Two independent clocks: a per-bookmark cooldown (latest_checked_at column, enforced by Store.DueForLatestCheck's WHERE clause) and a wake interval. The row is stamped before the fetch so a broken series waits out a full cooldown instead of retrying every tick, and writes go through Store.Get + Store.Upsert so a new chapter never reorders the list. Fetches use bogdanfinn/tls-client with a Chrome profile as defence in depth against fingerprint-based blocking; any failure logs and skips. See docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md. The poller's Store.Get + Store.Upsert is not wrapped in a transaction, so a userscript PUT that commits between the two can be overwritten by the poller's stale re-read — reverting that read progress and, since the stored value now differs, moving updated_at and reordering the list. This is a known, accepted limitation for a single-user deployment, not a bug to fix.
updated_at drives list order, so it moves only on real reading progress: the server applies its timestamp when the row is new or last_chapter_num changes, and otherwise keeps the stored value — favouriting a series or recording a newly published chapter must not reorder the list. PUT therefore returns the row as stored, and clients must adopt that response rather than their own payload. See plans/2026-07-25-bookmark-list-favorites-design.md §4.
Config via env: API_TOKEN, ALLOWED_ORIGINS (comma list), DB_PATH (default /data/bookmarks.db), PORT (default 8080), WEB_PASSWORD (gates the browser UI; unset disables it), LATEST_CHAPTER_POLL_ENABLED/_COOLDOWN/_INTERVAL/_BATCH/_STAGGER (background latest-chapter poller; defaults on, 1h/10m/14/20s).

Userscript structure (single IIFE, `manga-bookmark.user.js`)

Site adapters — one per host, detect(location, document) returns page type + IDs. Identify type/IDs from URL regex (most stable); pull title/cover from og:title/og:image meta tags, not CSS classes.
API client — apiGet/apiPut/apiDelete with bearer header; localStorage key mangabm:cache for instant render + offline fallback.
Progress logic — auto-upsert last_chapter only when chapterNum >= stored last_chapter_num (re-reading old chapters must not regress progress; unparseable -> set current). Manual panel override forces any value.
UI — rendered inside a Shadow DOM root to isolate from site CSS (critical on mobile).
SPA navigation — Asura is Astro, client-routed on the comic/chapter pages: patch history.pushState/replaceState + listen popstate, re-run detect() on URL change so auto-update fires without reload. Demonic uses classic reloads (initial document-idle run suffices).

Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trusting)

asurascans.com: series /comics/<slug> (slug carries a trailing hash-like suffix, e.g. -f886a8af), chapter /comics/<slug>/chapter/<n>. Astro-rendered; chapter links are present in raw server HTML (no client-side-only render blocking a server fetch).
demonicscans.org: series /manga/<slug> (slug may URL-encode punctuation, e.g. %2527 for '), chapter /title/<slug>/chapter/<n>/<page> (the older chaptered.php?manga=<id>&chapter=<n> form still exists as a redirect and is what series-page chapter-list anchors link through).

Commands (once code exists)

Backend (cd backend):

Test all: go test ./...
Single test: go test -run TestName ./...
Build static binary: CGO_ENABLED=0 go build

Local stack: docker compose up (named volume mounted at /data, restart: unless-stopped).

Smoke test: curl the endpoints with Authorization: Bearer <token>; confirm OPTIONS preflight returns CORS headers and /healthz returns 200.

Security invariants

Auth on /bookmarks*: require Authorization: Bearer <API_TOKEN>, constant-time compare, 401 otherwise.
CORS: reflect Origin only when in ALLOWED_ORIGINS; allow GET,PUT,DELETE,OPTIONS + headers Authorization,Content-Type; answer preflight OPTIONS with 204.

Relevant skills

multi-stage-dockerfile and docker-compose-orchestration for the container work (referenced in the plan).

graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.

Rules:

For codebase questions, first run graphify query "<question>" when graphify-out/graph.json exists. Use graphify path "<A>" "<B>" for relationships and graphify explain "<concept>" for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
After modifying code, run graphify update . to keep the graph current (AST-only, no API cost).

9.3 KiB Raw Blame History