Files

T

sulthan 6dce9fc481 Offline retry queue for userscript writes (#5 )

## The bug

Every mutation in the userscript is optimistic: it writes `state.list` and the `mangabm:cache` copy, re-renders, then PUTs. **If the PUT fails, nothing rolls back and nothing retries.** The cache now asserts something the server has never heard of, until the next successful `GET /bookmarks` silently overwrites it.

Walked end to end: phone loses signal mid-read, user taps **Archive**, the card moves to Archived and looks saved. `apiPut` throws. Local state is not rolled back. Signal returns, a later navigation calls `refresh()` → `apiGet()` → `setList()`, which replaces `state.list` wholesale. The series is back in All. No toast, no explanation, minutes later.

Four of the five call sites already toasted a promise of a retry that did not exist. This makes the existing copy honest rather than adding a new promise.

## The shape

**Markers, not payloads.** Every mutation already builds and PUTs the *whole* desired row, and `state.list` (mirrored into `mangabm:cache`) already *is* the desired state. So the queue stores only `{key, op, sendStatus, attempts}` in `localStorage` under `mangabm:queue`; the body is read from `state.byKey` at send time. That collapses four hard questions at once:

- **Ordering** — one entry per key, so two writes to the same series cannot replay out of order.
- **Coalescing** — archive-then-unarchive is not two writes, it is "the cache now says `reading`". Nothing to merge.
- **DELETE after PUT** — the delete entry *replaces* the put entry, so a replay cannot resurrect the row.
- **Staleness** — no snapshot can drift from the cache, because there is no snapshot.

**One write path.** `pushBookmark` / `pushDelete` are the only way a user-facing mutation reaches the API — not a fallback bolted onto each `catch`. That distinction is the whole point; see below.

**Drain triggers**, all cheap when the queue is empty (`drain()` returns on its first line): head of `refresh()`, `onNavigate` (drain only, *not* a full refresh — Asura is client-routed and an extra GET per route change is not wanted), a `window` `online` listener, and tapping the pending chip.

**Visibility.** A `⟳ N pending` chip in the existing `#nav` row, hidden entirely when the queue is empty. Silent convergence in the happy path; honest the moment something is stuck.

**Failure classes:** a `400` drops the entry and says so; a `401` aborts the whole pass and keeps the queue intact (fixing the token fixes everything); a `404` on DELETE is treated as success; network errors and `5xx` retry to a cap of 10 attempts. Every dropped write is announced — a queue that fails permanently and says nothing is the same class of bug being fixed.

## The sticky-`sendStatus` hole this closes

An empty `status` on the wire means "keep the stored bucket" server-side. A queue bolted onto each mutation's `catch` has a hole:

1. Offline. User archives X → entry `{X, put, sendStatus: true}` is queued.
2. Signal returns. No drain trigger has fired yet.
3. User reads a chapter of X → `syncUpsert` PUTs with `sendStatus: false` → **succeeds** → `upsertLocal(saved)` adopts a server row that still says `reading`.
4. The archive is gone from local state, and the pending entry now replays a row that no longer carries the intent. Silent un-archive.

Routing every write through `pushBookmark` — which ORs in any pending `sendStatus` and only ever *widens* it, never narrows it — is what closes that. A replayed progress write still omits `status`; a replayed archive still carries it.

`refresh()` drains before it fetches, then `overlayPending()` re-applies anything still pending over the fetched list before `setList` replaces `state.byKey`, so the card the user just changed never flaps back.

`applyLatestChapterIfChanged` deliberately stays **out** of the queue: it is background information the user never asked for, the server-side poller learns the same fact independently, and `backgroundRefreshLatest` already retries on a 4h throttle. Queueing it would let a stale local `latest_chapter` overwrite a fresher poller value on replay.

## Fixes from the final review (commits 6-8)

The whole-branch review found one Critical and two Important defects that only appear across commit boundaries:

- **Critical — the un-archive hole reopened through the latest-chapter exclusion.** `applyLatestChapterIfChanged` PUTs without `sendStatus`, so the server strips `status`, returns the stored `reading`, and `upsertLocal(saved)` writes that over a pending archive. `onNavigate` runs `maybeCaptureLatestOnSeriesPage()` *before* `drain()` with no await between them, so this was deterministic on any series-page visit, not a race — and the drain then sent `status:"reading"` explicitly, making it permanent. Fixed with a single `if (queueGet(bm.key)) return;` guard: the write stays unqueued as designed, it just no longer adopts a server row while a write is pending. `latest_chapter` still reaches the server via the drain, carrying the correct bucket.
- **Important — `draining` guarded drain-vs-drain but not drain-vs-mutation.** A tap during an in-flight same-key PUT started a second concurrent write; whichever response landed second won, and the loser's `queueDrop` could delete the entry the tap had just parked. Fixed with per-key in-flight tracking: a write for a key already in flight defers (parks a queue entry, sends nothing, returns `false` so the caller still toasts), and the landing flight suppresses its own `upsertLocal`/`queueDrop` when superseded — including on its failure path, so a failing flight cannot clobber a parked `op:"delete"` and resurrect a removed bookmark.
- **Important — `drain()` returned `undefined` while already draining**, so `refresh()`'s `await drain()` was a silent no-op and could adopt a pre-write list, flapping the card at boot. It now returns the in-flight promise. The empty-queue fast path is unchanged and still an immediate return.

Also: `render()` moved out of `pushBookmark`'s `try` (a render throw was re-queueing an already-successful write), and unawaited `drain()` rejections are swallowed.

## The backend is untouched

No file under `backend/` is in this diff. The `updated_at` rule and the empty-status keep rule stay solely in `Store.Upsert`; nothing client-side duplicates or works around them. A replayed PUT is an ordinary late write under the project's existing last-write-wins model. Regression check: `go test ./...` is `ok`, `CGO_ENABLED=0 go build ./...` succeeds.

Two accepted losses, marked with `ponytail:` comments at the replay site: a `latest_chapter` the poller learned while the client was offline can be overwritten by the client's older value (self-healing on the poller's next cooldown), and read progress made on another device between the failed write and the replay can be overwritten (single-user deployment).

## Verification status — read this before merging

`node --check` passes and every commit was reviewed, but **the 15-row manual DevTools checklist has NOT been run.** There is no test infrastructure for the userscript, and by design it gains none here — a pasted copy of the logic in a scratch node script would drift from the real file the moment either changed. The author is shipping to prod and verifying there.

The rows most worth checking first, because each maps to a specific defect the review caught:

- Offline → archive X → online → open X's **series page** and nothing else → X must stay Archived. *(the Critical above; nothing else exercises it)*
- Slow 3G → queue a write → tap the pending chip → immediately archive the same series → final state must match the last tap.
- Queue a write → reload on a slow link → the card must not flap back during `init`.
- Offline → archive X, then remove X → one entry, `op:"delete"` → after reconnecting, X must not reappear.
- Empty queue → navigate for a minute → no chip and **no extra network requests** from `onNavigate`.

## Known limitations, deliberately not fixed here

- A `400` drops the entry and toasts, but local state keeps asserting the lost change until the next successful `GET`.
- A `401` on a live tap toasts "will sync when online" rather than the auth message; the user learns the truth on the next drain.
- Two same-origin tabs clobber each other's `mangabm:queue` — the queue is read once at boot and each save writes the whole array. Same idiom as the pre-existing `saveCache`; low risk on mobile Bromite.
- A pending favourite floats to the top of the list until it syncs, then settles back. Consistent with how optimistic writes already behaved.

Reviewed-on: #5
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>

2026-07-27 17:52:16 +07:00

12 KiB

Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Status

Greenfield. Only plans/mangaBookmark.md exists — no code yet. That plan is the spec; read it before building. Two deliverables: a Go sync backend and a single Bromite-compatible userscript.

What this is

A manga read-progress tracker for a user reading on asurascans.com (the current domain; asuracomic.net 301s here) and demonicscans.org from Bromite (mobile Chromium). A userscript injects on-page UI (floating button + slide-in panel) and syncs progress to a self-hosted Go backend so bookmarks unify across both sites and across devices.

Hard constraints (these drive the design — do not violate)

Bromite uses Chromium's native userscript engine, not Tampermonkey:

No GM_* APIs anywhere. No GM_setValue/GM_getValue (use page localStorage), no GM_registerMenuCommand (inject on-page UI), no GM_xmlhttpRequest for cross-origin (use plain fetch()). Keeping the script GM-free also lets it run in desktop Tampermonkey/Violentmonkey for faster iteration.
Cross-origin fetch() works only against a CORS-enabled backend. Manga sites are https://, so backend must be HTTPS (mixed-content block otherwise).
Asura and Demonic are separate origins with separate localStorage — a shared remote store is the only way to unify bookmarks. Cloud sync is required, not optional.
Userscript runs in an isolated world, so the embedded API token is safe from the site's JS.
Cloudflare's block on fetching the manga sites is IP-reputation-based, not universal — and not reliably reproducible. Verified 2026-07-26: plain curl from both the CGNAT dev machine and the deployed VPS got clean 200s with real HTML on both asurascans.com and demonicscans.org (homepage, series, chapter pages) — no interactive Turnstile challenge from either IP at test time. This contradicts an earlier, untested assumption that the CGNAT dev IP would be blocked; it was not, at least on this date. Treat "does curl work right now" as a live, time-varying fact to re-check, not a fixed property of a given machine — Cloudflare's bot scoring can flip a previously-clean IP without notice. Any backend fetcher still needs a graceful-degrade path for when it does get challenged, and adapters should be verified against live pages (Playwright MCP, on-device devtools, or a direct probe) before finalizing, not assumed from a single earlier test.

Architecture

Bromite userscript (isolated world, per-site adapters, localStorage cache)
   -- fetch() HTTPS -->  reverse proxy (TLS + CORS)  -->  Go net/http  -->  SQLite (volume)

Backend (backend/): stdlib net/http (a handful of routes, no framework) + modernc.org/sqlite (pure Go, CGO_ENABLED=0 -> static binary -> distroless/scratch image). The reverse proxy terminates TLS; the Go service listens plain :8080.
Single-user store. One bookmarks table keyed <site>:<series_id> (asura|demonic). Sync is last-write-wins. Schema and endpoint list are in the plan.
Endpoints: GET /bookmarks, PUT /bookmarks/{key} (upsert; see updated_at rule below), DELETE /bookmarks/{key}, GET /healthz (no auth).
Web UI: the same binary serves a password-gated browser UI on a second hostname — GET / (list, or login page when there is no session), POST /login, POST /logout, GET /static/*, and htmx fragment endpoints under /ui/*. Templates and assets are go:embed-ed, so backend/Dockerfile must copy templates/ and static/ as well as *.go. Sessions are stateless HMAC cookies keyed off API_TOKEN; WEB_PASSWORD gates them and, when empty, the web routes are not registered at all. UI mutations read-modify-write through Store.Get + Store.Upsert so the updated_at rule stays in one place. See docs/superpowers/specs/2026-07-25-web-ui-design.md.
Latest-chapter poller: a ticker goroutine in the same binary re-checks each bookmarked series' newest published chapter from the backend's own network access, so latest_chapter stays fresh when the user is not browsing. It is a second, parallel signal — the userscript keeps its own maybeCaptureLatestOnSeriesPage/backgroundRefreshLatest logic unchanged. Two independent clocks: a per-bookmark cooldown (latest_checked_at column, enforced by Store.DueForLatestCheck's WHERE clause) and a wake interval. The row is stamped before the fetch so a broken series waits out a full cooldown instead of retrying every tick, and writes go through Store.Get + Store.Upsert so a new chapter never reorders the list. Fetches use bogdanfinn/tls-client with a Chrome profile as defence in depth against fingerprint-based blocking; any failure logs and skips. See docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md. The poller's Store.Get + Store.Upsert is not wrapped in a transaction, so a userscript PUT that commits between the two can be overwritten by the poller's stale re-read — reverting that read progress and, since the stored value now differs, moving updated_at and reordering the list. This is a known, accepted limitation for a single-user deployment, not a bug to fix.
updated_at drives list order, so it moves only on real reading progress: the server applies its timestamp when the row is new or last_chapter_num changes, and otherwise keeps the stored value — favouriting a series or recording a newly published chapter must not reorder the list. PUT therefore returns the row as stored, and clients must adopt that response rather than their own payload. See plans/2026-07-25-bookmark-list-favorites-design.md §4.
Lifecycle buckets: status on each bookmark is reading | archived | finished, orthogonal to favorite. Archived and finished appear only in their own tab — not in All, Updated, Favourites, or the recent strip. The poller keeps checking archived series and skips finished ones. finished is settable only from the web UI; PUT /bookmarks/{key} rejects it with 400. An empty incoming status means "keep the stored one" — resolved on the VALUES side of Store.Upsert, not in the conflict clause, because excluded.* is the post-evaluation row and a default applied there would wipe the bucket on every PUT from a client that predates the column. See docs/superpowers/specs/2026-07-27-status-buckets-design.md.
Config via env: API_TOKEN, ALLOWED_ORIGINS (comma list), DB_PATH (default /data/bookmarks.db), PORT (default 8080), WEB_PASSWORD (gates the browser UI; unset disables it), LATEST_CHAPTER_POLL_ENABLED/_COOLDOWN/_INTERVAL/_BATCH/_STAGGER (background latest-chapter poller; defaults on, 1h/10m/14/20s).

Userscript structure (single IIFE, `manga-bookmark.user.js`)

Site adapters — one per host, detect(location, document) returns page type + IDs. Identify type/IDs from URL regex (most stable); pull title/cover from og:title/og:image meta tags, not CSS classes.
API client — apiGet/apiPut/apiDelete with bearer header; localStorage key mangabm:cache for instant render + offline fallback.
Progress logic — auto-upsert last_chapter only when chapterNum >= stored last_chapter_num (re-reading old chapters must not regress progress; unparseable -> set current). Manual panel override forces any value.
Retry queue — every write goes through pushBookmark/pushDelete, so a failed mutation is parked in localStorage (mangabm:queue) and replayed on the next navigation, reconnect, or refresh(). Entries are markers ({key, op, sendStatus, attempts}), never payloads — the body is read from the cache at send time, so one entry per key gives ordering and coalescing for free. sendStatus is sticky: while an archive is pending, later writes to that key keep carrying the bucket, which is what stops a successful in-between write from silently un-archiving the series. refresh() drains before it fetches and overlays anything still pending, so the list never flaps. A 400 drops the entry, a 401 aborts the pass and keeps the queue, and transient failures retry to a cap of 10. Latest-chapter writes deliberately stay out of the queue. See docs/superpowers/specs/2026-07-27-offline-retry-queue-design.md.
UI — rendered inside a Shadow DOM root to isolate from site CSS (critical on mobile). Three tabs (All / Favourites / Archived) and a row of link chips to the web UI and both manga sites; WEB_BASE sits in the CONFIG block next to API_BASE.
SPA navigation — Asura is Astro, client-routed on the comic/chapter pages: patch history.pushState/replaceState + listen popstate, re-run detect() on URL change so auto-update fires without reload. Demonic uses classic reloads (initial document-idle run suffices).

Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trusting)

asurascans.com: series /comics/<slug> (slug carries a trailing hash-like suffix, e.g. -f886a8af), chapter /comics/<slug>/chapter/<n>. Astro-rendered; chapter links are present in raw server HTML (no client-side-only render blocking a server fetch).
demonicscans.org: series /manga/<slug> (slug may URL-encode punctuation, e.g. %2527 for '), chapter /title/<slug>/chapter/<n>/<page> (the older chaptered.php?manga=<id>&chapter=<n> form still exists as a redirect and is what series-page chapter-list anchors link through).

Commands (once code exists)

Backend (cd backend):

Test all: go test ./...
Single test: go test -run TestName ./...
Build static binary: CGO_ENABLED=0 go build

Local stack: docker compose up (named volume mounted at /data, restart: unless-stopped).

Smoke test: curl the endpoints with Authorization: Bearer <token>; confirm OPTIONS preflight returns CORS headers and /healthz returns 200.

Forge: Gitea, not GitHub

origin is a self-hosted Gitea instance (gitea.violetcrown.my.id), so gh does not work here — use tea (Gitea CLI) for anything past plain git. Common ones:

Open a PR: tea pr create --head <branch> --base main --title "..." --description "..."
List / view / check out: tea pr list, tea pr <n>, tea pr checkout <n>
Issues: tea issue create, tea issue list
Auth lives in tea login, not a GH_TOKEN env var.

tea prints its output as rendered boxes rather than plain text; the PR URL lands on the last line.

Security invariants

Auth on /bookmarks*: require Authorization: Bearer <API_TOKEN>, constant-time compare, 401 otherwise.
CORS: reflect Origin only when in ALLOWED_ORIGINS; allow GET,PUT,DELETE,OPTIONS + headers Authorization,Content-Type; answer preflight OPTIONS with 204.

Relevant skills

multi-stage-dockerfile and docker-compose-orchestration for the container work (referenced in the plan).

graphify

This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.

Rules:

For codebase questions, first run graphify query "<question>" when graphify-out/graph.json exists. Use graphify path "<A>" "<B>" for relationships and graphify explain "<concept>" for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
After modifying code, run graphify update . to keep the graph current (AST-only, no API cost).

12 KiB Raw Blame History