From 83c3ffe3adbd83ae3b690127da9b63928ae0ffa5 Mon Sep 17 00:00:00 2001 From: Sulthan Zaki Date: Sat, 25 Jul 2026 11:40:17 +0700 Subject: [PATCH] docs: add background opportunistic refresh to latest-chapter design Same-origin fetch() from the userscript, throttled per-bookmark, to reduce the "only fresh when you open the exact series page" gap without server-side polling (still blocked by Cloudflare). Also documents that no JSON API or RSS feed exists on either site, ruling out a more stable poll target. Co-Authored-By: Claude Sonnet 5 --- ...26-07-25-bookmark-list-favorites-design.md | 80 +++++++++++++++++-- 1 file changed, 74 insertions(+), 6 deletions(-) diff --git a/plans/2026-07-25-bookmark-list-favorites-design.md b/plans/2026-07-25-bookmark-list-favorites-design.md index 380380e..da9042e 100644 --- a/plans/2026-07-25-bookmark-list-favorites-design.md +++ b/plans/2026-07-25-bookmark-list-favorites-design.md @@ -48,12 +48,25 @@ unconditionally on every upsert. See section 4. parsed chapter numbers (rather than assuming list order) is used for robustness on this site. -**Conclusion: latest-available-chapter data can only be refreshed when the -user visits a bookmarked series' series page** (not on every chapter open). -This is a real, disclosed limitation — not solved by polling, since the -backend cannot fetch these sites itself (Cloudflare blocks server-side -fetch, per existing CLAUDE.md constraint) and there is no reader-page source -for it. +**Conclusion: capturing latest-available-chapter data requires fetching a +series page's HTML** (no reader-page source, no JSON API — see below). The +backend cannot do this itself (Cloudflare blocks server-side fetch, per +existing CLAUDE.md constraint), so it must happen from the userscript, +running in the user's real browser session. Section "Background opportunistic +refresh" below extends this beyond "only when you open that exact series +page." + +### No stable API exists (checked live, 2026-07-25) + +- The series page is server-rendered HTML (Next.js SSR) — network capture + during a series-page load shows no `_next/data` JSON call and no XHR/fetch + for chapter data (only unrelated `api.asurascans.com` calls for + announcements/promotions banners). Chapter data is embedded directly in + the HTML, not fetched separately. +- No RSS/feed exists: `asurascans.com/feed`, `/rss`, `/rss.xml` all 404. +- Conclusion: HTML scraping (DOM when live on the page, raw-text regex when + background-fetched — see below) is the only available data source on + either site. There is nothing more "API-like" to poll instead. ### Behavior @@ -77,6 +90,54 @@ for it. from last-read; otherwise unchanged (`"Chapter 12 · asura"` as today). Plain inline text, no separate badge/count UI. +### Background opportunistic refresh (closer to "live") + +Live-page capture above only refreshes a series when the user happens to +open that exact series page. To reduce that gap without server-side +polling (still blocked by Cloudflare — verified: server-side fetch is a +datacenter request with no browser session, this is unchanged and not +being revisited), the userscript also does same-origin background checks +using the user's own real browser session: + +- **Same-origin only.** `fetch()` issued from a page on `asurascans.com` + can only safely reach other `asurascans.com` paths (no CORS trouble, + looks like a normal authenticated browser request). It cannot reach + `demonicscans.org` or vice versa. So visiting any page on a site + opportunistically refreshes only that site's bookmarks — confirmed live + that same-origin `fetch()` from an already-loaded page succeeds cleanly + (tested against `asurascans.com/sitemap.xml` from a `comics/*` page). +- **Trigger:** on every page load, after `init()`/`refresh()`, run + `backgroundRefreshLatest()`. It picks bookmarks belonging to the + *current* site that haven't been checked within a throttle window + (`LATEST_CHECK_THROTTLE_MS`, proposed 4 hours), oldest-checked-first, and + checks at most `LATEST_CHECK_BATCH` of them (proposed 1) per page load — + so a normal reading session gradually keeps bookmarks fresh without ever + bursting requests. +- **Freshness tracking is local-only**, not synced: a small + `mangabm:lastchecked` localStorage map of `{ [key]: timestampMs }`. It's + device-local by nature (each device does its own background checks) and + keeping it out of the synced `Bookmark` record avoids polluting the + cross-device schema with a per-device value. +- **Fetch + parse:** `fetch(bookmark.series_url)` → `res.text()` → apply + the *same* regex rule already defined above (Asura: chapter-link href + + "Chapter N" text; Demonic: max of all `chaptered.php?...chapter=` + matches) against the raw HTML string instead of the live DOM. No + additional parsing logic — this reuses the exact same rule, just fed + fetched text instead of `document`. +- **Update path:** identical to live-page capture — PUT the bookmark with + new `latest_chapter`/`latest_chapter_num` if changed, no `updated_at` + bump (section 4). +- **Failure handling:** a failed/blocked background fetch is silently + skipped (no toast, no retry loop) — next eligible page load tries again + naturally once the throttle window passes. +- **What this buys:** instead of "only fresh if you opened that exact + series page," bookmarks on a site you're actively reading converge to + "checked within the last ~4 hours," which is meaningfully closer to live + given frequent reading — without needing server-side scraping that + Cloudflare would block anyway. It is still not push/instant; nothing + can notify the moment a new chapter is posted without the site itself + offering that (it doesn't — no RSS/webhooks, confirmed above). + ## 3. Favorites - New field `favorite: bool` on the bookmark record, synced through the @@ -147,6 +208,10 @@ SQLite migration: additive `ALTER TABLE bookmarks ADD COLUMN ...` for each, guarded against "duplicate column" errors so it's safe to run against the already-deployed database. +Userscript-local, not synced: `mangabm:lastchecked` localStorage key, a +`{ [bookmarkKey]: timestampMs }` map used only to throttle background +refresh (see section 2). + ## Testing - Go: table-driven tests for `Store.Upsert`'s conditional `updated_at` logic @@ -158,6 +223,9 @@ already-deployed database. plain re-visit or favorite toggle. - latest-chapter capture fires on series-page visits and displays correctly for both sites. + - background refresh: check one stale same-site bookmark per page load, + skips bookmarks checked within the throttle window, updates silently + without reordering the list, and doesn't fire cross-site. - favorite toggle persists across a panel close/reopen and a `refresh()` (i.e. round-trips through the backend correctly). - favorited manga still appears in "All" tab.