docs: add background opportunistic refresh to latest-chapter design

Same-origin fetch() from the userscript, throttled per-bookmark, to reduce
the "only fresh when you open the exact series page" gap without server-side
polling (still blocked by Cloudflare). Also documents that no JSON API or
RSS feed exists on either site, ruling out a more stable poll target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 11:40:17 +07:00
parent 0f7d611a60
commit 83c3ffe3ad
@@ -48,12 +48,25 @@ unconditionally on every upsert. See section 4.
parsed chapter numbers (rather than assuming list order) is used for
robustness on this site.
**Conclusion: latest-available-chapter data can only be refreshed when the
user visits a bookmarked series' series page** (not on every chapter open).
This is a real, disclosed limitation — not solved by polling, since the
backend cannot fetch these sites itself (Cloudflare blocks server-side
fetch, per existing CLAUDE.md constraint) and there is no reader-page source
for it.
**Conclusion: capturing latest-available-chapter data requires fetching a
series page's HTML** (no reader-page source, no JSON API — see below). The
backend cannot do this itself (Cloudflare blocks server-side fetch, per
existing CLAUDE.md constraint), so it must happen from the userscript,
running in the user's real browser session. Section "Background opportunistic
refresh" below extends this beyond "only when you open that exact series
page."
### No stable API exists (checked live, 2026-07-25)
- The series page is server-rendered HTML (Next.js SSR) — network capture
during a series-page load shows no `_next/data` JSON call and no XHR/fetch
for chapter data (only unrelated `api.asurascans.com` calls for
announcements/promotions banners). Chapter data is embedded directly in
the HTML, not fetched separately.
- No RSS/feed exists: `asurascans.com/feed`, `/rss`, `/rss.xml` all 404.
- Conclusion: HTML scraping (DOM when live on the page, raw-text regex when
background-fetched — see below) is the only available data source on
either site. There is nothing more "API-like" to poll instead.
### Behavior
@@ -77,6 +90,54 @@ for it.
from last-read; otherwise unchanged (`"Chapter 12 · asura"` as today).
Plain inline text, no separate badge/count UI.
### Background opportunistic refresh (closer to "live")
Live-page capture above only refreshes a series when the user happens to
open that exact series page. To reduce that gap without server-side
polling (still blocked by Cloudflare — verified: server-side fetch is a
datacenter request with no browser session, this is unchanged and not
being revisited), the userscript also does same-origin background checks
using the user's own real browser session:
- **Same-origin only.** `fetch()` issued from a page on `asurascans.com`
can only safely reach other `asurascans.com` paths (no CORS trouble,
looks like a normal authenticated browser request). It cannot reach
`demonicscans.org` or vice versa. So visiting any page on a site
opportunistically refreshes only that site's bookmarks — confirmed live
that same-origin `fetch()` from an already-loaded page succeeds cleanly
(tested against `asurascans.com/sitemap.xml` from a `comics/*` page).
- **Trigger:** on every page load, after `init()`/`refresh()`, run
`backgroundRefreshLatest()`. It picks bookmarks belonging to the
*current* site that haven't been checked within a throttle window
(`LATEST_CHECK_THROTTLE_MS`, proposed 4 hours), oldest-checked-first, and
checks at most `LATEST_CHECK_BATCH` of them (proposed 1) per page load —
so a normal reading session gradually keeps bookmarks fresh without ever
bursting requests.
- **Freshness tracking is local-only**, not synced: a small
`mangabm:lastchecked` localStorage map of `{ [key]: timestampMs }`. It's
device-local by nature (each device does its own background checks) and
keeping it out of the synced `Bookmark` record avoids polluting the
cross-device schema with a per-device value.
- **Fetch + parse:** `fetch(bookmark.series_url)` → `res.text()` → apply
the *same* regex rule already defined above (Asura: chapter-link href +
"Chapter N" text; Demonic: max of all `chaptered.php?...chapter=`
matches) against the raw HTML string instead of the live DOM. No
additional parsing logic — this reuses the exact same rule, just fed
fetched text instead of `document`.
- **Update path:** identical to live-page capture — PUT the bookmark with
new `latest_chapter`/`latest_chapter_num` if changed, no `updated_at`
bump (section 4).
- **Failure handling:** a failed/blocked background fetch is silently
skipped (no toast, no retry loop) — next eligible page load tries again
naturally once the throttle window passes.
- **What this buys:** instead of "only fresh if you opened that exact
series page," bookmarks on a site you're actively reading converge to
"checked within the last ~4 hours," which is meaningfully closer to live
given frequent reading — without needing server-side scraping that
Cloudflare would block anyway. It is still not push/instant; nothing
can notify the moment a new chapter is posted without the site itself
offering that (it doesn't — no RSS/webhooks, confirmed above).
## 3. Favorites
- New field `favorite: bool` on the bookmark record, synced through the
@@ -147,6 +208,10 @@ SQLite migration: additive `ALTER TABLE bookmarks ADD COLUMN ...` for each,
guarded against "duplicate column" errors so it's safe to run against the
already-deployed database.
Userscript-local, not synced: `mangabm:lastchecked` localStorage key, a
`{ [bookmarkKey]: timestampMs }` map used only to throttle background
refresh (see section 2).
## Testing
- Go: table-driven tests for `Store.Upsert`'s conditional `updated_at` logic
@@ -158,6 +223,9 @@ already-deployed database.
plain re-visit or favorite toggle.
- latest-chapter capture fires on series-page visits and displays
correctly for both sites.
- background refresh: check one stale same-site bookmark per page load,
skips bookmarks checked within the throttle window, updates silently
without reordering the list, and doesn't fire cross-site.
- favorite toggle persists across a panel close/reopen and a
`refresh()` (i.e. round-trips through the backend correctly).
- favorited manga still appears in "All" tab.