docs: add background opportunistic refresh to latest-chapter design
Same-origin fetch() from the userscript, throttled per-bookmark, to reduce the "only fresh when you open the exact series page" gap without server-side polling (still blocked by Cloudflare). Also documents that no JSON API or RSS feed exists on either site, ruling out a more stable poll target. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -48,12 +48,25 @@ unconditionally on every upsert. See section 4.
|
||||
parsed chapter numbers (rather than assuming list order) is used for
|
||||
robustness on this site.
|
||||
|
||||
**Conclusion: latest-available-chapter data can only be refreshed when the
|
||||
user visits a bookmarked series' series page** (not on every chapter open).
|
||||
This is a real, disclosed limitation — not solved by polling, since the
|
||||
backend cannot fetch these sites itself (Cloudflare blocks server-side
|
||||
fetch, per existing CLAUDE.md constraint) and there is no reader-page source
|
||||
for it.
|
||||
**Conclusion: capturing latest-available-chapter data requires fetching a
|
||||
series page's HTML** (no reader-page source, no JSON API — see below). The
|
||||
backend cannot do this itself (Cloudflare blocks server-side fetch, per
|
||||
existing CLAUDE.md constraint), so it must happen from the userscript,
|
||||
running in the user's real browser session. Section "Background opportunistic
|
||||
refresh" below extends this beyond "only when you open that exact series
|
||||
page."
|
||||
|
||||
### No stable API exists (checked live, 2026-07-25)
|
||||
|
||||
- The series page is server-rendered HTML (Next.js SSR) — network capture
|
||||
during a series-page load shows no `_next/data` JSON call and no XHR/fetch
|
||||
for chapter data (only unrelated `api.asurascans.com` calls for
|
||||
announcements/promotions banners). Chapter data is embedded directly in
|
||||
the HTML, not fetched separately.
|
||||
- No RSS/feed exists: `asurascans.com/feed`, `/rss`, `/rss.xml` all 404.
|
||||
- Conclusion: HTML scraping (DOM when live on the page, raw-text regex when
|
||||
background-fetched — see below) is the only available data source on
|
||||
either site. There is nothing more "API-like" to poll instead.
|
||||
|
||||
### Behavior
|
||||
|
||||
@@ -77,6 +90,54 @@ for it.
|
||||
from last-read; otherwise unchanged (`"Chapter 12 · asura"` as today).
|
||||
Plain inline text, no separate badge/count UI.
|
||||
|
||||
### Background opportunistic refresh (closer to "live")
|
||||
|
||||
Live-page capture above only refreshes a series when the user happens to
|
||||
open that exact series page. To reduce that gap without server-side
|
||||
polling (still blocked by Cloudflare — verified: server-side fetch is a
|
||||
datacenter request with no browser session, this is unchanged and not
|
||||
being revisited), the userscript also does same-origin background checks
|
||||
using the user's own real browser session:
|
||||
|
||||
- **Same-origin only.** `fetch()` issued from a page on `asurascans.com`
|
||||
can only safely reach other `asurascans.com` paths (no CORS trouble,
|
||||
looks like a normal authenticated browser request). It cannot reach
|
||||
`demonicscans.org` or vice versa. So visiting any page on a site
|
||||
opportunistically refreshes only that site's bookmarks — confirmed live
|
||||
that same-origin `fetch()` from an already-loaded page succeeds cleanly
|
||||
(tested against `asurascans.com/sitemap.xml` from a `comics/*` page).
|
||||
- **Trigger:** on every page load, after `init()`/`refresh()`, run
|
||||
`backgroundRefreshLatest()`. It picks bookmarks belonging to the
|
||||
*current* site that haven't been checked within a throttle window
|
||||
(`LATEST_CHECK_THROTTLE_MS`, proposed 4 hours), oldest-checked-first, and
|
||||
checks at most `LATEST_CHECK_BATCH` of them (proposed 1) per page load —
|
||||
so a normal reading session gradually keeps bookmarks fresh without ever
|
||||
bursting requests.
|
||||
- **Freshness tracking is local-only**, not synced: a small
|
||||
`mangabm:lastchecked` localStorage map of `{ [key]: timestampMs }`. It's
|
||||
device-local by nature (each device does its own background checks) and
|
||||
keeping it out of the synced `Bookmark` record avoids polluting the
|
||||
cross-device schema with a per-device value.
|
||||
- **Fetch + parse:** `fetch(bookmark.series_url)` → `res.text()` → apply
|
||||
the *same* regex rule already defined above (Asura: chapter-link href +
|
||||
"Chapter N" text; Demonic: max of all `chaptered.php?...chapter=`
|
||||
matches) against the raw HTML string instead of the live DOM. No
|
||||
additional parsing logic — this reuses the exact same rule, just fed
|
||||
fetched text instead of `document`.
|
||||
- **Update path:** identical to live-page capture — PUT the bookmark with
|
||||
new `latest_chapter`/`latest_chapter_num` if changed, no `updated_at`
|
||||
bump (section 4).
|
||||
- **Failure handling:** a failed/blocked background fetch is silently
|
||||
skipped (no toast, no retry loop) — next eligible page load tries again
|
||||
naturally once the throttle window passes.
|
||||
- **What this buys:** instead of "only fresh if you opened that exact
|
||||
series page," bookmarks on a site you're actively reading converge to
|
||||
"checked within the last ~4 hours," which is meaningfully closer to live
|
||||
given frequent reading — without needing server-side scraping that
|
||||
Cloudflare would block anyway. It is still not push/instant; nothing
|
||||
can notify the moment a new chapter is posted without the site itself
|
||||
offering that (it doesn't — no RSS/webhooks, confirmed above).
|
||||
|
||||
## 3. Favorites
|
||||
|
||||
- New field `favorite: bool` on the bookmark record, synced through the
|
||||
@@ -147,6 +208,10 @@ SQLite migration: additive `ALTER TABLE bookmarks ADD COLUMN ...` for each,
|
||||
guarded against "duplicate column" errors so it's safe to run against the
|
||||
already-deployed database.
|
||||
|
||||
Userscript-local, not synced: `mangabm:lastchecked` localStorage key, a
|
||||
`{ [bookmarkKey]: timestampMs }` map used only to throttle background
|
||||
refresh (see section 2).
|
||||
|
||||
## Testing
|
||||
|
||||
- Go: table-driven tests for `Store.Upsert`'s conditional `updated_at` logic
|
||||
@@ -158,6 +223,9 @@ already-deployed database.
|
||||
plain re-visit or favorite toggle.
|
||||
- latest-chapter capture fires on series-page visits and displays
|
||||
correctly for both sites.
|
||||
- background refresh: check one stale same-site bookmark per page load,
|
||||
skips bookmarks checked within the throttle window, updates silently
|
||||
without reordering the list, and doesn't fire cross-site.
|
||||
- favorite toggle persists across a panel close/reopen and a
|
||||
`refresh()` (i.e. round-trips through the backend correctly).
|
||||
- favorited manga still appears in "All" tab.
|
||||
|
||||
Reference in New Issue
Block a user