sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

lightnovelworld: chapter slug != series slug, so poller 404s on renamed novels #77

New Issue
Closed
opened 2026-08-10 19:29:32 +07:00 by sulthan · 2 comments
sulthan commented 2026-08-10 19:29:32 +07:00
Owner
Copy Link
Copy Source

Symptom

VPS poller log, 2026-08-10:

latest poll "lightnovelworld:my-longevity-simulation": fetch https://lightnovelworld.net/novel/my-longevity-simulation/: status 404

Root cause

Not a URL parse bug — the parse is fine, the assumption behind it is wrong.

novel-bookmark.user.js (lightnovelworld.detect, L143-158) synthesises the series
identity from the chapter path: /<slug>-chapter-<n>/ -> seriesId = <slug>,
seriesUrl = https://lightnovelworld.net/novel/<slug>/. That assumes the chapter slug
and the series slug are the same string.

On lightnovelworld they can diverge — a renamed novel keeps its old chapter slugs.
Verified live 2026-08-10:

URL Result
https://lightnovelworld.net/my-longevity-simulation-chapter-1/ 200
https://lightnovelworld.net/novel/my-longevity-simulation/ 404
https://lightnovelworld.net/novel/immortality-simulator/ 200, <h1 class="entry-title">Immortality Simulator</h1>

The chapter page's own breadcrumb points at the series:

<a itemprop="item" href="https://lightnovelworld.net/novel/immortality-simulator/">
<a href='https://lightnovelworld.net/novel/immortality-simulator/' aria-label='All Chapter'>

and that series page's 1234 chapter anchors are all
https://lightnovelworld.net/my-longevity-simulation-chapter-<n>/.

So for this novel: series slug immortality-simulator, chapter slug
my-longevity-simulation. Two independent facts, and the userscript stores only one.

Consequences

  1. Poller 404s forever. series_url is written once at creation (ADR-0003) and never
    corrected, so this series never gets a Latest Chapter or a cover, and burns a fetch
    every cooldown.

  2. Fixing only series_url is not enough. Both chapter scanners key the chapter
    pattern off the series slug:

    • backend/internal/latest/sites.go:128-137 — lnwSlugRe pulls <slug> out of
      /novel/<slug>/ and builds lightnovelworld\.net/<slug>-chapter-([0-9.]+)/.
    • novel-bookmark.user.js:177-182 — same regex, from seriesId.

    With series_url = /novel/immortality-simulator/ the page fetches 200 but matches
    zero anchors (they all say my-longevity-simulation-chapter-N). The scan needs the
    chapter slug, which the series URL does not carry.

  3. Duplicate rows. Bookmarking from the series page yields key
    lightnovelworld:immortality-simulator; from a chapter page,
    lightnovelworld:my-longevity-simulation. One novel, two Series rows.

Fix direction

The chapter page already links its own series — read it instead of synthesising it.
In lightnovelworld.detect's chapter branch, take seriesUrl/seriesId from the
breadcrumb anchor (a[href*="/novel/"], or link[rel=canonical] on the series page),
falling back to the current slug guess when absent. That fixes new bookmarks and the
duplicate-key hazard.

The scanner still needs the chapter slug as a separate fact. Cheapest option that keeps
the existing store shape: on lightnovelworld, drop the per-series scoping and match
lightnovelworld\.net/[a-z0-9-]+-chapter-([0-9.]+)/ against the series page only —
the page lists one novel's chapters, unlike novelfull/asura which carry
recommendation strips (a "latest updates" sidebar would need checking first). Otherwise
the chapter slug has to be stored alongside series_id, which is a schema change.

Existing rows are wrong on disk either way and need a one-off correction
(series_url + key) or a re-bookmark.

Scope

  • userscript/novel-bookmark.user.js (lightnovelworld.detect, latestChapterFromAnchors)
  • backend/internal/latest/sites.go (lnwSlugRe, latestChapterFrom lnw branch)
  • fixtures: userscript/test/novel-logic.test.js, backend/internal/latest/sites_test.go
    — both currently use a-will-eternal, where the two slugs happen to coincide, which is
    why this slipped through. Add a divergent-slug fixture.
## Symptom VPS poller log, 2026-08-10: ``` latest poll "lightnovelworld:my-longevity-simulation": fetch https://lightnovelworld.net/novel/my-longevity-simulation/: status 404 ``` ## Root cause Not a URL *parse* bug — the parse is fine, the **assumption** behind it is wrong. `novel-bookmark.user.js` (`lightnovelworld.detect`, L143-158) synthesises the series identity from the *chapter* path: `/<slug>-chapter-<n>/` -> `seriesId = <slug>`, `seriesUrl = https://lightnovelworld.net/novel/<slug>/`. That assumes the chapter slug and the series slug are the same string. On lightnovelworld they can diverge — a renamed novel keeps its old chapter slugs. Verified live 2026-08-10: | URL | Result | | --- | --- | | `https://lightnovelworld.net/my-longevity-simulation-chapter-1/` | 200 | | `https://lightnovelworld.net/novel/my-longevity-simulation/` | **404** | | `https://lightnovelworld.net/novel/immortality-simulator/` | 200, `<h1 class="entry-title">Immortality Simulator</h1>` | The chapter page's own breadcrumb points at the series: ```html <a itemprop="item" href="https://lightnovelworld.net/novel/immortality-simulator/"> <a href='https://lightnovelworld.net/novel/immortality-simulator/' aria-label='All Chapter'> ``` and that series page's 1234 chapter anchors are all `https://lightnovelworld.net/my-longevity-simulation-chapter-<n>/`. So for this novel: series slug `immortality-simulator`, chapter slug `my-longevity-simulation`. Two independent facts, and the userscript stores only one. ## Consequences 1. **Poller 404s forever.** `series_url` is written once at creation (ADR-0003) and never corrected, so this series never gets a Latest Chapter or a cover, and burns a fetch every cooldown. 2. **Fixing only `series_url` is not enough.** Both chapter scanners key the chapter pattern off the *series* slug: - `backend/internal/latest/sites.go:128-137` — `lnwSlugRe` pulls `<slug>` out of `/novel/<slug>/` and builds `lightnovelworld\.net/<slug>-chapter-([0-9.]+)/`. - `novel-bookmark.user.js:177-182` — same regex, from `seriesId`. With `series_url = /novel/immortality-simulator/` the page fetches 200 but matches **zero** anchors (they all say `my-longevity-simulation-chapter-N`). The scan needs the chapter slug, which the series URL does not carry. 3. **Duplicate rows.** Bookmarking from the series page yields key `lightnovelworld:immortality-simulator`; from a chapter page, `lightnovelworld:my-longevity-simulation`. One novel, two Series rows. ## Fix direction The chapter page already links its own series — read it instead of synthesising it. In `lightnovelworld.detect`'s chapter branch, take `seriesUrl`/`seriesId` from the breadcrumb anchor (`a[href*="/novel/"]`, or `link[rel=canonical]` on the series page), falling back to the current slug guess when absent. That fixes new bookmarks and the duplicate-key hazard. The scanner still needs the chapter slug as a separate fact. Cheapest option that keeps the existing store shape: on lightnovelworld, drop the per-series scoping and match `lightnovelworld\.net/[a-z0-9-]+-chapter-([0-9.]+)/` against the series page only — the page lists one novel's chapters, unlike novelfull/asura which carry recommendation strips (a "latest updates" sidebar would need checking first). Otherwise the chapter slug has to be stored alongside `series_id`, which is a schema change. Existing rows are wrong on disk either way and need a one-off correction (`series_url` + key) or a re-bookmark. ## Scope - `userscript/novel-bookmark.user.js` (`lightnovelworld.detect`, `latestChapterFromAnchors`) - `backend/internal/latest/sites.go` (`lnwSlugRe`, `latestChapterFrom` lnw branch) - fixtures: `userscript/test/novel-logic.test.js`, `backend/internal/latest/sites_test.go` — both currently use `a-will-eternal`, where the two slugs happen to coincide, which is why this slipped through. Add a divergent-slug fixture.
sulthan added the ready-for-agentbug labels 2026-08-10 19:29:32 +07:00
sulthan commented 2026-08-11 04:38:22 +07:00
Author
Owner
Copy Link
Copy Source

This was generated by AI during triage.

Research addendum — live survey of lightnovelworld, 2026-08-11

Full note committed at docs/research/lightnovelworld-chapter-vs-series-slug.md
(41 distinct novels sampled, plain curl, no Cloudflare challenge on any request).
It confirms the diagnosis and corrects three assumptions in the issue body.

Confirmed

  • /my-longevity-simulation-chapter-1/ 200, /novel/immortality-simulator/ 200,
    /novel/my-longevity-simulation/ 404, 0 redirects. The constructed series URL is
    permanently dead — no alias, no 301.
  • The chapter page does link its own series. Best pointer is a single element:
    <a href='https://lightnovelworld.net/novel/immortality-simulator/' aria-label='All Chapter'>
    (note single-quoted attributes). Microdata breadcrumb position 2 is an equally good
    second, and its <span itemprop="name"> yields the correct series title directly.

Correction 1 — the unscoped regex is SAFE, and the existing fixture is wrong

lnwSeriesFixture in sites_test.go:75-80 carries an overgeared-chapter-9999/ anchor
with the comment "The last anchor is another series'." No sampled lightnovelworld
series page contains a foreign chapter anchor. Across 28 real series pages, 27 had
exactly one distinct chapter-slug prefix and the 28th had two that both belong to that
same novel. Recommendation strips on a series page link /novel/<slug>/ only, which the
pattern cannot match.

The one widget that would carry foreign chapter links — "Latest Reading" — is an inert
client-side mustache template (#series-history is display:none with an empty <ul>,
rows live in #series-history-tpl with href="#/{{number}}") populated from the
visitor's local history. A server-side fetch never sees content there.

So: relax the regex, and fix the fixture rather than preserving the scoping. Caveat
worth a comment in the code: this is an empirical guarantee, not a structural one like
asura/novelfull. Narrowing the match to the ul.clstyle chapter-list container would
restore a structural guarantee cheaply.

Correction 2 — no stored-slug fix can work, because a slug can change mid-series

/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/ carries two chapter
slugs in one page: …-one-skill-not for chapters 1–99, …-one-skill-not-them-all for
100–423. Both resolve, both breadcrumb back to the same series. This kills the
"store the chapter slug alongside series_id" alternative outright — one novel legitimately
has more than one. The unscoped regex is the only option that handles it.

Correction 3 — the divergence has no derivable rule, in either direction

Not "chapters keep the old title". It goes both ways:

Series slug Chapter slug Which side is the polished title
immortality-simulator my-longevity-simulation series
the-sword-illuminates-the-great-wilderness radiant-blade-of-the-wilderness chapter
a-villains-will-to-survive the-villain-wants-to-live series

3 of 41 sampled novels (~7%) diverge at their latest chapter. There is no
"Alternative names" field on lightnovelworld series pages (the info panel exposes only
Author / Released / Type / Status), so the mapping exists only in the chapter anchors.
Neither slug is computable from the other.

Also settled

  • The full chapter list is inline in the series page HTML — 1232 chapters in one document
    for immortality-simulator, 2666 for greed-all-for-what. /novel/<slug>/chapters/ and
    /chapters/page-2 both 404; there is no pagination to follow.
  • All chapter hrefs on a series page are absolute; zero relative matches. A pattern
    anchored on the literal host is safe.
  • No chapter-slug → series-slug endpoint exists. /<chapter-slug>/ (no -chapter-N) 302s
    to chapter 1, which still requires parsing that chapter page.

Open decision for the implementer

series_id is currently the chapter slug, so the row key is
lightnovelworld:my-longevity-simulation. sites.go polls off the stored series_url,
not series_id, so storing the correct series_url alone is sufficient to fix the poller
without a re-key. Re-keying to the real series slug additionally fixes the duplicate-row
hazard (bookmarking from the series page vs. a chapter page) but requires migrating
existing rows. Worth a maintainer call before implementation — flagging rather than
deciding it here.

Note fetchableSeriesURL (poller.go:321-324) only checks the hostname, so it happily
accepts the dead /novel/<chapter-slug>/ URL; the gate is not a backstop for this.

> *This was generated by AI during triage.* ## Research addendum — live survey of lightnovelworld, 2026-08-11 Full note committed at `docs/research/lightnovelworld-chapter-vs-series-slug.md` (41 distinct novels sampled, plain `curl`, no Cloudflare challenge on any request). It confirms the diagnosis and **corrects three assumptions** in the issue body. ### Confirmed - `/my-longevity-simulation-chapter-1/` **200**, `/novel/immortality-simulator/` **200**, `/novel/my-longevity-simulation/` **404, 0 redirects**. The constructed series URL is permanently dead — no alias, no 301. - The chapter page does link its own series. Best pointer is a single element: `<a href='https://lightnovelworld.net/novel/immortality-simulator/' aria-label='All Chapter'>` (note **single-quoted** attributes). Microdata breadcrumb `position 2` is an equally good second, and its `<span itemprop="name">` yields the correct series title directly. ### Correction 1 — the unscoped regex is SAFE, and the existing fixture is wrong `lnwSeriesFixture` in `sites_test.go:75-80` carries an `overgeared-chapter-9999/` anchor with the comment *"The last anchor is another series'."* **No sampled lightnovelworld series page contains a foreign chapter anchor.** Across 28 real series pages, 27 had exactly one distinct chapter-slug prefix and the 28th had two that both belong to that same novel. Recommendation strips on a series page link `/novel/<slug>/` only, which the pattern cannot match. The one widget that would carry foreign chapter links — "Latest Reading" — is an inert client-side mustache template (`#series-history` is `display:none` with an empty `<ul>`, rows live in `#series-history-tpl` with `href="#/{{number}}"`) populated from the visitor's local history. A server-side fetch never sees content there. So: relax the regex, and **fix the fixture** rather than preserving the scoping. Caveat worth a comment in the code: this is an *empirical* guarantee, not a structural one like asura/novelfull. Narrowing the match to the `ul.clstyle` chapter-list container would restore a structural guarantee cheaply. ### Correction 2 — no stored-slug fix can work, because a slug can change mid-series `/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/` carries **two** chapter slugs in one page: `…-one-skill-not` for chapters 1–99, `…-one-skill-not-them-all` for 100–423. Both resolve, both breadcrumb back to the same series. This kills the "store the chapter slug alongside `series_id`" alternative outright — one novel legitimately has more than one. The unscoped regex is the only option that handles it. ### Correction 3 — the divergence has no derivable rule, in either direction Not "chapters keep the old title". It goes both ways: | Series slug | Chapter slug | Which side is the polished title | | --- | --- | --- | | `immortality-simulator` | `my-longevity-simulation` | series | | `the-sword-illuminates-the-great-wilderness` | `radiant-blade-of-the-wilderness` | chapter | | `a-villains-will-to-survive` | `the-villain-wants-to-live` | series | 3 of 41 sampled novels (~7%) diverge at their latest chapter. There is **no "Alternative names" field** on lightnovelworld series pages (the info panel exposes only Author / Released / Type / Status), so the mapping exists *only* in the chapter anchors. Neither slug is computable from the other. ### Also settled - The full chapter list is inline in the series page HTML — 1232 chapters in one document for `immortality-simulator`, 2666 for `greed-all-for-what`. `/novel/<slug>/chapters/` and `/chapters/page-2` both **404**; there is no pagination to follow. - All chapter hrefs on a series page are **absolute**; zero relative matches. A pattern anchored on the literal host is safe. - No chapter-slug → series-slug endpoint exists. `/<chapter-slug>/` (no `-chapter-N`) 302s to chapter 1, which still requires parsing that chapter page. ### Open decision for the implementer `series_id` is currently the **chapter** slug, so the row key is `lightnovelworld:my-longevity-simulation`. `sites.go` polls off the stored `series_url`, not `series_id`, so storing the correct `series_url` alone is sufficient to fix the poller without a re-key. Re-keying to the real series slug additionally fixes the duplicate-row hazard (bookmarking from the series page vs. a chapter page) but requires migrating existing rows. Worth a maintainer call before implementation — flagging rather than deciding it here. Note `fetchableSeriesURL` (`poller.go:321-324`) only checks the hostname, so it happily accepts the dead `/novel/<chapter-slug>/` URL; the gate is not a backstop for this.
sulthan commented 2026-08-11 09:27:59 +07:00
Author
Owner
Copy Link
Copy Source

Spec published as #80 (ready-for-agent). Design settled in a grilling session on 2026-08-11: identity is read from the chapter page's All Chapter pointer rather than derived from the address, the Poll's scan is unscoped and truncated before the comment thread, and existing rows are repaired client-side when the Reader next opens a chapter page. ADR-0008 records the decision; docs/research/lightnovelworld-chapter-vs-series-slug.md carries the evidence. Two corrections landed against this issue's own text: the ul.clstyle container named in the research note is the hidden empty template, not the chapter list, and the suggested series_url-only repair was rejected because it keeps an identity the Site does not guarantee.

Spec published as #80 (`ready-for-agent`). Design settled in a grilling session on 2026-08-11: identity is read from the chapter page's `All Chapter` pointer rather than derived from the address, the Poll's scan is unscoped and truncated before the comment thread, and existing rows are repaired client-side when the Reader next opens a chapter page. ADR-0008 records the decision; `docs/research/lightnovelworld-chapter-vs-series-slug.md` carries the evidence. Two corrections landed against this issue's own text: the `ul.clstyle` container named in the research note is the hidden empty template, not the chapter list, and the suggested `series_url`-only repair was rejected because it keeps an identity the Site does not guarantee.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is broken
 enhancement
New feature or improvement
 needs-info
Waiting on reporter for more information
 needs-triage
Maintainer needs to evaluate
 ready-for-agent
Fully specified, ready for an AFK agent
 ready-for-human
Needs human implementation
 wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wontfix
Will not be actioned
No Label bug ready-for-agent
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
sulthan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sulthan/mangaBookmark#77
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?