Poll's lnw chapter scan: unscoped and comment-truncated #87
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent
#80 (spec) — implements part of #77.
What to build
A lightnovelworld Series whose chapters are published under more than one Chapter Slug currently stalls: the Poll scopes its scan to the one slug it derived from the stored address, so chapters served under the other slug are invisible and the Latest Chapter freezes. One sampled novel serves chapters 1-99 under one Chapter Slug and 100-423 under another, both resolving, both pointing back at the same Series. No stored-slug approach can fix this — the scan must stop being scoped at all and match any chapter-shaped address on the host.
An unscoped scan reads too much, though: the Site server-renders a wpdiscuz comment thread below the chapter list, and a comment body is HTML that can carry an anchor. A stranger's comment would then be able to set the Latest Chapter on a row shared by every Reader who tracks that Series. So the scan runs against the body truncated at the first occurrence of the comment-thread marker. The marker is the one chosen by measurement — it occurs exactly once per page and follows every chapter anchor on all four Series pages sampled, whereas the two obvious alternatives occur 111 to 143 times per page including in
<head>, before the chapter list.Fail closed. Marker absent means skip the Series and log it. It does not mean scan the whole page — a Site redesign must degrade into staleness, never into a wrong shared value. The log line carries the body length, because the marker sits at roughly 94% of these documents (byte 646,329 of 685,023 on one page; 1,117,457 of 1,183,036 on another) and a body cut short by the size cap is otherwise indistinguishable from a markup change.
A container-scoped match was considered and rejected: the container the research note proposed turns out to be the hidden, empty "Latest Reading" template, and the real list is classless. The backend scans raw HTML with no parser, so container extraction means a second regex against class names — more fragile than truncation and protecting nothing extra.
While here: the body-size cap's comment claims roughly 10x headroom. On this Site it is about 3.5x. Correct the comment to the measured figure.
Only lightnovelworld changes. asura, demonic, comix, kagane and novelfull keep their current scoping.
Acceptance criteria
cd backend && go test ./...passesBlocked by
Landed on main (merge of ticket/87-lnw-scan-unscoped; commits
ecce453,51b0094).The lightnovelworld branch of latestChapterFrom no longer parses the stored address or compiles a per-Series pattern. It cuts the body at the first
wpd-threadsand scans the prefix with a package-level lnwChapterRe matching any chapter-shaped address on the host. Marker absent returns ok=false and logs the series URL, the marker and the body length — never a whole-page scan. The maxBodyBytes comment now records the measured ~3.5x headroom for this Site (685 KB and 1.18 MB pages) instead of the old 10x claim; the constant is unchanged.lnwSlugRe is deleted and the lnw trailing-slash table case removed: the branch no longer parses seriesURL at all, so both were dead. Clean cutover rather than dead code left behind.
21/21 TestLatestChapterFrom subtests pass, go test ./... green, gofmt -l internal/latest/ silent, and the asura, demonic, comix, kagane and novelfull branches are unchanged.
All six acceptance criteria met. The comment-poisoning case went red after the scoping came out and before the truncation went in, which is the proof that the truncation is load-bearing rather than decorative.