Research: completion markers on the six Sites' Series pages (#129)

docs/research/completion-markers.md

@@ -0,0 +1,375 @@
+# Machine-readable completion markers on the six Sites' Series pages
+
+Research note for Gitea issue #129 (part of #114, "dashboard suggests *potentially
+finished* Series"): for each of the six Sites, does the Series page publish a
+completion marker a Poll could read — an explicit status field, not a
+heuristic — and how trustworthy is it? The dashboard side (storage,
+presentation, containment) is issue #130's job, not this note's.
+
+All facts fetched live on **2026-08-19**. Sources are the live pages (URLs
+below, fetched 2026-08-19) and the in-tree adapter code
+(`backend/internal/latest/…`, branch `main` @ `f5d3fe5`). Every claim carries a
+URL or a `file:line`. Interpretation rather than observation is marked
+`[INFERENCE]`. Raw fetches archived in `/tmp/cm/` (throwaway).
+
+Fetch method per Site: plain TLS (`curl`) where the site serves real HTML;
+real Chrome via the repo's CDP sidecar pattern (Playwright) where a Cloudflare
+challenge blocks plain fetch — comix.to and kagane.to always, novelfull.com
+time-varying (today it 403'd plain fetch and cleared instantly in Chrome).
+
+---
+
+## 1. Summary verdict table
+
+| Site | Status signal | Fixed vocabulary | Hiatus/dropped ≠ completed | Where the marker lives | Adapter cost |
+|---|---|---|---|---|---|
+| asurascans.com | **Yes** — visible block + JSON in page state | `{ongoing, completed, hiatus, dropped}` (fixed, verified via browse filter) | **Yes** — four distinct values | Series page Status block; JSON in astro-island props; badge on listing cards; no chapter-list END row | **Low–Medium** |
+| demonicscans.org | **Yes** — info-block label/value pair | exactly `{Ongoing, Completed}` | **No** — the site cannot express hiatus/dropped | Series page info block only; search filter; no badges on listings, no END rows | **Medium** |
+| comix.to | **Yes** — `status` key in `initial-data` JSON (same blob the adapter parses) + header badge | `{releasing, finished, on_hiatus}` observed; 5 values in browse filter incl. `discontinued`, `not yet released` | **Yes** — `on_hiatus` distinct; `discontinued` a separate value | Series page header badge; `initial-data` JSON; browse-card chips | **Low** |
+| kagane.to | **Yes** — **two** explicit fields in the API JSON the adapter already fetches | `{Ongoing, Completed, Hiatus}` observed on both fields (99-series sample); no Dropped seen | **Yes for hiatus** (`upload_status`), **dropped unobservable** | Series API JSON (`publication_status`, `upload_status`); two badges on the page; no status filter in search | **Low** |
+| novelfull.com | **Yes** — info-panel status link into `/status/<value>` taxonomy | exactly `{Ongoing, Completed}` (`OnGoing`/`Ongoing` alias); `Hiatus`/`Dropped` 404 | **No** — not expressible | Series page info panel; `/status/<value>` taxonomy pages; no chapter-list END row | **Medium** |
+| lightnovelworld.net | **Yes** — visible class-named span **and** JSON-LD `creativeWorkStatus` | `{Ongoing, Completed, Hiatus}` (search filter radios; no dropped) | **Yes** — `PotentialActionStatus` (hiatus) ≠ `CompletedActionStatus` | Series page `.sertostat` span; `<script type="application/ld+json">` in the head; no badges on listings, no END rows | **Low–Medium** |
+
+Bottom line: **all six Sites publish an explicit, machine-readable completion
+marker on the Series page.** None requires inferring completion from absence of
+updates. Four of six (asura, comix, kagane, lnw) express hiatus as a distinct
+value; demonic and novelfull have a binary ongoing/completed vocabulary, which
+is *safe* for the Poll (a stalled-but-unlabelled series reads "ongoing") but
+cannot surface dropped/hiatus. Only kagane separates *work* status from
+*translation/upload* status — the exact distinction the false-positive
+question cares about. Details and quotes per Site in §2; the false-positive
+verdict in §3.
+
+---
+
+## 2. Per-Site findings
+
+### 2.1 asurascans.com — four-value status, JSON in page state
+
+Fetch: plain TLS, 200, no challenge (all pages below, 2026-08-19). Series URLs
+are `/comics/<slug>` where `<slug>` ends in a build hash the site appends
+(adapter strips it: `asuraBuildHash`, `sites.go:73`).
+
+**Series page — visible Status block** (ongoing sample,
+`https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af`):
+
+```html
+<div class="text-xs text-white/50 mb-1">Status</div>
+<div class="flex items-center gap-2">
+  <span class="w-2.5 h-2.5 rounded-full bg-[#A78BFA]"></span>
+  <span class="text-base font-bold text-[#A78BFA] capitalize"> ongoing </span>
+</div>
+```
+
+The completed sample (`https://asurascans.com/comics/solo-leveling`) renders the
+same block with the value `completed`. The label is a plain-text span — the
+value must be extracted from the text node, not a class.
+
+**Same value in machine-readable JSON.** The series page state (serialized into
+the astro-island props, HTML-escaped in the served document) carries
+`&quot;status&quot;:[0,&quot;completed&quot;]` (Solo Leveling) /
+`[0,&quot;ongoing&quot;]` (ongoing samples) — one key inside the same props
+object the adapter already scans for the chapter list.
+
+**Vocabulary: fixed, four values.** Homepage cards and browse pages embed one
+status per card; the browse filter enumerates the set:
+`https://asurascans.com/browse?status=completed`, `…?status=hiatus`,
+`…?status=dropped` each return 20-card lists whose blobs read
+`&quot;status&quot;:[0,&quot;completed&quot;]`, `[0,&quot;hiatus&quot;]`,
+`[0,&quot;dropped&quot;]`. Homepage: 72 `ongoing` + 4 `hiatus` card blobs.
+
+**Semantics of `dropped`** — scanlation-editorial, not work-ended
+`[INFERENCE, evidence below]`. `https://asurascans.com/comics/demon-king-b60d532c`
+is labelled `dropped` and its chapter links stop at `…/chapter/14`; the work
+continues elsewhere. So the Poll must treat **only `completed`** as
+potentially-finished; `dropped`/`hiatus` mean the site stopped releasing.
+
+**Other marker locations.** Listing cards carry a badge:
+`<span class="text-xs font-medium px-2 py-1 rounded capitalize bg-[#913FE2]/20 text-[#A78BFA]"> ongoing </span>`
+for ongoing, and `… bg-[#86EFAC]/20 text-[#86EFAC]"> completed </span>` for
+completed — the *color* does not discriminate (completed, hiatus and dropped
+all share the green `#86EFAC` badge; ongoing is purple `#913FE2`), so a badge
+reader must read the text. **No structured chapter-list END row**: Solo
+Leveling's final chapter is titled `Side Story 21 { THE END }` — the marker is
+prose inside a chapter title, unusable as a signal.
+
+**Adapter touchpoints.** `asuraLatestChapter` (`sites.go:158`) scopes chapter
+anchors by `asuraSlugRe` (`sites.go:67`); cover via `ogImageCover`
+(`sites.go:334`, wired `sites.go:450`). The `status` key lives in the same
+escaped props blob the chapter-list scan already walks — a new regex on the
+same body, or a parse of the visible Status block.
+
+### 2.2 demonicscans.org — binary vocabulary, nothing else expressible
+
+Fetch: plain TLS, 200 (2026-08-19). Series URLs `/manga/<title>`.
+
+**Series page — info block** (ongoing sample,
+`https://demonicscans.org/manga/Catastrophic-Necromancer`):
+
+```html
+<div class="flex flex-row">
+  <li style="width:150px;color:#b2b2b2;">Status</li>
+  <li>Ongoing</li>
+</div>
+```
+
+Completed sample (`https://demonicscans.org/manga/Solo-Leveling`): same markup,
+`<li>Completed</li>`. The row sits inside the same info-block `<div>` as
+Author / Rating / Last Update — a sibling `<li>` pair per field.
+
+**Vocabulary: exactly `{Ongoing, Completed}`.** The advanced-search filter
+(`https://demonicscans.org/advanced.php`) proves the set — the status `<select>`
+has two options:
+
+```html
+<select class="form-control" name="status">
+  <option value="all">All</option>
+  <option value="ongoing">Ongoing</option>
+  <option value="completed">Completed</option>
+</select>
+```
+
+There is no hiatus/dropped value anywhere on the site (`[INFERENCE]`: no such
+string in the filter or any series page sampled). A scanlation that stalls
+silently keeps reading `Ongoing`; the site cannot label a series
+dropped/on-hiatus.
+
+**Other marker locations.** None: homepage and `/lastupdates.php` show no
+status text or badges on their cards (2026-08-19). No chapter-list END rows
+(the last-chapter anchor is a plain `chaptered.php` link).
+
+**Adapter touchpoints.** `demonicLatestChapter` (`sites.go:177`) scans
+`chaptered.php?manga=<id>&chapter=<n>` anchors via `demonicChapterRe`
+(`sites.go:77`), unscoped; cover `ogImageCover` (`sites.go:334`, wired
+`sites.go:457`). A status reader adds a selector over the info-block `<li>`
+pair; the block is sloppy HTML (bare `<li>` inside `<div>`s) but stable across
+the two sampled themes/versions.
+
+### 2.3 comix.to — `status` key in the exact JSON the adapter already parses
+
+Fetch: Cloudflare JS challenge; cleared through a real Chrome (~12 s), then the
+adapter's own payload shape was fetched in-tab (2026-08-19). The payload is
+server-rendered HTML carrying `<script id="initial-data">…</script>` JSON
+(`comixInitialDataRe`, `sites.go:248`); the Series detail entry
+`["manga","detail","<slug>"]` contains `"status"` and the chapter list.
+
+Observed values (three sampled series, all 2026-08-19):
+
+| Series | URL | `status` | Header badge |
+|---|---|---|---|
+| Dungeons & Crayons (ongoing) | `https://comix.to/title/n8we-dungeons-and-crayons` | `"releasing"` | `RELEASING` |
+| Countach (completed) | `https://comix.to/title/q77m-countach` | `"finished"` | `FINISHED` |
+| How to Survive as the Daughter… (hiatus) | `https://comix.to/title/d0n08-how-to-survive-as-the-daughter-of-the-emperor-who-killed-me` | `"on_hiatus"` | `ON_HIATUS` |
+
+The visible header badge (hiatus sample):
+
+```html
+<span class="mpage__badge mpage__badge--status mpage__badge--status-on_hiatus">ON_HIATUS</span>
+```
+
+The class encodes the value (`--status-on_hiatus`), the text is the raw
+uppercase snake_case label.
+
+**Vocabulary: five fixed values** from the browse advanced filter ("RELEASE
+STATUS"): `Releasing`, `Finished`, `On hiatus`, `Discontinued`,
+`Not yet released`. The filter URL encodes snake_case:
+`https://comix.to/browse?statuses=on_hiatus` (plural `statuses`; a plain
+`?status=finished` probe did *not* filter the list — titles of other statuses
+remained). `discontinued` and `not yet released` were not observed on series
+pages but are site-selectable filter values `[INFERENCE: they map to
+`discontinued`/`not_yet_released` keys]`.
+
+**Hiatus/dropped ≠ completed: yes** — `on_hiatus` is a distinct JSON value and
+`discontinued` is a separate filter option.
+
+**Other marker locations.** Browse cards render status chips (FINISHED /
+RELEASING seen); the header badge above; the browse filter itself. No
+chapter-list END row (chapter entries are per-chapter rows).
+
+**Adapter touchpoints.** `comixLatestChapter` (`sites.go:185`) already parses
+this exact `["manga","detail","<slug>"]` entry for the chapter URL — `status`
+is a sibling key of the entry, so the read cost is near-zero. Cover
+`comixCoverEntry` (`sites.go:344`). Browser-backed read `comixRead`
+(`browser.go:154`), `Done` excludes the interstitial (`sites.go:471`).
+
+### 2.4 kagane.to — two explicit status fields in the API JSON the adapter already fetches
+
+Fetch: Cloudflare JS challenge; cleared through a real Chrome (12–32 s, flaky;
+the challenge occasionally re-fires). The adapter's payload shape was fetched
+in-tab: the Series page itself requests
+`https://kagane.to/api/v2/series/<uuid>` (`kaganeSeriesRe`, `browser.go:25`) and
+the poller consumes that JSON (`kaganeRead`, `browser.go:128`). Series URLs are
+`/series/<uuid>`, e.g. `https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b`
+(Infinite Decryption: The Strongest Level 0).
+
+The API JSON carries **two explicit status fields** plus year bounds:
+
+```json
+"publication_status": "Ongoing",
+"upload_status": "Ongoing",
+"start_year": 2025,
+"end_year": null,
+```
+
+- `publication_status` — the work's status (Ongoing / Completed / Hiatus).
+- `upload_status` — the site's own release status.
+
+**They diverge — proven.** "'Cause Calypso Can"
+(`https://kagane.to/series/019c2034-f431-73ac-9350-11bc498f8c40`):
+`"publication_status": "Ongoing"` but `"upload_status": "Hiatus"` — the work is
+ongoing, the site's release is paused. This is precisely the
+translation-stopped-vs-work-ended distinction the dashboard's false-positive
+question needs, and kagane is the **only** Site that separates the two.
+
+**Vocabulary.** Across a 99-series sample (search page 1, in-tab re-POST of the
+page's own `POST /api/v2/search/series?page=0&size=99` request with body
+`{"content_rating":["Safe","Suggestive"]}` — the page's own request returned
+38 results, a direct re-POST 403'd even with Referer), both fields take only
+`{Ongoing, Completed, Hiatus}`. **No Dropped value observed** on either field
+in the sample; a dropped release presumably stays `Hiatus` or `Ongoing`
+`[INFERENCE]`. The search page exposes no status filter (filters: Format,
+Genres, Tags, People, Sources, Content Rating, Language, Presets).
+
+**Page renders two badges** (value in text only, class carries no value), the
+hiatus one yellow:
+
+```html
+<span class="inline-flex items-center rounded-full border px-2.5 py-0.5 … bg-yellow-500/10 text-yellow-600 dark:text-yellow-400">Hiatus</span>
+```
+
+**Adapter touchpoints.** `kaganeLatestChapter` (`sites.go:197`) scans
+`"chapter_no":"<n>"` in the API body (`kaganeChapterRe`, `sites.go:98`);
+cover `kaganeCoverEntry` (`sites.go:349`). `publication_status`/`upload_status`
+are top-level keys of the same body — read cost near-zero.
+
+### 2.5 novelfull.com — binary status, taxonomy-linked
+
+Fetch: time-varying challenge. Plain TLS today returned HTTP 403 with a
+"Just a moment…" interstitial (~5.7 KB); the same URL cleared instantly through
+a real Chrome, and the cleared DOM is the adapter's payload shape (`novelfullRead`,
+`browser.go:141`; `Done` excludes the interstitial, `sites.go:501`).
+
+**Series page — info panel** (completed sample,
+`https://novelfull.com/reverend-insanity.html`):
+
+```html
+<div><h3>Status:</h3><a href="/status/Completed">Completed</a></div>
+```
+
+Ongoing sample (`https://novelfull.com/a-cunning-pervert-in-the-cultivation-world.html`):
+`<a href="/status/Ongoing">Ongoing</a>` in the same block.
+
+**Vocabulary: exactly `{Ongoing, Completed}`**, proven by the taxonomy pages:
+`https://novelfull.com/status/Ongoing` and `https://novelfull.com/status/OnGoing`
+both return 200 with identical listings (the two spellings alias; the page title
+normalizes to "Status: Ongoing"), while `https://novelfull.com/status/Hiatus` and
+`https://novelfull.com/status/Dropped` return 404. No hiatus/dropped state
+exists on the site.
+
+**Hiatus/dropped ≠ completed: no** — not expressible; a stalled novel keeps
+reading `Ongoing` indefinitely. Safe direction for the Poll (nothing falsely
+"completed"), but the site cannot surface stalled releases.
+
+**Other marker locations.** The `/status/<value>` taxonomy listing pages; the
+series-page info panel only. Homepage latest-updates cards carry no status
+(2026-08-19). **No chapter-list END row** (chapter rows are plain anchors;
+checked `reverend-insanity.html`'s cleared DOM).
+
+**Adapter touchpoints.** `novelfullLatestChapter` (`sites.go:204`) scopes
+chapter anchors by `novelfullSlugRe` (`sites.go:103`); cover `novelfullCoverEntry`
+(`sites.go:339`). The status `<a href="/status/…">` in the info panel is a new,
+independent selector on the same body.
+
+### 2.6 lightnovelworld.net — class-named span + schema.org JSON-LD
+
+Fetch: plain TLS, 200 (2026-08-19). Series URLs `/novel/<slug>/`.
+
+**Visible marker — `.sertostat` span, class = value** (completed sample,
+`https://lightnovelworld.net/novel/a-will-eternal/`):
+
+```html
+<div class="sertostat">
+  <span class="Completed">Completed</span>
+</div>
+```
+
+Ongoing sample (`https://lightnovelworld.net/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/`):
+`<span class="Ongoing">Ongoing</span>`; hiatus sample
+(`https://lightnovelworld.net/novel/rebuild-world-wn/`): `<span class="Hiatus">Hiatus</span>`.
+
+**Machine-readable marker — JSON-LD in the head.** The same pages carry a
+`<script type="application/ld+json">` block (line 469 of the a-will-eternal
+fetch) with a schema.org `Book` entry whose `creativeWorkStatus` is the
+**completion signal, mapped to the ActionStatus vocabulary**:
+
+| Sample | `creativeWorkStatus` |
+|---|---|
+| A Will Eternal (completed) | `"https://schema.org/CompletedActionStatus"` |
+| All Jobs and Classes… (ongoing) | `"https://schema.org/ActiveActionStatus"` |
+| Rebuild World (hiatus) | `"https://schema.org/PotentialActionStatus"` |
+
+**Vocabulary: `{Ongoing, Completed, Hiatus}`.** The homepage search filter
+enumerates it: radio inputs `id="anime_status-ongoing"`,
+`id="anime_status-completed"`, `id="anime_status-hiatus"` — no dropped option.
+
+**Hiatus ≠ completed: yes** — `PotentialActionStatus` (hiatus) is a distinct
+JSON-LD value from `CompletedActionStatus`; the Poll can treat only the latter
+as potentially-finished.
+
+**Other marker locations.** The `.sertostat` span only. Homepage/listing cards
+carry **no** status badges (2026-08-19 — earlier word hits on the homepage were
+the filter radios and a series title containing "Dropped"). **No chapter-list
+END row**: the chapter list is `epl-num`/`epl-title` rows with no end marker on
+the completed A Will Eternal.
+
+**Adapter touchpoints.** `lnwLatestChapter` (`sites.go:222`) scopes chapter
+anchors by `lnwChapterRe` (`sites.go:115`) and truncates at the
+`lnwCommentMarker` ("wpd-threads", `sites.go:124`); cover `ogImageCover`
+(`sites.go:334`, wired `sites.go:508`). The JSON-LD block is a separate script
+in the head — one regex or JSON parse away; the `.sertostat` span is in the
+article body.
+
+---
+
+## 3. The false-positive question — cross-site verdict
+
+The dashboard wants to *suggest* "potentially finished" without false
+positives (issue #129 context). Per Site, what would a Poll have to believe to
+mark a Series finished, and can that belief be wrong?
+
+| Site | Poll reads | False-positive exposure |
+|---|---|---|
+| asurascans.com | `status == completed` only | `dropped`/`hiatus` are distinct values — reading them as anything but completed is a false *negative*, not a false positive. `completed` is the site's own editorial call on a finished work (Solo Leveling). Safe. |
+| demonicscans.org | `Status == Completed` | Binary vocabulary: `Completed` is the site's label and hiatus/dropped are unexpressible, so a label never lies *about completion* — but "Completed" is site-editorial and could mark a translation ended early. Poll a labelled `Completed` series as a *suggestion*, not a fact. |
+| comix.to | `status == finished` | `on_hiatus` and `discontinued` are distinct values; only `finished` triggers. Same editorial caveat. Safe. |
+| kagane.to | `publication_status == Completed` **or** `upload_status == Completed` | **The only Site where the two questions are answered separately.** Work-finished should read `publication_status`; `upload_status == Completed` is the site's release-complete flag and can coexist with `publication_status == Ongoing` (inverse of the Calypso Can case `[INFERENCE]`). The Poll should decide which field drives "potentially finished". |
+| novelfull.com | `Status == Completed` | Binary vocabulary; hiatus/dropped unexpressible (taxonomy 404s). A labelled `Completed` (e.g. Reverend Insanity, The Legendary Mechanic) is the site's call. No hiatus false-positive path exists. |
+| lightnovelworld.net | `creativeWorkStatus == CompletedActionStatus` | `PotentialActionStatus` (hiatus) is distinct; only `CompletedActionStatus` triggers. Safest single-value reading of the six. |
+
+Cross-site verdict: **no Site's status field is owner-written
+`series.finished_at` data** (that stays the owner's prerogative, #114/#124
+context) — every value here is the site's editorial status. But **none of the
+six forces the Poll to guess**: all publish explicit markers, and only asura
+and kagane introduce *translation-state* vocabulary (`dropped` / `upload_status`)
+that a naive "status == completed" reader could misinterpret — in both cases
+the danger is a false *negative* (treating a dropped/hiatus series as still
+ongoing), not a false positive, **provided the Poll reads only the
+completed/finished value** and treats anything else as not-completed.
+
+---
+
+## 4. Adapter touchpoints (for #130's containment decision)
+
+| Site | Marker source | Existing parse this piggybacks on | New parse needed |
+|---|---|---|---|
+| asura | escaped `&quot;status&quot;:[0,&quot;…&quot;]` in astro-island props, or visible Status block | `asuraLatestChapter` walks the same props blob (`sites.go:158`) | one regex on the same body |
+| demonic | info-block `<li>` pair | — (`demonicLatestChapter` scans chapter anchors only, `sites.go:177`) | selector over the info block |
+| comix | `["manga","detail",…]."status"` in `initial-data` | `comixLatestChapter` parses that exact entry (`sites.go:185`, `sites.go:248`) | sibling key read |
+| kagane | `publication_status` / `upload_status` top-level keys | `kaganeLatestChapter` reads the same API body (`sites.go:197`) | sibling key read (×2) |
+| novelfull | info-panel `<a href="/status/…">` | — (`novelfullLatestChapter` scopes chapter anchors, `sites.go:204`) | selector over the info panel |
+| lightnovelworld | head JSON-LD `creativeWorkStatus`, or `.sertostat` span | — (`lnwLatestChapter` truncates at `wpd-threads`, `sites.go:222`) | one JSON-LD regex / parse |
+
+Challenge status for a future Poll: asura, demonic, lnw are plain-fetch; comix
+and kagane require the browser sidecar on every poll (Cloudflare challenge
+present today, 2026-08-19); novelfull requires the sidecar *whenever its
+challenge is up* (it was up today, 2026-08-19).