spec-134
120 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 sulthan
|
264839e798 | Merge ticket/147-lane-pause | ||
|
sulthan
|
cbe0a28921 |
feat(web,latest): pause and resume one Site's Lane with a mandatory expiry (#147)
Two owner-gated POST routes write the durable poll_lanes pause stamp the
poller's top-of-pass gate already reads: /admin/lanes/{site}/pause validates
the duration against the fixed 1h/6h/24h allow-list and the Site against the
registry, and /admin/lanes/{site}/resume zeroes the stamp. Both cap the form
body like the API path, answer with the freshly rendered Lanes block, and
never command the poller — the pause is a fact about the Site, so it
survives a restart. The Lanes page's c-ctrl slot now carries the pausebar:
Resume while paused, the duration select plus Pause while running, with the
paused phrase read from the live poll_lanes stamp so a press renders as
paused with no pass having run. Tests cover the round trips, rejections,
body caps, the pause-before-refusal ordering, fresh-poller survival, resume
restoring the full queue, and acquisition being unaffected.
|
||
|
sulthan
|
e45445cb20 | Merge ticket/146-forced-poll | ||
|
sulthan
|
4a95657425 | Merge ticket/143-overview | ||
|
sulthan
|
5b7adf5f2c |
refactor(web): share poll-state derivation; keep zebra parity across a row swap (#146)
Review round: pollState() unifies the CanPoll/Pending/Requested derivation used by the list row and the detail page, and the row anchor carries its band parity with the press (hx-vals) so the swapped answer keeps the alternation. |
||
|
sulthan
|
00fa237151 | fix(web): verdict space per design mock, single door() fig builder (#143) | ||
|
sulthan
|
fe2cd12049 |
feat(web,store,latest): forced poll — Check now stamps a Series for the next Lane pass (#146)
The control writes series.force_poll_at (column landed in migration 0014) and never commands the poller: pending is derived as force_poll_at > latest_checked_at and self-clears because the check stamp is written before the fetch. The due query's forced flag overrides the rest cutoff, the Sighting-deferral and finished-only clauses, jumps the queue, and wakes a sleeping browser Lane; it never overrides an empty series_url, the Bookmarks join, the refusal backoff, the sidecar-down skip or the Lane gap. ADRs: 0013-commands-through-the-database. |
||
|
sulthan
|
c432a3be30 | feat(web): overview page with verdict line and stats doors (#143) | ||
|
sulthan
|
6f9109c885 | Merge ticket/145-lanes-from-db | ||
|
sulthan
|
f5b52e48c8 | Merge ticket/144-series-detail | ||
|
sulthan
|
bf08d6e65c | Merge ticket/142-series-list | ||
|
sulthan
|
d72c48295d |
fix(web): statusline tokens, unreachable dot, reachability test spans (#145)
Second review round (spec + standards): - Polling-off and unreachable render as statusline tokens; the unreachable span drops mark-strong so the patina dot never sits next to red text. - Reachability tests assert on the rendered span, which 'reachable' and 'unreachable' substrings never could. - A pass-log query failure now reads as reachable (no evidence rule) instead of condemning the browser, and admin.go loses its dead time import. - startLatestPoller's doc no longer claims the admin page reads the poller; AGENTS.md carries the RefuseBackoff rename. |
||
|
sulthan
|
fc8a40cfc6 |
fix(latest,web): finish the atomic deletion and honest empty states (#145)
Review fixes on top of
|
||
|
sulthan
|
cb2b104e63 |
feat(web): read admin lanes from the durable pass log, not a poller snapshot (#145)
The Lanes page now projects store.LatestLanePasses and the owner-window outcomes (store.LanePassOutcomes) into per-Site rows instead of reading an in-memory Poller snapshot, so a deploy answers the instant the store is up. Browser configuration is a config fact and reachability is derived from recent browser-Site passes inside latest.RefuseBackoff. This atomically deletes the in-memory path in the same commit that makes the page read the DB: latest/status.go (LaneState, Status, LaneStatus, recordLaneState) and the web.LaneReporter seam plus fakeLanes are gone, and latest.refuseBackoff is renamed latest.RefuseBackoff at every callsite. ownerWindow (#142) is referenced, never declared (contract C2) |
||
|
sulthan
|
134c9307b1 | #142: fix stale filter count cutoff, tighten comments (review fixes) | ||
|
sulthan
|
09a094ca67 | fix(web): detail row lookup walks all pages of the Site read (#144) | ||
|
sulthan
|
e8a3c5f826 | #142: series list page with eight hygiene filters, site/library narrowing, paging | ||
|
sulthan
|
14990bde21 | feat(web): per-Series detail page keyed by site:series_id (#144) | ||
|
sulthan
|
503fb49d0a |
feat(latest): record one poll pass per exit with skip reason and outcome counts
Every way a Lane pass can end now writes exactly one durable row: a skip value naming the exit (paused, refusing, sidecar-down, no-fetcher, due-query, asleep, eligible-count, nothing-eligible, or empty for the loop), five outcome counts from the classification the Series read already makes, and carry-forward of the previous pass's figures exactly when the pass's own gap is zero. Refusal is durable through the poll_lanes row, so a restart does not re-probe a Site inside its backoff. Retention is 14 days. The mid-loop browser-unreachable return writes an empty skip by design: a tenth value is not invented here. (#141) |
||
|
sulthan
|
e0b9063d9e |
feat(store): cross-series admin read model with privacy in the projection (#140)
SeriesPage returns one 50-row page of Series matching one of eight named hygiene filters (all, no_series_url, never_read_a_chapter, no_readers, never_checked, stale, no_cover, reader_report), with a window-count total; SeriesShapes returns the per-Site aggregate, one grouped pass. The compound filter value object takes Site, Kind, Name, a caller-supplied staleness cutoff, and a 1-based page. The privacy boundary lives in the projection: adminSeriesColumns never selects latest_raised_by, and AdminSeries has no field for it — a SQL-computed boolean is all that crosses. LEFT JOIN surfaces orphans (reader_count 0); (site, series_id) tie-breaks the zero-stamp boundary so pages stay stable. Also lands 0014: the series(latest_checked_at) index and series.force_poll_at, both expand-step schema for later tickets in the series. |
||
|
sulthan
|
fd1131d11d | feat(store): persist poll lane state | ||
|
sulthan
|
030ffdc26e | feat: split admin surface into bookmarkable pages (#138) | ||
|
sulthan
|
0a245a0dde |
docs: name the Stall and the Correction in CONTEXT.md (#133)
Two domain terms the admin surfaces need and CONTEXT.md did not carry: - **Stall** — a Poll Lane that owed Polls, made none, and has nothing to say for it; distinct from a refusing Site and a Paused Lane. - **Correction** — an owner-set Latest Chapter for a Series no Poll can read; lower authority than a Sighting. Docs only. Branch cut fresh off `origin/main`, so it carries nothing from the research branch. Reviewed-on: #133 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
249f11e1fe |
docs: name the admin dashboard's domain terms in CONTEXT.md (#128)
Glossary-only change; no code touched. Charting the admin dashboard map (#114) settled four terms the glossary did not carry: - **Orphan Series** (#125) — a Series no Reader bookmarks; a state of the Series, never a Lifecycle bucket. - **Lane Pass** (#117) — one sweep of a Poll Lane, including a pass that declined to work and why. - **Forced Poll** (#119, #120) — a Poll the owner asks for by marking the Series, which jumps the waiting rules but never the Site's refusal, and which may replace a Cover. - **Paused Lane** (#119) — a bounded, restart-surviving stop on one Site, distinct from the deploy-time kill switch. **Acquisition** is amended in the same pass: establishing a Cover is no longer unique to it, since a Forced Poll can replace one. Reviewed-on: #128 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
f5d3fe58ec |
Add a gitea skill; point AGENTS.md at it (#126)
Forge usage lived in three places (`AGENTS.md`, `docs/agents/issue-tracker.md`, and habit). This moves the how-to-run-`tea` half into a model-invoked skill that fires on any issue/PR task, and reduces `AGENTS.md` to identity plus pointers. - **new** `.claude/skills/gitea/SKILL.md` — command table plus the traps `tea <cmd> --help` will not tell you. - `AGENTS.md` — Forge section is now one line: Gitea not GitHub, `gh` and the `issue://`/`pr://` URIs fail, then pointers to the skill, `docs/agents/issue-tracker.md`, and `docs/agents/triage-labels.md`. - `.claude/skills/implement-tickets/SKILL.md` — pointer split: tracker conventions to the doc, `tea` usage to the skill. Both `docs/agents/` files are untouched; the skill cites them instead of restating them. Facts in the skill are measured against `tea` 0.14.2 on 2026-08-17, not remembered: - `gh` is not installed, so `read issue://71` errors — there is no fallback to add. - **A bare read is a truncated read.** Without `--comments`, `tea issue <n>` drops every comment silently, with no prompt under a non-TTY: issue #123 prints 40 lines bare, 132 with the flag. The skill makes `--comments` mandatory for any read meant to understand a ticket, with `tea issue list --fields index,comments` as the checkable count. - Issues and PRs share one index space; output is rendered boxes so parsing needs `-o json`; `close` takes no `--comment`; labels never auto-create; multi-line bodies need a heredoc; `tea` exposes neither sub-issues nor dependencies. Unmeasured and marked as such: whether `--comments` covers a PR's review-comment stream — no PR in this repo has comments, so `tea pr review-comments <n>` is named without a claim about overlap. Reviewed-on: #126 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
766aa8f00d |
docs: make every AGENTS.md cite code, not docs or issues (#113)
Every `AGENTS.md` now cites code and nothing else.
## Why
Two rot mechanisms, same symptom — an agent confidently follows a stale statement:
1. **Non-code citations.** A spec, ADR, plan file, or issue records what was true when it was written. Nothing updates it when the decision reverses.
2. **Prose restating mechanism.** The code changes, the paragraph doesn't, and the next reader trusts the paragraph.
Code is the only source true at read time.
## What changed
**All three files:** removed every ADR ref, spec/plan pointer (`docs/superpowers/specs/*`, `plans/*`, `docs/research/*`), `DEPLOY.md`/`REDEPLOY.md`, `docs/agents/*`, and issue number. Facts those links carried are restated inline — the `tea` command set and the five triage label strings now live in the root Forge section. `### Domain docs` is deleted: it pointed only at `CONTEXT.md` and `docs/adr/`, neither of which exists.
**`backend/` and `userscript/`:** rewritten around derivability.
| Class | In code? | Treatment |
|---|---|---|
| Structure — packages, routes, env vars, columns | yes | name the symbol, nothing else |
| Mechanism — what a function does | yes | symbol + one line |
| Rationale — why, what a "simplify" breaks | **no** | written out |
| Measurement — observation against a service we don't control | **no** | written out, dated |
`backend/AGENTS.md` 20578 → 15512 bytes, `userscript/AGENTS.md` 7129 → 5912. Root grows 16905 → 19292: the cost of inlining the `docs/agents/*` facts plus the new rule.
**Rule** recorded in root as `## Writing an AGENTS.md`. Sole non-code exception is a sibling `AGENTS.md`. Closing clause: every symbol named must exist, since a dead pointer is a bug rather than a stale sentence.
**Harness-agnostic:** dropped the `Guidance for OpenCode (and Claude Code)` openers for plain scope lines.
## Verification
Applied the new rule to itself — extracted all 118 backticked identifiers across the three files and checked each against every `.go`, `.js`, `.sql`, `.html` and `.css` source. Zero repo symbols missing; the 8 non-matches are external (`GM_setValue`, `navigator.webdriver`, `HeadlessChrome`, `curl`, …).
That check caught a claim that was **already lying** on `main`: the cover section said `CoverFetcher` was gone, but `NewCoverFetcher`, `TLSCoverFetcher` and `BrowserCoverFetcher` are all live in `internal/latest`. Now names only the genuinely dead `/img/kagane/{id}` route. Exactly the failure the rule exists to prevent.
No code touched — documentation only, nothing to test.
Reviewed-on: #113
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
550b258c59 |
fix: give covers their own 10 MiB byte cap (#71) (#112)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.3 |
||
|
sulthan
|
3ac865cd08 |
chore: remove graphify (#111)
Removes the graphify integration. It was measured against this repo rather than assumed. ## Why `graphify query` returns a keyword-seeded BFS neighbourhood, not a location. Asked where CORS origin reflection is implemented, it returned 73 nodes — mostly `api_test.go` helpers, plus a `Reflection and Type Assertions` section from `.agents/skills/golang-performance/references/cpu.md` matched on the word "reflection" — and never named `httpmw/middleware.go:135` or `main.go:121`. `grep` returned both in 39ms. Same shape asking how the poller skips kagane: 145 nodes, top hits `poller_test.go` helpers and two nodes named `T`. `graphify explain "BrowserFetcher"` is sound (`browser.go L52`, 9 `EXTRACTED` edges), but that is what `lsp references` already answers, against live files instead of a snapshot. Staleness was never the problem — `graph.json` rebuilt 5s after `f568fb5`, so the git hooks worked. Retrieval quality was. ## What it cost - Two `PreToolUse` hooks injecting a "MANDATORY: run graphify query first" paragraph into context on **every** grep/find and every source-file read. - 685k input tokens across 5 build runs (`cost.json`). - 3.4MB of `graph.json` + `graph.html` tracked, across 11 commits of map-refresh churn. `AGENTS.md` is the stronger orientation artifact for a repo this size: it carries the CDP constraints, the UTC-clock finding, the per-site adapter list, and the security invariants — none of which an AST graph derives. Graphify earns its keep on repos too large to grep coherently and without curated docs; not this one. ## Changes - Delete the committed map (`graphify-out/`, -58k lines). - Drop the `## graphify` rules block from `AGENTS.md` (`CLAUDE.md` is a symlink, so both). - Drop the five `graphify-out/*` entries from `.gitignore`. - Empty the two `PreToolUse` hooks in `.claude/settings.json`. - Remove the stale `graphify query` instruction from `.claude/skills/implement-tickets/SKILL.md` — it pointed dispatched ticket-implementer agents at a binary that no longer exists. Uninstalled outside the tree (not in this diff): the `graphifyy` CLI, `~/.claude/skills/graphify/`, the global `~/.claude/CLAUDE.md` block, the `Bash(graphify query *)` permission in the git-ignored `.claude/settings.local.json`, and the `post-commit` / `post-checkout` git hooks. ## Verification `grep -ri graphify` over the worktree is clean; remaining hits are inside `.git/` (commit messages, two stale branch configs). No code touched — backend and userscript are untouched, so `go test ./...` is unaffected. Reviewed-on: #111 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
f568fb5e8c |
fix: an asleep browser Lane is not a stalled one on the admin page (#110)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.2 |
||
|
sulthan
|
20fff588cc |
fix: don't read Cloudflare's injected jsd script as a refusal (#109)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.1 |
||
|
sulthan
|
ba679223b2 |
Sightings: a Reader report defers a Poll of a solitary Series (#103) (#108)
Closes #103. A userscript PUT already carries the Latest Chapter the Reader's own browser read off the Series page. It may now stand in for a Poll, under one restriction and one ceiling: - **Solitary Series only** — a Series two Readers share is Polled on schedule however recently it was sighted, so one Reader's mistake can never reach another's list. - **One rest of standing**, and a **six-rest ceiling** (`sightingCeilingRests`, counted in the Site's own Rest): however many Sightings arrive, an unpolled Series is Polled. Both live in the due query's HAVING clause (`Store.DueForLatestCheck`) — the same place the schedule has always been decided, so no timer and no second code path can disagree with it. No new query per scheduler round. Judgement costs no extra request. `Poller.checkOne` already compares what the Site publishes against what is stored: a lower number contradicts the Sighting (Reader and both numbers logged), the same number confirms it, a higher number is the Site publishing and clears the attribution instead. Three contradictions stop that Reader deferring — their reports still write the Latest Chapter — and twenty consecutive confirmations forgive them, as does the owner's clear-marks control from #102. One client change was required: both userscripts skipped the PUT when the number had not moved, so the case the whole mechanism exists for — visiting a Series with nothing new — never reached the backend. `reportLatestChapter` sends it, skipping only the local write and the re-render. A numberless PUT (favourite toggle, progress from a chapter page) is no Sighting and defers nothing. Schema: migration `0011_series_sightings.sql` adds `series.latest_sighted_at` and `series.latest_raised_by`. Trust model, thresholds, and rejected alternatives with their citations: `docs/adr/0011-sighting-deferral-trust-model.md`. Reviewed on both axes (spec against #103, standards against the repo's rules); the blocker — attribution surviving a Poll that overtook the report — is fixed and has a test that fails without the fix. Verification: `go test ./...` green (needs Docker), `node --test userscript/test/*.test.js` 66 pass. Reviewed-on: #108 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.0 |
||
|
sulthan
|
1e6f1e985d |
Owner-only admin page: Reader roster plus Poll Lane status (#102) (#107)
Closes #102.
The only operational surface was /healthz and a fold-out roster inside the owner's own reading page. This adds /admin: an owner-only page carrying the Reader roster and one row per Poll Lane.
- **Poller seam.** `latest.Poller` records each Lane's last pass (`Site`, `Due`, `Checked`, `LastRun`, `Gap`, `Clamped`, `Browser`) and answers `LaneStatus()`; the page reads that snapshot, never a table. A pass that returns before computing its figures (refusal backoff, sidecar down) carries the previous pass's figures forward rather than recording zeroes, and a Lane that has never reached a pace renders no gap at all. Refusal and sidecar reachability are derived at snapshot time.
- **Owner gate at registration.** Every route reaching past the acting Reader lives in `adminRoutes()` and is wrapped in `requireOwner` when it is registered, so a missing gate is visible in the route list rather than hidden in a handler. `web.AdminPatterns()` is what the gate test walks, so a new route cannot be added without being tested. A non-owner gets 404, never 403.
- **Nil poller is a first-class state.** `main.newRouter` takes the reporter as an interface and converts a nil `*Poller` to a nil interface; no poller and no completed pass both render "No data yet" with the reason spelled out, rather than confident zeroes.
- **Roster moved** off the reading page onto /admin, with the Sighting counters and a confirm-gated `Clear marks` control. #103 fills those counters, so on delivery they read zero for everyone - deliberate ordering.
- **One accent, `--patina`** (verdigris, both colour branches): the far side of the wheel from ember's crimson and clear of the archive blue. Ember still means new chapter only; revocation still wears --danger.
Verification: `go vet ./...` and `go test ./...` green (Docker-backed); admin page screenshotted at 1100px and 390px in both colour schemes. Reviewed on both axes (spec, standards); findings on the accent hue, zero-figure honesty and three tests that could not fail are fixed in
|
||
|
sulthan
|
3303a55b20 |
feat: one Poll Lane per Site, replacing the shared pace (#100) (#106)
Closes #100. Each Site runs its own Poll Lane: an independent goroutine with its own rest and pace from the registry (`backend/internal/latest/sites.go`), replacing the shared cooldown/interval/stagger/batch configuration. Rest (1h, all six Sites including the browser trio) is enforced by the due query's WHERE clause; the Lane sleeps its effective gap between fetches — the registry 10s, or rest/eligible when a Site holds enough Series, floored at 1s with a Site-naming warning when the floor engages. Lane-local failure handling: - Two challenge-held results stop that Site's Lane for 15m; the probes keep their stamp, untried Series stay due. - A lost browser sets a shared Poller flag: the other browser Lanes skip their passes for the same 15m (no stamp-per-pass-per-Lane on a dead tab), then decay and probe again. - Browser wake gate preserved (5 due, or one waiting 15m, ADR-0005); one tab shared by the three browser Sites; "browser lane behind by X" logged every pass. - Cover work (healing a stored source URL and filling a blank from the series page) runs in the background so a slow CDN cannot consume a Lane's gap. Removed: `LATEST_CHAPTER_POLL_{COOLDOWN,BROWSER_COOLDOWN,INTERVAL,BATCH,STAGGER}` and the 6h browser rest. Only `LATEST_CHAPTER_POLL_ENABLED` remains; DEPLOY.md documents the exact `.env` edit. ADR-0010 records the decisions. Reviewed-on: #106 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
ddbd57070d |
Poll comix.to through the browser sidecar (#98) (#105)
Closes #98.
comix.to began answering plain-TLS fetches with a Cloudflare JavaScript
challenge on 2026-08-12, so every poll got a 403 interstitial. Its cover host
`static.comix.to` is gated the same way. comix therefore joins kagane and
novelfull as a browser-backed Site.
## What changed
- **Registry** (`internal/latest/sites.go`): comix gains a `Browser` entry —
`comixRead`, `Done: body != "" && !isInterstitial(body)`, `Fallback: false`.
Skip-when-no-browser falls out of the existing routing; no site-string compare
was added anywhere.
- **Read shape** (`internal/latest/browser.go`): an in-tab `fetch()` of the
Series URL, not a DOM render. comix is an SPA — rendering it costs ~65
requests for the same server-rendered HTML one fetch returns (24.5 KB,
~480 ms measured). `comixSeriesPageURL` pins scheme + host + `/title/<slug>`
and rebuilds the address, so a client-supplied `series_url` cannot aim the
browser anywhere else.
- **Cover bytes**: `comixImageURLRe` pins `https://static.comix.to/<path>.<ext>`;
`BrowserFetcher.Image` now gates on `browserOnlyCoverURL` rather than a
kagane-only regex, so both Sites' image URLs route through the one path.
Bytes come from direct navigation, not a page-context fetch — comix's Series
page sets `cross-origin-embedder-policy: require-corp`, which fails one.
- **Parsers and stored Series identity: untouched.** The in-tab body is the same
server-rendered HTML the existing fixtures were cut from.
## Verification
- `go test ./...` green (needs Docker).
- New seam tests: comix routes to the browser when one is configured, and is
not fetched at all when none is (`TestComixUsesBrowserFetcher`,
`TestComixSkippedWhenNoBrowserFetcher`); URL-pin and cover-gate table tests.
- Live proof against the real browser unit, `TestSmokeComix` (env-gated):
page 24793 bytes in one in-tab fetch, chapter 53, cover accepted by the pin,
26862 bytes of `image/jpg` retrieved.
- Two-axis review run; findings were stale comments on `BrowserFetcher`, `Get`
and the `Fallback` field, fixed in
|
||
|
sulthan
|
3f53c79cf4 |
docs: add Poll Lane and Sighting to the shared vocabulary (#104)
Two new glossary terms in `CONTEXT.md`, settled in a design session, plus the graphify refresh. - **Poll Lane** — one Site's own stream of Polls, carrying the pace at which that Site is willing to be asked. No Lane can slow, block or borrow from another's; a Reader never has one. - **Sighting** — what a Reader's browser happened to see of a Series's Latest Chapter. Reports the same fact as a Poll, carries none of its authority. - **Latest Chapter** amended: it no longer claims to be discovered without the reader present, since a Sighting establishes it between Polls. No code. The work these terms describe is specified in #98 (comix), #101 (Poll Lanes), #102 (admin page) and #103 (Sightings). Reviewed-on: #104 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
17ee0bd3f8 |
docs: correct the bot-score claims behind the browser poller (#99)
Docs only. No code changes - `git diff origin/main --stat` touches five Markdown files and adds one research note. ## What was wrong Several docs explained Cloudflare challenges as a "bot score" that our request rate could worsen. That mechanism does not exist on these sites. Researched live on 2026-08-12 against Cloudflare's own documentation and blog plus RFC 9309 - 22 primary pages, every claim carrying a source URL and read date, seven areas explicitly marked `Not publicly documented`. The note is `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. - The 1-99 bot score is **Enterprise Bot Management only**. A free-plan zone has no score at all; it gets Bot Fight Mode, which matches *signatures* (headless browsers, cloud-hosting IPs). - **No per-IP request rate is documented as an input to challenge issuance.** Volume is policed by Rate Limiting Rules, a separate opt-in product: one rule, IP-only counting, 10-second windows on Free. Published DDoS thresholds are ~1,000 errors/sec. - **`cf_clearance` defaults to 30 minutes**, so every cadence at or above 1 hour re-solves the challenge anyway. Cadence changes how many ~4s solves happen per day and nothing else. - The documented risk is **fingerprint quality**, which this repo already solved (real Chrome, stock UA, non-UTC clock). ## What changed | File | Correction | |---|---| | `AGENTS.md` | The block is per-zone configuration plus request fingerprint, not IP reputation. comix.to turning its gate on 2026-08-12 is the worked example. Residential egress avoids the cloud-hosting-IP *signature* rather than earning a better score. The UTC measurement stands; its mechanism is now marked undocumented. | | `backend/AGENTS.md` | Says why `_BROWSER_COOLDOWN` is longer: cost, not safety. | | `docs/adr/0003` | Dated correction - the sites do not "bot-score" the VPS IP. Decision stands on its sweep-depth argument. | | `docs/adr/0006` | Dated correction - no score to be better at. Decision stands on VPS memory. | | `DEPLOY.md` | A red kagane smoke run means the Site's settings or this Chrome's fingerprint moved, not "Cloudflare's scoring". | ADRs got dated `Corrected 2026-08-12:` paragraphs rather than silent rewrites - the record of what was decided stays intact, only the wrong mechanism is retracted. ## Deliberately not in this PR - **The 6h browser cooldown is unchanged.** I had lowered it to 1h and reverted that; cadence is a behaviour change and belongs with the comix work in #98, not in a docs correction. - **Two code comments still carry the myth**: `backend/main.go:83-84` ("a hammer against sites that are already bot-scoring us"). Left alone to keep this diff docs-only. Related: #98. Reviewed-on: #99 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
c62c3bb07b |
chore: drop asuracomic.net from the userscript, CORS allowlist and docs (#97)
Closes #96. ## What Removes every reference that still invites a Reader onto `asuracomic.net`. The domain's deep links 301 to the `asurascans.com` **root**, discarding the path (re-checked 2026-07-25), so a page on it never yields a series document client-side and a stored address on it never yields a series page server-side. #95 already pinned each Site to one hostname, so the backend rejects such an address cleanly; this is the cleanup around that. | File | Change | |---|---| | `userscript/manga-bookmark.user.js` | drops the `@match`, narrows the asura adapter to `/(^\|\.)asurascans\.com$/` | | `userscript/test/logic.test.js` | new test pinning the narrowed host match | | `.env.example`, `docker-compose.yml` | origin dropped from the `ALLOWED_ORIGINS` default | | `DEPLOY.md` | same, and the sample list gains the two novel origins it was missing | | `backend/api_test.go` | CORS fixtures and round-trip seed move to `asurascans.com` | | `README.md`, `AGENTS.md` | notes say the host is dropped, not "stays matched" | ## Behaviour - A Reader landing on `asuracomic.net` gets no userscript UI. Previously the script loaded and could do nothing useful — the redirect had already discarded the path. - A request whose `Origin` is `https://asuracomic.net` is no longer reflected by a deployment using the shipped defaults. - No backend logic changed: the CORS rule, the address gate and the poller are untouched. `AllowedOrigins` is data, not code. ## Security invariant preserved CORS still reflects `Origin` only when it appears in `ALLOWED_ORIGINS`, with `GET,PUT,DELETE,OPTIONS` and a `204` preflight — `TestCORSPreflight` and `TestCORSDisallowedOrigin` still pin both halves, now against a live origin. This change only removes a value from the allowlist, which is a narrowing. ## Verification - `go test ./...` — full backend suite green (real Postgres per package). - `node --test test/*.test.js` — 66/66 green, up one from the new match test. ## Deploy note (does not happen on merge) The live allowlist comes from the VPS `.env`, not from these defaults, so the origin must be dropped there in the same deploy. The one-off row repair for any stored `asuracomic.net` address is in #96. Reviewed-on: #97 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.3.0 |
||
|
sulthan
|
21615be2bd |
feat: one registry entry per Site, one shared Series-page read (#95)
Closes #94. ## What Two phases per the spec, in three feature commits plus two review-fix commits: **Phase one — one registry entry per Site** (`2d134fb`) The six per-site comparison points that used to live across three files collapse into one `sites` map in `backend/internal/latest/sites.go`: Latest Chapter parse, Cover parse, browser-backed list, fetcher route, host pins, and the browser payload read all become lookups into it. `browserBackedSites()` is derived from the registry (sorted, deterministic); `fetcherFor` and `fetchableSeriesURL` keep their signatures and become lookups; `BrowserFetcher.Get` dispatches through the entries' `Read`/`Done` while the tab lifecycle stays in `BrowserFetcher.run`. **Phase two — one shared Series-page read** (`f215130`) `readSeriesPage` (new `read.go`) performs the read the Poll and the Acquisition have in common: gate, route, fetch, parse Latest Chapter, parse Cover address. It returns facts only — polling and persistence policies (stamp order, cooldowns, cover policy) stay with the callers; `acquire.go` gained the comment naming the deliberate post-fetch stamp order. The poll's legacy cover heal and the no-chapter byte-count diagnostic were restored after review (`d998f87`) so the claims "the Poll keeps its own Cover policy" and "pinning is the only behavioural change" both hold. ## Behaviour - All six Sites now pin their host exactly; asura/demonic/comix previously accepted any https host. For asura this is a strict improvement: its dead old domain redirects deep links to the site root and would parse the wrong document. - Everything else is unchanged: existing parse tables, the challenge-body table and the gate table pass unmodified except the one deliberate exception — the gate table gains the three new pin cases. ## Security invariants preserved - The address gate is recognisably the same rule, now a single registry lookup: `https` + exact hostname match, all callers route through it. No fetch path was widened; asura/demonic/comix were narrowed. - The second host pin inside each browser entry's Read is retained deliberately (browser = strong SSRF primitive, `series_url` is client-supplied) and is not deduplicated against the shared gate. - Review hardening: `fetcherFor` now fails closed for unknown site strings (previously fell through to the TLS fetcher on an unreachable path), and the browser dispatch iterates a sorted list so outcomes cannot depend on map order. - The security review's log-injection finding was checked against Go's `url.Parse` and does not hold: control characters are rejected anywhere in a URL, so a client-supplied value in a log line cannot carry a newline. ## Review Reviewed on three axes (spec, standards, security) by read-only subagents over `672c16f..f1b26f4`. No blocking findings; all minor/nit findings addressed in `d998f87` and `700de20`. Verified end to end with `go test ./...` (Docker Postgres per test package) on every commit. ## Out of scope (tracked separately) - Dropping asuracomic.net (CORS allowlist, userscript match, API fixtures, live env) — separate issue, per spec. Reviewed-on: #95 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
672c16ffbf |
Remove the client latest-chapter scan for lightnovelworld (#91) (#93)
Follows the spec published on #91: remove the client-side latest-chapter scan for lightnovelworld rather than porting #87's truncation into a second codebase. - lightnovelworld.latestChapterFromAnchors deleted, not stubbed: absence is what the background-fetch guard keys off. - computeLatestChapter tolerates an adapter with no scanner (yields null) and is exported as the test seam. - backgroundRefreshLatest skips a scanner-less Site before the due filter: no Series page fetched, no freshness timestamp recorded, no batch slot consumed. The on-page path (maybeCaptureLatestOnSeriesPage) routes through the same null-tolerant computation. - novelfull's scanner, the shared max-chapter helper and all four manga Sites untouched. - userscript/AGENTS.md records the Poll-only contract for this Site and why. All seven acceptance criteria from the spec met. node --check clean; novel suite 30/30 (the regression pin fails if a lnw scan is reintroduced, scoped or not); manga suite 35/35, manga userscript byte-for-byte unchanged. Two-axis code review: no hard standard violations, spec-clean; one follow-up commit matching the sibling adapter guard from the manga script. Reviewed-on: #93 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e7e22a12a5 |
lightnovelworld Series identity is read from the chapter page (#80) (#92)
Implements spec #80 / ADR-0008 — Gitea issues #86, #87, #88, #89, #90, all closed. A Reader bookmarks a novel on lightnovelworld and it never shows a New Chapter, because the Series identity was derived from the chapter address instead of read from the page. One Series can publish under several Chapter Slugs, so the derived key points at a slug that 404s. - **#89** — the userscript's lnw adapter stops deriving `seriesUrl`/`seriesId` from the path. It reads the page's own pointer (`a[aria-label='All Chapter']`), falling back to the microdata breadcrumb's second crumb, and carries `chapterSlug` on the page object, stored nowhere. - **#87** — the Poll's lnw chapter scan is unscoped (no stored-slug pattern can cover a Series' whole list) and truncated at the `wpd-threads` comment thread, the one region a visitor can write to. Marker absent means skip and log with the body length, never scan whole. Corrects the `maxBodyBytes` headroom comment to the measured 3.5x. - **#86** — the scan fixture is now text trimmed from a real, wholly-fetched Series page instead of a hand-written cross-series anchor that no live page carries. - **#90** — stale stored rows repair themselves on the next chapter visit: a pure transform over cache, queue and last-checked map, silent to the Reader, with progress, favourite and lifecycle bucket preserved when two rows merge. - **#88** — an env-gated live canary (`SMOKE_LNW_SERIES_URL`) proving the marker still occurs exactly once and still follows the last chapter anchor, asserted against the production symbols themselves. Verified on the merged branch: `go test ./...` green, `gofmt -l internal/latest/` silent, both userscripts `node --check` clean, 35/35 + 29/29 logic tests. Live canary green (marker once at byte 612,182 of 651,795). #90 verified on device with Playwright. Open follow-up: **#91** — the userscript's client-side latest-chapter scan is still scoped to the derived slug. Reviewed-on: #92 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
90d8ab72ad |
chore: refresh graphify map and tidy AGENTS.md (#84)
graphify update regeneration: semantic hashes now populated in manifest.json, graph rebuilt (1634 nodes, 3179 edges). Track the map (5 curated files + .graphify_root) so a fresh checkout starts with it; cost.json, cache/, dated snapshots and .rebuild.lock stay ignored. AGENTS.md: drop stale Relevant skills and Notes sections; graphify rule now says to always query the graph first. Reviewed-on: #84 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
c400c91a80 |
Implement a batch of tickets through per-ticket subagents (#82)
## What this adds Two files that turn the one-ticket-at-a-time `/implement` loop into an orchestrated batch. **`.claude/skills/implement-tickets/SKILL.md`** — user-invoked (`disable-model-invocation: true`, so it costs no context until typed). The agent that runs it is an orchestrator, not an implementer: 1. Collect the tickets over `tea`, reading each `Blocked by` line. 2. Plan waves from the blocking edges, three tickets wide, and fix every cross-ticket contract (shared signature, JSON shape, column, token) before anything is dispatched. 3. Present the plan and stop for approval. 4. Per ticket: `git worktree add ../ticket-<n>`, copy the gitignored `.env`, claim the issue, write a brief to `.scratch/`, then dispatch the whole wave as one `task` batch. 5. Land each result — merge `--no-ff`, comment the report, close, remove the worktree. Textual conflicts are the orchestrator's; a semantic clash goes back to whichever ticket owns the contract. 6. Full suite once on the merged base. **`.omp/agents/ticket-implementer.md`** — the worker. Brief-driven, worktree-bound, and gated on review before it reports: it runs the `code-review` skill over its own diff with `cr-spec` and `cr-standards` on the two axes, fixes Critical and Important findings in at most two rounds, and returns a short status contract (`DONE` / `DONE_WITH_CONCERNS` / `BLOCKED` / `NEEDS_CONTEXT` / `REVIEW_BLOCKED`). The brief template makes the subagent read `tea issue <n> --comments` for its ticket and for the issue that ticket refers to — the comments carry decisions the body never got updated with — and names the `tdd` skill at each seam where a test comes first. Briefs are written in the ubiquitous language of `CONTEXT.md`; a brief that says "scrape" where the domain says Poll hands the subagent the wrong model of the system. ## Verification Dispatched a real `ticket-implementer` as a probe. The agent resolved from `.omp/agents`, and it spawned `cr-spec`, which replied. That was the one thing that could have silently killed the design: `task.maxRecursionDepth` defaults to 2, and the chain is session to orchestrator to implementer to reviewer. It clears. If that ever changes, the implementer returns `REVIEW_BLOCKED` and the orchestrator runs the review itself. Confirmed against the omp binary that `autoloadSkills: code-review, tdd` is split by `parseArrayOrCSV`, not swallowed as one unknown name. ## Notes - Agents are discovered from `.omp/agents`, never `.claude/agents` — the latter is deliberately skipped by omp because its frontmatter is a different contract. - No product code changes. `.gitignore` gains `.scratch/`, where briefs and reports live. - Not included: retry after a failed dispatch, a state file for resuming a crashed wave, a cheap model tier for mechanical tickets. Add them when a real batch needs them. Reviewed-on: #82 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
f1eb7d514c |
Record the lightnovelworld series-identity decision (#77) (#81)
Docs only. No code, no tests, nothing to run. Implementation is specified in #80. Outcome of a grilling session on 2026-08-11 against #77, backed by live measurement of lightnovelworld over 2026-08-10/11. ## What changed **`docs/adr/0008-series-identity-is-discovered-not-derived.md`** (new) A Series identity is discovered from the Site's own links, never derived from an address. On lightnovelworld the userscript reads the chapter page's `All Chapter` anchor instead of building a `/novel/<slug>/` address by string manipulation. A Chapter Slug is not an identity and is not stored. The backend's chapter scan drops its per-Series scoping and runs against the body truncated before the visitor comment thread. Evidence in the ADR: 3 of 41 sampled novels serve chapters under a slug that differs from their series slug, divergence runs in both directions, one novel serves chapters under two slugs, and neither slug is computable from the other. The pointer was checked on 8 chapter pages and agreed every time. Three narrower selectors are recorded as rejected, each with the measurement that killed it. Three rejected options are recorded with reasons: correcting the stored address only, which keeps an identity the Site does not guarantee; scoping the scan to a container, which the probe refuted; and a SQL migration, which is impossible because the database holds no source for the correct slug. **`CONTEXT.md`** - **Series** - identity is the canonical slug the Site publishes, never the title and never a Chapter Slug. - **Chapter Slug** - new term. A slug a Site builds its chapter addresses from. Not an identity: one Series may have several, and none is computable from another. - **Latest Chapter** - now the highest-numbered chapter, explicitly not a date and not the Site's own newest-chapter banner. Settles #79. **`docs/research/lightnovelworld-chapter-vs-series-slug.md`** (new, committed with its corrections) The 41-novel survey behind the ADR. Two claims are struck through and corrected in place, with the date and sample size of the probe that refuted each: the `ul.clstyle` container it named is the hidden, empty "Latest Reading" template rather than the chapter list, and its caveat about the comment region understated the risk, because that region is writable by any visitor while the scan takes an unbounded maximum into a Series row shared by every Reader (ADR-0003). ## Review notes Nothing here constrains code that exists today - the ADR describes work not yet written. The part worth disagreeing with, if any of it is wrong, is the fail-closed rule: a missing truncation marker means skip the Series and log, never scan the whole page. Related: #77 (the defect), #80 (the spec), #79 (the numbering anomaly, closed by decision), #71 (the same size cap seen from the cover side). Reviewed-on: #81 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
1ee5eb67ea |
Clear the stale Chrome singleton lock at browser boot (#76)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.2.1 |
||
|
sulthan
|
b22ae82897 |
Restore the novel script's lost module-scope constants (#74) (#75)
> *This was generated by AI during triage.* Fixes #74. `novel-bookmark.user.js` was split out of `manga-bookmark.user.js` and lost four module-scope constants. Every use of them is behind a `try/catch` or a fire-and-forget promise, so the `ReferenceError`s were swallowed rather than reported. | constant | used at | effect while missing | | --- | --- | --- | | `LATEST_CHECK_THROTTLE_MS` | `:722` | `backgroundRefreshLatest()` throws before computing `due` — no background latest-check ever runs for novels (the symptom in #74) | | `LATEST_CHECK_BATCH` | `:724` | same throw | | `CACHE_KEY` | `:215`, `:224` | `loadCache()` always returns `[]`, `saveCache()` silently no-ops — the local cache never persists | | `LASTCHECKED_KEY` | `:232`, `:241` | last-checked map never persists, so the throttle would not hold even once the first two are defined | #74 named only the two throttle constants. The two cache keys are the same lost lines with the same root cause, so they are restored here too — fixing only the pair the issue named would leave `backgroundRefreshLatest()` re-fetching every series on every navigation, because `saveLastChecked()` would still be a no-op. Values and comments copied verbatim from `manga-bookmark.user.js:37-43`; throttle 4h, batch 1. ## Verification - `node --check userscript/novel-bookmark.user.js` — clean. - `node --test userscript/test/logic.test.js userscript/test/novel-logic.test.js` — 47/47 pass. - New test `every SCREAMING_CASE constant the script uses is declared in it` scans both scripts (comments and string literals stripped first, so prose and SVG path data do not trip it). Confirmed it fails — 1 failing test — when `LATEST_CHECK_BATCH` is deleted again, and passes when restored. A behavioural test cannot reach this: the storage helpers and the background refresh are exactly the layers the harness does not cover (see the `testing-the-userscript` skill), and the errors are swallowed anyway. A static guard is the only instrument that sees this bug class. On-device confirmation that the ember now lights for novels is still outstanding — that needs Violentmonkey against a live novelfull/lightnovelworld page. Reviewed-on: #75 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.2.0 |
||
|
sulthan
|
7c7d597019 |
Delete the kagane-specific cover path (#63) (#73)
Closes #63 Deletes the second way to reach a Cover. Since #62, every Site's cover bytes land in the content-addressed store at creation or on the poll, and the one public route serves them all — nothing needs the kagane proxy anymore. ## What went - **Template-level rewrite:** `Bookmark.CoverURL()` and both templates' use of it. Cards and chrome now render `.Cover` — the wire value — and nothing else. `Bookmark.CoverSource` was dead once `CoverURL` went, so it and its `bookmarkColumns` entry are gone too. - **Kagane-only cover route and its identifier validation:** `GET /img/kagane/{id}`, `web.CoverFetcher`, `coverIDRe`, and the whole `internal/web/cover.go`. - **The proxy's persistence:** `store.KaganeImageID`, `GetKaganeCover`, `PutKaganeCover`, `kaganeCoverSourceURL`, `kaganeCoverRe`. - **The kagane-shaped branch in the byte-fetch routing:** `fetchCoverBytes` no longer takes a `site` argument and no longer names a Site. The URL shape kagane's API publishes is claimed by the browser module itself — `kaganeImageURLRe` + `browserCoverURL` live in `latest/browser.go` with the rest of the per-Site knowledge — and `BrowserFetcher.Image` is now URL-driven (it validates the URL it will navigate to, same SSRF discipline as before). The no-plain-TLS-fallback rule for a claimed URL is preserved: a claimed address with no browser is an error, never a challenge-page fetch. ## What stayed (deliberately) - `BrowserFetcher.Image` and the browser-backed acquisition path: kagane genuinely serves cover bytes behind the challenge + `cross-origin-resource-policy: same-origin`, so the sidecar remains the only fetcher for them — it just routes by URL claim now instead of by Site name. - `fetcherFor`'s per-Site page routing (kagane/novelfull page fetches) — that is the page path, not a cover path. ## Acceptance criteria - [x] Template-level kagane cover rewrite gone - [x] Kagane-only cover route and its identifier validation gone - [x] Tests removed/rewritten against the general route, guarantees kept: unstored + traversal-shaped addresses serve nothing (`TestPublicCoverRejectsUnknownAddress`), non-image content types never echoed (`TestPublicCoverNeverEchoesNonImage` — new; the store-side gate was already pinned by `TestCoverStoreAcceptsAnySourceURL`). Store reopen-persistence and filesystem content-addressing tests rewritten against `PutCover`/`GetCover`, no guarantee lost. - [x] No Site name in a cover code path outside the acquisition module (`grep kagane backend`: store/web/templates/api are clean; remaining hits are `latest/browser.go` + `latest/sites.go`, tests, docs) - [x] Web UI and panel render Covers for all six Sites (templates render the wire address; panel renders `b.cover` — untouched, it never had a kagane path) - [x] `go test ./...` green ## Verification - `go vet ./...` clean - `go test ./...` — all packages pass (root 16.9s, latest 12.7s, store 12.7s, web 0.004s) - `CGO_ENABLED=0 go build` produces the static binary - Cover-path tests run verbosely: `TestPublicCoverServesStoredBytesUnauthenticated`, `TestPublicCoverRejectsUnknownAddress` (unknown/malformed/traversal/empty), `TestPublicCoverNeverEchoesNonImage`, `TestListRendersAcquiredCover`, `TestAcquireKaganeCoverThroughBrowser`, `TestRunOncePrefetchesKaganeCover`, `TestRunOnceRoutesNonKaganeCoverToPublicFetcher` all pass; the three `SMOKE_*` tests skip without the browser sidecar, as designed Live browser verification of the "web UI and panel render Covers for all six Sites" criterion is being run separately with Playwright against real Site pages and a locally mocked backend. Reviewed-on: #73 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
78234f3c19 |
Browser-backed Sites join the Cover pipeline (#62) (#72)
Fixes #62
Browser-backed Sites join the Cover pipeline: kagane and novelfull Series now get their Covers at creation, through the same acquisition path as every other Site, instead of waiting for a poll pass.
## What changed
`latest.Acquirer` (creation-time acquisition, fired by the first Bookmark of a Series) previously skipped kagane and novelfull entirely — their pages only yield a Cloudflare challenge to the TLS client, so the request was spent for nothing. It now routes them like the poller does, with the two Sites split exactly as the issue demands:
- **kagane** — page fetched through the browser sidecar, cover URL extracted from the API JSON, bytes fetched through the browser sidecar (the only path that clears the challenge) into the content-addressed store. With no `BROWSER_WS_URL` configured, acquisition is skipped entirely and nothing falls back to a plain fetch.
- **novelfull** — page fetched through the browser sidecar, cover URL extracted from the HTML, bytes fetched over plain TLS through the ordinary gated fetcher (its image paths answer 200 with `access-control-allow-origin: *`, measured 2026-08-09). With no browser configured, the page fetch falls back to the TLS client — novelfull's challenge is a live time-varying fact (AGENTS.md), so when the page body answers, the Cover still lands; when it is challenged, nothing happens.
The byte-routing rule (kagane → browser, every other Site → TLS) is now one shared function (`latest.fetchCoverBytes`) used by both the Poller and the Acquirer, so the two cannot drift apart.
## Acceptance criteria
- [x] kagane cover bytes are fetched through the browser sidecar and stored in the content-addressed store — `TestAcquireKaganeCoverThroughBrowser`
- [x] novelfull cover URLs are extracted from the browser-fetched HTML, and its bytes are fetched over plain TLS — `TestAcquireNovelfullCoverOverPlainTLS`
- [x] With no browser sidecar configured, kagane Covers are absent and nothing falls back to a plain fetch — `TestAcquireKaganeSkippedWithoutBrowser`
- [x] With no browser sidecar configured, novelfull Covers still work if its page body is available — `TestAcquireNovelfullCoverWithoutBrowser`
- [x] Manually verified on-device: a kagane Series shows its Cover in the panel, not a broken-image glyph — being run by a separate manual-verification agent against a mocked scenario (no prod data); not part of this PR
- [x] `go test ./...` is green, with live-network checks gated behind `SMOKE_BROWSER_WS_URL` like the existing kagane image smoke test — new `TestSmokeAcquireKaganeCover` proves the end-to-end acquire path against the real browser when the env var is set
## Verification
- `go test ./...` green across all packages
- New unit tests exercise every routing decision with fakes — no network in the default suite
- Smoke test gated behind `SMOKE_BROWSER_WS_URL`, skipped by default
## Post-review changes (
|
||
|
sulthan
|
b9220b3dfc |
The Poll fills blank Covers for every Site and both Libraries (#70)
Closes #61. ## Summary Permanently-blank Series (the half of #47 that creation-time acquisition cannot reach) heal on the next due poll cycle. The cover path is no longer kagane-only: every Site and both Libraries fill a blank Cover from the series page the chapter poll already fetched, and never replace a Cover that already exists. ## What changed ### `backend/internal/latest/poller.go` - **`fillBlankCover`** — when `Cover` and `CoverAddress` are both blank, extract a source URL via `coverFrom` from the series-page body and store bytes through `SetSeriesCover`. Skips any Series that already has a source URL (owned by prefetch) or a stored address (never overwrite). - **`prefetchCover`** — source-URL healing path, now site-uniform. Kagane no longer special-cases into `PutKaganeCover` alone; every Site lands on `SetSeriesCover`, so the wire Cover becomes a content-addressed public URL. Reuses already-stored bytes when present. - **`storeCover` / `fetchCoverBytes`** — shared fetch+persist. Only kagane routes image bytes through the browser fetcher; every other Site uses plain TLS `CoverBytesFetch`. Failures log with the Series key and never return to the chapter path. - **`checkOne`** — after a successful series-page fetch, calls `fillBlankCover` once regardless of whether chapter extraction succeeded (cover fill is independent of the chapter signal). ### `backend/internal/latest/poller_test.go` Extended the existing poller harness (real store, fake fetchers) rather than a new one: - `TestRunOnceFillsBlankCoverFromSeriesPage` — asura manga, lightnovelworld novel, kagane manga; asserts wire Cover + correct fetcher routing. - `TestRunOnceDoesNotReplaceExistingCover` — second poll does not refetch. - `TestRunOnceRetriesFailedBlankCoverOnNextPoll` — failed fill stays blank, next due cycle retries (no separate queue). - `TestRunOnceBlankCoverFailureDoesNotBlockChapter` — chapter still lands; failure log carries the Series key. - Kagane prefetch test now also asserts the content-addressed wire Cover. ## Acceptance criteria (#61) | Criterion | Status | |---|---| | Cover prefetch runs for every Site | done | | Cover prefetch runs for both Libraries | done | | Poll fills a blank Cover | done | | Poll never replaces an existing Cover | done | | Failed cover fetch does not fail/block chapter poll | done | | Failed cover fetch retried next poll, no separate queue | done | | Failures logged with the Series | done | | Existing poller tests extended | done | | `go test ./...` green | done | | Manually verified: blank Series gets Cover after a poll cycle | **left for you** | ## Out of scope / not closed - Does **not** close #47 or #55 (per ticket). - No migration/backfill script — the Poll walks every Series already. - No admin refetch (#54). ## Review notes addressed - Removed the kagane-only `PutKaganeCover` branch from prefetch so source-URL healing also sets `CoverAddress` (wire Cover). - Guard so `fillBlankCover` does not double-fetch after `prefetchCover` healed the same snapshot. - Single `fillBlankCover` call site after the series-page fetch. ## Test plan - [x] `go test ./...` (backend; needs Docker/Postgres via `pgtest`) - [ ] After deploy: pick a Series that was blank, wait one poll cycle, confirm Cover in web UI and userscript panel Reviewed-on: #70 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e2c054e7ce |
Covers render in the userscript panel, from a public route (#60) (#69)
Closes #60. Spec: #55. Originating bug: #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Neither #47 nor #55 is closed from here. ## What this branch does The panel now renders Covers from the deployment's own origin, and both userscripts stop having an opinion about where a Cover lives. **The public route was already in place.** `GET /covers/{address}` landed with #59 (`92eba07`) and is registered on the bare mux, outside `httpmw.Auth` and outside the web UI's Discord session — `backend/main.go:210-214`, handler `backend/internal/api/handlers.go:142-158`. It reads no cookie and no header, answers `404` for an address that was never stored (and for a row whose file has gone missing — recorded-but-gone is not-found, never a fabricated body), refuses anything that is not `^[0-9a-f]{64}$` *before* the value becomes a path, and sets `Cache-Control: public, max-age=604800, immutable`. Those four properties are asserted by `backend/cover_test.go:231-278`. This branch re-verified them rather than re-implementing them; the only backend line it touches is a comment. **Both userscripts lose cover scraping entirely.** Every adapter's `cover:` field is gone, along with the two helpers that fed them: the manga script's `coverFromPage()` (the `img[alt]` DOM scan comix needed, because comix publishes no `og:image`) and the novel script's `metaName()` plus the now-callerless module-level `meta()`. Nothing under `userscript/` reads `og:image`, `meta[name=image]`, or `img[alt]` any more. **Nothing sends a cover either.** `delete body.cover` sits in `apiPut` — `manga-bookmark.user.js:486`, `novel-bookmark.user.js:275` — which is the single chokepoint every write passes through (`pushBookmark`, the retry-queue flush, `toggleFavorite`, `toggleArchive`). It operates on the `Object.assign` copy, so the in-memory row keeps the cover it renders with. This matters beyond tidiness: a Reader upgrading from an older copy has `localStorage` rows carrying third-party scraped URLs, and without the strip those would ride back up on the next write. The handler discards the field regardless (`handlers.go:53-59`) — it is permanently inert, not pending removal. **Failed loads get the designed empty state, not the broken-image glyph.** `onerror: (e) => e.target.replaceWith(el("div", { class: "cover ph" }))` on the cover `<img>` in both card renderers (`manga:1380-1390`, `novel:1134-1144`). The replacement is byte-identical to the existing no-cover branch on the very next line, so it picks up the `.cover.ph` styling already in the panel CSS — no new tokens, no new rule. `el()` routes any `on*` prop through `addEventListener`, so this is a listener, not an inline attribute string, and the swap is a `createElement` + DOM call with no markup parsing anywhere near it. This is the half of #47 that was visible on kagane. **The deleted scraping's tests went with it**: the two comix cover cases, the `pageImages` and `namedMetas` fixtures, the `img[alt]` and `meta[name=...]` stub branches, the now-dead `querySelectorAll` stub member, and every stale `og:image` fixture and `p.cover` assertion across both suites. The export lists needed no change and that was checked, not assumed — `coverFromPage` and `metaName` were module-private on `origin/main` and no cover symbol ever appeared in `module.exports`. Docs that described the deleted behaviour were corrected in the same breath, because leaving them would instruct the next agent to put the scraping back: `userscript/AGENTS.md` (adapter contract + the per-site notes for comix, kagane and novelfull), the README's adapter reference, and the userscript testing skill's stub table. ## Verification - `go test -count=1 ./...` — green across all nine packages (`backend` 29.8s, `latest`, `store`, `session`, `token`, `userscript`, `web`). - `node --check` clean on both userscripts; `node --test` on both logic suites — 46 tests, 46 pass. - `gofmt -l` clean; `go build ./...` clean. - The `onerror` swap is DOM behaviour and deliberately has no coverage in the Node harness — that harness stubs a browser precisely so it never needs a DOM, and #60 says not to invent coverage for it. It was instead exercised for real: the `el()` helper and the exact render expression were loaded into a headless Chromium with a deliberately unloadable `src`, and the resulting DOM was `<div class="cover ph"></div>`. Ad hoc, not committed. - **Not done, needs you:** the on-device criterion — a comix Series bookmarked mid-chapter showing its Cover in the panel. That needs a real install against the deployment and is the one box left unticked on #60. ## Reviewed Both `/code-review` axes ran against `cc0fa92`. Spec found no missed requirement and no scope creep; standards found the diff clean on the four areas it scrutinised (the `delete body.cover` placement, the `onerror` handler's DOM safety, comment quality, dead-code removal). Their combined findings — the dead `querySelectorAll` stub, the stale README and skill text, and the handler comment whose premise this change invalidates — are fixed in `8b58019`. ## Out of scope, deliberately The kagane-specific cover proxy still exists and still carries its session gate (#63 deletes it). The poll's blank-Cover fill (#61) and browser-backed Sites joining the pipeline (#62) are untouched. Reviewed-on: #69 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
92eba07da7 |
A newly bookmarked Series acquires its Cover at creation (#59) (#68)
Closes #59. Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55. ## What changed A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s. ### Store - Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on. - `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there. - `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address. - The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path. - `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load. ### Acquisition - `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position). - Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them. - Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid. - Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later. - Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62. ### Wire and route - `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk. - `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep. - `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading. ### Config `PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`. ## Acceptance criteria All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence. ## Verification - `go test ./...` green (Docker-backed Postgres suite). - Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type. - Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red. ## Reviewed Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail. ## Known sequencing A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly. Reviewed-on: #68 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |