caabdff15b72f360d993fbbf01ce08c48dafe9f8
111 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 sulthan
|
caabdff15b | Tidy blank line before RecordSeriesFailure (#165) | ||
|
sulthan
|
417f809d75 | Record poll failures as rows: the row is the failure state (#165) | ||
|
sulthan
|
4708703044 | Merge #168: All six Sites' completed marker, read by their adapters | ||
|
sulthan
|
7ef9f8c433 | Merge #164: A sixth outcome word: not_found splits out of errors | ||
|
sulthan
|
b38dfe54e8 |
Record per-site probe provenance in completed fixtures (#168)
Spec review flagged the live-verification acceptance criterion as unproven in the diff: the brief's adapter rule wants the probe recorded, not just the date. Each fixture comment now names how the live body was fetched on 2026-08-22 — curl probe for asura, demonic and lightnovelworld; cleared Chrome tab (CDP sidecar) for comix, kagane and novelfull, including the challenge/403 fallback story for novelfull. Also renamed lnwCompletedRe to lnwStatusRe for symmetry with the other site-marker vars. |
||
|
sulthan
|
f6782e212a | Note comix key-encoding fidelity in fixture comments (#168) | ||
|
sulthan
|
e69e513be4 |
Add completed markers for all six sites (#168)
site gains a Completed predicate answering whether a fetched body carries the Site's own completed value: asura escaped props status, demonic info-block Status pair, comix scoped detail entry (shared comixDetailQuery helper with the cover read), kagane publication_status only, novelfull status link, lightnovelworld JSON-LD creativeWorkStatus. siteCompletedFrom dispatches it; an absent hint stays false. Fixtures trimmed from live pages fetched 2026-08-22; TestSiteCompletedFrom covers completed, ongoing, selector-removed, and challenge bodies per site, the kagane upload/publication divergence, and an unknown site. |
||
|
sulthan
|
f4e4055a13 | Review: drop duplicate 404 log, scope comment wording to 4xx (Spec #164) | ||
|
sulthan
|
62c2cea74e | Spec #164: A sixth outcome word: not_found splits out of errors | ||
|
sulthan
|
8e4fa6448e |
Spec #136: Finished belongs to the Series — owner-owned poll gate, Lifecycle bucket dropped (#163)
Closes #136. Spec #136 end to end: `finished` becomes a fact about the Series, written only by the owner, and the reader-facing Lifecycle bucket is gone. ## What landed - **#157** — `series.finished_at bigint NOT NULL DEFAULT 0` plus the migration whose statement order is load-bearing (seed from the buckets, then flip them); both Lane queries lose the `HAVING COUNT(*) FILTER (WHERE b.status <> 'finished')` clause and gate on `finished_at = 0` instead, with the due-query/eligible-count force asymmetry kept deliberate and commented; `StatusFinished`, its API special-case 400, the web tab and the templates' Finished bucket deleted. - **#158** — owner Finish control on the Series detail page: confirm-gated finish, instant un-finish, admin accent (never ember, nothing is destroyed), `Store.SetSeriesFinished`, the two routes behind the owner gate, and the state displayed on the list row without offering the control there. - **#160** — reader side: derived `finished` bool on the flat Bookmark (`s.finished_at > 0`), rendered as a text-only label in both userscripts and on the web card; read-only inbound by omission from `Upsert`'s explicit `series` column list, same mechanism that already protects `cover`. - **#161** — glossary and the stale Reader-count divergence note catch up. - **#159** — `finished` joins the admin filter vocabulary (predicate `finished_at > 0`, label `Finished`, own aggregate count, figure last in the stats block as informational); the four clock-driven hygiene predicates (stale, never-checked, no-cover, no-chapter) exclude finished Series while unpollable, orphan and sighting-raised deliberately do not. ## Verification `go vet ./...` and `go test ./...` green on the merged branch (Docker-backed, throwaway `postgres:17-alpine` per package). Each ticket also passed a two-axis review (spec + standards) on its own branch before merge. Reviewed-on: #163 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
faa80c41ea |
Skills: split plan-tickets out of implement-tickets (#162)
Adds `.claude/skills/plan-tickets/SKILL.md` and trims `implement-tickets` to dispatch-only, with the matching `.omp/agents/ticket-implementer.md` update. Docs/skills only — no backend, userscript, or web changes. Reviewed-on: #162 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
4aaf1d4f91 |
Spec #135: owner data-correction actions — Latest Chapter, series_url, Cover, orphan removal (#156)
Implements spec #135 (spec 2 of 4, derived from wayfinder map #114; decisions settled in #120/#121/#125/#131). Blocked-by #134 is merged, so this lands on `main`. Four owner actions the dashboard can now perform, one ticket each: - **#149** — Latest Chapter correction: one numeric input, overwritten by the next machine write. - **#151** — Series URL repair: owner-typed, gated by the poller's own fetch gate. - **#150 / #153 / #154** — Cover replacement: addresses derived from bytes (`#150`), a Forced Poll replaces the Cover while an ordinary pass still only fills a blank one (`#153`), and byte reclamation is one guarded helper, file first / covers row last (`#154`). - **#155** — Orphan removal: one Series at a time, with the foreign key as the guard. Plus **#152** — Latest Chapter provenance: one derived line naming the actor class, so an owner can tell a hand-edited number from a machine read. - Migration `0015_latest_correction.sql` adds the correction/provenance columns; `0009` now derives cover addresses from bytes. - ADR `0014-cover-addresses-from-bytes.md` records the address scheme. Backend tests cover the store, poller, admin handlers, and web routes (`go test ./...`, needs Docker). Reviewed-on: #156 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
889f0f3f38 |
Admin dashboard: pages, Lane observability, poll pass log, per-Series intervention (#134) (#148)
Spec #134, all ten tickets. Closes #134. ## What ships The admin surface becomes four bookmarkable addresses behind one nav row, and Lane observability stops dying with the process. - **#138** `/admin` splits into Overview, Lanes, Readers, Series, each a real route with the active tab underlined. - **#139** `poll_passes` and `poll_lanes` land as durable tables with their store surface. - **#140** cross-Series admin read model, with the privacy boundary in the projection: the Reader id that raised a Latest Chapter never leaves the store package. - **#141** the poller records exactly one pass row per exit, with a skip reason and outcome counts. - **#142** Series list: eight hygiene filters, Site and Library narrowing, paging — all of it in the query string, so a filtered list is a bookmark. - **#143** Overview: a three-state verdict line and a stats block where every non-zero figure links to the list that counts it. - **#144** per-Series detail page, keyed by the `site:series_id` composite the rest of the system already uses. - **#145** the Lanes page reads the database; the in-memory Lane state, `web.LaneReporter` and `latest.Status` are deleted. - **#146** Forced Poll: *Check now* stamps `series.force_poll_at` and never commands the poller. - **#147** pause and resume one Site's Lane, with a mandatory 1h/6h/24h expiry. ## Shape of the design Two decisions carry the rest. **Commands go through the database, never at the poller**: both *Check now* and a Lane pause write a row the next pass reads, so they survive a restart and the whole surface stays testable with no poller running. And **pending is derived, never stored** — the request stamp being newer than the check stamp — which self-clears on the check stamp with no second write and no sweeper, because the check stamp is written before the fetch. ADRs: `docs/adr/0012-persisted-lane-state.md`, `docs/adr/0013-commands-through-the-database.md`. ## Verification `go test ./...` green on the merged base (`264839e`), all packages, Docker-backed. `gofmt -l` and `go vet` clean. Every ticket was reviewed on both axes (`cr-spec` + `cr-standards`) before merge. ## Known, non-blocking - **#143** the verdict ignores never-reported Lanes when other Lanes have reported, and the per-Site table lists Sites that have Series rather than the whole registry. The ticket prose asks for eight hygiene figures per Site; the design mock and the landed `.tbl.sites` grid both say six columns, and the mock won. - **#146** two `SeriesPage` scans per press instead of a keyed read — `ponytail:`-commented in-tree with the upgrade path. - **#147** a paused Site with no pass row yet renders no row and so no control, since the Lanes page lists Sites that have passed. - **#141** a sibling browser Lane declining at the top of a pass records as `sidecar-down`. Specified deliberately; the later spec in this series settles it. Reviewed-on: #148 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
0a245a0dde |
docs: name the Stall and the Correction in CONTEXT.md (#133)
Two domain terms the admin surfaces need and CONTEXT.md did not carry: - **Stall** — a Poll Lane that owed Polls, made none, and has nothing to say for it; distinct from a refusing Site and a Paused Lane. - **Correction** — an owner-set Latest Chapter for a Series no Poll can read; lower authority than a Sighting. Docs only. Branch cut fresh off `origin/main`, so it carries nothing from the research branch. Reviewed-on: #133 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
249f11e1fe |
docs: name the admin dashboard's domain terms in CONTEXT.md (#128)
Glossary-only change; no code touched. Charting the admin dashboard map (#114) settled four terms the glossary did not carry: - **Orphan Series** (#125) — a Series no Reader bookmarks; a state of the Series, never a Lifecycle bucket. - **Lane Pass** (#117) — one sweep of a Poll Lane, including a pass that declined to work and why. - **Forced Poll** (#119, #120) — a Poll the owner asks for by marking the Series, which jumps the waiting rules but never the Site's refusal, and which may replace a Cover. - **Paused Lane** (#119) — a bounded, restart-surviving stop on one Site, distinct from the deploy-time kill switch. **Acquisition** is amended in the same pass: establishing a Cover is no longer unique to it, since a Forced Poll can replace one. Reviewed-on: #128 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
f5d3fe58ec |
Add a gitea skill; point AGENTS.md at it (#126)
Forge usage lived in three places (`AGENTS.md`, `docs/agents/issue-tracker.md`, and habit). This moves the how-to-run-`tea` half into a model-invoked skill that fires on any issue/PR task, and reduces `AGENTS.md` to identity plus pointers. - **new** `.claude/skills/gitea/SKILL.md` — command table plus the traps `tea <cmd> --help` will not tell you. - `AGENTS.md` — Forge section is now one line: Gitea not GitHub, `gh` and the `issue://`/`pr://` URIs fail, then pointers to the skill, `docs/agents/issue-tracker.md`, and `docs/agents/triage-labels.md`. - `.claude/skills/implement-tickets/SKILL.md` — pointer split: tracker conventions to the doc, `tea` usage to the skill. Both `docs/agents/` files are untouched; the skill cites them instead of restating them. Facts in the skill are measured against `tea` 0.14.2 on 2026-08-17, not remembered: - `gh` is not installed, so `read issue://71` errors — there is no fallback to add. - **A bare read is a truncated read.** Without `--comments`, `tea issue <n>` drops every comment silently, with no prompt under a non-TTY: issue #123 prints 40 lines bare, 132 with the flag. The skill makes `--comments` mandatory for any read meant to understand a ticket, with `tea issue list --fields index,comments` as the checkable count. - Issues and PRs share one index space; output is rendered boxes so parsing needs `-o json`; `close` takes no `--comment`; labels never auto-create; multi-line bodies need a heredoc; `tea` exposes neither sub-issues nor dependencies. Unmeasured and marked as such: whether `--comments` covers a PR's review-comment stream — no PR in this repo has comments, so `tea pr review-comments <n>` is named without a claim about overlap. Reviewed-on: #126 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
766aa8f00d |
docs: make every AGENTS.md cite code, not docs or issues (#113)
Every `AGENTS.md` now cites code and nothing else.
## Why
Two rot mechanisms, same symptom — an agent confidently follows a stale statement:
1. **Non-code citations.** A spec, ADR, plan file, or issue records what was true when it was written. Nothing updates it when the decision reverses.
2. **Prose restating mechanism.** The code changes, the paragraph doesn't, and the next reader trusts the paragraph.
Code is the only source true at read time.
## What changed
**All three files:** removed every ADR ref, spec/plan pointer (`docs/superpowers/specs/*`, `plans/*`, `docs/research/*`), `DEPLOY.md`/`REDEPLOY.md`, `docs/agents/*`, and issue number. Facts those links carried are restated inline — the `tea` command set and the five triage label strings now live in the root Forge section. `### Domain docs` is deleted: it pointed only at `CONTEXT.md` and `docs/adr/`, neither of which exists.
**`backend/` and `userscript/`:** rewritten around derivability.
| Class | In code? | Treatment |
|---|---|---|
| Structure — packages, routes, env vars, columns | yes | name the symbol, nothing else |
| Mechanism — what a function does | yes | symbol + one line |
| Rationale — why, what a "simplify" breaks | **no** | written out |
| Measurement — observation against a service we don't control | **no** | written out, dated |
`backend/AGENTS.md` 20578 → 15512 bytes, `userscript/AGENTS.md` 7129 → 5912. Root grows 16905 → 19292: the cost of inlining the `docs/agents/*` facts plus the new rule.
**Rule** recorded in root as `## Writing an AGENTS.md`. Sole non-code exception is a sibling `AGENTS.md`. Closing clause: every symbol named must exist, since a dead pointer is a bug rather than a stale sentence.
**Harness-agnostic:** dropped the `Guidance for OpenCode (and Claude Code)` openers for plain scope lines.
## Verification
Applied the new rule to itself — extracted all 118 backticked identifiers across the three files and checked each against every `.go`, `.js`, `.sql`, `.html` and `.css` source. Zero repo symbols missing; the 8 non-matches are external (`GM_setValue`, `navigator.webdriver`, `HeadlessChrome`, `curl`, …).
That check caught a claim that was **already lying** on `main`: the cover section said `CoverFetcher` was gone, but `NewCoverFetcher`, `TLSCoverFetcher` and `BrowserCoverFetcher` are all live in `internal/latest`. Now names only the genuinely dead `/img/kagane/{id}` route. Exactly the failure the rule exists to prevent.
No code touched — documentation only, nothing to test.
Reviewed-on: #113
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
550b258c59 |
fix: give covers their own 10 MiB byte cap (#71) (#112)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.3 |
||
|
sulthan
|
3ac865cd08 |
chore: remove graphify (#111)
Removes the graphify integration. It was measured against this repo rather than assumed. ## Why `graphify query` returns a keyword-seeded BFS neighbourhood, not a location. Asked where CORS origin reflection is implemented, it returned 73 nodes — mostly `api_test.go` helpers, plus a `Reflection and Type Assertions` section from `.agents/skills/golang-performance/references/cpu.md` matched on the word "reflection" — and never named `httpmw/middleware.go:135` or `main.go:121`. `grep` returned both in 39ms. Same shape asking how the poller skips kagane: 145 nodes, top hits `poller_test.go` helpers and two nodes named `T`. `graphify explain "BrowserFetcher"` is sound (`browser.go L52`, 9 `EXTRACTED` edges), but that is what `lsp references` already answers, against live files instead of a snapshot. Staleness was never the problem — `graph.json` rebuilt 5s after `f568fb5`, so the git hooks worked. Retrieval quality was. ## What it cost - Two `PreToolUse` hooks injecting a "MANDATORY: run graphify query first" paragraph into context on **every** grep/find and every source-file read. - 685k input tokens across 5 build runs (`cost.json`). - 3.4MB of `graph.json` + `graph.html` tracked, across 11 commits of map-refresh churn. `AGENTS.md` is the stronger orientation artifact for a repo this size: it carries the CDP constraints, the UTC-clock finding, the per-site adapter list, and the security invariants — none of which an AST graph derives. Graphify earns its keep on repos too large to grep coherently and without curated docs; not this one. ## Changes - Delete the committed map (`graphify-out/`, -58k lines). - Drop the `## graphify` rules block from `AGENTS.md` (`CLAUDE.md` is a symlink, so both). - Drop the five `graphify-out/*` entries from `.gitignore`. - Empty the two `PreToolUse` hooks in `.claude/settings.json`. - Remove the stale `graphify query` instruction from `.claude/skills/implement-tickets/SKILL.md` — it pointed dispatched ticket-implementer agents at a binary that no longer exists. Uninstalled outside the tree (not in this diff): the `graphifyy` CLI, `~/.claude/skills/graphify/`, the global `~/.claude/CLAUDE.md` block, the `Bash(graphify query *)` permission in the git-ignored `.claude/settings.local.json`, and the `post-commit` / `post-checkout` git hooks. ## Verification `grep -ri graphify` over the worktree is clean; remaining hits are inside `.git/` (commit messages, two stale branch configs). No code touched — backend and userscript are untouched, so `go test ./...` is unaffected. Reviewed-on: #111 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
f568fb5e8c |
fix: an asleep browser Lane is not a stalled one on the admin page (#110)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.2 |
||
|
sulthan
|
20fff588cc |
fix: don't read Cloudflare's injected jsd script as a refusal (#109)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.1 |
||
|
sulthan
|
ba679223b2 |
Sightings: a Reader report defers a Poll of a solitary Series (#103) (#108)
Closes #103. A userscript PUT already carries the Latest Chapter the Reader's own browser read off the Series page. It may now stand in for a Poll, under one restriction and one ceiling: - **Solitary Series only** — a Series two Readers share is Polled on schedule however recently it was sighted, so one Reader's mistake can never reach another's list. - **One rest of standing**, and a **six-rest ceiling** (`sightingCeilingRests`, counted in the Site's own Rest): however many Sightings arrive, an unpolled Series is Polled. Both live in the due query's HAVING clause (`Store.DueForLatestCheck`) — the same place the schedule has always been decided, so no timer and no second code path can disagree with it. No new query per scheduler round. Judgement costs no extra request. `Poller.checkOne` already compares what the Site publishes against what is stored: a lower number contradicts the Sighting (Reader and both numbers logged), the same number confirms it, a higher number is the Site publishing and clears the attribution instead. Three contradictions stop that Reader deferring — their reports still write the Latest Chapter — and twenty consecutive confirmations forgive them, as does the owner's clear-marks control from #102. One client change was required: both userscripts skipped the PUT when the number had not moved, so the case the whole mechanism exists for — visiting a Series with nothing new — never reached the backend. `reportLatestChapter` sends it, skipping only the local write and the re-render. A numberless PUT (favourite toggle, progress from a chapter page) is no Sighting and defers nothing. Schema: migration `0011_series_sightings.sql` adds `series.latest_sighted_at` and `series.latest_raised_by`. Trust model, thresholds, and rejected alternatives with their citations: `docs/adr/0011-sighting-deferral-trust-model.md`. Reviewed on both axes (spec against #103, standards against the repo's rules); the blocker — attribution surviving a Poll that overtook the report — is fixed and has a test that fails without the fix. Verification: `go test ./...` green (needs Docker), `node --test userscript/test/*.test.js` 66 pass. Reviewed-on: #108 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.4.0 |
||
|
sulthan
|
1e6f1e985d |
Owner-only admin page: Reader roster plus Poll Lane status (#102) (#107)
Closes #102.
The only operational surface was /healthz and a fold-out roster inside the owner's own reading page. This adds /admin: an owner-only page carrying the Reader roster and one row per Poll Lane.
- **Poller seam.** `latest.Poller` records each Lane's last pass (`Site`, `Due`, `Checked`, `LastRun`, `Gap`, `Clamped`, `Browser`) and answers `LaneStatus()`; the page reads that snapshot, never a table. A pass that returns before computing its figures (refusal backoff, sidecar down) carries the previous pass's figures forward rather than recording zeroes, and a Lane that has never reached a pace renders no gap at all. Refusal and sidecar reachability are derived at snapshot time.
- **Owner gate at registration.** Every route reaching past the acting Reader lives in `adminRoutes()` and is wrapped in `requireOwner` when it is registered, so a missing gate is visible in the route list rather than hidden in a handler. `web.AdminPatterns()` is what the gate test walks, so a new route cannot be added without being tested. A non-owner gets 404, never 403.
- **Nil poller is a first-class state.** `main.newRouter` takes the reporter as an interface and converts a nil `*Poller` to a nil interface; no poller and no completed pass both render "No data yet" with the reason spelled out, rather than confident zeroes.
- **Roster moved** off the reading page onto /admin, with the Sighting counters and a confirm-gated `Clear marks` control. #103 fills those counters, so on delivery they read zero for everyone - deliberate ordering.
- **One accent, `--patina`** (verdigris, both colour branches): the far side of the wheel from ember's crimson and clear of the archive blue. Ember still means new chapter only; revocation still wears --danger.
Verification: `go vet ./...` and `go test ./...` green (Docker-backed); admin page screenshotted at 1100px and 390px in both colour schemes. Reviewed on both axes (spec, standards); findings on the accent hue, zero-figure honesty and three tests that could not fail are fixed in
|
||
|
sulthan
|
3303a55b20 |
feat: one Poll Lane per Site, replacing the shared pace (#100) (#106)
Closes #100. Each Site runs its own Poll Lane: an independent goroutine with its own rest and pace from the registry (`backend/internal/latest/sites.go`), replacing the shared cooldown/interval/stagger/batch configuration. Rest (1h, all six Sites including the browser trio) is enforced by the due query's WHERE clause; the Lane sleeps its effective gap between fetches — the registry 10s, or rest/eligible when a Site holds enough Series, floored at 1s with a Site-naming warning when the floor engages. Lane-local failure handling: - Two challenge-held results stop that Site's Lane for 15m; the probes keep their stamp, untried Series stay due. - A lost browser sets a shared Poller flag: the other browser Lanes skip their passes for the same 15m (no stamp-per-pass-per-Lane on a dead tab), then decay and probe again. - Browser wake gate preserved (5 due, or one waiting 15m, ADR-0005); one tab shared by the three browser Sites; "browser lane behind by X" logged every pass. - Cover work (healing a stored source URL and filling a blank from the series page) runs in the background so a slow CDN cannot consume a Lane's gap. Removed: `LATEST_CHAPTER_POLL_{COOLDOWN,BROWSER_COOLDOWN,INTERVAL,BATCH,STAGGER}` and the 6h browser rest. Only `LATEST_CHAPTER_POLL_ENABLED` remains; DEPLOY.md documents the exact `.env` edit. ADR-0010 records the decisions. Reviewed-on: #106 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
ddbd57070d |
Poll comix.to through the browser sidecar (#98) (#105)
Closes #98.
comix.to began answering plain-TLS fetches with a Cloudflare JavaScript
challenge on 2026-08-12, so every poll got a 403 interstitial. Its cover host
`static.comix.to` is gated the same way. comix therefore joins kagane and
novelfull as a browser-backed Site.
## What changed
- **Registry** (`internal/latest/sites.go`): comix gains a `Browser` entry —
`comixRead`, `Done: body != "" && !isInterstitial(body)`, `Fallback: false`.
Skip-when-no-browser falls out of the existing routing; no site-string compare
was added anywhere.
- **Read shape** (`internal/latest/browser.go`): an in-tab `fetch()` of the
Series URL, not a DOM render. comix is an SPA — rendering it costs ~65
requests for the same server-rendered HTML one fetch returns (24.5 KB,
~480 ms measured). `comixSeriesPageURL` pins scheme + host + `/title/<slug>`
and rebuilds the address, so a client-supplied `series_url` cannot aim the
browser anywhere else.
- **Cover bytes**: `comixImageURLRe` pins `https://static.comix.to/<path>.<ext>`;
`BrowserFetcher.Image` now gates on `browserOnlyCoverURL` rather than a
kagane-only regex, so both Sites' image URLs route through the one path.
Bytes come from direct navigation, not a page-context fetch — comix's Series
page sets `cross-origin-embedder-policy: require-corp`, which fails one.
- **Parsers and stored Series identity: untouched.** The in-tab body is the same
server-rendered HTML the existing fixtures were cut from.
## Verification
- `go test ./...` green (needs Docker).
- New seam tests: comix routes to the browser when one is configured, and is
not fetched at all when none is (`TestComixUsesBrowserFetcher`,
`TestComixSkippedWhenNoBrowserFetcher`); URL-pin and cover-gate table tests.
- Live proof against the real browser unit, `TestSmokeComix` (env-gated):
page 24793 bytes in one in-tab fetch, chapter 53, cover accepted by the pin,
26862 bytes of `image/jpg` retrieved.
- Two-axis review run; findings were stale comments on `BrowserFetcher`, `Get`
and the `Fallback` field, fixed in
|
||
|
sulthan
|
3f53c79cf4 |
docs: add Poll Lane and Sighting to the shared vocabulary (#104)
Two new glossary terms in `CONTEXT.md`, settled in a design session, plus the graphify refresh. - **Poll Lane** — one Site's own stream of Polls, carrying the pace at which that Site is willing to be asked. No Lane can slow, block or borrow from another's; a Reader never has one. - **Sighting** — what a Reader's browser happened to see of a Series's Latest Chapter. Reports the same fact as a Poll, carries none of its authority. - **Latest Chapter** amended: it no longer claims to be discovered without the reader present, since a Sighting establishes it between Polls. No code. The work these terms describe is specified in #98 (comix), #101 (Poll Lanes), #102 (admin page) and #103 (Sightings). Reviewed-on: #104 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
17ee0bd3f8 |
docs: correct the bot-score claims behind the browser poller (#99)
Docs only. No code changes - `git diff origin/main --stat` touches five Markdown files and adds one research note. ## What was wrong Several docs explained Cloudflare challenges as a "bot score" that our request rate could worsen. That mechanism does not exist on these sites. Researched live on 2026-08-12 against Cloudflare's own documentation and blog plus RFC 9309 - 22 primary pages, every claim carrying a source URL and read date, seven areas explicitly marked `Not publicly documented`. The note is `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. - The 1-99 bot score is **Enterprise Bot Management only**. A free-plan zone has no score at all; it gets Bot Fight Mode, which matches *signatures* (headless browsers, cloud-hosting IPs). - **No per-IP request rate is documented as an input to challenge issuance.** Volume is policed by Rate Limiting Rules, a separate opt-in product: one rule, IP-only counting, 10-second windows on Free. Published DDoS thresholds are ~1,000 errors/sec. - **`cf_clearance` defaults to 30 minutes**, so every cadence at or above 1 hour re-solves the challenge anyway. Cadence changes how many ~4s solves happen per day and nothing else. - The documented risk is **fingerprint quality**, which this repo already solved (real Chrome, stock UA, non-UTC clock). ## What changed | File | Correction | |---|---| | `AGENTS.md` | The block is per-zone configuration plus request fingerprint, not IP reputation. comix.to turning its gate on 2026-08-12 is the worked example. Residential egress avoids the cloud-hosting-IP *signature* rather than earning a better score. The UTC measurement stands; its mechanism is now marked undocumented. | | `backend/AGENTS.md` | Says why `_BROWSER_COOLDOWN` is longer: cost, not safety. | | `docs/adr/0003` | Dated correction - the sites do not "bot-score" the VPS IP. Decision stands on its sweep-depth argument. | | `docs/adr/0006` | Dated correction - no score to be better at. Decision stands on VPS memory. | | `DEPLOY.md` | A red kagane smoke run means the Site's settings or this Chrome's fingerprint moved, not "Cloudflare's scoring". | ADRs got dated `Corrected 2026-08-12:` paragraphs rather than silent rewrites - the record of what was decided stays intact, only the wrong mechanism is retracted. ## Deliberately not in this PR - **The 6h browser cooldown is unchanged.** I had lowered it to 1h and reverted that; cadence is a behaviour change and belongs with the comix work in #98, not in a docs correction. - **Two code comments still carry the myth**: `backend/main.go:83-84` ("a hammer against sites that are already bot-scoring us"). Left alone to keep this diff docs-only. Related: #98. Reviewed-on: #99 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
c62c3bb07b |
chore: drop asuracomic.net from the userscript, CORS allowlist and docs (#97)
Closes #96. ## What Removes every reference that still invites a Reader onto `asuracomic.net`. The domain's deep links 301 to the `asurascans.com` **root**, discarding the path (re-checked 2026-07-25), so a page on it never yields a series document client-side and a stored address on it never yields a series page server-side. #95 already pinned each Site to one hostname, so the backend rejects such an address cleanly; this is the cleanup around that. | File | Change | |---|---| | `userscript/manga-bookmark.user.js` | drops the `@match`, narrows the asura adapter to `/(^\|\.)asurascans\.com$/` | | `userscript/test/logic.test.js` | new test pinning the narrowed host match | | `.env.example`, `docker-compose.yml` | origin dropped from the `ALLOWED_ORIGINS` default | | `DEPLOY.md` | same, and the sample list gains the two novel origins it was missing | | `backend/api_test.go` | CORS fixtures and round-trip seed move to `asurascans.com` | | `README.md`, `AGENTS.md` | notes say the host is dropped, not "stays matched" | ## Behaviour - A Reader landing on `asuracomic.net` gets no userscript UI. Previously the script loaded and could do nothing useful — the redirect had already discarded the path. - A request whose `Origin` is `https://asuracomic.net` is no longer reflected by a deployment using the shipped defaults. - No backend logic changed: the CORS rule, the address gate and the poller are untouched. `AllowedOrigins` is data, not code. ## Security invariant preserved CORS still reflects `Origin` only when it appears in `ALLOWED_ORIGINS`, with `GET,PUT,DELETE,OPTIONS` and a `204` preflight — `TestCORSPreflight` and `TestCORSDisallowedOrigin` still pin both halves, now against a live origin. This change only removes a value from the allowlist, which is a narrowing. ## Verification - `go test ./...` — full backend suite green (real Postgres per package). - `node --test test/*.test.js` — 66/66 green, up one from the new match test. ## Deploy note (does not happen on merge) The live allowlist comes from the VPS `.env`, not from these defaults, so the origin must be dropped there in the same deploy. The one-off row repair for any stored `asuracomic.net` address is in #96. Reviewed-on: #97 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.3.0 |
||
|
sulthan
|
21615be2bd |
feat: one registry entry per Site, one shared Series-page read (#95)
Closes #94. ## What Two phases per the spec, in three feature commits plus two review-fix commits: **Phase one — one registry entry per Site** (`2d134fb`) The six per-site comparison points that used to live across three files collapse into one `sites` map in `backend/internal/latest/sites.go`: Latest Chapter parse, Cover parse, browser-backed list, fetcher route, host pins, and the browser payload read all become lookups into it. `browserBackedSites()` is derived from the registry (sorted, deterministic); `fetcherFor` and `fetchableSeriesURL` keep their signatures and become lookups; `BrowserFetcher.Get` dispatches through the entries' `Read`/`Done` while the tab lifecycle stays in `BrowserFetcher.run`. **Phase two — one shared Series-page read** (`f215130`) `readSeriesPage` (new `read.go`) performs the read the Poll and the Acquisition have in common: gate, route, fetch, parse Latest Chapter, parse Cover address. It returns facts only — polling and persistence policies (stamp order, cooldowns, cover policy) stay with the callers; `acquire.go` gained the comment naming the deliberate post-fetch stamp order. The poll's legacy cover heal and the no-chapter byte-count diagnostic were restored after review (`d998f87`) so the claims "the Poll keeps its own Cover policy" and "pinning is the only behavioural change" both hold. ## Behaviour - All six Sites now pin their host exactly; asura/demonic/comix previously accepted any https host. For asura this is a strict improvement: its dead old domain redirects deep links to the site root and would parse the wrong document. - Everything else is unchanged: existing parse tables, the challenge-body table and the gate table pass unmodified except the one deliberate exception — the gate table gains the three new pin cases. ## Security invariants preserved - The address gate is recognisably the same rule, now a single registry lookup: `https` + exact hostname match, all callers route through it. No fetch path was widened; asura/demonic/comix were narrowed. - The second host pin inside each browser entry's Read is retained deliberately (browser = strong SSRF primitive, `series_url` is client-supplied) and is not deduplicated against the shared gate. - Review hardening: `fetcherFor` now fails closed for unknown site strings (previously fell through to the TLS fetcher on an unreachable path), and the browser dispatch iterates a sorted list so outcomes cannot depend on map order. - The security review's log-injection finding was checked against Go's `url.Parse` and does not hold: control characters are rejected anywhere in a URL, so a client-supplied value in a log line cannot carry a newline. ## Review Reviewed on three axes (spec, standards, security) by read-only subagents over `672c16f..f1b26f4`. No blocking findings; all minor/nit findings addressed in `d998f87` and `700de20`. Verified end to end with `go test ./...` (Docker Postgres per test package) on every commit. ## Out of scope (tracked separately) - Dropping asuracomic.net (CORS allowlist, userscript match, API fixtures, live env) — separate issue, per spec. Reviewed-on: #95 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
672c16ffbf |
Remove the client latest-chapter scan for lightnovelworld (#91) (#93)
Follows the spec published on #91: remove the client-side latest-chapter scan for lightnovelworld rather than porting #87's truncation into a second codebase. - lightnovelworld.latestChapterFromAnchors deleted, not stubbed: absence is what the background-fetch guard keys off. - computeLatestChapter tolerates an adapter with no scanner (yields null) and is exported as the test seam. - backgroundRefreshLatest skips a scanner-less Site before the due filter: no Series page fetched, no freshness timestamp recorded, no batch slot consumed. The on-page path (maybeCaptureLatestOnSeriesPage) routes through the same null-tolerant computation. - novelfull's scanner, the shared max-chapter helper and all four manga Sites untouched. - userscript/AGENTS.md records the Poll-only contract for this Site and why. All seven acceptance criteria from the spec met. node --check clean; novel suite 30/30 (the regression pin fails if a lnw scan is reintroduced, scoped or not); manga suite 35/35, manga userscript byte-for-byte unchanged. Two-axis code review: no hard standard violations, spec-clean; one follow-up commit matching the sibling adapter guard from the manga script. Reviewed-on: #93 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e7e22a12a5 |
lightnovelworld Series identity is read from the chapter page (#80) (#92)
Implements spec #80 / ADR-0008 — Gitea issues #86, #87, #88, #89, #90, all closed. A Reader bookmarks a novel on lightnovelworld and it never shows a New Chapter, because the Series identity was derived from the chapter address instead of read from the page. One Series can publish under several Chapter Slugs, so the derived key points at a slug that 404s. - **#89** — the userscript's lnw adapter stops deriving `seriesUrl`/`seriesId` from the path. It reads the page's own pointer (`a[aria-label='All Chapter']`), falling back to the microdata breadcrumb's second crumb, and carries `chapterSlug` on the page object, stored nowhere. - **#87** — the Poll's lnw chapter scan is unscoped (no stored-slug pattern can cover a Series' whole list) and truncated at the `wpd-threads` comment thread, the one region a visitor can write to. Marker absent means skip and log with the body length, never scan whole. Corrects the `maxBodyBytes` headroom comment to the measured 3.5x. - **#86** — the scan fixture is now text trimmed from a real, wholly-fetched Series page instead of a hand-written cross-series anchor that no live page carries. - **#90** — stale stored rows repair themselves on the next chapter visit: a pure transform over cache, queue and last-checked map, silent to the Reader, with progress, favourite and lifecycle bucket preserved when two rows merge. - **#88** — an env-gated live canary (`SMOKE_LNW_SERIES_URL`) proving the marker still occurs exactly once and still follows the last chapter anchor, asserted against the production symbols themselves. Verified on the merged branch: `go test ./...` green, `gofmt -l internal/latest/` silent, both userscripts `node --check` clean, 35/35 + 29/29 logic tests. Live canary green (marker once at byte 612,182 of 651,795). #90 verified on device with Playwright. Open follow-up: **#91** — the userscript's client-side latest-chapter scan is still scoped to the derived slug. Reviewed-on: #92 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
90d8ab72ad |
chore: refresh graphify map and tidy AGENTS.md (#84)
graphify update regeneration: semantic hashes now populated in manifest.json, graph rebuilt (1634 nodes, 3179 edges). Track the map (5 curated files + .graphify_root) so a fresh checkout starts with it; cost.json, cache/, dated snapshots and .rebuild.lock stay ignored. AGENTS.md: drop stale Relevant skills and Notes sections; graphify rule now says to always query the graph first. Reviewed-on: #84 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
c400c91a80 |
Implement a batch of tickets through per-ticket subagents (#82)
## What this adds Two files that turn the one-ticket-at-a-time `/implement` loop into an orchestrated batch. **`.claude/skills/implement-tickets/SKILL.md`** — user-invoked (`disable-model-invocation: true`, so it costs no context until typed). The agent that runs it is an orchestrator, not an implementer: 1. Collect the tickets over `tea`, reading each `Blocked by` line. 2. Plan waves from the blocking edges, three tickets wide, and fix every cross-ticket contract (shared signature, JSON shape, column, token) before anything is dispatched. 3. Present the plan and stop for approval. 4. Per ticket: `git worktree add ../ticket-<n>`, copy the gitignored `.env`, claim the issue, write a brief to `.scratch/`, then dispatch the whole wave as one `task` batch. 5. Land each result — merge `--no-ff`, comment the report, close, remove the worktree. Textual conflicts are the orchestrator's; a semantic clash goes back to whichever ticket owns the contract. 6. Full suite once on the merged base. **`.omp/agents/ticket-implementer.md`** — the worker. Brief-driven, worktree-bound, and gated on review before it reports: it runs the `code-review` skill over its own diff with `cr-spec` and `cr-standards` on the two axes, fixes Critical and Important findings in at most two rounds, and returns a short status contract (`DONE` / `DONE_WITH_CONCERNS` / `BLOCKED` / `NEEDS_CONTEXT` / `REVIEW_BLOCKED`). The brief template makes the subagent read `tea issue <n> --comments` for its ticket and for the issue that ticket refers to — the comments carry decisions the body never got updated with — and names the `tdd` skill at each seam where a test comes first. Briefs are written in the ubiquitous language of `CONTEXT.md`; a brief that says "scrape" where the domain says Poll hands the subagent the wrong model of the system. ## Verification Dispatched a real `ticket-implementer` as a probe. The agent resolved from `.omp/agents`, and it spawned `cr-spec`, which replied. That was the one thing that could have silently killed the design: `task.maxRecursionDepth` defaults to 2, and the chain is session to orchestrator to implementer to reviewer. It clears. If that ever changes, the implementer returns `REVIEW_BLOCKED` and the orchestrator runs the review itself. Confirmed against the omp binary that `autoloadSkills: code-review, tdd` is split by `parseArrayOrCSV`, not swallowed as one unknown name. ## Notes - Agents are discovered from `.omp/agents`, never `.claude/agents` — the latter is deliberately skipped by omp because its frontmatter is a different contract. - No product code changes. `.gitignore` gains `.scratch/`, where briefs and reports live. - Not included: retry after a failed dispatch, a state file for resuming a crashed wave, a cheap model tier for mechanical tickets. Add them when a real batch needs them. Reviewed-on: #82 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
f1eb7d514c |
Record the lightnovelworld series-identity decision (#77) (#81)
Docs only. No code, no tests, nothing to run. Implementation is specified in #80. Outcome of a grilling session on 2026-08-11 against #77, backed by live measurement of lightnovelworld over 2026-08-10/11. ## What changed **`docs/adr/0008-series-identity-is-discovered-not-derived.md`** (new) A Series identity is discovered from the Site's own links, never derived from an address. On lightnovelworld the userscript reads the chapter page's `All Chapter` anchor instead of building a `/novel/<slug>/` address by string manipulation. A Chapter Slug is not an identity and is not stored. The backend's chapter scan drops its per-Series scoping and runs against the body truncated before the visitor comment thread. Evidence in the ADR: 3 of 41 sampled novels serve chapters under a slug that differs from their series slug, divergence runs in both directions, one novel serves chapters under two slugs, and neither slug is computable from the other. The pointer was checked on 8 chapter pages and agreed every time. Three narrower selectors are recorded as rejected, each with the measurement that killed it. Three rejected options are recorded with reasons: correcting the stored address only, which keeps an identity the Site does not guarantee; scoping the scan to a container, which the probe refuted; and a SQL migration, which is impossible because the database holds no source for the correct slug. **`CONTEXT.md`** - **Series** - identity is the canonical slug the Site publishes, never the title and never a Chapter Slug. - **Chapter Slug** - new term. A slug a Site builds its chapter addresses from. Not an identity: one Series may have several, and none is computable from another. - **Latest Chapter** - now the highest-numbered chapter, explicitly not a date and not the Site's own newest-chapter banner. Settles #79. **`docs/research/lightnovelworld-chapter-vs-series-slug.md`** (new, committed with its corrections) The 41-novel survey behind the ADR. Two claims are struck through and corrected in place, with the date and sample size of the probe that refuted each: the `ul.clstyle` container it named is the hidden, empty "Latest Reading" template rather than the chapter list, and its caveat about the comment region understated the risk, because that region is writable by any visitor while the scan takes an unbounded maximum into a Series row shared by every Reader (ADR-0003). ## Review notes Nothing here constrains code that exists today - the ADR describes work not yet written. The part worth disagreeing with, if any of it is wrong, is the fail-closed rule: a missing truncation marker means skip the Series and log, never scan the whole page. Related: #77 (the defect), #80 (the spec), #79 (the numbering anomaly, closed by decision), #71 (the same size cap seen from the cover side). Reviewed-on: #81 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
1ee5eb67ea |
Clear the stale Chrome singleton lock at browser boot (#76)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.2.1 |
||
|
sulthan
|
b22ae82897 |
Restore the novel script's lost module-scope constants (#74) (#75)
> *This was generated by AI during triage.* Fixes #74. `novel-bookmark.user.js` was split out of `manga-bookmark.user.js` and lost four module-scope constants. Every use of them is behind a `try/catch` or a fire-and-forget promise, so the `ReferenceError`s were swallowed rather than reported. | constant | used at | effect while missing | | --- | --- | --- | | `LATEST_CHECK_THROTTLE_MS` | `:722` | `backgroundRefreshLatest()` throws before computing `due` — no background latest-check ever runs for novels (the symptom in #74) | | `LATEST_CHECK_BATCH` | `:724` | same throw | | `CACHE_KEY` | `:215`, `:224` | `loadCache()` always returns `[]`, `saveCache()` silently no-ops — the local cache never persists | | `LASTCHECKED_KEY` | `:232`, `:241` | last-checked map never persists, so the throttle would not hold even once the first two are defined | #74 named only the two throttle constants. The two cache keys are the same lost lines with the same root cause, so they are restored here too — fixing only the pair the issue named would leave `backgroundRefreshLatest()` re-fetching every series on every navigation, because `saveLastChecked()` would still be a no-op. Values and comments copied verbatim from `manga-bookmark.user.js:37-43`; throttle 4h, batch 1. ## Verification - `node --check userscript/novel-bookmark.user.js` — clean. - `node --test userscript/test/logic.test.js userscript/test/novel-logic.test.js` — 47/47 pass. - New test `every SCREAMING_CASE constant the script uses is declared in it` scans both scripts (comments and string literals stripped first, so prose and SVG path data do not trip it). Confirmed it fails — 1 failing test — when `LATEST_CHECK_BATCH` is deleted again, and passes when restored. A behavioural test cannot reach this: the storage helpers and the background refresh are exactly the layers the harness does not cover (see the `testing-the-userscript` skill), and the errors are swallowed anyway. A static guard is the only instrument that sees this bug class. On-device confirmation that the ember now lights for novels is still outstanding — that needs Violentmonkey against a live novelfull/lightnovelworld page. Reviewed-on: #75 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.2.0 |
||
|
sulthan
|
7c7d597019 |
Delete the kagane-specific cover path (#63) (#73)
Closes #63 Deletes the second way to reach a Cover. Since #62, every Site's cover bytes land in the content-addressed store at creation or on the poll, and the one public route serves them all — nothing needs the kagane proxy anymore. ## What went - **Template-level rewrite:** `Bookmark.CoverURL()` and both templates' use of it. Cards and chrome now render `.Cover` — the wire value — and nothing else. `Bookmark.CoverSource` was dead once `CoverURL` went, so it and its `bookmarkColumns` entry are gone too. - **Kagane-only cover route and its identifier validation:** `GET /img/kagane/{id}`, `web.CoverFetcher`, `coverIDRe`, and the whole `internal/web/cover.go`. - **The proxy's persistence:** `store.KaganeImageID`, `GetKaganeCover`, `PutKaganeCover`, `kaganeCoverSourceURL`, `kaganeCoverRe`. - **The kagane-shaped branch in the byte-fetch routing:** `fetchCoverBytes` no longer takes a `site` argument and no longer names a Site. The URL shape kagane's API publishes is claimed by the browser module itself — `kaganeImageURLRe` + `browserCoverURL` live in `latest/browser.go` with the rest of the per-Site knowledge — and `BrowserFetcher.Image` is now URL-driven (it validates the URL it will navigate to, same SSRF discipline as before). The no-plain-TLS-fallback rule for a claimed URL is preserved: a claimed address with no browser is an error, never a challenge-page fetch. ## What stayed (deliberately) - `BrowserFetcher.Image` and the browser-backed acquisition path: kagane genuinely serves cover bytes behind the challenge + `cross-origin-resource-policy: same-origin`, so the sidecar remains the only fetcher for them — it just routes by URL claim now instead of by Site name. - `fetcherFor`'s per-Site page routing (kagane/novelfull page fetches) — that is the page path, not a cover path. ## Acceptance criteria - [x] Template-level kagane cover rewrite gone - [x] Kagane-only cover route and its identifier validation gone - [x] Tests removed/rewritten against the general route, guarantees kept: unstored + traversal-shaped addresses serve nothing (`TestPublicCoverRejectsUnknownAddress`), non-image content types never echoed (`TestPublicCoverNeverEchoesNonImage` — new; the store-side gate was already pinned by `TestCoverStoreAcceptsAnySourceURL`). Store reopen-persistence and filesystem content-addressing tests rewritten against `PutCover`/`GetCover`, no guarantee lost. - [x] No Site name in a cover code path outside the acquisition module (`grep kagane backend`: store/web/templates/api are clean; remaining hits are `latest/browser.go` + `latest/sites.go`, tests, docs) - [x] Web UI and panel render Covers for all six Sites (templates render the wire address; panel renders `b.cover` — untouched, it never had a kagane path) - [x] `go test ./...` green ## Verification - `go vet ./...` clean - `go test ./...` — all packages pass (root 16.9s, latest 12.7s, store 12.7s, web 0.004s) - `CGO_ENABLED=0 go build` produces the static binary - Cover-path tests run verbosely: `TestPublicCoverServesStoredBytesUnauthenticated`, `TestPublicCoverRejectsUnknownAddress` (unknown/malformed/traversal/empty), `TestPublicCoverNeverEchoesNonImage`, `TestListRendersAcquiredCover`, `TestAcquireKaganeCoverThroughBrowser`, `TestRunOncePrefetchesKaganeCover`, `TestRunOnceRoutesNonKaganeCoverToPublicFetcher` all pass; the three `SMOKE_*` tests skip without the browser sidecar, as designed Live browser verification of the "web UI and panel render Covers for all six Sites" criterion is being run separately with Playwright against real Site pages and a locally mocked backend. Reviewed-on: #73 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
78234f3c19 |
Browser-backed Sites join the Cover pipeline (#62) (#72)
Fixes #62
Browser-backed Sites join the Cover pipeline: kagane and novelfull Series now get their Covers at creation, through the same acquisition path as every other Site, instead of waiting for a poll pass.
## What changed
`latest.Acquirer` (creation-time acquisition, fired by the first Bookmark of a Series) previously skipped kagane and novelfull entirely — their pages only yield a Cloudflare challenge to the TLS client, so the request was spent for nothing. It now routes them like the poller does, with the two Sites split exactly as the issue demands:
- **kagane** — page fetched through the browser sidecar, cover URL extracted from the API JSON, bytes fetched through the browser sidecar (the only path that clears the challenge) into the content-addressed store. With no `BROWSER_WS_URL` configured, acquisition is skipped entirely and nothing falls back to a plain fetch.
- **novelfull** — page fetched through the browser sidecar, cover URL extracted from the HTML, bytes fetched over plain TLS through the ordinary gated fetcher (its image paths answer 200 with `access-control-allow-origin: *`, measured 2026-08-09). With no browser configured, the page fetch falls back to the TLS client — novelfull's challenge is a live time-varying fact (AGENTS.md), so when the page body answers, the Cover still lands; when it is challenged, nothing happens.
The byte-routing rule (kagane → browser, every other Site → TLS) is now one shared function (`latest.fetchCoverBytes`) used by both the Poller and the Acquirer, so the two cannot drift apart.
## Acceptance criteria
- [x] kagane cover bytes are fetched through the browser sidecar and stored in the content-addressed store — `TestAcquireKaganeCoverThroughBrowser`
- [x] novelfull cover URLs are extracted from the browser-fetched HTML, and its bytes are fetched over plain TLS — `TestAcquireNovelfullCoverOverPlainTLS`
- [x] With no browser sidecar configured, kagane Covers are absent and nothing falls back to a plain fetch — `TestAcquireKaganeSkippedWithoutBrowser`
- [x] With no browser sidecar configured, novelfull Covers still work if its page body is available — `TestAcquireNovelfullCoverWithoutBrowser`
- [x] Manually verified on-device: a kagane Series shows its Cover in the panel, not a broken-image glyph — being run by a separate manual-verification agent against a mocked scenario (no prod data); not part of this PR
- [x] `go test ./...` is green, with live-network checks gated behind `SMOKE_BROWSER_WS_URL` like the existing kagane image smoke test — new `TestSmokeAcquireKaganeCover` proves the end-to-end acquire path against the real browser when the env var is set
## Verification
- `go test ./...` green across all packages
- New unit tests exercise every routing decision with fakes — no network in the default suite
- Smoke test gated behind `SMOKE_BROWSER_WS_URL`, skipped by default
## Post-review changes (
|
||
|
sulthan
|
b9220b3dfc |
The Poll fills blank Covers for every Site and both Libraries (#70)
Closes #61. ## Summary Permanently-blank Series (the half of #47 that creation-time acquisition cannot reach) heal on the next due poll cycle. The cover path is no longer kagane-only: every Site and both Libraries fill a blank Cover from the series page the chapter poll already fetched, and never replace a Cover that already exists. ## What changed ### `backend/internal/latest/poller.go` - **`fillBlankCover`** — when `Cover` and `CoverAddress` are both blank, extract a source URL via `coverFrom` from the series-page body and store bytes through `SetSeriesCover`. Skips any Series that already has a source URL (owned by prefetch) or a stored address (never overwrite). - **`prefetchCover`** — source-URL healing path, now site-uniform. Kagane no longer special-cases into `PutKaganeCover` alone; every Site lands on `SetSeriesCover`, so the wire Cover becomes a content-addressed public URL. Reuses already-stored bytes when present. - **`storeCover` / `fetchCoverBytes`** — shared fetch+persist. Only kagane routes image bytes through the browser fetcher; every other Site uses plain TLS `CoverBytesFetch`. Failures log with the Series key and never return to the chapter path. - **`checkOne`** — after a successful series-page fetch, calls `fillBlankCover` once regardless of whether chapter extraction succeeded (cover fill is independent of the chapter signal). ### `backend/internal/latest/poller_test.go` Extended the existing poller harness (real store, fake fetchers) rather than a new one: - `TestRunOnceFillsBlankCoverFromSeriesPage` — asura manga, lightnovelworld novel, kagane manga; asserts wire Cover + correct fetcher routing. - `TestRunOnceDoesNotReplaceExistingCover` — second poll does not refetch. - `TestRunOnceRetriesFailedBlankCoverOnNextPoll` — failed fill stays blank, next due cycle retries (no separate queue). - `TestRunOnceBlankCoverFailureDoesNotBlockChapter` — chapter still lands; failure log carries the Series key. - Kagane prefetch test now also asserts the content-addressed wire Cover. ## Acceptance criteria (#61) | Criterion | Status | |---|---| | Cover prefetch runs for every Site | done | | Cover prefetch runs for both Libraries | done | | Poll fills a blank Cover | done | | Poll never replaces an existing Cover | done | | Failed cover fetch does not fail/block chapter poll | done | | Failed cover fetch retried next poll, no separate queue | done | | Failures logged with the Series | done | | Existing poller tests extended | done | | `go test ./...` green | done | | Manually verified: blank Series gets Cover after a poll cycle | **left for you** | ## Out of scope / not closed - Does **not** close #47 or #55 (per ticket). - No migration/backfill script — the Poll walks every Series already. - No admin refetch (#54). ## Review notes addressed - Removed the kagane-only `PutKaganeCover` branch from prefetch so source-URL healing also sets `CoverAddress` (wire Cover). - Guard so `fillBlankCover` does not double-fetch after `prefetchCover` healed the same snapshot. - Single `fillBlankCover` call site after the series-page fetch. ## Test plan - [x] `go test ./...` (backend; needs Docker/Postgres via `pgtest`) - [ ] After deploy: pick a Series that was blank, wait one poll cycle, confirm Cover in web UI and userscript panel Reviewed-on: #70 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e2c054e7ce |
Covers render in the userscript panel, from a public route (#60) (#69)
Closes #60. Spec: #55. Originating bug: #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Neither #47 nor #55 is closed from here. ## What this branch does The panel now renders Covers from the deployment's own origin, and both userscripts stop having an opinion about where a Cover lives. **The public route was already in place.** `GET /covers/{address}` landed with #59 (`92eba07`) and is registered on the bare mux, outside `httpmw.Auth` and outside the web UI's Discord session — `backend/main.go:210-214`, handler `backend/internal/api/handlers.go:142-158`. It reads no cookie and no header, answers `404` for an address that was never stored (and for a row whose file has gone missing — recorded-but-gone is not-found, never a fabricated body), refuses anything that is not `^[0-9a-f]{64}$` *before* the value becomes a path, and sets `Cache-Control: public, max-age=604800, immutable`. Those four properties are asserted by `backend/cover_test.go:231-278`. This branch re-verified them rather than re-implementing them; the only backend line it touches is a comment. **Both userscripts lose cover scraping entirely.** Every adapter's `cover:` field is gone, along with the two helpers that fed them: the manga script's `coverFromPage()` (the `img[alt]` DOM scan comix needed, because comix publishes no `og:image`) and the novel script's `metaName()` plus the now-callerless module-level `meta()`. Nothing under `userscript/` reads `og:image`, `meta[name=image]`, or `img[alt]` any more. **Nothing sends a cover either.** `delete body.cover` sits in `apiPut` — `manga-bookmark.user.js:486`, `novel-bookmark.user.js:275` — which is the single chokepoint every write passes through (`pushBookmark`, the retry-queue flush, `toggleFavorite`, `toggleArchive`). It operates on the `Object.assign` copy, so the in-memory row keeps the cover it renders with. This matters beyond tidiness: a Reader upgrading from an older copy has `localStorage` rows carrying third-party scraped URLs, and without the strip those would ride back up on the next write. The handler discards the field regardless (`handlers.go:53-59`) — it is permanently inert, not pending removal. **Failed loads get the designed empty state, not the broken-image glyph.** `onerror: (e) => e.target.replaceWith(el("div", { class: "cover ph" }))` on the cover `<img>` in both card renderers (`manga:1380-1390`, `novel:1134-1144`). The replacement is byte-identical to the existing no-cover branch on the very next line, so it picks up the `.cover.ph` styling already in the panel CSS — no new tokens, no new rule. `el()` routes any `on*` prop through `addEventListener`, so this is a listener, not an inline attribute string, and the swap is a `createElement` + DOM call with no markup parsing anywhere near it. This is the half of #47 that was visible on kagane. **The deleted scraping's tests went with it**: the two comix cover cases, the `pageImages` and `namedMetas` fixtures, the `img[alt]` and `meta[name=...]` stub branches, the now-dead `querySelectorAll` stub member, and every stale `og:image` fixture and `p.cover` assertion across both suites. The export lists needed no change and that was checked, not assumed — `coverFromPage` and `metaName` were module-private on `origin/main` and no cover symbol ever appeared in `module.exports`. Docs that described the deleted behaviour were corrected in the same breath, because leaving them would instruct the next agent to put the scraping back: `userscript/AGENTS.md` (adapter contract + the per-site notes for comix, kagane and novelfull), the README's adapter reference, and the userscript testing skill's stub table. ## Verification - `go test -count=1 ./...` — green across all nine packages (`backend` 29.8s, `latest`, `store`, `session`, `token`, `userscript`, `web`). - `node --check` clean on both userscripts; `node --test` on both logic suites — 46 tests, 46 pass. - `gofmt -l` clean; `go build ./...` clean. - The `onerror` swap is DOM behaviour and deliberately has no coverage in the Node harness — that harness stubs a browser precisely so it never needs a DOM, and #60 says not to invent coverage for it. It was instead exercised for real: the `el()` helper and the exact render expression were loaded into a headless Chromium with a deliberately unloadable `src`, and the resulting DOM was `<div class="cover ph"></div>`. Ad hoc, not committed. - **Not done, needs you:** the on-device criterion — a comix Series bookmarked mid-chapter showing its Cover in the panel. That needs a real install against the deployment and is the one box left unticked on #60. ## Reviewed Both `/code-review` axes ran against `cc0fa92`. Spec found no missed requirement and no scope creep; standards found the diff clean on the four areas it scrutinised (the `delete body.cover` placement, the `onerror` handler's DOM safety, comment quality, dead-code removal). Their combined findings — the dead `querySelectorAll` stub, the stale README and skill text, and the handler comment whose premise this change invalidates — are fixed in `8b58019`. ## Out of scope, deliberately The kagane-specific cover proxy still exists and still carries its session gate (#63 deletes it). The poll's blank-Cover fill (#61) and browser-backed Sites joining the pipeline (#62) are untouched. Reviewed-on: #69 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
92eba07da7 |
A newly bookmarked Series acquires its Cover at creation (#59) (#68)
Closes #59. Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55. ## What changed A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s. ### Store - Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on. - `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there. - `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address. - The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path. - `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load. ### Acquisition - `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position). - Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them. - Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid. - Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later. - Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62. ### Wire and route - `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk. - `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep. - `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading. ### Config `PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`. ## Acceptance criteria All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence. ## Verification - `go test ./...` green (Docker-backed Postgres suite). - Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type. - Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red. ## Reviewed Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail. ## Known sequencing A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly. Reviewed-on: #68 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
b6b88bde8a |
feat(latest): extract per-site covers (#58) (#67)
Closes #58 ## Summary - Add pure per-Site cover extraction beside latest-chapter parsing for all six Sites. - Read Asura, Demonic, LightNovelWorld, and NovelFull metadata; read the Comix target detail state; read Kagane's browser-fetched `series_covers[].image_id` JSON. - Preserve published cover URLs, percent-encode Demonic raw spaces, select Comix's smaller published `medium`, and avoid thumbnail rendition URL synthesis. - Add live-source fixtures plus no-cover and Cloudflare challenge coverage for every Site. ## Correctness - Scope Comix extraction to the requested series detail key, avoiding recommended posters. - Parse Kagane's current live API shape and emit its canonical compressed image route from the published image ID; unrelated JSON fields are ignored. - Validate Kagane image IDs against the existing UUID-shaped route constraint. - Keep extraction pure; storage, polling, and wire integration remain outside issue #58. ## Acceptance criteria - [x] Cover extraction exists for all six Sites in the existing latest parser module. - [x] Each Site has a live-source fixture with source URL and date. - [x] Comix reads the state blob, not metadata. - [x] Demonic raw spaces are percent-encoded. - [x] Comix returns the smaller published rendition. - [x] No-cover pages return empty. - [x] Cloudflare challenge pages return empty. - [x] No thumbnail URL is synthesized by editing a published URL. - [x] `go test ./...` passes. ## Verification - `go test ./...` - `go vet ./...` - `git diff --check` Parent issues #47 and #55 remain open as requested. Reviewed-on: #67 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
9d6d3bde72 |
Add gated cover byte fetcher (#66)
## Summary Adds a plain-TLS cover byte fetcher with a destination-class SSRF gate and wires public cover sources through the content-addressed filesystem store. ## Changes - Resolve hostnames before connecting; refuse non-HTTPS, loopback, private, link-local, unique-local, CGNAT, credentials, and mixed public/private DNS answers. - Re-check every redirect and resolve/classify again at dial time to close DNS rebinding. - Reuse `maxBodyBytes`; reject oversized responses and non-image content types before persistence. - Add generic `Store.GetCover`/`PutCover` source-URL storage while preserving the browser-backed kagane path. - Keep cover prefetch failures isolated from chapter polling. - Add observable tests for TLS, no-connection refusals, all refused address classes, redirect blocking, streaming body caps, non-image rejection, content-addressed persistence, DNS rebinding, and poller routing. ## Verification - `go test -count=1 ./...` - `go vet ./...` Both pass. No test touches the live network. Closes #57 Reviewed-on: #66 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e8d1cba6c5 |
Move cover bytes to content-addressed filesystem storage (#65)
Refs #56 ## Summary Moves Kagane cover bytes out of Postgres bytea storage into an immutable, content-addressed filesystem store. Reader-visible behavior remains unchanged: the existing session-gated route serves stored bytes, missing bytes use the existing browser fetch path, and no browser still returns a missing cover. ## Changes - Added migration 0008, which drops the legacy `covers` table and recreates it with only `address`, `path`, and `content_type`. Existing byte rows are intentionally dropped. - Added SHA-256 source-URL addressing with two-level sharding (`ab/cd/<sha256>`). Writes use a temp file plus atomic link; reads validate the stored relative path before opening it. - Made `COVER_DIR` required in runtime config and Compose. Compose passes it as a Docker build argument and volume target, so custom durable paths keep image ownership, runtime config, and the named `cover-data` volume aligned. - Updated every `store.Open` caller and documented configuration, deployment, backup, and troubleshooting behavior. - Added filesystem, restart, migration-drop, no-browser, and content-addressing coverage. ## Verification - `go test ./...` - `CGO_ENABLED=0 go build ./...` - `docker build --build-arg COVER_DIR=/data/covers -t manga-bookmark-cover-check-custom ./backend` - `docker compose config --format json` confirms custom `COVER_DIR` is the volume target - `git diff --check origin/main` - LSP diagnostics clean for touched Go files Parents #47 and #55 remain open as required by #56. Reviewed-on: #65 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
30c57bd39c |
Define Cover and record hosting its bytes (#47) (#64)
Defines **Cover** in the glossary and records ADR-0007, the decision behind #47's fix. ## Why these two files, and why now `CONTEXT.md` named Cover inside the **Series** entry — "facts true regardless of who is reading — title, cover, Latest Chapter" — but never said *what* one is. That gap is the bug. Nothing in the model distinguished "an address on a Site" from "an image a Reader's browser can display", so both clients were left to work it out independently, and one of them got it wrong. kagane serves covers with `cross-origin-resource-policy: same-origin`, the web UI rewrote them to a proxy in its templates, the JSON API did not, and the panel rendered a broken-image glyph. The new entry closes the ambiguity: *an address no client can load is not a Cover, it is a missing one.* ADR-0007 records what follows from that — the backend fetches, stores and serves every Site's cover bytes — plus the alternatives that were rejected and, more importantly, the two places this deliberately departs from existing precedent: - **Destination-class control instead of a host allowlist.** `fetchableSeriesURL` sets the allowlist precedent for `series_url`, and covers do not follow it. Cover hosts are CDNs that move independently of their Site — demonicscans serves its covers from `readermc.org` — so an allowlist would stop producing Covers the day a Site switched CDN, and that failure would look exactly like #47. The resolve-then-classify step is what actually stops the SSRF. - **A public cover route where the kagane proxy is session-gated.** An `<img>` cannot send a bearer token, and it cannot be given one either: the panel's shadow root is `mode: "open"`, so the host page's JavaScript can read any `src` the script sets. Both are security-adjacent departures, which is precisely why they are written down rather than left in a commit message. ## Scope Documentation only — no code, no schema, no behaviour. The implementation is #56–#63. ## Why this should merge promptly rather than sit All eight implementation tickets cite `docs/adr/0007-backend-hosts-cover-bytes.md` as the authority for decisions they must not relitigate, and they are written in the vocabulary this glossary entry defines. An agent picking up #56 reads both from `main`. Until this lands they get a 404 and either invent a rationale or stall — so this PR gates the tickets, not the other way round. Related: #47 (bug), #55 (spec), #54 (deferred admin refetch). Reviewed-on: #64 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
8081a0a5d8 |
Give the Tailscale ACL step a working policy file (#53)
Follow-up to #52, which merged before this landed. Docs only — no code, no compose changes. `DEPLOY.md` §7 told the operator to "tag the two machines" and showed a bare `acls` fragment. Following it literally does not work and is actively harmful: - the fragment references `tag:bookmark-api` / `tag:bookmark-browser` without a `tagOwners` section, so the policy is rejected on save; - it never says how a tag gets onto a device (`tailscale up --advertise-tags=...`, which re-authenticates); - replacing the tailnet's default allow-all with only that one rule **removes the operator's own SSH access to the browser machine**. Replaced with a complete, saveable policy file: `tagOwners`, the CDP rule, a second rule preserving own-device access including `:22`, and a `tests` block so a later edit that widens 9222 is rejected rather than silently applied. Also records two things that were assumed rather than stated: - **Why tagging is load-bearing.** Tailscale has no `deny`, so restricting 9222 means removing the blanket accept and enumerating what remains. That is only expressible if the browser machine falls outside a selector that still covers your own devices — which is exactly what a tag does, since a tagged device has no user and stops matching `autogroup:member` / `autogroup:self`. Without that, the whole step reads as arbitrary ceremony. - **Tagging replaces a device's user identity**, so it suits a dedicated box and disrupts a daily driver. Both paths are now written down. Finally, separates two checks the old text conflated: the existing `curl` runs on the home machine and proves only the **bind**, because node-local traffic is not filtered. Proving the **ACL** needs a third device, so that check is now its own step. Verified: `tailscale.com/docs/reference/syntax/policy-file` and `/docs/features/tags` (validated Apr 2026 / Dec 2025) for `tagOwners`, `autogroup:self` semantics vs tagged devices, `--advertise-tags` re-auth and key-expiry behaviour. Markdown fences balanced. Reviewed-on: #53 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
2a3bb6922d |
Move the browser off the VPS to its own unit (#46) (#52)
Closes #46 once deployed. The headless browser leaves the API stack and becomes its own compose unit (`chrome/docker-compose.yml`) intended for the home machine, reached over the tailnet. No fallback sidecar is left on the VPS. The backend needs no code change — `BROWSER_WS_URL` was already the only coupling. Its default is now empty rather than a pinned Docker IP, so an unconfigured or unreachable browser degrades exactly as it always has: plain-TLS libraries unaffected, kagane/novelfull logged and skipped, stored covers still served. ### What shipped - `chrome/docker-compose.yml` + `chrome/.env.example` — the browser unit, with the CDP port bound to `${BROWSER_BIND_ADDR}` (no default) and the resource limits from the epic: 512 MiB / 1 GiB memory+swap, `oom_score_adj 800`, halved CPU weight, shm 1 GiB -> 128 MiB. - API stack drops the service, its `depends_on` and the `browser` network. - `bookmark-api` gains the `default` network. Dropping `browser` had left it on `db` alone, which is `internal: true` — no published port and, worse, no egress for the poller at all. Caught by actually bringing the stack up. - ADR-0006 for the topology; `DEPLOY.md` §7 for first-time setup of the browser machine; `REDEPLOY.md` §8 for its independent update cadence; architecture diagrams, config tables and troubleshooting rows across README/AGENTS/env. ### Verified locally - Browser unit builds and runs: Chrome 151, UA carries no `HeadlessChrome`, all limits applied as declared. - **Live smoke passes through the new unit**: `TestSmokeKaganeImage` fetched 56710 bytes of `image/webp`, `TestSmokeKaganeGet` got a 200 with a real chapter list. The challenge cleared under the reduced 128 MiB shm. - Bind isolation proven: refused on the host's non-loopback address, accepted on the configured one. - 321 MiB peak of the 512 MiB cap after a full solve; 0 restarts, no OOM kill. - API stack comes up clean, `/healthz` 200; egress confirmed present on `default` and absent on `db`. - `go test ./...`, `go vet`, `gofmt` clean. ### Left to the operator Provisioning the home machine, the Tailscale ACL, setting `BROWSER_WS_URL` in production, and observing acceptance criteria 5-7 (covers with the machine off, several days of zero OOM/restarts, VPS memory improvement). `DEPLOY.md` §7 now carries the before/after `free -m` reading those need. Reviewed-on: #52 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>v1.1.0 |
||
|
sulthan
|
d1800d0707 |
Prefetch Kagane covers during latest polling (#51)
## Summary - Add an optional browser-backed cover fetcher to the latest-chapter poller. - Prefetch missing Kagane covers during the existing due-series cycle and persist them before a Reader opens the web UI. - Keep chapter polling, cooldown stamping, and on-first-view fallback independent from cover failures. ## Behavior and safety - Stored Kagane covers are detected before browser work, so later poll cycles do not refetch them. - Nil cover fetchers and non-Kagane series retain the existing behavior. - Shared Kagane image-id and content-type validation prevents challenge or non-image responses from poisoning persistent cover storage. - The browser is wired into both the chapter and cover poller paths from the composition root. ## Verification - `go test ./...` - Focused latest, store, and web package tests - Deterministic tests cover missing covers, cached covers, failed fetches, invalid content types, nil fetchers, and non-Kagane series. Closes #45 Reviewed-on: #51 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
84cfd1b2c1 |
Make browser sidecar on-demand (#44) (#50)
Closes #44. Chrome now starts on first CDP connection, tracks concurrent helpers, reaps after 300 seconds idle, preserves the named profile, and classifies reap interruptions. Shutdown stops Chrome's process group so cookie batches flush. ADR-0005 records the measured constraints and decisions. Verification: docker build, live CDP wake, graceful stop cleanup, sh -n, and go test ./... (7 packages, 3 no tests). Reviewed-on: #50 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
bfae84c5c3 |
Persist kagane covers in Postgres (#49)
Closes #43 Persist kagane cover bytes in a dedicated Postgres covers table keyed by image ID. The web handler reads storage before the browser, writes validated fetches through, and no longer keeps an in-process cover cache. Added migration, store persistence tests including reopen, handler coverage for stored/miss/rejected paths, and corrected repository guidance. Verification: - go test ./... - CGO_ENABLED=0 go build ./... Reviewed-on: #49 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |