a4da2e2f00be77262998fce678899cf507ab9060
43 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 sulthan
|
7c7d597019 |
Delete the kagane-specific cover path (#63) (#73)
Closes #63 Deletes the second way to reach a Cover. Since #62, every Site's cover bytes land in the content-addressed store at creation or on the poll, and the one public route serves them all — nothing needs the kagane proxy anymore. ## What went - **Template-level rewrite:** `Bookmark.CoverURL()` and both templates' use of it. Cards and chrome now render `.Cover` — the wire value — and nothing else. `Bookmark.CoverSource` was dead once `CoverURL` went, so it and its `bookmarkColumns` entry are gone too. - **Kagane-only cover route and its identifier validation:** `GET /img/kagane/{id}`, `web.CoverFetcher`, `coverIDRe`, and the whole `internal/web/cover.go`. - **The proxy's persistence:** `store.KaganeImageID`, `GetKaganeCover`, `PutKaganeCover`, `kaganeCoverSourceURL`, `kaganeCoverRe`. - **The kagane-shaped branch in the byte-fetch routing:** `fetchCoverBytes` no longer takes a `site` argument and no longer names a Site. The URL shape kagane's API publishes is claimed by the browser module itself — `kaganeImageURLRe` + `browserCoverURL` live in `latest/browser.go` with the rest of the per-Site knowledge — and `BrowserFetcher.Image` is now URL-driven (it validates the URL it will navigate to, same SSRF discipline as before). The no-plain-TLS-fallback rule for a claimed URL is preserved: a claimed address with no browser is an error, never a challenge-page fetch. ## What stayed (deliberately) - `BrowserFetcher.Image` and the browser-backed acquisition path: kagane genuinely serves cover bytes behind the challenge + `cross-origin-resource-policy: same-origin`, so the sidecar remains the only fetcher for them — it just routes by URL claim now instead of by Site name. - `fetcherFor`'s per-Site page routing (kagane/novelfull page fetches) — that is the page path, not a cover path. ## Acceptance criteria - [x] Template-level kagane cover rewrite gone - [x] Kagane-only cover route and its identifier validation gone - [x] Tests removed/rewritten against the general route, guarantees kept: unstored + traversal-shaped addresses serve nothing (`TestPublicCoverRejectsUnknownAddress`), non-image content types never echoed (`TestPublicCoverNeverEchoesNonImage` — new; the store-side gate was already pinned by `TestCoverStoreAcceptsAnySourceURL`). Store reopen-persistence and filesystem content-addressing tests rewritten against `PutCover`/`GetCover`, no guarantee lost. - [x] No Site name in a cover code path outside the acquisition module (`grep kagane backend`: store/web/templates/api are clean; remaining hits are `latest/browser.go` + `latest/sites.go`, tests, docs) - [x] Web UI and panel render Covers for all six Sites (templates render the wire address; panel renders `b.cover` — untouched, it never had a kagane path) - [x] `go test ./...` green ## Verification - `go vet ./...` clean - `go test ./...` — all packages pass (root 16.9s, latest 12.7s, store 12.7s, web 0.004s) - `CGO_ENABLED=0 go build` produces the static binary - Cover-path tests run verbosely: `TestPublicCoverServesStoredBytesUnauthenticated`, `TestPublicCoverRejectsUnknownAddress` (unknown/malformed/traversal/empty), `TestPublicCoverNeverEchoesNonImage`, `TestListRendersAcquiredCover`, `TestAcquireKaganeCoverThroughBrowser`, `TestRunOncePrefetchesKaganeCover`, `TestRunOnceRoutesNonKaganeCoverToPublicFetcher` all pass; the three `SMOKE_*` tests skip without the browser sidecar, as designed Live browser verification of the "web UI and panel render Covers for all six Sites" criterion is being run separately with Playwright against real Site pages and a locally mocked backend. Reviewed-on: #73 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
78234f3c19 |
Browser-backed Sites join the Cover pipeline (#62) (#72)
Fixes #62
Browser-backed Sites join the Cover pipeline: kagane and novelfull Series now get their Covers at creation, through the same acquisition path as every other Site, instead of waiting for a poll pass.
## What changed
`latest.Acquirer` (creation-time acquisition, fired by the first Bookmark of a Series) previously skipped kagane and novelfull entirely — their pages only yield a Cloudflare challenge to the TLS client, so the request was spent for nothing. It now routes them like the poller does, with the two Sites split exactly as the issue demands:
- **kagane** — page fetched through the browser sidecar, cover URL extracted from the API JSON, bytes fetched through the browser sidecar (the only path that clears the challenge) into the content-addressed store. With no `BROWSER_WS_URL` configured, acquisition is skipped entirely and nothing falls back to a plain fetch.
- **novelfull** — page fetched through the browser sidecar, cover URL extracted from the HTML, bytes fetched over plain TLS through the ordinary gated fetcher (its image paths answer 200 with `access-control-allow-origin: *`, measured 2026-08-09). With no browser configured, the page fetch falls back to the TLS client — novelfull's challenge is a live time-varying fact (AGENTS.md), so when the page body answers, the Cover still lands; when it is challenged, nothing happens.
The byte-routing rule (kagane → browser, every other Site → TLS) is now one shared function (`latest.fetchCoverBytes`) used by both the Poller and the Acquirer, so the two cannot drift apart.
## Acceptance criteria
- [x] kagane cover bytes are fetched through the browser sidecar and stored in the content-addressed store — `TestAcquireKaganeCoverThroughBrowser`
- [x] novelfull cover URLs are extracted from the browser-fetched HTML, and its bytes are fetched over plain TLS — `TestAcquireNovelfullCoverOverPlainTLS`
- [x] With no browser sidecar configured, kagane Covers are absent and nothing falls back to a plain fetch — `TestAcquireKaganeSkippedWithoutBrowser`
- [x] With no browser sidecar configured, novelfull Covers still work if its page body is available — `TestAcquireNovelfullCoverWithoutBrowser`
- [x] Manually verified on-device: a kagane Series shows its Cover in the panel, not a broken-image glyph — being run by a separate manual-verification agent against a mocked scenario (no prod data); not part of this PR
- [x] `go test ./...` is green, with live-network checks gated behind `SMOKE_BROWSER_WS_URL` like the existing kagane image smoke test — new `TestSmokeAcquireKaganeCover` proves the end-to-end acquire path against the real browser when the env var is set
## Verification
- `go test ./...` green across all packages
- New unit tests exercise every routing decision with fakes — no network in the default suite
- Smoke test gated behind `SMOKE_BROWSER_WS_URL`, skipped by default
## Post-review changes (
|
||
|
sulthan
|
b9220b3dfc |
The Poll fills blank Covers for every Site and both Libraries (#70)
Closes #61. ## Summary Permanently-blank Series (the half of #47 that creation-time acquisition cannot reach) heal on the next due poll cycle. The cover path is no longer kagane-only: every Site and both Libraries fill a blank Cover from the series page the chapter poll already fetched, and never replace a Cover that already exists. ## What changed ### `backend/internal/latest/poller.go` - **`fillBlankCover`** — when `Cover` and `CoverAddress` are both blank, extract a source URL via `coverFrom` from the series-page body and store bytes through `SetSeriesCover`. Skips any Series that already has a source URL (owned by prefetch) or a stored address (never overwrite). - **`prefetchCover`** — source-URL healing path, now site-uniform. Kagane no longer special-cases into `PutKaganeCover` alone; every Site lands on `SetSeriesCover`, so the wire Cover becomes a content-addressed public URL. Reuses already-stored bytes when present. - **`storeCover` / `fetchCoverBytes`** — shared fetch+persist. Only kagane routes image bytes through the browser fetcher; every other Site uses plain TLS `CoverBytesFetch`. Failures log with the Series key and never return to the chapter path. - **`checkOne`** — after a successful series-page fetch, calls `fillBlankCover` once regardless of whether chapter extraction succeeded (cover fill is independent of the chapter signal). ### `backend/internal/latest/poller_test.go` Extended the existing poller harness (real store, fake fetchers) rather than a new one: - `TestRunOnceFillsBlankCoverFromSeriesPage` — asura manga, lightnovelworld novel, kagane manga; asserts wire Cover + correct fetcher routing. - `TestRunOnceDoesNotReplaceExistingCover` — second poll does not refetch. - `TestRunOnceRetriesFailedBlankCoverOnNextPoll` — failed fill stays blank, next due cycle retries (no separate queue). - `TestRunOnceBlankCoverFailureDoesNotBlockChapter` — chapter still lands; failure log carries the Series key. - Kagane prefetch test now also asserts the content-addressed wire Cover. ## Acceptance criteria (#61) | Criterion | Status | |---|---| | Cover prefetch runs for every Site | done | | Cover prefetch runs for both Libraries | done | | Poll fills a blank Cover | done | | Poll never replaces an existing Cover | done | | Failed cover fetch does not fail/block chapter poll | done | | Failed cover fetch retried next poll, no separate queue | done | | Failures logged with the Series | done | | Existing poller tests extended | done | | `go test ./...` green | done | | Manually verified: blank Series gets Cover after a poll cycle | **left for you** | ## Out of scope / not closed - Does **not** close #47 or #55 (per ticket). - No migration/backfill script — the Poll walks every Series already. - No admin refetch (#54). ## Review notes addressed - Removed the kagane-only `PutKaganeCover` branch from prefetch so source-URL healing also sets `CoverAddress` (wire Cover). - Guard so `fillBlankCover` does not double-fetch after `prefetchCover` healed the same snapshot. - Single `fillBlankCover` call site after the series-page fetch. ## Test plan - [x] `go test ./...` (backend; needs Docker/Postgres via `pgtest`) - [ ] After deploy: pick a Series that was blank, wait one poll cycle, confirm Cover in web UI and userscript panel Reviewed-on: #70 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e2c054e7ce |
Covers render in the userscript panel, from a public route (#60) (#69)
Closes #60. Spec: #55. Originating bug: #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Neither #47 nor #55 is closed from here. ## What this branch does The panel now renders Covers from the deployment's own origin, and both userscripts stop having an opinion about where a Cover lives. **The public route was already in place.** `GET /covers/{address}` landed with #59 (`92eba07`) and is registered on the bare mux, outside `httpmw.Auth` and outside the web UI's Discord session — `backend/main.go:210-214`, handler `backend/internal/api/handlers.go:142-158`. It reads no cookie and no header, answers `404` for an address that was never stored (and for a row whose file has gone missing — recorded-but-gone is not-found, never a fabricated body), refuses anything that is not `^[0-9a-f]{64}$` *before* the value becomes a path, and sets `Cache-Control: public, max-age=604800, immutable`. Those four properties are asserted by `backend/cover_test.go:231-278`. This branch re-verified them rather than re-implementing them; the only backend line it touches is a comment. **Both userscripts lose cover scraping entirely.** Every adapter's `cover:` field is gone, along with the two helpers that fed them: the manga script's `coverFromPage()` (the `img[alt]` DOM scan comix needed, because comix publishes no `og:image`) and the novel script's `metaName()` plus the now-callerless module-level `meta()`. Nothing under `userscript/` reads `og:image`, `meta[name=image]`, or `img[alt]` any more. **Nothing sends a cover either.** `delete body.cover` sits in `apiPut` — `manga-bookmark.user.js:486`, `novel-bookmark.user.js:275` — which is the single chokepoint every write passes through (`pushBookmark`, the retry-queue flush, `toggleFavorite`, `toggleArchive`). It operates on the `Object.assign` copy, so the in-memory row keeps the cover it renders with. This matters beyond tidiness: a Reader upgrading from an older copy has `localStorage` rows carrying third-party scraped URLs, and without the strip those would ride back up on the next write. The handler discards the field regardless (`handlers.go:53-59`) — it is permanently inert, not pending removal. **Failed loads get the designed empty state, not the broken-image glyph.** `onerror: (e) => e.target.replaceWith(el("div", { class: "cover ph" }))` on the cover `<img>` in both card renderers (`manga:1380-1390`, `novel:1134-1144`). The replacement is byte-identical to the existing no-cover branch on the very next line, so it picks up the `.cover.ph` styling already in the panel CSS — no new tokens, no new rule. `el()` routes any `on*` prop through `addEventListener`, so this is a listener, not an inline attribute string, and the swap is a `createElement` + DOM call with no markup parsing anywhere near it. This is the half of #47 that was visible on kagane. **The deleted scraping's tests went with it**: the two comix cover cases, the `pageImages` and `namedMetas` fixtures, the `img[alt]` and `meta[name=...]` stub branches, the now-dead `querySelectorAll` stub member, and every stale `og:image` fixture and `p.cover` assertion across both suites. The export lists needed no change and that was checked, not assumed — `coverFromPage` and `metaName` were module-private on `origin/main` and no cover symbol ever appeared in `module.exports`. Docs that described the deleted behaviour were corrected in the same breath, because leaving them would instruct the next agent to put the scraping back: `userscript/AGENTS.md` (adapter contract + the per-site notes for comix, kagane and novelfull), the README's adapter reference, and the userscript testing skill's stub table. ## Verification - `go test -count=1 ./...` — green across all nine packages (`backend` 29.8s, `latest`, `store`, `session`, `token`, `userscript`, `web`). - `node --check` clean on both userscripts; `node --test` on both logic suites — 46 tests, 46 pass. - `gofmt -l` clean; `go build ./...` clean. - The `onerror` swap is DOM behaviour and deliberately has no coverage in the Node harness — that harness stubs a browser precisely so it never needs a DOM, and #60 says not to invent coverage for it. It was instead exercised for real: the `el()` helper and the exact render expression were loaded into a headless Chromium with a deliberately unloadable `src`, and the resulting DOM was `<div class="cover ph"></div>`. Ad hoc, not committed. - **Not done, needs you:** the on-device criterion — a comix Series bookmarked mid-chapter showing its Cover in the panel. That needs a real install against the deployment and is the one box left unticked on #60. ## Reviewed Both `/code-review` axes ran against `cc0fa92`. Spec found no missed requirement and no scope creep; standards found the diff clean on the four areas it scrutinised (the `delete body.cover` placement, the `onerror` handler's DOM safety, comment quality, dead-code removal). Their combined findings — the dead `querySelectorAll` stub, the stale README and skill text, and the handler comment whose premise this change invalidates — are fixed in `8b58019`. ## Out of scope, deliberately The kagane-specific cover proxy still exists and still carries its session gate (#63 deletes it). The poll's blank-Cover fill (#61) and browser-backed Sites joining the pipeline (#62) are untouched. Reviewed-on: #69 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
92eba07da7 |
A newly bookmarked Series acquires its Cover at creation (#59) (#68)
Closes #59. Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55. ## What changed A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s. ### Store - Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on. - `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there. - `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address. - The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path. - `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load. ### Acquisition - `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position). - Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them. - Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid. - Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later. - Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62. ### Wire and route - `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk. - `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep. - `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading. ### Config `PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`. ## Acceptance criteria All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence. ## Verification - `go test ./...` green (Docker-backed Postgres suite). - Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type. - Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red. ## Reviewed Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail. ## Known sequencing A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly. Reviewed-on: #68 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
b6b88bde8a |
feat(latest): extract per-site covers (#58) (#67)
Closes #58 ## Summary - Add pure per-Site cover extraction beside latest-chapter parsing for all six Sites. - Read Asura, Demonic, LightNovelWorld, and NovelFull metadata; read the Comix target detail state; read Kagane's browser-fetched `series_covers[].image_id` JSON. - Preserve published cover URLs, percent-encode Demonic raw spaces, select Comix's smaller published `medium`, and avoid thumbnail rendition URL synthesis. - Add live-source fixtures plus no-cover and Cloudflare challenge coverage for every Site. ## Correctness - Scope Comix extraction to the requested series detail key, avoiding recommended posters. - Parse Kagane's current live API shape and emit its canonical compressed image route from the published image ID; unrelated JSON fields are ignored. - Validate Kagane image IDs against the existing UUID-shaped route constraint. - Keep extraction pure; storage, polling, and wire integration remain outside issue #58. ## Acceptance criteria - [x] Cover extraction exists for all six Sites in the existing latest parser module. - [x] Each Site has a live-source fixture with source URL and date. - [x] Comix reads the state blob, not metadata. - [x] Demonic raw spaces are percent-encoded. - [x] Comix returns the smaller published rendition. - [x] No-cover pages return empty. - [x] Cloudflare challenge pages return empty. - [x] No thumbnail URL is synthesized by editing a published URL. - [x] `go test ./...` passes. ## Verification - `go test ./...` - `go vet ./...` - `git diff --check` Parent issues #47 and #55 remain open as requested. Reviewed-on: #67 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
9d6d3bde72 |
Add gated cover byte fetcher (#66)
## Summary Adds a plain-TLS cover byte fetcher with a destination-class SSRF gate and wires public cover sources through the content-addressed filesystem store. ## Changes - Resolve hostnames before connecting; refuse non-HTTPS, loopback, private, link-local, unique-local, CGNAT, credentials, and mixed public/private DNS answers. - Re-check every redirect and resolve/classify again at dial time to close DNS rebinding. - Reuse `maxBodyBytes`; reject oversized responses and non-image content types before persistence. - Add generic `Store.GetCover`/`PutCover` source-URL storage while preserving the browser-backed kagane path. - Keep cover prefetch failures isolated from chapter polling. - Add observable tests for TLS, no-connection refusals, all refused address classes, redirect blocking, streaming body caps, non-image rejection, content-addressed persistence, DNS rebinding, and poller routing. ## Verification - `go test -count=1 ./...` - `go vet ./...` Both pass. No test touches the live network. Closes #57 Reviewed-on: #66 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
e8d1cba6c5 |
Move cover bytes to content-addressed filesystem storage (#65)
Refs #56 ## Summary Moves Kagane cover bytes out of Postgres bytea storage into an immutable, content-addressed filesystem store. Reader-visible behavior remains unchanged: the existing session-gated route serves stored bytes, missing bytes use the existing browser fetch path, and no browser still returns a missing cover. ## Changes - Added migration 0008, which drops the legacy `covers` table and recreates it with only `address`, `path`, and `content_type`. Existing byte rows are intentionally dropped. - Added SHA-256 source-URL addressing with two-level sharding (`ab/cd/<sha256>`). Writes use a temp file plus atomic link; reads validate the stored relative path before opening it. - Made `COVER_DIR` required in runtime config and Compose. Compose passes it as a Docker build argument and volume target, so custom durable paths keep image ownership, runtime config, and the named `cover-data` volume aligned. - Updated every `store.Open` caller and documented configuration, deployment, backup, and troubleshooting behavior. - Added filesystem, restart, migration-drop, no-browser, and content-addressing coverage. ## Verification - `go test ./...` - `CGO_ENABLED=0 go build ./...` - `docker build --build-arg COVER_DIR=/data/covers -t manga-bookmark-cover-check-custom ./backend` - `docker compose config --format json` confirms custom `COVER_DIR` is the volume target - `git diff --check origin/main` - LSP diagnostics clean for touched Go files Parents #47 and #55 remain open as required by #56. Reviewed-on: #65 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
2a3bb6922d |
Move the browser off the VPS to its own unit (#46) (#52)
Closes #46 once deployed. The headless browser leaves the API stack and becomes its own compose unit (`chrome/docker-compose.yml`) intended for the home machine, reached over the tailnet. No fallback sidecar is left on the VPS. The backend needs no code change — `BROWSER_WS_URL` was already the only coupling. Its default is now empty rather than a pinned Docker IP, so an unconfigured or unreachable browser degrades exactly as it always has: plain-TLS libraries unaffected, kagane/novelfull logged and skipped, stored covers still served. ### What shipped - `chrome/docker-compose.yml` + `chrome/.env.example` — the browser unit, with the CDP port bound to `${BROWSER_BIND_ADDR}` (no default) and the resource limits from the epic: 512 MiB / 1 GiB memory+swap, `oom_score_adj 800`, halved CPU weight, shm 1 GiB -> 128 MiB. - API stack drops the service, its `depends_on` and the `browser` network. - `bookmark-api` gains the `default` network. Dropping `browser` had left it on `db` alone, which is `internal: true` — no published port and, worse, no egress for the poller at all. Caught by actually bringing the stack up. - ADR-0006 for the topology; `DEPLOY.md` §7 for first-time setup of the browser machine; `REDEPLOY.md` §8 for its independent update cadence; architecture diagrams, config tables and troubleshooting rows across README/AGENTS/env. ### Verified locally - Browser unit builds and runs: Chrome 151, UA carries no `HeadlessChrome`, all limits applied as declared. - **Live smoke passes through the new unit**: `TestSmokeKaganeImage` fetched 56710 bytes of `image/webp`, `TestSmokeKaganeGet` got a 200 with a real chapter list. The challenge cleared under the reduced 128 MiB shm. - Bind isolation proven: refused on the host's non-loopback address, accepted on the configured one. - 321 MiB peak of the 512 MiB cap after a full solve; 0 restarts, no OOM kill. - API stack comes up clean, `/healthz` 200; egress confirmed present on `default` and absent on `db`. - `go test ./...`, `go vet`, `gofmt` clean. ### Left to the operator Provisioning the home machine, the Tailscale ACL, setting `BROWSER_WS_URL` in production, and observing acceptance criteria 5-7 (covers with the machine off, several days of zero OOM/restarts, VPS memory improvement). `DEPLOY.md` §7 now carries the before/after `free -m` reading those need. Reviewed-on: #52 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
d1800d0707 |
Prefetch Kagane covers during latest polling (#51)
## Summary - Add an optional browser-backed cover fetcher to the latest-chapter poller. - Prefetch missing Kagane covers during the existing due-series cycle and persist them before a Reader opens the web UI. - Keep chapter polling, cooldown stamping, and on-first-view fallback independent from cover failures. ## Behavior and safety - Stored Kagane covers are detected before browser work, so later poll cycles do not refetch them. - Nil cover fetchers and non-Kagane series retain the existing behavior. - Shared Kagane image-id and content-type validation prevents challenge or non-image responses from poisoning persistent cover storage. - The browser is wired into both the chapter and cover poller paths from the composition root. ## Verification - `go test ./...` - Focused latest, store, and web package tests - Deterministic tests cover missing covers, cached covers, failed fetches, invalid content types, nil fetchers, and non-Kagane series. Closes #45 Reviewed-on: #51 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
84cfd1b2c1 |
Make browser sidecar on-demand (#44) (#50)
Closes #44. Chrome now starts on first CDP connection, tracks concurrent helpers, reaps after 300 seconds idle, preserves the named profile, and classifies reap interruptions. Shutdown stops Chrome's process group so cookie batches flush. ADR-0005 records the measured constraints and decisions. Verification: docker build, live CDP wake, graceful stop cleanup, sh -n, and go test ./... (7 packages, 3 no tests). Reviewed-on: #50 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
bfae84c5c3 |
Persist kagane covers in Postgres (#49)
Closes #43 Persist kagane cover bytes in a dedicated Postgres covers table keyed by image ID. The web handler reads storage before the browser, writes validated fetches through, and no longer keeps an in-process cover cache. Added migration, store persistence tests including reopen, handler coverage for stored/miss/rejected paths, and corrected repository guidance. Verification: - go test ./... - CGO_ENABLED=0 go build ./... Reviewed-on: #49 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
cd3a7e3d01 |
feat(latest): split browser poll cooldown (#48)
## Summary Split latest-chapter polling cooldowns by fetch cost. Browser-backed kagane and novelfull series now rest longer without changing the cadence of plain-TLS sites. ## Behavior - Plain-TLS series keep the 1h default cooldown. - Browser-backed series use `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, defaulting to 6h. - Both cooldowns share the existing 15m minimum floor; invalid values retain the existing fallback behavior. - The poller still selects both classes in one due query per cycle. - Existing ordering and exclusions remain unchanged: reader-count precedence, least-recently-checked ordering, finished exclusion, archived polling, and orphan exclusion. ## Implementation - Added the browser cooldown to backend configuration and passed it through production poller construction. - Added the browser-site list as the single routing source used for both due-query cutoff selection and fetcher choice. - Kept all query values parameterized; the site list is passed as a bound PostgreSQL array parameter. - Updated startup logging to report interval, plain cooldown, browser cooldown, batch, and stagger. - Documented the variable, default, and floor in `README.md`, `.env.example`, `backend/AGENTS.md`, and `docker-compose.yml`. ## Review findings addressed The first review found that configuration parsing was correct but `startLatestPoller` did not pass `BrowserCooldown` into `latest.Poller`; every browser-backed row would therefore have been due immediately. Production construction now goes through `newLatestPoller`, with a regression test covering both cooldown fields. The review also identified duplicated browser-site knowledge in fetch routing. `slices.Contains(browserBackedSites, site)` now reuses the same list already supplied to the store query. ## Verification - Focused backend tests pass: `go test ./internal/latest ./internal/store .`. - Full suite passes: `go test ./...`. - `graphify update .` completed. - Issue #42 was updated and closed. Reviewed-on: #48 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
741b23322b |
Fix comix titles and covers, kagane volume chapters, and kagane cover rendering (#37)
Fixes five reported symptoms across comix.to and kagane.to. Diagnosing them turned up two latent bugs underneath, both of which had to be fixed for the kagane cover work to function at all.
## Reported symptoms and their causes
| # | Symptom | Cause |
|---|---------|-------|
| 1 | comix bookmark titled `Comix - Read Comics online for free` | comix is an SPA that rewrites `document.title` on client routing but never touches the server-rendered `og:title`. The adapter read `og:title`, so a cold load stored the homepage's title. |
| 2 | next comix bookmark gets the *previous* series' title | Same cause. After an in-page hop, `og:title` still holds whatever page loaded first. |
| 3 | comix cover shows the placeholder | comix serves no `og:image` at all, so `coverFromPage()` had nothing to read. |
| 4 | kagane chapter never appears in the bookmark list | Reader URLs carry no chapter number, so it is parsed out of `og:title`. Volume-numbered series render `"<Series> - Volume <v> Chapter <n>"`, which the suffix regex did not match, so `chapterNum` came back null and nothing was recorded. |
| 5 | kagane title includes the chapter, e.g. `SP Baby - Volume 1 Chapter 1` | Same unmatched regex — the tail was never stripped. One fix covers 4 and 5. |
| 6 | kagane cover blocked in the web UI | kagane serves covers behind its Cloudflare challenge **and** with `cross-origin-resource-policy: same-origin`. No `<img>` on the UI's origin can load one even from a browser holding the clearance cookie. Hot-linking cannot be made to work. |
## What changed
**Userscript.** comix titles now come from `document.title` with the chapter page's `" - Ch.<n>"` tail stripped, and the cover is the `img` whose `alt` matches the cleaned title. comix fills `document.title` a beat *after* the URL changes — later than the nav watcher's 300 ms snapshot — so the watcher also re-detects when the `detect()` signature changes, not only when the URL does. The kagane suffix regex takes an optional `Volume <v> ` segment. All three page shapes were captured live on 2026-08-08 and pinned as regression tests.
**Cover proxy.** `Bookmark.CoverURL()` rewrites a stored kagane `og:image` to `/img/kagane/{id}`; templates render `.CoverURL` instead of `.Cover`. The endpoint is session-gated like every other UI route and fetches through the shared headless browser, which is same-origin with kagane and so satisfies both the challenge and the CORP header. Results are memoised in-process, so a cover costs one navigation per deployment lifetime. With `BROWSER_WS_URL` unset the endpoint answers 404 rather than reaching for a nil fetcher — the same degrade-to-userscript behaviour the poller already has.
The image id is matched against a UUID regex before it reaches the browser. That gate is load-bearing rather than tidiness: the cover is a stored client-supplied string, so an unvalidated one turns this endpoint into an SSRF primitive aimed at the deployment's own network. `ServeMux` path-cleans a traversal into a redirect before the handler runs, but the handler does not depend on that, and a test pins it.
## Two latent bugs found underneath
**`BrowserFetcher.run` never let a challenge solve.** It navigated, waited for `body`, read once, and closed the tab — roughly half a second end to end. The Cloudflare interstitial has a `body` too, so `WaitReady` was satisfied by the challenge page itself. This made the challenge *unclearable* rather than merely slow: an interstitial needs several seconds of a live page to solve itself and write clearance into the browser's shared cookie jar, so tearing the tab down first means every subsequent call is challenged exactly like the one before it. `run` now holds one tab and re-reads until the caller's predicate reports an answer, bounded by `challengeTimeout` and the caller's own deadline. Exhausting the budget maps back to the 403 the poller already expects, keeping a challenged site distinct from a broken transport.
**`chromedp/headless-shell` cannot clear kagane's challenge at all.** It is a stripped Chrome build and the tells are structural rather than a header: `navigator.webdriver` is true, the plugin list is empty, and the client hints are Chromium- rather than Chrome-branded. Overriding `webdriver` through CDP was tried on its own and changed nothing.
All measured 2026-08-08 from one IP against the same cover, so the comparisons are like for like:
| Browser | Result |
|---------|--------|
| `chromedp/headless-shell:stable` | never cleared (90 s) |
| `zenika/alpine-chrome` | never cleared — ships Chrome 124, old enough that Cloudflare refuses it and old enough to break chromedp's CDP structs |
| `google-chrome`, default UA | never cleared (60 s) — `--headless=new` advertises `HeadlessChrome` |
| `google-chrome`, stock UA, `TZ=UTC` | never cleared (90 s) |
| `google-chrome`, stock UA, any non-UTC `TZ` | **cleared in ~4 s** |
Both remaining tells are load-bearing, and each was tested in isolation. `chrome/` is a Debian image with `google-chrome-stable`, a UA whose version is read back out of the binary at startup (a hardcoded one would drift out of step with the `Sec-CH-UA` hints on the next Chrome update and become a fresh tell), and no `--enable-automation`.
### The timezone tell: UTC, not a country mismatch
The first pass concluded the zone had to match the egress IP's country. Re-measuring against the actual deployment case shows that was wrong, and the correction is in `1552dd1`.
The original inference read the host's `/etc/timezone` (`Asia/Bangkok`) and assumed a Thai egress. It isn't — this host egresses from an Indonesian IP. `Asia/Bangkok` cleared not because it matched a country but because it simply isn't UTC, and the two share +07, which hid the distinction. Same container, same Indonesian IP:
| `TZ` | Result |
|------|--------|
| `UTC` | never cleared (60 s, **twice**) |
| `Asia/Jakarta` | cleared in 4 s |
| `America/New_York` | cleared in 4 s |
`America/New_York` matches neither the country nor the offset nor the hemisphere and clears just as fast. A UTC clock is itself the bot signal — Cloudflare scores it as the datacenter default — and any real zone satisfies the check. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one, and a deployment that changes region need not keep it in sync.
One sharp edge remains: the usual `-v /etc/localtime:/etc/localtime:ro` does **not** work. Chrome resolves the zone through ICU, which takes the name from that path's symlink target and ignores the file's contents, so glibc reports the host zone while Chrome still reports UTC. `/etc/timezone` carries the name and is mounted instead.
Chrome also binds its DevTools port to loopback and silently ignores `--remote-debugging-address`, which is why headless-shell fronted it with socat. This image does the same, so it stays a drop-in: the compose service keeps the `headless-shell` name and its pinned address, and `BROWSER_WS_URL` is unchanged.
## Verification
```
go test ./... all packages ok
node --test 37 + 12 pass, 0 fail
SMOKE_BROWSER_WS_URL=... go test -run TestSmokeKagane ./internal/latest
TestSmokeKaganeImage PASS (5.29s) fetched 56710 bytes of image/webp
TestSmokeKaganeGet PASS (1.17s) status=200, real chapter-list JSON
```
The smoke test ran against the exact compose configuration — built image, empty `BROWSER_TZ`, `/etc/timezone` mounted, cold profile — hitting real kagane.to. It skips unless `SMOKE_BROWSER_WS_URL` names a sidecar, so `go test ./...` stays hermetic and Docker-only.
A red smoke run means the challenge is not clearing from that IP, which is a live, time-varying fact to re-check rather than necessarily a defect.
## Security invariants
- Auth unchanged. `/img/kagane/{id}` is session-gated by `requireSession`, the same guard as every other UI route.
- Outbound fetch gated: the id is UUID-validated before it reaches the browser, keeping the existing rule that a client-supplied string never selects a fetch target unchecked.
- No new secrets, no new logging of credentials, no change to CORS, sessions, or crypto.
- Templates still escape everything; `.CoverURL` returns a plain string and is not wrapped in `template.HTML`/`URL`.
- One new dependency-free image (`chrome/`) built from Debian plus Google's own apt repo; no new Go modules.
## Deploying
Needs `docker compose build headless-shell`.
**A UTC host must set `BROWSER_TZ`, or kagane silently stops working.** With it unset the sidecar falls back to the host's `/etc/timezone`; on a UTC server that yields UTC, which is the one value that never clears. Any real zone works — `BROWSER_TZ=Asia/Jakarta` for the current deployment. `.env.example` now documents this; it previously did not mention the knob at all.
Only the browser sidecar reads `BROWSER_TZ`. The backend keeps its UTC clock, and stored timestamps are unix ms, so nothing else shifts.
## Deliberately not done
Retry/backoff around the cover proxy, and a panel-side cover fix. The panel renders no covers, and covers cache in-process after the first fetch. Worth adding if kagane starts rate-limiting.
## Correction after review of the deployment case
`1552dd1` was added after the branch was first pushed: the deployment host runs UTC with an Indonesian egress IP, which prompted re-measuring the timezone claim and falsifying it. The earlier commits' reasoning is left intact rather than rebased away, so the diagnostic trail — including the wrong turn and what disproved it — stays readable.
Reviewed-on: #37
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
2ef769d421 |
Open registration to guild members (#27) (#36)
Closes #27. Guild membership is now the whole gate. `discordCallback` checks membership (and `DISCORD_REQUIRED_ROLE` when set), then `Store.EnsureReader` creates the Reader on first sight and returns the same row on every later login. The refusal returns before `EnsureReader`, so a turned-away sign-in leaves no row behind. `OWNER_DISCORD_ID` still seeds the owner, but only as the administrator — it no longer gates login. The cutover grace path goes with it: `API_TOKEN`, `API_TOKEN_GRACE_UNTIL` and the legacy branch in `httpmw.ResolveReader` are deleted, so a credential authenticates exactly one Reader or nothing. `userscript.Handler` drops its re-derivation too — the resolved path segment is already the credential. New surfaces: an empty library offers both install links (behind the tab-specific empty states, so "No favourites yet" still wins), and the owner alone gets a Readers panel with `POST /readers/{id}/revoke`. The owner's own row is not revocable — 404, not a self-logout. Isolation is asserted from both directions for read, modify and delete, and the shared-series invariant is pinned: two Readers on one series produce one series row, two independent progresses, one poll per due cycle, and one Reader's delete leaves the other's bookmark and the poll intact. Verified: `go test ./...` green; live smoke against a throwaway Postgres — empty-library state in both colour branches, roster rendering, a real revoke through the panel (target 401s next request, owner untouched), owner self-revoke refused 404, per-Reader `/u/<cred>` and bearer auth both 200 with 404 for an unknown credential. Reviewed-on: #36 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
c2b47eb05b |
Offer the userscripts as a download for mobile Violentmonkey (#26) (#35)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
1b1820d85a |
Cut production over: runbook corrections, env contract, Discord OAuth endpoint fix (#26) (#34)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
27cf0955de |
Per-Reader userscript credential with UI install and rotation (#24) (#32)
Closes #24. Child of #18; based on current main (includes Postgres, Reader table, Discord OAuth).
## What
Each Reader's userscript credential is derived from `TOKEN_KEY`, their Discord id and a token epoch (HMAC-SHA256, hex); only its SHA-256 sits in `readers.token_sha256` (new `token_epoch` column, migration 0006). One credential authenticates the script download path and the API bearer header.
- `internal/token`: derivation + hashing; the seed refreshes the owner's epoch-0 hash only before first rotation, so a restart can never resurrect a rotated-away credential
- `httpmw.Auth`/`ResolveReader`: acting Reader resolved from the credential hash, stashed in request context; the retired global `API_TOKEN` resolves to the owner until `API_TOKEN_GRACE_UNTIL` (enforced in code, logged per use) on both the bearer and script-download paths
- Userscript handler renders the bindmounted file with the resolved Reader's credential substituted for `__API_TOKEN__`; a legacy-path request during grace serves the derived credential, so installed devices self-migrate on their next update poll
- Web UI: "Userscripts" panel — session-gated install endpoints render the script directly (credential never in markup, address bar, or a redirect), confirm-gated rotation with an atomic epoch bump + hash rewrite and a reinstall warning
- Both userscripts carry `__API_TOKEN__` placeholders; the committed global-token literal is removed
## Design note
Credentials are derived rather than stored-random because the server must rebuild install URLs after restarts while the DB holds only hashes. HMAC output is high-entropy and unbrute-forceable; the AC's intent (unguessable, DB-leak-proof) is met.
## Deploy (also in DEPLOY.md)
1. Add `TOKEN_KEY` (`openssl rand -hex 32`) — required; changing it later invalidates every credential.
2. Keep `API_TOKEN` + set `API_TOKEN_GRACE_UNTIL` for the 14-day window.
3. After deploy, sign in → Userscripts → reinstall both scripts on every device. This also retires the old global credential for real — its literal survives in git history (present since
|
||
|
sulthan
|
bcc6b45515 |
feat(backend): Discord OAuth login with DB-backed sessions (#23) (#31)
Implements #23 per ADR-0002. - Discord authorization code grant (identify + guilds.members.read), form-encoded token exchange - Guild membership gate via the single-guild endpoint; optional DISCORD_REQUIRED_ROLE (empty default) - Owner Discord ID is the only identity allowed to sign in - Sessions are DB rows with opaque random ids; cookie carries only the id; expiry enforced; delete = revoke - HMAC session signing, derived key, and WEB_PASSWORD removed; no replacement signing secret - Login rate limiting preserved on the callback - Full flow tested through the real router against a local Discord stub (DISCORD_API_BASE) - Env: DISCORD_CLIENT_ID/_CLIENT_SECRET/_GUILD_ID/_REQUIRED_ROLE/_API_BASE/_REDIRECT_URI; docs updated go test ./... passes. Reviewed-on: #31 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
8cebb94b92 |
Give every Bookmark an owner (Reader table) (#30)
Closes #22 ## What A `readers` table appears; every Bookmark belongs to one. The owner is seeded as the first and only Reader, and all existing rows are attached to them. - **Migration 0003**: `readers` (discord_id UNIQUE, token_sha256 UNIQUE, created_at). - **Migration 0004** (run-once, version-table-gated): attaches existing bookmarks to the seeded owner, drops the surrogate `key` column, composite PK `(reader_id, site, series_id)`, FK to readers `ON DELETE CASCADE` — a duplicate Bookmark for one Reader and Series is impossible at the database level. - **Seed**: `Store.Open` runs schema to 0003, seeds exactly one owner row from `OWNER_DISCORD_ID` (hash = SHA-256 of `API_TOKEN`, refreshed on every start so rotation stays current), then migrates the rest. - **Scoping**: `List/Get/Upsert/Delete` take `readerID`; the wire `key` is derived as `site:series_id` on read. Handlers act as `Store.OwnerID()` while the global token remains the only credential. - **Unchanged**: authentication and the flat wire format — nothing observable changes from outside. - **New env** `OWNER_DISCORD_ID` (required): compose, .env.example, DEPLOY.md, README.md, backend/AGENTS.md updated. Series-level methods (due queue, mark-checked, set-latest-chapter) stay unscoped deliberately: series are shared rows polled once per due cycle, and the reader_count ordering requires cross-reader visibility (ADR-0003). ## Verification - `go test ./...` green, including new tests: seed idempotency + hash refresh, 0004 attach migration, DB-level duplicate impossibility, per-reader scoping, reader-delete cascade. - Live smoke test on fresh Postgres: seed → PUT/GET (flat wire intact) → restart idempotent; stored hash matches SHA-256 of the token. ## Deploy note `OWNER_DISCORD_ID` is required after this lands — the backend refuses to start without it. Set it to the owner's Discord snowflake (Settings → Advanced → Developer Mode → right-click name → Copy User ID). Reviewed-on: #30 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
984965ed9f |
Split Series from Bookmark, keeping the wire format flat (#21) (#29)
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
08749df050 |
feat(backend)!: run on Postgres with a migration-owned schema (#28)
Swap modernc.org/sqlite for jackc/pgx/v5 with no observable change: same endpoints, same wire format, same updated_at ordering rule. The schema now comes from numbered SQL embedded in the binary and applied on startup, one transaction each, recorded in schema_migrations. That replaces two pieces of SQLite-era machinery, both deleted rather than ported: the column probing (Postgres has ADD COLUMN IF NOT EXISTS, and there is no legacy database left to probe) and the Asura key rewrite, which has run clean on every start for months now that the userscripts strip build hashes before writing. Its regexp survives as latest.asuraBuildHash, where the poller still needs it to scope chapter links to a series whose slug carries a rotating hash. Types get real: favorite is a boolean, chapter numbers double precision, timestamps stay unix-ms bigint. SQLite's null-safe IS NOT becomes IS DISTINCT FROM, which is what implements the rule that only reading progress reorders a list. Inside COALESCE/NULLIF the status and kind parameters need an explicit ::text -- there is no target column to infer from and Postgres refuses to guess. Tests lose their free t.TempDir() database, so Docker is now a hard prerequisite for `go test ./...`: internal/pgtest starts one postgres:17-alpine per test binary and hands each test a database of its own. Also lands CONTEXT.md and the four ADRs written while scoping #18. BREAKING CHANGE: DB_PATH is retired for DATABASE_URL, which is required and has no default. Compose gains a postgres service on an internal network with its own volume; POSTGRES_PASSWORD joins .env. The old bookmarks-data volume is deliberately left undeclared so `docker compose down -v` cannot take the pre-migration database with it. main is not deployable until #25 and #26 land. Closes #20 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
b9f9aea82c |
docs: secure-coding rules for agents, and refresh stale AGENTS.md top-matter (#16)
Two doc commits: a new secure-coding rules section, plus a fix for top-matter the rebrand left stale.
## `9156525` — secure-coding rules
`AGENTS.md` carried two security invariants (bearer auth, CORS) but nothing about the code an agent actually writes here. That is the gap worth closing: measured rates for AI-generated web/backend code are ~40% vulnerable (Pearce et al.), 45% failing security tests (Veracode 2025), and users *with* assistants shipped SQLi at 36% vs 7% for the control group (Perry et al., Stanford). The failure classes cluster on broken access control, injection, session/error handling and invented dependencies — all live surfaces in this repo.
Rules were **extracted, not pasted**. Every one names a guard that already exists in-tree, so the instruction is *match this*, not *invent something*:
| Rule | Existing anchor |
| --- | --- |
| parameterized SQL only; constants may concatenate | `store.go` — all queries use `?` |
| `html/template` only; no `template.HTML` on stored data | `web.go:85` |
| client-supplied URLs pass the fetch gate | `poller.go:144 fetchableSeriesURL` |
| cap remote bodies | `fetch.go:16 maxBodyBytes` |
| `subtle.ConstantTimeCompare`, never `==` | `middleware.go:22`, `session.go:61` |
| generic error out, detail to log, never log the token | `web.go:151` |
| `X-Forwarded-Proto` for Secure; **rightmost** XFF for IP | `session.go:68,101` |
| cookie flags; expiry checked before signature | `session.go:72-94`, `Verify` |
| validate at handler boundary | `handlers.go:48` `MaxBytesReader` 64 KB, 400 on bad key/status/kind |
| site strings via `el({text})`, never `{html}` | `el()` in both userscripts |
| `fetch()`/`authHeaders()` → `API_BASE` only | existing `authHeaders` |
| `localStorage` = cache/queue, never credentials | shared with site JS |
Plus a dependency rule (stdlib first; verify a package exists before adding — ~20% of LLM-proposed packages don't resolve, which is the slopsquatting vector) and a review gate marking auth/CORS/session/crypto/fetch-gate as security-critical.
Deliberately **excluded**: container signing, k8s admission control, IaC scanning, PII/HIPAA/PCI, C/C++ memory safety. Per OpenSSF's guide for AI assistant instructions, irrelevant rules make a model generate code compensating for attacks that cannot happen. None of those apply to a single-user Go + SQLite + userscript stack.
Sources: OWASP AISVS 1.0 Appendix C, OWASP Top 10 / ASVS v5, OpenSSF *Security-Focused Guide for AI Code Assistant Instructions* (2025-08-01).
## `5d4d890` — stale top-matter
The rebrand rewrote root `AGENTS.md` as a compression pass and switched Bromite -> Violentmonkey, but left the project described as a manga-only tracker over two sites. Six sites, two libraries and two userscripts now exist.
Root `AGENTS.md`:
- *What this is* names both scripts with their site lists, the `kind` column, and the `<site>:<series_id>` key shape.
- Origins constraint generalised past Asura/Demonic.
- Records that **kagane and novelfull are reliably Cloudflare-challenged** and browser-polled over CDP. Without it that bullet list reads as contradicting the code, since the paragraph above asserts blocking is "not universal — and not reliably reproducible".
- Diagram says two userscripts.
`backend/AGENTS.md` — two instances of the same defect, found while verifying the above:
- Store key list gained `novelfull|lightnovelworld` and the `kind` column.
- **`NOVEL_USERSCRIPT_PATH` documented** — it shipped in `main.go:150` undocumented.
The dated Cloudflare paragraph is left verbatim: it is a timestamped observation ("Verified 2026-07-26"), so rewriting it would falsify a record rather than update it. `userscript/AGENTS.md` is untouched; it already documents both novel adapters and the `LIBRARY`/`STORE_PREFIX` split.
## Verification
Docs-only, no code touched. Every code reference above was read at `4229c17` before being cited — the fetch gate, body cap, constant-time compares, cookie flags, handler validation, `el()` helper and both route registrations. No invented line numbers.
## Not addressed here
The `API_TOKEN` literal is committed in plaintext in both userscripts and in their `@downloadURL`/`@updateURL` lines. The new rules say not to propagate it, but the actual remedy is rotation plus build-time substitution, since the value is already in git history. Separate change; flagging it so it does not get lost.
Reviewed-on: #16
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
4229c179b0 |
rebrand: MangaBM → BookmarkManager, add novel library support (#15)
Two intertwined changes — the rebrand and the novel library were developed on
the same branch because the novel UI plumbing is part of the new "Bookmark
Manager" wordmark in the web shell.
## What it does
- **Rebrand**: MangaBM → BookmarkManager across the Go module, compose stack,
env vars, Traefik hostnames, container/image names, userscript storage
prefixes (`mangabm:cache` → `bmgr:manga:cache`, `mangabm:queue` → `bmgr:manga:queue`),
and docs.
- **Novel library**: same backend, two libraries. New `kind` column splits
bookmarks into `manga` / `novel`; PUT validates it. Two userscripts:
- `manga-bookmark.user.js` — unchanged behaviour, just stamps its own `kind`.
- `novel-bookmark.user.js` — separate Violentmonkey install with adapters
for **novelfull.com** (polled via headless browser — Cloudflare JS
challenge) and **lightnovelworld.net** (polled via plain TLS).
- **Web UI**: library switch on the app shell. Login art, libswitch, and
novel-site colours from the Cinder design snapshot.
## Plumbing
- `addedColumns` ALTER for `kind` runs on first start after upgrade; every
pre-existing row is backfilled to `'manga'`. No manual SQL, no down-time.
- `ALLOWED_ORIGINS` gains the two novel sites.
- New `NOVEL_USERSCRIPT_PATH` env (default `/userscript/novel-bookmark.user.js`),
bindmounted alongside the manga script.
- Traefik router names `mangabm*` → `bmapi*` / `bmweb*`.
## Test status
- `go test ./...` — green
- `node --test userscript/test/logic.test.js` — 34 pass
- `node --test userscript/test/novel-logic.test.js` — 11 pass
- `node --check` on both userscripts — clean
## Notes for the redeploy
.env keys were renamed (`MANGA_API_HOST` → `BOOKMARK_API_HOST`,
`MANGA_WEB_HOST` → `BOOKMARK_WEB_HOST`). Update DNS / Traefik labels on the
prod override before pulling, otherwise the public hostnames go dark.
See the redeploy instructions I'll post next to this PR.
Reviewed-on: #15
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
4ee0b0f092 |
docs: split CLAUDE.md into per-directory guidance, add opencode agents
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
sulthan
|
ba23411a74 |
fix: address issues found in end-to-end verification
Chained defects made the kagane browser-fetch path completely non-functional in Docker Compose: headless-shell's compose command re-declared --remote-debugging-port, colliding with the image's own entrypoint/socat proxy (EOF on every dial); the sidecar was then only reachable by Docker DNS name, which Chrome's DevTools HTTP handler rejects with a 500 (Host-header/DNS-rebinding check); and NewBrowserFetcher's NoModifyURL option skipped /json/version discovery entirely, dialing a bare host:port that Chrome 404s since /devtools/browser/<uuid> is minted fresh per Chrome start. Fixed by trimming the redundant command flags, pinning headless-shell to a static IP so BROWSER_WS_URL can name it directly, and removing NoModifyURL so chromedp's discovery (which echoes the request's Host back into webSocketDebuggerUrl) does the right thing on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
sulthan
|
877d3df010 | feat(web): add comix and kagane site colours | ||
|
sulthan
|
2c7b4952f3 |
feat: wire headless-shell sidecar for challenge-gated polling
Also bumps backend/Dockerfile's build stage to golang:1.26-alpine — chromedp v0.16.0 and cdproto both require go 1.26, and the pinned 1.24-alpine base no longer builds the module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
sulthan
|
1d9b1200bb | feat(latest): add chromedp browser fetcher for challenge-gated sites | ||
|
sulthan
|
89eaef70d4 | feat(latest): allow comix and kagane, route kagane to a browser fetcher | ||
|
sulthan
|
2fcf882c49 | feat(latest): parse comix SSR state and kagane API for latest chapter | ||
|
sulthan
|
eeb601cbe2 |
Split backend into internal packages by responsibility
All Go files lived flat in backend/ as one package main. Move store,
latest-chapter polling, sessions, HTTP middleware, the JSON API, the
userscript handler, and the web UI (with its templates/static assets)
into backend/internal/{store,latest,session,httpmw,api,userscript,web},
each with an exported API. main.go becomes the composition root wiring
them into newRouter; root-level tests cover the assembled router while
package-local tests cover unit behavior. Update Dockerfile/.dockerignore
for the new internal/ tree and CLAUDE.md to describe the layout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
sulthan
|
3c935ba7c3 |
Action key under the tabs, plus the brand mark (#12)
The card action strip is icon-only, so nothing on screen said what the six
glyphs do. This adds a permanent key line under the tabs naming each one.
The key is tab-shaped, not row-shaped: archived and finished swap Archive
for Restore, and finished drops Done — the same conditions card.html already
uses for the buttons. It rides along in writeChromeOOB, otherwise an htmx
tab switch would leave the previous bucket's key behind.
Also from the Claude Design pass:
• per-action accent on hover/press — slate archive, moss finished, clay
chapter — so a press says which lane it belongs to; ember stays reserved for
new-chapter heat
• cover 74→80px desktop / 86→93px phone, title 19→21px, meta 10→11px
• favourite mark pinned to the right edge of the measure instead of trailing
the title, which drifted whenever a new-chapter title shrink-wrapped
• recent strip sizes now derive from --cover-w instead of repeating magic
numbers
### Brand mark
The design project grew a logo, so it lands in three places with three
different colour sources:
• `backend/static/logo.svg` — fixed palette, because a favicon has no page to
inherit from. Linked as `rel="icon"` from both the app and the login page.
• `{{define "mark"}}` in chrome.html — takes `--ink`/`--ember`/currentColor, so
the header mark flips with the light/dark theme. Used by app.html and
login.html.
• the userscript panel header — same drawing inside the shadow root, next to
the tokens it uses.
`.brand` becomes a flex row in both stylesheets and the topbar aligns centre
rather than baseline. The 5px stroke on the 200-unit grid thins out at brand
size, so it is nudged to 6.5 instead of scaled blindly. No mark on the edge
tab: a 7px sliver has nowhere to put one.
go test ./... passes (static-asset test now covers logo.svg), node --test on
the userscript logic suite passes 14/14. Verified live at 412px and desktop:
key renders, per-tab variant correct on /?tab=finished (play/star/pencil/undo/
trash, no Done), and the OOB swap fires on /ui/list.
Reviewed-on: #12
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
f3b55fd883 |
Work the design critique down: chapter format, colour law, search, strip, a11y (#11)
Two rounds of design-critique fixes on the web UI. Every visual change was verified at 390x844 and 1280x900 in both dark and light with screenshots; `go test ./...` is green throughout; no new dependencies. ## Earlier commits on this branch The two oldest commits predate this session and were never opened as their own PR, so they are under review here too: - Confirm-gate the lifecycle actions, cluster the action strip by consequence. - Fix the accessibility findings from the audit: contrast, focus, reduced motion. ## The rest **Chapter format.** The userscript and the poller both write `"Chapter N"`, and the templates prefixed `Ch ` again, so every real Asura row read `Ch Chapter 250` — while a manual edit stored a bare `250`, leaving two formats in one list. `DisplayChapter`/`DisplayLatest` on `Bookmark` now strip the lead-in and re-add exactly one `Ch `. **Zero-result search.** The client filter only toggled `card.hidden`, so a query matching nothing left a blank list under a fully populated, unfiltered "Continue reading" strip. There is now a no-match state with a Clear-search button, and the strip goes down while a filter is active. **The colour law.** `--ember` is documented as meaning "new chapter" and was spent on eight things, including setting "Nothing new." in the colour reserved for new chapters. Destruction moves to a new `--danger` token; text-input focus follows the searchbar idiom and turns `--paper`. Contrast, both themes: `--danger` on the page 4.82 / 6.65, the solid Remove button 4.94 / 7.30, the confirm question 9.00 / 7.98. **The remove confirm.** Buttons 40px 8px apart became 46px 12px apart, and the question names the series and the loss instead of asking "Remove this?". It opens with **Cancel** focused, not Remove — the two reversible rows still open on their affirmative. **The recent strip.** It was the head of the same `updated_at DESC` list rendered directly below it, on every tab, costing ~240px of the first phone screen. It is now scoped to series with a chapter waiting, and only on All. With nothing new anywhere it does not render — deliberate. **Stale chrome.** The strip and the Updated badge describe the whole library but live outside the swapped `#list`, so archiving a series left it under "Continue reading" with the badge still counting it, and `/?tab=all` reached by htmx differed from the same URL reloaded. Both regions move into `chrome.html` and refresh out of band on every mutation and every tab switch. **Accessibility and touch.** Esc closes any open panel and returns focus to the cell that owns it; opening a confirm moves focus into it; the inline error scrolls into view and no longer self-destructs after 5s; every tab and desktop action cell clears 44px; `role="alert"` on the login error; the card monogram is no longer announced; the busy bar is clipped by its own travel rather than by `overflow: hidden` on the card. **Chapter form label.** The panel's only visible text named the published chapter while the field held your progress. The field gets a real label; "Latest known" moves below it. **gzip.** Nothing was compressed. A stdlib middleware handles the four text types and leaves woff2 alone: style.css 21.8 -> 6.5 KB, htmx 50.9 -> 16.4, filter.js 7.6 -> 2.9. ## Not addressed The `role="status"` error slot is still mutated while hidden and then revealed, which is the non-announcing pattern the confirm rows were fixed for. Delete is still a silent vanish. Both are flagged in the critique snapshot under `.impeccable/critique/`. Design health went 24/36 (66.7%) to 29/40 (72.5%) between snapshots; the two P1s that survived were found and fixed after that run. Reviewed-on: #11 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
ac3ee9b298 |
Rebuild both UIs on the Cinder design (#10)
Implements the **Cinder** design (Claude Design doc `cfa39183`) across both UI surfaces, self-hosts the fonts it depends on, and writes down the two documents that keep the result maintainable. ## What changed **Web UI** — rebuilt on the design's visual language: editorial serif, containerless sheets divided by ash hairlines, one 760px measure, no radii and no shadows. The organising rule is that *heat is typographic*: only a series with an unread chapter is crimson (title on an ember underline, cover foot rule, `Ch N out`, play icon), and favourites get brass rather than borrowing the accent. Both states hang off two classes on the `<article>` (`is-new`, `is-dim`), so sub-elements inherit the state instead of re-deriving it. The six-cell action strip is `flex-basis: 100%` inside the row, which is what lets one piece of markup be a full-width strip with 46px thumb targets on a phone and a group of 40px squares beside the row on a desktop — no duplicate template branches. Icons moved to a sprite (`templates/icons.html`); htmx-swapped cards reference the page's symbols, so a row no longer carries a screenful of inline SVG. **Userscript panel** — repainted in the same tokens. The panel and the web UI are the same product on the same phone, and the old purple-on-charcoal panel read as a different application once the web UI moved. Structure, ids and classes are untouched, and the edge tab keeps its geometry, `touch-action` and `#hit` sizing. **Fonts are self-hosted** — five latin-subset woff2 files (~120 KB) embedded via the existing `//go:embed static`. Loading them from Google would lose the design's character exactly where it is used most: Bromite users routinely block Google's font domains, and the backend is reachable over a LAN with no internet route. `staticHandler` registers the `.woff2` MIME type, which Go's table lacks and the scratch image has no `/etc/mime.types` for. **Docs** — `docs/design-system.md` records the rules a stylesheet cannot state (what the ember is reserved for, why light mode is a re-tuning rather than an inversion, which details are load-bearing) so a future agent does not re-derive them from the CSS. `REDEPLOY.md` covers the operation actually performed every time, which `DEPLOY.md` reduced to two lines. ## Commits Each is one logical change and builds on its own: | | | |---|---| | `4d69e54` | `listView.NewCount` — data for the Updated badge, no markup | | `e940b96` | Web UI rebuilt on Cinder (CSS, templates, sprite, `filter.js`) | | `686fcc1` | Userscript panel repainted in the same tokens | | `ce7e93d` | Self-hosted webfonts + `.woff2` MIME registration | | `a547cc9` | `docs/design-system.md` | | `b20ecf1` | `REDEPLOY.md` | ## Verification - `go test ./...` — 187 pass. Userscript logic tests — 14 pass. - Screenshotted at 390px and 1180px, in dark and light, across All / Archived / Finished / empty / chapter-form / confirm-row. - Fonts: a run with `fonts.googleapis.com` and `fonts.gstatic.com` blocked still reports all five faces `loaded`; served as `200 font/woff2`. - The three `REDEPLOY.md` backup/restore commands were run, not assumed. `:ro` on the source volume fails (`unable to open database file` — WAL needs to create `-shm`), and a restored file lands root-owned while the container runs as uid 65532, so reads succeed and writes fail. Both are documented with the reason. ## Note for the reviewer The panel restyle is token-level only: the design doc covers the web UI, so the panel's *structure* has no reference to follow and was deliberately left alone. Deploying this needs a rebuild — templates, CSS and fonts are `//go:embed`ed, so pulling alone changes nothing. Reviewed-on: #10 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
0416354c06 |
Serve the userscript from the backend; card + loading fixes (#8)
Serves the userscript from the backend so Violentmonkey auto-updates it, plus two panel fixes.
## Backend: `GET /u/{token}/manga-bookmark.user.js`
The script is read off disk per request from `USERSCRIPT_PATH` and streamed back with its `@version` line rewritten.
- **Token in the path, not a header.** Violentmonkey's update poll sends no `Authorization` header, and the script embeds `API_TOKEN` in plain text — an open URL would hand that token to anyone who guessed it. Compare is constant-time.
- **404, never 401**, for both a wrong token and a missing file: a prober learns nothing about whether the route exists.
- Registered outside `withAuth` and outside the `WEB_PASSWORD` gate, so the script is installable on a deployment that never enabled the web UI.
- Stdlib only (`crypto/subtle`, `os`, `regexp`) — no new Go dependencies.
**The served `@version` is derived from the file's mtime** (`YYYY.MM.DD.HHMM`, UTC), discarding whatever the file body says. Violentmonkey only updates when the served version sorts higher than the installed one, so a body-derived version means one typo or accidental downgrade freezes updates forever. An mtime-derived version is monotonic by construction. A file with no `@version` line is served byte-identical. `os.Stat` runs before `os.ReadFile`, so a concurrent edit can only serve new content under an old stamp — which self-heals on the next poll — never the reverse.
## Bindmount
`./userscript` is bindmounted read-only at `/userscript`. The script is deliberately **not** copied into the image: the build context stays `./backend`, and widening it would churn every `COPY` path for a file the mount always supplies. Editing the file on the VPS is live on the next poll — no rebuild, no restart. `git pull` restores the committed version, so a redeploy always ships the repo's script; checkout sets mtime to now, so even a rollback serves a *higher* version and is adopted. Without the mount the endpoint 404s and logs it; bookmark sync is unaffected.
`@downloadURL` / `@updateURL` are literal URLs in the metadata block — it is parsed before any JS runs, so `API_BASE`/`API_TOKEN` cannot be interpolated. The token was already committed in this file, so this adds no new exposure.
## Userscript UI
- **Card actions moved under the subtitle.** Only the cover and the title continue reading now; the subtitle and the action row are inert siblings in the text column. A thumb that misses ★ lands on nothing, and Remove is never inside a link.
- **Loading spinner** while the first fetch is in flight — the panel used to read as frozen on the first open after a cold start. It draws only when there is nothing cached to draw instead, so a populated list never flaps.
## Verification
- `go test -count=1 ./...` — ok, 7.070s
- `node --check` clean; `node --test userscript/test/logic.test.js` — 14/14
- Live `docker compose` smoke: `/healthz` 200, wrong token 404, script served with a stamped `@version 2026.07.28.1057` and both metadata URLs present; `touch`ing the file advanced the served version to `2026.07.28.1100` with no restart.
Layout and spinner are verified on-device — there is deliberately no DOM test harness.
Reviewed-on: #8
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
206c447f36 |
Stable Asura series IDs: strip rotating build hash (#6)
Asura series slugs carry a site-wide build hash (-059befe1) that rotates on every redeploy, silently orphaning all asura bookmarks (old-hash URLs 302 to new-hash ones, so detect() yields keys that never match stored rows).
Backend:
- migrateAsuraKeys in OpenStore: one-off idempotent migration rewriting hashed asura keys to the stable hashless ID, merging collisions to newest updated_at; losers deleted before winner rewrite (PK-collision safe) — regression tests included
- poller latestChapterFrom: build hash made optional in chapter-scoping regex so latest_chapter survives redeploys
Userscript:
- stripBuildHash(/-[0-9a-f]{8}$/) applied to seriesId in both asura detect branches; URLs keep full slug (stale hashes 302)
- load-time migration rewrites cached + retry-queued asura keys to the stripped form so a queued PUT cannot resurrect an orphaned row
Docs: AGENTS.md + CLAUDE.md URL-shape notes; design spec at docs/superpowers/specs/2026-07-28-asura-stable-series-id-design.md
Verified: go test -count=1 ./... green (new rotation + collision tests), node --check green, regex verified against live asurascans.com slugs.
Deploy order: backend first (migration runs at OpenStore). Orphan rows created by not-yet-updated userscripts self-heal on next backend restart.
Reviewed-on: #6
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
6af49e6790 |
Archived and finished buckets, userscript nav chips (#4)
Gives every bookmark a lifecycle bucket — `reading`, `archived`, or `finished` — so on-hold series leave the main list while still being polled for new chapters, completed series get a web-only bucket, and the userscript panel gains quick links to the web UI and both manga sites.
Design: `docs/superpowers/specs/2026-07-27-status-buckets-design.md`
## Data model
One additive column through the existing `addedColumns` migration list:
```sql
ALTER TABLE bookmarks ADD COLUMN status TEXT NOT NULL DEFAULT 'reading'
```
The `DEFAULT` backfills every pre-existing row as `reading`, so there is no separate migration step. `favorite` is unchanged and orthogonal — a series can be an archived favourite.
Rollback is safe: an old binary against the new database omits `status` from its INSERT (it gets the DEFAULT) and never mentions it in the conflict clause, so buckets survive.
## The write rule
`PUT /bookmarks/{key}` decodes a whole `Bookmark` and `Upsert` writes every column it knows about. `latest_checked_at` escaped this by staying out of `bookmarkColumns` entirely — `status` cannot, because the userscript must be able to archive and restore.
So an empty incoming status means **"no opinion"**, not a value, and resolves on the `VALUES` side of the upsert:
```sql
COALESCE(NULLIF(?, ''), (SELECT status FROM bookmarks WHERE key = ?), 'reading')
```
with `DO UPDATE SET status = excluded.status`.
It has to be this way round. `excluded.*` is the row *after* the `VALUES` expressions are evaluated, so applying the default there and then reading `excluded.status` in the conflict clause would see `'reading'` rather than the empty string — and would overwrite an archived row on every progress PUT from a client that knows nothing about the column. One expression, evaluated once, covers insert and update alike. The subquery runs inside the transaction, so it sees the row the statement is about to conflict with.
`TestUpsertEmptyStatusPreservesStored` is the guard on this.
`updated_at` behaviour is unchanged: it moves only when `last_chapter_num` changes, so archiving, finishing, restoring, and favouriting never reorder the list.
## Validation
`PUT /bookmarks/{key}` returns 400 for any status outside `{"", "reading", "archived", "finished"}`, and for `"finished"` specifically. Finishing a series is a web-UI decision, enforced server-side rather than by trusting every client to leave the value alone. The `/ui/*` endpoints have their own session-guarded route and are unaffected.
## Visibility
| Surface | All | Updated | Favourites | Archived | Finished |
|---|---|---|---|---|---|
| Web | reading | reading | reading | archived | finished |
| Userscript | reading | — | reading | archived | not shown |
Archived and finished appear in their own tab and nowhere else — including the web UI's "Continue reading" strip, which is now built from reading-only rows before tab filtering. An archived favourite shows up under Archived only: Favourites means "favourites I am currently reading".
## Backend
- **`store.go`** — `Bookmark.Status`, the column in `schema` / `addedColumns` / `bookmarkColumns` / `scanBookmark` / `Upsert`. `scanBookmark` normalises anything outside the three known buckets to `reading`, so no row can land in no list at all.
- **`store.go`** — `DueForLatestCheck` gains `AND status IS NOT 'finished'`. Archived series keep being polled; that is the whole point of archiving rather than deleting. Finished ones have nothing coming, so polling them only burns fetches and risks a spurious "new chapter" badge. `IS NOT` is null-safe, so a hand-edited NULL still qualifies.
- **`handlers.go`** — status validation on `PUT`, before any write.
- **`web.go`** — `buildListView` filters the new tabs and excludes both buckets from `all` / `new` / `fav` and the recent strip; new `POST /ui/bookmarks/{key}/status`, session-guarded like its siblings, read-modify-writing through `Store.Get` + `Store.Upsert` so the `updated_at` rule stays in one place.
- **`templates/`** — two more tabs; per-card controls (reading → Archive + Finish, archived → Restore + Finish, finished → Restore); empty-state copy for both new tabs.
- **`static/style.css`** — five tabs no longer divide a phone's width legibly, so the row scrolls sideways instead of squeezing.
## Userscript (1.3.0)
- Third tab **Archived** beside All and Favourites. A missing `status` reads as `reading`, so a list cached by the previous version still renders. `finished` matches no tab and is invisible everywhere.
- Per-item **Archive / Unarchive** button on the existing optimistic path: mutate local state and cache, `apiPut`, adopt the server's returned row.
- Header chip row linking the web UI and both manga sites, each `target="_blank" rel="noopener"`.
- `apiPut` now omits `status` unless the caller opts in — see below.
Still free of every `GM_*` API: plain `fetch`, page `localStorage`, on-page UI only.
## One bug worth calling out
The userscript's other mutations (`updateToCurrentChapter`, `setChapterManual`, `toggleFavorite`, `applyLatestChapterIfChanged`) build their payload with `Object.assign({}, existing, …)`, so they echoed the cached `status` back to the server. `GET /bookmarks` has no status filter — finished rows are in `state.list` and only hidden at render time — which made two failures reachable:
1. Reading a chapter of a series marked finished sent `"status":"finished"`, which the API rejects with 400. Progress never synced, behind a misleading "Offline — saved locally, will retry" toast, permanently.
2. Archiving on desktop and then reading on a phone whose cache predated the archive sent `"status":"reading"` and silently un-archived the series — contradicting the README's "reading an archived series leaves it archived".
Fixed at the single choke point: `apiPut(key, obj, { sendStatus = false })` strips `status` from a copy of the body unless the caller opts in, and only `toggleArchive` opts in. Only an explicit archive/restore has an opinion about the bucket; everything else omits the field so the server's keep-on-empty rule applies. Stripping merely the *invalid* values would not have been enough — a stale cached `"reading"` still clobbers a remote archive.
Also: the userscript's `backgroundRefreshLatest` now skips finished series, matching the server poller, instead of spending batch slots fetching pages for a series that has nothing coming.
## Known limitations, deliberate
Both are marked in-code with `ponytail:` comments naming the ceiling and the upgrade path:
- The poller's `Store.Get` + `Store.Upsert` is not wrapped in a transaction, so a client PUT that commits between the two is lost to the stale re-read. Already documented for read progress in `CLAUDE.md`; it now costs a status change too. Accepted for a single-user deployment.
- A card whose new status no longer matches the active tab stays on screen until the next list load. The alternative is an out-of-band swap or a full list refresh per toggle, and the card visibly showing its new state is enough feedback.
## Testing
`go test ./...` passes; `CGO_ENABLED=0 go build ./...` clean.
- **`store_test.go`** — a fresh row defaults to `reading`; a legacy database gains the column with every row `reading`; an `Upsert` carrying `""` preserves the stored bucket while a value replaces it; a status change does not move `updated_at`; `DueForLatestCheck` returns archived and skips finished; the poller's `Get` → `Upsert` round trip preserves `archived`.
- **`main_test.go`** — `PUT` with `finished` or garbage is 400, `""` / `reading` / `archived` round-trip; a PUT that omits the `status` key entirely (what a pre-1.3.0 userscript sends) preserves an archived bucket *and* applies the chapter progress in the same request.
- **`web_test.go`** — each tab returns only its bucket; the recent strip excludes archived and finished; the status endpoint requires a session, rejects unknown values, and does not move `updated_at`; the card renders the right controls per bucket.
Userscript has no automated harness, so it was checked against a live `https://asurascans.com` page: the chips resolve, Archive moves a series out of All and Favourites into Archived, the state survives a full reload (so it came from the server, not local optimism), Unarchive returns it, a series marked finished in the web UI appears in no tab, and — captured on the wire — the archive PUT carries `"status":"archived"` while a favourite toggle on that same archived series carries no `status` key at all.
Reviewed-on: #4
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
a587b16423 |
Web UI: Updated tab, inline errors, mobile card fixes (#3)
Third pass on the password-gated web UI, on top of #1 and #2. ## Updated tab New `?tab=new` tab listing only series with an unread published chapter, plus per-tab empty states for Favourites and Updated. Covered by `TestUIListNewTab`. ## Inline error feedback htmx does not swap on a non-2xx response, so a failed favourite/chapter/delete looked like an ignored tap. Errors now render in a `.error-inline` slot on the card and clear after 5s. The chapter-edit form and delete-confirm row also close each other — only one per-card panel open at a time. ## Mobile fixes (P0) `.chapter-form` held three children on one unwrapped flex row, pushing Save off screen: **97px of page overflow at 390px, 127px at 360px**. That broke correcting a chapter number on the primary device class. - `.chapter-form` and `.confirm-row` wrap; hint and prompt take their own full-width row - `.chapter-form input` uses `flex: 1 1 0; min-width: 0` — with `flex: 1` (basis auto) a number input holds its ~20ch intrinsic width and refused to shrink, which pushed Save to a third row - `white-space: nowrap` on the confirm prompt alone reintroduced 26px of overflow; the full-width row is what actually fixes it Verified live: `document.body.scrollWidth <= window.innerWidth` with every chapter-form and confirm-row open, at 360/390/768/1280, light and dark. ## Icons `☆ ✎ 🗑 ▶` replaced with hand-authored inline SVG on `currentColor` — the emoji font rendered each in a different face, weight, and colour, ignoring the card's own type and colour system. `.icon.on` / `.icon.danger` / `.primary` keep driving colour. No icon font or library added. ## No-cover empty state Series whose source site gave no `og:image` render a title-initial monogram (`Bookmark.Initial()`) instead of a blank `--surface-2` rectangle that read as a cover still loading. Shared between the card cover and the Continue-reading strip. ## Verification - `go build ./... && go test ./...` — pass - Live interaction run: favourite toggle round-trips, chapter save 210.5→211, All/Updated/Favourites swap, delete-confirm removes the card, no console errors Reviewed-on: #3 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com> |
||
|
sulthan
|
62772e1eaa |
feat: server-side latest-chapter polling (#2)
Adds a background goroutine to the backend that re-checks each bookmarked series' newest published chapter on its own schedule, so `latest_chapter` stays fresh even when the manga sites are never opened in a browser.
This is a *second, parallel* signal, not a replacement: the userscript keeps its own `maybeCaptureLatestOnSeriesPage` / `backgroundRefreshLatest` logic, unchanged. `userscript/manga-bookmark.user.js` is byte-identical to `main`.
## How it works
One ticker goroutine in the same binary. Each wake it asks SQLite for bookmarks whose `latest_checked_at` has aged past a per-bookmark cooldown, fetches those series pages through a Chrome-fingerprinted HTTP client, extracts the max chapter number with a per-site regex, and writes it back through `Store.Get` + `Store.Upsert`. Every failure path logs and moves on.
Two independent clocks:
- **cooldown** — how long one bookmark rests between checks, enforced by the `WHERE` clause in `Store.DueForLatestCheck`, not by a timer.
- **interval** — how often the goroutine wakes and looks.
Shortening the interval therefore cannot shorten anyone's cooldown; it only makes the poller wake and find nothing due more often.
The row is stamped **before** the fetch, so an error, a timeout, or a shutdown mid-request still consumes the cooldown — a renamed or challenged series waits out a full cooldown instead of being retried every tick.
## Design decisions worth reviewing
**`latest_checked_at` is deliberately absent from the `Bookmark` struct and from `bookmarkColumns`.** `PUT /bookmarks/{key}` decodes a whole `Bookmark` and `Upsert` writes every column it knows about, so a userscript PUT — which has no idea this field exists — would write a zero and reset the cooldown, making the poller re-fetch that series on every tick for as long as the user kept reading it. Two tests guard this: `TestUpsertPreservesLatestCheckedAt` and `TestPutDoesNotClobberLatestCheckedAt`, the latter driving a real router PUT with a userscript-shaped body.
**`updated_at` never moves on a latest-chapter bump.** All chapter writes go through `Store.Get` + `Store.Upsert`, so the existing `CASE` keeps the stored timestamp when only `latest_chapter_num` changes and the bookmark list does not reorder. `TestRunOnceDoesNotReorderList` asserts both the timestamp and the `List()` head position.
**Fetches use `bogdanfinn/tls-client` with a Chrome profile.** Plain `net/http` was verified working against both sites on 2026-07-26, so this is not fixing an observed block — it is deliberate defence-in-depth against a future fingerprint-based one. The library is pure Go, so `CGO_ENABLED=0`, the static binary, and the distroless image are all unaffected. It does require the Go floor to move 1.23 → 1.24.
**`checkOne` validates before spending a request.** `series_url` is entirely client-supplied through `PUT /bookmarks/{key}`, so without a guard the poller would issue GETs from the server's own network position to any URL a token holder writes. The check requires a known site and an `https` URL with a non-empty host, and sits *after* the cooldown stamp so an unfetchable row is retried at cooldown pace rather than hot-looping.
## Config
Five new env vars, all with defaults sized for this deployment, all wired through `docker-compose.yml`:
| Variable | Default | Meaning |
| --- | --- | --- |
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | Kill switch |
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Per series, floored at `15m` |
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often to wake |
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake |
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch |
`batch × (cooldown / interval)` = 84 series hold a true cooldown cadence at these defaults. Past that nothing breaks: the cadence stretches uniformly and the oldest-checked-first ordering keeps it fair. Bad values log and fall back rather than failing startup — the poller is an enhancement, and a typo in one of its knobs must not stop bookmark sync.
## Known limitation (accepted, documented)
The poller's `Store.Get` + `Store.Upsert` is not wrapped in a single transaction. If a userscript `PUT` commits in the sub-millisecond window between the two, the poller writes back its stale re-read — reverting that progress and, since the stored `last_chapter_num` now differs, tripping the `updated_at` `CASE` and reordering the list.
Accepted rather than fixed for a single-user deployment: the window is one SELECT wide, the poller only writes when a chapter number actually changed, and the next read self-heals it. The alternative — a transactional read-modify-write — means moving or duplicating the `updated_at` `CASE` that four tests and the whole list-ordering invariant depend on. Recorded in `CLAUDE.md` next to the poller's architecture bullet so it is not a silent trap.
## Testing
- Full suite green, including `-race`; `go vet` clean; `CGO_ENABLED=0` static build and `docker compose build` both pass on the bumped `golang:1.24-alpine`.
- No test touches the network: the `fetcher` interface exists so tests inject a fake, and no test imports `tls-client` or reaches either manga site.
- Extraction is fixture-driven against markup trimmed from real pages (2026-07-26), including a Cloudflare challenge page, cross-series chapter links, decimal chapters, and both raw `&` and `&` forms.
- Poller tests cover the no-reorder invariant, cooldown enforcement across passes, batch limiting, one bad series not stalling a batch, downward correction on a retracted chapter, cancelled contexts, and all four failure shapes still consuming the cooldown.
- Migration from a pre-column database has its own test — `newTestStore` takes the `CREATE TABLE` path, so the `ALTER TABLE` path would otherwise be untested.
- **Live smoke test:** real server, real fetch of asurascans.com. Log showed `latest is now Chapter 181` and `due=1 checked=1`; `GET /bookmarks` returned `latest_chapter_num: 181` with `updated_at` byte-identical to the PUT that created the row — the no-reorder invariant confirmed against a live site, not just a fake.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: #2
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
ebc7a546c5 |
feat: password-gated web UI on the same backend (#1)
Adds a password-gated browser UI for the bookmark list, served by the same Go
binary and container as the userscript API.
## What
- `GET /` — list page, or the login page when there is no session (200, no redirect).
- `POST /login`, `POST /logout` — stateless HMAC session cookie, 60-day Max-Age.
- `GET /ui/list?tab=all|fav`, `POST /ui/bookmarks/{key}/favorite`,
`POST /ui/bookmarks/{key}/chapter`, `DELETE /ui/bookmarks/{key}` — htmx fragments.
- `GET /static/*` — embedded `style.css`, `htmx.min.js`, `filter.js`.
Mobile-first dark CSS, 2–3 column grid at ≥900px, "Continue reading" strip of the
five most recent series, NEW badge, client-side title search, no build step.
## Stack
Go `html/template` + htmx 2.0.4 (vendored, 50 KB) + plain CSS. No npm, no bundler.
Templates and assets are `go:embed`-ed, so `CGO_ENABLED=0` and the distroless
image still hold.
## Auth
`WEB_PASSWORD` gates the UI; unset means the web routes are never registered and
`/` returns 404. Session cookie is `HttpOnly`, `SameSite=Lax`, `Secure` when the
request is HTTPS. The signing key derives from `API_TOKEN` + `WEB_PASSWORD`, so
rotating either logs every browser out. Login is rate-limited to 10 failures per
20 minutes per client IP, keyed on the **rightmost** `X-Forwarded-For` entry
(Traefik appends the observed peer, so the leftmost is client-spoofable). CGNAT
lockout is a known, accepted limitation — the window self-heals.
## Invariants preserved
- A session cookie never authenticates `/bookmarks*`. That API stays JSON +
bearer token, unchanged, as does the userscript.
- `Store.Upsert` is byte-for-byte unmodified. Every UI write goes
read-modify-write through the new `Store.Get`, so the conditional-`updated_at`
rule (favouriting must not reorder the list, a chapter override must) lives in
exactly one function.
## Deployment
`docker-compose.prod.yml` gains a second Traefik router on `MANGA_WEB_HOST`
pointing at the same service — one container, one certificate resolver, no second
service. Both `MANGA_API_HOST` and `MANGA_WEB_HOST` are required (`:?`), with no
example fallback in `.env.example`: a placeholder there would make Traefik
silently publish the UI on a domain you do not own. Needs a DNS A/AAAA record for
`manga.<domain>`. See `DEPLOY.md` §1b.
## Docs
- Design: `docs/superpowers/specs/2026-07-25-web-ui-design.md`
- Plan: `plans/2026-07-25-web-ui-implementation-plan.md`
## Verification
`gofmt` clean, `go vet`, `go test -race ./...`, `CGO_ENABLED=0 go build`, a real
`docker build` + curl smoke test, and a Playwright pass covering login
reject/accept, favourite-without-reorder, chapter edit, delete-with-confirm,
search, tab switch + back button, 390px with no horizontal overflow, and zero JS
console errors.
Reviewed-on: #1
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
|
||
|
sulthan
|
01301805fb |
feat(backend): favorite + latest-chapter fields, conditional updated_at
Adds favorite, latest_chapter and latest_chapter_num to the bookmark record, with an idempotent ALTER TABLE migration so the already-deployed database picks them up. updated_at now moves only when a bookmark is new or last_chapter_num changes. Clients order their list by updated_at, so favoriting a series or recording a newly published chapter must not disturb that order. Upsert consequently returns the row as stored and the handler echoes that rather than the request payload, since the candidate timestamp it sends is often discarded. Scanning also tolerates NULL in the optional columns, which a database created before this code can legitimately contain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
 claude
|
f58d113934 |
feat: manga bookmark sync backend + Bromite userscript
Backend (Go, stdlib net/http + modernc.org/sqlite, CGO-free static binary):
- GET/PUT/DELETE /bookmarks{,/key} + /healthz
- bearer auth (constant-time), CORS origin reflection + 204 preflight
- SQLite store keyed <site>:<series_id>, last-write-wins, server-set updated_at
- httptest + temp-sqlite tests (auth, CORS, round-trip); go vet clean
- multi-stage Dockerfile (distroless static nonroot) + compose (base + prod proxy override)
Userscript (single Bromite-compatible IIFE, no GM_* APIs):
- Asura + Demonic adapters, URL-regex ids + og: title/cover
- Shadow-DOM floating UI, localStorage cache, optimistic sync
- auto-progress (no regress) + manual override; framework-agnostic nav watcher
Live-verified adapters (Playwright, 2026-07-24): asurascans.com /comics/<slug-hash>,
demonicscans.org /manga/<slug> + /title/<slug>/chapter/<n> — corrects the plan's
assumed /series/ paths and asuracomic.net domain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|