sulthan
e2c054e7ce
Closes #60. Spec: #55. Originating bug: #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Neither #47 nor #55 is closed from here. ## What this branch does The panel now renders Covers from the deployment's own origin, and both userscripts stop having an opinion about where a Cover lives. **The public route was already in place.** `GET /covers/{address}` landed with #59 (`92eba07`) and is registered on the bare mux, outside `httpmw.Auth` and outside the web UI's Discord session — `backend/main.go:210-214`, handler `backend/internal/api/handlers.go:142-158`. It reads no cookie and no header, answers `404` for an address that was never stored (and for a row whose file has gone missing — recorded-but-gone is not-found, never a fabricated body), refuses anything that is not `^[0-9a-f]{64}$` *before* the value becomes a path, and sets `Cache-Control: public, max-age=604800, immutable`. Those four properties are asserted by `backend/cover_test.go:231-278`. This branch re-verified them rather than re-implementing them; the only backend line it touches is a comment. **Both userscripts lose cover scraping entirely.** Every adapter's `cover:` field is gone, along with the two helpers that fed them: the manga script's `coverFromPage()` (the `img[alt]` DOM scan comix needed, because comix publishes no `og:image`) and the novel script's `metaName()` plus the now-callerless module-level `meta()`. Nothing under `userscript/` reads `og:image`, `meta[name=image]`, or `img[alt]` any more. **Nothing sends a cover either.** `delete body.cover` sits in `apiPut` — `manga-bookmark.user.js:486`, `novel-bookmark.user.js:275` — which is the single chokepoint every write passes through (`pushBookmark`, the retry-queue flush, `toggleFavorite`, `toggleArchive`). It operates on the `Object.assign` copy, so the in-memory row keeps the cover it renders with. This matters beyond tidiness: a Reader upgrading from an older copy has `localStorage` rows carrying third-party scraped URLs, and without the strip those would ride back up on the next write. The handler discards the field regardless (`handlers.go:53-59`) — it is permanently inert, not pending removal. **Failed loads get the designed empty state, not the broken-image glyph.** `onerror: (e) => e.target.replaceWith(el("div", { class: "cover ph" }))` on the cover `<img>` in both card renderers (`manga:1380-1390`, `novel:1134-1144`). The replacement is byte-identical to the existing no-cover branch on the very next line, so it picks up the `.cover.ph` styling already in the panel CSS — no new tokens, no new rule. `el()` routes any `on*` prop through `addEventListener`, so this is a listener, not an inline attribute string, and the swap is a `createElement` + DOM call with no markup parsing anywhere near it. This is the half of #47 that was visible on kagane. **The deleted scraping's tests went with it**: the two comix cover cases, the `pageImages` and `namedMetas` fixtures, the `img[alt]` and `meta[name=...]` stub branches, the now-dead `querySelectorAll` stub member, and every stale `og:image` fixture and `p.cover` assertion across both suites. The export lists needed no change and that was checked, not assumed — `coverFromPage` and `metaName` were module-private on `origin/main` and no cover symbol ever appeared in `module.exports`. Docs that described the deleted behaviour were corrected in the same breath, because leaving them would instruct the next agent to put the scraping back: `userscript/AGENTS.md` (adapter contract + the per-site notes for comix, kagane and novelfull), the README's adapter reference, and the userscript testing skill's stub table. ## Verification - `go test -count=1 ./...` — green across all nine packages (`backend` 29.8s, `latest`, `store`, `session`, `token`, `userscript`, `web`). - `node --check` clean on both userscripts; `node --test` on both logic suites — 46 tests, 46 pass. - `gofmt -l` clean; `go build ./...` clean. - The `onerror` swap is DOM behaviour and deliberately has no coverage in the Node harness — that harness stubs a browser precisely so it never needs a DOM, and #60 says not to invent coverage for it. It was instead exercised for real: the `el()` helper and the exact render expression were loaded into a headless Chromium with a deliberately unloadable `src`, and the resulting DOM was `<div class="cover ph"></div>`. Ad hoc, not committed. - **Not done, needs you:** the on-device criterion — a comix Series bookmarked mid-chapter showing its Cover in the panel. That needs a real install against the deployment and is the one box left unticked on #60. ## Reviewed Both `/code-review` axes ran against `cc0fa92`. Spec found no missed requirement and no scope creep; standards found the diff clean on the four areas it scrutinised (the `delete body.cover` placement, the `onerror` handler's DOM safety, comment quality, dead-code removal). Their combined findings — the dead `querySelectorAll` stub, the stale README and skill text, and the handler comment whose premise this change invalidates — are fixed in `8b58019`. ## Out of scope, deliberately The kagane-specific cover proxy still exists and still carries its session gate (#63 deletes it). The poll's blank-Cover fill (#61) and browser-backed Sites joining the pipeline (#62) are untouched. Reviewed-on: #69 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
293 lines
16 KiB
Markdown
293 lines
16 KiB
Markdown
# Manga Bookmark
|
||
|
||
Track manga read-progress on **asurascans.com** (a.k.a. asuracomic.net),
|
||
**demonicscans.org**, **comix.to**, and **kagane.to** from a phone (Bromite /
|
||
mobile Chromium), synced to a self-hosted Go backend so bookmarks unify across
|
||
all four sites and all devices.
|
||
|
||
Two parts:
|
||
|
||
- **`backend/`** — tiny Go (`net/http` + Postgres via pure-Go `pgx`) sync service. 4 routes,
|
||
static binary, distroless container.
|
||
- **`userscript/manga-bookmark.user.js`** — single Bromite-compatible userscript
|
||
(no `GM_*` APIs) that injects an on-page bookmark UI and syncs via `fetch()`.
|
||
|
||
```
|
||
Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
|
||
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
|
||
|
|
||
| CDP over the tailnet
|
||
v
|
||
headless Chrome, on-demand,
|
||
on a separate machine
|
||
(chrome/, ADR-0006)
|
||
```
|
||
|
||
Kagane and novelfull sit behind a Cloudflare JavaScript challenge no TLS
|
||
fingerprint clears, so the poller reaches those two through a real Chrome over
|
||
CDP. That browser is **not** part of the API stack: it is its own compose unit
|
||
on a second machine, spawned on the first connection and reaped when idle. The
|
||
API needs it only to discover new chapters and to fetch a kagane cover once —
|
||
covers are stored, so the library renders in full with the browser switched off.
|
||
|
||
---
|
||
|
||
## 1. Backend
|
||
|
||
### Config (env)
|
||
|
||
| Var | Default | Notes |
|
||
|-----|---------|-------|
|
||
| `TOKEN_KEY` | *(required)* | Secret every Reader's userscript credential is derived from (issue #24); only SHA-256 hashes of credentials are stored. |
|
||
| `OWNER_DISCORD_ID` | *(required)* | Discord user ID of the owner: seeded as the first Reader, owns every pre-registration bookmark, and is the only Reader who can revoke another's sessions. |
|
||
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
|
||
| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. |
|
||
| `COVER_DIR` | *(required)* | Filesystem volume for immutable, content-addressed Cover bytes. Compose builds the image and mounts `cover-data` at this path; standalone runs may choose another writable durable path. |
|
||
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
|
||
| `BROWSER_WS_URL` | empty | CDP endpoint of the remote browser (`ws://<tailnet IP>:9222`), used to poll Kagane/Novelfull past their JS challenge and to fetch uncached Kagane covers. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header, MagicDNS names included. Unset disables both; stored covers still serve. |
|
||
| `DISCORD_CLIENT_ID` | *(required)* | Discord application credentials for the browser sign-in (ADR-0002). |
|
||
| `DISCORD_CLIENT_SECRET` | *(required)* | As above. Never logged, never echoed in an error. |
|
||
| `DISCORD_GUILD_ID` | *(required)* | The one guild whose membership gates sign-in, checked at login only. Membership *is* registration: any member becomes a Reader on first login. |
|
||
| `DISCORD_REDIRECT_URI` | *(required)* | Exact callback URL; Discord matches it verbatim against the registered redirect. |
|
||
| `DISCORD_REQUIRED_ROLE` | empty | Role snowflake a member must additionally hold. Empty means guild membership alone suffices. |
|
||
| `DISCORD_API_BASE` | `https://discord.com/api/v10` | Test seam — tests point it at a local stub so the real token exchange runs. |
|
||
| `USERSCRIPT_PATH` | `/userscript/manga-bookmark.user.js` | Bindmounted file served at `/u/{token}/manga-bookmark.user.js`. |
|
||
| `NOVEL_USERSCRIPT_PATH` | `/userscript/novel-bookmark.user.js` | Same, for the novel library. |
|
||
| `LATEST_CHAPTER_POLL_ENABLED` | `1` | `0` turns the poller off entirely. |
|
||
| `LATEST_CHAPTER_POLL_COOLDOWN` | `1h` | Rest between checks of one plain-TLS series; floor `15m`. |
|
||
| `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN` | `6h` | Rest between checks of one browser-backed series; floor `15m`. |
|
||
| `LATEST_CHAPTER_POLL_INTERVAL` | `10m` | How often the poller wakes. Cannot shorten either cooldown. |
|
||
| `LATEST_CHAPTER_POLL_BATCH` | `14` | Series per wake. Keep `BATCH × STAGGER` under `INTERVAL`. |
|
||
| `LATEST_CHAPTER_POLL_STAGGER` | `20s` | Delay between fetches in a batch — this is the outbound request rate. |
|
||
|
||
Compose reads a few more from the same `.env` that the backend never sees:
|
||
`POSTGRES_PASSWORD` (required — `DATABASE_URL` is built from it, and Postgres
|
||
only applies it while `postgres-data` is empty), `BOOKMARK_API_HOST` and
|
||
`BOOKMARK_WEB_HOST` (required by the prod override), and the optional
|
||
`PROXY_NETWORK` / `TRAEFIK_ENTRYPOINT` / `TRAEFIK_CERTRESOLVER`. The browser
|
||
unit has its own `chrome/.env` on its own machine — `BROWSER_BIND_ADDR`
|
||
(required, the tailnet IP the CDP port is published on) and the optional
|
||
`BROWSER_TZ`. Full commentary is in `.env.example` and `chrome/.env.example`;
|
||
deployment order is `DEPLOY.md`.
|
||
|
||
### Endpoints
|
||
|
||
| Method | Path | Auth | Description |
|
||
|--------|------|------|-------------|
|
||
| `GET` | `/bookmarks` | Bearer | All bookmarks of the acting Reader. |
|
||
| `PUT` | `/bookmarks/{key}` | Bearer | Upsert one series; returns the row as stored. |
|
||
| `DELETE` | `/bookmarks/{key}` | Bearer | Remove one. |
|
||
| `GET` | `/healthz` | none | `200 ok`. |
|
||
| `GET` | `/u/{token}/manga-bookmark.user.js` | credential in path | Serves the userscript with the requesting Reader's credential substituted in and an mtime-derived `@version`. |
|
||
|
||
`key` is `<site>:<series_id>` — e.g. `asura:trash-of-the-counts-family-f886a8af`,
|
||
`demonic:Infinite-Level-Up-in-Murim`, `comix:12345`, or
|
||
`kagane:3fa85f64-5717-4562-b3fc-2c963f66afa6`. Sync is last-write-wins.
|
||
|
||
`updated_at` orders the bookmark list, so it moves only on real reading
|
||
progress: the server applies its timestamp when the row is new or
|
||
`last_chapter_num` changes, and otherwise keeps the stored one. Favouriting a
|
||
series or recording a newly published chapter therefore leaves the order alone.
|
||
Because the timestamp a client sends is only a candidate, `PUT` echoes the row
|
||
**as stored** and clients adopt that rather than their own payload.
|
||
|
||
### Develop / test
|
||
|
||
```bash
|
||
cd backend
|
||
go test ./... # unit + handler tests
|
||
CGO_ENABLED=0 go build # static binary
|
||
```
|
||
|
||
**`go test ./...` requires Docker.** The store talks to a real Postgres, so
|
||
each test package starts a throwaway `postgres:17-alpine` container and gives
|
||
every test its own database inside it (`internal/pgtest`). Nothing is stubbed
|
||
and nothing reaches the network beyond the local Docker daemon.
|
||
|
||
### Run the stack
|
||
|
||
```bash
|
||
cp .env.example .env
|
||
# edit .env: set TOKEN_KEY (openssl rand -hex 32) and
|
||
# POSTGRES_PASSWORD (openssl rand -hex 24)
|
||
|
||
docker compose up -d --build # binds 127.0.0.1:8080
|
||
```
|
||
|
||
That brings up two services — the API and Postgres. The browser is deliberately
|
||
not one of them; without `BROWSER_WS_URL` the poller logs and skips kagane and
|
||
novelfull, and everything else works. To run one locally, publish it on the
|
||
Docker bridge gateway so the API container can name it by IP:
|
||
|
||
```bash
|
||
cd chrome
|
||
echo 'BROWSER_BIND_ADDR=172.17.0.1' > .env
|
||
docker compose up -d --build
|
||
# then in the repo's own .env: BROWSER_WS_URL=ws://172.17.0.1:9222
|
||
```
|
||
|
||
Bind it to `127.0.0.1` instead if you only want to drive it from the host, e.g.
|
||
`SMOKE_BROWSER_WS_URL=ws://127.0.0.1:9222 go test -run TestSmokeKagane ./internal/latest`.
|
||
In production that address is the home machine's tailnet IP and nothing else —
|
||
see `DEPLOY.md` §7 and ADR-0006.
|
||
|
||
Smoke test:
|
||
|
||
```bash
|
||
# The credential is per Reader and derived, so there is no token in .env to
|
||
# grep. Take yours from the Userscripts panel's install link after signing in,
|
||
# or read it out of an installed script's API_TOKEN constant.
|
||
TOKEN=<your Reader credential>
|
||
curl -s localhost:8080/healthz # ok
|
||
curl -s localhost:8080/bookmarks # 401
|
||
curl -s -H "Authorization: Bearer $TOKEN" localhost:8080/bookmarks # []
|
||
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
||
-d '{"title":"Test","last_chapter":"Chapter 1","last_chapter_num":1}' \
|
||
localhost:8080/bookmarks/asura:test-1
|
||
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
|
||
-H 'Access-Control-Request-Method: PUT' \
|
||
localhost:8080/bookmarks/asura:test-1 | grep -i access-control # 204 + CORS headers
|
||
```
|
||
|
||
### Deploy behind your reverse proxy
|
||
|
||
Route `https://bookmark-api.<domain>` → the service on `:8080` (TLS at the proxy).
|
||
|
||
- **Host proxy** (nginx/Caddy on the host): the base compose already binds
|
||
`127.0.0.1:8080`; point the proxy `proxy_pass http://127.0.0.1:8080;`.
|
||
- **Docker proxy** (Traefik/nginx in a container on its own network): use the
|
||
override, which drops the published port and joins the shared network:
|
||
|
||
```bash
|
||
docker network create proxy # once, if it doesn't exist
|
||
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
|
||
```
|
||
Set `PROXY_NETWORK` in `.env` if your network isn't named `proxy`.
|
||
|
||
Verify: `https://bookmark-api.<domain>/healthz` returns `ok` over valid TLS (no
|
||
mixed-content), and an `OPTIONS` preflight from a real site origin returns the
|
||
CORS headers.
|
||
|
||
---
|
||
|
||
## 2. Userscript
|
||
|
||
### Install
|
||
|
||
Sign in to the web UI and open the **Userscripts** panel: it offers one
|
||
install link per library. Each link serves a script rendered with your own
|
||
credential already inside it — you never see, type or copy a credential. The
|
||
served script carries `@downloadURL`/`@updateURL` pointing at its
|
||
credential-bearing path, so Violentmonkey keeps auto-updating it.
|
||
|
||
The bindmounted files carry `__API_TOKEN__` placeholders; the backend
|
||
substitutes the requesting Reader's credential at serve time, so no real
|
||
credential is ever committed. Rotating the credential (same panel) invalidates
|
||
every installed copy immediately — reinstall on all devices.
|
||
|
||
### Install on Bromite (mobile)
|
||
|
||
Bromite runs Chromium's native userscript engine (no Tampermonkey needed):
|
||
|
||
1. Bromite → **Settings → User scripts** → enable user scripts (allow the
|
||
permission prompt).
|
||
2. Open the install link from the web UI — Bromite detects the `.user.js` and
|
||
offers to install it.
|
||
3. Confirm the install; the `@match` list covers both sites.
|
||
4. Open a series on either site — a 📑 button appears bottom-right.
|
||
|
||
> Exact menu wording varies by Bromite build; if "User scripts" is absent,
|
||
> update Bromite or use a build with userscript support.
|
||
|
||
### Desktop iteration (optional)
|
||
|
||
The script is `GM_*`-free, so it also runs in Tampermonkey/Violentmonkey on
|
||
desktop for faster testing — install the same file unchanged.
|
||
|
||
### Use
|
||
|
||
- **Bookmark**: on a series or chapter page, open the panel → **+ Bookmark this**.
|
||
- **Auto-progress**: opening a chapter of a bookmarked series records it when the
|
||
chapter number ≥ the stored one (re-reading older chapters never regresses
|
||
progress; unparseable numbers set the current chapter).
|
||
- **Manual override**: panel → **Edit** on any row forces a specific chapter.
|
||
- **Continue**: jumps to the last-read chapter (or the series page).
|
||
- **Latest chapter**: rows read `Read: … · Latest: …` once the newest published
|
||
chapter is known and it is ahead of your progress. See below for how that is
|
||
found.
|
||
- **Favourites**: the ☆ on any row toggles it; the **★ Favourites** tab narrows
|
||
the list. Favourited series still appear under **All**. The flag syncs, so it
|
||
follows you across devices; the chosen tab does not persist.
|
||
- **Archive**: the **Archive** button on any row parks a series — it drops out
|
||
of **All** and **★ Favourites** and moves to the **Archived** tab. The server
|
||
keeps checking it for new chapters, so it is worth coming back to. Archiving
|
||
does not touch read progress, and reading an archived series leaves it
|
||
archived.
|
||
- **Finished**: series you have completed live in a **Finished** tab in the web
|
||
UI only. It is set there and nowhere else — the API rejects the value — and
|
||
finished series are hidden from every userscript tab and are no longer polled
|
||
for new chapters.
|
||
- Bookmarks made on Asura appear when the panel is opened on Demonic, and vice
|
||
versa — the backend is the shared store.
|
||
|
||
Neither favouriting nor learning a new chapter reorders the list — only reading
|
||
progress does.
|
||
|
||
Offline / backend down: changes are cached in `localStorage` and retried on the
|
||
next successful load (last-write-wins).
|
||
|
||
#### How "latest chapter" is found
|
||
|
||
Only a series page lists every chapter (a reader page links just its
|
||
neighbours), and the backend cannot fetch either site — Cloudflare blocks
|
||
server-side requests, and neither site offers an API or feed to poll. So the
|
||
userscript does the looking, from your own browser session:
|
||
|
||
- Opening a bookmarked series page records its newest chapter directly.
|
||
- Otherwise it fetches series pages in the background — **same-origin only**, so
|
||
browsing Asura refreshes Asura bookmarks and Demonic refreshes Demonic. One
|
||
series per navigation, and at most one check per series every 4 hours
|
||
(`LATEST_CHECK_BATCH` / `LATEST_CHECK_THROTTLE_MS`). Failures are silent and
|
||
simply retried after the window.
|
||
|
||
Freshness is tracked per device in `localStorage` under `bmgr:manga:lastchecked`
|
||
and is deliberately not synced, since each device checks on its own.
|
||
|
||
This means a bookmark is as current as its last check — not the moment a
|
||
chapter drops. Nothing can be instant here: neither site offers push, feeds, or
|
||
an API.
|
||
|
||
---
|
||
|
||
## Adapter reference (verified live 2026-07-24)
|
||
|
||
The site adapters key everything off URL regex, with `title` from `og:title`
|
||
(or the page's own heading where a site ships none). No adapter reads a cover:
|
||
the backend acquires, stores and serves every Cover from its own origin
|
||
(ADR-0007). Confirmed against live pages via Playwright:
|
||
|
||
| Site | Series URL | Chapter URL | `series_id` |
|
||
|------|-----------|-------------|-------------|
|
||
| **Asura** (`asurascans.com`) | `/comics/<slug-hash>` | `/comics/<slug-hash>/chapter/<n>` | `<slug-hash>` |
|
||
| **Demonic** (`demonicscans.org`) | `/manga/<slug>` | `/title/<slug>/chapter/<n>/<page>` (`chaptered.php?manga=<id>&chapter=<n>` 301s here) | `<slug>` |
|
||
| **Comix** (`comix.to`) | `/title/<id>-<slug>` | `/title/<id>-<slug>/<uploadId>-chapter-<n>` | `<id>` |
|
||
| **Kagane** (`kagane.to`) | `/series/<uuid>` | `/series/<uuid>/reader/<bookUuid>` | `<uuid>` |
|
||
|
||
Notes:
|
||
- **`asuracomic.net` deep links are dead (re-checked 2026-07-25).** They 301 to
|
||
the `asurascans.com` **root**, discarding the path, at the edge — before the
|
||
userscript gets a document — so nothing client-side can rescue them. Reach
|
||
series through `asurascans.com`. The host stays matched in case the redirect
|
||
starts preserving paths again.
|
||
- Asura `og:title` carries a `Chapter N - Read Online \| Asura Scans` suffix that
|
||
the adapter strips; Demonic chapter `og:title` is `<Title> Chapter N`.
|
||
- Demonic's `<slug>` is identical on `/manga/…` and the canonical `/title/…`
|
||
reader, so a bookmark set from the series page and the auto-update from the
|
||
reader resolve to the **same key**.
|
||
- Asura showed no Next.js markers on the live site, so navigation uses a
|
||
framework-agnostic watcher (history patch + polling) rather than a Next-only
|
||
hook — works for client-routed and full-reload sites alike.
|
||
|
||
If either site changes its URL shape, update the regex in the matching adapter
|
||
in `userscript/manga-bookmark.user.js` and re-verify.
|