sulthan
741b23322b
Fixes five reported symptoms across comix.to and kagane.to. Diagnosing them turned up two latent bugs underneath, both of which had to be fixed for the kagane cover work to function at all.
## Reported symptoms and their causes
| # | Symptom | Cause |
|---|---------|-------|
| 1 | comix bookmark titled `Comix - Read Comics online for free` | comix is an SPA that rewrites `document.title` on client routing but never touches the server-rendered `og:title`. The adapter read `og:title`, so a cold load stored the homepage's title. |
| 2 | next comix bookmark gets the *previous* series' title | Same cause. After an in-page hop, `og:title` still holds whatever page loaded first. |
| 3 | comix cover shows the placeholder | comix serves no `og:image` at all, so `coverFromPage()` had nothing to read. |
| 4 | kagane chapter never appears in the bookmark list | Reader URLs carry no chapter number, so it is parsed out of `og:title`. Volume-numbered series render `"<Series> - Volume <v> Chapter <n>"`, which the suffix regex did not match, so `chapterNum` came back null and nothing was recorded. |
| 5 | kagane title includes the chapter, e.g. `SP Baby - Volume 1 Chapter 1` | Same unmatched regex — the tail was never stripped. One fix covers 4 and 5. |
| 6 | kagane cover blocked in the web UI | kagane serves covers behind its Cloudflare challenge **and** with `cross-origin-resource-policy: same-origin`. No `<img>` on the UI's origin can load one even from a browser holding the clearance cookie. Hot-linking cannot be made to work. |
## What changed
**Userscript.** comix titles now come from `document.title` with the chapter page's `" - Ch.<n>"` tail stripped, and the cover is the `img` whose `alt` matches the cleaned title. comix fills `document.title` a beat *after* the URL changes — later than the nav watcher's 300 ms snapshot — so the watcher also re-detects when the `detect()` signature changes, not only when the URL does. The kagane suffix regex takes an optional `Volume <v> ` segment. All three page shapes were captured live on 2026-08-08 and pinned as regression tests.
**Cover proxy.** `Bookmark.CoverURL()` rewrites a stored kagane `og:image` to `/img/kagane/{id}`; templates render `.CoverURL` instead of `.Cover`. The endpoint is session-gated like every other UI route and fetches through the shared headless browser, which is same-origin with kagane and so satisfies both the challenge and the CORP header. Results are memoised in-process, so a cover costs one navigation per deployment lifetime. With `BROWSER_WS_URL` unset the endpoint answers 404 rather than reaching for a nil fetcher — the same degrade-to-userscript behaviour the poller already has.
The image id is matched against a UUID regex before it reaches the browser. That gate is load-bearing rather than tidiness: the cover is a stored client-supplied string, so an unvalidated one turns this endpoint into an SSRF primitive aimed at the deployment's own network. `ServeMux` path-cleans a traversal into a redirect before the handler runs, but the handler does not depend on that, and a test pins it.
## Two latent bugs found underneath
**`BrowserFetcher.run` never let a challenge solve.** It navigated, waited for `body`, read once, and closed the tab — roughly half a second end to end. The Cloudflare interstitial has a `body` too, so `WaitReady` was satisfied by the challenge page itself. This made the challenge *unclearable* rather than merely slow: an interstitial needs several seconds of a live page to solve itself and write clearance into the browser's shared cookie jar, so tearing the tab down first means every subsequent call is challenged exactly like the one before it. `run` now holds one tab and re-reads until the caller's predicate reports an answer, bounded by `challengeTimeout` and the caller's own deadline. Exhausting the budget maps back to the 403 the poller already expects, keeping a challenged site distinct from a broken transport.
**`chromedp/headless-shell` cannot clear kagane's challenge at all.** It is a stripped Chrome build and the tells are structural rather than a header: `navigator.webdriver` is true, the plugin list is empty, and the client hints are Chromium- rather than Chrome-branded. Overriding `webdriver` through CDP was tried on its own and changed nothing.
All measured 2026-08-08 from one IP against the same cover, so the comparisons are like for like:
| Browser | Result |
|---------|--------|
| `chromedp/headless-shell:stable` | never cleared (90 s) |
| `zenika/alpine-chrome` | never cleared — ships Chrome 124, old enough that Cloudflare refuses it and old enough to break chromedp's CDP structs |
| `google-chrome`, default UA | never cleared (60 s) — `--headless=new` advertises `HeadlessChrome` |
| `google-chrome`, stock UA, `TZ=UTC` | never cleared (90 s) |
| `google-chrome`, stock UA, any non-UTC `TZ` | **cleared in ~4 s** |
Both remaining tells are load-bearing, and each was tested in isolation. `chrome/` is a Debian image with `google-chrome-stable`, a UA whose version is read back out of the binary at startup (a hardcoded one would drift out of step with the `Sec-CH-UA` hints on the next Chrome update and become a fresh tell), and no `--enable-automation`.
### The timezone tell: UTC, not a country mismatch
The first pass concluded the zone had to match the egress IP's country. Re-measuring against the actual deployment case shows that was wrong, and the correction is in `1552dd1`.
The original inference read the host's `/etc/timezone` (`Asia/Bangkok`) and assumed a Thai egress. It isn't — this host egresses from an Indonesian IP. `Asia/Bangkok` cleared not because it matched a country but because it simply isn't UTC, and the two share +07, which hid the distinction. Same container, same Indonesian IP:
| `TZ` | Result |
|------|--------|
| `UTC` | never cleared (60 s, **twice**) |
| `Asia/Jakarta` | cleared in 4 s |
| `America/New_York` | cleared in 4 s |
`America/New_York` matches neither the country nor the offset nor the hemisphere and clears just as fast. A UTC clock is itself the bot signal — Cloudflare scores it as the datacenter default — and any real zone satisfies the check. `BROWSER_TZ` therefore needs a plausible zone, not a geolocated one, and a deployment that changes region need not keep it in sync.
One sharp edge remains: the usual `-v /etc/localtime:/etc/localtime:ro` does **not** work. Chrome resolves the zone through ICU, which takes the name from that path's symlink target and ignores the file's contents, so glibc reports the host zone while Chrome still reports UTC. `/etc/timezone` carries the name and is mounted instead.
Chrome also binds its DevTools port to loopback and silently ignores `--remote-debugging-address`, which is why headless-shell fronted it with socat. This image does the same, so it stays a drop-in: the compose service keeps the `headless-shell` name and its pinned address, and `BROWSER_WS_URL` is unchanged.
## Verification
```
go test ./... all packages ok
node --test 37 + 12 pass, 0 fail
SMOKE_BROWSER_WS_URL=... go test -run TestSmokeKagane ./internal/latest
TestSmokeKaganeImage PASS (5.29s) fetched 56710 bytes of image/webp
TestSmokeKaganeGet PASS (1.17s) status=200, real chapter-list JSON
```
The smoke test ran against the exact compose configuration — built image, empty `BROWSER_TZ`, `/etc/timezone` mounted, cold profile — hitting real kagane.to. It skips unless `SMOKE_BROWSER_WS_URL` names a sidecar, so `go test ./...` stays hermetic and Docker-only.
A red smoke run means the challenge is not clearing from that IP, which is a live, time-varying fact to re-check rather than necessarily a defect.
## Security invariants
- Auth unchanged. `/img/kagane/{id}` is session-gated by `requireSession`, the same guard as every other UI route.
- Outbound fetch gated: the id is UUID-validated before it reaches the browser, keeping the existing rule that a client-supplied string never selects a fetch target unchecked.
- No new secrets, no new logging of credentials, no change to CORS, sessions, or crypto.
- Templates still escape everything; `.CoverURL` returns a plain string and is not wrapped in `template.HTML`/`URL`.
- One new dependency-free image (`chrome/`) built from Debian plus Google's own apt repo; no new Go modules.
## Deploying
Needs `docker compose build headless-shell`.
**A UTC host must set `BROWSER_TZ`, or kagane silently stops working.** With it unset the sidecar falls back to the host's `/etc/timezone`; on a UTC server that yields UTC, which is the one value that never clears. Any real zone works — `BROWSER_TZ=Asia/Jakarta` for the current deployment. `.env.example` now documents this; it previously did not mention the knob at all.
Only the browser sidecar reads `BROWSER_TZ`. The backend keeps its UTC clock, and stored timestamps are unix ms, so nothing else shifts.
## Deliberately not done
Retry/backoff around the cover proxy, and a panel-side cover fix. The panel renders no covers, and covers cache in-process after the first fetch. Worth adding if kagane starts rate-limiting.
## Correction after review of the deployment case
`1552dd1` was added after the branch was first pushed: the deployment host runs UTC with an Indonesian egress IP, which prompted re-measuring the timezone claim and falsifying it. The earlier commits' reasoning is left intact rather than rebased away, so the diagnostic trail — including the wrong turn and what disproved it — stays readable.
Reviewed-on: #37
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
Manga Bookmark
Track manga read-progress on asurascans.com (a.k.a. asuracomic.net), demonicscans.org, comix.to, and kagane.to from a phone (Bromite / mobile Chromium), synced to a self-hosted Go backend so bookmarks unify across all four sites and all devices.
Two parts:
backend/— tiny Go (net/http+ Postgres via pure-Gopgx) sync service. 4 routes, static binary, distroless container.userscript/manga-bookmark.user.js— single Bromite-compatible userscript (noGM_*APIs) that injects an on-page bookmark UI and syncs viafetch().
Bromite userscript (isolated world, Shadow DOM UI, localStorage cache)
-- fetch() HTTPS --> reverse proxy (TLS + CORS) --> Go net/http --> Postgres (volume)
1. Backend
Config (env)
| Var | Default | Notes |
|---|---|---|
TOKEN_KEY |
(required) | Secret every Reader's userscript credential is derived from (issue #24); only SHA-256 hashes of credentials are stored. |
OWNER_DISCORD_ID |
(required) | Discord user ID of the owner: seeded as the first Reader, owns every pre-registration bookmark, and is the only Reader who can revoke another's sessions. |
ALLOWED_ORIGINS |
Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
DATABASE_URL |
(required) | Postgres connection URL, e.g. postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable. Compose builds it from POSTGRES_PASSWORD. |
PORT |
8080 |
Plain HTTP; TLS terminated by the proxy. |
BROWSER_WS_URL |
ws://172.28.0.10:9222 |
Headless-shell CDP endpoint used to poll Kagane past its JS challenge. Must be an IP or localhost — Chrome's DevTools handler 500s any other Host header. |
DISCORD_CLIENT_ID |
(required) | Discord application credentials for the browser sign-in (ADR-0002). |
DISCORD_CLIENT_SECRET |
(required) | As above. Never logged, never echoed in an error. |
DISCORD_GUILD_ID |
(required) | The one guild whose membership gates sign-in, checked at login only. Membership is registration: any member becomes a Reader on first login. |
DISCORD_REDIRECT_URI |
(required) | Exact callback URL; Discord matches it verbatim against the registered redirect. |
DISCORD_REQUIRED_ROLE |
empty | Role snowflake a member must additionally hold. Empty means guild membership alone suffices. |
DISCORD_API_BASE |
https://discord.com/api/v10 |
Test seam — tests point it at a local stub so the real token exchange runs. |
USERSCRIPT_PATH |
/userscript/manga-bookmark.user.js |
Bindmounted file served at /u/{token}/manga-bookmark.user.js. |
NOVEL_USERSCRIPT_PATH |
/userscript/novel-bookmark.user.js |
Same, for the novel library. |
LATEST_CHAPTER_POLL_ENABLED |
1 |
0 turns the poller off entirely. |
LATEST_CHAPTER_POLL_COOLDOWN |
1h |
Rest between checks of one series; floor 15m. |
LATEST_CHAPTER_POLL_INTERVAL |
10m |
How often the poller wakes. Cannot shorten a cooldown. |
LATEST_CHAPTER_POLL_BATCH |
14 |
Series per wake. Keep BATCH × STAGGER under INTERVAL. |
LATEST_CHAPTER_POLL_STAGGER |
20s |
Delay between fetches in a batch — this is the outbound request rate. |
Compose reads a few more from the same .env that the backend never sees:
POSTGRES_PASSWORD (required — DATABASE_URL is built from it, and Postgres
only applies it while postgres-data is empty), BOOKMARK_API_HOST and
BOOKMARK_WEB_HOST (required by the prod override), and the optional
PROXY_NETWORK / TRAEFIK_ENTRYPOINT / TRAEFIK_CERTRESOLVER. Full commentary
is in .env.example; deployment order is DEPLOY.md.
Endpoints
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/bookmarks |
Bearer | All bookmarks of the acting Reader. |
PUT |
/bookmarks/{key} |
Bearer | Upsert one series; returns the row as stored. |
DELETE |
/bookmarks/{key} |
Bearer | Remove one. |
GET |
/healthz |
none | 200 ok. |
GET |
/u/{token}/manga-bookmark.user.js |
credential in path | Serves the userscript with the requesting Reader's credential substituted in and an mtime-derived @version. |
key is <site>:<series_id> — e.g. asura:trash-of-the-counts-family-f886a8af,
demonic:Infinite-Level-Up-in-Murim, comix:12345, or
kagane:3fa85f64-5717-4562-b3fc-2c963f66afa6. Sync is last-write-wins.
updated_at orders the bookmark list, so it moves only on real reading
progress: the server applies its timestamp when the row is new or
last_chapter_num changes, and otherwise keeps the stored one. Favouriting a
series or recording a newly published chapter therefore leaves the order alone.
Because the timestamp a client sends is only a candidate, PUT echoes the row
as stored and clients adopt that rather than their own payload.
Develop / test
cd backend
go test ./... # unit + handler tests
CGO_ENABLED=0 go build # static binary
go test ./... requires Docker. The store talks to a real Postgres, so
each test package starts a throwaway postgres:17-alpine container and gives
every test its own database inside it (internal/pgtest). Nothing is stubbed
and nothing reaches the network beyond the local Docker daemon.
Run the stack
cp .env.example .env
# edit .env: set TOKEN_KEY (openssl rand -hex 32) and
# POSTGRES_PASSWORD (openssl rand -hex 24)
docker compose up -d --build # binds 127.0.0.1:8080
Smoke test:
# The credential is per Reader and derived, so there is no token in .env to
# grep. Take yours from the Userscripts panel's install link after signing in,
# or read it out of an installed script's API_TOKEN constant.
TOKEN=<your Reader credential>
curl -s localhost:8080/healthz # ok
curl -s localhost:8080/bookmarks # 401
curl -s -H "Authorization: Bearer $TOKEN" localhost:8080/bookmarks # []
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"title":"Test","last_chapter":"Chapter 1","last_chapter_num":1}' \
localhost:8080/bookmarks/asura:test-1
curl -s -i -X OPTIONS -H 'Origin: https://asurascans.com' \
-H 'Access-Control-Request-Method: PUT' \
localhost:8080/bookmarks/asura:test-1 | grep -i access-control # 204 + CORS headers
Deploy behind your reverse proxy
Route https://bookmark-api.<domain> → the service on :8080 (TLS at the proxy).
-
Host proxy (nginx/Caddy on the host): the base compose already binds
127.0.0.1:8080; point the proxyproxy_pass http://127.0.0.1:8080;. -
Docker proxy (Traefik/nginx in a container on its own network): use the override, which drops the published port and joins the shared network:
docker network create proxy # once, if it doesn't exist docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --buildSet
PROXY_NETWORKin.envif your network isn't namedproxy.
Verify: https://bookmark-api.<domain>/healthz returns ok over valid TLS (no
mixed-content), and an OPTIONS preflight from a real site origin returns the
CORS headers.
2. Userscript
Install
Sign in to the web UI and open the Userscripts panel: it offers one
install link per library. Each link serves a script rendered with your own
credential already inside it — you never see, type or copy a credential. The
served script carries @downloadURL/@updateURL pointing at its
credential-bearing path, so Violentmonkey keeps auto-updating it.
The bindmounted files carry __API_TOKEN__ placeholders; the backend
substitutes the requesting Reader's credential at serve time, so no real
credential is ever committed. Rotating the credential (same panel) invalidates
every installed copy immediately — reinstall on all devices.
Install on Bromite (mobile)
Bromite runs Chromium's native userscript engine (no Tampermonkey needed):
- Bromite → Settings → User scripts → enable user scripts (allow the permission prompt).
- Open the install link from the web UI — Bromite detects the
.user.jsand offers to install it. - Confirm the install; the
@matchlist covers both sites. - Open a series on either site — a 📑 button appears bottom-right.
Exact menu wording varies by Bromite build; if "User scripts" is absent, update Bromite or use a build with userscript support.
Desktop iteration (optional)
The script is GM_*-free, so it also runs in Tampermonkey/Violentmonkey on
desktop for faster testing — install the same file unchanged.
Use
- Bookmark: on a series or chapter page, open the panel → + Bookmark this.
- Auto-progress: opening a chapter of a bookmarked series records it when the chapter number ≥ the stored one (re-reading older chapters never regresses progress; unparseable numbers set the current chapter).
- Manual override: panel → Edit on any row forces a specific chapter.
- Continue: jumps to the last-read chapter (or the series page).
- Latest chapter: rows read
Read: … · Latest: …once the newest published chapter is known and it is ahead of your progress. See below for how that is found. - Favourites: the ☆ on any row toggles it; the ★ Favourites tab narrows the list. Favourited series still appear under All. The flag syncs, so it follows you across devices; the chosen tab does not persist.
- Archive: the Archive button on any row parks a series — it drops out of All and ★ Favourites and moves to the Archived tab. The server keeps checking it for new chapters, so it is worth coming back to. Archiving does not touch read progress, and reading an archived series leaves it archived.
- Finished: series you have completed live in a Finished tab in the web UI only. It is set there and nowhere else — the API rejects the value — and finished series are hidden from every userscript tab and are no longer polled for new chapters.
- Bookmarks made on Asura appear when the panel is opened on Demonic, and vice versa — the backend is the shared store.
Neither favouriting nor learning a new chapter reorders the list — only reading progress does.
Offline / backend down: changes are cached in localStorage and retried on the
next successful load (last-write-wins).
How "latest chapter" is found
Only a series page lists every chapter (a reader page links just its neighbours), and the backend cannot fetch either site — Cloudflare blocks server-side requests, and neither site offers an API or feed to poll. So the userscript does the looking, from your own browser session:
- Opening a bookmarked series page records its newest chapter directly.
- Otherwise it fetches series pages in the background — same-origin only, so
browsing Asura refreshes Asura bookmarks and Demonic refreshes Demonic. One
series per navigation, and at most one check per series every 4 hours
(
LATEST_CHECK_BATCH/LATEST_CHECK_THROTTLE_MS). Failures are silent and simply retried after the window.
Freshness is tracked per device in localStorage under bmgr:manga:lastchecked
and is deliberately not synced, since each device checks on its own.
This means a bookmark is as current as its last check — not the moment a chapter drops. Nothing can be instant here: neither site offers push, feeds, or an API.
Adapter reference (verified live 2026-07-24)
The site adapters key everything off URL regex, with title/cover from
og:title / og:image. Confirmed against live pages via Playwright:
| Site | Series URL | Chapter URL | series_id |
|---|---|---|---|
Asura (asurascans.com) |
/comics/<slug-hash> |
/comics/<slug-hash>/chapter/<n> |
<slug-hash> |
Demonic (demonicscans.org) |
/manga/<slug> |
/title/<slug>/chapter/<n>/<page> (chaptered.php?manga=<id>&chapter=<n> 301s here) |
<slug> |
Comix (comix.to) |
/title/<id>-<slug> |
/title/<id>-<slug>/<uploadId>-chapter-<n> |
<id> |
Kagane (kagane.to) |
/series/<uuid> |
/series/<uuid>/reader/<bookUuid> |
<uuid> |
Notes:
asuracomic.netdeep links are dead (re-checked 2026-07-25). They 301 to theasurascans.comroot, discarding the path, at the edge — before the userscript gets a document — so nothing client-side can rescue them. Reach series throughasurascans.com. The host stays matched in case the redirect starts preserving paths again.- Asura
og:titlecarries aChapter N - Read Online \| Asura Scanssuffix that the adapter strips; Demonic chapterog:titleis<Title> Chapter N. - Demonic's
<slug>is identical on/manga/…and the canonical/title/…reader, so a bookmark set from the series page and the auto-update from the reader resolve to the same key. - Asura showed no Next.js markers on the live site, so navigation uses a framework-agnostic watcher (history patch + polling) rather than a Next-only hook — works for client-routed and full-reload sites alike.
If either site changes its URL shape, update the regex in the matching adapter
in userscript/manga-bookmark.user.js and re-verify.