feat: wire headless-shell sidecar for challenge-gated polling
Also bumps backend/Dockerfile's build stage to golang:1.26-alpine — chromedp v0.16.0 and cdproto both require go 1.26, and the pinned 1.24-alpine base no longer builds the module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -52,3 +52,7 @@ WEB_PASSWORD=
|
|||||||
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
|
# BATCH x (COOLDOWN / INTERVAL) series hold the cooldown cadence — 84 with these
|
||||||
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
|
# defaults. Beyond that the cadence stretches uniformly rather than breaking;
|
||||||
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
|
# raise BATCH or lower INTERVAL. Keep BATCH x STAGGER under INTERVAL.
|
||||||
|
|
||||||
|
# Headless-shell CDP endpoint for sites behind a JavaScript challenge (kagane).
|
||||||
|
# Unset disables browser polling; those sites then rely on the userscript alone.
|
||||||
|
BROWSER_WS_URL=ws://headless-shell:9222
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
|
|||||||
packages together into `newRouter`. Root-level `*_test.go` hold
|
packages together into `newRouter`. Root-level `*_test.go` hold
|
||||||
integration tests that exercise the full router; unit tests for a
|
integration tests that exercise the full router; unit tests for a
|
||||||
package live beside it under `internal/`.
|
package live beside it under `internal/`.
|
||||||
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`). Sync **last-write-wins**. Schema and endpoint list in plan.
|
- **Single-user store.** One `bookmarks` table keyed `<site>:<series_id>` (`asura`|`demonic`|`comix`|`kagane`). Sync **last-write-wins**. Schema and endpoint list in plan.
|
||||||
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
- **Endpoints:** `GET /bookmarks`, `PUT /bookmarks/{key}` (upsert; see `updated_at` rule below), `DELETE /bookmarks/{key}`, `GET /healthz` (no auth).
|
||||||
- **Web UI:** same binary serve password-gated browser UI on second
|
- **Web UI:** same binary serve password-gated browser UI on second
|
||||||
hostname — `GET /` (list, or login page when no session),
|
hostname — `GET /` (list, or login page when no session),
|
||||||
@@ -75,7 +75,10 @@ Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
|
|||||||
cooldown instead of retrying every tick, and writes go through
|
cooldown instead of retrying every tick, and writes go through
|
||||||
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
|
`Store.Get` + `Store.Upsert` so new chapter never reorders list.
|
||||||
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
Fetches use `bogdanfinn/tls-client` with Chrome profile as defence in depth
|
||||||
against fingerprint-based blocking; any failure log and skip. See
|
against fingerprint-based blocking; any failure log and skip. kagane sits
|
||||||
|
behind a Cloudflare JavaScript challenge the TLS client can't clear, so it is
|
||||||
|
browser-only: fetched over CDP via `BROWSER_WS_URL`, and simply not polled
|
||||||
|
when that's unset. See
|
||||||
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
`docs/superpowers/specs/2026-07-26-server-latest-chapter-polling-design.md`.
|
||||||
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
|
Poller's `Store.Get` + `Store.Upsert` not wrapped in transaction, so
|
||||||
userscript `PUT` that commits between the two can get overwritten by
|
userscript `PUT` that commits between the two can get overwritten by
|
||||||
@@ -100,6 +103,8 @@ Violentmonkey userscript (isolated world, per-site adapters, localStorage cache)
|
|||||||
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
(background latest-chapter poller; defaults on, `1h`/`10m`/`14`/`20s`).
|
||||||
`USERSCRIPT_PATH` (file served at `/u/{token}/manga-bookmark.user.js`,
|
`USERSCRIPT_PATH` (file served at `/u/{token}/manga-bookmark.user.js`,
|
||||||
default `/userscript/manga-bookmark.user.js`, supplied by bindmount).
|
default `/userscript/manga-bookmark.user.js`, supplied by bindmount).
|
||||||
|
`BROWSER_WS_URL` (headless-shell CDP endpoint for kagane; unset disables
|
||||||
|
browser polling and leaves that site to the userscript alone).
|
||||||
|
|
||||||
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
### Userscript structure (single IIFE, `manga-bookmark.user.js`)
|
||||||
|
|
||||||
|
|||||||
@@ -116,6 +116,12 @@ This merges the base file (build/image/env/volume) with the prod override
|
|||||||
(no host port, Traefik network + router labels). Always pass **both** `-f`
|
(no host port, Traefik network + router labels). Always pass **both** `-f`
|
||||||
flags — the prod file is not standalone.
|
flags — the prod file is not standalone.
|
||||||
|
|
||||||
|
Two services come up: `manga-api` (the backend) and `headless-shell`, a CDP
|
||||||
|
sidecar the poller uses to fetch kagane (behind a Cloudflare JS challenge).
|
||||||
|
It has no published port — only `manga-api` can reach it, over
|
||||||
|
`BROWSER_WS_URL`. Missing or unreachable, the poller just skips kagane and
|
||||||
|
logs it; nothing else is affected.
|
||||||
|
|
||||||
Check it's up and healthy:
|
Check it's up and healthy:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
# Manga Bookmark
|
# Manga Bookmark
|
||||||
|
|
||||||
Track manga read-progress on **asurascans.com** (a.k.a. asuracomic.net) and
|
Track manga read-progress on **asurascans.com** (a.k.a. asuracomic.net),
|
||||||
**demonicscans.org** from a phone (Bromite / mobile Chromium), synced to a
|
**demonicscans.org**, **comix.to**, and **kagane.to** from a phone (Bromite /
|
||||||
self-hosted Go backend so bookmarks unify across both sites and all devices.
|
mobile Chromium), synced to a self-hosted Go backend so bookmarks unify across
|
||||||
|
all four sites and all devices.
|
||||||
|
|
||||||
Two parts:
|
Two parts:
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1,7 +1,7 @@
|
|||||||
# syntax=docker/dockerfile:1
|
# syntax=docker/dockerfile:1
|
||||||
|
|
||||||
# --- build stage: compile a static, CGO-free binary ---
|
# --- build stage: compile a static, CGO-free binary ---
|
||||||
FROM golang:1.24-alpine AS build
|
FROM golang:1.26-alpine AS build
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# Dependencies first for layer caching (changes rarely).
|
# Dependencies first for layer caching (changes rarely).
|
||||||
|
|||||||
@@ -274,5 +274,20 @@ func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll) {
|
|||||||
Stagger: cfg.Stagger,
|
Stagger: cfg.Stagger,
|
||||||
Batch: cfg.Batch,
|
Batch: cfg.Batch,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Optional: without it, sites behind a JavaScript challenge are simply not
|
||||||
|
// polled, and their latest_chapter comes from the userscript alone — which
|
||||||
|
// is how the service behaved before the sidecar existed.
|
||||||
|
if ws := strings.TrimSpace(os.Getenv("BROWSER_WS_URL")); ws != "" {
|
||||||
|
bf, err := latest.NewBrowserFetcher(ws)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("latest-chapter poller: browser fetcher disabled: %v", err)
|
||||||
|
} else {
|
||||||
|
p.BrowserFetch = bf
|
||||||
|
context.AfterFunc(ctx, bf.Close)
|
||||||
|
log.Printf("latest-chapter poller: browser fetcher at %s", ws)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
go p.Run(ctx)
|
go p.Run(ctx)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,6 +17,10 @@ services:
|
|||||||
manga-api:
|
manga-api:
|
||||||
# Traffic arrives over the Traefik network, not a published port.
|
# Traffic arrives over the Traefik network, not a published port.
|
||||||
ports: !reset []
|
ports: !reset []
|
||||||
|
environment:
|
||||||
|
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://headless-shell:9222}
|
||||||
|
depends_on:
|
||||||
|
- headless-shell
|
||||||
networks:
|
networks:
|
||||||
- proxy
|
- proxy
|
||||||
labels:
|
labels:
|
||||||
@@ -36,6 +40,12 @@ services:
|
|||||||
- "traefik.http.routers.mangaweb.tls.certresolver=${TRAEFIK_CERTRESOLVER:-le}"
|
- "traefik.http.routers.mangaweb.tls.certresolver=${TRAEFIK_CERTRESOLVER:-le}"
|
||||||
- "traefik.http.routers.mangaweb.service=mangabm"
|
- "traefik.http.routers.mangaweb.service=mangabm"
|
||||||
|
|
||||||
|
# manga-api only joins `proxy` in this override (see above), so headless-shell
|
||||||
|
# has to follow it there too or DNS resolution of headless-shell:9222 breaks.
|
||||||
|
headless-shell:
|
||||||
|
networks:
|
||||||
|
- proxy
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
proxy:
|
proxy:
|
||||||
external: true
|
external: true
|
||||||
|
|||||||
@@ -29,6 +29,11 @@ services:
|
|||||||
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
|
LATEST_CHAPTER_POLL_INTERVAL: ${LATEST_CHAPTER_POLL_INTERVAL:-10m}
|
||||||
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
|
LATEST_CHAPTER_POLL_BATCH: ${LATEST_CHAPTER_POLL_BATCH:-14}
|
||||||
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
|
LATEST_CHAPTER_POLL_STAGGER: ${LATEST_CHAPTER_POLL_STAGGER:-20s}
|
||||||
|
# CDP endpoint for sites behind a JavaScript challenge (kagane). Unset
|
||||||
|
# disables browser polling for those sites; the userscript still covers them.
|
||||||
|
BROWSER_WS_URL: ${BROWSER_WS_URL:-ws://headless-shell:9222}
|
||||||
|
depends_on:
|
||||||
|
- headless-shell
|
||||||
volumes:
|
volumes:
|
||||||
- bookmarks-data:/data
|
- bookmarks-data:/data
|
||||||
# The userscript is served from here, read fresh on every request. Editing
|
# The userscript is served from here, read fresh on every request. Editing
|
||||||
@@ -41,5 +46,21 @@ services:
|
|||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:8080:8080"
|
- "127.0.0.1:8080:8080"
|
||||||
|
|
||||||
|
headless-shell:
|
||||||
|
image: chromedp/headless-shell:stable
|
||||||
|
restart: unless-stopped
|
||||||
|
# Chrome allocates shared memory per tab and dies on Docker's 64MB default.
|
||||||
|
shm_size: '1gb'
|
||||||
|
# Reaps zombie renderer processes, which otherwise accumulate for the
|
||||||
|
# container's lifetime.
|
||||||
|
init: true
|
||||||
|
# Deliberately no `ports:` — an exposed CDP endpoint is remote code
|
||||||
|
# execution. Only the backend on the internal network may reach it.
|
||||||
|
command:
|
||||||
|
- --remote-debugging-address=0.0.0.0
|
||||||
|
- --remote-debugging-port=9222
|
||||||
|
- --disable-gpu
|
||||||
|
- --no-sandbox
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
bookmarks-data:
|
bookmarks-data:
|
||||||
|
|||||||
Reference in New Issue
Block a user