Add comix.to and kagane.to support (#13)

Tracks read progress on comix.to and kagane.to alongside asura and demonic, in both the userscript and the backend. Implements `docs/superpowers/plans/2026-08-03-comix-kagane-support.md`. ## Userscript - `comix` adapter — `/title/<id>-<slug>`; only the id prefix is identity (the slug follows the title). No `og:image`, so the cover is matched by `alt`. - `kagane` adapter — reader URLs are uuids with no chapter number, so it comes out of `og:title`; anchor scanning is structurally impossible, replaced by `latestChapterFromApi` against kagane's same-origin JSON API. - `seriesId` threaded through `latestChapterFromAnchors` so comix can scope its scan to its own series and a recommendation strip cannot win the maximum. - `@match` for both hosts, panel chips, v1.6.0. ## Backend - `latestChapterFrom` cases: comix parses the SSR JSON state blob (`latestChapterUrl`, scoped to the series id); kagane parses API JSON (`chapter_no`). - Poller allowlist extended; `Poller.BrowserFetch` with `fetcherFor(site)` routes kagane to a browser fetcher. Nil means kagane is not polled at all — never a fallback to the TLS fetcher, which would only ever retrieve a challenge page. - `BrowserFetcher`: chromedp against a `headless-shell` sidecar. kagane sits behind a Cloudflare JS challenge that no TLS fingerprint clears, and the request is made inside the page rather than by replaying `cf_clearance`. - `BROWSER_WS_URL` wiring, sidecar in both compose files (no `ports:`, dedicated non-external network), Dockerfile on `golang:1.26-alpine` — chromedp requires go 1.26. - Web UI `--comix` / `--kagane` tokens in both colour branches. ## Notes for review - `series_url` is client-supplied and a headless browser is a strong SSRF primitive, so kagane's host is pinned twice: in `fetchableSeriesURL` and again in `kaganeAPIURL`. - Three chained defects found during verification made the browser path dead under Compose (sidecar flag collision, Chrome's Host-header DNS-rebinding check, the wrong chromedp option). Fixed; the compose comments record the wrong configurations too, so they don't get "simplified" back. - `ALLOWED_ORIGINS` now includes both new origins. Without it every write from comix/kagane silently fails CORS preflight, parks in the retry queue, and drops at the cap. ## Verification 221 backend tests, 32 userscript tests, static `CGO_ENABLED=0` build, both compose configs. Two gaps, both real: 1. The userscript on live pages via Violentmonkey needs a human browser profile — not run. Check: comix series page (title/cover, no chapter), comix chapter page (records the number; an *older* chapter must not regress it), comix SPA navigation without reload, kagane series page (og:image cover), kagane reader (number from `og:title`), both chips opening the right sites. 2. The kagane browser path has not completed end-to-end anywhere. Dial/navigate/fetch is confirmed, but Cloudflare 403'd headless-shell's Chrome on every attempt from the dev sandbox, and comix's poll-through-Docker was blocked by that environment's TLS interception. Both environment-dependent rather than branch defects — the first real deploy is the actual verification. Reviewed-on: #13 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-03 19:53:45 +07:00
parent 3c935ba7c3
commit 180ee78b1f
49 changed files with 2233 additions and 1027 deletions
@@ -0,0 +1,236 @@
+package latest
+
+import (
+	"context"
+	"log"
+	"net/url"
+	"time"
+
+	"mangabm/backend/internal/store"
+)
+
+// Fetcher retrieves a series page. It exists as an interface so tests can inject
+// a fake: nothing in the test suite may touch the network or the TLS client.
+type Fetcher interface {
+	Get(ctx context.Context, url string) (body string, status int, err error)
+}
+
+// Poller re-checks each bookmarked series' newest published chapter on a
+// schedule, independent of the userscript's own in-browser checks. The two run
+// in parallel and report the same observable fact, so whichever writes last wins
+// and neither needs to know about the other.
+//
+// Two clocks, deliberately independent:
+//
+//   - Interval is how often this goroutine wakes up and looks.
+//   - Cooldown is how long one bookmark rests since its own last check.
+//
+// Only the cooldown is per bookmark, and it is enforced by the WHERE clause in
+// DueForLatestCheck rather than by any timer. Shortening Interval therefore
+// cannot shorten anyone's cooldown; it only makes the poller wake up and find
+// nothing due more often.
+type Poller struct {
+	Store    *store.Store
+	Fetch    Fetcher
+	// BrowserFetch handles sites behind a JavaScript challenge that Fetch
+	// cannot clear. Nil disables those sites entirely rather than falling back
+	// to Fetch, which would only ever retrieve a challenge page.
+	BrowserFetch Fetcher
+	Now          func() time.Time // injected so tests can freeze it
+	Cooldown     time.Duration
+	Interval     time.Duration
+	Stagger      time.Duration
+	Batch        int
+}
+
+// fetcherFor returns the fetcher a site needs, or nil when the site cannot be
+// fetched at all right now. kagane sits behind a Cloudflare JavaScript
+// challenge that no TLS fingerprint clears — verified 2026-08-03 from the
+// deployment host with the same Chrome profile TLSFetcher uses — so it is
+// browser-only or nothing.
+func (p *Poller) fetcherFor(site string) Fetcher {
+	if site == "kagane" {
+		return p.BrowserFetch
+	}
+	return p.Fetch
+}
+
+// Run polls until ctx is cancelled.
+//
+// runOnce is called synchronously, so a batch that overruns the tick delays the
+// next one instead of stacking a second batch on top of it. That is the intended
+// failure mode for a misconfigured batch x stagger: a slower cadence, never
+// concurrent fetch storms.
+func (p *Poller) Run(ctx context.Context) {
+	log.Printf("latest-chapter poller: interval=%s cooldown=%s batch=%d stagger=%s",
+		p.Interval, p.Cooldown, p.Batch, p.Stagger)
+	t := time.NewTicker(p.Interval)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			log.Println("latest-chapter poller: stopped")
+			return
+		case <-t.C:
+			p.runOnce(ctx)
+		}
+	}
+}
+
+// runOnce processes one batch of due bookmarks.
+func (p *Poller) runOnce(ctx context.Context) {
+	cutoff := p.Now().Add(-p.Cooldown).UnixMilli()
+	due, err := p.Store.DueForLatestCheck(cutoff, p.Batch)
+	if err != nil {
+		log.Printf("latest poll: due query: %v", err)
+		return
+	}
+
+	checked := 0
+	for i, b := range due {
+		if ctx.Err() != nil {
+			break
+		}
+		// Staggered rather than fired together: a burst of simultaneous requests
+		// from one server IP is the traffic shape most likely to move that IP's
+		// bot score. This is the server-side analogue of the userscript's "one
+		// series per navigation ... indistinguishable from browsing" (L455-456).
+		stopped := false
+		if i > 0 && p.Stagger > 0 {
+			select {
+			case <-ctx.Done():
+				stopped = true
+			case <-time.After(p.Stagger):
+			}
+		}
+		if stopped {
+			break
+		}
+		p.checkOne(ctx, b)
+		checked++
+	}
+	// due vs checked is how you tell which constraint is binding: ticks that
+	// report due=0 mean the cooldown is the limit, ticks that report due==batch
+	// every time mean throughput is.
+	log.Printf("latest poll: due=%d checked=%d", len(due), checked)
+}
+
+// checkOne re-checks one series. Every failure path here is "log and move on":
+// the poller is a best-effort enhancement, and no single bad series may stall a
+// batch or take down the process.
+func (p *Poller) checkOne(ctx context.Context, b store.Bookmark) {
+	defer func() {
+		if r := recover(); r != nil {
+			log.Printf("latest poll %q: recovered from panic: %v", b.Key, r)
+		}
+	}()
+
+	// Stamped before the fetch, not after, so an error, a timeout, or a shutdown
+	// mid-request still consumes the cooldown. Otherwise a renamed or deleted
+	// series would be retried on every single tick forever. The userscript
+	// stamps in the same order and for the same reason (L471-473).
+	if err := p.Store.MarkLatestChecked(b.Key, p.Now().UnixMilli()); err != nil {
+		log.Printf("latest poll %q: mark checked: %v", b.Key, err)
+		return
+	}
+
+	// series_url is client-supplied (PUT /bookmarks/{key} accepts any string),
+	// so this is not just an optimisation against burning a request on an
+	// unknown site: without it, the server would issue a GET from its own
+	// network position to whatever URL a token-holder writes, including
+	// link-local/internal addresses or non-https schemes. The cooldown above
+	// is already consumed, so a row that never passes this check is retried at
+	// cooldown pace rather than hot-looping.
+	if !fetchableSeriesURL(b.Site, b.SeriesURL) {
+		log.Printf("latest poll %q: not fetchable: site=%q url=%q", b.Key, b.Site, b.SeriesURL)
+		return
+	}
+
+	f := p.fetcherFor(b.Site)
+	if f == nil {
+		log.Printf("latest poll %q: no fetcher for site %q", b.Key, b.Site)
+		return
+	}
+
+	body, status, err := f.Get(ctx, b.SeriesURL)
+	if err != nil {
+		log.Printf("latest poll %q: fetch %s: %v", b.Key, b.SeriesURL, err)
+		return
+	}
+	if status != 200 {
+		log.Printf("latest poll %q: fetch %s: status %d", b.Key, b.SeriesURL, status)
+		return
+	}
+
+	latest, ok := latestChapterFrom(b.Site, b.SeriesURL, body)
+	if !ok {
+		// Most likely a challenge page or a layout change. Either way the row is
+		// already stamped, so this waits out a cooldown instead of hot-looping.
+		log.Printf("latest poll %q: no chapter links in %d bytes", b.Key, len(body))
+		return
+	}
+
+	// Re-read: the row may have been updated or deleted while the fetch was in
+	// flight, and writing b back wholesale would undo that.
+	//
+	// ponytail: non-transactional read-modify-write, wrap Get+Upsert in a tx if
+	// this ever runs for more than one user. A client PUT that commits between
+	// these two statements is lost to the stale re-read — reverting read
+	// progress or a status change, and moving updated_at because the stored
+	// value now differs. Accepted for a single-user deployment: the window is
+	// milliseconds and the loser is one poll cycle.
+	cur, found, err := p.Store.Get(b.Key)
+	if err != nil {
+		log.Printf("latest poll %q: reread: %v", b.Key, err)
+		return
+	}
+	if !found {
+		return
+	}
+	// Equality, not >, mirroring the userscript (L427): a site that retracts a
+	// chapter should correct the stored number downward.
+	if cur.LatestChapterNum != nil && *cur.LatestChapterNum == latest.Num {
+		return
+	}
+
+	num := latest.Num
+	cur.LatestChapter = latest.Label
+	cur.LatestChapterNum = &num
+	// A candidate only. last_chapter_num is untouched, so the CASE in Upsert
+	// keeps the stored updated_at and the bookmark list does not reorder.
+	cur.UpdatedAt = p.Now().UnixMilli()
+	if _, err := p.Store.Upsert(cur); err != nil {
+		log.Printf("latest poll %q: upsert: %v", b.Key, err)
+		return
+	}
+	log.Printf("latest poll %q: latest is now %s", b.Key, latest.Label)
+}
+
+// fetchableSeriesURL reports whether site is a site latestChapterFrom knows how
+// to parse and seriesURL is safe to hand to a fetcher: an https URL with a
+// non-empty host. series_url comes from client-supplied PUT bodies, so this is
+// a defence against the poller being used to probe arbitrary hosts from the
+// server's own network position, not just a check against wasted requests.
+//
+// kagane is held to a stricter rule: it is fetched by a headless browser, which
+// executes JavaScript and carries cookies, and is therefore a far stronger SSRF
+// primitive than an HTTP GET. Its host must match exactly, not merely be
+// non-empty.
+func fetchableSeriesURL(site, seriesURL string) bool {
+	switch site {
+	case "asura", "demonic", "comix", "kagane":
+	default:
+		return false
+	}
+	u, err := url.Parse(seriesURL)
+	if err != nil {
+		return false
+	}
+	if u.Scheme != "https" || u.Host == "" {
+		return false
+	}
+	if site == "kagane" {
+		return u.Hostname() == "kagane.to"
+	}
+	return true
+}