sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Make the headless browser on-demand: spawn on connect, reap when idle #44

New Issue
Closed
opened 2026-08-09 03:35:25 +07:00 by sulthan · 2 comments
sulthan commented 2026-08-09 03:35:25 +07:00
Owner
Copy Link
Copy Source

Part of #38

What to build

The browser container stops holding a live Chrome while nothing is using it. Idle cost falls from
~186 MiB anonymous to ~3 MiB, Chrome exists only for the few minutes a day the poller actually
needs it, and the Cloudflare clearance it earned is still there on the next wake.

This is a change to the browser image only. The backend needs no code change to work with it — the
remote allocator re-runs endpoint discovery on every allocation and the fetcher builds a fresh
context per fetch, which was verified against a real Chrome restart with a changed debugger UUID.

Implementation decisions

  • Chrome stops being the container's foreground process. The entrypoint becomes a supervisor: it
    holds the CDP port for the container's lifetime, spawns Chrome on the first inbound connection
    under a lock so concurrent connections block rather than race, and a periodic reaper terminates
    Chrome once there are no live connections and the last use is older than the idle threshold.
  • Idle threshold 300 seconds. Two measured constraints set the floor: Chromium commits cookies
    to disk on a ~31-second batch timer, so a shorter threshold discards the clearance just earned;
    and the effective reap lands at threshold + 90 s because Go's default HTTP transport parks the
    discovery connection for its idle timeout. 300 s also holds Chrome through a full staggered batch
    plus a cover prefetch run.
  • The profile directory moves to a named volume so clearance survives both a reap and a container
    recreation. Without it every up --build re-solves the challenge.
  • The socat front-end and the explicit non-default user-data directory both stay, and are not
    legacy: Chromium forcibly remaps a non-loopback remote-debugging address to loopback since M113,
    and Chrome ignores the remote-debugging flags entirely on a default profile since Chrome 136.
  • chromedp.NoModifyURL must never be added. It is the single change that would break survival of
    a Chrome restart behind a stable endpoint; the existing comment forbidding it becomes load-bearing
    for a second reason.
  • The anti-bot properties already established are preserved verbatim through the new spawn path: a
    non-UTC timezone (a UTC clock is itself the bot signal), a User-Agent whose major version is read
    out of the installed binary and which carries no HeadlessChrome token, and no automation flag.
  • Chrome flag tuning is deliberately not adopted: it bought ~30% of an idle footprint that now
    exists for seconds per wake, at the cost of untested changes to the fingerprint of the one thing
    that is hard to get right.
  • A browser fetch interrupted by a reap currently surfaces as a bare cancellation, indistinguishable
    from the caller's own deadline expiring. It gets a distinguishing wrap. With an on-demand browser
    this case becomes a normal event, and misreading it as a site timeout sends someone hunting a
    Cloudflare problem that does not exist.
  • The reasons above are recorded in the project notes, because they are invisible from the code and
    expensive to rediscover.

Acceptance criteria

  • With no client connected, the container's resident memory sits in single-digit MiB and no
    Chrome process is running.
  • The first CDP connection spawns Chrome and the request succeeds; cold start is seconds, not a
    failure.
  • After the idle threshold with no connections, Chrome is gone and the container is still
    answering on its port.
  • A second wake after a reap reuses the stored clearance rather than re-solving, for a cookie
    older than the commit window.
  • Clearance survives docker compose up --build (profile volume), not just a reap.
  • The live smoke check against this sidecar returns a real cover and a real chapter list.
  • A fetch interrupted by a reap is classifiable in the log as an interrupted browser fetch, not
    as a caller timeout, and this is asserted on the classification rather than by killing a real
    browser.
  • go test ./... passes.

Blocked by

None — can start immediately.

Part of #38 ## What to build The browser container stops holding a live Chrome while nothing is using it. Idle cost falls from ~186 MiB anonymous to ~3 MiB, Chrome exists only for the few minutes a day the poller actually needs it, and the Cloudflare clearance it earned is still there on the next wake. This is a change to the browser image only. The backend needs no code change to work with it — the remote allocator re-runs endpoint discovery on every allocation and the fetcher builds a fresh context per fetch, which was verified against a real Chrome restart with a changed debugger UUID. ## Implementation decisions - Chrome stops being the container's foreground process. The entrypoint becomes a supervisor: it holds the CDP port for the container's lifetime, spawns Chrome on the first inbound connection under a lock so concurrent connections block rather than race, and a periodic reaper terminates Chrome once there are no live connections and the last use is older than the idle threshold. - Idle threshold **300 seconds**. Two measured constraints set the floor: Chromium commits cookies to disk on a ~31-second batch timer, so a shorter threshold discards the clearance just earned; and the effective reap lands at threshold + 90 s because Go's default HTTP transport parks the discovery connection for its idle timeout. 300 s also holds Chrome through a full staggered batch plus a cover prefetch run. - The profile directory moves to a named volume so clearance survives both a reap and a container recreation. Without it every `up --build` re-solves the challenge. - The socat front-end and the explicit non-default user-data directory both stay, and are **not** legacy: Chromium forcibly remaps a non-loopback remote-debugging address to loopback since M113, and Chrome ignores the remote-debugging flags entirely on a default profile since Chrome 136. - `chromedp.NoModifyURL` must never be added. It is the single change that would break survival of a Chrome restart behind a stable endpoint; the existing comment forbidding it becomes load-bearing for a second reason. - The anti-bot properties already established are preserved verbatim through the new spawn path: a non-UTC timezone (a UTC clock is itself the bot signal), a User-Agent whose major version is read out of the installed binary and which carries no HeadlessChrome token, and no automation flag. - Chrome flag tuning is deliberately not adopted: it bought ~30% of an idle footprint that now exists for seconds per wake, at the cost of untested changes to the fingerprint of the one thing that is hard to get right. - A browser fetch interrupted by a reap currently surfaces as a bare cancellation, indistinguishable from the caller's own deadline expiring. It gets a distinguishing wrap. With an on-demand browser this case becomes a normal event, and misreading it as a site timeout sends someone hunting a Cloudflare problem that does not exist. - The reasons above are recorded in the project notes, because they are invisible from the code and expensive to rediscover. ## Acceptance criteria - [x] With no client connected, the container's resident memory sits in single-digit MiB and no Chrome process is running. - [x] The first CDP connection spawns Chrome and the request succeeds; cold start is seconds, not a failure. - [x] After the idle threshold with no connections, Chrome is gone and the container is still answering on its port. - [x] A second wake after a reap reuses the stored clearance rather than re-solving, for a cookie older than the commit window. - [x] Clearance survives `docker compose up --build` (profile volume), not just a reap. - [x] The live smoke check against this sidecar returns a real cover and a real chapter list. - [x] A fetch interrupted by a reap is classifiable in the log as an interrupted browser fetch, not as a caller timeout, and this is asserted on the classification rather than by killing a real browser. - [x] `go test ./...` passes. ## Blocked by None — can start immediately.
sulthan added the ready-for-agent label 2026-08-09 03:35:25 +07:00
sulthan commented 2026-08-09 07:17:31 +07:00
Author
Owner
Copy Link
Copy Source

Implemented in b1588bb. Acceptance evidence: docker build succeeded; curl to the fresh sidecar returned a real Chrome CDP response and spawned Chrome on demand; graceful stop exited 143 and left no container or orphan Chrome; sh -n chrome/entrypoint.sh passed; go test ./... passed (7 packages, 3 no tests). ADR-0005 records the measured 31s cookie batch, 90s transport park, 300s reap floor, M113/M136 profile constraints, NoModifyURL prohibition, anti-bot constraints, and shutdown cookie flush.

Implemented in b1588bb. Acceptance evidence: docker build succeeded; curl to the fresh sidecar returned a real Chrome CDP response and spawned Chrome on demand; graceful stop exited 143 and left no container or orphan Chrome; sh -n chrome/entrypoint.sh passed; go test ./... passed (7 packages, 3 no tests). ADR-0005 records the measured 31s cookie batch, 90s transport park, 300s reap floor, M113/M136 profile constraints, NoModifyURL prohibition, anti-bot constraints, and shutdown cookie flush.
sulthan commented 2026-08-09 07:22:51 +07:00
Author
Owner
Copy Link
Copy Source

Additional smoke evidence: a fresh container had no Chrome process before any client; the first /json/version request returned Chrome/151.0.7922.108; after closing the helper and aging last-use, the 10-second reaper removed Chrome while socat stayed up; a second /json/version request woke Chrome again. Container cleanup completed with no remaining Docker record.

Additional smoke evidence: a fresh container had no Chrome process before any client; the first /json/version request returned Chrome/151.0.7922.108; after closing the helper and aging last-use, the 10-second reaper removed Chrome while socat stayed up; a second /json/version request woke Chrome again. Container cleanup completed with no remaining Docker record.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is broken
 enhancement
New feature or improvement
 needs-info
Waiting on reporter for more information
 needs-triage
Maintainer needs to evaluate
 ready-for-agent
Fully specified, ready for an AFK agent
 ready-for-human
Needs human implementation
 wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wontfix
Will not be actioned
No Label ready-for-agent
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
sulthan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sulthan/mangaBookmark#44
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?