sulthan
17ee0bd3f8
Docs only. No code changes - `git diff origin/main --stat` touches five Markdown files and adds one research note. ## What was wrong Several docs explained Cloudflare challenges as a "bot score" that our request rate could worsen. That mechanism does not exist on these sites. Researched live on 2026-08-12 against Cloudflare's own documentation and blog plus RFC 9309 - 22 primary pages, every claim carrying a source URL and read date, seven areas explicitly marked `Not publicly documented`. The note is `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`. - The 1-99 bot score is **Enterprise Bot Management only**. A free-plan zone has no score at all; it gets Bot Fight Mode, which matches *signatures* (headless browsers, cloud-hosting IPs). - **No per-IP request rate is documented as an input to challenge issuance.** Volume is policed by Rate Limiting Rules, a separate opt-in product: one rule, IP-only counting, 10-second windows on Free. Published DDoS thresholds are ~1,000 errors/sec. - **`cf_clearance` defaults to 30 minutes**, so every cadence at or above 1 hour re-solves the challenge anyway. Cadence changes how many ~4s solves happen per day and nothing else. - The documented risk is **fingerprint quality**, which this repo already solved (real Chrome, stock UA, non-UTC clock). ## What changed | File | Correction | |---|---| | `AGENTS.md` | The block is per-zone configuration plus request fingerprint, not IP reputation. comix.to turning its gate on 2026-08-12 is the worked example. Residential egress avoids the cloud-hosting-IP *signature* rather than earning a better score. The UTC measurement stands; its mechanism is now marked undocumented. | | `backend/AGENTS.md` | Says why `_BROWSER_COOLDOWN` is longer: cost, not safety. | | `docs/adr/0003` | Dated correction - the sites do not "bot-score" the VPS IP. Decision stands on its sweep-depth argument. | | `docs/adr/0006` | Dated correction - no score to be better at. Decision stands on VPS memory. | | `DEPLOY.md` | A red kagane smoke run means the Site's settings or this Chrome's fingerprint moved, not "Cloudflare's scoring". | ADRs got dated `Corrected 2026-08-12:` paragraphs rather than silent rewrites - the record of what was decided stays intact, only the wrong mechanism is retracted. ## Deliberately not in this PR - **The 6h browser cooldown is unchanged.** I had lowered it to 1h and reverted that; cadence is a behaviour change and belongs with the comix work in #98, not in a docs correction. - **Two code comments still carry the myth**: `backend/main.go:83-84` ("a hammer against sites that are already bot-scoring us"). Left alone to keep this diff docs-only. Related: #98. Reviewed-on: #99 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
82 lines
4.2 KiB
Markdown
82 lines
4.2 KiB
Markdown
# ADR-0006: The browser runs on the home machine, over the tailnet
|
|
|
|
Date: 2026-08-09
|
|
Status: accepted
|
|
|
|
## Decision
|
|
|
|
The headless browser is no longer part of the API stack. It is its own compose
|
|
unit (`chrome/docker-compose.yml`), deployed on the home machine, and the API on
|
|
the VPS reaches it over the existing tailnet through `BROWSER_WS_URL`. No
|
|
fallback sidecar remains on the VPS.
|
|
|
|
The backend needs no code change for this. The CDP endpoint was already a
|
|
configuration seam and the fetcher only ever holds the endpoint URL, so
|
|
relocation — and reversal — is one environment variable.
|
|
|
|
## Why
|
|
|
|
The sidecar held 471 MiB working set (645 MiB peak) on a 1974 MiB VPS with no
|
|
swap, which also hosts Traefik, Gitea and its Postgres. That is 24% of the host
|
|
and 86% of this project's memory, for a service that at the time answered zero
|
|
requests: the poller's due query joins bookmarks, production held four kagane
|
|
series and no bookmarks on any of them, and with no kagane bookmark the web UI
|
|
never rendered a kagane cover either.
|
|
|
|
The home machine has 5.9 GiB of swap and a residential egress, which avoids the
|
|
cloud-hosting-IP signature Cloudflare's Bot Fight Mode documentedly challenges. Both
|
|
machines were already on the tailnet.
|
|
|
|
Corrected 2026-08-12: the original wording said Cloudflare "scores" a residential
|
|
egress better than a datacenter IP. There is no score on a free-plan zone; what is
|
|
documented is signature matching, and hosting-provider IP space is one of the
|
|
signatures (`docs/research/cloudflare-bot-scoring-and-poll-cadence.md`). Memory was
|
|
the load-bearing reason regardless.
|
|
|
|
This move is only safe because covers are persisted (ADR-0005's sibling work,
|
|
issue #43/#45) and the browser is on-demand (ADR-0005). Without stored covers a
|
|
sleeping home machine would blank the library; without on-demand start the CI
|
|
runner that already holds ~1.2 GiB of that box's 1.8 GiB would be squeezed
|
|
around the clock.
|
|
|
|
## Constraints
|
|
|
|
**`BROWSER_WS_URL` must be the tailnet IP, never a MagicDNS hostname.** Chrome's
|
|
DevTools HTTP handler answers `/json/version` with a 500 for any `Host` header
|
|
that is not an IP or `localhost`. This is the same trap that previously forced a
|
|
pinned Docker IP; the pinned subnet is gone, the constraint is not.
|
|
|
|
**The CDP port binds to the tailnet address only, never `0.0.0.0`.** CDP
|
|
authenticates nothing: whatever reaches the port drives the browser and, through
|
|
it, the host. On the VPS the safety came from Docker network membership; the
|
|
home machine has a real LAN, so a `0.0.0.0` bind is a hole punched into it. The
|
|
bind address is the enforcement and Tailscale device identity plus a per-device
|
|
ACL is the policy. `BROWSER_BIND_ADDR` deliberately has no default, so an unset
|
|
value fails the deploy instead of publishing CDP to the LAN.
|
|
|
|
No bearer-token proxy is added in front of CDP. It would only defend against a
|
|
device already inside the tailnet, and it would be one more thing between the
|
|
poller and a browser that is already hard enough to keep clearing challenges.
|
|
|
|
**Resource limits are load-bearing, not decorative.** The browser is the
|
|
newcomer on that box, not the incumbent. A hard 512 MiB cap with 1 GiB
|
|
memory+swap makes Chrome reclaim its own cold pages onto the machine's SATA swap
|
|
instead of taking resident memory from the runner; untuned Chrome peaked at
|
|
645 MiB cgroup, which is more than is free there. `oom_score_adj` biases the
|
|
kernel to kill the browser first and never CI. Reduced CPU weight makes a
|
|
challenge solve yield to a running build — cold start degrades to about 3 s at
|
|
half a CPU, immaterial against a 45-second challenge budget. The shared-memory
|
|
reservation drops from 1 GiB to 128 MiB against a measured 19 MiB peak.
|
|
|
|
## Consequences
|
|
|
|
An unreachable browser degrades exactly as an unset `BROWSER_WS_URL` already
|
|
does: plain-TLS libraries are unaffected, kagane and novelfull log and skip, the
|
|
series waits out its cooldown, and stored covers keep serving. A power outage at
|
|
home costs chapter freshness on two sites, never the appearance of the library.
|
|
|
|
The two units are deployed and updated independently. `REDEPLOY.md` §8 covers
|
|
the browser; everything before it covers the API stack. A local `docker compose
|
|
up` now brings up two services, not three, and polls kagane only if
|
|
`BROWSER_WS_URL` is pointed somewhere.
|