fix: address issues found in end-to-end verification

Chained defects made the kagane browser-fetch path completely non-functional
in Docker Compose: headless-shell's compose command re-declared
--remote-debugging-port, colliding with the image's own entrypoint/socat
proxy (EOF on every dial); the sidecar was then only reachable by Docker DNS
name, which Chrome's DevTools HTTP handler rejects with a 500
(Host-header/DNS-rebinding check); and NewBrowserFetcher's NoModifyURL option
skipped /json/version discovery entirely, dialing a bare host:port that
Chrome 404s since /devtools/browser/<uuid> is minted fresh per Chrome start.
Fixed by trimming the redundant command flags, pinning headless-shell to a
static IP so BROWSER_WS_URL can name it directly, and removing NoModifyURL so
chromedp's discovery (which echoes the request's Host back into
webSocketDebuggerUrl) does the right thing on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-03 18:21:25 +07:00
parent 877d3df010
commit ba23411a74
3 changed files with 41 additions and 12 deletions
+15 -7
View File
@@ -46,18 +46,26 @@ type BrowserFetcher struct {
var _ Fetcher = (*BrowserFetcher)(nil)
// NewBrowserFetcher connects to a headless-shell over CDP. wsURL is the
// container's websocket endpoint, e.g. ws://headless-shell:9222.
// NewBrowserFetcher connects to a headless-shell over CDP. wsURL must name the
// sidecar by IP, e.g. ws://172.28.0.10:9222 — not by Docker DNS name. Chrome's
// DevTools HTTP handler 500s any /json/version request whose Host header
// isn't an IP or "localhost" (confirmed 2026-08-03 against
// chromedp/headless-shell:stable), so the compose network pins the sidecar's
// address for this to resolve at all.
//
// NoModifyURL is load-bearing: /json/version advertises a
// webSocketDebuggerUrl pointing at 127.0.0.1, which is meaningless from another
// container, and without this option chromedp follows it and hangs.
// Do not add chromedp.NoModifyURL here: that option skips the /json/version
// discovery request entirely and dials wsURL as if it were already the full
// debugger endpoint, but Chrome only accepts connections at
// /devtools/browser/<uuid>, a path chosen fresh at every Chrome start — dialing
// the bare host:port 404s. The default (discovery) path works precisely
// because Chrome's /json/version response echoes back the Host header of the
// discovery request in webSocketDebuggerUrl, so as long as wsURL is a
// container-reachable IP, the URL chromedp gets back already points at it.
func NewBrowserFetcher(wsURL string) (*BrowserFetcher, error) {
if wsURL == "" {
return nil, fmt.Errorf("empty browser websocket url")
}
ctx, cancel := chromedp.NewRemoteAllocator(
context.Background(), wsURL, chromedp.NoModifyURL)
ctx, cancel := chromedp.NewRemoteAllocator(context.Background(), wsURL)
return &BrowserFetcher{allocCtx: ctx, cancel: cancel}, nil
}