fix: give covers their own 10 MiB byte cap (#71) #112

2026-08-17T12:06:04+07:00

sulthan commented

2026-08-17 12:06:04 +07:00

Closes #71.

What

Cover fetches reused maxBodyBytes (4 MiB), the ceiling sized for series pages. Covers above it were rejected, logged, and retried every due cycle while the Series kept a monogram forever. backend/internal/latest/cover.go now uses its own maxCoverBytes = 10 << 20; maxBodyBytes is untouched and still governs pages.

Why 10 MiB

Live probe 2026-08-17, plain curl, no Cloudflare challenge:

Site	n	median	p90	max	> 4 MiB
asurascans	25	1,275,082	4,524,788	8,571,192	3 (12%)
demonicscans/readermc	78	63,061	206,994	801,200	0

The p90 already crosses the old cap, and two of the three rejected asura covers are JPEGs, not the animated GIF the issue names. 10 MiB clears 100% of the 103 sampled covers with ~18% headroom over the worst, and matches the caps GitHub (10 MB images/gifs) and Discord (10 MiB default) document for the same asset class.

The format gives no ceiling to lean on: GIF89a has no file-size field, frames are explicitly unlimited, and Go's image/gif bounds nothing. The full reading — spec, decoders, distribution, calibration — is in docs/research/gif-maximum-byte-size.md.

Memory: both cover paths buffer whole (cover.go fetch, handlers.go serve), so worst case is 2 x cap = 20 MiB per concurrent fetch+serve pair, ~10% of the 1974 MiB VPS at ten of each.

Rejected alternatives

Raise the shared constant — loosens the page guard for no benefit; pages measure 100 KB-1.2 MB.
Transcode/downscale — decoding is the OOM path: image/gif allocates width x height per frame with no dimension guard, so a legal 65535^2 GIF asks for ~4.29 GB on a swapless host. Also mutates content-addressed bytes.
Per-site override — only asura has a heavy tail today.

Not in scope

BrowserFetcher.Image (browser.go) caps nothing at all — a real gap, but a different failure mode (CDP data URL, not io.LimitReader) on the kagane/comix path, whose covers measure small. Worth its own issue.

Verification

go test ./... — 446 pass, 10 packages.
TestCoverFetcherAcceptsCoverOverPageCap (new): a cover above the page cap now round-trips with its content type.
TestCoverFetcherRejectsOversizedBody: re-anchored to maxCoverBytes, so the guard is still proven, including the unknown-Content-Length path.

Security invariants preserved: validateURL HTTPS/public-address gate unchanged, io.LimitReader cap still applied to every cover body with the post-read length re-check, no change to auth, CORS, or session handling.

Closes #71. ## What Cover fetches reused `maxBodyBytes` (4 MiB), the ceiling sized for series pages. Covers above it were rejected, logged, and retried every due cycle while the Series kept a monogram forever. `backend/internal/latest/cover.go` now uses its own `maxCoverBytes = 10 << 20`; `maxBodyBytes` is untouched and still governs pages. ## Why 10 MiB Live probe 2026-08-17, plain `curl`, no Cloudflare challenge: | Site | n | median | p90 | max | > 4 MiB | |---|---|---|---|---|---| | asurascans | 25 | 1,275,082 | 4,524,788 | 8,571,192 | 3 (12%) | | demonicscans/readermc | 78 | 63,061 | 206,994 | 801,200 | 0 | The p90 already crosses the old cap, and two of the three rejected asura covers are JPEGs, not the animated GIF the issue names. 10 MiB clears 100% of the 103 sampled covers with ~18% headroom over the worst, and matches the caps GitHub (10 MB images/gifs) and Discord (10 MiB default) document for the same asset class. The format gives no ceiling to lean on: GIF89a has no file-size field, frames are explicitly unlimited, and Go's `image/gif` bounds nothing. The full reading — spec, decoders, distribution, calibration — is in `docs/research/gif-maximum-byte-size.md`. Memory: both cover paths buffer whole (`cover.go` fetch, `handlers.go` serve), so worst case is 2 x cap = 20 MiB per concurrent fetch+serve pair, ~10% of the 1974 MiB VPS at ten of each. ## Rejected alternatives - **Raise the shared constant** — loosens the page guard for no benefit; pages measure 100 KB-1.2 MB. - **Transcode/downscale** — decoding is the OOM path: `image/gif` allocates width x height per frame with no dimension guard, so a legal 65535^2 GIF asks for ~4.29 GB on a swapless host. Also mutates content-addressed bytes. - **Per-site override** — only asura has a heavy tail today. ## Not in scope `BrowserFetcher.Image` (`browser.go`) caps nothing at all — a real gap, but a different failure mode (CDP data URL, not `io.LimitReader`) on the kagane/comix path, whose covers measure small. Worth its own issue. ## Verification - `go test ./...` — 446 pass, 10 packages. - `TestCoverFetcherAcceptsCoverOverPageCap` (new): a cover above the page cap now round-trips with its content type. - `TestCoverFetcherRejectsOversizedBody`: re-anchored to `maxCoverBytes`, so the guard is still proven, including the unknown-`Content-Length` path. Security invariants preserved: `validateURL` HTTPS/public-address gate unchanged, `io.LimitReader` cap still applied to every cover body with the post-read length re-check, no change to auth, CORS, or session handling.

sulthan added 1 commit 2026-08-17 12:06:05 +07:00

fix: give covers their own 10 MiB byte cap (#71 ) ac50c428a9

The cover fetch reused maxBodyBytes, the 4 MiB ceiling sized for series
pages, so any cover above it was rejected, logged, and retried forever
while the Series kept a monogram. Measured against asurascans on
2026-08-17 that is not an edge case: p90 is 4.52 MB and 3 of 25 covers
exceed 4 MiB, two of them plain JPEGs rather than the 8.57 MB animated
GIF the issue names.

Covers now have maxCoverBytes = 10 MiB, separate from the page cap: a
cover is one bounded binary asset, the page cap still has 3.5x headroom
over measured pages and should not be loosened along with it. 10 MiB is
~18% over the largest cover observed and matches the GitHub and Discord
image limits. The format offers no help in picking the number — GIF has
no maximum size at all — so docs/research/gif-maximum-byte-size.md
records the spec reading, the decoder behaviour, and the live
distribution the cap is derived from.

Transcoding was rejected: decoding is the OOM path, since Go's
image/gif allocates width x height per frame with no dimension guard
and a legal 65535^2 GIF would ask for ~4.29 GB on a 1974 MiB swapless
host.

sulthan merged commit 550b258c59 into main

2026-08-17 12:06:15 +07:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sulthan/mangaBookmark#112