sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
103 Commits 27 Branches 12 Tags
cb2b104e63455b488e068679e833a1a3c3b4581b
Commit Graph

10 Commits

Author SHA1 Message Date
sulthan cb2b104e63 feat(web): read admin lanes from the durable pass log, not a poller snapshot (#145) 
The Lanes page now projects store.LatestLanePasses and the owner-window
outcomes (store.LanePassOutcomes) into per-Site rows instead of reading an
in-memory Poller snapshot, so a deploy answers the instant the store is up.
Browser configuration is a config fact and reachability is derived from
recent browser-Site passes inside latest.RefuseBackoff.

This atomically deletes the in-memory path in the same commit that makes the
page read the DB: latest/status.go (LaneState, Status, LaneStatus,
recordLaneState) and the web.LaneReporter seam plus fakeLanes are gone, and
latest.refuseBackoff is renamed latest.RefuseBackoff at every callsite.
ownerWindow (#142) is referenced, never declared (contract C2)
 2026-08-21 20:25:38 +07:00
sulthan ba679223b2 Sightings: a Reader report defers a Poll of a solitary Series (#103) (#108) 
Closes #103.

A userscript PUT already carries the Latest Chapter the Reader's own browser read off the Series page. It may now stand in for a Poll, under one restriction and one ceiling:

- **Solitary Series only** — a Series two Readers share is Polled on schedule however recently it was sighted, so one Reader's mistake can never reach another's list.
- **One rest of standing**, and a **six-rest ceiling** (`sightingCeilingRests`, counted in the Site's own Rest): however many Sightings arrive, an unpolled Series is Polled.

Both live in the due query's HAVING clause (`Store.DueForLatestCheck`) — the same place the schedule has always been decided, so no timer and no second code path can disagree with it. No new query per scheduler round.

Judgement costs no extra request. `Poller.checkOne` already compares what the Site publishes against what is stored: a lower number contradicts the Sighting (Reader and both numbers logged), the same number confirms it, a higher number is the Site publishing and clears the attribution instead. Three contradictions stop that Reader deferring — their reports still write the Latest Chapter — and twenty consecutive confirmations forgive them, as does the owner's clear-marks control from #102.

One client change was required: both userscripts skipped the PUT when the number had not moved, so the case the whole mechanism exists for — visiting a Series with nothing new — never reached the backend. `reportLatestChapter` sends it, skipping only the local write and the re-render. A numberless PUT (favourite toggle, progress from a chapter page) is no Sighting and defers nothing.

Schema: migration `0011_series_sightings.sql` adds `series.latest_sighted_at` and `series.latest_raised_by`. Trust model, thresholds, and rejected alternatives with their citations: `docs/adr/0011-sighting-deferral-trust-model.md`.

Reviewed on both axes (spec against #103, standards against the repo's rules); the blocker — attribution surviving a Poll that overtook the report — is fixed and has a test that fails without the fix.

Verification: `go test ./...` green (needs Docker), `node --test userscript/test/*.test.js` 66 pass.
Reviewed-on: #108
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-16 20:10:59 +07:00
sulthan 3303a55b20 feat: one Poll Lane per Site, replacing the shared pace (#100) (#106) 
Closes #100.

Each Site runs its own Poll Lane: an independent goroutine with its own rest
and pace from the registry (`backend/internal/latest/sites.go`), replacing the
shared cooldown/interval/stagger/batch configuration. Rest (1h, all six Sites
including the browser trio) is enforced by the due query's WHERE clause; the
Lane sleeps its effective gap between fetches — the registry 10s, or
rest/eligible when a Site holds enough Series, floored at 1s with a
Site-naming warning when the floor engages.

Lane-local failure handling:
- Two challenge-held results stop that Site's Lane for 15m; the probes keep
  their stamp, untried Series stay due.
- A lost browser sets a shared Poller flag: the other browser Lanes skip
  their passes for the same 15m (no stamp-per-pass-per-Lane on a dead tab),
  then decay and probe again.
- Browser wake gate preserved (5 due, or one waiting 15m, ADR-0005); one tab
  shared by the three browser Sites; "browser lane behind by X" logged every
  pass.
- Cover work (healing a stored source URL and filling a blank from the series
  page) runs in the background so a slow CDN cannot consume a Lane's gap.

Removed: `LATEST_CHAPTER_POLL_{COOLDOWN,BROWSER_COOLDOWN,INTERVAL,BATCH,STAGGER}`
and the 6h browser rest. Only `LATEST_CHAPTER_POLL_ENABLED` remains; DEPLOY.md
documents the exact `.env` edit. ADR-0010 records the decisions.

Reviewed-on: #106
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-16 13:16:19 +07:00
sulthan 17ee0bd3f8 docs: correct the bot-score claims behind the browser poller (#99) 
Docs only. No code changes - `git diff origin/main --stat` touches five Markdown files and adds one research note.

## What was wrong

Several docs explained Cloudflare challenges as a "bot score" that our request rate could worsen. That mechanism does not exist on these sites.

Researched live on 2026-08-12 against Cloudflare's own documentation and blog plus RFC 9309 - 22 primary pages, every claim carrying a source URL and read date, seven areas explicitly marked `Not publicly documented`. The note is `docs/research/cloudflare-bot-scoring-and-poll-cadence.md`.

- The 1-99 bot score is **Enterprise Bot Management only**. A free-plan zone has no score at all; it gets Bot Fight Mode, which matches *signatures* (headless browsers, cloud-hosting IPs).
- **No per-IP request rate is documented as an input to challenge issuance.** Volume is policed by Rate Limiting Rules, a separate opt-in product: one rule, IP-only counting, 10-second windows on Free. Published DDoS thresholds are ~1,000 errors/sec.
- **`cf_clearance` defaults to 30 minutes**, so every cadence at or above 1 hour re-solves the challenge anyway. Cadence changes how many ~4s solves happen per day and nothing else.
- The documented risk is **fingerprint quality**, which this repo already solved (real Chrome, stock UA, non-UTC clock).

## What changed

| File | Correction |
|---|---|
| `AGENTS.md` | The block is per-zone configuration plus request fingerprint, not IP reputation. comix.to turning its gate on 2026-08-12 is the worked example. Residential egress avoids the cloud-hosting-IP *signature* rather than earning a better score. The UTC measurement stands; its mechanism is now marked undocumented. |
| `backend/AGENTS.md` | Says why `_BROWSER_COOLDOWN` is longer: cost, not safety. |
| `docs/adr/0003` | Dated correction - the sites do not "bot-score" the VPS IP. Decision stands on its sweep-depth argument. |
| `docs/adr/0006` | Dated correction - no score to be better at. Decision stands on VPS memory. |
| `DEPLOY.md` | A red kagane smoke run means the Site's settings or this Chrome's fingerprint moved, not "Cloudflare's scoring". |

ADRs got dated `Corrected 2026-08-12:` paragraphs rather than silent rewrites - the record of what was decided stays intact, only the wrong mechanism is retracted.

## Deliberately not in this PR

- **The 6h browser cooldown is unchanged.** I had lowered it to 1h and reverted that; cadence is a behaviour change and belongs with the comix work in #98, not in a docs correction.
- **Two code comments still carry the myth**: `backend/main.go:83-84` ("a hammer against sites that are already bot-scoring us"). Left alone to keep this diff docs-only.

Related: #98.
Reviewed-on: #99
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-12 09:32:07 +07:00
sulthan 21615be2bd feat: one registry entry per Site, one shared Series-page read (#95) 
Closes #94.

## What

Two phases per the spec, in three feature commits plus two review-fix commits:

**Phase one — one registry entry per Site** (`2d134fb`)
The six per-site comparison points that used to live across three files collapse
into one `sites` map in `backend/internal/latest/sites.go`: Latest Chapter parse,
Cover parse, browser-backed list, fetcher route, host pins, and the browser
payload read all become lookups into it. `browserBackedSites()` is derived from
the registry (sorted, deterministic); `fetcherFor` and `fetchableSeriesURL` keep
their signatures and become lookups; `BrowserFetcher.Get` dispatches through the
entries' `Read`/`Done` while the tab lifecycle stays in `BrowserFetcher.run`.

**Phase two — one shared Series-page read** (`f215130`)
`readSeriesPage` (new `read.go`) performs the read the Poll and the Acquisition
have in common: gate, route, fetch, parse Latest Chapter, parse Cover address.
It returns facts only — polling and persistence policies (stamp order, cooldowns,
cover policy) stay with the callers; `acquire.go` gained the comment naming the
deliberate post-fetch stamp order. The poll's legacy cover heal and the
no-chapter byte-count diagnostic were restored after review (`d998f87`) so the
claims "the Poll keeps its own Cover policy" and "pinning is the only
behavioural change" both hold.

## Behaviour

- All six Sites now pin their host exactly; asura/demonic/comix previously
  accepted any https host. For asura this is a strict improvement: its dead old
  domain redirects deep links to the site root and would parse the wrong
  document.
- Everything else is unchanged: existing parse tables, the challenge-body table
  and the gate table pass unmodified except the one deliberate exception — the
  gate table gains the three new pin cases.

## Security invariants preserved

- The address gate is recognisably the same rule, now a single registry lookup:
  `https` + exact hostname match, all callers route through it. No fetch path
  was widened; asura/demonic/comix were narrowed.
- The second host pin inside each browser entry's Read is retained deliberately
  (browser = strong SSRF primitive, `series_url` is client-supplied) and is not
  deduplicated against the shared gate.
- Review hardening: `fetcherFor` now fails closed for unknown site strings
  (previously fell through to the TLS fetcher on an unreachable path), and the
  browser dispatch iterates a sorted list so outcomes cannot depend on map order.
- The security review's log-injection finding was checked against Go's
  `url.Parse` and does not hold: control characters are rejected anywhere in a
  URL, so a client-supplied value in a log line cannot carry a newline.

## Review

Reviewed on three axes (spec, standards, security) by read-only subagents over
`672c16f..f1b26f4`. No blocking findings; all minor/nit findings addressed in
`d998f87` and `700de20`. Verified end to end with `go test ./...` (Docker
Postgres per test package) on every commit.

## Out of scope (tracked separately)

- Dropping asuracomic.net (CORS allowlist, userscript match, API fixtures,
  live env) — separate issue, per spec.

Reviewed-on: #95
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-12 05:43:37 +07:00
sulthan f1eb7d514c Record the lightnovelworld series-identity decision (#77) (#81) 
Docs only. No code, no tests, nothing to run. Implementation is specified in #80.

Outcome of a grilling session on 2026-08-11 against #77, backed by live measurement of lightnovelworld over 2026-08-10/11.

## What changed

**`docs/adr/0008-series-identity-is-discovered-not-derived.md`** (new)

A Series identity is discovered from the Site's own links, never derived from an address.
On lightnovelworld the userscript reads the chapter page's `All Chapter` anchor instead of
building a `/novel/<slug>/` address by string manipulation. A Chapter Slug is not an
identity and is not stored. The backend's chapter scan drops its per-Series scoping and
runs against the body truncated before the visitor comment thread.

Evidence in the ADR: 3 of 41 sampled novels serve chapters under a slug that differs from
their series slug, divergence runs in both directions, one novel serves chapters under two
slugs, and neither slug is computable from the other. The pointer was checked on 8 chapter
pages and agreed every time. Three narrower selectors are recorded as rejected, each with
the measurement that killed it.

Three rejected options are recorded with reasons: correcting the stored address only, which
keeps an identity the Site does not guarantee; scoping the scan to a container, which the
probe refuted; and a SQL migration, which is impossible because the database holds no
source for the correct slug.

**`CONTEXT.md`**

- **Series** - identity is the canonical slug the Site publishes, never the title and never a Chapter Slug.
- **Chapter Slug** - new term. A slug a Site builds its chapter addresses from. Not an identity: one Series may have several, and none is computable from another.
- **Latest Chapter** - now the highest-numbered chapter, explicitly not a date and not the Site's own newest-chapter banner. Settles #79.

**`docs/research/lightnovelworld-chapter-vs-series-slug.md`** (new, committed with its corrections)

The 41-novel survey behind the ADR. Two claims are struck through and corrected in place,
with the date and sample size of the probe that refuted each: the `ul.clstyle` container it
named is the hidden, empty "Latest Reading" template rather than the chapter list, and its
caveat about the comment region understated the risk, because that region is writable by
any visitor while the scan takes an unbounded maximum into a Series row shared by every
Reader (ADR-0003).

## Review notes

Nothing here constrains code that exists today - the ADR describes work not yet written.
The part worth disagreeing with, if any of it is wrong, is the fail-closed rule: a missing
truncation marker means skip the Series and log, never scan the whole page.

Related: #77 (the defect), #80 (the spec), #79 (the numbering anomaly, closed by decision),
#71 (the same size cap seen from the cover side).

Reviewed-on: #81
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-11 09:34:26 +07:00
sulthan 30c57bd39c Define Cover and record hosting its bytes (#47) (#64) 
Defines **Cover** in the glossary and records ADR-0007, the decision behind #47's fix.

## Why these two files, and why now

`CONTEXT.md` named Cover inside the **Series** entry — "facts true regardless of who is reading — title, cover, Latest Chapter" — but never said *what* one is. That gap is the bug. Nothing in the model distinguished "an address on a Site" from "an image a Reader's browser can display", so both clients were left to work it out independently, and one of them got it wrong. kagane serves covers with `cross-origin-resource-policy: same-origin`, the web UI rewrote them to a proxy in its templates, the JSON API did not, and the panel rendered a broken-image glyph. The new entry closes the ambiguity: *an address no client can load is not a Cover, it is a missing one.*

ADR-0007 records what follows from that — the backend fetches, stores and serves every Site's cover bytes — plus the alternatives that were rejected and, more importantly, the two places this deliberately departs from existing precedent:

- **Destination-class control instead of a host allowlist.** `fetchableSeriesURL` sets the allowlist precedent for `series_url`, and covers do not follow it. Cover hosts are CDNs that move independently of their Site — demonicscans serves its covers from `readermc.org` — so an allowlist would stop producing Covers the day a Site switched CDN, and that failure would look exactly like #47. The resolve-then-classify step is what actually stops the SSRF.
- **A public cover route where the kagane proxy is session-gated.** An `<img>` cannot send a bearer token, and it cannot be given one either: the panel's shadow root is `mode: "open"`, so the host page's JavaScript can read any `src` the script sets.

Both are security-adjacent departures, which is precisely why they are written down rather than left in a commit message.

## Scope

Documentation only — no code, no schema, no behaviour. The implementation is #56–#63.

## Why this should merge promptly rather than sit

All eight implementation tickets cite `docs/adr/0007-backend-hosts-cover-bytes.md` as the authority for decisions they must not relitigate, and they are written in the vocabulary this glossary entry defines. An agent picking up #56 reads both from `main`. Until this lands they get a 404 and either invent a rationale or stall — so this PR gates the tickets, not the other way round.

Related: #47 (bug), #55 (spec), #54 (deferred admin refetch).
Reviewed-on: #64
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-09 23:19:37 +07:00
sulthan 2a3bb6922d Move the browser off the VPS to its own unit (#46) (#52) 
Closes #46 once deployed.

The headless browser leaves the API stack and becomes its own compose unit
(`chrome/docker-compose.yml`) intended for the home machine, reached over the
tailnet. No fallback sidecar is left on the VPS.

The backend needs no code change — `BROWSER_WS_URL` was already the only
coupling. Its default is now empty rather than a pinned Docker IP, so an
unconfigured or unreachable browser degrades exactly as it always has: plain-TLS
libraries unaffected, kagane/novelfull logged and skipped, stored covers still
served.

### What shipped

- `chrome/docker-compose.yml` + `chrome/.env.example` — the browser unit, with
  the CDP port bound to `${BROWSER_BIND_ADDR}` (no default) and the resource
  limits from the epic: 512 MiB / 1 GiB memory+swap, `oom_score_adj 800`,
  halved CPU weight, shm 1 GiB -> 128 MiB.
- API stack drops the service, its `depends_on` and the `browser` network.
- `bookmark-api` gains the `default` network. Dropping `browser` had left it on
  `db` alone, which is `internal: true` — no published port and, worse, no
  egress for the poller at all. Caught by actually bringing the stack up.
- ADR-0006 for the topology; `DEPLOY.md` §7 for first-time setup of the browser
  machine; `REDEPLOY.md` §8 for its independent update cadence; architecture
  diagrams, config tables and troubleshooting rows across README/AGENTS/env.

### Verified locally

- Browser unit builds and runs: Chrome 151, UA carries no `HeadlessChrome`,
  all limits applied as declared.
- **Live smoke passes through the new unit**: `TestSmokeKaganeImage` fetched
  56710 bytes of `image/webp`, `TestSmokeKaganeGet` got a 200 with a real
  chapter list. The challenge cleared under the reduced 128 MiB shm.
- Bind isolation proven: refused on the host's non-loopback address, accepted
  on the configured one.
- 321 MiB peak of the 512 MiB cap after a full solve; 0 restarts, no OOM kill.
- API stack comes up clean, `/healthz` 200; egress confirmed present on
  `default` and absent on `db`.
- `go test ./...`, `go vet`, `gofmt` clean.

### Left to the operator

Provisioning the home machine, the Tailscale ACL, setting `BROWSER_WS_URL` in
production, and observing acceptance criteria 5-7 (covers with the machine off,
several days of zero OOM/restarts, VPS memory improvement). `DEPLOY.md` §7 now
carries the before/after `free -m` reading those need.

Reviewed-on: #52
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-09 15:28:21 +07:00
sulthan 84cfd1b2c1 Make browser sidecar on-demand (#44) (#50) 
Closes #44. Chrome now starts on first CDP connection, tracks concurrent helpers, reaps after 300 seconds idle, preserves the named profile, and classifies reap interruptions. Shutdown stops Chrome's process group so cookie batches flush. ADR-0005 records the measured constraints and decisions. Verification: docker build, live CDP wake, graceful stop cleanup, sh -n, and go test ./... (7 packages, 3 no tests).

Reviewed-on: #50
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-09 07:39:07 +07:00
sulthan 08749df050 feat(backend)!: run on Postgres with a migration-owned schema (#28) 
Swap modernc.org/sqlite for jackc/pgx/v5 with no observable change:
same endpoints, same wire format, same updated_at ordering rule.

The schema now comes from numbered SQL embedded in the binary and
applied on startup, one transaction each, recorded in
schema_migrations. That replaces two pieces of SQLite-era machinery,
both deleted rather than ported: the column probing (Postgres has ADD
COLUMN IF NOT EXISTS, and there is no legacy database left to probe)
and the Asura key rewrite, which has run clean on every start for
months now that the userscripts strip build hashes before writing. Its
regexp survives as latest.asuraBuildHash, where the poller still needs
it to scope chapter links to a series whose slug carries a rotating
hash.

Types get real: favorite is a boolean, chapter numbers double
precision, timestamps stay unix-ms bigint. SQLite's null-safe IS NOT
becomes IS DISTINCT FROM, which is what implements the rule that only
reading progress reorders a list. Inside COALESCE/NULLIF the status
and kind parameters need an explicit ::text -- there is no target
column to infer from and Postgres refuses to guess.

Tests lose their free t.TempDir() database, so Docker is now a hard
prerequisite for `go test ./...`: internal/pgtest starts one
postgres:17-alpine per test binary and hands each test a database of
its own.

Also lands CONTEXT.md and the four ADRs written while scoping #18.

BREAKING CHANGE: DB_PATH is retired for DATABASE_URL, which is
required and has no default. Compose gains a postgres service on an
internal network with its own volume; POSTGRES_PASSWORD joins .env.
The old bookmarks-data volume is deliberately left undeclared so
`docker compose down -v` cannot take the pre-migration database with
it. main is not deployable until #25 and #26 land.

Closes #20

Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
 2026-08-08 06:52:20 +07:00