Compare commits

..

22 Commits

Author SHA1 Message Date
sulthan 264839e798 Merge ticket/147-lane-pause 2026-08-22 01:19:51 +07:00
sulthan cbe0a28921 feat(web,latest): pause and resume one Site's Lane with a mandatory expiry (#147)
Two owner-gated POST routes write the durable poll_lanes pause stamp the
poller's top-of-pass gate already reads: /admin/lanes/{site}/pause validates
the duration against the fixed 1h/6h/24h allow-list and the Site against the
registry, and /admin/lanes/{site}/resume zeroes the stamp. Both cap the form
body like the API path, answer with the freshly rendered Lanes block, and
never command the poller — the pause is a fact about the Site, so it
survives a restart. The Lanes page's c-ctrl slot now carries the pausebar:
Resume while paused, the duration select plus Pause while running, with the
paused phrase read from the live poll_lanes stamp so a press renders as
paused with no pass having run. Tests cover the round trips, rejections,
body caps, the pause-before-refusal ordering, fresh-poller survival, resume
restoring the full queue, and acquisition being unaffected.
2026-08-22 01:15:57 +07:00
sulthan e45445cb20 Merge ticket/146-forced-poll 2026-08-22 01:05:09 +07:00
sulthan 4a95657425 Merge ticket/143-overview 2026-08-22 01:03:33 +07:00
sulthan 5b7adf5f2c refactor(web): share poll-state derivation; keep zebra parity across a row swap (#146)
Review round: pollState() unifies the CanPoll/Pending/Requested derivation
used by the list row and the detail page, and the row anchor carries its
band parity with the press (hx-vals) so the swapped answer keeps the
alternation.
2026-08-22 01:02:55 +07:00
sulthan 00fa237151 fix(web): verdict space per design mock, single door() fig builder (#143) 2026-08-22 00:56:49 +07:00
sulthan fe2cd12049 feat(web,store,latest): forced poll — Check now stamps a Series for the next Lane pass (#146)
The control writes series.force_poll_at (column landed in migration 0014)
and never commands the poller: pending is derived as force_poll_at >
latest_checked_at and self-clears because the check stamp is written before
the fetch. The due query's forced flag overrides the rest cutoff, the
Sighting-deferral and finished-only clauses, jumps the queue, and wakes a
sleeping browser Lane; it never overrides an empty series_url, the Bookmarks
join, the refusal backoff, the sidecar-down skip or the Lane gap.

ADRs: 0013-commands-through-the-database.
2026-08-22 00:53:51 +07:00
sulthan c432a3be30 feat(web): overview page with verdict line and stats doors (#143) 2026-08-22 00:49:39 +07:00
sulthan 6f9109c885 Merge ticket/145-lanes-from-db 2026-08-22 00:35:34 +07:00
sulthan f5b52e48c8 Merge ticket/144-series-detail 2026-08-22 00:34:14 +07:00
sulthan bf08d6e65c Merge ticket/142-series-list 2026-08-21 20:42:42 +07:00
sulthan d72c48295d fix(web): statusline tokens, unreachable dot, reachability test spans (#145)
Second review round (spec + standards):
- Polling-off and unreachable render as statusline tokens; the unreachable
  span drops mark-strong so the patina dot never sits next to red text.
- Reachability tests assert on the rendered span, which 'reachable' and
  'unreachable' substrings never could.
- A pass-log query failure now reads as reachable (no evidence rule) instead
  of condemning the browser, and admin.go loses its dead time import.
- startLatestPoller's doc no longer claims the admin page reads the poller;
  AGENTS.md carries the RefuseBackoff rename.
2026-08-21 20:41:45 +07:00
sulthan fc8a40cfc6 fix(latest,web): finish the atomic deletion and honest empty states (#145)
Review fixes on top of cb2b104:
- Delete the write-only in-memory refuseUntil map and setRefusalBackoff now
  that status.go is gone: the pass gate reads the durable stamp, so the map
  was half-deleted dead state (spec review C-1).
- An outcome-query failure no longer blanks the Lane table into the false
  'no data yet': rows render with 'none observed' chips instead (I-1).
- due-query and eligible-count skips get their own sentences instead of the
  merged 'check failed' (I-2).
- ADR-0012 constrain wording corrected and trailing newline added (M-1)
2026-08-21 20:31:50 +07:00
sulthan cb2b104e63 feat(web): read admin lanes from the durable pass log, not a poller snapshot (#145)
The Lanes page now projects store.LatestLanePasses and the owner-window
outcomes (store.LanePassOutcomes) into per-Site rows instead of reading an
in-memory Poller snapshot, so a deploy answers the instant the store is up.
Browser configuration is a config fact and reachability is derived from
recent browser-Site passes inside latest.RefuseBackoff.

This atomically deletes the in-memory path in the same commit that makes the
page read the DB: latest/status.go (LaneState, Status, LaneStatus,
recordLaneState) and the web.LaneReporter seam plus fakeLanes are gone, and
latest.refuseBackoff is renamed latest.RefuseBackoff at every callsite.
ownerWindow (#142) is referenced, never declared (contract C2)
2026-08-21 20:25:38 +07:00
sulthan 134c9307b1 #142: fix stale filter count cutoff, tighten comments (review fixes) 2026-08-21 19:48:24 +07:00
sulthan 09a094ca67 fix(web): detail row lookup walks all pages of the Site read (#144) 2026-08-21 19:42:45 +07:00
sulthan e8a3c5f826 #142: series list page with eight hygiene filters, site/library narrowing, paging 2026-08-21 19:40:45 +07:00
sulthan 14990bde21 feat(web): per-Series detail page keyed by site:series_id (#144) 2026-08-21 19:38:44 +07:00
sulthan 503fb49d0a feat(latest): record one poll pass per exit with skip reason and outcome counts
Every way a Lane pass can end now writes exactly one durable row: a skip
value naming the exit (paused, refusing, sidecar-down, no-fetcher,
due-query, asleep, eligible-count, nothing-eligible, or empty for the
loop), five outcome counts from the classification the Series read already
makes, and carry-forward of the previous pass's figures exactly when the
pass's own gap is zero. Refusal is durable through the poll_lanes row, so
a restart does not re-probe a Site inside its backoff. Retention is 14
days. The mid-loop browser-unreachable return writes an empty skip by
design: a tenth value is not invented here. (#141)
2026-08-21 18:48:30 +07:00
sulthan e0b9063d9e feat(store): cross-series admin read model with privacy in the projection (#140)
SeriesPage returns one 50-row page of Series matching one of eight named hygiene filters (all, no_series_url, never_read_a_chapter, no_readers, never_checked, stale, no_cover, reader_report), with a window-count total; SeriesShapes returns the per-Site aggregate, one grouped pass. The compound filter value object takes Site, Kind, Name, a caller-supplied staleness cutoff, and a 1-based page.

The privacy boundary lives in the projection: adminSeriesColumns never selects latest_raised_by, and AdminSeries has no field for it — a SQL-computed boolean is all that crosses. LEFT JOIN surfaces orphans (reader_count 0); (site, series_id) tie-breaks the zero-stamp boundary so pages stay stable.

Also lands 0014: the series(latest_checked_at) index and series.force_poll_at, both expand-step schema for later tickets in the series.
2026-08-21 18:20:59 +07:00
sulthan fd1131d11d feat(store): persist poll lane state 2026-08-21 17:22:23 +07:00
sulthan 030ffdc26e feat: split admin surface into bookmarkable pages (#138) 2026-08-21 17:01:46 +07:00
28 changed files with 232 additions and 2499 deletions
+87 -31
View File
@@ -1,49 +1,108 @@
---
name: implement-tickets
description: "Run a planned wave of tickets: one implementer subagent per ticket in its own worktree, then land, merge and close what comes back."
description: "Orchestrate a batch of tickets: plan the briefs, then hand each ticket to its own implementer subagent in its own worktree."
disable-model-invocation: true
---
# Implement tickets
You are the **orchestrator**. You dispatch, land results, and talk to the
tracker. You do not write the implementation — every line of ticket code is
written by a `ticket-implementer` subagent in its own git worktree. Reach for the
editor yourself only for a merge conflict resolution.
You are the **orchestrator**. You write briefs, dispatch, land results, and talk
to the tracker. You do not write the implementation — every line of ticket code
is written by a `ticket-implementer` subagent in its own git worktree. Reach for
the editor yourself only for a merge conflict resolution.
Ticket source and tracker conventions: `docs/agents/issue-tracker.md`. `tea` usage: skill `gitea`.
## 0. Load the plan
## 1. Collect the tickets
Your argument is the batch slug. Read `.scratch/<batch-slug>/plan.md` — it gives
the base branch, the waves, the contracts, and each ticket's brief path and
status. Run the earliest wave that is not landed.
The user's argument is the selector: issue numbers, a label, a parent issue, or
nothing. With nothing, take the open issues labelled `ready-for-agent`.
The briefs are the requirements and they are already approved: read each one you
are about to dispatch, but do not rewrite it, and do not fetch the tickets from
the tracker to second-guess it. A brief that is wrong or thin is a `plan-tickets`
problem — say so and stop, rather than patching it here.
Fetch each with `tea issue <n> --comments`, and read the **whole** body —
acceptance criteria and the `Blocked by` line are what the rest of this skill
runs on. A ticket whose blockers are still open is out of this batch unless a
blocker is also in it.
No plan file, or no briefs for the next wave: run `plan-tickets` first. That
skill owns wave membership, contracts, and every brief.
## 2. Plan the batch
## 1. Dispatch the wave
Explore enough of the codebase to write briefs a fresh context can act on: the
files each ticket lands in, the patterns it must follow, the `AGENTS.md`
invariants it touches.
Then decide three things:
- **Waves.** Blocking edges set the order; tickets with no open blocker inside
the batch share a wave. Cap each wave at **3** concurrent tickets unless the
user set another width.
- **Contracts.** Two tickets in one wave that meet at a function signature, a
JSON shape, a table column, or a token name: you decide the shape now and
write the identical wording into both briefs. A contract left for the
subagents to negotiate is a merge conflict you scheduled.
- **Splits.** A ticket too big for one fresh context window goes into the wave
as two briefs, or back to the user.
## 3. Get the plan approved
Present, and stop:
- the wave list, and for each ticket: number, title, one-line brief summary,
the files or areas it will touch, its verification commands
- every cross-ticket contract, verbatim as it will appear in the briefs
- anything you had to assume
Wait for approval. Apply the user's edits to the plan, do not relitigate them.
## 4. Run a wave
Per ticket, before dispatch:
```bash
git worktree add ../ticket-<n> -b ticket/<n>-<slug> <base> # base = the plan's base branch, checked out here
git worktree add ../ticket-<n> -b ticket/<n>-<slug> <base> # base = the branch you are on
cp .env ../ticket-<n>/ 2>/dev/null # gitignored, worktrees do not get it
tea issue edit <n> --add-assignees <your gitea username> # tea login list has it
```
Write the brief to `.scratch/<batch-slug>/t<n>-brief.md` using the template
below, in the ubiquitous language of `CONTEXT.md` — a brief that says "scrape"
where the domain says Poll hands the subagent the wrong model of the system.
Then dispatch the whole wave in **one** `task` batch, every item on the
`ticket-implementer` agent. Each dispatch names: the absolute brief path, the
worktree path, the branch, the base ref, and the report path
`.scratch/<batch-slug>/t<n>-report.md`. Mark each ticket `dispatched` in the plan
file.
`.scratch/<batch-slug>/t<n>-report.md`.
## 2. Land the wave
<brief-template>
# Ticket #<n> — <title>
**Read first.** `tea issue <n> --comments` for this ticket, then the issue it
refers to — the parent or spec — the same way. The comments carry decisions the
body never got updated with. This brief stays the requirements; those two reads
are the intent behind them.
**Goal.** The end-to-end behaviour this ticket makes work, from the user's side.
**Acceptance criteria.** Verbatim from the ticket.
**Contract.** The exact shared signatures / shapes / names this ticket must
implement or consume, and which sibling ticket is on the other end. Omit when
the ticket touches nothing shared.
**Where it lands.** The files and packages, and the existing pattern to follow
in each.
**Binding invariants.** The `AGENTS.md` rules this change can break — name them.
**TDD seams.** Where a test comes first — run the `tdd` skill at each one and
follow its red → green loop. Or "none — verify after".
**Verify.** The exact commands, e.g. `cd backend && go test ./...`,
`node --test userscript/test/logic.test.js`.
**Out of scope.** What not to touch, especially a sibling ticket's files.
</brief-template>
## 5. Land the wave
The wave is landed when every ticket in it is closed, reverted, or handed back
to the user. Per returned ticket:
@@ -61,17 +120,14 @@ clash — both sides green apart, wrong together — goes back to whichever tick
owns the contract, as a re-dispatch with the collision described.
Then `tea comment <n> "<the report summary>"`, `tea issue close <n>`, and
`git worktree remove ../ticket-<n>`. Keep the report file, and mark the ticket
`landed` or `handed back` in the plan file.
`git worktree remove ../ticket-<n>`. Keep the report file.
## 3. Hand back or close the batch
Only once the whole wave is landed does the next wave start — its briefs may
need what this one changed.
Waves left in the plan: stop and say which wave is next. Its briefs are written
by `plan-tickets` against the base you just changed — that is why they were not
written up front, and why you do not write them.
## 6. Close the batch
Last wave landed: run the full suite once on the merged base, and report a line
per ticket with its status, commits, and open concerns, plus anything still
assigned or open on the tracker. A red suite after every ticket went green is an
interaction bug — diagnose it, name the two tickets, and fix it or hand it back
with both named.
Run the full suite once on the merged base, and report: a line per ticket with
its status, commits, and open concerns, plus anything still assigned or open on
the tracker. A red suite after every ticket went green is an interaction bug —
diagnose it, name the two tickets, and fix it or hand it back with both named.
-125
View File
@@ -1,125 +0,0 @@
---
name: plan-tickets
description: "Plan a batch of tickets and write the briefs for its next wave: read the tracker, decide waves and contracts, get the plan approved."
disable-model-invocation: true
---
# Plan tickets
You produce the two artifacts the `implement-tickets` skill runs on: a **plan
file** and one **brief** per ticket in the next wave. You write no ticket code
and create no worktrees — that is the runner's half.
Ticket source and tracker conventions: `docs/agents/issue-tracker.md`. `tea` usage: skill `gitea`.
Called twice in a batch's life, at least: once to open it, then again after each
wave lands, because a later wave's briefs may need what the last one changed. On
a re-entry, read the existing `.scratch/<batch-slug>/plan.md` first and plan only
the next unlanded wave — the waves and contracts already approved there stand
unless the landed wave proved one wrong.
## 1. Collect the tickets
The user's argument is the selector: issue numbers, a label, a parent issue, or
nothing. With nothing, take the open issues labelled `ready-for-agent`.
Fetch each with `tea issue <n> --comments`, and read the **whole** body —
acceptance criteria and the `Blocked by` line are what the rest of this skill
runs on. A ticket whose blockers are still open is out of this batch unless a
blocker is also in it. Read the issue each ticket refers to, its parent or spec,
the same way: nothing on the implementation path reads the tracker after you —
only `cr-spec` does, at review time — so a decision that lives in a comment
reaches the implementer only if you carry it into the brief.
## 2. Plan the batch
Explore enough of the codebase to write briefs a fresh context can act on: the
files each ticket lands in, the patterns it must follow, the `AGENTS.md`
invariants it touches.
Then decide three things:
- **Waves.** Blocking edges set the order; tickets with no open blocker inside
the batch share a wave. Cap each wave at **3** concurrent tickets unless the
user set another width.
- **Contracts.** Two tickets in one wave that meet at a function signature, a
JSON shape, a table column, or a token name: you decide the shape now and
write the identical wording into both briefs. A contract left for the
subagents to negotiate is a merge conflict you scheduled.
- **Splits.** A ticket too big for one fresh context window goes into the wave
as two briefs, or back to the user.
## 3. Write the artifacts
Pick a `<batch-slug>` — short, from what the batch is about — and write
`.scratch/<batch-slug>/plan.md`. It is the handoff: the runner is a fresh context
that reads this and nothing of your reasoning.
<plan-template>
# Batch <batch-slug>
**Base branch.** The branch every worktree forks from and merges back into.
**Waves.** A table: wave number, ticket number, title, brief path, and status —
`planned` | `dispatched` | `landed` | `handed back`. Every ticket in the batch,
including waves not briefed yet.
**Contracts.** Each cross-ticket contract verbatim, naming both ticket numbers.
**Assumptions.** What you had to assume, and what the user corrected at approval.
</plan-template>
Then write a brief per ticket in the next wave to
`.scratch/<batch-slug>/t<n>-brief.md`, using the template below, in the
ubiquitous language of `CONTEXT.md` — a brief that says "scrape" where the domain
says Poll hands the subagent the wrong model of the system.
<brief-template>
# Ticket #<n> — <title>
**Decisions.** Whatever the ticket's comments or its parent spec settled that
the body never got updated with, restated verbatim. The implementer reads this
brief and the code, never the tracker — a decision missing here is lost to it,
and a stale comment thread is not a source you want a fresh context guessing
from. Omit only when the ticket has no comments.
**Goal.** The end-to-end behaviour this ticket makes work, from the user's side.
**Acceptance criteria.** Verbatim from the ticket.
**Contract.** The exact shared signatures / shapes / names this ticket must
implement or consume, and which sibling ticket is on the other end. Omit when
the ticket touches nothing shared.
**Where it lands.** The files and packages, and the existing pattern to follow
in each.
**Binding invariants.** The `AGENTS.md` rules this change can break — name them.
**TDD seams.** Where a test comes first — run the `tdd` skill at each one and
follow its red → green loop. Or "none — verify after".
**Verify.** The exact commands, e.g. `cd backend && go test ./...`,
`node --test userscript/test/logic.test.js`.
**Out of scope.** What not to touch, especially a sibling ticket's files.
</brief-template>
## 4. Get the plan approved
Present, and stop:
- the wave list, and for each ticket in the next wave: number, title, one-line
brief summary, the files or areas it will touch, its verification commands
- every cross-ticket contract, verbatim as it appears in the briefs
- anything you had to assume
Wait for approval. Apply the user's edits to the plan, do not relitigate them —
into the files, not just the reply, or the runner never sees them.
Then hand off: name the batch slug and the wave, and stop. Running the wave is
`implement-tickets`.
+15 -16
View File
@@ -24,35 +24,34 @@ Your branch is already checked out there. Never `git checkout`, `git switch`,
## Order of work
1. Read the brief file. It is the only statement of requirements you get — use
its exact values verbatim, and read nothing from the tracker. The
orchestrator has already read the ticket, its comments and its parent spec,
and folded every live decision into the brief; the threads themselves also
hold reversed and rejected ones you cannot tell apart from here.
2. Read `AGENTS.md` in the worktree, plus the nested `AGENTS.md` for the area
1. Read the brief file. It is the single source of requirements — use its exact
values verbatim.
2. Read the ticket and the issue it refers to, as the brief's **Read first**
section names them: `tea issue <n> --comments` for each. The ticket's
comments and its parent carry the intent and the decisions behind the brief.
Read no other ticket and no other brief.
3. Read `AGENTS.md` in the worktree, plus the nested `AGENTS.md` for the area
you touch. Its invariants bind you: security rules, design system, comment
policy.
3. Ask before writing code if requirements, acceptance criteria, approach, or
4. Ask before writing code if requirements, acceptance criteria, approach, or
dependencies are unclear. Asking is free; guessing is not.
4. Implement exactly what the brief specifies. At each TDD seam the brief names,
5. Implement exactly what the brief specifies. At each TDD seam the brief names,
run the `tdd` skill and follow its red → green loop.
Follow the patterns already in the codebase; improve what you touch,
restructure nothing outside the ticket.
5. Verify. Focused tests while iterating, the brief's full verification commands
6. Verify. Focused tests while iterating, the brief's full verification commands
once at the end. Test output must be pristine.
6. Commit to your branch. Reference the ticket number in the subject.
7. Review (below), fix, re-verify, commit the fixes.
8. Write the report file, then return the status contract.
7. Commit to your branch. Reference the ticket number in the subject.
8. Review (below), fix, re-verify, commit the fixes.
9. Write the report file, then return the status contract.
## Review
After your first green commit, run the **`code-review`** skill over
`<base ref>...HEAD` in the worktree, with two changes to how it dispatches:
use the **`cr-spec`** agent for the Spec axis and **`cr-standards`** for the
Standards axis, both in one batch, and hand the Spec axis your brief file as the
spec plus the ticket number from the brief's title, telling it to read the
ticket itself (`tea issue <n> --comments`). The tracker check belongs in that
read-only context, not in yours.
Standards axis, both in one batch, and give the Spec axis your brief file plus
the ticket body as the spec.
Fix every Critical and Important finding, then re-run the tests that cover the
amended code. Two fix rounds maximum: anything still open after that goes in the
+1 -1
View File
@@ -99,7 +99,7 @@ Go backend:
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
- Any outbound fetch of a client-supplied URL passes `FetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
- Compare secrets with `hmac.Equal` / `subtle.ConstantTimeCompare`, never `==`. A credential is matched by the SHA-256 the `readers` table holds, which is already a fixed-width equality — a new secret comparison must not regress to `==`.
- Errors: generic text to the client (`http.Error(w, "internal error", 500)`), detail to `log.Printf`. Never log `TOKEN_KEY`, a Reader's credential, `DISCORD_CLIENT_SECRET`, a session id, or a whole `Authorization` header.
+4 -4
View File
@@ -32,7 +32,7 @@ func TestPublicCoverServesStoredBytesUnauthenticated(t *testing.T) {
// The wire URL is what a client actually requests, so the path under test
// is taken from it rather than rebuilt by hand.
wire := st.CoverWireURL(store.CoverAddressForBytes([]byte("\x00webp-bytes")))
wire := st.CoverWireURL(store.CoverAddress(sourceURL))
path, ok := strings.CutPrefix(wire, testCoverBaseURL)
if !ok {
t.Fatalf("wire URL %q is not on the public origin %q", wire, testCoverBaseURL)
@@ -57,7 +57,7 @@ func TestPublicCoverServesStoredBytesUnauthenticated(t *testing.T) {
func TestPublicCoverRejectsUnknownAddress(t *testing.T) {
srv, _ := newWebTestServer(t, testConfig())
cases := map[string]string{
"unknown": "/covers/" + store.CoverAddressForBytes([]byte("never-stored")),
"unknown": "/covers/" + store.CoverAddress("https://cdn.example/never-stored.jpg"),
"malformed": "/covers/not-an-address",
"traversal": "/covers/../../etc/passwd",
"empty": "/covers/",
@@ -86,7 +86,7 @@ func TestPublicCoverNeverEchoesNonImage(t *testing.T) {
}
// A legitimate row, then the content type flipped behind the store's back:
// the bytes exist at the address, so only the type is hostile.
address := store.CoverAddressForBytes([]byte("<script>"))
address := store.CoverAddress(sourceURL)
if err := st.SetSeriesCover("asura", "solo", sourceURL, []byte("<script>"), "image/png"); err != nil {
t.Fatalf("seed row: %v", err)
}
@@ -127,7 +127,7 @@ func TestListRendersAcquiredCover(t *testing.T) {
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rr.Code)
}
want := `src="` + testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("\xff\xd8jpeg")) + `"`
want := `src="` + testCoverBaseURL + "/covers/" + store.CoverAddress(sourceURL) + `"`
if !strings.Contains(rr.Body.String(), want) {
t.Fatalf("rendered list does not contain %s", want)
}
+7 -7
View File
@@ -123,10 +123,10 @@ func TestAcquireFillsChapterAndCoverFromOneFetch(t *testing.T) {
if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 {
t.Fatalf("LatestChapterNum = %v, want 181", got.LatestChapterNum)
}
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(acquireCoverURL); got.Cover != want {
t.Fatalf("Cover = %q, want the absolute address %q", got.Cover, want)
}
body, contentType, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes")))
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(acquireCoverURL))
if err != nil || !ok {
t.Fatalf("CoverByAddress = %v, %v", ok, err)
}
@@ -182,7 +182,7 @@ func TestAcquireSkipsAnExistingSeries(t *testing.T) {
t.Fatalf("cover fetches = %d, want 1", got)
}
got := readBookmark(t, s, acquireKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(acquireCoverURL); got.Cover != want {
t.Fatalf("Cover = %q, want the acquired one %q", got.Cover, want)
}
}
@@ -343,10 +343,10 @@ func TestAcquireKaganeCoverThroughBrowser(t *testing.T) {
t.Fatalf("browser cover fetched URL %q, want %q", got, kaganeCoverSrc)
}
got := readBookmark(t, s, kaganeKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(kaganeCoverSrc); got.Cover != want {
t.Fatalf("Cover = %q, want the content-addressed URL %q", got.Cover, want)
}
body, contentType, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes")))
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(kaganeCoverSrc))
if err != nil || !ok {
t.Fatalf("CoverByAddress = %v, %v", ok, err)
}
@@ -381,7 +381,7 @@ func TestAcquireNovelfullCoverOverPlainTLS(t *testing.T) {
t.Fatalf("cover fetched from %q, want %q", got, novelfullCoverURL)
}
got := readBookmark(t, s, novelfullKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(novelfullCoverURL); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
}
@@ -451,7 +451,7 @@ func TestAcquireNovelfullCoverWithoutBrowser(t *testing.T) {
t.Fatalf("cover fetches = %d, want 1", got)
}
got := readBookmark(t, s, novelfullKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(novelfullCoverURL); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
}
+2 -2
View File
@@ -124,7 +124,7 @@ func (f *BrowserFetcher) Get(ctx context.Context, seriesURL string) (string, int
// request must be made from inside the page so it carries the clearance
// cookie, and the API is the only place the list exists. Refusing any other
// address is the per-Site half of the SSRF gate, kept deliberately behind
// FetchableSeriesURL (see browserRead.Read).
// fetchableSeriesURL (see browserRead.Read).
func kaganeRead(seriesURL string, out *string) (chromedp.Action, bool) {
apiURL, ok := kaganeAPIURL(seriesURL)
if !ok {
@@ -326,7 +326,7 @@ func (f *BrowserFetcher) run(ctx context.Context, target string, read chromedp.A
// kaganeAPIURL maps a stored series_url to the JSON endpoint carrying its
// chapter list. Returning false for anything else is a second line of defence
// behind FetchableSeriesURL: a headless browser is a strong SSRF primitive and
// behind fetchableSeriesURL: a headless browser is a strong SSRF primitive and
// series_url is client-supplied, so the host is pinned here too.
func kaganeAPIURL(seriesURL string) (string, bool) {
u, err := url.Parse(seriesURL)
+1 -1
View File
@@ -174,7 +174,7 @@ func (f *TLSCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte,
return body, contentType, nil
}
// This gate deliberately differs from FetchableSeriesURL: cover hosts are
// This gate deliberately differs from fetchableSeriesURL: cover hosts are
// site-independent CDNs, so a Site host allowlist would reject valid covers.
func (f *TLSCoverFetcher) validateURL(ctx context.Context, u *url.URL) error {
if u == nil || u.Scheme != "https" || u.Host == "" || u.User != nil {
+7 -58
View File
@@ -98,23 +98,6 @@ func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, cover stri
}()
}
// replaceCover is the Forced Poll's Cover path: the owner asked to accept the
// page as it now stands, so where fillBlankCover leaves a non-blank Cover
// alone (ADR-0007) this writes through whatever the page's Cover URL answers
// with, whether one exists or not. The accepted consequence (issue #135):
// refreshing the Cover and re-reading the chapters are one act — there is no
// Cover-only refetch.
func (p *Poller) replaceCover(ctx context.Context, sr store.Series, cover string) {
if cover == "" {
return
}
p.coverWG.Add(1)
go func() {
defer p.coverWG.Done()
p.storeCover(ctx, sr, cover)
}()
}
// prefetchCover heals Series that already carry a third-party source URL but
// no stored address — the state left by client-supplied covers before
// acquisition moved server-side. Every Site takes the same path; fetchCoverBytes
@@ -139,42 +122,17 @@ func (p *Poller) prefetchCover(ctx context.Context, sr store.Series) {
p.storeCover(ctx, sr, sr.Cover)
}
// storeCover fetches bytes for sourceURL and points the Series at them: a
// fill-only write for an ordinary pass, a write-through for a forced one
// (issue #135). Every failure is logged against the Series and swallowed so
// the chapter poll cannot see it.
// storeCover fetches bytes for sourceURL and points the Series at them. Every
// failure is logged against the Series and swallowed so the chapter poll
// cannot see it.
func (p *Poller) storeCover(ctx context.Context, sr store.Series, sourceURL string) {
bytes, contentType, err := fetchCoverBytes(ctx, sourceURL, p.CoverFetch, p.CoverBytesFetch)
if err != nil {
log.Printf("latest poll %q: fetch cover %s: %v", sr.Key(), sourceURL, err)
return
}
if !sr.Forced {
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType); err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
}
return
}
// The forced write replaces whether or not a Cover exists, and the row
// then tells the three outcomes apart: a blank filled, identical artwork
// re-served — an honest no-op — or a replacement whose previous address
// is stranded and reclaimed below. A failed reclaim is logged and the
// stranded bytes stay served until a later call reclaims them.
previous, current, err := p.Store.ReplaceSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType)
if err != nil {
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType); err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
return
}
switch {
case previous == "":
log.Printf("latest poll %q: cover filled at %s", sr.Key(), current)
case previous == current:
log.Printf("latest poll %q: cover unchanged, the site re-serves the same bytes", sr.Key())
default:
if err := p.Store.ReclaimCover(previous); err != nil {
log.Printf("latest poll %q: reclaim cover %s: %v", sr.Key(), previous, err)
}
log.Printf("latest poll %q: cover replaced %s -> %s", sr.Key(), previous, current)
}
}
@@ -663,12 +621,7 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOut
p.healCover(ctx, sr)
// Cover fill is independent of the chapter signal: a page that lost its
// chapter list may keep its og:image, and a blank Series heals either way.
// A forced pass writes the Cover through the replace path instead.
if sr.Forced {
p.replaceCover(ctx, sr, facts.Cover)
} else {
p.fillBlankCover(ctx, sr, facts.Cover)
}
p.fillBlankCover(ctx, sr, facts.Cover)
if !facts.HasLatest {
// Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a rest instead of hot-looping.
@@ -757,7 +710,7 @@ func (p *Poller) waitCovers() {
p.coverWG.Wait()
}
// FetchableSeriesURL reports whether site is a Site the registry knows and
// fetchableSeriesURL reports whether site is a Site the registry knows and
// seriesURL is safe to hand to a fetcher: an https URL whose host matches the
// Site's pinned hostname exactly. series_url comes from client-supplied PUT
// bodies, so this is a defence against the poller being used to probe
@@ -766,11 +719,7 @@ func (p *Poller) waitCovers() {
// browser Site guards a control that executes JavaScript and carries cookies,
// a parser Site guards a wasted request — but the rule is one rule, from the
// registry.
//
// The owner's series URL repair (issue #151) is a second caller: the web
// layer validates with this same gate before storing a repair, so there is
// never a second copy of it.
func FetchableSeriesURL(site, seriesURL string) bool {
func fetchableSeriesURL(site, seriesURL string) bool {
s, known := sites[site]
if !known {
return false
+18 -376
View File
@@ -6,10 +6,8 @@ import (
"database/sql"
"errors"
"fmt"
"io/fs"
"log"
"os"
"path/filepath"
"strings"
"sync"
"testing"
@@ -216,9 +214,9 @@ func TestRunOncePrefetchesPublicCover(t *testing.T) {
if got := covers.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1", got)
}
body, contentType, found, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes")))
body, contentType, found, err := s.GetCover(coverURL)
if err != nil || !found {
t.Fatalf("CoverByAddress: %v found=%v", err, found)
t.Fatalf("GetCover: %v found=%v", err, found)
}
if string(body) != "cover-bytes" || contentType != "image/jpeg" {
t.Fatalf("stored cover = (%q, %q), want (cover-bytes, image/jpeg)", body, contentType)
@@ -590,8 +588,8 @@ func TestFetchableSeriesURL(t *testing.T) {
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := FetchableSeriesURL(tt.site, tt.seriesURL); got != tt.want {
t.Errorf("FetchableSeriesURL(%q, %q) = %v, want %v",
if got := fetchableSeriesURL(tt.site, tt.seriesURL); got != tt.want {
t.Errorf("fetchableSeriesURL(%q, %q) = %v, want %v",
tt.site, tt.seriesURL, got, tt.want)
}
})
@@ -665,7 +663,7 @@ func TestNovelfullUsesTLSWhenNoBrowserFetcher(t *testing.T) {
if err != nil || !found {
t.Fatalf("Get: %v found=%v", err, found)
}
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(novelfullCoverURL); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
if got.LatestChapterNum == nil || *got.LatestChapterNum != 2334 {
@@ -813,7 +811,7 @@ func TestRunOncePrefetchesKaganeCover(t *testing.T) {
p.runOnce(context.Background())
p.waitCovers()
body, contentType, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes")))
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL))
if err != nil || !ok {
t.Fatalf("CoverByAddress: %v found=%v", err, ok)
}
@@ -823,7 +821,7 @@ func TestRunOncePrefetchesKaganeCover(t *testing.T) {
if got := covers.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1", got)
}
if got := readBookmark(t, s, key); got.Cover != testCoverBaseURL+"/covers/"+store.CoverAddressForBytes([]byte("cover-bytes")) {
if got := readBookmark(t, s, key); got.Cover != testCoverBaseURL+"/covers/"+store.CoverAddress(coverURL) {
t.Fatalf("wire Cover = %q, want content-addressed URL", got.Cover)
}
}
@@ -916,7 +914,7 @@ func TestRunOnceRejectsInvalidKaganeCover(t *testing.T) {
p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("not an image"))); err != nil || found {
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
t.Fatalf("invalid cover persisted = %v, err %v; want missing", found, err)
}
}
@@ -942,7 +940,7 @@ func TestRunOnceWithoutCoverFetcherStillPollsKagane(t *testing.T) {
p.runOnce(context.Background())
p.waitCovers()
if _, _, found, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes"))); err != nil || found {
if _, _, found, err := s.CoverByAddress(store.CoverAddress(coverURL)); err != nil || found {
t.Fatalf("cover after nil CoverFetch = found %v, err %v; want missing", found, err)
}
}
@@ -1021,8 +1019,8 @@ func TestFetchableSeriesURLPinsNovelHosts(t *testing.T) {
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := FetchableSeriesURL(tc.site, tc.url); got != tc.want {
t.Fatalf("FetchableSeriesURL(%q, %q) = %v, want %v", tc.site, tc.url, got, tc.want)
if got := fetchableSeriesURL(tc.site, tc.url); got != tc.want {
t.Fatalf("fetchableSeriesURL(%q, %q) = %v, want %v", tc.site, tc.url, got, tc.want)
}
})
}
@@ -1040,6 +1038,7 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
seriesURL string
kind string
body string
wantCover string
browser bool
}{
{
@@ -1048,12 +1047,14 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
seriesID: "chronicles-of-the-demon-faction-f886a8af",
seriesURL: "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af",
kind: store.KindManga, body: asuraSeriesFixture + asuraCoverFixture,
wantCover: "https://cdn.asurascans.com/asura-images/covers/chronicles-of-the-demon-faction.d4dcb8.webp",
},
{
name: "lightnovelworld novel",
key: "lightnovelworld:all-jobs-and-classes-i-just-wanted-one-skill-not-them-all", site: "lightnovelworld",
seriesID: "all-jobs-and-classes-i-just-wanted-one-skill-not-them-all", seriesURL: "https://lightnovelworld.net/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/",
kind: store.KindNovel, body: lnwSeriesFixture + lnwCoverFixture,
wantCover: "https://i1.wp.com/lightnovelworld.net/wp-content/uploads/2025/10/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all.jpg",
},
{
name: "kagane manga",
@@ -1061,6 +1062,7 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
seriesID: "019fe11a-8670-7cf3-8343-0b02057d3787",
seriesURL: "https://kagane.to/series/019fe11a-8670-7cf3-8343-0b02057d3787",
kind: store.KindManga, body: kaganeAPIFixtureWithCover, browser: true,
wantCover: "https://kagane.to/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed",
},
}
for _, tc := range cases {
@@ -1084,7 +1086,7 @@ func TestRunOnceFillsBlankCoverFromSeriesPage(t *testing.T) {
p.waitCovers()
got := readBookmark(t, s, tc.key)
wantWire := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes"))
wantWire := testCoverBaseURL + "/covers/" + store.CoverAddress(tc.wantCover)
if got.Cover != wantWire {
t.Fatalf("Cover = %q, want %q", got.Cover, wantWire)
}
@@ -1141,7 +1143,7 @@ func TestRunOnceDoesNotReplaceExistingCover(t *testing.T) {
t.Fatalf("cover fetch calls = %d, want 0", got)
}
got := readBookmark(t, s, key)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("first")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(first); got.Cover != want {
t.Fatalf("Cover = %q, want the first one %q", got.Cover, want)
}
}
@@ -1188,7 +1190,7 @@ func TestRunOnceRetriesFailedBlankCoverOnNextPoll(t *testing.T) {
t.Fatalf("cover fetch calls after retry = %d, want 2", got)
}
got := readBookmark(t, s, key)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(coverURL); got.Cover != want {
t.Fatalf("Cover after retry = %q, want %q", got.Cover, want)
}
}
@@ -2335,363 +2337,3 @@ func TestForcedSeriesWakesSleepingBrowser(t *testing.T) {
t.Fatalf("browser fetches with a forced series = %d, want 2 (the lane wakes)", got)
}
}
// A forced pass accepts the page as it now stands, so it writes the Cover
// through the replace path; an ordinary pass still only fills a blank one
// (issue #135).
func TestRunOnceForcedPassReplacesExistingCover(t *testing.T) {
s, _ := newTestStore(t)
const (
key = "asura:chronicles-of-the-demon-faction-f886a8af"
seriesID = "chronicles-of-the-demon-faction-f886a8af"
seriesURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
first = "https://cdn.example/covers/first.jpg"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID, SeriesURL: seriesURL, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.SetSeriesCover("asura", seriesID, first, []byte("first"), "image/jpeg"); err != nil {
t.Fatalf("seed cover: %v", err)
}
at := time.UnixMilli(5_000_000)
public := &fakeBytesCoverFetcher{body: []byte("second"), contentType: "image/jpeg"}
t.Run("unforced pass leaves the cover alone", func(t *testing.T) {
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: public,
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
if got := public.callCount(); got != 0 {
t.Fatalf("cover fetch calls = %d, want 0", got)
}
got := readBookmark(t, s, key)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("first")); got.Cover != want {
t.Fatalf("Cover = %q, want the first one %q", got.Cover, want)
}
})
t.Run("forced pass replaces the cover", func(t *testing.T) {
if err := s.ForceSeriesPoll("asura", seriesID, at.Add(time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: public,
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
if got := public.callCount(); got != 1 {
t.Fatalf("cover fetch calls = %d, want 1", got)
}
got := readBookmark(t, s, key)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("second")); got.Cover != want {
t.Fatalf("Cover = %q, want the second one %q", got.Cover, want)
}
body, _, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("second")))
if err != nil || !ok {
t.Fatalf("CoverByAddress: %v found=%v", err, ok)
}
if string(body) != "second" {
t.Fatalf("stored cover = %q, want second", body)
}
})
}
// Identical artwork re-served is an honest no-op the caller can tell apart
// from a replacement: the address comes from the bytes, so the row cannot
// change in substance, and the replace call site reports the three outcomes
// distinctly (issue #135).
func TestRunOnceForcedPassIdenticalBytesLogsNoOp(t *testing.T) {
s, _ := newTestStore(t)
const (
key = "asura:chronicles-of-the-demon-faction-f886a8af"
seriesID = "chronicles-of-the-demon-faction-f886a8af"
seriesURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
first = "https://cdn.example/covers/first.jpg"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID, SeriesURL: seriesURL, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.SetSeriesCover("asura", seriesID, first, []byte("cover-bytes"), "image/jpeg"); err != nil {
t.Fatalf("seed cover: %v", err)
}
at := time.UnixMilli(5_000_000)
if err := s.ForceSeriesPoll("asura", seriesID, at.Add(time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"},
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
got := readBookmark(t, s, key)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
t.Fatalf("Cover = %q, want unchanged %q", got.Cover, want)
}
if body, _, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes"))); err != nil || !ok || string(body) != "cover-bytes" {
t.Fatalf("stored cover after no-op: found=%v err=%v", ok, err)
}
logged := logs.String()
if !strings.Contains(logged, "cover unchanged") {
t.Fatalf("no-op not reported as unchanged; log:\n%s", logged)
}
if strings.Contains(logged, "cover replaced") {
t.Fatalf("no-op reported as a replacement; log:\n%s", logged)
}
}
// A Series whose sharded Cover file was unlinked out from under it is
// repaired by one forced pass: same bytes mean the same address and the file
// re-linked (issue #135, story 32).
func TestRunOnceForcedPassRelinksUnlinkedCoverFile(t *testing.T) {
coverDir := t.TempDir()
url := pgtest.URL(t)
s, err := store.Open(url, testOwner, coverDir, testCoverBaseURL)
if err != nil {
t.Fatalf("Open: %v", err)
}
t.Cleanup(func() { s.Close() })
const (
key = "asura:chronicles-of-the-demon-faction-f886a8af"
seriesID = "chronicles-of-the-demon-faction-f886a8af"
seriesURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
first = "https://cdn.example/covers/first.jpg"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID, SeriesURL: seriesURL, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.SetSeriesCover("asura", seriesID, first, []byte("cover-bytes"), "image/jpeg"); err != nil {
t.Fatalf("seed cover: %v", err)
}
address := store.CoverAddressForBytes([]byte("cover-bytes"))
coverPath := filepath.Join(coverDir, filepath.FromSlash(address[:2]+"/"+address[2:4]+"/"+address))
if err := os.Remove(coverPath); err != nil {
t.Fatalf("unlink cover file: %v", err)
}
if _, _, ok, err := s.CoverByAddress(address); err != nil || ok {
t.Fatalf("CoverByAddress after unlink = found %v err %v; want missing (file gone)", ok, err)
}
at := time.UnixMilli(5_000_000)
if err := s.ForceSeriesPoll("asura", seriesID, at.Add(time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("cover-bytes"), contentType: "image/jpeg"},
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
body, contentType, ok, err := s.CoverByAddress(address)
if err != nil || !ok {
t.Fatalf("CoverByAddress after forced pass: found=%v err=%v; want the file re-linked", ok, err)
}
if string(body) != "cover-bytes" || contentType != "image/jpeg" {
t.Fatalf("re-linked cover = (%q, %q), want (cover-bytes, image/jpeg)", body, contentType)
}
}
// A forced pass degrades exactly like an ordinary one when the cover sidecar
// is unreachable: the fetch is skipped and logged, the chapter poll is
// untouched, and the stored Cover is not moved (issue #135, story 6).
func TestRunOnceForcedPassWithoutCoverFetcherStillPolls(t *testing.T) {
s, _ := newTestStore(t)
const (
key = "kagane:019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
seriesID = "019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "kagane", SeriesID: seriesID,
SeriesURL: "https://kagane.to/series/" + seriesID, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.SetSeriesCover("kagane", seriesID, "https://kagane.to/api/v2/image/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b/compressed", []byte("existing"), "image/webp"); err != nil {
t.Fatalf("seed cover: %v", err)
}
at := time.UnixMilli(5_000_000)
if err := s.ForceSeriesPoll("kagane", seriesID, at.Add(time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
p := &Poller{
Store: s, BrowserFetch: &fakeFetcher{body: kaganeAPIFixtureWithCover, status: 200},
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
got := readBookmark(t, s, key)
if got.LatestChapterNum == nil || *got.LatestChapterNum != 41 {
t.Fatalf("LatestChapterNum = %v, want 41", got.LatestChapterNum)
}
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("existing")); got.Cover != want {
t.Fatalf("Cover = %q, want the existing one %q untouched", got.Cover, want)
}
if checked := readLatestCheckedAt(t, s, key); checked != at.UnixMilli() {
t.Fatalf("latest_checked_at = %d, want %d", checked, at.UnixMilli())
}
if !strings.Contains(logs.String(), "fetch cover") {
t.Fatalf("missing skipped-cover log; log:\n%s", logs.String())
}
}
// A forced replacement strands the previous address, and the poller reclaims
// it from the stranded branch: the old sharded file and covers row are both
// gone once the pass lands while the new bytes read back (issue #154). The
// identical-bytes no-op that follows reclaims nothing: previous == current
// there, and a reclamation would delete the Cover the pass just wrote.
func TestRunOnceForcedPassReclaimsSupersededCover(t *testing.T) {
coverDir := t.TempDir()
url := pgtest.URL(t)
s, err := store.Open(url, testOwner, coverDir, testCoverBaseURL)
if err != nil {
t.Fatalf("Open: %v", err)
}
t.Cleanup(func() { s.Close() })
const (
key = "asura:chronicles-of-the-demon-faction-f886a8af"
seriesID = "chronicles-of-the-demon-faction-f886a8af"
seriesURL = "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af"
first = "https://cdn.example/covers/first.jpg"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID, SeriesURL: seriesURL, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.SetSeriesCover("asura", seriesID, first, []byte("first"), "image/jpeg"); err != nil {
t.Fatalf("seed cover: %v", err)
}
stale := store.CoverAddressForBytes([]byte("first"))
stalePath := filepath.Join(coverDir, filepath.FromSlash(stale[:2]+"/"+stale[2:4]+"/"+stale))
at := time.UnixMilli(5_000_000)
if err := s.ForceSeriesPoll("asura", seriesID, at.Add(time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("second"), contentType: "image/jpeg"},
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
// The stranded address is gone on disk and in SQL; the new one reads back.
if _, err := os.Stat(stalePath); !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("stale sharded file after replacement = %v, want fs.ErrNotExist", err)
}
if _, _, ok, err := s.CoverByAddress(stale); err != nil || ok {
t.Fatalf("stale bytes after replacement = found %v err %v, want reclaimed", ok, err)
}
if body, _, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("second"))); err != nil || !ok || string(body) != "second" {
t.Fatalf("new bytes after replacement = found %v err %v, want served", ok, err)
}
// The identical-bytes pass is an honest no-op and reclaims nothing.
if err := s.ForceSeriesPoll("asura", seriesID, at.Add(2*time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
p.runOnce(context.Background())
p.waitCovers()
current := store.CoverAddressForBytes([]byte("second"))
currentPath := filepath.Join(coverDir, filepath.FromSlash(current[:2]+"/"+current[2:4]+"/"+current))
if _, err := os.Stat(currentPath); err != nil {
t.Fatalf("live sharded file after no-op pass = %v, want present", err)
}
if body, _, ok, err := s.CoverByAddress(current); err != nil || !ok || string(body) != "second" {
t.Fatalf("bytes after no-op pass = found %v err %v, want still served", ok, err)
}
}
// A reclamation that fails must not fail the Poll: the failure is logged
// against the Series and the replacement still lands, so the stranded bytes
// stay reachable for a retry and the owner's act succeeded (issue #154).
func TestRunOnceForcedPassReclaimFailureDoesNotFailPoll(t *testing.T) {
coverDir := t.TempDir()
url := pgtest.URL(t)
s, err := store.Open(url, testOwner, coverDir, testCoverBaseURL)
if err != nil {
t.Fatalf("Open: %v", err)
}
t.Cleanup(func() { s.Close() })
const (
key = "asura:chronicles-of-the-fallen-f886a8af"
seriesID = "chronicles-of-the-fallen-f886a8af"
seriesURL = "https://asurascans.com/comics/chronicles-of-the-fallen-f886a8af"
first = "https://cdn.example/covers/first.jpg"
)
if _, err := s.Upsert(s.OwnerID(), store.Bookmark{
Key: key, Site: "asura", SeriesID: seriesID, SeriesURL: seriesURL, UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.SetSeriesCover("asura", seriesID, first, []byte("first"), "image/jpeg"); err != nil {
t.Fatalf("seed cover: %v", err)
}
// Make the stranded file unremovable: a non-empty directory in its place.
stale := store.CoverAddressForBytes([]byte("first"))
stalePath := filepath.Join(coverDir, filepath.FromSlash(stale[:2]+"/"+stale[2:4]+"/"+stale))
if err := os.Remove(stalePath); err != nil {
t.Fatalf("clear file: %v", err)
}
if err := os.Mkdir(stalePath, 0o755); err != nil {
t.Fatalf("replace file with dir: %v", err)
}
if err := os.WriteFile(filepath.Join(stalePath, "blob"), []byte("x"), 0o644); err != nil {
t.Fatalf("fill dir: %v", err)
}
at := time.UnixMilli(5_000_000)
if err := s.ForceSeriesPoll("asura", seriesID, at.Add(time.Hour).UnixMilli()); err != nil {
t.Fatalf("ForceSeriesPoll: %v", err)
}
var logs strings.Builder
prev := log.Writer()
log.SetOutput(&logs)
t.Cleanup(func() { log.SetOutput(prev) })
p := &Poller{
Store: s, Fetch: &fakeFetcher{body: asuraSeriesFixture + asuraCoverFixture, status: 200},
CoverBytesFetch: &fakeBytesCoverFetcher{body: []byte("second"), contentType: "image/jpeg"},
Now: func() time.Time { return at },
}
p.runOnce(context.Background())
p.waitCovers()
logged := logs.String()
if !strings.Contains(logged, "reclaim cover") {
t.Fatalf("failed reclamation not logged; log:\n%s", logged)
}
if !strings.Contains(logged, "cover replaced") {
t.Fatalf("replacement not reported after a failed reclaim; log:\n%s", logged)
}
got := readBookmark(t, s, key)
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("second")); got.Cover != want {
t.Fatalf("Cover = %q, want the replacement %q", got.Cover, want)
}
}
+1 -1
View File
@@ -41,7 +41,7 @@ var (
// its own network position to whatever URL a token-holder writes, including
// link-local/internal addresses or non-https schemes.
func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fetcher) (seriesRead, error) {
if !FetchableSeriesURL(site, seriesURL) {
if !fetchableSeriesURL(site, seriesURL) {
return seriesRead{}, fmt.Errorf("%w: site=%q url=%q", errNotFetchable, site, seriesURL)
}
f := fetcherFor(site, browser, tls)
+1 -1
View File
@@ -46,7 +46,7 @@ type site struct {
type browserRead struct {
// Read builds the tab read for seriesURL, refusing (false) an address
// this Site will not open in a browser — the per-Site half of the SSRF
// gate, kept deliberately behind FetchableSeriesURL: a headless browser
// gate, kept deliberately behind fetchableSeriesURL: a headless browser
// executes JavaScript and carries cookies, and series_url is
// client-supplied.
Read func(seriesURL string, out *string) (chromedp.Action, bool)
+8 -12
View File
@@ -4,7 +4,6 @@ import (
"context"
"net/http"
"os"
"strings"
"testing"
"time"
@@ -109,7 +108,10 @@ func TestSmokeAcquireKaganeCover(t *testing.T) {
if ws == "" {
t.Skip("SMOKE_BROWSER_WS_URL unset")
}
const seriesID = "019fe11a-8670-7cf3-8343-0b02057d3787"
const (
seriesID = "019fe11a-8670-7cf3-8343-0b02057d3787"
coverURL = "https://kagane.to/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed"
)
s, _ := newTestStore(t)
bf, err := NewBrowserFetcher(ws)
if err != nil {
@@ -138,13 +140,12 @@ func TestSmokeAcquireKaganeCover(t *testing.T) {
if err != nil || !found {
t.Fatalf("Get: %v found=%v", err, found)
}
addr, ok := strings.CutPrefix(got.Cover, testCoverBaseURL+"/covers/")
if !ok {
t.Fatalf("Cover = %q, want an address on %q — the acquire path did not store the browser-fetched bytes", got.Cover, testCoverBaseURL+"/covers/")
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(coverURL); got.Cover != want {
t.Fatalf("Cover = %q, want %q — the acquire path did not store the browser-fetched bytes", got.Cover, want)
}
body, contentType, ok, err := s.CoverByAddress(addr)
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL))
if err != nil || !ok {
t.Fatalf("CoverByAddress(%q): %v found=%v", addr, err, ok)
t.Fatalf("CoverByAddress: %v found=%v", err, ok)
}
if len(body) < 1000 {
t.Fatalf("stored cover is %d bytes, want a real image", len(body))
@@ -152,10 +153,5 @@ func TestSmokeAcquireKaganeCover(t *testing.T) {
if contentType != "image/webp" {
t.Fatalf("content type = %q, want image/webp", contentType)
}
// The address the row carries is the bytes' own SHA-256: a re-art behind
// the same URL would be a different address, which is the whole point.
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes(body); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
t.Logf("stored %d bytes of %s", len(body), contentType)
}
+1 -1
View File
@@ -40,7 +40,7 @@ func TestSmokeLnwCommentBoundary(t *testing.T) {
if seriesURL == "" {
t.Skip("SMOKE_LNW_SERIES_URL unset")
}
if !FetchableSeriesURL("lightnovelworld", seriesURL) {
if !fetchableSeriesURL("lightnovelworld", seriesURL) {
t.Fatalf("%q is not a fetchable lightnovelworld series URL", seriesURL)
}
+6 -13
View File
@@ -41,8 +41,7 @@ type SeriesFilter struct {
// must never leave the store package — so the projection does not select it,
// and only the anonymous boolean in raisedByReaderAnswer crosses it.
const adminSeriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.force_poll_at,
s.latest_corrected_at`
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.force_poll_at`
// raisedByReaderAnswer answers "did a Reader's report set this number" without
// naming which Reader. Kept apart from adminSeriesColumns so the column list —
@@ -72,15 +71,9 @@ type AdminSeries struct {
// ForcePollAt is the owner's "check now" request stamp (issue #146), zero
// meaning never asked. Pending is derived, never stored: a request is
// pending while ForcePollAt is newer than LatestCheckedAt.
ForcePollAt int64
// LatestCorrectedAt is the correction stamp (issue #149): non-zero means
// the Latest Chapter is the owner's, zero means never corrected. The
// provenance line (#152) derives from it, so the zero-means-never meaning
// is load-bearing.
LatestCorrectedAt int64
ReaderCount int
RaisedByReader bool // a Reader's report set LatestChapterNum
ForcePollAt int64
ReaderCount int
RaisedByReader bool // a Reader's report set LatestChapterNum
}
// SeriesPage is one page of the owner's filtered Series list plus the count
@@ -196,7 +189,7 @@ func (s *Store) SeriesPage(f SeriesFilter) (SeriesPage, error) {
`+where+`
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
s.force_poll_at, s.latest_corrected_at, s.latest_raised_by
s.force_poll_at, s.latest_raised_by
`+having+`
ORDER BY s.latest_checked_at, s.site, s.series_id
LIMIT $`+strconv.Itoa(base+1)+` OFFSET $`+strconv.Itoa(base+2), args...)
@@ -271,7 +264,7 @@ func scanAdminSeries(scan func(...any) error) (AdminSeries, int, error) {
if err := scan(
&a.Site, &a.SeriesID, &a.Title, &a.SeriesURL, &a.CoverAddress,
&a.Kind, &a.LatestChapter, &latestChapterNum, &a.LatestCheckedAt,
&a.ForcePollAt, &a.LatestCorrectedAt,
&a.ForcePollAt,
&a.RaisedByReader, &a.ReaderCount, &total,
); err != nil {
return AdminSeries{}, 0, err
@@ -1,15 +1,8 @@
-- The Cover splits into two facts. `cover` keeps the third-party address the
-- bytes come from, which is what the refetch path dedupes on; `cover_address`
-- is the content address of the bytes once they are actually stored, and is
-- what the wire's absolute URL is built from.
-- bytes come from, which is what the acquisition path refetches and dedupes
-- on; `cover_address` is the content address of the bytes once they are
-- actually stored, and is what the wire's absolute URL is built from.
--
-- Empty `cover_address` therefore means "no Cover yet" rather than "a Cover
-- that 404s", which is the distinction the API and the UI both depend on.
--
-- The content address was originally the hex SHA-256 of the source URL
-- (ADR-0007). Since ADR-0014 it is the hex SHA-256 of the bytes themselves,
-- so a re-art behind the same URL is a new address. Rows written before
-- ADR-0014 keep their URL-derived addresses; they are never rehashed and heal
-- into byte addressing on their first forced replacement. Both derivations
-- share the 64-hex-digit shape, so the serving guard is unchanged.
ALTER TABLE series ADD COLUMN cover_address text NOT NULL DEFAULT '';
@@ -1,4 +0,0 @@
-- latest_corrected_at is the "the current Latest Chapter is the owner's" stamp
-- (#149). Written by the Correction; zeroed by every machine write of the
-- value. Zero means never corrected.
ALTER TABLE series ADD COLUMN latest_corrected_at bigint NOT NULL DEFAULT 0;
+37 -204
View File
@@ -16,7 +16,6 @@ import (
"strconv"
"strings"
"github.com/jackc/pgx/v5/pgconn"
_ "github.com/jackc/pgx/v5/stdlib"
)
@@ -61,11 +60,10 @@ type Bookmark struct {
// exists once no matter how many bookmarks point at it (ADR-0003).
//
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
// existing Series has them ignored, and only the backend's own Poll may
// change them. The one exception is SeriesURL, which the owner's
// SetSeriesURL may repair (issue #151). Kind and the latest-chapter fields
// are last-write-wins like the bookmark's own fields. Never serialized: the
// wire format is the flat Bookmark (ADR-0004).
// existing Series has them ignored, and only the backend's own Poll may change
// them. Kind and the latest-chapter fields are last-write-wins like the
// bookmark's own fields. Never serialized: the wire format is the flat
// Bookmark (ADR-0004).
type Series struct {
Site string
SeriesID string
@@ -706,112 +704,67 @@ func (s *Store) getCoverByAddress(address string) ([]byte, string, bool, error)
return body, contentType, true, nil
}
func (s *Store) putCover(sourceURL string, body []byte, contentType string) (string, error) {
func (s *Store) putCover(sourceURL string, body []byte, contentType string) error {
stored, ok := CoverContentType(contentType)
if !ok {
return "", fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
return fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
}
contentType = stored
address := CoverAddressForBytes(body)
address := coverSourceAddress(sourceURL)
relativePath := coverRelativePath(address)
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(relativePath))
if err := os.MkdirAll(filepath.Dir(coverPath), 0o755); err != nil {
return "", fmt.Errorf("create cover shard: %w", err)
return fmt.Errorf("create cover shard: %w", err)
}
tmp, err := os.CreateTemp(filepath.Dir(coverPath), ".cover-*")
if err != nil {
return "", fmt.Errorf("create cover temp file: %w", err)
return fmt.Errorf("create cover temp file: %w", err)
}
tmpName := tmp.Name()
defer os.Remove(tmpName)
if _, err := tmp.Write(body); err != nil {
tmp.Close()
return "", fmt.Errorf("write cover temp file: %w", err)
return fmt.Errorf("write cover temp file: %w", err)
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return "", fmt.Errorf("sync cover temp file: %w", err)
return fmt.Errorf("sync cover temp file: %w", err)
}
if err := tmp.Close(); err != nil {
return "", fmt.Errorf("close cover temp file: %w", err)
return fmt.Errorf("close cover temp file: %w", err)
}
if err := os.Link(tmpName, coverPath); err != nil && !errors.Is(err, fs.ErrExist) {
return "", fmt.Errorf("install cover file: %w", err)
return fmt.Errorf("install cover file: %w", err)
}
if _, err := s.db.Exec(`
INSERT INTO covers (address, path, content_type)
VALUES ($1, $2, $3)
ON CONFLICT (address) DO NOTHING`, address, relativePath, contentType); err != nil {
return "", fmt.Errorf("record cover %q: %w", address, err)
}
return address, nil
}
// ReclaimCover permanently removes a Cover nothing references: the sharded
// file first, the covers row last. A blank address is a no-op, and so is any
// address a Series row still points at — byte-identical artwork is one row by
// construction (ADR-0014), so reclaiming one Series' stranded bytes must not
// blank another's. The file goes first because the covers row is the handle:
// an interrupted run stays findable in SQL — covers rows unreferenced by any
// series cover_address — and re-running finishes the job, whereas deleting
// the row first would leave a file nothing names. A concurrent Forced Poll
// repointing a live Series at this address between the guard and the unlink
// is the repairable case: the missing file reads as ok=false and the next
// pass re-installs it. Failures are returned, never logged here — the caller
// logs and carries on — and a failed unlink leaves the row in place for a
// retry. A whole-table sweep, if ever wanted, is one SQL query over covers,
// not a tree walk and not this function.
func (s *Store) ReclaimCover(address string) error {
if address == "" {
return nil
}
var referenced int
err := s.db.QueryRow(`SELECT 1 FROM series WHERE cover_address = $1 LIMIT 1`, address).Scan(&referenced)
if err == nil {
return nil
}
if !errors.Is(err, sql.ErrNoRows) {
return fmt.Errorf("guard reclaim of cover %q: %w", address, err)
}
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(coverRelativePath(address)))
if err := os.Remove(coverPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("remove cover file %q: %w", address, err)
}
if _, err := s.db.Exec(`DELETE FROM covers WHERE address = $1`, address); err != nil {
return fmt.Errorf("delete cover row %q: %w", address, err)
return fmt.Errorf("record cover %q: %w", address, err)
}
return nil
}
// GetCover returns the immutable object a source URL's own hash names. Rows
// written before byte addressing (ADR-0014) are the only ones that ever reach
// it; it hashes the URL, so a byte-addressed Cover is invisible to it. Missing
// GetCover returns the immutable object addressed by its source URL. Missing
// files are reported with ok=false so callers can retry acquisition later.
func (s *Store) GetCover(sourceURL string) ([]byte, string, bool, error) {
return s.getCover(sourceURL)
}
// PutCover persists bytes under their own content address (ADR-0014). A later
// write of the same bytes cannot replace the immutable object.
// PutCover persists bytes under the source URL's content address. A later
// write for the same URL cannot replace the immutable object.
func (s *Store) PutCover(sourceURL string, body []byte, contentType string) error {
_, err := s.putCover(sourceURL, body, contentType)
return err
return s.putCover(sourceURL, body, contentType)
}
// CoverAddressForBytes is the content address body is stored under: the hex
// SHA-256 of the bytes, so identical artwork is one address and a re-art a
// new one. Legacy rows were addressed from their source URL instead and are
// never rehashed — both derivations coexist (ADR-0014).
func CoverAddressForBytes(body []byte) string {
sum := sha256.Sum256(body)
return hex.EncodeToString(sum[:])
}
// CoverAddress is the content address bytes fetched from sourceURL are stored
// under. It is a pure function of the URL, so the acquisition path can name a
// Cover before it has the bytes.
func CoverAddress(sourceURL string) string { return coverSourceAddress(sourceURL) }
// coverAddressRe is the shape of a stored address: 64 lowercase hex digits —
// the hex SHA-256 of the cover bytes, or of the source URL for legacy rows
// (ADR-0014). Request paths reach CoverByAddress, so the shape is checked
// before the value is ever turned into a filesystem path; byte-derived
// addresses keep the same shape, so the guard is unchanged.
// coverAddressRe is the shape of a stored address: the hex SHA-256 of a source
// URL. Request paths reach CoverByAddress, so the shape is checked before the
// value is ever turned into a filesystem path.
var coverAddressRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
// CoverByAddress returns the immutable object at one content address. An
@@ -835,68 +788,24 @@ func (s *Store) CoverWireURL(address string) string {
return s.coverBaseURL + "/covers/" + address
}
// SetSeriesCover stores the bytes and points the Series at their address, but
// only while the Series has no Cover: acquisition at creation and the poll
// both call this, and whichever arrives second must not overwrite the first.
// The bytes themselves are content-addressed and immutable, so storing them
// twice is free. See ReplaceSeriesCover for the write that may move a Cover
// once one exists (ADR-0014).
// SetSeriesCover stores the bytes and points the Series at them, but only
// while the Series has no Cover: acquisition at creation and the poll both
// call this, and whichever arrives second must not overwrite the first. The
// bytes themselves are content-addressed and immutable, so storing them twice
// is free.
func (s *Store) SetSeriesCover(site, seriesID, sourceURL string, body []byte, contentType string) error {
address, err := s.putCover(sourceURL, body, contentType)
if err != nil {
if err := s.putCover(sourceURL, body, contentType); err != nil {
return err
}
if _, err := s.db.Exec(`
UPDATE series SET cover = $3, cover_address = $4
WHERE site = $1 AND series_id = $2 AND cover_address = ''`,
site, seriesID, sourceURL, address); err != nil {
site, seriesID, sourceURL, coverSourceAddress(sourceURL)); err != nil {
return fmt.Errorf("set cover for %q: %w", site+":"+seriesID, err)
}
return nil
}
// ReplaceSeriesCover stores the bytes and points the Series at their address
// whether or not one already exists, writing the current source URL alongside
// — the Forced Poll's installer and the only write that may move a Cover once
// one exists (ADR-0014). previous is the address the row held before the write
// ("" if it had none) and current the address of the bytes just stored; both
// are read and written in one transaction, so a concurrent replacement reports
// the exact displacement. previous == current means the Site served identical
// artwork, an honest no-op; otherwise previous is stranded — the row no
// longer points at it, and reclaiming its bytes is the caller's separate act
// (the poller's replace path calls ReclaimCover on it). This write itself
// removes nothing.
func (s *Store) ReplaceSeriesCover(site, seriesID, sourceURL string, body []byte, contentType string) (previous, current string, err error) {
current, err = s.putCover(sourceURL, body, contentType)
if err != nil {
return "", "", err
}
tx, err := s.db.Begin()
if err != nil {
return "", "", fmt.Errorf("begin replace cover for %q: %w", site+":"+seriesID, err)
}
defer tx.Rollback()
err = tx.QueryRow(`
SELECT cover_address FROM series
WHERE site = $1 AND series_id = $2 FOR UPDATE`,
site, seriesID).Scan(&previous)
if errors.Is(err, sql.ErrNoRows) {
previous = ""
} else if err != nil {
return "", "", fmt.Errorf("read cover for %q: %w", site+":"+seriesID, err)
}
if _, err := tx.Exec(`
UPDATE series SET cover = $3, cover_address = $4
WHERE site = $1 AND series_id = $2`,
site, seriesID, sourceURL, current); err != nil {
return "", "", fmt.Errorf("replace cover for %q: %w", site+":"+seriesID, err)
}
if err := tx.Commit(); err != nil {
return "", "", fmt.Errorf("commit cover replace for %q: %w", site+":"+seriesID, err)
}
return previous, current, nil
}
// List returns every bookmark of one reader, newest activity first.
// Series-owned fields are joined in, so each Bookmark reads back whole and
// flat (ADR-0004).
@@ -995,11 +904,6 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
// xmax is zero only on a row this statement inserted, which is how a
// Series nobody had bookmarked before is told apart from one that already
// existed — DO UPDATE returns a row either way.
// latest_corrected_at is the one clause conditional on the value moving
// (#149): after a Correction a Reader's cached row holds the corrected
// number and resends it on the next Progress PUT, so unconditional
// zeroing would erase the fact while the value is still the owner's. The
// stamp survives a same-number PUT and dies the moment the number moves.
var created bool
if err := tx.QueryRow(`
INSERT INTO series (site, series_id, title, series_url, kind,
@@ -1010,10 +914,7 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
ON CONFLICT (site, series_id) DO UPDATE SET
kind=excluded.kind,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num,
latest_corrected_at = CASE
WHEN series.latest_chapter_num IS DISTINCT FROM excluded.latest_chapter_num
THEN 0 ELSE series.latest_corrected_at END
latest_chapter_num=excluded.latest_chapter_num
RETURNING xmax = 0`,
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Kind,
b.LatestChapter, latestNum).Scan(&created); err != nil {
@@ -1083,37 +984,6 @@ func (s *Store) Delete(readerID int64, key string) error {
return nil
}
// pgForeignKeyViolation is the SQLSTATE the driver surfaces when a Bookmark
// row refuses a Series delete (bookmarks_series_fk). pgconn exports no named
// constant for it, so the store names it here.
const pgForeignKeyViolation = "23503"
// ErrSeriesHasBookmarks is RemoveSeries' refusal: a Reader still holds the
// Series, so the owner's removal must not reach past that record. The
// delete is the check — no NOT EXISTS pre-check that can race the insert —
// and the driver's foreign-key violation is translated here so no driver
// type escapes the store (issue #155).
var ErrSeriesHasBookmarks = errors.New("series has bookmarks")
// RemoveSeries deletes one Series row by (site, series_id). It is refused
// while any Bookmark references the row; deleting an absent key is not an
// error, matching Delete. The caller owns the stranded Cover: read the row's
// cover_address before the delete and call ReclaimCover after it — the
// helper's guard cannot pass while the series row still points at the
// address, so the order is the sequence, not a preference.
func (s *Store) RemoveSeries(site, seriesID string) error {
if _, err := s.db.Exec(
`DELETE FROM series WHERE site = $1 AND series_id = $2`,
site, seriesID); err != nil {
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == pgForeignKeyViolation {
return ErrSeriesHasBookmarks
}
return fmt.Errorf("remove series %s:%s: %w", site, seriesID, err)
}
return nil
}
// RecordLanePass appends one pass and prunes every older row in the same
// transaction. retainBefore is supplied by the poller's clock.
func (s *Store) RecordLanePass(p LanePass, retainBefore int64) error {
@@ -1450,13 +1320,10 @@ func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
// SetLatestChapter records the newest chapter the poll found on a series page.
// The poller walks Series rather than Bookmarks, so this is a series-level
// write: the row is shared, and updating it once refreshes every bookmark that
// joins to it. Touching a missing series is not an error. The correction stamp
// is zeroed unconditionally: checkOne only calls this when the number differs,
// so a second copy of the condition would drift (#149).
// joins to it. Touching a missing series is not an error.
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
if _, err := s.db.Exec(
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4,
latest_corrected_at = 0
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
WHERE site = $1 AND series_id = $2`,
site, seriesID, label, num); err != nil {
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
@@ -1464,42 +1331,8 @@ func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) erro
return nil
}
// SetSeriesURL stores the owner's repair for a Series' source address
// (issue #151): the one write that lifts the write-once rule documented on
// Series.SeriesURL. It is a store, not a verification — the caller has
// already passed the poller's fetch gate. The handler 404s on an unknown row
// before calling; the write itself is a plain single-column UPDATE like
// MarkLatestChecked.
func (s *Store) SetSeriesURL(site, seriesID, seriesURL string) error {
if _, err := s.db.Exec(
`UPDATE series SET series_url = $3 WHERE site = $1 AND series_id = $2`,
site, seriesID, seriesURL); err != nil {
return fmt.Errorf("set series url %s:%s: %w", site, seriesID, err)
}
return nil
}
// CorrectLatestChapter makes the Latest Chapter the owner's: one UPDATE
// carrying the number, the derived label and the correction stamp. The label
// shape is the poller's and the userscript's ("Chapter " + the number as
// printed), so chapterLeadIn strips it and the UI renders "Ch N" with no
// special case. latest_checked_at is not touched: a Correction is not a check.
// A raising Reader is cleared without judgement: the number is the owner's
// now, and no Sighting counter moves (spec #135).
func (s *Store) CorrectLatestChapter(site, seriesID string, num float64, at int64) error {
if _, err := s.db.Exec(
`UPDATE series SET
latest_chapter = $3,
latest_chapter_num = $4,
latest_corrected_at = $5,
latest_raised_by = NULL
WHERE site = $1 AND series_id = $2`,
site, seriesID, "Chapter "+strconv.FormatFloat(num, 'f', -1, 64), num, at); err != nil {
return fmt.Errorf("correct latest chapter %s:%s: %w", site, seriesID, err)
}
return nil
}
// RecordSighting notes that a Reader's browser reported this Series' Latest
// Chapter, which is the half of a Sighting the client body cannot express
// (issue #103). It must be called *before* the Upsert that stores the reported
// value: the raise test compares against what is still on the row, and after
// the Upsert there is nothing left to compare with. A Series that does not
+12 -562
View File
@@ -5,8 +5,6 @@ import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"errors"
"io/fs"
"os"
"path/filepath"
"strconv"
@@ -852,7 +850,7 @@ func TestSetSeriesCoverDoesNotOverwrite(t *testing.T) {
if err != nil || !ok {
t.Fatalf("Get = %v, %v", ok, err)
}
if want := "https://bookmarks.test/covers/" + CoverAddressForBytes([]byte("first")); got.Cover != want {
if want := "https://bookmarks.test/covers/" + CoverAddress(first); got.Cover != want {
t.Fatalf("Cover = %q, want the first one %q", got.Cover, want)
}
}
@@ -871,7 +869,7 @@ func TestCoverByAddress(t *testing.T) {
t.Fatalf("SetSeriesCover: %v", err)
}
body, contentType, ok, err := store.CoverByAddress(CoverAddressForBytes([]byte("bytes")))
body, contentType, ok, err := store.CoverByAddress(CoverAddress(source))
if err != nil || !ok {
t.Fatalf("CoverByAddress = %v, %v", ok, err)
}
@@ -879,8 +877,8 @@ func TestCoverByAddress(t *testing.T) {
t.Fatalf("CoverByAddress = %q, %q, want the stored bytes", body, contentType)
}
for _, address := range []string{"", "../../etc/passwd", "ZZ" + CoverAddressForBytes([]byte("bytes"))[2:],
CoverAddressForBytes([]byte("never stored"))} {
for _, address := range []string{"", "../../etc/passwd", "ZZ" + CoverAddress(source)[2:],
CoverAddress("never stored")} {
_, _, ok, err := store.CoverByAddress(address)
if err != nil || ok {
t.Fatalf("CoverByAddress(%q) = %v, %v, want a clean miss", address, ok, err)
@@ -915,7 +913,7 @@ func TestUpsertExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
if err != nil {
t.Fatalf("Upsert: %v", err)
}
wantCover := "https://bookmarks.test/covers/" + CoverAddressForBytes([]byte("bytes"))
wantCover := "https://bookmarks.test/covers/" + CoverAddress(acquired)
if got.Title != "Solo Leveling" || got.SeriesURL != "https://asurascans.com/comics/solo" ||
got.Cover != wantCover {
t.Fatalf("stored = %+v, want original title/url/cover kept", got)
@@ -983,7 +981,7 @@ func TestDeleteKeepsSeriesRow(t *testing.T) {
if err != nil {
t.Fatalf("re-upsert: %v", err)
}
wantCover := "https://bookmarks.test/covers/" + CoverAddressForBytes([]byte("bytes"))
wantCover := "https://bookmarks.test/covers/" + CoverAddress(acquired)
if stored.Title != "Solo Leveling" || stored.Cover != wantCover {
t.Fatalf("re-bookmark = %+v, want title/cover from the surviving series row", stored)
}
@@ -1500,9 +1498,9 @@ func TestCoverPersistsAcrossReopen(t *testing.T) {
t.Fatalf("reopen: %v", err)
}
defer second.Close()
got, contentType, ok, err := second.CoverByAddress(CoverAddressForBytes(body))
got, contentType, ok, err := second.GetCover(sourceURL)
if err != nil {
t.Fatalf("CoverByAddress: %v", err)
t.Fatalf("GetCover: %v", err)
}
if !ok || !bytes.Equal(got, body) || contentType != "image/webp" {
t.Fatalf("stored cover = (%q, %q, %v), want (%q, image/webp, true)", got, contentType, ok, body)
@@ -1541,7 +1539,7 @@ func TestCoverIsContentAddressedOnFilesystem(t *testing.T) {
}
defer first.Close()
addressBytes := sha256.Sum256(body)
addressBytes := sha256.Sum256([]byte(sourceURL))
address := hex.EncodeToString(addressBytes[:])
wantPath := filepath.Join(address[:2], address[2:4], address)
@@ -1572,9 +1570,9 @@ func TestCoverStoreAcceptsAnySourceURL(t *testing.T) {
if err := s.PutCover(sourceURL, want, "image/jpeg"); err != nil {
t.Fatalf("PutCover: %v", err)
}
got, contentType, ok, err := s.CoverByAddress(CoverAddressForBytes(want))
got, contentType, ok, err := s.GetCover(sourceURL)
if err != nil {
t.Fatalf("CoverByAddress: %v", err)
t.Fatalf("GetCover: %v", err)
}
if !ok || !bytes.Equal(got, want) || contentType != "image/jpeg" {
t.Fatalf("GetCover = (%q, %q, %v), want (%q, image/jpeg, true)", got, contentType, ok, want)
@@ -1582,7 +1580,7 @@ func TestCoverStoreAcceptsAnySourceURL(t *testing.T) {
if err := s.PutCover("https://cdn.example/not-image", []byte("html"), "text/html"); err == nil {
t.Fatal("PutCover accepted a non-image")
}
if _, _, ok, err := s.CoverByAddress(CoverAddressForBytes([]byte("html"))); err != nil || ok {
if _, _, ok, err := s.GetCover("https://cdn.example/not-image"); err != nil || ok {
t.Fatalf("rejected cover = found %v, err %v; want missing", ok, err)
}
}
@@ -1901,551 +1899,3 @@ func TestDueForLatestCheckForcedDoesNotOverrideURLOrJoin(t *testing.T) {
t.Fatalf("due = %v, want neither the URL-less nor the orphan series", due)
}
}
// A Correction writes the number, the derived label and the stamp, clears the
// raising Reader, and never touches either Sighting counter or the check
// stamp — a Correction is not a check and never judges a Reader (#149).
func TestCorrectLatestChapterStampsClearsAndDoesNotTouchCheckOrMarks(t *testing.T) {
s := newTestStore(t)
other := secondReader(t, s)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 4321_000)
// A Reader raised the number, and carries a mark for it.
if err := s.RecordSighting(other, "asura", "solo", num2(3), 1000); err != nil {
t.Fatalf("RecordSighting: %v", err)
}
if _, err := s.db.Exec(`
UPDATE readers SET sighting_agreements = 5, sighting_disagreements = 2
WHERE id = $1`, other); err != nil {
t.Fatalf("mark reader: %v", err)
}
if err := s.CorrectLatestChapter("asura", "solo", 12.5, 9000); err != nil {
t.Fatalf("CorrectLatestChapter: %v", err)
}
var chapter string
var num float64
var stamp, checkedAt int64
var raisedBy any
if err := s.db.QueryRow(`
SELECT latest_chapter, latest_chapter_num, latest_corrected_at,
latest_checked_at, latest_raised_by
FROM series WHERE site = 'asura' AND series_id = 'solo'`).
Scan(&chapter, &num, &stamp, &checkedAt, &raisedBy); err != nil {
t.Fatalf("read back: %v", err)
}
if chapter != "Chapter 12.5" {
t.Errorf("latest_chapter = %q, want the derived label %q", chapter, "Chapter 12.5")
}
if num != 12.5 {
t.Errorf("latest_chapter_num = %v, want 12.5", num)
}
if stamp != 9000 {
t.Errorf("latest_corrected_at = %d, want 9000", stamp)
}
if checkedAt != 4321_000 {
t.Errorf("latest_checked_at = %d, want the untouched 4321000", checkedAt)
}
if raisedBy != nil {
t.Errorf("latest_raised_by = %v, want the attribution cleared", raisedBy)
}
readers, err := s.Readers()
if err != nil {
t.Fatalf("Readers: %v", err)
}
for _, r := range readers {
if r.ID == other && (r.Agreements != 5 || r.Disagreements != 2) {
t.Errorf("raising reader's marks = %+v, want agreements 5, disagreements 2 unchanged", r)
}
}
}
// The stamp follows the number (spec #135): an Upsert resending the corrected
// value — a Reader's cached row after a correction — keeps it, and an Upsert
// that actually moves the number kills it. Unconditional zeroing would erase
// the fact while the value is still the owner's; that is the whole point of
// the clause.
func TestUpsertCorrectionStampFollowsTheNumber(t *testing.T) {
s := newTestStore(t)
base := Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", Kind: KindManga,
SeriesURL: "https://asurascans.com/comics/solo", UpdatedAt: 1000,
}
if _, err := s.Upsert(s.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err)
}
if err := s.CorrectLatestChapter("asura", "solo", 5, 9000); err != nil {
t.Fatalf("CorrectLatestChapter: %v", err)
}
// Same number back: the value is still the owner's.
same := base
same.LatestChapterNum = num2(5)
if _, err := s.Upsert(s.OwnerID(), same); err != nil {
t.Fatalf("same-number upsert: %v", err)
}
if got := s.latestCorrectedAt(t, "asura", "solo"); got != 9000 {
t.Fatalf("stamp after same-number Upsert = %d, want 9000 kept", got)
}
// A different number: a machine (or a Reader) wrote the value.
moved := base
moved.LatestChapterNum = num2(7)
if _, err := s.Upsert(s.OwnerID(), moved); err != nil {
t.Fatalf("moved upsert: %v", err)
}
if got := s.latestCorrectedAt(t, "asura", "solo"); got != 0 {
t.Fatalf("stamp after moved Upsert = %d, want zeroed", got)
}
}
// The poller's chapter setter zeroes the stamp unconditionally: checkOne only
// calls it when the number differs, so the condition lives upstream and a
// second copy here would drift (#149).
func TestSetLatestChapterZeroesCorrectionStamp(t *testing.T) {
s := newTestStore(t)
seedForCheck(t, s, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := s.CorrectLatestChapter("asura", "solo", 5, 9000); err != nil {
t.Fatalf("CorrectLatestChapter: %v", err)
}
if err := s.SetLatestChapter("asura", "solo", "Chapter 6", 6); err != nil {
t.Fatalf("SetLatestChapter: %v", err)
}
if got := s.latestCorrectedAt(t, "asura", "solo"); got != 0 {
t.Fatalf("stamp after a machine write = %d, want zeroed", got)
}
}
// latestCorrectedAt reads the stamp column for the assertion above.
func (s *Store) latestCorrectedAt(t *testing.T, site, seriesID string) int64 {
t.Helper()
var stamp int64
if err := s.db.QueryRow(
`SELECT latest_corrected_at FROM series WHERE site = $1 AND series_id = $2`,
site, seriesID).Scan(&stamp); err != nil {
t.Fatalf("read stamp: %v", err)
}
return stamp
}
// num2 boxes a chapter number for the Bookmark fields that take a pointer.
func num2(f float64) *float64 { return &f }
// --- Cover addressing (ADR-0014): the address is the bytes' SHA-256 ---
// The address is what makes a re-art visible at all, so the same bytes must
// always name the same address and different bytes different ones — and the
// address must keep the 64-hex-digit shape CoverByAddress's guard still checks
// before any request-supplied value becomes a filesystem path.
func TestCoverAddressForBytesIsDeterministicAndDistinct(t *testing.T) {
first := CoverAddressForBytes([]byte("art"))
again := CoverAddressForBytes([]byte("art"))
other := CoverAddressForBytes([]byte("artwork"))
if first != again {
t.Fatalf("same bytes gave %q then %q, want one address", first, again)
}
if first == other {
t.Fatalf("different bytes gave the same address %q", first)
}
if !coverAddressRe.MatchString(first) {
t.Fatalf("address %q is not the 64-hex-digit shape the serving guard checks", first)
}
}
// ReplaceSeriesCover is the forced-replacement installer: it moves a Cover
// whether or not one exists, writes the source URL alongside it, and reports
// the three outcomes the Forced Poll has to tell apart.
func TestReplaceSeriesCover(t *testing.T) {
store := newTestStore(t)
if _, err := store.Upsert(store.OwnerID(), Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
}); err != nil {
t.Fatalf("seed: %v", err)
}
// A blank Cover: previous is "", and the row points at the new bytes.
previous, current, err := store.ReplaceSeriesCover("asura", "solo",
"https://cdn.asurascans.com/covers/solo.webp", []byte("first-art"), "image/webp")
if err != nil {
t.Fatalf("ReplaceSeriesCover on a blank: %v", err)
}
if previous != "" {
t.Fatalf("previous on a blank = %q, want empty", previous)
}
if want := CoverAddressForBytes([]byte("first-art")); current != want {
t.Fatalf("current = %q, want %q", current, want)
}
if sr := readSeries(t, store, "asura", "solo"); sr.CoverAddress != current ||
sr.Cover != "https://cdn.asurascans.com/covers/solo.webp" {
t.Fatalf("series after blank fill = %+v, want the new address and source URL", sr)
}
// A re-art: previous is the stranded address, current the new one.
previous, current, err = store.ReplaceSeriesCover("asura", "solo",
"https://cdn.asurascans.com/covers/solo-rebrand.webp", []byte("second-art"), "image/jpeg")
if err != nil {
t.Fatalf("ReplaceSeriesCover over a filled Cover: %v", err)
}
if want := CoverAddressForBytes([]byte("first-art")); previous != want {
t.Fatalf("previous = %q, want the replaced address %q", previous, want)
}
if want := CoverAddressForBytes([]byte("second-art")); current != want {
t.Fatalf("current = %q, want %q", current, want)
}
if sr := readSeries(t, store, "asura", "solo"); sr.CoverAddress != current ||
sr.Cover != "https://cdn.asurascans.com/covers/solo-rebrand.webp" {
t.Fatalf("series after replacement = %+v, want the new address and source URL", sr)
}
// The replaced bytes stay served under their old address: ReplaceSeriesCover
// itself reclaims nothing, reclamation is the caller's separate act (#154).
if _, _, ok, err := store.CoverByAddress(CoverAddressForBytes([]byte("first-art"))); err != nil || !ok {
t.Fatalf("superseded bytes = found %v, err %v, want still served", ok, err)
}
// The Site is serving the same artwork again: previous == current is the
// honest no-op the caller reports as "unchanged".
previous, current, err = store.ReplaceSeriesCover("asura", "solo",
"https://cdn.asurascans.com/covers/solo-rebrand.webp", []byte("second-art"), "image/jpeg")
if err != nil {
t.Fatalf("ReplaceSeriesCover over identical bytes: %v", err)
}
if previous != current {
t.Fatalf("identical bytes: previous = %q, current = %q, want one address", previous, current)
}
if want := CoverAddressForBytes([]byte("second-art")); current != want {
t.Fatalf("current = %q, want %q", current, want)
}
}
// Rows written before byte addressing hold the hash of their source URL and
// are never rehashed: GetCover — the poller's heal path — keeps resolving
// them through coverSourceAddress.
func TestGetCoverResolvesLegacyURLDerivedAddress(t *testing.T) {
store := newTestStore(t)
source := "https://cdn.example/legacy.jpg"
legacy := coverSourceAddress(source)
relativePath := coverRelativePath(legacy)
coverPath := filepath.Join(store.coverDir, filepath.FromSlash(relativePath))
if err := os.MkdirAll(filepath.Dir(coverPath), 0o755); err != nil {
t.Fatalf("create shard dir: %v", err)
}
if err := os.WriteFile(coverPath, []byte("legacy-bytes"), 0o644); err != nil {
t.Fatalf("write legacy file: %v", err)
}
if _, err := store.db.Exec(
`INSERT INTO covers (address, path, content_type) VALUES ($1, $2, $3)`,
legacy, relativePath, "image/jpeg"); err != nil {
t.Fatalf("plant legacy row: %v", err)
}
body, contentType, ok, err := store.GetCover(source)
if err != nil || !ok {
t.Fatalf("GetCover on a legacy row = %v, %v, want found", ok, err)
}
if string(body) != "legacy-bytes" || contentType != "image/jpeg" {
t.Fatalf("legacy cover = (%q, %q), want the planted bytes", body, contentType)
}
}
// SetSeriesURL is the one write that lifts the write-once rule of
// Series.SeriesURL (issue #151): a client PUT naming an existing Series still
// has its new URL dropped, yet the owner's repair lands where the Upsert
// would have ignored it.
func TestSetSeriesURLWritesWhereUpsertIgnores(t *testing.T) {
store := newTestStore(t)
base := Bookmark{
Key: "asura:solo", Site: "asura", SeriesID: "solo",
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
UpdatedAt: 1000,
}
if _, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("seed: %v", err)
}
// A client PUT naming the existing Series is refused: the row is shared,
// so the stored URL stands.
base.SeriesURL = "https://evil.example/solo"
if got, err := store.Upsert(store.OwnerID(), base); err != nil {
t.Fatalf("Upsert: %v", err)
} else if got.SeriesURL != "https://asurascans.com/comics/solo" {
t.Fatalf("Upsert stored %q, want the original URL untouched", got.SeriesURL)
}
// The owner's repair writes where the Upsert would have ignored it.
repair := "https://asurascans.com/comics/solo-renumbered"
if err := store.SetSeriesURL("asura", "solo", repair); err != nil {
t.Fatalf("SetSeriesURL: %v", err)
}
sr := readSeries(t, store, "asura", "solo")
if sr.SeriesURL != repair {
t.Fatalf("stored URL = %q, want %q", sr.SeriesURL, repair)
}
}
// --- Cover byte reclamation (issue #154): one guarded helper, file first ---
// coverShardPath is the on-disk location of one address's bytes, built the
// same way getCoverByAddress reads them.
func coverShardPath(t *testing.T, s *Store, address string) string {
t.Helper()
return filepath.Join(s.coverDir, filepath.FromSlash(coverRelativePath(address)))
}
// ReclaimCover removes a Cover nothing references: the row alone is not the
// point — the sharded file must be gone too, because the file is the reclaimed
// disk space.
func TestReclaimCoverRemovesUnreferencedBytes(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := store.SetSeriesCover("asura", "solo", "https://cdn.example/covers/old.jpg", []byte("old-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover: %v", err)
}
old := CoverAddressForBytes([]byte("old-art"))
if _, _, err := store.ReplaceSeriesCover("asura", "solo", "https://cdn.example/covers/new.jpg", []byte("new-art"), "image/jpeg"); err != nil {
t.Fatalf("replace cover: %v", err)
}
if err := store.ReclaimCover(old); err != nil {
t.Fatalf("ReclaimCover: %v", err)
}
if _, err := os.Stat(coverShardPath(t, store, old)); !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("sharded path after reclaim = %v, want fs.ErrNotExist", err)
}
if _, _, ok, err := store.CoverByAddress(old); err != nil || ok {
t.Fatalf("covers row after reclaim = found %v err %v, want gone", ok, err)
}
// The live Cover survives the reclamation of the stranded one.
if body, _, ok, err := store.CoverByAddress(CoverAddressForBytes([]byte("new-art"))); err != nil || !ok || string(body) != "new-art" {
t.Fatalf("new bytes after reclaim = found %v err %v, want still served", ok, err)
}
}
// The guard is the whole design: byte-identical artwork is one covers row by
// construction (ADR-0014), so a second Series pointing at the address must
// keep the bytes — reclaiming one Series' stranded artwork may not blank
// another's.
func TestReclaimCoverSparesReferencedAddress(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
const src = "https://cdn.example/covers/shared.jpg"
addr := CoverAddressForBytes([]byte("shared-art"))
if err := store.SetSeriesCover("asura", "solo", src, []byte("shared-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover: %v", err)
}
// One Series pointing at the address is enough for the guard.
if err := store.ReclaimCover(addr); err != nil {
t.Fatalf("ReclaimCover on a referenced address: %v", err)
}
if body, _, ok, err := store.CoverByAddress(addr); err != nil || !ok || string(body) != "shared-art" {
t.Fatalf("bytes after no-op = found %v err %v, want still served", ok, err)
}
if _, err := os.Stat(coverShardPath(t, store, addr)); err != nil {
t.Fatalf("sharded file after no-op: %v, want present", err)
}
// A second Series serving identical bytes shares the row by construction.
seedForCheck(t, store, "asura:second", "https://asurascans.com/comics/second", 0)
if err := store.SetSeriesCover("asura", "second", src, []byte("shared-art"), "image/jpeg"); err != nil {
t.Fatalf("share cover: %v", err)
}
if err := store.ReclaimCover(addr); err != nil {
t.Fatalf("ReclaimCover on a shared address: %v", err)
}
if body, _, ok, err := store.CoverByAddress(addr); err != nil || !ok || string(body) != "shared-art" {
t.Fatalf("shared bytes after no-op = found %v err %v, want still served", ok, err)
}
if _, err := os.Stat(coverShardPath(t, store, addr)); err != nil {
t.Fatalf("sharded file after shared no-op: %v, want present", err)
}
}
// A blank address is the wire value for "no Cover" (ADR-0007), never a
// reclaimable one.
func TestReclaimCoverBlankAddressIsNoOp(t *testing.T) {
store := newTestStore(t)
if err := store.ReclaimCover(""); err != nil {
t.Fatalf("ReclaimCover(\"\") = %v, want nil", err)
}
}
// An interrupted reclamation is the state the file-first order exists for:
// the row is the handle, so the unreferenced-covers query finds the torn
// Cover and re-running ReclaimCover finishes the job — a missing file is
// "already gone", which counts as success.
func TestReclaimCoverInterruptedRunIsFindableAndFinishes(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := store.SetSeriesCover("asura", "solo", "https://cdn.example/covers/torn.jpg", []byte("torn-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover: %v", err)
}
torn := CoverAddressForBytes([]byte("torn-art"))
if _, _, err := store.ReplaceSeriesCover("asura", "solo", "https://cdn.example/covers/new.jpg", []byte("new-art"), "image/jpeg"); err != nil {
t.Fatalf("replace cover: %v", err)
}
if err := os.Remove(coverShardPath(t, store, torn)); err != nil {
t.Fatalf("unlink mid-reclamation: %v", err)
}
var found string
err := store.db.QueryRow(`
SELECT address FROM covers c
WHERE NOT EXISTS (SELECT 1 FROM series s WHERE s.cover_address = c.address)
LIMIT 1`).Scan(&found)
if err != nil || found != torn {
t.Fatalf("unreferenced-covers query = (%q, %v), want the torn row %q", found, err, torn)
}
if err := store.ReclaimCover(torn); err != nil {
t.Fatalf("re-run over a missing file: %v", err)
}
if _, _, ok, err := store.CoverByAddress(torn); err != nil || ok {
t.Fatalf("row after re-run = found %v err %v, want gone", ok, err)
}
}
// A failed file removal is the one state that is not self-cleaning: the
// covers row must survive so a retry can finish the job, and the store
// returns the error rather than logging — each caller logs and carries on,
// so the failure has no user-facing surface.
func TestReclaimCoverFailedUnlinkKeepsRow(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := store.SetSeriesCover("asura", "solo", "https://cdn.example/covers/stuck.jpg", []byte("stuck-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover: %v", err)
}
stuck := CoverAddressForBytes([]byte("stuck-art"))
if _, _, err := store.ReplaceSeriesCover("asura", "solo", "https://cdn.example/covers/other.jpg", []byte("other-art"), "image/jpeg"); err != nil {
t.Fatalf("replace cover: %v", err)
}
// Make the unlink fail: the sharded path becomes a non-empty directory,
// which os.Remove refuses.
shard := coverShardPath(t, store, stuck)
if err := os.Remove(shard); err != nil {
t.Fatalf("clear file: %v", err)
}
if err := os.Mkdir(shard, 0o755); err != nil {
t.Fatalf("replace file with dir: %v", err)
}
if err := os.WriteFile(filepath.Join(shard, "blob"), []byte("x"), 0o644); err != nil {
t.Fatalf("fill dir: %v", err)
}
if err := store.ReclaimCover(stuck); err == nil {
t.Fatal("ReclaimCover over an unremovable file = nil, want the error")
}
var one int
if err := store.db.QueryRow(`SELECT 1 FROM covers WHERE address = $1`, stuck).Scan(&one); err != nil {
t.Fatal("covers row after failed unlink is gone; want it left for a retry")
}
}
// RemoveSeries is the orphan removal (#155): one Series, one delete, refused
// by the database while any Bookmark points at it. The store translates the
// foreign-key violation into its own sentinel so no driver type escapes, and
// the caller reaps the stranded Cover through ReclaimCover.
func TestRemoveSeriesRemovesOrphanAndReclaimsCover(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := store.Delete(store.OwnerID(), "asura:solo"); err != nil {
t.Fatalf("orphan the series: %v", err)
}
if err := store.SetSeriesCover("asura", "solo", "https://cdn.example/covers/old.jpg", []byte("old-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover: %v", err)
}
addr := CoverAddressForBytes([]byte("old-art"))
if err := store.RemoveSeries("asura", "solo"); err != nil {
t.Fatalf("RemoveSeries: %v", err)
}
// The caller's sequence: the row is deleted first, then the address is
// reclaimed — the guard cannot pass while the row still points at it.
if err := store.ReclaimCover(addr); err != nil {
t.Fatalf("ReclaimCover: %v", err)
}
var one int
if err := store.db.QueryRow(`SELECT 1 FROM series WHERE site = $1 AND series_id = $2`, "asura", "solo").Scan(&one); err != sql.ErrNoRows {
t.Fatalf("series row after remove = %v, want sql.ErrNoRows", err)
}
if _, _, ok, err := store.CoverByAddress(addr); err != nil || ok {
t.Fatalf("covers row after remove = found %v err %v, want gone", ok, err)
}
if _, err := os.Stat(coverShardPath(t, store, addr)); !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("sharded file after remove = %v, want fs.ErrNotExist", err)
}
}
// The refusal is the whole point of the sentinel: a Series a Reader still
// holds is not removed, its row is untouched and its Cover keeps serving.
func TestRemoveSeriesRefusedWhileBookmarked(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
if err := store.SetSeriesCover("asura", "solo", "https://cdn.example/covers/kept.jpg", []byte("kept-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover: %v", err)
}
addr := CoverAddressForBytes([]byte("kept-art"))
if err := store.RemoveSeries("asura", "solo"); !errors.Is(err, ErrSeriesHasBookmarks) {
t.Fatalf("RemoveSeries on a bookmarked series = %v, want ErrSeriesHasBookmarks", err)
}
var held int
if err := store.db.QueryRow(`SELECT 1 FROM series WHERE site = $1 AND series_id = $2`, "asura", "solo").Scan(&held); err != nil {
t.Fatal("series row after refusal is gone; want it untouched")
}
if body, _, ok, err := store.CoverByAddress(addr); err != nil || !ok || string(body) != "kept-art" {
t.Fatalf("cover after refusal = found %v err %v, want still served", ok, err)
}
}
// A Series sharing its Cover address with a second Series is removed while
// the artwork stays readable through CoverByAddress: the series row stops
// referencing the address first, so ReclaimCover's guard passes for this
// caller without touching the shared bytes (ADR-0014).
func TestRemoveSeriesSparesSharedCover(t *testing.T) {
store := newTestStore(t)
seedForCheck(t, store, "asura:solo", "https://asurascans.com/comics/solo", 0)
seedForCheck(t, store, "asura:second", "https://asurascans.com/comics/second", 0)
if err := store.Delete(store.OwnerID(), "asura:solo"); err != nil {
t.Fatalf("orphan solo: %v", err)
}
if err := store.Delete(store.OwnerID(), "asura:second"); err != nil {
t.Fatalf("orphan second: %v", err)
}
const src = "https://cdn.example/covers/shared.jpg"
if err := store.SetSeriesCover("asura", "solo", src, []byte("shared-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover on solo: %v", err)
}
if err := store.SetSeriesCover("asura", "second", src, []byte("shared-art"), "image/jpeg"); err != nil {
t.Fatalf("put cover on second: %v", err)
}
addr := CoverAddressForBytes([]byte("shared-art"))
if err := store.RemoveSeries("asura", "solo"); err != nil {
t.Fatalf("RemoveSeries: %v", err)
}
// The guard spares the shared bytes even though this caller reclaims.
if err := store.ReclaimCover(addr); err != nil {
t.Fatalf("ReclaimCover over a shared address: %v", err)
}
if body, _, ok, err := store.CoverByAddress(addr); err != nil || !ok || string(body) != "shared-art" {
t.Fatalf("shared bytes after remove = found %v err %v, want still served", ok, err)
}
if _, err := os.Stat(coverShardPath(t, store, addr)); err != nil {
t.Fatalf("sharded file after remove: %v, want present", err)
}
var one int
if err := store.db.QueryRow(`SELECT 1 FROM series WHERE site = $1 AND series_id = $2`, "asura", "second").Scan(&one); err != nil {
t.Fatal("the second series row vanished with the first")
}
}
// Deleting an absent key removes nothing and is not an error, matching the
// Delete precedent — the handler's own lookups turn the absent case into the
// 404 before the store ever sees it.
func TestRemoveSeriesMissingKeyIsCleanNoOp(t *testing.T) {
store := newTestStore(t)
if err := store.RemoveSeries("asura", "ghost"); err != nil {
t.Fatalf("RemoveSeries on a missing key = %v, want nil", err)
}
}
-3
View File
@@ -47,9 +47,6 @@ func (h *Handler) adminRoutes() []adminRoute {
{"GET /admin/series", h.adminSeries},
{"GET /admin/series/{key}", h.adminSeriesDetail},
{"POST /admin/series/{key}/poll", h.adminSeriesPoll},
{"POST /admin/series/{key}/latest", h.adminSeriesCorrectLatest},
{"POST /admin/series/{key}/series-url", h.adminSeriesSetURL},
{"POST /admin/series/{key}/remove", h.adminSeriesRemove},
{"POST /admin/lanes/{site}/pause", h.adminLanePause},
{"POST /admin/lanes/{site}/resume", h.adminLaneResume},
{"GET /ui/admin/lanes", h.uiLanes},
+1 -275
View File
@@ -1,10 +1,8 @@
package web
import (
"errors"
"fmt"
"log"
"math"
"net/http"
"net/url"
"strconv"
@@ -49,6 +47,7 @@ var seriesFilterOrder = []string{
}
// seriesListView is the Series list page's data. The template renders strings
// and flags, and every judgement about what a value means is made here.
type seriesListView struct {
Filters []seriesFilterOption
Sites []string
@@ -57,9 +56,6 @@ type seriesListView struct {
FilterLabel string
Rows []seriesRowView
Total int
// OOB marks the out-of-band copy of the heading the removal answer
// carries; on the page itself it is false (issue #155).
OOB bool
// KindBoth / KindManga / KindNovel are the Library segment links, and
// PrevHref / NextHref the pager's, all carrying the active filter, Site
// and Kind so narrowing never drops state.
@@ -104,12 +100,6 @@ type seriesRowView struct {
CanPoll bool
Pending bool
Requested string // "requested 3m ago", rendered only while pending
// CanRemove is the Remove control's visibility: only a Series no Reader
// holds can be removed, so the owner is never offered a button that the
// database will always refuse (issue #155). RemovalRefused marks the one
// raced answer: the row stays and says a fresh Bookmark caught the press.
CanRemove bool
RemovalRefused bool
}
// adminSeries renders the filterable, bookmarkable Series list: filter, Site,
@@ -182,269 +172,6 @@ func (h *Handler) adminSeriesPoll(w http.ResponseWriter, r *http.Request) {
h.render(w, http.StatusOK, "series-row", seriesRow(a, band, time.Now()))
}
// adminSeriesCorrectLatest is the Latest Chapter correction: the owner types
// one number and the Series' Latest Chapter becomes it, stamped as a
// Correction. The number must be a finite float greater than zero — a
// non-numeric, zero or negative value answers 400 and never reaches the
// store, because a bad value would become every Reader's problem. The press
// answers with the freshly rendered meta fragment, so the figures describe
// the state after the press. The owner gate is the route's, not this
// handler's; the body is capped like the API path caps its bodies.
func (h *Handler) adminSeriesCorrectLatest(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
num, err := strconv.ParseFloat(r.PostFormValue("chapter"), 64)
if err != nil || math.IsNaN(num) || math.IsInf(num, 0) || num <= 0 {
http.Error(w, "chapter must be a finite number greater than zero", http.StatusBadRequest)
return
}
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
log.Printf("series correction %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
} else if !found {
http.NotFound(w, r)
return
}
if err := h.store.CorrectLatestChapter(site, seriesID, num, time.Now().UnixMilli()); err != nil {
log.Printf("series correction %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Re-read after the write: the answer must describe the state after the
// press, so the marker reads "corrected just now".
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series correction %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
}
// adminSeriesSetURL is the series URL repair: the owner types one address
// and the Series' Poll fetches it from then on, verified by the same gate
// the poller uses before it fetches anything — a URL failing
// latest.FetchableSeriesURL answers 400 and never reaches the store. The
// repair is a store, not a verification: it performs no outbound fetch, and
// the owner presses Check now afterwards. This lifts the write-once rule of
// Series.SeriesURL for the owner only — a Reader's PUT is still ignored. The
// owner gate is the route's, not this handler's; the body is capped like the
// API path caps its bodies; the key is validated here — a malformed key is a
// 400 and an unknown one a 404.
func (h *Handler) adminSeriesSetURL(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
seriesURL := r.PostFormValue("series_url")
if !latest.FetchableSeriesURL(site, seriesURL) {
http.Error(w, "series URL must be an https address on this site's host", http.StatusBadRequest)
return
}
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
log.Printf("series url %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
} else if !found {
http.NotFound(w, r)
return
}
if err := h.store.SetSeriesURL(site, seriesID, seriesURL); err != nil {
log.Printf("series url %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Re-read after the write: the answer must describe the state after the
// press, like the correction's answer does.
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series url %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
}
// seriesListHeadView is the list heading's data. The template renders it
// inline at the top of the Series list and out of band in the removal answer
// (OOB true, like the chrome partials' OOB flag): the count and the filter
// label are one fact (issue #155).
type seriesListHeadView struct {
Total int
FilterLabel string
OOB bool
}
// adminSeriesRemove is the orphan removal: one Series, one delete, refused by
// the database while any Bookmark exists (translated by the store, never a
// driver error on the page). The owner gate is the route's, not this
// handler's; the body is capped like the API path caps its bodies; the key is
// validated here — a malformed key is a 400 and an unknown one a 404.
//
// The Cover is read from the row before the delete and reclaimed after it:
// ReclaimCover's guard cannot pass while a series row still points at the
// address, so the order is the sequence, not a preference. A reclamation
// failure is not a removal failure — the row is gone and the covers row
// survives for a retry; the handler logs and answers success, because the
// failure has no user-facing surface.
//
// Two callers, one handler, branched on HX-Target like adminSeriesPoll. The
// detail page's remove answers with a navigation — to the No-Readers list on
// success, back to the detail page when a fresh Bookmark raced the press,
// where the new count is visible. The list row's answers with the removed
// row's fragment and the heading re-rendered with the fresh count out of
// band; HX-Reswap deletes the row through the same button that swaps the
// refusal back in, and the count query runs over the press's own filter
// state, so the heading describes the list the owner is looking at.
func (h *Handler) adminSeriesRemove(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
// The row's Cover address is read before the delete because the delete is
// what makes it reclaimable.
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series remove %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
cover := a.CoverAddress
if err := h.store.RemoveSeries(site, seriesID); err != nil {
if errors.Is(err, store.ErrSeriesHasBookmarks) {
// A Bookmark landed between the owner's read and the press: the
// row stays, answered at its new count with the fact spelled
// out — never a 500, and never a deleted row.
if r.Header.Get("HX-Target") == "detail-meta" {
seriesRemoveNavigation(w, r, "/admin/series/"+site+":"+seriesID)
return
}
fresh, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series remove %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
// A second press removed it while this one was refused; the
// row has nothing left to say.
http.NotFound(w, r)
return
}
band := 0
if r.PostFormValue("band") == "1" {
band = 1
}
row := seriesRow(fresh, band, time.Now())
row.RemovalRefused = true
h.render(w, http.StatusOK, "series-row", row)
return
}
log.Printf("series remove %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if err := h.store.ReclaimCover(cover); err != nil {
log.Printf("series remove %s: reclaim cover: %v", site+":"+seriesID, err)
}
if r.Header.Get("HX-Target") == "detail-meta" {
seriesRemoveNavigation(w, r, "/admin/series?filter="+store.SeriesFilterNoReaders)
return
}
// The list answer: the removed row's fragment, plus the heading
// re-rendered with the fresh count. HX-Reswap deletes the row through the
// same button that swaps the refusal back in. The count query failing
// does not undo the removal — log it and answer the row alone.
w.Header().Set("HX-Reswap", "delete")
band := 0
if r.PostFormValue("band") == "1" {
band = 1
}
h.render(w, http.StatusOK, "series-row", seriesRow(a, band, time.Now()))
if head, err := h.seriesListHeadView(r); err != nil {
log.Printf("series remove %s: %v", site+":"+seriesID, err)
} else {
h.render(w, http.StatusOK, "series-list-head", head)
}
}
// seriesListHeadView is the list heading with the count as it stands after a
// removal: the same filter, Site and Kind the press's row carried (the list
// row's button hx-includes the filterbar), so the figure describes the list
// the owner is looking at — the All filter and an unknown one stay the
// absent case. The count is the store's window total, one query.
func (h *Handler) seriesListHeadView(r *http.Request) (seriesListHeadView, error) {
filter := r.PostFormValue("filter")
if _, ok := seriesFilterLabels[filter]; !ok {
filter = store.SeriesFilterAll
}
site := r.PostFormValue("site")
kind := r.PostFormValue("kind")
if kind != store.KindManga && kind != store.KindNovel {
kind = ""
}
data, err := h.store.SeriesPage(store.SeriesFilter{
Site: site,
Kind: kind,
Name: filter,
Cutoff: time.Now().Add(-ownerWindow).UnixMilli(),
Page: 1,
})
if err != nil {
return seriesListHeadView{}, err
}
return seriesListHeadView{
Total: data.Total,
FilterLabel: seriesFilterLabels[filter],
OOB: true,
}, nil
}
// seriesRemoveNavigation answers a removal from the detail page. htmx gets a
// full navigation (HX-Redirect): a bare 303 would be followed by the request
// and the landing page swapped into the press's target, so the header is the
// redirect htmx can see; plain clients get the 303 the ticket names.
func seriesRemoveNavigation(w http.ResponseWriter, r *http.Request, to string) {
if r.Header.Get("HX-Request") != "" {
w.Header().Set("HX-Redirect", to)
return
}
http.Redirect(w, r, to, http.StatusSeeOther)
}
// seriesListView assembles one Series list view from the request's query
// string. An unknown filter value is the absent All case, never an error: the
// select's options are not the only way this URL can be reached.
@@ -569,7 +296,6 @@ func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
Readers: a.ReaderCount,
Band: i%2 == 1,
CanPoll: canPoll,
CanRemove: a.ReaderCount == 0,
Pending: pending,
Requested: requested,
}
+1 -48
View File
@@ -24,26 +24,14 @@ type seriesDetailView struct {
Cover string
Chapter string // Latest Chapter number, or "—" before the first capture
Checked string // how long ago the poller last checked, or "never"
// URL is the stored source address, prefilled into the repair input —
// the one stored string this page renders back into a form (issue #151).
URL string
Readers int
// Corrected is the correction marker's text, "" while no Correction
// stands: "corrected <age> ago" — the copy that says the value is the
// owner's, and it dies with the stamp (a machine write of the number).
Corrected string
// Provenance is the actor class behind the current Chapter: "correction",
// "sighting" or "machine read"; "" while the Series was never read, when
// the line is not rendered. Derived from the same anonymous stamps the
// marks above read — no Reader identity crosses here.
Provenance string
// Marks, one per hygiene fact, rendered only while it holds.
Unpollable bool // no SeriesURL to fetch
NoCover bool
Orphan bool // no Reader holds the Series
SightingRaised bool // a Reader's Sighting set the Latest Chapter
// Poll is the Check now control and the pending marker (issue #146): the
// same derivation and visibility as the list row. CanPoll is false on a
// Series with no page to fetch and on an orphan; Pending is derived —
@@ -52,10 +40,6 @@ type seriesDetailView struct {
CanPoll bool
Pending bool
Requested string
// CanRemove is the Remove control's visibility (issue #155): only a
// Series no Reader holds can be removed, so the owner is never offered a
// button the database will always refuse.
CanRemove bool
}
// adminSeriesDetail renders one Series' page, keyed by the composite
@@ -119,14 +103,12 @@ func (h *Handler) seriesDetailView(a store.AdminSeries) seriesDetailView {
Kind: a.Kind,
Title: a.Title,
Cover: h.store.CoverWireURL(a.CoverAddress),
URL: a.SeriesURL,
Readers: a.ReaderCount,
Unpollable: a.SeriesURL == "",
NoCover: a.CoverAddress == "",
Orphan: a.ReaderCount == 0,
SightingRaised: a.RaisedByReader,
CanPoll: canPoll,
CanRemove: a.ReaderCount == 0,
Pending: pending,
Requested: requested,
}
@@ -140,34 +122,5 @@ func (h *Handler) seriesDetailView(a store.AdminSeries) seriesDetailView {
} else {
v.Checked = since(time.Now(), time.UnixMilli(a.LatestCheckedAt))
}
v.Corrected = correctedAge(time.Now(), a.LatestCorrectedAt)
// Provenance: the actor class behind the current number, evaluated in the
// order the classes outrank one another — the owner's stamp, which a
// Correction leaves standing and a machine write clears (issue #149); a
// raising Reader, which a Correction drops; then any check stamp at all.
// An Acquisition reads as a machine read because it stamps
// latest_checked_at exactly as a Poll does, so the two are
// indistinguishable the moment it finishes; telling them apart would need
// the column this project declines to add (spec #135), and the one
// actionable case — acquired once, never read again — is already the
// unchecked filter.
if a.LatestCorrectedAt != 0 {
v.Provenance = "correction"
} else if a.RaisedByReader {
v.Provenance = "sighting"
} else if a.LatestCheckedAt != 0 {
v.Provenance = "machine read"
}
return v
}
// correctedAge is the correction marker's text: "corrected <age> ago" while
// the stamp is set, "" when zero — zero means never corrected, and the marker
// must not read as history once a machine wrote the number.
func correctedAge(now time.Time, at int64) string {
if at == 0 {
return ""
}
return "corrected " + since(now, time.UnixMilli(at))
}
@@ -1,34 +0,0 @@
package web
import (
"testing"
"bookmarkmanager/backend/internal/store"
)
// seriesDetailView derives the provenance line from the three stamps the
// admin projection already carries: the correction stamp outranks a raising
// Reader, which outranks a check stamp, and a Series with none of the three
// renders no line at all — it was never read, and no actor class is true of
// it. Acquisition stamps latest_checked_at exactly as a Poll does, so an
// acquired value lands in the same "machine read" class (#152).
func TestSeriesDetailViewProvenance(t *testing.T) {
cases := []struct {
name string
a store.AdminSeries
want string
}{
{"correction stamp", store.AdminSeries{LatestCorrectedAt: 1}, "correction"},
{"raising reader only", store.AdminSeries{RaisedByReader: true}, "sighting"},
{"check stamp only", store.AdminSeries{LatestCheckedAt: 1}, "machine read"},
{"correction outranks sighting", store.AdminSeries{LatestCorrectedAt: 1, RaisedByReader: true}, "correction"},
{"none of the three", store.AdminSeries{}, ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := (&Handler{}).seriesDetailView(tc.a).Provenance; got != tc.want {
t.Fatalf("Provenance = %q, want %q", got, tc.want)
}
})
}
}
-28
View File
@@ -653,13 +653,6 @@
margin-left: auto;
}
.admin-sheet .tbl.series .row-msg {
grid-column: 1 / -1;
margin-top: 6px;
color: var(--danger-soft);
font: 400 13px/1.4 var(--font-body);
}
.admin-sheet .detail-back {
display: inline-block;
margin: 18px 0 0;
@@ -727,27 +720,6 @@
margin-top: 8px;
}
.admin-sheet .dform input {
min-width: 0;
padding: 8px 10px;
border: 1px solid var(--field-line);
background: var(--ink);
color: var(--paper);
font: 500 14px var(--font-mono);
outline: none;
}
/* Focus follows the chapter form's idiom — paper, not heat: a red border on
a valid number field reads as "invalid". */
.admin-sheet .dform input:focus {
border-color: var(--paper);
}
.admin-sheet .dform .hint {
margin: 0;
color: var(--mute-2);
font: 500 12px/1.4 var(--font-mono);
letter-spacing: .04em;
}
.admin-sheet .pausebar {
display: flex;
align-items: center;
@@ -1,9 +1,8 @@
{{/* Per-Series page: one address per Series, keyed "<site>:<series_id>" so the
list row is one hop from it. Everything here is a Series-level fact plus
the anonymous Reader count. Check now lands in its own .dform below the
.detail-grid; the correction form is the grid's first column and the URL
repair the second (issue #151). The pending and corrected markers ride the
meta line with the other marks. */}}
the anonymous Reader count. The Check now control lands in its own .dform
below the (empty) .detail-grid; the pending marker rides the meta line
with the other marks. */}}
{{define "series-detail"}}
<a class="ghost detail-back" href="/admin/series">← Series</a>
<h1 class="detail-title">{{.Title}}</h1>
@@ -11,52 +10,26 @@
{{if .Cover}}<div class="cover"><img src="{{.Cover}}" alt="" loading="lazy"></div>
{{else}}<div class="cover"></div>{{end}}
{{template "series-detail-meta" .}}
<div class="detail-grid">
<form class="dform" hx-post="/admin/series/{{.Key}}/latest" hx-target="#detail-meta" hx-swap="outerHTML">
<h3>Correct latest chapter</h3>
<p class="hint">The next successful Poll overwrites this value.</p>
<div class="field">
<input type="number" name="chapter" step="any" placeholder="{{.Chapter}}" required>
<button type="submit" class="ghost">Set</button>
</div>
</form>
<form class="dform" hx-post="/admin/series/{{.Key}}/series-url" hx-target="#detail-meta" hx-swap="outerHTML">
<h3>Repair series URL</h3>
<p class="hint">The Poll fetches this address — storing is not verifying it. A Site-wide host change is a SQL migration, not two hundred forms.</p>
<div class="field">
<input type="url" name="series_url" value="{{.URL}}" required>
<button type="submit" class="ghost">Set</button>
</div>
</form>
</div>
<div class="detail-grid"></div>
{{if .CanPoll}}
<div class="dform">
<div class="field"><a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="#detail-meta" hx-swap="outerHTML" href="#">Check now</a></div>
</div>
{{end}}
{{if .CanRemove}}
<div class="dform">
<div class="field"><button class="ghost danger" hx-post="/admin/series/{{.Key}}/remove" hx-target="#detail-meta" hx-confirm="Removes this series and its stored cover. No Reader has it bookmarked; one re-bookmarking it recreates the row.">Remove</button></div>
</div>
{{end}}
{{end}}
{{/* series-detail-meta is the meta line, and the answer a Check now or
correction press on the detail page swaps into its place: the same marks,
re-rendered after the stamp so the pending and corrected markers — and
the provenance line beside the number — describe the value they sit
next to. */}}
{{/* series-detail-meta is the meta line, and the answer a Check now press on
the detail page swaps into its place: the same marks, re-rendered after
the stamp so the pending marker shows. */}}
{{define "series-detail-meta"}}
<div class="detail-meta" id="detail-meta">
<span>ch {{.Chapter}}</span>
{{if .Provenance}}<span>{{.Provenance}}</span>{{end}}
<span>checked {{.Checked}}</span>
<span>{{.Readers}} readers</span>
{{if .Corrected}}<span class="mark">{{.Corrected}}</span>{{end}}
{{if .Pending}}<span class="mark">{{.Requested}}</span>{{end}}
{{if .Unpollable}}<span class="mark">unpollable</span>{{end}}
{{if .NoCover}}<span class="mark">no cover</span>{{end}}
{{if .Orphan}}<span class="mark">orphan</span>{{end}}
{{if .SightingRaised}}<span class="mark">sighting-raised</span>{{end}}
</div>
{{end}}
{{end}}
@@ -4,7 +4,7 @@
that can be bookmarked: the two selects submit the GET form, and the
Library segment links and the pager preserve the filter and Site. */}}
{{define "series-list"}}
<form class="filterbar" id="filterbar" method="get" action="/admin/series">
<form class="filterbar" method="get" action="/admin/series">
<input type="hidden" name="kind" value="{{.Kind}}">
<label class="fsel"><span>Show</span><select name="filter" onchange="this.form.submit()">
{{range .Filters}}<option value="{{.Name}}"{{if .Selected}} selected{{end}}>{{.Label}} ({{.Count}})</option>{{end}}
@@ -19,7 +19,7 @@
<a href="{{.KindNovel}}"{{if eq .Kind "novel"}} class="active"{{end}}>novels</a>
</span>
</form>
{{template "series-list-head" .}}
<div class="listhead">{{.Total}} series <span class="lbl">· <em>{{.FilterLabel}}</em></span></div>
{{if .Rows}}
<div class="tbl series">
<div class="thead"><span>Site</span><span class="c-ch">Ch</span><span>Checked</span><span class="c-rd">Readers</span><span>Notes</span><span></span></div>
@@ -48,16 +48,6 @@
<span>{{.Age}}</span>
<span class="c-rd">{{.Readers}}</span>
<span class="c-note">{{range .Notes}}<span class="mark">{{.}}</span>{{end}}{{if .More}}<span class="mark mark-faint">+{{.More}}</span>{{end}}</span>
<span class="c-act">{{if .CanPoll}}<a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="closest .trow" hx-swap="outerHTML" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' href="#">Check now</a>{{end}}{{if .CanRemove}}<button class="ghost danger" hx-post="/admin/series/{{.Key}}/remove" hx-target="closest .trow" hx-swap="outerHTML" hx-include="#filterbar" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' hx-confirm="Removes this series and its stored cover. No Reader has it bookmarked; one re-bookmarking it recreates the row.">Remove</button>{{end}}</span>
{{if .RemovalRefused}}<span class="row-msg">a Reader has bookmarked this Series again</span>{{end}}
<span class="c-act">{{if .CanPoll}}<a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="closest .trow" hx-swap="outerHTML" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' href="#">Check now</a>{{end}}</span>
</div>
{{end}}
{{/* series-list-head is the list's heading — the count and the label are one
fact. The removal answer renders it out of band (the OOB flag, like the
chrome partials) so the heading never lies past the row that made it,
and inline here it is the page's own heading. The id is the OOB swap's
hook; hx-swap-oob sits on the element the answer carries. */}}
{{define "series-list-head"}}
<div class="listhead" id="series-listhead"{{if .OOB}} hx-swap-oob="true"{{end}}>{{.Total}} series <span class="lbl">· <em>{{.FilterLabel}}</em></span></div>
{{end}}
+8 -548
View File
@@ -2557,21 +2557,18 @@ func TestSeriesListRowShape(t *testing.T) {
if !strings.Contains(body, `class="c-site site-asura"`) {
t.Errorf("the site cell lacks its site class:\n%s", body)
}
// The action cell carries the Check now control on pollable rows and the
// Remove control on the orphan (#155) — a Series no Reader holds can be
// removed, so the orphan's cell is never empty. No confirm row renders
// in this batch: the confirm gate is htmx's own, not a toggled cell.
if !strings.Contains(body, `<span class="c-act">`) {
t.Errorf("the action cell is not present:\n%s", body)
// The action cell carries the Check now control on pollable rows and is
// empty on the orphan (no URL, no Readers — a button that can never do
// anything is not offered), and no Remove control or confirm row renders
// in this batch.
if !strings.Contains(body, `<span class="c-act"></span>`) {
t.Errorf("the orphan's action cell is not present and empty:\n%s", body)
}
if got := strings.Count(body, "Check now"); got != 2 {
t.Errorf("Check now control count = %d, want 2 (only the two pollable rows):\n%s", got, body)
}
if got := strings.Count(body, ">Remove<"); got != 1 {
t.Errorf("Remove control count = %d, want 1 (only the orphan):\n%s", got, body)
}
if strings.Contains(body, "confirm-row") {
t.Errorf("a confirm row renders in this batch:\n%s", body)
if strings.Contains(body, "Remove") || strings.Contains(body, "confirm-row") {
t.Errorf("a Remove control or confirm row renders in this batch:\n%s", body)
}
// No ember: the new-chapter signal stays off the admin surface. Scoped to
// the page content — the shell's brand mark legitimately wears the ember
@@ -2725,60 +2722,6 @@ func TestAdminSeriesDetailRendersMarks(t *testing.T) {
}
}
// The provenance line beside the chapter names the actor class behind the
// value — "machine read" for a checked Series, "correction" for the owner's
// stamp, "sighting" for a Reader-raised one — and appears nowhere in the
// rendered Series list: an actor class is context for the Series the owner is
// already looking at, never a population to sweep (#152).
func TestAdminSeriesDetailProvenanceLine(t *testing.T) {
router, st := newWebTestServer(t, testConfig())
seed(t, st, store.Bookmark{
Key: "asura:machine", Site: "asura", SeriesID: "machine",
Title: "Machine", SeriesURL: "https://asurascans.com/series/machine",
Kind: "manga", LatestChapter: "45", LatestChapterNum: floatPtr(45),
})
if err := st.MarkLatestChecked("asura", "machine", time.Now().Add(-time.Hour).UnixMilli()); err != nil {
t.Fatalf("MarkLatestChecked: %v", err)
}
seed(t, st, store.Bookmark{
Key: "asura:hand", Site: "asura", SeriesID: "hand",
Title: "Hand", SeriesURL: "https://asurascans.com/series/hand",
Kind: "manga", LatestChapter: "12", LatestChapterNum: floatPtr(12),
})
if err := st.CorrectLatestChapter("asura", "hand", 13, time.Now().UnixMilli()); err != nil {
t.Fatalf("CorrectLatestChapter: %v", err)
}
seed(t, st, store.Bookmark{
Key: "demonic:raised", Site: "demonic", SeriesID: "raised",
Title: "Raised", SeriesURL: "https://demonicscans.org/series/raised",
Kind: "manga",
})
if err := st.RecordSighting(st.OwnerID(), "demonic", "raised", floatPtr(7), time.Now().UnixMilli()); err != nil {
t.Fatalf("RecordSighting: %v", err)
}
for _, tc := range []struct{ key, want string }{
{"asura:machine", "machine read"},
{"asura:hand", "correction"},
{"demonic:raised", "sighting"},
} {
body := seriesDetailPage(t, router, st, tc.key)
if !strings.Contains(body, "<span>"+tc.want+"</span>") {
t.Errorf("detail %s lacks the %q provenance line:\n%s", tc.key, tc.want, body)
}
}
listBody := adminSeriesPage(t, router, st, "")
for _, word := range []string{"machine read", "correction", "sighting"} {
if strings.Contains(listBody, "<span>"+word+"</span>") {
t.Errorf("Series list carries a %q provenance line:\n%s", word, listBody)
}
}
}
// A well-formed key naming no row is a 404, and so is a key with no ":",
// an empty Site or an empty SeriesID — the detail page never answers 500 for
// an address nobody can reach.
@@ -3295,486 +3238,3 @@ func TestSeriesPollCapsBody(t *testing.T) {
t.Errorf("an oversized body still stamped the request:\n%s", body)
}
}
// The correction route validates at the boundary: a non-numeric, zero,
// negative or non-finite chapter answers 400 and never reaches the store, and
// a finite number greater than zero stores the number, the derived label and
// the stamp. The answer is the freshly rendered meta fragment, so the figures
// describe the state after the press (#149).
func TestCorrectLatestChapterRoute(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{
key: "asura:solo", url: "u", checkedAt: 9000, bookmarks: 1, latestNum: floatPtr(3),
})
router := newRouter(st, testConfig())
cookie := sessionCookie(t, st)
for _, body := range []string{
"chapter=abc", "chapter=", "chapter=0", "chapter=-1", "chapter=NaN", "chapter=Inf",
} {
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:solo/latest", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(cookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusBadRequest {
t.Errorf("POST latest with body %q: status = %d, want 400", body, rr.Code)
}
}
// Nothing reached the store: the seeded number stands, unstamped.
var num float64
var stamp int64
if err := db.QueryRow(`
SELECT latest_chapter_num, latest_corrected_at
FROM series WHERE site = 'asura' AND series_id = 'solo'`).
Scan(&num, &stamp); err != nil {
t.Fatalf("read back: %v", err)
}
if num != 3 || stamp != 0 {
t.Fatalf("after 400s the row is num %v, stamp %d; want 3, 0", num, stamp)
}
// A good press stores the number, the derived label and the stamp, and
// answers with the meta fragment describing the state after the press.
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:solo/latest", strings.NewReader("chapter=12.5"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(cookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("POST latest status = %d, want 200 (body %s)", rr.Code, rr.Body.String())
}
body := rr.Body.String()
if !strings.Contains(body, `id="detail-meta"`) {
t.Errorf("correction answer is not the meta fragment:\n%s", body)
}
if !strings.Contains(body, `<span class="mark">corrected `) {
t.Errorf("correction answer lacks the fresh corrected marker:\n%s", body)
}
var label string
if err := db.QueryRow(`
SELECT latest_chapter, latest_chapter_num, latest_corrected_at
FROM series WHERE site = 'asura' AND series_id = 'solo'`).
Scan(&label, &num, &stamp); err != nil {
t.Fatalf("read back: %v", err)
}
if label != "Chapter 12.5" || num != 12.5 {
t.Errorf("stored = %q, %v; want the derived label and 12.5", label, num)
}
if stamp == 0 {
t.Error("stamp = 0, want the correction stamp written")
}
}
// The detail page offers the one-input correction with the plain copy, and
// the corrected marker rides the meta line while the stamp is set — then
// disappears the moment a machine writes the number (#149).
func TestAdminSeriesDetailCorrectionMarker(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:solo", url: "u", checkedAt: 9000, bookmarks: 1})
router := newRouter(st, testConfig())
cookie := sessionCookie(t, st)
body := seriesDetailPage(t, router, st, "asura:solo")
for _, want := range []string{
`name="chapter"`,
`hx-post="/admin/series/asura:solo/latest"`,
"The next successful Poll overwrites this value.",
} {
if !strings.Contains(body, want) {
t.Errorf("detail page lacks %q:\n%s", want, body)
}
}
if strings.Contains(body, "corrected ") {
t.Errorf("uncorrected detail already carries the marker:\n%s", body)
}
// The press lands the marker on the meta line.
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:solo/latest", strings.NewReader("chapter=7"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(cookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("POST latest status = %d, want 200", rr.Code)
}
body = seriesDetailPage(t, router, st, "asura:solo")
if !strings.Contains(body, `<span class="mark">corrected `) {
t.Errorf("detail page lacks the corrected marker after the press:\n%s", body)
}
if !strings.Contains(body, "ch 7") {
t.Errorf("detail page does not show the corrected number:\n%s", body)
}
// A machine write (the poller's setter) kills the marker.
if err := st.SetLatestChapter("asura", "solo", "Chapter 8", 8); err != nil {
t.Fatalf("SetLatestChapter: %v", err)
}
body = seriesDetailPage(t, router, st, "asura:solo")
if strings.Contains(body, "corrected ") {
t.Errorf("marker survives a machine write:\n%s", body)
}
if !strings.Contains(body, "ch 8") {
t.Errorf("detail page does not show the machine-written number:\n%s", body)
}
}
// The series URL repair validates with the poller's own fetch gate and
// answers 400 before anything reaches the store; a URL that passes the gate
// is stored where an Upsert would have ignored it. The request performs no
// outbound fetch — no fetcher is ever constructed on this path (the web
// router has no fetcher seam at all, and the handler only calls the store),
// so "storing is not verifying" is enforced by construction (#151).
func TestSeriesURLRepairRoute(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{
key: "asura:solo", url: "https://asurascans.com/comics/solo", checkedAt: 9000, bookmarks: 1,
})
router := newRouter(st, testConfig())
cookie := sessionCookie(t, st)
// A URL the gate refuses — foreign host, http scheme, host of another
// Site — answers 400 and never reaches the store.
for _, body := range []string{
"series_url=https://evil.example/solo",
"series_url=http://asurascans.com/stories/solo",
"series_url=https://kagane.to/series/solo",
"series_url=",
} {
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:solo/series-url", strings.NewReader(body))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(cookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusBadRequest {
t.Errorf("POST series-url with body %q: status = %d, want 400", body, rr.Code)
}
}
capReq := httptest.NewRequest(http.MethodPost, "/admin/series/asura:solo/series-url",
strings.NewReader("series_url=https://asurascans.com/stories/"+strings.Repeat("a", 1<<17)))
capReq.Header.Set("Content-Type", "application/x-www-form-urlencoded")
capReq.AddCookie(cookie)
capRR := httptest.NewRecorder()
router.ServeHTTP(capRR, capReq)
if capRR.Code != http.StatusBadRequest {
t.Errorf("POST series-url with an oversized body: status = %d, want 400", capRR.Code)
}
var stored string
if err := db.QueryRow(`SELECT series_url FROM series WHERE site = 'asura' AND series_id = 'solo'`).
Scan(&stored); err != nil {
t.Fatalf("read back: %v", err)
}
if stored != "https://asurascans.com/comics/solo" {
t.Fatalf("after 400s the stored URL = %q, want the seeded one untouched", stored)
}
// A URL that passes the gate lands, and the press answers with the meta
// fragment just like the other detail-page actions.
repair := "https://asurascans.com/stories/solo-renumbered"
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:solo/series-url",
strings.NewReader("series_url="+repair))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(cookie)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("POST series-url status = %d, want 200 (body %s)", rr.Code, rr.Body.String())
}
if body := rr.Body.String(); !strings.Contains(body, `id="detail-meta"`) {
t.Errorf("repair answer is not the meta fragment:\n%s", body)
}
if err := db.QueryRow(`SELECT series_url FROM series WHERE site = 'asura' AND series_id = 'solo'`).
Scan(&stored); err != nil {
t.Fatalf("read back: %v", err)
}
if stored != repair {
t.Fatalf("stored URL = %q, want %q", stored, repair)
}
}
// The detail page offers the repair input prefilled with the stored address,
// states the honest limit — a Site-wide host change is a SQL migration, not a
// per-Series form — and a stored string renders back into the input escaped
// (issue #151).
func TestAdminSeriesDetailRepairForm(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{
key: "asura:solo", url: "https://asurascans.com/stories/solo", bookmarks: 1,
})
seedSeriesRow(t, st, db, seriesRowSeed{
key: "asura:evil", url: `https://asurascans.com/x"><script>alert(1)</script>`, bookmarks: 1,
})
router := newRouter(st, testConfig())
body := seriesDetailPage(t, router, st, "asura:solo")
for _, want := range []string{
`name="series_url"`,
`hx-post="/admin/series/asura:solo/series-url"`,
`value="https://asurascans.com/stories/solo"`,
"A Site-wide host change", "SQL migration",
} {
if !strings.Contains(body, want) {
t.Errorf("detail page lacks %q:\n%s", want, body)
}
}
// The stored value that is markup stays markup in the input's value
// attribute, never executable HTML.
body = seriesDetailPage(t, router, st, "asura:evil")
if strings.Contains(body, `<script>alert(1)</script>`) {
t.Errorf("repair input renders stored URL unescaped:\n%s", body)
}
if !strings.Contains(body, `value="https://asurascans.com/x&#34;&gt;&lt;script&gt;alert(1)&lt;/script&gt;"`) {
t.Errorf("repair input does not carry the escaped stored URL:\n%s", body)
}
}
// The Remove control is offered only to the owner, and only on a Series no
// Reader holds: on the orphan's list row and on the orphan's detail page,
// nowhere else (#155).
func TestSeriesRemoveRendersOnlyOnOrphans(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:ok", url: "u", checkedAt: 9000, bookmarks: 1})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:orphan", url: "u", checkedAt: 9000, bookmarks: 0})
router := newRouter(st, testConfig())
body := adminSeriesPage(t, router, st, "")
if got := strings.Count(body, ">Remove<"); got != 1 {
t.Errorf("list offers Remove %d times, want 1 (only the orphan):\n%s", got, body)
}
for _, tc := range []struct {
key string
want bool
}{
{"asura:ok", false},
{"asura:orphan", true},
} {
body := seriesDetailPage(t, router, st, tc.key)
if got := strings.Contains(body, ">Remove<"); got != tc.want {
t.Errorf("%s detail offers Remove = %v, want %v", tc.key, got, tc.want)
}
}
}
// A removal from the list answers with the removed row's fragment and the
// heading re-rendered out of band with the fresh count: the row and the
// count are one fact. The row's press carries the list's filter state, so
// the count describes the list the owner is looking at, and the HX-Reswap
// header deletes the row through the same button that swaps the refusal in.
func TestRemoveFromListAnswersRowAndFreshHeading(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:a", url: "u", checkedAt: 9000, bookmarks: 0})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:b", url: "u", checkedAt: 9000, bookmarks: 0})
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:held", url: "u", checkedAt: 9000, bookmarks: 1})
router := newRouter(st, testConfig())
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:a/remove",
strings.NewReader("filter=no_readers&band=0"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("removal status = %d, want 200", rr.Code)
}
if got := rr.Header().Get("HX-Reswap"); got != "delete" {
t.Errorf("response does not ask htmx to delete the row (HX-Reswap = %q)", got)
}
body := rr.Body.String()
if !strings.Contains(body, "Title of asura:a") {
t.Errorf("answer does not carry the removed row's fragment:\n%s", body)
}
if !strings.Contains(body, `hx-swap-oob="true"`) ||
!strings.Contains(body, "1 series") || !strings.Contains(body, "No Readers") {
t.Errorf("answer does not re-render the heading out of band with the fresh count:\n%s", body)
}
// Gone from the store, gone from the list, and the heading lies no longer.
var one int
if err := db.QueryRow(`SELECT 1 FROM series WHERE site = 'asura' AND series_id = 'a'`).Scan(&one); err != sql.ErrNoRows {
t.Fatalf("series row after removal = %v, want sql.ErrNoRows", err)
}
body = adminSeriesPage(t, router, st, "?filter=no_readers")
if strings.Contains(body, "Title of asura:a") || !strings.Contains(body, "1 series") {
t.Errorf("list after removal is not the fresh view:\n%s", body)
}
}
// A removal from the detail page navigates to the No-Readers list: htmx gets
// a full navigation (HX-Redirect — a 303 would be followed by the request
// and the list page swapped into the press's target), plain clients the 303
// the ticket names, to the wire filter the orphan list actually is.
func TestRemoveFromDetailRedirectsToNoReadersList(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:gone", url: "u", checkedAt: 9000, bookmarks: 0})
// A second orphan for the htmx dialect's request, whose row must still
// exist after the first request removed its own.
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:gone2", url: "u", checkedAt: 9000, bookmarks: 0})
router := newRouter(st, testConfig())
target := "/admin/series?filter=" + store.SeriesFilterNoReaders
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:gone/remove", nil)
req.Header.Set("HX-Target", "detail-meta")
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusSeeOther {
t.Fatalf("detail removal status = %d, want 303", rr.Code)
}
if got := rr.Header().Get("Location"); got != target {
t.Errorf("Location = %q, want %q", got, target)
}
var one int
if err := db.QueryRow(`SELECT 1 FROM series WHERE site = 'asura' AND series_id = 'gone'`).Scan(&one); err != sql.ErrNoRows {
t.Fatalf("series row after detail removal = %v, want sql.ErrNoRows", err)
}
req = httptest.NewRequest(http.MethodPost, "/admin/series/asura:gone2/remove", nil)
req.Header.Set("HX-Request", "true")
req.Header.Set("HX-Target", "detail-meta")
req.AddCookie(sessionCookie(t, st))
rr = httptest.NewRecorder()
router.ServeHTTP(rr, req)
if got := rr.Header().Get("HX-Redirect"); got != target {
t.Errorf("HX-Redirect = %q, want %q", got, target)
}
}
// A removal that races a fresh Bookmark is a refusal, not an error: the row
// is rendered again at its new count with the fact spelled out, never a 500,
// and it must not vanish from the list — the delete never happened. The
// detail-surface refusal navigates back to the detail page, where the same
// fresh count is visible.
func TestRemoveRacedBookmarkIsRefusedNotError(t *testing.T) {
st, dsn := newTestStoreURL(t)
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatalf("open %s: %v", dsn, err)
}
defer db.Close()
seedSeriesRow(t, st, db, seriesRowSeed{key: "asura:raced", url: "u", checkedAt: 9000, bookmarks: 1})
router := newRouter(st, testConfig())
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:raced/remove",
strings.NewReader("filter=no_readers&band=0"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("refusal status = %d, want 200 (never a 500)", rr.Code)
}
if got := rr.Header().Get("HX-Reswap"); got != "" {
t.Errorf("refusal carries HX-Reswap = %q, want none (the row must stay)", got)
}
body := rr.Body.String()
if !strings.Contains(body, "a Reader has bookmarked this Series again") {
t.Errorf("refusal does not say what happened:\n%s", body)
}
if !strings.Contains(body, `class="c-rd">1</span>`) {
t.Errorf("refusal does not render the fresh count:\n%s", body)
}
var one int
if err := db.QueryRow(`SELECT 1 FROM series WHERE site = 'asura' AND series_id = 'raced'`).Scan(&one); err != nil {
t.Fatalf("series row after refusal = %v, want present", err)
}
list := adminSeriesPage(t, router, st, "")
if !strings.Contains(list, "Title of asura:raced") {
t.Fatal("row vanished from the list after a refused removal")
}
if strings.Contains(list, ">Remove<") {
t.Errorf("a held series still offers Remove:\n%s", list)
}
// The detail-surface refusal navigates back to the detail page.
req = httptest.NewRequest(http.MethodPost, "/admin/series/asura:raced/remove", nil)
req.Header.Set("HX-Target", "detail-meta")
req.AddCookie(sessionCookie(t, st))
rr = httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusSeeOther {
t.Fatalf("detail refusal status = %d, want 303", rr.Code)
}
if got := rr.Header().Get("Location"); got != "/admin/series/asura:raced" {
t.Errorf("refusal Location = %q, want the detail page", got)
}
}
// The remove route trusts the same way the poll route does: a malformed key
// is a 400 and an unknown Series a 404, and an oversized body is a 400 that
// removes nothing.
func TestRemoveRejectsBadKeysAndCapsBody(t *testing.T) {
router, st := newWebTestServer(t, testConfig())
seed(t, st, store.Bookmark{
Key: "asura:x", Site: "asura", SeriesID: "x",
Title: "Title of asura:x", SeriesURL: "u",
})
for _, tc := range []struct {
path string
want int
}{
{"/admin/series/nocolon/remove", http.StatusBadRequest},
{"/admin/series/:x/remove", http.StatusBadRequest},
{"/admin/series/asura:/remove", http.StatusBadRequest},
{"/admin/series/asura:ghost/remove", http.StatusNotFound},
} {
req := httptest.NewRequest(http.MethodPost, tc.path, nil)
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != tc.want {
t.Errorf("POST %s status = %d, want %d", tc.path, rr.Code, tc.want)
}
}
big := strings.Repeat("a", 1<<17) // 128 KiB, over the 64 KiB cap
req := httptest.NewRequest(http.MethodPost, "/admin/series/asura:x/remove", strings.NewReader(big))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(sessionCookie(t, st))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if rr.Code != http.StatusBadRequest {
t.Fatalf("oversized body status = %d, want 400", rr.Code)
}
list := adminSeriesPage(t, router, st, "")
if !strings.Contains(list, "Title of asura:x") {
t.Errorf("an oversized body still removed the row:\n%s", list)
}
}
@@ -1,86 +0,0 @@
# ADR-0014: Cover addresses derived from the bytes, not the source URL
Date: 2026-08-22
Status: accepted
## Decision
A Cover's content address is the hex SHA-256 of its **bytes**, not of the
source URL it was fetched from. `CoverAddressForBytes(body)` names the address
`putCover` stores under, `SetSeriesCover` and `ReplaceSeriesCover` point the
Series row at it, and the wire URL is built from it exactly as before — same
route, same 64-hex-digit shape, same immutability, only the input to the hash
changes. Rows written before this ADR keep their URL-derived addresses
forever: they are never rehashed on read, and they heal into byte addressing
only when a Forced Poll replaces them.
`ReplaceSeriesCover(site, seriesID, sourceURL, body, contentType)`
`(previous, current, error)` is the one write that may move a Cover once one
exists. It stores the bytes, then in one transaction locks the Series row,
reads the old `cover_address`, writes the new one and the source URL, and
reports both addresses: `previous == ""` means there was no Cover,
`previous == current` means the Site served identical artwork, and any other
pair names the stranded address.
## Why a future reader will find this surprising
The address is what makes a re-art visible at all. URL addressing collapses
every image behind a stable URL into one address, so a Series whose Cover
changes (a big-budget CPI blitz on a light novel is the standing example)
keeps serving its original cover bytes: the poll refetches the same URL,
hashes it, and the store records the same address, everyone happy except the
Reader. Nothing in the system can detect the change, because the address is a
pure function of the fetch target, and identical bytes written 1,000 times
are one blob on disk. Storing bytes we already know how to store is only a
few lines of work. **Rejecting that work is the surprising part, and the
answer is the Forced Poll wave**: for a corrupt/blank cover the poll's
fill-if-blank installer already worked, but for a *wrong but non-blank* cover
there was no write that would move it at all — only a manual truth in
`series.cover_address`, which is exactly the thing that must never be set by
hand. Byte addressing gives the replacement write a **new address to write**,
and with it a legitimate, transaction-safe mover.
## Considered options
**Keep URL addressing and add a generic "clear the cover" write.**
Rejected: clearing is a two-phase action (blank it, wait for the poll to
re-fill, hope the bytes changed in between) that cannot report what the
write did, and it makes the Series render cover-less in between. The
replacement write is atomic, reports its displacement, and has one effect:
the Series now points at bytes that actually came from its source URL.
**Address by URL, but salt it so a re-art is a new address.**
Rejected: the salt would have to live somewhere addressable (a stored per-
Series nonce), turning the address from a content fact into a mutable fact —
two rows could then hold identical bytes under different addresses and the
invariant "same bytes object" is gone.
## Consequences
- `store.CoverAddress` (URL-hash) is deleted; `CoverAddressForBytes` is
public so tests and the forced-poll wave can predict addresses from the
bytes fakes serve.
- Legacy URL-addressed rows are read-only facts: `GetCover(sourceURL)` keeps
resolving them (the poll heal path), and they are re-addressed only by a
forced replacement. Until one happens, they are invisible to byte-derived
lookups — the reverse direction was always true, so this side has no
migration and no lookup fan-out.
- A replaced Cover's old bytes stay on disk under their address (the `covers`
row is untouched — only the Series row moves). Nothing reclaims them
today; a later sweep is a small query over `covers` addresses not
referenced by any `series` row.
- `SetSeriesCover` keeps its `cover_address = ''` guard untouched: the
acquisition-at-creation and poll fill paths still may not overwrite a
non-blank Cover. The two installers are now deliberately different
functions instead of one function with a conditional.
- The address is still a filesystem path (≤64 hex chars, no separators), so
`coverAddressRe` and the sharding stay exactly as they are.
## Cost of reversing
The URL-hash side of the current rows is uncomputable from the rows alone: a
rollback would need every stored blob's source URL, a join to a table that
does not store it, or a refetch of every Series. Keeping both derivations
resolvable is cheaper than either, so the two derivations are documented in
the 0009 migration comment: no component may assume which derivation a
stored address came from, because the 64-hex shape hides it.