Compare commits

...

8 Commits

Author SHA1 Message Date
sulthan cddd16bcdc Merge pull request 'fix: series removal sent its status line twice' (#176) from fix/series-remove-double-writeheader into main 2026-08-23 14:02:45 +07:00
sulthan c1616b3162 fix: series removal sent its status line twice
adminSeriesRemove answered the list surface with two h.render calls -- the
row fragment and the out-of-band heading -- and h.render writes a status
line each time, so every removal logged "superfluous
response.WriteHeader call". The heading is an append to a response
already committed, so it now executes straight onto w, the way
writeChromeOOB already does it.

The regression test runs the router under a real server with a captured
ErrorLog: a ResponseRecorder never sees this warning, which is why the
existing removal test did not catch it.
2026-08-23 14:02:27 +07:00
sulthan e1ba7fdabb fix: pgtest left an anonymous volume behind on every run (#175)
## Problem

`go test ./...` leaked one Docker volume per test package. `pgtest.Main` tore its throwaway container down with `docker rm -f` — no `-v`. The `postgres:17-alpine` image declares a `VOLUME` for `/var/lib/postgresql/data`, and an explicit `rm` without `-v` orphans that anonymous volume even though the container was created with `--rm`.

Found in passing: a 13-hour-old orphaned pgtest container (AutoRemove, `fsync=off`, one mount) still running from a killed test binary — the same leak's other half. Removed on the dev machine.

## Change

- `backend/internal/pgtest/pgtest.go`: `-v` on all three `docker rm -f` teardown sites (`Main`'s defer, and both error paths in `start`).
- `AGENTS.md`: cleanup expectation under Commands — `docker compose down -v` for a stack, `docker rm -f -v` for a hand-run container, then check `docker volume ls` and `docker system df`. Explicitly out of bounds: removing the user's `postgres-data` volume, or a blanket `docker system prune` of their images and build cache.

## Verification

`docker volume ls` snapshot diffed across a real `./internal/store` run: no new volumes, `docker system df` reports 0 local volumes.

Reviewed-on: #175
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-23 13:14:49 +07:00
sulthan 2ac1f0c507 fix: BROWSER_WS_URL reaches the container again (#171 dropped the line) 2026-08-23 12:55:26 +07:00
sulthan e93e79c1bb Spec #137: per-Series poll failure state, completion hint, and outbound owner notification (#174)
Implements spec #137 (spec 4 of 4 from wayfinder map #114).

Closes #137.

Tickets: #164, #165, #166, #167, #168, #169, #170, #171, #172, #173 — all closed, landed on this branch.

## Summary
- #164/#168: sixth outcome word `not_found`; per-Site completed marker predicate.
- #165/#169: `poll_failures` row is the failure state; the pass remembers a Site-reported completion.
- #166/#171: two new admin filters (`failing`, `unverified`); outbound owner notification + stall condition.
- #167/#170: a failure names itself on the Series page; the completion hint reaches the owner and decides nothing.
- #172: the other three fault conditions (no-browser-route, sidecar-down, adapter-broken) feeding the notifier.
- #173: the landing verdict line shares the same `latest.FaultsFrom` judgement the notifier uses, so the page and the push cannot disagree.

`cd backend && go test ./...` green on the merged branch (8 packages).

Reviewed-on: #174
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-23 11:31:10 +07:00
sulthan 8e4fa6448e Spec #136: Finished belongs to the Series — owner-owned poll gate, Lifecycle bucket dropped (#163)
Closes #136.

Spec #136 end to end: `finished` becomes a fact about the Series, written only by the owner, and the reader-facing Lifecycle bucket is gone.

## What landed

- **#157** — `series.finished_at bigint NOT NULL DEFAULT 0` plus the migration whose statement order is load-bearing (seed from the buckets, then flip them); both Lane queries lose the `HAVING COUNT(*) FILTER (WHERE b.status <> 'finished')` clause and gate on `finished_at = 0` instead, with the due-query/eligible-count force asymmetry kept deliberate and commented; `StatusFinished`, its API special-case 400, the web tab and the templates' Finished bucket deleted.
- **#158** — owner Finish control on the Series detail page: confirm-gated finish, instant un-finish, admin accent (never ember, nothing is destroyed), `Store.SetSeriesFinished`, the two routes behind the owner gate, and the state displayed on the list row without offering the control there.
- **#160** — reader side: derived `finished` bool on the flat Bookmark (`s.finished_at > 0`), rendered as a text-only label in both userscripts and on the web card; read-only inbound by omission from `Upsert`'s explicit `series` column list, same mechanism that already protects `cover`.
- **#161** — glossary and the stale Reader-count divergence note catch up.
- **#159** — `finished` joins the admin filter vocabulary (predicate `finished_at > 0`, label `Finished`, own aggregate count, figure last in the stats block as informational); the four clock-driven hygiene predicates (stale, never-checked, no-cover, no-chapter) exclude finished Series while unpollable, orphan and sighting-raised deliberately do not.

## Verification

`go vet ./...` and `go test ./...` green on the merged branch (Docker-backed, throwaway `postgres:17-alpine` per package). Each ticket also passed a two-axis review (spec + standards) on its own branch before merge.

Reviewed-on: #163
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-22 17:31:52 +07:00
sulthan faa80c41ea Skills: split plan-tickets out of implement-tickets (#162)
Adds `.claude/skills/plan-tickets/SKILL.md` and trims `implement-tickets` to dispatch-only, with the matching `.omp/agents/ticket-implementer.md` update.

Docs/skills only — no backend, userscript, or web changes.

Reviewed-on: #162
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-22 16:43:42 +07:00
sulthan 4aaf1d4f91 Spec #135: owner data-correction actions — Latest Chapter, series_url, Cover, orphan removal (#156)
Implements spec #135 (spec 2 of 4, derived from wayfinder map #114; decisions settled in #120/#121/#125/#131). Blocked-by #134 is merged, so this lands on `main`.

Four owner actions the dashboard can now perform, one ticket each:

- **#149** — Latest Chapter correction: one numeric input, overwritten by the next machine write.
- **#151** — Series URL repair: owner-typed, gated by the poller's own fetch gate.
- **#150 / #153 / #154** — Cover replacement: addresses derived from bytes (`#150`), a Forced Poll replaces the Cover while an ordinary pass still only fills a blank one (`#153`), and byte reclamation is one guarded helper, file first / covers row last (`#154`).
- **#155** — Orphan removal: one Series at a time, with the foreign key as the guard.

Plus **#152** — Latest Chapter provenance: one derived line naming the actor class, so an owner can tell a hand-edited number from a machine read.

- Migration `0015_latest_correction.sql` adds the correction/provenance columns; `0009` now derives cover addresses from bytes.
- ADR `0014-cover-addresses-from-bytes.md` records the address scheme.

Backend tests cover the store, poller, admin handlers, and web routes (`go test ./...`, needs Docker).

Reviewed-on: #156
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
2026-08-22 12:10:40 +07:00
67 changed files with 7493 additions and 609 deletions
+31 -87
View File
@@ -1,108 +1,49 @@
---
name: implement-tickets
description: "Orchestrate a batch of tickets: plan the briefs, then hand each ticket to its own implementer subagent in its own worktree."
description: "Run a planned wave of tickets: one implementer subagent per ticket in its own worktree, then land, merge and close what comes back."
disable-model-invocation: true
---
# Implement tickets
You are the **orchestrator**. You write briefs, dispatch, land results, and talk
to the tracker. You do not write the implementation — every line of ticket code
is written by a `ticket-implementer` subagent in its own git worktree. Reach for
the editor yourself only for a merge conflict resolution.
You are the **orchestrator**. You dispatch, land results, and talk to the
tracker. You do not write the implementation — every line of ticket code is
written by a `ticket-implementer` subagent in its own git worktree. Reach for the
editor yourself only for a merge conflict resolution.
Ticket source and tracker conventions: `docs/agents/issue-tracker.md`. `tea` usage: skill `gitea`.
## 1. Collect the tickets
## 0. Load the plan
The user's argument is the selector: issue numbers, a label, a parent issue, or
nothing. With nothing, take the open issues labelled `ready-for-agent`.
Your argument is the batch slug. Read `.scratch/<batch-slug>/plan.md` — it gives
the base branch, the waves, the contracts, and each ticket's brief path and
status. Run the earliest wave that is not landed.
Fetch each with `tea issue <n> --comments`, and read the **whole** body —
acceptance criteria and the `Blocked by` line are what the rest of this skill
runs on. A ticket whose blockers are still open is out of this batch unless a
blocker is also in it.
The briefs are the requirements and they are already approved: read each one you
are about to dispatch, but do not rewrite it, and do not fetch the tickets from
the tracker to second-guess it. A brief that is wrong or thin is a `plan-tickets`
problem — say so and stop, rather than patching it here.
## 2. Plan the batch
No plan file, or no briefs for the next wave: run `plan-tickets` first. That
skill owns wave membership, contracts, and every brief.
Explore enough of the codebase to write briefs a fresh context can act on: the
files each ticket lands in, the patterns it must follow, the `AGENTS.md`
invariants it touches.
Then decide three things:
- **Waves.** Blocking edges set the order; tickets with no open blocker inside
the batch share a wave. Cap each wave at **3** concurrent tickets unless the
user set another width.
- **Contracts.** Two tickets in one wave that meet at a function signature, a
JSON shape, a table column, or a token name: you decide the shape now and
write the identical wording into both briefs. A contract left for the
subagents to negotiate is a merge conflict you scheduled.
- **Splits.** A ticket too big for one fresh context window goes into the wave
as two briefs, or back to the user.
## 3. Get the plan approved
Present, and stop:
- the wave list, and for each ticket: number, title, one-line brief summary,
the files or areas it will touch, its verification commands
- every cross-ticket contract, verbatim as it will appear in the briefs
- anything you had to assume
Wait for approval. Apply the user's edits to the plan, do not relitigate them.
## 4. Run a wave
## 1. Dispatch the wave
Per ticket, before dispatch:
```bash
git worktree add ../ticket-<n> -b ticket/<n>-<slug> <base> # base = the branch you are on
git worktree add ../ticket-<n> -b ticket/<n>-<slug> <base> # base = the plan's base branch, checked out here
cp .env ../ticket-<n>/ 2>/dev/null # gitignored, worktrees do not get it
tea issue edit <n> --add-assignees <your gitea username> # tea login list has it
```
Write the brief to `.scratch/<batch-slug>/t<n>-brief.md` using the template
below, in the ubiquitous language of `CONTEXT.md` — a brief that says "scrape"
where the domain says Poll hands the subagent the wrong model of the system.
Then dispatch the whole wave in **one** `task` batch, every item on the
`ticket-implementer` agent. Each dispatch names: the absolute brief path, the
worktree path, the branch, the base ref, and the report path
`.scratch/<batch-slug>/t<n>-report.md`.
`.scratch/<batch-slug>/t<n>-report.md`. Mark each ticket `dispatched` in the plan
file.
<brief-template>
# Ticket #<n> — <title>
**Read first.** `tea issue <n> --comments` for this ticket, then the issue it
refers to — the parent or spec — the same way. The comments carry decisions the
body never got updated with. This brief stays the requirements; those two reads
are the intent behind them.
**Goal.** The end-to-end behaviour this ticket makes work, from the user's side.
**Acceptance criteria.** Verbatim from the ticket.
**Contract.** The exact shared signatures / shapes / names this ticket must
implement or consume, and which sibling ticket is on the other end. Omit when
the ticket touches nothing shared.
**Where it lands.** The files and packages, and the existing pattern to follow
in each.
**Binding invariants.** The `AGENTS.md` rules this change can break — name them.
**TDD seams.** Where a test comes first — run the `tdd` skill at each one and
follow its red → green loop. Or "none — verify after".
**Verify.** The exact commands, e.g. `cd backend && go test ./...`,
`node --test userscript/test/logic.test.js`.
**Out of scope.** What not to touch, especially a sibling ticket's files.
</brief-template>
## 5. Land the wave
## 2. Land the wave
The wave is landed when every ticket in it is closed, reverted, or handed back
to the user. Per returned ticket:
@@ -120,14 +61,17 @@ clash — both sides green apart, wrong together — goes back to whichever tick
owns the contract, as a re-dispatch with the collision described.
Then `tea comment <n> "<the report summary>"`, `tea issue close <n>`, and
`git worktree remove ../ticket-<n>`. Keep the report file.
`git worktree remove ../ticket-<n>`. Keep the report file, and mark the ticket
`landed` or `handed back` in the plan file.
Only once the whole wave is landed does the next wave start — its briefs may
need what this one changed.
## 3. Hand back or close the batch
## 6. Close the batch
Waves left in the plan: stop and say which wave is next. Its briefs are written
by `plan-tickets` against the base you just changed — that is why they were not
written up front, and why you do not write them.
Run the full suite once on the merged base, and report: a line per ticket with
its status, commits, and open concerns, plus anything still assigned or open on
the tracker. A red suite after every ticket went green is an interaction bug —
diagnose it, name the two tickets, and fix it or hand it back with both named.
Last wave landed: run the full suite once on the merged base, and report a line
per ticket with its status, commits, and open concerns, plus anything still
assigned or open on the tracker. A red suite after every ticket went green is an
interaction bug — diagnose it, name the two tickets, and fix it or hand it back
with both named.
+125
View File
@@ -0,0 +1,125 @@
---
name: plan-tickets
description: "Plan a batch of tickets and write the briefs for its next wave: read the tracker, decide waves and contracts, get the plan approved."
disable-model-invocation: true
---
# Plan tickets
You produce the two artifacts the `implement-tickets` skill runs on: a **plan
file** and one **brief** per ticket in the next wave. You write no ticket code
and create no worktrees — that is the runner's half.
Ticket source and tracker conventions: `docs/agents/issue-tracker.md`. `tea` usage: skill `gitea`.
Called twice in a batch's life, at least: once to open it, then again after each
wave lands, because a later wave's briefs may need what the last one changed. On
a re-entry, read the existing `.scratch/<batch-slug>/plan.md` first and plan only
the next unlanded wave — the waves and contracts already approved there stand
unless the landed wave proved one wrong.
## 1. Collect the tickets
The user's argument is the selector: issue numbers, a label, a parent issue, or
nothing. With nothing, take the open issues labelled `ready-for-agent`.
Fetch each with `tea issue <n> --comments`, and read the **whole** body —
acceptance criteria and the `Blocked by` line are what the rest of this skill
runs on. A ticket whose blockers are still open is out of this batch unless a
blocker is also in it. Read the issue each ticket refers to, its parent or spec,
the same way: nothing on the implementation path reads the tracker after you —
only `cr-spec` does, at review time — so a decision that lives in a comment
reaches the implementer only if you carry it into the brief.
## 2. Plan the batch
Explore enough of the codebase to write briefs a fresh context can act on: the
files each ticket lands in, the patterns it must follow, the `AGENTS.md`
invariants it touches.
Then decide three things:
- **Waves.** Blocking edges set the order; tickets with no open blocker inside
the batch share a wave. Cap each wave at **3** concurrent tickets unless the
user set another width.
- **Contracts.** Two tickets in one wave that meet at a function signature, a
JSON shape, a table column, or a token name: you decide the shape now and
write the identical wording into both briefs. A contract left for the
subagents to negotiate is a merge conflict you scheduled.
- **Splits.** A ticket too big for one fresh context window goes into the wave
as two briefs, or back to the user.
## 3. Write the artifacts
Pick a `<batch-slug>` — short, from what the batch is about — and write
`.scratch/<batch-slug>/plan.md`. It is the handoff: the runner is a fresh context
that reads this and nothing of your reasoning.
<plan-template>
# Batch <batch-slug>
**Base branch.** The branch every worktree forks from and merges back into.
**Waves.** A table: wave number, ticket number, title, brief path, and status —
`planned` | `dispatched` | `landed` | `handed back`. Every ticket in the batch,
including waves not briefed yet.
**Contracts.** Each cross-ticket contract verbatim, naming both ticket numbers.
**Assumptions.** What you had to assume, and what the user corrected at approval.
</plan-template>
Then write a brief per ticket in the next wave to
`.scratch/<batch-slug>/t<n>-brief.md`, using the template below, in the
ubiquitous language of `CONTEXT.md` — a brief that says "scrape" where the domain
says Poll hands the subagent the wrong model of the system.
<brief-template>
# Ticket #<n> — <title>
**Decisions.** Whatever the ticket's comments or its parent spec settled that
the body never got updated with, restated verbatim. The implementer reads this
brief and the code, never the tracker — a decision missing here is lost to it,
and a stale comment thread is not a source you want a fresh context guessing
from. Omit only when the ticket has no comments.
**Goal.** The end-to-end behaviour this ticket makes work, from the user's side.
**Acceptance criteria.** Verbatim from the ticket.
**Contract.** The exact shared signatures / shapes / names this ticket must
implement or consume, and which sibling ticket is on the other end. Omit when
the ticket touches nothing shared.
**Where it lands.** The files and packages, and the existing pattern to follow
in each.
**Binding invariants.** The `AGENTS.md` rules this change can break — name them.
**TDD seams.** Where a test comes first — run the `tdd` skill at each one and
follow its red → green loop. Or "none — verify after".
**Verify.** The exact commands, e.g. `cd backend && go test ./...`,
`node --test userscript/test/logic.test.js`.
**Out of scope.** What not to touch, especially a sibling ticket's files.
</brief-template>
## 4. Get the plan approved
Present, and stop:
- the wave list, and for each ticket in the next wave: number, title, one-line
brief summary, the files or areas it will touch, its verification commands
- every cross-ticket contract, verbatim as it appears in the briefs
- anything you had to assume
Wait for approval. Apply the user's edits to the plan, do not relitigate them —
into the files, not just the reply, or the runner never sees them.
Then hand off: name the batch slug and the wave, and stop. Running the wave is
`implement-tickets`.
+7 -1
View File
@@ -100,7 +100,13 @@ DISCORD_REDIRECT_URI=
# every kagane poll. Left unset here on purpose — a wrong default would poll a
# stranger's address, and "no browser" is a safe, self-announcing state.
# BROWSER_WS_URL=ws://100.x.y.z:9222
#
# Discord webhook for owner notices (outbound alerting when a poll Lane
# stalls). Unset means the whole path is off — a local stack needs no webhook,
# exactly as the browser URL behaves. The address is a secret in the class of
# TOKEN_KEY: never commit it, never paste it anywhere public.
# DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
#
# Zone the backend stamps its log lines in. Cosmetic only. Nothing else in
# the service has a zone: bookmark timestamps are unix ms, and the two real
# time columns are timestamptz. Defaults to Asia/Jakarta; set to UTC for the
+16 -15
View File
@@ -24,34 +24,35 @@ Your branch is already checked out there. Never `git checkout`, `git switch`,
## Order of work
1. Read the brief file. It is the single source of requirements — use its exact
values verbatim.
2. Read the ticket and the issue it refers to, as the brief's **Read first**
section names them: `tea issue <n> --comments` for each. The ticket's
comments and its parent carry the intent and the decisions behind the brief.
Read no other ticket and no other brief.
3. Read `AGENTS.md` in the worktree, plus the nested `AGENTS.md` for the area
1. Read the brief file. It is the only statement of requirements you get — use
its exact values verbatim, and read nothing from the tracker. The
orchestrator has already read the ticket, its comments and its parent spec,
and folded every live decision into the brief; the threads themselves also
hold reversed and rejected ones you cannot tell apart from here.
2. Read `AGENTS.md` in the worktree, plus the nested `AGENTS.md` for the area
you touch. Its invariants bind you: security rules, design system, comment
policy.
4. Ask before writing code if requirements, acceptance criteria, approach, or
3. Ask before writing code if requirements, acceptance criteria, approach, or
dependencies are unclear. Asking is free; guessing is not.
5. Implement exactly what the brief specifies. At each TDD seam the brief names,
4. Implement exactly what the brief specifies. At each TDD seam the brief names,
run the `tdd` skill and follow its red → green loop.
Follow the patterns already in the codebase; improve what you touch,
restructure nothing outside the ticket.
6. Verify. Focused tests while iterating, the brief's full verification commands
5. Verify. Focused tests while iterating, the brief's full verification commands
once at the end. Test output must be pristine.
7. Commit to your branch. Reference the ticket number in the subject.
8. Review (below), fix, re-verify, commit the fixes.
9. Write the report file, then return the status contract.
6. Commit to your branch. Reference the ticket number in the subject.
7. Review (below), fix, re-verify, commit the fixes.
8. Write the report file, then return the status contract.
## Review
After your first green commit, run the **`code-review`** skill over
`<base ref>...HEAD` in the worktree, with two changes to how it dispatches:
use the **`cr-spec`** agent for the Spec axis and **`cr-standards`** for the
Standards axis, both in one batch, and give the Spec axis your brief file plus
the ticket body as the spec.
Standards axis, both in one batch, and hand the Spec axis your brief file as the
spec plus the ticket number from the brief's title, telling it to read the
ticket itself (`tea issue <n> --comments`). The tracker check belongs in that
read-only context, not in yours.
Fix every Critical and Important finding, then re-run the tests that cover the
amended code. Two fix rounds maximum: anything still open after that goes in the
+10 -1
View File
@@ -50,6 +50,15 @@ Backend (`cd backend`):
Local stack: `docker compose up` (bookmark-api + postgres only; `postgres-data` named volume, `restart: unless-stopped`). No browser — without `BROWSER_WS_URL` the poller logs and skips kagane and comix. To run one: `cd chrome && BROWSER_BIND_ADDR=172.17.0.1 docker compose up -d --build`, then `BROWSER_WS_URL=ws://172.17.0.1:9222` in the root `.env` (bridge gateway, so the API container can name it by IP).
**Clean up Docker after testing.** Storage on the dev machine is scarce, so
anything you started for a test you also tear down before calling the work
done: `docker compose down -v` for a stack you brought up, `docker rm -f -v`
for a container you ran by hand (`-v`, or the image's anonymous data volume
survives). Then check `docker volume ls` and `docker system df` for leftovers
and reclaim them — a dangling volume nobody notices is the leak that fills the
disk. Never remove the `postgres-data` volume of a stack the user is actually
running, and never blanket-`docker system prune` their images or build cache.
Live CDP proof (needs that browser and network, skipped otherwise):
`SMOKE_BROWSER_WS_URL=ws://<ip>:<port> go test -run 'TestSmokeKagane|TestSmokeComix' ./internal/latest`
— fetches a real kagane and comix cover and chapter list. A red run means the challenge is
@@ -99,7 +108,7 @@ Go backend:
- SQL always parameterized (`$N`). Only compile-time constants (`bookmarkColumns`) may be concatenated into query text — never a request value, not even a validated one.
- `html/template` only for anything a browser parses, never `text/template`. Never wrap stored or fetched strings in `template.HTML`/`JS`/`URL`; that switches off the escaping every template depends on.
- Any outbound fetch of a client-supplied URL passes `fetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
- Any outbound fetch of a client-supplied URL passes `FetchableSeriesURL` (site + `https` + host check) first. `series_url` arrives in a PUT body, so without the gate the poller will probe arbitrary hosts from the server's own network position. New fetch path reuses the gate rather than re-deriving one.
- Cap every remote body with `io.LimitReader` (`maxBodyBytes`). An unbounded read is an OOM handed to whatever is on the other end.
- Compare secrets with `hmac.Equal` / `subtle.ConstantTimeCompare`, never `==`. A credential is matched by the SHA-256 the `readers` table holds, which is already a fixed-width equality — a new secret comparison must not regress to `==`.
- Errors: generic text to the client (`http.Error(w, "internal error", 500)`), detail to `log.Printf`. Never log `TOKEN_KEY`, a Reader's credential, `DISCORD_CLIENT_SECRET`, a session id, or a whole `Authorization` header.
+15 -3
View File
@@ -95,7 +95,7 @@ _Avoid_: run, cycle, tick, batch, poll history
**Forced Poll**:
A Poll the owner asks for by hand instead of waiting for the Series's turn. It jumps its
Lane's queue and ignores every waiting rule — the rest between Polls, a Sighting standing
in for a check, a Series only finished Readers hold — but never overrules a Site that is
in for a check, a finished Series — but never overrules a Site that is
refusing us, the Lane's spacing between fetches, or a Series with no page to fetch. Asked
for by marking the Series, never by commanding the poller, so it happens on the Lane's
next pass rather than at the moment of asking.
@@ -149,10 +149,22 @@ accent is permitted to signal.
_Avoid_: unread, update available
**Lifecycle bucket**:
Which of three mutually exclusive states a Bookmark sits in — reading, archived, or
finished. A Bookmark is in exactly one. Orthogonal to being a favourite.
Which of the two states a Bookmark sits in — reading or archived. A Bookmark is in exactly
one. Orthogonal to being a favourite. Finished is not a bucket: it is a fact about the
Series (see `series.finished_at`), owned by the owner and stamped once, and every Bookmark
on a finished Series is archived.
_Avoid_: state, status (as a domain word), list
**Finished Series**:
A Series the owner has marked finished, stamped once in `series.finished_at`
(epoch ms, zero means not finished). The owner is its only writer — no
adapter, no Reader, no Poll can set it — and a Forced Poll reads a finished
Series once for that pass and never clears the flag. It is a fact about the
Series, not a Bookmark bucket: every Bookmark on a finished Series is
archived, the Lane stops polling it (the due gate reads `finished_at = 0`),
and Readers see a label and nothing more.
_Avoid_: completed, done, dropped, shelved (that is Archived), ended
**Favourite**:
A reader's manual pin on a Bookmark. Orthogonal to the Lifecycle bucket, and never a
reason to reorder the list.
+2 -2
View File
@@ -29,8 +29,8 @@ Not a public reading tracker or social app — a private, self-hosted sync layer
## Capabilities and Constraints
- Two libraries (manga, novels) with lifecycle tabs: All / Updated / Favourites / Archived / Finished. Search-filter by title (client-side, `filter.js`).
- Card actions: continue (opens source site), toggle favourite, manual chapter override, archive, finish, remove — each move out of the list confirm-gated.
- Two libraries (manga, novels) with lifecycle tabs: All / Updated / Favourites / Archived. Search-filter by title (client-side, `filter.js`).
- Card actions: continue (opens source site), toggle favourite, manual chapter override, archive, remove — each move out of the list confirm-gated.
- "Continue reading" horizontal strip for series with an unread chapter.
- A Reader with no bookmarks at all sees a deliberate empty library offering both userscript install links, not an error and not a blank page.
- Isolation is the load-bearing invariant: two Readers cannot see or change each other's bookmarks. A series both track is one shared row polled once, with independent progress on each side.
+4 -4
View File
@@ -218,10 +218,10 @@ desktop for faster testing — install the same file unchanged.
keeps checking it for new chapters, so it is worth coming back to. Archiving
does not touch read progress, and reading an archived series leaves it
archived.
- **Finished**: series you have completed live in a **Finished** tab in the web
UI only. It is set there and nowhere else — the API rejects the value — and
finished series are hidden from every userscript tab and are no longer polled
for new chapters.
- **Finished**: the owner marks a Series finished from its detail page; the
backend stops polling it, and every Reader sees a read-only label. It is a
fact about the Series, not a Reader's bucket: old `Finished` bookmarks were
folded into Archived in the cutover, so there is no Finished tab.
- Bookmarks made on Asura appear when the panel is opened on Demonic, and vice
versa — the backend is the shared store.
+18 -5
View File
@@ -159,6 +159,15 @@ failures. The flag decays and they probe again.
- Browser Lanes wake Chrome only when 5+ Series are due or one has waited 15m,
and cover work runs in the background so a slow CDN can't eat a Lane's gap.
### Owner notices — `internal/notify`, `latest.Fault`, `latest.Notifier`, `latest.FaultsFrom`
The poller's outbound owner-notice path (issue #171): one condition today
(the stall), judged from the durable pass log alone so the poller and any
future reader of the same judgement cannot disagree. The webhook address is a
secret in the class of TOKEN_KEY — never logged, never rendered, never
carried in an error. The `owner_notices` suppression table (one row per
condition + site) is the only state; every threshold is derived, not stored.
### Covers — `Store.OnSeriesCreated`, `latest.Acquirer`, `latest.CoverBytesFetcher`, `Store.SetSeriesCover`
Acquired once when the first Bookmark of a Series is created, then served from
@@ -200,12 +209,16 @@ stored** and clients must adopt that response rather than their own payload.
### Lifecycle buckets — `status` on each bookmark
`reading` | `archived` | `finished`, orthogonal to `favorite`. Archived and
finished appear only in their own tab, never in All, Updated, Favourites or the
recent strip. The poller keeps checking archived series and skips finished ones.
`reading` | `archived`, orthogonal to `favorite`. Archived rows appear only
in their own tab, never in All, Updated, Favourites or the recent strip. The
poller keeps checking archived series; a finished Series (issue #157) is a
`series.finished_at` fact the Lane gate reads, with every bookmark on it
archived.
- `finished` is settable only from the web UI; `PUT /bookmarks/{key}` rejects
it with 400.
- `PUT /bookmarks/{key}` accepts only the two values; anything else —
`finished` included — is a plain 400, and the web UI's own status control
validates the same way. The 0016 migration is the only writer of the flag
today; the undo is writing 0.
- **An empty incoming status means "keep the stored one"**, and it is resolved
on the `VALUES` side of `Store.Upsert`, not in the conflict clause:
`excluded.*` is the post-evaluation row, so a default applied there would
+48 -1
View File
@@ -2,6 +2,7 @@ package main
import (
"bytes"
"database/sql"
"encoding/json"
"fmt"
"net/http"
@@ -298,7 +299,7 @@ func TestFlatWireFieldSet(t *testing.T) {
"series_url": true, "cover": true, "last_chapter": true,
"last_chapter_num": true, "last_chapter_url": true, "favorite": true,
"latest_chapter": true, "latest_chapter_num": true, "updated_at": true,
"status": true, "kind": true,
"status": true, "kind": true, "finished": true,
}
checkFlat := func(t *testing.T, payload []byte) map[string]json.RawMessage {
t.Helper()
@@ -362,6 +363,52 @@ func TestFlatWireFieldSet(t *testing.T) {
checkFlat(t, body2)
}
// finished is derived on the wire and read-only: a client PUT echoing a cached
// value, forward progress or not, must not change the Series' retired state,
// so GET still reports the truth after the echo (issues #157, #160).
func TestFinishedWireRoundTrip(t *testing.T) {
s, url := newTestStoreURL(t)
srv := newRouter(s, testConfig())
key := "asura:done"
putBookmark(t, srv, key, store.Bookmark{
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/done",
LastChapterNum: 10,
})
// The store writer for the flag is the admin surface's own and lands in
// the same wave (#158), so seed the fact with SQL, like store_test.go.
db, err := sql.Open("pgx", url)
if err != nil {
t.Fatalf("open db: %v", err)
}
defer db.Close()
if _, err := db.Exec(
`UPDATE series SET finished_at = 1000 WHERE site = 'asura' AND series_id = 'done'`); err != nil {
t.Fatalf("seed finished: %v", err)
}
got := getBookmarks(t, srv)
if len(got) != 1 || !got[0].Finished {
t.Fatalf("GET = %+v, want one bookmark carrying finished: true", got)
}
// A stale cache echoing the flag cannot un-finish (or finish) the Series.
for _, sent := range []bool{false, true} {
echoed := putBookmark(t, srv, key, store.Bookmark{
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/done",
LastChapterNum: 11, Finished: sent,
})
if !echoed.Finished {
t.Fatalf("PUT echoing Finished: %v reported finished = false, want true", sent)
}
got = getBookmarks(t, srv)
if !got[0].Finished {
t.Fatalf("GET after echoing Finished: %v = false, want the Series state preserved", sent)
}
}
}
// A PUT naming an existing series must ignore client-supplied title, cover and
// URL — the security boundary from ADR-0003, where a hostile site's scraped
// values could otherwise land on a shared row — while progress still lands.
+4 -4
View File
@@ -32,7 +32,7 @@ func TestPublicCoverServesStoredBytesUnauthenticated(t *testing.T) {
// The wire URL is what a client actually requests, so the path under test
// is taken from it rather than rebuilt by hand.
wire := st.CoverWireURL(store.CoverAddress(sourceURL))
wire := st.CoverWireURL(store.CoverAddressForBytes([]byte("\x00webp-bytes")))
path, ok := strings.CutPrefix(wire, testCoverBaseURL)
if !ok {
t.Fatalf("wire URL %q is not on the public origin %q", wire, testCoverBaseURL)
@@ -57,7 +57,7 @@ func TestPublicCoverServesStoredBytesUnauthenticated(t *testing.T) {
func TestPublicCoverRejectsUnknownAddress(t *testing.T) {
srv, _ := newWebTestServer(t, testConfig())
cases := map[string]string{
"unknown": "/covers/" + store.CoverAddress("https://cdn.example/never-stored.jpg"),
"unknown": "/covers/" + store.CoverAddressForBytes([]byte("never-stored")),
"malformed": "/covers/not-an-address",
"traversal": "/covers/../../etc/passwd",
"empty": "/covers/",
@@ -86,7 +86,7 @@ func TestPublicCoverNeverEchoesNonImage(t *testing.T) {
}
// A legitimate row, then the content type flipped behind the store's back:
// the bytes exist at the address, so only the type is hostile.
address := store.CoverAddress(sourceURL)
address := store.CoverAddressForBytes([]byte("<script>"))
if err := st.SetSeriesCover("asura", "solo", sourceURL, []byte("<script>"), "image/png"); err != nil {
t.Fatalf("seed row: %v", err)
}
@@ -127,7 +127,7 @@ func TestListRendersAcquiredCover(t *testing.T) {
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rr.Code)
}
want := `src="` + testCoverBaseURL + "/covers/" + store.CoverAddress(sourceURL) + `"`
want := `src="` + testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("\xff\xd8jpeg")) + `"`
if !strings.Contains(rr.Body.String(), want) {
t.Fatalf("rendered list does not contain %s", want)
}
+3 -6
View File
@@ -73,14 +73,11 @@ func (h *Handler) Put(w http.ResponseWriter, r *http.Request) {
}
// An empty status is "no opinion" and Upsert keeps the stored bucket.
// Finishing a series is a web-UI decision, so the JSON API refuses it
// rather than trusting every client to leave it alone.
// Anything else outside the two lifecycle buckets is a client bug, not
// something to silently coerce — finished included, which is no longer a
// bucket at all (issue #157).
switch b.Status {
case "", store.StatusReading, store.StatusArchived:
case store.StatusFinished:
http.Error(w, "status "+store.StatusFinished+" can only be set from the web UI",
http.StatusBadRequest)
return
default:
http.Error(w, "invalid status", http.StatusBadRequest)
return
+7 -7
View File
@@ -123,10 +123,10 @@ func TestAcquireFillsChapterAndCoverFromOneFetch(t *testing.T) {
if got.LatestChapterNum == nil || *got.LatestChapterNum != 181 {
t.Fatalf("LatestChapterNum = %v, want 181", got.LatestChapterNum)
}
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(acquireCoverURL); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
t.Fatalf("Cover = %q, want the absolute address %q", got.Cover, want)
}
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(acquireCoverURL))
body, contentType, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes")))
if err != nil || !ok {
t.Fatalf("CoverByAddress = %v, %v", ok, err)
}
@@ -182,7 +182,7 @@ func TestAcquireSkipsAnExistingSeries(t *testing.T) {
t.Fatalf("cover fetches = %d, want 1", got)
}
got := readBookmark(t, s, acquireKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(acquireCoverURL); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
t.Fatalf("Cover = %q, want the acquired one %q", got.Cover, want)
}
}
@@ -343,10 +343,10 @@ func TestAcquireKaganeCoverThroughBrowser(t *testing.T) {
t.Fatalf("browser cover fetched URL %q, want %q", got, kaganeCoverSrc)
}
got := readBookmark(t, s, kaganeKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(kaganeCoverSrc); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
t.Fatalf("Cover = %q, want the content-addressed URL %q", got.Cover, want)
}
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(kaganeCoverSrc))
body, contentType, ok, err := s.CoverByAddress(store.CoverAddressForBytes([]byte("cover-bytes")))
if err != nil || !ok {
t.Fatalf("CoverByAddress = %v, %v", ok, err)
}
@@ -381,7 +381,7 @@ func TestAcquireNovelfullCoverOverPlainTLS(t *testing.T) {
t.Fatalf("cover fetched from %q, want %q", got, novelfullCoverURL)
}
got := readBookmark(t, s, novelfullKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(novelfullCoverURL); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
}
@@ -451,7 +451,7 @@ func TestAcquireNovelfullCoverWithoutBrowser(t *testing.T) {
t.Fatalf("cover fetches = %d, want 1", got)
}
got := readBookmark(t, s, novelfullKey)
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(novelfullCoverURL); got.Cover != want {
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes([]byte("cover-bytes")); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
}
+2 -2
View File
@@ -124,7 +124,7 @@ func (f *BrowserFetcher) Get(ctx context.Context, seriesURL string) (string, int
// request must be made from inside the page so it carries the clearance
// cookie, and the API is the only place the list exists. Refusing any other
// address is the per-Site half of the SSRF gate, kept deliberately behind
// fetchableSeriesURL (see browserRead.Read).
// FetchableSeriesURL (see browserRead.Read).
func kaganeRead(seriesURL string, out *string) (chromedp.Action, bool) {
apiURL, ok := kaganeAPIURL(seriesURL)
if !ok {
@@ -326,7 +326,7 @@ func (f *BrowserFetcher) run(ctx context.Context, target string, read chromedp.A
// kaganeAPIURL maps a stored series_url to the JSON endpoint carrying its
// chapter list. Returning false for anything else is a second line of defence
// behind fetchableSeriesURL: a headless browser is a strong SSRF primitive and
// behind FetchableSeriesURL: a headless browser is a strong SSRF primitive and
// series_url is client-supplied, so the host is pinned here too.
func kaganeAPIURL(seriesURL string) (string, bool) {
u, err := url.Parse(seriesURL)
+1 -1
View File
@@ -174,7 +174,7 @@ func (f *TLSCoverFetcher) Fetch(ctx context.Context, sourceURL string) ([]byte,
return body, contentType, nil
}
// This gate deliberately differs from fetchableSeriesURL: cover hosts are
// This gate deliberately differs from FetchableSeriesURL: cover hosts are
// site-independent CDNs, so a Site host allowlist would reject valid covers.
func (f *TLSCoverFetcher) validateURL(ctx context.Context, u *url.URL) error {
if u == nil || u.Scheme != "https" || u.Host == "" || u.User != nil {
+210
View File
@@ -0,0 +1,210 @@
package latest
import (
"context"
"fmt"
"time"
"bookmarkmanager/backend/internal/store"
)
// ConditionStall is the owner-notice machine word for a Lane that owed Polls,
// made none, and has nothing to say for it. The word is the message's footer
// and its suppression key; it is wire-stable. #172 declares the other three
// words (no-browser-route, sidecar-down, adapter-broken); this ticket
// declares only the stall.
const ConditionStall = "stall"
// ConditionNoBrowserRoute is the owner-notice machine word for a Site whose
// challenge refuses for longer than the owner window with no browser route
// to clear it — the 403-with-interstitial the reader maps to
// errChallengeHeld (read.go), which plain TLS cannot clear. The word is the
// message's footer and its suppression key; it is wire-stable.
const ConditionNoBrowserRoute = "no-browser-route"
// ConditionSidecarDown is the owner-notice machine word for a browser
// sidecar no Lane has reached for longer than the owner window: every
// browser-backed Lane's latest pass is a sidecar skip. The word is the
// message's footer and its suppression key; it is wire-stable, and its
// suppression row holds the empty Site (AC4).
const ConditionSidecarDown = "sidecar-down"
// ConditionAdapterBroken is the owner-notice machine word for a Site whose
// adapter stopped finding chapters: more than half of its Series hold an
// old no-chapter failure row. The word is the message's footer and its
// suppression key; it is wire-stable.
const ConditionAdapterBroken = "adapter-broken"
// OwnerWindow is the class-level staleness boundary every owner-notice
// condition measures against — the same twelve hours the Lanes page's
// "not checked in 12h" filter uses (internal/web/admin.go). Declared here
// once so #172's three conditions and the admin filters share one figure.
const OwnerWindow = 12 * time.Hour
// Fault is one condition the owner is told about, judged from durable rows
// alone. Site is "" for a fault that is not one Site's.
type Fault struct {
Condition string // one of the Condition* words
Site string
Since int64 // unix ms the episode began; the message's age
}
// FaultInput is everything the judgement reads. A struct so #172's three
// conditions can add inputs without changing either caller.
type FaultInput struct {
Passes []store.LanePass
// RefusingSince is, per Site, the unix ms when that Site's current
// unbroken run of refusing passes began, or absent when its latest pass
// did not refuse. Its zero value is an empty map, which contributes no
// fault.
RefusingSince map[string]int64
// SidecarOK is, per browser-backed Site, the unix ms of that Site's most
// recent pass that actually reached the sidecar. Zero when the pass log
// holds none — an asleep Lane never reached it and never counts as
// evidence either way. Its zero value is an empty map.
SidecarOK map[string]int64
// NoChapterShare is, per Site, the share of that Site's Series holding a
// no-chapter failure row older than the owner window. Its zero value is
// an empty map, which contributes no fault.
NoChapterShare map[string]float64
}
// FaultsFrom judges the owner-notice conditions from durable rows alone, so
// the poller and the landing page cannot disagree about what a fault is.
//
// A Lane stalls when its latest pass shows due > 0, none checked, no skip
// value and no refusal — exactly the row the mid-loop browser loss writes
// (see the comment at the outcomeUnreachable return in runLanePass), so a
// Lane that owed Polls, made none, and has nothing to say for it is a fault.
// The three #172 conditions share the same OwnerWindow boundary and the same
// fail-open shape: an input's absence contributes no fault, never a false
// one.
//
// - no-browser-route: a Site whose refusing run began before the window
// and that has no browser route to clear the challenge (AC2). Both
// refusal shapes count — the gate's skip='refusing' rows and the loop's
// refused>0 rows (the twice-refused break writes skip="") — so the run
// stays unbroken across them, and one healthy pass ends it.
// - sidecar-down: every browser-backed Site's latest pass is a sidecar
// skip — SkipSidecarDown or SkipNoFetcher, the two early returns that
// record a skip value — and each Site's most recent sidecar-reaching
// pass is older than the window (AC4). The skip clause is what keeps
// SkipAsleep out: a Lane under both wake thresholds is the commonest
// healthy state, never reached the sidecar, and would otherwise age into
// a false alarm. Emitted once, with Site "" (AC4's one row per episode).
// - adapter-broken: more than half of one Site's Series hold a no-chapter
// failure row older than the window (AC6). Strictly above half: the
// filter is the tool, and one Site change makes hundreds of rows, so a
// single failing Series never fires (AC7). The episode's age is the
// window — the old rows prove the episode is at least that old.
func FaultsFrom(in FaultInput, now time.Time) []Fault {
var faults []Fault
for _, p := range in.Passes {
if p.Due > 0 && p.Checked == 0 && p.Skip == "" && p.Refused == 0 {
faults = append(faults, Fault{Condition: ConditionStall, Site: p.Site, Since: p.RanAt})
}
}
cutoff := now.Add(-OwnerWindow).UnixMilli()
for site, since := range in.RefusingSince {
if since < cutoff && !isBrowserSite(site) {
faults = append(faults, Fault{Condition: ConditionNoBrowserRoute, Site: site, Since: since})
}
}
if since, down := sidecarDownSince(in.Passes, in.SidecarOK, cutoff); down {
faults = append(faults, Fault{Condition: ConditionSidecarDown, Site: "", Since: since})
}
for site, share := range in.NoChapterShare {
if share > 0.5 {
faults = append(faults, Fault{Condition: ConditionAdapterBroken, Site: site, Since: now.Add(-OwnerWindow).UnixMilli()})
}
}
return faults
}
// sidecarDownSince reports whether no browser Lane has reached the sidecar
// for longer than the owner window and, when it has, the last moment any
// Lane reached it. The skip clause — every browser-backed Site's latest pass
// must be SkipSidecarDown or SkipNoFetcher — keeps SkipAsleep out (see
// FaultsFrom). A Site with no pass row at all is not judged down either: a
// fresh database is not a dead sidecar.
func sidecarDownSince(passes []store.LanePass, ok map[string]int64, cutoff int64) (since int64, down bool) {
latest := make(map[string]store.LanePass, len(passes))
for _, p := range passes {
latest[p.Site] = p
}
for _, site := range browserBackedSites() {
p, found := latest[site]
if !found || (p.Skip != SkipSidecarDown && p.Skip != SkipNoFetcher) {
return 0, false
}
reached := ok[site]
if reached > 0 && reached >= cutoff {
return 0, false
}
if reached > since {
since = reached
}
}
// No Lane's retained pass log shows a sidecar reach: the honest age is
// the window itself — "at least twelve hours" — not the epoch, which
// humanAge would render as tens of thousands of days.
if since == 0 {
since = cutoff
}
return since, true
}
// Notifier delivers one owner notice. The poller neither retries nor queues:
// an error is logged and the suppression row left unwritten, so the next pass
// tries again while the condition holds.
type Notifier interface {
Notify(ctx context.Context, f Fault, sentence, href string) error
}
// ownerNoticeConditions is every condition this package judges, for the
// clear loop in recordPass: a condition absent from a pass's fault list
// forgets its episode, so the next occurrence sends again. #172 extends the
// list when it adds its conditions.
var ownerNoticeConditions = []string{ConditionStall, ConditionNoBrowserRoute, ConditionSidecarDown, ConditionAdapterBroken}
// noticeFor renders one fault's message: the description sentence — condition,
// age, repair — and the deep link the embed's title points at. Each condition
// provides its own wording; the stall is the only one today (issue #171).
func noticeFor(f Fault, row store.LanePass, now time.Time) (sentence, href string) {
switch f.Condition {
case ConditionStall:
return fmt.Sprintf(
"%s owed %d Polls and made none — %s; check the Lane's browser sidecar and the Site's challenge state",
f.Site, row.Due, humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
case ConditionNoBrowserRoute:
return fmt.Sprintf(
"%s has refused for %s with no browser route — the challenge does not clear on plain TLS; redeploy or add a browser route",
f.Site, humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
case ConditionSidecarDown:
return fmt.Sprintf(
"no browser Lane has reached the sidecar for %s — the browser sidecar is down; start or repair the browser machine",
humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
case ConditionAdapterBroken:
return fmt.Sprintf(
"more than half of %s's Series have failed no-chapter reads for at least %s — the Site's layout changed and the adapter is broken",
f.Site, humanAge(now.Sub(time.UnixMilli(f.Since)))), "/admin/lanes"
}
return "", ""
}
// humanAge renders a duration the way an owner reads it in a message: minutes
// under an hour, then hours, then days; a stall that was just born reads
// "just now".
func humanAge(d time.Duration) string {
switch {
case d < time.Minute:
return "just now"
case d < time.Hour:
return fmt.Sprintf("%dm", int(d.Minutes()))
case d < 24*time.Hour:
return fmt.Sprintf("%dh", int(d.Hours()))
default:
return fmt.Sprintf("%dd", int(d.Hours()/24))
}
}
+247 -19
View File
@@ -46,7 +46,10 @@ type Poller struct {
// CoverBytesFetch is optional; it handles plain-TLS sources through the
// same failure-isolated prefetch path.
CoverBytesFetch CoverBytesFetcher
Now func() time.Time // injected so tests can freeze it
// Notify delivers owner notices. Nil disables the whole path (issue #171):
// the poller is not the place a missing webhook becomes an error.
Notify Notifier
Now func() time.Time // injected so tests can freeze it
// eligibleCount reports how many of a Site's Series are eligible for
// polling, defaulting to Store.EligibleSeriesCount. Injected so tests can
// fail the count alone: the eligible query shares the due query's tables,
@@ -98,6 +101,23 @@ func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, cover stri
}()
}
// replaceCover is the Forced Poll's Cover path: the owner asked to accept the
// page as it now stands, so where fillBlankCover leaves a non-blank Cover
// alone (ADR-0007) this writes through whatever the page's Cover URL answers
// with, whether one exists or not. The accepted consequence (issue #135):
// refreshing the Cover and re-reading the chapters are one act — there is no
// Cover-only refetch.
func (p *Poller) replaceCover(ctx context.Context, sr store.Series, cover string) {
if cover == "" {
return
}
p.coverWG.Add(1)
go func() {
defer p.coverWG.Done()
p.storeCover(ctx, sr, cover)
}()
}
// prefetchCover heals Series that already carry a third-party source URL but
// no stored address — the state left by client-supplied covers before
// acquisition moved server-side. Every Site takes the same path; fetchCoverBytes
@@ -122,17 +142,42 @@ func (p *Poller) prefetchCover(ctx context.Context, sr store.Series) {
p.storeCover(ctx, sr, sr.Cover)
}
// storeCover fetches bytes for sourceURL and points the Series at them. Every
// failure is logged against the Series and swallowed so the chapter poll
// cannot see it.
// storeCover fetches bytes for sourceURL and points the Series at them: a
// fill-only write for an ordinary pass, a write-through for a forced one
// (issue #135). Every failure is logged against the Series and swallowed so
// the chapter poll cannot see it.
func (p *Poller) storeCover(ctx context.Context, sr store.Series, sourceURL string) {
bytes, contentType, err := fetchCoverBytes(ctx, sourceURL, p.CoverFetch, p.CoverBytesFetch)
if err != nil {
log.Printf("latest poll %q: fetch cover %s: %v", sr.Key(), sourceURL, err)
return
}
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType); err != nil {
if !sr.Forced {
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType); err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
}
return
}
// The forced write replaces whether or not a Cover exists, and the row
// then tells the three outcomes apart: a blank filled, identical artwork
// re-served — an honest no-op — or a replacement whose previous address
// is stranded and reclaimed below. A failed reclaim is logged and the
// stranded bytes stay served until a later call reclaims them.
previous, current, err := p.Store.ReplaceSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType)
if err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
return
}
switch {
case previous == "":
log.Printf("latest poll %q: cover filled at %s", sr.Key(), current)
case previous == current:
log.Printf("latest poll %q: cover unchanged, the site re-serves the same bytes", sr.Key())
default:
if err := p.Store.ReclaimCover(previous); err != nil {
log.Printf("latest poll %q: reclaim cover %s: %v", sr.Key(), previous, err)
}
log.Printf("latest poll %q: cover replaced %s -> %s", sr.Key(), previous, current)
}
}
@@ -231,7 +276,9 @@ const (
// (issue #141). The classification the read already makes is counted, never a
// second taxonomy: refused is the Site holding a challenge, unreachable the
// browser interrupting, noChapter a 200 with real HTML but no chapter links,
// unfetchable the host pin or a missing fetcher, and errors everything else.
// unfetchable the host pin or a missing fetcher, notFound a 4xx other than
// the 403 refusal — the Site answered with a client status — and errors
// everything else.
type readOutcome int
const (
@@ -241,14 +288,33 @@ const (
outcomeNoChapter
outcomeUnfetchable
outcomeError
outcomeNotFound
)
// word returns the wire spelling this outcome stores in poll_failures — the
// same strings the pass log's columns use (C1, issue #164). Success, refusal
// and browser loss return "" so recordFailure's "no statement" case is one
// return: a challenge or a lost sidecar is no evidence about any particular
// Series (ADR-0016).
func (o readOutcome) word() string {
switch o {
case outcomeNotFound:
return "not_found"
case outcomeNoChapter:
return "no_chapter"
case outcomeUnfetchable:
return "unfetchable"
case outcomeError:
return "errors"
}
return ""
}
// outcomeCounts are the five named outcome counts of one pass. A success
// count is derived, never stored: checked minus the four, with unreachable
// excluded because the sidecar-loss path returns before the checked counter
// increments (issue #141).
// outcomeCounts are the six named outcome counts of one pass. A success
// count is derived, never stored: checked minus the five named failures,
// with unreachable excluded because the sidecar-loss path returns before the
// checked counter increments (issue #141).
type outcomeCounts struct {
refused, unreachable, noChapter, unfetchable, errors int
refused, unreachable, noChapter, unfetchable, errors, notFound int
}
func (c *outcomeCounts) add(o readOutcome) {
@@ -261,6 +327,8 @@ func (c *outcomeCounts) add(o readOutcome) {
c.noChapter++
case outcomeUnfetchable:
c.unfetchable++
case outcomeNotFound:
c.notFound++
case outcomeError:
c.errors++
}
@@ -292,7 +360,7 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
// carries the previous pass's forward inside recordPass.
fig := passFigures{}
rec := passRecord{site: name, ranAt: now.UnixMilli()}
defer func() { p.recordPass(rec, fig) }()
defer func() { p.recordPass(ctx, rec, fig) }()
// One Lane row read at the top of a pass, serving two gates (issue #139).
// Both stamps outlive our process, so the gates read the durable row
@@ -409,6 +477,7 @@ func (p *Poller) runLanePass(ctx context.Context, name string, paced bool) time.
}
}
outcome := p.checkOne(ctx, sr)
p.recordFailure(sr, outcome)
if outcome == outcomeUnreachable {
// The mid-loop browser loss writes an empty skip on purpose: the
// pass returns before the checked counter increments, so its row
@@ -461,8 +530,9 @@ type passFigures struct {
// pass's due, gap, clamped and checked forward rather than stating zeroes it
// did not measure; the skip column says why it declined, so the zeroes that
// remain (due-query, no-fetcher) read as explanations rather than
// measurements.
func (p *Poller) recordPass(rec passRecord, fig passFigures) {
// measurements. Once the row is durable, the owner-notice judgement runs
// beside it (issue #171).
func (p *Poller) recordPass(ctx context.Context, rec passRecord, fig passFigures) {
row := store.LanePass{
Site: rec.site,
RanAt: rec.ranAt,
@@ -475,6 +545,7 @@ func (p *Poller) recordPass(rec passRecord, fig passFigures) {
Unreachable: rec.counts.unreachable,
NoChapter: rec.counts.noChapter,
Unfetchable: rec.counts.unfetchable,
NotFound: rec.counts.notFound,
Errors: rec.counts.errors,
}
if row.GapMS == 0 {
@@ -490,7 +561,111 @@ func (p *Poller) recordPass(rec passRecord, fig passFigures) {
}
if err := p.Store.RecordLanePass(row, rec.ranAt-lanePassRetention.Milliseconds()); err != nil {
log.Printf("latest poll %s: record lane pass: %v", rec.site, err)
return
}
p.ownerNotices(ctx, row)
}
// ownerNotices judges the owner-notice conditions for the pass just recorded
// and fires (issue #171). It sits in recordPass because that deferred call is
// the one place every return path passes through: three of the four
// conditions occur on early returns and the success path can never see them.
// The judgement reads the whole pass log plus three derived reads — the
// other Lanes' latest passes, each Site's refusing-run start, its last
// sidecar-reaching pass, and its no-chapter share — so one pass judges every
// condition (issue #172). Per fault: NoticeSent → send → MarkNoticeSent, so
// a fault lasting a month sends one message, not one per pass; a condition
// absent from this pass's fault list forgets its episode, so the next
// occurrence sends again. Everything here is best-effort: a failed send, a
// failed store read and a failed notice write are all logged and never
// change the pass's outcome counts or its return value. The clear runs even
// when Notify is nil, so a deployment that turns the webhook off does not
// leave stale rows that suppress the first real notice after it is turned
// back on.
func (p *Poller) ownerNotices(ctx context.Context, row store.LanePass) {
now := p.Now()
in := FaultInput{Passes: []store.LanePass{row}}
// Each read fails independently: a failure logs and contributes no fault,
// never a false one.
if passes, err := p.Store.LatestLanePasses(); err != nil {
log.Printf("latest poll %s: latest lane passes: %v", row.Site, err)
} else {
in.Passes = passes
}
if since, err := p.Store.RefusingSince(now.UnixMilli()); err != nil {
log.Printf("latest poll %s: refusing since: %v", row.Site, err)
} else {
in.RefusingSince = since
}
if ok, err := p.Store.SidecarOK(browserBackedSites()); err != nil {
log.Printf("latest poll %s: sidecar ok: %v", row.Site, err)
} else {
in.SidecarOK = ok
}
if share, err := p.Store.NoChapterShare(now.Add(-OwnerWindow).UnixMilli()); err != nil {
log.Printf("latest poll %s: no-chapter share: %v", row.Site, err)
} else {
in.NoChapterShare = share
}
faults := FaultsFrom(in, now)
bySite := make(map[string]store.LanePass, len(in.Passes))
for _, pass := range in.Passes {
bySite[pass.Site] = pass
}
for _, f := range faults {
if p.Notify == nil {
continue
}
sent, err := p.Store.NoticeSent(f.Condition, f.Site)
if err != nil {
log.Printf("latest poll %s: notice sent: %v", row.Site, err)
continue
}
if sent {
continue
}
// The fault's own Site's pass renders its sentence — a stall judged
// from another Lane's pass must not quote this pass's figures.
pass, ok := bySite[f.Site]
if !ok {
pass = row
}
sentence, href := noticeFor(f, pass, now)
if err := p.Notify.Notify(ctx, f, sentence, href); err != nil {
// The stamp stays unset: no queue, no backoff — the condition is
// durable, so the next pass tries again while it holds.
log.Printf("latest poll %s: owner notice %s: %v", row.Site, f.Condition, err)
continue
}
if err := p.Store.MarkNoticeSent(f.Condition, f.Site, now.UnixMilli()); err != nil {
log.Printf("latest poll %s: mark notice sent: %v", row.Site, err)
}
}
for _, cond := range ownerNoticeConditions {
if !hasFault(faults, cond, row.Site) {
if err := p.Store.ClearNotice(cond, row.Site); err != nil {
log.Printf("latest poll %s: clear owner notice: %v", row.Site, err)
}
}
}
// sidecar-down suppresses under the empty Site — one row across all
// browser Lanes (AC4) — so clear that row when it is absent from this
// pass's fault list, and a lifted sidecar fires again when it returns.
if !hasFault(faults, ConditionSidecarDown, "") {
if err := p.Store.ClearNotice(ConditionSidecarDown, ""); err != nil {
log.Printf("latest poll %s: clear owner notice: %v", row.Site, err)
}
}
}
// hasFault reports whether faults hold the given condition for the site.
func hasFault(faults []Fault, condition, site string) bool {
for _, f := range faults {
if f.Condition == condition && f.Site == site {
return true
}
}
return false
}
// countEligible routes the eligible count through the test seam when one is
@@ -565,13 +740,35 @@ func maxSeriesWait(due []store.Series, now time.Time, rest time.Duration) time.D
}
return oldest
}
// recordFailure keeps one Series' failure row in step with its read
// (ADR-0016): a failure word is upserted, a successful read deletes the row,
// and refused or unreachable issue no statement at all. Called for every
// outcome from the pass loop, so the four failure words and the success path
// share one write point, and a forced Poll that reads the page clears through
// the ordinary success path — no branch of its own. Log a store failure and
// carry on: this is best-effort, and no single bad Series may stall a Lane.
func (p *Poller) recordFailure(sr store.Series, outcome readOutcome) {
if outcome == outcomeSuccess {
if err := p.Store.ClearSeriesFailure(sr.Site, sr.SeriesID); err != nil {
log.Printf("latest poll %q: clear failure: %v", sr.Key(), err)
}
return
}
word := outcome.word()
if word == "" {
return
}
if err := p.Store.RecordSeriesFailure(sr.Site, sr.SeriesID, word, p.Now().UnixMilli()); err != nil {
log.Printf("latest poll %q: record failure: %v", sr.Key(), err)
}
}
// checkOne re-checks one series. Every failure path here is "log and move on":
// the poller is a best-effort enhancement, and no single bad series may stall a
// Lane or take down the process. The returned outcome classifies the read for
// the pass row (issue #141), so the Lane can count a refusal, a lost browser,
// a chapter-less page, an unfetchable address or a transport error without
// re-deriving the taxonomy.
// a chapter-less page, an unfetchable address, a missing page or a transport
// error without re-deriving the taxonomy.
func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOutcome) {
defer func() {
if r := recover(); r != nil {
@@ -615,13 +812,40 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) (outcome readOut
if errors.Is(err, errBrowserInterrupted) {
return outcomeUnreachable
}
if errors.Is(err, errNotFound) {
return outcomeNotFound
}
return outcomeError
}
// A legacy cover source is healed independently of the page read.
p.healCover(ctx, sr)
// Cover fill is independent of the chapter signal: a page that lost its
// chapter list may keep its og:image, and a blank Series heals either way.
p.fillBlankCover(ctx, sr, facts.Cover)
// A forced pass writes the Cover through the replace path instead.
if sr.Forced {
p.replaceCover(ctx, sr, facts.Cover)
} else {
p.fillBlankCover(ctx, sr, facts.Cover)
}
// Learned from the successful read: the write sits after the error switch
// (a refused, unreachable or errored read reaches nothing) and before the
// returns below — a completed page whose chapter number did not change
// still has to write. The transition is zero-versus-nonzero, not the
// stamp's value: a Series still completed keeps its original stamp, so the
// age #170 prints is "since the Site first said so"; one that stopped
// being completed is zeroed.
stamp := int64(0)
if facts.SiteCompleted {
stamp = p.Now().UnixMilli()
}
if (sr.SiteCompletedAt == 0) != (stamp == 0) {
if err := p.Store.SetSiteCompletedAt(sr.Site, sr.SeriesID, stamp); err != nil {
// Best-effort, like every poller write: never change the outcome
// word the pass counts.
log.Printf("latest poll %q: set site completed: %v", sr.Key(), err)
}
}
if !facts.HasLatest {
// Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a rest instead of hot-looping.
@@ -710,7 +934,7 @@ func (p *Poller) waitCovers() {
p.coverWG.Wait()
}
// fetchableSeriesURL reports whether site is a Site the registry knows and
// FetchableSeriesURL reports whether site is a Site the registry knows and
// seriesURL is safe to hand to a fetcher: an https URL whose host matches the
// Site's pinned hostname exactly. series_url comes from client-supplied PUT
// bodies, so this is a defence against the poller being used to probe
@@ -719,7 +943,11 @@ func (p *Poller) waitCovers() {
// browser Site guards a control that executes JavaScript and carries cookies,
// a parser Site guards a wasted request — but the rule is one rule, from the
// registry.
func fetchableSeriesURL(site, seriesURL string) bool {
//
// The owner's series URL repair (issue #151) is a second caller: the web
// layer validates with this same gate before storing a repair, so there is
// never a second copy of it.
func FetchableSeriesURL(site, seriesURL string) bool {
s, known := sites[site]
if !known {
return false
File diff suppressed because it is too large Load Diff
+25 -8
View File
@@ -6,15 +6,19 @@ import (
"fmt"
)
// seriesRead carries the two facts the poll and the acquirer both extract
// from a series page. Persistence, stamps and scheduling stay with the
// callers, so the policies that keep the two flows distinct (stamp order,
// rests) are not swallowed by the module.
// seriesRead carries the facts the poll and the acquirer both extract from a
// series page. Persistence, stamps and scheduling stay with the callers, so
// the policies that keep the two flows distinct (stamp order, rests) are not
// swallowed by the module.
type seriesRead struct {
Latest latestChapter
HasLatest bool
Cover string
HasCover bool
// SiteCompleted is whether the Site's own completed value was on the page.
// A challenge body and a redesign both read false — an absent hint, never
// a claim (issue #168).
SiteCompleted bool
// BodyLen is the fetched body's length, surfaced because the no-chapter
// log uses it to tell a markup change from a body the size cap cut short.
BodyLen int
@@ -25,15 +29,18 @@ type seriesRead struct {
// errChallengeHeld (browser.go) is the outcome of a Site that answered with
// its interstitial — status 403 (cf-mitigated) or a challenge page body — and
// is how a Lane tells a refusal from an ordinary failure (issue #100).
// errNotFound marks any other 4xx: the page is gone — a fact the owner can act
// on — as distinct from a Site or database that is merely unwell (issue #164).
var (
errNotFetchable = errors.New("series url not fetchable")
errNoFetcher = errors.New("no fetcher for site")
errNotFound = errors.New("series page not found")
)
// readSeriesPage performs the series-page read the poll and the acquirer have
// in common: gate the address, choose the route, fetch the page, extract the
// Latest Chapter and the Cover address. It persists nothing and stamps
// nothing.
// Latest Chapter, the Cover address and the Site's completed value. It
// persists nothing and stamps nothing.
//
// series_url arrives in a client-supplied PUT body (PUT /bookmarks/{key}
// accepts any string), so the gate is not an optimisation against burning a
@@ -41,7 +48,7 @@ var (
// its own network position to whatever URL a token-holder writes, including
// link-local/internal addresses or non-https schemes.
func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fetcher) (seriesRead, error) {
if !fetchableSeriesURL(site, seriesURL) {
if !FetchableSeriesURL(site, seriesURL) {
return seriesRead{}, fmt.Errorf("%w: site=%q url=%q", errNotFetchable, site, seriesURL)
}
f := fetcherFor(site, browser, tls)
@@ -59,6 +66,9 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errChallengeHeld, seriesURL, status)
}
if status != 200 {
if status >= 400 && status < 500 {
return seriesRead{}, fmt.Errorf("%w: fetch %s: status %d", errNotFound, seriesURL, status)
}
return seriesRead{}, fmt.Errorf("fetch %s: status %d", seriesURL, status)
}
if isInterstitial(body) {
@@ -69,5 +79,12 @@ func readSeriesPage(ctx context.Context, site, seriesURL string, browser, tls Fe
}
latest, hasLatest := latestChapterFrom(site, seriesURL, body)
cover, hasCover := coverFrom(site, seriesURL, body)
return seriesRead{Latest: latest, HasLatest: hasLatest, Cover: cover, HasCover: hasCover, BodyLen: len(body)}, nil
return seriesRead{
Latest: latest,
HasLatest: hasLatest,
Cover: cover,
HasCover: hasCover,
SiteCompleted: siteCompletedFrom(site, seriesURL, body),
BodyLen: len(body),
}, nil
}
+120 -15
View File
@@ -22,10 +22,10 @@ type latestChapter struct {
// site answers the fixed questions every series-page read asks of its Site
// (ADR-0009): the host its addresses must carry, how to find the Latest
// Chapter and the Cover address in a body, and — for a Site behind a
// JavaScript challenge — how to read its payload from a cleared tab. One
// entry describes everything about one Site, and nowhere else gets to compare
// the site string.
// Chapter and the Cover address in a body, whether the Site calls the work
// completed, and — for a Site behind a JavaScript challenge — how to read its
// payload from a cleared tab. One entry describes everything about one Site,
// and nowhere else gets to compare the site string.
type site struct {
// Host is the exact hostname a series_url for this Site must carry.
Host string
@@ -33,6 +33,10 @@ type site struct {
LatestChapter func(seriesURL, body string) (latestChapter, bool)
// Cover finds the Cover address in a fetched body.
Cover func(seriesURL, body string) (string, bool)
// Completed reports whether this body carries the Site's own completed
// value. False for a body that carries any other value, and false for a
// failed extraction — never an error and never a third state.
Completed func(seriesURL, body string) bool
// Rest is how long a Series of this Site rests between Polls.
Rest time.Duration
// Gap is the Lane's strictest pace: at least one second must pass between
@@ -46,7 +50,7 @@ type site struct {
type browserRead struct {
// Read builds the tab read for seriesURL, refusing (false) an address
// this Site will not open in a browser — the per-Site half of the SSRF
// gate, kept deliberately behind fetchableSeriesURL: a headless browser
// gate, kept deliberately behind FetchableSeriesURL: a headless browser
// executes JavaScript and carries cookies, and series_url is
// client-supplied.
Read func(seriesURL string, out *string) (chromedp.Action, bool)
@@ -299,34 +303,42 @@ func kaganeCoverURL(body string) string {
return ""
}
func comixCoverURL(seriesURL, body string) string {
// comixDetailQuery returns the ["manga","detail","<id>"] query entry of
// comix's initial-data JSON, or nil. The cover and completed reads share the
// scoped lookup so a "recommended" strip entry can never contribute either
// answer.
func comixDetailQuery(seriesURL, body string) json.RawMessage {
id, ok := comixSeriesID(seriesURL)
if !ok {
return ""
return nil
}
data := comixInitialDataRe.FindStringSubmatch(body)
if data == nil {
return ""
return nil
}
var state struct {
Queries map[string]json.RawMessage `json:"queries"`
}
if err := json.Unmarshal([]byte(data[1]), &state); err != nil {
return nil
}
return state.Queries[`["manga","detail","`+id+`"]`]
}
func comixCoverURL(seriesURL, body string) string {
detail := comixDetailQuery(seriesURL, body)
if len(detail) == 0 {
return ""
}
raw := state.Queries[`["manga","detail","`+id+`"]`]
if len(raw) == 0 {
return ""
}
var detail struct {
var entry struct {
Poster struct {
Medium string `json:"medium"`
} `json:"poster"`
}
if err := json.Unmarshal(raw, &detail); err != nil {
if err := json.Unmarshal(detail, &entry); err != nil {
return ""
}
return publishedCoverURL(detail.Poster.Medium)
return publishedCoverURL(entry.Poster.Medium)
}
// ogImageCover reads the og:image metadata shared by asura, demonic and
@@ -360,6 +372,87 @@ func coverFrom(site, seriesURL, body string) (string, bool) {
return "", false
}
// asuraStatusRe matches the status value inside the escaped astro-island
// props blob, in both the &quot; form the served document carries and the "
// form a decoded copy would. "completed" is the only true value: "dropped"
// is scanlation editorial (the work itself continues elsewhere) and hiatus is
// its own value.
var asuraStatusRe = regexp.MustCompile(`(?:&quot;|")status(?:&quot;|"):\[0,(?:&quot;|")completed(?:&quot;|")\]`)
func asuraCompleted(_, body string) bool {
return asuraStatusRe.MatchString(body)
}
// demonicStatusRe matches the info block's status pair: a Status label <li>
// immediately followed by the value <li>. The site's whole status vocabulary
// is {Ongoing, Completed} (its advanced-search status filter), so the literal
// Completed value is the entire signal.
var demonicStatusRe = regexp.MustCompile(`<li[^>]*>\s*Status\s*</li>\s*<li[^>]*>\s*Completed\s*</li>`)
func demonicCompleted(_, body string) bool {
return demonicStatusRe.MatchString(body)
}
// comixCompleted reads "status" from the scoped detail entry only;
// "finished" is the completed value, and on_hiatus and discontinued are
// distinct values.
func comixCompleted(seriesURL, body string) bool {
detail := comixDetailQuery(seriesURL, body)
if len(detail) == 0 {
return false
}
var entry struct {
Status string `json:"status"`
}
if err := json.Unmarshal(detail, &entry); err != nil {
return false
}
return entry.Status == "finished"
}
// kaganeCompleted reads publication_status only: upload_status is the
// release's state, and the two provably diverge ('Cause Calypso Can,
// 2026-08-19: publication Ongoing, upload Hiatus), so a Completed upload
// must never read as a Completed work.
func kaganeCompleted(_, body string) bool {
var series struct {
PublicationStatus string `json:"publication_status"`
}
if err := json.Unmarshal([]byte(body), &series); err != nil {
return false
}
return series.PublicationStatus == "Completed"
}
// novelfullStatusRe matches the info panel's status link. The page's whole
// status vocabulary is {Ongoing, Completed} (the "OnGoing" spelling aliases
// "Ongoing" on the taxonomy), so the Completed href is the signal.
var novelfullStatusRe = regexp.MustCompile(`href="/status/Completed"`)
func novelfullCompleted(_, body string) bool {
return novelfullStatusRe.MatchString(body)
}
// lnwCompleted matches creativeWorkStatus in the head's JSON-LD block. The
// whole body is scanned and the comment marker is not required, unlike
// lnwLatestChapter: the status block sits ahead of the visitor-writable
// thread, and hiatus maps to a distinct PotentialActionStatus value.
var lnwStatusRe = regexp.MustCompile(`"creativeWorkStatus"\s*:\s*"https://schema\.org/CompletedActionStatus"`)
func lnwCompleted(_, body string) bool {
return lnwStatusRe.MatchString(body)
}
// siteCompletedFrom reports whether the Site calls this work completed, via
// the Site's registry entry. False for an unknown site, a challenge body and
// a redesign alike: an absent hint, never a claim.
func siteCompletedFrom(site, seriesURL, body string) bool {
if fn := sites[site].Completed; fn != nil {
return fn(seriesURL, body)
}
return false
}
// metaContent returns the content of the first <meta> whose attrName is
// attrValue. It keeps scanning after an empty match so a later published cover
// is not hidden by an empty tag.
@@ -449,6 +542,7 @@ var sites = map[string]site{
Host: "asurascans.com",
LatestChapter: asuraLatestChapter,
Cover: ogImageCover,
Completed: asuraCompleted,
Rest: defaultRest,
Gap: defaultGap,
},
@@ -456,6 +550,7 @@ var sites = map[string]site{
Host: "demonicscans.org",
LatestChapter: demonicLatestChapter,
Cover: ogImageCover,
Completed: demonicCompleted,
Rest: defaultRest,
Gap: defaultGap,
},
@@ -463,6 +558,7 @@ var sites = map[string]site{
Host: "comix.to",
LatestChapter: comixLatestChapter,
Cover: comixCoverEntry,
Completed: comixCompleted,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
@@ -479,6 +575,7 @@ var sites = map[string]site{
Host: "kagane.to",
LatestChapter: kaganeLatestChapter,
Cover: kaganeCoverEntry,
Completed: kaganeCompleted,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
@@ -493,6 +590,7 @@ var sites = map[string]site{
Host: "novelfull.com",
LatestChapter: novelfullLatestChapter,
Cover: novelfullCoverEntry,
Completed: novelfullCompleted,
Rest: defaultRest,
Gap: defaultGap,
Browser: &browserRead{
@@ -507,6 +605,7 @@ var sites = map[string]site{
Host: "lightnovelworld.net",
LatestChapter: lnwLatestChapter,
Cover: ogImageCover,
Completed: lnwCompleted,
Rest: defaultRest,
Gap: defaultGap,
},
@@ -524,6 +623,12 @@ func SiteNames() []string {
return names
}
// BrowserBackedSites is derived from the registry: the Sites whose pages are
// read through the browser sidecar. Sorted so callers that range it (the
// browser fetcher's dispatch) see a stable order instead of map-iteration
// noise. Exported so the web layer shares the same set the poller does.
func BrowserBackedSites() []string { return browserBackedSites() }
// browserBackedSites is derived from the registry: the Sites whose pages are
// read through the browser sidecar. Sorted so callers that range it (the
// browser fetcher's dispatch) see a stable order instead of map-iteration
+273
View File
@@ -188,6 +188,104 @@ const lnwSeriesFixture = `
</div>
`
// Completed-marker fixtures, trimmed from the live pages fetched 2026-08-22
// for the verification step below. Selector-removed rows delete the marker
// from the consts and must answer false.
// Trimmed from https://asurascans.com/comics/solo-leveling (301 to
// /comics/solo-leveling-b60d532c) fetched 2026-08-22 by a curl probe from
// this machine. The astro-island props are HTML-escaped in the served
// document, so the quotes arrive as &quot;.
const asuraCompletedFixture = `
&quot;bookmarkCount&quot;:[0,39128],&quot;status&quot;:[0,&quot;completed&quot;],&quot;type&quot;:[0,&quot;manhwa&quot;]
`
// Trimmed from https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af
// fetched 2026-08-22 by a curl probe.
const asuraOngoingFixture = `
&quot;bookmarkCount&quot;:[0,54027],&quot;status&quot;:[0,&quot;ongoing&quot;],&quot;type&quot;:[0,&quot;manhwa&quot;]
`
// Trimmed from https://demonicscans.org/manga/Solo-Leveling fetched
// 2026-08-22 by a curl probe. The info block is sloppy: bare <li>s inside a
// <div>, label and value as a sibling pair.
const demonicCompletedFixture = `
<div class="flex flex-row">
<li style="width:150px;color:#b2b2b2;">Status</li>
<li>Completed</li>
</div>
`
// Trimmed from https://demonicscans.org/manga/Catastrophic-Necromancer
// fetched 2026-08-22 by a curl probe. Same block, ongoing value.
const demonicOngoingFixture = `
<div class="flex flex-row">
<li style="width:150px;color:#b2b2b2;">Status</li>
<li>Ongoing</li>
</div>
`
// Trimmed from https://comix.to/title/q77m-countach fetched 2026-08-22 by a
// cleared Chrome tab (the CDP sidecar: the Series URL is fetched in-tab, the
// same server-rendered payload the poll reads). The served page escapes the
// query map keys as \u0022; the fixture carries the decoded form, which
// parses to the same key. The recommended strip on this very page carries a
// finished entry; only the ["manga","detail","q77m"] entry counts.
const comixCompletedFixture = `<script type="application/json" id="initial-data">{"queries":{"[\"manga\",\"recommended\",\"q77m\",1]":{"items":[{"hid":"pz65","title":"Wangan Midnight: C1 Runner","status":"finished"}]},"[\"manga\",\"detail\",\"q77m\"]":{"id":13211,"hid":"q77m","title":"Countach","status":"finished","originalLanguage":"ja"}}}</script>`
// Trimmed from https://comix.to/title/n8we-dungeons-and-crayons fetched
// 2026-08-22 the same way (keys in their decoded form, as above). The
// recommended strip includes a finished entry; the target detail is
// releasing, and only the detail read counts.
const comixOngoingFixture = `<script type="application/json" id="initial-data">{"queries":{"[\"manga\",\"recommended\",\"n8we\",1]":{"items":[{"hid":"g2rk","title":"On the Way to Meet Mom","status":"finished"}]},"[\"manga\",\"detail\",\"n8we\"]":{"id":1429,"hid":"n8we","title":"Dungeons and Crayons","status":"releasing","originalLanguage":"ko"}}}</script>`
// Trimmed from GET https://kagane.to/api/v2/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b
// fetched 2026-08-22 in a cleared Chrome tab (plain TLS serves the Cloudflare
// challenge).
const kaganeOngoingFixture = `{"series_id":"019f84bc-9ba0-7ed9-86f5-8b905ec7c28b","title":"Infinite Decryption: The Strongest Level 0","publication_status":"Ongoing","upload_status":"Ongoing"}`
// Trimmed from GET https://kagane.to/api/v2/series/019c29c3-5abd-70e6-9efc-1f1f22d839f6
// fetched 2026-08-22 the same way.
const kaganeCompletedFixture = `{"series_id":"019c29c3-5abd-70e6-9efc-1f1f22d839f6","title":"Real Account 1-17","publication_status":"Completed","upload_status":"Completed"}`
// Composed, not a single live trim: upload Completed alongside a
// non-Completed publication status is the research note's inferred inverse
// case and did not turn up in the ~1900-series live scan of 2026-08-22 (both
// field vocabularies are live-verified; the note's live divergence is 'Cause
// Calypso Can, publication Ongoing + upload Hiatus). The predicate must read
// publication_status only.
const kaganeDivergentFixture = `{"series_id":"019f84bc-9ba0-7ed9-86f5-8b905ec7c28b","title":"Infinite Decryption: The Strongest Level 0","publication_status":"Ongoing","upload_status":"Completed"}`
// Trimmed from https://novelfull.com/reverend-insanity.html fetched
// 2026-08-22 in a cleared Chrome tab after the plain-TLS probe was served the
// challenge (time-varying; plain curl answered 403 the same day).
const novelfullCompletedFixture = `<div><h3>Status:</h3><a href="/status/Completed">Completed</a></div>`
// Trimmed from https://novelfull.com/a-cunning-pervert-in-the-cultivation-world.html
// fetched 2026-08-22 the same way.
const novelfullOngoingFixture = `<div><h3>Status:</h3><a href="/status/Ongoing">Ongoing</a></div>`
// Trimmed from the JSON-LD block in the head of
// https://lightnovelworld.net/novel/a-will-eternal/ fetched 2026-08-22. The
// block sits ahead of the wpdiscuz thread, so the predicate reads the whole
// body and needs no comment marker, unlike lnwLatestChapter.
const lnwCompletedFixture = `<script type="application/ld+json">{
"@context": "https://schema.org",
"@type": "Book",
"name": "A Will Eternal",
"creativeWorkStatus": "https://schema.org/CompletedActionStatus"
}</script>`
// Trimmed from
// https://lightnovelworld.net/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/
// fetched 2026-08-22. Same block, ongoing value.
const lnwOngoingFixture = `<script type="application/ld+json">{
"@context": "https://schema.org",
"@type": "Book",
"name": "All Jobs and Classes I Just Wanted One Skill Not Them All",
"creativeWorkStatus": "https://schema.org/ActiveActionStatus"
}</script>`
// Trimmed from https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af
// (redirected to ...-00dcbf97) on 2026-08-10.
const asuraCoverFixture = `<meta property="og:image" content="https://cdn.asurascans.com/asura-images/covers/chronicles-of-the-demon-faction.d4dcb8.webp">`
@@ -473,3 +571,178 @@ func TestLatestChapterFrom(t *testing.T) {
})
}
}
func TestSiteCompletedFrom(t *testing.T) {
const asuraURL = "https://asurascans.com/comics/solo-leveling-b60d532c"
const demonicURL = "https://demonicscans.org/manga/Solo-Leveling"
const comixURL = "https://comix.to/title/q77m-countach"
const kaganeURL = "https://kagane.to/series/019f84bc-9ba0-7ed9-86f5-8b905ec7c28b"
const novelfullURL = "https://novelfull.com/reverend-insanity.html"
const lnwURL = "https://lightnovelworld.net/novel/a-will-eternal/"
tests := []struct {
name string
site string
seriesURL string
body string
want bool
}{
{
name: "asura completed via escaped props status",
site: "asura", seriesURL: asuraURL, body: asuraCompletedFixture,
want: true,
},
{
name: "asura ongoing value is not completed",
site: "asura",
seriesURL: "https://asurascans.com/comics/chronicles-of-the-demon-faction-f886a8af",
body: asuraOngoingFixture,
want: false,
},
{
name: "asura with the status key removed",
site: "asura", seriesURL: asuraURL,
body: strings.ReplaceAll(asuraCompletedFixture, `&quot;status&quot;:[0,&quot;completed&quot;]`, ""),
want: false,
},
{
name: "asura challenge page",
site: "asura", seriesURL: asuraURL, body: challengeFixture,
want: false,
},
{
name: "demonic completed info-block pair",
site: "demonic", seriesURL: demonicURL, body: demonicCompletedFixture,
want: true,
},
{
name: "demonic ongoing value is not completed",
site: "demonic",
seriesURL: "https://demonicscans.org/manga/Catastrophic-Necromancer",
body: demonicOngoingFixture,
want: false,
},
{
name: "demonic with the value li removed",
site: "demonic", seriesURL: demonicURL,
body: strings.ReplaceAll(demonicCompletedFixture, "<li>Completed</li>", ""),
want: false,
},
{
name: "comix recommended finished does not count",
site: "comix",
seriesURL: "https://comix.to/title/n8we-dungeons-and-crayons",
body: comixOngoingFixture,
want: false,
},
{
name: "comix detail finished wins over a finished recommended strip",
site: "comix",
seriesURL: comixURL,
body: comixCompletedFixture,
want: true,
},
{
name: "comix with the detail status removed",
site: "comix", seriesURL: comixURL,
body: strings.ReplaceAll(comixCompletedFixture, `"status":"finished",`, ""),
want: false,
},
{
name: "comix challenge page",
site: "comix", seriesURL: comixURL, body: challengeFixture,
want: false,
},
{
name: "kagane publication Completed",
site: "kagane",
seriesURL: "https://kagane.to/series/019c29c3-5abd-70e6-9efc-1f1f22d839f6",
body: kaganeCompletedFixture,
want: true,
},
{
name: "kagane upload Completed does not count",
site: "kagane", seriesURL: kaganeURL, body: kaganeDivergentFixture,
want: false,
},
{
name: "kagane ongoing values are not completed",
site: "kagane", seriesURL: kaganeURL, body: kaganeOngoingFixture,
want: false,
},
{
name: "kagane with publication_status removed",
site: "kagane", seriesURL: kaganeURL,
body: strings.ReplaceAll(kaganeCompletedFixture, `"publication_status":"Completed",`, ""),
want: false,
},
{
name: "kagane challenge page",
site: "kagane", seriesURL: kaganeURL, body: challengeFixture,
want: false,
},
{
name: "novelfull status link Completed",
site: "novelfull",
seriesURL: novelfullURL,
body: novelfullCompletedFixture,
want: true,
},
{
name: "novelfull ongoing link is not completed",
site: "novelfull",
seriesURL: "https://novelfull.com/a-cunning-pervert-in-the-cultivation-world.html",
body: novelfullOngoingFixture,
want: false,
},
{
name: "novelfull with the status link removed",
site: "novelfull", seriesURL: novelfullURL,
body: strings.ReplaceAll(novelfullCompletedFixture, `<a href="/status/Completed">Completed</a>`, ""),
want: false,
},
{
name: "novelfull challenge page",
site: "novelfull", seriesURL: novelfullURL, body: challengeFixture,
want: false,
},
{
name: "lightnovelworld JSON-LD CompletedActionStatus",
site: "lightnovelworld",
seriesURL: lnwURL,
body: lnwCompletedFixture,
want: true,
},
{
name: "lightnovelworld ActiveActionStatus is not completed",
site: "lightnovelworld",
seriesURL: "https://lightnovelworld.net/novel/all-jobs-and-classes-i-just-wanted-one-skill-not-them-all/",
body: lnwOngoingFixture,
want: false,
},
{
name: "lightnovelworld with creativeWorkStatus removed",
site: "lightnovelworld", seriesURL: lnwURL,
body: strings.ReplaceAll(lnwCompletedFixture, `"creativeWorkStatus": "https://schema.org/CompletedActionStatus"`, ""),
want: false,
},
{
name: "lightnovelworld challenge page",
site: "lightnovelworld", seriesURL: lnwURL, body: challengeFixture,
want: false,
},
{
name: "unknown site",
site: "mangadex", seriesURL: "https://mangadex.org/title/x", body: asuraCompletedFixture,
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := siteCompletedFrom(tt.site, tt.seriesURL, tt.body); got != tt.want {
t.Errorf("siteCompletedFrom(%q, %q, body) = %v, want %v", tt.site, tt.seriesURL, got, tt.want)
}
})
}
}
+12 -8
View File
@@ -4,6 +4,7 @@ import (
"context"
"net/http"
"os"
"strings"
"testing"
"time"
@@ -108,10 +109,7 @@ func TestSmokeAcquireKaganeCover(t *testing.T) {
if ws == "" {
t.Skip("SMOKE_BROWSER_WS_URL unset")
}
const (
seriesID = "019fe11a-8670-7cf3-8343-0b02057d3787"
coverURL = "https://kagane.to/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed"
)
const seriesID = "019fe11a-8670-7cf3-8343-0b02057d3787"
s, _ := newTestStore(t)
bf, err := NewBrowserFetcher(ws)
if err != nil {
@@ -140,12 +138,13 @@ func TestSmokeAcquireKaganeCover(t *testing.T) {
if err != nil || !found {
t.Fatalf("Get: %v found=%v", err, found)
}
if want := testCoverBaseURL + "/covers/" + store.CoverAddress(coverURL); got.Cover != want {
t.Fatalf("Cover = %q, want %q — the acquire path did not store the browser-fetched bytes", got.Cover, want)
addr, ok := strings.CutPrefix(got.Cover, testCoverBaseURL+"/covers/")
if !ok {
t.Fatalf("Cover = %q, want an address on %q — the acquire path did not store the browser-fetched bytes", got.Cover, testCoverBaseURL+"/covers/")
}
body, contentType, ok, err := s.CoverByAddress(store.CoverAddress(coverURL))
body, contentType, ok, err := s.CoverByAddress(addr)
if err != nil || !ok {
t.Fatalf("CoverByAddress: %v found=%v", err, ok)
t.Fatalf("CoverByAddress(%q): %v found=%v", addr, err, ok)
}
if len(body) < 1000 {
t.Fatalf("stored cover is %d bytes, want a real image", len(body))
@@ -153,5 +152,10 @@ func TestSmokeAcquireKaganeCover(t *testing.T) {
if contentType != "image/webp" {
t.Fatalf("content type = %q, want image/webp", contentType)
}
// The address the row carries is the bytes' own SHA-256: a re-art behind
// the same URL would be a different address, which is the whole point.
if want := testCoverBaseURL + "/covers/" + store.CoverAddressForBytes(body); got.Cover != want {
t.Fatalf("Cover = %q, want %q", got.Cover, want)
}
t.Logf("stored %d bytes of %s", len(body), contentType)
}
+1 -1
View File
@@ -40,7 +40,7 @@ func TestSmokeLnwCommentBoundary(t *testing.T) {
if seriesURL == "" {
t.Skip("SMOKE_LNW_SERIES_URL unset")
}
if !fetchableSeriesURL("lightnovelworld", seriesURL) {
if !FetchableSeriesURL("lightnovelworld", seriesURL) {
t.Fatalf("%q is not a fetchable lightnovelworld series URL", seriesURL)
}
+122
View File
@@ -0,0 +1,122 @@
// Package notify posts owner-notice embeds to a Discord webhook. It is
// deliberately small and stdlib-only: one POST of a JSON body needs no
// Discord library, and the path imports nothing of this repo's session or
// OAuth packages — that independence is why a webhook was chosen over a bot
// (issue #171, AC9).
package notify
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
"bookmarkmanager/backend/internal/latest"
)
// dangerColor is the dark design branch's --danger, #cf5c4d = 13589581. The
// integer is unreadable, so a future edit will otherwise "fix" it — do not:
// --ember means new chapter only, and a fault wearing ember would tell the
// owner a stall is a release. This is the one colour a fault wears.
const dangerColor = 13589581
// Client posts owner notices to one Discord webhook. The webhook address is a
// secret in the class of TOKEN_KEY: it is never logged, never rendered, and
// never carried in a returned error, which the poller logs.
type Client struct {
webhookURL string
baseURL string // the deployment's public origin; embed URLs resolve against it
http *http.Client
}
// New returns a Client posting to webhookURL. baseURL is the deployment's
// public origin (Config.PublicBaseURL); the embed's deep-linked title is
// built from it.
func New(webhookURL, baseURL string) *Client {
return &Client{
webhookURL: webhookURL,
baseURL: strings.TrimSuffix(baseURL, "/"),
http: &http.Client{Timeout: 10 * time.Second},
}
}
// Notify posts one owner notice as a Discord embed: the danger colour, the
// subject as a deep-linked title, the sentence as the description, the pass
// time as the timestamp and the machine word in the footer. The send is
// wrapped in a deadline so a hanging Discord cannot hold a poller Lane. On
// failure the error carries no part of the webhook address (the poller logs
// it), and the caller leaves the suppression row unwritten so the next pass
// retries while the condition holds.
func (c *Client) Notify(ctx context.Context, f latest.Fault, sentence, href string) error {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
body, err := json.Marshal(c.payload(f, sentence, href))
if err != nil {
return fmt.Errorf("owner notice: marshal: %w", err)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.webhookURL, strings.NewReader(string(body)))
if err != nil {
return errors.New("owner notice: build request")
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
if err != nil {
// The transport error embeds the webhook address; the address is a
// secret in the class of TOKEN_KEY and the poller logs this error.
return errors.New("owner notice: send failed")
}
defer resp.Body.Close()
// Cap the read: a Discord error page is enough, and draining the body
// lets the connection be reused.
io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return fmt.Errorf("owner notice: webhook status %d", resp.StatusCode)
}
return nil
}
// payload is the webhook body: one embed and nothing else. No fields, no
// thumbnail, no author block — the wire shape is what Discord reads.
func (c *Client) payload(f latest.Fault, sentence, href string) webhookPayload {
return webhookPayload{Embeds: []embed{{
Color: dangerColor,
Title: subject(f),
URL: c.baseURL + href,
Description: sentence,
Timestamp: time.UnixMilli(f.Since).UTC().Format(time.RFC3339),
Footer: embedFooter{Text: f.Condition},
}}}
}
// subject renders the embed's title: the Site the fault is about, with the
// machine word so the title needs no per-condition wording here — #172's
// conditions pass different sentences, not a different builder. For a fault
// that is not one Site's, the machine word stands alone.
func subject(f latest.Fault) string {
if f.Site == "" {
return f.Condition
}
return f.Site + ": " + f.Condition
}
type webhookPayload struct {
Embeds []embed `json:"embeds"`
}
type embed struct {
Color int `json:"color"`
Title string `json:"title"`
URL string `json:"url"`
Description string `json:"description"`
Timestamp string `json:"timestamp"`
Footer embedFooter `json:"footer"`
}
type embedFooter struct {
Text string `json:"text"`
}
+100
View File
@@ -0,0 +1,100 @@
package notify_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/notify"
)
// TestNotifyEmbedShape pins the wire shape Discord actually reads: one embed
// in the danger colour with the subject as a deep-linked title built from the
// base URL, the sentence as the description, the pass time as an RFC3339
// timestamp, the machine word in the footer, and no fields grid (nor
// thumbnail, nor author block) at all. The webhook is a local server, so no
// test can reach Discord.
func TestNotifyEmbedShape(t *testing.T) {
var body map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if ct := r.Header.Get("Content-Type"); ct != "application/json" {
t.Errorf("Content-Type = %q, want application/json", ct)
}
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Errorf("decode request body: %v", err)
}
w.WriteHeader(http.StatusNoContent)
}))
defer srv.Close()
passTime := time.UnixMilli(5_000_000).UTC()
c := notify.New(srv.URL, "https://bookmarks.test/")
err := c.Notify(context.Background(),
latest.Fault{Condition: latest.ConditionStall, Site: "comix", Since: passTime.UnixMilli()},
"sentence", "/admin/lanes")
if err != nil {
t.Fatalf("Notify: %v", err)
}
embeds, ok := body["embeds"].([]any)
if !ok || len(embeds) != 1 {
t.Fatalf("embeds = %#v, want exactly one embed", body["embeds"])
}
embed, ok := embeds[0].(map[string]any)
if !ok {
t.Fatalf("embed = %#v, want an object", embeds[0])
}
if color, ok := embed["color"].(float64); !ok || int(color) != 13589581 {
t.Fatalf("color = %#v, want 13589581 (--danger #cf5c4d)", embed["color"])
}
if url := embed["url"]; url != "https://bookmarks.test/admin/lanes" {
t.Fatalf("url = %#v, want the title deep-linked off the base URL", url)
}
if desc := embed["description"]; desc != "sentence" {
t.Fatalf("description = %#v, want the sentence", desc)
}
footer, ok := embed["footer"].(map[string]any)
if !ok || footer["text"] != latest.ConditionStall {
t.Fatalf("footer = %#v, want the machine word in the footer", embed["footer"])
}
ts, ok := embed["timestamp"].(string)
if !ok {
t.Fatalf("timestamp = %#v, want an RFC3339 string", embed["timestamp"])
}
parsed, err := time.Parse(time.RFC3339, ts)
if err != nil || !parsed.Equal(passTime) {
t.Fatalf("timestamp = %q, want %s (the pass time, RFC3339)", ts, passTime.Format(time.RFC3339))
}
for _, banned := range []string{"fields", "thumbnail", "author"} {
if _, ok := embed[banned]; ok {
t.Fatalf("embed has %q, want it absent (no field grid, no thumbnail, no author block)", banned)
}
}
}
// A non-2xx answer is an error the caller logs, and the error never carries
// the webhook address — a secret in the class of TOKEN_KEY, and the poller
// logs every notify error.
func TestNotifyNon2xxIsErrorWithoutTheAddress(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer srv.Close()
c := notify.New(srv.URL, "https://bookmarks.test")
err := c.Notify(context.Background(),
latest.Fault{Condition: latest.ConditionStall, Site: "comix", Since: 5_000_000},
"sentence", "/admin/lanes")
if err == nil {
t.Fatal("Notify = nil, want an error for a 500")
}
if strings.Contains(err.Error(), srv.URL) {
t.Fatalf("error %q leaks the webhook address", err)
}
}
+5 -3
View File
@@ -40,7 +40,9 @@ func Main(m *testing.M) int {
fmt.Println("pgtest:", err)
return 1
}
defer exec.Command("docker", "rm", "-f", id).Run()
// -v: the postgres image declares a VOLUME, so an explicit rm without it
// leaves the anonymous data volume behind on every test run.
defer exec.Command("docker", "rm", "-f", "-v", id).Run()
adminURL = url
return m.Run()
@@ -85,7 +87,7 @@ func start() (id, url string, err error) {
port, err := exec.Command("docker", "port", id, "5432/tcp").Output()
if err != nil {
exec.Command("docker", "rm", "-f", id).Run()
exec.Command("docker", "rm", "-f", "-v", id).Run()
return "", "", fmt.Errorf("docker port: %w", err)
}
// "0.0.0.0:32768" (and possibly a second, IPv6 line); the port is all we want.
@@ -94,7 +96,7 @@ func start() (id, url string, err error) {
first[strings.LastIndex(first, ":")+1:])
if err := waitReady(url); err != nil {
exec.Command("docker", "rm", "-f", id).Run()
exec.Command("docker", "rm", "-f", "-v", id).Run()
return "", "", err
}
return id, url, nil
+103 -37
View File
@@ -7,31 +7,40 @@ import (
"strings"
)
// Series filter names (issue #140), ordered permanent-then-fixable — the
// repairs nothing will ever undo first, the ones a Poll can make right after.
// A name is the repair a row needs, not the SQL that finds it; the values are
// the wire form the Series list URL carries (#142). "all" is the absent and
// Series filter names (issue #140): the seven repair filters are ordered
// permanent-then-fixable — the repairs nothing will ever undo first, the
// ones a Poll can make right after. SeriesFilterFinished and
// SeriesFilterSiteCompleted are not part of that ordering: a finished
// Series is a deliberate state and a Site-completed one is the Site's own
// marker, not repairs, so the pair rides the tail, informational. A name is
// the repair a row needs, not the SQL that finds it; the values are the
// wire form the Series list URL carries (#142). "all" is the absent and
// unknown case: every Series.
const (
SeriesFilterAll = "all"
SeriesFilterNoURL = "no_series_url"
SeriesFilterNoChapter = "never_read_a_chapter"
SeriesFilterNoReaders = "no_readers"
SeriesFilterNeverChecked = "never_checked"
SeriesFilterStale = "stale"
SeriesFilterNoCover = "no_cover"
SeriesFilterReaderReport = "reader_report"
SeriesFilterAll = "all"
SeriesFilterNoURL = "no_series_url"
SeriesFilterNoChapter = "never_read_a_chapter"
SeriesFilterNoReaders = "no_readers"
SeriesFilterNeverChecked = "never_checked"
SeriesFilterStale = "stale"
SeriesFilterNoCover = "no_cover"
SeriesFilterReaderReport = "reader_report"
SeriesFilterFinished = "finished"
SeriesFilterSiteCompleted = "site_completed"
SeriesFilterFailing = "failing"
SeriesFilterUnverified = "unverified"
)
// SeriesFilter is one named hygiene predicate over the whole library. Site
// SeriesFilter is one named filter predicate over the whole library. Site
// and Kind narrow the row read; Name picks the predicate; Cutoff is the
// staleness boundary the "stale" filter compares against, supplied by the
// caller's clock — the store has no clock; Page is 1-based.
// staleness boundary the age-based filters — "stale" and the failing pair —
// compare against, supplied by the caller's clock — the store has no clock;
// Page is 1-based.
type SeriesFilter struct {
Site string // "" = every Site
Kind string // "" = both library buckets' series
Name string // one of the SeriesFilter* constants; "" = SeriesFilterAll
Cutoff int64 // unix ms; "stale" reads it, the store never does
Cutoff int64 // unix ms; the age-based filters read it, the store never does
Page int // 1-based page of the row read; default 1
}
@@ -41,7 +50,8 @@ type SeriesFilter struct {
// must never leave the store package — so the projection does not select it,
// and only the anonymous boolean in raisedByReaderAnswer crosses it.
const adminSeriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.force_poll_at`
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.force_poll_at,
s.latest_corrected_at, s.finished_at, s.site_completed_at`
// raisedByReaderAnswer answers "did a Reader's report set this number" without
// naming which Reader. Kept apart from adminSeriesColumns so the column list —
@@ -50,6 +60,12 @@ const adminSeriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover_
// Reader behind them.
const raisedByReaderAnswer = `(s.latest_raised_by IS NOT NULL) AS raised_by_reader`
// failureAnswer is the poll-failures row's answer (issue #165), kept apart
// from adminSeriesColumns like raisedByReaderAnswer: outcome and failing_since
// are not Series columns, and the COALESCE keeps the row scannable when the
// LEFT JOIN finds no failure row.
const failureAnswer = `COALESCE(f.outcome, ''), COALESCE(f.failing_since, 0)`
// seriesPageSize is the row read's page length. The tie-break in the query's
// ORDER BY is what makes this a stable page boundary — see SeriesPage.
const seriesPageSize = 50
@@ -71,9 +87,27 @@ type AdminSeries struct {
// ForcePollAt is the owner's "check now" request stamp (issue #146), zero
// meaning never asked. Pending is derived, never stored: a request is
// pending while ForcePollAt is newer than LatestCheckedAt.
ForcePollAt int64
ReaderCount int
RaisedByReader bool // a Reader's report set LatestChapterNum
ForcePollAt int64
// LatestCorrectedAt is the correction stamp (issue #149): non-zero means
// the Latest Chapter is the owner's, zero means never corrected. The
// provenance line (#152) derives from it, so the zero-means-never meaning
// is load-bearing.
LatestCorrectedAt int64
ReaderCount int
RaisedByReader bool // a Reader's report set LatestChapterNum
// FailureOutcome is the outcome word of the Series' standing failure, ""
// when no failure row stands. FailingSince is when the run of failures
// began, zero with no row. Both come from the LEFT JOIN, not the Series
// row (issue #165: the row's existence is the state).
FailureOutcome string
FailingSince int64
// FinishedAt is the owner's finish stamp: unix ms, zero while the Series is
// not finished — the same shape as the Correction stamp, and its own undo.
FinishedAt int64
// SiteCompletedAt is when the last successful Poll read saw the Site's own
// completed value, zero meaning it did not (issue #168). The Site's marker,
// never a Lifecycle decision: the owner's Finish is the only retirement.
SiteCompletedAt int64
}
// SeriesPage is one page of the owner's filtered Series list plus the count
@@ -106,10 +140,28 @@ func (a AdminSeries) Key() string { return a.Site + ":" + a.SeriesID }
// The WHERE set is: no URL (an empty URL only — the host-failing-the-fetch-
// gate case is invisible to SQL, needs the Site registry in Go, and belongs to
// a later repair), never-read-a-chapter and never-checked as disjoint halves
// (non-zero versus zero check stamp), stale, no cover, and Reader-report.
// (non-zero versus zero check stamp), stale, no cover, finished (the
// retirement stamp, read directly), site completed (the Site's marker, read
// directly), Reader-report, and the failing pair — failing (the failure row
// exists, a chapter exists, and failing_since is past the cutoff) and
// unverified (the Reader-attributed subset of failing).
// failing's chapter IS NOT NULL is the exact complement of never-read-a-
// chapter's IS NULL half, so the two are disjoint by construction.
// no_readers is the one HAVING predicate: it is the orphan test, an aggregate
// over the LEFT JOIN, where a bare WHERE has no row to test.
//
// The clock-versus-outcome split decides which predicates exclude finished
// Series (`s.finished_at = 0` in each of the five): the clock-driven ones —
// never-checked, stale, no-chapter, no-cover — keep ticking after the last
// Poll, so they would report a retired row as a problem no Poll is coming to
// fix; site-completed's stamp is the Site's, and it keeps standing after the
// owner retires the row, so a finished Series would be reported as work
// nobody is going to do. The outcome-driven ones — no-URL, no-readers,
// Reader-report, failing, unverified — read stored facts that simply stop
// arriving, so a finished Series needing a genuine repair still shows up
// under them. The failing pair carries no finished guard for exactly that
// contrast: a failure row is a stored outcome, not a ticking clock.
//
// stale is the checked-but-old half of the stamp partition — because the
// verdict line wants "not checked in twelve hours" as one figure, and a never
// checked Series is already counted on its own "waiting"/never-checked
@@ -126,16 +178,26 @@ func adminFilter(f SeriesFilter) (where, having string, args []any, err error) {
case SeriesFilterNoURL:
clauses = append(clauses, `s.series_url = ''`)
case SeriesFilterNoChapter:
clauses = append(clauses, `s.latest_checked_at <> 0 AND s.latest_chapter_num IS NULL`)
clauses = append(clauses, `s.latest_checked_at <> 0 AND s.latest_chapter_num IS NULL AND s.finished_at = 0`)
case SeriesFilterNeverChecked:
clauses = append(clauses, `s.latest_checked_at = 0`)
clauses = append(clauses, `s.latest_checked_at = 0 AND s.finished_at = 0`)
case SeriesFilterStale:
clauses = append(clauses, `s.latest_checked_at > 0 AND s.latest_checked_at < $`+strconv.Itoa(len(args)+1))
clauses = append(clauses, `s.latest_checked_at > 0 AND s.latest_checked_at < $`+strconv.Itoa(len(args)+1)+` AND s.finished_at = 0`)
args = append(args, f.Cutoff)
case SeriesFilterNoCover:
clauses = append(clauses, `s.cover_address = ''`)
clauses = append(clauses, `s.cover_address = '' AND s.finished_at = 0`)
case SeriesFilterReaderReport:
clauses = append(clauses, `s.latest_raised_by IS NOT NULL`)
case SeriesFilterFailing:
clauses = append(clauses, `f.site IS NOT NULL AND s.latest_chapter_num IS NOT NULL AND f.failing_since < $`+strconv.Itoa(len(args)+1))
args = append(args, f.Cutoff)
case SeriesFilterUnverified:
clauses = append(clauses, `f.site IS NOT NULL AND s.latest_chapter_num IS NOT NULL AND f.failing_since < $`+strconv.Itoa(len(args)+1)+` AND s.latest_raised_by IS NOT NULL`)
args = append(args, f.Cutoff)
case SeriesFilterFinished:
clauses = append(clauses, `s.finished_at > 0`)
case SeriesFilterSiteCompleted:
clauses = append(clauses, `s.site_completed_at > 0 AND s.finished_at = 0`)
case SeriesFilterNoReaders:
having = `HAVING COUNT(b.reader_id) = 0`
default:
@@ -150,9 +212,9 @@ func adminFilter(f SeriesFilter) (where, having string, args []any, err error) {
// SeriesPage returns one page of the Series matching the filter, least
// recently checked first. The LEFT JOIN to Bookmarks is what surfaces the
// orphans that hygiene has to find — an inner join would hide them, exactly
// as the Lane's join does. ReaderCount is a plain count of every Bookmark on
// the Series, which knowingly disagrees with the two Lane queries for as long
// as the finished lifecycle bucket exists (#140).
// as the Lane's join does. Every bookmark keeps its Series polled now that
// finished is a Series flag, so this plain ReaderCount agrees with the Lane
// queries (issue #157).
//
// The tie-break is mandatory, not decorative: every unpollable Series shares a
// zero check stamp, so ordering on that column alone gives no stable page
@@ -181,15 +243,18 @@ func (s *Store) SeriesPage(f SeriesFilter) (SeriesPage, error) {
base := len(args)
args = append(args, seriesPageSize, seriesPageSize*(f.Page-1))
rows, err := s.db.Query(`
SELECT `+adminSeriesColumns+`, `+raisedByReaderAnswer+`,
SELECT `+adminSeriesColumns+`, `+raisedByReaderAnswer+`, `+failureAnswer+`,
COUNT(b.reader_id) AS reader_count,
COUNT(*) OVER () AS filtered_total
FROM series s
LEFT JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
LEFT JOIN poll_failures f ON f.site = s.site AND f.series_id = s.series_id
`+where+`
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
s.force_poll_at, s.latest_raised_by
s.force_poll_at, s.latest_corrected_at, s.finished_at, s.site_completed_at,
s.latest_raised_by,
f.outcome, f.failing_since
`+having+`
ORDER BY s.latest_checked_at, s.site, s.series_id
LIMIT $`+strconv.Itoa(base+1)+` OFFSET $`+strconv.Itoa(base+2), args...)
@@ -228,6 +293,7 @@ func (s *Store) SeriesShapes(f SeriesFilter) ([]SiteSeriesShape, error) {
SELECT s.site, s.kind
FROM series s
LEFT JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
LEFT JOIN poll_failures f ON f.site = s.site AND f.series_id = s.series_id
`+where+`
GROUP BY s.site, s.series_id, s.kind
`+having+`
@@ -250,11 +316,11 @@ func (s *Store) SeriesShapes(f SeriesFilter) ([]SiteSeriesShape, error) {
return out, rows.Err()
}
// scanAdminSeries reads one row in adminSeriesColumns + raisedByReaderAnswer
// order, plus the query's reader_count and filtered_total columns, and returns
// the window total alongside the row. latest_chapter_num is NULL until first
// captured — the "never read a chapter" state. The Sighting-raiser column is
// never among the scanned columns.
// scanAdminSeries reads one row in adminSeriesColumns + raisedByReaderAnswer +
// failureAnswer order, plus the query's reader_count and filtered_total
// columns, and returns the window total alongside the row. latest_chapter_num
// is NULL until first captured — the "never read a chapter" state. The
// Sighting-raiser column is never among the scanned columns.
func scanAdminSeries(scan func(...any) error) (AdminSeries, int, error) {
var (
a AdminSeries
@@ -264,8 +330,8 @@ func scanAdminSeries(scan func(...any) error) (AdminSeries, int, error) {
if err := scan(
&a.Site, &a.SeriesID, &a.Title, &a.SeriesURL, &a.CoverAddress,
&a.Kind, &a.LatestChapter, &latestChapterNum, &a.LatestCheckedAt,
&a.ForcePollAt,
&a.RaisedByReader, &a.ReaderCount, &total,
&a.ForcePollAt, &a.LatestCorrectedAt, &a.FinishedAt, &a.SiteCompletedAt,
&a.RaisedByReader, &a.FailureOutcome, &a.FailingSince, &a.ReaderCount, &total,
); err != nil {
return AdminSeries{}, 0, err
}
+351 -11
View File
@@ -12,14 +12,15 @@ import (
// Upsert path could not produce together: an orphan has no bookmark, and a
// Reader-raised Latest Chapter needs a Sighting the store does not create.
type seriesSeed struct {
key string
kind string
url string
cover string // cover_address
checkedAt int64
latestNum *float64
bookmarks int // readers that hold it; 0 = orphan
raisedBy bool // a Reader's report is attributed as the raiser
key string
kind string
url string
cover string // cover_address
checkedAt int64
latestNum *float64
bookmarks int // readers that hold it; 0 = orphan
raisedBy bool // a Reader's report is attributed as the raiser
siteCompletedAt int64 // the Site's own marker (issue #168), 0 = not set
}
// seedAdminSeries inserts one series row and its bookmarks (owner first, then
@@ -39,10 +40,10 @@ func seedAdminSeries(t *testing.T, s *Store, seed seriesSeed) {
}
if _, err := s.db.Exec(`
INSERT INTO series (site, series_id, title, kind, series_url, cover_address,
latest_checked_at, latest_chapter, latest_chapter_num)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
latest_checked_at, latest_chapter, latest_chapter_num, site_completed_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
site, seriesID, "Title of "+seed.key, seed.kind, seed.url, seed.cover,
seed.checkedAt, latestChapter, seed.latestNum); err != nil {
seed.checkedAt, latestChapter, seed.latestNum, seed.siteCompletedAt); err != nil {
t.Fatalf("seed series %q: %v", seed.key, err)
}
for i := range seed.bookmarks {
@@ -128,6 +129,85 @@ func TestAdminSeriesFilters(t *testing.T) {
}
}
// A finished Series is the owner's deliberate state, not a problem a Poll
// will fix: the four clock-driven hygiene predicates exclude it (their
// stamps stop advancing at the last Poll, so without the guard a retired row
// is reported forever), the three outcome-driven ones still include it, and
// the finished filter returns exactly the retired rows.
func TestAdminFinishedSeriesFilters(t *testing.T) {
s := newTestStore(t)
// Each fin-* row is shaped to trip exactly one predicate if its guard
// fails: checked-but-old for stale, a zero stamp for never-checked, a
// stamp with no chapter for no-chapter, an empty cover for no-cover, and
// the unguarded three shaped to trip their own. A healthy, unfinished
// neighbour keeps the exclusion checks honest: a filter that regressed to
// matching nothing would pass a bare "no finished rows" assertion.
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-stale", url: "u", cover: "c", checkedAt: 2000, latestNum: new(4.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-neverchecked", url: "u", cover: "c", checkedAt: 0, latestNum: new(4.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-nochapter", url: "u", cover: "c", checkedAt: 9000, bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-nocover", url: "u", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-nourl", url: "", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-orphan", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 0})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-report", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1, raisedBy: true})
seedAdminSeries(t, s, seriesSeed{key: "asura:healthy", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
finished := []string{
"asura:fin-stale", "asura:fin-neverchecked", "asura:fin-nochapter",
"asura:fin-nocover", "asura:fin-nourl", "asura:fin-orphan", "asura:fin-report",
}
for _, key := range finished {
site, id, _ := strings.Cut(key, ":")
if err := s.SetSeriesFinished(site, id, 1000); err != nil {
t.Fatalf("finish %s: %v", key, err)
}
}
cases := []struct {
name string
f SeriesFilter
want []string
}{
{"stale excludes finished", SeriesFilter{Name: SeriesFilterStale, Cutoff: 5000}, nil},
{"never checked excludes finished", SeriesFilter{Name: SeriesFilterNeverChecked}, nil},
{"no chapter excludes finished", SeriesFilter{Name: SeriesFilterNoChapter}, nil},
{"no cover excludes finished", SeriesFilter{Name: SeriesFilterNoCover}, nil},
{"no url includes finished", SeriesFilter{Name: SeriesFilterNoURL}, []string{"asura:fin-nourl"}},
{"no readers includes finished", SeriesFilter{Name: SeriesFilterNoReaders}, []string{"asura:fin-orphan"}},
{"reader report includes finished", SeriesFilter{Name: SeriesFilterReaderReport}, []string{"asura:fin-report"}},
{"finished returns the retired rows", SeriesFilter{Name: SeriesFilterFinished}, finished},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := pageKeys(t, s, tc.f)
want := map[string]bool{}
for _, k := range tc.want {
want[k] = true
}
if len(got) != len(want) {
t.Fatalf("%+v returned %v, want exactly %v", tc.f, got, want)
}
for k := range want {
if !got[k] {
t.Fatalf("%+v dropped %q (got %v)", tc.f, k, got)
}
}
})
}
// The aggregate's finished total counts every retired row — the same
// predicate the Overview's finished figure is summed from.
shapes, err := s.SeriesShapes(SeriesFilter{Name: SeriesFilterFinished})
if err != nil {
t.Fatalf("SeriesShapes(finished): %v", err)
}
sum := 0
for _, sh := range shapes {
sum += sh.Total
}
if sum != len(finished) {
t.Fatalf("finished aggregate = %d, want %d", sum, len(finished))
}
}
// "Never read a chapter" and "never checked" are disjoint by construction:
// the first requires a non-zero check stamp, the second a zero one. Over a
// mix that should satisfy both, no row may be counted twice.
@@ -394,3 +474,263 @@ func TestAdminSeriesCarriesForcePollAt(t *testing.T) {
t.Fatalf("row = %+v, want ForcePollAt 5000", page.Rows)
}
}
// SetSeriesFinished is the owner's finish stamp write: finishing writes the
// given ms, un-finishing writes zero — the one undo, the same shape as the
// correction stamp. Touching a missing series is not an error: the row may
// have been orphaned, and the caller's read decides what exists.
func TestSetSeriesFinishedStampsAndClears(t *testing.T) {
s := newTestStore(t)
seedAdminSeries(t, s, seriesSeed{key: "asura:x", url: "u", checkedAt: 9000, bookmarks: 1})
if err := s.SetSeriesFinished("asura", "x", 42); err != nil {
t.Fatalf("SetSeriesFinished: %v", err)
}
var got int64
if err := s.db.QueryRow(
`SELECT finished_at FROM series WHERE site = 'asura' AND series_id = 'x'`).Scan(&got); err != nil {
t.Fatalf("read finished_at: %v", err)
}
if got != 42 {
t.Fatalf("finished_at = %d, want 42", got)
}
if err := s.SetSeriesFinished("asura", "x", 0); err != nil {
t.Fatalf("SetSeriesFinished un-finish: %v", err)
}
if err := s.db.QueryRow(
`SELECT finished_at FROM series WHERE site = 'asura' AND series_id = 'x'`).Scan(&got); err != nil {
t.Fatalf("read finished_at after un-finish: %v", err)
}
if got != 0 {
t.Fatalf("finished_at = %d, want 0 (un-finish writes zero)", got)
}
if err := s.SetSeriesFinished("asura", "ghost", 42); err != nil {
t.Fatalf("SetSeriesFinished missing: %v", err)
}
}
// The admin projection carries the finish stamp so the web layer can render
// the finished state without a second read.
func TestAdminSeriesCarriesFinishedAt(t *testing.T) {
s := newTestStore(t)
seedAdminSeries(t, s, seriesSeed{key: "asura:x", url: "u", checkedAt: 1000, bookmarks: 1})
if err := s.SetSeriesFinished("asura", "x", 5000); err != nil {
t.Fatalf("SetSeriesFinished: %v", err)
}
page, err := s.SeriesPage(SeriesFilter{})
if err != nil {
t.Fatalf("SeriesPage: %v", err)
}
if len(page.Rows) != 1 || page.Rows[0].FinishedAt != 5000 {
t.Fatalf("row = %+v, want FinishedAt 5000", page.Rows)
}
}
// The failing pair reads the failure row's age, not the Series row (issue
// #165): failing requires the joined row, a Latest Chapter, and failing_since
// past the cutoff — the same constant stale reads; unverified is the
// Reader-attributed subset of the same test. A failure inside the window is
// in neither — one bad fetch is not a fault to correct — and a Series that
// never captured a chapter is never failing, the exact complement of
// never-read-a-chapter's IS NULL half, so the two filters are disjoint by
// construction. A finished failing Series still appears: this pair reads
// stored outcomes, which simply stop arriving, and carries no finished guard.
func TestAdminFailingAndUnverifiedFilters(t *testing.T) {
s := newTestStore(t)
const cutoff = 5000
seedAdminSeries(t, s, seriesSeed{key: "asura:failing", url: "u", cover: "c", checkedAt: 9000, latestNum: new(10.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:recent", url: "u", cover: "c", checkedAt: 9000, latestNum: new(9.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:nochapter", url: "u", cover: "c", checkedAt: 9000, bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:reader", url: "u", cover: "c", checkedAt: 9000, latestNum: new(8.0), bookmarks: 1, raisedBy: true})
seedAdminSeries(t, s, seriesSeed{key: "asura:polled", url: "u", cover: "c", checkedAt: 9000, latestNum: new(7.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fin-failing", url: "u", cover: "c", checkedAt: 9000, latestNum: new(6.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:healthy", url: "u", cover: "c", checkedAt: 9000, latestNum: new(5.0), bookmarks: 1})
// Failure rows seed the run's start stamp; the cutoff decides the age.
for _, f := range []struct {
key, word string
since int64
}{
{"asura:failing", "not_found", 2000},
{"asura:recent", "not_found", 9000},
{"asura:nochapter", "not_found", 2000},
{"asura:reader", "not_found", 2000},
{"asura:polled", "errors", 2000},
{"asura:fin-failing", "not_found", 2000},
} {
site, id, _ := strings.Cut(f.key, ":")
if err := s.RecordSeriesFailure(site, id, f.word, f.since); err != nil {
t.Fatalf("seed failure %s: %v", f.key, err)
}
}
if err := s.SetSeriesFinished("asura", "fin-failing", 1000); err != nil {
t.Fatalf("finish asura:fin-failing: %v", err)
}
cases := []struct {
name string
f SeriesFilter
want []string
}{
{"failing", SeriesFilter{Name: SeriesFilterFailing, Cutoff: cutoff}, []string{"asura:failing", "asura:reader", "asura:polled", "asura:fin-failing"}},
{"unverified", SeriesFilter{Name: SeriesFilterUnverified, Cutoff: cutoff}, []string{"asura:reader"}},
{"never read a chapter", SeriesFilter{Name: SeriesFilterNoChapter}, []string{"asura:nochapter"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := pageKeys(t, s, tc.f)
want := map[string]bool{}
for _, k := range tc.want {
want[k] = true
}
if len(got) != len(want) {
t.Fatalf("%+v returned %v, want exactly %v", tc.f, got, want)
}
for k := range want {
if !got[k] {
t.Fatalf("%+v dropped %q (got %v)", tc.f, k, got)
}
}
})
}
// Disjointness: the failing pair and never-read-a-chapter are disjoint by
// the chapter column — IS NOT NULL here, IS NULL there — so no row may be
// counted under both.
failing := pageKeys(t, s, SeriesFilter{Name: SeriesFilterFailing, Cutoff: cutoff})
noChapter := pageKeys(t, s, SeriesFilter{Name: SeriesFilterNoChapter})
for k := range failing {
if noChapter[k] {
t.Fatalf("row %q matches both failing and never-read-a-chapter", k)
}
}
if failing["asura:nochapter"] {
t.Fatal("a Series with no chapter ever captured appears in failing")
}
if !noChapter["asura:nochapter"] {
t.Fatal("the no-chapter Series vanished from never-read-a-chapter")
}
// The aggregates count the same rows the lists do: the landing page's
// figures and the select's options come from these two passes.
for _, name := range []string{SeriesFilterFailing, SeriesFilterUnverified} {
shapes, err := s.SeriesShapes(SeriesFilter{Name: name, Cutoff: cutoff})
if err != nil {
t.Fatalf("SeriesShapes(%s): %v", name, err)
}
sum := 0
for _, sh := range shapes {
sum += sh.Total
}
want := len(pageKeys(t, s, SeriesFilter{Name: name, Cutoff: cutoff}))
if sum != want {
t.Fatalf("%s aggregate sum = %d, want %d (aggregate disagrees with row query)", name, sum, want)
}
}
}
// The projection carries the failure facts from the LEFT JOIN, not the
// Series row: a failing Series reads back its outcome word and the stamp its
// run began at; a Series with no failure row reads empty and zero. Nothing
// new is projected from the Series row itself.
func TestAdminFailureProjection(t *testing.T) {
s := newTestStore(t)
seedAdminSeries(t, s, seriesSeed{key: "asura:broken", url: "u", cover: "c", checkedAt: 9000, latestNum: new(9.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:fine", url: "u", cover: "c", checkedAt: 9000, latestNum: new(8.0), bookmarks: 1})
if err := s.RecordSeriesFailure("asura", "broken", "not_found", 2000); err != nil {
t.Fatalf("seed failure row: %v", err)
}
page, err := s.SeriesPage(SeriesFilter{})
if err != nil {
t.Fatalf("SeriesPage: %v", err)
}
byKey := map[string]AdminSeries{}
for _, a := range page.Rows {
byKey[a.Key()] = a
}
if byKey["asura:broken"].FailureOutcome != "not_found" || byKey["asura:broken"].FailingSince != 2000 {
t.Fatalf("broken row = %+v, want FailureOutcome not_found, FailingSince 2000", byKey["asura:broken"])
}
if byKey["asura:fine"].FailureOutcome != "" || byKey["asura:fine"].FailingSince != 0 {
t.Fatalf("fine row = %+v, want empty outcome and zero stamp", byKey["asura:fine"])
}
}
// The site-completed filter (issue #170) lists the Site's own marker as work
// to work through, carrying the same finished guard the clock-driven
// predicates gained: the Site's stamp keeps standing after the owner retires
// the row, so a finished Series would be reported as work nobody is going to
// do. A zero stamp never appears. Un-finishing puts the Series back in the
// Poll query — the finished_at gate is #157's own, asserted through
// DueForLatestCheck rather than re-derived here.
func TestAdminSiteCompletedFilter(t *testing.T) {
s := newTestStore(t)
seedAdminSeries(t, s, seriesSeed{key: "asura:done", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1, siteCompletedAt: 5000})
seedAdminSeries(t, s, seriesSeed{key: "asura:never", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
seedAdminSeries(t, s, seriesSeed{key: "asura:retired", url: "u", cover: "c", checkedAt: 0, latestNum: new(4.0), bookmarks: 1, siteCompletedAt: 5000})
seedAdminSeries(t, s, seriesSeed{key: "asura:healthy", url: "u", cover: "c", checkedAt: 9000, latestNum: new(4.0), bookmarks: 1})
if err := s.SetSeriesFinished("asura", "retired", 1000); err != nil {
t.Fatalf("finish asura:retired: %v", err)
}
got := pageKeys(t, s, SeriesFilter{Name: SeriesFilterSiteCompleted})
if len(got) != 1 || !got["asura:done"] {
t.Fatalf("site-completed returned %v, want only asura:done", got)
}
// The projection carries the stamp so the detail page can age it.
page, err := s.SeriesPage(SeriesFilter{Name: SeriesFilterSiteCompleted})
if err != nil {
t.Fatalf("SeriesPage(site_completed): %v", err)
}
if len(page.Rows) != 1 || page.Rows[0].SiteCompletedAt != 5000 {
t.Fatalf("row = %+v, want SiteCompletedAt 5000", page.Rows)
}
// The aggregate counts the same row: the landing figure and the select's
// option come from this pass, so they cannot disagree with the list.
shapes, err := s.SeriesShapes(SeriesFilter{Name: SeriesFilterSiteCompleted})
if err != nil {
t.Fatalf("SeriesShapes(site_completed): %v", err)
}
sum := 0
for _, sh := range shapes {
sum += sh.Total
}
if sum != 1 {
t.Fatalf("site-completed aggregate = %d, want 1", sum)
}
// Un-finishing puts the Series back in the Poll query: the due read's
// finished_at gate (issue #157) admits it again — asserted through the
// Lane's own read, not re-derived here.
due, err := s.DueForLatestCheck("asura", 1000, noCeiling)
if err != nil {
t.Fatalf("DueForLatestCheck: %v", err)
}
for _, sr := range due {
if sr.Key() == "asura:retired" {
t.Fatalf("a finished Series is still due for a Poll:\n%+v", due)
}
}
if err := s.SetSeriesFinished("asura", "retired", 0); err != nil {
t.Fatalf("un-finish asura:retired: %v", err)
}
due, err = s.DueForLatestCheck("asura", 1000, noCeiling)
if err != nil {
t.Fatalf("DueForLatestCheck after un-finish: %v", err)
}
found := false
for _, sr := range due {
if sr.Key() == "asura:retired" {
found = true
}
}
if !found {
t.Fatalf("un-finished Series is not due for a Poll:\n%+v", due)
}
}
@@ -1,8 +1,15 @@
-- The Cover splits into two facts. `cover` keeps the third-party address the
-- bytes come from, which is what the acquisition path refetches and dedupes
-- on; `cover_address` is the content address of the bytes once they are
-- actually stored, and is what the wire's absolute URL is built from.
-- bytes come from, which is what the refetch path dedupes on; `cover_address`
-- is the content address of the bytes once they are actually stored, and is
-- what the wire's absolute URL is built from.
--
-- Empty `cover_address` therefore means "no Cover yet" rather than "a Cover
-- that 404s", which is the distinction the API and the UI both depend on.
--
-- The content address was originally the hex SHA-256 of the source URL
-- (ADR-0007). Since ADR-0014 it is the hex SHA-256 of the bytes themselves,
-- so a re-art behind the same URL is a new address. Rows written before
-- ADR-0014 keep their URL-derived addresses; they are never rehashed and heal
-- into byte addressing on their first forced replacement. Both derivations
-- share the 64-hex-digit shape, so the serving guard is unchanged.
ALTER TABLE series ADD COLUMN cover_address text NOT NULL DEFAULT '';
@@ -0,0 +1,4 @@
-- latest_corrected_at is the "the current Latest Chapter is the owner's" stamp
-- (#149). Written by the Correction; zeroed by every machine write of the
-- value. Zero means never corrected.
ALTER TABLE series ADD COLUMN latest_corrected_at bigint NOT NULL DEFAULT 0;
@@ -0,0 +1,24 @@
-- finished_at is "the owner marked this Series finished" (#157): epoch ms,
-- zero means not finished, and it doubles as the undo (write zero). The poll
-- gate reads it — a Series is polled only while finished_at = 0 — never a
-- bookmark's status.
ALTER TABLE series ADD COLUMN finished_at bigint NOT NULL DEFAULT 0;
-- Column first, seed second, flip third — the order is load-bearing: a seed
-- that ran after the flip would read the buckets it just destroyed, declare
-- nothing finished, and silently resume polling on Series nobody chose to
-- resume. The seed mirrors the pre-cutover due gate exactly: a Series stays
-- polled while any bookmark is outside the finished bucket, so a Series whose
-- every bookmark sits in it is stamped, one click from being read again
-- afterwards. The stamp is the only memory of the bucket the flip is about to
-- erase.
UPDATE series s SET finished_at = (EXTRACT(EPOCH FROM now()) * 1000)::bigint
WHERE EXISTS (SELECT 1 FROM bookmarks b
WHERE b.site = s.site AND b.series_id = s.series_id)
AND NOT EXISTS (SELECT 1 FROM bookmarks b
WHERE b.site = s.site AND b.series_id = s.series_id
AND b.status <> 'finished');
-- The Lifecycle bucket is gone; a finished bookmark is an archived one. The
-- flip must come after the seed, which still reads the bucket.
UPDATE bookmarks SET status = 'archived' WHERE status = 'finished';
@@ -0,0 +1,4 @@
-- A 4xx other than the 403 refusal is the page being gone, not the Site being
-- unwell; the pass log counts it separately so the Lanes page can say "not
-- found" (#164). DEFAULT 0 keeps pre-existing rows readable.
ALTER TABLE poll_passes ADD COLUMN not_found integer NOT NULL DEFAULT 0;
@@ -0,0 +1,13 @@
-- One row per Series that is failing right now (ADR-0016): the row's
-- existence is the failure state, failing_since ages the run of failures,
-- and a correct read deletes the row. Keyed by the same (site, series_id)
-- composite the rest of the system uses, with the cascade so deleting a
-- Series takes its failure row and orphan removal stays a single statement.
CREATE TABLE poll_failures (
site text NOT NULL,
series_id text NOT NULL,
outcome text NOT NULL,
failing_since bigint NOT NULL,
PRIMARY KEY (site, series_id),
FOREIGN KEY (site, series_id) REFERENCES series (site, series_id) ON DELETE CASCADE
);
@@ -0,0 +1,4 @@
-- When the last successful Poll read saw the Site's own completed value
-- (#168): epoch-ms, zero meaning it did not. DEFAULT 0 keeps pre-existing
-- rows readable.
ALTER TABLE series ADD COLUMN site_completed_at bigint NOT NULL DEFAULT 0;
@@ -0,0 +1,12 @@
-- Owner notices (issue #171): one row per episode, remembered only as "the
-- owner was told". The row's presence is the whole state — the poller checks
-- it before sending, writes it after a successful send, and clears it when
-- the condition no longer holds. site is '' for a fault that is not one
-- Site's; the composite primary key is what makes the row a lock against a
-- second message for the same episode.
CREATE TABLE owner_notices (
condition text NOT NULL,
site text NOT NULL,
notified_at bigint NOT NULL,
PRIMARY KEY (condition, site)
);
+458 -82
View File
@@ -16,6 +16,7 @@ import (
"strconv"
"strings"
"github.com/jackc/pgx/v5/pgconn"
_ "github.com/jackc/pgx/v5/stdlib"
)
@@ -46,13 +47,16 @@ type Bookmark struct {
LatestChapter string `json:"latest_chapter"`
LatestChapterNum *float64 `json:"latest_chapter_num"` // nil until first captured
UpdatedAt int64 `json:"updated_at"` // unix ms; see Upsert
// Status is the lifecycle bucket: reading, archived, or finished.
// Archived series stay polled for new chapters; finished ones do not.
// Status is the lifecycle bucket: reading or archived.
// Archived series stay polled for new chapters.
// Empty on the way in means "no opinion" — see Upsert.
Status string `json:"status"`
// Kind is the library bucket: manga or novel. Empty on the way in means
// "no opinion" — see Upsert.
Kind string `json:"kind"`
// Finished is the owner's retirement of the Series, derived: the flag is a
// Series fact and a client cannot write it — see Upsert.
Finished bool `json:"finished"`
}
// Series is one distinct work, shared by every bookmark that tracks it. It is
@@ -60,10 +64,11 @@ type Bookmark struct {
// exists once no matter how many bookmarks point at it (ADR-0003).
//
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
// existing Series has them ignored, and only the backend's own Poll may change
// them. Kind and the latest-chapter fields are last-write-wins like the
// bookmark's own fields. Never serialized: the wire format is the flat
// Bookmark (ADR-0004).
// existing Series has them ignored, and only the backend's own Poll may
// change them. The one exception is SeriesURL, which the owner's
// SetSeriesURL may repair (issue #151). Kind and the latest-chapter fields
// are last-write-wins like the bookmark's own fields. Never serialized: the
// wire format is the flat Bookmark (ADR-0004).
type Series struct {
Site string
SeriesID string
@@ -79,6 +84,10 @@ type Series struct {
LatestChapter string
LatestChapterNum *float64 // nil until first captured
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
// SiteCompletedAt is when the last successful Poll read saw the Site's
// own completed value, zero meaning it did not (issue #168). A poller
// fact, not reading progress: Upsert never touches it.
SiteCompletedAt int64
// LatestRaisedBy is the Reader whose Sighting last raised LatestChapter,
// and nil when the stored value is a Poll's own finding. It is what lets a
// Poll that contradicts the value downwards name a Reader instead of
@@ -105,7 +114,7 @@ type LanePass struct {
GapMS int64
Clamped bool
Refused, Unreachable, NoChapter int
Unfetchable, Errors int
Unfetchable, Errors, NotFound int
PausedUntil, RefuseUntil int64
}
@@ -114,7 +123,7 @@ type LanePass struct {
type SiteOutcomes struct {
Site string
Refused, Unreachable, NoChapter int
Unfetchable, Errors int
Unfetchable, Errors, NotFound int
}
// LanePause is one persisted Lane pause stamp.
@@ -212,10 +221,11 @@ const (
)
// Lifecycle buckets. A bookmark is in exactly one; favorite is orthogonal.
// Finished is not a bucket: it is a fact about the Series (series.finished_at),
// never about a Reader's bookmark.
const (
StatusReading = "reading"
StatusArchived = "archived"
StatusFinished = "finished"
)
//go:embed migrations/*.sql
@@ -227,16 +237,18 @@ var migrations embed.FS
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover_address,
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind,
s.finished_at > 0`
// seriesColumns is the series row in scanSeries order, used by the poller's
// due query. latest_checked_at lives only on series — see MarkLatestChecked
// for why it stays off every client-visible write.
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.latest_raised_by`
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at, s.latest_raised_by,
s.site_completed_at`
const lanePassColumns = `p.site, p.ran_at, p.skip, p.due, p.checked, p.gap_ms, p.clamped,
p.refused, p.unreachable, p.no_chapter, p.unfetchable, p.errors,
p.refused, p.unreachable, p.no_chapter, p.unfetchable, p.errors, p.not_found,
COALESCE(l.paused_until, 0), COALESCE(l.refuse_until, 0)`
// Owner is the person running the service: the first Reader, seeded at startup
@@ -606,6 +618,7 @@ func (s *Store) scanBookmark(scan func(...any) error) (Bookmark, error) {
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &coverAddress,
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
&b.Finished,
); err != nil {
return Bookmark{}, err
}
@@ -617,9 +630,9 @@ func (s *Store) scanBookmark(scan func(...any) error) (Bookmark, error) {
// bookmark is keyed (reader_id, site, series_id) (issue #22).
b.Key = b.Site + ":" + b.SeriesID
// An unrecognised bucket (a hand-edited row) would leave the row in no list
// at all, so anything outside the three known buckets reads as the default
// at all, so anything outside the two known buckets reads as the default
// rather than being passed through.
if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished {
if b.Status != StatusReading && b.Status != StatusArchived {
b.Status = StatusReading
}
return b, nil
@@ -638,7 +651,7 @@ func scanSeries(scan func(...any) error) (Series, error) {
if err := scan(
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover, &sr.CoverAddress,
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt, &latestRaisedBy,
&sr.Forced, &sr.readerCount,
&sr.SiteCompletedAt, &sr.Forced, &sr.readerCount,
); err != nil {
return Series{}, err
}
@@ -655,7 +668,7 @@ func scanLanePass(scan func(...any) error) (LanePass, error) {
var p LanePass
if err := scan(
&p.Site, &p.RanAt, &p.Skip, &p.Due, &p.Checked, &p.GapMS, &p.Clamped,
&p.Refused, &p.Unreachable, &p.NoChapter, &p.Unfetchable, &p.Errors,
&p.Refused, &p.Unreachable, &p.NoChapter, &p.Unfetchable, &p.Errors, &p.NotFound,
&p.PausedUntil, &p.RefuseUntil,
); err != nil {
return LanePass{}, err
@@ -704,67 +717,112 @@ func (s *Store) getCoverByAddress(address string) ([]byte, string, bool, error)
return body, contentType, true, nil
}
func (s *Store) putCover(sourceURL string, body []byte, contentType string) error {
func (s *Store) putCover(sourceURL string, body []byte, contentType string) (string, error) {
stored, ok := CoverContentType(contentType)
if !ok {
return fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
return "", fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
}
contentType = stored
address := coverSourceAddress(sourceURL)
address := CoverAddressForBytes(body)
relativePath := coverRelativePath(address)
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(relativePath))
if err := os.MkdirAll(filepath.Dir(coverPath), 0o755); err != nil {
return fmt.Errorf("create cover shard: %w", err)
return "", fmt.Errorf("create cover shard: %w", err)
}
tmp, err := os.CreateTemp(filepath.Dir(coverPath), ".cover-*")
if err != nil {
return fmt.Errorf("create cover temp file: %w", err)
return "", fmt.Errorf("create cover temp file: %w", err)
}
tmpName := tmp.Name()
defer os.Remove(tmpName)
if _, err := tmp.Write(body); err != nil {
tmp.Close()
return fmt.Errorf("write cover temp file: %w", err)
return "", fmt.Errorf("write cover temp file: %w", err)
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return fmt.Errorf("sync cover temp file: %w", err)
return "", fmt.Errorf("sync cover temp file: %w", err)
}
if err := tmp.Close(); err != nil {
return fmt.Errorf("close cover temp file: %w", err)
return "", fmt.Errorf("close cover temp file: %w", err)
}
if err := os.Link(tmpName, coverPath); err != nil && !errors.Is(err, fs.ErrExist) {
return fmt.Errorf("install cover file: %w", err)
return "", fmt.Errorf("install cover file: %w", err)
}
if _, err := s.db.Exec(`
INSERT INTO covers (address, path, content_type)
VALUES ($1, $2, $3)
ON CONFLICT (address) DO NOTHING`, address, relativePath, contentType); err != nil {
return fmt.Errorf("record cover %q: %w", address, err)
return "", fmt.Errorf("record cover %q: %w", address, err)
}
return address, nil
}
// ReclaimCover permanently removes a Cover nothing references: the sharded
// file first, the covers row last. A blank address is a no-op, and so is any
// address a Series row still points at — byte-identical artwork is one row by
// construction (ADR-0014), so reclaiming one Series' stranded bytes must not
// blank another's. The file goes first because the covers row is the handle:
// an interrupted run stays findable in SQL — covers rows unreferenced by any
// series cover_address — and re-running finishes the job, whereas deleting
// the row first would leave a file nothing names. A concurrent Forced Poll
// repointing a live Series at this address between the guard and the unlink
// is the repairable case: the missing file reads as ok=false and the next
// pass re-installs it. Failures are returned, never logged here — the caller
// logs and carries on — and a failed unlink leaves the row in place for a
// retry. A whole-table sweep, if ever wanted, is one SQL query over covers,
// not a tree walk and not this function.
func (s *Store) ReclaimCover(address string) error {
if address == "" {
return nil
}
var referenced int
err := s.db.QueryRow(`SELECT 1 FROM series WHERE cover_address = $1 LIMIT 1`, address).Scan(&referenced)
if err == nil {
return nil
}
if !errors.Is(err, sql.ErrNoRows) {
return fmt.Errorf("guard reclaim of cover %q: %w", address, err)
}
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(coverRelativePath(address)))
if err := os.Remove(coverPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("remove cover file %q: %w", address, err)
}
if _, err := s.db.Exec(`DELETE FROM covers WHERE address = $1`, address); err != nil {
return fmt.Errorf("delete cover row %q: %w", address, err)
}
return nil
}
// GetCover returns the immutable object addressed by its source URL. Missing
// GetCover returns the immutable object a source URL's own hash names. Rows
// written before byte addressing (ADR-0014) are the only ones that ever reach
// it; it hashes the URL, so a byte-addressed Cover is invisible to it. Missing
// files are reported with ok=false so callers can retry acquisition later.
func (s *Store) GetCover(sourceURL string) ([]byte, string, bool, error) {
return s.getCover(sourceURL)
}
// PutCover persists bytes under the source URL's content address. A later
// write for the same URL cannot replace the immutable object.
// PutCover persists bytes under their own content address (ADR-0014). A later
// write of the same bytes cannot replace the immutable object.
func (s *Store) PutCover(sourceURL string, body []byte, contentType string) error {
return s.putCover(sourceURL, body, contentType)
_, err := s.putCover(sourceURL, body, contentType)
return err
}
// CoverAddress is the content address bytes fetched from sourceURL are stored
// under. It is a pure function of the URL, so the acquisition path can name a
// Cover before it has the bytes.
func CoverAddress(sourceURL string) string { return coverSourceAddress(sourceURL) }
// CoverAddressForBytes is the content address body is stored under: the hex
// SHA-256 of the bytes, so identical artwork is one address and a re-art a
// new one. Legacy rows were addressed from their source URL instead and are
// never rehashed — both derivations coexist (ADR-0014).
func CoverAddressForBytes(body []byte) string {
sum := sha256.Sum256(body)
return hex.EncodeToString(sum[:])
}
// coverAddressRe is the shape of a stored address: the hex SHA-256 of a source
// URL. Request paths reach CoverByAddress, so the shape is checked before the
// value is ever turned into a filesystem path.
// coverAddressRe is the shape of a stored address: 64 lowercase hex digits —
// the hex SHA-256 of the cover bytes, or of the source URL for legacy rows
// (ADR-0014). Request paths reach CoverByAddress, so the shape is checked
// before the value is ever turned into a filesystem path; byte-derived
// addresses keep the same shape, so the guard is unchanged.
var coverAddressRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
// CoverByAddress returns the immutable object at one content address. An
@@ -788,24 +846,68 @@ func (s *Store) CoverWireURL(address string) string {
return s.coverBaseURL + "/covers/" + address
}
// SetSeriesCover stores the bytes and points the Series at them, but only
// while the Series has no Cover: acquisition at creation and the poll both
// call this, and whichever arrives second must not overwrite the first. The
// bytes themselves are content-addressed and immutable, so storing them twice
// is free.
// SetSeriesCover stores the bytes and points the Series at their address, but
// only while the Series has no Cover: acquisition at creation and the poll
// both call this, and whichever arrives second must not overwrite the first.
// The bytes themselves are content-addressed and immutable, so storing them
// twice is free. See ReplaceSeriesCover for the write that may move a Cover
// once one exists (ADR-0014).
func (s *Store) SetSeriesCover(site, seriesID, sourceURL string, body []byte, contentType string) error {
if err := s.putCover(sourceURL, body, contentType); err != nil {
address, err := s.putCover(sourceURL, body, contentType)
if err != nil {
return err
}
if _, err := s.db.Exec(`
UPDATE series SET cover = $3, cover_address = $4
WHERE site = $1 AND series_id = $2 AND cover_address = ''`,
site, seriesID, sourceURL, coverSourceAddress(sourceURL)); err != nil {
site, seriesID, sourceURL, address); err != nil {
return fmt.Errorf("set cover for %q: %w", site+":"+seriesID, err)
}
return nil
}
// ReplaceSeriesCover stores the bytes and points the Series at their address
// whether or not one already exists, writing the current source URL alongside
// — the Forced Poll's installer and the only write that may move a Cover once
// one exists (ADR-0014). previous is the address the row held before the write
// ("" if it had none) and current the address of the bytes just stored; both
// are read and written in one transaction, so a concurrent replacement reports
// the exact displacement. previous == current means the Site served identical
// artwork, an honest no-op; otherwise previous is stranded — the row no
// longer points at it, and reclaiming its bytes is the caller's separate act
// (the poller's replace path calls ReclaimCover on it). This write itself
// removes nothing.
func (s *Store) ReplaceSeriesCover(site, seriesID, sourceURL string, body []byte, contentType string) (previous, current string, err error) {
current, err = s.putCover(sourceURL, body, contentType)
if err != nil {
return "", "", err
}
tx, err := s.db.Begin()
if err != nil {
return "", "", fmt.Errorf("begin replace cover for %q: %w", site+":"+seriesID, err)
}
defer tx.Rollback()
err = tx.QueryRow(`
SELECT cover_address FROM series
WHERE site = $1 AND series_id = $2 FOR UPDATE`,
site, seriesID).Scan(&previous)
if errors.Is(err, sql.ErrNoRows) {
previous = ""
} else if err != nil {
return "", "", fmt.Errorf("read cover for %q: %w", site+":"+seriesID, err)
}
if _, err := tx.Exec(`
UPDATE series SET cover = $3, cover_address = $4
WHERE site = $1 AND series_id = $2`,
site, seriesID, sourceURL, current); err != nil {
return "", "", fmt.Errorf("replace cover for %q: %w", site+":"+seriesID, err)
}
if err := tx.Commit(); err != nil {
return "", "", fmt.Errorf("commit cover replace for %q: %w", site+":"+seriesID, err)
}
return previous, current, nil
}
// List returns every bookmark of one reader, newest activity first.
// Series-owned fields are joined in, so each Bookmark reads back whole and
// flat (ADR-0004).
@@ -904,6 +1006,11 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
// xmax is zero only on a row this statement inserted, which is how a
// Series nobody had bookmarked before is told apart from one that already
// existed — DO UPDATE returns a row either way.
// latest_corrected_at is the one clause conditional on the value moving
// (#149): after a Correction a Reader's cached row holds the corrected
// number and resends it on the next Progress PUT, so unconditional
// zeroing would erase the fact while the value is still the owner's. The
// stamp survives a same-number PUT and dies the moment the number moves.
var created bool
if err := tx.QueryRow(`
INSERT INTO series (site, series_id, title, series_url, kind,
@@ -914,7 +1021,10 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
ON CONFLICT (site, series_id) DO UPDATE SET
kind=excluded.kind,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num
latest_chapter_num=excluded.latest_chapter_num,
latest_corrected_at = CASE
WHEN series.latest_chapter_num IS DISTINCT FROM excluded.latest_chapter_num
THEN 0 ELSE series.latest_corrected_at END
RETURNING xmax = 0`,
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Kind,
b.LatestChapter, latestNum).Scan(&created); err != nil {
@@ -984,6 +1094,37 @@ func (s *Store) Delete(readerID int64, key string) error {
return nil
}
// pgForeignKeyViolation is the SQLSTATE the driver surfaces when a Bookmark
// row refuses a Series delete (bookmarks_series_fk). pgconn exports no named
// constant for it, so the store names it here.
const pgForeignKeyViolation = "23503"
// ErrSeriesHasBookmarks is RemoveSeries' refusal: a Reader still holds the
// Series, so the owner's removal must not reach past that record. The
// delete is the check — no NOT EXISTS pre-check that can race the insert —
// and the driver's foreign-key violation is translated here so no driver
// type escapes the store (issue #155).
var ErrSeriesHasBookmarks = errors.New("series has bookmarks")
// RemoveSeries deletes one Series row by (site, series_id). It is refused
// while any Bookmark references the row; deleting an absent key is not an
// error, matching Delete. The caller owns the stranded Cover: read the row's
// cover_address before the delete and call ReclaimCover after it — the
// helper's guard cannot pass while the series row still points at the
// address, so the order is the sequence, not a preference.
func (s *Store) RemoveSeries(site, seriesID string) error {
if _, err := s.db.Exec(
`DELETE FROM series WHERE site = $1 AND series_id = $2`,
site, seriesID); err != nil {
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == pgForeignKeyViolation {
return ErrSeriesHasBookmarks
}
return fmt.Errorf("remove series %s:%s: %w", site, seriesID, err)
}
return nil
}
// RecordLanePass appends one pass and prunes every older row in the same
// transaction. retainBefore is supplied by the poller's clock.
func (s *Store) RecordLanePass(p LanePass, retainBefore int64) error {
@@ -995,10 +1136,10 @@ func (s *Store) RecordLanePass(p LanePass, retainBefore int64) error {
if _, err := tx.Exec(`
INSERT INTO poll_passes
(site, ran_at, skip, due, checked, gap_ms, clamped,
refused, unreachable, no_chapter, unfetchable, errors)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)`,
refused, unreachable, no_chapter, unfetchable, errors, not_found)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)`,
p.Site, p.RanAt, p.Skip, p.Due, p.Checked, p.GapMS, p.Clamped,
p.Refused, p.Unreachable, p.NoChapter, p.Unfetchable, p.Errors); err != nil {
p.Refused, p.Unreachable, p.NoChapter, p.Unfetchable, p.Errors, p.NotFound); err != nil {
return fmt.Errorf("insert lane pass %s at %d: %w", p.Site, p.RanAt, err)
}
if _, err := tx.Exec(`DELETE FROM poll_passes WHERE ran_at < $1`, retainBefore); err != nil {
@@ -1035,7 +1176,7 @@ func (s *Store) LatestLanePasses() ([]LanePass, error) {
FROM (
SELECT DISTINCT ON (site)
site, ran_at, skip, due, checked, gap_ms, clamped,
refused, unreachable, no_chapter, unfetchable, errors
refused, unreachable, no_chapter, unfetchable, errors, not_found
FROM poll_passes
ORDER BY site, ran_at DESC
) p
@@ -1062,7 +1203,7 @@ func (s *Store) LatestLanePasses() ([]LanePass, error) {
func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
rows, err := s.db.Query(`
SELECT site, SUM(refused), SUM(unreachable), SUM(no_chapter),
SUM(unfetchable), SUM(errors)
SUM(unfetchable), SUM(errors), SUM(not_found)
FROM poll_passes
WHERE ran_at >= $1
GROUP BY site
@@ -1077,7 +1218,7 @@ func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
var outcomes SiteOutcomes
if err := rows.Scan(
&outcomes.Site, &outcomes.Refused, &outcomes.Unreachable,
&outcomes.NoChapter, &outcomes.Unfetchable, &outcomes.Errors,
&outcomes.NoChapter, &outcomes.Unfetchable, &outcomes.Errors, &outcomes.NotFound,
); err != nil {
return nil, fmt.Errorf("scan lane pass outcomes: %w", err)
}
@@ -1086,6 +1227,101 @@ func (s *Store) LanePassOutcomes(since int64) ([]SiteOutcomes, error) {
return out, rows.Err()
}
// RefusingSince reports, per Site, when that Site's current unbroken run of
// refusing passes began: a pass refuses when the Lane gate returned early
// (skip = 'refusing') or the pass loop counted refusals (refused > 0), so a
// Site parked in refusal backoff keeps its run unbroken. A Site whose
// latest pass did not refuse is absent. The run is bounded by the pass
// log's retention — a run older than the log answers with the oldest row
// present — and nothing after now counts: the cutoff is the caller's clock.
func (s *Store) RefusingSince(now int64) (map[string]int64, error) {
rows, err := s.db.Query(`
SELECT site, MIN(ran_at)
FROM poll_passes p
WHERE ran_at <= $1
AND (refused > 0 OR skip = 'refusing')
AND ran_at > COALESCE((
SELECT MAX(q.ran_at) FROM poll_passes q
WHERE q.site = p.site AND q.ran_at <= $1
AND NOT (q.refused > 0 OR q.skip = 'refusing')
), 0)
GROUP BY site`, now)
if err != nil {
return nil, fmt.Errorf("query refusing since: %w", err)
}
defer rows.Close()
out := map[string]int64{}
for rows.Next() {
var site string
var since int64
if err := rows.Scan(&site, &since); err != nil {
return nil, fmt.Errorf("scan refusing since: %w", err)
}
out[site] = since
}
return out, rows.Err()
}
// SidecarOK reports, per Site in sites, the unix ms of that Site's most
// recent pass that actually reached the sidecar: the pass ran its loop
// (skip = '') and no Series read lost Chrome (unreachable = 0) — a
// challenge answer still reached the sidecar, a lost sidecar did not. A
// Site with no such pass is absent: an asleep Lane never reached it, so it
// is absent too and cannot age into a sidecar-down alarm by itself.
func (s *Store) SidecarOK(sites []string) (map[string]int64, error) {
rows, err := s.db.Query(`
SELECT DISTINCT ON (site) site, ran_at
FROM poll_passes
WHERE site = ANY($1) AND skip = '' AND unreachable = 0
ORDER BY site, ran_at DESC`, sites)
if err != nil {
return nil, fmt.Errorf("query sidecar ok: %w", err)
}
defer rows.Close()
out := map[string]int64{}
for rows.Next() {
var site string
var ranAt int64
if err := rows.Scan(&site, &ranAt); err != nil {
return nil, fmt.Errorf("scan sidecar ok: %w", err)
}
out[site] = ranAt
}
return out, rows.Err()
}
// NoChapterShare reports, per Site, the share of that Site's Series holding
// a no-chapter failure row older than cutoff: old rows over the Site's
// whole Series count, so a four-Series Site and a 200-Series Site are
// judged on the same scale. A Site with no Series has no share and is
// absent.
func (s *Store) NoChapterShare(cutoff int64) (map[string]float64, error) {
rows, err := s.db.Query(`
SELECT s.site,
(COUNT(*) FILTER (WHERE f.outcome = 'no_chapter' AND f.failing_since < $1))::float8
/ (COUNT(*)::float8)
FROM series s
LEFT JOIN poll_failures f ON f.site = s.site AND f.series_id = s.series_id
GROUP BY s.site
ORDER BY s.site`, cutoff)
if err != nil {
return nil, fmt.Errorf("query no-chapter share: %w", err)
}
defer rows.Close()
out := map[string]float64{}
for rows.Next() {
var site string
var share float64
if err := rows.Scan(&site, &share); err != nil {
return nil, fmt.Errorf("scan no-chapter share: %w", err)
}
out[site] = share
}
return out, rows.Err()
}
// SetLaneRefusal persists a Site's refusal backoff stamp without touching its
// pause. until is supplied by the caller's clock.
func (s *Store) SetLaneRefusal(site string, until int64) error {
@@ -1162,6 +1398,72 @@ func (s *Store) LaneGates(site string) (pausedUntil, refuseUntil int64, err erro
return pausedUntil, refuseUntil, nil
}
// RecordSeriesFailure upserts one Series' failure row: the outcome word is
// updated when it changes, failing_since is never overwritten, and an
// unchanged word writes nothing. One statement, so the identical-word no-op
// and the keep-the-stamp rule are the same guarantee: the SET arm fires only
// when the word differs, and failing_since is simply absent from it
// (ADR-0016).
func (s *Store) RecordSeriesFailure(site, seriesID, outcome string, now int64) error {
if _, err := s.db.Exec(`
INSERT INTO poll_failures (site, series_id, outcome, failing_since) VALUES ($1, $2, $3, $4)
ON CONFLICT (site, series_id) DO UPDATE SET outcome = EXCLUDED.outcome
WHERE poll_failures.outcome <> EXCLUDED.outcome`, site, seriesID, outcome, now); err != nil {
return fmt.Errorf("record series failure %s:%s: %w", site, seriesID, err)
}
return nil
}
// ClearSeriesFailure deletes one Series' failure row. A row that is not there
// is not an error.
func (s *Store) ClearSeriesFailure(site, seriesID string) error {
if _, err := s.db.Exec(
`DELETE FROM poll_failures WHERE site = $1 AND series_id = $2`, site, seriesID); err != nil {
return fmt.Errorf("clear series failure %s:%s: %w", site, seriesID, err)
}
return nil
}
// NoticeSent reports whether the owner has already been told about this
// episode (issue #171). The row's presence is the whole state, so a missing
// row reads false without error.
func (s *Store) NoticeSent(condition, site string) (bool, error) {
var at int64
err := s.db.QueryRow(
`SELECT notified_at FROM owner_notices WHERE condition = $1 AND site = $2`,
condition, site).Scan(&at)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("notice sent %s/%s: %w", condition, site, err)
}
return true, nil
}
// MarkNoticeSent records that the owner was told. Called only after a
// successful send. Re-marking the same episode just moves the stamp: the row
// is a lock against a second message, not a log.
func (s *Store) MarkNoticeSent(condition, site string, at int64) error {
if _, err := s.db.Exec(`
INSERT INTO owner_notices (condition, site, notified_at) VALUES ($1, $2, $3)
ON CONFLICT (condition, site) DO UPDATE SET notified_at = EXCLUDED.notified_at`,
condition, site, at); err != nil {
return fmt.Errorf("mark notice sent %s/%s: %w", condition, site, err)
}
return nil
}
// ClearNotice forgets an episode, so the condition's next occurrence sends
// again. A row that is not there is not an error.
func (s *Store) ClearNotice(condition, site string) error {
if _, err := s.db.Exec(
`DELETE FROM owner_notices WHERE condition = $1 AND site = $2`, condition, site); err != nil {
return fmt.Errorf("clear notice %s/%s: %w", condition, site, err)
}
return nil
}
// DueForLatestCheck returns one Site's series whose server-side
// latest-chapter check has aged past cutoffMs, ordered by how many bookmarks
// reference them (descending) then least-recently-checked first. One Site per
@@ -1172,10 +1474,12 @@ func (s *Store) LaneGates(site string) (pausedUntil, refuseUntil int64, err erro
//
// A forced Series (force_poll_at newer than latest_checked_at, issue #146)
// overrides exactly three gates: the rest cutoff, the Sighting-deferral
// clause and the finished-only bucket. It never overrides an empty
// series_url or the Bookmarks join — nothing to fetch, and no consumer for
// the result — so those stay unconditional. Forced rows sort to the front of
// the queue; the reader-count-then-age ordering among the rest is ADR-0003.
// clause and a finished Series. It never overrides an empty series_url or the
// Bookmarks join — nothing to fetch, and no consumer for the result — so
// those stay unconditional, and it never clears the finish: nothing here
// writes finished_at, and pending force clears itself when the pass stamps
// the check timestamp. Forced rows sort to the front of the queue; the
// reader-count-then-age ordering among the rest is ADR-0003.
//
// The reader_count ordering is the point of the split (ADR-0003): a series
// shared by several readers is fetched once per due cycle, and the popular
@@ -1183,14 +1487,15 @@ func (s *Store) LaneGates(site string) (pausedUntil, refuseUntil int64, err erro
// reader count, oldest-first keeps the poll fair when the backlog outgrows
// throughput: the most neglected series is always next, so a large collection
// refreshes uniformly slower rather than leaving a tail that never refreshes
// at all. The userscript sorts its own queue the same way (L453).
// at all. The userscript sorts its own queue the same way.
//
// Series with no series_url are skipped — there is nothing to fetch, which is
// the same filter the userscript applies at L452. Series whose only bookmarks
// are finished are skipped too: nothing more is coming, so fetching them only
// burns requests. Archived bookmarks still count — knowing what a shelved
// series is up to is the whole reason for archiving instead of deleting.
// A series with no bookmarks at all never appears: the join excludes it.
// the same filter the userscript applies before refreshing. A finished Series is
// skipped unless forced: nothing more is coming, so fetching it only burns
// requests (issue #157). Archived bookmarks still count — knowing what a
// shelved series is up to is the whole reason for archiving instead of
// deleting. A series with no bookmarks at all never appears: the join
// excludes it.
//
// ceilingMs is the Sighting deferral ceiling (issue #103): a Series whose last
// real Poll is older than it appears however recently it was sighted. That is
@@ -1199,10 +1504,10 @@ func (s *Store) LaneGates(site string) (pausedUntil, refuseUntil int64, err erro
// decided here, from two facts the query already computes, so a Lane gains no
// query per round: a Sighting younger than cutoffMs holds the Series back, but
// only while COUNT(*) is 1. A Series a second Reader bookmarks is Polled on
// schedule, so a wrong value the whole guild can see is corrected by a check
// that was never postponed; on a solitary Series the only person a wrong value
// reaches is the Reader who reported it. Whether the reporting Reader is
// allowed to defer at all was settled when the Sighting was recorded — see
// the schedule, so a wrong value the guild can see is corrected by a check
// that was never postponed; on a solitary Series the only person a wrong
// value reaches is the Reader who reported it. Whether the reporting Reader
// is allowed to defer at all was settled when the Sighting was recorded — see
// RecordSighting.
func (s *Store) DueForLatestCheck(site string, cutoffMs, ceilingMs int64) ([]Series, error) {
rows, err := s.db.Query(`SELECT `+seriesColumns+`,
@@ -1214,12 +1519,11 @@ func (s *Store) DueForLatestCheck(site string, cutoffMs, ceilingMs int64) ([]Ser
AND s.series_url <> ''
AND (s.latest_checked_at <= $2::bigint
OR s.force_poll_at > s.latest_checked_at)
AND (s.finished_at = 0 OR s.force_poll_at > s.latest_checked_at)
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at,
s.force_poll_at
HAVING (COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
OR s.force_poll_at > s.latest_checked_at)
AND (COUNT(*) > 1
s.site_completed_at, s.force_poll_at, s.finished_at
HAVING (COUNT(*) > 1
OR s.latest_sighted_at <= $2::bigint
OR s.latest_checked_at <= $3::bigint
OR s.force_poll_at > s.latest_checked_at)
@@ -1241,14 +1545,18 @@ func (s *Store) DueForLatestCheck(site string, cutoffMs, ceilingMs int64) ([]Ser
return out, rows.Err()
}
// EligibleSeriesCount returns how many of a Site's Series still have at least
// one bookmark outside the finished bucket. It is the denominator of the
// Lane's pace (issue #100): the effective gap is the smaller of the registry
// gap and one hour divided by this count, so Series that will never be Polled
// do not make the Lane faster than it needs to be, and counting every eligible
// Series rather than only those currently due keeps the pace steady — the
// single worst moment to be fastest is startup, when everything is due at
// once.
// EligibleSeriesCount returns how many of a Site's Series are not finished —
// the flag, never a Reader vote. It is the denominator of the Lane's pace
// (issue #100): the effective gap is the smaller of the registry gap and one
// hour divided by this count, so Series that will never be Polled do not make
// the Lane faster than it needs to be, and counting every eligible Series
// rather than only those currently due keeps the pace steady — the single
// worst moment to be fastest is startup, when everything is due at once.
//
// The deliberate asymmetry with DueForLatestCheck's WHERE: a forced Series
// is due but never admitted here, because a forced pass must not speed up
// every other fetch on the Site — one impassioned press is not a reason to
// hammer the Site (issue #157).
func (s *Store) EligibleSeriesCount(site string) (int, error) {
var n int
err := s.db.QueryRow(`SELECT COUNT(*) FROM (
@@ -1256,8 +1564,8 @@ func (s *Store) EligibleSeriesCount(site string) (int, error) {
FROM series s
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
WHERE s.site = $1
AND s.finished_at = 0
GROUP BY s.site, s.series_id
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
) e`, site).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count eligible series %s: %w", site, err)
@@ -1303,6 +1611,21 @@ func (s *Store) ForceSeriesPoll(site, seriesID string, at int64) error {
return nil
}
// SetSeriesFinished stamps or clears the owner's finish. at is unix ms to
// finish, zero to un-finish. A finished Series drops out of the Lane's reads
// (issue #157), and nothing else writes this column: it is the only writer
// outside migration 0016, so a machine write can never retire a Series
// silently. Touching a missing series is not an error: the row may have been
// orphaned, and the caller's read decides what exists.
func (s *Store) SetSeriesFinished(site, seriesID string, at int64) error {
if _, err := s.db.Exec(
`UPDATE series SET finished_at = $1 WHERE site = $2 AND series_id = $3`,
at, site, seriesID); err != nil {
return fmt.Errorf("set series finished %s:%s: %w", site, seriesID, err)
}
return nil
}
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
// tests outside this package (the poller's own tests assert on rest
// bookkeeping) — see MarkLatestChecked for why the field stays off the
@@ -1320,10 +1643,13 @@ func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
// SetLatestChapter records the newest chapter the poll found on a series page.
// The poller walks Series rather than Bookmarks, so this is a series-level
// write: the row is shared, and updating it once refreshes every bookmark that
// joins to it. Touching a missing series is not an error.
// joins to it. Touching a missing series is not an error. The correction stamp
// is zeroed unconditionally: checkOne only calls this when the number differs,
// so a second copy of the condition would drift (#149).
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
if _, err := s.db.Exec(
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4,
latest_corrected_at = 0
WHERE site = $1 AND series_id = $2`,
site, seriesID, label, num); err != nil {
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
@@ -1331,8 +1657,58 @@ func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) erro
return nil
}
// RecordSighting notes that a Reader's browser reported this Series' Latest
// Chapter, which is the half of a Sighting the client body cannot express
// SetSiteCompletedAt stores when the last successful read saw the Site's
// completed value, zero meaning it did not. Series-level, like the Latest
// Chapter: it is a fact about the work, not about one Reader's bookmark, and
// the bookmark's updated_at is never touched — this is not reading progress
// and must not reorder any Reader's list, the same rule SetLatestChapter's
// comment states. Touching a missing series is not an error: the row may
// have been orphaned, and the caller's read decides what exists.
func (s *Store) SetSiteCompletedAt(site, seriesID string, at int64) error {
if _, err := s.db.Exec(
`UPDATE series SET site_completed_at = $1 WHERE site = $2 AND series_id = $3`,
at, site, seriesID); err != nil {
return fmt.Errorf("set site completed %s:%s: %w", site, seriesID, err)
}
return nil
}
// SetSeriesURL stores the owner's repair for a Series' source address
// (issue #151): the one write that lifts the write-once rule documented on
// Series.SeriesURL. It is a store, not a verification — the caller has
// already passed the poller's fetch gate. The handler 404s on an unknown row
// before calling; the write itself is a plain single-column UPDATE like
// MarkLatestChecked.
func (s *Store) SetSeriesURL(site, seriesID, seriesURL string) error {
if _, err := s.db.Exec(
`UPDATE series SET series_url = $3 WHERE site = $1 AND series_id = $2`,
site, seriesID, seriesURL); err != nil {
return fmt.Errorf("set series url %s:%s: %w", site, seriesID, err)
}
return nil
}
// CorrectLatestChapter makes the Latest Chapter the owner's: one UPDATE
// carrying the number, the derived label and the correction stamp. The label
// shape is the poller's and the userscript's ("Chapter " + the number as
// printed), so chapterLeadIn strips it and the UI renders "Ch N" with no
// special case. latest_checked_at is not touched: a Correction is not a check.
// A raising Reader is cleared without judgement: the number is the owner's
// now, and no Sighting counter moves (spec #135).
func (s *Store) CorrectLatestChapter(site, seriesID string, num float64, at int64) error {
if _, err := s.db.Exec(
`UPDATE series SET
latest_chapter = $3,
latest_chapter_num = $4,
latest_corrected_at = $5,
latest_raised_by = NULL
WHERE site = $1 AND series_id = $2`,
site, seriesID, "Chapter "+strconv.FormatFloat(num, 'f', -1, 64), num, at); err != nil {
return fmt.Errorf("correct latest chapter %s:%s: %w", site, seriesID, err)
}
return nil
}
// (issue #103). It must be called *before* the Upsert that stores the reported
// value: the raise test compares against what is still on the row, and after
// the Upsert there is nothing left to compare with. A Series that does not
File diff suppressed because it is too large Load Diff
+10 -3
View File
@@ -4,14 +4,16 @@ import (
"log"
"net/http"
"strconv"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store"
)
// ownerWindow is the staleness boundary the Series list's "not checked in
// 12h" filter compares against. Declared once; later admin tickets read it.
const ownerWindow = 12 * time.Hour
// 12h" filter compares against. It reads latest.OwnerWindow — the one place
// the class-level twelve hours lives, shared with the owner-notice
// conditions (issue #171).
const ownerWindow = latest.OwnerWindow
// adminView is the shared shell data for an administrative page and the roster
// fragment returned after a Reader action.
@@ -47,6 +49,11 @@ func (h *Handler) adminRoutes() []adminRoute {
{"GET /admin/series", h.adminSeries},
{"GET /admin/series/{key}", h.adminSeriesDetail},
{"POST /admin/series/{key}/poll", h.adminSeriesPoll},
{"POST /admin/series/{key}/finish", h.adminSeriesFinish},
{"POST /admin/series/{key}/unfinish", h.adminSeriesUnfinish},
{"POST /admin/series/{key}/latest", h.adminSeriesCorrectLatest},
{"POST /admin/series/{key}/series-url", h.adminSeriesSetURL},
{"POST /admin/series/{key}/remove", h.adminSeriesRemove},
{"POST /admin/lanes/{site}/pause", h.adminLanePause},
{"POST /admin/lanes/{site}/resume", h.adminLaneResume},
{"GET /ui/admin/lanes", h.uiLanes},
+18 -6
View File
@@ -56,6 +56,10 @@ type laneRow struct {
// offer Resume from the moment the owner presses Pause, with no pass
// having run to record it (issue #147).
Paused bool
// FailingHref is the one navigation the row offers: the Site's name
// links to that Site's failing Series. The chips beside it stay
// unlinked; the withdrawn promise lives on outcomeChips (issue #167).
FailingHref string
}
// chip is one named outcome count over the owner's window.
@@ -223,6 +227,7 @@ func buildLaneRow(p store.LanePass, o store.SiteOutcomes, now time.Time) laneRow
if p.GapMS > 0 {
row.Gap = (time.Duration(p.GapMS) * time.Millisecond).Truncate(time.Second).String()
}
row.FailingHref = seriesListHref(store.SeriesFilterFailing, p.Site, "", 0)
row.Chips = outcomeChips(o)
row.HasChips = len(row.Chips) > 0
row.StatePhrase, row.StateGood, row.Attention = laneState(p, now)
@@ -232,17 +237,24 @@ func buildLaneRow(p store.LanePass, o store.SiteOutcomes, now time.Time) laneRow
// outcomeChips lists a Site's nonzero window sums in the taxonomy's fixed
// order, so the chips never reorder as the window changes. None observed is
// written by the template, not drawn as a confident zero count.
// written by the template, not drawn as a confident zero count. The names are
// spelled through outcomeWord, the one vocabulary the failure surface shares
// with the Series list's fact line (issue #167). The counts stay unlinked
// permanently — a withdrawn promise: two of the six words write no per-Series
// state, and the other four count attempts inside the owner window while the
// failing filter lists Series failing now, so neither set contains the other
// and no chip can be a door to its list.
func outcomeChips(o store.SiteOutcomes) []chip {
fixed := []struct {
name string
count int
}{
{"refused", o.Refused},
{"unreachable", o.Unreachable},
{"no chapter", o.NoChapter},
{"unfetchable", o.Unfetchable},
{"errors", o.Errors},
{outcomeWord("refused"), o.Refused},
{outcomeWord("unreachable"), o.Unreachable},
{outcomeWord("no_chapter"), o.NoChapter},
{outcomeWord("unfetchable"), o.Unfetchable},
{outcomeWord("not_found"), o.NotFound},
{outcomeWord("errors"), o.Errors},
}
var out []chip
for _, f := range fixed {
+53 -23
View File
@@ -2,8 +2,10 @@ package web
import (
"fmt"
"log"
"time"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/store"
)
@@ -23,8 +25,9 @@ type overviewView struct {
// as stale + never_checked: a never-checked Series is already counted on
// its own filter, and the verdict wants the inclusive number.
Unchecked int
// Hygiene is the seven problem filters in the Series list's own render
// order; Library is the library split plus the roster. Every figure is a
// Hygiene is the problem filters in the Series list's own render order
// plus the informational tail — finished, then site completed — riding
// last; Library is the library split plus the roster. Every figure is a
// door into the list that counts it, except a zero.
Hygiene []fig
Library []fig
@@ -59,10 +62,13 @@ type siteRow struct {
}
// overviewView assembles the landing page from the store's read model: one
// SeriesShapes pass per filter summed in Go (the shipped surface offers eight
// SeriesShapes pass per filter summed in Go (the shipped surface offers ten
// grouped passes, not a stats query — #140), the pass log's latest pass per
// Site, and the roster. A failure in any read is a 500 with a logged reason,
// never a page of silent zeroes.
// Site, and the roster. A failure in the SeriesShapes, pass or roster reads
// is a 500 with a logged reason, never a page of silent zeroes. The three
// fault-input reads (RefusingSince, SidecarOK, NoChapterShare) fail open:
// a failing read logs and contributes no fault, so the landing page still
// renders — the same fail-open the poller uses for owner notices.
func (h *Handler) overviewView() (overviewView, error) {
now := time.Now()
cutoff := now.Add(-ownerWindow).UnixMilli()
@@ -91,10 +97,36 @@ func (h *Handler) overviewView() (overviewView, error) {
view := overviewView{Waiting: waiting(passes)}
view.Unchecked = totals[store.SeriesFilterStale] + totals[store.SeriesFilterNeverChecked]
view.Verdict, view.HasCounts = overviewVerdict(passes, now)
var refusingSince map[string]int64
if m, err := h.store.RefusingSince(now.UnixMilli()); err != nil {
log.Printf("admin overview: refusing since: %v", err)
} else {
refusingSince = m
}
var sidecarOK map[string]int64
if m, err := h.store.SidecarOK(latest.BrowserBackedSites()); err != nil {
log.Printf("admin overview: sidecar ok: %v", err)
} else {
sidecarOK = m
}
var noChapterShare map[string]float64
if m, err := h.store.NoChapterShare(cutoff); err != nil {
log.Printf("admin overview: no-chapter share: %v", err)
} else {
noChapterShare = m
}
faults := latest.FaultsFrom(latest.FaultInput{
Passes: passes,
RefusingSince: refusingSince,
SidecarOK: sidecarOK,
NoChapterShare: noChapterShare,
}, now)
view.Verdict, view.HasCounts = overviewVerdict(passes, faults)
// The seven problem filters, in seriesFilterOrder's permanent-then-fixable
// order; the All filter's count belongs to the Library block, not to a
// The hygiene figures, in seriesFilterOrder's tail: the problem filters
// in permanent-then-fixable order, then the informational tail — finished
// and site completed — riding last because seriesFilterOrder appends them
// there. The All filter's count belongs to the Library block, not to a
// "hygiene" figure.
hygiene := make([]fig, 0, len(seriesFilterOrder)-1)
for _, name := range seriesFilterOrder[1:] {
@@ -162,28 +194,26 @@ func door(label string, count int, href string) fig {
return fig{Label: label, Href: href, Count: count}
}
// overviewVerdict decides the landing page's one line from the latest pass
// per Site: no passes at all is "no Lane has reported yet" — never confident
// zeroes; otherwise the count of Lanes whose last pass needs the owner, or
// "all lanes healthy". The count comes from the same laneState judgement the
// Lanes page colours on, so the two pages cannot disagree on what a fault is.
func overviewVerdict(passes []store.LanePass, now time.Time) (phrase string, counts bool) {
// overviewVerdict decides the landing page's one line: no passes at all is
// "no Lane has reported yet" — never confident zeroes; otherwise the count of
// faults from the shared FaultsFrom judgement, so the page and the push
// cannot disagree. Zero faults is "all lanes healthy". A sidecar-down fault
// carries Site == "" and is still one fault. The Lanes page's per-row
// laneState is a different question (is this Lane's last pass healthy) from
// the notice class, and its known false positives live there deliberately, so
// the two now differ.
func overviewVerdict(passes []store.LanePass, faults []latest.Fault) (phrase string, counts bool) {
if len(passes) == 0 {
return "no Lane has reported yet", false
}
attention := 0
for _, p := range passes {
if _, _, attn := laneState(p, now); attn {
attention++
}
}
if attention == 0 {
n := len(faults)
if n == 0 {
return "all lanes healthy", true
}
if attention == 1 {
if n == 1 {
return "1 lane needs a look", true
}
return fmt.Sprintf("%d lanes need a look", attention), true
return fmt.Sprintf("%d lanes need a look", n), true
}
// waiting sums Due over the latest pass per Site: how many Series the Lanes
+423 -17
View File
@@ -1,8 +1,10 @@
package web
import (
"errors"
"fmt"
"log"
"math"
"net/http"
"net/url"
"strconv"
@@ -18,23 +20,30 @@ import (
// the wrong page. The store does not export it (#140).
const seriesPageSize = 50
// seriesFilterLabels names every hygiene filter for the Series list select,
// keyed by the wire constant the URL carries. The render order is
// seriesFilterOrder; the labels are read by later admin tickets too, so the
// map and the constants cannot drift apart.
// seriesFilterLabels names every Series filter for the list select, keyed by
// the wire constant the URL carries. The render order is seriesFilterOrder;
// the labels are read by later admin tickets too, so the map and the
// constants cannot drift apart.
var seriesFilterLabels = map[string]string{
store.SeriesFilterAll: "All series",
store.SeriesFilterNoURL: "No series URL",
store.SeriesFilterNoChapter: "Never read a chapter",
store.SeriesFilterNoReaders: "No Readers",
store.SeriesFilterNeverChecked: "Never checked",
store.SeriesFilterStale: "Not checked in 12h",
store.SeriesFilterNoCover: "No cover",
store.SeriesFilterReaderReport: "Latest from a Reader",
store.SeriesFilterAll: "All series",
store.SeriesFilterNoURL: "No series URL",
store.SeriesFilterNoChapter: "Never read a chapter",
store.SeriesFilterNoReaders: "No Readers",
store.SeriesFilterNeverChecked: "Never checked",
store.SeriesFilterStale: "Not checked in 12h",
store.SeriesFilterNoCover: "No cover",
store.SeriesFilterReaderReport: "Latest from a Reader",
store.SeriesFilterFailing: "Failing over 12h",
store.SeriesFilterUnverified: "Unverified Reader number",
store.SeriesFilterFinished: "Finished",
store.SeriesFilterSiteCompleted: "Site says completed",
}
// seriesFilterOrder is the select's render order: All first, then the
// permanent repairs, then the fixable ones (issue #140).
// permanent repairs, then the fixable ones (issue #140). Finished and the
// site-completed hint ride the tail — deliberate, not repairs — and the
// Overview's stats block renders the same tail, which is what sits the
// finished and site-completed figures last there.
var seriesFilterOrder = []string{
store.SeriesFilterAll,
store.SeriesFilterNoURL,
@@ -44,10 +53,13 @@ var seriesFilterOrder = []string{
store.SeriesFilterStale,
store.SeriesFilterNoCover,
store.SeriesFilterReaderReport,
store.SeriesFilterFailing,
store.SeriesFilterUnverified,
store.SeriesFilterFinished,
store.SeriesFilterSiteCompleted,
}
// seriesListView is the Series list page's data. The template renders strings
// and flags, and every judgement about what a value means is made here.
type seriesListView struct {
Filters []seriesFilterOption
Sites []string
@@ -56,6 +68,9 @@ type seriesListView struct {
FilterLabel string
Rows []seriesRowView
Total int
// OOB marks the out-of-band copy of the heading the removal answer
// carries; on the page itself it is false (issue #155).
OOB bool
// KindBoth / KindManga / KindNovel are the Library segment links, and
// PrevHref / NextHref the pager's, all carrying the active filter, Site
// and Kind so narrowing never drops state.
@@ -100,6 +115,22 @@ type seriesRowView struct {
CanPoll bool
Pending bool
Requested string // "requested 3m ago", rendered only while pending
// CanRemove is the Remove control's visibility: only a Series no Reader
// holds can be removed, so the owner is never offered a button that the
// database will always refuse (issue #155). RemovalRefused marks the one
// raced answer: the row stays and says a fresh Bookmark caught the press.
CanRemove bool
RemovalRefused bool
// Finished is the row's display of the owner's finish stamp: the list row
// shows the state and never offers the control — that lives on the detail
// page, where a press that retires a Series from the Lane is on purpose
// and confirm-gated (issue #158).
Finished bool
// Failure names the standing failure and how long it has stood — "not
// found · 3d ago", "" while no failure row stands. Its own field, never a
// Notes chip: the chips cap at two plus a tail, so the one fact that
// names the failure would be the most likely to be truncated away.
Failure string
}
// adminSeries renders the filterable, bookmarkable Series list: filter, Site,
@@ -172,6 +203,362 @@ func (h *Handler) adminSeriesPoll(w http.ResponseWriter, r *http.Request) {
h.render(w, http.StatusOK, "series-row", seriesRow(a, band, time.Now()))
}
// adminSeriesFinish is the owner's Finish control: it stamps the Series'
// finished_at and answers with the freshly rendered meta fragment, so the
// "finished <age> ago" line describes the state after the press. The Lane's
// next pass reads the stamp and stops polling the Series (issue #157). The
// owner gate is the route's, not this handler's; the body is capped like the
// API path caps its bodies; the key is validated here — a malformed key is a
// 400 and an unknown one a 404.
func (h *Handler) adminSeriesFinish(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
log.Printf("series finish %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
} else if !found {
http.NotFound(w, r)
return
}
if err := h.store.SetSeriesFinished(site, seriesID, time.Now().UnixMilli()); err != nil {
log.Printf("series finish %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Re-read after the stamp: the answer must describe the state after the
// press, so the line reads "finished just now". The control's one caller
// is the detail page, which swaps the meta fragment in place.
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series finish %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
}
// adminSeriesUnfinish is the reversal of the Finish control: it clears the
// stamp (writes zero) and answers with the freshly rendered meta fragment, so
// the Series is back in the Lane's queue from its next pass. Reversal, so it
// fires instantly with no confirm row (issue #158).
func (h *Handler) adminSeriesUnfinish(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
log.Printf("series unfinish %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
} else if !found {
http.NotFound(w, r)
return
}
if err := h.store.SetSeriesFinished(site, seriesID, 0); err != nil {
log.Printf("series unfinish %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Re-read after the write: the answer must describe the state after the
// press, so the fragment no longer carries the finished line.
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series unfinish %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
}
// adminSeriesCorrectLatest is the Latest Chapter correction: the owner types
// one number and the Series' Latest Chapter becomes it, stamped as a
// Correction. The number must be a finite float greater than zero — a
// non-numeric, zero or negative value answers 400 and never reaches the
// store, because a bad value would become every Reader's problem. The press
// answers with the freshly rendered meta fragment, so the figures describe
// the state after the press. The owner gate is the route's, not this
// handler's; the body is capped like the API path caps its bodies.
func (h *Handler) adminSeriesCorrectLatest(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
num, err := strconv.ParseFloat(r.PostFormValue("chapter"), 64)
if err != nil || math.IsNaN(num) || math.IsInf(num, 0) || num <= 0 {
http.Error(w, "chapter must be a finite number greater than zero", http.StatusBadRequest)
return
}
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
log.Printf("series correction %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
} else if !found {
http.NotFound(w, r)
return
}
if err := h.store.CorrectLatestChapter(site, seriesID, num, time.Now().UnixMilli()); err != nil {
log.Printf("series correction %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Re-read after the write: the answer must describe the state after the
// press, so the marker reads "corrected just now".
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series correction %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
}
// adminSeriesSetURL is the series URL repair: the owner types one address
// and the Series' Poll fetches it from then on, verified by the same gate
// the poller uses before it fetches anything — a URL failing
// latest.FetchableSeriesURL answers 400 and never reaches the store. The
// repair is a store, not a verification: it performs no outbound fetch, and
// the owner presses Check now afterwards. This lifts the write-once rule of
// Series.SeriesURL for the owner only — a Reader's PUT is still ignored. The
// owner gate is the route's, not this handler's; the body is capped like the
// API path caps its bodies; the key is validated here — a malformed key is a
// 400 and an unknown one a 404.
func (h *Handler) adminSeriesSetURL(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
seriesURL := r.PostFormValue("series_url")
if !latest.FetchableSeriesURL(site, seriesURL) {
http.Error(w, "series URL must be an https address on this site's host", http.StatusBadRequest)
return
}
if _, found, err := h.adminSeriesByKey(site, seriesID); err != nil {
log.Printf("series url %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
} else if !found {
http.NotFound(w, r)
return
}
if err := h.store.SetSeriesURL(site, seriesID, seriesURL); err != nil {
log.Printf("series url %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Re-read after the write: the answer must describe the state after the
// press, like the correction's answer does.
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series url %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
h.render(w, http.StatusOK, "series-detail-meta", h.seriesDetailView(a))
}
// seriesListHeadView is the list heading's data. The template renders it
// inline at the top of the Series list and out of band in the removal answer
// (OOB true, like the chrome partials' OOB flag): the count and the filter
// label are one fact (issue #155).
type seriesListHeadView struct {
Total int
FilterLabel string
OOB bool
}
// adminSeriesRemove is the orphan removal: one Series, one delete, refused by
// the database while any Bookmark exists (translated by the store, never a
// driver error on the page). The owner gate is the route's, not this
// handler's; the body is capped like the API path caps its bodies; the key is
// validated here — a malformed key is a 400 and an unknown one a 404.
//
// The Cover is read from the row before the delete and reclaimed after it:
// ReclaimCover's guard cannot pass while a series row still points at the
// address, so the order is the sequence, not a preference. A reclamation
// failure is not a removal failure — the row is gone and the covers row
// survives for a retry; the handler logs and answers success, because the
// failure has no user-facing surface.
//
// Two callers, one handler, branched on HX-Target like adminSeriesPoll. The
// detail page's remove answers with a navigation — to the No-Readers list on
// success, back to the detail page when a fresh Bookmark raced the press,
// where the new count is visible. The list row's answers with the removed
// row's fragment and the heading re-rendered with the fresh count out of
// band; HX-Reswap deletes the row through the same button that swaps the
// refusal back in, and the count query runs over the press's own filter
// state, so the heading describes the list the owner is looking at.
func (h *Handler) adminSeriesRemove(w http.ResponseWriter, r *http.Request) {
site, seriesID, ok := strings.Cut(r.PathValue("key"), ":")
if !ok || site == "" || seriesID == "" {
http.Error(w, "bad series key", http.StatusBadRequest)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<16)
if err := r.ParseForm(); err != nil {
http.Error(w, "invalid form", http.StatusBadRequest)
return
}
// The row's Cover address is read before the delete because the delete is
// what makes it reclaimable.
a, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series remove %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
http.NotFound(w, r)
return
}
cover := a.CoverAddress
if err := h.store.RemoveSeries(site, seriesID); err != nil {
if errors.Is(err, store.ErrSeriesHasBookmarks) {
// A Bookmark landed between the owner's read and the press: the
// row stays, answered at its new count with the fact spelled
// out — never a 500, and never a deleted row.
if r.Header.Get("HX-Target") == "detail-meta" {
seriesRemoveNavigation(w, r, "/admin/series/"+site+":"+seriesID)
return
}
fresh, found, err := h.adminSeriesByKey(site, seriesID)
if err != nil {
log.Printf("series remove %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if !found {
// A second press removed it while this one was refused; the
// row has nothing left to say.
http.NotFound(w, r)
return
}
band := 0
if r.PostFormValue("band") == "1" {
band = 1
}
row := seriesRow(fresh, band, time.Now())
row.RemovalRefused = true
h.render(w, http.StatusOK, "series-row", row)
return
}
log.Printf("series remove %s: %v", site+":"+seriesID, err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if err := h.store.ReclaimCover(cover); err != nil {
log.Printf("series remove %s: reclaim cover: %v", site+":"+seriesID, err)
}
if r.Header.Get("HX-Target") == "detail-meta" {
seriesRemoveNavigation(w, r, "/admin/series?filter="+store.SeriesFilterNoReaders)
return
}
// The list answer: the removed row's fragment, plus the heading
// re-rendered with the fresh count. HX-Reswap deletes the row through the
// same button that swaps the refusal back in. The count query failing
// does not undo the removal — log it and answer the row alone.
w.Header().Set("HX-Reswap", "delete")
band := 0
if r.PostFormValue("band") == "1" {
band = 1
}
h.render(w, http.StatusOK, "series-row", seriesRow(a, band, time.Now()))
// The heading is an out-of-band append to a response whose status line has
// already gone out with the row, so it is executed straight onto w —
// h.render would send a second WriteHeader.
if head, err := h.seriesListHeadView(r); err != nil {
log.Printf("series remove %s: %v", site+":"+seriesID, err)
} else if err := h.tmpl.ExecuteTemplate(w, "series-list-head", head); err != nil {
log.Printf("series remove %s: render series-list-head oob: %v", site+":"+seriesID, err)
}
}
// seriesListHeadView is the list heading with the count as it stands after a
// removal: the same filter, Site and Kind the press's row carried (the list
// row's button hx-includes the filterbar), so the figure describes the list
// the owner is looking at — the All filter and an unknown one stay the
// absent case. The count is the store's window total, one query.
func (h *Handler) seriesListHeadView(r *http.Request) (seriesListHeadView, error) {
filter := r.PostFormValue("filter")
if _, ok := seriesFilterLabels[filter]; !ok {
filter = store.SeriesFilterAll
}
site := r.PostFormValue("site")
kind := r.PostFormValue("kind")
if kind != store.KindManga && kind != store.KindNovel {
kind = ""
}
data, err := h.store.SeriesPage(store.SeriesFilter{
Site: site,
Kind: kind,
Name: filter,
Cutoff: time.Now().Add(-ownerWindow).UnixMilli(),
Page: 1,
})
if err != nil {
return seriesListHeadView{}, err
}
return seriesListHeadView{
Total: data.Total,
FilterLabel: seriesFilterLabels[filter],
OOB: true,
}, nil
}
// seriesRemoveNavigation answers a removal from the detail page. htmx gets a
// full navigation (HX-Redirect): a bare 303 would be followed by the request
// and the landing page swapped into the press's target, so the header is the
// redirect htmx can see; plain clients get the 303 the ticket names.
func seriesRemoveNavigation(w http.ResponseWriter, r *http.Request, to string) {
if r.Header.Get("HX-Request") != "" {
w.Header().Set("HX-Redirect", to)
return
}
http.Redirect(w, r, to, http.StatusSeeOther)
}
// seriesListView assembles one Series list view from the request's query
// string. An unknown filter value is the absent All case, never an error: the
// select's options are not the only way this URL can be reached.
@@ -243,9 +630,9 @@ func (h *Handler) seriesListView(r *http.Request) (seriesListView, error) {
return view, nil
}
// seriesFilterOptions renders every hygiene filter with its library-wide
// count, one SeriesShapes pass per filter summed in Go — the shipped surface
// offers eight grouped passes, not a single stats query (#140). The counts
// seriesFilterOptions renders every filter with its library-wide count, one
// SeriesShapes pass per filter summed in Go — the shipped surface offers nine
// grouped passes, not a single stats query (#140). The counts
// are library-wide because the select sits next to the Site narrowing and
// must not shift as the owner narrows the list itself. Cutoff travels with
// the stale filter, or its count would always be zero.
@@ -296,8 +683,10 @@ func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
Readers: a.ReaderCount,
Band: i%2 == 1,
CanPoll: canPoll,
CanRemove: a.ReaderCount == 0,
Pending: pending,
Requested: requested,
Finished: a.FinishedAt > 0,
}
if a.LatestChapterNum != nil {
row.Ch = strconv.FormatFloat(*a.LatestChapterNum, 'f', -1, 64)
@@ -305,6 +694,9 @@ func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
row.Ch = "—"
}
row.Age = checkedAge(now, a.LatestCheckedAt)
if a.FailureOutcome != "" {
row.Failure = outcomeWord(a.FailureOutcome) + " · " + checkedAge(now, a.FailingSince)
}
notes := seriesNotes(a, now)
if n := len(notes); n > 2 {
row.Notes, row.More = notes[:2], n-2
@@ -315,6 +707,20 @@ func seriesRow(a store.AdminSeries, i int, now time.Time) seriesRowView {
return row
}
// outcomeWord spells the wire failure word as the Lanes chips spell it — a
// space, not the underscore: not_found reads "not found", no_chapter "no
// chapter". The remaining words are their own spelling, so an unknown word
// degrades to itself rather than vanishing from the page.
func outcomeWord(wire string) string {
switch wire {
case "not_found":
return "not found"
case "no_chapter":
return "no chapter"
}
return wire
}
// seriesNotes are a row's hygiene chips in the design's order: no URL, no
// cover, orphan, stale, reader sighting.
func seriesNotes(a store.AdminSeries, now time.Time) []string {
+88 -1
View File
@@ -24,14 +24,26 @@ type seriesDetailView struct {
Cover string
Chapter string // Latest Chapter number, or "—" before the first capture
Checked string // how long ago the poller last checked, or "never"
// URL is the stored source address, prefilled into the repair input —
// the one stored string this page renders back into a form (issue #151).
URL string
Readers int
// Corrected is the correction marker's text, "" while no Correction
// stands: "corrected <age> ago" — the copy that says the value is the
// owner's, and it dies with the stamp (a machine write of the number).
Corrected string
// Provenance is the actor class behind the current Chapter: "correction",
// "sighting" or "machine read"; "" while the Series was never read, when
// the line is not rendered. Derived from the same anonymous stamps the
// marks above read — no Reader identity crosses here.
Provenance string
// Marks, one per hygiene fact, rendered only while it holds.
Unpollable bool // no SeriesURL to fetch
NoCover bool
Orphan bool // no Reader holds the Series
SightingRaised bool // a Reader's Sighting set the Latest Chapter
// Poll is the Check now control and the pending marker (issue #146): the
// same derivation and visibility as the list row. CanPoll is false on a
// Series with no page to fetch and on an orphan; Pending is derived —
@@ -40,6 +52,27 @@ type seriesDetailView struct {
CanPoll bool
Pending bool
Requested string
// CanRemove is the Remove control's visibility (issue #155): only a
// Series no Reader holds can be removed, so the owner is never offered a
// button the database will always refuse.
CanRemove bool
// Finished is the owner's finish stamp rendered for the control: while it
// stands, the page offers the instant Un-finish, not the confirm-gated
// Finish (issue #158).
Finished bool
// FinishedSince is the "finished <age> ago" line, "" while no finish
// stands. It rides the meta fragment both presses swap, so the answer
// itself shows how long the Series has been finished.
FinishedSince string
// Unverified is the sentence beside the Latest Chapter correction
// control while a Reader's number stands behind a failure past the
// owner window: the value is unconfirmed and the owner should not trust
// it. It never names the Reader. "" otherwise.
Unverified string
// SiteCompleted is the hint's line, "" while the Site has said nothing or
// the owner has finished the Series: "the site says this work is
// completed (since 3d ago)". A hint, never a control (issue #170).
SiteCompleted string
}
// adminSeriesDetail renders one Series' page, keyed by the composite
@@ -103,12 +136,14 @@ func (h *Handler) seriesDetailView(a store.AdminSeries) seriesDetailView {
Kind: a.Kind,
Title: a.Title,
Cover: h.store.CoverWireURL(a.CoverAddress),
URL: a.SeriesURL,
Readers: a.ReaderCount,
Unpollable: a.SeriesURL == "",
NoCover: a.CoverAddress == "",
Orphan: a.ReaderCount == 0,
SightingRaised: a.RaisedByReader,
CanPoll: canPoll,
CanRemove: a.ReaderCount == 0,
Pending: pending,
Requested: requested,
}
@@ -122,5 +157,57 @@ func (h *Handler) seriesDetailView(a store.AdminSeries) seriesDetailView {
} else {
v.Checked = since(time.Now(), time.UnixMilli(a.LatestCheckedAt))
}
v.Corrected = correctedAge(time.Now(), a.LatestCorrectedAt)
v.Finished = a.FinishedAt != 0
v.FinishedSince = finishedAge(time.Now(), a.FinishedAt)
// Unverified: a Reader's number standing behind a failure that has outlived
// the owner window is a number nobody has re-checked since — say so next to
// the correction control, without naming the Reader. A machine read or a
// failure still inside the window carries no sentence; the failure itself
// has not yet outlived the twelve hours' worth of trust.
if a.RaisedByReader && a.FailureOutcome != "" && a.FailingSince < time.Now().Add(-ownerWindow).UnixMilli() {
v.Unverified = "Unverified Reader number: the page has been failing for over 12h, so this Reader-reported chapter is unconfirmed."
}
if a.SiteCompletedAt != 0 && a.FinishedAt == 0 {
v.SiteCompleted = "the site says this work is completed (since " + checkedAge(time.Now(), a.SiteCompletedAt) + ")"
}
// Provenance: the actor class behind the current number, evaluated in the
// order the classes outrank one another — the owner's stamp, which a
// Correction leaves standing and a machine write clears (issue #149); a
// raising Reader, which a Correction drops; then any check stamp at all.
// An Acquisition reads as a machine read because it stamps
// latest_checked_at exactly as a Poll does, so the two are
// indistinguishable the moment it finishes; telling them apart would need
// the column this project declines to add (spec #135), and the one
// actionable case — acquired once, never read again — is already the
// unchecked filter.
if a.LatestCorrectedAt != 0 {
v.Provenance = "correction"
} else if a.RaisedByReader {
v.Provenance = "sighting"
} else if a.LatestCheckedAt != 0 {
v.Provenance = "machine read"
}
return v
}
// correctedAge is the correction marker's text: "corrected <age> ago" while
// the stamp is set, "" when zero — zero means never corrected, and the marker
// must not read as history once a machine wrote the number.
func correctedAge(now time.Time, at int64) string {
if at == 0 {
return ""
}
return "corrected " + since(now, time.UnixMilli(at))
}
// finishedAge is the finish marker's text: "finished <age> ago" while the
// stamp is set, "" when zero — zero means never finished, and the reversal
// (un-finish) must not read as history after a press (issue #158).
func finishedAge(now time.Time, at int64) string {
if at == 0 {
return ""
}
return "finished " + since(now, time.UnixMilli(at))
}
@@ -0,0 +1,34 @@
package web
import (
"testing"
"bookmarkmanager/backend/internal/store"
)
// seriesDetailView derives the provenance line from the three stamps the
// admin projection already carries: the correction stamp outranks a raising
// Reader, which outranks a check stamp, and a Series with none of the three
// renders no line at all — it was never read, and no actor class is true of
// it. Acquisition stamps latest_checked_at exactly as a Poll does, so an
// acquired value lands in the same "machine read" class (#152).
func TestSeriesDetailViewProvenance(t *testing.T) {
cases := []struct {
name string
a store.AdminSeries
want string
}{
{"correction stamp", store.AdminSeries{LatestCorrectedAt: 1}, "correction"},
{"raising reader only", store.AdminSeries{RaisedByReader: true}, "sighting"},
{"check stamp only", store.AdminSeries{LatestCheckedAt: 1}, "machine read"},
{"correction outranks sighting", store.AdminSeries{LatestCorrectedAt: 1, RaisedByReader: true}, "correction"},
{"none of the three", store.AdminSeries{}, ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := (&Handler{}).seriesDetailView(tc.a).Provenance; got != tc.want {
t.Fatalf("Provenance = %q, want %q", got, tc.want)
}
})
}
}
+44
View File
@@ -653,6 +653,29 @@
margin-left: auto;
}
/* The finish confirm is the one calm row on the admin sheet: a reversible
move wears the recessed ash, never the remove wash above, and its
affirmative takes the sheet's patina accent (issue #158). */
.admin-sheet .confirm-row.calm {
background: var(--ash);
}
.admin-sheet .confirm-row.calm span {
color: var(--paper-dim);
}
.admin-sheet .confirm-row.calm .go {
background: var(--patina);
color: var(--ink);
}
.admin-sheet .tbl.series .row-msg {
grid-column: 1 / -1;
margin-top: 6px;
color: var(--danger-soft);
font: 400 13px/1.4 var(--font-body);
}
.admin-sheet .detail-back {
display: inline-block;
margin: 18px 0 0;
@@ -720,6 +743,27 @@
margin-top: 8px;
}
.admin-sheet .dform input {
min-width: 0;
padding: 8px 10px;
border: 1px solid var(--field-line);
background: var(--ink);
color: var(--paper);
font: 500 14px var(--font-mono);
outline: none;
}
/* Focus follows the chapter form's idiom — paper, not heat: a red border on
a valid number field reads as "invalid". */
.admin-sheet .dform input:focus {
border-color: var(--paper);
}
.admin-sheet .dform .hint {
margin: 0;
color: var(--mute-2);
font: 500 12px/1.4 var(--font-mono);
letter-spacing: .04em;
}
.admin-sheet .pausebar {
display: flex;
align-items: center;
+3 -3
View File
@@ -61,7 +61,7 @@ function setActiveTab(el) {
document.dispatchEvent(new Event("bmgr:refilter"));
}
// The chapter-edit form and the archive/finish/remove confirm rows are the
// The chapter-edit form and the archive/remove confirm rows are the
// per-card disclosure panels; only one makes sense open at a time. The button
// that owns an open panel carries .open, which is how the strip shows which
// cell the panel belongs to.
@@ -97,10 +97,10 @@ function toggleChapterForm(key) {
if (form && !form.hidden) form.querySelector("input").focus();
}
// kind is "archive" | "finish" | "remove" — the panel id and the owning action
// kind is "archive" | "remove" — the panel id and the owning action
// cell share it.
function toggleConfirmRow(key, kind) {
var cls = { archive: ".box", finish: ".finish", remove: ".remove" }[kind];
var cls = { archive: ".box", remove: ".remove" }[kind];
var row = togglePanel(key, "confirm-" + kind + "-" + key, ".actions " + cls);
// Focus the answer rather than trusting aria-live on a container that merely
// unhides: it makes the announcement deterministic, keeps tab order inside
+9 -10
View File
@@ -54,7 +54,7 @@
--ink: #100f0e; /* page */
--ash: #161413; /* recessed panel (chapter form) */
--dim: #0d0c0b; /* archived / finished rows sink */
--dim: #0d0c0b; /* archived rows sink */
--rule: #221f1d; /* hairline between sheets */
--rule-soft: #1a1817; /* measure edges */
--field-line: #2c2926;
@@ -550,6 +550,7 @@ button { cursor: pointer; }
.new-chapter { color: var(--ember); }
.state { display: flex; align-items: center; gap: 4px; color: var(--mute); }
.state svg { width: 10px; height: 10px; }
.state.finished { color: var(--moss); }
.is-dim .meta { color: var(--mute-2); }
.is-dim .site-asura, .is-dim .site-demonic,
.is-dim .site-comix, .is-dim .site-kagane,
@@ -612,13 +613,13 @@ button { cursor: pointer; }
.actions > *:last-child { border-right: none; }
.actions svg { width: 17px; height: 17px; }
.actions > *:hover { color: var(--paper); }
/* Per-action accent on hover and press: gold favourite, slate archive, moss
finished, clay chapter. Remove keeps --danger, play keeps paper/ember. */
/* Per-action accent on hover and press: gold favourite, slate archive, clay
chapter. Remove keeps --danger, play keeps paper/ember. */
.actions .fav:hover, .actions .fav:active, .actions .fav:focus-visible { color: var(--brass); }
.actions .pencil:hover, .actions .pencil:active, .actions .pencil:focus-visible { color: var(--clay); }
.actions .box:hover, .actions .box:active, .actions .box:focus-visible { color: var(--slate); }
.actions .finish:hover, .actions .finish:active, .actions .finish:focus-visible { color: var(--moss); }
.actions .play { color: var(--paper); }
.is-new .actions .play { color: var(--ember); }
.actions .play:hover { background: var(--hover); }
.actions .on { color: var(--brass); }
@@ -632,8 +633,8 @@ button { cursor: pointer; }
.actions .remove.open { background: var(--danger-wash); color: var(--danger); }
.is-dim .actions > * { color: var(--mute-2); }
/* Three clusters by consequence: navigate (play) | organize (favourite,
chapter) | lifecycle (archive/restore, finish, remove). The lifecycle cells
/* Two clusters by consequence: navigate (play) | organize (favourite,
chapter) | lifecycle (archive/restore, remove). The lifecycle cells
sit on a recessed ground so the thumb reads "this one moves the series"
before it reads which icon it landed on. */
.actions > .lifecycle { background: var(--ash); }
@@ -712,7 +713,7 @@ button { cursor: pointer; }
color: var(--danger-ink);
font-weight: 600;
}
/* Archive and finish are reversible, so their confirm asks in grey — only the
/* Archive is reversible, so its confirm asks in grey — only the
irreversible remove gets the danger wash. */
.confirm-row.calm { background: var(--ash); }
.confirm-row.calm span { color: var(--paper-dim); }
@@ -917,9 +918,7 @@ button { cursor: pointer; }
/* The cell border follows the icon on hover, so the accent reads as a state
rather than a stray colour. */
.actions .fav:hover, .actions .pencil:hover,
.actions .box:hover, .actions .finish:hover { border-color: currentColor; }
/* Panels line up with the body text, i.e. past the cover and its gap. */
.actions .box:hover { border-color: currentColor; }
.chapter-form, .confirm-row, .error-inline {
margin-left: calc(var(--cover-w) + var(--row-gap));
}
-4
View File
@@ -71,10 +71,6 @@
{{if eq .Tab "archived"}}aria-current="page"{{end}}
hx-get="{{.ListURL "archived"}}" hx-target="#list" hx-swap="innerHTML"
hx-push-url="{{.PageURL "archived"}}" hx-on::after-request="setActiveTab(this)">Archived</a>
<a href="{{.PageURL "finished"}}" class="{{if eq .Tab "finished"}}active{{end}}"
{{if eq .Tab "finished"}}aria-current="page"{{end}}
hx-get="{{.ListURL "finished"}}" hx-target="#list" hx-swap="innerHTML"
hx-push-url="{{.PageURL "finished"}}" hx-on::after-request="setActiveTab(this)">Finished</a>
</nav>
</div>
+5 -23
View File
@@ -1,6 +1,6 @@
{{define "card"}}
{{/* One sheet per series. is-new turns the title crimson over an ember rule;
is-dim sinks archived and finished rows into italic grey. */}}
is-dim sinks archived rows into italic grey. */}}
<article class="card{{if eq .Status "reading"}}{{if .HasNewChapter}} is-new{{end}}{{else}} is-dim{{end}}"
id="card-{{.Key}}" data-title="{{.Title}}">
<div class="row">
@@ -32,9 +32,10 @@
{{if eq .Status "archived"}}
<span class="sep">/</span>
<span class="state">archived</span>
{{else if eq .Status "finished"}}
{{end}}
{{if .Finished}}
<span class="sep">/</span>
<span class="state"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-check"/></svg>finished</span>
<span class="state finished">finished</span>
{{end}}
</p>
</div>
@@ -56,7 +57,7 @@
<svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-pencil"/></svg>
</button>
{{/* Restore is a reversal, so it fires straight away; every move *out* of
the list (archive, finish, remove) goes through a confirm row. */}}
the list (archive, remove) goes through a confirm row. */}}
{{if eq .Status "reading"}}
<button class="lifecycle box" title="Archive" aria-label="Archive"
aria-expanded="false" aria-controls="confirm-archive-{{.Key}}"
@@ -71,13 +72,6 @@
<svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-undo"/></svg>
</button>
{{end}}
{{if ne .Status "finished"}}
<button class="lifecycle finish" title="Mark finished" aria-label="Mark finished"
aria-expanded="false" aria-controls="confirm-finish-{{.Key}}"
onclick="toggleConfirmRow('{{.Key}}', 'finish')">
<svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-check"/></svg>
</button>
{{end}}
<button class="lifecycle remove" title="Remove" aria-label="Remove"
aria-expanded="false" aria-controls="confirm-remove-{{.Key}}"
onclick="toggleConfirmRow('{{.Key}}', 'remove')">
@@ -116,18 +110,6 @@
</div>
</div>
{{end}}
{{if ne .Status "finished"}}
<div class="confirm-row calm" id="confirm-finish-{{.Key}}" role="group" aria-live="polite" hidden>
<span>Mark finished?</span>
<div>
<button class="go"
hx-post="/ui/bookmarks/{{.Key}}/status" hx-vals='{"status":"finished"}'
hx-target="[id='card-{{.Key}}']" hx-swap="outerHTML"
hx-indicator="[id='card-{{.Key}}']" hx-disabled-elt="this">Finish</button>
<button type="button" onclick="toggleConfirmRow('{{.Key}}', 'finish')">Cancel</button>
</div>
</div>
{{end}}
<div class="confirm-row" id="confirm-remove-{{.Key}}" role="group" aria-live="polite" hidden>
<span>Remove “{{.Title}}”? Chapter progress is lost.</span>
<div>
+2 -6
View File
@@ -29,21 +29,17 @@
{{/* The action key. The icon strip on a card is unlabelled, so one permanent
line under the tabs names every glyph. It follows the tab rather than the
row: the archived and finished buckets swap Archive for Restore, and a
finished series has no Done to offer. */}}
row: the archived bucket swaps Archive for Restore. */}}
{{define "keyrow"}}
<div class="keyrow" id="keyrow" aria-label="Action key"{{if .OOB}} hx-swap-oob="true"{{end}}>
<span class="pair"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-play"/></svg><span>Read</span></span>
<span class="pair brass"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-star"/></svg><span>Fav</span></span>
<span class="pair"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-pencil"/></svg><span>Chapter</span></span>
{{if or (eq .Tab "archived") (eq .Tab "finished")}}
{{if eq .Tab "archived"}}
<span class="pair"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-undo"/></svg><span>Restore</span></span>
{{else}}
<span class="pair"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-box"/></svg><span>Archive</span></span>
{{end}}
{{if ne .Tab "finished"}}
<span class="pair"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-check"/></svg><span>Done</span></span>
{{end}}
<span class="pair trash"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="#i-trash"/></svg><span>Delete</span></span>
</div>
{{end}}
@@ -7,7 +7,6 @@
<symbol id="i-star-on" viewBox="0 0 24 24"><path d="M12 3.6l2.6 5.6 6 .8-4.4 4.2 1.1 6-5.3-2.9-5.3 2.9 1.1-6-4.4-4.2 6-.8z" fill="currentColor" stroke="currentColor" stroke-width="1.7" stroke-linejoin="round"/></symbol>
<symbol id="i-pencil" viewBox="0 0 24 24"><path d="M4 20h4L19 9l-4-4L4 16z" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linejoin="round"/></symbol>
<symbol id="i-box" viewBox="0 0 24 24"><path d="M3 7h18v4H3zM5 11v9h14v-9M10 15h4" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linejoin="round"/></symbol>
<symbol id="i-check" viewBox="0 0 24 24"><path d="M4 12.5l5.2 5.5L20 6.5" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/></symbol>
<symbol id="i-trash" viewBox="0 0 24 24"><path d="M4 7h16M9.5 7V4h5v3M6.5 7l1 13h9l1-13M10.5 11v6M13.5 11v6" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"/></symbol>
<symbol id="i-undo" viewBox="0 0 24 24"><path d="M4 9h9.5a5 5 0 010 10H8M4 9l4.2-4.2M4 9l4.2 4.2" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"/></symbol>
<symbol id="i-search" viewBox="0 0 24 24"><circle cx="10.5" cy="10.5" r="6.5" fill="none" stroke="currentColor" stroke-width="1.8"/><path d="M15.3 15.3L20 20" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round"/></symbol>
+1 -1
View File
@@ -26,7 +26,7 @@
</div>
{{range .Rows}}
<div class="trow{{if .Attention}} attention{{end}}">
<span class="c-site">{{.Site}}</span>
<a class="c-site" href="{{.FailingHref}}">{{.Site}}</a>
<span>{{.Due}}</span>
<span>{{.Checked}}</span>
<span>{{.Gap}}</span>
-2
View File
@@ -14,8 +14,6 @@
<div class="empty"><strong>Nothing new.</strong><p>Every series is caught up to its latest chapter.</p></div>
{{else if eq .Tab "archived"}}
<div class="empty"><strong>Nothing archived.</strong><p>Shelve a series to park it here — it keeps getting checked for new chapters.</p></div>
{{else if eq .Tab "finished"}}
<div class="empty"><strong>Nothing finished yet.</strong><p>Mark a series finished and it moves out of your reading list.</p></div>
{{else if .EmptyLibrary}}
{{/* Nothing in either library, so the links are the only thing this page can
usefully say. Both scripts: the two libraries are separate installs. */}}
@@ -1,8 +1,9 @@
{{/* Per-Series page: one address per Series, keyed "<site>:<series_id>" so the
list row is one hop from it. Everything here is a Series-level fact plus
the anonymous Reader count. The Check now control lands in its own .dform
below the (empty) .detail-grid; the pending marker rides the meta line
with the other marks. */}}
the anonymous Reader count. Check now lands in its own .dform below the
.detail-grid; the correction form is the grid's first column and the URL
repair the second (issue #151). The pending and corrected markers ride the
meta line with the other marks. */}}
{{define "series-detail"}}
<a class="ghost detail-back" href="/admin/series">← Series</a>
<h1 class="detail-title">{{.Title}}</h1>
@@ -10,26 +11,71 @@
{{if .Cover}}<div class="cover"><img src="{{.Cover}}" alt="" loading="lazy"></div>
{{else}}<div class="cover"></div>{{end}}
{{template "series-detail-meta" .}}
<div class="detail-grid"></div>
<div class="detail-grid">
<form class="dform" hx-post="/admin/series/{{.Key}}/latest" hx-target="#detail-meta" hx-swap="outerHTML">
<h3>Correct latest chapter</h3>
<p class="hint">The next successful Poll overwrites this value.</p>
<div class="field">
<input type="number" name="chapter" step="any" placeholder="{{.Chapter}}" required>
<button type="submit" class="ghost">Set</button>
</div>
{{if .Unverified}}<p class="hint">{{.Unverified}}</p>{{end}}
</form>
<form class="dform" hx-post="/admin/series/{{.Key}}/series-url" hx-target="#detail-meta" hx-swap="outerHTML">
<h3>Repair series URL</h3>
<p class="hint">The Poll fetches this address — storing is not verifying it. A Site-wide host change is a SQL migration, not two hundred forms.</p>
<div class="field">
<input type="url" name="series_url" value="{{.URL}}" required>
<button type="submit" class="ghost">Set</button>
</div>
</form>
</div>
{{if .CanPoll}}
<div class="dform">
<div class="field"><a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="#detail-meta" hx-swap="outerHTML" href="#">Check now</a></div>
</div>
{{end}}
{{if .CanRemove}}
<div class="dform">
<div class="field"><button class="ghost danger" hx-post="/admin/series/{{.Key}}/remove" hx-target="#detail-meta" hx-confirm="Removes this series and its stored cover. No Reader has it bookmarked; one re-bookmarking it recreates the row.">Remove</button></div>
</div>
{{end}}
{{if .Finished}}
<div class="dform">
<div class="field"><button type="button" class="ghost" hx-post="/admin/series/{{.Key}}/unfinish" hx-target="#detail-meta" hx-swap="outerHTML">Un-finish</button></div>
</div>
{{else}}
<div class="dform">
<div class="field"><button type="button" class="ghost" onclick="document.getElementById('confirm-finish').hidden = false">Finish</button></div>
{{if .SiteCompleted}}<p class="hint">{{.SiteCompleted}}</p>{{end}}
<div class="confirm-row calm" id="confirm-finish" role="group" aria-live="polite" hidden>
<span>Mark this Series finished?</span>
<div>
<button type="button" class="go" hx-post="/admin/series/{{.Key}}/finish" hx-target="#detail-meta" hx-swap="outerHTML">Finish</button>
<button type="button" onclick="document.getElementById('confirm-finish').hidden = true">Cancel</button>
</div>
</div>
</div>
{{end}}
{{end}}
{{/* series-detail-meta is the meta line, and the answer a Check now press on
the detail page swaps into its place: the same marks, re-rendered after
the stamp so the pending marker shows. */}}
{{/* series-detail-meta is the meta line, and the answer a Check now or
correction press on the detail page swaps into its place: the same marks,
re-rendered after the stamp so the pending and corrected markers — and
the provenance line beside the number — describe the value they sit
next to. */}}
{{define "series-detail-meta"}}
<div class="detail-meta" id="detail-meta">
<span>ch {{.Chapter}}</span>
{{if .Provenance}}<span>{{.Provenance}}</span>{{end}}
<span>checked {{.Checked}}</span>
<span>{{.Readers}} readers</span>
{{if .Corrected}}<span class="mark">{{.Corrected}}</span>{{end}}
{{if .Pending}}<span class="mark">{{.Requested}}</span>{{end}}
{{if .FinishedSince}}<span class="mark">{{.FinishedSince}}</span>{{end}}
{{if .Unpollable}}<span class="mark">unpollable</span>{{end}}
{{if .NoCover}}<span class="mark">no cover</span>{{end}}
{{if .Orphan}}<span class="mark">orphan</span>{{end}}
{{if .SightingRaised}}<span class="mark">sighting-raised</span>{{end}}
</div>
{{end}}
{{end}}
@@ -4,7 +4,7 @@
that can be bookmarked: the two selects submit the GET form, and the
Library segment links and the pager preserve the filter and Site. */}}
{{define "series-list"}}
<form class="filterbar" method="get" action="/admin/series">
<form class="filterbar" id="filterbar" method="get" action="/admin/series">
<input type="hidden" name="kind" value="{{.Kind}}">
<label class="fsel"><span>Show</span><select name="filter" onchange="this.form.submit()">
{{range .Filters}}<option value="{{.Name}}"{{if .Selected}} selected{{end}}>{{.Label}} ({{.Count}})</option>{{end}}
@@ -19,7 +19,7 @@
<a href="{{.KindNovel}}"{{if eq .Kind "novel"}} class="active"{{end}}>novels</a>
</span>
</form>
<div class="listhead">{{.Total}} series <span class="lbl">· <em>{{.FilterLabel}}</em></span></div>
{{template "series-list-head" .}}
{{if .Rows}}
<div class="tbl series">
<div class="thead"><span>Site</span><span class="c-ch">Ch</span><span>Checked</span><span class="c-rd">Readers</span><span>Notes</span><span></span></div>
@@ -47,7 +47,17 @@
<span class="c-ch">{{.Ch}}</span>
<span>{{.Age}}</span>
<span class="c-rd">{{.Readers}}</span>
<span class="c-note">{{range .Notes}}<span class="mark">{{.}}</span>{{end}}{{if .More}}<span class="mark mark-faint">+{{.More}}</span>{{end}}</span>
<span class="c-act">{{if .CanPoll}}<a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="closest .trow" hx-swap="outerHTML" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' href="#">Check now</a>{{end}}</span>
<span class="c-note">{{if .Failure}}<span class="mark">{{.Failure}}</span>{{end}}{{range .Notes}}<span class="mark">{{.}}</span>{{end}}{{if .More}}<span class="mark mark-faint">+{{.More}}</span>{{end}}{{if .Finished}}<span class="mark mark-faint">finished</span>{{end}}</span>
<span class="c-act">{{if .CanPoll}}<a class="ghost act" hx-post="/admin/series/{{.Key}}/poll" hx-target="closest .trow" hx-swap="outerHTML" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' href="#">Check now</a>{{end}}{{if .CanRemove}}<button class="ghost danger" hx-post="/admin/series/{{.Key}}/remove" hx-target="closest .trow" hx-swap="outerHTML" hx-include="#filterbar" hx-vals='{"band":{{if .Band}}1{{else}}0{{end}}}' hx-confirm="Removes this series and its stored cover. No Reader has it bookmarked; one re-bookmarking it recreates the row.">Remove</button>{{end}}</span>
{{if .RemovalRefused}}<span class="row-msg">a Reader has bookmarked this Series again</span>{{end}}
</div>
{{end}}
{{/* series-list-head is the list's heading — the count and the label are one
fact. The removal answer renders it out of band (the OOB flag, like the
chrome partials) so the heading never lies past the row that made it,
and inline here it is the page's own heading. The id is the OOB swap's
hook; hx-swap-oob sits on the element the answer carries. */}}
{{define "series-list-head"}}
<div class="listhead" id="series-listhead"{{if .OOB}} hx-swap-oob="true"{{end}}>{{.Total}} series <span class="lbl">· <em>{{.FilterLabel}}</em></span></div>
{{end}}
+10 -11
View File
@@ -293,10 +293,11 @@ func libOf(q string) string {
// buildListView loads one reader's list once and derives both the tab-filtered
// items and the recent strip from it.
//
// Archived and finished series appear in their own tab and nowhere else — not
// in All, not in Updated, not in Favourites, and not in the recent strip. An
// archived favourite therefore shows only under Archived: Favourites means
// "favourites I am currently reading".
// Archived series appear in their own tab and nowhere else — not in All, not
// in Updated, not in Favourites, and not in the recent strip. An archived
// favourite therefore shows only under Archived: Favourites means "favourites
// I am currently reading". There is no Finished tab: finished is a fact about
// the Series, not a bookmark bucket (issue #157).
func (h *Handler) buildListView(readerID int64, lib, tab string) (listView, error) {
all, err := h.store.List(readerID) // already ordered updated_at DESC
if err != nil {
@@ -327,8 +328,6 @@ func (h *Handler) buildListView(readerID int64, lib, tab string) (listView, erro
items = withNew
case "archived":
items = filterBookmarks(all, func(b store.Bookmark) bool { return b.Status == store.StatusArchived })
case "finished":
items = filterBookmarks(all, func(b store.Bookmark) bool { return b.Status == store.StatusFinished })
default:
tab = "all"
items = reading
@@ -392,7 +391,7 @@ func currentLib(r *http.Request) string {
// writeChromeOOB appends the regions that live outside #list — the recent
// strip, the Updated badge and the action key — as out-of-band swaps, so a
// mutation cannot leave them describing the library as it was before the tap.
// The key is in here because it is tab-shaped too: archived and finished swap
// The key is in here because it is tab-shaped too: the archived bucket swaps
// Archive for Restore.
func (h *Handler) writeChromeOOB(w http.ResponseWriter, view listView) {
view.OOB = true
@@ -492,9 +491,9 @@ func (h *Handler) uiFavorite(w http.ResponseWriter, r *http.Request) {
h.saveAndRenderCard(w, r, b)
}
// uiStatus moves a bookmark between lifecycle buckets. This is the only place
// a series can be marked finished — the JSON API refuses that value, so the
// userscript cannot set it even by accident.
// uiStatus moves a bookmark between the two lifecycle buckets. Finished is not
// one of them: it is a fact about the Series, decided from the admin surface,
// so the web UI's per-bookmark control cannot set it (issue #157).
//
// last_chapter_num is untouched, so Upsert keeps the stored updated_at and the
// list does not reorder.
@@ -508,7 +507,7 @@ func (h *Handler) uiStatus(w http.ResponseWriter, r *http.Request) {
return
}
switch s := r.PostFormValue("status"); s {
case store.StatusReading, store.StatusArchived, store.StatusFinished:
case store.StatusReading, store.StatusArchived:
b.Status = s
default:
http.Error(w, "invalid status", http.StatusBadRequest)
+28 -4
View File
@@ -14,6 +14,7 @@ import (
"bookmarkmanager/backend/internal/api"
"bookmarkmanager/backend/internal/httpmw"
"bookmarkmanager/backend/internal/latest"
"bookmarkmanager/backend/internal/notify"
"bookmarkmanager/backend/internal/store"
"bookmarkmanager/backend/internal/token"
"bookmarkmanager/backend/internal/userscript"
@@ -59,6 +60,12 @@ type Config struct {
// the Lanes page reports the fact and derives reachability from the pass
// log rather than asking the poller (issue #145).
BrowserWSURL string
// DiscordWebhookURL is the webhook owner notices post to (issue #171).
// Unset means the whole path is off — a local stack needs no webhook,
// exactly as the browser URL behaves. The address is a secret in the
// class of TOKEN_KEY: never logged, and it must not reach any line that
// prints configuration.
DiscordWebhookURL string
// LatestPoll configures the background latest-chapter fetcher.
LatestPoll LatestPoll
}
@@ -114,6 +121,7 @@ func loadConfig() Config {
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
NovelUserscriptPath: envOr("NOVEL_USERSCRIPT_PATH", "/userscript/novel-bookmark.user.js"),
BrowserWSURL: os.Getenv("BROWSER_WS_URL"),
DiscordWebhookURL: os.Getenv("DISCORD_WEBHOOK_URL"),
LatestPoll: loadLatestPoll(),
}
c.Discord = web.DiscordConfig{
@@ -288,10 +296,22 @@ func main() {
}
s.OnSeriesCreated = acq.Acquire
}
// Owner notices (issue #171): a configured webhook makes the poller tell
// the owner about stalled Lanes. Unset means the whole path is off — a
// local stack needs no webhook, exactly as the browser URL behaves. Only
// the presence is logged; the address itself is a secret in the class of
// TOKEN_KEY.
var notifier latest.Notifier
if u := strings.TrimSpace(cfg.DiscordWebhookURL); u != "" {
notifier = notify.New(u, cfg.PublicBaseURL)
log.Println("owner notices: enabled")
} else {
log.Println("owner notices: disabled (DISCORD_WEBHOOK_URL unset)")
}
// The poller's only connection to the web layer is the database now: it is
// started for its own sake, and the Lanes page reads the pass rows it
// records (issue #145).
startLatestPoller(pollCtx, s, cfg.LatestPoll, browser)
startLatestPoller(pollCtx, s, cfg.LatestPoll, browser, notifier)
srv := &http.Server{
Addr: ":" + cfg.Port,
@@ -324,7 +344,9 @@ func main() {
// newLatestPoller wires the fetcher seams into the poller. Pace is registry
// property, not config (issue #100), so there are no knobs to pass through.
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher) *latest.Poller {
// notifier is nil when no webhook is configured: a missing webhook is a
// silent off switch, not an error (issue #171).
func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetcher, notifier latest.Notifier) *latest.Poller {
var covers latest.BrowserCoverFetcher
if f, ok := browser.(latest.BrowserCoverFetcher); ok {
covers = f
@@ -335,6 +357,7 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
BrowserFetch: browser,
CoverFetch: covers,
CoverBytesFetch: latest.NewCoverFetcher(),
Notify: notifier,
Now: time.Now,
}
}
@@ -345,7 +368,8 @@ func newLatestPoller(s *store.Store, cfg LatestPoll, fetch, browser latest.Fetch
// tracking, which is exactly how it behaved before. It returns the running
// Poller, or nil when there is none; the caller starts it for its own sake —
// the Lanes page reads the pass log, so no return value is wired anywhere.
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher) *latest.Poller {
// notifier is nil when DISCORD_WEBHOOK_URL is unset (issue #171).
func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, browser latest.Fetcher, notifier latest.Notifier) *latest.Poller {
if !cfg.Enabled {
log.Println("latest-chapter poller: disabled by config")
return nil
@@ -358,7 +382,7 @@ func startLatestPoller(ctx context.Context, s *store.Store, cfg LatestPoll, brow
// Nil browser: sites behind a JavaScript challenge are simply not polled,
// and their latest_chapter comes from the userscript alone — which is how
// the service behaved before the sidecar existed.
p := newLatestPoller(s, cfg, f, browser)
p := newLatestPoller(s, cfg, f, browser, notifier)
go p.Run(ctx)
return p
+24 -2
View File
@@ -28,6 +28,20 @@ func TestLoadConfigReadsCoverDirectory(t *testing.T) {
}
}
func TestLoadConfigReadsDiscordWebhook(t *testing.T) {
// The address is read, never defaulted: unset stays empty (the whole
// path is off), set flows into the Config for the poller's notifier.
const url = "https://discord.com/api/webhooks/000000/secret"
t.Setenv("DISCORD_WEBHOOK_URL", url)
if got := loadConfig().DiscordWebhookURL; got != url {
t.Fatalf("DiscordWebhookURL = %q, want %q", got, url)
}
t.Setenv("DISCORD_WEBHOOK_URL", "")
if got := loadConfig().DiscordWebhookURL; got != "" {
t.Fatalf("DiscordWebhookURL = %q, want empty when unset", got)
}
}
func TestLoadLatestPollEnabledParsing(t *testing.T) {
tests := []struct {
raw string
@@ -51,7 +65,8 @@ func TestLoadLatestPollEnabledParsing(t *testing.T) {
// nothing here sizes a cooldown any more.
func TestNewLatestPollerWiresFetchers(t *testing.T) {
tls := &latest.TLSFetcher{}
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil)
notifier := &stubNotifier{}
p := newLatestPoller(nil, LatestPoll{Enabled: true}, tls, nil, notifier)
if p.Fetch != tls {
t.Fatalf("Fetch not wired")
}
@@ -67,8 +82,15 @@ func TestNewLatestPollerWiresFetchers(t *testing.T) {
if p.Now == nil {
t.Fatalf("Now = nil, want the live clock")
}
if p.Notify != notifier {
t.Fatalf("Notify = %v, want the configured notifier", p.Notify)
}
}
// stubNotifier satisfies latest.Notifier so newLatestPoller's wiring can be
// asserted; it is never called.
type stubNotifier struct{ latest.Notifier }
func TestPutStatusValidation(t *testing.T) {
cases := []struct {
name string
@@ -78,7 +100,7 @@ func TestPutStatusValidation(t *testing.T) {
{"empty is no opinion", "", http.StatusOK},
{"reading", "reading", http.StatusOK},
{"archived", "archived", http.StatusOK},
{"finished is web-only", "finished", http.StatusBadRequest},
{"finished is no longer a bucket", "finished", http.StatusBadRequest},
{"garbage", "dropped", http.StatusBadRequest},
}
for _, tc := range cases {
+1335 -48
View File
File diff suppressed because it is too large Load Diff
+6
View File
@@ -76,6 +76,12 @@ services:
# independent of chromedp's own dial logic. The same trap that used to
# force a pinned Docker IP now forbids the tailnet name.
BROWSER_WS_URL: ${BROWSER_WS_URL:-}
#
# Owner-notice webhook (issue #171). Empty default, never a
# required-guard: unset means the whole path is off, so a local stack
# runs exactly as it does today. An env var not listed here never
# reaches the container.
DISCORD_WEBHOOK_URL: ${DISCORD_WEBHOOK_URL:-}
depends_on:
# The migration runner is the first thing the binary does, so a Postgres
# that is still initialising means a crash-loop until it is not.
@@ -0,0 +1,86 @@
# ADR-0014: Cover addresses derived from the bytes, not the source URL
Date: 2026-08-22
Status: accepted
## Decision
A Cover's content address is the hex SHA-256 of its **bytes**, not of the
source URL it was fetched from. `CoverAddressForBytes(body)` names the address
`putCover` stores under, `SetSeriesCover` and `ReplaceSeriesCover` point the
Series row at it, and the wire URL is built from it exactly as before — same
route, same 64-hex-digit shape, same immutability, only the input to the hash
changes. Rows written before this ADR keep their URL-derived addresses
forever: they are never rehashed on read, and they heal into byte addressing
only when a Forced Poll replaces them.
`ReplaceSeriesCover(site, seriesID, sourceURL, body, contentType)`
`(previous, current, error)` is the one write that may move a Cover once one
exists. It stores the bytes, then in one transaction locks the Series row,
reads the old `cover_address`, writes the new one and the source URL, and
reports both addresses: `previous == ""` means there was no Cover,
`previous == current` means the Site served identical artwork, and any other
pair names the stranded address.
## Why a future reader will find this surprising
The address is what makes a re-art visible at all. URL addressing collapses
every image behind a stable URL into one address, so a Series whose Cover
changes (a big-budget CPI blitz on a light novel is the standing example)
keeps serving its original cover bytes: the poll refetches the same URL,
hashes it, and the store records the same address, everyone happy except the
Reader. Nothing in the system can detect the change, because the address is a
pure function of the fetch target, and identical bytes written 1,000 times
are one blob on disk. Storing bytes we already know how to store is only a
few lines of work. **Rejecting that work is the surprising part, and the
answer is the Forced Poll wave**: for a corrupt/blank cover the poll's
fill-if-blank installer already worked, but for a *wrong but non-blank* cover
there was no write that would move it at all — only a manual truth in
`series.cover_address`, which is exactly the thing that must never be set by
hand. Byte addressing gives the replacement write a **new address to write**,
and with it a legitimate, transaction-safe mover.
## Considered options
**Keep URL addressing and add a generic "clear the cover" write.**
Rejected: clearing is a two-phase action (blank it, wait for the poll to
re-fill, hope the bytes changed in between) that cannot report what the
write did, and it makes the Series render cover-less in between. The
replacement write is atomic, reports its displacement, and has one effect:
the Series now points at bytes that actually came from its source URL.
**Address by URL, but salt it so a re-art is a new address.**
Rejected: the salt would have to live somewhere addressable (a stored per-
Series nonce), turning the address from a content fact into a mutable fact —
two rows could then hold identical bytes under different addresses and the
invariant "same bytes object" is gone.
## Consequences
- `store.CoverAddress` (URL-hash) is deleted; `CoverAddressForBytes` is
public so tests and the forced-poll wave can predict addresses from the
bytes fakes serve.
- Legacy URL-addressed rows are read-only facts: `GetCover(sourceURL)` keeps
resolving them (the poll heal path), and they are re-addressed only by a
forced replacement. Until one happens, they are invisible to byte-derived
lookups — the reverse direction was always true, so this side has no
migration and no lookup fan-out.
- A replaced Cover's old bytes stay on disk under their address (the `covers`
row is untouched — only the Series row moves). Nothing reclaims them
today; a later sweep is a small query over `covers` addresses not
referenced by any `series` row.
- `SetSeriesCover` keeps its `cover_address = ''` guard untouched: the
acquisition-at-creation and poll fill paths still may not overwrite a
non-blank Cover. The two installers are now deliberately different
functions instead of one function with a conditional.
- The address is still a filesystem path (≤64 hex chars, no separators), so
`coverAddressRe` and the sharding stay exactly as they are.
## Cost of reversing
The URL-hash side of the current rows is uncomputable from the rows alone: a
rollback would need every stored blob's source URL, a join to a table that
does not store it, or a refetch of every Series. Keeping both derivations
resolvable is cheaper than either, so the two derivations are documented in
the 0009 migration comment: no component may assume which derivation a
stored address came from, because the 64-hex shape hides it.
+109
View File
@@ -0,0 +1,109 @@
# ADR-0015: Finished is a fact about the Series, not a bookmark bucket
Date: 2026-08-22
Status: accepted
## Decision
"Finished" moves from the per-Reader `bookmarks.status` bucket to a
Series-owned flag: `series.finished_at`, unix ms, zero while the Series is
still running. The Lane's gate reads the flag — a Series is polled only while
`finished_at = 0` — and `bookmarks.status` keeps exactly two values,
`reading` and `archived`.
The cutover is one-way, done by migration 0016 in three load-bearing
statements:
1. `ALTER TABLE series ADD COLUMN finished_at bigint NOT NULL DEFAULT 0`.
2. Seed it from the bookmarks: a Series is stamped finished when no bookmark
on it is outside the `finished` bucket. This mirrors the pre-cutover due
gate exactly — the old query skipped a Series only while
`COUNT(*) FILTER (WHERE status <> 'finished') = 0` — so no Series changes
polling state at the cutover.
3. Rewrite every `finished` bookmark to `archived`. The bucket is gone; the
seed ran first because it is the only statement that can still read it.
The JSON API rejects a `finished` status with the same plain 400 as any
unknown value, and the web UI no longer offers a Finished tab, a finish
button, or a finished state badge.
## Why a future reader will find this surprising
The bucket looked Reader-shaped but described a Series fact. A Series is
finished once, and every Reader reading it is then on a finished Series —
yet the bucket carried three copies of the answer, one per Reader, free to
disagree. The disagreement is not theoretical: a second Reader who merely
kept the Series (or never read it) kept it in `reading`, so the poll gate
kept fetching a Series the first Reader had closed out, forever. Worse, the
disagreement was never resolvable — nothing in the system could say "this
Series is finished" without rewriting every bookmark, which silently edits
another Reader's progress state.
The flag is also the only memory of the bucket after the flip. `finished`
bookmarks become `archived` because a two-value status needs no third
value, and an archived row must keep meaning "shelved, but the Series is
being watched" — which is what the row says. The migration's seed is what
keeps the legacy meaning: a Series every Reader finished is stamped, so the
Lane stops polling it just as it would have pre-cutover; a Series any
Reader still reads is left alone, exactly as the old gate left it. A
Series whose every Reader only shelved (archived) continues to be polled,
because an archived bookmark is *supposed* to be polled — the cutover
changes the answer, it does not invent it. And because the flag is a series
fact, the cutover also repairs the disagreement case: the moment one Reader
has the Series open, it reads as finished to everyone.
The migration is the only writer of the flag today; the undo is writing 0,
which returns the Series to the poll. An owner-facing "mark finished" write
is deliberately not part of this change — the gate is what this ticket
rewrites, and the write can land on top of it without touching anything
here.
The userscript merge ranks `archived > reading` now. `finished` is not a
value the wire can carry, so the merge cannot un-finish a row — it cannot
even name the state it is protecting.
## Considered options
**Keep the bucket and add the flag alongside it, both live.**
Rejected: two sources of truth for one fact, with the Lane forced to
resolve "any Reader finished?" on every due query and every Reader write
still able to resurrect a finished Series. The whole point of the change is
that the finished state survives Readers.
**Stamp a Series finished when every bookmark is finished *or* archived.**
Rejected: it flips polling state at the cutover. Shelved-only Series were
polled before; making them finished stops the checks the reader knowingly
asked to keep.
**Finish as "no bookmark is reading", leaving the buckets untouched.**
Rejected for the same reason plus one: `archived` is a Reader's own state
and the flip is what makes the flag the *only* source of finished. Keeping
the `finished` value in the table would force every status validation,
merge and UI branch to keep handling a value no write can produce.
## Consequences
- `bookmarks.status` is validated to `reading | archived`, empty meaning
"keep the stored value"; the API's 400 for `finished` is now the generic
invalid-status rejection rather than a special case, and the web UI's own
status control rejects it the same way.
- The Lane due query and the eligible count read `finished_at`; a Forced
Poll (issue #146) still overrides the flag — the owner asked, so the Lane
looks — and the pending force clears itself when the pass stamps the
check timestamp, never by touching `finished_at`.
- The web UI has no Finished tab; finished Series render in Archived with
their archived badge, dimmed like any shelved row.
- Migration 0016 stamps `finished_at` with the migration's own clock
(`now()` ms), which is also the undo: write 0 and the Series returns to
the poll.
## Cost of reversing
The finished buckets are destroyed by the flip; reversing means re-deriving
per-Reader finished state from a Series fact that now encodes the
majority-agreement snapshot plus whatever the owner reset since. The stamp
differentiates "finished at the cutover and untouched" from "finished
later", but not which Reader's choice each Series carried, and any Series
the owner has since restored is gone from the derivation entirely. The ADR
is a statement of intent to ship the one-way cutover and live with its
consequences; the per-Reader history is not kept anywhere in the schema.
@@ -0,0 +1,147 @@
# ADR-0016: The failure row is the state
Date: 2026-08-22
Status: accepted
## Decision
A Series that used to work and has stopped is recorded in one table,
`poll_failures`, keyed by the same `(site, series_id)` composite the rest of
the system uses. One row per failing Series, carrying the outcome word and a
`failing_since` stamp. The row's **existence** is the failure state: there is
no success sentinel, no counter, no history, and nothing added to the Series
row. A correct read deletes the row, and a repeated identical failure writes
nothing — the stamp ages the run of failures, not the current word.
The write is one upsert, from the poll pass loop:
```sql
INSERT INTO poll_failures (site, series_id, outcome, failing_since)
VALUES ($1, $2, $3, $4)
ON CONFLICT (site, series_id) DO UPDATE SET outcome = EXCLUDED.outcome
WHERE poll_failures.outcome <> EXCLUDED.outcome
```
`failing_since` is simply absent from the `SET` arm, so it is never
overwritten, and the `WHERE` makes an unchanged word write nothing at all —
one statement, no read-then-write race. The clear is a plain `DELETE`; a row
that is not there is not an error, because the poller clears on every
successful read and most clears find nothing. The composite foreign key
cascades: deleting a Series takes its failure row, so orphan removal stays a
single statement.
## Why a future reader will find this surprising
The failure state lives in a table of its own, not on the Series row and not
in the pass log. A flag on `series` would be the familiar shape, and the
pass log already records outcomes per pass — why a third place?
Because the failure must outlive the pass that observed it and mean
something the pass row cannot say. The pass log is a per-Lane stream: it
records that a Site returned N `errors` and M `not_found` on a given run,
but "this exact Series has been failing for three months" is not a question
any single pass row answers — it is a question across rows, and the Lanes
page is a live view of the last 14 days, not a Series index. A Series-level
fact needs Series-level storage, and a flag on the Series row is the wrong
shape too: the failure is *transient by definition* (a correct read ends it)
and *repeatable* (the same Series can fail again next year), so the honest
record of "failing since" is a stamp that moves, not a column that flips.
A row that is created and deleted by the read's outcome is that stamp, and
nothing else: the moment a read succeeds the row is gone, so "is this Series
failing?" is answered by one indexed existence check — no success sentinel
to keep consistent with the failure, no counter to reset, no history to
prune. The state cannot drift out of step with the reads that maintain it,
because the reads *are* the maintenance.
The two no-write outcomes are the second surprise. `refused` (the Site is
holding a challenge) and `unreachable` (the browser sidecar was lost
mid-loop) issue **no statement at all** — neither an upsert nor a delete.
The direction matters, and both directions are wrong to touch:
- **Writing would condemn a whole library for one Site's bad day.** A
refusal is the Site's mood, not a fact about any particular Series: when
Cloudflare turns a zone's JS detection on, every Series on that Site reads
refused in the same pass. Writing those rows would stamp every Series in
the library as failing on the evidence of one Site's configuration, and
#166's worklist would present a Site-wide outage as thousands of broken
Series.
- **Deleting would claim a recovery nothing read.** A lost sidecar tells us
nothing about the page — the read never happened. Deleting the row would
report the Series healthy, and worse, it would reset `failing_since`: the
age that makes a three-month failure findable would start over on a
browser restart, erasing evidence no page read contradicted.
So a refusal or an unreachable pass leaves the table exactly as it found
it — the pass neither adds rows that would condemn nor deletes rows that
would claim recovery. The four outcomes that are real evidence about the
page (`not_found`, `no_chapter`, `unfetchable`, `errors`) write; the two
that are not write nothing; success deletes. There is no third case.
## Considered options
**A `failing_since` column on the Series row, alongside the failure word.**
Rejected: it is two more columns on a table every due query and every
bookmark join already touches, for a state that is transient and repeatable.
The column would need its own "clear on success" writer anyway — the same
maintenance the row has — while permanently widening the hottest table in
the system for a value that is usually absent. And the worklist's join
would have to distinguish "never failed" from "currently healthy", which
is exactly the sentinel problem the row avoids: an empty column means both.
**A counter or last-outcome timestamp instead of (or beside) the row.**
Rejected: nothing in the system consumes a failure *count* or a
last-outcome time — the worklist needs "failing and since when". A counter
invites "three failures = something" thresholds that the ticket explicitly
keeps out of scope, and a last-outcome timestamp conflates "the word
changed" with "the run restarted", which the age is deliberately defined
against. The stamp is the only number that means anything, and the row
carries exactly that.
**Write into `poll_passes` and derive the failure state from the pass
stream.**
Rejected: a pass row is per-Lane and per-run; deriving "this Series is
failing since" means scanning 14 days of outcome counts per Series and
guessing at continuity across retention boundaries. The pass log answers
"what did this Site's lane do recently"; the failure table answers "which
Series are broken right now". Two questions, two tables — the pass log is
already pruned on a fixed cutoff, which would silently reset every
failure's age the moment the evidence aged out.
**Refused/unreachable write nothing, but the delete happens anyway (or the
upsert happens, with no delete).**
Rejected in both directions, above: writing condemns a library for one
Site's mood; deleting claims a recovery nothing read and resets the age
that makes a long failure findable. The asymmetry is the point — the two
outcomes are evidence about the *environment*, never about a page.
## Consequences
- The poller writes from the pass loop, one call after `checkOne`, and
clears through the ordinary success path: a Forced Poll that reads the
page deletes the row with no forced branch of its own, and a Forced Poll
*request* — which only stamps the request — clears nothing.
- `poll_failures` is a poll-write table like `poll_lanes` and
`poll_passes`: the store's two methods live next to the Lane-pass
writers, and neither runs on a request path. A store failure is logged
and the poll continues — a failure row is best-effort, and no single bad
Series may stall a Lane.
- The due query does not join the table. A failing Series is polled at the
same pace as any other, because the moment the query started pacing by
failure state, the age would stop meaning what the worklist reads it as.
- Orphan removal stays a single statement: the composite foreign key's
`ON DELETE CASCADE` is what takes the failure row with the Series.
- The table starts empty at deploy, so nothing is findable for the first
twelve hours after deploy and a pre-existing breakage reads as new.
Accepted; the alternative is inventing history.
## Cost of reversing
The failure state is derived, not stored: a rollback drops the table and
every in-progress failure age with it, leaving the pass log's outcome
counts as the only trace — which is exactly the "log line nobody reads"
the ticket set out to replace. Recreating the table later starts the ages
over, so a reversal that is later reversed loses the evidence of the
intervening failures. The failure state itself, however, is the one thing
that is *not* lost by reversing: it is re-derived from the next pass, in
the same direction the original design derives it — the row is
recreated by the next failing read and deleted by the next good one.
+14 -14
View File
@@ -2,7 +2,7 @@
Source of truth: the Claude Design project **BookmarkManager Web UI**
(`969ac210-fe02-4c01-ae1b-9a271dcc779a`, `index.html` + siblings
`archived.html`/`fav.html`/`finished.html`/`new.html`/`login.html`/`mobile.html`,
`archived.html`/`fav.html`/`new.html`/`login.html`/`mobile.html`,
`style.css`, `filter.js`). This file records the rules that got implemented so
a future agent can extend the UI without re-reading the design.
@@ -48,7 +48,7 @@ Defined once in `backend/internal/web/static/style.css` `:root`, mirrored in the
| --- | --- | --- | --- |
| `--ink` | `#100f0e` | `#f7f4ef` | page |
| `--ash` | `#161413` | `#efeae3` | recessed panel (chapter form) |
| `--dim` | `#0d0c0b` | `#f1ede7` | archived / finished row background |
| `--dim` | `#0d0c0b` | `#f1ede7` | archived row background |
| `--rule` | `#221f1d` | `#e0dad2` | hairline between sheets, button borders |
| `--rule-soft` | `#1a1817` | `#e8e3dc` | the measure's own side edges |
| `--field-line` | `#2c2926` | `#d4cdc4` | input borders, ghost-button underline |
@@ -70,7 +70,7 @@ Defined once in `backend/internal/web/static/style.css` `:root`, mirrored in the
| `--danger-soft` | `#e2aaa1` | `#7c2c22` | text on danger wash |
| `--brass` | `#b8912f` | `#8a681c` | favourite — a cooler second metal |
| `--slate` | `#7fa0c0` | `#3f6689` | archive accent |
| `--moss` | `#7fae86` | `#3d6c46` | finished accent |
| `--moss` | `#7fae86` | `#3d6c46` | finished Series label |
| `--clay` | `#b5906f` | `#7c5533` | set-chapter accent |
| `--trash` | `#977671` | `#8c6558` | remove, at rest — icons need 3:1, not 4.5:1 |
| `--patina` | `#5fb3a6` | `#1f6f66` | admin page only — a Poll Lane needing attention, a Reader whose reports are blocked |
@@ -164,9 +164,9 @@ rather than scaling the artwork down.
**Action key** (`.keyrow`): one permanent line under the tabs naming what
every icon in `.actions` does — Read / Fav / Chapter / Archive / Done /
Delete — so the icon strip on a card is never a guess. The key follows the tab,
not the row: Archive becomes Restore under Archived and Finished, and Finished
drops Done. On a phone each pair stacks icon-over-word
Delete — so the icon strip on a card is never a guess. The key follows the
tab, not the row: under Archived, Archive becomes Restore. On a phone each
pair stacks icon-over-word
(`flex-direction: column`) so the word gets the full cell width; ≥720px it lays
out icon-beside-word at the same wording. `.pair.brass` and `.pair.trash`
carry their icon's resting accent so the key itself teaches the colour
@@ -179,7 +179,7 @@ article.card[.is-new|.is-dim]#card-<key>[data-title]
.row
a.cover[tabindex="-1" aria-hidden] img | span.monogram, + span.foot-rule[.brass]
.body .title-line (h3.title + svg.fav-mark) , p.meta
.actions play, favourite, chapter | lifecycle: archive/restore, finish, remove
.actions play, favourite, chapter | lifecycle: archive/restore, remove
form.chapter-form[hidden] .hint + .field(input + Save) + .hint (latest known)
.confirm-row[.calm][hidden] × one per lifecycle action, span + (go/danger-solid, Cancel)
p.error-inline[hidden]
@@ -188,7 +188,7 @@ article.card[.is-new|.is-dim]#card-<key>[data-title]
Rules that are easy to break:
- `.is-new` only when `Status == reading && HasNewChapter`; `.is-dim` for
`archived` and `finished`. Both are set on the `<article>` — every heat and
`archived`. Both are set on the `<article>` — every heat and
dim rule is a descendant selector off those two classes, so a new sub-element
inherits the state for free.
- `.actions` is `flex: 1 0 100%` inside `.row`, which is what makes it a
@@ -196,19 +196,19 @@ Rules that are easy to break:
the row at ≥720px. Cells are 46px tall on phone (thumb target) and divided by
`border-right: 1px var(--rule)`, last child none.
- Three clusters by consequence, in this order: navigate (`.play`) | organize
(`.fav`, `.pencil`) | lifecycle (`.box`/`.restore`, `.finish`, `.remove`,
(`.fav`, `.pencil`) | lifecycle (`.box`/`.restore`, `.remove`,
each carrying the `.lifecycle` class). Lifecycle cells sit on a recessed
`--ash` ground so the thumb reads "this one moves the series" before it
reads which icon it landed on; ≥720px they separate by a 10px gap instead of
the phone's inset hairline.
- Every lifecycle button that moves a series out of the list is
**confirm-gated**: it opens its own `.confirm-row` (`archive`, `finish`,
`remove` — `toggleConfirmRow(key, kind)` in `filter.js`). Archive and finish
ask in `.calm` grey since they're reversible; remove alone gets the
**confirm-gated**: it opens its own `.confirm-row` (`archive`,
`remove` — `toggleConfirmRow(key, kind)` in `filter.js`). Archive asks in
`.calm` grey since it's reversible; remove alone gets the
`--danger-wash` treatment and names the series in its question. Restore
fires instantly — no confirm — because it's the reversal.
- Per-action hover/press accent: `.fav` → `--brass`, `.pencil` → `--clay`,
`.box` → `--slate`, `.finish` → `--moss`. `.play` stays paper/ember (ember
`.box` → `--slate`. `.play` stays paper/ember (ember
only when `.is-new`). `.remove` stays `--trash` at rest, `--danger` on
hover. Desktop cell borders follow the same accent on hover
(`border-color: currentColor`); the two coloured *resting* states
@@ -286,7 +286,7 @@ No transforms on hover, no scale, no easing curves beyond `ease-out`/`linear`.
never reuse `--ember` or `--danger` for anything but their one meaning.
3. If it is per-series, hang it off `.is-new` / `.is-dim` rather than adding a
third state class.
4. If it removes a series from the current view (archive/finish/remove-shaped),
4. If it removes a series from the current view (archive/remove-shaped),
it is confirm-gated via its own `.confirm-row` — no exceptions, restore is
the only instant action because it's the one that's reversible by nature.
5. Icon → `templates/icons.html`; nothing inlines SVG paths. Brand mark stays
+23 -10
View File
@@ -471,9 +471,10 @@
}
// Only an explicit archive/restore has an opinion about the bucket. Every
// other write omits `status`, so the server keeps the stored one — otherwise
// a cached value would resend "finished" (which the API rejects with 400) or
// silently un-archive a series archived on another device.
// other write omits `status`, so the server keeps the stored one —
// otherwise a cached value would resend a stale status (the API rejects
// unknown values with 400) or silently un-archive a series archived on
// another device.
async function apiPut(key, obj, { sendStatus = false } = {}) {
const body = Object.assign({}, obj);
if (!sendStatus) delete body.status;
@@ -550,6 +551,12 @@
return b.kind || "manga";
}
// finished is a fact about the Series, decided from the owner's side and
// derived on the wire (issue #157); the panel only labels it.
function finishedLabel(b) {
return b && b.finished ? "Finished" : "";
}
// ============================================================
// Retry queue
//
@@ -888,8 +895,8 @@
}
// Archive parks a series: it leaves All and Favourites but the server keeps
// polling it for new chapters. "finished" is deliberately not reachable from
// here — the API rejects that value, it is a web-UI decision.
// polling it for new chapters. finished is not a bucket the script can
// reach — it is a fact about the Series now (issue #157).
async function toggleArchive(key) {
const existing = state.byKey[key];
if (!existing) return;
@@ -966,7 +973,6 @@
const now = Date.now();
const due = state.list
.filter((b) => b.site === site && b.series_url)
.filter((b) => statusOf(b) !== "finished")
.filter((b) => now - (checked[b.key] || 0) >= LATEST_CHECK_THROTTLE_MS)
.sort((a, b) => (checked[a.key] || 0) - (checked[b.key] || 0))
.slice(0, LATEST_CHECK_BATCH);
@@ -1335,8 +1341,8 @@
}
// Tabs narrow what is drawn; state.list always holds every bookmark.
// Archived rows are hidden from All and Favourites, and finished ones —
// which only the web UI can set — are hidden from every tab.
// Archived rows are hidden from All and Favourites; a Series that is
// finished is the web UI's business and arrives archived (issue #157).
for (const [id, tab] of [["tabAll", "all"], ["tabFav", "favorites"], ["tabArc", "archived"]]) {
root.getElementById(id).classList.toggle("active", activeTab === tab);
}
@@ -1399,6 +1405,7 @@
el("div", { class: "meta" }, [
el("a", { class: "go-t t", href: cont, text: b.title || b.series_id }),
el("div", { class: "c" + (behind ? " behind" : ""), text: sub }),
b.finished && el("span", { class: "finished", text: finishedLabel(b) }),
el("div", { class: "actions" }, [
el("button", {
class: "btn small star" + (b.favorite ? " on" : ""),
@@ -1669,7 +1676,7 @@
--paper: #f2ece5; --paper-hot: #f0d3cb; --paper-dim: #ddd5cb;
--mute: #8d857c; --mute-2: #5a5450; --faint: #3a3733; --faint-2: #57504b;
--ember: #e0452c; --ember-wash: #1a1211; --ember-ink: #150907;
--ember-soft: #eda798; --brass: #b8912f; --trash: #6b5450;
--ember-soft: #eda798; --brass: #b8912f; --trash: #6b5450; --moss: #7fae86;
--font-display: Georgia, "Times New Roman", serif;
--font-mono: ui-monospace, SFMono-Regular, Menlo, monospace;
--font-body: system-ui, -apple-system, sans-serif;
@@ -1788,6 +1795,12 @@
font: 400 17px/1.2 var(--font-display); color: var(--paper-dim);
max-width: 100%; white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
}
/* finished: a Series fact, not a state class (issue #157) — same meta
typography as .c, moss instead of mute. */
.finished {
margin: 0; color: var(--moss);
font: 500 10px/1.4 var(--font-mono); letter-spacing: .12em; text-transform: uppercase;
}
.item.hot .t { color: var(--paper-hot); border-bottom: 1px solid var(--ember); padding-bottom: 3px; }
.item.dim .t { font-style: italic; color: var(--mute); }
.c {
@@ -1838,7 +1851,7 @@
// Exposes pure logic only — see userscript/test/logic.test.js.
// ============================================================
if (typeof window === "undefined" && typeof module === "object" && module.exports) {
module.exports = { stripBuildHash, comixSeriesId, asura, demonic, comix, kagane, anchorsFromHTML, statusOf, kindOf };
module.exports = { stripBuildHash, comixSeriesId, asura, demonic, comix, kagane, anchorsFromHTML, statusOf, kindOf, finishedLabel };
}
// ============================================================
+27 -13
View File
@@ -268,8 +268,9 @@
// Duplicate case (spec user story 2): one row must survive, and it is
// the one already under the repaired key — with the stale row's
// progress carried across when it is ahead, favourite OR'd, and the
// stronger lifecycle bucket kept (finished > archived > reading, so a
// merge can never silently un-archive or un-finish a row).
// stronger lifecycle bucket kept (archived > reading, so a merge can
// never silently un-archive a row). finished is not a bucket the
// script can see (issue #157).
const merged = Object.assign({}, existing);
if (
existing.last_chapter_num == null ||
@@ -280,7 +281,7 @@
merged.last_chapter_url = stale.last_chapter_url;
}
merged.favorite = !!(existing.favorite || stale.favorite);
const rank = (s) => ({ finished: 2, archived: 1, reading: 0 }[s || "reading"] || 0);
const rank = (s) => ({ archived: 1, reading: 0 }[s || "reading"] || 0);
merged.status = rank(stale.status) > rank(existing.status) ? stale.status : existing.status;
merged.updated_at = Math.max(existing.updated_at || 0, stale.updated_at || 0);
listOut = list.filter((b) => b.key !== oldKey).map((b) => (b.key === newKey ? merged : b));
@@ -405,9 +406,10 @@
}
// Only an explicit archive/restore has an opinion about the bucket. Every
// other write omits `status`, so the server keeps the stored one — otherwise
// a cached value would resend "finished" (which the API rejects with 400) or
// silently un-archive a series archived on another device.
// other write omits `status`, so the server keeps the stored one —
// otherwise a cached value would resend a stale status (the API rejects
// unknown values with 400) or silently un-archive a series archived on
// another device.
async function apiPut(key, obj, { sendStatus = false } = {}) {
const body = Object.assign({}, obj);
if (!sendStatus) delete body.status;
@@ -484,6 +486,12 @@
return b.kind || "manga";
}
// finished is a fact about the Series, decided from the owner's side and
// derived on the wire (issue #157); the panel only labels it.
function finishedLabel(b) {
return b && b.finished ? "Finished" : "";
}
// ============================================================
// Retry queue
//
@@ -792,8 +800,8 @@
}
// Archive parks a series: it leaves All and Favourites but the server keeps
// polling it for new chapters. "finished" is deliberately not reachable from
// here — the API rejects that value, it is a web-UI decision.
// polling it for new chapters. finished is not a bucket the script can
// reach — it is a fact about the Series now (issue #157).
async function toggleArchive(key) {
const existing = state.byKey[key];
if (!existing) return;
@@ -878,7 +886,6 @@
const now = Date.now();
const due = state.list
.filter((b) => b.site === site && b.series_url)
.filter((b) => statusOf(b) !== "finished")
.filter((b) => now - (checked[b.key] || 0) >= LATEST_CHECK_THROTTLE_MS)
.sort((a, b) => (checked[a.key] || 0) - (checked[b.key] || 0))
.slice(0, LATEST_CHECK_BATCH);
@@ -1241,8 +1248,8 @@
}
// Tabs narrow what is drawn; state.list always holds every bookmark.
// Archived rows are hidden from All and Favourites, and finished ones —
// which only the web UI can set — are hidden from every tab.
// Archived rows are hidden from All and Favourites; a Series that is
// finished is the web UI's business and arrives archived (issue #157).
for (const [id, tab] of [["tabAll", "all"], ["tabFav", "favorites"], ["tabArc", "archived"]]) {
root.getElementById(id).classList.toggle("active", activeTab === tab);
}
@@ -1305,6 +1312,7 @@
el("div", { class: "meta" }, [
el("a", { class: "go-t t", href: cont, text: b.title || b.series_id }),
el("div", { class: "c" + (behind ? " behind" : ""), text: sub }),
b.finished && el("span", { class: "finished", text: finishedLabel(b) }),
el("div", { class: "actions" }, [
el("button", {
class: "btn small star" + (b.favorite ? " on" : ""),
@@ -1593,7 +1601,7 @@
--paper: #f2ece5; --paper-hot: #f0d3cb; --paper-dim: #ddd5cb;
--mute: #8d857c; --mute-2: #5a5450; --faint: #3a3733; --faint-2: #57504b;
--ember: #e0452c; --ember-wash: #1a1211; --ember-ink: #150907;
--ember-soft: #eda798; --brass: #b8912f; --trash: #6b5450;
--ember-soft: #eda798; --brass: #b8912f; --trash: #6b5450; --moss: #7fae86;
--font-display: Georgia, "Times New Roman", serif;
--font-mono: ui-monospace, SFMono-Regular, Menlo, monospace;
--font-body: system-ui, -apple-system, sans-serif;
@@ -1719,6 +1727,12 @@
font: 500 10px/1.4 var(--font-mono); letter-spacing: .12em; text-transform: uppercase;
}
.c.behind { color: var(--ember); }
/* finished: a Series fact, not a state class (issue #157) — same meta
typography as .c, moss instead of mute. */
.finished {
margin: 0; color: var(--moss);
font: 500 10px/1.4 var(--font-mono); letter-spacing: .12em; text-transform: uppercase;
}
.actions { display: flex; gap: 0; flex-wrap: wrap; margin-top: 2px; }
.btn {
background: none; color: var(--mute); border: 1px solid var(--rule);
@@ -1762,7 +1776,7 @@
// Exposes pure logic only — see userscript/test/novel-logic.test.js.
// ============================================================
if (typeof window === "undefined" && typeof module === "object" && module.exports) {
module.exports = { novelfull, lightnovelworld, anchorsFromHTML, statusOf, kindOf, maxChapter, escapeRe, computeLatestChapter, repairLnwStaleRow };
module.exports = { novelfull, lightnovelworld, anchorsFromHTML, statusOf, kindOf, maxChapter, escapeRe, computeLatestChapter, repairLnwStaleRow, finishedLabel };
}
// ============================================================
+13 -1
View File
@@ -58,6 +58,7 @@ const {
anchorsFromHTML,
statusOf,
kindOf,
finishedLabel,
} = require("../manga-bookmark.user.js");
// detect() reads only these four properties off location.
@@ -395,7 +396,18 @@ test("statusOf defaults a missing status to reading", () => {
assert.equal(statusOf({}), "reading");
assert.equal(statusOf({ status: "" }), "reading");
assert.equal(statusOf({ status: "archived" }), "archived");
assert.equal(statusOf({ status: "finished" }), "finished");
});
// ============================================================
// finishedLabel — finished is a fact about the Series (issue #157), which is
// the server's business; the panel only names it. The label is text through
// the el() helper, never markup.
// ============================================================
test("finishedLabel names a finished series and nothing else", () => {
assert.equal(finishedLabel({ finished: true }), "Finished");
assert.equal(finishedLabel({ finished: false }), "");
assert.equal(finishedLabel({}), "");
});
// ============================================================
+30
View File
@@ -49,6 +49,7 @@ const {
kindOf,
maxChapter,
repairLnwStaleRow,
finishedLabel,
} = require("../novel-bookmark.user.js");
function loc(href) {
@@ -563,6 +564,23 @@ test("repairLnwStaleRow does not regress progress when the repaired-key row is a
assert.equal(out.list[0].last_chapter_num, 100);
});
// ============================================================
// Merge rank — exactly two values: archived beats reading, and nothing else
// can ever win (issue #157 retired the finished bucket).
// ============================================================
test("repairLnwStaleRow merge keeps archived over reading and lets no third value win", () => {
const canonical = (over) => staleRow(Object.assign({ key: "lightnovelworld:immortality-simulator" }, over));
// archived beats reading whether it arrives as the stale row or the repaired one
let out = repairLnwStaleRow([canonical({ status: "archived" }), staleRow({ status: "reading" })], [], {}, lnwPage());
assert.equal(out.list[0].status, "archived");
out = repairLnwStaleRow([staleRow({ status: "archived" }), canonical({ status: "reading" })], [], {}, lnwPage());
assert.equal(out.list[0].status, "archived");
// "finished" is not a bucket anymore: a row carrying it reads as the
// default, so it can never win (issue #157).
out = repairLnwStaleRow([staleRow({ status: "finished" }), canonical({ status: "reading" })], [], {}, lnwPage());
assert.equal(out.list[0].status, "reading");
});
// ============================================================
// kindOf
// ============================================================
@@ -575,6 +593,18 @@ test("kindOf passes through novel", () => {
assert.equal(kindOf({ kind: "novel" }), "novel");
});
// ============================================================
// finishedLabel — finished is a fact about the Series (issue #157), which is
// the server's business; the panel only names it. The label is text through
// the el() helper, never markup.
// ============================================================
test("finishedLabel names a finished series and nothing else", () => {
assert.equal(finishedLabel({ finished: true }), "Finished");
assert.equal(finishedLabel({ finished: false }), "");
assert.equal(finishedLabel({}), "");
});
// ============================================================
// Source guard
//