cd3a7e3d01
## Summary Split latest-chapter polling cooldowns by fetch cost. Browser-backed kagane and novelfull series now rest longer without changing the cadence of plain-TLS sites. ## Behavior - Plain-TLS series keep the 1h default cooldown. - Browser-backed series use `LATEST_CHAPTER_POLL_BROWSER_COOLDOWN`, defaulting to 6h. - Both cooldowns share the existing 15m minimum floor; invalid values retain the existing fallback behavior. - The poller still selects both classes in one due query per cycle. - Existing ordering and exclusions remain unchanged: reader-count precedence, least-recently-checked ordering, finished exclusion, archived polling, and orphan exclusion. ## Implementation - Added the browser cooldown to backend configuration and passed it through production poller construction. - Added the browser-site list as the single routing source used for both due-query cutoff selection and fetcher choice. - Kept all query values parameterized; the site list is passed as a bound PostgreSQL array parameter. - Updated startup logging to report interval, plain cooldown, browser cooldown, batch, and stagger. - Documented the variable, default, and floor in `README.md`, `.env.example`, `backend/AGENTS.md`, and `docker-compose.yml`. ## Review findings addressed The first review found that configuration parsing was correct but `startLatestPoller` did not pass `BrowserCooldown` into `latest.Poller`; every browser-backed row would therefore have been due immediately. Production construction now goes through `newLatestPoller`, with a regression test covering both cooldown fields. The review also identified duplicated browser-site knowledge in fetch routing. `slices.Contains(browserBackedSites, site)` now reuses the same list already supplied to the store query. ## Verification - Focused backend tests pass: `go test ./internal/latest ./internal/store .`. - Full suite passes: `go test ./...`. - `graphify update .` completed. - Issue #42 was updated and closed. Reviewed-on: #48 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
787 lines
31 KiB
Go
787 lines
31 KiB
Go
package store
|
|
|
|
import (
|
|
"database/sql"
|
|
"embed"
|
|
"errors"
|
|
"fmt"
|
|
"io/fs"
|
|
"path"
|
|
"regexp"
|
|
"slices"
|
|
"strconv"
|
|
"strings"
|
|
|
|
"github.com/jackc/pgx/v5/pgtype"
|
|
_ "github.com/jackc/pgx/v5/stdlib"
|
|
)
|
|
|
|
// Bookmark is one tracked series, keyed "<site>:<series_id>" across both sites.
|
|
//
|
|
// LastChapter* is the user's read progress; LatestChapter* is the newest
|
|
// chapter the site has published, captured opportunistically by the userscript.
|
|
//
|
|
// Title, SeriesURL, Cover, Kind and LatestChapter* live on the shared Series
|
|
// row (ADR-0003) and are joined in on read; Bookmark carries only what differs
|
|
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
|
|
// format stays flat regardless — see ADR-0004.
|
|
type Bookmark struct {
|
|
Key string `json:"key"`
|
|
Site string `json:"site"`
|
|
SeriesID string `json:"series_id"`
|
|
Title string `json:"title"`
|
|
SeriesURL string `json:"series_url"`
|
|
Cover string `json:"cover"`
|
|
LastChapter string `json:"last_chapter"`
|
|
LastChapterNum float64 `json:"last_chapter_num"`
|
|
LastChapterURL string `json:"last_chapter_url"`
|
|
Favorite bool `json:"favorite"`
|
|
LatestChapter string `json:"latest_chapter"`
|
|
LatestChapterNum *float64 `json:"latest_chapter_num"` // nil until first captured
|
|
UpdatedAt int64 `json:"updated_at"` // unix ms; see Upsert
|
|
// Status is the lifecycle bucket: reading, archived, or finished.
|
|
// Archived series stay polled for new chapters; finished ones do not.
|
|
// Empty on the way in means "no opinion" — see Upsert.
|
|
Status string `json:"status"`
|
|
// Kind is the library bucket: manga or novel. Empty on the way in means
|
|
// "no opinion" — see Upsert.
|
|
Kind string `json:"kind"`
|
|
}
|
|
|
|
// Series is one distinct work, shared by every bookmark that tracks it. It is
|
|
// keyed (site, series_id) — the pair a bookmark key decomposes into — and
|
|
// exists once no matter how many bookmarks point at it (ADR-0003).
|
|
//
|
|
// Title, SeriesURL and Cover are written once, at creation: a PUT naming an
|
|
// existing Series has them ignored, and only the backend's own Poll may change
|
|
// them. Kind and the latest-chapter fields are last-write-wins like the
|
|
// bookmark's own fields. Never serialized: the wire format is the flat
|
|
// Bookmark (ADR-0004).
|
|
type Series struct {
|
|
Site string
|
|
SeriesID string
|
|
Title string
|
|
SeriesURL string
|
|
Cover string
|
|
Kind string
|
|
LatestChapter string
|
|
LatestChapterNum *float64 // nil until first captured
|
|
LatestCheckedAt int64 // unix ms; see MarkLatestChecked
|
|
|
|
// readerCount is the number of bookmarks referencing this series, filled
|
|
// only by the due-queue query that orders on it.
|
|
readerCount int
|
|
}
|
|
|
|
// Key returns the canonical identity in bookmark-key form ("<site>:<series_id>"),
|
|
// used by the poller's logs and by tests asserting on the due queue.
|
|
func (s Series) Key() string { return s.Site + ":" + s.SeriesID }
|
|
|
|
// HasNewChapter reports whether the site has published past the read point.
|
|
// A nil LatestChapterNum means nothing has been captured yet, which is not the
|
|
// same as "nothing new".
|
|
func (b Bookmark) HasNewChapter() bool {
|
|
return b.LatestChapterNum != nil && *b.LatestChapterNum > b.LastChapterNum
|
|
}
|
|
|
|
// chapterLeadIn matches the prefix the userscript and the poller both write
|
|
// ("Chapter 250"), so the UI can add exactly one "Ch " of its own instead of
|
|
// doubling it. A manual edit through the web UI stores a bare "250", which is
|
|
// the same string minus the lead-in.
|
|
var chapterLeadIn = regexp.MustCompile(`(?i)^\s*(?:chapter|ch\.?)\s*`)
|
|
|
|
func displayChapter(raw string, num float64) string {
|
|
rest := strings.TrimSpace(chapterLeadIn.ReplaceAllString(raw, ""))
|
|
if rest == "" {
|
|
rest = strconv.FormatFloat(num, 'f', -1, 64)
|
|
}
|
|
// "Ch " only makes sense in front of a number; anything else is a label the
|
|
// site gave us, so pass it through as written.
|
|
if rest[0] < '0' || rest[0] > '9' {
|
|
return rest
|
|
}
|
|
return "Ch " + rest
|
|
}
|
|
|
|
// DisplayChapter is the read-progress line: one canonical "Ch N" whatever
|
|
// format the write came in as.
|
|
func (b Bookmark) DisplayChapter() string {
|
|
return displayChapter(b.LastChapter, b.LastChapterNum)
|
|
}
|
|
|
|
// DisplayLatest is the same for the newest published chapter, which arrives
|
|
// with the same "Chapter N" lead-in from both the userscript and the poller.
|
|
func (b Bookmark) DisplayLatest() string {
|
|
var num float64
|
|
if b.LatestChapterNum != nil {
|
|
num = *b.LatestChapterNum
|
|
}
|
|
return displayChapter(b.LatestChapter, num)
|
|
}
|
|
|
|
// ContinueURL is where the Continue button points: the chapter last read, or
|
|
// the series page when no chapter URL was ever captured.
|
|
func (b Bookmark) ContinueURL() string {
|
|
if b.LastChapterURL != "" {
|
|
return b.LastChapterURL
|
|
}
|
|
return b.SeriesURL
|
|
}
|
|
|
|
// Initial is the monogram the web UI shows in place of a cover when the
|
|
// source site never gave us an og:image. First rune, uppercased; "?" when even
|
|
// the title is missing, so the slot is never empty.
|
|
func (b Bookmark) Initial() string {
|
|
for _, r := range b.Title {
|
|
return strings.ToUpper(string(r))
|
|
}
|
|
return "?"
|
|
}
|
|
|
|
// kaganeCoverRe matches the cover URL kagane's og:image carries, which is what
|
|
// the userscript stores for that site.
|
|
var kaganeCoverRe = regexp.MustCompile(`^https://kagane\.to/api/v2/image/([0-9a-f-]{36})/compressed$`)
|
|
|
|
// CoverURL is the src the web UI puts in an <img>. For every site but kagane
|
|
// that is Cover as stored. kagane serves its images behind a Cloudflare
|
|
// challenge *and* with `cross-origin-resource-policy: same-origin`, so no page
|
|
// on another origin can load one however it asks (verified 2026-08-08); those
|
|
// go through the backend's own proxy instead.
|
|
func (b Bookmark) CoverURL() string {
|
|
if m := kaganeCoverRe.FindStringSubmatch(b.Cover); m != nil {
|
|
return "/img/kagane/" + m[1]
|
|
}
|
|
return b.Cover
|
|
}
|
|
|
|
// Library buckets. A bookmark is in exactly one. This cannot be derived from
|
|
// Site: asurascans serves manga and novels from the same /comics/ path, so the
|
|
// userscript that recorded the page is the only party that knows which.
|
|
const (
|
|
KindManga = "manga"
|
|
KindNovel = "novel"
|
|
)
|
|
|
|
// Lifecycle buckets. A bookmark is in exactly one; favorite is orthogonal.
|
|
const (
|
|
StatusReading = "reading"
|
|
StatusArchived = "archived"
|
|
StatusFinished = "finished"
|
|
)
|
|
|
|
//go:embed migrations/*.sql
|
|
var migrations embed.FS
|
|
|
|
// bookmarkColumns is the only value ever concatenated into query text. It is a
|
|
// compile-time constant; every request value is bound as a parameter. The
|
|
// series-owned fields are joined in from the series table, in scanBookmark
|
|
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
|
|
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover,
|
|
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
|
|
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
|
|
|
|
// seriesColumns is the series row in scanSeries order, used by the poller's
|
|
// due query. latest_checked_at lives only on series — see MarkLatestChecked
|
|
// for why it stays off every client-visible write.
|
|
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
|
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
|
|
|
|
// Owner is the person running the service: the first Reader, seeded at startup
|
|
// so a fresh deployment has a library before anyone logs in. The seed makes
|
|
// sure exactly one readers row matches their Discord ID, carrying the SHA-256
|
|
// of their epoch-0 userscript credential (derived by internal/token). Every
|
|
// other Reader is created by their own first login (EnsureReader).
|
|
type Owner struct {
|
|
DiscordID string
|
|
// TokenHash is the SHA-256 of the epoch-0 credential; the array shape
|
|
// makes it a compile error to store anything that is not a hash.
|
|
TokenHash [32]byte
|
|
}
|
|
|
|
// Store is the Postgres-backed bookmark store.
|
|
type Store struct {
|
|
db *sql.DB
|
|
// ownerID is the seeded owner Reader (issue #22) — the only Reader with
|
|
// administrative reach (revoking another Reader's sessions). Every store
|
|
// method takes a reader id explicitly, so ownership is never implicit.
|
|
ownerID int64
|
|
}
|
|
|
|
// OwnerID returns the seeded owner Reader's id: the administrator, and the
|
|
// Reader every pre-registration bookmark belongs to.
|
|
func (s *Store) OwnerID() int64 { return s.ownerID }
|
|
|
|
// ReaderIDForTokenHash resolves the Reader whose stored credential hash
|
|
// matches, reporting absence with ok=false. The comparison is an equality on
|
|
// the 32-byte SHA-256 of the presented credential — never on the credential
|
|
// itself — and the indexed lookup reveals only whether some Reader matches,
|
|
// which the 401/200 split has to reveal anyway. An attacker's probe is the
|
|
// hash of their guess, so even the index's prefix comparisons leak nothing
|
|
// about the real credential.
|
|
func (s *Store) ReaderIDForTokenHash(hash [32]byte) (int64, bool, error) {
|
|
var id int64
|
|
err := s.db.QueryRow(
|
|
`SELECT id FROM readers WHERE token_sha256 = $1`, hash[:]).Scan(&id)
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return 0, false, nil
|
|
}
|
|
if err != nil {
|
|
return 0, false, fmt.Errorf("reader by token hash: %w", err)
|
|
}
|
|
return id, true, nil
|
|
}
|
|
|
|
// ReaderTokenInfo returns the identity halves a Reader's credential is
|
|
// derived from (internal/token.Token): their Discord id and token epoch. The
|
|
// web UI needs these to rebuild the install URL — the only place a credential
|
|
// is ever produced in plaintext.
|
|
func (s *Store) ReaderTokenInfo(readerID int64) (string, int64, error) {
|
|
var (
|
|
discordID string
|
|
epoch int64
|
|
)
|
|
err := s.db.QueryRow(
|
|
`SELECT discord_id, token_epoch FROM readers WHERE id = $1`, readerID).
|
|
Scan(&discordID, &epoch)
|
|
if err != nil {
|
|
return "", 0, fmt.Errorf("reader %d token info: %w", readerID, err)
|
|
}
|
|
return discordID, epoch, nil
|
|
}
|
|
|
|
// RotateToken bumps a Reader's token epoch and rewrites the stored hash in
|
|
// one statement, so the new hash always matches the new epoch. expectedEpoch
|
|
// is the epoch the caller derived newHash for (ReaderTokenInfo + 1); a
|
|
// concurrent rotation — or an unknown reader — leaves the row untouched and
|
|
// is reported as an error rather than silently succeeding.
|
|
func (s *Store) RotateToken(readerID, expectedEpoch int64, newHash [32]byte) error {
|
|
var epoch int64
|
|
err := s.db.QueryRow(`
|
|
UPDATE readers SET token_epoch = token_epoch + 1, token_sha256 = $3
|
|
WHERE id = $1 AND token_epoch = $2
|
|
RETURNING token_epoch`, readerID, expectedEpoch, newHash[:]).Scan(&epoch)
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return fmt.Errorf("rotate token for reader %d: concurrent rotation or unknown reader", readerID)
|
|
}
|
|
if err != nil {
|
|
return fmt.Errorf("rotate token for reader %d: %w", readerID, err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// EnsureReader returns the Reader registered to discordID, creating the row on
|
|
// first sight. Registration is open to every guild member (issue #27), and the
|
|
// Discord identity is the only thing that decides which Reader a login is: one
|
|
// code path serves the first login and every later one, so a returning Reader
|
|
// can never end up with a second library.
|
|
//
|
|
// epochZeroHash is only used for a brand-new row. An existing row keeps its
|
|
// stored hash untouched, or a login would silently undo a rotation and revive
|
|
// the credential the Reader rotated away from.
|
|
func (s *Store) EnsureReader(discordID string, epochZeroHash [32]byte) (int64, error) {
|
|
var id int64
|
|
// DO UPDATE rather than DO NOTHING because only an updated row is
|
|
// returned by RETURNING; assigning the column to itself is the no-op that
|
|
// makes the existing id come back.
|
|
err := s.db.QueryRow(`
|
|
INSERT INTO readers (discord_id, token_sha256) VALUES ($1, $2)
|
|
ON CONFLICT (discord_id) DO UPDATE SET discord_id = readers.discord_id
|
|
RETURNING id`, discordID, epochZeroHash[:]).Scan(&id)
|
|
if err != nil {
|
|
return 0, fmt.Errorf("ensure reader: %w", err)
|
|
}
|
|
return id, nil
|
|
}
|
|
|
|
// ReaderSummary is one Reader as the owner's administration panel sees them:
|
|
// who they are and how many live sessions they hold. No credential material,
|
|
// hashed or otherwise, is exposed.
|
|
type ReaderSummary struct {
|
|
ID int64
|
|
DiscordID string
|
|
// Sessions counts unexpired session rows — what the owner revokes.
|
|
Sessions int
|
|
}
|
|
|
|
// Readers lists every Reader with their live session count, oldest first, so
|
|
// the owner row (always the oldest) heads the list.
|
|
func (s *Store) Readers() ([]ReaderSummary, error) {
|
|
rows, err := s.db.Query(`
|
|
SELECT r.id, r.discord_id,
|
|
count(sess.id) FILTER (WHERE sess.expires_at > now()) AS sessions
|
|
FROM readers r
|
|
LEFT JOIN sessions sess ON sess.reader_id = r.id
|
|
GROUP BY r.id, r.discord_id
|
|
ORDER BY r.id`)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("query readers: %w", err)
|
|
}
|
|
defer rows.Close()
|
|
|
|
out := []ReaderSummary{}
|
|
for rows.Next() {
|
|
var r ReaderSummary
|
|
if err := rows.Scan(&r.ID, &r.DiscordID, &r.Sessions); err != nil {
|
|
return nil, fmt.Errorf("scan reader: %w", err)
|
|
}
|
|
out = append(out, r)
|
|
}
|
|
return out, rows.Err()
|
|
}
|
|
|
|
// readersMigration is the version that creates the readers table. The owner
|
|
// seed runs between two migrate passes, so that the run-once migration which
|
|
// attaches existing bookmarks (0004) finds the owner row.
|
|
const readersMigration = 3
|
|
|
|
// allMigrations is the migrate() cap that applies every pending version.
|
|
const allMigrations = 0
|
|
|
|
// Open connects to Postgres at url — a libpq connection URL such as
|
|
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — brings its
|
|
// schema up to date, and seeds the owner Reader.
|
|
func Open(url string, owner Owner) (*Store, error) {
|
|
db, err := sql.Open("pgx", url)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("open postgres: %w", err)
|
|
}
|
|
// Schema runs in two passes with the seed between: 0003 creates the
|
|
// readers table, the owner row must exist before 0004 attaches the
|
|
// existing bookmarks to it. Anything past 0004 is applied by the second
|
|
// pass.
|
|
if err := migrate(db, readersMigration); err != nil {
|
|
db.Close()
|
|
return nil, fmt.Errorf("migrate schema: %w", err)
|
|
}
|
|
// The owner row must exist before 0004 attaches the existing bookmarks to
|
|
// it. The hash refresh is a separate statement after all migrations: the
|
|
// token_epoch column 0006 adds does not exist yet at this point, and the
|
|
// refresh only ever concerns rows that have never been rotated.
|
|
if err := seedOwner(db, owner); err != nil {
|
|
db.Close()
|
|
return nil, fmt.Errorf("seed owner: %w", err)
|
|
}
|
|
if err := migrate(db, allMigrations); err != nil {
|
|
db.Close()
|
|
return nil, fmt.Errorf("migrate: %w", err)
|
|
}
|
|
if err := refreshOwnerToken(db, owner); err != nil {
|
|
db.Close()
|
|
return nil, fmt.Errorf("refresh owner token: %w", err)
|
|
}
|
|
var ownerID int64
|
|
if err := db.QueryRow(
|
|
`SELECT id FROM readers WHERE discord_id = $1`, owner.DiscordID).Scan(&ownerID); err != nil {
|
|
db.Close()
|
|
return nil, fmt.Errorf("resolve owner: %w", err)
|
|
}
|
|
return &Store{db: db, ownerID: ownerID}, nil
|
|
}
|
|
|
|
// seedOwner makes sure the configured owner exists as exactly one readers row.
|
|
// The hash is only ever written here for a brand-new row; existing rows keep
|
|
// what they have until refreshOwnerToken decides otherwise, so the seed can
|
|
// never clobber a rotation.
|
|
func seedOwner(db *sql.DB, o Owner) error {
|
|
if _, err := db.Exec(`
|
|
INSERT INTO readers (discord_id, token_sha256) VALUES ($1, $2)
|
|
ON CONFLICT (discord_id) DO NOTHING`,
|
|
o.DiscordID, o.TokenHash[:]); err != nil {
|
|
return fmt.Errorf("seed owner: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// refreshOwnerToken brings a never-rotated owner row's hash current with the
|
|
// configured credential. That is the cutover path: a database seeded under
|
|
// the retired global token still carries its hash at epoch 0, and the
|
|
// epoch-0 derivation is the caller's TokenHash. A rotated row (epoch > 0) is
|
|
// left alone — a restart must not resurrect the old credential by
|
|
// overwriting the hash a rotation wrote.
|
|
func refreshOwnerToken(db *sql.DB, o Owner) error {
|
|
if _, err := db.Exec(`
|
|
UPDATE readers SET token_sha256 = $2
|
|
WHERE discord_id = $1 AND token_epoch = 0`,
|
|
o.DiscordID, o.TokenHash[:]); err != nil {
|
|
return fmt.Errorf("refresh owner token: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// migrate applies every embedded migration this database has not recorded, in
|
|
// filename order, each in its own transaction. upto caps the highest version
|
|
// applied; 0 means all. Files are named "<version>_<name>.sql" and are
|
|
// append-only: editing an applied file changes nothing, because
|
|
// schema_migrations is how a database remembers what it ran. Runs on every
|
|
// start and is a no-op once current.
|
|
func migrate(db *sql.DB, upto int64) error {
|
|
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
|
|
version bigint PRIMARY KEY,
|
|
applied_at timestamptz NOT NULL DEFAULT now())`); err != nil {
|
|
return fmt.Errorf("create version table: %w", err)
|
|
}
|
|
|
|
names, err := fs.Glob(migrations, "migrations/*.sql")
|
|
if err != nil {
|
|
return err
|
|
}
|
|
slices.Sort(names)
|
|
|
|
for _, name := range names {
|
|
version, err := strconv.ParseInt(strings.SplitN(path.Base(name), "_", 2)[0], 10, 64)
|
|
if err != nil {
|
|
return fmt.Errorf("migration %q: filename must start with a version number", name)
|
|
}
|
|
if upto > 0 && version > upto {
|
|
continue
|
|
}
|
|
body, err := migrations.ReadFile(name)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := applyMigration(db, version, string(body)); err != nil {
|
|
return fmt.Errorf("migration %q: %w", name, err)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// applyMigration runs one migration and records its version in the same
|
|
// transaction, so an interrupted start leaves neither half behind.
|
|
func applyMigration(db *sql.DB, version int64, body string) error {
|
|
tx, err := db.Begin()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer tx.Rollback()
|
|
|
|
var applied bool
|
|
if err := tx.QueryRow(
|
|
`SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1)`,
|
|
version).Scan(&applied); err != nil {
|
|
return err
|
|
}
|
|
if applied {
|
|
return nil
|
|
}
|
|
// No parameters, so this goes over the simple protocol and a migration may
|
|
// hold more than one statement.
|
|
if _, err := tx.Exec(body); err != nil {
|
|
return err
|
|
}
|
|
if _, err := tx.Exec(`INSERT INTO schema_migrations (version) VALUES ($1)`, version); err != nil {
|
|
return err
|
|
}
|
|
return tx.Commit()
|
|
}
|
|
|
|
// scanBookmark reads one row in bookmarkColumns order. Every column is NOT
|
|
// NULL except latest_chapter_num, where NULL means "never captured" — a
|
|
// distinct state from chapter zero, and the reason for the pointer.
|
|
func scanBookmark(scan func(...any) error) (Bookmark, error) {
|
|
var (
|
|
b Bookmark
|
|
latestChapterNum sql.NullFloat64
|
|
)
|
|
if err := scan(
|
|
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover,
|
|
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
|
|
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
|
|
); err != nil {
|
|
return Bookmark{}, err
|
|
}
|
|
if latestChapterNum.Valid {
|
|
b.LatestChapterNum = &latestChapterNum.Float64
|
|
}
|
|
// The wire identity is derived: there is no stored key column, the
|
|
// bookmark is keyed (reader_id, site, series_id) (issue #22).
|
|
b.Key = b.Site + ":" + b.SeriesID
|
|
// An unrecognised bucket (a hand-edited row) would leave the row in no list
|
|
// at all, so anything outside the three known buckets reads as the default
|
|
// rather than being passed through.
|
|
if b.Status != StatusReading && b.Status != StatusArchived && b.Status != StatusFinished {
|
|
b.Status = StatusReading
|
|
}
|
|
return b, nil
|
|
}
|
|
|
|
// scanSeries reads one row in seriesColumns order, plus the due query's
|
|
// reader_count column. latest_chapter_num is NULL until the first capture,
|
|
// same as on the bookmark read path.
|
|
func scanSeries(scan func(...any) error) (Series, error) {
|
|
var (
|
|
sr Series
|
|
latestChapterNum sql.NullFloat64
|
|
)
|
|
if err := scan(
|
|
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
|
|
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
|
|
&sr.readerCount,
|
|
); err != nil {
|
|
return Series{}, err
|
|
}
|
|
if latestChapterNum.Valid {
|
|
sr.LatestChapterNum = &latestChapterNum.Float64
|
|
}
|
|
return sr, nil
|
|
}
|
|
|
|
// Close releases the underlying database handle.
|
|
func (s *Store) Close() error { return s.db.Close() }
|
|
|
|
// List returns every bookmark of one reader, newest activity first.
|
|
// Series-owned fields are joined in, so each Bookmark reads back whole and
|
|
// flat (ADR-0004).
|
|
func (s *Store) List(readerID int64) ([]Bookmark, error) {
|
|
rows, err := s.db.Query(`SELECT `+bookmarkColumns+`
|
|
FROM bookmarks b
|
|
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
|
WHERE b.reader_id = $1
|
|
ORDER BY b.updated_at DESC`, readerID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("query bookmarks: %w", err)
|
|
}
|
|
defer rows.Close()
|
|
|
|
out := []Bookmark{}
|
|
for rows.Next() {
|
|
b, err := scanBookmark(rows.Scan)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("scan bookmark: %w", err)
|
|
}
|
|
out = append(out, b)
|
|
}
|
|
return out, rows.Err()
|
|
}
|
|
|
|
// Get returns one bookmark of one reader by key. A missing key is not an
|
|
// error: ok is false and err is nil. UI mutations read-modify-write through
|
|
// this so they preserve the fields they do not touch.
|
|
func (s *Store) Get(readerID int64, key string) (Bookmark, bool, error) {
|
|
site, seriesID, ok := strings.Cut(key, ":")
|
|
if !ok {
|
|
return Bookmark{}, false, nil
|
|
}
|
|
b, err := scanBookmark(s.db.QueryRow(
|
|
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
|
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
|
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
|
|
readerID, site, seriesID).Scan)
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return Bookmark{}, false, nil
|
|
}
|
|
if err != nil {
|
|
return Bookmark{}, false, fmt.Errorf("get %q: %w", key, err)
|
|
}
|
|
return b, true, nil
|
|
}
|
|
|
|
// Upsert inserts or replaces one reader's bookmark by key (last-write-wins)
|
|
// and returns the row as actually stored — one flat object with the
|
|
// series-owned fields joined in, exactly as GET reports it (ADR-0004). A
|
|
// bookmark is keyed (reader_id, site, series_id), so the same key upserts two
|
|
// independent rows for two readers.
|
|
//
|
|
// The flat body is decomposed across two tables in one transaction. The series
|
|
// row is written first (the bookmarks FK requires it to exist), then the
|
|
// bookmark row. On the series side, title/series_url/cover are applied only
|
|
// when the row is brand new: once a series exists, client-supplied values are
|
|
// ignored, because the row is shared and the values are scraped page content —
|
|
// see ADR-0003. Kind and the latest-chapter fields are last-write-wins.
|
|
//
|
|
// b.UpdatedAt is only a candidate: it is applied when the row is new or when
|
|
// last_chapter_num changes, and otherwise the stored value is kept. Clients
|
|
// order their list by updated_at, so favoriting a series or recording a newly
|
|
// published chapter must not disturb that order — only real reading progress
|
|
// does. Callers must therefore use the returned bookmark, not the argument.
|
|
func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
|
|
tx, err := s.db.Begin()
|
|
if err != nil {
|
|
return Bookmark{}, fmt.Errorf("begin %q: %w", b.Key, err)
|
|
}
|
|
defer tx.Rollback()
|
|
|
|
var latestNum any
|
|
if b.LatestChapterNum != nil {
|
|
latestNum = *b.LatestChapterNum
|
|
}
|
|
|
|
// The kind column resolves on the VALUES side, not in the conflict clause:
|
|
// excluded.* is the row *after* these expressions are evaluated, so a
|
|
// default applied there would look identical to a real 'manga' and would
|
|
// overwrite a novel series on every PUT from a client that knows nothing
|
|
// about the column. Resolved once here, an empty incoming kind means "keep
|
|
// what is stored", and only a brand-new row falls through to the literal
|
|
// default. The subquery runs inside this transaction, so it sees the row
|
|
// this statement is about to conflict with. Same pattern as the status
|
|
// COALESCE on the bookmark insert below.
|
|
//
|
|
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no
|
|
// target column to infer the parameter type from, and Postgres rejects the
|
|
// statement rather than guessing.
|
|
if _, err := tx.Exec(`
|
|
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
|
latest_chapter, latest_chapter_num)
|
|
VALUES ($1, $2, $3, $4, $5,
|
|
COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
|
|
$7, $8)
|
|
ON CONFLICT (site, series_id) DO UPDATE SET
|
|
kind=excluded.kind,
|
|
latest_chapter=excluded.latest_chapter,
|
|
latest_chapter_num=excluded.latest_chapter_num`,
|
|
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
|
|
b.LatestChapter, latestNum); err != nil {
|
|
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
|
|
}
|
|
|
|
// IS DISTINCT FROM is Postgres's null-safe comparison, and it is what
|
|
// implements the ordering rule. Within DO UPDATE, a bare column is the
|
|
// stored row and excluded.* is the incoming one; a brand-new key never
|
|
// reaches this clause, so it keeps the fresh timestamp from VALUES.
|
|
if _, err := tx.Exec(`
|
|
INSERT INTO bookmarks (reader_id, site, series_id, last_chapter, last_chapter_num,
|
|
last_chapter_url, favorite, status, updated_at)
|
|
VALUES ($1, $2, $3, $4, $5, $6, $7,
|
|
COALESCE(NULLIF($8::text, ''), (SELECT status FROM bookmarks WHERE reader_id = $1 AND site = $2 AND series_id = $3), 'reading'),
|
|
$9)
|
|
ON CONFLICT (reader_id, site, series_id) DO UPDATE SET
|
|
last_chapter=excluded.last_chapter, last_chapter_num=excluded.last_chapter_num,
|
|
last_chapter_url=excluded.last_chapter_url,
|
|
favorite=excluded.favorite,
|
|
status=excluded.status,
|
|
updated_at=CASE
|
|
WHEN bookmarks.last_chapter_num IS DISTINCT FROM excluded.last_chapter_num
|
|
THEN excluded.updated_at
|
|
ELSE bookmarks.updated_at
|
|
END`,
|
|
readerID, b.Site, b.SeriesID,
|
|
b.LastChapter, b.LastChapterNum, b.LastChapterURL,
|
|
b.Favorite, b.Status, b.UpdatedAt); err != nil {
|
|
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
|
|
}
|
|
|
|
stored, err := scanBookmark(tx.QueryRow(
|
|
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
|
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
|
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
|
|
readerID, b.Site, b.SeriesID).Scan)
|
|
if err != nil {
|
|
return Bookmark{}, fmt.Errorf("read back %q: %w", b.Key, err)
|
|
}
|
|
if err := tx.Commit(); err != nil {
|
|
return Bookmark{}, fmt.Errorf("commit %q: %w", b.Key, err)
|
|
}
|
|
return stored, nil
|
|
}
|
|
|
|
// Delete removes one reader's bookmark by key. Deleting a missing key is not
|
|
// an error.
|
|
func (s *Store) Delete(readerID int64, key string) error {
|
|
site, seriesID, ok := strings.Cut(key, ":")
|
|
if !ok {
|
|
return nil
|
|
}
|
|
if _, err := s.db.Exec(
|
|
`DELETE FROM bookmarks WHERE reader_id = $1 AND site = $2 AND series_id = $3`,
|
|
readerID, site, seriesID); err != nil {
|
|
return fmt.Errorf("delete %q: %w", key, err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// DueForLatestCheck returns series whose server-side latest-chapter check has
|
|
// aged past the appropriate cutoff, ordered by how many bookmarks reference
|
|
// them (descending) then least-recently-checked first, at most limit of them.
|
|
// Browser-backed sites use browserCutoffMs; every other site uses cutoffMs.
|
|
//
|
|
// The reader_count ordering is the point of the split (ADR-0003): a series
|
|
// shared by several readers is fetched once per due cycle, and the popular
|
|
// ones stay freshest while the long tail absorbs any shortfall. Within one
|
|
// reader count, oldest-first keeps the poll fair when the backlog outgrows
|
|
// throughput: the most neglected series is always next, so a large collection
|
|
// refreshes uniformly slower rather than leaving a tail that never refreshes
|
|
// at all. The userscript sorts its own queue the same way (L453).
|
|
//
|
|
// Series with no series_url are skipped — there is nothing to fetch, which is
|
|
// the same filter the userscript applies at L452. Series whose only bookmarks
|
|
// are finished are skipped too: nothing more is coming, so fetching them only
|
|
// burns requests. Archived bookmarks still count — knowing what a shelved
|
|
// series is up to is the whole reason for archiving instead of deleting.
|
|
// A series with no bookmarks at all never appears: the join excludes it.
|
|
func (s *Store) DueForLatestCheck(cutoffMs, browserCutoffMs int64, browserSites []string, limit int) ([]Series, error) {
|
|
rows, err := s.db.Query(`SELECT `+seriesColumns+`, COUNT(*) AS reader_count
|
|
FROM series s
|
|
JOIN bookmarks b ON b.site = s.site AND b.series_id = s.series_id
|
|
WHERE s.series_url <> ''
|
|
AND s.latest_checked_at <= CASE
|
|
WHEN s.site = ANY($3::text[]) THEN $2::bigint
|
|
ELSE $1::bigint
|
|
END
|
|
GROUP BY s.site, s.series_id, s.title, s.series_url, s.cover,
|
|
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at
|
|
HAVING COUNT(*) FILTER (WHERE b.status <> 'finished') > 0
|
|
ORDER BY reader_count DESC, s.latest_checked_at ASC
|
|
LIMIT $4`, cutoffMs, browserCutoffMs, pgtype.FlatArray[string](browserSites), limit)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("query due series: %w", err)
|
|
}
|
|
defer rows.Close()
|
|
|
|
out := []Series{}
|
|
for rows.Next() {
|
|
sr, err := scanSeries(rows.Scan)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("scan due series: %w", err)
|
|
}
|
|
out = append(out, sr)
|
|
}
|
|
return out, rows.Err()
|
|
}
|
|
|
|
// MarkLatestChecked records that the server looked at a series at ts, whatever
|
|
// the look turned up. Marking a missing series is not an error: the row may
|
|
// have been orphaned while a fetch was in flight.
|
|
//
|
|
// This is the one write that does not go through Upsert, and the column is kept
|
|
// out of the client-visible read path on purpose. PUT /bookmarks/{key} decodes
|
|
// a whole Bookmark from the client and Upsert writes every series column it
|
|
// knows about, so a userscript PUT — which has no idea this field exists —
|
|
// would write a zero and reset the cooldown, making the poller re-fetch that
|
|
// series every tick for as long as the user kept reading it.
|
|
func (s *Store) MarkLatestChecked(site, seriesID string, ts int64) error {
|
|
if _, err := s.db.Exec(
|
|
`UPDATE series SET latest_checked_at = $1 WHERE site = $2 AND series_id = $3`,
|
|
ts, site, seriesID); err != nil {
|
|
return fmt.Errorf("mark checked %s:%s: %w", site, seriesID, err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// LatestCheckedAt reads the column MarkLatestChecked writes. It exists for
|
|
// tests outside this package (the poller's own tests assert on cooldown
|
|
// bookkeeping) — see MarkLatestChecked for why the field stays off the
|
|
// client-visible row.
|
|
func (s *Store) LatestCheckedAt(site, seriesID string) (int64, error) {
|
|
var ts int64
|
|
if err := s.db.QueryRow(
|
|
`SELECT latest_checked_at FROM series WHERE site = $1 AND series_id = $2`,
|
|
site, seriesID).Scan(&ts); err != nil {
|
|
return 0, fmt.Errorf("latest checked at %s:%s: %w", site, seriesID, err)
|
|
}
|
|
return ts, nil
|
|
}
|
|
|
|
// SetLatestChapter records the newest chapter the poll found on a series page.
|
|
// The poller walks Series rather than Bookmarks, so this is a series-level
|
|
// write: the row is shared, and updating it once refreshes every bookmark that
|
|
// joins to it. Touching a missing series is not an error.
|
|
func (s *Store) SetLatestChapter(site, seriesID, label string, num float64) error {
|
|
if _, err := s.db.Exec(
|
|
`UPDATE series SET latest_chapter = $3, latest_chapter_num = $4
|
|
WHERE site = $1 AND series_id = $2`,
|
|
site, seriesID, label, num); err != nil {
|
|
return fmt.Errorf("set latest chapter %s:%s: %w", site, seriesID, err)
|
|
}
|
|
return nil
|
|
}
|