sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

A newly bookmarked Series acquires its Cover at creation (#59) (#68)

Browse Source 
Closes #59.

Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55.

## What changed

A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s.

### Store

- Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on.
- `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there.
- `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address.
- The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path.
- `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load.

### Acquisition

- `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position).
- Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them.
- Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid.
- Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later.
- Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62.

### Wire and route

- `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk.
- `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep.
- `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading.

### Config

`PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`.

## Acceptance criteria

All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence.

## Verification

- `go test ./...` green (Docker-backed Postgres suite).
- Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type.
- Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red.

## Reviewed

Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail.

## Known sequencing

A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly.

Reviewed-on: #68
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
This commit was merged in pull request #68.
This commit is contained in:
Sulthan Zaki
2026-08-10 04:07:53 +07:00
committed by sulthan
parent b6b88bde8a
commit 92eba07da7
18 changed files with 971 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/internal/store/store.go
+152 -40
View File
@@ -30,12 +30,20 @@ import (
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
// format stays flat regardless — see ADR-0004.
type Bookmark struct {
Key string `json:"key"`
Site string `json:"site"`
SeriesID string `json:"series_id"`
Title string `json:"title"`
SeriesURL string `json:"series_url"`
Cover string `json:"cover"`
Key string `json:"key"`
Site string `json:"site"`
SeriesID string `json:"series_id"`
Title string `json:"title"`
SeriesURL string `json:"series_url"`
// Cover is the wire value: an absolute URL on this deployment's own
// origin once the bytes exist, and "" until they do — never a third-party
// address and never an address that 404s (ADR-0007). A client may still
// send this field and it is discarded on the way in; see Upsert.
Cover string `json:"cover"`
// CoverSource is the third-party address the bytes were fetched from. It
// stays off the wire: it is the acquisition path's dedupe key, and no
// client is ever asked to render one.
CoverSource string `json:"-"`
LastChapter string `json:"last_chapter"`
LastChapterNum float64 `json:"last_chapter_num"`
LastChapterURL string `json:"last_chapter_url"`
@@ -62,11 +70,16 @@ type Bookmark struct {
// bookmark's own fields. Never serialized: the wire format is the flat
// Bookmark (ADR-0004).
type Series struct {
Site string
SeriesID string
Title string
SeriesURL string
Site string
SeriesID string
Title string
SeriesURL string
// Cover is the third-party source address the bytes come from, and
// CoverAddress the content address they are stored under. A blank
// CoverAddress is what "no Cover yet" means: the poll fills it and never
// replaces a filled one (ADR-0007).
Cover string
CoverAddress string
Kind string
LatestChapter string
LatestChapterNum *float64 // nil until first captured
@@ -156,23 +169,27 @@ func KaganeImageID(cover string) (string, bool) {
return m[1], true
}
// IsCoverContentType reports whether a fetched response is safe to store and serve.
func IsCoverContentType(contentType string) bool {
// CoverContentType canonicalises a fetched response's media type and reports
// whether the bytes are safe to store and serve. comix answers "image/jpg",
// which no standard lists but browsers accept; it is stored as the real name
// rather than passed through, so one image never lands under two spellings.
func CoverContentType(contentType string) (string, bool) {
switch contentType {
case "image/jpg":
return "image/jpeg", true
case "image/webp", "image/jpeg", "image/png", "image/avif", "image/gif":
return true
return contentType, true
default:
return false
return "", false
}
}
// CoverURL is the src the web UI puts in an <img>. For every site but kagane
// that is Cover as stored. kagane serves its images behind a Cloudflare
// challenge *and* with `cross-origin-resource-policy: same-origin`, so no page
// on another origin can load one however it asks (verified 2026-08-08); those
// go through the backend's own proxy instead.
// CoverURL is the src the web UI puts in an <img>. Cover already is an address
// on this origin, so for every site but kagane it is used as-is. kagane's
// bytes still arrive through the browser-backed proxy, which is keyed by image
// id rather than by content address until #62 moves it onto the same path.
func (b Bookmark) CoverURL() string {
if imageID, ok := KaganeImageID(b.Cover); ok {
if imageID, ok := KaganeImageID(b.CoverSource); ok {
return "/img/kagane/" + imageID
}
return b.Cover
@@ -200,14 +217,14 @@ var migrations embed.FS
// compile-time constant; every request value is bound as a parameter. The
// series-owned fields are joined in from the series table, in scanBookmark
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover,
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover, s.cover_address,
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
// seriesColumns is the series row in scanSeries order, used by the poller's
// due query. latest_checked_at lives only on series — see MarkLatestChecked
// for why it stays off every client-visible write.
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover, s.cover_address,
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
// Owner is the person running the service: the first Reader, seeded at startup
@@ -230,6 +247,16 @@ type Store struct {
// method takes a reader id explicitly, so ownership is never implicit.
ownerID int64
coverDir string
// coverBaseURL is this deployment's public origin. Cover addresses are
// absolute because the userscript renders them on third-party origins,
// where a relative path would resolve against the Site (ADR-0007).
coverBaseURL string
// OnSeriesCreated fires once, after commit, for a Series no Reader had
// bookmarked before. It is how creation-time Cover and Latest Chapter
// acquisition is triggered without the write waiting on a third-party
// Site; nil disables it, which is what every test that does not care
// about acquisition leaves it as.
OnSeriesCreated func(Series)
}
// OwnerID returns the seeded owner Reader's id: the administrator, and the
@@ -365,10 +392,19 @@ const allMigrations = 0
// Open connects to Postgres at url — a libpq connection URL such as
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — brings its
// schema up to date, seeds the owner Reader, and prepares cover storage.
func Open(url string, owner Owner, coverDir string) (*Store, error) {
func Open(url string, owner Owner, coverDir, coverBaseURL string) (*Store, error) {
if strings.TrimSpace(coverDir) == "" {
return nil, errors.New("cover directory is required")
}
// Every wire Cover is this string with a path glued on, rendered by a
// userscript on a Site's own origin: anything but an absolute origin
// produces addresses no client can load, silently (ADR-0007).
base := strings.TrimRight(coverBaseURL, "/")
if host, ok := strings.CutPrefix(base, "https://"); !ok || host == "" {
if host, ok := strings.CutPrefix(base, "http://"); !ok || host == "" {
return nil, fmt.Errorf("cover base URL %q is not an absolute http(s) origin", coverBaseURL)
}
}
if err := os.MkdirAll(coverDir, 0o755); err != nil {
return nil, fmt.Errorf("create cover directory: %w", err)
}
@@ -414,7 +450,9 @@ func Open(url string, owner Owner, coverDir string) (*Store, error) {
db.Close()
return nil, fmt.Errorf("resolve owner: %w", err)
}
return &Store{db: db, ownerID: ownerID, coverDir: coverDir}, nil
return &Store{
db: db, ownerID: ownerID, coverDir: coverDir, coverBaseURL: base,
}, nil
}
// seedOwner makes sure the configured owner exists as exactly one readers row.
@@ -517,18 +555,20 @@ func applyMigration(db *sql.DB, version int64, body string) error {
// scanBookmark reads one row in bookmarkColumns order. Every column is NOT
// NULL except latest_chapter_num, where NULL means "never captured" — a
// distinct state from chapter zero, and the reason for the pointer.
func scanBookmark(scan func(...any) error) (Bookmark, error) {
func (s *Store) scanBookmark(scan func(...any) error) (Bookmark, error) {
var (
b Bookmark
coverAddress string
latestChapterNum sql.NullFloat64
)
if err := scan(
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover,
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.CoverSource, &coverAddress,
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
); err != nil {
return Bookmark{}, err
}
b.Cover = s.CoverWireURL(coverAddress)
if latestChapterNum.Valid {
b.LatestChapterNum = &latestChapterNum.Float64
}
@@ -553,7 +593,7 @@ func scanSeries(scan func(...any) error) (Series, error) {
latestChapterNum sql.NullFloat64
)
if err := scan(
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover, &sr.CoverAddress,
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
&sr.readerCount,
); err != nil {
@@ -582,7 +622,10 @@ func kaganeCoverSourceURL(imageID string) string {
}
func (s *Store) getCover(sourceURL string) ([]byte, string, bool, error) {
address := coverSourceAddress(sourceURL)
return s.getCoverByAddress(coverSourceAddress(sourceURL))
}
func (s *Store) getCoverByAddress(address string) ([]byte, string, bool, error) {
var relativePath, contentType string
err := s.db.QueryRow(
`SELECT path, content_type FROM covers WHERE address = $1`, address,
@@ -608,9 +651,11 @@ func (s *Store) getCover(sourceURL string) ([]byte, string, bool, error) {
}
func (s *Store) putCover(sourceURL string, body []byte, contentType string) error {
if !IsCoverContentType(contentType) {
stored, ok := CoverContentType(contentType)
if !ok {
return fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
}
contentType = stored
address := coverSourceAddress(sourceURL)
relativePath := coverRelativePath(address)
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(relativePath))
@@ -670,6 +715,55 @@ func (s *Store) PutKaganeCover(imageID string, body []byte, contentType string)
return s.putCover(kaganeCoverSourceURL(imageID), body, contentType)
}
// CoverAddress is the content address bytes fetched from sourceURL are stored
// under. It is a pure function of the URL, so the acquisition path can name a
// Cover before it has the bytes.
func CoverAddress(sourceURL string) string { return coverSourceAddress(sourceURL) }
// coverAddressRe is the shape of a stored address: the hex SHA-256 of a source
// URL. Request paths reach CoverByAddress, so the shape is checked before the
// value is ever turned into a filesystem path.
var coverAddressRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
// CoverByAddress returns the immutable object at one content address. An
// address that is not a stored one - malformed, unknown, or recorded but with
// its file gone - is reported with ok=false rather than as an error.
func (s *Store) CoverByAddress(address string) ([]byte, string, bool, error) {
if !coverAddressRe.MatchString(address) {
return nil, "", false, nil
}
return s.getCoverByAddress(address)
}
// CoverWireURL is the absolute URL a client renders for a stored Cover, and ""
// for a Series that has none yet. A blank is a real state, not a placeholder
// address: it is what tells both clients to draw their own fallback instead of
// requesting bytes that do not exist (ADR-0007).
func (s *Store) CoverWireURL(address string) string {
if address == "" {
return ""
}
return s.coverBaseURL + "/covers/" + address
}
// SetSeriesCover stores the bytes and points the Series at them, but only
// while the Series has no Cover: acquisition at creation and the poll both
// call this, and whichever arrives second must not overwrite the first. The
// bytes themselves are content-addressed and immutable, so storing them twice
// is free.
func (s *Store) SetSeriesCover(site, seriesID, sourceURL string, body []byte, contentType string) error {
if err := s.putCover(sourceURL, body, contentType); err != nil {
return err
}
if _, err := s.db.Exec(`
UPDATE series SET cover = $3, cover_address = $4
WHERE site = $1 AND series_id = $2 AND cover_address = ''`,
site, seriesID, sourceURL, coverSourceAddress(sourceURL)); err != nil {
return fmt.Errorf("set cover for %q: %w", site+":"+seriesID, err)
}
return nil
}
// List returns every bookmark of one reader, newest activity first.
// Series-owned fields are joined in, so each Bookmark reads back whole and
// flat (ADR-0004).
@@ -686,7 +780,7 @@ func (s *Store) List(readerID int64) ([]Bookmark, error) {
out := []Bookmark{}
for rows.Next() {
b, err := scanBookmark(rows.Scan)
b, err := s.scanBookmark(rows.Scan)
if err != nil {
return nil, fmt.Errorf("scan bookmark: %w", err)
}
@@ -703,7 +797,7 @@ func (s *Store) Get(readerID int64, key string) (Bookmark, bool, error) {
if !ok {
return Bookmark{}, false, nil
}
b, err := scanBookmark(s.db.QueryRow(
b, err := s.scanBookmark(s.db.QueryRow(
`SELECT `+bookmarkColumns+` FROM bookmarks b
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
@@ -760,18 +854,28 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no
// target column to infer the parameter type from, and Postgres rejects the
// statement rather than guessing.
if _, err := tx.Exec(`
INSERT INTO series (site, series_id, title, series_url, cover, kind,
//
// The cover columns are absent on purpose: the Cover is acquired
// server-side (ADR-0007), so a client-supplied one is not written even
// when the row is brand new.
//
// xmax is zero only on a row this statement inserted, which is how a
// Series nobody had bookmarked before is told apart from one that already
// existed — DO UPDATE returns a row either way.
var created bool
if err := tx.QueryRow(`
INSERT INTO series (site, series_id, title, series_url, kind,
latest_chapter, latest_chapter_num)
VALUES ($1, $2, $3, $4, $5,
COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
$7, $8)
VALUES ($1, $2, $3, $4,
COALESCE(NULLIF($5::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
$6, $7)
ON CONFLICT (site, series_id) DO UPDATE SET
kind=excluded.kind,
latest_chapter=excluded.latest_chapter,
latest_chapter_num=excluded.latest_chapter_num`,
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
b.LatestChapter, latestNum); err != nil {
latest_chapter_num=excluded.latest_chapter_num
RETURNING xmax = 0`,
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Kind,
b.LatestChapter, latestNum).Scan(&created); err != nil {
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
}
@@ -801,7 +905,7 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
}
stored, err := scanBookmark(tx.QueryRow(
stored, err := s.scanBookmark(tx.QueryRow(
`SELECT `+bookmarkColumns+` FROM bookmarks b
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
@@ -812,6 +916,14 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
if err := tx.Commit(); err != nil {
return Bookmark{}, fmt.Errorf("commit %q: %w", b.Key, err)
}
// After commit, never inside the transaction: the hook reaches a
// third-party Site, and the Reader's write must not wait on it.
if created && s.OnSeriesCreated != nil {
s.OnSeriesCreated(Series{
Site: b.Site, SeriesID: b.SeriesID, Title: stored.Title,
SeriesURL: stored.SeriesURL, Kind: stored.Kind,
})
}
return stored, nil
}