Closes #59. Part of spec #55, and the ticket that fixes the reported bug #47. Architecture: `docs/adr/0007-backend-hosts-cover-bytes.md`. Does not close #47 or #55. ## What changed A Reader bookmarks a Series nobody holds yet — the exact case in #47 — and within seconds the list shows its artwork instead of a broken image. The first Bookmark to create a Series fires `Store.OnSeriesCreated` after commit, and the new `latest.Acquirer` turns that into **one** series-page fetch that yields both the Latest Chapter and the cover URL. The bytes go through the gated cover fetcher from #57 and are stored content-addressed through #56, so the wire carries an absolute URL on this deployment's own origin — never a third-party address, and never one that 404s. ### Store - Migration `0009_series_cover_address.sql` adds `series.cover_address`. The two facts are now split: `series.cover` is the third-party source address the bytes came from (the acquisition path's dedupe key), `series.cover_address` is the SHA-256 they are stored under. An empty `cover_address` is precisely what "no Cover yet" means, which is the distinction both the API and the UI depend on. - `SetSeriesCover` writes the address only after the bytes are on disk, so the wire can never name an object that is not there. - `CoverWireURL` builds `PUBLIC_BASE_URL + /covers/<sha256>` for every scanned row, and returns `""` for a blank address. - The cover columns are gone from `Upsert`'s `INSERT` and its `DO UPDATE`. A client-supplied cover cannot reach the shared Series row on any path, not just the creation path. - `Open` now rejects a base URL that is not an absolute `http(s)` origin: `PUBLIC_BASE_URL=bookmarks.example.com` would otherwise start cleanly and emit addresses no browser can load. ### Acquisition - `internal/latest/acquire.go`: one fetch, gated by the poller's own `fetchableSeriesURL` (a `series_url` arrives in a client-supplied PUT body, so without the gate a token-holder chooses what the server fetches from its own network position). - Asynchronous and log-and-drop. The Bookmark, its progress and its Latest Chapter are already committed; a Site that is down or a cover that cannot be produced disturbs none of them. - Bounded by a two-slot semaphore. A bulk sync creating N Series would otherwise fire N simultaneous requests from one IP — the traffic shape the poller's stagger exists to avoid. - Cancelled at shutdown (shares the poller's context) and stamps `latest_checked_at`, so the poller does not refetch the same page a tick later. - Browser-backed Sites (kagane, novelfull) are deliberately skipped: their pages only yield a Cloudflare challenge to the TLS client, so the request would be spent for nothing. They arrive in #62. ### Wire and route - `GET /covers/{address}` serves the bytes publicly and uncredentialed with `Cache-Control: public, max-age=604800, immutable`. The address is gated by a `^[0-9a-f]{64}$` pattern and cross-checked against a pure function of itself before any filesystem read, so no request shaped like a traversal reaches disk. - `PUT /bookmarks/{key}` still accepts a `cover` field and discards it, permanently. Rejecting it would break every installed userscript the moment this deploys, and ADR-0004's compatibility argument depends on those scripts continuing to work. The decode site says so in place of a TODO nobody intends to keep. - `store.CoverContentType` canonicalises comix's non-standard `image/jpg` to `image/jpeg`, so one image cannot land under two spellings. This one was found by the live smoke test, not by reading. ### Config `PUBLIC_BASE_URL` is new and required (cover URLs must go out absolute — the userscript renders them on third-party origins, where a relative path resolves against the Site). Documented in `.env.example`, `docker-compose.yml` (`:?` so compose fails too), `DEPLOY.md` and `backend/AGENTS.md`. ## Acceptance criteria All twelve of #59's criteria are met; the checklist on the issue is ticked with the evidence. ## Verification - `go test ./...` green (Docker-backed Postgres suite). - Live smoke against a real backend + Postgres: bookmarking `comix:n8we-dungeons-and-crayons` produced `"cover": "http://127.0.0.1:8099/covers/8ce74d80…"` and `"latest_chapter": "Chapter 81"` within seconds of the PUT; `curl` on that address returned `200`, `Content-Type: image/jpeg`, `Cache-Control: public, max-age=604800, immutable`, and a 280x420 JPEG. That run is what surfaced the `image/jpg` content type. - Mutation-checked the asynchrony test: removing the `go` from `Acquire` turns `TestAcquireDoesNotBlockTheWrite` red. ## Reviewed Both axes of `/code-review` were run against this diff before commit. Their findings that were actionable here are folded in: the concurrency bound, the shutdown tie, the `PUBLIC_BASE_URL` validation, the missing `latest_checked_at` stamp, and a test that could not fail. ## Known sequencing A kagane/novelfull Series created between this deploy and #62 has no cover source at all: the acquisition skips those Sites and `Upsert` no longer persists the userscript-scraped address. This is #59's stated boundary rather than a defect, but it is a user-visible gap on two Sites and should order #62 accordingly. Reviewed-on: #68 Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com> Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
This commit was merged in pull request #68.
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
-- The Cover splits into two facts. `cover` keeps the third-party address the
|
||||
-- bytes come from, which is what the acquisition path refetches and dedupes
|
||||
-- on; `cover_address` is the content address of the bytes once they are
|
||||
-- actually stored, and is what the wire's absolute URL is built from.
|
||||
--
|
||||
-- Empty `cover_address` therefore means "no Cover yet" rather than "a Cover
|
||||
-- that 404s", which is the distinction the API and the UI both depend on.
|
||||
ALTER TABLE series ADD COLUMN cover_address text NOT NULL DEFAULT '';
|
||||
+152
-40
@@ -30,12 +30,20 @@ import (
|
||||
// between readers: progress, favourite, lifecycle bucket, updated_at. The wire
|
||||
// format stays flat regardless — see ADR-0004.
|
||||
type Bookmark struct {
|
||||
Key string `json:"key"`
|
||||
Site string `json:"site"`
|
||||
SeriesID string `json:"series_id"`
|
||||
Title string `json:"title"`
|
||||
SeriesURL string `json:"series_url"`
|
||||
Cover string `json:"cover"`
|
||||
Key string `json:"key"`
|
||||
Site string `json:"site"`
|
||||
SeriesID string `json:"series_id"`
|
||||
Title string `json:"title"`
|
||||
SeriesURL string `json:"series_url"`
|
||||
// Cover is the wire value: an absolute URL on this deployment's own
|
||||
// origin once the bytes exist, and "" until they do — never a third-party
|
||||
// address and never an address that 404s (ADR-0007). A client may still
|
||||
// send this field and it is discarded on the way in; see Upsert.
|
||||
Cover string `json:"cover"`
|
||||
// CoverSource is the third-party address the bytes were fetched from. It
|
||||
// stays off the wire: it is the acquisition path's dedupe key, and no
|
||||
// client is ever asked to render one.
|
||||
CoverSource string `json:"-"`
|
||||
LastChapter string `json:"last_chapter"`
|
||||
LastChapterNum float64 `json:"last_chapter_num"`
|
||||
LastChapterURL string `json:"last_chapter_url"`
|
||||
@@ -62,11 +70,16 @@ type Bookmark struct {
|
||||
// bookmark's own fields. Never serialized: the wire format is the flat
|
||||
// Bookmark (ADR-0004).
|
||||
type Series struct {
|
||||
Site string
|
||||
SeriesID string
|
||||
Title string
|
||||
SeriesURL string
|
||||
Site string
|
||||
SeriesID string
|
||||
Title string
|
||||
SeriesURL string
|
||||
// Cover is the third-party source address the bytes come from, and
|
||||
// CoverAddress the content address they are stored under. A blank
|
||||
// CoverAddress is what "no Cover yet" means: the poll fills it and never
|
||||
// replaces a filled one (ADR-0007).
|
||||
Cover string
|
||||
CoverAddress string
|
||||
Kind string
|
||||
LatestChapter string
|
||||
LatestChapterNum *float64 // nil until first captured
|
||||
@@ -156,23 +169,27 @@ func KaganeImageID(cover string) (string, bool) {
|
||||
return m[1], true
|
||||
}
|
||||
|
||||
// IsCoverContentType reports whether a fetched response is safe to store and serve.
|
||||
func IsCoverContentType(contentType string) bool {
|
||||
// CoverContentType canonicalises a fetched response's media type and reports
|
||||
// whether the bytes are safe to store and serve. comix answers "image/jpg",
|
||||
// which no standard lists but browsers accept; it is stored as the real name
|
||||
// rather than passed through, so one image never lands under two spellings.
|
||||
func CoverContentType(contentType string) (string, bool) {
|
||||
switch contentType {
|
||||
case "image/jpg":
|
||||
return "image/jpeg", true
|
||||
case "image/webp", "image/jpeg", "image/png", "image/avif", "image/gif":
|
||||
return true
|
||||
return contentType, true
|
||||
default:
|
||||
return false
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// CoverURL is the src the web UI puts in an <img>. For every site but kagane
|
||||
// that is Cover as stored. kagane serves its images behind a Cloudflare
|
||||
// challenge *and* with `cross-origin-resource-policy: same-origin`, so no page
|
||||
// on another origin can load one however it asks (verified 2026-08-08); those
|
||||
// go through the backend's own proxy instead.
|
||||
// CoverURL is the src the web UI puts in an <img>. Cover already is an address
|
||||
// on this origin, so for every site but kagane it is used as-is. kagane's
|
||||
// bytes still arrive through the browser-backed proxy, which is keyed by image
|
||||
// id rather than by content address until #62 moves it onto the same path.
|
||||
func (b Bookmark) CoverURL() string {
|
||||
if imageID, ok := KaganeImageID(b.Cover); ok {
|
||||
if imageID, ok := KaganeImageID(b.CoverSource); ok {
|
||||
return "/img/kagane/" + imageID
|
||||
}
|
||||
return b.Cover
|
||||
@@ -200,14 +217,14 @@ var migrations embed.FS
|
||||
// compile-time constant; every request value is bound as a parameter. The
|
||||
// series-owned fields are joined in from the series table, in scanBookmark
|
||||
// order, so the flat Bookmark reads back whole despite the split (ADR-0004).
|
||||
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover,
|
||||
const bookmarkColumns = `b.site, b.series_id, s.title, s.series_url, s.cover, s.cover_address,
|
||||
b.last_chapter, b.last_chapter_num, b.last_chapter_url,
|
||||
b.favorite, s.latest_chapter, s.latest_chapter_num, b.updated_at, b.status, s.kind`
|
||||
|
||||
// seriesColumns is the series row in scanSeries order, used by the poller's
|
||||
// due query. latest_checked_at lives only on series — see MarkLatestChecked
|
||||
// for why it stays off every client-visible write.
|
||||
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover,
|
||||
const seriesColumns = `s.site, s.series_id, s.title, s.series_url, s.cover, s.cover_address,
|
||||
s.kind, s.latest_chapter, s.latest_chapter_num, s.latest_checked_at`
|
||||
|
||||
// Owner is the person running the service: the first Reader, seeded at startup
|
||||
@@ -230,6 +247,16 @@ type Store struct {
|
||||
// method takes a reader id explicitly, so ownership is never implicit.
|
||||
ownerID int64
|
||||
coverDir string
|
||||
// coverBaseURL is this deployment's public origin. Cover addresses are
|
||||
// absolute because the userscript renders them on third-party origins,
|
||||
// where a relative path would resolve against the Site (ADR-0007).
|
||||
coverBaseURL string
|
||||
// OnSeriesCreated fires once, after commit, for a Series no Reader had
|
||||
// bookmarked before. It is how creation-time Cover and Latest Chapter
|
||||
// acquisition is triggered without the write waiting on a third-party
|
||||
// Site; nil disables it, which is what every test that does not care
|
||||
// about acquisition leaves it as.
|
||||
OnSeriesCreated func(Series)
|
||||
}
|
||||
|
||||
// OwnerID returns the seeded owner Reader's id: the administrator, and the
|
||||
@@ -365,10 +392,19 @@ const allMigrations = 0
|
||||
// Open connects to Postgres at url — a libpq connection URL such as
|
||||
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — brings its
|
||||
// schema up to date, seeds the owner Reader, and prepares cover storage.
|
||||
func Open(url string, owner Owner, coverDir string) (*Store, error) {
|
||||
func Open(url string, owner Owner, coverDir, coverBaseURL string) (*Store, error) {
|
||||
if strings.TrimSpace(coverDir) == "" {
|
||||
return nil, errors.New("cover directory is required")
|
||||
}
|
||||
// Every wire Cover is this string with a path glued on, rendered by a
|
||||
// userscript on a Site's own origin: anything but an absolute origin
|
||||
// produces addresses no client can load, silently (ADR-0007).
|
||||
base := strings.TrimRight(coverBaseURL, "/")
|
||||
if host, ok := strings.CutPrefix(base, "https://"); !ok || host == "" {
|
||||
if host, ok := strings.CutPrefix(base, "http://"); !ok || host == "" {
|
||||
return nil, fmt.Errorf("cover base URL %q is not an absolute http(s) origin", coverBaseURL)
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(coverDir, 0o755); err != nil {
|
||||
return nil, fmt.Errorf("create cover directory: %w", err)
|
||||
}
|
||||
@@ -414,7 +450,9 @@ func Open(url string, owner Owner, coverDir string) (*Store, error) {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("resolve owner: %w", err)
|
||||
}
|
||||
return &Store{db: db, ownerID: ownerID, coverDir: coverDir}, nil
|
||||
return &Store{
|
||||
db: db, ownerID: ownerID, coverDir: coverDir, coverBaseURL: base,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// seedOwner makes sure the configured owner exists as exactly one readers row.
|
||||
@@ -517,18 +555,20 @@ func applyMigration(db *sql.DB, version int64, body string) error {
|
||||
// scanBookmark reads one row in bookmarkColumns order. Every column is NOT
|
||||
// NULL except latest_chapter_num, where NULL means "never captured" — a
|
||||
// distinct state from chapter zero, and the reason for the pointer.
|
||||
func scanBookmark(scan func(...any) error) (Bookmark, error) {
|
||||
func (s *Store) scanBookmark(scan func(...any) error) (Bookmark, error) {
|
||||
var (
|
||||
b Bookmark
|
||||
coverAddress string
|
||||
latestChapterNum sql.NullFloat64
|
||||
)
|
||||
if err := scan(
|
||||
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.Cover,
|
||||
&b.Site, &b.SeriesID, &b.Title, &b.SeriesURL, &b.CoverSource, &coverAddress,
|
||||
&b.LastChapter, &b.LastChapterNum, &b.LastChapterURL,
|
||||
&b.Favorite, &b.LatestChapter, &latestChapterNum, &b.UpdatedAt, &b.Status, &b.Kind,
|
||||
); err != nil {
|
||||
return Bookmark{}, err
|
||||
}
|
||||
b.Cover = s.CoverWireURL(coverAddress)
|
||||
if latestChapterNum.Valid {
|
||||
b.LatestChapterNum = &latestChapterNum.Float64
|
||||
}
|
||||
@@ -553,7 +593,7 @@ func scanSeries(scan func(...any) error) (Series, error) {
|
||||
latestChapterNum sql.NullFloat64
|
||||
)
|
||||
if err := scan(
|
||||
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover,
|
||||
&sr.Site, &sr.SeriesID, &sr.Title, &sr.SeriesURL, &sr.Cover, &sr.CoverAddress,
|
||||
&sr.Kind, &sr.LatestChapter, &latestChapterNum, &sr.LatestCheckedAt,
|
||||
&sr.readerCount,
|
||||
); err != nil {
|
||||
@@ -582,7 +622,10 @@ func kaganeCoverSourceURL(imageID string) string {
|
||||
}
|
||||
|
||||
func (s *Store) getCover(sourceURL string) ([]byte, string, bool, error) {
|
||||
address := coverSourceAddress(sourceURL)
|
||||
return s.getCoverByAddress(coverSourceAddress(sourceURL))
|
||||
}
|
||||
|
||||
func (s *Store) getCoverByAddress(address string) ([]byte, string, bool, error) {
|
||||
var relativePath, contentType string
|
||||
err := s.db.QueryRow(
|
||||
`SELECT path, content_type FROM covers WHERE address = $1`, address,
|
||||
@@ -608,9 +651,11 @@ func (s *Store) getCover(sourceURL string) ([]byte, string, bool, error) {
|
||||
}
|
||||
|
||||
func (s *Store) putCover(sourceURL string, body []byte, contentType string) error {
|
||||
if !IsCoverContentType(contentType) {
|
||||
stored, ok := CoverContentType(contentType)
|
||||
if !ok {
|
||||
return fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
|
||||
}
|
||||
contentType = stored
|
||||
address := coverSourceAddress(sourceURL)
|
||||
relativePath := coverRelativePath(address)
|
||||
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(relativePath))
|
||||
@@ -670,6 +715,55 @@ func (s *Store) PutKaganeCover(imageID string, body []byte, contentType string)
|
||||
return s.putCover(kaganeCoverSourceURL(imageID), body, contentType)
|
||||
}
|
||||
|
||||
// CoverAddress is the content address bytes fetched from sourceURL are stored
|
||||
// under. It is a pure function of the URL, so the acquisition path can name a
|
||||
// Cover before it has the bytes.
|
||||
func CoverAddress(sourceURL string) string { return coverSourceAddress(sourceURL) }
|
||||
|
||||
// coverAddressRe is the shape of a stored address: the hex SHA-256 of a source
|
||||
// URL. Request paths reach CoverByAddress, so the shape is checked before the
|
||||
// value is ever turned into a filesystem path.
|
||||
var coverAddressRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||
|
||||
// CoverByAddress returns the immutable object at one content address. An
|
||||
// address that is not a stored one - malformed, unknown, or recorded but with
|
||||
// its file gone - is reported with ok=false rather than as an error.
|
||||
func (s *Store) CoverByAddress(address string) ([]byte, string, bool, error) {
|
||||
if !coverAddressRe.MatchString(address) {
|
||||
return nil, "", false, nil
|
||||
}
|
||||
return s.getCoverByAddress(address)
|
||||
}
|
||||
|
||||
// CoverWireURL is the absolute URL a client renders for a stored Cover, and ""
|
||||
// for a Series that has none yet. A blank is a real state, not a placeholder
|
||||
// address: it is what tells both clients to draw their own fallback instead of
|
||||
// requesting bytes that do not exist (ADR-0007).
|
||||
func (s *Store) CoverWireURL(address string) string {
|
||||
if address == "" {
|
||||
return ""
|
||||
}
|
||||
return s.coverBaseURL + "/covers/" + address
|
||||
}
|
||||
|
||||
// SetSeriesCover stores the bytes and points the Series at them, but only
|
||||
// while the Series has no Cover: acquisition at creation and the poll both
|
||||
// call this, and whichever arrives second must not overwrite the first. The
|
||||
// bytes themselves are content-addressed and immutable, so storing them twice
|
||||
// is free.
|
||||
func (s *Store) SetSeriesCover(site, seriesID, sourceURL string, body []byte, contentType string) error {
|
||||
if err := s.putCover(sourceURL, body, contentType); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := s.db.Exec(`
|
||||
UPDATE series SET cover = $3, cover_address = $4
|
||||
WHERE site = $1 AND series_id = $2 AND cover_address = ''`,
|
||||
site, seriesID, sourceURL, coverSourceAddress(sourceURL)); err != nil {
|
||||
return fmt.Errorf("set cover for %q: %w", site+":"+seriesID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// List returns every bookmark of one reader, newest activity first.
|
||||
// Series-owned fields are joined in, so each Bookmark reads back whole and
|
||||
// flat (ADR-0004).
|
||||
@@ -686,7 +780,7 @@ func (s *Store) List(readerID int64) ([]Bookmark, error) {
|
||||
|
||||
out := []Bookmark{}
|
||||
for rows.Next() {
|
||||
b, err := scanBookmark(rows.Scan)
|
||||
b, err := s.scanBookmark(rows.Scan)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("scan bookmark: %w", err)
|
||||
}
|
||||
@@ -703,7 +797,7 @@ func (s *Store) Get(readerID int64, key string) (Bookmark, bool, error) {
|
||||
if !ok {
|
||||
return Bookmark{}, false, nil
|
||||
}
|
||||
b, err := scanBookmark(s.db.QueryRow(
|
||||
b, err := s.scanBookmark(s.db.QueryRow(
|
||||
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
||||
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
|
||||
@@ -760,18 +854,28 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
|
||||
// The ::text casts are load-bearing: inside COALESCE/NULLIF there is no
|
||||
// target column to infer the parameter type from, and Postgres rejects the
|
||||
// statement rather than guessing.
|
||||
if _, err := tx.Exec(`
|
||||
INSERT INTO series (site, series_id, title, series_url, cover, kind,
|
||||
//
|
||||
// The cover columns are absent on purpose: the Cover is acquired
|
||||
// server-side (ADR-0007), so a client-supplied one is not written even
|
||||
// when the row is brand new.
|
||||
//
|
||||
// xmax is zero only on a row this statement inserted, which is how a
|
||||
// Series nobody had bookmarked before is told apart from one that already
|
||||
// existed — DO UPDATE returns a row either way.
|
||||
var created bool
|
||||
if err := tx.QueryRow(`
|
||||
INSERT INTO series (site, series_id, title, series_url, kind,
|
||||
latest_chapter, latest_chapter_num)
|
||||
VALUES ($1, $2, $3, $4, $5,
|
||||
COALESCE(NULLIF($6::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
|
||||
$7, $8)
|
||||
VALUES ($1, $2, $3, $4,
|
||||
COALESCE(NULLIF($5::text, ''), (SELECT kind FROM series WHERE site = $1 AND series_id = $2), 'manga'),
|
||||
$6, $7)
|
||||
ON CONFLICT (site, series_id) DO UPDATE SET
|
||||
kind=excluded.kind,
|
||||
latest_chapter=excluded.latest_chapter,
|
||||
latest_chapter_num=excluded.latest_chapter_num`,
|
||||
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Cover, b.Kind,
|
||||
b.LatestChapter, latestNum); err != nil {
|
||||
latest_chapter_num=excluded.latest_chapter_num
|
||||
RETURNING xmax = 0`,
|
||||
b.Site, b.SeriesID, b.Title, b.SeriesURL, b.Kind,
|
||||
b.LatestChapter, latestNum).Scan(&created); err != nil {
|
||||
return Bookmark{}, fmt.Errorf("upsert series for %q: %w", b.Key, err)
|
||||
}
|
||||
|
||||
@@ -801,7 +905,7 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
|
||||
return Bookmark{}, fmt.Errorf("upsert %q: %w", b.Key, err)
|
||||
}
|
||||
|
||||
stored, err := scanBookmark(tx.QueryRow(
|
||||
stored, err := s.scanBookmark(tx.QueryRow(
|
||||
`SELECT `+bookmarkColumns+` FROM bookmarks b
|
||||
JOIN series s ON s.site = b.site AND s.series_id = b.series_id
|
||||
WHERE b.reader_id = $1 AND b.site = $2 AND b.series_id = $3`,
|
||||
@@ -812,6 +916,14 @@ func (s *Store) Upsert(readerID int64, b Bookmark) (Bookmark, error) {
|
||||
if err := tx.Commit(); err != nil {
|
||||
return Bookmark{}, fmt.Errorf("commit %q: %w", b.Key, err)
|
||||
}
|
||||
// After commit, never inside the transaction: the hook reaches a
|
||||
// third-party Site, and the Reader's write must not wait on it.
|
||||
if created && s.OnSeriesCreated != nil {
|
||||
s.OnSeriesCreated(Series{
|
||||
Site: b.Site, SeriesID: b.SeriesID, Title: stored.Title,
|
||||
SeriesURL: stored.SeriesURL, Kind: stored.Kind,
|
||||
})
|
||||
}
|
||||
return stored, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -21,9 +21,12 @@ func TestMain(m *testing.M) { os.Exit(pgtest.Main(m)) }
|
||||
// reader register one (see secondReader).
|
||||
var testOwner = Owner{DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("owner-token-hash"))}
|
||||
|
||||
// testCoverBaseURL is the public origin every stored cover URL is built from.
|
||||
const testCoverBaseURL = "https://bookmarks.test"
|
||||
|
||||
func newTestStore(t *testing.T) *Store {
|
||||
t.Helper()
|
||||
store, err := Open(pgtest.URL(t), testOwner, t.TempDir())
|
||||
store, err := Open(pgtest.URL(t), testOwner, t.TempDir(), testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -49,7 +52,7 @@ func secondReader(t *testing.T, s *Store) int64 {
|
||||
func TestOpenIsIdempotent(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, testOwner, coverDir)
|
||||
first, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -60,7 +63,7 @@ func TestOpenIsIdempotent(t *testing.T) {
|
||||
}
|
||||
first.Close()
|
||||
|
||||
second, err := Open(url, testOwner, coverDir)
|
||||
second, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -113,7 +116,7 @@ func TestReaderTokenInfo(t *testing.T) {
|
||||
func TestRotateTokenInvalidatesOldAndSurvivesRestart(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
store, err := Open(url, testOwner, coverDir)
|
||||
store, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -145,7 +148,7 @@ func TestRotateTokenInvalidatesOldAndSurvivesRestart(t *testing.T) {
|
||||
}
|
||||
store.Close()
|
||||
|
||||
reopened, err := Open(url, testOwner, coverDir)
|
||||
reopened, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -547,32 +550,39 @@ func TestDisplayChapter(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// CoverURL reads the source address for the kagane branch and the wire value
|
||||
// otherwise, so both are set the way scanBookmark sets them.
|
||||
func TestCoverURL(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
cover string
|
||||
want string
|
||||
name string
|
||||
coverSource string
|
||||
cover string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
"kagane routes through the proxy",
|
||||
"https://kagane.to/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed",
|
||||
"https://bookmarks.test/covers/" + CoverAddress("kagane"),
|
||||
"/img/kagane/019fe11a-84c3-7fc3-a84b-88787374b617",
|
||||
},
|
||||
{
|
||||
"another site is served as stored",
|
||||
"https://gg.asuracomic.net/storage/media/1/conversions/cover.webp",
|
||||
"another site is served from our own origin",
|
||||
"https://gg.asuracomic.net/storage/media/1/conversions/cover.webp",
|
||||
"https://bookmarks.test/covers/" + CoverAddress("asura"),
|
||||
"https://bookmarks.test/covers/" + CoverAddress("asura"),
|
||||
},
|
||||
{
|
||||
"a lookalike host is not rewritten",
|
||||
"https://evil.example/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed",
|
||||
"https://evil.example/api/v2/image/019fe11a-84c3-7fc3-a84b-88787374b617/compressed",
|
||||
"https://bookmarks.test/covers/" + CoverAddress("evil"),
|
||||
"https://bookmarks.test/covers/" + CoverAddress("evil"),
|
||||
},
|
||||
{"no cover stays empty", "", ""},
|
||||
{"no cover stays empty", "", "", ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := (Bookmark{Cover: tc.cover}).CoverURL(); got != tc.want {
|
||||
b := Bookmark{CoverSource: tc.coverSource, Cover: tc.cover}
|
||||
if got := b.CoverURL(); got != tc.want {
|
||||
t.Errorf("CoverURL() = %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
@@ -672,7 +682,7 @@ func TestMigration0002BackfillsExistingBookmarks(t *testing.T) {
|
||||
// Bring it current through the production path: Open runs the schema to
|
||||
// 0003, seeds the owner, then applies 0004 which attaches this row. 0002
|
||||
// must have backfilled the series row, not lost data.
|
||||
st, err := Open(url, testOwner, t.TempDir())
|
||||
st, err := Open(url, testOwner, t.TempDir(), testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open after migrate: %v", err)
|
||||
}
|
||||
@@ -756,39 +766,143 @@ func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
|
||||
return sr
|
||||
}
|
||||
|
||||
// The first PUT for a series creates its row from the client's title, cover
|
||||
// and URL — there is no other source for them (ADR-0003).
|
||||
// The first PUT for a series creates its row from the client's title and URL —
|
||||
// there is no other source for them (ADR-0003). The Cover is not among them:
|
||||
// it is acquired server-side, so a client-supplied one is dropped even on a
|
||||
// brand-new row (ADR-0007).
|
||||
func TestUpsertCreatesSeriesFromClient(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
if _, err := store.Upsert(store.OwnerID(), Bookmark{
|
||||
stored, err := store.Upsert(store.OwnerID(), Bookmark{
|
||||
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
|
||||
Cover: "https://asurascans.com/covers/solo.jpg", Kind: KindManga,
|
||||
UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Upsert: %v", err)
|
||||
}
|
||||
if stored.Cover != "" {
|
||||
t.Fatalf("Cover = %q, want empty — a client cover is never stored", stored.Cover)
|
||||
}
|
||||
|
||||
sr := readSeries(t, store, "asura", "solo")
|
||||
if sr.Title != "Solo Leveling" || sr.SeriesURL != "https://asurascans.com/comics/solo" ||
|
||||
sr.Cover != "https://asurascans.com/covers/solo.jpg" {
|
||||
t.Fatalf("series = %+v, want client title/url/cover stored", sr)
|
||||
if sr.Title != "Solo Leveling" || sr.SeriesURL != "https://asurascans.com/comics/solo" {
|
||||
t.Fatalf("series = %+v, want client title/url stored", sr)
|
||||
}
|
||||
if sr.Cover != "" {
|
||||
t.Fatalf("series cover = %q, want empty", sr.Cover)
|
||||
}
|
||||
}
|
||||
|
||||
// A PUT naming an existing series must not overwrite its title, cover or URL:
|
||||
// the row is shared, and those values are scraped page content (ADR-0003).
|
||||
// The hook is what starts creation-time acquisition, so it must fire exactly
|
||||
// once per Series — on the PUT that created it, and on no later one, whichever
|
||||
// Reader sends it.
|
||||
func TestOnSeriesCreatedFiresOnceForANewSeries(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
var created []Series
|
||||
store.OnSeriesCreated = func(sr Series) { created = append(created, sr) }
|
||||
|
||||
b := Bookmark{
|
||||
Key: "comix:solo", Site: "comix", SeriesID: "solo", Title: "Solo Leveling",
|
||||
SeriesURL: "https://comix.to/series/solo", Kind: KindManga, UpdatedAt: 1000,
|
||||
}
|
||||
if _, err := store.Upsert(store.OwnerID(), b); err != nil {
|
||||
t.Fatalf("Upsert: %v", err)
|
||||
}
|
||||
b.LastChapterNum = 12
|
||||
b.UpdatedAt = 2000
|
||||
if _, err := store.Upsert(store.OwnerID(), b); err != nil {
|
||||
t.Fatalf("second Upsert: %v", err)
|
||||
}
|
||||
if _, err := store.Upsert(secondReader(t, store), b); err != nil {
|
||||
t.Fatalf("second reader Upsert: %v", err)
|
||||
}
|
||||
|
||||
if len(created) != 1 {
|
||||
t.Fatalf("hook fired %d times, want 1: %+v", len(created), created)
|
||||
}
|
||||
if created[0].Site != "comix" || created[0].SeriesID != "solo" ||
|
||||
created[0].SeriesURL != "https://comix.to/series/solo" {
|
||||
t.Fatalf("hook got %+v, want the created series' identity and URL", created[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Acquisition at creation and the poll both write covers, and whichever
|
||||
// arrives second must leave the first one alone: a Cover is replaced by
|
||||
// nothing short of the series row being rebuilt.
|
||||
func TestSetSeriesCoverDoesNotOverwrite(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
if _, err := store.Upsert(store.OwnerID(), Bookmark{
|
||||
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
first := "https://asurascans.com/covers/first.jpg"
|
||||
if err := store.SetSeriesCover("asura", "solo", first, []byte("first"), "image/jpeg"); err != nil {
|
||||
t.Fatalf("SetSeriesCover: %v", err)
|
||||
}
|
||||
if err := store.SetSeriesCover("asura", "solo", "https://asurascans.com/covers/second.jpg",
|
||||
[]byte("second"), "image/jpeg"); err != nil {
|
||||
t.Fatalf("second SetSeriesCover: %v", err)
|
||||
}
|
||||
|
||||
got, ok, err := store.Get(store.OwnerID(), "asura:solo")
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("Get = %v, %v", ok, err)
|
||||
}
|
||||
if want := "https://bookmarks.test/covers/" + CoverAddress(first); got.Cover != want {
|
||||
t.Fatalf("Cover = %q, want the first one %q", got.Cover, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The address comes straight off a public request path, so anything that is
|
||||
// not a stored address must be a miss rather than a filesystem lookup.
|
||||
func TestCoverByAddress(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
if _, err := store.Upsert(store.OwnerID(), Bookmark{
|
||||
Key: "asura:solo", Site: "asura", SeriesID: "solo", UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
source := "https://asurascans.com/covers/solo.jpg"
|
||||
if err := store.SetSeriesCover("asura", "solo", source, []byte("bytes"), "image/jpeg"); err != nil {
|
||||
t.Fatalf("SetSeriesCover: %v", err)
|
||||
}
|
||||
|
||||
body, contentType, ok, err := store.CoverByAddress(CoverAddress(source))
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("CoverByAddress = %v, %v", ok, err)
|
||||
}
|
||||
if string(body) != "bytes" || contentType != "image/jpeg" {
|
||||
t.Fatalf("CoverByAddress = %q, %q, want the stored bytes", body, contentType)
|
||||
}
|
||||
|
||||
for _, address := range []string{"", "../../etc/passwd", "ZZ" + CoverAddress(source)[2:],
|
||||
CoverAddress("never stored")} {
|
||||
_, _, ok, err := store.CoverByAddress(address)
|
||||
if err != nil || ok {
|
||||
t.Fatalf("CoverByAddress(%q) = %v, %v, want a clean miss", address, ok, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A PUT naming an existing series must not overwrite its title or URL: the row
|
||||
// is shared, and those values are scraped page content (ADR-0003). An acquired
|
||||
// Cover is likewise untouched by any client.
|
||||
func TestUpsertExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
base := Bookmark{
|
||||
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||
Title: "Solo Leveling", SeriesURL: "https://asurascans.com/comics/solo",
|
||||
Cover: "https://asurascans.com/covers/solo.jpg", LastChapterNum: 10,
|
||||
UpdatedAt: 1000,
|
||||
LastChapterNum: 10, UpdatedAt: 1000,
|
||||
}
|
||||
if _, err := store.Upsert(store.OwnerID(), base); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
acquired := "https://asurascans.com/covers/solo.jpg"
|
||||
if err := store.SetSeriesCover("asura", "solo", acquired, []byte("bytes"), "image/jpeg"); err != nil {
|
||||
t.Fatalf("SetSeriesCover: %v", err)
|
||||
}
|
||||
|
||||
// Same series, hostile/compromised values, real progress advance.
|
||||
base.Title = "Scraped Rename"
|
||||
@@ -799,8 +913,9 @@ func TestUpsertExistingSeriesIgnoresClientTitleCoverURL(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Upsert: %v", err)
|
||||
}
|
||||
wantCover := "https://bookmarks.test/covers/" + CoverAddress(acquired)
|
||||
if got.Title != "Solo Leveling" || got.SeriesURL != "https://asurascans.com/comics/solo" ||
|
||||
got.Cover != "https://asurascans.com/covers/solo.jpg" {
|
||||
got.Cover != wantCover {
|
||||
t.Fatalf("stored = %+v, want original title/url/cover kept", got)
|
||||
}
|
||||
if got.LastChapterNum != 11 {
|
||||
@@ -836,16 +951,19 @@ func TestUpsertExistingSeriesAcceptsKindAndLatest(t *testing.T) {
|
||||
}
|
||||
|
||||
// Deleting the last bookmark must leave the series row behind, so a later
|
||||
// re-bookmark shows title and cover immediately instead of waiting for a poll.
|
||||
// re-bookmark shows title and cover immediately instead of re-acquiring them.
|
||||
func TestDeleteKeepsSeriesRow(t *testing.T) {
|
||||
store := newTestStore(t)
|
||||
if _, err := store.Upsert(store.OwnerID(), Bookmark{
|
||||
Key: "asura:solo", Site: "asura", SeriesID: "solo",
|
||||
Title: "Solo Leveling", Cover: "https://asurascans.com/covers/solo.jpg",
|
||||
UpdatedAt: 1000,
|
||||
Title: "Solo Leveling", UpdatedAt: 1000,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
acquired := "https://asurascans.com/covers/solo.jpg"
|
||||
if err := store.SetSeriesCover("asura", "solo", acquired, []byte("bytes"), "image/jpeg"); err != nil {
|
||||
t.Fatalf("SetSeriesCover: %v", err)
|
||||
}
|
||||
if err := store.Delete(store.OwnerID(), "asura:solo"); err != nil {
|
||||
t.Fatalf("Delete: %v", err)
|
||||
}
|
||||
@@ -863,7 +981,8 @@ func TestDeleteKeepsSeriesRow(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("re-upsert: %v", err)
|
||||
}
|
||||
if stored.Title != "Solo Leveling" || stored.Cover != "https://asurascans.com/covers/solo.jpg" {
|
||||
wantCover := "https://bookmarks.test/covers/" + CoverAddress(acquired)
|
||||
if stored.Title != "Solo Leveling" || stored.Cover != wantCover {
|
||||
t.Fatalf("re-bookmark = %+v, want title/cover from the surviving series row", stored)
|
||||
}
|
||||
}
|
||||
@@ -940,14 +1059,14 @@ func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
|
||||
func TestSeedOwnerIdempotentAndRefreshesTokenHash(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v1"))}, coverDir)
|
||||
first, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v1"))}, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
ownerID := first.OwnerID()
|
||||
first.Close()
|
||||
|
||||
second, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v2"))}, coverDir)
|
||||
second, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v2"))}, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -1004,7 +1123,7 @@ func TestMigration0004AttachesBookmarksToOwner(t *testing.T) {
|
||||
t.Fatalf("migrate to 0002: %v", err)
|
||||
}
|
||||
|
||||
st, err := Open(url, testOwner, t.TempDir())
|
||||
st, err := Open(url, testOwner, t.TempDir(), testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -1281,7 +1400,7 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
|
||||
func TestKaganeCoverPersistsAcrossReopen(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, testOwner, coverDir)
|
||||
first, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -1293,7 +1412,7 @@ func TestKaganeCoverPersistsAcrossReopen(t *testing.T) {
|
||||
t.Fatalf("close first store: %v", err)
|
||||
}
|
||||
|
||||
second, err := Open(url, testOwner, coverDir)
|
||||
second, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -1308,15 +1427,26 @@ func TestKaganeCoverPersistsAcrossReopen(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestOpenRequiresCoverDirectory(t *testing.T) {
|
||||
if _, err := Open(pgtest.URL(t), testOwner, ""); err == nil || !strings.Contains(err.Error(), "cover directory is required") {
|
||||
if _, err := Open(pgtest.URL(t), testOwner, "", testCoverBaseURL); err == nil || !strings.Contains(err.Error(), "cover directory is required") {
|
||||
t.Fatalf("Open without cover directory = %v, want required-directory error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A base URL without a scheme reads like a hostname and starts cleanly, but
|
||||
// every Cover it puts on the wire is an address no browser can resolve.
|
||||
func TestOpenRequiresAbsoluteCoverBaseURL(t *testing.T) {
|
||||
for _, base := range []string{"", "bookmarks.test", "https://", "ftp://bookmarks.test"} {
|
||||
if _, err := Open(pgtest.URL(t), testOwner, t.TempDir(), base); err == nil ||
|
||||
!strings.Contains(err.Error(), "absolute http(s) origin") {
|
||||
t.Fatalf("Open with base %q = %v, want absolute-origin error", base, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestKaganeCoverIsContentAddressedOnFilesystem(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, testOwner, coverDir)
|
||||
first, err := Open(url, testOwner, coverDir, testCoverBaseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user