sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

The Poll fills blank Covers for every Site and both Libraries (#70)

Browse Source 
Closes #61.

## Summary

Permanently-blank Series (the half of #47 that creation-time acquisition cannot reach) heal on the next due poll cycle. The cover path is no longer kagane-only: every Site and both Libraries fill a blank Cover from the series page the chapter poll already fetched, and never replace a Cover that already exists.

## What changed

### `backend/internal/latest/poller.go`

- **`fillBlankCover`** — when `Cover` and `CoverAddress` are both blank, extract a source URL via `coverFrom` from the series-page body and store bytes through `SetSeriesCover`. Skips any Series that already has a source URL (owned by prefetch) or a stored address (never overwrite).
- **`prefetchCover`** — source-URL healing path, now site-uniform. Kagane no longer special-cases into `PutKaganeCover` alone; every Site lands on `SetSeriesCover`, so the wire Cover becomes a content-addressed public URL. Reuses already-stored bytes when present.
- **`storeCover` / `fetchCoverBytes`** — shared fetch+persist. Only kagane routes image bytes through the browser fetcher; every other Site uses plain TLS `CoverBytesFetch`. Failures log with the Series key and never return to the chapter path.
- **`checkOne`** — after a successful series-page fetch, calls `fillBlankCover` once regardless of whether chapter extraction succeeded (cover fill is independent of the chapter signal).

### `backend/internal/latest/poller_test.go`

Extended the existing poller harness (real store, fake fetchers) rather than a new one:

- `TestRunOnceFillsBlankCoverFromSeriesPage` — asura manga, lightnovelworld novel, kagane manga; asserts wire Cover + correct fetcher routing.
- `TestRunOnceDoesNotReplaceExistingCover` — second poll does not refetch.
- `TestRunOnceRetriesFailedBlankCoverOnNextPoll` — failed fill stays blank, next due cycle retries (no separate queue).
- `TestRunOnceBlankCoverFailureDoesNotBlockChapter` — chapter still lands; failure log carries the Series key.
- Kagane prefetch test now also asserts the content-addressed wire Cover.

## Acceptance criteria (#61)

| Criterion | Status |
|---|---|
| Cover prefetch runs for every Site | done |
| Cover prefetch runs for both Libraries | done |
| Poll fills a blank Cover | done |
| Poll never replaces an existing Cover | done |
| Failed cover fetch does not fail/block chapter poll | done |
| Failed cover fetch retried next poll, no separate queue | done |
| Failures logged with the Series | done |
| Existing poller tests extended | done |
| `go test ./...` green | done |
| Manually verified: blank Series gets Cover after a poll cycle | **left for you** |

## Out of scope / not closed

- Does **not** close #47 or #55 (per ticket).
- No migration/backfill script — the Poll walks every Series already.
- No admin refetch (#54).

## Review notes addressed

- Removed the kagane-only `PutKaganeCover` branch from prefetch so source-URL healing also sets `CoverAddress` (wire Cover).
- Guard so `fillBlankCover` does not double-fetch after `prefetchCover` healed the same snapshot.
- Single `fillBlankCover` call site after the series-page fetch.

## Test plan

- [x] `go test ./...` (backend; needs Docker/Postgres via `pgtest`)
- [ ] After deploy: pick a Series that was blank, wait one poll cycle, confirm Cover in web UI and userscript panel

Reviewed-on: #70
Co-authored-by: Sulthan Zaki <sultankiki05@gmail.com>
Co-committed-by: Sulthan Zaki <sultankiki05@gmail.com>
This commit was merged in pull request #70.
This commit is contained in:
Sulthan Zaki
2026-08-10 10:14:27 +07:00
committed by sulthan
parent e2c054e7ce
commit b9220b3dfc
2 changed files with 288 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/internal/latest/poller.go
+69 -40
View File
@@ -2,6 +2,7 @@ package latest
import (
"context"
"errors"
"log"
"net/url"
"slices"
@@ -57,60 +58,85 @@ type Poller struct {
var browserBackedSites = []string{"kagane", "novelfull"}
// fillBlankCover gives a Series its Cover when it has none. The blank state is
// what "no Cover yet" means on the wire (ADR-0007): permanently-blank rows
// created before acquisition existed, and rows whose creation-time fetch
// failed, both heal here. A non-blank CoverAddress is left alone — refetching
// would add a request per Series per cycle and change artwork under the Reader
// for no visible reason. A row that already carries a source URL is owned by
// prefetchCover instead; this path only extracts from the series page.
//
// Failures are logged against the Series and never returned: the chapter poll
// must not notice. A failed fill is retried the next time this Series is due;
// there is no separate retry queue.
func (p *Poller) fillBlankCover(ctx context.Context, sr store.Series, body string) {
if sr.CoverAddress != "" || sr.Cover != "" {
return
}
cover, ok := coverFrom(sr.Site, sr.SeriesURL, body)
if !ok {
return
}
p.storeCover(ctx, sr, cover)
}
// prefetchCover heals Series that already carry a third-party source URL but
// no stored address — the state left by client-supplied covers before
// acquisition moved server-side. Every Site takes the same path; only the
// byte fetcher differs (kagane needs the browser). New blanks have no source
// URL and go through fillBlankCover from the series page instead.
func (p *Poller) prefetchCover(ctx context.Context, sr store.Series) {
if sr.Cover == "" {
if sr.Cover == "" || sr.CoverAddress != "" {
return
}
if sr.Site == "kagane" {
if p.CoverFetch == nil {
return
}
imageID, ok := store.KaganeImageID(sr.Cover)
if !ok {
return
}
_, _, found, err := p.Store.GetKaganeCover(imageID)
if err != nil {
log.Printf("latest poll %q: read cover: %v", sr.Key(), err)
return
}
if found {
return
}
body, contentType, err := p.CoverFetch.Image(ctx, imageID)
if err != nil {
log.Printf("latest poll %q: fetch cover: %v", sr.Key(), err)
return
}
if err := p.Store.PutKaganeCover(imageID, body, contentType); err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
}
return
}
if p.CoverBytesFetch == nil || sr.CoverAddress != "" {
return
}
// Bytes may already be stored from an earlier poll that ran before the
// Series carried an address; storing them again is free (they are
// content-addressed and immutable), and the point of the second call is
// the address, which is what makes the Cover visible on the wire.
body, contentType, found, err := p.Store.GetCover(sr.Cover)
if err != nil {
log.Printf("latest poll %q: read cover: %v", sr.Key(), err)
return
}
if !found {
body, contentType, err = p.CoverBytesFetch.Fetch(ctx, sr.Cover)
if err != nil {
log.Printf("latest poll %q: fetch cover: %v", sr.Key(), err)
return
if found {
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sr.Cover, body, contentType); err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
}
return
}
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sr.Cover, body, contentType); err != nil {
p.storeCover(ctx, sr, sr.Cover)
}
// storeCover fetches bytes for sourceURL and points the Series at them. Every
// failure is logged against the Series and swallowed so the chapter poll
// cannot see it.
func (p *Poller) storeCover(ctx context.Context, sr store.Series, sourceURL string) {
bytes, contentType, err := p.fetchCoverBytes(ctx, sr, sourceURL)
if err != nil {
log.Printf("latest poll %q: fetch cover %s: %v", sr.Key(), sourceURL, err)
return
}
if err := p.Store.SetSeriesCover(sr.Site, sr.SeriesID, sourceURL, bytes, contentType); err != nil {
log.Printf("latest poll %q: persist cover: %v", sr.Key(), err)
}
}
// fetchCoverBytes routes by Site: only kagane needs the browser for image
// bytes; every other Site's CDN answers plain TLS. Missing fetchers degrade to
// a blank Cover rather than falling back onto a path that cannot succeed.
func (p *Poller) fetchCoverBytes(ctx context.Context, sr store.Series, cover string) ([]byte, string, error) {
if sr.Site == "kagane" {
if p.CoverFetch == nil {
return nil, "", errors.New("no cover fetcher")
}
imageID, ok := store.KaganeImageID(cover)
if !ok {
return nil, "", errors.New("invalid kagane cover URL")
}
return p.CoverFetch.Image(ctx, imageID)
}
if p.CoverBytesFetch == nil {
return nil, "", errors.New("no cover fetcher")
}
return p.CoverBytesFetch.Fetch(ctx, cover)
}
// fetcherFor returns the fetcher a site needs, or nil when the site cannot be
// fetched at all right now. kagane and novelfull both sit behind a Cloudflare
// JavaScript challenge that no TLS fingerprint clears — kagane verified
@@ -234,6 +260,9 @@ func (p *Poller) checkOne(ctx context.Context, sr store.Series) {
}
latest, ok := latestChapterFrom(sr.Site, sr.SeriesURL, body)
// Cover fill is independent of the chapter signal: a page that lost its
// chapter list may keep its og:image, and a blank Series heals either way.
p.fillBlankCover(ctx, sr, body)
if !ok {
// Most likely a challenge page or a layout change. Either way the row is
// already stamped, so this waits out a cooldown instead of hot-looping.