Move cover bytes to content-addressed filesystem storage #65
@@ -25,6 +25,10 @@ POSTGRES_PASSWORD=changeme-generate-a-long-random-password
|
||||
# Override only to point the backend at a Postgres compose does not run.
|
||||
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
||||
|
||||
# Directory inside bookmark-api for immutable, content-addressed Cover bytes.
|
||||
# Compose builds the image and mounts its named volume at this path.
|
||||
COVER_DIR=/covers
|
||||
|
||||
# --- Prod override (Traefik) only ---
|
||||
# Subdomain Traefik routes to this service (required by the prod override).
|
||||
# BOOKMARK_API_HOST=bookmark-api.example.com
|
||||
|
||||
@@ -53,6 +53,10 @@ POSTGRES_PASSWORD=<paste output of: openssl rand -hex 24>
|
||||
# not run; it then replaces the URL built from POSTGRES_PASSWORD above.
|
||||
# DATABASE_URL=postgres://user:pass@host:5432/bookmarks?sslmode=require
|
||||
|
||||
# Required path inside bookmark-api. Compose builds the image and mounts the
|
||||
# named cover-data volume at this path.
|
||||
COVER_DIR=/covers
|
||||
|
||||
# Required for the Traefik override. Both have no fallback — compose refuses
|
||||
# to start without them. BOOKMARK_WEB_HOST is required even if the web UI
|
||||
# were unused; see 1b.
|
||||
|
||||
@@ -42,6 +42,7 @@ covers are stored, so the library renders in full with the browser switched off.
|
||||
| `OWNER_DISCORD_ID` | *(required)* | Discord user ID of the owner: seeded as the first Reader, owns every pre-registration bookmark, and is the only Reader who can revoke another's sessions. |
|
||||
| `ALLOWED_ORIGINS` | Asura + Demonic + Comix + Kagane origins | Comma-separated CORS allowlist. |
|
||||
| `DATABASE_URL` | *(required)* | Postgres connection URL, e.g. `postgres://bookmarks:…@postgres:5432/bookmarks?sslmode=disable`. Compose builds it from `POSTGRES_PASSWORD`. |
|
||||
| `COVER_DIR` | *(required)* | Filesystem volume for immutable, content-addressed Cover bytes. Compose builds the image and mounts `cover-data` at this path; standalone runs may choose another writable durable path. |
|
||||
| `PORT` | `8080` | Plain HTTP; TLS terminated by the proxy. |
|
||||
| `BROWSER_WS_URL` | empty | CDP endpoint of the remote browser (`ws://<tailnet IP>:9222`), used to poll Kagane/Novelfull past their JS challenge and to fetch uncached Kagane covers. Must be an IP or `localhost` — Chrome's DevTools handler 500s any other Host header, MagicDNS names included. Unset disables both; stored covers still serve. |
|
||||
| `DISCORD_CLIENT_ID` | *(required)* | Discord application credentials for the browser sign-in (ADR-0002). |
|
||||
|
||||
+5
-1
@@ -62,6 +62,10 @@ $COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c '\dt'
|
||||
# -> bookmarks, covers, readers, schema_migrations, series, sessions
|
||||
```
|
||||
|
||||
The `covers` table is metadata only after the filesystem cutover: bytes live in
|
||||
the separate `cover-data` volume. Back that volume up with the database dump;
|
||||
restoring only Postgres leaves stored Cover addresses without files.
|
||||
|
||||
Inside the container that connects over the local socket as the `bookmarks`
|
||||
superuser, so no password is needed anywhere in this section. `-T` is not
|
||||
optional: without it Compose allocates a TTY, which rewrites `\n` to `\r\n` and
|
||||
@@ -455,7 +459,7 @@ and skipped, stored covers still served.
|
||||
| Dump is 0 bytes, or `pg_restore`: `did not find magic string in file header` | You ran `exec` without `-T`. The allocated TTY rewrites newlines in the binary stream and corrupts the archive in flight (§1). |
|
||||
| `git pull`: `could not read Username for 'https://…'` | The checkout's remote is the HTTPS clone URL and the server has no credential helper, so the pull prompts into a closed stdin. Switch it to SSH once — `git remote set-url origin ssh://git@gitea.violetcrown.my.id:2222/sulthan/mangaBookmark.git`. Gitea's SSH listens on **2222**, not 22; port 22 is the host's own sshd and answers `Permission denied (publickey)` no matter which key is registered. |
|
||||
| kagane rows stopped updating after a redeploy | Check `BROWSER_WS_URL` survived the `.env` edit and still names the home machine's tailnet **IP**. A hostname 500s at `/json/version`; an empty value disables the browser silently. Plain-TLS sites keep working either way, which is why this is easy to miss. |
|
||||
| kagane covers went blank in the web UI | They should not — covers are rows in `covers`, not an in-process cache. `$COMPOSE exec -T postgres psql -U bookmarks -d bookmarks -c 'select count(*) from covers'`. Zero after a restore means the dump predates the covers table; they refill on the next poll of each series. |
|
||||
| kagane covers went blank in the web UI | Covers use the `cover-data` volume now. Restore/check that volume alongside Postgres; rows in `covers` are metadata only. If the database has rows but files are missing, the next browser-backed request refetches them; without a browser it remains a 404. |
|
||||
| Browser unit will not start: `set BROWSER_BIND_ADDR to this machine's tailnet IP` | `chrome/.env` is missing or the variable is empty. It has no default on purpose — an unset value must fail the deploy rather than publish an unauthenticated CDP port to the LAN. |
|
||||
| `bookmark-browser` shows `OOMKilled true` | The cap did its job. Read `docker logs bookmark-browser` before raising it — the sizing and what the cap protects are in ADR-0006. |
|
||||
|
||||
|
||||
@@ -114,6 +114,7 @@ Guidance for OpenCode (and Claude Code) working under `backend/`. See root `AGEN
|
||||
the owner of every pre-registration bookmark; required),
|
||||
`ALLOWED_ORIGINS` (comma list),
|
||||
`DATABASE_URL` (Postgres connection URL, required — no default),
|
||||
`COVER_DIR` (required filesystem volume for content-addressed Cover bytes),
|
||||
`PORT` (default `8080`), `DISCORD_CLIENT_ID`/`_CLIENT_SECRET`/`_GUILD_ID`/
|
||||
`_REDIRECT_URI` (required; Discord OAuth for the browser UI),
|
||||
`DISCORD_REQUIRED_ROLE` (optional role gate, empty by default),
|
||||
|
||||
+6
-1
@@ -2,6 +2,7 @@
|
||||
|
||||
# --- build stage: compile a static, CGO-free binary ---
|
||||
FROM golang:1.26-alpine AS build
|
||||
ARG COVER_DIR=/covers
|
||||
WORKDIR /src
|
||||
|
||||
# Dependencies first for layer caching (changes rarely).
|
||||
@@ -19,11 +20,15 @@ COPY internal/ ./internal/
|
||||
# -trimpath + -ldflags strip paths and debug info for a smaller image.
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server .
|
||||
|
||||
# Create the source directory; runtime COPY sets ownership for the named volume.
|
||||
RUN mkdir -p "$COVER_DIR"
|
||||
|
||||
# --- runtime stage: distroless static, non-root ---
|
||||
FROM gcr.io/distroless/static:nonroot
|
||||
ARG COVER_DIR=/covers
|
||||
WORKDIR /
|
||||
COPY --from=build --chown=65532:65532 ${COVER_DIR} ${COVER_DIR}
|
||||
COPY --from=build /out/server /server
|
||||
|
||||
EXPOSE 8080
|
||||
USER nonroot:nonroot
|
||||
ENV PORT=8080
|
||||
|
||||
+1
-1
@@ -60,7 +60,7 @@ func newTestStoreURL(t *testing.T) (*store.Store, string) {
|
||||
url := pgtest.URL(t)
|
||||
s, err := store.Open(url, store.Owner{
|
||||
DiscordID: testDiscordID, TokenHash: token.Hash(ownerCredential()),
|
||||
})
|
||||
}, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("store.Open: %v", err)
|
||||
}
|
||||
|
||||
@@ -101,11 +101,12 @@ func TestKaganeCoverServesStoredBytesWithoutBrowser(t *testing.T) {
|
||||
|
||||
func TestKaganeCoverServesPersistedBytesAfterRestart(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
owner := store.Owner{
|
||||
DiscordID: "cover-owner",
|
||||
TokenHash: sha256.Sum256([]byte("cover-owner-token")),
|
||||
}
|
||||
first, err := store.Open(url, owner)
|
||||
first, err := store.Open(url, owner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -117,7 +118,7 @@ func TestKaganeCoverServesPersistedBytesAfterRestart(t *testing.T) {
|
||||
t.Fatalf("close first store: %v", err)
|
||||
}
|
||||
|
||||
second, err := store.Open(url, owner)
|
||||
second, err := store.Open(url, owner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
|
||||
@@ -27,7 +27,7 @@ var testOwner = store.Owner{DiscordID: "test-owner", TokenHash: sha256.Sum256([]
|
||||
func newTestStore(t *testing.T) (*store.Store, string) {
|
||||
t.Helper()
|
||||
url := pgtest.URL(t)
|
||||
s, err := store.Open(url, testOwner)
|
||||
s, err := store.Open(url, testOwner, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -274,7 +274,7 @@ func TestRunOnceFetchesSharedSeriesOnce(t *testing.T) {
|
||||
// A second reader tracks the same series. The seed is the only
|
||||
// reader-creation path, so a second Open as a different owner is how a
|
||||
// test gets a second reader on the same database.
|
||||
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))})
|
||||
other, err := store.Open(url, store.Owner{DiscordID: "second-reader", TokenHash: sha256.Sum256([]byte("second-token-hash"))}, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("Open second reader: %v", err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
-- Cover bytes move out of Postgres. Existing rows are intentionally dropped:
|
||||
-- the old kagane path already refetches missing Covers on demand.
|
||||
DROP TABLE covers;
|
||||
|
||||
CREATE TABLE covers (
|
||||
address text PRIMARY KEY,
|
||||
path text NOT NULL,
|
||||
content_type text NOT NULL
|
||||
);
|
||||
@@ -1,12 +1,16 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"embed"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
@@ -225,7 +229,8 @@ type Store struct {
|
||||
// ownerID is the seeded owner Reader (issue #22) — the only Reader with
|
||||
// administrative reach (revoking another Reader's sessions). Every store
|
||||
// method takes a reader id explicitly, so ownership is never implicit.
|
||||
ownerID int64
|
||||
ownerID int64
|
||||
coverDir string
|
||||
}
|
||||
|
||||
// OwnerID returns the seeded owner Reader's id: the administrator, and the
|
||||
@@ -360,8 +365,22 @@ const allMigrations = 0
|
||||
|
||||
// Open connects to Postgres at url — a libpq connection URL such as
|
||||
// "postgres://user:pass@host:5432/bookmarks?sslmode=disable" — brings its
|
||||
// schema up to date, and seeds the owner Reader.
|
||||
func Open(url string, owner Owner) (*Store, error) {
|
||||
// schema up to date, seeds the owner Reader, and prepares cover storage.
|
||||
func Open(url string, owner Owner, coverDir string) (*Store, error) {
|
||||
if strings.TrimSpace(coverDir) == "" {
|
||||
return nil, errors.New("cover directory is required")
|
||||
}
|
||||
if err := os.MkdirAll(coverDir, 0o755); err != nil {
|
||||
return nil, fmt.Errorf("create cover directory: %w", err)
|
||||
}
|
||||
info, err := os.Stat(coverDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stat cover directory: %w", err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return nil, fmt.Errorf("cover directory %q is not a directory", coverDir)
|
||||
}
|
||||
|
||||
db, err := sql.Open("pgx", url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open postgres: %w", err)
|
||||
@@ -396,7 +415,7 @@ func Open(url string, owner Owner) (*Store, error) {
|
||||
db.Close()
|
||||
return nil, fmt.Errorf("resolve owner: %w", err)
|
||||
}
|
||||
return &Store{db: db, ownerID: ownerID}, nil
|
||||
return &Store{db: db, ownerID: ownerID, coverDir: coverDir}, nil
|
||||
}
|
||||
|
||||
// seedOwner makes sure the configured owner exists as exactly one readers row.
|
||||
@@ -550,40 +569,96 @@ func scanSeries(scan func(...any) error) (Series, error) {
|
||||
// Close releases the underlying database handle.
|
||||
func (s *Store) Close() error { return s.db.Close() }
|
||||
|
||||
// GetKaganeCover returns one persisted cover. Missing covers are reported with
|
||||
// ok=false rather than as an error so the web handler can fetch them once.
|
||||
func (s *Store) GetKaganeCover(imageID string) ([]byte, string, bool, error) {
|
||||
var (
|
||||
body []byte
|
||||
contentType string
|
||||
)
|
||||
func coverSourceAddress(sourceURL string) string {
|
||||
sum := sha256.Sum256([]byte(sourceURL))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func coverRelativePath(address string) string {
|
||||
return address[:2] + "/" + address[2:4] + "/" + address
|
||||
}
|
||||
|
||||
func kaganeCoverSourceURL(imageID string) string {
|
||||
return "https://kagane.to/api/v2/image/" + imageID + "/compressed"
|
||||
}
|
||||
|
||||
func (s *Store) getCover(sourceURL string) ([]byte, string, bool, error) {
|
||||
address := coverSourceAddress(sourceURL)
|
||||
var relativePath, contentType string
|
||||
err := s.db.QueryRow(
|
||||
`SELECT body, content_type FROM covers WHERE image_id = $1`, imageID,
|
||||
).Scan(&body, &contentType)
|
||||
`SELECT path, content_type FROM covers WHERE address = $1`, address,
|
||||
).Scan(&relativePath, &contentType)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, "", false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", false, fmt.Errorf("get kagane cover %q: %w", imageID, err)
|
||||
return nil, "", false, fmt.Errorf("get cover %q: %w", address, err)
|
||||
}
|
||||
expectedPath := coverRelativePath(address)
|
||||
if relativePath != expectedPath {
|
||||
return nil, "", false, fmt.Errorf("cover %q has unexpected path %q", address, relativePath)
|
||||
}
|
||||
body, err := os.ReadFile(filepath.Join(s.coverDir, filepath.FromSlash(relativePath)))
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil, "", false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", false, fmt.Errorf("read cover %q: %w", address, err)
|
||||
}
|
||||
return body, contentType, true, nil
|
||||
}
|
||||
|
||||
// PutKaganeCover persists one fetched cover. Image ids are immutable, so a
|
||||
// later fetch cannot replace the bytes already made durable.
|
||||
func (s *Store) PutKaganeCover(imageID string, body []byte, contentType string) error {
|
||||
func (s *Store) putCover(sourceURL string, body []byte, contentType string) error {
|
||||
if !IsKaganeCoverContentType(contentType) {
|
||||
return fmt.Errorf("put kagane cover %q: unsupported content type %q", imageID, contentType)
|
||||
return fmt.Errorf("put cover %q: unsupported content type %q", sourceURL, contentType)
|
||||
}
|
||||
address := coverSourceAddress(sourceURL)
|
||||
relativePath := coverRelativePath(address)
|
||||
coverPath := filepath.Join(s.coverDir, filepath.FromSlash(relativePath))
|
||||
if err := os.MkdirAll(filepath.Dir(coverPath), 0o755); err != nil {
|
||||
return fmt.Errorf("create cover shard: %w", err)
|
||||
}
|
||||
tmp, err := os.CreateTemp(filepath.Dir(coverPath), ".cover-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("create cover temp file: %w", err)
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
defer os.Remove(tmpName)
|
||||
if _, err := tmp.Write(body); err != nil {
|
||||
tmp.Close()
|
||||
return fmt.Errorf("write cover temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Sync(); err != nil {
|
||||
tmp.Close()
|
||||
return fmt.Errorf("sync cover temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close cover temp file: %w", err)
|
||||
}
|
||||
if err := os.Link(tmpName, coverPath); err != nil && !errors.Is(err, fs.ErrExist) {
|
||||
return fmt.Errorf("install cover file: %w", err)
|
||||
}
|
||||
if _, err := s.db.Exec(`
|
||||
INSERT INTO covers (image_id, body, content_type)
|
||||
INSERT INTO covers (address, path, content_type)
|
||||
VALUES ($1, $2, $3)
|
||||
ON CONFLICT (image_id) DO NOTHING`, imageID, body, contentType); err != nil {
|
||||
return fmt.Errorf("put kagane cover %q: %w", imageID, err)
|
||||
ON CONFLICT (address) DO NOTHING`, address, relativePath, contentType); err != nil {
|
||||
return fmt.Errorf("record cover %q: %w", address, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetKaganeCover returns one persisted cover. Missing covers are reported with
|
||||
// ok=false rather than as an error so the web handler can fetch them once.
|
||||
func (s *Store) GetKaganeCover(imageID string) ([]byte, string, bool, error) {
|
||||
return s.getCover(kaganeCoverSourceURL(imageID))
|
||||
}
|
||||
|
||||
// PutKaganeCover persists one fetched cover. The source URL's content address
|
||||
// makes each stored object immutable, so later writes for that URL are ignored.
|
||||
func (s *Store) PutKaganeCover(imageID string, body []byte, contentType string) error {
|
||||
return s.putCover(kaganeCoverSourceURL(imageID), body, contentType)
|
||||
}
|
||||
|
||||
// List returns every bookmark of one reader, newest activity first.
|
||||
// Series-owned fields are joined in, so each Bookmark reads back whole and
|
||||
// flat (ADR-0004).
|
||||
|
||||
@@ -4,7 +4,9 @@ import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -21,7 +23,7 @@ var testOwner = Owner{DiscordID: "test-owner", TokenHash: sha256.Sum256([]byte("
|
||||
|
||||
func newTestStore(t *testing.T) *Store {
|
||||
t.Helper()
|
||||
store, err := Open(pgtest.URL(t), testOwner)
|
||||
store, err := Open(pgtest.URL(t), testOwner, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -46,7 +48,8 @@ func secondReader(t *testing.T, s *Store) int64 {
|
||||
// error, and must leave the rows alone.
|
||||
func TestOpenIsIdempotent(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
first, err := Open(url, testOwner)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -57,7 +60,7 @@ func TestOpenIsIdempotent(t *testing.T) {
|
||||
}
|
||||
first.Close()
|
||||
|
||||
second, err := Open(url, testOwner)
|
||||
second, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -109,7 +112,8 @@ func TestReaderTokenInfo(t *testing.T) {
|
||||
// epoch-0 hash only while the row has never been rotated.
|
||||
func TestRotateTokenInvalidatesOldAndSurvivesRestart(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
store, err := Open(url, testOwner)
|
||||
coverDir := t.TempDir()
|
||||
store, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -141,7 +145,7 @@ func TestRotateTokenInvalidatesOldAndSurvivesRestart(t *testing.T) {
|
||||
}
|
||||
store.Close()
|
||||
|
||||
reopened, err := Open(url, testOwner)
|
||||
reopened, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -668,7 +672,7 @@ func TestMigration0002BackfillsExistingBookmarks(t *testing.T) {
|
||||
// Bring it current through the production path: Open runs the schema to
|
||||
// 0003, seeds the owner, then applies 0004 which attaches this row. 0002
|
||||
// must have backfilled the series row, not lost data.
|
||||
st, err := Open(url, testOwner)
|
||||
st, err := Open(url, testOwner, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("Open after migrate: %v", err)
|
||||
}
|
||||
@@ -697,6 +701,48 @@ func TestMigration0002BackfillsExistingBookmarks(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigration0008DropsLegacyCoverRows(t *testing.T) {
|
||||
db, err := sql.Open("pgx", pgtest.URL(t))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
if err := migrate(db, readersMigration); err != nil {
|
||||
t.Fatalf("migrate readers: %v", err)
|
||||
}
|
||||
if err := seedOwner(db, testOwner); err != nil {
|
||||
t.Fatalf("seed owner: %v", err)
|
||||
}
|
||||
if err := migrate(db, 7); err != nil {
|
||||
t.Fatalf("migrate legacy covers: %v", err)
|
||||
}
|
||||
if _, err := db.Exec(`
|
||||
INSERT INTO covers (image_id, body, content_type)
|
||||
VALUES ('legacy-image', 'legacy-bytes', 'image/jpeg')`); err != nil {
|
||||
t.Fatalf("seed legacy cover: %v", err)
|
||||
}
|
||||
if err := migrate(db, 0); err != nil {
|
||||
t.Fatalf("migrate filesystem covers: %v", err)
|
||||
}
|
||||
|
||||
var count int
|
||||
if err := db.QueryRow(`SELECT count(*) FROM covers`).Scan(&count); err != nil {
|
||||
t.Fatalf("count covers: %v", err)
|
||||
}
|
||||
if count != 0 {
|
||||
t.Fatalf("legacy covers = %d, want 0", count)
|
||||
}
|
||||
var bodyColumn int
|
||||
if err := db.QueryRow(`SELECT count(*) FROM information_schema.columns
|
||||
WHERE table_name = 'covers' AND column_name = 'body'`).Scan(&bodyColumn); err != nil {
|
||||
t.Fatalf("cover columns: %v", err)
|
||||
}
|
||||
if bodyColumn != 0 {
|
||||
t.Fatal("legacy covers table still has body column")
|
||||
}
|
||||
}
|
||||
|
||||
// readSeries reads the series row directly, for asserting on what Upsert
|
||||
// actually stored rather than what the joined Bookmark reports.
|
||||
func readSeries(t *testing.T, s *Store, site, seriesID string) Series {
|
||||
@@ -893,14 +939,15 @@ func TestDueForLatestCheckExcludesOrphanSeries(t *testing.T) {
|
||||
// rotations.
|
||||
func TestSeedOwnerIdempotentAndRefreshesTokenHash(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
first, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v1"))})
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v1"))}, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
ownerID := first.OwnerID()
|
||||
first.Close()
|
||||
|
||||
second, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v2"))})
|
||||
second, err := Open(url, Owner{DiscordID: "owner", TokenHash: sha256.Sum256([]byte("hash-v2"))}, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -957,7 +1004,7 @@ func TestMigration0004AttachesBookmarksToOwner(t *testing.T) {
|
||||
t.Fatalf("migrate to 0002: %v", err)
|
||||
}
|
||||
|
||||
st, err := Open(url, testOwner)
|
||||
st, err := Open(url, testOwner, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -1233,7 +1280,8 @@ func TestTwoReadersShareOneSeriesWithIndependentProgress(t *testing.T) {
|
||||
}
|
||||
func TestKaganeCoverPersistsAcrossReopen(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
first, err := Open(url, testOwner)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
@@ -1245,7 +1293,7 @@ func TestKaganeCoverPersistsAcrossReopen(t *testing.T) {
|
||||
t.Fatalf("close first store: %v", err)
|
||||
}
|
||||
|
||||
second, err := Open(url, testOwner)
|
||||
second, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("reopen: %v", err)
|
||||
}
|
||||
@@ -1258,3 +1306,49 @@ func TestKaganeCoverPersistsAcrossReopen(t *testing.T) {
|
||||
t.Fatalf("stored cover = (%q, %q, %v), want (%q, image/webp, true)", got, contentType, ok, body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenRequiresCoverDirectory(t *testing.T) {
|
||||
if _, err := Open(pgtest.URL(t), testOwner, ""); err == nil || !strings.Contains(err.Error(), "cover directory is required") {
|
||||
t.Fatalf("Open without cover directory = %v, want required-directory error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKaganeCoverIsContentAddressedOnFilesystem(t *testing.T) {
|
||||
url := pgtest.URL(t)
|
||||
coverDir := t.TempDir()
|
||||
first, err := Open(url, testOwner, coverDir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
body := []byte("stored-cover")
|
||||
const imageID = "019fe11a-84c3-7fc3-a84b-88787374b617"
|
||||
if err := first.PutKaganeCover(imageID, body, "image/webp"); err != nil {
|
||||
first.Close()
|
||||
t.Fatalf("PutKaganeCover: %v", err)
|
||||
}
|
||||
defer first.Close()
|
||||
|
||||
sourceURL := "https://kagane.to/api/v2/image/" + imageID + "/compressed"
|
||||
addressBytes := sha256.Sum256([]byte(sourceURL))
|
||||
address := hex.EncodeToString(addressBytes[:])
|
||||
wantPath := filepath.Join(address[:2], address[2:4], address)
|
||||
|
||||
var gotPath, contentType string
|
||||
if err := first.db.QueryRow(`SELECT path, content_type FROM covers WHERE address = $1`, address).Scan(&gotPath, &contentType); err != nil {
|
||||
t.Fatalf("cover row: %v", err)
|
||||
}
|
||||
if gotPath != wantPath || contentType != "image/webp" {
|
||||
t.Fatalf("cover row = (%q, %q), want (%q, image/webp)", gotPath, contentType, wantPath)
|
||||
}
|
||||
if got, err := os.ReadFile(filepath.Join(coverDir, gotPath)); err != nil || !bytes.Equal(got, body) {
|
||||
t.Fatalf("cover file = (%q, %v), want (%q, nil)", got, err, body)
|
||||
}
|
||||
|
||||
var bodyColumn int
|
||||
if err := first.db.QueryRow(`SELECT count(*) FROM information_schema.columns WHERE table_name = 'covers' AND column_name = 'body'`).Scan(&bodyColumn); err != nil {
|
||||
t.Fatalf("cover columns: %v", err)
|
||||
}
|
||||
if bodyColumn != 0 {
|
||||
t.Fatalf("covers still has body column")
|
||||
}
|
||||
}
|
||||
|
||||
+10
-2
@@ -30,7 +30,11 @@ type Config struct {
|
||||
// DatabaseURL is the Postgres connection URL; required, no default,
|
||||
// because a wrong guess would silently start on an empty database.
|
||||
DatabaseURL string
|
||||
Port string
|
||||
// CoverDir is the filesystem volume for immutable cover bytes. Required:
|
||||
// serving a stored address without durable bytes would be worse than a
|
||||
// startup failure.
|
||||
CoverDir string
|
||||
Port string
|
||||
// OwnerDiscordID identifies the seeded owner Reader (issue #22). Required:
|
||||
// bookmarks are scoped to a Reader, and a fresh deployment needs one
|
||||
// before anybody logs in. The owner is also the only Reader who can revoke
|
||||
@@ -167,6 +171,7 @@ func loadConfig() Config {
|
||||
c := Config{
|
||||
TokenKey: os.Getenv("TOKEN_KEY"),
|
||||
DatabaseURL: os.Getenv("DATABASE_URL"),
|
||||
CoverDir: os.Getenv("COVER_DIR"),
|
||||
Port: envOr("PORT", "8080"),
|
||||
OwnerDiscordID: os.Getenv("OWNER_DISCORD_ID"),
|
||||
UserscriptPath: envOr("USERSCRIPT_PATH", "/userscript/manga-bookmark.user.js"),
|
||||
@@ -254,6 +259,9 @@ func main() {
|
||||
if cfg.DatabaseURL == "" {
|
||||
log.Fatal("DATABASE_URL is required")
|
||||
}
|
||||
if cfg.CoverDir == "" {
|
||||
log.Fatal("COVER_DIR is required")
|
||||
}
|
||||
// The web UI signs in through Discord, so a deployment without the OAuth
|
||||
// application is misconfigured rather than passwordless.
|
||||
for key, v := range map[string]string{
|
||||
@@ -274,7 +282,7 @@ func main() {
|
||||
TokenHash: token.Hash(token.Token([]byte(cfg.TokenKey), cfg.OwnerDiscordID, 0)),
|
||||
}
|
||||
|
||||
s, err := store.Open(cfg.DatabaseURL, owner)
|
||||
s, err := store.Open(cfg.DatabaseURL, owner, cfg.CoverDir)
|
||||
if err != nil {
|
||||
log.Fatalf("open store: %v", err)
|
||||
}
|
||||
|
||||
@@ -37,6 +37,13 @@ func TestLoadLatestPollDefaults(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfigReadsCoverDirectory(t *testing.T) {
|
||||
t.Setenv("COVER_DIR", "/covers")
|
||||
if got := loadConfig().CoverDir; got != "/covers" {
|
||||
t.Fatalf("CoverDir = %q, want /covers", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadLatestPollEnabledParsing(t *testing.T) {
|
||||
tests := []struct {
|
||||
raw string
|
||||
|
||||
+10
-1
@@ -11,7 +11,10 @@
|
||||
|
||||
services:
|
||||
bookmark-api:
|
||||
build: ./backend
|
||||
build:
|
||||
context: ./backend
|
||||
args:
|
||||
COVER_DIR: ${COVER_DIR:?set COVER_DIR in .env}
|
||||
image: bookmarkmanager-backend:latest
|
||||
container_name: bookmark-api
|
||||
restart: unless-stopped
|
||||
@@ -26,6 +29,9 @@ services:
|
||||
# The bookmarks database. Host is the compose service name; the password
|
||||
# comes from .env so it is never committed.
|
||||
DATABASE_URL: ${DATABASE_URL:-postgres://bookmarks:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/bookmarks?sslmode=disable}
|
||||
# Required path inside the API. The build seeds ownership at this path
|
||||
# and the named volume below mounts there.
|
||||
COVER_DIR: ${COVER_DIR:?set COVER_DIR in .env}
|
||||
PORT: "8080"
|
||||
# Log timestamps only. Go's `log` stamps lines in local time, and this
|
||||
# service has no other use for a zone: bookmark timestamps are unix ms
|
||||
@@ -80,6 +86,8 @@ services:
|
||||
# no rebuild, no restart. `git pull` restores the committed version, which
|
||||
# is why a redeploy always ships the repo's script.
|
||||
- ./userscript:/userscript:ro
|
||||
# Content-addressed cover bytes survive API restarts and redeploys.
|
||||
- cover-data:${COVER_DIR:?set COVER_DIR in .env}
|
||||
# Bound to loopback only: the proxy (or curl during smoke test) reaches it,
|
||||
# the public internet does not.
|
||||
ports:
|
||||
@@ -112,6 +120,7 @@ services:
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
cover-data:
|
||||
# The pre-Postgres SQLite volume (bookmarks-data) is deliberately no longer
|
||||
# declared here: undeclared means `docker compose down -v` cannot take it
|
||||
# with the rest, so the old database survives the cutover until someone
|
||||
|
||||
Reference in New Issue
Block a user