sulthan/mangaBookmark
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: one Poll Lane per Site, replacing the shared pace (#100)

Browse Source 
Each Site runs its own poll goroutine, paced by Rest and Gap from the
registry (sites.go) instead of the shared cooldown/interval/stagger/batch
config. Rest is enforced by the due query's WHERE clause; the Lane sleeps
its effective gap between fetches (hour/eligible, floored at 1s).

Lane-local failures: two refusals stop that Site for 15m, a lost browser
stops only the round's remaining browser Lanes, and cover heals moved to
background goroutines so a slow CDN cannot consume a Lane's gap. The five
LATEST_CHAPTER_POLL_* pace env vars are gone; only the kill switch
remains.
This commit is contained in:
Sulthan Zaki
2026-08-16 12:51:01 +07:00
parent ddbd57070d
commit 90ce14aec6
22 changed files with 3645 additions and 1884 deletions
Download Patch File Download Diff File Expand all files Collapse all files
graphify-out/GRAPH_REPORT.md
+57 -52
View File
@@ -1,16 +1,16 @@
# Graph Report - mangaBookmark (2026-08-16)
## Corpus Check
- 112 files · ~274,851 words
- 113 files · ~279,161 words
- Verdict: corpus is large enough that graph structure adds value.
## Summary
- 1640 nodes · 3294 edges · 221 communities (67 shown, 154 thin omitted)
- Extraction: 90% EXTRACTED · 10% INFERRED · 0% AMBIGUOUS · INFERRED: 313 edges (avg confidence: 0.77)
- 1668 nodes · 3379 edges · 222 communities (68 shown, 154 thin omitted)
- Extraction: 91% EXTRACTED · 9% INFERRED · 0% AMBIGUOUS · INFERRED: 320 edges (avg confidence: 0.77)
- Token cost: 0 input · 0 output
## Graph Freshness
- Built from commit: `f000cc7e`
- Built from commit: `4f1cbcfd`
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
- Run `graphify update .` after code changes (no API cost).
@@ -45,7 +45,8 @@
- [[_COMMUNITY_Find Skills Guide|Find Skills Guide]]
- [[_COMMUNITY_Allocation Patterns|Allocation Patterns]]
- [[_COMMUNITY_Observability & Alerting|Observability & Alerting]]
- [[_COMMUNITY_Repo Hard Constraints|Repo Hard Constraints]]
- [[_COMMUNITY_Store|Store]]
- [[_COMMUNITY_Open|Open]]
- [[_COMMUNITY_Go Testing Guide|Go Testing Guide]]
- [[_COMMUNITY_Session Store|Session Store]]
- [[_COMMUNITY_Web UI Filter Logic|Web UI Filter Logic]]
@@ -54,7 +55,7 @@
- [[_COMMUNITY_novel-logic.test.js|novel-logic.test.js]]
- [[_COMMUNITY_UI Critique 2026-07-26A|UI Critique 2026-07-26A]]
- [[_COMMUNITY_UI Critique 2026-07-26B|UI Critique 2026-07-26B]]
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
- [[_COMMUNITY_pgtest.go|pgtest.go]]
- [[_COMMUNITY_Issue Tracker & Triage|Issue Tracker & Triage]]
- [[_COMMUNITY_Ticket Workflow|Ticket Workflow]]
- [[_COMMUNITY_Go Perf Alert Rules|Go Perf Alert Rules]]
@@ -137,6 +138,7 @@
- [[_COMMUNITY_Identity comes from Discord OAuth; we store no passwords and send no email|Identity comes from Discord OAuth; we store no passwords and send no email]]
- [[_COMMUNITY_The wire format stays flat and deliberately does not mirror the schema|The wire format stays flat and deliberately does not mirror the schema]]
- [[_COMMUNITY_ADR-0005 On-demand browser sidecar|ADR-0005: On-demand browser sidecar]]
- [[_COMMUNITY_sessions_test.go|sessions_test.go]]
- [[_COMMUNITY_Bookmark Manager|Bookmark Manager]]
- [[_COMMUNITY_triage-labels|triage-labels.md]]
- [[_COMMUNITY_Cross-Ticket Contract|Cross-Ticket Contract]]
@@ -164,7 +166,6 @@
- [[_COMMUNITY_Reviewer Subagent (opencode)|Reviewer Subagent (opencode)]]
- [[_COMMUNITY_Finding Severity Rubric|Finding Severity Rubric]]
- [[_COMMUNITY_Spec Compliance Review|Spec Compliance Review]]
- [[_COMMUNITY_T|T]]
- [[_COMMUNITY_Why Use samberoops|Why Use samber/oops]]
- [[_COMMUNITY_singleflight Cache Stampede Prevention|singleflight Cache Stampede Prevention]]
- [[_COMMUNITY_Struct Field Alignment|Struct Field Alignment]]
@@ -239,14 +240,14 @@
## God Nodes (most connected - your core abstractions)
1. `testConfig()` - 53 edges
2. `newWebTestServer()` - 49 edges
3. `newTestStore()` - 43 edges
4. `newTestStore()` - 42 edges
3. `newTestStore()` - 48 edges
4. `newTestStore()` - 43 edges
5. `e()` - 33 edges
6. `Handler` - 29 edges
7. `ne()` - 28 edges
8. `De()` - 28 edges
9. `Open()` - 27 edges
10. `se()` - 27 edges
6. `Open()` - 29 edges
7. `Handler` - 29 edges
8. `ne()` - 28 edges
9. `De()` - 28 edges
10. `Store` - 27 edges
## Surprising Connections (you probably didn't know these)
- `el()` --indirect_call--> `c()` [INFERRED]
@@ -272,7 +273,7 @@
- **Headless browser infrastructure (sidecar, on-demand, home deployment)** — docs_adr_0005_on_demand_browser_headless_shell, docs_adr_0005_on_demand_browser_cdp, docs_adr_0005_on_demand_browser_on_demand_start, docs_adr_0006_browser_on_the_home_machine_home_machine_rationale [INFERRED 0.85]
- **lightnovelworld series-identity investigation and fix** — docs_research_lightnovelworld_chapter_vs_series_slug_issue_77, docs_research_lightnovelworld_chapter_vs_series_slug_unscoped_regex, docs_adr_0008_series_identity_is_discovered_not_derived_discovered_identity [INFERRED 0.85]
## Communities (221 total, 154 thin omitted)
## Communities (222 total, 154 thin omitted)
### Community 0 - "HTMX Library Internals"
Cohesion: 0.08
@@ -284,30 +285,30 @@ Nodes (84): floatPtr(), testConfig(), getCover(), Cookie, Handler, ResponseRecor
### Community 2 - "Manga Userscript Adapters"
Cohesion: 0.06
Nodes (77): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+69 more)
Nodes (76): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+68 more)
### Community 3 - "Novel Userscript Adapters"
Cohesion: 0.06
Nodes (78): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+70 more)
Nodes (79): adapterFor(), anchorsFromDocument(), anchorsFromHTML(), apiDelete(), apiGet(), apiPut(), applyFabPos(), applyLatestChapterIfChanged() (+71 more)
### Community 4 - "Series Acquisition Tests"
Cohesion: 0.09
Nodes (69): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+61 more)
Cohesion: 0.08
Nodes (78): bookmarkNewKaganeSeries(), bookmarkNewNovelfullSeries(), bookmarkNewSeries(), Context, Store, T, newAcquirer(), readBookmark() (+70 more)
### Community 5 - "Bookmarks API Tests"
Cohesion: 0.08
Nodes (67): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+59 more)
Cohesion: 0.06
Nodes (74): auth(), getBookmarks(), Handler, Request, Store, T, newTestServer(), newTestStore() (+66 more)
### Community 7 - "Cover & Acquire Internals"
Cohesion: 0.10
Nodes (30): Addr, Context, Store, defaultCoverResolver(), fetchCoverBytes(), Client, Context, NewCoverFetcher() (+22 more)
Cohesion: 0.08
Nodes (37): Addr, Context, Store, WaitGroup, isInterstitial(), defaultCoverResolver(), fetchCoverBytes(), Client (+29 more)
### Community 8 - "System Architecture Concepts"
Cohesion: 0.13
Nodes (20): app.html — App Shell Template, Manga/Novel Library Switch, Bookmark Bucket Tabs, Confirm Row (Archive/Finish/Remove), card.html — Series Card Template, htmx /ui/* Mutation Endpoints, chrome.html — Out-of-Band Regions, Action Key (+12 more)
### Community 9 - "Session Middleware"
Cohesion: 0.07
Cohesion: 0.08
Nodes (35): ClearCookie(), ClientIP(), Duration, Mutex, Request, ResponseWriter, Time, isHTTPS() (+27 more)
### Community 10 - "Go Test Helpers"
@@ -315,12 +316,12 @@ Cohesion: 0.05
Nodes (39): Test Helpers, Test Timeout, Basic Handler Test, HTTP Handler Testing, Query Parameters and Headers, Docker Compose Fixture, Integration Testing, SQL Schema Fixture (+31 more)
### Community 11 - "Store Tests"
Cohesion: 0.15
Nodes (43): Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck(), seedSecondReader() (+35 more)
Cohesion: 0.14
Nodes (45): scanSeries(), Store, T, newTestStore(), readLatestCheckedAt(), readSeries(), secondReader(), seedForCheck() (+37 more)
### Community 12 - "Bookmarks API Handler"
Cohesion: 0.11
Nodes (24): Auth(), compressible(), CORS(), Handler, ResponseWriter, Store, Gzip(), ResolveReader() (+16 more)
Cohesion: 0.08
Nodes (32): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Auth(), compressible() (+24 more)
### Community 13 - "Web UI Handlers"
Cohesion: 0.14
@@ -331,8 +332,8 @@ Cohesion: 0.06
Nodes (33): Creating Errors, Custom Error Types, Custom types that wrap other errors, Decision table: which error strategy to use, Error Creation, Error String Conventions, Errors as Values, `errors.New` — static error messages (+25 more)
### Community 15 - "CDP Browser Client"
Cohesion: 0.07
Nodes (36): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+28 more)
Cohesion: 0.06
Nodes (56): awaitPromise(), browserConnectionLost(), classifyBrowserError(), comixRead(), comixSeriesPageURL(), Action, Context, Mutex (+48 more)
### Community 16 - "Cloudflare bot scoring and poll cadence — what is actually documented"
Cohesion: 0.06
@@ -343,8 +344,8 @@ Cohesion: 0.08
Nodes (23): Code Style Details, Extract Complex Conditions, Value vs Pointer Arguments, Code Organization Within Files, Complex Conditions & Init Scope, Composite Literals, Control Flow, Cross-References (+15 more)
### Community 19 - "Store"
Cohesion: 0.09
Nodes (8): coverRelativePath(), coverSourceAddress(), displayChapter(), Store, scanSeries(), TestDisplayChapter(), Bookmark, ReaderSummary
Cohesion: 0.33
Nodes (5): ADR-0010: Poll Lanes — one independent Poll stream per Site, Constraints preserved, Decision, Tradeoffs and rejections, Why
### Community 20 - "I/O Performance Patterns"
Cohesion: 0.11
@@ -371,24 +372,28 @@ Cohesion: 0.33
Nodes (12): coverResponse(), Request, T, TestCoverFetcherCanonicalisesJpgAlias(), TestCoverFetcherFetchesPublicHTTPSImage(), TestCoverFetcherRefusesUnsafeDestinationsBeforeRequest(), TestCoverFetcherRejectsNonImage(), TestCoverFetcherRejectsOversizedBody() (+4 more)
### Community 26 - "Open"
Cohesion: 0.20
Nodes (17): applyMigration(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen(), TestMigration0002BackfillsExistingBookmarks() (+9 more)
Cohesion: 0.40
Nodes (5): Map of pointers for large, frequently updated structs, Memory Layout, Pointer receivers for large structs, Struct field alignment, Zero-size field at end of struct
### Community 27 - "Find Skills Guide"
Cohesion: 0.14
Nodes (13): Common Skill Categories, Find Skills, How to Help Users Find Skills, Step 1: Understand What They Need, Step 2: Check the Leaderboard First, Step 3: Search for Skills, Step 4: Verify Quality Before Recommending, Step 5: Present Options to the User (+5 more)
### Community 28 - "Allocation Patterns"
Cohesion: 0.11
Nodes (19): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map of pointers for large, frequently updated structs, Map size hints (+11 more)
Cohesion: 0.14
Nodes (14): Allocation Patterns, Backing Array Leaks, Direct indexing vs append, Eliminate redundant map lookups, Interface boxing, Map never shrinks, Map size hints, Memory Optimization (+6 more)
### Community 29 - "Observability & Alerting"
Cohesion: 0.22
Nodes (9): Alerting rules (examples), CPU saturation, GC pressure, Goroutine leaks, Grafana Dashboards, Memory leaks, Prometheus Metrics for Go, PromQL Queries for Performance Diagnosis (+1 more)
### Community 32 - "Repo Hard Constraints"
Cohesion: 0.18
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
### Community 31 - "Store"
Cohesion: 0.12
Nodes (3): coverRelativePath(), coverSourceAddress(), Store
### Community 32 - "Open"
Cohesion: 0.14
Nodes (20): applyMigration(), displayChapter(), migrate(), Open(), refreshOwnerToken(), seedOwner(), TestCoverIsContentAddressedOnFilesystem(), TestCoverPersistsAcrossReopen() (+12 more)
### Community 33 - "Go Testing Guide"
Cohesion: 0.20
@@ -418,9 +423,9 @@ Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations,
Cohesion: 0.29
Nodes (6): Design Health Score, Design Specificity Verdict, Minor Observations, Persona Red Flags, Priority Issues, Questions to Consider
### Community 41 - "sessions_test.go"
Cohesion: 0.48
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
### Community 42 - "pgtest.go"
Cohesion: 0.18
Nodes (12): M, TestMain(), M, TestMain(), M, Main(), start(), URL() (+4 more)
### Community 45 - "Go Perf Alert Rules"
Cohesion: 0.50
@@ -530,28 +535,28 @@ Nodes (3): Consequence, The wire format stays flat and deliberately does not mir
Cohesion: 0.50
Nodes (3): ADR-0005: On-demand browser sidecar, Constraints, Decision
### Community 158 - "T"
Cohesion: 0.08
Nodes (38): Handler, Request, ResponseWriter, Store, Healthz(), writeJSON(), Request, ReaderID() (+30 more)
### Community 125 - "sessions_test.go"
Cohesion: 0.48
Nodes (6): T, TestCreateAndGetSession(), TestDeleteSessionIsPerReader(), TestDeleteSessionRevokes(), TestExpiredSessionIsGone(), TestGetSessionUnknownID()
### Community 273 - "AGENTS.md"
Cohesion: 0.50
Nodes (3): Live URL shapes (verified 2026-07-26, may drift — re-check against live pages before trust), Second script: `novel-bookmark.user.js`, Userscript structure (single IIFE, `manga-bookmark.user.js`)
## Knowledge Gaps
- **520 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+515 more)
- **524 isolated node(s):** `bookmarkmanager/backend`, `ctxKey`, `loginView`, `ctxKey`, `test` (+519 more)
These have ≤1 connection - possible missing edges or undocumented components.
- **154 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
## Suggested Questions
_Questions this graph is uniquely positioned to answer:_
- **Why does `New()` connect `Series Acquisition Tests` to `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`, `Open`?**
_High betweenness centrality (0.051) - this node is a cross-community bridge._
- **Why does `Open()` connect `Open` to `Repo Hard Constraints`, `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `Store Tests`, `Store`?**
_High betweenness centrality (0.038) - this node is a cross-community bridge._
- **Why does `New()` connect `Series Acquisition Tests` to `Open`, `Bookmarks API Tests`, `Cover & Acquire Internals`, `Session Middleware`, `Web UI Handlers`?**
_High betweenness centrality (0.045) - this node is a cross-community bridge._
- **Why does `Open()` connect `Open` to `Cover Fetch Test Helpers`, `Series Acquisition Tests`, `Bookmarks API Tests`, `pgtest.go`, `Store Tests`, `Store`?**
_High betweenness centrality (0.033) - this node is a cross-community bridge._
- **Why does `newRouter()` connect `Bookmarks API Tests` to `Cover Fetch Test Helpers`, `Bookmarks API Handler`, `Series Acquisition Tests`?**
_High betweenness centrality (0.030) - this node is a cross-community bridge._
_High betweenness centrality (0.027) - this node is a cross-community bridge._
- **Are the 47 inferred relationships involving `testConfig()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverNeverEchoesNonImage()`) actually correct?**
_`testConfig()` has 47 INFERRED edges - model-reasoned connections that need verification._
- **Are the 8 inferred relationships involving `newWebTestServer()` (e.g. with `TestListRendersAcquiredCover()` and `TestPublicCoverRejectsUnknownAddress()`) actually correct?**
@@ -559,4 +564,4 @@ _Questions this graph is uniquely positioned to answer:_
- **Are the 12 inferred relationships involving `newTestStore()` (e.g. with `TestAcquireDoesNotBlockTheWrite()` and `TestAcquireFailureLeavesTheBookmarkIntact()`) actually correct?**
_`newTestStore()` has 12 INFERRED edges - model-reasoned connections that need verification._
- **What connects `bookmarkmanager/backend`, `ctxKey`, `loginView` to the rest of the system?**
_553 weakly-connected nodes found - possible documentation gaps or missing edges._
_557 weakly-connected nodes found - possible documentation gaps or missing edges._