fix(attribution): sum attribution in the database, page the daily rollup (prod OOM) #16

2026-10-16T19:36:37Z

KofTwentyTwo commented

2026-10-16 19:36:37 +00:00

(Migrated from github.com)

Why

On 2026-10-16 at 19:20:18Z the kof22 web pod website-backend-74bdc7754c-dc7wc died with OutOfMemoryError: Java heap space. One admin request (req=14ad5fa6…, Database Performance home) rendered the Attribution panel, which ran an unbounded QueryAction on db_perf_attribution for the last 24 hours:

19:19:23 Large number of records in QueryOutputList  noRecords=50000   table=db_perf_attribution
19:19:24 ...                                          noRecords=100000
19:19:28 ...                                          noRecords=250000
   at DbPerfStore.query(DbPerfStore.java:65)
   at DbPerfWidgetRenderer.attribution(DbPerfWidgetRenderer.java:288)
19:20:18 Terminating due to java.lang.OutOfMemoryError: Java heap space

Container working set was flat at ~620 MiB for 35 minutes, then jumped to 1,849 MiB in the minute of the crash. Every other request on the pod (4-row Site API queries) took 4-6 s during the GC thrash. The table holds ~571,000 rows for the last 24 h (538 MB in PostgreSQL; ~30,000 rows/hour since the web/worker split), so the widget can never fit in a 1.5 GB heap.

Two worker paths have the same shape:

dbPerfAnalyze (hourly) loads the whole day through RuleContext.window("attribution", 1). On 2026-10-16 the worker working set jumped from about 660 MiB to about 1.84 GiB at every :07-:08 run.
dbPerfRollupAndPurge logged 50k and then 100k attribution rows at 08:15 while it rolled up a day.

What

DbPerfWidgetRenderer.attribution() uses AggregateAction: SUM(statements), SUM(totalMs) grouped by unitKind, unitName, tableName, ordered by time, capped at 5,000 groups (prod has ~1,000). The heap holds aggregate rows only.
DbPerfStore.forEachPage(table, filter, pageSize, consumer): keyset-paged reads (id > last, ordered, limit).
RollupService.rollupAttribution folds the day in 1,000-row pages, so memory is bounded by distinct dimension keys (~8,500 in prod), not by the day's detail rows.
RuleContext.window("attribution", days) returns AggregateAction sums grouped by unitKind, unitName, tableName and operation (the only fields the rules read), not the detail rows.

Tests

AttributionBoundedReadTest puts a post-query customizer on db_perf_attribution that counts every record QQQ materializes:

the widget loads zero detail rows for 3,000 inserted rows and the sums, shares and top table are still right;
the 24-hour window still excludes older detail;
the analysis RuleContext window loads zero detail rows and returns the same totals;
forEachPage never returns more than the page size and sees each row once;
the rollup over 2,500 rows never holds more than BATCH_SIZE at once and its sums cover every row across page boundaries.

./mvnw verify passes (24 classes, JaCoCo gate).

Follow-up (not in this PR)

SnapshotEvaluator.spikes loads 7 days of db_perf_query_snapshot (~33k rows/day in kof22, so ~230k rows by day 7), plus every db_perf_statement row, on each snapshot. That is bounded by time, but large. It should read per-statement history in pages, or select only the fields it needs.

Attribution detail retention is the shared detailRetentionDays (default 30). At kof22's rate that is ~20M rows / ~16 GB. Consider a separate, shorter attribution retention or coarser flush keys.

🤖 Generated with Claude Code

## Why On 2026-10-16 at 19:20:18Z the kof22 web pod `website-backend-74bdc7754c-dc7wc` died with `OutOfMemoryError: Java heap space`. One admin request (`req=14ad5fa6…`, Database Performance home) rendered the **Attribution** panel, which ran an unbounded `QueryAction` on `db_perf_attribution` for the last 24 hours: ``` 19:19:23 Large number of records in QueryOutputList noRecords=50000 table=db_perf_attribution 19:19:24 ... noRecords=100000 19:19:28 ... noRecords=250000 at DbPerfStore.query(DbPerfStore.java:65) at DbPerfWidgetRenderer.attribution(DbPerfWidgetRenderer.java:288) 19:20:18 Terminating due to java.lang.OutOfMemoryError: Java heap space ``` Container working set was flat at ~620 MiB for 35 minutes, then jumped to 1,849 MiB in the minute of the crash. Every other request on the pod (4-row Site API queries) took 4-6 s during the GC thrash. The table holds ~571,000 rows for the last 24 h (538 MB in PostgreSQL; ~30,000 rows/hour since the web/worker split), so the widget can never fit in a 1.5 GB heap. Two worker paths have the same shape: - `dbPerfAnalyze` (hourly) loads the whole day through `RuleContext.window("attribution", 1)`. On 2026-10-16 the worker working set jumped from about 660 MiB to about 1.84 GiB at every :07-:08 run. - `dbPerfRollupAndPurge` logged 50k and then 100k attribution rows at 08:15 while it rolled up a day. ## What - `DbPerfWidgetRenderer.attribution()` uses `AggregateAction`: `SUM(statements)`, `SUM(totalMs)` grouped by `unitKind, unitName, tableName`, ordered by time, capped at 5,000 groups (prod has ~1,000). The heap holds aggregate rows only. - `DbPerfStore.forEachPage(table, filter, pageSize, consumer)`: keyset-paged reads (id > last, ordered, limit). - `RollupService.rollupAttribution` folds the day in 1,000-row pages, so memory is bounded by distinct dimension keys (~8,500 in prod), not by the day's detail rows. - `RuleContext.window("attribution", days)` returns `AggregateAction` sums grouped by unitKind, unitName, tableName and operation (the only fields the rules read), not the detail rows. ## Tests `AttributionBoundedReadTest` puts a post-query customizer on `db_perf_attribution` that counts every record QQQ materializes: - the widget loads **zero** detail rows for 3,000 inserted rows and the sums, shares and top table are still right; - the 24-hour window still excludes older detail; - the analysis `RuleContext` window loads **zero** detail rows and returns the same totals; - `forEachPage` never returns more than the page size and sees each row once; - the rollup over 2,500 rows never holds more than `BATCH_SIZE` at once and its sums cover every row across page boundaries. `./mvnw verify` passes (24 classes, JaCoCo gate). ## Follow-up (not in this PR) - `SnapshotEvaluator.spikes` loads 7 days of `db_perf_query_snapshot` (~33k rows/day in kof22, so ~230k rows by day 7), plus every `db_perf_statement` row, on each snapshot. That is bounded by time, but large. It should read per-statement history in pages, or select only the fields it needs. Attribution detail retention is the shared `detailRetentionDays` (default 30). At kof22's rate that is ~20M rows / ~16 GB. Consider a separate, shorter attribution retention or coarser flush keys. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-db-performance#16