feat(api): /qqq/v1 data-analysis extension, core pushdown backend, analyze permission #5

2026-10-13T22:44:19Z

KofTwentyTwo commented

2026-10-13 22:44:19 +00:00

(Migrated from github.com)

Summary

Phase (b) part 2 of docs/DATA-ANALYSIS.md (§3, §4.3, §8): the spec'd UI API, core pushdown, and the analyze permission through core.

UI API extension data-analysis 1.0.0 (/qqq/v1/qbit/data-analysis, the #1128 extension point). It is registered only when qqq-middleware-javalin is present.

Method	Path
GET	`/tables/{table}/capabilities` (200 with `canAnalyze=false` when not allowed, so the UI can hide the action)
POST	`/tables/{table}/column-profile`
POST	`/tables/{table}/distinct-values`
POST	`/tables/{table}/table-profile`
POST	`/tables/{table}/duplicates`
POST	`/runs` (start an exact run)
GET	`/runs/{runId}`

Schemas: 26 DataAnalysis…V1 schemas. Filters reuse core's filter schema, and every operation has an assistant-oriented summary and description.
Contract: the fragment is committed at src/main/resources/openapi/qqq-v1/data-analysis.yaml, and a test regenerates it and fails on drift (-Dopenapi.write=true rewrites it).
Errors: 403 for permission, 400 for a bad field or body, 404 for an unknown table.
Analyze permission: <table>.analyze is registered through core TableFeaturePermissionsMetaData (#1138) and checked with PermissionsHelper.hasTableFeaturePermission, so it shows in the authentik policy editor (qbit-authentik#19). The stand-in is gone.
CoreAnalysisBackend:
- Pushes percentiles, stddev, null/blank counts, numeric and date buckets and lengths down through #1139. Patterns and length histograms stay streamed.
- Backends without capabilities keep the streamed stand-in.
Scalable estimates:
- Native backends: aggregates run with withSamplePercent, counts are scaled to the population ("≈"), ratios are unscaled, and the row count is the true total.
- Stand-in: uses one primary-key <= bound for "first N rows by id".
- No IN lists, so the Postgres bind limit is never hit.
- Estimate drills cover the full population; exact results keep strict drill equivalence.

Requires QQQ 4.1.0-RC.6, with #1128, #1138, #1139 and #1145 (H2 default-mode grouping). The pom targets RC.6, so this stays a draft until RC.6 is published. Local builds use -Dqqq.snapshot.version=4.1.0-DATAANALYSIS-SNAPSHOT, which is develop plus #1145.

Test plan

./mvnw verify locally against the snapshot, with Docker: 123 tests, all passing, including 21 on Postgres 17. A 50,000-row estimate regression covers the bind limit. JaCoCo 98% instructions / 100% classes; checkstyle clean.
HTTP server tests:
- every endpoint's happy path;
- 403 without .analyze, and canAnalyze=false;
- 400 for a field behind a field security lock;
- a drill filter from a response, posted to core /qqq/v1/table/{t}/count, returns the same number;
- an async run completes;
- the operations appear in /qqq/v1/openapi.json and in UiApiInvoker.listOperations() (assistant tools).
The engine contract suite runs for both backends on memory, H2 (default and PostgreSQL mode) and Postgres
CI after RC.6 is published

🤖 Generated with Claude Code

## Summary Phase (b) part 2 of `docs/DATA-ANALYSIS.md` (§3, §4.3, §8): the spec'd UI API, core pushdown, and the analyze permission through core. **UI API extension `data-analysis` 1.0.0** (`/qqq/v1/qbit/data-analysis`, the #1128 extension point). It is registered only when `qqq-middleware-javalin` is present. | Method | Path | |---|---| | GET | `/tables/{table}/capabilities` (200 with `canAnalyze=false` when not allowed, so the UI can hide the action) | | POST | `/tables/{table}/column-profile` | | POST | `/tables/{table}/distinct-values` | | POST | `/tables/{table}/table-profile` | | POST | `/tables/{table}/duplicates` | | POST | `/runs` (start an exact run) | | GET | `/runs/{runId}` | - **Schemas:** 26 `DataAnalysis…V1` schemas. Filters reuse core's filter schema, and every operation has an assistant-oriented summary and description. - **Contract:** the fragment is committed at `src/main/resources/openapi/qqq-v1/data-analysis.yaml`, and a test regenerates it and fails on drift (`-Dopenapi.write=true` rewrites it). - **Errors:** 403 for permission, 400 for a bad field or body, 404 for an unknown table. - **Analyze permission:** `<table>.analyze` is registered through core `TableFeaturePermissionsMetaData` (#1138) and checked with `PermissionsHelper.hasTableFeaturePermission`, so it shows in the authentik policy editor (qbit-authentik#19). The stand-in is gone. - **`CoreAnalysisBackend`:** - Pushes percentiles, stddev, null/blank counts, numeric and date buckets and lengths down through #1139. Patterns and length histograms stay streamed. - Backends without capabilities keep the streamed stand-in. - **Scalable estimates:** - Native backends: aggregates run with `withSamplePercent`, counts are scaled to the population ("≈"), ratios are unscaled, and the row count is the true total. - Stand-in: uses one primary-key `<=` bound for "first N rows by id". - No IN lists, so the Postgres bind limit is never hit. - Estimate drills cover the full population; exact results keep strict drill equivalence. **Requires QQQ 4.1.0-RC.6**, with #1128, #1138, #1139 and **#1145** (H2 default-mode grouping). The pom targets RC.6, so this stays a **draft until RC.6 is published**. Local builds use `-Dqqq.snapshot.version=4.1.0-DATAANALYSIS-SNAPSHOT`, which is develop plus #1145. ## Test plan - [x] `./mvnw verify` locally against the snapshot, with Docker: **123 tests, all passing, including 21 on Postgres 17**. A 50,000-row estimate regression covers the bind limit. JaCoCo 98% instructions / 100% classes; checkstyle clean. - [x] HTTP server tests: - every endpoint's happy path; - 403 without `.analyze`, and `canAnalyze=false`; - 400 for a field behind a field security lock; - a drill filter from a response, posted to core `/qqq/v1/table/{t}/count`, returns the same number; - an async run completes; - the operations appear in `/qqq/v1/openapi.json` and in `UiApiInvoker.listOperations()` (assistant tools). - [x] The engine contract suite runs for both backends on memory, H2 (default and PostgreSQL mode) and Postgres - [ ] CI after RC.6 is published 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-data-analysis#5