feat(compare): compare, distribution shift, correlation, scatter and crosstabs #8

2026-10-14T00:37:08Z

KofTwentyTwo commented

2026-10-14 00:37:08 +00:00

(Migrated from github.com)

Summary

Phase (d) of docs/DATA-ANALYSIS.md (§6). Stacked on #6 (pivot), which is stacked on #5. Draft until those merge after RC.6.

Extension 1.2.0, all as the user, FULL fields only, with drills and the privacy guard:

Endpoint	What it does
`POST /tables/{t}/compare`	Filter A vs B (or vs the whole table), per field: side-by-side metrics, delta, % delta, and significance. Categorical fields use chi-square homogeneity over the top 10 plus Other; numeric fields use Welch's t from pushed-down COUNT/AVG/VAR_SAMP. Two proportions use a z test.
`POST /tables/{t}/shift`	Category shares or quartiles per time bucket, and PSI between the first and last complete bucket
`POST /tables/{t}/correlation`	Pearson matrix (2–12 numeric fields) pushed down with `CORR`, with n per pair; heatmap
`POST /tables/{t}/scatter`	Sampled points (max 2,000) with a least-squares trend line. No point drills or points under the privacy guard.
`POST /tables/{t}/crosstab`	Counts with row and column shares, chi-square independence and Cramér's V; small cells merged or suppressed

The stats package is pure Java, including the regularized incomplete beta and gamma. It applies validity rules (expected count < 5 is merged, otherwise "not applicable") and writes plain sentences ("A and B differ (p = 0.003)"). Nothing is called significant at p ≥ 0.05.

Test plan

./mvnw verify with Docker: green on memory, H2 and Postgres (202+ tests). JaCoCo ~98% instructions / 100% classes.
Statistics against scipy reference values (Welch t, df and p; chi-square; PSI; z)
Compare (numeric and categorical), shift, correlation vs a Java Pearson, scatter limit and trend, crosstab: exact numbers on the fixture, drill equivalence for every count, privacy guard, field access (400), 403, HTTP

🤖 Generated with Claude Code

## Summary Phase (d) of `docs/DATA-ANALYSIS.md` (§6). **Stacked on #6 (pivot), which is stacked on #5.** Draft until those merge after RC.6. Extension 1.2.0, all as the user, FULL fields only, with drills and the privacy guard: | Endpoint | What it does | |---|---| | `POST /tables/{t}/compare` | Filter A vs B (or vs the whole table), per field: side-by-side metrics, delta, % delta, and significance. Categorical fields use chi-square homogeneity over the top 10 plus Other; numeric fields use Welch's t from pushed-down COUNT/AVG/VAR_SAMP. Two proportions use a z test. | | `POST /tables/{t}/shift` | Category shares or quartiles per time bucket, and PSI between the first and last complete bucket | | `POST /tables/{t}/correlation` | Pearson matrix (2–12 numeric fields) pushed down with `CORR`, with n per pair; heatmap | | `POST /tables/{t}/scatter` | Sampled points (max 2,000) with a least-squares trend line. No point drills or points under the privacy guard. | | `POST /tables/{t}/crosstab` | Counts with row and column shares, chi-square independence and Cramér's V; small cells merged or suppressed | The `stats` package is pure Java, including the regularized incomplete beta and gamma. It applies validity rules (expected count < 5 is merged, otherwise "not applicable") and writes plain sentences ("A and B differ (p = 0.003)"). Nothing is called significant at p ≥ 0.05. ## Test plan - [x] `./mvnw verify` with Docker: green on memory, H2 and Postgres (202+ tests). JaCoCo ~98% instructions / 100% classes. - [x] Statistics against scipy reference values (Welch t, df and p; chi-square; PSI; z) - [x] Compare (numeric and categorical), shift, correlation vs a Java Pearson, scatter limit and trend, crosstab: exact numbers on the fixture, drill equivalence for every count, privacy guard, field access (400), 403, HTTP 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-data-analysis#8