feat(index): never index PII/PHI-classified or secured fields (secured data §15) #55

2026-10-14T16:13:30Z

KofTwentyTwo commented

2026-10-14 16:13:30 +00:00

(Migrated from github.com)

Secured-data spec §15: Quick Search must never index PII/PHI-classified or secured fields.

Rule, in one place (QuickSearchFieldRules, on the released QQQ 4.1.0-RC.6 APIs, no SNAPSHOT):

isProtected(table, field) is true when the field is classified PII or PHI (AiDataClassifications.ofField, falling back to the table's classification), has a field security lock (a shared index can't honour per-user field access), is isSensitive, or is a masked/password type.
whyNotIndexable refuses protected fields in addition to the existing reasons (hidden, blob, boolean and so on).

Where it applies: real-time indexing, basepull and full reindex document building, the configured searchable and snippet fields (protected entries are dropped with a once-per-table warning that never logs a value), config writes and the primer, record labels (a label built from a protected field falls back), the version/timestamp field, delete ids (a table with a protected primary key, such as a PHI table, isn't indexed at all), results and highlights, and diagnostics. Results from tables that have a protected field are rebuilt from allowed values only, so documents indexed before this change can't leak through old labels or snippets. Tables whose only non-indexable fields are harmless (booleans, blobs) keep their labels and aren't filtered.

Canary: ClassifiedDataCanaryTest gives a test table a PHI-classified and a PII field holding marker strings, runs real-time indexing, basepull and search through the captured OpenSearch seam, and asserts the markers appear in no request body, result or snippet, while an unclassified value is indexed.

RC.7 follow-up (noted in the javadoc): when the qbit moves to QQQ 4.1.0-RC.7, also honour core ClassifiedValues.isClassified (the host's protector, qqq#1182) and FieldAccessHelper lock tiers (qqq#1172). I built on RC.6 because RC.7 isn't released yet (qqq#1185 is open), and everything above uses RC.6 APIs.

Owner step: existing indexes keep whatever they already hold until reindexed. Results are filtered now, but run a full reindex after deploy to purge stored classified values.

Tests: mvnw -o -B verify green: 483 unit tests, with the architecture and coverage gates. The 41 Docker integration tests were skipped locally; CI runs them.

🤖 Generated with Claude Code

Secured-data spec §15: Quick Search must never index PII/PHI-classified or secured fields. **Rule, in one place** (`QuickSearchFieldRules`, on the released QQQ 4.1.0-RC.6 APIs, no SNAPSHOT): - `isProtected(table, field)` is true when the field is classified PII or PHI (`AiDataClassifications.ofField`, falling back to the table's classification), has a field security lock (a shared index can't honour per-user field access), is `isSensitive`, or is a masked/password type. - `whyNotIndexable` refuses protected fields in addition to the existing reasons (hidden, blob, boolean and so on). **Where it applies:** real-time indexing, basepull and full reindex document building, the configured searchable and snippet fields (protected entries are dropped with a once-per-table warning that never logs a value), config writes and the primer, record labels (a label built from a protected field falls back), the version/timestamp field, delete ids (a table with a protected primary key, such as a PHI table, isn't indexed at all), results and highlights, and diagnostics. Results from tables that have a protected field are rebuilt from allowed values only, so documents indexed before this change can't leak through old labels or snippets. Tables whose only non-indexable fields are harmless (booleans, blobs) keep their labels and aren't filtered. **Canary:** `ClassifiedDataCanaryTest` gives a test table a PHI-classified and a PII field holding marker strings, runs real-time indexing, basepull and search through the captured OpenSearch seam, and asserts the markers appear in no request body, result or snippet, while an unclassified value is indexed. **RC.7 follow-up** (noted in the javadoc): when the qbit moves to QQQ 4.1.0-RC.7, also honour core `ClassifiedValues.isClassified` (the host's protector, qqq#1182) and `FieldAccessHelper` lock tiers (qqq#1172). I built on RC.6 because RC.7 isn't released yet (qqq#1185 is open), and everything above uses RC.6 APIs. **Owner step:** existing indexes keep whatever they already hold until reindexed. Results are filtered now, but run a full reindex after deploy to purge stored classified values. **Tests:** `mvnw -o -B verify` green: 483 unit tests, with the architecture and coverage gates. The 41 Docker integration tests were skipped locally; CI runs them. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-quick-search#55