From 4e20070ae19178c8c321b14af232873beac8b838 Mon Sep 17 00:00:00 2001 From: James Maes Date: Fri, 20 Nov 2026 09:01:39 -0600 Subject: [PATCH 1/2] feat: a create date and a modify date on every table Every table of the QBit has a create date and a modify date: DATE_TIME, not editable, with core's CREATE_DATE and MODIFY_DATE behaviors, in the columns create_date and modify_date. Seven tables did not have them: reveal grant, break-glass session, access log, wipe run, wipe marker, legal hold and audit key. Their fields are named createDate and modifyDate. Classification, policy and retention rule already had create_date and modify_date, set by the table customizer. They keep those field names, because a rename would change what a host reads and what their audits record, and the two fields are now not editable and have the behaviors. Writes. Every write of the QBit is a core insert or update action, so the dates are in the statement that writes the row. The table customizer sets them, as it did for the three tables: a new row gets both, equal, from the QBit's clock, and a changed row a new modify date. The access-log writer still runs one insert per row and no other statement, lock or transaction. The access log and wipe runs are only ever inserted, so their dates stay equal. Core uses a DATE_TIME field with the MODIFY_DATE behavior as the record version of its table. That now holds for all ten tables: on an update core sets the modify date itself, after the customizer has run, and an update that sends the version it read is refused when the row has changed since. Updates that send no version, the QBit's own among them, behave as before. Schema. docs/ddl/secured-data-postgres.sql has a new last block that adds the two columns to the seven tables: nullable, defaulting to the current time, with the rows that were already there given the row's own time. No CREATE TABLE statement is edited. An existing database runs that block alone, before this release is deployed. The block rewrites no table, can be run while the release before this one is running (its inserts name neither column and get both from the default) and can be run again. This release does not run without the columns. README: "Upgrading an existing database". Unchanged. The dates have no classification, policy or assistant exposure of their own: they are CONFIDENTIAL with their table and get the tier of its other fields. Nothing signed or verified covers them: a wrapped data key is bound to its table, field and record id, and the fingerprints of a log row are made from search values. A changed policy or classification writes the same core audit, field by field, and the same POLICY_CHANGE or CLASSIFICATION_CHANGE row. Tests. SecuredDataStandardDatesTest: every table the producer declares has the two dates (the tables are taken from the instance, so a table added later without them fails); the dates resolve to the tier of the other fields of their table for five kinds of session; every writer sets both and only a change moves the modify date (access-log writer, policy, classification, legal hold, reveal, break-glass start, end and expiry on the memory backend; audit key, wipe run, wipe marker and retention rule in the shipped schema); the access-log writer runs exactly one insert per row; a changed policy and classification write the audit and log rows they wrote before; rows written before the columns existed are unchanged by the upgrade block, their wrapped keys unwrap to the same data keys and their audit token verifies the same value and no other. SecuredDataDdlTest: the shipped DDL gives every declared table the two columns, the upgrade block adds them where they were missing, an insert that names neither succeeds, and the block can be run twice. Seven of the eight new tests fail without the change; the eighth is the record of what a changed policy wrote before and passes on both sides. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 7 + README.md | 30 +- docs/ddl/secured-data-postgres.sql | 49 ++ .../secureddata/metadata/RetentionTables.java | 26 +- .../metadata/SecuredDataTableCustomizers.java | 33 +- .../metadata/SecuredDataTables.java | 74 +- .../qbits/secureddata/SecuredDataDdlTest.java | 69 ++ .../SecuredDataQBitProducerTest.java | 4 +- .../SecuredDataStandardDatesTest.java | 774 ++++++++++++++++++ 9 files changed, 1043 insertions(+), 23 deletions(-) create mode 100644 src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 927316c..4b51317 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ All notable changes are documented here, following [Keep a Changelog](https://ke ## [Unreleased] +### Added: a create date and a modify date on every table +- Every table has a create date and a modify date: `DATE_TIME`, not editable, with QQQ's `CREATE_DATE` and `MODIFY_DATE` behaviors, in the columns `create_date` and `modify_date`. Seven tables did not have them: reveal grant, break-glass session, access log, wipe run, wipe marker, legal hold and audit key. Their fields are named `createDate` and `modifyDate`. Classification, policy and retention rule already had `create_date` and `modify_date`; they keep those field names, and the two fields are now not editable and have the behaviors. +- A new row gets both dates, equal, in the insert that writes it. A changed row keeps its create date and gets a later modify date. The access log and wipe runs are only ever inserted, so their dates stay equal. The access-log writer still runs one insert per row, with no other statement, lock or transaction. +- **Upgrade: run the DDL first.** Before deploying this release on an existing database, run the last block of `docs/ddl/secured-data-postgres.sql` (from the line `-- Standard dates:` to the end). It adds the two nullable columns to the seven tables, defaults them to the current time and gives existing rows the row's own time. It rewrites no table and can be run while the release before this one is running, which goes on working with the columns present. This release does not run without them. See "Upgrading an existing database" in the README. +- The dates have no classification, policy or assistant exposure of their own, and nothing that is signed or verified covers them. Rows written before the columns existed are unchanged apart from the two columns: wrapped data keys unwrap as before and audit tokens verify as before. +- QQQ uses a field with the `MODIFY_DATE` behavior as the record version of its table. That now holds for every table of the QBit, classification, policy and retention rule included: on an update core sets the modify date itself, and an update that sends the version it read is refused if the row has changed since. Updates that send no version, the QBit's own among them, behave as before. + ### Fixed: a reveal and the access-log writer waiting for each other - A reveal, or the start, end or expiry of a break-glass session, no longer deadlocks with a thread that asks for a classification while it holds another lock of this QBit. The access-log writer is such a thread: it holds the anomaly counters while it inserts a row, and that insert asks for classifications (it counts earlier rows and writes an audit, both queries with criteria). A reveal holds the runtime's monitor and then inserts its own access-log row. When both happened at once, neither finished, and neither did any later request, because every request asks for a classification. A person who opens a record with a masked field and reveals it straight away does both at once. - The rule rows a node keeps (classifications, active retention rules) have a lock of their own, held only while they are answered or read. Asking for a classification no longer takes the monitor that reveal and break-glass hold for their whole operation. diff --git a/README.md b/README.md index fb9f5fc..1ddc692 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,29 @@ new SecuredDataQBitProducer() Register the host backend and application tables before producing the QBit. Enable core permissions (`qInstance.withEnforcePermissionsInCoreActions(true)`) at application entry points. Only one secured-data runtime is supported per instance. A namespace labels the QBit identity; table names remain fixed. -The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata; an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode. +The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata (the fields `createDate` and `modifyDate` are the columns `create_date` and `modify_date`); an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode. + +### Upgrading an existing database: create and modify dates + +Every table has the columns `create_date` and `modify_date`, and the QBit reads and writes them on all ten tables. Seven tables were first created without them: `secured_data_reveal_grant`, `secured_data_break_glass`, `secured_data_access_log`, `secured_data_wipe_run`, `secured_data_wipe_marker`, `secured_data_legal_hold` and `secured_data_audit_key`. + +Before deploying a release that has the two dates on every table, run the last block of [the DDL](docs/ddl/secured-data-postgres.sql) on the existing database: everything from the line `-- Standard dates:` to the end of the file. The order matters. The release before it runs unchanged once the columns are there. The release with the dates does not run without them: its reads of the seven tables fail, and with them every request that needs a field decision. The block does three things to each of the seven tables: + +- It adds the two columns, nullable. That changes the catalog only; the table is not rewritten. +- It gives each column a default of the current time, so a release that does not know the columns goes on inserting while both releases run. +- It gives the rows that were already there the row's own time (below). The `UPDATE` touches only rows whose `create_date` is null, so the block can be run again and then changes nothing. It takes row locks only, and on a large access log it can be run in ranges of `id`. + +| Table | `create_date` of an existing row | `modify_date` of an existing row | +|---|---|---| +| `secured_data_reveal_grant` | `granted_at` | `granted_at` | +| `secured_data_break_glass` | `started_at` | `ended_at`, else `started_at` | +| `secured_data_access_log` | `created_at` | `created_at` | +| `secured_data_wipe_run` | `finished_at`, else `started_at` | the same | +| `secured_data_wipe_marker` | `wiped_at` | `wiped_at` | +| `secured_data_legal_hold` | `placed_at`, else the time of the upgrade | `released_at`, else `placed_at`, else the time of the upgrade | +| `secured_data_audit_key` | `created_at` | `created_at` | + +Nothing else changes in a row: wrapped data keys, search fingerprints and every other column are left as they are. ## Tables, screens and permissions @@ -44,6 +66,12 @@ The host supplies database connection metadata. Physical table names and field b Every table has audit rules, explicit record sections and CONFIDENTIAL code classification. The Secured Data app navigates to Classification, Policy, Access log and Break-glass session. Reveal grants are a read-only child list joined by `access_log_id`; no process or home widget appears in navigation. The classification target `(table_name, field_name)` is unique, including the null field used for a table-wide override. +Every table has a create date and a modify date: date-time fields that nobody edits, in the columns `create_date` and `modify_date`. The fields are named `createDate` and `modifyDate`. Classification, Policy and Retention rules had the two dates before the other tables did and keep the field names `create_date` and `modify_date`. A new row gets both, equal. A changed row keeps its create date and gets a later modify date. Every write of the QBit is a QQQ insert or update action, so the dates are in the one statement that writes the row, the access-log writer's insert included. The access log and wipe runs are only ever inserted, so their two dates stay equal. + +The two dates have no classification of their own. They are CONFIDENTIAL with the rest of their table, are given the tier of its other fields, and are closed to the assistant with it. Nothing that is signed or verified covers them: a wrapped data key is bound to its table, field and record id, and the search fingerprints of a log row are made from the search values. + +The modify date has QQQ's modify-date behavior, and QQQ uses such a field as the record version of its table. On an update core sets it to the current time (or one millisecond after the stored value, if that is later). When a caller sends the version it read, core refuses the update if the row has changed since. The QBit's own updates send no version. + `secured_data.admin` gates configuration editing and activity reads through a standard QQQ custom permission checker. It gives no implicit access to classified fields on application tables. `secured_data.hasAccess` is the catalog base permission; the custom checker requires admin. Activity writes are refused for ordinary users even when they hold admin. Customizers also protect system-only writes, and update/delete capabilities are disabled on the access log, including system calls. Both table features are registered for **all tables**: `secured_data_full` (Secured data: full) and `secured_data_reveal` (Secured data: reveal). This deliberate catalog choice supports new admin classification rows without a restart. Core's permission-base, READ prerequisite and NOT_PROTECTED semantics are preserved: NOT_PROTECTED application tables grant their features without an explicit permission. Use protected table permission rules when field policies must restrict users. Feature grants have no standard WRITE/admin implication. diff --git a/docs/ddl/secured-data-postgres.sql b/docs/ddl/secured-data-postgres.sql index 1b220ef..8c51664 100644 --- a/docs/ddl/secured-data-postgres.sql +++ b/docs/ddl/secured-data-postgres.sql @@ -138,3 +138,52 @@ CREATE TABLE secured_data_audit_key ( CREATE INDEX secured_data_hold_lookup ON secured_data_legal_hold (table_name, status, record_id); CREATE INDEX secured_data_retention_lookup ON secured_data_retention_rule (table_name, is_active); CREATE INDEX secured_data_wipe_rule_lookup ON secured_data_wipe_marker (rule_id); + +-- Standard dates: create_date and modify_date on the seven tables created above without them. +-- A new database runs the whole file. An existing database runs this block alone, BEFORE the release +-- that writes the two columns is deployed. The columns are nullable and default to the current time, so +-- a release that does not know them goes on inserting while both run. Adding a nullable column and +-- setting a default change the catalog only (no table rewrite). Each UPDATE gives the rows that were +-- already there the row's own time, once: it touches only rows whose create_date is still null, takes +-- row locks only, and on a large access log can be run in ranges of id. +ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_reveal_grant ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_reveal_grant ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_reveal_grant SET create_date = granted_at, modify_date = granted_at WHERE create_date IS NULL; + +ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_break_glass ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_break_glass ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_break_glass SET create_date = started_at, modify_date = COALESCE(ended_at, started_at) WHERE create_date IS NULL; + +ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_access_log ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_access_log ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_access_log SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL; + +ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_wipe_run ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_wipe_run ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_wipe_run SET create_date = COALESCE(finished_at, started_at), modify_date = COALESCE(finished_at, started_at) WHERE create_date IS NULL; + +ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_wipe_marker ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_wipe_marker ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_wipe_marker SET create_date = wiped_at, modify_date = wiped_at WHERE create_date IS NULL; + +ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_legal_hold ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_legal_hold ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_legal_hold SET create_date = COALESCE(placed_at, CURRENT_TIMESTAMP), modify_date = COALESCE(released_at, placed_at, CURRENT_TIMESTAMP) WHERE create_date IS NULL; + +ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; +ALTER TABLE secured_data_audit_key ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; +ALTER TABLE secured_data_audit_key ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP; +UPDATE secured_data_audit_key SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL; diff --git a/src/main/java/com/kingsrook/qbits/secureddata/metadata/RetentionTables.java b/src/main/java/com/kingsrook/qbits/secureddata/metadata/RetentionTables.java index 37b8f1b..5bb3da1 100644 --- a/src/main/java/com/kingsrook/qbits/secureddata/metadata/RetentionTables.java +++ b/src/main/java/com/kingsrook/qbits/secureddata/metadata/RetentionTables.java @@ -31,12 +31,16 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory; import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.AUDIT_KEY; +import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.CREATE_DATE; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.LEGAL_HOLD; +import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.MODIFY_DATE; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.RETENTION_RULE; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_MARKER; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_RUN; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.base; +import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.createDate; import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.field; +import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.modifyDate; /******************************************************************************* ** Retention policies, value-free history, legal holds and mutable audit keys. @@ -74,8 +78,8 @@ public final class RetentionTables .withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true)) .withField(field("last_run_at", QFieldType.DATE_TIME).withIsEditable(false)) .withField(field("last_run_count", QFieldType.INTEGER).withIsEditable(false)) - .withField(field("create_date", QFieldType.DATE_TIME)) - .withField(field("modify_date", QFieldType.DATE_TIME)) + .withField(createDate("create_date")) + .withField(modifyDate("modify_date")) .withField(field("modified_by", QFieldType.STRING).withMaxLength(255)); rule.withSection(SectionFactory.defaultT1("id", "name", "table_name", "field_names")); rule.withSection(SectionFactory.defaultT2("age_field_name", "age_days", "action", "overwrite_text", "audit_treatment", "archive_storage_name", "batch_size", "is_active")); @@ -92,8 +96,11 @@ public final class RetentionTables .withField(field("action", QFieldType.STRING).withMaxLength(40).withIsRequired(true)) .withField(field("wiped_at", QFieldType.DATE_TIME).withIsRequired(true)) .withField(field("run_id", QFieldType.INTEGER).withIsRequired(true)) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)) .withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name"))); marker.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "rule_id", "action", "wiped_at", "run_id")); + marker.withSection(SectionFactory.defaultT3(CREATE_DATE, MODIFY_DATE)); QTableMetaData hold = base(config, LEGAL_HOLD, "Legal hold") .withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables")) .withField(field("record_id", QFieldType.STRING).withMaxLength(255)) @@ -102,9 +109,11 @@ public final class RetentionTables .withField(field("placed_at", QFieldType.DATE_TIME)) .withField(field("released_by", QFieldType.STRING).withMaxLength(255)) .withField(field("released_at", QFieldType.DATE_TIME)) - .withField(field("status", QFieldType.STRING).withMaxLength(20).withDefaultValue("ACTIVE")); + .withField(field("status", QFieldType.STRING).withMaxLength(20).withDefaultValue("ACTIVE")) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)); hold.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "reason", "status")); - hold.withSection(SectionFactory.defaultT3("placed_by", "placed_at", "released_by", "released_at")); + hold.withSection(SectionFactory.defaultT3("placed_by", "placed_at", "released_by", "released_at", CREATE_DATE, MODIFY_DATE)); QTableMetaData run = base(config, WIPE_RUN, "Wipe run") .withField(field("rule_id", QFieldType.INTEGER).withIsRequired(true)) .withField(field("started_at", QFieldType.DATE_TIME).withIsRequired(true)) @@ -115,9 +124,11 @@ public final class RetentionTables .withField(field("wiped_count", QFieldType.INTEGER)) .withField(field("error_count", QFieldType.INTEGER)) .withField(field("record_ids", QFieldType.STRING).withMaxLength(4000)) - .withField(field("message", QFieldType.STRING).withMaxLength(500)); + .withField(field("message", QFieldType.STRING).withMaxLength(500)) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)); run.withSection(SectionFactory.defaultT1("id", "rule_id", "status", "eligible_count", "held_count", "wiped_count", "error_count")); - run.withSection(SectionFactory.defaultT3("started_at", "finished_at", "record_ids", "message")); + run.withSection(SectionFactory.defaultT3("started_at", "finished_at", "record_ids", "message", CREATE_DATE, MODIFY_DATE)); QTableMetaData key = base(config, AUDIT_KEY, "Audit key") .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE)) .withoutCapabilities(Capability.TABLE_EXPORT) @@ -127,8 +138,11 @@ public final class RetentionTables .withField(field("wrapped_key", QFieldType.TEXT).withIsRequired(true).withIsSensitive(true).withIsHidden(true) .withFieldSecurityLock(new FieldSecurityLock().withSecurityKeyType(KEY_LOCK).withDefaultBehavior(FieldSecurityLock.Behavior.DENY).withOverrideValues(List.of(KEY_LOCK_NEVER_GRANTED)))) .withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true)) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)) .withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name"))); key.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "created_at")); + key.withSection(SectionFactory.defaultT3(CREATE_DATE, MODIFY_DATE)); return (List.of(rule, marker, hold, run, key)); } } diff --git a/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java b/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java index c36d921..75a1f4a 100644 --- a/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java +++ b/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java @@ -68,6 +68,7 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface record.getValues().forEach(effective::setValue); validate(table, record, effective); Instant now = SecuredDataRuntime.current().config().getClock().instant(); + dates(table, record, old == null, now); if(SecuredDataTables.LEGAL_HOLD.equals(table)) { if(old == null) @@ -82,19 +83,41 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface } continue; } - if(old == null) - { - record.setValue("create_date", now); - } - record.setValue("modify_date", now); record.setValue("modified_by", SecuredDataResolver.userId()); } } + else + { + Instant now = SecuredDataRuntime.current().config().getClock().instant(); + for(QRecord record : records) + { + dates(table, record, input instanceof InsertInput, now); + } + } return (records); } + /******************************************************************************* + ** The two dates every table has, set in the write that is already being made: + ** a new row gets both, equal, from the clock every other time of this QBit + ** comes from, and a changed row a new modify date. Core has the last word on + ** a change: the modify date is the record version of its table, and core + ** moves it forward after this has run. A row of a table that is only ever + ** inserted (the access log, a wipe run) so keeps equal dates. + *******************************************************************************/ + private static void dates(String table, QRecord record, boolean inserted, Instant now) + { + if(inserted) + { + record.setValue(SecuredDataTables.createDateField(table), now); + } + record.setValue(SecuredDataTables.modifyDateField(table), now); + } + + + /******************************************************************************* ** Access logs are append-only even for the system. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTables.java b/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTables.java index 86a9965..af8692a 100644 --- a/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTables.java +++ b/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTables.java @@ -32,6 +32,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel; import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules; import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn; @@ -66,6 +67,8 @@ public final class SecuredDataTables public static final List ALL = List.of(CLASSIFICATION, POLICY, REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, RETENTION_RULE, WIPE_MARKER, LEGAL_HOLD, WIPE_RUN, AUDIT_KEY); public static final List MAIN = List.of(CLASSIFICATION, POLICY, ACCESS_LOG, BREAK_GLASS, RETENTION_RULE); public static final List SYSTEM_WRITTEN = List.of(REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, WIPE_MARKER, WIPE_RUN, AUDIT_KEY); + public static final String CREATE_DATE = "createDate"; + public static final String MODIFY_DATE = "modifyDate"; public static final String REVEALS_JOIN = "secured_data_access_log_reveals"; public static final String REVEALS_WIDGET = "securedDataReveals"; @@ -133,6 +136,51 @@ public final class SecuredDataTables + /******************************************************************************* + ** When a row was written. Every table has this field and a modify date; + ** neither is editable, and the columns are create_date and modify_date. + *******************************************************************************/ + static QFieldMetaData createDate(String name) + { + return (field(name, QFieldType.DATE_TIME).withBackendName("create_date").withLabel("create date") + .withIsEditable(false).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE)); + } + + + + /******************************************************************************* + ** When a row last changed: its create date until something changes it. + *******************************************************************************/ + static QFieldMetaData modifyDate(String name) + { + return (field(name, QFieldType.DATE_TIME).withBackendName("modify_date").withLabel("modify date") + .withIsEditable(false).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE)); + } + + + + /******************************************************************************* + ** The name of a table's create date field. Classification, policy and + ** retention rule had the two dates before the other tables did and keep the + ** field names they have always had. + *******************************************************************************/ + static String createDateField(String table) + { + return (List.of(CLASSIFICATION, POLICY, RETENTION_RULE).contains(table) ? "create_date" : CREATE_DATE); + } + + + + /******************************************************************************* + ** The name of a table's modify date field (see createDateField). + *******************************************************************************/ + static String modifyDateField(String table) + { + return (List.of(CLASSIFICATION, POLICY, RETENTION_RULE).contains(table) ? "modify_date" : MODIFY_DATE); + } + + + /******************************************************************************* ** Define Classification. *******************************************************************************/ @@ -145,8 +193,8 @@ public final class SecuredDataTables .withField(field("mask_style", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("secured_data_mask_style")) .withField(field("low_cardinality", QFieldType.BOOLEAN)) .withField(field("notes", QFieldType.TEXT)) - .withField(field("create_date", QFieldType.DATE_TIME)) - .withField(field("modify_date", QFieldType.DATE_TIME)) + .withField(createDate("create_date")) + .withField(modifyDate("modify_date")) .withField(field("modified_by", QFieldType.STRING).withMaxLength(255)); table.withSection(SectionFactory.defaultT1("id", "table_name", "field_name", "classification")); table.withSection(SectionFactory.defaultT2("mask_style", "low_cardinality", "notes")); @@ -179,8 +227,8 @@ public final class SecuredDataTables .withField(field("distinct_value_alert", QFieldType.INTEGER).withDefaultValue(15)) .withField(field("min_entropy_enabled", QFieldType.BOOLEAN).withDefaultValue(true)) .withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true)) - .withField(field("create_date", QFieldType.DATE_TIME)) - .withField(field("modify_date", QFieldType.DATE_TIME)) + .withField(createDate("create_date")) + .withField(modifyDate("modify_date")) .withField(field("modified_by", QFieldType.STRING).withMaxLength(255)); table.withSection(SectionFactory.defaultT1("id", "name", "permission_name", "classification")); table.withSection(SectionFactory.defaultT2("table_name", "tier", "reveal_minutes", "daily_reveal_cap", "reason_options", "allow_free_text_reason", "require_free_text", "client_channel", "exact_match_per_hour", "miss_alert_consecutive", "distinct_value_alert", "min_entropy_enabled", "is_active")); @@ -205,10 +253,12 @@ public final class SecuredDataTables .withField(field("reason_text", QFieldType.TEXT)) .withField(field("granted_at", QFieldType.DATE_TIME).withIsRequired(true)) .withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true)) - .withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true)); + .withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true)) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)); table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "table_name")); table.withSection(SectionFactory.defaultT2("record_id", "field_name", "reason", "reason_text", "access_log_id")); - table.withSection(SectionFactory.defaultT3("granted_at", "expires_at")); + table.withSection(SectionFactory.defaultT3("granted_at", "expires_at", CREATE_DATE, MODIFY_DATE)); return (table); } @@ -230,10 +280,12 @@ public final class SecuredDataTables .withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true)) .withField(field("ended_at", QFieldType.DATE_TIME)) .withField(field("ended_by", QFieldType.STRING).withMaxLength(255)) - .withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status")); + .withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status")) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)); table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "reason")); table.withSection(SectionFactory.defaultT2("reason_text", "classification_ceiling", "table_name", "ended_by", "status")); - table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at")); + table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at", CREATE_DATE, MODIFY_DATE)); return (table); } @@ -261,10 +313,12 @@ public final class SecuredDataTables .withField(field("break_glass_id", QFieldType.INTEGER)) .withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true)) .withField(field("search_value_fingerprints", QFieldType.TEXT)) - .withField(field("rule_id", QFieldType.INTEGER)); + .withField(field("rule_id", QFieldType.INTEGER)) + .withField(createDate(CREATE_DATE)) + .withField(modifyDate(MODIFY_DATE)); table.withSection(SectionFactory.defaultT1("id", "event_type", "user_id", "user_name")); table.withSection(SectionFactory.defaultT2("table_name", "record_id", "field_name", "record_count", "action_name", "reason", "reason_text", "client_channel", "outcome", "anomaly_flag", "break_glass_id", "rule_id", "search_value_fingerprints")); - table.withSection(SectionFactory.defaultT3("created_at")); + table.withSection(SectionFactory.defaultT3("created_at", CREATE_DATE, MODIFY_DATE)); table.withSection(SectionFactory.customT2("reveals", new QIcon("visibility")).withLabel("Reveals").withWidgetName(REVEALS_WIDGET)); return (table); } diff --git a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataDdlTest.java b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataDdlTest.java index 1218a31..ee1ac12 100644 --- a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataDdlTest.java +++ b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataDdlTest.java @@ -120,4 +120,73 @@ class SecuredDataDdlTest .isInstanceOf(java.sql.SQLException.class); } } + + + + /******************************************************************************* + ** Every table the producer declares has the columns create_date and + ** modify_date in a database made from the DDL. The tables that were first + ** created without them get them from the last block of the file, which is what + ** an existing database runs: nullable columns that default to the current + ** time, so an insert that names neither still succeeds, and a block that can + ** be run again and then changes nothing. + *******************************************************************************/ + @Test + void postgresDdlGivesEveryDeclaredTableTheTwoDateColumns() throws Exception + { + String url = "jdbc:h2:mem:secured_data_dates;MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1"; + QInstance instance = SecuredDataTestSupport.setup(); + try(Connection connection = DriverManager.getConnection(url, "sa", "")) + { + SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(false)); + java.util.List added = new java.util.ArrayList<>(); + for(var table : SecuredDataStandardDatesTest.declared(instance)) + { + if(!hasColumn(connection, table.getName(), "create_date")) + { + assertThat(hasColumn(connection, table.getName(), "modify_date")).as(table.getName()).isFalse(); + added.add(table.getName()); + } + } + assertThat(added).hasSize(7); + connection.createStatement().execute("INSERT INTO secured_data_legal_hold (table_name, reason) VALUES ('person', 'before the columns')"); + + SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(true)); + for(var table : SecuredDataStandardDatesTest.declared(instance)) + { + for(String column : java.util.List.of("create_date", "modify_date")) + { + try(java.sql.ResultSet columns = connection.getMetaData().getColumns(null, null, table.getName(), column)) + { + assertThat(columns.next()).as(table.getName() + "." + column).isTrue(); + assertThat(columns.getString("TYPE_NAME")).as(table.getName() + "." + column).isEqualTo("TIMESTAMP WITH TIME ZONE"); + assertThat(columns.getString("IS_NULLABLE")).as(table.getName() + "." + column).isEqualTo("YES"); + if(added.contains(table.getName())) + { + assertThat(columns.getString("COLUMN_DEF")).as(table.getName() + "." + column).isEqualTo("CURRENT_TIMESTAMP"); + } + } + } + } + + connection.createStatement().execute("INSERT INTO secured_data_legal_hold (table_name, reason) VALUES ('person', 'by a release that does not know the columns')"); + var holds = SecuredDataStandardDatesTest.rows(connection, SecuredDataTables.LEGAL_HOLD, false); + assertThat(holds).hasSize(2).allSatisfy(hold -> assertThat(hold.get("create_date")).isNotNull().isEqualTo(hold.get("modify_date"))); + SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(true)); + assertThat(SecuredDataStandardDatesTest.rows(connection, SecuredDataTables.LEGAL_HOLD, false)).isEqualTo(holds); + } + } + + + + /******************************************************************************* + ** Whether a table has a column. + *******************************************************************************/ + private static boolean hasColumn(Connection connection, String table, String column) throws java.sql.SQLException + { + try(java.sql.ResultSet columns = connection.getMetaData().getColumns(null, null, table, column)) + { + return (columns.next()); + } + } } diff --git a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataQBitProducerTest.java b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataQBitProducerTest.java index 47a78ca..96867bd 100644 --- a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataQBitProducerTest.java +++ b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataQBitProducerTest.java @@ -95,7 +95,9 @@ class SecuredDataQBitProducerTest assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(name.equals(SecuredDataTables.AUDIT_KEY) ? AuditLevel.NONE : AuditLevel.FIELD); assertThat(table.getSections()).isNotEmpty(); assertThat(SecuredDataMetadata.classification(table, null)).isEqualTo(DataClassification.CONFIDENTIAL); - table.getFields().values().forEach(field -> assertThat(field.getBackendName()).isEqualTo(field.getName())); + // A column has the name of its field. The two standard dates are always the columns create_date and modify_date. + table.getFields().values().forEach(field -> assertThat(field.getBackendName()) + .isEqualTo("createDate".equals(field.getName()) ? "create_date" : "modifyDate".equals(field.getName()) ? "modify_date" : field.getName())); } assertThat(SecuredDataTables.esbTableNames()).containsExactlyElementsOf(SecuredDataTables.MAIN); assertThat(SecuredDataEnums.sources()).hasSize(6); diff --git a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java new file mode 100644 index 0000000..39061df --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java @@ -0,0 +1,774 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.secureddata; + + +import java.io.IOException; +import java.io.Serializable; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.sql.Timestamp; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService; +import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables; +import com.kingsrook.qbits.secureddata.retention.WipeService; +import com.kingsrook.qbits.secureddata.reveal.RevealService; +import com.kingsrook.qbits.secureddata.support.SystemWrites; +import com.kingsrook.qqq.backend.core.actions.ai.AiExposure; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; +import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens; +import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent; +import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType; +import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome; +import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier; +import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Every table has a create date and a modify date: what the metadata says of + ** them, who may read them, every writer of this QBit setting them, what the + ** access-log writer runs for a row, the records of a changed policy, and rows + ** that were written before the two columns existed. + *******************************************************************************/ +class SecuredDataStandardDatesTest +{ + static final String DDL = "docs/ddl/secured-data-postgres.sql"; + + /** The first line of the block of the DDL that an existing database runs to get the two columns. */ + static final String DDL_BLOCK = "-- Standard dates:"; + + /** The tables that had the two dates before the other tables did: they keep the field names they have always had. */ + private static final Set SNAKE_CASE_NAMES = Set.of("secured_data_classification", "secured_data_policy", "secured_data_retention_rule"); + + private static final Instant LATER = NOW.plusSeconds(90); + + + + /******************************************************************************* + ** Cleanup. + *******************************************************************************/ + @AfterEach + void cleanup() + { + RetentionTestSupport.close(); + } + + + + /******************************************************************************* + ** The tables the producer put in an instance, whichever class declares them, + ** so that a table added later is covered without being named here. + *******************************************************************************/ + static List declared(QInstance instance) + { + String qBit = instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getName(); + List tables = instance.getTables().values().stream().filter(table -> qBit.equals(table.getSourceQBitName())).toList(); + assertThat(tables).extracting(QTableMetaData::getName).containsAll(SecuredDataTables.ALL); + return (tables); + } + + + + /******************************************************************************* + ** Create date field name. + *******************************************************************************/ + private static String createDate(String table) + { + return (SNAKE_CASE_NAMES.contains(table) ? "create_date" : "createDate"); + } + + + + /******************************************************************************* + ** Modify date field name. + *******************************************************************************/ + private static String modifyDate(String table) + { + return (SNAKE_CASE_NAMES.contains(table) ? "modify_date" : "modifyDate"); + } + + + + /******************************************************************************* + ** What the QBit declares, before core enriches anything: core gives a field + ** named createDate or modifyDate its behavior by itself, unless a host turned + ** that off, and gives the snake_case names nothing. + *******************************************************************************/ + @Test + void everyDeclaredTableHasACreateDateAndAModifyDate() throws Exception + { + QInstance instance = base(); + instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class)); + instance.addTable(person("memory")); + new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withClock(CLOCK).withAccessLogFlushIntervalMillis(60000)).produce(instance); + QContext.init(instance, user("u1")); + RUNTIMES.add(SecuredDataRuntime.current()); + + for(QTableMetaData table : declared(instance)) + { + assertDate(table, createDate(table.getName()), "create_date", DynamicDefaultValueBehavior.CREATE_DATE); + assertDate(table, modifyDate(table.getName()), "modify_date", DynamicDefaultValueBehavior.MODIFY_DATE); + } + } + + + + /******************************************************************************* + ** One date field: a date-time nobody edits, kept by core, in a section. + *******************************************************************************/ + private static void assertDate(QTableMetaData table, String name, String column, DynamicDefaultValueBehavior behavior) + { + String where = table.getName() + "." + name; + assertThat(table.getFields()).as(where).containsKey(name); + QFieldMetaData field = table.getField(name); + assertThat(field.getType()).as(where).isEqualTo(QFieldType.DATE_TIME); + assertThat(field.getIsEditable()).as(where).isFalse(); + assertThat(field.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class)).as(where).isEqualTo(behavior); + assertThat(field.getBackendName()).as(where).isEqualTo(column); + assertThat(table.getFields().values()).as(where).filteredOn(other -> other.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class) == behavior).hasSize(1); + assertThat(table.getSections()).as(where).anySatisfy(section -> assertThat(section.getFieldNames()).contains(name)); + } + + + + /******************************************************************************* + ** The two dates have no classification, policy or assistant exposure of their + ** own. Whoever asks, they are given the tier of every other field of their + ** table that has none of its own either, and the assistant is given nothing. + *******************************************************************************/ + @Test + void theDatesAreReadableExactlyWhereTheOtherFieldsOfTheirTableAre() throws Exception + { + QInstance instance = setup(config -> config.withDefaultTier(DataClassification.CONFIDENTIAL, FieldAccessTier.MASKED)); + insert(SecuredDataTables.POLICY, policyRow("hidden", "CONFIDENTIAL", "HIDDEN")); + insert(SecuredDataTables.POLICY, policyRow("full", "CONFIDENTIAL", "FULL")); + insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "FULL")); + insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "HIDDEN").withValue("client_channel", "web")); + + Map sessions = new LinkedHashMap<>(); + sessions.put("the default tier", user("u1")); + sessions.put("a policy that hides", user("u2", "hidden")); + sessions.put("a policy that shows", user("u3", "full")); + sessions.put("a channel cap", SecuredDataTestSupport.channelUser("web", "capped")); + sessions.put("the administrator", user("u4", "secured_data.admin")); + Map expected = Map.of("the default tier", FieldAccessTier.MASKED, "a policy that hides", FieldAccessTier.HIDDEN, + "a policy that shows", FieldAccessTier.FULL, "a channel cap", FieldAccessTier.HIDDEN, "the administrator", FieldAccessTier.FULL); + + for(Map.Entry session : sessions.entrySet()) + { + QContext.setQSession(session.getValue()); + for(QTableMetaData table : declared(instance)) + { + for(String name : List.of(createDate(table.getName()), modifyDate(table.getName()))) + { + String where = session.getKey() + ": " + table.getName() + "." + name; + QFieldMetaData date = table.getField(name); + assertThat(AiFieldMetaData.of(date)).as(where).isNull(); + assertThat(SecuredDataClassifications.effective(table, name)).as(where).isEqualTo(DataClassification.CONFIDENTIAL); + FieldAccessTier tier = SecuredDataRuntime.current().resolver().resolve(table, date).tier(); + assertThat(tier).as(where).isEqualTo(expected.get(session.getKey())); + for(QFieldMetaData other : table.getFields().values()) + { + if(!other.getName().equals(table.getPrimaryKeyField())) + { + assertThat(SecuredDataRuntime.current().resolver().resolve(table, other).tier()).as(where + " and " + other.getName()).isEqualTo(tier); + } + } + } + assertThat(AiTableMetaData.of(table).getExposure()).as(table.getName()).isEqualTo(AiExposure.NONE); + } + } + } + + + + /******************************************************************************* + ** Every writer of this QBit, on the memory backend: the access-log writer, a + ** changed policy and classification with their own log rows, a legal hold, a + ** reveal and a break-glass session from its start to its end and its expiry. + ** A new row has both dates, equal, from the clock of the QBit. A changed row + ** keeps its create date and has a later modify date, which core chooses (the + ** modify date is the record version of its table). + *******************************************************************************/ + @Test + void everyWriterSetsBothDatesAndOnlyAChangeMovesTheModifyDate() throws Exception + { + setup(); + + //////////////////////////////////////////////////////////////////////////// + // the access-log writer: created_at is when the access happened, the two // + // dates are when the writer's one insert wrote the row // + //////////////////////////////////////////////////////////////////////////// + SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hmac")); + CLOCK.now = LATER; + SecuredDataRuntime.current().logger().flush(); + QRecord logged = logs().getLast(); + assertThat(logged.getValueString("event_type")).isEqualTo("EXACT_MATCH_FILTER"); + assertThat(logged.getValueInstant("created_at")).isEqualTo(NOW); + assertDates(logged, SecuredDataTables.ACCESS_LOG, LATER, LATER); + assertThatThrownBy(() -> update(SecuredDataTables.ACCESS_LOG, new QRecord().withValue("id", logged.getValue("id")).withValue("outcome", "DENIED"))).isInstanceOf(Exception.class); + assertDates(row(SecuredDataTables.ACCESS_LOG, logged.getValue("id")), SecuredDataTables.ACCESS_LOG, LATER, LATER); + CLOCK.now = NOW; + + /////////////////////////////////////////////////////////////////////// + // the rows the rules are read from, and the log row of every change // + /////////////////////////////////////////////////////////////////////// + QRecord policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("reason_options", "Support").withValue("allow_free_text_reason", false)); + QRecord classification = classification("notes", "PII"); + QRecord hold = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "person").withValue("reason", "Preservation")); + Map written = new LinkedHashMap<>(); + written.put(SecuredDataTables.POLICY, policy.getValue("id")); + written.put(SecuredDataTables.CLASSIFICATION, classification.getValue("id")); + written.put(SecuredDataTables.LEGAL_HOLD, hold.getValue("id")); + for(Map.Entry row : written.entrySet()) + { + assertDates(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW); + } + assertDates(log("POLICY_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW); + assertDates(log("CLASSIFICATION_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW); + + CLOCK.now = LATER; + update(SecuredDataTables.POLICY, new QRecord().withValue("id", policy.getValue("id")).withValue("name", "renamed")); + update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", classification.getValue("id")).withValue("notes", "reviewed")); + update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("status", "RELEASED")); + for(Map.Entry row : written.entrySet()) + { + assertChanged(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW); + } + assertDates(log("POLICY_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER); + assertDates(log("CLASSIFICATION_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER); + CLOCK.now = NOW; + + //////////////////////////////////////////// + // a reveal: its grant and its log row // + //////////////////////////////////////////// + Serializable person = insert("person", new QRecord().withValue("email", "someone@example.invalid")).getValue("id"); + new RevealService().reveal("person", person, "email", "Support", null); + assertDates(SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()).getFirst(), SecuredDataTables.REVEAL_GRANT, NOW, NOW); + assertDates(log("REVEAL", null), SecuredDataTables.ACCESS_LOG, NOW, NOW); + + ////////////////////////////////////////////////////////////////////// + // a break-glass session: started, then ended; started, then expired // + ////////////////////////////////////////////////////////////////////// + BreakGlassService breaks = new BreakGlassService(); + QContext.setQSession(user("u1", "cs_admin", BreakGlassService.START + ".hasAccess")); + Serializable ended = breaks.start("Investigating urgent incident", 60, null, null).getValue("id"); + assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW); + CLOCK.now = LATER; + breaks.end(ended); + Instant endedAt = assertChanged(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW); + assertDates(log("BREAK_GLASS_END", null), SecuredDataTables.ACCESS_LOG, LATER, LATER); + + Serializable expired = breaks.start("Investigating another incident", 1, null, null).getValue("id"); + assertDates(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER); + CLOCK.now = LATER.plusSeconds(60); + assertThat(breaks.expire()).isEqualTo(1); + assertChanged(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER); + assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, endedAt); + + ///////////////////////////////////////////////////////////////////////// + // the access log is only ever inserted, so no row's modify date moved // + ///////////////////////////////////////////////////////////////////////// + assertThat(logs()).hasSizeGreaterThan(8).allSatisfy(row -> + { + assertThat(row.getValueInstant("createDate")).isNotNull(); + assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("createDate")); + }); + } + + + + /******************************************************************************* + ** The writers of a wipe, in the shipped schema: the audit key, the run, the + ** marker (written once, then changed by a later wipe of the same field), the + ** log row, and the rule the run is recorded on. A time is compared with what + ** the same database gives back for another time written from the same clock, + ** not with the clock: the test database shifts a time by the zone of the + ** machine it runs on. + *******************************************************************************/ + @Test + void aWipeSetsBothDatesInTheShippedSchema() throws Exception + { + RetentionTestSupport.setup(); + QRecord rule = RetentionTestSupport.rule("OVERWRITE_TEXT", "secret"); + QRecord reading = RetentionTestSupport.reading(120); + assertThat(SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", reading.getValue("id"), true)).isPresent(); + QRecord key = RetentionTestSupport.rows(SecuredDataTables.AUDIT_KEY).getFirst(); + assertDates(key, SecuredDataTables.AUDIT_KEY, key.getValueInstant("created_at"), key.getValueInstant("created_at")); + Instant ruleWritten = row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")).getValueInstant("create_date"); + assertDates(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten); + + CLOCK.now = LATER; + QRecord run = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id"))); + assertThat(run.getValueInteger("wiped_count")).isEqualTo(1); + Instant wiped = ruleWritten.plusSeconds(90); + QRecord firstRun = row(SecuredDataTables.WIPE_RUN, run.getValue("id")); + assertThat(firstRun.getValueInstant("finished_at")).isEqualTo(wiped); + assertDates(firstRun, SecuredDataTables.WIPE_RUN, wiped, wiped); + QRecord marker = RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER).getFirst(); + assertThat(marker.getValueInstant("wiped_at")).isEqualTo(wiped); + assertDates(marker, SecuredDataTables.WIPE_MARKER, wiped, wiped); + assertThat(log("WIPE", "OVERWRITE_TEXT").getValueInstant("created_at")).isEqualTo(wiped); + assertDates(log("WIPE", "OVERWRITE_TEXT"), SecuredDataTables.ACCESS_LOG, wiped, wiped); + Instant ruleRun = assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten); + + /////////////////////////////////////////////////////////////////////////// + // the field is given a value again and wiped again: the marker of that // + // record and field is changed, and no second marker is written // + /////////////////////////////////////////////////////////////////////////// + CLOCK.now = LATER.plusSeconds(90); + Instant again = wiped.plusSeconds(90); + update("reading", new QRecord().withValue("id", reading.getValue("id")).withValue("secret", "another-test-secret")); + QRecord second = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id"))); + assertThat(second.getValueInteger("wiped_count")).isEqualTo(1); + assertThat(RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER)).hasSize(1); + QRecord changed = row(SecuredDataTables.WIPE_MARKER, marker.getValue("id")); + assertThat(changed.getValue("run_id")).isEqualTo(second.getValue("id")); + assertThat(changed.getValueInstant("wiped_at")).isEqualTo(again); + assertChanged(changed, SecuredDataTables.WIPE_MARKER, wiped, wiped); + assertDates(row(SecuredDataTables.WIPE_RUN, run.getValue("id")), SecuredDataTables.WIPE_RUN, wiped, wiped); + assertDates(row(SecuredDataTables.WIPE_RUN, second.getValue("id")), SecuredDataTables.WIPE_RUN, again, again); + assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleRun); + } + + + + /******************************************************************************* + ** The access-log writer runs concurrently with reveals, so the dates must cost + ** it nothing: for each row it still runs one insert, which now names the two + ** columns, and no other statement that writes the log. + *******************************************************************************/ + @Test + void theAccessLogWriterStillRunsOneInsertForARow() throws Exception + { + RetentionTestSupport.setup(); + try(Connection connection = DriverManager.getConnection(url(), "sa", "")) + { + connection.createStatement().execute("SET QUERY_STATISTICS TRUE"); + for(int i = 0; i < 3; i++) + { + SecuredDataRuntime.current().logger().onAccess(new FieldAccessEvent().withType(FieldAccessEventType.EXACT_MATCH_FILTER).withOutcome(FieldAccessOutcome.MATCH) + .withTableName("reading").withFieldNames(List.of("secret")).withActionName("query").withRecordCount(1).withSearchValueFingerprints(List.of("hmac" + i))); + } + SecuredDataRuntime.current().logger().flush(); + assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).hasSize(3).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull().isEqualTo(row.get("modify_date"))); + + Map writes = new LinkedHashMap<>(); + try(ResultSet statistics = connection.createStatement().executeQuery("SELECT SQL_STATEMENT, EXECUTION_COUNT FROM INFORMATION_SCHEMA.QUERY_STATISTICS")) + { + while(statistics.next()) + { + String sql = statistics.getString(1).strip(); + if(sql.contains(SecuredDataTables.ACCESS_LOG) && !sql.toUpperCase().startsWith("SELECT")) + { + writes.merge(sql, statistics.getInt(2), Integer::sum); + } + } + } + assertThat(writes).hasSize(1); + assertThat(writes.keySet().iterator().next()).startsWithIgnoringCase("INSERT INTO").contains("create_date", "modify_date"); + assertThat(writes.values()).containsExactly(3); + } + } + + + + /******************************************************************************* + ** A changed policy or classification writes what it wrote before the tables + ** had the standard dates: its core audit, field by field, and its + ** POLICY_CHANGE or CLASSIFICATION_CHANGE row in the access log. + *******************************************************************************/ + @Test + void aChangedPolicyOrClassificationWritesItsAuditAndItsLogRowAsBefore() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + QRecord policy = new InsertAction().execute(new InsertInput(SecuredDataTables.POLICY).withRecord(policyRow("viewer", "PHI", "MASKED"))).getRecords().getFirst(); + QRecord classification = new InsertAction().execute(new InsertInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("table_name", "reading") + .withValue("field_name", "notes").withValue("classification", "PHI"))).getRecords().getFirst(); + assertThat(policy.getErrors()).isNullOrEmpty(); + assertThat(classification.getErrors()).isNullOrEmpty(); + CLOCK.now = LATER; + assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.POLICY).withRecord(new QRecord().withValue("id", policy.getValue("id")).withValue("tier", "HIDDEN"))) + .getRecords().getFirst().getErrors()).isNullOrEmpty(); + assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("id", classification.getValue("id")) + .withValue("classification", "PII"))).getRecords().getFirst().getErrors()).isNullOrEmpty(); + new DeleteAction().execute(new DeleteInput(SecuredDataTables.POLICY).withPrimaryKeys(List.of(policy.getValue("id")))); + new DeleteAction().execute(new DeleteInput(SecuredDataTables.CLASSIFICATION).withPrimaryKeys(List.of(classification.getValue("id")))); + + /////////////////////////////////////////////////////////////////////////// + // the log rows: one per change, with every value they have always had // + /////////////////////////////////////////////////////////////////////////// + List changes = new ArrayList<>(); + List logged = RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG); + Instant inserted = logged.getFirst().getValueInstant("created_at"); + for(QRecord row : logged) + { + changes.add(String.join(" ", row.getValueString("event_type"), row.getValueString("table_name"), row.getValueString("record_id"), row.getValueString("action_name"), + row.getValueString("outcome"), row.getValueString("user_id"), row.getValueString("user_name"), + "after " + Duration.between(inserted, row.getValueInstant("created_at")).toSeconds() + " s", String.valueOf(row.getValueBoolean("anomaly_flag")))); + for(String empty : List.of("field_name", "record_count", "reason", "reason_text", "client_channel", "break_glass_id", "search_value_fingerprints", "rule_id")) + { + assertThat(row.getValue(empty)).as(empty).isNull(); + } + } + String policyId = policy.getValueString("id"); + String classificationId = classification.getValueString("id"); + assertThat(changes).containsExactly( + "POLICY_CHANGE secured_data_policy " + policyId + " INSERT ALLOWED admin admin after 0 s false", + "CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " INSERT ALLOWED admin admin after 0 s false", + "POLICY_CHANGE secured_data_policy " + policyId + " UPDATE ALLOWED admin admin after 90 s false", + "CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " UPDATE ALLOWED admin admin after 90 s false", + "POLICY_CHANGE secured_data_policy " + policyId + " DELETE ALLOWED admin admin after 90 s false", + "CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " DELETE ALLOWED admin admin after 90 s false"); + + ////////////////////////////////////////////////////////////////////////////// + // the core audits of the two rows: the same audits with the same details, // + // the two dates among them under the names they have always been audited by // + ////////////////////////////////////////////////////////////////////////////// + assertThat(audits(instance, SecuredDataTables.POLICY, policy.getValue("id"))).containsExactly( + "Record was Inserted: allow_free_text_reason [Set allow free text reason to Yes], classification [Set classification to PHI], create_date," + + " distinct_value_alert [Set distinct value alert to 15], exact_match_per_hour [Set exact match per hour to 20], id [Set Id to " + policyId + "]," + + " is_active [Set is active to Yes], min_entropy_enabled [Set min entropy enabled to Yes], miss_alert_consecutive [Set miss alert consecutive to 5]," + + " modified_by [Set modified by to admin], modify_date, name [Set name to viewer PHI], permission_name [Set permission name to viewer]," + + " require_free_text [Set require free text to No], reveal_minutes [Set reveal minutes to 5], tier [Set tier to MASKED]", + "Record was Edited: modify_date, tier [Changed tier from \"MASKED\" to \"HIDDEN\"]", + "Record was Deleted: "); + assertThat(audits(instance, SecuredDataTables.CLASSIFICATION, classification.getValue("id"))).containsExactly( + "Record was Inserted: classification [Set classification to PHI], create_date, field_name [Set field name to notes], id [Set Id to " + classificationId + "]," + + " modified_by [Set modified by to admin], modify_date, table_name [Set table name to Reading]", + "Record was Edited: classification [Changed classification from \"PHI\" to \"PII\"], modify_date", + "Record was Deleted: "); + } + + + + /******************************************************************************* + ** The core audits of one record, oldest first: the message of each and, in + ** the order of the field names, the fields it has a detail for and what the + ** detail says. The two dates are named only: how a time reads depends on the + ** zone of the machine. + *******************************************************************************/ + private static List audits(QInstance instance, String table, Serializable id) + { + Serializable auditTable = SystemWrites.query("auditTable", new QQueryFilter().withCriteria(new QFilterCriteria("name", QCriteriaOperator.EQUALS, table))).getFirst().getValue("id"); + List audits = new ArrayList<>(); + for(QRecord audit : SystemWrites.query("audit", new QQueryFilter().withCriteria(new QFilterCriteria("auditTableId", QCriteriaOperator.EQUALS, auditTable)) + .withCriteria(new QFilterCriteria("recordId", QCriteriaOperator.EQUALS, id)).withOrderBy(new QFilterOrderBy("id", true)))) + { + List fields = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("auditId", QCriteriaOperator.EQUALS, audit.getValue("id")))) + .stream().map(detail -> detail.getValueString("fieldName") + (detail.getValueString("fieldName").endsWith("_date") ? "" + : " [" + detail.getValueString("message") + "]")).sorted().toList(); + audits.add(audit.getValueString("message") + ": " + String.join(", ", fields)); + } + assertThat(instance.getTable(table)).isNotNull(); + return (audits); + } + + + + /******************************************************************************* + ** Rows written before the two columns existed. A reading is audited (its + ** audit token is made with a data key of its own, and that key is stored + ** wrapped, bound to its record and field), the log has rows, a rule and a + ** hold exist. The two columns are then taken away, which leaves the database + ** as the release before this one left it, and a key and a log row are + ** written the way that release wrote them. The block an existing database + ** runs must change nothing but the two columns: the same rows, the same + ** wrapped keys, which still unwrap to the same data keys, and the same + ** token, which still verifies the value and no other. + *******************************************************************************/ + @Test + void rowsWrittenBeforeTheDatesExistedVerifyUnchangedAfterTheyAreAdded() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + QTableMetaData reading = instance.getTable("reading"); + SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL"); + SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL"); + QRecord rule = RetentionTestSupport.rule("NULL_WITH_MARKER", "systolic"); + insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", "404").withValue("reason", "Preservation")); + Serializable id = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 120).withValue("secret", "s") + .withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id"); + Serializable wiped = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 130).withValue("secret", "s") + .withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id"); + new WipeService().batch(rule.getValue("id"), List.of(wiped)); + + SecuredDataKeySource keys = SecuredDataRuntime.current().keys(); + byte[] dataKey = keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key(); + List tokens = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("fieldName", QCriteriaOperator.EQUALS, "systolic"))).stream() + .map(detail -> detail.getValueString("newValue")).filter(value -> value != null && value.startsWith("shred:v1:")).toList(); + String token = tokens.stream().filter(candidate -> ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, candidate)).findFirst().orElseThrow(); + assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse(); + + List tables = declared(instance).stream().map(QTableMetaData::getName).filter(name -> !SNAKE_CASE_NAMES.contains(name)).toList(); + byte[] earlierKey = new byte[32]; + Arrays.fill(earlierKey, (byte) 7); + Instant earlier = NOW.minusSeconds(86400); + try(Connection connection = DriverManager.getConnection(url(), "sa", "")) + { + for(String table : tables) + { + assertThat(rows(connection, table, false)).as(table).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull()); + connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN create_date"); + connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN modify_date"); + } + try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_audit_key (table_name, record_id, field_name, wrapped_key, created_at) VALUES ('reading', '9001', 'systolic', ?, ?)")) + { + insert.setString(1, Base64.getEncoder().encodeToString(SecuredDataKeySource.encrypt(earlierKey, "reading\u001fsystolic\u001f9001"))); + insert.setTimestamp(2, Timestamp.from(earlier)); + insert.executeUpdate(); + } + try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_access_log (event_type, user_id, table_name, field_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u1', 'reading', 'systolic', 'ALLOWED', false, ?)")) + { + insert.setTimestamp(1, Timestamp.from(earlier)); + insert.executeUpdate(); + } + Map>> before = new LinkedHashMap<>(); + for(String table : tables) + { + before.put(table, rows(connection, table, false)); + } + assertThat(before.get(SecuredDataTables.ACCESS_LOG)).hasSizeGreaterThan(3); + assertThat(before.get(SecuredDataTables.AUDIT_KEY)).hasSize(2); + for(String table : List.of(SecuredDataTables.WIPE_RUN, SecuredDataTables.WIPE_MARKER, SecuredDataTables.LEGAL_HOLD)) + { + assertThat(before.get(table)).as(table).hasSize(1); + } + + run(connection, ddl(true)); + + for(String table : tables) + { + assertThat(rows(connection, table, true)).as(table).isEqualTo(before.get(table)); + assertThat(rows(connection, table, false)).as(table).allSatisfy(row -> + { + assertThat(row.get("create_date")).isNotNull(); + assertThat(row.get("modify_date")).isNotNull(); + }); + } + assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).allSatisfy(row -> + { + assertThat(row.get("create_date")).isEqualTo(row.get("created_at")); + assertThat(row.get("modify_date")).isEqualTo(row.get("created_at")); + }); + assertThat(rows(connection, SecuredDataTables.AUDIT_KEY, false)).allSatisfy(row -> + { + assertThat(row.get("create_date")).isEqualTo(row.get("created_at")); + assertThat(row.get("modify_date")).isEqualTo(row.get("created_at")); + }); + + ///////////////////////////////////////////////////////////////////////////// + // the release before this one goes on writing while both run: its insert // + // names neither column, and the row has both, equal, all the same // + ///////////////////////////////////////////////////////////////////////////// + connection.createStatement().execute("INSERT INTO secured_data_access_log (event_type, user_id, table_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u2', 'reading', 'ALLOWED', false, CURRENT_TIMESTAMP)"); + Map concurrent = rows(connection, SecuredDataTables.ACCESS_LOG, false).getLast(); + assertThat(concurrent.get("user_id")).isEqualTo("u2"); + assertThat(concurrent.get("create_date")).isNotNull().isEqualTo(concurrent.get("modify_date")); + } + + /////////////////////////////////////////////////////////////////////////// + // nothing kept in memory: every key is unwrapped again from its row // + /////////////////////////////////////////////////////////////////////////// + keys.close(); + assertThat(keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key()).isEqualTo(dataKey); + assertThat(keys.keyForRecordField("reading", "systolic", 9001, false).orElseThrow().key()).isEqualTo(earlierKey); + assertThat(keys.keyForRecordField("reading", "systolic", wiped, false)).isEmpty(); + assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isTrue(); + assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse(); + + //////////////////////////////////////////////////////////////////////////// + // and the rows that were there are read and written on as they were: the // + // earlier log row is counted, and the reading is wiped under its rule // + //////////////////////////////////////////////////////////////////////////// + assertThat(RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG)).anySatisfy(row -> + { + assertThat(row.getValueString("user_id")).isEqualTo("u1"); + assertThat(row.getValueInstant("createDate")).isNotNull().isEqualTo(row.getValueInstant("created_at")); + assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("created_at")); + }); + CLOCK.now = LATER; + assertThat(new WipeService().batch(rule.getValue("id"), List.of(id)).getValueInteger("wiped_count")).isEqualTo(1); + assertThat(keys.keyForRecordField("reading", "systolic", id, false)).isEmpty(); + assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isFalse(); + } + + + + /******************************************************************************* + ** The database RetentionTestSupport made from the shipped DDL. + *******************************************************************************/ + private static String url() + { + return ("jdbc:h2:mem:" + RetentionTestSupport.BACKEND + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1"); + } + + + + /******************************************************************************* + ** The statements of the shipped DDL: those of the block an existing database + ** runs to get the two columns, or those before it (the schema without them). + *******************************************************************************/ + static List ddl(boolean datesBlock) throws IOException + { + String ddl = Files.readString(Path.of(DDL)); + int block = ddl.indexOf(DDL_BLOCK); + assertThat(block).as("the block of " + DDL + " that begins " + DDL_BLOCK).isPositive().isEqualTo(ddl.lastIndexOf(DDL_BLOCK)); + return (Arrays.stream((datesBlock ? ddl.substring(block) : ddl.substring(0, block)).split(";")).filter(sql -> !sql.isBlank()).toList()); + } + + + + /******************************************************************************* + ** Run statements. + *******************************************************************************/ + static void run(Connection connection, List statements) throws SQLException + { + for(String sql : statements) + { + connection.createStatement().execute(sql); + } + } + + + + /******************************************************************************* + ** Every row of a table as the database holds it, column by column, in id + ** order, with or without the two date columns. + *******************************************************************************/ + static List> rows(Connection connection, String table, boolean withoutDates) throws SQLException + { + List> rows = new ArrayList<>(); + try(ResultSet results = connection.createStatement().executeQuery("SELECT * FROM " + table + " ORDER BY id")) + { + ResultSetMetaData columns = results.getMetaData(); + while(results.next()) + { + Map row = new LinkedHashMap<>(); + for(int i = 1; i <= columns.getColumnCount(); i++) + { + String column = columns.getColumnName(i); + if(!withoutDates || !Set.of("create_date", "modify_date").contains(column)) + { + row.put(column, results.getString(i)); + } + } + rows.add(row); + } + } + return (rows); + } + + + + /******************************************************************************* + ** One stored row. + *******************************************************************************/ + private static QRecord row(String table, Serializable id) + { + return (SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id))).getFirst()); + } + + + + /******************************************************************************* + ** The newest log row of an event, and of an action where one is given. + *******************************************************************************/ + private static QRecord log(String event, String action) + { + return (logs().stream().filter(row -> event.equals(row.getValueString("event_type")) && (action == null || action.equals(row.getValueString("action_name")))) + .reduce((first, second) -> second).orElseThrow()); + } + + + + /******************************************************************************* + ** Both dates of a stored row. + *******************************************************************************/ + private static void assertDates(QRecord row, String table, Instant created, Instant modified) + { + assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created); + assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isEqualTo(modified); + } + + + + /******************************************************************************* + ** Both dates of a stored row that was changed: the create date it had, and a + ** modify date later than the one it had, which is returned. + *******************************************************************************/ + private static Instant assertChanged(QRecord row, String table, Instant created, Instant modifiedBefore) + { + assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created); + assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isAfter(modifiedBefore); + return (row.getValueInstant(modifyDate(table))); + } +} -- 2.54.0 From 0a8c0fba6f8a2581970a0827c6e746bd6f4fb80c Mon Sep 17 00:00:00 2001 From: James Maes Date: Fri, 20 Nov 2026 10:40:48 -0600 Subject: [PATCH 2/2] fix: an edit over HTTP is an edit of its row, not a new row The table customizer finds the stored row of an update by its id, and compared the two ids as they came: the id of a request over HTTP is text (it is the id in the path), the stored id is a number. They were never equal, so an edit made through the API or the UI had no stored row and was validated and stamped as an insert. The uniqueness check of a classification compared ids the same way. Both now compare the ids as text. This is older than the standard dates, but they build on it: the customizer gives a new row its create date, and a client that knows record versions edits a table with a modify date by sending only the changed values and the version it read. What an edit over HTTP did before: - A classification, policy or retention rule got a new create_date. - Releasing a legal hold overwrote when and by whom it was placed (and, since the hold has one, its create date). - Editing the reason of a legal hold without sending its status failed with a server error (the validation read the status of a row it did not have). - Editing a classification while naming its table and field was refused as a second classification of that field. - Editing a retention rule without sending every field was refused. Tests (SecuredDataHttpEditTest, through the real middleware on the shipped schema, a day after the rows were written): the reason of a hold is edited with the changed value and the version, and the stale version is then refused; a hold is released and keeps when and by whom it was placed and its create date; a classification is edited naming its table and field; a policy and a retention rule are renamed and keep their create dates; a row with neither date is read and edited. All five fail without the fix. They run in UTC: where the default zone of the JVM is not UTC, core refuses every update that sends a version, which is core's to fix and is kept out of these tests. Notes for an upgrade (shipped DDL block and README): set a lock_timeout before the ALTER statements and run the block again when one is cancelled; on a large access log run its backfill UPDATE in ranges of id, or leave it out, since rows without the two dates are read and written like any others. Changelog: the fix, that an update naming the create date still sets it, and that a wipe moves the record version of its retention rule. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 9 +- README.md | 22 +- docs/ddl/secured-data-postgres.sql | 16 +- .../metadata/SecuredDataTableCustomizers.java | 8 +- .../secureddata/SecuredDataHttpEditTest.java | 352 ++++++++++++++++++ .../SecuredDataStandardDatesTest.java | 2 +- 6 files changed, 399 insertions(+), 10 deletions(-) create mode 100644 src/test/java/com/kingsrook/qbits/secureddata/SecuredDataHttpEditTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b51317..1470cc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,10 +6,15 @@ All notable changes are documented here, following [Keep a Changelog](https://ke ### Added: a create date and a modify date on every table - Every table has a create date and a modify date: `DATE_TIME`, not editable, with QQQ's `CREATE_DATE` and `MODIFY_DATE` behaviors, in the columns `create_date` and `modify_date`. Seven tables did not have them: reveal grant, break-glass session, access log, wipe run, wipe marker, legal hold and audit key. Their fields are named `createDate` and `modifyDate`. Classification, policy and retention rule already had `create_date` and `modify_date`; they keep those field names, and the two fields are now not editable and have the behaviors. -- A new row gets both dates, equal, in the insert that writes it. A changed row keeps its create date and gets a later modify date. The access log and wipe runs are only ever inserted, so their dates stay equal. The access-log writer still runs one insert per row, with no other statement, lock or transaction. +- A new row gets both dates, equal, in the insert that writes it. A changed row keeps its create date and gets a later modify date (for an edit that arrives over HTTP this holds from this release on: see the fix below). An update that names the create date itself still sets it: not editable is what the metadata tells a client. The access log and wipe runs are only ever inserted, so their dates stay equal. The access-log writer still runs one insert per row, with no other statement, lock or transaction. - **Upgrade: run the DDL first.** Before deploying this release on an existing database, run the last block of `docs/ddl/secured-data-postgres.sql` (from the line `-- Standard dates:` to the end). It adds the two nullable columns to the seven tables, defaults them to the current time and gives existing rows the row's own time. It rewrites no table and can be run while the release before this one is running, which goes on working with the columns present. This release does not run without them. See "Upgrading an existing database" in the README. - The dates have no classification, policy or assistant exposure of their own, and nothing that is signed or verified covers them. Rows written before the columns existed are unchanged apart from the two columns: wrapped data keys unwrap as before and audit tokens verify as before. -- QQQ uses a field with the `MODIFY_DATE` behavior as the record version of its table. That now holds for every table of the QBit, classification, policy and retention rule included: on an update core sets the modify date itself, and an update that sends the version it read is refused if the row has changed since. Updates that send no version, the QBit's own among them, behave as before. +- QQQ uses a field with the `MODIFY_DATE` behavior as the record version of its table. That now holds for every table of the QBit, classification, policy and retention rule included: on an update core sets the modify date itself, and an update that sends the version it read is refused (409) if the row has changed since. Updates that send no version, the QBit's own among them, behave as before. The QBit's own updates move the version all the same: a wipe, the scheduled one included, records its run on its retention rule, so an API client that reads a rule and edits it with that version after a wipe has run gets 409 and has to read the rule again. +- On a live database, set a `lock_timeout` before running the upgrade block and run it again when a statement is cancelled (an `ALTER` waits behind an open transaction on its table, and other readers and writers wait behind it). On a large access log, run its backfill `UPDATE` in ranges of `id`, or leave it out: rows without the two dates are read and written like any others. The README has both. + +### Fixed: an edit over HTTP was treated as a new row +- The table customizer matched an update with its stored row by comparing ids of different types: the id of a request over HTTP is text, the stored id a number. An edit made through the API or the UI was therefore validated and stamped as an insert. Ids are now compared as text. +- What that did, for an edit over HTTP: a classification, policy or retention rule got a new `create_date` on every edit. Releasing a legal hold overwrote when and by whom it was placed. Editing a classification while naming its table and field was refused as a second classification of the same field. Editing a retention rule without sending every field was refused. Editing a legal hold without sending its status failed with a server error; a client that sends only the changed fields of a table with a modify date does exactly that. ### Fixed: a reveal and the access-log writer waiting for each other - A reveal, or the start, end or expiry of a break-glass session, no longer deadlocks with a thread that asks for a classification while it holds another lock of this QBit. The access-log writer is such a thread: it holds the anomaly counters while it inserts a row, and that insert asks for classifications (it counts earlier rows and writes an audit, both queries with criteria). A reveal holds the runtime's monitor and then inserts its own access-log row. When both happened at once, neither finished, and neither did any later request, because every request asks for a classification. A person who opens a record with a masked field and reveals it straight away does both at once. diff --git a/README.md b/README.md index 1ddc692..6f37a7c 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,23 @@ Before deploying a release that has the two dates on every table, run the last b - It adds the two columns, nullable. That changes the catalog only; the table is not rewritten. - It gives each column a default of the current time, so a release that does not know the columns goes on inserting while both releases run. -- It gives the rows that were already there the row's own time (below). The `UPDATE` touches only rows whose `create_date` is null, so the block can be run again and then changes nothing. It takes row locks only, and on a large access log it can be run in ranges of `id`. +- It gives the rows that were already there the row's own time (below). The `UPDATE` touches only rows whose `create_date` is null and takes row locks only. + +Every statement of the block can be run again and then changes nothing. Two things to plan for on a live database: + +- **Locks.** Each `ALTER` takes a brief exclusive lock on its table. It waits behind any open transaction that has touched the table (one that has inserted a log row, for one), and every other reader and writer of the table then waits behind the `ALTER`. Set a lock timeout for the session first, and run the block again when a statement is cancelled: + + ```sql + SET lock_timeout = '5s'; + ``` + +- **A large access log.** The `UPDATE` of `secured_data_access_log` writes every existing row in one transaction. One measurement on PostgreSQL 16 with 2,000,000 rows: 24 s, and the table grew from 146 MB to 324 MB. On a large log run it in ranges of `id` instead, one range per transaction, up to the highest `id`: + + ```sql + UPDATE secured_data_access_log SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL AND id BETWEEN 1 AND 100000; + ``` + + Or leave it out. A row without the two dates is read and written like any other, and the columns are filled for every row inserted from then on. | Table | `create_date` of an existing row | `modify_date` of an existing row | |---|---|---| @@ -66,11 +82,11 @@ Nothing else changes in a row: wrapped data keys, search fingerprints and every Every table has audit rules, explicit record sections and CONFIDENTIAL code classification. The Secured Data app navigates to Classification, Policy, Access log and Break-glass session. Reveal grants are a read-only child list joined by `access_log_id`; no process or home widget appears in navigation. The classification target `(table_name, field_name)` is unique, including the null field used for a table-wide override. -Every table has a create date and a modify date: date-time fields that nobody edits, in the columns `create_date` and `modify_date`. The fields are named `createDate` and `modifyDate`. Classification, Policy and Retention rules had the two dates before the other tables did and keep the field names `create_date` and `modify_date`. A new row gets both, equal. A changed row keeps its create date and gets a later modify date. Every write of the QBit is a QQQ insert or update action, so the dates are in the one statement that writes the row, the access-log writer's insert included. The access log and wipe runs are only ever inserted, so their two dates stay equal. +Every table has a create date and a modify date: date-time fields that nobody edits, in the columns `create_date` and `modify_date`. The fields are named `createDate` and `modifyDate`. Classification, Policy and Retention rules had the two dates before the other tables did and keep the field names `create_date` and `modify_date`. A new row gets both, equal. A changed row keeps its create date and gets a later modify date. (Not editable is what the metadata tells a client, as everywhere in QQQ: an update that names the create date itself still sets it.) Every write of the QBit is a QQQ insert or update action, so the dates are in the one statement that writes the row, the access-log writer's insert included. The access log and wipe runs are only ever inserted, so their two dates stay equal. The two dates have no classification of their own. They are CONFIDENTIAL with the rest of their table, are given the tier of its other fields, and are closed to the assistant with it. Nothing that is signed or verified covers them: a wrapped data key is bound to its table, field and record id, and the search fingerprints of a log row are made from the search values. -The modify date has QQQ's modify-date behavior, and QQQ uses such a field as the record version of its table. On an update core sets it to the current time (or one millisecond after the stored value, if that is later). When a caller sends the version it read, core refuses the update if the row has changed since. The QBit's own updates send no version. +The modify date has QQQ's modify-date behavior, and QQQ uses such a field as the record version of its table. On an update core sets it to the current time (or one millisecond after the stored value, if that is later). When a caller sends the version it read, core refuses the update (409) if the row has changed since. The QBit's own updates send no version, but they move it: a wipe records its run on its retention rule, so a client that reads a rule and saves it with that version after a wipe has run is refused and has to read the rule again. `secured_data.admin` gates configuration editing and activity reads through a standard QQQ custom permission checker. It gives no implicit access to classified fields on application tables. `secured_data.hasAccess` is the catalog base permission; the custom checker requires admin. Activity writes are refused for ordinary users even when they hold admin. Customizers also protect system-only writes, and update/delete capabilities are disabled on the access log, including system calls. diff --git a/docs/ddl/secured-data-postgres.sql b/docs/ddl/secured-data-postgres.sql index 8c51664..a5a9094 100644 --- a/docs/ddl/secured-data-postgres.sql +++ b/docs/ddl/secured-data-postgres.sql @@ -144,8 +144,20 @@ CREATE INDEX secured_data_wipe_rule_lookup ON secured_data_wipe_marker (rule_id) -- that writes the two columns is deployed. The columns are nullable and default to the current time, so -- a release that does not know them goes on inserting while both run. Adding a nullable column and -- setting a default change the catalog only (no table rewrite). Each UPDATE gives the rows that were --- already there the row's own time, once: it touches only rows whose create_date is still null, takes --- row locks only, and on a large access log can be run in ranges of id. +-- already there the row's own time, once: it touches only rows whose create_date is still null and +-- takes row locks only. Every statement can be run again and then changes nothing. +-- +-- Locks. Each ALTER takes a brief exclusive lock on its table. It waits behind any open transaction +-- that has touched the table (one that has inserted a log row, for one), and every other reader and +-- writer of the table then waits behind the ALTER. On a live database set a lock timeout for the +-- session first (SET lock_timeout = '5s') and run the block again when a statement is cancelled. +-- +-- A large access log. The UPDATE of secured_data_access_log writes every existing row in one +-- transaction. One measurement on PostgreSQL 16 with 2,000,000 rows: 24 s, and the table grew from +-- 146 MB to 324 MB. On a large log run it in ranges of id instead, one range per transaction (add +-- AND id BETWEEN 1 AND 100000 to its WHERE, then the next range, up to the highest id), or leave it +-- out: a row without the two dates is read and written like any other, and the columns are filled +-- for every row inserted from then on. ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE; ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE; ALTER TABLE secured_data_reveal_grant ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP; diff --git a/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java b/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java index 75a1f4a..39fe6f6 100644 --- a/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java +++ b/src/main/java/com/kingsrook/qbits/secureddata/metadata/SecuredDataTableCustomizers.java @@ -63,7 +63,11 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface { for(QRecord record : records) { - QRecord old = oldRecords.orElse(List.of()).stream().filter(row -> Objects.equals(row.getValue("id"), record.getValue("id"))).findFirst().orElse(null); + /////////////////////////////////////////////////////////////////////////// + // ids are compared as text: a request over HTTP names its row by the id // + // in its path, which is text, while the stored id is a number // + /////////////////////////////////////////////////////////////////////////// + QRecord old = oldRecords.orElse(List.of()).stream().filter(row -> Objects.equals(row.getValueString("id"), record.getValueString("id"))).findFirst().orElse(null); QRecord effective = old == null ? new QRecord() : new QRecord(old); record.getValues().forEach(effective::setValue); validate(table, record, effective); @@ -177,7 +181,7 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface { for(QRecord row : SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, target)))) { - if(Objects.equals(field, row.getValueString("field_name")) && !Objects.equals(row.getValue("id"), effective.getValue("id"))) + if(Objects.equals(field, row.getValueString("field_name")) && !Objects.equals(row.getValueString("id"), effective.getValueString("id"))) { record.addError(new BadInputStatusMessage("A classification for this table and field already exists.")); } diff --git a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataHttpEditTest.java b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataHttpEditTest.java new file mode 100644 index 0000000..166216c --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataHttpEditTest.java @@ -0,0 +1,352 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.secureddata; + + +import java.io.Serializable; +import java.net.URI; +import java.net.URLEncoder; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.nio.charset.StandardCharsets; +import java.sql.Connection; +import java.sql.DriverManager; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TimeZone; +import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables; +import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface; +import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer; +import com.kingsrook.qqq.middleware.javalin.QJavalinImplementation; +import org.h2.util.DateTimeUtils; +import org.json.JSONObject; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert; +import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** A person edits a row of the QBit over HTTP, a day after it was written. The + ** request names the row by the id in its path, which arrives as text, while + ** the stored id is a number: the edit must still be an edit of that row. A + ** table with a modify date is edited the way a client that knows record + ** versions edits it, with the changed values only and the version it read. + *******************************************************************************/ +class SecuredDataHttpEditTest +{ + private QApplicationJavalinServer server; + private int port; + private final TimeZone zone = TimeZone.getDefault(); + + + + /******************************************************************************* + ** These tests run in UTC, whatever the zone of the machine. Core compares the + ** version a client sends with the stored modify date in the database, and + ** where the default zone of the JVM is not UTC that comparison never matches + ** (a version is then refused with 409 for a row nobody changed). That is + ** core's to fix. It is kept out of what is tested here, which is that an + ** edit is an edit of its row. The test database keeps the zone it first saw, + ** so it is told of each change. + *******************************************************************************/ + @BeforeEach + void utc() + { + TimeZone.setDefault(TimeZone.getTimeZone("UTC")); + DateTimeUtils.resetCalendar(); + } + + + + /******************************************************************************* + ** Cleanup. + *******************************************************************************/ + @AfterEach + void cleanup() + { + TimeZone.setDefault(zone); + DateTimeUtils.resetCalendar(); + if(server != null) + { + server.stop(); + } + QJavalinImplementation.setSessionCookieHttpOnly(false); + RetentionTestSupport.close(); + } + + + + /******************************************************************************* + ** The session of every request: an administrator of the QBit. + *******************************************************************************/ + public static class Sessions implements QAuthenticationModuleCustomizerInterface + { + /******************************************************************************* + ** Customize session. + *******************************************************************************/ + public void customizeSession(QInstance instance, QSession session, Map context) + { + session.setPermissions(Set.of("secured_data.admin", "secured_data.hasAccess", "secured_data.retention", "reading.read", "reading.write")); + } + } + + + + /******************************************************************************* + ** Serve the instance, a day after its rows were written. + *******************************************************************************/ + private void serveADayLater(QInstance instance) throws Exception + { + instance.getAuthentication().setType(QAuthenticationType.MOCK); + instance.getAuthentication().setCustomizer(new QCodeReference(Sessions.class)); + server = new QApplicationJavalinServer(new AbstractQQQApplication() + { + public QInstance defineQInstance() + { + return (instance); + } + }).withPort(0).withServeFrontendNext(false).withServeFrontendMaterialDashboard(false).withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true); + server.start(); + var serviceField = QApplicationJavalinServer.class.getDeclaredField("service"); + serviceField.setAccessible(true); + port = ((io.javalin.Javalin) serviceField.get(server)).port(); + CLOCK.now = NOW.plusSeconds(86400); + } + + + + /******************************************************************************* + ** Request. + *******************************************************************************/ + private HttpResponse request(String method, String path, String body) throws Exception + { + var builder = HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + port + "/qqq/v1" + path)).header("Cookie", "sessionUUID=edit-user"); + if(body != null) + { + builder.header("Content-Type", "application/json").method(method, HttpRequest.BodyPublishers.ofString(body)); + } + return (HttpClient.newHttpClient().send(builder.build(), HttpResponse.BodyHandlers.ofString())); + } + + + + /******************************************************************************* + ** The version a client reads with a row: the value of its modify date. + *******************************************************************************/ + private String version(String table, Serializable id, String modifyDate) throws Exception + { + HttpResponse get = request("GET", "/table/" + table + "/" + id, null); + assertThat(get.statusCode()).as(get.body()).isEqualTo(200); + JSONObject body = new JSONObject(get.body()); + JSONObject values = body.has("record") ? body.getJSONObject("record").getJSONObject("values") : body.getJSONObject("values"); + return (values.isNull(modifyDate) ? null : String.valueOf(values.get(modifyDate))); + } + + + + /******************************************************************************* + ** Edit a row: the changed values, and the version that was read where the + ** client sends one. + *******************************************************************************/ + private HttpResponse patch(String table, Serializable id, String version, String body) throws Exception + { + return (request("PATCH", "/table/" + table + "/" + id + (version == null ? "" : "?expectedVersion=" + URLEncoder.encode(version, StandardCharsets.UTF_8)), body)); + } + + + + /******************************************************************************* + ** One row as the database holds it, column by column. + *******************************************************************************/ + private static Map stored(String table, Serializable id) throws Exception + { + try(Connection connection = DriverManager.getConnection(SecuredDataStandardDatesTest.url(), "sa", "")) + { + return (SecuredDataStandardDatesTest.rows(connection, table, false).stream().filter(row -> String.valueOf(id).equals(row.get("id"))).findFirst().orElseThrow()); + } + } + + + + /******************************************************************************* + ** The columns an edit must leave as they were. + *******************************************************************************/ + private static void assertKept(Map before, Map after, String... columns) + { + for(String column : columns) + { + assertThat(before.get(column)).as(column + " before").isNotNull(); + assertThat(after.get(column)).as(column).isEqualTo(before.get(column)); + } + } + + + + /******************************************************************************* + ** The reason of a hold is edited: only the reason is sent, with the version. + ** The hold stays the hold it was, and the version it was edited with is then + ** no longer its version. + *******************************************************************************/ + @Test + void aPersonEditsTheReasonOfAHold() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + Serializable id = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("reason", "Preservation")).getValue("id"); + serveADayLater(instance); + Map placed = stored(SecuredDataTables.LEGAL_HOLD, id); + String version = version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate"); + assertThat(version).isNotNull(); + + HttpResponse edit = patch(SecuredDataTables.LEGAL_HOLD, id, version, "{\"reason\":\"Preservation, extended\"}"); + assertThat(edit.statusCode()).as(edit.body()).isEqualTo(200); + Map edited = stored(SecuredDataTables.LEGAL_HOLD, id); + assertThat(edited.get("reason")).isEqualTo("Preservation, extended"); + assertKept(placed, edited, "table_name", "status", "placed_at", "placed_by", "create_date"); + assertThat(edited.get("released_at")).isNull(); + assertThat(edited.get("modify_date")).isNotNull().isNotEqualTo(placed.get("modify_date")); + + HttpResponse stale = patch(SecuredDataTables.LEGAL_HOLD, id, version, "{\"reason\":\"Preservation, extended again\"}"); + assertThat(stale.statusCode()).as(stale.body()).isEqualTo(409); + assertThat(stored(SecuredDataTables.LEGAL_HOLD, id)).isEqualTo(edited); + } + + + + /******************************************************************************* + ** A hold is released a day after it was placed: when and by whom it was + ** placed, and its create date, are those of the day it was placed. + *******************************************************************************/ + @Test + void aHoldReleasedADayLaterKeepsWhenAndByWhomItWasPlaced() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + Serializable id = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", "7").withValue("reason", "Preservation")).getValue("id"); + serveADayLater(instance); + Map placed = stored(SecuredDataTables.LEGAL_HOLD, id); + + HttpResponse release = patch(SecuredDataTables.LEGAL_HOLD, id, version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate"), "{\"status\":\"RELEASED\"}"); + assertThat(release.statusCode()).as(release.body()).isEqualTo(200); + Map released = stored(SecuredDataTables.LEGAL_HOLD, id); + assertThat(released.get("status")).isEqualTo("RELEASED"); + assertKept(placed, released, "table_name", "record_id", "reason", "placed_at", "placed_by", "create_date"); + assertThat(released.get("released_at")).isNotNull().isNotEqualTo(placed.get("placed_at")); + assertThat(released.get("released_by")).isNotNull(); + assertThat(released.get("modify_date")).isNotNull().isNotEqualTo(placed.get("modify_date")); + } + + + + /******************************************************************************* + ** A classification is edited by a client that sends its table and field back + ** with the change: the row is not taken for a second classification of the + ** same field, and it keeps its create date. + *******************************************************************************/ + @Test + void aClassificationIsEditedNamingItsTableAndField() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + Serializable id = insert(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "reading").withValue("field_name", "notes").withValue("classification", "PII")).getValue("id"); + serveADayLater(instance); + Map written = stored(SecuredDataTables.CLASSIFICATION, id); + + HttpResponse edit = patch(SecuredDataTables.CLASSIFICATION, id, null, "{\"table_name\":\"reading\",\"field_name\":\"notes\",\"classification\":\"PHI\"}"); + assertThat(edit.statusCode()).as(edit.body()).isEqualTo(200); + Map edited = stored(SecuredDataTables.CLASSIFICATION, id); + assertThat(edited.get("classification")).isEqualTo("PHI"); + assertKept(written, edited, "table_name", "field_name", "create_date"); + assertThat(edited.get("modify_date")).isNotNull().isNotEqualTo(written.get("modify_date")); + } + + + + /******************************************************************************* + ** A policy and a retention rule are renamed: each keeps its create date and + ** everything else it had, and the rule is validated as the rule it is. + *******************************************************************************/ + @Test + void aRenamedPolicyOrRetentionRuleKeepsItsCreateDate() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + Serializable policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE")).getValue("id"); + Serializable rule = RetentionTestSupport.rule("OVERWRITE_TEXT", "secret").getValue("id"); + serveADayLater(instance); + Map policyWritten = stored(SecuredDataTables.POLICY, policy); + Map ruleWritten = stored(SecuredDataTables.RETENTION_RULE, rule); + + HttpResponse policyEdit = patch(SecuredDataTables.POLICY, policy, null, "{\"name\":\"renamed\"}"); + assertThat(policyEdit.statusCode()).as(policyEdit.body()).isEqualTo(200); + Map policyEdited = stored(SecuredDataTables.POLICY, policy); + assertThat(policyEdited.get("name")).isEqualTo("renamed"); + assertKept(policyWritten, policyEdited, "permission_name", "classification", "tier", "create_date"); + assertThat(policyEdited.get("modify_date")).isNotNull().isNotEqualTo(policyWritten.get("modify_date")); + + HttpResponse ruleEdit = patch(SecuredDataTables.RETENTION_RULE, rule, null, "{\"name\":\"renamed\"}"); + assertThat(ruleEdit.statusCode()).as(ruleEdit.body()).isEqualTo(200); + Map ruleEdited = stored(SecuredDataTables.RETENTION_RULE, rule); + assertThat(ruleEdited.get("name")).isEqualTo("renamed"); + assertKept(ruleWritten, ruleEdited, "table_name", "field_names", "age_field_name", "action", "create_date"); + assertThat(ruleEdited.get("modify_date")).isNotNull().isNotEqualTo(ruleWritten.get("modify_date")); + } + + + + /******************************************************************************* + ** A row that has neither date, as a row is that was written before the two + ** columns existed and was not given its own time afterwards: it is read and + ** edited like any other, and gets a modify date when it is changed. + *******************************************************************************/ + @Test + void aRowWithoutDatesIsReadAndEdited() throws Exception + { + QInstance instance = RetentionTestSupport.setup(); + Serializable id = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("reason", "Preservation")).getValue("id"); + try(Connection connection = DriverManager.getConnection(SecuredDataStandardDatesTest.url(), "sa", "")) + { + for(String table : List.of(SecuredDataTables.LEGAL_HOLD, SecuredDataTables.ACCESS_LOG, SecuredDataTables.BREAK_GLASS)) + { + connection.createStatement().execute("UPDATE " + table + " SET create_date = NULL, modify_date = NULL"); + } + } + serveADayLater(instance); + assertThat(version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate")).isNull(); + + HttpResponse edit = patch(SecuredDataTables.LEGAL_HOLD, id, null, "{\"reason\":\"Preservation, extended\"}"); + assertThat(edit.statusCode()).as(edit.body()).isEqualTo(200); + Map edited = stored(SecuredDataTables.LEGAL_HOLD, id); + assertThat(edited.get("reason")).isEqualTo("Preservation, extended"); + assertThat(edited.get("create_date")).isNull(); + assertThat(edited.get("modify_date")).isNotNull(); + assertThat(version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate")).isNotNull(); + } +} diff --git a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java index 39061df..6c11b4b 100644 --- a/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java +++ b/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java @@ -666,7 +666,7 @@ class SecuredDataStandardDatesTest /******************************************************************************* ** The database RetentionTestSupport made from the shipped DDL. *******************************************************************************/ - private static String url() + static String url() { return ("jdbc:h2:mem:" + RetentionTestSupport.BACKEND + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1"); } -- 2.54.0