Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9f04dd3e91
|
No files matched your search
@@ -1,12 +1,12 @@
|
||||
version: 2.1
|
||||
setup: true
|
||||
orbs:
|
||||
qqq-orb: kingsrook/qqq-orb@0.8
|
||||
qqq-orb: kingsrook/qqq-orb@0.7.12
|
||||
workflows:
|
||||
setup:
|
||||
jobs:
|
||||
- qqq-orb/generate_pipeline:
|
||||
orb_version: '0.8'
|
||||
orb_version: 0.7.12
|
||||
filters:
|
||||
branches:
|
||||
ignore: automation/qbit-rc-pending
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
name: build
|
||||
on:
|
||||
push:
|
||||
jobs:
|
||||
maven:
|
||||
uses: self:KofTwentyTwo/ci-workflows/.gitea/workflows/maven.yaml@v1
|
||||
secrets: inherit
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
profile: qbit
|
||||
orb_version: '0.8'
|
||||
orb_version: 0.7.12
|
||||
default_branch: develop
|
||||
java_version: '21'
|
||||
browsers: true
|
||||
|
||||
@@ -4,23 +4,6 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added: a create date and a modify date on every table
|
||||
- Every table has a create date and a modify date: `DATE_TIME`, not editable, with QQQ's `CREATE_DATE` and `MODIFY_DATE` behaviors, in the columns `create_date` and `modify_date`. Seven tables did not have them: reveal grant, break-glass session, access log, wipe run, wipe marker, legal hold and audit key. Their fields are named `createDate` and `modifyDate`. Classification, policy and retention rule already had `create_date` and `modify_date`; they keep those field names, and the two fields are now not editable and have the behaviors.
|
||||
- A new row gets both dates, equal, in the insert that writes it. A changed row keeps its create date and gets a later modify date. The access log and wipe runs are only ever inserted, so their dates stay equal. The access-log writer still runs one insert per row, with no other statement, lock or transaction.
|
||||
- **Upgrade: run the DDL first.** Before deploying this release on an existing database, run the last block of `docs/ddl/secured-data-postgres.sql` (from the line `-- Standard dates:` to the end). It adds the two nullable columns to the seven tables, defaults them to the current time and gives existing rows the row's own time. It rewrites no table and can be run while the release before this one is running, which goes on working with the columns present. This release does not run without them. See "Upgrading an existing database" in the README.
|
||||
- The dates have no classification, policy or assistant exposure of their own, and nothing that is signed or verified covers them. Rows written before the columns existed are unchanged apart from the two columns: wrapped data keys unwrap as before and audit tokens verify as before.
|
||||
- QQQ uses a field with the `MODIFY_DATE` behavior as the record version of its table. That now holds for every table of the QBit, classification, policy and retention rule included: on an update core sets the modify date itself, and an update that sends the version it read is refused if the row has changed since. Updates that send no version, the QBit's own among them, behave as before.
|
||||
|
||||
### Fixed: a reveal and the access-log writer waiting for each other
|
||||
- A reveal, or the start, end or expiry of a break-glass session, no longer deadlocks with a thread that asks for a classification while it holds another lock of this QBit. The access-log writer is such a thread: it holds the anomaly counters while it inserts a row, and that insert asks for classifications (it counts earlier rows and writes an audit, both queries with criteria). A reveal holds the runtime's monitor and then inserts its own access-log row. When both happened at once, neither finished, and neither did any later request, because every request asks for a classification. A person who opens a record with a masked field and reveals it straight away does both at once.
|
||||
- The rule rows a node keeps (classifications, active retention rules) have a lock of their own, held only while they are answered or read. Asking for a classification no longer takes the monitor that reveal and break-glass hold for their whole operation.
|
||||
|
||||
### Fixed: rule reads that ask for their own rows
|
||||
- Reading the rule rows a node keeps (classifications, active retention rules, policies) no longer recurses when code that runs inside that read asks for the same rows. Another QBit's virtual field value provider, or a post-query customizer, on a secured-data table that ran a query with filter criteria did exactly that: with one classification row present, every request that needed a classification ended in a `StackOverflowError`.
|
||||
- The QBit's system reads of its own tables (`SystemWrites.query`, the audit key lookup) name the stored fields of the table, so core runs no virtual field value provider inside them.
|
||||
- A question asked from inside a rule read is answered from the rows held when that read began (none before the first read) and starts no other read. Only the reading thread, inside the QBit's system read, can be given that answer; every request gets the rows the read returns. The same holds for a transaction's read of its own retention rule edits.
|
||||
- A rule read that fails is tried again by the next question instead of failing or recursing from then on.
|
||||
|
||||
### Fixed: access-log lifecycle
|
||||
- The access-log worker and JVM shutdown hook hold the logger weakly, so an instance the host drops (tests, reloads) can be collected; its worker stops at the next tick.
|
||||
- `withRegisterShutdownHook(false)` for hosts that call `SecuredDataRuntime.close()` themselves; `SecuredDataAccessLogger.isWorkerRunning()`.
|
||||
|
||||
@@ -30,29 +30,7 @@ new SecuredDataQBitProducer()
|
||||
|
||||
Register the host backend and application tables before producing the QBit. Enable core permissions (`qInstance.withEnforcePermissionsInCoreActions(true)`) at application entry points. Only one secured-data runtime is supported per instance. A namespace labels the QBit identity; table names remain fixed.
|
||||
|
||||
The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata (the fields `createDate` and `modifyDate` are the columns `create_date` and `modify_date`); an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode.
|
||||
|
||||
### Upgrading an existing database: create and modify dates
|
||||
|
||||
Every table has the columns `create_date` and `modify_date`, and the QBit reads and writes them on all ten tables. Seven tables were first created without them: `secured_data_reveal_grant`, `secured_data_break_glass`, `secured_data_access_log`, `secured_data_wipe_run`, `secured_data_wipe_marker`, `secured_data_legal_hold` and `secured_data_audit_key`.
|
||||
|
||||
Before deploying a release that has the two dates on every table, run the last block of [the DDL](docs/ddl/secured-data-postgres.sql) on the existing database: everything from the line `-- Standard dates:` to the end of the file. The order matters. The release before it runs unchanged once the columns are there. The release with the dates does not run without them: its reads of the seven tables fail, and with them every request that needs a field decision. The block does three things to each of the seven tables:
|
||||
|
||||
- It adds the two columns, nullable. That changes the catalog only; the table is not rewritten.
|
||||
- It gives each column a default of the current time, so a release that does not know the columns goes on inserting while both releases run.
|
||||
- It gives the rows that were already there the row's own time (below). The `UPDATE` touches only rows whose `create_date` is null, so the block can be run again and then changes nothing. It takes row locks only, and on a large access log it can be run in ranges of `id`.
|
||||
|
||||
| Table | `create_date` of an existing row | `modify_date` of an existing row |
|
||||
|---|---|---|
|
||||
| `secured_data_reveal_grant` | `granted_at` | `granted_at` |
|
||||
| `secured_data_break_glass` | `started_at` | `ended_at`, else `started_at` |
|
||||
| `secured_data_access_log` | `created_at` | `created_at` |
|
||||
| `secured_data_wipe_run` | `finished_at`, else `started_at` | the same |
|
||||
| `secured_data_wipe_marker` | `wiped_at` | `wiped_at` |
|
||||
| `secured_data_legal_hold` | `placed_at`, else the time of the upgrade | `released_at`, else `placed_at`, else the time of the upgrade |
|
||||
| `secured_data_audit_key` | `created_at` | `created_at` |
|
||||
|
||||
Nothing else changes in a row: wrapped data keys, search fingerprints and every other column are left as they are.
|
||||
The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata; an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode.
|
||||
|
||||
## Tables, screens and permissions
|
||||
|
||||
@@ -66,12 +44,6 @@ Nothing else changes in a row: wrapped data keys, search fingerprints and every
|
||||
|
||||
Every table has audit rules, explicit record sections and CONFIDENTIAL code classification. The Secured Data app navigates to Classification, Policy, Access log and Break-glass session. Reveal grants are a read-only child list joined by `access_log_id`; no process or home widget appears in navigation. The classification target `(table_name, field_name)` is unique, including the null field used for a table-wide override.
|
||||
|
||||
Every table has a create date and a modify date: date-time fields that nobody edits, in the columns `create_date` and `modify_date`. The fields are named `createDate` and `modifyDate`. Classification, Policy and Retention rules had the two dates before the other tables did and keep the field names `create_date` and `modify_date`. A new row gets both, equal. A changed row keeps its create date and gets a later modify date. Every write of the QBit is a QQQ insert or update action, so the dates are in the one statement that writes the row, the access-log writer's insert included. The access log and wipe runs are only ever inserted, so their two dates stay equal.
|
||||
|
||||
The two dates have no classification of their own. They are CONFIDENTIAL with the rest of their table, are given the tier of its other fields, and are closed to the assistant with it. Nothing that is signed or verified covers them: a wrapped data key is bound to its table, field and record id, and the search fingerprints of a log row are made from the search values.
|
||||
|
||||
The modify date has QQQ's modify-date behavior, and QQQ uses such a field as the record version of its table. On an update core sets it to the current time (or one millisecond after the stored value, if that is later). When a caller sends the version it read, core refuses the update if the row has changed since. The QBit's own updates send no version.
|
||||
|
||||
`secured_data.admin` gates configuration editing and activity reads through a standard QQQ custom permission checker. It gives no implicit access to classified fields on application tables. `secured_data.hasAccess` is the catalog base permission; the custom checker requires admin. Activity writes are refused for ordinary users even when they hold admin. Customizers also protect system-only writes, and update/delete capabilities are disabled on the access log, including system calls.
|
||||
|
||||
Both table features are registered for **all tables**: `secured_data_full` (Secured data: full) and `secured_data_reveal` (Secured data: reveal). This deliberate catalog choice supports new admin classification rows without a restart. Core's permission-base, READ prerequisite and NOT_PROTECTED semantics are preserved: NOT_PROTECTED application tables grant their features without an explicit permission. Use protected table permission rules when field policies must restrict users. Feature grants have no standard WRITE/admin implication.
|
||||
|
||||
@@ -138,52 +138,3 @@ CREATE TABLE secured_data_audit_key (
|
||||
CREATE INDEX secured_data_hold_lookup ON secured_data_legal_hold (table_name, status, record_id);
|
||||
CREATE INDEX secured_data_retention_lookup ON secured_data_retention_rule (table_name, is_active);
|
||||
CREATE INDEX secured_data_wipe_rule_lookup ON secured_data_wipe_marker (rule_id);
|
||||
|
||||
-- Standard dates: create_date and modify_date on the seven tables created above without them.
|
||||
-- A new database runs the whole file. An existing database runs this block alone, BEFORE the release
|
||||
-- that writes the two columns is deployed. The columns are nullable and default to the current time, so
|
||||
-- a release that does not know them goes on inserting while both run. Adding a nullable column and
|
||||
-- setting a default change the catalog only (no table rewrite). Each UPDATE gives the rows that were
|
||||
-- already there the row's own time, once: it touches only rows whose create_date is still null, takes
|
||||
-- row locks only, and on a large access log can be run in ranges of id.
|
||||
ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_reveal_grant ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_reveal_grant ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_reveal_grant SET create_date = granted_at, modify_date = granted_at WHERE create_date IS NULL;
|
||||
|
||||
ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_break_glass ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_break_glass ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_break_glass SET create_date = started_at, modify_date = COALESCE(ended_at, started_at) WHERE create_date IS NULL;
|
||||
|
||||
ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_access_log ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_access_log ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_access_log SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL;
|
||||
|
||||
ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_wipe_run ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_wipe_run ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_wipe_run SET create_date = COALESCE(finished_at, started_at), modify_date = COALESCE(finished_at, started_at) WHERE create_date IS NULL;
|
||||
|
||||
ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_wipe_marker ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_wipe_marker ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_wipe_marker SET create_date = wiped_at, modify_date = wiped_at WHERE create_date IS NULL;
|
||||
|
||||
ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_legal_hold ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_legal_hold ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_legal_hold SET create_date = COALESCE(placed_at, CURRENT_TIMESTAMP), modify_date = COALESCE(released_at, placed_at, CURRENT_TIMESTAMP) WHERE create_date IS NULL;
|
||||
|
||||
ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
||||
ALTER TABLE secured_data_audit_key ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
ALTER TABLE secured_data_audit_key ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
||||
UPDATE secured_data_audit_key SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL;
|
||||
@@ -22,10 +22,9 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
|
||||
<properties>
|
||||
<qqq.version>4.1.0-RC.26</qqq.version> <!-- feeds qbit-build-parent's qqq-bom-pom import -->
|
||||
<!-- QQQ 4.1.0-RC.15 ships UI API 1.6.0, field access and audit value protection -->
|
||||
<qqq.snapshot.version>4.1.0-RC.26</qqq.snapshot.version>
|
||||
<revision>0.1.0-RC.8</revision>
|
||||
<qqq.snapshot.version>4.1.0-RC.15</qqq.snapshot.version>
|
||||
<revision>0.1.0-SNAPSHOT</revision>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
<jacoco.version>0.8.15</jacoco.version>
|
||||
@@ -35,7 +34,7 @@
|
||||
<parent>
|
||||
<groupId>com.kingsrook</groupId>
|
||||
<artifactId>qbit-build-parent</artifactId>
|
||||
<version>2.2.1</version>
|
||||
<version>2.0.0</version>
|
||||
</parent>
|
||||
|
||||
<groupId>com.kingsrook.qbits</groupId>
|
||||
@@ -93,7 +92,7 @@
|
||||
<dependency>
|
||||
<groupId>com.kingsrook.qqq</groupId>
|
||||
<artifactId>qqq-frontend-next</artifactId>
|
||||
<version>1.0.0-RC.38</version>
|
||||
<version>1.0.0-RC.14</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
|
||||
@@ -1,103 +0,0 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.secureddata;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import java.util.function.Supplier;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One kind of rule rows a node keeps in memory (classifications, policies or
|
||||
** active retention rules): read once per version, never used unread past
|
||||
** SecuredDataRuntime.MAX_RULE_AGE, and safe against a read that asks for the
|
||||
** same rows again.
|
||||
**
|
||||
** Not synchronized: every call must be made holding the owner's lock (the
|
||||
** runtime's rule lock for classifications and retention rules, the resolver's
|
||||
** monitor for policies). That lock is what makes the re-entry answer below
|
||||
** private to one thread: while a read is running, only the thread running it
|
||||
** can get in here; every other thread waits for the lock and is given the rows
|
||||
** that read returns.
|
||||
**
|
||||
** Re-entry: reading rule rows is itself a query, and anything that runs inside
|
||||
** it (a table customizer, a record-security lookup, a value provider) may ask a
|
||||
** question that needs the very rows being read. Such a nested question is
|
||||
** answered with the rows this node held when the read began - the answer every
|
||||
** request was given until now, and no rows before the first read - and never
|
||||
** starts another read. The request that started the read gets the new rows.
|
||||
**
|
||||
** Failure: a read that throws leaves the previous rows and version in place
|
||||
** and is not remembered, so the next question reads again. The previous rows
|
||||
** are never handed to a request in place of rows that could not be read.
|
||||
*******************************************************************************/
|
||||
final class RuleRows
|
||||
{
|
||||
private List<QRecord> rows = List.of();
|
||||
private long version;
|
||||
private long loadedAt;
|
||||
private boolean loaded;
|
||||
private boolean reading;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The rows of this version, reading them when the rows held are of another
|
||||
** version, too old, or were never read.
|
||||
**
|
||||
** @param currentVersion the owner's invalidation clock, read by the caller
|
||||
** before this call: a change during the read leaves a
|
||||
** version that no longer matches, so the next question
|
||||
** reads again
|
||||
** @param read the query; whatever it throws is thrown to the caller
|
||||
*******************************************************************************/
|
||||
List<QRecord> get(long currentVersion, Supplier<List<QRecord>> read)
|
||||
{
|
||||
if(loaded && version == currentVersion && !SecuredDataRuntime.tooOld(loadedAt))
|
||||
{
|
||||
return (rows);
|
||||
}
|
||||
if(reading)
|
||||
{
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
// the read that is running on this thread asked for its own rows: give //
|
||||
// it the rows held when it began, and do not start another read //
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
return (rows);
|
||||
}
|
||||
reading = true;
|
||||
try
|
||||
{
|
||||
long startedAt = System.nanoTime();
|
||||
List<QRecord> fresh = List.copyOf(read.get());
|
||||
rows = fresh;
|
||||
version = currentVersion;
|
||||
loadedAt = startedAt;
|
||||
loaded = true;
|
||||
}
|
||||
finally
|
||||
{
|
||||
reading = false;
|
||||
}
|
||||
return (rows);
|
||||
}
|
||||
}
|
||||
@@ -124,7 +124,7 @@ public final class SecuredDataKeySource implements ClassifiedValueKeySource, Aut
|
||||
try
|
||||
{
|
||||
List<QRecord> rows = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)
|
||||
.withFilter(recordField(table, field, pk)).withShouldOmitHiddenFields(false).withFieldNamesToInclude(SystemWrites.storedFields(SecuredDataTables.AUDIT_KEY))
|
||||
.withFilter(recordField(table, field, pk)).withShouldOmitHiddenFields(false)
|
||||
.withTransaction(QContext.getQBackendTransaction())).getRecords());
|
||||
if(rows.isEmpty())
|
||||
{
|
||||
|
||||
@@ -74,14 +74,10 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
|
||||
{
|
||||
}
|
||||
|
||||
/** Used only inside methods synchronized on this resolver. */
|
||||
private final RuleRows policyRows = new RuleRows();
|
||||
private long cachedVersion;
|
||||
private List<QRecord> policies = List.of();
|
||||
private final Map<String, UserState> users = new HashMap<>();
|
||||
|
||||
/** Request-scoped (QContext) holder of the access fingerprint. */
|
||||
private static final String FINGERPRINT_KEY = SecuredDataResolver.class.getName() + ".accessFingerprint";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
@@ -292,19 +288,16 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
|
||||
|
||||
/*******************************************************************************
|
||||
** Refresh policy rows once per version and matched state per relevant permissions.
|
||||
** A read that fails throws and is tried again by the next call. A question
|
||||
** asked from inside the read itself (see RuleRows) is answered from the
|
||||
** policies held when the read began; whatever it matched for its user is
|
||||
** dropped with every other user's state when the read returns.
|
||||
*******************************************************************************/
|
||||
private synchronized UserState state()
|
||||
{
|
||||
policies = policyRows.get(runtime.policyVersion(), () ->
|
||||
long version = runtime.policyVersion();
|
||||
if(cachedVersion != version)
|
||||
{
|
||||
List<QRecord> rows = SystemWrites.query(SecuredDataTables.POLICY, new QQueryFilter());
|
||||
policies = List.copyOf(SystemWrites.query(SecuredDataTables.POLICY, new QQueryFilter()));
|
||||
users.clear();
|
||||
return (rows);
|
||||
});
|
||||
cachedVersion = version;
|
||||
}
|
||||
QSession session = QContext.getQSession();
|
||||
TreeSet<String> held = new TreeSet<>();
|
||||
for(QRecord policy : policies)
|
||||
@@ -315,7 +308,7 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
|
||||
held.add(permission);
|
||||
}
|
||||
}
|
||||
String key = userId() + '\u0000' + String.join("\u0000", held);
|
||||
String key = JsonUtils.toJson(List.of(userId(), held));
|
||||
UserState state = users.get(key);
|
||||
if(state == null)
|
||||
{
|
||||
@@ -363,82 +356,11 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
|
||||
** Stable across unrelated permissions/records; includes channel and exact expiry.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getAccessFingerprint()
|
||||
public synchronized String getAccessFingerprint()
|
||||
{
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
// core asks for the fingerprint before every field decision (once per //
|
||||
// field of every record of a result), so the value is kept for the //
|
||||
// request - and only while everything it is derived from is unchanged //
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
QSession session = QContext.getQSession();
|
||||
String channel = Objects.requireNonNullElse(SecuredDataMetadata.channel(session), "");
|
||||
Instant now = runtime.config().getClock().instant();
|
||||
RequestFingerprint kept = QContext.getObject(FINGERPRINT_KEY, RequestFingerprint.class).orElse(null);
|
||||
if(kept != null && kept.matches(this, session, channel, runtime.policyVersion(), runtime.classificationVersion(), now))
|
||||
{
|
||||
return (kept.value());
|
||||
}
|
||||
RequestFingerprint computed = computeAccessFingerprint(session, channel, now);
|
||||
QContext.setObject(FINGERPRINT_KEY, computed);
|
||||
return (computed.value());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint and what it was derived from. It stops matching when the
|
||||
** session, its permissions, the channel, a policy or classification version
|
||||
** changes, or when the first active break-glass grant expires.
|
||||
*******************************************************************************/
|
||||
private record RequestFingerprint(int resolver, QSession session, int permissionsIdentity, int permissionCount, String channel, long policyVersion,
|
||||
long classificationVersion, Instant validUntil, String value) implements Serializable
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Whether the kept value is still the one a fresh computation would give.
|
||||
*******************************************************************************/
|
||||
boolean matches(SecuredDataResolver owner, QSession current, String currentChannel, long currentPolicyVersion, long currentClassificationVersion, Instant now)
|
||||
{
|
||||
return (resolver == System.identityHashCode(owner) && session == current && permissionsIdentity == identityOfPermissions(current) && permissionCount == countOfPermissions(current)
|
||||
&& channel.equals(currentChannel) && policyVersion == currentPolicyVersion && classificationVersion == currentClassificationVersion && now.isBefore(validUntil));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The permission set object of a session (it is replaced, not edited, when a
|
||||
** session is given new permissions).
|
||||
*******************************************************************************/
|
||||
private static int identityOfPermissions(QSession session)
|
||||
{
|
||||
return (session == null || session.getPermissions() == null ? 0 : System.identityHashCode(session.getPermissions()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** How many permissions a session holds (an edit in place changes it).
|
||||
*******************************************************************************/
|
||||
private static int countOfPermissions(QSession session)
|
||||
{
|
||||
return (session == null || session.getPermissions() == null ? -1 : session.getPermissions().size());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable across unrelated permissions/records; includes channel and exact expiry.
|
||||
*******************************************************************************/
|
||||
private synchronized RequestFingerprint computeAccessFingerprint(QSession session, String channel, Instant now)
|
||||
{
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
// versions are read before the rows they describe, so a change during the //
|
||||
// computation leaves a kept value that no longer matches //
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
long policyVersion = runtime.policyVersion();
|
||||
long classificationVersion = runtime.classificationVersion();
|
||||
state();
|
||||
TreeSet<String> relevant = new TreeSet<>();
|
||||
List<QRecord> breakGlass = state().breakGlass();
|
||||
QSession session = QContext.getQSession();
|
||||
for(QRecord row : policies)
|
||||
{
|
||||
String permission = row.getValueString("permission_name");
|
||||
@@ -458,15 +380,12 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
|
||||
}
|
||||
}
|
||||
}
|
||||
List<QRecord> activeBreaks = breakGlass.stream().filter(row -> "ACTIVE".equals(row.getValueString("status")))
|
||||
.filter(row -> row.getValueInstant("expires_at") != null && row.getValueInstant("expires_at").isAfter(now)).toList();
|
||||
List<String> breaks = activeBreaks.stream().map(row -> row.getValueString("id")).sorted().toList();
|
||||
Instant validUntil = activeBreaks.stream().map(row -> row.getValueInstant("expires_at")).min(Instant::compareTo).orElse(Instant.MAX);
|
||||
String payload = JsonUtils.toJson(List.of(userId(), relevant, policyVersion, classificationVersion, breaks, channel));
|
||||
List<String> breaks = activeBreakGlass().stream().map(row -> row.getValueString("id")).sorted().toList();
|
||||
String payload = JsonUtils.toJson(List.of(userId(), relevant, runtime.policyVersion(), runtime.classificationVersion(), breaks,
|
||||
Objects.requireNonNullElse(SecuredDataMetadata.channel(session), "")));
|
||||
try
|
||||
{
|
||||
String value = Base64.getUrlEncoder().withoutPadding().encodeToString(MessageDigest.getInstance("SHA-256").digest(payload.getBytes(StandardCharsets.UTF_8)));
|
||||
return (new RequestFingerprint(System.identityHashCode(this), session, identityOfPermissions(session), countOfPermissions(session), channel, policyVersion, classificationVersion, validUntil, value));
|
||||
return (Base64.getUrlEncoder().withoutPadding().encodeToString(MessageDigest.getInstance("SHA-256").digest(payload.getBytes(StandardCharsets.UTF_8))));
|
||||
}
|
||||
catch(java.security.NoSuchAlgorithmException e)
|
||||
{
|
||||
|
||||
@@ -21,21 +21,9 @@
|
||||
package com.kingsrook.qbits.secureddata;
|
||||
|
||||
|
||||
import java.lang.ref.WeakReference;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
|
||||
import com.kingsrook.qbits.secureddata.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.cluster.ClusterBus;
|
||||
import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;
|
||||
import com.kingsrook.qqq.backend.core.cluster.ClusterEvent;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData;
|
||||
|
||||
@@ -46,58 +34,16 @@ import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaDa
|
||||
public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCloseable
|
||||
{
|
||||
public static final String NAME = "securedDataRuntime";
|
||||
|
||||
private static final QLogger LOG = QLogger.getLogger(SecuredDataRuntime.class);
|
||||
|
||||
/////////////////////////////////////////////////////////////////////////////////
|
||||
// the three sets of rule rows a node keeps in memory. A change clears them on //
|
||||
// this node at once and, through QQQ's cluster bus (ClusterCaches), on every //
|
||||
// other node. MAX_RULE_AGE is the safety net for a lost event: no node uses //
|
||||
// rule rows older than that without reading them again. //
|
||||
/////////////////////////////////////////////////////////////////////////////////
|
||||
public static final String POLICY_CACHE = "securedData.policies";
|
||||
public static final String CLASSIFICATION_CACHE = "securedData.classifications";
|
||||
public static final String RETENTION_CACHE = "securedData.retentionRules";
|
||||
|
||||
public static final Duration MAX_RULE_AGE = Duration.ofSeconds(30);
|
||||
|
||||
static
|
||||
{
|
||||
ClusterCaches.shared(POLICY_CACHE, POLICY_CACHE, () -> clearOnThisNode(POLICY_CACHE), null);
|
||||
ClusterCaches.shared(CLASSIFICATION_CACHE, CLASSIFICATION_CACHE, () -> clearOnThisNode(CLASSIFICATION_CACHE), null);
|
||||
ClusterCaches.shared(RETENTION_CACHE, RETENTION_CACHE, () -> clearOnThisNode(RETENTION_CACHE), null);
|
||||
}
|
||||
|
||||
private static final AtomicLong STARTUP_VERSION = new AtomicLong(System.currentTimeMillis());
|
||||
private final AtomicLong classificationVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
|
||||
private final AtomicLong policyVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
|
||||
private final AtomicLong retentionVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
|
||||
private final transient WeakReference<QInstance> owner;
|
||||
private final transient SecuredDataQBitConfig config;
|
||||
private final transient SecuredDataResolver resolver;
|
||||
private final transient SecuredDataKeySource keys = new SecuredDataKeySource();
|
||||
private final transient SecuredDataAccessLogger logger;
|
||||
private final transient SecuredDataAnomalies anomalies;
|
||||
/////////////////////////////////////////////////////////////////////////////////
|
||||
// both are used only while holding ruleLock: one lock for the two, because //
|
||||
// reading either may ask for the other. //
|
||||
// //
|
||||
// It is a lock of its own, never this runtime's monitor. A reveal and the //
|
||||
// start, end and expiry of a break-glass session hold the runtime's monitor //
|
||||
// for their whole operation, and every thread asks for a classification all //
|
||||
// the time (core asks for every field it decides on and for every criterion //
|
||||
// of every query), including threads that hold another lock of this QBit: the //
|
||||
// access-log writer inside the anomaly counters, the resolver, the key //
|
||||
// source. When the question needed the runtime's monitor, such a thread and //
|
||||
// a reveal that needed its lock next waited for each other for ever. //
|
||||
// //
|
||||
// ruleLock is held only while rule rows are answered or read. A read is a //
|
||||
// query made as the system: it takes none of the QBit's other locks, so no //
|
||||
// thread holding ruleLock ever waits for one of them. //
|
||||
/////////////////////////////////////////////////////////////////////////////////
|
||||
private final transient Object ruleLock = new Object();
|
||||
private final transient RuleRows classificationRows = new RuleRows();
|
||||
private final transient RuleRows activeRetentionRules = new RuleRows();
|
||||
private transient long cachedClassificationVersion;
|
||||
private transient List<com.kingsrook.qqq.backend.core.model.data.QRecord> classificationRows = List.of();
|
||||
|
||||
|
||||
|
||||
@@ -107,7 +53,6 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
|
||||
public SecuredDataRuntime(QInstance instance, SecuredDataQBitConfig config)
|
||||
{
|
||||
this.config = config;
|
||||
this.owner = new WeakReference<>(instance);
|
||||
this.resolver = new SecuredDataResolver(this);
|
||||
this.anomalies = new SecuredDataAnomalies(config);
|
||||
this.logger = new SecuredDataAccessLogger(instance, this);
|
||||
@@ -192,7 +137,6 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
|
||||
public void invalidateClassifications()
|
||||
{
|
||||
classificationVersion.incrementAndGet();
|
||||
announce(CLASSIFICATION_CACHE);
|
||||
}
|
||||
|
||||
|
||||
@@ -203,115 +147,23 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
|
||||
public void invalidatePolicies()
|
||||
{
|
||||
policyVersion.incrementAndGet();
|
||||
announce(POLICY_CACHE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A retention rule changed: forget the active rules here and on every node.
|
||||
** Read classification rows once per version, always as the system.
|
||||
*******************************************************************************/
|
||||
public void invalidateRetentionRules()
|
||||
synchronized List<com.kingsrook.qqq.backend.core.model.data.QRecord> classificationRows()
|
||||
{
|
||||
retentionVersion.incrementAndGet();
|
||||
announce(RETENTION_CACHE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Version of the retention rules this node holds.
|
||||
*******************************************************************************/
|
||||
public long retentionVersion()
|
||||
{
|
||||
return (retentionVersion.get());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether rows loaded at this System.nanoTime are too old to use unread.
|
||||
*******************************************************************************/
|
||||
static boolean tooOld(long loadedAtNanos)
|
||||
{
|
||||
return (System.nanoTime() - loadedAtNanos > MAX_RULE_AGE.toNanos());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every active retention rule, read once per version and at most MAX_RULE_AGE
|
||||
** old. A read that fails throws and leaves nothing in place of the rules:
|
||||
** callers never see "no rules" because the rules could not be read, and the
|
||||
** next call reads again. A question asked from inside the read itself (see
|
||||
** RuleRows) is answered with the rules held when the read began.
|
||||
*******************************************************************************/
|
||||
public List<QRecord> activeRetentionRules()
|
||||
{
|
||||
synchronized(ruleLock)
|
||||
long version = classificationVersion();
|
||||
if(cachedClassificationVersion != version)
|
||||
{
|
||||
return (activeRetentionRules.get(retentionVersion(), () -> SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter()
|
||||
.withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true)))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Tell the other nodes (this node has already moved its version). A failure
|
||||
** to send is logged: the other nodes then read again within MAX_RULE_AGE.
|
||||
*******************************************************************************/
|
||||
private void announce(String cache)
|
||||
{
|
||||
try
|
||||
{
|
||||
QInstance instance = owner.get();
|
||||
if(instance != null)
|
||||
{
|
||||
ClusterBus.publish(instance, new ClusterEvent(ClusterEvent.TYPE_MEMO_INVALIDATE, cache, null));
|
||||
}
|
||||
}
|
||||
catch(RuntimeException e)
|
||||
{
|
||||
LOG.warn("Could not announce a secured-data rule change; other nodes read again within " + MAX_RULE_AGE.toSeconds() + " seconds", e);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A rule change announced on the cluster bus (also by this node): move the
|
||||
** version of the runtime of the instance the event arrived on.
|
||||
*******************************************************************************/
|
||||
private static void clearOnThisNode(String cache)
|
||||
{
|
||||
SecuredDataRuntime runtime = current();
|
||||
if(runtime == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
switch(cache)
|
||||
{
|
||||
case POLICY_CACHE -> runtime.policyVersion.incrementAndGet();
|
||||
case CLASSIFICATION_CACHE -> runtime.classificationVersion.incrementAndGet();
|
||||
default -> runtime.retentionVersion.incrementAndGet();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read classification rows once per version, always as the system. A read
|
||||
** that fails throws and is tried again by the next call. A question asked from
|
||||
** inside the read itself (see RuleRows) is answered with the rows held when
|
||||
** the read began, and never starts another read.
|
||||
*******************************************************************************/
|
||||
List<QRecord> classificationRows()
|
||||
{
|
||||
synchronized(ruleLock)
|
||||
{
|
||||
return (classificationRows.get(classificationVersion(), () -> SystemWrites.query(SecuredDataTables.CLASSIFICATION, new QQueryFilter())));
|
||||
classificationRows = List.copyOf(com.kingsrook.qbits.secureddata.support.SystemWrites.query(
|
||||
com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.CLASSIFICATION, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter()));
|
||||
cachedClassificationVersion = version;
|
||||
}
|
||||
return (classificationRows);
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -31,16 +31,12 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.AUDIT_KEY;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.CREATE_DATE;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.LEGAL_HOLD;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.MODIFY_DATE;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.RETENTION_RULE;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_MARKER;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_RUN;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.base;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.createDate;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.field;
|
||||
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.modifyDate;
|
||||
|
||||
/*******************************************************************************
|
||||
** Retention policies, value-free history, legal holds and mutable audit keys.
|
||||
@@ -78,8 +74,8 @@ public final class RetentionTables
|
||||
.withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true))
|
||||
.withField(field("last_run_at", QFieldType.DATE_TIME).withIsEditable(false))
|
||||
.withField(field("last_run_count", QFieldType.INTEGER).withIsEditable(false))
|
||||
.withField(createDate("create_date"))
|
||||
.withField(modifyDate("modify_date"))
|
||||
.withField(field("create_date", QFieldType.DATE_TIME))
|
||||
.withField(field("modify_date", QFieldType.DATE_TIME))
|
||||
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
|
||||
rule.withSection(SectionFactory.defaultT1("id", "name", "table_name", "field_names"));
|
||||
rule.withSection(SectionFactory.defaultT2("age_field_name", "age_days", "action", "overwrite_text", "audit_treatment", "archive_storage_name", "batch_size", "is_active"));
|
||||
@@ -96,11 +92,8 @@ public final class RetentionTables
|
||||
.withField(field("action", QFieldType.STRING).withMaxLength(40).withIsRequired(true))
|
||||
.withField(field("wiped_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
.withField(field("run_id", QFieldType.INTEGER).withIsRequired(true))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE))
|
||||
.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name")));
|
||||
marker.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "rule_id", "action", "wiped_at", "run_id"));
|
||||
marker.withSection(SectionFactory.defaultT3(CREATE_DATE, MODIFY_DATE));
|
||||
QTableMetaData hold = base(config, LEGAL_HOLD, "Legal hold")
|
||||
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
|
||||
.withField(field("record_id", QFieldType.STRING).withMaxLength(255))
|
||||
@@ -109,11 +102,9 @@ public final class RetentionTables
|
||||
.withField(field("placed_at", QFieldType.DATE_TIME))
|
||||
.withField(field("released_by", QFieldType.STRING).withMaxLength(255))
|
||||
.withField(field("released_at", QFieldType.DATE_TIME))
|
||||
.withField(field("status", QFieldType.STRING).withMaxLength(20).withDefaultValue("ACTIVE"))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE));
|
||||
.withField(field("status", QFieldType.STRING).withMaxLength(20).withDefaultValue("ACTIVE"));
|
||||
hold.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "reason", "status"));
|
||||
hold.withSection(SectionFactory.defaultT3("placed_by", "placed_at", "released_by", "released_at", CREATE_DATE, MODIFY_DATE));
|
||||
hold.withSection(SectionFactory.defaultT3("placed_by", "placed_at", "released_by", "released_at"));
|
||||
QTableMetaData run = base(config, WIPE_RUN, "Wipe run")
|
||||
.withField(field("rule_id", QFieldType.INTEGER).withIsRequired(true))
|
||||
.withField(field("started_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
@@ -124,11 +115,9 @@ public final class RetentionTables
|
||||
.withField(field("wiped_count", QFieldType.INTEGER))
|
||||
.withField(field("error_count", QFieldType.INTEGER))
|
||||
.withField(field("record_ids", QFieldType.STRING).withMaxLength(4000))
|
||||
.withField(field("message", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE));
|
||||
.withField(field("message", QFieldType.STRING).withMaxLength(500));
|
||||
run.withSection(SectionFactory.defaultT1("id", "rule_id", "status", "eligible_count", "held_count", "wiped_count", "error_count"));
|
||||
run.withSection(SectionFactory.defaultT3("started_at", "finished_at", "record_ids", "message", CREATE_DATE, MODIFY_DATE));
|
||||
run.withSection(SectionFactory.defaultT3("started_at", "finished_at", "record_ids", "message"));
|
||||
QTableMetaData key = base(config, AUDIT_KEY, "Audit key")
|
||||
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
|
||||
.withoutCapabilities(Capability.TABLE_EXPORT)
|
||||
@@ -138,11 +127,8 @@ public final class RetentionTables
|
||||
.withField(field("wrapped_key", QFieldType.TEXT).withIsRequired(true).withIsSensitive(true).withIsHidden(true)
|
||||
.withFieldSecurityLock(new FieldSecurityLock().withSecurityKeyType(KEY_LOCK).withDefaultBehavior(FieldSecurityLock.Behavior.DENY).withOverrideValues(List.of(KEY_LOCK_NEVER_GRANTED))))
|
||||
.withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE))
|
||||
.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name")));
|
||||
key.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "created_at"));
|
||||
key.withSection(SectionFactory.defaultT3(CREATE_DATE, MODIFY_DATE));
|
||||
return (List.of(rule, marker, hold, run, key));
|
||||
}
|
||||
}
|
||||
+6
-34
@@ -68,7 +68,6 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
|
||||
record.getValues().forEach(effective::setValue);
|
||||
validate(table, record, effective);
|
||||
Instant now = SecuredDataRuntime.current().config().getClock().instant();
|
||||
dates(table, record, old == null, now);
|
||||
if(SecuredDataTables.LEGAL_HOLD.equals(table))
|
||||
{
|
||||
if(old == null)
|
||||
@@ -83,41 +82,19 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if(old == null)
|
||||
{
|
||||
record.setValue("create_date", now);
|
||||
}
|
||||
record.setValue("modify_date", now);
|
||||
record.setValue("modified_by", SecuredDataResolver.userId());
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
Instant now = SecuredDataRuntime.current().config().getClock().instant();
|
||||
for(QRecord record : records)
|
||||
{
|
||||
dates(table, record, input instanceof InsertInput, now);
|
||||
}
|
||||
}
|
||||
return (records);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The two dates every table has, set in the write that is already being made:
|
||||
** a new row gets both, equal, from the clock every other time of this QBit
|
||||
** comes from, and a changed row a new modify date. Core has the last word on
|
||||
** a change: the modify date is the record version of its table, and core
|
||||
** moves it forward after this has run. A row of a table that is only ever
|
||||
** inserted (the access log, a wipe run) so keeps equal dates.
|
||||
*******************************************************************************/
|
||||
private static void dates(String table, QRecord record, boolean inserted, Instant now)
|
||||
{
|
||||
if(inserted)
|
||||
{
|
||||
record.setValue(SecuredDataTables.createDateField(table), now);
|
||||
}
|
||||
record.setValue(SecuredDataTables.modifyDateField(table), now);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Access logs are append-only even for the system.
|
||||
*******************************************************************************/
|
||||
@@ -261,8 +238,7 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
|
||||
}
|
||||
SecuredDataRuntime runtime = SecuredDataRuntime.current();
|
||||
Runnable invalidate = SecuredDataTables.CLASSIFICATION.equals(table) ? runtime::invalidateClassifications
|
||||
: SecuredDataTables.POLICY.equals(table) || SecuredDataTables.BREAK_GLASS.equals(table) ? runtime::invalidatePolicies
|
||||
: SecuredDataTables.RETENTION_RULE.equals(table) ? runtime::invalidateRetentionRules : null;
|
||||
: SecuredDataTables.POLICY.equals(table) || SecuredDataTables.BREAK_GLASS.equals(table) ? runtime::invalidatePolicies : null;
|
||||
if(invalidate != null)
|
||||
{
|
||||
QBackendTransaction active = transaction == null ? QContext.getQBackendTransaction() : transaction;
|
||||
@@ -272,10 +248,6 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
|
||||
}
|
||||
else
|
||||
{
|
||||
if(SecuredDataTables.RETENTION_RULE.equals(table))
|
||||
{
|
||||
com.kingsrook.qbits.secureddata.retention.RetentionRules.editedInTransaction();
|
||||
}
|
||||
active.addAfterCommitCallback(invalidate);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,7 +32,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
|
||||
@@ -67,8 +66,6 @@ public final class SecuredDataTables
|
||||
public static final List<String> ALL = List.of(CLASSIFICATION, POLICY, REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, RETENTION_RULE, WIPE_MARKER, LEGAL_HOLD, WIPE_RUN, AUDIT_KEY);
|
||||
public static final List<String> MAIN = List.of(CLASSIFICATION, POLICY, ACCESS_LOG, BREAK_GLASS, RETENTION_RULE);
|
||||
public static final List<String> SYSTEM_WRITTEN = List.of(REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, WIPE_MARKER, WIPE_RUN, AUDIT_KEY);
|
||||
public static final String CREATE_DATE = "createDate";
|
||||
public static final String MODIFY_DATE = "modifyDate";
|
||||
public static final String REVEALS_JOIN = "secured_data_access_log_reveals";
|
||||
public static final String REVEALS_WIDGET = "securedDataReveals";
|
||||
|
||||
@@ -136,51 +133,6 @@ public final class SecuredDataTables
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** When a row was written. Every table has this field and a modify date;
|
||||
** neither is editable, and the columns are create_date and modify_date.
|
||||
*******************************************************************************/
|
||||
static QFieldMetaData createDate(String name)
|
||||
{
|
||||
return (field(name, QFieldType.DATE_TIME).withBackendName("create_date").withLabel("create date")
|
||||
.withIsEditable(false).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** When a row last changed: its create date until something changes it.
|
||||
*******************************************************************************/
|
||||
static QFieldMetaData modifyDate(String name)
|
||||
{
|
||||
return (field(name, QFieldType.DATE_TIME).withBackendName("modify_date").withLabel("modify date")
|
||||
.withIsEditable(false).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The name of a table's create date field. Classification, policy and
|
||||
** retention rule had the two dates before the other tables did and keep the
|
||||
** field names they have always had.
|
||||
*******************************************************************************/
|
||||
static String createDateField(String table)
|
||||
{
|
||||
return (List.of(CLASSIFICATION, POLICY, RETENTION_RULE).contains(table) ? "create_date" : CREATE_DATE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The name of a table's modify date field (see createDateField).
|
||||
*******************************************************************************/
|
||||
static String modifyDateField(String table)
|
||||
{
|
||||
return (List.of(CLASSIFICATION, POLICY, RETENTION_RULE).contains(table) ? "modify_date" : MODIFY_DATE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Define Classification.
|
||||
*******************************************************************************/
|
||||
@@ -193,8 +145,8 @@ public final class SecuredDataTables
|
||||
.withField(field("mask_style", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("secured_data_mask_style"))
|
||||
.withField(field("low_cardinality", QFieldType.BOOLEAN))
|
||||
.withField(field("notes", QFieldType.TEXT))
|
||||
.withField(createDate("create_date"))
|
||||
.withField(modifyDate("modify_date"))
|
||||
.withField(field("create_date", QFieldType.DATE_TIME))
|
||||
.withField(field("modify_date", QFieldType.DATE_TIME))
|
||||
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
|
||||
table.withSection(SectionFactory.defaultT1("id", "table_name", "field_name", "classification"));
|
||||
table.withSection(SectionFactory.defaultT2("mask_style", "low_cardinality", "notes"));
|
||||
@@ -227,8 +179,8 @@ public final class SecuredDataTables
|
||||
.withField(field("distinct_value_alert", QFieldType.INTEGER).withDefaultValue(15))
|
||||
.withField(field("min_entropy_enabled", QFieldType.BOOLEAN).withDefaultValue(true))
|
||||
.withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true))
|
||||
.withField(createDate("create_date"))
|
||||
.withField(modifyDate("modify_date"))
|
||||
.withField(field("create_date", QFieldType.DATE_TIME))
|
||||
.withField(field("modify_date", QFieldType.DATE_TIME))
|
||||
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
|
||||
table.withSection(SectionFactory.defaultT1("id", "name", "permission_name", "classification"));
|
||||
table.withSection(SectionFactory.defaultT2("table_name", "tier", "reveal_minutes", "daily_reveal_cap", "reason_options", "allow_free_text_reason", "require_free_text", "client_channel", "exact_match_per_hour", "miss_alert_consecutive", "distinct_value_alert", "min_entropy_enabled", "is_active"));
|
||||
@@ -253,12 +205,10 @@ public final class SecuredDataTables
|
||||
.withField(field("reason_text", QFieldType.TEXT))
|
||||
.withField(field("granted_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
.withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
.withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE));
|
||||
.withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true));
|
||||
table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "table_name"));
|
||||
table.withSection(SectionFactory.defaultT2("record_id", "field_name", "reason", "reason_text", "access_log_id"));
|
||||
table.withSection(SectionFactory.defaultT3("granted_at", "expires_at", CREATE_DATE, MODIFY_DATE));
|
||||
table.withSection(SectionFactory.defaultT3("granted_at", "expires_at"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
@@ -280,12 +230,10 @@ public final class SecuredDataTables
|
||||
.withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
.withField(field("ended_at", QFieldType.DATE_TIME))
|
||||
.withField(field("ended_by", QFieldType.STRING).withMaxLength(255))
|
||||
.withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status"))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE));
|
||||
.withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status"));
|
||||
table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "reason"));
|
||||
table.withSection(SectionFactory.defaultT2("reason_text", "classification_ceiling", "table_name", "ended_by", "status"));
|
||||
table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at", CREATE_DATE, MODIFY_DATE));
|
||||
table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
@@ -313,12 +261,10 @@ public final class SecuredDataTables
|
||||
.withField(field("break_glass_id", QFieldType.INTEGER))
|
||||
.withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true))
|
||||
.withField(field("search_value_fingerprints", QFieldType.TEXT))
|
||||
.withField(field("rule_id", QFieldType.INTEGER))
|
||||
.withField(createDate(CREATE_DATE))
|
||||
.withField(modifyDate(MODIFY_DATE));
|
||||
.withField(field("rule_id", QFieldType.INTEGER));
|
||||
table.withSection(SectionFactory.defaultT1("id", "event_type", "user_id", "user_name"));
|
||||
table.withSection(SectionFactory.defaultT2("table_name", "record_id", "field_name", "record_count", "action_name", "reason", "reason_text", "client_channel", "outcome", "anomaly_flag", "break_glass_id", "rule_id", "search_value_fingerprints"));
|
||||
table.withSection(SectionFactory.defaultT3("created_at", CREATE_DATE, MODIFY_DATE));
|
||||
table.withSection(SectionFactory.defaultT3("created_at"));
|
||||
table.withSection(SectionFactory.customT2("reveals", new QIcon("visibility")).withLabel("Reveals").withWidgetName(REVEALS_WIDGET));
|
||||
return (table);
|
||||
}
|
||||
|
||||
@@ -58,86 +58,15 @@ public final class RetentionRules
|
||||
return (names == null ? List.of() : Arrays.stream(names.split(",", -1)).map(String::trim).toList());
|
||||
}
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////
|
||||
// set on a thread that changed a rule inside a transaction: until that //
|
||||
// transaction ends, the thread reads the rules through it (its own edits), //
|
||||
// not the rules every other thread shares //
|
||||
///////////////////////////////////////////////////////////////////////////////
|
||||
private static final ThreadLocal<Boolean> EDITED_IN_TRANSACTION = ThreadLocal.withInitial(() -> false);
|
||||
|
||||
/** Set while this thread reads its own edits, so that a question from inside that read does not read them again. */
|
||||
private static final ThreadLocal<Boolean> READING_OWN_EDITS = ThreadLocal.withInitial(() -> false);
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The active rules of a table that name this field. Core asks this for every
|
||||
** field of every audited write and every filter criterion of every query, so
|
||||
** the active rules are read once (SecuredDataRuntime.activeRetentionRules: per
|
||||
** change, and at most MAX_RULE_AGE old) instead of once per call. A committed
|
||||
** change is seen at once on the node that made it and on every node the
|
||||
** cluster event reaches; a transaction that edits rules reads its own edits.
|
||||
** A failed read throws: it is never "no rules". A question asked from inside
|
||||
** a read of the rules is answered from the rules held before that read (see
|
||||
** RuleRows) and never starts another.
|
||||
** Fresh rules in the current transaction; edits take effect without a stale cache.
|
||||
*******************************************************************************/
|
||||
public static List<QRecord> active(String table, String field)
|
||||
{
|
||||
SecuredDataRuntime runtime = SecuredDataRuntime.current();
|
||||
List<QRecord> rules;
|
||||
if(runtime != null && (!readsOwnEdits() || READING_OWN_EDITS.get()))
|
||||
{
|
||||
//////////////////////////////////////////////////////////////////////////
|
||||
// also the answer to a question asked from inside this thread's read //
|
||||
// of its own edits (code that runs inside that query): the rules //
|
||||
// every other thread shares, and never another read of the edits //
|
||||
//////////////////////////////////////////////////////////////////////////
|
||||
rules = runtime.activeRetentionRules();
|
||||
}
|
||||
else
|
||||
{
|
||||
READING_OWN_EDITS.set(true);
|
||||
try
|
||||
{
|
||||
rules = SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter()
|
||||
.withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true))
|
||||
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table)));
|
||||
}
|
||||
finally
|
||||
{
|
||||
READING_OWN_EDITS.remove();
|
||||
}
|
||||
}
|
||||
return (rules.stream().filter(rule -> table != null && table.equals(rule.getValueString("table_name"))).filter(rule -> fields(rule).contains(field)).toList());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A rule was written on this thread inside a transaction that is still open.
|
||||
*******************************************************************************/
|
||||
public static void editedInTransaction()
|
||||
{
|
||||
EDITED_IN_TRANSACTION.set(true);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether this thread must read the rules through its open transaction.
|
||||
*******************************************************************************/
|
||||
private static boolean readsOwnEdits()
|
||||
{
|
||||
if(!EDITED_IN_TRANSACTION.get())
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
if(QContext.getQBackendTransaction() == null)
|
||||
{
|
||||
EDITED_IN_TRANSACTION.remove();
|
||||
return (false);
|
||||
}
|
||||
return (true);
|
||||
return (SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter()
|
||||
.withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true))
|
||||
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table)))
|
||||
.stream().filter(rule -> fields(rule).contains(field)).toList());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
package com.kingsrook.qbits.secureddata.support;
|
||||
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
@@ -34,7 +32,6 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
|
||||
|
||||
@@ -103,15 +100,13 @@ public final class SystemWrites
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Query without user locks or field access recursion, for the stored fields of
|
||||
** the table only (see storedFields).
|
||||
** Query without user locks or field access recursion.
|
||||
*******************************************************************************/
|
||||
public static List<QRecord> query(String table, QQueryFilter filter)
|
||||
{
|
||||
try
|
||||
{
|
||||
return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withFieldNamesToInclude(storedFields(table))
|
||||
.withTransaction(QContext.getQBackendTransaction())).getRecords()));
|
||||
return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withTransaction(QContext.getQBackendTransaction())).getRecords()));
|
||||
}
|
||||
catch(QException e)
|
||||
{
|
||||
@@ -121,22 +116,6 @@ public final class SystemWrites
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The names of the stored fields of a table, as the projection of a system
|
||||
** read. A query that names its fields is not given virtual fields, so core
|
||||
** runs none of the value providers another QBit or the host may have added to
|
||||
** a secured-data table: this QBit reads its own columns and nothing else, and
|
||||
** a provider cannot be made to run inside a rule read. Null (every field, and
|
||||
** the error core reports) for a table the instance does not have.
|
||||
*******************************************************************************/
|
||||
public static Set<String> storedFields(String tableName)
|
||||
{
|
||||
QTableMetaData table = QContext.getQInstance().getTable(tableName);
|
||||
return (table == null ? null : new HashSet<>(table.getFields().keySet()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Insert and reject backend record errors.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -120,73 +120,4 @@ class SecuredDataDdlTest
|
||||
.isInstanceOf(java.sql.SQLException.class);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every table the producer declares has the columns create_date and
|
||||
** modify_date in a database made from the DDL. The tables that were first
|
||||
** created without them get them from the last block of the file, which is what
|
||||
** an existing database runs: nullable columns that default to the current
|
||||
** time, so an insert that names neither still succeeds, and a block that can
|
||||
** be run again and then changes nothing.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void postgresDdlGivesEveryDeclaredTableTheTwoDateColumns() throws Exception
|
||||
{
|
||||
String url = "jdbc:h2:mem:secured_data_dates;MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1";
|
||||
QInstance instance = SecuredDataTestSupport.setup();
|
||||
try(Connection connection = DriverManager.getConnection(url, "sa", ""))
|
||||
{
|
||||
SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(false));
|
||||
java.util.List<String> added = new java.util.ArrayList<>();
|
||||
for(var table : SecuredDataStandardDatesTest.declared(instance))
|
||||
{
|
||||
if(!hasColumn(connection, table.getName(), "create_date"))
|
||||
{
|
||||
assertThat(hasColumn(connection, table.getName(), "modify_date")).as(table.getName()).isFalse();
|
||||
added.add(table.getName());
|
||||
}
|
||||
}
|
||||
assertThat(added).hasSize(7);
|
||||
connection.createStatement().execute("INSERT INTO secured_data_legal_hold (table_name, reason) VALUES ('person', 'before the columns')");
|
||||
|
||||
SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(true));
|
||||
for(var table : SecuredDataStandardDatesTest.declared(instance))
|
||||
{
|
||||
for(String column : java.util.List.of("create_date", "modify_date"))
|
||||
{
|
||||
try(java.sql.ResultSet columns = connection.getMetaData().getColumns(null, null, table.getName(), column))
|
||||
{
|
||||
assertThat(columns.next()).as(table.getName() + "." + column).isTrue();
|
||||
assertThat(columns.getString("TYPE_NAME")).as(table.getName() + "." + column).isEqualTo("TIMESTAMP WITH TIME ZONE");
|
||||
assertThat(columns.getString("IS_NULLABLE")).as(table.getName() + "." + column).isEqualTo("YES");
|
||||
if(added.contains(table.getName()))
|
||||
{
|
||||
assertThat(columns.getString("COLUMN_DEF")).as(table.getName() + "." + column).isEqualTo("CURRENT_TIMESTAMP");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
connection.createStatement().execute("INSERT INTO secured_data_legal_hold (table_name, reason) VALUES ('person', 'by a release that does not know the columns')");
|
||||
var holds = SecuredDataStandardDatesTest.rows(connection, SecuredDataTables.LEGAL_HOLD, false);
|
||||
assertThat(holds).hasSize(2).allSatisfy(hold -> assertThat(hold.get("create_date")).isNotNull().isEqualTo(hold.get("modify_date")));
|
||||
SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(true));
|
||||
assertThat(SecuredDataStandardDatesTest.rows(connection, SecuredDataTables.LEGAL_HOLD, false)).isEqualTo(holds);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether a table has a column.
|
||||
*******************************************************************************/
|
||||
private static boolean hasColumn(Connection connection, String table, String column) throws java.sql.SQLException
|
||||
{
|
||||
try(java.sql.ResultSet columns = connection.getMetaData().getColumns(null, null, table, column))
|
||||
{
|
||||
return (columns.next());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,271 +0,0 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.secureddata;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
|
||||
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
|
||||
import com.kingsrook.qbits.secureddata.reveal.RevealResult;
|
||||
import com.kingsrook.qbits.secureddata.reveal.RevealService;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.Timeout;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.fail;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A reveal, and the start, end and expiry of a break-glass session, hold the
|
||||
** runtime's monitor from their first check to their last write, so that two of
|
||||
** them cannot both pass a check (the daily allowance, one active session).
|
||||
** Every other thread asks this QBit for a classification all the time: core
|
||||
** asks for every field it decides on and for every filter criterion of every
|
||||
** query, including the queries made inside this QBit's own writes.
|
||||
**
|
||||
** When that question needed the same monitor, a thread that asked it while
|
||||
** holding another of this QBit's locks could never be answered during a
|
||||
** reveal that was about to need that lock: the access-log writer holds the
|
||||
** anomaly counters' monitor while it inserts a row (the insert counts earlier
|
||||
** rows and writes an audit, both queries with criteria), and a reveal inserts
|
||||
** its own access-log row. Reveal: runtime, then counters. Writer: counters,
|
||||
** then runtime. Both waited for ever, and so did every later request, since
|
||||
** every request asks for a classification.
|
||||
**
|
||||
** The rule rows now have a lock of their own that is held only while they are
|
||||
** read. These tests run on a relational database (H2 in PostgreSQL mode, with
|
||||
** the audit tables), where an insert really writes its audit. Each parks one
|
||||
** thread inside the operation, holding the runtime's monitor, and requires
|
||||
** the access-log writer and a classification question on other threads to
|
||||
** finish meanwhile; then the parked operation finishes too. Without the fix
|
||||
** the writer does not finish and the test fails after WAIT_SECONDS.
|
||||
*******************************************************************************/
|
||||
@Timeout(180)
|
||||
class SecuredDataLockOrderTest
|
||||
{
|
||||
private static final long WAIT_SECONDS = 20;
|
||||
|
||||
private final ExecutorService threads = Executors.newCachedThreadPool(work ->
|
||||
{
|
||||
Thread thread = new Thread(work, "lock-order-test");
|
||||
thread.setDaemon(true);
|
||||
return (thread);
|
||||
});
|
||||
|
||||
private QInstance instance;
|
||||
private SecuredDataRuntime runtime;
|
||||
private QSession person;
|
||||
private QTableMetaData reading;
|
||||
private Serializable pk;
|
||||
|
||||
private final CountDownLatch parked = new CountDownLatch(1);
|
||||
private final CountDownLatch resume = new CountDownLatch(1);
|
||||
private final AtomicReference<Thread> working = new AtomicReference<>();
|
||||
private final AtomicBoolean heldTheRuntimesMonitor = new AtomicBoolean();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Let a parked thread go, then close; a runtime whose threads are stuck was
|
||||
** already taken out of the list (see abandon).
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
resume.countDown();
|
||||
threads.shutdownNow();
|
||||
RetentionTestSupport.close();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A reveal in progress holds up neither the access-log writer nor a question.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theAccessLogWriterAndAClassificationQuestionFinishWhileARevealIsInProgress() throws Exception
|
||||
{
|
||||
setup();
|
||||
Future<RevealResult> reveal = parkInside(() -> new RevealService().reveal("reading", pk, "notes", null, "Synthetic reveal; lock order test."));
|
||||
|
||||
othersFinishMeanwhile();
|
||||
|
||||
resume.countDown();
|
||||
assertThat(reveal.get(WAIT_SECONDS, TimeUnit.SECONDS).value()).isEqualTo("test-notes");
|
||||
assertThat(logs()).extracting(row -> row.getValueString("event_type")).contains("EXACT_MATCH_FILTER", "REVEAL");
|
||||
assertThat(runtime.resolver().isRevealActive("reading", "notes", pk)).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The same for the start of a break-glass session, which holds the same monitor.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theAccessLogWriterAndAClassificationQuestionFinishWhileBreakGlassStarts() throws Exception
|
||||
{
|
||||
setup();
|
||||
person.getPermissions().add(BreakGlassService.START + ".hasAccess");
|
||||
Future<QRecord> started = parkInside(() -> new BreakGlassService().start("Synthetic break-glass session; lock order test.", 5, "reading", DataClassification.PHI));
|
||||
|
||||
othersFinishMeanwhile();
|
||||
|
||||
resume.countDown();
|
||||
assertThat(started.get(WAIT_SECONDS, TimeUnit.SECONDS).getValueString("status")).isEqualTo("ACTIVE");
|
||||
assertThat(logs()).extracting(row -> row.getValueString("event_type")).contains("EXACT_MATCH_FILTER", "BREAK_GLASS_START");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A relational instance with audit tables; a PII field one policy makes
|
||||
** revealable to the person; one row; and one search of that field waiting in
|
||||
** the access-log queue, as it waits after a person filters a list by it.
|
||||
*******************************************************************************/
|
||||
private void setup() throws Exception
|
||||
{
|
||||
instance = RetentionTestSupport.setup();
|
||||
runtime = SecuredDataRuntime.current();
|
||||
person = QContext.getQSession();
|
||||
reading = instance.getTable("reading");
|
||||
insert(SecuredDataTables.POLICY, policyRow("secured_data.admin", "PII", "REVEALABLE"));
|
||||
pk = RetentionTestSupport.reading(120).getValue("id");
|
||||
|
||||
runtime.resolver().onAccess(new FieldAccessEvent().withType(FieldAccessEventType.EXACT_MATCH_FILTER).withOutcome(FieldAccessOutcome.MATCH)
|
||||
.withTableName("reading").withFieldNames(List.of("notes")).withActionName("query").withRecordCount(1)
|
||||
.withSearchValueFingerprints(List.of(runtime.resolver().fingerprintSearchValue("known"))));
|
||||
assertThat(logs()).as("the search is queued, not yet written").extracting(row -> row.getValueString("event_type")).doesNotContain("EXACT_MATCH_FILTER");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Run the operation as the person on another thread and return once it is
|
||||
** parked at its first reading of the clock, which both operations make
|
||||
** inside their synchronized block.
|
||||
*******************************************************************************/
|
||||
private <T> Future<T> parkInside(Callable<T> operation) throws Exception
|
||||
{
|
||||
CLOCK.beforeInstant = () ->
|
||||
{
|
||||
if(Thread.currentThread() == working.get() && parked.getCount() == 1)
|
||||
{
|
||||
heldTheRuntimesMonitor.set(Thread.holdsLock(runtime));
|
||||
parked.countDown();
|
||||
try
|
||||
{
|
||||
resume.await();
|
||||
}
|
||||
catch(InterruptedException e)
|
||||
{
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
};
|
||||
Future<T> future = threads.submit(() ->
|
||||
{
|
||||
QContext.init(instance, person);
|
||||
try
|
||||
{
|
||||
working.set(Thread.currentThread());
|
||||
return (operation.call());
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
});
|
||||
assertThat(parked.await(WAIT_SECONDS, TimeUnit.SECONDS)).as("the operation read the clock").isTrue();
|
||||
assertThat(heldTheRuntimesMonitor).as("the operation is parked holding the runtime's monitor").isTrue();
|
||||
return (future);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** While the operation is parked: the access-log writer drains its queue (what
|
||||
** its worker does on every tick), and a thread that holds no lock is told
|
||||
** the classification of a field and the active retention rules.
|
||||
*******************************************************************************/
|
||||
private void othersFinishMeanwhile() throws Exception
|
||||
{
|
||||
Future<?> writer = threads.submit(() -> runtime.logger().flush());
|
||||
Future<DataClassification> question = threads.submit(() ->
|
||||
{
|
||||
QContext.init(instance, person);
|
||||
try
|
||||
{
|
||||
runtime.activeRetentionRules();
|
||||
return (SecuredDataClassifications.effective(reading, "notes"));
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
});
|
||||
|
||||
try
|
||||
{
|
||||
writer.get(WAIT_SECONDS, TimeUnit.SECONDS);
|
||||
assertThat(question.get(WAIT_SECONDS, TimeUnit.SECONDS)).isEqualTo(DataClassification.PII);
|
||||
}
|
||||
catch(TimeoutException e)
|
||||
{
|
||||
abandon();
|
||||
fail("A thread waited " + WAIT_SECONDS + " seconds for the monitor the parked operation holds (writer done: " + writer.isDone() + ", question done: " + question.isDone()
|
||||
+ "): the operation needs the lock that thread holds next, so neither would ever finish");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Threads of this runtime are stuck holding its locks: closing it would wait
|
||||
** for them, so it is left to the daemon threads instead of being closed.
|
||||
*******************************************************************************/
|
||||
private void abandon()
|
||||
{
|
||||
SecuredDataTestSupport.RUNTIMES.remove(runtime);
|
||||
}
|
||||
}
|
||||
@@ -95,9 +95,7 @@ class SecuredDataQBitProducerTest
|
||||
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(name.equals(SecuredDataTables.AUDIT_KEY) ? AuditLevel.NONE : AuditLevel.FIELD);
|
||||
assertThat(table.getSections()).isNotEmpty();
|
||||
assertThat(SecuredDataMetadata.classification(table, null)).isEqualTo(DataClassification.CONFIDENTIAL);
|
||||
// A column has the name of its field. The two standard dates are always the columns create_date and modify_date.
|
||||
table.getFields().values().forEach(field -> assertThat(field.getBackendName())
|
||||
.isEqualTo("createDate".equals(field.getName()) ? "create_date" : "modifyDate".equals(field.getName()) ? "modify_date" : field.getName()));
|
||||
table.getFields().values().forEach(field -> assertThat(field.getBackendName()).isEqualTo(field.getName()));
|
||||
}
|
||||
assertThat(SecuredDataTables.esbTableNames()).containsExactlyElementsOf(SecuredDataTables.MAIN);
|
||||
assertThat(SecuredDataEnums.sources()).hasSize(6);
|
||||
|
||||
@@ -1,261 +0,0 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.secureddata;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
|
||||
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
|
||||
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
|
||||
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
|
||||
import com.kingsrook.qqq.backend.core.cluster.ClusterBus;
|
||||
import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;
|
||||
import com.kingsrook.qqq.backend.core.cluster.ClusterEvent;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.delete;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The rule rows a node keeps (retention rules, policies, classifications) and
|
||||
** the access fingerprint of a request: read once, cleared by every change on
|
||||
** every node, never older than the maximum age, and never an empty answer when
|
||||
** the rules cannot be read.
|
||||
*******************************************************************************/
|
||||
class SecuredDataRuleCachesTest
|
||||
{
|
||||
private static final List<String> ANNOUNCED = new ArrayList<>();
|
||||
|
||||
static
|
||||
{
|
||||
ClusterBus.on(ClusterEvent.TYPE_MEMO_INVALIDATE, (instance, event) -> ANNOUNCED.add(event.getName()));
|
||||
}
|
||||
|
||||
QInstance instance;
|
||||
QTableMetaData person;
|
||||
SecuredDataRuntime runtime;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A person table with a date, so a retention rule on it is valid.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void init() throws Exception
|
||||
{
|
||||
instance = setup();
|
||||
person = instance.getTable("person");
|
||||
person.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("createDate"));
|
||||
runtime = SecuredDataRuntime.current();
|
||||
MemoryRecordStore.setCollectStatistics(true);
|
||||
MemoryRecordStore.resetStatistics();
|
||||
ANNOUNCED.clear();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Cleanup.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
MemoryRecordStore.setCollectStatistics(false);
|
||||
close();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An active rule on one field of the person table.
|
||||
*******************************************************************************/
|
||||
private static QRecord rule(String field) throws Exception
|
||||
{
|
||||
return (insert(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("name", "30 days: " + field).withValue("table_name", "person")
|
||||
.withValue("field_names", field).withValue("age_field_name", "createDate").withValue("age_days", 30).withValue("action", "NULL_WITH_MARKER")));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Queries the in-memory store has run since the last reset.
|
||||
*******************************************************************************/
|
||||
private static int queries()
|
||||
{
|
||||
return (MemoryRecordStore.getStatistics().getOrDefault(MemoryRecordStore.STAT_QUERIES_RAN, 0));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Core asks for the rules of a field on every audited field and filter
|
||||
** criterion: many questions are one read, and every committed change (insert,
|
||||
** update, delete) is seen by the next question.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void activeRetentionRulesAreReadOncePerChange() throws Exception
|
||||
{
|
||||
QRecord notes = rule("notes");
|
||||
assertThat(notes.getErrors()).isNullOrEmpty();
|
||||
MemoryRecordStore.resetStatistics();
|
||||
for(int i = 0; i < 50; i++)
|
||||
{
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
assertThat(RetentionRules.active("person", "email")).isEmpty();
|
||||
assertThat(RetentionRules.active("anotherTable", "notes")).isEmpty();
|
||||
assertThat(RetentionRules.changedOnly("person", "notes")).isFalse();
|
||||
}
|
||||
assertThat(queries()).isLessThanOrEqualTo(1);
|
||||
|
||||
QRecord ssn = rule("ssn");
|
||||
assertThat(RetentionRules.active("person", "ssn")).hasSize(1);
|
||||
|
||||
update(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("id", notes.getValue("id")).withValue("is_active", false));
|
||||
assertThat(RetentionRules.active("person", "notes")).isEmpty();
|
||||
|
||||
delete(SecuredDataTables.RETENTION_RULE, ssn);
|
||||
assertThat(RetentionRules.active("person", "ssn")).isEmpty();
|
||||
assertThat(ANNOUNCED).contains(SecuredDataRuntime.RETENTION_CACHE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A rule change made on another node arrives as a cluster cache event: each
|
||||
** of the three kinds of rules is read again here. A change made here is
|
||||
** announced to the other nodes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aChangeOnAnyNodeClearsTheRulesOnEveryNode() throws Exception
|
||||
{
|
||||
rule("notes");
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
|
||||
long retention = runtime.retentionVersion();
|
||||
long policies = runtime.policyVersion();
|
||||
long classifications = runtime.classificationVersion();
|
||||
ClusterCaches.invalidate(SecuredDataRuntime.RETENTION_CACHE);
|
||||
ClusterCaches.invalidate(SecuredDataRuntime.POLICY_CACHE);
|
||||
ClusterCaches.invalidate(SecuredDataRuntime.CLASSIFICATION_CACHE);
|
||||
assertThat(runtime.retentionVersion()).isGreaterThan(retention);
|
||||
assertThat(runtime.policyVersion()).isGreaterThan(policies);
|
||||
assertThat(runtime.classificationVersion()).isGreaterThan(classifications);
|
||||
|
||||
MemoryRecordStore.resetStatistics();
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
assertThat(queries()).isGreaterThanOrEqualTo(1);
|
||||
MemoryRecordStore.resetStatistics();
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
assertThat(queries()).isZero();
|
||||
|
||||
ANNOUNCED.clear();
|
||||
runtime.invalidatePolicies();
|
||||
runtime.invalidateClassifications();
|
||||
runtime.invalidateRetentionRules();
|
||||
assertThat(ANNOUNCED).contains(SecuredDataRuntime.POLICY_CACHE, SecuredDataRuntime.CLASSIFICATION_CACHE, SecuredDataRuntime.RETENTION_CACHE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fail closed: when the rules cannot be read, asking for them throws. The
|
||||
** rules read before are not used, and the answer is never an empty list.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void rulesThatCannotBeReadAreNeverAnEmptyAnswer() throws Exception
|
||||
{
|
||||
rule("notes");
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
|
||||
QTableMetaData ruleTable = instance.getTables().remove(SecuredDataTables.RETENTION_RULE);
|
||||
try
|
||||
{
|
||||
runtime.invalidateRetentionRules();
|
||||
assertThatThrownBy(() -> RetentionRules.active("person", "notes")).isInstanceOf(RuntimeException.class);
|
||||
assertThatThrownBy(() -> RetentionRules.active("person", "notes")).isInstanceOf(RuntimeException.class);
|
||||
}
|
||||
finally
|
||||
{
|
||||
instance.getTables().put(SecuredDataTables.RETENTION_RULE, ruleTable);
|
||||
}
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The safety net for a lost cluster event: rows are read again once they are
|
||||
** older than the maximum age.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void ruleRowsAreNotUsedPastTheirMaximumAge()
|
||||
{
|
||||
assertThat(SecuredDataRuntime.tooOld(System.nanoTime())).isFalse();
|
||||
assertThat(SecuredDataRuntime.tooOld(System.nanoTime() - SecuredDataRuntime.MAX_RULE_AGE.plusSeconds(1).toNanos())).isTrue();
|
||||
assertThat(SecuredDataRuntime.MAX_RULE_AGE.toSeconds()).isLessThanOrEqualTo(60);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Core asks for the access fingerprint before every field decision. Within a
|
||||
** request the same value object is returned; a permission that matters, given
|
||||
** to the same session object, changes it at once.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theAccessFingerprintIsKeptForTheRequestAndTracksThePermissions()
|
||||
{
|
||||
SecuredDataResolver resolver = runtime.resolver();
|
||||
QSession session = QContext.getQSession();
|
||||
String first = resolver.getAccessFingerprint();
|
||||
for(int i = 0; i < 20; i++)
|
||||
{
|
||||
assertThat(resolver.getAccessFingerprint()).isSameAs(first);
|
||||
}
|
||||
|
||||
session.getPermissions().add("unrelated.permission");
|
||||
assertThat(resolver.getAccessFingerprint()).isEqualTo(first);
|
||||
|
||||
session.getPermissions().add(SecuredDataMetadata.permissionBase(person) + "." + SecuredDataPermissions.FULL);
|
||||
String granted = resolver.getAccessFingerprint();
|
||||
assertThat(granted).isNotEqualTo(first);
|
||||
|
||||
session.setPermissions(new HashSet<>(List.of("person.read")));
|
||||
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(granted);
|
||||
}
|
||||
}
|
||||
@@ -1,682 +0,0 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.secureddata;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.Callable;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
|
||||
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
|
||||
import com.kingsrook.qbits.secureddata.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessDecision;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.query.VirtualFieldValueProviderInterface;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reading rule rows (classifications, retention rules, policies) is a query on
|
||||
** a secured-data table, and other code runs inside that query: a virtual field
|
||||
** value provider or a post-query customizer that another QBit or the host put
|
||||
** on the table. When that code runs a query with filter criteria, core asks
|
||||
** this QBit for a classification - which needs the rows being read.
|
||||
**
|
||||
** These tests put such code on the rule tables and prove: the read asks for
|
||||
** stored fields only, so no value provider runs inside it; a question asked
|
||||
** inside a read is answered from the rows held before and starts no other
|
||||
** read; only the reading thread, as the system, can be given that answer; and
|
||||
** a read that fails is tried again by the next question.
|
||||
*******************************************************************************/
|
||||
class SecuredDataRuleReloadTest
|
||||
{
|
||||
QInstance instance;
|
||||
QTableMetaData person;
|
||||
SecuredDataRuntime runtime;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Something to do inside a query on a rule table.
|
||||
*******************************************************************************/
|
||||
@FunctionalInterface
|
||||
interface NestedWork
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Do it.
|
||||
*******************************************************************************/
|
||||
void run() throws Exception;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stands in for another QBit's virtual field on a secured-data table: filling
|
||||
** it runs a query that has filter criteria.
|
||||
*******************************************************************************/
|
||||
public static class NestedQueryProvider implements VirtualFieldValueProviderInterface
|
||||
{
|
||||
static final AtomicInteger RUNS = new AtomicInteger();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fill the field from a query with criteria, as a real provider does.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void fill(QTableMetaData table, String fieldName, List<QRecord> records, QueryInput input) throws QException
|
||||
{
|
||||
RUNS.incrementAndGet();
|
||||
queryWithCriteria();
|
||||
records.forEach(record -> record.setValue(fieldName, "filled"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A post-query customizer on a secured-data table. Core runs it for every
|
||||
** query on the table, whatever fields the query names.
|
||||
*******************************************************************************/
|
||||
public static class NestedWorkCustomizer implements TableCustomizerInterface
|
||||
{
|
||||
static final AtomicInteger RUNS = new AtomicInteger();
|
||||
static volatile NestedWork work;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Run the work the test asked for, inside the query.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> postQuery(QueryOrGetInputInterface queryInput, List<QRecord> records) throws QException
|
||||
{
|
||||
NestedWork toRun = work;
|
||||
if(toRun != null)
|
||||
{
|
||||
RUNS.incrementAndGet();
|
||||
try
|
||||
{
|
||||
toRun.run();
|
||||
}
|
||||
catch(QException | RuntimeException e)
|
||||
{
|
||||
throw (e);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new QException("Nested work failed", e));
|
||||
}
|
||||
}
|
||||
return (records);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A person table with a date, so a retention rule on it is valid.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void init() throws Exception
|
||||
{
|
||||
instance = setup();
|
||||
person = instance.getTable("person");
|
||||
person.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("createDate"));
|
||||
runtime = SecuredDataRuntime.current();
|
||||
NestedQueryProvider.RUNS.set(0);
|
||||
NestedWorkCustomizer.RUNS.set(0);
|
||||
NestedWorkCustomizer.work = null;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Cleanup.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
NestedWorkCustomizer.work = null;
|
||||
close();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A query with filter criteria on a host table: core validates the operand
|
||||
** of a classified field, so it asks for the field's classification - and
|
||||
** this QBit then asks whether a retention rule names the field.
|
||||
*******************************************************************************/
|
||||
private static void queryWithCriteria() throws QException
|
||||
{
|
||||
new QueryAction().execute(new QueryInput("person").withFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.EQUALS, 1))));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Another QBit's virtual field on a secured-data table.
|
||||
*******************************************************************************/
|
||||
private void addProvider(String table)
|
||||
{
|
||||
instance.getTable(table).addVirtualField(new QVirtualFieldMetaData("labels", QFieldType.STRING).withIsQuerySelectable(true)
|
||||
.withValueProvider(new QCodeReference(NestedQueryProvider.class)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A post-query customizer on a secured-data table.
|
||||
*******************************************************************************/
|
||||
private void addCustomizer(String table)
|
||||
{
|
||||
instance.getTable(table).withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(NestedWorkCustomizer.class));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A row as another node writes it: straight into the store. None of this
|
||||
** node's customizers run and nothing here reads the rule rows; this node
|
||||
** learns of the change when its version moves (the cluster event).
|
||||
*******************************************************************************/
|
||||
private static QRecord writtenElsewhere(String table, QRecord row) throws QException
|
||||
{
|
||||
return (MemoryRecordStore.getInstance().insert(new InsertInput(table).withRecord(row), true).getFirst());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A classification of a person field, written by another node.
|
||||
*******************************************************************************/
|
||||
private QRecord classifiedElsewhere(String field, String classification) throws QException
|
||||
{
|
||||
QRecord row = writtenElsewhere(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "person")
|
||||
.withValue("field_name", field).withValue("classification", classification));
|
||||
runtime.invalidateClassifications();
|
||||
return (row);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A classification row changed by another node.
|
||||
*******************************************************************************/
|
||||
private void reclassifiedElsewhere(QRecord row, String classification) throws QException
|
||||
{
|
||||
MemoryRecordStore.getInstance().update(new UpdateInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord()
|
||||
.withValue("id", row.getValue("id")).withValue("classification", classification)), false);
|
||||
runtime.invalidateClassifications();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An active retention rule on one field of the person table, written by
|
||||
** another node.
|
||||
*******************************************************************************/
|
||||
private void ruleWrittenElsewhere(String field) throws QException
|
||||
{
|
||||
writtenElsewhere(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("name", "30 days: " + field).withValue("table_name", "person")
|
||||
.withValue("field_names", field).withValue("age_field_name", "createDate").withValue("age_days", 30).withValue("action", "NULL_WITH_MARKER")
|
||||
.withValue("audit_treatment", "SHREDDABLE").withValue("is_active", true));
|
||||
runtime.invalidateRetentionRules();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The classification of a person field.
|
||||
*******************************************************************************/
|
||||
private DataClassification classificationOf(String field)
|
||||
{
|
||||
return (SecuredDataClassifications.effective(person, field));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The reported failure: a value provider on the classification table whose
|
||||
** work is a query with criteria, and one classification row. Before the fix
|
||||
** the next classification question recursed until the stack overflowed, and
|
||||
** so did every question after it. Now the rule read names its stored fields,
|
||||
** so the provider does not run inside it at all.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aValueProviderOnTheClassificationTableDoesNotRunInsideTheRuleRead() throws Exception
|
||||
{
|
||||
addProvider(SecuredDataTables.CLASSIFICATION);
|
||||
classification("notes", "PHI");
|
||||
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
|
||||
queryWithCriteria();
|
||||
assertThat(NestedQueryProvider.RUNS).hasValue(0);
|
||||
assertThat(SystemWrites.query(SecuredDataTables.CLASSIFICATION, new QQueryFilter())).singleElement()
|
||||
.satisfies(row -> assertThat(row.getValues()).doesNotContainKey("labels").containsEntry("field_name", "notes"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The virtual field is still there for a query that asks for every field (a
|
||||
** screen listing the classifications). Its provider's query with criteria
|
||||
** asks for a classification while the rows are out of date: the rows are
|
||||
** read once, inside the provider, and nothing recurses.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void anOrdinaryQueryOnTheClassificationTableStillFillsTheVirtualField() throws Exception
|
||||
{
|
||||
addProvider(SecuredDataTables.CLASSIFICATION);
|
||||
classifiedElsewhere("notes", "PHI");
|
||||
|
||||
List<QRecord> rows = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.CLASSIFICATION)).getRecords());
|
||||
assertThat(rows).singleElement().satisfies(row -> assertThat(row.getValueString("labels")).isEqualTo("filled"));
|
||||
assertThat(NestedQueryProvider.RUNS).hasValue(1);
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A post-query customizer runs inside the rule read whatever fields the read
|
||||
** names. Its query with criteria asks for a classification in the middle of
|
||||
** the read of the classifications: before the fix that started the read again
|
||||
** and recursed until the stack overflowed. Now the rows are read once.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aQuestionAskedInsideTheClassificationReadDoesNotStartAnotherRead() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.CLASSIFICATION);
|
||||
classifiedElsewhere("notes", "PHI");
|
||||
|
||||
NestedWorkCustomizer.work = SecuredDataRuleReloadTest::queryWithCriteria;
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
|
||||
|
||||
assertThat(classificationOf("email")).isEqualTo(DataClassification.PII);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** What a question asked inside the read is told, and who can ask it.
|
||||
**
|
||||
** The nested question gets the rows this node held when the read began (no
|
||||
** rows, the first time; then the rows of the previous version), in both
|
||||
** directions of a change. It can only be asked by the thread doing the read,
|
||||
** which is inside the QBit's system read: the session is the system session
|
||||
** and the resolver answers FULL there without looking at any classification,
|
||||
** so no person's field access is decided from that answer. The question that
|
||||
** started the read gets the new rows.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aQuestionAskedInsideTheReadIsAnsweredFromTheRowsHeldBefore() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.CLASSIFICATION);
|
||||
assertThat(classificationOf("notes")).isNull();
|
||||
|
||||
List<DataClassification> nestedAnswers = new ArrayList<>();
|
||||
List<Boolean> nestedAsSystem = new ArrayList<>();
|
||||
List<FieldAccessDecision> nestedDecisions = new ArrayList<>();
|
||||
NestedWork ask = () ->
|
||||
{
|
||||
nestedAnswers.add(classificationOf("notes"));
|
||||
nestedAsSystem.add(QContext.getQSession() instanceof QSystemUserSession && SystemWrites.active());
|
||||
nestedDecisions.add(runtime.resolver().resolve(person, person.getField("notes")));
|
||||
};
|
||||
|
||||
/////////////////////////////////////////////////////////////////////////
|
||||
// a row raises the field: inside the read it is still unclassified, //
|
||||
// and the question that started the read is answered PHI //
|
||||
/////////////////////////////////////////////////////////////////////////
|
||||
QRecord row = classifiedElsewhere("notes", "PHI");
|
||||
NestedWorkCustomizer.work = ask;
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
|
||||
NestedWorkCustomizer.work = null;
|
||||
assertThat(nestedAnswers).containsExactly((DataClassification) null);
|
||||
|
||||
/////////////////////////////////////////////////////////////////////////
|
||||
// the row is changed to lower the field: inside the read it is still //
|
||||
// PHI, and the question that started the read is answered PUBLIC //
|
||||
/////////////////////////////////////////////////////////////////////////
|
||||
reclassifiedElsewhere(row, "PUBLIC");
|
||||
NestedWorkCustomizer.work = ask;
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PUBLIC);
|
||||
NestedWorkCustomizer.work = null;
|
||||
assertThat(nestedAnswers).containsExactly(null, DataClassification.PHI);
|
||||
|
||||
assertThat(nestedAsSystem).containsExactly(true, true);
|
||||
assertThat(nestedDecisions).containsExactly(FieldAccessDecision.FULL, FieldAccessDecision.FULL);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(2);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A request on another thread is never given the rows held before: while the
|
||||
** read is running it waits, and then it is answered from the rows the read
|
||||
** returned.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aRequestOnAnotherThreadWaitsForTheReadAndIsGivenItsRows() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.CLASSIFICATION);
|
||||
QRecord row = classifiedElsewhere("notes", "PUBLIC");
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PUBLIC);
|
||||
reclassifiedElsewhere(row, "PHI");
|
||||
|
||||
CountDownLatch readIsRunning = new CountDownLatch(1);
|
||||
CountDownLatch letReadFinish = new CountDownLatch(1);
|
||||
NestedWorkCustomizer.work = () ->
|
||||
{
|
||||
readIsRunning.countDown();
|
||||
assertThat(letReadFinish.await(30, TimeUnit.SECONDS)).isTrue();
|
||||
};
|
||||
|
||||
ExecutorService pool = Executors.newFixedThreadPool(2);
|
||||
try
|
||||
{
|
||||
Future<DataClassification> reader = pool.submit(asUser("u2", () -> classificationOf("notes")));
|
||||
assertThat(readIsRunning.await(30, TimeUnit.SECONDS)).isTrue();
|
||||
|
||||
Future<DataClassification> other = pool.submit(asUser("u3", () -> classificationOf("notes")));
|
||||
assertThatThrownBy(() -> other.get(500, TimeUnit.MILLISECONDS)).isInstanceOf(TimeoutException.class);
|
||||
|
||||
letReadFinish.countDown();
|
||||
assertThat(reader.get(30, TimeUnit.SECONDS)).isEqualTo(DataClassification.PHI);
|
||||
assertThat(other.get(30, TimeUnit.SECONDS)).isEqualTo(DataClassification.PHI);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
letReadFinish.countDown();
|
||||
pool.shutdownNow();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A person's request on its own thread.
|
||||
*******************************************************************************/
|
||||
private <T> Callable<T> asUser(String id, Callable<T> work)
|
||||
{
|
||||
return (() ->
|
||||
{
|
||||
QContext.init(instance, user(id));
|
||||
try
|
||||
{
|
||||
return (work.call());
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A read that fails (here, in code running inside it, after that code asked a
|
||||
** nested question) throws to the request that started it and leaves nothing
|
||||
** behind: the next request reads again and gets the current rows - not the
|
||||
** rows held before, and not another failure. An Error, such as the stack
|
||||
** overflow this defect used to end in, is no different.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aClassificationReadThatFailsIsTriedAgainByTheNextQuestion() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.CLASSIFICATION);
|
||||
QRecord row = classifiedElsewhere("notes", "PUBLIC");
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PUBLIC);
|
||||
reclassifiedElsewhere(row, "PHI");
|
||||
|
||||
NestedWorkCustomizer.work = () ->
|
||||
{
|
||||
classificationOf("notes");
|
||||
throw (new QException("The store is unavailable"));
|
||||
};
|
||||
assertThatThrownBy(() -> classificationOf("notes")).isInstanceOf(IllegalStateException.class);
|
||||
|
||||
NestedWorkCustomizer.work = () ->
|
||||
{
|
||||
classificationOf("notes");
|
||||
throw (new StackOverflowError());
|
||||
};
|
||||
assertThatThrownBy(() -> classificationOf("notes")).isInstanceOf(StackOverflowError.class);
|
||||
|
||||
NestedWorkCustomizer.work = null;
|
||||
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The active retention rules have the same shape: core's classification
|
||||
** question for a host field also asks whether a retention rule names the
|
||||
** field. A value provider on the retention rule table, with one active rule,
|
||||
** recursed the same way before the fix; it does not run inside the rule read.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aValueProviderOnTheRetentionRuleTableDoesNotRunInsideTheRuleRead() throws Exception
|
||||
{
|
||||
addProvider(SecuredDataTables.RETENTION_RULE);
|
||||
ruleWrittenElsewhere("notes");
|
||||
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
queryWithCriteria();
|
||||
assertThat(NestedQueryProvider.RUNS).hasValue(0);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A question asked inside the read of the retention rules starts no other
|
||||
** read and is answered from the rules held before; the question that started
|
||||
** the read gets the new rules. A read that fails is tried again.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aQuestionAskedInsideTheRetentionRuleReadIsAnsweredFromTheRulesHeldBefore() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.RETENTION_RULE);
|
||||
assertThat(RetentionRules.active("person", "notes")).isEmpty();
|
||||
ruleWrittenElsewhere("notes");
|
||||
|
||||
List<Integer> nestedAnswers = new ArrayList<>();
|
||||
NestedWorkCustomizer.work = () ->
|
||||
{
|
||||
queryWithCriteria();
|
||||
nestedAnswers.add(RetentionRules.active("person", "notes").size());
|
||||
if(nestedAnswers.size() == 1)
|
||||
{
|
||||
throw (new QException("The store is unavailable"));
|
||||
}
|
||||
};
|
||||
assertThatThrownBy(() -> RetentionRules.active("person", "notes")).isInstanceOf(IllegalStateException.class);
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
assertThat(nestedAnswers).containsExactly(0, 0);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(2);
|
||||
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(2);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A transaction that edited a rule reads the rules through itself, not from
|
||||
** the rules the node keeps. A question asked inside that read is answered
|
||||
** from the rules every thread shares: before the fix it read the edits again,
|
||||
** and recursed until the stack overflowed.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aQuestionAskedInsideAReadOfATransactionsOwnEditsDoesNotReadThemAgain() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.RETENTION_RULE);
|
||||
ruleWrittenElsewhere("notes");
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
|
||||
QContext.setTransaction(new QBackendTransaction());
|
||||
RetentionRules.editedInTransaction();
|
||||
try
|
||||
{
|
||||
NestedWorkCustomizer.work = SecuredDataRuleReloadTest::queryWithCriteria;
|
||||
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
////////////////////////////////////////////////////////////////////////
|
||||
// the next question outside a transaction ends this thread's edits //
|
||||
////////////////////////////////////////////////////////////////////////
|
||||
NestedWorkCustomizer.work = null;
|
||||
QContext.setTransaction(null);
|
||||
RetentionRules.active("person", "notes");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The policies are read the same way (rows first, version afterwards), so
|
||||
** they are guarded the same way: code inside the policy read that asks the
|
||||
** resolver a policy question starts no other read, and a read that fails is
|
||||
** tried again.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aQuestionAskedInsideThePolicyReadDoesNotStartAnotherRead() throws Exception
|
||||
{
|
||||
addCustomizer(SecuredDataTables.POLICY);
|
||||
addProvider(SecuredDataTables.POLICY);
|
||||
assertThat(runtime.resolver().revealPolicies(person, "email")).isEmpty();
|
||||
writtenElsewhere(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE"));
|
||||
runtime.invalidatePolicies();
|
||||
|
||||
List<Integer> nestedAnswers = new ArrayList<>();
|
||||
NestedWorkCustomizer.work = () ->
|
||||
{
|
||||
nestedAnswers.add(runtime.resolver().revealPolicies(person, "email").size());
|
||||
if(nestedAnswers.size() == 1)
|
||||
{
|
||||
throw (new QException("The store is unavailable"));
|
||||
}
|
||||
};
|
||||
assertThatThrownBy(() -> runtime.resolver().revealPolicies(person, "email")).isInstanceOf(IllegalStateException.class);
|
||||
assertThat(runtime.resolver().revealPolicies(person, "email")).hasSize(1);
|
||||
assertThat(nestedAnswers).containsExactly(0, 0);
|
||||
assertThat(NestedQueryProvider.RUNS).hasValue(0);
|
||||
|
||||
NestedWorkCustomizer.RUNS.set(0);
|
||||
assertThat(runtime.resolver().revealPolicies(person, "email")).hasSize(1);
|
||||
assertThat(NestedWorkCustomizer.RUNS).hasValue(0);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The rows themselves, without a table: nothing is held before the first
|
||||
** read, a nested question never reads, a failed read is forgotten, and a new
|
||||
** version is read once.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void ruleRowsAreReadOncePerVersionAndNeverFromInsideTheirOwnRead()
|
||||
{
|
||||
RuleRows rows = new RuleRows();
|
||||
AtomicInteger reads = new AtomicInteger();
|
||||
List<QRecord> first = List.of(new QRecord().withValue("id", 1));
|
||||
List<QRecord> second = List.of(new QRecord().withValue("id", 2));
|
||||
|
||||
assertThatThrownBy(() -> rows.get(1, () ->
|
||||
{
|
||||
reads.incrementAndGet();
|
||||
assertThat(rows.get(1, () -> fail("a nested question must not read"))).isEmpty();
|
||||
throw (new IllegalStateException("The store is unavailable"));
|
||||
})).isInstanceOf(IllegalStateException.class);
|
||||
|
||||
assertThat(rows.get(1, () ->
|
||||
{
|
||||
reads.incrementAndGet();
|
||||
assertThat(rows.get(1, () -> fail("a nested question must not read"))).isEmpty();
|
||||
return (first);
|
||||
})).isEqualTo(first);
|
||||
assertThat(rows.get(1, () -> fail("rows of this version are held"))).isEqualTo(first);
|
||||
|
||||
assertThat(rows.get(2, () ->
|
||||
{
|
||||
reads.incrementAndGet();
|
||||
assertThat(rows.get(2, () -> fail("a nested question must not read"))).isEqualTo(first);
|
||||
return (second);
|
||||
})).isEqualTo(second);
|
||||
assertThat(reads).hasValue(3);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A read that must not happen.
|
||||
*******************************************************************************/
|
||||
private static List<QRecord> fail(String why)
|
||||
{
|
||||
throw (new AssertionError(why));
|
||||
}
|
||||
}
|
||||
@@ -1,774 +0,0 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.secureddata;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.Serializable;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.sql.Connection;
|
||||
import java.sql.DriverManager;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.ResultSetMetaData;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Timestamp;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.Base64;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
|
||||
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
|
||||
import com.kingsrook.qbits.secureddata.retention.WipeService;
|
||||
import com.kingsrook.qbits.secureddata.reveal.RevealService;
|
||||
import com.kingsrook.qbits.secureddata.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
|
||||
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every table has a create date and a modify date: what the metadata says of
|
||||
** them, who may read them, every writer of this QBit setting them, what the
|
||||
** access-log writer runs for a row, the records of a changed policy, and rows
|
||||
** that were written before the two columns existed.
|
||||
*******************************************************************************/
|
||||
class SecuredDataStandardDatesTest
|
||||
{
|
||||
static final String DDL = "docs/ddl/secured-data-postgres.sql";
|
||||
|
||||
/** The first line of the block of the DDL that an existing database runs to get the two columns. */
|
||||
static final String DDL_BLOCK = "-- Standard dates:";
|
||||
|
||||
/** The tables that had the two dates before the other tables did: they keep the field names they have always had. */
|
||||
private static final Set<String> SNAKE_CASE_NAMES = Set.of("secured_data_classification", "secured_data_policy", "secured_data_retention_rule");
|
||||
|
||||
private static final Instant LATER = NOW.plusSeconds(90);
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Cleanup.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
RetentionTestSupport.close();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The tables the producer put in an instance, whichever class declares them,
|
||||
** so that a table added later is covered without being named here.
|
||||
*******************************************************************************/
|
||||
static List<QTableMetaData> declared(QInstance instance)
|
||||
{
|
||||
String qBit = instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getName();
|
||||
List<QTableMetaData> tables = instance.getTables().values().stream().filter(table -> qBit.equals(table.getSourceQBitName())).toList();
|
||||
assertThat(tables).extracting(QTableMetaData::getName).containsAll(SecuredDataTables.ALL);
|
||||
return (tables);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Create date field name.
|
||||
*******************************************************************************/
|
||||
private static String createDate(String table)
|
||||
{
|
||||
return (SNAKE_CASE_NAMES.contains(table) ? "create_date" : "createDate");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Modify date field name.
|
||||
*******************************************************************************/
|
||||
private static String modifyDate(String table)
|
||||
{
|
||||
return (SNAKE_CASE_NAMES.contains(table) ? "modify_date" : "modifyDate");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** What the QBit declares, before core enriches anything: core gives a field
|
||||
** named createDate or modifyDate its behavior by itself, unless a host turned
|
||||
** that off, and gives the snake_case names nothing.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void everyDeclaredTableHasACreateDateAndAModifyDate() throws Exception
|
||||
{
|
||||
QInstance instance = base();
|
||||
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
|
||||
instance.addTable(person("memory"));
|
||||
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withClock(CLOCK).withAccessLogFlushIntervalMillis(60000)).produce(instance);
|
||||
QContext.init(instance, user("u1"));
|
||||
RUNTIMES.add(SecuredDataRuntime.current());
|
||||
|
||||
for(QTableMetaData table : declared(instance))
|
||||
{
|
||||
assertDate(table, createDate(table.getName()), "create_date", DynamicDefaultValueBehavior.CREATE_DATE);
|
||||
assertDate(table, modifyDate(table.getName()), "modify_date", DynamicDefaultValueBehavior.MODIFY_DATE);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One date field: a date-time nobody edits, kept by core, in a section.
|
||||
*******************************************************************************/
|
||||
private static void assertDate(QTableMetaData table, String name, String column, DynamicDefaultValueBehavior behavior)
|
||||
{
|
||||
String where = table.getName() + "." + name;
|
||||
assertThat(table.getFields()).as(where).containsKey(name);
|
||||
QFieldMetaData field = table.getField(name);
|
||||
assertThat(field.getType()).as(where).isEqualTo(QFieldType.DATE_TIME);
|
||||
assertThat(field.getIsEditable()).as(where).isFalse();
|
||||
assertThat(field.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class)).as(where).isEqualTo(behavior);
|
||||
assertThat(field.getBackendName()).as(where).isEqualTo(column);
|
||||
assertThat(table.getFields().values()).as(where).filteredOn(other -> other.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class) == behavior).hasSize(1);
|
||||
assertThat(table.getSections()).as(where).anySatisfy(section -> assertThat(section.getFieldNames()).contains(name));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The two dates have no classification, policy or assistant exposure of their
|
||||
** own. Whoever asks, they are given the tier of every other field of their
|
||||
** table that has none of its own either, and the assistant is given nothing.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theDatesAreReadableExactlyWhereTheOtherFieldsOfTheirTableAre() throws Exception
|
||||
{
|
||||
QInstance instance = setup(config -> config.withDefaultTier(DataClassification.CONFIDENTIAL, FieldAccessTier.MASKED));
|
||||
insert(SecuredDataTables.POLICY, policyRow("hidden", "CONFIDENTIAL", "HIDDEN"));
|
||||
insert(SecuredDataTables.POLICY, policyRow("full", "CONFIDENTIAL", "FULL"));
|
||||
insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "FULL"));
|
||||
insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "HIDDEN").withValue("client_channel", "web"));
|
||||
|
||||
Map<String, QSession> sessions = new LinkedHashMap<>();
|
||||
sessions.put("the default tier", user("u1"));
|
||||
sessions.put("a policy that hides", user("u2", "hidden"));
|
||||
sessions.put("a policy that shows", user("u3", "full"));
|
||||
sessions.put("a channel cap", SecuredDataTestSupport.channelUser("web", "capped"));
|
||||
sessions.put("the administrator", user("u4", "secured_data.admin"));
|
||||
Map<String, FieldAccessTier> expected = Map.of("the default tier", FieldAccessTier.MASKED, "a policy that hides", FieldAccessTier.HIDDEN,
|
||||
"a policy that shows", FieldAccessTier.FULL, "a channel cap", FieldAccessTier.HIDDEN, "the administrator", FieldAccessTier.FULL);
|
||||
|
||||
for(Map.Entry<String, QSession> session : sessions.entrySet())
|
||||
{
|
||||
QContext.setQSession(session.getValue());
|
||||
for(QTableMetaData table : declared(instance))
|
||||
{
|
||||
for(String name : List.of(createDate(table.getName()), modifyDate(table.getName())))
|
||||
{
|
||||
String where = session.getKey() + ": " + table.getName() + "." + name;
|
||||
QFieldMetaData date = table.getField(name);
|
||||
assertThat(AiFieldMetaData.of(date)).as(where).isNull();
|
||||
assertThat(SecuredDataClassifications.effective(table, name)).as(where).isEqualTo(DataClassification.CONFIDENTIAL);
|
||||
FieldAccessTier tier = SecuredDataRuntime.current().resolver().resolve(table, date).tier();
|
||||
assertThat(tier).as(where).isEqualTo(expected.get(session.getKey()));
|
||||
for(QFieldMetaData other : table.getFields().values())
|
||||
{
|
||||
if(!other.getName().equals(table.getPrimaryKeyField()))
|
||||
{
|
||||
assertThat(SecuredDataRuntime.current().resolver().resolve(table, other).tier()).as(where + " and " + other.getName()).isEqualTo(tier);
|
||||
}
|
||||
}
|
||||
}
|
||||
assertThat(AiTableMetaData.of(table).getExposure()).as(table.getName()).isEqualTo(AiExposure.NONE);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every writer of this QBit, on the memory backend: the access-log writer, a
|
||||
** changed policy and classification with their own log rows, a legal hold, a
|
||||
** reveal and a break-glass session from its start to its end and its expiry.
|
||||
** A new row has both dates, equal, from the clock of the QBit. A changed row
|
||||
** keeps its create date and has a later modify date, which core chooses (the
|
||||
** modify date is the record version of its table).
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void everyWriterSetsBothDatesAndOnlyAChangeMovesTheModifyDate() throws Exception
|
||||
{
|
||||
setup();
|
||||
|
||||
////////////////////////////////////////////////////////////////////////////
|
||||
// the access-log writer: created_at is when the access happened, the two //
|
||||
// dates are when the writer's one insert wrote the row //
|
||||
////////////////////////////////////////////////////////////////////////////
|
||||
SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hmac"));
|
||||
CLOCK.now = LATER;
|
||||
SecuredDataRuntime.current().logger().flush();
|
||||
QRecord logged = logs().getLast();
|
||||
assertThat(logged.getValueString("event_type")).isEqualTo("EXACT_MATCH_FILTER");
|
||||
assertThat(logged.getValueInstant("created_at")).isEqualTo(NOW);
|
||||
assertDates(logged, SecuredDataTables.ACCESS_LOG, LATER, LATER);
|
||||
assertThatThrownBy(() -> update(SecuredDataTables.ACCESS_LOG, new QRecord().withValue("id", logged.getValue("id")).withValue("outcome", "DENIED"))).isInstanceOf(Exception.class);
|
||||
assertDates(row(SecuredDataTables.ACCESS_LOG, logged.getValue("id")), SecuredDataTables.ACCESS_LOG, LATER, LATER);
|
||||
CLOCK.now = NOW;
|
||||
|
||||
///////////////////////////////////////////////////////////////////////
|
||||
// the rows the rules are read from, and the log row of every change //
|
||||
///////////////////////////////////////////////////////////////////////
|
||||
QRecord policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("reason_options", "Support").withValue("allow_free_text_reason", false));
|
||||
QRecord classification = classification("notes", "PII");
|
||||
QRecord hold = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "person").withValue("reason", "Preservation"));
|
||||
Map<String, Serializable> written = new LinkedHashMap<>();
|
||||
written.put(SecuredDataTables.POLICY, policy.getValue("id"));
|
||||
written.put(SecuredDataTables.CLASSIFICATION, classification.getValue("id"));
|
||||
written.put(SecuredDataTables.LEGAL_HOLD, hold.getValue("id"));
|
||||
for(Map.Entry<String, Serializable> row : written.entrySet())
|
||||
{
|
||||
assertDates(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW);
|
||||
}
|
||||
assertDates(log("POLICY_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW);
|
||||
assertDates(log("CLASSIFICATION_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW);
|
||||
|
||||
CLOCK.now = LATER;
|
||||
update(SecuredDataTables.POLICY, new QRecord().withValue("id", policy.getValue("id")).withValue("name", "renamed"));
|
||||
update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", classification.getValue("id")).withValue("notes", "reviewed"));
|
||||
update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("status", "RELEASED"));
|
||||
for(Map.Entry<String, Serializable> row : written.entrySet())
|
||||
{
|
||||
assertChanged(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW);
|
||||
}
|
||||
assertDates(log("POLICY_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER);
|
||||
assertDates(log("CLASSIFICATION_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER);
|
||||
CLOCK.now = NOW;
|
||||
|
||||
////////////////////////////////////////////
|
||||
// a reveal: its grant and its log row //
|
||||
////////////////////////////////////////////
|
||||
Serializable person = insert("person", new QRecord().withValue("email", "someone@example.invalid")).getValue("id");
|
||||
new RevealService().reveal("person", person, "email", "Support", null);
|
||||
assertDates(SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()).getFirst(), SecuredDataTables.REVEAL_GRANT, NOW, NOW);
|
||||
assertDates(log("REVEAL", null), SecuredDataTables.ACCESS_LOG, NOW, NOW);
|
||||
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
// a break-glass session: started, then ended; started, then expired //
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
BreakGlassService breaks = new BreakGlassService();
|
||||
QContext.setQSession(user("u1", "cs_admin", BreakGlassService.START + ".hasAccess"));
|
||||
Serializable ended = breaks.start("Investigating urgent incident", 60, null, null).getValue("id");
|
||||
assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW);
|
||||
CLOCK.now = LATER;
|
||||
breaks.end(ended);
|
||||
Instant endedAt = assertChanged(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW);
|
||||
assertDates(log("BREAK_GLASS_END", null), SecuredDataTables.ACCESS_LOG, LATER, LATER);
|
||||
|
||||
Serializable expired = breaks.start("Investigating another incident", 1, null, null).getValue("id");
|
||||
assertDates(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER);
|
||||
CLOCK.now = LATER.plusSeconds(60);
|
||||
assertThat(breaks.expire()).isEqualTo(1);
|
||||
assertChanged(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER);
|
||||
assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, endedAt);
|
||||
|
||||
/////////////////////////////////////////////////////////////////////////
|
||||
// the access log is only ever inserted, so no row's modify date moved //
|
||||
/////////////////////////////////////////////////////////////////////////
|
||||
assertThat(logs()).hasSizeGreaterThan(8).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.getValueInstant("createDate")).isNotNull();
|
||||
assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("createDate"));
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The writers of a wipe, in the shipped schema: the audit key, the run, the
|
||||
** marker (written once, then changed by a later wipe of the same field), the
|
||||
** log row, and the rule the run is recorded on. A time is compared with what
|
||||
** the same database gives back for another time written from the same clock,
|
||||
** not with the clock: the test database shifts a time by the zone of the
|
||||
** machine it runs on.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aWipeSetsBothDatesInTheShippedSchema() throws Exception
|
||||
{
|
||||
RetentionTestSupport.setup();
|
||||
QRecord rule = RetentionTestSupport.rule("OVERWRITE_TEXT", "secret");
|
||||
QRecord reading = RetentionTestSupport.reading(120);
|
||||
assertThat(SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", reading.getValue("id"), true)).isPresent();
|
||||
QRecord key = RetentionTestSupport.rows(SecuredDataTables.AUDIT_KEY).getFirst();
|
||||
assertDates(key, SecuredDataTables.AUDIT_KEY, key.getValueInstant("created_at"), key.getValueInstant("created_at"));
|
||||
Instant ruleWritten = row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")).getValueInstant("create_date");
|
||||
assertDates(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten);
|
||||
|
||||
CLOCK.now = LATER;
|
||||
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id")));
|
||||
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
|
||||
Instant wiped = ruleWritten.plusSeconds(90);
|
||||
QRecord firstRun = row(SecuredDataTables.WIPE_RUN, run.getValue("id"));
|
||||
assertThat(firstRun.getValueInstant("finished_at")).isEqualTo(wiped);
|
||||
assertDates(firstRun, SecuredDataTables.WIPE_RUN, wiped, wiped);
|
||||
QRecord marker = RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER).getFirst();
|
||||
assertThat(marker.getValueInstant("wiped_at")).isEqualTo(wiped);
|
||||
assertDates(marker, SecuredDataTables.WIPE_MARKER, wiped, wiped);
|
||||
assertThat(log("WIPE", "OVERWRITE_TEXT").getValueInstant("created_at")).isEqualTo(wiped);
|
||||
assertDates(log("WIPE", "OVERWRITE_TEXT"), SecuredDataTables.ACCESS_LOG, wiped, wiped);
|
||||
Instant ruleRun = assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten);
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
// the field is given a value again and wiped again: the marker of that //
|
||||
// record and field is changed, and no second marker is written //
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
CLOCK.now = LATER.plusSeconds(90);
|
||||
Instant again = wiped.plusSeconds(90);
|
||||
update("reading", new QRecord().withValue("id", reading.getValue("id")).withValue("secret", "another-test-secret"));
|
||||
QRecord second = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id")));
|
||||
assertThat(second.getValueInteger("wiped_count")).isEqualTo(1);
|
||||
assertThat(RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER)).hasSize(1);
|
||||
QRecord changed = row(SecuredDataTables.WIPE_MARKER, marker.getValue("id"));
|
||||
assertThat(changed.getValue("run_id")).isEqualTo(second.getValue("id"));
|
||||
assertThat(changed.getValueInstant("wiped_at")).isEqualTo(again);
|
||||
assertChanged(changed, SecuredDataTables.WIPE_MARKER, wiped, wiped);
|
||||
assertDates(row(SecuredDataTables.WIPE_RUN, run.getValue("id")), SecuredDataTables.WIPE_RUN, wiped, wiped);
|
||||
assertDates(row(SecuredDataTables.WIPE_RUN, second.getValue("id")), SecuredDataTables.WIPE_RUN, again, again);
|
||||
assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleRun);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The access-log writer runs concurrently with reveals, so the dates must cost
|
||||
** it nothing: for each row it still runs one insert, which now names the two
|
||||
** columns, and no other statement that writes the log.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theAccessLogWriterStillRunsOneInsertForARow() throws Exception
|
||||
{
|
||||
RetentionTestSupport.setup();
|
||||
try(Connection connection = DriverManager.getConnection(url(), "sa", ""))
|
||||
{
|
||||
connection.createStatement().execute("SET QUERY_STATISTICS TRUE");
|
||||
for(int i = 0; i < 3; i++)
|
||||
{
|
||||
SecuredDataRuntime.current().logger().onAccess(new FieldAccessEvent().withType(FieldAccessEventType.EXACT_MATCH_FILTER).withOutcome(FieldAccessOutcome.MATCH)
|
||||
.withTableName("reading").withFieldNames(List.of("secret")).withActionName("query").withRecordCount(1).withSearchValueFingerprints(List.of("hmac" + i)));
|
||||
}
|
||||
SecuredDataRuntime.current().logger().flush();
|
||||
assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).hasSize(3).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull().isEqualTo(row.get("modify_date")));
|
||||
|
||||
Map<String, Integer> writes = new LinkedHashMap<>();
|
||||
try(ResultSet statistics = connection.createStatement().executeQuery("SELECT SQL_STATEMENT, EXECUTION_COUNT FROM INFORMATION_SCHEMA.QUERY_STATISTICS"))
|
||||
{
|
||||
while(statistics.next())
|
||||
{
|
||||
String sql = statistics.getString(1).strip();
|
||||
if(sql.contains(SecuredDataTables.ACCESS_LOG) && !sql.toUpperCase().startsWith("SELECT"))
|
||||
{
|
||||
writes.merge(sql, statistics.getInt(2), Integer::sum);
|
||||
}
|
||||
}
|
||||
}
|
||||
assertThat(writes).hasSize(1);
|
||||
assertThat(writes.keySet().iterator().next()).startsWithIgnoringCase("INSERT INTO").contains("create_date", "modify_date");
|
||||
assertThat(writes.values()).containsExactly(3);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A changed policy or classification writes what it wrote before the tables
|
||||
** had the standard dates: its core audit, field by field, and its
|
||||
** POLICY_CHANGE or CLASSIFICATION_CHANGE row in the access log.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aChangedPolicyOrClassificationWritesItsAuditAndItsLogRowAsBefore() throws Exception
|
||||
{
|
||||
QInstance instance = RetentionTestSupport.setup();
|
||||
QRecord policy = new InsertAction().execute(new InsertInput(SecuredDataTables.POLICY).withRecord(policyRow("viewer", "PHI", "MASKED"))).getRecords().getFirst();
|
||||
QRecord classification = new InsertAction().execute(new InsertInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("table_name", "reading")
|
||||
.withValue("field_name", "notes").withValue("classification", "PHI"))).getRecords().getFirst();
|
||||
assertThat(policy.getErrors()).isNullOrEmpty();
|
||||
assertThat(classification.getErrors()).isNullOrEmpty();
|
||||
CLOCK.now = LATER;
|
||||
assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.POLICY).withRecord(new QRecord().withValue("id", policy.getValue("id")).withValue("tier", "HIDDEN")))
|
||||
.getRecords().getFirst().getErrors()).isNullOrEmpty();
|
||||
assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("id", classification.getValue("id"))
|
||||
.withValue("classification", "PII"))).getRecords().getFirst().getErrors()).isNullOrEmpty();
|
||||
new DeleteAction().execute(new DeleteInput(SecuredDataTables.POLICY).withPrimaryKeys(List.of(policy.getValue("id"))));
|
||||
new DeleteAction().execute(new DeleteInput(SecuredDataTables.CLASSIFICATION).withPrimaryKeys(List.of(classification.getValue("id"))));
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
// the log rows: one per change, with every value they have always had //
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
List<String> changes = new ArrayList<>();
|
||||
List<QRecord> logged = RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG);
|
||||
Instant inserted = logged.getFirst().getValueInstant("created_at");
|
||||
for(QRecord row : logged)
|
||||
{
|
||||
changes.add(String.join(" ", row.getValueString("event_type"), row.getValueString("table_name"), row.getValueString("record_id"), row.getValueString("action_name"),
|
||||
row.getValueString("outcome"), row.getValueString("user_id"), row.getValueString("user_name"),
|
||||
"after " + Duration.between(inserted, row.getValueInstant("created_at")).toSeconds() + " s", String.valueOf(row.getValueBoolean("anomaly_flag"))));
|
||||
for(String empty : List.of("field_name", "record_count", "reason", "reason_text", "client_channel", "break_glass_id", "search_value_fingerprints", "rule_id"))
|
||||
{
|
||||
assertThat(row.getValue(empty)).as(empty).isNull();
|
||||
}
|
||||
}
|
||||
String policyId = policy.getValueString("id");
|
||||
String classificationId = classification.getValueString("id");
|
||||
assertThat(changes).containsExactly(
|
||||
"POLICY_CHANGE secured_data_policy " + policyId + " INSERT ALLOWED admin admin after 0 s false",
|
||||
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " INSERT ALLOWED admin admin after 0 s false",
|
||||
"POLICY_CHANGE secured_data_policy " + policyId + " UPDATE ALLOWED admin admin after 90 s false",
|
||||
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " UPDATE ALLOWED admin admin after 90 s false",
|
||||
"POLICY_CHANGE secured_data_policy " + policyId + " DELETE ALLOWED admin admin after 90 s false",
|
||||
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " DELETE ALLOWED admin admin after 90 s false");
|
||||
|
||||
//////////////////////////////////////////////////////////////////////////////
|
||||
// the core audits of the two rows: the same audits with the same details, //
|
||||
// the two dates among them under the names they have always been audited by //
|
||||
//////////////////////////////////////////////////////////////////////////////
|
||||
assertThat(audits(instance, SecuredDataTables.POLICY, policy.getValue("id"))).containsExactly(
|
||||
"Record was Inserted: allow_free_text_reason [Set allow free text reason to Yes], classification [Set classification to PHI], create_date,"
|
||||
+ " distinct_value_alert [Set distinct value alert to 15], exact_match_per_hour [Set exact match per hour to 20], id [Set Id to " + policyId + "],"
|
||||
+ " is_active [Set is active to Yes], min_entropy_enabled [Set min entropy enabled to Yes], miss_alert_consecutive [Set miss alert consecutive to 5],"
|
||||
+ " modified_by [Set modified by to admin], modify_date, name [Set name to viewer PHI], permission_name [Set permission name to viewer],"
|
||||
+ " require_free_text [Set require free text to No], reveal_minutes [Set reveal minutes to 5], tier [Set tier to MASKED]",
|
||||
"Record was Edited: modify_date, tier [Changed tier from \"MASKED\" to \"HIDDEN\"]",
|
||||
"Record was Deleted: ");
|
||||
assertThat(audits(instance, SecuredDataTables.CLASSIFICATION, classification.getValue("id"))).containsExactly(
|
||||
"Record was Inserted: classification [Set classification to PHI], create_date, field_name [Set field name to notes], id [Set Id to " + classificationId + "],"
|
||||
+ " modified_by [Set modified by to admin], modify_date, table_name [Set table name to Reading]",
|
||||
"Record was Edited: classification [Changed classification from \"PHI\" to \"PII\"], modify_date",
|
||||
"Record was Deleted: ");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The core audits of one record, oldest first: the message of each and, in
|
||||
** the order of the field names, the fields it has a detail for and what the
|
||||
** detail says. The two dates are named only: how a time reads depends on the
|
||||
** zone of the machine.
|
||||
*******************************************************************************/
|
||||
private static List<String> audits(QInstance instance, String table, Serializable id)
|
||||
{
|
||||
Serializable auditTable = SystemWrites.query("auditTable", new QQueryFilter().withCriteria(new QFilterCriteria("name", QCriteriaOperator.EQUALS, table))).getFirst().getValue("id");
|
||||
List<String> audits = new ArrayList<>();
|
||||
for(QRecord audit : SystemWrites.query("audit", new QQueryFilter().withCriteria(new QFilterCriteria("auditTableId", QCriteriaOperator.EQUALS, auditTable))
|
||||
.withCriteria(new QFilterCriteria("recordId", QCriteriaOperator.EQUALS, id)).withOrderBy(new QFilterOrderBy("id", true))))
|
||||
{
|
||||
List<String> fields = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("auditId", QCriteriaOperator.EQUALS, audit.getValue("id"))))
|
||||
.stream().map(detail -> detail.getValueString("fieldName") + (detail.getValueString("fieldName").endsWith("_date") ? ""
|
||||
: " [" + detail.getValueString("message") + "]")).sorted().toList();
|
||||
audits.add(audit.getValueString("message") + ": " + String.join(", ", fields));
|
||||
}
|
||||
assertThat(instance.getTable(table)).isNotNull();
|
||||
return (audits);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Rows written before the two columns existed. A reading is audited (its
|
||||
** audit token is made with a data key of its own, and that key is stored
|
||||
** wrapped, bound to its record and field), the log has rows, a rule and a
|
||||
** hold exist. The two columns are then taken away, which leaves the database
|
||||
** as the release before this one left it, and a key and a log row are
|
||||
** written the way that release wrote them. The block an existing database
|
||||
** runs must change nothing but the two columns: the same rows, the same
|
||||
** wrapped keys, which still unwrap to the same data keys, and the same
|
||||
** token, which still verifies the value and no other.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void rowsWrittenBeforeTheDatesExistedVerifyUnchangedAfterTheyAreAdded() throws Exception
|
||||
{
|
||||
QInstance instance = RetentionTestSupport.setup();
|
||||
QTableMetaData reading = instance.getTable("reading");
|
||||
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
|
||||
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
|
||||
QRecord rule = RetentionTestSupport.rule("NULL_WITH_MARKER", "systolic");
|
||||
insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", "404").withValue("reason", "Preservation"));
|
||||
Serializable id = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 120).withValue("secret", "s")
|
||||
.withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id");
|
||||
Serializable wiped = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 130).withValue("secret", "s")
|
||||
.withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id");
|
||||
new WipeService().batch(rule.getValue("id"), List.of(wiped));
|
||||
|
||||
SecuredDataKeySource keys = SecuredDataRuntime.current().keys();
|
||||
byte[] dataKey = keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key();
|
||||
List<String> tokens = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("fieldName", QCriteriaOperator.EQUALS, "systolic"))).stream()
|
||||
.map(detail -> detail.getValueString("newValue")).filter(value -> value != null && value.startsWith("shred:v1:")).toList();
|
||||
String token = tokens.stream().filter(candidate -> ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, candidate)).findFirst().orElseThrow();
|
||||
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse();
|
||||
|
||||
List<String> tables = declared(instance).stream().map(QTableMetaData::getName).filter(name -> !SNAKE_CASE_NAMES.contains(name)).toList();
|
||||
byte[] earlierKey = new byte[32];
|
||||
Arrays.fill(earlierKey, (byte) 7);
|
||||
Instant earlier = NOW.minusSeconds(86400);
|
||||
try(Connection connection = DriverManager.getConnection(url(), "sa", ""))
|
||||
{
|
||||
for(String table : tables)
|
||||
{
|
||||
assertThat(rows(connection, table, false)).as(table).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull());
|
||||
connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN create_date");
|
||||
connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN modify_date");
|
||||
}
|
||||
try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_audit_key (table_name, record_id, field_name, wrapped_key, created_at) VALUES ('reading', '9001', 'systolic', ?, ?)"))
|
||||
{
|
||||
insert.setString(1, Base64.getEncoder().encodeToString(SecuredDataKeySource.encrypt(earlierKey, "reading\u001fsystolic\u001f9001")));
|
||||
insert.setTimestamp(2, Timestamp.from(earlier));
|
||||
insert.executeUpdate();
|
||||
}
|
||||
try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_access_log (event_type, user_id, table_name, field_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u1', 'reading', 'systolic', 'ALLOWED', false, ?)"))
|
||||
{
|
||||
insert.setTimestamp(1, Timestamp.from(earlier));
|
||||
insert.executeUpdate();
|
||||
}
|
||||
Map<String, List<Map<String, String>>> before = new LinkedHashMap<>();
|
||||
for(String table : tables)
|
||||
{
|
||||
before.put(table, rows(connection, table, false));
|
||||
}
|
||||
assertThat(before.get(SecuredDataTables.ACCESS_LOG)).hasSizeGreaterThan(3);
|
||||
assertThat(before.get(SecuredDataTables.AUDIT_KEY)).hasSize(2);
|
||||
for(String table : List.of(SecuredDataTables.WIPE_RUN, SecuredDataTables.WIPE_MARKER, SecuredDataTables.LEGAL_HOLD))
|
||||
{
|
||||
assertThat(before.get(table)).as(table).hasSize(1);
|
||||
}
|
||||
|
||||
run(connection, ddl(true));
|
||||
|
||||
for(String table : tables)
|
||||
{
|
||||
assertThat(rows(connection, table, true)).as(table).isEqualTo(before.get(table));
|
||||
assertThat(rows(connection, table, false)).as(table).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.get("create_date")).isNotNull();
|
||||
assertThat(row.get("modify_date")).isNotNull();
|
||||
});
|
||||
}
|
||||
assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.get("create_date")).isEqualTo(row.get("created_at"));
|
||||
assertThat(row.get("modify_date")).isEqualTo(row.get("created_at"));
|
||||
});
|
||||
assertThat(rows(connection, SecuredDataTables.AUDIT_KEY, false)).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.get("create_date")).isEqualTo(row.get("created_at"));
|
||||
assertThat(row.get("modify_date")).isEqualTo(row.get("created_at"));
|
||||
});
|
||||
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
// the release before this one goes on writing while both run: its insert //
|
||||
// names neither column, and the row has both, equal, all the same //
|
||||
/////////////////////////////////////////////////////////////////////////////
|
||||
connection.createStatement().execute("INSERT INTO secured_data_access_log (event_type, user_id, table_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u2', 'reading', 'ALLOWED', false, CURRENT_TIMESTAMP)");
|
||||
Map<String, String> concurrent = rows(connection, SecuredDataTables.ACCESS_LOG, false).getLast();
|
||||
assertThat(concurrent.get("user_id")).isEqualTo("u2");
|
||||
assertThat(concurrent.get("create_date")).isNotNull().isEqualTo(concurrent.get("modify_date"));
|
||||
}
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
// nothing kept in memory: every key is unwrapped again from its row //
|
||||
///////////////////////////////////////////////////////////////////////////
|
||||
keys.close();
|
||||
assertThat(keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key()).isEqualTo(dataKey);
|
||||
assertThat(keys.keyForRecordField("reading", "systolic", 9001, false).orElseThrow().key()).isEqualTo(earlierKey);
|
||||
assertThat(keys.keyForRecordField("reading", "systolic", wiped, false)).isEmpty();
|
||||
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isTrue();
|
||||
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse();
|
||||
|
||||
////////////////////////////////////////////////////////////////////////////
|
||||
// and the rows that were there are read and written on as they were: the //
|
||||
// earlier log row is counted, and the reading is wiped under its rule //
|
||||
////////////////////////////////////////////////////////////////////////////
|
||||
assertThat(RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG)).anySatisfy(row ->
|
||||
{
|
||||
assertThat(row.getValueString("user_id")).isEqualTo("u1");
|
||||
assertThat(row.getValueInstant("createDate")).isNotNull().isEqualTo(row.getValueInstant("created_at"));
|
||||
assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("created_at"));
|
||||
});
|
||||
CLOCK.now = LATER;
|
||||
assertThat(new WipeService().batch(rule.getValue("id"), List.of(id)).getValueInteger("wiped_count")).isEqualTo(1);
|
||||
assertThat(keys.keyForRecordField("reading", "systolic", id, false)).isEmpty();
|
||||
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isFalse();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The database RetentionTestSupport made from the shipped DDL.
|
||||
*******************************************************************************/
|
||||
private static String url()
|
||||
{
|
||||
return ("jdbc:h2:mem:" + RetentionTestSupport.BACKEND + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The statements of the shipped DDL: those of the block an existing database
|
||||
** runs to get the two columns, or those before it (the schema without them).
|
||||
*******************************************************************************/
|
||||
static List<String> ddl(boolean datesBlock) throws IOException
|
||||
{
|
||||
String ddl = Files.readString(Path.of(DDL));
|
||||
int block = ddl.indexOf(DDL_BLOCK);
|
||||
assertThat(block).as("the block of " + DDL + " that begins " + DDL_BLOCK).isPositive().isEqualTo(ddl.lastIndexOf(DDL_BLOCK));
|
||||
return (Arrays.stream((datesBlock ? ddl.substring(block) : ddl.substring(0, block)).split(";")).filter(sql -> !sql.isBlank()).toList());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Run statements.
|
||||
*******************************************************************************/
|
||||
static void run(Connection connection, List<String> statements) throws SQLException
|
||||
{
|
||||
for(String sql : statements)
|
||||
{
|
||||
connection.createStatement().execute(sql);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every row of a table as the database holds it, column by column, in id
|
||||
** order, with or without the two date columns.
|
||||
*******************************************************************************/
|
||||
static List<Map<String, String>> rows(Connection connection, String table, boolean withoutDates) throws SQLException
|
||||
{
|
||||
List<Map<String, String>> rows = new ArrayList<>();
|
||||
try(ResultSet results = connection.createStatement().executeQuery("SELECT * FROM " + table + " ORDER BY id"))
|
||||
{
|
||||
ResultSetMetaData columns = results.getMetaData();
|
||||
while(results.next())
|
||||
{
|
||||
Map<String, String> row = new LinkedHashMap<>();
|
||||
for(int i = 1; i <= columns.getColumnCount(); i++)
|
||||
{
|
||||
String column = columns.getColumnName(i);
|
||||
if(!withoutDates || !Set.of("create_date", "modify_date").contains(column))
|
||||
{
|
||||
row.put(column, results.getString(i));
|
||||
}
|
||||
}
|
||||
rows.add(row);
|
||||
}
|
||||
}
|
||||
return (rows);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One stored row.
|
||||
*******************************************************************************/
|
||||
private static QRecord row(String table, Serializable id)
|
||||
{
|
||||
return (SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id))).getFirst());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The newest log row of an event, and of an action where one is given.
|
||||
*******************************************************************************/
|
||||
private static QRecord log(String event, String action)
|
||||
{
|
||||
return (logs().stream().filter(row -> event.equals(row.getValueString("event_type")) && (action == null || action.equals(row.getValueString("action_name"))))
|
||||
.reduce((first, second) -> second).orElseThrow());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Both dates of a stored row.
|
||||
*******************************************************************************/
|
||||
private static void assertDates(QRecord row, String table, Instant created, Instant modified)
|
||||
{
|
||||
assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created);
|
||||
assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isEqualTo(modified);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Both dates of a stored row that was changed: the create date it had, and a
|
||||
** modify date later than the one it had, which is returned.
|
||||
*******************************************************************************/
|
||||
private static Instant assertChanged(QRecord row, String table, Instant created, Instant modifiedBefore)
|
||||
{
|
||||
assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created);
|
||||
assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isAfter(modifiedBefore);
|
||||
return (row.getValueInstant(modifyDate(table)));
|
||||
}
|
||||
}
|
||||
@@ -234,7 +234,6 @@ final class SecuredDataTestSupport
|
||||
*******************************************************************************/
|
||||
static void close()
|
||||
{
|
||||
CLOCK.beforeInstant = null;
|
||||
RUNTIMES.forEach(SecuredDataRuntime::close);
|
||||
RUNTIMES.clear();
|
||||
QContext.clear();
|
||||
@@ -249,9 +248,6 @@ final class SecuredDataTestSupport
|
||||
{
|
||||
Instant now = NOW;
|
||||
|
||||
/** Run on the reading thread before every reading of the clock; a test parks a thread at a known point with it. */
|
||||
volatile Runnable beforeInstant;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
@@ -279,11 +275,6 @@ final class SecuredDataTestSupport
|
||||
*******************************************************************************/
|
||||
public Instant instant()
|
||||
{
|
||||
Runnable hook = beforeInstant;
|
||||
if(hook != null)
|
||||
{
|
||||
hook.run();
|
||||
}
|
||||
return (now);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user