Files
qbit-secured-data/src/test/java/com/kingsrook/qbits/secureddata/SecuredDataStandardDatesTest.java
T
james.maesandClaude Fable 5.2 4e20070ae1
maven / build (push) Successful in 1m56s
build / maven (push) Successful in 1m55s
feat: a create date and a modify date on every table
Every table of the QBit has a create date and a modify date: DATE_TIME, not
editable, with core's CREATE_DATE and MODIFY_DATE behaviors, in the columns
create_date and modify_date.

Seven tables did not have them: reveal grant, break-glass session, access
log, wipe run, wipe marker, legal hold and audit key. Their fields are named
createDate and modifyDate. Classification, policy and retention rule already
had create_date and modify_date, set by the table customizer. They keep those
field names, because a rename would change what a host reads and what their
audits record, and the two fields are now not editable and have the
behaviors.

Writes. Every write of the QBit is a core insert or update action, so the
dates are in the statement that writes the row. The table customizer sets
them, as it did for the three tables: a new row gets both, equal, from the
QBit's clock, and a changed row a new modify date. The access-log writer
still runs one insert per row and no other statement, lock or transaction.
The access log and wipe runs are only ever inserted, so their dates stay
equal.

Core uses a DATE_TIME field with the MODIFY_DATE behavior as the record
version of its table. That now holds for all ten tables: on an update core
sets the modify date itself, after the customizer has run, and an update
that sends the version it read is refused when the row has changed since.
Updates that send no version, the QBit's own among them, behave as before.

Schema. docs/ddl/secured-data-postgres.sql has a new last block that adds
the two columns to the seven tables: nullable, defaulting to the current
time, with the rows that were already there given the row's own time. No
CREATE TABLE statement is edited. An existing database runs that block
alone, before this release is deployed. The block rewrites no table, can be
run while the release before this one is running (its inserts name neither
column and get both from the default) and can be run again. This release
does not run without the columns. README: "Upgrading an existing database".

Unchanged. The dates have no classification, policy or assistant exposure of
their own: they are CONFIDENTIAL with their table and get the tier of its
other fields. Nothing signed or verified covers them: a wrapped data key is
bound to its table, field and record id, and the fingerprints of a log row
are made from search values. A changed policy or classification writes the
same core audit, field by field, and the same POLICY_CHANGE or
CLASSIFICATION_CHANGE row.

Tests. SecuredDataStandardDatesTest: every table the producer declares has
the two dates (the tables are taken from the instance, so a table added
later without them fails); the dates resolve to the tier of the other fields
of their table for five kinds of session; every writer sets both and only a
change moves the modify date (access-log writer, policy, classification,
legal hold, reveal, break-glass start, end and expiry on the memory backend;
audit key, wipe run, wipe marker and retention rule in the shipped schema);
the access-log writer runs exactly one insert per row; a changed policy and
classification write the audit and log rows they wrote before; rows written
before the columns existed are unchanged by the upgrade block, their wrapped
keys unwrap to the same data keys and their audit token verifies the same
value and no other. SecuredDataDdlTest: the shipped DDL gives every declared
table the two columns, the upgrade block adds them where they were missing,
an insert that names neither succeeds, and the block can be run twice. Seven
of the eight new tests fail without the change; the eighth is the record of
what a changed policy wrote before and passes on both sides.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 09:01:43 -06:00

775 lines
46 KiB
Java

/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.IOException;
import java.io.Serializable;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.WipeService;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Every table has a create date and a modify date: what the metadata says of
** them, who may read them, every writer of this QBit setting them, what the
** access-log writer runs for a row, the records of a changed policy, and rows
** that were written before the two columns existed.
*******************************************************************************/
class SecuredDataStandardDatesTest
{
static final String DDL = "docs/ddl/secured-data-postgres.sql";
/** The first line of the block of the DDL that an existing database runs to get the two columns. */
static final String DDL_BLOCK = "-- Standard dates:";
/** The tables that had the two dates before the other tables did: they keep the field names they have always had. */
private static final Set<String> SNAKE_CASE_NAMES = Set.of("secured_data_classification", "secured_data_policy", "secured_data_retention_rule");
private static final Instant LATER = NOW.plusSeconds(90);
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
RetentionTestSupport.close();
}
/*******************************************************************************
** The tables the producer put in an instance, whichever class declares them,
** so that a table added later is covered without being named here.
*******************************************************************************/
static List<QTableMetaData> declared(QInstance instance)
{
String qBit = instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getName();
List<QTableMetaData> tables = instance.getTables().values().stream().filter(table -> qBit.equals(table.getSourceQBitName())).toList();
assertThat(tables).extracting(QTableMetaData::getName).containsAll(SecuredDataTables.ALL);
return (tables);
}
/*******************************************************************************
** Create date field name.
*******************************************************************************/
private static String createDate(String table)
{
return (SNAKE_CASE_NAMES.contains(table) ? "create_date" : "createDate");
}
/*******************************************************************************
** Modify date field name.
*******************************************************************************/
private static String modifyDate(String table)
{
return (SNAKE_CASE_NAMES.contains(table) ? "modify_date" : "modifyDate");
}
/*******************************************************************************
** What the QBit declares, before core enriches anything: core gives a field
** named createDate or modifyDate its behavior by itself, unless a host turned
** that off, and gives the snake_case names nothing.
*******************************************************************************/
@Test
void everyDeclaredTableHasACreateDateAndAModifyDate() throws Exception
{
QInstance instance = base();
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
instance.addTable(person("memory"));
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withClock(CLOCK).withAccessLogFlushIntervalMillis(60000)).produce(instance);
QContext.init(instance, user("u1"));
RUNTIMES.add(SecuredDataRuntime.current());
for(QTableMetaData table : declared(instance))
{
assertDate(table, createDate(table.getName()), "create_date", DynamicDefaultValueBehavior.CREATE_DATE);
assertDate(table, modifyDate(table.getName()), "modify_date", DynamicDefaultValueBehavior.MODIFY_DATE);
}
}
/*******************************************************************************
** One date field: a date-time nobody edits, kept by core, in a section.
*******************************************************************************/
private static void assertDate(QTableMetaData table, String name, String column, DynamicDefaultValueBehavior behavior)
{
String where = table.getName() + "." + name;
assertThat(table.getFields()).as(where).containsKey(name);
QFieldMetaData field = table.getField(name);
assertThat(field.getType()).as(where).isEqualTo(QFieldType.DATE_TIME);
assertThat(field.getIsEditable()).as(where).isFalse();
assertThat(field.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class)).as(where).isEqualTo(behavior);
assertThat(field.getBackendName()).as(where).isEqualTo(column);
assertThat(table.getFields().values()).as(where).filteredOn(other -> other.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class) == behavior).hasSize(1);
assertThat(table.getSections()).as(where).anySatisfy(section -> assertThat(section.getFieldNames()).contains(name));
}
/*******************************************************************************
** The two dates have no classification, policy or assistant exposure of their
** own. Whoever asks, they are given the tier of every other field of their
** table that has none of its own either, and the assistant is given nothing.
*******************************************************************************/
@Test
void theDatesAreReadableExactlyWhereTheOtherFieldsOfTheirTableAre() throws Exception
{
QInstance instance = setup(config -> config.withDefaultTier(DataClassification.CONFIDENTIAL, FieldAccessTier.MASKED));
insert(SecuredDataTables.POLICY, policyRow("hidden", "CONFIDENTIAL", "HIDDEN"));
insert(SecuredDataTables.POLICY, policyRow("full", "CONFIDENTIAL", "FULL"));
insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "FULL"));
insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "HIDDEN").withValue("client_channel", "web"));
Map<String, QSession> sessions = new LinkedHashMap<>();
sessions.put("the default tier", user("u1"));
sessions.put("a policy that hides", user("u2", "hidden"));
sessions.put("a policy that shows", user("u3", "full"));
sessions.put("a channel cap", SecuredDataTestSupport.channelUser("web", "capped"));
sessions.put("the administrator", user("u4", "secured_data.admin"));
Map<String, FieldAccessTier> expected = Map.of("the default tier", FieldAccessTier.MASKED, "a policy that hides", FieldAccessTier.HIDDEN,
"a policy that shows", FieldAccessTier.FULL, "a channel cap", FieldAccessTier.HIDDEN, "the administrator", FieldAccessTier.FULL);
for(Map.Entry<String, QSession> session : sessions.entrySet())
{
QContext.setQSession(session.getValue());
for(QTableMetaData table : declared(instance))
{
for(String name : List.of(createDate(table.getName()), modifyDate(table.getName())))
{
String where = session.getKey() + ": " + table.getName() + "." + name;
QFieldMetaData date = table.getField(name);
assertThat(AiFieldMetaData.of(date)).as(where).isNull();
assertThat(SecuredDataClassifications.effective(table, name)).as(where).isEqualTo(DataClassification.CONFIDENTIAL);
FieldAccessTier tier = SecuredDataRuntime.current().resolver().resolve(table, date).tier();
assertThat(tier).as(where).isEqualTo(expected.get(session.getKey()));
for(QFieldMetaData other : table.getFields().values())
{
if(!other.getName().equals(table.getPrimaryKeyField()))
{
assertThat(SecuredDataRuntime.current().resolver().resolve(table, other).tier()).as(where + " and " + other.getName()).isEqualTo(tier);
}
}
}
assertThat(AiTableMetaData.of(table).getExposure()).as(table.getName()).isEqualTo(AiExposure.NONE);
}
}
}
/*******************************************************************************
** Every writer of this QBit, on the memory backend: the access-log writer, a
** changed policy and classification with their own log rows, a legal hold, a
** reveal and a break-glass session from its start to its end and its expiry.
** A new row has both dates, equal, from the clock of the QBit. A changed row
** keeps its create date and has a later modify date, which core chooses (the
** modify date is the record version of its table).
*******************************************************************************/
@Test
void everyWriterSetsBothDatesAndOnlyAChangeMovesTheModifyDate() throws Exception
{
setup();
////////////////////////////////////////////////////////////////////////////
// the access-log writer: created_at is when the access happened, the two //
// dates are when the writer's one insert wrote the row //
////////////////////////////////////////////////////////////////////////////
SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hmac"));
CLOCK.now = LATER;
SecuredDataRuntime.current().logger().flush();
QRecord logged = logs().getLast();
assertThat(logged.getValueString("event_type")).isEqualTo("EXACT_MATCH_FILTER");
assertThat(logged.getValueInstant("created_at")).isEqualTo(NOW);
assertDates(logged, SecuredDataTables.ACCESS_LOG, LATER, LATER);
assertThatThrownBy(() -> update(SecuredDataTables.ACCESS_LOG, new QRecord().withValue("id", logged.getValue("id")).withValue("outcome", "DENIED"))).isInstanceOf(Exception.class);
assertDates(row(SecuredDataTables.ACCESS_LOG, logged.getValue("id")), SecuredDataTables.ACCESS_LOG, LATER, LATER);
CLOCK.now = NOW;
///////////////////////////////////////////////////////////////////////
// the rows the rules are read from, and the log row of every change //
///////////////////////////////////////////////////////////////////////
QRecord policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("reason_options", "Support").withValue("allow_free_text_reason", false));
QRecord classification = classification("notes", "PII");
QRecord hold = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "person").withValue("reason", "Preservation"));
Map<String, Serializable> written = new LinkedHashMap<>();
written.put(SecuredDataTables.POLICY, policy.getValue("id"));
written.put(SecuredDataTables.CLASSIFICATION, classification.getValue("id"));
written.put(SecuredDataTables.LEGAL_HOLD, hold.getValue("id"));
for(Map.Entry<String, Serializable> row : written.entrySet())
{
assertDates(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW);
}
assertDates(log("POLICY_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW);
assertDates(log("CLASSIFICATION_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW);
CLOCK.now = LATER;
update(SecuredDataTables.POLICY, new QRecord().withValue("id", policy.getValue("id")).withValue("name", "renamed"));
update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", classification.getValue("id")).withValue("notes", "reviewed"));
update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("status", "RELEASED"));
for(Map.Entry<String, Serializable> row : written.entrySet())
{
assertChanged(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW);
}
assertDates(log("POLICY_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER);
assertDates(log("CLASSIFICATION_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER);
CLOCK.now = NOW;
////////////////////////////////////////////
// a reveal: its grant and its log row //
////////////////////////////////////////////
Serializable person = insert("person", new QRecord().withValue("email", "someone@example.invalid")).getValue("id");
new RevealService().reveal("person", person, "email", "Support", null);
assertDates(SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()).getFirst(), SecuredDataTables.REVEAL_GRANT, NOW, NOW);
assertDates(log("REVEAL", null), SecuredDataTables.ACCESS_LOG, NOW, NOW);
//////////////////////////////////////////////////////////////////////
// a break-glass session: started, then ended; started, then expired //
//////////////////////////////////////////////////////////////////////
BreakGlassService breaks = new BreakGlassService();
QContext.setQSession(user("u1", "cs_admin", BreakGlassService.START + ".hasAccess"));
Serializable ended = breaks.start("Investigating urgent incident", 60, null, null).getValue("id");
assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW);
CLOCK.now = LATER;
breaks.end(ended);
Instant endedAt = assertChanged(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW);
assertDates(log("BREAK_GLASS_END", null), SecuredDataTables.ACCESS_LOG, LATER, LATER);
Serializable expired = breaks.start("Investigating another incident", 1, null, null).getValue("id");
assertDates(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER);
CLOCK.now = LATER.plusSeconds(60);
assertThat(breaks.expire()).isEqualTo(1);
assertChanged(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER);
assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, endedAt);
/////////////////////////////////////////////////////////////////////////
// the access log is only ever inserted, so no row's modify date moved //
/////////////////////////////////////////////////////////////////////////
assertThat(logs()).hasSizeGreaterThan(8).allSatisfy(row ->
{
assertThat(row.getValueInstant("createDate")).isNotNull();
assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("createDate"));
});
}
/*******************************************************************************
** The writers of a wipe, in the shipped schema: the audit key, the run, the
** marker (written once, then changed by a later wipe of the same field), the
** log row, and the rule the run is recorded on. A time is compared with what
** the same database gives back for another time written from the same clock,
** not with the clock: the test database shifts a time by the zone of the
** machine it runs on.
*******************************************************************************/
@Test
void aWipeSetsBothDatesInTheShippedSchema() throws Exception
{
RetentionTestSupport.setup();
QRecord rule = RetentionTestSupport.rule("OVERWRITE_TEXT", "secret");
QRecord reading = RetentionTestSupport.reading(120);
assertThat(SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", reading.getValue("id"), true)).isPresent();
QRecord key = RetentionTestSupport.rows(SecuredDataTables.AUDIT_KEY).getFirst();
assertDates(key, SecuredDataTables.AUDIT_KEY, key.getValueInstant("created_at"), key.getValueInstant("created_at"));
Instant ruleWritten = row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")).getValueInstant("create_date");
assertDates(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten);
CLOCK.now = LATER;
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id")));
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
Instant wiped = ruleWritten.plusSeconds(90);
QRecord firstRun = row(SecuredDataTables.WIPE_RUN, run.getValue("id"));
assertThat(firstRun.getValueInstant("finished_at")).isEqualTo(wiped);
assertDates(firstRun, SecuredDataTables.WIPE_RUN, wiped, wiped);
QRecord marker = RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER).getFirst();
assertThat(marker.getValueInstant("wiped_at")).isEqualTo(wiped);
assertDates(marker, SecuredDataTables.WIPE_MARKER, wiped, wiped);
assertThat(log("WIPE", "OVERWRITE_TEXT").getValueInstant("created_at")).isEqualTo(wiped);
assertDates(log("WIPE", "OVERWRITE_TEXT"), SecuredDataTables.ACCESS_LOG, wiped, wiped);
Instant ruleRun = assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten);
///////////////////////////////////////////////////////////////////////////
// the field is given a value again and wiped again: the marker of that //
// record and field is changed, and no second marker is written //
///////////////////////////////////////////////////////////////////////////
CLOCK.now = LATER.plusSeconds(90);
Instant again = wiped.plusSeconds(90);
update("reading", new QRecord().withValue("id", reading.getValue("id")).withValue("secret", "another-test-secret"));
QRecord second = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id")));
assertThat(second.getValueInteger("wiped_count")).isEqualTo(1);
assertThat(RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER)).hasSize(1);
QRecord changed = row(SecuredDataTables.WIPE_MARKER, marker.getValue("id"));
assertThat(changed.getValue("run_id")).isEqualTo(second.getValue("id"));
assertThat(changed.getValueInstant("wiped_at")).isEqualTo(again);
assertChanged(changed, SecuredDataTables.WIPE_MARKER, wiped, wiped);
assertDates(row(SecuredDataTables.WIPE_RUN, run.getValue("id")), SecuredDataTables.WIPE_RUN, wiped, wiped);
assertDates(row(SecuredDataTables.WIPE_RUN, second.getValue("id")), SecuredDataTables.WIPE_RUN, again, again);
assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleRun);
}
/*******************************************************************************
** The access-log writer runs concurrently with reveals, so the dates must cost
** it nothing: for each row it still runs one insert, which now names the two
** columns, and no other statement that writes the log.
*******************************************************************************/
@Test
void theAccessLogWriterStillRunsOneInsertForARow() throws Exception
{
RetentionTestSupport.setup();
try(Connection connection = DriverManager.getConnection(url(), "sa", ""))
{
connection.createStatement().execute("SET QUERY_STATISTICS TRUE");
for(int i = 0; i < 3; i++)
{
SecuredDataRuntime.current().logger().onAccess(new FieldAccessEvent().withType(FieldAccessEventType.EXACT_MATCH_FILTER).withOutcome(FieldAccessOutcome.MATCH)
.withTableName("reading").withFieldNames(List.of("secret")).withActionName("query").withRecordCount(1).withSearchValueFingerprints(List.of("hmac" + i)));
}
SecuredDataRuntime.current().logger().flush();
assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).hasSize(3).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull().isEqualTo(row.get("modify_date")));
Map<String, Integer> writes = new LinkedHashMap<>();
try(ResultSet statistics = connection.createStatement().executeQuery("SELECT SQL_STATEMENT, EXECUTION_COUNT FROM INFORMATION_SCHEMA.QUERY_STATISTICS"))
{
while(statistics.next())
{
String sql = statistics.getString(1).strip();
if(sql.contains(SecuredDataTables.ACCESS_LOG) && !sql.toUpperCase().startsWith("SELECT"))
{
writes.merge(sql, statistics.getInt(2), Integer::sum);
}
}
}
assertThat(writes).hasSize(1);
assertThat(writes.keySet().iterator().next()).startsWithIgnoringCase("INSERT INTO").contains("create_date", "modify_date");
assertThat(writes.values()).containsExactly(3);
}
}
/*******************************************************************************
** A changed policy or classification writes what it wrote before the tables
** had the standard dates: its core audit, field by field, and its
** POLICY_CHANGE or CLASSIFICATION_CHANGE row in the access log.
*******************************************************************************/
@Test
void aChangedPolicyOrClassificationWritesItsAuditAndItsLogRowAsBefore() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
QRecord policy = new InsertAction().execute(new InsertInput(SecuredDataTables.POLICY).withRecord(policyRow("viewer", "PHI", "MASKED"))).getRecords().getFirst();
QRecord classification = new InsertAction().execute(new InsertInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("table_name", "reading")
.withValue("field_name", "notes").withValue("classification", "PHI"))).getRecords().getFirst();
assertThat(policy.getErrors()).isNullOrEmpty();
assertThat(classification.getErrors()).isNullOrEmpty();
CLOCK.now = LATER;
assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.POLICY).withRecord(new QRecord().withValue("id", policy.getValue("id")).withValue("tier", "HIDDEN")))
.getRecords().getFirst().getErrors()).isNullOrEmpty();
assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("id", classification.getValue("id"))
.withValue("classification", "PII"))).getRecords().getFirst().getErrors()).isNullOrEmpty();
new DeleteAction().execute(new DeleteInput(SecuredDataTables.POLICY).withPrimaryKeys(List.of(policy.getValue("id"))));
new DeleteAction().execute(new DeleteInput(SecuredDataTables.CLASSIFICATION).withPrimaryKeys(List.of(classification.getValue("id"))));
///////////////////////////////////////////////////////////////////////////
// the log rows: one per change, with every value they have always had //
///////////////////////////////////////////////////////////////////////////
List<String> changes = new ArrayList<>();
List<QRecord> logged = RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG);
Instant inserted = logged.getFirst().getValueInstant("created_at");
for(QRecord row : logged)
{
changes.add(String.join(" ", row.getValueString("event_type"), row.getValueString("table_name"), row.getValueString("record_id"), row.getValueString("action_name"),
row.getValueString("outcome"), row.getValueString("user_id"), row.getValueString("user_name"),
"after " + Duration.between(inserted, row.getValueInstant("created_at")).toSeconds() + " s", String.valueOf(row.getValueBoolean("anomaly_flag"))));
for(String empty : List.of("field_name", "record_count", "reason", "reason_text", "client_channel", "break_glass_id", "search_value_fingerprints", "rule_id"))
{
assertThat(row.getValue(empty)).as(empty).isNull();
}
}
String policyId = policy.getValueString("id");
String classificationId = classification.getValueString("id");
assertThat(changes).containsExactly(
"POLICY_CHANGE secured_data_policy " + policyId + " INSERT ALLOWED admin admin after 0 s false",
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " INSERT ALLOWED admin admin after 0 s false",
"POLICY_CHANGE secured_data_policy " + policyId + " UPDATE ALLOWED admin admin after 90 s false",
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " UPDATE ALLOWED admin admin after 90 s false",
"POLICY_CHANGE secured_data_policy " + policyId + " DELETE ALLOWED admin admin after 90 s false",
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " DELETE ALLOWED admin admin after 90 s false");
//////////////////////////////////////////////////////////////////////////////
// the core audits of the two rows: the same audits with the same details, //
// the two dates among them under the names they have always been audited by //
//////////////////////////////////////////////////////////////////////////////
assertThat(audits(instance, SecuredDataTables.POLICY, policy.getValue("id"))).containsExactly(
"Record was Inserted: allow_free_text_reason [Set allow free text reason to Yes], classification [Set classification to PHI], create_date,"
+ " distinct_value_alert [Set distinct value alert to 15], exact_match_per_hour [Set exact match per hour to 20], id [Set Id to " + policyId + "],"
+ " is_active [Set is active to Yes], min_entropy_enabled [Set min entropy enabled to Yes], miss_alert_consecutive [Set miss alert consecutive to 5],"
+ " modified_by [Set modified by to admin], modify_date, name [Set name to viewer PHI], permission_name [Set permission name to viewer],"
+ " require_free_text [Set require free text to No], reveal_minutes [Set reveal minutes to 5], tier [Set tier to MASKED]",
"Record was Edited: modify_date, tier [Changed tier from \"MASKED\" to \"HIDDEN\"]",
"Record was Deleted: ");
assertThat(audits(instance, SecuredDataTables.CLASSIFICATION, classification.getValue("id"))).containsExactly(
"Record was Inserted: classification [Set classification to PHI], create_date, field_name [Set field name to notes], id [Set Id to " + classificationId + "],"
+ " modified_by [Set modified by to admin], modify_date, table_name [Set table name to Reading]",
"Record was Edited: classification [Changed classification from \"PHI\" to \"PII\"], modify_date",
"Record was Deleted: ");
}
/*******************************************************************************
** The core audits of one record, oldest first: the message of each and, in
** the order of the field names, the fields it has a detail for and what the
** detail says. The two dates are named only: how a time reads depends on the
** zone of the machine.
*******************************************************************************/
private static List<String> audits(QInstance instance, String table, Serializable id)
{
Serializable auditTable = SystemWrites.query("auditTable", new QQueryFilter().withCriteria(new QFilterCriteria("name", QCriteriaOperator.EQUALS, table))).getFirst().getValue("id");
List<String> audits = new ArrayList<>();
for(QRecord audit : SystemWrites.query("audit", new QQueryFilter().withCriteria(new QFilterCriteria("auditTableId", QCriteriaOperator.EQUALS, auditTable))
.withCriteria(new QFilterCriteria("recordId", QCriteriaOperator.EQUALS, id)).withOrderBy(new QFilterOrderBy("id", true))))
{
List<String> fields = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("auditId", QCriteriaOperator.EQUALS, audit.getValue("id"))))
.stream().map(detail -> detail.getValueString("fieldName") + (detail.getValueString("fieldName").endsWith("_date") ? ""
: " [" + detail.getValueString("message") + "]")).sorted().toList();
audits.add(audit.getValueString("message") + ": " + String.join(", ", fields));
}
assertThat(instance.getTable(table)).isNotNull();
return (audits);
}
/*******************************************************************************
** Rows written before the two columns existed. A reading is audited (its
** audit token is made with a data key of its own, and that key is stored
** wrapped, bound to its record and field), the log has rows, a rule and a
** hold exist. The two columns are then taken away, which leaves the database
** as the release before this one left it, and a key and a log row are
** written the way that release wrote them. The block an existing database
** runs must change nothing but the two columns: the same rows, the same
** wrapped keys, which still unwrap to the same data keys, and the same
** token, which still verifies the value and no other.
*******************************************************************************/
@Test
void rowsWrittenBeforeTheDatesExistedVerifyUnchangedAfterTheyAreAdded() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
QTableMetaData reading = instance.getTable("reading");
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
QRecord rule = RetentionTestSupport.rule("NULL_WITH_MARKER", "systolic");
insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", "404").withValue("reason", "Preservation"));
Serializable id = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 120).withValue("secret", "s")
.withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id");
Serializable wiped = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 130).withValue("secret", "s")
.withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id");
new WipeService().batch(rule.getValue("id"), List.of(wiped));
SecuredDataKeySource keys = SecuredDataRuntime.current().keys();
byte[] dataKey = keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key();
List<String> tokens = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("fieldName", QCriteriaOperator.EQUALS, "systolic"))).stream()
.map(detail -> detail.getValueString("newValue")).filter(value -> value != null && value.startsWith("shred:v1:")).toList();
String token = tokens.stream().filter(candidate -> ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, candidate)).findFirst().orElseThrow();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse();
List<String> tables = declared(instance).stream().map(QTableMetaData::getName).filter(name -> !SNAKE_CASE_NAMES.contains(name)).toList();
byte[] earlierKey = new byte[32];
Arrays.fill(earlierKey, (byte) 7);
Instant earlier = NOW.minusSeconds(86400);
try(Connection connection = DriverManager.getConnection(url(), "sa", ""))
{
for(String table : tables)
{
assertThat(rows(connection, table, false)).as(table).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull());
connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN create_date");
connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN modify_date");
}
try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_audit_key (table_name, record_id, field_name, wrapped_key, created_at) VALUES ('reading', '9001', 'systolic', ?, ?)"))
{
insert.setString(1, Base64.getEncoder().encodeToString(SecuredDataKeySource.encrypt(earlierKey, "reading\u001fsystolic\u001f9001")));
insert.setTimestamp(2, Timestamp.from(earlier));
insert.executeUpdate();
}
try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_access_log (event_type, user_id, table_name, field_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u1', 'reading', 'systolic', 'ALLOWED', false, ?)"))
{
insert.setTimestamp(1, Timestamp.from(earlier));
insert.executeUpdate();
}
Map<String, List<Map<String, String>>> before = new LinkedHashMap<>();
for(String table : tables)
{
before.put(table, rows(connection, table, false));
}
assertThat(before.get(SecuredDataTables.ACCESS_LOG)).hasSizeGreaterThan(3);
assertThat(before.get(SecuredDataTables.AUDIT_KEY)).hasSize(2);
for(String table : List.of(SecuredDataTables.WIPE_RUN, SecuredDataTables.WIPE_MARKER, SecuredDataTables.LEGAL_HOLD))
{
assertThat(before.get(table)).as(table).hasSize(1);
}
run(connection, ddl(true));
for(String table : tables)
{
assertThat(rows(connection, table, true)).as(table).isEqualTo(before.get(table));
assertThat(rows(connection, table, false)).as(table).allSatisfy(row ->
{
assertThat(row.get("create_date")).isNotNull();
assertThat(row.get("modify_date")).isNotNull();
});
}
assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).allSatisfy(row ->
{
assertThat(row.get("create_date")).isEqualTo(row.get("created_at"));
assertThat(row.get("modify_date")).isEqualTo(row.get("created_at"));
});
assertThat(rows(connection, SecuredDataTables.AUDIT_KEY, false)).allSatisfy(row ->
{
assertThat(row.get("create_date")).isEqualTo(row.get("created_at"));
assertThat(row.get("modify_date")).isEqualTo(row.get("created_at"));
});
/////////////////////////////////////////////////////////////////////////////
// the release before this one goes on writing while both run: its insert //
// names neither column, and the row has both, equal, all the same //
/////////////////////////////////////////////////////////////////////////////
connection.createStatement().execute("INSERT INTO secured_data_access_log (event_type, user_id, table_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u2', 'reading', 'ALLOWED', false, CURRENT_TIMESTAMP)");
Map<String, String> concurrent = rows(connection, SecuredDataTables.ACCESS_LOG, false).getLast();
assertThat(concurrent.get("user_id")).isEqualTo("u2");
assertThat(concurrent.get("create_date")).isNotNull().isEqualTo(concurrent.get("modify_date"));
}
///////////////////////////////////////////////////////////////////////////
// nothing kept in memory: every key is unwrapped again from its row //
///////////////////////////////////////////////////////////////////////////
keys.close();
assertThat(keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key()).isEqualTo(dataKey);
assertThat(keys.keyForRecordField("reading", "systolic", 9001, false).orElseThrow().key()).isEqualTo(earlierKey);
assertThat(keys.keyForRecordField("reading", "systolic", wiped, false)).isEmpty();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isTrue();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse();
////////////////////////////////////////////////////////////////////////////
// and the rows that were there are read and written on as they were: the //
// earlier log row is counted, and the reading is wiped under its rule //
////////////////////////////////////////////////////////////////////////////
assertThat(RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG)).anySatisfy(row ->
{
assertThat(row.getValueString("user_id")).isEqualTo("u1");
assertThat(row.getValueInstant("createDate")).isNotNull().isEqualTo(row.getValueInstant("created_at"));
assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("created_at"));
});
CLOCK.now = LATER;
assertThat(new WipeService().batch(rule.getValue("id"), List.of(id)).getValueInteger("wiped_count")).isEqualTo(1);
assertThat(keys.keyForRecordField("reading", "systolic", id, false)).isEmpty();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isFalse();
}
/*******************************************************************************
** The database RetentionTestSupport made from the shipped DDL.
*******************************************************************************/
private static String url()
{
return ("jdbc:h2:mem:" + RetentionTestSupport.BACKEND + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1");
}
/*******************************************************************************
** The statements of the shipped DDL: those of the block an existing database
** runs to get the two columns, or those before it (the schema without them).
*******************************************************************************/
static List<String> ddl(boolean datesBlock) throws IOException
{
String ddl = Files.readString(Path.of(DDL));
int block = ddl.indexOf(DDL_BLOCK);
assertThat(block).as("the block of " + DDL + " that begins " + DDL_BLOCK).isPositive().isEqualTo(ddl.lastIndexOf(DDL_BLOCK));
return (Arrays.stream((datesBlock ? ddl.substring(block) : ddl.substring(0, block)).split(";")).filter(sql -> !sql.isBlank()).toList());
}
/*******************************************************************************
** Run statements.
*******************************************************************************/
static void run(Connection connection, List<String> statements) throws SQLException
{
for(String sql : statements)
{
connection.createStatement().execute(sql);
}
}
/*******************************************************************************
** Every row of a table as the database holds it, column by column, in id
** order, with or without the two date columns.
*******************************************************************************/
static List<Map<String, String>> rows(Connection connection, String table, boolean withoutDates) throws SQLException
{
List<Map<String, String>> rows = new ArrayList<>();
try(ResultSet results = connection.createStatement().executeQuery("SELECT * FROM " + table + " ORDER BY id"))
{
ResultSetMetaData columns = results.getMetaData();
while(results.next())
{
Map<String, String> row = new LinkedHashMap<>();
for(int i = 1; i <= columns.getColumnCount(); i++)
{
String column = columns.getColumnName(i);
if(!withoutDates || !Set.of("create_date", "modify_date").contains(column))
{
row.put(column, results.getString(i));
}
}
rows.add(row);
}
}
return (rows);
}
/*******************************************************************************
** One stored row.
*******************************************************************************/
private static QRecord row(String table, Serializable id)
{
return (SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id))).getFirst());
}
/*******************************************************************************
** The newest log row of an event, and of an action where one is given.
*******************************************************************************/
private static QRecord log(String event, String action)
{
return (logs().stream().filter(row -> event.equals(row.getValueString("event_type")) && (action == null || action.equals(row.getValueString("action_name"))))
.reduce((first, second) -> second).orElseThrow());
}
/*******************************************************************************
** Both dates of a stored row.
*******************************************************************************/
private static void assertDates(QRecord row, String table, Instant created, Instant modified)
{
assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created);
assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isEqualTo(modified);
}
/*******************************************************************************
** Both dates of a stored row that was changed: the create date it had, and a
** modify date later than the one it had, which is returned.
*******************************************************************************/
private static Instant assertChanged(QRecord row, String table, Instant created, Instant modifiedBefore)
{
assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created);
assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isAfter(modifiedBefore);
return (row.getValueInstant(modifyDate(table)));
}
}