Every table of the QBit has a create date and a modify date: DATE_TIME, not editable, with core's CREATE_DATE and MODIFY_DATE behaviors, in the columns create_date and modify_date. Seven tables did not have them: reveal grant, break-glass session, access log, wipe run, wipe marker, legal hold and audit key. Their fields are named createDate and modifyDate. Classification, policy and retention rule already had create_date and modify_date, set by the table customizer. They keep those field names, because a rename would change what a host reads and what their audits record, and the two fields are now not editable and have the behaviors. Writes. Every write of the QBit is a core insert or update action, so the dates are in the statement that writes the row. The table customizer sets them, as it did for the three tables: a new row gets both, equal, from the QBit's clock, and a changed row a new modify date. The access-log writer still runs one insert per row and no other statement, lock or transaction. The access log and wipe runs are only ever inserted, so their dates stay equal. Core uses a DATE_TIME field with the MODIFY_DATE behavior as the record version of its table. That now holds for all ten tables: on an update core sets the modify date itself, after the customizer has run, and an update that sends the version it read is refused when the row has changed since. Updates that send no version, the QBit's own among them, behave as before. Schema. docs/ddl/secured-data-postgres.sql has a new last block that adds the two columns to the seven tables: nullable, defaulting to the current time, with the rows that were already there given the row's own time. No CREATE TABLE statement is edited. An existing database runs that block alone, before this release is deployed. The block rewrites no table, can be run while the release before this one is running (its inserts name neither column and get both from the default) and can be run again. This release does not run without the columns. README: "Upgrading an existing database". Unchanged. The dates have no classification, policy or assistant exposure of their own: they are CONFIDENTIAL with their table and get the tier of its other fields. Nothing signed or verified covers them: a wrapped data key is bound to its table, field and record id, and the fingerprints of a log row are made from search values. A changed policy or classification writes the same core audit, field by field, and the same POLICY_CHANGE or CLASSIFICATION_CHANGE row. Tests. SecuredDataStandardDatesTest: every table the producer declares has the two dates (the tables are taken from the instance, so a table added later without them fails); the dates resolve to the tier of the other fields of their table for five kinds of session; every writer sets both and only a change moves the modify date (access-log writer, policy, classification, legal hold, reveal, break-glass start, end and expiry on the memory backend; audit key, wipe run, wipe marker and retention rule in the shipped schema); the access-log writer runs exactly one insert per row; a changed policy and classification write the audit and log rows they wrote before; rows written before the columns existed are unchanged by the upgrade block, their wrapped keys unwrap to the same data keys and their audit token verifies the same value and no other. SecuredDataDdlTest: the shipped DDL gives every declared table the two columns, the upgrade block adds them where they were missing, an insert that names neither succeeds, and the block can be run twice. Seven of the eight new tests fail without the change; the eighth is the record of what a changed policy wrote before and passes on both sides. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
190 lines
11 KiB
SQL
190 lines
11 KiB
SQL
-- PostgreSQL 15+ (NULLS NOT DISTINCT makes the table-wide classification unique).
|
|
-- Extra value-free fingerprint column supports enumeration detection across restarts/nodes.
|
|
|
|
CREATE TABLE secured_data_classification (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
|
table_name VARCHAR(255) NOT NULL,
|
|
field_name VARCHAR(255),
|
|
classification VARCHAR(255) NOT NULL,
|
|
mask_style VARCHAR(255),
|
|
low_cardinality BOOLEAN,
|
|
notes TEXT,
|
|
create_date TIMESTAMP WITH TIME ZONE,
|
|
modify_date TIMESTAMP WITH TIME ZONE,
|
|
modified_by VARCHAR(255),
|
|
CONSTRAINT secured_data_classification_target UNIQUE NULLS NOT DISTINCT (table_name, field_name)
|
|
);
|
|
|
|
CREATE TABLE secured_data_policy (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
|
name VARCHAR(255) NOT NULL,
|
|
permission_name VARCHAR(255) NOT NULL,
|
|
classification VARCHAR(255) NOT NULL,
|
|
table_name VARCHAR(255),
|
|
tier VARCHAR(255) NOT NULL,
|
|
reveal_minutes INTEGER DEFAULT 5,
|
|
daily_reveal_cap INTEGER,
|
|
reason_options TEXT,
|
|
allow_free_text_reason BOOLEAN DEFAULT true,
|
|
require_free_text BOOLEAN DEFAULT false,
|
|
client_channel VARCHAR(255),
|
|
exact_match_per_hour INTEGER DEFAULT 20,
|
|
miss_alert_consecutive INTEGER DEFAULT 5,
|
|
distinct_value_alert INTEGER DEFAULT 15,
|
|
min_entropy_enabled BOOLEAN DEFAULT true,
|
|
is_active BOOLEAN DEFAULT true,
|
|
create_date TIMESTAMP WITH TIME ZONE,
|
|
modify_date TIMESTAMP WITH TIME ZONE,
|
|
modified_by VARCHAR(255),
|
|
CHECK (reveal_minutes BETWEEN 1 AND 60),
|
|
CHECK (exact_match_per_hour >= 0 AND miss_alert_consecutive >= 0 AND distinct_value_alert >= 0)
|
|
);
|
|
|
|
CREATE TABLE secured_data_reveal_grant (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
|
user_id VARCHAR(255) NOT NULL,
|
|
user_name VARCHAR(255),
|
|
table_name VARCHAR(255) NOT NULL,
|
|
record_id VARCHAR(255) NOT NULL,
|
|
field_name VARCHAR(255) NOT NULL,
|
|
reason VARCHAR(500),
|
|
reason_text TEXT,
|
|
granted_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
|
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
|
access_log_id INTEGER NOT NULL
|
|
);
|
|
|
|
CREATE TABLE secured_data_break_glass (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
|
user_id VARCHAR(255) NOT NULL,
|
|
user_name VARCHAR(255),
|
|
reason VARCHAR(500),
|
|
reason_text TEXT,
|
|
classification_ceiling VARCHAR(255) DEFAULT 'PHI',
|
|
table_name VARCHAR(255),
|
|
started_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
|
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
|
ended_at TIMESTAMP WITH TIME ZONE,
|
|
ended_by VARCHAR(255),
|
|
status VARCHAR(255) DEFAULT 'ACTIVE'
|
|
);
|
|
|
|
CREATE TABLE secured_data_access_log (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
|
event_type VARCHAR(255) NOT NULL,
|
|
user_id VARCHAR(255),
|
|
user_name VARCHAR(255),
|
|
table_name VARCHAR(255) NOT NULL,
|
|
record_id TEXT,
|
|
field_name TEXT,
|
|
record_count INTEGER,
|
|
action_name VARCHAR(255),
|
|
reason VARCHAR(500),
|
|
reason_text TEXT,
|
|
client_channel VARCHAR(255),
|
|
outcome VARCHAR(255),
|
|
anomaly_flag BOOLEAN DEFAULT false,
|
|
break_glass_id INTEGER,
|
|
created_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
|
search_value_fingerprints TEXT,
|
|
rule_id INTEGER
|
|
);
|
|
|
|
CREATE INDEX secured_data_policy_lookup ON secured_data_policy (is_active, permission_name, classification, table_name);
|
|
CREATE INDEX secured_data_break_glass_lookup ON secured_data_break_glass (user_id, status, expires_at);
|
|
CREATE INDEX secured_data_reveal_lookup ON secured_data_reveal_grant (user_id, table_name, field_name, record_id, expires_at);
|
|
CREATE INDEX secured_data_access_lookup ON secured_data_access_log (user_id, table_name, field_name, event_type, created_at);
|
|
|
|
-- Phase (e): bounded user counters and home time windows.
|
|
CREATE INDEX secured_data_anomaly_lookup ON secured_data_access_log (user_id, event_type, created_at);
|
|
CREATE INDEX secured_data_home_lookup ON secured_data_access_log (event_type, created_at);
|
|
|
|
|
|
-- Phase (f): all policy targets and audit tables must share this transactional backend.
|
|
CREATE TABLE secured_data_retention_rule (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
|
name VARCHAR(255) NOT NULL, table_name VARCHAR(255) NOT NULL, field_names TEXT NOT NULL,
|
|
age_field_name VARCHAR(255) NOT NULL, age_days INTEGER NOT NULL CHECK (age_days >= 1),
|
|
action VARCHAR(40) DEFAULT 'NULL_WITH_MARKER' NOT NULL CHECK (action IN ('NULL_WITH_MARKER','OVERWRITE_TEXT','KEYED_HASH','DELETE_RECORD','ARCHIVE_THEN_WIPE')),
|
|
overwrite_text TEXT DEFAULT '[wiped per policy]',
|
|
audit_treatment VARCHAR(30) DEFAULT 'SHREDDABLE' NOT NULL CHECK (audit_treatment IN ('SHREDDABLE','CHANGED_ONLY')),
|
|
archive_storage_name VARCHAR(255), batch_size INTEGER DEFAULT 500 NOT NULL CHECK (batch_size >= 1),
|
|
is_active BOOLEAN DEFAULT true, last_run_at TIMESTAMP WITH TIME ZONE, last_run_count INTEGER,
|
|
create_date TIMESTAMP WITH TIME ZONE, modify_date TIMESTAMP WITH TIME ZONE, modified_by VARCHAR(255)
|
|
);
|
|
CREATE TABLE secured_data_wipe_run (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, rule_id INTEGER NOT NULL REFERENCES secured_data_retention_rule(id),
|
|
started_at TIMESTAMP WITH TIME ZONE NOT NULL, finished_at TIMESTAMP WITH TIME ZONE, status VARCHAR(30),
|
|
eligible_count INTEGER, held_count INTEGER, wiped_count INTEGER, error_count INTEGER,
|
|
record_ids VARCHAR(4000), message VARCHAR(500)
|
|
);
|
|
CREATE TABLE secured_data_wipe_marker (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL, record_id VARCHAR(255) NOT NULL,
|
|
field_name VARCHAR(255) NOT NULL, rule_id INTEGER NOT NULL REFERENCES secured_data_retention_rule(id), action VARCHAR(40) NOT NULL,
|
|
wiped_at TIMESTAMP WITH TIME ZONE NOT NULL, run_id INTEGER NOT NULL REFERENCES secured_data_wipe_run(id),
|
|
UNIQUE (table_name, record_id, field_name)
|
|
);
|
|
CREATE TABLE secured_data_legal_hold (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL, record_id VARCHAR(255), reason TEXT NOT NULL,
|
|
placed_by VARCHAR(255), placed_at TIMESTAMP WITH TIME ZONE, released_by VARCHAR(255), released_at TIMESTAMP WITH TIME ZONE,
|
|
status VARCHAR(20) DEFAULT 'ACTIVE' NOT NULL CHECK (status IN ('ACTIVE','RELEASED'))
|
|
);
|
|
-- Mutable key material: never register this table in ESB, WORM or long-term logical backups.
|
|
CREATE TABLE secured_data_audit_key (
|
|
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL,
|
|
record_id VARCHAR(255) NOT NULL, field_name VARCHAR(255) NOT NULL, wrapped_key TEXT NOT NULL,
|
|
created_at TIMESTAMP WITH TIME ZONE NOT NULL, UNIQUE (table_name, record_id, field_name)
|
|
);
|
|
CREATE INDEX secured_data_hold_lookup ON secured_data_legal_hold (table_name, status, record_id);
|
|
CREATE INDEX secured_data_retention_lookup ON secured_data_retention_rule (table_name, is_active);
|
|
CREATE INDEX secured_data_wipe_rule_lookup ON secured_data_wipe_marker (rule_id);
|
|
|
|
-- Standard dates: create_date and modify_date on the seven tables created above without them.
|
|
-- A new database runs the whole file. An existing database runs this block alone, BEFORE the release
|
|
-- that writes the two columns is deployed. The columns are nullable and default to the current time, so
|
|
-- a release that does not know them goes on inserting while both run. Adding a nullable column and
|
|
-- setting a default change the catalog only (no table rewrite). Each UPDATE gives the rows that were
|
|
-- already there the row's own time, once: it touches only rows whose create_date is still null, takes
|
|
-- row locks only, and on a large access log can be run in ranges of id.
|
|
ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_reveal_grant ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_reveal_grant ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_reveal_grant SET create_date = granted_at, modify_date = granted_at WHERE create_date IS NULL;
|
|
|
|
ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_break_glass ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_break_glass ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_break_glass SET create_date = started_at, modify_date = COALESCE(ended_at, started_at) WHERE create_date IS NULL;
|
|
|
|
ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_access_log ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_access_log ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_access_log SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL;
|
|
|
|
ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_wipe_run ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_wipe_run ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_wipe_run SET create_date = COALESCE(finished_at, started_at), modify_date = COALESCE(finished_at, started_at) WHERE create_date IS NULL;
|
|
|
|
ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_wipe_marker ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_wipe_marker ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_wipe_marker SET create_date = wiped_at, modify_date = wiped_at WHERE create_date IS NULL;
|
|
|
|
ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_legal_hold ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_legal_hold ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_legal_hold SET create_date = COALESCE(placed_at, CURRENT_TIMESTAMP), modify_date = COALESCE(released_at, placed_at, CURRENT_TIMESTAMP) WHERE create_date IS NULL;
|
|
|
|
ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
|
|
ALTER TABLE secured_data_audit_key ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
ALTER TABLE secured_data_audit_key ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
|
|
UPDATE secured_data_audit_key SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL;
|