fix: a reveal and the access-log writer no longer wait for each other (release/0.1 port of #45) #46
Open
james.maes
wants to merge 1 commits from
fix/reveal-access-log-deadlock-0.1 into release/0.1
pull from: fix/reveal-access-log-deadlock-0.1
merge into: QRun-IO:release/0.1
QRun-IO:develop
QRun-IO:fix/reveal-access-log-deadlock
QRun-IO:release/0.1
QRun-IO:fix/reentrant-rule-reload-0.1
QRun-IO:fix/reentrant-rule-reload
QRun-IO:merge/develop-into-release-0.1-rc6
QRun-IO:perf/request-scoped-resolver-and-cached-rules
QRun-IO:main
QRun-IO:dependabot/maven/develop/maven-minor-and-patch-47b49de09a
QRun-IO:dependabot/github_actions/develop/QRun-IO/qqq-orb/dot-github/workflows/qbit-codeql.yml-1c389210e6dfda1630d053c07ae4dae523dd5576
QRun-IO:dependabot/github_actions/develop/QRun-IO/qqq-orb/dot-github/workflows/qbit-release-rc.yml-1c389210e6dfda1630d053c07ae4dae523dd5576
QRun-IO:feature/orb-0.8-track
QRun-IO:feature/orb-0.7.18
QRun-IO:feature/orb-0.7.17
QRun-IO:feature/orb-standardize-release-0.1
QRun-IO:feature/orb-main-standardization
QRun-IO:feature/shared-codeql
QRun-IO:feature/orb-0712-stable-gate
QRun-IO:feature/orb-rc-worker
QRun-IO:feature/orb-0.7.11
QRun-IO:feature/central-ci
QRun-IO:feature/access-log-lifecycle
QRun-IO:feature/virtual-fields-and-idle-flush
QRun-IO:dependabot/maven/develop/com.puppycrawl.tools-checkstyle-14.3.0
QRun-IO:ci/release-rc-secrets-inherit
QRun-IO:feature/ui-api-1.6
QRun-IO:feature/remove-standins
QRun-IO:feature/orb-0.6.9
QRun-IO:feature/retention-and-wipe
QRun-IO:feature/app-home
QRun-IO:feature/reveal-and-break-glass
QRun-IO:feature/policy-and-access-log
QRun-IO:fix/pin-qqq-rc
QRun-IO:feature/scaffold
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c2fde973a5
|
fix: a reveal and the access-log writer no longer wait for each other
A reveal, and the start, end and expiry of a break-glass session, hold the
runtime's monitor from their first check to their last write. Asking for a
classification (SecuredDataRuntime.classificationRows, and
activeRetentionRules) took the same monitor, and every thread asks: core
asks for every field it decides on and for every filter criterion of every
query, including the queries made inside this QBit's own writes.
So a thread that asked while holding another lock of this QBit could not be
answered during a reveal, and if the reveal needed that lock next, both
waited for ever. The access-log writer is such a thread. It holds the
anomaly counters while it inserts a row; the insert counts earlier rows and
writes an audit, both queries with criteria. A reveal inserts its own
access-log row.
reveal: runtime's monitor, then the anomaly counters
writer: the anomaly counters, then the runtime's monitor
Seen in a host on PostgreSQL: a person read a record with a masked field and
revealed it at once; the writer's tick drained the read's log rows at the
same moment. The request and the writer deadlocked (thread dump: the request
in SystemWrites.insertOne under RevealService.reveal, the writer in
SecuredDataRuntime.classificationRows under AuditAction.fetchIdFromName
under its own insert), and every later request blocked in
classificationRows. The same order existed between a reveal and the
resolver's monitor, and the key source's.
- The rule rows have a lock of their own (ruleLock), held only while they
are answered or read. A read is a query made as the system and takes none
of the QBit's other locks, so nothing that holds ruleLock waits for one.
- The runtime's monitor is left to what it was for: one reveal or
break-glass change at a time.
Tests (SecuredDataLockOrderTest, on H2 in PostgreSQL mode with the audit
tables): a thread is parked inside a reveal, and inside the start of a
break-glass session, holding the runtime's monitor; the access-log writer
must drain its queue and another thread must be told a classification and
the active retention rules meanwhile, and then the parked operation must
finish. Both tests fail without the fix (neither thread finishes).
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|