perf: skip tag lookups for tables with no tag assignments #67

2026-11-10T21:26:05Z

james.maes commented

2026-11-10 21:26:05 +00:00

Problem

Hosts that enable TAGS on every table (GG Forge does: all record features are on by default) pay one collaboration_record_tag statement on almost every read of every table, whether or not anything was ever tagged, and whether or not the UI API is on.

Why: TagsMetaDataProvider adds the selectable virtual field collaborationTags to every table with TAGS enabled. Core (VirtualFieldValueProviders.selected) selects every selectable provider field when a query or get names no fields, and calls the provider once per QueryAction result page and once per GetAction (QueryAction.postRecordActions, GetAction.postRecordActions). TagValueProvider.fill then queried collaboration_record_tag unconditionally. The UI API flag only registers HTTP endpoints; it has no bearing on this field.

So the lookup is already batched for one list query (one statement per 500 parents), but every per-record read made by anything else (a get, the stored-record read before an update, a per-row lookup in another qbit or a job) drags one more statement with it. That matches what Forge's Database Performance tables show: collaboration_record_tag counts that track the number of per-record reads of other tables.

Change

TaggedTables (new): each node remembers, per table, whether the table has any tag assignment. TagValueProvider skips the lookup for a table known to have none. A table that has assignments is read exactly as before (one assignment lookup and one vocabulary lookup per 500 parents); record tags themselves are never cached.
ParentRowCustomizer and AttachmentPreviewCustomizer: when one of the qbit's child tables is browsed, each distinct parent is read once per result page to authorize its rows, instead of once per row. Same check, same outcome.

No API change except the new optional CollaborationQBitConfig.withTagLookupCacheSeconds (default 30, 0 disables). No schema change.

Measured (TagLookupStatementBudgetTest, memory backend; one backend query is one SQL statement on an RDBMS backend)

600 added parents, table never tagged, statements against this qbit's tables:

Read	Before (1.2.0-RC.2)	After
One list of 602 rows	2	1 the first time (the probe), then 0
600 gets	600	0
100 single-record updates	100	0

Tagged table, list of 602 rows: 2 assignment + 2 vocabulary statements before and after (plus 1 probe when the node has to ask again).

Browsing 61 tasks on two parents: 61 parent reads before, 2 after.

The cache, stated

Mechanism: core Memoization per QInstance, registered with core ClusterCaches as collaboration.taggedTables. Not qbit-redis: a Redis round trip per read would cost what the statement it replaces costs, and the fact cached is tiny and safe to hold per node.
What is cached: one boolean per table name. Bounded at 10,000 entries per instance (there cannot be more entries than tables).
Learned by: one SELECT id ... WHERE table_name = ? LIMIT 1, served by the existing unique index (table_name, record_id, tag_id).
Security: the probe runs on a private metadata copy without row locks, parent checks or customizers and returns no data, so the answer is the same for every session. A test proves that a caller who cannot see a table's only tagged record does not hide its tags from one who can. When the answer is "has assignments", the unchanged, lock-enforcing query runs.
Invalidation: any insert or update of collaboration_record_tag through QQQ (CollaborationRealtime, the listener that already publishes tags.changed) clears the table's entry on the node at once, so the writing transaction reads its own tag, and again after commit, so a probe that ran in between cannot outlive the commit. Both publish core's memo-invalidate cluster event.
Several nodes: with a ClusterBus transport the other nodes clear immediately. Without one (events stay on the node), another node shows a table's very first tag after at most the timeout. Only the "no assignments" answer can be stale, and only for a table going from zero assignments to one; later tags on that table show at once. The same bound covers writes made outside QQQ and restores that suppress change events.
Failure: a failed probe is thrown, never remembered, exactly as a failed assignment query was before. withTagLookupCacheSeconds(0) restores a lookup on every read.

Found and left alone (not on the measured path; reported for follow-up)

CustomFieldValueProvider.fill: one definition statement and one value statement per EAV custom field per result page (and per get). Only tables with active EAV fields; core's provider interface is per field.
Host deletes: CollaborationCleanup, AttachmentCleanup, WorkCleanup and TagsService.onParentsDeleted run 12 lookups after each delete call on any feature-enabled table even when there is nothing to clean (counted in the code, not measured; batched by ids, not per record). The cleanup lookups on collaboration_mention and collaboration_comment_revision filter by (table_name, record_id), which neither table indexes.
TagsService.recompute loads every assignment of a tag to count it on each apply; merge reads assignments once per link.
CollaborationRecords.query and the providers clone table metadata per call (CPU only, no statement).

Gates

./mvnw verify (Checkstyle at validate, architecture guardrails, all tests including the Docker-backed suites, JaCoCo 70% instructions / 90% classes): see the comment below for the run on this head.

Release

Merging into release/1.2 makes the shared Maven workflow publish 1.2.0-RC.3 to the QRun-IO Gitea registry and commit the version bump. The same two commits apply unchanged to develop (the touched files are identical there) and should follow so the next line keeps the fix.

🤖 Generated with Claude Code

## Problem Hosts that enable TAGS on every table (GG Forge does: all record features are on by default) pay one `collaboration_record_tag` statement on almost every read of every table, whether or not anything was ever tagged, and whether or not the UI API is on. Why: `TagsMetaDataProvider` adds the selectable virtual field `collaborationTags` to every table with TAGS enabled. Core (`VirtualFieldValueProviders.selected`) selects every selectable provider field when a query or get names no fields, and calls the provider once per `QueryAction` result page and once per `GetAction` (`QueryAction.postRecordActions`, `GetAction.postRecordActions`). `TagValueProvider.fill` then queried `collaboration_record_tag` unconditionally. The UI API flag only registers HTTP endpoints; it has no bearing on this field. So the lookup is already batched for one list query (one statement per 500 parents), but every per-record read made by anything else (a get, the stored-record read before an update, a per-row lookup in another qbit or a job) drags one more statement with it. That matches what Forge's Database Performance tables show: `collaboration_record_tag` counts that track the number of per-record reads of other tables. ## Change 1. `TaggedTables` (new): each node remembers, per table, whether the table has any tag assignment. `TagValueProvider` skips the lookup for a table known to have none. A table that has assignments is read exactly as before (one assignment lookup and one vocabulary lookup per 500 parents); record tags themselves are never cached. 2. `ParentRowCustomizer` and `AttachmentPreviewCustomizer`: when one of the qbit's child tables is browsed, each distinct parent is read once per result page to authorize its rows, instead of once per row. Same check, same outcome. No API change except the new optional `CollaborationQBitConfig.withTagLookupCacheSeconds` (default 30, 0 disables). No schema change. ## Measured (TagLookupStatementBudgetTest, memory backend; one backend query is one SQL statement on an RDBMS backend) 600 added parents, table never tagged, statements against this qbit's tables: | Read | Before (1.2.0-RC.2) | After | | --- | --- | --- | | One list of 602 rows | 2 | 1 the first time (the probe), then 0 | | 600 gets | 600 | 0 | | 100 single-record updates | 100 | 0 | Tagged table, list of 602 rows: 2 assignment + 2 vocabulary statements before and after (plus 1 probe when the node has to ask again). Browsing 61 tasks on two parents: 61 parent reads before, 2 after. ## The cache, stated - Mechanism: core `Memoization` per `QInstance`, registered with core `ClusterCaches` as `collaboration.taggedTables`. Not qbit-redis: a Redis round trip per read would cost what the statement it replaces costs, and the fact cached is tiny and safe to hold per node. - What is cached: one boolean per table name. Bounded at 10,000 entries per instance (there cannot be more entries than tables). - Learned by: one `SELECT id ... WHERE table_name = ? LIMIT 1`, served by the existing unique index `(table_name, record_id, tag_id)`. - Security: the probe runs on a private metadata copy without row locks, parent checks or customizers and returns no data, so the answer is the same for every session. A test proves that a caller who cannot see a table's only tagged record does not hide its tags from one who can. When the answer is "has assignments", the unchanged, lock-enforcing query runs. - Invalidation: any insert or update of `collaboration_record_tag` through QQQ (`CollaborationRealtime`, the listener that already publishes `tags.changed`) clears the table's entry on the node at once, so the writing transaction reads its own tag, and again after commit, so a probe that ran in between cannot outlive the commit. Both publish core's `memo-invalidate` cluster event. - Several nodes: with a ClusterBus transport the other nodes clear immediately. Without one (events stay on the node), another node shows a table's very first tag after at most the timeout. Only the "no assignments" answer can be stale, and only for a table going from zero assignments to one; later tags on that table show at once. The same bound covers writes made outside QQQ and restores that suppress change events. - Failure: a failed probe is thrown, never remembered, exactly as a failed assignment query was before. `withTagLookupCacheSeconds(0)` restores a lookup on every read. ## Found and left alone (not on the measured path; reported for follow-up) - `CustomFieldValueProvider.fill`: one definition statement and one value statement per EAV custom field per result page (and per get). Only tables with active EAV fields; core's provider interface is per field. - Host deletes: `CollaborationCleanup`, `AttachmentCleanup`, `WorkCleanup` and `TagsService.onParentsDeleted` run 12 lookups after each delete call on any feature-enabled table even when there is nothing to clean (counted in the code, not measured; batched by ids, not per record). The cleanup lookups on `collaboration_mention` and `collaboration_comment_revision` filter by `(table_name, record_id)`, which neither table indexes. - `TagsService.recompute` loads every assignment of a tag to count it on each apply; `merge` reads assignments once per link. - `CollaborationRecords.query` and the providers clone table metadata per call (CPU only, no statement). ## Gates `./mvnw verify` (Checkstyle at validate, architecture guardrails, all tests including the Docker-backed suites, JaCoCo 70% instructions / 90% classes): see the comment below for the run on this head. ## Release Merging into `release/1.2` makes the shared Maven workflow publish `1.2.0-RC.3` to the QRun-IO Gitea registry and commit the version bump. The same two commits apply unchanged to `develop` (the touched files are identical there) and should follow so the next line keeps the fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 2 commits 2026-11-10 21:26:05 +00:00

perf(tags): no assignment lookup for tables that have no tags

maven / build (push) Successful in 7m38s

Details

build / maven (push) Successful in 7m38s

Details

50226fbabc

Core selects the Tags virtual field for every query or get that names no
fields, so every such read of a TAGS-enabled table ran one
collaboration_record_tag statement, including reads made one record at a
time and tables that were never tagged. Measured with 600 added parents
(TagLookupStatementBudgetTest): a 602-row list ran 2 statements, 600 gets
ran 600 and 100 single-record updates ran 100; now 1 probe, then 0.

TaggedTables remembers, per node and per table, whether any assignment
exists (core Memoization registered with ClusterCaches, 30 s by default,
withTagLookupCacheSeconds, 0 disables). Writing an assignment clears it at
once and after commit and publishes memo-invalidate. The probe ignores row
locks and returns no data, so the answer is the same for every session.
Tables with assignments are read as before: one lookup per 500 parents.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

perf(security): authorize each distinct parent once per child result page

maven / build (push) Successful in 6m40s

Details

build / maven (push) Successful in 6m40s

Details

b9137b3379

ParentRowCustomizer and AttachmentPreviewCustomizer read the parent of
every child row to decide whether the row is visible, so rows that share a
parent repeated the same read: 61 tasks on two parents ran 61 parent reads
(TagLookupStatementBudgetTest), now 2. The answer is kept only for the one
postQuery call, where the session and transaction cannot change.

Also correct the README: the Tags field is selectable.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

james.maes commented

2026-11-10 21:32:05 +00:00

Gates on head b9137b3 (no skip flags):

Local ./mvnw -B verify on this branch: BUILD SUCCESS in 6m10s. Checkstyle 0 violations; both architecture guardrail executions passed; 344 tests, 0 failures, 0 errors, 26 skipped (the MySQL and PostgreSQL upload-journal suites, which are opt-in through mysqlJournalTests / postgresJournalTests and skip the same way on the base branch); JaCoCo: all coverage checks met. The Docker-backed suites ran (AttachmentTransactionsPostgresTest, CustomFieldValuesPostgresTest, MyWorkPostgresTest, RedisPresenceStoreTest).
Hosted maven / build (push): success in 6m40s.

This repository's build defines no SpotBugs or PMD execution (parent qbit-build-parent 2.2.1), so those gates do not exist here.

Gates on head b9137b3 (no skip flags): - Local `./mvnw -B verify` on this branch: BUILD SUCCESS in 6m10s. Checkstyle 0 violations; both architecture guardrail executions passed; 344 tests, 0 failures, 0 errors, 26 skipped (the MySQL and PostgreSQL upload-journal suites, which are opt-in through `mysqlJournalTests` / `postgresJournalTests` and skip the same way on the base branch); JaCoCo: all coverage checks met. The Docker-backed suites ran (AttachmentTransactionsPostgresTest, CustomFieldValuesPostgresTest, MyWorkPostgresTest, RedisPresenceStoreTest). - Hosted `maven / build (push)`: success in 6m40s. This repository's build defines no SpotBugs or PMD execution (parent qbit-build-parent 2.2.1), so those gates do not exist here.

james.maes changed title from ~~WIP: perf(tags): no assignment lookup for untagged tables; one parent check per child page [release/1.2]~~ to perf: skip tag lookups for tables with no tag assignments

2026-11-10 21:39:07 +00:00

james.maes merged commit 38d54231a5 into release/1.2

2026-11-10 21:39:10 +00:00

james.maes referenced this pull request

2026-11-10 21:58:24 +00:00

perf: skip tag lookups for tables with no tag assignments (forward-port of #67) #68

james.maes referenced this issue from a commit

2026-11-10 22:11:03 +00:00

Merge pull request 'perf: skip tag lookups for tables with no tag assignments (forward-port of #67)' (#68) from fix/read-path-statements-develop into develop

Sign in to join this conversation.