Compare commits

..
Author SHA1 Message Date
james.maesandClaude Fable 5.1 fe91118908 fix(enterprise): guard column statistics by field access, drop the owner table
8979964 gave the columnStats process a hidden, unrestricted owner table
so its table-less result records would not be withheld. That also
bypassed the field-access filter on those results: a caller could read
min, max and the most common values of a field whose access tier they do
not hold in full. The owner table is gone.

The process now runs GuardedColumnStatsStep instead:
- before anything is computed, it refuses (QPermissionDeniedException) a
  field the caller cannot read in full (REVEALABLE, MASKED or HIDDEN),
  including joined fields, which are checked on their own table
- its result records are tagged with the analysed table, so the UI API
  applies that table's field-access policy to them; table-less process
  records are still withheld as before

GuardedColumnStatsStepTest covers a field behind a field security lock:
refused without the key, allowed with it, and an unrestricted field
allowed. The process shape itself belongs in qqq core (QRun-IO/qqq#1367).

Refs QRun-IO/qqq#1367

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 06:50:58 -06:00
james.maesandClaude Fable 5.1 897996401a fix(enterprise): let column statistics reach the caller
With QUERY_STATS on and the columnStats process registered, the
statistics still came back empty: the process's result records belong to
no table, and the UI API withholds such process records once any field in
the instance has a restricted access tier (qbit-secured-data, the
sample's secret fields). The process now has a hidden, unrestricted owner
table, so `qqq table stats person firstName` returns counts, min, max and
the distribution.

Refs QRun-IO/qqq#1367

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 06:37:15 -06:00
james.maesandClaude Fable 5.1 3233a885f7 feat(enterprise): back collaboration attachments with object storage on MinIO
Uploading an attachment failed with "The Object Storage qbit is not
installed": qbit-collaboration keeps attachments in object storage and
the enterprise profile had none. When S3_ENDPOINT is set, the profile now
installs qbit-object-storage with one MinIO connection and the private
"collaboration" bucket alias that collaboration expects, plus its Storage
admin app. Credentials are ${env.S3_*} references only (the qbit's default
allowed patterns); qqq-cli's scripts/enterprise-server.sh starts the MinIO
container, creates the bucket and generates the credentials outside git.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 06:33:06 -06:00
james.maesandClaude Fable 5.1 468c7c639c feat(enterprise): offer column statistics and audits on the data tables
QQQ turns QUERY_STATS off on every backend by default, so no table of the
enterprise profile offered column statistics: `qqq table stats person
firstName` answered "Person does not offer column statistics (no
QUERY_STATS capability)" and the web UI hid its column statistics.

The enterprise profile now enables QUERY_STATS on the backends whose
module can aggregate (RDBMS: H2, SQLite and the merged sample app's
in-memory H2; and the memory backend). Filesystem and API backends cannot
aggregate and stay without it. The columnStats process that serves the
statistics over the UI API is registered too (the instance had none).
TABLE_COUNT and TABLE_EXPORT were already
on. Field-level audits now also cover orderLine and the sample's person,
pet, petNote and carrier tables, besides customer, order and address.

The qqq sample app's own backend has the same default (its person table on
the 8077 sample server lacks QUERY_STATS too); that belongs in qqq's
sample project, not here.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 06:31:11 -06:00
james.maesandClaude Fable 5.1 74837f5db2 feat: enterprise profile with the full QBit set for terminal-client testing (#48)
Adds QQQ_ALL_PROFILE=enterprise composing the QBit set an enterprise rollout would run (jobs, collaboration, alerts, data-analysis, db-performance, sql-workbench, help, release-notes, qa-capture, secured-data, email, slack, cluster, cdn, backups, file-gateway, session-store, worm-audit, assistant, quick-search 1.1.0, redis realtime, the core profile's seven QBits, the sample app, core audits/scripts/saved views/reports/bulk-load/sharing/query stats/durable jobs/health) with MOCK auth and an in-memory Quartz scheduler. Fixes found by the sweep: health advertised, calculateOrderTotal input form, record labels, mutable health indicators, quoted H2 reserved words, webhook test-event ordering workaround, jobs-permission workaround, quick-search admin permission for the full profile's smoke.

Closes #44, closes #45, closes #46. Refs #47, QRun-IO/qqq#1361, QRun-IO/qqq#1362.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 06:21:34 -06:00
james.maesandCodex 0252c6811b ci: adopt managed orb 0.8 patch track (#40)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 16:04:22 -05:00
james.maesandCodex 73bdcc1a6f ci: adopt shared orb 0.7.17 (#39)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 14:25:53 -05:00
james.maesandCodex d937b19050 ci: use central source scanning (#37)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 11:52:26 -05:00
15 changed files with 1090 additions and 15 deletions

No files matched your search

+2 -2
View File
@@ -1,12 +1,12 @@
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.7.12
qqq-orb: kingsrook/qqq-orb@0.8
workflows:
setup:
jobs:
- qqq-orb/generate_pipeline:
orb_version: 0.7.12
orb_version: '0.8'
filters:
branches:
ignore: automation/qbit-rc-pending
+1 -1
View File
@@ -1,5 +1,5 @@
profile: all
orb_version: 0.7.12
orb_version: '0.8'
default_branch: develop
publish:
snapshot: true
@@ -0,0 +1,13 @@
-- Quick Search 1.x protects its admin processes (quickSearchFullReindex and friends) and its
-- admin app with the quick_search_admin permission; the full-profile admin runs the reindex.
INSERT INTO permission (name, object_type, object_label)
VALUES
('quick_search_admin.hasAccess', 'process', 'Quick Search Admin')
ON CONFLICT (name) DO NOTHING;
INSERT INTO role_permission_int (role_id, permission_id)
SELECT role.id, permission.id
FROM role CROSS JOIN permission
WHERE permission.name = 'quick_search_admin.hasAccess'
AND role.name = 'admin'
ON CONFLICT (role_id, permission_id) DO NOTHING;
+4
View File
@@ -108,6 +108,10 @@ For an existing volume, apply the idempotent grants and then sign in again:
COMPOSE_PROFILES=full docker compose exec -T postgres sh -c 'psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' < infra/postgres/05-runtime-demo-permissions.sql
```
Quick Search 1.x protects `quickSearchFullReindex` and its other admin processes with
`quick_search_admin.hasAccess`; fresh volumes grant it to the admin role through
`infra/postgres/06-quick-search-permissions.sql`. Apply that file the same way on an existing volume.
`mvn -pl qqq-all-app -am verify` includes native local API, calculation/permission,
memory restart, and packaged-JAR checks. `scripts/run_core_smoke.py` and
`scripts/run_full_smoke.py` verify these routes with actual Admin and restricted
+23 -1
View File
@@ -42,13 +42,35 @@
<dependency><groupId>com.kingsrook.qqq</groupId><artifactId>qqq-esb</artifactId></dependency>
<dependency><groupId>com.kingsrook.qqq</groupId><artifactId>qqq-frontend-next</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-user-role-permissions</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-quick-search</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-quick-search</artifactId><version>1.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-customizable-table-views</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-standard-process-trace</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-webhooks</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-workflows</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-geo-data</artifactId></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-sftp-data-integration</artifactId></dependency>
<!-- Enterprise profile (QQQ_ALL_PROFILE=enterprise): every locally runnable qbit, for terminal-client testing. -->
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-alerts</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-assistant</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-backups</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-cdn</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-cluster</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-collaboration</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-data-analysis</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-db-performance</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-email</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-file-gateway</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-help</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-jobs</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-object-storage</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-qa-capture</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-redis</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-release-notes</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-secured-data</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-session-store</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-slack</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-sql-workbench</artifactId><version>1.0.0-SNAPSHOT</version></dependency>
<dependency><groupId>com.kingsrook.qbits</groupId><artifactId>qbit-worm-audit</artifactId><version>0.1.0-SNAPSHOT</version></dependency>
<dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId></dependency>
<dependency><groupId>com.mysql</groupId><artifactId>mysql-connector-j</artifactId></dependency>
<dependency><groupId>com.h2database</groupId><artifactId>h2</artifactId></dependency>
@@ -34,6 +34,18 @@ final class DemoQBitData
"redirect_state", "user", "role", "permission", "role_permission_int", "user_role_int",
"user_permission_int");
/** H2 (MySQL mode) keywords that qbit tables use as column or table names. */
private static final Set<String> H2_RESERVED_WORDS = Set.of("VALUE", "KEY", "VALUES", "USER", "ORDER", "GROUP",
"LIMIT", "OFFSET", "YEAR", "MONTH", "DAY", "HOUR", "MINUTE", "SECOND", "ROW", "ROWS", "END", "FROM", "TO",
"ALL", "AND", "OR", "NOT", "NULL", "TRUE", "FALSE", "IF", "IN", "IS", "AS", "ON", "BY", "SELECT", "WHERE",
"TABLE", "INTERVAL", "DEFAULT", "CHECK", "CONSTRAINT", "PRIMARY", "UNIQUE", "FOREIGN", "INDEX", "JOIN",
"LEFT", "RIGHT", "INNER", "OUTER", "CROSS", "NATURAL", "UNION", "EXCEPT", "INTERSECT", "MINUS", "DISTINCT",
"HAVING", "QUALIFY", "WINDOW", "CASE", "WHEN", "THEN", "ELSE", "FETCH", "FOR", "WITH", "CURRENT_DATE",
"CURRENT_TIME", "CURRENT_TIMESTAMP", "CURRENT_USER", "LOCALTIME", "LOCALTIMESTAMP", "ARRAY", "BETWEEN",
"EXISTS", "LIKE", "ILIKE", "REGEXP", "SET", "UNKNOWN", "USING", "TOP", "SYSDATE", "SYSTIME", "TODAY",
"_ROWID_", "ROWNUM", "LISTAGG", "INTERSECTS", "ANY", "SOME", "ASYMMETRIC", "SYMMETRIC", "AUTHORIZATION",
"CURRENT_CATALOG", "CURRENT_SCHEMA", "CURRENT_PATH", "CURRENT_ROLE", "LATERAL", "UESCAPE");
private DemoQBitData()
{
}
@@ -231,7 +243,7 @@ final class DemoQBitData
{
throw new IllegalArgumentException("Unsupported demo schema identifier");
}
if("value".equalsIgnoreCase(value) || "key".equalsIgnoreCase(value))
if(H2_RESERVED_WORDS.contains(value.toUpperCase()))
{
return "`" + value + "`";
}
@@ -11,6 +11,7 @@ import com.kingsrook.qbits.customizabletableviews.CustomizableTableViewsQBitConf
import com.kingsrook.qbits.customizabletableviews.CustomizableTableViewsQBitProducer;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitProducer;
import com.kingsrook.qbits.quicksearch.QuickSearchStartupMode;
import com.kingsrook.qbits.quicksearch.SearchableFieldConfig;
import com.kingsrook.qbits.sftpdataintegration.SFTPDataIntegrationQBitConfig;
import com.kingsrook.qbits.sftpdataintegration.SFTPDataIntegrationQBitProducer;
@@ -34,8 +35,11 @@ import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.ProvidedOrSuppliedTableConfig;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
@@ -111,6 +115,8 @@ final class DemoQBits
.withCategory(WebhookEventCategory.STORE).withTableName("order")
.withCustomizer(new QCodeReference(DemoOrderWebhookContent.class)));
moveTestEventFormAfterPreview(instance);
new WorkflowsQBitProducer()
.withQBitConfig(new WorkflowsQBitConfig().withTableMetaDataCustomizer(tableCustomizer))
.produce(instance).addSelfToInstance(instance);
@@ -156,11 +162,14 @@ final class DemoQBits
.withOpensearchUsername(settings.get("OPENSEARCH_USER"))
.withOpensearchPassword(settings.get("OPENSEARCH_PASSWORD"))
.withUseSsl(Boolean.parseBoolean(settings.getOrDefault("OPENSEARCH_SSL", "false")))
// quick search 1.x connects at startup unless degraded; 0.2.x did not (the enterprise profile bumped it)
.withStartupMode(QuickSearchStartupMode.DEGRADED)
.withSearchableTable("customer", List.of(new SearchableFieldConfig("name"),
new SearchableFieldConfig("email")));
////////////////////////////////////////////////////////////////////
// quick search 1.x registers its own QBitMetaData when produced //
////////////////////////////////////////////////////////////////////
new QuickSearchQBitProducer().withConfig(searchConfig).produce(instance);
instance.addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits")
.withArtifactId("quick-search").withVersion("0.2.1-SNAPSHOT").withConfig(searchConfig));
new SavedBulkLoadProfileMetaDataProvider().defineAll(instance, "postgres", null);
new SavedReportsMetaDataProvider().defineAll(instance, "postgres", "sftp", null);
@@ -181,4 +190,27 @@ final class DemoQBits
return table;
})).produce(instance, "demo-import");
}
/**
* qbit-webhooks 0.x puts SendTestEventToWebhook's input form before the
* preview step, so the records chosen at init are lost and the run fails
* with "No records were selected" (QRun-IO/qqq#1361; fixed on qbit-webhooks
* fix/test-event-records). Until a release carries it, move the form after
* the first backend step here.
*/
static void moveTestEventFormAfterPreview(QInstance instance)
{
QProcessMetaData process = instance.getProcess("SendTestEventToWebhook");
if(process == null || process.getStepList() == null || process.getStepList().size() < 2)
{
return;
}
List<QStepMetaData> steps = process.getStepList();
if(steps.get(0) instanceof QFrontendStepMetaData form && "input".equals(form.getName()) && steps.get(1) instanceof QBackendStepMetaData)
{
steps.remove(0);
steps.add(1, form);
}
}
}
@@ -21,6 +21,10 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionOutputMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
@@ -62,13 +66,25 @@ final class DemoRuntimeFeatures
instance.withRuntimeService(new QCodeReference(TransientNotesService.class));
instance.addProcess(new QProcessMetaData().withName("calculateOrderTotal").withLabel("Calculate Order Total (JavaScript)")
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION))
// the inputs as a form, so interactive clients ask for them (QRun-IO/qqq-all#45)
.withStep(new QFrontendStepMetaData().withName("input").withLabel("Order line")
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM))
.withFormField(new QFieldMetaData("quantity", QFieldType.INTEGER).withLabel("Quantity").withIsRequired(true)
.withHelpContent(new QHelpContent("1 to 1000")))
.withFormField(new QFieldMetaData("unitPriceCents", QFieldType.INTEGER).withLabel("Unit price (cents)").withIsRequired(true)
.withHelpContent(new QHelpContent("0 to 1000000"))))
.withStep(new QBackendStepMetaData().withName("calculate")
.withCode(new QCodeReference(CalculateOrderTotalStep.class))
.withInputData(new QFunctionInputMetaData().withFieldList(List.of(
new QFieldMetaData("quantity", QFieldType.INTEGER),
new QFieldMetaData("unitPriceCents", QFieldType.INTEGER))))
.withOutputMetaData(new QFunctionOutputMetaData().withFieldList(List.of(
new QFieldMetaData("totalCents", QFieldType.INTEGER))))));
new QFieldMetaData("totalCents", QFieldType.INTEGER)))))
.withStep(new QFrontendStepMetaData().withName("result").withLabel("Total")
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.VIEW_FORM))
.withViewField(new QFieldMetaData("quantity", QFieldType.INTEGER).withLabel("Quantity"))
.withViewField(new QFieldMetaData("unitPriceCents", QFieldType.INTEGER).withLabel("Unit price (cents)"))
.withViewField(new QFieldMetaData("totalCents", QFieldType.INTEGER).withLabel("Total (cents)"))));
QJavalinMetaData.ofOrWithNew(instance).withAdditionalRouteProviderReference(new QCodeReference(DemoCatalogRoutes.class));
}
@@ -9,7 +9,9 @@ package com.kingsrook.qqq.all;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.session.QSession;
@@ -54,7 +56,7 @@ public class DemoUsers implements QAuthenticationModuleCustomizerInterface
public void customizeSession(QInstance instance, QSession session, Map<String, Object> context)
{
session.withPermissions("apiCatalog.read", "demoNote.read");
Boolean admin = ADMIN_SESSION.equals(session.getUuid());
Boolean admin = ADMIN_SESSION.equals(session.getUuid()) || EnterpriseFeatures.isEnabled();
String name = admin ? "Admin" : "Demo User";
String userId = admin ? "core:admin" : "core:demo";
session.getUser().setIdReference(userId);
@@ -84,6 +86,30 @@ public class DemoUsers implements QAuthenticationModuleCustomizerInterface
session.withPermission(processName + ".hasAccess");
}
}
if(EnterpriseFeatures.isEnabled())
{
//////////////////////////////////////////////////////////////////////
// enterprise test server: every session is an all-access admin, so //
// the terminal clients can reach every qbit surface //
//////////////////////////////////////////////////////////////////////
session.withPermissions(PermissionsHelper.getAllAvailablePermissionNames(instance));
//////////////////////////////////////////////////////////////////////
// permissions some modules check by literal name, outside the //
// table/process permission rules (QRun-IO/qqq#1362) //
//////////////////////////////////////////////////////////////////////
session.withPermissions("jobs_job.read", "jobs_admin", "data_analysis.admin");
for(QSecurityKeyType keyType : instance.getSecurityKeyTypes().values())
{
if(keyType.getAllAccessKeyName() != null)
{
session.withSecurityKeyValue(keyType.getAllAccessKeyName(), true);
}
//////////////////////////////////////////////////////////////////
// per-user keys (owner locks of qbit tables) take the user id //
//////////////////////////////////////////////////////////////////
session.withSecurityKeyValue(keyType.getName(), userId);
}
}
session.withValueForFrontend("user", new LinkedHashMap<>(Map.of("name", name, "email", userId + "@example.test")));
}
}
@@ -0,0 +1,652 @@
/*
* Copyright 2026 Kingsrook, LLC
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0
*/
package com.kingsrook.qqq.all;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.TreeMap;
import java.util.Properties;
import java.util.Set;
import java.util.function.Consumer;
import java.util.stream.Collectors;
import com.kingsrook.qbits.alerts.AlertsQBitConfig;
import com.kingsrook.qbits.alerts.AlertsQBitProducer;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.AssistantQBitProducer;
import com.kingsrook.qbits.backups.BackupsQBitConfig;
import com.kingsrook.qbits.backups.BackupsQBitProducer;
import com.kingsrook.qbits.backups.storage.InMemoryBackupStorage;
import com.kingsrook.qbits.cdn.CdnQBitConfig;
import com.kingsrook.qbits.cdn.CdnQBitProducer;
import com.kingsrook.qbits.cluster.ClusterQBitConfig;
import com.kingsrook.qbits.cluster.ClusterQBitProducer;
import com.kingsrook.qbits.collaboration.CollaborationQBitConfig;
import com.kingsrook.qbits.collaboration.CollaborationQBitProducer;
import com.kingsrook.qbits.dataanalysis.DataAnalysisQBitConfig;
import com.kingsrook.qbits.dataanalysis.DataAnalysisQBitProducer;
import com.kingsrook.qbits.dbperformance.DbPerfQBitConfig;
import com.kingsrook.qbits.dbperformance.DbPerfQBitProducer;
import com.kingsrook.qbits.email.EmailQBitConfig;
import com.kingsrook.qbits.email.EmailQBitProducer;
import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig;
import com.kingsrook.qbits.filegateway.FileGatewayQBitProducer;
import com.kingsrook.qbits.help.HelpQBitConfig;
import com.kingsrook.qbits.help.HelpQBitProducer;
import com.kingsrook.qbits.jobs.JobsQBitConfig;
import com.kingsrook.qbits.jobs.JobsQBitProducer;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitProducer;
import com.kingsrook.qbits.objectstorage.config.AddressingStyle;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.config.ProviderType;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
import com.kingsrook.qbits.qacapture.QaCaptureQBitConfig;
import com.kingsrook.qbits.qacapture.QaCaptureQBitProducer;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitProducer;
import com.kingsrook.qbits.quicksearch.SearchableFieldConfig;
import com.kingsrook.qbits.redis.RedisConnectionConfig;
import com.kingsrook.qbits.redis.RedisQBitConfig;
import com.kingsrook.qbits.redis.RedisQBitProducer;
import com.kingsrook.qbits.releasenotes.ReleaseNotesQBitConfig;
import com.kingsrook.qbits.releasenotes.ReleaseNotesQBitProducer;
import com.kingsrook.qbits.releasenotes.sources.ComponentVersionsSource;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.SecuredDataQBitProducer;
import com.kingsrook.qbits.sessionstore.QSessionStoreProviderType;
import com.kingsrook.qbits.sessionstore.QSessionStoreQBitConfig;
import com.kingsrook.qbits.sessionstore.QSessionStoreQBitProducer;
import com.kingsrook.qbits.slack.SlackQBitConfig;
import com.kingsrook.qbits.slack.SlackQBitProducer;
import com.kingsrook.qbits.sqlworkbench.SqlWorkbenchQBitConfig;
import com.kingsrook.qbits.sqlworkbench.SqlWorkbenchQBitProducer;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditQBitProducer;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
import com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider;
import com.kingsrook.qqq.backend.core.jobs.DurableJobsMetaData;
import com.kingsrook.qqq.backend.core.jobs.DurableJobsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import com.kingsrook.qqq.backend.core.processes.implementations.columnstats.ColumnStatsStep;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.quartz.QuartzSchedulerMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppChildMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.querystats.QueryStatMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.savedreports.SavedReportsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.savedviews.SavedViewsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.scripts.ScriptsMetaDataProvider;
import com.kingsrook.qqq.backend.core.actions.preferences.UserPreferencesKeyValueStoreMetaDataProvider;
import com.kingsrook.qqq.backend.core.processes.implementations.sharing.SharingMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature;
import com.kingsrook.qqq.backend.module.filesystem.base.model.metadata.AbstractFilesystemTableBackendDetails;
import com.kingsrook.qqq.backend.module.filesystem.base.model.metadata.Cardinality;
import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemBackendMetaData;
import com.kingsrook.qqq.backend.module.filesystem.local.model.metadata.FilesystemTableBackendDetails;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails;
/*******************************************************************************
** The "enterprise" profile (QQQ_ALL_PROFILE=enterprise): the core profile plus
** every qbit and core module an enterprise rollout runs, all on the local H2
** backend, for exercising the terminal clients against a large instance.
**
** Each feature is named; QQQ_ALL_ENTERPRISE_EXCLUDE (comma separated names)
** leaves features out. Quick search needs QQQ_ALL_OPENSEARCH_HOST and redis
** needs QQQ_ALL_REDIS_HOST; without them those two are left out.
*******************************************************************************/
final class EnterpriseFeatures
{
static final String PROFILE = "enterprise";
private static final String SAMPLE_PROVIDER = "com.kingsrook.sampleapp.metadata.SampleMetaDataProvider";
private interface Feature
{
void add(QInstance instance) throws Exception;
}
private EnterpriseFeatures()
{
}
/*******************************************************************************
** True when the process runs the enterprise profile.
*******************************************************************************/
static boolean isEnabled()
{
return PROFILE.equals(System.getenv("QQQ_ALL_PROFILE"));
}
/*******************************************************************************
** Names the operator excluded.
*******************************************************************************/
static Set<String> excluded()
{
String value = System.getenv().getOrDefault("QQQ_ALL_ENTERPRISE_EXCLUDE", "");
return Arrays.stream(value.split(",")).map(String::trim).filter(s -> !s.isEmpty()).collect(Collectors.toSet());
}
/*******************************************************************************
** Add every enterprise feature that is not excluded.
*******************************************************************************/
static void add(QInstance instance, String backendName, int port) throws QException
{
String schedulerName = "demoScheduler";
String baseUrl = "http://127.0.0.1:" + port;
//////////////////////////////////////////////////////////////////////////
// qbits schedule with cron expressions, which the simple scheduler of //
// the core profile cannot run: swap it for an in-memory Quartz //
//////////////////////////////////////////////////////////////////////////
Properties quartzProperties = new Properties();
quartzProperties.put("org.quartz.scheduler.instanceName", "qqqAllEnterprise");
quartzProperties.put("org.quartz.threadPool.threadCount", "4");
quartzProperties.put("org.quartz.jobStore.class", "org.quartz.simpl.RAMJobStore");
instance.getSchedulers().put(schedulerName, new QuartzSchedulerMetaData().withProperties(quartzProperties).withName(schedulerName));
MetaDataCustomizerInterface<QTableMetaData> tableCustomizer = (i, table) ->
{
if(table.getBackendName() == null || backendName.equals(table.getBackendName()))
{
table.setBackendName(backendName);
table.setBackendDetails(new RDBMSTableBackendDetails()
.withTableName(QInstanceEnricher.inferBackendName(table.getName())));
QInstanceEnricher.setInferredFieldBackendNames(table);
}
return table;
};
Consumer<QTableMetaData> enricher = table ->
{
try
{
tableCustomizer.customizeMetaData(instance, table);
}
catch(QException e)
{
throw new IllegalStateException(e);
}
};
Map<String, Feature> features = new LinkedHashMap<>();
/////////////////////////
// core qqq modules //
/////////////////////////
features.put("audits", i ->
{
new AuditsMetaDataProvider().defineAll(i, backendName, enricher);
for(String tableName : List.of("customer", "order", "address"))
{
i.getTable(tableName).setAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD));
}
});
features.put("scripts", i ->
{
////////////////////////////////////////////////////////////////////
// the scripts provider defines its own "tables" possible values //
////////////////////////////////////////////////////////////////////
i.getPossibleValueSources().remove("tables");
new ScriptsMetaDataProvider().defineAll(i, backendName, enricher);
});
features.put("savedViews", i -> new SavedViewsMetaDataProvider().defineAll(i, backendName, enricher));
features.put("savedReports", i ->
{
new SavedReportsMetaDataProvider().defineAll(i, backendName, "files", enricher);
//////////////////////////////////////////////////////////////////
// rendered reports are files, one per record, under the files //
// backend (the provider leaves the storage details to the host) //
//////////////////////////////////////////////////////////////////
QTableMetaData reportStorage = i.getTable("reportStorage");
if(reportStorage != null && reportStorage.getBackendDetails() == null)
{
reportStorage.setBackendDetails(new FilesystemTableBackendDetails().withBasePath("reportStorage").withCardinality(Cardinality.ONE).withFileNameFieldName("reference").withContentsFieldName("contents"));
}
ensureFilesystemDirectories(i);
});
features.put("bulkLoadProfiles", i -> new SavedBulkLoadProfileMetaDataProvider().defineAll(i, backendName, enricher));
features.put("sharing", i -> new SharingMetaDataProvider().defineAll(i, null));
features.put("queryStats", i -> new QueryStatMetaDataProvider().defineAll(i, backendName, enricher));
features.put("userPreferences", i -> new UserPreferencesKeyValueStoreMetaDataProvider().defineAll(i, backendName, null));
//////////////////////////////////////////////////////////////////////////
// central health: its tables turn on the health UI API extension, so //
// /version lists it (QRun-IO/qqq-all#44). Enterprise only for now: in //
// the core profile its backend checks share the cached sqlite //
// connection across the test suite's runtimes //
//////////////////////////////////////////////////////////////////////////
features.put("health", i -> new HealthMetaDataProvider().defineAll(i, backendName, enricher));
features.put("durableJobs", i ->
{
new DurableJobsMetaDataProvider().defineAll(i, backendName, enricher);
i.withSupplementalMetaData(new DurableJobsMetaData().withWorkerThreads(2));
});
/////////////////////////
// qbits //
/////////////////////////
//////////////////////////////////////////////////////////////////////////
// object storage on a local MinIO (enterprise-server.sh starts it): //
// collaboration attachments live in its private "collaboration" //
// bucket. Credentials are references to the server's env only. //
//////////////////////////////////////////////////////////////////////////
String s3Endpoint = System.getenv("S3_ENDPOINT");
if(s3Endpoint != null && !s3Endpoint.isBlank())
{
features.put("objectStorage", i -> new ObjectStorageQBitProducer().withQBitConfig(new ObjectStorageQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName).withAdminApp(true)
.withConnection(new StorageConnectionConfig("local").withProviderType(ProviderType.MINIO)
.withEndpoint("${env.S3_ENDPOINT}").withRegion("us-east-1").withAddressingStyle(AddressingStyle.PATH)
.withAccessKeyRef("${env.S3_ACCESS_KEY}").withSecretKeyRef("${env.S3_SECRET_KEY}"))
.withBucket(new StorageBucketConfig("collaboration").withConnection("local")
.withBucketName("${env.S3_ATTACHMENTS_BUCKET}").withRole(BucketRole.PRIVATE).withIndexed(false)))
.produce(i).addSelfToInstance(i));
}
features.put("jobs", i -> new JobsQBitProducer().withQBitConfig(new JobsQBitConfig()
.withSchedulerName(schedulerName)).produce(i).addSelfToInstance(i));
features.put("collaboration", i -> new CollaborationQBitProducer().withQBitConfig(new CollaborationQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName).withUiApi(true).withAdminApp(true)
.withDefaultFeatures(RecordFeature.values())
.withFrontendBaseUrl(baseUrl)).produce(i).addSelfToInstance(i));
features.put("alerts", i -> new AlertsQBitProducer().withQBitConfig(new AlertsQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName)
.withPublicBaseUrl(baseUrl)).produce(i).addSelfToInstance(i));
features.put("dataAnalysis", i -> new DataAnalysisQBitProducer().withQBitConfig(new DataAnalysisQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName)).produce(i).addSelfToInstance(i));
features.put("dbPerformance", i -> new DbPerfQBitProducer().withQBitConfig(new DbPerfQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName)).produce(i).addSelfToInstance(i));
features.put("sqlWorkbench", i -> new SqlWorkbenchQBitProducer().withQBitConfig(new SqlWorkbenchQBitConfig()
.withBackendName(backendName).withTableMetaDataCustomizer(tableCustomizer)
.withAllowedBackendNames(List.of(backendName))).produce(i).addSelfToInstance(i));
features.put("help", i -> new HelpQBitProducer().withQBitConfig(new HelpQBitConfig()
.withBackendName(backendName).withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i));
features.put("releaseNotes", i -> new ReleaseNotesQBitProducer().withQBitConfig(new ReleaseNotesQBitConfig()
.withBackendName(backendName).withApplicationName("qqqAll").withApplicationLabel("QQQ All")
.withAppVersion("4.1.0").withAllowNonReleaseVersions(true)
.withSource(new ComponentVersionsSource())).produce(i).addSelfToInstance(i));
features.put("qaCapture", i -> new QaCaptureQBitProducer().withQBitConfig(new QaCaptureQBitConfig()
.withBackendName(backendName).withEnvironment("local").withBackendVersion("4.1.0-SNAPSHOT"))
.produce(i).addSelfToInstance(i));
features.put("securedData", i -> new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName)
.withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i));
features.put("email", i -> new EmailQBitProducer().withQBitConfig(new EmailQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName).withPublicBaseUrl(baseUrl)
.withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i));
features.put("slack", i -> new SlackQBitProducer().withQBitConfig(new SlackQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName).withPublicBaseUrl(baseUrl)
.withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i));
features.put("cluster", i -> new ClusterQBitProducer().withQBitConfig(new ClusterQBitConfig()
.withBackendName(backendName)).produce(i).addSelfToInstance(i));
features.put("cdn", i -> new CdnQBitProducer().withQBitConfig(new CdnQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName)
.withPurgeableTables(List.of("customer")).withTableMetaDataCustomizer(tableCustomizer))
.produce(i).addSelfToInstance(i));
features.put("backups", i -> new BackupsQBitProducer().withQBitConfig(new BackupsQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName)
.withStorageResolver(name -> new InMemoryBackupStorage())).produce(i).addSelfToInstance(i));
features.put("fileGateway", i -> new FileGatewayQBitProducer().withQBitConfig(new FileGatewayQBitConfig()
.withBackendName(backendName).withSftpEnabled(false).withSchedulerName(schedulerName)
.withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i));
features.put("sessionStore", i ->
{
new QSessionStoreQBitProducer().withConfig(new QSessionStoreQBitConfig()
.withProviderType(QSessionStoreProviderType.TABLE_BASED).withBackendName(backendName)
.withDefaultTtl(Duration.ofHours(8)).withEnableSlidingExpiration(true)).produce(i, "session-store");
if(i.getProcess("cleanExpiredSessions") != null && i.getProcess("cleanExpiredSessions").getSchedule() != null)
{
i.getProcess("cleanExpiredSessions").getSchedule().setSchedulerName(schedulerName);
}
});
features.put("wormAudit", i ->
{
KeyPairGenerator generator = KeyPairGenerator.getInstance("Ed25519");
KeyPair keyPair = generator.generateKeyPair();
new WormAuditQBitProducer().withQBitConfig(new WormAuditQBitConfig()
.withBackendName(backendName).withSchedulerName(schedulerName).withAppRole("app")
.withSigningKey(() -> new WormAuditQBitConfig.SigningKey("local", keyPair.getPrivate()))
.withTrustedPublicKeys(Map.of("local", keyPair.getPublic()))
.withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i);
});
features.put("assistant", i -> new AssistantQBitProducer().withQBitConfig(new AssistantQBitConfig()
.withBackendName(backendName).withUiApiBaseUrl(baseUrl)
.withTableMetaDataCustomizer(tableCustomizer)).produce(i).addSelfToInstance(i));
if(isSampleAppOnClasspath())
{
features.put("sampleApp", EnterpriseFeatures::mergeSampleApp);
}
String opensearchHost = System.getenv("QQQ_ALL_OPENSEARCH_HOST");
if(opensearchHost != null && !opensearchHost.isBlank())
{
features.put("quickSearch", i ->
{
QuickSearchQBitConfig searchConfig = new QuickSearchQBitConfig()
.withBackendName(backendName)
.withOpensearchHost(opensearchHost)
.withOpensearchPort(Integer.valueOf(System.getenv().getOrDefault("QQQ_ALL_OPENSEARCH_PORT", "9200")))
.withOpensearchIndexName(System.getenv().getOrDefault("QQQ_ALL_OPENSEARCH_INDEX", "qqq-all-enterprise"))
.withUseSsl(false)
.withAllowPlaintextCredentials(true)
.withSchedulerName(schedulerName)
.withTableMetaDataCustomizer(tableCustomizer)
.withSearchableTable("customer", List.of(new SearchableFieldConfig("name"), new SearchableFieldConfig("email")))
.withSearchableTable("order", List.of(new SearchableFieldConfig("orderNo"), new SearchableFieldConfig("status")));
new QuickSearchQBitProducer().withConfig(searchConfig).produce(i).addSelfToInstance(i);
});
}
String redisHost = System.getenv("QQQ_ALL_REDIS_HOST");
if(redisHost != null && !redisHost.isBlank())
{
features.put("redis", i -> new RedisQBitProducer().withQBitConfig(new RedisQBitConfig()
.withConnection("realtime", new RedisConnectionConfig().withHost(redisHost)
.withPort(Integer.valueOf(System.getenv().getOrDefault("QQQ_ALL_REDIS_PORT", "6379")))
.withTls(false))
.withRealtime("realtime", "local", "qqq-all")).produce(i).addSelfToInstance(i));
}
Set<String> excluded = excluded();
for(Map.Entry<String, Feature> entry : features.entrySet())
{
if(excluded.contains(entry.getKey()))
{
System.out.println("[enterprise] skipped " + entry.getKey());
continue;
}
try
{
entry.getValue().add(instance);
System.out.println("[enterprise] added " + entry.getKey());
}
catch(Exception e)
{
if("true".equals(System.getenv("QQQ_ALL_ENTERPRISE_LENIENT")))
{
System.out.println("[enterprise] FAILED " + entry.getKey() + ": " + e);
Arrays.stream(e.getStackTrace()).limit(6).forEach(frame -> System.out.println("[enterprise] at " + frame));
continue;
}
throw new QException("Enterprise feature " + entry.getKey() + " failed: " + e.getMessage(), e);
}
}
enableColumnStatistics(instance);
auditDataTables(instance);
addNavigationApps(instance);
}
/*******************************************************************************
** QQQ turns QUERY_STATS (column statistics: counts, distinct values, min/max
** and averages per field) off on every backend by default. Turn it on for the
** backends whose module can aggregate, RDBMS (H2, SQLite, the sample's
** in-memory H2) and the memory backend, so their tables offer statistics as
** the web UI and the terminal clients expect; filesystem and API backends
** cannot aggregate and stay without it. Tables of those backends that list
** QUERY_STATS as disabled keep it off. The columnStats process that serves
** the statistics is added when the instance lacks it, and runs through
** GuardedColumnStatsStep (only fully readable fields).
*******************************************************************************/
static void enableColumnStatistics(QInstance instance)
{
////////////////////////////////////////////////////////////////////
// the UI API serves column statistics through this process //
////////////////////////////////////////////////////////////////////
if(instance.getProcess(ColumnStatsStep.getProcessMetaData().getName()) == null)
{
instance.addProcess(ColumnStatsStep.getProcessMetaData());
}
////////////////////////////////////////////////////////////////////////
// run it through the guarded step: fields the caller cannot read in //
// full are refused, and the results carry the analysed table so its //
// field-access policy applies (QRun-IO/qqq#1367) //
////////////////////////////////////////////////////////////////////////
QProcessMetaData process = instance.getProcess(ColumnStatsStep.getProcessMetaData().getName());
for(QStepMetaData step : process.getStepList())
{
if(step instanceof QBackendStepMetaData backendStep && backendStep.getCode() != null
&& ColumnStatsStep.class.getName().equals(backendStep.getCode().getName()))
{
backendStep.setCode(new QCodeReference(GuardedColumnStatsStep.class));
}
}
for(QBackendMetaData backend : instance.getBackends().values())
{
if(backend instanceof RDBMSBackendMetaData || new MemoryBackendModule().getBackendType().equals(backend.getBackendType()))
{
backend.withCapability(Capability.QUERY_STATS);
}
}
}
/*******************************************************************************
** Field-level audits on the demo and sample data tables (customer, order and
** address get them with the audits feature), when audits are installed.
*******************************************************************************/
static void auditDataTables(QInstance instance)
{
if(instance.getTable("audit") == null)
{
return;
}
for(String tableName : List.of("customer", "order", "orderLine", "address", "person", "pet", "petNote", "carrier"))
{
QTableMetaData table = instance.getTable(tableName);
if(table != null && table.getAuditRules() == null)
{
table.setAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD));
}
}
}
/*******************************************************************************
** The core profile has no apps and several qbits only add tables. Put the
** demo tables in a "QQQ All" app and every visible table that no app reaches
** into an "Enterprise Admin" app, one section per table-name prefix, so the
** clients' app tree reaches everything.
*******************************************************************************/
private static void addNavigationApps(QInstance instance)
{
List<String> demoTables = List.of("customer", "order", "orderLine", "product", "address", "apiCatalog", "demoNote",
"shipping_country", "shipping_stateProvince", "shipping_city");
List<QAppChildMetaData> demoChildren = new ArrayList<>();
demoTables.stream().map(instance::getTable).filter(Objects::nonNull).forEach(demoChildren::add);
List.of("syncOrder", "calculateOrderTotal").stream().map(instance::getProcess).filter(Objects::nonNull).forEach(demoChildren::add);
instance.addApp(new QAppMetaData().withName("qqqAll").withLabel("QQQ All").withIcon(new QIcon("storefront"))
.withSectionOfChildren(new QAppSection().withName("demoData").withLabel("Demo Data").withIcon(new QIcon("table_chart")), demoChildren));
Set<String> reached = new HashSet<>();
for(QAppMetaData app : instance.getApps().values())
{
if(app.getChildren() != null)
{
app.getChildren().stream().filter(c -> c instanceof QTableMetaData).forEach(c -> reached.add(c.getName()));
}
if(app.getSections() != null)
{
app.getSections().stream().filter(section -> section.getTables() != null).forEach(section -> reached.addAll(section.getTables()));
}
}
Map<String, List<QAppChildMetaData>> byPrefix = new TreeMap<>();
for(QTableMetaData table : instance.getTables().values())
{
if(reached.contains(table.getName()) || table.getIsHidden())
{
continue;
}
String backendName = QInstanceEnricher.inferBackendName(table.getName());
String prefix = backendName.contains("_") ? backendName.substring(0, backendName.indexOf('_')) : backendName;
byPrefix.computeIfAbsent(prefix, k -> new ArrayList<>()).add(table);
}
QAppMetaData admin = new QAppMetaData().withName("enterpriseAdmin").withLabel("Enterprise Admin").withIcon(new QIcon("admin_panel_settings"));
byPrefix.forEach((prefix, tables) -> admin.withSectionOfChildren(new QAppSection().withName("admin_" + prefix)
.withLabel(QInstanceEnricher.nameToLabel(prefix)).withIcon(new QIcon("folder")), tables));
instance.addApp(admin);
}
/*******************************************************************************
** Filesystem tables (rendered report storage) answer 500 when their directory
** does not exist yet: create each one under the files backend.
*******************************************************************************/
private static void ensureFilesystemDirectories(QInstance instance) throws Exception
{
if(!(instance.getBackend("files") instanceof FilesystemBackendMetaData files))
{
return;
}
for(QTableMetaData table : instance.getTables().values())
{
if("files".equals(table.getBackendName()) && table.getBackendDetails() instanceof AbstractFilesystemTableBackendDetails details && details.getBasePath() != null)
{
Files.createDirectories(Path.of(files.getBasePath(), details.getBasePath()));
}
}
}
/*******************************************************************************
** True when the qqq sample project's classes are on the classpath (the start
** script adds qqq-sample-project/target/classes).
*******************************************************************************/
private static boolean isSampleAppOnClasspath()
{
try
{
Class.forName(SAMPLE_PROVIDER);
return true;
}
catch(ClassNotFoundException e)
{
return false;
}
}
/*******************************************************************************
** Merge the standard qqq sample app (person, pet, carrier, city files, the
** field and adornment labs, its processes, widgets and apps) into this
** instance. Its ESB wiring is dropped (this instance has its own broker) and
** names this instance already defines win. Reflection keeps the sample
** project out of the build; it is a classpath add-on.
*******************************************************************************/
private static void mergeSampleApp(QInstance instance) throws Exception
{
Class<?> provider = Class.forName(SAMPLE_PROVIDER);
QInstance sample = (QInstance) provider.getMethod("defineInstance").invoke(null);
provider.getMethod("primeTestDatabase", String.class).invoke(null, "prime-test-database.sql");
//////////////////////////////////////////////////////////////////////////
// the sample's sales module defines its own "customer"; this instance //
// already has one, so the sales module stays out //
//////////////////////////////////////////////////////////////////////////
Set<String> skip = Set.of("sales", "customer", "sku", "tag", "sales_order", "sales_order_line", "sales_order_tag",
"salesOrderJoinLine", "salesOrderJoinTag", "salesOrderLines", "salesOrderTags");
sample.getTables().keySet().removeAll(skip);
sample.getPossibleValueSources().keySet().removeAll(skip);
sample.getJoins().keySet().removeAll(skip);
sample.getWidgets().keySet().removeAll(skip);
sample.getApps().keySet().removeAll(skip);
sample.getApps().values().forEach(app ->
{
if(app.getChildren() != null)
{
app.getChildren().removeIf(child -> skip.contains(child.getName()));
}
if(app.getSections() != null)
{
app.getSections().forEach(section ->
{
if(section.getTables() != null)
{
section.getTables().removeIf(skip::contains);
}
});
}
});
sample.getBackends().forEach((name, value) -> instance.getBackends().putIfAbsent(name, value));
sample.getPossibleValueSources().forEach((name, value) -> instance.getPossibleValueSources().putIfAbsent(name, value));
sample.getSecurityKeyTypes().forEach((name, value) -> instance.getSecurityKeyTypes().putIfAbsent(name, value));
sample.getTables().forEach((name, table) ->
{
if(table.getSupplementalMetaData() != null)
{
table.getSupplementalMetaData().values().removeIf(EnterpriseFeatures::isEsb);
}
instance.getTables().putIfAbsent(name, table);
});
sample.getProcesses().forEach((name, process) ->
{
if(process.getSupplementalMetaData() != null)
{
process.getSupplementalMetaData().values().removeIf(EnterpriseFeatures::isEsb);
}
instance.getProcesses().putIfAbsent(name, process);
});
sample.getJoins().forEach((name, value) -> instance.getJoins().putIfAbsent(name, value));
sample.getReports().forEach((name, value) -> instance.getReports().putIfAbsent(name, value));
sample.getWidgets().forEach((name, value) -> instance.getWidgets().putIfAbsent(name, value));
sample.getApps().forEach((name, value) -> instance.getApps().putIfAbsent(name, value));
}
/*******************************************************************************
** ESB supplemental meta data of the sample points at its own destinations.
*******************************************************************************/
private static boolean isEsb(Object supplemental)
{
return supplemental != null && supplemental.getClass().getName().startsWith("com.kingsrook.qqq.esb.");
}
}
@@ -0,0 +1,98 @@
/*
* Copyright 2026 Kingsrook, LLC
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0
*/
package com.kingsrook.qqq.all;
import java.io.Serializable;
import java.util.Collection;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.columnstats.ColumnStatsStep;
/**
* Column statistics only for fields the caller may read in full.
*
* Statistics expose values (min, max, the most common values), so a field
* whose access tier for the caller is REVEALABLE, MASKED or HIDDEN is refused
* with a permission error before anything is computed. The result records are
* then tagged with the analysed table, so the UI API applies that table's
* field-access policy to them instead of withholding table-less records.
* The process shape belongs in qqq core (QRun-IO/qqq#1367); this is the host's
* safe check until then.
*/
public class GuardedColumnStatsStep extends ColumnStatsStep
{
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
String tableName = input.getValueString("tableName");
String fieldName = input.getValueString("fieldName");
QTableMetaData table = tableName == null ? null : QContext.getQInstance().getTable(tableName);
assertFullAccess(table, fieldName);
super.run(input, output);
if(table != null)
{
for(Serializable value : output.getValues().values())
{
tag(table.getName(), value);
}
}
}
/**
* Refuse statistics for a field the caller cannot read in full; a joined
* field ("otherTable.field") is checked on its own table.
*/
static void assertFullAccess(QTableMetaData table, String fieldName) throws QPermissionDeniedException
{
if(table == null || fieldName == null)
{
return;
}
QTableMetaData fieldTable = table;
String name = fieldName;
int dot = fieldName.indexOf('.');
if(dot > 0 && QContext.getQInstance().getTable(fieldName.substring(0, dot)) != null)
{
fieldTable = QContext.getQInstance().getTable(fieldName.substring(0, dot));
name = fieldName.substring(dot + 1);
}
FieldAccessTier tier = FieldAccessHelper.getTier(fieldTable, name);
if(tier != FieldAccessTier.FULL)
{
String label = fieldTable.getField(name) == null ? name : fieldTable.getField(name).getLabel();
throw new QPermissionDeniedException("Column statistics are not available for " + label
+ ": you do not have full access to this field.");
}
}
private static void tag(String tableName, Serializable value)
{
if(value instanceof QRecord record && record.getTableName() == null)
{
record.setTableName(tableName);
}
else if(value instanceof Collection<?> collection)
{
for(Object element : collection)
{
if(element instanceof QRecord record && record.getTableName() == null)
{
record.setTableName(tableName);
}
}
}
}
}
@@ -26,9 +26,9 @@ public final class ProfileLauncher
static String profile(Map<String, String> environment)
{
String selected = environment.getOrDefault("QQQ_ALL_PROFILE", "core");
if(!"core".equals(selected) && !"full".equals(selected))
if(!"core".equals(selected) && !"full".equals(selected) && !EnterpriseFeatures.PROFILE.equals(selected))
{
throw new IllegalArgumentException("QQQ_ALL_PROFILE must be core or full");
throw new IllegalArgumentException("QQQ_ALL_PROFILE must be core, full or enterprise");
}
return selected;
}
@@ -38,7 +38,7 @@ public final class ProfileLauncher
String selected = profile(environment);
Path dataDirectory = Path.of(environment.getOrDefault("QQQ_ALL_DATA_DIR", "./data"));
Integer port = Integer.valueOf(environment.getOrDefault("QQQ_ALL_PORT", "8080"));
if("core".equals(selected))
if("core".equals(selected) || EnterpriseFeatures.PROFILE.equals(selected))
{
CoreRuntime.launch(dataDirectory, port, environment.getOrDefault("QQQ_ALL_BIND_HOST", "127.0.0.1"));
return;
@@ -7,6 +7,7 @@
package com.kingsrook.qqq.all;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
@@ -110,12 +111,13 @@ public final class QqqAllApplication extends AbstractQQQApplication
instance.addTable(new QTableMetaData().withName("customer").withLabel("Customer")
.withBackendName("h2").withPrimaryKeyField("id")
.withRecordLabelFormat("%s").withRecordLabelFields("name")
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withField(new QFieldMetaData("email", QFieldType.STRING)));
instance.addTable(new QTableMetaData().withName("order").withLabel("Order")
.withBackendName("h2").withBackendDetails(new RDBMSTableBackendDetails().withTableName("orders"))
.withPrimaryKeyField("id")
.withPrimaryKeyField("id").withRecordLabelFormat("%s").withRecordLabelFields("orderNo")
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("orderNo", QFieldType.STRING).withBackendName("order_no"))
.withField(new QFieldMetaData("customerId", QFieldType.INTEGER).withBackendName("customer_id"))
@@ -156,12 +158,16 @@ public final class QqqAllApplication extends AbstractQQQApplication
new EsbJavalinMetaDataProducer().produce(instance);
instance.withSupplementalMetaData(new HealthCheckMetaData().withEnabled(true)
.withEndpointPath("/health").withIndicators(List.of(new MemoryHealthIndicator().withThreshold(99))));
.withEndpointPath("/health").withIndicators(new ArrayList<>(List.of(new MemoryHealthIndicator().withThreshold(99)))));
QJavalinMetaData.ofOrWithNew(instance)
.withAdditionalRouteProviderReference(new QCodeReference(JavalinHealthRouteProvider.class));
if(includeDemoQbits)
{
DemoQBits.addCore(instance, "h2");
if(EnterpriseFeatures.isEnabled())
{
EnterpriseFeatures.add(instance, "h2", httpPort);
}
DemoUsers.protectDemoTables(instance);
QJavalinMetaData.ofOrWithNew(instance)
.withAdditionalRouteProviderReference(new QCodeReference(DemoOrderWebhookReceiver.class));
@@ -0,0 +1,81 @@
/*
* Copyright 2026 Kingsrook, LLC
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0
*/
package com.kingsrook.qqq.all;
import java.nio.file.Path;
import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** The enterprise profile offers column statistics wherever the backend can aggregate. */
class EnterpriseColumnStatsTest
{
@TempDir Path dataDirectory;
@Test
void queryStatsOnAggregatingBackendsOnly() throws Exception
{
QInstance instance = new QqqAllApplication(dataDirectory, "tcp://127.0.0.1:61616").defineQInstance();
for(String tableName : new String[] {"customer", "order", "orderLine", "demoNote"})
{
assertFalse(hasQueryStats(instance, tableName), tableName + " before");
}
EnterpriseFeatures.enableColumnStatistics(instance);
assertNotNull(instance.getProcess("columnStats"));
///////////////////////////////////////////////////////////////////
// H2 (customer, order), SQLite (orderLine) and memory (demoNote) //
///////////////////////////////////////////////////////////////////
for(String tableName : new String[] {"customer", "order", "orderLine", "demoNote"})
{
assertTrue(hasQueryStats(instance, tableName), tableName);
assertTrue(hasCapability(instance, tableName, Capability.TABLE_COUNT), tableName);
assertTrue(hasCapability(instance, tableName, Capability.TABLE_EXPORT), tableName);
}
////////////////////////////////////////////////////////////////
// the filesystem (product) and API (apiCatalog) backends do //
// not aggregate, so they stay without statistics //
////////////////////////////////////////////////////////////////
assertFalse(hasQueryStats(instance, "product"));
assertFalse(hasQueryStats(instance, "apiCatalog"));
}
@Test
void dataTablesAreAuditedWhenAuditsAreInstalled() throws Exception
{
QInstance instance = new QqqAllApplication(dataDirectory, "tcp://127.0.0.1:61616").defineQInstance();
new AuditsMetaDataProvider().defineAll(instance, "h2", null);
EnterpriseFeatures.auditDataTables(instance);
for(String tableName : new String[] {"customer", "order", "orderLine", "address"})
{
assertEquals(AuditLevel.FIELD, instance.getTable(tableName).getAuditRules().getAuditLevel(), tableName);
}
}
private static boolean hasQueryStats(QInstance instance, String tableName)
{
return hasCapability(instance, tableName, Capability.QUERY_STATS);
}
private static boolean hasCapability(QInstance instance, String tableName, Capability capability)
{
QTableMetaData table = instance.getTable(tableName);
QBackendMetaData backend = instance.getBackendForTable(tableName);
return table.isCapabilityEnabled(backend, capability);
}
}
@@ -0,0 +1,113 @@
/*
* Copyright 2026 Kingsrook, LLC
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0
*/
package com.kingsrook.qqq.all;
import java.math.BigDecimal;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.AuthScope;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock;
import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** Column statistics are refused for a field the caller cannot read in full. */
class GuardedColumnStatsStepTest
{
private static final String KEY = "salaryViewer";
private QInstance instance;
@BeforeEach
void setUp() throws QException
{
MemoryRecordStore.fullReset();
instance = new QInstance();
instance.registerAuthenticationProvider(AuthScope.instanceDefault(), new QAuthenticationMetaData().withName("mock").withType(QAuthenticationType.MOCK));
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
instance.addSecurityKeyType(new QSecurityKeyType().withName(KEY));
instance.addTable(new QTableMetaData().withName("employee").withBackendName("memory").withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withField(new QFieldMetaData("salary", QFieldType.DECIMAL).withLabel("Salary")
.withFieldSecurityLock(new FieldSecurityLock().withSecurityKeyType(KEY)
.withDefaultBehavior(FieldSecurityLock.Behavior.MASK).withOverrideValues(List.of(true)))));
EnterpriseFeatures.enableColumnStatistics(instance);
QContext.init(instance, new QSystemUserSession());
new InsertAction().execute(new InsertInput("employee").withRecords(List.of(
new QRecord().withValue("name", "Ada").withValue("salary", new BigDecimal("100")),
new QRecord().withValue("name", "Grace").withValue("salary", new BigDecimal("200")))));
}
@AfterEach
void tearDown()
{
QContext.clear();
MemoryRecordStore.fullReset();
}
@Test
void restrictedFieldIsRefusedWithoutTheTier() throws QException
{
QContext.init(instance, new QSession());
QPermissionDeniedException denied = assertThrows(QPermissionDeniedException.class, () -> run("salary"));
assertTrue(denied.getMessage().contains("Column statistics are not available for Salary"), denied.getMessage());
}
@Test
void restrictedFieldIsAllowedWithTheTier() throws QException
{
QContext.init(instance, new QSession().withSecurityKeyValue(KEY, true));
RunProcessOutput output = run("salary");
assertTrue(output.getException().isEmpty(), () -> output.getException().get().getMessage());
QRecord stats = (QRecord) output.getValues().get("statsRecord");
assertEquals(2, stats.getValueInteger("count"));
assertEquals("employee", stats.getTableName());
}
@Test
void unrestrictedFieldIsAllowed() throws QException
{
QContext.init(instance, new QSession());
RunProcessOutput output = run("name");
assertTrue(output.getException().isEmpty(), () -> output.getException().get().getMessage());
}
private RunProcessOutput run(String fieldName) throws QException
{
RunProcessInput input = new RunProcessInput();
input.setProcessName("columnStats");
input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
input.addValue("tableName", "employee");
input.addValue("fieldName", fieldName);
return (new RunProcessAction().execute(input));
}
}