Compare commits

...
Author SHA1 Message Date
james.maesandCodex 2504129672 ci: use the shared branch-aware CodeQL worker
Co-Authored-By: Codex <noreply@openai.com>
2026-10-21 10:24:19 -05:00
james.maesandCodex b1ede9f887 ci: adopt central orb 0.7.12 and stable gates (#26)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 08:56:56 -05:00
james.maesandCodex 81552784e9 ci: centralize rc release completion in qqq orb (#25)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 02:26:29 -05:00
james.maesandCodex a68d557fb2 ci: adopt shared orb 0.7.11 (#24)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 01:01:55 -05:00
james.maes 36527b06ff Merge pull request #23 from QRun-IO/feature/central-ci
ci: adopt shared QQQ orb standard
2026-10-20 18:57:32 -05:00
james.maesandCodex 7c78003765 ci: adopt verified central QQQ orb 0.7.3
Co-Authored-By: Codex <noreply@openai.com>
2026-10-20 18:52:18 -05:00
james.maes ce127037e4 Merge pull request #19 from QRun-IO/feature/service-principal
fix(schedules): schedules and watches run as the data-analysis service principal
2026-10-16 13:33:34 -05:00
james.maesandClaude Opus 5.5 20545be7fc build: QQQ 4.1.0-RC.19
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 13:19:12 -05:00
james.maes 96d2e48e47 Merge pull request #20 from QRun-IO/feature/data-analysis-business-home
feat(home): move Setup to admin and omit unpermitted home content
2026-10-16 10:19:27 -05:00
james.maesandClaude Sonnet 5.5 1d2c2b0029 feat(home): move Setup to admin and omit unpermitted home content
The Data Analysis home showed the engine Setup card (max rows, sampling,
cache, scheduler state) to every SAVED_READ user, and rendered
"Unavailable: READ permission required." shells for unreadable groups.

- dataAnalysisSetup is off the home widget list; it stays registered, gated
  to data_analysis.admin with DenyBehavior.HIDDEN. Hosts place
  DataAnalysisQBitProducer.SYSTEM_WIDGETS on their Admin area.
- Each home data widget requires READ on its table and is omitted from
  metadata otherwise.
- Overview groups the user cannot read are left out of the payload; no
  "Unavailable" text remains on the home.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-16 09:18:26 -05:00
james.maesandClaude Opus 5.5 21671b3df3 fix(schedules): run schedules and watches as the data-analysis service principal
The self-made data-analysis-scheduler session is rejected under OAuth2,
so every scheduled run failed. With the host's principal declared,
inventory runs as it and owner/recipient runs go on their behalf through
it (current access intersected). Undeclared: unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:27:23 -05:00
james.maes 9d1d712040 Merge pull request #18 from QRun-IO/ci/release-rc-secrets-inherit
ci(release-rc): pass secrets so the shared workflow can sign the RC tag
2026-10-16 08:03:37 -05:00
james.maesandClaude Opus 5.5 2c1fa83cfd ci(release-rc): pass secrets to the shared workflow so it can sign the RC tag
QRun-IO/.github#2 adds a tag-and-release job that signs vX with the
QRUN_TAG_SIGNING_KEY secret after Maven Central confirms the artifact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:19:18 -05:00
james.maes 25d03c1514 Merge pull request #17 from QRun-IO/feature/ui-api-1.6
feat(ui-api): extension for QQQ UI API 1.6.0 (operationIds, BasicErrorResponseV1, QQQ 4.1.0-RC.15)
2026-10-15 16:12:58 -05:00
james.maesandClaude Opus 5.5 442eb4398c chore: target QQQ 4.1.0-RC.15 (UI API 1.6.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:27:05 -05:00
james.maesandClaude Opus 5.5 3aa728d88a feat(ui-api): extension for QQQ UI API 1.6.0 (operationIds, BasicErrorResponseV1, QQQ 4.1.0-RC.14)
Regenerated against UI API 1.6.0: every operation has an operationId and
the standard error responses (BasicErrorResponseV1: error, code, message,
fieldErrors, traceId); the extension's own error schema is kept but
deprecated; hand-written error bodies go through QJavalinUtils so they carry
code/message/traceId. Extension MINOR bumped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:43:37 -05:00
james.maes 51ab908b11 Merge pull request #16 from QRun-IO/fix/producer-version-from-build
fix: qbit metadata version comes from the build
2026-10-15 01:42:56 -05:00
james.maesandClaude Opus 5.5 3474b0c188 fix: the qbit metadata version comes from the build, not a hardcoded constant
The producer read a literal version, so released artifacts reported a stale
or -SNAPSHOT version. qbit-data-analysis.properties (version=${project.version}) is
Maven-filtered and the producer loads it; tests assert producer == bundle ==
artifact version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 01:33:59 -05:00
james.maes b12cadfc75 Merge pull request #15 from QRun-IO/fix/help-bundle-version
fix(help): help bundle version comes from the build, not a hardcoded SNAPSHOT
2026-10-14 22:37:20 -05:00
james.maesandClaude Opus 5.5 6065d2d1c1 fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT
On release branches the bundle's version and the producer's differed, so the
bundle test that asserts they match failed, and released bundles recorded
SNAPSHOT as their provenance. help/*.yaml is now filtered:
version: ${project.version}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 22:06:05 -05:00
james.maes 0e0cc0b83d Merge pull request #13 from QRun-IO/feature/help-bundle
feat(help): qbit-data-analysis help bundle via QQQ help bundles, with coverage gate
2026-10-14 20:45:30 -05:00
james.maesandClaude Opus 5.5 73de290a93 feat(help): ship the qbit-data-analysis help bundle through QQQ help bundles, with coverage gate
Host-neutral help (deployment-specific facts removed), registered with
HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:27:13 -05:00
26 changed files with 2124 additions and 310 deletions

No files matched your search

+11 -57
View File
@@ -1,60 +1,14 @@
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
steps:
- run:
name: Keep package installation unattended
command: |
# These build machines are discarded after the job.
sudo install -d /etc/needrestart/conf.d
printf '%s\n' '$nrconf{restart} = "l";' | sudo tee /etc/needrestart/conf.d/99-qqq-ci.conf >/dev/null
qqq-orb: kingsrook/qqq-orb@0.7.12
workflows:
test_only:
jobs:
- qqq-orb/mvn_test_only:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
filters:
branches:
ignore: /(develop|release\/.*)/
publish_snapshot:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: snapshot
filters:
branches:
only: [develop]
publish_release_candidate:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: release_candidate
filters:
branches:
only: [/release\/.*/]
publish_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: release
filters:
branches:
ignore: /.*/
tags:
only: [/^v\d+\.\d+\.\d+$/]
setup:
jobs:
- qqq-orb/generate_pipeline:
orb_version: 0.7.12
filters:
branches:
ignore: automation/qbit-rc-pending
tags:
only: /^v\d+\.\d+\.\d+$/
+23
View File
@@ -0,0 +1,23 @@
name: CodeQL
'on':
push:
branches:
- main
- develop
- release/**
- hotfix/**
pull_request: {}
schedule:
- cron: 41 16 * * 0
workflow_dispatch: {}
permissions:
contents: read
actions: read
security-events: write
packages: read
concurrency:
group: codeql-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
analyze:
uses: QRun-IO/qqq-orb/.github/workflows/qbit-codeql.yml@c288a8864fe9054f68410e1d78f5863aea4c4d22
+62 -13
View File
@@ -1,25 +1,74 @@
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
# the release line and the release/* push runs CircleCI publish_release_candidate.
name: Release candidate
on:
'on':
workflow_dispatch:
inputs:
dry_run:
description: Print the plan without merging or creating branches
description: Plan a cut without changing release state
type: boolean
default: false
finalize:
description: Retry finalization of an existing cut; do not cut another RC
type: boolean
default: false
recover:
description: Recover the same published RC after a deployment timeout; never
reupload
type: boolean
default: false
circle_build:
description: Original Circle publisher build number (recovery only)
type: string
default: ''
merge_sha:
description: Existing cut merge_sha (finalize retry only)
type: string
default: ''
bump_sha:
description: Existing cut bump_sha (finalize retry only)
type: string
default: ''
branch:
description: Existing cut branch (finalize retry only)
type: string
default: ''
version:
description: Existing cut version (finalize retry only)
type: string
default: ''
pull_request:
types: [closed]
branches: [develop]
types:
- closed
branches:
- develop
repository_dispatch:
types:
- qbit-rc-published
permissions:
contents: write
actions: read
checks: read
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
if: github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch'
|| (github.event_name == 'pull_request' && github.event.pull_request.merged
== true && github.event.pull_request.base.ref == 'develop' && contains(github.event.pull_request.labels.*.name,
'release:rc'))
uses: QRun-IO/qqq-orb/.github/workflows/qbit-release-rc.yml@cc30f599e63c8e41629c99d23395bac1adb8bc44
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || false
}}
finalize: ${{ github.event_name == 'repository_dispatch' || (github.event_name
== 'workflow_dispatch' && inputs.finalize) }}
recover: ${{ github.event_name == 'workflow_dispatch' && inputs.recover || false
}}
circle_build: ${{ github.event_name == 'workflow_dispatch' && inputs.circle_build
|| '' }}
merge_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.merge_sha
|| inputs.merge_sha || '' }}
bump_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.bump_sha
|| inputs.bump_sha || '' }}
branch: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.branch
|| inputs.branch || '' }}
version: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.version
|| inputs.version || '' }}
secrets: inherit
+16
View File
@@ -0,0 +1,16 @@
profile: qbit
orb_version: 0.7.12
default_branch: develop
java_version: '21'
browsers: true
api_checks: false
docs_only: false
release_on_main: false
release_tag_policy: stable
test_branch_policy: non-release
publish:
snapshot: true
release_candidate: true
release: true
asciidoc: false
release_completion: true
+12 -1
View File
@@ -4,8 +4,16 @@ Following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Semantic
## [Unreleased]
### Fixed
- Scheduled analyses and watches work under OAuth2. When the host declares the `data-analysis` service principal (QQQ service identities, `DataAnalysisQBitConfig.withServicePrincipal`), schedule and watch inventory runs as that principal, and owner and recipient runs go on their behalf through it: their current access intersected with the principal's. The self-made `data-analysis-scheduler` session was rejected under OAuth2 (`Invalid session in request` on every `data_analysis_run_schedules` run). With no principal declared, behavior is unchanged. Needs the QQQ release with service identities (qqq#1255, #1256).
- QBit metadata now reports the Maven artifact version, including release versions, from a filtered build resource.
- The Data Analysis home no longer shows the engine Setup card or "Unavailable: READ permission required." shells. Setup is admin-only (hidden from everyone else) and off the home; hosts place `DataAnalysisQBitProducer.SYSTEM_WIDGETS` on their Admin area. Home widgets the user cannot read are omitted from metadata, and unreadable groups are left out of the overview.
### Added
- Complete host-neutral in-app help bundle with live metadata references and a core help coverage gate.
- Analysis Settings table action **Create rows for analyzable tables**: writes the effective settings as editable `data_analysis_table_config` rows (insert-only, no change in behavior).
- Setup card on the Analyses home (settings, result cache, saved analyses, watches, schedules).
- Help bundle `help/qbit-data-analysis.yaml`: every table says what it is for and how its rows get created.
@@ -40,12 +48,15 @@ Following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Semantic
### Changed
- UI API extension data-analysis 1.5.0 (requires QQQ 4.1.0-RC.15, UI API 1.6.0): operationIds, standard error responses with BasicErrorResponseV1, extension tags, contract validation for operationIds and schema references, and HTTP error envelope regression assertions.
- Pin QQQ 4.1.0-RC.7 and replace the temporary help loader with core bundle registration and startup sync.
- Boolean pivot/crosstab headers now say Yes / No / (Blank). Pivot measures use human labels such as Count, Average Amount, Sum of Amount, Distinct Email, Median Amount and 90th percentile of Amount.
- Watch DDL includes owner/date fields and last value/message; existing databases need an owner/date backfill before enabling watches.
- Scalable estimates sample each native aggregate without primary-key IN lists, scale counts using observed sample sizes, retain exact population totals and unscaled ratios, and drill over the full population. Portable estimates use one primary-key boundary and report unscaled prefix counts.
- Target QQQ 4.1.0-RC.6; local verification uses `-Dqqq.snapshot.version=4.1.0-DATAANALYSIS-SNAPSHOT`.
- Empty-string and temporal drills round-trip through core HTTP filter interpretation.
### Previously added
+18 -5
View File
@@ -10,7 +10,7 @@ Twenty-one HTTP operations are available under `/qqq/v1/qbit/data-analysis` (ext
## Requirements
- Java 21; QQQ **4.1.0-RC.6+**.
- Java 21; QQQ **4.1.0-RC.7+**.
- A configured QQQ backend for the qbit tables and authenticated sessions with permissions and security keys.
- Core aggregate and table-feature permission APIs ([qqq#1139](https://github.com/QRun-IO/qqq/pull/1139)) and the UI API extension point ([qqq#1128](https://github.com/QRun-IO/qqq/pull/1128)). Middleware Javalin remains optional; the producer registers endpoints only when it is present.
@@ -187,7 +187,7 @@ Requests and drills reuse core's `QueryFilter` JSON shape and schema reference.
Regenerate deliberately after an API change:
```bash
./mvnw -o -q test -Dtest=DataAnalysisApiTest#contract -Dopenapi.write=true -Dqqq.snapshot.version=4.1.0-DATAANALYSIS-SNAPSHOT
./mvnw -q test -Dtest=DataAnalysisApiTest#contract -Dopenapi.write=true
```
## Pivot backend and field access
@@ -202,7 +202,7 @@ The planner first requests count-ordered top-N groups for every dimension (defau
Hosts can supply `withFieldAnalysisAccessResolver(() -> resolver)`. FULL fields alone appear in capabilities and may be selected. EXACT_MATCH_ONLY and NONE get the same 400 as an unknown field; user-authored filters are passed through for core enforcement. Generated drills using a restricted field (including representative primary keys) are disabled. Related PVS display labels also require FULL access. Exact-run cache identities hash the sorted FULL-field list, the optional core access fingerprint, and related label access.
The default resolver discovers the optional core `fieldaccess.FieldAccessHelper` class and its static methods once. FULL maps to FULL, REVEALABLE to EXACT_MATCH_ONLY, MASKED/HIDDEN to NONE. Without that class, legacy field locks supply FULL/NONE. Replace the reflective calls when the core package ships. Reflection tests use a local stand-in helper rather than shadowing a future real core class at its FQN; the installed snapshot exercises the actual fallback selection. Exact-run persistence continues to support profiles only; pivot results are synchronous.
The default resolver discovers the optional core `fieldaccess.FieldAccessHelper` class and its static methods once. FULL maps to FULL, REVEALABLE to EXACT_MATCH_ONLY, MASKED/HIDDEN to NONE. Without that class, legacy field locks supply FULL/NONE. Replace the reflective calls when the core package ships. Reflection tests use a local stand-in helper rather than shadowing a future real core class at its FQN; the pinned RC.7 exercises the actual core tier resolver selection. Exact-run persistence continues to support profiles only; pivot results are synchronous.
## Backend selection
@@ -304,13 +304,26 @@ The Analyses home uses standard `multiStatistics` and `table` widgets: mine/shar
Backend phases **b–f** are implemented, including saved/shared/starred analyses, schedules, standard widgets, home, export and SQL handoff. The separate profile, pivot and relationship frontend remains outside this repository.
## In-app help
The qbit registers `src/main/resources/help/qbit-data-analysis.yaml` as a QBIT bundle.
Hosts with the core `helpContent` table receive it through QQQ's startup sync; registration
is harmless without that table. The app help follows `appName` when you configure a custom name.
To override text, register your own `layer: HOST` bundle with the same entry key and role
(using your configured app name for app help). HOST entries take precedence over QBIT entries;
QQQ preserves manually edited help during sync.
`DataAnalysisTablesGuardrailTest` uses `HelpBundleAssertions` to check owned metadata,
summary length and live references, and checks widget and possible-value-source coverage.
Update the bundle's literal version alongside the POM version when releasing.
## Build
```bash
./mvnw -o -q verify -Dqqq.snapshot.version=4.1.0-DATAANALYSIS-SNAPSHOT
./mvnw -q verify
```
The wrapper pins Maven 3.9.11. On Windows use `mvnw.cmd`. Verification runs Checkstyle, a shared JUnit 5/AssertJ contract suite on memory and H2, PostgreSQL via Testcontainers when Docker is available, and JaCoCo gates of 70% instructions and 90% classes. The suite checks numerical references, every exact count/drill pair, full-population estimate drills, security, guarded cache JSON, native sampling, asynchronous jobs and cache invalidation. The POM targets RC.6; until its release, always supply the locally installed snapshot override shown above. RDBMS, H2, Testcontainers PostgreSQL and the PostgreSQL driver are test-scoped; middleware Javalin is optional.
The wrapper pins Maven 3.9.11. On Windows use `mvnw.cmd`. Verification runs Checkstyle, a shared JUnit 5/AssertJ contract suite on memory and H2, PostgreSQL via Testcontainers when Docker is available, and JaCoCo gates of 70% instructions and 90% classes. The suite checks numerical references, every exact count/drill pair, full-population estimate drills, security, guarded cache JSON, native sampling, asynchronous jobs and cache invalidation. The POM pins QQQ 4.1.0-RC.7 from Maven Central. RDBMS, H2, Testcontainers PostgreSQL and the PostgreSQL driver are test-scoped; middleware Javalin is optional.
## License
+29 -1
View File
@@ -128,7 +128,35 @@
</dependencies>
<build>
<resources>
<!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
<includes>
<include>help/*.yaml</include>
<include>qbit-data-analysis.properties</include>
</includes>
</resource>
<resource>
<directory>src/main/resources</directory>
<filtering>false</filtering>
<excludes>
<exclude>help/*.yaml</exclude>
<exclude>qbit-data-analysis.properties</exclude>
</excludes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables combine.children="append">
<qbit.artifactVersion>${project.version}</qbit.artifactVersion>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
@@ -215,7 +243,7 @@
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.19</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -427,6 +427,28 @@ public class DataAnalysisQBitConfig implements QBitConfig
return (this);
}
private String servicePrincipal = "data-analysis";
/*******************************************************************************
** The service principal schedules and watches run as when the host declares
** it (QQQ service identities): inventory as the principal, owner and
** recipient runs on their behalf through it. Undeclared: the previous local
** sessions.
*******************************************************************************/
public String getServicePrincipal()
{
return (servicePrincipal);
}
/*******************************************************************************
** Fluent setter for servicePrincipal (default data-analysis).
*******************************************************************************/
public DataAnalysisQBitConfig withServicePrincipal(String servicePrincipal)
{
this.servicePrincipal = servicePrincipal;
return (this);
}
private String messagingProviderName;
private String fromAddress;
private transient java.util.function.Function<String, String> recipientAddressSupplier;
@@ -21,10 +21,16 @@
package com.kingsrook.qbits.dataanalysis;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Properties;
import com.kingsrook.qbits.dataanalysis.api.DataAnalysisApi;
import com.kingsrook.qbits.dataanalysis.app.DataAnalysisHome;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermission;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermissionsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
@@ -39,13 +45,48 @@ public class DataAnalysisQBitProducer implements QBitMetaDataProducer<DataAnalys
{
public static final String GROUP_ID = "com.kingsrook.qbits";
public static final String ARTIFACT_ID = "qbit-data-analysis";
public static final String VERSION = "0.1.0";
public static final String VERSION = loadVersion();
/////////////////////////////////////////////////////////////////////////////////
// Widgets that show engine configuration. They are registered with the qbit //
// (admin-only, hidden from everyone else) but kept off the business home; the //
// host places them on its Admin area, e.g. withWidgets(SYSTEM_WIDGETS). //
/////////////////////////////////////////////////////////////////////////////////
public static final List<String> SYSTEM_WIDGETS = List.of(DataAnalysisHome.SETUP);
private DataAnalysisQBitConfig qBitConfig;
private String namespace;
/*******************************************************************************
** Read the artifact version from the Maven-filtered properties resource.
*******************************************************************************/
private static String loadVersion()
{
try(InputStream inputStream = DataAnalysisQBitProducer.class.getClassLoader().getResourceAsStream("qbit-data-analysis.properties"))
{
if(inputStream == null)
{
throw (new IllegalStateException("Missing qbit-data-analysis.properties; build with Maven to generate the artifact version"));
}
Properties properties = new Properties();
properties.load(inputStream);
String version = properties.getProperty("version");
if(version == null || version.isBlank() || version.contains("${"))
{
throw (new IllegalStateException("Missing or unfiltered version in qbit-data-analysis.properties; build with Maven resource filtering"));
}
return (version);
}
catch(IOException e)
{
throw (new IllegalStateException("Could not read artifact version from qbit-data-analysis.properties", e));
}
}
/*******************************************************************************
** Fluent configuration setter.
*******************************************************************************/
@@ -171,7 +212,11 @@ public class DataAnalysisQBitProducer implements QBitMetaDataProducer<DataAnalys
{
instance.getApp(qBitConfig.getParentAppName()).withChild(instance.getApp(qBitConfig.getAppName()));
}
com.kingsrook.qbits.dataanalysis.help.DataAnalysisHelp.apply(instance, qBitConfig);
// Registration is metadata-only; core startup sync guards hosts without helpContent.
HelpBundles.register(instance, "help/qbit-data-analysis.yaml");
HelpBundles.of(instance).getBundles().getLast().getEntries().stream()
.filter(entry -> "app:dataAnalysis;slot:help".equals(entry.getKey()))
.forEach(entry -> entry.setKey("app:" + qBitConfig.getAppName() + ";slot:help"));
return (new MetaDataProducerMultiOutput());
}
}
@@ -76,7 +76,7 @@ public class DataAnalysisApi
*******************************************************************************/
public static MiddlewareSpecExtension extension()
{
return (new MiddlewareSpecExtension().withName("data-analysis").withSemanticVersion("1.4.0")
return (new MiddlewareSpecExtension().withName("data-analysis").withSemanticVersion("1.5.0")
.withDescription("Profiles, pivots, comparisons, distribution shifts, correlations, scatter samples, crosstabs and anomaly watch previews as the signed-in user.")
.withSpec(SavedApi.Run.class).withSpec(SavedApi.ListSaved.class).withSpec(SavedApi.Star.class).withSpec(SavedApi.Unstar.class)
.withSpec(SavedApi.Sql.class).withSpec(AnalysisFileSpec.class).withSpec(AnalysisFileSpec.Saved.class)
@@ -28,14 +28,17 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.dataanalysis.DataAnalysisQBitConfig;
import com.kingsrook.qbits.dataanalysis.DataAnalysisQBitProducer;
import com.kingsrook.qbits.dataanalysis.DataAnalysisRuntime;
import com.kingsrook.qbits.dataanalysis.engine.DrillFilters;
import com.kingsrook.qbits.dataanalysis.model.DataAnalysisTables;
import com.kingsrook.qbits.dataanalysis.permissions.DataAnalysisPermissions;
import com.kingsrook.qbits.dataanalysis.saved.SavedAnalyses;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.actions.permissions.UseOtherPermissionNameCustomPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -54,6 +57,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.DenyBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@@ -82,7 +86,39 @@ public class DataAnalysisHome extends AbstractWidgetRenderer
*******************************************************************************/
public static List<QWidgetMetaData> widgets()
{
return (List.of(widget(SETUP, "Setup", WidgetType.TABLE), widget(), widget(STARRED, "Starred analyses", WidgetType.TABLE), widget(FIRING, "Firing watches", WidgetType.TABLE), widget(RECENT, "My recent runs", WidgetType.TABLE)));
return (List.of(setupWidget(), widget().withPermissionRules(readRules(DataAnalysisTables.SAVED)),
widget(STARRED, "Starred analyses", WidgetType.TABLE).withPermissionRules(readRules(DataAnalysisTables.SAVED)),
widget(FIRING, "Firing watches", WidgetType.TABLE).withPermissionRules(readRules(DataAnalysisTables.WATCH)),
widget(RECENT, "My recent runs", WidgetType.TABLE).withPermissionRules(readRules(DataAnalysisTables.RUN))));
}
/*******************************************************************************
** Names of the widgets placed on the business home. Setup is system content and
** is deliberately absent; hosts place {@link DataAnalysisQBitProducer#SYSTEM_WIDGETS}
** on their Admin area instead.
*******************************************************************************/
public static List<String> homeWidgetNames()
{
return (widgets().stream().map(QWidgetMetaData::getName).filter(name -> !DataAnalysisQBitProducer.SYSTEM_WIDGETS.contains(name)).toList());
}
/*******************************************************************************
** The Setup card shows engine configuration, so only Data Analysis admins get it;
** everyone else has it omitted from metadata.
*******************************************************************************/
private static QWidgetMetaData setupWidget()
{
return (widget(SETUP, "Setup", WidgetType.TABLE).withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
.withDenyBehavior(DenyBehavior.HIDDEN).withCustomPermissionChecker(UseOtherPermissionNameCustomPermissionChecker.build(DataAnalysisPermissions.ADMIN))));
}
/*******************************************************************************
** A data widget is omitted from metadata for users who cannot read its table.
*******************************************************************************/
private static QPermissionRules readRules(String table)
{
return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION).withDenyBehavior(DenyBehavior.HIDDEN)
.withCustomPermissionChecker(TableReadWidgetPermissionChecker.build(table)));
}
/*******************************************************************************
@@ -101,7 +137,7 @@ public class DataAnalysisHome extends AbstractWidgetRenderer
public static QAppMetaData app(DataAnalysisQBitConfig config, List<QTableMetaData> tables)
{
QAppMetaData app = new QAppMetaData().withName(config.getAppName()).withLabel(config.getAppLabel())
.withIcon(new QIcon("insights")).withWidgets(widgets().stream().map(QWidgetMetaData::getName).toList())
.withIcon(new QIcon("insights")).withWidgets(homeWidgetNames())
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED))
.withSection(new QAppSection().withName("analyses").withLabel("Analyses").withIcon(new QIcon("insights")).withTables(DataAnalysisTables.NAV_TABLES));
app.setParentAppName(config.getParentAppName());
@@ -197,46 +233,45 @@ public class DataAnalysisHome extends AbstractWidgetRenderer
}
/*******************************************************************************
** One standard group per subject, with real counts or an unavailable explanation.
** One standard group per subject the viewer can read; unreadable subjects are left out.
*******************************************************************************/
public static MultiStatisticsData statistics(Instant now) throws QException
{
List<MultiStatisticsData.StatisticsGroupData> groups = new ArrayList<>();
var saved = group("Saved analyses", DataAnalysisTables.SAVED);
if(canRead(DataAnalysisTables.SAVED))
{
var saved = group("Saved analyses");
List<SavedAnalyses.Entry> entries = DataAnalysisRuntime.saved().list();
String user = SavedAnalyses.user();
saved.withStatisticList(List.of(stat("Mine", (int) entries.stream().filter(e -> e.owner().equals(user)).count(), DataAnalysisTables.SAVED),
stat("Shared", (int) entries.stream().filter(e -> Boolean.TRUE.equals(e.shared())).count(), DataAnalysisTables.SAVED),
stat("Starred", (int) entries.stream().filter(SavedAnalyses.Entry::starred).count(), DataAnalysisTables.SAVED)));
groups.add(saved);
}
groups.add(saved);
var watches = group("Watches", DataAnalysisTables.WATCH);
if(canRead(DataAnalysisTables.WATCH))
{
var watches = group("Watches");
watches.withStatisticList(List.of(stat("Enabled", count(DataAnalysisTables.WATCH, equal("enabled", true)), DataAnalysisTables.WATCH),
stat("Firing", count(DataAnalysisTables.WATCH, equal("state", "FIRING")), DataAnalysisTables.WATCH)));
groups.add(watches);
}
groups.add(watches);
var runs = group("My runs", DataAnalysisTables.RUN);
if(canRead(DataAnalysisTables.RUN))
{
var runs = group("My runs");
runs.withStatisticList(List.of(stat("Last 7 days", count(DataAnalysisTables.RUN, equal("user_id", SavedAnalyses.user())
.withCriteria("started_at", QCriteriaOperator.GREATER_THAN_OR_EQUALS, now.minus(7, ChronoUnit.DAYS))
.withCriteria("started_at", QCriteriaOperator.LESS_THAN_OR_EQUALS, now)), DataAnalysisTables.RUN)));
groups.add(runs);
}
groups.add(runs);
return (new MultiStatisticsData("Analyses", groups));
}
/*******************************************************************************
** Empty/denied data are different messages, not synthetic zeroes.
** A group is only built for a readable subject, so its subheader is always true.
*******************************************************************************/
private static MultiStatisticsData.StatisticsGroupData group(String label, String table)
private static MultiStatisticsData.StatisticsGroupData group(String label)
{
return (new MultiStatisticsData.StatisticsGroupData().withHeader(label).withStatisticList(List.of())
.withSubheader(canRead(table) ? "Visible to you" : "Unavailable: READ permission required."));
return (new MultiStatisticsData.StatisticsGroupData().withHeader(label).withStatisticList(List.of()).withSubheader("Visible to you"));
}
/*******************************************************************************
@@ -257,7 +292,7 @@ public class DataAnalysisHome extends AbstractWidgetRenderer
{
columns.add(new TableData.Column(i == 0 ? "html" : "text", headers[i], "c" + i, null, "left"));
}
return (new TableData(label, columns, new ArrayList<>()).withNoRowsFoundHTML(canRead(table) ? empty : "Unavailable: READ permission required.")
return (new TableData(label, columns, new ArrayList<>()).withNoRowsFoundHTML(empty)
.withLinkText("View all").withLinkURL(QContext.getQInstance().getTablePath(table)).withRowsPerPage(10));
}
@@ -0,0 +1,88 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.dataanalysis.app;
import java.io.Serializable;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.AbstractTableActionInput;
import com.kingsrook.qqq.backend.core.model.metadata.code.InitializableViaCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceWithProperties;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import com.kingsrook.qqq.backend.core.utils.collections.MapBuilder;
/*******************************************************************************
** A widget is available only to sessions that can READ the table it summarizes.
** Contributes no catalog entry of its own: the table's READ permission is the
** permission.
*******************************************************************************/
public class TableReadWidgetPermissionChecker implements CustomPermissionChecker, InitializableViaCodeReference
{
private static final String TABLE_NAME_PROPERTY = "tableName";
private String tableName;
/*******************************************************************************
** Code reference for the given table.
*******************************************************************************/
public static QCodeReferenceWithProperties build(String tableName)
{
return (new QCodeReferenceWithProperties(TableReadWidgetPermissionChecker.class, MapBuilder.of(TABLE_NAME_PROPERTY, tableName)));
}
/*******************************************************************************
** Read the table name from the code reference properties.
*******************************************************************************/
@Override
public void initialize(QCodeReference codeReference)
{
if(codeReference instanceof QCodeReferenceWithProperties withProperties)
{
Map<String, Serializable> properties = withProperties.getProperties();
tableName = ValueUtils.getValueAsString(properties.get(TABLE_NAME_PROPERTY));
}
}
/*******************************************************************************
** Deny unless the session may READ the table.
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput actionInput, MetaDataWithPermissionRules metaDataWithPermissionRules) throws QPermissionDeniedException
{
AbstractTableActionInput tableInput = new AbstractTableActionInput();
tableInput.setTableName(tableName);
PermissionsHelper.checkTablePermissionThrowing(tableInput, TablePermissionSubType.READ);
}
}
@@ -1,132 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.dataanalysis.help;
import java.io.IOException;
import java.io.InputStream;
import java.util.LinkedHashMap;
import java.util.Map;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import com.kingsrook.qbits.dataanalysis.DataAnalysisQBitConfig;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Applies help/qbit-data-analysis.yaml (the QQQ help-bundle format) to the
** Analyses app and its data_analysis_* tables: what each table is for and
** how its rows get created, shown on the table screen and when it is empty.
** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with HelpBundles.register once the pinned
** QQQ release includes it; the bundle file stays the same.
*******************************************************************************/
public final class DataAnalysisHelp
{
public static final String RESOURCE = "/help/qbit-data-analysis.yaml";
public static final String SLOT = "help";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private DataAnalysisHelp()
{
}
/*******************************************************************************
** The bundle's entries by key, validated (fails startup if malformed).
*******************************************************************************/
public static Map<String, JsonNode> entries() throws QException
{
try(InputStream stream = DataAnalysisHelp.class.getResourceAsStream(RESOURCE))
{
if(stream == null)
{
throw (new IOException("missing resource"));
}
JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream);
if(!"qbit-data-analysis".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText()))
{
throw (new IOException("invalid bundle header"));
}
Map<String, JsonNode> entries = new LinkedHashMap<>();
for(JsonNode entry : bundle.path("entries"))
{
String key = entry.path("key").asText();
String summary = entry.path("summary").asText();
if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null)
{
throw (new IOException("invalid or duplicate entry: " + key));
}
}
return (entries);
}
catch(IOException | RuntimeException e)
{
throw (new QException("Could not load " + RESOURCE, e));
}
}
/*******************************************************************************
** Apply the help to the app and every data_analysis_* table in the instance.
** A host's own help on a table is kept (the bundle only fills the slot).
*******************************************************************************/
public static void apply(QInstance instance, DataAnalysisQBitConfig config) throws QException
{
Map<String, JsonNode> entries = entries();
for(QTableMetaData table : instance.getTables().values())
{
JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT)))
{
table.withHelpContent(SLOT, content(entry));
}
}
QAppMetaData app = instance.getApp(config.getAppName());
JsonNode appEntry = entries.get("app:dataAnalysis;slot:" + SLOT);
if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT)))
{
app.withHelpContent(SLOT, content(appEntry));
}
}
/*******************************************************************************
** Summary as the first paragraph, then the content, as markdown.
*******************************************************************************/
static QHelpContent content(JsonNode entry)
{
return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip()));
}
}
@@ -0,0 +1,90 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.dataanalysis.schedule;
import java.util.function.Supplier;
import com.kingsrook.qbits.dataanalysis.DataAnalysisQBitConfig;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher;
import com.kingsrook.qqq.backend.core.serviceidentity.ServiceIdentities;
/*******************************************************************************
** Sessions for scheduled analysis work. With the host's data-analysis service
** principal declared (QQQ service identities): inventory runs as the
** principal, and owner or recipient runs go on their behalf through it (their
** current access intersected with the principal's). Without it: the previous
** local sessions (single-node default unchanged). Never a system session.
*******************************************************************************/
public final class AnalysisSessions
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private AnalysisSessions()
{
}
/*******************************************************************************
** The session for schedule/watch inventory and status updates.
*******************************************************************************/
public static QSession maintenance(DataAnalysisQBitConfig config, Supplier<QSession> legacy) throws QException
{
if(usesPrincipal(config))
{
return (ServiceIdentities.session(QContext.getQInstance(), config.getServicePrincipal()));
}
return (legacy.get());
}
/*******************************************************************************
** The session for running as a user (owner or recipient).
*******************************************************************************/
public static QSession forUser(DataAnalysisQBitConfig config, String userId) throws QException
{
if(config.getOwnerSessionSupplier() != null)
{
return (config.getOwnerSessionSupplier().apply(userId));
}
if(usesPrincipal(config))
{
return (ServiceIdentities.sessionOnBehalfOf(QContext.getQInstance(), config.getServicePrincipal(), userId));
}
return (new QAuthenticationModuleDispatcher().getQModule(QContext.getQInstance().getAuthentication()).createAutomatedSessionForUser(QContext.getQInstance(), userId));
}
/*******************************************************************************
** Whether the host declared the configured principal.
*******************************************************************************/
private static boolean usesPrincipal(DataAnalysisQBitConfig config)
{
return (config.getServicePrincipal() != null && ServiceIdentities.isDeclared(QContext.getQInstance(), config.getServicePrincipal()));
}
}
@@ -58,7 +58,6 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QUser;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher;
/*******************************************************************************
** Scheduled work uses owner and recipient sessions; maintenance only inventories rows.
@@ -83,13 +82,17 @@ public class ScheduleRunner
*******************************************************************************/
public synchronized Integer runDue(Instant now) throws QException
{
QSession maintenance = new QSession().withUser(new QUser().withIdReference("data-analysis-scheduler"))
.withPermissions("data_analysis.saved.read", "data_analysis.saved.edit")
.withSecurityKeyValue(DataAnalysisPermissions.ADMIN, true);
if(QContext.getQSession().getBackendVariants() != null)
QSession maintenance = AnalysisSessions.maintenance(config, () ->
{
maintenance.setBackendVariants(new java.util.HashMap<>(QContext.getQSession().getBackendVariants()));
}
QSession legacy = new QSession().withUser(new QUser().withIdReference("data-analysis-scheduler"))
.withPermissions("data_analysis.saved.read", "data_analysis.saved.edit")
.withSecurityKeyValue(DataAnalysisPermissions.ADMIN, true);
if(QContext.getQSession().getBackendVariants() != null)
{
legacy.setBackendVariants(new java.util.HashMap<>(QContext.getQSession().getBackendVariants()));
}
return (legacy);
});
int[] count = {0};
QContext.withTemporaryContext(new CapturedContext(QContext.getQInstance(), maintenance), () ->
{
@@ -161,9 +164,7 @@ public class ScheduleRunner
QSession session;
try
{
session = config.getOwnerSessionSupplier() == null
? new QAuthenticationModuleDispatcher().getQModule(QContext.getQInstance().getAuthentication()).createAutomatedSessionForUser(QContext.getQInstance(), id)
: config.getOwnerSessionSupplier().apply(id);
session = AnalysisSessions.forUser(config, id);
}
catch(Exception e)
{
@@ -49,7 +49,6 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QUser;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher;
/*******************************************************************************
** Transition watches through the core alerts SPI, computing exclusively as owner.
@@ -94,13 +93,17 @@ public class WatchEvaluator
*******************************************************************************/
public synchronized Integer evaluateEnabled(Instant now) throws QException
{
QSession maintenance = new QSession().withUser(new QUser().withIdReference("data-analysis-scheduler"))
.withPermissions("data_analysis.watch.read", "data_analysis.watch.edit")
.withSecurityKeyValue(DataAnalysisPermissions.ADMIN, true);
if(QContext.getQSession().getBackendVariants() != null)
QSession maintenance = com.kingsrook.qbits.dataanalysis.schedule.AnalysisSessions.maintenance(config, () ->
{
maintenance.setBackendVariants(new java.util.HashMap<>(QContext.getQSession().getBackendVariants()));
}
QSession legacy = new QSession().withUser(new QUser().withIdReference("data-analysis-scheduler"))
.withPermissions("data_analysis.watch.read", "data_analysis.watch.edit")
.withSecurityKeyValue(DataAnalysisPermissions.ADMIN, true);
if(QContext.getQSession().getBackendVariants() != null)
{
legacy.setBackendVariants(new java.util.HashMap<>(QContext.getQSession().getBackendVariants()));
}
return (legacy);
});
int[] count = {0};
QContext.withTemporaryContext(new CapturedContext(QContext.getQInstance(), maintenance), () ->
{
@@ -128,9 +131,7 @@ public class WatchEvaluator
QSession session;
try
{
session = config.getOwnerSessionSupplier() == null
? new QAuthenticationModuleDispatcher().getQModule(QContext.getQInstance().getAuthentication()).createAutomatedSessionForUser(QContext.getQInstance(), owner)
: config.getOwnerSessionSupplier().apply(owner);
session = com.kingsrook.qbits.dataanalysis.schedule.AnalysisSessions.forUser(config, owner);
}
catch(Exception e)
{
+399 -34
View File
@@ -1,43 +1,408 @@
bundle: qbit-data-analysis
version: 1
version: ${project.version}
layer: QBIT
# Help for the Analyses app and every data_analysis_* table, in the QQQ help-bundle
# format (qqq#1198). Each table entry says what the table is for and how its rows get
# created, so an empty table explains itself. DataAnalysisHelp applies it until the core
# HelpBundles registry ships in a QQQ release (STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8)).
# Summaries stay under 120 characters (DataAnalysisTablesGuardrailTest).
entries:
- key: app:dataAnalysis;slot:help
summary: "Profile, pivot, compare and watch any table, and tune how analysis behaves per table."
content: |
The Setup card on the home lists each part, its state and the next step. Analysis works on every analyzable table from the start with built-in limits, so nothing is required. Open Analysis Settings and run Create rows for analyzable tables to see those limits as rows you can edit per table.
- key: widget:data_analysis_saved_to_data_analysis_saved_star;slot:label
summary: "See stars associated with this saved analysis."
- key: widget:data_analysis_saved_to_data_analysis_schedule;slot:label
summary: "Review delivery schedules for this saved analysis."
- key: widget:data_analysis_saved_to_data_analysis_watch;slot:label
summary: "Review watches associated with this saved analysis."
- key: widget:dataAnalysisHome;slot:label
summary: "See an overview of saved analyses, watches and runs you can access."
- key: widget:dataAnalysisStarred;slot:label
summary: "Open saved analyses you have starred."
- key: widget:dataAnalysisFiring;slot:label
summary: "Review watches that are currently firing."
- key: widget:dataAnalysisRecent;slot:label
summary: "Review your recent analysis runs."
- key: widget:DATA_ANALYSIS_SAVED;slot:label
summary: "View the chart produced by a saved analysis."
- key: app:dataAnalysis;slot:help
summary: "Explore your data, revisit saved analyses and monitor changes with watches."
content: |
Use Analyses to return to saved work, view starred analyses and review watches that are firing. Analyze a table to explore its records, then save the choices you want to revisit. Analysis Settings controls availability and limits for each table. Analysis Runs helps you review exact results and investigate failures.
- key: table:data_analysis_table_config;slot:help
summary: "Control analysis availability, row limits, sampling and result reuse for each table."
content: |
**How rows get created:** Run **Create rows for analyzable tables** to add the effective settings for tables that do not have a row, or choose New for one table. Existing rows are left unchanged. Without a row, your configured defaults apply.
- key: table:data_analysis_table_config;slot:help
summary: "Per-table analysis limits: on or off, row limits, sampling, smallest group, cache minutes and field lists."
content: |
**How rows get created:** run the table action Create rows for analyzable tables once. It adds one row per table holding the settings that already apply, so nothing changes until you edit a row. Or add a row with New for just one table. A table without a row uses the defaults from the configuration, so an empty table means every table is on with the defaults. Cache minutes controls how long an exact result is reused (0 turns caching off). Smallest group hides groups with fewer records.
Use Analysis Settings to adjust how a table can be analyzed. You can enable analysis, set row and sample limits, choose allowed or excluded fields, and hide groups smaller than a chosen size. Cache Ttl Minutes controls how long completed exact results can be reused. Tables without an entry use the configured defaults.
- key: table:data_analysis_table_config;field:table_name
summary: "The table whose records you want to analyze."
- key: table:data_analysis_table_config;field:enabled
summary: "Whether you can analyze records from this table."
- key: table:data_analysis_table_config;field:max_rows
summary: "The row limit used when analysis reads individual records."
- key: table:data_analysis_table_config;field:sample_threshold_rows
summary: "The row count above which an estimated analysis may use a sample."
- key: table:data_analysis_table_config;field:sample_rows
summary: "The number of rows to include in a sample."
- key: table:data_analysis_table_config;field:min_group_size
summary: "The smallest group of records that may appear in results."
content: |
Min Group Size prevents very small groups from appearing in analysis results. Choose the minimum number of records a group must contain before it can be shown. If expected categories are missing, review this limit along with your filters before assuming that the underlying records are absent.
- key: table:data_analysis_table_config;field:analyzable_fields
summary: "The fields allowed to be analyzed for this table."
- key: table:data_analysis_table_config;field:excluded_fields
summary: "The fields excluded from analysis for this table."
- key: table:data_analysis_table_config;field:cache_ttl_minutes
summary: "How many minutes completed exact results can be reused; zero turns reuse off."
- key: table:data_analysis_saved;slot:help
summary: "Return to analyses you saved with their table, filters, view and chart choices."
content: |
**How rows get created:** Save your analysis choices to create a saved analysis. An empty list means you have no saved analyses available to you.
- key: table:data_analysis_saved;slot:help
summary: "Analyses people saved: the table, filter and view they want to come back to."
content: |
**How rows get created:** a person saves an analysis from the Analyze screen on any table. Each saved analysis belongs to its owner; stars, schedules and watches hang off it. Empty means nobody has saved one yet.
Save an analysis from Analyze to keep the table, filters and analysis choices you want to revisit. Each saved analysis has an owner and can be starred for quick access. Open its Schedules and Watches lists to review related deliveries and checks. Sharing the definition does not remove access restrictions on the underlying data.
- key: table:data_analysis_saved;field:name
summary: "The name you use to find this saved analysis."
- key: table:data_analysis_saved;field:description
summary: "A description that helps you understand this entry."
- key: table:data_analysis_saved;field:table_name
summary: "The table whose records you want to analyze."
- key: table:data_analysis_saved;field:filter_json
summary: "The saved record filters, stored as JSON."
- key: table:data_analysis_saved;field:kind
summary: "The type of analysis, such as a profile, pivot or comparison."
- key: table:data_analysis_saved;field:definition_json
summary: "The saved analysis choices, stored as JSON."
- key: table:data_analysis_saved;field:chart_kind
summary: "The chart style used to display the analysis."
- key: table:data_analysis_saved;field:owner_user_id
summary: "The person who owns and manages this analysis or watch."
- key: table:data_analysis_saved;field:shared
summary: "Whether other permitted people can read the saved analysis definition."
- key: table:data_analysis_saved_star;slot:help
summary: "Keep track of the saved analyses you have starred."
content: |
**How rows get created:** Star a saved analysis to create your personal star. An empty list means you have no stars available here.
- key: table:data_analysis_saved_star;slot:help
summary: "Which saved analyses each person starred."
content: |
**How rows get created:** a person stars a saved analysis, and the row is theirs. Starred analyses show on the Analyses home. Empty means nothing is starred yet.
Star a saved analysis to make it easier to find on the Analyses home. Each star links one person to one saved analysis, so your starred list is personal. These entries are created through the star control. An empty list means you have not starred an analysis available here.
- key: table:data_analysis_saved_star;field:saved_id
summary: "The saved analysis this entry belongs to."
- key: table:data_analysis_saved_star;field:user_id
summary: "The person this entry belongs to."
- key: table:data_analysis_schedule;slot:help
summary: "Run saved analyses at chosen times and deliver their results to selected people."
content: |
**How rows get created:** Add a schedule to a saved analysis to create a delivery definition. An empty list means you have no schedules available here.
- key: table:data_analysis_schedule;slot:help
summary: "Schedules that run a saved analysis daily, weekly or monthly and send the result."
content: |
**How rows get created:** a person adds a schedule on a saved analysis (the Schedules list on the saved analysis). Schedules run only when the host configures a scheduler for the qbit. Empty means no analysis is scheduled.
Add a schedule to a saved analysis to send results daily, weekly or monthly. Choose the local time, time zone, recipients and output format, then enable it when ready. Automatic runs require scheduling to be configured. Use Send now for an immediate delivery and review Last Status for the outcome.
- key: table:data_analysis_schedule;field:saved_id
summary: "The saved analysis this entry belongs to."
- key: table:data_analysis_schedule;field:cron
summary: "An advanced schedule expression that takes precedence over Frequency, Day and Time."
content: |
Leave Cron blank when you want to schedule deliveries with Frequency, Day and Time. If you supply an advanced expression, it uses the schedule's Time Zone and replaces those calendar choices. You must use a valid Quartz schedule expression. Check Next Run At after saving to confirm the resulting delivery time.
- key: table:data_analysis_schedule;field:frequency
summary: "Choose daily, weekly or monthly delivery."
- key: table:data_analysis_schedule;field:day
summary: "The weekday number or day of the month used by the schedule."
content: |
For a weekly schedule, enter a weekday number from 1 for Monday through 7 for Sunday. For a monthly schedule, enter a day from 1 through 31. If that day does not exist in a month, delivery uses the last day of that month. Daily schedules do not need a day.
- key: table:data_analysis_schedule;field:time
summary: "The local time of day when the analysis should run."
- key: table:data_analysis_schedule;field:time_zone
summary: "The time zone used to interpret the scheduled time."
- key: table:data_analysis_schedule;field:format
summary: "Choose an HTML summary, CSV file or Excel workbook for delivery."
- key: table:data_analysis_schedule;field:recipients
summary: "The people receiving the result, listed by user ID in a JSON array."
content: |
Recipients accepts a list of user IDs, not email addresses. Supply a JSON array containing between one and one hundred nonblank IDs. Duplicate IDs are treated as one recipient. Review the people in the list before you enable the schedule or choose Send now, since those actions deliver the analysis result.
- key: table:data_analysis_schedule;field:enabled
summary: "Whether the schedule is allowed to run automatically."
- key: table:data_analysis_schedule;field:last_run_at
summary: "When the schedule last ran."
- key: table:data_analysis_schedule;field:last_status
summary: "The outcome of the most recent scheduled delivery."
- key: table:data_analysis_schedule;field:next_run_at
summary: "When the schedule is next due to run."
- key: table:data_analysis_watch;slot:help
summary: "Check analysis results for spikes, drops, new categories or other selected conditions."
content: |
**How rows get created:** Add a watch for a saved analysis or a table and metric. An empty list means you have no watches available here.
- key: table:data_analysis_watch;slot:help
summary: "Watches that alert when a saved analysis spikes, drops, goes out of range or gets a new category."
content: |
**How rows get created:** a person adds a watch on a saved analysis (the Watches list on the saved analysis). Enabled watches are evaluated every 15 minutes when a scheduler is configured; Evaluate now checks one by hand. A firing watch raises an alert. Empty means nothing is being watched.
Create a watch to check a saved analysis or a table and metric you specify. Choose the detector, its parameters, the date field and time grouping. Enabled watches are checked automatically when scheduling is configured, and you can use Evaluate now to check immediately. Review State and Last Message to understand the result.
- key: table:data_analysis_watch;field:saved_id
summary: "The saved analysis this entry belongs to."
- key: table:data_analysis_watch;field:table_name
summary: "The table whose records you want to analyze."
- key: table:data_analysis_watch;field:filter_json
summary: "The saved record filters, stored as JSON."
- key: table:data_analysis_watch;field:metric_json
summary: "The measurement checked by the watch, stored as JSON."
- key: table:data_analysis_watch;field:detector
summary: "The condition to detect, such as a spike, drop or new category."
- key: table:data_analysis_watch;field:parameters_json
summary: "The settings used by the selected detector, stored as JSON."
- key: table:data_analysis_watch;field:date_field
summary: "The date field used to place records into time groups."
- key: table:data_analysis_watch;field:bucket
summary: "The time interval used to group records for the watch."
- key: table:data_analysis_watch;field:severity
summary: "The importance assigned to an alert raised by the watch."
- key: table:data_analysis_watch;field:owner_user_id
summary: "The person who owns and manages this analysis or watch."
- key: table:data_analysis_watch;field:enabled
summary: "Whether the watch is included in automatic checks."
- key: table:data_analysis_watch;field:last_evaluated_at
summary: "When the watch was last checked."
- key: table:data_analysis_watch;field:last_value
summary: "The latest measured value from the watch."
- key: table:data_analysis_watch;field:last_message
summary: "The latest explanation of the watch result."
- key: table:data_analysis_watch;field:state
summary: "Whether the watch is currently OK or firing."
- key: table:data_analysis_watch;field:alert_key
summary: "The identifier used to associate the watch with its alert."
- key: table:data_analysis_run;slot:help
summary: "Review exact analysis runs, their results, timing and any errors."
content: |
**How rows get created:** Start an exact analysis to record its execution and reusable result. An empty list means you have no recorded runs available here.
- key: table:data_analysis_run;slot:help
summary: "Each exact analysis run: who, what, when and the stored result that is reused as a cache."
content: |
**How rows get created:** automatically when someone starts an exact analysis. A finished run is reused for the table's cache minutes (Analysis Settings). Empty means nobody has run an exact analysis yet.
Analysis Runs records exact analyses as they run. You can review who requested an analysis, its table and operation, the number of rows scanned and its status. Completed results can be reused for the time allowed in Analysis Settings. If a run fails, read Error before trying the analysis again.
- key: table:data_analysis_run;field:table_name
summary: "The table whose records you want to analyze."
- key: table:data_analysis_run;field:operation
summary: "The analysis operation performed by the run."
- key: table:data_analysis_run;field:request_hash
summary: "A fingerprint of the analysis request used to match reusable results."
- key: table:data_analysis_run;field:filter_hash
summary: "A fingerprint of the filters used to match reusable results."
- key: table:data_analysis_run;field:data_version
summary: "The data version associated with the stored result."
- key: table:data_analysis_run;field:status
summary: "Whether the analysis is running, complete or failed."
- key: table:data_analysis_run;field:started_at
summary: "When the analysis run started."
- key: table:data_analysis_run;field:finished_at
summary: "When the analysis run finished."
- key: table:data_analysis_run;field:rows_scanned
summary: "The number of rows examined by the analysis."
- key: table:data_analysis_run;field:sampled
summary: "Whether the result was calculated from a sample of rows."
- key: table:data_analysis_run;field:result_json
summary: "The stored analysis result in JSON form."
- key: table:data_analysis_run;field:error
summary: "The reason the analysis run failed."
- key: table:data_analysis_run;field:user_id
summary: "The person this entry belongs to."
- key: process:data_analysis_schedule_send;slot:help
summary: "Run selected analysis schedules and send their results now."
content: |
Use Send now on selected Schedules when you need a delivery without waiting for the next scheduled time. Review the schedules before continuing. The action uses each schedule’s saved analysis, recipients and format. Check the result and Last Status afterward to confirm which deliveries completed or failed.
- key: process:data_analysis_run_schedules;slot:help
summary: "Run analysis schedules whose next delivery time is due."
content: |
Run analysis schedules is the background action that checks for scheduled deliveries. Configure and enable schedules on a saved analysis to use automatic delivery. You can inspect Next Run At, Last Run At and Last Status on Schedules to understand timing and outcomes. Automatic execution requires a configured scheduler.
- key: process:data_analysis_watch_evaluate;slot:help
summary: "Check selected watches now and update their results."
content: |
Use Evaluate now to check selected Watches without waiting for the next automatic check. Review the selection before continuing. After the action, inspect State, Last Value and Last Message to understand each result. A watch that detects its configured condition can raise an alert with the selected severity.
- key: process:data_analysis_watch_enable;slot:help
summary: "Include selected watches in automatic checks."
content: |
Use Enable when selected Watches are ready for automatic checking. Review the detector, parameters, date field and time grouping before enabling them. The action checks your access to each watch and reports the outcome. Automatic checks require a configured scheduler; use Evaluate now when you need an immediate result.
- key: process:data_analysis_watch_disable;slot:help
summary: "Exclude selected watches from automatic checks."
content: |
Use Disable when you want selected Watches to stop participating in automatic checks. Review the selection before continuing and read the result for each watch. The watch definition remains available for inspection and later use. Use Enable when you are ready to include it in automatic checks again.
- key: process:data_analysis_evaluate_watches;slot:help
summary: "Check enabled watches and update their alert conditions."
content: |
Evaluate watches is the background action that checks enabled Watches. With scheduling configured, it runs every fifteen minutes. You can review Last Evaluated At, State and Last Message to see what happened. Use Evaluate now on a watch when you need to check its condition before the next scheduled run.
- key: table:data_analysis_table_config;section:identity
summary: "The identifying details for this entry."
- key: table:data_analysis_table_config;section:details
summary: "The settings and details associated with this entry."
- key: table:data_analysis_table_config;section:dates
summary: "When this entry was created and last changed."
- key: table:data_analysis_saved;section:identity
summary: "The identifying details for this entry."
- key: table:data_analysis_saved;section:details
summary: "The settings and details associated with this entry."
- key: table:data_analysis_saved;section:dates
summary: "When this entry was created and last changed."
- key: table:data_analysis_saved_star;section:identity
summary: "The identifying details for this entry."
- key: table:data_analysis_saved_star;section:dates
summary: "When this entry was created and last changed."
- key: table:data_analysis_schedule;section:identity
summary: "The identifying details for this entry."
- key: table:data_analysis_schedule;section:details
summary: "The settings and details associated with this entry."
- key: table:data_analysis_schedule;section:dates
summary: "When this entry was created and last changed."
- key: table:data_analysis_watch;section:identity
summary: "The identifying details for this entry."
- key: table:data_analysis_watch;section:details
summary: "The settings and details associated with this entry."
- key: table:data_analysis_watch;section:dates
summary: "When this entry was created and last changed."
- key: table:data_analysis_run;section:identity
summary: "The identifying details for this entry."
- key: table:data_analysis_run;section:details
summary: "The settings and details associated with this entry."
- key: table:data_analysis_run;section:dates
summary: "When this entry was created and last changed."
- key: table:data_analysis_saved;section:data_analysis_saved_to_data_analysis_saved_star
summary: "People who starred this saved analysis."
- key: table:data_analysis_saved;section:data_analysis_saved_to_data_analysis_schedule
summary: "Delivery schedules associated with this saved analysis."
- key: table:data_analysis_saved;section:data_analysis_saved_to_data_analysis_watch
summary: "Watches associated with this saved analysis."
- key: widget:dataAnalysisSetup;slot:label
summary: "Review analysis setup and the next steps available to you."
- key: process:data_analysis_table_config_create_rows;slot:help
summary: "Create editable settings rows for analyzable tables that do not already have one."
content: |
Use **Create rows for analyzable tables** in Analysis Settings to copy the effective settings into new rows.
Review the tables before continuing. Existing settings rows are left unchanged.
You need permission to insert settings. Your analysis behavior stays the same until you edit a setting.
- key: possibleValueSource:data_analysis_kind;slot:help
summary: "Choose from the available analysis types."
content: |
Your available choices are {{values:data_analysis_kind}}.
- key: possibleValueSource:data_analysis_chart_kind;slot:help
summary: "Choose from the available chart styles."
content: |
Your available choices are {{values:data_analysis_chart_kind}}.
- key: possibleValueSource:data_analysis_detector;slot:help
summary: "Choose from the available watch detectors."
content: |
Your available choices are {{values:data_analysis_detector}}.
- key: possibleValueSource:data_analysis_watch_state;slot:help
summary: "Choose from the available watch states."
content: |
Your available choices are {{values:data_analysis_watch_state}}.
- key: possibleValueSource:data_analysis_watch_bucket;slot:help
summary: "Choose from the available time intervals."
content: |
Your available choices are {{values:data_analysis_watch_bucket}}.
- key: possibleValueSource:data_analysis_watch_severity;slot:help
summary: "Choose from the available alert severities."
content: |
Your available choices are {{values:data_analysis_watch_severity}}.
- key: possibleValueSource:data_analysis_run_status;slot:help
summary: "Choose from the available run states."
content: |
Your available choices are {{values:data_analysis_run_status}}.
- key: possibleValueSource:data_analysis_schedule_frequency;slot:help
summary: "Choose from the available delivery frequencies."
content: |
Your available choices are {{values:data_analysis_schedule_frequency}}.
- key: possibleValueSource:data_analysis_schedule_format;slot:help
summary: "Choose from the available delivery formats."
content: |
Your available choices are {{values:data_analysis_schedule_format}}.
- key: possibleValueSource:data_analysis_table_config;slot:help
summary: "Select a record from {{label:data_analysis_table_config}}."
- key: possibleValueSource:data_analysis_saved;slot:help
summary: "Select a record from {{label:data_analysis_saved}}."
- key: possibleValueSource:data_analysis_saved_star;slot:help
summary: "Select a record from {{label:data_analysis_saved_star}}."
- key: possibleValueSource:data_analysis_schedule;slot:help
summary: "Select a record from {{label:data_analysis_schedule}}."
- key: possibleValueSource:data_analysis_watch;slot:help
summary: "Select a record from {{label:data_analysis_watch}}."
- key: possibleValueSource:data_analysis_run;slot:help
summary: "Select a record from {{label:data_analysis_run}}."
- key: possibleValueSource:data_analysis_fields;slot:help
summary: "Choose fields from the selected table for your analysis settings."
- key: process:data_analysis_table_config.bulkInsert;slot:help
summary: "Import new records from a file."
content: |
Use **{{label:data_analysis_table_config.bulkInsert}}** to add records to {{label:data_analysis_table_config}} from a file. Match your file columns to the available fields and review the input before continuing. Check the result for validation errors and records you need to correct.
- key: process:data_analysis_table_config.bulkEdit;slot:help
summary: "Apply changes to multiple selected records."
content: |
Use **{{label:data_analysis_table_config.bulkEdit}}** to change fields on your selected records in {{label:data_analysis_table_config}}. Review your selection and the values you want to change before continuing. Check the result for records that could not be updated.
- key: process:data_analysis_table_config.bulkEditWithFile;slot:help
summary: "Update existing records using values from a file."
content: |
Use **{{label:data_analysis_table_config.bulkEditWithFile}}** to update records in {{label:data_analysis_table_config}}. Match your file columns to fields and identify the existing records to update. Review the input before continuing and check the result for errors.
- key: process:data_analysis_table_config.bulkDelete;slot:help
summary: "Delete multiple selected records."
content: |
Use **{{label:data_analysis_table_config.bulkDelete}}** to remove selected records from {{label:data_analysis_table_config}}. Review your selection carefully before confirming deletion. Check the result for records that could not be deleted.
- key: process:data_analysis_saved.bulkInsert;slot:help
summary: "Import new records from a file."
content: |
Use **{{label:data_analysis_saved.bulkInsert}}** to add records to {{label:data_analysis_saved}} from a file. Match your file columns to the available fields and review the input before continuing. Check the result for validation errors and records you need to correct.
- key: process:data_analysis_saved.bulkEdit;slot:help
summary: "Apply changes to multiple selected records."
content: |
Use **{{label:data_analysis_saved.bulkEdit}}** to change fields on your selected records in {{label:data_analysis_saved}}. Review your selection and the values you want to change before continuing. Check the result for records that could not be updated.
- key: process:data_analysis_saved.bulkEditWithFile;slot:help
summary: "Update existing records using values from a file."
content: |
Use **{{label:data_analysis_saved.bulkEditWithFile}}** to update records in {{label:data_analysis_saved}}. Match your file columns to fields and identify the existing records to update. Review the input before continuing and check the result for errors.
- key: process:data_analysis_saved.bulkDelete;slot:help
summary: "Delete multiple selected records."
content: |
Use **{{label:data_analysis_saved.bulkDelete}}** to remove selected records from {{label:data_analysis_saved}}. Review your selection carefully before confirming deletion. Check the result for records that could not be deleted.
- key: process:data_analysis_saved_star.bulkInsert;slot:help
summary: "Import new records from a file."
content: |
Use **{{label:data_analysis_saved_star.bulkInsert}}** to add records to {{label:data_analysis_saved_star}} from a file. Match your file columns to the available fields and review the input before continuing. Check the result for validation errors and records you need to correct.
- key: process:data_analysis_saved_star.bulkEdit;slot:help
summary: "Apply changes to multiple selected records."
content: |
Use **{{label:data_analysis_saved_star.bulkEdit}}** to change fields on your selected records in {{label:data_analysis_saved_star}}. Review your selection and the values you want to change before continuing. Check the result for records that could not be updated.
- key: process:data_analysis_saved_star.bulkEditWithFile;slot:help
summary: "Update existing records using values from a file."
content: |
Use **{{label:data_analysis_saved_star.bulkEditWithFile}}** to update records in {{label:data_analysis_saved_star}}. Match your file columns to fields and identify the existing records to update. Review the input before continuing and check the result for errors.
- key: process:data_analysis_saved_star.bulkDelete;slot:help
summary: "Delete multiple selected records."
content: |
Use **{{label:data_analysis_saved_star.bulkDelete}}** to remove selected records from {{label:data_analysis_saved_star}}. Review your selection carefully before confirming deletion. Check the result for records that could not be deleted.
- key: process:data_analysis_schedule.bulkInsert;slot:help
summary: "Import new records from a file."
content: |
Use **{{label:data_analysis_schedule.bulkInsert}}** to add records to {{label:data_analysis_schedule}} from a file. Match your file columns to the available fields and review the input before continuing. Check the result for validation errors and records you need to correct.
- key: process:data_analysis_schedule.bulkEdit;slot:help
summary: "Apply changes to multiple selected records."
content: |
Use **{{label:data_analysis_schedule.bulkEdit}}** to change fields on your selected records in {{label:data_analysis_schedule}}. Review your selection and the values you want to change before continuing. Check the result for records that could not be updated.
- key: process:data_analysis_schedule.bulkEditWithFile;slot:help
summary: "Update existing records using values from a file."
content: |
Use **{{label:data_analysis_schedule.bulkEditWithFile}}** to update records in {{label:data_analysis_schedule}}. Match your file columns to fields and identify the existing records to update. Review the input before continuing and check the result for errors.
- key: process:data_analysis_schedule.bulkDelete;slot:help
summary: "Delete multiple selected records."
content: |
Use **{{label:data_analysis_schedule.bulkDelete}}** to remove selected records from {{label:data_analysis_schedule}}. Review your selection carefully before confirming deletion. Check the result for records that could not be deleted.
- key: process:data_analysis_watch.bulkInsert;slot:help
summary: "Import new records from a file."
content: |
Use **{{label:data_analysis_watch.bulkInsert}}** to add records to {{label:data_analysis_watch}} from a file. Match your file columns to the available fields and review the input before continuing. Check the result for validation errors and records you need to correct.
- key: process:data_analysis_watch.bulkEdit;slot:help
summary: "Apply changes to multiple selected records."
content: |
Use **{{label:data_analysis_watch.bulkEdit}}** to change fields on your selected records in {{label:data_analysis_watch}}. Review your selection and the values you want to change before continuing. Check the result for records that could not be updated.
- key: process:data_analysis_watch.bulkEditWithFile;slot:help
summary: "Update existing records using values from a file."
content: |
Use **{{label:data_analysis_watch.bulkEditWithFile}}** to update records in {{label:data_analysis_watch}}. Match your file columns to fields and identify the existing records to update. Review the input before continuing and check the result for errors.
- key: process:data_analysis_watch.bulkDelete;slot:help
summary: "Delete multiple selected records."
content: |
Use **{{label:data_analysis_watch.bulkDelete}}** to remove selected records from {{label:data_analysis_watch}}. Review your selection carefully before confirming deletion. Check the result for records that could not be deleted.
- key: process:data_analysis_run.bulkInsert;slot:help
summary: "Import new records from a file."
content: |
Use **{{label:data_analysis_run.bulkInsert}}** to add records to {{label:data_analysis_run}} from a file. Match your file columns to the available fields and review the input before continuing. Check the result for validation errors and records you need to correct.
- key: process:data_analysis_run.bulkEdit;slot:help
summary: "Apply changes to multiple selected records."
content: |
Use **{{label:data_analysis_run.bulkEdit}}** to change fields on your selected records in {{label:data_analysis_run}}. Review your selection and the values you want to change before continuing. Check the result for records that could not be updated.
- key: process:data_analysis_run.bulkEditWithFile;slot:help
summary: "Update existing records using values from a file."
content: |
Use **{{label:data_analysis_run.bulkEditWithFile}}** to update records in {{label:data_analysis_run}}. Match your file columns to fields and identify the existing records to update. Review the input before continuing and check the result for errors.
- key: process:data_analysis_run.bulkDelete;slot:help
summary: "Delete multiple selected records."
content: |
Use **{{label:data_analysis_run.bulkDelete}}** to remove selected records from {{label:data_analysis_run}}. Review your selection carefully before confirming deletion. Check the result for records that could not be deleted.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1 @@
version=${project.version}
@@ -136,6 +136,13 @@ class DataAnalysisApiTest extends EngineFixture
}
var response = invoker.invoke(request);
assertThat(response.getStatus()).as(response.getBody()).isEqualTo(status);
if(status >= 400)
{
for(String field : List.of("error", "code", "message", "traceId"))
{
assertThat(response.getJson().path(field).asText()).as(field).isNotBlank();
}
}
return (response.getJson());
}
@@ -282,7 +289,7 @@ class DataAnalysisApiTest extends EngineFixture
}
/*******************************************************************************
** Generated fragment is committed verbatim and has all fourteen operations.
** Generated fragment matches the committed contract and validates all references.
*******************************************************************************/
@Test
void contract() throws Exception
@@ -296,8 +303,9 @@ class DataAnalysisApiTest extends EngineFixture
Files.createDirectories(path.getParent());
Files.writeString(path, yaml);
}
assertThat(Files.exists(path)).as("Regenerate with ./mvnw -o test -Dtest=DataAnalysisApiTest#contract -Dopenapi.write=true -Dqqq.snapshot.version=4.1.0-DATAANALYSIS-SNAPSHOT").isTrue();
assertThat(Files.exists(path)).as("Regenerate with ./mvnw -o test -Dtest=DataAnalysisApiTest#contract -Dopenapi.write=true -Dqqq.snapshot.version=4.1.99-SNAPSHOT").isTrue();
assertThat(Files.readString(path).replace("\r\n", "\n")).as("OpenAPI changed: regenerate with -Dopenapi.write=true").isEqualTo(yaml.replace("\r\n", "\n"));
assertThat(UiApiContract.check(yaml, Files.readString(path), null, 1, "-Dopenapi.write=true")).isEmpty();
}
@Test
void pivotHttp() throws Exception
@@ -364,7 +372,7 @@ class DataAnalysisApiTest extends EngineFixture
}
relationshipCountDrills(call(HttpMethod.POST, "/tables/{table}/crosstab", Map.of("rowField", "status", "columnField", "customer"), 200));
// All new schemas are discoverable from the same served OpenAPI catalog.
assertThat(com.kingsrook.qbits.dataanalysis.api.DataAnalysisApi.extension().getSemanticVersion()).isEqualTo("1.4.0");
assertThat(com.kingsrook.qbits.dataanalysis.api.DataAnalysisApi.extension().getSemanticVersion()).isEqualTo("1.5.0");
}
/*******************************************************************************
@@ -25,6 +25,9 @@ import java.time.temporal.ChronoUnit;
import java.util.List;
import com.kingsrook.qbits.dataanalysis.app.DataAnalysisHome;
import com.kingsrook.qbits.dataanalysis.model.DataAnalysisTables;
import com.kingsrook.qbits.dataanalysis.permissions.DataAnalysisPermissions;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.MultiStatisticsData;
@@ -69,12 +72,43 @@ class DataAnalysisHomeTest
}
}
QContext.init(instance, TestSupport.user("alice"));
assertThat(DataAnalysisHome.statistics(Instant.now()).getStatisticsGroupData()).allSatisfy(g ->
assertThat(DataAnalysisHome.statistics(Instant.now()).getStatisticsGroupData()).isEmpty();
assertThat(((TableData) DataAnalysisHome.render(DataAnalysisHome.FIRING, Instant.now())).getNoRowsFoundHTML()).doesNotContain("Unavailable");
}
@Test
void setupIsOffTheBusinessHomeAndAdminOnly() throws Exception
{
var instance = TestSupport.instance();
assertThat(DataAnalysisHome.homeWidgetNames()).doesNotContain(DataAnalysisHome.SETUP).containsExactly(DataAnalysisHome.HOME_WIDGET,
DataAnalysisHome.STARRED, DataAnalysisHome.FIRING, DataAnalysisHome.RECENT);
assertThat(instance.getApp("dataAnalysis").getWidgets()).doesNotContain(DataAnalysisHome.SETUP);
assertThat(DataAnalysisQBitProducer.SYSTEM_WIDGETS).containsExactly(DataAnalysisHome.SETUP);
assertThat(instance.getWidgets()).containsKey(DataAnalysisHome.SETUP);
QContext.init(instance, TestSupport.admin());
assertThat(PermissionsHelper.hasWidgetPermission(new AbstractActionInput(), DataAnalysisHome.SETUP)).isTrue();
assertThat(DataAnalysisHome.setup().getRows()).hasSize(5);
assertThat(new DataAnalysisHome().render(new RenderWidgetInput().withWidgetMetaData(instance.getWidget(DataAnalysisHome.SETUP))).getWidgetData()).isInstanceOf(TableData.class);
QContext.init(instance, TestSupport.user("alice").withPermissions(DataAnalysisPermissions.SAVED_READ));
assertThat(PermissionsHelper.hasWidgetPermission(new AbstractActionInput(), DataAnalysisHome.SETUP)).isFalse();
}
@Test
void savedReadOnlyUserGetsNoLockedShells() throws Exception
{
var instance = TestSupport.instance();
QContext.init(instance, TestSupport.user("alice").withPermissions(DataAnalysisPermissions.SAVED_READ));
List<String> visible = DataAnalysisHome.homeWidgetNames().stream().filter(name -> PermissionsHelper.hasWidgetPermission(new AbstractActionInput(), name)).toList();
assertThat(visible).containsExactly(DataAnalysisHome.HOME_WIDGET, DataAnalysisHome.STARRED);
for(String name : visible)
{
assertThat(g.getStatisticList()).isEmpty();
assertThat(g.getSubheader()).contains("Unavailable");
});
assertThat(((TableData) DataAnalysisHome.render(DataAnalysisHome.FIRING, Instant.now())).getNoRowsFoundHTML()).contains("Unavailable");
var data = DataAnalysisHome.render(name, Instant.now());
assertThat(com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(data)).doesNotContain("Unavailable").doesNotContain("permission required");
}
var groups = DataAnalysisHome.statistics(Instant.now()).getStatisticsGroupData();
assertThat(groups).extracting(g -> g.getHeader()).containsExactly("Saved analyses");
}
@Test
@@ -32,6 +32,7 @@ import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfigValidationException;
@@ -115,11 +116,17 @@ class DataAnalysisMetadataTest
assertThat(producer.isEnabled()).isTrue();
assertThat(producer.getNamespace()).isEqualTo("analysis");
assertThat(producer.getQBitConfig()).isSameAs(config);
assertThat(producer.getQBitMetaData().getVersion()).isEqualTo("0.1.0");
String artifactVersion = System.getProperty("qbit.artifactVersion");
assertThat(artifactVersion).isNotBlank();
assertThat(producer.getQBitMetaData().getVersion()).isEqualTo(artifactVersion).isEqualTo(DataAnalysisQBitProducer.VERSION);
assertThat(producer.getQBitMetaData().getGroupId()).isEqualTo("com.kingsrook.qbits");
assertThat(producer.getQBitMetaData().getArtifactId()).isEqualTo("qbit-data-analysis");
assertThat(producer.getPackageNameForFindingMetaDataProducers()).endsWith(".metadata");
producer.produce(instance);
assertThat(HelpBundles.of(instance).getBundles().getFirst().getVersion())
.isEqualTo(artifactVersion).isEqualTo(producer.getQBitMetaData().getVersion());
assertThat(HelpBundles.of(instance).getBundles().getFirst().getEntries()).anyMatch(entry -> entry.getKey().equals("app:customAnalysis;slot:help"))
.noneMatch(entry -> entry.getKey().equals("app:dataAnalysis;slot:help"));
assertThat(instance.getApp("customAnalysis").getParentAppName()).isEqualTo("admin");
assertThat(instance.getApp("admin").getChildren()).contains(instance.getApp("customAnalysis"));
assertThat(new DataAnalysisQBitProducer().isEnabled()).isFalse();
@@ -30,12 +30,14 @@ import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;
import com.kingsrook.qbits.dataanalysis.help.DataAnalysisHelp;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.testing.HelpBundleAssertions;
import com.kingsrook.qbits.dataanalysis.model.DataAnalysisTables;
import com.kingsrook.qbits.dataanalysis.processes.TableConfigDefaultsProcess;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.actions.help.HelpTargetCatalog;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundleEntry;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
@@ -144,7 +146,7 @@ class DataAnalysisTablesGuardrailTest
///////////////////////////////////////////////////////////////////////////////
assertThat(instance.getProcess(TableConfigDefaultsProcess.NAME).getTableName()).isEqualTo(DataAnalysisTables.TABLE_CONFIG);
assertThat(instance.getProcess(TableConfigDefaultsProcess.NAME).getMinInputRecords()).isEqualTo(0);
assertThat(help(DataAnalysisTables.TABLE_CONFIG).getFirst().getContent()).contains(instance.getProcess(TableConfigDefaultsProcess.NAME).getLabel());
assertThat(help(DataAnalysisTables.TABLE_CONFIG).getContent()).contains(instance.getProcess(TableConfigDefaultsProcess.NAME).getLabel());
}
@@ -155,39 +157,50 @@ class DataAnalysisTablesGuardrailTest
@Test
void everyTableHasEmptyStateHelp() throws Exception
{
assertThat(DataAnalysisHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120));
HelpBundleAssertions.forInstance(instance).bundles("help/qbit-data-analysis.yaml")
.owning(target -> target.name().startsWith("data_analysis_") || target.name().startsWith("dataAnalysis") || target.name().equals("DATA_ANALYSIS_SAVED"))
.assertComplete();
for(String table : analysisTables())
{
List<QHelpContent> help = help(table);
assertThat(help).as(table + " table help").isNotEmpty();
assertThat(help.getFirst().getContent()).as(table + " help says how rows get created").contains("How rows get created");
assertThat(help(table).getContent()).as(table + " help says how rows get created").contains("How rows get created");
}
assertThat(instance.getApp("dataAnalysis").getHelpContent()).containsKey(DataAnalysisHelp.SLOT);
assertThat(DataAnalysisHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || instance.getTable(key.substring(6, key.indexOf(';'))) != null,
"help names only real tables");
// RC.7's core gate requires apps, tables, processes and fields; also cover our widgets and value sources.
List<String> keys = HelpBundles.of(instance).getBundles().getFirst().getEntries().stream().map(HelpBundleEntry::getKey).toList();
HelpTargetCatalog.collect(instance).stream()
.filter(target -> List.of("widget", "possibleValueSource").contains(target.kind()))
.filter(target -> target.name().startsWith("data_analysis_") || target.name().startsWith("dataAnalysis") || target.name().equals("DATA_ANALYSIS_SAVED"))
.forEach(target -> assertThat(keys).contains(target.key()));
assertThat(keys).doesNotHaveDuplicates();
assertThat(instance.getTable("helpContent")).isNull();
HelpBundles.syncOnStartup(instance);
assertThat(HelpBundles.of(instance).getBundles()).hasSize(1);
}
/*******************************************************************************
** A host's own help on a table is kept.
** Host bundles override the qbit without replacing its resource.
*******************************************************************************/
@Test
void hostHelpIsNotOverridden() throws Exception
void hostBundleOverridesQbit()
{
instance.getTable(DataAnalysisTables.SAVED).getHelpContent().put(DataAnalysisHelp.SLOT, List.of(new QHelpContent().withContentAsMarkdown("host text")));
DataAnalysisHelp.apply(instance, new DataAnalysisQBitConfig());
assertThat(help(DataAnalysisTables.SAVED).getFirst().getContent()).isEqualTo("host text");
HelpBundles.register(instance, "help/host-override.yaml");
var winner = HelpBundles.of(instance).resolveWinningEntries()
.get(new HelpBundles.EntryKey("table:data_analysis_saved;slot:help", "ALL_SCREENS"));
assertThat(winner.bundle().getId()).isEqualTo("test-host");
assertThat(winner.entry().getSummary()).isEqualTo("Your host's saved analysis guidance.");
assertThat(help(DataAnalysisTables.SAVED).getContent()).contains("How rows get created");
}
/*******************************************************************************
**
** Read the registered bundle without requiring host help storage.
*******************************************************************************/
private List<QHelpContent> help(String table)
private HelpBundleEntry help(String table)
{
return (instance.getTable(table).getHelpContent() == null ? null : instance.getTable(table).getHelpContent().get(DataAnalysisHelp.SLOT));
return (HelpBundles.of(instance).getBundles().getFirst().getEntries().stream()
.filter(entry -> entry.getKey().equals("table:" + table + ";slot:help")).findFirst().orElseThrow());
}
@@ -107,8 +107,11 @@ class FieldAnalysisAccessTest extends EngineFixture
var fallback = new FieldSecurityLockFieldAccessResolver();
assertThat(fallback.resolve(table, table.getField("amount"))).isEqualTo(FieldAnalysisAccess.FULL);
assertThat(fallback.resolve(table, table.getField("secret"))).isEqualTo(FieldAnalysisAccess.NONE);
// The snapshot has no tier package. Test the actual default selection.
assertThat(CoreTierFieldAccessResolver.defaultResolver()).isInstanceOf(FieldSecurityLockFieldAccessResolver.class);
// RC.7 supplies the core tier API; verify discovery and effective access.
var defaultResolver = CoreTierFieldAccessResolver.defaultResolver();
assertThat(defaultResolver).isInstanceOf(CoreTierFieldAccessResolver.class);
assertThat(defaultResolver.resolve(table, table.getField("amount"))).isEqualTo(FieldAnalysisAccess.FULL);
assertThat(defaultResolver.resolve(table, table.getField("secret"))).isEqualTo(FieldAnalysisAccess.NONE);
var resolver = new CoreTierFieldAccessResolver(TierApi.class);
try
{
@@ -26,6 +26,7 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.dataanalysis.model.DataAnalysisTables;
import com.kingsrook.qbits.dataanalysis.permissions.DataAnalysisPermissions;
import com.kingsrook.qbits.dataanalysis.processes.ScheduleActions;
import com.kingsrook.qbits.dataanalysis.saved.AnalysisExport;
import com.kingsrook.qbits.dataanalysis.saved.SavedDefinition;
@@ -47,6 +48,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.messaging.QMessagingProvide
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.messaging.MessagingProviderInterface;
import com.kingsrook.qqq.backend.core.modules.messaging.QMessagingProviderDispatcher;
import com.kingsrook.qqq.backend.core.serviceidentity.QServicePrincipalMetaData;
import com.kingsrook.qqq.backend.core.serviceidentity.ServiceIdentitiesMetaData;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
@@ -160,6 +163,27 @@ class SchedulesTest extends EngineFixture
assertThat(WatchesTest.Sink.raised).isEmpty();
}
/*******************************************************************************
** With the host's data-analysis principal declared, schedule inventory runs
** as the principal (its grants decide - a principal without them finds
** nothing due), never as a self-made session.
*******************************************************************************/
@Test
void inventoryRunsAsTheDeclaredPrincipal() throws Exception
{
schedule("PIVOT", Map.of("rows", List.of(Map.of("field", "status")), "values", List.of(Map.of("operator", "COUNT"))), "CSV");
config.withServicePrincipal("analysis-bare");
ServiceIdentitiesMetaData.declare(instance, new QServicePrincipalMetaData().withName("analysis-bare"));
assertThatThrownBy(() -> runner.runDue(NOW)).isInstanceOf(QException.class);
assertThat(Mail.sent).isEmpty();
ServiceIdentitiesMetaData.declare(instance, new QServicePrincipalMetaData().withName("data-analysis")
.withLocalPermissions("data_analysis.saved.read", "data_analysis.saved.edit").withLocalSecurityKeyValue(DataAnalysisPermissions.ADMIN, true));
config.withServicePrincipal("data-analysis");
assertThat(runner.runDue(NOW)).isEqualTo(1);
assertThat(Mail.sent).hasSize(2);
}
@Test
void providerFailureAndMissingOwnerRaiseVisibleAlerts() throws Exception
{
@@ -0,0 +1,7 @@
bundle: test-host
version: 1
layer: HOST
entries:
- key: table:data_analysis_saved;slot:help
summary: "Your host's saved analysis guidance."
content: "Use your host's saved analysis workflow."