Compare commits

..
Author SHA1 Message Date
james.maesandCodex d2c2f8a9b8 ci: centralize QBit source scanning
Co-Authored-By: Codex <noreply@openai.com>
2026-10-21 10:20:46 -05:00
11 changed files with 148 additions and 198 deletions

No files matched your search

+2 -2
View File
@@ -1,12 +1,12 @@
version: 2.1 version: 2.1
setup: true setup: true
orbs: orbs:
qqq-orb: kingsrook/qqq-orb@0.8 qqq-orb: kingsrook/qqq-orb@0.7.12
workflows: workflows:
setup: setup:
jobs: jobs:
- qqq-orb/generate_pipeline: - qqq-orb/generate_pipeline:
orb_version: '0.8' orb_version: 0.7.12
filters: filters:
branches: branches:
ignore: automation/qbit-rc-pending ignore: automation/qbit-rc-pending
-7
View File
@@ -1,7 +0,0 @@
name: build
on:
push:
jobs:
maven:
uses: self:KofTwentyTwo/ci-workflows/.gitea/workflows/maven.yaml@v1
secrets: inherit
+1 -1
View File
@@ -1,5 +1,5 @@
profile: qbit profile: qbit
orb_version: '0.8' orb_version: 0.7.12
default_branch: develop default_branch: develop
java_version: '21' java_version: '21'
browsers: true browsers: true
-4
View File
@@ -2,10 +2,6 @@
## Unreleased / 0.2.0 ## Unreleased / 0.2.0
- Register CRM health checks (`crm.email`, `crm.sequence_lag`, `crm.lead_score`) with the QQQ 4.1 core health registry (`HealthChecksMetaData`, qqq#1199), replacing the `qqq-middleware-health` indicator stand-in and its optional dependency.
- Lead scoring keeps its incremental watermark at millisecond precision, matching `modifyDate`, so a contact changed in the same millisecond a pass starts is not skipped.
- Read the CRM QBit metadata version from the Maven-filtered artifact version so release metadata matches the build. - Read the CRM QBit metadata version from the Maven-filtered artifact version so release metadata matches the build.
- Adopt QQQ 4.1.0-RC.7 native help bundles, replacing the RC.6 stand-in loader with host-neutral CRM help, live references and the core coverage gate. - Adopt QQQ 4.1.0-RC.7 native help bundles, replacing the RC.6 stand-in loader with host-neutral CRM help, live references and the core coverage gate.
+2 -2
View File
@@ -62,9 +62,9 @@ qInstance.setEnforcePermissionsInCoreActions(true);
- `withApiName(...)` and `withApiVersion(...)`: publish main and settings tables into an existing host API definition. The host supplies `ApiInstanceMetaData` and includes optional `qqq-middleware-api` at runtime. Main tables declare core search fields. - `withApiName(...)` and `withApiVersion(...)`: publish main and settings tables into an existing host API definition. The host supplies `ApiInstanceMetaData` and includes optional `qqq-middleware-api` at runtime. Main tables declare core search fields.
- `withEsbDestinationName(...)`: reserved but rejected on RC.6. Its ESB event factory copies all values and has no classified-field exclusion; CRM does not publish those payloads. - `withEsbDestinationName(...)`: reserved but rejected on RC.6. Its ESB event factory copies all values and has no classified-field exclusion; CRM does not publish those payloads.
- `CrmBulkLoadProfiles.ensure()`: call once in host startup after registering core's `SavedBulkLoadProfileMetaDataProvider`. It seeds three CSV mappings by label without replacing edited profiles. CRM does not register the core profile tables. - `CrmBulkLoadProfiles.ensure()`: call once in host startup after registering core's `SavedBulkLoadProfileMetaDataProvider`. It seeds three CSV mappings by label without replacing edited profiles. CRM does not register the core profile tables.
- Health: the producer registers `crm.email`, `crm.sequence_lag` and `crm.lead_score` with the core health registry (`HealthChecksMetaData`). Hosts that enable core health monitoring (`HealthMetaDataProvider`) get them scheduled by the core `HealthRunner`, which runs each check as the core system user. `CrmHealthChecks.run()` evaluates them once in the caller's context. No check adds permissions or tenant keys or returns personal values. - `CrmHealthChecks.indicators(monitoringSession)`: register the returned indicators with the host health middleware. Include optional `qqq-middleware-health`; the supplied session determines tenant scope and is restored on worker threads. `run()` uses the caller's context. No check adds permissions or tenant keys.
Help and health both use the core registries (`HelpBundles`, `HealthChecksMetaData`). See [P4B implementation notes](docs/P4B-IMPLEMENTATION.md) for RC.6 limits. The shipped help bundle and health adapter are temporary core integration adapters; their removal target is RC.9. See [P4B implementation notes](docs/P4B-IMPLEMENTATION.md) for RC.6 limits.
| Scope | Permissions | | Scope | Permissions |
|---|---| |---|---|
+11 -11
View File
@@ -89,8 +89,8 @@ These parts are real and are kept, adapted to 0.2.0:
| Imports | `importContacts` | core bulk load + saved bulk-load profiles | | Imports | `importContacts` | core bulk load + saved bulk-load profiles |
| PII | none | `DataClassification.PII` on every personal field; secured-data tiers apply | | PII | none | `DataClassification.PII` on every personal field; secured-data tiers apply |
| Reporting | 14 bespoke widgets | home widgets for the daily view; trend reports via data analysis (`crm_*.analyze`) | | Reporting | 14 bespoke widgets | home widgets for the daily view; trend reports via data analysis (`crm_*.analyze`) |
| Health | none | `crm.*` checks in the core health registry (`HealthChecksMetaData`) | | Health | none | `crm` health checks (stand-in until core #1199) |
| Help | none | `help/qbit-crm.yaml` bundle registered with core `HelpBundles` | | Help | none | `help/qbit-crm.yaml` bundle (stand-in loader until core #1198) |
| Tenancy | clientId lock on 3 tables | configurable tenant key type on every table; the host maps Authentik claims to keys | | Tenancy | clientId lock on 3 tables | configurable tenant key type on every table; the host maps Authentik claims to keys |
## Data model (33 tables: 6 main, 15 settings, 12 sub-tables) ## Data model (33 tables: 6 main, 15 settings, 12 sub-tables)
@@ -268,14 +268,13 @@ signals are logged.
### Health ### Health
`CrmHealthChecks` registers three checks with the core health registry `CrmHealthChecks` reports three checks:
(`HealthChecksMetaData`, component `crm`): - email transport readiness
- `crm.email`: email transport readiness - sequence lag (the oldest due enrollment more than 30 min late)
- `crm.sequence_lag`: the oldest due enrollment more than 30 min late - the scoring job's last success time
- `crm.lead_score`: the scoring job's last success time
The core `HealthRunner` schedules them on hosts that enable health monitoring and runs each one It is a `STAND-IN(core=QRun-IO/qqq#1199, removeBy=4.1.0-RC.9)` adapter onto the
as the core system user. A failing check reports UNKNOWN with a fixed message. `qqq-middleware-health` `HealthIndicator` until the central `HealthCheck` SPI ships.
### Help ### Help
@@ -283,8 +282,9 @@ as the core system user. A failing check reports UNKNOWN with a fixed message.
- the app - the app
- every table, field, action and possible-value source - every table, field, action and possible-value source
The producer registers it with core `HelpBundles.register`; `HelpBundleAssertions` checks that it `HelpBundleAssertions` checks that it is complete. Until `HelpBundles.register` is in a core RC,
is complete. `CrmHelpBundleStandIn` (`STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.9)`) loads the same
file into `QHelpContent`.
### Contingent dropdowns ### Contingent dropdowns
+8 -4
View File
@@ -22,7 +22,6 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties> <properties>
<qqq.version>4.1.0-RC.26</qqq.version> <!-- feeds qbit-build-parent's qqq-bom-pom import -->
<revision>0.2.0-SNAPSHOT</revision> <revision>0.2.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source> <maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target> <maven.compiler.target>21</maven.compiler.target>
@@ -34,7 +33,7 @@
<parent> <parent>
<groupId>com.kingsrook</groupId> <groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId> <artifactId>qbit-build-parent</artifactId>
<version>2.2.1</version> <version>2.1.2</version>
</parent> </parent>
<groupId>com.kingsrook.qbits</groupId> <groupId>com.kingsrook.qbits</groupId>
@@ -79,11 +78,16 @@
<artifactId>qqq-middleware-api</artifactId> <artifactId>qqq-middleware-api</artifactId>
<optional>true</optional> <optional>true</optional>
</dependency> </dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-middleware-health</artifactId>
<optional>true</optional>
</dependency>
<dependency> <dependency>
<groupId>com.kingsrook.qbits</groupId> <groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-email</artifactId> <artifactId>qbit-email</artifactId>
<version>0.1.0-RC.11</version> <version>0.1.0-RC.3</version>
<optional>true</optional> <optional>true</optional>
</dependency> </dependency>
<dependency> <dependency>
@@ -235,7 +239,7 @@
<activeByDefault>true</activeByDefault> <activeByDefault>true</activeByDefault>
</activation> </activation>
<properties> <properties>
<qqq.snapshot.version>4.1.0-RC.26</qqq.snapshot.version> <qqq.snapshot.version>4.1.0-RC.7</qqq.snapshot.version>
</properties> </properties>
<dependencyManagement> <dependencyManagement>
<dependencies> <dependencies>
@@ -46,7 +46,6 @@ import com.kingsrook.qbits.crm.deals.customizers.DealStageChangeCustomizer;
import com.kingsrook.qbits.crm.deals.model.Deal; import com.kingsrook.qbits.crm.deals.model.Deal;
import com.kingsrook.qbits.crm.deals.model.DealProduct; import com.kingsrook.qbits.crm.deals.model.DealProduct;
import com.kingsrook.qbits.crm.email.CrmEmailProcessIdentity; import com.kingsrook.qbits.crm.email.CrmEmailProcessIdentity;
import com.kingsrook.qbits.crm.health.CrmHealthChecks;
import com.kingsrook.qbits.crm.scoring.AutoAssignCustomizer; import com.kingsrook.qbits.crm.scoring.AutoAssignCustomizer;
import com.kingsrook.qbits.crm.security.CrmDataPolicy; import com.kingsrook.qbits.crm.security.CrmDataPolicy;
import com.kingsrook.qbits.crm.security.CrmPermissions; import com.kingsrook.qbits.crm.security.CrmPermissions;
@@ -152,7 +151,6 @@ public class CrmQBitProducer implements QBitMetaDataProducer<CrmQBitConfig>
} }
CrmDataPolicy.apply(metaDataProducerMultiOutput, qInstance, qBitConfig); CrmDataPolicy.apply(metaDataProducerMultiOutput, qInstance, qBitConfig);
CrmIntegrations.apply(metaDataProducerMultiOutput, qBitConfig); CrmIntegrations.apply(metaDataProducerMultiOutput, qBitConfig);
CrmHealthChecks.register(qInstance, qBitConfig);
for(QWidgetMetaData widget : metaDataProducerMultiOutput.getEach(QWidgetMetaData.class)) for(QWidgetMetaData widget : metaDataProducerMultiOutput.getEach(QWidgetMetaData.class))
{ {
widget.setName(snakeCase(widget.getName())); widget.setName(snakeCase(widget.getName()));
@@ -23,7 +23,6 @@ package com.kingsrook.qbits.crm.health;
import java.time.Instant; import java.time.Instant;
import java.time.temporal.ChronoUnit; import java.time.temporal.ChronoUnit;
import java.util.LinkedHashMap;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import com.kingsrook.qbits.crm.CrmQBitConfig; import com.kingsrook.qbits.crm.CrmQBitConfig;
@@ -33,15 +32,9 @@ import com.kingsrook.qbits.crm.email.CrmEmailTransport;
import com.kingsrook.qbits.crm.scoring.JobState; import com.kingsrook.qbits.crm.scoring.JobState;
import com.kingsrook.qbits.crm.scoring.processes.CalculateLeadScoresProcess; import com.kingsrook.qbits.crm.scoring.processes.CalculateLeadScoresProcess;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity; import com.kingsrook.qqq.backend.core.context.CapturedContext;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.health.HealthCategory;
import com.kingsrook.qqq.backend.core.health.HealthCheck;
import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
@@ -49,25 +42,24 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance; import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.middleware.health.HealthIndicator;
import com.kingsrook.qqq.middleware.health.model.metadata.HealthCheckResult;
import com.kingsrook.qqq.middleware.health.model.metadata.HealthStatus;
/******************************************************************************* /*******************************************************************************
** CRM checks registered with the core health registry (HealthChecksMetaData). ** STAND-IN(core=QRun-IO/qqq#1199, removeBy=4.1.0-RC.9)
** The core HealthRunner schedules them and runs each one in its own automated ** CRM checks for hosts to register with their health middleware. Run in the
** system-user context; no check returns personal values. ** caller's tenant context when one is supplied, else as the core system user;
** no check returns personal values.
*******************************************************************************/ *******************************************************************************/
public class CrmHealthChecks public class CrmHealthChecks
{ {
public static final String EMAIL = "crm.email";
public static final String SEQUENCE_LAG = "crm.sequence_lag";
public static final String LEAD_SCORE = "crm.lead_score";
private static final String COMPONENT = "crm";
private final CrmEmailTransport transport; private final CrmEmailTransport transport;
/******************************************************************************* /*******************************************************************************
** Use the same readiness decision as actual email delivery. ** Use the same readiness decision as actual email delivery.
*******************************************************************************/ *******************************************************************************/
@@ -77,7 +69,6 @@ public class CrmHealthChecks
} }
/******************************************************************************* /*******************************************************************************
** Allow a host transport adapter. ** Allow a host transport adapter.
*******************************************************************************/ *******************************************************************************/
@@ -87,141 +78,121 @@ public class CrmHealthChecks
} }
/******************************************************************************* /*******************************************************************************
** Register the CRM checks with the instance's core health registry. ** Run all checks without registering tables or starting background workers.
** Production may run repeatedly; an already registered name is kept.
*******************************************************************************/ *******************************************************************************/
public static void register(QInstance instance, CrmQBitConfig config) public Map<String, HealthCheckResult> run() throws QException
{ {
new CrmHealthChecks(config).registerWith(instance); return (Map.of("crm.email", email(), "crm.sequenceLag", sequenceLag(), "crm.leadScore", leadScore()));
} }
/******************************************************************************* /*******************************************************************************
** Register these checks (with this transport) with the core health registry. ** Expose individual checks through the RC.6 health SPI. With no caller
** session (registration at startup), each check runs as the core system
** user: token-based authentication modules (OAuth2, Auth0) reject any other
** tokenless session, and the checks report only UP/DEGRADED, never values.
*******************************************************************************/ *******************************************************************************/
public void registerWith(QInstance instance) public List<HealthIndicator> indicators()
{ {
HealthChecksMetaData registry = HealthChecksMetaData.ofOrWithNew(instance); return (indicators(QContext.getQSession()));
add(registry, EMAIL, "CRM email transport", HealthCategory.EMAIL, 300);
add(registry, SEQUENCE_LAG, "CRM sequence lag", HealthCategory.JOBS, 300);
add(registry, LEAD_SCORE, "CRM lead scoring", HealthCategory.JOBS, 900);
} }
/*******************************************************************************
** Supply the host's monitoring session explicitly for middleware worker threads.
** Its permissions and tenant keys are preserved; this adapter never grants keys.
** A null session runs each check as the core system user.
*******************************************************************************/
public List<HealthIndicator> indicators(QSession session)
{
return (List.of(new Indicator("crm.email", session), new Indicator("crm.sequenceLag", session), new Indicator("crm.leadScore", session)));
}
/******************************************************************************* /*******************************************************************************
** Run all checks once in the caller's context, without registering anything. ** One independently evaluated middleware indicator.
*******************************************************************************/ *******************************************************************************/
public Map<String, HealthCheckOutcome> run() throws QException private class Indicator implements HealthIndicator
{ {
Map<String, HealthCheckOutcome> outcomes = new LinkedHashMap<>(); private final String name;
for(String name : List.of(EMAIL, SEQUENCE_LAG, LEAD_SCORE)) private final QSession session;
/*******************************************************************************
** Select a check.
*******************************************************************************/
private Indicator(String name, QSession session)
{ {
outcomes.put(name, check(name).run(new HealthCheckInput())); this.name = name;
this.session = session;
} }
return (outcomes);
}
@Override
/******************************************************************************* public String getName()
** The core SPI body for one named check. Failures are reported as UNKNOWN
** with a fixed message, so no exception text (or value) leaves the check.
*******************************************************************************/
public HealthCheck check(String name)
{
return (new HealthCheck()
{ {
/******************************************************************* return (name);
** Evaluate the named check in the current QContext. }
*******************************************************************/
@Override
public HealthCheckOutcome run(HealthCheckInput input) @Override
public HealthCheckResult check(QInstance instance)
{
CapturedContext original = QContext.capture();
try
{ {
try QContext.init(instance, session != null ? session : new QSystemUserSession(instance));
return (switch(name)
{ {
return (switch(name) case "crm.email" -> email();
{ case "crm.sequenceLag" -> sequenceLag();
case EMAIL -> email(); default -> leadScore();
case SEQUENCE_LAG -> sequenceLag(); });
case LEAD_SCORE -> leadScore(); }
default -> throw (new IllegalArgumentException("Unknown CRM health check")); catch(QException | RuntimeException e)
}); {
} return (new HealthCheckResult().withStatus(HealthStatus.UNKNOWN).withDetail("reason", "CRM check could not be completed"));
catch(QException | RuntimeException e) }
{ finally
return (HealthCheckOutcome.unknown("CRM check could not be completed")); {
} QContext.init(original);
} }
});
}
/*******************************************************************************
** Add one check definition unless the name is already registered.
*******************************************************************************/
private void add(HealthChecksMetaData registry, String name, String label, HealthCategory category, int intervalSeconds)
{
if(registry.getChecks().stream().anyMatch(existing -> name.equals(existing.getName())))
{
return;
} }
registry.withCheck(new HealthCheckMetaData()
.withName(name)
.withLabel(label)
.withComponent(COMPONENT)
.withCategory(category)
.withKind(HealthCheckKind.DEPENDENCY)
.withSeverity(AlertSeverity.WARNING)
.withIntervalSeconds(intervalSeconds)
.withTimeoutMs(5000)
.withCheckInstance(check(name)));
} }
/******************************************************************************* /*******************************************************************************
** An unconfigured transport is degraded, not a claim that email can be sent. ** An unconfigured transport is degraded, not a claim that email can be sent.
*******************************************************************************/ *******************************************************************************/
private HealthCheckOutcome email() throws QException private HealthCheckResult email() throws QException
{ {
return (transport.readiness() == CrmReadiness.NOT_CONFIGURED return (new HealthCheckResult().withStatus(transport.readiness() == CrmReadiness.NOT_CONFIGURED ? HealthStatus.DEGRADED : HealthStatus.UP));
? HealthCheckOutcome.degraded("Email transport not configured")
: HealthCheckOutcome.up("Email transport ready"));
} }
/******************************************************************************* /*******************************************************************************
** Query just the oldest active due enrollment. ** Query just the oldest active due enrollment.
*******************************************************************************/ *******************************************************************************/
private HealthCheckOutcome sequenceLag() throws QException private HealthCheckResult sequenceLag() throws QException
{ {
List<QRecord> rows = new QueryAction().execute(new QueryInput("crm_sequence_enrollment").withFilter(new QQueryFilter() List<QRecord> rows = new QueryAction().execute(new QueryInput("crm_sequence_enrollment").withFilter(new QQueryFilter()
.withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, CrmEnrollmentStatus.ACTIVE.getId())) .withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, CrmEnrollmentStatus.ACTIVE.getId()))
.withCriteria(new QFilterCriteria("nextStepDate", QCriteriaOperator.LESS_THAN, Instant.now().minus(30, ChronoUnit.MINUTES))) .withCriteria(new QFilterCriteria("nextStepDate", QCriteriaOperator.LESS_THAN, Instant.now().minus(30, ChronoUnit.MINUTES)))
.withOrderBy(new QFilterOrderBy("nextStepDate")).withLimit(1))).getRecords(); .withOrderBy(new QFilterOrderBy("nextStepDate")).withLimit(1))).getRecords();
return (rows.isEmpty() return (new HealthCheckResult().withStatus(rows.isEmpty() ? HealthStatus.UP : HealthStatus.DEGRADED));
? HealthCheckOutcome.up("No sequence step is more than 30 minutes late")
: HealthCheckOutcome.degraded("A sequence step is more than 30 minutes late"));
} }
/******************************************************************************* /*******************************************************************************
** No successful score pass, or one more than three hours old, needs attention. ** No successful score pass, or one more than three hours old, needs attention.
*******************************************************************************/ *******************************************************************************/
private HealthCheckOutcome leadScore() throws QException private HealthCheckResult leadScore() throws QException
{ {
List<QRecord> rows = new QueryAction().execute(new QueryInput(JobState.TABLE_NAME).withFilter(new QQueryFilter( List<QRecord> rows = new QueryAction().execute(new QueryInput(JobState.TABLE_NAME).withFilter(new QQueryFilter(
new QFilterCriteria("name", QCriteriaOperator.EQUALS, CalculateLeadScoresProcess.NAME)).withLimit(1))).getRecords(); new QFilterCriteria("name", QCriteriaOperator.EQUALS, CalculateLeadScoresProcess.NAME)).withLimit(1))).getRecords();
Instant last = rows.isEmpty() ? null : rows.getFirst().getValueInstant("lastSuccess"); Instant last = rows.isEmpty() ? null : rows.getFirst().getValueInstant("lastSuccess");
return (last == null || last.isBefore(Instant.now().minus(3, ChronoUnit.HOURS)) return (new HealthCheckResult().withStatus(last == null || last.isBefore(Instant.now().minus(3, ChronoUnit.HOURS))
? HealthCheckOutcome.degraded("No lead score pass succeeded in the last 3 hours") ? HealthStatus.DEGRADED : HealthStatus.UP));
: HealthCheckOutcome.up("Lead scores are current"));
} }
} }
@@ -105,11 +105,7 @@ public class CalculateLeadScoresProcess implements BackendStep, MetaDataProducer
@Override @Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{ {
//////////////////////////////////////////////////////////////////////// Instant startedAt = Instant.now();
// modifyDate is kept at millisecond precision; a microsecond watermark //
// would skip a change made in the same millisecond the pass started. //
////////////////////////////////////////////////////////////////////////
Instant startedAt = Instant.now().truncatedTo(ChronoUnit.MILLIS);
List<QRecord> states = new QueryAction().execute(new QueryInput(JobState.TABLE_NAME) List<QRecord> states = new QueryAction().execute(new QueryInput(JobState.TABLE_NAME)
.withFilter(new QQueryFilter(new QFilterCriteria("name", QCriteriaOperator.EQUALS, NAME)))).getRecords(); .withFilter(new QQueryFilter(new QFilterCriteria("name", QCriteriaOperator.EQUALS, NAME)))).getRecords();
QRecord state = states.isEmpty() ? new QRecord().withValue("name", NAME) : states.getFirst(); QRecord state = states.isEmpty() ? new QRecord().withValue("name", NAME) : states.getFirst();
@@ -33,19 +33,14 @@ import com.kingsrook.qbits.crm.health.CrmHealthChecks;
import com.kingsrook.qbits.crm.scoring.processes.CalculateLeadScoresProcess; import com.kingsrook.qbits.crm.scoring.processes.CalculateLeadScoresProcess;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
import com.kingsrook.qqq.backend.core.health.HealthStatus;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData; import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider; import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile; import com.kingsrook.qqq.backend.core.processes.implementations.bulk.insert.model.BulkLoadProfile;
import com.kingsrook.qqq.middleware.health.model.metadata.HealthStatus;
import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThat;
@@ -56,41 +51,27 @@ import static org.assertj.core.api.Assertions.assertThat;
class CrmHealthBulkLoadTest extends BaseTest class CrmHealthBulkLoadTest extends BaseTest
{ {
/******************************************************************************* /*******************************************************************************
** No transport or score run needs attention; the producer registered the ** No transport or score run needs attention.
** same checks with the core health registry.
*******************************************************************************/ *******************************************************************************/
@Test @Test
void noTransportOrScoreRunNeedsAttention() throws Exception void noTransportOrScoreRunNeedsAttention() throws Exception
{ {
var checks = new CrmHealthChecks(new CrmQBitConfig()); var checks = new CrmHealthChecks(new CrmQBitConfig());
var results = checks.run(); var results = checks.run();
assertThat(results.get(CrmHealthChecks.EMAIL).getStatus()).isEqualTo(HealthStatus.DEGRADED); assertThat(results.get("crm.email").getStatus()).isEqualTo(HealthStatus.DEGRADED);
assertThat(results.get(CrmHealthChecks.LEAD_SCORE).getStatus()).isEqualTo(HealthStatus.DEGRADED); assertThat(results.get("crm.leadScore").getStatus()).isEqualTo(HealthStatus.DEGRADED);
assertThat(results.get(CrmHealthChecks.SEQUENCE_LAG).getStatus()).isEqualTo(HealthStatus.UP); assertThat(results.get("crm.sequenceLag").getStatus()).isEqualTo(HealthStatus.UP);
for(HealthCheckMetaData registered : crmChecks(QContext.getQInstance())) checks.indicators().forEach(indicator ->
{ {
assertThat(registered.getCheckInstance().run(new HealthCheckInput()).getStatus()).isEqualTo(results.get(registered.getName()).getStatus()); try
} {
} assertThat(indicator.check(QContext.getQInstance()).getStatus()).isEqualTo(results.get(indicator.getName()).getStatus());
}
catch(Exception e)
/******************************************************************************* {
** The producer registers exactly the three CRM checks as valid core throw (new AssertionError(e));
** definitions, and re-registration does not duplicate them. }
*******************************************************************************/
@Test
void producerRegistersCoreHealthChecks()
{
QInstance instance = QContext.getQInstance();
assertThat(crmChecks(instance)).extracting(HealthCheckMetaData::getName)
.containsExactlyInAnyOrder(CrmHealthChecks.EMAIL, CrmHealthChecks.SEQUENCE_LAG, CrmHealthChecks.LEAD_SCORE);
assertThat(crmChecks(instance)).allSatisfy(check ->
{
assertThat(check.getCheckInstance()).isNotNull();
assertThat(check.getTimeoutMs()).isLessThan(check.getIntervalSeconds() * 1000);
}); });
CrmHealthChecks.register(instance, new CrmQBitConfig());
assertThat(crmChecks(instance)).hasSize(3);
} }
@@ -120,8 +101,8 @@ class CrmHealthBulkLoadTest extends BaseTest
.withValue("lastSuccess", Instant.now().minus(181, ChronoUnit.MINUTES))); .withValue("lastSuccess", Instant.now().minus(181, ChronoUnit.MINUTES)));
com.kingsrook.qbits.crm.core.CrmRecords.update("crm_sequence_enrollment", new QRecord().withValue("id", 1) com.kingsrook.qbits.crm.core.CrmRecords.update("crm_sequence_enrollment", new QRecord().withValue("id", 1)
.withValue("nextStepDate", Instant.now().minus(31, ChronoUnit.MINUTES))); .withValue("nextStepDate", Instant.now().minus(31, ChronoUnit.MINUTES)));
assertThat(checks.run().get(CrmHealthChecks.SEQUENCE_LAG).getStatus()).isEqualTo(HealthStatus.DEGRADED); assertThat(checks.run().get("crm.sequenceLag").getStatus()).isEqualTo(HealthStatus.DEGRADED);
assertThat(checks.run().get(CrmHealthChecks.LEAD_SCORE).getStatus()).isEqualTo(HealthStatus.DEGRADED); assertThat(checks.run().get("crm.leadScore").getStatus()).isEqualTo(HealthStatus.DEGRADED);
} }
@@ -139,7 +120,7 @@ class CrmHealthBulkLoadTest extends BaseTest
throw (new IllegalStateException("unavailable")); throw (new IllegalStateException("unavailable"));
} }
}); });
assertThat(checks.check(CrmHealthChecks.EMAIL).run(new HealthCheckInput()).getStatus()).isEqualTo(HealthStatus.UNKNOWN); assertThat(checks.indicators().getFirst().check(QContext.getQInstance()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
} }
@@ -182,36 +163,27 @@ class CrmHealthBulkLoadTest extends BaseTest
} }
/******************************************************************************* /*******************************************************************************
** The core HealthRunner runs each check on a worker thread with no caller ** Health middleware invokes checks in worker threads without a caller QContext.
** QContext, as a QSystemUserSession. A registered CRM check must succeed that
** way on a host whose authentication rejects tokenless sessions (OAuth2, Auth0).
*******************************************************************************/ *******************************************************************************/
@Test @Test
void registeredChecksRunAsCoreSystemUserOnTokenRequiredHost() throws Exception void indicatorsCarryMonitoringSessionAndRestoreWorkerContext() throws Exception
{ {
var instance = QContext.getQInstance(); var instance = QContext.getQInstance();
instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData() var indicator = new CrmHealthChecks(new CrmQBitConfig()).indicators(QContext.getQSession()).get(1);
.withName("token-required")
.withType(QAuthenticationType.MOCK)
.withModuleCodeReference(new QCodeReference(TokenRequiredAuthenticationModule.class)));
HealthCheckMetaData sequenceLag = crmChecks(instance).stream()
.filter(check -> CrmHealthChecks.SEQUENCE_LAG.equals(check.getName())).findFirst().orElseThrow();
var status = java.util.concurrent.CompletableFuture.supplyAsync(() -> var status = java.util.concurrent.CompletableFuture.supplyAsync(() ->
{ {
assertThat(QContext.getQInstance()).isNull();
QContext.init(instance, new QSystemUserSession(instance));
try try
{ {
return (sequenceLag.getCheckInstance().run(new HealthCheckInput().withQInstance(instance)).getStatus()); assertThat(QContext.getQInstance()).isNull();
var result = indicator.check(instance);
assertThat(QContext.getQInstance()).isNull();
assertThat(QContext.getQSession()).isNull();
return (result.getStatus());
} }
catch(Exception e) catch(Exception e)
{ {
throw (new AssertionError(e)); throw (new AssertionError(e));
} }
finally
{
QContext.clear();
}
}).get(); }).get();
assertThat(status).isEqualTo(HealthStatus.UP); assertThat(status).isEqualTo(HealthStatus.UP);
assertThat(QContext.getQInstance()).isSameAs(instance); assertThat(QContext.getQInstance()).isSameAs(instance);
@@ -220,12 +192,32 @@ class CrmHealthBulkLoadTest extends BaseTest
/******************************************************************************* /*******************************************************************************
** The CRM definitions in the core health registry. ** Regression: indicators registered with no caller session (at host startup)
** must still run on a host whose authentication rejects tokenless sessions
** (OAuth2, Auth0). They used a plain new QSession(), so every check failed
** session validation and reported UNKNOWN.
*******************************************************************************/ *******************************************************************************/
private static List<HealthCheckMetaData> crmChecks(QInstance instance) @Test
void contextlessIndicatorsRunOnTokenRequiredHost() throws Exception
{ {
return (HealthChecksMetaData.ofOrWithNew(instance).getChecks().stream() var instance = QContext.getQInstance();
.filter(check -> "crm".equals(check.getComponent())).toList()); instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData()
.withName("token-required")
.withType(QAuthenticationType.MOCK)
.withModuleCodeReference(new QCodeReference(TokenRequiredAuthenticationModule.class)));
var status = java.util.concurrent.CompletableFuture.supplyAsync(() ->
{
try
{
var indicator = new CrmHealthChecks(new CrmQBitConfig()).indicators().get(1);
return (indicator.check(instance).getStatus());
}
catch(Exception e)
{
throw (new AssertionError(e));
}
}).get();
assertThat(status).isEqualTo(HealthStatus.UP);
} }
} }