fix(health): real CloudFront health probe; UP/not-testable instead of UNKNOWN #29

2026-10-16T16:39:05Z

KofTwentyTwo commented

2026-10-16 16:39:05 +00:00

(Migrated from github.com)

Problem

kof22 prod (qbit-cdn 0.1.8): health check cdn.provider.1 is stuck at UNKNOWN, message "CDN provider does not support connection tests", details {}.

Cause

CdnProviderHealthChecks.run returns that exact UNKNOWN only when the adapter for the row's providerType does not implement CdnOperations. In 0.1.8 that is only NONE (NoneCdnProvider) and SOUIN. CloudFrontCdnProvider already implements CdnOperations in 0.1.8. A CLOUDFRONT row would have run the connection test (GetDistributionConfig + ListInvalidations, both granted) and come back UP or DEGRADED. So provider row 1 is almost certainly the "No CDN (record only)" NONE placeholder that the "Create default provider" action seeds. It is probably not the CloudFront row. The existing test reloadsProviderAndHandlesUnsupported asserted exactly this NONE -> UNKNOWN behaviour.

UNKNOWN can never be useful here. In QQQ core HealthTransitionEvaluator, "UNKNOWN breaks consecutive streaks but never changes state or raises alerts", so a check that is always UNKNOWN alerts nobody and adds noise to the dashboard.

Design

New optional SPI CdnHealthProbe / CdnHealth (state UP/DEGRADED/DOWN, fixed message, safe details). The central check prefers it. It falls back to the existing connection test, and otherwise reports UP with testable=false. NONE gives "No CDN configured; purges are recorded only". Other adapters without a connection test give "CDN provider configured; not testable (no connection test)". The deadline, virtual-thread and cancel handling is unchanged and now shared.
CloudFront implements checkHealth, using the existing CloudFrontClientFactory (AWS SDK v2, same credentials as invalidations):
- GetDistribution:
  - UP when Enabled and Deployed.
  - DEGRADED when InProgress ("deploying a change (InProgress)").
  - DOWN on NoSuchDistribution or Enabled=false.
- Throttling, AWS 5xx or network errors give DEGRADED ("throttled or unavailable; distribution status not checked", transient=true).
- Rejected credentials or other 4xx give DOWN. Missing env credentials or a missing distribution id give DOWN.
- Origin-verify header missing gives DEGRADED, the same as before.
- AccessDenied on GetDistribution falls back to GetDistributionConfig. That call is granted and still proves existence and Enabled. The check reports UP, message "CloudFront configured; distribution status not testable with current permissions", details {"testable":false,"reason":"credentials lack cloudfront:GetDistribution", ...}. If both calls are denied, it reports UP with testable=false and the combined reason.
- CloudWatch AWS/CloudFront 5xxErrorRate (us-east-1, DistributionId + Region=Global, Average, 5-minute periods, last 30 minutes) is added as errorRate5xxPercent. Above 5% the check is DEGRADED. It is best effort: no datapoints or a failed read leave the field out. No new dependency was needed, because cloudwatch was already used for stats import and the kof22 key already has cloudwatch:GetMetricData.
Messages are fixed text. Details hold only booleans, numbers and known states (Deployed/InProgress/UNKNOWN). No AWS message, id, hostname or credential reaches the health tables.

Why UP + not-testable rather than UNKNOWN

UNKNOWN that never changes is useless and alerts nobody. UP with testable=false honestly says "configured; this part cannot be verified". Real failures stay visible elsewhere:

Failed invalidations are recorded as FAILED (retrying) and then DEAD rows in cdn_purge_request with errorMessage (CdnPurgeDispatcher.fail).
They are counted for 24h and 7d on the CDN home (CdnHomeData) and logged (LOG.warn in the dispatcher).
Note: purge failures do not currently raise a central health alert. That is a possible follow-up and out of scope here.

Permissions (kof22)

KofTwentyTwo/Praetor terraform/cdn-cloudfront/iam.tf (user cdn-qbit-kof22) grants these on the distribution: CreateInvalidation, GetInvalidation, ListInvalidations, GetDistributionConfig. It also grants cloudwatch:GetMetricData/ListMetrics on *. It does not grant cloudfront:GetDistribution, so until that is added kof22 gets UP/testable=false, with Enabled and origin header still checked and the 5xx rate included. A separate Praetor PR proposes adding cloudfront:GetDistribution.

Tests

CloudFrontHealthTest (new, 13 tests) uses mocked CloudFrontClient and CloudWatchClient, with no network. It covers:
- Deployed gives UP, and the metric query shape is correct.
- InProgress gives DEGRADED. Disabled gives DOWN.
- Origin header missing gives DEGRADED.
- 5xx rate: the 5% boundary, average, no data and a failed read.
- AccessDenied falls back to config (UP, not testable, and DOWN when disabled). Both denied gives UP, not testable.
- NoSuchDistribution, InvalidClientTokenId, Throttling and ServiceUnavailable, both directly and after the fallback.
- Network, no-credentials, no-id and generic SDK failures.
- No secrets in messages or details.
CdnProviderHealthChecksTest checks that NONE and SOUIN are now UP with testable=false, and that the probe is preferred over the connection test with state, message and details passed through.
./mvnw verify passes locally: 137 tests, JaCoCo gate, guardrails.

Release

kof22 pins qbit-cdn 0.1.8. After merge, cut 0.1.9 and bump qbit-cdn.version in Website-Backend.

🤖 Generated with Claude Code

## Problem kof22 prod (qbit-cdn 0.1.8): health check `cdn.provider.1` is stuck at **UNKNOWN**, message "CDN provider does not support connection tests", details `{}`. ## Cause `CdnProviderHealthChecks.run` returns that exact UNKNOWN only when the adapter for the row's `providerType` does not implement `CdnOperations`. In 0.1.8 that is only **NONE** (`NoneCdnProvider`) and **SOUIN**. `CloudFrontCdnProvider` already implements `CdnOperations` in 0.1.8. A CLOUDFRONT row would have run the connection test (GetDistributionConfig + ListInvalidations, both granted) and come back UP or DEGRADED. So provider row 1 is almost certainly the **"No CDN (record only)" NONE placeholder** that the "Create default provider" action seeds. It is probably not the CloudFront row. The existing test `reloadsProviderAndHandlesUnsupported` asserted exactly this NONE -> UNKNOWN behaviour. UNKNOWN can never be useful here. In QQQ core `HealthTransitionEvaluator`, "UNKNOWN breaks consecutive streaks but never changes state or raises alerts", so a check that is always UNKNOWN alerts nobody and adds noise to the dashboard. ## Design - **New optional SPI** `CdnHealthProbe` / `CdnHealth` (state UP/DEGRADED/DOWN, fixed message, safe details). The central check prefers it. It falls back to the existing connection test, and otherwise reports **UP with `testable=false`**. NONE gives "No CDN configured; purges are recorded only". Other adapters without a connection test give "CDN provider configured; not testable (no connection test)". The deadline, virtual-thread and cancel handling is unchanged and now shared. - **CloudFront implements `checkHealth`**, using the existing `CloudFrontClientFactory` (AWS SDK v2, same credentials as invalidations): - `GetDistribution`: - UP when Enabled and `Deployed`. - DEGRADED when `InProgress` ("deploying a change (InProgress)"). - DOWN on `NoSuchDistribution` or Enabled=false. - Throttling, AWS 5xx or network errors give DEGRADED ("throttled or unavailable; distribution status not checked", `transient=true`). - Rejected credentials or other 4xx give DOWN. Missing env credentials or a missing distribution id give DOWN. - Origin-verify header missing gives DEGRADED, the same as before. - **AccessDenied on GetDistribution** falls back to `GetDistributionConfig`. That call is granted and still proves existence and Enabled. The check reports **UP**, message "CloudFront configured; distribution status not testable with current permissions", details `{"testable":false,"reason":"credentials lack cloudfront:GetDistribution", ...}`. If both calls are denied, it reports UP with `testable=false` and the combined reason. - **CloudWatch** `AWS/CloudFront` `5xxErrorRate` (us-east-1, DistributionId + Region=Global, Average, 5-minute periods, last 30 minutes) is added as `errorRate5xxPercent`. Above 5% the check is DEGRADED. It is best effort: no datapoints or a failed read leave the field out. No new dependency was needed, because `cloudwatch` was already used for stats import and the kof22 key already has `cloudwatch:GetMetricData`. - Messages are fixed text. Details hold only booleans, numbers and known states (`Deployed`/`InProgress`/`UNKNOWN`). No AWS message, id, hostname or credential reaches the health tables. ### Why UP + not-testable rather than UNKNOWN UNKNOWN that never changes is useless and alerts nobody. UP with `testable=false` honestly says "configured; this part cannot be verified". Real failures stay visible elsewhere: - Failed invalidations are recorded as `FAILED` (retrying) and then `DEAD` rows in `cdn_purge_request` with `errorMessage` (`CdnPurgeDispatcher.fail`). - They are counted for 24h and 7d on the CDN home (`CdnHomeData`) and logged (`LOG.warn` in the dispatcher). - Note: purge failures do not currently raise a central health alert. That is a possible follow-up and out of scope here. ## Permissions (kof22) `KofTwentyTwo/Praetor` `terraform/cdn-cloudfront/iam.tf` (user `cdn-qbit-kof22`) grants these on the distribution: `CreateInvalidation`, `GetInvalidation`, `ListInvalidations`, `GetDistributionConfig`. It also grants `cloudwatch:GetMetricData`/`ListMetrics` on `*`. It does **not** grant `cloudfront:GetDistribution`, so until that is added kof22 gets UP/testable=false, with Enabled and origin header still checked and the 5xx rate included. A separate Praetor PR proposes adding `cloudfront:GetDistribution`. ## Tests - `CloudFrontHealthTest` (new, 13 tests) uses mocked `CloudFrontClient` and `CloudWatchClient`, with no network. It covers: - Deployed gives UP, and the metric query shape is correct. - InProgress gives DEGRADED. Disabled gives DOWN. - Origin header missing gives DEGRADED. - 5xx rate: the 5% boundary, average, no data and a failed read. - AccessDenied falls back to config (UP, not testable, and DOWN when disabled). Both denied gives UP, not testable. - NoSuchDistribution, InvalidClientTokenId, Throttling and ServiceUnavailable, both directly and after the fallback. - Network, no-credentials, no-id and generic SDK failures. - No secrets in messages or details. - `CdnProviderHealthChecksTest` checks that NONE and SOUIN are now UP with `testable=false`, and that the probe is preferred over the connection test with state, message and details passed through. - `./mvnw verify` passes locally: 137 tests, JaCoCo gate, guardrails. ## Release kof22 pins `qbit-cdn` 0.1.8. After merge, cut 0.1.9 and bump `qbit-cdn.version` in Website-Backend. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-cdn#29