From 5415cdb2807880ff701a21204ecf8b42cd4f2948 Mon Sep 17 00:00:00 2001 From: James Maes Date: Tue, 13 Oct 2026 19:27:30 -0500 Subject: [PATCH] docs: qbit-email design spec Outbox on core's messaging SPI, providers as data rows (SMTP, SES, Postmark, SendGrid, Mailgun, Gmail), Markdown and MJML/HTML templates with versions and dark-mode-safe branding, delivery tracking through signed webhooks, suppression, RFC 8058 unsubscribe, PHI rules, consumer migrations and the inbound phase. Owner decisions (2026-10-12): all recommendations accepted. Co-Authored-By: Claude Opus 5.5 --- docs/DESIGN.md | 447 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 445 insertions(+), 2 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 42e3824..b1c7e76 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -1,3 +1,446 @@ -# QBit Email design +# qbit-email design -The design spec arrives in the spec PR (`feature/design-spec`). +**Status:** spec. Phases (a) to (d) build outbound; phase (e) is inbound (section 12, design only). +**Brief:** owner decisions of 2026-10-13 (providers, outbound first, both template kinds). +**Owner decisions (2026-10-12): all recommendations accepted.** Every "Recommended" choice and +every decision below stands; it is adjusted once it can be seen working. + +One email service for a whole QQQ system. Every qbit and app that sends mail queues it through +this qbit's outbox and gets the same providers, templates, branding, tracking, suppression, +unsubscribe and PHI rules. + +| # | Decision | +|---|---| +| D1 | Build on core's messaging SPI (`MessagingProviderInterface`, `SendMessageAction`). The qbit registers one provider type, `QBIT_EMAIL`, that writes to the outbox, so existing `SendMessageAction` callers move by pointing their `messagingProviderName` at it (section 2.4). Core's built-in `EMAIL` and `SES` types are not overridden. | +| D2 | Delivery transports live in the qbit (section 3): core's SMTP provider has no auth, STARTTLS or timeouts, and its SES provider uses AWS SDK v1 with static keys in metadata. No core PR is required; the transports can move to core later without changing the tables. | +| D3 | Providers are data rows (`email_provider`). Credentials are environment-variable NAMES, never values (the qbit-assistant rule). | +| D4 | Every send goes through the outbox (`email_message`); nothing sends inline. Async, at least once, retries with backoff, dead letters, idempotency keys. | +| D5 | Two template kinds: Markdown with `{{variables}}` inside the branded responsive layout (the default), and raw MJML or HTML (marketing). Templates are versioned. | +| D6 | Tracking events (SES via SNS, Postmark, SendGrid, Mailgun) arrive at `/qqq/hooks/email/...` (not the UI API), are signature-verified, and land in a durable inbox before processing. | +| D7 | Hard bounces and complaints suppress the address. One-click unsubscribe (RFC 8058) records a suppression for the message's category, and users see it as a Preferences section (core #1162). | +| D8 | PHI-safe by default: PII/PHI-classified variables are refused unless the template is `phi_approved`. The recommended PHI pattern is a "secure message" (a sign-in link, no PHI in the body). | +| D9 | Gmail: SMTP with an app password in phase (b), as a preset of the SMTP transport. The Gmail API with OAuth is specified (section 3.6) and built only on request. | +| D10 | MJML is compiled when a version is activated, by a pluggable compiler (default: the `mjml` CLI when configured). Sends never need the compiler; HTML templates need nothing. | + +## 1. Flow + +``` + EmailService.send(EmailRequest) ─┐ + email_send_templated (process) ─┤ + SendMessageAction (type QBIT_EMAIL) ─┼─► EmailOutbox.enqueue ─► email_message (QUEUED; rendered, redacted) + record action "Send Test Email" ─┘ idempotency key unique; suppression and PHI checks first + │ + scheduler (every 15 s) ─► EmailDispatcher.runOnce + claim attempt n (unique insert into email_message_attempt) + ─► provider rate limit ─► EmailTransport.send (SMTP | SES | Postmark | SendGrid | Mailgun | Gmail) + ─► SENT (provider message id) │ RETRY_WAIT (backoff) │ DEAD_LETTER + │ + provider webhook ─► /qqq/hooks/email/{kind}/{providerName} ─► verify ─► email_webhook_inbox (durable) + ─► inbox processor ─► email_event (DELIVERED, BOUNCED, COMPLAINED, OPENED, CLICKED, ...) + ─► message status ─► email_suppression (hard bounce, complaint) + List-Unsubscribe one-click POST ─► /qqq/hooks/email/unsubscribe/{token} ─► email_suppression (category) +``` + +## 2. Sending API + +### 2.1 `EmailService` (Java, for every consumer) + +```java +EmailSendResult result = EmailService.send(new EmailRequest() + .withTemplateName("alerts_notification") // or withSubject + withMarkdownBody / withHtmlBody for ad-hoc mail + .withTo("pat@example.com") // withCc, withBcc, withReplyTo; "addr" or "Name " + .withVariable("count", 3) // plain values, checked against the template's schema + .withRecordVariable("order", "order", 1234) // a record, read as the sender; classification known + .withCategory(EmailCategory.NOTIFICATION) // default: the template's + .withProviderName("pmg") // optional; else the default provider + .withBrandName("kof22") // optional; else the template's, else the default brand + .withIdempotencyKey("alerts:notif:991") // optional; a repeat returns the existing message + .withLinkedRecord("alerts_alert", 991) // shown as a link on the message record + .withSender("qbit-alerts", "alerts_notify") // sender qbit and process + .withSendAfter(instant)); // optional deferral +// EmailSendResult { messageId, status (QUEUED | DUPLICATE | SUPPRESSED | REFUSED), reasons } +``` + +- Runs inside the caller's transaction when the backends match (the row commits with the caller's + write), else after commit, as qbit-webhooks does. +- Validation is synchronous: unknown template, missing required variables, bad addresses, a + PII/PHI variable on a template that is not `phi_approved` (section 7), and a disabled or missing + provider all return `REFUSED` with plain reasons, and nothing is queued. +- All recipients suppressed for the category: `SUPPRESSED`; the message is stored with that status + (visible in the outbox) and not sent. Partly suppressed: those addresses are dropped and listed + on the message. + +### 2.2 Process `email_send_templated` +The same request as a backend process (template, to, cc, bcc, variables JSON, linked table and +id, idempotency key, provider, brand) for scripts, the API and other qbits. Not in the nav; +callable through the API and `/qqq/v1/processes`. + +### 2.3 Permissions +`email.view` (read every `email_` table), `email.send` (send through the API or the process; +system callers use the authorized scope), `email.manage` (templates, brands, suppressions, resend, +cancel), `email.admin` (providers, test send, PHI approval, removing complaint suppressions). A +template may set `edit_permission_name` to restrict who edits it (e.g. marketing templates to +`marketing.manage`). + +### 2.4 Core messaging bridge +- `EmailOutboxMessagingProviderMetaData` (type `QBIT_EMAIL`, a `QMessagingProviderMetaData`) + registers `EmailOutboxMessagingProvider` with `QMessagingProviderDispatcher`. The producer adds + one instance named `qbitEmail` (configurable). +- `sendMessage(SendMessageInput)` maps the parties (`EmailPartyRole` TO/CC/BCC/FROM), subject and + contents (`EmailContentRole` TEXT/HTML) to an ad-hoc `EmailRequest` (no template, category + TRANSACTIONAL unless configured) and enqueues it. It throws on `REFUSED`, so callers still see + failures; delivery is then asynchronous. +- Attachments are stored through qbit-object-storage when it is installed, else refused with a + plain message (no blobs in the outbox). +- Callers (qbit-alerts `EmailChannel` and `NotificationEmailer`, qbit-marketing + `QqqMessagingEmailTransport`) can switch by configuration at once, then move to `EmailService` + in phase (d) for templates, categories, idempotency and linked records. + +## 3. Providers and transports + +### 3.1 `email_provider` +| Field | Notes | +|---|---| +| name, label | unique name, used by `withProviderName` | +| kind | SMTP, SES, POSTMARK, SENDGRID, MAILGUN, GMAIL_SMTP, GMAIL_API | +| enabled, is_default | exactly one enabled default (validated) | +| from_address, from_name, reply_to | defaults for messages | +| sending_domain | the domain the provider is verified for; other from domains are refused | +| host, port, security | SMTP: NONE, STARTTLS, TLS; timeouts fixed (10 s connect, 15 s read and write) | +| region | SES region; Mailgun region (US or EU) | +| username_env_name, secret_env_name | env-var NAMES only (`^[A-Z_][A-Z0-9_]{0,99}$`, and in the config allowlist) | +| webhook_secret_env_name | env-var NAME for webhook verification (Mailgun signing key, SendGrid public key, Postmark basic-auth password) | +| sns_topic_arn, configuration_set, message_stream | SES events topic and configuration set; Postmark message stream | +| max_per_minute, max_per_day | counted from `email_message_attempt` in the database, so they hold across replicas | +| track_opens, track_clicks | off by default; refused for `phi_approved` templates | +| phi_allowed | admin attestation that the provider may carry PHI (BAA in place, or an internal relay) | +| health_status, last_success_date, last_failure_date, last_failure_message | kept by the dispatcher | + +Secrets resolve through `EmailQBitConfig.secretResolver` (default `System::getenv`) at send time. +They are never stored, logged, audited, put in process output or returned. The record screen shows +whether each named variable is set (yes or no), never the value. + +**Record action "Send Test Email"** (`email.admin`): queues a fixed test message to a given +address through that provider and reports the outcome on the standard result screen. + +### 3.2 `EmailTransport` SPI (in the qbit) +```java +public interface EmailTransport +{ + EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException; +} +``` +`EmailEnvelope` holds the rendered MIME parts: from, to, cc, bcc, reply-to, subject, HTML, text, +and headers (`List-Unsubscribe`, `Auto-Submitted`, `X-QQQ-Email-Id`). The result is the +provider's message id. The exception says whether the failure is **permanent** (rejected address, +sender refused, 4xx validation) or **transient** (connection, 5xx, throttling with an optional +retry-after). + +### 3.3 SMTP (phase a) +`jakarta.mail` (already in core): auth when a username env name is set, STARTTLS or implicit TLS, +connect/read/write timeouts (the kof22 `Mailer` lesson: a stuck relay must not pin a thread). +Covers PMG (`smtp.galaxy.lan:26`, no auth), any relay, and Gmail SMTP. + +### 3.4 SES (phase a) +AWS SDK v2 `sesv2` `SendEmail` with raw content, so our MIME and headers are kept. Credentials: +the default AWS chain when no env names are set (IRSA, instance role, `AWS_*`), else the named +access-key and secret env vars. The configuration set routes events to SNS (section 6). + +### 3.5 Postmark, SendGrid, Mailgun (phase b) +HTTP JSON APIs through `java.net.http`, no vendor SDKs: Postmark `POST /email` (server-token +header, message stream), SendGrid `POST /v3/mail/send` (bearer key; `X-Message-Id`), Mailgun +`POST /v3/{domain}/messages.mime` (basic auth `api:key`, US or EU base). 429 and 5xx are +transient; other 4xx are permanent. + +### 3.6 Gmail +- **Phase (b): `GMAIL_SMTP`**, a preset of SMTP (`smtp.gmail.com:587`, STARTTLS) with the account + as username and an app password env var. `docs/PROVIDERS.md` documents it: it needs 2-step + verification and an app password; Google's daily limits (about 500 for a personal account, 2000 + for Workspace) go in `max_per_day`. +- **`GMAIL_API` (specified, built on request):** OAuth client id, secret and refresh token (env + names), `users.messages.send` with the raw MIME. It needs a Google Cloud project and consent + screen and gives system mail nothing the app password does not, so it is deferred. + +## 4. Outbox and delivery + +### 4.1 `email_message` +| Field | Notes | +|---|---| +| status | QUEUED, SENDING, SENT, DELIVERED, RETRY_WAIT, DEAD_LETTER, BOUNCED, COMPLAINED, SUPPRESSED, CANCELLED | +| to_addresses, cc_addresses, bcc_addresses, from_address, reply_to | addresses are PII; the table is classified PII (section 7) | +| subject | rendered; encrypted for `phi_approved` templates (section 7.3) | +| template_id, template_version_id, brand_id, category, locale | | +| variables_snapshot | JSON, redacted per section 7.3 | +| html_body, text_body | rendered at queue time so retries send the same bytes; encrypted for `phi_approved`; cleared after `bodyRetentionDays` (default 30) once final, keeping `body_sha256` | +| idempotency_key | unique when present | +| provider_id, provider_message_id | | +| attempt_count, next_attempt_date, lease_expires_date, last_error, sent_date, delivered_date, dead_letter_date | | +| linked_table_name, linked_record_id | rendered as a link on the record screen | +| sender_qbit, sender_process | | +| suppressed_addresses | addresses dropped at queue time | + +Sub-tables, shown as relations on the message: `email_message_attempt` (message_id and +attempt_number unique as a pair, outcome, provider_id, claimed_by, start and end dates, duration, +error) and `email_event` (section 6). + +### 4.2 Semantics (as qbit-webhooks, proven across replicas) +- **Claim:** to send attempt *n* a runner inserts (`message_id`, *n*) into + `email_message_attempt`; the pair is unique, so exactly one runner sends. +- **Lease:** the winner marks the message SENDING with a 120 s lease; an expired lease is retried + as attempt *n+1*. +- **At least once:** a runner that dies after the provider accepted, but before recording it, + causes at most one repeat. None of the providers offers a send-idempotency key, so the spec says + so plainly. +- **Outcomes:** success → SENT. Transient → RETRY_WAIT after `retryMinutes` (1, 5, 15, 60, 240, + 720; retry-after honoured, capped at 1 h). Permanent → DEAD_LETTER at once. After `maxAttempts` + (6) → DEAD_LETTER. +- **Rate limits** per provider per minute and per day, from the database. Over a limit the message + waits; that is not an attempt. +- **Provider health:** HEALTHY → UNHEALTHY after 10 consecutive transient failures; while + unhealthy one message is tried every 5 minutes; a success → HEALTHY. Alerts (optional, section + 9): `email.provider.unhealthy` (ERROR, resolved on recovery) and `email.message.dead_letter` + (WARNING, deduplicated per provider per hour). +- **Record actions:** Resend (dead letters and bounces: a new linked message with the same + content), Cancel (queued or waiting), Send Now (waiting). Standard streamed-ETL Review and Result + screens. +- **Retention:** a daily job clears bodies after `bodyRetentionDays` and deletes final messages + older than `messageRetentionDays` (0 = keep forever; kof22 keeps). + +## 5. Templates and branding + +### 5.1 Tables +- `email_template`: name (unique, snake_case), label, category (TRANSACTIONAL, NOTIFICATION, + MARKETING, SECURITY), kind (MARKDOWN, MJML, HTML), brand_id, active_version_id, phi_approved, + phi_approved_by, phi_approved_date, edit_permission_name, owner_qbit, description. +- `email_template_version` (child list on the template): template_id, version_number, status + (DRAFT, ACTIVE, RETIRED), locale (default `en`), subject, preheader, body_source, text_source + (optional), compiled_html (MJML), variables_schema (JSON), sample_variables (JSON), created_by, + activated_date. +- `email_brand`: name, label, logo_url, logo_dark_url, logo_alt, primary_color, text_color, + background_color, dark_background_color, dark_text_color, button_text_color, footer_markdown, + postal_address (shown on marketing mail, for CAN-SPAM), social_links (JSON list of label and + url), site_url, is_default, app_name (optional: the brand for that app's sends). + +An ACTIVE version cannot be edited. **Record action "New Version"** copies it to a DRAFT; +**"Activate Version"** validates it (schema, a render with the samples, the PHI check, MJML +compile, brand contrast) and makes it active, retiring the previous one. A message always records +the exact version it used. + +### 5.2 Rendering +- **Variables:** `{{name}}` and dotted paths (`{{order.number}}`), always HTML-escaped; + `{{#if name}}…{{/if}}` and `{{#each items}}…{{/each}}`; `{{name|fallback text}}`. No unescaped + output and no helpers that run code. Unknown variables fail activation, not sends. +- **Schema:** each variable has name, label, type (STRING, NUMBER, DATE, DATE_TIME, BOOLEAN, URL, + LIST, RECORD), required, an optional classification, and an optional bound `table` and `field` + (the classification then comes from the field, section 7). +- **Markdown** (commonmark-java with GFM tables and autolinks; raw HTML in Markdown disabled) + renders into the branded layout: a fixed, tested, table-based responsive shell (600 px max, + single column, inline styles plus one `