# QBit Email
One email service for a QQQ system: a durable outbox (`email_message`) with retries, dead
letters, idempotency keys, rate limits and provider health; SMTP, Amazon SES, Postmark,
SendGrid, Mailgun and Gmail SMTP transports; versioned Markdown, HTML and MJML templates with
dark-mode-safe branding; classified variables and PHI protection; preview widgets; a
real-data home; delivery events, suppression and one-click unsubscribe; and the core messaging bridge. Every send goes through the outbox.
The [design](docs/DESIGN.md) also specifies the deferred Preferences UI (core #1162),
consumer migrations and inbound mail. Preview uses the standard widget endpoint; the
dedicated preview API arrives with QQQ RC.6. Attachments are refused until
qbit-object-storage is wired in.
## Install
Requires Java 21, QQQ **4.1.0-RC.8**, and a transactional RDBMS with the supplied unique
constraints. Apply [the PostgreSQL DDL](docs/ddl/email-postgres.sql) before producing the qbit. SQL timestamps store UTC values without a timezone, matching QQQ RC.5.
The physical table names equal their metadata names; `backendName` selects the host's
QQQ database backend, as it does in qbit-webhooks.
```xml
com.kingsrook.qbits
qbit-email
${qbit-email.version}
```
```java
new EmailQBitProducer()
.withQBitConfig(new EmailQBitConfig()
.withBackendName("rdbms")
.withSchedulerName("scheduler"))
.produce(qInstance);
```
The scheduler must already exist. Without a scheduler, callers can enqueue and run
`EmailDispatcher` or Send Now manually; automatic dispatch is not installed.
Configuration is looked up from the current `QInstance` on every call, never cached
in a static variable.
## Starter defaults and setup
Include order: the DDL first, then the starter defaults from the host's master changelog (the qbit itself never runs DDL):
```yaml
- include: { file: db/liquibase/qbit-email/email-defaults.changelog.sql }
```
They are insert-only, hold no secrets, skip a host without the tables, and never change how mail looks on deploy. They add one brand, `neutral` (white background, near-black text, blue buttons, light and dark schemes, every pair at least 4.5:1 contrast), and only when the host has no brand at all. It is not the default brand: copy it for your own, or tick Default Brand on it.
Everything else starts empty on purpose, and each table's help says why: a **provider** is your own connection (a server and the NAMES of its secret variables), so nothing can be sent until you add one; messages, attempts, events, the webhook inbox and suppressions are runtime data; the secure-message template is created by the qbit on first use and when the Email home opens, and other qbits register theirs with `EmailApi.ensureTemplate`.
Every table has help that says what it is for and **how its rows get created**, available through the host's help bundle sync. Email admins also get a **Setup** card first on the home: provider, sender addresses, brands, templates, marketing readiness, delivery tracking, suppression list and scheduler, each with its state and the next step. It uses the same provider and brand resolution as the sender and `EmailApi.readiness`.
### In-app help
The producer registers [help/qbit-email.yaml](src/main/resources/help/qbit-email.yaml) as a `QBIT` bundle using QQQ's help bundle API. Registration is metadata-only and works without a `helpContent` table; hosts with help enabled load and sync registered bundles through QQQ. The app entry follows `EmailQBitConfig.appName`.
To override wording or add deployment-specific instructions, register a separate `layer: HOST` bundle with the same entry keys and roles using `HelpBundles.register(qInstance, "help/your-host.yaml")`. HOST entries take precedence over QBIT entries; UI edits remain protected by the core sync. Keep the bundle version aligned with the POM's artifact version when releasing.
`EmailTablesGuardrailTest.everyTableHasEmptyStateHelp` runs `HelpBundleAssertions` against the produced instance to check owned objects, non-trivial fields, summaries and live references, with no exclusions. `EmailQBitProducerTest` checks registration without a help table and with a custom app name. Both run during `./mvnw verify`.
A template's **Owner Qbit** (the qbit that registered it) is now used: messages sent from the template name that qbit as their sender unless the request names one.
## Tables and permissions
| Table | Navigation | Writes |
|---|---|---|
| `email_message` | Messages | Qbit only; people use record actions |
| `email_message_attempt` | Attempts on a message | Qbit only |
| `email_event` | Events on a message | Qbit only; readable with `email.view` |
| `email_webhook_inbox` | Inbox on a provider | Qbit only; readable with `email.view`; Reprocess requires `email.manage` |
| `email_template` | Templates | `email.manage`; template edit grants also apply |
| `email_template_version` | Versions on a template | Draft edits only; lifecycle uses record actions |
| `email_brand` | Brands | `email.manage`; template edit grants also apply |
| `email_suppression` | Suppressions | `email.manage`; changing/removing a complaint needs `email.admin` |
| `email_provider` | Providers, with child Messages | `email.admin`, including reads and test sends |
| `email_provider_sender_address` | Sender Addresses on a provider | `email.admin`, including reads |
`email.view` reads messages, templates, versions, brands and suppressions.
`email.send` permits the Java service, bridge and backend send process. Reading
messages requires `email.view`, `email.manage` or `email.admin`; send alone grants no read access.
`email.manage` also allows sending, Resend, Cancel and Send Now. `email.admin`
implies all permissions. Grant `email.hasAccess` for the app's derived base access.
Enable the host's `QInstance.enforcePermissionsInCoreActions` for direct Java action
permission enforcement; the tables and processes use the shared custom checker.
Trusted system callers use `SystemWrites.call(...)`, which opens core's authorized
scope and still preserves record security locks.
All tables and columns follow the `email_`/snake_case database convention. Java field
names are camelCase, matching the webhooks reference. System tables disable direct
insert/update/delete capabilities. Labels contain no status decorations.
## Providers and credentials
See the [provider setup guide](docs/PROVIDERS.md) for row fields, DNS/domain setup,
limits, Gmail app passwords and provider webhook setup.
| Kind | Transport | Status |
|---|---|---|
| SMTP | Relay or authenticated SMTP, STARTTLS / TLS | Available |
| SES | AWS SES v2, raw MIME | Available |
| POSTMARK | HTTPS JSON, server token and message stream | Available |
| SENDGRID | HTTPS JSON, bearer API key | Available |
| MAILGUN | HTTPS MIME multipart, US / EU | Available |
| GMAIL_SMTP | smtp.gmail.com:587, STARTTLS and app password | Available |
| GMAIL_API | OAuth Gmail API | Deferred; use GMAIL_SMTP |
Create the first enabled provider with `isDefault=true`. When any providers are
enabled, exactly one must be the enabled default. Switch defaults in a batch update,
clearing the old default first. A nullable unique `default_slot` backs the validation.
Providers with message history cannot be deleted; disable them instead.
**Sender addresses.** A provider can own any number of verified from addresses, kept in
`email_provider_sender_address` and shown as a Sender Addresses list on the provider record
(not in the nav; `email.admin` reads and writes, like the provider). Each row has an
`address` (validated, stored lowercase, line breaks refused), an optional `displayName`,
`isDefault`, `enabled` and a `note`. An address is unique per provider, at most one enabled
row per provider is the default, and when the provider has a `sendingDomain` every address must
be in that domain (changing the domain with other-domain rows is refused). Deleting a provider
deletes its rows. Existing databases add the table and the nullable
`email_message.from_address_id` column from `docs/ddl/email-postgres.sql`.
The provider's own `fromAddress` / `fromName` stay the fallback, so nothing changes until a
row exists. A row marked default wins over the fallback. On a send, `withFromAddressId(id)`
(or `fromAddressId` on the send processes) must name an enabled row of the resolved provider,
and once a provider has rows an explicit `withFrom(...)` must match one of them by address;
either failure is `REFUSED` with "That from address is not set up for provider ". With
no rows, `withFrom(...)` behaves as before. The chosen row is recorded on the message as
`fromAddressId`, and a resend reuses it.
The from address pickers are contingent: `fromAddressId` on the send processes lists only the
rows of the chosen `providerId` (`providerName` is still accepted for API callers, and the two
must agree), and the Send Test Email action lists only the rows of the provider being tested.
That action's form has no record id, so a prepare step publishes the single selected provider id
as the process value `possibleValueFilterValueId`, which the picker's filter reads after `${input.id}`.
For an unauthenticated relay, use kind `SMTP`, host/port, security `NONE`, and a
`fromAddress`. Leave credential names empty. Authenticated SMTP uses `usernameEnvName`
and `secretEnvName`; security is `NONE`, `STARTTLS` (required) or `TLS`. Timeouts are
10 seconds to connect and 15 seconds each to read and write. The Gmail SMTP preset
uses `smtp.gmail.com:587` with required STARTTLS.
SES needs a `region`, optionally `configurationSet`. With no credential references
it uses the AWS default chain. Otherwise both username and secret references are
required and resolve to the access key and secret key. SES sends raw MIME with explicit
envelope destinations, keeping BCC recipients out of the raw message headers. SDK
retries are disabled so outbox attempts own retry accounting; an API call is bounded
at 40 seconds.
Credential fields contain **environment variable names**, matching
`^[A-Z_][A-Z0-9_]{0,99}$`. The effective allowlist is `allowedEnvNames` plus these defaults
for the selected kind:
| Kind | Default allowed names |
|---|---|
| SMTP / GMAIL_SMTP | `EMAIL_SMTP_USERNAME`, `EMAIL_SMTP_PASSWORD`, `EMAIL_GMAIL_USERNAME`, `EMAIL_GMAIL_APP_PASSWORD`, `GMAIL_USERNAME`, `GMAIL_APP_PASSWORD` |
| SES | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` |
| POSTMARK | `POSTMARK_SERVER_TOKEN`, `POSTMARK_WEBHOOK_PASSWORD`; legacy `EMAIL_POSTMARK_SERVER_TOKEN`, `EMAIL_POSTMARK_WEBHOOK_PASSWORD` |
| SENDGRID | `SENDGRID_API_KEY`, `SENDGRID_WEBHOOK_PUBLIC_KEY`; legacy `EMAIL_SENDGRID_API_KEY`, `EMAIL_SENDGRID_WEBHOOK_PUBLIC_KEY` |
| MAILGUN | `MAILGUN_API_KEY`, `MAILGUN_WEBHOOK_SIGNING_KEY`; legacy `EMAIL_MAILGUN_API_KEY`, `EMAIL_MAILGUN_SIGNING_KEY` |
| GMAIL_API | `EMAIL_GMAIL_CLIENT_ID`, `EMAIL_GMAIL_CLIENT_SECRET`, `EMAIL_GMAIL_REFRESH_TOKEN` |
Allowlisting does not automatically select a name. Empty SMTP references mean no auth;
empty SES references mean the default chain. Explicit SES references use a static
access/secret pair; temporary session credentials belong in the default AWS chain.
`withSecretResolver(Function)` defaults to `System::getenv`. Values are
resolved during an attempt and never retained, audited, returned or logged. The
provider record's display-only fields return only `Set` / `Not set`; checking presence
necessarily invokes the resolver too. Enqueue and internal queries do not resolve it.
POSTMARK, SENDGRID and MAILGUN require a secret environment name. Mailgun also
requires a sending domain and US/EU region. Gmail SMTP requires both credential
names and STARTTLS. GMAIL_API is refused with
`Gmail API sending is specified but not built; use GMAIL_SMTP`.
HTTP transports use fixed HTTPS endpoints, 10-second connect and 30-second request
timeouts, and no redirects. HTTP 429/5xx retry with optional Retry-After; account
and credential failures pause the provider. Tracking is explicitly controlled by
`trackOpens` / `trackClicks`, off by default. The producer registers the phase-(c)
receiver idempotently when the provided Javalin middleware is present.
## Health checks
The producer registers these checks with QQQ central health monitoring:
| Name | Kind | Interval |
|---|---|---|
| `email.provider.` (each enabled provider) | DEPENDENCY | 300 seconds |
| `email.readiness` | DEPENDENCY | 300 seconds |
Both use component `email`, category `EMAIL`, and a 5-second timeout. Provider
checks read passive delivery state only: HEALTHY maps to UP, UNHEALTHY to DOWN,
and missing/disabled/unknown state to UNKNOWN. The transport SPI has no
non-sending connectivity method; these checks never contact providers or send
mail. Readiness reuses `EmailApi.readiness` for every `EmailApiCategory`; any
unmet prerequisite is DEGRADED, with only enum names in `failedChecks`. Existing
passive state and `email.provider.unhealthy` alerts remain in place.
The Email home uses a standard status widget reading the central `health_component`
rollup, with a link to System Health's checks filtered to `email`. It no longer
lists provider diagnostics. The host must install central health metadata and its
schema; central reads use the viewer's health permissions.
This integration uses the core health API released in QQQ `4.1.0-RC.8`.
Verify offline with `./mvnw -o verify` (`mvnw.cmd` on Windows).
## Sending from another qbit
There are two supported routes. Both queue through the durable outbox; a successful
call acknowledges enqueueing, not delivery.
**1. QQQ core only, for simple subject/body email.** The host installs qbit-email;
the consuming qbit needs no qbit-email dependency or imports. Use
`SendMessageAction` with `messagingProviderName` set to `qbitEmail` (the default
bridge name; hosts can configure another name):
```java
import com.kingsrook.qqq.backend.core.actions.messaging.SendMessageAction;
import com.kingsrook.qqq.backend.core.model.actions.messaging.Content;
import com.kingsrook.qqq.backend.core.model.actions.messaging.Party;
import com.kingsrook.qqq.backend.core.model.actions.messaging.SendMessageInput;
import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailContentRole;
new SendMessageAction().execute(new SendMessageInput()
.withMessagingProviderName("qbitEmail")
.withTo(new Party().withAddress("pat@example.com"))
.withSubject("Ready")
.withContent(new Content().withContentRole(EmailContentRole.TEXT).withBody("Your export is ready.")));
```
**2. The published `com.kingsrook.qbits.email.api` package**, for templates,
idempotency, categories, linked records, sender selection and delivery lookups.
Consumers with a qbit-email dependency import only this package:
```java
import com.kingsrook.qbits.email.api.EmailApi;
import com.kingsrook.qbits.email.api.EmailApiRequest;
import com.kingsrook.qbits.email.api.EmailApiResult;
EmailApiResult result = EmailApi.send(new EmailApiRequest()
.withTo("pat@example.com")
.withTemplateName("order_ready")
.withVariable("order_number", "A-1")
.withVariable("customer_name", "Pat")
.withIdempotencyKey("orders:42:ready")
.withLinkedRecord("orders", 42)
.withSender("orders", "ready"));
```
`EmailApi.ensureTemplate(EmailApiTemplate)` creates a template once; see
[Templates from another qbit](#templates-from-another-qbit).
For durable job effects, use `EmailApi.sendOnce(request, effectKey)`. It assigns
the effect key as the request's idempotency key and returns `EmailApiEffectResult`
with `EmailApiEffectOutcome.QUEUED_OR_DUPLICATE` or `REFUSED` and a nullable
`EmailApiMessage`. Success means the outbox has committed; duplicate handoffs return
the same row and RFC Message-ID. Null requests, null/blank/over-500-character keys,
conflicting request keys and caller transactions are refused. Suppressed rows remain
refused on replay and include their audit metadata; refusals without a row have no
message. Database failures throw `QException` for retry or reconciliation. Ordinary
`EmailApi.send` accepts `request.withTransaction(transaction, backendName)` when a
caller-owned or cross-backend deferred enqueue is needed. See [durable job effects](docs/DURABLE-JOBS.md).
`EmailApi.isInstalled(instance)` detects whether the qbit was produced.
`defaultProviderName(instance)` resolves an enabled configured/default provider;
`listSenderAddresses(providerName)` lists enabled sender rows (null selects the
default). `readiness(instance, category)` reports configuration prerequisites,
including MARKETING's public URL, unsubscribe key and default brand postal address.
For a PHI-approved template, use `readiness(instance, category, true)` to also check
the body key and provider PHI/tracking policy. PHI is a template approval state,
not a category. Readiness is a preflight, not proof that a particular send will pass.
`getMessage(id)`, `getMessageByIdempotencyKey(key)`, `getMessageByRfcMessageId(rfcMessageId)`
and `findMessagesByIdempotencyKeyPrefix(prefix, limit)` expose only
id, delivery status, category, idempotency key, RFC Message-ID, provider message id, template name,
linked table/id and create/sent/delivered timestamps. They never return addresses,
subjects, bodies or variable snapshots. Prefixes are literal and nonblank; limits
must be positive; results are ordered by ascending message id. Missing messages
return an empty Optional/list. Exact-key lookups also return empty for null/blank
values; RFC lookup takes the stored bracketed header, not a provider response id.
Ordinary enqueue results have `QUEUED`, `DUPLICATE`,
`SUPPRESSED` or `REFUSED`; delivery status is a separate string on `EmailApiMessage`.
Database calls require the caller's initialized `QContext`; methods accepting an
installed instance require it to be the current context instance. Lookups run as
the email system, so callers must authorize any exposure to their own users.
Sending retains the existing permission, schema, PHI, suppression and recipient
rules. Database failures propagate as core `QException`; invalid template definitions
and lookup arguments throw `IllegalArgumentException`. No secret values are exposed.
The API uses only JDK, QQQ core and its own API types in published signatures.
Changes within a `0.x` minor release line are additive only. Internal packages
(including runtime, configuration, outbox, metadata, support and template classes)
are not API and may change. `EmailEffects`, `EmailEffectOutcome` and `EmailMessageRef`
are deprecated internals; `EmailApi` is the sole public qbit-email surface.
The internal service details below describe host and
qbit-email implementation behavior, not additional cross-qbit contracts.
## Internal sending service
```java
EmailSendResult result = EmailService.send(new EmailRequest()
.withTo("Pat ")
.withCc("copy@example.com")
.withSubject("Your export is ready")
.withMarkdownBody("[Open your export](https://app.example.com/exports/42)")
.withCategory(EmailCategory.NOTIFICATION)
.withIdempotencyKey("export:42")
.withLinkedRecord("export", 42)
.withSender("exports", "complete"));
```
Use `withHtmlBody` or `withTextBody` instead of Markdown, with optional explicit text
alongside HTML. `withFrom`, `withReplyTo`, `withBcc`, `withProviderName`, `withSendAfter`,
and `withLocale` are supported. `sendingDomain`, when present, must match the sender's
domain. Addresses are parsed strictly; header newlines are refused. Markdown raw HTML
and unsafe URL schemes are disabled. Templates always use the validated template renderer. The ad-hoc renderer extension
remains compatible. Both alternatives are stored at enqueue time and reused on retries.
Results expose `messageId()`, `status()` (`QUEUED`, `DUPLICATE`, `SUPPRESSED`, `REFUSED`)
and `reasons()`. Idempotency is global within the outbox. A repeated key returns the
original id; unique constraints protect races. Idempotent inserts briefly lock the
oldest provider row so concurrent service calls do not abort a caller-owned SQL
transaction with a duplicate-key error.
`withTransaction(transaction, backendName)` joins a matching backend transaction.
The one-argument overload assumes the email backend. A different backend registers
an after-commit insert, discarded on rollback. Its immediate result has a null id
and a reason explaining that enqueue is deferred: an auto-generated id does not
exist before that callback runs. Cross-backend callbacks have the same crash window
as qbit-webhooks' fallback; use the same backend when atomic enqueue is required.
Without a caller transaction, the service owns and commits its transaction.
For multi-tenant hosts, configure `securityFields` and `recordSecurityLocks`, add the
matching columns to the DDL, and provide the request's `securityValues`. Security
values propagate to attempts and resends.
Suppression compares canonical lowercase mailboxes. ALL applies across categories,
except UNSUBSCRIBE cannot block TRANSACTIONAL or SECURITY. Category suppressions apply
to NOTIFICATION and MARKETING. Expired rows are ignored. Partial suppression drops
only matching recipients and records them in `suppressedAddresses`; all-suppressed
messages remain visible with SUPPRESSED status and never consume an attempt.
## Templates, variables and branding
Create an `email_template`, then use **New Version** to copy the active or latest
version into a DRAFT. Edit its source, schema and synthetic sample values. **Activate
Version** validates syntax, samples, classifications, brand contrast and optional
MJML compilation, then retires the previous active version for that locale in one
transaction. ACTIVE and RETIRED versions are immutable. Template-specific
`editPermissionName` applies to CRUD and actions, in addition to the email permission;
even email administrators need that explicit grant. Trusted system writes may bypass it.
```java
EmailService.send(new EmailRequest()
.withTemplateName("order_ready")
.withTo("pat@example.com")
.withRecordVariable("order", "order", 123)
.withVariable("count", 3)
.withLocale("en"));
```
A schema is a JSON list, for example:
```json
[
{"name":"order","label":"Order","type":"RECORD","required":true},
{"name":"count","label":"Items","type":"NUMBER","required":true},
{"name":"link","type":"URL","classification":"PUBLIC"}
]
```
Types are STRING, NUMBER, DATE, DATE_TIME, BOOLEAN, URL, LIST and RECORD. Optional
`table`/`field` bind a variable's classification to metadata. URL values must be
absolute HTTP(S) URLs. Use `{{name}}`, `{{order.number}}`, `{{name|fallback text}}`,
`{{#if count}}...{{/if}}`, and `{{#each items}}{{@index}}: {{this.name}}{{/each}}`.
`this` also supports scalar list elements. Empty/null/false/zero values are false.
Unknown variables, unsupported helpers, unclosed expressions and mismatched blocks
produce line-numbered errors. Records and lists allow dynamic child paths; declare
specific dotted entries when child types/requiredness matter. There is no raw output.
HTML values are escaped; Markdown values cannot inject Markdown syntax. Subject and
text contexts use plain values, with subject newlines refused.
Built-ins are `brand.*` (snake_case names such as `label`, `site_url`, `primary_color`),
`message.id` (the actual outbox primary key), `now` (UTC), and `unsubscribe_url`.
The latter is empty for TRANSACTIONAL/SECURITY and when no notification opt-out key
is configured. The requested locale resolves to an ACTIVE version, falling back to
`en`. The template supplies the category unless the request overrides it.
Brand precedence is request name, template brand, the configured app's brand,
default brand, then the built-in neutral brand. Markdown uses CommonMark tables and
autolinks; raw HTML is escaped. `[Label](https://example.com){.button}` creates a table
button. The 600px shell includes preheader, logo variants, footer, social links and
light/dark colors; marketing requires a postal address. Brand saves and activation
require 4.5:1 text and button contrast in both modes. Optional `darkPrimaryColor` and
`darkButtonTextColor` fall back to their light counterparts. Colors use `#RRGGBB`.
HTML versions retain authored HTML, with escaped variables and an opt-out footer
before `