diff --git a/AGENTS.md b/AGENTS.md index 3b749e9..26e29c6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,15 @@ Design: `docs/DESIGN.md`. ## Architecture - **Entry point:** `EmailQBitProducer implements QBitMetaDataProducer`, package `com.kingsrook.qbits.email`. -- Packages are added per phase as `docs/DESIGN.md` describes. +- `metadata`: seven tables, enum sources, relations and provider/suppression validation. +- `outbox`: EmailService, requests/results and the replaceable ad-hoc renderer. +- `delivery`: durable dispatcher, attempt leases, rate limits, health and optional alerts. +- `transport`: shared MIME, SMTP/SES transports, settings and registry. +- `bridge`: QBIT_EMAIL adapter for core SendMessageAction. +- `processes`: Review/Result record actions, backend send API and scheduled jobs. +- `permissions`, `app`, `support`: custom permission checker, app/navigation and authorized database operations. +- `EmailRuntime` resolves config from the current instance; `EmailIntegrations` declares ESB, API and Quick Search tables. +- Templates/branding/PHI/full home rendering are part 2; their tables exist now. ## Invariants diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ff8586..40b8461 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,4 +6,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and ## [Unreleased] ### Added +- Phase (b) Postmark, SendGrid and Mailgun HTTP transports with fixed HTTPS endpoints, bounded requests, tracking options, safe error classification and Retry-After support; no new dependencies or filesystem storage. +- Strict Gmail SMTP credential/STARTTLS validation, provider-specific credential and webhook environment defaults, and the [provider setup guide](docs/PROVIDERS.md). +- HTTP request-contract tests and local-server end-to-end outbox tests covering success, provider failures, permanent rejection, retries and secret-safe stored errors. +- Phase (a), part 1 email tables, PostgreSQL DDL, enum sources, child relations, permissions and Email app navigation. +- Provider validation with environment-name allowlists and presence-only credential displays. +- Transactional ad-hoc EmailService, idempotency, suppression and a replaceable Markdown/HTML/text renderer. +- Durable dispatcher with unique attempt claims, leases, retry/dead-letter handling, shared provider rate limits, health probes and optional alerts. +- SMTP with authentication/TLS/timeouts, SES v2 raw MIME, and a Gmail SMTP preset. +- QBIT_EMAIL core messaging bridge, Review/Result actions, backend send process, dispatch/maintenance/retention schedules and host integration declarations. +- H2 concurrency and transaction tests, GreenMail end-to-end tests, fake SES tests, metadata/permission checks and process-runner coverage. - Repository scaffold to the QRun qbit repository standard (Apache-2.0, CircleCI qqq-orb 0.6.8 with the `configure-package-install` pre-step, GA-only `publish_release`, the `release-rc` caller workflow). diff --git a/CLAUDE.md b/CLAUDE.md index 35c7f8b..116f242 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,15 @@ Design: `docs/DESIGN.md`. ## Architecture - **Entry point:** `EmailQBitProducer implements QBitMetaDataProducer`, package `com.kingsrook.qbits.email`. -- Packages are added per phase as `docs/DESIGN.md` describes. +- `metadata`: seven tables, enum sources, relations and provider/suppression validation. +- `outbox`: EmailService, requests/results and the replaceable ad-hoc renderer. +- `delivery`: durable dispatcher, attempt leases, rate limits, health and optional alerts. +- `transport`: shared MIME, SMTP/SES transports, settings and registry. +- `bridge`: QBIT_EMAIL adapter for core SendMessageAction. +- `processes`: Review/Result record actions, backend send API and scheduled jobs. +- `permissions`, `app`, `support`: custom permission checker, app/navigation and authorized database operations. +- `EmailRuntime` resolves config from the current instance; `EmailIntegrations` declares ESB, API and Quick Search tables. +- Templates/branding/PHI/full home rendering are part 2; their tables exist now. ## Invariants diff --git a/README.md b/README.md index 8aa9933..5d72005 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,24 @@ # QBit Email -One email service for a QQQ system: an outbox with async sending, retries, dead letters and -idempotency keys; providers as data rows (SMTP relay, Amazon SES, Postmark, SendGrid, Mailgun, -Gmail); Markdown and MJML/HTML templates with versions and branding; delivery tracking from -provider webhooks; a suppression list; RFC 8058 one-click unsubscribe; and PHI-safe sending. -Inbound mail is phase 2. +Phases (a) and (b): a durable email outbox for QQQ, with SMTP, Amazon SES, Postmark, +SendGrid, Mailgun and Gmail SMTP transports, +suppression, retries, provider health, record actions, and the core messaging bridge. +Every send is queued in `email_message` before a dispatcher contacts a provider. -Design: [`docs/DESIGN.md`](docs/DESIGN.md). - -## Requirements - -- Java 21 -- QQQ 4.1 (`qqq-bom-pom` 4.1.0-RC.4 or later) -- An RDBMS backend for the `email_` tables +The approved [design](docs/DESIGN.md) covers later phases too. This implementation +accepts ad-hoc Markdown, HTML and text. Template rendering, branding, PHI handling and +the full home dashboard belong to part 2. Template and variable-bearing requests are +refused in part 1; attachments are refused. Tracking webhooks and unsubscribe remain +later phases. ## Install +Requires Java 21, QQQ **4.1.0-RC.5**, and a transactional RDBMS with the supplied unique +constraints. Apply [the PostgreSQL DDL](docs/ddl/email-postgres.sql) before producing +the qbit. SQL timestamps store UTC values without a timezone, matching QQQ RC.5. +The physical table names equal their metadata names; `backendName` selects the host's +QQQ database backend, as it does in qbit-webhooks. + ```xml com.kingsrook.qbits @@ -26,27 +29,253 @@ Design: [`docs/DESIGN.md`](docs/DESIGN.md). ```java new EmailQBitProducer() - .withQBitConfig(new EmailQBitConfig().withBackendName("rdbms")) + .withQBitConfig(new EmailQBitConfig() + .withBackendName("rdbms") + .withSchedulerName("scheduler")) .produce(qInstance); ``` -## Tables, permissions and configuration +The scheduler must already exist. Without a scheduler, callers can enqueue and run +`EmailDispatcher` or Send Now manually; automatic dispatch is not installed. +Configuration is looked up from the current `QInstance` on every call, never cached +in a static variable. -See [`docs/DESIGN.md`](docs/DESIGN.md). Tables are prefixed `email_`; permissions use the full -qbit name (`email.view`, `email.send`, `email.manage`, `email.admin`). +## Tables and permissions -## Adopting in another app +| Table | Navigation | Writes | +|---|---|---| +| `email_message` | Messages | Qbit only; people use record actions | +| `email_message_attempt` | Attempts on a message | Qbit only | +| `email_template` | Templates | `email.manage`; rendering arrives in part 2 | +| `email_template_version` | Versions on a template | `email.manage`; activation arrives in part 2 | +| `email_brand` | Brands | `email.manage`; rendering arrives in part 2 | +| `email_suppression` | Suppressions | `email.manage`; changing/removing a complaint needs `email.admin` | +| `email_provider` | Providers, with child Messages | `email.admin`, including reads and test sends | -Add the dependency, produce the qbit against your RDBMS backend, add a provider row, and send -through `EmailService` (or the `email_send_templated` process). Code that already calls core's -`SendEmailAction` can route through the outbox with the messaging bridge (DESIGN section 2). +`email.view` reads messages, templates, versions, brands and suppressions. +`email.send` permits the Java service, bridge and backend send process. Reading +messages requires `email.view`, `email.manage` or `email.admin`; send alone grants no read access. +`email.manage` also allows sending, Resend, Cancel and Send Now. `email.admin` +implies all permissions. Grant `email.hasAccess` for the app's derived base access. +Enable the host's `QInstance.enforcePermissionsInCoreActions` for direct Java action +permission enforcement; the tables and processes use the shared custom checker. +Trusted system callers use `SystemWrites.call(...)`, which opens core's authorized +scope and still preserves record security locks. + +All tables and columns follow the `email_`/snake_case database convention. Java field +names are camelCase, matching the webhooks reference. System tables disable direct +insert/update/delete capabilities. Labels contain no status decorations. + +## Providers and credentials + +See the [provider setup guide](docs/PROVIDERS.md) for row fields, DNS/domain setup, +limits, Gmail app passwords and the webhook configuration planned for phase (c). + +| Kind | Transport | Status | +|---|---|---| +| SMTP | Relay or authenticated SMTP, STARTTLS / TLS | Available | +| SES | AWS SES v2, raw MIME | Available | +| POSTMARK | HTTPS JSON, server token and message stream | Available | +| SENDGRID | HTTPS JSON, bearer API key | Available | +| MAILGUN | HTTPS MIME multipart, US / EU | Available | +| GMAIL_SMTP | smtp.gmail.com:587, STARTTLS and app password | Available | +| GMAIL_API | OAuth Gmail API | Deferred; use GMAIL_SMTP | + +Create the first enabled provider with `isDefault=true`. When any providers are +enabled, exactly one must be the enabled default. Switch defaults in a batch update, +clearing the old default first. A nullable unique `default_slot` backs the validation. +Providers with message history cannot be deleted; disable them instead. + +For an unauthenticated relay, use kind `SMTP`, host/port, security `NONE`, and a +`fromAddress`. Leave credential names empty. Authenticated SMTP uses `usernameEnvName` +and `secretEnvName`; security is `NONE`, `STARTTLS` (required) or `TLS`. Timeouts are +10 seconds to connect and 15 seconds each to read and write. The Gmail SMTP preset +uses `smtp.gmail.com:587` with required STARTTLS. + +SES needs a `region`, optionally `configurationSet`. With no credential references +it uses the AWS default chain. Otherwise both username and secret references are +required and resolve to the access key and secret key. SES sends raw MIME with explicit +envelope destinations, keeping BCC recipients out of the raw message headers. SDK +retries are disabled so outbox attempts own retry accounting; an API call is bounded +at 40 seconds. + +Credential fields contain **environment variable names**, matching +`^[A-Z_][A-Z0-9_]{0,99}$`. The effective allowlist is `allowedEnvNames` plus these defaults +for the selected kind: + +| Kind | Default allowed names | +|---|---| +| SMTP / GMAIL_SMTP | `EMAIL_SMTP_USERNAME`, `EMAIL_SMTP_PASSWORD`, `EMAIL_GMAIL_USERNAME`, `EMAIL_GMAIL_APP_PASSWORD`, `GMAIL_USERNAME`, `GMAIL_APP_PASSWORD` | +| SES | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` | +| POSTMARK | `POSTMARK_SERVER_TOKEN`, `POSTMARK_WEBHOOK_PASSWORD`; legacy `EMAIL_POSTMARK_SERVER_TOKEN`, `EMAIL_POSTMARK_WEBHOOK_PASSWORD` | +| SENDGRID | `SENDGRID_API_KEY`, `SENDGRID_WEBHOOK_PUBLIC_KEY`; legacy `EMAIL_SENDGRID_API_KEY`, `EMAIL_SENDGRID_WEBHOOK_PUBLIC_KEY` | +| MAILGUN | `MAILGUN_API_KEY`, `MAILGUN_WEBHOOK_SIGNING_KEY`; legacy `EMAIL_MAILGUN_API_KEY`, `EMAIL_MAILGUN_SIGNING_KEY` | +| GMAIL_API | `EMAIL_GMAIL_CLIENT_ID`, `EMAIL_GMAIL_CLIENT_SECRET`, `EMAIL_GMAIL_REFRESH_TOKEN` | + +Allowlisting does not automatically select a name. Empty SMTP references mean no auth; +empty SES references mean the default chain. Explicit SES references use a static +access/secret pair; temporary session credentials belong in the default AWS chain. +`withSecretResolver(Function)` defaults to `System::getenv`. Values are +resolved during an attempt and never retained, audited, returned or logged. The +provider record's display-only fields return only `Set` / `Not set`; checking presence +necessarily invokes the resolver too. Enqueue and internal queries do not resolve it. + +POSTMARK, SENDGRID and MAILGUN require a secret environment name. Mailgun also +requires a sending domain and US/EU region. Gmail SMTP requires both credential +names and STARTTLS. GMAIL_API is refused with +`Gmail API sending is specified but not built; use GMAIL_SMTP`. + +HTTP transports use fixed HTTPS endpoints, 10-second connect and 30-second request +timeouts, and no redirects. HTTP 429/5xx retry with optional Retry-After; account +and credential failures pause the provider. Tracking is explicitly controlled by +`trackOpens` / `trackClicks`, off by default. Outbound tracking settings do not +install the event webhook receiver; that arrives in phase (c). + +## Sending + +```java +EmailSendResult result = EmailService.send(new EmailRequest() + .withTo("Pat ") + .withCc("copy@example.com") + .withSubject("Your export is ready") + .withMarkdownBody("[Open your export](https://app.example.com/exports/42)") + .withCategory(EmailCategory.NOTIFICATION) + .withIdempotencyKey("export:42") + .withLinkedRecord("export", 42) + .withSender("exports", "complete")); +``` + +Use `withHtmlBody` or `withTextBody` instead of Markdown, with optional explicit text +alongside HTML. `withFrom`, `withReplyTo`, `withBcc`, `withProviderName`, `withSendAfter`, +and `withLocale` are supported. `sendingDomain`, when present, must match the sender's +domain. Addresses are parsed strictly; header newlines are refused. Markdown raw HTML +and unsafe URL schemes are disabled. `EmailRenderer` is the replacement seam for +part 2. Both alternatives are stored at enqueue time and reused on retries. + +Results expose `messageId()`, `status()` (`QUEUED`, `DUPLICATE`, `SUPPRESSED`, `REFUSED`) +and `reasons()`. Idempotency is global within the outbox. A repeated key returns the +original id; unique constraints protect races. Idempotent inserts briefly lock the +oldest provider row so concurrent service calls do not abort a caller-owned SQL +transaction with a duplicate-key error. + +`withTransaction(transaction, backendName)` joins a matching backend transaction. +The one-argument overload assumes the email backend. A different backend registers +an after-commit insert, discarded on rollback. Its immediate result has a null id +and a reason explaining that enqueue is deferred: an auto-generated id does not +exist before that callback runs. Cross-backend callbacks have the same crash window +as qbit-webhooks' fallback; use the same backend when atomic enqueue is required. +Without a caller transaction, the service owns and commits its transaction. + +For multi-tenant hosts, configure `securityFields` and `recordSecurityLocks`, add the +matching columns to the DDL, and provide the request's `securityValues`. Security +values propagate to attempts and resends. + +Suppression compares canonical lowercase mailboxes. ALL applies across categories, +except UNSUBSCRIBE cannot block TRANSACTIONAL or SECURITY. Category suppressions apply +to NOTIFICATION and MARKETING. Expired rows are ignored. Partial suppression drops +only matching recipients and records them in `suppressedAddresses`; all-suppressed +messages remain visible with SUPPRESSED status and never consume an attempt. + +## Delivery and actions + +Each claim inserts a unique `(message_id, attempt_number)` row and commits SENDING +with a lease before network I/O. Provider-row database locks serialize rate checks +and claims across replicas; they are released before contacting the provider. Limits +count attempts in rolling minute and 24-hour windows. Waiting consumes no attempt. +Per-message SMTP 5xx and SES MessageRejected/InvalidParameterValue failures become +DEAD_LETTER, as do invalid MIME addresses or headers; transient failures +use the configured backoff. Explicit retry-after values are capped at one hour; +the configured backoff itself is not capped. + +Expired attempts become LEASE_EXPIRED, then retry or dead-letter at the attempt limit. +Late completion from an expired runner cannot overwrite the newer attempt's outcome. +Delivery is **at least once**: provider acceptance followed by a runner crash can +produce a repeat. Credential, authentication and SES account failures immediately mark +the provider UNHEALTHY and stop its remaining deliveries for that dispatch pass. +These attempts are recorded as PROVIDER_FAILURE and keep messages in RETRY_WAIT with +backoff. They increment the historical attempt count but do not consume the message's +maxAttempts budget (including during lease recovery). Ten consecutive other transient failures mark a provider UNHEALTHY; +one queued message probes it every five minutes, and success restores HEALTHY. +Optional core alerts raise `email.provider.unhealthy` and `email.message.dead-letter`, +with dead-letter deduplication per provider/hour. No alert sink is required. +Alert failures cannot fail an already committed delivery. + +SMTP sign-in requires STARTTLS or TLS. Plaintext authentication is refused at provider +validation and again before connecting, except for loopback (`localhost`, `127.0.0.1`, +`::1`) or an explicit `withAllowPlaintextSmtpAuth(true)` host configuration (default false). +Provider sender/reply-to fields and MIME subjects/addresses cannot contain CR or LF. + +Messages offer standard Review/Result actions: Resend (DEAD_LETTER, BOUNCED, +COMPLAINED, with retained content), Cancel (QUEUED/RETRY_WAIT), and Send Now +(QUEUED/RETRY_WAIT). Resend creates a new linked message and rechecks suppression. +Send Now overrides deferral but preserves rate limits, health and active leases. +Providers offer Send Test Email with a recipient input and a reported outcome. + +`email_send_templated` is a hidden backend API process. Inputs include `templateName`, +`to`, `cc`, `bcc`, `replyTo`, `subject`, `markdownBody`, `htmlBody`, `textBody`, +`providerName`, `brandName`, `category`, `variablesJson`, `linkedTableName`, +`linkedRecordId`, and `idempotencyKey`. Ad-hoc content works now; template/brand/variable +inputs return REFUSED until part 2. Output includes `messageId`, `status`, and `reasons`. + +Hidden schedules are `email_dispatch` (15 s by default), `email_maintain` (300 s, lease +recovery including disabled providers, and health probes), and `email_retention` +(daily). Retention clears final-message bodies after 30 days, preserving `bodySha256`; +message deletion is disabled by default. `finishedDate` records finalization independently +of enqueue and later retention updates. + +## Core messaging bridge + +The producer adds messaging provider `qbitEmail` (configurable), type `QBIT_EMAIL`. +Existing callers switch only `messagingProviderName`: + +```java +new SendMessageAction().execute(new SendMessageInput() + .withMessagingProviderName("qbitEmail") + .withTo(new Party().withAddress("pat@example.com")) + .withSubject("Ready") + .withContent(new Content().withContentRole(EmailContentRole.TEXT).withBody("Your export is ready."))); +``` + +The bridge recursively maps Party/MultiParty TO, CC, BCC, FROM and REPLY_TO roles, plus +TEXT/HTML alternatives. REFUSED throws `QException`; successful calls acknowledge +queueing, not delivery. The built-in EMAIL and SES types are unchanged. + +## Configuration + +All settings have fluent `withX` setters and getters. + +| Setting | Default / purpose | +|---|---| +| `backendName` | Required existing backend | +| `schedulerName` | Optional existing scheduler; required for automatic dispatch | +| `publicBaseUrl` | Reserved for later unsubscribe links | +| `defaultProviderName` | Optional named provider override; otherwise enabled default row | +| `bridgeProviderName` | `qbitEmail` | +| `secretResolver`, `allowedEnvNames` | `System::getenv`, empty additional allowlist | +| `dispatchIntervalSeconds`, `batchSize` | 15, 100 | +| `maxAttempts`, `retryMinutes` | 6; 1, 5, 15, 60, 240, 720 | +| `leaseSeconds` | 120; minimum 60 | +| `bodyRetentionDays`, `messageRetentionDays` | 30, 0 (keep message rows forever) | +| `appName`, `appLabel` | `email`, `Email` | +| `securityFields`, `recordSecurityLocks` | Empty host additions | +| `tableMetaDataCustomizer` | Optional final metadata customization | + +`EmailIntegrations.esbTableNames()`, `apiTableNames()` and `quickSearchTables()` expose +host integration declarations for messages, templates and suppressions. Quick Search +weights exclude subjects, bodies and variable snapshots. Hosts wire their ESB/API/search +modules to these declarations. ## Build ```bash -./mvnw verify # tests, Checkstyle and the JaCoCo gate +./mvnw -o -q verify ``` +Windows can use `./mvnw.cmd -o -q verify`. No new dependencies are required. Tests use +H2, GreenMail, a fake SES client and local HTTP servers, including transaction rollback, concurrent claims, +concurrent idempotency, shared rate budgets, health recovery and process screens. +See [implementation notes](docs/PART1-NOTES.md) for design/API accommodations. + ## License Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE). diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md new file mode 100644 index 0000000..c593dcf --- /dev/null +++ b/docs/PROVIDERS.md @@ -0,0 +1,215 @@ +# Email providers + +Create an enabled `email_provider` row and choose exactly one enabled default. +Provider edits and Send Test Email require `email.admin`. Test sends queue through +the outbox and obey the same rate limits as application messages. + +Field names here are database columns. QQQ record/API names are camelCase: +`secret_env_name` is `secretEnvName`, for example. All kinds use `name`, `label`, +`kind`, `enabled`, `is_default`, `from_address`, optional `from_name` / `reply_to`, +and optional `max_per_minute` / `max_per_day`. When set, `sending_domain` must +match the From address domain exactly. + +## Credentials and delivery + +Credential fields contain environment variable **names**, never values. Names +must match `^[A-Z_][A-Z0-9_]{0,99}$` and the effective allowlist. The defaults +below supplement `EmailQBitConfig.allowedEnvNames`; explicitly allow custom names. +Allowlisting does not automatically select a name for a row. + +| Kind | `username_env_name` | `secret_env_name` | `webhook_secret_env_name` | +|---|---|---|---| +| SMTP | `EMAIL_SMTP_USERNAME` (optional) | `EMAIL_SMTP_PASSWORD` | Not used | +| SES | `AWS_ACCESS_KEY_ID` (optional) | `AWS_SECRET_ACCESS_KEY` (optional pair) | Not used; `sns_topic_arn` is a plain ARN | +| POSTMARK | Not used | `POSTMARK_SERVER_TOKEN` | `POSTMARK_WEBHOOK_PASSWORD` | +| SENDGRID | Not used | `SENDGRID_API_KEY` | `SENDGRID_WEBHOOK_PUBLIC_KEY` | +| MAILGUN | Not used | `MAILGUN_API_KEY` | `MAILGUN_WEBHOOK_SIGNING_KEY` | +| GMAIL_SMTP | `GMAIL_USERNAME` | `GMAIL_APP_PASSWORD` | Not used | + +Legacy `EMAIL_POSTMARK_SERVER_TOKEN`, `EMAIL_POSTMARK_WEBHOOK_PASSWORD`, +`EMAIL_SENDGRID_API_KEY`, `EMAIL_SENDGRID_WEBHOOK_PUBLIC_KEY`, `EMAIL_MAILGUN_API_KEY`, +`EMAIL_MAILGUN_SIGNING_KEY`, `EMAIL_GMAIL_USERNAME` and `EMAIL_GMAIL_APP_PASSWORD` +remain allowlisted for their kinds. SES also allowlists `AWS_SESSION_TOKEN` for +the default credential chain; explicit row credentials are an access/secret pair. + +Values resolve through `secretResolver` (default `System::getenv`) for each send +attempt. Rotation needs no service restart. Provider record presence indicators +also resolve values, returning only Set / Not set; enqueue and internal queries +do not resolve credentials. Values and provider response diagnostics are never +logged, stored in errors, returned in exceptions or cached by these transports. + +HTTP uses fixed HTTPS endpoints, a 10-second connect timeout, a 30-second request +timeout, no redirects and no client retries. The outbox owns retries. MIME and +multipart bodies are built entirely in memory; services use no filesystem storage. + +HTTP 429 and 5xx are transient. Retry-After accepts seconds or an HTTP date; +the dispatcher caps explicit delays at one hour. Authentication/account failures +mark the provider UNHEALTHY, stop its remaining deliveries for the dispatch pass, +and retain messages for retry without consuming their failure budget. Permanent +message errors become DEAD_LETTER. Exceptions use fixed text and numeric codes; +stored errors use fixed messages, never response bodies. + +Acceptance means SENT, not delivered. Delivery is at least once: a crash after +acceptance can cause a duplicate. Limits count attempts across replicas in rolling +minute and 24-hour windows, not recipients or successful deliveries. Leave room +for retries, recipient quotas and other applications using the same account. + +## SMTP + +Set `kind=SMTP`, `host`, `port`, and `security=NONE`, `STARTTLS`, or `TLS`. For +authenticated SMTP, set both credential names; for an unauthenticated relay, +leave both empty. Authentication requires encryption except for loopback or an +explicit host `allowPlaintextSmtpAuth` configuration. + +Configure the relay's allowed sender domains and route. For public delivery, +verify the domain with the operator, publish its DKIM and SPF records, and align +the From domain with DMARC. Update the existing SPF policy rather than adding a +second SPF record. Set limits below the relay's quota. SMTP has 10-second connect +and 15-second read/write timeouts; there is no provider event webhook here. + +## Amazon SES + +Set `kind=SES`, `region`, and a verified `from_address`. Optionally set +`sending_domain`, `configuration_set` and `sns_topic_arn`. Leave credential names +empty for the AWS default chain (workload/instance role or AWS environment); +otherwise supply both access-key and secret-key environment names. + +Verify the sender identity in the sending region and publish SES's DKIM records. +Configure SPF for a custom MAIL FROM domain if used. Request production access +when leaving the sandbox. Configure the configuration set's SNS event destination +and put the topic ARN in `sns_topic_arn`, not a secret field. + +Sending quotas are regional and account-specific, with sandbox, recipient and +rate restrictions. Set both rate fields to fit those quotas. SES sends raw MIME +with explicit envelope destinations and strips Bcc from its MIME headers. + +## Postmark + +Set `kind=POSTMARK`, `secret_env_name=POSTMARK_SERVER_TOKEN`, a verified +`from_address`, and optionally `sending_domain`. Use a server token, not an +account token. `message_stream` defaults to `outbound`; another transactional or +broadcast stream must exist on that server. `track_opens` and `track_clicks` +default to false; click tracking maps to HtmlAndText or None. + +Verify the domain or sender signature, publish Postmark's DKIM and recommended +Return-Path/SPF DNS records, obtain account approval and create the desired +message stream. The endpoint is `https://api.postmarkapp.com/email`; +the returned `MessageID` is retained. + +| Signal | Outcome | +|---|---| +| ErrorCode 10 | Provider failure: bad token | +| ErrorCode 300 / 406 | Permanent: invalid request / inactive recipient | +| ErrorCode 400/401, 405, 412, 413, 422 | Provider failure: signature, sending approval or server/account configuration | +| ErrorCode 1235/1236 / 1480 | Provider failure: stream / IP configuration | +| HTTP 401/403 | Provider failure | +| HTTP 429 / 5xx | Transient; takes precedence over a body code | +| Other request errors | Permanent; malformed successful responses are transient | + +API ErrorCode 413 is an account problem; HTTP 413 is a payload-size error. +Check plan, recipient and payload limits before setting the rate fields. The +single-message API documents a 10 MB message limit. See +[Postmark API errors and limits](https://postmarkapp.com/developer/api/overview). + +## SendGrid + +Set `kind=SENDGRID`, `secret_env_name=SENDGRID_API_KEY`, a verified `from_address` +and optionally `sending_domain`. Create a key with Mail Send permission. +Authenticate the domain and publish SendGrid's DKIM/SPF DNS records; single-sender +verification can be used for initial tests. Confirm the account can send. + +`track_opens` controls open tracking; `track_clicks` controls HTML and plain-text +click tracking. Explicit false values override account tracking defaults. +The global endpoint is `https://api.sendgrid.com/v3/mail/send`; only HTTP 202 +counts as success and `X-Message-Id` becomes the provider message ID. + +One personalization contains To/Cc/Bcc with display names. `custom_args` carries +`qqq_email_id`. Reserved headers are filtered case-insensitively: `x-sg-id`, +`x-sg-eid`, `received`, `dkim-signature`, `Content-Type`, `Content-Transfer-Encoding`, +`To`, `From`, `Subject`, `Reply-To`, `CC`, `BCC`. `List-Unsubscribe` and +`List-Unsubscribe-Post` are retained. + +HTTP 401/403 are provider failures, 400/413 permanent, and 429/5xx transient. +Set rate limits for the account's plan/API quota. Payload and recipient limits +apply separately from outbox attempt limits. See +[SendGrid Mail Send](https://www.twilio.com/docs/sendgrid/api-reference/mail-send/mail-send). + +## Mailgun + +Set `kind=MAILGUN`, `secret_env_name=MAILGUN_API_KEY`, `sending_domain` (required, +a DNS name without a scheme/path), and `region=US` or `EU` (required). +The From address domain must match the sending domain. + +Add and verify the domain in that region. Publish Mailgun's SPF/DKIM records and +its requested tracking DNS records if enabling tracking. Use an API key allowed +to send for that domain. Sandbox domains restrict recipients; check production +status and plan/rate quotas before choosing `max_per_minute` / `max_per_day`. +Those fields do not enforce MIME-size or recipient limits. + +The endpoint is `https://api.mailgun.net/v3/{sending_domain}/messages.mime`, or +`https://api.eu.mailgun.net/v3/{sending_domain}/messages.mime` for EU. Basic auth +uses `api:`. Multipart includes all To/Cc/Bcc mailboxes as envelope +destinations, MIME `message` with Bcc stripped, `o:tracking-opens`, +`o:tracking-clicks` (explicit yes/no from the row), and `v:qqq_email_id`. +The response `id` is stored without surrounding angle brackets. + +HTTP 401/403 are provider failures, 400 permanent, and 429/5xx transient. + +## Gmail SMTP + +| Field | Value | +|---|---| +| `kind` | `GMAIL_SMTP` | +| `username_env_name` | `GMAIL_USERNAME`: value is the full account email address | +| `secret_env_name` | `GMAIL_APP_PASSWORD`: value is the generated app password | +| `security` | `STARTTLS` | +| `from_address` | Account address or a permitted configured alias | +| `max_per_day` | At most about 500 personal / 2000 paid Workspace; lower with headroom | + +The preset always uses `smtp.gmail.com:587` with required STARTTLS, ignoring +custom host/port values. Both env names are required; other security values are +refused even if plaintext auth is enabled for other relays. An omitted security +value uses the preset at the settings layer, but provider forms default to NONE, +so select STARTTLS when creating the row. + +Enable 2-step verification and generate an app password for the service; do not +use the normal account password. Account/admin security policy may disable app +passwords. A Google password change revokes existing app passwords, requiring a +new environment value. See +[Google app password setup](https://support.google.com/accounts/answer/185833). + +Google manages domain authentication for personal gmail.com senders. For +Workspace, verify the domain and configure Google's SPF/DKIM records and DMARC. +Trials, new accounts, SMTP recipient quotas and other account users can reduce +available capacity. The 500/2000 limit is an upper planning bound, not guaranteed +throughput. See [Workspace limits](https://support.google.com/a/answer/166852) +and [personal Gmail limits](https://support.google.com/mail/answer/22839). + +`GMAIL_API` is refused with: +`Gmail API sending is specified but not built; use GMAIL_SMTP`. +It would require a Google Cloud project, OAuth consent, client credentials and +refresh-token management. It adds no needed system-mail capability over the app +password preset in this phase and is built only on request (DESIGN 3.6). + +## Event webhooks: phase (c) + +Event ingestion is **not installed by phase (b)**. After phase (c) is deployed, +configure provider event delivery at: + +`https:///qqq/hooks/email/{kind}/{providerName}` + +Use the configured provider name and provider-kind route defined by phase (c). +Do not expect delivery/bounce/complaint/open/click updates until the receiver +exists. Tracking options can operate on the provider side now. + +| Provider | Event setup to prepare | +|---|---| +| Postmark | Server/stream event webhooks with Basic auth; `POSTMARK_WEBHOOK_PASSWORD` | +| SendGrid | Signed Event Webhook; verification key in `SENDGRID_WEBHOOK_PUBLIC_KEY` | +| Mailgun | Regional domain event webhooks; `MAILGUN_WEBHOOK_SIGNING_KEY`, separate from the sending key | +| SES | Configuration-set events to SNS, HTTP subscription and `sns_topic_arn`; signature/topic verification arrives in phase (c) | +| SMTP / Gmail SMTP | No native provider event webhook in this design | + +Webhook environment names are allowlisted now so rows can be prepared; outbound +sends never resolve them. Verification, durable event processing and unsubscribe +handling arrive in later phases. diff --git a/docs/ddl/email-postgres.sql b/docs/ddl/email-postgres.sql new file mode 100644 index 0000000..12e7909 --- /dev/null +++ b/docs/ddl/email-postgres.sql @@ -0,0 +1,178 @@ +-- Phase (a), part 1. Apply before producing the qbit. Host security columns must be added by the host. + +CREATE TABLE email_provider ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + name VARCHAR(500), + label VARCHAR(500), + kind VARCHAR(500), + enabled BOOLEAN, + is_default BOOLEAN, + from_address VARCHAR(500), + from_name VARCHAR(500), + reply_to VARCHAR(500), + sending_domain VARCHAR(500), + host VARCHAR(500), + port INTEGER, + security VARCHAR(500), + region VARCHAR(500), + username_env_name VARCHAR(500), + secret_env_name VARCHAR(500), + webhook_secret_env_name VARCHAR(500), + sns_topic_arn VARCHAR(500), + configuration_set VARCHAR(500), + message_stream VARCHAR(500), + max_per_minute INTEGER, + max_per_day INTEGER, + track_opens BOOLEAN, + track_clicks BOOLEAN, + phi_allowed BOOLEAN, + health_status VARCHAR(500), + last_success_date TIMESTAMP, + last_failure_date TIMESTAMP, + last_failure_message VARCHAR(500), + dispatch_lock INTEGER, + default_slot INTEGER, + UNIQUE (name), + UNIQUE (default_slot) +); + +CREATE TABLE email_message ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + status VARCHAR(500), + to_addresses TEXT, + cc_addresses TEXT, + bcc_addresses TEXT, + from_address VARCHAR(500), + reply_to VARCHAR(500), + subject TEXT, + template_id INTEGER, + template_version_id INTEGER, + brand_id INTEGER, + category VARCHAR(500), + locale VARCHAR(500), + variables_snapshot TEXT, + html_body TEXT, + text_body TEXT, + body_sha256 VARCHAR(500), + idempotency_key VARCHAR(500), + provider_id INTEGER, + provider_message_id VARCHAR(500), + attempt_count INTEGER, + next_attempt_date TIMESTAMP, + lease_expires_date TIMESTAMP, + last_error VARCHAR(500), + finished_date TIMESTAMP, + sent_date TIMESTAMP, + delivered_date TIMESTAMP, + dead_letter_date TIMESTAMP, + linked_table_name VARCHAR(500), + linked_record_id VARCHAR(500), + sender_qbit VARCHAR(500), + sender_process VARCHAR(500), + suppressed_addresses TEXT, + send_after TIMESTAMP, + UNIQUE (idempotency_key) +); + +CREATE TABLE email_message_attempt ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + message_id INTEGER NOT NULL, + attempt_number INTEGER NOT NULL, + outcome VARCHAR(500), + provider_id INTEGER, + claimed_by VARCHAR(500), + start_date TIMESTAMP, + end_date TIMESTAMP, + lease_expires_date TIMESTAMP, + duration_millis INTEGER, + error VARCHAR(500), + UNIQUE (message_id, attempt_number) +); + +CREATE TABLE email_template ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + name VARCHAR(500), + label VARCHAR(500), + category VARCHAR(500), + kind VARCHAR(500), + brand_id INTEGER, + active_version_id INTEGER, + phi_approved BOOLEAN, + phi_approved_by VARCHAR(500), + phi_approved_date TIMESTAMP, + edit_permission_name VARCHAR(500), + owner_qbit VARCHAR(500), + description TEXT, + UNIQUE (name) +); + +CREATE TABLE email_template_version ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + template_id INTEGER, + version_number INTEGER, + status VARCHAR(500), + locale VARCHAR(500), + subject TEXT, + preheader TEXT, + body_source TEXT, + text_source TEXT, + compiled_html TEXT, + variables_schema TEXT, + sample_variables TEXT, + created_by VARCHAR(500), + activated_date TIMESTAMP, + UNIQUE (template_id, version_number, locale) +); + +CREATE TABLE email_brand ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + name VARCHAR(500), + label VARCHAR(500), + logo_url VARCHAR(500), + logo_dark_url VARCHAR(500), + logo_alt VARCHAR(500), + primary_color VARCHAR(500), + text_color VARCHAR(500), + background_color VARCHAR(500), + dark_background_color VARCHAR(500), + dark_text_color VARCHAR(500), + button_text_color VARCHAR(500), + footer_markdown TEXT, + postal_address TEXT, + social_links TEXT, + site_url VARCHAR(500), + is_default BOOLEAN, + app_name VARCHAR(500), + UNIQUE (name) +); + +CREATE TABLE email_suppression ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + address VARCHAR(500), + scope VARCHAR(500), + reason VARCHAR(500), + source_message_id INTEGER, + source_event_id INTEGER, + note TEXT, + created_date TIMESTAMP, + expires_date TIMESTAMP, + UNIQUE (address, scope) +); + +CREATE INDEX email_message_due ON email_message (status, next_attempt_date); +CREATE INDEX email_attempt_provider_start ON email_message_attempt (provider_id, start_date); +CREATE INDEX email_attempt_provider_end ON email_message_attempt (provider_id, end_date, id); diff --git a/pom.xml b/pom.xml index 452723b..2f79dee 100644 --- a/pom.xml +++ b/pom.xml @@ -27,6 +27,9 @@ 21 0.8.15 2.2.224 + 0.27.1 + 2.41.34 + 2.0.1 @@ -77,14 +80,46 @@ qqq-backend-core + + + org.commonmark + commonmark + ${commonmark.version} + + + org.commonmark + commonmark-ext-gfm-tables + ${commonmark.version} + + + org.commonmark + commonmark-ext-autolink + ${commonmark.version} + + + + + software.amazon.awssdk + sesv2 + ${awssdk.version} + + + + + com.kingsrook.qqq + qqq-middleware-javalin + provided + + com.kingsrook.qqq qqq-backend-module-rdbms test - com.kingsrook.qqq - qqq-middleware-javalin + com.icegreen + greenmail-junit5 + ${greenmail.version} test @@ -195,7 +230,7 @@ true - 4.1.0-RC.4 + 4.1.0-RC.5 diff --git a/src/main/java/com/kingsrook/qbits/email/EmailIntegrations.java b/src/main/java/com/kingsrook/qbits/email/EmailIntegrations.java new file mode 100644 index 0000000..bfc61ba --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/EmailIntegrations.java @@ -0,0 +1,94 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email; + + +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.email.metadata.EmailTables; + + +/******************************************************************************* + ** What a host wires into its ESB, Quick Search and API, without this qbit + ** depending on those modules (the same opt-in hooks as qbit-sql-workbench). + ** + **
    + **
  • {@link #esbTableNames()}: publish these tables' inserts, updates and + ** deletes to an ESB topic (the main tables).
    • + **
  • {@link #quickSearchTables()}: index these in Quick Search, with field + ** weights.
    • + **
  • {@link #apiTableNames()}: expose these in the host's QQQ API. The API + ** gets exactly the admin's rules: the same permissions, customizers and + ** read-only tables.
    • + **
+ *******************************************************************************/ +public final class EmailIntegrations +{ + /******************************************************************************* + ** A table for Quick Search: field weights, the timestamp for incremental + ** indexing, and the record label. + *******************************************************************************/ + public record SearchableTable(String tableName, Map fieldWeights, String basepullTimestamp, String recordLabelFormat, List recordLabelFields) + { + } + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailIntegrations() + { + } + + + + /******************************************************************************* + ** Tables to publish on the ESB. + *******************************************************************************/ + public static List esbTableNames() + { + return (List.of(EmailTables.MESSAGE, EmailTables.TEMPLATE, EmailTables.SUPPRESSION)); + } + + + + /******************************************************************************* + ** Tables for the host's QQQ API. + *******************************************************************************/ + public static List apiTableNames() + { + return (List.of(EmailTables.MESSAGE, EmailTables.TEMPLATE, EmailTables.SUPPRESSION)); + } + + + + /******************************************************************************* + ** Tables for Quick Search. + *******************************************************************************/ + public static List quickSearchTables() + { + return (List.of( + new SearchableTable(EmailTables.MESSAGE, Map.of("id", 10, "providerMessageId", 8, "status", 4, "linkedRecordId", 3), "modifyDate", "%s", List.of("id")), + new SearchableTable(EmailTables.TEMPLATE, Map.of("name", 10, "label", 8, "description", 3), "modifyDate", "%s", List.of("name")), + new SearchableTable(EmailTables.SUPPRESSION, Map.of("address", 10, "scope", 4, "reason", 3), "modifyDate", "%s", List.of("address")))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/EmailQBitConfig.java b/src/main/java/com/kingsrook/qbits/email/EmailQBitConfig.java index 58982ba..0ef8c05 100644 --- a/src/main/java/com/kingsrook/qbits/email/EmailQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/email/EmailQBitConfig.java @@ -21,41 +21,138 @@ package com.kingsrook.qbits.email; +import java.util.HashSet; import java.util.List; +import java.util.Set; +import java.util.function.Function; +import java.util.regex.Pattern; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig; -import com.kingsrook.qqq.backend.core.utils.StringUtils; +import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; /******************************************************************************* - ** Configuration for the email qbit. See docs/DESIGN.md. + ** Configuration for phase (a) email delivery. *******************************************************************************/ public class EmailQBitConfig implements QBitConfig { private String backendName; + private String schedulerName; + private String publicBaseUrl; + private String defaultProviderName; + private String bridgeProviderName = "qbitEmail"; + private Function secretResolver = System::getenv; + private boolean allowPlaintextSmtpAuth = false; + private Set allowedEnvNames = Set.of(); + private Integer dispatchIntervalSeconds = 15; + private Integer batchSize = 100; + private Integer maxAttempts = 6; + private List retryMinutes = List.of(1, 5, 15, 60, 240, 720); + private Integer leaseSeconds = 120; + private Integer bodyRetentionDays = 30; + private Integer messageRetentionDays = 0; + private String appName = "email"; + private String appLabel = "Email"; + private List securityFields = List.of(); + private List recordSecurityLocks = List.of(); + private MetaDataCustomizerInterface tableMetaDataCustomizer; - + public static final Pattern ENV_NAME = Pattern.compile("^[A-Z_][A-Z0-9_]{0,99}$"); /******************************************************************************* - ** Validate the configuration against the instance. + ** Whether the host explicitly permits SMTP credentials without TLS. + *******************************************************************************/ + public boolean getAllowPlaintextSmtpAuth() + { + return (allowPlaintextSmtpAuth); + } + + /******************************************************************************* + ** Opt in to plaintext SMTP authentication outside loopback. + *******************************************************************************/ + public EmailQBitConfig withAllowPlaintextSmtpAuth(boolean allowPlaintextSmtpAuth) + { + this.allowPlaintextSmtpAuth = allowPlaintextSmtpAuth; + return (this); + } + + /******************************************************************************* + ** Validate against the host instance. *******************************************************************************/ @Override - public void validate(QInstance qInstance, List errors) + public void validate(QInstance instance, List errors) { - if(!StringUtils.hasContent(backendName)) + if(backendName == null || backendName.isBlank()) { - errors.add("EmailQBitConfig: backendName is required"); + errors.add("backendName is required"); } - else if(qInstance.getBackend(backendName) == null) + else if(instance.getBackend(backendName) == null) { - errors.add("EmailQBitConfig: backend not found: " + backendName); + errors.add("backend not found: " + backendName); + } + if(schedulerName != null && instance.getScheduler(schedulerName) == null) + { + errors.add("schedulerName is not a scheduler in the instance"); + } + if(maxAttempts == null || maxAttempts < 1 || batchSize == null || batchSize < 1 || dispatchIntervalSeconds == null || dispatchIntervalSeconds < 1) + { + errors.add("Attempts, batch size and dispatch interval must be positive"); + } + if(leaseSeconds == null || leaseSeconds < 60) + { + errors.add("leaseSeconds must be at least 60, longer than transport timeouts"); + } + if(retryMinutes == null || retryMinutes.isEmpty() || retryMinutes.stream().anyMatch(n -> n == null || n < 1)) + { + errors.add("retryMinutes must contain positive delays"); + } + if(bodyRetentionDays == null || bodyRetentionDays < 0 || messageRetentionDays == null || messageRetentionDays < 0) + { + errors.add("Retention days must not be negative"); + } + if(secretResolver == null || allowedEnvNames == null || allowedEnvNames.stream().anyMatch(n -> n == null || !ENV_NAME.matcher(n).matches())) + { + errors.add("A secret resolver and valid allowed environment names are required"); + } + if(bridgeProviderName == null || bridgeProviderName.isBlank() || appName == null || appName.isBlank()) + { + errors.add("Bridge provider and app names are required"); } } + /******************************************************************************* + ** Retry minutes after a one-based attempt, repeating the final delay. + *******************************************************************************/ + public int retryMinutesAfterAttempt(int number) + { + return (retryMinutes.get(Math.max(0, Math.min(number - 1, retryMinutes.size() - 1)))); + } + + /******************************************************************************* + ** Allowed names include the documented defaults for this provider kind. + *******************************************************************************/ + public Set effectiveAllowedEnvNames(String kind) + { + Set names = new HashSet<>(allowedEnvNames); + names.addAll(switch(kind == null ? "" : kind) + { + case "SMTP", "GMAIL_SMTP" -> Set.of("EMAIL_SMTP_USERNAME", "EMAIL_SMTP_PASSWORD", "EMAIL_GMAIL_USERNAME", "EMAIL_GMAIL_APP_PASSWORD", "GMAIL_USERNAME", "GMAIL_APP_PASSWORD"); + case "SES" -> Set.of("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"); + case "POSTMARK" -> Set.of("EMAIL_POSTMARK_SERVER_TOKEN", "EMAIL_POSTMARK_WEBHOOK_PASSWORD", "POSTMARK_SERVER_TOKEN", "POSTMARK_WEBHOOK_PASSWORD"); + case "SENDGRID" -> Set.of("EMAIL_SENDGRID_API_KEY", "EMAIL_SENDGRID_WEBHOOK_PUBLIC_KEY", "SENDGRID_API_KEY", "SENDGRID_WEBHOOK_PUBLIC_KEY"); + case "MAILGUN" -> Set.of("EMAIL_MAILGUN_API_KEY", "EMAIL_MAILGUN_SIGNING_KEY", "MAILGUN_API_KEY", "MAILGUN_WEBHOOK_SIGNING_KEY"); + case "GMAIL_API" -> Set.of("EMAIL_GMAIL_CLIENT_ID", "EMAIL_GMAIL_CLIENT_SECRET", "EMAIL_GMAIL_REFRESH_TOKEN"); + default -> Set.of(); + }); + return (names); + } /******************************************************************************* - ** Getter for backendName + ** Getter for backendName. *******************************************************************************/ public String getBackendName() { @@ -63,13 +160,354 @@ public class EmailQBitConfig implements QBitConfig } - /******************************************************************************* - ** Fluent setter for backendName + ** Fluent setter for backendName. *******************************************************************************/ public EmailQBitConfig withBackendName(String backendName) { this.backendName = backendName; return (this); } + + + /******************************************************************************* + ** Getter for schedulerName. + *******************************************************************************/ + public String getSchedulerName() + { + return (schedulerName); + } + + + /******************************************************************************* + ** Fluent setter for schedulerName. + *******************************************************************************/ + public EmailQBitConfig withSchedulerName(String schedulerName) + { + this.schedulerName = schedulerName; + return (this); + } + + + /******************************************************************************* + ** Getter for publicBaseUrl. + *******************************************************************************/ + public String getPublicBaseUrl() + { + return (publicBaseUrl); + } + + + /******************************************************************************* + ** Fluent setter for publicBaseUrl. + *******************************************************************************/ + public EmailQBitConfig withPublicBaseUrl(String publicBaseUrl) + { + this.publicBaseUrl = publicBaseUrl; + return (this); + } + + + /******************************************************************************* + ** Getter for defaultProviderName. + *******************************************************************************/ + public String getDefaultProviderName() + { + return (defaultProviderName); + } + + + /******************************************************************************* + ** Fluent setter for defaultProviderName. + *******************************************************************************/ + public EmailQBitConfig withDefaultProviderName(String defaultProviderName) + { + this.defaultProviderName = defaultProviderName; + return (this); + } + + + /******************************************************************************* + ** Getter for bridgeProviderName. + *******************************************************************************/ + public String getBridgeProviderName() + { + return (bridgeProviderName); + } + + + /******************************************************************************* + ** Fluent setter for bridgeProviderName. + *******************************************************************************/ + public EmailQBitConfig withBridgeProviderName(String bridgeProviderName) + { + this.bridgeProviderName = bridgeProviderName; + return (this); + } + + + /******************************************************************************* + ** Getter for secretResolver. + *******************************************************************************/ + public Function getSecretResolver() + { + return (secretResolver); + } + + + /******************************************************************************* + ** Fluent setter for secretResolver. + *******************************************************************************/ + public EmailQBitConfig withSecretResolver(Function secretResolver) + { + this.secretResolver = secretResolver; + return (this); + } + + + /******************************************************************************* + ** Getter for allowedEnvNames. + *******************************************************************************/ + public Set getAllowedEnvNames() + { + return (allowedEnvNames); + } + + + /******************************************************************************* + ** Fluent setter for allowedEnvNames. + *******************************************************************************/ + public EmailQBitConfig withAllowedEnvNames(Set allowedEnvNames) + { + this.allowedEnvNames = allowedEnvNames; + return (this); + } + + + /******************************************************************************* + ** Getter for dispatchIntervalSeconds. + *******************************************************************************/ + public Integer getDispatchIntervalSeconds() + { + return (dispatchIntervalSeconds); + } + + + /******************************************************************************* + ** Fluent setter for dispatchIntervalSeconds. + *******************************************************************************/ + public EmailQBitConfig withDispatchIntervalSeconds(Integer dispatchIntervalSeconds) + { + this.dispatchIntervalSeconds = dispatchIntervalSeconds; + return (this); + } + + + /******************************************************************************* + ** Getter for batchSize. + *******************************************************************************/ + public Integer getBatchSize() + { + return (batchSize); + } + + + /******************************************************************************* + ** Fluent setter for batchSize. + *******************************************************************************/ + public EmailQBitConfig withBatchSize(Integer batchSize) + { + this.batchSize = batchSize; + return (this); + } + + + /******************************************************************************* + ** Getter for maxAttempts. + *******************************************************************************/ + public Integer getMaxAttempts() + { + return (maxAttempts); + } + + + /******************************************************************************* + ** Fluent setter for maxAttempts. + *******************************************************************************/ + public EmailQBitConfig withMaxAttempts(Integer maxAttempts) + { + this.maxAttempts = maxAttempts; + return (this); + } + + + /******************************************************************************* + ** Getter for retryMinutes. + *******************************************************************************/ + public List getRetryMinutes() + { + return (retryMinutes); + } + + + /******************************************************************************* + ** Fluent setter for retryMinutes. + *******************************************************************************/ + public EmailQBitConfig withRetryMinutes(List retryMinutes) + { + this.retryMinutes = retryMinutes; + return (this); + } + + + /******************************************************************************* + ** Getter for leaseSeconds. + *******************************************************************************/ + public Integer getLeaseSeconds() + { + return (leaseSeconds); + } + + + /******************************************************************************* + ** Fluent setter for leaseSeconds. + *******************************************************************************/ + public EmailQBitConfig withLeaseSeconds(Integer leaseSeconds) + { + this.leaseSeconds = leaseSeconds; + return (this); + } + + + /******************************************************************************* + ** Getter for bodyRetentionDays. + *******************************************************************************/ + public Integer getBodyRetentionDays() + { + return (bodyRetentionDays); + } + + + /******************************************************************************* + ** Fluent setter for bodyRetentionDays. + *******************************************************************************/ + public EmailQBitConfig withBodyRetentionDays(Integer bodyRetentionDays) + { + this.bodyRetentionDays = bodyRetentionDays; + return (this); + } + + + /******************************************************************************* + ** Getter for messageRetentionDays. + *******************************************************************************/ + public Integer getMessageRetentionDays() + { + return (messageRetentionDays); + } + + + /******************************************************************************* + ** Fluent setter for messageRetentionDays. + *******************************************************************************/ + public EmailQBitConfig withMessageRetentionDays(Integer messageRetentionDays) + { + this.messageRetentionDays = messageRetentionDays; + return (this); + } + + + /******************************************************************************* + ** Getter for appName. + *******************************************************************************/ + public String getAppName() + { + return (appName); + } + + + /******************************************************************************* + ** Fluent setter for appName. + *******************************************************************************/ + public EmailQBitConfig withAppName(String appName) + { + this.appName = appName; + return (this); + } + + + /******************************************************************************* + ** Getter for appLabel. + *******************************************************************************/ + public String getAppLabel() + { + return (appLabel); + } + + + /******************************************************************************* + ** Fluent setter for appLabel. + *******************************************************************************/ + public EmailQBitConfig withAppLabel(String appLabel) + { + this.appLabel = appLabel; + return (this); + } + + + /******************************************************************************* + ** Getter for securityFields. + *******************************************************************************/ + public List getSecurityFields() + { + return (securityFields); + } + + + /******************************************************************************* + ** Fluent setter for securityFields. + *******************************************************************************/ + public EmailQBitConfig withSecurityFields(List securityFields) + { + this.securityFields = securityFields; + return (this); + } + + + /******************************************************************************* + ** Getter for recordSecurityLocks. + *******************************************************************************/ + public List getRecordSecurityLocks() + { + return (recordSecurityLocks); + } + + + /******************************************************************************* + ** Fluent setter for recordSecurityLocks. + *******************************************************************************/ + public EmailQBitConfig withRecordSecurityLocks(List recordSecurityLocks) + { + this.recordSecurityLocks = recordSecurityLocks; + return (this); + } + + + /******************************************************************************* + ** Getter for tableMetaDataCustomizer. + *******************************************************************************/ + public MetaDataCustomizerInterface getTableMetaDataCustomizer() + { + return (tableMetaDataCustomizer); + } + + + /******************************************************************************* + ** Fluent setter for tableMetaDataCustomizer. + *******************************************************************************/ + public EmailQBitConfig withTableMetaDataCustomizer(MetaDataCustomizerInterface tableMetaDataCustomizer) + { + this.tableMetaDataCustomizer = tableMetaDataCustomizer; + return (this); + } } diff --git a/src/main/java/com/kingsrook/qbits/email/EmailQBitProducer.java b/src/main/java/com/kingsrook/qbits/email/EmailQBitProducer.java index e51eb72..bdd787b 100644 --- a/src/main/java/com/kingsrook/qbits/email/EmailQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/email/EmailQBitProducer.java @@ -22,17 +22,30 @@ package com.kingsrook.qbits.email; import java.util.ArrayList; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.email.app.EmailApp; +import com.kingsrook.qbits.email.bridge.EmailOutboxMessagingProviderMetaData; +import com.kingsrook.qbits.email.delivery.EmailAlerts; +import com.kingsrook.qbits.email.metadata.EmailTableCustomizers; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.processes.EmailProcesses; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSectionAlternativeType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; /******************************************************************************* - ** Entry point of the email qbit (scaffold: the tables, providers and app - ** arrive in phase (a), see docs/DESIGN.md). + ** Entry point of the email outbox qbit. *******************************************************************************/ public class EmailQBitProducer implements QBitMetaDataProducer { @@ -112,8 +125,7 @@ public class EmailQBitProducer implements QBitMetaDataProducer /******************************************************************************* - ** Validate the config and register the qbit (scaffold: the email_ tables, - ** providers, processes and app arrive in phase (a), docs/DESIGN.md). + ** Validate and register the tables, bridge, processes, alerts and app. *******************************************************************************/ @Override public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException @@ -125,6 +137,59 @@ public class EmailQBitProducer implements QBitMetaDataProducer throw (new QException("Email qbit configuration is invalid: " + String.join("; ", errors))); } qInstance.addQBit(getQBitMetaData()); + MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput(); + EmailTables.possibleValueSources().forEach(output::add); + Map tables = new LinkedHashMap<>(); + for(QTableMetaData table : EmailTables.tables(qBitConfig)) + { + Class customizer = EmailTables.PROVIDER.equals(table.getName()) ? EmailTableCustomizers.ProviderCustomizer.class + : EmailTables.SUPPRESSION.equals(table.getName()) ? EmailTableCustomizers.SuppressionCustomizer.class : null; + if(customizer != null) + { + table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(customizer)) + .withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(customizer)) + .withCustomizer(TableCustomizers.PRE_DELETE_RECORD, new QCodeReference(customizer)); + } + if(EmailTables.MESSAGE.equals(table.getName())) + { + table.withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(EmailTableCustomizers.MessageLinksCustomizer.class)); + } + if(EmailTables.PROVIDER.equals(table.getName())) + { + table.withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(EmailTableCustomizers.ProviderCustomizer.class)); + for(String field : List.of("usernameEnvName", "secretEnvName", "webhookSecretEnvName")) + { + table.withVirtualField(new QVirtualFieldMetaData(field + "Status", QFieldType.STRING) + .withLabel(field.equals("usernameEnvName") ? "Username set" : field.equals("secretEnvName") ? "Secret set" : "Webhook secret set") + .withIsEditable(false).withIsQuerySelectable(true)); + } + table.getSections().get(1).withAlternative(QFieldSectionAlternativeType.RECORD_VIEW, section -> + { + List fields = new ArrayList<>(section.getFieldNames()); + fields.addAll(List.of("usernameEnvNameStatus", "secretEnvNameStatus", "webhookSecretEnvNameStatus")); + section.setFieldNames(fields); + }); + } + if(qBitConfig.getTableMetaDataCustomizer() != null) + { + table = qBitConfig.getTableMetaDataCustomizer().customizeMetaData(qInstance, table); + } + table.setSourceQBitName(getQBitMetaData().getName()); + tables.put(table.getName(), table); + output.add(table); + } + EmailTables.joins().forEach(output::add); + EmailTables.childWidgets().forEach(output::add); + EmailApp.widgets().forEach(output::add); + EmailProcesses.processes(qBitConfig).forEach(process -> + { + process.setSourceQBitName(getQBitMetaData().getName()); + output.add(process); + }); + EmailAlerts.types().forEach(output::add); + output.add(EmailApp.app(qBitConfig, tables)); + output.add(new EmailOutboxMessagingProviderMetaData().withName(qBitConfig.getBridgeProviderName())); + output.addSelfToInstance(qInstance); return (new MetaDataProducerMultiOutput()); } } diff --git a/src/main/java/com/kingsrook/qbits/email/EmailRuntime.java b/src/main/java/com/kingsrook/qbits/email/EmailRuntime.java new file mode 100644 index 0000000..531997c --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/EmailRuntime.java @@ -0,0 +1,126 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email; + + +import java.lang.management.ManagementFactory; +import java.time.Clock; +import java.util.UUID; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; + + +/******************************************************************************* + ** Runtime lookups for the qbit. + ** + **

The config is read from the current QInstance on every call (never held + ** in a static), so reconfiguring, or two instances in one JVM, never see + ** stale settings. The only JVM-wide state is the runner identity + ** and the clock, which tests replace.

+ *******************************************************************************/ +public final class EmailRuntime +{ + + private static final String NODE_ID = ManagementFactory.getRuntimeMXBean().getName() + "/" + UUID.randomUUID().toString().substring(0, 8); + + private static volatile Clock clock = Clock.systemUTC(); + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailRuntime() + { + } + + + + /******************************************************************************* + ** The config in the current instance, or null when the qbit is not installed. + *******************************************************************************/ + public static EmailQBitConfig config() + { + return (config(QContext.getQInstance())); + } + + + + /******************************************************************************* + ** The config in an instance, or null when the qbit is not installed. + *******************************************************************************/ + public static EmailQBitConfig config(QInstance qInstance) + { + if(qInstance == null || qInstance.getQBits() == null) + { + return (null); + } + for(QBitMetaData qBit : qInstance.getQBits().values()) + { + if(EmailQBitProducer.ARTIFACT_ID.equals(qBit.getArtifactId()) && qBit.getConfig() instanceof EmailQBitConfig config) + { + return (config); + } + } + return (null); + } + + + + /******************************************************************************* + ** True when the qbit is installed in the current instance. + *******************************************************************************/ + public static boolean isInstalled() + { + return (config() != null); + } + + + + /******************************************************************************* + ** The clock (tests replace it). + *******************************************************************************/ + public static Clock clock() + { + return (clock); + } + + + + /******************************************************************************* + ** Replace the clock (tests). + *******************************************************************************/ + public static void setClock(Clock newClock) + { + clock = newClock == null ? Clock.systemUTC() : newClock; + } + + + + /******************************************************************************* + ** Runner identity stored on claimed attempts. + *******************************************************************************/ + public static String nodeId() + { + return (NODE_ID); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/app/EmailApp.java b/src/main/java/com/kingsrook/qbits/email/app/EmailApp.java new file mode 100644 index 0000000..2b0e03b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/app/EmailApp.java @@ -0,0 +1,114 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.app; + + +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput; +import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; + + +/******************************************************************************* + ** Email app with a placeholder overview and the five main tables in navigation. + ** Versions and attempts are child lists, and processes are record actions. + *******************************************************************************/ +public final class EmailApp +{ + public static final String HOME_WIDGET = "emailHome"; + + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailApp() + { + } + + + + /******************************************************************************* + ** Placeholder home widget. + *******************************************************************************/ + public static List widgets() + { + return (List.of( + new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Overview").withType(WidgetType.HTML.getType()) + .withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false) + .withPermissionRules(EmailTables.permissionRules()))); + } + + + + /******************************************************************************* + ** The app, with the main tables as children. + *******************************************************************************/ + public static QAppMetaData app(EmailQBitConfig config, Map tables) + { + QAppMetaData app = new QAppMetaData() + .withName(config.getAppName()) + .withLabel(config.getAppLabel()) + .withIcon(new QIcon("mail")) + .withPermissionRules(EmailTables.permissionRules()) + .withWidgets(List.of(HOME_WIDGET)); + for(String name : EmailTables.MAIN) + { + if(tables.containsKey(name)) + { + app.withChild(tables.get(name)); + } + } + return (app); + } + + + + /******************************************************************************* + ** Home renderer. + *******************************************************************************/ + public static class HomeRenderer extends AbstractWidgetRenderer + { + /******************************************************************************* + ** Render. + *******************************************************************************/ + @Override + public RenderWidgetOutput render(RenderWidgetInput input) throws QException + { + return (new RenderWidgetOutput(new RawHTML(null, "

Use Messages to follow queued email and Providers to manage delivery.

"))); + } + } + + + +} diff --git a/src/main/java/com/kingsrook/qbits/email/bridge/EmailOutboxMessagingProvider.java b/src/main/java/com/kingsrook/qbits/email/bridge/EmailOutboxMessagingProvider.java new file mode 100644 index 0000000..a657a9f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/bridge/EmailOutboxMessagingProvider.java @@ -0,0 +1,134 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.bridge; + + +import com.kingsrook.qbits.email.outbox.EmailRequest; +import com.kingsrook.qbits.email.outbox.EmailSendResult; +import com.kingsrook.qbits.email.outbox.EmailService; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.messaging.MultiParty; +import com.kingsrook.qqq.backend.core.model.actions.messaging.Party; +import com.kingsrook.qqq.backend.core.model.actions.messaging.SendMessageInput; +import com.kingsrook.qqq.backend.core.model.actions.messaging.SendMessageOutput; +import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailContentRole; +import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailPartyRole; +import com.kingsrook.qqq.backend.core.modules.messaging.MessagingProviderInterface; +import jakarta.mail.internet.InternetAddress; + + +/******************************************************************************* + ** Adapts core messaging calls to durable asynchronous outbox inserts. + *******************************************************************************/ +public class EmailOutboxMessagingProvider implements MessagingProviderInterface +{ + /******************************************************************************* + ** Distinct provider type. + *******************************************************************************/ + @Override + public String getType() + { + return (EmailOutboxMessagingProviderMetaData.TYPE); + } + + /******************************************************************************* + ** Map core parties and alternatives; refuse unsupported attachments. + *******************************************************************************/ + @Override + public SendMessageOutput sendMessage(SendMessageInput input) throws QException + { + if(input.getAttachmentList() != null && !input.getAttachmentList().isEmpty()) + { + throw (new QException("Email attachments are not supported in part 1")); + } + EmailRequest request = new EmailRequest().withSubject(input.getSubject()); + parties(request, input.getTo(), EmailPartyRole.TO); + parties(request, input.getFrom(), EmailPartyRole.FROM); + if(input.getContentList() != null) + { + for(var content : input.getContentList()) + { + if(content.getContentRole() == EmailContentRole.HTML) + { + request.withHtmlBody(content.getBody()); + } + else if(content.getContentRole() == EmailContentRole.TEXT || content.getContentRole() == null) + { + request.withTextBody(content.getBody()); + } + else + { + throw (new QException("Unsupported email content role")); + } + } + } + EmailSendResult result = EmailService.send(request); + if(result.status() == EmailSendResult.Status.REFUSED) + { + throw (new QException(String.join("; ", result.reasons()))); + } + return (new SendMessageOutput()); + } + + /******************************************************************************* + ** Walk nested MultiParty inputs, inheriting a parent role when omitted. + *******************************************************************************/ + private static void parties(EmailRequest request, Party party, EmailPartyRole inherited) throws QException + { + if(party == null) + { + return; + } + EmailPartyRole role = party.getRole() instanceof EmailPartyRole explicit ? explicit : inherited; + if(party instanceof MultiParty multi) + { + if(multi.getPartyList() != null) + { + for(Party child : multi.getPartyList()) + { + parties(request, child, role); + } + } + return; + } + String address = party.getAddress(); + try + { + if(party.getLabel() != null) + { + address = new InternetAddress(address, party.getLabel(), "UTF-8").toString(); + } + } + catch(Exception e) + { + throw (new QException("Invalid email party")); + } + switch(role) + { + case TO -> request.withTo(address); + case CC -> request.withCc(address); + case BCC -> request.withBcc(address); + case FROM -> request.withFrom(address); + case REPLY_TO -> request.withReplyTo(address); + default -> throw (new QException("Unsupported email party role")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/bridge/EmailOutboxMessagingProviderMetaData.java b/src/main/java/com/kingsrook/qbits/email/bridge/EmailOutboxMessagingProviderMetaData.java new file mode 100644 index 0000000..486e6b1 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/bridge/EmailOutboxMessagingProviderMetaData.java @@ -0,0 +1,47 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.bridge; + + +import com.kingsrook.qqq.backend.core.model.metadata.messaging.QMessagingProviderMetaData; +import com.kingsrook.qqq.backend.core.modules.messaging.QMessagingProviderDispatcher; + + +/******************************************************************************* + ** Core messaging provider type registered independently of EMAIL and SES. + *******************************************************************************/ +public class EmailOutboxMessagingProviderMetaData extends QMessagingProviderMetaData +{ + public static final String TYPE = "QBIT_EMAIL"; + + static + { + QMessagingProviderDispatcher.registerMessagingProvider(new EmailOutboxMessagingProvider()); + } + + /******************************************************************************* + ** The bridge type. + *******************************************************************************/ + public EmailOutboxMessagingProviderMetaData() + { + setType(TYPE); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/delivery/EmailAlerts.java b/src/main/java/com/kingsrook/qbits/email/delivery/EmailAlerts.java new file mode 100644 index 0000000..7d2a291 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/delivery/EmailAlerts.java @@ -0,0 +1,76 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.delivery; + + +import java.util.List; +import com.kingsrook.qbits.email.EmailRuntime; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qqq.backend.core.alerts.AlertSeverity; +import com.kingsrook.qqq.backend.core.alerts.AlertSignal; +import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData; +import com.kingsrook.qqq.backend.core.alerts.QAlerts; + + +/******************************************************************************* + ** Optional alert sink integration with content-free signals. + *******************************************************************************/ +public final class EmailAlerts +{ + public static final String UNHEALTHY = "email.provider.unhealthy"; + public static final String DEAD_LETTER = "email.message.dead-letter"; + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailAlerts() + { + } + + /******************************************************************************* + ** Declare alert types using RC.5 dotted-name syntax. + *******************************************************************************/ + public static List types() + { + return (List.of(new QAlertTypeMetaData().withName(UNHEALTHY).withLabel("Email provider unhealthy").withDefaultSeverity(AlertSeverity.ERROR) + .withSubjectTableName(EmailTables.PROVIDER).withVisibilityPermission("email.admin").withSourceName("email"), + new QAlertTypeMetaData().withName(DEAD_LETTER).withLabel("Email dead letters").withDefaultSeverity(AlertSeverity.WARNING) + .withSubjectTableName(EmailTables.MESSAGE).withVisibilityPermission("email.view").withSourceName("email"))); + } + + /******************************************************************************* + ** Raise without addresses, provider diagnostics or content. + *******************************************************************************/ + public static void raise(String type, Integer providerId) + { + String key = "provider:" + providerId + (DEAD_LETTER.equals(type) ? ":" + EmailRuntime.clock().instant().getEpochSecond() / 3600 : ""); + QAlerts.raise(new AlertSignal(type).withDedupKey(key).withTitle(UNHEALTHY.equals(type) ? "Email provider unhealthy" : "Email message dead-lettered") + .withSummary("Review the Email tables for status and retry actions.")); + } + + /******************************************************************************* + ** Resolve on recovery; core guards a missing or failing sink. + *******************************************************************************/ + public static void recovered(Integer providerId) + { + QAlerts.resolve(UNHEALTHY, "provider:" + providerId, "An email attempt succeeded"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java b/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java new file mode 100644 index 0000000..7d95f4c --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java @@ -0,0 +1,496 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.delivery; + + +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qbits.email.EmailRuntime; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qbits.email.transport.EmailEnvelope; +import com.kingsrook.qbits.email.transport.EmailProviderSettings; +import com.kingsrook.qbits.email.transport.EmailTransportException; +import com.kingsrook.qbits.email.transport.EmailTransportRegistry; +import com.kingsrook.qbits.email.transport.EmailTransportResult; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair; + + +/******************************************************************************* + ** Durable at-least-once delivery with unique attempt claims and database leases. + *******************************************************************************/ +public class EmailDispatcher +{ + private static final QLogger LOG = QLogger.getLogger(EmailDispatcher.class); + + public static final List DUE = List.of("QUEUED", "RETRY_WAIT"); + public static final List FINAL = List.of("SENT", "DELIVERED", "DEAD_LETTER", "BOUNCED", "COMPLAINED", "SUPPRESSED", "CANCELLED"); + private final EmailTransportRegistry transports; + private final String runner = EmailRuntime.nodeId() + "/" + UUID.randomUUID(); + + /******************************************************************************* + ** Default delivery transports. + *******************************************************************************/ + public EmailDispatcher() + { + this(new EmailTransportRegistry()); + } + + /******************************************************************************* + ** Inject transport registry for host extensions and tests. + *******************************************************************************/ + public EmailDispatcher(EmailTransportRegistry transports) + { + this.transports = transports; + } + + /******************************************************************************* + ** One message delivery result. + *******************************************************************************/ + public enum SendResult + { + SENT, + RETRY_WAIT, + DEAD_LETTER, + PROVIDER_FAILURE, + NOT_SENT + } + + /******************************************************************************* + ** Counts from a dispatch pass. + *******************************************************************************/ + public record RunSummary(int sent, int failed, int skipped) + { + } + + /******************************************************************************* + ** A database claim, fully committed before network I/O. + *******************************************************************************/ + private record Claim(QRecord message, QRecord provider, QRecord attempt) + { + } + + /******************************************************************************* + ** Dispatch the configured batch. + *******************************************************************************/ + public RunSummary runOnce() throws QException + { + return (run(null)); + } + + /******************************************************************************* + ** Dispatch due message ids, or all due messages when null. + *******************************************************************************/ + public RunSummary run(List ids) throws QException + { + if(EmailRuntime.config() == null || (ids != null && ids.isEmpty())) + { + return (new RunSummary(0, 0, 0)); + } + QQueryFilter filter = new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.IN, "QUEUED", "RETRY_WAIT", "SENDING")) + .withCriteria(new QFilterCriteria("nextAttemptDate", QCriteriaOperator.LESS_THAN_OR_EQUALS, EmailRuntime.clock().instant())) + .withOrderBy(new QFilterOrderBy("id")).withLimit(EmailRuntime.config().getBatchSize()); + if(ids != null) + { + filter.withCriteria(new QFilterCriteria("id", QCriteriaOperator.IN, new ArrayList<>(ids))); + } + int sent = 0; + int failed = 0; + int skipped = 0; + Integer afterId = 0; + Set failedProviders = new HashSet<>(); + while(sent + failed < EmailRuntime.config().getBatchSize()) + { + QQueryFilter pageFilter = filter.clone().withCriteria(new QFilterCriteria("id", QCriteriaOperator.GREATER_THAN, afterId)); + List page = SystemWrites.query(EmailTables.MESSAGE, pageFilter); + if(page.isEmpty()) + { + break; + } + for(QRecord row : page) + { + afterId = row.getValueInteger("id"); + if(failedProviders.contains(row.getValueInteger("providerId"))) + { + skipped++; + continue; + } + try + { + SendResult result = attempt(afterId, false); + switch(result) + { + case SENT -> sent++; + case RETRY_WAIT, DEAD_LETTER -> failed++; + case PROVIDER_FAILURE -> + { + failed++; + failedProviders.add(row.getValueInteger("providerId")); + } + default -> skipped++; + } + } + catch(QException | RuntimeException e) + { + failed++; + LOG.warn("Email attempt failed", logPair("messageId", afterId)); + } + if(sent + failed >= EmailRuntime.config().getBatchSize()) + { + break; + } + } + } + return (new RunSummary(sent, failed, skipped)); + } + + /******************************************************************************* + ** Manual send bypasses due time but preserves rate limits, health and leases. + *******************************************************************************/ + public SendResult sendNow(Integer id) throws QException + { + SendResult result = attempt(id, true); + return (result == SendResult.PROVIDER_FAILURE ? SendResult.RETRY_WAIT : result); + } + + /******************************************************************************* + ** Claim, send outside the transaction, then persist the fenced result. + *******************************************************************************/ + protected SendResult attempt(Integer id, boolean now) throws QException + { + Claim claim = claim(id, now); + if(claim == null) + { + return (SendResult.NOT_SENT); + } + EmailTransportResult result = null; + EmailTransportException error = null; + try + { + result = transports.get(claim.provider().getValueString("kind")).send(envelope(claim.message()), new EmailProviderSettings(claim.provider(), EmailRuntime.config())); + } + catch(EmailTransportException e) + { + error = e; + } + catch(RuntimeException e) + { + error = new EmailTransportException("Transport temporarily unavailable", false); + } + return (finish(claim, result, error)); + } + + /******************************************************************************* + ** Unique attempt insertion under a short provider lock, including rate checks. + *******************************************************************************/ + private Claim claim(Integer id, boolean immediate) throws QException + { + EmailQBitConfig config = EmailRuntime.config(); + QRecord initial = EmailStore.get(EmailTables.MESSAGE, id, null); + if(initial == null || config == null) + { + return (null); + } + try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER))) + { + QRecord provider = EmailStore.lockProvider(initial.getValueInteger("providerId"), tx); + QRecord message = EmailStore.get(EmailTables.MESSAGE, id, tx); + Instant time = EmailRuntime.clock().instant(); + if(provider == null || message == null || !Boolean.TRUE.equals(provider.getValueBoolean("enabled"))) + { + return (null); + } + String status = message.getValueString("status"); + int previous = message.getValueInteger("attemptCount") == null ? 0 : message.getValueInteger("attemptCount"); + if("SENDING".equals(status)) + { + Instant lease = message.getValueInstant("leaseExpiresDate"); + if(lease != null && lease.isAfter(time)) + { + return (null); + } + expire(message, tx, time); + if(budgetAttempts(id, previous, tx) >= config.getMaxAttempts()) + { + EmailStore.update(EmailTables.MESSAGE, new QRecord().withValue("id", id).withValue("status", "DEAD_LETTER") + .withValue("deadLetterDate", time).withValue("finishedDate", time).withValue("leaseExpiresDate", null).withValue("nextAttemptDate", null).withValue("lastError", "Attempt lease expired"), tx); + tx.commit(); + alert(() -> EmailAlerts.raise(EmailAlerts.DEAD_LETTER, provider.getValueInteger("id"))); + return (null); + } + } + else if(!DUE.contains(status)) + { + return (null); + } + if(!immediate && ((message.getValueInstant("sendAfter") != null && message.getValueInstant("sendAfter").isAfter(time)) + || (message.getValueInstant("nextAttemptDate") != null && message.getValueInstant("nextAttemptDate").isAfter(time)))) + { + return (null); + } + if(limited(provider, tx, time)) + { + return (null); + } + Instant lease = time.plusSeconds(config.getLeaseSeconds()); + QRecord attempt = new QRecord().withValue("messageId", id).withValue("providerId", provider.getValue("id")) + .withValue("attemptNumber", previous + 1).withValue("outcome", "IN_FLIGHT").withValue("claimedBy", runner) + .withValue("startDate", time).withValue("leaseExpiresDate", lease); + for(var field : config.getSecurityFields()) + { + attempt.setValue(field.getName(), message.getValue(field.getName())); + } + QRecord inserted = SystemWrites.insert(EmailTables.ATTEMPT, List.of(attempt), tx).getRecords().get(0); + if(inserted.getValueInteger("id") == null || (inserted.getErrors() != null && !inserted.getErrors().isEmpty())) + { + return (null); + } + EmailStore.update(EmailTables.MESSAGE, new QRecord().withValue("id", id).withValue("status", "SENDING").withValue("attemptCount", previous + 1) + .withValue("leaseExpiresDate", lease).withValue("nextAttemptDate", lease), tx); + tx.commit(); + return (new Claim(message, provider, inserted)); + } + } + + /******************************************************************************* + ** Shared rate and unhealthy-provider probe budgets, counted from attempts. + *******************************************************************************/ + private static boolean limited(QRecord provider, QBackendTransaction transaction, Instant now) throws QException + { + for(String field : List.of("maxPerMinute", "maxPerDay")) + { + Integer limit = provider.getValueInteger(field); + Instant after = now.minusSeconds("maxPerMinute".equals(field) ? 60 : 86400); + if(limit != null && SystemWrites.count(EmailTables.ATTEMPT, EmailStore.equal("providerId", provider.getValueInteger("id")) + .withCriteria(new QFilterCriteria("startDate", QCriteriaOperator.GREATER_THAN, after)), transaction) >= limit) + { + return (true); + } + } + if("UNHEALTHY".equals(provider.getValueString("healthStatus"))) + { + return (SystemWrites.count(EmailTables.ATTEMPT, EmailStore.equal("providerId", provider.getValueInteger("id")) + .withCriteria(new QFilterCriteria("startDate", QCriteriaOperator.GREATER_THAN, now.minusSeconds(300))), transaction) > 0); + } + return (false); + } + + /******************************************************************************* + ** Mark the abandoned attempt before retrying. + *******************************************************************************/ + private static void expire(QRecord message, QBackendTransaction tx, Instant now) throws QException + { + for(QRecord attempt : SystemWrites.query(EmailTables.ATTEMPT, EmailStore.equal("messageId", message.getValueInteger("id")) + .withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "IN_FLIGHT")), tx)) + { + EmailStore.update(EmailTables.ATTEMPT, new QRecord().withValue("id", attempt.getValue("id")).withValue("outcome", "LEASE_EXPIRED") + .withValue("endDate", now).withValue("error", "Attempt lease expired"), tx); + } + } + + /******************************************************************************* + ** Fence late completions by attempt number and persist outcome atomically. + *******************************************************************************/ + private SendResult finish(Claim claim, EmailTransportResult result, EmailTransportException error) throws QException + { + try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER))) + { + Integer providerId = claim.provider().getValueInteger("id"); + QRecord provider = EmailStore.lockProvider(providerId, tx); + QRecord message = EmailStore.get(EmailTables.MESSAGE, claim.message().getValueInteger("id"), tx); + int number = claim.attempt().getValueInteger("attemptNumber"); + if(message == null || !"SENDING".equals(message.getValueString("status")) || !Integer.valueOf(number).equals(message.getValueInteger("attemptCount"))) + { + return (SendResult.NOT_SENT); + } + Instant now = EmailRuntime.clock().instant(); + boolean success = error == null && result != null; + boolean permanent = error != null && error.getPermanent(); + boolean providerLevel = error != null && error.getProviderLevel(); + String safeError = error == null ? null : permanent ? "Provider rejected the message" : "Provider temporarily unavailable"; + if(providerLevel) + { + safeError = "Credentials missing or refused"; + } + if(error != null && "Gmail API sending is specified but not built; use GMAIL_SMTP".equals(error.getMessage())) + { + safeError = "Gmail API sending is specified but not built; use GMAIL_SMTP"; + } + EmailStore.update(EmailTables.ATTEMPT, new QRecord().withValue("id", claim.attempt().getValue("id")) + .withValue("outcome", success ? "SUCCEEDED" : providerLevel ? "PROVIDER_FAILURE" : permanent ? "PERMANENT_FAILURE" : "TRANSIENT_FAILURE") + .withValue("endDate", now).withValue("durationMillis", (int) Math.max(0, Duration.between(claim.attempt().getValueInstant("startDate"), now).toMillis())) + .withValue("error", safeError), tx); + QRecord update = new QRecord().withValue("id", message.getValue("id")).withValue("leaseExpiresDate", null).withValue("lastError", safeError); + SendResult outcome; + if(success) + { + outcome = SendResult.SENT; + update.withValue("status", "SENT").withValue("sentDate", now).withValue("finishedDate", now).withValue("providerMessageId", result.providerMessageId()).withValue("nextAttemptDate", null); + EmailStore.update(EmailTables.PROVIDER, new QRecord().withValue("id", providerId).withValue("healthStatus", "HEALTHY").withValue("lastSuccessDate", now), tx); + } + else + { + boolean dead = permanent || (!providerLevel && budgetAttempts(message.getValueInteger("id"), number, tx) >= EmailRuntime.config().getMaxAttempts()); + outcome = dead ? SendResult.DEAD_LETTER : SendResult.RETRY_WAIT; + long seconds = 60L * EmailRuntime.config().retryMinutesAfterAttempt(number); + if(error != null && error.getRetryAfter() != null) + { + seconds = Math.max(1, Math.min(3600, error.getRetryAfter().getSeconds())); + } + update.withValue("status", outcome.name()).withValue("deadLetterDate", dead ? now : null).withValue("finishedDate", dead ? now : null).withValue("nextAttemptDate", dead ? null : now.plusSeconds(seconds)); + QRecord health = new QRecord().withValue("id", providerId).withValue("lastFailureDate", now).withValue("lastFailureMessage", safeError); + if(providerLevel || (!permanent && consecutiveFailures(providerId, tx) >= 10)) + { + health.withValue("healthStatus", "UNHEALTHY"); + } + EmailStore.update(EmailTables.PROVIDER, health, tx); + } + EmailStore.update(EmailTables.MESSAGE, update, tx); + QRecord updatedProvider = EmailStore.get(EmailTables.PROVIDER, providerId, tx); + tx.commit(); + if(success && "UNHEALTHY".equals(provider.getValueString("healthStatus"))) + { + alert(() -> EmailAlerts.recovered(providerId)); + } + if("UNHEALTHY".equals(updatedProvider.getValueString("healthStatus")) && !"UNHEALTHY".equals(provider.getValueString("healthStatus"))) + { + alert(() -> EmailAlerts.raise(EmailAlerts.UNHEALTHY, providerId)); + } + if(outcome == SendResult.DEAD_LETTER) + { + alert(() -> EmailAlerts.raise(EmailAlerts.DEAD_LETTER, providerId)); + } + LOG.info("Email attempt completed", logPair("messageId", message.getValue("id")), logPair("status", outcome.name())); + return (providerLevel ? SendResult.PROVIDER_FAILURE : outcome); + } + } + + /******************************************************************************* + ** Keep claim numbering monotonic while excluding provider failures from limits. + *******************************************************************************/ + private static long budgetAttempts(Integer messageId, int attempts, QBackendTransaction tx) throws QException + { + return (attempts - SystemWrites.count(EmailTables.ATTEMPT, EmailStore.equal("messageId", messageId) + .withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "PROVIDER_FAILURE")), tx)); + } + + /******************************************************************************* + ** Optional alerting must never change the result of a committed delivery. + *******************************************************************************/ + private static void alert(Runnable notification) + { + try + { + notification.run(); + } + catch(RuntimeException e) + { + LOG.warn("Email alert unavailable"); + } + } + + /******************************************************************************* + ** Consecutive completed transient failures; a success or permanent rejection breaks the streak. + *******************************************************************************/ + private static int consecutiveFailures(Integer providerId, QBackendTransaction tx) throws QException + { + int failures = 0; + for(QRecord attempt : SystemWrites.query(EmailTables.ATTEMPT, EmailStore.equal("providerId", providerId) + .withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.NOT_EQUALS, "IN_FLIGHT")) + .withOrderBy(new QFilterOrderBy("endDate", false)).withOrderBy(new QFilterOrderBy("id", false)).withLimit(10), tx)) + { + if(!"TRANSIENT_FAILURE".equals(attempt.getValueString("outcome"))) + { + break; + } + failures++; + } + return (failures); + } + + /******************************************************************************* + ** The persisted alternatives are reused verbatim on each attempt. + *******************************************************************************/ + private static EmailEnvelope envelope(QRecord message) + { + return (new EmailEnvelope().withMessageId(message.getValueString("id")).withFrom(message.getValueString("fromAddress")) + .withTo(split(message.getValueString("toAddresses"))).withCc(split(message.getValueString("ccAddresses"))).withBcc(split(message.getValueString("bccAddresses"))) + .withReplyTo(message.getValueString("replyTo")).withSubject(message.getValueString("subject")) + .withHtmlBody(message.getValueString("htmlBody")).withTextBody(message.getValueString("textBody"))); + } + + /******************************************************************************* + ** Preserve quoted display names by treating the stored list as one RFC address list. + *******************************************************************************/ + private static List split(String value) + { + return (value == null || value.isBlank() ? List.of() : List.of(value)); + } + + /******************************************************************************* + ** Lease recovery and five-minute provider probes share the normal claim path. + *******************************************************************************/ + public RunSummary maintain() throws QException + { + Instant now = EmailRuntime.clock().instant(); + for(QRecord candidate : SystemWrites.query(EmailTables.MESSAGE, EmailStore.equal("status", "SENDING") + .withCriteria(new QFilterCriteria("leaseExpiresDate", QCriteriaOperator.LESS_THAN_OR_EQUALS, now)))) + { + try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER))) + { + EmailStore.lockProvider(candidate.getValueInteger("providerId"), tx); + QRecord current = EmailStore.get(EmailTables.MESSAGE, candidate.getValueInteger("id"), tx); + if(current == null || !"SENDING".equals(current.getValueString("status")) || current.getValueInstant("leaseExpiresDate").isAfter(now)) + { + continue; + } + expire(current, tx, now); + boolean dead = budgetAttempts(current.getValueInteger("id"), current.getValueInteger("attemptCount"), tx) >= EmailRuntime.config().getMaxAttempts(); + EmailStore.update(EmailTables.MESSAGE, new QRecord().withValue("id", current.getValue("id")) + .withValue("status", dead ? "DEAD_LETTER" : "RETRY_WAIT").withValue("nextAttemptDate", dead ? null : now) + .withValue("leaseExpiresDate", null).withValue("lastError", "Attempt lease expired") + .withValue("deadLetterDate", dead ? now : null).withValue("finishedDate", dead ? now : null), tx); + tx.commit(); + if(dead) + { + alert(() -> EmailAlerts.raise(EmailAlerts.DEAD_LETTER, candidate.getValueInteger("providerId"))); + } + } + } + return (runOnce()); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/metadata/EmailEnums.java b/src/main/java/com/kingsrook/qbits/email/metadata/EmailEnums.java new file mode 100644 index 0000000..9d35729 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/metadata/EmailEnums.java @@ -0,0 +1,315 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.metadata; + + +import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum; + + +/******************************************************************************* + ** Enumerated database values and their display labels. + *******************************************************************************/ +public final class EmailEnums +{ + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailEnums() + { + } + + + /******************************************************************************* + ** ProviderKind values. + *******************************************************************************/ + public enum ProviderKind implements PossibleValueEnum + { + SMTP, + SES, + POSTMARK, + SENDGRID, + MAILGUN, + GMAIL_SMTP, + GMAIL_API; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** Security values. + *******************************************************************************/ + public enum Security implements PossibleValueEnum + { + NONE, + STARTTLS, + TLS; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** MessageStatus values. + *******************************************************************************/ + public enum MessageStatus implements PossibleValueEnum + { + QUEUED, + SENDING, + SENT, + DELIVERED, + RETRY_WAIT, + DEAD_LETTER, + BOUNCED, + COMPLAINED, + SUPPRESSED, + CANCELLED; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** AttemptOutcome values. + *******************************************************************************/ + public enum AttemptOutcome implements PossibleValueEnum + { + IN_FLIGHT, + SUCCEEDED, + TRANSIENT_FAILURE, + PROVIDER_FAILURE, + PERMANENT_FAILURE, + LEASE_EXPIRED; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** HealthStatus values. + *******************************************************************************/ + public enum HealthStatus implements PossibleValueEnum + { + HEALTHY, + UNHEALTHY; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** TemplateKind values. + *******************************************************************************/ + public enum TemplateKind implements PossibleValueEnum + { + MARKDOWN, + MJML, + HTML; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** VersionStatus values. + *******************************************************************************/ + public enum VersionStatus implements PossibleValueEnum + { + DRAFT, + ACTIVE, + RETIRED; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** SuppressionScope values. + *******************************************************************************/ + public enum SuppressionScope implements PossibleValueEnum + { + ALL, + TRANSACTIONAL, + NOTIFICATION, + MARKETING, + SECURITY; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } + + + /******************************************************************************* + ** SuppressionReason values. + *******************************************************************************/ + public enum SuppressionReason implements PossibleValueEnum + { + HARD_BOUNCE, + COMPLAINT, + UNSUBSCRIBE, + MANUAL; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Plain display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' ')); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/metadata/EmailTableCustomizers.java b/src/main/java/com/kingsrook/qbits/email/metadata/EmailTableCustomizers.java new file mode 100644 index 0000000..c083699 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/metadata/EmailTableCustomizers.java @@ -0,0 +1,358 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.metadata; + + +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import java.util.Objects; +import java.util.Optional; +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qbits.email.EmailRuntime; +import com.kingsrook.qbits.email.permissions.EmailPermissionChecker; +import com.kingsrook.qbits.email.permissions.EmailPermissions; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qbits.email.transport.EmailProviderSettings; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage; +import jakarta.mail.internet.InternetAddress; + + +/******************************************************************************* + ** Provider and suppression validation shared by UI, API and code. + *******************************************************************************/ +public final class EmailTableCustomizers +{ + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailTableCustomizers() + { + } + + /******************************************************************************* + ** Merged values for sparse updates. + *******************************************************************************/ + private static QRecord effective(QRecord record, Optional> oldRecords) + { + QRecord merged = new QRecord(); + if(oldRecords != null && oldRecords.isPresent()) + { + oldRecords.get().stream().filter(r -> Objects.equals(r.getValue("id"), record.getValue("id"))).findFirst() + .ifPresent(r -> merged.getValues().putAll(r.getValues())); + } + merged.getValues().putAll(record.getValues()); + return (merged); + } + + /******************************************************************************* + ** Validate a provider without resolving any credentials. + *******************************************************************************/ + public static List validateProvider(QRecord row, EmailQBitConfig config) + { + List errors = new ArrayList<>(); + String kind = row.getValueString("kind"); + if("GMAIL_API".equals(kind)) + { + errors.add("Gmail API sending is specified but not built; use GMAIL_SMTP"); + } + if(List.of("POSTMARK", "SENDGRID", "MAILGUN", "GMAIL_SMTP").contains(kind == null ? "" : kind) + && (row.getValueString("secretEnvName") == null || row.getValueString("secretEnvName").isBlank())) + { + errors.add("Provider needs a secret environment name"); + } + if("GMAIL_SMTP".equals(kind)) + { + if(row.getValueString("usernameEnvName") == null || row.getValueString("usernameEnvName").isBlank()) + { + errors.add("Gmail SMTP needs an account address environment name"); + } + if(row.getValueString("security") != null && !"STARTTLS".equals(row.getValueString("security"))) + { + errors.add("Gmail SMTP requires STARTTLS"); + } + } + if("MAILGUN".equals(kind)) + { + if(!EmailProviderSettings.validSendingDomain(row.getValueString("sendingDomain"))) + { + errors.add("Mailgun needs a valid sending domain"); + } + if(!List.of("US", "EU").contains(row.getValueString("region") == null ? "" : row.getValueString("region"))) + { + errors.add("Mailgun region must be US or EU"); + } + } + if(("SMTP".equals(kind) || "GMAIL_SMTP".equals(kind)) + && !new EmailProviderSettings(row, config).allowsSmtpAuth()) + { + errors.add("SMTP sign-in needs STARTTLS or TLS"); + } + for(String field : List.of("fromName", "fromAddress", "replyTo")) + { + String value = row.getValueString(field); + if(value != null && (value.contains("\r") || value.contains("\n"))) + { + errors.add("Provider sender and reply-to fields must not contain line breaks"); + } + } + for(String field : List.of("usernameEnvName", "secretEnvName", "webhookSecretEnvName")) + { + String name = row.getValueString(field); + if(name != null && (!EmailQBitConfig.ENV_NAME.matcher(name).matches() || !config.effectiveAllowedEnvNames(kind).contains(name))) + { + errors.add("Credential references must be allowed environment variable names"); + } + } + if("SMTP".equals(kind) && (row.getValueString("host") == null || row.getValueString("host").isBlank() + || row.getValueInteger("port") == null || row.getValueInteger("port") < 1 || row.getValueInteger("port") > 65535)) + { + errors.add("SMTP needs a host and a port between 1 and 65535"); + } + if("SES".equals(kind) && (row.getValueString("region") == null || row.getValueString("region").isBlank())) + { + errors.add("SES needs a region"); + } + if("SES".equals(kind) && (row.getValueString("usernameEnvName") == null) != (row.getValueString("secretEnvName") == null)) + { + errors.add("SES needs both credential environment names or neither for the default chain"); + } + for(String field : List.of("maxPerMinute", "maxPerDay")) + { + if(row.getValueInteger(field) != null && row.getValueInteger(field) < 1) + { + errors.add("Rate limits must be positive or empty"); + } + } + try + { + InternetAddress from = new InternetAddress(row.getValueString("fromAddress"), true); + from.validate(); + String domain = row.getValueString("sendingDomain"); + if(domain != null && !domain.equalsIgnoreCase(from.getAddress().substring(from.getAddress().lastIndexOf('@') + 1))) + { + errors.add("From address domain must match sending domain"); + } + } + catch(Exception e) + { + errors.add("A valid from address is required"); + } + return (errors); + } + + /******************************************************************************* + ** Provider validation and secret-presence display. + *******************************************************************************/ + public static class ProviderCustomizer implements TableCustomizerInterface + { + /******************************************************************************* + ** Validate the resulting provider set, including sparse and batch writes. + *******************************************************************************/ + @Override + public List preInsertOrUpdate(AbstractActionInput input, List records, boolean preview, Optional> oldRecords) throws QException + { + if(SystemWrites.active()) + { + return (records); + } + List resulting = new ArrayList<>(SystemWrites.query(EmailTables.PROVIDER, new QQueryFilter(), + input instanceof com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput insert ? insert.getTransaction() + : input instanceof com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput update ? update.getTransaction() : null)); + for(QRecord record : records) + { + QRecord merged = effective(record, oldRecords); + validateProvider(merged, EmailRuntime.config()).forEach(error -> record.addError(new BadInputStatusMessage(error))); + for(String field : List.of("usernameEnvName", "secretEnvName", "webhookSecretEnvName")) + { + String name = record.getValueString(field); + if(name != null && !EmailQBitConfig.ENV_NAME.matcher(name).matches()) + { + // An accidentally pasted credential must not be echoed in the failed write. + record.setValue(field, null); + } + } + resulting.removeIf(r -> Objects.equals(r.getValue("id"), record.getValue("id"))); + resulting.add(merged); + record.setValue("defaultSlot", Boolean.TRUE.equals(merged.getValueBoolean("enabled")) && Boolean.TRUE.equals(merged.getValueBoolean("isDefault")) ? 1 : null); + } + long enabled = resulting.stream().filter(r -> Boolean.TRUE.equals(r.getValueBoolean("enabled"))).count(); + long defaults = resulting.stream().filter(r -> Boolean.TRUE.equals(r.getValueBoolean("enabled")) && Boolean.TRUE.equals(r.getValueBoolean("isDefault"))).count(); + if(enabled > 0 && defaults != 1) + { + records.forEach(r -> r.addError(new BadInputStatusMessage("Exactly one enabled provider must be the default"))); + } + return (records); + } + + /******************************************************************************* + ** Prevent deleting the default while other enabled providers remain. + *******************************************************************************/ + @Override + public List preDelete(DeleteInput input, List records, boolean preview) throws QException + { + for(QRecord row : records) + { + if(!SystemWrites.query(EmailTables.MESSAGE, com.kingsrook.qbits.email.support.EmailStore.equal("providerId", row.getValueInteger("id")).withLimit(1), input.getTransaction()).isEmpty()) + { + row.addError(new BadInputStatusMessage("Providers with message history cannot be deleted; disable the provider instead")); + } + } + List remaining = new ArrayList<>(SystemWrites.query(EmailTables.PROVIDER, new QQueryFilter(), input.getTransaction())); + remaining.removeIf(r -> records.stream().anyMatch(d -> Objects.equals(d.getValue("id"), r.getValue("id")))); + if(remaining.stream().anyMatch(r -> Boolean.TRUE.equals(r.getValueBoolean("enabled"))) + && remaining.stream().noneMatch(r -> Boolean.TRUE.equals(r.getValueBoolean("enabled")) && Boolean.TRUE.equals(r.getValueBoolean("isDefault")))) + { + records.forEach(r -> r.addError(new BadInputStatusMessage("Choose another enabled default before deleting this provider"))); + } + return (records); + } + + /******************************************************************************* + ** Presence only; internal queries never resolve credentials. + *******************************************************************************/ + @Override + public List postQuery(QueryOrGetInputInterface input, List records) + { + if(!SystemWrites.active()) + { + for(QRecord row : records) + { + for(String field : List.of("usernameEnvName", "secretEnvName", "webhookSecretEnvName")) + { + String value = null; + String name = row.getValueString(field); + EmailQBitConfig config = EmailRuntime.config(); + if(name != null && EmailQBitConfig.ENV_NAME.matcher(name).matches() && config.effectiveAllowedEnvNames(row.getValueString("kind")).contains(name)) + { + try + { + value = config.getSecretResolver().apply(name); + } + catch(RuntimeException e) + { + // A failing resolver must never leak exception text or values. + } + } + row.setValue(field + "Status", value == null || value.isBlank() ? "Not set" : "Set"); + } + } + } + return (records); + } + } + + /******************************************************************************* + ** Canonical addresses and administrator protection for complaint removal. + *******************************************************************************/ + public static class SuppressionCustomizer implements TableCustomizerInterface + { + /******************************************************************************* + ** Normalize and protect existing complaint rows from being weakened. + *******************************************************************************/ + @Override + public List preInsertOrUpdate(AbstractActionInput input, List records, boolean preview, Optional> oldRecords) + { + for(QRecord record : records) + { + QRecord merged = effective(record, oldRecords); + if(merged.getValue("createdDate") == null) + { + record.setValue("createdDate", EmailRuntime.clock().instant()); + } + if(record.getValueString("address") != null) + { + try + { + InternetAddress address = new InternetAddress(record.getValueString("address"), true); + address.validate(); + record.setValue("address", address.getAddress().toLowerCase(Locale.ROOT)); + } + catch(Exception e) + { + record.addError(new BadInputStatusMessage("A valid suppression address is required")); + } + } + if(oldRecords != null && oldRecords.isPresent() && oldRecords.get().stream().anyMatch(r -> Objects.equals(r.getValue("id"), merged.getValue("id")) && "COMPLAINT".equals(r.getValueString("reason")))) + { + protect(record); + } + } + return (records); + } + + /******************************************************************************* + ** Only administrators remove complaint suppressions. + *******************************************************************************/ + @Override + public List preDelete(DeleteInput input, List records, boolean preview) + { + records.stream().filter(r -> "COMPLAINT".equals(r.getValueString("reason"))).forEach(SuppressionCustomizer::protect); + return (records); + } + + /******************************************************************************* + ** Apply the complaint permission even if core enforcement is disabled. + *******************************************************************************/ + private static void protect(QRecord record) + { + if(!SystemWrites.active() && !EmailPermissionChecker.has(EmailPermissions.ADMIN)) + { + record.addError(new BadInputStatusMessage("Changing a complaint suppression needs email.admin")); + } + } + } + + /******************************************************************************* + ** Link application records through the standard dynamic record-link adornment. + *******************************************************************************/ + public static class MessageLinksCustomizer implements TableCustomizerInterface + { + /******************************************************************************* + ** Supply only the target table name; no linked record content is loaded. + *******************************************************************************/ + @Override + public List postQuery(QueryOrGetInputInterface input, List records) + { + if(input.getShouldGenerateDisplayValues()) + { + for(QRecord row : records) + { + String table = row.getValueString("linkedTableName"); + if(table != null && row.getValue("linkedRecordId") != null + && com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getTable(table) != null) + { + row.setDisplayValue("linkedRecordId:toRecordFromTableDynamic", table); + } + } + } + return (records); + } + } + +} diff --git a/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java b/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java new file mode 100644 index 0000000..dce42c5 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java @@ -0,0 +1,415 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.metadata; + + +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qbits.email.outbox.EmailCategory; +import com.kingsrook.qbits.email.permissions.EmailPermissionChecker; +import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer; +import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType; +import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn; +import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType; +import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory; +import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey; + + +/******************************************************************************* + ** Standard visible tables, with system-owned messages and attempt claims. + *******************************************************************************/ +public final class EmailTables +{ + public static final String PROVIDER = "email_provider"; + public static final String MESSAGE = "email_message"; + public static final String ATTEMPT = "email_message_attempt"; + public static final String TEMPLATE = "email_template"; + public static final String VERSION = "email_template_version"; + public static final String BRAND = "email_brand"; + public static final String SUPPRESSION = "email_suppression"; + public static final List ALL = List.of(PROVIDER, MESSAGE, ATTEMPT, TEMPLATE, VERSION, BRAND, SUPPRESSION); + public static final List MAIN = List.of(MESSAGE, TEMPLATE, BRAND, SUPPRESSION, PROVIDER); + public static final List SYSTEM_WRITTEN = List.of(MESSAGE, ATTEMPT); + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailTables() + { + } + + /******************************************************************************* + ** Shared permission base and custom action checker. + *******************************************************************************/ + public static QPermissionRules permissionRules() + { + return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION).withPermissionBaseName("email") + .withCustomPermissionChecker(new QCodeReference(EmailPermissionChecker.class))); + } + + /******************************************************************************* + ** All enum and table possible-value sources. + *******************************************************************************/ + public static List possibleValueSources() + { + List result = new ArrayList<>(); + result.add(QPossibleValueSource.newForEnum("email_provider_kind", EmailEnums.ProviderKind.values())); + result.add(QPossibleValueSource.newForEnum("email_security", EmailEnums.Security.values())); + result.add(QPossibleValueSource.newForEnum("email_health_status", EmailEnums.HealthStatus.values())); + result.add(QPossibleValueSource.newForEnum("email_message_status", EmailEnums.MessageStatus.values())); + result.add(QPossibleValueSource.newForEnum("email_attempt_outcome", EmailEnums.AttemptOutcome.values())); + result.add(QPossibleValueSource.newForEnum("email_template_kind", EmailEnums.TemplateKind.values())); + result.add(QPossibleValueSource.newForEnum("email_version_status", EmailEnums.VersionStatus.values())); + result.add(QPossibleValueSource.newForEnum("email_suppression_scope", EmailEnums.SuppressionScope.values())); + result.add(QPossibleValueSource.newForEnum("email_suppression_reason", EmailEnums.SuppressionReason.values())); + result.add(QPossibleValueSource.newForEnum("email_category", EmailCategory.values())); + ALL.forEach(name -> result.add(QPossibleValueSource.newForTable(name))); + return (result); + } + + /******************************************************************************* + ** The seven phase (a) tables. + *******************************************************************************/ + public static List tables(EmailQBitConfig config) + { + List tables = new ArrayList<>(); + tables.add(provider(config)); + tables.add(message(config)); + tables.add(messageattempt(config)); + tables.add(template(config)); + tables.add(templateversion(config)); + tables.add(brand(config)); + tables.add(suppression(config)); + for(QTableMetaData table : tables) + { + for(QFieldMetaData field : config.getSecurityFields()) + { + table.addField(field.clone()); + table.getSections().get(0).getFieldNames().add(field.getName()); + } + config.getRecordSecurityLocks().forEach(table::withRecordSecurityLock); + if(SYSTEM_WRITTEN.contains(table.getName())) + { + table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE); + table.getFields().values().forEach(f -> f.setIsEditable(false)); + } + table.getFields().values().forEach(f -> f.setBackendName(QInstanceEnricher.inferBackendName(f.getName()))); + } + return (tables); + } + + /******************************************************************************* + ** Common fields and record label. + *******************************************************************************/ + private static QTableMetaData base(EmailQBitConfig config, String name, String label) + { + return (new QTableMetaData().withName(name).withLabel(label).withBackendName(config.getBackendName()) + .withPrimaryKeyField("id").withIcon(new QIcon("mail")).withPermissionRules(permissionRules()) + .withRecordLabelFormatAndFields("%s", "id") + .withField(field("id", QFieldType.INTEGER).withIsEditable(false)) + .withField(field("createDate", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))); + } + + /******************************************************************************* + ** Field with a plain English label. + *******************************************************************************/ + private static QFieldMetaData field(String name, QFieldType type) + { + String words = name.replaceAll("([a-z])([A-Z])", "$1 $2"); + return (new QFieldMetaData(name, type).withLabel(Character.toUpperCase(words.charAt(0)) + words.substring(1))); + } + + + /******************************************************************************* + ** Metadata for email_provider. + *******************************************************************************/ + private static QTableMetaData provider(EmailQBitConfig config) + { + QTableMetaData table = base(config, PROVIDER, "Provider") + .withField(field("name", QFieldType.STRING).withIsRequired(true).withMaxLength(500)) + .withField(field("label", QFieldType.STRING).withMaxLength(500)) + .withField(field("kind", QFieldType.STRING).withPossibleValueSourceName("email_provider_kind").withIsRequired(true).withMaxLength(500)) + .withField(field("enabled", QFieldType.BOOLEAN).withDefaultValue(true)) + .withField(field("isDefault", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("fromAddress", QFieldType.STRING).withIsSensitive(true).withMaxLength(500)) + .withField(field("fromName", QFieldType.STRING).withMaxLength(500)) + .withField(field("replyTo", QFieldType.STRING).withIsSensitive(true).withMaxLength(500)) + .withField(field("sendingDomain", QFieldType.STRING).withMaxLength(500)) + .withField(field("host", QFieldType.STRING).withMaxLength(500)) + .withField(field("port", QFieldType.INTEGER)) + .withField(field("security", QFieldType.STRING).withPossibleValueSourceName("email_security").withDefaultValue("NONE").withMaxLength(500)) + .withField(field("region", QFieldType.STRING).withMaxLength(500)) + .withField(field("usernameEnvName", QFieldType.STRING).withMaxLength(500)) + .withField(field("secretEnvName", QFieldType.STRING).withMaxLength(500)) + .withField(field("webhookSecretEnvName", QFieldType.STRING).withMaxLength(500)) + .withField(field("snsTopicArn", QFieldType.STRING).withMaxLength(500)) + .withField(field("configurationSet", QFieldType.STRING).withMaxLength(500)) + .withField(field("messageStream", QFieldType.STRING).withMaxLength(500)) + .withField(field("maxPerMinute", QFieldType.INTEGER)) + .withField(field("maxPerDay", QFieldType.INTEGER)) + .withField(field("trackOpens", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("trackClicks", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("phiAllowed", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("healthStatus", QFieldType.STRING).withPossibleValueSourceName("email_health_status").withDefaultValue("HEALTHY").withMaxLength(500)) + .withField(field("lastSuccessDate", QFieldType.DATE_TIME)) + .withField(field("lastFailureDate", QFieldType.DATE_TIME)) + .withField(field("lastFailureMessage", QFieldType.STRING).withMaxLength(500)) + .withField(field("dispatchLock", QFieldType.INTEGER).withDefaultValue(0).withIsHidden(true).withIsEditable(false)) + .withField(field("defaultSlot", QFieldType.INTEGER).withIsHidden(true).withIsEditable(false)); + table.withRecordLabelFormatAndFields("%s", "name"); + table.withUniqueKey(new UniqueKey("name")); + table.withUniqueKey(new UniqueKey("defaultSlot")); + table.withSection(SectionFactory.defaultT1("id", "name", "label", "kind", "enabled", "isDefault")); + table.withSection(SectionFactory.defaultT2("fromAddress", "fromName", "replyTo", "sendingDomain", "host", "port", "security", "region", "usernameEnvName", "secretEnvName", "webhookSecretEnvName", "snsTopicArn", "configurationSet", "messageStream", "maxPerMinute", "maxPerDay", "trackOpens", "trackClicks", "phiAllowed", "healthStatus", "lastSuccessDate", "lastFailureDate", "lastFailureMessage")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withSection(SectionFactory.customT2("children", new QIcon("list")).withLabel("Messages").withWidgetName(PROVIDER + "_children")); + return (table); + } + + + /******************************************************************************* + ** Metadata for email_message. + *******************************************************************************/ + private static QTableMetaData message(EmailQBitConfig config) + { + QTableMetaData table = base(config, MESSAGE, "Message") + .withField(field("status", QFieldType.STRING).withPossibleValueSourceName("email_message_status").withMaxLength(500)) + .withField(field("toAddresses", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("ccAddresses", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("bccAddresses", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("fromAddress", QFieldType.STRING).withIsSensitive(true).withMaxLength(500)) + .withField(field("replyTo", QFieldType.STRING).withIsSensitive(true).withMaxLength(500)) + .withField(field("subject", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("templateId", QFieldType.INTEGER).withPossibleValueSourceName(TEMPLATE)) + .withField(field("templateVersionId", QFieldType.INTEGER).withPossibleValueSourceName(VERSION)) + .withField(field("brandId", QFieldType.INTEGER).withPossibleValueSourceName(BRAND)) + .withField(field("category", QFieldType.STRING).withPossibleValueSourceName("email_category").withMaxLength(500)) + .withField(field("locale", QFieldType.STRING).withDefaultValue("en").withMaxLength(500)) + .withField(field("variablesSnapshot", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("htmlBody", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("textBody", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("bodySha256", QFieldType.STRING).withMaxLength(500)) + .withField(field("idempotencyKey", QFieldType.STRING).withMaxLength(500)) + .withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER)) + .withField(field("providerMessageId", QFieldType.STRING).withMaxLength(500)) + .withField(field("attemptCount", QFieldType.INTEGER).withDefaultValue(0)) + .withField(field("nextAttemptDate", QFieldType.DATE_TIME)) + .withField(field("leaseExpiresDate", QFieldType.DATE_TIME)) + .withField(field("lastError", QFieldType.STRING).withMaxLength(500)) + .withField(field("finishedDate", QFieldType.DATE_TIME)) + .withField(field("sentDate", QFieldType.DATE_TIME)) + .withField(field("deliveredDate", QFieldType.DATE_TIME)) + .withField(field("deadLetterDate", QFieldType.DATE_TIME)) + .withField(field("linkedTableName", QFieldType.STRING).withMaxLength(500)) + .withField(field("linkedRecordId", QFieldType.STRING).withMaxLength(500) + .withFieldAdornment(new FieldAdornment(AdornmentType.LINK).withValue(AdornmentType.LinkValues.TO_RECORD_FROM_TABLE_DYNAMIC, true))) + .withField(field("senderQbit", QFieldType.STRING).withMaxLength(500)) + .withField(field("senderProcess", QFieldType.STRING).withMaxLength(500)) + .withField(field("suppressedAddresses", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("sendAfter", QFieldType.DATE_TIME)); + table.withUniqueKey(new UniqueKey("idempotencyKey")); + table.withSection(SectionFactory.defaultT1("id", "status", "toAddresses", "ccAddresses", "bccAddresses", "fromAddress")); + table.withSection(SectionFactory.defaultT2("replyTo", "subject", "templateId", "templateVersionId", "brandId", "category", "locale", "variablesSnapshot", "htmlBody", "textBody", "bodySha256", "idempotencyKey", "providerId", "providerMessageId", "attemptCount", "nextAttemptDate", "leaseExpiresDate", "lastError", "finishedDate", "sentDate", "deliveredDate", "deadLetterDate", "linkedTableName", "linkedRecordId", "senderQbit", "senderProcess", "suppressedAddresses", "sendAfter")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withSection(SectionFactory.customT2("children", new QIcon("list")).withLabel("Attempts").withWidgetName(MESSAGE + "_children")); + return (table); + } + + + /******************************************************************************* + ** Metadata for email_message_attempt. + *******************************************************************************/ + private static QTableMetaData messageattempt(EmailQBitConfig config) + { + QTableMetaData table = base(config, ATTEMPT, "Message Attempt") + .withField(field("messageId", QFieldType.INTEGER).withPossibleValueSourceName(MESSAGE).withIsRequired(true)) + .withField(field("attemptNumber", QFieldType.INTEGER).withIsRequired(true)) + .withField(field("outcome", QFieldType.STRING).withPossibleValueSourceName("email_attempt_outcome").withMaxLength(500)) + .withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER)) + .withField(field("claimedBy", QFieldType.STRING).withMaxLength(500)) + .withField(field("startDate", QFieldType.DATE_TIME)) + .withField(field("endDate", QFieldType.DATE_TIME)) + .withField(field("leaseExpiresDate", QFieldType.DATE_TIME)) + .withField(field("durationMillis", QFieldType.INTEGER)) + .withField(field("error", QFieldType.STRING).withMaxLength(500)); + table.withUniqueKey(new UniqueKey("messageId", "attemptNumber")); + table.withSection(SectionFactory.defaultT1("id", "messageId", "attemptNumber", "outcome", "providerId", "claimedBy")); + table.withSection(SectionFactory.defaultT2("startDate", "endDate", "leaseExpiresDate", "durationMillis", "error")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + return (table); + } + + + /******************************************************************************* + ** Metadata for email_template. + *******************************************************************************/ + private static QTableMetaData template(EmailQBitConfig config) + { + QTableMetaData table = base(config, TEMPLATE, "Template") + .withField(field("name", QFieldType.STRING).withIsRequired(true).withMaxLength(500)) + .withField(field("label", QFieldType.STRING).withMaxLength(500)) + .withField(field("category", QFieldType.STRING).withPossibleValueSourceName("email_category").withMaxLength(500)) + .withField(field("kind", QFieldType.STRING).withPossibleValueSourceName("email_template_kind").withIsRequired(true).withMaxLength(500)) + .withField(field("brandId", QFieldType.INTEGER).withPossibleValueSourceName(BRAND)) + .withField(field("activeVersionId", QFieldType.INTEGER).withPossibleValueSourceName(VERSION)) + .withField(field("phiApproved", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("phiApprovedBy", QFieldType.STRING).withMaxLength(500)) + .withField(field("phiApprovedDate", QFieldType.DATE_TIME)) + .withField(field("editPermissionName", QFieldType.STRING).withMaxLength(500)) + .withField(field("ownerQbit", QFieldType.STRING).withMaxLength(500)) + .withField(field("description", QFieldType.TEXT)); + table.withRecordLabelFormatAndFields("%s", "name"); + table.withUniqueKey(new UniqueKey("name")); + table.withSection(SectionFactory.defaultT1("id", "name", "label", "category", "kind", "brandId")); + table.withSection(SectionFactory.defaultT2("activeVersionId", "phiApproved", "phiApprovedBy", "phiApprovedDate", "editPermissionName", "ownerQbit", "description")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withSection(SectionFactory.customT2("children", new QIcon("list")).withLabel("Versions").withWidgetName(TEMPLATE + "_children")); + return (table); + } + + + /******************************************************************************* + ** Metadata for email_template_version. + *******************************************************************************/ + private static QTableMetaData templateversion(EmailQBitConfig config) + { + QTableMetaData table = base(config, VERSION, "Template Version") + .withField(field("templateId", QFieldType.INTEGER).withPossibleValueSourceName(TEMPLATE)) + .withField(field("versionNumber", QFieldType.INTEGER)) + .withField(field("status", QFieldType.STRING).withPossibleValueSourceName("email_version_status").withMaxLength(500)) + .withField(field("locale", QFieldType.STRING).withDefaultValue("en").withMaxLength(500)) + .withField(field("subject", QFieldType.TEXT).withIsSensitive(true)) + .withField(field("preheader", QFieldType.TEXT)) + .withField(field("bodySource", QFieldType.TEXT)) + .withField(field("textSource", QFieldType.TEXT)) + .withField(field("compiledHtml", QFieldType.TEXT)) + .withField(field("variablesSchema", QFieldType.TEXT)) + .withField(field("sampleVariables", QFieldType.TEXT)) + .withField(field("createdBy", QFieldType.STRING).withMaxLength(500)) + .withField(field("activatedDate", QFieldType.DATE_TIME)); + table.withUniqueKey(new UniqueKey("templateId", "versionNumber", "locale")); + table.withSection(SectionFactory.defaultT1("id", "templateId", "versionNumber", "status", "locale", "subject")); + table.withSection(SectionFactory.defaultT2("preheader", "bodySource", "textSource", "compiledHtml", "variablesSchema", "sampleVariables", "createdBy", "activatedDate")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + return (table); + } + + + /******************************************************************************* + ** Metadata for email_brand. + *******************************************************************************/ + private static QTableMetaData brand(EmailQBitConfig config) + { + QTableMetaData table = base(config, BRAND, "Brand") + .withField(field("name", QFieldType.STRING).withIsRequired(true).withMaxLength(500)) + .withField(field("label", QFieldType.STRING).withMaxLength(500)) + .withField(field("logoUrl", QFieldType.STRING).withMaxLength(500)) + .withField(field("logoDarkUrl", QFieldType.STRING).withMaxLength(500)) + .withField(field("logoAlt", QFieldType.STRING).withMaxLength(500)) + .withField(field("primaryColor", QFieldType.STRING).withMaxLength(500)) + .withField(field("textColor", QFieldType.STRING).withMaxLength(500)) + .withField(field("backgroundColor", QFieldType.STRING).withMaxLength(500)) + .withField(field("darkBackgroundColor", QFieldType.STRING).withMaxLength(500)) + .withField(field("darkTextColor", QFieldType.STRING).withMaxLength(500)) + .withField(field("buttonTextColor", QFieldType.STRING).withMaxLength(500)) + .withField(field("footerMarkdown", QFieldType.TEXT)) + .withField(field("postalAddress", QFieldType.TEXT)) + .withField(field("socialLinks", QFieldType.TEXT)) + .withField(field("siteUrl", QFieldType.STRING).withMaxLength(500)) + .withField(field("isDefault", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("appName", QFieldType.STRING).withMaxLength(500)); + table.withRecordLabelFormatAndFields("%s", "name"); + table.withUniqueKey(new UniqueKey("name")); + table.withSection(SectionFactory.defaultT1("id", "name", "label", "logoUrl", "logoDarkUrl", "logoAlt")); + table.withSection(SectionFactory.defaultT2("primaryColor", "textColor", "backgroundColor", "darkBackgroundColor", "darkTextColor", "buttonTextColor", "footerMarkdown", "postalAddress", "socialLinks", "siteUrl", "isDefault", "appName")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + return (table); + } + + + /******************************************************************************* + ** Metadata for email_suppression. + *******************************************************************************/ + private static QTableMetaData suppression(EmailQBitConfig config) + { + QTableMetaData table = base(config, SUPPRESSION, "Suppression") + .withField(field("address", QFieldType.STRING).withIsRequired(true).withMaxLength(500)) + .withField(field("scope", QFieldType.STRING).withPossibleValueSourceName("email_suppression_scope").withIsRequired(true).withMaxLength(500)) + .withField(field("reason", QFieldType.STRING).withPossibleValueSourceName("email_suppression_reason").withIsRequired(true).withMaxLength(500)) + .withField(field("sourceMessageId", QFieldType.INTEGER)) + .withField(field("sourceEventId", QFieldType.INTEGER)) + .withField(field("note", QFieldType.TEXT)) + .withField(field("createdDate", QFieldType.DATE_TIME)) + .withField(field("expiresDate", QFieldType.DATE_TIME)); + table.withUniqueKey(new UniqueKey("address", "scope")); + table.withSection(SectionFactory.defaultT1("id", "address", "scope", "reason", "sourceMessageId", "sourceEventId")); + table.withSection(SectionFactory.defaultT2("note", "createdDate", "expiresDate")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + return (table); + } + + /******************************************************************************* + ** Relations used by child lists. + *******************************************************************************/ + public static List joins() + { + return (List.of(join(PROVIDER, MESSAGE, "providerId"), join(MESSAGE, ATTEMPT, "messageId"), join(TEMPLATE, VERSION, "templateId"))); + } + + /******************************************************************************* + ** Parent to child relation. + *******************************************************************************/ + private static QJoinMetaData join(String parent, String child, String field) + { + return (new QJoinMetaData().withName(parent + "_join_" + child).withType(JoinType.ONE_TO_MANY) + .withLeftTable(parent).withRightTable(child).withJoinOn(new JoinOn("id", field))); + } + + /******************************************************************************* + ** Child lists live on the parent record, never in navigation. + *******************************************************************************/ + public static List childWidgets() + { + List widgets = new ArrayList<>(); + for(QJoinMetaData join : joins()) + { + widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(join).withName(join.getLeftTable() + "_children") + .withLabel(join.getRightTable().equals(ATTEMPT) ? "Attempts" : join.getRightTable().equals(VERSION) ? "Versions" : "Messages") + .withMaxRows(50).getWidgetMetaData().withGridColumns(12).withPermissionRules(permissionRules())); + } + return (widgets); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/AdHocEmailRenderer.java b/src/main/java/com/kingsrook/qbits/email/outbox/AdHocEmailRenderer.java new file mode 100644 index 0000000..cdb7ae6 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/AdHocEmailRenderer.java @@ -0,0 +1,74 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import java.util.List; +import org.commonmark.ext.autolink.AutolinkExtension; +import org.commonmark.ext.gfm.tables.TablesExtension; +import org.commonmark.parser.Parser; +import org.commonmark.renderer.html.HtmlRenderer; + + +/******************************************************************************* + ** Minimal part 1 layout. Markdown raw HTML and unsafe URL schemes are disabled. + *******************************************************************************/ +public class AdHocEmailRenderer implements EmailRenderer +{ + /******************************************************************************* + ** Render markdown, HTML, or plain text into both alternatives. + *******************************************************************************/ + @Override + public Rendered render(EmailRequest request) + { + String html = request.getHtmlBody(); + if(html == null) + { + if(request.getMarkdownBody() != null) + { + var extensions = List.of(TablesExtension.create(), AutolinkExtension.create()); + html = HtmlRenderer.builder().escapeHtml(true).sanitizeUrls(true).extensions(extensions).build() + .render(Parser.builder().extensions(extensions).build().parse(request.getMarkdownBody())); + } + else + { + html = "
" + escape(request.getTextBody()) + "
"; + } + } + String text = request.getTextBody(); + if(text == null) + { + text = html.replaceAll("(?is)<(script|style)[^>]*>.*?", "") + .replaceAll("(?i)|", "\n") + .replaceAll("<[^>]+>", "").replace("<", "<").replace(">", ">").replace(""", "\"") + .replace("'", "'").replace(" ", " ").replace("&", "&").strip(); + } + return (new Rendered("
" + html + "
", text)); + } + + /******************************************************************************* + ** Escape plain text for the fallback HTML part. + *******************************************************************************/ + private static String escape(String text) + { + return (text == null ? "" : text.replace("&", "&").replace("<", "<").replace(">", ">")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailCategory.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailCategory.java new file mode 100644 index 0000000..5fc38c5 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailCategory.java @@ -0,0 +1,54 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum; + + +/******************************************************************************* + ** Purpose of a message; transactional and security messages cannot be opted out. + *******************************************************************************/ +public enum EmailCategory implements PossibleValueEnum +{ + TRANSACTIONAL, + NOTIFICATION, + MARKETING, + SECURITY; + + /******************************************************************************* + ** Stable database value. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (name()); + } + + /******************************************************************************* + ** Display label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailRenderer.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailRenderer.java new file mode 100644 index 0000000..220ed5a --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailRenderer.java @@ -0,0 +1,43 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + + + +/******************************************************************************* + ** Replaceable rendering seam for the versioned template and brand renderer. + *******************************************************************************/ +@FunctionalInterface +public interface EmailRenderer +{ + /******************************************************************************* + ** Render already validated ad-hoc content. + *******************************************************************************/ + Rendered render(EmailRequest request); + + /******************************************************************************* + ** Rendered alternatives persisted once for all attempts. + *******************************************************************************/ + record Rendered(String html, String text) + { + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailRequest.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailRequest.java new file mode 100644 index 0000000..f4b5e49 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailRequest.java @@ -0,0 +1,593 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import java.io.Serializable; +import java.time.Instant; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; + + +/******************************************************************************* + ** Synchronous enqueue request. Templates and classified variables arrive in part 2. + *******************************************************************************/ +public class EmailRequest +{ + private List to = new ArrayList<>(); + private List cc = new ArrayList<>(); + private List bcc = new ArrayList<>(); + private String from; + private String replyTo; + private String subject; + private String markdownBody; + private String htmlBody; + private String textBody; + private String providerName; + private String templateName; + private String brandName; + private String locale = "en"; + private EmailCategory category = EmailCategory.TRANSACTIONAL; + private String idempotencyKey; + private Instant sendAfter; + private String linkedTableName; + private String linkedRecordId; + private String senderQbit; + private String senderProcess; + private QBackendTransaction transaction; + private String transactionBackendName; + private Map variables = new LinkedHashMap<>(); + private Map securityValues = new LinkedHashMap<>(); + + /******************************************************************************* + ** Append to recipients. + *******************************************************************************/ + public EmailRequest withTo(String... addresses) + { + to.addAll(List.of(addresses)); + return (this); + } + + /******************************************************************************* + ** Append cc recipients. + *******************************************************************************/ + public EmailRequest withCc(String... addresses) + { + cc.addAll(List.of(addresses)); + return (this); + } + + /******************************************************************************* + ** Append bcc recipients. + *******************************************************************************/ + public EmailRequest withBcc(String... addresses) + { + bcc.addAll(List.of(addresses)); + return (this); + } + + /******************************************************************************* + ** Link the message to an application record. + *******************************************************************************/ + public EmailRequest withLinkedRecord(String table, Object id) + { + linkedTableName = table; + linkedRecordId = id == null ? null : String.valueOf(id); + return (this); + } + + /******************************************************************************* + ** Identify the producing qbit and process. + *******************************************************************************/ + public EmailRequest withSender(String qbit, String process) + { + senderQbit = qbit; + senderProcess = process; + return (this); + } + + /******************************************************************************* + ** Declare the backend of the caller transaction; QBackendTransaction does not expose it. + *******************************************************************************/ + public EmailRequest withTransaction(QBackendTransaction transaction, String backendName) + { + this.transaction = transaction; + this.transactionBackendName = backendName; + return (this); + } + + /******************************************************************************* + ** Accept template variables as a seam; part 1 refuses variable-bearing requests. + *******************************************************************************/ + public EmailRequest withVariable(String name, Serializable value) + { + variables.put(name, value); + return (this); + } + + /******************************************************************************* + ** Accept a record reference without reading or snapshotting the record. + *******************************************************************************/ + public EmailRequest withRecordVariable(String name, String table, Serializable id) + { + variables.put(name, new LinkedHashMap<>(Map.of("table", table, "id", id))); + return (this); + } + + + /******************************************************************************* + ** Getter for to. + *******************************************************************************/ + public List getTo() + { + return (to); + } + + + /******************************************************************************* + ** Fluent setter for to. + *******************************************************************************/ + public EmailRequest withTo(List to) + { + this.to = to; + return (this); + } + + + /******************************************************************************* + ** Getter for cc. + *******************************************************************************/ + public List getCc() + { + return (cc); + } + + + /******************************************************************************* + ** Fluent setter for cc. + *******************************************************************************/ + public EmailRequest withCc(List cc) + { + this.cc = cc; + return (this); + } + + + /******************************************************************************* + ** Getter for bcc. + *******************************************************************************/ + public List getBcc() + { + return (bcc); + } + + + /******************************************************************************* + ** Fluent setter for bcc. + *******************************************************************************/ + public EmailRequest withBcc(List bcc) + { + this.bcc = bcc; + return (this); + } + + + /******************************************************************************* + ** Getter for from. + *******************************************************************************/ + public String getFrom() + { + return (from); + } + + + /******************************************************************************* + ** Fluent setter for from. + *******************************************************************************/ + public EmailRequest withFrom(String from) + { + this.from = from; + return (this); + } + + + /******************************************************************************* + ** Getter for replyTo. + *******************************************************************************/ + public String getReplyTo() + { + return (replyTo); + } + + + /******************************************************************************* + ** Fluent setter for replyTo. + *******************************************************************************/ + public EmailRequest withReplyTo(String replyTo) + { + this.replyTo = replyTo; + return (this); + } + + + /******************************************************************************* + ** Getter for subject. + *******************************************************************************/ + public String getSubject() + { + return (subject); + } + + + /******************************************************************************* + ** Fluent setter for subject. + *******************************************************************************/ + public EmailRequest withSubject(String subject) + { + this.subject = subject; + return (this); + } + + + /******************************************************************************* + ** Getter for markdownBody. + *******************************************************************************/ + public String getMarkdownBody() + { + return (markdownBody); + } + + + /******************************************************************************* + ** Fluent setter for markdownBody. + *******************************************************************************/ + public EmailRequest withMarkdownBody(String markdownBody) + { + this.markdownBody = markdownBody; + return (this); + } + + + /******************************************************************************* + ** Getter for htmlBody. + *******************************************************************************/ + public String getHtmlBody() + { + return (htmlBody); + } + + + /******************************************************************************* + ** Fluent setter for htmlBody. + *******************************************************************************/ + public EmailRequest withHtmlBody(String htmlBody) + { + this.htmlBody = htmlBody; + return (this); + } + + + /******************************************************************************* + ** Getter for textBody. + *******************************************************************************/ + public String getTextBody() + { + return (textBody); + } + + + /******************************************************************************* + ** Fluent setter for textBody. + *******************************************************************************/ + public EmailRequest withTextBody(String textBody) + { + this.textBody = textBody; + return (this); + } + + + /******************************************************************************* + ** Getter for providerName. + *******************************************************************************/ + public String getProviderName() + { + return (providerName); + } + + + /******************************************************************************* + ** Fluent setter for providerName. + *******************************************************************************/ + public EmailRequest withProviderName(String providerName) + { + this.providerName = providerName; + return (this); + } + + + /******************************************************************************* + ** Getter for templateName. + *******************************************************************************/ + public String getTemplateName() + { + return (templateName); + } + + + /******************************************************************************* + ** Fluent setter for templateName. + *******************************************************************************/ + public EmailRequest withTemplateName(String templateName) + { + this.templateName = templateName; + return (this); + } + + + /******************************************************************************* + ** Getter for brandName. + *******************************************************************************/ + public String getBrandName() + { + return (brandName); + } + + + /******************************************************************************* + ** Fluent setter for brandName. + *******************************************************************************/ + public EmailRequest withBrandName(String brandName) + { + this.brandName = brandName; + return (this); + } + + + /******************************************************************************* + ** Getter for locale. + *******************************************************************************/ + public String getLocale() + { + return (locale); + } + + + /******************************************************************************* + ** Fluent setter for locale. + *******************************************************************************/ + public EmailRequest withLocale(String locale) + { + this.locale = locale; + return (this); + } + + + /******************************************************************************* + ** Getter for category. + *******************************************************************************/ + public EmailCategory getCategory() + { + return (category); + } + + + /******************************************************************************* + ** Fluent setter for category. + *******************************************************************************/ + public EmailRequest withCategory(EmailCategory category) + { + this.category = category; + return (this); + } + + + /******************************************************************************* + ** Getter for idempotencyKey. + *******************************************************************************/ + public String getIdempotencyKey() + { + return (idempotencyKey); + } + + + /******************************************************************************* + ** Fluent setter for idempotencyKey. + *******************************************************************************/ + public EmailRequest withIdempotencyKey(String idempotencyKey) + { + this.idempotencyKey = idempotencyKey; + return (this); + } + + + /******************************************************************************* + ** Getter for sendAfter. + *******************************************************************************/ + public Instant getSendAfter() + { + return (sendAfter); + } + + + /******************************************************************************* + ** Fluent setter for sendAfter. + *******************************************************************************/ + public EmailRequest withSendAfter(Instant sendAfter) + { + this.sendAfter = sendAfter; + return (this); + } + + + /******************************************************************************* + ** Getter for linkedTableName. + *******************************************************************************/ + public String getLinkedTableName() + { + return (linkedTableName); + } + + + /******************************************************************************* + ** Fluent setter for linkedTableName. + *******************************************************************************/ + public EmailRequest withLinkedTableName(String linkedTableName) + { + this.linkedTableName = linkedTableName; + return (this); + } + + + /******************************************************************************* + ** Getter for linkedRecordId. + *******************************************************************************/ + public String getLinkedRecordId() + { + return (linkedRecordId); + } + + + /******************************************************************************* + ** Fluent setter for linkedRecordId. + *******************************************************************************/ + public EmailRequest withLinkedRecordId(String linkedRecordId) + { + this.linkedRecordId = linkedRecordId; + return (this); + } + + + /******************************************************************************* + ** Getter for senderQbit. + *******************************************************************************/ + public String getSenderQbit() + { + return (senderQbit); + } + + + /******************************************************************************* + ** Fluent setter for senderQbit. + *******************************************************************************/ + public EmailRequest withSenderQbit(String senderQbit) + { + this.senderQbit = senderQbit; + return (this); + } + + + /******************************************************************************* + ** Getter for senderProcess. + *******************************************************************************/ + public String getSenderProcess() + { + return (senderProcess); + } + + + /******************************************************************************* + ** Fluent setter for senderProcess. + *******************************************************************************/ + public EmailRequest withSenderProcess(String senderProcess) + { + this.senderProcess = senderProcess; + return (this); + } + + + /******************************************************************************* + ** Getter for transaction. + *******************************************************************************/ + public QBackendTransaction getTransaction() + { + return (transaction); + } + + + /******************************************************************************* + ** Fluent setter for transaction. + *******************************************************************************/ + public EmailRequest withTransaction(QBackendTransaction transaction) + { + this.transaction = transaction; + return (this); + } + + + /******************************************************************************* + ** Getter for transactionBackendName. + *******************************************************************************/ + public String getTransactionBackendName() + { + return (transactionBackendName); + } + + + /******************************************************************************* + ** Fluent setter for transactionBackendName. + *******************************************************************************/ + public EmailRequest withTransactionBackendName(String transactionBackendName) + { + this.transactionBackendName = transactionBackendName; + return (this); + } + + + /******************************************************************************* + ** Getter for variables. + *******************************************************************************/ + public Map getVariables() + { + return (variables); + } + + + /******************************************************************************* + ** Fluent setter for variables. + *******************************************************************************/ + public EmailRequest withVariables(Map variables) + { + this.variables = variables; + return (this); + } + + + /******************************************************************************* + ** Getter for securityValues. + *******************************************************************************/ + public Map getSecurityValues() + { + return (securityValues); + } + + + /******************************************************************************* + ** Fluent setter for securityValues. + *******************************************************************************/ + public EmailRequest withSecurityValues(Map securityValues) + { + this.securityValues = securityValues; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailSendResult.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailSendResult.java new file mode 100644 index 0000000..86c622c --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailSendResult.java @@ -0,0 +1,42 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import java.util.List; + + +/******************************************************************************* + ** Queue result. Cross-backend deferred inserts initially have no message id. + *******************************************************************************/ +public record EmailSendResult(Integer messageId, Status status, List reasons) +{ + /******************************************************************************* + ** Enqueue outcomes. + *******************************************************************************/ + public enum Status + { + QUEUED, + DUPLICATE, + SUPPRESSED, + REFUSED + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java new file mode 100644 index 0000000..620b923 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java @@ -0,0 +1,384 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HexFormat; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Locale; +import java.util.Set; +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qbits.email.EmailRuntime; +import com.kingsrook.qbits.email.metadata.EmailTableCustomizers; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.permissions.EmailPermissionChecker; +import com.kingsrook.qbits.email.permissions.EmailPermissions; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.messaging.SendMessageInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import jakarta.mail.internet.InternetAddress; + + +/******************************************************************************* + ** Validated transactional outbox API. This class never performs network delivery. + *******************************************************************************/ +public final class EmailService +{ + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailService() + { + } + + /******************************************************************************* + ** Queue with the part 1 renderer. + *******************************************************************************/ + public static EmailSendResult send(EmailRequest request) throws QException + { + return (send(request, new AdHocEmailRenderer())); + } + + /******************************************************************************* + ** Queue with a renderer supplied by the host or a later phase. + *******************************************************************************/ + public static EmailSendResult send(EmailRequest request, EmailRenderer renderer) throws QException + { + EmailQBitConfig config = EmailRuntime.config(); + if(config == null) + { + return (refused("Email qbit is not installed")); + } + if(CorePermissionEnforcer.shouldEnforce(new SendMessageInput()) && !EmailPermissionChecker.has(EmailPermissions.SEND)) + { + return (refused("Sending email needs email.send")); + } + if(request == null) + { + return (refused("An email request is required")); + } + if(request.getTemplateName() != null) + { + return (refused("templates arrive in part 2")); + } + if(!request.getVariables().isEmpty() || request.getBrandName() != null) + { + return (refused("Template variables and branding arrive in part 2")); + } + if(request.getSubject() == null || request.getSubject().contains("\r") || request.getSubject().contains("\n") + || (request.getMarkdownBody() == null && request.getHtmlBody() == null && request.getTextBody() == null)) + { + return (refused("A subject and message body are required; subject must be one line")); + } + EmailCategory category = request.getCategory() == null ? EmailCategory.TRANSACTIONAL : request.getCategory(); + QBackendTransaction transaction = request.getTransaction(); + boolean sameBackend = transaction == null || request.getTransactionBackendName() == null || config.getBackendName().equals(request.getTransactionBackendName()); + QBackendTransaction queryTransaction = sameBackend ? transaction : null; + QRecord duplicate = duplicate(request.getIdempotencyKey(), queryTransaction); + if(duplicate != null) + { + return (new EmailSendResult(duplicate.getValueInteger("id"), EmailSendResult.Status.DUPLICATE, List.of())); + } + String providerName = request.getProviderName() == null ? config.getDefaultProviderName() : request.getProviderName(); + QQueryFilter providerFilter = new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true)); + providerFilter.withCriteria(new QFilterCriteria(providerName == null ? "isDefault" : "name", QCriteriaOperator.EQUALS, providerName == null ? true : providerName)); + List providers = SystemWrites.query(EmailTables.PROVIDER, providerFilter, queryTransaction); + if(providers.size() != 1) + { + return (refused("An enabled named or default provider is required")); + } + QRecord provider = providers.get(0); + List providerErrors = EmailTableCustomizers.validateProvider(provider, config); + if(!providerErrors.isEmpty()) + { + return (new EmailSendResult(null, EmailSendResult.Status.REFUSED, providerErrors)); + } + List to; + List cc; + List bcc; + String from; + String replyTo; + try + { + to = addresses(request.getTo()); + cc = addresses(request.getCc()); + bcc = addresses(request.getBcc()); + String sender = request.getFrom() == null ? provider.getValueString("fromAddress") : request.getFrom(); + from = single(sender); + if(request.getFrom() == null && provider.getValueString("fromName") != null) + { + from = new InternetAddress(new InternetAddress(from).getAddress(), provider.getValueString("fromName"), "UTF-8").toString(); + } + String reply = request.getReplyTo() == null ? provider.getValueString("replyTo") : request.getReplyTo(); + replyTo = reply == null ? null : single(reply); + String domain = provider.getValueString("sendingDomain"); + String address = new InternetAddress(from).getAddress(); + if(domain != null && !domain.equalsIgnoreCase(address.substring(address.lastIndexOf('@') + 1))) + { + return (refused("From address domain must match sending domain")); + } + } + catch(Exception e) + { + return (refused("One or more email addresses are invalid")); + } + if(to.isEmpty() && cc.isEmpty() && bcc.isEmpty()) + { + return (refused("At least one recipient is required")); + } + Set blocked = suppressed(category, queryTransaction); + List dropped = new ArrayList<>(); + for(List addresses : List.of(to, cc, bcc)) + { + addresses.removeIf(address -> + { + try + { + if(blocked.contains(new InternetAddress(address).getAddress().toLowerCase(Locale.ROOT))) + { + dropped.add(address); + return (true); + } + } + catch(Exception e) + { + // Addresses were validated above. + } + return (false); + }); + } + boolean allSuppressed = to.isEmpty() && cc.isEmpty() && bcc.isEmpty(); + EmailRenderer.Rendered rendered = renderer.render(request); + Instant now = EmailRuntime.clock().instant(); + QRecord row = new QRecord().withValue("providerId", provider.getValue("id")) + .withValue("status", allSuppressed ? "SUPPRESSED" : "QUEUED").withValue("finishedDate", allSuppressed ? now : null).withValue("attemptCount", 0) + .withValue("toAddresses", String.join(",", to)).withValue("ccAddresses", String.join(",", cc)).withValue("bccAddresses", String.join(",", bcc)) + .withValue("fromAddress", from).withValue("replyTo", replyTo).withValue("subject", request.getSubject()) + .withValue("htmlBody", rendered.html()).withValue("textBody", rendered.text()).withValue("bodySha256", digest(rendered)) + .withValue("category", category.name()).withValue("locale", request.getLocale()).withValue("idempotencyKey", request.getIdempotencyKey()) + .withValue("linkedTableName", request.getLinkedTableName()).withValue("linkedRecordId", request.getLinkedRecordId()) + .withValue("senderQbit", request.getSenderQbit()).withValue("senderProcess", request.getSenderProcess()) + .withValue("sendAfter", request.getSendAfter()).withValue("nextAttemptDate", request.getSendAfter() == null ? now : request.getSendAfter()) + .withValue("suppressedAddresses", String.join(",", dropped)); + for(var field : config.getSecurityFields()) + { + row.setValue(field.getName(), request.getSecurityValues().get(field.getName())); + } + if(transaction != null && !sameBackend) + { + transaction.addAfterCommitCallback(() -> + { + try + { + if(insert(row, null).status() == EmailSendResult.Status.REFUSED) + { + throw (new QException("Email after-commit enqueue failed")); + } + } + catch(QException e) + { + // Core logs callback failures; propagate only a sanitized message. + throw (new IllegalStateException("Email after-commit enqueue failed")); + } + }); + return (new EmailSendResult(null, allSuppressed ? EmailSendResult.Status.SUPPRESSED : EmailSendResult.Status.QUEUED, + List.of("Enqueue deferred until caller transaction commits"))); + } + return (insert(row, transaction)); + } + + /******************************************************************************* + ** Insert once; the database unique key arbitrates concurrent requests. + *******************************************************************************/ + private static EmailSendResult insert(QRecord row, QBackendTransaction caller) throws QException + { + if(caller != null) + { + return (insertInTransaction(row, caller)); + } + try(QBackendTransaction transaction = QBackendTransaction.openFor(new InsertInput(EmailTables.MESSAGE))) + { + EmailSendResult result = insertInTransaction(row, transaction); + if(result.status() == EmailSendResult.Status.REFUSED) + { + transaction.rollback(); + } + else + { + transaction.commit(); + return (result); + } + } + QRecord existing = duplicate(row.getValueString("idempotencyKey"), null); + return (existing == null ? refused("Could not queue the email message") : new EmailSendResult(existing.getValueInteger("id"), EmailSendResult.Status.DUPLICATE, List.of())); + } + + /******************************************************************************* + ** Insert using the caller transaction, returning only safe diagnostics. + *******************************************************************************/ + private static EmailSendResult insertInTransaction(QRecord row, QBackendTransaction transaction) throws QException + { + // A shared provider-row gate prevents a duplicate-key failure from aborting + // a caller-owned SQL transaction. The unique index remains the final arbiter. + if(row.getValueString("idempotencyKey") != null) + { + List providers = SystemWrites.query(EmailTables.PROVIDER, new QQueryFilter().withOrderBy(new QFilterOrderBy("id")).withLimit(1), transaction); + if(!providers.isEmpty()) + { + EmailStore.lockProvider(providers.get(0).getValueInteger("id"), transaction); + } + } + QRecord existing = duplicate(row.getValueString("idempotencyKey"), transaction); + if(existing != null) + { + return (new EmailSendResult(existing.getValueInteger("id"), EmailSendResult.Status.DUPLICATE, List.of())); + } + QRecord inserted; + try + { + inserted = SystemWrites.insert(EmailTables.MESSAGE, List.of(row), transaction).getRecords().get(0); + } + catch(QException e) + { + return (refused("Could not queue the email message")); + } + if(inserted.getErrors() != null && !inserted.getErrors().isEmpty()) + { + return (refused("Could not queue the email message")); + } + return (new EmailSendResult(inserted.getValueInteger("id"), "SUPPRESSED".equals(row.getValueString("status")) + ? EmailSendResult.Status.SUPPRESSED : EmailSendResult.Status.QUEUED, List.of())); + } + + /******************************************************************************* + ** Look up an idempotency key in the appropriate transaction. + *******************************************************************************/ + private static QRecord duplicate(String key, QBackendTransaction transaction) throws QException + { + if(key == null) + { + return (null); + } + List rows = SystemWrites.query(EmailTables.MESSAGE, new QQueryFilter(new QFilterCriteria("idempotencyKey", QCriteriaOperator.EQUALS, key)), transaction); + return (rows.isEmpty() ? null : rows.get(0)); + } + + /******************************************************************************* + ** Suppression matching; opt-outs never suppress transactional or security mail. + *******************************************************************************/ + private static Set suppressed(EmailCategory category, QBackendTransaction transaction) throws QException + { + Set result = new LinkedHashSet<>(); + boolean optOut = category == EmailCategory.NOTIFICATION || category == EmailCategory.MARKETING; + for(QRecord row : SystemWrites.query(EmailTables.SUPPRESSION, new QQueryFilter(), transaction)) + { + Instant expires = row.getValueInstant("expiresDate"); + if(expires != null && !expires.isAfter(EmailRuntime.clock().instant())) + { + continue; + } + if(!optOut && "UNSUBSCRIBE".equals(row.getValueString("reason"))) + { + continue; + } + if("ALL".equals(row.getValueString("scope")) || (optOut && category.name().equals(row.getValueString("scope")))) + { + result.add(row.getValueString("address").toLowerCase(Locale.ROOT)); + } + } + return (result); + } + + /******************************************************************************* + ** Strict address parsing, rejecting header injection and local-only addresses. + *******************************************************************************/ + public static List addresses(List input) throws Exception + { + List result = new ArrayList<>(); + for(String value : input) + { + if(value == null || value.contains("\r") || value.contains("\n")) + { + throw (new IllegalArgumentException("Invalid address")); + } + for(InternetAddress address : InternetAddress.parse(value, true)) + { + address.validate(); + if(address.isGroup() || !address.getAddress().contains("@")) + { + throw (new IllegalArgumentException("Invalid address")); + } + result.add(address.toString()); + } + } + return (result); + } + + /******************************************************************************* + ** One sender or reply-to mailbox. + *******************************************************************************/ + private static String single(String value) throws Exception + { + List addresses = addresses(List.of(value)); + if(addresses.size() != 1) + { + throw (new IllegalArgumentException("One address required")); + } + return (addresses.get(0)); + } + + /******************************************************************************* + ** Keep a digest after body retention clears both alternatives. + *******************************************************************************/ + private static String digest(EmailRenderer.Rendered rendered) + { + try + { + return (HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest((rendered.html() + "\u0000" + rendered.text()).getBytes(StandardCharsets.UTF_8)))); + } + catch(java.security.NoSuchAlgorithmException e) + { + throw (new IllegalStateException("SHA-256 unavailable")); + } + } + + /******************************************************************************* + ** Refusal with no stored row. + *******************************************************************************/ + private static EmailSendResult refused(String reason) + { + return (new EmailSendResult(null, EmailSendResult.Status.REFUSED, List.of(reason))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/permissions/EmailPermissionChecker.java b/src/main/java/com/kingsrook/qbits/email/permissions/EmailPermissionChecker.java new file mode 100644 index 0000000..f457249 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/permissions/EmailPermissionChecker.java @@ -0,0 +1,128 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.permissions; + + +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.processes.EmailProcesses; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; + + +/******************************************************************************* + ** Maps every action on the qbit's tables, processes, widgets and app to one + ** of {@code email.view}, {@code email.manage} or {@code email.admin} + ** (admin implies manage, manage implies view). + ** + **

Deliveries and attempts are written only by the qbit (its own writes run + ** in an authorized scope); people and the API can read them and act on them + ** through the processes.

+ *******************************************************************************/ +public class EmailPermissionChecker implements CustomPermissionChecker +{ + /******************************************************************************* + ** Check, or throw. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput actionInput, MetaDataWithPermissionRules metaData) throws QPermissionDeniedException + { + String needed = required(actionInput, metaData); + if(needed == null) + { + throw (new QPermissionDeniedException("Email messages and attempts are written only by the email qbit. Use Send Now, Resend or Cancel.")); + } + if(!has(needed)) + { + throw (new QPermissionDeniedException("This needs the " + needed + " permission.")); + } + } + + + + /******************************************************************************* + ** The permission an action needs (null: nobody may do it directly). + *******************************************************************************/ + static String required(AbstractActionInput actionInput, MetaDataWithPermissionRules metaData) + { + boolean write = actionInput instanceof InsertInput || actionInput instanceof UpdateInput || actionInput instanceof DeleteInput; + if(metaData instanceof QTableMetaData table) + { + if(EmailTables.PROVIDER.equals(table.getName())) + { + return (EmailPermissions.ADMIN); + } + if(!write) + { + return (EmailPermissions.VIEW); + } + if(EmailTables.SYSTEM_WRITTEN.contains(table.getName())) + { + return (null); + } + if(EmailTables.PROVIDER.equals(table.getName())) + { + return (EmailPermissions.ADMIN); + } + return (EmailPermissions.MANAGE); + } + if(metaData instanceof QProcessMetaData process) + { + return (EmailProcesses.permissionFor(process.getName())); + } + return (EmailPermissions.VIEW); + } + + + + /******************************************************************************* + ** True when the current session holds the permission (or a higher one). + *******************************************************************************/ + public static boolean has(String permission) + { + QSession session = QContext.getQSession(); + if(session == null) + { + return (false); + } + if(session.hasPermission(EmailPermissions.ADMIN)) + { + return (true); + } + if(EmailPermissions.ADMIN.equals(permission)) + { + return (false); + } + if(session.hasPermission(EmailPermissions.MANAGE)) + { + return (true); + } + return (session.hasPermission(permission)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/permissions/EmailPermissions.java b/src/main/java/com/kingsrook/qbits/email/permissions/EmailPermissions.java new file mode 100644 index 0000000..aaf00fb --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/permissions/EmailPermissions.java @@ -0,0 +1,49 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.permissions; + + +import java.util.List; + + +/******************************************************************************* + ** Email permissions: view, send, manage and admin, plus the derived app access. + ** Administrators imply all permissions; managers may also send and view. + *******************************************************************************/ +public final class EmailPermissions +{ + public static final String SEND = "email.send"; + public static final String VIEW = "email.view"; + public static final String MANAGE = "email.manage"; + public static final String ADMIN = "email.admin"; + public static final String HAS_ACCESS = "email.hasAccess"; + + public static final List ALL = List.of(VIEW, SEND, MANAGE, ADMIN, HAS_ACCESS); + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailPermissions() + { + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/processes/EmailProcesses.java b/src/main/java/com/kingsrook/qbits/email/processes/EmailProcesses.java new file mode 100644 index 0000000..0d628fb --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/processes/EmailProcesses.java @@ -0,0 +1,439 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.processes; + + +import java.io.Serializable; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qbits.email.EmailRuntime; +import com.kingsrook.qbits.email.delivery.EmailDispatcher; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.outbox.EmailCategory; +import com.kingsrook.qbits.email.outbox.EmailRenderer; +import com.kingsrook.qbits.email.outbox.EmailRequest; +import com.kingsrook.qbits.email.outbox.EmailSendResult; +import com.kingsrook.qbits.email.outbox.EmailService; +import com.kingsrook.qbits.email.permissions.EmailPermissions; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.processes.Status; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; + + +/******************************************************************************* + ** Standard Review/Result record actions and hidden scheduled maintenance. + *******************************************************************************/ +public final class EmailProcesses +{ + public static final String RESEND = "email_resend"; + public static final String CANCEL = "email_cancel"; + public static final String SEND_NOW = "email_send_now"; + public static final String SEND_TEST = "email_send_test"; + public static final String SEND_TEMPLATED = "email_send_templated"; + public static final String DISPATCH = "email_dispatch"; + public static final String MAINTAIN = "email_maintain"; + public static final String RETENTION = "email_retention"; + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailProcesses() + { + } + + /******************************************************************************* + ** Required action permission. + *******************************************************************************/ + public static String permissionFor(String name) + { + return (switch(name) + { + case RESEND, CANCEL, SEND_NOW -> EmailPermissions.MANAGE; + case SEND_TEMPLATED -> EmailPermissions.SEND; + default -> EmailPermissions.ADMIN; + }); + } + + /******************************************************************************* + ** All actions and schedules; none is an app child. + *******************************************************************************/ + public static List processes(EmailQBitConfig config) + { + QProcessMetaData test = action(SEND_TEST, "Send Test Email", EmailTables.PROVIDER); + test.withStep(0, new QFrontendStepMetaData().withName("input").withLabel("Recipient") + .withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM)) + .withFormField(new QFieldMetaData("toAddress", QFieldType.STRING).withLabel("To address").withIsRequired(true))); + QFunctionInputMetaData inputs = new QFunctionInputMetaData(); + for(String field : List.of("templateName", "to", "cc", "bcc", "replyTo", "subject", "markdownBody", "htmlBody", "textBody", "providerName", "brandName", + "category", "variablesJson", "linkedTableName", "linkedRecordId", "idempotencyKey")) + { + inputs.withField(new QFieldMetaData(field, QFieldType.STRING)); + } + QProcessMetaData send = new QProcessMetaData().withName(SEND_TEMPLATED).withLabel("Send Email").withIsHidden(true) + .withPermissionRules(EmailTables.permissionRules()) + .withStep(new QBackendStepMetaData().withName("send").withCode(new QCodeReference(SendStep.class)).withInputData(inputs)); + return (List.of(action(RESEND, "Resend", EmailTables.MESSAGE), action(CANCEL, "Cancel", EmailTables.MESSAGE), + action(SEND_NOW, "Send Now", EmailTables.MESSAGE), test, send, + scheduled(config, DISPATCH, "Dispatch Email", config.getDispatchIntervalSeconds()), + scheduled(config, MAINTAIN, "Maintain Email", 300), scheduled(config, RETENTION, "Email Retention", 86400))); + } + + /******************************************************************************* + ** A standard streamed-ETL action. + *******************************************************************************/ + private static QProcessMetaData action(String name, String label, String table) + { + Map values = new HashMap<>(); + values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, table); + values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, table); + values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_FETCH_HEAVY_FIELDS, true); + values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, label + " for the selected records."); + return (StreamedETLWithFrontendProcess.defineProcessMetaData(ExtractViaQueryStep.class, ActionTransform.class, ActionLoad.class, values) + .withName(name).withLabel(label).withIcon(new QIcon("mail")).withTableName(table).withPermissionRules(EmailTables.permissionRules())); + } + + /******************************************************************************* + ** Hidden backend schedule; without a scheduler manual dispatch remains possible. + *******************************************************************************/ + private static QProcessMetaData scheduled(EmailQBitConfig config, String name, String label, int seconds) + { + QProcessMetaData process = new QProcessMetaData().withName(name).withLabel(label).withIsHidden(true) + .withPermissionRules(EmailTables.permissionRules()).withStep(new QBackendStepMetaData().withName("run").withCode(new QCodeReference(ScheduledStep.class))); + if(config.getSchedulerName() != null) + { + process.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(seconds).withInitialDelaySeconds(Math.min(seconds, 30))); + } + return (process); + } + + /******************************************************************************* + ** Whether a selected row can participate in the action. + *******************************************************************************/ + private static boolean eligible(String process, QRecord row) + { + if(SEND_TEST.equals(process)) + { + return (Boolean.TRUE.equals(row.getValueBoolean("enabled"))); + } + String status = row.getValueString("status"); + if(RESEND.equals(process)) + { + return (List.of("DEAD_LETTER", "BOUNCED", "COMPLAINED").contains(status) + && (row.getValueString("htmlBody") != null || row.getValueString("textBody") != null)); + } + return (EmailDispatcher.DUE.contains(status)); + } + + /******************************************************************************* + ** Review screen validation without side effects. + *******************************************************************************/ + public static class ActionTransform extends AbstractTransformStep implements ProcessSummaryProviderInterface + { + private final ArrayList lines = new ArrayList<>(); + + /******************************************************************************* + ** Filter rows and explain skipped records. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) + { + for(QRecord row : input.getRecords()) + { + boolean valid = eligible(input.getProcessName(), row); + lines.add(new ProcessSummaryLine(valid ? Status.OK : Status.INFO, 1, valid ? "record will be processed" : "record skipped: status or retained content does not permit this action")); + if(valid) + { + output.getRecords().add(row); + } + } + } + + /******************************************************************************* + ** Review lines. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + return (lines); + } + } + + /******************************************************************************* + ** Execute only after review; re-read rows to avoid stale actions. + *******************************************************************************/ + public static class ActionLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface + { + private final ArrayList lines = new ArrayList<>(); + + /******************************************************************************* + ** Perform and report one selected row at a time. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + String process = input.getProcessName(); + for(QRecord row : input.getRecords()) + { + Integer id = row.getValueInteger("id"); + QRecord current = EmailStore.get(SEND_TEST.equals(process) ? EmailTables.PROVIDER : EmailTables.MESSAGE, id, null); + String result = "Skipped"; + if(current != null && eligible(process, current)) + { + result = switch(process) + { + case SEND_NOW -> new EmailDispatcher().sendNow(id).name(); + case CANCEL -> cancel(id) ? "CANCELLED" : "Skipped"; + case RESEND -> resend(current).status().name(); + case SEND_TEST -> sendTest(current, input.getValueString("toAddress")); + default -> "Skipped"; + }; + } + lines.add(new ProcessSummaryLine(Status.INFO, 1, "record: " + result)); + output.getRecords().add(row); + } + } + + /******************************************************************************* + ** Result screen lines. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + return (lines); + } + } + + /******************************************************************************* + ** Cancel under the same database gate as claims. + *******************************************************************************/ + public static boolean cancel(Integer id) throws QException + { + QRecord row = EmailStore.get(EmailTables.MESSAGE, id, null); + if(row == null) + { + return (false); + } + try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER))) + { + EmailStore.lockProvider(row.getValueInteger("providerId"), tx); + row = EmailStore.get(EmailTables.MESSAGE, id, tx); + if(!EmailDispatcher.DUE.contains(row.getValueString("status"))) + { + return (false); + } + EmailStore.update(EmailTables.MESSAGE, new QRecord().withValue("id", id).withValue("status", "CANCELLED").withValue("finishedDate", EmailRuntime.clock().instant()).withValue("nextAttemptDate", null), tx); + tx.commit(); + return (true); + } + } + + /******************************************************************************* + ** A fresh, linked message through suppression and provider validation again. + *******************************************************************************/ + private static EmailSendResult resend(QRecord row) throws QException + { + QRecord provider = EmailStore.get(EmailTables.PROVIDER, row.getValueInteger("providerId"), null); + EmailRequest request = new EmailRequest().withSubject(row.getValueString("subject")).withHtmlBody(row.getValueString("htmlBody")) + .withTextBody(row.getValueString("textBody")).withFrom(row.getValueString("fromAddress")).withReplyTo(row.getValueString("replyTo")) + .withProviderName(provider.getValueString("name")).withCategory(EmailCategory.valueOf(row.getValueString("category"))) + .withLinkedRecord(EmailTables.MESSAGE, row.getValue("id")).withSender("email", RESEND); + for(String field : List.of("toAddresses", "ccAddresses", "bccAddresses")) + { + String value = row.getValueString(field); + if(value != null && !value.isBlank()) + { + switch(field) + { + case "toAddresses" -> request.withTo(value); + case "ccAddresses" -> request.withCc(value); + default -> request.withBcc(value); + } + } + } + for(var field : EmailRuntime.config().getSecurityFields()) + { + request.getSecurityValues().put(field.getName(), row.getValue(field.getName())); + } + return (EmailService.send(request, ignored -> new EmailRenderer.Rendered(row.getValueString("htmlBody"), row.getValueString("textBody")))); + } + + /******************************************************************************* + ** Queue and dispatch a fixed provider test message. + *******************************************************************************/ + private static String sendTest(QRecord provider, String to) throws QException + { + if(to == null) + { + return ("REFUSED: a to address is required"); + } + EmailSendResult result = EmailService.send(new EmailRequest().withProviderName(provider.getValueString("name")).withTo(to) + .withSubject("Email delivery test").withMarkdownBody("This is a test of your email provider.").withSender("email", SEND_TEST)); + return (result.status() == EmailSendResult.Status.QUEUED ? new EmailDispatcher().sendNow(result.messageId()).name() : result.status().name()); + } + + /******************************************************************************* + ** Backend sending API with inputs declared as QQQ fields. + *******************************************************************************/ + public static class SendStep implements BackendStep + { + /******************************************************************************* + ** Map process inputs to the same service API. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + EmailRequest request = new EmailRequest().withTemplateName(input.getValueString("templateName")) + .withSubject(input.getValueString("subject")).withMarkdownBody(input.getValueString("markdownBody")) + .withHtmlBody(input.getValueString("htmlBody")).withTextBody(input.getValueString("textBody")) + .withReplyTo(input.getValueString("replyTo")).withProviderName(input.getValueString("providerName")) + .withBrandName(input.getValueString("brandName")).withIdempotencyKey(input.getValueString("idempotencyKey")) + .withLinkedRecord(input.getValueString("linkedTableName"), input.getValueString("linkedRecordId")); + if(input.getValueString("to") != null) + { + request.withTo(input.getValueString("to")); + } + if(input.getValueString("cc") != null) + { + request.withCc(input.getValueString("cc")); + } + if(input.getValueString("bcc") != null) + { + request.withBcc(input.getValueString("bcc")); + } + if(input.getValueString("variablesJson") != null) + { + request.withVariable("variablesJson", input.getValueString("variablesJson")); + } + if(input.getValueString("category") != null) + { + try + { + request.withCategory(EmailCategory.valueOf(input.getValueString("category"))); + } + catch(IllegalArgumentException e) + { + throw (new QException("Unknown email category")); + } + } + EmailSendResult result = EmailService.send(request); + output.addValue("messageId", result.messageId()); + output.addValue("status", result.status().name()); + output.addValue("reasons", String.join("; ", result.reasons())); + } + } + + /******************************************************************************* + ** Scheduler entry point. + *******************************************************************************/ + public static class ScheduledStep implements BackendStep + { + /******************************************************************************* + ** Dispatch, recover leases and probe health, or retain final messages. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + switch(input.getProcessName()) + { + case RETENTION -> retain(); + case MAINTAIN -> new EmailDispatcher().maintain(); + default -> new EmailDispatcher().runOnce(); + } + } + } + + /******************************************************************************* + ** Clear retained bodies and optionally delete final messages and their attempts. + *******************************************************************************/ + public static void retain() throws QException + { + EmailQBitConfig config = EmailRuntime.config(); + Instant now = EmailRuntime.clock().instant(); + for(QRecord row : SystemWrites.query(EmailTables.MESSAGE, new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.IN, new ArrayList<>(EmailDispatcher.FINAL))))) + { + Instant finalDate = row.getValueInstant("finishedDate"); + if(finalDate == null) + { + finalDate = row.getValueInstant("sentDate"); + } + if(finalDate == null) + { + finalDate = row.getValueInstant("deadLetterDate"); + } + if(finalDate == null) + { + finalDate = row.getValueInstant("createDate"); + } + if(finalDate == null) + { + continue; + } + if(config.getMessageRetentionDays() > 0 && finalDate.isBefore(now.minus(Duration.ofDays(config.getMessageRetentionDays())))) + { + try(QBackendTransaction transaction = QBackendTransaction.openFor(new UpdateInput(EmailTables.MESSAGE))) + { + SystemWrites.delete(EmailTables.ATTEMPT, EmailStore.equal("messageId", row.getValueInteger("id")), transaction); + SystemWrites.delete(EmailTables.MESSAGE, EmailStore.equal("id", row.getValueInteger("id")), transaction); + transaction.commit(); + } + } + else if(finalDate.isBefore(now.minus(Duration.ofDays(config.getBodyRetentionDays()))) && (row.getValue("htmlBody") != null || row.getValue("textBody") != null)) + { + SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", row.getValue("id")).withValue("htmlBody", null).withValue("textBody", null)); + } + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/support/EmailStore.java b/src/main/java/com/kingsrook/qbits/email/support/EmailStore.java new file mode 100644 index 0000000..b46ecba --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/support/EmailStore.java @@ -0,0 +1,88 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.support; + + +import java.io.Serializable; +import java.util.List; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + + +/******************************************************************************* + ** Small database helpers shared by dispatcher and actions. + *******************************************************************************/ +public final class EmailStore +{ + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailStore() + { + } + + /******************************************************************************* + ** Equality filter. + *******************************************************************************/ + public static QQueryFilter equal(String field, Serializable value) + { + return (new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.EQUALS, value))); + } + + /******************************************************************************* + ** Read a row in a transaction. + *******************************************************************************/ + public static QRecord get(String table, Integer id, QBackendTransaction transaction) throws QException + { + List rows = SystemWrites.query(table, equal("id", id), transaction); + return (rows.isEmpty() ? null : rows.get(0)); + } + + /******************************************************************************* + ** Check write errors without exposing record contents. + *******************************************************************************/ + public static void update(String table, QRecord row, QBackendTransaction transaction) throws QException + { + List rows = SystemWrites.update(table, List.of(row), transaction); + if(rows.isEmpty() || (rows.get(0).getErrors() != null && !rows.get(0).getErrors().isEmpty())) + { + throw (new QException("Email database update failed")); + } + } + + /******************************************************************************* + ** Serialize provider rate checks and claims using the database row lock. + *******************************************************************************/ + public static QRecord lockProvider(Integer id, QBackendTransaction transaction) throws QException + { + if(get(EmailTables.PROVIDER, id, transaction) == null) + { + return (null); + } + update(EmailTables.PROVIDER, new QRecord().withValue("id", id).withValue("dispatchLock", 0), transaction); + return (get(EmailTables.PROVIDER, id, transaction)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/support/SystemWrites.java b/src/main/java/com/kingsrook/qbits/email/support/SystemWrites.java new file mode 100644 index 0000000..1744b85 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/support/SystemWrites.java @@ -0,0 +1,207 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.support; + + +import java.util.List; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer; +import com.kingsrook.qqq.backend.core.actions.tables.CountAction; +import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + + +/******************************************************************************* + ** The qbit's own reads and writes (the delivery engine, the outbox, health + ** and rotation). They run in core's authorized scope, so they work with core + ** permission enforcement on, and they mark the thread so the tables' + ** customizers can tell a qbit write from a person's or the API's. + *******************************************************************************/ +public final class SystemWrites +{ + private static final ThreadLocal DEPTH = ThreadLocal.withInitial(() -> 0); + + + + /******************************************************************************* + ** Work that may throw. + *******************************************************************************/ + @FunctionalInterface + public interface Work + { + /******************************************************************************* + ** Do it. + *******************************************************************************/ + T run() throws QException; + } + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private SystemWrites() + { + } + + + + /******************************************************************************* + ** Run work as the qbit. + *******************************************************************************/ + public static T call(Work work) throws QException + { + DEPTH.set(DEPTH.get() + 1); + try(CorePermissionEnforcer.AuthorizedScope ignored = CorePermissionEnforcer.openAuthorizedScope()) + { + return (work.run()); + } + finally + { + DEPTH.set(DEPTH.get() - 1); + } + } + + + + /******************************************************************************* + ** True while the qbit itself is writing on this thread. + *******************************************************************************/ + public static boolean active() + { + return (DEPTH.get() > 0); + } + + + + /******************************************************************************* + ** Insert, returning the output (records may carry errors). + *******************************************************************************/ + public static InsertOutput insert(String table, List records, QBackendTransaction transaction) throws QException + { + return (call(() -> new InsertAction().execute(new InsertInput(table).withRecords(records).withTransaction(transaction)))); + } + + + + /******************************************************************************* + ** Insert one record, returning it (with errors if it failed). + *******************************************************************************/ + public static QRecord insertOne(String table, QRecord record) throws QException + { + return (insert(table, List.of(record), null).getRecords().get(0)); + } + + + + /******************************************************************************* + ** Update. + *******************************************************************************/ + public static List update(String table, List records, QBackendTransaction transaction) throws QException + { + if(records.isEmpty()) + { + return (records); + } + return (call(() -> new UpdateAction().execute(new UpdateInput(table).withRecords(records).withTransaction(transaction)).getRecords())); + } + + + + /******************************************************************************* + ** Update one record. + *******************************************************************************/ + public static void updateOne(String table, QRecord record) throws QException + { + update(table, List.of(record), null); + } + + + + /******************************************************************************* + ** Query. + *******************************************************************************/ + public static List query(String table, QQueryFilter filter) throws QException + { + return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter)).getRecords())); + } + + + + /******************************************************************************* + ** Query in a transaction. + *******************************************************************************/ + public static List query(String table, QQueryFilter filter, QBackendTransaction transaction) throws QException + { + return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withTransaction(transaction)).getRecords())); + } + + + + /******************************************************************************* + ** Count. + *******************************************************************************/ + public static int count(String table, QQueryFilter filter) throws QException + { + return (count(table, filter, null)); + } + + + /******************************************************************************* + ** Count within the claim transaction so rate budgets remain shared. + *******************************************************************************/ + public static int count(String table, QQueryFilter filter, QBackendTransaction transaction) throws QException + { + Integer count = call(() -> new CountAction().execute(new CountInput(table).withFilter(filter).withTransaction(transaction)).getCount()); + return (count == null ? 0 : count); + } + + + + /******************************************************************************* + ** Delete by filter, returning the number deleted. + *******************************************************************************/ + public static int delete(String table, QQueryFilter filter) throws QException + { + return (delete(table, filter, null)); + } + + + /******************************************************************************* + ** Delete within a transaction, including retention's parent and child rows. + *******************************************************************************/ + public static int delete(String table, QQueryFilter filter, QBackendTransaction transaction) throws QException + { + Integer deleted = call(() -> new DeleteAction().execute(new DeleteInput(table).withQueryFilter(filter).withTransaction(transaction)).getDeletedRecordCount()); + return (deleted == null ? 0 : deleted); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java new file mode 100644 index 0000000..379073a --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java @@ -0,0 +1,234 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.util.List; +import java.util.Map; + + +/******************************************************************************* + ** Rendered, immutable-by-convention content passed to a transport. + *******************************************************************************/ +public class EmailEnvelope +{ + private String messageId; + private String from; + private List to = List.of(); + private List cc = List.of(); + private List bcc = List.of(); + private String replyTo; + private String subject; + private String htmlBody; + private String textBody; + private Map headers = Map.of(); + + + + /******************************************************************************* + ** Getter for messageId. + *******************************************************************************/ + public String getMessageId() + { + return (messageId); + } + + + /******************************************************************************* + ** Fluent setter for messageId. + *******************************************************************************/ + public EmailEnvelope withMessageId(String messageId) + { + this.messageId = messageId; + return (this); + } + + + /******************************************************************************* + ** Getter for from. + *******************************************************************************/ + public String getFrom() + { + return (from); + } + + + /******************************************************************************* + ** Fluent setter for from. + *******************************************************************************/ + public EmailEnvelope withFrom(String from) + { + this.from = from; + return (this); + } + + + /******************************************************************************* + ** Getter for to. + *******************************************************************************/ + public List getTo() + { + return (to); + } + + + /******************************************************************************* + ** Fluent setter for to. + *******************************************************************************/ + public EmailEnvelope withTo(List to) + { + this.to = to; + return (this); + } + + + /******************************************************************************* + ** Getter for cc. + *******************************************************************************/ + public List getCc() + { + return (cc); + } + + + /******************************************************************************* + ** Fluent setter for cc. + *******************************************************************************/ + public EmailEnvelope withCc(List cc) + { + this.cc = cc; + return (this); + } + + + /******************************************************************************* + ** Getter for bcc. + *******************************************************************************/ + public List getBcc() + { + return (bcc); + } + + + /******************************************************************************* + ** Fluent setter for bcc. + *******************************************************************************/ + public EmailEnvelope withBcc(List bcc) + { + this.bcc = bcc; + return (this); + } + + + /******************************************************************************* + ** Getter for replyTo. + *******************************************************************************/ + public String getReplyTo() + { + return (replyTo); + } + + + /******************************************************************************* + ** Fluent setter for replyTo. + *******************************************************************************/ + public EmailEnvelope withReplyTo(String replyTo) + { + this.replyTo = replyTo; + return (this); + } + + + /******************************************************************************* + ** Getter for subject. + *******************************************************************************/ + public String getSubject() + { + return (subject); + } + + + /******************************************************************************* + ** Fluent setter for subject. + *******************************************************************************/ + public EmailEnvelope withSubject(String subject) + { + this.subject = subject; + return (this); + } + + + /******************************************************************************* + ** Getter for htmlBody. + *******************************************************************************/ + public String getHtmlBody() + { + return (htmlBody); + } + + + /******************************************************************************* + ** Fluent setter for htmlBody. + *******************************************************************************/ + public EmailEnvelope withHtmlBody(String htmlBody) + { + this.htmlBody = htmlBody; + return (this); + } + + + /******************************************************************************* + ** Getter for textBody. + *******************************************************************************/ + public String getTextBody() + { + return (textBody); + } + + + /******************************************************************************* + ** Fluent setter for textBody. + *******************************************************************************/ + public EmailEnvelope withTextBody(String textBody) + { + this.textBody = textBody; + return (this); + } + + + /******************************************************************************* + ** Getter for headers. + *******************************************************************************/ + public Map getHeaders() + { + return (headers); + } + + + /******************************************************************************* + ** Fluent setter for headers. + *******************************************************************************/ + public EmailEnvelope withHeaders(Map headers) + { + this.headers = headers; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailHttp.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailHttp.java new file mode 100644 index 0000000..605846b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailHttp.java @@ -0,0 +1,70 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.io.IOException; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.nio.charset.StandardCharsets; +import java.time.Duration; + + +/******************************************************************************* + ** Package-visible HTTP seam; no provider SDK, credential cache or filesystem. + *******************************************************************************/ +@FunctionalInterface +interface EmailHttp +{ + /******************************************************************************* + ** Exchange one request without retries. + *******************************************************************************/ + HttpResponse send(HttpRequest request) throws IOException, InterruptedException; + + + + /******************************************************************************* + ** Production requests use HTTPS and never forward credentials on redirects. + *******************************************************************************/ + static HttpResponse sendJdk(HttpRequest request) throws IOException, InterruptedException + { + if(!"https".equalsIgnoreCase(request.uri().getScheme())) + { + throw (new IOException("Email HTTP requires HTTPS")); + } + try(HttpClient client = newClient()) + { + return (client.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8))); + } + } + + + + /******************************************************************************* + ** A bounded, stateless JDK client; the caller closes it after each exchange. + *******************************************************************************/ + static HttpClient newClient() + { + return (HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).followRedirects(HttpClient.Redirect.NEVER).build()); + } +} + diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java new file mode 100644 index 0000000..5398644 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java @@ -0,0 +1,230 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.io.IOException; +import java.net.URI; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.time.Duration; +import java.time.Instant; +import java.time.ZonedDateTime; +import java.time.format.DateTimeFormatter; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Properties; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import jakarta.mail.MessagingException; +import jakarta.mail.Session; + + +/******************************************************************************* + ** Shared serialization, timeouts and sanitized HTTP failures. + *******************************************************************************/ +final class EmailHttpSupport +{ + static final ObjectMapper JSON = new ObjectMapper(); + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailHttpSupport() + { + } + + + + /******************************************************************************* + ** Use the shared envelope validation even for JSON APIs. + *******************************************************************************/ + static void validate(EmailEnvelope envelope) throws EmailTransportException + { + try + { + EmailMime.build(Session.getInstance(new Properties()), envelope); + } + catch(MessagingException | RuntimeException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + } + + + + /******************************************************************************* + ** Preserve application headers and the stable correlation identifier. + *******************************************************************************/ + static Map headers(EmailEnvelope envelope) + { + Map headers = new LinkedHashMap<>(); + envelope.getHeaders().forEach((name, value) -> + { + if(!name.equalsIgnoreCase("Bcc") && !name.equalsIgnoreCase("Auto-Submitted") && !name.equalsIgnoreCase("X-QQQ-Email-Id")) + { + headers.put(name, value); + } + }); + headers.put("Auto-Submitted", "auto-generated"); + headers.put("X-QQQ-Email-Id", envelope.getMessageId()); + return (headers); + } + + + + /******************************************************************************* + ** Serialize only in memory using Jackson already supplied by core. + *******************************************************************************/ + static byte[] json(Object value) throws EmailTransportException + { + try + { + return (JSON.writeValueAsBytes(value)); + } + catch(IOException | RuntimeException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + } + + + + /******************************************************************************* + ** Resolve auth before calling; never attach raw exceptions to a safe failure. + *******************************************************************************/ + static HttpResponse post(EmailHttp http, String url, String contentType, String authHeader, String auth, byte[] body) throws EmailTransportException + { + try + { + HttpRequest request = HttpRequest.newBuilder(URI.create(url)).timeout(Duration.ofSeconds(30)) + .header("Content-Type", contentType).header("Accept", "application/json").header(authHeader, auth) + .POST(HttpRequest.BodyPublishers.ofByteArray(body)).build(); + return (http.send(request)); + } + catch(InterruptedException e) + { + Thread.currentThread().interrupt(); + throw (new EmailTransportException("Email HTTP request interrupted", false)); + } + catch(IOException e) + { + throw (new EmailTransportException("Email HTTP temporarily unavailable", false)); + } + catch(RuntimeException e) + { + throw (new EmailTransportException("Email HTTP configuration invalid", false).withProviderLevel(true)); + } + } + + + + /******************************************************************************* + ** Unparseable replies must not expose bodies or cause permanent message loss. + *******************************************************************************/ + static JsonNode responseJson(HttpResponse response) throws EmailTransportException + { + try + { + JsonNode json = JSON.readTree(response.body()); + if(json != null && json.isObject()) + { + return (json); + } + } + catch(IOException | RuntimeException e) + { + // Bodies and parser diagnostics may contain recipient addresses. + } + throw (new EmailTransportException("Email provider returned an invalid response", false)); + } + + + + /******************************************************************************* + ** Parse either Retry-After syntax; the dispatcher applies the one-hour cap. + *******************************************************************************/ + static Duration retryAfter(HttpResponse response) + { + String value = response.headers().firstValue("Retry-After").orElse("").trim(); + try + { + long seconds = Long.parseLong(value); + return (seconds < 0 ? null : Duration.ofSeconds(seconds)); + } + catch(RuntimeException e) + { + try + { + Duration delay = Duration.between(Instant.now(), ZonedDateTime.parse(value, DateTimeFormatter.RFC_1123_DATE_TIME).toInstant()); + return (delay.isNegative() ? Duration.ZERO : delay); + } + catch(RuntimeException ignored) + { + return (null); + } + } + } + + + + /******************************************************************************* + ** Only numeric status/code and fixed text escape the transport. + *******************************************************************************/ + static EmailTransportException failure(HttpResponse response, String provider, Integer code, boolean permanent, boolean providerLevel) + { + return (new EmailTransportException(provider + " request failed (HTTP " + response.statusCode() + + (code == null ? "" : ", code " + code) + ")", permanent).withProviderLevel(providerLevel).withRetryAfter(retryAfter(response))); + } + + + + /******************************************************************************* + ** Common HTTP precedence: throttling and server errors always remain retryable. + *******************************************************************************/ + static void checkStatus(HttpResponse response, String provider, int success) throws EmailTransportException + { + int status = response.statusCode(); + if(status != success) + { + boolean transientFailure = status == 408 || status == 429 || status >= 500; + boolean providerLevel = status == 401 || status == 403 || (status >= 300 && status < 400); + throw (failure(response, provider, null, !transientFailure && !providerLevel && status >= 400, providerLevel)); + } + } + + + + /******************************************************************************* + ** A success must contain a usable id; never mistake a malformed body for success. + *******************************************************************************/ + static EmailTransportResult result(String id) throws EmailTransportException + { + if(id == null || id.isBlank() || id.length() > 500 || id.contains("\r") || id.contains("\n")) + { + throw (new EmailTransportException("Email provider returned an invalid message id", false)); + } + return (new EmailTransportResult(id)); + } +} + diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java new file mode 100644 index 0000000..ac6534b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java @@ -0,0 +1,114 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.util.Date; +import java.util.List; +import jakarta.mail.Message; +import jakarta.mail.MessagingException; +import jakarta.mail.Session; +import jakarta.mail.internet.InternetAddress; +import jakarta.mail.internet.MimeBodyPart; +import jakarta.mail.internet.MimeMessage; +import jakarta.mail.internet.MimeMultipart; + + +/******************************************************************************* + ** Shared UTF-8 multipart MIME construction for SMTP and SES. + *******************************************************************************/ +final class EmailMime +{ + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailMime() + { + } + + /******************************************************************************* + ** Construct both alternative parts and stable application headers. + *******************************************************************************/ + static MimeMessage build(Session session, EmailEnvelope envelope) throws MessagingException + { + noLineBreaks(envelope.getSubject()); + noLineBreaks(envelope.getFrom()); + noLineBreaks(envelope.getReplyTo()); + noLineBreaks(envelope.getMessageId()); + MimeMessage message = new MimeMessage(session); + message.setFrom(new InternetAddress(envelope.getFrom(), true)); + recipients(message, Message.RecipientType.TO, envelope.getTo()); + recipients(message, Message.RecipientType.CC, envelope.getCc()); + recipients(message, Message.RecipientType.BCC, envelope.getBcc()); + if(envelope.getReplyTo() != null) + { + message.setReplyTo(InternetAddress.parse(envelope.getReplyTo(), true)); + } + message.setSubject(envelope.getSubject(), "UTF-8"); + message.setSentDate(new Date()); + MimeMultipart alternative = new MimeMultipart("alternative"); + MimeBodyPart text = new MimeBodyPart(); + text.setText(envelope.getTextBody() == null ? "" : envelope.getTextBody(), "UTF-8"); + alternative.addBodyPart(text); + MimeBodyPart html = new MimeBodyPart(); + html.setContent(envelope.getHtmlBody() == null ? "" : envelope.getHtmlBody(), "text/html; charset=UTF-8"); + alternative.addBodyPart(html); + message.setContent(alternative); + for(var header : envelope.getHeaders().entrySet()) + { + if(!header.getKey().matches("[A-Za-z0-9-]+") || header.getValue().contains("\r") || header.getValue().contains("\n")) + { + throw (new MessagingException("Invalid envelope header")); + } + message.setHeader(header.getKey(), header.getValue()); + } + message.setHeader("Auto-Submitted", "auto-generated"); + message.setHeader("X-QQQ-Email-Id", envelope.getMessageId()); + message.saveChanges(); + return (message); + } + + /******************************************************************************* + ** Add one recipient role. + *******************************************************************************/ + private static void recipients(MimeMessage message, Message.RecipientType type, List addresses) throws MessagingException + { + for(String address : addresses) + { + noLineBreaks(address); + } + if(!addresses.isEmpty()) + { + message.setRecipients(type, InternetAddress.parse(String.join(",", addresses), true)); + } + } + + /******************************************************************************* + ** Reject header injection even for envelopes that bypass queue validation. + *******************************************************************************/ + private static void noLineBreaks(String value) throws MessagingException + { + if(value != null && (value.contains("\r") || value.contains("\n"))) + { + throw (new MessagingException("Invalid envelope header")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailProviderSettings.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailProviderSettings.java new file mode 100644 index 0000000..6d4d470 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailProviderSettings.java @@ -0,0 +1,147 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import com.kingsrook.qbits.email.EmailQBitConfig; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + + +/******************************************************************************* + ** Provider row and resolver. Resolved secrets are never retained. + *******************************************************************************/ +public final class EmailProviderSettings +{ + private final QRecord row; + private final EmailQBitConfig config; + + /******************************************************************************* + ** Settings for one attempt, without resolving secrets. + *******************************************************************************/ + public EmailProviderSettings(QRecord row, EmailQBitConfig config) + { + this.row = row; + this.config = config; + } + + /******************************************************************************* + ** Nonsecret provider setting. + *******************************************************************************/ + public String value(String name) + { + return (row.getValueString(name)); + } + + /******************************************************************************* + ** SMTP port, including the optional Gmail preset. + *******************************************************************************/ + public int port() + { + return ("GMAIL_SMTP".equals(value("kind")) ? 587 : row.getValueInteger("port")); + } + + /******************************************************************************* + ** SMTP host. + *******************************************************************************/ + public String host() + { + return ("GMAIL_SMTP".equals(value("kind")) ? "smtp.gmail.com" : value("host")); + } + + /******************************************************************************* + ** SMTP security. + *******************************************************************************/ + public String security() + { + return ("GMAIL_SMTP".equals(value("kind")) ? "STARTTLS" : value("security")); + } + + /******************************************************************************* + ** Shared validation and send-time guard, without resolving credentials or DNS. + *******************************************************************************/ + public boolean allowsSmtpAuth() + { + boolean credentials = value("usernameEnvName") != null || value("secretEnvName") != null; + String security = value("security") == null ? security() : value("security"); + return (!credentials || "STARTTLS".equals(security) || "TLS".equals(security) + || config.getAllowPlaintextSmtpAuth() || "localhost".equalsIgnoreCase(host()) + || "127.0.0.1".equals(host()) || "::1".equals(host())); + } + + /******************************************************************************* + ** A domain is a single DNS name, never a URL or a path component escape. + *******************************************************************************/ + public static boolean validSendingDomain(String domain) + { + return (domain != null && domain.length() <= 253 + && domain.matches("(?i)[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)+")); + } + + /******************************************************************************* + ** Tracking is opt-in; null database values mean disabled. + *******************************************************************************/ + boolean enabled(String field) + { + return (Boolean.TRUE.equals(row.getValueBoolean(field))); + } + + /******************************************************************************* + ** HTTP providers must never send a missing token as the literal string null. + *******************************************************************************/ + String requiredSecret() throws EmailTransportException + { + String secret = resolve("secretEnvName"); + if(secret == null) + { + throw (new EmailTransportException("Provider needs a secret environment name", false).withProviderLevel(true)); + } + return (secret); + } + + /******************************************************************************* + ** Resolve an explicitly named secret immediately before sending. + *******************************************************************************/ + String resolve(String field) throws EmailTransportException + { + String name = value(field); + if(name == null) + { + return (null); + } + if(!EmailQBitConfig.ENV_NAME.matcher(name).matches() || !config.effectiveAllowedEnvNames(value("kind")).contains(name)) + { + throw (new EmailTransportException("Credential environment name is not allowed", false).withProviderLevel(true)); + } + try + { + String value = config.getSecretResolver().apply(name); + if(value != null && !value.isBlank()) + { + return (value); + } + } + catch(RuntimeException e) + { + // Resolver exception text may contain a secret. + } + throw (new EmailTransportException("Credential environment variable is not set", false).withProviderLevel(true)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransport.java new file mode 100644 index 0000000..55e21ab --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransport.java @@ -0,0 +1,37 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import com.kingsrook.qbits.email.transport.EmailTransportException; + + +/******************************************************************************* + ** Delivery transport; called only for a claimed outbox attempt. + *******************************************************************************/ +@FunctionalInterface +public interface EmailTransport +{ + /******************************************************************************* + ** Send a previously rendered envelope. + *******************************************************************************/ + EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException; +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportException.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportException.java new file mode 100644 index 0000000..c9b2b56 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportException.java @@ -0,0 +1,99 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.time.Duration; + + +/******************************************************************************* + ** Sanitized transport failure. Never retain a provider exception or credential. + *******************************************************************************/ +public class EmailTransportException extends Exception +{ + private boolean permanent; + private boolean providerLevel; + private Duration retryAfter; + + /******************************************************************************* + ** Create a safe failure with a permanent or transient classification. + *******************************************************************************/ + public EmailTransportException(String message, boolean permanent) + { + super(message); + this.permanent = permanent; + } + + + /******************************************************************************* + ** Getter for permanent. + *******************************************************************************/ + public boolean getPermanent() + { + return (permanent && !providerLevel); + } + + /******************************************************************************* + ** Provider failures are retryable without consuming the message retry budget. + *******************************************************************************/ + public boolean getProviderLevel() + { + return (providerLevel); + } + + /******************************************************************************* + ** Classify a failure affecting the provider rather than this message. + *******************************************************************************/ + public EmailTransportException withProviderLevel(boolean providerLevel) + { + this.providerLevel = providerLevel; + return (this); + } + + + /******************************************************************************* + ** Fluent setter for permanent. + *******************************************************************************/ + public EmailTransportException withPermanent(boolean permanent) + { + this.permanent = permanent; + return (this); + } + + + /******************************************************************************* + ** Getter for retryAfter. + *******************************************************************************/ + public Duration getRetryAfter() + { + return (retryAfter); + } + + + /******************************************************************************* + ** Fluent setter for retryAfter. + *******************************************************************************/ + public EmailTransportException withRetryAfter(Duration retryAfter) + { + this.retryAfter = retryAfter; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportRegistry.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportRegistry.java new file mode 100644 index 0000000..6cf887d --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportRegistry.java @@ -0,0 +1,70 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.util.HashMap; +import java.util.Map; + + +/******************************************************************************* + ** Per-dispatcher registry; no configuration or credential caches. + *******************************************************************************/ +public class EmailTransportRegistry +{ + private final Map transports = new HashMap<>(); + + /******************************************************************************* + ** Install the outbound transports. + *******************************************************************************/ + public EmailTransportRegistry() + { + transports.put("SMTP", new SmtpEmailTransport()); + transports.put("GMAIL_SMTP", new SmtpEmailTransport()); + transports.put("SES", new SesEmailTransport()); + transports.put("POSTMARK", new PostmarkEmailTransport()); + transports.put("SENDGRID", new SendGridEmailTransport()); + transports.put("MAILGUN", new MailgunEmailTransport()); + } + + /******************************************************************************* + ** Host or test transport override. + *******************************************************************************/ + public EmailTransportRegistry withTransport(String kind, EmailTransport transport) + { + transports.put(kind, transport); + return (this); + } + + /******************************************************************************* + ** Resolve a kind, explicitly refusing future transports. + *******************************************************************************/ + public EmailTransport get(String kind) throws EmailTransportException + { + EmailTransport transport = transports.get(kind); + if(transport == null) + { + throw (new EmailTransportException("GMAIL_API".equals(kind) + ? "Gmail API sending is specified but not built; use GMAIL_SMTP" : "Unknown email provider kind", true)); + } + return (transport); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportResult.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportResult.java new file mode 100644 index 0000000..2026e39 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailTransportResult.java @@ -0,0 +1,31 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + + + +/******************************************************************************* + ** Provider acknowledgment, never a delivery receipt. + *******************************************************************************/ +public record EmailTransportResult(String providerMessageId) +{ +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java new file mode 100644 index 0000000..89d3492 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java @@ -0,0 +1,132 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.net.http.HttpResponse; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Base64; +import java.util.List; +import java.util.Properties; +import java.util.UUID; +import jakarta.mail.MessagingException; +import jakarta.mail.Session; +import jakarta.mail.internet.InternetAddress; + + +/******************************************************************************* + ** Mailgun MIME upload, entirely in memory, with explicit blind destinations. + *******************************************************************************/ +public class MailgunEmailTransport implements EmailTransport +{ + private final EmailHttp http; + + + + /******************************************************************************* + ** Fixed regional production endpoints and JDK HTTP implementation. + *******************************************************************************/ + public MailgunEmailTransport() + { + this(EmailHttp::sendJdk); + } + + + + /******************************************************************************* + ** Package-visible request/response test seam. + *******************************************************************************/ + MailgunEmailTransport(EmailHttp http) + { + this.http = http; + } + + + + /******************************************************************************* + ** Envelope destinations include Bcc; serialized MIME never includes its header. + *******************************************************************************/ + @Override + public EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException + { + String domain = settings.value("sendingDomain"); + String region = settings.value("region"); + if(!EmailProviderSettings.validSendingDomain(domain) || (!"US".equals(region) && !"EU".equals(region))) + { + throw (new EmailTransportException("Mailgun needs a sending domain and US or EU region", false).withProviderLevel(true)); + } + String boundary = "qqq-email-" + UUID.randomUUID(); + ByteArrayOutputStream body = new ByteArrayOutputStream(); + try + { + List destinations = new ArrayList<>(); + for(List role : List.of(envelope.getTo(), envelope.getCc(), envelope.getBcc())) + { + for(String value : role) + { + for(InternetAddress address : InternetAddress.parse(value, true)) + { + address.validate(); + destinations.add(address.getAddress()); + } + } + } + field(body, boundary, "to", String.join(",", destinations)); + field(body, boundary, "o:tracking-opens", settings.enabled("trackOpens") ? "yes" : "no"); + field(body, boundary, "o:tracking-clicks", settings.enabled("trackClicks") ? "yes" : "no"); + field(body, boundary, "v:qqq_email_id", envelope.getMessageId()); + body.writeBytes(("--" + boundary + "\r\nContent-Disposition: form-data; name=\"message\"; filename=\"message.eml\"\r\n" + + "Content-Type: message/rfc822\r\n\r\n").getBytes(StandardCharsets.UTF_8)); + EmailMime.build(Session.getInstance(new Properties()), envelope).writeTo(body, new String[] {"Bcc"}); + body.writeBytes(("\r\n--" + boundary + "--\r\n").getBytes(StandardCharsets.UTF_8)); + } + catch(MessagingException | IOException | RuntimeException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + String auth = Base64.getEncoder().encodeToString(("api:" + settings.requiredSecret()).getBytes(StandardCharsets.UTF_8)); + String host = "EU".equals(region) ? "https://api.eu.mailgun.net" : "https://api.mailgun.net"; + HttpResponse response = EmailHttpSupport.post(http, host + "/v3/" + domain + "/messages.mime", + "multipart/form-data; boundary=" + boundary, "Authorization", "Basic " + auth, body.toByteArray()); + EmailHttpSupport.checkStatus(response, "Mailgun", 200); + String id = EmailHttpSupport.responseJson(response).path("id").textValue(); + if(id != null && id.startsWith("<") && id.endsWith(">")) + { + id = id.substring(1, id.length() - 1); + } + return (EmailHttpSupport.result(id)); + } + + + + /******************************************************************************* + ** Encode fixed-name UTF-8 multipart fields without touching the filesystem. + *******************************************************************************/ + private static void field(ByteArrayOutputStream body, String boundary, String name, String value) + { + body.writeBytes(("--" + boundary + "\r\nContent-Disposition: form-data; name=\"" + name + "\"\r\n\r\n" + + value + "\r\n").getBytes(StandardCharsets.UTF_8)); + } +} + diff --git a/src/main/java/com/kingsrook/qbits/email/transport/PostmarkEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/PostmarkEmailTransport.java new file mode 100644 index 0000000..e053fc9 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/PostmarkEmailTransport.java @@ -0,0 +1,129 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.net.http.HttpResponse; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Set; +import com.fasterxml.jackson.databind.JsonNode; + + +/******************************************************************************* + ** Postmark's single-message API with per-attempt server token resolution. + *******************************************************************************/ +public class PostmarkEmailTransport implements EmailTransport +{ + private final EmailHttp http; + + + + /******************************************************************************* + ** Fixed production endpoint and JDK HTTP implementation. + *******************************************************************************/ + public PostmarkEmailTransport() + { + this(EmailHttp::sendJdk); + } + + + + /******************************************************************************* + ** Package-visible request/response test seam. + *******************************************************************************/ + PostmarkEmailTransport(EmailHttp http) + { + this.http = http; + } + + + + /******************************************************************************* + ** Send a rendered message; provider response diagnostics never escape. + *******************************************************************************/ + @Override + public EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException + { + EmailHttpSupport.validate(envelope); + Map body = new LinkedHashMap<>(); + body.put("From", envelope.getFrom()); + body.put("To", String.join(",", envelope.getTo())); + body.put("Cc", String.join(",", envelope.getCc())); + body.put("Bcc", String.join(",", envelope.getBcc())); + if(envelope.getReplyTo() != null) + { + body.put("ReplyTo", envelope.getReplyTo()); + } + body.put("Subject", envelope.getSubject()); + if(envelope.getHtmlBody() != null) + { + body.put("HtmlBody", envelope.getHtmlBody()); + } + if(envelope.getTextBody() != null) + { + body.put("TextBody", envelope.getTextBody()); + } + body.put("Headers", EmailHttpSupport.headers(envelope).entrySet().stream() + .map(h -> Map.of("Name", h.getKey(), "Value", h.getValue())).toList()); + String stream = settings.value("messageStream"); + body.put("MessageStream", stream == null || stream.isBlank() ? "outbound" : stream); + body.put("TrackOpens", settings.enabled("trackOpens")); + body.put("TrackLinks", settings.enabled("trackClicks") ? "HtmlAndText" : "None"); + HttpResponse response = EmailHttpSupport.post(http, "https://api.postmarkapp.com/email", "application/json", + "X-Postmark-Server-Token", settings.requiredSecret(), EmailHttpSupport.json(body)); + int status = response.statusCode(); + if(status == 408 || status == 429 || status >= 500 || status == 401 || status == 403 || (status >= 300 && status < 400)) + { + EmailHttpSupport.checkStatus(response, "Postmark", 200); + } + JsonNode json; + try + { + json = EmailHttpSupport.responseJson(response); + } + catch(EmailTransportException e) + { + EmailHttpSupport.checkStatus(response, "Postmark", 200); + throw (e); + } + Integer code = json.path("ErrorCode").canConvertToInt() ? json.path("ErrorCode").intValue() : null; + if(status == 200 && code == null) + { + throw (new EmailTransportException("Email provider returned an invalid response", false)); + } + if(status != 200 || code == null || code != 0) + { + // Postmark ErrorCode (not HTTP status) | Classification + // 10 bad token | provider-level + // 300 invalid request, 406 inactive | permanent message failure + // 400/401 sender signature, 405 send disabled, 412 pending approval, + // 413 account unapproved, 422 invalid server, 1235/1236 stream, + // 1480 IP restriction | provider-level + // HTTP 429 / 5xx | transient, takes precedence + // Other request codes | permanent (unknown 200 is transient) + boolean providerLevel = code != null && Set.of(10, 400, 401, 405, 412, 413, 422, 1235, 1236, 1480).contains(code); + throw (EmailHttpSupport.failure(response, "Postmark", code, !providerLevel && (status >= 400 || code != null && code != 0), providerLevel)); + } + return (EmailHttpSupport.result(json.path("MessageID").textValue())); + } +} + diff --git a/src/main/java/com/kingsrook/qbits/email/transport/SendGridEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/SendGridEmailTransport.java new file mode 100644 index 0000000..b3a6242 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/SendGridEmailTransport.java @@ -0,0 +1,145 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.net.http.HttpResponse; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import jakarta.mail.internet.AddressException; +import jakarta.mail.internet.InternetAddress; + + +/******************************************************************************* + ** SendGrid v3 mail send with explicit per-message tracking settings. + *******************************************************************************/ +public class SendGridEmailTransport implements EmailTransport +{ + private static final Set RESERVED_HEADERS = Set.of("x-sg-id", "x-sg-eid", "received", "dkim-signature", + "content-type", "content-transfer-encoding", "to", "from", "subject", "reply-to", "cc", "bcc"); + private final EmailHttp http; + + + + /******************************************************************************* + ** Fixed production endpoint and JDK HTTP implementation. + *******************************************************************************/ + public SendGridEmailTransport() + { + this(EmailHttp::sendJdk); + } + + + + /******************************************************************************* + ** Package-visible request/response test seam. + *******************************************************************************/ + SendGridEmailTransport(EmailHttp http) + { + this.http = http; + } + + + + /******************************************************************************* + ** Build one personalization, keeping blind recipients out of custom headers. + *******************************************************************************/ + @Override + public EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException + { + EmailHttpSupport.validate(envelope); + Map body = new LinkedHashMap<>(); + Map personalization = new LinkedHashMap<>(); + personalization.put("to", addresses(envelope.getTo())); + if(!envelope.getCc().isEmpty()) + { + personalization.put("cc", addresses(envelope.getCc())); + } + if(!envelope.getBcc().isEmpty()) + { + personalization.put("bcc", addresses(envelope.getBcc())); + } + body.put("personalizations", List.of(personalization)); + body.put("from", addresses(List.of(envelope.getFrom())).getFirst()); + if(envelope.getReplyTo() != null) + { + body.put("reply_to", addresses(List.of(envelope.getReplyTo())).getFirst()); + } + body.put("subject", envelope.getSubject()); + List> content = new ArrayList<>(); + if(envelope.getTextBody() != null && !envelope.getTextBody().isEmpty()) + { + content.add(Map.of("type", "text/plain", "value", envelope.getTextBody())); + } + if(envelope.getHtmlBody() != null && !envelope.getHtmlBody().isEmpty()) + { + content.add(Map.of("type", "text/html", "value", envelope.getHtmlBody())); + } + body.put("content", content); + Map headers = EmailHttpSupport.headers(envelope); + headers.keySet().removeIf(name -> RESERVED_HEADERS.contains(name.toLowerCase(Locale.ROOT))); + body.put("headers", headers); + body.put("custom_args", Map.of("qqq_email_id", envelope.getMessageId())); + body.put("tracking_settings", Map.of("open_tracking", Map.of("enable", settings.enabled("trackOpens")), + "click_tracking", Map.of("enable", settings.enabled("trackClicks"), "enable_text", settings.enabled("trackClicks")))); + HttpResponse response = EmailHttpSupport.post(http, "https://api.sendgrid.com/v3/mail/send", "application/json", + "Authorization", "Bearer " + settings.requiredSecret(), EmailHttpSupport.json(body)); + EmailHttpSupport.checkStatus(response, "SendGrid", 202); + return (EmailHttpSupport.result(response.headers().firstValue("X-Message-Id").orElse(null))); + } + + + + /******************************************************************************* + ** Convert mailboxes without losing their decoded display names. + *******************************************************************************/ + private static List> addresses(List values) throws EmailTransportException + { + List> result = new ArrayList<>(); + try + { + for(String value : values) + { + for(InternetAddress address : InternetAddress.parse(value, true)) + { + address.validate(); + Map entry = new LinkedHashMap<>(); + entry.put("email", address.getAddress()); + if(address.getPersonal() != null) + { + entry.put("name", address.getPersonal()); + } + result.add(entry); + } + } + } + catch(AddressException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + return (result); + } +} + diff --git a/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java new file mode 100644 index 0000000..671c5aa --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java @@ -0,0 +1,163 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Properties; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.Function; +import jakarta.mail.MessagingException; +import jakarta.mail.Session; +import jakarta.mail.internet.AddressException; +import jakarta.mail.internet.InternetAddress; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.core.SdkBytes; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.sesv2.SesV2Client; +import software.amazon.awssdk.services.sesv2.SesV2ClientBuilder; +import software.amazon.awssdk.services.sesv2.model.Destination; +import software.amazon.awssdk.services.sesv2.model.EmailContent; +import software.amazon.awssdk.services.sesv2.model.MessageRejectedException; +import software.amazon.awssdk.services.sesv2.model.RawMessage; +import software.amazon.awssdk.services.sesv2.model.SendEmailRequest; +import software.amazon.awssdk.services.sesv2.model.SesV2Exception; + + +/******************************************************************************* + ** SES v2 raw MIME transport; credentials are resolved only for an attempt. + *******************************************************************************/ +public class SesEmailTransport implements EmailTransport +{ + private final Function clientFactory; + + /******************************************************************************* + ** Production client factory. + *******************************************************************************/ + public SesEmailTransport() + { + this(SesV2ClientBuilder::build); + } + + /******************************************************************************* + ** Package-visible fake-client seam. + *******************************************************************************/ + SesEmailTransport(Function clientFactory) + { + this.clientFactory = clientFactory; + } + + /******************************************************************************* + ** Send the same MIME parts and headers as SMTP. + *******************************************************************************/ + @Override + public EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException + { + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try + { + EmailMime.build(Session.getInstance(new Properties()), envelope).writeTo(bytes, new String[] {"Bcc"}); + } + catch(MessagingException | IOException | RuntimeException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + AtomicBoolean credentialsFailed = new AtomicBoolean(); + try(DefaultCredentialsProvider defaultCredentials = DefaultCredentialsProvider.builder().build()) + { + SesV2ClientBuilder builder = SesV2Client.builder().region(Region.of(settings.value("region"))) + .overrideConfiguration(c -> c.apiCallTimeout(Duration.ofSeconds(40)).apiCallAttemptTimeout(Duration.ofSeconds(35)) + .retryPolicy(software.amazon.awssdk.core.retry.RetryPolicy.none())); + if(settings.value("usernameEnvName") != null) + { + builder.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(settings.resolve("usernameEnvName"), settings.resolve("secretEnvName")))); + } + else + { + builder.credentialsProvider(() -> + { + try + { + return (defaultCredentials.resolveCredentials()); + } + catch(RuntimeException e) + { + credentialsFailed.set(true); + throw (e); + } + }); + } + SendEmailRequest request = SendEmailRequest.builder() + .fromEmailAddress(envelope.getFrom()) + .destination(Destination.builder().toAddresses(destinations(envelope.getTo())).ccAddresses(destinations(envelope.getCc())).bccAddresses(destinations(envelope.getBcc())).build()) + .configurationSetName(settings.value("configurationSet")) + .content(EmailContent.builder().raw(RawMessage.builder().data(SdkBytes.fromByteArray(bytes.toByteArray())).build()).build()).build(); + try(SesV2Client client = clientFactory.apply(builder)) + { + return (new EmailTransportResult(client.sendEmail(request).messageId())); + } + } + catch(EmailTransportException e) + { + throw (e); + } + catch(SesV2Exception e) + { + String code = e.awsErrorDetails() == null ? null : e.awsErrorDetails().errorCode(); + boolean clientFailure = e.statusCode() >= 400 && e.statusCode() < 500 && e.statusCode() != 429 && !e.isThrottlingException(); + boolean permanent = clientFailure && e.statusCode() != 401 && e.statusCode() != 403 + && (e instanceof MessageRejectedException || "MessageRejected".equals(code) || "InvalidParameterValue".equals(code)); + throw (new EmailTransportException(permanent ? "SES rejected the message" : "SES temporarily unavailable", permanent) + .withProviderLevel(clientFailure && !permanent)); + } + catch(AddressException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + catch(Exception e) + { + throw (new EmailTransportException("SES temporarily unavailable", false).withProviderLevel(credentialsFailed.get())); + } + } + + + /******************************************************************************* + ** SES envelope destinations are individual mailboxes, without display names. + *******************************************************************************/ + private static List destinations(List values) throws AddressException + { + List addresses = new ArrayList<>(); + for(String value : values) + { + for(InternetAddress address : InternetAddress.parse(value, true)) + { + addresses.add(address.getAddress()); + } + } + return (addresses); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/transport/SmtpEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/SmtpEmailTransport.java new file mode 100644 index 0000000..6eadf07 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/transport/SmtpEmailTransport.java @@ -0,0 +1,135 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.transport; + + +import java.util.HashSet; +import java.util.Properties; +import java.util.Set; +import com.sun.mail.smtp.SMTPAddressFailedException; +import com.sun.mail.smtp.SMTPSendFailedException; +import jakarta.mail.AuthenticationFailedException; +import jakarta.mail.MessagingException; +import jakarta.mail.Session; +import jakarta.mail.Transport; +import jakarta.mail.internet.MimeMessage; + + +/******************************************************************************* + ** Authenticated or relay SMTP with bounded connect, read and write timeouts. + *******************************************************************************/ +public class SmtpEmailTransport implements EmailTransport +{ + /******************************************************************************* + ** Submit one claimed message. Provider diagnostics are never retained. + *******************************************************************************/ + @Override + public EmailTransportResult send(EmailEnvelope envelope, EmailProviderSettings settings) throws EmailTransportException + { + if(!settings.allowsSmtpAuth()) + { + throw (new EmailTransportException("SMTP sign-in needs STARTTLS or TLS", false).withProviderLevel(true)); + } + if("GMAIL_SMTP".equals(settings.value("kind")) + && (settings.value("usernameEnvName") == null || settings.value("secretEnvName") == null + || (settings.value("security") != null && !"STARTTLS".equals(settings.value("security"))))) + { + throw (new EmailTransportException("Gmail SMTP needs both credential names and STARTTLS", false).withProviderLevel(true)); + } + Properties properties = new Properties(); + properties.setProperty("mail.smtp.connectiontimeout", "10000"); + properties.setProperty("mail.smtp.timeout", "15000"); + properties.setProperty("mail.smtp.writetimeout", "15000"); + properties.setProperty("mail.smtp.auth", String.valueOf(settings.value("usernameEnvName") != null)); + properties.setProperty("mail.smtp.starttls.enable", String.valueOf("STARTTLS".equals(settings.security()))); + properties.setProperty("mail.smtp.starttls.required", String.valueOf("STARTTLS".equals(settings.security()))); + properties.setProperty("mail.smtp.ssl.enable", String.valueOf("TLS".equals(settings.security()))); + properties.setProperty("mail.smtp.ssl.checkserveridentity", "true"); + Session session = Session.getInstance(properties); + MimeMessage message; + try + { + message = EmailMime.build(session, envelope); + } + catch(MessagingException | RuntimeException e) + { + throw (new EmailTransportException("Invalid email message", true)); + } + try(Transport transport = session.getTransport("smtp")) + { + String username = settings.resolve("usernameEnvName"); + String password = username == null ? null : settings.resolve("secretEnvName"); + transport.connect(settings.host(), settings.port(), username, password); + transport.sendMessage(message, message.getAllRecipients()); + return (new EmailTransportResult(message.getMessageID())); + } + catch(MessagingException e) + { + boolean permanent = permanent(e); + throw (new EmailTransportException(permanent ? "SMTP rejected the message" : "SMTP temporarily unavailable", permanent).withProviderLevel(providerLevel(e))); + } + } + + /******************************************************************************* + ** Inspect chained SMTP reply codes without exposing addresses or diagnostics. + *******************************************************************************/ + static boolean permanent(Throwable error) + { + if(providerLevel(error)) + { + return (false); + } + Set seen = new HashSet<>(); + while(error != null && seen.add(error)) + { + if(error instanceof SMTPAddressFailedException address && address.getReturnCode() >= 500) + { + return (true); + } + if(error instanceof SMTPSendFailedException send && send.getReturnCode() >= 500) + { + return (true); + } + error = error instanceof MessagingException messaging && messaging.getNextException() != null ? messaging.getNextException() : error.getCause(); + } + return (false); + } + + /******************************************************************************* + ** Authentication failures affect every message, including temporary refusals. + *******************************************************************************/ + static boolean providerLevel(Throwable error) + { + Set seen = new HashSet<>(); + while(error != null && seen.add(error)) + { + if(error instanceof AuthenticationFailedException + || error instanceof SMTPSendFailedException send && (send.getReturnCode() == 530 || send.getReturnCode() == 535) + || error instanceof SMTPAddressFailedException address && (address.getReturnCode() == 530 || address.getReturnCode() == 535)) + { + return (true); + } + error = error instanceof MessagingException messaging && messaging.getNextException() != null ? messaging.getNextException() : error.getCause(); + } + return (false); + } +} + diff --git a/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java b/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java new file mode 100644 index 0000000..1492106 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java @@ -0,0 +1,927 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email; + +import java.net.ServerSocket; +import java.time.*; +import java.util.*; +import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicInteger; +import com.icegreen.greenmail.util.GreenMail; +import com.icegreen.greenmail.util.ServerSetup; +import com.kingsrook.qbits.email.delivery.*; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.outbox.*; +import com.kingsrook.qbits.email.processes.EmailProcesses; +import com.kingsrook.qbits.email.support.*; +import com.kingsrook.qbits.email.transport.*; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.messaging.SendMessageAction; +import com.kingsrook.qqq.backend.core.alerts.AlertResolution; +import com.kingsrook.qqq.backend.core.alerts.AlertSignal; +import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface; +import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.messaging.*; +import com.kingsrook.qqq.backend.core.model.actions.messaging.email.*; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import jakarta.mail.Multipart; +import org.junit.jupiter.api.*; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import static org.assertj.core.api.Assertions.*; + +/******************************************************************************* + ** Email Delivery Test. + *******************************************************************************/ +class EmailDeliveryTest +{ + GreenMail mail; + Instant now = Instant.parse("2026-10-13T12:00:00Z"); + + /******************************************************************************* + ** Setup. + *******************************************************************************/ + @BeforeEach + void setup() throws Exception + { + EmailRuntime.setClock(Clock.fixed(now, ZoneOffset.UTC)); + H2Support.setup(c -> + { + }); + mail = new GreenMail(new ServerSetup(0, "127.0.0.1", "smtp")); + mail.start(); + H2Support.provider(mail.getSmtp().getPort()); + } + + /******************************************************************************* + ** Cleanup. + *******************************************************************************/ + @AfterEach + void cleanup() + { + if(mail != null) + { + mail.stop(); + } + QContext.clear(); + EmailRuntime.setClock(null); + } + + /******************************************************************************* + ** Request. + *******************************************************************************/ + EmailRequest request() + { + return (new EmailRequest() + .withTo("Person ") + .withSubject("Hello") + .withMarkdownBody("# Hello\n\nA **message**.")); + } + + /******************************************************************************* + ** Message. + *******************************************************************************/ + QRecord message(Integer id) throws Exception + { + return (EmailStore.get(EmailTables.MESSAGE, id, null)); + } + + /******************************************************************************* + ** Fake. + *******************************************************************************/ + EmailDispatcher fake(EmailTransport transport) + { + return (new EmailDispatcher(new EmailTransportRegistry().withTransport("SMTP", transport))); + } + + /******************************************************************************* + ** Missing credentials and SMTP 535 pause a provider until credentials recover. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void providerCredentialsRecoverWithoutExhaustingMessageBudget(boolean refused) throws Exception + { + mail.setUser("person@example.com", "login", "password"); + EmailRuntime.config().withMaxAttempts(1).withSecretResolver(name -> refused ? "wrong" : null); + SystemWrites.updateOne(EmailTables.PROVIDER, new QRecord().withValue("id", 1) + .withValue("usernameEnvName", "EMAIL_SMTP_USERNAME").withValue("secretEnvName", "EMAIL_SMTP_PASSWORD")); + var first = EmailService.send(request()); + var second = EmailService.send(request()); + var dispatcher = new EmailDispatcher(); + for(int i = 0; i < 2; i++) + { + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(i * 301L), ZoneOffset.UTC)); + assertThat(dispatcher.runOnce()).isEqualTo(new EmailDispatcher.RunSummary(0, 1, 1)); + assertThat(message(first.messageId()).getValueString("status")).isEqualTo("RETRY_WAIT"); + assertThat(message(first.messageId()).getValueInteger("attemptCount")).isEqualTo(i + 1); + assertThat(message(first.messageId()).getValueInstant("nextAttemptDate")) + .isEqualTo(now.plusSeconds(i * 301L + (i == 0 ? 60 : 300))); + assertThat(message(second.messageId()).getValueInteger("attemptCount")).isZero(); + var provider = H2Support.all(EmailTables.PROVIDER).getFirst(); + assertThat(provider.getValueString("healthStatus")).isEqualTo("UNHEALTHY"); + assertThat(provider.getValueString("lastFailureMessage")).isEqualTo("Credentials missing or refused"); + } + EmailRuntime.config().withSecretResolver(name -> name.endsWith("USERNAME") ? "login" : "password"); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(602), ZoneOffset.UTC)); + assertThat(dispatcher.runOnce().sent()).isEqualTo(2); + assertThat(message(first.messageId()).getValueString("status")).isEqualTo("SENT"); + assertThat(H2Support.all(EmailTables.PROVIDER).getFirst().getValueString("healthStatus")).isEqualTo("HEALTHY"); + assertThat(H2Support.all(EmailTables.ATTEMPT)).extracting(row -> row.getValueString("outcome")) + .containsExactly("PROVIDER_FAILURE", "PROVIDER_FAILURE", "SUCCEEDED", "SUCCEEDED"); + } + + /******************************************************************************* + ** Ordinary transient failures still have their full budget after provider repair. + *******************************************************************************/ + @Test + void providerFailuresDoNotConsumeLaterTransientBudget() throws Exception + { + EmailRuntime.config().withMaxAttempts(2); + var queued = EmailService.send(request()); + var broken = fake((e, settings) -> + { + throw new EmailTransportException("credentials", false).withProviderLevel(true); + }); + for(int i = 0; i < 3; i++) + { + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(i * 901L), ZoneOffset.UTC)); + assertThat(broken.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT); + } + var temporary = fake((e, settings) -> + { + throw new EmailTransportException("temporary", false); + }); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(2703), ZoneOffset.UTC)); + assertThat(temporary.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(3004), ZoneOffset.UTC)); + assertThat(temporary.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.DEAD_LETTER); + assertThat(message(queued.messageId()).getValueInteger("attemptCount")).isEqualTo(5); + } + + /******************************************************************************* + ** Both lease recovery paths exclude earlier provider failures from the budget. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void leaseRecoveryExcludesProviderFailures(boolean maintenance) throws Exception + { + EmailRuntime.config().withMaxAttempts(2); + var queued = EmailService.send(request()); + for(int i = 0; i < 2; i++) + { + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(i * 301L), ZoneOffset.UTC)); + assertThat(fake((e, settings) -> + { + throw new EmailTransportException("credentials", false).withProviderLevel(true); + }).sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT); + } + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(602), ZoneOffset.UTC)); + var slow = fake((e, settings) -> + { + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(903), ZoneOffset.UTC)); + var recovery = fake((envelope, provider) -> new EmailTransportResult("recovered")); + try + { + assertThat((maintenance ? recovery.maintain() : recovery.runOnce()).sent()).isEqualTo(1); + } + catch(Exception ex) + { + throw new RuntimeException(ex); + } + return (new EmailTransportResult("late")); + }); + assertThat(slow.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.NOT_SENT); + assertThat(message(queued.messageId()).getValueString("providerMessageId")).isEqualTo("recovered"); + assertThat(H2Support.all(EmailTables.ATTEMPT)).extracting(row -> row.getValueString("outcome")) + .containsExactly("PROVIDER_FAILURE", "PROVIDER_FAILURE", "LEASE_EXPIRED", "SUCCEEDED"); + } + + /******************************************************************************* + ** Even a slow tick stops the failed provider while processing another provider. + *******************************************************************************/ + @Test + void providerFailureStopsTheTickAfterProbeWindowPasses() throws Exception + { + EmailService.send(request()); + EmailService.send(request()); + H2Support.insert(EmailTables.PROVIDER, new QRecord().withValue("name", "other").withValue("kind", "SMTP") + .withValue("host", "localhost").withValue("port", 25).withValue("fromAddress", "sender@example.com") + .withValue("enabled", true).withValue("isDefault", false)); + var other = EmailService.send(request().withProviderName("other")); + AtomicInteger calls = new AtomicInteger(); + var dispatcher = new EmailDispatcher(new EmailTransportRegistry().withTransport("SMTP", (e, settings) -> + { + calls.incrementAndGet(); + if("relay".equals(settings.value("name"))) + { + throw new EmailTransportException("credentials", false).withProviderLevel(true); + } + return (new EmailTransportResult("ok")); + })) + { + /******************************************************************************* + ** Advance time only after the failed attempt has committed. + *******************************************************************************/ + @Override + protected SendResult attempt(Integer id, boolean immediate) throws QException + { + SendResult result = super.attempt(id, immediate); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(601), ZoneOffset.UTC)); + return (result); + } + }; + assertThat(dispatcher.runOnce()).isEqualTo(new EmailDispatcher.RunSummary(1, 1, 1)); + assertThat(calls.get()).isEqualTo(2); + assertThat(message(other.messageId()).getValueString("status")).isEqualTo("SENT"); + } + + /******************************************************************************* + ** Corrupt queued MIME is dead-lettered and does not prevent the next delivery. + *******************************************************************************/ + @Test + void messageBuildFailureIsPermanent() throws Exception + { + var invalidAddress = EmailService.send(request()); + var invalidSubject = EmailService.send(request()); + var valid = EmailService.send(request()); + SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", invalidAddress.messageId()).withValue("fromAddress", "broken <")); + SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", invalidSubject.messageId()).withValue("subject", "bad\r\nheader")); + assertThat(new EmailDispatcher().runOnce()).isEqualTo(new EmailDispatcher.RunSummary(1, 2, 0)); + assertThat(message(invalidAddress.messageId()).getValueString("status")).isEqualTo("DEAD_LETTER"); + assertThat(message(invalidSubject.messageId()).getValueString("status")).isEqualTo("DEAD_LETTER"); + assertThat(message(valid.messageId()).getValueString("status")).isEqualTo("SENT"); + } + + /******************************************************************************* + ** A database or runtime failure on one message cannot abort a batch. + *******************************************************************************/ + @Test + void isolatesCheckedAndRuntimeAttemptFailures() throws Exception + { + var checked = EmailService.send(request()); + var unchecked = EmailService.send(request()); + var valid = EmailService.send(request()); + var dispatcher = new EmailDispatcher() + { + /******************************************************************************* + ** Inject failures at the same boundary as database claim/completion errors. + *******************************************************************************/ + @Override + protected SendResult attempt(Integer id, boolean immediate) throws QException + { + if(id.equals(checked.messageId())) + { + throw new QException("test database failure"); + } + if(id.equals(unchecked.messageId())) + { + throw new IllegalStateException("test runtime failure"); + } + return (super.attempt(id, immediate)); + } + }; + assertThat(dispatcher.runOnce()).isEqualTo(new EmailDispatcher.RunSummary(1, 2, 0)); + assertThat(message(valid.messageId()).getValueString("status")).isEqualTo("SENT"); + } + + /******************************************************************************* + ** Both unhealthy raises and recovery resolutions tolerate a throwing sink. + *******************************************************************************/ + @Test + void failingAlertSinkCannotFailCommittedOutcomes() throws Exception + { + ThrowingAlertSink.raised = 0; + ThrowingAlertSink.resolved = 0; + AlertsInstanceMetaData.ofOrWithNew(QContext.getQInstance()) + .withSink(new QCodeReference(ThrowingAlertSink.class)); + var queued = EmailService.send(request()); + assertThat(fake((e, settings) -> + { + throw new EmailTransportException("credentials", false).withProviderLevel(true); + }).sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT); + assertThat(ThrowingAlertSink.raised).isEqualTo(1); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(301), ZoneOffset.UTC)); + assertThat(new EmailDispatcher().sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.SENT); + assertThat(ThrowingAlertSink.resolved).isEqualTo(1); + assertThat(message(queued.messageId()).getValueString("status")).isEqualTo("SENT"); + var rejected = EmailService.send(request()); + assertThat(fake((e, settings) -> + { + throw new EmailTransportException("rejected", true); + }).sendNow(rejected.messageId())).isEqualTo(EmailDispatcher.SendResult.DEAD_LETTER); + assertThat(ThrowingAlertSink.raised).isEqualTo(2); + } + + /******************************************************************************* + ** Loadable sink that records invocation before simulating an alert outage. + *******************************************************************************/ + public static class ThrowingAlertSink implements AlertSinkInterface + { + static int raised; + static int resolved; + + /******************************************************************************* + ** Simulate a failed alert raise. + *******************************************************************************/ + @Override + public void raise(AlertSignal signal) + { + raised++; + throw new IllegalStateException("test alert outage"); + } + + /******************************************************************************* + ** Simulate a failed recovery resolution. + *******************************************************************************/ + @Override + public void resolve(AlertResolution resolution) + { + resolved++; + throw new IllegalStateException("test alert outage"); + } + + /******************************************************************************* + ** Heartbeats are not used by email delivery. + *******************************************************************************/ + @Override + public void heartbeat(String source, Instant timestamp) + { + } + } + + /******************************************************************************* + ** Smtp End To End And Idempotency. + *******************************************************************************/ + @Test + void smtpEndToEndAndIdempotency() throws Exception + { + var queued = EmailService.send(request().withIdempotencyKey("one")); + assertThat(queued.status()).isEqualTo(EmailSendResult.Status.QUEUED); + assertThat(mail.getReceivedMessages()).isEmpty(); + var duplicate = EmailService.send(request().withIdempotencyKey("one")); + assertThat(duplicate.status()).isEqualTo(EmailSendResult.Status.DUPLICATE); + assertThat(duplicate.messageId()).isEqualTo(queued.messageId()); + assertThat(new EmailDispatcher().runOnce().sent()).isEqualTo(1); + assertThat(mail.waitForIncomingEmail(2000, 1)).isTrue(); + var received = mail.getReceivedMessages()[0]; + Multipart parts = (Multipart) received.getContent(); + assertThat(parts.getCount()).isEqualTo(2); + assertThat(parts.getBodyPart(0).getContent().toString()).contains("Hello", "message"); + assertThat(parts.getBodyPart(1).getContent().toString()).contains("message"); + assertThat(received.getHeader("Auto-Submitted")).containsExactly("auto-generated"); + assertThat(received.getHeader("X-QQQ-Email-Id")).containsExactly(queued.messageId().toString()); + assertThat(message(queued.messageId()).getValueString("providerMessageId")) + .isEqualTo(received.getMessageID()); + assertThat(message(queued.messageId()).getValueString("status")).isEqualTo("SENT"); + assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(1); + } + + /******************************************************************************* + ** Smtp Authentication And Secret Resolution Only At Send. + *******************************************************************************/ + @Test + void smtpAuthenticationAndSecretResolutionOnlyAtSend() throws Exception + { + mail.setUser("person@example.com", "login", "password"); + AtomicInteger resolutions = new AtomicInteger(); + EmailRuntime.config() + .withSecretResolver( + name -> + { + resolutions.incrementAndGet(); + return (name.endsWith("USERNAME") ? "login" : "password"); + }); + SystemWrites.updateOne( + EmailTables.PROVIDER, + new QRecord() + .withValue("id", 1) + .withValue("usernameEnvName", "EMAIL_SMTP_USERNAME") + .withValue("secretEnvName", "EMAIL_SMTP_PASSWORD")); + var queued = EmailService.send(request()); + assertThat(resolutions.get()).isZero(); + assertThat(new EmailDispatcher().sendNow(queued.messageId())) + .isEqualTo(EmailDispatcher.SendResult.SENT); + assertThat(resolutions.get()).isEqualTo(2); + assertThat(mail.waitForIncomingEmail(2000, 1)).isTrue(); + } + + /******************************************************************************* + ** Unreachable Smtp Retries. + *******************************************************************************/ + @Test + void unreachableSmtpRetries() throws Exception + { + int port; + try(ServerSocket socket = new ServerSocket(0)) + { + port = socket.getLocalPort(); + } + SystemWrites.updateOne( + EmailTables.PROVIDER, new QRecord().withValue("id", 1).withValue("port", port)); + var queued = EmailService.send(request()); + assertThat(new EmailDispatcher().sendNow(queued.messageId())) + .isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT); + assertThat(message(queued.messageId()).getValueInstant("nextAttemptDate")) + .isEqualTo(now.plusSeconds(60)); + assertThat(message(queued.messageId()).getValueString("lastError")) + .doesNotContain("person", "Hello"); + } + + /******************************************************************************* + ** Retries Backoff Max Attempts And Retry After. + *******************************************************************************/ + @Test + void retriesBackoffMaxAttemptsAndRetryAfter() throws Exception + { + EmailRuntime.config().withMaxAttempts(3); + var queued = EmailService.send(request()); + var dispatcher = + fake( + (e, s) -> + { + throw new EmailTransportException("secret diagnostic", false); + }); + assertThat(dispatcher.runOnce().failed()).isEqualTo(1); + assertThat(message(queued.messageId()).getValueInstant("nextAttemptDate")) + .isEqualTo(now.plusSeconds(60)); + assertThat(dispatcher.runOnce().failed()).isZero(); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(60), ZoneOffset.UTC)); + dispatcher.runOnce(); + assertThat(message(queued.messageId()).getValueInstant("nextAttemptDate")) + .isEqualTo(now.plusSeconds(360)); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(360), ZoneOffset.UTC)); + assertThat(dispatcher.runOnce().failed()).isEqualTo(1); + assertThat(message(queued.messageId()).getValueString("status")).isEqualTo("DEAD_LETTER"); + var second = EmailService.send(request()); + fake((e, s) -> + { + throw new EmailTransportException("throttle", false).withRetryAfter(Duration.ofHours(2)); + }) + .runOnce(); + assertThat(message(second.messageId()).getValueInstant("nextAttemptDate")) + .isEqualTo(now.plusSeconds(3960)); + } + + /******************************************************************************* + ** Permanent Failure And Future Transport Are Dead Letters. + *******************************************************************************/ + @Test + void permanentFailureAndFutureTransportAreDeadLetters() throws Exception + { + var queued = EmailService.send(request()); + assertThat( + fake((e, s) -> + { + throw new EmailTransportException("rejected", true); + }) + .sendNow(queued.messageId())) + .isEqualTo(EmailDispatcher.SendResult.DEAD_LETTER); + var future = EmailService.send(request()); + SystemWrites.updateOne( + EmailTables.PROVIDER, new QRecord().withValue("id", 1).withValue("kind", "GMAIL_API")); + assertThat(new EmailDispatcher().sendNow(future.messageId())) + .isEqualTo(EmailDispatcher.SendResult.DEAD_LETTER); + assertThatThrownBy(() -> new EmailTransportRegistry().get("GMAIL_API")) + .hasMessage("Gmail API sending is specified but not built; use GMAIL_SMTP"); + } + + /******************************************************************************* + ** Expired Lease Retries And Stale Completion Cannot Overwrite. + *******************************************************************************/ + @Test + void expiredLeaseRetriesAndStaleCompletionCannotOverwrite() throws Exception + { + var queued = EmailService.send(request()); + var recovery = fake((e, s) -> new EmailTransportResult("recovered")); + var slow = + fake( + (e, s) -> + { + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(121), ZoneOffset.UTC)); + try + { + assertThat(recovery.runOnce().sent()).isEqualTo(1); + } + catch(Exception ex) + { + throw new RuntimeException(ex); + } + return (new EmailTransportResult("late")); + }); + assertThat(slow.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.NOT_SENT); + assertThat(message(queued.messageId()).getValueString("providerMessageId")) + .isEqualTo("recovered"); + assertThat(H2Support.all(EmailTables.ATTEMPT)) + .extracting(r -> r.getValueString("outcome")) + .containsExactly("LEASE_EXPIRED", "SUCCEEDED"); + } + + /******************************************************************************* + ** Suppression Scope Category Expiry And Partial. + *******************************************************************************/ + @Test + void suppressionScopeCategoryExpiryAndPartial() throws Exception + { + H2Support.insert( + EmailTables.SUPPRESSION, + new QRecord() + .withValue("address", "PERSON@example.com") + .withValue("scope", "ALL") + .withValue("reason", "HARD_BOUNCE")); + var suppressed = EmailService.send(request()); + assertThat(suppressed.status()).isEqualTo(EmailSendResult.Status.SUPPRESSED); + assertThat(new EmailDispatcher().runOnce().sent()).isZero(); + var partial = EmailService.send(request().withCc("other@example.com")); + assertThat(partial.status()).isEqualTo(EmailSendResult.Status.QUEUED); + assertThat(message(partial.messageId()).getValueString("toAddresses")).isEmpty(); + assertThat(message(partial.messageId()).getValueString("suppressedAddresses")) + .contains("person@example.com"); + SystemWrites.delete( + EmailTables.SUPPRESSION, + new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter()); + H2Support.insert( + EmailTables.SUPPRESSION, + new QRecord() + .withValue("address", "person@example.com") + .withValue("scope", "NOTIFICATION") + .withValue("reason", "UNSUBSCRIBE")); + assertThat(EmailService.send(request()).status()).isEqualTo(EmailSendResult.Status.QUEUED); + assertThat(EmailService.send(request().withCategory(EmailCategory.NOTIFICATION)).status()) + .isEqualTo(EmailSendResult.Status.SUPPRESSED); + SystemWrites.updateOne( + EmailTables.SUPPRESSION, new QRecord().withValue("id", 2).withValue("scope", "ALL")); + assertThat(EmailService.send(request().withCategory(EmailCategory.SECURITY)).status()) + .isEqualTo(EmailSendResult.Status.QUEUED); + SystemWrites.updateOne( + EmailTables.SUPPRESSION, + new QRecord().withValue("id", 2).withValue("expiresDate", now.minusSeconds(1))); + assertThat(EmailService.send(request().withCategory(EmailCategory.MARKETING)).status()) + .isEqualTo(EmailSendResult.Status.QUEUED); + } + + /******************************************************************************* + ** Rate Limits Do Not Consume Attempts And Send After Is Honored. + *******************************************************************************/ + @Test + void rateLimitsDoNotConsumeAttemptsAndSendAfterIsHonored() throws Exception + { + SystemWrites.updateOne( + EmailTables.PROVIDER, + new QRecord().withValue("id", 1).withValue("maxPerMinute", 1).withValue("maxPerDay", 2)); + EmailService.send(request()); + EmailService.send(request()); + var deferred = EmailService.send(request().withSendAfter(now.plusSeconds(1000))); + var dispatcher = fake((e, s) -> new EmailTransportResult("ok")); + assertThat(dispatcher.runOnce().sent()).isEqualTo(1); + assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(1); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(61), ZoneOffset.UTC)); + assertThat(dispatcher.runOnce().sent()).isEqualTo(1); + assertThat(dispatcher.sendNow(deferred.messageId())) + .isEqualTo(EmailDispatcher.SendResult.NOT_SENT); + } + + /******************************************************************************* + ** Transaction Commit Rollback And Cross Backend Callback. + *******************************************************************************/ + @Test + void transactionCommitRollbackAndCrossBackendCallback() throws Exception + { + try(var tx = QBackendTransaction.openFor(new InsertInput(EmailTables.MESSAGE))) + { + assertThat(EmailService.send(request().withTransaction(tx, "rdbms")).status()) + .isEqualTo(EmailSendResult.Status.QUEUED); + tx.rollback(); + } + assertThat(H2Support.all(EmailTables.MESSAGE)).isEmpty(); + try(var tx = QBackendTransaction.openFor(new InsertInput(EmailTables.MESSAGE))) + { + EmailService.send(request().withTransaction(tx, "rdbms")); + tx.commit(); + } + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1); + var other = new QBackendTransaction(); + var result = EmailService.send(request().withTransaction(other, "elsewhere")); + assertThat(result.messageId()).isNull(); + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1); + other.commit(); + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(2); + var rolledBack = new QBackendTransaction(); + EmailService.send(request().withTransaction(rolledBack, "elsewhere")); + rolledBack.rollback(); + rolledBack.commit(); + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(2); + } + + /******************************************************************************* + ** Bridge Core Action Queues And Delivers. + *******************************************************************************/ + @Test + void bridgeCoreActionQueuesAndDelivers() throws Exception + { + new SendMessageAction() + .execute( + new SendMessageInput() + .withMessagingProviderName("qbitEmail") + .withSubject("Bridge") + .withTo( + new MultiParty() + .withParty( + new Party() + .withAddress("person@example.com") + .withRole(EmailPartyRole.TO)) + .withParty( + new Party() + .withAddress("copy@example.com") + .withRole(EmailPartyRole.CC))) + .withFrom(new Party().withAddress("sender@example.com").withLabel("Sender")) + .withContent( + new Content().withContentRole(EmailContentRole.HTML).withBody("

Bridge

")) + .withContent( + new Content().withContentRole(EmailContentRole.TEXT).withBody("Bridge"))); + assertThat(H2Support.all(EmailTables.MESSAGE)) + .hasSize(1) + .allMatch(r -> "QUEUED".equals(r.getValueString("status"))); + assertThat(mail.getReceivedMessages()).isEmpty(); + assertThat(new EmailDispatcher().runOnce().sent()).isEqualTo(1); + assertThat(mail.waitForIncomingEmail(2000, 2)).isTrue(); + } + + /******************************************************************************* + ** Provider Health Ten Failures Probe And Recovery. + *******************************************************************************/ + @Test + void providerHealthTenFailuresProbeAndRecovery() throws Exception + { + EmailRuntime.config().withMaxAttempts(20); + for(int i = 0; i < 11; i++) + { + EmailService.send(request()); + } + var dispatcher = + fake( + (e, s) -> + { + throw new EmailTransportException("temporary", false); + }); + assertThat(dispatcher.runOnce().failed()).isEqualTo(10); + assertThat(H2Support.all(EmailTables.PROVIDER).getFirst().getValueString("healthStatus")) + .isEqualTo("UNHEALTHY"); + assertThat(dispatcher.runOnce().failed()).isZero(); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(301), ZoneOffset.UTC)); + assertThat(dispatcher.maintain().failed()).isEqualTo(1); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(602), ZoneOffset.UTC)); + assertThat(fake((e, s) -> new EmailTransportResult("ok")).runOnce().sent()).isGreaterThan(0); + assertThat(H2Support.all(EmailTables.PROVIDER).getFirst().getValueString("healthStatus")) + .isEqualTo("HEALTHY"); + } + + /******************************************************************************* + ** Retention Keeps Digest And Unfinished Messages. + *******************************************************************************/ + @Test + void retentionKeepsDigestAndUnfinishedMessages() throws Exception + { + var sent = EmailService.send(request()); + fake((e, s) -> new EmailTransportResult("ok")).sendNow(sent.messageId()); + var waiting = EmailService.send(request()); + EmailRuntime.setClock(Clock.fixed(now.plus(Duration.ofDays(31)), ZoneOffset.UTC)); + EmailProcesses.retain(); + assertThat(message(sent.messageId()).getValueString("htmlBody")).isNull(); + assertThat(message(sent.messageId()).getValueString("bodySha256")).hasSize(64); + assertThat(message(waiting.messageId()).getValueString("htmlBody")).isNotNull(); + EmailRuntime.config().withMessageRetentionDays(30); + EmailProcesses.retain(); + assertThat(message(sent.messageId())).isNull(); + assertThat(H2Support.all(EmailTables.ATTEMPT)).isEmpty(); + } + + /******************************************************************************* + ** Two Replicas Claim Each Message Once And Share Rate Budget. + *******************************************************************************/ + @Test + void twoReplicasClaimEachMessageOnceAndShareRateBudget() throws Exception + { + var instanceA = QContext.getQInstance(); + var backend = + (com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData) + instanceA.getBackend("rdbms"); + var instanceB = H2Support.instance(backend, c -> + { + }); + for(int i = 0; i < 25; i++) + { + EmailService.send(request()); + } + var received = new ConcurrentHashMap(); + var ready = new CyclicBarrier(2); + try(var pool = Executors.newFixedThreadPool(2)) + { + List> futures = new ArrayList<>(); + for(var instance : List.of(instanceA, instanceB)) + { + futures.add( + pool.submit( + () -> + { + QContext.init(instance, H2Support.admin()); + try + { + ready.await(); + var worker = + fake( + (e, s) -> + { + received + .computeIfAbsent(e.getMessageId(), k -> new AtomicInteger()) + .incrementAndGet(); + return (new EmailTransportResult("ok")); + }); + for(int i = 0; i < 3; i++) + { + worker.runOnce(); + } + } + catch(Exception e) + { + throw new RuntimeException(e); + } + finally + { + QContext.clear(); + } + })); + } + for(var future : futures) + { + future.get(30, TimeUnit.SECONDS); + } + } + assertThat(received).hasSize(25); + assertThat(received.values()).allMatch(n -> n.get() == 1); + assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(25); + } + + /******************************************************************************* + ** Concurrent Idempotency Joins Caller Transactions. + *******************************************************************************/ + @Test + void concurrentIdempotencyJoinsCallerTransactions() throws Exception + { + var instance = QContext.getQInstance(); + try(var pool = Executors.newFixedThreadPool(4)) + { + var barrier = new CyclicBarrier(4); + List> results = new ArrayList<>(); + for(int i = 0; i < 4; i++) + { + results.add( + pool.submit( + () -> + { + QContext.init(instance, H2Support.admin()); + try(var tx = QBackendTransaction.openFor(new InsertInput(EmailTables.MESSAGE))) + { + barrier.await(); + var result = + EmailService.send( + request() + .withIdempotencyKey("concurrent") + .withTransaction(tx, "rdbms")); + tx.commit(); + return (result); + } + finally + { + QContext.clear(); + } + })); + } + List outcomes = new ArrayList<>(); + for(var f : results) + { + outcomes.add(f.get(20, TimeUnit.SECONDS)); + } + assertThat(outcomes).filteredOn(r -> r.status() == EmailSendResult.Status.QUEUED).hasSize(1); + assertThat(outcomes) + .filteredOn(r -> r.status() == EmailSendResult.Status.DUPLICATE) + .hasSize(3); + assertThat(outcomes.stream().map(EmailSendResult::messageId).distinct()).hasSize(1); + } + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1); + } + + /******************************************************************************* + ** Replicas Share Rate Limit And Limited Provider Does Not Starve Others. + *******************************************************************************/ + @Test + void replicasShareRateLimitAndLimitedProviderDoesNotStarveOthers() throws Exception + { + EmailRuntime.config().withBatchSize(1); + SystemWrites.updateOne( + EmailTables.PROVIDER, new QRecord().withValue("id", 1).withValue("maxPerMinute", 1)); + var other = + H2Support.insert( + EmailTables.PROVIDER, + new QRecord() + .withValue("name", "other") + .withValue("kind", "SMTP") + .withValue("host", "127.0.0.1") + .withValue("port", 25) + .withValue("fromAddress", "sender@example.com") + .withValue("isDefault", false) + .withValue("enabled", true)); + for(int i = 0; i < 3; i++) + { + EmailService.send(request()); + } + var otherMessage = EmailService.send(request().withProviderName("other")); + var instance = QContext.getQInstance(); + try(var pool = Executors.newFixedThreadPool(2)) + { + List> futures = new ArrayList<>(); + for(int i = 0; i < 2; i++) + { + futures.add( + pool.submit( + () -> + { + QContext.init(instance, H2Support.admin()); + try + { + fake((e, settings) -> new EmailTransportResult("ok")).runOnce(); + } + catch(Exception e) + { + throw new RuntimeException(e); + } + finally + { + QContext.clear(); + } + })); + } + for(var f : futures) + { + f.get(20, TimeUnit.SECONDS); + } + } + assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(2); + assertThat(message(otherMessage.messageId()).getValueString("status")).isEqualTo("SENT"); + } + + /******************************************************************************* + ** Maintenance Recovers Disabled Provider Lease And Exhausted Attempts. + *******************************************************************************/ + @Test + void maintenanceRecoversDisabledProviderLeaseAndExhaustedAttempts() throws Exception + { + EmailRuntime.config().withMaxAttempts(1); + var queued = EmailService.send(request()); + var inFlight = + fake( + (e, settings) -> + { + try + { + SystemWrites.updateOne( + EmailTables.PROVIDER, + new QRecord().withValue("id", 1).withValue("enabled", false)); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(121), ZoneOffset.UTC)); + new EmailDispatcher().maintain(); + } + catch(Exception ex) + { + throw new RuntimeException(ex); + } + return (new EmailTransportResult("too-late")); + }); + assertThat(inFlight.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.NOT_SENT); + assertThat(message(queued.messageId()).getValueString("status")).isEqualTo("DEAD_LETTER"); + assertThat(message(queued.messageId()).getValueInstant("finishedDate")) + .isEqualTo(now.plusSeconds(121)); + assertThat(H2Support.all(EmailTables.ATTEMPT).getFirst().getValueString("outcome")) + .isEqualTo("LEASE_EXPIRED"); + } +} diff --git a/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java b/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java new file mode 100644 index 0000000..80588b9 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java @@ -0,0 +1,502 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email; + +import java.util.*; +import com.kingsrook.qbits.email.app.EmailApp; +import com.kingsrook.qbits.email.metadata.*; +import com.kingsrook.qbits.email.outbox.*; +import com.kingsrook.qbits.email.permissions.*; +import com.kingsrook.qbits.email.processes.*; +import com.kingsrook.qbits.email.support.*; +import com.kingsrook.qqq.backend.core.actions.tables.*; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.fields.*; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; +import org.junit.jupiter.api.*; +import static org.assertj.core.api.Assertions.*; + +/******************************************************************************* + ** Email Metadata Test. + *******************************************************************************/ +class EmailMetadataTest +{ + /******************************************************************************* + ** Setup. + *******************************************************************************/ + @BeforeEach + void setup() throws Exception + { + H2Support.setup(c -> + { + }); + } + + /******************************************************************************* + ** Teardown. + *******************************************************************************/ + @AfterEach + void teardown() + { + QContext.clear(); + } + + /******************************************************************************* + ** Producer Standards And Integration Fields. + *******************************************************************************/ + @Test + void producerStandardsAndIntegrationFields() throws Exception + { + var instance = QContext.getQInstance(); + assertThat(instance.getTables()).hasSize(7); + for(var table : instance.getTables().values()) + { + assertThat(table.getName()).matches("email_[a-z_]+"); + assertThat(table.getPermissionRules().getPermissionBaseName()).isEqualTo("email"); + assertThat(table.getBackendName()).isEqualTo("rdbms"); + assertThat(table.getIsHidden()).isFalse(); + for(var field : table.getFields().values()) + { + assertThat(field.getBackendName()).matches("[a-z][a-z0-9_]*"); + } + } + var children = instance.getApp("email").getChildren(); + assertThat(children).extracting(c -> c.getName()).containsExactlyElementsOf(EmailTables.MAIN); + assertThat(EmailTables.SYSTEM_WRITTEN) + .allSatisfy( + t -> + assertThat(instance.getTable(t).getDisabledCapabilities()) + .contains( + Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE)); + assertThat(instance.getMessagingProvider("qbitEmail").getType()).isEqualTo("QBIT_EMAIL"); + assertThat(EmailIntegrations.esbTableNames()) + .containsExactly(EmailTables.MESSAGE, EmailTables.TEMPLATE, EmailTables.SUPPRESSION); + assertThat(EmailIntegrations.apiTableNames()).isEqualTo(EmailIntegrations.esbTableNames()); + assertThat(EmailIntegrations.quickSearchTables()) + .allSatisfy( + t -> + assertThat(t.fieldWeights()) + .doesNotContainKeys("subject", "htmlBody", "textBody", "variablesSnapshot")); + assertThat( + new EmailApp.HomeRenderer() + .render( + new com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput())) + .isNotNull(); + assertThat(EmailRuntime.isInstalled()).isTrue(); + assertThat(EmailRuntime.config(new QInstance())).isNull(); + assertThat(EmailRuntime.config(null)).isNull(); + } + + /******************************************************************************* + ** Provider. + *******************************************************************************/ + QRecord provider(String name) + { + return (new QRecord() + .withValue("name", name) + .withValue("kind", "SMTP") + .withValue("host", "localhost") + .withValue("port", 25) + .withValue("fromAddress", "sender@example.com") + .withValue("enabled", true) + .withValue("isDefault", true)); + } + + /******************************************************************************* + ** Insert. + *******************************************************************************/ + QRecord insert(QRecord row) throws Exception + { + return (new InsertAction() + .execute(new InsertInput(EmailTables.PROVIDER).withRecords(List.of(row))) + .getRecords() + .get(0)); + } + + /******************************************************************************* + ** Validates Env Name Allowlist And Never Returns Values. + *******************************************************************************/ + @Test + void validatesEnvNameAllowlistAndNeverReturnsValues() throws Exception + { + EmailRuntime.config().withSecretResolver(n -> "top-secret-do-not-expose"); + assertThat(insert(provider("bad").withValue("secretEnvName", "literal-password")).getErrors()) + .isNotEmpty(); + assertThat(insert(provider("bad2").withValue("secretEnvName", "OTHER_SECRET")).getErrors()) + .isNotEmpty(); + assertThat( + insert(provider("domain").withValue("sendingDomain", "different.example")).getErrors()) + .isNotEmpty(); + assertThat(insert(provider("port").withValue("port", 0)).getErrors()).isNotEmpty(); + assertThat(insert(provider("no-default").withValue("isDefault", false)).getErrors()) + .isNotEmpty(); + var good = insert(provider("relay").withValue("secretEnvName", "EMAIL_SMTP_PASSWORD")); + assertThat(good.getErrors()).isNullOrEmpty(); + var queried = + new QueryAction().execute(new QueryInput(EmailTables.PROVIDER)).getRecords().getFirst(); + assertThat(queried.getValueString("secretEnvNameStatus")).isEqualTo("Set"); + assertThat(queried.getValueString("usernameEnvNameStatus")).isEqualTo("Not set"); + assertThat(queried.getValues().values().toString()).doesNotContain("top-secret"); + EmailRuntime.config() + .withSecretResolver( + n -> + { + throw new RuntimeException("secret in exception"); + }); + queried = + new QueryAction().execute(new QueryInput(EmailTables.PROVIDER)).getRecords().getFirst(); + assertThat(queried.getValueString("secretEnvNameStatus")).isEqualTo("Not set"); + assertThat(insert(provider("duplicate-default")).getErrors()).isNotEmpty(); + assertThat(insert(provider("secondary").withValue("isDefault", false)).getErrors()) + .isNullOrEmpty(); + var updated = + new UpdateAction() + .execute( + new UpdateInput(EmailTables.PROVIDER) + .withRecords( + List.of( + new QRecord() + .withValue("id", good.getValue("id")) + .withValue("label", "New label")))); + assertThat(updated.getRecords().getFirst().getErrors()).isNullOrEmpty(); + var invalid = + new UpdateAction() + .execute( + new UpdateInput(EmailTables.PROVIDER) + .withRecords( + List.of( + new QRecord() + .withValue("id", good.getValue("id")) + .withValue("enabled", false)))); + assertThat(invalid.getRecords().getFirst().getErrors()).isNotEmpty(); + var delete = + new DeleteAction() + .execute( + new DeleteInput(EmailTables.PROVIDER) + .withQueryFilter(EmailStore.equal("id", good.getValueInteger("id")))); + assertThat(H2Support.all(EmailTables.PROVIDER)).hasSize(2); + } + + /******************************************************************************* + ** Service Refuses Invalid Inputs And Rendering Escapes Raw Html. + *******************************************************************************/ + @Test + void serviceRefusesInvalidInputsAndRenderingEscapesRawHtml() throws Exception + { + H2Support.provider(25); + var good = + new EmailRequest() + .withTo("a@example.com") + .withSubject("s") + .withMarkdownBody("\n\n**Hello**"); + assertThat(EmailService.send(new EmailRequest().withTemplateName("later")).reasons()) + .contains("templates arrive in part 2"); + assertThat(EmailService.send(null).status()).isEqualTo(EmailSendResult.Status.REFUSED); + assertThat(EmailService.send(new EmailRequest()).status()) + .isEqualTo(EmailSendResult.Status.REFUSED); + assertThat(EmailService.send(new EmailRequest().withSubject("s").withTextBody("body")).status()) + .isEqualTo(EmailSendResult.Status.REFUSED); + assertThat( + EmailService.send( + new EmailRequest().withTo("not-address").withSubject("s").withTextBody("body")) + .status()) + .isEqualTo(EmailSendResult.Status.REFUSED); + assertThat( + EmailService.send( + new EmailRequest() + .withTo("a@example.com\r\nBcc:x@example.com") + .withSubject("s") + .withTextBody("body")) + .status()) + .isEqualTo(EmailSendResult.Status.REFUSED); + assertThat( + EmailService.send( + new EmailRequest() + .withTo("a@example.com") + .withSubject("s") + .withTextBody("body") + .withVariable("phi", "no")) + .status()) + .isEqualTo(EmailSendResult.Status.REFUSED); + var rendered = new AdHocEmailRenderer().render(good); + assertThat(rendered.html()).contains("<script>").doesNotContain("