Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc23fbdae7
|
||
|
|
40f6935c19
|
||
|
|
351a8070db | ||
|
|
f87e70dd53 | ||
|
|
061416be84 | ||
|
|
2688f02a9a
|
||
|
|
0aa9b5907e
|
||
|
|
f8f03b1416
|
||
|
|
40d54226b2
|
||
|
|
4988a0740f
|
||
|
|
6af5062aa7
|
||
|
|
17c7dc1140
|
No files matched your search
@@ -6,6 +6,15 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- Persisted unique RFC Message-IDs, stable across delivery attempts, with SMTP/Gmail SMTP, SES raw MIME and Postmark/SendGrid/Mailgun request propagation (SES replaces the header downstream).
|
||||
- Core-independent `EmailEffects` handoff, reference lookups by idempotency key or RFC Message-ID, refusal mapping and an RC.7 binding design sketch; offline retry and provider-contract tests.
|
||||
- Checkstyle now checks test sources as well as main sources; existing test imports and formatting corrected.
|
||||
- Provider sender addresses: child table `email_provider_sender_address` (validated lowercase address, display name, one default, enabled, note; unique per provider; domain bound to the provider's sending domain; admin-only), shown on the provider record. `EmailRequest.withFromAddressId`, `from_address_id` on messages, refusal of a from address that is not an enabled row of the resolved provider, and the provider `from_address` / `from_name` kept as the fallback.
|
||||
- Contingent possible-value filters on the new from-address pickers (send processes by `providerId`, Send Test Email by the acted-on provider, message `fromAddressId` by `providerId`) and on the attempt and event message pickers (by `providerId`) and the suppression source event (by `sourceMessageId`); the send processes also take `providerId`.
|
||||
- Phase (c): authenticated SNS, Postmark, SendGrid and Mailgun webhooks, durable inbox deduplication and Mailgun nonce replay protection; configurable Javalin routes.
|
||||
- Atomic scheduled event processing, per-recipient event history, monotonic delivery state, category-aware suppression, optional daily complaint alerts and FAILED inbox Reprocess with Review/Result.
|
||||
- One-click unsubscribe and escaped self-contained light/dark confirmation pages using the existing envelope-bound signer; account/security tokens refused.
|
||||
- Event-based home totals, PostgreSQL/H2 tracking DDL, provider setup instructions, test-only SNS certificate/key and real HTTP, crypto, parser, concurrency, retry and privacy tests. Preferences stays deferred until core #1162 reaches an RC.
|
||||
- Phase (a), part 2: versioned Markdown/HTML/MJML templates, typed variable schemas, escaped blocks/loops/fallbacks, branded light/dark layout and contrast validation.
|
||||
- New Version, Activate Version and version-bound Approve For PHI actions, edit permissions, immutable history and parent-scoped version selectors.
|
||||
- Sender-scoped record variables, classification stand-in, HMAC snapshot tokens, AES-GCM bodies, encrypted query displays and protected resend.
|
||||
|
||||
@@ -4,9 +4,9 @@ One email service for a QQQ system: a durable outbox (`email_message`) with retr
|
||||
letters, idempotency keys, rate limits and provider health; SMTP, Amazon SES, Postmark,
|
||||
SendGrid, Mailgun and Gmail SMTP transports; versioned Markdown, HTML and MJML templates with
|
||||
dark-mode-safe branding; classified variables and PHI protection; preview widgets; a
|
||||
real-data home; and the core messaging bridge. Every send goes through the outbox.
|
||||
real-data home; delivery events, suppression and one-click unsubscribe; and the core messaging bridge. Every send goes through the outbox.
|
||||
|
||||
The [design](docs/DESIGN.md) also covers delivery tracking, unsubscribe, Preferences, the
|
||||
The [design](docs/DESIGN.md) also specifies the deferred Preferences UI (core #1162),
|
||||
consumer migrations and inbound mail. Preview uses the standard widget endpoint; the
|
||||
dedicated preview API arrives with QQQ RC.6. Attachments are refused until
|
||||
qbit-object-storage is wired in.
|
||||
@@ -45,11 +45,14 @@ in a static variable.
|
||||
|---|---|---|
|
||||
| `email_message` | Messages | Qbit only; people use record actions |
|
||||
| `email_message_attempt` | Attempts on a message | Qbit only |
|
||||
| `email_event` | Events on a message | Qbit only; readable with `email.view` |
|
||||
| `email_webhook_inbox` | Inbox on a provider | Qbit only; readable with `email.view`; Reprocess requires `email.manage` |
|
||||
| `email_template` | Templates | `email.manage`; template edit grants also apply |
|
||||
| `email_template_version` | Versions on a template | Draft edits only; lifecycle uses record actions |
|
||||
| `email_brand` | Brands | `email.manage`; template edit grants also apply |
|
||||
| `email_suppression` | Suppressions | `email.manage`; changing/removing a complaint needs `email.admin` |
|
||||
| `email_provider` | Providers, with child Messages | `email.admin`, including reads and test sends |
|
||||
| `email_provider_sender_address` | Sender Addresses on a provider | `email.admin`, including reads |
|
||||
|
||||
`email.view` reads messages, templates, versions, brands and suppressions.
|
||||
`email.send` permits the Java service, bridge and backend send process. Reading
|
||||
@@ -68,7 +71,7 @@ insert/update/delete capabilities. Labels contain no status decorations.
|
||||
## Providers and credentials
|
||||
|
||||
See the [provider setup guide](docs/PROVIDERS.md) for row fields, DNS/domain setup,
|
||||
limits, Gmail app passwords and the webhook configuration planned for phase (c).
|
||||
limits, Gmail app passwords and provider webhook setup.
|
||||
|
||||
| Kind | Transport | Status |
|
||||
|---|---|---|
|
||||
@@ -85,6 +88,30 @@ enabled, exactly one must be the enabled default. Switch defaults in a batch upd
|
||||
clearing the old default first. A nullable unique `default_slot` backs the validation.
|
||||
Providers with message history cannot be deleted; disable them instead.
|
||||
|
||||
**Sender addresses.** A provider can own any number of verified from addresses, kept in
|
||||
`email_provider_sender_address` and shown as a Sender Addresses list on the provider record
|
||||
(not in the nav; `email.admin` reads and writes, like the provider). Each row has an
|
||||
`address` (validated, stored lowercase, line breaks refused), an optional `displayName`,
|
||||
`isDefault`, `enabled` and a `note`. An address is unique per provider, at most one enabled
|
||||
row per provider is the default, and when the provider has a `sendingDomain` every address must
|
||||
be in that domain (changing the domain with other-domain rows is refused). Deleting a provider
|
||||
deletes its rows. Existing databases add the table and the nullable
|
||||
`email_message.from_address_id` column from `docs/ddl/email-postgres.sql`.
|
||||
|
||||
The provider's own `fromAddress` / `fromName` stay the fallback, so nothing changes until a
|
||||
row exists. A row marked default wins over the fallback. On a send, `withFromAddressId(id)`
|
||||
(or `fromAddressId` on the send processes) must name an enabled row of the resolved provider,
|
||||
and once a provider has rows an explicit `withFrom(...)` must match one of them by address;
|
||||
either failure is `REFUSED` with "That from address is not set up for provider <name>". With
|
||||
no rows, `withFrom(...)` behaves as before. The chosen row is recorded on the message as
|
||||
`fromAddressId`, and a resend reuses it.
|
||||
|
||||
The from address pickers are contingent: `fromAddressId` on the send processes lists only the
|
||||
rows of the chosen `providerId` (`providerName` is still accepted for API callers, and the two
|
||||
must agree), and the Send Test Email action lists only the rows of the provider being tested.
|
||||
That action's form has no record id, so a prepare step publishes the single selected provider id
|
||||
as the process value `possibleValueFilterValueId`, which the picker's filter reads after `${input.id}`.
|
||||
|
||||
For an unauthenticated relay, use kind `SMTP`, host/port, security `NONE`, and a
|
||||
`fromAddress`. Leave credential names empty. Authenticated SMTP uses `usernameEnvName`
|
||||
and `secretEnvName`; security is `NONE`, `STARTTLS` (required) or `TLS`. Timeouts are
|
||||
@@ -127,8 +154,8 @@ names and STARTTLS. GMAIL_API is refused with
|
||||
HTTP transports use fixed HTTPS endpoints, 10-second connect and 30-second request
|
||||
timeouts, and no redirects. HTTP 429/5xx retry with optional Retry-After; account
|
||||
and credential failures pause the provider. Tracking is explicitly controlled by
|
||||
`trackOpens` / `trackClicks`, off by default. Outbound tracking settings do not
|
||||
install the event webhook receiver; that arrives in phase (c).
|
||||
`trackOpens` / `trackClicks`, off by default. The producer registers the phase-(c)
|
||||
receiver idempotently when the provided Javalin middleware is present.
|
||||
|
||||
## Sending
|
||||
|
||||
@@ -241,6 +268,35 @@ sanitizer removes iframes and style blocks, so this preview is not an exact mail
|
||||
emulator. Draft MJML is compiled in memory for preview when a compiler is configured;
|
||||
preview never stores the result or changes the version.
|
||||
|
||||
### Templates from another qbit
|
||||
|
||||
A qbit that sends its own mail ships the template with its code instead of asking an
|
||||
administrator to create it. Describe it with `EmailTemplateDefinition` and call
|
||||
`EmailTemplates.ensure` at startup or before first use:
|
||||
|
||||
```java
|
||||
boolean created = EmailTemplates.ensure(new EmailTemplateDefinition()
|
||||
.withName("order_ready")
|
||||
.withLabel("Order ready")
|
||||
.withCategory(EmailCategory.TRANSACTIONAL)
|
||||
.withSubject("Order {{order_number}} is ready")
|
||||
.withBody("Hi {{customer_name}}, order {{order_number}} is ready for pickup.")
|
||||
.withVariable("order_number", "Order number", "STRING", true, "PUBLIC")
|
||||
.withVariable("customer_name", "Customer name", "STRING", true, null)
|
||||
.withSampleVariables("{\"order_number\":\"A-1\",\"customer_name\":\"Pat\"}")
|
||||
.withOwnerQbit("orders")
|
||||
.withEditPermissionName("orders.edit"));
|
||||
```
|
||||
|
||||
`kind` defaults to MARKDOWN (MJML and HTML are also accepted), `withTextBody` is
|
||||
optional, and `withVariablesSchema(json)` replaces `withVariable` when you already have
|
||||
the JSON. When no template of that name exists, `ensure` validates the definition like
|
||||
Activate Version (syntax, schema, a render with the samples, MJML compile), creates the
|
||||
template and an ACTIVE version 1 as the qbit, and returns true. When one exists it does
|
||||
nothing and returns false, so people's edits and later versions are never overwritten;
|
||||
the unique name settles concurrent callers. An invalid definition throws
|
||||
`IllegalArgumentException` with plain reasons and writes nothing.
|
||||
|
||||
## PHI and PII
|
||||
|
||||
`withRecordVariable` reads through `QueryAction` in the sender's QContext; it does not
|
||||
@@ -279,15 +335,17 @@ and content. Keys resolve through `secretResolver`; no secret values are stored.
|
||||
|
||||
NOTIFICATION/MARKETING templates prepare footer links and headers using
|
||||
`QQQ_EMAIL_UNSUBSCRIBE_KEY` and `publicBaseUrl`; marketing refuses without them.
|
||||
Tokens bind the message, recipient-envelope hash and category. The phase-(c)
|
||||
unsubscribe route and Preferences are still pending: signed links are prepared here,
|
||||
but this qbit alone does not yet serve the unsubscribe destination. Do not deploy
|
||||
opt-out mail without that receiving handler.
|
||||
Tokens bind the message, recipient-envelope hash and category. The receiving route
|
||||
is implemented: GET only shows a light/dark confirmation page; explicit POST upserts
|
||||
category suppression and records UNSUBSCRIBED. Invalid or account/security tokens
|
||||
return 404. The legacy token covers every To/Cc/Bcc recipient, so use one mailbox
|
||||
per opt-out message for individual preferences. Preferences UI remains deferred
|
||||
until core #1162 ships in an RC.
|
||||
|
||||
The home uses viewer-scoped counts and queries, UTC hour/day charts, provider health
|
||||
(admins), and 20 recent messages with masked recipients. No sample data or sampled
|
||||
counts are used. Bounce/complaint totals use dedicated event timestamps, populated
|
||||
by the future webhook phase or host integrations.
|
||||
counts are used. Delivered/bounced/complained totals count recipient events and
|
||||
use their occurred dates; sent/dead-letter totals count messages.
|
||||
|
||||
## Delivery and actions
|
||||
|
||||
@@ -331,12 +389,49 @@ Providers offer Send Test Email with a recipient input and a reported outcome.
|
||||
record bindings use `{"patient":{"table":"patient","id":123}}` in
|
||||
`recordVariablesJson`. Output includes `messageId`, `status`, and `reasons`.
|
||||
|
||||
Hidden schedules are `email_dispatch` (15 s by default), `email_maintain` (300 s, lease
|
||||
Hidden schedules include `email_process_webhooks` (30 s), `email_dispatch` (15 s by default), `email_maintain` (300 s, lease
|
||||
recovery including disabled providers, and health probes), and `email_retention`
|
||||
(daily). Retention clears final-message bodies after 30 days, preserving `bodySha256`;
|
||||
message deletion is disabled by default. `finishedDate` records finalization independently
|
||||
of enqueue and later retention updates.
|
||||
|
||||
## Delivery tracking
|
||||
|
||||
Apply the new `email_webhook_inbox` and `email_event` DDL plus indexes for an existing
|
||||
phase-(a)/(b) database. Configure the authenticated HTTPS endpoints using
|
||||
[the provider guide](docs/PROVIDERS.md#event-webhooks). Verification supports SNS
|
||||
RSA v1/v2 with pinned topic and AWS certificate hosts, Postmark Basic auth,
|
||||
SendGrid ECDSA and Mailgun HMAC with durable nonce replay protection.
|
||||
|
||||
The 1 MiB receiver persists before acknowledging. DB transactions serialize
|
||||
processing across replicas and commit event rows, message state and suppression
|
||||
atomically. Events in a payload are applied independently: matched events commit at
|
||||
once, and only unmatched events (or events whose message is still SENDING) retry the
|
||||
row with configured backoff. After six attempts they are skipped and the row finishes
|
||||
PROCESSED with a note ("N events did not match a message"). Rows that cannot be parsed
|
||||
or written fail after six attempts; **Reprocess** on FAILED inbox rows uses the
|
||||
standard Review/Result flow. An event whose recipient is not on the matched message
|
||||
is ignored, and an SES bounce or complaint without its recipient list is skipped.
|
||||
SNS notifications older than an hour or more than five minutes ahead are refused, and
|
||||
certificate fetch failures are cached for five minutes per URL. Disabled tracking is
|
||||
ignored. Per-event recipient keys deduplicate
|
||||
overlapping batches. All new service state is in the DB; certificate caches are in
|
||||
memory. No runtime filesystem or new dependencies are used.
|
||||
|
||||
Hard bounce/complaint suppress ALL; provider unsubscribe suppresses the message's
|
||||
opt-out category. Message status never regresses from COMPLAINED to BOUNCED or
|
||||
DELIVERED. Optional `email.complaint` alerts warn after more than three complaints
|
||||
in 24 hours, with daily deduplication. Provider details mask addresses and are capped
|
||||
at 500 characters; raw authenticated inbox payloads are sensitive DB data.
|
||||
|
||||
NOTIFICATION and MARKETING messages with unsubscribe configured require exactly one
|
||||
To recipient and no Cc/Bcc; send one message per recipient. The token binds that
|
||||
mailbox, and an unsubscribe POST suppresses only that address for the message's
|
||||
category. Multi-recipient requests are refused before suppression filtering.
|
||||
TRANSACTIONAL and SECURITY messages, and NOTIFICATION messages without unsubscribe
|
||||
configured, may still have multiple recipients. Existing single-recipient links
|
||||
remain valid; old multi-recipient links are rejected.
|
||||
|
||||
## Core messaging bridge
|
||||
|
||||
The producer adds messaging provider `qbitEmail` (configurable), type `QBIT_EMAIL`.
|
||||
@@ -375,6 +470,7 @@ All settings have fluent `withX` setters and getters.
|
||||
| `backendName` | Required existing backend |
|
||||
| `schedulerName` | Optional existing scheduler; required for automatic dispatch |
|
||||
| `publicBaseUrl` | Public app URL for signed unsubscribe links |
|
||||
| `webhookBasePath` | `/qqq/hooks/email`; absolute path outside `/qqq/v1` |
|
||||
| `mjmlCompiler` | Optional `MjmlCompilerInterface`; required for MJML activation |
|
||||
| `classificationOverrides` | Map of `table.field` to `EmailClassifications.Level` for RC.5 |
|
||||
| `defaultProviderName` | Optional named provider override; otherwise enabled default row |
|
||||
@@ -405,6 +501,26 @@ concurrent idempotency, shared rate budgets, health recovery and process screens
|
||||
The tests also exercise template lifecycle, parent-scoped possible values, rendering,
|
||||
PHI canary egress, encryption, preview widgets and home data.
|
||||
|
||||
## Durable jobs
|
||||
|
||||
`EmailEffects.send(request, effectIdempotencyKey)` hands a message to the committed
|
||||
email outbox exactly once per key and returns an `EmailEffects.Result` containing
|
||||
`EmailEffectOutcome` and an `EmailMessageRef`. `QUEUED_OR_DUPLICATE` means the outbox
|
||||
accepted responsibility; `REFUSED` means a definite refusal, including suppression.
|
||||
Use `findByIdempotencyKey` or `findByRfcMessageId` to reconcile a handoff. These are
|
||||
trusted system reads returning only references. Database failures propagate.
|
||||
Requests must not carry a caller transaction; the effect dispatcher calls after its
|
||||
own chunk commit. Existing `EmailService` transaction support is unchanged.
|
||||
|
||||
Each new message stores a unique `rfcMessageId` once. SMTP (including the Gmail preset),
|
||||
Postmark, SendGrid and Mailgun carry it on every attempt. SES receives it in the raw
|
||||
request but replaces it downstream; every transport retains `X-QQQ-Email-Id`.
|
||||
Message-ID enables correlation and receiver deduplication where supported; it does
|
||||
not promise exactly-once inbox delivery.
|
||||
|
||||
See [the handoff contract, provider references, schema upgrade and RC.7 binding sketch](docs/DURABLE-JOBS.md).
|
||||
The build remains on released QQQ **4.1.0-RC.5**, without durable-jobs dependencies.
|
||||
|
||||
## License
|
||||
|
||||
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
|
||||
+56
-14
@@ -1,6 +1,6 @@
|
||||
# qbit-email design
|
||||
|
||||
**Status:** phase (a) implemented; later phases remain specification. Phases (a) to (d) build outbound; phase (e) is inbound (section 12, design only).
|
||||
**Status:** phases (a), (b) and (c) implemented, except Preferences (8.3), deferred until core #1162 ships in an RC. Phase (d) consumer migrations and phase (e) inbound remain specification.
|
||||
**Brief:** owner decisions of 2026-10-13 (providers, outbound first, both template kinds).
|
||||
**Owner decisions (2026-10-12): all recommendations accepted.** Every "Recommended" choice and
|
||||
every decision below stands; it is adjusted once it can be seen working.
|
||||
@@ -54,6 +54,7 @@ EmailSendResult result = EmailService.send(new EmailRequest()
|
||||
.withRecordVariable("order", "order", 1234) // a record, read as the sender; classification known
|
||||
.withCategory(EmailCategory.NOTIFICATION) // default: the template's
|
||||
.withProviderName("pmg") // optional; else the default provider
|
||||
.withFromAddressId(4) // optional enabled email_provider_sender_address row of that provider
|
||||
.withBrandName("kof22") // optional; else the template's, else the default brand
|
||||
.withIdempotencyKey("alerts:notif:991") // optional; a repeat returns the existing message
|
||||
.withLinkedRecord("alerts_alert", 991) // shown as a link on the message record
|
||||
@@ -73,7 +74,8 @@ EmailSendResult result = EmailService.send(new EmailRequest()
|
||||
|
||||
### 2.2 Process `email_send_templated`
|
||||
The same request as a backend process (template, to, cc, bcc, variables JSON, linked table and
|
||||
id, idempotency key, provider, brand) for scripts, the API and other qbits. Not in the nav;
|
||||
id, idempotency key, provider by `providerId` or `providerName`, `fromAddressId` filtered by the
|
||||
chosen `providerId`, brand) for scripts, the API and other qbits. Not in the nav;
|
||||
callable through the API and `/qqq/v1/processes`.
|
||||
|
||||
### 2.3 Permissions
|
||||
@@ -122,7 +124,16 @@ They are never stored, logged, audited, put in process output or returned. The r
|
||||
whether each named variable is set (yes or no), never the value.
|
||||
|
||||
**Record action "Send Test Email"** (`email.admin`): queues a fixed test message to a given
|
||||
address through that provider and reports the outcome on the standard result screen.
|
||||
address through that provider and reports the outcome on the standard result screen. An optional
|
||||
`fromAddressId` picker lists only that provider's sender addresses.
|
||||
|
||||
**Sender addresses** (`email_provider_sender_address`, a child list on the provider, `email.admin`):
|
||||
provider_id, address (validated, lowercase), display_name, is_default (at most one enabled default
|
||||
per provider), enabled, note; unique (provider_id, address); the domain must equal `sending_domain`
|
||||
when that is set. The provider's `from_address` / `from_name` remain the fallback; an enabled default
|
||||
row wins when present. A chosen id, or an explicit from text once the provider has rows, must match an
|
||||
enabled row of the resolved provider or the send is `REFUSED` ("That from address is not set up for
|
||||
provider <name>"). The row used is kept as `email_message.from_address_id`.
|
||||
|
||||
### 3.2 `EmailTransport` SPI (in the qbit)
|
||||
```java
|
||||
@@ -174,6 +185,7 @@ transient; other 4xx are permanent.
|
||||
| variables_snapshot | JSON, redacted per section 7.3 |
|
||||
| html_body, text_body | rendered at queue time so retries send the same bytes; encrypted for `phi_approved`; cleared after `bodyRetentionDays` (default 30) once final, keeping `body_sha256` |
|
||||
| idempotency_key | unique when present |
|
||||
| rfc_message_id | STRING(500), unique; stored once at enqueue and reused on every attempt; see [durable job effects](DURABLE-JOBS.md) |
|
||||
| provider_id, provider_message_id | |
|
||||
| attempt_count, next_attempt_date, lease_expires_date, last_error, sent_date, delivered_date, dead_letter_date | |
|
||||
| linked_table_name, linked_record_id | rendered as a link on the record screen |
|
||||
@@ -205,7 +217,9 @@ error) and `email_event` (section 6).
|
||||
content), Cancel (queued or waiting), Send Now (waiting). Standard streamed-ETL Review and Result
|
||||
screens.
|
||||
- **Retention:** a daily job clears bodies after `bodyRetentionDays` and deletes final messages
|
||||
older than `messageRetentionDays` (0 = keep forever; kof22 keeps).
|
||||
older than `messageRetentionDays` (0 = keep forever; kof22 keeps). NOTIFICATION and MARKETING
|
||||
messages are never deleted, because their unsubscribe links resolve through the message row
|
||||
(8.2); at `messageRetentionDays` retention only clears their bodies and variable snapshot.
|
||||
|
||||
## 5. Templates and branding
|
||||
|
||||
@@ -283,11 +297,10 @@ id, or unsaved source, plus variables) returns subject, html, text and warnings.
|
||||
token snapshot, linking to the original message for the token binding.
|
||||
- The home follows the newer Alerts pattern: standard QQQ chart widgets plus inline,
|
||||
theme-neutral cards. All queries/counts use the viewer's session. Bounce/complaint
|
||||
dates have dedicated columns for phase-(c) events; no dates or metrics are invented.
|
||||
- Part 2 signs opt-out links and prepares headers; the receiving unsubscribe handler
|
||||
remains phase (c). Without its key/base URL, notification omits opt-out links and
|
||||
dates are populated by phase-(c) events; home totals count those recipient events.
|
||||
- Part 2 signs opt-out links and prepares headers; phase (c) now serves the receiver. Without its key/base URL, notification omits opt-out links and
|
||||
headers, while marketing refuses as specified in 8.2. Do not use opt-out delivery
|
||||
without a receiving handler. The token's address hash covers the recipient envelope.
|
||||
without a receiving handler. The token's address hash binds the single recipient mailbox.
|
||||
|
||||
## 6. Delivery tracking
|
||||
|
||||
@@ -312,6 +325,32 @@ DEFERRED, BOUNCED, COMPLAINED, OPENED, CLICKED, UNSUBSCRIBED, REJECTED), bounce_
|
||||
UNDETERMINED), recipient, occurred_date, detail (the provider's reason, truncated; never body
|
||||
content), url (clicks only). Opens and clicks are stored only when the provider row enables them.
|
||||
|
||||
### 6.3 Phase (c) implementation
|
||||
- Inbox states: PENDING, PROCESSED, FAILED, IGNORED. Retry after configured backoff,
|
||||
FAILED after six attempts, and a standard Reprocess action resets FAILED to PENDING.
|
||||
- Events in one payload are applied independently. Only unmatched events (or events whose
|
||||
message is still SENDING) defer the row; on the sixth attempt they are skipped and the row is
|
||||
PROCESSED with the note "N events did not match a message". Suppression applies only to a
|
||||
recipient in the matched message's to, cc or bcc; an SES bounce or complaint without its
|
||||
recipient list is skipped, never widened to all destinations. SNS notifications must carry a
|
||||
Timestamp no older than 1 hour and no more than 5 minutes ahead, and certificate fetch
|
||||
failures are cached for 5 minutes per URL.
|
||||
- A short DB transaction locks the common provider gate then the actual provider;
|
||||
it rechecks the observed inbox attempt and commits all changes together. No worker
|
||||
filesystem, extra claim table, in-process scheduler or network operation is needed
|
||||
during processing. A crash rolls back the transaction and releases the claim.
|
||||
- `email_event.event_key` provides unique recipient/event deduplication across
|
||||
overlapping batches, alongside inbox uniqueness on provider/event and provider/nonce.
|
||||
- Message correlation always stays within the authenticated provider. Unknown message
|
||||
ids retry (including callbacks before SENT); unsupported event kinds are ignored.
|
||||
- Opens/clicks require current provider flags and are also dropped for PHI-approved
|
||||
messages. Event details mask addresses before truncation to 500 characters.
|
||||
- The unsubscribe signer binds one mailbox. A valid POST suppresses only that address
|
||||
for MARKETING/NOTIFICATION; the page never displays addresses. Legacy links remain
|
||||
valid only for messages with one To recipient and no Cc/Bcc.
|
||||
- PostgreSQL and H2 definitions, crypto fixtures and real random-port Javalin tests
|
||||
are included. Certificate caching is memory-only. Setup is in `PROVIDERS.md`.
|
||||
|
||||
## 7. PHI and PII
|
||||
|
||||
### 7.1 Classification
|
||||
@@ -371,6 +410,7 @@ MANUAL), source_message_id, source_event_id, note, created_date, expires_date (o
|
||||
respects HARD_BOUNCE and COMPLAINT.
|
||||
|
||||
### 8.2 One-click unsubscribe (RFC 8058, RFC 2369)
|
||||
- NOTIFICATION or MARKETING messages carrying an unsubscribe link/header must have exactly one To recipient and no Cc/Bcc; send one message per recipient.
|
||||
- MARKETING and NOTIFICATION messages carry
|
||||
`List-Unsubscribe: <https://{publicBaseUrl}/qqq/hooks/email/unsubscribe/{token}>` and
|
||||
`List-Unsubscribe-Post: List-Unsubscribe=One-Click`, and the footer links to the same URL.
|
||||
@@ -379,11 +419,11 @@ MANUAL), source_message_id, source_event_id, note, created_date, expires_date (o
|
||||
MARKETING sends are refused. Tokens do not expire (RFC 8058 expects the link to keep working).
|
||||
- `POST` with `List-Unsubscribe=One-Click` records the suppression and returns 200. `GET` returns a
|
||||
minimal page with one "Unsubscribe" button that POSTs, because mail scanners prefetch GETs; then
|
||||
"You are unsubscribed from {category}", with a link to Preferences.
|
||||
"You are unsubscribed from {category}". A Preferences link awaits core #1162.
|
||||
- An `UNSUBSCRIBED` event is recorded on the message.
|
||||
|
||||
### 8.3 Preferences (core #1162)
|
||||
The qbit registers a Preferences section `email` ("Email") whose provider
|
||||
**Deferred: core #1162 is not in an RC.** The planned qbit registers a Preferences section `email` ("Email") whose provider
|
||||
(`EmailPreferenceProvider implements UserPreferenceProviderInterface`) maps the signed-in user's
|
||||
address (`UserPreferenceRequestContext.getUserEmailAddress()`, never an address from the request)
|
||||
to `email_suppression` rows: one toggle per opt-out-able category (NOTIFICATION,
|
||||
@@ -452,13 +492,15 @@ section's description explains the split with qbit-alerts' Notifications section
|
||||
| `email_brand` | yes | people |
|
||||
| `email_suppression` | yes | people and the qbit |
|
||||
| `email_provider` | yes | admins |
|
||||
| `email_provider_sender_address` | relation (on the provider) | admins |
|
||||
| `email_webhook_inbox` | relation (on the provider) | the qbit |
|
||||
| `email_inbound`, `email_inbound_attachment` | phase (e) | the qbit |
|
||||
|
||||
Postgres DDL in `docs/ddl/email-postgres.sql` (phase a).
|
||||
Postgres DDL in `docs/ddl/email-postgres.sql` (phases a-c).
|
||||
|
||||
## 14. Configuration (`EmailQBitConfig`)
|
||||
backendName (required), schedulerName (required to send), publicBaseUrl (unsubscribe links),
|
||||
backendName (required), schedulerName (required for automatic processing), publicBaseUrl (unsubscribe links),
|
||||
webhookBasePath (`/qqq/hooks/email`, shared by tracking and unsubscribe),
|
||||
defaultProviderName, bridgeProviderName (`qbitEmail`), secretResolver, allowedEnvNames,
|
||||
dispatchIntervalSeconds (15), batchSize (100), maxAttempts (6), retryMinutes, leaseSeconds (120),
|
||||
bodyRetentionDays (30), messageRetentionDays (0), mjmlCompiler, classificationOverrides, appName
|
||||
@@ -469,12 +511,12 @@ and appLabel, securityFields and recordSecurityLocks (multi-tenant hosts).
|
||||
|---|---|---|
|
||||
| (a) | tables, outbox and dispatcher, SMTP and SES, Markdown/HTML/MJML templates and versions, brands and the layout, preview, EmailService, the process, the core bridge, PHI refusal and redaction, the app home | GreenMail SMTP sink end to end; SES through a mocked `SesV2Client`; the claim race (two runners, each message sent once); retry, dead letter, idempotency; rendering (variables, escaping, `#if`/`#each`, plain text, dark-mode contrast); PHI refusal; snapshot redaction; the canary test; producer standards (names, nav, permissions) |
|
||||
| (b) | Postmark, SendGrid, Mailgun, Gmail SMTP | HTTP mocks per provider (success, 4xx permanent, 429/5xx transient); the Gmail preset. HTML/MJML moved into part (a), part 2 at owner request. |
|
||||
| (c) | webhook route and inbox, events, suppression, unsubscribe, Preferences section | signature verification per provider (valid, tampered, stale, missing secret); the SNS certificate host rule; suppression on hard bounce and complaint; one-click POST and the GET page; category rules |
|
||||
| (c) | webhook route and inbox, events, suppression, unsubscribe (Preferences deferred) | signature verification per provider (valid, tampered, stale, missing secret); the SNS certificate host rule; suppression on hard bounce and complaint; one-click POST and the GET page; category rules |
|
||||
| (d) | consumer migrations | each consumer's own tests; a contract test of `EmailService` |
|
||||
| (e) | inbound | design only for now |
|
||||
|
||||
Dependencies: phase (a) builds on QQQ 4.1.0-RC.5 with the classification stand-in, and moves to
|
||||
RC.6 for `DataClassification` and Preferences.
|
||||
later RCs for `DataClassification` and Preferences when their APIs are released.
|
||||
|
||||
## 16. Owner steps (outside code)
|
||||
- **PMG (kof22):** none; the relay already accepts the cluster.
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
# Durable job email effects
|
||||
|
||||
This implements the email-side contract for QRun-IO/qqq#1191, spec section 2.13,
|
||||
without importing core jobs types. QQQ stays at released 4.1.0-RC.5. The
|
||||
`EmailJobEffectTarget implements EffectTarget` binding is deferred until RC.7 ships.
|
||||
|
||||
## Stored identity and schema upgrade
|
||||
|
||||
`email_message.rfc_message_id` (`rfcMessageId` in metadata) is a unique STRING(500)
|
||||
shown in the message T2 details. Before insertion, the service stores
|
||||
`<token@domain>`, where `token` is the first 32 lowercase hex characters of SHA-256
|
||||
of the UTF-8 idempotency key. Without a key, the seed is `qbit-email:` plus a fresh
|
||||
UUID, since the row id is not yet allocated. The domain is the normalized From
|
||||
mailbox domain, excluding display names; unavailable or unsuitable domains use
|
||||
`qbit-email.local`. The key itself is never included in the header.
|
||||
|
||||
The dispatcher reads this stored value on every attempt. JavaMail's
|
||||
`updateMessageID()` is overridden so repeated `saveChanges()` calls cannot replace
|
||||
it. A manual **Resend** creates a new outbox row and a new identity; a delivery retry
|
||||
keeps its identity. `providerMessageId` remains the provider's response identifier,
|
||||
distinct from `rfcMessageId` and the numeric outbox `id`.
|
||||
|
||||
New installations use [email-postgres.sql](ddl/email-postgres.sql). Existing
|
||||
PostgreSQL installations must pause enqueue and dispatch, then apply
|
||||
[email-message-id-upgrade.sql](ddl/email-message-id-upgrade.sql) before starting
|
||||
this version. The migration backfills old rows using the existing key, or
|
||||
`qbit-email:` plus the already allocated row id, and the stored From domain.
|
||||
It cannot recover previously generated random transport Message-IDs. Review host
|
||||
schema adaptations first; host security columns are unaffected.
|
||||
|
||||
## Provider behavior
|
||||
|
||||
| Transport | Request mechanism | Identity preservation |
|
||||
|---|---|---|
|
||||
| SMTP / Gmail SMTP preset | MIME `Message-ID` | Sent unchanged by this transport; downstream relay policy still applies |
|
||||
| SES v2 | Raw MIME `Message-ID` | **SES replaces Message-ID downstream** |
|
||||
| Postmark | `Headers: [{Name: "Message-ID", Value: "<...>"}]` | Custom Message-ID supported |
|
||||
| SendGrid | `headers: {"Message-ID": "<...>"}` | Sent as a custom header |
|
||||
| Mailgun | MIME Message-ID plus multipart `h:Message-Id` | Sent in the `/messages.mime` request |
|
||||
|
||||
All transports retain `X-QQQ-Email-Id` with the numeric outbox id. Postmark's
|
||||
`MessageID`, SendGrid's response `X-Message-Id`, and SES's returned id are provider
|
||||
references, not our RFC header. A fixed header enables recognition and receiver
|
||||
deduplication where supported, but is not an exactly-once inbox guarantee.
|
||||
|
||||
Provider references: [Postmark custom Message-ID](https://postmarkapp.com/developer/user-guide/send-email-with-api/send-a-single-email),
|
||||
[SendGrid custom headers](https://www.twilio.com/docs/sendgrid/api-reference/mail-send/mail-send),
|
||||
[Mailgun sending parameters](https://documentation.mailgun.com/docs/mailgun/api-reference/send/mailgun/messages/post-v3--domain-name--messages),
|
||||
and [SES header replacement](https://docs.aws.amazon.com/ses/latest/APIReference/API_SendRawEmail.html).
|
||||
|
||||
## Handoff and reconcile
|
||||
|
||||
`EmailEffects.send(request, effectIdempotencyKey)` sets the request's key, refusing
|
||||
a different pre-existing key, a missing/blank/overlength effect key, a null request
|
||||
or a caller transaction. The service performs its usual validation and send
|
||||
permission checks. Success is returned only after the outbox transaction commits;
|
||||
there is no transport call here. Use ordinary `EmailService.send` for caller-owned
|
||||
or cross-backend deferred transactions.
|
||||
|
||||
`EmailEffects.Result(outcome, message)` contains an `EmailEffectOutcome` and a
|
||||
nullable `EmailMessageRef(Integer id, String status, String rfcMessageId,
|
||||
String providerMessageId)`:
|
||||
|
||||
- `QUEUED_OR_DUPLICATE`: one committed row accepted the handoff, including replays.
|
||||
- `REFUSED`: invalid or suppressed request. A suppressed audit row has a reference;
|
||||
refusals without a stored row have a null reference. Replaying a suppressed row
|
||||
stays refused even though `EmailService` returns `DUPLICATE`.
|
||||
- Database/query failures throw `QException`; callers must retry or reconcile,
|
||||
never turn an uncertain outcome into a permanent refusal.
|
||||
|
||||
`findByIdempotencyKey(key)` and `findByRfcMessageId(messageId)` return
|
||||
`Optional<EmailMessageRef>` from system reads, like other internal outbox reads.
|
||||
They return empty for null/blank/missing values. These are trusted backend APIs,
|
||||
not viewer-scoped public endpoints. Lookups reflect current status and provider
|
||||
reference and do not expose bodies, recipients or variables.
|
||||
|
||||
The effect's `SENT` means successful durable handoff, not mailbox delivery. The
|
||||
email dispatcher owns later delivery/retry failures. Reconcile acknowledges an
|
||||
existing outbox row (including an audit row); it must never create another email
|
||||
merely because that row is terminal. Retain message rows and idempotency keys for
|
||||
at least the maximum effect retry/reconciliation horizon.
|
||||
|
||||
## RC.7 binding design note (not compiled)
|
||||
|
||||
Mapping: `deliver -> QUEUED_OR_DUPLICATE => SENT(ref.id)`, `REFUSED => FAILED`;
|
||||
`reconcile -> findByIdempotencyKey present => FOUND(ref.id), absent => NOT_FOUND`.
|
||||
Exceptions escape to core's retry/unknown handling. Payload decoding must preserve
|
||||
the existing PHI/variable policies and must not attach a caller transaction.
|
||||
|
||||
This 25-line sketch is documentation-only. Match core names and signatures to the
|
||||
published RC.7 contract when it ships; `decode` represents payload conversion.
|
||||
|
||||
```java
|
||||
// Future RC.7 binding; no core jobs imports in the RC.5 implementation.
|
||||
final class EmailJobEffectTarget implements EffectTarget
|
||||
{
|
||||
// SENT acknowledges the committed qbit outbox row.
|
||||
@Override
|
||||
public DeliveryResult deliver(Effect effect) throws Exception
|
||||
{
|
||||
EmailRequest request = decode(effect.payload());
|
||||
EmailEffects.Result result = EmailEffects.send(request, effect.idempotencyKey());
|
||||
if(result.outcome() == EmailEffectOutcome.REFUSED)
|
||||
{
|
||||
return (DeliveryResult.failed("Email refused"));
|
||||
}
|
||||
return (DeliveryResult.sent(result.message().id().toString()));
|
||||
}
|
||||
|
||||
// Existence reconciles handoff independently of mailbox delivery.
|
||||
@Override
|
||||
public ReconcileResult reconcile(Effect effect) throws Exception
|
||||
{
|
||||
return (EmailEffects.findByIdempotencyKey(effect.idempotencyKey())
|
||||
.map(ref -> ReconcileResult.found(ref.id().toString()))
|
||||
.orElseGet(ReconcileResult::notFound));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Tests use H2, the existing GreenMail SMTP sink, fake SES and HTTP request seams;
|
||||
no vendor endpoints are called. The PostgreSQL migration is supplied for host
|
||||
deployment and is not executed by the offline Java test gate.
|
||||
+68
-18
@@ -191,25 +191,75 @@ It would require a Google Cloud project, OAuth consent, client credentials and
|
||||
refresh-token management. It adds no needed system-mail capability over the app
|
||||
password preset in this phase and is built only on request (DESIGN 3.6).
|
||||
|
||||
## Event webhooks: phase (c)
|
||||
## Event webhooks
|
||||
|
||||
Event ingestion is **not installed by phase (b)**. After phase (c) is deployed,
|
||||
configure provider event delivery at:
|
||||
Phase (c) installs authenticated provider receivers when the host includes Javalin.
|
||||
Use `https://<app>{base}/{kind}/{providerName}`, where `base` is
|
||||
`EmailQBitConfig.webhookBasePath` (default `/qqq/hooks/email`), `kind` is `ses`,
|
||||
`postmark`, `sendgrid` or `mailgun`, and `providerName` is the row's unique name.
|
||||
The row must be enabled and its kind must match. The configurable base must start
|
||||
with `/`, have no trailing slash, and stay outside `/qqq/v1`.
|
||||
|
||||
`https://<app>/qqq/hooks/email/{kind}/{providerName}`
|
||||
|
||||
Use the configured provider name and provider-kind route defined by phase (c).
|
||||
Do not expect delivery/bounce/complaint/open/click updates until the receiver
|
||||
exists. Tracking options can operate on the provider side now.
|
||||
|
||||
| Provider | Event setup to prepare |
|
||||
| Provider | Setup and events |
|
||||
|---|---|
|
||||
| Postmark | Server/stream event webhooks with Basic auth; `POSTMARK_WEBHOOK_PASSWORD` |
|
||||
| SendGrid | Signed Event Webhook; verification key in `SENDGRID_WEBHOOK_PUBLIC_KEY` |
|
||||
| Mailgun | Regional domain event webhooks; `MAILGUN_WEBHOOK_SIGNING_KEY`, separate from the sending key |
|
||||
| SES | Configuration-set events to SNS, HTTP subscription and `sns_topic_arn`; signature/topic verification arrives in phase (c) |
|
||||
| SMTP / Gmail SMTP | No native provider event webhook in this design |
|
||||
| SES | Send configuration-set Delivery, Bounce, Complaint, Reject and DeliveryDelay events to SNS; enable Open/Click only when desired. Set `sns_topic_arn` to that exact topic. Subscribe the HTTPS `/ses/{providerName}` URL with raw delivery **off**, so SNS sends its signed JSON envelope. No webhook secret env is used. Verified subscription confirmations are fetched automatically and retained as IGNORED inbox rows. |
|
||||
| Postmark | Configure server/message-stream webhooks for Delivery, Bounce, SpamComplaint and SubscriptionChange; optionally Open and Click. Use HTTP Basic username **`postmark`** and the password resolved from `webhook_secret_env_name` (normally `POSTMARK_WEBHOOK_PASSWORD`). Configure those credentials in Postmark's webhook settings/URL; keep the stored provider row limited to the environment name. |
|
||||
| SendGrid | Enable the Signed Event Webhook for delivered, bounce, dropped, deferred, spamreport and unsubscribe (group_unsubscribe is also accepted); optionally open/click. Store the base64 DER verification public key in `SENDGRID_WEBHOOK_PUBLIC_KEY`, referenced by `webhook_secret_env_name`. Preserve both Twilio signature/timestamp headers through the proxy. Batches are authenticated over timestamp plus the unchanged raw body. |
|
||||
| Mailgun | Configure regional domain webhooks for delivered, permanent/temporary failed, complained and unsubscribed; optionally opened/clicked. Set `webhook_secret_env_name=MAILGUN_WEBHOOK_SIGNING_KEY` to the account's HTTP webhook signing key, **not** its sending API key. Preserve the JSON `signature` object and `event-data`. |
|
||||
| SMTP / Gmail SMTP | No native event receiver. One-click unsubscribe remains available for opt-out mail. |
|
||||
|
||||
Webhook environment names are allowlisted now so rows can be prepared; outbound
|
||||
sends never resolve them. Verification, durable event processing and unsubscribe
|
||||
handling arrive in later phases.
|
||||
SNS accepts SignatureVersion 1 and 2, pins TopicArn, and fetches certificates only
|
||||
from HTTPS `sns.<region>.amazonaws.com` or `.amazonaws.com.cn` hosts with `.pem`
|
||||
paths. Certificates are cached only in memory; redirects are disabled. Confirmation
|
||||
URLs use the same HTTPS host restriction. Canonicalization follows
|
||||
[AWS's signature rules](https://docs.aws.amazon.com/sns/latest/dg/sns-verify-signature-of-message-verify-message-signature.html).
|
||||
SNS retries can be old, so freshness is enforced only for SendGrid and Mailgun:
|
||||
timestamps must be within five minutes in either direction. Keep host clocks synchronized.
|
||||
Postmark authenticates the request with Basic auth and supplies no signed timestamp.
|
||||
|
||||
Requests are capped at 1 MiB, including chunked requests. Responses are JSON:
|
||||
200 `received` or `duplicate`, 401 invalid verification/Mailgun nonce replay,
|
||||
404 missing/disabled/mismatched provider, 413 oversized input, and 503 unavailable
|
||||
verification configuration, SNS fetch or database. No request bodies, recipient
|
||||
addresses or credential values are logged. Raw payloads are stored in the DB inbox
|
||||
as sensitive data, readable with `email.view`.
|
||||
|
||||
The hidden `email_process_webhooks` schedule runs every 30 seconds. One bounded
|
||||
transaction applies each inbox row, including events, message state and suppressions.
|
||||
The common provider lock serializes suppression upserts across replicas/providers;
|
||||
no network I/O occurs under that lock. Failed attempts back off with `retryMinutes`
|
||||
and become FAILED after six attempts. Use the inbox's **Reprocess** Review/Result
|
||||
action (`email.manage`) after resolving the cause. Unmatched messages retry, including
|
||||
a webhook racing the transport's SENT commit. Unknown/disabled tracking events are
|
||||
IGNORED. Per-event recipient keys prevent overlapping SendGrid batches from duplicating
|
||||
history. Mailgun's `(provider_id, nonce)` uniqueness is distinct from event deduplication:
|
||||
a stored nonce is refused with 401 even if its event is already known.
|
||||
|
||||
Hard bounces and complaints upsert ALL suppression; complaints take precedence over
|
||||
hard-bounce reasons. Unsubscribe events suppress only MARKETING or NOTIFICATION for
|
||||
the matched message. Delivery advances SENT to DELIVERED; hard bounce and complaint
|
||||
advance delivery state without reversing a later complaint. Opens/clicks require the
|
||||
corresponding provider flag and are never retained for PHI-approved messages.
|
||||
Provider reason text is address-masked and limited to 500 characters; only clicks
|
||||
retain URLs. More than three complaints in 24 hours raises optional WARNING alert
|
||||
`email.complaint`, deduplicated per UTC day by core's alert sink.
|
||||
|
||||
## One-click unsubscribe
|
||||
|
||||
Set `publicBaseUrl` to the public HTTPS app origin and supply
|
||||
`QQQ_EMAIL_UNSUBSCRIBE_KEY` through the secret resolver. The configured webhook base
|
||||
also controls newly signed unsubscribe URLs. Keep the key and published path stable:
|
||||
tokens have no expiry, but cannot resolve after their message row is deleted.
|
||||
|
||||
GET `{base}/unsubscribe/{token}` displays a self-contained, escaped light/dark form
|
||||
and never changes state. POST `List-Unsubscribe=One-Click` returns an empty 200;
|
||||
the page's form POST returns confirmation HTML. Both atomically upsert category
|
||||
suppression and record an UNSUBSCRIBED event. Repeated POSTs do not duplicate events.
|
||||
Invalid tokens return a generic 404. TRANSACTIONAL and SECURITY tokens are neither
|
||||
issued nor accepted.
|
||||
|
||||
The existing v1 token binds the message, category and **complete To/Cc/Bcc envelope
|
||||
hash**, rather than one mailbox. Following that link opts out every mailbox in the
|
||||
bound envelope for that category; the page reveals none of those addresses. Send
|
||||
one recipient per opt-out message when individual preference control is needed.
|
||||
Preferences UI remains deferred until core #1162 ships in an RC.
|
||||
@@ -0,0 +1,11 @@
|
||||
-- Existing PostgreSQL installations: pause enqueue and dispatch before applying.
|
||||
-- SHA-256 is PostgreSQL's built-in bytea function; no extension is required.
|
||||
BEGIN;
|
||||
ALTER TABLE email_message ADD COLUMN rfc_message_id VARCHAR(500);
|
||||
UPDATE email_message
|
||||
SET rfc_message_id = '<' || left(encode(sha256(convert_to(
|
||||
coalesce(idempotency_key, 'qbit-email:' || id::text), 'UTF8')), 'hex'), 32)
|
||||
|| '@' || coalesce(lower(substring(from_address from
|
||||
'@([A-Za-z0-9]+(?:[.-][A-Za-z0-9]+)*)(?:>|$)')), 'qbit-email.local') || '>';
|
||||
ALTER TABLE email_message ADD CONSTRAINT email_message_rfc_message_id_unique UNIQUE (rfc_message_id);
|
||||
COMMIT;
|
||||
@@ -1,4 +1,4 @@
|
||||
-- Phase (a), parts 1 and 2. Apply before producing the qbit. Host security columns must be added by the host.
|
||||
-- Phases (a), (b) and (c). Apply before producing the qbit. Host security columns must be added by the host.
|
||||
|
||||
CREATE TABLE email_provider (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
@@ -38,6 +38,19 @@ CREATE TABLE email_provider (
|
||||
UNIQUE (default_slot)
|
||||
);
|
||||
|
||||
CREATE TABLE email_provider_sender_address (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
modify_date TIMESTAMP,
|
||||
provider_id INTEGER NOT NULL,
|
||||
address VARCHAR(500) NOT NULL,
|
||||
display_name VARCHAR(500),
|
||||
is_default BOOLEAN,
|
||||
enabled BOOLEAN,
|
||||
note TEXT,
|
||||
UNIQUE (provider_id, address)
|
||||
);
|
||||
|
||||
CREATE TABLE email_message (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
@@ -47,6 +60,7 @@ CREATE TABLE email_message (
|
||||
cc_addresses TEXT,
|
||||
bcc_addresses TEXT,
|
||||
from_address VARCHAR(500),
|
||||
from_address_id INTEGER,
|
||||
reply_to VARCHAR(500),
|
||||
subject TEXT,
|
||||
template_id INTEGER,
|
||||
@@ -61,6 +75,7 @@ CREATE TABLE email_message (
|
||||
phi_approved BOOLEAN,
|
||||
unsubscribe_url TEXT,
|
||||
idempotency_key VARCHAR(500),
|
||||
rfc_message_id VARCHAR(500),
|
||||
provider_id INTEGER,
|
||||
provider_message_id VARCHAR(500),
|
||||
attempt_count INTEGER,
|
||||
@@ -79,7 +94,8 @@ CREATE TABLE email_message (
|
||||
sender_process VARCHAR(500),
|
||||
suppressed_addresses TEXT,
|
||||
send_after TIMESTAMP,
|
||||
UNIQUE (idempotency_key)
|
||||
UNIQUE (idempotency_key),
|
||||
UNIQUE (rfc_message_id)
|
||||
);
|
||||
|
||||
CREATE TABLE email_message_attempt (
|
||||
@@ -184,3 +200,42 @@ CREATE TABLE email_suppression (
|
||||
CREATE INDEX email_message_due ON email_message (status, next_attempt_date);
|
||||
CREATE INDEX email_attempt_provider_start ON email_message_attempt (provider_id, start_date);
|
||||
CREATE INDEX email_attempt_provider_end ON email_message_attempt (provider_id, end_date, id);
|
||||
|
||||
CREATE TABLE email_webhook_inbox (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
modify_date TIMESTAMP,
|
||||
provider_id INTEGER NOT NULL,
|
||||
kind VARCHAR(500) NOT NULL,
|
||||
external_event_id VARCHAR(500) NOT NULL,
|
||||
nonce VARCHAR(500),
|
||||
status VARCHAR(500) NOT NULL,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_date TIMESTAMP,
|
||||
last_error VARCHAR(500),
|
||||
payload_json TEXT,
|
||||
payload_sha256 VARCHAR(500),
|
||||
received_date TIMESTAMP,
|
||||
processed_date TIMESTAMP,
|
||||
UNIQUE (provider_id, external_event_id),
|
||||
UNIQUE (provider_id, nonce)
|
||||
);
|
||||
|
||||
CREATE TABLE email_event (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
modify_date TIMESTAMP,
|
||||
message_id INTEGER NOT NULL,
|
||||
provider_id INTEGER,
|
||||
event_type VARCHAR(500) NOT NULL,
|
||||
bounce_type VARCHAR(500),
|
||||
recipient VARCHAR(500),
|
||||
occurred_date TIMESTAMP,
|
||||
detail VARCHAR(500),
|
||||
url TEXT,
|
||||
event_key VARCHAR(500) NOT NULL UNIQUE
|
||||
);
|
||||
CREATE INDEX email_inbox_due ON email_webhook_inbox (status, next_attempt_date);
|
||||
CREATE INDEX email_event_message ON email_event (message_id);
|
||||
CREATE INDEX email_event_type_date ON email_event (event_type, occurred_date);
|
||||
CREATE INDEX email_message_provider_id ON email_message (provider_id, provider_message_id);
|
||||
@@ -168,7 +168,7 @@
|
||||
<failsOnError>false</failsOnError>
|
||||
<failOnViolation>true</failOnViolation>
|
||||
<violationSeverity>warning</violationSeverity>
|
||||
<includeTestSourceDirectory>false</includeTestSourceDirectory>
|
||||
<includeTestSourceDirectory>true</includeTestSourceDirectory>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
|
||||
@@ -46,6 +46,7 @@ public class EmailQBitConfig implements QBitConfig
|
||||
private String backendName;
|
||||
private String schedulerName;
|
||||
private String publicBaseUrl;
|
||||
private String webhookBasePath = "/qqq/hooks/email";
|
||||
private String defaultProviderName;
|
||||
private String bridgeProviderName = "qbitEmail";
|
||||
private Function<String, String> secretResolver = System::getenv;
|
||||
@@ -89,6 +90,11 @@ public class EmailQBitConfig implements QBitConfig
|
||||
@Override
|
||||
public void validate(QInstance instance, List<String> errors)
|
||||
{
|
||||
if(webhookBasePath == null || !webhookBasePath.matches("/(?:[A-Za-z0-9_-]+/)*[A-Za-z0-9_-]+")
|
||||
|| webhookBasePath.equals("/qqq/v1") || webhookBasePath.startsWith("/qqq/v1/"))
|
||||
{
|
||||
errors.add("webhookBasePath must be an absolute path outside /qqq/v1, without a trailing slash");
|
||||
}
|
||||
if(backendName == null || backendName.isBlank())
|
||||
{
|
||||
errors.add("backendName is required");
|
||||
@@ -201,6 +207,23 @@ public class EmailQBitConfig implements QBitConfig
|
||||
return (publicBaseUrl);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Public provider and unsubscribe route base.
|
||||
*******************************************************************************/
|
||||
public String getWebhookBasePath()
|
||||
{
|
||||
return (webhookBasePath);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the route base and queue-time unsubscribe URLs together.
|
||||
*******************************************************************************/
|
||||
public EmailQBitConfig withWebhookBasePath(String webhookBasePath)
|
||||
{
|
||||
this.webhookBasePath = webhookBasePath;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for publicBaseUrl.
|
||||
|
||||
@@ -143,6 +143,7 @@ public class EmailQBitProducer implements QBitMetaDataProducer<EmailQBitConfig>
|
||||
for(QTableMetaData table : EmailTables.tables(qBitConfig))
|
||||
{
|
||||
Class<?> customizer = EmailTables.PROVIDER.equals(table.getName()) ? EmailTableCustomizers.ProviderCustomizer.class
|
||||
: EmailTables.SENDER.equals(table.getName()) ? EmailTableCustomizers.SenderAddressCustomizer.class
|
||||
: EmailTables.SUPPRESSION.equals(table.getName()) ? EmailTableCustomizers.SuppressionCustomizer.class
|
||||
: List.of(EmailTables.TEMPLATE, EmailTables.VERSION, EmailTables.BRAND).contains(table.getName()) ? com.kingsrook.qbits.email.template.EmailTemplateCustomizers.class : null;
|
||||
if(customizer != null)
|
||||
@@ -195,6 +196,24 @@ public class EmailQBitProducer implements QBitMetaDataProducer<EmailQBitConfig>
|
||||
output.add(EmailApp.app(qBitConfig, tables));
|
||||
output.add(new EmailOutboxMessagingProviderMetaData().withName(qBitConfig.getBridgeProviderName()));
|
||||
output.addSelfToInstance(qInstance);
|
||||
registerWebhooks(qInstance);
|
||||
return (new MetaDataProducerMultiOutput());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Keep optional provided middleware linkage outside non-web applications.
|
||||
*******************************************************************************/
|
||||
private static void registerWebhooks(QInstance instance) throws QException
|
||||
{
|
||||
try
|
||||
{
|
||||
Class.forName("io.javalin.Javalin");
|
||||
Class.forName("com.kingsrook.qqq.middleware.javalin.QJavalinMetaData");
|
||||
}
|
||||
catch(ClassNotFoundException e)
|
||||
{
|
||||
return;
|
||||
}
|
||||
com.kingsrook.qbits.email.webhook.EmailWebhookRouteProvider.register(instance);
|
||||
}
|
||||
}
|
||||
@@ -68,8 +68,21 @@ public final class EmailHome
|
||||
String[] fields = {"sentDate", "deliveredDate", "deadLetterDate", "bouncedDate", "complainedDate"};
|
||||
for(int i = 0; i < fields.length; i++)
|
||||
{
|
||||
totals.add(count(EmailTables.MESSAGE, since(fields[i], now.minus(i < 3 ? 1 : 7, ChronoUnit.DAYS))
|
||||
.withCriteria(fields[i], QCriteriaOperator.LESS_THAN_OR_EQUALS, now)));
|
||||
String eventType = switch(i)
|
||||
{
|
||||
case 1 -> "DELIVERED";
|
||||
case 3 -> "BOUNCED";
|
||||
case 4 -> "COMPLAINED";
|
||||
default -> null;
|
||||
};
|
||||
String field = eventType == null ? fields[i] : "occurredDate";
|
||||
QQueryFilter filter = since(field, now.minus(i < 3 ? 1 : 7, ChronoUnit.DAYS))
|
||||
.withCriteria(field, QCriteriaOperator.LESS_THAN_OR_EQUALS, now);
|
||||
if(eventType != null)
|
||||
{
|
||||
filter.withCriteria("eventType", QCriteriaOperator.EQUALS, eventType);
|
||||
}
|
||||
totals.add(count(eventType == null ? EmailTables.MESSAGE : EmailTables.EVENT, filter));
|
||||
}
|
||||
totals.add(count(EmailTables.SUPPRESSION, new QQueryFilter()));
|
||||
List<QRecord> messages = query(EmailTables.MESSAGE, new QQueryFilter().withOrderBy(new QFilterOrderBy("createDate", false))
|
||||
|
||||
@@ -35,6 +35,7 @@ import com.kingsrook.qqq.backend.core.alerts.QAlerts;
|
||||
*******************************************************************************/
|
||||
public final class EmailAlerts
|
||||
{
|
||||
public static final String COMPLAINT = "email.complaint";
|
||||
public static final String UNHEALTHY = "email.provider.unhealthy";
|
||||
public static final String DEAD_LETTER = "email.message.dead-letter";
|
||||
|
||||
@@ -50,7 +51,9 @@ public final class EmailAlerts
|
||||
*******************************************************************************/
|
||||
public static List<QAlertTypeMetaData> types()
|
||||
{
|
||||
return (List.of(new QAlertTypeMetaData().withName(UNHEALTHY).withLabel("Email provider unhealthy").withDefaultSeverity(AlertSeverity.ERROR)
|
||||
return (List.of(new QAlertTypeMetaData().withName(COMPLAINT).withLabel("Email complaints").withDefaultSeverity(AlertSeverity.WARNING)
|
||||
.withSubjectTableName(EmailTables.EVENT).withVisibilityPermission("email.view").withSourceName("email"),
|
||||
new QAlertTypeMetaData().withName(UNHEALTHY).withLabel("Email provider unhealthy").withDefaultSeverity(AlertSeverity.ERROR)
|
||||
.withSubjectTableName(EmailTables.PROVIDER).withVisibilityPermission("email.admin").withSourceName("email"),
|
||||
new QAlertTypeMetaData().withName(DEAD_LETTER).withLabel("Email dead letters").withDefaultSeverity(AlertSeverity.WARNING)
|
||||
.withSubjectTableName(EmailTables.MESSAGE).withVisibilityPermission("email.view").withSourceName("email")));
|
||||
@@ -62,7 +65,11 @@ public final class EmailAlerts
|
||||
public static void raise(String type, Integer providerId)
|
||||
{
|
||||
String key = "provider:" + providerId + (DEAD_LETTER.equals(type) ? ":" + EmailRuntime.clock().instant().getEpochSecond() / 3600 : "");
|
||||
QAlerts.raise(new AlertSignal(type).withDedupKey(key).withTitle(UNHEALTHY.equals(type) ? "Email provider unhealthy" : "Email message dead-lettered")
|
||||
if(COMPLAINT.equals(type))
|
||||
{
|
||||
key = "complaints:" + EmailRuntime.clock().instant().getEpochSecond() / 86400;
|
||||
}
|
||||
QAlerts.raise(new AlertSignal(type).withDedupKey(key).withTitle(COMPLAINT.equals(type) ? "More than three email complaints in 24 hours" : UNHEALTHY.equals(type) ? "Email provider unhealthy" : "Email message dead-lettered")
|
||||
.withSummary("Review the Email tables for status and retry actions."));
|
||||
}
|
||||
|
||||
|
||||
@@ -451,7 +451,7 @@ public class EmailDispatcher
|
||||
*******************************************************************************/
|
||||
private static EmailEnvelope envelope(QRecord message)
|
||||
{
|
||||
return (new EmailEnvelope().withMessageId(message.getValueString("id")).withFrom(message.getValueString("fromAddress"))
|
||||
return (new EmailEnvelope().withMessageId(message.getValueString("id")).withRfcMessageId(message.getValueString("rfcMessageId")).withFrom(message.getValueString("fromAddress"))
|
||||
.withTo(split(message.getValueString("toAddresses"))).withCc(split(message.getValueString("ccAddresses"))).withBcc(split(message.getValueString("bccAddresses")))
|
||||
.withReplyTo(message.getValueString("replyTo")).withSubject(content(message, "subject"))
|
||||
.withHtmlBody(content(message, "htmlBody"))
|
||||
|
||||
@@ -312,4 +312,88 @@ public final class EmailEnums
|
||||
return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' '));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** InboxStatus values.
|
||||
*******************************************************************************/
|
||||
public enum InboxStatus implements PossibleValueEnum<String>
|
||||
{
|
||||
PENDING, PROCESSED, FAILED, IGNORED;
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable stored identifier.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getPossibleValueId()
|
||||
{
|
||||
return (name());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Display label.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getPossibleValueLabel()
|
||||
{
|
||||
return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' '));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** EventType values.
|
||||
*******************************************************************************/
|
||||
public enum EventType implements PossibleValueEnum<String>
|
||||
{
|
||||
DELIVERED, DEFERRED, BOUNCED, COMPLAINED, OPENED, CLICKED, UNSUBSCRIBED, REJECTED;
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable stored identifier.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getPossibleValueId()
|
||||
{
|
||||
return (name());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Display label.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getPossibleValueLabel()
|
||||
{
|
||||
return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' '));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** BounceType values.
|
||||
*******************************************************************************/
|
||||
public enum BounceType implements PossibleValueEnum<String>
|
||||
{
|
||||
HARD, SOFT, UNDETERMINED;
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable stored identifier.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getPossibleValueId()
|
||||
{
|
||||
return (name());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Display label.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String getPossibleValueLabel()
|
||||
{
|
||||
return (name().charAt(0) + name().substring(1).toLowerCase(java.util.Locale.ROOT).replace('_', ' '));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -21,23 +21,32 @@
|
||||
package com.kingsrook.qbits.email.metadata;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Objects;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.email.EmailQBitConfig;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissionChecker;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissions;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qbits.email.transport.EmailProviderSettings;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
|
||||
import jakarta.mail.internet.InternetAddress;
|
||||
@@ -197,6 +206,13 @@ public final class EmailTableCustomizers
|
||||
record.setValue(field, null);
|
||||
}
|
||||
}
|
||||
String sendingDomain = merged.getValueString("sendingDomain");
|
||||
if(sendingDomain != null && !sendingDomain.isBlank() && merged.getValue("id") != null
|
||||
&& SystemWrites.query(EmailTables.SENDER, EmailStore.equal("providerId", merged.getValueInteger("id")), null).stream()
|
||||
.anyMatch(s -> !sendingDomain.equalsIgnoreCase(s.getValueString("address").substring(s.getValueString("address").lastIndexOf('@') + 1))))
|
||||
{
|
||||
record.addError(new BadInputStatusMessage("Sender addresses must match the sending domain"));
|
||||
}
|
||||
resulting.removeIf(r -> Objects.equals(r.getValue("id"), record.getValue("id")));
|
||||
resulting.add(merged);
|
||||
record.setValue("defaultSlot", Boolean.TRUE.equals(merged.getValueBoolean("enabled")) && Boolean.TRUE.equals(merged.getValueBoolean("isDefault")) ? 1 : null);
|
||||
@@ -230,6 +246,13 @@ public final class EmailTableCustomizers
|
||||
{
|
||||
records.forEach(r -> r.addError(new BadInputStatusMessage("Choose another enabled default before deleting this provider")));
|
||||
}
|
||||
for(QRecord row : records)
|
||||
{
|
||||
if(!preview && row.getErrors().isEmpty())
|
||||
{
|
||||
SystemWrites.delete(EmailTables.SENDER, EmailStore.equal("providerId", row.getValueInteger("id")), input.getTransaction());
|
||||
}
|
||||
}
|
||||
return (records);
|
||||
}
|
||||
|
||||
@@ -272,6 +295,106 @@ public final class EmailTableCustomizers
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sender addresses: canonical lowercase address, the provider's sending domain,
|
||||
** unique per provider and at most one enabled default per provider.
|
||||
*******************************************************************************/
|
||||
public static class SenderAddressCustomizer implements TableCustomizerInterface
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Validate each row against its provider and the resulting set of rows.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords) throws QException
|
||||
{
|
||||
if(SystemWrites.active())
|
||||
{
|
||||
return (records);
|
||||
}
|
||||
QBackendTransaction transaction = input instanceof InsertInput insert ? insert.getTransaction() : input instanceof UpdateInput update ? update.getTransaction() : null;
|
||||
List<QRecord> resulting = new ArrayList<>();
|
||||
Set<Serializable> providerIds = new LinkedHashSet<>();
|
||||
for(QRecord record : records)
|
||||
{
|
||||
QRecord merged = effective(record, oldRecords);
|
||||
if(record.getValueString("address") != null)
|
||||
{
|
||||
try
|
||||
{
|
||||
String text = record.getValueString("address");
|
||||
if(text.contains("\r") || text.contains("\n"))
|
||||
{
|
||||
throw (new IllegalArgumentException("Sender address must not contain line breaks"));
|
||||
}
|
||||
InternetAddress address = new InternetAddress(text, true);
|
||||
address.validate();
|
||||
record.setValue("address", address.getAddress().toLowerCase(Locale.ROOT));
|
||||
merged.setValue("address", record.getValue("address"));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
record.addError(new BadInputStatusMessage("A valid sender address is required"));
|
||||
}
|
||||
}
|
||||
String name = merged.getValueString("displayName");
|
||||
if(name != null && (name.contains("\r") || name.contains("\n")))
|
||||
{
|
||||
record.addError(new BadInputStatusMessage("Display name must not contain line breaks"));
|
||||
}
|
||||
QRecord provider = merged.getValue("providerId") == null ? null : EmailStore.get(EmailTables.PROVIDER, merged.getValueInteger("providerId"), transaction);
|
||||
if(provider == null)
|
||||
{
|
||||
record.addError(new BadInputStatusMessage("An existing provider is required"));
|
||||
}
|
||||
else if(merged.getValueString("address") != null && record.getErrors().isEmpty())
|
||||
{
|
||||
String domain = provider.getValueString("sendingDomain");
|
||||
String address = merged.getValueString("address");
|
||||
if(domain != null && !domain.isBlank() && !domain.equalsIgnoreCase(address.substring(address.lastIndexOf('@') + 1)))
|
||||
{
|
||||
record.addError(new BadInputStatusMessage("Sender address domain must match the provider sending domain"));
|
||||
}
|
||||
}
|
||||
if(Boolean.TRUE.equals(merged.getValueBoolean("isDefault")) && !Boolean.TRUE.equals(merged.getValueBoolean("enabled")))
|
||||
{
|
||||
record.addError(new BadInputStatusMessage("A default sender address must be enabled"));
|
||||
}
|
||||
if(merged.getValue("providerId") != null)
|
||||
{
|
||||
providerIds.add(merged.getValue("providerId"));
|
||||
}
|
||||
resulting.add(merged);
|
||||
}
|
||||
if(!providerIds.isEmpty())
|
||||
{
|
||||
List<QRecord> existing = SystemWrites.query(EmailTables.SENDER, new QQueryFilter(new QFilterCriteria("providerId", QCriteriaOperator.IN, new ArrayList<>(providerIds))), transaction);
|
||||
existing.removeIf(r -> records.stream().anyMatch(n -> Objects.equals(n.getValue("id"), r.getValue("id"))));
|
||||
for(int i = 0; i < records.size(); i++)
|
||||
{
|
||||
QRecord merged = resulting.get(i);
|
||||
List<QRecord> peers = new ArrayList<>(existing);
|
||||
for(int j = 0; j < resulting.size(); j++)
|
||||
{
|
||||
if(j != i)
|
||||
{
|
||||
peers.add(resulting.get(j));
|
||||
}
|
||||
}
|
||||
peers.removeIf(r -> !Objects.equals(r.getValue("providerId"), merged.getValue("providerId")));
|
||||
if(peers.stream().anyMatch(r -> Objects.equals(r.getValueString("address"), merged.getValueString("address"))))
|
||||
{
|
||||
records.get(i).addError(new BadInputStatusMessage("That sender address already exists for this provider"));
|
||||
}
|
||||
if(Boolean.TRUE.equals(merged.getValueBoolean("isDefault")) && peers.stream().anyMatch(r -> Boolean.TRUE.equals(r.getValueBoolean("isDefault"))))
|
||||
{
|
||||
records.get(i).addError(new BadInputStatusMessage("Only one sender address per provider can be the default"));
|
||||
}
|
||||
}
|
||||
}
|
||||
return (records);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Canonical addresses and administrator protection for complaint removal.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -26,6 +26,7 @@ import java.util.List;
|
||||
import com.kingsrook.qbits.email.EmailQBitConfig;
|
||||
import com.kingsrook.qbits.email.outbox.EmailCategory;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissionChecker;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
|
||||
import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
@@ -53,15 +54,18 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
|
||||
public final class EmailTables
|
||||
{
|
||||
public static final String PROVIDER = "email_provider";
|
||||
public static final String SENDER = "email_provider_sender_address";
|
||||
public static final String MESSAGE = "email_message";
|
||||
public static final String ATTEMPT = "email_message_attempt";
|
||||
public static final String TEMPLATE = "email_template";
|
||||
public static final String VERSION = "email_template_version";
|
||||
public static final String BRAND = "email_brand";
|
||||
public static final String SUPPRESSION = "email_suppression";
|
||||
public static final List<String> ALL = List.of(PROVIDER, MESSAGE, ATTEMPT, TEMPLATE, VERSION, BRAND, SUPPRESSION);
|
||||
public static final String INBOX = "email_webhook_inbox";
|
||||
public static final String EVENT = "email_event";
|
||||
public static final List<String> ALL = List.of(PROVIDER, SENDER, MESSAGE, ATTEMPT, TEMPLATE, VERSION, BRAND, SUPPRESSION, INBOX, EVENT);
|
||||
public static final List<String> MAIN = List.of(MESSAGE, TEMPLATE, BRAND, SUPPRESSION, PROVIDER);
|
||||
public static final List<String> SYSTEM_WRITTEN = List.of(MESSAGE, ATTEMPT);
|
||||
public static final List<String> SYSTEM_WRITTEN = List.of(MESSAGE, ATTEMPT, INBOX, EVENT);
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
@@ -95,12 +99,16 @@ public final class EmailTables
|
||||
result.add(QPossibleValueSource.newForEnum("email_suppression_scope", EmailEnums.SuppressionScope.values()));
|
||||
result.add(QPossibleValueSource.newForEnum("email_suppression_reason", EmailEnums.SuppressionReason.values()));
|
||||
result.add(QPossibleValueSource.newForEnum("email_category", EmailCategory.values()));
|
||||
result.add(QPossibleValueSource.newForEnum("email_inbox_status", EmailEnums.InboxStatus.values()));
|
||||
result.add(QPossibleValueSource.newForEnum("email_event_type", EmailEnums.EventType.values()));
|
||||
result.add(QPossibleValueSource.newForEnum("email_bounce_type", EmailEnums.BounceType.values()));
|
||||
ALL.forEach(name -> result.add(QPossibleValueSource.newForTable(name)));
|
||||
for(QPossibleValueSource source : result)
|
||||
{
|
||||
if(List.of(PROVIDER, MESSAGE, VERSION).contains(source.getName()))
|
||||
if(List.of(PROVIDER, SENDER, MESSAGE, VERSION).contains(source.getName()))
|
||||
{
|
||||
source.withSearchFields(PROVIDER.equals(source.getName()) ? List.of("label", "name") : MESSAGE.equals(source.getName()) ? List.of("subject") : List.of("versionNumber", "locale"))
|
||||
source.withSearchFields(PROVIDER.equals(source.getName()) ? List.of("label", "name") : SENDER.equals(source.getName()) ? List.of("address", "displayName")
|
||||
: MESSAGE.equals(source.getName()) ? List.of("subject") : List.of("versionNumber", "locale"))
|
||||
.withOrderByField("id");
|
||||
}
|
||||
}
|
||||
@@ -108,18 +116,21 @@ public final class EmailTables
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The seven phase (a) tables.
|
||||
** Outbound delivery and tracking tables.
|
||||
*******************************************************************************/
|
||||
public static List<QTableMetaData> tables(EmailQBitConfig config)
|
||||
{
|
||||
List<QTableMetaData> tables = new ArrayList<>();
|
||||
tables.add(provider(config));
|
||||
tables.add(sender(config));
|
||||
tables.add(message(config));
|
||||
tables.add(messageattempt(config));
|
||||
tables.add(template(config));
|
||||
tables.add(templateversion(config));
|
||||
tables.add(brand(config));
|
||||
tables.add(suppression(config));
|
||||
tables.add(inbox(config));
|
||||
tables.add(event(config));
|
||||
for(QTableMetaData table : tables)
|
||||
{
|
||||
for(QFieldMetaData field : config.getSecurityFields())
|
||||
@@ -203,7 +214,31 @@ public final class EmailTables
|
||||
table.withSection(SectionFactory.defaultT1("id", "name", "label", "kind", "enabled", "isDefault"));
|
||||
table.withSection(SectionFactory.defaultT2("fromAddress", "fromName", "replyTo", "sendingDomain", "host", "port", "security", "region", "usernameEnvName", "secretEnvName", "webhookSecretEnvName", "snsTopicArn", "configurationSet", "messageStream", "maxPerMinute", "maxPerDay", "trackOpens", "trackClicks", "phiAllowed", "healthStatus", "lastSuccessDate", "lastFailureDate", "lastFailureMessage"));
|
||||
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
|
||||
table.withSection(SectionFactory.customT2("senders", new QIcon("list")).withLabel("Sender Addresses").withWidgetName(SENDER + "_children"));
|
||||
table.withSection(SectionFactory.customT2("children", new QIcon("list")).withLabel("Messages").withWidgetName(PROVIDER + "_children"));
|
||||
table.withSection(SectionFactory.customT2("inbox", new QIcon("list")).withLabel("Webhook Inbox").withWidgetName(INBOX + "_children"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Metadata for email_provider_sender_address. The provider list in the
|
||||
** nav is the only entry point; rows are managed as a child list on the provider.
|
||||
*******************************************************************************/
|
||||
private static QTableMetaData sender(EmailQBitConfig config)
|
||||
{
|
||||
QTableMetaData table = base(config, SENDER, "Provider Sender Address")
|
||||
.withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER).withIsRequired(true))
|
||||
.withField(field("address", QFieldType.STRING).withIsRequired(true).withMaxLength(500))
|
||||
.withField(field("displayName", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("isDefault", QFieldType.BOOLEAN).withDefaultValue(false))
|
||||
.withField(field("enabled", QFieldType.BOOLEAN).withDefaultValue(true))
|
||||
.withField(field("note", QFieldType.TEXT));
|
||||
table.withRecordLabelFormat(null).withRecordLabelFields(List.of());
|
||||
table.withUniqueKey(new UniqueKey("providerId", "address"));
|
||||
table.withSection(SectionFactory.defaultT1("id", "providerId", "address", "displayName", "isDefault", "enabled"));
|
||||
table.withSection(SectionFactory.defaultT2("note"));
|
||||
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
@@ -219,10 +254,11 @@ public final class EmailTables
|
||||
.withField(field("ccAddresses", QFieldType.TEXT).withIsSensitive(true))
|
||||
.withField(field("bccAddresses", QFieldType.TEXT).withIsSensitive(true))
|
||||
.withField(field("fromAddress", QFieldType.STRING).withIsSensitive(true).withMaxLength(500))
|
||||
.withField(field("fromAddressId", QFieldType.INTEGER).withPossibleValueSourceName(SENDER).withPossibleValueSourceFilter(EmailStore.equal("providerId", "${input.providerId}")))
|
||||
.withField(field("replyTo", QFieldType.STRING).withIsSensitive(true).withMaxLength(500))
|
||||
.withField(field("subject", QFieldType.TEXT).withIsSensitive(true))
|
||||
.withField(field("templateId", QFieldType.INTEGER).withPossibleValueSourceName(TEMPLATE))
|
||||
.withField(field("templateVersionId", QFieldType.INTEGER).withPossibleValueSourceName(VERSION).withPossibleValueSourceFilter(com.kingsrook.qbits.email.support.EmailStore.equal("templateId", "${input.templateId}")))
|
||||
.withField(field("templateVersionId", QFieldType.INTEGER).withPossibleValueSourceName(VERSION).withPossibleValueSourceFilter(EmailStore.equal("templateId", "${input.templateId}")))
|
||||
.withField(field("brandId", QFieldType.INTEGER).withPossibleValueSourceName(BRAND))
|
||||
.withField(field("category", QFieldType.STRING).withPossibleValueSourceName("email_category").withMaxLength(500))
|
||||
.withField(field("locale", QFieldType.STRING).withDefaultValue("en").withMaxLength(500))
|
||||
@@ -233,6 +269,7 @@ public final class EmailTables
|
||||
.withField(field("unsubscribeUrl", QFieldType.TEXT))
|
||||
.withField(field("bodySha256", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("idempotencyKey", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("rfcMessageId", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER))
|
||||
.withField(field("providerMessageId", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("attemptCount", QFieldType.INTEGER).withDefaultValue(0))
|
||||
@@ -255,10 +292,12 @@ public final class EmailTables
|
||||
table.withRecordLabelFormat(null).withRecordLabelFields(List.of());
|
||||
table.getField("subject").setIsHeavy(false);
|
||||
table.withUniqueKey(new UniqueKey("idempotencyKey"));
|
||||
table.withUniqueKey(new UniqueKey("rfcMessageId"));
|
||||
table.withSection(SectionFactory.defaultT1("id", "status", "toAddresses", "ccAddresses", "bccAddresses", "fromAddress"));
|
||||
table.withSection(SectionFactory.defaultT2("replyTo", "subject", "templateId", "templateVersionId", "brandId", "category", "locale", "variablesSnapshot", "htmlBody", "textBody", "phiApproved", "unsubscribeUrl", "bodySha256", "idempotencyKey", "providerId", "providerMessageId", "attemptCount", "nextAttemptDate", "leaseExpiresDate", "lastError", "finishedDate", "sentDate", "deliveredDate", "bouncedDate", "complainedDate", "deadLetterDate", "linkedTableName", "linkedRecordId", "senderQbit", "senderProcess", "suppressedAddresses", "sendAfter"));
|
||||
table.withSection(SectionFactory.defaultT2("fromAddressId", "replyTo", "subject", "templateId", "templateVersionId", "brandId", "category", "locale", "variablesSnapshot", "htmlBody", "textBody", "phiApproved", "unsubscribeUrl", "bodySha256", "idempotencyKey", "rfcMessageId", "providerId", "providerMessageId", "attemptCount", "nextAttemptDate", "leaseExpiresDate", "lastError", "finishedDate", "sentDate", "deliveredDate", "bouncedDate", "complainedDate", "deadLetterDate", "linkedTableName", "linkedRecordId", "senderQbit", "senderProcess", "suppressedAddresses", "sendAfter"));
|
||||
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
|
||||
table.withSection(SectionFactory.customT2("children", new QIcon("list")).withLabel("Attempts").withWidgetName(MESSAGE + "_children"));
|
||||
table.withSection(SectionFactory.customT2("events", new QIcon("list")).withLabel("Events").withWidgetName(EVENT + "_children"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
@@ -269,7 +308,7 @@ public final class EmailTables
|
||||
private static QTableMetaData messageattempt(EmailQBitConfig config)
|
||||
{
|
||||
QTableMetaData table = base(config, ATTEMPT, "Message Attempt")
|
||||
.withField(field("messageId", QFieldType.INTEGER).withPossibleValueSourceName(MESSAGE).withIsRequired(true))
|
||||
.withField(field("messageId", QFieldType.INTEGER).withPossibleValueSourceName(MESSAGE).withIsRequired(true).withPossibleValueSourceFilter(EmailStore.equal("providerId", "${input.providerId}")))
|
||||
.withField(field("attemptNumber", QFieldType.INTEGER).withIsRequired(true))
|
||||
.withField(field("outcome", QFieldType.STRING).withPossibleValueSourceName("email_attempt_outcome").withMaxLength(500))
|
||||
.withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER))
|
||||
@@ -298,8 +337,8 @@ public final class EmailTables
|
||||
.withField(field("category", QFieldType.STRING).withPossibleValueSourceName("email_category").withMaxLength(500))
|
||||
.withField(field("kind", QFieldType.STRING).withPossibleValueSourceName("email_template_kind").withIsRequired(true).withMaxLength(500))
|
||||
.withField(field("brandId", QFieldType.INTEGER).withPossibleValueSourceName(BRAND))
|
||||
.withField(field("activeVersionId", QFieldType.INTEGER).withPossibleValueSourceName(VERSION).withIsEditable(false).withPossibleValueSourceFilter(com.kingsrook.qbits.email.support.EmailStore.equal("templateId", "${input.id}")))
|
||||
.withField(field("approvedVersionId", QFieldType.INTEGER).withIsEditable(false).withPossibleValueSourceName(VERSION).withPossibleValueSourceFilter(com.kingsrook.qbits.email.support.EmailStore.equal("templateId", "${input.id}")))
|
||||
.withField(field("activeVersionId", QFieldType.INTEGER).withPossibleValueSourceName(VERSION).withIsEditable(false).withPossibleValueSourceFilter(EmailStore.equal("templateId", "${input.id}")))
|
||||
.withField(field("approvedVersionId", QFieldType.INTEGER).withIsEditable(false).withPossibleValueSourceName(VERSION).withPossibleValueSourceFilter(EmailStore.equal("templateId", "${input.id}")))
|
||||
.withField(field("versionLock", QFieldType.INTEGER).withIsHidden(true).withIsEditable(false))
|
||||
.withField(field("phiApproved", QFieldType.BOOLEAN).withDefaultValue(false).withIsEditable(false))
|
||||
.withField(field("phiApprovedBy", QFieldType.STRING).withMaxLength(500).withIsEditable(false))
|
||||
@@ -390,7 +429,7 @@ public final class EmailTables
|
||||
.withField(field("scope", QFieldType.STRING).withPossibleValueSourceName("email_suppression_scope").withIsRequired(true).withMaxLength(500))
|
||||
.withField(field("reason", QFieldType.STRING).withPossibleValueSourceName("email_suppression_reason").withIsRequired(true).withMaxLength(500))
|
||||
.withField(field("sourceMessageId", QFieldType.INTEGER))
|
||||
.withField(field("sourceEventId", QFieldType.INTEGER))
|
||||
.withField(field("sourceEventId", QFieldType.INTEGER).withPossibleValueSourceName(EVENT).withPossibleValueSourceFilter(EmailStore.equal("messageId", "${input.sourceMessageId}")))
|
||||
.withField(field("note", QFieldType.TEXT))
|
||||
.withField(field("createdDate", QFieldType.DATE_TIME))
|
||||
.withField(field("expiresDate", QFieldType.DATE_TIME));
|
||||
@@ -401,12 +440,59 @@ public final class EmailTables
|
||||
return (table);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** System-written webhook inbox records.
|
||||
*******************************************************************************/
|
||||
private static QTableMetaData inbox(EmailQBitConfig config)
|
||||
{
|
||||
QTableMetaData table = base(config, INBOX, "Webhook Inbox")
|
||||
.withField(field("providerId", QFieldType.INTEGER).withIsRequired(true).withPossibleValueSourceName(PROVIDER))
|
||||
.withField(field("kind", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("externalEventId", QFieldType.STRING).withMaxLength(500).withIsRequired(true))
|
||||
.withField(field("nonce", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("status", QFieldType.STRING).withMaxLength(500).withIsRequired(true).withPossibleValueSourceName("email_inbox_status"))
|
||||
.withField(field("attempts", QFieldType.INTEGER))
|
||||
.withField(field("nextAttemptDate", QFieldType.DATE_TIME))
|
||||
.withField(field("lastError", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("payloadJson", QFieldType.TEXT).withIsSensitive(true))
|
||||
.withField(field("payloadSha256", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("receivedDate", QFieldType.DATE_TIME))
|
||||
.withField(field("processedDate", QFieldType.DATE_TIME));
|
||||
table.withUniqueKey(new UniqueKey("providerId", "externalEventId"));
|
||||
table.withUniqueKey(new UniqueKey("providerId", "nonce"));
|
||||
table.withSection(SectionFactory.defaultT1("id", "providerId", "kind", "externalEventId", "nonce", "status", "attempts", "nextAttemptDate", "lastError", "payloadJson", "payloadSha256", "receivedDate", "processedDate"));
|
||||
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** System-written event records.
|
||||
*******************************************************************************/
|
||||
private static QTableMetaData event(EmailQBitConfig config)
|
||||
{
|
||||
QTableMetaData table = base(config, EVENT, "Event")
|
||||
.withField(field("messageId", QFieldType.INTEGER).withIsRequired(true).withPossibleValueSourceName(MESSAGE).withPossibleValueSourceFilter(EmailStore.equal("providerId", "${input.providerId}")))
|
||||
.withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER))
|
||||
.withField(field("eventType", QFieldType.STRING).withMaxLength(500).withIsRequired(true).withPossibleValueSourceName("email_event_type"))
|
||||
.withField(field("bounceType", QFieldType.STRING).withMaxLength(500).withPossibleValueSourceName("email_bounce_type"))
|
||||
.withField(field("recipient", QFieldType.STRING).withMaxLength(500).withIsSensitive(true))
|
||||
.withField(field("occurredDate", QFieldType.DATE_TIME))
|
||||
.withField(field("detail", QFieldType.STRING).withMaxLength(500))
|
||||
.withField(field("url", QFieldType.TEXT).withIsSensitive(true))
|
||||
.withField(field("eventKey", QFieldType.STRING).withMaxLength(500).withIsRequired(true).withIsHidden(true));
|
||||
table.withUniqueKey(new UniqueKey("eventKey"));
|
||||
table.withSection(SectionFactory.defaultT1("id", "messageId", "providerId", "eventType", "bounceType", "recipient", "occurredDate", "detail", "url"));
|
||||
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
|
||||
return (table);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Relations used by child lists.
|
||||
*******************************************************************************/
|
||||
public static List<QJoinMetaData> joins()
|
||||
{
|
||||
return (List.of(join(PROVIDER, MESSAGE, "providerId"), join(MESSAGE, ATTEMPT, "messageId"), join(TEMPLATE, VERSION, "templateId")));
|
||||
return (List.of(join(PROVIDER, MESSAGE, "providerId"), join(MESSAGE, ATTEMPT, "messageId"), join(TEMPLATE, VERSION, "templateId"), join(PROVIDER, INBOX, "providerId"), join(PROVIDER, SENDER, "providerId"), join(MESSAGE, EVENT, "messageId")));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
@@ -426,8 +512,8 @@ public final class EmailTables
|
||||
List<QWidgetMetaData> widgets = new ArrayList<>();
|
||||
for(QJoinMetaData join : joins())
|
||||
{
|
||||
widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(join).withName(join.getLeftTable() + "_children")
|
||||
.withLabel(join.getRightTable().equals(ATTEMPT) ? "Attempts" : join.getRightTable().equals(VERSION) ? "Versions" : "Messages")
|
||||
widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(join).withName(List.of(INBOX, EVENT, SENDER).contains(join.getRightTable()) ? join.getRightTable() + "_children" : join.getLeftTable() + "_children")
|
||||
.withLabel(join.getRightTable().equals(INBOX) ? "Webhook Inbox" : join.getRightTable().equals(SENDER) ? "Sender Addresses" : join.getRightTable().equals(EVENT) ? "Events" : join.getRightTable().equals(ATTEMPT) ? "Attempts" : join.getRightTable().equals(VERSION) ? "Versions" : "Messages")
|
||||
.withMaxRows(50).getWidgetMetaData().withGridColumns(12).withPermissionRules(permissionRules()));
|
||||
}
|
||||
return (widgets);
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.outbox;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Durable handoff outcomes. Operational failures propagate to the caller.
|
||||
*******************************************************************************/
|
||||
public enum EmailEffectOutcome
|
||||
{
|
||||
QUEUED_OR_DUPLICATE,
|
||||
REFUSED
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.outbox;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Outbox handoff and system-read reconciliation for trusted effect dispatchers.
|
||||
** This API has no dependency on core durable-job types. Sending retains the
|
||||
** EmailService permission checks; callers must supply a request without a transaction.
|
||||
*******************************************************************************/
|
||||
public final class EmailEffects
|
||||
{
|
||||
/*******************************************************************************
|
||||
** A refusal can have a reference when suppression retained an audit row.
|
||||
*******************************************************************************/
|
||||
public record Result(EmailEffectOutcome outcome, EmailMessageRef message)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private EmailEffects()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Hand off once with the effect's identity. Refuse invalid or conflicting keys
|
||||
** and caller transactions: success must mean the outbox has committed already.
|
||||
** Database failures throw QException and must not become a permanent refusal.
|
||||
*******************************************************************************/
|
||||
public static Result send(EmailRequest request, String effectIdempotencyKey) throws QException
|
||||
{
|
||||
if(request == null || effectIdempotencyKey == null || effectIdempotencyKey.isBlank() || effectIdempotencyKey.length() > 500
|
||||
|| request.getTransaction() != null
|
||||
|| request.getIdempotencyKey() != null && !request.getIdempotencyKey().equals(effectIdempotencyKey))
|
||||
{
|
||||
return (new Result(EmailEffectOutcome.REFUSED, null));
|
||||
}
|
||||
request.withIdempotencyKey(effectIdempotencyKey);
|
||||
EmailSendResult result = EmailService.send(request);
|
||||
EmailMessageRef message = result.messageId() == null ? null : find("id", result.messageId()).orElseThrow(() -> new QException("Queued email reference is unavailable"));
|
||||
boolean refused = result.status() == EmailSendResult.Status.REFUSED || result.status() == EmailSendResult.Status.SUPPRESSED
|
||||
|| message != null && "SUPPRESSED".equals(message.status());
|
||||
return (new Result(refused ? EmailEffectOutcome.REFUSED : EmailEffectOutcome.QUEUED_OR_DUPLICATE, message));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Look up a committed outbox row by its exact effect or application key.
|
||||
*******************************************************************************/
|
||||
public static Optional<EmailMessageRef> findByIdempotencyKey(String key) throws QException
|
||||
{
|
||||
return (key == null || key.isBlank() ? Optional.empty() : find("idempotencyKey", key));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Look up the exact bracketed RFC Message-ID stored at enqueue time.
|
||||
*******************************************************************************/
|
||||
public static Optional<EmailMessageRef> findByRfcMessageId(String messageId) throws QException
|
||||
{
|
||||
return (messageId == null || messageId.isBlank() ? Optional.empty() : find("rfcMessageId", messageId));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Use the same authorized read scope as the dispatcher; expose only references.
|
||||
*******************************************************************************/
|
||||
private static Optional<EmailMessageRef> find(String field, java.io.Serializable value) throws QException
|
||||
{
|
||||
List<QRecord> rows = SystemWrites.query(EmailTables.MESSAGE, EmailStore.equal(field, value).withLimit(1));
|
||||
if(rows.isEmpty())
|
||||
{
|
||||
return (Optional.empty());
|
||||
}
|
||||
QRecord row = rows.getFirst();
|
||||
return (Optional.of(new EmailMessageRef(row.getValueInteger("id"), row.getValueString("status"), row.getValueString("rfcMessageId"), row.getValueString("providerMessageId"))));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.outbox;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reconciliation reference without message bodies, recipients or credentials.
|
||||
*******************************************************************************/
|
||||
public record EmailMessageRef(Integer id, String status, String rfcMessageId, String providerMessageId)
|
||||
{
|
||||
}
|
||||
@@ -39,6 +39,7 @@ public class EmailRequest
|
||||
private List<String> cc = new ArrayList<>();
|
||||
private List<String> bcc = new ArrayList<>();
|
||||
private String from;
|
||||
private Integer fromAddressId;
|
||||
private String replyTo;
|
||||
private String subject;
|
||||
private String markdownBody;
|
||||
@@ -213,6 +214,26 @@ public class EmailRequest
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for fromAddressId.
|
||||
*******************************************************************************/
|
||||
public Integer getFromAddressId()
|
||||
{
|
||||
return (fromAddressId);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for fromAddressId, a row of email_provider_sender_address that
|
||||
** must belong to the resolved provider and be enabled.
|
||||
*******************************************************************************/
|
||||
public EmailRequest withFromAddressId(Integer fromAddressId)
|
||||
{
|
||||
this.fromAddressId = fromAddressId;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for replyTo.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -30,12 +30,14 @@ import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import com.kingsrook.qbits.email.EmailQBitConfig;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTableCustomizers;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissionChecker;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissions;
|
||||
import com.kingsrook.qbits.email.render.EmailUnsubscribeLinks;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qbits.email.template.EmailTemplateRenderer;
|
||||
@@ -163,16 +165,29 @@ public final class EmailService
|
||||
List<String> bcc;
|
||||
String from;
|
||||
String replyTo;
|
||||
QRecord senderRow;
|
||||
try
|
||||
{
|
||||
senderRow = senderRow(provider, SystemWrites.query(EmailTables.SENDER, EmailStore.equal("providerId", provider.getValue("id")), queryTransaction), request);
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
return (refused(e.getMessage()));
|
||||
}
|
||||
try
|
||||
{
|
||||
to = addresses(request.getTo());
|
||||
cc = addresses(request.getCc());
|
||||
bcc = addresses(request.getBcc());
|
||||
String sender = request.getFrom() == null ? provider.getValueString("fromAddress") : request.getFrom();
|
||||
String sender = request.getFrom() != null ? request.getFrom() : senderRow != null ? senderRow.getValueString("address") : provider.getValueString("fromAddress");
|
||||
from = single(sender);
|
||||
if(request.getFrom() == null && provider.getValueString("fromName") != null)
|
||||
if(request.getFrom() == null)
|
||||
{
|
||||
from = new InternetAddress(new InternetAddress(from).getAddress(), provider.getValueString("fromName"), "UTF-8").toString();
|
||||
String name = senderRow != null ? senderRow.getValueString("displayName") : provider.getValueString("fromName");
|
||||
if(name != null && !name.isBlank())
|
||||
{
|
||||
from = new InternetAddress(new InternetAddress(from).getAddress(), name, "UTF-8").toString();
|
||||
}
|
||||
}
|
||||
String reply = request.getReplyTo() == null ? provider.getValueString("replyTo") : request.getReplyTo();
|
||||
replyTo = reply == null ? null : single(reply);
|
||||
@@ -187,6 +202,18 @@ public final class EmailService
|
||||
{
|
||||
return (refused("One or more email addresses are invalid"));
|
||||
}
|
||||
try
|
||||
{
|
||||
String unsubscribe = EmailUnsubscribeLinks.create("pending", String.join(",", to), category.name());
|
||||
if(unsubscribe != null || retained != null && retained.getValueString("unsubscribeUrl") != null)
|
||||
{
|
||||
EmailUnsubscribeLinks.recipient(String.join(",", to), String.join(",", cc), String.join(",", bcc));
|
||||
}
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
return (refused(e.getMessage()));
|
||||
}
|
||||
if(to.isEmpty() && cc.isEmpty() && bcc.isEmpty())
|
||||
{
|
||||
return (refused("At least one recipient is required"));
|
||||
@@ -214,9 +241,14 @@ public final class EmailService
|
||||
}
|
||||
boolean allSuppressed = to.isEmpty() && cc.isEmpty() && bcc.isEmpty();
|
||||
EmailRenderer.Rendered rendered;
|
||||
QRecord adHocBrand = null;
|
||||
try
|
||||
{
|
||||
rendered = templateContent == null ? renderer.render(request) : new EmailRenderer.Rendered(templateContent.getValueString("htmlBody"), templateContent.getValueString("textBody"));
|
||||
if(templateContent == null)
|
||||
{
|
||||
adHocBrand = com.kingsrook.qbits.email.render.EmailBrands.resolve(request.getBrandName(), null);
|
||||
}
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
@@ -226,9 +258,11 @@ public final class EmailService
|
||||
QRecord row = new QRecord().withValue("providerId", provider.getValue("id"))
|
||||
.withValue("status", allSuppressed ? "SUPPRESSED" : "QUEUED").withValue("finishedDate", allSuppressed ? now : null).withValue("attemptCount", 0)
|
||||
.withValue("toAddresses", String.join(",", to)).withValue("ccAddresses", String.join(",", cc)).withValue("bccAddresses", String.join(",", bcc))
|
||||
.withValue("fromAddress", from).withValue("replyTo", replyTo).withValue("subject", request.getSubject())
|
||||
.withValue("fromAddress", from).withValue("fromAddressId", senderRow == null ? null : senderRow.getValue("id")).withValue("replyTo", replyTo).withValue("subject", request.getSubject())
|
||||
.withValue("htmlBody", rendered.html()).withValue("textBody", rendered.text()).withValue("bodySha256", digest(rendered))
|
||||
.withValue("brandId", adHocBrand == null ? null : adHocBrand.getValue("id"))
|
||||
.withValue("category", category.name()).withValue("locale", request.getLocale()).withValue("idempotencyKey", request.getIdempotencyKey())
|
||||
.withValue("rfcMessageId", messageId(request.getIdempotencyKey(), from))
|
||||
.withValue("linkedTableName", request.getLinkedTableName()).withValue("linkedRecordId", request.getLinkedRecordId())
|
||||
.withValue("senderQbit", request.getSenderQbit()).withValue("senderProcess", request.getSenderProcess())
|
||||
.withValue("sendAfter", request.getSendAfter()).withValue("nextAttemptDate", request.getSendAfter() == null ? now : request.getSendAfter())
|
||||
@@ -237,14 +271,6 @@ public final class EmailService
|
||||
{
|
||||
row.getValues().putAll(templateContent.getValues());
|
||||
}
|
||||
try
|
||||
{
|
||||
com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("pending", String.join(",", to), category.name());
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
return (refused(e.getMessage()));
|
||||
}
|
||||
for(var field : config.getSecurityFields())
|
||||
{
|
||||
row.setValue(field.getName(), request.getSecurityValues().get(field.getName()));
|
||||
@@ -293,7 +319,7 @@ public final class EmailService
|
||||
return (refused("Provider is unavailable"));
|
||||
}
|
||||
EmailRequest request = new EmailRequest().withSubject(row.getValueString("subject")).withHtmlBody(row.getValueString("htmlBody"))
|
||||
.withTextBody(row.getValueString("textBody")).withFrom(row.getValueString("fromAddress")).withReplyTo(row.getValueString("replyTo"))
|
||||
.withTextBody(row.getValueString("textBody")).withFrom(row.getValueString("fromAddress")).withFromAddressId(row.getValueInteger("fromAddressId")).withReplyTo(row.getValueString("replyTo"))
|
||||
.withProviderName(provider.getValueString("name")).withCategory(EmailCategory.valueOf(row.getValueString("category")))
|
||||
.withLinkedRecord(EmailTables.MESSAGE, id).withSender("email", "email_resend");
|
||||
if(row.getValueString("toAddresses") != null && !row.getValueString("toAddresses").isBlank())
|
||||
@@ -367,26 +393,18 @@ public final class EmailService
|
||||
{
|
||||
return (new EmailSendResult(existing.getValueInteger("id"), EmailSendResult.Status.DUPLICATE, List.of()));
|
||||
}
|
||||
QRecord inserted;
|
||||
try
|
||||
{
|
||||
inserted = SystemWrites.insert(EmailTables.MESSAGE, List.of(row), transaction).getRecords().get(0);
|
||||
}
|
||||
catch(QException e)
|
||||
{
|
||||
return (refused("Could not queue the email message"));
|
||||
}
|
||||
QRecord inserted = SystemWrites.insert(EmailTables.MESSAGE, List.of(row), transaction).getRecords().get(0);
|
||||
if(inserted.getErrors() != null && !inserted.getErrors().isEmpty())
|
||||
{
|
||||
return (refused("Could not queue the email message"));
|
||||
throw (new QException("Could not queue the email message"));
|
||||
}
|
||||
try
|
||||
{
|
||||
String unsubscribe = row.getValueString("unsubscribeUrl");
|
||||
if(unsubscribe == null && (templateRenderer != null || !Boolean.TRUE.equals(row.getValueBoolean("phiApproved"))))
|
||||
if(unsubscribe == null && !"SUPPRESSED".equals(row.getValueString("status"))
|
||||
&& (templateRenderer != null || !Boolean.TRUE.equals(row.getValueBoolean("phiApproved"))))
|
||||
{
|
||||
unsubscribe = com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create(inserted.getValueString("id"),
|
||||
row.getValueString("toAddresses") + "," + row.getValueString("ccAddresses") + "," + row.getValueString("bccAddresses"), row.getValueString("category"));
|
||||
unsubscribe = EmailUnsubscribeLinks.create(inserted.getValueString("id"), row.getValueString("toAddresses"), row.getValueString("category"));
|
||||
}
|
||||
QRecord content;
|
||||
if(templateRenderer != null)
|
||||
@@ -468,6 +486,57 @@ public final class EmailService
|
||||
return (result);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The provider sender-address row a request uses, or null for the provider's own
|
||||
** from address. A chosen id, or a from text, must name an enabled row of this
|
||||
** provider; once a provider has rows, an explicit from text must match one.
|
||||
** With neither, an enabled default row wins over the provider's from address.
|
||||
*******************************************************************************/
|
||||
private static QRecord senderRow(QRecord provider, List<QRecord> rows, EmailRequest request)
|
||||
{
|
||||
String refusal = "That from address is not set up for provider " + provider.getValueString("name");
|
||||
String wanted = null;
|
||||
if(request.getFrom() != null && (request.getFromAddressId() != null || !rows.isEmpty()))
|
||||
{
|
||||
try
|
||||
{
|
||||
wanted = new InternetAddress(request.getFrom()).getAddress().toLowerCase(Locale.ROOT);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalArgumentException("One or more email addresses are invalid"));
|
||||
}
|
||||
}
|
||||
for(QRecord row : rows)
|
||||
{
|
||||
boolean enabled = Boolean.TRUE.equals(row.getValueBoolean("enabled"));
|
||||
if(request.getFromAddressId() != null)
|
||||
{
|
||||
if(request.getFromAddressId().equals(row.getValueInteger("id")))
|
||||
{
|
||||
if(!enabled || wanted != null && !wanted.equals(row.getValueString("address")))
|
||||
{
|
||||
throw (new IllegalArgumentException(refusal));
|
||||
}
|
||||
return (row);
|
||||
}
|
||||
}
|
||||
else if(wanted != null && enabled && wanted.equals(row.getValueString("address")))
|
||||
{
|
||||
return (row);
|
||||
}
|
||||
else if(request.getFrom() == null && enabled && Boolean.TRUE.equals(row.getValueBoolean("isDefault")))
|
||||
{
|
||||
return (row);
|
||||
}
|
||||
}
|
||||
if(request.getFromAddressId() != null || wanted != null)
|
||||
{
|
||||
throw (new IllegalArgumentException(refusal));
|
||||
}
|
||||
return (null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Strict address parsing, rejecting header injection and local-only addresses.
|
||||
*******************************************************************************/
|
||||
@@ -521,6 +590,42 @@ public final class EmailService
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Create once before insertion. A UUID seeds messages without an idempotency key.
|
||||
** The mailbox parser ignores display names; malformed domains use a local fallback.
|
||||
*******************************************************************************/
|
||||
static String messageId(String key, String from)
|
||||
{
|
||||
String domain = "qbit-email.local";
|
||||
try
|
||||
{
|
||||
String address = new InternetAddress(from).getAddress();
|
||||
String candidate = address.substring(address.lastIndexOf('@') + 1);
|
||||
if(address.contains("@") && candidate.matches("[A-Za-z0-9]+(?:[.-][A-Za-z0-9]+)*"))
|
||||
{
|
||||
domain = candidate.toLowerCase(Locale.ROOT);
|
||||
}
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
// A fallback also supports callers preparing legacy message rows.
|
||||
}
|
||||
try
|
||||
{
|
||||
String seed = key == null ? "qbit-email:" + UUID.randomUUID() : key;
|
||||
String token = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(seed.getBytes(StandardCharsets.UTF_8))).substring(0, 32);
|
||||
return ("<" + token + "@" + domain + ">");
|
||||
}
|
||||
catch(java.security.NoSuchAlgorithmException e)
|
||||
{
|
||||
throw (new IllegalStateException("SHA-256 unavailable"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Refusal with no stored row.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -74,7 +74,7 @@ public class EmailPermissionChecker implements CustomPermissionChecker
|
||||
boolean write = actionInput instanceof InsertInput || actionInput instanceof UpdateInput || actionInput instanceof DeleteInput;
|
||||
if(metaData instanceof QTableMetaData table)
|
||||
{
|
||||
if(EmailTables.PROVIDER.equals(table.getName()))
|
||||
if(EmailTables.PROVIDER.equals(table.getName()) || EmailTables.SENDER.equals(table.getName()))
|
||||
{
|
||||
return (EmailPermissions.ADMIN);
|
||||
}
|
||||
|
||||
@@ -86,8 +86,13 @@ public final class EmailProcesses
|
||||
public static final String SEND_TEMPLATED = "email_send_templated";
|
||||
public static final String DISPATCH = "email_dispatch";
|
||||
public static final String MAINTAIN = "email_maintain";
|
||||
public static final String REPROCESS = "email_reprocess_webhook";
|
||||
public static final String PROCESS_WEBHOOKS = "email_process_webhooks";
|
||||
public static final String RETENTION = "email_retention";
|
||||
|
||||
/** Process value the send-test prepare step sets to the selected provider id; the from address filter falls back to it. */
|
||||
public static final String PROVIDER_FILTER_VALUE = "possibleValueFilterValueId";
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
@@ -102,7 +107,7 @@ public final class EmailProcesses
|
||||
{
|
||||
return (switch(name)
|
||||
{
|
||||
case RESEND, CANCEL, SEND_NOW, NEW_VERSION, ACTIVATE_VERSION -> EmailPermissions.MANAGE;
|
||||
case REPROCESS, RESEND, CANCEL, SEND_NOW, NEW_VERSION, ACTIVATE_VERSION -> EmailPermissions.MANAGE;
|
||||
case SEND_TEMPLATED, SEND_AD_HOC -> EmailPermissions.SEND;
|
||||
default -> EmailPermissions.ADMIN;
|
||||
});
|
||||
@@ -114,9 +119,12 @@ public final class EmailProcesses
|
||||
public static List<QProcessMetaData> processes(EmailQBitConfig config)
|
||||
{
|
||||
QProcessMetaData test = action(SEND_TEST, "Send Test Email", EmailTables.PROVIDER);
|
||||
test.withStep(0, new QFrontendStepMetaData().withName("input").withLabel("Recipient")
|
||||
test.withStep(0, new QBackendStepMetaData().withName("prepare").withCode(new QCodeReference(SendTestPrepareStep.class)));
|
||||
test.withStep(1, new QFrontendStepMetaData().withName("input").withLabel("Recipient")
|
||||
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM))
|
||||
.withFormField(new QFieldMetaData("toAddress", QFieldType.STRING).withLabel("To address").withIsRequired(true)));
|
||||
.withFormField(new QFieldMetaData("toAddress", QFieldType.STRING).withLabel("To address").withIsRequired(true))
|
||||
.withFormField(new QFieldMetaData("fromAddressId", QFieldType.INTEGER).withLabel("From address").withPossibleValueSourceName(EmailTables.SENDER)
|
||||
.withPossibleValueSourceFilter(EmailStore.equal("providerId", "${input.id}??${processValues." + PROVIDER_FILTER_VALUE + "}"))));
|
||||
QFunctionInputMetaData inputs = new QFunctionInputMetaData();
|
||||
QFunctionInputMetaData adHocInputs = new QFunctionInputMetaData();
|
||||
for(String field : List.of("templateName", "to", "cc", "bcc", "replyTo", "subject", "markdownBody", "htmlBody", "textBody", "providerName", "brandName",
|
||||
@@ -126,6 +134,12 @@ public final class EmailProcesses
|
||||
adHocInputs.withField(new QFieldMetaData(field, QFieldType.STRING));
|
||||
}
|
||||
adHocInputs.withField(new QFieldMetaData("category", QFieldType.STRING));
|
||||
for(QFunctionInputMetaData target : List.of(inputs, adHocInputs))
|
||||
{
|
||||
target.withField(new QFieldMetaData("providerId", QFieldType.INTEGER).withPossibleValueSourceName(EmailTables.PROVIDER));
|
||||
target.withField(new QFieldMetaData("fromAddressId", QFieldType.INTEGER).withPossibleValueSourceName(EmailTables.SENDER)
|
||||
.withPossibleValueSourceFilter(EmailStore.equal("providerId", "${input.providerId}")));
|
||||
}
|
||||
QProcessMetaData adHoc = new QProcessMetaData().withName(SEND_AD_HOC).withLabel("Send Ad Hoc Email").withIsHidden(true)
|
||||
.withPermissionRules(EmailTables.permissionRules())
|
||||
.withStep(new QBackendStepMetaData().withName("send").withCode(new QCodeReference(SendStep.class)).withInputData(adHocInputs));
|
||||
@@ -136,6 +150,7 @@ public final class EmailProcesses
|
||||
action(SEND_NOW, "Send Now", EmailTables.MESSAGE), test, send, adHoc,
|
||||
action(NEW_VERSION, "New Version", EmailTables.TEMPLATE), action(ACTIVATE_VERSION, "Activate Version", EmailTables.VERSION),
|
||||
action(APPROVE_PHI, "Approve For PHI", EmailTables.VERSION),
|
||||
action(REPROCESS, "Reprocess", EmailTables.INBOX), scheduled(config, PROCESS_WEBHOOKS, "Process Email Webhooks", 30),
|
||||
scheduled(config, DISPATCH, "Dispatch Email", config.getDispatchIntervalSeconds()),
|
||||
scheduled(config, MAINTAIN, "Maintain Email", 300), scheduled(config, RETENTION, "Email Retention", 86400)));
|
||||
}
|
||||
@@ -175,6 +190,10 @@ public final class EmailProcesses
|
||||
*******************************************************************************/
|
||||
private static boolean eligible(String process, QRecord row)
|
||||
{
|
||||
if(REPROCESS.equals(process))
|
||||
{
|
||||
return ("FAILED".equals(row.getValueString("status")));
|
||||
}
|
||||
if(NEW_VERSION.equals(process))
|
||||
{
|
||||
return (true);
|
||||
@@ -266,7 +285,7 @@ public final class EmailProcesses
|
||||
{
|
||||
Integer id = row.getValueInteger("id");
|
||||
QRecord current = EmailStore.get(NEW_VERSION.equals(process) ? EmailTables.TEMPLATE : List.of(ACTIVATE_VERSION, APPROVE_PHI).contains(process) ? EmailTables.VERSION
|
||||
: SEND_TEST.equals(process) ? EmailTables.PROVIDER : EmailTables.MESSAGE, id, null);
|
||||
: REPROCESS.equals(process) ? EmailTables.INBOX : SEND_TEST.equals(process) ? EmailTables.PROVIDER : EmailTables.MESSAGE, id, null);
|
||||
String result = "Skipped";
|
||||
if(current != null && eligible(process, current))
|
||||
{
|
||||
@@ -283,10 +302,11 @@ public final class EmailProcesses
|
||||
com.kingsrook.qbits.email.template.EmailVersions.approve(id);
|
||||
yield "Approved for PHI";
|
||||
}
|
||||
case REPROCESS -> com.kingsrook.qbits.email.tracking.EmailInboxProcessor.reprocess(id) ? "PENDING" : "Skipped";
|
||||
case SEND_NOW -> new EmailDispatcher().sendNow(id).name();
|
||||
case CANCEL -> cancel(id) ? "CANCELLED" : "Skipped";
|
||||
case RESEND -> resend(current).status().name();
|
||||
case SEND_TEST -> sendTest(current, input.getValueString("toAddress"));
|
||||
case SEND_TEST -> sendTest(current, input.getValueString("toAddress"), input.getValueInteger("fromAddressId"));
|
||||
default -> "Skipped";
|
||||
};
|
||||
}
|
||||
@@ -340,17 +360,45 @@ public final class EmailProcesses
|
||||
/*******************************************************************************
|
||||
** Queue and dispatch a fixed provider test message.
|
||||
*******************************************************************************/
|
||||
private static String sendTest(QRecord provider, String to) throws QException
|
||||
private static String sendTest(QRecord provider, String to, Integer fromAddressId) throws QException
|
||||
{
|
||||
if(to == null)
|
||||
{
|
||||
return ("REFUSED: a to address is required");
|
||||
}
|
||||
EmailSendResult result = EmailService.send(new EmailRequest().withProviderName(provider.getValueString("name")).withTo(to)
|
||||
EmailSendResult result = EmailService.send(new EmailRequest().withProviderName(provider.getValueString("name")).withTo(to).withFromAddressId(fromAddressId)
|
||||
.withSubject("Email delivery test").withMarkdownBody("This is a test of your email provider.").withSender("email", SEND_TEST));
|
||||
return (result.status() == EmailSendResult.Status.QUEUED ? new EmailDispatcher().sendNow(result.messageId()).name() : result.status().name());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Seeds the from address possible-value filter: a record action's form has no
|
||||
** record id, so a single selected provider id becomes a process value that the
|
||||
** filter reads after the form's own ${input.id}.
|
||||
*******************************************************************************/
|
||||
public static class SendTestPrepareStep implements BackendStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Expose the selected provider id when exactly one provider is selected.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
String ids = input.getValueString("recordIds");
|
||||
if(ids != null && !ids.contains(","))
|
||||
{
|
||||
try
|
||||
{
|
||||
output.addValue(PROVIDER_FILTER_VALUE, Integer.valueOf(ids.trim()));
|
||||
}
|
||||
catch(NumberFormatException e)
|
||||
{
|
||||
// Not a single id: the filter stays open and the review step still validates the selection.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Backend sending API with inputs declared as QQQ fields.
|
||||
*******************************************************************************/
|
||||
@@ -362,10 +410,20 @@ public final class EmailProcesses
|
||||
@Override
|
||||
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
EmailRequest request = new EmailRequest().withTemplateName(input.getValueString("templateName"))
|
||||
String providerName = input.getValueString("providerName");
|
||||
if(input.getValueInteger("providerId") != null)
|
||||
{
|
||||
QRecord provider = EmailStore.get(EmailTables.PROVIDER, input.getValueInteger("providerId"), null);
|
||||
if(provider == null || providerName != null && !providerName.equals(provider.getValueString("name")))
|
||||
{
|
||||
throw (new QException("providerId does not match an existing provider named providerName"));
|
||||
}
|
||||
providerName = provider.getValueString("name");
|
||||
}
|
||||
EmailRequest request = new EmailRequest().withTemplateName(input.getValueString("templateName")).withFromAddressId(input.getValueInteger("fromAddressId"))
|
||||
.withSubject(input.getValueString("subject")).withMarkdownBody(input.getValueString("markdownBody"))
|
||||
.withHtmlBody(input.getValueString("htmlBody")).withTextBody(input.getValueString("textBody"))
|
||||
.withReplyTo(input.getValueString("replyTo")).withProviderName(input.getValueString("providerName"))
|
||||
.withReplyTo(input.getValueString("replyTo")).withProviderName(providerName)
|
||||
.withBrandName(input.getValueString("brandName")).withIdempotencyKey(input.getValueString("idempotencyKey"))
|
||||
.withLinkedRecord(input.getValueString("linkedTableName"), input.getValueString("linkedRecordId"));
|
||||
if(input.getValueString("to") != null)
|
||||
@@ -431,6 +489,7 @@ public final class EmailProcesses
|
||||
{
|
||||
switch(input.getProcessName())
|
||||
{
|
||||
case PROCESS_WEBHOOKS -> new com.kingsrook.qbits.email.tracking.EmailInboxProcessor().runOnce();
|
||||
case RETENTION -> retain();
|
||||
case MAINTAIN -> new EmailDispatcher().maintain();
|
||||
default -> new EmailDispatcher().runOnce();
|
||||
@@ -440,6 +499,8 @@ public final class EmailProcesses
|
||||
|
||||
/*******************************************************************************
|
||||
** Clear retained bodies and optionally delete final messages and their attempts.
|
||||
** NOTIFICATION and MARKETING messages are never deleted (their unsubscribe links must
|
||||
** keep working); at message retention only their bodies and variable snapshot are cleared.
|
||||
*******************************************************************************/
|
||||
public static void retain() throws QException
|
||||
{
|
||||
@@ -464,18 +525,35 @@ public final class EmailProcesses
|
||||
{
|
||||
continue;
|
||||
}
|
||||
if(config.getMessageRetentionDays() > 0 && finalDate.isBefore(now.minus(Duration.ofDays(config.getMessageRetentionDays()))))
|
||||
boolean expired = config.getMessageRetentionDays() > 0 && finalDate.isBefore(now.minus(Duration.ofDays(config.getMessageRetentionDays())));
|
||||
boolean optOut = List.of("NOTIFICATION", "MARKETING").contains(row.getValueString("category"));
|
||||
if(expired && !optOut)
|
||||
{
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openFor(new UpdateInput(EmailTables.MESSAGE)))
|
||||
{
|
||||
if(row.getValueInteger("providerId") != null)
|
||||
{
|
||||
com.kingsrook.qbits.email.tracking.EmailTrackingStore.lock(row.getValueInteger("providerId"), transaction);
|
||||
}
|
||||
SystemWrites.delete(EmailTables.EVENT, EmailStore.equal("messageId", row.getValueInteger("id")), transaction);
|
||||
SystemWrites.delete(EmailTables.ATTEMPT, EmailStore.equal("messageId", row.getValueInteger("id")), transaction);
|
||||
SystemWrites.delete(EmailTables.MESSAGE, EmailStore.equal("id", row.getValueInteger("id")), transaction);
|
||||
transaction.commit();
|
||||
}
|
||||
}
|
||||
else if(finalDate.isBefore(now.minus(Duration.ofDays(config.getBodyRetentionDays()))) && (row.getValue("htmlBody") != null || row.getValue("textBody") != null))
|
||||
else if(expired || finalDate.isBefore(now.minus(Duration.ofDays(config.getBodyRetentionDays()))))
|
||||
{
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", row.getValue("id")).withValue("htmlBody", null).withValue("textBody", null));
|
||||
// Notification and marketing rows are never deleted, so their unsubscribe links keep working; only content goes.
|
||||
boolean clearSnapshot = expired && row.getValue("variablesSnapshot") != null;
|
||||
if(row.getValue("htmlBody") != null || row.getValue("textBody") != null || clearSnapshot)
|
||||
{
|
||||
QRecord update = new QRecord().withValue("id", row.getValue("id")).withValue("htmlBody", null).withValue("textBody", null);
|
||||
if(clearSnapshot)
|
||||
{
|
||||
update.withValue("variablesSnapshot", null);
|
||||
}
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, update);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -27,13 +27,15 @@ import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.Base64;
|
||||
import java.util.HexFormat;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.template.VariableSchema;
|
||||
import jakarta.mail.internet.InternetAddress;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Queue-time link signing from DESIGN 8.2. The receiving route is phase (c).
|
||||
** Queue-time link signing and verification from DESIGN 8.2.
|
||||
*******************************************************************************/
|
||||
public final class EmailUnsubscribeLinks
|
||||
{
|
||||
@@ -45,39 +47,104 @@ public final class EmailUnsubscribeLinks
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Bind the message, recipient-envelope hash and category without exposing PII.
|
||||
** Bind the message, single mailbox hash and category without exposing PII.
|
||||
** No key means no link/header; marketing must refuse rather than omit opt-out.
|
||||
*******************************************************************************/
|
||||
public static String create(String messageId, String recipients, String category)
|
||||
public static String create(String messageId, String recipient, String category)
|
||||
{
|
||||
if(!"NOTIFICATION".equals(category) && !"MARKETING".equals(category))
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
String key;
|
||||
try
|
||||
{
|
||||
String key = EmailRuntime.config().getSecretResolver().apply("QQQ_EMAIL_UNSUBSCRIBE_KEY");
|
||||
String base = EmailRuntime.config().getPublicBaseUrl();
|
||||
if(key == null || key.isBlank() || base == null || base.isBlank())
|
||||
key = EmailRuntime.config().getSecretResolver().apply("QQQ_EMAIL_UNSUBSCRIBE_KEY");
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalArgumentException("Opt-out email needs publicBaseUrl and QQQ_EMAIL_UNSUBSCRIBE_KEY"));
|
||||
}
|
||||
String base = EmailRuntime.config().getPublicBaseUrl();
|
||||
if(key == null || key.isBlank() || base == null || base.isBlank())
|
||||
{
|
||||
if("MARKETING".equals(category))
|
||||
{
|
||||
if("MARKETING".equals(category))
|
||||
{
|
||||
throw (new IllegalArgumentException());
|
||||
}
|
||||
return (null);
|
||||
throw (new IllegalArgumentException("Opt-out email needs publicBaseUrl and QQQ_EMAIL_UNSUBSCRIBE_KEY"));
|
||||
}
|
||||
return (null);
|
||||
}
|
||||
String address = recipient(recipient, null, null);
|
||||
try
|
||||
{
|
||||
VariableSchema.url(base);
|
||||
String hash = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(recipients.getBytes(StandardCharsets.UTF_8)));
|
||||
String hash = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(address.getBytes(StandardCharsets.UTF_8)));
|
||||
String payload = Base64.getUrlEncoder().withoutPadding().encodeToString(VariableSchema.json(Map.of("messageId", messageId, "addressHash", hash, "category", category)).getBytes(StandardCharsets.UTF_8));
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(key.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
String signed = "v1." + payload;
|
||||
String signature = Base64.getUrlEncoder().withoutPadding().encodeToString(mac.doFinal(signed.getBytes(StandardCharsets.UTF_8)));
|
||||
return (base.replaceAll("/+$", "") + "/qqq/hooks/email/unsubscribe/" + signed + "." + signature);
|
||||
return (base.replaceAll("/+$", "") + EmailRuntime.config().getWebhookBasePath() + "/unsubscribe/" + signed + "." + signature);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalArgumentException("Opt-out email needs publicBaseUrl and QQQ_EMAIL_UNSUBSCRIBE_KEY"));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Exactly one To mailbox, canonicalized for signing and category suppression.
|
||||
*******************************************************************************/
|
||||
public static String recipient(String to, String cc, String bcc)
|
||||
{
|
||||
try
|
||||
{
|
||||
InternetAddress[] addresses = InternetAddress.parse(to == null ? "" : to, true);
|
||||
if(addresses.length == 1 && !addresses[0].isGroup() && addresses[0].getAddress().contains("@")
|
||||
&& (cc == null || cc.isBlank()) && (bcc == null || bcc.isBlank()))
|
||||
{
|
||||
addresses[0].validate();
|
||||
return (addresses[0].getAddress().toLowerCase(Locale.ROOT));
|
||||
}
|
||||
}
|
||||
catch(jakarta.mail.internet.AddressException e)
|
||||
{
|
||||
// Malformed stored envelopes cannot authorize an opt-out.
|
||||
}
|
||||
throw (new IllegalArgumentException("Notification and marketing email with an unsubscribe link goes to one recipient per message; send one message per recipient."));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify the signed format before any lookup or side effect.
|
||||
** Invalid and non-opt-out categories are indistinguishable to callers.
|
||||
*******************************************************************************/
|
||||
public static org.json.JSONObject verify(String token)
|
||||
{
|
||||
try
|
||||
{
|
||||
if(token == null || token.length() > 4096)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
String[] parts = token.split("\\.", -1);
|
||||
String key = EmailRuntime.config().getSecretResolver().apply("QQQ_EMAIL_UNSUBSCRIBE_KEY");
|
||||
if(parts.length != 3 || !"v1".equals(parts[0]) || key == null || key.isBlank())
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(key.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
if(!MessageDigest.isEqual(mac.doFinal((parts[0] + "." + parts[1]).getBytes(StandardCharsets.UTF_8)), Base64.getUrlDecoder().decode(parts[2])))
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
var payload = new org.json.JSONObject(new String(Base64.getUrlDecoder().decode(parts[1]), StandardCharsets.UTF_8));
|
||||
String category = payload.getString("category");
|
||||
return (("MARKETING".equals(category) || "NOTIFICATION".equals(category)) ? payload : null);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.template;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.email.outbox.EmailCategory;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A template another qbit ships with its own code: its name, content and
|
||||
** variable contract. Hand it to {@link EmailTemplates#ensure} at startup or on
|
||||
** first use; it is created once as an ACTIVE version 1 and never overwritten.
|
||||
*******************************************************************************/
|
||||
public final class EmailTemplateDefinition
|
||||
{
|
||||
private String name;
|
||||
private String label;
|
||||
private EmailCategory category;
|
||||
private String kind = "MARKDOWN";
|
||||
private String subject;
|
||||
private String body;
|
||||
private String textBody;
|
||||
private String variablesSchema;
|
||||
private final List<Map<String, Object>> variables = new ArrayList<>();
|
||||
private String sampleVariables;
|
||||
private String ownerQbit;
|
||||
private String editPermissionName;
|
||||
|
||||
/*******************************************************************************
|
||||
** Unique snake_case template name, as used by EmailRequest.withTemplateName.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withName(String name)
|
||||
{
|
||||
this.name = name;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Human label shown in the template table.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withLabel(String label)
|
||||
{
|
||||
this.label = label;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Template category; it decides opt-out and PHI rules.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withCategory(EmailCategory category)
|
||||
{
|
||||
this.category = category;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** MARKDOWN (the default), MJML or HTML.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withKind(String kind)
|
||||
{
|
||||
this.kind = kind;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Subject source, which may use variables.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withSubject(String subject)
|
||||
{
|
||||
this.subject = subject;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Body source in the template's kind.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withBody(String body)
|
||||
{
|
||||
this.body = body;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Optional plain-text source; omitted, it is derived from the body.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withTextBody(String textBody)
|
||||
{
|
||||
this.textBody = textBody;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The variables schema as the JSON list the template table stores. Use this or
|
||||
** withVariable, not both.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withVariablesSchema(String variablesSchema)
|
||||
{
|
||||
this.variablesSchema = variablesSchema;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Declare one variable (type: STRING, NUMBER, DATE, DATE_TIME, BOOLEAN, URL,
|
||||
** LIST or RECORD; classification: PUBLIC, CONFIDENTIAL, PII, PHI or null).
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withVariable(String name, String label, String type, boolean required, String classification)
|
||||
{
|
||||
Map<String, Object> entry = new LinkedHashMap<>();
|
||||
entry.put("name", name);
|
||||
entry.put("label", label);
|
||||
entry.put("type", type);
|
||||
entry.put("required", required);
|
||||
entry.put("classification", classification);
|
||||
variables.add(entry);
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Synthetic sample values as a JSON object, used to validate and preview.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withSampleVariables(String sampleVariables)
|
||||
{
|
||||
this.sampleVariables = sampleVariables;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The qbit that owns this template.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withOwnerQbit(String ownerQbit)
|
||||
{
|
||||
this.ownerQbit = ownerQbit;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Optional permission, besides the email permission, needed to edit the template.
|
||||
*******************************************************************************/
|
||||
public EmailTemplateDefinition withEditPermissionName(String editPermissionName)
|
||||
{
|
||||
this.editPermissionName = editPermissionName;
|
||||
return (this);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The template name.
|
||||
*******************************************************************************/
|
||||
public String getName()
|
||||
{
|
||||
return (name);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The template label.
|
||||
*******************************************************************************/
|
||||
public String getLabel()
|
||||
{
|
||||
return (label);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The template category.
|
||||
*******************************************************************************/
|
||||
public EmailCategory getCategory()
|
||||
{
|
||||
return (category);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The template kind.
|
||||
*******************************************************************************/
|
||||
public String getKind()
|
||||
{
|
||||
return (kind);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The subject source.
|
||||
*******************************************************************************/
|
||||
public String getSubject()
|
||||
{
|
||||
return (subject);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The body source.
|
||||
*******************************************************************************/
|
||||
public String getBody()
|
||||
{
|
||||
return (body);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The optional plain-text source.
|
||||
*******************************************************************************/
|
||||
public String getTextBody()
|
||||
{
|
||||
return (textBody);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The schema JSON: the explicit text, or the typed variables as a JSON list.
|
||||
*******************************************************************************/
|
||||
public String getVariablesSchema()
|
||||
{
|
||||
if(variablesSchema != null && !variablesSchema.isBlank())
|
||||
{
|
||||
return (variablesSchema);
|
||||
}
|
||||
return (VariableSchema.json(variables));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The sample values JSON, defaulting to an empty object.
|
||||
*******************************************************************************/
|
||||
public String getSampleVariables()
|
||||
{
|
||||
return (sampleVariables == null || sampleVariables.isBlank() ? "{}" : sampleVariables);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The owning qbit.
|
||||
*******************************************************************************/
|
||||
public String getOwnerQbit()
|
||||
{
|
||||
return (ownerQbit);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The optional edit permission.
|
||||
*******************************************************************************/
|
||||
public String getEditPermissionName()
|
||||
{
|
||||
return (editPermissionName);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Plain-language reasons the definition is incomplete or malformed, before any
|
||||
** rendering check. Empty when it passes.
|
||||
*******************************************************************************/
|
||||
public List<String> problems()
|
||||
{
|
||||
List<String> reasons = new ArrayList<>();
|
||||
if(name == null || !name.matches("[a-z][a-z0-9_]{0,99}"))
|
||||
{
|
||||
reasons.add("Name must be snake_case: lowercase letters, digits and underscores, starting with a letter");
|
||||
}
|
||||
if(label == null || label.isBlank())
|
||||
{
|
||||
reasons.add("Label is required");
|
||||
}
|
||||
if(category == null)
|
||||
{
|
||||
reasons.add("Category is required");
|
||||
}
|
||||
if(kind == null || !List.of("MARKDOWN", "MJML", "HTML").contains(kind))
|
||||
{
|
||||
reasons.add("Kind must be MARKDOWN, MJML or HTML");
|
||||
}
|
||||
if(subject == null || subject.isBlank())
|
||||
{
|
||||
reasons.add("Subject is required");
|
||||
}
|
||||
if(body == null || body.isBlank())
|
||||
{
|
||||
reasons.add("Body is required");
|
||||
}
|
||||
if(variablesSchema != null && !variablesSchema.isBlank() && !variables.isEmpty())
|
||||
{
|
||||
reasons.add("Declare variables with withVariablesSchema or withVariable, not both");
|
||||
}
|
||||
else
|
||||
{
|
||||
try
|
||||
{
|
||||
new VariableSchema(getVariablesSchema());
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
reasons.add(e.getMessage());
|
||||
}
|
||||
}
|
||||
try
|
||||
{
|
||||
VariableSchema.values(sampleVariables);
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
reasons.add(e.getMessage());
|
||||
}
|
||||
return (reasons);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.template;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Entry point for other qbits to ship their templates. A template is created
|
||||
** once; after that this class never touches it, so people's edits are safe.
|
||||
*******************************************************************************/
|
||||
public final class EmailTemplates
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private EmailTemplates()
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Create the template and an ACTIVE version 1 when no template of that name
|
||||
** exists. The definition is validated like Activate Version (syntax, schema, a
|
||||
** render with the samples, MJML compile); an invalid one is refused with
|
||||
** plain reasons and nothing is written. Returns true only when this call
|
||||
** created the template; the unique name arbitrates concurrent callers.
|
||||
*******************************************************************************/
|
||||
public static boolean ensure(EmailTemplateDefinition definition) throws QException
|
||||
{
|
||||
List<String> reasons = definition.problems();
|
||||
if(!reasons.isEmpty())
|
||||
{
|
||||
throw (new IllegalArgumentException("Template definition refused: " + String.join("; ", reasons)));
|
||||
}
|
||||
return (SystemWrites.call(() -> create(definition)));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The checked, transactional creation, run as the qbit.
|
||||
*******************************************************************************/
|
||||
private static boolean create(EmailTemplateDefinition definition) throws QException
|
||||
{
|
||||
if(exists(definition.getName()))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
QRecord template = new QRecord().withValue("name", definition.getName()).withValue("label", definition.getLabel())
|
||||
.withValue("category", definition.getCategory().name()).withValue("kind", definition.getKind()).withValue("phiApproved", false)
|
||||
.withValue("ownerQbit", definition.getOwnerQbit()).withValue("editPermissionName", definition.getEditPermissionName());
|
||||
QRecord version = new QRecord().withValue("versionNumber", 1).withValue("status", "ACTIVE").withValue("locale", "en")
|
||||
.withValue("subject", definition.getSubject()).withValue("bodySource", definition.getBody()).withValue("textSource", definition.getTextBody())
|
||||
.withValue("variablesSchema", definition.getVariablesSchema()).withValue("sampleVariables", definition.getSampleVariables());
|
||||
try
|
||||
{
|
||||
EmailVersions.validate(template, version);
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
throw (new IllegalArgumentException("Template definition refused: " + e.getMessage()));
|
||||
}
|
||||
try(QBackendTransaction tx = QBackendTransaction.openFor(new InsertInput(EmailTables.TEMPLATE)))
|
||||
{
|
||||
QRecord stored = SystemWrites.insert(EmailTables.TEMPLATE, List.of(template), tx).getRecords().getFirst();
|
||||
if(stored.getValue("id") == null || stored.getErrors() != null && !stored.getErrors().isEmpty())
|
||||
{
|
||||
tx.rollback();
|
||||
if(exists(definition.getName()))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
throw (new QException("Could not create template " + definition.getName()));
|
||||
}
|
||||
version.withValue("templateId", stored.getValue("id"));
|
||||
QRecord storedVersion = SystemWrites.insert(EmailTables.VERSION, List.of(version), tx).getRecords().getFirst();
|
||||
if(storedVersion.getValue("id") == null || storedVersion.getErrors() != null && !storedVersion.getErrors().isEmpty())
|
||||
{
|
||||
throw (new QException("Could not create template version for " + definition.getName()));
|
||||
}
|
||||
EmailStore.update(EmailTables.TEMPLATE, new QRecord().withValue("id", stored.getValue("id")).withValue("activeVersionId", storedVersion.getValue("id")), tx);
|
||||
tx.commit();
|
||||
return (true);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether a template of that name is already present.
|
||||
*******************************************************************************/
|
||||
private static boolean exists(String name) throws QException
|
||||
{
|
||||
return (!SystemWrites.query(EmailTables.TEMPLATE, EmailStore.equal("name", name)).isEmpty());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,302 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Instant;
|
||||
import java.time.OffsetDateTime;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import org.json.JSONArray;
|
||||
import org.json.JSONObject;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Provider payloads become small recipient-level events, never message bodies.
|
||||
*******************************************************************************/
|
||||
public final class EmailEventParser
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Provider identifiers remain scoped to the provider during database matching.
|
||||
*******************************************************************************/
|
||||
public record Event(String externalId, String messageId, String providerMessageId, String type, String bounceType,
|
||||
String recipient, Instant occurred, String detail, String url)
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Parse one provider request, including all SendGrid events and SES recipients.
|
||||
*******************************************************************************/
|
||||
public List<Event> parse(String kind, String body, Instant received)
|
||||
{
|
||||
List<Event> events = new ArrayList<>();
|
||||
switch(kind)
|
||||
{
|
||||
case "SES" -> ses(new JSONObject(body), received, events);
|
||||
case "POSTMARK" -> postmark(new JSONObject(body), received, events);
|
||||
case "MAILGUN" -> mailgun(new JSONObject(body).getJSONObject("event-data"), received, events);
|
||||
case "SENDGRID" ->
|
||||
{
|
||||
JSONArray batch = new JSONArray(body);
|
||||
for(int i = 0; i < batch.length(); i++)
|
||||
{
|
||||
sendgrid(batch.getJSONObject(i), received, events);
|
||||
}
|
||||
}
|
||||
default -> throw new IllegalArgumentException("Unsupported email event kind");
|
||||
}
|
||||
return (events);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** SES recipients live under the specific event, with mail.destination fallback for non-suppressive events only.
|
||||
*******************************************************************************/
|
||||
private static void ses(JSONObject envelope, Instant received, List<Event> events)
|
||||
{
|
||||
if(!"Notification".equals(envelope.getString("Type")))
|
||||
{
|
||||
return;
|
||||
}
|
||||
JSONObject payload = new JSONObject(envelope.getString("Message"));
|
||||
String rawType = payload.optString("eventType", payload.optString("notificationType"));
|
||||
String type = switch(rawType)
|
||||
{
|
||||
case "Delivery" -> "DELIVERED";
|
||||
case "Bounce" -> "BOUNCED";
|
||||
case "Complaint" -> "COMPLAINED";
|
||||
case "Reject" -> "REJECTED";
|
||||
case "DeliveryDelay" -> "DEFERRED";
|
||||
case "Open" -> "OPENED";
|
||||
case "Click" -> "CLICKED";
|
||||
default -> null;
|
||||
};
|
||||
if(type == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
JSONObject mail = payload.getJSONObject("mail");
|
||||
String key = "DeliveryDelay".equals(rawType) ? "deliveryDelay" : rawType.toLowerCase(Locale.ROOT);
|
||||
JSONObject data = object(payload, key);
|
||||
String bounce = "BOUNCED".equals(type) ? switch(data.optString("bounceType"))
|
||||
{
|
||||
case "Permanent" -> "HARD";
|
||||
case "Transient" -> "SOFT";
|
||||
default -> "UNDETERMINED";
|
||||
} : null;
|
||||
String messageId = null;
|
||||
JSONArray headers = mail.optJSONArray("headers");
|
||||
if(headers != null)
|
||||
{
|
||||
for(int i = 0; i < headers.length(); i++)
|
||||
{
|
||||
JSONObject entry = headers.getJSONObject(i);
|
||||
if("X-QQQ-Email-Id".equalsIgnoreCase(entry.optString("name")))
|
||||
{
|
||||
messageId = entry.optString("value", null);
|
||||
}
|
||||
}
|
||||
}
|
||||
JSONArray recipients = data.optJSONArray(switch(type)
|
||||
{
|
||||
case "BOUNCED" -> "bouncedRecipients";
|
||||
case "COMPLAINED" -> "complainedRecipients";
|
||||
case "DEFERRED" -> "delayedRecipients";
|
||||
default -> "recipients";
|
||||
});
|
||||
if(recipients == null && ("BOUNCED".equals(type) || "COMPLAINED".equals(type)))
|
||||
{
|
||||
// Never widen a bounce or complaint to every destination; the event carries no recipient and is skipped.
|
||||
events.add(new Event(envelope.getString("MessageId"), messageId, mail.getString("messageId"), type, bounce, null,
|
||||
time(data.opt("timestamp"), received), null, null));
|
||||
return;
|
||||
}
|
||||
if(recipients == null)
|
||||
{
|
||||
recipients = mail.getJSONArray("destination");
|
||||
}
|
||||
for(int i = 0; i < recipients.length(); i++)
|
||||
{
|
||||
Object recipient = recipients.get(i);
|
||||
String address = recipient instanceof JSONObject row ? row.getString("emailAddress") : recipient.toString();
|
||||
String reason = recipient instanceof JSONObject row ? row.optString("diagnosticCode", data.optString("bounceSubType", null))
|
||||
: data.optString("reason", data.optString("smtpResponse", data.optString("delayType", data.optString("complaintFeedbackType", null))));
|
||||
events.add(new Event(envelope.getString("MessageId"), messageId, mail.getString("messageId"), type, bounce, address,
|
||||
time(data.opt("timestamp"), received), detail(reason), "CLICKED".equals(type) ? data.optString("link", null) : null));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Postmark subscription reactivation is informational, never an unsubscribe.
|
||||
*******************************************************************************/
|
||||
private static void postmark(JSONObject data, Instant received, List<Event> events)
|
||||
{
|
||||
String type = switch(data.getString("RecordType"))
|
||||
{
|
||||
case "Delivery" -> "DELIVERED";
|
||||
case "Bounce" -> "BOUNCED";
|
||||
case "SpamComplaint" -> "COMPLAINED";
|
||||
case "Open" -> "OPENED";
|
||||
case "Click" -> "CLICKED";
|
||||
case "SubscriptionChange" -> data.optBoolean("SuppressSending", false) ? "UNSUBSCRIBED" : null;
|
||||
default -> null;
|
||||
};
|
||||
if(type == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
String bounce = "BOUNCED".equals(type) ? switch(data.optString("Type"))
|
||||
{
|
||||
case "HardBounce", "SpamNotification", "SpamComplaint", "BadEmailAddress", "ManuallyDeactivated" -> "HARD";
|
||||
case "SoftBounce", "Transient", "DnsError", "MailboxFull" -> "SOFT";
|
||||
default -> "UNDETERMINED";
|
||||
} : null;
|
||||
Object date = data.opt("DeliveredAt");
|
||||
for(String field : List.of("BouncedAt", "ReceivedAt", "ChangedAt"))
|
||||
{
|
||||
if(data.has(field))
|
||||
{
|
||||
date = data.get(field);
|
||||
}
|
||||
}
|
||||
String id = data.has("ID") ? data.get("RecordType") + ":" + data.get("ID") : fingerprint(data);
|
||||
events.add(new Event(id, null, data.optString("MessageID", null), type, bounce,
|
||||
data.optString("Recipient", data.optString("Email", null)), time(date, received),
|
||||
detail(data.optString("Description", data.optString("Details", data.optString("SuppressionReason", null)))),
|
||||
"CLICKED".equals(type) ? data.optString("OriginalLink", null) : null));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** SendGrid retains the provider message id prefix before its delivery suffix.
|
||||
*******************************************************************************/
|
||||
private static void sendgrid(JSONObject data, Instant received, List<Event> events)
|
||||
{
|
||||
String type = switch(data.getString("event"))
|
||||
{
|
||||
case "delivered" -> "DELIVERED";
|
||||
case "bounce" -> "BOUNCED";
|
||||
case "dropped" -> "REJECTED";
|
||||
case "deferred" -> "DEFERRED";
|
||||
case "spamreport" -> "COMPLAINED";
|
||||
case "open" -> "OPENED";
|
||||
case "click" -> "CLICKED";
|
||||
case "unsubscribe", "group_unsubscribe" -> "UNSUBSCRIBED";
|
||||
default -> null;
|
||||
};
|
||||
if(type == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
String bounce = "BOUNCED".equals(type) ? ("blocked".equals(data.optString("type")) ? "SOFT" : "HARD") : null;
|
||||
String providerMessage = data.optString("sg_message_id", null);
|
||||
if(providerMessage != null && providerMessage.contains("."))
|
||||
{
|
||||
providerMessage = providerMessage.substring(0, providerMessage.indexOf('.'));
|
||||
}
|
||||
events.add(new Event(data.optString("sg_event_id", fingerprint(data)),
|
||||
data.optString("qqq_email_id", object(data, "custom_args").optString("qqq_email_id", null)), providerMessage, type, bounce,
|
||||
data.optString("email", null), time(data.opt("timestamp"), received), detail(data.optString("reason", data.optString("response", null))),
|
||||
"CLICKED".equals(type) ? data.optString("url", null) : null));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Mailgun MIME ids are normalized identically to the outbound transport.
|
||||
*******************************************************************************/
|
||||
private static void mailgun(JSONObject data, Instant received, List<Event> events)
|
||||
{
|
||||
String type = switch(data.getString("event"))
|
||||
{
|
||||
case "delivered" -> "DELIVERED";
|
||||
case "failed" -> "BOUNCED";
|
||||
case "complained" -> "COMPLAINED";
|
||||
case "opened" -> "OPENED";
|
||||
case "clicked" -> "CLICKED";
|
||||
case "unsubscribed" -> "UNSUBSCRIBED";
|
||||
default -> null;
|
||||
};
|
||||
if(type == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
String bounce = "BOUNCED".equals(type) ? switch(data.optString("severity"))
|
||||
{
|
||||
case "permanent" -> "HARD";
|
||||
case "temporary" -> "SOFT";
|
||||
default -> "UNDETERMINED";
|
||||
} : null;
|
||||
JSONObject variables = object(data, "user-variables");
|
||||
String providerMessage = object(object(data, "message"), "headers").optString("message-id", null);
|
||||
if(providerMessage != null)
|
||||
{
|
||||
providerMessage = providerMessage.replaceAll("^<|>$", "");
|
||||
}
|
||||
events.add(new Event(data.optString("id", fingerprint(data)), variables.optString("qqq_email_id", variables.optString("v:qqq_email_id", null)),
|
||||
providerMessage, type, bounce, data.optString("recipient", null), time(data.opt("timestamp"), received),
|
||||
detail(data.optString("reason", object(data, "delivery-status").optString("message", null))), "CLICKED".equals(type) ? data.optString("url", null) : null));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Optional nested objects contain no executable data.
|
||||
*******************************************************************************/
|
||||
private static JSONObject object(JSONObject parent, String key)
|
||||
{
|
||||
JSONObject value = parent.optJSONObject(key);
|
||||
return (value == null ? new JSONObject() : value);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fall back to receipt time only when the provider supplied no timestamp.
|
||||
*******************************************************************************/
|
||||
private static Instant time(Object value, Instant fallback)
|
||||
{
|
||||
if(value == null || JSONObject.NULL.equals(value))
|
||||
{
|
||||
return (fallback);
|
||||
}
|
||||
if(value instanceof Number number)
|
||||
{
|
||||
return (Instant.ofEpochMilli((long) (number.doubleValue() * 1000)));
|
||||
}
|
||||
return (OffsetDateTime.parse(value.toString()).toInstant());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fallback identity for providers without a distinct event identifier.
|
||||
*******************************************************************************/
|
||||
private static String fingerprint(JSONObject data)
|
||||
{
|
||||
return (WebhookVerification.hash(data.toString().getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Mask complete addresses before truncation, including quoted local parts.
|
||||
*******************************************************************************/
|
||||
public static String detail(String reason)
|
||||
{
|
||||
if(reason == null)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
String masked = reason.replaceAll("(?i)(?:\"[^\"]*\"|[^\\s<>(),;:]+)@[^\\s<>(),;:]+", "[address]");
|
||||
return (masked.substring(0, Math.min(500, masked.length())));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.delivery.EmailAlerts;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import jakarta.mail.internet.InternetAddress;
|
||||
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Bounded, replica-safe database transactions apply inbox rows exactly once.
|
||||
** No network or external effects occur while the row's provider is locked.
|
||||
*******************************************************************************/
|
||||
public final class EmailInboxProcessor
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(EmailInboxProcessor.class);
|
||||
|
||||
private static final int MAX_ATTEMPTS = 6;
|
||||
|
||||
/*******************************************************************************
|
||||
** Poll due rows; transaction rollback releases claims after a worker crash.
|
||||
*******************************************************************************/
|
||||
public int runOnce() throws QException
|
||||
{
|
||||
int processed = 0;
|
||||
for(QRecord row : SystemWrites.query(EmailTables.INBOX, EmailStore.equal("status", "PENDING")
|
||||
.withCriteria("nextAttemptDate", QCriteriaOperator.LESS_THAN_OR_EQUALS, EmailRuntime.clock().instant())
|
||||
.withOrderBy(new QFilterOrderBy("id")).withLimit(EmailRuntime.config().getBatchSize())))
|
||||
{
|
||||
if(process(row))
|
||||
{
|
||||
processed++;
|
||||
}
|
||||
}
|
||||
return (processed);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Re-read eligibility after the lock, including the observed attempt number.
|
||||
*******************************************************************************/
|
||||
private static boolean due(QRecord current, QRecord observed)
|
||||
{
|
||||
return (current != null && "PENDING".equals(current.getValueString("status"))
|
||||
&& Objects.equals(current.getValue("attempts"), observed.getValue("attempts"))
|
||||
&& current.getValueInstant("nextAttemptDate") != null && !current.getValueInstant("nextAttemptDate").isAfter(EmailRuntime.clock().instant()));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** All event, message, suppression and receipt changes commit together.
|
||||
*******************************************************************************/
|
||||
private boolean process(QRecord observed) throws QException
|
||||
{
|
||||
boolean complaint = false;
|
||||
try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER)))
|
||||
{
|
||||
QRecord provider = EmailTrackingStore.lock(observed.getValueInteger("providerId"), tx);
|
||||
QRecord row = EmailStore.get(EmailTables.INBOX, observed.getValueInteger("id"), tx);
|
||||
if(!due(row, observed))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
int applied = 0;
|
||||
int unmatched = 0;
|
||||
int skipped = 0;
|
||||
for(EmailEventParser.Event event : new EmailEventParser().parse(row.getValueString("kind"), row.getValueString("payloadJson"), row.getValueInstant("receivedDate")))
|
||||
{
|
||||
if(("OPENED".equals(event.type()) && !Boolean.TRUE.equals(provider.getValueBoolean("trackOpens")))
|
||||
|| ("CLICKED".equals(event.type()) && !Boolean.TRUE.equals(provider.getValueBoolean("trackClicks"))))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
QRecord message = match(event, provider.getValueInteger("id"), tx);
|
||||
if(message == null || "SENDING".equals(message.getValueString("status")))
|
||||
{
|
||||
unmatched++;
|
||||
continue;
|
||||
}
|
||||
String recipient = normalize(event.recipient());
|
||||
if(recipient == null || !recipients(message).contains(recipient))
|
||||
{
|
||||
LOG.info("Ignoring an email event whose recipient is not on the message", logPair("messageId", message.getValue("id")));
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
if(Boolean.TRUE.equals(message.getValueBoolean("phiApproved")) && List.of("OPENED", "CLICKED").contains(event.type()))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
String key = provider.getValueInteger("id") + ":" + WebhookVerification.hash((event.externalId() + "\n" + event.type() + "\n"
|
||||
+ recipient).getBytes(StandardCharsets.UTF_8));
|
||||
QRecord stored = EmailTrackingStore.event(message, event, key, tx);
|
||||
if(stored == null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
applied++;
|
||||
updateMessage(message, stored, tx);
|
||||
if("COMPLAINED".equals(event.type()) || ("BOUNCED".equals(event.type()) && "HARD".equals(event.bounceType())))
|
||||
{
|
||||
EmailTrackingStore.suppress(message, stored, "ALL", "COMPLAINED".equals(event.type()) ? "COMPLAINT" : "HARD_BOUNCE", tx);
|
||||
complaint |= "COMPLAINED".equals(event.type());
|
||||
}
|
||||
else if("UNSUBSCRIBED".equals(event.type()))
|
||||
{
|
||||
EmailTrackingStore.suppress(message, stored, message.getValueString("category"), "UNSUBSCRIBE", tx);
|
||||
}
|
||||
}
|
||||
|
||||
int attempts = row.getValueInteger("attempts") + 1;
|
||||
boolean retry = unmatched > 0 && attempts < MAX_ATTEMPTS;
|
||||
List<String> notes = new ArrayList<>();
|
||||
if(unmatched > 0)
|
||||
{
|
||||
notes.add(unmatched + (unmatched == 1 ? " event did not match a message" : " events did not match a message"));
|
||||
}
|
||||
if(skipped > 0)
|
||||
{
|
||||
notes.add(skipped + (skipped == 1 ? " event was skipped (recipient not on the message)" : " events were skipped (recipient not on the message)"));
|
||||
}
|
||||
EmailStore.update(EmailTables.INBOX, new QRecord().withValue("id", row.getValue("id"))
|
||||
.withValue("status", retry ? "PENDING" : (applied == 0 && unmatched == 0 ? "IGNORED" : "PROCESSED")).withValue("attempts", attempts)
|
||||
.withValue("processedDate", retry ? null : EmailRuntime.clock().instant())
|
||||
.withValue("nextAttemptDate", retry ? EmailRuntime.clock().instant().plusSeconds(60L * EmailRuntime.config().retryMinutesAfterAttempt(attempts)) : null)
|
||||
.withValue("lastError", notes.isEmpty() ? null : String.join("; ", notes)), tx);
|
||||
tx.commit();
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
fail(observed);
|
||||
return (false);
|
||||
}
|
||||
if(complaint)
|
||||
{
|
||||
complaintAlert();
|
||||
}
|
||||
return (true);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Canonical mailbox, or null when the provider supplied none that is valid.
|
||||
*******************************************************************************/
|
||||
private static String normalize(String value)
|
||||
{
|
||||
if(value == null || value.isBlank())
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
try
|
||||
{
|
||||
return (EmailTrackingStore.address(value));
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The canonical to, cc and bcc mailboxes of a message; only these may be
|
||||
** suppressed by an event matched to it.
|
||||
*******************************************************************************/
|
||||
private static Set<String> recipients(QRecord message)
|
||||
{
|
||||
Set<String> set = new HashSet<>();
|
||||
for(String field : List.of("toAddresses", "ccAddresses", "bccAddresses"))
|
||||
{
|
||||
String value = message.getValueString(field);
|
||||
if(value == null || value.isBlank())
|
||||
{
|
||||
continue;
|
||||
}
|
||||
try
|
||||
{
|
||||
for(InternetAddress address : InternetAddress.parse(value, false))
|
||||
{
|
||||
set.add(address.getAddress().toLowerCase(Locale.ROOT));
|
||||
}
|
||||
}
|
||||
catch(jakarta.mail.internet.AddressException e)
|
||||
{
|
||||
// A malformed stored envelope contributes no recipients.
|
||||
}
|
||||
}
|
||||
return (set);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Match our id first, then the provider id; neither may cross providers.
|
||||
*******************************************************************************/
|
||||
private static QRecord match(EmailEventParser.Event event, Integer providerId, QBackendTransaction tx) throws QException
|
||||
{
|
||||
if(event.messageId() != null && event.messageId().matches("[0-9]{1,10}") && Long.parseLong(event.messageId()) <= Integer.MAX_VALUE)
|
||||
{
|
||||
QRecord message = EmailStore.get(EmailTables.MESSAGE, Integer.valueOf(event.messageId()), tx);
|
||||
if(message != null && providerId.equals(message.getValueInteger("providerId")))
|
||||
{
|
||||
return (message);
|
||||
}
|
||||
}
|
||||
if(event.providerMessageId() == null || event.providerMessageId().isBlank())
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
List<QRecord> rows = SystemWrites.query(EmailTables.MESSAGE, EmailStore.equal("providerId", providerId)
|
||||
.withCriteria("providerMessageId", QCriteriaOperator.EQUALS, event.providerMessageId()), tx);
|
||||
return (rows.size() == 1 ? rows.getFirst() : null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Complaint outranks hard bounce, which outranks delivery; never regress.
|
||||
*******************************************************************************/
|
||||
private static void updateMessage(QRecord message, QRecord event, QBackendTransaction tx) throws QException
|
||||
{
|
||||
String type = event.getValueString("eventType");
|
||||
String status = message.getValueString("status");
|
||||
String field = switch(type)
|
||||
{
|
||||
case "DELIVERED" -> "SENT".equals(status) ? "deliveredDate" : null;
|
||||
case "BOUNCED" -> "HARD".equals(event.getValueString("bounceType")) && List.of("SENT", "DELIVERED").contains(status) ? "bouncedDate" : null;
|
||||
case "COMPLAINED" -> List.of("SENT", "DELIVERED", "BOUNCED").contains(status) ? "complainedDate" : null;
|
||||
default -> null;
|
||||
};
|
||||
if(field != null)
|
||||
{
|
||||
EmailStore.update(EmailTables.MESSAGE, new QRecord().withValue("id", message.getValue("id")).withValue("status", type)
|
||||
.withValue(field, event.getValue("occurredDate")), tx);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** After rollback, only the still-due observed attempt may consume retry budget.
|
||||
*******************************************************************************/
|
||||
private static void fail(QRecord observed) throws QException
|
||||
{
|
||||
try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER)))
|
||||
{
|
||||
EmailTrackingStore.lock(observed.getValueInteger("providerId"), tx);
|
||||
QRecord row = EmailStore.get(EmailTables.INBOX, observed.getValueInteger("id"), tx);
|
||||
if(due(row, observed))
|
||||
{
|
||||
int attempts = row.getValueInteger("attempts") + 1;
|
||||
EmailStore.update(EmailTables.INBOX, new QRecord().withValue("id", row.getValue("id")).withValue("attempts", attempts)
|
||||
.withValue("status", attempts >= MAX_ATTEMPTS ? "FAILED" : "PENDING").withValue("lastError", "Email event processing failed; check message matching and payload validity.")
|
||||
.withValue("nextAttemptDate", attempts >= MAX_ATTEMPTS ? null : EmailRuntime.clock().instant().plusSeconds(60L * EmailRuntime.config().retryMinutesAfterAttempt(attempts))), tx);
|
||||
tx.commit();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Standard Reprocess action rechecks FAILED under the same transaction lock.
|
||||
*******************************************************************************/
|
||||
public static boolean reprocess(Integer id) throws QException
|
||||
{
|
||||
QRecord selected = EmailStore.get(EmailTables.INBOX, id, null);
|
||||
if(selected == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER)))
|
||||
{
|
||||
EmailTrackingStore.lock(selected.getValueInteger("providerId"), tx);
|
||||
QRecord row = EmailStore.get(EmailTables.INBOX, id, tx);
|
||||
if(!"FAILED".equals(row.getValueString("status")))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
EmailStore.update(EmailTables.INBOX, new QRecord().withValue("id", id).withValue("status", "PENDING").withValue("attempts", 0)
|
||||
.withValue("nextAttemptDate", EmailRuntime.clock().instant()).withValue("lastError", null).withValue("processedDate", null), tx);
|
||||
tx.commit();
|
||||
return (true);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Optional alerting cannot turn a committed event into a failed inbox row.
|
||||
*******************************************************************************/
|
||||
private static void complaintAlert()
|
||||
{
|
||||
try
|
||||
{
|
||||
Class.forName("com.kingsrook.qqq.backend.core.alerts.QAlerts");
|
||||
if(SystemWrites.count(EmailTables.EVENT, EmailStore.equal("eventType", "COMPLAINED")
|
||||
.withCriteria("occurredDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, EmailRuntime.clock().instant().minusSeconds(86400))
|
||||
.withCriteria("occurredDate", QCriteriaOperator.LESS_THAN_OR_EQUALS, EmailRuntime.clock().instant())) > 3)
|
||||
{
|
||||
EmailAlerts.raise(EmailAlerts.COMPLAINT, 0);
|
||||
}
|
||||
}
|
||||
catch(Exception | LinkageError e)
|
||||
{
|
||||
// Alert integration is optional; event data is already committed.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import jakarta.mail.internet.InternetAddress;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** DB-only receipt, event deduplication and suppression writes.
|
||||
*******************************************************************************/
|
||||
public final class EmailTrackingStore
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Distinguish an authenticated retry from reuse of a Mailgun nonce.
|
||||
*******************************************************************************/
|
||||
public enum Receipt
|
||||
{
|
||||
RECEIVED, DUPLICATE, REPLAY
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private EmailTrackingStore()
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Serialize address upserts across providers, then acquire the dispatcher lock.
|
||||
** Every tracking writer uses this order and does no network I/O while locked.
|
||||
*******************************************************************************/
|
||||
public static QRecord lock(Integer providerId, QBackendTransaction tx) throws QException
|
||||
{
|
||||
List<QRecord> providers = SystemWrites.query(EmailTables.PROVIDER, new QQueryFilter().withOrderBy(new QFilterOrderBy("id")).withLimit(1), tx);
|
||||
if(providers.isEmpty())
|
||||
{
|
||||
throw (new QException("Email provider unavailable"));
|
||||
}
|
||||
EmailStore.lockProvider(providers.getFirst().getValueInteger("id"), tx);
|
||||
QRecord provider = EmailStore.lockProvider(providerId, tx);
|
||||
if(provider == null)
|
||||
{
|
||||
throw (new QException("Email provider unavailable"));
|
||||
}
|
||||
return (provider);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A provider lock avoids expected duplicate SQL errors (and payload logging).
|
||||
** Unique constraints remain the final database guard.
|
||||
*******************************************************************************/
|
||||
public static Receipt receive(QRecord provider, String externalId, String nonce, byte[] bytes, String note) throws QException
|
||||
{
|
||||
Integer providerId = provider.getValueInteger("id");
|
||||
try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER)))
|
||||
{
|
||||
EmailStore.lockProvider(providerId, tx);
|
||||
QQueryFilter filter = EmailStore.equal("providerId", providerId);
|
||||
if(nonce != null && SystemWrites.count(EmailTables.INBOX, filter.clone().withCriteria("nonce", QCriteriaOperator.EQUALS, nonce), tx) > 0)
|
||||
{
|
||||
return (Receipt.REPLAY);
|
||||
}
|
||||
if(SystemWrites.count(EmailTables.INBOX, filter.withCriteria("externalEventId", QCriteriaOperator.EQUALS, externalId), tx) > 0)
|
||||
{
|
||||
return (Receipt.DUPLICATE);
|
||||
}
|
||||
var now = EmailRuntime.clock().instant();
|
||||
QRecord row = new QRecord().withValue("providerId", providerId).withValue("kind", provider.getValue("kind"))
|
||||
.withValue("externalEventId", externalId).withValue("nonce", nonce).withValue("status", note == null ? "PENDING" : "IGNORED")
|
||||
.withValue("attempts", 0).withValue("nextAttemptDate", note == null ? now : null).withValue("lastError", note)
|
||||
.withValue("payloadJson", new String(bytes, StandardCharsets.UTF_8)).withValue("payloadSha256", WebhookVerification.hash(bytes))
|
||||
.withValue("receivedDate", now).withValue("processedDate", note == null ? null : now);
|
||||
copySecurity(provider, row);
|
||||
insert(EmailTables.INBOX, row, tx);
|
||||
tx.commit();
|
||||
return (Receipt.RECEIVED);
|
||||
}
|
||||
catch(QException e)
|
||||
{
|
||||
// Re-query after rollback: a unique violation aborts a PostgreSQL transaction.
|
||||
if(nonce != null && SystemWrites.count(EmailTables.INBOX, EmailStore.equal("providerId", providerId)
|
||||
.withCriteria("nonce", QCriteriaOperator.EQUALS, nonce)) > 0)
|
||||
{
|
||||
return (Receipt.REPLAY);
|
||||
}
|
||||
if(SystemWrites.count(EmailTables.INBOX, EmailStore.equal("providerId", providerId)
|
||||
.withCriteria("externalEventId", QCriteriaOperator.EQUALS, externalId)) > 0)
|
||||
{
|
||||
return (Receipt.DUPLICATE);
|
||||
}
|
||||
throw (new QException("Email inbox write failed"));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fail atomically without exposing input values or database diagnostics.
|
||||
*******************************************************************************/
|
||||
public static QRecord insert(String table, QRecord row, QBackendTransaction tx) throws QException
|
||||
{
|
||||
List<QRecord> records = SystemWrites.insert(table, List.of(row), tx).getRecords();
|
||||
if(records.isEmpty() || records.getFirst().getValueInteger("id") == null || (records.getFirst().getErrors() != null && !records.getFirst().getErrors().isEmpty()))
|
||||
{
|
||||
throw (new QException("Email tracking write failed"));
|
||||
}
|
||||
return (records.getFirst());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Preserve host tenant security on child and suppression records.
|
||||
*******************************************************************************/
|
||||
public static void copySecurity(QRecord source, QRecord target)
|
||||
{
|
||||
for(var field : EmailRuntime.config().getSecurityFields())
|
||||
{
|
||||
target.withValue(field.getName(), source.getValue(field.getName()));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Insert a recipient event once even across overlapping provider batches.
|
||||
*******************************************************************************/
|
||||
public static QRecord event(QRecord message, EmailEventParser.Event event, String key, QBackendTransaction tx) throws QException
|
||||
{
|
||||
if(SystemWrites.count(EmailTables.EVENT, EmailStore.equal("eventKey", key), tx) > 0)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
QRecord row = new QRecord().withValue("messageId", message.getValue("id")).withValue("providerId", message.getValue("providerId"))
|
||||
.withValue("eventType", event.type()).withValue("bounceType", event.bounceType()).withValue("recipient", address(event.recipient()))
|
||||
.withValue("occurredDate", event.occurred()).withValue("detail", event.detail()).withValue("url", "CLICKED".equals(event.type()) ? event.url() : null)
|
||||
.withValue("eventKey", key);
|
||||
copySecurity(message, row);
|
||||
return (insert(EmailTables.EVENT, row, tx));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Upsert without weakening a complaint or shortening an existing suppression.
|
||||
** All callers hold the common tracking lock throughout their transaction.
|
||||
*******************************************************************************/
|
||||
public static void suppress(QRecord message, QRecord event, String scope, String reason, QBackendTransaction tx) throws QException
|
||||
{
|
||||
if("UNSUBSCRIBE".equals(reason) && !List.of("MARKETING", "NOTIFICATION").contains(scope == null ? "" : scope))
|
||||
{
|
||||
return;
|
||||
}
|
||||
String address = event.getValueString("recipient");
|
||||
List<QRecord> existing = SystemWrites.query(EmailTables.SUPPRESSION, EmailStore.equal("address", address)
|
||||
.withCriteria("scope", QCriteriaOperator.EQUALS, scope), tx);
|
||||
QRecord row = new QRecord().withValue("address", address).withValue("scope", scope).withValue("reason", reason)
|
||||
.withValue("sourceMessageId", message.getValue("id")).withValue("sourceEventId", event.getValue("id")).withValue("expiresDate", null);
|
||||
copySecurity(message, row);
|
||||
if(existing.isEmpty())
|
||||
{
|
||||
row.withValue("createdDate", EmailRuntime.clock().instant());
|
||||
insert(EmailTables.SUPPRESSION, row, tx);
|
||||
}
|
||||
else
|
||||
{
|
||||
QRecord old = existing.getFirst();
|
||||
if("COMPLAINT".equals(old.getValueString("reason")))
|
||||
{
|
||||
row.withValue("reason", "COMPLAINT");
|
||||
}
|
||||
row.withValue("id", old.getValue("id"));
|
||||
EmailStore.update(EmailTables.SUPPRESSION, row, tx);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Canonical mailbox for event deduplication and suppression lookup.
|
||||
*******************************************************************************/
|
||||
public static String address(String value)
|
||||
{
|
||||
try
|
||||
{
|
||||
InternetAddress address = new InternetAddress(value, true);
|
||||
address.validate();
|
||||
return (address.getAddress().toLowerCase(Locale.ROOT));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalArgumentException("Email event recipient is invalid"));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.render.EmailUnsubscribeLinks;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.template.EmailTemplateEngine;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.json.JSONObject;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stateless token validation and atomic, idempotent category opt-out.
|
||||
*******************************************************************************/
|
||||
public final class EmailUnsubscribe
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Resolve the signed message and its single recipient, without writes.
|
||||
*******************************************************************************/
|
||||
public QRecord resolve(String token) throws QException
|
||||
{
|
||||
return (resolve(token, null));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Recheck the binding against the same transaction used for suppression.
|
||||
*******************************************************************************/
|
||||
private QRecord resolve(String token, QBackendTransaction transaction) throws QException
|
||||
{
|
||||
JSONObject payload = EmailUnsubscribeLinks.verify(token);
|
||||
if(payload == null)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
try
|
||||
{
|
||||
QRecord message = EmailStore.get(EmailTables.MESSAGE, Integer.valueOf(payload.getString("messageId")), transaction);
|
||||
if(message == null || !payload.getString("category").equals(message.getValueString("category")))
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
String recipient = EmailUnsubscribeLinks.recipient(message.getValueString("toAddresses"), message.getValueString("ccAddresses"), message.getValueString("bccAddresses"));
|
||||
String hash = payload.getString("addressHash");
|
||||
// Safe legacy links remain valid only after the single-mailbox check above.
|
||||
String legacyEnvelope = message.getValueString("toAddresses") + "," + message.getValueString("ccAddresses") + "," + message.getValueString("bccAddresses");
|
||||
boolean matches = WebhookVerification.equal(hash, WebhookVerification.hash(recipient.getBytes(StandardCharsets.UTF_8)))
|
||||
|| WebhookVerification.equal(hash, WebhookVerification.hash(legacyEnvelope.getBytes(StandardCharsets.UTF_8)));
|
||||
return (matches ? message : null);
|
||||
}
|
||||
catch(IllegalArgumentException | org.json.JSONException e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Opt out only the signed mailbox; reject changed or multi-recipient envelopes.
|
||||
** Never display those addresses on the public page or emit them to logs.
|
||||
*******************************************************************************/
|
||||
public void unsubscribe(String token) throws QException
|
||||
{
|
||||
QRecord initial = resolve(token);
|
||||
if(initial == null)
|
||||
{
|
||||
throw (new IllegalArgumentException("Invalid unsubscribe link"));
|
||||
}
|
||||
try(QBackendTransaction tx = QBackendTransaction.openFor(new UpdateInput(EmailTables.PROVIDER)))
|
||||
{
|
||||
EmailTrackingStore.lock(initial.getValueInteger("providerId"), tx);
|
||||
QRecord message = resolve(token, tx);
|
||||
if(message == null || !initial.getValueInteger("providerId").equals(message.getValueInteger("providerId")))
|
||||
{
|
||||
throw (new IllegalArgumentException("Invalid unsubscribe link"));
|
||||
}
|
||||
String recipient = EmailUnsubscribeLinks.recipient(message.getValueString("toAddresses"), message.getValueString("ccAddresses"), message.getValueString("bccAddresses"));
|
||||
String key = "unsubscribe:" + message.getValueInteger("id") + ":" + WebhookVerification.hash(recipient.getBytes(StandardCharsets.UTF_8));
|
||||
var event = new EmailEventParser.Event(key, message.getValueString("id"), null, "UNSUBSCRIBED", null, recipient, EmailRuntime.clock().instant(), null, null);
|
||||
QRecord stored = EmailTrackingStore.event(message, event, key, tx);
|
||||
if(stored == null)
|
||||
{
|
||||
stored = com.kingsrook.qbits.email.support.SystemWrites.query(EmailTables.EVENT, EmailStore.equal("eventKey", key), tx).getFirst();
|
||||
}
|
||||
EmailTrackingStore.suppress(message, stored, message.getValueString("category"), "UNSUBSCRIBE", tx);
|
||||
tx.commit();
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Self-contained accessible page; an empty form action posts to this same URL.
|
||||
*******************************************************************************/
|
||||
public String page(QRecord message, boolean confirmed) throws QException
|
||||
{
|
||||
QRecord brand = message.getValueInteger("brandId") == null ? null : EmailStore.get(EmailTables.BRAND, message.getValueInteger("brandId"), null);
|
||||
String label = brand == null || brand.getValueString("label") == null ? EmailRuntime.config().getAppLabel() : brand.getValueString("label");
|
||||
String category = "MARKETING".equals(message.getValueString("category")) ? "Marketing" : "Notification";
|
||||
String question = confirmed ? "You are unsubscribed from " + category + " email from " + label + "."
|
||||
: "Unsubscribe from " + category + " email from " + label + "?";
|
||||
return ("<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">"
|
||||
+ "<meta name=\"color-scheme\" content=\"light dark\"><title>Email preferences</title><style>"
|
||||
+ ":root{color-scheme:light dark}*{box-sizing:border-box}body{margin:0;padding:20px;font:18px/1.5 system-ui;background:#fff;color:#172033}"
|
||||
+ "main{max-width:560px;margin:40px auto;overflow-wrap:anywhere}h1{font-size:1.5rem}button{max-width:100%;padding:12px 20px;font:inherit;background:#174ea6;color:#fff;border:0;border-radius:6px;cursor:pointer}"
|
||||
+ "@media(prefers-color-scheme:dark){body{background:#111827;color:#f9fafb}button{background:#a9c7ff;color:#111827}}</style></head><body><main><p>"
|
||||
+ EmailTemplateEngine.escape(label) + "</p><h1>" + EmailTemplateEngine.escape(question) + "</h1>"
|
||||
+ (confirmed ? "" : "<form method=\"post\"><input type=\"hidden\" name=\"confirm\" value=\"unsubscribe\"><button type=\"submit\">Unsubscribe</button></form>")
|
||||
+ "</main></body></html>");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Instant;
|
||||
import java.util.HexFormat;
|
||||
import org.json.JSONObject;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Authenticate Mailgun; the durable inbox separately arbitrates nonce replay.
|
||||
*******************************************************************************/
|
||||
public final class MailgunVerifier
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Verify timestamp, token and hexadecimal HMAC without retaining the key.
|
||||
*******************************************************************************/
|
||||
public boolean verify(JSONObject payload, String secret, Instant now)
|
||||
{
|
||||
if(secret == null || secret.isBlank())
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
try
|
||||
{
|
||||
JSONObject signed = payload.getJSONObject("signature");
|
||||
String timestamp = signed.get("timestamp").toString();
|
||||
String token = signed.getString("token");
|
||||
if(token.isBlank() || token.length() > 500 || !WebhookVerification.fresh(timestamp, now))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
String expected = HexFormat.of().formatHex(mac.doFinal((timestamp + token).getBytes(StandardCharsets.UTF_8)));
|
||||
return (WebhookVerification.equal(expected, signed.getString("signature")));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.Base64;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Postmark authenticates with fixed username postmark and a resolved password.
|
||||
*******************************************************************************/
|
||||
public final class PostmarkVerifier
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Verify HTTP Basic credentials without logging or returning either value.
|
||||
*******************************************************************************/
|
||||
public boolean verify(String authorization, String secret)
|
||||
{
|
||||
if(secret == null || secret.isBlank() || authorization == null || !authorization.regionMatches(true, 0, "Basic ", 0, 6))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
try
|
||||
{
|
||||
String credentials = new String(Base64.getDecoder().decode(authorization.substring(6)), StandardCharsets.UTF_8);
|
||||
return (WebhookVerification.equal("postmark:" + secret, credentials));
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.KeyFactory;
|
||||
import java.security.Signature;
|
||||
import java.security.spec.X509EncodedKeySpec;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify SendGrid's timestamp and raw body with its DER encoded EC public key.
|
||||
*******************************************************************************/
|
||||
public final class SendGridVerifier
|
||||
{
|
||||
/*******************************************************************************
|
||||
** The timestamp bytes precede the unchanged request body.
|
||||
*******************************************************************************/
|
||||
public boolean verify(byte[] body, String signature, String timestamp, String secret, Instant now)
|
||||
{
|
||||
if(secret == null || secret.isBlank() || signature == null || !WebhookVerification.fresh(timestamp, now))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
try
|
||||
{
|
||||
var key = KeyFactory.getInstance("EC").generatePublic(new X509EncodedKeySpec(Base64.getDecoder().decode(secret)));
|
||||
Signature verifier = Signature.getInstance("SHA256withECDSA");
|
||||
verifier.initVerify(key);
|
||||
verifier.update(timestamp.getBytes(StandardCharsets.UTF_8));
|
||||
verifier.update(body);
|
||||
return (verifier.verify(Base64.getDecoder().decode(signature)));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.Signature;
|
||||
import java.security.cert.CertificateFactory;
|
||||
import java.security.cert.X509Certificate;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.Date;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import com.kingsrook.qbits.email.transport.EmailHttp;
|
||||
import org.json.JSONObject;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** SNS verification with HTTPS AWS-only certificate retrieval, memory caching (including
|
||||
** a five-minute cache of fetch failures) and a freshness window on the signed Timestamp.
|
||||
*******************************************************************************/
|
||||
public final class SnsVerifier
|
||||
{
|
||||
private final EmailHttp http;
|
||||
private static final Duration MAX_AGE = Duration.ofHours(1);
|
||||
private static final Duration MAX_FUTURE = Duration.ofMinutes(5);
|
||||
private static final Duration FAILURE_TTL = Duration.ofMinutes(5);
|
||||
|
||||
private final Map<String, X509Certificate> certificates = new ConcurrentHashMap<>();
|
||||
private final Map<String, Instant> failures = new ConcurrentHashMap<>();
|
||||
|
||||
/*******************************************************************************
|
||||
** Default bounded HTTPS transport without redirects.
|
||||
*******************************************************************************/
|
||||
public SnsVerifier()
|
||||
{
|
||||
this(EmailHttp::sendJdk);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Inject HTTP for deterministic certificate and confirmation tests.
|
||||
*******************************************************************************/
|
||||
public SnsVerifier(EmailHttp http)
|
||||
{
|
||||
this.http = http;
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Strict URL boundary applied before every network operation.
|
||||
*******************************************************************************/
|
||||
public static boolean allowedUrl(String value, boolean certificate)
|
||||
{
|
||||
try
|
||||
{
|
||||
URI uri = URI.create(value);
|
||||
return ("https".equals(uri.getScheme()) && uri.getHost() != null
|
||||
&& uri.getHost().matches("^sns\\.[a-z0-9-]+\\.amazonaws\\.com(\\.cn)?$")
|
||||
&& uri.getUserInfo() == null && (uri.getPort() == -1 || uri.getPort() == 443) && uri.getFragment() == null
|
||||
&& (!certificate || (uri.getPath().endsWith(".pem") && uri.getQuery() == null)));
|
||||
}
|
||||
catch(RuntimeException e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** AWS's fixed order includes the final newline and optional Subject only.
|
||||
*******************************************************************************/
|
||||
public static String canonical(JSONObject payload)
|
||||
{
|
||||
String type = payload.getString("Type");
|
||||
List<String> fields;
|
||||
if("Notification".equals(type))
|
||||
{
|
||||
fields = payload.has("Subject") ? List.of("Message", "MessageId", "Subject", "Timestamp", "TopicArn", "Type")
|
||||
: List.of("Message", "MessageId", "Timestamp", "TopicArn", "Type");
|
||||
}
|
||||
else if("SubscriptionConfirmation".equals(type) || "UnsubscribeConfirmation".equals(type))
|
||||
{
|
||||
fields = List.of("Message", "MessageId", "SubscribeURL", "Timestamp", "Token", "TopicArn", "Type");
|
||||
}
|
||||
else
|
||||
{
|
||||
throw (new IllegalArgumentException("Unsupported SNS envelope"));
|
||||
}
|
||||
StringBuilder value = new StringBuilder();
|
||||
for(String field : fields)
|
||||
{
|
||||
value.append(field).append('\n').append(payload.getString(field)).append('\n');
|
||||
}
|
||||
return (value.toString());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Authenticate the topic and signature; no confirmation before success.
|
||||
*******************************************************************************/
|
||||
public boolean verify(JSONObject payload, String topic, Instant now) throws IOException, InterruptedException
|
||||
{
|
||||
try
|
||||
{
|
||||
String url = payload.getString("SigningCertURL");
|
||||
if(topic == null || topic.isBlank() || !topic.equals(payload.getString("TopicArn")) || !allowedUrl(url, true))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
String algorithm = switch(payload.getString("SignatureVersion"))
|
||||
{
|
||||
case "1" -> "SHA1withRSA";
|
||||
case "2" -> "SHA256withRSA";
|
||||
default -> null;
|
||||
};
|
||||
if(algorithm == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
Instant stamp = Instant.parse(payload.getString("Timestamp"));
|
||||
if(stamp.isBefore(now.minus(MAX_AGE)) || stamp.isAfter(now.plus(MAX_FUTURE)))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
String canonical = canonical(payload);
|
||||
X509Certificate certificate = certificates.get(url);
|
||||
if(certificate == null || certificate.getNotAfter().toInstant().isBefore(now))
|
||||
{
|
||||
Instant failed = failures.get(url);
|
||||
if(failed != null && failed.plus(FAILURE_TTL).isAfter(now))
|
||||
{
|
||||
throw (new IOException("SNS verification service unavailable"));
|
||||
}
|
||||
byte[] pem;
|
||||
try
|
||||
{
|
||||
pem = get(url).getBytes(StandardCharsets.UTF_8);
|
||||
}
|
||||
catch(IOException e)
|
||||
{
|
||||
if(failures.size() >= 64)
|
||||
{
|
||||
failures.clear();
|
||||
}
|
||||
failures.put(url, now);
|
||||
throw (e);
|
||||
}
|
||||
certificate = (X509Certificate) CertificateFactory.getInstance("X.509").generateCertificate(new ByteArrayInputStream(pem));
|
||||
certificate.checkValidity(Date.from(now));
|
||||
if(certificates.size() >= 64)
|
||||
{
|
||||
certificates.clear();
|
||||
}
|
||||
certificates.put(url, certificate);
|
||||
}
|
||||
certificate.checkValidity(Date.from(now));
|
||||
Signature verifier = Signature.getInstance(algorithm);
|
||||
verifier.initVerify(certificate.getPublicKey());
|
||||
verifier.update(canonical.getBytes(StandardCharsets.UTF_8));
|
||||
return (verifier.verify(Base64.getDecoder().decode(payload.getString("Signature"))));
|
||||
}
|
||||
catch(IOException | InterruptedException e)
|
||||
{
|
||||
throw (e);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Caller invokes this only after verifying the signed confirmation envelope.
|
||||
*******************************************************************************/
|
||||
public void confirm(JSONObject payload) throws IOException, InterruptedException
|
||||
{
|
||||
String url = payload.getString("SubscribeURL");
|
||||
if(!allowedUrl(url, false))
|
||||
{
|
||||
throw (new IllegalArgumentException("Invalid SNS confirmation URL"));
|
||||
}
|
||||
get(url);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fixed request timeout, no redirects, and no provider response diagnostics.
|
||||
*******************************************************************************/
|
||||
private String get(String url) throws IOException, InterruptedException
|
||||
{
|
||||
var response = http.send(HttpRequest.newBuilder(URI.create(url)).timeout(Duration.ofSeconds(30)).GET().build());
|
||||
if(response.statusCode() != 200 || response.body() == null || response.body().length() > 65536)
|
||||
{
|
||||
throw (new IOException("SNS verification service unavailable"));
|
||||
}
|
||||
return (response.body());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Instant;
|
||||
import java.util.HexFormat;
|
||||
import com.kingsrook.qbits.email.EmailQBitConfig;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Small shared verification primitives. Errors never retain request data.
|
||||
*******************************************************************************/
|
||||
public final class WebhookVerification
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private WebhookVerification()
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve only an allowlisted webhook secret; null means unavailable.
|
||||
*******************************************************************************/
|
||||
public static String secret(QRecord provider)
|
||||
{
|
||||
EmailQBitConfig config = EmailRuntime.config();
|
||||
String name = provider.getValueString("webhookSecretEnvName");
|
||||
try
|
||||
{
|
||||
if(name != null && EmailQBitConfig.ENV_NAME.matcher(name).matches() && config.effectiveAllowedEnvNames(provider.getValueString("kind")).contains(name))
|
||||
{
|
||||
String value = config.getSecretResolver().apply(name);
|
||||
return (value == null || value.isBlank() ? null : value);
|
||||
}
|
||||
}
|
||||
catch(RuntimeException e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
return (null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Compare bytes without data-dependent early exits.
|
||||
*******************************************************************************/
|
||||
public static boolean equal(String expected, String actual)
|
||||
{
|
||||
return (expected != null && actual != null && MessageDigest.isEqual(expected.getBytes(StandardCharsets.UTF_8), actual.getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Reject timestamps outside five minutes, including future timestamps.
|
||||
*******************************************************************************/
|
||||
public static boolean fresh(String timestamp, Instant now)
|
||||
{
|
||||
try
|
||||
{
|
||||
Instant time = Instant.ofEpochSecond(Long.parseLong(timestamp));
|
||||
return (!time.isBefore(now.minusSeconds(300)) && !time.isAfter(now.plusSeconds(300)));
|
||||
}
|
||||
catch(RuntimeException e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable payload and event fingerprints, never payload diagnostics.
|
||||
*******************************************************************************/
|
||||
public static String hash(byte[] bytes)
|
||||
{
|
||||
try
|
||||
{
|
||||
return (HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalStateException("SHA-256 unavailable"));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -31,6 +31,7 @@ import java.util.Map;
|
||||
public class EmailEnvelope
|
||||
{
|
||||
private String messageId;
|
||||
private String rfcMessageId;
|
||||
private String from;
|
||||
private List<String> to = List.of();
|
||||
private List<String> cc = List.of();
|
||||
@@ -62,6 +63,27 @@ public class EmailEnvelope
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for the persisted RFC 5322 Message-ID, distinct from the outbox row id.
|
||||
*******************************************************************************/
|
||||
public String getRfcMessageId()
|
||||
{
|
||||
return (rfcMessageId);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for the persisted RFC 5322 Message-ID.
|
||||
*******************************************************************************/
|
||||
public EmailEnvelope withRfcMessageId(String rfcMessageId)
|
||||
{
|
||||
this.rfcMessageId = rfcMessageId;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for from.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -30,10 +30,10 @@ import java.time.Duration;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Package-visible HTTP seam; no provider SDK, credential cache or filesystem.
|
||||
** Shared HTTP seam; no provider SDK, credential cache or filesystem.
|
||||
*******************************************************************************/
|
||||
@FunctionalInterface
|
||||
interface EmailHttp
|
||||
public interface EmailHttp
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Exchange one request without retries.
|
||||
|
||||
@@ -81,13 +81,17 @@ final class EmailHttpSupport
|
||||
Map<String, String> headers = new LinkedHashMap<>();
|
||||
envelope.getHeaders().forEach((name, value) ->
|
||||
{
|
||||
if(!name.equalsIgnoreCase("Bcc") && !name.equalsIgnoreCase("Auto-Submitted") && !name.equalsIgnoreCase("X-QQQ-Email-Id"))
|
||||
if(!name.equalsIgnoreCase("Bcc") && !name.equalsIgnoreCase("Auto-Submitted") && !name.equalsIgnoreCase("X-QQQ-Email-Id") && !name.equalsIgnoreCase("Message-ID"))
|
||||
{
|
||||
headers.put(name, value);
|
||||
}
|
||||
});
|
||||
headers.put("Auto-Submitted", "auto-generated");
|
||||
headers.put("X-QQQ-Email-Id", envelope.getMessageId());
|
||||
if(envelope.getRfcMessageId() != null)
|
||||
{
|
||||
headers.put("Message-ID", envelope.getRfcMessageId());
|
||||
}
|
||||
return (headers);
|
||||
}
|
||||
|
||||
|
||||
@@ -53,7 +53,26 @@ final class EmailMime
|
||||
noLineBreaks(envelope.getFrom());
|
||||
noLineBreaks(envelope.getReplyTo());
|
||||
noLineBreaks(envelope.getMessageId());
|
||||
MimeMessage message = new MimeMessage(session);
|
||||
noLineBreaks(envelope.getRfcMessageId());
|
||||
String rfcMessageId = envelope.getRfcMessageId();
|
||||
MimeMessage message = new MimeMessage(session)
|
||||
{
|
||||
/*******************************************************************************
|
||||
** JavaMail may save again when sending; keep the persisted identity each time.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
protected void updateMessageID() throws MessagingException
|
||||
{
|
||||
if(rfcMessageId == null)
|
||||
{
|
||||
super.updateMessageID();
|
||||
}
|
||||
else
|
||||
{
|
||||
setHeader("Message-ID", rfcMessageId);
|
||||
}
|
||||
}
|
||||
};
|
||||
message.setFrom(new InternetAddress(envelope.getFrom(), true));
|
||||
recipients(message, Message.RecipientType.TO, envelope.getTo());
|
||||
recipients(message, Message.RecipientType.CC, envelope.getCc());
|
||||
|
||||
@@ -96,6 +96,10 @@ public class MailgunEmailTransport implements EmailTransport
|
||||
field(body, boundary, "o:tracking-opens", settings.enabled("trackOpens") ? "yes" : "no");
|
||||
field(body, boundary, "o:tracking-clicks", settings.enabled("trackClicks") ? "yes" : "no");
|
||||
field(body, boundary, "v:qqq_email_id", envelope.getMessageId());
|
||||
if(envelope.getRfcMessageId() != null)
|
||||
{
|
||||
field(body, boundary, "h:Message-Id", envelope.getRfcMessageId());
|
||||
}
|
||||
body.writeBytes(("--" + boundary + "\r\nContent-Disposition: form-data; name=\"message\"; filename=\"message.eml\"\r\n"
|
||||
+ "Content-Type: message/rfc822\r\n\r\n").getBytes(StandardCharsets.UTF_8));
|
||||
EmailMime.build(Session.getInstance(new Properties()), envelope).writeTo(body, new String[] {"Bcc"});
|
||||
|
||||
@@ -50,6 +50,9 @@ import software.amazon.awssdk.services.sesv2.model.SesV2Exception;
|
||||
|
||||
/*******************************************************************************
|
||||
** SES v2 raw MIME transport; credentials are resolved only for an attempt.
|
||||
** The raw request carries the stored Message-ID, but SES replaces it downstream.
|
||||
** Receiver deduplication by that header is unavailable with SES; X-QQQ-Email-Id
|
||||
** remains the application correlation header. See docs/DURABLE-JOBS.md.
|
||||
*******************************************************************************/
|
||||
public class SesEmailTransport implements EmailTransport
|
||||
{
|
||||
|
||||
@@ -0,0 +1,292 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.webhook;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qbits.email.tracking.EmailTrackingStore;
|
||||
import com.kingsrook.qbits.email.tracking.EmailUnsubscribe;
|
||||
import com.kingsrook.qbits.email.tracking.MailgunVerifier;
|
||||
import com.kingsrook.qbits.email.tracking.PostmarkVerifier;
|
||||
import com.kingsrook.qbits.email.tracking.SendGridVerifier;
|
||||
import com.kingsrook.qbits.email.tracking.SnsVerifier;
|
||||
import com.kingsrook.qbits.email.tracking.WebhookVerification;
|
||||
import com.kingsrook.qqq.backend.core.context.CapturedContext;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
|
||||
import com.kingsrook.qqq.middleware.javalin.QJavalinMetaData;
|
||||
import com.kingsrook.qqq.middleware.javalin.QJavalinRouteProviderInterface;
|
||||
import io.javalin.config.JavalinConfig;
|
||||
import io.javalin.http.Context;
|
||||
import org.json.JSONArray;
|
||||
import org.json.JSONObject;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Authenticated provider ingress and signed public opt-out outside the UI API.
|
||||
** No bodies, tokens, addresses or exception diagnostics are logged here.
|
||||
*******************************************************************************/
|
||||
public class EmailWebhookRouteProvider implements QJavalinRouteProviderInterface
|
||||
{
|
||||
private static final int MAX_BODY = 1024 * 1024;
|
||||
private QInstance instance;
|
||||
private final SnsVerifier sns;
|
||||
|
||||
/*******************************************************************************
|
||||
** Middleware constructs the default route provider.
|
||||
*******************************************************************************/
|
||||
public EmailWebhookRouteProvider()
|
||||
{
|
||||
this(new SnsVerifier());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Inject the existing HTTP seam through the SNS verifier for local tests.
|
||||
*******************************************************************************/
|
||||
public EmailWebhookRouteProvider(SnsVerifier sns)
|
||||
{
|
||||
this.sns = sns;
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Idempotent registration even when a producer is run repeatedly.
|
||||
*******************************************************************************/
|
||||
public static void register(QInstance instance)
|
||||
{
|
||||
QJavalinMetaData metadata = QJavalinMetaData.ofOrWithNew(instance);
|
||||
if(CollectionUtils.nonNullList(metadata.getAdditionalRouteProviderReferences()).stream()
|
||||
.noneMatch(reference -> reference != null && EmailWebhookRouteProvider.class.getName().equals(reference.getName())))
|
||||
{
|
||||
metadata.withAdditionalRouteProviderReference(new QCodeReference(EmailWebhookRouteProvider.class));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Supplied by the host middleware.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void setQInstance(QInstance instance)
|
||||
{
|
||||
this.instance = instance;
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Specific unsubscribe routes precede the provider wildcard route.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void acceptJavalinConfig(JavalinConfig config)
|
||||
{
|
||||
String base = EmailRuntime.config(instance).getWebhookBasePath();
|
||||
config.routes.get(base + "/unsubscribe/{token}", this::unsubscribe);
|
||||
config.routes.post(base + "/unsubscribe/{token}", this::unsubscribe);
|
||||
config.routes.post(base + "/{kind}/{providerName}", this::receive);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Bound streaming input even when Content-Length is absent or dishonest.
|
||||
*******************************************************************************/
|
||||
private static byte[] body(Context context) throws java.io.IOException
|
||||
{
|
||||
if(context.req().getContentLengthLong() > MAX_BODY)
|
||||
{
|
||||
respond(context, 413, "error", "Payload too large");
|
||||
return (null);
|
||||
}
|
||||
byte[] bytes = context.req().getInputStream().readNBytes(MAX_BODY + 1);
|
||||
if(bytes.length > MAX_BODY)
|
||||
{
|
||||
respond(context, 413, "error", "Payload too large");
|
||||
return (null);
|
||||
}
|
||||
return (bytes);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Authenticate before storing. Missing configuration and DB outages are retryable.
|
||||
*******************************************************************************/
|
||||
private void receive(Context context)
|
||||
{
|
||||
CapturedContext previous = QContext.capture();
|
||||
try
|
||||
{
|
||||
QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession(instance));
|
||||
byte[] bytes = body(context);
|
||||
if(bytes == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
String kind = context.pathParam("kind").toUpperCase(Locale.ROOT);
|
||||
List<QRecord> providers = SystemWrites.query(EmailTables.PROVIDER, EmailStore.equal("name", context.pathParam("providerName")));
|
||||
if(providers.size() != 1 || !Boolean.TRUE.equals(providers.getFirst().getValueBoolean("enabled"))
|
||||
|| !kind.equals(providers.getFirst().getValueString("kind")) || !List.of("SES", "POSTMARK", "SENDGRID", "MAILGUN").contains(kind))
|
||||
{
|
||||
respond(context, 404, "error", "Provider not found");
|
||||
return;
|
||||
}
|
||||
QRecord provider = providers.getFirst();
|
||||
String secret = "SES".equals(kind) ? provider.getValueString("snsTopicArn") : WebhookVerification.secret(provider);
|
||||
if(secret == null || secret.isBlank())
|
||||
{
|
||||
respond(context, 503, "error", "Webhook verification unavailable");
|
||||
return;
|
||||
}
|
||||
String payload = new String(bytes, StandardCharsets.UTF_8);
|
||||
JSONObject json = "SENDGRID".equals(kind) ? new JSONObject() : new JSONObject(payload);
|
||||
boolean verified = switch(kind)
|
||||
{
|
||||
case "SES" -> sns.verify(json, secret, EmailRuntime.clock().instant());
|
||||
case "POSTMARK" -> new PostmarkVerifier().verify(context.header("Authorization"), secret);
|
||||
case "SENDGRID" -> new SendGridVerifier().verify(bytes, context.header("X-Twilio-Email-Event-Webhook-Signature"),
|
||||
context.header("X-Twilio-Email-Event-Webhook-Timestamp"), secret, EmailRuntime.clock().instant());
|
||||
case "MAILGUN" -> new MailgunVerifier().verify(json, secret, EmailRuntime.clock().instant());
|
||||
default -> false;
|
||||
};
|
||||
if(!verified)
|
||||
{
|
||||
respond(context, 401, "error", "Webhook verification failed");
|
||||
return;
|
||||
}
|
||||
String id = WebhookVerification.hash(bytes);
|
||||
String nonce = null;
|
||||
String note = null;
|
||||
if("SES".equals(kind))
|
||||
{
|
||||
id = json.getString("MessageId");
|
||||
if(!"Notification".equals(json.getString("Type")))
|
||||
{
|
||||
if(!SnsVerifier.allowedUrl(json.getString("SubscribeURL"), false))
|
||||
{
|
||||
respond(context, 401, "error", "Webhook verification failed");
|
||||
return;
|
||||
}
|
||||
if("SubscriptionConfirmation".equals(json.getString("Type")))
|
||||
{
|
||||
sns.confirm(json);
|
||||
}
|
||||
note = "SNS subscription control message verified";
|
||||
}
|
||||
}
|
||||
else if("MAILGUN".equals(kind))
|
||||
{
|
||||
id = json.getJSONObject("event-data").getString("id");
|
||||
nonce = json.getJSONObject("signature").getString("token");
|
||||
}
|
||||
else if("POSTMARK".equals(kind) && json.has("ID"))
|
||||
{
|
||||
id = json.getString("RecordType") + ":" + json.get("ID");
|
||||
}
|
||||
else if("SENDGRID".equals(kind))
|
||||
{
|
||||
new JSONArray(payload);
|
||||
}
|
||||
if(id.isBlank() || id.length() > 500)
|
||||
{
|
||||
respond(context, 401, "error", "Webhook verification failed");
|
||||
return;
|
||||
}
|
||||
EmailTrackingStore.Receipt outcome = EmailTrackingStore.receive(provider, id, nonce, bytes, note);
|
||||
respond(context, outcome == EmailTrackingStore.Receipt.REPLAY ? 401 : 200,
|
||||
outcome == EmailTrackingStore.Receipt.REPLAY ? "error" : outcome == EmailTrackingStore.Receipt.DUPLICATE ? "duplicate" : "received", true);
|
||||
}
|
||||
catch(org.json.JSONException e)
|
||||
{
|
||||
respond(context, 401, "error", "Webhook verification failed");
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
if(e instanceof InterruptedException)
|
||||
{
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
respond(context, 503, "error", "Email inbox unavailable");
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.init(previous);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** GET is read-only. Only an explicit one-click or page form POST opts out.
|
||||
*******************************************************************************/
|
||||
private void unsubscribe(Context context)
|
||||
{
|
||||
CapturedContext previous = QContext.capture();
|
||||
try
|
||||
{
|
||||
QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession(instance));
|
||||
context.header("Cache-Control", "no-store").header("Referrer-Policy", "no-referrer")
|
||||
.header("Content-Security-Policy", "default-src 'none'; style-src 'unsafe-inline'; form-action 'self'; frame-ancestors 'none'");
|
||||
byte[] bytes = body(context);
|
||||
if(bytes == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
String token = context.pathParam("token");
|
||||
EmailUnsubscribe service = new EmailUnsubscribe();
|
||||
QRecord message = service.resolve(token);
|
||||
if(message == null)
|
||||
{
|
||||
context.status(404).contentType("text/html; charset=utf-8").result("<!doctype html><html lang=\"en\"><meta charset=\"utf-8\"><title>Link not found</title><p>This link is unavailable.</p></html>");
|
||||
return;
|
||||
}
|
||||
boolean post = "POST".equals(context.req().getMethod());
|
||||
boolean oneClick = false;
|
||||
if(post)
|
||||
{
|
||||
String form = new String(bytes, StandardCharsets.UTF_8);
|
||||
oneClick = "List-Unsubscribe=One-Click".equals(form);
|
||||
if(!oneClick && !"confirm=unsubscribe".equals(form))
|
||||
{
|
||||
respond(context, 400, "error", "Unsubscribe confirmation required");
|
||||
return;
|
||||
}
|
||||
service.unsubscribe(token);
|
||||
}
|
||||
context.status(200).contentType(oneClick ? "text/plain" : "text/html; charset=utf-8").result(oneClick ? "" : service.page(message, post));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
respond(context, 503, "error", "Unsubscribe unavailable");
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.init(previous);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fixed JSON responses contain no request or credential data.
|
||||
*******************************************************************************/
|
||||
private static void respond(Context context, int status, String key, Object value)
|
||||
{
|
||||
context.status(status).contentType("application/json").result(new JSONObject().put(key, value).toString());
|
||||
}
|
||||
}
|
||||
@@ -21,18 +21,34 @@
|
||||
package com.kingsrook.qbits.email;
|
||||
|
||||
import java.net.ServerSocket;
|
||||
import java.time.*;
|
||||
import java.util.*;
|
||||
import java.util.concurrent.*;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.CyclicBarrier;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import com.icegreen.greenmail.util.GreenMail;
|
||||
import com.icegreen.greenmail.util.ServerSetup;
|
||||
import com.kingsrook.qbits.email.delivery.*;
|
||||
import com.kingsrook.qbits.email.delivery.EmailDispatcher;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.outbox.*;
|
||||
import com.kingsrook.qbits.email.outbox.EmailCategory;
|
||||
import com.kingsrook.qbits.email.outbox.EmailRequest;
|
||||
import com.kingsrook.qbits.email.outbox.EmailSendResult;
|
||||
import com.kingsrook.qbits.email.outbox.EmailService;
|
||||
import com.kingsrook.qbits.email.processes.EmailProcesses;
|
||||
import com.kingsrook.qbits.email.support.*;
|
||||
import com.kingsrook.qbits.email.transport.*;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qbits.email.transport.EmailTransport;
|
||||
import com.kingsrook.qbits.email.transport.EmailTransportException;
|
||||
import com.kingsrook.qbits.email.transport.EmailTransportRegistry;
|
||||
import com.kingsrook.qbits.email.transport.EmailTransportResult;
|
||||
import com.kingsrook.qbits.email.transport.SmtpEmailTransport;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.messaging.SendMessageAction;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
|
||||
@@ -41,16 +57,23 @@ import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.*;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.email.*;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.Content;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.MultiParty;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.Party;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.SendMessageInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailContentRole;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailPartyRole;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import jakarta.mail.Multipart;
|
||||
import org.junit.jupiter.api.*;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Email Delivery Test.
|
||||
@@ -398,6 +421,39 @@ class EmailDeliveryTest
|
||||
assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(1);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A lost acknowledgement retries the stored identity through the real SMTP sink.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void smtpRetryPreservesStoredMessageId() throws Exception
|
||||
{
|
||||
var queued = EmailService.send(request().withIdempotencyKey("job:1:row:mail"));
|
||||
String stored = message(queued.messageId()).getValueString("rfcMessageId");
|
||||
AtomicInteger calls = new AtomicInteger();
|
||||
var dispatcher = fake((envelope, settings) ->
|
||||
{
|
||||
assertThat(envelope.getRfcMessageId()).isEqualTo(stored);
|
||||
var sent = new SmtpEmailTransport().send(envelope, settings);
|
||||
if(calls.incrementAndGet() == 1)
|
||||
{
|
||||
throw new EmailTransportException("Lost acknowledgement", false);
|
||||
}
|
||||
return (sent);
|
||||
});
|
||||
assertThat(dispatcher.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT);
|
||||
EmailRuntime.setClock(Clock.fixed(now.plusSeconds(301), ZoneOffset.UTC));
|
||||
assertThat(dispatcher.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.SENT);
|
||||
assertThat(mail.waitForIncomingEmail(2000, 2)).isTrue();
|
||||
assertThat(mail.getReceivedMessages()).hasSize(2);
|
||||
for(var received : mail.getReceivedMessages())
|
||||
{
|
||||
assertThat(received.getMessageID()).isEqualTo(stored);
|
||||
assertThat(received.getHeader("X-QQQ-Email-Id")).containsExactly(queued.messageId().toString());
|
||||
}
|
||||
assertThat(message(queued.messageId()).getValueString("rfcMessageId")).isEqualTo(stored);
|
||||
assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(2);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Smtp Authentication And Secret Resolution Only At Send.
|
||||
*******************************************************************************/
|
||||
@@ -477,8 +533,8 @@ class EmailDeliveryTest
|
||||
var second = EmailService.send(request());
|
||||
fake((e, s) ->
|
||||
{
|
||||
throw new EmailTransportException("throttle", false).withRetryAfter(Duration.ofHours(2));
|
||||
})
|
||||
throw new EmailTransportException("throttle", false).withRetryAfter(Duration.ofHours(2));
|
||||
})
|
||||
.runOnce();
|
||||
assertThat(message(second.messageId()).getValueInstant("nextAttemptDate"))
|
||||
.isEqualTo(now.plusSeconds(3960));
|
||||
@@ -494,8 +550,8 @@ class EmailDeliveryTest
|
||||
assertThat(
|
||||
fake((e, s) ->
|
||||
{
|
||||
throw new EmailTransportException("rejected", true);
|
||||
})
|
||||
throw new EmailTransportException("rejected", true);
|
||||
})
|
||||
.sendNow(queued.messageId()))
|
||||
.isEqualTo(EmailDispatcher.SendResult.DEAD_LETTER);
|
||||
var future = EmailService.send(request());
|
||||
@@ -583,6 +639,48 @@ class EmailDeliveryTest
|
||||
.isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Opt-out messages are one mailbox per send, before suppression can trim them.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"NOTIFICATION", "MARKETING"})
|
||||
void unsubscribeRequiresOneToRecipient(String category) throws Exception
|
||||
{
|
||||
EmailRuntime.config().withPublicBaseUrl("https://app.example.com").withSecretResolver(name -> "test-unsubscribe-key");
|
||||
H2Support.insert(EmailTables.BRAND, new QRecord().withValue("name", "postal").withValue("isDefault", true).withValue("postalAddress", "123 Test Street"));
|
||||
for(EmailRequest invalid : List.of(request().withTo(List.of("person@example.com", "other@example.com")),
|
||||
request().withTo("person@example.com, other@example.com"), request().withCc("other@example.com"),
|
||||
request().withBcc("other@example.com"), request().withTo(List.of()).withCc("other@example.com"),
|
||||
request().withTo(List.of()).withBcc("other@example.com"), request().withTo(List.of()), request().withCc("person@example.com")))
|
||||
{
|
||||
var result = EmailService.send(invalid.withCategory(EmailCategory.valueOf(category)));
|
||||
assertThat(result.status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(result.reasons()).containsExactly("Notification and marketing email with an unsubscribe link goes to one recipient per message; send one message per recipient.");
|
||||
}
|
||||
H2Support.insert(EmailTables.SUPPRESSION, new QRecord().withValue("address", "other@example.com").withValue("scope", "ALL").withValue("reason", "HARD_BOUNCE"));
|
||||
assertThat(EmailService.send(request().withCc("other@example.com").withCategory(EmailCategory.valueOf(category))).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).isEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Transactional mail and notification mail without opt-out retain multi-party delivery.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void multiRecipientWithoutUnsubscribeStillQueues() throws Exception
|
||||
{
|
||||
EmailRuntime.config().withPublicBaseUrl("https://app.example.com").withSecretResolver(name -> "test-unsubscribe-key");
|
||||
var result = EmailService.send(request().withTo(List.of("person@example.com, other@example.com")).withCc("copy@example.com").withBcc("blind@example.com"));
|
||||
assertThat(result.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
assertThat(message(result.messageId()).getValueString("unsubscribeUrl")).isNull();
|
||||
assertThat(new EmailDispatcher().runOnce().sent()).isEqualTo(1);
|
||||
assertThat(mail.getReceivedMessages()).hasSize(4);
|
||||
assertThat(mail.getReceivedMessages()[0].getAllRecipients()).hasSize(3);
|
||||
EmailRuntime.config().withSecretResolver(name -> null);
|
||||
result = EmailService.send(request().withCc("copy@example.com").withBcc("blind@example.com").withCategory(EmailCategory.NOTIFICATION));
|
||||
assertThat(result.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
assertThat(message(result.messageId()).getValueString("unsubscribeUrl")).isNull();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Rate Limits Do Not Consume Attempts And Send After Is Honored.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -30,9 +30,9 @@ import java.util.Set;
|
||||
import com.kingsrook.qbits.email.app.EmailHome;
|
||||
import com.kingsrook.qbits.email.metadata.EmailEnums;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.outbox.EmailCategory;
|
||||
import com.kingsrook.qbits.email.render.ContrastCheck;
|
||||
import com.kingsrook.qbits.email.render.EmailLayout;
|
||||
import com.kingsrook.qbits.email.outbox.EmailCategory;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.dashboard.RenderWidgetAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
|
||||
@@ -66,7 +66,9 @@ class EmailHomePolishTest
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
H2Support.setup(c -> { });
|
||||
H2Support.setup(c ->
|
||||
{
|
||||
});
|
||||
EmailRuntime.setClock(Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
H2Support.provider(25);
|
||||
}
|
||||
@@ -91,6 +93,12 @@ class EmailHomePolishTest
|
||||
.withValue("sentDate", NOW.minusSeconds(86400)).withValue("deliveredDate", NOW.plusSeconds(1))
|
||||
.withValue("deadLetterDate", NOW).withValue("bouncedDate", NOW.minusSeconds(7 * 86400))
|
||||
.withValue("complainedDate", NOW.minusSeconds(7 * 86400 + 1)));
|
||||
for(String type : List.of("DELIVERED", "BOUNCED", "COMPLAINED"))
|
||||
{
|
||||
SystemWrites.insertOne(EmailTables.EVENT, new QRecord().withValue("messageId", 1).withValue("eventType", type).withValue("eventKey", type)
|
||||
.withValue("occurredDate", "DELIVERED".equals(type) ? NOW.plusSeconds(1)
|
||||
: NOW.minusSeconds(7 * 86400 + ("COMPLAINED".equals(type) ? 1 : 0))));
|
||||
}
|
||||
H2Support.insert(EmailTables.SUPPRESSION, new QRecord().withValue("address", "a@example.test").withValue("scope", "ALL").withValue("reason", "MANUAL"));
|
||||
var document = Jsoup.parse(EmailHome.overview().getHtml());
|
||||
var tiles = document.select("body > div > div > div");
|
||||
|
||||
@@ -20,14 +20,24 @@
|
||||
|
||||
package com.kingsrook.qbits.email;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.email.app.EmailApp;
|
||||
import com.kingsrook.qbits.email.metadata.*;
|
||||
import com.kingsrook.qbits.email.outbox.*;
|
||||
import com.kingsrook.qbits.email.permissions.*;
|
||||
import com.kingsrook.qbits.email.processes.*;
|
||||
import com.kingsrook.qbits.email.support.*;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.*;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTableCustomizers;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.outbox.AdHocEmailRenderer;
|
||||
import com.kingsrook.qbits.email.outbox.EmailRequest;
|
||||
import com.kingsrook.qbits.email.outbox.EmailSendResult;
|
||||
import com.kingsrook.qbits.email.outbox.EmailService;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissionChecker;
|
||||
import com.kingsrook.qbits.email.processes.EmailProcesses;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
@@ -36,10 +46,14 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.*;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
|
||||
import org.junit.jupiter.api.*;
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Email Metadata Test.
|
||||
@@ -73,7 +87,7 @@ class EmailMetadataTest
|
||||
void producerStandardsAndIntegrationFields() throws Exception
|
||||
{
|
||||
var instance = QContext.getQInstance();
|
||||
assertThat(instance.getTables()).hasSize(7);
|
||||
assertThat(instance.getTables()).hasSize(10);
|
||||
for(var table : instance.getTables().values())
|
||||
{
|
||||
assertThat(table.getName()).matches("email_[a-z_]+");
|
||||
@@ -412,7 +426,7 @@ class EmailMetadataTest
|
||||
"MAILGUN",
|
||||
"GMAIL_API",
|
||||
"future"))
|
||||
{
|
||||
{
|
||||
assertThat(
|
||||
new EmailQBitConfig()
|
||||
.withAllowedEnvNames(Set.of("CUSTOM"))
|
||||
@@ -495,8 +509,8 @@ class EmailMetadataTest
|
||||
config.validate(QContext.getQInstance(), errors);
|
||||
assertThat(errors).isEmpty();
|
||||
var scheduled = EmailProcesses.processes(config).stream().filter(p -> p.getSchedule() != null).toList();
|
||||
assertThat(scheduled).hasSize(3).allMatch(p -> p.getIsHidden());
|
||||
assertThat(scheduled).extracting(p -> p.getSchedule().getRepeatSeconds()).containsExactly(15, 300, 86400);
|
||||
assertThat(scheduled).hasSize(4).allMatch(p -> p.getIsHidden());
|
||||
assertThat(scheduled).extracting(p -> p.getSchedule().getRepeatSeconds()).containsExactly(30, 15, 300, 86400);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,199 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email;
|
||||
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.outbox.EmailCategory;
|
||||
import com.kingsrook.qbits.email.outbox.EmailRequest;
|
||||
import com.kingsrook.qbits.email.outbox.EmailSendResult;
|
||||
import com.kingsrook.qbits.email.outbox.EmailService;
|
||||
import com.kingsrook.qbits.email.processes.EmailProcesses;
|
||||
import com.kingsrook.qbits.email.render.EmailUnsubscribeLinks;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qbits.email.template.EmailTemplateDefinition;
|
||||
import com.kingsrook.qbits.email.template.EmailTemplates;
|
||||
import com.kingsrook.qbits.email.template.EmailVersions;
|
||||
import com.kingsrook.qbits.email.tracking.EmailUnsubscribe;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Retention of opt-out mail and the consumer-facing template API.
|
||||
*******************************************************************************/
|
||||
class EmailPhaseCReviewTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Isolated database with opt-out links configured and message retention on.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
H2Support.setup(c -> c.withPublicBaseUrl("https://app.example.com").withSecretResolver(name -> SECRET).withMessageRetentionDays(30));
|
||||
H2Support.provider(25);
|
||||
EmailRuntime.setClock(Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Clear thread and clock state.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
QContext.clear();
|
||||
EmailRuntime.setClock(null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** An old final message with content, in the given category.
|
||||
*******************************************************************************/
|
||||
private static QRecord oldMessage(String category, String address) throws Exception
|
||||
{
|
||||
return (SystemWrites.insertOne(EmailTables.MESSAGE, new QRecord().withValue("providerId", H2Support.all(EmailTables.PROVIDER).getFirst().getValue("id"))
|
||||
.withValue("status", "SENT").withValue("category", category)
|
||||
.withValue("toAddresses", address).withValue("sentDate", NOW.minus(Duration.ofDays(100))).withValue("htmlBody", "<p>hi</p>")
|
||||
.withValue("textBody", "hi").withValue("variablesSnapshot", "{\"name\":\"x\"}")));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Old marketing and notification mail is kept (content cleared) so its
|
||||
** unsubscribe link works; transactional mail is deleted.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void retentionKeepsOptOutMessagesAndTheirUnsubscribeLinks() throws Exception
|
||||
{
|
||||
QRecord marketing = oldMessage("MARKETING", "pat@example.com");
|
||||
QRecord notification = oldMessage("NOTIFICATION", "sam@example.com");
|
||||
QRecord transactional = oldMessage("TRANSACTIONAL", "lee@example.com");
|
||||
String url = EmailUnsubscribeLinks.create(marketing.getValueString("id"), "pat@example.com", "MARKETING");
|
||||
assertThat(url).isNotNull();
|
||||
EmailProcesses.retain();
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, transactional.getValueInteger("id"), null)).isNull();
|
||||
for(QRecord kept : List.of(marketing, notification))
|
||||
{
|
||||
QRecord row = EmailStore.get(EmailTables.MESSAGE, kept.getValueInteger("id"), null);
|
||||
assertThat(row).isNotNull();
|
||||
assertThat(row.getValueString("htmlBody")).isNull();
|
||||
assertThat(row.getValueString("textBody")).isNull();
|
||||
assertThat(row.getValueString("variablesSnapshot")).isNull();
|
||||
}
|
||||
new EmailUnsubscribe().unsubscribe(url.substring(url.lastIndexOf('/') + 1));
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.getValueString("address")).isEqualTo("pat@example.com");
|
||||
assertThat(row.getValueString("scope")).isEqualTo("MARKETING");
|
||||
});
|
||||
EmailProcesses.retain();
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, marketing.getValueInteger("id"), null)).isNotNull();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A complete definition used by the template tests.
|
||||
*******************************************************************************/
|
||||
private static EmailTemplateDefinition definition()
|
||||
{
|
||||
return (new EmailTemplateDefinition().withName("order_ready").withLabel("Order ready").withCategory(EmailCategory.TRANSACTIONAL)
|
||||
.withSubject("Order {{order_number}} is ready").withBody("Hi {{customer_name}}, order {{order_number}} is ready.")
|
||||
.withVariable("order_number", "Order number", "STRING", true, "PUBLIC")
|
||||
.withVariable("customer_name", "Customer name", "STRING", true, null)
|
||||
.withSampleVariables("{\"order_number\":\"A-1\",\"customer_name\":\"Pat\"}").withOwnerQbit("orders"));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Created once as an ACTIVE version 1 that sends; the second call changes nothing.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void ensureCreatesOnceAndIsIdempotent() throws Exception
|
||||
{
|
||||
assertThat(EmailTemplates.ensure(definition())).isTrue();
|
||||
assertThat(EmailTemplates.ensure(definition())).isFalse();
|
||||
QRecord template = H2Support.all(EmailTables.TEMPLATE).getFirst();
|
||||
assertThat(H2Support.all(EmailTables.TEMPLATE)).hasSize(1);
|
||||
assertThat(template.getValueString("ownerQbit")).isEqualTo("orders");
|
||||
assertThat(template.getValueString("kind")).isEqualTo("MARKDOWN");
|
||||
assertThat(template.getValueBoolean("phiApproved")).isFalse();
|
||||
List<QRecord> versions = H2Support.all(EmailTables.VERSION);
|
||||
assertThat(versions).hasSize(1);
|
||||
assertThat(versions.getFirst().getValueString("status")).isEqualTo("ACTIVE");
|
||||
assertThat(versions.getFirst().getValueInteger("versionNumber")).isEqualTo(1);
|
||||
assertThat(versions.getFirst().getValueString("variablesSchema")).contains("order_number").contains("PUBLIC");
|
||||
assertThat(template.getValue("activeVersionId")).isEqualTo(versions.getFirst().getValue("id"));
|
||||
EmailSendResult result = EmailService.send(new EmailRequest().withTemplateName("order_ready").withTo("pat@example.com")
|
||||
.withVariable("order_number", "A-1").withVariable("customer_name", "Pat"));
|
||||
assertThat(result.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** An edited template keeps its edits when the defining qbit calls ensure again.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void ensureDoesNotTouchAnEditedTemplate() throws Exception
|
||||
{
|
||||
assertThat(EmailTemplates.ensure(definition())).isTrue();
|
||||
QRecord template = H2Support.all(EmailTables.TEMPLATE).getFirst();
|
||||
QRecord draft = EmailVersions.newVersion(template.getValueInteger("id"));
|
||||
new UpdateAction().execute(new UpdateInput(EmailTables.VERSION).withRecords(List.of(new QRecord().withValue("id", draft.getValue("id"))
|
||||
.withValue("bodySource", "Edited by a person for {{order_number}}"))));
|
||||
EmailVersions.activate(draft.getValueInteger("id"));
|
||||
assertThat(EmailTemplates.ensure(definition().withBody("Changed again {{order_number}} {{customer_name}}"))).isFalse();
|
||||
assertThat(H2Support.all(EmailTables.VERSION)).hasSize(2);
|
||||
QRecord active = EmailStore.get(EmailTables.VERSION, EmailStore.get(EmailTables.TEMPLATE, template.getValueInteger("id"), null).getValueInteger("activeVersionId"), null);
|
||||
assertThat(active.getValueInteger("versionNumber")).isEqualTo(2);
|
||||
assertThat(active.getValueString("bodySource")).isEqualTo("Edited by a person for {{order_number}}");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Invalid definitions are refused with plain reasons and write nothing.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void ensureRefusesInvalidDefinitions() throws Exception
|
||||
{
|
||||
assertThatThrownBy(() -> EmailTemplates.ensure(new EmailTemplateDefinition().withName("Bad Name").withKind("PDF")))
|
||||
.isInstanceOf(IllegalArgumentException.class)
|
||||
.hasMessageContaining("Name must be snake_case").hasMessageContaining("Label is required").hasMessageContaining("Category is required")
|
||||
.hasMessageContaining("Kind must be").hasMessageContaining("Subject is required").hasMessageContaining("Body is required");
|
||||
assertThatThrownBy(() -> EmailTemplates.ensure(definition().withVariablesSchema("[]")))
|
||||
.hasMessageContaining("not both");
|
||||
assertThatThrownBy(() -> EmailTemplates.ensure(new EmailTemplateDefinition().withName("x").withLabel("X").withCategory(EmailCategory.SECURITY)
|
||||
.withSubject("S").withBody("B").withVariablesSchema("{").withSampleVariables("[")))
|
||||
.hasMessageContaining("Invalid variables schema").hasMessageContaining("Variables must be a JSON object");
|
||||
assertThatThrownBy(() -> EmailTemplates.ensure(definition().withBody("Hi {{undeclared_thing}}")))
|
||||
.isInstanceOf(IllegalArgumentException.class).hasMessageContaining("refused");
|
||||
assertThatThrownBy(() -> EmailTemplates.ensure(definition().withSampleVariables("{}")))
|
||||
.isInstanceOf(IllegalArgumentException.class).hasMessageContaining("refused");
|
||||
assertThat(H2Support.all(EmailTables.TEMPLATE)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.VERSION)).isEmpty();
|
||||
}
|
||||
}
|
||||
@@ -20,20 +20,28 @@
|
||||
|
||||
package com.kingsrook.qbits.email;
|
||||
|
||||
import java.time.*;
|
||||
import java.util.*;
|
||||
import com.icegreen.greenmail.util.*;
|
||||
import com.kingsrook.qbits.email.delivery.*;
|
||||
import com.kingsrook.qbits.email.metadata.*;
|
||||
import com.kingsrook.qbits.email.outbox.*;
|
||||
import com.kingsrook.qbits.email.processes.*;
|
||||
import com.kingsrook.qbits.email.support.*;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.Map;
|
||||
import com.icegreen.greenmail.util.GreenMail;
|
||||
import com.icegreen.greenmail.util.ServerSetup;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.outbox.EmailRequest;
|
||||
import com.kingsrook.qbits.email.outbox.EmailService;
|
||||
import com.kingsrook.qbits.email.processes.EmailProcesses;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.*;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.junit.jupiter.api.*;
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/*******************************************************************************
|
||||
** Email Processes Test.
|
||||
|
||||
@@ -109,7 +109,7 @@ class EmailReviewFixesTest
|
||||
var schema = new VariableSchema("[{\"name\":\"items\",\"type\":\"LIST\"},{\"name\":\"value\",\"type\":\"STRING\"}]");
|
||||
var loop = engine.parse("{{#each items}}{{#each items}}{{/each}}{{/each}}", schema);
|
||||
assertThatThrownBy(() -> loop.render(Map.of("items", Collections.nCopies(100, "")), EmailTemplateEngine.Context.TEXT)).hasMessage("Template output is too large");
|
||||
assertThatThrownBy(() -> engine.parse("{{value}}", schema).render(Map.of("value", "\u00e9".repeat(600_000)), EmailTemplateEngine.Context.TEXT))
|
||||
assertThatThrownBy(() -> engine.parse("{{value}}", schema).render(Map.of("value", "é".repeat(600_000)), EmailTemplateEngine.Context.TEXT))
|
||||
.hasMessage("Template output is too large");
|
||||
assertThat(engine.parse("{{#if value}}".repeat(8) + "ok" + "{{/if}}".repeat(8), schema).render(Map.of("value", true), EmailTemplateEngine.Context.TEXT)).isEqualTo("ok");
|
||||
assertThatThrownBy(() -> engine.parse("{{#if value}}".repeat(9) + "{{/if}}".repeat(9), schema)).hasMessageContaining("8 levels");
|
||||
@@ -179,7 +179,10 @@ class EmailReviewFixesTest
|
||||
{
|
||||
assertThatThrownBy(() -> EmailTemplateEngine.validateHtml(source)).hasMessageContaining("must be quoted");
|
||||
}
|
||||
EmailRuntime.config().withMjmlCompiler(source -> { throw new AssertionError("Unsafe MJML reached compiler"); });
|
||||
EmailRuntime.config().withMjmlCompiler(source ->
|
||||
{
|
||||
throw new AssertionError("Unsafe MJML reached compiler");
|
||||
});
|
||||
QRecord mjml = EmailTemplatesTest.draft("includes", "MJML", "[]", "{}", "<mjml><mj-include path='secret'/></mjml>");
|
||||
assertThatThrownBy(() -> EmailVersions.activate(mjml.getValueInteger("id"))).hasMessage("MJML includes are not allowed");
|
||||
assertThat(preview(mjml)).contains("MJML includes are not allowed");
|
||||
@@ -192,7 +195,10 @@ class EmailReviewFixesTest
|
||||
void viewerNeverCompiles() throws Exception
|
||||
{
|
||||
QRecord version = EmailTemplatesTest.draft("viewer", "MJML", "[]", "{}", "<mjml/>");
|
||||
EmailRuntime.config().withMjmlCompiler(source -> { throw new AssertionError("Viewer compiled MJML"); });
|
||||
EmailRuntime.config().withMjmlCompiler(source ->
|
||||
{
|
||||
throw new AssertionError("Viewer compiled MJML");
|
||||
});
|
||||
QContext.getQSession().setPermissions(Set.of("email.view"));
|
||||
assertThat(preview(version)).contains("Not compiled yet. Activate the version or use Preview as a manager.");
|
||||
QContext.getQSession().setPermissions(Set.of("email.manage"));
|
||||
@@ -270,7 +276,10 @@ class EmailReviewFixesTest
|
||||
QRecord original = EmailStore.get(EmailTables.MESSAGE, first.messageId(), null);
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", second.messageId()).withValue("htmlBody", original.getValue("htmlBody")));
|
||||
var dispatcher = new com.kingsrook.qbits.email.delivery.EmailDispatcher(new com.kingsrook.qbits.email.transport.EmailTransportRegistry()
|
||||
.withTransport("SMTP", (envelope, settings) -> { throw new AssertionError("Swapped ciphertext reached transport"); }));
|
||||
.withTransport("SMTP", (envelope, settings) ->
|
||||
{
|
||||
throw new AssertionError("Swapped ciphertext reached transport");
|
||||
}));
|
||||
assertThat(dispatcher.sendNow(second.messageId())).isEqualTo(com.kingsrook.qbits.email.delivery.EmailDispatcher.SendResult.RETRY_WAIT);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,489 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.outbox.EmailRequest;
|
||||
import com.kingsrook.qbits.email.outbox.EmailSendResult;
|
||||
import com.kingsrook.qbits.email.outbox.EmailService;
|
||||
import com.kingsrook.qbits.email.permissions.EmailPermissionChecker;
|
||||
import com.kingsrook.qbits.email.processes.EmailProcesses;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.values.SearchPossibleValueSourceAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueSearchFilterUseCase;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Provider sender addresses: validation, sending rules, the fallback to the
|
||||
** provider's own from address and every contingent possible-value pair.
|
||||
*******************************************************************************/
|
||||
class EmailSenderAddressTest
|
||||
{
|
||||
private QRecord relay;
|
||||
private QRecord other;
|
||||
|
||||
/*******************************************************************************
|
||||
** Two providers on an isolated database.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
H2Support.setup(c ->
|
||||
{
|
||||
});
|
||||
relay = H2Support.provider(25);
|
||||
other = H2Support.insert(EmailTables.PROVIDER, new QRecord().withValue("name", "other").withValue("kind", "SMTP").withValue("host", "127.0.0.1")
|
||||
.withValue("port", 25).withValue("security", "NONE").withValue("fromAddress", "other@example.com").withValue("enabled", true).withValue("isDefault", false));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Clear thread state.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A sender address row for a provider.
|
||||
*******************************************************************************/
|
||||
private static QRecord row(QRecord provider, String address)
|
||||
{
|
||||
return (new QRecord().withValue("providerId", provider.getValue("id")).withValue("address", address).withValue("enabled", true).withValue("isDefault", false));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Insert through the table action and return the (possibly errored) result.
|
||||
*******************************************************************************/
|
||||
private static QRecord tryInsert(QRecord row) throws Exception
|
||||
{
|
||||
return (new InsertAction().execute(new InsertInput(EmailTables.SENDER).withRecords(List.of(row))).getRecords().get(0));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A one-recipient request.
|
||||
*******************************************************************************/
|
||||
private static EmailRequest request(String providerName)
|
||||
{
|
||||
return (new EmailRequest().withProviderName(providerName).withTo("person@example.net").withSubject("Hello").withTextBody("Body"));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Addresses are canonical lowercase and line breaks, bad addresses are refused.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void addressIsCanonicalAndSafe() throws Exception
|
||||
{
|
||||
QRecord saved = H2Support.insert(EmailTables.SENDER, row(relay, "Sales@Example.COM"));
|
||||
assertThat(saved.getValueString("address")).isEqualTo("sales@example.com");
|
||||
assertThat(tryInsert(row(relay, "a@example.com\nBcc: x@example.net")).getErrors()).isNotEmpty();
|
||||
assertThat(tryInsert(row(relay, "not an address")).getErrors()).isNotEmpty();
|
||||
assertThat(tryInsert(row(relay, "ok@example.com").withValue("displayName", "Evil\r\nBcc: x@example.net")).getErrors()).isNotEmpty();
|
||||
assertThat(tryInsert(new QRecord().withValue("providerId", 9999).withValue("address", "x@example.com")).getErrors()).isNotEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The domain must equal the provider sending domain when one is set.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void domainMustMatchSendingDomain() throws Exception
|
||||
{
|
||||
SystemWrites.updateOne(EmailTables.PROVIDER, new QRecord().withValue("id", relay.getValue("id")).withValue("sendingDomain", "example.com"));
|
||||
assertThat(tryInsert(row(relay, "x@other.org")).getErrors()).isNotEmpty();
|
||||
assertThat(tryInsert(row(relay, "x@EXAMPLE.com")).getErrors()).isNullOrEmpty();
|
||||
QRecord off = tryInsert(row(other, "x@other.org"));
|
||||
assertThat(off.getErrors()).isNullOrEmpty();
|
||||
var update = new UpdateAction().execute(new UpdateInput(EmailTables.PROVIDER).withRecords(List.of(new QRecord().withValue("id", other.getValue("id")).withValue("sendingDomain", "example.com"))));
|
||||
assertThat(update.getRecords().get(0).getErrors()).isNotEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Unique per provider; the same address may belong to another provider.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void addressIsUniquePerProvider() throws Exception
|
||||
{
|
||||
H2Support.insert(EmailTables.SENDER, row(relay, "sales@example.com"));
|
||||
assertThat(tryInsert(row(relay, "SALES@example.com")).getErrors()).isNotEmpty();
|
||||
assertThat(tryInsert(row(other, "sales@example.com")).getErrors()).isNullOrEmpty();
|
||||
var both = new InsertAction().execute(new InsertInput(EmailTables.SENDER).withRecords(List.of(row(relay, "twin@example.com"), row(relay, "twin@example.com")))).getRecords();
|
||||
assertThat(both.stream().filter(r -> r.getErrors() != null && !r.getErrors().isEmpty())).isNotEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** At most one default per provider, and a default must be enabled.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void singleEnabledDefault() throws Exception
|
||||
{
|
||||
H2Support.insert(EmailTables.SENDER, row(relay, "a@example.com").withValue("isDefault", true));
|
||||
assertThat(tryInsert(row(relay, "b@example.com").withValue("isDefault", true)).getErrors()).isNotEmpty();
|
||||
assertThat(tryInsert(row(other, "c@example.com").withValue("isDefault", true)).getErrors()).isNullOrEmpty();
|
||||
assertThat(tryInsert(row(relay, "d@example.com").withValue("isDefault", true).withValue("enabled", false)).getErrors()).isNotEmpty();
|
||||
QRecord plain = H2Support.insert(EmailTables.SENDER, row(relay, "e@example.com"));
|
||||
var promote = new UpdateAction().execute(new UpdateInput(EmailTables.SENDER).withRecords(List.of(new QRecord().withValue("id", plain.getValue("id")).withValue("isDefault", true))));
|
||||
assertThat(promote.getRecords().get(0).getErrors()).isNotEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Deleting a provider removes its sender addresses.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void deletingProviderRemovesSenderAddresses() throws Exception
|
||||
{
|
||||
H2Support.insert(EmailTables.SENDER, row(other, "gone@example.com"));
|
||||
new DeleteAction().execute(new DeleteInput(EmailTables.PROVIDER).withPrimaryKey(other.getValue("id")));
|
||||
assertThat(H2Support.all(EmailTables.SENDER)).isEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sender rows are admin-only, like providers.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void senderRowsAreAdminOnly() throws Exception
|
||||
{
|
||||
var table = QContext.getQInstance().getTable(EmailTables.SENDER);
|
||||
assertThat(table.getPermissionRules().getPermissionBaseName()).isEqualTo("email");
|
||||
assertThat(EmailTables.MAIN).doesNotContain(EmailTables.SENDER);
|
||||
assertThat(QContext.getQInstance().getJoin(EmailTables.PROVIDER + "_join_" + EmailTables.SENDER)).isNotNull();
|
||||
assertThat(QContext.getQInstance().getWidget(EmailTables.SENDER + "_children")).isNotNull();
|
||||
QContext.getQSession().setPermissions(java.util.Set.of("email.manage"));
|
||||
var checker = new EmailPermissionChecker();
|
||||
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new InsertInput(EmailTables.SENDER), table)).hasMessageContaining("email.admin");
|
||||
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new QueryInput(EmailTables.SENDER), table)).hasMessageContaining("email.admin");
|
||||
QContext.getQSession().setPermissions(java.util.Set.of("email.admin"));
|
||||
checker.checkPermissionsThrowing(new InsertInput(EmailTables.SENDER), table);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sending with a chosen id: ok, another provider's row, and a disabled row.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sendWithFromAddressId() throws Exception
|
||||
{
|
||||
QRecord mine = H2Support.insert(EmailTables.SENDER, row(relay, "Sales@example.com").withValue("displayName", "Sales Desk"));
|
||||
QRecord theirs = H2Support.insert(EmailTables.SENDER, row(other, "help@example.com"));
|
||||
QRecord off = H2Support.insert(EmailTables.SENDER, row(relay, "old@example.com").withValue("enabled", false));
|
||||
EmailSendResult ok = EmailService.send(request("relay").withFromAddressId(mine.getValueInteger("id")));
|
||||
assertThat(ok.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
QRecord stored = EmailStore.get(EmailTables.MESSAGE, ok.messageId(), null);
|
||||
assertThat(stored.getValueString("fromAddress")).contains("sales@example.com").contains("Sales Desk");
|
||||
assertThat(stored.getValueInteger("fromAddressId")).isEqualTo(mine.getValueInteger("id"));
|
||||
EmailSendResult wrong = EmailService.send(request("relay").withFromAddressId(theirs.getValueInteger("id")));
|
||||
assertThat(wrong.status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(wrong.reasons()).containsExactly("That from address is not set up for provider relay");
|
||||
EmailSendResult disabled = EmailService.send(request("relay").withFromAddressId(off.getValueInteger("id")));
|
||||
assertThat(disabled.status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(disabled.reasons()).containsExactly("That from address is not set up for provider relay");
|
||||
assertThat(EmailService.send(request("relay").withFromAddressId(4242)).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(EmailService.send(request("relay").withFromAddressId(mine.getValueInteger("id")).withFrom("other@example.com")).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Once a provider has rows an explicit from text must match an enabled one.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void fromTextMustMatchRowsWhenTheyExist() throws Exception
|
||||
{
|
||||
QRecord listed = H2Support.insert(EmailTables.SENDER, row(relay, "sales@example.com"));
|
||||
H2Support.insert(EmailTables.SENDER, row(relay, "old@example.com").withValue("enabled", false));
|
||||
EmailSendResult ok = EmailService.send(request("relay").withFrom("Sales Team <SALES@example.com>"));
|
||||
assertThat(ok.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
QRecord stored = EmailStore.get(EmailTables.MESSAGE, ok.messageId(), null);
|
||||
assertThat(stored.getValueString("fromAddress")).contains("Sales Team");
|
||||
assertThat(stored.getValueInteger("fromAddressId")).isEqualTo(listed.getValueInteger("id"));
|
||||
EmailSendResult unlisted = EmailService.send(request("relay").withFrom("random@example.com"));
|
||||
assertThat(unlisted.status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(unlisted.reasons()).containsExactly("That from address is not set up for provider relay");
|
||||
assertThat(EmailService.send(request("relay").withFrom("old@example.com")).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(EmailService.send(request("relay").withFrom("sender@example.com")).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(EmailService.send(request("relay").withFrom("not an address")).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Without rows, behavior is unchanged: any from text, and the provider's from address and name.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void fallsBackToProviderFromAddress() throws Exception
|
||||
{
|
||||
SystemWrites.updateOne(EmailTables.PROVIDER, new QRecord().withValue("id", relay.getValue("id")).withValue("fromName", "Relay Team"));
|
||||
EmailSendResult plain = EmailService.send(request("relay"));
|
||||
QRecord stored = EmailStore.get(EmailTables.MESSAGE, plain.messageId(), null);
|
||||
assertThat(stored.getValueString("fromAddress")).contains("sender@example.com").contains("Relay Team");
|
||||
assertThat(stored.getValue("fromAddressId")).isNull();
|
||||
EmailSendResult explicit = EmailService.send(request("relay").withFrom("anyone@example.org"));
|
||||
assertThat(explicit.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, explicit.messageId(), null).getValueString("fromAddress")).isEqualTo("anyone@example.org");
|
||||
H2Support.insert(EmailTables.SENDER, row(other, "help@example.com"));
|
||||
EmailSendResult unaffected = EmailService.send(request("relay").withFrom("anyone@example.org"));
|
||||
assertThat(unaffected.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** An enabled default row wins over the provider's own from address; a disabled one does not.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void defaultRowWins() throws Exception
|
||||
{
|
||||
QRecord preferred = H2Support.insert(EmailTables.SENDER, row(relay, "team@example.com").withValue("displayName", "The Team").withValue("isDefault", true));
|
||||
EmailSendResult result = EmailService.send(request("relay"));
|
||||
QRecord stored = EmailStore.get(EmailTables.MESSAGE, result.messageId(), null);
|
||||
assertThat(stored.getValueString("fromAddress")).contains("team@example.com").contains("The Team");
|
||||
assertThat(stored.getValueInteger("fromAddressId")).isEqualTo(preferred.getValueInteger("id"));
|
||||
SystemWrites.updateOne(EmailTables.SENDER, new QRecord().withValue("id", preferred.getValue("id")).withValue("enabled", false));
|
||||
QRecord back = EmailStore.get(EmailTables.MESSAGE, EmailService.send(request("relay")).messageId(), null);
|
||||
assertThat(back.getValueString("fromAddress")).isEqualTo("sender@example.com");
|
||||
assertThat(back.getValue("fromAddressId")).isNull();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A resend keeps the original sender row.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void resendKeepsSenderRow() throws Exception
|
||||
{
|
||||
QRecord mine = H2Support.insert(EmailTables.SENDER, row(relay, "sales@example.com"));
|
||||
EmailSendResult sent = EmailService.send(request("relay").withFromAddressId(mine.getValueInteger("id")));
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", sent.messageId()).withValue("status", "DEAD_LETTER"));
|
||||
EmailSendResult again = EmailService.resend(sent.messageId());
|
||||
assertThat(again.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, again.messageId(), null).getValueInteger("fromAddressId")).isEqualTo(mine.getValueInteger("id"));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The templated and ad hoc send processes accept providerId and fromAddressId.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sendProcessAcceptsProviderAndFromAddressIds() throws Exception
|
||||
{
|
||||
QRecord mine = H2Support.insert(EmailTables.SENDER, row(relay, "sales@example.com"));
|
||||
QRecord theirs = H2Support.insert(EmailTables.SENDER, row(other, "help@example.com"));
|
||||
Map<String, Serializable> values = Map.of("to", "person@example.net", "subject", "Hi", "textBody", "Body", "providerId", relay.getValueInteger("id"), "fromAddressId", mine.getValueInteger("id"));
|
||||
var input = new RunProcessInput();
|
||||
input.setProcessName(EmailProcesses.SEND_AD_HOC);
|
||||
values.forEach(input::addValue);
|
||||
assertThat(new RunProcessAction().execute(input).getValues().get("status")).isEqualTo("QUEUED");
|
||||
var wrong = new RunProcessInput();
|
||||
wrong.setProcessName(EmailProcesses.SEND_AD_HOC);
|
||||
values.forEach(wrong::addValue);
|
||||
wrong.addValue("fromAddressId", theirs.getValueInteger("id"));
|
||||
assertThat(new RunProcessAction().execute(wrong).getValues().get("status")).isEqualTo("REFUSED");
|
||||
var named = new RunProcessInput();
|
||||
named.setProcessName(EmailProcesses.SEND_AD_HOC);
|
||||
values.forEach(named::addValue);
|
||||
named.addValue("providerName", "relay");
|
||||
assertThat(new RunProcessAction().execute(named).getValues().get("status")).isEqualTo("QUEUED");
|
||||
var mismatch = new RunProcessInput();
|
||||
mismatch.setProcessName(EmailProcesses.SEND_AD_HOC);
|
||||
values.forEach(mismatch::addValue);
|
||||
mismatch.addValue("providerName", "other");
|
||||
assertThatThrownBy(() -> new RunProcessAction().execute(mismatch)).hasMessageContaining("providerId does not match");
|
||||
var missing = new RunProcessInput();
|
||||
missing.setProcessName(EmailProcesses.SEND_AD_HOC);
|
||||
values.forEach(missing::addValue);
|
||||
missing.addValue("providerId", 4242);
|
||||
assertThatThrownBy(() -> new RunProcessAction().execute(missing)).hasMessageContaining("providerId does not match");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Send Test Email takes an optional from address that must belong to the provider.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sendTestUsesFromAddress() throws Exception
|
||||
{
|
||||
QRecord mine = H2Support.insert(EmailTables.SENDER, row(relay, "sales@example.com"));
|
||||
QRecord theirs = H2Support.insert(EmailTables.SENDER, row(other, "help@example.com"));
|
||||
var wrong = ProcessRunner.runWithValues(EmailProcesses.SEND_TEST, Map.of("toAddress", "test@example.com", "fromAddressId", theirs.getValueInteger("id")), relay.getValue("id"));
|
||||
assertThat(wrong.result()).anyMatch(s -> s.contains("REFUSED"));
|
||||
var ok = ProcessRunner.runWithValues(EmailProcesses.SEND_TEST, Map.of("toAddress", "test@example.com", "fromAddressId", mine.getValueInteger("id")), relay.getValue("id"));
|
||||
assertThat(ok.result()).anyMatch(s -> s.contains("FAILED") || s.contains("SENT") || s.contains("RETRY") || s.contains("QUEUED"));
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).anyMatch(m -> mine.getValue("id").equals(m.getValue("fromAddressId")));
|
||||
}
|
||||
|
||||
// ----- contingent possible-value pairs -----
|
||||
|
||||
/*******************************************************************************
|
||||
** Assert the filter on a field and that a possible-value search with the parent
|
||||
** value returns only that parent's rows. The search runs through core's action.
|
||||
*******************************************************************************/
|
||||
private static void assertContingent(QFieldMetaData field, String inputName, Serializable parentValue, List<Serializable> expected, List<Serializable> excluded) throws Exception
|
||||
{
|
||||
assertThat(field.getPossibleValueSourceFilter()).isNotNull();
|
||||
var filter = field.getPossibleValueSourceFilter().clone();
|
||||
assertThat(filter.toString()).contains("${input." + inputName + "}");
|
||||
filter.interpretValues(Map.of("input", Map.of(inputName, parentValue), "processValues", Map.<String, Serializable>of()), PossibleValueSearchFilterUseCase.FORM);
|
||||
var found = new SearchPossibleValueSourceAction().execute(new SearchPossibleValueSourceInput().withPossibleValueSourceName(field.getPossibleValueSourceName()).withDefaultQueryFilter(filter));
|
||||
List<String> ids = found.getResults().stream().map(v -> v.getId().toString()).toList();
|
||||
assertThat(ids).containsAll(expected.stream().map(String::valueOf).toList());
|
||||
assertThat(ids).doesNotContainAnyElementsOf(excluded.stream().map(String::valueOf).toList());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** One row each for the two providers.
|
||||
*******************************************************************************/
|
||||
private QRecord[] senders() throws Exception
|
||||
{
|
||||
return (new QRecord[] { H2Support.insert(EmailTables.SENDER, row(relay, "a@example.com")), H2Support.insert(EmailTables.SENDER, row(other, "b@example.com")) });
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Templated send: fromAddressId follows providerId.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void templatedSendFromAddressFollowsProvider() throws Exception
|
||||
{
|
||||
QRecord[] rows = senders();
|
||||
QFieldMetaData field = QContext.getQInstance().getProcess(EmailProcesses.SEND_TEMPLATED).getBackendStep("send").getInputMetaData().getFieldThrowing("fromAddressId");
|
||||
assertThat(field.getPossibleValueSourceName()).isEqualTo(EmailTables.SENDER);
|
||||
assertContingent(field, "providerId", relay.getValueInteger("id"), List.of(rows[0].getValueInteger("id")), List.of(rows[1].getValueInteger("id")));
|
||||
assertContingent(field, "providerId", other.getValueInteger("id"), List.of(rows[1].getValueInteger("id")), List.of(rows[0].getValueInteger("id")));
|
||||
assertThat(QContext.getQInstance().getProcess(EmailProcesses.SEND_TEMPLATED).getBackendStep("send").getInputMetaData().getFieldThrowing("providerId").getPossibleValueSourceName()).isEqualTo(EmailTables.PROVIDER);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Ad hoc send: fromAddressId follows providerId.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void adHocSendFromAddressFollowsProvider() throws Exception
|
||||
{
|
||||
QRecord[] rows = senders();
|
||||
QFieldMetaData field = QContext.getQInstance().getProcess(EmailProcesses.SEND_AD_HOC).getBackendStep("send").getInputMetaData().getFieldThrowing("fromAddressId");
|
||||
assertContingent(field, "providerId", relay.getValueInteger("id"), List.of(rows[0].getValueInteger("id")), List.of(rows[1].getValueInteger("id")));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Send Test Email: fromAddressId follows the provider being acted on, which a
|
||||
** record action's form only gets from the prepare step's process value.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sendTestFromAddressFollowsActedOnProvider() throws Exception
|
||||
{
|
||||
QRecord[] rows = senders();
|
||||
QFieldMetaData field = QContext.getQInstance().getProcess(EmailProcesses.SEND_TEST).getFrontendStep("input").getFormFields().stream()
|
||||
.filter(f -> "fromAddressId".equals(f.getName())).findFirst().orElseThrow();
|
||||
assertContingent(field, "id", relay.getValueInteger("id"), List.of(rows[0].getValueInteger("id")), List.of(rows[1].getValueInteger("id")));
|
||||
var input = new RunProcessInput();
|
||||
input.setProcessName(EmailProcesses.SEND_TEST);
|
||||
input.addValue("recordsParam", "recordIds");
|
||||
input.addValue("recordIds", String.valueOf(other.getValue("id")));
|
||||
input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK);
|
||||
var seeded = new RunProcessAction().execute(input).getValues().get(EmailProcesses.PROVIDER_FILTER_VALUE);
|
||||
assertThat(seeded).isEqualTo(other.getValue("id"));
|
||||
var filter = field.getPossibleValueSourceFilter().clone();
|
||||
filter.interpretValues(Map.of("input", Map.<String, Serializable>of(), "processValues", Map.of(EmailProcesses.PROVIDER_FILTER_VALUE, (Serializable) seeded)), PossibleValueSearchFilterUseCase.FORM);
|
||||
var found = new SearchPossibleValueSourceAction().execute(new SearchPossibleValueSourceInput().withPossibleValueSourceName(EmailTables.SENDER).withDefaultQueryFilter(filter));
|
||||
assertThat(found.getResults().stream().map(v -> v.getId().toString()).toList()).containsExactly(rows[1].getValueString("id"));
|
||||
var several = new RunProcessInput();
|
||||
several.setProcessName(EmailProcesses.SEND_TEST);
|
||||
several.addValue("recordsParam", "recordIds");
|
||||
several.addValue("recordIds", "1,2");
|
||||
several.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK);
|
||||
assertThat(new RunProcessAction().execute(several).getValues()).doesNotContainKey(EmailProcesses.PROVIDER_FILTER_VALUE);
|
||||
var none = new RunProcessInput();
|
||||
none.setProcessName(EmailProcesses.SEND_TEST);
|
||||
none.addValue("recordsParam", "recordIds");
|
||||
none.addValue("recordIds", "x");
|
||||
none.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK);
|
||||
assertThat(new RunProcessAction().execute(none).getValues()).doesNotContainKey(EmailProcesses.PROVIDER_FILTER_VALUE);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Message: fromAddressId follows providerId, and is read-only.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void messageFromAddressFollowsProvider() throws Exception
|
||||
{
|
||||
QRecord[] rows = senders();
|
||||
QFieldMetaData field = QContext.getQInstance().getTable(EmailTables.MESSAGE).getField("fromAddressId");
|
||||
assertThat(field.getIsEditable()).isFalse();
|
||||
assertContingent(field, "providerId", relay.getValueInteger("id"), List.of(rows[0].getValueInteger("id")), List.of(rows[1].getValueInteger("id")));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Attempt and event: the message follows the provider.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void attemptAndEventMessageFollowProvider() throws Exception
|
||||
{
|
||||
QRecord fromRelay = EmailStore.get(EmailTables.MESSAGE, EmailService.send(request("relay")).messageId(), null);
|
||||
QRecord fromOther = EmailStore.get(EmailTables.MESSAGE, EmailService.send(request("other")).messageId(), null);
|
||||
for(String table : List.of(EmailTables.ATTEMPT, EmailTables.EVENT))
|
||||
{
|
||||
QFieldMetaData field = QContext.getQInstance().getTable(table).getField("messageId");
|
||||
assertContingent(field, "providerId", relay.getValueInteger("id"), List.of(fromRelay.getValueInteger("id")), List.of(fromOther.getValueInteger("id")));
|
||||
assertContingent(field, "providerId", other.getValueInteger("id"), List.of(fromOther.getValueInteger("id")), List.of(fromRelay.getValueInteger("id")));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Suppression: the source event follows the source message.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void suppressionSourceEventFollowsSourceMessage() throws Exception
|
||||
{
|
||||
QRecord first = EmailStore.get(EmailTables.MESSAGE, EmailService.send(request("relay")).messageId(), null);
|
||||
QRecord second = EmailStore.get(EmailTables.MESSAGE, EmailService.send(request("other")).messageId(), null);
|
||||
QRecord e1 = SystemWrites.insertOne(EmailTables.EVENT, new QRecord().withValue("messageId", first.getValue("id")).withValue("eventType", "BOUNCED").withValue("eventKey", "k1"));
|
||||
QRecord e2 = SystemWrites.insertOne(EmailTables.EVENT, new QRecord().withValue("messageId", second.getValue("id")).withValue("eventType", "BOUNCED").withValue("eventKey", "k2"));
|
||||
QFieldMetaData field = QContext.getQInstance().getTable(EmailTables.SUPPRESSION).getField("sourceEventId");
|
||||
assertContingent(field, "sourceMessageId", first.getValueInteger("id"), List.of(e1.getValueInteger("id")), List.of(e2.getValueInteger("id")));
|
||||
assertContingent(field, "sourceMessageId", second.getValueInteger("id"), List.of(e2.getValueInteger("id")), List.of(e1.getValueInteger("id")));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fields without a parent scope stay unfiltered.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void independentRelationshipsStayUnfiltered() throws Exception
|
||||
{
|
||||
var instance = QContext.getQInstance();
|
||||
assertThat(instance.getTable(EmailTables.TEMPLATE).getField("brandId").getPossibleValueSourceFilter()).isNull();
|
||||
assertThat(instance.getTable(EmailTables.MESSAGE).getField("brandId").getPossibleValueSourceFilter()).isNull();
|
||||
assertThat(instance.getTable(EmailTables.MESSAGE).getField("providerId").getPossibleValueSourceFilter()).isNull();
|
||||
assertThat(instance.getTable(EmailTables.MESSAGE).getField("templateId").getPossibleValueSourceFilter()).isNull();
|
||||
assertThat(instance.getTable(EmailTables.VERSION).getField("templateId").getPossibleValueSourceFilter()).isNull();
|
||||
assertThat(instance.getTable(EmailTables.INBOX).getField("providerId").getPossibleValueSourceFilter()).isNull();
|
||||
assertThat(instance.getTable(EmailTables.SENDER).getField("providerId").getPossibleValueSourceFilter()).isNull();
|
||||
}
|
||||
}
|
||||
@@ -148,7 +148,10 @@ class EmailTemplatesTest
|
||||
assertThat(((RawHTML) preview.getWidgetData()).getHtml()).contains("<p>Email</p>");
|
||||
assertThat(EmailStore.get(EmailTables.VERSION, version.getValueInteger("id"), null).getValue("compiledHtml")).isNull();
|
||||
EmailVersions.activate(version.getValueInteger("id"));
|
||||
EmailRuntime.config().withMjmlCompiler(source -> { throw new AssertionError("Compiler ran at send time"); });
|
||||
EmailRuntime.config().withMjmlCompiler(source ->
|
||||
{
|
||||
throw new AssertionError("Compiler ran at send time");
|
||||
});
|
||||
var result = EmailService.send(new EmailRequest().withTemplateName("mjml").withLocale("fr").withTo("p@example.com"));
|
||||
assertThat(result.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
QRecord row = EmailStore.get(EmailTables.MESSAGE, result.messageId(), null);
|
||||
@@ -157,6 +160,33 @@ class EmailTemplatesTest
|
||||
assertThat(row.getValueString("subject")).endsWith(result.messageId().toString());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Template categories and retained resend links enforce the same mailbox boundary.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unsubscribeTemplateCategoryAndResend() throws Exception
|
||||
{
|
||||
EmailRuntime.config().withSecretResolver(name -> "test-unsubscribe-key");
|
||||
QRecord version = draft("notice", "MARKDOWN", "[]", "{}", "A notification");
|
||||
SystemWrites.updateOne(EmailTables.TEMPLATE, new QRecord().withValue("id", version.getValue("templateId")).withValue("category", "NOTIFICATION"));
|
||||
EmailVersions.activate(version.getValueInteger("id"));
|
||||
var refused = EmailService.send(new EmailRequest().withTemplateName("notice").withTo("person@example.com").withBcc("other@example.com"));
|
||||
assertThat(refused.status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
assertThat(refused.reasons()).containsExactly("Notification and marketing email with an unsubscribe link goes to one recipient per message; send one message per recipient.");
|
||||
var queued = EmailService.send(new EmailRequest().withTemplateName("notice").withTo("\"Person, One\" <PERSON@example.com>"));
|
||||
assertThat(queued.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
var resent = EmailService.resend(queued.messageId());
|
||||
assertThat(resent.status()).isEqualTo(EmailSendResult.Status.QUEUED);
|
||||
QRecord message = EmailStore.get(EmailTables.MESSAGE, queued.messageId(), null);
|
||||
String url = message.getValueString("unsubscribeUrl");
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, resent.messageId(), null).getValueString("unsubscribeUrl")).isEqualTo(url);
|
||||
new com.kingsrook.qbits.email.tracking.EmailUnsubscribe().unsubscribe(url.substring(url.lastIndexOf('/') + 1));
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1).allSatisfy(row -> assertThat(row.getValueString("address")).isEqualTo("person@example.com"));
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", queued.messageId()).withValue("ccAddresses", "other@example.com"));
|
||||
EmailRuntime.config().withSecretResolver(name -> null);
|
||||
assertThat(EmailService.resend(queued.messageId()).status()).isEqualTo(EmailSendResult.Status.REFUSED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Branded rendered templates reach a real SMTP sink with both alternatives.
|
||||
*******************************************************************************/
|
||||
@@ -220,8 +250,15 @@ class EmailTemplatesTest
|
||||
SystemWrites.insertOne(EmailTables.ATTEMPT, new QRecord().withValue("messageId", 1).withValue("providerId", 1).withValue("attemptNumber", 1)
|
||||
.withValue("outcome", "SUCCEEDED").withValue("startDate", now.minusSeconds(10)).withValue("endDate", now));
|
||||
H2Support.insert(EmailTables.SUPPRESSION, new QRecord().withValue("address", "x@example.com").withValue("scope", "ALL").withValue("reason", "MANUAL"));
|
||||
for(String type : List.of("DELIVERED", "BOUNCED", "COMPLAINED"))
|
||||
{
|
||||
SystemWrites.insertOne(EmailTables.EVENT, new QRecord().withValue("messageId", 1).withValue("eventType", type).withValue("eventKey", type)
|
||||
.withValue("occurredDate", "DELIVERED".equals(type) ? now : now.minusSeconds(("BOUNCED".equals(type) ? 2 : 8) * 86400)));
|
||||
}
|
||||
SystemWrites.insertOne(EmailTables.EVENT, new QRecord().withValue("messageId", 1).withValue("eventType", "DELIVERED").withValue("eventKey", "second-delivery")
|
||||
.withValue("occurredDate", now.minusSeconds(60)));
|
||||
var tiles = org.jsoup.Jsoup.parse(EmailHome.overview().getHtml()).select("body > div > div > div");
|
||||
assertThat(tiles).extracting(org.jsoup.nodes.Element::text).containsExactly("1 Sent (24 h)", "1 Delivered (24 h)", "0 Dead letters (24 h)", "1 Bounced (7 d)", "0 Complaints (7 d)", "1 Suppressed addresses");
|
||||
assertThat(tiles).extracting(org.jsoup.nodes.Element::text).containsExactly("1 Sent (24 h)", "2 Delivered (24 h)", "0 Dead letters (24 h)", "1 Bounced (7 d)", "0 Complaints (7 d)", "1 Suppressed addresses");
|
||||
assertThat(EmailHome.hourly().getChartData().getDatasets().getFirst().getData()).contains(1);
|
||||
assertThat(EmailHome.suppressions().getChartData().getDatasets().get(3).getData()).contains(1);
|
||||
QRecord defaultBrand = H2Support.insert(EmailTables.BRAND, new QRecord().withValue("name", "default").withValue("label", "Default").withValue("isDefault", true));
|
||||
@@ -261,6 +298,12 @@ class EmailTemplatesTest
|
||||
EmailRuntime.config().withSecretResolver(name -> null);
|
||||
assertThat(com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("1", "address", "NOTIFICATION")).isNull();
|
||||
assertThatThrownBy(() -> com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("1", "address", "MARKETING")).hasMessageContaining("QQQ_EMAIL_UNSUBSCRIBE_KEY");
|
||||
EmailRuntime.config().withSecretResolver(name ->
|
||||
{
|
||||
throw new IllegalArgumentException("resolver detail");
|
||||
});
|
||||
assertThatThrownBy(() -> com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("1", "person@example.com", "NOTIFICATION"))
|
||||
.hasMessage("Opt-out email needs publicBaseUrl and QQQ_EMAIL_UNSUBSCRIBE_KEY");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
|
||||
@@ -85,7 +85,7 @@ public final class H2Support
|
||||
*******************************************************************************/
|
||||
public static QInstance instance(RDBMSBackendMetaData backend, Consumer<EmailQBitConfig> tweak)
|
||||
throws Exception
|
||||
{
|
||||
{
|
||||
QInstance instance = new QInstance();
|
||||
instance.withInstanceDefaultAuthentication(
|
||||
new QAuthenticationMetaData()
|
||||
|
||||
@@ -81,7 +81,17 @@ class PhiCanaryEgressTest
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
canary = "CANARY-PHI-" + UUID.randomUUID();
|
||||
////////////////////////////////////////////////////////////////////////
|
||||
// the random part is mapped from hex to the letters g-v: a five-char //
|
||||
// window of a hex canary could match any hex digest (body hashes, //
|
||||
// Message-IDs) by chance; these windows can only match real leakage //
|
||||
////////////////////////////////////////////////////////////////////////
|
||||
StringBuilder random = new StringBuilder();
|
||||
for(char c : UUID.randomUUID().toString().replace("-", "").toCharArray())
|
||||
{
|
||||
random.append((char) ('g' + Character.digit(c, 16)));
|
||||
}
|
||||
canary = "CANARY-PHI-" + random;
|
||||
H2Support.setup(c -> c.withMaxAttempts(2).withSecretResolver(PhiCanaryEgressTest::secret)
|
||||
.withClassificationOverrides(Map.of("test_patient.name", EmailClassifications.Level.PHI)));
|
||||
H2Support.provider(25);
|
||||
|
||||
@@ -66,7 +66,7 @@ public final class ProcessRunner
|
||||
public static Lines runWithValues(
|
||||
String process, java.util.Map<String, Serializable> values, Serializable... ids)
|
||||
throws Exception
|
||||
{
|
||||
{
|
||||
RunProcessInput input = new RunProcessInput();
|
||||
input.setProcessName(process);
|
||||
input.addValue("recordsParam", "recordIds");
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.outbox;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.email.H2Support;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Durable handoff semantics and stored identity using the real H2 outbox.
|
||||
*******************************************************************************/
|
||||
class EmailEffectsTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Prepare a provider without opening a transport connection.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
H2Support.setup(config ->
|
||||
{
|
||||
});
|
||||
H2Support.provider(25);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Release the per-thread context.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void cleanup()
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A plain message uses the configured sender.
|
||||
*******************************************************************************/
|
||||
private EmailRequest request()
|
||||
{
|
||||
return (new EmailRequest().withTo("person@example.com").withSubject("Effect").withTextBody("Body"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Replays return the same persisted reference without another row.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void handoffDeduplicatesAndLookupsReturnCurrentReferences() throws Exception
|
||||
{
|
||||
String key = "job:2:row:email";
|
||||
EmailRequest request = request();
|
||||
var first = EmailEffects.send(request, key);
|
||||
var second = EmailEffects.send(request(), key);
|
||||
assertThat(request.getIdempotencyKey()).isEqualTo(key);
|
||||
assertThat(first.outcome()).isEqualTo(EmailEffectOutcome.QUEUED_OR_DUPLICATE);
|
||||
assertThat(second).isEqualTo(first);
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1);
|
||||
String token = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(key.getBytes(StandardCharsets.UTF_8))).substring(0, 32);
|
||||
assertThat(first.message().rfcMessageId()).isEqualTo("<" + token + "@example.com>");
|
||||
assertThat(first.message().status()).isEqualTo("QUEUED");
|
||||
assertThat(first.message().providerMessageId()).isNull();
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", first.message().id())
|
||||
.withValue("status", "SENT").withValue("providerMessageId", "provider-reference"));
|
||||
QContext.init(QContext.getQInstance(), new QSession().withPermissions(List.of()));
|
||||
var found = EmailEffects.findByIdempotencyKey(key).orElseThrow();
|
||||
assertThat(found.id()).isEqualTo(first.message().id());
|
||||
assertThat(found.status()).isEqualTo("SENT");
|
||||
assertThat(found.providerMessageId()).isEqualTo("provider-reference");
|
||||
assertThat(EmailEffects.findByRfcMessageId(first.message().rfcMessageId())).contains(found);
|
||||
assertThat(EmailEffects.findByIdempotencyKey("missing")).isEmpty();
|
||||
assertThat(EmailEffects.findByRfcMessageId("<missing@example.com>")).isEmpty();
|
||||
assertThat(EmailEffects.findByIdempotencyKey(null)).isEmpty();
|
||||
assertThat(EmailEffects.findByRfcMessageId(null)).isEmpty();
|
||||
assertThat(EmailEffects.findByIdempotencyKey(" ")).isEmpty();
|
||||
assertThat(EmailEffects.findByRfcMessageId(" ")).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Conflicting or unusable requests never enqueue or change an existing key.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void refusesConflictingKeysAndInvalidRequests() throws Exception
|
||||
{
|
||||
EmailRequest conflict = request().withIdempotencyKey("other");
|
||||
assertThat(EmailEffects.send(conflict, "effect").outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(conflict.getIdempotencyKey()).isEqualTo("other");
|
||||
assertThat(EmailEffects.send(request(), null).outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(EmailEffects.send(request(), " ").outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(EmailEffects.send(request(), "a".repeat(501)).outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(EmailEffects.send(null, "effect").outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
var invalid = EmailEffects.send(request().withSubject(null), "invalid");
|
||||
assertThat(invalid.outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(invalid.message()).isNull();
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).isEmpty();
|
||||
assertThat(EmailEffects.send(request().withIdempotencyKey("same"), "same").outcome()).isEqualTo(EmailEffectOutcome.QUEUED_OR_DUPLICATE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A suppressed audit row stays refused on replay, including duplicate results.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void suppressedRecipientsAreRefusedOnEveryHandoff() throws Exception
|
||||
{
|
||||
H2Support.insert(EmailTables.SUPPRESSION, new QRecord().withValue("address", "person@example.com")
|
||||
.withValue("scope", "ALL").withValue("reason", "HARD_BOUNCE"));
|
||||
var first = EmailEffects.send(request(), "suppressed");
|
||||
var second = EmailEffects.send(request(), "suppressed");
|
||||
assertThat(first.outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(first.message().status()).isEqualTo("SUPPRESSED");
|
||||
assertThat(second).isEqualTo(first);
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Never acknowledge an uncommitted or deferred handoff to the jobs dispatcher.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void callerTransactionsAreRefused() throws Exception
|
||||
{
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openFor(new InsertInput(EmailTables.MESSAGE)))
|
||||
{
|
||||
assertThat(EmailEffects.send(request().withTransaction(transaction, "rdbms"), "same-backend").outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
assertThat(EmailEffects.send(request().withTransaction(transaction, "other"), "other-backend").outcome()).isEqualTo(EmailEffectOutcome.REFUSED);
|
||||
transaction.commit();
|
||||
}
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Database uniqueness failures must escape, rather than become terminal refusals.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void storageFailurePropagatesAndRfcIdIsUnique() throws Exception
|
||||
{
|
||||
var first = EmailEffects.send(request(), "collision");
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", first.message().id()).withValue("idempotencyKey", "changed"));
|
||||
assertThatThrownBy(() -> EmailEffects.send(request(), "collision")).isInstanceOf(QException.class);
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1);
|
||||
assertThat(EmailEffects.findByIdempotencyKey("collision")).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Identity hashing is deterministic, ignores display names and has a safe fallback.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void generationAndUnkeyedMessages() throws Exception
|
||||
{
|
||||
assertThat(EmailService.messageId("key", "Display <sender@EXAMPLE.com>"))
|
||||
.isEqualTo(EmailService.messageId("key", "sender@example.com"));
|
||||
for(String from : new String[] {null, "local", "broken <", "sender@[]"})
|
||||
{
|
||||
assertThat(EmailService.messageId("key", from)).matches("<[0-9a-f]{32}@qbit-email\\.local>");
|
||||
}
|
||||
var first = EmailService.send(request());
|
||||
var second = EmailService.send(request());
|
||||
var rows = H2Support.all(EmailTables.MESSAGE);
|
||||
assertThat(rows).hasSize(2);
|
||||
assertThat(rows.get(0).getValueString("rfcMessageId")).matches("<[0-9a-f]{32}@example\\.com>")
|
||||
.isNotEqualTo(rows.get(1).getValueString("rfcMessageId"));
|
||||
assertThat(first.messageId()).isNotEqualTo(second.messageId());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
import java.util.List;
|
||||
import org.json.JSONArray;
|
||||
import org.json.JSONObject;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.message;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.provider;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Provider-specific normalization contracts use realistic fixture envelopes.
|
||||
*******************************************************************************/
|
||||
class EmailEventParserTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** All SES event types, with notificationType fallback and header correlation.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@CsvSource({"Delivery,DELIVERED", "Bounce,BOUNCED", "Complaint,COMPLAINED", "Reject,REJECTED", "DeliveryDelay,DEFERRED", "Open,OPENED", "Click,CLICKED"})
|
||||
void sesTypes(String raw, String expected) throws Exception
|
||||
{
|
||||
JSONObject envelope = new JSONObject(resource("/webhooks/ses-bounce.json"));
|
||||
JSONObject body = new JSONObject(envelope.getString("Message"));
|
||||
body.remove("eventType");
|
||||
body.put("notificationType", raw);
|
||||
String field = "DeliveryDelay".equals(raw) ? "deliveryDelay" : raw.toLowerCase();
|
||||
JSONObject data = new JSONObject().put("timestamp", NOW.toString()).put("link", "https://example.com/page").put("bounceType", "Transient");
|
||||
if("Bounce".equals(raw) || "Complaint".equals(raw))
|
||||
{
|
||||
data.put("Bounce".equals(raw) ? "bouncedRecipients" : "complainedRecipients", new JSONArray()
|
||||
.put(new JSONObject().put("emailAddress", "recipient@example.com")).put(new JSONObject().put("emailAddress", "second@example.com")));
|
||||
}
|
||||
body.put(field, data);
|
||||
envelope.put("Message", body.toString());
|
||||
var events = new EmailEventParser().parse("SES", envelope.toString(), NOW);
|
||||
assertThat(events).hasSize(2).allSatisfy(event ->
|
||||
{
|
||||
assertThat(event.type()).isEqualTo(expected);
|
||||
assertThat(event.occurred()).isEqualTo(NOW);
|
||||
assertThat(event.messageId()).isEqualTo("1");
|
||||
assertThat(event.url()).isEqualTo("CLICKED".equals(expected) ? "https://example.com/page" : null);
|
||||
});
|
||||
if("BOUNCED".equals(expected))
|
||||
{
|
||||
assertThat(events.getFirst().bounceType()).isEqualTo("SOFT");
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Postmark mappings keep subscriber reactivation out of suppression.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@CsvSource({"Delivery,DELIVERED", "Bounce,BOUNCED", "SpamComplaint,COMPLAINED", "Open,OPENED", "Click,CLICKED", "SubscriptionChange,UNSUBSCRIBED"})
|
||||
void postmarkTypes(String raw, String expected) throws Exception
|
||||
{
|
||||
JSONObject body = new JSONObject(resource("/webhooks/postmark-delivery.json")).put("RecordType", raw).put("SuppressSending", true)
|
||||
.put("OriginalLink", "https://example.com/page").put("Type", "Transient").put("ChangedAt", NOW.toString());
|
||||
var event = new EmailEventParser().parse("POSTMARK", body.toString(), NOW).getFirst();
|
||||
assertThat(event.type()).isEqualTo(expected);
|
||||
assertThat(event.occurred()).isEqualTo(NOW);
|
||||
assertThat(event.providerMessageId()).isEqualTo("provider-message-1");
|
||||
assertThat(event.url()).isEqualTo("CLICKED".equals(expected) ? "https://example.com/page" : null);
|
||||
if("BOUNCED".equals(expected))
|
||||
{
|
||||
assertThat(event.bounceType()).isEqualTo("SOFT");
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** SendGrid event mappings retain reason and provider-message prefix correlation.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@CsvSource({"delivered,DELIVERED", "bounce,BOUNCED", "dropped,REJECTED", "deferred,DEFERRED", "spamreport,COMPLAINED", "open,OPENED", "click,CLICKED", "unsubscribe,UNSUBSCRIBED"})
|
||||
void sendgridTypes(String raw, String expected) throws Exception
|
||||
{
|
||||
JSONObject body = new JSONArray(resource("/webhooks/sendgrid-batch.json")).getJSONObject(0).put("event", raw)
|
||||
.put("custom_args", new JSONObject().put("qqq_email_id", "25")).put("url", "https://example.com/page");
|
||||
var event = new EmailEventParser().parse("SENDGRID", new JSONArray().put(body).toString(), NOW).getFirst();
|
||||
assertThat(event.type()).isEqualTo(expected);
|
||||
assertThat(event.messageId()).isEqualTo("25");
|
||||
assertThat(event.providerMessageId()).isEqualTo("provider-message-1");
|
||||
assertThat(event.url()).isEqualTo("CLICKED".equals(expected) ? "https://example.com/page" : null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Mailgun's nested MIME id and both documented variable spellings correlate.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@CsvSource({"delivered,DELIVERED", "failed,BOUNCED", "complained,COMPLAINED", "opened,OPENED", "clicked,CLICKED", "unsubscribed,UNSUBSCRIBED"})
|
||||
void mailgunTypes(String raw, String expected) throws Exception
|
||||
{
|
||||
JSONObject body = new JSONObject(resource("/webhooks/mailgun-failed.json"));
|
||||
body.getJSONObject("event-data").put("event", raw).put("severity", "temporary").put("url", "https://example.com/page")
|
||||
.put("user-variables", new JSONObject().put("v:qqq_email_id", "25"));
|
||||
var event = new EmailEventParser().parse("MAILGUN", body.toString(), NOW).getFirst();
|
||||
assertThat(event.type()).isEqualTo(expected);
|
||||
assertThat(event.messageId()).isEqualTo("25");
|
||||
assertThat(event.providerMessageId()).isEqualTo("provider-message-1");
|
||||
assertThat(event.url()).isEqualTo("CLICKED".equals(expected) ? "https://example.com/page" : null);
|
||||
if("BOUNCED".equals(expected))
|
||||
{
|
||||
assertThat(event.bounceType()).isEqualTo("SOFT");
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Unknown events are ignored; incomplete known events fail for bounded retry.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unknownEventsAndMissingFields() throws Exception
|
||||
{
|
||||
var parser = new EmailEventParser();
|
||||
assertThat(parser.parse("SES", "{\"Type\":\"SubscriptionConfirmation\"}", NOW)).isEmpty();
|
||||
assertThat(parser.parse("SES", new JSONObject().put("Type", "Notification").put("Message", "{\"eventType\":\"Send\"}").toString(), NOW)).isEmpty();
|
||||
assertThat(parser.parse("POSTMARK", "{\"RecordType\":\"Unknown\"}", NOW)).isEmpty();
|
||||
assertThat(parser.parse("SENDGRID", "[{\"event\":\"processed\"}]", NOW)).isEmpty();
|
||||
assertThat(parser.parse("MAILGUN", "{\"event-data\":{\"event\":\"accepted\"}}", NOW)).isEmpty();
|
||||
assertThatThrownBy(() -> parser.parse("POSTMARK", "invalid", NOW)).isInstanceOf(RuntimeException.class);
|
||||
assertThatThrownBy(() -> parser.parse("OTHER", "{}", NOW)).isInstanceOf(IllegalArgumentException.class);
|
||||
for(String kind : List.of("SES", "POSTMARK", "SENDGRID", "MAILGUN"))
|
||||
{
|
||||
assertThatThrownBy(() -> parser.parse(kind, "{}", NOW)).isInstanceOf(RuntimeException.class);
|
||||
}
|
||||
assertThat(EmailEventParser.detail("550 \"name with spaces\"@example.com unavailable " + "x".repeat(600))).hasSize(500).doesNotContain("example.com", "name with spaces");
|
||||
assertThat(EmailEventParser.detail("bad person+tag@example.com and other@example.org")).isEqualTo("bad [address] and [address]");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Explicit uncertain bounce types remain uncertain, not hard suppression.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void undeterminedBounceTypes() throws Exception
|
||||
{
|
||||
var parser = new EmailEventParser();
|
||||
JSONObject ses = new JSONObject(resource("/webhooks/ses-bounce.json"));
|
||||
JSONObject inner = new JSONObject(ses.getString("Message"));
|
||||
inner.getJSONObject("bounce").put("bounceType", "Undetermined");
|
||||
assertThat(parser.parse("SES", ses.put("Message", inner.toString()).toString(), NOW).getFirst().bounceType()).isEqualTo("UNDETERMINED");
|
||||
JSONObject postmark = new JSONObject(resource("/webhooks/postmark-bounce.json")).put("Type", "Unknown");
|
||||
assertThat(parser.parse("POSTMARK", postmark.toString(), NOW).getFirst().bounceType()).isEqualTo("UNDETERMINED");
|
||||
JSONObject mailgun = new JSONObject(resource("/webhooks/mailgun-failed.json"));
|
||||
mailgun.getJSONObject("event-data").put("severity", "unknown");
|
||||
assertThat(parser.parse("MAILGUN", mailgun.toString(), NOW).getFirst().bounceType()).isEqualTo("UNDETERMINED");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,453 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Clock;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.H2Support;
|
||||
import com.kingsrook.qbits.email.ProcessRunner;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.processes.EmailProcesses;
|
||||
import com.kingsrook.qbits.email.support.EmailStore;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import org.json.JSONArray;
|
||||
import org.json.JSONObject;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.CsvSource;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.message;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.provider;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/*******************************************************************************
|
||||
** Durable processing, rollback, suppression and race acceptance on real H2.
|
||||
*******************************************************************************/
|
||||
class EmailInboxProcessorTest
|
||||
{
|
||||
private QInstance instance;
|
||||
|
||||
/*******************************************************************************
|
||||
** Use a fixed clock for retry and provider event times.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
instance = H2Support.setup(c -> c.withSecretResolver(name -> SECRET));
|
||||
EmailRuntime.setClock(Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Restore host-independent runtime state.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void teardown()
|
||||
{
|
||||
QContext.clear();
|
||||
EmailRuntime.setClock(null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Ingest the same authenticated payload boundary used by HTTP.
|
||||
*******************************************************************************/
|
||||
private static void receive(QRecord provider, String id, String body) throws Exception
|
||||
{
|
||||
assertThat(EmailTrackingStore.receive(provider, id, null, body.getBytes(StandardCharsets.UTF_8), null)).isEqualTo(EmailTrackingStore.Receipt.RECEIVED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** All provider fixtures produce hard-bounce suppression and masked diagnostics.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@CsvSource({"SES,ses-bounce.json,2", "POSTMARK,postmark-bounce.json,1", "SENDGRID,sendgrid-batch.json,2", "MAILGUN,mailgun-failed.json,1"})
|
||||
void realisticFixtures(String kind, String fixture, int count) throws Exception
|
||||
{
|
||||
QRecord provider = provider(kind);
|
||||
QRecord message = message(provider);
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", message.getValue("id")).withValue("toAddresses", "recipient@example.com, second@example.com"));
|
||||
receive(provider, "fixture", resource("/webhooks/" + fixture));
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isEqualTo(1);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(count);
|
||||
assertThat(H2Support.all(EmailTables.INBOX).getFirst().getValueString("status")).isEqualTo("PROCESSED");
|
||||
QRecord updated = EmailStore.get(EmailTables.MESSAGE, message.getValueInteger("id"), null);
|
||||
assertThat(updated.getValueString("status")).isEqualTo("BOUNCED");
|
||||
assertThat(updated.getValueInstant("bouncedDate")).isNotNull();
|
||||
for(QRecord suppression : H2Support.all(EmailTables.SUPPRESSION))
|
||||
{
|
||||
assertThat(suppression.getValueString("scope")).isEqualTo("ALL");
|
||||
assertThat(suppression.getValueString("reason")).isEqualTo("HARD_BOUNCE");
|
||||
assertThat(suppression.getValueInteger("sourceMessageId")).isEqualTo(message.getValueInteger("id"));
|
||||
assertThat(suppression.getValueInteger("sourceEventId")).isPositive();
|
||||
}
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).allSatisfy(event -> assertThat(event.getValueString("detail")).doesNotContain("recipient@example.com", "second@example.com"));
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isZero();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Soft bounces do not suppress; complaint dominates all later delivery events.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void statusOrderSuppressionUpsertsAndOverlappingBatches() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = message(provider);
|
||||
JSONObject event = new JSONObject().put("email", "Recipient@EXAMPLE.COM").put("qqq_email_id", message.getValueString("id"))
|
||||
.put("timestamp", NOW.getEpochSecond()).put("event", "bounce").put("type", "blocked").put("sg_event_id", "soft");
|
||||
receive(provider, "one", new JSONArray().put(event).toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE).getFirst().getValueString("status")).isEqualTo("SENT");
|
||||
event.put("type", "bounce").put("sg_event_id", "hard");
|
||||
receive(provider, "two", new JSONArray().put(event).toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
event.put("event", "spamreport").put("sg_event_id", "complaint");
|
||||
receive(provider, "three", new JSONArray().put(event).toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE).getFirst().getValueString("status")).isEqualTo("COMPLAINED");
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE).getFirst().getValueInstant("complainedDate")).isEqualTo(NOW);
|
||||
event.put("event", "delivered").put("sg_event_id", "late-delivery");
|
||||
JSONArray overlap = new JSONArray(resource("/webhooks/sendgrid-batch.json")).put(event);
|
||||
receive(provider, "four", overlap.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
receive(provider, "five", overlap.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.INBOX).getLast().getValueString("status")).isEqualTo("IGNORED");
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(6);
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE).getFirst().getValueString("status")).isEqualTo("COMPLAINED");
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION).getFirst().getValueString("reason")).isEqualTo("COMPLAINT");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Provider unsubscribes cannot suppress account/security mail.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"MARKETING", "NOTIFICATION", "TRANSACTIONAL", "SECURITY"})
|
||||
void categoryRule(String category) throws Exception
|
||||
{
|
||||
QRecord provider = provider("POSTMARK");
|
||||
QRecord message = message(provider);
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", message.getValue("id")).withValue("category", category));
|
||||
JSONObject event = new JSONObject(resource("/webhooks/postmark-delivery.json")).put("RecordType", "SubscriptionChange").put("SuppressSending", true);
|
||||
receive(provider, "unsubscribe", event.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
if(List.of("MARKETING", "NOTIFICATION").contains(category))
|
||||
{
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION).getFirst().getValueString("scope")).isEqualTo(category);
|
||||
}
|
||||
else
|
||||
{
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
}
|
||||
receive(provider, "reactivation", event.put("SuppressSending", false).toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Tracking flags apply before matching, and protected mail never stores tracking.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void opensAndClicksOnlyWhenEnabled() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = message(provider);
|
||||
JSONArray batch = new JSONArray();
|
||||
for(String type : List.of("open", "click"))
|
||||
{
|
||||
batch.put(new JSONObject().put("event", type).put("email", "recipient@example.com").put("qqq_email_id", message.getValue("id"))
|
||||
.put("sg_event_id", type).put("url", "https://example.com/link").put("timestamp", NOW.getEpochSecond()));
|
||||
}
|
||||
receive(provider, "off", batch.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
SystemWrites.updateOne(EmailTables.PROVIDER, new QRecord().withValue("id", provider.getValue("id")).withValue("trackOpens", true).withValue("trackClicks", true));
|
||||
receive(provider, "on", batch.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(2);
|
||||
assertThat(H2Support.all(EmailTables.EVENT).getFirst().getValueString("url")).isNull();
|
||||
assertThat(H2Support.all(EmailTables.EVENT).getLast().getValueString("url")).isEqualTo("https://example.com/link");
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", message.getValue("id")).withValue("phiApproved", true));
|
||||
batch.getJSONObject(0).put("sg_event_id", "protected-open");
|
||||
batch.getJSONObject(1).put("sg_event_id", "protected-click");
|
||||
receive(provider, "protected", batch.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(2);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Partial batch failure rolls back events, suppression and message status.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void retriesSixTimesThenReviewReprocess() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = message(provider);
|
||||
receive(provider, "retry", "[{not json");
|
||||
for(int i = 1; i <= 6; i++)
|
||||
{
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isZero();
|
||||
QRecord row = H2Support.all(EmailTables.INBOX).getFirst();
|
||||
assertThat(row.getValueInteger("attempts")).isEqualTo(i);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE).getFirst().getValueString("status")).isEqualTo("SENT");
|
||||
if(i < 6)
|
||||
{
|
||||
assertThat(row.getValueString("status")).isEqualTo("PENDING");
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isZero();
|
||||
EmailRuntime.setClock(Clock.fixed(row.getValueInstant("nextAttemptDate"), ZoneOffset.UTC));
|
||||
}
|
||||
}
|
||||
QRecord failed = H2Support.all(EmailTables.INBOX).getFirst();
|
||||
assertThat(failed.getValueString("status")).isEqualTo("FAILED");
|
||||
var lines = ProcessRunner.run(EmailProcesses.REPROCESS, failed.getValue("id"));
|
||||
assertThat(lines.review().toString()).contains("will be processed");
|
||||
assertThat(lines.result().toString()).contains("PENDING");
|
||||
assertThat(EmailInboxProcessor.reprocess(failed.getValueInteger("id"))).isFalse();
|
||||
assertThat(EmailInboxProcessor.reprocess(9999)).isFalse();
|
||||
SystemWrites.updateOne(EmailTables.INBOX, new QRecord().withValue("id", failed.getValue("id")).withValue("payloadJson", resource("/webhooks/sendgrid-batch.json")));
|
||||
new EmailProcesses.ScheduledStep().run(new RunBackendStepInput().withProcessName(EmailProcesses.PROCESS_WEBHOOKS), new RunBackendStepOutput());
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(2);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A foreign event never blocks a real hard bounce in the same SendGrid batch; the
|
||||
** foreign one retries, then the row finishes PROCESSED with a note, not FAILED.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unmatchedEventDoesNotBlockMatchedBounce() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = message(provider);
|
||||
JSONArray batch = new JSONArray(resource("/webhooks/sendgrid-batch.json"));
|
||||
batch.put(new JSONObject().put("email", "stranger@example.com").put("timestamp", NOW.getEpochSecond()).put("event", "bounce")
|
||||
.put("type", "bounce").put("sg_event_id", "foreign").put("sg_message_id", "someone-elses-message.filter009"));
|
||||
receive(provider, "mixed", batch.toString());
|
||||
for(int i = 1; i <= 6; i++)
|
||||
{
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isEqualTo(1);
|
||||
QRecord row = H2Support.all(EmailTables.INBOX).getFirst();
|
||||
assertThat(row.getValueInteger("attempts")).isEqualTo(i);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(2);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, message.getValueInteger("id"), null).getValueString("status")).isEqualTo("BOUNCED");
|
||||
if(i < 6)
|
||||
{
|
||||
assertThat(row.getValueString("status")).isEqualTo("PENDING");
|
||||
assertThat(row.getValueString("lastError")).isEqualTo("1 event did not match a message");
|
||||
EmailRuntime.setClock(Clock.fixed(row.getValueInstant("nextAttemptDate"), ZoneOffset.UTC));
|
||||
}
|
||||
}
|
||||
QRecord done = H2Support.all(EmailTables.INBOX).getFirst();
|
||||
assertThat(done.getValueString("status")).isEqualTo("PROCESSED");
|
||||
assertThat(done.getValueString("lastError")).isEqualTo("1 event did not match a message");
|
||||
assertThat(done.getValueInstant("processedDate")).isNotNull();
|
||||
assertThat(done.getValue("nextAttemptDate")).isNull();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Only a recipient of the matched message can be suppressed, and an SES bounce
|
||||
** without bouncedRecipients is skipped rather than widened to all destinations.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void suppressionOnlyForRecipientsOfTheMessage() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = message(provider);
|
||||
JSONObject event = new JSONObject().put("event", "bounce").put("type", "bounce").put("email", "intruder@example.com")
|
||||
.put("qqq_email_id", message.getValueString("id")).put("timestamp", NOW.getEpochSecond()).put("sg_event_id", "intruder");
|
||||
receive(provider, "intruder", new JSONArray().put(event).toString());
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isEqualTo(1);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
QRecord row = H2Support.all(EmailTables.INBOX).getFirst();
|
||||
assertThat(row.getValueString("status")).isEqualTo("IGNORED");
|
||||
assertThat(row.getValueString("lastError")).contains("1 event was skipped");
|
||||
assertThat(row.getValueString("lastError")).doesNotContain("intruder");
|
||||
|
||||
QRecord ses = provider("SES");
|
||||
QRecord sesMessage = message(ses);
|
||||
JSONObject envelope = new JSONObject(resource("/webhooks/ses-bounce.json"));
|
||||
JSONObject body = new JSONObject(envelope.getString("Message"));
|
||||
body.getJSONObject("mail").getJSONArray("headers").getJSONObject(0).put("value", sesMessage.getValueString("id"));
|
||||
body.getJSONObject("bounce").remove("bouncedRecipients");
|
||||
receive(ses, "no-recipients", envelope.put("Message", body.toString()).toString());
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isEqualTo(1);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, sesMessage.getValueInteger("id"), null).getValueString("status")).isEqualTo("SENT");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Our custom id cannot match another provider's message; callbacks may race SENT.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void providerBoundaryAndSendingRace() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = message(provider("POSTMARK"));
|
||||
JSONObject event = new JSONObject().put("event", "delivered").put("qqq_email_id", message.getValue("id"))
|
||||
.put("email", "recipient@example.com").put("sg_event_id", "cross-provider");
|
||||
receive(provider, "cross", new JSONArray().put(event).toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
QRecord actual = message(provider);
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", actual.getValue("id")).withValue("status", "SENDING"));
|
||||
receive(provider, "racing", new JSONArray().put(event.put("qqq_email_id", actual.getValue("id")).put("sg_event_id", "racing")).toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", actual.getValue("id")).withValue("status", "SENT"));
|
||||
EmailRuntime.setClock(Clock.fixed(NOW.plusSeconds(61), ZoneOffset.UTC));
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
assertThat(EmailStore.get(EmailTables.MESSAGE, actual.getValueInteger("id"), null).getValueString("status")).isEqualTo("DELIVERED");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Custom arguments support the entire integer primary-key range.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void fullIntegerMessageId() throws Exception
|
||||
{
|
||||
QRecord provider = provider("SENDGRID");
|
||||
QRecord message = SystemWrites.insertOne(EmailTables.MESSAGE, new QRecord().withValue("id", 2000000000)
|
||||
.withValue("providerId", provider.getValue("id")).withValue("status", "SENT").withValue("category", "NOTIFICATION")
|
||||
.withValue("toAddresses", "recipient@example.com"));
|
||||
assertThat(message.getValueInteger("id")).isEqualTo(2000000000);
|
||||
JSONObject event = new JSONObject().put("event", "delivered").put("qqq_email_id", "2000000000")
|
||||
.put("email", "recipient@example.com").put("sg_event_id", "large-message-id");
|
||||
receive(provider, "large", new JSONArray().put(event).toString());
|
||||
assertThat(new EmailInboxProcessor().runOnce()).isEqualTo(1);
|
||||
assertThat(H2Support.all(EmailTables.EVENT).getFirst().getValueInteger("messageId")).isEqualTo(2000000000);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Two processor replicas observe one receipt but only one applies it.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void concurrentProcessors() throws Exception
|
||||
{
|
||||
QRecord provider = provider("POSTMARK");
|
||||
message(provider);
|
||||
receive(provider, "race", resource("/webhooks/postmark-bounce.json"));
|
||||
try(var workers = Executors.newFixedThreadPool(2))
|
||||
{
|
||||
var tasks = new ArrayList<java.util.concurrent.Future<Integer>>();
|
||||
for(int i = 0; i < 2; i++)
|
||||
{
|
||||
tasks.add(workers.submit(() ->
|
||||
{
|
||||
QContext.init(instance, H2Support.admin());
|
||||
try
|
||||
{
|
||||
return (new EmailInboxProcessor().runOnce());
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
}));
|
||||
}
|
||||
assertThat(tasks.get(0).get(20, TimeUnit.SECONDS) + tasks.get(1).get(20, TimeUnit.SECONDS)).isEqualTo(1);
|
||||
}
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Core's optional alert sink receives a day-scoped warning after threshold.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void complaintAlertThreshold() throws Exception
|
||||
{
|
||||
QRecord provider = provider("POSTMARK");
|
||||
message(provider);
|
||||
Alerts.signals.clear();
|
||||
com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData.ofOrWithNew(instance)
|
||||
.withSink(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(Alerts.class));
|
||||
for(int i = 0; i < 5; i++)
|
||||
{
|
||||
JSONObject event = new JSONObject(resource("/webhooks/postmark-bounce.json")).put("RecordType", "SpamComplaint").put("ID", i);
|
||||
receive(provider, "complaint-" + i, event.toString());
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(Alerts.signals).hasSize(Math.max(0, i - 2));
|
||||
}
|
||||
assertThat(Alerts.signals).allSatisfy(signal ->
|
||||
{
|
||||
assertThat(signal.getAlertTypeName()).isEqualTo("email.complaint");
|
||||
assertThat(signal.getDedupKey()).startsWith("complaints:");
|
||||
});
|
||||
assertThat(Alerts.signals.get(0).getDedupKey()).isEqualTo(Alerts.signals.get(1).getDedupKey());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Captures content-free core alert signals.
|
||||
*******************************************************************************/
|
||||
public static class Alerts implements com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface
|
||||
{
|
||||
static final List<com.kingsrook.qqq.backend.core.alerts.AlertSignal> signals = new ArrayList<>();
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
|
||||
@Override
|
||||
public void raise(com.kingsrook.qqq.backend.core.alerts.AlertSignal signal)
|
||||
{
|
||||
signals.add(signal);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void resolve(com.kingsrook.qqq.backend.core.alerts.AlertResolution resolution)
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void heartbeat(String source, java.time.Instant timestamp)
|
||||
{
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,521 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.KeyPairGenerator;
|
||||
import java.security.Signature;
|
||||
import java.time.Clock;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Base64;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
import com.kingsrook.qbits.email.H2Support;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.render.EmailUnsubscribeLinks;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qbits.email.webhook.EmailWebhookRouteProvider;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.middleware.javalin.QJavalinMetaData;
|
||||
import io.javalin.Javalin;
|
||||
import org.json.JSONObject;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.TOPIC;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.certificateTime;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.message;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.provider;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.response;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signMailgun;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signSns;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Real random-port HTTP exercises the exact route provider and database boundary.
|
||||
*******************************************************************************/
|
||||
class EmailWebhookRoutesTest
|
||||
{
|
||||
private Javalin server;
|
||||
private HttpClient client;
|
||||
private QInstance instance;
|
||||
private final List<String> fetched = new java.util.concurrent.CopyOnWriteArrayList<>();
|
||||
private static final String BASE = "/custom/email";
|
||||
|
||||
/*******************************************************************************
|
||||
** Test the configurable path and injected SNS HTTP seam.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setup() throws Exception
|
||||
{
|
||||
instance = H2Support.setup(c -> c.withWebhookBasePath(BASE).withPublicBaseUrl("https://app.example.com").withSecretResolver(name -> SECRET));
|
||||
EmailRuntime.setClock(Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
String pem = resource("/sns-test/test-only-certificate.pem");
|
||||
var routes = new EmailWebhookRouteProvider(new SnsVerifier(request ->
|
||||
{
|
||||
fetched.add(request.uri().toString());
|
||||
return (response(request, 200, request.uri().getPath().endsWith(".pem") ? pem : "confirmed"));
|
||||
}));
|
||||
routes.setQInstance(instance);
|
||||
server = Javalin.create(routes::acceptJavalinConfig).start(0);
|
||||
client = HttpClient.newHttpClient();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Release every server, connection and thread-local test context.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void teardown()
|
||||
{
|
||||
server.stop();
|
||||
client.close();
|
||||
EmailRuntime.setClock(null);
|
||||
QContext.clear();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** HTTP request helper, with no application code bypass.
|
||||
*******************************************************************************/
|
||||
private HttpResponse<String> request(String path, String method, String body, String... headers) throws Exception
|
||||
{
|
||||
var builder = HttpRequest.newBuilder(URI.create("http://localhost:" + server.port() + BASE + path));
|
||||
if(headers.length > 0)
|
||||
{
|
||||
builder.headers(headers);
|
||||
}
|
||||
builder.method(method, body == null ? HttpRequest.BodyPublishers.noBody() : HttpRequest.BodyPublishers.ofString(body));
|
||||
return (client.send(builder.build(), HttpResponse.BodyHandlers.ofString()));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Fixed Basic auth used only by synthetic Postmark requests.
|
||||
*******************************************************************************/
|
||||
private static String auth()
|
||||
{
|
||||
return ("Basic " + Base64.getEncoder().encodeToString(("postmark:" + SECRET).getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** All status codes and streaming size caps before any authentication lookup.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void postmarkStatusesAndDedup() throws Exception
|
||||
{
|
||||
QRecord provider = provider("POSTMARK");
|
||||
String body = resource("/webhooks/postmark-delivery.json");
|
||||
assertThat(request("/postmark/postmark", "POST", body, "Authorization", auth()).body()).contains("received");
|
||||
assertThat(request("/postmark/postmark", "POST", body, "Authorization", auth()).body()).contains("duplicate");
|
||||
assertThat(H2Support.all(EmailTables.INBOX)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
assertThat(request("/postmark/postmark", "POST", body).statusCode()).isEqualTo(401);
|
||||
assertThat(request("/postmark/postmark", "POST", "invalid", "Authorization", auth()).statusCode()).isEqualTo(401);
|
||||
assertThat(request("/mailgun/postmark", "POST", body).statusCode()).isEqualTo(404);
|
||||
assertThat(request("/postmark/missing", "POST", body).statusCode()).isEqualTo(404);
|
||||
assertThat(request("/postmark/postmark", "POST", "x".repeat(1024 * 1024 + 1)).statusCode()).isEqualTo(413);
|
||||
var chunked = HttpRequest.newBuilder(URI.create("http://localhost:" + server.port() + BASE + "/postmark/postmark"))
|
||||
.POST(HttpRequest.BodyPublishers.ofInputStream(() -> new java.io.ByteArrayInputStream(new byte[1024 * 1024 + 1]))).build();
|
||||
assertThat(client.send(chunked, HttpResponse.BodyHandlers.ofString()).statusCode()).isEqualTo(413);
|
||||
EmailRuntime.config().withSecretResolver(name -> null);
|
||||
assertThat(request("/postmark/postmark", "POST", body).statusCode()).isEqualTo(503);
|
||||
SystemWrites.updateOne(EmailTables.PROVIDER, new QRecord().withValue("id", provider.getValue("id")).withValue("enabled", false).withValue("isDefault", false));
|
||||
assertThat(request("/postmark/postmark", "POST", body).statusCode()).isEqualTo(404);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Missing per-provider verification configuration is explicitly retryable.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"SES", "SENDGRID", "MAILGUN"})
|
||||
void missingSecretOrTopic(String kind) throws Exception
|
||||
{
|
||||
QRecord provider = provider(kind);
|
||||
if("SES".equals(kind))
|
||||
{
|
||||
SystemWrites.updateOne(EmailTables.PROVIDER, new QRecord().withValue("id", provider.getValue("id")).withValue("snsTopicArn", null));
|
||||
}
|
||||
EmailRuntime.config().withSecretResolver(name -> null);
|
||||
assertThat(request("/" + kind.toLowerCase() + "/" + kind.toLowerCase(), "POST", "{}").statusCode()).isEqualTo(503);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** DB failure is 503, and idempotent registration never adds duplicate routes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unavailableStoreAndRegistration() throws Exception
|
||||
{
|
||||
provider("POSTMARK");
|
||||
EmailWebhookRouteProvider.register(instance);
|
||||
EmailWebhookRouteProvider.register(instance);
|
||||
assertThat(QJavalinMetaData.ofOrWithNew(instance).getAdditionalRouteProviderReferences()).hasSize(1);
|
||||
var removed = instance.getTables().remove(EmailTables.INBOX);
|
||||
try
|
||||
{
|
||||
assertThat(request("/postmark/postmark", "POST", resource("/webhooks/postmark-delivery.json"), "Authorization", auth()).statusCode()).isEqualTo(503);
|
||||
}
|
||||
finally
|
||||
{
|
||||
instance.addTable(removed);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** SNS verifies before confirming, with a durable ignored control message.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void snsNotificationConfirmationAndTopicMismatch() throws Exception
|
||||
{
|
||||
provider("SES");
|
||||
EmailRuntime.setClock(Clock.fixed(certificateTime(), ZoneOffset.UTC));
|
||||
JSONObject payload = signSns(new JSONObject(resource("/webhooks/ses-bounce.json")));
|
||||
assertThat(request("/ses/ses", "POST", payload.toString()).statusCode()).isEqualTo(200);
|
||||
assertThat(request("/ses/ses", "POST", payload.toString()).body()).contains("duplicate");
|
||||
payload.put("MessageId", "subscription").put("Type", "SubscriptionConfirmation").put("Token", "test-token")
|
||||
.put("SubscribeURL", "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&Token=test");
|
||||
signSns(payload);
|
||||
assertThat(request("/ses/ses", "POST", payload.toString()).statusCode()).isEqualTo(200);
|
||||
assertThat(H2Support.all(EmailTables.INBOX).get(1).getValueString("status")).isEqualTo("IGNORED");
|
||||
assertThat(fetched).anyMatch(url -> url.contains("ConfirmSubscription"));
|
||||
assertThat(request("/ses/ses", "POST", signSns(payload.put("SubscribeURL", "http://evil.example/")).toString()).statusCode()).isEqualTo(401);
|
||||
assertThat(request("/ses/ses", "POST", signSns(payload.put("TopicArn", TOPIC + "other")).toString()).statusCode()).isEqualTo(401);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Signed raw SendGrid body is accepted and tampering leaves no extra receipt.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sendgridSignedBatch() throws Exception
|
||||
{
|
||||
provider("SENDGRID");
|
||||
var keys = KeyPairGenerator.getInstance("EC").generateKeyPair();
|
||||
EmailRuntime.config().withSecretResolver(name -> Base64.getEncoder().encodeToString(keys.getPublic().getEncoded()));
|
||||
String body = resource("/webhooks/sendgrid-batch.json");
|
||||
String timestamp = Long.toString(NOW.getEpochSecond());
|
||||
Signature signer = Signature.getInstance("SHA256withECDSA");
|
||||
signer.initSign(keys.getPrivate());
|
||||
signer.update((timestamp + body).getBytes(StandardCharsets.UTF_8));
|
||||
String signature = Base64.getEncoder().encodeToString(signer.sign());
|
||||
String[] headers = {"X-Twilio-Email-Event-Webhook-Signature", signature, "X-Twilio-Email-Event-Webhook-Timestamp", timestamp};
|
||||
assertThat(request("/sendgrid/sendgrid", "POST", body, headers).statusCode()).isEqualTo(200);
|
||||
assertThat(request("/sendgrid/sendgrid", "POST", body + " ", headers).statusCode()).isEqualTo(401);
|
||||
assertThat(request("/sendgrid/sendgrid", "POST", body, headers).body()).contains("duplicate");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Mailgun replay is 401, but a new signed nonce with an old event id is 200.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void mailgunReplayAndDuplicateEvent() throws Exception
|
||||
{
|
||||
provider("MAILGUN");
|
||||
JSONObject payload = signMailgun(new JSONObject(resource("/webhooks/mailgun-failed.json")), "nonce-one", NOW);
|
||||
assertThat(request("/mailgun/mailgun", "POST", payload.toString()).body()).contains("received");
|
||||
assertThat(request("/mailgun/mailgun", "POST", payload.toString()).statusCode()).isEqualTo(401);
|
||||
signMailgun(payload, "nonce-two", NOW);
|
||||
assertThat(request("/mailgun/mailgun", "POST", payload.toString()).body()).contains("duplicate");
|
||||
signMailgun(payload, "stale", NOW.minusSeconds(301));
|
||||
assertThat(request("/mailgun/mailgun", "POST", payload.toString()).statusCode()).isEqualTo(401);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Concurrent receipt dedup is durable across separate HTTP worker threads.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void concurrentDuplicates() throws Exception
|
||||
{
|
||||
provider("POSTMARK");
|
||||
String body = resource("/webhooks/postmark-bounce.json");
|
||||
try(var workers = Executors.newFixedThreadPool(2))
|
||||
{
|
||||
var first = workers.submit(() -> request("/postmark/postmark", "POST", body, "Authorization", auth()));
|
||||
var second = workers.submit(() -> request("/postmark/postmark", "POST", body, "Authorization", auth()));
|
||||
assertThat(first.get(20, TimeUnit.SECONDS).statusCode()).isEqualTo(200);
|
||||
assertThat(second.get(20, TimeUnit.SECONDS).statusCode()).isEqualTo(200);
|
||||
assertThat(H2Support.all(EmailTables.INBOX)).hasSize(1);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** GET never opts out, page content is escaped and one-click returns no body.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unsubscribeGetAndBothPosts() throws Exception
|
||||
{
|
||||
QRecord message = message(provider("POSTMARK"));
|
||||
QRecord brand = H2Support.insert(EmailTables.BRAND, new QRecord().withValue("name", "brand").withValue("label", "<script>Brand & Co</script>"));
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", message.getValue("id")).withValue("brandId", brand.getValue("id")));
|
||||
String token = token(message);
|
||||
var get = request("/unsubscribe/" + token, "GET", null);
|
||||
assertThat(get.statusCode()).isEqualTo(200);
|
||||
assertThat(get.body()).contains("<script>Brand & Co</script>", "Unsubscribe from Marketing", "color-scheme:light dark", "prefers-color-scheme:dark", "<form method=\"post\">")
|
||||
.doesNotContain("<script>", "recipient@example.com", "http://", "https://");
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
assertThat(request("/unsubscribe/" + token, "POST", "anything").statusCode()).isEqualTo(400);
|
||||
var click = request("/unsubscribe/" + token, "POST", "List-Unsubscribe=One-Click", "Content-Type", "application/x-www-form-urlencoded");
|
||||
assertThat(click.statusCode()).isEqualTo(200);
|
||||
assertThat(click.body()).isEmpty();
|
||||
assertThat(request("/unsubscribe/" + token, "POST", "confirm=unsubscribe").body()).contains("You are unsubscribed").doesNotContain("<form");
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION).getFirst().getValueString("scope")).isEqualTo("MARKETING");
|
||||
assertThat(H2Support.all(EmailTables.MESSAGE).getFirst().getValueString("status")).isEqualTo("SENT");
|
||||
assertThat(request("/unsubscribe/" + token, "POST", "x".repeat(1024 * 1024 + 1)).statusCode()).isEqualTo(413);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Real enqueue uses the existing signer and preserves the ad-hoc brand label.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void queuedAdHocBrandAndSigner() throws Exception
|
||||
{
|
||||
provider("POSTMARK");
|
||||
QRecord brand = H2Support.insert(EmailTables.BRAND, new QRecord().withValue("name", "chosen-brand").withValue("label", "Chosen Brand"));
|
||||
var result = com.kingsrook.qbits.email.outbox.EmailService.send(new com.kingsrook.qbits.email.outbox.EmailRequest()
|
||||
.withTo("recipient@example.com").withSubject("Notice").withMarkdownBody("A notice").withBrandName("chosen-brand")
|
||||
.withCategory(com.kingsrook.qbits.email.outbox.EmailCategory.NOTIFICATION));
|
||||
assertThat(result.status()).isEqualTo(com.kingsrook.qbits.email.outbox.EmailSendResult.Status.QUEUED);
|
||||
QRecord stored = H2Support.all(EmailTables.MESSAGE).getFirst();
|
||||
assertThat(stored.getValueInteger("brandId")).isEqualTo(brand.getValueInteger("id"));
|
||||
String url = stored.getValueString("unsubscribeUrl");
|
||||
String path = "/unsubscribe/" + url.substring(url.lastIndexOf('/') + 1);
|
||||
assertThat(request(path, "GET", null).body()).contains("Notification email from Chosen Brand?");
|
||||
assertThat(request(path, "POST", "List-Unsubscribe=One-Click").statusCode()).isEqualTo(200);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION).getFirst().getValueString("scope")).isEqualTo("NOTIFICATION");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** One signed mailbox is suppressed, while repeat POST restores its opt-out.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unsubscribeSingleRecipientAndRepeatedSuppression() throws Exception
|
||||
{
|
||||
QRecord provider = provider("POSTMARK");
|
||||
QRecord message = message(provider);
|
||||
message.withValue("toAddresses", "\"Recipient, One\" <Recipient@example.com>").withValue("category", "NOTIFICATION");
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, message);
|
||||
QRecord other = message(provider);
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, other.withValue("toAddresses", "other@example.com"));
|
||||
String signed = token(message);
|
||||
assertThat(EmailUnsubscribeLinks.verify(signed).getString("addressHash"))
|
||||
.isEqualTo(WebhookVerification.hash("recipient@example.com".getBytes(StandardCharsets.UTF_8)));
|
||||
String path = "/unsubscribe/" + signed;
|
||||
assertThat(request(path, "POST", "confirm=unsubscribe").statusCode()).isEqualTo(200);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.getValueString("scope")).isEqualTo("NOTIFICATION");
|
||||
assertThat(row.getValueString("address")).isEqualTo("recipient@example.com");
|
||||
});
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
SystemWrites.delete(EmailTables.SUPPRESSION, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter());
|
||||
assertThat(request(path, "POST", "List-Unsubscribe=One-Click").statusCode()).isEqualTo(200);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).hasSize(1);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Legacy single-recipient links keep working; legacy multi-recipient links fail closed.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void legacyUnsubscribeRequiresSingleRecipient() throws Exception
|
||||
{
|
||||
QRecord message = message(provider("POSTMARK"));
|
||||
String safe = legacyToken(message);
|
||||
assertThat(request("/unsubscribe/" + safe, "GET", null).statusCode()).isEqualTo(200);
|
||||
for(String field : List.of("toAddresses", "ccAddresses", "bccAddresses"))
|
||||
{
|
||||
QRecord changed = new QRecord().withValue("id", message.getValue("id")).withValue("toAddresses", "recipient@example.com")
|
||||
.withValue("ccAddresses", null).withValue("bccAddresses", null)
|
||||
.withValue(field, "toAddresses".equals(field) ? "recipient@example.com, other@example.com" : "other@example.com");
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, changed);
|
||||
changed.withValue("category", "MARKETING");
|
||||
String legacy = legacyToken(changed);
|
||||
for(String signed : List.of(safe, legacy))
|
||||
{
|
||||
assertThat(request("/unsubscribe/" + signed, "GET", null).statusCode()).isEqualTo(404);
|
||||
assertThat(request("/unsubscribe/" + signed, "POST", "List-Unsubscribe=One-Click").statusCode()).isEqualTo(404);
|
||||
}
|
||||
}
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
assertThat(H2Support.all(EmailTables.EVENT)).isEmpty();
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, message.withValue("ccAddresses", null).withValue("bccAddresses", null));
|
||||
assertThat(request("/unsubscribe/" + safe, "POST", "List-Unsubscribe=One-Click").statusCode()).isEqualTo(200);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).hasSize(1);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Reproduce the old signature without relying on the current signer's validation.
|
||||
*******************************************************************************/
|
||||
private static String legacyToken(QRecord message) throws Exception
|
||||
{
|
||||
String envelope = message.getValueString("toAddresses") + "," + message.getValueString("ccAddresses") + "," + message.getValueString("bccAddresses");
|
||||
JSONObject payload = new JSONObject().put("messageId", message.getValueString("id")).put("category", message.getValueString("category"))
|
||||
.put("addressHash", WebhookVerification.hash(envelope.getBytes(StandardCharsets.UTF_8)));
|
||||
String signed = "v1." + Base64.getUrlEncoder().withoutPadding().encodeToString(payload.toString().getBytes(StandardCharsets.UTF_8));
|
||||
javax.crypto.Mac mac = javax.crypto.Mac.getInstance("HmacSHA256");
|
||||
mac.init(new javax.crypto.spec.SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
return (signed + "." + Base64.getUrlEncoder().withoutPadding().encodeToString(mac.doFinal(signed.getBytes(StandardCharsets.UTF_8))));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Signer and receiver reject non-opt-out categories, including valid HMACs.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void invalidAndTransactionalTokens() throws Exception
|
||||
{
|
||||
QRecord message = message(provider("POSTMARK"));
|
||||
String token = token(message);
|
||||
assertThat(request("/unsubscribe/bad-token", "GET", null).statusCode()).isEqualTo(404);
|
||||
assertThat(request("/unsubscribe/" + token + "x", "POST", "List-Unsubscribe=One-Click").statusCode()).isEqualTo(404);
|
||||
for(String category : List.of("TRANSACTIONAL", "SECURITY"))
|
||||
{
|
||||
assertThat(EmailUnsubscribeLinks.create("1", "recipient@example.com", category)).isNull();
|
||||
JSONObject data = EmailUnsubscribeLinks.verify(token).put("category", category);
|
||||
String signed = "v1." + Base64.getUrlEncoder().withoutPadding().encodeToString(data.toString().getBytes(StandardCharsets.UTF_8));
|
||||
javax.crypto.Mac mac = javax.crypto.Mac.getInstance("HmacSHA256");
|
||||
mac.init(new javax.crypto.spec.SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
String forged = signed + "." + Base64.getUrlEncoder().withoutPadding().encodeToString(mac.doFinal(signed.getBytes(StandardCharsets.UTF_8)));
|
||||
assertThat(request("/unsubscribe/" + forged, "POST", "List-Unsubscribe=One-Click").statusCode()).isEqualTo(404);
|
||||
}
|
||||
SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", message.getValue("id")).withValue("toAddresses", "changed@example.com"));
|
||||
assertThat(request("/unsubscribe/" + token, "GET", null).statusCode()).isEqualTo(404);
|
||||
assertThat(H2Support.all(EmailTables.SUPPRESSION)).isEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Logs at normal operational levels never contain addresses or payload bodies.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void logsContainNoAddresses() throws Exception
|
||||
{
|
||||
QRecord provider = provider("POSTMARK");
|
||||
message(provider);
|
||||
List<String> logs = new java.util.concurrent.CopyOnWriteArrayList<>();
|
||||
var capture = new org.apache.logging.log4j.core.appender.AbstractAppender("tracking-privacy", null, null, true, null)
|
||||
{
|
||||
@Override
|
||||
public void append(org.apache.logging.log4j.core.LogEvent event)
|
||||
{
|
||||
// Match log4j2-test.xml: core's explicitly OFF diagnostics are not output.
|
||||
if(event.getLevel() == org.apache.logging.log4j.Level.OFF)
|
||||
{
|
||||
return;
|
||||
}
|
||||
logs.add(event.getMessage().getFormattedMessage());
|
||||
if(event.getThrown() != null)
|
||||
{
|
||||
logs.add(event.getThrown().toString());
|
||||
}
|
||||
}
|
||||
};
|
||||
capture.start();
|
||||
var context = (org.apache.logging.log4j.core.LoggerContext) org.apache.logging.log4j.LogManager.getContext(false);
|
||||
var root = context.getConfiguration().getRootLogger();
|
||||
var previousLevel = root.getLevel();
|
||||
root.addAppender(capture, org.apache.logging.log4j.Level.ALL, null);
|
||||
root.setLevel(org.apache.logging.log4j.Level.INFO);
|
||||
context.updateLoggers();
|
||||
try
|
||||
{
|
||||
String body = resource("/webhooks/postmark-bounce.json");
|
||||
assertThat(request("/postmark/postmark", "POST", body, "Authorization", auth()).statusCode()).isEqualTo(200);
|
||||
assertThat(request("/postmark/postmark", "POST", body, "Authorization", auth()).statusCode()).isEqualTo(200);
|
||||
assertThat(request("/postmark/postmark", "POST", body).statusCode()).isEqualTo(401);
|
||||
new EmailInboxProcessor().runOnce();
|
||||
String broken = new JSONObject(body).put("ID", 999).put("MessageID", "missing").toString();
|
||||
assertThat(request("/postmark/postmark", "POST", broken, "Authorization", auth()).statusCode()).isEqualTo(200);
|
||||
new EmailInboxProcessor().runOnce();
|
||||
assertThat(String.join("\n", logs)).doesNotContain("recipient@example.com", "sender@example.com", SECRET, "RecordType", "550 mailbox");
|
||||
assertThat(H2Support.all(EmailTables.INBOX).getLast().getValueString("lastError")).doesNotContain("@", "missing");
|
||||
}
|
||||
finally
|
||||
{
|
||||
root.removeAppender(capture.getName());
|
||||
root.setLevel(previousLevel);
|
||||
context.updateLoggers();
|
||||
capture.stop();
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Path validation excludes UI API routes and route metacharacters.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void routeConfigurationValidation()
|
||||
{
|
||||
for(String path : List.of("relative", "/qqq/v1", "/qqq/v1/email", "/hooks/{param}", "//hooks", "/hooks/"))
|
||||
{
|
||||
List<String> errors = new ArrayList<>();
|
||||
EmailRuntime.config().withWebhookBasePath(path).validate(instance, errors);
|
||||
assertThat(errors).anyMatch(error -> error.contains("webhookBasePath"));
|
||||
}
|
||||
List<String> errors = new ArrayList<>();
|
||||
EmailRuntime.config().withWebhookBasePath("/qqq/hooks/email").validate(instance, errors);
|
||||
assertThat(errors).isEmpty();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Both system tables are readable by viewers and reject direct user writes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void systemTablePermissions() throws Exception
|
||||
{
|
||||
QContext.getQSession().setPermissions(java.util.Set.of("email.view"));
|
||||
var checker = new com.kingsrook.qbits.email.permissions.EmailPermissionChecker();
|
||||
for(String table : List.of(EmailTables.EVENT, EmailTables.INBOX))
|
||||
{
|
||||
var metadata = instance.getTable(table);
|
||||
checker.checkPermissionsThrowing(new com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput(table), metadata);
|
||||
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput(table), metadata))
|
||||
.isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class);
|
||||
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput(table), metadata))
|
||||
.isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class);
|
||||
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput(table), metadata))
|
||||
.isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class);
|
||||
}
|
||||
assertThat(instance.getTable(EmailTables.PROVIDER).getSections()).anyMatch(section -> (EmailTables.INBOX + "_children").equals(section.getWidgetName()));
|
||||
assertThat(instance.getTable(EmailTables.MESSAGE).getSections()).anyMatch(section -> (EmailTables.EVENT + "_children").equals(section.getWidgetName()));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sign the same single To mailbox as enqueue.
|
||||
*******************************************************************************/
|
||||
private static String token(QRecord message)
|
||||
{
|
||||
String url = EmailUnsubscribeLinks.create(message.getValueString("id"), message.getValueString("toAddresses"), message.getValueString("category"));
|
||||
assertThat(url).startsWith("https://app.example.com" + BASE + "/unsubscribe/");
|
||||
return (url.substring(url.lastIndexOf('/') + 1));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import javax.net.ssl.SSLSession;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpHeaders;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.KeyFactory;
|
||||
import java.security.PrivateKey;
|
||||
import java.security.Signature;
|
||||
import java.security.cert.CertificateFactory;
|
||||
import java.security.cert.X509Certificate;
|
||||
import java.security.spec.PKCS8EncodedKeySpec;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import com.kingsrook.qbits.email.H2Support;
|
||||
import com.kingsrook.qbits.email.metadata.EmailTables;
|
||||
import com.kingsrook.qbits.email.support.SystemWrites;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.json.JSONObject;
|
||||
|
||||
/*******************************************************************************
|
||||
** Synthetic provider and crypto fixtures, always non-production.
|
||||
*******************************************************************************/
|
||||
public final class TrackingTestSupport
|
||||
{
|
||||
public static final Instant NOW = Instant.parse("2026-10-13T12:05:00Z");
|
||||
public static final String SECRET = "test-only-webhook-secret";
|
||||
public static final String TOPIC = "arn:aws:sns:us-east-1:123456789012:email-events";
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private TrackingTestSupport()
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Load a synthetic JSON or test-only PEM fixture from the classpath.
|
||||
*******************************************************************************/
|
||||
public static String resource(String name) throws Exception
|
||||
{
|
||||
try(var stream = TrackingTestSupport.class.getResourceAsStream(name))
|
||||
{
|
||||
return (new String(stream.readAllBytes(), StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure a fully valid provider row without resolving its sending key.
|
||||
*******************************************************************************/
|
||||
public static QRecord provider(String kind) throws Exception
|
||||
{
|
||||
return (H2Support.insert(EmailTables.PROVIDER, new QRecord().withValue("name", kind.toLowerCase(java.util.Locale.ROOT))
|
||||
.withValue("kind", kind).withValue("enabled", true).withValue("isDefault", H2Support.all(EmailTables.PROVIDER).isEmpty())
|
||||
.withValue("fromAddress", "sender@example.com").withValue("sendingDomain", "example.com").withValue("region", "SES".equals(kind) ? "us-east-1" : "US")
|
||||
.withValue("secretEnvName", switch(kind)
|
||||
{
|
||||
case "POSTMARK" -> "POSTMARK_SERVER_TOKEN";
|
||||
case "SENDGRID" -> "SENDGRID_API_KEY";
|
||||
case "MAILGUN" -> "MAILGUN_API_KEY";
|
||||
default -> null;
|
||||
})
|
||||
.withValue("webhookSecretEnvName", switch(kind)
|
||||
{
|
||||
case "POSTMARK" -> "POSTMARK_WEBHOOK_PASSWORD";
|
||||
case "SENDGRID" -> "SENDGRID_WEBHOOK_PUBLIC_KEY";
|
||||
case "MAILGUN" -> "MAILGUN_WEBHOOK_SIGNING_KEY";
|
||||
default -> null;
|
||||
})
|
||||
.withValue("snsTopicArn", TOPIC)));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Persist an accepted outbox row through the same authorized scope as delivery.
|
||||
*******************************************************************************/
|
||||
public static QRecord message(QRecord provider) throws Exception
|
||||
{
|
||||
return (SystemWrites.insertOne(EmailTables.MESSAGE, new QRecord().withValue("providerId", provider.getValue("id"))
|
||||
.withValue("providerMessageId", "provider-message-1").withValue("status", "SENT").withValue("category", "MARKETING")
|
||||
.withValue("toAddresses", "recipient@example.com").withValue("sentDate", NOW.minusSeconds(600))));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test certificate controls the verification date independent of execution day.
|
||||
*******************************************************************************/
|
||||
public static Instant certificateTime() throws Exception
|
||||
{
|
||||
try(var stream = TrackingTestSupport.class.getResourceAsStream("/sns-test/test-only-certificate.pem"))
|
||||
{
|
||||
var cert = (X509Certificate) CertificateFactory.getInstance("X.509").generateCertificate(stream);
|
||||
return (cert.getNotBefore().toInstant().plusSeconds(60));
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sign independently using AWS's required order, not the implementation helper.
|
||||
*******************************************************************************/
|
||||
public static JSONObject signSns(JSONObject payload) throws Exception
|
||||
{
|
||||
return (signSns(payload, certificateTime()));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sign with an explicit envelope Timestamp (the verifier enforces freshness).
|
||||
*******************************************************************************/
|
||||
public static JSONObject signSns(JSONObject payload, Instant timestamp) throws Exception
|
||||
{
|
||||
payload.put("Timestamp", timestamp.toString());
|
||||
String pem = resource("/sns-test/test-only-private-key.pem").replaceAll("-----[^-]+-----", "").replaceAll("\\s", "");
|
||||
PrivateKey key = KeyFactory.getInstance("RSA").generatePrivate(new PKCS8EncodedKeySpec(Base64.getDecoder().decode(pem)));
|
||||
List<String> fields = "Notification".equals(payload.getString("Type")) ? List.of("Message", "MessageId", "Subject", "Timestamp", "TopicArn", "Type")
|
||||
: List.of("Message", "MessageId", "SubscribeURL", "Timestamp", "Token", "TopicArn", "Type");
|
||||
StringBuilder canonical = new StringBuilder();
|
||||
for(String field : fields)
|
||||
{
|
||||
if(payload.has(field))
|
||||
{
|
||||
canonical.append(field).append('\n').append(payload.getString(field)).append('\n');
|
||||
}
|
||||
}
|
||||
Signature signer = Signature.getInstance("1".equals(payload.getString("SignatureVersion")) ? "SHA1withRSA" : "SHA256withRSA");
|
||||
signer.initSign(key);
|
||||
signer.update(canonical.toString().getBytes(StandardCharsets.UTF_8));
|
||||
return (payload.put("Signature", Base64.getEncoder().encodeToString(signer.sign())));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Sign Mailgun's timestamp and nonce with a clearly test-only HMAC key.
|
||||
*******************************************************************************/
|
||||
public static JSONObject signMailgun(JSONObject body, String nonce, Instant now) throws Exception
|
||||
{
|
||||
String timestamp = Long.toString(now.getEpochSecond());
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(SECRET.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
|
||||
body.put("signature", new JSONObject().put("timestamp", timestamp).put("token", nonce)
|
||||
.put("signature", java.util.HexFormat.of().formatHex(mac.doFinal((timestamp + nonce).getBytes(StandardCharsets.UTF_8)))));
|
||||
return (body);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Memory-only HTTP response for the existing EmailHttp seam.
|
||||
*******************************************************************************/
|
||||
public static HttpResponse<String> response(HttpRequest request, int status, String body)
|
||||
{
|
||||
return (new HttpResponse<>()
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public int statusCode()
|
||||
{
|
||||
return (status);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public HttpRequest request()
|
||||
{
|
||||
return (request);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public Optional<HttpResponse<String>> previousResponse()
|
||||
{
|
||||
return (Optional.empty());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public HttpHeaders headers()
|
||||
{
|
||||
return (HttpHeaders.of(Map.of(), (a, b) -> true));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String body()
|
||||
{
|
||||
return (body);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public Optional<SSLSession> sslSession()
|
||||
{
|
||||
return (Optional.empty());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public URI uri()
|
||||
{
|
||||
return (request.uri());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Test adapter implementation.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public java.net.http.HttpClient.Version version()
|
||||
{
|
||||
return (java.net.http.HttpClient.Version.HTTP_1_1);
|
||||
}
|
||||
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.email.tracking;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.KeyPairGenerator;
|
||||
import java.security.Signature;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import org.json.JSONObject;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.TOPIC;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.certificateTime;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.response;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signMailgun;
|
||||
import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signSns;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/*******************************************************************************
|
||||
** Crypto boundaries use genuine signatures and a public test-only RSA fixture.
|
||||
*******************************************************************************/
|
||||
class WebhookVerifiersTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Postmark's fixed user and constant-time credential verification.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void postmarkValidTamperedMissing()
|
||||
{
|
||||
var verifier = new PostmarkVerifier();
|
||||
String auth = "Basic " + Base64.getEncoder().encodeToString(("postmark:" + SECRET).getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(verifier.verify(auth, SECRET)).isTrue();
|
||||
assertThat(verifier.verify(auth, "wrong")).isFalse();
|
||||
assertThat(verifier.verify(auth, null)).isFalse();
|
||||
assertThat(verifier.verify(auth, " ")).isFalse();
|
||||
assertThat(verifier.verify(null, SECRET)).isFalse();
|
||||
assertThat(verifier.verify("Bearer abc", SECRET)).isFalse();
|
||||
assertThat(verifier.verify("Basic ?", SECRET)).isFalse();
|
||||
assertThat(verifier.verify("Basic " + Base64.getEncoder().encodeToString(("other:" + SECRET).getBytes(StandardCharsets.UTF_8)), SECRET)).isFalse();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Genuine ECDSA validates unchanged bytes, timestamp freshness and DER key.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sendgridValidTamperedStaleMissing() throws Exception
|
||||
{
|
||||
var keys = KeyPairGenerator.getInstance("EC").generateKeyPair();
|
||||
String publicKey = Base64.getEncoder().encodeToString(keys.getPublic().getEncoded());
|
||||
byte[] body = "[{\"event\":\"delivered\"}]".getBytes(StandardCharsets.UTF_8);
|
||||
String timestamp = Long.toString(NOW.getEpochSecond());
|
||||
Signature signer = Signature.getInstance("SHA256withECDSA");
|
||||
signer.initSign(keys.getPrivate());
|
||||
signer.update(timestamp.getBytes(StandardCharsets.UTF_8));
|
||||
signer.update(body);
|
||||
String signed = Base64.getEncoder().encodeToString(signer.sign());
|
||||
var verifier = new SendGridVerifier();
|
||||
assertThat(verifier.verify(body, signed, timestamp, publicKey, NOW)).isTrue();
|
||||
assertThat(verifier.verify("tampered".getBytes(StandardCharsets.UTF_8), signed, timestamp, publicKey, NOW)).isFalse();
|
||||
assertThat(verifier.verify(body, signed, timestamp, publicKey, NOW.plusSeconds(301))).isFalse();
|
||||
assertThat(verifier.verify(body, signed, timestamp, publicKey, NOW.minusSeconds(301))).isFalse();
|
||||
assertThat(verifier.verify(body, signed, "9223372036854775807", publicKey, NOW)).isFalse();
|
||||
assertThat(verifier.verify(body, signed, timestamp, null, NOW)).isFalse();
|
||||
assertThat(verifier.verify(body, null, timestamp, publicKey, NOW)).isFalse();
|
||||
assertThat(verifier.verify(body, "?", timestamp, publicKey, NOW)).isFalse();
|
||||
assertThat(verifier.verify(body, signed, timestamp, "?", NOW)).isFalse();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Mailgun validates the signed nonce, not its unsigned event body.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void mailgunValidTamperedStaleMissing() throws Exception
|
||||
{
|
||||
var verifier = new MailgunVerifier();
|
||||
JSONObject data = signMailgun(new JSONObject(), "test-nonce", NOW);
|
||||
assertThat(verifier.verify(data, SECRET, NOW)).isTrue();
|
||||
assertThat(verifier.verify(data, SECRET, NOW.plusSeconds(301))).isFalse();
|
||||
assertThat(verifier.verify(data, SECRET, NOW.minusSeconds(301))).isFalse();
|
||||
assertThat(verifier.verify(data, null, NOW)).isFalse();
|
||||
assertThat(verifier.verify(data, "wrong", NOW)).isFalse();
|
||||
data.getJSONObject("signature").put("token", "changed");
|
||||
assertThat(verifier.verify(data, SECRET, NOW)).isFalse();
|
||||
assertThat(verifier.verify(new JSONObject(), SECRET, NOW)).isFalse();
|
||||
data.getJSONObject("signature").put("token", "");
|
||||
assertThat(verifier.verify(data, SECRET, NOW)).isFalse();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Both SNS algorithms, optional Subject and certificate caching.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"1", "2"})
|
||||
void snsSignatureAndCache(String version) throws Exception
|
||||
{
|
||||
String pem = resource("/sns-test/test-only-certificate.pem");
|
||||
AtomicInteger requests = new AtomicInteger();
|
||||
var verifier = new SnsVerifier(request ->
|
||||
{
|
||||
requests.incrementAndGet();
|
||||
return (response(request, 200, pem));
|
||||
});
|
||||
JSONObject data = signSns(new JSONObject(resource("/webhooks/ses-bounce.json")).put("SignatureVersion", version));
|
||||
Instant now = certificateTime();
|
||||
assertThat(verifier.verify(data, TOPIC, now)).isTrue();
|
||||
assertThat(verifier.verify(signSns(data.put("Subject", "Test subject")), TOPIC, now)).isTrue();
|
||||
assertThat(requests.get()).isEqualTo(1);
|
||||
assertThat(verifier.verify(data, TOPIC + "other", now)).isFalse();
|
||||
assertThat(verifier.verify(data, null, now)).isFalse();
|
||||
data.put("Message", "tampered");
|
||||
assertThat(verifier.verify(data, TOPIC, now)).isFalse();
|
||||
data.put("SignatureVersion", "3");
|
||||
assertThat(verifier.verify(data, TOPIC, now)).isFalse();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Notifications outside the one hour past / five minute future window are refused
|
||||
** before any certificate fetch.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void snsTimestampWindow() throws Exception
|
||||
{
|
||||
AtomicInteger requests = new AtomicInteger();
|
||||
String pem = resource("/sns-test/test-only-certificate.pem");
|
||||
var verifier = new SnsVerifier(request ->
|
||||
{
|
||||
requests.incrementAndGet();
|
||||
return (response(request, 200, pem));
|
||||
});
|
||||
Instant now = certificateTime();
|
||||
JSONObject base = new JSONObject(resource("/webhooks/ses-bounce.json"));
|
||||
assertThat(verifier.verify(signSns(new JSONObject(base.toString()), now.minusSeconds(3601)), TOPIC, now)).isFalse();
|
||||
assertThat(verifier.verify(signSns(new JSONObject(base.toString()), now.plusSeconds(301)), TOPIC, now)).isFalse();
|
||||
assertThat(requests.get()).isZero();
|
||||
assertThat(verifier.verify(signSns(new JSONObject(base.toString()), now.minusSeconds(3599)), TOPIC, now)).isTrue();
|
||||
assertThat(verifier.verify(signSns(new JSONObject(base.toString()), now.plusSeconds(299)), TOPIC, now)).isTrue();
|
||||
assertThat(verifier.verify(signSns(new JSONObject(base.toString())).put("Timestamp", "yesterday"), TOPIC, now)).isFalse();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A failed certificate fetch is remembered per URL for five minutes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void snsCachesCertificateFetchFailures() throws Exception
|
||||
{
|
||||
AtomicInteger requests = new AtomicInteger();
|
||||
var verifier = new SnsVerifier(request ->
|
||||
{
|
||||
requests.incrementAndGet();
|
||||
return (response(request, 503, "down"));
|
||||
});
|
||||
Instant now = certificateTime();
|
||||
JSONObject data = signSns(new JSONObject(resource("/webhooks/ses-bounce.json")));
|
||||
assertThatThrownBy(() -> verifier.verify(data, TOPIC, now)).isInstanceOf(java.io.IOException.class);
|
||||
assertThatThrownBy(() -> verifier.verify(data, TOPIC, now.plusSeconds(299))).isInstanceOf(java.io.IOException.class);
|
||||
assertThat(requests.get()).isEqualTo(1);
|
||||
assertThatThrownBy(() -> verifier.verify(signSns(new JSONObject(data.toString()), now.plusSeconds(300)), TOPIC, now.plusSeconds(300))).isInstanceOf(java.io.IOException.class);
|
||||
assertThat(requests.get()).isEqualTo(2);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Strict AWS URL host, scheme, path and authority before network access.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@ValueSource(strings = {"http://sns.us-east-1.amazonaws.com/cert.pem", "https://evil.example/cert.pem", "https://sns.us-east-1.amazonaws.com/cert.txt",
|
||||
"https://sns.us-east-1.amazonaws.com.evil.example/cert.pem", "https://user@sns.us-east-1.amazonaws.com/cert.pem", "https://sns.us-east-1.amazonaws.com:8443/cert.pem",
|
||||
"https://sns.us-east-1.amazonaws.com/cert.pem?x=1", "not a URI"})
|
||||
void snsRejectsCertificateUrls(String url) throws Exception
|
||||
{
|
||||
var verifier = new SnsVerifier(request ->
|
||||
{
|
||||
throw (new AssertionError("Must not fetch untrusted URL"));
|
||||
});
|
||||
JSONObject data = new JSONObject(resource("/webhooks/ses-bounce.json")).put("SigningCertURL", url);
|
||||
assertThat(verifier.verify(data, TOPIC, certificateTime())).isFalse();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Confirmations use AWS field order and only verified AWS URLs are fetched.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void snsConfirmationsAndFailures() throws Exception
|
||||
{
|
||||
String pem = resource("/sns-test/test-only-certificate.pem");
|
||||
AtomicInteger confirms = new AtomicInteger();
|
||||
var verifier = new SnsVerifier(request ->
|
||||
{
|
||||
if(request.uri().getPath().endsWith(".pem"))
|
||||
{
|
||||
return (response(request, 200, pem));
|
||||
}
|
||||
confirms.incrementAndGet();
|
||||
return (response(request, 200, "confirmed"));
|
||||
});
|
||||
for(String type : List.of("SubscriptionConfirmation", "UnsubscribeConfirmation"))
|
||||
{
|
||||
JSONObject data = signSns(new JSONObject(resource("/webhooks/ses-bounce.json")).put("Type", type)
|
||||
.put("SubscribeURL", "https://sns.us-east-1.amazonaws.com/?Action=ConfirmSubscription&Token=test")
|
||||
.put("Token", "test-token"));
|
||||
assertThat(verifier.verify(data, TOPIC, certificateTime())).isTrue();
|
||||
verifier.confirm(data);
|
||||
assertThatThrownBy(() -> verifier.confirm(data.put("SubscribeURL", "https://evil.example/"))).isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
assertThat(confirms.get()).isEqualTo(2);
|
||||
assertThat(SnsVerifier.allowedUrl("https://sns.cn-north-1.amazonaws.com.cn/test.pem", true)).isTrue();
|
||||
JSONObject data = signSns(new JSONObject(resource("/webhooks/ses-bounce.json")));
|
||||
assertThat(new SnsVerifier(request -> response(request, 200, "bad cert")).verify(data, TOPIC, certificateTime())).isFalse();
|
||||
assertThatThrownBy(() -> new SnsVerifier(request -> response(request, 503, "error")).verify(data, TOPIC, certificateTime())).isInstanceOf(java.io.IOException.class);
|
||||
assertThat(verifier.verify(data, TOPIC, Instant.parse("2000-01-01T00:00:00Z"))).isFalse();
|
||||
data.put("Type", "Unsupported");
|
||||
assertThat(verifier.verify(data, TOPIC, certificateTime())).isFalse();
|
||||
}
|
||||
}
|
||||
@@ -21,6 +21,7 @@
|
||||
package com.kingsrook.qbits.email.transport;
|
||||
|
||||
|
||||
import javax.net.ssl.SSLSession;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.InetSocketAddress;
|
||||
@@ -44,7 +45,6 @@ import java.util.Properties;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import javax.net.ssl.SSLSession;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.kingsrook.qbits.email.EmailQBitConfig;
|
||||
import com.kingsrook.qbits.email.EmailRuntime;
|
||||
@@ -147,9 +147,9 @@ class EmailHttpTransportsTest
|
||||
*******************************************************************************/
|
||||
private static EmailEnvelope envelope()
|
||||
{
|
||||
return (new EmailEnvelope().withMessageId("42").withFrom("Sender <sender@example.com>")
|
||||
return (new EmailEnvelope().withMessageId("42").withRfcMessageId("<0123456789abcdef0123456789abcdef@example.com>").withFrom("Sender <sender@example.com>")
|
||||
.withTo(List.of("Person <to@example.com>")).withCc(List.of("cc@example.com")).withBcc(List.of("bcc@example.com"))
|
||||
.withReplyTo("Replies <reply@example.com>").withSubject("Hello \u00e9").withTextBody("Plain \u00e9").withHtmlBody("<b>Hello \u00e9</b>")
|
||||
.withReplyTo("Replies <reply@example.com>").withSubject("Hello é").withTextBody("Plain é").withHtmlBody("<b>Hello é</b>")
|
||||
.withHeaders(Map.of("List-Unsubscribe", "<https://example.com/unsubscribe>",
|
||||
"List-Unsubscribe-Post", "List-Unsubscribe=One-Click", "X-Custom", "kept")));
|
||||
}
|
||||
@@ -283,6 +283,11 @@ class EmailHttpTransportsTest
|
||||
assertThat(json.path("MessageStream").asText()).isEqualTo(tracking ? "broadcasts" : "outbound");
|
||||
assertThat(json.path("TrackOpens").asBoolean()).isEqualTo(tracking);
|
||||
assertThat(json.path("TrackLinks").asText()).isEqualTo(tracking ? "HtmlAndText" : "None");
|
||||
assertThat(json.path("Headers")).anySatisfy(header ->
|
||||
{
|
||||
assertThat(header.path("Name").asText()).isEqualTo("Message-ID");
|
||||
assertThat(header.path("Value").asText()).isEqualTo(envelope().getRfcMessageId());
|
||||
});
|
||||
assertThat(json.path("Headers").toString()).contains("List-Unsubscribe", "List-Unsubscribe-Post", "X-QQQ-Email-Id", "42", "Auto-Submitted");
|
||||
assertThat(json.toString()).doesNotContain(SECRET);
|
||||
}
|
||||
@@ -303,6 +308,7 @@ class EmailHttpTransportsTest
|
||||
return (response(202, "", Map.of("x-message-id", List.of("sg-id"))));
|
||||
});
|
||||
Map<String, String> headers = new java.util.HashMap<>(envelope().getHeaders());
|
||||
headers.put("message-id", "<override@example.com>");
|
||||
for(String reserved : List.of("X-SG-ID", "x-sg-eid", "Received", "DKIM-Signature", "Content-Type",
|
||||
"Content-Transfer-Encoding", "To", "From", "Subject", "Reply-To", "CC", "BcC"))
|
||||
{
|
||||
@@ -329,6 +335,8 @@ class EmailHttpTransportsTest
|
||||
assertThat(json.at("/content/1/type").asText()).isEqualTo("text/html");
|
||||
assertThat(json.at("/content/1/value").asText()).isEqualTo(envelope().getHtmlBody());
|
||||
assertThat(json.at("/custom_args/qqq_email_id").asText()).isEqualTo("42");
|
||||
assertThat(json.at("/headers/Message-ID").asText()).isEqualTo(envelope().getRfcMessageId());
|
||||
assertThat(json.path("headers").has("message-id")).isFalse();
|
||||
assertThat(json.path("headers").toString()).doesNotContain("must-be-removed").contains("List-Unsubscribe", "List-Unsubscribe-Post");
|
||||
assertThat(json.at("/tracking_settings/open_tracking/enable").asBoolean()).isEqualTo(tracking);
|
||||
assertThat(json.at("/tracking_settings/click_tracking/enable").asBoolean()).isEqualTo(tracking);
|
||||
@@ -371,6 +379,8 @@ class EmailHttpTransportsTest
|
||||
assertThat(mime).doesNotContain("bcc@example.com", "Bcc:", SECRET);
|
||||
MimeMessage message = new MimeMessage(Session.getInstance(new Properties()), new ByteArrayInputStream(mime.getBytes(StandardCharsets.UTF_8)));
|
||||
assertThat(message.getSubject()).isEqualTo(envelope().getSubject());
|
||||
assertThat(message.getMessageID()).isEqualTo(envelope().getRfcMessageId());
|
||||
assertThat(multipart).contains("name=\"h:Message-Id\"\r\n\r\n" + envelope().getRfcMessageId() + "\r\n");
|
||||
assertThat(message.getHeader("X-QQQ-Email-Id", null)).isEqualTo("42");
|
||||
assertThat(message.getHeader("List-Unsubscribe-Post", null)).isEqualTo("List-Unsubscribe=One-Click");
|
||||
assertThat(message.getRecipients(jakarta.mail.Message.RecipientType.CC)[0].toString()).isEqualTo("cc@example.com");
|
||||
|
||||
@@ -47,8 +47,15 @@ import jakarta.mail.internet.InternetAddress;
|
||||
import jakarta.mail.internet.MimeMessage;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import software.amazon.awssdk.services.sesv2.SesV2Client;
|
||||
import software.amazon.awssdk.services.sesv2.model.*;
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
import software.amazon.awssdk.services.sesv2.model.Message;
|
||||
import software.amazon.awssdk.services.sesv2.model.MessageRejectedException;
|
||||
import software.amazon.awssdk.services.sesv2.model.SendEmailRequest;
|
||||
import software.amazon.awssdk.services.sesv2.model.SendEmailResponse;
|
||||
import software.amazon.awssdk.services.sesv2.model.SesV2Exception;
|
||||
import software.amazon.awssdk.services.sesv2.model.TooManyRequestsException;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.fail;
|
||||
|
||||
/*******************************************************************************
|
||||
** Email Transports Test.
|
||||
@@ -61,7 +68,7 @@ class EmailTransportsTest
|
||||
EmailEnvelope envelope()
|
||||
{
|
||||
return (new EmailEnvelope()
|
||||
.withMessageId("123")
|
||||
.withMessageId("123").withRfcMessageId("<0123456789abcdef0123456789abcdef@example.com>")
|
||||
.withFrom("sender@example.com")
|
||||
.withTo(List.of("a@example.com"))
|
||||
.withBcc(List.of("hidden@example.com"))
|
||||
@@ -135,10 +142,26 @@ class EmailTransportsTest
|
||||
Session.getInstance(new Properties()),
|
||||
new ByteArrayInputStream(captured.get().content().raw().data().asByteArray()));
|
||||
assertThat(mime.getHeader("Bcc")).isNull();
|
||||
assertThat(mime.getMessageID()).isEqualTo(envelope().getRfcMessageId());
|
||||
assertThat(mime.getHeader("X-QQQ-Email-Id")).containsExactly("123");
|
||||
assertThat(mime.getHeader("X-Test")).containsExactly("yes");
|
||||
assertThat(mime.getHeader("Auto-Submitted")).containsExactly("auto-generated");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Repeated JavaMail saves cannot replace the identity or accept a custom override.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void mimeIdentitySurvivesSaveChanges() throws Exception
|
||||
{
|
||||
var envelope = envelope().withHeaders(Map.of("message-id", "<override@example.com>"));
|
||||
var message = EmailMime.build(Session.getInstance(new Properties()), envelope);
|
||||
message.saveChanges();
|
||||
message.saveChanges();
|
||||
assertThat(message.getHeader("Message-ID")).containsExactly(envelope.getRfcMessageId());
|
||||
assertThat(EmailMime.build(Session.getInstance(new Properties()), envelope().withRfcMessageId(null)).getMessageID()).isNotBlank();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Ses Classifies Throttling Rejection And Server Failure.
|
||||
*******************************************************************************/
|
||||
@@ -150,7 +173,7 @@ class EmailTransportsTest
|
||||
TooManyRequestsException.builder().statusCode(429).message("secret").build(),
|
||||
MessageRejectedException.builder().statusCode(400).message("secret").build(),
|
||||
SesV2Exception.builder().statusCode(503).message("secret").build()))
|
||||
{
|
||||
{
|
||||
var transport =
|
||||
new SesEmailTransport(
|
||||
builder ->
|
||||
@@ -313,7 +336,7 @@ class EmailTransportsTest
|
||||
for(String newline : List.of("\r", "\n"))
|
||||
{
|
||||
String bad = "a@example.com" + newline + "Bcc: hidden@example.com";
|
||||
for(EmailEnvelope envelope : List.of(envelope().withSubject(bad), envelope().withFrom(bad), envelope().withReplyTo(bad),
|
||||
for(EmailEnvelope envelope : List.of(envelope().withRfcMessageId(bad), envelope().withSubject(bad), envelope().withFrom(bad), envelope().withReplyTo(bad),
|
||||
envelope().withTo(List.of(bad)), envelope().withCc(List.of(bad)), envelope().withBcc(List.of(bad)),
|
||||
envelope().withHeaders(Map.of("X-Test", bad)), envelope().withFrom("broken <")))
|
||||
{
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
-- Phase (a), parts 1 and 2. Apply before producing the qbit. Host security columns must be added by the host.
|
||||
-- Phases (a), (b) and (c). Apply before producing the qbit. Host security columns must be added by the host.
|
||||
|
||||
CREATE TABLE email_provider (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
@@ -38,6 +38,19 @@ CREATE TABLE email_provider (
|
||||
UNIQUE (default_slot)
|
||||
);
|
||||
|
||||
CREATE TABLE email_provider_sender_address (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
modify_date TIMESTAMP,
|
||||
provider_id INTEGER NOT NULL,
|
||||
address VARCHAR(500) NOT NULL,
|
||||
display_name VARCHAR(500),
|
||||
is_default BOOLEAN,
|
||||
enabled BOOLEAN,
|
||||
note TEXT,
|
||||
UNIQUE (provider_id, address)
|
||||
);
|
||||
|
||||
CREATE TABLE email_message (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
@@ -47,6 +60,7 @@ CREATE TABLE email_message (
|
||||
cc_addresses TEXT,
|
||||
bcc_addresses TEXT,
|
||||
from_address VARCHAR(500),
|
||||
from_address_id INTEGER,
|
||||
reply_to VARCHAR(500),
|
||||
subject TEXT,
|
||||
template_id INTEGER,
|
||||
@@ -61,6 +75,7 @@ CREATE TABLE email_message (
|
||||
phi_approved BOOLEAN,
|
||||
unsubscribe_url TEXT,
|
||||
idempotency_key VARCHAR(500),
|
||||
rfc_message_id VARCHAR(500),
|
||||
provider_id INTEGER,
|
||||
provider_message_id VARCHAR(500),
|
||||
attempt_count INTEGER,
|
||||
@@ -79,7 +94,8 @@ CREATE TABLE email_message (
|
||||
sender_process VARCHAR(500),
|
||||
suppressed_addresses TEXT,
|
||||
send_after TIMESTAMP,
|
||||
UNIQUE (idempotency_key)
|
||||
UNIQUE (idempotency_key),
|
||||
UNIQUE (rfc_message_id)
|
||||
);
|
||||
|
||||
CREATE TABLE email_message_attempt (
|
||||
@@ -184,3 +200,42 @@ CREATE TABLE email_suppression (
|
||||
CREATE INDEX email_message_due ON email_message (status, next_attempt_date);
|
||||
CREATE INDEX email_attempt_provider_start ON email_message_attempt (provider_id, start_date);
|
||||
CREATE INDEX email_attempt_provider_end ON email_message_attempt (provider_id, end_date, id);
|
||||
|
||||
CREATE TABLE email_webhook_inbox (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
modify_date TIMESTAMP,
|
||||
provider_id INTEGER NOT NULL,
|
||||
kind VARCHAR(500) NOT NULL,
|
||||
external_event_id VARCHAR(500) NOT NULL,
|
||||
nonce VARCHAR(500),
|
||||
status VARCHAR(500) NOT NULL,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_date TIMESTAMP,
|
||||
last_error VARCHAR(500),
|
||||
payload_json TEXT,
|
||||
payload_sha256 VARCHAR(500),
|
||||
received_date TIMESTAMP,
|
||||
processed_date TIMESTAMP,
|
||||
UNIQUE (provider_id, external_event_id),
|
||||
UNIQUE (provider_id, nonce)
|
||||
);
|
||||
|
||||
CREATE TABLE email_event (
|
||||
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
|
||||
create_date TIMESTAMP,
|
||||
modify_date TIMESTAMP,
|
||||
message_id INTEGER NOT NULL,
|
||||
provider_id INTEGER,
|
||||
event_type VARCHAR(500) NOT NULL,
|
||||
bounce_type VARCHAR(500),
|
||||
recipient VARCHAR(500),
|
||||
occurred_date TIMESTAMP,
|
||||
detail VARCHAR(500),
|
||||
url TEXT,
|
||||
event_key VARCHAR(500) NOT NULL UNIQUE
|
||||
);
|
||||
CREATE INDEX email_inbox_due ON email_webhook_inbox (status, next_attempt_date);
|
||||
CREATE INDEX email_event_message ON email_event (message_id);
|
||||
CREATE INDEX email_event_type_date ON email_event (event_type, occurred_date);
|
||||
CREATE INDEX email_message_provider_id ON email_message (provider_id, provider_message_id);
|
||||
@@ -0,0 +1,13 @@
|
||||
# Test-only SNS certificate and private key
|
||||
|
||||
These are public test fixtures, never production credentials. Generated at authoring
|
||||
time with OpenSSL; tests do not spawn OpenSSL or write certificate files.
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days 36500 \
|
||||
-subj "/CN=TEST ONLY - SNS webhook verification fixture" \
|
||||
-keyout test-only-private-key.pem -out test-only-certificate.pem
|
||||
```
|
||||
|
||||
The fake EmailHttp serves this certificate for an AWS-shaped test URL. SNS tests
|
||||
sign with this key in memory. Never trust this certificate outside the tests.
|
||||
@@ -0,0 +1,20 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIDUTCCAjmgAwIBAgIUMxFivUctd7E0lQz94zgDEkGMm8EwDQYJKoZIhvcNAQEL
|
||||
BQAwNzE1MDMGA1UEAwwsVEVTVCBPTkxZIC0gU05TIHdlYmhvb2sgdmVyaWZpY2F0
|
||||
aW9uIGZpeHR1cmUwIBcNMjYxMDE0MDI0OTM4WhgPMjEyNjA5MjAwMjQ5MzhaMDcx
|
||||
NTAzBgNVBAMMLFRFU1QgT05MWSAtIFNOUyB3ZWJob29rIHZlcmlmaWNhdGlvbiBm
|
||||
aXh0dXJlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwDnXnAbhkNrq
|
||||
Hi05Rl33RojxWRN8N81yEISzyrp8zTBXg15Icp8dkLIEkTYrcZv5GDZ/ROE95nHG
|
||||
Ue9ODyK3ga8X4yyHZQEtc3F0SyCwwCg/sRIx2F217cnrr2jbF57mZBiUqcr0+naa
|
||||
ZyqpBsV4iBnZSIQchi0RWTkPI7kj0oJi9uWBOf7sVQBlKFyboTRloFLxnB4XdBQ5
|
||||
PYNiNT9pv8drCe3B9OcAu9odhRuI/fvws1C+zlDxqxlCvYGonUvGD9ob9x5NkuUg
|
||||
hABxyUYcZWgMzSBTcrL7vIZyTLsgEqRB2IZQDeqZhpcEc2ewrWvnTPzPVc9rzew1
|
||||
8hltQd4E2wIDAQABo1MwUTAdBgNVHQ4EFgQUIfc+4qImdzoXevD8eAK8iFynrTQw
|
||||
HwYDVR0jBBgwFoAUIfc+4qImdzoXevD8eAK8iFynrTQwDwYDVR0TAQH/BAUwAwEB
|
||||
/zANBgkqhkiG9w0BAQsFAAOCAQEAFaUW8M16oVhV/lERu+JXeasCtvLrudrufomH
|
||||
HybCP+YHbgee4RnIBDjKOzDZRpbjJmi/a7rJidtpdnD8Dn1lexB7wEDmgDdyB4oT
|
||||
4R62dKuOQgnL/RuDUUp3YYMZliSppSNQRgOYl7Islq3zmpYBCBAV9s4fiUHW8uBy
|
||||
ZYRQbF49xxPINpSreMJtrf+Pq9E1As4cCc3ViSS1ITsH4R515xJvQg64HQtunl8W
|
||||
+rSzTfsAADCqqFapmdLQiHSJWIEXxbNv1B0vBSIJCQkNsjTceblgqYTAdK1Kufsw
|
||||
s5ZWUbQHAIXODa3VRvaislvH+ygHAtIBi1kyV33c2AiuPPVuqg==
|
||||
-----END CERTIFICATE-----
|
||||
@@ -0,0 +1,28 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDAOdecBuGQ2uoe
|
||||
LTlGXfdGiPFZE3w3zXIQhLPKunzNMFeDXkhynx2QsgSRNitxm/kYNn9E4T3mccZR
|
||||
704PIreBrxfjLIdlAS1zcXRLILDAKD+xEjHYXbXtyeuvaNsXnuZkGJSpyvT6dppn
|
||||
KqkGxXiIGdlIhByGLRFZOQ8juSPSgmL25YE5/uxVAGUoXJuhNGWgUvGcHhd0FDk9
|
||||
g2I1P2m/x2sJ7cH05wC72h2FG4j9+/CzUL7OUPGrGUK9gaidS8YP2hv3Hk2S5SCE
|
||||
AHHJRhxlaAzNIFNysvu8hnJMuyASpEHYhlAN6pmGlwRzZ7Cta+dM/M9Vz2vN7DXy
|
||||
GW1B3gTbAgMBAAECggEATrQgHEkQ9+855iscBTyZ7u5I6vC4blyWtD6xInMigByn
|
||||
6CilPxD42PKaPh5QwU5cOdauWzRh1MWGrqttaXMMcXOQMzpbITbPNjIgrFXxKGwS
|
||||
T9ykPfN4dqiy7/ndopkzuY366F4p2ZbP9yeXVydpB0iWF0PILgOr070x/q1iBmEX
|
||||
zY/rTgSO3xhS6fJOtZdCFvSY7Geg8pPYJ1pAYBaz2nPL1WYXN4M0qpllBLKBwwnf
|
||||
i6PRxLAyATLpOJgmI9sGmbfvu7yKRIAvVItR+2r5v03Fh7brKEOq9YmmzY7Hx62L
|
||||
VWsr6CMzUrsUm7R54epPeBBvhZ0AIWAf9gIDemvmUQKBgQDz4lms/uFjn6h9aoNX
|
||||
Xk/AdZGKp44d3vXF7X3MrKB10zPPg13n/Wwp0DrGP08aZMMbWevROIHfzToTFHkW
|
||||
sYiwmazF9y0+6tf0BmV8EuR7vJtIhKwTJAgdFbbTylXUdNc+5rh+O6IoImlpCD75
|
||||
rQfPe3fezlxzYmC43Q68WfRsYwKBgQDJxoRl8/a0jNx4ZFZzYfxk/uG5/HdY+dSS
|
||||
SecJvj9M6CTDbA8k8wBWgdsgEgvB5qCjTe7Bci9dagDcL4qUsAl4H05I09VVIbUd
|
||||
3h7JCWwVFr/Dvn6L7tqeJXlq5yDPfYDPvbe2EK+x/Zq/QTVtPYFqtlH83U78cU4N
|
||||
3g9mP1aDKQKBgCQU+O9/RhlXJbZoa8YhTetLeZzp7Ty+H96IpHQDljnYZXHuHZaV
|
||||
B8un4woxk6J0BemmBB2QHqbd1hm8ja2t1mtKUSV4zFjipdCUoa8ZnHQyE0dHWa37
|
||||
RrgG3mW3KbulDbNgryr955/Id4/yiF6oGt+3hKSSI1tu0qCHFAcJx//BAoGAJE4j
|
||||
5HRRH4t43/Ct3XT0BPgXhW93yfqp1OiXOZmRqjTa6I646t0OYoJa+xpEI09r9/Y9
|
||||
z3DMNPKt1PB47uf9zC9V1qTG7dpyzDz7LREU8XZ2arIL8wG3NF1py2Jq9OqQEtEA
|
||||
DTJTaqNHBMKm6lD2yEyl0+m+HI4GjVg3TR54YoECgYEA3kOMd1NuVDNHQU6qRD54
|
||||
oGvym00jaCYCH1mQvS7VWOkXfmT6ZnKw+qRLcoUGTtP76pcWNYQQMyUDfbdtQp1u
|
||||
gdrlS/zASsQhuDaKrCiuwOIlTvlx7InRi2MqpN+ztus8hZzyBHg9EHUWGykZ7rRh
|
||||
XtLanuf3Tn3MvyiHNaVPyQs=
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"signature": {
|
||||
"timestamp": "1791892800",
|
||||
"token": "test-mailgun-nonce",
|
||||
"signature": "signed-in-test"
|
||||
},
|
||||
"event-data": {
|
||||
"id": "mailgun-event-1",
|
||||
"event": "failed",
|
||||
"severity": "permanent",
|
||||
"timestamp": 1791892800,
|
||||
"recipient": "recipient@example.com",
|
||||
"message": {
|
||||
"headers": {
|
||||
"message-id": "<provider-message-1>"
|
||||
}
|
||||
},
|
||||
"delivery-status": {
|
||||
"message": "550 recipient@example.com unavailable"
|
||||
},
|
||||
"user-variables": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"RecordType": "Bounce",
|
||||
"ID": 10001,
|
||||
"MessageID": "provider-message-1",
|
||||
"Type": "HardBounce",
|
||||
"Email": "recipient@example.com",
|
||||
"BouncedAt": "2026-10-13T12:01:00Z",
|
||||
"Description": "550 mailbox recipient@example.com is unavailable"
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"RecordType": "Delivery",
|
||||
"MessageID": "provider-message-1",
|
||||
"Recipient": "recipient@example.com",
|
||||
"DeliveredAt": "2026-10-13T12:00:00Z",
|
||||
"Details": "250 accepted"
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
[
|
||||
{
|
||||
"email": "recipient@example.com",
|
||||
"timestamp": 1791892800,
|
||||
"event": "delivered",
|
||||
"sg_event_id": "sg-event-1",
|
||||
"sg_message_id": "provider-message-1.filter001",
|
||||
"response": "250 accepted"
|
||||
},
|
||||
{
|
||||
"email": "recipient@example.com",
|
||||
"timestamp": 1791892860,
|
||||
"event": "bounce",
|
||||
"type": "bounce",
|
||||
"sg_event_id": "sg-event-2",
|
||||
"sg_message_id": "provider-message-1.filter001",
|
||||
"reason": "550 recipient@example.com does not exist"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"Type": "Notification",
|
||||
"MessageId": "sns-envelope-1",
|
||||
"TopicArn": "arn:aws:sns:us-east-1:123456789012:email-events",
|
||||
"Timestamp": "2026-10-13T12:00:00Z",
|
||||
"SignatureVersion": "2",
|
||||
"SigningCertURL": "https://sns.us-east-1.amazonaws.com/test-only.pem",
|
||||
"Signature": "signed-in-test",
|
||||
"Message": "{\"eventType\": \"Bounce\", \"mail\": {\"messageId\": \"provider-message-1\", \"destination\": [\"recipient@example.com\", \"second@example.com\"], \"headers\": [{\"name\": \"X-QQQ-Email-Id\", \"value\": \"1\"}]}, \"bounce\": {\"bounceType\": \"Permanent\", \"bounceSubType\": \"General\", \"timestamp\": \"2026-10-13T12:01:00Z\", \"bouncedRecipients\": [{\"emailAddress\": \"recipient@example.com\", \"diagnosticCode\": \"smtp; 550 recipient@example.com unavailable\"}, {\"emailAddress\": \"second@example.com\", \"diagnosticCode\": \"smtp; 550 second@example.com unavailable\"}]}}"
|
||||
}
|
||||
Reference in new issue
Block a user