diff --git a/CHANGELOG.md b/CHANGELOG.md index bbd13d9..850e7c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and ## [Unreleased] ### Added +- Persisted unique RFC Message-IDs, stable across delivery attempts, with SMTP/Gmail SMTP, SES raw MIME and Postmark/SendGrid/Mailgun request propagation (SES replaces the header downstream). +- Core-independent `EmailEffects` handoff, reference lookups by idempotency key or RFC Message-ID, refusal mapping and an RC.7 binding design sketch; offline retry and provider-contract tests. +- Checkstyle now checks test sources as well as main sources; existing test imports and formatting corrected. - Provider sender addresses: child table `email_provider_sender_address` (validated lowercase address, display name, one default, enabled, note; unique per provider; domain bound to the provider's sending domain; admin-only), shown on the provider record. `EmailRequest.withFromAddressId`, `from_address_id` on messages, refusal of a from address that is not an enabled row of the resolved provider, and the provider `from_address` / `from_name` kept as the fallback. - Contingent possible-value filters on the new from-address pickers (send processes by `providerId`, Send Test Email by the acted-on provider, message `fromAddressId` by `providerId`) and on the attempt and event message pickers (by `providerId`) and the suppression source event (by `sourceMessageId`); the send processes also take `providerId`. - Phase (c): authenticated SNS, Postmark, SendGrid and Mailgun webhooks, durable inbox deduplication and Mailgun nonce replay protection; configurable Javalin routes. diff --git a/README.md b/README.md index 0c2218a..8ad9e26 100644 --- a/README.md +++ b/README.md @@ -501,6 +501,26 @@ concurrent idempotency, shared rate budgets, health recovery and process screens The tests also exercise template lifecycle, parent-scoped possible values, rendering, PHI canary egress, encryption, preview widgets and home data. +## Durable jobs + +`EmailEffects.send(request, effectIdempotencyKey)` hands a message to the committed +email outbox exactly once per key and returns an `EmailEffects.Result` containing +`EmailEffectOutcome` and an `EmailMessageRef`. `QUEUED_OR_DUPLICATE` means the outbox +accepted responsibility; `REFUSED` means a definite refusal, including suppression. +Use `findByIdempotencyKey` or `findByRfcMessageId` to reconcile a handoff. These are +trusted system reads returning only references. Database failures propagate. +Requests must not carry a caller transaction; the effect dispatcher calls after its +own chunk commit. Existing `EmailService` transaction support is unchanged. + +Each new message stores a unique `rfcMessageId` once. SMTP (including the Gmail preset), +Postmark, SendGrid and Mailgun carry it on every attempt. SES receives it in the raw +request but replaces it downstream; every transport retains `X-QQQ-Email-Id`. +Message-ID enables correlation and receiver deduplication where supported; it does +not promise exactly-once inbox delivery. + +See [the handoff contract, provider references, schema upgrade and RC.7 binding sketch](docs/DURABLE-JOBS.md). +The build remains on released QQQ **4.1.0-RC.5**, without durable-jobs dependencies. + ## License Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE). diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 47524f8..9b68dcf 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -185,6 +185,7 @@ transient; other 4xx are permanent. | variables_snapshot | JSON, redacted per section 7.3 | | html_body, text_body | rendered at queue time so retries send the same bytes; encrypted for `phi_approved`; cleared after `bodyRetentionDays` (default 30) once final, keeping `body_sha256` | | idempotency_key | unique when present | +| rfc_message_id | STRING(500), unique; stored once at enqueue and reused on every attempt; see [durable job effects](DURABLE-JOBS.md) | | provider_id, provider_message_id | | | attempt_count, next_attempt_date, lease_expires_date, last_error, sent_date, delivered_date, dead_letter_date | | | linked_table_name, linked_record_id | rendered as a link on the record screen | diff --git a/docs/DURABLE-JOBS.md b/docs/DURABLE-JOBS.md new file mode 100644 index 0000000..8311c9f --- /dev/null +++ b/docs/DURABLE-JOBS.md @@ -0,0 +1,123 @@ +# Durable job email effects + +This implements the email-side contract for QRun-IO/qqq#1191, spec section 2.13, +without importing core jobs types. QQQ stays at released 4.1.0-RC.5. The +`EmailJobEffectTarget implements EffectTarget` binding is deferred until RC.7 ships. + +## Stored identity and schema upgrade + +`email_message.rfc_message_id` (`rfcMessageId` in metadata) is a unique STRING(500) +shown in the message T2 details. Before insertion, the service stores +``, where `token` is the first 32 lowercase hex characters of SHA-256 +of the UTF-8 idempotency key. Without a key, the seed is `qbit-email:` plus a fresh +UUID, since the row id is not yet allocated. The domain is the normalized From +mailbox domain, excluding display names; unavailable or unsuitable domains use +`qbit-email.local`. The key itself is never included in the header. + +The dispatcher reads this stored value on every attempt. JavaMail's +`updateMessageID()` is overridden so repeated `saveChanges()` calls cannot replace +it. A manual **Resend** creates a new outbox row and a new identity; a delivery retry +keeps its identity. `providerMessageId` remains the provider's response identifier, +distinct from `rfcMessageId` and the numeric outbox `id`. + +New installations use [email-postgres.sql](ddl/email-postgres.sql). Existing +PostgreSQL installations must pause enqueue and dispatch, then apply +[email-message-id-upgrade.sql](ddl/email-message-id-upgrade.sql) before starting +this version. The migration backfills old rows using the existing key, or +`qbit-email:` plus the already allocated row id, and the stored From domain. +It cannot recover previously generated random transport Message-IDs. Review host +schema adaptations first; host security columns are unaffected. + +## Provider behavior + +| Transport | Request mechanism | Identity preservation | +|---|---|---| +| SMTP / Gmail SMTP preset | MIME `Message-ID` | Sent unchanged by this transport; downstream relay policy still applies | +| SES v2 | Raw MIME `Message-ID` | **SES replaces Message-ID downstream** | +| Postmark | `Headers: [{Name: "Message-ID", Value: "<...>"}]` | Custom Message-ID supported | +| SendGrid | `headers: {"Message-ID": "<...>"}` | Sent as a custom header | +| Mailgun | MIME Message-ID plus multipart `h:Message-Id` | Sent in the `/messages.mime` request | + +All transports retain `X-QQQ-Email-Id` with the numeric outbox id. Postmark's +`MessageID`, SendGrid's response `X-Message-Id`, and SES's returned id are provider +references, not our RFC header. A fixed header enables recognition and receiver +deduplication where supported, but is not an exactly-once inbox guarantee. + +Provider references: [Postmark custom Message-ID](https://postmarkapp.com/developer/user-guide/send-email-with-api/send-a-single-email), +[SendGrid custom headers](https://www.twilio.com/docs/sendgrid/api-reference/mail-send/mail-send), +[Mailgun sending parameters](https://documentation.mailgun.com/docs/mailgun/api-reference/send/mailgun/messages/post-v3--domain-name--messages), +and [SES header replacement](https://docs.aws.amazon.com/ses/latest/APIReference/API_SendRawEmail.html). + +## Handoff and reconcile + +`EmailEffects.send(request, effectIdempotencyKey)` sets the request's key, refusing +a different pre-existing key, a missing/blank/overlength effect key, a null request +or a caller transaction. The service performs its usual validation and send +permission checks. Success is returned only after the outbox transaction commits; +there is no transport call here. Use ordinary `EmailService.send` for caller-owned +or cross-backend deferred transactions. + +`EmailEffects.Result(outcome, message)` contains an `EmailEffectOutcome` and a +nullable `EmailMessageRef(Integer id, String status, String rfcMessageId, +String providerMessageId)`: + +- `QUEUED_OR_DUPLICATE`: one committed row accepted the handoff, including replays. +- `REFUSED`: invalid or suppressed request. A suppressed audit row has a reference; + refusals without a stored row have a null reference. Replaying a suppressed row + stays refused even though `EmailService` returns `DUPLICATE`. +- Database/query failures throw `QException`; callers must retry or reconcile, + never turn an uncertain outcome into a permanent refusal. + +`findByIdempotencyKey(key)` and `findByRfcMessageId(messageId)` return +`Optional` from system reads, like other internal outbox reads. +They return empty for null/blank/missing values. These are trusted backend APIs, +not viewer-scoped public endpoints. Lookups reflect current status and provider +reference and do not expose bodies, recipients or variables. + +The effect's `SENT` means successful durable handoff, not mailbox delivery. The +email dispatcher owns later delivery/retry failures. Reconcile acknowledges an +existing outbox row (including an audit row); it must never create another email +merely because that row is terminal. Retain message rows and idempotency keys for +at least the maximum effect retry/reconciliation horizon. + +## RC.7 binding design note (not compiled) + +Mapping: `deliver -> QUEUED_OR_DUPLICATE => SENT(ref.id)`, `REFUSED => FAILED`; +`reconcile -> findByIdempotencyKey present => FOUND(ref.id), absent => NOT_FOUND`. +Exceptions escape to core's retry/unknown handling. Payload decoding must preserve +the existing PHI/variable policies and must not attach a caller transaction. + +This 25-line sketch is documentation-only. Match core names and signatures to the +published RC.7 contract when it ships; `decode` represents payload conversion. + +```java +// Future RC.7 binding; no core jobs imports in the RC.5 implementation. +final class EmailJobEffectTarget implements EffectTarget +{ + // SENT acknowledges the committed qbit outbox row. + @Override + public DeliveryResult deliver(Effect effect) throws Exception + { + EmailRequest request = decode(effect.payload()); + EmailEffects.Result result = EmailEffects.send(request, effect.idempotencyKey()); + if(result.outcome() == EmailEffectOutcome.REFUSED) + { + return (DeliveryResult.failed("Email refused")); + } + return (DeliveryResult.sent(result.message().id().toString())); + } + + // Existence reconciles handoff independently of mailbox delivery. + @Override + public ReconcileResult reconcile(Effect effect) throws Exception + { + return (EmailEffects.findByIdempotencyKey(effect.idempotencyKey()) + .map(ref -> ReconcileResult.found(ref.id().toString())) + .orElseGet(ReconcileResult::notFound)); + } +} +``` + +Tests use H2, the existing GreenMail SMTP sink, fake SES and HTTP request seams; +no vendor endpoints are called. The PostgreSQL migration is supplied for host +deployment and is not executed by the offline Java test gate. diff --git a/docs/ddl/email-message-id-upgrade.sql b/docs/ddl/email-message-id-upgrade.sql new file mode 100644 index 0000000..6da735d --- /dev/null +++ b/docs/ddl/email-message-id-upgrade.sql @@ -0,0 +1,11 @@ +-- Existing PostgreSQL installations: pause enqueue and dispatch before applying. +-- SHA-256 is PostgreSQL's built-in bytea function; no extension is required. +BEGIN; +ALTER TABLE email_message ADD COLUMN rfc_message_id VARCHAR(500); +UPDATE email_message +SET rfc_message_id = '<' || left(encode(sha256(convert_to( + coalesce(idempotency_key, 'qbit-email:' || id::text), 'UTF8')), 'hex'), 32) + || '@' || coalesce(lower(substring(from_address from + '@([A-Za-z0-9]+(?:[.-][A-Za-z0-9]+)*)(?:>|$)')), 'qbit-email.local') || '>'; +ALTER TABLE email_message ADD CONSTRAINT email_message_rfc_message_id_unique UNIQUE (rfc_message_id); +COMMIT; diff --git a/docs/ddl/email-postgres.sql b/docs/ddl/email-postgres.sql index ae1ea2e..264fdd4 100644 --- a/docs/ddl/email-postgres.sql +++ b/docs/ddl/email-postgres.sql @@ -75,6 +75,7 @@ CREATE TABLE email_message ( phi_approved BOOLEAN, unsubscribe_url TEXT, idempotency_key VARCHAR(500), + rfc_message_id VARCHAR(500), provider_id INTEGER, provider_message_id VARCHAR(500), attempt_count INTEGER, @@ -93,7 +94,8 @@ CREATE TABLE email_message ( sender_process VARCHAR(500), suppressed_addresses TEXT, send_after TIMESTAMP, - UNIQUE (idempotency_key) + UNIQUE (idempotency_key), + UNIQUE (rfc_message_id) ); CREATE TABLE email_message_attempt ( diff --git a/pom.xml b/pom.xml index 0243ef0..5aeb48e 100644 --- a/pom.xml +++ b/pom.xml @@ -168,7 +168,7 @@ false true warning - false + true diff --git a/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java b/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java index f5a8be4..b6c18dd 100644 --- a/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java +++ b/src/main/java/com/kingsrook/qbits/email/delivery/EmailDispatcher.java @@ -451,7 +451,7 @@ public class EmailDispatcher *******************************************************************************/ private static EmailEnvelope envelope(QRecord message) { - return (new EmailEnvelope().withMessageId(message.getValueString("id")).withFrom(message.getValueString("fromAddress")) + return (new EmailEnvelope().withMessageId(message.getValueString("id")).withRfcMessageId(message.getValueString("rfcMessageId")).withFrom(message.getValueString("fromAddress")) .withTo(split(message.getValueString("toAddresses"))).withCc(split(message.getValueString("ccAddresses"))).withBcc(split(message.getValueString("bccAddresses"))) .withReplyTo(message.getValueString("replyTo")).withSubject(content(message, "subject")) .withHtmlBody(content(message, "htmlBody")) diff --git a/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java b/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java index f484348..f428922 100644 --- a/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java +++ b/src/main/java/com/kingsrook/qbits/email/metadata/EmailTables.java @@ -269,6 +269,7 @@ public final class EmailTables .withField(field("unsubscribeUrl", QFieldType.TEXT)) .withField(field("bodySha256", QFieldType.STRING).withMaxLength(500)) .withField(field("idempotencyKey", QFieldType.STRING).withMaxLength(500)) + .withField(field("rfcMessageId", QFieldType.STRING).withMaxLength(500)) .withField(field("providerId", QFieldType.INTEGER).withPossibleValueSourceName(PROVIDER)) .withField(field("providerMessageId", QFieldType.STRING).withMaxLength(500)) .withField(field("attemptCount", QFieldType.INTEGER).withDefaultValue(0)) @@ -291,8 +292,9 @@ public final class EmailTables table.withRecordLabelFormat(null).withRecordLabelFields(List.of()); table.getField("subject").setIsHeavy(false); table.withUniqueKey(new UniqueKey("idempotencyKey")); + table.withUniqueKey(new UniqueKey("rfcMessageId")); table.withSection(SectionFactory.defaultT1("id", "status", "toAddresses", "ccAddresses", "bccAddresses", "fromAddress")); - table.withSection(SectionFactory.defaultT2("fromAddressId", "replyTo", "subject", "templateId", "templateVersionId", "brandId", "category", "locale", "variablesSnapshot", "htmlBody", "textBody", "phiApproved", "unsubscribeUrl", "bodySha256", "idempotencyKey", "providerId", "providerMessageId", "attemptCount", "nextAttemptDate", "leaseExpiresDate", "lastError", "finishedDate", "sentDate", "deliveredDate", "bouncedDate", "complainedDate", "deadLetterDate", "linkedTableName", "linkedRecordId", "senderQbit", "senderProcess", "suppressedAddresses", "sendAfter")); + table.withSection(SectionFactory.defaultT2("fromAddressId", "replyTo", "subject", "templateId", "templateVersionId", "brandId", "category", "locale", "variablesSnapshot", "htmlBody", "textBody", "phiApproved", "unsubscribeUrl", "bodySha256", "idempotencyKey", "rfcMessageId", "providerId", "providerMessageId", "attemptCount", "nextAttemptDate", "leaseExpiresDate", "lastError", "finishedDate", "sentDate", "deliveredDate", "bouncedDate", "complainedDate", "deadLetterDate", "linkedTableName", "linkedRecordId", "senderQbit", "senderProcess", "suppressedAddresses", "sendAfter")); table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); table.withSection(SectionFactory.customT2("children", new QIcon("list")).withLabel("Attempts").withWidgetName(MESSAGE + "_children")); table.withSection(SectionFactory.customT2("events", new QIcon("list")).withLabel("Events").withWidgetName(EVENT + "_children")); diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailEffectOutcome.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailEffectOutcome.java new file mode 100644 index 0000000..2fef1f3 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailEffectOutcome.java @@ -0,0 +1,31 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +/******************************************************************************* + ** Durable handoff outcomes. Operational failures propagate to the caller. + *******************************************************************************/ +public enum EmailEffectOutcome +{ + QUEUED_OR_DUPLICATE, + REFUSED +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailEffects.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailEffects.java new file mode 100644 index 0000000..0b02e6d --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailEffects.java @@ -0,0 +1,114 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import java.util.List; +import java.util.Optional; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + + +/******************************************************************************* + ** Outbox handoff and system-read reconciliation for trusted effect dispatchers. + ** This API has no dependency on core durable-job types. Sending retains the + ** EmailService permission checks; callers must supply a request without a transaction. + *******************************************************************************/ +public final class EmailEffects +{ + /******************************************************************************* + ** A refusal can have a reference when suppression retained an audit row. + *******************************************************************************/ + public record Result(EmailEffectOutcome outcome, EmailMessageRef message) + { + } + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private EmailEffects() + { + } + + + + /******************************************************************************* + ** Hand off once with the effect's identity. Refuse invalid or conflicting keys + ** and caller transactions: success must mean the outbox has committed already. + ** Database failures throw QException and must not become a permanent refusal. + *******************************************************************************/ + public static Result send(EmailRequest request, String effectIdempotencyKey) throws QException + { + if(request == null || effectIdempotencyKey == null || effectIdempotencyKey.isBlank() || effectIdempotencyKey.length() > 500 + || request.getTransaction() != null + || request.getIdempotencyKey() != null && !request.getIdempotencyKey().equals(effectIdempotencyKey)) + { + return (new Result(EmailEffectOutcome.REFUSED, null)); + } + request.withIdempotencyKey(effectIdempotencyKey); + EmailSendResult result = EmailService.send(request); + EmailMessageRef message = result.messageId() == null ? null : find("id", result.messageId()).orElseThrow(() -> new QException("Queued email reference is unavailable")); + boolean refused = result.status() == EmailSendResult.Status.REFUSED || result.status() == EmailSendResult.Status.SUPPRESSED + || message != null && "SUPPRESSED".equals(message.status()); + return (new Result(refused ? EmailEffectOutcome.REFUSED : EmailEffectOutcome.QUEUED_OR_DUPLICATE, message)); + } + + + + /******************************************************************************* + ** Look up a committed outbox row by its exact effect or application key. + *******************************************************************************/ + public static Optional findByIdempotencyKey(String key) throws QException + { + return (key == null || key.isBlank() ? Optional.empty() : find("idempotencyKey", key)); + } + + + + /******************************************************************************* + ** Look up the exact bracketed RFC Message-ID stored at enqueue time. + *******************************************************************************/ + public static Optional findByRfcMessageId(String messageId) throws QException + { + return (messageId == null || messageId.isBlank() ? Optional.empty() : find("rfcMessageId", messageId)); + } + + + + /******************************************************************************* + ** Use the same authorized read scope as the dispatcher; expose only references. + *******************************************************************************/ + private static Optional find(String field, java.io.Serializable value) throws QException + { + List rows = SystemWrites.query(EmailTables.MESSAGE, EmailStore.equal(field, value).withLimit(1)); + if(rows.isEmpty()) + { + return (Optional.empty()); + } + QRecord row = rows.getFirst(); + return (Optional.of(new EmailMessageRef(row.getValueInteger("id"), row.getValueString("status"), row.getValueString("rfcMessageId"), row.getValueString("providerMessageId")))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailMessageRef.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailMessageRef.java new file mode 100644 index 0000000..71e70bd --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailMessageRef.java @@ -0,0 +1,29 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +/******************************************************************************* + ** Reconciliation reference without message bodies, recipients or credentials. + *******************************************************************************/ +public record EmailMessageRef(Integer id, String status, String rfcMessageId, String providerMessageId) +{ +} diff --git a/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java b/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java index 4b39b42..5d78b06 100644 --- a/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java +++ b/src/main/java/com/kingsrook/qbits/email/outbox/EmailService.java @@ -30,6 +30,7 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Set; +import java.util.UUID; import com.kingsrook.qbits.email.EmailQBitConfig; import com.kingsrook.qbits.email.EmailRuntime; import com.kingsrook.qbits.email.metadata.EmailTableCustomizers; @@ -261,6 +262,7 @@ public final class EmailService .withValue("htmlBody", rendered.html()).withValue("textBody", rendered.text()).withValue("bodySha256", digest(rendered)) .withValue("brandId", adHocBrand == null ? null : adHocBrand.getValue("id")) .withValue("category", category.name()).withValue("locale", request.getLocale()).withValue("idempotencyKey", request.getIdempotencyKey()) + .withValue("rfcMessageId", messageId(request.getIdempotencyKey(), from)) .withValue("linkedTableName", request.getLinkedTableName()).withValue("linkedRecordId", request.getLinkedRecordId()) .withValue("senderQbit", request.getSenderQbit()).withValue("senderProcess", request.getSenderProcess()) .withValue("sendAfter", request.getSendAfter()).withValue("nextAttemptDate", request.getSendAfter() == null ? now : request.getSendAfter()) @@ -391,18 +393,10 @@ public final class EmailService { return (new EmailSendResult(existing.getValueInteger("id"), EmailSendResult.Status.DUPLICATE, List.of())); } - QRecord inserted; - try - { - inserted = SystemWrites.insert(EmailTables.MESSAGE, List.of(row), transaction).getRecords().get(0); - } - catch(QException e) - { - return (refused("Could not queue the email message")); - } + QRecord inserted = SystemWrites.insert(EmailTables.MESSAGE, List.of(row), transaction).getRecords().get(0); if(inserted.getErrors() != null && !inserted.getErrors().isEmpty()) { - return (refused("Could not queue the email message")); + throw (new QException("Could not queue the email message")); } try { @@ -596,6 +590,42 @@ public final class EmailService } } + + + /******************************************************************************* + ** Create once before insertion. A UUID seeds messages without an idempotency key. + ** The mailbox parser ignores display names; malformed domains use a local fallback. + *******************************************************************************/ + static String messageId(String key, String from) + { + String domain = "qbit-email.local"; + try + { + String address = new InternetAddress(from).getAddress(); + String candidate = address.substring(address.lastIndexOf('@') + 1); + if(address.contains("@") && candidate.matches("[A-Za-z0-9]+(?:[.-][A-Za-z0-9]+)*")) + { + domain = candidate.toLowerCase(Locale.ROOT); + } + } + catch(Exception e) + { + // A fallback also supports callers preparing legacy message rows. + } + try + { + String seed = key == null ? "qbit-email:" + UUID.randomUUID() : key; + String token = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(seed.getBytes(StandardCharsets.UTF_8))).substring(0, 32); + return ("<" + token + "@" + domain + ">"); + } + catch(java.security.NoSuchAlgorithmException e) + { + throw (new IllegalStateException("SHA-256 unavailable")); + } + } + + + /******************************************************************************* ** Refusal with no stored row. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java index 379073a..d7d03ee 100644 --- a/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailEnvelope.java @@ -31,6 +31,7 @@ import java.util.Map; public class EmailEnvelope { private String messageId; + private String rfcMessageId; private String from; private List to = List.of(); private List cc = List.of(); @@ -62,6 +63,27 @@ public class EmailEnvelope } + + /******************************************************************************* + ** Getter for the persisted RFC 5322 Message-ID, distinct from the outbox row id. + *******************************************************************************/ + public String getRfcMessageId() + { + return (rfcMessageId); + } + + + + /******************************************************************************* + ** Fluent setter for the persisted RFC 5322 Message-ID. + *******************************************************************************/ + public EmailEnvelope withRfcMessageId(String rfcMessageId) + { + this.rfcMessageId = rfcMessageId; + return (this); + } + + /******************************************************************************* ** Getter for from. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java index 5398644..746885a 100644 --- a/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailHttpSupport.java @@ -81,13 +81,17 @@ final class EmailHttpSupport Map headers = new LinkedHashMap<>(); envelope.getHeaders().forEach((name, value) -> { - if(!name.equalsIgnoreCase("Bcc") && !name.equalsIgnoreCase("Auto-Submitted") && !name.equalsIgnoreCase("X-QQQ-Email-Id")) + if(!name.equalsIgnoreCase("Bcc") && !name.equalsIgnoreCase("Auto-Submitted") && !name.equalsIgnoreCase("X-QQQ-Email-Id") && !name.equalsIgnoreCase("Message-ID")) { headers.put(name, value); } }); headers.put("Auto-Submitted", "auto-generated"); headers.put("X-QQQ-Email-Id", envelope.getMessageId()); + if(envelope.getRfcMessageId() != null) + { + headers.put("Message-ID", envelope.getRfcMessageId()); + } return (headers); } diff --git a/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java b/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java index ac6534b..c49240e 100644 --- a/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java +++ b/src/main/java/com/kingsrook/qbits/email/transport/EmailMime.java @@ -53,7 +53,26 @@ final class EmailMime noLineBreaks(envelope.getFrom()); noLineBreaks(envelope.getReplyTo()); noLineBreaks(envelope.getMessageId()); - MimeMessage message = new MimeMessage(session); + noLineBreaks(envelope.getRfcMessageId()); + String rfcMessageId = envelope.getRfcMessageId(); + MimeMessage message = new MimeMessage(session) + { + /******************************************************************************* + ** JavaMail may save again when sending; keep the persisted identity each time. + *******************************************************************************/ + @Override + protected void updateMessageID() throws MessagingException + { + if(rfcMessageId == null) + { + super.updateMessageID(); + } + else + { + setHeader("Message-ID", rfcMessageId); + } + } + }; message.setFrom(new InternetAddress(envelope.getFrom(), true)); recipients(message, Message.RecipientType.TO, envelope.getTo()); recipients(message, Message.RecipientType.CC, envelope.getCc()); diff --git a/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java index 89d3492..877ab19 100644 --- a/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java +++ b/src/main/java/com/kingsrook/qbits/email/transport/MailgunEmailTransport.java @@ -96,6 +96,10 @@ public class MailgunEmailTransport implements EmailTransport field(body, boundary, "o:tracking-opens", settings.enabled("trackOpens") ? "yes" : "no"); field(body, boundary, "o:tracking-clicks", settings.enabled("trackClicks") ? "yes" : "no"); field(body, boundary, "v:qqq_email_id", envelope.getMessageId()); + if(envelope.getRfcMessageId() != null) + { + field(body, boundary, "h:Message-Id", envelope.getRfcMessageId()); + } body.writeBytes(("--" + boundary + "\r\nContent-Disposition: form-data; name=\"message\"; filename=\"message.eml\"\r\n" + "Content-Type: message/rfc822\r\n\r\n").getBytes(StandardCharsets.UTF_8)); EmailMime.build(Session.getInstance(new Properties()), envelope).writeTo(body, new String[] {"Bcc"}); diff --git a/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java b/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java index e499533..bc61557 100644 --- a/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java +++ b/src/main/java/com/kingsrook/qbits/email/transport/SesEmailTransport.java @@ -50,6 +50,9 @@ import software.amazon.awssdk.services.sesv2.model.SesV2Exception; /******************************************************************************* ** SES v2 raw MIME transport; credentials are resolved only for an attempt. + ** The raw request carries the stored Message-ID, but SES replaces it downstream. + ** Receiver deduplication by that header is unavailable with SES; X-QQQ-Email-Id + ** remains the application correlation header. See docs/DURABLE-JOBS.md. *******************************************************************************/ public class SesEmailTransport implements EmailTransport { diff --git a/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java b/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java index 856cdef..3c24bc8 100644 --- a/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java +++ b/src/test/java/com/kingsrook/qbits/email/EmailDeliveryTest.java @@ -21,18 +21,34 @@ package com.kingsrook.qbits.email; import java.net.ServerSocket; -import java.time.*; -import java.util.*; -import java.util.concurrent.*; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import com.icegreen.greenmail.util.GreenMail; import com.icegreen.greenmail.util.ServerSetup; -import com.kingsrook.qbits.email.delivery.*; +import com.kingsrook.qbits.email.delivery.EmailDispatcher; import com.kingsrook.qbits.email.metadata.EmailTables; -import com.kingsrook.qbits.email.outbox.*; +import com.kingsrook.qbits.email.outbox.EmailCategory; +import com.kingsrook.qbits.email.outbox.EmailRequest; +import com.kingsrook.qbits.email.outbox.EmailSendResult; +import com.kingsrook.qbits.email.outbox.EmailService; import com.kingsrook.qbits.email.processes.EmailProcesses; -import com.kingsrook.qbits.email.support.*; -import com.kingsrook.qbits.email.transport.*; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qbits.email.transport.EmailTransport; +import com.kingsrook.qbits.email.transport.EmailTransportException; +import com.kingsrook.qbits.email.transport.EmailTransportRegistry; +import com.kingsrook.qbits.email.transport.EmailTransportResult; +import com.kingsrook.qbits.email.transport.SmtpEmailTransport; import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; import com.kingsrook.qqq.backend.core.actions.messaging.SendMessageAction; import com.kingsrook.qqq.backend.core.alerts.AlertResolution; @@ -41,16 +57,23 @@ import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface; import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData; import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.exceptions.QException; -import com.kingsrook.qqq.backend.core.model.actions.messaging.*; -import com.kingsrook.qqq.backend.core.model.actions.messaging.email.*; +import com.kingsrook.qqq.backend.core.model.actions.messaging.Content; +import com.kingsrook.qqq.backend.core.model.actions.messaging.MultiParty; +import com.kingsrook.qqq.backend.core.model.actions.messaging.Party; +import com.kingsrook.qqq.backend.core.model.actions.messaging.SendMessageInput; +import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailContentRole; +import com.kingsrook.qqq.backend.core.model.actions.messaging.email.EmailPartyRole; import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; import jakarta.mail.Multipart; -import org.junit.jupiter.api.*; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; -import static org.assertj.core.api.Assertions.*; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; /******************************************************************************* ** Email Delivery Test. @@ -398,6 +421,39 @@ class EmailDeliveryTest assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(1); } + /******************************************************************************* + ** A lost acknowledgement retries the stored identity through the real SMTP sink. + *******************************************************************************/ + @Test + void smtpRetryPreservesStoredMessageId() throws Exception + { + var queued = EmailService.send(request().withIdempotencyKey("job:1:row:mail")); + String stored = message(queued.messageId()).getValueString("rfcMessageId"); + AtomicInteger calls = new AtomicInteger(); + var dispatcher = fake((envelope, settings) -> + { + assertThat(envelope.getRfcMessageId()).isEqualTo(stored); + var sent = new SmtpEmailTransport().send(envelope, settings); + if(calls.incrementAndGet() == 1) + { + throw new EmailTransportException("Lost acknowledgement", false); + } + return (sent); + }); + assertThat(dispatcher.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.RETRY_WAIT); + EmailRuntime.setClock(Clock.fixed(now.plusSeconds(301), ZoneOffset.UTC)); + assertThat(dispatcher.sendNow(queued.messageId())).isEqualTo(EmailDispatcher.SendResult.SENT); + assertThat(mail.waitForIncomingEmail(2000, 2)).isTrue(); + assertThat(mail.getReceivedMessages()).hasSize(2); + for(var received : mail.getReceivedMessages()) + { + assertThat(received.getMessageID()).isEqualTo(stored); + assertThat(received.getHeader("X-QQQ-Email-Id")).containsExactly(queued.messageId().toString()); + } + assertThat(message(queued.messageId()).getValueString("rfcMessageId")).isEqualTo(stored); + assertThat(H2Support.all(EmailTables.ATTEMPT)).hasSize(2); + } + /******************************************************************************* ** Smtp Authentication And Secret Resolution Only At Send. *******************************************************************************/ @@ -477,8 +533,8 @@ class EmailDeliveryTest var second = EmailService.send(request()); fake((e, s) -> { - throw new EmailTransportException("throttle", false).withRetryAfter(Duration.ofHours(2)); - }) + throw new EmailTransportException("throttle", false).withRetryAfter(Duration.ofHours(2)); + }) .runOnce(); assertThat(message(second.messageId()).getValueInstant("nextAttemptDate")) .isEqualTo(now.plusSeconds(3960)); @@ -494,8 +550,8 @@ class EmailDeliveryTest assertThat( fake((e, s) -> { - throw new EmailTransportException("rejected", true); - }) + throw new EmailTransportException("rejected", true); + }) .sendNow(queued.messageId())) .isEqualTo(EmailDispatcher.SendResult.DEAD_LETTER); var future = EmailService.send(request()); diff --git a/src/test/java/com/kingsrook/qbits/email/EmailHomePolishTest.java b/src/test/java/com/kingsrook/qbits/email/EmailHomePolishTest.java index 660fa5c..cac3e59 100644 --- a/src/test/java/com/kingsrook/qbits/email/EmailHomePolishTest.java +++ b/src/test/java/com/kingsrook/qbits/email/EmailHomePolishTest.java @@ -30,9 +30,9 @@ import java.util.Set; import com.kingsrook.qbits.email.app.EmailHome; import com.kingsrook.qbits.email.metadata.EmailEnums; import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.outbox.EmailCategory; import com.kingsrook.qbits.email.render.ContrastCheck; import com.kingsrook.qbits.email.render.EmailLayout; -import com.kingsrook.qbits.email.outbox.EmailCategory; import com.kingsrook.qbits.email.support.SystemWrites; import com.kingsrook.qqq.backend.core.actions.dashboard.RenderWidgetAction; import com.kingsrook.qqq.backend.core.actions.tables.GetAction; @@ -66,7 +66,9 @@ class EmailHomePolishTest @BeforeEach void setup() throws Exception { - H2Support.setup(c -> { }); + H2Support.setup(c -> + { + }); EmailRuntime.setClock(Clock.fixed(NOW, ZoneOffset.UTC)); H2Support.provider(25); } diff --git a/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java b/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java index d95fa2b..c4929ee 100644 --- a/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java +++ b/src/test/java/com/kingsrook/qbits/email/EmailMetadataTest.java @@ -20,14 +20,24 @@ package com.kingsrook.qbits.email; -import java.util.*; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; import com.kingsrook.qbits.email.app.EmailApp; -import com.kingsrook.qbits.email.metadata.*; -import com.kingsrook.qbits.email.outbox.*; -import com.kingsrook.qbits.email.permissions.*; -import com.kingsrook.qbits.email.processes.*; -import com.kingsrook.qbits.email.support.*; -import com.kingsrook.qqq.backend.core.actions.tables.*; +import com.kingsrook.qbits.email.metadata.EmailTableCustomizers; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.outbox.AdHocEmailRenderer; +import com.kingsrook.qbits.email.outbox.EmailRequest; +import com.kingsrook.qbits.email.outbox.EmailSendResult; +import com.kingsrook.qbits.email.outbox.EmailService; +import com.kingsrook.qbits.email.permissions.EmailPermissionChecker; +import com.kingsrook.qbits.email.processes.EmailProcesses; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; @@ -36,10 +46,14 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; -import com.kingsrook.qqq.backend.core.model.metadata.fields.*; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; -import org.junit.jupiter.api.*; -import static org.assertj.core.api.Assertions.*; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; /******************************************************************************* ** Email Metadata Test. @@ -412,7 +426,7 @@ class EmailMetadataTest "MAILGUN", "GMAIL_API", "future")) - { + { assertThat( new EmailQBitConfig() .withAllowedEnvNames(Set.of("CUSTOM")) diff --git a/src/test/java/com/kingsrook/qbits/email/EmailProcessesTest.java b/src/test/java/com/kingsrook/qbits/email/EmailProcessesTest.java index 6f92573..a287e95 100644 --- a/src/test/java/com/kingsrook/qbits/email/EmailProcessesTest.java +++ b/src/test/java/com/kingsrook/qbits/email/EmailProcessesTest.java @@ -20,20 +20,28 @@ package com.kingsrook.qbits.email; -import java.time.*; -import java.util.*; -import com.icegreen.greenmail.util.*; -import com.kingsrook.qbits.email.delivery.*; -import com.kingsrook.qbits.email.metadata.*; -import com.kingsrook.qbits.email.outbox.*; -import com.kingsrook.qbits.email.processes.*; -import com.kingsrook.qbits.email.support.*; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.Map; +import com.icegreen.greenmail.util.GreenMail; +import com.icegreen.greenmail.util.ServerSetup; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.outbox.EmailRequest; +import com.kingsrook.qbits.email.outbox.EmailService; +import com.kingsrook.qbits.email.processes.EmailProcesses; +import com.kingsrook.qbits.email.support.EmailStore; +import com.kingsrook.qbits.email.support.SystemWrites; import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction; import com.kingsrook.qqq.backend.core.context.QContext; -import com.kingsrook.qqq.backend.core.model.actions.processes.*; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput; import com.kingsrook.qqq.backend.core.model.data.QRecord; -import org.junit.jupiter.api.*; -import static org.assertj.core.api.Assertions.*; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; /******************************************************************************* ** Email Processes Test. diff --git a/src/test/java/com/kingsrook/qbits/email/EmailReviewFixesTest.java b/src/test/java/com/kingsrook/qbits/email/EmailReviewFixesTest.java index 58f4c0d..d0e5fb8 100644 --- a/src/test/java/com/kingsrook/qbits/email/EmailReviewFixesTest.java +++ b/src/test/java/com/kingsrook/qbits/email/EmailReviewFixesTest.java @@ -109,7 +109,7 @@ class EmailReviewFixesTest var schema = new VariableSchema("[{\"name\":\"items\",\"type\":\"LIST\"},{\"name\":\"value\",\"type\":\"STRING\"}]"); var loop = engine.parse("{{#each items}}{{#each items}}{{/each}}{{/each}}", schema); assertThatThrownBy(() -> loop.render(Map.of("items", Collections.nCopies(100, "")), EmailTemplateEngine.Context.TEXT)).hasMessage("Template output is too large"); - assertThatThrownBy(() -> engine.parse("{{value}}", schema).render(Map.of("value", "\u00e9".repeat(600_000)), EmailTemplateEngine.Context.TEXT)) + assertThatThrownBy(() -> engine.parse("{{value}}", schema).render(Map.of("value", "é".repeat(600_000)), EmailTemplateEngine.Context.TEXT)) .hasMessage("Template output is too large"); assertThat(engine.parse("{{#if value}}".repeat(8) + "ok" + "{{/if}}".repeat(8), schema).render(Map.of("value", true), EmailTemplateEngine.Context.TEXT)).isEqualTo("ok"); assertThatThrownBy(() -> engine.parse("{{#if value}}".repeat(9) + "{{/if}}".repeat(9), schema)).hasMessageContaining("8 levels"); @@ -179,7 +179,10 @@ class EmailReviewFixesTest { assertThatThrownBy(() -> EmailTemplateEngine.validateHtml(source)).hasMessageContaining("must be quoted"); } - EmailRuntime.config().withMjmlCompiler(source -> { throw new AssertionError("Unsafe MJML reached compiler"); }); + EmailRuntime.config().withMjmlCompiler(source -> + { + throw new AssertionError("Unsafe MJML reached compiler"); + }); QRecord mjml = EmailTemplatesTest.draft("includes", "MJML", "[]", "{}", ""); assertThatThrownBy(() -> EmailVersions.activate(mjml.getValueInteger("id"))).hasMessage("MJML includes are not allowed"); assertThat(preview(mjml)).contains("MJML includes are not allowed"); @@ -192,7 +195,10 @@ class EmailReviewFixesTest void viewerNeverCompiles() throws Exception { QRecord version = EmailTemplatesTest.draft("viewer", "MJML", "[]", "{}", ""); - EmailRuntime.config().withMjmlCompiler(source -> { throw new AssertionError("Viewer compiled MJML"); }); + EmailRuntime.config().withMjmlCompiler(source -> + { + throw new AssertionError("Viewer compiled MJML"); + }); QContext.getQSession().setPermissions(Set.of("email.view")); assertThat(preview(version)).contains("Not compiled yet. Activate the version or use Preview as a manager."); QContext.getQSession().setPermissions(Set.of("email.manage")); @@ -270,7 +276,10 @@ class EmailReviewFixesTest QRecord original = EmailStore.get(EmailTables.MESSAGE, first.messageId(), null); SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", second.messageId()).withValue("htmlBody", original.getValue("htmlBody"))); var dispatcher = new com.kingsrook.qbits.email.delivery.EmailDispatcher(new com.kingsrook.qbits.email.transport.EmailTransportRegistry() - .withTransport("SMTP", (envelope, settings) -> { throw new AssertionError("Swapped ciphertext reached transport"); })); + .withTransport("SMTP", (envelope, settings) -> + { + throw new AssertionError("Swapped ciphertext reached transport"); + })); assertThat(dispatcher.sendNow(second.messageId())).isEqualTo(com.kingsrook.qbits.email.delivery.EmailDispatcher.SendResult.RETRY_WAIT); } diff --git a/src/test/java/com/kingsrook/qbits/email/EmailTemplatesTest.java b/src/test/java/com/kingsrook/qbits/email/EmailTemplatesTest.java index bff3466..cc08cb9 100644 --- a/src/test/java/com/kingsrook/qbits/email/EmailTemplatesTest.java +++ b/src/test/java/com/kingsrook/qbits/email/EmailTemplatesTest.java @@ -148,7 +148,10 @@ class EmailTemplatesTest assertThat(((RawHTML) preview.getWidgetData()).getHtml()).contains("

Email

"); assertThat(EmailStore.get(EmailTables.VERSION, version.getValueInteger("id"), null).getValue("compiledHtml")).isNull(); EmailVersions.activate(version.getValueInteger("id")); - EmailRuntime.config().withMjmlCompiler(source -> { throw new AssertionError("Compiler ran at send time"); }); + EmailRuntime.config().withMjmlCompiler(source -> + { + throw new AssertionError("Compiler ran at send time"); + }); var result = EmailService.send(new EmailRequest().withTemplateName("mjml").withLocale("fr").withTo("p@example.com")); assertThat(result.status()).isEqualTo(EmailSendResult.Status.QUEUED); QRecord row = EmailStore.get(EmailTables.MESSAGE, result.messageId(), null); @@ -295,7 +298,10 @@ class EmailTemplatesTest EmailRuntime.config().withSecretResolver(name -> null); assertThat(com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("1", "address", "NOTIFICATION")).isNull(); assertThatThrownBy(() -> com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("1", "address", "MARKETING")).hasMessageContaining("QQQ_EMAIL_UNSUBSCRIBE_KEY"); - EmailRuntime.config().withSecretResolver(name -> { throw new IllegalArgumentException("resolver detail"); }); + EmailRuntime.config().withSecretResolver(name -> + { + throw new IllegalArgumentException("resolver detail"); + }); assertThatThrownBy(() -> com.kingsrook.qbits.email.render.EmailUnsubscribeLinks.create("1", "person@example.com", "NOTIFICATION")) .hasMessage("Opt-out email needs publicBaseUrl and QQQ_EMAIL_UNSUBSCRIBE_KEY"); } diff --git a/src/test/java/com/kingsrook/qbits/email/H2Support.java b/src/test/java/com/kingsrook/qbits/email/H2Support.java index e1083b9..a0970c2 100644 --- a/src/test/java/com/kingsrook/qbits/email/H2Support.java +++ b/src/test/java/com/kingsrook/qbits/email/H2Support.java @@ -85,7 +85,7 @@ public final class H2Support *******************************************************************************/ public static QInstance instance(RDBMSBackendMetaData backend, Consumer tweak) throws Exception - { + { QInstance instance = new QInstance(); instance.withInstanceDefaultAuthentication( new QAuthenticationMetaData() diff --git a/src/test/java/com/kingsrook/qbits/email/PhiCanaryEgressTest.java b/src/test/java/com/kingsrook/qbits/email/PhiCanaryEgressTest.java index e744078..d1c4d42 100644 --- a/src/test/java/com/kingsrook/qbits/email/PhiCanaryEgressTest.java +++ b/src/test/java/com/kingsrook/qbits/email/PhiCanaryEgressTest.java @@ -81,7 +81,17 @@ class PhiCanaryEgressTest @BeforeEach void setup() throws Exception { - canary = "CANARY-PHI-" + UUID.randomUUID(); + //////////////////////////////////////////////////////////////////////// + // the random part is mapped from hex to the letters g-v: a five-char // + // window of a hex canary could match any hex digest (body hashes, // + // Message-IDs) by chance; these windows can only match real leakage // + //////////////////////////////////////////////////////////////////////// + StringBuilder random = new StringBuilder(); + for(char c : UUID.randomUUID().toString().replace("-", "").toCharArray()) + { + random.append((char) ('g' + Character.digit(c, 16))); + } + canary = "CANARY-PHI-" + random; H2Support.setup(c -> c.withMaxAttempts(2).withSecretResolver(PhiCanaryEgressTest::secret) .withClassificationOverrides(Map.of("test_patient.name", EmailClassifications.Level.PHI))); H2Support.provider(25); diff --git a/src/test/java/com/kingsrook/qbits/email/ProcessRunner.java b/src/test/java/com/kingsrook/qbits/email/ProcessRunner.java index 3110a8a..2fa0b19 100644 --- a/src/test/java/com/kingsrook/qbits/email/ProcessRunner.java +++ b/src/test/java/com/kingsrook/qbits/email/ProcessRunner.java @@ -66,7 +66,7 @@ public final class ProcessRunner public static Lines runWithValues( String process, java.util.Map values, Serializable... ids) throws Exception - { + { RunProcessInput input = new RunProcessInput(); input.setProcessName(process); input.addValue("recordsParam", "recordIds"); diff --git a/src/test/java/com/kingsrook/qbits/email/outbox/EmailEffectsTest.java b/src/test/java/com/kingsrook/qbits/email/outbox/EmailEffectsTest.java new file mode 100644 index 0000000..fe145e6 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/email/outbox/EmailEffectsTest.java @@ -0,0 +1,212 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.email.outbox; + + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.util.HexFormat; +import java.util.List; +import com.kingsrook.qbits.email.H2Support; +import com.kingsrook.qbits.email.metadata.EmailTables; +import com.kingsrook.qbits.email.support.SystemWrites; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Durable handoff semantics and stored identity using the real H2 outbox. + *******************************************************************************/ +class EmailEffectsTest +{ + /******************************************************************************* + ** Prepare a provider without opening a transport connection. + *******************************************************************************/ + @BeforeEach + void setup() throws Exception + { + H2Support.setup(config -> + { + }); + H2Support.provider(25); + } + + + + /******************************************************************************* + ** Release the per-thread context. + *******************************************************************************/ + @AfterEach + void cleanup() + { + QContext.clear(); + } + + + + /******************************************************************************* + ** A plain message uses the configured sender. + *******************************************************************************/ + private EmailRequest request() + { + return (new EmailRequest().withTo("person@example.com").withSubject("Effect").withTextBody("Body")); + } + + + + /******************************************************************************* + ** Replays return the same persisted reference without another row. + *******************************************************************************/ + @Test + void handoffDeduplicatesAndLookupsReturnCurrentReferences() throws Exception + { + String key = "job:2:row:email"; + EmailRequest request = request(); + var first = EmailEffects.send(request, key); + var second = EmailEffects.send(request(), key); + assertThat(request.getIdempotencyKey()).isEqualTo(key); + assertThat(first.outcome()).isEqualTo(EmailEffectOutcome.QUEUED_OR_DUPLICATE); + assertThat(second).isEqualTo(first); + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1); + String token = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(key.getBytes(StandardCharsets.UTF_8))).substring(0, 32); + assertThat(first.message().rfcMessageId()).isEqualTo("<" + token + "@example.com>"); + assertThat(first.message().status()).isEqualTo("QUEUED"); + assertThat(first.message().providerMessageId()).isNull(); + SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", first.message().id()) + .withValue("status", "SENT").withValue("providerMessageId", "provider-reference")); + QContext.init(QContext.getQInstance(), new QSession().withPermissions(List.of())); + var found = EmailEffects.findByIdempotencyKey(key).orElseThrow(); + assertThat(found.id()).isEqualTo(first.message().id()); + assertThat(found.status()).isEqualTo("SENT"); + assertThat(found.providerMessageId()).isEqualTo("provider-reference"); + assertThat(EmailEffects.findByRfcMessageId(first.message().rfcMessageId())).contains(found); + assertThat(EmailEffects.findByIdempotencyKey("missing")).isEmpty(); + assertThat(EmailEffects.findByRfcMessageId("")).isEmpty(); + assertThat(EmailEffects.findByIdempotencyKey(null)).isEmpty(); + assertThat(EmailEffects.findByRfcMessageId(null)).isEmpty(); + assertThat(EmailEffects.findByIdempotencyKey(" ")).isEmpty(); + assertThat(EmailEffects.findByRfcMessageId(" ")).isEmpty(); + } + + + + /******************************************************************************* + ** Conflicting or unusable requests never enqueue or change an existing key. + *******************************************************************************/ + @Test + void refusesConflictingKeysAndInvalidRequests() throws Exception + { + EmailRequest conflict = request().withIdempotencyKey("other"); + assertThat(EmailEffects.send(conflict, "effect").outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(conflict.getIdempotencyKey()).isEqualTo("other"); + assertThat(EmailEffects.send(request(), null).outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(EmailEffects.send(request(), " ").outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(EmailEffects.send(request(), "a".repeat(501)).outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(EmailEffects.send(null, "effect").outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + var invalid = EmailEffects.send(request().withSubject(null), "invalid"); + assertThat(invalid.outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(invalid.message()).isNull(); + assertThat(H2Support.all(EmailTables.MESSAGE)).isEmpty(); + assertThat(EmailEffects.send(request().withIdempotencyKey("same"), "same").outcome()).isEqualTo(EmailEffectOutcome.QUEUED_OR_DUPLICATE); + } + + + + /******************************************************************************* + ** A suppressed audit row stays refused on replay, including duplicate results. + *******************************************************************************/ + @Test + void suppressedRecipientsAreRefusedOnEveryHandoff() throws Exception + { + H2Support.insert(EmailTables.SUPPRESSION, new QRecord().withValue("address", "person@example.com") + .withValue("scope", "ALL").withValue("reason", "HARD_BOUNCE")); + var first = EmailEffects.send(request(), "suppressed"); + var second = EmailEffects.send(request(), "suppressed"); + assertThat(first.outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(first.message().status()).isEqualTo("SUPPRESSED"); + assertThat(second).isEqualTo(first); + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1); + } + + + + /******************************************************************************* + ** Never acknowledge an uncommitted or deferred handoff to the jobs dispatcher. + *******************************************************************************/ + @Test + void callerTransactionsAreRefused() throws Exception + { + try(QBackendTransaction transaction = QBackendTransaction.openFor(new InsertInput(EmailTables.MESSAGE))) + { + assertThat(EmailEffects.send(request().withTransaction(transaction, "rdbms"), "same-backend").outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + assertThat(EmailEffects.send(request().withTransaction(transaction, "other"), "other-backend").outcome()).isEqualTo(EmailEffectOutcome.REFUSED); + transaction.commit(); + } + assertThat(H2Support.all(EmailTables.MESSAGE)).isEmpty(); + } + + + + /******************************************************************************* + ** Database uniqueness failures must escape, rather than become terminal refusals. + *******************************************************************************/ + @Test + void storageFailurePropagatesAndRfcIdIsUnique() throws Exception + { + var first = EmailEffects.send(request(), "collision"); + SystemWrites.updateOne(EmailTables.MESSAGE, new QRecord().withValue("id", first.message().id()).withValue("idempotencyKey", "changed")); + assertThatThrownBy(() -> EmailEffects.send(request(), "collision")).isInstanceOf(QException.class); + assertThat(H2Support.all(EmailTables.MESSAGE)).hasSize(1); + assertThat(EmailEffects.findByIdempotencyKey("collision")).isEmpty(); + } + + + + /******************************************************************************* + ** Identity hashing is deterministic, ignores display names and has a safe fallback. + *******************************************************************************/ + @Test + void generationAndUnkeyedMessages() throws Exception + { + assertThat(EmailService.messageId("key", "Display ")) + .isEqualTo(EmailService.messageId("key", "sender@example.com")); + for(String from : new String[] {null, "local", "broken <", "sender@[]"}) + { + assertThat(EmailService.messageId("key", from)).matches("<[0-9a-f]{32}@qbit-email\\.local>"); + } + var first = EmailService.send(request()); + var second = EmailService.send(request()); + var rows = H2Support.all(EmailTables.MESSAGE); + assertThat(rows).hasSize(2); + assertThat(rows.get(0).getValueString("rfcMessageId")).matches("<[0-9a-f]{32}@example\\.com>") + .isNotEqualTo(rows.get(1).getValueString("rfcMessageId")); + assertThat(first.messageId()).isNotEqualTo(second.messageId()); + } +} diff --git a/src/test/java/com/kingsrook/qbits/email/tracking/EmailEventParserTest.java b/src/test/java/com/kingsrook/qbits/email/tracking/EmailEventParserTest.java index a9b3a99..b85af86 100644 --- a/src/test/java/com/kingsrook/qbits/email/tracking/EmailEventParserTest.java +++ b/src/test/java/com/kingsrook/qbits/email/tracking/EmailEventParserTest.java @@ -26,8 +26,12 @@ import org.json.JSONObject; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; -import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.*; -import static org.assertj.core.api.Assertions.*; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.message; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.provider; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; /******************************************************************************* ** Provider-specific normalization contracts use realistic fixture envelopes. diff --git a/src/test/java/com/kingsrook/qbits/email/tracking/EmailInboxProcessorTest.java b/src/test/java/com/kingsrook/qbits/email/tracking/EmailInboxProcessorTest.java index ef4160e..84e04ff 100644 --- a/src/test/java/com/kingsrook/qbits/email/tracking/EmailInboxProcessorTest.java +++ b/src/test/java/com/kingsrook/qbits/email/tracking/EmailInboxProcessorTest.java @@ -47,8 +47,12 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.CsvSource; import org.junit.jupiter.params.provider.ValueSource; -import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.*; -import static org.assertj.core.api.Assertions.*; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.message; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.provider; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource; +import static org.assertj.core.api.Assertions.assertThat; /******************************************************************************* ** Durable processing, rollback, suppression and race acceptance on real H2. @@ -422,6 +426,7 @@ class EmailInboxProcessorTest /******************************************************************************* ** Test adapter implementation. *******************************************************************************/ + @Override public void raise(com.kingsrook.qqq.backend.core.alerts.AlertSignal signal) { diff --git a/src/test/java/com/kingsrook/qbits/email/tracking/EmailWebhookRoutesTest.java b/src/test/java/com/kingsrook/qbits/email/tracking/EmailWebhookRoutesTest.java index 7efa2ef..515f1d9 100644 --- a/src/test/java/com/kingsrook/qbits/email/tracking/EmailWebhookRoutesTest.java +++ b/src/test/java/com/kingsrook/qbits/email/tracking/EmailWebhookRoutesTest.java @@ -32,7 +32,6 @@ import java.time.ZoneOffset; import java.util.ArrayList; import java.util.Base64; import java.util.List; -import java.util.Map; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; import com.kingsrook.qbits.email.EmailRuntime; @@ -52,8 +51,18 @@ import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; -import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.*; -import static org.assertj.core.api.Assertions.*; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.TOPIC; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.certificateTime; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.message; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.provider; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.response; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signMailgun; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signSns; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; /******************************************************************************* ** Real random-port HTTP exercises the exact route provider and database boundary. diff --git a/src/test/java/com/kingsrook/qbits/email/tracking/TrackingTestSupport.java b/src/test/java/com/kingsrook/qbits/email/tracking/TrackingTestSupport.java index 9064302..0676a25 100644 --- a/src/test/java/com/kingsrook/qbits/email/tracking/TrackingTestSupport.java +++ b/src/test/java/com/kingsrook/qbits/email/tracking/TrackingTestSupport.java @@ -20,6 +20,9 @@ package com.kingsrook.qbits.email.tracking; +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; +import javax.net.ssl.SSLSession; import java.net.URI; import java.net.http.HttpHeaders; import java.net.http.HttpRequest; @@ -36,9 +39,6 @@ import java.util.Base64; import java.util.List; import java.util.Map; import java.util.Optional; -import javax.crypto.Mac; -import javax.crypto.spec.SecretKeySpec; -import javax.net.ssl.SSLSession; import com.kingsrook.qbits.email.H2Support; import com.kingsrook.qbits.email.metadata.EmailTables; import com.kingsrook.qbits.email.support.SystemWrites; diff --git a/src/test/java/com/kingsrook/qbits/email/tracking/WebhookVerifiersTest.java b/src/test/java/com/kingsrook/qbits/email/tracking/WebhookVerifiersTest.java index ae2d02d..652aa77 100644 --- a/src/test/java/com/kingsrook/qbits/email/tracking/WebhookVerifiersTest.java +++ b/src/test/java/com/kingsrook/qbits/email/tracking/WebhookVerifiersTest.java @@ -31,8 +31,16 @@ import org.json.JSONObject; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.ValueSource; -import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.*; -import static org.assertj.core.api.Assertions.*; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.NOW; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.SECRET; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.TOPIC; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.certificateTime; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.resource; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.response; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signMailgun; +import static com.kingsrook.qbits.email.tracking.TrackingTestSupport.signSns; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; /******************************************************************************* ** Crypto boundaries use genuine signatures and a public test-only RSA fixture. diff --git a/src/test/java/com/kingsrook/qbits/email/transport/EmailHttpTransportsTest.java b/src/test/java/com/kingsrook/qbits/email/transport/EmailHttpTransportsTest.java index b7e2477..e8fd735 100644 --- a/src/test/java/com/kingsrook/qbits/email/transport/EmailHttpTransportsTest.java +++ b/src/test/java/com/kingsrook/qbits/email/transport/EmailHttpTransportsTest.java @@ -21,6 +21,7 @@ package com.kingsrook.qbits.email.transport; +import javax.net.ssl.SSLSession; import java.io.ByteArrayInputStream; import java.io.IOException; import java.net.InetSocketAddress; @@ -44,7 +45,6 @@ import java.util.Properties; import java.util.Set; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; -import javax.net.ssl.SSLSession; import com.fasterxml.jackson.databind.JsonNode; import com.kingsrook.qbits.email.EmailQBitConfig; import com.kingsrook.qbits.email.EmailRuntime; @@ -147,9 +147,9 @@ class EmailHttpTransportsTest *******************************************************************************/ private static EmailEnvelope envelope() { - return (new EmailEnvelope().withMessageId("42").withFrom("Sender ") + return (new EmailEnvelope().withMessageId("42").withRfcMessageId("<0123456789abcdef0123456789abcdef@example.com>").withFrom("Sender ") .withTo(List.of("Person ")).withCc(List.of("cc@example.com")).withBcc(List.of("bcc@example.com")) - .withReplyTo("Replies ").withSubject("Hello \u00e9").withTextBody("Plain \u00e9").withHtmlBody("Hello \u00e9") + .withReplyTo("Replies ").withSubject("Hello é").withTextBody("Plain é").withHtmlBody("Hello é") .withHeaders(Map.of("List-Unsubscribe", "", "List-Unsubscribe-Post", "List-Unsubscribe=One-Click", "X-Custom", "kept"))); } @@ -283,6 +283,11 @@ class EmailHttpTransportsTest assertThat(json.path("MessageStream").asText()).isEqualTo(tracking ? "broadcasts" : "outbound"); assertThat(json.path("TrackOpens").asBoolean()).isEqualTo(tracking); assertThat(json.path("TrackLinks").asText()).isEqualTo(tracking ? "HtmlAndText" : "None"); + assertThat(json.path("Headers")).anySatisfy(header -> + { + assertThat(header.path("Name").asText()).isEqualTo("Message-ID"); + assertThat(header.path("Value").asText()).isEqualTo(envelope().getRfcMessageId()); + }); assertThat(json.path("Headers").toString()).contains("List-Unsubscribe", "List-Unsubscribe-Post", "X-QQQ-Email-Id", "42", "Auto-Submitted"); assertThat(json.toString()).doesNotContain(SECRET); } @@ -303,6 +308,7 @@ class EmailHttpTransportsTest return (response(202, "", Map.of("x-message-id", List.of("sg-id")))); }); Map headers = new java.util.HashMap<>(envelope().getHeaders()); + headers.put("message-id", ""); for(String reserved : List.of("X-SG-ID", "x-sg-eid", "Received", "DKIM-Signature", "Content-Type", "Content-Transfer-Encoding", "To", "From", "Subject", "Reply-To", "CC", "BcC")) { @@ -329,6 +335,8 @@ class EmailHttpTransportsTest assertThat(json.at("/content/1/type").asText()).isEqualTo("text/html"); assertThat(json.at("/content/1/value").asText()).isEqualTo(envelope().getHtmlBody()); assertThat(json.at("/custom_args/qqq_email_id").asText()).isEqualTo("42"); + assertThat(json.at("/headers/Message-ID").asText()).isEqualTo(envelope().getRfcMessageId()); + assertThat(json.path("headers").has("message-id")).isFalse(); assertThat(json.path("headers").toString()).doesNotContain("must-be-removed").contains("List-Unsubscribe", "List-Unsubscribe-Post"); assertThat(json.at("/tracking_settings/open_tracking/enable").asBoolean()).isEqualTo(tracking); assertThat(json.at("/tracking_settings/click_tracking/enable").asBoolean()).isEqualTo(tracking); @@ -371,6 +379,8 @@ class EmailHttpTransportsTest assertThat(mime).doesNotContain("bcc@example.com", "Bcc:", SECRET); MimeMessage message = new MimeMessage(Session.getInstance(new Properties()), new ByteArrayInputStream(mime.getBytes(StandardCharsets.UTF_8))); assertThat(message.getSubject()).isEqualTo(envelope().getSubject()); + assertThat(message.getMessageID()).isEqualTo(envelope().getRfcMessageId()); + assertThat(multipart).contains("name=\"h:Message-Id\"\r\n\r\n" + envelope().getRfcMessageId() + "\r\n"); assertThat(message.getHeader("X-QQQ-Email-Id", null)).isEqualTo("42"); assertThat(message.getHeader("List-Unsubscribe-Post", null)).isEqualTo("List-Unsubscribe=One-Click"); assertThat(message.getRecipients(jakarta.mail.Message.RecipientType.CC)[0].toString()).isEqualTo("cc@example.com"); diff --git a/src/test/java/com/kingsrook/qbits/email/transport/EmailTransportsTest.java b/src/test/java/com/kingsrook/qbits/email/transport/EmailTransportsTest.java index 6d7bcc6..a5911a1 100644 --- a/src/test/java/com/kingsrook/qbits/email/transport/EmailTransportsTest.java +++ b/src/test/java/com/kingsrook/qbits/email/transport/EmailTransportsTest.java @@ -47,8 +47,15 @@ import jakarta.mail.internet.InternetAddress; import jakarta.mail.internet.MimeMessage; import org.junit.jupiter.api.Test; import software.amazon.awssdk.services.sesv2.SesV2Client; -import software.amazon.awssdk.services.sesv2.model.*; -import static org.assertj.core.api.Assertions.*; +import software.amazon.awssdk.services.sesv2.model.Message; +import software.amazon.awssdk.services.sesv2.model.MessageRejectedException; +import software.amazon.awssdk.services.sesv2.model.SendEmailRequest; +import software.amazon.awssdk.services.sesv2.model.SendEmailResponse; +import software.amazon.awssdk.services.sesv2.model.SesV2Exception; +import software.amazon.awssdk.services.sesv2.model.TooManyRequestsException; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.fail; /******************************************************************************* ** Email Transports Test. @@ -61,7 +68,7 @@ class EmailTransportsTest EmailEnvelope envelope() { return (new EmailEnvelope() - .withMessageId("123") + .withMessageId("123").withRfcMessageId("<0123456789abcdef0123456789abcdef@example.com>") .withFrom("sender@example.com") .withTo(List.of("a@example.com")) .withBcc(List.of("hidden@example.com")) @@ -135,10 +142,26 @@ class EmailTransportsTest Session.getInstance(new Properties()), new ByteArrayInputStream(captured.get().content().raw().data().asByteArray())); assertThat(mime.getHeader("Bcc")).isNull(); + assertThat(mime.getMessageID()).isEqualTo(envelope().getRfcMessageId()); + assertThat(mime.getHeader("X-QQQ-Email-Id")).containsExactly("123"); assertThat(mime.getHeader("X-Test")).containsExactly("yes"); assertThat(mime.getHeader("Auto-Submitted")).containsExactly("auto-generated"); } + /******************************************************************************* + ** Repeated JavaMail saves cannot replace the identity or accept a custom override. + *******************************************************************************/ + @Test + void mimeIdentitySurvivesSaveChanges() throws Exception + { + var envelope = envelope().withHeaders(Map.of("message-id", "")); + var message = EmailMime.build(Session.getInstance(new Properties()), envelope); + message.saveChanges(); + message.saveChanges(); + assertThat(message.getHeader("Message-ID")).containsExactly(envelope.getRfcMessageId()); + assertThat(EmailMime.build(Session.getInstance(new Properties()), envelope().withRfcMessageId(null)).getMessageID()).isNotBlank(); + } + /******************************************************************************* ** Ses Classifies Throttling Rejection And Server Failure. *******************************************************************************/ @@ -150,7 +173,7 @@ class EmailTransportsTest TooManyRequestsException.builder().statusCode(429).message("secret").build(), MessageRejectedException.builder().statusCode(400).message("secret").build(), SesV2Exception.builder().statusCode(503).message("secret").build())) - { + { var transport = new SesEmailTransport( builder -> @@ -313,7 +336,7 @@ class EmailTransportsTest for(String newline : List.of("\r", "\n")) { String bad = "a@example.com" + newline + "Bcc: hidden@example.com"; - for(EmailEnvelope envelope : List.of(envelope().withSubject(bad), envelope().withFrom(bad), envelope().withReplyTo(bad), + for(EmailEnvelope envelope : List.of(envelope().withRfcMessageId(bad), envelope().withSubject(bad), envelope().withFrom(bad), envelope().withReplyTo(bad), envelope().withTo(List.of(bad)), envelope().withCc(List.of(bad)), envelope().withBcc(List.of(bad)), envelope().withHeaders(Map.of("X-Test", bad)), envelope().withFrom("broken <"))) { diff --git a/src/test/resources/email-h2.sql b/src/test/resources/email-h2.sql index ae1ea2e..264fdd4 100644 --- a/src/test/resources/email-h2.sql +++ b/src/test/resources/email-h2.sql @@ -75,6 +75,7 @@ CREATE TABLE email_message ( phi_approved BOOLEAN, unsubscribe_url TEXT, idempotency_key VARCHAR(500), + rfc_message_id VARCHAR(500), provider_id INTEGER, provider_message_id VARCHAR(500), attempt_count INTEGER, @@ -93,7 +94,8 @@ CREATE TABLE email_message ( sender_process VARCHAR(500), suppressed_addresses TEXT, send_after TIMESTAMP, - UNIQUE (idempotency_key) + UNIQUE (idempotency_key), + UNIQUE (rfc_message_id) ); CREATE TABLE email_message_attempt (