Compare commits

..
Author SHA1 Message Date
james.maesandClaude Opus 5.5 4471a13e78 test(help): compare the help bundle version with the artifact version, not a hardcoded SNAPSHOT
The bundle version is now Maven-filtered, so a literal SNAPSHOT in the test
fails every release build. Surefire passes the artifact version in
qbit.artifactVersion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:07:33 -05:00
james.maes c39d78901c Merge pull request #20 from QRun-IO/fix/help-bundle-version
fix(help): help bundle version comes from the build, not a hardcoded SNAPSHOT
2026-10-14 22:37:20 -05:00
james.maesandClaude Opus 5.5 1e25fe54d2 fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT
On release branches the bundle's version and the producer's differed, so the
bundle test that asserts they match failed, and released bundles recorded
SNAPSHOT as their provenance. help/*.yaml is now filtered:
version: ${project.version}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 22:09:48 -05:00
james.maes 0fe99db48d Merge pull request #19 from QRun-IO/feature/help-bundle
feat(help): qbit-edi help bundle via QQQ help bundles, with coverage gate
2026-10-14 20:45:30 -05:00
james.maesandClaude Opus 5.5 b08057180b feat(help): ship the qbit-edi help bundle through QQQ help bundles, with coverage gate
Host-neutral help (deployment-specific facts removed), registered with
HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:27:22 -05:00
james.maes 79b85a163d Merge pull request #18 from QRun-IO/feature/tables-populated-defaults
feat: every edi table explains itself; Setup status; Install sample maps
2026-10-14 17:42:45 -05:00
james.maesandClaude Opus 5.5 4e32aebc00 fix: X12 routing honors identity isActive; tests for draft maps and host help
- X12 inbound routing and outbound building ignore inactive identities, as
  EDIFACT does (UPGRADE note).
- Test that a draft map bound to an outbound document is never used (it
  already was not) and that inactive identities stop sending.
- hostHelpIsKept test; drop the unfounded concurrency claim on Install
  sample maps; plain wording in the help bundle header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:32:54 -05:00
james.maesandClaude Opus 5.5 7bafab62cc docs: upgrade note wording
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:17:54 -05:00
james.maesandClaude Opus 5.5 a1914f7553 feat: table help, Setup status, Install sample maps, tables guardrail
Every edi_* table and the app get help bundle text saying how rows get
created and the setup order. The home gets a Setup status section. A new
Install sample maps table action adds the bundled synthetic maps as
inactive, unbound drafts (insert-only by name).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:17:41 -05:00
james.maes 51a5eaef74 Merge pull request #17 from QRun-IO/fix/repeat-schedule-time-zone
fix: no cron time zone on repeat-seconds worker schedules
2026-10-14 15:10:02 -05:00
james.maesandClaude Opus 5.5 47c3aac1b2 fix: no cron time zone on repeat-seconds worker schedules
OutboundProcesses.process set cronTimeZoneId("UTC") on every schedule.
Core instance validation refuses a time zone on a non-cron schedule, so a
host that configures a scheduler (kof22) failed validation at start. Only
cron schedules get UTC now; OutboundTest covers both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:59:12 -05:00
james.maes bf8f40959a Merge pull request #16 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:57:01 -05:00
james.maesandClaude Opus 5.5 f7c031b793 ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:06:51 -05:00
james.maes cd9423e966 Merge pull request #15 from QRun-IO/feature/phase-f-mapper-api
feat: phase f1 - spec'd UI API extension for the EDI mapper and sandbox
2026-10-14 09:58:48 -05:00
james.maes d97418447b Merge develop into feature/phase-f-mapper-api 2026-10-14 09:49:11 -05:00
james.maesandClaude Opus 5.5 c15872a373 feat: phase f1 - spec'd UI API extension for the EDI mapper
- MiddlewareSpecExtension "edi" 1.0.0 under /qqq/v1/qbit/edi:
  GET guides/{guideId}/tree, GET maps/{mapId}/target,
  POST sandbox/preview (unsaved rule overrides, base64 uploads up to
  5 MB, PHI suppression) and POST maps/{mapId}/validate.
- MapPreviewService shared by the Preview Map action and the endpoint;
  MapRuleValidator for paths, transforms and arguments.
- Committed fragment src/main/resources/openapi/qqq-v1/edi.yaml with a
  contract test; HTTP tests in strict raw-route mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 09:48:59 -05:00
james.maes fc800b8124 Merge pull request #14 from QRun-IO/feature/qqq-rc6
build: move to QQQ 4.1.0-RC.6 and remove RC.5 stand-ins
2026-10-14 09:26:57 -05:00
james.maesandClaude Opus 5.5 30ac598952 build: move to QQQ 4.1.0-RC.6 and remove RC.5 stand-ins
- qqq-bom 4.1.0-RC.6 (guardrail G7).
- Replace the EdiClassifications stand-in with core DataClassification
  on edi_ tables and fields and on the healthcare sample tables.
- Value-free QAlerts.notify notifications for exception assignment and
  partner exceptions.
- Fix the shared-transport possible-value filter grouping.
- PhiLogScope kept (RC.6 has no query-log redaction), now marked for
  core #1147 by RC.7; file-gateway stand-ins kept until that api ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 09:17:37 -05:00
james.maes 21b0e61cbb Merge pull request #13 from QRun-IO/feature/phase-g-edifact
feat: phase g - EDIFACT
2026-10-14 08:59:26 -05:00
james.maesandClaude Opus 5.5 5afeefb1c2 Merge develop into feature/phase-g-edifact
Resolve guide loader, validator, demo and docs conflicts with phases c and e;
HealthcareValidation now takes the standard-neutral EdiDocument.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 08:50:06 -05:00
james.maes b9ea954e1f Merge pull request #11 from QRun-IO/feature/phase-e-healthcare
feat: phase e - HIPAA healthcare documents, SNIP 3-5, PHI handling, canary tests
2026-10-14 08:46:36 -05:00
james.maesandClaude Opus 5.5 ea4c48ad66 feat: phase e - HIPAA healthcare documents, SNIP 3-5, PHI handling
- In-house 005010 baseline guides for 837P, 837I, 835, 270/271, 276/277
  and 834; 999 for HIPAA documents with IK3/IK4/CTX/IK5.
- SNIP 3 balancing (837 claim vs line charges, 835 payment vs claims and
  PLB, CAS adjustments), SNIP 4 situational rules, SNIP 5 format checks
  (ICD-10 shape, NPI check digit, ZIP) and an ExternalCodeSetProvider SPI.
- PHI classification via the RC.5 stand-in; healthcare keys go to
  phiBusinessKey; no element values in errors, events, alerts, logs or
  acks for PHI documents; audited Download Raw Document action.
- Healthcare sample maps, canary PHI egress tests, demo clearinghouse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 08:38:24 -05:00
james.maes 09b77ec122 Merge pull request #10 from QRun-IO/feature/phase-c-as2
feat: phase c - AS2 transport
2026-10-14 08:27:25 -05:00
james.maesandClaude Opus 5.5 4f8916246a feat: phase g - EDIFACT
- EDIFACT envelope reader and writer (UNA, UNB/UNZ, UNG/UNE, UNH/UNT,
  release-character escaping, syntax 3 and 4) with envelope findings.
- Routing by UNB identities, EDIFACT control references and duplicate
  detection; mixed X12 and EDIFACT inbox processing.
- In-house baseline guides (D96A, D01B) for ORDERS, ORDRSP, DESADV,
  INVOIC and CONTRL; CONTRL generation and inbound reconciliation.
- Sample maps (ORDERS in; ORDRSP, DESADV, INVOIC out) and demo partner
  Lindqvist Retail AB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 08:26:05 -05:00
james.maesandClaude Opus 5.5 a4dca9046b feat: phase c - AS2 transport
- AS2 send and receive with signing, encryption, compression, MIC, sync
  and async MDN (as2-lib algorithms and MIC types, BouncyCastle S/MIME
  and MDN handling in memory; no temp files or keystores).
- /qqq/hooks/edi/as2/{transport} receiver and /mdn async receipt route;
  retransmits get the original MDN; failure MDNs for decryption,
  authentication and integrity errors; async MDN delivery queue.
- edi_as2_mdn table; certificate parsing on save, keys from env vars,
  ACTIVE/NEXT rollover with Promote Certificate and Export Public
  Certificate; NEXT expiry and edi.mdn.overdue alerts; VAN over AS2.
- Demo partner Pacific Wholesale over AS2 against an in-process peer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 08:09:55 -05:00
james.maes 2122fafed5 Merge pull request #9 from QRun-IO/feature/phase-d-logistics
feat: phase d - logistics documents 940, 945, 214, 204
2026-10-14 07:47:12 -05:00
james.maesandClaude Opus 5.5 0a98f34c5e feat: phase d - logistics documents 940, 945, 214, 204
- In-house baseline guides for 940, 945, 214 and 204 (004010, 005010)
  with control-total, stop and status situational rules.
- Sample maps: 940 in, 945 out and in, 214 in, 204 in and out; DATEPART
  and TIMEPART outbound transforms and "+" joined inbound paths.
- Fix: InboundApplier no longer fails on tables without associations.
- Logistics tests (round trips, invalid documents, routing) and demo
  partners Harbor Logistics (940/945) and Ridgeline Freight (204/214).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:41:41 -05:00
james.maes 00b91243a6 Merge pull request #7 from QRun-IO/feature/phase-b-retail-outbound
feat: phase b - retail documents, outbound pipeline, transports, ack monitor, demo
2026-10-14 07:09:35 -05:00
james.maes 0d5d24392e Merge pull request #6 from QRun-IO/feature/phase-a2-inbound
feat: phase a2 - inbound vertical slice (inbox, routing, maps, apply, gateway intake, app home)
2026-10-14 07:09:21 -05:00
james.maes 88c6c8e278 Merge pull request #5 from QRun-IO/feature/phase-a1-x12-core
feat: phase a1 - EDI data model, X12 parsing, guide validation, TA1/997/999
2026-10-14 07:09:09 -05:00
james.maesandClaude Opus 5.5 fece01d7df fix: widget serialization, labels and a readable app home
- Keep runtime services (clock, resolvers, suppliers, stores) out of
  EdiQBitConfig JSON: every child-record-list widget returned 500 because
  the qbit config was serialized into widget responses.
- Title Case table labels, plural child-list labels, acronym-correct
  field labels, PVS on every edi_ foreign key, meaningful record labels
  (no PHI), operator-first field order, newest-first activity lists.
- Home: six focused statistics; exception queue, overdue acks and recent
  errors as narrow tables with partner names and humanized ages; volume
  by partner and named document with totals; one compact System table.
  No inner horizontal scroll at 390 px.
- Tests for config serialization, every widget over core rendering and
  HTTP, labels, ordering and home formatting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:08:09 -05:00
james.maesandClaude Opus 5.5 2915f47740 feat: phase b - retail documents, outbound pipeline, transports, ack monitor
- Baseline guides and sample maps for 855, 856, 810, 846, 860 (004010
  and 005010); outbound map engine with SEQUENCE and HL numbering.
- Outbound triggers (record-change transactional outbox, ESB enqueue
  process, opt-in Send As EDI), outbound worker (map, validate, batch,
  envelope, store) and send worker with retry, backoff and dead letter.
- Transport plugin SPI and registry; SFTP push and poll (streamed, host
  key pinned), file-gateway OUTBOX through the agreed facade stand-in,
  HTTPS out (basic, bearer, HMAC) and the /qqq/hooks/edi/https receiver.
- Overdue-ack and certificate-expiry monitor via QAlerts; Resend, Resend
  As New and Cancel actions; outbound queue statistics on the home.
- Test-scope demo server with a realistic synthetic seed (docs/DEMO.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 22:14:08 -05:00
james.maesandClaude Opus 5.5 49bc88ba3d feat: phase a2 - inbound vertical slice
- Durable edi_inbox with InboundSink intake, claimed scheduled worker,
  backoff, stale-claim reclaim and dead letter.
- Routing by ISA identities and usage, duplicate handling, per-ST
  transactions, validation errors, 997/999/TA1 queued as outbound
  messages with allocated control numbers, inbound ack reconciliation
  with QAlerts raise/resolve.
- Versioned data-driven maps (edi_map, rules, samples), EDI/QQQ paths,
  transforms, and apply through Insert/UpdateAction as a host-supplied
  service session with enforcement on.
- qbit-file-gateway consumer "edi" against a marked stand-in of the
  agreed api; optional object-storage payload store (api package only).
- EDI app with Activity/Setup nav and a typed-widget home; record actions
  (Reprocess, Preview Map, Assign/Ignore/Resolve, Create Partner
  Override, Activate Guide, Activate Map) with standard Review/Result.
- Contingent partner-scoped possible-value filters with tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:23:53 -05:00
james.maesandClaude Opus 5.5 b35d0440dd feat: phase a1 - EDI data model, X12 parsing, guide validation, acks
- 21 edi_ tables (partners, identities, agreements, agreement documents,
  control numbers, transports, certificates, guides, code lists,
  interchanges, groups, documents, errors, events, exceptions, inbox,
  payloads) with record screens, PVS, joins and child lists.
- StAEDI-based streaming X12 envelope reader (multi-interchange,
  separators, envelope findings with TA1/AK9/AK5 codes) and writer.
- Implementation guides as data: JSON baseline guides (850 004010/005010,
  997, 999), loader with revisions, structural validator and rule
  language for situational rules, SNIP 1-2.
- TA1, 997 and 999 builders at each ack level; inbound ack reader.
- Atomic control-number allocation and duplicate detection.
- AES-256-GCM envelope-encrypted database payload store with rotation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:31:14 -05:00
james.maes bddba1eb8c Merge pull request #4 from QRun-IO/feature/design-spec
docs: qbit-edi design spec
2026-10-13 19:43:49 -05:00
james.maesandClaude Opus 5.5 6ddc8d5e50 docs: qbit-edi design spec
Trading partners and agreements, pluggable transports (AS2, SFTP via
qbit-file-gateway, SFTP push and VAN polling, HTTPS, email), StAEDI
parsing, implementation guides and maps as versioned data, inbound and
outbound flows with TA1/997/999, HIPAA PHI handling and SNIP levels,
database-and-S3-only payload storage, the app home, and phases a-g.

Owner decisions (2026-10-12): all recommendations accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:41:43 -05:00
james.maes 94c459b150 Merge pull request #1 from QRun-IO/feature/repo-standard-scaffold
chore: bring the scaffold up to the QRun qbit repository standard
2026-10-13 19:34:42 -05:00
james.maesandClaude Opus 5.5 f45e126a62 chore: bring the scaffold up to the QRun qbit repository standard
Replace the application template's example sources with the EDI producer
and config, and add the standard files: qbit-build-parent 2.0.0 POM on
QQQ 4.1.0-RC.5 with Checkstyle and the JaCoCo gate, Maven wrapper,
CircleCI qqq-orb 0.6.8 workflows, the release-candidate workflow,
Dependabot, community pointer files, CHANGELOG, CLAUDE.md/AGENTS.md and a
DESIGN.md placeholder for the spec PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:31:04 -05:00
310 changed files with 80567 additions and 1131 deletions

No files matched your search

+57 -11
View File
@@ -1,14 +1,60 @@
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.8
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
steps:
- run:
name: Keep package installation unattended
command: |
# These build machines are discarded after the job.
sudo install -d /etc/needrestart/conf.d
printf '%s\n' '$nrconf{restart} = "l";' | sudo tee /etc/needrestart/conf.d/99-qqq-ci.conf >/dev/null
workflows:
setup:
jobs:
- qqq-orb/generate_pipeline:
orb_version: '0.8'
filters:
branches:
ignore: automation/qbit-rc-pending
tags:
ignore: /.*/
test_only:
jobs:
- qqq-orb/mvn_test_only:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
filters:
branches:
ignore: /(develop|release\/.*)/
publish_snapshot:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: snapshot
filters:
branches:
only: [develop]
publish_release_candidate:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: release_candidate
filters:
branches:
only: [/release\/.*/]
publish_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: release
filters:
branches:
ignore: /.*/
tags:
only: [/^v\d+\.\d+\.\d+$/]
+20
View File
@@ -0,0 +1,20 @@
# QRun-IO qbit .editorconfig: the Kingsrook layout (3-space indentation, never tabs).
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 3
tab_width = 3
[*.md]
trim_trailing_whitespace = false
[*.{yml,yaml}]
indent_size = 2
[*.{cmd,bat}]
end_of_line = crlf
+3
View File
@@ -0,0 +1,3 @@
* text=auto eol=lf
*.cmd text eol=crlf
mvnw text eol=lf
-24
View File
@@ -1,24 +0,0 @@
# Standard QRun release caller (v2, thin contract): every push and manual dispatch delegates to the protected qrun-release-v1 channel. Do not edit per repository.
name: qrun-release
on:
push: {}
workflow_dispatch: {}
permissions:
contents: read
actions: read
jobs:
release:
uses: self:KofTwentyTwo/platform-ci/.gitea/workflows/java-maven.yaml@qrun-release-v1
with:
runner: linux-small
browsers: true
browser-runner: browser-build
postpublish:
if: ${{ needs.release.outputs.capture == 'true' }}
needs: release
permissions:
contents: write
actions: write
uses: self:KofTwentyTwo/platform-ci/.gitea/workflows/qrun-source-version.yaml@qrun-release-v1
with:
runner: linux-small
+25
View File
@@ -0,0 +1,25 @@
# Weekly, grouped dependency updates. Minor and patch updates arrive as one PR per ecosystem;
# major updates stay separate so each gets its own review. At most three version-update PRs per
# ecosystem are open at once. Security updates are enabled in the repository settings and are
# not limited by this schedule.
version: 2
updates:
- package-ecosystem: maven
directory: /
target-branch: develop
schedule:
interval: weekly
open-pull-requests-limit: 3
groups:
maven-minor-and-patch:
update-types: [minor, patch]
- package-ecosystem: github-actions
directory: /
target-branch: develop
schedule:
interval: weekly
open-pull-requests-limit: 3
groups:
github-actions:
patterns: ['*']
-23
View File
@@ -1,23 +0,0 @@
name: CodeQL
'on':
push:
branches:
- main
- develop
- release/**
- hotfix/**
pull_request: {}
schedule:
- cron: 41 16 * * 0
workflow_dispatch: {}
permissions:
contents: read
actions: read
security-events: write
packages: read
concurrency:
group: codeql-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
analyze:
uses: QRun-IO/qqq-orb/.github/workflows/qbit-codeql.yml@c288a8864fe9054f68410e1d78f5863aea4c4d22
+25
View File
@@ -0,0 +1,25 @@
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
# the release line and the release/* push runs CircleCI publish_release_candidate.
name: Release candidate
on:
workflow_dispatch:
inputs:
dry_run:
description: Print the plan without merging or creating branches
type: boolean
default: false
pull_request:
types: [closed]
branches: [develop]
permissions:
contents: write
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
+15 -25
View File
@@ -1,31 +1,21 @@
# Maven
target/
pom.xml.tag
pom.xml.releaseBackup
pom.xml.versionsBackup
pom.xml.next
release.properties
# IDE
.idea/
.env
.idea
*.iml
.project
.classpath
.settings/
.vscode/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Build
*.class
*.log
*.jar
*.war
*.nar
*.ear
# Logs
*.log
logs/
*.zip
*.tar.gz
*.rar
hs_err_pid*
.DS_Store
*.sw*
.flattened-pom.xml
dependency-reduced-pom.xml
/.env.local
/.cache/
.claude/settings.local.json
+3
View File
@@ -0,0 +1,3 @@
wrapperVersion=3.3.4
distributionType=only-script
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.11/apache-maven-3.9.11-bin.zip
-15
View File
@@ -1,15 +0,0 @@
profile: qbit
orb_version: '0.8'
default_branch: develop
java_version: '21'
browsers: true
api_checks: false
docs_only: false
release_on_main: false
release_tag_policy: none
test_branch_policy: all
publish:
snapshot: false
release_candidate: false
release: false
asciidoc: false
+62
View File
@@ -0,0 +1,62 @@
# AGENTS.md
This file provides guidance to Codex when working with code in this repository.
## Project Overview
**QBit EDI**: a full EDI application qbit for QQQ. Trading partners and agreements, pluggable
transports (AS2, SFTP/FTPS, VAN, HTTPS, email), X12 (004010, 005010; EDIFACT later) parsing and
validation against data-driven implementation guides, 997/999/TA1, versioned maps to QQQ records,
inbound and outbound flows, and HIPAA documents with PHI classification. Design: `docs/DESIGN.md`.
## Build Commands
```bash
./mvnw compile
./mvnw test
./mvnw verify # Checkstyle, tests, JaCoCo: 70% instructions, 90% classes
```
Always use the wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
## Architecture
Entry point: `EdiQBitProducer implements QBitMetaDataProducer<EdiQBitConfig>` in
`com.kingsrook.qbits.edi`. Component producers are scanned only in `com.kingsrook.qbits.edi.metadata`.
The package layout per phase is in `docs/DESIGN.md`.
## Invariants
- Data lives in standard visible QQQ tables with record screens, secured by permissions and record locks.
- Processes are table or record actions with plain labels, never in the nav; record actions use the
standard streamed-ETL Review/Result summary.
- One app home that owns the EDI data, built from typed widgets (no `RawHTML`, guardrail G5).
- Inbound transport endpoints are non-UI routes under `/qqq/hooks/edi/...`, never under `/qqq/v1`.
UI endpoints are spec'd `MiddlewareSpecExtension`s under `/qqq/v1/qbit/edi/...` with a committed fragment.
- Records are created and updated through normal QQQ actions as a service account.
- Raw documents and PHI fields are classified (`DataClassification`), stored encrypted.
- Secrets are referenced by environment-variable name only. No real PHI in tests or samples.
- No dependency on another qbit's internals (guardrail G1); notifications go through core `QAlerts`.
## Naming
- Tables: `edi_<name>`, lowercase snake_case; SQL names equal QQQ table names.
- Permissions: full name, `edi.*`.
## Code Style
- Copy the Apache header from `checkstyle/license.txt`; 3-space indent, Allman braces, `return (x);`.
- Flower-box Javadoc on every class and method; fluent `withX()` setters.
- Wrapper types in config and entities; JUnit 5 and AssertJ tests.
## Dependencies
Java 21; parent `com.kingsrook:qbit-build-parent:2.0.0`. Default-active `qqq-snapshot` imports QQQ
`4.1.0-RC.6` ahead of the parent's 4.0.0 BOM (ADR-0007); move to each new RC within a week (G7).
Middleware Javalin is optional. Also StAEDI (X12) and as2-lib with BouncyCastle (AS2), all Apache-2.0/MIT-style licensed.
## Release
CircleCI qqq-orb 0.6.8: `develop` publishes SNAPSHOTs, `release/*` publishes RCs (orb version
bumps; cut by the `release-rc.yml` workflow), `vX.Y.Z` tags publish releases. No manual publishing.
Branches are `feature/*` or `fix/*` only; never push `hotfix/*` or `release/*`.
+82
View File
@@ -0,0 +1,82 @@
# Changelog
Following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Semantic Versioning](https://semver.org/).
## [Unreleased]
### Changed
- Require QQQ 4.1.0-RC.7 for core help-bundle registration and coverage validation.
- X12 routing and sending honor identity `isActive`, matching EDIFACT.
### Added
- Complete, host-neutral `help/qbit-edi.yaml` bundle registered with QQQ core: app, tables, fields, actions, widgets and possible-value sources, with a coverage gate and HOST overrides. Table/app fallback remains available without `helpContent`.
- Setup status section on the EDI home (partner, identities, agreement, agreement documents, transport, certificates, guides, maps) with the next step; it counts only the active rows the runtime uses.
- "Install sample maps" action on Maps: standard streamed ETL, insert-only by name, installs the 32 bundled synthetic maps as draft, unbound maps bound to the built-in guide.
- `EdiTablesGuardrailTest`: every table is used at runtime, seeded or listed as starting empty, and has help.
- Phase f1 `edi` 1.0.0 spec'd UI API: guide tree, map target/transform metadata, encrypted base64 sample preview with unsaved rule overrides, and per-rule validation under `/qqq/v1/qbit/edi`.
- Versioned `Edi...V1` schemas and generated OpenAPI fragment with the RC.6 contract/compatibility checks; real HTTP strict-mode permission, PHI, upload, isolation and validation tests.
- Shared sandbox/record-action preview service, current/original guide selection, caller-session transforms, structured revision/expected diffs and value-free PHI diagnostics.
- Transform registry argument schemas, including host-supplied schemas. Preview Map record actions now enforce the documented exact `edi.map.preview` permission through core's full-name permission checker.
- Phase g EDIFACT syntax 3/4 envelopes (UNA/UNB, optional UNG/UNE, UNH/UNT, UNZ), release escaping, native envelope findings, UNB identity routing, 14-digit UN control allocation and duplicate detection.
- Original D96A/D01B ORDERS, ORDRSP, DESADV, INVOIC and CONTRL guides; ORDERS inbound and retail response outbound sample maps with composite fields and segment groups.
- Standard-neutral document validation/mapping, mixed X12/EDIFACT inbox framing, CONTRL generation and reconciliation with overdue resolution and rejection alerts, native outbound envelopes and archived resend-as-new.
- Lindqvist Retail AB demo partner with ORDERS application, ORDRSP/DESADV/INVOIC delivery and CONTRL in both directions; EDIFACT map preview and HTTPS media type.
- Phase e original HIPAA baselines for 837P/I, 835, 270/271, 276/277 and 834; exact SNIP 3 balancing, occurrence-scoped SNIP 4 rules, SNIP 5 formats and a host `ExternalCodeSetProvider` SPI.
- Test-only healthcare maps, classified sample tables, seeded synthetic fixtures, Cedar Health Clearinghouse demo traffic and a PHI canary scanner covering both ingress paths, 999s, apply/replay/resend, widgets, query exports, alerts and logs.
- Permission-checked, session-bound Download Raw Document actions for inbox/interchange rows, with a value-free user audit event.
- Phase d warehouse and logistics baseline guides for 940, 945, 214 and 204 in 004010 and 005010 (syntax notes, situational control-total and stop rules, AT7 pairing), inbound/outbound sample maps (940 in, 945 out/in, 214 in, 204 in/out), DATEPART/TIMEPART outbound transforms and `+`-joined inbound paths for date/time pairs, and Harbor Logistics and Ridgeline Freight demo traffic.
- Phase c AS2: `As2Transport` send and receive (RFC 4130) with S/MIME signing (SHA-256/384/512; SHA1 flagged), encryption (AES-128/256 CBC or GCM; 3DES flagged), optional compression, synchronous and asynchronous signed MDNs with MIC verification, Test Connection, optional TLS pinning and VAN transports with no partner. Built on as2-lib 5.1.8 and BouncyCastle, entirely in memory (no keystore or temporary files).
- `POST /qqq/hooks/edi/as2/{transport}` and `/{transport}/mdn` route provider: identity, decryption (ACTIVE or NEXT keys), signature and MIC checks, Message-ID dedupe returning the original MDN, error MDNs with no inbox row, and a queued, retried asynchronous MDN sender.
- Read-only `edi_as2_mdn` table with "AS2 MDNs" child lists on outbound and inbound messages; raw MDNs in the encrypted payload store; `edi_outbound_message.transport_mic`; `edi_transport.weak_crypto`; `EdiQBitConfig.publicBaseUrl`.
- Certificates: parsed on save (private keys refused), keys from environment PEM PKCS#8 or base64 PKCS#12, ACTIVE/NEXT rollover, record actions Promote Certificate and Export Public Certificate, expiry alerts for ACTIVE and NEXT, and the `edi.mdn.overdue` alert.
- Demo: Pacific Wholesale over AS2 against an in-process BouncyCastle peer with in-memory certificates; AS2 acceptance tests in both directions against our own route and an independent peer.
- Phase b retail baseline guides for 855/856/810/846/860 in 004010 and 005010; inbound/outbound sample maps and nested outbound associations with SEQUENCE/HL transforms.
- Transactional RECORD_CHANGE and manual enqueue, host-wired ESB process, batch schedules, validated encrypted envelopes, send workers, retries and dead-letter alerts.
- Instance transport registry, MINA SFTP streaming push/poll with pinned host keys, gateway OUTBOX stand-in/download delivery, and authenticated HTTPS ingress/egress using core Standard Webhooks.
- Ack and certificate monitoring, functional-ack exception resolution, resend/re-envelope/cancel actions, transport tests and outbound queue home statistics.
- Test-scope H2/Next UI demo with synthetic retail traffic, in-memory gateway and public self-signed certificates; local HTTP/SFTP and application acceptance tests.
- Phase a2 inbound vertical slice: gateway intake, fenced inbox claims/retries, dead letters and core alerts; routing, validation, duplicates, safe events and exception occurrence counts.
- Versioned inbound maps, wildcard associations, transforms, permission-enforced transactional upsert and original/current revision replay.
- Queued enveloped TA1/997/999 and inbound acknowledgement reconciliation; outbound request/message tables for phase b.
- Public-API-only encrypted object-storage adapter and marked file-gateway contract stand-in.
- EDI app, typed home with permission-scoped counts, main-table search metadata, optional ESB events with identifier-only document payloads, and standard Review/Result actions.
- Synthetic sample upload, preview comparisons and encrypted JSON downloads; guide overrides and guide/map activation.
- Contingent possible values on existing/new tables, including actual filtered-search tests.
- PostgreSQL schema additions and UTC zone-less timestamps matching RC.5 RDBMS bindings.
- Phase a1 data model: 21 annotated entities, QQQ tables, enum/table value sources, joins, child lists, audit, permissions, partner locks, and PostgreSQL DDL with metadata-derived H2 integration tests.
- StAEDI 1.26.4 X12 streaming envelopes, separator detection, bounded transaction buffers, structured envelope findings, and fixed-width ISA serialization.
- Original 850/997/999 baseline guides, versioned database loading, SNIP 1–2 validation, syntax notes, and the documented situational rule language.
- TA1, 997 and 999 builders with acknowledgement levels, positional situational context, reply envelopes, and inbound acknowledgement parsing.
- Atomic RDBMS control-number allocation with first-use creation, wrap/exhaustion handling, and interchange/group/transaction duplicate detection.
- Database payload storage with AES-256-GCM envelope encryption, SHA-256 integrity metadata, key rotation, and replaceable environment-secret resolution.
- Per-instance runtime services and configuration, synthetic generators, exact acknowledgement fixtures, concurrent SQL tests, and payload tests that verify no temporary files are created.
- Repository scaffold to the QRun qbit repository standard: `qbit-build-parent` 2.0.0, QQQ 4.1.0-RC.5, Maven wrapper, Checkstyle, JaCoCo gate, CircleCI qqq-orb 0.6.8, release-candidate workflow, Dependabot, community files.
- `EdiQBitProducer` and `EdiQBitConfig` (table prefix `edi_`, permission base `edi`).
### Changed
- QQQ baseline is now 4.1.0-RC.6. EDI tables/fields and healthcare test fixtures use core `DataClassification` through `AiTableMetaData`/`AiFieldMetaData`, preserving PHI, PII, CONFIDENTIAL and PHI AI-exposure restrictions.
- Exception Assign and new/reopened partner rejection/dead-letter exceptions send value-free core notifications after commit, with capturing-sink and PHI canary coverage.
- Retain `PhiLogScope` for core #1147 until RC.7: RC.6 still logs RDBMS query filters and has no redaction hook or `FieldAccessHelper`. Contingent dependency-behavior APIs are also absent; shared-choice filters remain tested.
### Removed
- Reflective `EdiClassifications` RC.5 stand-in. File-gateway stand-ins remain until its API is published; the object-storage API integration is unchanged.
### Fixed
- Shared-choice filters group partner-or-shared alternatives beneath an AND root, so core search criteria cannot admit another partner's transports.
- 999 now accepts SNIP 3+ findings with `E` instead of rejecting every ERROR. HIPAA acknowledgement selection follows the implementation version and enabled agreement expectation; outbound HIPAA ST carries its implementation identifier.
- Qualified HL/NM1 anchors distinguish healthcare hierarchy levels; repeated loop names under different parents retain separate occurrences. Situational rules inherit direct ancestor segments without borrowing values from sibling occurrences.
- Review/Result records expose identifiers only. A marked RC.5 PHI logging bridge prevents core upsert query diagnostics from copying claim/member values into logs.
- Inbound apply no longer fails with a null-pointer error on a target table that has no associations (flat tables such as the 214 status sample).
+54 -17
View File
@@ -1,25 +1,62 @@
# qbit-template-application
# CLAUDE.md
GitHub template repository for scaffolding Application QBits (complete mini-applications
with tables, processes, widgets, and UI navigation) on the QQQ platform.
This file provides guidance to Claude Code when working with code in this repository.
## QQQ 4.0 baseline — 2026-09-24
## Project Overview
The historical review below predates the 4.0 migration. This template targets Java 21 and the public QQQ 4.0.0 release from Maven Central. The corrected APIs, license references and actual template commands are in README.md and docs/00-getting-started.md. Preserve the documented multi-instance and scaffold limitations.
**QBit EDI**: a full EDI application qbit for QQQ. Trading partners and agreements, pluggable
transports (AS2, SFTP/FTPS, VAN, HTTPS, email), X12 (004010, 005010; EDIFACT later) parsing and
validation against data-driven implementation guides, 997/999/TA1, versioned maps to QQQ records,
inbound and outbound flows, and HIPAA documents with PHI classification. Design: `docs/DESIGN.md`.
## Knowledge base
## Build Commands
Reviewed dossier and platform knowledge live in the second-brain vault:
```bash
./mvnw compile
./mvnw test
./mvnw verify # Checkstyle, tests, JaCoCo: 70% instructions, 90% classes
```
- Hub: `R:/Git.Local/KofTwentyTwo/second-brain/knowledge/qqq/qqq-hub.md`
- This repo's dossier: `R:/Git.Local/KofTwentyTwo/second-brain/knowledge/qqq/repos/qbit-template-application.md`
- QBit mechanics refresher: `R:/Git.Local/KofTwentyTwo/second-brain/knowledge/qqq/architecture/metadata-model.md`
Always use the wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
Reviewed at commit `fb9395ac0d07` (branch `main`, 2026-07-04).
## Architecture
Key caveat from that review: the scaffold in `src/main` references six qqq APIs that do not
exist in any qqq version (wrong packages for `MetaDataProducerInterface` /
`AbstractWidgetRenderer` / `RenderWidgetInput`, plus nonexistent `QInstance.addAppSection`,
`QProcessMetaData.withInputFields`, `QAppSection.withSourceQBitName`) — it has never compiled,
and CI on `main` is red. Verify against the qqq checkout (`R:/Git.Local/QRunIO/qqq`) before
trusting any API usage shown here.
Entry point: `EdiQBitProducer implements QBitMetaDataProducer<EdiQBitConfig>` in
`com.kingsrook.qbits.edi`. Component producers are scanned only in `com.kingsrook.qbits.edi.metadata`.
The package layout per phase is in `docs/DESIGN.md`.
## Invariants
- Data lives in standard visible QQQ tables with record screens, secured by permissions and record locks.
- Processes are table or record actions with plain labels, never in the nav; record actions use the
standard streamed-ETL Review/Result summary.
- One app home that owns the EDI data, built from typed widgets (no `RawHTML`, guardrail G5).
- Inbound transport endpoints are non-UI routes under `/qqq/hooks/edi/...`, never under `/qqq/v1`.
UI endpoints are spec'd `MiddlewareSpecExtension`s under `/qqq/v1/qbit/edi/...` with a committed fragment.
- Records are created and updated through normal QQQ actions as a service account.
- Raw documents and PHI fields are classified (`DataClassification`), stored encrypted.
- Secrets are referenced by environment-variable name only. No real PHI in tests or samples.
- No dependency on another qbit's internals (guardrail G1); notifications go through core `QAlerts`.
## Naming
- Tables: `edi_<name>`, lowercase snake_case; SQL names equal QQQ table names.
- Permissions: full name, `edi.*`.
## Code Style
- Copy the Apache header from `checkstyle/license.txt`; 3-space indent, Allman braces, `return (x);`.
- Flower-box Javadoc on every class and method; fluent `withX()` setters.
- Wrapper types in config and entities; JUnit 5 and AssertJ tests.
## Dependencies
Java 21; parent `com.kingsrook:qbit-build-parent:2.0.0`. Default-active `qqq-snapshot` imports QQQ
`4.1.0-RC.6` ahead of the parent's 4.0.0 BOM (ADR-0007); move to each new RC within a week (G7).
Middleware Javalin is optional. Also StAEDI (X12) and as2-lib with BouncyCastle (AS2), all Apache-2.0/MIT-style licensed.
## Release
CircleCI qqq-orb 0.6.8: `develop` publishes SNAPSHOTs, `release/*` publishes RCs (orb version
bumps; cut by the `release-rc.yml` workflow), `vX.Y.Z` tags publish releases. No manual publishing.
Branches are `feature/*` or `fix/*` only; never push `hotfix/*` or `release/*`.
+3
View File
@@ -0,0 +1,3 @@
# Code of Conduct
See Project level [Code of Conduct](https://github.com/QRun-IO/qqq/blob/develop/CODE_OF_CONDUCT.md)
+3
View File
@@ -0,0 +1,3 @@
# Code Style Guide
* See Project [Code Style Guide](https://github.com/QRun-IO/qqq/blob/develop/CODE_STYLE.md)
+5
View File
@@ -0,0 +1,5 @@
# Contributing
Follow [QQQ contributing guidance](https://github.com/QRun-IO/qqq/blob/develop/CONTRIBUTING.md).
Use Java 21 and `./mvnw verify`, and follow local Checkstyle. Create feature branches from `develop` and target `develop` with PRs. See [AGENTS.md](AGENTS.md) for architecture, naming, security and release conventions, and [the design](https://github.com/QRun-IO/qqq/pull/1137) before extending the qbit.
+1 -1
View File
@@ -176,7 +176,7 @@
END OF TERMS AND CONDITIONS
Copyright 2022-2025 QRunIO, LLC
Copyright 2024 Kingsrook, LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+3 -3
View File
@@ -1,6 +1,6 @@
qBit Template - Application
Copyright 2022-2025 QRunIO, LLC
QBit EDI
Copyright 2026 Kingsrook, LLC
This product includes software developed at QRunIO, LLC (https://qrun.io/).
This product includes software developed at Kingsrook, LLC (https://kingsrook.com/).
Licensed under the Apache License, Version 2.0.
+422 -41
View File
@@ -1,55 +1,436 @@
# QBit Template - Application
# QBit EDI
Template repository for creating **Application QBits** - complete, self-contained applications or major functional modules for the QQQ framework.
EDI for QQQ: trading partners, agreements, X12 and EDIFACT validation, encrypted payloads, inbound mapping
and application, and TA1/997/999/CONTRL acknowledgements. See [the design](docs/DESIGN.md).
## When to Use This Template
## Status
Use this template when building:
- Complete mini-applications (WMS, OMS, CRM)
- Major functional modules with multiple tables
- QBits that require UI navigation (QAppSection)
- Self-contained features with business logic
Phases a1 to e and g implement inbound/outbound retail, logistics and healthcare EDI (X12 and EDIFACT) over gateway, SFTP, HTTPS and AS2. Original baseline guides cover
850, 855, 856, 810, 846 and 860 in 004010/005010, plus 997/999. Source records and associations
produce validated, encrypted outbound interchanges, with control-number tracking, retries,
acknowledgement monitoring and operator resend/cancel actions. AS2 (signed, encrypted, compressed, synchronous and
asynchronous MDN, certificate rollover) is built in; email sending remains disabled until the qbit-email API ships.
A runnable synthetic [demo](docs/DEMO.md) includes Next UI and needs no external services.
## Quick Start
EDIFACT syntax versions 3 and 4 support UNA/UNB, optional UNG groups, UNH messages and their
trailers, release-character escaping, mixed-standard payloads, and CONTRL syntax responses.
Original D96A/D01B baselines cover ORDERS, ORDRSP, DESADV, INVOIC and CONTRL. ORDERS maps to
sample orders and lines; the other three retail documents map outbound. APERAK remains future work.
Requires **Java 21**, **Maven 3.8+**, and QQQ **4.0.0**. Create a repository with **Use this template**, customize its Maven coordinates and Java package/classes, then run `mvn clean verify`. Follow the [Getting Started guide](docs/00-getting-started.md) for the full sequence.
Configure `standard=EDIFACT` on identities and agreements. `interchangeId` and
`interchangeQualifier` carry the UNB party identifiers and qualifiers. Set `interchangeVersion`
to `3` or `4`, `syntaxIdentifier` to `UNOC`, and outbound separators explicitly: element `+`,
component `:`, segment `'`, release `?`, decimal `.` (or `,`); syntax 4 also uses the configured
repetition separator. Document versions are `D96A` or `D01B`, with `ackExpectation=CONTRL`.
UNB11=`1` uses a test agreement and respects `applyTestDocuments`.
## Structure
Existing schemas need these additional nullable agreement columns before using this version:
```
src/main/java/com/kingsrook/qbits/example/
├── ExampleAppQBitConfig.java
├── ExampleAppQBitProducer.java
├── model/
│ ├── ExampleEntity.java
│ └── ExampleChildEntity.java
├── processes/
│ ├── ExampleProcessStep.java
│ └── ExampleProcessMetaDataProducer.java
└── widgets/
├── ExampleDashboardWidgetRenderer.java
└── ExampleDashboardWidgetMetaDataProducer.java
```sql
ALTER TABLE edi_agreement ADD COLUMN release_character VARCHAR(255);
ALTER TABLE edi_agreement ADD COLUMN decimal_mark VARCHAR(255);
ALTER TABLE edi_agreement ADD COLUMN syntax_identifier VARCHAR(255);
```
## Key Characteristics
## Requirements and installation
| Feature | Description |
|---------|-------------|
| QAppSection | Required - provides UI navigation |
| Tables | Multiple interconnected business tables |
| Processes | Full CRUD with business logic |
| Widgets | Dashboard components |
| Security | Permission key support |
| Integration | Hooks for other QBits |
Java 21, QQQ **4.1.0-RC.7+**, an RDBMS backend, and `com.kingsrook.qbits:qbit-edi:0.1.0-SNAPSHOT`.
Apply [the PostgreSQL schema](docs/ddl/edi-postgres.sql). Targets and their associations must
share the EDI backend so each transaction set can commit or roll back atomically.
## Documentation
```java
new EdiQBitProducer()
.withQBitConfig(new EdiQBitConfig()
.withBackendName("rdbms")
.withSchedulerName("scheduler") // optional
.withSendAsEdiTable("sample_order") // host-defined table, optional manual action
.withServiceSessionSupplier(() -> makeEdiServiceSession()))
.produce(qInstance);
```
- [Getting Started](docs/00-getting-started.md)
- [Tables and Relationships](docs/01-tables-and-relationships.md)
- [Processes](docs/02-processes.md)
- [Widgets and UI](docs/03-widgets-and-ui.md)
Call `EdiRuntime.current().initialize()` in the host's authorized startup context to install
baseline guides and data-driven schedules. After defining batch/poll settings, call
`OutboundProcesses.installSchedules()` before starting the host scheduler; rebuild/restart scheduling
after changing those settings. Define partners, identities, active agreements, inbound agreement documents
and active maps as normal QQQ records. `makeEdiServiceSession()` is a host function returning
a least-privilege `QSession` with target and association permissions. Applying without a supplier
produces an APPLY exception: `no service session configured`.
## See Also
The `edi` app has Activity/Setup sections, typed home widgets, record screens and standard
Review/Result actions. Main tables have RC.6 search-field metadata; `EdiIntegrations.mainTables()`
also exposes the list for host search registration.
- [qbit-template-extension](https://github.com/QRun-IO/qbit-template-extension) - For infrastructure extensions
- [qbit-template-data](https://github.com/QRun-IO/qbit-template-data) - For reference data providers
## In-app help
The producer registers [help/qbit-edi.yaml](src/main/resources/help/qbit-edi.yaml), a `QBIT`
bundle versioned with this artifact. It covers the EDI app, tables, fields, actions, widgets,
and possible-value sources. Live `{{...}}` references follow the current metadata.
Hosts can register their own `layer: HOST` bundle with `HelpBundles.register(qInstance,
"help/my-host.yaml")`, using the same keys and roles to override selected entries. Enable
core's `helpContent` provider with provenance fields, then call
`QInstanceHelpContentManager.loadHelpContent(qInstance)` and `HelpBundles.syncOnStartup(qInstance)`
after defining metadata. Core preserves user edits during sync. If you configure a different
EDI app name, use that name in your host's app help key.
Registration is safe without `helpContent`; those hosts retain the table/app help fallback.
`EdiHelpBundleTest` runs the core `HelpBundleAssertions` coverage gate and checks widget and
possible-value-source coverage during `./mvnw verify`, without host-specific exclusions.
## Tables
There are **26 persisted tables**, whose SQL names equal their QQQ names.
| Area | Tables |
| --- | --- |
| Setup | `edi_partner`, `edi_identity`, `edi_agreement`, `edi_agreement_document`, `edi_control_number`, `edi_transport`, `edi_certificate` |
| Guides | `edi_guide`, `edi_guide_segment`, `edi_guide_element`, `edi_guide_rule`, `edi_code_list`, `edi_code_list_value` |
| Maps | `edi_map`, `edi_map_rule`, `edi_map_sample` |
| Processing | `edi_inbox`, `edi_interchange`, `edi_functional_group`, `edi_transaction`, `edi_transaction_error`, `edi_exception`, `edi_event` |
| Storage/queue | `edi_payload`, `edi_outbound_request`, `edi_outbound_message` |
`edi_preview_download` is a **storage-only metadata alias**, not a SQL table. Supporting tables
appear as child lists. Partner-scoped form choices use RC.6 contingent filters, including shared
transports/guides and local certificates/identities.
## UI API
With optional Javalin present, the producer registers the `edi` extension, version `1.0.0`,
under `/qqq/v1/qbit/edi`. All routes are endpoint specs and pass strict UI-route auditing.
| Method | Path after `/qqq/v1/qbit/edi` | Result |
|---|---|---|
| GET | `/guides/{guideId}/tree` | Nested loops, ordered segments, elements and up to 200 active codes per element |
| GET | `/maps/{mapId}/target` | Target fields, direct associations, current rules and registered transform argument schemas |
| POST | `/sandbox/preview` | Parse counts, positional findings, acknowledgement, mapped records and revision/expected diffs |
| POST | `/maps/{mapId}/validate` | Per-rule path, transform and required-argument errors |
Preview accepts `{mapId, sampleId?, sampleBase64?, sampleName?, ruleOverrides?, guideRevision?}`.
Supply exactly one sample source. Uploads are base64 JSON, at most **5 MiB decoded**, archived as
encrypted `edi_map_sample` records through the same storage as Preview Map. No runtime files are
created. Configure the host's Javalin `http.maxRequestSize` to at least 8 MiB to allow base64
overhead and request metadata. Oversized decoded samples return 400; a smaller host limit returns 413.
`ruleOverrides` replaces the entire rule list in memory, including an empty list; it never saves
rules. `CURRENT` (default) selects the newest active revision of the pinned guide's name, standard,
version, transaction set and partner; `ORIGINAL` uses the map's pinned guide. Validate accepts
`{rules: [...]}` and checks the pinned guide. Rule fields and `transformArgs` have the same shape
as core table CRUD (`transformArgs` is a JSON object encoded as a string).
Every read enforces table permissions and partner locks with the caller's session, regardless of
the host's default core-enforcement setting. Preview additionally requires **`edi.map.preview`**,
shared with both Preview Map record actions. Reading target metadata requires target/association
table read permissions; guide definitions, code lists, map rules and stored samples require their
own table reads. LOOKUP and host transforms run as the caller, without service-account elevation.
Only the internal PHI classification check and encrypted sample storage use the EDI system writer.
For PHI, `result` and `ack.text` are null and both diff lists are empty; findings retain positions
and codes with generic messages. Mapping failures are value-free `MAP` findings. Preview never
writes target records. Diffs contain differing document records as `{documentIndex, before, after}`;
acknowledgements for multiple groups are concatenated, with `type: MIXED` for different ack types.
The mapper saves `edi_map_rule` through the **core table API**, using its normal
`edi.map_rule.insert|edit|delete` permissions and record locks. There are no custom rule-write routes.
The Next mapper page is separate phase-f work.
The generated contract is `src/main/resources/openapi/qqq-v1/edi.yaml`; all component names are
`Edi...V1`. Regenerate intentionally with:
```powershell
./mvnw.cmd -B '-Dtest=EdiApiContractTest' '-Dedi.openapi.update=true' test
```
RC.6 provides `UiApiContract.check`, used to compare generated and checked-in YAML and run
`OpenApiCompatibilityChecker` against this initial fragment baseline. The extension also appears
in the host's `/qqq/v1/openapi.yaml` and `/qqq/v1/version`.
## Inbound operation
`InboundSink.receive(InboundPayload)` stores encrypted bytes and SHA-256 before returning a durable
inbox record. Transport/external ID retries are idempotent; matching bytes within the 365-day intake
window create DUPLICATE rows. `EdiFileGatewayConsumer` uses the marked internal stand-in contract,
with upload ID plus digest as external ID, and returns `edi_inbox:<id>` without parsing.
Storage failure requests retry in one minute.
`InboxWorker.runOnce()` processes up to 20 messages with compare-and-set claims, fenced database
checkpoints and five-minute stale-claim recovery. Retry delays are 1m, 5m, 15m, 1h and 6h; the next
failure becomes DEAD_LETTER. Each mapped accepted ST or UNH applies in its own transaction through QQQ
actions with explicit permission enforcement. ISA15=T or UNB11=1 applies only with `applyTestDocuments`.
Routing checks ISA or UNB identities, qualifiers, standard and usage. Linked gateway usernames must match
the routed partner. Findings and failures become exceptions with occurrence counts and value-free
timeline events. Replies allocate controls and remain QUEUED, recording `ACK_QUEUED`. Incoming
acknowledgements reconcile outbound groups and resolve overdue alerts.
Reprocess restarts the containing inbox from encrypted storage, using original or current guide/map
revisions. Source-position checkpoints and target upsert keys preserve idempotency, stale findings
are replaced, and the same reply is not queued twice. A live PROCESSING claim cannot be reset.
## Maps and preview
EDI paths reuse the guide rule language. Targets support `field` and `association.field`;
wildcards such as `PO1[*]/PO107` produce child records. `matchKeyFields` controls upsert, with
`onMatch` UPDATE/SKIP/EXCEPTION. [MAPS.md](docs/MAPS.md) documents transform arguments.
Resources under `/edi/maps` include inbound 850/860 and outbound 855/856/810/846 maps.
The order, shipment/carton/item, invoice/line and inventory schemas exist **only in test sources**.
Outbound maps support nested association paths; inbound target paths retain the one-association limit.
Preview shows parse counts, SNIP findings, the would-be ack, mapped JSON, expected-result and
previous-revision comparisons. It writes no target records. PHI previews suppress document values.
A session-authorized encrypted JSON report is downloadable. Select a stored sample or upload
synthetic EDI on the standard Review step. Browser uploads require the host's RC.6 Javalin
`uploadedFileArchiveTableName` to be `edi_preview_download` and multipart handling without disk
staging. The alias encrypts streams and persists samples; this process accepts no filesystem archive.
Samples must contain synthetic data only.
## Setup and starter data
No Liquibase changelog or host include is needed: the qbit seeds its own reference data. When the
host calls `EdiRuntime.initialize()` at startup (after the schema exists) the 41 built-in guides
load, with their segments, elements, rules, code lists and code values. It only changes a guide when
the bundled content changes, so edits to your own rows are kept.
Everything else is host or runtime data, and every table explains itself when empty (help bundle
`help/qbit-edi.yaml`). Set up in this order:
1. Partner.
2. Identities: one of yours (no partner) and one per partner.
3. Agreement: pairs your identity with the partner's, for a standard and test or production.
4. Agreement documents: one per transaction set and direction, with the guide and optionally a map.
5. Transport (AS2, SFTP, HTTPS), linked on the agreement. Certificates only for AS2 signing and encryption.
The **Setup status** section on the home shows each step, whether it is done (counting only the
active rows the runtime uses) and the next step.
Maps start empty. The **Install sample maps** action on Maps adds the 32 bundled synthetic examples
as draft maps with no partner and no agreement document, by name and never over an existing map. Point
a map at your own table, preview it, activate it, then select it on an agreement document.
## Permissions
Full table permissions include `edi.partner.read|insert|edit|delete`, `edi.map.*`,
`edi.transaction.read` and `edi.exception.read|edit`. Multiword names use dots
(e.g. `edi.map.rule.read`). Runtime tables disable people-facing mutations.
| Action | Permission |
| --- | --- |
| Reprocess | `edi.reprocess` |
| Preview Map | `edi.map.preview` |
| Assign / Ignore / Mark Resolved | `edi.exception.assign` / `edi.exception.ignore` / `edi.exception.resolve` |
| Create Partner Override / Activate Guide | `edi.guide.override` / `edi.guide.activate` |
| Activate Map | `edi.map.activate` |
| Send As EDI | `edi.send` |
| Resend / Resend As New | `edi.resend` |
| Cancel | `edi.cancel` |
| Test Connection | `edi.transport.test` |
| Promote Certificate | `edi.certificate.promote` |
| Export Public Certificate | `edi.certificate.read` |
| Scheduled workers / ESB enqueue | `edi.admin` |
Partner locks use `edi_partner` with `edi.admin` all-access. Home queries explicitly enforce
the viewing user's permissions. Processes are table/record actions, never navigation entries.
## Configuration
| Setting | Default / behavior |
| --- | --- |
| `backendName` | Required host RDBMS backend |
| `appName`, `appLabel`, `parentAppName` | `edi`, `EDI`, no parent |
| `schedulerName` | Null; configure for 15-second polling |
| `serviceSessionSupplier` | Null; required for inbound application and outbound source reads/mapping |
| `withSendAsEdiTable(name)` | Opt a host table into Send As EDI |
| `partnerLocks` | `false` |
| `maxSegmentsPerTransaction` | 100000 |
| `payloadStore` | `DATABASE` or `OBJECT_STORAGE` |
| `payloadBucketAlias` | `edi` |
| `secretsResolver` | Environment-variable lookup |
| `publicBaseUrl` | Null; the deployment base URL (for example `https://edi.example.com`), required for ASYNC AS2 receipts |
| `clock` | UTC system clock |
| `esbDestinationName` | Null; optional core ESB destination |
Set `QQQ_EDI_PAYLOAD_KEY` to a base64-encoded 32-byte master key; retain the old key in
`QQQ_EDI_PAYLOAD_KEY_PREVIOUS` during rotation. AES-256-GCM envelope encryption uses random data
keys and SHA-256 verification. Both stores use the same envelope and currently buffer a payload
in memory. Database storage remains the default for demos/tests.
For object storage, add/install optional `qbit-object-storage:0.1.0-RC.2`, configure its bucket
alias and select `OBJECT_STORAGE`. EDI uses only its public API and system access, with keys
`edi/yyyy/MM/dd/kind/uuid`. Its transitive QQQ dependencies are excluded to retain RC.6.
Optional `qqq-esb` publishes setup/activity events after commit. Document events contain only ID
and status because RC.6's standard factory otherwise copies PHI fields. Alert names are
`edi.inbox.dead-letter`, `edi.ack.overdue`, `edi.ack.rejected`, `edi.send.failed`,
`edi.certificate.expiring`, `edi.certificate.expired` and `edi.mdn.overdue` (RC.6 requires dotted names).
Core `DataClassification` metadata marks PHI/PII/CONFIDENTIAL fields through `AiFieldMetaData`,
with a CONFIDENTIAL table default through `AiTableMetaData`. PHI fields deny AI exposure.
Hosts must supply secured-data protection, database encryption at rest and access policies for PHI columns; markers alone do not mask fields.
The home reports installed integration and storage state.
Assign sends a value-free `QAlerts.notify` notification to the assignee after commit. Newly opened
or reopened REJECTED_ACK/DEAD_LETTER exceptions notify distinct assignees of OPEN/IN_PROGRESS
exceptions for the same partner; these existing assignments define the partner's recipients.
Repeated occurrences do not send another notification. Titles and bodies contain no document,
partner or exception-summary values. Without an alerts integration, core uses its logging sink.
## Outbound and transports
Active OUTBOUND agreement documents select the source table, map, guide and trigger. RECORD_CHANGE
uses core's instance listener and the writer's transaction. Source and EDI tables must share the
backend. `triggerFilter` is a serialized QQQ `QQueryFilter`; MANUAL also honors that filter.
RC.6 catches listener errors without failing the source write: successful enqueues roll back with
the source transaction, but hosts must monitor failed listener execution.
`edi_outbound_enqueue` is the ESB boundary. Configure an RC.6 `EsbTrigger` to invoke it with
`sourceTableName`, `sourceRecordId` and `eventId`; broker setup belongs to the host. Event IDs are
unique per agreement document. RC.6's consumer runner uses JMS, so broker-free acceptance calls
this process directly.
`edi_outbound_worker` and `edi_send_worker` run every 15 seconds when a scheduler is configured.
IMMEDIATE produces one interchange per request. BATCH cuts up to 100 requests for one agreement
document at its UTC `batchSchedule` into one group/interchange. Mapping/enveloping and sending
use database row locks to fence replicas; crashes release the lock. Remote delivery is at least
once: a lost connection after remote acceptance can resend the same bytes and control numbers.
Retry waits are 1m, 5m, 15m, 1h, 3h, 6h, 12h, 24h, capped at the last delay; `maxAttempts` defaults
to 8. Permanent transport rejection is FAILED; retry exhaustion is DEAD_LETTER. Both open a
TRANSPORT exception and raise `edi.send.failed`.
| Transport | Behavior and settings |
| --- | --- |
| GATEWAY | `gatewayEndpointName`; marked `FileGatewayOutbox` stand-in with a provider registration hook. Put is SENT; `EdiFileGatewayConsumer.onDownloaded` is DELIVERED. |
| SFTP | MINA client from core's filesystem module; required `sftpHostKeyFingerprint` in MINA SHA256 format, username and password/private-key env name. Streams upload to `.tmp`, then rename. `sftpPollCron` schedules remote pickup; `sftpFilePattern` is a Java regex (default `.*\.x12`). DELETE/MOVE occurs only after durable inbox commit. |
| HTTPS | POST with NONE/BASIC/BEARER/HMAC; TLS required except loopback. BASIC's secret is `user:password`; BEARER's is the token; HMAC uses core Standard Webhooks `whsec_` format. `httpsHeadersJson` holds nonsecret headers only. Redirects are refused; 2xx is DELIVERED, 429/5xx retry. |
| AS2 | See [AS2](#as2). |
| EMAIL | Settings validation only. Sending returns a permanent unavailable result until the qbit-email API ships. |
Optional Javalin registers `POST /qqq/hooks/edi/https/{transport}`. `httpsInboundAuth` must be
BASIC, BEARER or HMAC with `httpsSecretEnvVar`. It authenticates exact bytes, caps intake at
50 MiB and returns 202 only after durable storage. HMAC verifies timestamps and deduplicates by
transport plus webhook ID. TLS terminates at the host ingress.
The transport registry is `EdiTransportPluginsMetaData`, attached to `QInstance`. Hosts can
register/replace implementations with `withPlugin`. All transports open encrypted PayloadStore
streams; production code never writes local files.
Ack deadlines start on the first successful send. `edi_ack_monitor` runs every five minutes,
raises overdue alerts and exceptions, and checks certificate expiry at 30/14/7 days and expiry.
A functional ack resolves its alert and exception; TA1 updates interchange acknowledgement only.
Resend retains archived bytes/control numbers. Resend As New re-envelopes the archived document
without re-running its map and links `originalInterchangeId`. Cancel accepts QUEUED/RETRY_WAIT only.
Existing a2 databases need `edi_interchange.original_interchange_id INTEGER` and a unique index
on `edi_outbound_request(agreement_document_id, event_id)` in addition to their prior schema.
## AS2
`As2Transport` (type `AS2`) sends and receives AS2 (RFC 4130) with S/MIME signing and encryption
(SHA-256/384/512, AES-128/256 CBC or GCM; SHA1 and 3DES only when set explicitly and flagged by
the read-only `weakCrypto` field), optional zlib compression, and synchronous or asynchronous
signed MDNs. Everything is in memory: no keystore, key or temporary file is ever written.
**Transport settings** (`edi_transport`, type AS2): `as2LocalId`, `as2PartnerId`, `as2Url` (HTTPS;
plain HTTP only on loopback), `as2Subject`, `signAlgorithm` (blank = unsigned), `encryptAlgorithm`
(blank = unencrypted), `compress`, `mdnMode` (`SYNC` default, `ASYNC`, `NONE`), `mdnSigned`,
`localCertificateId` (ours), `partnerCertificateId`, optional `partnerTlsCertificateId` (pins the
partner TLS certificate), and optional `httpAuthUserEnvVar` / `httpAuthPasswordEnvVar` for HTTP
Basic. For inbound, a set `signAlgorithm` / `encryptAlgorithm` means signed / encrypted messages are
required (otherwise `insufficient-message-security`). A VAN is an AS2 transport with `isVan` true
and no `partnerId`; its partner certificate belongs to the VAN operator's partner record, and
routing to trading partners stays by ISA identity.
**Certificates** (`edi_certificate`): paste the public PEM only; subject, issuer, serial,
SHA-256 fingerprint and validity are parsed on save, and a pasted private key is refused. Our private
key is never stored: `privateKeyEnvVar` names an environment variable holding a PEM PKCS#8 key or a
base64 PKCS#12 store (with `privateKeyPasswordEnvVar` for its password). A missing or mismatched
variable fails plainly, naming only the variable. Status `ACTIVE` signs and encrypts outbound;
inbound accepts `ACTIVE` and `NEXT` (ours for decryption, the partner's for verification), so a
rollover is: add the new certificate as `NEXT`, tell the partner, then run **Promote Certificate**
(`NEXT` becomes `ACTIVE`, the previous `ACTIVE` of the same partner and usage becomes `RETIRED`).
**Export Public Certificate** downloads the PEM to send to a partner. The monitor alerts at 30, 14
and 7 days before expiry and at expiry for `ACTIVE` and `NEXT` certificates.
**Hook URLs** (outside `/qqq/v1`, authenticated per transport by AS2 identity, signature and
certificate; give your partner the first, and the second is sent in `Receipt-Delivery-Option`):
| URL | Purpose |
| --- | --- |
| `POST {base}/qqq/hooks/edi/as2/{transport}` | Receive messages; answers with the MDN, or 200 and a queued asynchronous MDN |
| `POST {base}/qqq/hooks/edi/as2/{transport}/mdn` | Receive asynchronous MDNs for messages we sent |
`{transport}` is the transport `name`. Set `EdiQBitConfig.withPublicBaseUrl(...)` so `ASYNC`
transports can announce the second URL. Bodies are limited by the largest `maxInterchangeBytes` of
the agreements on the transport (50 MB default). A retransmitted Message-ID is answered with the
original MDN and is not reprocessed; a bad signature, decryption or MIC returns an MDN with
`processed/error: authentication-failed`, `decryption-failed` or `integrity-check-failed` and creates
no inbox row; a storage failure returns `unexpected-processing-error` so the partner retries. Every MDN
sent or received is kept in the read-only `edi_as2_mdn` table (AS2 MDNs child lists on the
outbound message and the inbound message) with the raw MDN in the encrypted payload store.
`ASYNC` messages stay SENT until the MDN arrives; the `edi_as2_mdn_sender` step delivers the MDNs we
owe, retrying with the send backoff, and `edi.mdn.overdue` warns when an `ASYNC` MDN is later than
the agreement document `ackDueMinutes` (60 when unset).
Existing databases need the new `edi_as2_mdn` table, `edi_outbound_message.transport_mic` and
`edi_transport.weak_crypto` (see `docs/ddl/edi-postgres.sql`).
## HIPAA and PHI
Original **QQQ baseline guides** cover 837P (005010X222A1), 837I (005010X223A2),
835 (005010X221A1), 270/271 (005010X279A1), 276/277 (005010X212), 834 (005010X220A1)
and 999 (005010X231A1). These are in-house structure and usage baselines, not licensed TR3s.
No TR3 text or full licensed code lists ship. Hosts may import their licensed guides.
SNIP 1–2 covers envelopes and the active guide. SNIP 3 uses exact decimals for claim charges,
claim/line CAS adjustments and payments less signed PLB adjustments (zero tolerance). SNIP 4
rules run per loop occurrence, including dependent patients, claim/line dates, diagnoses,
subscriber demographics and benefits-or-rejection responses. SNIP 3+ findings produce `E` and
an exception; SNIP 1–2 errors produce `R`. HIPAA replies use 999 when acknowledgement is enabled.
SNIP 5 includes ICD-10 diagnosis shape, NPI Luhn with the 80840 prefix, and US ZIP shape.
`ExternalCodeSetProvider.isValid(codeSet, code, asOf)` supplies licensed membership checks for
EXTERNAL code lists; without it, only formats are checked. The System widget reports that state.
Healthcare agreement documents default to PHI and SNIP 4. Clinical keys go only to
`edi_transaction.phiBusinessKey`; validation excerpts, acknowledgements, events, exceptions,
alerts and process summaries contain no submitted clinical values. Healthcare sample maps and
host tables live **only in test sources/resources** and use seeded synthetic patients and claims.
Their NPI `1234567893` is a checksum test fixture, not an asserted provider identity.
**The host must encrypt the database at rest and should install qbit-secured-data.** Payloads
are always AES-GCM encrypted; PHI columns rely on host database encryption. RC.6 core
classification metadata marks PHI/PII/CONFIDENTIAL and denies PHI AI exposure, but does not
provide secured-data field tiers. Without secured-data policies, a user with table access can
read PHI columns.
The home explicitly reports this limitation. Use record-level audits on host PHI target tables
until core classified-value audit protection is available.
RC.6 still includes query criteria in RDBMS failure logs and has no logging redaction hook.
The marked `PhiLogScope` bridge suppresses synchronous logger diagnostics during PHI mapping/application and preview; ordinary value-free EDI events
remain persisted. It changes no logger levels and does not suppress other threads. The bridge
tracks core #1147 (secured-data logging), with a removal/review deadline of 4.1.0-RC.7.
Map samples use record-level audits to avoid copying expected-result PHI into audit details. Host code outside these scopes must protect its
own logging, audits and asynchronous work.
Download Raw Document uses a session-bound process grant, rechecks `edi.raw.download` and record
read/partner access at download time, streams decrypted bytes without local files, and records
`RAW_DOWNLOADED` with the user and row references, never content. Download filenames are generic.
`PhiCanaryTest` scans every five-character marker window through gateway/HTTPS intake, forced
SNIP 2/3/4 errors, mapping/apply, 999, replay/resend, Review/Result, alerts/notifications, DEBUG logs, all EDI
query exports and all home/child widgets. FULL exports exempt only PHI-classified columns;
RC.6 HIDDEN field-lock exports have no exemptions. MASKED tiers and the optional Quick Search
qbit require the newer secured-data integration and are not claimed by these RC.6 tests.
## Build and verification
```bash
./mvnw -B verify
```
Use the wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM. Verification runs Checkstyle,
synthetic H2 tests and JaCoCo (70% instructions, 90% classes). Object-storage key/envelope metadata
are unit-tested. Live S3, a deployed gateway/scheduler and browser layout are not exercised.
The public object-storage facade is final with restricted constructors; tests avoid provider internals.
## License
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
+4
View File
@@ -0,0 +1,4 @@
# Security Policy
* See Project [Security Policy](https://github.com/QRun-IO/qqq/blob/develop/SECURITY.md)
* Report a vulnerability privately through this repository's **Security** tab ("Report a vulnerability").
+9
View File
@@ -0,0 +1,9 @@
# Upgrade notes
## Unreleased
- No schema change and no Liquibase include.
- Behavior change: an inactive identity (`isActive` false) now stops X12 routing and sending, as it already did for EDIFACT. Inbound documents from or to it are rejected as an unknown partner, and outbound requests for its agreements wait until it is active. Identities default to active, so only identities someone switched off are affected.
- Hosts that already call `EdiRuntime.initialize()` at startup keep getting the 41 built-in guides and code lists. A host that never called it will now see "Needs setup" for Guides and code lists on the home.
- New home section "Setup status" (widget `edi_home_setup`).
- New table action "Install sample maps" on Maps, permission `edi.admin`. It is never run automatically and only adds draft maps whose name does not exist yet.
+246
View File
@@ -0,0 +1,246 @@
<?xml version="1.0"?>
<!--
~ QQQ - Low-code Application Framework for Engineers.
~ Copyright (C) 2021-2022. Kingsrook, LLC
~ 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
~ contact@kingsrook.com
~ https://github.com/Kingsrook/
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ https://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://checkstyle.sourceforge.net/dtds/configuration_1_3.dtd">
<!--
See reference at https://checkstyle.sourceforge.io/checks.html
-->
<module name = "Checker">
<property name="charset" value="UTF-8"/>
<property name="severity" value="warning"/>
<property name="fileExtensions" value="java, properties, xml"/>
<!-- Checks for whitespace -->
<!-- See http://checkstyle.sf.net/config_whitespace.html -->
<module name="FileTabCharacter">
<property name="eachLine" value="true"/>
</module>
<!--
<module name="LineLength">
<property name="max" value="100"/>
<property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
</module>
-->
<module name="TreeWalker">
<module name="SuppressWarningsHolder"/>
<module name="OuterTypeFilename"/>
<module name="IllegalTokenText">
<property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/>
<property name="format" value="\\u00(09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/>
<property name="message" value="Consider using special escape sequence instead of octal value or Unicode escaped value."/>
</module>
<module name="AvoidEscapedUnicodeCharacters">
<property name="allowEscapesForControlCharacters" value="true"/>
<property name="allowByTailComment" value="true"/>
<property name="allowNonPrintableEscapes" value="true"/>
</module>
<module name="AvoidStarImport"/>
<module name="OneTopLevelClass"/>
<module name="NoLineWrap"/>
<module name="EmptyBlock">
<property name="option" value="TEXT"/>
<property name="tokens" value="LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH"/>
</module>
<module name="NeedBraces"/>
<module name="LeftCurly">
<property name="option" value="nl"/>
</module>
<module name="RightCurly">
<property name="option" value="alone"/>
</module>
<!--
<module name="WhitespaceAround">
<property name="allowEmptyConstructors" value="true"/>
<property name="allowEmptyMethods" value="true"/>
<property name="allowEmptyTypes" value="true"/>
<property name="allowEmptyLoops" value="true"/>
<message key="ws.notFollowed"
value="WhitespaceAround: ''{0}'' is not followed by whitespace. Empty blocks may only be represented as '{}' when not part of a multi-block statement (4.1.3)"/>
<message key="ws.notPreceded"
value="WhitespaceAround: ''{0}'' is not preceded with whitespace."/>
</module>
-->
<module name="OneStatementPerLine"/>
<module name="MultipleVariableDeclarations"/>
<module name="ArrayTypeStyle"/>
<module name="MissingSwitchDefault"/>
<module name="FallThrough"/>
<module name="UpperEll"/>
<module name="ModifierOrder"/>
<module name="EmptyLineSeparator">
<property name="allowNoEmptyLineBetweenFields" value="true"/>
</module>
<module name="SeparatorWrap">
<property name="id" value="SeparatorWrapDot"/>
<property name="tokens" value="DOT"/>
<property name="option" value="nl"/>
</module>
<module name="SeparatorWrap">
<property name="id" value="SeparatorWrapComma"/>
<property name="tokens" value="COMMA"/>
<property name="option" value="EOL"/>
</module>
<module name="SeparatorWrap">
<!-- ELLIPSIS is EOL until https://github.com/google/styleguide/issues/258 -->
<property name="id" value="SeparatorWrapEllipsis"/>
<property name="tokens" value="ELLIPSIS"/>
<property name="option" value="EOL"/>
</module>
<module name="SeparatorWrap">
<!-- ARRAY_DECLARATOR is EOL until https://github.com/google/styleguide/issues/259 -->
<property name="id" value="SeparatorWrapArrayDeclarator"/>
<property name="tokens" value="ARRAY_DECLARATOR"/>
<property name="option" value="EOL"/>
</module>
<module name="SeparatorWrap">
<property name="id" value="SeparatorWrapMethodRef"/>
<property name="tokens" value="METHOD_REF"/>
<property name="option" value="nl"/>
</module>
<module name="PackageName">
<property name="format" value="^[a-z]+(\.[a-z][a-z0-9]*)*$"/>
<message key="name.invalidPattern"
value="Package name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="TypeName">
<message key="name.invalidPattern"
value="Type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="MemberName">
<property name="format" value="^[_a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Member name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="ParameterName">
<property name="format" value="^[a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Parameter name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="CatchParameterName">
<property name="format" value="^[a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Catch parameter name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="LocalVariableName">
<property name="tokens" value="VARIABLE_DEF"/>
<property name="format" value="^[a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Local variable name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="ClassTypeParameterName">
<property name="format" value="(^[A-Z][0-9]?)$|([A-Z][_A-Z0-9]*$)"/>
<message key="name.invalidPattern" value="Class type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="MethodTypeParameterName">
<property name="format" value="(^[A-Z][0-9]?)$|([A-Z][_A-Z0-9]*$)"/>
<message key="name.invalidPattern" value="Method type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="InterfaceTypeParameterName">
<property name="format" value="(^[A-Z][0-9]?)$|([A-Z][_A-Z0-9]*$)"/>
<message key="name.invalidPattern" value="Interface type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="NoFinalizer"/>
<module name="GenericWhitespace">
<message key="ws.followed" value="GenericWhitespace ''{0}'' is followed by whitespace."/>
<message key="ws.preceded" value="GenericWhitespace ''{0}'' is preceded with whitespace."/>
<message key="ws.illegalFollow" value="GenericWhitespace ''{0}'' should followed by whitespace."/>
<message key="ws.notPreceded" value="GenericWhitespace ''{0}'' is not preceded with whitespace."/>
</module>
<module name="Indentation">
<property name="basicOffset" value="3"/>
<property name="braceAdjustment" value="0"/>
<property name="caseIndent" value="3"/>
<property name="throwsIndent" value="6"/>
<property name="lineWrappingIndentation" value="3"/>
<property name="arrayInitIndent" value="6"/>
</module>
<!--
<module name="AbbreviationAsWordInName">
<property name="ignoreFinal" value="false"/>
<property name="allowedAbbreviationLength" value="1"/>
</module>
<module name="OverloadMethodsDeclarationOrder"/>
<module name="VariableDeclarationUsageDistance"/>
<module name="CustomImportOrder">
<property name="sortImportsInGroupAlphabetically" value="true"/>
<property name="separateLineBetweenGroups" value="true"/>
<property name="customImportOrderRules" value="STATIC###THIRD_PARTY_PACKAGE"/>
</module>
-->
<module name="CustomImportOrder">
<property name="customImportOrderRules" value="SPECIAL_IMPORTS###STANDARD_JAVA_PACKAGE###THIRD_PARTY_PACKAGE###STATIC"/>
<property name="specialImportsRegExp" value="^javax\."/>
<property name="standardPackageRegExp" value="^java\."/>
<property name="sortImportsInGroupAlphabetically" value="true"/>
<property name="separateLineBetweenGroups" value="false"/>
</module>
<module name="MethodParamPad"/>
<module name="NoWhitespaceBefore">
<property name="tokens" value="COMMA, SEMI, POST_INC, POST_DEC, DOT, ELLIPSIS, METHOD_REF"/>
<property name="allowLineBreaks" value="true"/>
</module>
<module name="ParenPad"/>
<module name="OperatorWrap">
<property name="option" value="NL"/>
<property name="tokens" value="BAND, BOR, BSR, BXOR, DIV, EQUAL, GE, GT, LAND, LE, LITERAL_INSTANCEOF, LOR, LT, MINUS, MOD, NOT_EQUAL, PLUS, QUESTION, SL, SR, STAR, METHOD_REF "/>
</module>
<module name="AnnotationLocation">
<property name="id" value="AnnotationLocationMostCases"/>
<property name="tokens" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF"/>
</module>
<module name="AnnotationLocation">
<property name="id" value="AnnotationLocationVariables"/>
<property name="tokens" value="VARIABLE_DEF"/>
<property name="allowSamelineMultipleAnnotations" value="true"/>
</module>
<module name="JavadocMethod">
<property name="allowMissingParamTags" value="true"/>
<property name="allowMissingReturnTag" value="true"/>
<property name="allowedAnnotations" value="Override, Test"/>
</module>
<module name="MissingJavadocMethod">
<property name="scope" value="private"/>
</module>
<module name="MissingJavadocType">
<property name="scope" value="private"/>
</module>
<module name="MethodName">
<property name="format" value="^[a-z][a-zA-Z0-9_]*$"/>
<message key="name.invalidPattern"
value="Method name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="EmptyCatchBlock">
<property name="exceptionVariableName" value="expected"/>
</module>
<module name="CommentsIndentation"/>
<!-- this would be nice, but requires the the @inheritDoc javadoc tag... -->
<module name="MissingOverride"/>
</module>
<module name="Header">
<property name="headerFile" value="checkstyle/license.txt"/>
<property name="fileExtensions" value="java"/>
<property name="ignoreLines" value="3"/>
</module>
<module name="SuppressWarningsFilter"/>
</module>
+19
View File
@@ -0,0 +1,19 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-30
View File
@@ -1,30 +0,0 @@
# Getting Started
Targets Java 21, Maven 3.8 or later, and QQQ 4.0.0. Maven resolves the framework dependencies from Maven Central.
1. Choose **Use this template** on GitHub, then clone the repository you created.
2. In `pom.xml`, set your own `groupId`, `artifactId`, name, description, and version.
3. Use your IDE's package refactoring to rename `com.kingsrook.qbits.example` throughout `src/`. Rename the `Example*` classes and update their imports and references with the IDE's rename refactoring.
4. Update the QBit producer's `GROUP_ID`, `ARTIFACT_ID`, and `VERSION` constants to match your project. Give the metadata names their own stable names before combining the QBit with other examples.
5. Build the generated project:
```bash
mvn clean verify
```
The repository provides example Java sources; customization uses normal package/class refactoring. Keep example coordinates unpublished. After customizing, add tests for your QBit's behavior before setting up its publishing workflow.
## Register with a host application
The following uses the original example names; substitute the names chosen above:
```java
new ExampleAppQBitProducer()
.withConfig(new ExampleAppQBitConfig()
.withBackendName("rdbms"))
.produce(qInstance, "my-application");
```
Register the configured backend in the host first. If you add table prefixes or multiple QBit instances, update related table references, possible-value-source names, and process/widget names consistently. A table prefix alone does not namespace all metadata.
[Tables and Relationships](01-tables-and-relationships.md) describes the model; the `processes/` and `widgets/` directories contain the remaining examples.
-50
View File
@@ -1,50 +0,0 @@
# Tables and Relationships
Application QBits typically have multiple interconnected tables representing a business domain.
## Entity Design
Use `@QMetaDataProducingEntity` for automatic metadata generation:
```java
@QMetaDataProducingEntity(producePossibleValueSource = true)
public class Order extends QRecordEntity
{
public static final String TABLE_NAME = "order";
@QField(isPrimaryKey = true)
private Integer id;
@QField(isRequired = true, possibleValueSourceName = "customer")
private Integer customerId;
@QField(isRequired = true)
private String status;
}
```
## Relationships
Define foreign keys using `possibleValueSourceName`:
```java
@QField(isRequired = true, possibleValueSourceName = "exampleEntity")
private Integer exampleEntityId;
```
## Table Prefixing
Application QBits support optional table prefixing:
```java
// Without prefix: order, orderLine
// With prefix "sales": sales_order, sales_orderLine
new MyAppQBitProducer()
.withConfig(new MyAppQBitConfig()
.withBackendName("rdbms")
.withTableNamePrefix("sales"))
.produce(qInstance, "sales-app");
```
The producer applies the prefix to table names. When extending this pattern, also update possible-value-source names, relationship targets, process tables, and navigation references; table renaming alone does not provide complete multi-instance support.
-7
View File
@@ -1,7 +0,0 @@
# Processes
Start with [ExampleProcessMetaDataProducer](../src/main/java/com/kingsrook/qbits/example/processes/ExampleProcessMetaDataProducer.java) and [ExampleProcessStep](../src/main/java/com/kingsrook/qbits/example/processes/ExampleProcessStep.java). The producer defines the process, its table, inputs, and backend step; the step transforms `PENDING` records to `PROCESSED` and returns them in the output.
In QQQ 4.0, import `MetaDataProducerInterface` from `com.kingsrook.qqq.backend.core.model.metadata`. Define process inputs through `withInputMetaData(new QFunctionInputMetaData().withField(...))`. An `AbstractTransformStep` must implement both `runOnePage(RunBackendStepInput, RunBackendStepOutput)` and `getProcessSummary(RunBackendStepOutput, boolean)`; keep the example's implementations when renaming it.
To make the process discoverable in navigation, add its registered name to a `QAppSection` through `withProcesses(List.of(ExampleProcessMetaDataProducer.NAME))`, then include the section in a `QAppMetaData` registered with `qInstance.addApp(...)`. Use the actual registered name if you introduce a prefix. Add tests for record changes and failure cases when replacing the example business logic.
-23
View File
@@ -1,23 +0,0 @@
# Widgets and UI
The application producer registers a `QAppMetaData` containing a `QAppSection`. The section references metadata names already registered in the instance; it is not registered directly on `QInstance`.
```java
QAppSection section = new QAppSection()
.withName("exampleSection")
.withLabel("Example Application")
.withTables(List.of(ExampleEntity.TABLE_NAME, ExampleChildEntity.TABLE_NAME))
.withProcesses(List.of(ExampleProcessMetaDataProducer.NAME))
.withWidgets(List.of(ExampleDashboardWidgetMetaDataProducer.NAME));
qInstance.addApp(new QAppMetaData()
.withName("exampleApp")
.withLabel("Example Application")
.withSection(section));
```
Use [ExampleAppQBitProducer](../src/main/java/com/kingsrook/qbits/example/ExampleAppQBitProducer.java) as the starting point. Keep these references consistent when enabling/disabling tables or introducing prefixes.
The widget has two parts: [its metadata producer](../src/main/java/com/kingsrook/qbits/example/widgets/ExampleDashboardWidgetMetaDataProducer.java) declares a statistics widget and its renderer, and [its renderer](../src/main/java/com/kingsrook/qbits/example/widgets/ExampleDashboardWidgetRenderer.java) returns a `RenderWidgetOutput` containing `StatisticsData`. Replace the example count with a query or calculation. Choose a widget type that matches the returned data model.
For QQQ 4.0, `AbstractWidgetRenderer` is in `com.kingsrook.qqq.backend.core.actions.dashboard.widgets`; `RenderWidgetInput` and `RenderWidgetOutput` are in `com.kingsrook.qqq.backend.core.model.actions.widgets`.
+82
View File
@@ -0,0 +1,82 @@
# EDI local demo
From this worktree with Java 21:
```powershell
./mvnw.cmd -B test-compile exec:java '-Dexec.classpathScope=test' '-Dexec.mainClass=com.kingsrook.qbits.edi.demo.EdiDemoServer' '-Dexec.args=8099'
```
Open http://localhost:8099/app/edi. Stop with Ctrl+C. On Unix use `./mvnw`.
Next UI table routes are `/app/<tableName>` and record routes are `/app/<tableName>/<id>`.
Each launch creates a fresh in-memory H2 database, an in-memory synthetic payload key and a
fully anonymous demo session. No external credentials or services are required. Demo auth is
in test sources only and must not be installed in a real host.
The seed includes Northwind Retail (gateway), Harbor Logistics (SFTP) and Summit Market (HTTPS),
with codes NORTHWIND, HARBOR and SUMMIT and synthetic contacts at `example.invalid`.
It includes identities/agreements, guides/maps, orders with lines, shipments with cartons/items,
invoices and inventory. Forty synthetic inbound documents go through intake and the real inbox worker:
850s, 860s, validation errors, an unknown sender and a byte duplicate. Northwind's 855/856/810/846
pass the outbound worker and an in-memory gateway; incoming 997s reconcile some sent groups.
An overdue group, an exhausted send and several open exceptions provide operational examples.
Public self-signed certificates are generated in memory (two for expiry alerts, three for AS2); private keys are never persisted.
Pacific Wholesale (code PACIFIC) is the AS2 partner. An in-process BouncyCastle peer listens on a
random loopback port and the seed generates synthetic RSA certificates in memory (our ACTIVE and NEXT
certificates and the partner's; the private keys live only in the demo secrets map and are never
written to disk). The seed sends three 850s to the application's own AS2 hook (signed, encrypted;
two with base64 transfer encoding), which answers with signed MDNs (`edi_as2_mdn` rows, shown as
"AS2 MDNs" on the inbound message); the 997s go back over AS2 as well. An 855 built from a source record is sent
over AS2 to the peer, which verifies the signature and MIC and returns a signed synchronous MDN
(the outbound message becomes DELIVERED with its `transport_mic`). One more message goes to the
asynchronous transport `pacific-as2-async`: it stays SENT until the peer posts the MDN to
`/qqq/hooks/edi/as2/pacific-as2-async/mdn`. The running demo serves
`POST /qqq/hooks/edi/as2/pacific-as2` (and `.../mdn`) on the same port; Test Connection,
Export Public Certificate and Promote Certificate (on `demo-as2-next`) work on the AS2 records.
Phase d adds warehouse and logistics traffic, seeded last so the retail scenarios above are
unchanged. Harbor Logistics (the 3PL) gets a gateway OUTBOX transport (the partner pulls over SFTP)
and sends three valid 940 shipping orders and one with a bad unit; the orders and lines land in
`sample_wms_order`, and two shipments (one short) go out as 945s. A fourth partner, Ridgeline Freight
(code RIDGELINE, gateway transport), receives two 204 load tenders (the second a change) and sends
four 214 status events (one rejected with a bad status code) that build `sample_shipment_status`.
The in-memory gateway accepts every send, and the rejected documents produce rejecting 997s.
Phase g adds a fifth partner, **Lindqvist Retail AB** (LINDQVIST), after the existing scenarios.
Its D96A test agreement receives three synthetic ORDERS documents through the durable inbox;
orders and lines appear in `sample_order`, and CONTRLs enter the gateway queue. ORDRSP, DESADV
and INVOIC are mapped from the retail sample records, delivered through the in-memory gateway,
and reconciled by incoming accepted CONTRLs. Look for ORDERS with status APPLIED and the three
outbound message types with status ACKNOWLEDGED. Released delimiters in buyer/item identifiers
exercise the same parser used for partner input. All contacts remain synthetic `example.invalid`.
The SFTP/HTTPS partner settings are illustrative `.invalid` destinations. Their agreements have
no outbound transport, so they cannot contact a real partner; resulting acknowledgement send
retries are visible. Northwind, Harbor, Ridgeline and Lindqvist have configured outbound documents; the healthcare scenario adds its own. The in-memory gateway
is restored after the intentional dead-letter scenario. Standard Send As EDI, Test Connection,
Resend, Resend As New and Cancel actions are available. The gateway stores bytes in memory;
it does not expose a separate partner download UI.
Acceptance boots a random-port QApplicationJavalinServer, loads Next UI, renders and serializes
every registered widget through core RenderWidgetAction/JsonUtils, and POSTs every home and
child-list widget over HTTP using seeded parent records. It also checks configuration JSON,
metadata, table queries, labels, field order, foreign-key value sources and PHI-safe record labels.
Next UI defaults table queries to descending ID (newest records first); activity child lists use
descending creation date and ID. The home exception queue puts OPEN before IN_PROGRESS, newest
first within each status. RC.5 has no table-level default-sort metadata to override Next's sort.
Inbound message labels use external IDs because filenames are classified as PHI.
Light/dark visual layout and production scheduler/gateway deployment remain host acceptance.
Phase e adds a fifth partner, **Cedar Health Clearinghouse** (CEDAR, PHI enabled), after the existing
retail and logistics scenarios. It sends seeded synthetic professional and institutional claims,
a remittance, a professional claim with a one-dollar balancing error (SNIP 3), and a professional
claim missing its dependent patient loop (SNIP 4). Both business-rule errors are accepted with
errors (`E`) and open exceptions. Value-free 999s go to the in-memory clearinghouse outbox.
A synthetic eligibility request goes out as a 270 and a matching-trace 271 arrives. Clinical keys
appear only in the classified PHI key column, not the ordinary document key or record labels.
The healthcare sample maps and tables are test-only. Patients are seeded `Testpatient Alpha`
fixtures, member IDs begin with `SYN`, and `1234567893` is a documented NPI checksum test value.
The host database must be encrypted for real PHI; this disposable demo uses synthetic data only.
The System widget shows format-only external code validation until a provider is registered.
Inbox and interchange records offer the audited **Download Raw Document** action.
+1068
View File
File diff suppressed because it is too large. Load diff
+78
View File
@@ -0,0 +1,78 @@
# EDI guide JSON
The files in `src/main/resources/edi/guides/` are original QQQ baseline guides, not licensed
X12/TR3 documentation or complete code sets. JSON loads into `guide.Guide` with
`BuiltInGuideLoader.read(InputStream)`. `GuideRepository.load(id)` reconstructs the same model from
the normalized `edi_guide*` and code-list tables.
The root has `name`, `standard`, `version`, `transactionSet`, `functionalIdCode`, ordered
`segments`, and optional `rules`. A segment has:
- `segmentId`, `usage` (`REQUIRED`, `SITUATIONAL`, `NOT_USED`), `minUse`, `maxUse`;
- optional `loopId`, `parentLoopId`, and `loopMaxRepeat`; the first segment starts each loop
occurrence, with parents declared before children;
- optional boolean `condition`, `elements`, and `rules`.
An element has one-based `position` and optional `componentPosition`, `elementId`, `dataType`
(`AN`, `ID`, `N0`–`N9`, `R`, `DT`, `TM`, `B`), `minLength`, `maxLength`, `usage`, `fixedValue`,
`repeatMax`, and `condition`. `codes` is an optional array of permitted ID values. Omitted/null
means no code constraint; an explicitly empty array permits no values. DT accepts YYMMDD with
a 1950–2049 century window or CCYYMMDD; TM accepts HHMM, HHMMSS, and one/two fractional digits.
N types accept integer lexical values with an implied decimal point.
Rules use `type` (`PAIRED`, `REQUIRED`, `EXCLUSION`, `CONDITIONAL`, `LIST_CONDITIONAL`, or
`SITUATIONAL`), `positions` for syntax notes, or `expression` for situational rules. Optional
`snipLevel`, `severity`, `message`, `ak3Code`, `ak4Code`, `ik3Code`, and `ik4Code` customize findings.
Syntax-note positions are one-based elements. Segment/element `condition` means “required when
this expression is true.” Explicit `when(...)` rules express prohibitions and other relationships.
Paths include `BEG03`, `REF01-2`, `N1/N101`, `N1[N101=ST]/N301`, `PO1[*]/PO102`, and
`N1[2]/N101`. Indexes start at one; unindexed paths select all occurrences. Loop scopes come from
the guide, so a qualified parent selects its child segments. Functions are `present`, `absent`,
`count`, `sum`, `eq`, `in`, `matches`, `date`, `and`, `or`, `not`, and `when`. String literals use
single quotes; numeric literals use decimal notation. `date` tests calendar validity. Examples:
```text
when(eq(N101, 'ST'), present(N3))
eq(sum(PO102), 12)
and(in(BEG01, '00', '01'), date(BEG05))
```
No script engine runs. PHI validation suppresses all excerpts and custom rule messages. Functional
acks omit submitted values; situational 999 CTX records carry only trigger positions, following
[X12's CTX interpretation](https://x12.org/resources/requests-for-interpretation/rfi-1311-999ctx-context).
At host startup, after schema creation and service-session initialization, call
`EdiRuntime.current().initialize()`. Unchanged resource checksums are no-ops. A changed resource
creates a new revision and retires the former active built-in revision in one database transaction;
historical rows and their children remain addressable by id. A reverted resource also creates a
new revision. Built-in guide writes require the loader's security key (`edi.guide.loader`), or a
system service session. Configuration editors can receive `CUSTOM`/`IMPORTED` values of the
`edi_guide_source` key without receiving permission to edit `BUILT_IN` rows.
## EDIFACT baselines
Set `standard` to `EDIFACT`, `transactionSet` to the UNH message type, and `version` to the
concatenated UNH version/release (`D96A`, `D01B`). The ten bundled resources are in-house retail
subsets with short field names and only the code values needed by the maps. They contain no
copied directory descriptions and are not complete UNECE directories or EANCOM certifications.
UNH and UNT appear as actual guide segments. UNB/UNZ and UNG/UNE are checked by the envelope
reader. UNH association codes, such as `EAN008`, remain available as component 5 of UNH02.
EDIFACT mandatory (`M`) maps to `REQUIRED`; conditional (`C`) maps to `SITUATIONAL` with
`condition` or rules where needed. A component uses the existing `position` plus
`componentPosition`, for example NAD02 component 1. EDIFACT decimal values accept `.` or `,`;
DECIMAL/UNIT inbound transforms normalize them and outbound DECIMAL uses the agreement's mark.
Segment groups use the existing loop tree: SG1 starts with RFF, SG2 with NAD, and the retail
line loop SG25 starts with LIN. These are stable local guide loop labels, not a claim that every
message/directory assigns the same published group numbers. DESADV uses SG10 (CPS) containing
SG17 (LIN), matching carton/item associations. Map examples are `NAD[NAD01=BY]/NAD02-1`,
`SG25/LIN/LIN03-1` and `SG25[*]/QTY/QTY01-2`. Named loops take precedence over element parsing,
so `SG25` is a loop rather than an element of a hypothetical `SG` segment.
CONTRL baselines include UCI, optional UCF group responses, UCM message responses and nested
UCS/UCD positional errors. Shared validator diagnostics are translated to EDIFACT 0085 codes;
submitted erroneous values are not copied into the acknowledgement. `GROUP_ONLY` reports the
interchange outcome, `ALL_TRANSACTIONS` includes accepted messages, and `ERRORS_ONLY` includes
rejected message details. A CONTRL is never itself acknowledged. APERAK is deferred.
+91
View File
@@ -0,0 +1,91 @@
# EDI maps
A map comprises an `edi_map` row and ordered `edi_map_rule` rows. EDI paths resolve through the
guide's loop tree; target paths support one association level. `condition`, `defaultValue` and
`required` apply within each loop occurrence. Missing optional values preserve child alignment.
`transformArgs` is a JSON object with string values. Omitted transforms copy the first value.
An inbound `ediPath` may join several element paths with `+` (outside any `[...]` predicate); their
values are concatenated in order. This is how a date and a time held in separate elements reach
`DATETIME`: `G62[G6201=10]/G6202+G62[G6201=10]/G6204`. Inside a `[*]` loop path only the part after
the anchor is split (`S5[*]/G6202+G6204`). A `condition` such as `present(G62[G6201=10]/G6202)` skips
the rule when the elements are absent; a transform with no input otherwise fails the rule.
| Transform | Arguments / result |
| --- | --- |
| DATE | CCYYMMDD or YYMMDD; fixed 1950–2049 century window; QQQ date |
| TIME | HHMM, HHMMSS, fractional seconds; normalized string |
| DATETIME | Date and time values, or `time` argument; `timezone` defaults UTC |
| DATEPART / TIMEPART | Outbound halves of a DATETIME: a QQQ timestamp (an instant, an offset timestamp, or a zone-less one read as UTC) rendered as CCYYMMDD or HHMMSS in the `timezone` argument (default UTC) |
| DECIMAL | `scale` implied decimal places, default 0 |
| QUALIFIER | Sibling predicate in `ediPath`, e.g. `N1[N101=ST]/N104` |
| CODE | Map-local translation object, e.g. `{"EA":"each","CA":"case"}` |
| UNIT | `factor`, `divisor`, `scale` (defaults 1, 1, 6), HALF_UP |
| CONSTANT | `value`; needs no EDI input |
| CONCAT | `separator`, default empty; joins selected values |
| SPLIT | Literal `separator` (comma), zero-based `index` (0) |
| SUBSTRING | Zero-based `start` inclusive, `end` exclusive |
| TRIM / UPPER | Whitespace trim / locale-independent uppercase |
| LOOKUP | `table`, `key`, `result` (default id); exactly one permitted row |
| SEQUENCE | Outbound loop occurrence, starting at `start` (default 1) |
| HL | Outbound HL01 counter, HL02 nearest parent, HL03 `code`, HL04 child indicator |
Supply the partner timezone explicitly for DATETIME. UNIT uses configured conversion factors/pack
sizes; host-dependent field lookups and code-list translation can register named transforms through
`EdiRuntime.current().transforms().register(name, transform)`. Transforms execute as the service
session. LOOKUP explicitly enforces permissions.
Samples reference encrypted PayloadStore objects. Preview JSON contains `values` and recursive
`associations`; `expectedResultJson` uses the same shape. Comparisons show structured before/after
JSON as Result lines. Preview never invokes target application. See the synthetic 850 resources
under `src/main/resources/edi/maps` and [GUIDE-FORMAT.md](GUIDE-FORMAT.md).
## Outbound maps
OUTBOUND rules are evaluated in sequence order. EDI path prefixes create nested loop scopes;
the directly mapped QQQ association drives each occurrence. Nested QQQ paths such as
`cartons.items.sku` are supported and fetched recursively by the worker. Constants within a
loop can carry an association field as `qqqPath` to identify their scope. See the 856 samples
for Shipment/Order/Pack/Item loops and parent numbering.
DATE/TIME format QQQ values as X12 dates/times; DECIMAL multiplies by the implied `scale`.
Other registered transforms operate on the source value. Conditions use the existing EDI rule
language over segments already emitted. A missing required mapping input fails mapping;
structurally invalid mapped output opens VALIDATION and is never sent. ST/SE or UNH/UNT are generated by
the mapper; the worker allocates controls. Component-target paths use `-`, for example
`LIN03-1` and `LIN03-2`, and retain components separately so the EDIFACT writer can escape data.
## Warehouse and logistics samples
Phase d adds baseline guides and sample maps for four documents; test sources define the synthetic
`sample_wms_*`, `sample_shipment_status` and `sample_load*` tables. Maps with a second direction use
an `-in` or `-out` file suffix.
| File | Direction | Target | Business key |
| --- | --- | --- | --- |
| `940-*-sample.json` | INBOUND | `sample_wms_order` + `lines` | W0502 (secondary W0503) |
| `945-*-sample.json` / `945-in-*` | OUTBOUND / INBOUND | `sample_wms_shipment` + `lines` | W0602 |
| `214-*-sample.json` | INBOUND | `sample_shipment_status` (flat, first LX event) | B1002 (secondary B1001) |
| `204-*-sample.json` / `204-out-*` | INBOUND / OUTBOUND | `sample_load` + `stops` | B204 |
Order dates and appointments arrive as `DATETIME` from a qualified G62 (`G62[G6201=10]/...`) or from
the stop's G62 and leave as `DATEPART` plus `TIMEPART`. The 204 stop address is read from the nested
loop (`S5[*]/S5N1/N102`; the guide gives the stop-level N1 loop its own id, `S5N1`, so it does not
collide with the header N1 loop) and the stop sequence is mapped from S501. A 214 can carry several LX
events; the sample map records the first (`LX[1]/...`) and a host that wants all of them maps an
association instead. Optional outbound qualifiers such as MAN01 and L1102 are constants, so the
sample source rows must carry the value they qualify (the carton SSCC, the bill of lading).
## EDIFACT retail samples
Each of D96A and D01B includes `ORDERS-<version>-sample.json` inbound to `sample_order` and
`lines`, and outbound `ORDRSP` from `sample_order`, `DESADV` from `sample_shipment` with
`cartons.items`, and `INVOIC` from `sample_invoice` with `lines`. Test sources define these
synthetic tables. ORDERS carries its business key in BGM02, date in qualified DTM01 component 2,
buyer in qualified NAD02 component 1, and items in SG25. BGM's document codes are 220, 231, 351
and 380 respectively. These subsets intentionally carry only the fields exercised by the samples.
Composite paths address values, not delimiter-joined strings. An identifier containing `+`, `:`
or `'` remains a single value; the writer supplies release escapes. The outbound agreement owns
UNA settings. Separate map resources per directory keep future version-specific changes isolated.
+570
View File
@@ -0,0 +1,570 @@
CREATE TABLE edi_agreement (
apply_test_documents BOOLEAN,
component_separator VARCHAR(255),
control_number_scope VARCHAR(255),
create_date TIMESTAMP,
decimal_mark VARCHAR(255),
duplicate_window_days INTEGER,
element_separator VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbound_transport_id INTEGER,
interchange_version VARCHAR(255),
is_active BOOLEAN,
line_ending VARCHAR(255),
local_identity_id INTEGER,
max_interchange_bytes BIGINT,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
notes VARCHAR(10000),
on_duplicate VARCHAR(255),
outbound_transport_id INTEGER,
partner_id INTEGER,
partner_identity_id INTEGER,
release_character VARCHAR(255),
repetition_separator VARCHAR(255),
request_ta1 BOOLEAN,
segment_terminator VARCHAR(255),
send_ta1 VARCHAR(255),
standard VARCHAR(255),
syntax_identifier VARCHAR(255),
usage_indicator VARCHAR(255),
UNIQUE (local_identity_id, partner_identity_id, standard, usage_indicator)
);
CREATE TABLE edi_agreement_document (
ack_due_minutes INTEGER,
ack_expectation VARCHAR(255),
ack_level VARCHAR(255),
agreement_id INTEGER,
batch_mode VARCHAR(255),
batch_schedule VARCHAR(255),
contains_phi BOOLEAN,
create_date TIMESTAMP,
direction VARCHAR(255),
functional_id_code VARCHAR(255),
guide_id INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
is_active BOOLEAN,
map_id INTEGER,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
outbound_trigger VARCHAR(255),
snip_level INTEGER,
source_table_name VARCHAR(255),
target_table_name VARCHAR(255),
transaction_set VARCHAR(255),
trigger_filter VARCHAR(10000),
version VARCHAR(255),
UNIQUE (agreement_id, direction, transaction_set, version)
);
CREATE TABLE edi_as2_mdn (
create_date TIMESTAMP,
delivered_at TIMESTAMP,
delivery_attempts INTEGER,
delivery_status VARCHAR(20),
delivery_url VARCHAR(1000),
direction VARCHAR(20),
disposition VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbox_id INTEGER,
message_id VARCHAR(255),
mic_algorithm VARCHAR(50),
mic_value VARCHAR(255),
modify_date TIMESTAMP,
next_attempt_at TIMESTAMP,
original_message_id VARCHAR(255),
outbound_message_id INTEGER,
partner_id INTEGER,
payload_ref VARCHAR(255),
received_at TIMESTAMP,
signature_valid BOOLEAN,
signed BOOLEAN,
transport_id INTEGER
);
CREATE TABLE edi_certificate (
alias VARCHAR(255),
certificate_pem VARCHAR(10000),
create_date TIMESTAMP,
fingerprint_sha256 VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
issuer VARCHAR(255),
modified_by VARCHAR(255),
modify_date TIMESTAMP,
not_after TIMESTAMP,
not_before TIMESTAMP,
partner_id INTEGER,
private_key_env_var VARCHAR(255),
private_key_password_env_var VARCHAR(255),
serial_number VARCHAR(255),
status VARCHAR(20),
subject VARCHAR(255),
usage VARCHAR(255)
);
CREATE TABLE edi_code_list (
create_date TIMESTAMP,
element_id VARCHAR(255),
external_code_set VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
source VARCHAR(255)
);
CREATE TABLE edi_code_list_value (
code VARCHAR(255),
code_list_id INTEGER,
create_date TIMESTAMP,
description VARCHAR(10000),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
is_active BOOLEAN,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
UNIQUE (code_list_id, code)
);
CREATE TABLE edi_control_number (
agreement_id INTEGER,
create_date TIMESTAMP,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
last_value BIGINT,
level VARCHAR(255),
local_identity_id INTEGER,
max_value BIGINT,
min_value BIGINT,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
scope_key VARCHAR(255),
wrap_allowed BOOLEAN,
UNIQUE (scope_key, level)
);
CREATE TABLE edi_event (
at TIMESTAMP,
create_date TIMESTAMP,
detail VARCHAR(10000),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbox_id INTEGER,
interchange_id INTEGER,
modify_date TIMESTAMP,
transaction_id INTEGER,
type VARCHAR(255),
user_name VARCHAR(255)
);
CREATE TABLE edi_exception (
assigned_to_user_id VARCHAR(255),
create_date TIMESTAMP,
detail VARCHAR(10000),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbox_id INTEGER,
interchange_id INTEGER,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
occurrences INTEGER,
outbound_message_id INTEGER,
partner_id INTEGER,
resolution VARCHAR(10000),
resolved_at TIMESTAMP,
resolved_by VARCHAR(255),
status VARCHAR(255),
summary VARCHAR(255),
transaction_id INTEGER,
type VARCHAR(255)
);
CREATE TABLE edi_functional_group (
ack_due_at TIMESTAMP,
ack_received_at TIMESTAMP,
ack_status VARCHAR(255),
ack_transaction_id INTEGER,
control_number VARCHAR(255),
create_date TIMESTAMP,
functional_id_code VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbound_position INTEGER,
interchange_id INTEGER,
modify_date TIMESTAMP,
sender_id VARCHAR(255),
transaction_count INTEGER,
version VARCHAR(255)
);
CREATE TABLE edi_guide (
compiled_checksum VARCHAR(255),
create_date TIMESTAMP,
description VARCHAR(10000),
functional_id_code VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
parent_guide_id INTEGER,
partner_id INTEGER,
revision INTEGER,
source VARCHAR(255),
standard VARCHAR(255),
status VARCHAR(255),
transaction_set VARCHAR(255),
version VARCHAR(255),
UNIQUE (name, revision)
);
CREATE TABLE edi_guide_element (
code_list_id INTEGER,
component_position INTEGER,
create_date TIMESTAMP,
data_type VARCHAR(255),
element_id VARCHAR(255),
fixed_value VARCHAR(255),
guide_segment_id INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
max_length INTEGER,
min_length INTEGER,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
position INTEGER,
repeat_max INTEGER,
situational_rule_id INTEGER,
usage VARCHAR(255)
);
CREATE TABLE edi_guide_rule (
ak3_code VARCHAR(255),
ak4_code VARCHAR(255),
create_date TIMESTAMP,
expression VARCHAR(10000),
guide_id INTEGER,
guide_segment_id INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
ik3_code VARCHAR(255),
ik4_code VARCHAR(255),
message VARCHAR(10000),
modified_by VARCHAR(255),
modify_date TIMESTAMP,
positions VARCHAR(255),
severity VARCHAR(255),
snip_level INTEGER,
type VARCHAR(255)
);
CREATE TABLE edi_guide_segment (
create_date TIMESTAMP,
guide_id INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
loop_id VARCHAR(255),
loop_max_repeat INTEGER,
max_use INTEGER,
min_use INTEGER,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
notes VARCHAR(10000),
parent_loop_id VARCHAR(255),
position VARCHAR(255),
segment_id VARCHAR(255),
sequence INTEGER,
situational_rule_id INTEGER,
usage VARCHAR(255)
);
CREATE TABLE edi_identity (
application_code VARCHAR(255),
create_date TIMESTAMP,
description VARCHAR(10000),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
interchange_id VARCHAR(255),
interchange_qualifier VARCHAR(255),
is_active BOOLEAN,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
partner_id INTEGER,
standard VARCHAR(255),
UNIQUE (standard, interchange_qualifier, interchange_id, application_code)
);
CREATE TABLE edi_inbox (
attempts INTEGER,
claimed_at TIMESTAMP,
create_date TIMESTAMP,
external_id VARCHAR(255),
file_name VARCHAR(255),
gateway_user_id VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
last_error VARCHAR(255),
modify_date TIMESTAMP,
next_attempt_at TIMESTAMP,
partner_id INTEGER,
payload_ref VARCHAR(255),
payload_sha256 VARCHAR(255),
processed_at TIMESTAMP,
received_at TIMESTAMP,
remote_address VARCHAR(255),
reprocess BOOLEAN,
size_bytes BIGINT,
status VARCHAR(255),
transport_id INTEGER,
transport_type VARCHAR(255),
use_current_revisions BOOLEAN,
username VARCHAR(255),
UNIQUE (transport_type, external_id)
);
CREATE TABLE edi_interchange (
agreement_id INTEGER,
contains_phi BOOLEAN,
control_number VARCHAR(255),
create_date TIMESTAMP,
direction VARCHAR(255),
group_count INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbound_position INTEGER,
inbox_id INTEGER,
interchange_date TIMESTAMP,
interchange_version VARCHAR(255),
modify_date TIMESTAMP,
original_interchange_id INTEGER,
outbound_message_id INTEGER,
partner_id INTEGER,
payload_ref VARCHAR(255),
payload_sha256 VARCHAR(255),
receiver_id VARCHAR(255),
receiver_qualifier VARCHAR(255),
sender_id VARCHAR(255),
sender_qualifier VARCHAR(255),
size_bytes BIGINT,
standard VARCHAR(255),
status VARCHAR(255),
ta1_status VARCHAR(255),
transport_id INTEGER,
usage_indicator VARCHAR(255)
);
CREATE TABLE edi_map (
business_key_path VARCHAR(255),
business_key_type VARCHAR(255),
create_date TIMESTAMP,
description VARCHAR(10000),
direction VARCHAR(255),
guide_id INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
match_key_fields VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
on_match VARCHAR(255),
parent_map_id INTEGER,
partner_id INTEGER,
revision INTEGER,
secondary_key_path VARCHAR(255),
standard VARCHAR(255),
status VARCHAR(255),
table_name VARCHAR(255),
transaction_set VARCHAR(255),
version VARCHAR(255)
);
CREATE TABLE edi_map_rule (
condition VARCHAR(10000),
create_date TIMESTAMP,
default_value VARCHAR(255),
edi_path VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
map_id INTEGER,
modify_date TIMESTAMP,
notes VARCHAR(10000),
qqq_path VARCHAR(255),
required BOOLEAN,
sequence INTEGER,
transform VARCHAR(255),
transform_args VARCHAR(10000)
);
CREATE TABLE edi_map_sample (
create_date TIMESTAMP,
expected_result_json VARCHAR(10000),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
last_preview_at TIMESTAMP,
last_preview_status VARCHAR(255),
map_id INTEGER,
modify_date TIMESTAMP,
name VARCHAR(255),
payload_ref VARCHAR(255)
);
CREATE TABLE edi_outbound_message (
ack_for_group_id INTEGER,
ack_for_interchange_id INTEGER,
agreement_id INTEGER,
attempts INTEGER,
claimed_at TIMESTAMP,
create_date TIMESTAMP,
delivered_at TIMESTAMP,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
interchange_id INTEGER,
last_error VARCHAR(255),
modify_date TIMESTAMP,
next_attempt_at TIMESTAMP,
outbound_request_id INTEGER,
partner_id INTEGER,
payload_ref VARCHAR(255),
payload_sha256 VARCHAR(255),
sent_at TIMESTAMP,
size_bytes BIGINT,
status VARCHAR(255),
transport_id INTEGER,
transport_message_id VARCHAR(255),
transport_mic VARCHAR(255)
);
CREATE TABLE edi_outbound_request (
agreement_document_id INTEGER,
attempts INTEGER,
claimed_at TIMESTAMP,
create_date TIMESTAMP,
event_id VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
last_error VARCHAR(255),
modify_date TIMESTAMP,
next_attempt_at TIMESTAMP,
source_record_id VARCHAR(255),
source_table_name VARCHAR(255),
status VARCHAR(255),
UNIQUE (agreement_document_id, event_id)
);
CREATE TABLE edi_partner (
code VARCHAR(255),
contact_email VARCHAR(255),
contact_name VARCHAR(255),
contact_phone VARCHAR(255),
contains_phi BOOLEAN,
create_date TIMESTAMP,
default_timezone VARCHAR(255),
gateway_username VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
notes VARCHAR(10000),
status VARCHAR(255),
UNIQUE (name),
UNIQUE (code)
);
CREATE TABLE edi_payload (
ciphertext BYTEA,
create_date TIMESTAMP,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
key_id VARCHAR(255),
kind VARCHAR(255),
modify_date TIMESTAMP,
sha256 VARCHAR(255),
size_bytes BIGINT
);
CREATE TABLE edi_transaction (
ack_status VARCHAR(255),
agreement_document_id INTEGER,
business_key VARCHAR(255),
business_key_type VARCHAR(255),
contains_phi BOOLEAN,
control_number VARCHAR(255),
create_date TIMESTAMP,
direction VARCHAR(255),
error_count INTEGER,
functional_group_id INTEGER,
guide_id INTEGER,
guide_revision INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
inbound_position INTEGER,
interchange_id INTEGER,
map_id INTEGER,
map_revision INTEGER,
modify_date TIMESTAMP,
partner_id INTEGER,
phi_business_key VARCHAR(255),
secondary_key VARCHAR(255),
segment_count INTEGER,
status VARCHAR(255),
target_record_id VARCHAR(255),
target_table_name VARCHAR(255),
transaction_set VARCHAR(255),
version VARCHAR(255),
warning_count INTEGER
);
CREATE TABLE edi_transaction_error (
ack_code VARCHAR(255),
component_position INTEGER,
create_date TIMESTAMP,
element_position INTEGER,
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
loop_id VARCHAR(255),
message VARCHAR(10000),
modify_date TIMESTAMP,
segment_id VARCHAR(255),
segment_position INTEGER,
severity VARCHAR(255),
snip_level INTEGER,
transaction_id INTEGER,
value_excerpt VARCHAR(255)
);
CREATE TABLE edi_transport (
as2_local_id VARCHAR(255),
as2_partner_id VARCHAR(255),
as2_subject VARCHAR(255),
as2_url VARCHAR(255),
compress BOOLEAN,
create_date TIMESTAMP,
direction VARCHAR(255),
email_attachment_name VARCHAR(255),
email_from_alias VARCHAR(255),
email_provider_name VARCHAR(255),
email_to VARCHAR(255),
encrypt_algorithm VARCHAR(255),
gateway_endpoint_name VARCHAR(255),
http_auth_password_env_var VARCHAR(255),
http_auth_user_env_var VARCHAR(255),
https_auth_type VARCHAR(255),
https_headers_json VARCHAR(10000),
https_inbound_auth VARCHAR(255),
https_secret_env_var VARCHAR(255),
https_url VARCHAR(255),
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
is_active BOOLEAN,
is_van BOOLEAN,
local_certificate_id INTEGER,
max_attempts INTEGER,
mdn_mode VARCHAR(255),
mdn_signed BOOLEAN,
modified_by VARCHAR(255),
modify_date TIMESTAMP,
name VARCHAR(255),
notes VARCHAR(10000),
partner_certificate_id INTEGER,
partner_id INTEGER,
partner_tls_certificate_id INTEGER,
sftp_after_pickup VARCHAR(255),
sftp_archive_path VARCHAR(255),
sftp_file_pattern VARCHAR(255),
sftp_host VARCHAR(255),
sftp_host_key_fingerprint VARCHAR(255),
sftp_inbound_path VARCHAR(255),
sftp_outbound_path VARCHAR(255),
sftp_password_env_var VARCHAR(255),
sftp_poll_cron VARCHAR(255),
sftp_port INTEGER,
sftp_private_key_env_var VARCHAR(255),
sftp_temp_suffix VARCHAR(255),
sftp_username VARCHAR(255),
sign_algorithm VARCHAR(255),
type VARCHAR(255),
van_name VARCHAR(255),
weak_crypto BOOLEAN,
UNIQUE (name)
);
Vendored Executable
+295
View File
@@ -0,0 +1,295 @@
#!/bin/sh
# ----------------------------------------------------------------------------
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# ----------------------------------------------------------------------------
# ----------------------------------------------------------------------------
# Apache Maven Wrapper startup batch script, version 3.3.4
#
# Optional ENV vars
# -----------------
# JAVA_HOME - location of a JDK home dir, required when download maven via java source
# MVNW_REPOURL - repo url base for downloading maven distribution
# MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
# MVNW_VERBOSE - true: enable verbose log; debug: trace the mvnw script; others: silence the output
# ----------------------------------------------------------------------------
set -euf
[ "${MVNW_VERBOSE-}" != debug ] || set -x
# OS specific support.
native_path() { printf %s\\n "$1"; }
case "$(uname)" in
CYGWIN* | MINGW*)
[ -z "${JAVA_HOME-}" ] || JAVA_HOME="$(cygpath --unix "$JAVA_HOME")"
native_path() { cygpath --path --windows "$1"; }
;;
esac
# set JAVACMD and JAVACCMD
set_java_home() {
# For Cygwin and MinGW, ensure paths are in Unix format before anything is touched
if [ -n "${JAVA_HOME-}" ]; then
if [ -x "$JAVA_HOME/jre/sh/java" ]; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
JAVACCMD="$JAVA_HOME/jre/sh/javac"
else
JAVACMD="$JAVA_HOME/bin/java"
JAVACCMD="$JAVA_HOME/bin/javac"
if [ ! -x "$JAVACMD" ] || [ ! -x "$JAVACCMD" ]; then
echo "The JAVA_HOME environment variable is not defined correctly, so mvnw cannot run." >&2
echo "JAVA_HOME is set to \"$JAVA_HOME\", but \"\$JAVA_HOME/bin/java\" or \"\$JAVA_HOME/bin/javac\" does not exist." >&2
return 1
fi
fi
else
JAVACMD="$(
'set' +e
'unset' -f command 2>/dev/null
'command' -v java
)" || :
JAVACCMD="$(
'set' +e
'unset' -f command 2>/dev/null
'command' -v javac
)" || :
if [ ! -x "${JAVACMD-}" ] || [ ! -x "${JAVACCMD-}" ]; then
echo "The java/javac command does not exist in PATH nor is JAVA_HOME set, so mvnw cannot run." >&2
return 1
fi
fi
}
# hash string like Java String::hashCode
hash_string() {
str="${1:-}" h=0
while [ -n "$str" ]; do
char="${str%"${str#?}"}"
h=$(((h * 31 + $(LC_CTYPE=C printf %d "'$char")) % 4294967296))
str="${str#?}"
done
printf %x\\n $h
}
verbose() { :; }
[ "${MVNW_VERBOSE-}" != true ] || verbose() { printf %s\\n "${1-}"; }
die() {
printf %s\\n "$1" >&2
exit 1
}
trim() {
# MWRAPPER-139:
# Trims trailing and leading whitespace, carriage returns, tabs, and linefeeds.
# Needed for removing poorly interpreted newline sequences when running in more
# exotic environments such as mingw bash on Windows.
printf "%s" "${1}" | tr -d '[:space:]'
}
scriptDir="$(dirname "$0")"
scriptName="$(basename "$0")"
# parse distributionUrl and optional distributionSha256Sum, requires .mvn/wrapper/maven-wrapper.properties
while IFS="=" read -r key value; do
case "${key-}" in
distributionUrl) distributionUrl=$(trim "${value-}") ;;
distributionSha256Sum) distributionSha256Sum=$(trim "${value-}") ;;
esac
done <"$scriptDir/.mvn/wrapper/maven-wrapper.properties"
[ -n "${distributionUrl-}" ] || die "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties"
case "${distributionUrl##*/}" in
maven-mvnd-*bin.*)
MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/
case "${PROCESSOR_ARCHITECTURE-}${PROCESSOR_ARCHITEW6432-}:$(uname -a)" in
*AMD64:CYGWIN* | *AMD64:MINGW*) distributionPlatform=windows-amd64 ;;
:Darwin*x86_64) distributionPlatform=darwin-amd64 ;;
:Darwin*arm64) distributionPlatform=darwin-aarch64 ;;
:Linux*x86_64*) distributionPlatform=linux-amd64 ;;
*)
echo "Cannot detect native platform for mvnd on $(uname)-$(uname -m), use pure java version" >&2
distributionPlatform=linux-amd64
;;
esac
distributionUrl="${distributionUrl%-bin.*}-$distributionPlatform.zip"
;;
maven-mvnd-*) MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ ;;
*) MVN_CMD="mvn${scriptName#mvnw}" _MVNW_REPO_PATTERN=/org/apache/maven/ ;;
esac
# apply MVNW_REPOURL and calculate MAVEN_HOME
# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-<version>,maven-mvnd-<version>-<platform>}/<hash>
[ -z "${MVNW_REPOURL-}" ] || distributionUrl="$MVNW_REPOURL$_MVNW_REPO_PATTERN${distributionUrl#*"$_MVNW_REPO_PATTERN"}"
distributionUrlName="${distributionUrl##*/}"
distributionUrlNameMain="${distributionUrlName%.*}"
distributionUrlNameMain="${distributionUrlNameMain%-bin}"
MAVEN_USER_HOME="${MAVEN_USER_HOME:-${HOME}/.m2}"
MAVEN_HOME="${MAVEN_USER_HOME}/wrapper/dists/${distributionUrlNameMain-}/$(hash_string "$distributionUrl")"
exec_maven() {
unset MVNW_VERBOSE MVNW_USERNAME MVNW_PASSWORD MVNW_REPOURL || :
exec "$MAVEN_HOME/bin/$MVN_CMD" "$@" || die "cannot exec $MAVEN_HOME/bin/$MVN_CMD"
}
if [ -d "$MAVEN_HOME" ]; then
verbose "found existing MAVEN_HOME at $MAVEN_HOME"
exec_maven "$@"
fi
case "${distributionUrl-}" in
*?-bin.zip | *?maven-mvnd-?*-?*.zip) ;;
*) die "distributionUrl is not valid, must match *-bin.zip or maven-mvnd-*.zip, but found '${distributionUrl-}'" ;;
esac
# prepare tmp dir
if TMP_DOWNLOAD_DIR="$(mktemp -d)" && [ -d "$TMP_DOWNLOAD_DIR" ]; then
clean() { rm -rf -- "$TMP_DOWNLOAD_DIR"; }
trap clean HUP INT TERM EXIT
else
die "cannot create temp dir"
fi
mkdir -p -- "${MAVEN_HOME%/*}"
# Download and Install Apache Maven
verbose "Couldn't find MAVEN_HOME, downloading and installing it ..."
verbose "Downloading from: $distributionUrl"
verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName"
# select .zip or .tar.gz
if ! command -v unzip >/dev/null; then
distributionUrl="${distributionUrl%.zip}.tar.gz"
distributionUrlName="${distributionUrl##*/}"
fi
# verbose opt
__MVNW_QUIET_WGET=--quiet __MVNW_QUIET_CURL=--silent __MVNW_QUIET_UNZIP=-q __MVNW_QUIET_TAR=''
[ "${MVNW_VERBOSE-}" != true ] || __MVNW_QUIET_WGET='' __MVNW_QUIET_CURL='' __MVNW_QUIET_UNZIP='' __MVNW_QUIET_TAR=v
# normalize http auth
case "${MVNW_PASSWORD:+has-password}" in
'') MVNW_USERNAME='' MVNW_PASSWORD='' ;;
has-password) [ -n "${MVNW_USERNAME-}" ] || MVNW_USERNAME='' MVNW_PASSWORD='' ;;
esac
if [ -z "${MVNW_USERNAME-}" ] && command -v wget >/dev/null; then
verbose "Found wget ... using wget"
wget ${__MVNW_QUIET_WGET:+"$__MVNW_QUIET_WGET"} "$distributionUrl" -O "$TMP_DOWNLOAD_DIR/$distributionUrlName" || die "wget: Failed to fetch $distributionUrl"
elif [ -z "${MVNW_USERNAME-}" ] && command -v curl >/dev/null; then
verbose "Found curl ... using curl"
curl ${__MVNW_QUIET_CURL:+"$__MVNW_QUIET_CURL"} -f -L -o "$TMP_DOWNLOAD_DIR/$distributionUrlName" "$distributionUrl" || die "curl: Failed to fetch $distributionUrl"
elif set_java_home; then
verbose "Falling back to use Java to download"
javaSource="$TMP_DOWNLOAD_DIR/Downloader.java"
targetZip="$TMP_DOWNLOAD_DIR/$distributionUrlName"
cat >"$javaSource" <<-END
public class Downloader extends java.net.Authenticator
{
protected java.net.PasswordAuthentication getPasswordAuthentication()
{
return new java.net.PasswordAuthentication( System.getenv( "MVNW_USERNAME" ), System.getenv( "MVNW_PASSWORD" ).toCharArray() );
}
public static void main( String[] args ) throws Exception
{
setDefault( new Downloader() );
java.nio.file.Files.copy( java.net.URI.create( args[0] ).toURL().openStream(), java.nio.file.Paths.get( args[1] ).toAbsolutePath().normalize() );
}
}
END
# For Cygwin/MinGW, switch paths to Windows format before running javac and java
verbose " - Compiling Downloader.java ..."
"$(native_path "$JAVACCMD")" "$(native_path "$javaSource")" || die "Failed to compile Downloader.java"
verbose " - Running Downloader.java ..."
"$(native_path "$JAVACMD")" -cp "$(native_path "$TMP_DOWNLOAD_DIR")" Downloader "$distributionUrl" "$(native_path "$targetZip")"
fi
# If specified, validate the SHA-256 sum of the Maven distribution zip file
if [ -n "${distributionSha256Sum-}" ]; then
distributionSha256Result=false
if [ "$MVN_CMD" = mvnd.sh ]; then
echo "Checksum validation is not supported for maven-mvnd." >&2
echo "Please disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2
exit 1
elif command -v sha256sum >/dev/null; then
if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | sha256sum -c - >/dev/null 2>&1; then
distributionSha256Result=true
fi
elif command -v shasum >/dev/null; then
if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | shasum -a 256 -c >/dev/null 2>&1; then
distributionSha256Result=true
fi
else
echo "Checksum validation was requested but neither 'sha256sum' or 'shasum' are available." >&2
echo "Please install either command, or disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2
exit 1
fi
if [ $distributionSha256Result = false ]; then
echo "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised." >&2
echo "If you updated your Maven version, you need to update the specified distributionSha256Sum property." >&2
exit 1
fi
fi
# unzip and move
if command -v unzip >/dev/null; then
unzip ${__MVNW_QUIET_UNZIP:+"$__MVNW_QUIET_UNZIP"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -d "$TMP_DOWNLOAD_DIR" || die "failed to unzip"
else
tar xzf${__MVNW_QUIET_TAR:+"$__MVNW_QUIET_TAR"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -C "$TMP_DOWNLOAD_DIR" || die "failed to untar"
fi
# Find the actual extracted directory name (handles snapshots where filename != directory name)
actualDistributionDir=""
# First try the expected directory name (for regular distributions)
if [ -d "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" ]; then
if [ -f "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain/bin/$MVN_CMD" ]; then
actualDistributionDir="$distributionUrlNameMain"
fi
fi
# If not found, search for any directory with the Maven executable (for snapshots)
if [ -z "$actualDistributionDir" ]; then
# enable globbing to iterate over items
set +f
for dir in "$TMP_DOWNLOAD_DIR"/*; do
if [ -d "$dir" ]; then
if [ -f "$dir/bin/$MVN_CMD" ]; then
actualDistributionDir="$(basename "$dir")"
break
fi
fi
done
set -f
fi
if [ -z "$actualDistributionDir" ]; then
verbose "Contents of $TMP_DOWNLOAD_DIR:"
verbose "$(ls -la "$TMP_DOWNLOAD_DIR")"
die "Could not find Maven distribution directory in extracted archive"
fi
verbose "Found extracted Maven distribution directory: $actualDistributionDir"
printf %s\\n "$distributionUrl" >"$TMP_DOWNLOAD_DIR/$actualDistributionDir/mvnw.url"
mv -- "$TMP_DOWNLOAD_DIR/$actualDistributionDir" "$MAVEN_HOME" || [ -d "$MAVEN_HOME" ] || die "fail to move MAVEN_HOME"
clean || :
exec_maven "$@"
Vendored
+189
View File
@@ -0,0 +1,189 @@
<# : batch portion
@REM ----------------------------------------------------------------------------
@REM Licensed to the Apache Software Foundation (ASF) under one
@REM or more contributor license agreements. See the NOTICE file
@REM distributed with this work for additional information
@REM regarding copyright ownership. The ASF licenses this file
@REM to you under the Apache License, Version 2.0 (the
@REM "License"); you may not use this file except in compliance
@REM with the License. You may obtain a copy of the License at
@REM
@REM http://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing,
@REM software distributed under the License is distributed on an
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@REM KIND, either express or implied. See the License for the
@REM specific language governing permissions and limitations
@REM under the License.
@REM ----------------------------------------------------------------------------
@REM ----------------------------------------------------------------------------
@REM Apache Maven Wrapper startup batch script, version 3.3.4
@REM
@REM Optional ENV vars
@REM MVNW_REPOURL - repo url base for downloading maven distribution
@REM MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
@REM MVNW_VERBOSE - true: enable verbose log; others: silence the output
@REM ----------------------------------------------------------------------------
@IF "%__MVNW_ARG0_NAME__%"=="" (SET __MVNW_ARG0_NAME__=%~nx0)
@SET __MVNW_CMD__=
@SET __MVNW_ERROR__=
@SET __MVNW_PSMODULEP_SAVE=%PSModulePath%
@SET PSModulePath=
@FOR /F "usebackq tokens=1* delims==" %%A IN (`powershell -noprofile "& {$scriptDir='%~dp0'; $script='%__MVNW_ARG0_NAME__%'; icm -ScriptBlock ([Scriptblock]::Create((Get-Content -Raw '%~f0'))) -NoNewScope}"`) DO @(
IF "%%A"=="MVN_CMD" (set __MVNW_CMD__=%%B) ELSE IF "%%B"=="" (echo %%A) ELSE (echo %%A=%%B)
)
@SET PSModulePath=%__MVNW_PSMODULEP_SAVE%
@SET __MVNW_PSMODULEP_SAVE=
@SET __MVNW_ARG0_NAME__=
@SET MVNW_USERNAME=
@SET MVNW_PASSWORD=
@IF NOT "%__MVNW_CMD__%"=="" ("%__MVNW_CMD__%" %*)
@echo Cannot start maven from wrapper >&2 && exit /b 1
@GOTO :EOF
: end batch / begin powershell #>
$ErrorActionPreference = "Stop"
if ($env:MVNW_VERBOSE -eq "true") {
$VerbosePreference = "Continue"
}
# calculate distributionUrl, requires .mvn/wrapper/maven-wrapper.properties
$distributionUrl = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionUrl
if (!$distributionUrl) {
Write-Error "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties"
}
switch -wildcard -casesensitive ( $($distributionUrl -replace '^.*/','') ) {
"maven-mvnd-*" {
$USE_MVND = $true
$distributionUrl = $distributionUrl -replace '-bin\.[^.]*$',"-windows-amd64.zip"
$MVN_CMD = "mvnd.cmd"
break
}
default {
$USE_MVND = $false
$MVN_CMD = $script -replace '^mvnw','mvn'
break
}
}
# apply MVNW_REPOURL and calculate MAVEN_HOME
# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-<version>,maven-mvnd-<version>-<platform>}/<hash>
if ($env:MVNW_REPOURL) {
$MVNW_REPO_PATTERN = if ($USE_MVND -eq $False) { "/org/apache/maven/" } else { "/maven/mvnd/" }
$distributionUrl = "$env:MVNW_REPOURL$MVNW_REPO_PATTERN$($distributionUrl -replace "^.*$MVNW_REPO_PATTERN",'')"
}
$distributionUrlName = $distributionUrl -replace '^.*/',''
$distributionUrlNameMain = $distributionUrlName -replace '\.[^.]*$','' -replace '-bin$',''
$MAVEN_M2_PATH = "$HOME/.m2"
if ($env:MAVEN_USER_HOME) {
$MAVEN_M2_PATH = "$env:MAVEN_USER_HOME"
}
if (-not (Test-Path -Path $MAVEN_M2_PATH)) {
New-Item -Path $MAVEN_M2_PATH -ItemType Directory | Out-Null
}
$MAVEN_WRAPPER_DISTS = $null
if ((Get-Item $MAVEN_M2_PATH).Target[0] -eq $null) {
$MAVEN_WRAPPER_DISTS = "$MAVEN_M2_PATH/wrapper/dists"
} else {
$MAVEN_WRAPPER_DISTS = (Get-Item $MAVEN_M2_PATH).Target[0] + "/wrapper/dists"
}
$MAVEN_HOME_PARENT = "$MAVEN_WRAPPER_DISTS/$distributionUrlNameMain"
$MAVEN_HOME_NAME = ([System.Security.Cryptography.SHA256]::Create().ComputeHash([byte[]][char[]]$distributionUrl) | ForEach-Object {$_.ToString("x2")}) -join ''
$MAVEN_HOME = "$MAVEN_HOME_PARENT/$MAVEN_HOME_NAME"
if (Test-Path -Path "$MAVEN_HOME" -PathType Container) {
Write-Verbose "found existing MAVEN_HOME at $MAVEN_HOME"
Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD"
exit $?
}
if (! $distributionUrlNameMain -or ($distributionUrlName -eq $distributionUrlNameMain)) {
Write-Error "distributionUrl is not valid, must end with *-bin.zip, but found $distributionUrl"
}
# prepare tmp dir
$TMP_DOWNLOAD_DIR_HOLDER = New-TemporaryFile
$TMP_DOWNLOAD_DIR = New-Item -Itemtype Directory -Path "$TMP_DOWNLOAD_DIR_HOLDER.dir"
$TMP_DOWNLOAD_DIR_HOLDER.Delete() | Out-Null
trap {
if ($TMP_DOWNLOAD_DIR.Exists) {
try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null }
catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" }
}
}
New-Item -Itemtype Directory -Path "$MAVEN_HOME_PARENT" -Force | Out-Null
# Download and Install Apache Maven
Write-Verbose "Couldn't find MAVEN_HOME, downloading and installing it ..."
Write-Verbose "Downloading from: $distributionUrl"
Write-Verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName"
$webclient = New-Object System.Net.WebClient
if ($env:MVNW_USERNAME -and $env:MVNW_PASSWORD) {
$webclient.Credentials = New-Object System.Net.NetworkCredential($env:MVNW_USERNAME, $env:MVNW_PASSWORD)
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$webclient.DownloadFile($distributionUrl, "$TMP_DOWNLOAD_DIR/$distributionUrlName") | Out-Null
# If specified, validate the SHA-256 sum of the Maven distribution zip file
$distributionSha256Sum = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionSha256Sum
if ($distributionSha256Sum) {
if ($USE_MVND) {
Write-Error "Checksum validation is not supported for maven-mvnd. `nPlease disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties."
}
Import-Module $PSHOME\Modules\Microsoft.PowerShell.Utility -Function Get-FileHash
if ((Get-FileHash "$TMP_DOWNLOAD_DIR/$distributionUrlName" -Algorithm SHA256).Hash.ToLower() -ne $distributionSha256Sum) {
Write-Error "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised. If you updated your Maven version, you need to update the specified distributionSha256Sum property."
}
}
# unzip and move
Expand-Archive "$TMP_DOWNLOAD_DIR/$distributionUrlName" -DestinationPath "$TMP_DOWNLOAD_DIR" | Out-Null
# Find the actual extracted directory name (handles snapshots where filename != directory name)
$actualDistributionDir = ""
# First try the expected directory name (for regular distributions)
$expectedPath = Join-Path "$TMP_DOWNLOAD_DIR" "$distributionUrlNameMain"
$expectedMvnPath = Join-Path "$expectedPath" "bin/$MVN_CMD"
if ((Test-Path -Path $expectedPath -PathType Container) -and (Test-Path -Path $expectedMvnPath -PathType Leaf)) {
$actualDistributionDir = $distributionUrlNameMain
}
# If not found, search for any directory with the Maven executable (for snapshots)
if (!$actualDistributionDir) {
Get-ChildItem -Path "$TMP_DOWNLOAD_DIR" -Directory | ForEach-Object {
$testPath = Join-Path $_.FullName "bin/$MVN_CMD"
if (Test-Path -Path $testPath -PathType Leaf) {
$actualDistributionDir = $_.Name
}
}
}
if (!$actualDistributionDir) {
Write-Error "Could not find Maven distribution directory in extracted archive"
}
Write-Verbose "Found extracted Maven distribution directory: $actualDistributionDir"
Rename-Item -Path "$TMP_DOWNLOAD_DIR/$actualDistributionDir" -NewName $MAVEN_HOME_NAME | Out-Null
try {
Move-Item -Path "$TMP_DOWNLOAD_DIR/$MAVEN_HOME_NAME" -Destination $MAVEN_HOME_PARENT | Out-Null
} catch {
if (! (Test-Path -Path "$MAVEN_HOME" -PathType Container)) {
Write-Error "fail to move MAVEN_HOME"
}
} finally {
try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null }
catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" }
}
Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD"
+205 -37
View File
@@ -1,37 +1,120 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<!--
~ Copyright 2026 Kingsrook, LLC
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-example-app</artifactId>
<version>0.1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>QBit Example Application</name>
<description>Template for Application QBits - complete mini-applications with UI navigation</description>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<revision>0.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<qqq.version>4.0.0</qqq.version>
<jacoco.version>0.8.15</jacoco.version>
</properties>
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-edi</artifactId>
<version>${revision}</version>
<packaging>jar</packaging>
<name>QBit EDI</name>
<description>QQQ QBit: EDI (X12 and EDIFACT) trading partners, transports (AS2, SFTP, VAN, HTTPS, email), acknowledgements, data-driven implementation guides and maps</description>
<url>https://github.com/QRun-IO/qbit-edi</url>
<modelVersion>4.0.0</modelVersion>
<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
<developers>
<developer>
<id>kingsrook</id>
<name>Kingsrook Team</name>
<email>contact@kingsrook.com</email>
<organization>Kingsrook, LLC</organization>
<organizationUrl>https://kingsrook.com</organizationUrl>
</developer>
</developers>
<scm>
<connection>scm:git:git://github.com/QRun-IO/qbit-edi.git</connection>
<developerConnection>scm:git:ssh://github.com:QRun-IO/qbit-edi.git</developerConnection>
<url>https://github.com/QRun-IO/qbit-edi</url>
<tag>HEAD</tag>
</scm>
<dependencies>
<!-- QQQ Backend Core -->
<dependency><groupId>com.kingsrook.qqq</groupId><artifactId>qqq-frontend-next</artifactId><version>1.0.0-RC.14</version><scope>test</scope></dependency>
<dependency><groupId>org.bouncycastle</groupId><artifactId>bcpkix-jdk18on</artifactId><version>1.85</version></dependency>
<dependency><groupId>com.helger.as2</groupId><artifactId>as2-lib</artifactId><version>5.1.8</version></dependency>
<dependency><groupId>com.kingsrook.qqq</groupId><artifactId>qqq-backend-module-filesystem</artifactId></dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-esb</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-object-storage</artifactId>
<version>0.1.0-RC.2</version>
<optional>true</optional>
<exclusions>
<exclusion><groupId>com.kingsrook.qqq</groupId><artifactId>*</artifactId></exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.xlate</groupId>
<artifactId>staedi</artifactId>
<version>1.26.4</version>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>2.4.240</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-middleware-javalin</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-module-rdbms</artifactId>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-core</artifactId>
<version>${qqq.version}</version>
</dependency>
<!-- Testing -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.10.0</version>
<scope>test</scope>
</dependency>
<dependency>
@@ -40,51 +123,136 @@
<version>3.27.7</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-module-rdbms</artifactId>
<version>${qqq.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>2.2.224</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<resources>
<!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
<includes>
<include>help/*.yaml</include>
</includes>
</resource>
<resource>
<directory>src/main/resources</directory>
<filtering>false</filtering>
<excludes>
<exclude>help/*.yaml</exclude>
</excludes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.11.0</version>
</plugin>
<!-- the artifact version, for the help bundle version test -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.1.2</version>
<configuration>
<systemPropertyVariables combine.children="append">
<qbit.artifactVersion>${project.version}</qbit.artifactVersion>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.3.1</version>
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>10.12.6</version>
</dependency>
</dependencies>
<configuration>
<configLocation>checkstyle/config.xml</configLocation>
<headerLocation>checkstyle/license.txt</headerLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failsOnError>false</failsOnError>
<failOnViolation>true</failOnViolation>
<violationSeverity>warning</violationSeverity>
<includeTestSourceDirectory>false</includeTestSourceDirectory>
</configuration>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>test</phase>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<execution>
<id>check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>INSTRUCTION</counter>
<value>COVEREDRATIO</value>
<minimum>0.70</minimum>
</limit>
<limit>
<counter>CLASS</counter>
<value>COVEREDRATIO</value>
<minimum>0.90</minimum>
</limit>
</limits>
</rule>
</rules>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.7</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-bom-pom</artifactId>
<version>${qqq.snapshot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
</profile>
</profiles>
</project>
@@ -0,0 +1,393 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi;
import java.util.List;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
/*******************************************************************************
** Code configuration for the EDI qbit. Runtime settings (partners, agreements,
** transports, guides and maps) are data in the edi_ tables, not here.
*******************************************************************************/
public class EdiQBitConfig implements QBitConfig
{
public static final String TABLE_NAME_PREFIX = "edi_";
public static final String PERMISSION_BASE_NAME = "edi";
private final java.util.Set<String> sendAsEdiTables = new java.util.LinkedHashSet<>();
/*******************************************************************************
** Opt a source table into the manual Send As EDI action.
*******************************************************************************/
public EdiQBitConfig withSendAsEdiTable(String tableName)
{
sendAsEdiTables.add(tableName);
return (this);
}
/*******************************************************************************
** Explicitly opted-in manual source tables.
*******************************************************************************/
public java.util.Set<String> getSendAsEdiTables()
{
return (java.util.Set.copyOf(sendAsEdiTables));
}
private com.kingsrook.qbits.edi.guide.ExternalCodeSetProvider externalCodeSetProvider;
/*******************************************************************************
** Optional host provider for licensed external code sets.
*******************************************************************************/
@JsonIgnore
public com.kingsrook.qbits.edi.guide.ExternalCodeSetProvider getExternalCodeSetProvider()
{
return (externalCodeSetProvider);
}
/*******************************************************************************
** Register a provider; null retains format-only validation.
*******************************************************************************/
public EdiQBitConfig withExternalCodeSetProvider(com.kingsrook.qbits.edi.guide.ExternalCodeSetProvider value)
{
externalCodeSetProvider = value;
return (this);
}
private String payloadBucketAlias = "edi";
private java.util.function.Supplier<com.kingsrook.qqq.backend.core.model.session.QSession> serviceSessionSupplier = null;
private java.time.Clock clock = java.time.Clock.systemUTC();
private String esbDestinationName;
private String schedulerName;
private Integer maxSegmentsPerTransaction = 100000;
private String payloadStore = "DATABASE";
private com.kingsrook.qbits.edi.security.EdiSecrets secretsResolver = System::getenv;
private Boolean partnerLocks = false;
private String backendName;
private String appName = "edi";
private String appLabel = "EDI";
private String parentAppName;
private String publicBaseUrl;
/*******************************************************************************
** Validate the configuration, collecting all errors.
*******************************************************************************/
@Override
public void validate(QInstance instance, List<String> errors)
{
if(backendName == null || instance.getBackend(backendName) == null)
{
errors.add("backendName must name an existing backend");
}
if(maxSegmentsPerTransaction == null || maxSegmentsPerTransaction < 2)
{
errors.add("maxSegmentsPerTransaction must be at least 2");
}
if(!List.of("DATABASE", "OBJECT_STORAGE").contains(payloadStore))
{
errors.add("payloadStore must be DATABASE or OBJECT_STORAGE");
}
if(secretsResolver == null || partnerLocks == null)
{
errors.add("secretsResolver and partnerLocks are required");
}
if(appName == null || appName.isBlank())
{
errors.add("appName is required");
}
}
/*******************************************************************************
** The backend for the qbit's tables.
*******************************************************************************/
@Override
public String getDefaultBackendNameForTables()
{
return (backendName);
}
/*******************************************************************************
** Getter for backendName
*******************************************************************************/
public String getBackendName()
{
return (this.backendName);
}
/*******************************************************************************
** Fluent setter for backendName
*******************************************************************************/
public EdiQBitConfig withBackendName(String backendName)
{
this.backendName = backendName;
return (this);
}
/*******************************************************************************
** Getter for appName
*******************************************************************************/
public String getAppName()
{
return (this.appName);
}
/*******************************************************************************
** Fluent setter for appName
*******************************************************************************/
public EdiQBitConfig withAppName(String appName)
{
this.appName = appName;
return (this);
}
/*******************************************************************************
** Getter for appLabel
*******************************************************************************/
public String getAppLabel()
{
return (this.appLabel);
}
/*******************************************************************************
** Fluent setter for appLabel
*******************************************************************************/
public EdiQBitConfig withAppLabel(String appLabel)
{
this.appLabel = appLabel;
return (this);
}
/*******************************************************************************
** Getter for parentAppName
*******************************************************************************/
public String getParentAppName()
{
return (this.parentAppName);
}
/*******************************************************************************
** Fluent setter for parentAppName
*******************************************************************************/
public EdiQBitConfig withParentAppName(String parentAppName)
{
this.parentAppName = parentAppName;
return (this);
}
/*******************************************************************************
** Get schedulerName.
*******************************************************************************/
public String getSchedulerName()
{
return (schedulerName);
}
/*******************************************************************************
** Configure schedulerName.
*******************************************************************************/
public EdiQBitConfig withSchedulerName(String value)
{
schedulerName = value;
return (this);
}
/*******************************************************************************
** Get maxSegmentsPerTransaction.
*******************************************************************************/
public Integer getMaxSegmentsPerTransaction()
{
return (maxSegmentsPerTransaction);
}
/*******************************************************************************
** Configure maxSegmentsPerTransaction.
*******************************************************************************/
public EdiQBitConfig withMaxSegmentsPerTransaction(Integer value)
{
maxSegmentsPerTransaction = value;
return (this);
}
/*******************************************************************************
** Get payloadStore.
*******************************************************************************/
@JsonIgnore
public String getPayloadStore()
{
return (payloadStore);
}
/*******************************************************************************
** Configure payloadStore.
*******************************************************************************/
public EdiQBitConfig withPayloadStore(String value)
{
payloadStore = value;
return (this);
}
/*******************************************************************************
** Get secretsResolver.
*******************************************************************************/
@JsonIgnore
public com.kingsrook.qbits.edi.security.EdiSecrets getSecretsResolver()
{
return (secretsResolver);
}
/*******************************************************************************
** Configure secretsResolver.
*******************************************************************************/
public EdiQBitConfig withSecretsResolver(com.kingsrook.qbits.edi.security.EdiSecrets value)
{
secretsResolver = value;
return (this);
}
/*******************************************************************************
** Get partnerLocks.
*******************************************************************************/
public Boolean getPartnerLocks()
{
return (partnerLocks);
}
/*******************************************************************************
** Configure partnerLocks.
*******************************************************************************/
public EdiQBitConfig withPartnerLocks(Boolean value)
{
partnerLocks = value;
return (this);
}
/*******************************************************************************
** Get payloadBucketAlias.
*******************************************************************************/
public String getPayloadBucketAlias()
{
return (payloadBucketAlias);
}
/*******************************************************************************
** Configure payloadBucketAlias.
*******************************************************************************/
public EdiQBitConfig withPayloadBucketAlias(String value)
{
payloadBucketAlias = value;
return (this);
}
/*******************************************************************************
** Get serviceSessionSupplier.
*******************************************************************************/
@JsonIgnore
public java.util.function.Supplier<com.kingsrook.qqq.backend.core.model.session.QSession> getServiceSessionSupplier()
{
return (serviceSessionSupplier);
}
/*******************************************************************************
** Configure serviceSessionSupplier.
*******************************************************************************/
public EdiQBitConfig withServiceSessionSupplier(java.util.function.Supplier<com.kingsrook.qqq.backend.core.model.session.QSession> value)
{
serviceSessionSupplier = value;
return (this);
}
/*******************************************************************************
** Get clock.
*******************************************************************************/
@JsonIgnore
public java.time.Clock getClock()
{
return (clock);
}
/*******************************************************************************
** Configure clock.
*******************************************************************************/
public EdiQBitConfig withClock(java.time.Clock value)
{
clock = value;
return (this);
}
/*******************************************************************************
** Optional destination for core table events on main tables.
*******************************************************************************/
public String getEsbDestinationName()
{
return (esbDestinationName);
}
/*******************************************************************************
** Enable standard table publications to a host-configured destination.
*******************************************************************************/
public EdiQBitConfig withEsbDestinationName(String value)
{
esbDestinationName = value;
return (this);
}
/*******************************************************************************
** Public base URL of this deployment (for example https://edi.example.com),
** used to build the asynchronous AS2 receipt address
** {base}/qqq/hooks/edi/as2/{transport}/mdn.
*******************************************************************************/
public String getPublicBaseUrl()
{
return (publicBaseUrl);
}
/*******************************************************************************
** Set the public base URL.
*******************************************************************************/
public EdiQBitConfig withPublicBaseUrl(String value)
{
publicBaseUrl = value;
return (this);
}
}
@@ -0,0 +1,163 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
/*******************************************************************************
** Entry point for the EDI qbit; component producers are scanned only in the
** metadata package.
*******************************************************************************/
public class EdiQBitProducer implements QBitMetaDataProducer<EdiQBitConfig>
{
public static final String GROUP_ID = "com.kingsrook.qbits";
public static final String ARTIFACT_ID = "qbit-edi";
public static final String VERSION = "0.1.0";
private EdiQBitConfig qBitConfig;
private String namespace;
/*******************************************************************************
** Fluent configuration setter.
*******************************************************************************/
public EdiQBitProducer withQBitConfig(EdiQBitConfig config)
{
qBitConfig = config;
return (this);
}
/*******************************************************************************
** Configuration for component producers.
*******************************************************************************/
@Override
public EdiQBitConfig getQBitConfig()
{
return (qBitConfig);
}
/*******************************************************************************
** Fluent namespace setter.
*******************************************************************************/
public EdiQBitProducer withNamespace(String namespace)
{
this.namespace = namespace;
return (this);
}
/*******************************************************************************
** Qbit namespace.
*******************************************************************************/
@Override
public String getNamespace()
{
return (namespace);
}
/*******************************************************************************
** Qbit identity.
*******************************************************************************/
@Override
public QBitMetaData getQBitMetaData()
{
return (new QBitMetaData().withGroupId(GROUP_ID).withArtifactId(ARTIFACT_ID).withVersion(VERSION).withNamespace(namespace).withConfig(qBitConfig));
}
/*******************************************************************************
** Unconfigured producers discovered by host scanning are disabled.
*******************************************************************************/
@Override
public boolean isEnabled()
{
return (qBitConfig != null);
}
/*******************************************************************************
** Only the metadata package holds component producers.
*******************************************************************************/
@Override
public String getPackageNameForFindingMetaDataProducers()
{
return ("com.kingsrook.qbits.edi.metadata");
}
/*******************************************************************************
** Validate the configuration and install the qbit's metadata.
*******************************************************************************/
@Override
public MetaDataProducerMultiOutput produce(QInstance instance) throws QException
{
qBitConfig.validate(instance);
MetaDataProducerMultiOutput output = QBitMetaDataProducer.super.produce(instance);
output.addSelfToInstance(instance);
EdiRuntime.install(instance, qBitConfig);
// Registration is metadata-only; core startup sync skips hosts without helpContent.
HelpBundles.register(instance, "help/qbit-edi.yaml");
// The app name is configurable; its bundle key must follow the installed app.
HelpBundles.of(instance).getBundles().getLast().getEntries().stream()
.filter(entry -> "app:edi;slot:help".equals(entry.getKey()))
.forEach(entry -> entry.setKey("app:" + qBitConfig.getAppName() + ";slot:help"));
if(instance.getTable("helpContent") == null)
{
com.kingsrook.qbits.edi.metadata.EdiHelp.apply(instance, qBitConfig.getAppName());
}
instance.withSupplementalMetaData(new com.kingsrook.qbits.edi.metadata.EdiRecordLabels());
instance.withRecordChangeListener(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(com.kingsrook.qbits.edi.flow.OutboundEnqueue.class));
instance.withSupplementalMetaData(new com.kingsrook.qbits.edi.transport.EdiTransportPluginsMetaData()
.withPlugin(new com.kingsrook.qbits.edi.transport.gateway.GatewayTransport())
.withPlugin(new com.kingsrook.qbits.edi.transport.HttpsTransport()).withPlugin(new com.kingsrook.qbits.edi.transport.SftpTransport())
.withPlugin(new com.kingsrook.qbits.edi.as2.As2Transport()).withPlugin(new com.kingsrook.qbits.edi.transport.UnavailableTransport("EMAIL")));
try
{
com.kingsrook.qbits.edi.api.EdiApi.register(instance);
com.kingsrook.qqq.middleware.javalin.QJavalinMetaData.ofOrWithNew(instance).withAdditionalRouteProviderReference(
new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(com.kingsrook.qbits.edi.transport.HttpsRouteProvider.class))
.withAdditionalRouteProviderReference(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(com.kingsrook.qbits.edi.as2.As2RouteProvider.class));
}
catch(NoClassDefFoundError optionalMiddlewareAbsent)
{
// Machine hooks are installed only when optional Javalin is present.
}
return (new MetaDataProducerMultiOutput());
}
}
@@ -0,0 +1,215 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.WeakHashMap;
import com.kingsrook.qbits.edi.ack.Ack997Builder;
import com.kingsrook.qbits.edi.ack.Ack999Builder;
import com.kingsrook.qbits.edi.ack.InboundAckReader;
import com.kingsrook.qbits.edi.ack.TA1Builder;
import com.kingsrook.qbits.edi.guide.BuiltInGuideLoader;
import com.kingsrook.qbits.edi.guide.GuideValidator;
import com.kingsrook.qbits.edi.storage.DatabasePayloadStore;
import com.kingsrook.qbits.edi.storage.PayloadCrypto;
import com.kingsrook.qbits.edi.x12.ControlNumberAllocator;
import com.kingsrook.qbits.edi.x12.DuplicateDetector;
import com.kingsrook.qbits.edi.x12.X12EnvelopeReader;
import com.kingsrook.qbits.edi.x12.X12Writer;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
/*******************************************************************************
** Per-QInstance service holder; metadata installation does not access the database.
*******************************************************************************/
public final class EdiRuntime
{
private static final QLogger LOG = QLogger.getLogger(EdiRuntime.class);
private static final Map<QInstance, EdiRuntime> INSTANCES = Collections.synchronizedMap(new WeakHashMap<>());
private final X12EnvelopeReader reader;
private final X12Writer writer = new X12Writer();
private final GuideValidator validator;
private final BuiltInGuideLoader guides = new BuiltInGuideLoader();
private final com.kingsrook.qbits.edi.storage.PayloadStore payloads;
private final EdiQBitConfig config;
private final com.kingsrook.qbits.edi.map.TransformRegistry transforms = new com.kingsrook.qbits.edi.map.TransformRegistry();
private final DuplicateDetector duplicates = new DuplicateDetector();
private final ControlNumberAllocator controls;
private final Ack997Builder ack997 = new Ack997Builder();
private final Ack999Builder ack999 = new Ack999Builder();
private final TA1Builder ta1 = new TA1Builder();
private final InboundAckReader acknowledgements = new InboundAckReader();
/*******************************************************************************
** Construct services without resolving keys until content is stored or opened.
*******************************************************************************/
private EdiRuntime(QInstance instance, EdiQBitConfig config)
{
this.config = config;
validator = new GuideValidator(config);
reader = new X12EnvelopeReader(config.getMaxSegmentsPerTransaction());
payloads = "OBJECT_STORAGE".equals(config.getPayloadStore())
? new com.kingsrook.qbits.edi.storage.ObjectStoragePayloadStore(new PayloadCrypto(config.getSecretsResolver()), config.getPayloadBucketAlias(), config.getClock())
: new DatabasePayloadStore(new PayloadCrypto(config.getSecretsResolver()));
controls = instance.getBackend(config.getBackendName()) instanceof RDBMSBackendMetaData backend ? new ControlNumberAllocator(backend) : null;
}
/*******************************************************************************
** Install for a host instance.
*******************************************************************************/
public static void install(QInstance instance, EdiQBitConfig config)
{
INSTANCES.put(instance, new EdiRuntime(instance, config));
}
/*******************************************************************************
** Get services for the current QQQ context.
*******************************************************************************/
public static EdiRuntime current()
{
return (Objects.requireNonNull(INSTANCES.get(QContext.getQInstance()), "EDI runtime has not been installed"));
}
/*******************************************************************************
** Call at host startup after DDL, with the service session established.
*******************************************************************************/
public List<Integer> initialize() throws Exception
{
if("DATABASE".equals(config.getPayloadStore()))
{
LOG.warn("EDI database payload storage is enabled; object storage is recommended for production");
}
var ids = guides.load();
com.kingsrook.qbits.edi.processes.OutboundProcesses.installSchedules();
return (ids);
}
/*******************************************************************************
** Access reader.
*******************************************************************************/
public X12EnvelopeReader reader()
{
return (reader);
}
/*******************************************************************************
** Access writer.
*******************************************************************************/
public X12Writer writer()
{
return (writer);
}
/*******************************************************************************
** Access validator.
*******************************************************************************/
public GuideValidator validator()
{
return (validator);
}
/*******************************************************************************
** Access guides.
*******************************************************************************/
public BuiltInGuideLoader guides()
{
return (guides);
}
/*******************************************************************************
** Access payloads.
*******************************************************************************/
public com.kingsrook.qbits.edi.storage.PayloadStore payloads()
{
return (payloads);
}
/*******************************************************************************
** Access duplicates.
*******************************************************************************/
public DuplicateDetector duplicates()
{
return (duplicates);
}
/*******************************************************************************
** Access controls.
*******************************************************************************/
public ControlNumberAllocator controls()
{
return (Objects.requireNonNull(controls, "Control allocation requires an RDBMS backend"));
}
/*******************************************************************************
** Access ack997.
*******************************************************************************/
public Ack997Builder ack997()
{
return (ack997);
}
/*******************************************************************************
** Access ack999.
*******************************************************************************/
public Ack999Builder ack999()
{
return (ack999);
}
/*******************************************************************************
** Access ta1.
*******************************************************************************/
public TA1Builder ta1()
{
return (ta1);
}
/*******************************************************************************
** Access acknowledgements.
*******************************************************************************/
public InboundAckReader acknowledgements()
{
return (acknowledgements);
}
/*******************************************************************************
** Installed host configuration.
*******************************************************************************/
public EdiQBitConfig config()
{
return (config);
}
/*******************************************************************************
** Host-extensible inbound transformations.
*******************************************************************************/
public com.kingsrook.qbits.edi.map.TransformRegistry transforms()
{
return (transforms);
}
}
@@ -0,0 +1,43 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.guide.ValidationFinding;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
/*******************************************************************************
** Generate an X12 997 functional acknowledgement.
*******************************************************************************/
public class Ack997Builder
{
/*******************************************************************************
** Build transaction detail at the configured acknowledgement level.
*******************************************************************************/
public X12TransactionSet build(X12FunctionalGroup group, Map<String, List<ValidationFinding>> findings, AckLevel level, String controlNumber)
{
return (FunctionalAckBuilder.build(false, group, findings, level, controlNumber));
}
}
@@ -0,0 +1,43 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.guide.ValidationFinding;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
/*******************************************************************************
** Generate an X12 999 functional acknowledgement.
*******************************************************************************/
public class Ack999Builder
{
/*******************************************************************************
** Build transaction detail at the configured acknowledgement level.
*******************************************************************************/
public X12TransactionSet build(X12FunctionalGroup group, Map<String, List<ValidationFinding>> findings, AckLevel level, String controlNumber)
{
return (FunctionalAckBuilder.build(true, group, findings, level, controlNumber));
}
}
@@ -0,0 +1,60 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.List;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12Interchange;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
/*******************************************************************************
** Address acknowledgements to the original sender using newly allocated controls.
*******************************************************************************/
public final class AckEnvelope
{
/*******************************************************************************
** Swap interchange identities while preserving standard version and usage.
*******************************************************************************/
public static X12Segment replyIsa(X12Segment original, String control, LocalDateTime at)
{
return (X12Segment.of("ISA", 1, "00", "", "00", "", original.value(7), original.value(8), original.value(5), original.value(6),
at.format(DateTimeFormatter.ofPattern("yyMMdd")), at.format(DateTimeFormatter.ofPattern("HHmm")), original.value(11), original.value(12),
"0".repeat(Math.max(0, 9 - control.length())) + control, "0", original.value(15), original.value(16)));
}
/*******************************************************************************
** Wrap an acknowledgement transaction in a reply functional group and interchange.
*******************************************************************************/
public static X12Interchange wrap(X12Interchange received, X12FunctionalGroup group, X12TransactionSet acknowledgement, String isaControl, String gsControl, LocalDateTime at)
{
X12Segment isa = replyIsa(received.isa(), isaControl, at);
String version = acknowledgement.st().value(1).equals("999") ? "005010X231A1" : group.gs().value(8);
X12Segment gs = X12Segment.of("GS", 2, "FA", group.gs().value(3), group.gs().value(2), at.format(DateTimeFormatter.BASIC_ISO_DATE),
at.format(DateTimeFormatter.ofPattern("HHmm")), gsControl, "X", version);
X12FunctionalGroup reply = new X12FunctionalGroup(gs, X12Segment.of("GE", 0, "1", gsControl), List.of(acknowledgement), List.of());
return (new X12Interchange(received.separators(), isa, X12Segment.of("IEA", 0, "1", isa.value(13)), List.of(reply), List.of(), List.of()));
}
}
@@ -0,0 +1,32 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
/*******************************************************************************
** Transaction detail included in a functional acknowledgement.
*******************************************************************************/
public enum AckLevel
{
GROUP_ONLY, ALL_TRANSACTIONS, ERRORS_ONLY
}
@@ -0,0 +1,137 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.guide.ValidationFinding;
import com.kingsrook.qbits.edi.x12.EnvelopeFinding;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
/*******************************************************************************
** Shared 997/999 status, counts and detail generation.
*******************************************************************************/
final class FunctionalAckBuilder
{
/*******************************************************************************
** Build an acknowledgement transaction with its own ST control number.
*******************************************************************************/
static X12TransactionSet build(boolean implementation, X12FunctionalGroup group, Map<String, List<ValidationFinding>> findings, AckLevel level, String control)
{
List<X12Segment> segments = new ArrayList<>();
if(implementation)
{
add(segments, "ST", "999", control, "005010X231A1");
add(segments, "AK1", group.gs().value(1), group.gs().value(6), group.gs().value(8));
}
else
{
add(segments, "ST", "997", control);
add(segments, "AK1", group.gs().value(1), group.gs().value(6));
}
int accepted = 0;
boolean hasWarnings = false;
for(X12TransactionSet transaction : group.transactions())
{
List<ValidationFinding> errors = findings.getOrDefault(transaction.st().value(2), List.of());
boolean rejected = !transaction.findings().isEmpty() || errors.stream().anyMatch(f -> "ERROR".equals(f.severity()) && f.snipLevel() <= 2);
String status = rejected ? "R" : errors.isEmpty() ? "A" : "E";
if(!rejected)
{
accepted++;
}
hasWarnings |= status.equals("E");
if(level == AckLevel.GROUP_ONLY || level == AckLevel.ERRORS_ONLY && status.equals("A"))
{
continue;
}
if(implementation && !transaction.st().value(3).isEmpty())
{
add(segments, "AK2", transaction.st().value(1), transaction.st().value(2), transaction.st().value(3));
}
else
{
add(segments, "AK2", transaction.st().value(1), transaction.st().value(2));
}
for(ValidationFinding finding : errors)
{
add(segments, implementation ? "IK3" : "AK3", finding.segmentId(), String.valueOf(finding.segmentPosition()),
finding.loopId() == null ? "" : finding.loopId(), implementation ? finding.ik3Code() : finding.ak3Code());
if(finding.elementPosition() != null)
{
List<String> position = new ArrayList<>(List.of(finding.elementPosition().toString()));
if(finding.componentPosition() != null)
{
position.add(finding.componentPosition().toString());
}
segments.add(new X12Segment(implementation ? "IK4" : "AK4", segments.size() + 1,
List.of(List.of(position), List.of(List.of("")), List.of(List.of(implementation ? finding.ik4Code() : finding.ak4Code())))));
}
if(implementation && finding.context() != null)
{
var context = finding.context();
List<List<List<String>>> values = new ArrayList<>();
for(String value : List.of("SITUATIONAL TRIGGER", context.segmentId(), String.valueOf(context.segmentPosition()), context.loopId() == null ? "" : context.loopId()))
{
values.add(List.of(List.of(value)));
}
if(context.elementPosition() != null)
{
List<String> position = new ArrayList<>(List.of(context.elementPosition().toString()));
if(context.componentPosition() != null)
{
position.add(context.componentPosition().toString());
}
values.add(List.of(position));
}
segments.add(new X12Segment("CTX", segments.size() + 1, values));
}
}
List<String> statusElements = new ArrayList<>(List.of(status));
transaction.findings().stream().map(EnvelopeFinding::code).distinct().limit(5).forEach(statusElements::add);
if(rejected && statusElements.size() == 1)
{
statusElements.add("5");
}
add(segments, implementation ? "IK5" : "AK5", statusElements.toArray(String[]::new));
}
String status = !group.findings().isEmpty() || accepted == 0 && !group.transactions().isEmpty() ? "R"
: accepted < group.transactions().size() ? "P" : hasWarnings ? "E" : "A";
List<String> summary = new ArrayList<>(List.of(status, group.ge().value(1), String.valueOf(group.transactions().size()), String.valueOf(accepted)));
group.findings().stream().map(EnvelopeFinding::code).distinct().limit(5).forEach(summary::add);
add(segments, "AK9", summary.toArray(String[]::new));
add(segments, "SE", String.valueOf(segments.size() + 1), control);
return (new X12TransactionSet(segments.getFirst(), segments.getLast(), List.copyOf(segments), List.of()));
}
/*******************************************************************************
** Append a positioned simple segment.
*******************************************************************************/
private static void add(List<X12Segment> segments, String id, String... values)
{
segments.add(X12Segment.of(id, segments.size() + 1, values));
}
}
@@ -0,0 +1,107 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12Interchange;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
/*******************************************************************************
** Read acknowledgement controls and outcomes for later reconciliation.
*******************************************************************************/
public class InboundAckReader
{
/*******************************************************************************
** Parsed functional or interchange acknowledgement.
*******************************************************************************/
public record Result(String type, String controlNumber, String status, Map<String, String> transactionStatuses, String noteCode)
{
}
/*******************************************************************************
** Read all received acknowledgements in an interchange.
*******************************************************************************/
public List<Result> read(X12Interchange interchange)
{
List<Result> results = new ArrayList<>();
for(X12Segment segment : interchange.serviceSegments())
{
if(segment.id().equals("TA1"))
{
results.add(new Result("TA1", segment.value(1), segment.value(4), Map.of(), segment.value(5)));
}
}
for(X12FunctionalGroup group : interchange.groups())
{
for(X12TransactionSet transaction : group.transactions())
{
if(List.of("997", "999").contains(transaction.st().value(1)))
{
results.add(read(transaction));
}
}
}
return (results);
}
/*******************************************************************************
** Extract AK1, AK2, AK5/IK5 and AK9 without acknowledging the ack.
*******************************************************************************/
public Result read(X12TransactionSet transaction)
{
String control = null;
String status = null;
String stControl = null;
Map<String, String> statuses = new LinkedHashMap<>();
for(X12Segment segment : transaction.segments())
{
switch(segment.id())
{
case "AK1" -> control = segment.value(2);
case "AK2" -> stControl = segment.value(2);
case "AK5", "IK5" ->
{
if(stControl != null)
{
statuses.put(stControl, segment.value(1));
}
}
case "AK9" -> status = segment.value(1);
default ->
{
// Detail segments do not change reconciliation status.
}
}
}
if(control == null || status == null)
{
throw new IllegalArgumentException("Incomplete functional acknowledgement");
}
return (new Result(transaction.st().value(1), control, status, Map.copyOf(statuses), null));
}
}
@@ -0,0 +1,59 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.ack;
import java.time.LocalDateTime;
import java.util.List;
import com.kingsrook.qbits.edi.x12.X12Interchange;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** TA1 is an interchange-level service segment, never an ST/SE transaction.
*******************************************************************************/
public class TA1Builder
{
/*******************************************************************************
** Select the first interchange finding, or acceptance when there are none.
*******************************************************************************/
public X12Segment build(X12Segment isa, List<com.kingsrook.qbits.edi.x12.EnvelopeFinding> findings)
{
return (build(isa, findings.stream().filter(f -> "TA1".equals(f.level())).map(com.kingsrook.qbits.edi.x12.EnvelopeFinding::code).findFirst().orElse("000")));
}
/*******************************************************************************
** Acknowledge original ISA control, date and time with a TA1 note code.
*******************************************************************************/
public X12Segment build(X12Segment isa, String noteCode)
{
return (X12Segment.of("TA1", 2, isa.value(13), isa.value(9), isa.value(10), "000".equals(noteCode) ? "A" : "R", noteCode));
}
/*******************************************************************************
** Generate a reply interchange with swapped identities and its own ISA13.
*******************************************************************************/
public X12Interchange interchange(X12Interchange received, String control, String noteCode, LocalDateTime at)
{
X12Segment isa = AckEnvelope.replyIsa(received.isa(), control, at);
return (new X12Interchange(received.separators(), isa, X12Segment.of("IEA", 3, "0", isa.value(13)), List.of(), List.of(build(received.isa(), noteCode)), List.of()));
}
}
@@ -0,0 +1,293 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.api;
import java.util.List;
import java.util.Map;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.MapperFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.cfg.CoercionAction;
import com.fasterxml.jackson.databind.cfg.CoercionInputShape;
import com.fasterxml.jackson.databind.json.JsonMapper;
import com.fasterxml.jackson.databind.type.LogicalType;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation;
import com.kingsrook.qqq.middleware.javalin.specs.BasicResponse;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtension;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtensionsMetaData;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import com.kingsrook.qqq.openapi.model.In;
import com.kingsrook.qqq.openapi.model.Parameter;
import com.kingsrook.qqq.openapi.model.Schema;
import com.kingsrook.qqq.openapi.model.Type;
import io.javalin.http.HttpStatus;
/*******************************************************************************
** The EDI 1.0.0 UI API: four spec'd operations in the default qbit namespace.
*******************************************************************************/
public final class EdiApi
{
/*******************************************************************************
** Static registration only.
*******************************************************************************/
private EdiApi()
{
}
/*******************************************************************************
** Install with optional Javalin from the qbit producer.
*******************************************************************************/
public static void register(QInstance instance)
{
MiddlewareSpecExtensionsMetaData.ofOrWithNew(instance).withExtension(extension());
}
/*******************************************************************************
** Shared descriptor for route binding and the committed contract.
*******************************************************************************/
public static MiddlewareSpecExtension extension()
{
return (new MiddlewareSpecExtension().withName("edi").withSemanticVersion("1.0.0")
.withDescription("EDI sandbox and mapper, using the signed-in user's permissions.")
.withSpec(GuideTreeSpec.class).withSpec(MapTargetSpec.class).withSpec(PreviewSpec.class).withSpec(ValidateSpec.class));
}
/*******************************************************************************
** Common typed JSON contract; core owns authentication and error mapping.
*******************************************************************************/
public abstract static class EdiSpec extends AbstractJsonEndpointSpec
{
private final String path;
private final String response;
private final String request;
private final String summary;
/*******************************************************************************
** Immutable operation metadata.
*******************************************************************************/
protected EdiSpec(String path, String response, String request, String summary)
{
this.path = path;
this.response = response;
this.request = request;
this.summary = summary;
}
/*******************************************************************************
** Publish the method, path and permission expectations.
*******************************************************************************/
@Override
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/edi" + path).withHttpMethod(request == null ? HttpMethod.GET : HttpMethod.POST)
.withShortSummary(summary).withLongDescription(summary + ". Uses the caller session and table read permissions. Preview also requires edi.map.preview. Rule writes use core table CRUD."));
}
/*******************************************************************************
** Declare positive integer record identifiers.
*******************************************************************************/
@Override
public List<Parameter> defineRequestParameters()
{
String name = path.contains("{guideId}") ? "guideId" : path.contains("{mapId}") ? "mapId" : null;
return (name == null ? List.of() : List.of(new Parameter().withName(name).withIn(In.PATH).withRequired(true)
.withDescription("Positive record identifier.").withSchema(new Schema().withType(Type.INTEGER))));
}
/*******************************************************************************
** All schemas use the Edi prefix and V1 suffix.
*******************************************************************************/
@Override
public Map<String, Schema> defineComponentSchemas()
{
return (path.startsWith("/guides/") ? EdiSchemas.all() : Map.of());
}
/*******************************************************************************
** Response component name.
*******************************************************************************/
@Override
protected String getResponseSchemaName()
{
return (response);
}
/*******************************************************************************
** Optional request body contract.
*******************************************************************************/
@Override
protected Schema defineRequestBodySchema()
{
return (request == null ? null : EdiSchemas.ref(request));
}
/*******************************************************************************
** Core middleware errors are explicitly documented.
*******************************************************************************/
@Override
public List<BasicResponse> defineAdditionalBasicResponses()
{
return (List.of(new BasicResponse(HttpStatus.BAD_REQUEST, "Invalid request or sample exceeds 5 MiB", "EdiErrorV1"),
new BasicResponse(HttpStatus.UNAUTHORIZED, "Sign in required", "EdiErrorV1"),
new BasicResponse(HttpStatus.FORBIDDEN, "Permission denied", "EdiErrorV1"),
new BasicResponse(HttpStatus.NOT_FOUND, "Missing or invisible record", "EdiErrorV1"),
new BasicResponse(HttpStatus.CONTENT_TOO_LARGE, "Host HTTP request size limit exceeded", "EdiErrorV1")));
}
/*******************************************************************************
** Reject malformed or overflowing identifiers without echoing request content.
*******************************************************************************/
protected Integer id(JsonInput input, String name) throws QBadRequestException
{
try
{
int value = Integer.parseInt(input.getPathParam(name));
if(value < 1)
{
throw (new NumberFormatException());
}
return (value);
}
catch(NumberFormatException e)
{
throw (new QBadRequestException("A positive integer identifier is required"));
}
}
/*******************************************************************************
** Strict scalar typing prevents coercion of unsaved edits and sample identifiers.
*******************************************************************************/
protected <T> T body(JsonInput input, Class<T> type) throws QBadRequestException
{
ObjectMapper mapper = JsonMapper.builder().disable(MapperFeature.ALLOW_COERCION_OF_SCALARS)
.disable(DeserializationFeature.ACCEPT_FLOAT_AS_INT).build();
mapper.coercionConfigFor(LogicalType.Textual).setCoercion(CoercionInputShape.Integer, CoercionAction.Fail)
.setCoercion(CoercionInputShape.Float, CoercionAction.Fail).setCoercion(CoercionInputShape.Boolean, CoercionAction.Fail);
try
{
if(input.getBody() == null)
{
throw (new IllegalArgumentException("Object body required"));
}
return (mapper.convertValue(input.getBody(), type));
}
catch(IllegalArgumentException e)
{
throw (new QBadRequestException("Invalid body; check property names and types"));
}
}
}
/*******************************************************************************
** Read the guide tree.
*******************************************************************************/
public static class GuideTreeSpec extends EdiSpec
{
/*******************************************************************************
** Public constructor for core spec discovery.
*******************************************************************************/
public GuideTreeSpec()
{
super("/guides/{guideId}/tree", "EdiGuideTreeV1", null, "Read the guide tree");
}
/*******************************************************************************
** Execute with the authenticated caller.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
return (new EdiApiService().tree(id(input, "guideId")));
}
}
/*******************************************************************************
** Read target metadata and map rules.
*******************************************************************************/
public static class MapTargetSpec extends EdiSpec
{
/*******************************************************************************
** Public constructor for core spec discovery.
*******************************************************************************/
public MapTargetSpec()
{
super("/maps/{mapId}/target", "EdiTargetV1", null, "Read target metadata and map rules");
}
/*******************************************************************************
** Execute with the authenticated caller.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
return (new EdiApiService().target(id(input, "mapId")));
}
}
/*******************************************************************************
** Preview an encrypted sample with optional unsaved rules.
*******************************************************************************/
public static class PreviewSpec extends EdiSpec
{
/*******************************************************************************
** Public constructor for core spec discovery.
*******************************************************************************/
public PreviewSpec()
{
super("/sandbox/preview", "EdiPreviewV1", "EdiPreviewRequestV1", "Preview an encrypted sample with optional unsaved rules");
}
/*******************************************************************************
** Execute with the authenticated caller.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
return (new EdiApiService().preview(body(input, EdiSchemas.EdiPreviewRequestV1.class)));
}
}
/*******************************************************************************
** Validate unsaved map rules.
*******************************************************************************/
public static class ValidateSpec extends EdiSpec
{
/*******************************************************************************
** Public constructor for core spec discovery.
*******************************************************************************/
public ValidateSpec()
{
super("/maps/{mapId}/validate", "EdiValidationV1", "EdiValidateRequestV1", "Validate unsaved map rules");
}
/*******************************************************************************
** Execute with the authenticated caller.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
return (new EdiApiService().validate(id(input, "mapId"), body(input, EdiSchemas.EdiValidateRequestV1.class)));
}
}
}
@@ -0,0 +1,321 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.api;
import java.io.ByteArrayInputStream;
import java.util.ArrayList;
import java.util.Base64;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiAssociationV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiElementV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiFieldV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiGuideTreeV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiLoopV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiPreviewRequestV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiPreviewV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiRuleErrorsV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiRuleV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiSegmentV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiTargetV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiTransformV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiValidateRequestV1;
import com.kingsrook.qbits.edi.api.EdiSchemas.EdiValidationV1;
import com.kingsrook.qbits.edi.map.MapPreviewService;
import com.kingsrook.qbits.edi.map.MapRuleValidator;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadKind;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Caller-authorized projections and sandbox orchestration, using normal QQQ actions.
*******************************************************************************/
public class EdiApiService
{
public static final int MAX_SAMPLE_BYTES = 5 * 1024 * 1024;
private static final ObjectMapper JSON = new ObjectMapper().findAndRegisterModules();
/*******************************************************************************
** Enforced reads preserve table permissions, field locks and partner record locks.
*******************************************************************************/
private static List<QRecord> query(String table, QQueryFilter filter) throws Exception
{
return (EdiRecords.queryEnforced(table, filter));
}
/*******************************************************************************
** A missing or invisible record has the same response.
*******************************************************************************/
private static QRecord row(String table, Integer id) throws Exception
{
if(id == null || id < 1)
{
throw (new QBadRequestException("A positive identifier is required"));
}
var records = query(table, EdiRecords.equal("id", id));
if(records.isEmpty())
{
throw (new QNotFoundException("Not found"));
}
return (records.getFirst());
}
/*******************************************************************************
** Read a guide as a nested tree, preserving duplicate loop ids in separate parents.
*******************************************************************************/
public EdiGuideTreeV1 tree(Integer id) throws Exception
{
row("edi_guide", id);
List<EdiLoopV1> roots = new ArrayList<>();
List<EdiSegmentV1> rootSegments = new ArrayList<>();
Map<String, EdiLoopV1> loops = new LinkedHashMap<>();
Map<String, String> latestPaths = new LinkedHashMap<>();
var segments = query("edi_guide_segment", EdiRecords.equal("guideId", id));
segments.sort(Comparator.comparing(r -> r.getValueInteger("sequence")));
for(var segment : segments)
{
List<EdiElementV1> elements = new ArrayList<>();
var rows = query("edi_guide_element", EdiRecords.equal("guideSegmentId", segment.getValue("id")));
rows.sort(Comparator.comparing((QRecord r) -> r.getValueInteger("position")).thenComparing(r -> Objects.requireNonNullElse(r.getValueInteger("componentPosition"), 0)));
for(var element : rows)
{
List<String> codes = List.of();
Integer codeList = element.getValueInteger("codeListId");
if(codeList != null)
{
row("edi_code_list", codeList);
codes = query("edi_code_list_value", EdiRecords.equal("codeListId", codeList)
.withCriteria("isActive", com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.EQUALS, true))
.stream().map(r -> r.getValueString("code")).sorted().limit(200).toList();
}
elements.add(new EdiElementV1(element.getValueInteger("position"), element.getValueInteger("componentPosition"),
element.getValueString("elementId"), element.getValueString("name"), element.getValueString("dataType"),
element.getValueInteger("minLength"), element.getValueInteger("maxLength"), element.getValueString("usage"), codeList, codes));
}
EdiSegmentV1 node = new EdiSegmentV1(segment.getValueString("segmentId"), segment.getValueString("position"), segment.getValueString("usage"),
segment.getValueInteger("minUse"), segment.getValueInteger("maxUse"), segment.getValueString("name"), elements);
String loopId = segment.getValueString("loopId");
if(loopId == null || loopId.isBlank())
{
rootSegments.add(node);
}
else
{
String parent = latestPaths.get(segment.getValueString("parentLoopId"));
String path = (parent == null ? "" : parent + "/") + loopId;
EdiLoopV1 loop = loops.get(path);
if(loop == null)
{
loop = new EdiLoopV1(loopId, segment.getValueInteger("loopMaxRepeat"), new ArrayList<>(), new ArrayList<>());
loops.put(path, loop);
(parent == null ? roots : loops.get(parent).loops()).add(loop);
}
latestPaths.put(loopId, path);
loop.segments().add(node);
}
}
return (new EdiGuideTreeV1(roots, rootSegments));
}
/*******************************************************************************
** Return visible target metadata, rules and the same registry used by mapping.
*******************************************************************************/
public EdiTargetV1 target(Integer id) throws Exception
{
QRecord map = row("edi_map", id);
QTableMetaData table = targetTable(map);
List<EdiAssociationV1> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(var association : table.getAssociations())
{
QTableMetaData child = QContext.getQInstance().getTable(association.getAssociatedTableName());
associations.add(new EdiAssociationV1(association.getName(), child.getName(), Objects.requireNonNullElse(child.getLabel(), association.getName()), fields(child)));
}
}
List<EdiRuleV1> rules = new ArrayList<>();
for(QRecord rule : query("edi_map_rule", EdiRecords.equal("mapId", id)))
{
rules.add(rule(rule));
}
rules.sort(Comparator.comparing(r -> Objects.requireNonNullElse(r.sequence(), 0)));
return (new EdiTargetV1(table.getName(), fields(table), associations, rules,
EdiRuntime.current().transforms().argumentSchemas().entrySet().stream().map(e -> new EdiTransformV1(e.getKey(), e.getValue())).toList()));
}
/*******************************************************************************
** Resolve the configured target without accepting a client-supplied table.
*******************************************************************************/
private QTableMetaData targetTable(QRecord map) throws Exception
{
QTableMetaData table = QContext.getQInstance().getTable(map.getValueString("tableName"));
if(table == null)
{
throw (new QBadRequestException("The map target table is unavailable"));
}
return (table);
}
/*******************************************************************************
** Table metadata is only exposed with the table's read permission.
*******************************************************************************/
private List<EdiFieldV1> fields(QTableMetaData table) throws Exception
{
QueryInput input = new QueryInput(table.getName());
input.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
PermissionsHelper.checkTablePermissionThrowing(input, TablePermissionSubType.READ);
return (table.getFields().values().stream().map(f -> new EdiFieldV1(f.getName(), f.getLabel(), f.getType().name(), f.getIsRequired(), f.getPossibleValueSourceName())).toList());
}
/*******************************************************************************
** Copy only documented rule fields; do not leak persistence or diagnostic fields.
*******************************************************************************/
private EdiRuleV1 rule(QRecord r)
{
return (new EdiRuleV1(r.getValueInteger("id"), r.getValueInteger("sequence"), r.getValueString("ediPath"), r.getValueString("qqqPath"),
r.getValueString("transform"), r.getValueString("transformArgs"), r.getValueBoolean("required"), r.getValueString("condition"), r.getValueString("defaultValue")));
}
/*******************************************************************************
** Convert unsaved edits to detached records; mapId is never accepted from a rule.
*******************************************************************************/
private List<QRecord> records(List<EdiRuleV1> rules) throws Exception
{
if(rules == null || rules.size() > 10000 || rules.stream().anyMatch(Objects::isNull))
{
throw (new QBadRequestException("Supply an array of at most 10000 rules"));
}
List<QRecord> result = new ArrayList<>();
for(EdiRuleV1 rule : rules)
{
result.add(new QRecord().withValue("id", rule.id()).withValue("sequence", rule.sequence()).withValue("ediPath", rule.ediPath())
.withValue("qqqPath", rule.qqqPath()).withValue("transform", rule.transform()).withValue("transformArgs", rule.transformArgs())
.withValue("required", rule.required()).withValue("condition", rule.condition()).withValue("defaultValue", rule.defaultValue()));
}
return (result);
}
/*******************************************************************************
** Validate all submitted rules, with stable index correlation and no persistence.
*******************************************************************************/
public EdiValidationV1 validate(Integer id, EdiValidateRequestV1 request) throws Exception
{
QRecord map = row("edi_map", id);
targetTable(map);
var guide = MapPreviewService.guide(map, "ORIGINAL");
List<QRecord> rules = records(request.rules());
List<EdiRuleErrorsV1> results = new ArrayList<>();
for(int i = 0; i < rules.size(); i++)
{
QRecord rule = rules.get(i);
results.add(new EdiRuleErrorsV1(i, rule.getValueInteger("id"), rule.getValueInteger("sequence"),
new MapRuleValidator().validate(rule, map, guide, EdiRuntime.current().transforms())));
}
return (new EdiValidationV1(results));
}
/*******************************************************************************
** Authorize before decoding or saving an upload; previews never change map rules.
*******************************************************************************/
public EdiPreviewV1 preview(EdiPreviewRequestV1 request) throws Exception
{
QRecord map = row("edi_map", request.mapId());
AbstractActionInput permission = new AbstractActionInput();
permission.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
PermissionsHelper.checkProcessPermissionThrowing(permission, "edi_map_preview");
String revision = Objects.requireNonNullElse(request.guideRevision(), "CURRENT");
if(!List.of("CURRENT", "ORIGINAL").contains(revision))
{
throw (new QBadRequestException("guideRevision must be CURRENT or ORIGINAL"));
}
if((request.sampleId() == null) == (request.sampleBase64() == null))
{
throw (new QBadRequestException("Supply exactly one of sampleId and sampleBase64"));
}
List<QRecord> overrides = request.ruleOverrides() == null ? null : records(request.ruleOverrides());
QRecord sample;
if(request.sampleId() != null)
{
sample = row("edi_map_sample", request.sampleId());
if(!Objects.equals(sample.getValueInteger("mapId"), request.mapId()))
{
throw (new QNotFoundException("Not found"));
}
}
else
{
String encoded = request.sampleBase64();
if(encoded.length() > ((MAX_SAMPLE_BYTES + 2) / 3) * 4)
{
throw (new QBadRequestException("Sample exceeds the 5 MiB limit"));
}
byte[] bytes;
try
{
bytes = Base64.getDecoder().decode(encoded);
}
catch(IllegalArgumentException e)
{
throw (new QBadRequestException("sampleBase64 must be valid base64"));
}
if(bytes.length == 0 || bytes.length > MAX_SAMPLE_BYTES)
{
throw (new QBadRequestException("Sample must contain 1 byte through 5 MiB"));
}
if(request.sampleName() != null && request.sampleName().length() > 255)
{
throw (new QBadRequestException("sampleName must be at most 255 characters"));
}
// As in the record action, only encrypted sample archival uses the EDI writer.
sample = SystemWrites.call(() ->
{
var ref = EdiRuntime.current().payloads().put(PayloadKind.SAMPLE, new ByteArrayInputStream(bytes));
try
{
return (SystemWrites.insert("edi_map_sample", new QRecord().withValue("mapId", request.mapId())
.withValue("name", Objects.requireNonNullElse(request.sampleName(), "Synthetic upload")).withValue("payloadRef", ref.value())));
}
catch(Exception e)
{
EdiRuntime.current().payloads().delete(ref);
throw (e);
}
});
}
return (JSON.convertValue(new MapPreviewService().preview(map, sample, overrides, revision), EdiPreviewV1.class));
}
}
@@ -0,0 +1,528 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.api;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qqq.middleware.javalin.schemabuilder.ToSchema;
import com.kingsrook.qqq.openapi.model.Schema;
import com.kingsrook.qqq.openapi.model.Type;
/*******************************************************************************
** Versioned wire types; schemas describe values without exposing QRecord internals.
*******************************************************************************/
public final class EdiSchemas
{
/*******************************************************************************
** Static schema catalog.
*******************************************************************************/
private EdiSchemas()
{
}
/*******************************************************************************
** A reference to another versioned wire type.
*******************************************************************************/
public static Schema ref(String name)
{
return (new Schema().withRefToSchema(name));
}
/*******************************************************************************
** Array schema with an explicit item shape.
*******************************************************************************/
private static Schema array(Schema item)
{
return (new Schema().withType(Type.ARRAY).withItems(item));
}
/*******************************************************************************
** Ordered guide tree. Root segments are outside loops; nested loops retain parent scope.
*******************************************************************************/
public record EdiGuideTreeV1(List<EdiLoopV1> loops, List<EdiSegmentV1> segments) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Ordered guide tree. Root segments are outside loops; nested loops retain parent scope.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("loops", array(ref("EdiLoopV1"))),
Map.entry("segments", array(ref("EdiSegmentV1")))))));
}
}
/*******************************************************************************
** Guide loop, nested within its parent; repeat is the maximum occurrence count.
*******************************************************************************/
public record EdiLoopV1(String id, Integer repeat, List<EdiLoopV1> loops, List<EdiSegmentV1> segments) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Guide loop, nested within its parent; repeat is the maximum occurrence count.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("id", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("repeat", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("loops", array(ref("EdiLoopV1"))),
Map.entry("segments", array(ref("EdiSegmentV1")))))));
}
}
/*******************************************************************************
** Segment definition in guide sequence order.
*******************************************************************************/
public record EdiSegmentV1(String id, String position, String usage, Integer min, Integer max, String name, List<EdiElementV1> elements) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Segment definition in guide sequence order.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("id", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("position", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("usage", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("min", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("max", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("name", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("elements", array(ref("EdiElementV1")))))));
}
}
/*******************************************************************************
** Element or composite component; codes contain at most 200 active codes in lexical order.
*******************************************************************************/
public record EdiElementV1(Integer position, Integer component, String id, String name, String type, Integer minLength, Integer maxLength, String usage, Integer codeListId, List<String> codes) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Element or composite component; codes contain at most 200 active codes in lexical order.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("position", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("component", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("id", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("name", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("type", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("minLength", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("maxLength", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("usage", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("codeListId", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("codes", array(new Schema().withType(Type.STRING).withNullable(true)))))));
}
}
/*******************************************************************************
** Target metadata visible to the caller, current rules and registered transforms.
*******************************************************************************/
public record EdiTargetV1(String table, List<EdiFieldV1> fields, List<EdiAssociationV1> associations, List<EdiRuleV1> rules, List<EdiTransformV1> transforms) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Target metadata visible to the caller, current rules and registered transforms.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("table", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("fields", array(ref("EdiFieldV1"))),
Map.entry("associations", array(ref("EdiAssociationV1"))),
Map.entry("rules", array(ref("EdiRuleV1"))),
Map.entry("transforms", array(ref("EdiTransformV1")))))));
}
}
/*******************************************************************************
** QQQ target field metadata.
*******************************************************************************/
public record EdiFieldV1(String name, String label, String type, Boolean required, String possibleValueSourceName) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("QQQ target field metadata.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("name", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("label", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("type", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("required", new Schema().withType(Type.BOOLEAN)),
Map.entry("possibleValueSourceName", new Schema().withType(Type.STRING).withNullable(true))))));
}
}
/*******************************************************************************
** Direct QQQ association and its target fields.
*******************************************************************************/
public record EdiAssociationV1(String name, String table, String label, List<EdiFieldV1> fields) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Direct QQQ association and its target fields.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("name", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("table", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("label", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("fields", array(ref("EdiFieldV1")))))));
}
}
/*******************************************************************************
** Map rule. id is optional for unsaved rules. transformArgs is a JSON object encoded as a string, as in core table CRUD.
*******************************************************************************/
public record EdiRuleV1(Integer id, Integer sequence, String ediPath, String qqqPath, String transform, String transformArgs, Boolean required, String condition, String defaultValue) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Map rule. id is optional for unsaved rules. transformArgs is a JSON object encoded as a string, as in core table CRUD.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("id", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("sequence", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("ediPath", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("qqqPath", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("transform", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("transformArgs", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("required", new Schema().withType(Type.BOOLEAN).withNullable(true)),
Map.entry("condition", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("defaultValue", new Schema().withType(Type.STRING).withNullable(true))))));
}
}
/*******************************************************************************
** Registered transform and JSON Schema describing its string-valued arguments.
*******************************************************************************/
public record EdiTransformV1(String name, Map<String, Object> argumentSchema) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Registered transform and JSON Schema describing its string-valued arguments.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("name", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("argumentSchema", new Schema().withType(Type.OBJECT).withAdditionalProperties(true).withDescription("JSON Schema: type, properties, required and additionalProperties. All transform arguments are strings."))))));
}
}
/*******************************************************************************
** mapId is required. Supply exactly one of sampleId or sampleBase64. Uploads are at most 5 MiB decoded, stored encrypted. ruleOverrides replaces the whole rule set in memory. CURRENT (default) uses the newest active guide in the pinned guide family; ORIGINAL uses the pinned guide.
*******************************************************************************/
public record EdiPreviewRequestV1(Integer mapId, Integer sampleId, String sampleBase64, String sampleName, List<EdiRuleV1> ruleOverrides, String guideRevision) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("mapId is required. Supply exactly one of sampleId or sampleBase64. Uploads are at most 5 MiB decoded, stored encrypted. ruleOverrides replaces the whole rule set in memory. CURRENT (default) uses the newest active guide in the pinned guide family; ORIGINAL uses the pinned guide.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("mapId", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("sampleId", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("sampleBase64", new Schema().withType(Type.STRING).withNullable(true).withFormat("byte").withMaxLength(6990508)),
Map.entry("sampleName", new Schema().withType(Type.STRING).withNullable(true).withMaxLength(255)),
Map.entry("ruleOverrides", array(ref("EdiRuleV1")).withNullable(true)),
Map.entry("guideRevision", new Schema().withType(Type.STRING).withNullable(true).withEnumValues(List.of("CURRENT", "ORIGINAL")))))));
}
}
/*******************************************************************************
** Parse, validate, acknowledge and map without target writes. PHI suppresses result, ack text, and both diffs.
*******************************************************************************/
public record EdiPreviewV1(EdiParseV1 parse, List<EdiFindingV1> findings, EdiAckV1 ack, EdiResultV1 result, EdiDiffV1 diff, Boolean phiSuppressed) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Parse, validate, acknowledge and map without target writes. PHI suppresses result, ack text, and both diffs.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("parse", ref("EdiParseV1")),
Map.entry("findings", array(ref("EdiFindingV1"))),
Map.entry("ack", ref("EdiAckV1")),
Map.entry("result", ref("EdiResultV1").withNullable(true)),
Map.entry("diff", ref("EdiDiffV1")),
Map.entry("phiSuppressed", new Schema().withType(Type.BOOLEAN))))));
}
}
/*******************************************************************************
** Counts of wire interchanges, groups and documents.
*******************************************************************************/
public record EdiParseV1(Integer interchanges, Integer groups, Integer documents) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Counts of wire interchanges, groups and documents.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("interchanges", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("groups", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("documents", new Schema().withType(Type.INTEGER).withNullable(true))))));
}
}
/*******************************************************************************
** Value-free diagnostic. PHI messages are generic; only positions and codes are retained.
*******************************************************************************/
public record EdiFindingV1(Integer snipLevel, String severity, Integer segmentPosition, String segmentId, String loopId, Integer elementPosition, String code, String message) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Value-free diagnostic. PHI messages are generic; only positions and codes are retained.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("snipLevel", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("severity", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("segmentPosition", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("segmentId", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("loopId", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("elementPosition", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("code", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("message", new Schema().withType(Type.STRING).withNullable(true))))));
}
}
/*******************************************************************************
** Concatenated acknowledgement previews; type is 997, 999, TA1, CONTRL, NONE or MIXED. text is null for PHI.
*******************************************************************************/
public record EdiAckV1(String type, String text) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Concatenated acknowledgement previews; type is 997, 999, TA1, CONTRL, NONE or MIXED. text is null for PHI.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("type", new Schema().withType(Type.STRING).withNullable(true)),
Map.entry("text", new Schema().withType(Type.STRING).withNullable(true))))));
}
}
/*******************************************************************************
** Mapped records in document order; no target writes.
*******************************************************************************/
public record EdiResultV1(List<EdiRecordV1> records) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Mapped records in document order; no target writes.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("records", array(ref("EdiRecordV1")))))));
}
}
/*******************************************************************************
** Field values and recursively associated records. Values use the target table field types.
*******************************************************************************/
public record EdiRecordV1(Map<String, Object> values, Map<String, List<EdiRecordV1>> associations) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Field values and recursively associated records. Values use the target table field types.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("values", new Schema().withType(Type.OBJECT).withAdditionalProperties(true)),
Map.entry("associations", new Schema().withType(Type.OBJECT).withAdditionalProperties(array(ref("EdiRecordV1"))))))));
}
}
/*******************************************************************************
** Only differing records are returned. Both lists are empty for PHI.
*******************************************************************************/
public record EdiDiffV1(List<EdiDifferenceV1> againstPreviousRevision, List<EdiDifferenceV1> againstExpected) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Only differing records are returned. Both lists are empty for PHI.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("againstPreviousRevision", array(ref("EdiDifferenceV1"))),
Map.entry("againstExpected", array(ref("EdiDifferenceV1")))))));
}
}
/*******************************************************************************
** One document comparison; documentIndex is zero based.
*******************************************************************************/
public record EdiDifferenceV1(Integer documentIndex, Object before, Object after) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("One document comparison; documentIndex is zero based.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("documentIndex", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("before", new Schema().withNullable(true).withDescription("Expected JSON or a previous mapped record.")),
Map.entry("after", new Schema().withType(Type.OBJECT).withAdditionalProperties(true).withNullable(true))))));
}
}
/*******************************************************************************
** rules is required and may be empty; rules are never persisted.
*******************************************************************************/
public record EdiValidateRequestV1(List<EdiRuleV1> rules) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("rules is required and may be empty; rules are never persisted.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("rules", array(ref("EdiRuleV1")))))));
}
}
/*******************************************************************************
** One result per submitted rule, including rules with no errors.
*******************************************************************************/
public record EdiValidationV1(List<EdiRuleErrorsV1> rules) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("One result per submitted rule, including rules with no errors.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("rules", array(ref("EdiRuleErrorsV1")))))));
}
}
/*******************************************************************************
** Correlate errors using zero-based index, optional id and sequence.
*******************************************************************************/
public record EdiRuleErrorsV1(Integer index, Integer id, Integer sequence, List<String> errors) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Correlate errors using zero-based index, optional id and sequence.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("index", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("id", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("sequence", new Schema().withType(Type.INTEGER).withNullable(true)),
Map.entry("errors", array(new Schema().withType(Type.STRING).withNullable(true)))))));
}
}
/*******************************************************************************
** Core middleware error body.
*******************************************************************************/
public record EdiErrorV1(String error) implements ToSchema
{
/*******************************************************************************
** Describe this wire object for generated clients.
*******************************************************************************/
@Override
public Schema toSchema()
{
return (new Schema().withType(Type.OBJECT).withDescription("Core middleware error body.")
.withProperties(new TreeMap<>(Map.ofEntries(
Map.entry("error", new Schema().withType(Type.STRING).withNullable(true))))));
}
}
/*******************************************************************************
** Complete deterministic schema catalog for all four operations.
*******************************************************************************/
public static Map<String, Schema> all()
{
Map<String, Schema> result = new TreeMap<>();
result.put("EdiGuideTreeV1", new EdiGuideTreeV1(null, null).toSchema());
result.put("EdiLoopV1", new EdiLoopV1(null, null, null, null).toSchema());
result.put("EdiSegmentV1", new EdiSegmentV1(null, null, null, null, null, null, null).toSchema());
result.put("EdiElementV1", new EdiElementV1(null, null, null, null, null, null, null, null, null, null).toSchema());
result.put("EdiTargetV1", new EdiTargetV1(null, null, null, null, null).toSchema());
result.put("EdiFieldV1", new EdiFieldV1(null, null, null, null, null).toSchema());
result.put("EdiAssociationV1", new EdiAssociationV1(null, null, null, null).toSchema());
result.put("EdiRuleV1", new EdiRuleV1(null, null, null, null, null, null, null, null, null).toSchema());
result.put("EdiTransformV1", new EdiTransformV1(null, null).toSchema());
result.put("EdiPreviewRequestV1", new EdiPreviewRequestV1(null, null, null, null, null, null).toSchema());
result.put("EdiPreviewV1", new EdiPreviewV1(null, null, null, null, null, null).toSchema());
result.put("EdiParseV1", new EdiParseV1(null, null, null).toSchema());
result.put("EdiFindingV1", new EdiFindingV1(null, null, null, null, null, null, null, null).toSchema());
result.put("EdiAckV1", new EdiAckV1(null, null).toSchema());
result.put("EdiResultV1", new EdiResultV1(null).toSchema());
result.put("EdiRecordV1", new EdiRecordV1(null, null).toSchema());
result.put("EdiDiffV1", new EdiDiffV1(null, null).toSchema());
result.put("EdiDifferenceV1", new EdiDifferenceV1(null, null, null).toSchema());
result.put("EdiValidateRequestV1", new EdiValidateRequestV1(null).toSchema());
result.put("EdiValidationV1", new EdiValidationV1(null).toSchema());
result.put("EdiRuleErrorsV1", new EdiRuleErrorsV1(null, null, null, null).toSchema());
result.put("EdiErrorV1", new EdiErrorV1(null).toSchema());
return (result);
}
}
@@ -0,0 +1,47 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.app;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryOutput;
/*******************************************************************************
** Explicitly enforce table and field permissions inside the widget's action scope.
** RC.5 treats nested DEFAULT queries as trusted, so the generic renderer alone
** does not remove fields denied by a FieldSecurityLock.
*******************************************************************************/
public class EdiChildRecordListRenderer extends ChildRecordListRenderer
{
/*******************************************************************************
** Use the standard typed child list with an enforced query at its public boundary.
*******************************************************************************/
@Override
protected QueryOutput executeQuery(QueryInput input) throws QException
{
input.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
return (super.executeQuery(input));
}
}
@@ -0,0 +1,522 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.app;
import java.io.Serializable;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.edi.EdiQBitConfig;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.TableData;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Typed EDI home. Every data query explicitly enforces the viewing user's access.
*******************************************************************************/
public class EdiHome extends AbstractWidgetRenderer
{
public static final List<String> ACTIVITY = List.of("edi_transaction", "edi_interchange", "edi_exception");
public static final List<String> SETUP = List.of("edi_partner", "edi_agreement", "edi_transport", "edi_certificate", "edi_guide", "edi_code_list", "edi_map");
private static final Map<String, String> LABELS = new LinkedHashMap<>();
private static final Map<String, String> DOCUMENT_NAMES = new HashMap<>();
private static final List<String> TABLES = List.of("setup", "queue", "acks", "errors", "volume", "system");
static
{
DOCUMENT_NAMES.putAll(Map.ofEntries(Map.entry("810", "Invoice"), Map.entry("846", "Inventory Advice"), Map.entry("850", "Purchase Order"),
Map.entry("855", "PO Acknowledgment"), Map.entry("856", "Ship Notice"), Map.entry("860", "PO Change"), Map.entry("940", "Warehouse Shipping Order"),
Map.entry("945", "Warehouse Shipping Advice"), Map.entry("214", "Shipment Status"), Map.entry("204", "Load Tender"), Map.entry("837", "Claim"),
Map.entry("835", "Remittance"), Map.entry("270", "Eligibility Inquiry"), Map.entry("271", "Eligibility Response"), Map.entry("276", "Claim Status Inquiry"),
Map.entry("277", "Claim Status Response"), Map.entry("834", "Enrollment"), Map.entry("997", "Functional Ack"), Map.entry("999", "Implementation Ack"),
Map.entry("TA1", "Interchange Ack")));
LABELS.put("in_today", "Inbound today");
LABELS.put("out_today", "Outbound today");
LABELS.put("exceptions", "Open exceptions");
LABELS.put("overdue", "Overdue acks");
LABELS.put("dead", "Dead letters");
LABELS.put("certificates", "Certificates expiring (30 days)");
LABELS.put("setup", "Setup status");
LABELS.put("queue", "Exception queue");
LABELS.put("acks", "Overdue acknowledgements");
LABELS.put("errors", "Recent errors (24 hours)");
LABELS.put("volume", "Volume by partner and document (7 days)");
LABELS.put("system", "System");
}
/*******************************************************************************
** Activity and setup only; all other tables remain parent record children.
*******************************************************************************/
public static QAppMetaData app(EdiQBitConfig config, List<QTableMetaData> tables)
{
QAppMetaData app = new QAppMetaData().withName(config.getAppName()).withLabel(config.getAppLabel()).withIcon(new QIcon("swap_horiz"))
.withWidgets(LABELS.keySet().stream().map(n -> "edi_home_" + n).toList())
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED))
.withSection(new QAppSection().withName("activity").withLabel("Activity").withTables(ACTIVITY))
.withSection(new QAppSection().withName("setup").withLabel("Setup").withTables(SETUP));
app.setParentAppName(config.getParentAppName());
tables.stream().filter(t -> ACTIVITY.contains(t.getName()) || SETUP.contains(t.getName())).forEach(app::withChild);
return (app);
}
/*******************************************************************************
** RC.5 typed statistics and tables; no raw markup in metadata or widget data.
*******************************************************************************/
public static List<QWidgetMetaData> widgets()
{
List<QWidgetMetaData> widgets = new ArrayList<>();
LABELS.forEach((name, label) -> widgets.add(new QWidgetMetaData().withName("edi_home_" + name).withLabel(label)
.withType((TABLES.contains(name) ? WidgetType.TABLE : WidgetType.STATISTICS).getType())
.withGridColumns(TABLES.contains(name) ? 12 : 2).withCodeReference(new QCodeReference(EdiHome.class))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED))));
return (widgets);
}
/*******************************************************************************
** Direct rendering also enforces table permissions, including nested process calls.
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
String name = input.getWidgetMetaData().getName().substring("edi_home_".length());
Instant now = EdiRuntime.current().config().getClock().instant();
return (switch(name)
{
case "in_today", "out_today" -> count("edi_transaction", new QQueryFilter()
.withCriteria("direction", QCriteriaOperator.EQUALS, name.startsWith("in_") ? "INBOUND" : "OUTBOUND")
.withCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, now.truncatedTo(ChronoUnit.DAYS)));
case "exceptions" -> count("edi_exception", openExceptions());
case "overdue" -> count("edi_functional_group", overdueAcks(now));
case "dead" -> count("edi_exception", openExceptions().withCriteria("type", QCriteriaOperator.IN, "DEAD_LETTER", "TRANSPORT"));
case "certificates" -> count("edi_certificate", new QQueryFilter().withCriteria("notAfter", QCriteriaOperator.LESS_THAN_OR_EQUALS, now.plus(30, ChronoUnit.DAYS))
.withCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"));
case "setup" -> setup();
case "queue" -> queue(now);
case "acks" -> acks(now);
case "errors" -> errors(now);
case "volume" -> volume(now);
case "system" -> system(now);
default -> throw new QException("Unknown EDI home widget " + name);
});
}
/*******************************************************************************
** A counted statistic linking to the same filter.
*******************************************************************************/
private static RenderWidgetOutput count(String table, QQueryFilter filter) throws QException
{
return (new RenderWidgetOutput(new StatisticsData().withCount(countOf(table, filter)).withCountURL(link(table, filter))));
}
/*******************************************************************************
** Count with enforcement.
*******************************************************************************/
private static Integer countOf(String table, QQueryFilter filter) throws QException
{
CountInput countInput = new CountInput(table).withFilter(filter);
countInput.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
return (new CountAction().execute(countInput).getCount());
}
/*******************************************************************************
** Open and in-progress exceptions.
*******************************************************************************/
private static QQueryFilter openExceptions()
{
return (new QQueryFilter().withCriteria("status", QCriteriaOperator.IN, "OPEN", "IN_PROGRESS"));
}
/*******************************************************************************
** Outbound groups whose expected acknowledgement is past due.
*******************************************************************************/
private static QQueryFilter overdueAcks(Instant now)
{
return (new QQueryFilter().withCriteria("ackStatus", QCriteriaOperator.EQUALS, "PENDING").withCriteria("ackDueAt", QCriteriaOperator.LESS_THAN, now));
}
/*******************************************************************************
** The setup steps in order, with their status and the next step. Counts are
** taken as the viewer; a table the viewer cannot read shows as not visible.
*******************************************************************************/
private static RenderWidgetOutput setup() throws QException
{
QQueryFilter active = new QQueryFilter().withCriteria("isActive", QCriteriaOperator.EQUALS, true);
EdiSetupStatus.Counts counts = new EdiSetupStatus.Counts(safeCount("edi_partner", new QQueryFilter()),
safeCount("edi_identity", new QQueryFilter().withCriteria("isActive", QCriteriaOperator.EQUALS, true).withCriteria("partnerId", QCriteriaOperator.IS_BLANK)),
safeCount("edi_identity", new QQueryFilter().withCriteria("isActive", QCriteriaOperator.EQUALS, true).withCriteria("partnerId", QCriteriaOperator.IS_NOT_BLANK)),
safeCount("edi_agreement", new QQueryFilter()), safeCount("edi_agreement", active),
safeCount("edi_agreement_document", new QQueryFilter()), safeCount("edi_agreement_document", active),
safeCount("edi_transport", new QQueryFilter()), safeCount("edi_transport", active),
safeCount("edi_certificate", new QQueryFilter()), safeCount("edi_guide", new QQueryFilter()), safeCount("edi_code_list", new QQueryFilter()),
safeCount("edi_map", new QQueryFilter()), safeCount("edi_map", new QQueryFilter().withCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE")));
List<Map<String, Object>> rows = new ArrayList<>();
for(EdiSetupStatus.Item item : EdiSetupStatus.items(counts))
{
rows.add(row("Step", item.step(), "Status", item.state().label() + (item.detail().isEmpty() ? "" : ": " + item.detail()), "Next step", item.nextStep().isEmpty() ? "Nothing to do." : item.nextStep()));
}
return (table(List.of("Step", "Status", "Next step"), rows, "edi_partner", new QQueryFilter(), "Open partners"));
}
/*******************************************************************************
** A count the viewer may not be allowed to read: -1 instead of an error.
*******************************************************************************/
private static int safeCount(String table, QQueryFilter filter)
{
try
{
return (countOf(table, filter));
}
catch(QException | RuntimeException e)
{
return (-1);
}
}
/*******************************************************************************
** Open exceptions, oldest first, in four narrow columns.
*******************************************************************************/
private static RenderWidgetOutput queue(Instant now) throws QException
{
Map<Integer, String> partners = partnerNames();
List<Map<String, Object>> rows = new ArrayList<>();
for(QRecord row : query("edi_exception", openExceptions().withOrderBy(new QFilterOrderBy("createDate", true)).withOrderBy(new QFilterOrderBy("id", true)).withLimit(20)))
{
rows.add(row("Exception", plain(row.getValueString("type")) + ": " + row.getValueString("summary"),
"Partner", partnerName(partners, row.getValueInteger("partnerId")), "Age", humanize(Duration.between(row.getValueInstant("createDate"), now))));
}
return (table(List.of("Exception", "Partner", "Age"), rows, "edi_exception", openExceptions(), "View all exceptions"));
}
/*******************************************************************************
** Overdue acknowledgements with partner names and humanized ages.
*******************************************************************************/
private static RenderWidgetOutput acks(Instant now) throws QException
{
Map<Integer, String> partners = partnerNames();
List<Map<String, Object>> rows = new ArrayList<>();
for(QRecord group : query("edi_functional_group", overdueAcks(now).withOrderBy(new QFilterOrderBy("ackDueAt", true)).withLimit(20)))
{
Integer partnerId = null;
Instant sent = null;
List<QRecord> interchanges = query("edi_interchange", new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, group.getValue("interchangeId")));
if(!interchanges.isEmpty())
{
partnerId = interchanges.getFirst().getValueInteger("partnerId");
Serializable messageId = interchanges.getFirst().getValue("outboundMessageId");
if(messageId != null)
{
List<QRecord> messages = query("edi_outbound_message", new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, messageId));
sent = messages.isEmpty() ? null : messages.getFirst().getValueInstant("sentAt");
}
}
Instant due = group.getValueInstant("ackDueAt");
rows.add(row("Partner", partnerName(partners, partnerId), "Group", "GS " + group.getValueString("controlNumber"),
"Sent", sent == null ? "Not sent" : humanize(Duration.between(sent, now)) + " ago", "Overdue by", due == null ? "" : humanize(Duration.between(due, now))));
}
return (table(List.of("Partner", "Group", "Sent", "Overdue by"), rows, "edi_functional_group", overdueAcks(now), "View functional groups"));
}
/*******************************************************************************
** Exceptions raised in the last day, newest first.
*******************************************************************************/
private static RenderWidgetOutput errors(Instant now) throws QException
{
Map<Integer, String> partners = partnerNames();
List<Map<String, Object>> rows = new ArrayList<>();
for(QRecord row : query("edi_exception", lastDay(now).withOrderBy(new QFilterOrderBy("createDate", false)).withOrderBy(new QFilterOrderBy("id", false)).withLimit(20)))
{
rows.add(row("Error", plain(row.getValueString("type")) + ": " + row.getValueString("summary"),
"Partner", partnerName(partners, row.getValueInteger("partnerId")), "When", humanize(Duration.between(row.getValueInstant("createDate"), now)) + " ago"));
}
return (table(List.of("Error", "Partner", "When"), rows, "edi_exception", lastDay(now), "View exceptions"));
}
/*******************************************************************************
** Created within the last day.
*******************************************************************************/
private static QQueryFilter lastDay(Instant now)
{
return (new QQueryFilter().withCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, now.minus(1, ChronoUnit.DAYS)));
}
/*******************************************************************************
** Documents by partner and transaction set over seven days, with a total row.
*******************************************************************************/
private static RenderWidgetOutput volume(Instant now) throws QException
{
Map<Integer, String> partners = partnerNames();
QQueryFilter filter = new QQueryFilter().withCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, now.minus(7, ChronoUnit.DAYS));
Map<List<String>, int[]> counts = new TreeMap<>((a, b) -> a.get(0).equals(b.get(0)) ? a.get(1).compareTo(b.get(1)) : a.get(0).compareTo(b.get(0)));
for(QRecord row : query("edi_transaction", filter))
{
int[] c = counts.computeIfAbsent(List.of(partnerName(partners, row.getValueInteger("partnerId")), documentName(row.getValueString("transactionSet"))), k -> new int[3]);
c["INBOUND".equals(row.getValueString("direction")) ? 0 : 1]++;
if(List.of("INVALID", "REJECTED").contains(row.getValueString("status")))
{
c[2]++;
}
}
List<Map<String, Object>> rows = new ArrayList<>();
int[] totals = new int[3];
for(Map.Entry<List<String>, int[]> entry : counts.entrySet())
{
int[] c = entry.getValue();
rows.add(row("Partner", entry.getKey().get(0), "Document", entry.getKey().get(1), "In / out / rejected", c[0] + " / " + c[1] + " / " + c[2]));
for(int i = 0; i < 3; i++)
{
totals[i] += c[i];
}
}
rows.add(row("Partner", "Total", "Document", "", "In / out / rejected", totals[0] + " / " + totals[1] + " / " + totals[2]));
return (table(List.of("Partner", "Document", "In / out / rejected"), rows, "edi_transaction", filter, "View documents"));
}
/*******************************************************************************
** Integration and queue state as one compact two-column table.
*******************************************************************************/
private static RenderWidgetOutput system(Instant now) throws QException
{
boolean secured = com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getQBits().values().stream().anyMatch(q -> "qbit-secured-data".equals(q.getArtifactId()));
String store = EdiRuntime.current().config().getPayloadStore();
List<QRecord> oldestIn = query("edi_inbox", inboxBacklog().withOrderBy(new QFilterOrderBy("receivedAt", true)).withLimit(1));
List<QRecord> oldestOut = query("edi_outbound_message", outboundQueue().withOrderBy(new QFilterOrderBy("createDate", true)).withLimit(1));
List<Map<String, Object>> rows = new ArrayList<>();
rows.add(row("Item", "Payload storage", "Status", "DATABASE".equals(store) ? "Database, encrypted (demos and tests)" : "Object storage, encrypted"));
rows.add(row("Item", "Secured data", "Status", secured ? "Installed" : "Not installed: PHI fields are not masked"));
rows.add(row("Item", "External code sets", "Status", EdiRuntime.current().config().getExternalCodeSetProvider() == null ? "Format only: no provider configured" : "Provider configured"));
rows.add(row("Item", "Inbound backlog", "Status", countOf("edi_inbox", inboxBacklog())
+ (oldestIn.isEmpty() ? "" : ", oldest " + humanize(Duration.between(oldestIn.getFirst().getValueInstant("receivedAt"), now)))));
rows.add(row("Item", "Outbound queue", "Status", countOf("edi_outbound_message", outboundQueue())
+ (oldestOut.isEmpty() ? "" : ", oldest " + humanize(Duration.between(oldestOut.getFirst().getValueInstant("createDate"), now)))));
return (table(List.of("Item", "Status"), rows, "edi_outbound_message", outboundQueue(), "View outbound queue"));
}
/*******************************************************************************
** Inbox rows not yet processed.
*******************************************************************************/
private static QQueryFilter inboxBacklog()
{
return (new QQueryFilter().withCriteria("status", QCriteriaOperator.IN, "PENDING", "PROCESSING", "RETRY_WAIT"));
}
/*******************************************************************************
** Outbound messages waiting to send.
*******************************************************************************/
private static QQueryFilter outboundQueue()
{
return (new QQueryFilter().withCriteria("status", QCriteriaOperator.IN, "QUEUED", "RETRY_WAIT"));
}
/*******************************************************************************
** Partner names readable by the viewing user.
*******************************************************************************/
private static Map<Integer, String> partnerNames() throws QException
{
Map<Integer, String> names = new HashMap<>();
for(QRecord partner : query("edi_partner", new QQueryFilter()))
{
names.put(partner.getValueInteger("id"), partner.getValueString("name"));
}
return (names);
}
/*******************************************************************************
** A partner's name, or a plain placeholder.
*******************************************************************************/
private static String partnerName(Map<Integer, String> names, Integer id)
{
return (id == null ? "Unrouted" : names.getOrDefault(id, "Partner " + id));
}
/*******************************************************************************
** A transaction set with its plain name.
*******************************************************************************/
static String documentName(String transactionSet)
{
String name = DOCUMENT_NAMES.get(transactionSet);
return (name == null ? String.valueOf(transactionSet) : transactionSet + " " + name);
}
/*******************************************************************************
** UPPER_SNAKE enum value as a sentence-case label.
*******************************************************************************/
static String plain(String value)
{
if(value == null || value.isEmpty())
{
return ("");
}
String lower = value.toLowerCase().replace('_', ' ');
return (Character.toUpperCase(lower.charAt(0)) + lower.substring(1));
}
/*******************************************************************************
** Compact human duration: 45 s, 12 m, 3 h 5 m, 2 d 4 h.
*******************************************************************************/
static String humanize(Duration duration)
{
long seconds = Math.max(0L, duration.getSeconds());
if(seconds < 60)
{
return (seconds + " s");
}
long minutes = seconds / 60;
if(minutes < 60)
{
return (minutes + " m");
}
long hours = minutes / 60;
if(hours < 24)
{
return (hours + " h" + (minutes % 60 == 0 ? "" : " " + (minutes % 60) + " m"));
}
return (hours / 24 + " d" + (hours % 24 == 0 ? "" : " " + (hours % 24) + " h"));
}
/*******************************************************************************
** An ordered row from alternating keys and values.
*******************************************************************************/
private static Map<String, Object> row(Object... keysAndValues)
{
Map<String, Object> row = new LinkedHashMap<>();
for(int i = 0; i < keysAndValues.length; i += 2)
{
row.put(String.valueOf(keysAndValues[i]), keysAndValues[i + 1]);
}
return (row);
}
/*******************************************************************************
** A typed table with a link to the filtered table.
*******************************************************************************/
private static RenderWidgetOutput table(List<String> columns, List<Map<String, Object>> rows, String linkTable, QQueryFilter linkFilter, String linkText) throws QException
{
return (new RenderWidgetOutput(tableData(columns, rows).withLinkURL(link(linkTable, linkFilter)).withLinkText(linkText)));
}
/*******************************************************************************
** Plain cells allow typed rendering and never contain HTML.
*******************************************************************************/
private static TableData tableData(List<String> columns, List<Map<String, Object>> rows)
{
return (new TableData(null, columns.stream().map(c -> new TableData.Column("text", c, c, null, "left")).toList(), rows));
}
/*******************************************************************************
** Explicit permission enforcement is required inside widget renderers.
*******************************************************************************/
private static List<QRecord> query(String table, QQueryFilter filter) throws QException
{
QueryInput query = new QueryInput(table).withFilter(filter);
query.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
return (new QueryAction().execute(query).getRecords());
}
/*******************************************************************************
** Standard filtered-table URL used by statistic and table navigation.
*******************************************************************************/
private static String link(String table, QQueryFilter filter) throws QException
{
try
{
return ("/app/" + table + "?filter=" + URLEncoder.encode(new ObjectMapper().findAndRegisterModules().writeValueAsString(filter), StandardCharsets.UTF_8));
}
catch(Exception e)
{
throw new QException("Could not create EDI table link");
}
}
}
@@ -0,0 +1,156 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.app;
import java.util.ArrayList;
import java.util.List;
/*******************************************************************************
** The Setup status on the EDI home: each setup step in the order it has to be
** done (partner, identity, agreement, document, transport), whether it is
** done, and the next step. The counts use the same conditions the runtime
** uses to route documents (active rows), so the card never reports "set up"
** for rows the runtime ignores.
*******************************************************************************/
public final class EdiSetupStatus
{
/*******************************************************************************
** DONE: working. TODO: the exchange needs it. BUILT_IN: provided by the
** qbit. OPTIONAL: only some setups need it. UNKNOWN: the viewer cannot see it.
*******************************************************************************/
public enum State
{
DONE("Set up"), TODO("Needs setup"), BUILT_IN("Built in"), OPTIONAL("Optional"), UNKNOWN("Not visible to you");
private final String label;
/*******************************************************************************
**
*******************************************************************************/
State(String label)
{
this.label = label;
}
/*******************************************************************************
** The label shown on the card.
*******************************************************************************/
public String label()
{
return (label);
}
}
/*******************************************************************************
** One line of the card.
*******************************************************************************/
public record Item(String step, String tableName, State state, String detail, String nextStep)
{
}
/*******************************************************************************
** The row counts the card is built from; -1 means the viewer cannot read it.
** Active counts are rows with isActive true (what the runtime uses).
*******************************************************************************/
public record Counts(int partners, int ownIdentities, int partnerIdentities, int agreements, int activeAgreements, int agreementDocuments,
int activeAgreementDocuments, int transports, int activeTransports, int certificates, int guides, int codeLists, int maps, int activeMaps)
{
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private EdiSetupStatus()
{
}
/*******************************************************************************
** The items, in setup order.
*******************************************************************************/
public static List<Item> items(Counts c)
{
List<Item> items = new ArrayList<>();
items.add(item("1. Partner", "edi_partner", c.partners(), c.partners(), "partner", "Add the trading partner with New on Partners."));
items.add(identities(c));
items.add(item("3. Agreement", "edi_agreement", c.agreements(), c.activeAgreements(), "agreement",
"Add an agreement on Agreements: your identity, the partner's identity, the standard and test or production. Make sure Active is on."));
items.add(item("4. Agreement documents", "edi_agreement_document", c.agreementDocuments(), c.activeAgreementDocuments(), "agreement document",
"Add one per transaction set and direction on Agreement Documents: pick the guide, and a map if it should become records. A document type with no row is not processed."));
items.add(item("5. Transport", "edi_transport", c.transports(), c.activeTransports(), "transport",
"Add the AS2, SFTP or HTTPS connection on Transports (secrets as variable names), link it on the agreement, then run Test Connection."));
items.add(c.certificates() < 0 ? new Item("Certificates", "edi_certificate", State.UNKNOWN, "", "")
: c.certificates() == 0
? new Item("Certificates", "edi_certificate", State.OPTIONAL, "None.", "Needed only for AS2 signing and encryption: add yours and the partner's on Certificates.")
: new Item("Certificates", "edi_certificate", State.DONE, c.certificates() + (c.certificates() == 1 ? " certificate." : " certificates."), ""));
items.add(c.guides() < 0 || c.codeLists() < 0 ? new Item("Guides and code lists", "edi_guide", State.UNKNOWN, "", "")
: c.guides() == 0 || c.codeLists() == 0
? new Item("Guides and code lists", "edi_guide", State.TODO, "The built-in guides are not loaded.", "The host must call EdiRuntime.initialize() at startup, after the schema exists.")
: new Item("Guides and code lists", "edi_guide", State.BUILT_IN, c.guides() + " guides and " + c.codeLists() + " code lists.", ""));
items.add(c.maps() < 0 ? new Item("Maps", "edi_map", State.UNKNOWN, "", "")
: c.maps() == 0
? new Item("Maps", "edi_map", State.OPTIONAL, "None.", "Needed to turn documents into records. Run Install sample maps on Maps to start from an example.")
: c.activeMaps() == 0
? new Item("Maps", "edi_map", State.TODO, c.maps() + " maps, none active.", "Point a map at your table, preview it, then run Activate Map.")
: new Item("Maps", "edi_map", State.DONE, c.activeMaps() + " active of " + c.maps() + ".", ""));
return (items);
}
/*******************************************************************************
** Step 2: your own identity (no partner) and the partner's.
*******************************************************************************/
private static Item identities(Counts c)
{
String step = "2. Identities";
if(c.ownIdentities() < 0 || c.partnerIdentities() < 0)
{
return (new Item(step, "edi_identity", State.UNKNOWN, "", ""));
}
if(c.ownIdentities() == 0)
{
return (new Item(step, "edi_identity", State.TODO, "None of yours.", "Add your own ISA or UNB qualifier and ID on Identities, with no partner."));
}
if(c.partnerIdentities() == 0)
{
return (new Item(step, "edi_identity", State.TODO, "Yours is set; no partner has one.", "Add the partner's qualifier and ID on Identities, with the partner selected."));
}
return (new Item(step, "edi_identity", State.DONE, c.ownIdentities() + " of yours, " + c.partnerIdentities() + " for partners.", ""));
}
/*******************************************************************************
** A step that needs at least one active row.
*******************************************************************************/
private static Item item(String step, String table, int total, int active, String noun, String next)
{
if(total < 0 || active < 0)
{
return (new Item(step, table, State.UNKNOWN, "", ""));
}
if(total == 0)
{
return (new Item(step, table, State.TODO, "None.", next));
}
if(active == 0)
{
return (new Item(step, table, State.TODO, total + " " + noun + (total == 1 ? "" : "s") + ", none active.", "Turn Active on for the one to use."));
}
return (new Item(step, table, State.DONE, active + " active.", ""));
}
}
@@ -0,0 +1,646 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import com.helger.as2lib.crypto.BCCryptoHelper;
import com.helger.as2lib.crypto.ECryptoAlgorithmCrypt;
import com.helger.as2lib.crypto.ECryptoAlgorithmSign;
import com.helger.as2lib.crypto.MIC;
import com.helger.mail.cte.EContentTransferEncoding;
import com.kingsrook.qbits.edi.as2.CertificateService.LocalKey;
import jakarta.activation.DataHandler;
import jakarta.mail.internet.ContentType;
import jakarta.mail.internet.MimeBodyPart;
import jakarta.mail.internet.MimeMultipart;
import jakarta.mail.internet.MimeUtility;
import jakarta.mail.util.ByteArrayDataSource;
import org.bouncycastle.cms.CMSEnvelopedData;
import org.bouncycastle.cms.CMSEnvelopedDataGenerator;
import org.bouncycastle.cms.CMSProcessableByteArray;
import org.bouncycastle.cms.CMSSignerDigestMismatchException;
import org.bouncycastle.cms.RecipientInformation;
import org.bouncycastle.cms.SignerInformation;
import org.bouncycastle.cms.jcajce.JcaSimpleSignerInfoVerifierBuilder;
import org.bouncycastle.cms.jcajce.JceCMSContentEncryptorBuilder;
import org.bouncycastle.cms.jcajce.JceKeyTransEnvelopedRecipient;
import org.bouncycastle.cms.jcajce.JceKeyTransRecipientId;
import org.bouncycastle.cms.jcajce.JceKeyTransRecipientInfoGenerator;
import org.bouncycastle.cms.jcajce.ZlibCompressor;
import org.bouncycastle.cms.jcajce.ZlibExpanderProvider;
import org.bouncycastle.mail.smime.SMIMECompressed;
import org.bouncycastle.mail.smime.SMIMECompressedGenerator;
import org.bouncycastle.mail.smime.SMIMEEnveloped;
import org.bouncycastle.mail.smime.SMIMESigned;
/*******************************************************************************
** The AS2 cryptographic layer, entirely in memory. S/MIME signing and the
** algorithm tables use as2-lib's BouncyCastle helper; encryption, decryption,
** signature verification, compression and the MIC use BouncyCastle directly,
** because as2-lib's decrypt and verify stage content in temporary files and this
** qbit never writes local files. The MIC is computed over the exact bytes of the
** signed (or, when unsigned, the transmitted) MIME entity, headers included, so
** both ends agree whatever transfer encoding was used. Messages follow RFC 4130
** (AS2), RFC 5751 (S/MIME) and RFC 5402 (compression).
*******************************************************************************/
public final class As2Crypto
{
private static final BCCryptoHelper HELPER = new BCCryptoHelper();
private static final String PROVIDER = BCCryptoHelper.DEFAULT_SECURITY_PROVIDER_NAME;
private static final byte[] HEADER_END = "\r\n\r\n".getBytes(StandardCharsets.US_ASCII);
private static final byte[] LINE_END = "\r\n".getBytes(StandardCharsets.US_ASCII);
public static final long MAX_EXPANDED_BYTES = 256L * 1024 * 1024;
/*******************************************************************************
** The result of unwrapping an inbound message: the EDI payload, what protection
** was applied and the MIC of the signed (or received) content.
*******************************************************************************/
public record Decoded(byte[] payload, String fileName, Boolean encrypted, Boolean signed, Boolean compressed, MIC mic)
{
}
/*******************************************************************************
** A MIME entity split for HTTP: the Content-Type, transfer encoding and
** disposition go in HTTP headers, the body is the HTTP body.
*******************************************************************************/
public record Wire(String contentType, String transferEncoding, String contentDisposition, byte[] body)
{
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private As2Crypto()
{
}
/*******************************************************************************
** Map the signAlgorithm setting; null means unsigned.
*******************************************************************************/
public static ECryptoAlgorithmSign signAlgorithm(String setting)
{
if(setting == null || setting.isBlank())
{
return (null);
}
return (switch(setting)
{
case "SHA256" -> ECryptoAlgorithmSign.DIGEST_SHA_256;
case "SHA384" -> ECryptoAlgorithmSign.DIGEST_SHA_384;
case "SHA512" -> ECryptoAlgorithmSign.DIGEST_SHA_512;
case "SHA1" -> ECryptoAlgorithmSign.DIGEST_SHA_1;
default -> throw new IllegalArgumentException("Unsupported signAlgorithm");
});
}
/*******************************************************************************
** Map the encryptAlgorithm setting; null means unencrypted.
*******************************************************************************/
public static ECryptoAlgorithmCrypt encryptAlgorithm(String setting)
{
if(setting == null || setting.isBlank())
{
return (null);
}
return (switch(setting)
{
case "AES128_CBC" -> ECryptoAlgorithmCrypt.CRYPT_AES128_CBC;
case "AES256_CBC" -> ECryptoAlgorithmCrypt.CRYPT_AES256_CBC;
case "AES128_GCM" -> ECryptoAlgorithmCrypt.CRYPT_AES128_GCM;
case "AES256_GCM" -> ECryptoAlgorithmCrypt.CRYPT_AES256_GCM;
case "3DES" -> ECryptoAlgorithmCrypt.CRYPT_3DES;
default -> throw new IllegalArgumentException("Unsupported encryptAlgorithm");
});
}
/*******************************************************************************
** SHA1 signatures and 3DES encryption are accepted only when set explicitly,
** and are flagged as weak.
*******************************************************************************/
public static boolean isWeak(String sign, String encrypt)
{
return ("SHA1".equals(sign) || "3DES".equals(encrypt));
}
/*******************************************************************************
** Parse a complete MIME entity (headers, blank line, body) held in memory.
*******************************************************************************/
public static MimeBodyPart parse(byte[] entity) throws As2Exception
{
try
{
return (new MimeBodyPart(new ByteArrayInputStream(entity)));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 message is not a valid MIME entity", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Rebuild the MIME entity of an HTTP request or response from its Content-Type,
** Content-Transfer-Encoding and Content-Disposition headers and its body.
*******************************************************************************/
public static MimeBodyPart fromHttp(String contentType, String transferEncoding, String contentDisposition, byte[] body) throws As2Exception
{
return (parse(entity(contentType, transferEncoding, contentDisposition, body)));
}
/*******************************************************************************
** The MIME entity bytes (headers, blank line, body) for HTTP-level parts. Header
** values with line breaks are refused.
*******************************************************************************/
public static byte[] entity(String contentType, String transferEncoding, String contentDisposition, byte[] body) throws As2Exception
{
for(String value : Arrays.asList(contentType, transferEncoding, contentDisposition))
{
if(value != null && (value.contains("\r") || value.contains("\n")))
{
throw (new As2Exception("The AS2 message has an invalid header", As2Exception.UNEXPECTED_PROCESSING));
}
}
if(contentType == null || contentType.isBlank())
{
throw (new As2Exception("The AS2 message has no valid Content-Type", As2Exception.UNEXPECTED_PROCESSING));
}
ByteArrayOutputStream entity = new ByteArrayOutputStream();
String headers = "Content-Type: " + contentType + "\r\n" + (transferEncoding == null || transferEncoding.isBlank() ? "" : "Content-Transfer-Encoding: " + transferEncoding + "\r\n")
+ (contentDisposition == null || contentDisposition.isBlank() ? "" : "Content-Disposition: " + contentDisposition + "\r\n") + "\r\n";
entity.writeBytes(headers.getBytes(StandardCharsets.ISO_8859_1));
entity.writeBytes(body);
return (entity.toByteArray());
}
/*******************************************************************************
** Serialize a MIME entity (headers and encoded body).
*******************************************************************************/
public static byte[] toBytes(MimeBodyPart part) throws As2Exception
{
try
{
ByteArrayOutputStream out = new ByteArrayOutputStream();
part.writeTo(out);
return (out.toByteArray());
}
catch(Exception e)
{
throw (new As2Exception("The AS2 message could not be serialized", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Split a MIME entity into the HTTP headers that describe it and its body.
*******************************************************************************/
public static Wire toWire(MimeBodyPart part) throws As2Exception
{
return (splitRaw(toBytes(part)));
}
/*******************************************************************************
** Split already serialized MIME entity bytes into the Content-Type,
** Content-Transfer-Encoding and Content-Disposition values and the body,
** without re-serializing (which could invalidate a signature).
*******************************************************************************/
public static Wire splitRaw(byte[] bytes) throws As2Exception
{
int split = indexOf(bytes, HEADER_END, 0);
if(split < 0)
{
throw (new As2Exception("The AS2 message has no body", As2Exception.UNEXPECTED_PROCESSING));
}
String contentType = null;
String encoding = null;
String disposition = null;
String current = null;
StringBuilder value = new StringBuilder();
for(String line : (new String(bytes, 0, split, StandardCharsets.ISO_8859_1) + "\r\nX-End: end").split("\r\n"))
{
if((line.startsWith(" ") || line.startsWith("\t")) && current != null)
{
value.append(' ').append(line.trim());
continue;
}
if(current != null)
{
contentType = "content-type".equals(current) ? value.toString() : contentType;
encoding = "content-transfer-encoding".equals(current) ? value.toString() : encoding;
disposition = "content-disposition".equals(current) ? value.toString() : disposition;
}
int colon = line.indexOf(':');
current = colon < 0 ? null : line.substring(0, colon).trim().toLowerCase(java.util.Locale.ROOT);
value = new StringBuilder(colon < 0 ? "" : line.substring(colon + 1).trim());
}
byte[] body = new byte[bytes.length - split - HEADER_END.length];
System.arraycopy(bytes, split + HEADER_END.length, body, 0, body.length);
return (new Wire(contentType, encoding, disposition, body));
}
/*******************************************************************************
** The EDI document as a binary MIME part.
*******************************************************************************/
public static MimeBodyPart payloadPart(byte[] payload, String contentType, String fileName) throws As2Exception
{
try
{
MimeBodyPart part = new MimeBodyPart();
part.setDataHandler(new DataHandler(new ByteArrayDataSource(payload, contentType)));
part.setHeader("Content-Type", contentType);
part.setHeader("Content-Transfer-Encoding", "binary");
part.setHeader("Content-Disposition", "attachment; filename=\"" + fileName + "\"");
return (part);
}
catch(Exception e)
{
throw (new As2Exception("The EDI document could not be wrapped for AS2", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Compress a part (RFC 5402, zlib).
*******************************************************************************/
public static MimeBodyPart compress(MimeBodyPart part) throws As2Exception
{
try
{
SMIMECompressedGenerator generator = new SMIMECompressedGenerator();
generator.setContentTransferEncoding("binary");
return (generator.generate(part, new ZlibCompressor()));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 message could not be compressed", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** The MIC of a MIME entity: the digest of its exact bytes, headers included
** (RFC 4130 section 7.3.1).
*******************************************************************************/
public static MIC mic(byte[] entity, ECryptoAlgorithmSign algorithm) throws As2Exception
{
try
{
return (new MIC(MessageDigest.getInstance(algorithm.getID().toUpperCase(java.util.Locale.ROOT)).digest(entity), algorithm));
}
catch(Exception e)
{
throw (new As2Exception("The message integrity check could not be computed", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** The exact bytes of the first body part of a multipart entity: the signed
** content of multipart/signed, without the delimiter lines around it.
*******************************************************************************/
public static byte[] firstPart(String contentType, byte[] body) throws As2Exception
{
try
{
String boundary = new ContentType(contentType).getParameter("boundary");
int open = indexOf(body, ("--" + boundary).getBytes(StandardCharsets.ISO_8859_1), 0);
int start = open < 0 ? -1 : indexOf(body, LINE_END, open) + LINE_END.length;
int end = start < 0 ? -1 : indexOf(body, ("\r\n--" + boundary).getBytes(StandardCharsets.ISO_8859_1), start - LINE_END.length);
if(end < 0)
{
throw (new IllegalArgumentException("no signed content"));
}
return (Arrays.copyOfRange(body, start, end));
}
catch(Exception e)
{
throw (new As2Exception("The signed AS2 message is malformed", As2Exception.AUTHENTICATION_FAILED));
}
}
/*******************************************************************************
** Sign a part into multipart/signed with the local key.
*******************************************************************************/
public static MimeBodyPart sign(MimeBodyPart part, LocalKey key, ECryptoAlgorithmSign algorithm) throws As2Exception
{
try
{
return (HELPER.sign(part, key.certificate(), key.key(), algorithm, true, algorithm == ECryptoAlgorithmSign.DIGEST_SHA_1, false, EContentTransferEncoding.BINARY));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 message could not be signed with certificate " + key.row().getValueString("alias") + ": " + e.getClass().getSimpleName(), As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Encrypt the exact bytes of a MIME entity to the partner certificate (CMS
** enveloped data), returning the application/pkcs7-mime wire message.
*******************************************************************************/
public static Wire encrypt(byte[] entity, X509Certificate certificate, ECryptoAlgorithmCrypt algorithm) throws As2Exception
{
try
{
CMSEnvelopedDataGenerator generator = new CMSEnvelopedDataGenerator();
generator.addRecipientInfoGenerator(new JceKeyTransRecipientInfoGenerator(certificate).setProvider(PROVIDER));
CMSEnvelopedData enveloped = generator.generate(new CMSProcessableByteArray(entity), new JceCMSContentEncryptorBuilder(algorithm.getOID()).setProvider(PROVIDER).build());
return (new Wire("application/pkcs7-mime; smime-type=enveloped-data; name=smime.p7m", "binary", "attachment; filename=\"smime.p7m\"", enveloped.getEncoded()));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 message could not be encrypted: " + e.getClass().getSimpleName(), As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Unwrap an inbound entity: decrypt with any of our keys, verify the signature
** against the partner certificates, decompress, and compute the MIC over the
** exact signed bytes (or, when unsigned, the entity after decryption). Layers
** may nest in any order (RFC 5402 allows compression before or after signing).
*******************************************************************************/
public static Decoded decode(byte[] entity, List<LocalKey> keys, List<X509Certificate> partnerCertificates, ECryptoAlgorithmSign micAlgorithm, Date now) throws As2Exception
{
byte[] raw = entity;
byte[] micBytes = raw;
MimeBodyPart part = parse(raw);
boolean encrypted = false;
boolean signed = false;
boolean compressed = false;
for(int depth = 0; depth < 6; depth++)
{
if(isEncrypted(part))
{
raw = decrypt(part, keys);
part = parse(raw);
micBytes = signed ? micBytes : raw;
encrypted = true;
}
else if(isSigned(part))
{
verify(part, partnerCertificates, now);
Wire wire = splitRaw(raw);
raw = firstPart(wire.contentType(), wire.body());
micBytes = raw;
part = parse(raw);
signed = true;
}
else if(isCompressed(part))
{
raw = expand(part);
part = parse(raw);
compressed = true;
}
else
{
break;
}
}
try
{
MIC mic = micAlgorithm == null ? null : mic(micBytes, micAlgorithm);
return (new Decoded(part.getInputStream().readAllBytes(), part.getFileName(), encrypted, signed, compressed, mic));
}
catch(As2Exception e)
{
throw (e);
}
catch(Exception e)
{
throw (new As2Exception("The AS2 payload could not be read", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Verify a multipart/signed entity against the candidate certificates and
** return the signed content. A digest that does not match the content is an
** integrity failure; a signature no candidate certificate verifies is an
** authentication failure. Expired certificates are not accepted.
*******************************************************************************/
public static MimeBodyPart verify(MimeBodyPart part, List<X509Certificate> certificates, Date now) throws As2Exception
{
SMIMESigned signed;
try
{
signed = new SMIMESigned(new MimeMultipart(new ByteArrayDataSource(part.getRawInputStream().readAllBytes(), MimeUtility.unfold(part.getContentType()))));
}
catch(Exception e)
{
throw (new As2Exception("The signed AS2 message is malformed", As2Exception.AUTHENTICATION_FAILED));
}
for(SignerInformation signer : signed.getSignerInfos().getSigners())
{
for(X509Certificate certificate : certificates)
{
if(certificate.getNotAfter().before(now))
{
continue;
}
try
{
if(signer.verify(new JcaSimpleSignerInfoVerifierBuilder().setProvider(PROVIDER).build(certificate)))
{
return (signed.getContent());
}
}
catch(CMSSignerDigestMismatchException e)
{
throw (new As2Exception("The message digest does not match the signed content", As2Exception.INTEGRITY_CHECK_FAILED));
}
catch(Exception e)
{
// This certificate cannot verify the signature; try the next.
continue;
}
}
}
throw (new As2Exception("The signature could not be verified with the partner certificates", As2Exception.AUTHENTICATION_FAILED));
}
/*******************************************************************************
** Decrypt with the first of our keys the message was encrypted to, returning
** the clear MIME entity bytes.
*******************************************************************************/
private static byte[] decrypt(MimeBodyPart part, List<LocalKey> keys) throws As2Exception
{
SMIMEEnveloped enveloped;
try
{
enveloped = new SMIMEEnveloped(part);
}
catch(Exception e)
{
throw (new As2Exception("The encrypted AS2 message is malformed", As2Exception.DECRYPTION_FAILED));
}
for(LocalKey key : keys)
{
try
{
RecipientInformation recipient = enveloped.getRecipientInfos().get(new JceKeyTransRecipientId(key.certificate()));
if(recipient != null)
{
return (recipient.getContent(new JceKeyTransEnvelopedRecipient(key.key()).setProvider(PROVIDER)));
}
}
catch(Exception e)
{
// The key matched the recipient but could not decrypt; try the next key.
continue;
}
}
throw (new As2Exception("The message was not encrypted to an accepted certificate", As2Exception.DECRYPTION_FAILED));
}
/*******************************************************************************
** Inflate a compressed-data entity, bounded to prevent decompression bombs.
*******************************************************************************/
private static byte[] expand(MimeBodyPart part) throws As2Exception
{
try
{
return (new SMIMECompressed(part).getContent(new ZlibExpanderProvider(MAX_EXPANDED_BYTES)));
}
catch(Exception e)
{
throw (new As2Exception("The compressed AS2 message could not be expanded", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** True for application/pkcs7-mime enveloped-data.
*******************************************************************************/
private static boolean isEncrypted(MimeBodyPart part) throws As2Exception
{
try
{
return (HELPER.isEncrypted(part));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 Content-Type is invalid", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** True for multipart/signed.
*******************************************************************************/
private static boolean isSigned(MimeBodyPart part) throws As2Exception
{
try
{
return (HELPER.isSigned(part));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 Content-Type is invalid", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** True for application/pkcs7-mime compressed-data.
*******************************************************************************/
private static boolean isCompressed(MimeBodyPart part) throws As2Exception
{
try
{
return (HELPER.isCompressed(part.getContentType()));
}
catch(Exception e)
{
throw (new As2Exception("The AS2 Content-Type is invalid", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Index of a byte sequence at or after a position, or -1.
*******************************************************************************/
private static int indexOf(byte[] data, byte[] pattern, int from)
{
for(int i = Math.max(from, 0); i <= data.length - pattern.length; i++)
{
int j = 0;
while(j < pattern.length && data[i + j] == pattern[j])
{
j++;
}
if(j == pattern.length)
{
return (i);
}
}
return (-1);
}
}
@@ -0,0 +1,69 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
/*******************************************************************************
** A plain, value-free AS2 failure. The message never carries key material,
** secrets or payload content; the optional code is the RFC 4130 disposition
** modifier a receiver reports to the sender (for example authentication-failed).
*******************************************************************************/
public class As2Exception extends Exception
{
public static final String DECRYPTION_FAILED = "decryption-failed";
public static final String AUTHENTICATION_FAILED = "authentication-failed";
public static final String INTEGRITY_CHECK_FAILED = "integrity-check-failed";
public static final String INSUFFICIENT_SECURITY = "insufficient-message-security";
public static final String UNEXPECTED_PROCESSING = "unexpected-processing-error";
private final String code;
/*******************************************************************************
** Failure without a disposition code (configuration and local problems).
*******************************************************************************/
public As2Exception(String message)
{
this(message, null);
}
/*******************************************************************************
** Failure carrying the RFC 4130 disposition modifier.
*******************************************************************************/
public As2Exception(String message, String code)
{
super(message);
this.code = code;
}
/*******************************************************************************
** The RFC 4130 disposition modifier, or null when the failure is local.
*******************************************************************************/
public String getCode()
{
return (code);
}
}
@@ -0,0 +1,234 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.time.Duration;
import java.util.Base64;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
/*******************************************************************************
** HTTP client helpers for AS2: bounded timeouts, no redirects, optional pinning
** of the partner TLS certificate (otherwise the JVM trust store applies), and
** bounded response bodies.
*******************************************************************************/
public final class As2Http
{
public static final int MAX_RESPONSE_BYTES = 10 * 1024 * 1024;
/*******************************************************************************
** Status, lower-case header names and the (bounded) body of a response.
*******************************************************************************/
public record Response(int status, Map<String, String> headers, byte[] body)
{
}
/*******************************************************************************
** Accepts only the pinned certificate as the server leaf; the hostname is
** still checked by the TLS stack.
*******************************************************************************/
private static final class PinnedTrustManager implements X509TrustManager
{
private final byte[] pinned;
/*******************************************************************************
** Pin the SHA-256 of the encoded certificate.
*******************************************************************************/
private PinnedTrustManager(X509Certificate certificate) throws Exception
{
pinned = MessageDigest.getInstance("SHA-256").digest(certificate.getEncoded());
}
/*******************************************************************************
** This client never authenticates with a client certificate.
*******************************************************************************/
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException
{
throw (new CertificateException("Client certificates are not accepted"));
}
/*******************************************************************************
** Require the leaf to be exactly the pinned certificate.
*******************************************************************************/
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException
{
try
{
if(chain == null || chain.length == 0 || !MessageDigest.isEqual(pinned, MessageDigest.getInstance("SHA-256").digest(chain[0].getEncoded())))
{
throw (new CertificateException("The server certificate does not match the pinned partner TLS certificate"));
}
}
catch(CertificateException e)
{
throw (e);
}
catch(Exception e)
{
throw (new CertificateException("The server certificate could not be checked against the pin"));
}
}
/*******************************************************************************
** No trust anchors beyond the pin.
*******************************************************************************/
@Override
public X509Certificate[] getAcceptedIssuers()
{
return (new X509Certificate[0]);
}
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private As2Http()
{
}
/*******************************************************************************
** A client using the JVM trust store, or only the pinned certificate.
*******************************************************************************/
public static HttpClient client(X509Certificate pinned) throws Exception
{
HttpClient.Builder builder = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(15)).followRedirects(HttpClient.Redirect.NEVER);
if(pinned != null)
{
SSLContext context = SSLContext.getInstance("TLS");
context.init(null, new TrustManager[] {new PinnedTrustManager(pinned)}, new SecureRandom());
builder.sslContext(context);
}
return (builder.build());
}
/*******************************************************************************
** Trust manager for pin checks in tests and diagnostics.
*******************************************************************************/
public static X509TrustManager pinnedTrustManager(X509Certificate pinned) throws Exception
{
return (new PinnedTrustManager(pinned));
}
/*******************************************************************************
** POST bytes with the given headers.
*******************************************************************************/
public static Response post(HttpClient client, URI uri, Map<String, String> headers, byte[] body, Duration timeout) throws Exception
{
HttpRequest.Builder request = HttpRequest.newBuilder(uri).timeout(timeout).POST(HttpRequest.BodyPublishers.ofByteArray(body));
headers.forEach(request::header);
return (read(client.send(request.build(), HttpResponse.BodyHandlers.ofInputStream())));
}
/*******************************************************************************
** OPTIONS probe: no body is sent.
*******************************************************************************/
public static Response options(HttpClient client, URI uri, Map<String, String> headers, Duration timeout) throws Exception
{
HttpRequest.Builder request = HttpRequest.newBuilder(uri).timeout(timeout).method("OPTIONS", HttpRequest.BodyPublishers.noBody());
headers.forEach(request::header);
return (read(client.send(request.build(), HttpResponse.BodyHandlers.ofInputStream())));
}
/*******************************************************************************
** Read the bounded body and lower-case the header names.
*******************************************************************************/
private static Response read(HttpResponse<InputStream> response) throws Exception
{
try(InputStream stream = response.body())
{
Map<String, String> headers = new TreeMap<>();
response.headers().map().forEach((name, values) -> headers.put(name.toLowerCase(Locale.ROOT), values.getFirst()));
return (new Response(response.statusCode(), headers, stream.readNBytes(MAX_RESPONSE_BYTES)));
}
}
/*******************************************************************************
** An Authorization header value for HTTP Basic authentication.
*******************************************************************************/
public static String basic(String user, String password)
{
return ("Basic " + Base64.getEncoder().encodeToString((user + ":" + password).getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
** TLS is required except on loopback, which the tests and local peers use.
*******************************************************************************/
public static boolean isAllowedUrl(String url)
{
try
{
URI uri = URI.create(url);
return (uri.getHost() != null && uri.getUserInfo() == null
&& ("https".equals(uri.getScheme()) || "http".equals(uri.getScheme()) && List.of("127.0.0.1", "localhost", "[::1]").contains(uri.getHost())));
}
catch(Exception e)
{
return (false);
}
}
}
@@ -0,0 +1,261 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.security.cert.X509Certificate;
import java.util.Date;
import java.util.List;
import com.helger.as2lib.crypto.ECryptoAlgorithmSign;
import com.helger.as2lib.crypto.MIC;
import com.kingsrook.qbits.edi.as2.CertificateService.LocalKey;
import jakarta.mail.internet.InternetHeaders;
import jakarta.mail.internet.MimeBodyPart;
import jakarta.mail.internet.MimeMultipart;
import jakarta.mail.internet.MimeUtility;
import jakarta.mail.util.ByteArrayDataSource;
/*******************************************************************************
** Builds and parses AS2 message disposition notifications (RFC 4130 section 7,
** RFC 8098): multipart/report with a message/disposition-notification part,
** optionally wrapped in multipart/signed. Memory only.
*******************************************************************************/
public final class As2Mdn
{
public static final String PROCESSED = "automatic-action/MDN-sent-automatically; processed";
/*******************************************************************************
** A receipt ready to send or store: HTTP-level parts and the complete MIME
** entity (headers and body) kept as the stored raw MDN.
*******************************************************************************/
public record Built(String messageId, As2Crypto.Wire wire, byte[] entity, Boolean signed)
{
}
/*******************************************************************************
** A receipt read from a partner. Fields come from the disposition notification;
** signed and signatureValid describe the signature, when there was one.
*******************************************************************************/
public record Parsed(String originalMessageId, String disposition, String micText, MIC mic, Boolean signed, Boolean signatureValid)
{
/*******************************************************************************
** Success means the partner processed the message (a warning still counts).
*******************************************************************************/
public boolean processed()
{
if(disposition == null)
{
return (false);
}
int separator = disposition.indexOf(';');
String status = (separator < 0 ? disposition : disposition.substring(separator + 1)).trim().toLowerCase(java.util.Locale.ROOT);
return (status.equals("processed") || status.startsWith("processed/warning"));
}
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private As2Mdn()
{
}
/*******************************************************************************
** Disposition text for a failure, e.g. processed/error: authentication-failed.
*******************************************************************************/
public static String failure(String code)
{
return ("automatic-action/MDN-sent-automatically; processed/error: " + code);
}
/*******************************************************************************
** Normalize an AS2 identifier or Message-ID: trim, drop angle brackets and
** quotes.
*******************************************************************************/
public static String normalizeId(String value)
{
if(value == null)
{
return (null);
}
String id = value.trim();
while(id.length() >= 2 && ((id.startsWith("<") && id.endsWith(">")) || (id.startsWith("\"") && id.endsWith("\""))))
{
id = id.substring(1, id.length() - 1).trim();
}
return (id);
}
/*******************************************************************************
** Keep only characters that are safe to show and store from a remote string.
*******************************************************************************/
public static String sanitize(String value)
{
if(value == null)
{
return (null);
}
String clean = value.replaceAll("[^A-Za-z0-9 /;:._,=+@<>-]", "");
return (clean.length() <= 200 ? clean : clean.substring(0, 200));
}
/*******************************************************************************
** Build a receipt. The MIC is included for processed messages; a signing key
** and algorithm produce a signed receipt.
*******************************************************************************/
public static Built build(String messageId, String ourId, String originalMessageId, String disposition, MIC mic, LocalKey key, ECryptoAlgorithmSign algorithm) throws As2Exception
{
try
{
String fields = "Reporting-UA: qbit-edi\r\nOriginal-Recipient: rfc822; " + ourId + "\r\nFinal-Recipient: rfc822; " + ourId
+ "\r\nOriginal-Message-ID: <" + originalMessageId + ">\r\nDisposition: " + disposition + "\r\n"
+ (mic == null ? "" : "Received-Content-MIC: " + mic.getAsAS2String() + "\r\n");
MimeMultipart report = new MimeMultipart("report; report-type=disposition-notification");
MimeBodyPart text = new MimeBodyPart();
text.setText("The AS2 message " + originalMessageId + " was processed with disposition: " + disposition + "\r\n", "us-ascii");
text.setHeader("Content-Transfer-Encoding", "7bit");
report.addBodyPart(text);
InternetHeaders headers = new InternetHeaders();
headers.addHeader("Content-Type", "message/disposition-notification");
headers.addHeader("Content-Transfer-Encoding", "7bit");
report.addBodyPart(new MimeBodyPart(headers, fields.getBytes(StandardCharsets.US_ASCII)));
MimeBodyPart outer = new MimeBodyPart();
outer.setContent(report);
outer.setHeader("Content-Type", report.getContentType());
MimeBodyPart result = key == null ? outer : As2Crypto.sign(outer, key, algorithm);
byte[] entity = As2Crypto.toBytes(result);
return (new Built(messageId, As2Crypto.splitRaw(entity), entity, key != null));
}
catch(As2Exception e)
{
throw (e);
}
catch(Exception e)
{
throw (new As2Exception("The MDN could not be built", As2Exception.UNEXPECTED_PROCESSING));
}
}
/*******************************************************************************
** Parse a receipt. A signature is verified against the candidate certificates
** and reported in signatureValid; an invalid one still yields the readable
** content so the failure can be recorded, but callers must not trust it.
*******************************************************************************/
public static Parsed parse(String contentType, String transferEncoding, byte[] body, List<X509Certificate> certificates, Date now) throws As2Exception
{
MimeBodyPart part = As2Crypto.fromHttp(contentType, transferEncoding, null, body);
boolean signed = false;
Boolean valid = null;
try
{
MimeBodyPart content = part;
if(part.isMimeType("multipart/signed"))
{
signed = true;
try
{
content = As2Crypto.verify(part, certificates, now);
valid = true;
}
catch(As2Exception e)
{
valid = false;
content = (MimeBodyPart) new MimeMultipart(new ByteArrayDataSource(part.getRawInputStream().readAllBytes(), MimeUtility.unfold(part.getContentType()))).getBodyPart(0);
}
}
if(!content.isMimeType("multipart/report"))
{
throw (new As2Exception("The response is not a message disposition notification"));
}
MimeMultipart report = new MimeMultipart(new ByteArrayDataSource(content.getRawInputStream().readAllBytes(), MimeUtility.unfold(content.getContentType())));
for(int i = 0; i < report.getCount(); i++)
{
if(report.getBodyPart(i).isMimeType("message/disposition-notification"))
{
InternetHeaders fields = new InternetHeaders(new ByteArrayInputStream(report.getBodyPart(i).getInputStream().readAllBytes()));
return (fromFields(fields, signed, valid));
}
}
throw (new As2Exception("The response has no disposition notification"));
}
catch(As2Exception e)
{
throw (e);
}
catch(Exception e)
{
throw (new As2Exception("The response is not a readable message disposition notification"));
}
}
/*******************************************************************************
** Read the notification fields.
*******************************************************************************/
private static Parsed fromFields(InternetHeaders fields, boolean signed, Boolean valid)
{
String original = normalizeId(first(fields, "Original-Message-ID"));
String disposition = sanitize(first(fields, "Disposition"));
String micText = sanitize(first(fields, "Received-Content-MIC"));
MIC mic = null;
try
{
mic = MIC.parse(micText);
}
catch(Exception e)
{
// An unreadable MIC is reported as a mismatch by the caller.
mic = null;
}
return (new Parsed(original == null ? null : sanitize(original), disposition, micText, mic, signed, valid));
}
/*******************************************************************************
** First value of a notification field, unfolded, or null.
*******************************************************************************/
private static String first(InternetHeaders fields, String name)
{
String[] values = fields.getHeader(name);
return (values == null || values.length == 0 ? null : MimeUtility.unfold(values[0]).trim());
}
}
@@ -0,0 +1,163 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.security.cert.X509Certificate;
import java.util.Date;
import java.util.List;
import java.util.Map;
import com.helger.as2lib.crypto.MIC;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.flow.EdiJournal;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Receives asynchronous MDNs for messages we sent: parses the receipt, verifies
** its signature against the partner certificates, matches Original-Message-ID to
** an outbound message, compares the MIC, then marks the message DELIVERED or
** FAILED, stores the raw MDN, writes a timeline event and resolves the overdue
** alert. Callers run inside the system context.
*******************************************************************************/
public class As2MdnReceiver
{
private static final QLogger LOG = QLogger.getLogger(As2MdnReceiver.class);
/*******************************************************************************
** Handle one inbound MDN; header names are lower case. A receipt that cannot be
** authenticated changes nothing.
*******************************************************************************/
public As2Receiver.Result receive(EdiTransport transport, Map<String, String> headers, byte[] body) throws Exception
{
String from = As2Mdn.normalizeId(headers.get("as2-from"));
String to = As2Mdn.normalizeId(headers.get("as2-to"));
if(from == null || to == null || !from.equalsIgnoreCase(transport.getAs2PartnerId()) || !to.equalsIgnoreCase(transport.getAs2LocalId()))
{
LOG.warn("AS2 MDN refused for transport " + transport.getName() + ": unknown AS2 identity");
return (As2Receiver.Result.text(403, "Unknown AS2 identity"));
}
As2Mdn.Parsed mdn;
try
{
List<X509Certificate> certificates = transport.getPartnerCertificateId() == null ? List.of() : new CertificateService().partnerVerificationCertificates(transport);
mdn = As2Mdn.parse(headers.get("content-type"), headers.get("content-transfer-encoding"), body, certificates, Date.from(EdiRuntime.current().config().getClock().instant()));
}
catch(As2Exception e)
{
return (As2Receiver.Result.text(400, "The MDN could not be read"));
}
if((Boolean.TRUE.equals(mdn.signed()) && !Boolean.TRUE.equals(mdn.signatureValid())) || (Boolean.TRUE.equals(transport.getMdnSigned()) && !Boolean.TRUE.equals(mdn.signed())))
{
LOG.warn("AS2 MDN refused for transport " + transport.getName() + ": signature check failed");
return (As2Receiver.Result.text(401, "The MDN signature could not be verified"));
}
List<QRecord> messages = mdn.originalMessageId() == null ? List.of() : EdiRecords.query("edi_outbound_message", EdiRecords.equal("transportMessageId", mdn.originalMessageId())
.withCriteria("transportId", QCriteriaOperator.EQUALS, transport.getId()));
if(messages.isEmpty())
{
return (As2Receiver.Result.text(404, "No message was sent with that Original-Message-ID"));
}
String mdnId = As2Mdn.sanitize(As2Mdn.normalizeId(headers.get("message-id")));
if(mdnId != null && !EdiRecords.query(As2Store.TABLE, EdiRecords.equal("direction", "INBOUND").withCriteria("messageId", QCriteriaOperator.EQUALS, mdnId)
.withCriteria("originalMessageId", QCriteriaOperator.EQUALS, mdn.originalMessageId())).isEmpty())
{
return (new As2Receiver.Result(200, Map.of(), new byte[0]));
}
settle(transport, messages.getFirst(), mdn, mdnId, headers, body);
return (new As2Receiver.Result(200, Map.of(), new byte[0]));
}
/*******************************************************************************
** Record the receipt and settle the message it answers.
*******************************************************************************/
private void settle(EdiTransport transport, QRecord message, As2Mdn.Parsed mdn, String mdnId, Map<String, String> headers, byte[] body) throws Exception
{
MIC ours = null;
try
{
ours = MIC.parse(message.getValueString("transportMic"));
}
catch(Exception e)
{
LOG.warn("The MIC stored for outbound message " + message.getValue("id") + " could not be read");
}
String failure = As2Transport.judge(transport, mdn, ours);
byte[] raw = As2Crypto.entity(headers.get("content-type"), headers.get("content-transfer-encoding"), null, body);
SystemWrites.transaction(() ->
{
PayloadRef ref = As2Store.storeRaw(raw);
As2Store.insert(new QRecord().withValue("direction", "INBOUND").withValue("messageId", mdnId).withValue("originalMessageId", mdn.originalMessageId())
.withValue("disposition", mdn.disposition()).withValue("micAlgorithm", mdn.mic() == null ? null : mdn.mic().getDigestAlgorithm().getID())
.withValue("micValue", As2Store.micValue(mdn.mic())).withValue("signed", mdn.signed()).withValue("signatureValid", mdn.signatureValid()).withValue("payloadRef", ref.value())
.withValue("transportId", transport.getId()).withValue("partnerId", message.getValue("partnerId")).withValue("outboundMessageId", message.getValue("id")).withValue("deliveryStatus", "NONE"));
Integer interchangeId = message.getValueInteger("interchangeId");
EdiJournal.event(null, interchangeId, null, "MDN");
if(SystemWrites.guard("UPDATE edi_outbound_message SET id=id WHERE id=? AND status='SENT'", message.getValue("id")))
{
var now = EdiRuntime.current().config().getClock().instant();
String status = failure == null ? "DELIVERED" : "FAILED";
QRecord update = new QRecord().withValue("id", message.getValue("id")).withValue("status", status).withValue("lastError", failure == null ? null : truncate(failure));
if(failure == null)
{
update.withValue("deliveredAt", now);
}
SystemWrites.update("edi_outbound_message", update);
if(interchangeId != null)
{
SystemWrites.update("edi_interchange", new QRecord().withValue("id", interchangeId).withValue("status", status));
}
EdiJournal.event(null, interchangeId, null, status);
if(failure != null)
{
EdiJournal.exception(null, interchangeId, null, message.getValueInteger("partnerId"), "TRANSPORT", "Outbound delivery failed");
QAlerts.raise(new AlertSignal("edi.send.failed").withSeverity(AlertSeverity.ERROR).withDedupKey("edi_outbound_message:" + message.getValue("id"))
.withTitle("EDI delivery failed").withSummary("Review the outbound message"));
}
}
return (null);
});
QAlerts.resolve("edi.mdn.overdue", "edi_outbound_message:" + message.getValue("id"), "MDN received");
}
/*******************************************************************************
** Keep an error within its column.
*******************************************************************************/
private static String truncate(String text)
{
return (text.length() <= 255 ? text : text.substring(0, 255));
}
}
@@ -0,0 +1,158 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.time.Duration;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.flow.OutboundWorker;
import com.kingsrook.qbits.edi.flow.SendWorker;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Delivers the asynchronous MDNs we owe partners. The edi_as2_mdn row is the
** queue entry (deliveryStatus PENDING): a scheduled step posts the stored MDN to
** the address the partner gave in Receipt-Delivery-Option, retries with the send
** backoff, and fails (with an alert) after the transport maxAttempts. Row locks
** fence concurrent senders.
*******************************************************************************/
public class As2MdnSender
{
private static final QLogger LOG = QLogger.getLogger(As2MdnSender.class);
/*******************************************************************************
** Deliver pending MDNs that are due; returns how many were attempted.
*******************************************************************************/
public Integer runOnce() throws QException
{
return (SystemWrites.call(() ->
{
int count = 0;
var due = new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR).withCriteria("nextAttemptAt", QCriteriaOperator.IS_BLANK)
.withCriteria("nextAttemptAt", QCriteriaOperator.LESS_THAN_OR_EQUALS, EdiRuntime.current().config().getClock().instant());
for(QRecord row : EdiRecords.query(As2Store.TABLE, new QQueryFilter().withCriteria("deliveryStatus", QCriteriaOperator.EQUALS, "PENDING").withSubFilter(due).withLimit(20)))
{
count += deliver(row);
}
return (count);
}));
}
/*******************************************************************************
** One delivery attempt, fenced by a row lock on the selected state.
*******************************************************************************/
private Integer deliver(QRecord selected) throws QException
{
return (SystemWrites.transaction(() ->
{
if(!SystemWrites.guard("UPDATE edi_as2_mdn SET id=id WHERE id=? AND delivery_status='PENDING' AND COALESCE(delivery_attempts,0)=?", selected.getValue("id"),
Objects.requireNonNullElse(selected.getValueInteger("deliveryAttempts"), 0)))
{
return (0);
}
QRecord row = OutboundWorker.row(As2Store.TABLE, selected.getValue("id"));
EdiTransport transport = OutboundWorker.row("edi_transport", row.getValue("transportId")).toEntity(EdiTransport.class);
int attempts = Objects.requireNonNullElse(row.getValueInteger("deliveryAttempts"), 0) + 1;
boolean delivered = post(row, transport);
int maximum = Objects.requireNonNullElse(transport.getMaxAttempts(), 8);
Instant now = EdiRuntime.current().config().getClock().instant();
String status = delivered ? "SENT" : attempts >= maximum ? "FAILED" : "PENDING";
QRecord update = new QRecord().withValue("id", row.getValue("id")).withValue("deliveryStatus", status).withValue("deliveryAttempts", attempts)
.withValue("nextAttemptAt", "PENDING".equals(status) ? now.plusSeconds(SendWorker.BACKOFF.get(Math.min(attempts - 1, SendWorker.BACKOFF.size() - 1))) : null);
if(delivered)
{
update.withValue("deliveredAt", now);
}
SystemWrites.update(As2Store.TABLE, update);
if("FAILED".equals(status))
{
QAlerts.raise(new AlertSignal("edi.send.failed").withSeverity(AlertSeverity.ERROR).withDedupKey("edi_as2_mdn:" + row.getValue("id"))
.withTitle("AS2 receipt delivery failed").withSummary("Review the AS2 receipt"));
}
return (1);
}));
}
/*******************************************************************************
** POST the stored MDN to the partner receipt address; true on any 2xx.
*******************************************************************************/
private boolean post(QRecord row, EdiTransport transport)
{
try
{
byte[] raw;
try(InputStream stream = EdiRuntime.current().payloads().open(new PayloadRef(row.getValueString("payloadRef"), null, 0)))
{
raw = stream.readAllBytes();
}
As2Crypto.Wire wire = As2Crypto.splitRaw(raw);
Map<String, String> headers = new LinkedHashMap<>();
headers.put("AS2-Version", "1.2");
headers.put("AS2-From", As2Transport.asId(transport.getAs2LocalId()));
headers.put("AS2-To", As2Transport.asId(transport.getAs2PartnerId()));
headers.put("Message-ID", "<" + row.getValueString("messageId") + ">");
headers.put("MIME-Version", "1.0");
headers.put("Content-Type", wire.contentType());
if(wire.transferEncoding() != null)
{
headers.put("Content-Transfer-Encoding", wire.transferEncoding());
}
if(transport.getHttpAuthUserEnvVar() != null)
{
var secrets = EdiRuntime.current().config().getSecretsResolver();
headers.put("Authorization", As2Http.basic(secrets.resolve(transport.getHttpAuthUserEnvVar()), secrets.resolve(transport.getHttpAuthPasswordEnvVar())));
}
HttpClient client = As2Http.client(null);
int status = As2Http.post(client, URI.create(row.getValueString("deliveryUrl")), headers, wire.body(), Duration.ofSeconds(60)).status();
return (status >= 200 && status < 300);
}
catch(Exception e)
{
LOG.warn("AS2 receipt delivery attempt failed for receipt " + row.getValue("id"));
return (false);
}
}
}
@@ -0,0 +1,356 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.security.cert.X509Certificate;
import java.time.Instant;
import java.util.Date;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import com.helger.as2lib.crypto.ECryptoAlgorithmSign;
import com.helger.as2lib.crypto.MIC;
import com.helger.as2lib.disposition.DispositionOptions;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.as2.CertificateService.LocalKey;
import com.kingsrook.qbits.edi.flow.EdiJournal;
import com.kingsrook.qbits.edi.flow.InboundPayload;
import com.kingsrook.qbits.edi.flow.InboundSink;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import jakarta.mail.internet.MimeBodyPart;
/*******************************************************************************
** Receives AS2 messages for one transport: checks the AS2 identities, decrypts
** with our ACTIVE or NEXT keys, verifies the signature against the partner
** certificates, computes the MIC, hands the document to the inbox, and answers
** with a synchronous MDN or queues an asynchronous one. A retransmitted
** Message-ID gets the original MDN again and is not reprocessed. Callers run
** inside the system context; failures are reported as RFC 4130 dispositions.
*******************************************************************************/
public class As2Receiver
{
private static final QLogger LOG = QLogger.getLogger(As2Receiver.class);
/*******************************************************************************
** An HTTP answer: status, headers and body.
*******************************************************************************/
public record Result(int status, Map<String, String> headers, byte[] body)
{
/*******************************************************************************
** A plain-text answer.
*******************************************************************************/
static Result text(int status, String text)
{
return (new Result(status, Map.of("Content-Type", "text/plain; charset=utf-8"), text.getBytes(StandardCharsets.UTF_8)));
}
}
/*******************************************************************************
** What was asked of us for this message.
*******************************************************************************/
private record Request(String messageId, boolean mdnRequested, String deliveryUrl, boolean signMdn, ECryptoAlgorithmSign micAlgorithm)
{
/*******************************************************************************
** The same request, answered in the HTTP response only. Used before the sender
** is authenticated, so an unauthenticated sender cannot aim a receipt at an
** address of its choosing.
*******************************************************************************/
Request synchronous()
{
return (new Request(messageId, mdnRequested, null, signMdn, micAlgorithm));
}
}
/*******************************************************************************
** Handle one inbound message; header names are lower case.
*******************************************************************************/
public Result receive(EdiTransport transport, Map<String, String> headers, byte[] body, String remoteAddress) throws Exception
{
String messageId = As2Mdn.normalizeId(headers.get("message-id"));
String from = As2Mdn.normalizeId(headers.get("as2-from"));
String to = As2Mdn.normalizeId(headers.get("as2-to"));
if(messageId == null || messageId.isBlank() || messageId.length() > 250 || !messageId.matches("[\\x21-\\x7E]+") || from == null || to == null)
{
return (Result.text(400, "The AS2-From, AS2-To and Message-ID headers are required"));
}
if(!from.equalsIgnoreCase(transport.getAs2PartnerId()) || !to.equalsIgnoreCase(transport.getAs2LocalId()))
{
LOG.warn("AS2 message refused for transport " + transport.getName() + ": unknown AS2 identity");
return (Result.text(403, "Unknown AS2 identity"));
}
Request request = request(transport, headers, messageId);
if(request.deliveryUrl() != null && !deliveryUrlAllowed(transport, request.deliveryUrl()))
{
return (Result.text(400, "The Receipt-Delivery-Option is not acceptable"));
}
CertificateService certificates = new CertificateService();
CertificateService.KeySet keys = certificates.localDecryptionKeys();
As2Crypto.Decoded decoded;
try
{
List<X509Certificate> partnerCertificates = transport.getPartnerCertificateId() == null ? List.of() : certificates.partnerVerificationCertificates(transport);
byte[] top = As2Crypto.entity(headers.get("content-type"), headers.get("content-transfer-encoding"), headers.get("content-disposition"), body);
decoded = As2Crypto.decode(top, keys.keys(), partnerCertificates, request.micAlgorithm(), Date.from(EdiRuntime.current().config().getClock().instant()));
if((transport.getSignAlgorithm() != null && !decoded.signed()) || (transport.getEncryptAlgorithm() != null && !decoded.encrypted()))
{
throw (new As2Exception("The message does not meet the transport security requirements", As2Exception.INSUFFICIENT_SECURITY));
}
}
catch(As2Exception e)
{
boolean ourFault = e.getCode() == null || (As2Exception.DECRYPTION_FAILED.equals(e.getCode()) && !keys.problems().isEmpty());
return (reject(transport, request.synchronous(), ourFault ? As2Exception.UNEXPECTED_PROCESSING : e.getCode(), certificates));
}
String externalId = externalId(transport, messageId);
QRecord replay = alreadyAnswered(transport, request, externalId);
if(replay != null)
{
return (replay(transport, request, replay));
}
QRecord inbox;
try
{
inbox = new InboundSink(EdiRuntime.current().payloads()).receive(new InboundPayload("AS2", externalId, decoded.fileName(), remoteAddress, new ByteArrayInputStream(decoded.payload())));
if(inbox.getValue("transportId") == null)
{
SystemWrites.update("edi_inbox", new QRecord().withValue("id", inbox.getValue("id")).withValue("transportId", transport.getId()));
}
}
catch(Exception e)
{
LOG.warn("AS2 intake failed for transport " + transport.getName());
return (reject(transport, request, As2Exception.UNEXPECTED_PROCESSING, certificates));
}
if(!request.mdnRequested())
{
return (new Result(200, Map.of(), new byte[0]));
}
try
{
return (answer(transport, request, As2Mdn.PROCESSED, decoded.mic(), inbox.getValueInteger("id"), certificates));
}
catch(As2Exception e)
{
LOG.warn("AS2 receipt could not be produced for transport " + transport.getName() + ": " + e.getMessage());
return (Result.text(500, "The receipt could not be produced"));
}
}
/*******************************************************************************
** Read what the message asks for: whether a receipt is wanted, where an
** asynchronous one goes, and whether and how to sign it.
*******************************************************************************/
private Request request(EdiTransport transport, Map<String, String> headers, String messageId)
{
DispositionOptions options = new DispositionOptions();
try
{
options = DispositionOptions.createFromString(headers.get("disposition-notification-options"));
}
catch(Exception e)
{
LOG.warn("AS2 disposition options ignored for transport " + transport.getName());
}
ECryptoAlgorithmSign algorithm = options.getFirstMICAlg();
if(algorithm == null)
{
algorithm = As2Crypto.signAlgorithm(transport.getSignAlgorithm()) == null ? ECryptoAlgorithmSign.DIGEST_SHA_256 : As2Crypto.signAlgorithm(transport.getSignAlgorithm());
}
String delivery = headers.get("receipt-delivery-option");
return (new Request(messageId, headers.get("disposition-notification-to") != null, delivery == null || delivery.isBlank() ? null : delivery.trim(),
options.getProtocol() != null || Boolean.TRUE.equals(transport.getMdnSigned()), algorithm));
}
/*******************************************************************************
** An asynchronous receipt address must use TLS (loopback excepted) and, unless
** the transport requires signed messages, must be on the partner's own host,
** so an unauthenticated sender cannot aim our receipts at third parties.
*******************************************************************************/
static boolean deliveryUrlAllowed(EdiTransport transport, String url)
{
if(!As2Http.isAllowedUrl(url))
{
return (false);
}
if(transport.getSignAlgorithm() != null)
{
return (true);
}
try
{
return (transport.getAs2Url() != null && java.net.URI.create(url).getHost().equalsIgnoreCase(java.net.URI.create(transport.getAs2Url()).getHost()));
}
catch(Exception e)
{
return (false);
}
}
/*******************************************************************************
** The inbox dedupe key: the Message-ID. Message-IDs are unique per sender, so if
** the same Message-ID was already received on a different transport (another
** partner), the key is scoped by the transport id so neither message is lost.
*******************************************************************************/
private static String externalId(EdiTransport transport, String messageId) throws Exception
{
List<QRecord> rows = EdiRecords.query("edi_inbox", EdiRecords.equal("transportType", "AS2").withCriteria("externalId", QCriteriaOperator.EQUALS, messageId));
boolean foreign = !rows.isEmpty() && rows.getFirst().getValue("transportId") != null && !rows.getFirst().getValue("transportId").equals(transport.getId());
return (foreign ? transport.getId() + ":" + messageId : messageId);
}
/*******************************************************************************
** The receipt row we already sent for this Message-ID (a retransmission), or
** null when the original was never answered.
*******************************************************************************/
private QRecord alreadyAnswered(EdiTransport transport, Request request, String externalId) throws Exception
{
if(EdiRecords.query("edi_inbox", EdiRecords.equal("transportType", "AS2").withCriteria("externalId", QCriteriaOperator.EQUALS, externalId)).isEmpty())
{
return (null);
}
List<QRecord> rows = EdiRecords.query(As2Store.TABLE, EdiRecords.equal("direction", "OUTBOUND").withCriteria("originalMessageId", QCriteriaOperator.EQUALS, request.messageId())
.withCriteria("transportId", QCriteriaOperator.EQUALS, transport.getId()));
return (rows.isEmpty() ? null : rows.getFirst());
}
/*******************************************************************************
** Answer a retransmission with the original receipt, unchanged. An
** asynchronous receipt is queued to be delivered again.
*******************************************************************************/
private Result replay(EdiTransport transport, Request request, QRecord original) throws Exception
{
if(!request.mdnRequested())
{
return (new Result(200, Map.of(), new byte[0]));
}
if(request.deliveryUrl() != null)
{
SystemWrites.update(As2Store.TABLE, new QRecord().withValue("id", original.getValue("id")).withValue("deliveryUrl", request.deliveryUrl()).withValue("deliveryStatus", "PENDING")
.withValue("deliveryAttempts", 0).withValue("nextAttemptAt", null));
return (new Result(200, Map.of(), new byte[0]));
}
byte[] raw;
try(InputStream stream = EdiRuntime.current().payloads().open(new PayloadRef(original.getValueString("payloadRef"), null, 0)))
{
raw = stream.readAllBytes();
}
return (syncResult(transport, original.getValueString("messageId"), As2Crypto.splitRaw(raw)));
}
/*******************************************************************************
** Reject a message that could not be accepted. With a receipt requested the
** failure is reported in an MDN so the partner can act or retry; otherwise an
** HTTP error answers. No inbox row exists for a rejected message.
*******************************************************************************/
private Result reject(EdiTransport transport, Request request, String code, CertificateService certificates) throws Exception
{
LOG.warn("AS2 message rejected for transport " + transport.getName() + ": " + code);
if(!request.mdnRequested())
{
return (Result.text(As2Exception.UNEXPECTED_PROCESSING.equals(code) ? 500 : 400, "AS2 message rejected: " + code));
}
try
{
return (answer(transport, request, As2Mdn.failure(code), null, null, certificates));
}
catch(As2Exception e)
{
LOG.warn("AS2 receipt could not be produced for transport " + transport.getName() + ": " + e.getMessage());
return (Result.text(500, "The receipt could not be produced"));
}
}
/*******************************************************************************
** Build, store and answer or queue the MDN.
*******************************************************************************/
private Result answer(EdiTransport transport, Request request, String disposition, MIC mic, Integer inboxId, CertificateService certificates) throws Exception
{
LocalKey key = request.signMdn() ? certificates.activeLocal(transport) : null;
String mdnId = "edi-mdn-" + UUID.randomUUID() + "@" + transport.getAs2LocalId().replaceAll("[^A-Za-z0-9.-]", "-");
As2Mdn.Built built = As2Mdn.build(mdnId, transport.getAs2LocalId(), request.messageId(), disposition, mic, key, request.micAlgorithm());
PayloadRef raw = As2Store.storeRaw(built.entity());
Instant now = EdiRuntime.current().config().getClock().instant();
QRecord row = As2Store.insert(new QRecord().withValue("direction", "OUTBOUND").withValue("messageId", mdnId).withValue("originalMessageId", request.messageId())
.withValue("disposition", disposition).withValue("micAlgorithm", mic == null ? null : mic.getDigestAlgorithm().getID()).withValue("micValue", As2Store.micValue(mic))
.withValue("signed", built.signed()).withValue("payloadRef", raw.value()).withValue("receivedAt", now).withValue("transportId", transport.getId())
.withValue("partnerId", transport.getPartnerId()).withValue("inboxId", inboxId).withValue("deliveryStatus", request.deliveryUrl() == null ? "NONE" : "PENDING")
.withValue("deliveryUrl", request.deliveryUrl()).withValue("deliveryAttempts", 0).withValue("nextAttemptAt", null));
if(inboxId != null)
{
EdiJournal.event(inboxId, null, null, "MDN");
}
LOG.info("AS2 receipt " + row.getValue("id") + " " + (request.deliveryUrl() == null ? "returned" : "queued") + " for transport " + transport.getName());
return (request.deliveryUrl() != null ? new Result(200, Map.of(), new byte[0]) : syncResult(transport, mdnId, built.wire()));
}
/*******************************************************************************
** The synchronous MDN response with its AS2 headers.
*******************************************************************************/
private Result syncResult(EdiTransport transport, String mdnId, As2Crypto.Wire wire)
{
Map<String, String> headers = new LinkedHashMap<>();
headers.put("AS2-Version", "1.2");
headers.put("AS2-From", As2Transport.asId(transport.getAs2LocalId()));
headers.put("AS2-To", As2Transport.asId(transport.getAs2PartnerId()));
headers.put("Message-ID", "<" + mdnId + ">");
headers.put("MIME-Version", "1.0");
headers.put("Content-Type", wire.contentType());
if(wire.transferEncoding() != null)
{
headers.put("Content-Transfer-Encoding", wire.transferEncoding());
}
return (new Result(200, headers, wire.body()));
}
}
@@ -0,0 +1,148 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.middleware.javalin.QJavalinRouteProviderInterface;
import io.javalin.config.JavalinConfig;
import io.javalin.http.Context;
/*******************************************************************************
** AS2 machine endpoints outside the UI API, authenticated per transport by AS2
** identity, S/MIME signature and certificates (never a user session):
** POST /qqq/hooks/edi/as2/{transport} receives messages and answers with the
** MDN, and POST /qqq/hooks/edi/as2/{transport}/mdn receives asynchronous MDNs.
*******************************************************************************/
public class As2RouteProvider implements QJavalinRouteProviderInterface
{
private static final QLogger LOG = QLogger.getLogger(As2RouteProvider.class);
public static final long DEFAULT_MAX_BODY_BYTES = 50L * 1024 * 1024;
public static final long ABSOLUTE_MAX_BODY_BYTES = 200L * 1024 * 1024;
private QInstance instance;
/*******************************************************************************
** Host instance supplied by Javalin.
*******************************************************************************/
public void setQInstance(QInstance instance)
{
this.instance = instance;
}
/*******************************************************************************
** Register the non-UI hooks.
*******************************************************************************/
public void acceptJavalinConfig(JavalinConfig config)
{
config.routes.post("/qqq/hooks/edi/as2/{transport}", context -> handle(context, false));
config.routes.post("/qqq/hooks/edi/as2/{transport}/mdn", context -> handle(context, true));
}
/*******************************************************************************
** Find the active AS2 transport, read the bounded body and delegate.
*******************************************************************************/
private void handle(Context context, boolean mdn)
{
var previous = QContext.capture();
try
{
QContext.init(instance, null);
As2Receiver.Result result = SystemWrites.call(() -> process(context, mdn));
context.status(result.status());
result.headers().forEach(context::header);
context.result(result.body());
}
catch(Exception e)
{
LOG.warn("AS2 hook failed: " + e.getClass().getSimpleName());
context.status(503);
context.result("The AS2 request could not be processed".getBytes(java.nio.charset.StandardCharsets.UTF_8));
}
finally
{
QContext.init(previous);
}
}
/*******************************************************************************
** The hook body, inside the system context.
*******************************************************************************/
private As2Receiver.Result process(Context context, boolean mdn) throws Exception
{
List<QRecord> rows = EdiRecords.query("edi_transport", EdiRecords.equal("name", context.pathParam("transport")).withCriteria("type", QCriteriaOperator.EQUALS, "AS2")
.withCriteria("isActive", QCriteriaOperator.EQUALS, true));
if(rows.size() != 1 || (mdn ? "INBOUND" : "OUTBOUND").equals(rows.getFirst().getValueString("direction")))
{
return (As2Receiver.Result.text(404, "Unknown AS2 transport"));
}
EdiTransport transport = rows.getFirst().toEntity(EdiTransport.class);
long limit = mdn ? As2Http.MAX_RESPONSE_BYTES : maxBodyBytes(transport);
if(context.req().getContentLengthLong() > limit)
{
return (As2Receiver.Result.text(413, "The message is too large"));
}
byte[] body = context.req().getInputStream().readNBytes((int) limit + 1);
if(body.length > limit)
{
return (As2Receiver.Result.text(413, "The message is too large"));
}
Map<String, String> headers = new TreeMap<>();
context.headerMap().forEach((name, value) -> headers.put(name.toLowerCase(Locale.ROOT), value));
return (mdn ? new As2MdnReceiver().receive(transport, headers, body) : new As2Receiver().receive(transport, headers, body, context.ip()));
}
/*******************************************************************************
** The largest interchange any agreement on this transport allows, or 50 MB.
*******************************************************************************/
private static long maxBodyBytes(EdiTransport transport) throws Exception
{
long limit = 0;
for(QRecord agreement : EdiRecords.query("edi_agreement", EdiRecords.equal("inboundTransportId", transport.getId())))
{
limit = Math.max(limit, agreement.getValueLong("maxInterchangeBytes") == null ? 0 : agreement.getValueLong("maxInterchangeBytes"));
}
return (limit <= 0 ? DEFAULT_MAX_BODY_BYTES : Math.min(limit + limit / 2 + 65536, ABSOLUTE_MAX_BODY_BYTES));
}
}
@@ -0,0 +1,86 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.ByteArrayInputStream;
import java.time.Instant;
import com.helger.as2lib.crypto.MIC;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.PayloadKind;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Persistence helpers for AS2 receipts: the raw MDN goes to the encrypted payload
** store and a read-only edi_as2_mdn row describes it. Callers run inside the
** system context.
*******************************************************************************/
public final class As2Store
{
public static final String TABLE = "edi_as2_mdn";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private As2Store()
{
}
/*******************************************************************************
** Store the raw MDN entity (headers and body) encrypted.
*******************************************************************************/
public static PayloadRef storeRaw(byte[] entity) throws Exception
{
return (EdiRuntime.current().payloads().put(PayloadKind.MDN, new ByteArrayInputStream(entity)));
}
/*******************************************************************************
** The base64 value of a MIC without its algorithm.
*******************************************************************************/
public static String micValue(MIC mic)
{
return (mic == null ? null : java.util.Base64.getEncoder().encodeToString(mic.micBytes()));
}
/*******************************************************************************
** Insert a receipt row and return it.
*******************************************************************************/
public static QRecord insert(QRecord row) throws QException
{
if(row.getValue("receivedAt") == null)
{
row.setValue("receivedAt", Instant.now(EdiRuntime.current().config().getClock()));
}
return (SystemWrites.insert(TABLE, row));
}
}
@@ -0,0 +1,584 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.security.MessageDigest;
import java.security.cert.X509Certificate;
import java.time.Duration;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Date;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.helger.as2lib.crypto.ECryptoAlgorithmCrypt;
import com.helger.as2lib.crypto.ECryptoAlgorithmSign;
import com.helger.as2lib.crypto.MIC;
import com.helger.as2lib.disposition.DispositionOptions;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.as2.CertificateService.LocalKey;
import com.kingsrook.qbits.edi.flow.EdiJournal;
import com.kingsrook.qbits.edi.model.EdiOutboundMessage;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qbits.edi.transport.EdiSendResult;
import com.kingsrook.qbits.edi.transport.EdiTestResult;
import com.kingsrook.qbits.edi.transport.EdiTransportPlugin;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** AS2 (RFC 4130) transport. Sending builds the message (compress, sign, then
** encrypt), POSTs it, and for a synchronous receipt verifies the MDN signature,
** the received-content MIC and the disposition. An asynchronous receipt leaves
** the message SENT until the MDN arrives at the receipt hook. No local files are
** used; keys come from environment variables through the secrets resolver.
*******************************************************************************/
public class As2Transport implements EdiTransportPlugin
{
public static final String TYPE = "AS2";
private static final String ID_PATTERN = "[A-Za-z0-9 ._@:/-]{1,128}";
private static final Duration TIMEOUT = Duration.ofSeconds(60);
/*******************************************************************************
** The prepared wire message and the MIC computed for it.
*******************************************************************************/
private record Prepared(As2Crypto.Wire wire, MIC mic)
{
}
/*******************************************************************************
** Registry key.
*******************************************************************************/
@Override
public String getType()
{
return (TYPE);
}
/*******************************************************************************
** Check identifiers, URL, algorithms, receipt mode and certificate ownership
** without resolving any secret. SHA1 and 3DES are accepted only when set
** explicitly; they set the weakCrypto flag on the record.
*******************************************************************************/
@Override
public void validate(EdiTransport transport, List<String> errors)
{
boolean outbound = !"INBOUND".equals(transport.getDirection());
if(transport.getAs2LocalId() == null || !transport.getAs2LocalId().matches(ID_PATTERN))
{
errors.add("as2LocalId is required (letters, digits, spaces and . _ @ : / - only)");
}
if(transport.getAs2PartnerId() == null || !transport.getAs2PartnerId().matches(ID_PATTERN))
{
errors.add("as2PartnerId is required (letters, digits, spaces and . _ @ : / - only)");
}
if(outbound && (transport.getAs2Url() == null || !As2Http.isAllowedUrl(transport.getAs2Url())))
{
errors.add("as2Url must use TLS (HTTP is allowed on loopback only) and carry no credentials");
}
try
{
As2Crypto.signAlgorithm(transport.getSignAlgorithm());
}
catch(IllegalArgumentException e)
{
errors.add("Unsupported signAlgorithm");
}
try
{
As2Crypto.encryptAlgorithm(transport.getEncryptAlgorithm());
}
catch(IllegalArgumentException e)
{
errors.add("Unsupported encryptAlgorithm");
}
if(transport.getMdnMode() != null && !List.of("SYNC", "ASYNC", "NONE").contains(transport.getMdnMode()))
{
errors.add("Unsupported mdnMode");
}
if(Boolean.TRUE.equals(transport.getCompress()) && transport.getSignAlgorithm() == null && transport.getEncryptAlgorithm() == null)
{
errors.add("compress requires signing or encryption");
}
if(transport.getPartnerId() == null && !Boolean.TRUE.equals(transport.getIsVan()))
{
errors.add("partnerId is required unless the transport is a VAN (isVan)");
}
if((transport.getHttpAuthUserEnvVar() == null) != (transport.getHttpAuthPasswordEnvVar() == null))
{
errors.add("httpAuthUserEnvVar and httpAuthPasswordEnvVar must be set together");
}
if(outbound && "ASYNC".equals(transport.getMdnMode()) && baseUrl() == null)
{
errors.add("ASYNC receipts need the qbit publicBaseUrl configuration");
}
try
{
validateCertificates(transport, errors);
}
catch(Exception e)
{
errors.add("Certificates could not be checked");
}
}
/*******************************************************************************
** Certificate references must exist and have the right owner: ours (no
** partner, with a key variable) as the local certificate; the partner's own, or
** for a VAN a certificate that is not ours, as the partner certificate; a TLS
** certificate for pinning. Retired certificates are refused.
*******************************************************************************/
private static void validateCertificates(EdiTransport transport, List<String> errors) throws Exception
{
boolean secured = transport.getSignAlgorithm() != null || transport.getEncryptAlgorithm() != null || Boolean.TRUE.equals(transport.getMdnSigned());
QRecord local = certificate(transport.getLocalCertificateId());
QRecord partner = certificate(transport.getPartnerCertificateId());
QRecord tls = certificate(transport.getPartnerTlsCertificateId());
if(transport.getLocalCertificateId() == null ? secured : local == null)
{
errors.add(transport.getLocalCertificateId() == null ? "localCertificateId is required when signing or encrypting" : "localCertificateId does not exist");
}
else if(local != null && (local.getValue("partnerId") != null || local.getValueString("privateKeyEnvVar") == null || "RETIRED".equals(local.getValueString("status"))))
{
errors.add("localCertificateId must be one of our own certificates (no partner, with a private key variable) that is not retired");
}
if(transport.getPartnerCertificateId() == null ? secured : partner == null)
{
errors.add(transport.getPartnerCertificateId() == null ? "partnerCertificateId is required when signing or encrypting" : "partnerCertificateId does not exist");
}
else if(partner != null)
{
boolean ownedByPartner = transport.getPartnerId() == null || transport.getPartnerId().equals(partner.getValueInteger("partnerId"));
if(!ownedByPartner || partner.getValueString("privateKeyEnvVar") != null || "RETIRED".equals(partner.getValueString("status")))
{
errors.add("partnerCertificateId must be a certificate of the partner (or, for a VAN, of the VAN) that is not retired and holds no private key");
}
}
if(transport.getPartnerTlsCertificateId() != null && (tls == null || !"TLS".equals(tls.getValueString("usage"))))
{
errors.add("partnerTlsCertificateId must be a certificate with usage TLS");
}
}
/*******************************************************************************
** Certificate row by id, or null.
*******************************************************************************/
private static QRecord certificate(Integer id) throws Exception
{
if(id == null)
{
return (null);
}
return (SystemWrites.call(() -> EdiRecords.query(CertificateService.TABLE, EdiRecords.equal("id", id)).stream().findFirst().orElse(null)));
}
/*******************************************************************************
** The deployment public base URL without a trailing slash, or null.
*******************************************************************************/
static String baseUrl()
{
try
{
String url = EdiRuntime.current().config().getPublicBaseUrl();
return (url == null || url.isBlank() ? null : url.replaceAll("/+$", ""));
}
catch(Exception e)
{
return (null);
}
}
/*******************************************************************************
** Build, sign, encrypt and POST the message; settle it from the receipt.
*******************************************************************************/
@Override
public EdiSendResult send(EdiOutboundMessage message, PayloadRef payload, EdiTransport transport) throws Exception
{
return (SystemWrites.call(() -> deliver(message, payload, transport)));
}
/*******************************************************************************
** The send itself, inside the system context. Configuration problems (a
** missing variable, an expired or missing certificate) are RETRY results that
** name only the problem, so the retry policy and alerts apply once fixed.
*******************************************************************************/
private EdiSendResult deliver(EdiOutboundMessage message, PayloadRef payload, EdiTransport transport) throws Exception
{
byte[] edi;
try(InputStream stream = EdiRuntime.current().payloads().open(payload))
{
edi = stream.readAllBytes();
}
String messageId = messageId(message.getId(), transport, edi);
CertificateService certificates = new CertificateService();
String mode = mode(transport);
As2Http.Response response;
Prepared prepared;
try
{
prepared = prepare(message, transport, edi, messageId, certificates);
HttpClient client = As2Http.client(certificates.tlsPin(transport));
response = As2Http.post(client, URI.create(transport.getAs2Url()), headers(transport, messageId, prepared, mode), prepared.wire().body(), TIMEOUT);
}
catch(As2Exception e)
{
return (new EdiSendResult("RETRY", null, e.getMessage()));
}
catch(java.io.IOException e)
{
return (new EdiSendResult("RETRY", null, "The AS2 endpoint could not be reached"));
}
if(response.status() >= 500 || response.status() == 429)
{
return (new EdiSendResult("RETRY", null, "HTTP " + response.status()));
}
if(response.status() < 200 || response.status() >= 300)
{
return (new EdiSendResult("FAILED", null, "HTTP " + response.status()));
}
SystemWrites.update("edi_outbound_message", new QRecord().withValue("id", message.getId()).withValue("transportMic", prepared.mic().getAsAS2String()));
if(!"SYNC".equals(mode))
{
return (new EdiSendResult("SENT", messageId, "ASYNC".equals(mode) ? "Awaiting the asynchronous MDN" : "No receipt requested"));
}
return (settleSync(message, transport, messageId, prepared, response, certificates));
}
/*******************************************************************************
** Compress, sign and encrypt as configured, computing the MIC over the content
** that is signed.
*******************************************************************************/
private Prepared prepare(EdiOutboundMessage message, EdiTransport transport, byte[] edi, String messageId, CertificateService certificates) throws Exception
{
ECryptoAlgorithmSign signAlgorithm = As2Crypto.signAlgorithm(transport.getSignAlgorithm());
ECryptoAlgorithmCrypt encryptAlgorithm = As2Crypto.encryptAlgorithm(transport.getEncryptAlgorithm());
LocalKey key = signAlgorithm == null ? null : certificates.activeLocal(transport);
X509Certificate partner = encryptAlgorithm == null ? null : certificates.activePartnerCertificate(transport);
checkValid("Local certificate", key == null ? null : key.certificate());
checkValid("Partner certificate", partner);
String contentType = edi.length >= 3 && edi[0] == 'U' && edi[1] == 'N' ? "application/edifact" : "application/edi-x12";
var part = As2Crypto.payloadPart(edi, contentType, "edi-" + message.getId() + ".edi");
if(Boolean.TRUE.equals(transport.getCompress()))
{
part = As2Crypto.compress(part);
}
ECryptoAlgorithmSign micAlgorithm = signAlgorithm == null ? ECryptoAlgorithmSign.DIGEST_SHA_256 : signAlgorithm;
byte[] clear;
MIC mic;
if(signAlgorithm == null)
{
clear = As2Crypto.toBytes(part);
mic = As2Crypto.mic(clear, micAlgorithm);
}
else
{
clear = As2Crypto.toBytes(As2Crypto.sign(part, key, signAlgorithm));
As2Crypto.Wire signed = As2Crypto.splitRaw(clear);
mic = As2Crypto.mic(As2Crypto.firstPart(signed.contentType(), signed.body()), micAlgorithm);
}
return (new Prepared(encryptAlgorithm == null ? As2Crypto.splitRaw(clear) : As2Crypto.encrypt(clear, partner, encryptAlgorithm), mic));
}
/*******************************************************************************
** Certificates outside their validity period are not used.
*******************************************************************************/
private static void checkValid(String what, X509Certificate certificate) throws As2Exception
{
if(certificate != null)
{
try
{
certificate.checkValidity();
}
catch(Exception e)
{
throw (new As2Exception(what + " " + certificate.getSerialNumber().toString(16) + " is expired or not yet valid"));
}
}
}
/*******************************************************************************
** The AS2 request headers.
*******************************************************************************/
private Map<String, String> headers(EdiTransport transport, String messageId, Prepared prepared, String mode) throws Exception
{
Map<String, String> headers = new LinkedHashMap<>();
headers.put("AS2-Version", "1.2");
headers.put("AS2-From", asId(transport.getAs2LocalId()));
headers.put("AS2-To", asId(transport.getAs2PartnerId()));
headers.put("Message-ID", "<" + messageId + ">");
headers.put("Subject", subject(transport));
headers.put("Date", DateTimeFormatter.RFC_1123_DATE_TIME.format(ZonedDateTime.now(EdiRuntime.current().config().getClock().withZone(ZoneOffset.UTC))));
headers.put("MIME-Version", "1.0");
headers.put("User-Agent", "qbit-edi");
headers.put("Content-Type", prepared.wire().contentType());
if(prepared.wire().transferEncoding() != null)
{
headers.put("Content-Transfer-Encoding", prepared.wire().transferEncoding());
}
if(prepared.wire().contentDisposition() != null)
{
headers.put("Content-Disposition", prepared.wire().contentDisposition());
}
if(!"NONE".equals(mode))
{
headers.put("Disposition-Notification-To", transport.getAs2LocalId());
if(Boolean.TRUE.equals(transport.getMdnSigned()))
{
ECryptoAlgorithmSign alg = As2Crypto.signAlgorithm(transport.getSignAlgorithm()) == null ? ECryptoAlgorithmSign.DIGEST_SHA_256 : As2Crypto.signAlgorithm(transport.getSignAlgorithm());
headers.put("Disposition-Notification-Options", new DispositionOptions().setProtocolImportance(DispositionOptions.IMPORTANCE_REQUIRED)
.setProtocol(DispositionOptions.PROTOCOL_PKCS7_SIGNATURE).setMICAlgImportance(DispositionOptions.IMPORTANCE_REQUIRED).setMICAlg(alg).getAsString());
}
}
if("ASYNC".equals(mode))
{
headers.put("Receipt-Delivery-Option", baseUrl() + "/qqq/hooks/edi/as2/" + transport.getName() + "/mdn");
}
if(transport.getHttpAuthUserEnvVar() != null)
{
headers.put("Authorization", As2Http.basic(secret(transport.getHttpAuthUserEnvVar()), secret(transport.getHttpAuthPasswordEnvVar())));
}
return (headers);
}
/*******************************************************************************
** Evaluate a synchronous receipt: signature, disposition and MIC.
*******************************************************************************/
private EdiSendResult settleSync(EdiOutboundMessage message, EdiTransport transport, String messageId, Prepared prepared, As2Http.Response response, CertificateService certificates) throws Exception
{
String contentType = response.headers().get("content-type");
if(response.body().length == 0 || contentType == null)
{
return (new EdiSendResult("RETRY", messageId, "The partner returned no MDN"));
}
As2Mdn.Parsed mdn;
try
{
mdn = As2Mdn.parse(contentType, response.headers().get("content-transfer-encoding"), response.body(), certificates.partnerVerificationCertificates(transport),
Date.from(EdiRuntime.current().config().getClock().instant()));
}
catch(As2Exception e)
{
return (new EdiSendResult("RETRY", messageId, e.getMessage()));
}
String failure = judge(transport, mdn, prepared.mic());
PayloadRef raw = As2Store.storeRaw(As2Crypto.entity(contentType, response.headers().get("content-transfer-encoding"), null, response.body()));
As2Store.insert(new QRecord().withValue("direction", "INBOUND").withValue("messageId", As2Mdn.sanitize(As2Mdn.normalizeId(response.headers().get("message-id"))))
.withValue("originalMessageId", messageId).withValue("disposition", mdn.disposition()).withValue("micAlgorithm", mdn.mic() == null ? null : mdn.mic().getDigestAlgorithm().getID())
.withValue("micValue", As2Store.micValue(mdn.mic())).withValue("signed", mdn.signed()).withValue("signatureValid", mdn.signatureValid()).withValue("payloadRef", raw.value())
.withValue("transportId", transport.getId()).withValue("partnerId", message.getPartnerId()).withValue("outboundMessageId", message.getId()).withValue("deliveryStatus", "NONE"));
EdiJournal.event(null, message.getInterchangeId(), null, "MDN");
if(failure != null)
{
return (new EdiSendResult("FAILED", messageId, failure));
}
QAlerts.resolve("edi.mdn.overdue", "edi_outbound_message:" + message.getId(), "MDN received");
return (new EdiSendResult("DELIVERED", messageId, "MDN processed"));
}
/*******************************************************************************
** The reason a receipt does not confirm delivery, or null when it does.
*******************************************************************************/
static String judge(EdiTransport transport, As2Mdn.Parsed mdn, MIC ours)
{
if(Boolean.TRUE.equals(mdn.signed()) && !Boolean.TRUE.equals(mdn.signatureValid()))
{
return ("The MDN signature could not be verified");
}
if(Boolean.TRUE.equals(transport.getMdnSigned()) && !Boolean.TRUE.equals(mdn.signed()))
{
return ("The MDN is not signed");
}
if(!mdn.processed())
{
return ("Partner disposition: " + mdn.disposition());
}
if(ours != null && (mdn.mic() == null ? Boolean.TRUE.equals(transport.getMdnSigned()) || mdn.micText() != null : !mdn.mic().equals(ours)))
{
return ("The MDN Received-Content-MIC does not match the message sent");
}
return (null);
}
/*******************************************************************************
** Probe configuration without sending any document: resolve keys and
** certificates, then OPTIONS the endpoint.
*******************************************************************************/
@Override
public EdiTestResult test(EdiTransport transport)
{
try
{
return (SystemWrites.call(() -> probe(transport)));
}
catch(Exception e)
{
return (new EdiTestResult(false, "AS2 test could not run; check the database and configuration"));
}
}
/*******************************************************************************
** Resolve keys and certificates and reach the endpoint.
*******************************************************************************/
private EdiTestResult probe(EdiTransport transport) throws Exception
{
CertificateService certificates = new CertificateService();
List<String> notes = new java.util.ArrayList<>();
try
{
boolean secured = transport.getSignAlgorithm() != null || transport.getEncryptAlgorithm() != null || Boolean.TRUE.equals(transport.getMdnSigned());
if(secured || transport.getLocalCertificateId() != null)
{
LocalKey key = certificates.activeLocal(transport);
notes.add("Local key " + key.row().getValueString("alias") + " resolved (valid until " + key.certificate().getNotAfter().toInstant() + ")");
}
if(secured || transport.getPartnerCertificateId() != null)
{
certificates.activePartnerCertificate(transport);
notes.add(certificates.partnerVerificationCertificates(transport).size() + " partner certificate(s) accepted for verification");
}
HttpClient client = As2Http.client(certificates.tlsPin(transport));
if(!"INBOUND".equals(transport.getDirection()) && transport.getAs2Url() != null)
{
int status = As2Http.options(client, URI.create(transport.getAs2Url()), Map.of(), Duration.ofSeconds(15)).status();
notes.add("Endpoint responded with HTTP " + status);
if(status >= 500)
{
return (new EdiTestResult(false, String.join("; ", notes)));
}
}
}
catch(As2Exception e)
{
notes.add(e.getMessage());
return (new EdiTestResult(false, String.join("; ", notes)));
}
catch(java.io.IOException e)
{
notes.add("The AS2 endpoint could not be reached");
return (new EdiTestResult(false, String.join("; ", notes)));
}
if(As2Crypto.isWeak(transport.getSignAlgorithm(), transport.getEncryptAlgorithm()))
{
notes.add("Warning: weak crypto (SHA1 or 3DES) is configured");
}
return (new EdiTestResult(true, String.join("; ", notes)));
}
/*******************************************************************************
** Receipt mode, defaulting to SYNC.
*******************************************************************************/
static String mode(EdiTransport transport)
{
return (transport.getMdnMode() == null ? "SYNC" : transport.getMdnMode());
}
/*******************************************************************************
** A stable Message-ID: retries of the same outbound message reuse it, so the
** partner can recognize them.
*******************************************************************************/
static String messageId(Integer messageId, EdiTransport transport, byte[] edi) throws Exception
{
String digest = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(edi)).substring(0, 12);
return ("edi-" + messageId + "-" + digest + "@" + transport.getAs2LocalId().replaceAll("[^A-Za-z0-9.-]", "-"));
}
/*******************************************************************************
** An AS2 identifier as a header value; identifiers with spaces are quoted.
*******************************************************************************/
static String asId(String id)
{
return (id.contains(" ") ? "\"" + id + "\"" : id);
}
/*******************************************************************************
** Subject header without line breaks.
*******************************************************************************/
private static String subject(EdiTransport transport)
{
String subject = transport.getAs2Subject() == null || transport.getAs2Subject().isBlank() ? "EDI interchange" : transport.getAs2Subject();
return (subject.replaceAll("[\\r\\n]+", " "));
}
/*******************************************************************************
** Resolve a secret by environment variable name; failure names only the name.
*******************************************************************************/
private static String secret(String variable) throws As2Exception
{
String value = EdiRuntime.current().config().getSecretsResolver().resolve(variable);
if(value == null)
{
throw (new As2Exception("environment variable " + variable + " is not set"));
}
return (value);
}
}
@@ -0,0 +1,102 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.security.cert.X509Certificate;
import java.time.Instant;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
/*******************************************************************************
** Pre-insert and pre-update customizer for edi_certificate. It parses the pasted
** public certificate and fills subject, issuer, serial number, fingerprint and
** validity dates, and refuses a private key pasted into certificatePem (the
** private key belongs in the environment variable named by privateKeyEnvVar).
*******************************************************************************/
public class CertificateCustomizer implements TableCustomizerInterface
{
/*******************************************************************************
** Parse the certificate on save; reject private keys and key variables on
** partner certificates without echoing any value.
*******************************************************************************/
@Override
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords) throws QException
{
for(QRecord record : records)
{
QRecord merged = new QRecord();
oldRecords.orElse(List.of()).stream().filter(old -> Objects.equals(old.getValue("id"), record.getValue("id")))
.findFirst().ifPresent(old -> merged.getValues().putAll(old.getValues()));
merged.getValues().putAll(record.getValues());
boolean insert = oldRecords.isEmpty() || record.getValue("id") == null;
if(insert || record.getValues().containsKey("certificatePem"))
{
try
{
X509Certificate certificate = CertificateService.parse(merged.getValueString("certificatePem"));
record.setValue("subject", abbreviate(certificate.getSubjectX500Principal().getName()));
record.setValue("issuer", abbreviate(certificate.getIssuerX500Principal().getName()));
record.setValue("serialNumber", certificate.getSerialNumber().toString(16).toUpperCase());
record.setValue("fingerprintSha256", CertificateService.fingerprint(certificate));
record.setValue("notBefore", certificate.getNotBefore().toInstant());
record.setValue("notAfter", certificate.getNotAfter().toInstant());
merged.setValue("notAfter", certificate.getNotAfter().toInstant());
}
catch(As2Exception e)
{
record.addError(new BadInputStatusMessage(e.getMessage()));
}
}
if(insert && merged.getValueString("status") == null)
{
record.setValue("status", "ACTIVE");
}
if(merged.getValueInteger("partnerId") != null && merged.getValueString("privateKeyEnvVar") != null && !merged.getValueString("privateKeyEnvVar").isBlank())
{
record.addError(new BadInputStatusMessage("privateKeyEnvVar is only for our own certificates; leave it blank on a partner certificate"));
}
Instant notAfter = merged.getValueInstant("notAfter");
if(notAfter != null && merged.getValueInstant("notBefore") != null && !notAfter.isAfter(merged.getValueInstant("notBefore")))
{
record.addError(new BadInputStatusMessage("The certificate validity period is empty"));
}
}
return (records);
}
/*******************************************************************************
** Keep parsed names within the column length.
*******************************************************************************/
private static String abbreviate(String value)
{
return (value.length() <= 255 ? value : value.substring(0, 255));
}
}
@@ -0,0 +1,474 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.as2;
import java.io.ByteArrayInputStream;
import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.security.KeyStore;
import java.security.MessageDigest;
import java.security.PrivateKey;
import java.security.Signature;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.Base64;
import java.util.Collection;
import java.util.Collections;
import java.util.HexFormat;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.bouncycastle.asn1.pkcs.PrivateKeyInfo;
import org.bouncycastle.openssl.PEMKeyPair;
import org.bouncycastle.openssl.PEMParser;
import org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter;
import org.bouncycastle.openssl.jcajce.JceOpenSSLPKCS8DecryptorProviderBuilder;
import org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo;
/*******************************************************************************
** Resolves the certificates and keys an AS2 transport uses. Public certificates
** come from edi_certificate rows; our private keys come only from environment
** variables (a PEM PKCS#8 key or a base64 PKCS#12 store) through the replaceable
** secrets resolver, are matched to the certificate, and are never cached,
** logged, stored or returned. Callers run inside the system context.
*******************************************************************************/
public final class CertificateService
{
private static final QLogger LOG = QLogger.getLogger(CertificateService.class);
public static final String TABLE = "edi_certificate";
/*******************************************************************************
** Our certificate with its matched private key; the key is excluded from any
** string form of this object.
*******************************************************************************/
public record LocalKey(QRecord row, X509Certificate certificate, PrivateKey key)
{
/*******************************************************************************
** Never render key material.
*******************************************************************************/
@Override
public String toString()
{
return ("LocalKey[" + row.getValueString("alias") + "]");
}
}
/*******************************************************************************
** Local keys usable for decryption, plus the problems (naming environment
** variables only) for keys that could not be loaded.
*******************************************************************************/
public record KeySet(List<LocalKey> keys, List<String> problems)
{
}
/*******************************************************************************
** True when the text contains any private key block.
*******************************************************************************/
public static boolean containsPrivateKey(String text)
{
return (text != null && text.toUpperCase().contains("PRIVATE KEY"));
}
/*******************************************************************************
** Parse the first X.509 certificate of a PEM text; private keys are refused
** without echoing any content.
*******************************************************************************/
public static X509Certificate parse(String pem) throws As2Exception
{
if(pem == null || pem.isBlank())
{
throw (new As2Exception("certificatePem is required"));
}
if(containsPrivateKey(pem))
{
throw (new As2Exception("certificatePem must hold only the public certificate; keep the private key in the environment variable named by privateKeyEnvVar"));
}
if(!pem.contains("-----BEGIN CERTIFICATE-----"))
{
throw (new As2Exception("certificatePem must be a PEM encoded X.509 certificate"));
}
Collection<?> certificates;
try
{
certificates = CertificateFactory.getInstance("X.509").generateCertificates(new ByteArrayInputStream(pem.getBytes(StandardCharsets.UTF_8)));
}
catch(Exception e)
{
throw (new As2Exception("certificatePem could not be parsed as an X.509 certificate"));
}
if(certificates.isEmpty() || !(certificates.iterator().next() instanceof X509Certificate certificate))
{
throw (new As2Exception("certificatePem does not contain an X.509 certificate"));
}
return (certificate);
}
/*******************************************************************************
** Upper-case colon separated SHA-256 fingerprint of the certificate.
*******************************************************************************/
public static String fingerprint(X509Certificate certificate) throws As2Exception
{
try
{
return (HexFormat.ofDelimiter(":").withUpperCase().formatHex(MessageDigest.getInstance("SHA-256").digest(certificate.getEncoded())));
}
catch(Exception e)
{
throw (new As2Exception("certificate fingerprint could not be computed"));
}
}
/*******************************************************************************
** Resolve the key used to sign messages and receipts for the transport: the
** referenced certificate when it is ACTIVE, otherwise the ACTIVE certificate
** of ours with the same usage (so a promoted rollover takes effect at once).
*******************************************************************************/
public LocalKey activeLocal(EdiTransport transport) throws As2Exception, QException
{
QRecord referenced = byId(transport.getLocalCertificateId());
if(referenced == null)
{
throw (new As2Exception("The local certificate of transport " + transport.getName() + " is not configured"));
}
QRecord active = "ACTIVE".equals(referenced.getValueString("status")) ? referenced : findActive(null, referenced.getValueString("usage"), true);
if(active == null)
{
throw (new As2Exception("No ACTIVE local certificate is available for transport " + transport.getName()));
}
return (load(active));
}
/*******************************************************************************
** Every local key inbound decryption accepts: our ACTIVE and NEXT certificates
** that can decrypt. A key that cannot be loaded is skipped and reported by
** environment variable name only.
*******************************************************************************/
public KeySet localDecryptionKeys() throws QException
{
List<LocalKey> keys = new ArrayList<>();
List<String> problems = new ArrayList<>();
QQueryFilter filter = new QQueryFilter().withCriteria("status", QCriteriaOperator.IN, "ACTIVE", "NEXT")
.withCriteria("partnerId", QCriteriaOperator.IS_BLANK).withCriteria("privateKeyEnvVar", QCriteriaOperator.IS_NOT_BLANK);
for(QRecord row : EdiRecords.query(TABLE, filter))
{
if(!List.of("ENCRYPTION", "SIGNING_AND_ENCRYPTION").contains(row.getValueString("usage")))
{
continue;
}
try
{
keys.add(load(row));
}
catch(As2Exception e)
{
problems.add(e.getMessage());
LOG.warn("Local AS2 key skipped: " + e.getMessage());
}
}
return (new KeySet(keys, problems));
}
/*******************************************************************************
** The partner certificate outbound messages are encrypted to: the referenced
** certificate when ACTIVE, otherwise the partner ACTIVE certificate of the
** same usage.
*******************************************************************************/
public X509Certificate activePartnerCertificate(EdiTransport transport) throws As2Exception, QException
{
QRecord referenced = byId(transport.getPartnerCertificateId());
if(referenced == null)
{
throw (new As2Exception("The partner certificate of transport " + transport.getName() + " is not configured"));
}
QRecord active = "ACTIVE".equals(referenced.getValueString("status")) ? referenced : findActive(referenced.getValueInteger("partnerId"), referenced.getValueString("usage"), false);
if(active == null)
{
throw (new As2Exception("No ACTIVE partner certificate is available for transport " + transport.getName()));
}
return (parse(active.getValueString("certificatePem")));
}
/*******************************************************************************
** Partner certificates a signature may verify against: the referenced one and
** the same partner other ACTIVE or NEXT signing certificates. A partner
** certificate with no partner (a shared VAN) has no rollover candidates.
*******************************************************************************/
public List<X509Certificate> partnerVerificationCertificates(EdiTransport transport) throws As2Exception, QException
{
QRecord referenced = byId(transport.getPartnerCertificateId());
if(referenced == null)
{
throw (new As2Exception("The partner certificate of transport " + transport.getName() + " is not configured"));
}
List<QRecord> rows = new ArrayList<>();
if(List.of("ACTIVE", "NEXT").contains(referenced.getValueString("status")))
{
rows.add(referenced);
}
if(referenced.getValueInteger("partnerId") != null)
{
QQueryFilter filter = EdiRecords.equal("partnerId", referenced.getValueInteger("partnerId")).withCriteria("status", QCriteriaOperator.IN, "ACTIVE", "NEXT")
.withCriteria("usage", QCriteriaOperator.IN, "SIGNING", "SIGNING_AND_ENCRYPTION").withCriteria("privateKeyEnvVar", QCriteriaOperator.IS_BLANK);
for(QRecord row : EdiRecords.query(TABLE, filter))
{
if(rows.stream().noneMatch(existing -> Objects.equals(existing.getValue("id"), row.getValue("id"))))
{
rows.add(row);
}
}
}
List<X509Certificate> certificates = new ArrayList<>();
for(QRecord row : rows)
{
certificates.add(parse(row.getValueString("certificatePem")));
}
return (certificates);
}
/*******************************************************************************
** The pinned TLS certificate of the partner endpoint, or null when no pin is set.
*******************************************************************************/
public X509Certificate tlsPin(EdiTransport transport) throws As2Exception, QException
{
QRecord row = byId(transport.getPartnerTlsCertificateId());
if(transport.getPartnerTlsCertificateId() != null && row == null)
{
throw (new As2Exception("The partner TLS certificate of transport " + transport.getName() + " does not exist"));
}
return (row == null ? null : parse(row.getValueString("certificatePem")));
}
/*******************************************************************************
** Promote a NEXT certificate to ACTIVE and retire the previous ACTIVE one of
** the same partner and usage. Returns false when the certificate is not NEXT.
*******************************************************************************/
public boolean promote(QRecord row) throws QException
{
if(!"NEXT".equals(row.getValueString("status")))
{
return (false);
}
boolean local = row.getValueInteger("partnerId") == null && row.getValueString("privateKeyEnvVar") != null;
QRecord previous = findActive(row.getValueInteger("partnerId"), row.getValueString("usage"), local);
while(previous != null)
{
SystemWrites.update(TABLE, new QRecord().withValue("id", previous.getValue("id")).withValue("status", "RETIRED"));
previous = findActive(row.getValueInteger("partnerId"), row.getValueString("usage"), local);
}
SystemWrites.update(TABLE, new QRecord().withValue("id", row.getValue("id")).withValue("status", "ACTIVE"));
return (true);
}
/*******************************************************************************
** Certificate row by id, or null.
*******************************************************************************/
private QRecord byId(Integer id) throws QException
{
if(id == null)
{
return (null);
}
List<QRecord> rows = EdiRecords.query(TABLE, EdiRecords.equal("id", id));
return (rows.isEmpty() ? null : rows.getFirst());
}
/*******************************************************************************
** The ACTIVE certificate of a partner (null partner means ours or a shared
** one) and usage, local ones being those with a private key variable.
*******************************************************************************/
private QRecord findActive(Integer partnerId, String usage, boolean local) throws QException
{
QQueryFilter filter = EdiRecords.equal("status", "ACTIVE").withCriteria("partnerId", partnerId == null ? QCriteriaOperator.IS_BLANK : QCriteriaOperator.EQUALS, partnerId)
.withCriteria("usage", QCriteriaOperator.EQUALS, usage).withCriteria("privateKeyEnvVar", local ? QCriteriaOperator.IS_NOT_BLANK : QCriteriaOperator.IS_BLANK);
List<QRecord> rows = EdiRecords.query(TABLE, filter);
return (rows.stream().min((a, b) -> Integer.compare(a.getValueInteger("id"), b.getValueInteger("id"))).orElse(null));
}
/*******************************************************************************
** Parse the certificate, read its private key from the environment and prove
** the two belong together by signing and verifying a probe.
*******************************************************************************/
private LocalKey load(QRecord row) throws As2Exception
{
X509Certificate certificate = parse(row.getValueString("certificatePem"));
String variable = row.getValueString("privateKeyEnvVar");
if(variable == null || variable.isBlank())
{
throw (new As2Exception("Certificate " + row.getValueString("alias") + " has no privateKeyEnvVar"));
}
PrivateKey key = readKey(variable, row.getValueString("privateKeyPasswordEnvVar"));
if(!matches(key, certificate))
{
throw (new As2Exception("The private key in environment variable " + variable + " does not match certificate " + row.getValueString("alias")));
}
return (new LocalKey(row, certificate, key));
}
/*******************************************************************************
** Read the key from the named environment variable; failures name only the
** variable.
*******************************************************************************/
private static PrivateKey readKey(String variable, String passwordVariable) throws As2Exception
{
var secrets = EdiRuntime.current().config().getSecretsResolver();
String value = secrets.resolve(variable);
if(value == null || value.isBlank())
{
throw (new As2Exception("environment variable " + variable + " is not set"));
}
char[] password = new char[0];
if(passwordVariable != null && !passwordVariable.isBlank())
{
String resolved = secrets.resolve(passwordVariable);
if(resolved == null)
{
throw (new As2Exception("environment variable " + passwordVariable + " is not set"));
}
password = resolved.toCharArray();
}
try
{
return (parseKey(value, password));
}
catch(Exception e)
{
throw (new As2Exception("The private key in environment variable " + variable + " could not be read as a PEM PKCS#8 key or a base64 PKCS#12 store"));
}
}
/*******************************************************************************
** Parse a PEM key (PKCS#8, encrypted PKCS#8 or traditional) or a base64 PKCS#12
** store, entirely in memory.
*******************************************************************************/
static PrivateKey parseKey(String raw, char[] password) throws Exception
{
String text = raw.trim();
if(text.indexOf('\n') < 0 && text.contains("\\n"))
{
text = text.replace("\\n", "\n");
}
if(text.contains("-----BEGIN"))
{
JcaPEMKeyConverter converter = new JcaPEMKeyConverter();
try(PEMParser parser = new PEMParser(new StringReader(text)))
{
Object object = parser.readObject();
while(object != null)
{
if(object instanceof PrivateKeyInfo info)
{
return (converter.getPrivateKey(info));
}
if(object instanceof PEMKeyPair pair)
{
return (converter.getKeyPair(pair).getPrivate());
}
if(object instanceof PKCS8EncryptedPrivateKeyInfo encrypted)
{
return (converter.getPrivateKey(encrypted.decryptPrivateKeyInfo(new JceOpenSSLPKCS8DecryptorProviderBuilder().build(password))));
}
object = parser.readObject();
}
}
throw (new IllegalArgumentException("no private key block"));
}
KeyStore store = KeyStore.getInstance("PKCS12");
store.load(new ByteArrayInputStream(Base64.getMimeDecoder().decode(text)), password);
for(String alias : Collections.list(store.aliases()))
{
if(store.isKeyEntry(alias))
{
return ((PrivateKey) store.getKey(alias, password));
}
}
throw (new IllegalArgumentException("no key entry"));
}
/*******************************************************************************
** Sign a probe with the key and verify it with the certificate public key.
*******************************************************************************/
private static boolean matches(PrivateKey key, X509Certificate certificate)
{
try
{
String algorithm = "EC".equals(key.getAlgorithm()) ? "SHA256withECDSA" : "SHA256with" + key.getAlgorithm();
byte[] probe = "qbit-edi-key-match".getBytes(StandardCharsets.UTF_8);
Signature signer = Signature.getInstance(algorithm);
signer.initSign(key);
signer.update(probe);
byte[] signature = signer.sign();
Signature verifier = Signature.getInstance(algorithm);
verifier.initVerify(certificate.getPublicKey());
verifier.update(probe);
return (verifier.verify(signature));
}
catch(Exception e)
{
return (false);
}
}
}
@@ -0,0 +1,46 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.document;
import java.util.List;
import com.kingsrook.qbits.edi.x12.EnvelopeFinding;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** Standard-neutral document view over the existing immutable segment value model.
*******************************************************************************/
public interface EdiDocument
{
/*******************************************************************************
** Return the actual message header, including its original composite values.
*******************************************************************************/
X12Segment header();
/*******************************************************************************
** Return ordered segments including the message envelope.
*******************************************************************************/
List<X12Segment> segments();
/*******************************************************************************
** Return envelope diagnostics without submitted values.
*******************************************************************************/
List<EnvelopeFinding> findings();
}
@@ -0,0 +1,120 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.document;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import com.kingsrook.qbits.edi.edifact.EdifactSeparators;
/*******************************************************************************
** Frame concatenated X12 and EDIFACT envelopes without delimiter guessing in data.
*******************************************************************************/
public class EnvelopeReader
{
/*******************************************************************************
** Read one envelope, respecting UNA and released terminators; null means EOF.
*******************************************************************************/
public byte[] next(InputStream input) throws IOException
{
int first;
do
{
first = input.read();
}
while(first != -1 && Character.isWhitespace(first));
if(first == -1)
{
return (null);
}
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bytes.write(first);
bytes.write(input.readNBytes(2));
String tag = bytes.toString(StandardCharsets.US_ASCII);
char terminator;
char release = 0;
String end;
StringBuilder prefix = new StringBuilder();
if("ISA".equals(tag))
{
bytes.write(input.readNBytes(103));
if(bytes.size() != 106)
{
return (bytes.toByteArray());
}
terminator = (char) bytes.toByteArray()[105];
end = "IEA";
}
else if("UNA".equals(tag) || "UNB".equals(tag))
{
EdifactSeparators separators = EdifactSeparators.DEFAULT;
if("UNA".equals(tag))
{
bytes.write(input.readNBytes(6));
separators = EdifactSeparators.fromUna(bytes.toString(StandardCharsets.US_ASCII));
}
else
{
prefix.append("UNB");
}
terminator = separators.segment();
release = separators.release();
end = "UNZ";
}
else
{
throw new IOException("Unknown EDI envelope");
}
boolean escaped = false;
int value;
while((value = input.read()) != -1)
{
bytes.write(value);
if(bytes.size() > 50 * 1024 * 1024)
{
throw new IOException("EDI envelope exceeds size limit");
}
if(escaped)
{
escaped = false;
continue;
}
if(value == release)
{
escaped = true;
}
else if(value == terminator)
{
if(prefix.toString().equals(end))
{
break;
}
prefix.setLength(0);
}
else if(prefix.length() < 3 && !(prefix.isEmpty() && Character.isWhitespace(value)))
{
prefix.append((char) value);
}
}
return (bytes.toByteArray());
}
}
@@ -0,0 +1,152 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.ack.AckLevel;
import com.kingsrook.qbits.edi.guide.ValidationFinding;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** Positional CONTRL syntax responses; business application failures are not syntax errors.
*******************************************************************************/
public class ContrlBuilder
{
/*******************************************************************************
** Build directory-specific acknowledgement details without echoing data values.
*******************************************************************************/
public EdifactMessage build(EdifactInterchange input, Map<String, List<ValidationFinding>> findings, AckLevel level, String reference, String version)
{
List<X12Segment> segments = new ArrayList<>();
segments.add(EdifactWriter.segment("UNH", 1, List.of(reference), List.of("CONTRL", version.substring(0, 1), version.substring(1), "UN")));
boolean rejected = !input.findings().isEmpty() || level == AckLevel.GROUP_ONLY && (input.groups().stream().anyMatch(g -> !g.findings().isEmpty())
|| input.allMessages().stream().anyMatch(m -> rejected(m, findings)));
var uci = new ArrayList<List<String>>();
uci.add(List.of(input.unb().value(5)));
uci.add(input.unb().elements().get(1).getFirst());
uci.add(input.unb().elements().get(2).getFirst());
uci.add(List.of(rejected ? "4" : "7"));
if(!input.findings().isEmpty())
{
uci.add(List.of(input.findings().getFirst().code()));
}
segments.add(new X12Segment("UCI", 2, uci.stream().map(List::of).toList()));
if(level != AckLevel.GROUP_ONLY && input.findings().isEmpty())
{
messages(input.messages(), findings, level, segments);
for(var group : input.groups())
{
var response = EdifactWriter.segment("UCF", segments.size() + 1, List.of(group.ung().value(5)), group.ung().elements().get(1).getFirst(),
group.ung().elements().get(2).getFirst(), List.of(group.findings().isEmpty() ? "7" : "4"));
if(!group.findings().isEmpty())
{
var fields = new ArrayList<>(response.elements());
fields.add(List.of(List.of(group.findings().getFirst().code())));
response = new X12Segment("UCF", response.position(), fields);
}
segments.add(response);
messages(group.messages(), findings, level, segments);
}
}
segments.add(X12Segment.of("UNT", segments.size() + 1, String.valueOf(segments.size() + 1), reference));
return (new EdifactMessage(segments.getFirst(), segments.getLast(), List.copyOf(segments), List.of()));
}
/*******************************************************************************
** Suppress accepted message detail only under ERRORS_ONLY.
*******************************************************************************/
private void messages(List<EdifactMessage> messages, Map<String, List<ValidationFinding>> findings, AckLevel level, List<X12Segment> segments)
{
for(var message : messages)
{
if("CONTRL".equals(message.type()))
{
continue;
}
boolean rejected = rejected(message, findings);
if(level == AckLevel.ERRORS_ONLY && !rejected)
{
continue;
}
var fields = new ArrayList<List<List<String>>>();
fields.add(List.of(List.of(message.reference())));
fields.add(message.header().elements().get(1));
fields.add(List.of(List.of(rejected ? "4" : "7")));
if(!message.findings().isEmpty())
{
fields.add(List.of(List.of(message.findings().getFirst().code())));
}
segments.add(new X12Segment("UCM", segments.size() + 1, fields));
for(var finding : findings.getOrDefault(message.reference(), List.of()))
{
if(!"ERROR".equals(finding.severity()))
{
continue;
}
String code = syntaxCode(finding);
segments.add(finding.elementPosition() == null
? X12Segment.of("UCS", segments.size() + 1, String.valueOf(finding.segmentPosition()), code)
: X12Segment.of("UCS", segments.size() + 1, String.valueOf(finding.segmentPosition())));
if(finding.elementPosition() != null)
{
List<String> position = finding.componentPosition() == null ? List.of(finding.elementPosition().toString())
: List.of(finding.elementPosition().toString(), finding.componentPosition().toString());
segments.add(EdifactWriter.segment("UCD", segments.size() + 1, List.of(code), position));
}
}
}
}
/*******************************************************************************
** Envelope and structural errors reject the message, warnings do not.
*******************************************************************************/
private boolean rejected(EdifactMessage message, Map<String, List<ValidationFinding>> findings)
{
return (!message.findings().isEmpty() || findings.getOrDefault(message.reference(), List.of()).stream().anyMatch(f -> "ERROR".equals(f.severity())));
}
/*******************************************************************************
** Translate shared validator diagnostics to service syntax codes (0085).
*******************************************************************************/
public static String syntaxCode(ValidationFinding finding)
{
String code = finding.elementPosition() == null ? finding.ak3Code() : finding.ak4Code();
if(code == null)
{
return ("12");
}
return (finding.elementPosition() == null ? switch(code)
{
case "3" -> "13";
case "4", "5" -> "35";
case "2", "6", "7" -> "33";
default -> "12";
} : switch(code)
{
case "1", "2" -> "13";
case "4" -> "14";
case "5" -> "15";
default -> "12";
});
}
}
@@ -0,0 +1,143 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.flow.EdiJournal;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Reconcile addressed CONTRL controls only within the routed outbound agreement.
*******************************************************************************/
public class ContrlReader
{
/*******************************************************************************
** Apply functional acceptance/rejection; receipt-only code 8 leaves deadlines open.
*******************************************************************************/
public void reconcile(EdifactMessage message, QRecord agreement, QRecord received) throws Exception
{
X12Segment uci = message.segments().stream().filter(s -> "UCI".equals(s.id())).findFirst().orElseThrow();
if("8".equals(uci.value(4)))
{
return;
}
String overall = status(uci.value(4));
var candidates = EdiRecords.query("edi_interchange", EdiRecords.equal("agreementId", agreement.getValue("id")).withCriteria("direction", QCriteriaOperator.EQUALS, "OUTBOUND")
.withCriteria("standard", QCriteriaOperator.EQUALS, "EDIFACT").withCriteria("controlNumber", QCriteriaOperator.EQUALS, uci.value(1))
.withCriteria("senderId", QCriteriaOperator.EQUALS, uci.value(2)).withCriteria("senderQualifier", QCriteriaOperator.EQUALS, uci.value(2, 1, 2))
.withCriteria("receiverId", QCriteriaOperator.EQUALS, uci.value(3)).withCriteria("receiverQualifier", QCriteriaOperator.EQUALS, uci.value(3, 1, 2)));
if(candidates.size() != 1)
{
throw new IllegalArgumentException("Acknowledgement reference is unknown or ambiguous");
}
QRecord interchange = candidates.getFirst();
Map<String, X12Segment> details = new LinkedHashMap<>();
Map<String, String> groupStatuses = new LinkedHashMap<>();
String currentGroup = "";
for(var segment : message.segments())
{
if("UCF".equals(segment.id()))
{
currentGroup = segment.value(1);
groupStatuses.put(currentGroup, status(segment.value(4)));
}
if("UCM".equals(segment.id()))
{
String key = currentGroup + ":" + segment.value(1);
if(details.put(key, segment) != null)
{
throw new IllegalArgumentException("Duplicate acknowledgement message reference");
}
status(segment.value(3));
}
}
boolean rejected = "R".equals(overall);
var groups = EdiRecords.query("edi_functional_group", EdiRecords.equal("interchangeId", interchange.getValue("id")));
for(var group : groups)
{
String scope = interchange.getValueInteger("groupCount") == 0 ? "" : group.getValueString("controlNumber");
String groupStatus = "R".equals(overall) ? "R" : groupStatuses.getOrDefault(scope, overall);
boolean groupRejected = "R".equals(groupStatus);
for(var tx : EdiRecords.query("edi_transaction", EdiRecords.equal("functionalGroupId", group.getValue("id"))))
{
X12Segment detail = details.remove(scope + ":" + tx.getValueString("controlNumber"));
if(detail != null && (!detail.value(2).equals(tx.getValueString("transactionSet"))
|| !(detail.value(2, 1, 2) + detail.value(2, 1, 3)).equals(tx.getValueString("version"))))
{
throw new IllegalArgumentException("Acknowledgement message identification mismatch");
}
String result = "R".equals(groupStatus) || detail != null && "R".equals(status(detail.value(3))) ? "R" : "A";
groupRejected |= "R".equals(result);
SystemWrites.update("edi_transaction", new QRecord().withValue("id", tx.getValue("id")).withValue("ackStatus", result)
.withValue("status", "R".equals(result) ? "REJECTED" : "ACKNOWLEDGED"));
}
rejected |= groupRejected;
SystemWrites.update("edi_functional_group", new QRecord().withValue("id", group.getValue("id")).withValue("ackStatus", groupRejected ? "R" : "A")
.withValue("ackReceivedAt", EdiRuntime.current().config().getClock().instant()).withValue("ackTransactionId", received.getValue("id")));
}
if(!details.isEmpty() || groupStatuses.keySet().stream().anyMatch(c -> groups.stream().noneMatch(g -> c.equals(g.getValueString("controlNumber")))))
{
throw new IllegalArgumentException("Acknowledgement contains an unknown message or group reference");
}
SystemWrites.update("edi_interchange", new QRecord().withValue("id", interchange.getValue("id")).withValue("status", rejected ? "REJECTED" : "ACKNOWLEDGED"));
SystemWrites.update("edi_transaction", new QRecord().withValue("id", received.getValue("id")).withValue("status", "ACKNOWLEDGED").withValue("ackStatus", rejected ? "R" : "A"));
for(var group : groups)
{
QAlerts.resolve("edi.ack.overdue", "edi_functional_group:" + group.getValue("id"), "CONTRL received");
}
for(var exception : EdiRecords.query("edi_exception", EdiRecords.equal("interchangeId", interchange.getValue("id")).withCriteria("type", QCriteriaOperator.EQUALS, "OVERDUE_ACK")
.withCriteria("status", QCriteriaOperator.IN, "OPEN", "IN_PROGRESS")))
{
SystemWrites.update("edi_exception", new QRecord().withValue("id", exception.getValue("id")).withValue("status", "RESOLVED")
.withValue("resolvedAt", EdiRuntime.current().config().getClock().instant()).withValue("resolution", "CONTRL received"));
}
if(rejected)
{
EdiJournal.exception(null, interchange.getValueInteger("id"), null, agreement.getValueInteger("partnerId"), "REJECTED_ACK", "Partner rejected an outbound EDIFACT document");
QAlerts.raise(new AlertSignal("edi.ack.rejected").withSeverity(AlertSeverity.ERROR).withDedupKey("edi_interchange:" + interchange.getValue("id"))
.withTitle("EDI acknowledgement rejected").withSummary("Review the outbound interchange"));
}
EdiJournal.event(null, interchange.getValueInteger("id"), received.getValueInteger("id"), "ACK_RECEIVED");
}
/*******************************************************************************
** Only functional acceptance and rejection can complete an expected response.
*******************************************************************************/
private String status(String action)
{
return (switch(action)
{
case "7" -> "A";
case "4" -> "R";
default -> throw new IllegalArgumentException("Unsupported CONTRL action");
});
}
}
@@ -0,0 +1,267 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import com.kingsrook.qbits.edi.document.EnvelopeReader;
import com.kingsrook.qbits.edi.x12.EnvelopeFinding;
import com.kingsrook.qbits.edi.x12.X12Segment;
import io.xlate.edi.stream.EDIInputFactory;
import io.xlate.edi.stream.EDIStreamEvent;
import io.xlate.edi.stream.EDIStreamException;
import io.xlate.edi.stream.EDIStreamReader;
import io.xlate.edi.stream.EDIStreamValidationError;
/*******************************************************************************
** StAEDI tokenizer with explicit UN envelope checks and bounded message retention.
*******************************************************************************/
public class EdifactEnvelopeReader
{
private final int maxSegments;
/*******************************************************************************
** Bound retained message segments, including UNH and UNT.
*******************************************************************************/
public EdifactEnvelopeReader(int maxSegments)
{
if(maxSegments < 2)
{
throw new IllegalArgumentException("At least two segments are required");
}
this.maxSegments = maxSegments;
}
/*******************************************************************************
** Read concatenated EDIFACT interchanges without closing the caller stream.
*******************************************************************************/
public List<EdifactInterchange> read(InputStream input) throws IOException
{
List<EdifactInterchange> result = new ArrayList<>();
EnvelopeReader framing = new EnvelopeReader();
byte[] bytes;
while((bytes = framing.next(input)) != null)
{
result.add(parse(bytes));
}
return (result);
}
/*******************************************************************************
** Parse one framed interchange; diagnostics never include submitted values.
*******************************************************************************/
private EdifactInterchange parse(byte[] bytes) throws IOException
{
String prefix = new String(bytes, 0, Math.min(9, bytes.length), StandardCharsets.US_ASCII);
EdifactSeparators separators = prefix.startsWith("UNA") ? EdifactSeparators.fromUna(prefix) : EdifactSeparators.DEFAULT;
List<EnvelopeFinding> errors = new ArrayList<>();
List<EnvelopeFinding> groupErrors = new ArrayList<>();
List<EnvelopeFinding> messageErrors = new ArrayList<>();
List<EdifactGroup> groups = new ArrayList<>();
List<EdifactMessage> messages = new ArrayList<>();
List<EdifactMessage> grouped = new ArrayList<>();
List<X12Segment> segments = new ArrayList<>();
var references = new HashSet<String>();
X12Segment unb = null;
X12Segment unz = null;
X12Segment ung = null;
X12Segment unh = null;
String tag = "";
int count = 0;
List<List<List<String>>> values = new ArrayList<>();
EDIInputFactory factory = EDIInputFactory.newFactory();
factory.setProperty(EDIInputFactory.EDI_IGNORE_EXTRANEOUS_CHARACTERS, true);
factory.setProperty(EDIInputFactory.EDI_VALIDATE_CONTROL_CODE_VALUES, false);
try(EDIStreamReader reader = factory.createEDIStreamReader(new ByteArrayInputStream(bytes)))
{
while(reader.hasNext())
{
EDIStreamEvent event = reader.next();
if(event == EDIStreamEvent.START_SEGMENT)
{
tag = reader.getText();
values = new ArrayList<>();
}
else if(event == EDIStreamEvent.ELEMENT_DATA)
{
int e = reader.getLocation().getElementPosition();
int r = Math.max(1, reader.getLocation().getElementOccurrence());
int c = Math.max(1, reader.getLocation().getComponentPosition());
while(values.size() < e)
{
values.add(new ArrayList<>());
}
while(values.get(e - 1).size() < r)
{
values.get(e - 1).add(new ArrayList<>());
}
var components = values.get(e - 1).get(r - 1);
while(components.size() < c)
{
components.add("");
}
components.set(c - 1, reader.getText());
}
else if(event.isError() && reader.getErrorType() != EDIStreamValidationError.CONTROL_REFERENCE_MISMATCH
&& reader.getErrorType() != EDIStreamValidationError.CONTROL_COUNT_DOES_NOT_MATCH_ACTUAL_COUNT)
{
if(unh != null)
{
messageErrors.add(error("UNH", Math.max(1, count), "12"));
}
else if(ung != null)
{
groupErrors.add(error("UNG", count, "12"));
}
else
{
errors.add(error("UNB", count, "12"));
}
}
else if(event == EDIStreamEvent.END_SEGMENT)
{
if("UNA".equals(tag))
{
continue;
}
count = "UNH".equals(tag) ? 1 : count + 1;
X12Segment segment = new X12Segment(tag, count, values);
switch(tag)
{
case "UNB" ->
{
unb = segment;
if("4".equals(unb.value(1, 1, 2)) && !prefix.startsWith("UNA"))
{
separators = new EdifactSeparators(':', '+', '.', '?', '*', '\'');
}
if(!List.of("3", "4").contains(unb.value(1, 1, 2)) || !List.of("", "1").contains(unb.value(11))
|| unb.value(5).isEmpty() || unb.value(5).length() > 14)
{
errors.add(error("UNB", 1, "12"));
}
}
case "UNG" ->
{
ung = segment;
grouped = new ArrayList<>();
groupErrors = new ArrayList<>();
}
case "UNH" ->
{
unh = segment;
segments = new ArrayList<>();
segments.add(segment);
messageErrors = new ArrayList<>();
if(!references.add(unh.value(1)))
{
messageErrors.add(error("UNH", 1, "19"));
}
}
case "UNT" ->
{
if(unh == null)
{
errors.add(error("UNB", count, "33"));
break;
}
check(messageErrors, "UNH", segment, segment.value(1), String.valueOf(count), "29");
check(messageErrors, "UNH", segment, segment.value(2), unh.value(1), "28");
if(segments.size() < maxSegments)
{
segments.add(segment);
}
(ung == null ? messages : grouped).add(new EdifactMessage(unh, segment, List.copyOf(segments), List.copyOf(messageErrors)));
unh = null;
}
case "UNE" ->
{
if(ung == null)
{
errors.add(error("UNB", count, "33"));
break;
}
check(groupErrors, "UNG", segment, segment.value(1), String.valueOf(grouped.size()), "29");
check(groupErrors, "UNG", segment, segment.value(2), ung.value(5), "28");
groups.add(new EdifactGroup(ung, segment, List.copyOf(grouped), List.copyOf(groupErrors)));
ung = null;
}
case "UNZ" ->
{
unz = segment;
check(errors, "UNB", segment, segment.value(1), String.valueOf(groups.isEmpty() ? messages.size() : groups.size()), "29");
check(errors, "UNB", segment, segment.value(2), unb == null ? "" : unb.value(5), "28");
}
default ->
{
if(unh == null)
{
errors.add(error("UNB", count, "33"));
}
else if(segments.size() < maxSegments - 1)
{
segments.add(segment);
}
else if(messageErrors.stream().noneMatch(f -> "35".equals(f.code())))
{
messageErrors.add(error("UNH", count, "35"));
}
}
}
}
}
}
catch(EDIStreamException e)
{
errors.add(error("UNB", 0, "12"));
}
if(unb == null || unz == null || ung != null || unh != null || !groups.isEmpty() && !messages.isEmpty())
{
errors.add(error("UNB", 0, "13"));
}
return (new EdifactInterchange(separators, unb, unz, List.copyOf(groups), List.copyOf(messages), List.copyOf(errors)));
}
/*******************************************************************************
** Count fields accept zero padding; references retain their exact spelling.
*******************************************************************************/
private static void check(List<EnvelopeFinding> findings, String level, X12Segment segment, String actual, String expected, String code)
{
boolean equal = "29".equals(code) ? actual.matches("[0-9]+") && new java.math.BigInteger(actual).toString().equals(expected) : actual.equals(expected);
if(!equal)
{
findings.add(error(level, segment.position(), code));
}
}
/*******************************************************************************
** Service syntax error code 0085 with a value-free diagnostic.
*******************************************************************************/
private static EnvelopeFinding error(String level, int position, String code)
{
return (new EnvelopeFinding(level, position, code, "EDIFACT envelope failed validation"));
}
}
@@ -0,0 +1,32 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.List;
import com.kingsrook.qbits.edi.x12.EnvelopeFinding;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** Optional UNG/UNE functional group; ungrouped messages belong to the interchange.
*******************************************************************************/
public record EdifactGroup(X12Segment ung, X12Segment une, List<EdifactMessage> messages, List<EnvelopeFinding> findings)
{
}
@@ -0,0 +1,307 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.ack.AckLevel;
import com.kingsrook.qbits.edi.flow.EdiJournal;
import com.kingsrook.qbits.edi.flow.InboundProcessor;
import com.kingsrook.qbits.edi.flow.InboxWorker;
import com.kingsrook.qbits.edi.flow.OutboundWorker;
import com.kingsrook.qbits.edi.guide.GuideRepository;
import com.kingsrook.qbits.edi.guide.ValidationFinding;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadStore;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Native UNB routing and CONTRL with the shared revision, validation and apply checkpoints.
*******************************************************************************/
public class EdifactInbound extends InboundProcessor
{
private final QRecord inbox;
private final Instant lease;
/*******************************************************************************
** Bind every write to the same inbox lease used by the X12 worker.
*******************************************************************************/
public EdifactInbound(QRecord inbox, Instant lease, PayloadStore store)
{
super(inbox, lease, store);
this.inbox = inbox;
this.lease = lease;
}
/*******************************************************************************
** Route UNB identities before validating, acknowledging or applying messages.
*******************************************************************************/
public void process(EdifactInterchange input, int position) throws Exception
{
if(input.unb() == null)
{
InboxWorker.fenced(inbox, lease, () ->
{
EdiJournal.exception(inbox.getValueInteger("id"), null, null, null, "ENVELOPE", "Invalid EDIFACT header");
return (null);
});
return;
}
QRecord agreement = route(input);
QRecord interchange = InboxWorker.fenced(inbox, lease, () -> interchange(input, agreement, position));
if(agreement == null)
{
problem(interchange, "UNKNOWN_PARTNER");
return;
}
if(inbox.getValueString("username") != null && EdiRecords.query("edi_partner", EdiRecords.equal("gatewayUsername", inbox.getValue("username"))).stream()
.anyMatch(p -> !p.getValue("id").equals(agreement.getValue("partnerId"))))
{
problem(interchange, "PARTNER_MISMATCH");
return;
}
if(inbox.getValueLong("sizeBytes") != null && inbox.getValueLong("sizeBytes") > Objects.requireNonNullElse(agreement.getValueLong("maxInterchangeBytes"), 52428800L))
{
problem(interchange, "ENVELOPE");
return;
}
boolean duplicate = "DUPLICATE".equals(interchange.getValueString("status"));
boolean ackInput = !input.allMessages().isEmpty() && input.allMessages().stream().allMatch(m -> "CONTRL".equals(m.type()));
if(duplicate)
{
problem(interchange, "DUPLICATE");
if(!"PROCESS".equals(agreement.getValueString("onDuplicate")))
{
if(!ackInput && "REJECT_TA1".equals(agreement.getValueString("onDuplicate")))
{
var rejected = new EdifactInterchange(input.separators(), input.unb(), input.unz(), input.groups(), input.messages(),
List.of(new com.kingsrook.qbits.edi.x12.EnvelopeFinding("UNB", 1, "19", "Duplicate interchange")));
acknowledge(rejected, agreement, interchange, Map.of(), AckLevel.GROUP_ONLY);
}
return;
}
}
if(!input.findings().isEmpty())
{
problem(interchange, "ENVELOPE");
if(!ackInput)
{
acknowledge(input, agreement, interchange, Map.of(), AckLevel.GROUP_ONLY);
}
return;
}
Map<String, List<ValidationFinding>> findings = new LinkedHashMap<>();
AckLevel level = AckLevel.ERRORS_ONLY;
boolean sendAck = false;
List<EdifactGroup> batches = input.groups().isEmpty() ? List.of(new EdifactGroup(null, null, input.messages(), List.of())) : input.groups();
int groupPosition = 0;
for(var batch : batches)
{
int gp = ++groupPosition;
QRecord group = InboxWorker.fenced(inbox, lease, () ->
{
var prior = EdiRecords.query("edi_functional_group", EdiRecords.equal("interchangeId", interchange.getValue("id")).withCriteria("inboundPosition", QCriteriaOperator.EQUALS, gp));
return (prior.isEmpty() ? SystemWrites.insert("edi_functional_group", new QRecord().withValue("interchangeId", interchange.getValue("id")).withValue("inboundPosition", gp)
.withValue("senderId", input.unb().value(2)).withValue("controlNumber", batch.ung() == null ? input.unb().value(5) : batch.ung().value(5))
.withValue("version", batch.messages().isEmpty() ? null : batch.messages().getFirst().version())
.withValue("functionalIdCode", batch.ung() == null ? null : batch.ung().value(1)).withValue("transactionCount", batch.messages().size()).withValue("ackStatus", "NONE")) : prior.getFirst());
});
int messagePosition = 0;
for(var message : batch.messages())
{
QRecord document = document(agreement, message);
int mp = ++messagePosition;
QRecord configured = document;
QRecord tx = InboxWorker.fenced(inbox, lease, () ->
{
var prior = EdiRecords.query("edi_transaction", EdiRecords.equal("functionalGroupId", group.getValue("id")).withCriteria("inboundPosition", QCriteriaOperator.EQUALS, mp));
return (prior.isEmpty() ? SystemWrites.insert("edi_transaction", new QRecord().withValue("interchangeId", interchange.getValue("id")).withValue("functionalGroupId", group.getValue("id"))
.withValue("inboundPosition", mp).withValue("partnerId", agreement.getValue("partnerId")).withValue("direction", "INBOUND").withValue("transactionSet", message.type())
.withValue("version", message.version()).withValue("controlNumber", message.reference()).withValue("segmentCount", message.segments().size()).withValue("status", "RECEIVED")
.withValue("agreementDocumentId", configured == null ? null : configured.getValue("id"))
.withValue("containsPhi", configured != null && Boolean.TRUE.equals(configured.getValueBoolean("containsPhi")))) : prior.getFirst());
});
document = revisions(document, tx);
List<ValidationFinding> result = new ArrayList<>();
if(document == null || document.getValue("guideId") == null)
{
result.add(finding("No implementation guide is configured"));
problem(interchange, "NOT_CONFIGURED");
}
else
{
result.addAll(EdiRuntime.current().validator().validate(message, new GuideRepository().load(document.getValueInteger("guideId")), Boolean.TRUE.equals(document.getValueBoolean("containsPhi"))));
}
findings.put(message.reference(), List.copyOf(result));
if(!batch.findings().isEmpty() || !message.findings().isEmpty())
{
result.add(finding("Envelope counts or references failed validation"));
}
QRecord effective = document;
InboxWorker.fenced(inbox, lease, () ->
{
validation(tx, result, effective);
return (null);
});
if("CONTRL".equals(message.type()))
{
if(result.isEmpty())
{
InboxWorker.fenced(inbox, lease, () ->
{
new ContrlReader().reconcile(message, agreement, tx);
return (null);
});
}
continue;
}
if(document != null && "CONTRL".equals(document.getValueString("ackExpectation")))
{
sendAck = true;
AckLevel selected = AckLevel.valueOf(Objects.requireNonNullElse(document.getValueString("ackLevel"), "ERRORS_ONLY"));
if(selected == AckLevel.ALL_TRANSACTIONS || level != AckLevel.ALL_TRANSACTIONS && selected == AckLevel.GROUP_ONLY)
{
level = selected;
}
}
if(document != null && result.stream().noneMatch(f -> "ERROR".equals(f.severity()))
&& (!"T".equals(input.usage()) || Boolean.TRUE.equals(agreement.getValueBoolean("applyTestDocuments"))))
{
apply(message, tx, document, agreement);
}
}
}
if(!ackInput && (sendAck || "1".equals(input.unb().value(9))))
{
acknowledge(input, agreement, interchange, findings, level);
}
}
/*******************************************************************************
** Match qualifier, identifier, standard, usage and identity ownership exactly.
*******************************************************************************/
private QRecord route(EdifactInterchange input) throws Exception
{
for(var agreement : EdiRecords.query("edi_agreement", EdiRecords.equal("isActive", true).withCriteria("standard", QCriteriaOperator.EQUALS, "EDIFACT")
.withCriteria("usageIndicator", QCriteriaOperator.EQUALS, input.usage())))
{
QRecord sender = OutboundWorker.row("edi_identity", agreement.getValue("partnerIdentityId"));
QRecord receiver = OutboundWorker.row("edi_identity", agreement.getValue("localIdentityId"));
if(Boolean.TRUE.equals(sender.getValueBoolean("isActive")) && Boolean.TRUE.equals(receiver.getValueBoolean("isActive"))
&& "EDIFACT".equals(sender.getValueString("standard")) && "EDIFACT".equals(receiver.getValueString("standard"))
&& Objects.equals(sender.getValue("partnerId"), agreement.getValue("partnerId")) && receiver.getValue("partnerId") == null
&& input.unb().value(2).equals(sender.getValueString("interchangeId")) && input.unb().value(3).equals(receiver.getValueString("interchangeId"))
&& input.unb().value(2, 1, 2).equals(Objects.requireNonNullElse(sender.getValueString("interchangeQualifier"), ""))
&& input.unb().value(3, 1, 2).equals(Objects.requireNonNullElse(receiver.getValueString("interchangeQualifier"), "")))
{
return (agreement);
}
}
return (null);
}
/*******************************************************************************
** Preserve retries and serialize reference duplicate checks on the agreement.
*******************************************************************************/
private QRecord interchange(EdifactInterchange input, QRecord agreement, int position) throws Exception
{
var prior = EdiRecords.query("edi_interchange", EdiRecords.equal("inboxId", inbox.getValue("id")).withCriteria("inboundPosition", QCriteriaOperator.EQUALS, position));
if(!prior.isEmpty())
{
QRecord existing = prior.getFirst();
if(agreement != null && existing.getValue("agreementId") == null)
{
existing.withValue("agreementId", agreement.getValue("id")).withValue("partnerId", agreement.getValue("partnerId"));
SystemWrites.update("edi_interchange", existing);
}
return (existing);
}
boolean duplicate = false;
if(agreement != null)
{
SystemWrites.guard("UPDATE edi_agreement SET id=id WHERE id=?", agreement.getValue("id"));
duplicate = EdiRuntime.current().duplicates().edifact(agreement.getValueInteger("id"), input, Objects.requireNonNullElse(agreement.getValueInteger("duplicateWindowDays"), 365), EdiRuntime.current().config().getClock().instant());
SystemWrites.update("edi_inbox", new QRecord().withValue("id", inbox.getValue("id")).withValue("partnerId", agreement.getValue("partnerId")));
}
return (SystemWrites.insert("edi_interchange", new QRecord().withValue("inboxId", inbox.getValue("id")).withValue("inboundPosition", position).withValue("direction", "INBOUND")
.withValue("agreementId", agreement == null ? null : agreement.getValue("id")).withValue("partnerId", agreement == null ? null : agreement.getValue("partnerId"))
.withValue("standard", "EDIFACT").withValue("senderId", input.unb().value(2)).withValue("senderQualifier", input.unb().value(2, 1, 2))
.withValue("receiverId", input.unb().value(3)).withValue("receiverQualifier", input.unb().value(3, 1, 2)).withValue("controlNumber", input.unb().value(5))
.withValue("interchangeVersion", input.unb().value(1, 1, 2)).withValue("usageIndicator", input.usage()).withValue("groupCount", input.groups().size())
.withValue("payloadRef", inbox.getValue("payloadRef")).withValue("payloadSha256", inbox.getValue("payloadSha256")).withValue("sizeBytes", inbox.getValue("sizeBytes"))
.withValue("status", duplicate ? "DUPLICATE" : input.findings().isEmpty() ? "ACCEPTED" : "REJECTED")));
}
/*******************************************************************************
** Match message type and directory independently for every UNH.
*******************************************************************************/
private QRecord document(QRecord agreement, EdifactMessage message) throws Exception
{
var rows = EdiRecords.query("edi_agreement_document", EdiRecords.equal("agreementId", agreement.getValue("id")).withCriteria("direction", QCriteriaOperator.EQUALS, "INBOUND")
.withCriteria("transactionSet", QCriteriaOperator.EQUALS, message.type()).withCriteria("version", QCriteriaOperator.EQUALS, message.version()).withCriteria("isActive", QCriteriaOperator.EQUALS, true));
return (rows.isEmpty() ? null : rows.getFirst());
}
/*******************************************************************************
** Queue at most one CONTRL per received interchange, never acknowledge CONTRL.
*******************************************************************************/
private void acknowledge(EdifactInterchange input, QRecord agreement, QRecord received, Map<String, List<ValidationFinding>> findings, AckLevel level) throws Exception
{
InboxWorker.fenced(inbox, lease, () ->
{
if(EdiRecords.query("edi_outbound_message", EdiRecords.equal("ackForInterchangeId", received.getValue("id"))).isEmpty())
{
String version = input.allMessages().isEmpty() ? "D96A" : input.allMessages().getFirst().version();
var message = new ContrlBuilder().build(input, findings, level, EdifactOutbound.allocate(agreement, "UNH"), version);
EdifactOutbound.envelope(agreement, null, null, List.of(), List.of(message), null, received.getValueInteger("id"));
EdiJournal.event(inbox.getValueInteger("id"), received.getValueInteger("id"), null, "ACK_QUEUED");
}
return (null);
});
}
/*******************************************************************************
** Record a safe positional structural failure.
*******************************************************************************/
private ValidationFinding finding(String text)
{
return (new ValidationFinding(1, "UNH", null, null, null, 1, "3", null, "3", null, "ERROR", text, null));
}
/*******************************************************************************
** Journal failures only while holding the current inbox claim.
*******************************************************************************/
private void problem(QRecord interchange, String type) throws Exception
{
InboxWorker.fenced(inbox, lease, () ->
{
EdiJournal.exception(inbox.getValueInteger("id"), interchange.getValueInteger("id"), null, interchange.getValueInteger("partnerId"), type, "EDIFACT processing requires attention: " + type);
return (null);
});
}
}
@@ -0,0 +1,49 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.List;
import java.util.stream.Stream;
import com.kingsrook.qbits.edi.x12.EnvelopeFinding;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** EDIFACT service envelope with optional groups and ungrouped messages.
*******************************************************************************/
public record EdifactInterchange(EdifactSeparators separators, X12Segment unb, X12Segment unz,
List<EdifactGroup> groups, List<EdifactMessage> messages, List<EnvelopeFinding> findings)
{
/*******************************************************************************
** Visit every message without inventing a functional group on the wire.
*******************************************************************************/
public List<EdifactMessage> allMessages()
{
return (Stream.concat(messages.stream(), groups.stream().flatMap(g -> g.messages().stream())).toList());
}
/*******************************************************************************
** Normalize the UNB test flag to the agreement's usage vocabulary.
*******************************************************************************/
public String usage()
{
return ("1".equals(unb.value(11)) ? "T" : "P");
}
}
@@ -0,0 +1,78 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import com.kingsrook.qbits.edi.map.OutboundMapper;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Reuse mapping transforms and associations with native UNH/UNT envelopes.
*******************************************************************************/
public class EdifactMapper extends OutboundMapper
{
private final char decimalMark;
/*******************************************************************************
** Default numeric formatting follows the default UNA decimal mark.
*******************************************************************************/
public EdifactMapper()
{
this('.');
}
/*******************************************************************************
** Use the agreement's decimal service character for outbound numeric transforms.
*******************************************************************************/
public EdifactMapper(char decimalMark)
{
this.decimalMark = decimalMark;
}
/*******************************************************************************
** Render numeric values using the negotiated decimal mark.
*******************************************************************************/
@Override
protected String decimal(String value)
{
return (value.replace('.', decimalMark));
}
/*******************************************************************************
** Build one directory-specific message with an allocated UNH reference.
*******************************************************************************/
public EdifactMessage message(QRecord source, QRecord map, List<QRecord> rules, String reference) throws Exception
{
if(!"OUTBOUND".equals(map.getValueString("direction")))
{
throw new IllegalArgumentException("An outbound map is required");
}
String version = map.getValueString("version");
List<X12Segment> segments = new ArrayList<>();
segments.add(EdifactWriter.segment("UNH", 1, List.of(reference), List.of(map.getValueString("transactionSet"), version.substring(0, 1), version.substring(1), "UN")));
emit(source, rules.stream().sorted(Comparator.comparing(r -> r.getValueInteger("sequence"))).toList(), "", "", 1, "", new int[] {0}, segments);
segments.add(X12Segment.of("UNT", segments.size() + 1, String.valueOf(segments.size() + 1), reference));
return (new EdifactMessage(segments.getFirst(), segments.getLast(), List.copyOf(segments), List.of()));
}
}
@@ -0,0 +1,56 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.List;
import com.kingsrook.qbits.edi.document.EdiDocument;
import com.kingsrook.qbits.edi.x12.EnvelopeFinding;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** UNH/UNT message with its complete directory and association identification.
*******************************************************************************/
public record EdifactMessage(X12Segment header, X12Segment unt, List<X12Segment> segments, List<EnvelopeFinding> findings) implements EdiDocument
{
/*******************************************************************************
** Agreement document key, for example ORDERS.
*******************************************************************************/
public String type()
{
return (header.value(2));
}
/*******************************************************************************
** Agreement directory key, for example D96A.
*******************************************************************************/
public String version()
{
return (header.value(2, 1, 2) + header.value(2, 1, 3));
}
/*******************************************************************************
** Message reference, unique within its interchange.
*******************************************************************************/
public String reference()
{
return (header.value(1));
}
}
@@ -0,0 +1,168 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.flow.EdiJournal;
import com.kingsrook.qbits.edi.flow.OutboundWorker;
import com.kingsrook.qbits.edi.guide.RuleEvaluator;
import com.kingsrook.qbits.edi.storage.PayloadKind;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** EDIFACT outbound envelopes and durable tracking through the existing send queue.
*******************************************************************************/
public class EdifactOutbound
{
/*******************************************************************************
** Allocate independent UNB/UNH sequences within the configured control scope.
*******************************************************************************/
public static String allocate(QRecord agreement, String level) throws Exception
{
boolean local = "LOCAL_IDENTITY".equals(agreement.getValueString("controlNumberScope"));
return (String.valueOf(EdiRuntime.current().controls().allocate(local ? null : agreement.getValueInteger("id"),
local ? agreement.getValueInteger("localIdentityId") : null, level, 1, 99999999999999L, true)));
}
/*******************************************************************************
** Build an ungrouped interchange; UNZ counts messages when UNG is absent.
*******************************************************************************/
public static EdifactInterchange wrap(QRecord agreement, QRecord sender, QRecord recipient, List<EdifactMessage> messages, String control, LocalDateTime at)
{
String syntax = Objects.requireNonNullElse(agreement.getValueString("interchangeVersion"), "3");
if(!List.of("3", "4").contains(syntax))
{
throw new IllegalArgumentException("EDIFACT syntax version must be 3 or 4");
}
X12Segment unb = EdifactWriter.segment("UNB", 1, List.of(Objects.requireNonNullElse(agreement.getValueString("syntaxIdentifier"), "UNOC"), syntax),
List.of(sender.getValueString("interchangeId"), Objects.requireNonNullElse(sender.getValueString("interchangeQualifier"), "")),
List.of(recipient.getValueString("interchangeId"), Objects.requireNonNullElse(recipient.getValueString("interchangeQualifier"), "")),
List.of(at.format(DateTimeFormatter.ofPattern("4".equals(syntax) ? "yyyyMMdd" : "yyMMdd")), at.format(DateTimeFormatter.ofPattern("HHmm"))),
List.of(control), List.of(""), List.of(""), List.of(""), List.of(""), List.of(""), List.of("T".equals(agreement.getValueString("usageIndicator")) ? "1" : ""));
return (new EdifactInterchange(EdifactSeparators.fromAgreement(agreement), unb, X12Segment.of("UNZ", 0, String.valueOf(messages.size()), control), List.of(), List.copyOf(messages), List.of()));
}
/*******************************************************************************
** Store bytes and record checkpoints atomically with outbound requests or ack fencing.
** A logical tracking group carries ack deadlines even when UNG is absent on the wire.
*******************************************************************************/
public static QRecord envelope(QRecord agreement, QRecord document, QRecord map, List<QRecord> requests, List<EdifactMessage> messages, Integer original, Integer ackFor) throws Exception
{
var runtime = EdiRuntime.current();
var interchange = wrap(agreement, OutboundWorker.row("edi_identity", agreement.getValue("localIdentityId")), OutboundWorker.row("edi_identity", agreement.getValue("partnerIdentityId")),
messages, allocate(agreement, "UNB"), LocalDateTime.ofInstant(runtime.config().getClock().instant(), ZoneOffset.UTC));
String ending = switch(Objects.requireNonNullElse(agreement.getValueString("lineEnding"), "NONE"))
{
case "LF" -> "\n";
case "CRLF" -> "\r\n";
default -> "";
};
var payload = runtime.payloads().put(PayloadKind.OUTBOUND, new ByteArrayInputStream(new EdifactWriter().write(interchange, ending).getBytes(StandardCharsets.UTF_8)));
QRecord persisted = SystemWrites.insert("edi_interchange", new QRecord().withValue("direction", "OUTBOUND").withValue("partnerId", agreement.getValue("partnerId"))
.withValue("agreementId", agreement.getValue("id")).withValue("transportId", agreement.getValue("outboundTransportId")).withValue("standard", "EDIFACT")
.withValue("senderQualifier", interchange.unb().value(2, 1, 2)).withValue("senderId", interchange.unb().value(2))
.withValue("receiverQualifier", interchange.unb().value(3, 1, 2)).withValue("receiverId", interchange.unb().value(3))
.withValue("controlNumber", interchange.unb().value(5)).withValue("interchangeVersion", interchange.unb().value(1, 1, 2)).withValue("usageIndicator", interchange.usage())
.withValue("interchangeDate", runtime.config().getClock().instant()).withValue("groupCount", 0).withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256())
.withValue("sizeBytes", payload.sizeBytes()).withValue("status", "ENVELOPED").withValue("originalInterchangeId", original)
.withValue("containsPhi", document != null && Boolean.TRUE.equals(document.getValueBoolean("containsPhi"))));
boolean expects = document != null && "CONTRL".equals(document.getValueString("ackExpectation"));
QRecord group = SystemWrites.insert("edi_functional_group", new QRecord().withValue("interchangeId", persisted.getValue("id")).withValue("senderId", interchange.unb().value(2))
.withValue("controlNumber", interchange.unb().value(5)).withValue("version", messages.getFirst().version()).withValue("transactionCount", messages.size()).withValue("ackStatus", expects ? "PENDING" : "NONE"));
for(int i = 0; i < messages.size(); i++)
{
var message = messages.get(i);
String key = map == null || map.getValueString("businessKeyPath") == null ? null : new RuleEvaluator(message.segments(), Map.of()).values(map.getValueString("businessKeyPath")).stream().findFirst().orElse(null);
SystemWrites.insert("edi_transaction", new QRecord().withValue("interchangeId", persisted.getValue("id")).withValue("functionalGroupId", group.getValue("id"))
.withValue("direction", "OUTBOUND").withValue("partnerId", agreement.getValue("partnerId")).withValue("agreementDocumentId", document == null ? null : document.getValue("id"))
.withValue("transactionSet", message.type()).withValue("version", message.version()).withValue("controlNumber", message.reference()).withValue("status", "QUEUED")
.withValue("guideId", document == null ? null : document.getValue("guideId"))
.withValue("guideRevision", document == null || document.getValue("guideId") == null ? null : OutboundWorker.row("edi_guide", document.getValue("guideId")).getValue("revision")).withValue("mapId", map == null ? null : map.getValue("id"))
.withValue("mapRevision", map == null ? null : map.getValue("revision"))
.withValue(document != null && Boolean.TRUE.equals(document.getValueBoolean("containsPhi")) ? "phiBusinessKey" : "businessKey", key)
.withValue("businessKeyType", map == null ? null : map.getValue("businessKeyType"))
.withValue("containsPhi", document != null && Boolean.TRUE.equals(document.getValueBoolean("containsPhi"))).withValue("segmentCount", message.segments().size())
.withValue("targetTableName", i < requests.size() ? requests.get(i).getValue("sourceTableName") : null).withValue("targetRecordId", i < requests.size() ? requests.get(i).getValue("sourceRecordId") : null));
if(i < requests.size())
{
SystemWrites.update("edi_outbound_request", new QRecord().withValue("id", requests.get(i).getValue("id")).withValue("status", "ENVELOPED"));
}
}
QRecord queued = SystemWrites.insert("edi_outbound_message", new QRecord().withValue("interchangeId", persisted.getValue("id")).withValue("agreementId", agreement.getValue("id"))
.withValue("partnerId", agreement.getValue("partnerId")).withValue("transportId", agreement.getValue("outboundTransportId")).withValue("status", "QUEUED").withValue("attempts", 0)
.withValue("ackForInterchangeId", ackFor).withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes()));
SystemWrites.update("edi_interchange", new QRecord().withValue("id", persisted.getValue("id")).withValue("outboundMessageId", queued.getValue("id")));
EdiJournal.event(null, persisted.getValueInteger("id"), null, "ENVELOPED");
return (queued);
}
/*******************************************************************************
** Re-envelope archived outbound messages without reading changed source records.
*******************************************************************************/
public static void resendAsNew(QRecord original) throws Exception
{
QRecord agreement = OutboundWorker.row("edi_agreement", original.getValue("agreementId"));
QRecord oldInterchange = OutboundWorker.row("edi_interchange", original.getValue("interchangeId"));
EdifactInterchange archive;
try(var stream = EdiRuntime.current().payloads().open(new com.kingsrook.qbits.edi.storage.PayloadRef(original.getValueString("payloadRef"), null, 0)))
{
archive = new EdifactEnvelopeReader(EdiRuntime.current().config().getMaxSegmentsPerTransaction()).read(stream).getFirst();
}
List<EdifactMessage> messages = new java.util.ArrayList<>();
for(var message : archive.allMessages())
{
String reference = allocate(agreement, "UNH");
var segments = new java.util.ArrayList<>(message.segments());
var elements = new java.util.ArrayList<>(message.header().elements());
elements.set(0, List.of(List.of(reference)));
segments.set(0, new X12Segment("UNH", 1, elements));
segments.set(segments.size() - 1, X12Segment.of("UNT", segments.size(), String.valueOf(segments.size()), reference));
messages.add(new EdifactMessage(segments.getFirst(), segments.getLast(), List.copyOf(segments), List.of()));
}
var oldTransactions = com.kingsrook.qbits.edi.storage.EdiRecords.query("edi_transaction", com.kingsrook.qbits.edi.storage.EdiRecords.equal("interchangeId", oldInterchange.getValue("id")));
QRecord first = oldTransactions.getFirst();
QRecord document = first.getValue("agreementDocumentId") == null ? null : OutboundWorker.row("edi_agreement_document", first.getValue("agreementDocumentId"));
QRecord queued = envelope(agreement, document, null, List.of(), messages, oldInterchange.getValueInteger("id"), null);
var newTransactions = com.kingsrook.qbits.edi.storage.EdiRecords.query("edi_transaction", com.kingsrook.qbits.edi.storage.EdiRecords.equal("interchangeId", queued.getValue("interchangeId")));
for(int i = 0; i < archive.allMessages().size(); i++)
{
String reference = archive.allMessages().get(i).reference();
QRecord old = oldTransactions.stream().filter(t -> reference.equals(t.getValueString("controlNumber"))).findFirst().orElseThrow();
QRecord update = new QRecord().withValue("id", newTransactions.get(i).getValue("id"));
for(String field : List.of("guideId", "guideRevision", "mapId", "mapRevision", "businessKey", "phiBusinessKey", "containsPhi", "businessKeyType", "secondaryKey", "targetTableName", "targetRecordId", "agreementDocumentId"))
{
update.withValue(field, old.getValue(field));
}
SystemWrites.update("edi_transaction", update);
}
}
}
@@ -0,0 +1,89 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** UNA service characters; the fifth character is reserved in syntax version 3.
*******************************************************************************/
public record EdifactSeparators(char component, char element, char decimal, char release, char repetition, char segment)
{
public static final EdifactSeparators DEFAULT = new EdifactSeparators(':', '+', '.', '?', ' ', '\'');
/*******************************************************************************
** Ambiguous service characters cannot be safely parsed or escaped.
*******************************************************************************/
public EdifactSeparators
{
var values = List.of(component, element, decimal, release, repetition, segment);
if(!List.of('.', ',').contains(decimal) || new HashSet<>(values).size() != 6 || values.stream().anyMatch(c -> c > 127 || Character.isISOControl(c)))
{
throw new IllegalArgumentException("Invalid EDIFACT service characters");
}
}
/*******************************************************************************
** Read exactly six UNA service characters.
*******************************************************************************/
public static EdifactSeparators fromUna(String una)
{
if(una.length() != 9 || !una.startsWith("UNA"))
{
throw new IllegalArgumentException("Invalid UNA");
}
return (new EdifactSeparators(una.charAt(3), una.charAt(4), una.charAt(5), una.charAt(6), una.charAt(7), una.charAt(8)));
}
/*******************************************************************************
** Use the agreement's EDIFACT service characters.
*******************************************************************************/
public static EdifactSeparators fromAgreement(QRecord agreement)
{
return (new EdifactSeparators(character(agreement, "componentSeparator", ":"), character(agreement, "elementSeparator", "+"),
character(agreement, "decimalMark", "."), character(agreement, "releaseCharacter", "?"),
"4".equals(agreement.getValueString("interchangeVersion")) ? character(agreement, "repetitionSeparator", "*") : ' ', character(agreement, "segmentTerminator", "'")));
}
/*******************************************************************************
** A separator setting must contain exactly one character.
*******************************************************************************/
private static char character(QRecord agreement, String field, String fallback)
{
String value = Objects.requireNonNullElse(agreement.getValueString(field), fallback);
if(value.length() != 1)
{
throw new IllegalArgumentException("Invalid service character setting");
}
return (value.charAt(0));
}
/*******************************************************************************
** Render service string advice, including its segment terminator.
*******************************************************************************/
public String una()
{
return ("UNA" + component + element + decimal + release + repetition + segment);
}
}
@@ -0,0 +1,115 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.edifact;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** In-memory EDIFACT serialization; escape values, never composite structure.
*******************************************************************************/
public class EdifactWriter
{
/*******************************************************************************
** Render a full interchange with explicit UNA advice.
*******************************************************************************/
public String write(EdifactInterchange interchange)
{
return (write(interchange, ""));
}
/*******************************************************************************
** Retain optional groups and caller-selected line endings.
*******************************************************************************/
public String write(EdifactInterchange interchange, String ending)
{
List<X12Segment> segments = new ArrayList<>();
segments.add(interchange.unb());
interchange.messages().forEach(m -> segments.addAll(m.segments()));
for(var group : interchange.groups())
{
segments.add(group.ung());
group.messages().forEach(m -> segments.addAll(m.segments()));
segments.add(group.une());
}
segments.add(interchange.unz());
return (interchange.separators().una() + ending + write(segments, interchange.separators(), ending));
}
/*******************************************************************************
** Serialize component and repetition lists without losing released delimiters.
*******************************************************************************/
public String write(List<X12Segment> segments, EdifactSeparators separators, String ending)
{
if(!List.of("", "\n", "\r\n").contains(ending))
{
throw new IllegalArgumentException("Unsupported line ending");
}
StringBuilder out = new StringBuilder();
for(var segment : segments)
{
out.append(segment.id());
for(var element : segment.elements())
{
out.append(separators.element());
for(int r = 0; r < element.size(); r++)
{
if(r > 0)
{
if(separators.repetition() == ' ')
{
throw new IllegalArgumentException("Repetition requires syntax version 4");
}
out.append(separators.repetition());
}
for(int c = 0; c < element.get(r).size(); c++)
{
if(c > 0)
{
out.append(separators.component());
}
for(char value : element.get(r).get(c).toCharArray())
{
if(value == separators.release() || value == separators.element() || value == separators.component()
|| value == separators.segment() || separators.repetition() != ' ' && value == separators.repetition())
{
out.append(separators.release());
}
out.append(value);
}
}
}
}
out.append(separators.segment()).append(ending);
}
return (out.toString());
}
/*******************************************************************************
** Construct a segment from explicit component lists, with no delimiter parsing.
*******************************************************************************/
@SafeVarargs
public static X12Segment segment(String tag, int position, List<String>... elements)
{
return (new X12Segment(tag, position, java.util.Arrays.stream(elements).map(List::of).toList()));
}
}
@@ -0,0 +1,145 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.time.temporal.ChronoUnit;
import java.util.List;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Five-minute acknowledgement and public-certificate expiry monitor.
*******************************************************************************/
public class AckMonitor
{
/*******************************************************************************
** Deduplicate alerts and open exceptions without embedding document values.
*******************************************************************************/
public Integer runOnce() throws QException
{
return (SystemWrites.call(() ->
{
int count = 0;
var now = EdiRuntime.current().config().getClock().instant();
for(QRecord group : EdiRecords.query("edi_functional_group", EdiRecords.equal("ackStatus", "PENDING")
.withCriteria("ackDueAt", QCriteriaOperator.LESS_THAN, now).withCriteria("ackReceivedAt", QCriteriaOperator.IS_BLANK)))
{
QRecord interchange = OutboundWorker.row("edi_interchange", group.getValue("interchangeId"));
if(!"OUTBOUND".equals(interchange.getValueString("direction")))
{
continue;
}
String key = "edi_functional_group:" + group.getValue("id");
QAlerts.raise(new AlertSignal("edi.ack.overdue").withSeverity(AlertSeverity.WARNING).withDedupKey(key)
.withRecordRef(new com.kingsrook.qqq.backend.core.alerts.AlertRecordRef("edi_functional_group", group.getValue("id")))
.withTitle("EDI acknowledgement overdue").withSummary("Review the outbound functional group"));
if(EdiRecords.query("edi_exception", EdiRecords.equal("interchangeId", interchange.getValue("id")).withCriteria("type", QCriteriaOperator.EQUALS, "OVERDUE_ACK")
.withCriteria("status", QCriteriaOperator.IN, "OPEN", "IN_PROGRESS")).isEmpty())
{
EdiJournal.exception(null, interchange.getValueInteger("id"), null, interchange.getValueInteger("partnerId"), "OVERDUE_ACK", "Expected acknowledgement has not arrived");
}
count++;
}
for(QRecord certificate : EdiRecords.query("edi_certificate", new QQueryFilter().withCriteria("status", QCriteriaOperator.IN, "ACTIVE", "NEXT")))
{
if(certificate.getValueInstant("notAfter") == null)
{
continue;
}
long days = ChronoUnit.DAYS.between(now, certificate.getValueInstant("notAfter"));
if(!certificate.getValueInstant("notAfter").isAfter(now))
{
expiry(certificate, "edi.certificate.expired", "expired", AlertSeverity.ERROR);
}
else
{
for(int threshold : List.of(30, 14, 7))
{
if(days < threshold)
{
expiry(certificate, "edi.certificate.expiring", String.valueOf(threshold), AlertSeverity.WARNING);
}
}
}
}
return (count + overdueMdns(now));
}));
}
/*******************************************************************************
** SENT messages on an asynchronous-MDN AS2 transport whose MDN has not arrived
** within the agreement document ackDueMinutes (60 when none is set) raise a
** warning, deduplicated per message and resolved when the MDN arrives.
*******************************************************************************/
private static int overdueMdns(java.time.Instant now) throws QException
{
List<java.io.Serializable> transports = new java.util.ArrayList<>();
for(QRecord transport : EdiRecords.query("edi_transport", EdiRecords.equal("type", "AS2").withCriteria("mdnMode", QCriteriaOperator.EQUALS, "ASYNC")))
{
transports.add(transport.getValue("id"));
}
if(transports.isEmpty())
{
return (0);
}
int count = 0;
for(QRecord message : EdiRecords.query("edi_outbound_message", EdiRecords.equal("status", "SENT").withCriteria("transportId", QCriteriaOperator.IN, transports)
.withCriteria("sentAt", QCriteriaOperator.IS_NOT_BLANK)))
{
int minutes = 60;
Integer smallest = null;
for(QRecord document : message.getValue("agreementId") == null ? List.<QRecord>of()
: EdiRecords.query("edi_agreement_document", EdiRecords.equal("agreementId", message.getValue("agreementId")).withCriteria("direction", QCriteriaOperator.EQUALS, "OUTBOUND")))
{
Integer due = document.getValueInteger("ackDueMinutes");
smallest = due == null ? smallest : smallest == null ? due : Math.min(smallest, due);
}
minutes = smallest == null ? minutes : smallest;
if(message.getValueInstant("sentAt").plus(minutes, ChronoUnit.MINUTES).isAfter(now))
{
continue;
}
QAlerts.raise(new AlertSignal("edi.mdn.overdue").withSeverity(AlertSeverity.WARNING).withDedupKey("edi_outbound_message:" + message.getValue("id"))
.withRecordRef(new com.kingsrook.qqq.backend.core.alerts.AlertRecordRef("edi_outbound_message", message.getValue("id")))
.withTitle("AS2 receipt overdue").withSummary("The asynchronous MDN has not arrived; review the outbound message"));
count++;
}
return (count);
}
/*******************************************************************************
** Each certificate threshold has its own stable deduplication key.
*******************************************************************************/
private static void expiry(QRecord certificate, String type, String threshold, AlertSeverity severity)
{
QAlerts.raise(new AlertSignal(type).withSeverity(severity).withDedupKey("edi_certificate:" + certificate.getValue("id") + ":" + threshold)
.withRecordRef(new com.kingsrook.qqq.backend.core.alerts.AlertRecordRef("edi_certificate", certificate.getValue("id")))
.withTitle("EDI certificate expiry").withSummary("Review the certificate validity dates"));
}
}
@@ -0,0 +1,89 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.util.List;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.alerts.NotificationAudience;
import com.kingsrook.qqq.backend.core.alerts.NotificationSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Value-free operational timeline and coalesced exception work queue.
*******************************************************************************/
public final class EdiJournal
{
/*******************************************************************************
** Events contain only step names, record references and timestamps.
*******************************************************************************/
public static void event(Integer inbox, Integer interchange, Integer transaction, String type) throws Exception
{
SystemWrites.insert("edi_event", new QRecord().withValue("inboxId", inbox).withValue("interchangeId", interchange)
.withValue("transactionId", transaction).withValue("at", EdiRuntime.current().config().getClock().instant())
.withValue("type", type).withValue("detail", type).withValue("userName", "edi"));
}
/*******************************************************************************
** Repeated failures of the same item increment occurrences without exposing values.
*******************************************************************************/
public static void exception(Integer inbox, Integer interchange, Integer transaction, Integer partner, String type, String summary) throws Exception
{
QQueryFilter filter = EdiRecords.equal("type", type);
for(var pair : List.of(new Object[] {"inboxId", inbox}, new Object[] {"interchangeId", interchange}, new Object[] {"transactionId", transaction}))
{
filter.withCriteria(pair[0].toString(), pair[1] == null ? QCriteriaOperator.IS_BLANK : QCriteriaOperator.EQUALS, (Integer) pair[1]);
}
List<QRecord> matches = EdiRecords.query("edi_exception", filter);
QRecord exception;
boolean opened = matches.isEmpty();
if(opened)
{
exception = SystemWrites.insert("edi_exception", new QRecord().withValue("inboxId", inbox).withValue("interchangeId", interchange).withValue("transactionId", transaction)
.withValue("partnerId", partner).withValue("type", type).withValue("summary", summary).withValue("status", "OPEN").withValue("occurrences", 1));
}
else
{
QRecord row = matches.getFirst();
exception = row;
opened = List.of("RESOLVED", "IGNORED").contains(row.getValueString("status"));
SystemWrites.update("edi_exception", new QRecord().withValue("id", row.getValue("id")).withValue("occurrences", row.getValueInteger("occurrences") + 1).withValue("status", "OPEN"));
}
if(opened && partner != null && List.of("REJECTED_ACK", "DEAD_LETTER").contains(type))
{
String[] assignees = EdiRecords.query("edi_exception", EdiRecords.equal("partnerId", partner)
.withCriteria("status", QCriteriaOperator.IN, "OPEN", "IN_PROGRESS")).stream()
.map(row -> row.getValueString("assignedToUserId")).filter(user -> user != null && !user.isBlank()).distinct().toArray(String[]::new);
if(assignees.length > 0)
{
QAlerts.notify(new NotificationSignal("EDI exception opened")
.withBody("An acknowledgement rejection or dead-letter exception needs attention.")
.withAudience(NotificationAudience.users(assignees))
.withSourceName("edi").withSourceTableName("edi_exception").withSourceRecordId(exception.getValueString("id")));
}
}
event(inbox, interchange, transaction, "EXCEPTION");
}
}
@@ -0,0 +1,38 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.io.InputStream;
/*******************************************************************************
** Transport-neutral stream. The receiving transport owns and closes the stream.
*******************************************************************************/
public record InboundPayload(String transportType, String externalId, String fileName, String remoteAddress,
InputStream stream, String gatewayUserId, String username)
{
/*******************************************************************************
** Payload without gateway identity.
*******************************************************************************/
public InboundPayload(String transportType, String externalId, String fileName, String remoteAddress, InputStream stream)
{
this(transportType, externalId, fileName, remoteAddress, stream, null, null);
}
}
@@ -0,0 +1,592 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.time.LocalDateTime;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.ack.AckEnvelope;
import com.kingsrook.qbits.edi.ack.AckLevel;
import com.kingsrook.qbits.edi.guide.Guide;
import com.kingsrook.qbits.edi.guide.GuideRepository;
import com.kingsrook.qbits.edi.guide.ValidationFinding;
import com.kingsrook.qbits.edi.map.InboundApplier;
import com.kingsrook.qbits.edi.map.InboundMapper;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadKind;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.storage.PayloadStore;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12Interchange;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Routing, validation, acknowledgements and service-session application checkpoints.
*******************************************************************************/
public class InboundProcessor
{
private final QRecord inbox;
private final Instant lease;
private final PayloadStore store;
private final EdiRuntime runtime = EdiRuntime.current();
/*******************************************************************************
** One processor per claimed payload; database checkpoints fence every side effect.
*******************************************************************************/
public InboundProcessor(QRecord inbox, Instant lease, PayloadStore store)
{
this.inbox = inbox;
this.lease = lease;
this.store = store;
}
/*******************************************************************************
** Route before mapping; unknown and mismatched identities never reach target writes.
*******************************************************************************/
public void process(X12Interchange x12, int position) throws Exception
{
QRecord agreement = route(x12);
QRecord interchange = InboxWorker.fenced(inbox, lease, () -> interchange(x12, agreement, position));
Integer id = interchange.getValueInteger("id");
if(agreement == null)
{
problem(id, null, null, "UNKNOWN_PARTNER", "No active agreement matches the interchange identities and usage");
return;
}
Integer partner = agreement.getValueInteger("partnerId");
if(inbox.getValueString("username") != null)
{
var linked = EdiRecords.query("edi_partner", EdiRecords.equal("gatewayUsername", inbox.getValue("username")));
if(linked.stream().anyMatch(p -> !partner.equals(p.getValueInteger("id"))))
{
problem(id, null, partner, "PARTNER_MISMATCH", "Gateway identity does not match the routed trading partner");
return;
}
}
String code = x12.findings().stream().filter(f -> "TA1".equals(f.level())).map(f -> f.code()).findFirst().orElse("000");
boolean duplicate = "DUPLICATE".equals(interchange.getValueString("status"));
if(duplicate)
{
code = "025";
problem(id, null, partner, "DUPLICATE", "Duplicate interchange control number");
}
boolean ackInput = x12.serviceSegments().stream().anyMatch(s -> "TA1".equals(s.id())) || x12.groups().stream().anyMatch(g -> !g.transactions().isEmpty()) && x12.groups().stream().flatMap(g -> g.transactions().stream()).allMatch(t -> List.of("997", "999").contains(t.st().value(1)));
String ta1Code = code;
String policy = agreement.getValueString("sendTa1");
boolean send = !ackInput && ("ALWAYS".equals(policy) || "ON_REQUEST".equals(policy) && "1".equals(x12.isa().value(14))
|| "ON_ERROR".equals(policy) && !"000".equals(code));
if(duplicate && "REJECT_TA1".equals(agreement.getValueString("onDuplicate")))
{
send = true;
}
if(send)
{
InboxWorker.fenced(inbox, lease, () ->
{
if(EdiRecords.query("edi_outbound_message", EdiRecords.equal("ackForInterchangeId", id)).isEmpty())
{
queue(runtime.ta1().interchange(x12, control(agreement, "ISA"), ta1Code, LocalDateTime.now(runtime.config().getClock())), agreement, id, null, "TA1");
}
return (null);
});
}
if(duplicate && !"PROCESS".equals(agreement.getValueString("onDuplicate")))
{
return;
}
if(!"000".equals(code) && !duplicate)
{
problem(id, null, partner, "ENVELOPE", "Interchange envelope failed validation");
return;
}
int groupPosition = 0;
for(X12FunctionalGroup group : x12.groups())
{
int currentGroupPosition = ++groupPosition;
QRecord groupRow = InboxWorker.fenced(inbox, lease, () -> group(group, interchange, agreement, currentGroupPosition));
Map<String, List<ValidationFinding>> findings = new LinkedHashMap<>();
QRecord ackDocument = null;
int transactionPosition = 0;
for(X12TransactionSet transaction : group.transactions())
{
QRecord document = document(agreement, transaction, group);
if(document != null && !"NONE".equals(document.getValueString("ackExpectation")))
{
ackDocument = document;
}
QRecord configuredDocument = document;
int currentTransactionPosition = ++transactionPosition;
QRecord tx = InboxWorker.fenced(inbox, lease, () -> transaction(transaction, groupRow, interchange, configuredDocument, currentTransactionPosition));
if(List.of("997", "999").contains(transaction.st().value(1)))
{
continue;
}
document = revisions(document, tx);
QRecord effectiveDocument = document;
List<ValidationFinding> result = validate(transaction, group, document, groupRow);
findings.put(transaction.st().value(2), result);
InboxWorker.fenced(inbox, lease, () ->
{
validation(tx, result, effectiveDocument);
return (null);
});
if(document == null)
{
problem(id, tx.getValueInteger("id"), partner, "NOT_CONFIGURED", "Inbound document has no active agreement configuration");
continue;
}
if(result.stream().anyMatch(f -> "ERROR".equals(f.severity()) && f.snipLevel() <= 2))
{
continue;
}
if("T".equals(x12.isa().value(15)) && !Boolean.TRUE.equals(agreement.getValueBoolean("applyTestDocuments")))
{
continue;
}
apply(transaction, tx, document, agreement);
}
if(!ackInput && ackDocument != null)
{
QRecord config = ackDocument;
InboxWorker.fenced(inbox, lease, () ->
{
if(EdiRecords.query("edi_outbound_message", EdiRecords.equal("ackForGroupId", groupRow.getValue("id"))).isEmpty())
{
AckLevel level = AckLevel.valueOf(Objects.requireNonNullElse(config.getValueString("ackLevel"), "ERRORS_ONLY"));
boolean hipaa = group.gs().value(8).startsWith("005010X") || "999".equals(config.getValueString("ackExpectation"));
var ack = hipaa ? runtime.ack999().build(group, findings, level, "0001") : runtime.ack997().build(group, findings, level, "0001");
queue(AckEnvelope.wrap(x12, group, ack, control(agreement, "ISA"), control(agreement, "GS"), LocalDateTime.now(runtime.config().getClock())),
agreement, id, groupRow.getValueInteger("id"), hipaa ? "999" : "997");
}
return (null);
});
}
}
if(ackInput)
{
InboxWorker.fenced(inbox, lease, () ->
{
if(x12.serviceSegments().stream().anyMatch(s -> "TA1".equals(s.id()))
&& EdiRecords.query("edi_transaction", EdiRecords.equal("interchangeId", id)).isEmpty())
{
SystemWrites.insert("edi_transaction", new QRecord().withValue("interchangeId", id).withValue("partnerId", partner)
.withValue("direction", "INBOUND").withValue("transactionSet", "TA1").withValue("controlNumber", x12.isa().value(13)).withValue("status", "RECEIVED"));
}
reconcile(x12, agreement, id);
return (null);
});
}
}
/*******************************************************************************
** Route on sender, receiver, qualifier, standard and test/production indicator.
*******************************************************************************/
private QRecord route(X12Interchange x12) throws Exception
{
for(QRecord agreement : EdiRecords.query("edi_agreement", EdiRecords.equal("isActive", true).withCriteria("standard", QCriteriaOperator.EQUALS, "X12")
.withCriteria("usageIndicator", QCriteriaOperator.EQUALS, x12.isa().value(15))))
{
QRecord sender = row("edi_identity", agreement.getValueInteger("partnerIdentityId"));
QRecord receiver = row("edi_identity", agreement.getValueInteger("localIdentityId"));
if(sender != null && receiver != null && !Boolean.FALSE.equals(sender.getValueBoolean("isActive")) && !Boolean.FALSE.equals(receiver.getValueBoolean("isActive"))
&& Objects.equals(sender.getValueInteger("partnerId"), agreement.getValueInteger("partnerId"))
&& receiver.getValue("partnerId") == null && x12.isa().value(5).equals(sender.getValueString("interchangeQualifier"))
&& x12.isa().value(6).trim().equals(sender.getValueString("interchangeId")) && x12.isa().value(7).equals(receiver.getValueString("interchangeQualifier"))
&& x12.isa().value(8).trim().equals(receiver.getValueString("interchangeId")))
{
return (agreement);
}
}
return (null);
}
/*******************************************************************************
** Persist each interchange once; serialize duplicate checks on its agreement.
*******************************************************************************/
private QRecord interchange(X12Interchange x12, QRecord agreement, int position) throws Exception
{
var prior = EdiRecords.query("edi_interchange", EdiRecords.equal("inboxId", inbox.getValue("id")).withCriteria("inboundPosition", QCriteriaOperator.EQUALS, position));
if(!prior.isEmpty())
{
QRecord existing = prior.getFirst();
if(agreement != null && existing.getValue("agreementId") == null)
{
existing.withValue("agreementId", agreement.getValue("id")).withValue("partnerId", agreement.getValue("partnerId"));
SystemWrites.update("edi_interchange", existing);
SystemWrites.update("edi_inbox", new QRecord().withValue("id", inbox.getValue("id")).withValue("partnerId", agreement.getValue("partnerId")));
}
return (existing);
}
boolean duplicate = false;
if(agreement != null)
{
SystemWrites.guard("UPDATE edi_agreement SET id=id WHERE id=?", agreement.getValue("id"));
duplicate = runtime.duplicates().interchange(agreement.getValueInteger("id"), x12.isa(), agreement.getValueInteger("duplicateWindowDays"), runtime.config().getClock().instant());
SystemWrites.update("edi_inbox", new QRecord().withValue("id", inbox.getValue("id")).withValue("partnerId", agreement.getValue("partnerId")));
}
QRecord row = SystemWrites.insert("edi_interchange", new QRecord().withValue("inboundPosition", position).withValue("inboxId", inbox.getValue("id")).withValue("direction", "INBOUND")
.withValue("agreementId", agreement == null ? null : agreement.getValue("id")).withValue("partnerId", agreement == null ? null : agreement.getValue("partnerId"))
.withValue("standard", "X12").withValue("senderQualifier", x12.isa().value(5)).withValue("senderId", x12.isa().value(6).trim())
.withValue("receiverQualifier", x12.isa().value(7)).withValue("receiverId", x12.isa().value(8).trim()).withValue("controlNumber", x12.isa().value(13))
.withValue("interchangeVersion", x12.isa().value(12)).withValue("usageIndicator", x12.isa().value(15)).withValue("groupCount", x12.groups().size())
.withValue("payloadRef", inbox.getValue("payloadRef")).withValue("payloadSha256", inbox.getValue("payloadSha256")).withValue("sizeBytes", inbox.getValue("sizeBytes"))
.withValue("containsPhi", x12.groups().stream().anyMatch(g -> g.gs().value(8).startsWith("005010X")))
.withValue("status", duplicate ? "DUPLICATE" : x12.findings().isEmpty() ? "ACCEPTED" : "REJECTED"));
EdiJournal.event(inbox.getValueInteger("id"), row.getValueInteger("id"), null, "PARSED");
return (row);
}
/*******************************************************************************
** Group checkpoints retain sender and control duplicate evidence.
*******************************************************************************/
private QRecord group(X12FunctionalGroup group, QRecord interchange, QRecord agreement, int position) throws Exception
{
QQueryFilter filter = EdiRecords.equal("interchangeId", interchange.getValue("id")).withCriteria("inboundPosition", QCriteriaOperator.EQUALS, position);
var prior = EdiRecords.query("edi_functional_group", filter);
if(!prior.isEmpty())
{
return (prior.getFirst());
}
boolean duplicate = runtime.duplicates().group(group.gs().value(2), group.gs().value(6), agreement.getValueInteger("duplicateWindowDays"), runtime.config().getClock().instant());
return (SystemWrites.insert("edi_functional_group", new QRecord().withValue("inboundPosition", position).withValue("interchangeId", interchange.getValue("id")).withValue("senderId", group.gs().value(2))
.withValue("functionalIdCode", group.gs().value(1)).withValue("controlNumber", group.gs().value(6)).withValue("version", group.gs().value(8))
.withValue("transactionCount", group.transactions().size()).withValue("ackStatus", duplicate && !"PROCESS".equals(agreement.getValueString("onDuplicate")) ? "R" : "NONE")));
}
/*******************************************************************************
** Select the inbound document; outbound ack expectation never controls our reply.
*******************************************************************************/
private QRecord document(QRecord agreement, X12TransactionSet tx, X12FunctionalGroup group) throws Exception
{
var rows = EdiRecords.query("edi_agreement_document", EdiRecords.equal("agreementId", agreement.getValue("id"))
.withCriteria("direction", QCriteriaOperator.EQUALS, "INBOUND").withCriteria("transactionSet", QCriteriaOperator.EQUALS, tx.st().value(1))
.withCriteria("version", QCriteriaOperator.EQUALS, group.gs().value(8)).withCriteria("isActive", QCriteriaOperator.EQUALS, true));
return (rows.isEmpty() ? null : rows.getFirst());
}
/*******************************************************************************
** Transaction checkpoints preserve guide and map references across worker retries.
*******************************************************************************/
private QRecord transaction(X12TransactionSet tx, QRecord group, QRecord interchange, QRecord document, int position) throws Exception
{
var rows = EdiRecords.query("edi_transaction", EdiRecords.equal("functionalGroupId", group.getValue("id")).withCriteria("inboundPosition", QCriteriaOperator.EQUALS, position));
if(!rows.isEmpty())
{
return (rows.getFirst());
}
return (SystemWrites.insert("edi_transaction", new QRecord().withValue("inboundPosition", position).withValue("interchangeId", interchange.getValue("id")).withValue("functionalGroupId", group.getValue("id"))
.withValue("direction", "INBOUND").withValue("partnerId", interchange.getValue("partnerId")).withValue("transactionSet", tx.st().value(1))
.withValue("version", group.getValue("version")).withValue("controlNumber", tx.st().value(2)).withValue("segmentCount", tx.segments().size())
.withValue("agreementDocumentId", document == null ? null : document.getValue("id")).withValue("containsPhi", group.getValueString("version").startsWith("005010X") || document != null && Boolean.TRUE.equals(document.getValueBoolean("containsPhi")))
.withValue("status", "RECEIVED")));
}
/*******************************************************************************
** Replays select original persisted definitions or the currently active revision.
*******************************************************************************/
protected QRecord revisions(QRecord document, QRecord transaction) throws Exception
{
if(document == null)
{
return (null);
}
QRecord effective = new QRecord();
effective.setValues(new java.util.HashMap<>(document.getValues()));
for(String kind : List.of("guide", "map"))
{
Integer originalId = transaction.getValueInteger(kind + "Id");
boolean current = Boolean.TRUE.equals(inbox.getValueBoolean("useCurrentRevisions"));
if(originalId != null && !current)
{
effective.withValue(kind + "Id", originalId);
}
else if(current)
{
QRecord referenced = row("edi_" + kind, document.getValueInteger(kind + "Id"));
if(referenced != null)
{
var active = EdiRecords.query("edi_" + kind, EdiRecords.equal("name", referenced.getValue("name")).withCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"));
active.stream().filter(r -> Objects.equals(r.getValue("partnerId"), referenced.getValue("partnerId"))).findFirst()
.ifPresent(r -> effective.withValue(kind + "Id", r.getValue("id")));
}
}
}
return (effective);
}
/*******************************************************************************
** Gather positional findings, including group and repeated transaction controls.
*******************************************************************************/
private List<ValidationFinding> validate(X12TransactionSet tx, X12FunctionalGroup group, QRecord document, QRecord groupRow) throws Exception
{
List<ValidationFinding> findings = new ArrayList<>();
if(document != null && document.getValueInteger("guideId") != null)
{
findings.addAll(runtime.validator().validate(tx, new GuideRepository().load(document.getValueInteger("guideId")), Boolean.TRUE.equals(document.getValueBoolean("containsPhi")) || group.gs().value(8).startsWith("005010X")));
int snip = Objects.requireNonNullElse(document.getValueInteger("snipLevel"), 2);
findings.removeIf(f -> f.snipLevel() > snip);
}
else
{
findings.add(finding("No implementation guide is configured", "3"));
}
if("R".equals(groupRow.getValueString("ackStatus")) || runtime.duplicates().transactions(group).contains(tx.st().value(2)))
{
findings.add(finding("Duplicate control number", "7"));
}
if(!group.findings().isEmpty() || !tx.findings().isEmpty())
{
findings.add(finding("Envelope counts or controls failed validation", "7"));
}
return (findings);
}
/*******************************************************************************
** Construct a value-free structural error.
*******************************************************************************/
private static ValidationFinding finding(String message, String code)
{
return (new ValidationFinding(1, "ST", null, null, null, 1, code, null, code, null, "ERROR", message, null));
}
/*******************************************************************************
** Persist findings without repeatedly adding error rows on storage retries.
*******************************************************************************/
protected void validation(QRecord tx, List<ValidationFinding> findings, QRecord document) throws Exception
{
boolean rejected = findings.stream().anyMatch(f -> "ERROR".equals(f.severity()) && f.snipLevel() <= 2);
if(!"APPLIED".equals(tx.getValueString("status")) || Boolean.TRUE.equals(inbox.getValueBoolean("reprocess")))
{
QRecord guide = document == null ? null : row("edi_guide", document.getValueInteger("guideId"));
SystemWrites.update("edi_transaction", new QRecord().withValue("id", tx.getValue("id")).withValue("status", rejected ? "INVALID" : "VALID")
.withValue("ackStatus", rejected ? "R" : findings.isEmpty() ? "A" : "E").withValue("errorCount", (int) findings.stream().filter(f -> "ERROR".equals(f.severity())).count())
.withValue("warningCount", (int) findings.stream().filter(f -> !"ERROR".equals(f.severity())).count())
.withValue("guideId", guide == null ? null : guide.getValue("id")).withValue("guideRevision", guide == null ? null : guide.getValue("revision"))
.withValue("mapId", document == null ? null : document.getValue("mapId"))
.withValue("mapRevision", document == null || row("edi_map", document.getValueInteger("mapId")) == null ? null : row("edi_map", document.getValueInteger("mapId")).getValue("revision")));
}
if(Boolean.TRUE.equals(inbox.getValueBoolean("reprocess")))
{
new com.kingsrook.qqq.backend.core.actions.tables.DeleteAction().execute(new com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput("edi_transaction_error")
.withQueryFilter(EdiRecords.equal("transactionId", tx.getValue("id"))).withTransaction(com.kingsrook.qqq.backend.core.context.QContext.getQBackendTransaction()));
}
if(EdiRecords.query("edi_transaction_error", EdiRecords.equal("transactionId", tx.getValue("id"))).isEmpty())
{
for(ValidationFinding f : findings)
{
SystemWrites.insert("edi_transaction_error", new QRecord().withValue("transactionId", tx.getValue("id")).withValue("segmentPosition", f.segmentPosition())
.withValue("segmentId", f.segmentId()).withValue("loopId", f.loopId()).withValue("elementPosition", f.elementPosition()).withValue("componentPosition", f.componentPosition())
.withValue("snipLevel", f.snipLevel()).withValue("ackCode", f.ak4Code() == null ? f.ak3Code() : f.ak4Code()).withValue("severity", f.severity())
.withValue("message", f.message()).withValue("valueExcerpt", Boolean.TRUE.equals(tx.getValueBoolean("containsPhi")) ? null : f.valueExcerpt()));
}
}
EdiJournal.event(inbox.getValueInteger("id"), tx.getValueInteger("interchangeId"), tx.getValueInteger("id"), "VALIDATED");
if(!findings.isEmpty())
{
EdiJournal.exception(inbox.getValueInteger("id"), tx.getValueInteger("interchangeId"), tx.getValueInteger("id"), tx.getValueInteger("partnerId"), "VALIDATION", "Document validation has findings");
}
}
/*******************************************************************************
** A failed target write rolls back only this set, then records an APPLY exception.
*******************************************************************************/
protected void apply(com.kingsrook.qbits.edi.document.EdiDocument x12, QRecord tx, QRecord document, QRecord agreement) throws Exception
{
if("APPLIED".equals(tx.getValueString("status")) && !Boolean.TRUE.equals(inbox.getValueBoolean("reprocess")))
{
return;
}
QRecord map = row("edi_map", document.getValueInteger("mapId"));
if(map == null || !("ACTIVE".equals(map.getValueString("status")) || Boolean.TRUE.equals(inbox.getValueBoolean("reprocess")) && !Boolean.TRUE.equals(inbox.getValueBoolean("useCurrentRevisions"))))
{
return;
}
String[] stage = {"MAPPING"};
try(var logging = com.kingsrook.qbits.edi.security.PhiLogScope.open(Boolean.TRUE.equals(tx.getValueBoolean("containsPhi"))))
{
InboxWorker.fenced(inbox, lease, () ->
{
Guide guide = new GuideRepository().load(document.getValueInteger("guideId"));
List<QRecord> rules = EdiRecords.query("edi_map_rule", EdiRecords.equal("mapId", map.getValue("id")));
rules.sort(java.util.Comparator.comparing(r -> r.getValueInteger("sequence")));
if(runtime.config().getServiceSessionSupplier() == null)
{
stage[0] = "APPLY";
}
InboundMapper.Result mapped = InboundApplier.asPartner(runtime.config().getServiceSessionSupplier(), agreement.getValueInteger("partnerId"), () -> new InboundMapper(runtime.transforms()).map(x12, guide, map, rules));
stage[0] = "APPLY";
QRecord target = InboundApplier.asPartner(runtime.config().getServiceSessionSupplier(), agreement.getValueInteger("partnerId"), () -> new InboundApplier().apply(map, mapped.record()));
String primaryKey = com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getTable(map.getValueString("tableName")).getPrimaryKeyField();
boolean phi = Boolean.TRUE.equals(tx.getValueBoolean("containsPhi"));
SystemWrites.update("edi_transaction", new QRecord().withValue("id", tx.getValue("id")).withValue("status", "APPLIED")
.withValue("mapId", map.getValue("id")).withValue("mapRevision", map.getValue("revision")).withValue("businessKeyType", map.getValue("businessKeyType"))
.withValue(phi ? "phiBusinessKey" : "businessKey", mapped.businessKey()).withValue("secondaryKey", phi ? null : mapped.secondaryKey())
.withValue("targetTableName", map.getValue("tableName")).withValue("targetRecordId", target.getValueString(primaryKey)));
EdiJournal.event(inbox.getValueInteger("id"), tx.getValueInteger("interchangeId"), tx.getValueInteger("id"), "MAPPED");
EdiJournal.event(inbox.getValueInteger("id"), tx.getValueInteger("interchangeId"), tx.getValueInteger("id"), "APPLIED");
return (null);
});
}
catch(Exception e)
{
InboxWorker.fenced(inbox, lease, () ->
{
SystemWrites.update("edi_transaction", new QRecord().withValue("id", tx.getValue("id")).withValue("status", "EXCEPTION"));
EdiJournal.exception(inbox.getValueInteger("id"), tx.getValueInteger("interchangeId"), tx.getValueInteger("id"), agreement.getValueInteger("partnerId"), stage[0],
runtime.config().getServiceSessionSupplier() == null ? "no service session configured" : "Inbound " + stage[0].toLowerCase(java.util.Locale.ROOT) + " failed");
return (null);
});
}
}
/*******************************************************************************
** Allocate controls from the agreement or local identity's configured scope.
*******************************************************************************/
private String control(QRecord agreement, String level) throws Exception
{
boolean local = "LOCAL_IDENTITY".equals(agreement.getValueString("controlNumberScope"));
return (String.valueOf(runtime.controls().allocate(local ? null : agreement.getValueInteger("id"), local ? agreement.getValueInteger("localIdentityId") : null, level, 1, 999999999, true)));
}
/*******************************************************************************
** Persist an encrypted, fully enveloped acknowledgement; sending is phase b.
*******************************************************************************/
private void queue(X12Interchange ack, QRecord agreement, Integer receivedId, Integer groupId, String type) throws Exception
{
PayloadRef payload = store.put(PayloadKind.OUTBOUND, new ByteArrayInputStream(runtime.writer().write(ack).getBytes(StandardCharsets.US_ASCII)));
QRecord message = SystemWrites.insert("edi_outbound_message", new QRecord().withValue("agreementId", agreement.getValue("id")).withValue("partnerId", agreement.getValue("partnerId"))
.withValue("transportId", agreement.getValue("outboundTransportId")).withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes())
.withValue("status", "QUEUED").withValue("attempts", 0).withValue("ackForGroupId", groupId).withValue("ackForInterchangeId", groupId == null ? receivedId : null));
QRecord interchange = SystemWrites.insert("edi_interchange", new QRecord().withValue("outboundMessageId", message.getValue("id")).withValue("direction", "OUTBOUND")
.withValue("agreementId", agreement.getValue("id")).withValue("partnerId", agreement.getValue("partnerId")).withValue("controlNumber", ack.isa().value(13))
.withValue("senderQualifier", ack.isa().value(5)).withValue("senderId", ack.isa().value(6).trim()).withValue("receiverQualifier", ack.isa().value(7)).withValue("receiverId", ack.isa().value(8).trim())
.withValue("standard", "X12").withValue("interchangeVersion", ack.isa().value(12)).withValue("usageIndicator", ack.isa().value(15)).withValue("groupCount", ack.groups().size())
.withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes()).withValue("status", "ENVELOPED"));
SystemWrites.update("edi_outbound_message", new QRecord().withValue("id", message.getValue("id")).withValue("interchangeId", interchange.getValue("id")));
Integer outboundGroup = null;
if(!ack.groups().isEmpty())
{
var gs = ack.groups().getFirst().gs();
outboundGroup = SystemWrites.insert("edi_functional_group", new QRecord().withValue("interchangeId", interchange.getValue("id")).withValue("functionalIdCode", "FA")
.withValue("senderId", gs.value(2)).withValue("controlNumber", gs.value(6)).withValue("version", gs.value(8)).withValue("transactionCount", 1).withValue("ackStatus", "NONE")).getValueInteger("id");
}
SystemWrites.insert("edi_transaction", new QRecord().withValue("interchangeId", interchange.getValue("id")).withValue("functionalGroupId", outboundGroup)
.withValue("direction", "OUTBOUND").withValue("partnerId", agreement.getValue("partnerId")).withValue("transactionSet", type).withValue("status", "QUEUED")
.withValue("controlNumber", outboundGroup == null ? ack.isa().value(13) : "0001").withValue("version", ack.groups().isEmpty() ? ack.isa().value(12) : ack.groups().getFirst().gs().value(8)));
EdiJournal.event(inbox.getValueInteger("id"), receivedId, null, "ACK_QUEUED");
}
/*******************************************************************************
** Reconcile only outbound controls on this agreement; acknowledgements get no ack.
*******************************************************************************/
private void reconcile(X12Interchange x12, QRecord agreement, Integer received) throws Exception
{
for(var ack : runtime.acknowledgements().read(x12))
{
var receivedAcks = EdiRecords.query("edi_transaction", EdiRecords.equal("interchangeId", received).withCriteria("transactionSet", QCriteriaOperator.EQUALS, ack.type()));
Integer ackTransactionId = receivedAcks.isEmpty() ? null : receivedAcks.getFirst().getValueInteger("id");
if(ackTransactionId != null)
{
SystemWrites.update("edi_transaction", new QRecord().withValue("id", ackTransactionId).withValue("status", "ACKNOWLEDGED").withValue("ackStatus", ack.status()));
}
for(QRecord interchange : EdiRecords.query("edi_interchange", EdiRecords.equal("agreementId", agreement.getValue("id")).withCriteria("direction", QCriteriaOperator.EQUALS, "OUTBOUND")))
{
boolean ta1 = "TA1".equals(ack.type());
if(ta1 && !ack.controlNumber().equals(interchange.getValueString("controlNumber")))
{
continue;
}
if(ta1)
{
SystemWrites.update("edi_interchange", new QRecord().withValue("id", interchange.getValue("id")).withValue("ta1Status", ack.status()));
continue;
}
for(QRecord group : EdiRecords.query("edi_functional_group", EdiRecords.equal("interchangeId", interchange.getValue("id"))))
{
if(!ta1 && !ack.controlNumber().equals(group.getValueString("controlNumber")))
{
continue;
}
SystemWrites.update("edi_functional_group", new QRecord().withValue("id", group.getValue("id")).withValue("ackStatus", ack.status()).withValue("ackReceivedAt", runtime.config().getClock().instant()).withValue("ackTransactionId", ackTransactionId));
for(QRecord transaction : EdiRecords.query("edi_transaction", EdiRecords.equal("functionalGroupId", group.getValue("id"))))
{
String status = ack.transactionStatuses().getOrDefault(transaction.getValueString("controlNumber"), ack.status());
SystemWrites.update("edi_transaction", new QRecord().withValue("id", transaction.getValue("id")).withValue("ackStatus", status)
.withValue("status", "R".equals(status) ? "REJECTED" : "ACKNOWLEDGED"));
}
QAlerts.resolve("edi.ack.overdue", "edi_functional_group:" + group.getValue("id"), "Acknowledgement received");
for(QRecord exception : EdiRecords.query("edi_exception", EdiRecords.equal("interchangeId", interchange.getValue("id"))
.withCriteria("type", QCriteriaOperator.EQUALS, "OVERDUE_ACK").withCriteria("status", QCriteriaOperator.IN, "OPEN", "IN_PROGRESS")))
{
SystemWrites.update("edi_exception", new QRecord().withValue("id", exception.getValue("id")).withValue("status", "RESOLVED")
.withValue("resolvedAt", runtime.config().getClock().instant()).withValue("resolution", "Acknowledgement received"));
}
if(List.of("R", "P").contains(ack.status()))
{
EdiJournal.exception(inbox.getValueInteger("id"), received, null, agreement.getValueInteger("partnerId"), "REJECTED_ACK", "Partner rejected an outbound group");
QAlerts.raise(new AlertSignal("edi.ack.rejected").withSeverity(AlertSeverity.ERROR).withDedupKey(group.getValueString("id"))
.withTitle("EDI acknowledgement rejected").withSummary("Review the outbound functional group"));
}
}
}
}
EdiJournal.event(inbox.getValueInteger("id"), received, null, "ACK_RECEIVED");
}
/*******************************************************************************
** Persist safe failures with the current claim held.
*******************************************************************************/
private void problem(Integer interchange, Integer transaction, Integer partner, String type, String summary) throws Exception
{
InboxWorker.fenced(inbox, lease, () ->
{
EdiJournal.exception(inbox.getValueInteger("id"), interchange, transaction, partner, type, summary);
return (null);
});
}
/*******************************************************************************
** Optional foreign-key lookup.
*******************************************************************************/
private static QRecord row(String table, Integer id) throws Exception
{
if(id == null)
{
return (null);
}
var rows = EdiRecords.query(table, EdiRecords.equal("id", id));
return (rows.isEmpty() ? null : rows.getFirst());
}
}
@@ -0,0 +1,97 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.List;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadKind;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.storage.PayloadStore;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Store first, then acknowledge durable intake. Never parse in the receive path.
*******************************************************************************/
public class InboundSink
{
private final PayloadStore store;
/*******************************************************************************
** Configure the storage boundary.
*******************************************************************************/
public InboundSink(PayloadStore store)
{
this.store = store;
}
/*******************************************************************************
** Unique external ids survive retries; plaintext digests catch byte duplicates.
*******************************************************************************/
public QRecord receive(InboundPayload payload) throws Exception
{
if(payload.transportType() == null || payload.externalId() == null || payload.externalId().isBlank())
{
throw new IllegalArgumentException("Transport and external id are required");
}
return (SystemWrites.call(() ->
{
QQueryFilter key = EdiRecords.equal("transportType", payload.transportType()).withCriteria("externalId", QCriteriaOperator.EQUALS, payload.externalId());
List<QRecord> existing = EdiRecords.query("edi_inbox", key);
if(!existing.isEmpty())
{
return (existing.getFirst());
}
try
{
return (SystemWrites.transaction(() ->
{
PayloadRef ref = store.put(PayloadKind.INBOUND, payload.stream());
Instant now = EdiRuntime.current().config().getClock().instant();
boolean duplicate = !EdiRecords.query("edi_inbox", EdiRecords.equal("payloadSha256", ref.sha256())
.withCriteria("receivedAt", QCriteriaOperator.GREATER_THAN_OR_EQUALS, now.minus(365, ChronoUnit.DAYS))).isEmpty();
QRecord row = SystemWrites.insert("edi_inbox", new QRecord().withValue("transportType", payload.transportType()).withValue("externalId", payload.externalId())
.withValue("fileName", payload.fileName()).withValue("remoteAddress", payload.remoteAddress()).withValue("gatewayUserId", payload.gatewayUserId())
.withValue("username", payload.username()).withValue("payloadRef", ref.value()).withValue("payloadSha256", ref.sha256()).withValue("sizeBytes", ref.sizeBytes())
.withValue("receivedAt", now).withValue("status", duplicate ? "DUPLICATE" : "PENDING").withValue("attempts", 0).withValue("nextAttemptAt", now));
EdiJournal.event(row.getValueInteger("id"), null, null, "RECEIVED");
EdiJournal.event(row.getValueInteger("id"), null, null, "STORED");
return (row);
}));
}
catch(Exception e)
{
// A competing receiver may have committed the unique external id.
existing = EdiRecords.query("edi_inbox", key);
if(!existing.isEmpty())
{
return (existing.getFirst());
}
throw e;
}
}));
}
}
@@ -0,0 +1,187 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.io.InputStream;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.storage.PayloadStore;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Bounded durable worker with database leases and independently committed sets.
*******************************************************************************/
public class InboxWorker
{
public static final String DEAD_LETTER_ALERT = "edi.inbox.dead-letter";
private static final List<Long> BACKOFF = List.of(60L, 300L, 900L, 3600L, 21600L);
private final PayloadStore store;
/*******************************************************************************
** Default runtime storage.
*******************************************************************************/
public InboxWorker()
{
this(EdiRuntime.current().payloads());
}
/*******************************************************************************
** Injectable storage boundary for transport/storage failure tests.
*******************************************************************************/
public InboxWorker(PayloadStore store)
{
this.store = store;
}
/*******************************************************************************
** Process at most twenty due messages as EDI's internal writer.
*******************************************************************************/
public Integer runOnce() throws QException
{
return (SystemWrites.call(() ->
{
Instant now = now();
QQueryFilter due = EdiRecords.equal("status", "RETRY_WAIT")
.withSubFilter(new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR)
.withCriteria("nextAttemptAt", QCriteriaOperator.IS_BLANK).withCriteria("nextAttemptAt", QCriteriaOperator.LESS_THAN_OR_EQUALS, now));
QQueryFilter stale = EdiRecords.equal("status", "PROCESSING").withCriteria("claimedAt", QCriteriaOperator.LESS_THAN, now.minusSeconds(300));
List<QRecord> rows = EdiRecords.query("edi_inbox", new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR).withSubFilter(due).withSubFilter(EdiRecords.equal("status", "PENDING")).withSubFilter(stale)
.withOrderBy(new QFilterOrderBy("receivedAt")).withLimit(20));
int completed = 0;
for(QRecord row : rows)
{
Instant lease = now().truncatedTo(ChronoUnit.MILLIS);
if(!claim(row, lease))
{
continue;
}
try(InputStream stream = store.open(new PayloadRef(row.getValueString("payloadRef"), row.getValueString("payloadSha256"), row.getValueLong("sizeBytes"))))
{
var framing = new com.kingsrook.qbits.edi.document.EnvelopeReader();
byte[] envelope;
int position = 0;
while((envelope = framing.next(stream)) != null)
{
position++;
if(envelope[0] == 'U')
{
var parsed = new com.kingsrook.qbits.edi.edifact.EdifactEnvelopeReader(EdiRuntime.current().config().getMaxSegmentsPerTransaction()).read(new java.io.ByteArrayInputStream(envelope)).getFirst();
new com.kingsrook.qbits.edi.edifact.EdifactInbound(row, lease, store).process(parsed, position);
}
else
{
var parsed = EdiRuntime.current().reader().read(new java.io.ByteArrayInputStream(envelope)).getFirst();
new InboundProcessor(row, lease, store).process(parsed, position);
}
}
fenced(row, lease, () ->
{
SystemWrites.update("edi_inbox", new QRecord().withValue("id", row.getValue("id")).withValue("status", "PROCESSED")
.withValue("processedAt", now()).withValue("nextAttemptAt", null).withValue("lastError", null).withValue("reprocess", false));
return (null);
});
completed++;
}
catch(Exception e)
{
fail(row, lease);
}
}
return (completed);
}));
}
/*******************************************************************************
** Compare selected status and lease before claiming; only the winner proceeds.
*******************************************************************************/
private boolean claim(QRecord row, Instant lease) throws QException
{
return (SystemWrites.transaction(() ->
{
boolean won = row.getValue("claimedAt") == null
? SystemWrites.guard("UPDATE edi_inbox SET id=id WHERE id=? AND status=? AND claimed_at IS NULL", row.getValue("id"), row.getValue("status"))
: SystemWrites.guard("UPDATE edi_inbox SET id=id WHERE id=? AND status=? AND claimed_at=?", row.getValue("id"), row.getValue("status"), row.getValueInstant("claimedAt"));
if(won)
{
SystemWrites.update("edi_inbox", new QRecord().withValue("id", row.getValue("id")).withValue("status", "PROCESSING").withValue("claimedAt", lease));
}
return (won);
}));
}
/*******************************************************************************
** Every checkpoint holds the inbox lock and verifies ownership until commit.
*******************************************************************************/
public static <T> T fenced(QRecord inbox, Instant lease, SystemWrites.Work<T> work) throws QException
{
return (SystemWrites.transaction(() ->
{
if(!SystemWrites.guard("UPDATE edi_inbox SET id=id WHERE id=? AND status='PROCESSING' AND claimed_at=?", inbox.getValue("id"), lease))
{
throw new QException("Inbox claim was lost");
}
return (work.run());
}));
}
/*******************************************************************************
** Persist retry outcome separately from failed set writes, with no exception text.
*******************************************************************************/
private void fail(QRecord row, Instant lease) throws QException
{
fenced(row, lease, () ->
{
int attempts = Objects.requireNonNullElse(row.getValueInteger("attempts"), 0) + 1;
boolean dead = attempts > BACKOFF.size();
SystemWrites.update("edi_inbox", new QRecord().withValue("id", row.getValue("id")).withValue("attempts", attempts)
.withValue("status", dead ? "DEAD_LETTER" : "RETRY_WAIT").withValue("nextAttemptAt", dead ? null : now().plusSeconds(BACKOFF.get(attempts - 1)))
.withValue("lastError", "Inbound processing failed; retry the stored payload"));
if(dead)
{
EdiJournal.exception(row.getValueInteger("id"), null, null, row.getValueInteger("partnerId"), "DEAD_LETTER", "Inbound processing exhausted retries");
QAlerts.raise(new AlertSignal(DEAD_LETTER_ALERT).withSeverity(AlertSeverity.ERROR).withDedupKey(row.getValueString("id"))
.withTitle("EDI inbox exhausted retries").withSummary("Review the dead-letter inbox record"));
}
return (null);
});
}
/*******************************************************************************
** Host clock makes retry and stale-claim behavior deterministic.
*******************************************************************************/
private static Instant now()
{
return (EdiRuntime.current().config().getClock().instant());
}
}
@@ -0,0 +1,159 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.util.List;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Guarded resend/cancel operations used after the standard Review step.
*******************************************************************************/
public final class OutboundActions
{
/*******************************************************************************
** Call within SystemWrites.transaction so validation and mutation share a lock.
*******************************************************************************/
public static void execute(String table, QRecord selected, String operation) throws Exception
{
QRecord message = table.equals("edi_interchange") ? OutboundWorker.row("edi_outbound_message", selected.getValue("outboundMessageId")) : selected;
if(!SystemWrites.guard("UPDATE edi_outbound_message SET id=id WHERE id=? AND status=?", message.getValue("id"), message.getValue("status")))
{
throw new IllegalStateException("Outbound message changed; refresh the record");
}
message = OutboundWorker.row("edi_outbound_message", message.getValue("id"));
if(operation.equals("cancel"))
{
if(!List.of("QUEUED", "RETRY_WAIT").contains(message.getValueString("status")))
{
throw new IllegalStateException("Only queued or retrying messages can be cancelled");
}
SystemWrites.update("edi_outbound_message", new QRecord().withValue("id", message.getValue("id")).withValue("status", "CANCELLED").withValue("nextAttemptAt", null));
}
else if(operation.equals("resend"))
{
SystemWrites.update("edi_outbound_message", new QRecord().withValue("id", message.getValue("id")).withValue("status", "QUEUED")
.withValue("attempts", 0).withValue("nextAttemptAt", null).withValue("lastError", null));
}
else if(operation.equals("new"))
{
reEnvelope(message);
}
else
{
throw new IllegalArgumentException("Unknown outbound operation");
}
EdiJournal.event(null, message.getValueInteger("interchangeId"), null, operation.toUpperCase(java.util.Locale.ROOT));
}
/*******************************************************************************
** Re-envelope the archived payload and tracking snapshots, including TA1/997/999.
** Maps are not re-run: source edits require a new outbound request instead.
*******************************************************************************/
private static void reEnvelope(QRecord message) throws Exception
{
if("EDIFACT".equals(OutboundWorker.row("edi_agreement", message.getValue("agreementId")).getValueString("standard")))
{
com.kingsrook.qbits.edi.edifact.EdifactOutbound.resendAsNew(message);
return;
}
var runtime = EdiRuntime.current();
QRecord originalRow = OutboundWorker.row("edi_interchange", message.getValue("interchangeId"));
QRecord agreement = OutboundWorker.row("edi_agreement", message.getValue("agreementId"));
boolean local = "LOCAL_IDENTITY".equals(agreement.getValueString("controlNumberScope"));
String control = String.format("%09d", runtime.controls().allocate(local ? null : agreement.getValueInteger("id"), local ? agreement.getValueInteger("localIdentityId") : null, "ISA", 1, 999999999, true));
com.kingsrook.qbits.edi.x12.X12Interchange original;
try(var stream = runtime.payloads().open(new PayloadRef(message.getValueString("payloadRef"), message.getValueString("payloadSha256"), message.getValueLong("sizeBytes"))))
{
original = runtime.reader().read(stream).getFirst();
}
var groups = new java.util.ArrayList<com.kingsrook.qbits.edi.x12.X12FunctionalGroup>();
for(var group : original.groups())
{
String groupControl = String.valueOf(runtime.controls().allocate(local ? null : agreement.getValueInteger("id"), local ? agreement.getValueInteger("localIdentityId") : null, "GS", 1, 999999999, true));
groups.add(new com.kingsrook.qbits.edi.x12.X12FunctionalGroup(replace(group.gs(), 6, groupControl), replace(group.ge(), 2, groupControl), group.transactions(), List.of()));
}
var rebuilt = new com.kingsrook.qbits.edi.x12.X12Interchange(original.separators(), replace(original.isa(), 13, control), replace(original.iea(), 2, control), groups, original.serviceSegments(), List.of());
var payload = runtime.payloads().put(com.kingsrook.qbits.edi.storage.PayloadKind.OUTBOUND,
new java.io.ByteArrayInputStream(runtime.writer().write(rebuilt).getBytes(java.nio.charset.StandardCharsets.UTF_8)));
QRecord newInterchange = copy(originalRow).withValue("originalInterchangeId", originalRow.getValue("id")).withValue("controlNumber", control)
.withValue("status", "ENVELOPED").withValue("ta1Status", null).withValue("outboundMessageId", null).withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes());
newInterchange = SystemWrites.insert("edi_interchange", newInterchange);
var groupIds = new java.util.HashMap<Integer, Integer>();
for(QRecord oldGroup : EdiRecords.query("edi_functional_group", EdiRecords.equal("interchangeId", originalRow.getValue("id"))))
{
int index = -1;
for(int i = 0; i < original.groups().size(); i++)
{
if(original.groups().get(i).gs().value(6).equals(oldGroup.getValueString("controlNumber")))
{
index = i;
break;
}
}
if(index < 0)
{
throw new IllegalStateException("Stored group does not match archived envelope");
}
QRecord newGroup = SystemWrites.insert("edi_functional_group", copy(oldGroup).withValue("interchangeId", newInterchange.getValue("id"))
.withValue("controlNumber", groups.get(index).gs().value(6)).withValue("ackDueAt", null).withValue("ackReceivedAt", null).withValue("ackTransactionId", null)
.withValue("ackStatus", "NONE".equals(oldGroup.getValueString("ackStatus")) ? "NONE" : "PENDING"));
groupIds.put(oldGroup.getValueInteger("id"), newGroup.getValueInteger("id"));
}
for(QRecord transaction : EdiRecords.query("edi_transaction", EdiRecords.equal("interchangeId", originalRow.getValue("id"))))
{
SystemWrites.insert("edi_transaction", copy(transaction).withValue("interchangeId", newInterchange.getValue("id"))
.withValue("functionalGroupId", groupIds.get(transaction.getValueInteger("functionalGroupId"))).withValue("status", "QUEUED").withValue("ackStatus", null));
}
QRecord newMessage = SystemWrites.insert("edi_outbound_message", copy(message).withValue("interchangeId", newInterchange.getValue("id"))
.withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes())
.withValue("status", "QUEUED").withValue("attempts", 0).withValue("nextAttemptAt", null).withValue("sentAt", null).withValue("deliveredAt", null).withValue("claimedAt", null)
.withValue("transportMessageId", null).withValue("lastError", null).withValue("ackForGroupId", null).withValue("ackForInterchangeId", null));
SystemWrites.update("edi_interchange", new QRecord().withValue("id", newInterchange.getValue("id")).withValue("outboundMessageId", newMessage.getValue("id")));
}
/*******************************************************************************
** Preserve components and repetitions while replacing one envelope element.
*******************************************************************************/
private static com.kingsrook.qbits.edi.x12.X12Segment replace(com.kingsrook.qbits.edi.x12.X12Segment segment, int position, String value)
{
var elements = new java.util.ArrayList<>(segment.elements());
elements.set(position - 1, List.of(List.of(value)));
return (new com.kingsrook.qbits.edi.x12.X12Segment(segment.id(), segment.position(), elements));
}
/*******************************************************************************
** Clone business metadata without identity and audit timestamps.
*******************************************************************************/
private static QRecord copy(QRecord source)
{
QRecord copy = new QRecord();
copy.getValues().putAll(source.getValues());
for(String field : List.of("id", "createDate", "modifyDate"))
{
copy.getValues().remove(field);
}
return (copy);
}
}
@@ -0,0 +1,116 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.util.List;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.tables.listeners.RecordChangeEvent;
import com.kingsrook.qqq.backend.core.actions.tables.listeners.RecordChangeListenerInterface;
import com.kingsrook.qqq.backend.core.actions.tables.listeners.RecordChangeType;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Transactional source listener and explicit enqueue boundary for MANUAL/ESB.
*******************************************************************************/
public class OutboundEnqueue implements RecordChangeListenerInterface
{
/*******************************************************************************
** Ignore internal tables and deletes; definitions are read at event time.
*******************************************************************************/
public boolean appliesTo(String table, RecordChangeType type)
{
return (!table.startsWith("edi_") && type != RecordChangeType.DELETE);
}
/*******************************************************************************
** Use the event transaction, even when the caller did not put it in QContext.
*******************************************************************************/
public void onRecordsChanged(RecordChangeEvent event) throws QException
{
var previous = QContext.getQBackendTransaction();
QContext.setTransaction(event.getTransaction());
try
{
for(QRecord record : event.getRecords())
{
enqueue(event.getTableName(), record, "RECORD_CHANGE", null);
}
}
finally
{
QContext.setTransaction(previous);
}
}
/*******************************************************************************
** Enqueue matching active outbound definitions using the writer's transaction.
*******************************************************************************/
public static Integer enqueue(String table, QRecord source, String trigger, String eventId) throws QException
{
return (SystemWrites.call(() ->
{
int count = 0;
var definitions = EdiRecords.query("edi_agreement_document", EdiRecords.equal("sourceTableName", table)
.withCriteria("direction", QCriteriaOperator.EQUALS, "OUTBOUND").withCriteria("isActive", QCriteriaOperator.EQUALS, true));
for(QRecord document : definitions)
{
if(!"MANUAL".equals(trigger) && !trigger.equals(document.getValueString("outboundTrigger")))
{
continue;
}
QRecord agreement = OutboundWorker.row("edi_agreement", document.getValueInteger("agreementId"));
if(!Boolean.TRUE.equals(agreement.getValueBoolean("isActive")))
{
continue;
}
var metadata = QContext.getQInstance().getTable(table);
if(!metadata.getBackendName().equals(EdiRuntime.current().config().getBackendName()))
{
throw new QException("Outbound source and EDI must share the transactional backend");
}
String filter = document.getValueString("triggerFilter");
if(filter != null && !filter.isBlank() && EdiRecords.query(table, EdiRecords.equal(metadata.getPrimaryKeyField(), source.getValue(metadata.getPrimaryKeyField()))
.withSubFilter(new ObjectMapper().readValue(filter, QQueryFilter.class))).isEmpty())
{
continue;
}
if(eventId != null && !EdiRecords.query("edi_outbound_request", EdiRecords.equal("eventId", eventId)
.withCriteria("agreementDocumentId", QCriteriaOperator.EQUALS, document.getValue("id"))).isEmpty())
{
continue;
}
SystemWrites.insert("edi_outbound_request", new QRecord().withValue("agreementDocumentId", document.getValue("id"))
.withValue("sourceTableName", table).withValue("sourceRecordId", source.getValueString(metadata.getPrimaryKeyField()))
.withValue("eventId", eventId).withValue("status", "PENDING").withValue("attempts", 0)
.withValue("nextAttemptAt", EdiRuntime.current().config().getClock().instant()));
count++;
}
return (count);
}));
}
}
@@ -0,0 +1,276 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.io.ByteArrayInputStream;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.guide.GuideRepository;
import com.kingsrook.qbits.edi.map.InboundApplier;
import com.kingsrook.qbits.edi.map.OutboundMapper;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadKind;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qbits.edi.x12.X12FunctionalGroup;
import com.kingsrook.qbits.edi.x12.X12Interchange;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qbits.edi.x12.X12Separators;
import com.kingsrook.qbits.edi.x12.X12TransactionSet;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Transactional map/validate/envelope worker. Row locks fence replicas; a crash
** rolls the whole envelope back, leaving requests available for the next run.
*******************************************************************************/
public class OutboundWorker
{
/*******************************************************************************
** Process immediate definitions; batch processes call runDocument explicitly.
*******************************************************************************/
public Integer runOnce() throws QException
{
return (SystemWrites.call(() ->
{
int count = 0;
for(QRecord document : EdiRecords.query("edi_agreement_document", EdiRecords.equal("direction", "OUTBOUND").withCriteria("isActive", QCriteriaOperator.EQUALS, true)))
{
if(!"BATCH".equals(document.getValueString("batchMode")))
{
count += runDocument(document.getValueInteger("id"));
}
}
return (count);
}));
}
/*******************************************************************************
** A scheduled batch invocation cuts one bounded batch for its definition.
*******************************************************************************/
public Integer runDocument(Integer id) throws QException
{
return (SystemWrites.call(() -> SystemWrites.transaction(() ->
{
QRecord document = row("edi_agreement_document", id);
QRecord agreement = row("edi_agreement", document.getValueInteger("agreementId"));
if(!Boolean.TRUE.equals(document.getValueBoolean("isActive")) || !Boolean.TRUE.equals(agreement.getValueBoolean("isActive")))
{
return (0);
}
QRecord localIdentity = row("edi_identity", agreement.getValueInteger("localIdentityId"));
QRecord partnerIdentity = row("edi_identity", agreement.getValueInteger("partnerIdentityId"));
if(localIdentity == null || partnerIdentity == null || Boolean.FALSE.equals(localIdentity.getValueBoolean("isActive")) || Boolean.FALSE.equals(partnerIdentity.getValueBoolean("isActive")))
{
///////////////////////////////////////////////////////////////////////////
// an inactive identity stops sending, as it stops routing; requests wait //
///////////////////////////////////////////////////////////////////////////
return (0);
}
var requests = EdiRecords.query("edi_outbound_request", EdiRecords.equal("agreementDocumentId", id).withCriteria("status", QCriteriaOperator.EQUALS, "PENDING").withLimit(100));
if(requests.isEmpty())
{
return (0);
}
List<QRecord> accepted = new ArrayList<>();
List<com.kingsrook.qbits.edi.document.EdiDocument> sets = new ArrayList<>();
QRecord map = row("edi_map", document.getValueInteger("mapId"));
var rules = EdiRecords.query("edi_map_rule", EdiRecords.equal("mapId", map.getValue("id")));
var associations = new java.util.LinkedHashSet<String>();
for(QRecord rule : rules)
{
String path = rule.getValueString("qqqPath");
if(path != null)
{
for(int dot = path.indexOf('.'); dot >= 0; dot = path.indexOf('.', dot + 1))
{
associations.add(path.substring(0, dot));
}
}
}
var guide = new GuideRepository().load(document.getValueInteger("guideId"));
for(QRecord request : requests)
{
if(!SystemWrites.guard("UPDATE edi_outbound_request SET id=id WHERE id=? AND status='PENDING'", request.getValue("id")))
{
continue;
}
try(var logging = com.kingsrook.qbits.edi.security.PhiLogScope.open(Boolean.TRUE.equals(document.getValueBoolean("containsPhi"))))
{
com.kingsrook.qbits.edi.document.EdiDocument set = InboundApplier.asPartner(EdiRuntime.current().config().getServiceSessionSupplier(), agreement.getValueInteger("partnerId"), () ->
{
String table = request.getValueString("sourceTableName");
String primaryKey = QContext.getQInstance().getTable(table).getPrimaryKeyField();
var query = new QueryInput(table).withFilter(EdiRecords.equal(primaryKey, request.getValue("sourceRecordId")))
.withIncludeAssociations(true).withAssociationNamesToInclude(associations).withTransaction(QContext.getQBackendTransaction());
query.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
var records = new QueryAction().execute(query).getRecords();
if(records.size() != 1 || (!"ACTIVE".equals(map.getValueString("status")) || !table.equals(map.getValueString("tableName"))))
{
throw new IllegalArgumentException("Source record or active map unavailable");
}
return ("EDIFACT".equals(agreement.getValueString("standard"))
? new com.kingsrook.qbits.edi.edifact.EdifactMapper(com.kingsrook.qbits.edi.edifact.EdifactSeparators.fromAgreement(agreement).decimal()).message(records.getFirst(), map, rules, com.kingsrook.qbits.edi.edifact.EdifactOutbound.allocate(agreement, "UNH"))
: new OutboundMapper().build(records.getFirst(), map, rules, String.format("%04d", sets.size() + 1)));
});
if(EdiRuntime.current().validator().validate(set, guide, Boolean.TRUE.equals(document.getValueBoolean("containsPhi"))).stream().anyMatch(f -> "ERROR".equals(f.severity())))
{
reject(request, agreement, "VALIDATION");
continue;
}
accepted.add(request);
sets.add(set);
}
catch(Exception e)
{
reject(request, agreement, "MAPPING");
}
if(!"BATCH".equals(document.getValueString("batchMode")) && !sets.isEmpty())
{
envelope(agreement, document, map, accepted, sets, null);
accepted.clear();
sets.clear();
}
}
if(!sets.isEmpty())
{
envelope(agreement, document, map, accepted, sets, null);
}
return (requests.size());
})));
}
/*******************************************************************************
** Invalid documents are terminal requests and never enter the send queue.
*******************************************************************************/
private void reject(QRecord request, QRecord agreement, String type) throws Exception
{
SystemWrites.update("edi_outbound_request", new QRecord().withValue("id", request.getValue("id")).withValue("status", "FAILED").withValue("lastError", type));
EdiJournal.exception(null, null, null, agreement.getValueInteger("partnerId"), type, "Outbound request failed " + type.toLowerCase(java.util.Locale.ROOT));
}
/*******************************************************************************
** Allocate controls once, store exact bytes, then persist all tracking records.
*******************************************************************************/
public static QRecord envelope(QRecord agreement, QRecord document, QRecord map, List<QRecord> requests, List<? extends com.kingsrook.qbits.edi.document.EdiDocument> sets, Integer original) throws Exception
{
if("EDIFACT".equals(agreement.getValueString("standard")))
{
return (com.kingsrook.qbits.edi.edifact.EdifactOutbound.envelope(agreement, document, map, requests,
sets.stream().map(s -> (com.kingsrook.qbits.edi.edifact.EdifactMessage) s).toList(), original, null));
}
var runtime = EdiRuntime.current();
QRecord local = row("edi_identity", agreement.getValueInteger("localIdentityId"));
QRecord partner = row("edi_identity", agreement.getValueInteger("partnerIdentityId"));
LocalDateTime at = LocalDateTime.ofInstant(runtime.config().getClock().instant(), ZoneOffset.UTC);
String isaControl = String.format("%09d", allocate(agreement, "ISA"));
String gsControl = String.valueOf(allocate(agreement, "GS"));
String version = Objects.requireNonNullElse(agreement.getValueString("interchangeVersion"), "00401");
String component = value(agreement, "componentSeparator", ":");
String repetition = value(agreement, "repetitionSeparator", "^");
X12Segment isa = X12Segment.of("ISA", 1, "00", "", "00", "", local.getValueString("interchangeQualifier"), local.getValueString("interchangeId"),
partner.getValueString("interchangeQualifier"), partner.getValueString("interchangeId"), at.format(DateTimeFormatter.ofPattern("yyMMdd")), at.format(DateTimeFormatter.ofPattern("HHmm")),
version.compareTo("00501") >= 0 ? repetition : "U", version, isaControl, Boolean.TRUE.equals(agreement.getValueBoolean("requestTa1")) ? "1" : "0", agreement.getValueString("usageIndicator"), component);
X12Segment gs = X12Segment.of("GS", 2, document.getValueString("functionalIdCode"), local.getValueString("applicationCode"), partner.getValueString("applicationCode"), at.format(DateTimeFormatter.BASIC_ISO_DATE),
at.format(DateTimeFormatter.ofPattern("HHmm")), gsControl, "X", document.getValueString("version"));
var interchange = new X12Interchange(new X12Separators(value(agreement, "elementSeparator", "*").charAt(0), value(agreement, "segmentTerminator", "~").charAt(0), component.charAt(0), version.compareTo("00501") >= 0 ? repetition.charAt(0) : null),
isa, X12Segment.of("IEA", 0, "1", isaControl), List.of(new X12FunctionalGroup(gs, X12Segment.of("GE", 0, String.valueOf(sets.size()), gsControl), sets.stream().map(s -> (X12TransactionSet) s).toList(), List.of())), List.of(), List.of());
String ending = switch(value(agreement, "lineEnding", "NONE"))
{
case "LF" -> "\n";
case "CRLF" -> "\r\n";
default -> "";
};
var payload = runtime.payloads().put(PayloadKind.OUTBOUND, new ByteArrayInputStream(runtime.writer().write(interchange, ending).getBytes(StandardCharsets.UTF_8)));
QRecord persisted = SystemWrites.insert("edi_interchange", new QRecord().withValue("direction", "OUTBOUND").withValue("partnerId", agreement.getValue("partnerId"))
.withValue("agreementId", agreement.getValue("id")).withValue("transportId", agreement.getValue("outboundTransportId")).withValue("standard", "X12")
.withValue("senderQualifier", isa.value(5)).withValue("senderId", isa.value(6)).withValue("receiverQualifier", isa.value(7)).withValue("receiverId", isa.value(8))
.withValue("controlNumber", isaControl).withValue("interchangeVersion", version).withValue("usageIndicator", isa.value(15)).withValue("interchangeDate", runtime.config().getClock().instant())
.withValue("groupCount", 1).withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes())
.withValue("status", "ENVELOPED").withValue("containsPhi", document.getValue("containsPhi")).withValue("originalInterchangeId", original));
boolean expectsAck = document.getValueString("ackExpectation") != null && !"NONE".equals(document.getValueString("ackExpectation"));
QRecord group = SystemWrites.insert("edi_functional_group", new QRecord().withValue("interchangeId", persisted.getValue("id")).withValue("senderId", gs.value(2))
.withValue("functionalIdCode", gs.value(1)).withValue("controlNumber", gsControl).withValue("version", gs.value(8)).withValue("transactionCount", sets.size())
.withValue("ackStatus", expectsAck ? "PENDING" : "NONE"));
for(int i = 0; i < sets.size(); i++)
{
var set = sets.get(i);
String businessKey = map.getValueString("businessKeyPath") == null ? null : new com.kingsrook.qbits.edi.guide.RuleEvaluator(set.segments(), java.util.Map.of()).values(map.getValueString("businessKeyPath")).stream().findFirst().orElse(null);
SystemWrites.insert("edi_transaction", new QRecord().withValue("interchangeId", persisted.getValue("id")).withValue("functionalGroupId", group.getValue("id"))
.withValue("direction", "OUTBOUND").withValue("partnerId", agreement.getValue("partnerId")).withValue("agreementDocumentId", document.getValue("id"))
.withValue("transactionSet", set.header().value(1)).withValue("version", gs.value(8)).withValue("controlNumber", set.header().value(2)).withValue("status", "QUEUED")
.withValue("guideId", document.getValue("guideId")).withValue("guideRevision", document.getValue("guideId") == null ? null : row("edi_guide", document.getValue("guideId")).getValue("revision"))
.withValue("targetTableName", i < requests.size() ? requests.get(i).getValue("sourceTableName") : null).withValue("targetRecordId", i < requests.size() ? requests.get(i).getValue("sourceRecordId") : null).withValue("mapId", map.getValue("id")).withValue("mapRevision", map.getValue("revision"))
.withValue("businessKeyType", map.getValue("businessKeyType")).withValue(Boolean.TRUE.equals(document.getValueBoolean("containsPhi")) ? "phiBusinessKey" : "businessKey", businessKey)
.withValue("containsPhi", document.getValue("containsPhi")).withValue("segmentCount", set.segments().size()));
if(i < requests.size())
{
SystemWrites.update("edi_outbound_request", new QRecord().withValue("id", requests.get(i).getValue("id")).withValue("status", "ENVELOPED"));
}
}
QRecord message = SystemWrites.insert("edi_outbound_message", new QRecord().withValue("interchangeId", persisted.getValue("id")).withValue("agreementId", agreement.getValue("id"))
.withValue("partnerId", agreement.getValue("partnerId")).withValue("transportId", agreement.getValue("outboundTransportId")).withValue("status", "QUEUED").withValue("attempts", 0)
.withValue("payloadRef", payload.value()).withValue("payloadSha256", payload.sha256()).withValue("sizeBytes", payload.sizeBytes()));
SystemWrites.update("edi_interchange", new QRecord().withValue("id", persisted.getValue("id")).withValue("outboundMessageId", message.getValue("id")));
EdiJournal.event(null, persisted.getValueInteger("id"), null, "ENVELOPED");
return (message);
}
/*******************************************************************************
** Allocate in the configured agreement or identity scope.
*******************************************************************************/
private static long allocate(QRecord agreement, String level) throws Exception
{
boolean local = "LOCAL_IDENTITY".equals(agreement.getValueString("controlNumberScope"));
return (EdiRuntime.current().controls().allocate(local ? null : agreement.getValueInteger("id"), local ? agreement.getValueInteger("localIdentityId") : null, level, 1, 999999999, true));
}
/*******************************************************************************
** Lookup required configuration without leaking submitted values.
*******************************************************************************/
public static QRecord row(String table, Serializable id) throws QException
{
if(id == null)
{
throw new QException("Required EDI record reference is absent");
}
return (EdiRecords.query(table, EdiRecords.equal("id", id)).getFirst());
}
/*******************************************************************************
** Default optional envelope settings.
*******************************************************************************/
private static String value(QRecord record, String field, String fallback)
{
return (Objects.requireNonNullElse(record.getValueString(field), fallback));
}
}
@@ -0,0 +1,166 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.flow;
import java.time.Instant;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qbits.edi.EdiRuntime;
import com.kingsrook.qbits.edi.model.EdiOutboundMessage;
import com.kingsrook.qbits.edi.model.EdiTransport;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadRef;
import com.kingsrook.qbits.edi.support.SystemWrites;
import com.kingsrook.qbits.edi.transport.EdiSendResult;
import com.kingsrook.qbits.edi.transport.EdiTransportPluginsMetaData;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Durable at-least-once sender. A database row lock fences concurrent senders;
** transport timeouts bound each attempt. Stable bytes/ids make retries detectable.
*******************************************************************************/
public class SendWorker
{
public static final List<Long> BACKOFF = List.of(60L, 300L, 900L, 3600L, 10800L, 21600L, 43200L, 86400L);
/*******************************************************************************
** Process only queued or due retry rows, including inbound-generated replies.
*******************************************************************************/
public Integer runOnce() throws QException
{
return (SystemWrites.call(() ->
{
int count = 0;
var due = new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR).withCriteria("nextAttemptAt", QCriteriaOperator.IS_BLANK)
.withCriteria("nextAttemptAt", QCriteriaOperator.LESS_THAN_OR_EQUALS, now());
for(QRecord message : EdiRecords.query("edi_outbound_message", new QQueryFilter().withCriteria("status", QCriteriaOperator.IN, "QUEUED", "RETRY_WAIT").withSubFilter(due).withLimit(20)))
{
count += send(message);
}
return (count);
}));
}
/*******************************************************************************
** Compare selected status and attempts under lock before performing remote I/O.
*******************************************************************************/
private Integer send(QRecord selected) throws QException
{
return (SystemWrites.transaction(() ->
{
if(!SystemWrites.guard("UPDATE edi_outbound_message SET id=id WHERE id=? AND status=? AND COALESCE(attempts,0)=?", selected.getValue("id"), selected.getValue("status"), Objects.requireNonNullElse(selected.getValueInteger("attempts"), 0)))
{
return (0);
}
QRecord message = OutboundWorker.row("edi_outbound_message", selected.getValue("id"));
int attempts = Objects.requireNonNullElse(message.getValueInteger("attempts"), 0) + 1;
int maximum = 8;
EdiSendResult result;
try
{
QRecord transport = OutboundWorker.row("edi_transport", message.getValue("transportId"));
maximum = Objects.requireNonNullElse(transport.getValueInteger("maxAttempts"), 8);
if(!Boolean.TRUE.equals(transport.getValueBoolean("isActive")) || "INBOUND".equals(transport.getValueString("direction")))
{
result = new EdiSendResult("FAILED", null, "Outbound transport is disabled");
}
else
{
result = EdiTransportPluginsMetaData.current().get(transport.getValueString("type")).send(message.toEntity(EdiOutboundMessage.class),
new PayloadRef(message.getValueString("payloadRef"), message.getValueString("payloadSha256"), message.getValueLong("sizeBytes")), transport.toEntity(EdiTransport.class));
}
}
catch(Exception e)
{
result = new EdiSendResult("RETRY", null, "Transport attempt failed");
}
boolean sent = List.of("SENT", "DELIVERED").contains(result.status());
String status = sent ? result.status() : "FAILED".equals(result.status()) ? "FAILED" : attempts >= maximum ? "DEAD_LETTER" : "RETRY_WAIT";
QRecord update = new QRecord().withValue("id", message.getValue("id")).withValue("status", status).withValue("attempts", attempts)
.withValue("claimedAt", null).withValue("lastError", sent ? null : failureText(result))
.withValue("nextAttemptAt", "RETRY_WAIT".equals(status) ? now().plusSeconds(BACKOFF.get(Math.min(attempts - 1, BACKOFF.size() - 1))) : null);
if(sent)
{
update.withValue("transportMessageId", result.messageId()).withValue("sentAt", now());
if("DELIVERED".equals(status))
{
update.withValue("deliveredAt", now());
}
}
SystemWrites.update("edi_outbound_message", update);
Integer interchangeId = message.getValueInteger("interchangeId");
if(interchangeId != null && sent)
{
SystemWrites.update("edi_interchange", new QRecord().withValue("id", interchangeId).withValue("status", status));
for(QRecord group : EdiRecords.query("edi_functional_group", EdiRecords.equal("interchangeId", interchangeId)))
{
for(QRecord transaction : EdiRecords.query("edi_transaction", EdiRecords.equal("functionalGroupId", group.getValue("id"))))
{
if("QUEUED".equals(transaction.getValueString("status")))
{
SystemWrites.update("edi_transaction", new QRecord().withValue("id", transaction.getValue("id")).withValue("status", "SENT"));
}
if("PENDING".equals(group.getValueString("ackStatus")) && group.getValue("ackDueAt") == null)
{
QRecord document = OutboundWorker.row("edi_agreement_document", transaction.getValue("agreementDocumentId"));
SystemWrites.update("edi_functional_group", new QRecord().withValue("id", group.getValue("id"))
.withValue("ackDueAt", now().plusSeconds(60L * Objects.requireNonNullElse(document.getValueInteger("ackDueMinutes"), 1440))));
}
}
}
}
if(List.of("FAILED", "DEAD_LETTER").contains(status))
{
SystemWrites.insert("edi_exception", new QRecord().withValue("outboundMessageId", message.getValue("id")).withValue("interchangeId", interchangeId)
.withValue("partnerId", message.getValue("partnerId")).withValue("type", "TRANSPORT").withValue("status", "OPEN").withValue("occurrences", 1).withValue("summary", "Outbound delivery failed"));
QAlerts.raise(new AlertSignal("edi.send.failed").withSeverity(AlertSeverity.ERROR).withDedupKey("edi_outbound_message:" + message.getValue("id"))
.withTitle("EDI delivery failed").withSummary("Review the outbound message"));
}
EdiJournal.event(null, interchangeId, null, status);
return (1);
}));
}
/*******************************************************************************
** The recorded failure: the transport reason (value-free by contract, for
** example an HTTP status, an AS2 disposition or a missing variable name), or a
** generic text.
*******************************************************************************/
private static String failureText(EdiSendResult result)
{
String reason = result.reason();
return (reason == null || reason.isBlank() ? "Transport delivery failed" : reason.length() <= 255 ? reason : reason.substring(0, 255));
}
/*******************************************************************************
** Clock controlled by the host and deterministic tests.
*******************************************************************************/
private static Instant now()
{
return (EdiRuntime.current().config().getClock().instant());
}
}
@@ -0,0 +1,31 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
/*******************************************************************************
** Positional situational trigger for 999 CTX; never includes business or element values.
*******************************************************************************/
public record AckContext(String segmentId, int segmentPosition, String loopId, Integer elementPosition, Integer componentPosition)
{
}
@@ -0,0 +1,105 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.storage.PayloadCrypto;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Checksum-based installation of original baseline guide resources.
*******************************************************************************/
public class BuiltInGuideLoader
{
public static final List<String> RESOURCES = List.of("850-004010", "850-005010", "997-004010", "997-005010", "999-005010X231A1", "855-004010", "855-005010", "810-004010", "810-005010", "846-004010", "846-005010", "860-004010", "860-005010", "856-004010", "856-005010", "940-004010", "940-005010", "945-004010", "945-005010", "214-004010", "214-005010", "204-004010", "204-005010", "ORDERS-D96A", "ORDRSP-D96A", "DESADV-D96A", "INVOIC-D96A", "CONTRL-D96A", "ORDERS-D01B", "ORDRSP-D01B", "DESADV-D01B", "INVOIC-D01B", "CONTRL-D01B", "837P-005010X222A1", "837I-005010X223A2", "835-005010X221A1", "270-005010X279A1", "271-005010X279A1", "276-005010X212", "277-005010X212", "834-005010X220A1");
/*******************************************************************************
** Read the documented JSON format in memory.
*******************************************************************************/
public Guide read(InputStream input) throws IOException
{
return (new ObjectMapper().readValue(input, Guide.class));
}
/*******************************************************************************
** Read one bundled baseline.
*******************************************************************************/
public Guide resource(String name) throws IOException
{
try(InputStream input = BuiltInGuideLoader.class.getResourceAsStream("/edi/guides/" + name + ".json"))
{
if(input == null)
{
throw new IOException("Unknown built-in guide");
}
return (read(input));
}
}
/*******************************************************************************
** Install changed guides after the host schema and service session are ready.
*******************************************************************************/
public List<Integer> load() throws Exception
{
List<Integer> ids = new ArrayList<>();
for(String resource : RESOURCES)
{
try(InputStream input = BuiltInGuideLoader.class.getResourceAsStream("/edi/guides/" + resource + ".json"))
{
ids.add(upsert(input.readAllBytes()));
}
}
return (ids);
}
/*******************************************************************************
** Retain previous revisions; identical content is a no-op.
*******************************************************************************/
public Integer upsert(byte[] json) throws Exception
{
Guide guide = read(new ByteArrayInputStream(json));
String checksum = PayloadCrypto.digest(json);
List<QRecord> revisions = EdiRecords.query("edi_guide", EdiRecords.equal("name", guide.name()));
int revision = 0;
QRecord latest = null;
for(QRecord row : revisions)
{
if("BUILT_IN".equals(row.getValueString("source")) && (latest == null || row.getValueInteger("revision") > latest.getValueInteger("revision")))
{
latest = row;
}
revision = Math.max(revision, row.getValueInteger("revision"));
}
if(latest != null && checksum.equals(latest.getValueString("compiledChecksum")))
{
return (latest.getValueInteger("id"));
}
return (new GuideRepository().save(guide, checksum, revision + 1));
}
}
@@ -0,0 +1,37 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.time.LocalDate;
/*******************************************************************************
** Host-owned, licensed code sets. Implementations must not log submitted values.
*******************************************************************************/
@FunctionalInterface
public interface ExternalCodeSetProvider
{
/*******************************************************************************
** Check membership on the document's effective service or payment date.
*******************************************************************************/
boolean isValid(String codeSet, String code, LocalDate asOf);
}
@@ -0,0 +1,32 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.util.List;
/*******************************************************************************
** In-house or imported implementation guide. Segment order is significant.
*******************************************************************************/
public record Guide(String name, String standard, String version, String transactionSet, String functionalIdCode, List<GuideSegment> segments, List<GuideRule> rules)
{
}
@@ -0,0 +1,39 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.util.List;
/*******************************************************************************
** An element or composite component constraint.
*******************************************************************************/
public record GuideElement(Integer position, Integer componentPosition, String elementId, String dataType, Integer minLength, Integer maxLength, String usage, List<String> codes, String fixedValue, Integer repeatMax, String condition, String source, String externalCodeSet)
{
/*******************************************************************************
** Preserve the original constructor for host-authored inline constraints.
*******************************************************************************/
public GuideElement(Integer position, Integer componentPosition, String elementId, String dataType, Integer minLength, Integer maxLength, String usage, List<String> codes, String fixedValue, Integer repeatMax, String condition)
{
this(position, componentPosition, elementId, dataType, minLength, maxLength, usage, codes, fixedValue, repeatMax, condition, null, null);
}
}
@@ -0,0 +1,247 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Load and persist versioned normalized guide rows through standard QQQ actions.
*******************************************************************************/
public class GuideRepository
{
/*******************************************************************************
** Read a guide and all its ordered definitions.
*******************************************************************************/
public Guide load(Integer id) throws Exception
{
return (load(id, false));
}
/*******************************************************************************
** UI preview explicitly enforces every guide read, independently of host defaults.
*******************************************************************************/
public Guide load(Integer id, boolean enforcePermissions) throws Exception
{
var guideRows = rows("edi_guide", "id", id, enforcePermissions);
if(guideRows.isEmpty())
{
throw (new com.kingsrook.qqq.backend.core.exceptions.QNotFoundException("Not found"));
}
QRecord guide = guideRows.getFirst();
List<QRecord> definitions = new ArrayList<>(rows("edi_guide_segment", "guideId", id, enforcePermissions));
definitions.sort(Comparator.comparing(r -> r.getValueInteger("sequence")));
List<GuideSegment> segments = new ArrayList<>();
List<QRecord> rules = rows("edi_guide_rule", "guideId", id, enforcePermissions);
for(QRecord definition : definitions)
{
Integer segmentId = definition.getValueInteger("id");
List<GuideElement> elements = new ArrayList<>();
for(QRecord element : rows("edi_guide_element", "guideSegmentId", segmentId, enforcePermissions))
{
List<String> codes = null;
QRecord codeList = null;
if(element.getValueInteger("codeListId") != null)
{
codeList = rows("edi_code_list", "id", element.getValueInteger("codeListId"), enforcePermissions).getFirst();
codes = new ArrayList<>();
for(QRecord code : rows("edi_code_list_value", "codeListId", element.getValueInteger("codeListId"), enforcePermissions))
{
if(Boolean.TRUE.equals(code.getValueBoolean("isActive")))
{
codes.add(code.getValueString("code"));
}
}
}
elements.add(new GuideElement(element.getValueInteger("position"), element.getValueInteger("componentPosition"), element.getValueString("elementId"),
element.getValueString("dataType"), element.getValueInteger("minLength"), element.getValueInteger("maxLength"), element.getValueString("usage"), codes,
element.getValueString("fixedValue"), element.getValueInteger("repeatMax"), condition(rules, element.getValueInteger("situationalRuleId")), codeList == null ? null : codeList.getValueString("source"), codeList == null ? null : codeList.getValueString("externalCodeSet")));
}
List<GuideRule> segmentRules = rules.stream().filter(r -> segmentId.equals(r.getValueInteger("guideSegmentId")) && !"condition".equals(r.getValueString("message"))).map(GuideRepository::rule).toList();
segments.add(new GuideSegment(definition.getValueString("segmentId"), definition.getValueString("loopId"), definition.getValueString("parentLoopId"),
definition.getValueInteger("loopMaxRepeat"), definition.getValueString("usage"), definition.getValueInteger("minUse"), definition.getValueInteger("maxUse"),
elements, segmentRules, condition(rules, definition.getValueInteger("situationalRuleId"))));
}
return (new Guide(guide.getValueString("name"), guide.getValueString("standard"), guide.getValueString("version"), guide.getValueString("transactionSet"),
guide.getValueString("functionalIdCode"), segments, rules.stream().filter(r -> r.getValueInteger("guideSegmentId") == null).map(GuideRepository::rule).toList()));
}
/*******************************************************************************
** Insert a complete new revision atomically, retaining every previous revision.
*******************************************************************************/
public Integer save(Guide guide, String checksum, int revision) throws Exception
{
try(QBackendTransaction transaction = QBackendTransaction.openFor(new InsertInput("edi_guide")))
{
try
{
QRecord row = insert("edi_guide", new QRecord().withValue("name", guide.name()).withValue("standard", guide.standard()).withValue("version", guide.version())
.withValue("transactionSet", guide.transactionSet()).withValue("functionalIdCode", guide.functionalIdCode()).withValue("revision", revision)
.withValue("status", "ACTIVE").withValue("source", "BUILT_IN").withValue("compiledChecksum", checksum), transaction);
Integer id = row.getValueInteger("id");
int sequence = 0;
for(GuideSegment segment : guide.segments())
{
QRecord segmentRow = insert("edi_guide_segment", new QRecord().withValue("guideId", id).withValue("sequence", ++sequence)
.withValue("segmentId", segment.segmentId()).withValue("name", segment.segmentId()).withValue("position", String.format("%04d", sequence * 10))
.withValue("loopId", segment.loopId()).withValue("parentLoopId", segment.parentLoopId()).withValue("loopMaxRepeat", segment.loopMaxRepeat())
.withValue("usage", segment.usage()).withValue("minUse", segment.minUse()).withValue("maxUse", segment.maxUse()), transaction);
Integer segmentId = segmentRow.getValueInteger("id");
for(GuideElement element : list(segment.elements()))
{
Integer codeListId = null;
if(element.codes() != null || "EXTERNAL".equals(element.source()))
{
codeListId = insert("edi_code_list", new QRecord().withValue("name", guide.name() + "/" + revision + "/" + sequence + "/" + element.position())
.withValue("elementId", element.elementId()).withValue("source", element.source() == null ? "BUILT_IN" : element.source()).withValue("externalCodeSet", element.externalCodeSet()), transaction).getValueInteger("id");
for(String code : list(element.codes()))
{
insert("edi_code_list_value", new QRecord().withValue("codeListId", codeListId).withValue("code", code).withValue("isActive", true), transaction);
}
}
insert("edi_guide_element", new QRecord().withValue("guideSegmentId", segmentId).withValue("position", element.position())
.withValue("componentPosition", element.componentPosition()).withValue("elementId", element.elementId()).withValue("dataType", element.dataType())
.withValue("minLength", element.minLength()).withValue("maxLength", element.maxLength()).withValue("usage", element.usage())
.withValue("codeListId", codeListId).withValue("fixedValue", element.fixedValue()).withValue("repeatMax", element.repeatMax())
.withValue("situationalRuleId", saveCondition(id, segmentId, element.condition(), transaction)), transaction);
}
if(segment.condition() != null)
{
new UpdateAction().execute(new UpdateInput("edi_guide_segment").withTransaction(transaction).withRecord(new QRecord().withValue("id", segmentId)
.withValue("situationalRuleId", saveCondition(id, segmentId, segment.condition(), transaction))));
}
for(GuideRule rule : list(segment.rules()))
{
saveRule(id, segmentId, rule, transaction);
}
}
for(GuideRule rule : list(guide.rules()))
{
saveRule(id, null, rule, transaction);
}
for(QRecord previous : EdiRecords.query("edi_guide", EdiRecords.equal("name", guide.name())))
{
if("BUILT_IN".equals(previous.getValueString("source")) && "ACTIVE".equals(previous.getValueString("status")) && !id.equals(previous.getValueInteger("id")))
{
QRecord updated = new UpdateAction().execute(new UpdateInput("edi_guide").withTransaction(transaction)
.withRecord(new QRecord().withValue("id", previous.getValueInteger("id")).withValue("status", "RETIRED"))).getRecords().getFirst();
if(updated.getErrors() != null && !updated.getErrors().isEmpty())
{
throw new QException("Previous guide revision could not be retired");
}
}
}
transaction.commit();
return (id);
}
catch(Exception e)
{
transaction.rollback();
throw e;
}
}
}
/*******************************************************************************
** Store a condition as a referenced guide rule.
*******************************************************************************/
private static Integer saveCondition(Integer guideId, Integer segmentId, String expression, QBackendTransaction transaction) throws Exception
{
if(expression == null)
{
return (null);
}
return (saveRule(guideId, segmentId, new GuideRule("SITUATIONAL", List.of(), expression, 2, "ERROR", "condition", "8", "2", "8", "2"), transaction));
}
/*******************************************************************************
** Store syntax and situational rules without losing ack codes.
*******************************************************************************/
private static Integer saveRule(Integer guideId, Integer segmentId, GuideRule rule, QBackendTransaction transaction) throws Exception
{
return (insert("edi_guide_rule", new QRecord().withValue("guideId", guideId).withValue("guideSegmentId", segmentId).withValue("type", rule.type())
.withValue("positions", list(rule.positions()).stream().map(Object::toString).collect(Collectors.joining(","))).withValue("expression", rule.expression())
.withValue("snipLevel", rule.snipLevel()).withValue("severity", rule.severity()).withValue("message", rule.message())
.withValue("ak3Code", rule.ak3Code()).withValue("ak4Code", rule.ak4Code()).withValue("ik3Code", rule.ik3Code()).withValue("ik4Code", rule.ik4Code()), transaction).getValueInteger("id"));
}
/*******************************************************************************
** Reconstruct a syntax or situational rule.
*******************************************************************************/
private static GuideRule rule(QRecord row)
{
String positions = row.getValueString("positions");
return (new GuideRule(row.getValueString("type"), positions == null || positions.isBlank() ? List.of() : Arrays.stream(positions.split(",")).map(Integer::valueOf).toList(),
row.getValueString("expression"), row.getValueInteger("snipLevel"), row.getValueString("severity"), row.getValueString("message"), row.getValueString("ak3Code"),
row.getValueString("ak4Code"), row.getValueString("ik3Code"), row.getValueString("ik4Code")));
}
/*******************************************************************************
** Resolve a condition by its explicit row id.
*******************************************************************************/
private static String condition(List<QRecord> rules, Integer id)
{
return (rules.stream().filter(r -> Objects.equals(r.getValueInteger("id"), id)).map(r -> r.getValueString("expression")).findFirst().orElse(null));
}
/*******************************************************************************
** Query normalized child records.
*******************************************************************************/
private static List<QRecord> rows(String table, String field, Integer id, boolean enforcePermissions) throws Exception
{
return (enforcePermissions ? EdiRecords.queryEnforced(table, EdiRecords.equal(field, id)) : EdiRecords.query(table, EdiRecords.equal(field, id)));
}
/*******************************************************************************
** Insert within the owning guide transaction, checking validation errors.
*******************************************************************************/
private static QRecord insert(String table, QRecord row, QBackendTransaction transaction) throws Exception
{
QRecord result = new InsertAction().execute(new InsertInput(table).withTransaction(transaction).withRecord(row)).getRecords().getFirst();
if(result.getErrors() != null && !result.getErrors().isEmpty())
{
throw new QException("Guide record validation failed for " + table);
}
return (result);
}
/*******************************************************************************
** Treat omitted optional lists as empty.
*******************************************************************************/
private static <T> List<T> list(List<T> list)
{
return (list == null ? List.of() : list);
}
}
@@ -0,0 +1,32 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.util.List;
/*******************************************************************************
** A syntax note or small-language expression; no executable host code.
*******************************************************************************/
public record GuideRule(String type, List<Integer> positions, String expression, Integer snipLevel, String severity, String message, String ak3Code, String ak4Code, String ik3Code, String ik4Code)
{
}
@@ -0,0 +1,32 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.util.List;
/*******************************************************************************
** Ordered segment definition; the first segment of a loop starts each occurrence.
*******************************************************************************/
public record GuideSegment(String segmentId, String loopId, String parentLoopId, Integer loopMaxRepeat, String usage, Integer minUse, Integer maxUse, List<GuideElement> elements, List<GuideRule> rules, String condition)
{
}
@@ -0,0 +1,506 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.document.EdiDocument;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** SNIP 1-2 structural and semantic validation over StAEDI-tokenized segments.
*******************************************************************************/
public class GuideValidator
{
private final com.kingsrook.qbits.edi.EdiQBitConfig config;
/*******************************************************************************
** Standalone validation uses built-in formats without a licensed provider.
*******************************************************************************/
public GuideValidator()
{
this(new com.kingsrook.qbits.edi.EdiQBitConfig());
}
/*******************************************************************************
** Read the host provider at validation time, allowing runtime registration.
*******************************************************************************/
public GuideValidator(com.kingsrook.qbits.edi.EdiQBitConfig config)
{
this.config = config;
}
/*******************************************************************************
** Validate one transaction; diagnostic text never contains submitted values.
*******************************************************************************/
public List<ValidationFinding> validate(EdiDocument transaction, Guide guide, boolean containsPhi)
{
State state = new State(transaction.segments(), containsPhi, "EDIFACT".equals(guide.standard()));
List<Node> roots = tree(guide);
consume(roots, state, false);
while(state.index < state.segments.size())
{
X12Segment segment = state.segments.get(state.index++);
boolean known = guide.segments().stream().anyMatch(s -> s.segmentId().equals(segment.id()));
state.add(segment, null, null, known ? "7" : "2", null, known ? "Segment out of order" : "Unexpected segment", null);
}
RuleEvaluator evaluator = new RuleEvaluator(state.segments, state.loops);
for(GuideSegment definition : guide.segments())
{
if(definition.condition() != null)
{
List<List<X12Segment>> scopes = definition.loopId() == null ? List.of(state.segments) : state.loops.getOrDefault(definition.loopId(), List.of(state.segments));
for(List<X12Segment> scope : scopes)
{
RuleEvaluator scoped = new RuleEvaluator(scope, state.loops);
if(safeTest(scoped, definition.condition()) && state.bindings.stream().noneMatch(b -> b.definition() == definition && scope.contains(b.segment())))
{
state.add(X12Segment.of(definition.segmentId(), scope.isEmpty() ? 1 : scope.getLast().position() + 1), definition.loopId(), null, "3", null, "Situational segment required", null);
int last = state.findings.size() - 1;
state.findings.set(last, state.findings.get(last).withContext(scoped.context(definition.condition())));
}
}
}
}
for(GuideRule rule : list(guide.rules()))
{
rule(rule, transaction.header(), null, evaluator, state);
}
for(Binding binding : state.bindings)
{
GuideSegment definition = binding.definition();
X12Segment segment = binding.segment();
elements(segment, definition, evaluator, state);
external(segment, definition, state);
for(GuideRule rule : list(definition.rules()))
{
rule(rule, segment, definition.loopId(), "SITUATIONAL".equals(rule.type()) && definition.loopId() != null ? scoped(binding, state) : evaluator, state);
}
}
if(guide.version().startsWith("005010X"))
{
state.findings.addAll(new HealthcareValidation().validate(transaction, guide, state.loops));
}
return (List.copyOf(state.findings));
}
/*******************************************************************************
** Resolve guide-defined loop occurrences for the mapper using the validator grammar.
*******************************************************************************/
public Map<String, List<List<X12Segment>>> loops(EdiDocument transaction, Guide guide)
{
State state = new State(transaction.segments(), true, "EDIFACT".equals(guide.standard()));
List<Node> roots = tree(guide);
consume(roots, state, false);
return (state.loops);
}
/*******************************************************************************
** The same loop name can occur under subscriber and dependent patient parents.
*******************************************************************************/
private static List<Node> tree(Guide guide)
{
List<Node> roots = new ArrayList<>();
Map<String, Node> latest = new LinkedHashMap<>();
for(GuideSegment segment : guide.segments())
{
List<Node> destination = roots;
if(segment.loopId() != null && !segment.loopId().isBlank())
{
Node parent = latest.get(segment.parentLoopId());
List<Node> siblings = parent == null ? roots : parent.children;
Node loop = siblings.stream().filter(n -> segment.loopId().equals(n.loopId)).findFirst().orElse(null);
if(loop == null)
{
loop = new Node(segment.loopId(), null);
siblings.add(loop);
}
latest.put(segment.loopId(), loop);
destination = loop.children;
}
destination.add(new Node(null, segment));
}
return (roots);
}
/*******************************************************************************
** Situational rules see their occurrence plus direct ancestor segments, never siblings.
*******************************************************************************/
private static RuleEvaluator scoped(Binding binding, State state)
{
List<X12Segment> scope = state.loops.getOrDefault(binding.definition().loopId(), List.of()).stream()
.filter(o -> o.contains(binding.segment())).min(java.util.Comparator.comparingInt(List::size)).orElse(state.segments);
List<X12Segment> visible = new ArrayList<>(scope);
for(List<X12Segment> ancestor : state.loops.values().stream().flatMap(List::stream).toList())
{
if(ancestor.size() > scope.size() && ancestor.contains(scope.getFirst()))
{
for(X12Segment segment : ancestor)
{
boolean nested = state.loops.values().stream().flatMap(List::stream)
.anyMatch(o -> o.size() < ancestor.size() && ancestor.contains(o.getFirst()) && o.contains(segment));
if(!nested && !visible.contains(segment))
{
visible.add(segment);
}
}
}
}
visible.sort(java.util.Comparator.comparingInt(X12Segment::position));
return (new RuleEvaluator(visible, state.loops));
}
/*******************************************************************************
** Consume ordered segments and nested loop occurrences.
*******************************************************************************/
private static void consume(List<Node> nodes, State state, boolean loopOccurrence)
{
for(Node node : nodes)
{
if(node.definition == null)
{
GuideSegment head = node.children.getFirst().definition;
int count = 0;
while(state.matches(head))
{
int start = state.index;
consume(node.children, state, true);
if(start == state.index)
{
break;
}
state.loops.computeIfAbsent(node.loopId, key -> new ArrayList<>()).add(state.segments.subList(start, state.index));
if(++count > number(head.loopMaxRepeat(), 1))
{
state.add(state.segments.get(start), node.loopId, null, "4", null, "Loop repeat limit exceeded", null);
}
}
if(count == 0 && "REQUIRED".equals(head.usage()))
{
state.add(X12Segment.of(head.segmentId(), state.position()), node.loopId, null, "3", null, "Required loop missing", null);
}
}
else
{
GuideSegment definition = node.definition;
int count = 0;
int maximum = number(definition.maxUse(), 1);
while(state.matches(definition.segmentId()) && (node != nodes.getFirst() || !loopOccurrence || state.matches(definition)) && (count == 0 || !loopOccurrence || node != nodes.getFirst()))
{
X12Segment segment = state.segments.get(state.index++);
state.bindings.add(new Binding(segment, definition));
count++;
if("NOT_USED".equals(definition.usage()))
{
state.add(segment, definition.loopId(), null, "6", null, "Segment is not used", null);
}
if(count > maximum)
{
state.add(segment, definition.loopId(), null, "5", null, "Segment use limit exceeded", null);
}
}
int minimum = Math.max(number(definition.minUse(), 0), "REQUIRED".equals(definition.usage()) ? 1 : 0);
if(count < minimum)
{
state.add(X12Segment.of(definition.segmentId(), state.position()), definition.loopId(), null, "3", null, "Required segment missing", null);
}
}
}
}
/*******************************************************************************
** Apply usage, lengths, data types, code lists, fixed values and repetition bounds.
*******************************************************************************/
private static void elements(X12Segment segment, GuideSegment definition, RuleEvaluator evaluator, State state)
{
for(GuideElement element : list(definition.elements()))
{
int repetitions = element.position() <= segment.elements().size() ? segment.elements().get(element.position() - 1).size() : 1;
if(repetitions > number(element.repeatMax(), 1))
{
state.add(segment, definition.loopId(), element, "8", "12", "Element repetition limit exceeded", null);
}
for(int r = 1; r <= repetitions; r++)
{
String value = segment.value(element.position(), r, number(element.componentPosition(), 1));
boolean required = "REQUIRED".equals(element.usage()) || element.condition() != null && safeTest(evaluator, element.condition());
if(value.isEmpty())
{
if(required)
{
state.add(segment, definition.loopId(), element, "8", "1", "Required element missing", null);
if(element.condition() != null)
{
int last = state.findings.size() - 1;
state.findings.set(last, state.findings.get(last).withContext(evaluator.context(element.condition())));
}
}
continue;
}
if("NOT_USED".equals(element.usage()))
{
state.add(segment, definition.loopId(), element, "8", "10", "Element is not used", value);
}
if(value.length() < number(element.minLength(), 0))
{
state.add(segment, definition.loopId(), element, "8", "4", "Element too short", value);
}
if(value.length() > number(element.maxLength(), Integer.MAX_VALUE))
{
state.add(segment, definition.loopId(), element, "8", "5", "Element too long", value);
}
if(!validType(element.dataType(), state.edifact && "R".equals(element.dataType()) ? value.replace(',', '.') : value))
{
String code = "DT".equals(element.dataType()) ? "8" : "TM".equals(element.dataType()) ? "9" : "6";
state.add(segment, definition.loopId(), element, "8", code, "Invalid element data type", value);
}
if(element.fixedValue() != null && !element.fixedValue().equals(value)
|| !"EXTERNAL".equals(element.source()) && "ID".equals(element.dataType()) && element.codes() != null && !element.codes().contains(value))
{
state.add(segment, definition.loopId(), element, "8", "7", "Invalid code or fixed value", value);
}
}
}
}
/*******************************************************************************
** External membership is SNIP 5 and is optional; formats are always built in.
*******************************************************************************/
private void external(X12Segment segment, GuideSegment definition, State state)
{
if(config.getExternalCodeSetProvider() == null)
{
return;
}
for(GuideElement element : list(definition.elements()))
{
if(!"EXTERNAL".equals(element.source()))
{
continue;
}
int repetitions = element.position() <= segment.elements().size() ? segment.elements().get(element.position() - 1).size() : 1;
for(int repetition = 1; repetition <= repetitions; repetition++)
{
String value = segment.value(element.position(), repetition, number(element.componentPosition(), 1));
if(!value.isEmpty())
{
boolean valid;
try(var logging = com.kingsrook.qbits.edi.security.PhiLogScope.open(state.phi))
{
valid = config.getExternalCodeSetProvider().isValid(element.externalCodeSet(), value,
HealthcareValidation.asOf(state.segments, config.getClock()));
}
catch(RuntimeException e)
{
valid = false;
}
if(!valid)
{
state.findings.add(new ValidationFinding(segment.position(), segment.id(), definition.loopId(), element.position(), element.componentPosition(),
5, "8", "7", "8", "7", "ERROR", "External code validation failed", null));
}
}
}
}
}
/*******************************************************************************
** Validate X12 lexical data types without coercing bad values.
*******************************************************************************/
public static boolean validType(String type, String value)
{
if(type == null || type.equals("AN") || type.equals("ID") || type.equals("B"))
{
return (value.chars().noneMatch(Character::isISOControl));
}
if(type.matches("N[0-9]"))
{
return (value.matches("-?[0-9]+"));
}
if(type.equals("R"))
{
return (value.matches("-?(?:[0-9]+(?:\\.[0-9]*)?|\\.[0-9]+)"));
}
if(type.equals("DT"))
{
return (RuleEvaluator.validDate(value));
}
if(type.equals("TM"))
{
return (value.matches("(?:[01][0-9]|2[0-3])[0-5][0-9](?:[0-5][0-9](?:[0-9]{1,2})?)?"));
}
throw new IllegalArgumentException("Unknown guide data type");
}
/*******************************************************************************
** Evaluate syntax notes and explicit expressions; PHI rules use a safe message.
*******************************************************************************/
private static void rule(GuideRule rule, X12Segment segment, String loopId, RuleEvaluator evaluator, State state)
{
List<Integer> positions = list(rule.positions());
long present = positions.stream().filter(p -> !segment.value(p).isEmpty()).count();
boolean first = !positions.isEmpty() && !segment.value(positions.getFirst()).isEmpty();
boolean passed = switch(rule.type())
{
case "PAIRED", "P" -> present == 0 || present == positions.size();
case "REQUIRED", "R" -> present > 0;
case "EXCLUSION", "E" -> present <= 1;
case "CONDITIONAL", "C" -> !first || present == positions.size();
case "LIST_CONDITIONAL", "L" -> !first || present > 1;
case "SITUATIONAL" -> safeTest(evaluator, rule.expression());
default -> throw new IllegalArgumentException("Unknown syntax note");
};
if(!passed)
{
state.findings.add(new ValidationFinding(segment.position(), segment.id(), loopId, positions.isEmpty() ? null : positions.getFirst(), null,
number(rule.snipLevel(), 2), fallback(rule.ak3Code(), "8"), fallback(rule.ak4Code(), "2"), fallback(rule.ik3Code(), "8"), fallback(rule.ik4Code(), "2"),
fallback(rule.severity(), "ERROR"), state.phi ? "Guide rule failed" : fallback(rule.message(), "Guide rule failed"), null)
.withContext("SITUATIONAL".equals(rule.type()) ? evaluator.context(rule.expression()) : null));
}
}
/*******************************************************************************
** Bad data cannot leak through numeric or regular-expression exception messages.
*******************************************************************************/
private static boolean safeTest(RuleEvaluator evaluator, String expression)
{
try
{
return (evaluator.test(expression));
}
catch(IllegalArgumentException e)
{
return (false);
}
}
/*******************************************************************************
** Use a default for omitted optional numeric constraints.
*******************************************************************************/
private static int number(Integer value, int fallback)
{
return (value == null ? fallback : value);
}
/*******************************************************************************
** Use a default for omitted optional text.
*******************************************************************************/
private static String fallback(String value, String fallback)
{
return (value == null ? fallback : value);
}
/*******************************************************************************
** Normalize omitted JSON lists.
*******************************************************************************/
private static <T> List<T> list(List<T> value)
{
return (value == null ? List.of() : value);
}
/*******************************************************************************
** A guide tree node.
*******************************************************************************/
private static class Node
{
private final String loopId;
private final GuideSegment definition;
private final List<Node> children = new ArrayList<>();
/*******************************************************************************
** Create a loop or segment node.
*******************************************************************************/
private Node(String loopId, GuideSegment definition)
{
this.loopId = loopId;
this.definition = definition;
}
}
/*******************************************************************************
** Bind a parsed segment to its guide definition.
*******************************************************************************/
private record Binding(X12Segment segment, GuideSegment definition)
{
}
/*******************************************************************************
** Per-call state; validator instances are safe to share.
*******************************************************************************/
private static class State
{
private final List<X12Segment> segments;
private final boolean phi;
private final boolean edifact;
private final List<ValidationFinding> findings = new ArrayList<>();
private final List<Binding> bindings = new ArrayList<>();
private final Map<String, List<List<X12Segment>>> loops = new LinkedHashMap<>();
private int index;
/*******************************************************************************
** Initialize a transaction validation.
*******************************************************************************/
private State(List<X12Segment> segments, boolean phi, boolean edifact)
{
this.segments = segments;
this.phi = phi;
this.edifact = edifact;
}
/*******************************************************************************
** Check the next segment.
*******************************************************************************/
private boolean matches(String id)
{
return (index < segments.size() && segments.get(index).id().equals(id));
}
/*******************************************************************************
** Fixed qualifiers distinguish adjacent HL and NM1 loop anchors.
*******************************************************************************/
private boolean matches(GuideSegment definition)
{
return (matches(definition.segmentId()) && list(definition.elements()).stream().filter(e -> e.fixedValue() != null)
.allMatch(e -> e.fixedValue().equals(segments.get(index).value(e.position(), 1, number(e.componentPosition(), 1)))));
}
/*******************************************************************************
** Locate a missing segment relative to the buffered transaction.
*******************************************************************************/
private int position()
{
return (index < segments.size() ? segments.get(index).position() : segments.size() + 1);
}
/*******************************************************************************
** Add a diagnostic with a bounded, optionally suppressed excerpt.
*******************************************************************************/
private void add(X12Segment segment, String loopId, GuideElement element, String segmentCode, String elementCode, String message, String value)
{
findings.add(new ValidationFinding(segment.position(), segment.id(), loopId, element == null ? null : element.position(), element == null ? null : element.componentPosition(),
2, segmentCode, elementCode, segmentCode, elementCode, "ERROR", message, phi || value == null ? null : value.substring(0, Math.min(64, value.length()))));
}
}
}
@@ -0,0 +1,250 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.math.BigDecimal;
import java.time.Clock;
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.document.EdiDocument;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** Original healthcare balancing and format checks, without licensed code content.
*******************************************************************************/
public class HealthcareValidation
{
/*******************************************************************************
** Exact decimal arithmetic; every diagnostic contains only positions and codes.
*******************************************************************************/
public List<ValidationFinding> validate(EdiDocument transaction, Guide guide, Map<String, List<List<X12Segment>>> loops)
{
List<ValidationFinding> findings = new ArrayList<>();
List<X12Segment> segments = transaction.segments();
if("837".equals(guide.transactionSet()))
{
for(List<X12Segment> claim : blocks(segments, "CLM"))
{
X12Segment head = claim.getFirst();
try
{
BigDecimal lines = BigDecimal.ZERO;
for(X12Segment line : claim)
{
if(line.id().equals("SV1") || line.id().equals("SV2"))
{
lines = lines.add(amount(line, line.id().equals("SV1") ? 2 : 3));
}
}
if(amount(head, 2).compareTo(lines) != 0)
{
findings.add(finding(head, "2300", 2, null, 3, "Claim charges do not balance"));
}
}
catch(NumberFormatException e)
{
findings.add(finding(head, "2300", 2, null, 3, "Claim balance cannot be calculated"));
}
}
}
if("835".equals(guide.transactionSet()))
{
remittance(segments, findings);
}
for(X12Segment segment : segments)
{
String loop = loops.entrySet().stream().filter(e -> e.getValue().stream().anyMatch(o -> o.contains(segment)))
.min(java.util.Comparator.comparingInt(e -> e.getValue().stream().filter(o -> o.contains(segment)).mapToInt(List::size).min().orElse(Integer.MAX_VALUE)))
.map(Map.Entry::getKey).orElse(null);
if(segment.id().equals("NM1") && segment.value(8).equals("XX") && !validNpi(segment.value(9)))
{
findings.add(finding(segment, loop, 9, null, 5, "Invalid NPI format or check digit"));
}
if(segment.id().equals("N4") && !segment.value(3).isEmpty() && (segment.value(4).isEmpty() || segment.value(4).equals("US"))
&& !segment.value(3).matches("[0-9]{5}([0-9]{4}|-[0-9]{4})?"))
{
findings.add(finding(segment, loop, 3, null, 5, "Invalid postal format"));
}
if(segment.id().equals("HI"))
{
for(int position = 1; position <= segment.elements().size(); position++)
{
for(int repetition = 1; repetition <= segment.elements().get(position - 1).size(); repetition++)
{
if(List.of("ABK", "ABF", "ABJ", "APR").contains(segment.value(position, repetition, 1))
&& !segment.value(position, repetition, 2).matches("[A-Z][0-9][A-Z0-9](?:[A-Z0-9]{0,4}|\\.[A-Z0-9]{1,4})"))
{
findings.add(finding(segment, loop, position, 2, 5, "Invalid ICD-10 diagnosis format"));
}
}
}
}
}
return (findings);
}
/*******************************************************************************
** Claim and line CAS monetary positions are 3, 6, 9, 12, 15 and 18.
*******************************************************************************/
private static void remittance(List<X12Segment> segments, List<ValidationFinding> findings)
{
for(List<X12Segment> claim : blocks(segments, "CLP"))
{
X12Segment head = claim.getFirst();
try
{
BigDecimal adjustments = BigDecimal.ZERO;
for(X12Segment segment : claim)
{
if(segment.id().equals("CAS"))
{
for(int p = 3; p <= segment.elements().size(); p += 3)
{
adjustments = adjustments.add(amount(segment, p));
}
}
}
if(amount(head, 3).subtract(amount(head, 4)).compareTo(adjustments) != 0)
{
findings.add(finding(head, "2100", 3, null, 3, "Claim adjustments do not balance"));
}
}
catch(NumberFormatException e)
{
findings.add(finding(head, "2100", 3, null, 3, "Claim balance cannot be calculated"));
}
}
for(X12Segment payment : segments.stream().filter(s -> s.id().equals("BPR")).toList())
{
try
{
BigDecimal total = BigDecimal.ZERO;
for(X12Segment segment : segments)
{
if(segment.id().equals("CLP"))
{
total = total.add(amount(segment, 4));
}
if(segment.id().equals("PLB"))
{
for(int p = 4; p <= segment.elements().size(); p += 2)
{
total = total.subtract(amount(segment, p));
}
}
}
if(amount(payment, 2).compareTo(total) != 0)
{
findings.add(finding(payment, null, 2, null, 3, "Payment does not balance"));
}
}
catch(NumberFormatException e)
{
findings.add(finding(payment, null, 2, null, 3, "Payment balance cannot be calculated"));
}
}
}
/*******************************************************************************
** Isolate each claim so one claim cannot compensate for another's imbalance.
*******************************************************************************/
private static List<List<X12Segment>> blocks(List<X12Segment> segments, String anchor)
{
List<List<X12Segment>> blocks = new ArrayList<>();
List<X12Segment> current = null;
for(X12Segment segment : segments)
{
if(segment.id().equals(anchor))
{
current = new ArrayList<>();
blocks.add(current);
}
if(current != null)
{
current.add(segment);
}
}
return (blocks);
}
/*******************************************************************************
** Missing monetary elements fail balancing rather than silently becoming zero.
*******************************************************************************/
private static BigDecimal amount(X12Segment segment, int position)
{
return (new BigDecimal(segment.value(position)));
}
/*******************************************************************************
** Position-only finding with a positional trigger for the implementation ack.
*******************************************************************************/
private static ValidationFinding finding(X12Segment segment, String loop, Integer element, Integer component, int level, String message)
{
return (new ValidationFinding(segment.position(), segment.id(), loop, element, component, level,
"8", "7", "8", "7", "ERROR", message, null)
.withContext(level == 3 ? new AckContext(segment.id(), segment.position(), loop, element, component) : null));
}
/*******************************************************************************
** NPI uses the Luhn checksum over the 80840 prefix and all ten digits.
*******************************************************************************/
public static boolean validNpi(String value)
{
if(!value.matches("[12][0-9]{9}"))
{
return (false);
}
String digits = "80840" + value;
int sum = 0;
for(int i = digits.length() - 1, n = 0; i >= 0; i--, n++)
{
int digit = digits.charAt(i) - '0';
if(n % 2 == 1)
{
digit *= 2;
}
sum += digit > 9 ? digit - 9 : digit;
}
return (sum % 10 == 0);
}
/*******************************************************************************
** Service date, then payment date, then the configured clock for undated documents.
*******************************************************************************/
public static LocalDate asOf(List<X12Segment> segments, Clock clock)
{
for(X12Segment segment : segments)
{
String date = segment.id().equals("DTP") && segment.value(1).equals("472") ? segment.value(3)
: segment.id().equals("BPR") ? segment.value(16) : "";
if(RuleEvaluator.validDate(date) && date.length() == 8)
{
return (LocalDate.parse(date, DateTimeFormatter.BASIC_ISO_DATE));
}
}
return (LocalDate.now(clock));
}
}
@@ -0,0 +1,363 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
import java.math.BigDecimal;
import java.time.DateTimeException;
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.kingsrook.qbits.edi.x12.X12Segment;
/*******************************************************************************
** Parser and evaluator for the closed EDI rule language. No scripting or reflection.
*******************************************************************************/
public class RuleEvaluator
{
private final List<X12Segment> segments;
private final Map<String, List<List<X12Segment>>> loops;
/*******************************************************************************
** Bind document segments and guide-derived loop occurrences.
*******************************************************************************/
public RuleEvaluator(List<X12Segment> segments, Map<String, List<List<X12Segment>>> loops)
{
this.segments = segments;
this.loops = loops;
}
/*******************************************************************************
** Require the expression to return a boolean.
*******************************************************************************/
public boolean test(String expression)
{
Object result = evaluate(expression.trim());
if(!(result instanceof Boolean))
{
throw new IllegalArgumentException("Rule must return a boolean");
}
return ((Boolean) result);
}
/*******************************************************************************
** Evaluate one expression recursively with explicit functions only.
*******************************************************************************/
public Object evaluate(String expression)
{
String text = expression.trim();
if(text.startsWith("'") && text.endsWith("'"))
{
return (text.substring(1, text.length() - 1).replace("\'", "'"));
}
if(text.matches("-?[0-9]+(?:\\.[0-9]+)?"))
{
return (new BigDecimal(text));
}
int open = text.indexOf('(');
if(open < 0)
{
return (values(text));
}
if(!text.endsWith(")"))
{
throw new IllegalArgumentException("Unclosed rule function");
}
String function = text.substring(0, open).trim();
List<String> args = arguments(text.substring(open + 1, text.length() - 1));
int arity = switch(function)
{
case "present", "absent", "count", "sum", "date", "not" -> 1;
case "eq", "matches", "when" -> 2;
case "in" -> -2;
case "and", "or" -> -1;
default -> throw new IllegalArgumentException("Unknown rule function");
};
if(arity > 0 && args.size() != arity || arity < 0 && args.size() < -arity)
{
throw new IllegalArgumentException("Invalid rule argument count");
}
return (switch(function)
{
case "present" -> !asValues(evaluate(arg(args, 0))).isEmpty();
case "absent" -> asValues(evaluate(arg(args, 0))).isEmpty();
case "count" -> BigDecimal.valueOf(asValues(evaluate(arg(args, 0))).size());
case "sum" -> asValues(evaluate(arg(args, 0))).stream().map(BigDecimal::new).reduce(BigDecimal.ZERO, BigDecimal::add);
case "eq" -> equal(evaluate(arg(args, 0)), evaluate(arg(args, 1)));
case "in" -> args.subList(1, args.size()).stream().anyMatch(a -> equal(evaluate(arg(args, 0)), evaluate(a)));
case "matches" -> asValues(evaluate(arg(args, 0))).stream().allMatch(v -> v.matches(scalar(evaluate(arg(args, 1)))));
case "date" -> validDate(scalar(evaluate(arg(args, 0))));
case "and" -> args.stream().allMatch(this::test);
case "or" -> args.stream().anyMatch(this::test);
case "not" -> !test(arg(args, 0));
case "when" -> !test(arg(args, 0)) || test(arg(args, 1));
default -> throw new IllegalArgumentException("Unknown rule function");
});
}
/*******************************************************************************
** Extract a required argument.
*******************************************************************************/
private static String arg(List<String> args, int index)
{
if(index >= args.size())
{
throw new IllegalArgumentException("Missing rule argument");
}
return (args.get(index));
}
/*******************************************************************************
** Split function arguments outside nested calls, predicates and quoted strings.
*******************************************************************************/
private static List<String> arguments(String text)
{
List<String> result = new ArrayList<>();
boolean quoted = false;
int depth = 0;
int start = 0;
for(int i = 0; i < text.length(); i++)
{
char c = text.charAt(i);
if(c == '\'' && (i == 0 || text.charAt(i - 1) != '\\'))
{
quoted = !quoted;
}
if(!quoted)
{
if(c == '(' || c == '[')
{
depth++;
}
if(c == ')' || c == ']')
{
depth--;
}
if(c == ',' && depth == 0)
{
result.add(text.substring(start, i).trim());
start = i + 1;
}
}
}
if(quoted || depth != 0)
{
throw new IllegalArgumentException("Invalid rule syntax");
}
result.add(text.substring(start).trim());
return (result);
}
/*******************************************************************************
** Resolve loop paths, segment predicates, one-based indexes and components.
*******************************************************************************/
public List<String> values(String path)
{
List<X12Segment> scope = segments;
String[] steps = path.split("/");
for(int i = 0; i < steps.length; i++)
{
String step = steps[i];
Matcher selector = Pattern.compile("([^\\[]+)(?:\\[(.*)])?").matcher(step);
if(!selector.matches())
{
throw new IllegalArgumentException("Invalid EDI path");
}
String name = selector.group(1);
String qualifier = selector.group(2);
Matcher element = Pattern.compile("([A-Z][A-Z0-9]{1,2}?)([0-9]{2})(?:-([0-9]+))?").matcher(name);
if(element.matches() && !loops.containsKey(name))
{
List<X12Segment> selected = select(scope, element.group(1), qualifier);
int position = Integer.parseInt(element.group(2));
int component = element.group(3) == null ? 1 : Integer.parseInt(element.group(3));
List<String> result = new ArrayList<>();
for(X12Segment segment : selected)
{
if(position <= segment.elements().size())
{
for(int r = 1; r <= segment.elements().get(position - 1).size(); r++)
{
String value = segment.value(position, r, component);
if(!value.isEmpty())
{
result.add(value);
}
}
}
}
return (result);
}
List<List<X12Segment>> occurrences = loops.get(name);
if(occurrences == null)
{
List<List<X12Segment>> byAnchor = loops.values().stream().flatMap(List::stream)
.filter(o -> !o.isEmpty() && o.getFirst().id().equals(name)).distinct()
.sorted(java.util.Comparator.comparingInt(o -> o.getFirst().position())).toList();
if(!byAnchor.isEmpty())
{
occurrences = byAnchor;
}
}
if(occurrences != null)
{
List<X12Segment> priorScope = scope;
occurrences = occurrences.stream().filter(o -> !o.isEmpty() && priorScope.contains(o.getFirst())).toList();
List<X12Segment> heads = select(occurrences.stream().map(List::getFirst).toList(), null, qualifier);
scope = occurrences.stream().filter(o -> heads.contains(o.getFirst())).flatMap(List::stream).toList();
if(i == steps.length - 1)
{
return (heads.stream().map(X12Segment::id).toList());
}
}
else
{
scope = select(scope, name, qualifier);
}
}
return (scope.stream().map(X12Segment::id).toList());
}
/*******************************************************************************
** Apply segment selection without evaluating arbitrary expressions.
*******************************************************************************/
private static List<X12Segment> select(List<X12Segment> scope, String name, String qualifier)
{
List<X12Segment> result = scope.stream().filter(s -> name == null || name.equals(s.id())).toList();
if(qualifier == null || qualifier.equals("*"))
{
return (result);
}
if(qualifier.matches("[0-9]+"))
{
int index = Integer.parseInt(qualifier) - 1;
return (index >= 0 && index < result.size() ? List.of(result.get(index)) : List.of());
}
Matcher predicate = Pattern.compile("[A-Z][A-Z0-9]{1,2}?([0-9]{2})(?:-([0-9]+))?=(.*)").matcher(qualifier);
if(!predicate.matches())
{
throw new IllegalArgumentException("Invalid path predicate");
}
int position = Integer.parseInt(predicate.group(1));
int component = predicate.group(2) == null ? 1 : Integer.parseInt(predicate.group(2));
String expected = predicate.group(3).replace("'", "");
return (result.stream().filter(s -> s.value(position, 1, component).equals(expected)).toList());
}
/*******************************************************************************
** Convert values to a scalar for scalar-only functions.
*******************************************************************************/
private static String scalar(Object value)
{
List<String> values = asValues(value);
return (values.isEmpty() ? "" : values.getFirst());
}
/*******************************************************************************
** Normalize scalar and path results.
*******************************************************************************/
private static List<String> asValues(Object value)
{
if(value instanceof List<?> list)
{
return (list.stream().map(Object::toString).toList());
}
return (List.of(value.toString()));
}
/*******************************************************************************
** Compare numerically when either argument is numeric, otherwise by value.
*******************************************************************************/
private static boolean equal(Object left, Object right)
{
if(left instanceof BigDecimal || right instanceof BigDecimal)
{
try
{
return (new BigDecimal(scalar(left)).compareTo(new BigDecimal(scalar(right))) == 0);
}
catch(NumberFormatException e)
{
return (false);
}
}
return (asValues(left).stream().anyMatch(asValues(right)::contains));
}
/*******************************************************************************
** Validate six- or eight-digit calendar dates; YY uses the 1950-2049 window.
*******************************************************************************/
public static boolean validDate(String value)
{
if(!value.matches("[0-9]{6}|[0-9]{8}"))
{
return (false);
}
if(value.length() == 6)
{
value = (Integer.parseInt(value.substring(0, 2)) >= 50 ? "19" : "20") + value;
}
try
{
LocalDate.parse(value, DateTimeFormatter.BASIC_ISO_DATE);
return (true);
}
catch(DateTimeException e)
{
return (false);
}
}
/*******************************************************************************
** Locate the first present trigger referenced by a situational condition; values never leave the evaluator.
*******************************************************************************/
public AckContext context(String expression)
{
String condition = expression.trim();
if(condition.startsWith("when("))
{
condition = arguments(condition.substring(5, condition.length() - 1)).getFirst();
}
// Remove literals before recognizing paths so quoted business text cannot become context.
condition = condition.replaceAll("'[^']*'", "");
Matcher matcher = Pattern.compile("([A-Z][A-Z0-9]{1,2}?)([0-9]{2})(?:-([0-9]+))?").matcher(condition);
while(matcher.find())
{
String id = matcher.group(1);
int position = Integer.parseInt(matcher.group(2));
Integer component = matcher.group(3) == null ? null : Integer.valueOf(matcher.group(3));
for(X12Segment segment : segments)
{
if(segment.id().equals(id) && !segment.value(position, 1, component == null ? 1 : component).isEmpty())
{
String loop = loops.entrySet().stream().filter(e -> e.getValue().stream().anyMatch(o -> o.contains(segment))).map(Map.Entry::getKey).findFirst().orElse(null);
return (new AckContext(id, segment.position(), loop, position, component));
}
}
}
return (null);
}
}
@@ -0,0 +1,47 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.guide;
/*******************************************************************************
** Structured validation result; PHI values must never appear in text or excerpts.
*******************************************************************************/
public record ValidationFinding(int segmentPosition, String segmentId, String loopId, Integer elementPosition, Integer componentPosition, int snipLevel, String ak3Code, String ak4Code, String ik3Code, String ik4Code, String severity, String message, String valueExcerpt, AckContext context)
{
/*******************************************************************************
** Construct a finding without situational trigger context.
*******************************************************************************/
public ValidationFinding(int segmentPosition, String segmentId, String loopId, Integer elementPosition, Integer componentPosition, int snipLevel,
String ak3Code, String ak4Code, String ik3Code, String ik4Code, String severity, String message, String valueExcerpt)
{
this(segmentPosition, segmentId, loopId, elementPosition, componentPosition, snipLevel, ak3Code, ak4Code, ik3Code, ik4Code, severity, message, valueExcerpt, null);
}
/*******************************************************************************
** Attach only positional acknowledgement context.
*******************************************************************************/
public ValidationFinding withContext(AckContext context)
{
return (new ValidationFinding(segmentPosition, segmentId, loopId, elementPosition, componentPosition, snipLevel, ak3Code, ak4Code, ik3Code, ik4Code, severity, message, valueExcerpt, context));
}
}
@@ -0,0 +1,166 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.map;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Permission-enforced target writes, including association writes and upsert reads.
*******************************************************************************/
public class InboundApplier
{
/*******************************************************************************
** Run mapping and applying with the supplied least-privilege service identity.
*******************************************************************************/
public static <T> T asService(Supplier<QSession> supplier, com.kingsrook.qbits.edi.support.SystemWrites.Work<T> work) throws Exception
{
if(supplier == null)
{
throw new IllegalStateException("no service session configured");
}
QSession session = supplier.get();
if(session == null)
{
throw new IllegalStateException("no service session configured");
}
QSession previous = QContext.getQSession();
QContext.setQSession(session);
try
{
return (work.run());
}
finally
{
QContext.setQSession(previous);
}
}
/*******************************************************************************
** Add this transaction's partner key to a copy of the configured session.
*******************************************************************************/
public static <T> T asPartner(Supplier<QSession> supplier, Integer partner, com.kingsrook.qbits.edi.support.SystemWrites.Work<T> work) throws Exception
{
return (asService(supplier, () ->
{
QSession prior = QContext.getQSession();
QSession session = prior.clone();
Map<String, List<java.io.Serializable>> keys = new java.util.HashMap<>();
if(prior.getSecurityKeyValues() != null)
{
prior.getSecurityKeyValues().forEach((key, values) -> keys.put(key, new java.util.ArrayList<>(values)));
}
session.setSecurityKeyValues(keys);
if(partner != null)
{
session.withSecurityKeyValue("edi_partner", partner);
}
QContext.setQSession(session);
try
{
return (work.run());
}
finally
{
QContext.setQSession(prior);
}
}));
}
/*******************************************************************************
** The caller owns the per-transaction-set database transaction.
*******************************************************************************/
public QRecord apply(QRecord map, QRecord record) throws Exception
{
String table = map.getValueString("tableName");
var targetTable = QContext.getQInstance().getTable(table);
String backend = com.kingsrook.qbits.edi.EdiRuntime.current().config().getBackendName();
if(targetTable == null || !backend.equals(targetTable.getBackendName())
|| targetTable.getAssociations() != null && targetTable.getAssociations().stream().anyMatch(a -> !backend.equals(QContext.getQInstance().getTable(a.getAssociatedTableName()).getBackendName())))
{
throw new IllegalArgumentException("Inbound targets and associations must share the EDI transactional backend");
}
String keys = map.getValueString("matchKeyFields");
List<QRecord> matches = List.of();
if(keys != null && !keys.isBlank())
{
QQueryFilter filter = new QQueryFilter().withLimit(2);
for(String key : keys.split(","))
{
key = key.trim();
if(record.getValue(key) == null)
{
throw new IllegalArgumentException("Upsert key is missing");
}
filter.withCriteria(key, QCriteriaOperator.EQUALS, record.getValue(key));
}
QueryInput query = new QueryInput(table).withFilter(filter).withTransaction(QContext.getQBackendTransaction());
query.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
matches = new QueryAction().execute(query).getRecords();
}
if(matches.size() > 1)
{
throw new IllegalArgumentException("Upsert key is ambiguous");
}
QRecord result;
if(matches.isEmpty())
{
InsertInput insert = new InsertInput(table).withRecord(record).withTransaction(QContext.getQBackendTransaction());
insert.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
result = new InsertAction().execute(insert).getRecords().getFirst();
}
else
{
String onMatch = map.getValueString("onMatch");
if("SKIP".equals(onMatch))
{
return (matches.getFirst());
}
if(!"UPDATE".equals(onMatch))
{
throw new IllegalArgumentException("Existing target record is not updateable by this map");
}
String primaryKey = QContext.getQInstance().getTable(table).getPrimaryKeyField();
record.withValue(primaryKey, matches.getFirst().getValue(primaryKey));
UpdateInput update = new UpdateInput(table).withRecord(record).withTransaction(QContext.getQBackendTransaction());
update.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
result = new UpdateAction().execute(update).getRecords().getFirst();
}
if(result.getErrors() != null && !result.getErrors().isEmpty())
{
throw new IllegalArgumentException("Target write failed validation");
}
return (result);
}
}
@@ -0,0 +1,185 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.edi.map;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.edi.document.EdiDocument;
import com.kingsrook.qbits.edi.guide.Guide;
import com.kingsrook.qbits.edi.guide.GuideValidator;
import com.kingsrook.qbits.edi.guide.RuleEvaluator;
import com.kingsrook.qbits.edi.storage.EdiRecords;
import com.kingsrook.qbits.edi.x12.X12Segment;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Inbound mapping preserving loop occurrences, including missing optional values.
*******************************************************************************/
public class InboundMapper
{
private final TransformRegistry transforms;
/*******************************************************************************
** Use the host's transform registry.
*******************************************************************************/
public InboundMapper(TransformRegistry transforms)
{
this.transforms = transforms;
}
/*******************************************************************************
** Mapped records and extracted business identifiers; never log this object.
*******************************************************************************/
public record Result(QRecord record, String businessKey, String secondaryKey)
{
}
/*******************************************************************************
** Load ordered rules for a persisted map.
*******************************************************************************/
public Result map(EdiDocument transaction, Guide guide, QRecord map) throws Exception
{
List<QRecord> rules = new ArrayList<>(EdiRecords.query("edi_map_rule", EdiRecords.equal("mapId", map.getValue("id"))));
rules.sort(Comparator.comparing(r -> r.getValueInteger("sequence")));
return (map(transaction, guide, map, rules));
}
/*******************************************************************************
** Map in memory; wildcard rules sharing an association use the same loop scope.
*******************************************************************************/
public Result map(EdiDocument transaction, Guide guide, QRecord map, List<QRecord> rules) throws Exception
{
Map<String, List<List<X12Segment>>> loops = new GuideValidator().loops(transaction, guide);
RuleEvaluator evaluator = new RuleEvaluator(transaction.segments(), loops);
QRecord result = new QRecord();
for(QRecord rule : rules)
{
try
{
String path = rule.getValueString("ediPath");
String target = rule.getValueString("qqqPath");
String[] fields = target.split("\\.");
if(fields.length > 2 || fields.length == 0)
{
throw new IllegalArgumentException("Invalid target path");
}
if(fields.length == 1)
{
apply(rule, evaluator, result, fields[0], path, "EDIFACT".equals(guide.standard()));
}
else
{
if(path == null || !path.contains("[*]/"))
{
throw new IllegalArgumentException("Association path requires a wildcard loop");
}
String anchor = path.substring(0, path.indexOf("[*]/"));
String relative = path.substring(path.indexOf("[*]/") + 4);
List<List<X12Segment>> occurrences = loops.get(anchor);
if(occurrences == null)
{
occurrences = loops.values().stream().flatMap(List::stream).filter(l -> !l.isEmpty() && l.getFirst().id().equals(anchor)).distinct().toList();
}
List<QRecord> children = result.getAssociatedRecords() == null ? null : result.getAssociatedRecords().get(fields[0]);
if(children == null)
{
children = new ArrayList<>();
for(int i = 0; i < occurrences.size(); i++)
{
children.add(new QRecord());
}
result.withAssociatedRecords(fields[0], children);
}
if(children.size() != occurrences.size())
{
throw new IllegalArgumentException("Association rules use inconsistent loops");
}
for(int i = 0; i < occurrences.size(); i++)
{
apply(rule, new RuleEvaluator(occurrences.get(i), loops), children.get(i), fields[1], relative, "EDIFACT".equals(guide.standard()));
}
}
}
catch(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException e)
{
throw (e);
}
catch(Exception e)
{
throw new IllegalArgumentException("Mapping failed at rule " + rule.getValueInteger("sequence") + " (" + rule.getValueString("ediPath") + " -> " + rule.getValueString("qqqPath") + ")");
}
}
return (new Result(result, key(evaluator, map.getValueString("businessKeyPath")), key(evaluator, map.getValueString("secondaryKeyPath"))));
}
/*******************************************************************************
** Conditions and defaults are evaluated within the current loop occurrence.
*******************************************************************************/
private void apply(QRecord rule, RuleEvaluator evaluator, QRecord target, String field, String path, boolean edifact) throws Exception
{
String condition = rule.getValueString("condition");
if(condition != null && !condition.isBlank() && !evaluator.test(condition))
{
return;
}
List<String> values = path == null || path.isBlank() ? List.of() : values(evaluator, path);
if(values.isEmpty() && rule.getValueString("defaultValue") != null)
{
values = List.of(rule.getValueString("defaultValue"));
}
if(edifact && List.of("DECIMAL", "UNIT").contains(java.util.Objects.requireNonNullElse(rule.getValueString("transform"), "")))
{
values = values.stream().map(v -> v.replace(',', '.')).toList();
}
Serializable value = transforms.apply(rule.getValueString("transform"), rule.getValueString("transformArgs"), values);
if(Boolean.TRUE.equals(rule.getValueBoolean("required")) && (value == null || value.toString().isBlank()))
{
throw new IllegalArgumentException("Required mapping input is absent");
}
target.withValue(field, value);
}
/*******************************************************************************
** A path may join several element paths with a plus outside any predicate, in order, so
** a transform such as DATETIME receives a date followed by a time from separate elements.
*******************************************************************************/
private static List<String> values(RuleEvaluator evaluator, String path)
{
List<String> values = new ArrayList<>();
for(String part : path.split("\\+(?![^\\[]*\\])"))
{
values.addAll(evaluator.values(part.trim()));
}
return (values);
}
/*******************************************************************************
** Missing optional business keys remain null.
*******************************************************************************/
private static String key(RuleEvaluator evaluator, String path)
{
List<String> values = path == null || path.isBlank() ? List.of() : evaluator.values(path);
return (values.isEmpty() ? null : values.getFirst());
}
}
Loaded 100 of 310 files, more files were not shown because too many files have changed in this diff. Show more