Files
james.maesandClaude Sonnet 5.5 af162ed066
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / Scoped workflow [workflow 9cb8c13124a99d18da83cbcd4faa795859ac9e55] (pull_request) Successful in 10s
QRun-IO/ci-policy: native / native-result (pull_request) Successful in 7s
QRun-IO/ci-policy: ci / head-profile (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (pull_request) Skipped
QRun-IO/ci-policy: ci / node-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-9 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-10 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-13 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-14 (pull_request) Skipped
QRun-IO/ci-policy: ci / android-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-business (pull_request) Skipped
QRun-IO/ci-policy: ci / original-mcp (pull_request) Skipped
QRun-IO/ci-policy: ci / original-voyage (pull_request) Skipped
QRun-IO/ci-policy: ci / original-cli (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: ci / shared-pipeline (pull_request) Skipped
QRun-IO/ci-policy: ci / ci-result (pull_request) Waiting to run
maven / build (push) Failing after 3m18s
build / maven (push) Failing after 3m18s
build: pin QQQ BOM to 4.1.0-RC.22, drop Central snapshot profile, update status
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-06 17:19:57 -06:00

156 lines
3.9 KiB
Markdown

# qbit-sftp-data-integration
SFTP file transfer and processing for QQQ applications.
**For:** QQQ developers who need to import data from or export data to SFTP servers
**Status:** Stable
## Why This Exists
Enterprise integrations often rely on file drops. Partners send CSV files to an SFTP server. Nightly jobs export reports to remote locations. Building this means handling connections, parsing files, tracking what's been processed, and managing errors.
This QBit provides SFTP connectivity with file polling, parsing, and processing built in. Configure a connection, define file formats, and let QQQ handle the rest.
## Features
- **SFTP Connectivity** - Connect with password or key-based authentication
- **File Polling** - Scheduled checks for new files
- **Format Parsing** - CSV, fixed-width, and delimited file support
- **Record Mapping** - Map file columns to entity fields
- **Processing Tracking** - Track which files have been processed
- **Error Handling** - Move failed files to error directories
## Quick Start
### Prerequisites
- QQQ application (v4.0.0+)
- Database backend configured
- SFTP server credentials
### Installation
Add to your `pom.xml`:
```xml
<dependency>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-sftp-data-integration</artifactId>
<version>0.3.0</version>
</dependency>
```
### Register the QBit
```java
public class AppMetaProvider extends QMetaProvider {
@Override
public void configure(QInstance qInstance) {
new SftpDataIntegrationQBit().configure(qInstance);
}
}
```
### Configure an SFTP Connection
```java
new QSftpConnectionMetaData()
.withName("partnerSftp")
.withHost("sftp.partner.com")
.withPort(22)
.withUsername("integration")
.withPassword("${SFTP_PASSWORD}"); // From environment
```
## Usage
### Import Files
```java
new QSftpImportMetaData()
.withName("dailyOrderImport")
.withConnection("partnerSftp")
.withRemotePath("/outbound/orders/")
.withFilePattern("orders_*.csv")
.withTargetTable("order")
.withMapping(new QFieldMapping()
.withColumn("order_id", "orderNumber")
.withColumn("customer_email", "customerEmail")
.withColumn("total", "orderTotal"));
```
### Export Files
```java
new QSftpExportMetaData()
.withName("nightlyInventoryExport")
.withConnection("partnerSftp")
.withRemotePath("/inbound/inventory/")
.withFileNamePattern("inventory_{date}.csv")
.withSourceTable("product")
.withFilter(new QQueryFilter()
.withCriteria("stockLevel", Operator.LESS_THAN, 10));
```
### Scheduled Polling
```java
new QSftpImportMetaData()
.withName("hourlyUpdates")
.withSchedule("0 0 * * * *") // Every hour
.withConnection("partnerSftp")
.withRemotePath("/updates/");
```
### Key-Based Authentication
```java
new QSftpConnectionMetaData()
.withName("secureSftp")
.withHost("secure.example.com")
.withUsername("service")
.withPrivateKeyPath("/path/to/id_rsa")
.withPrivateKeyPassphrase("${KEY_PASSPHRASE}");
```
## Configuration
The QBit creates these tables:
| Table | Purpose |
|-------|---------|
| `sftp_connection` | Connection configurations |
| `sftp_file_log` | Processed file history |
| `sftp_error_log` | Failed file details |
### Post-Processing
```java
new QSftpImportMetaData()
.withName("orderImport")
.withPostProcessAction(PostProcessAction.MOVE)
.withProcessedPath("/archive/")
.withErrorPath("/errors/");
```
## Project Status
Release candidate. Builds against QQQ `4.1.0-RC.22`; release candidates (`0.5.0-RC.N`) are published to the Gitea Maven registry at `https://git.kof22.com/api/packages/QRun-IO/maven`.
### Roadmap
- S3 and Azure Blob support
- PGP encryption/decryption
- File validation rules
## Contributing
1. Fork the repository
2. Create a feature branch
3. Run tests: `mvn clean verify`
4. Submit a pull request
## License
Apache-2.0 - See [LICENSE](LICENSE) and [NOTICE](NOTICE).