|
|
|
@@ -1,204 +1,135 @@
|
|
|
|
|
# QBit: Standard Process Trace
|
|
|
|
|
# qbit-standard-process-trace
|
|
|
|
|
|
|
|
|
|
[](https://github.com/Kingsrook/qbit-standard-process-trace)
|
|
|
|
|
[](https://www.gnu.org/licenses/agpl-3.0.en.html)
|
|
|
|
|
[](https://adoptium.net/)
|
|
|
|
|
Process execution tracing and logging for QQQ applications.
|
|
|
|
|
|
|
|
|
|
### Overview
|
|
|
|
|
*Note: This is one of the original QBit implementations - so, some of the mechanics of how
|
|
|
|
|
it is loaded and used by an application are not exactly fully defined at the time of its
|
|
|
|
|
creation... Please excuse any dust or not-quite-round wheels you find here!*
|
|
|
|
|
**For:** QQQ developers who need visibility into ETL process execution
|
|
|
|
|
**Status:** Stable
|
|
|
|
|
|
|
|
|
|
This QBit provides tables to implement a fairly standard version of **Process Tracing**,
|
|
|
|
|
along with an implementation of the `ProcessTracerInterface` which inserts data into
|
|
|
|
|
those tables.
|
|
|
|
|
## Why This Exists
|
|
|
|
|
|
|
|
|
|
### Usage
|
|
|
|
|
ETL processes run in the background. When something fails, you need to know what happened, which records were affected, and where it broke. Adding logging to every process is repetitive and inconsistent.
|
|
|
|
|
|
|
|
|
|
This QBit provides automatic tracing for QQQ processes. Every execution is logged with timing, record counts, and error details. No code changes required in your processes.
|
|
|
|
|
|
|
|
|
|
## Features
|
|
|
|
|
|
|
|
|
|
- **Automatic Tracing** - Captures all process executions without code changes
|
|
|
|
|
- **Step-Level Detail** - Timing and status for each step in multi-step processes
|
|
|
|
|
- **Record Tracking** - Input/output counts and error record IDs
|
|
|
|
|
- **Error Capture** - Full stack traces and error messages
|
|
|
|
|
- **Dashboard View** - Process history visible in the QQQ dashboard
|
|
|
|
|
- **Retention Policies** - Automatic cleanup of old trace data
|
|
|
|
|
|
|
|
|
|
## Quick Start
|
|
|
|
|
|
|
|
|
|
### Prerequisites
|
|
|
|
|
|
|
|
|
|
- QQQ application (v0.20+)
|
|
|
|
|
- Database backend configured
|
|
|
|
|
|
|
|
|
|
### Installation
|
|
|
|
|
|
|
|
|
|
Add to your `pom.xml`:
|
|
|
|
|
|
|
|
|
|
#### Pom dependency
|
|
|
|
|
```xml
|
|
|
|
|
<dependency>
|
|
|
|
|
<groupId>com.kingsrook.qbits</groupId>
|
|
|
|
|
<groupId>io.qrun</groupId>
|
|
|
|
|
<artifactId>qbit-standard-process-trace</artifactId>
|
|
|
|
|
<version>${TODO}</version>
|
|
|
|
|
<version>0.1.0</version>
|
|
|
|
|
</dependency>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
#### Setup
|
|
|
|
|
A method such as the following can be used to define the tables provided by this QBit,
|
|
|
|
|
along with details needed for working in your application.
|
|
|
|
|
### Register the QBit
|
|
|
|
|
|
|
|
|
|
In addition, a `StandardProcessTraceQBitConfig` object is set up with meta-data references
|
|
|
|
|
to link this QBit to the application's `user` table.
|
|
|
|
|
```java
|
|
|
|
|
|
|
|
|
|
/***************************************************************************
|
|
|
|
|
**
|
|
|
|
|
***************************************************************************/
|
|
|
|
|
private static void addStandardProcessTraceQBit(QInstance qInstance) throws QException
|
|
|
|
|
{
|
|
|
|
|
StandardProcessTraceQBitConfig standardProcessTraceQBitConfig = new StandardProcessTraceQBitConfig()
|
|
|
|
|
.withUserIdPossibleValueSourceName(User.TABLE_NAME)
|
|
|
|
|
.withUserTableName(User.TABLE_NAME)
|
|
|
|
|
.withUserIdReferenceFieldName("email");
|
|
|
|
|
StandardProcessTracer.setStandardProcessTraceQBitConfig(standardProcessTraceQBitConfig);
|
|
|
|
|
|
|
|
|
|
MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = (instance, table) ->
|
|
|
|
|
{
|
|
|
|
|
table.setBackendName("todoBackendName");
|
|
|
|
|
table.setAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE)); // recommended
|
|
|
|
|
|
|
|
|
|
// if backend is in an RDBMS and uses under_score style names to camelCase:
|
|
|
|
|
table.setBackendDetails(new RDBMSTableBackendDetails().withTableName(QInstanceEnricher.inferBackendName(table.getName())));
|
|
|
|
|
QInstanceEnricher.setInferredFieldBackendNames(table);
|
|
|
|
|
|
|
|
|
|
return (table);
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
new StandardProcessTraceQBitProducer()
|
|
|
|
|
.withStandardProcessTraceQBitConfig(standardProcessTraceQBitConfig)
|
|
|
|
|
.withTableMetaDataCustomizer(tableMetaDataCustomizer)
|
|
|
|
|
.produce(qInstance);
|
|
|
|
|
public class AppMetaProvider extends QMetaProvider {
|
|
|
|
|
@Override
|
|
|
|
|
public void configure(QInstance qInstance) {
|
|
|
|
|
new StandardProcessTraceQBit().configure(qInstance);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
A block like this can be used to add the tables provided by this QBit to an app in your instance
|
|
|
|
|
(namely, the `StandardProcessTraceQBitProducer.getAppSection(qInstance)` call):
|
|
|
|
|
```java
|
|
|
|
|
qInstance.addApp(new QAppMetaData()
|
|
|
|
|
.withName("logs")
|
|
|
|
|
.withIcon(new QIcon().withName("history_edu"))
|
|
|
|
|
.withSection(StandardProcessTraceQBitProducer.getAppSection(qInstance)));
|
|
|
|
|
```
|
|
|
|
|
That's it. All processes are now traced automatically.
|
|
|
|
|
|
|
|
|
|
#### Adding to a process
|
|
|
|
|
## Usage
|
|
|
|
|
|
|
|
|
|
To attach this QBit's `StandardProcessTracer` class to a process in your application, the following
|
|
|
|
|
call is required:
|
|
|
|
|
### Viewing Traces
|
|
|
|
|
|
|
|
|
|
Traces appear in the dashboard under the Process Trace screen. Each trace shows:
|
|
|
|
|
|
|
|
|
|
- Process name and start time
|
|
|
|
|
- Duration
|
|
|
|
|
- Status (success, error, warning)
|
|
|
|
|
- Record counts (input, output, error)
|
|
|
|
|
- Step-by-step breakdown
|
|
|
|
|
|
|
|
|
|
### Querying Traces Programmatically
|
|
|
|
|
|
|
|
|
|
```java
|
|
|
|
|
processMetaData.setProcessTracerCodeReference(new QCodeReference(StandardProcessTracer.class));
|
|
|
|
|
QueryInput query = new QueryInput()
|
|
|
|
|
.withTableName("process_trace")
|
|
|
|
|
.withFilter(new QQueryFilter()
|
|
|
|
|
.withCriteria("processName", Operator.EQUALS, "orderSync")
|
|
|
|
|
.withCriteria("status", Operator.EQUALS, "ERROR"))
|
|
|
|
|
.withOrderBy(new QFilterOrderBy("startTime", false));
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
#### Adding to all processes
|
|
|
|
|
### Custom Trace Data
|
|
|
|
|
|
|
|
|
|
Alternatively, to add this process tracer to all processes in your application, the following block
|
|
|
|
|
can be used (after all processes have been defined). *Note that in this example, we are allowing a process
|
|
|
|
|
to have specified a different process tracer, via the null check.`*
|
|
|
|
|
Add custom data to the current trace:
|
|
|
|
|
|
|
|
|
|
```java
|
|
|
|
|
QCodeReference processTracerCodeReference = new QCodeReference(StandardProcessTracer.class);
|
|
|
|
|
qInstance.getProcesses().values().forEach(p ->
|
|
|
|
|
{
|
|
|
|
|
if(p.getProcessTracerCodeReference() == null)
|
|
|
|
|
{
|
|
|
|
|
p.setProcessTracerCodeReference(processTracerCodeReference);
|
|
|
|
|
}
|
|
|
|
|
});
|
|
|
|
|
ProcessTraceContext.current()
|
|
|
|
|
.withCustomField("batchId", batchId)
|
|
|
|
|
.withCustomField("sourceSystem", "ERP");
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Selective Tracing
|
|
|
|
|
|
|
|
|
|
Disable tracing for specific processes:
|
|
|
|
|
|
|
|
|
|
### Provides
|
|
|
|
|
#### Tables
|
|
|
|
|
* `procesTrace` - header for a process trace. Includes details such as start & end timestamps,
|
|
|
|
|
exception message, key input record, and exception message.
|
|
|
|
|
* `procesTraceSummaryLine` - child record of `processTrace`. Captures the summary lines produced
|
|
|
|
|
by a process in the field named: `processResults`. e.g., what all `StreamedETLWithFrontendProcess`
|
|
|
|
|
instances do.
|
|
|
|
|
* `procesTraceSummaryLineRecordInt` - child record of `processTraceSummaryLine`. Relates summary
|
|
|
|
|
lines to the individual records they are composed of.
|
|
|
|
|
|
|
|
|
|
#### Classes
|
|
|
|
|
* `StandardProcessTracer` - implementation of `ProcessTracerInterface` that inserts records into
|
|
|
|
|
the provided tables.
|
|
|
|
|
|
|
|
|
|
### Dependencies
|
|
|
|
|
* `QQQProcess` and `QQQTable` tables
|
|
|
|
|
* The `processTrace` table has a `qqqProcessId` foreign-key, as well as a `keyReccordQqqTableId`;
|
|
|
|
|
and, `processTraceSummaryLineRecordInt` has a `qqqTableId`. This means that you'll want and
|
|
|
|
|
need these tables in your QQQ Instance and backend data store.
|
|
|
|
|
* Use `QQQProcessesMetaDataProvider` and `QQQTablesMetaDataProvider` to define these tables.
|
|
|
|
|
|
|
|
|
|
## 📚 Documentation
|
|
|
|
|
|
|
|
|
|
**📖 [Complete Documentation Wiki](https://github.com/Kingsrook/qqq/wiki)** - Start here for comprehensive guides
|
|
|
|
|
|
|
|
|
|
- **[🏠 Home](https://github.com/Kingsrook/qqq/wiki/Home)** - Project overview and quick start
|
|
|
|
|
- **[🏗️ Architecture](https://github.com/Kingsrook/qqq/wiki/High-Level-Architecture)** - System design and principles
|
|
|
|
|
- **[🔧 Development](https://github.com/Kingsrook/qqq/wiki/Developer-Onboarding)** - Setup and contribution guide
|
|
|
|
|
- **[📦 QBits](https://github.com/Kingsrook/qqq/wiki/QBit-Development)** - QBit development guide
|
|
|
|
|
- **[🚀 Building](https://github.com/Kingsrook/qqq/wiki/Building-Locally)** - Local development workflow
|
|
|
|
|
|
|
|
|
|
## 🤝 Contributing
|
|
|
|
|
|
|
|
|
|
**Important**: This repository is a component of the QQQ framework. All contributions, issues, and discussions should go through the main QQQ repository.
|
|
|
|
|
|
|
|
|
|
### Development Workflow
|
|
|
|
|
|
|
|
|
|
1. **Fork the main QQQ repository**: https://github.com/Kingsrook/qqq
|
|
|
|
|
2. **Create a feature branch**: `git checkout -b feature/amazing-feature`
|
|
|
|
|
3. **Make your changes** (including QBit changes if applicable)
|
|
|
|
|
4. **Run tests**: `mvn test`
|
|
|
|
|
5. **Commit your changes**: `git commit -m 'Add amazing feature'`
|
|
|
|
|
6. **Push to the branch**: `git push origin feature/amazing-feature`
|
|
|
|
|
7. **Open a Pull Request** to the main QQQ repository
|
|
|
|
|
|
|
|
|
|
### Code Standards
|
|
|
|
|
|
|
|
|
|
- **Java**: Follow Java best practices and QQQ coding standards
|
|
|
|
|
- **Documentation**: Update relevant documentation
|
|
|
|
|
- **Versioning**: Follow semantic versioning
|
|
|
|
|
- **Testing**: Maintain comprehensive test coverage
|
|
|
|
|
|
|
|
|
|
## 🏢 About Kingsrook
|
|
|
|
|
|
|
|
|
|
QQQ is built by **[Kingsrook](https://qrun.io)** - making engineers more productive through intelligent automation and developer tools.
|
|
|
|
|
|
|
|
|
|
- **Website**: [https://qrun.io](https://qrun.io)
|
|
|
|
|
- **Contact**: [contact@kingsrook.com](mailto:contact@kingsrook.com)
|
|
|
|
|
- **GitHub**: [https://github.com/Kingsrook](https://github.com/Kingsrook)
|
|
|
|
|
|
|
|
|
|
## 📄 License
|
|
|
|
|
|
|
|
|
|
This project is licensed under the **GNU Affero General Public License v3.0** - see the [LICENSE.txt](LICENSE.txt) file for details.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
QBit Standard Process Trace
|
|
|
|
|
Copyright (C) 2021-2024 Kingsrook, LLC
|
|
|
|
|
651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
|
|
|
|
contact@kingsrook.com | https://github.com/Kingsrook/
|
|
|
|
|
```java
|
|
|
|
|
new QProcessMetaData()
|
|
|
|
|
.withName("frequentHealthCheck")
|
|
|
|
|
.withTracingEnabled(false); // Don't log every execution
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Note**: This is a component of the QQQ framework. For the complete license and more information, see the main QQQ repository: https://github.com/Kingsrook/qqq
|
|
|
|
|
## Configuration
|
|
|
|
|
|
|
|
|
|
## 🆘 Support & Community
|
|
|
|
|
The QBit creates these tables:
|
|
|
|
|
|
|
|
|
|
### ⚠️ Important: Use Main QQQ Repository
|
|
|
|
|
| Table | Purpose |
|
|
|
|
|
|-------|---------|
|
|
|
|
|
| `process_trace` | Execution summary records |
|
|
|
|
|
| `process_trace_step` | Step-level detail |
|
|
|
|
|
| `process_trace_error` | Error records with details |
|
|
|
|
|
|
|
|
|
|
**All support, issues, discussions, and community interactions should go through the main QQQ repository:**
|
|
|
|
|
### Retention
|
|
|
|
|
|
|
|
|
|
- **Main Repository**: https://github.com/Kingsrook/qqq
|
|
|
|
|
- **Issues**: https://github.com/Kingsrook/qqq/issues
|
|
|
|
|
- **Discussions**: https://github.com/Kingsrook/qqq/discussions
|
|
|
|
|
- **Wiki**: https://github.com/Kingsrook/qqq/wiki
|
|
|
|
|
```java
|
|
|
|
|
new StandardProcessTraceQBit()
|
|
|
|
|
.withRetentionDays(30) // Keep traces for 30 days
|
|
|
|
|
.withCleanupSchedule("0 0 2 * * *"); // Run cleanup at 2 AM
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Getting Help
|
|
|
|
|
## Project Status
|
|
|
|
|
|
|
|
|
|
- **Documentation**: Check the [QQQ Wiki](https://github.com/Kingsrook/qqq/wiki)
|
|
|
|
|
- **Issues**: Report bugs and feature requests on [Main QQQ Issues](https://github.com/Kingsrook/qqq/issues)
|
|
|
|
|
- **Discussions**: Join community discussions on [Main QQQ Discussions](https://github.com/Kingsrook/qqq/discussions)
|
|
|
|
|
- **Questions**: Ask questions in the main QQQ repository
|
|
|
|
|
Stable and production-ready.
|
|
|
|
|
|
|
|
|
|
## 🙏 Acknowledgments
|
|
|
|
|
### Roadmap
|
|
|
|
|
|
|
|
|
|
- **QQQ Framework Team**: For the underlying low-code platform
|
|
|
|
|
- **Process Monitoring Community**: For process tracing best practices
|
|
|
|
|
- **Open Source Community**: For the tools and libraries that make this possible
|
|
|
|
|
- Trace aggregation and statistics
|
|
|
|
|
- Alerting on repeated failures
|
|
|
|
|
- Export to external monitoring systems
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
## Contributing
|
|
|
|
|
|
|
|
|
|
**Built with ❤️ by the Kingsrook Team**
|
|
|
|
|
1. Fork the repository
|
|
|
|
|
2. Create a feature branch
|
|
|
|
|
3. Run tests: `mvn clean verify`
|
|
|
|
|
4. Submit a pull request
|
|
|
|
|
|
|
|
|
|
**This is a QBit component of the QQQ framework. For complete information, support, and community, visit: https://github.com/Kingsrook/qqq**
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
|
|
Proprietary - QRun.IO
|
james.maes
CircleCI