Data Transform Refinery (DTR) conceptual overview

At the highest level, DTR is composed of:

• Refineries
• Pipelines
• Nodes
• Transformers
• Rules (and Sub-rules)
• Data Farms

Each layer has a clearly defined responsibility and execution order. Processing is always explicitly sequenced and fully observable.

Sequencing Model
Sequencing is a fundamental design principle in DTR and exists at four levels:
1. Refinery → Node sequence
2. Node → Transformer sequence
3. Transformer → Rule sequence
4. Rule → Sub-rule sequence
This guarantees:
• Predictable outcomes
• Traceability
• Reproducibility

Execution Transparency
At any point during execution, DTR exposes:
• Input and output Data Farms
• Transformation effects per rule
• Row-level statistics
• Structural changes across nodes
The configured Refinery is the executable specification. No separate technical documentation is required.

Refineries and Pipelines, Nodes, Transformers, Rules (and Sub-rules), Data Farms

Refineries and Pipelines

Refinery
A Refinery is the top-level container.
It defines:
• The complete data preparation logic
• The orchestration of one or more pipelines
• The execution sequence across all components
A Refinery represents a self-contained, executable data refinement system.

Pipelines
A Pipeline is a horizontal sequence of nodes that processes data from ingestion to output.
Pipelines can be used for:
• Main processing flows
• Attribution flows
• Data preparation prior to joining
• Intermediate refinement stages
Multiple pipelines can exist within a single Refinery and can interact through Joiner and Attribution mechanisms...

Nodes

Nodes are the primary structural building blocks within a pipeline.
Ingest Node
• Entry point of any pipeline
• Connects to external data sources
• Maps source fields to the DTR semantic data model
• Produces the initial Data Farm
Supported connection types include files, databases, APIs, and application logs.

Processor Node
• Core processing unit of DTR
• Contains one or more Transformers
• Executes transformers sequentially
• Produces a refined output Data Farm
Processor Nodes are fully configurable and form the majority of transformation logic.

Joiner Node
• Specialized node with a single function
• Joins multiple input Data Farms into one output farm
• Supports conditional inclusion of rows
• Used to consolidate data from multiple pipelines
Technically equivalent to controlled concatenation with filtering logic.

Farm Converter Node
• Converts one Data Farm type into another
• Typical conversions:
o Subtasks → Tasks
o Tasks → Activities
• Performs aggregation and restructuring
• Reduces row counts while preserving semantic meaning

Data Farms

A Data Farm is a temporary, in-memory table passed between nodes.

Characteristics:
• Column-based structure
• Typed according to the DTR semantic model
• Exists only during pipeline execution
• Fully inspectable at every stage

Main Data Farm Types

Timestamped Farms
• Subtask Farm
• Task Farm
• Activity Farm

Current State Farms
• Case Farm
• User Farm
• Master / Reference Farm

Timestamped farms form the basis for process analysis, while current-state farms are typically used for attribution

Transformers

Transformers perform the actual data manipulation within a Processor Node.

Each transformer:
• Targets a specific class of data problems
• Is rule-based
• Executes deterministically and sequentially

Row-based Transformers
Designed for timestamped, event-level data:
• Row Extractor
• Row Filler
• Event Namer
• Event Group Namer
• Group Attributor
• Activity Namer
These transformers operate row-by-row and are optimized for high-volume event data.

Column-based Transformers
Designed for attribute-level manipulation:
• Column Cleaner
• Column Lookup
• Anonymizer
Column transformers operate across entire fields and support compound logic and sequencing.

Rules

Transformers are configured through Rules.
• Rules execute sequentially
• Rows matched by a rule are excluded from subsequent rules
• Each rule produces execution statistics
Some rules support compound sub-rules, allowing complex conditional logic within a single rule while preserving execution order.

Summary

The Data Transform Refinery is a component-based, rule-driven data processing engine built around:
• Explicit pipelines
• Typed data farms
• Deterministic sequencing
• High-performance row and column transformations
Its architecture is designed to support complex, multi-stage data refinement while remaining transparent, inspectable, and reproducible.