Transform Ops

What it does: A Transform step applies a sequence of deterministic data operations to the outputs of one or more upstream steps. No LLM call is made.

---

How Transforms Work

A transform step collects its inputs, then applies each op in ops: left-to-right. Each op receives the output of the previous op, so the sequence acts as a pipeline. Ops that take an expr field use JMESPath expressions.

- id: process
  transform:
    inputs:
      - from: "{{ step.fetch.items }}"  # reference a completed step's output
    ops:
      - filter:
          expr: "score > `0.5`"
      - sort:
          field: "score"
          order: "desc"

When multiple inputs are listed, they are deep-merged into a single object (right-to-left, right wins) before the first op runs.

The transform step emits {"result": <finalValue>}. Downstream steps reference it as step.<id>.result.

---

The ops: Array

Each element of ops: is a map with exactly one key — the op name — whose value is a sub-map of op-specific fields (or an array for merge).

ops:
  - <op_name>:
      <field>: <value>

---

Op Reference

merge

Deep-merges the outputs of the listed steps into the current data object. Map keys from later steps override earlier ones; arrays are concatenated.

Sub-fields: an array of step IDs.

ops:
  - merge:
      - step_a
      - step_b

Sub-field	Type	Description
(value)	array of strings	Ordered list of step IDs to merge.

---

filter

Keeps only array items where the JMESPath expression evaluates to a truthy value. Non-array input is coerced to a single-element array first.

ops:
  - filter:
      expr: "confidence > `0.5`"

Sub-field	Type	Description
expr	string	JMESPath expression. Items where it is truthy are kept.

---

sort

Sorts an array by a field value. Numeric fields are compared numerically; all others are compared as strings. nil values sort last.

ops:
  - sort:
      field: "score"
      order: "desc"

Sub-field	Type	Description
field	string	Field name (or JMESPath) to sort by.
order	string	"asc" (default) or "desc".

---

map

Projects each array item through a JMESPath expression, replacing it with the expression's result. Non-array input is coerced to a single-element array first.

ops:
  - map:
      expr: "{ title: title, score: confidence }"

Sub-field	Type	Description
expr	string	JMESPath expression evaluated against each item.

---

flatten

Flattens one level of nesting: each element that is itself an array is spread into the parent array. Non-array elements are kept as-is.

Sub-fields: none.

ops:
  - flatten: {}

Input [[1, 2], [3], 4] → output [1, 2, 3, 4].

---

deduplicate

Removes duplicate array items, keeping the first occurrence. Two items are considered duplicates if the given field has the same string representation.

ops:
  - deduplicate:
      field: "url"

Sub-field	Type	Description
field	string	Field name (or JMESPath) used as the deduplication key.

---

Complete Pipeline Example

A transform step that takes fanout results, filters out failures and low-confidence items, removes duplicates by URL, and sorts by score descending.

pipeline:
  - id: score_urls
    agent: "./agents/scorer.agent.yaml"
    params:
      url: "{{ item }}"
    for_each:
      from: "step.fetch.urls"
      concurrency: 10
      max_failures: -1

  - id: clean_results
    transform:
      inputs:
        - from: score_urls
      ops:
        # Unwrap the fanout results array
        - map:
            expr: "results"

        # Drop failed iterations and low-confidence scores
        - filter:
            expr: "score > `0.6` && !ktsu_error"

        # One result per URL
        - deduplicate:
            field: "url"

        # Highest confidence first
        - sort:
            field: "score"
            order: "desc"
    depends_on: [score_urls]

clean_results emits {"result": [...]} — a deduplicated, sorted array of passing scored objects.

---

See also Pipeline Primitives for the full list of step types, and Fanout for running an agent step over an array.

Revised April 2026