Pipeline Transformation System

Pipeline Transformation System

New in v3.14.0 — Upgraded in v4.0.0 - Execute complex multi-step file transformations via batch_operations({ pipeline_json }), reducing token consumption by 4x (from ~4000-8000 to ~1000-2000 tokens per workflow).

Overview

The pipeline system allows you to chain multiple file operations together with automatic data flow, safety features, and comprehensive error handling.

Key Benefits

• Token Efficiency: 4x reduction in token usage for multi-step workflows
• Atomic Operations: Automatic backup and rollback on failure
• Risk Assessment: Intelligent analysis of operation impact (LOW/MEDIUM/HIGH/CRITICAL)
• Data Flow: Seamless passing of results between steps via input_from and input_from_all
• Conditional Logic: Skip steps based on prior results (9 condition types)
• Template Variables: Reference prior step results with {{step_id.field}} syntax
• Parallel Execution: Independent steps run concurrently via DAG scheduling
• Safety First: Dry-run mode, validation, and configurable error handling

Quick Start

{
  "name": "refactor-namespace",
  "steps": [
    {
      "id": "find",
      "action": "search",
      "params": {
        "path": "./src",
        "pattern": "oldNamespace"
      }
    },
    {
      "id": "count",
      "action": "count_occurrences",
      "input_from": "find",
      "params": { "pattern": "oldNamespace" }
    },
    {
      "id": "replace",
      "action": "edit",
      "input_from": "find",
      "condition": { "type": "count_gt", "step_ref": "count", "value": 0 },
      "params": {
        "old_text": "oldNamespace",
        "new_text": "newNamespace"
      }
    },
    {
      "id": "verify",
      "action": "count_occurrences",
      "input_from": "find",
      "params": {
        "pattern": "newNamespace"
      }
    }
  ]
}

Supported Actions

1. Search

Find files matching a pattern.

{
  "id": "find-files",
  "action": "search",
  "params": {
    "path": "./src",
    "pattern": "TODO",
    "file_types": [".js", ".ts"]
  }
}

Outputs: files_matched[] - List of matching file paths

---

2. Read Ranges

Read file contents from previous step results.

{
  "id": "read-content",
  "action": "read_ranges",
  "input_from": "find-files"
}

Outputs: content{} - Map of file path → content

---

3. Edit

Simple find/replace operation.

{
  "id": "fix-typo",
  "action": "edit",
  "input_from": "find-files",
  "params": {
    "old_text": "recieve",
    "new_text": "receive"
  }
}

Outputs: edits_applied, counts{}

---

4. Multi Edit

Multiple edits in one operation.

{
  "id": "bulk-replace",
  "action": "multi_edit",
  "input_from": "find-files",
  "params": {
    "edits": [
      {"old_text": "var", "new_text": "let"},
      {"old_text": "function", "new_text": "const"}
    ]
  }
}

Outputs: edits_applied, counts{}

---

5. Count Occurrences

Count pattern occurrences in files.

{
  "id": "count-todos",
  "action": "count_occurrences",
  "input_from": "find-files",
  "params": {
    "pattern": "TODO"
  }
}

Outputs: counts{} - Map of file path → count

---

6. Regex Transform

Advanced regex-based transformations.

{
  "id": "modernize-syntax",
  "action": "regex_transform",
  "input_from": "find-files",
  "params": {
    "patterns": [
      {
        "pattern": "function (\\w+)\\(",
        "replacement": "const $1 = ("
      }
    ]
  }
}

Outputs: edits_applied, counts{}

---

7. Copy

Copy files to a destination.

{
  "id": "backup-files",
  "action": "copy",
  "input_from": "find-files",
  "params": {
    "destination": "./backups"
  }
}

Outputs: files_matched[] - New file paths

---

8. Rename

Rename or move files.

{
  "id": "move-files",
  "action": "rename",
  "input_from": "find-files",
  "params": {
    "destination": "./archived"
  }
}

Outputs: files_matched[] - New file paths

---

9. Delete

Soft-delete files (moved to trash/recycle bin).

{
  "id": "remove-temp",
  "action": "delete",
  "input_from": "find-files"
}

Outputs: files_matched[] - Deleted file paths

---

10. Aggregate (new in v4.1.0)

Combine content and file lists from multiple steps.

{
  "id": "combine",
  "action": "aggregate",
  "input_from_all": ["search-src", "search-tests"]
}

Outputs: files_matched[] - Combined file list, aggregated_content - Merged content

---

11. Diff (new in v4.1.0)

Generate a unified diff between two files.

{
  "id": "compare",
  "action": "diff",
  "params": {
    "file_a": "./src/old.go",
    "file_b": "./src/new.go"
  }
}

Outputs: aggregated_content - Unified diff output, counts{"changes"} - Number of changed sections

---

12. Merge (new in v4.1.0)

Union or intersection of file lists from multiple steps.

{
  "id": "common-files",
  "action": "merge",
  "input_from_all": ["search-a", "search-b"],
  "params": { "mode": "intersection" }
}

Outputs: files_matched[] - Resulting file list. Modes: union (default), intersection

Conditional Steps (new in v4.1.0)

Steps can be conditionally executed based on prior step results:

{
  "id": "fix",
  "action": "edit",
  "input_from": "find",
  "condition": {
    "type": "count_gt",
    "step_ref": "count-step",
    "value": 5
  },
  "params": { "old_text": "TODO", "new_text": "DONE" }
}

Condition Types

Type	Description	Required Fields
has_matches	Step found files	step_ref
no_matches	Step found no files	step_ref
count_gt	Total count > value	step_ref, value
count_lt	Total count < value	step_ref, value
count_eq	Total count = value	step_ref, value
file_exists	File exists on disk	path
file_not_exists	File does not exist	path
step_succeeded	Step completed OK	step_ref
step_failed	Step failed	step_ref

Skipped steps show skipped: true and skip_reason in results. They count as successful (not failures).

Template Variables (new in v4.1.0)

Reference prior step results in params using {{step_id.field}}:

{
  "id": "report",
  "action": "search",
  "params": {
    "path": "./src",
    "pattern": "Found {{count-step.count}} occurrences in {{find.files_count}} files"
  }
}

Available Fields

Field	Description	Example Value
count	Sum of all counts	42
files_count	Number of matched files	5
files	Comma-separated file list	a.go,b.go
risk	Risk level	MEDIUM
edits	Number of edits applied	15

Templates resolve recursively in strings, slices, and nested maps.

Parallel Execution (new in v4.1.0)

Enable parallel execution with "parallel": true:

{
  "name": "parallel-analysis",
  "parallel": true,
  "steps": [
    {"id": "s1", "action": "search", "params": {"path": "./src", "pattern": "TODO"}},
    {"id": "s2", "action": "search", "params": {"path": "./src", "pattern": "FIXME"}},
    {"id": "merge", "action": "merge", "input_from_all": ["s1", "s2"],
     "params": {"mode": "union"}}
  ]
}

How It Works

1. DAG Analysis: Steps are analyzed for dependencies (input_from, input_from_all, condition.step_ref)
2. Level Grouping: Independent steps are grouped into execution levels via topological sort
3. Parallel Dispatch: Steps within a level run concurrently using the worker pool
4. Safety Splitting: Destructive actions (edit, delete, rename) within a level are serialized into sub-levels

In the example above, s1 and s2 run in parallel (no dependencies), then merge runs after both complete.

Multi-Step References (new in v4.1.0)

Use input_from_all to reference multiple prior steps:

{
  "id": "combined",
  "action": "aggregate",
  "input_from_all": ["search-src", "search-tests", "search-docs"]
}

This is required for aggregate and merge actions. All referenced step IDs must exist and precede the current step.

Pipeline Configuration

Top-Level Options

{
  "name": "my-pipeline",           // Required: Pipeline name
  "dry_run": false,                 // Optional: Preview without changes
  "force": false,                   // Optional: Bypass risk warnings
  "stop_on_error": true,            // Optional: Stop on first error (default: true)
  "create_backup": true,            // Optional: Auto-backup (default: true for destructive ops)
  "verbose": false,                 // Optional: Return intermediate data (contents, per-file counts)
  "parallel": false,                // Optional: Enable DAG-based parallel execution (v4.1.0)
  "steps": [...]                    // Required: Pipeline steps
}

Step Structure

{
  "id": "step-id",                  // Required: Unique alphanumeric + - _
  "action": "search",               // Required: One of 12 supported actions
  "input_from": "previous-step",    // Optional: Get files from previous step
  "input_from_all": ["s1", "s2"],   // Optional: Get files from multiple steps (v4.1.0)
  "condition": {                    // Optional: Conditional execution (v4.1.0)
    "type": "count_gt",
    "step_ref": "count-step",
    "value": 5
  },
  "params": {                       // Required: Action-specific parameters
    "param1": "value1"
  }
}

Data Flow & Dependencies

Steps can pass data to subsequent steps via input_from (single step) or input_from_all (multiple steps):

{
  "steps": [
    {"id": "find", "action": "search", ...},
    {"id": "edit", "action": "edit", "input_from": "find", ...},
    {"id": "verify", "action": "count_occurrences", "input_from": "find", ...},
    {"id": "combined", "action": "aggregate", "input_from_all": ["find", "verify"]}
  ]
}

Rules:

• input_from and input_from_all must reference previous steps (backward references only)
• Referenced step must have files_matched[] output
• No forward references or circular dependencies
• condition.step_ref also creates a dependency (used by parallel scheduler)

Safety Features

Automatic Backup

For destructive operations (edit, multi_edit, regex_transform, delete, rename), the pipeline automatically:

1. Pre-scans to identify affected files
2. Creates a batch backup before execution
3. Stores backup ID in result

Restore a backup:

// Via backup tool
backup({ action: "restore", backup_id: "20260213_120530_abc123def" })

Risk Assessment

Pipelines assess risk based on:

• Files affected: 30+ = MEDIUM, 50+ = HIGH, 80+ = CRITICAL
• Total edits: 100+ = MEDIUM, 500+ = HIGH, 1000+ = CRITICAL

High/Critical operations require force: true unless in dry-run mode.

Rollback on Failure

When stop_on_error: true (default):

1. Pipeline stops at first error
2. If backup exists, automatically restores files
3. Result indicates rollback_performed: true

Dry-Run Mode

Preview changes without modifying files:

{
  "name": "preview-changes",
  "dry_run": true,
  "steps": [...]
}

Dry-run results show:

• Which files would be affected
• How many edits would be made
• Risk level assessment
• No actual changes applied

Verbose Mode

By default, pipeline output is compact (one-line summary). Use verbose: true to get intermediate data:

{
  "name": "inspect-code",
  "verbose": true,
  "steps": [
    { "id": "find", "action": "search", "params": { "path": "src/", "pattern": "TODO" } },
    { "id": "read", "action": "read_ranges", "input_from": "find" },
    { "id": "count", "action": "count_occurrences", "input_from": "find", "params": { "pattern": "TODO" } }
  ]
}

Compact output (verbose: false):

OK: 3/3 steps | 5 files | 0 edits

Verbose output (verbose: true):

• Full step-by-step results with durations
• File contents from read_ranges (truncated at 50 lines per file)
• Per-file occurrence counts from count_occurrences
• Complete file lists (even when >5 files)

Use compact for edit workflows (minimize tokens). Use verbose when you need to inspect or analyze results.

Limits & Validation

• Max steps per pipeline: 20
• Max files affected: 100 (configurable, requires force to exceed)
• Step IDs: Alphanumeric, -, _ only
• Forward references: Not allowed
• Circular dependencies: Prevented by validation

Example Workflows

1. Refactoring Workflow

{
  "name": "rename-class",
  "steps": [
    {
      "id": "find",
      "action": "search",
      "params": {"path": "./src", "pattern": "class OldName"}
    },
    {
      "id": "rename-class",
      "action": "edit",
      "input_from": "find",
      "params": {"old_text": "OldName", "new_text": "NewName"}
    },
    {
      "id": "rename-imports",
      "action": "search",
      "params": {"path": "./src", "pattern": "from .* import OldName"}
    },
    {
      "id": "fix-imports",
      "action": "edit",
      "input_from": "rename-imports",
      "params": {"old_text": "OldName", "new_text": "NewName"}
    }
  ]
}

2. Code Quality Workflow

{
  "name": "fix-console-logs",
  "steps": [
    {
      "id": "find-logs",
      "action": "search",
      "params": {"path": "./src", "pattern": "console.log"}
    },
    {
      "id": "count-before",
      "action": "count_occurrences",
      "input_from": "find-logs",
      "params": {"pattern": "console.log"}
    },
    {
      "id": "remove-logs",
      "action": "regex_transform",
      "input_from": "find-logs",
      "params": {
        "patterns": [
          {"pattern": "\\s*console\\.log\\([^)]*\\);?\\n?", "replacement": ""}
        ]
      }
    }
  ]
}

3. Migration Workflow

{
  "name": "migrate-components",
  "steps": [
    {
      "id": "find-old",
      "action": "search",
      "params": {"path": "./old-components", "pattern": ".*", "file_types": [".jsx"]}
    },
    {
      "id": "copy-to-new",
      "action": "copy",
      "input_from": "find-old",
      "params": {"destination": "./new-components"}
    },
    {
      "id": "modernize-syntax",
      "action": "multi_edit",
      "input_from": "copy-to-new",
      "params": {
        "edits": [
          {"old_text": "React.Component", "new_text": "React.FC"},
          {"old_text": "componentDidMount", "new_text": "useEffect"}
        ]
      }
    }
  ]
}

Performance

Token Savings

Before (separate calls):

• search: ~500 tokens
• read_file (with start_line/end_line) x10: ~3000 tokens
• edit_file x10: ~4000 tokens
• Total: ~7500 tokens

After (pipeline):

• batch_operations: ~1500 tokens
• Savings: 5x reduction

Execution Speed

• Single network round-trip vs. 20+ separate calls
• Automatic parallelization of independent operations with parallel: true
• DAG-based scheduling runs independent steps concurrently
• In-memory data passing between steps

Output Formats

Compact Mode

OK: 4/4 steps | 12 files | 45 edits | medium risk

Verbose Mode

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📋 Pipeline: refactor-todos
✅ Success: true
📊 Steps: 4/4 completed
⏱️  Duration: 1.2s
💾 Backup: 20250213_120530_abc123def

📝 Step Results:

1. find [search] ✅
   Duration: 234ms
   Files: 12 matched

2. replace [edit] ✅
   Duration: 567ms
   Edits: 45 replacements
   ⚠️  Risk: MEDIUM

3. verify [count_occurrences] ✅
   Duration: 123ms
   Counts: 45 total occurrences

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📁 Files affected: 12
✏️  Total edits: 45
⚠️  Overall risk: MEDIUM

Error Handling

Stop on Error (default)

{"stop_on_error": true}

• Pipeline stops at first error
• Previous steps remain applied (if no rollback)
• Returns partial results

Continue on Error

{"stop_on_error": false}

• Pipeline continues despite errors
• Marks individual steps as failed
• Returns all results (success + failures)

Best Practices

1. Start with dry-run - Always preview changes first
2. Small batches - Keep pipelines under 10 steps for clarity
3. Descriptive IDs - Use clear step names: find-todos, fix-typos
4. Validate data flow - Ensure input_from references make sense
5. Test incrementally - Build pipelines step-by-step
6. Use backups - Don’t disable create_backup for destructive ops
7. Monitor risk - Pay attention to risk level warnings

Troubleshooting

”forward reference” Error

Step references a step that comes later in the pipeline. Fix: Reorder steps so input_from points backward.

”too many files affected” Error

Pipeline would affect >100 files without force: true. Fix: Add "force": true or reduce scope.

”operation blocked due to HIGH risk”

Destructive operation affects many files. Fix: Use "dry_run": true to preview, or add "force": true.

Rollback Failed

Backup couldn’t be restored after failure. Fix: Manually restore using backup({ action:"restore", backup_id }) with the backup ID.

API Reference

See Pipeline Tools API for complete parameter documentation.

See Also

• Batch Operations - Multi-file operations
• Backup & Recovery - Backup system details
• Safe Editing - Editing best practices