KDL Pipeline Definition Language

This document describes the design of a KDL-based language to define and modify Xvc pipelines as text, and how it maps onto the existing pipeline machinery.

Motivation

Today a pipeline is built imperatively, one CLI invocation at a time (xvc pipeline step new, step dependency, step output, ...), or declaratively via xvc pipeline export / import with JSON or YAML. The JSON/YAML schema is a direct serialization of internal structures: it leaks digest fields, is verbose, and doesn't express the graph users think in.

KDL is a small document language designed for exactly this kind of configuration. The KDL pipeline definition is a third, human-first format for export / import that describes a pipeline the way users reason about it:

• Nodes are dependencies — the data the pipeline observes: files, globs, parameters, regex/line fragments, URLs, database queries, generic commands.
• Edges are commands — a step connects the nodes it consumes to the outputs it produces.

Design principles

• One deep module. The whole feature is a single adapter module in xvc-pipeline with a two-function interface: pipeline_schema_from_kdl(&str) -> XvcPipelineSchema and pipeline_schema_to_kdl(&XvcPipelineSchema) -> String. Everything else — DAG construction, step state machine, ECS stores, import/export plumbing — is reused unchanged. The KDL layer knows nothing about execution; the executor knows nothing about KDL.
• Information hiding. Runtime bookkeeping (content digests, metadata, entity ids) never appears in a KDL document. A KDL file contains only what a user would type.
• Define errors out of existence. Node ids default to the dependency's own text (a file node for data.csv is named data.csv unless renamed); one-off dependencies can be declared inline in a step without naming a node; when defaults to by_dependencies; metric formats are inferred from file extensions.
• Somewhat general-purpose. The language maps to XvcPipelineSchema, not to CLI flags, so future dependency types get a node type for free once they are added to XvcDependency.

The graph model

A pipeline is a directed graph.

graph LR
    images["node images<br/>(glob-items data/images/*)"] -->|preprocess| prep["data/train.bin"]
    params["node params<br/>(param params.yaml::train)"] -->|train| model["models/model.pt"]
    prep -->|train| model
    prep -->|train| metrics["metric results/metrics.json"]

• A node declares a dependency: something whose change invalidates the steps that consume it.
• A step is an edge bundle: it consumes a set of nodes (deps), runs a command, and produces outputs (outs).
• Outputs are the step's outgoing endpoints. When a produced file is also consumed by another step's file/glob node, the two steps are ordered implicitly — the runner already derives these edges from outputs (dependencies_to_path). Explicit step-to-step edges are written with after, which maps to the existing step dependency type.

Language

One KDL document defines one pipeline.

version 1

pipeline "train-model" {
    workdir "."

    // ── Nodes: what the pipeline observes ────────────────────────────
    node "images" glob-items="data/images/*"
    node "params" param="params.yaml::train"
    node "top-users" sqlite-file="people.db" sqlite-query="SELECT count(*) FROM people"

    // ── Edges: commands that connect nodes to outputs ────────────────
    step "preprocess" command="python src/preprocess.py" {
        deps "images"
        outs {
            file "data/train.bin"
        }
    }

    step "train" command="python src/train.py" when="by_dependencies" {
        deps "params" file="data/train.bin"
        after "preprocess"
        outs {
            file "models/model.pt"
            metric "results/metrics.json"
            image "plots/loss.png"
        }
    }
}

version

Top-level, optional, defaults to 1. Reserved so the language can evolve without breaking existing pipeline files: a future incompatible change bumps this number and the parser can dispatch on it.

pipeline

Top-level, exactly one. The single argument is the pipeline name (the CLI --pipeline-name flag overrides it, as with JSON/YAML import). Optional child workdir "dir" sets the pipeline run directory relative to the repository root (default: repository root).

node — dependencies

node "<id>" <type>="<spec>" declares a dependency node. <id> is the name steps use to reference it and must be unique in the pipeline; when omitted, the id is the <spec> text itself. Exactly one type property must be present. The <spec> syntax of every type is identical to the corresponding xvc pipeline step dependency CLI flag:

The -items variants track individual items (files, lines) so the step command can use ${XVC_GLOB_ITEMS}-style environment variables; the plain variants only track a digest. This mirrors the CLI exactly.

step — commands

step "<name>" command="<shell command>" [when="..."] declares an edge. when is one of by_dependencies (default), always, never — the same values as step new --when / step update --when. Children:

• deps — arguments are references to declared node ids; properties declare anonymous inline nodes with the same type properties as node. Multiple deps children are allowed and merge. Referencing an undeclared id is an error.
• after "<step>" ... — explicit step dependencies (CLI --step). Kept distinct from deps because it is an edge between commands, not a data node.
• outs — children declare outputs, matching step output: file "path" (--output-file), metric "path" [format="json|csv|tsv"] (--output-metric, format inferred from extension when omitted), and image "path" (--output-image).

Feature parity

Every pipeline-shaping CLI command has a KDL counterpart; read-only and execution commands operate on the imported result and need none:

CLI integration

XvcSchemaSerializationFormat gains a Kdl variant recognized by the .kdl extension. No new subcommands:

$ xvc pipeline export --file pipeline.kdl        # write current pipeline as KDL
$ xvc pipeline import --file pipeline.kdl        # define a new pipeline
$ xvc pipeline import --file pipeline.kdl --overwrite   # modify: edit file, re-import
$ xvc pipeline run                               # unchanged

Export emits the canonical graph form: one named node per distinct dependency (shared dependencies become shared nodes), after for step dependencies, and no digest/metadata noise. export | import round-trips.