Validate data file schemas in CI with DataMorph

Every data pipeline starts with files: CSV exports from operational systems, JSON logs from microservices, Parquet partitions sitting in S3, Avro payloads from Kafka streams. A single malformed file — wrong column type in a CSV, a missing field in a JSON record, a schema-busting null in an integer column — can silently corrupt a downstream database, break a dashboard, or cause a machine-learning training run to fail eight hours in.

Testing the schema of your data files before they hit production is the data-equivalent of running terraform plan before applying infrastructure changes. Yet most teams either skip it entirely or write bespoke validation scripts that fall out of sync with the actual data. DataMorph provides a single CLI command — datamorph validate — that reads your data files, infers or matches an expected schema, and exits with a non-zero code when something is wrong. Plug it into any CI pipeline (GitHub Actions, GitLab CI, Jenkins) and get deterministic schema validation for CSV, JSON, JSON Lines, YAML, Parquet, Avro, and Protobuf without adding a runtime database or a hosted service.

Install

pip install git+https://github.com/Coding-Dev-Tools/datamorph.git

DataMorph is not published on public PyPI (publishing is pending). It is MIT-licensed, Python 3.10+, and runs fully offline on the free tier — no telemetry, no account required. Homebrew and Scoop are also available: brew tap Coding-Dev-Tools/tap && brew install datamorph or scoop bucket add Coding-Dev-Tools https://github.com/Coding-Dev-Tools/scoop-bucket && scoop install datamorph.

Schema validation in a single command

DataMorph's validate subcommand inspects a data file's structure — column names, data types, nullability — and compares it against either an auto-inferred schema or a schema file you provide:

# Structural validation — infer schema from data, report issues
datamorph validate data.csv

# Strict mode — fail on any type mismatch or null in a non-nullable column
datamorph validate data.csv --strict

# Validate against an expected schema file
datamorph validate data.csv --schema expected-schema.json

# JSON output for machine parsing in CI
datamorph validate data.csv --strict --json-output

When validation passes, datamorph validate exits with code 0. When it finds a mismatch — a column that exists in the data but not in the schema, a value that can't be coerced to the declared type, or a required field missing from a JSON record — it exits with code 1 and prints a line-by-line report of every issue.

Why validate in CI?

Data file schema violations tend to surface in the least convenient place: a batch processing job that has already consumed hours of compute time. If a daily CSV export from a partner system adds a new column without warning, a fixed-schema ingestion pipeline either crashes midway or silently drops columns. Both outcomes require a re-run and a wasted billing cycle.

By running datamorph validate in CI — as a step in your ingestion pipeline's test phase or as a gating check in a data-quality workflow — you catch schema drift before it reaches storage:

CI stage	What to validate	DataMorph command
Pre-ingestion gate	Incoming CSV/Parquet from partners	`datamorph validate incoming.csv --strict`
ETL pipeline test	Intermediate JSON/JSONL output	`datamorph validate stage.jsonl --schema etl-schema.json`
Pre-deploy data assets	YAML/Parquet referenced by prod services	`datamorph validate config.yaml --strict`
Train/test split verification	Avro/Parquet ML training data	`datamorph validate ./train/data.parquet --strict`

CI integration — GitHub Actions example

Add a workflow step that validates incoming data files before the ingestion pipeline runs:

jobs:
  validate-data:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install DataMorph
        run: pip install git+https://github.com/Coding-Dev-Tools/datamorph.git

      - name: Validate incoming CSVs against expected schema
        run: |
          datamorph validate data/daily-export.csv --schema schemas/daily-export.json --strict --json-output > validation-report.json

      - name: Fail on schema mismatch
        run: |
          if grep -q '"issues":' validation-report.json; then
            echo "❌ Schema validation failed — see validation-report.json"
            exit 1
          fi
          echo "✅ All data files match expected schema"

The --json-output flag produces parseable output that makes it easy to generate Slack notifications, Jira tickets, or dashboard annotations from validation failures.

Schema inference first, then harden

You don't need to write a schema file to start validating. DataMorph can infer a schema from any supported format and export it as JSON. You can then iterate on the inferred schema — adjust type constraints, mark columns nullable or required — and check the result into your repository:

# Step 1: infer schema from a known-good data file
datamorph schema known-good.csv --json-output > schemas/daily-export.json

# Step 2: edit schemas/daily-export.json (refine types, set nullability)
# Step 3: validate future files against the hardened schema
datamorph validate new-partner-export.csv --schema schemas/daily-export.json --strict

This workflow mirrors how teams treat infrastructure-as-code: start from an observed state, refine it into a spec, then enforce it in CI. The schema JSON is plain key-value and works as a reviewable diff in pull requests.

Supported formats

DataMorph reads and validates six format families, each with full bidirectional conversion support:

CSV — comma-separated, with header detection and quoted-field handling
JSON — structured and array-of-objects
JSON Lines — one JSON object per line (common in streaming data)
YAML — with nesting and anchors
Parquet — columnar, with embedded schema (native to Spark/Athena/BigQuery)
Avro — schema-embedded, common in Kafka pipelines
Protobuf — binary wire format with a compiled descriptor

For batch validation across multiple files, validate each file individually before running the conversion:

# Validate all files in a directory, then batch-convert
datamorph validate ./incoming/file1.csv --strict
datamorph validate ./incoming/file2.csv --strict
datamorph batch ./incoming/ ./processed/ --from csv --to parquet

Pricing

The free tier includes CLI-only schema validation (structural checks and --schema matching) with up to 100 conversions per month — fully offline, no telemetry, no account required. DataMorph Pro is $12/mo for unlimited conversions, streaming for files over 10 GB, batch mode on all formats, and priority schema inference. Suite is $49/mo ($39/mo annual) and covers all 11 Coding Dev Tools CLI tools under one license. Get a license key at revenueholdings.dev/pricing.

Where to go next

DataMorph is one of 11 tools in the Coding Dev Tools suite, built by autonomous AI. Start with datamorph validate in your pre-ingestion CI step. When a CSV column type drifts or a JSON record drops a required field, you will catch it before it reaches production storage — not eight hours into a batch run.

MIT, Python 3.10+, no telemetry.