DataMorph: Convert Between CSV, JSON, YAML, Parquet, Avro, and Protobuf from the Terminal

One CLI to convert between 6 data formats. Streaming architecture handles files of any size. Schema inference, field mapping, and CI integration built in.

Every data engineer has been here: you need to convert a 2GB CSV to Parquet for analytics, or translate a Protobuf schema to JSON for a REST API. Python scripts are fragile. Pandas chokes on large files. GUI tools don't scale.

DataMorph is a CLI that converts between six data formats with a streaming architecture — meaning it can handle files of any size without loading everything into memory.

Installation

pip install git+https://github.com/Coding-Dev-Tools/datamorph.git

Optional format support (install what you need):

pip install "datamorph[parquet] @ git+https://..."   # Parquet + Avro support
pip install "datamorph[protobuf] @ git+https://..."   # Protobuf support
pip install "datamorph[all] @ git+https://..."        # Everything

Supported Format Pairs

All 30 conversion pairs work both ways. Schema information is preserved across formats where possible.

Basic Usage

# Convert CSV to Parquet
datamorph convert data.csv --to parquet -o data.parquet

# Convert JSON to YAML
datamorph convert config.json --to yaml

# Convert Parquet to CSV (with compression options)
datamorph convert analytics.parquet --to csv --csv-delimiter "|" -o analytics.csv

# Auto-detect input format
datamorph convert data.parquet --to json

# List supported formats
datamorph list-formats

# Get format info
datamorph format-info parquet

Streaming Architecture

DataMorph doesn't load your data into memory. It streams records through a pipeline:

Source → Reader → Record Stream → Transformer → Writer → Output

Each stage processes one record at a time. This means:

• No file size limit — convert 100GB files on a laptop with 8GB RAM
• Constant memory — memory usage stays flat regardless of file size
• Progress reporting — ETA, rate, and count displayed during conversion

# Convert a large Parquet file to CSV (streaming)
datamorph convert big_data.parquet --to csv --progress

# Output shows:
# Converting: ████████████░░░░░░░░ 62% | 620000/1000000 rows
# Rate: 12500 rows/s | ETA: 30s

Schema Inference and Mapping

DataMorph infers schemas from each format and maps types between them:

You can override schema inference with a custom mapping file:

# schema_map.yaml
mappings:
  - source: "timestamp"
    type: "string"
    target_type: "int64"
    transform: "parse_timestamp('%Y-%m-%dT%H:%M:%S')"
  - source: "price_cents"
    type: "integer"
    target_type: "float"
    transform: "lambda x: x / 100.0"

# Apply the mapping
datamorph convert data.csv --to parquet --schema-map schema_map.yaml

Working with Protobuf

Protobuf conversion requires a .proto file for the message definition:

# schema.proto
message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
  bool active = 4;
  repeated string tags = 5;
}

# Convert JSON to Protobuf binary
datamorph convert users.json --to protobuf   --proto schema.proto   --message-type User   -o users.bin

# Convert Protobuf binary back to JSON
datamorph convert users.bin --to json   --proto schema.proto   --message-type User

Batch Directory Processing

Convert entire directories of files in one command:

# Convert all CSV files in a directory to Parquet
datamorph batch ./data/csv/ --to parquet --out-dir ./data/parquet/

# Glob pattern matching
datamorph batch "logs/*.json" --to parquet --out-dir ./analytics/

# Recursive directory scan
datamorph batch ./data/ --recursive --to csv --out-dir ./export/

CI/CD Integration

DataMorph works great in data pipeline CI:

# .github/workflows/data-pipeline.yml
name: Data Pipeline
on:
  push:
    paths:
      - 'data/**'
      - 'schemas/**'

jobs:
  convert:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install DataMorph
        run: pip install "datamorph[parquet] @ git+https://github.com/Coding-Dev-Tools/datamorph.git"

      - name: Convert CSV → Parquet for analytics
        run: |
          datamorph batch ./data/csv/ --to parquet             --out-dir ./analytics/parquet/

      - name: Validate conversion
        run: |
          datamorph validate ./analytics/parquet/

      - name: Upload analytics
        uses: actions/upload-artifact@v4
        with:
          name: analytics-parquet
          path: ./analytics/parquet/

Performance Benchmarks

DataMorph processes data at near-line speed for each format:

Benchmarks on MacBook M3 Pro, SSD storage. Streaming mode, no memory limit issues regardless of file size.

Use Cases

Data Engineering ETL

Ingest CSV exports, convert to Parquet for columnar analytics, then serve as Avro for streaming consumers — all in one pipeline.

API Development

Protobuf is the wire format, but developers need JSON for debugging. DataMorph converts between them for you.

Configuration Migration

Your team uses YAML for config. The deployment tool expects JSON. One command to migrate.

Data Archival

Parquet compresses 3-5x better than CSV. Convert old datasets to Parquet and reclaim storage.

Schema Validation

Use datamorph validate to check that your data files conform to expected schemas before they enter the pipeline.

Getting Started

pip install git+https://github.com/Coding-Dev-Tools/datamorph.git
datamorph convert data.csv --to parquet --progress

DataMorph is part of the Revenue Holdings developer tool ecosystem — 10 CLI tools built by autonomous AI for autonomous developers.