Duckle Lets Engineers Build Data Pipelines Once and Run Them Anywhere 🔗
The Rust-based tool compiles visual or SQL pipelines to DuckDB SQL for self-hosted execution with full audit trails and AI agent integration.
Duckle is an open-source ETL and ELT platform built in Rust that enables data engineers to author pipelines locally using a visual canvas, Python, or SQL, then deploy the same pipeline file to any infrastructure—on-premises servers, Docker containers, or personal machines—without vendor lock-in or per-row billing. Every pipeline is stored as a single file in Git, ensuring reproducibility and long-term maintainability independent of individual contributors.
At its core, Duckle compiles pipeline graphs into optimized SQL executed on DuckDB, leveraging the engine’s columnar performance to process large workloads efficiently—such as moving 96 million rows from Postgres to Parquet in under 40 seconds on a capable instance. The platform includes 385 pre-built components covering CDC, data quality, reverse ETL, lineage, and dbt integration, with support for importing legacy Talend jobs directly into the canvas.
A standout feature is its MCP (Model Context Protocol) server, which allows AI agents like Claude or Cursor to interact with pipelines as tools, enabling natural language-driven data workflows. Duckle also provides a web console for scheduling, role-based access, and audit trails, making it suitable for team environments.
The catch: While Duckle offers strong local-first flexibility and avoids cloud vendor dependencies, its reliance on DuckDB means it may not suit teams requiring native integration with distributed compute engines like Spark or Flink for massive-scale, multi-terabyte workloads, and its maturity in enterprise-grade security and compliance certifications remains unproven at this stage.
Why this leads today Duckle unifies fragmented ETL workflows in DuckDB, reducing setup time and adding real-time quality, lineage, and AI integration without vendor lock-in.
- Data engineers migrate Talend jobs to self-hosted pipelines
- Teams build auditable ETL workflows versioned in Git
- Developers connect AI agents to data pipelines via MCP server
Source: slothflowlabs/duckle — based on the README and release notes.