Building a Correct-by-Design Lakehouse. Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents

TL;DR

This paper introduces Bauplan, a code-first lakehouse system that enhances correctness through data contracts, versioning, and transactional pipelines, reducing runtime errors and divergence between development and production.

Contribution

Bauplan integrates typed data contracts, Git-like versioning, and transactional execution into lakehouses, providing a unified model for reliable analytics and AI workflows.

Findings

Validated key correctness intuitions with Alloy model

Identified subtle issues in transactional branch visibility

Demonstrated early system design benefits

Abstract

Lakehouses are now the default substrate for analytics and AI, but they remain fragile under concurrent, untrusted change: schema mismatches often surface only at runtime, development and production easily diverge, and multi-table pipelines can expose partial results after failure. We present Bauplan, a code-first lakehouse that aims to eliminate a broad class of these failures by construction. Bauplan builds on a storage substrate that already provides atomic single-table snapshot evolution, and adds three pipeline-level correctness mechanisms: typed table contracts to make transformation boundaries checkable, Git-like data versioning to support reproducible collaboration and review, and transactional runs that guarantee atomic publication of an entire pipeline execution. We describe the system design, show how these abstractions fit together into a unified programming model for humans and agents, and report early results from a lightweight Alloy model that both validates key intuitions and exposes subtle counterexamples around transactional branch visibility. Our experience suggests that correctness in the lakehouse is best addressed not by patching failures after the fact, but by restricting the programming model so that many illegal states become unrepresentable.