The Future of the Lakehouse: Delta Lake, Rust, and Data Platforms at Scale

In this episode of the Data Engineering Central Podcast, I sit down with Ethan, a maintainer of delta-rs and an expert in modern lakehouse architecture working in the pharmaceutical industry.

We discuss Ethan’s journey into tech and data engineering, the evolution of open table formats like Delta Lake and Apache Iceberg, and what it actually takes to build scalable enterprise data platforms in highly regulated environments like big pharma.

We also dive into:

* delta-rs and the future of Delta Lake outside Spark

* Lakehouse architecture and open catalogs

* Rust in the modern data ecosystem

* Data platform governance and scalability

* Enterprise analytics and infrastructure

* The future of agentic analytics and AI-enabled data systems

* Lessons learned building large-scale data platforms

If you’re interested in modern data engineering, open source infrastructure, lakehouses, or the future of analytics engineering, this is a great conversation.

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe