Data Engineering Central Podcast

What does it actually take to build AI systems that work in production?

In this episode of Data Engineering Central, I sit down with Maria Vechtomova, AI Engineering Lead, co-founder of CAUCHY, Databricks MVP, and author of MLOps with Databricks.

Maria has spent more than 12 years working across data, machine learning, MLOps, and AI engineering. We talk about how production ML has evolved from the early days of homegrown model platforms to today’s world of LLMs, agents, Databricks, MLflow, and AI-assisted coding.

We dig into why LLM applications can be significantly harder to...

I sat down with Erfan Hesami, founder of Pipeline to Insights, to talk about where data engineering is headed as AI and agents become part of everyday engineering work.

We dig into the changing role of the data engineer, the growing overlap between data and AI engineering, and why strong fundamentals may actually become more important as AI makes writing code cheaper and easier.

We also talk about AI agents inside modern data platforms, governance and security, AI-generated code and technical debt, controlling AI costs, the importance of human judgment, and why communication and business...

What happens when you spend decades building a successful career in tech, finally reach Big Tech, and then suddenly get laid off?

In this episode of the Data Engineering Central Podcast, I sit down with AsianDadEnergy to talk about his journey from writing BASIC on an IBM 386 and working with COBOL mainframes to consulting, becoming a chief architect in Big Tech, and eventually losing his job after seven years.

But the layoff wasn’t the end of the story.

Years earlier, the 2008 financial crisis convinced him and his wife to live below th...

In this episode, I sit down with Shehab Amin, co-founder and CEO of LakeSail, to talk about what happens when you decide to rebuild one of data engineering’s most important technologies, Apache Spark, in Rust … sorta.

We dig into Shehab’s journey from writing C++ as a kid to studying computer science at Berkeley, becoming a founder, and eventually building Sail. We talk about why Spark has remained so sticky, what Rust, Apache Arrow, and DataFusion are changing about data infrastructure, and why rebuilding Spark compatibility turned out to be much harder—and more interesting—than expected.<...

AI can now write SQL, generate Python, review pull requests, and build entire applications with a few well-crafted prompts. Yet despite all of that progress, data engineering still feels oddly resistant to the full AI revolution. Pipelines break. Data models drift. Production data needs validation. Someone still has to close the loop.

In this episode of the Data Engineering Central Podcast, I sat down with Hugo Lu, founder and CEO of Orchestra, to talk about what “agentic data engineering” actually means beyond the buzzwords. Rather than another conversation about AI replacing engineers, we dug into the infrastructure that...

When people think about starting a software company, they usually imagine raising venture capital, hiring engineers, and growing a team as quickly as possible.

Michael Drogalis chose a different path.

After helping build technology in the Kafka ecosystem, founding a startup that was ultimately acquired by Confluent, and leading product for stream processing, he walked away from big tech to see if one person could build a serious B2B software company.

* The result became ShadowTraffic, a product that helps engineering teams generate realistic production traffic for testing, demos, and development.

<...

The creator of Pandas and co-creator of Apache Arrow, Wes McKinney, joins the Data Engineering Central Podcast for an in-depth conversation about how modern data engineering came to exist, where AI is taking software development, and why good engineering still matters more than ever.

We start with Wes’ journey from building GoldenEye fan websites as a teenager to creating Pandas while working at a quantitative hedge fund, and eventually launching Apache Arrow, one of the foundational technologies behind today’s modern data ecosystem. Along the way, we discuss Cloudera, Parquet, DuckDB, DataFusion, Spark, and how the industry evol...

What happens when someone who started programming on a Commodore 64 watches AI reshape the entire data industry?

In this episode of the Data Engineering Central Podcast, I sit down with Dave Langer to explore nearly three decades of experience across software engineering, business intelligence, analytics, data science, and AI.

Dave’s career spans COBOL programming on IBM mainframes, enterprise architecture, Microsoft’s Xbox division, machine learning, startup leadership, authorship, and building one of the largest personal brands in the data space.

We discuss why many of the biggest problems in data haven’t change...

In this episode of the Data Engineering Central Podcast, I sit down with David Jaitillake to explore the future of data engineering, analytics, and AI. David has spent nearly two decades working across data teams, from analyst roles in the early SQL Server days to leading teams, founding startups, serving as VP of AI at Cube, and now co-founding Quarry.

We discuss why semantic layers have suddenly become one of the most important concepts in modern data platforms, how tools like Claude Code are transforming engineering workflows, and why the core problems in data haven’t really ch...

In this episode of the Data Engineering Central Podcast, I sit down with Ethan, a maintainer of delta-rs and an expert in modern lakehouse architecture working in the pharmaceutical industry.

We discuss Ethan’s journey into tech and data engineering, the evolution of open table formats like Delta Lake and Apache Iceberg, and what it actually takes to build scalable enterprise data platforms in highly regulated environments like big pharma.

We also dive into:

* delta-rs and the future of Delta Lake outside Spark

* Lakehouse architecture and open catalogs

* Rust in...

Victor Moreno went from failing out of a top CS program to becoming a senior engineer at AWS, and his story says a lot about what actually matters in software engineering today.

In this conversation, we go deep into the reality behind the AI hype, what makes engineers valuable (it’s not writing more code), and why the future of the field looks very different from what most people think.

We talk about the shift from coding to system thinking, why fundamentals matter more in the age of AI, and how junior engineers will need to...

In this episode of the Data Engineering Central Podcast, I sit down with Yordan Ivanov, Head of Data Engineering at a growing fintech company, to talk through what it actually looks like to build and run real data platforms in production.

Yordan’s story starts like many of mine, early programming, gaming, PHP, Linux servers—but what makes this conversation interesting is how he evolved from a generalist software engineer into a data engineering leader without even realizing it at first.

We spend a lot of time digging into what actually matters in modern data engi...

In this episode of the Data Engineering Central Podcast, I sit down with Jacob Matson, Developer Advocate at MotherDuck, to unpack one of the most interesting shifts happening in data engineering right now.

Jacob didn’t start in tech the way most people expect. He began in accounting, working with Excel and financial systems, before slowly realizing that the real problem he loved solving wasn’t finance, it was data pipelines. That path eventually led him deep into SQL Server, data warehousing, and ultimately to DuckDB, a tool that fundamentally changed how he thought about processing data.

In this episode of the Data Engineering Central Podcast, I sit down with Ben Rogojan to talk about the real story behind data engineering careers, Big Tech, and what’s changing right now.

Ben shares how he went from working in kitchens… to data engineering… to Facebook… and eventually walking away from it all to build his own consulting business.

And yeah, it wasn’t all glamorous.

“I was making the same money as Facebook… and I hated my life.”

We get into the stuff most people don’t talk about:

* What it’s a...

Most companies don’t have a tooling problem. They have a foundation problem.

In this episode, I sit down with Matthew Housley, a famed co-author of Data Engineering Fundamentals and former CTO of Ternary Data, to talk about what actually makes data teams successful and why so many organizations get it wrong despite having modern stacks, cloud platforms, and expensive dashboards.

* Matthew’s path is a little different than most. He started in academia as a mathematics instructor before moving into industry as a data scientist at Overstock.com, and eventually leading data strategy and anal...

AI isn’t just changing how we write code. It’s changing what it even means to build software.

In this episode of the Data Engineering Central Podcast, I sit down with Neil Roberts — a developer who’s been through every major wave of the web, from BASIC on an Atari to modern TypeScript, and now deep into LLMs and agentic workflows.

This is not another surface-level “AI will change everything” conversation. We get into what is actually happening right now, where it works, where it completely breaks, and what developers are getting wrong.

* We talk...

In this episode of the Data Engineering Central Podcast, I sit down with Andreas Kretz to break down what is really happening in the industry right now. We go far beyond surface-level AI hype and talk about how data engineering actually works in the real world, what skills still matter, and where most engineers are wasting time.

Andreas shares his full journey from industrial IoT and working at Bosch to building one of the largest data engineering education platforms in the world, training over 2,000 students and reaching more than 100,000 engineers globally. We get into what production data...

Most data teams think they’re building value. In reality, they’ve become ticket queues.

In this episode, Chris Gambill explains his storied career in tech and data through the years, dealing with data at Fortune 500 company scale, and breaking out on his own.

We cover career growth, what separates senior engineers from true strategic operators, and the biggest mistakes people make early on. We discuss the classic problems that have plagued data teams for decades and why it’s all still a struggle.

Today’s podcast is sponsored by Estuary.

Without...

In this episode, Robert Pack walks through his journey from engineering and simulation work to building large-scale data systems across 900+ plants at BASF.

We break down what those systems actually looked like, including ingestion, modeling, and the realities of batch vs real-time in industrial environments.

We also dive into:

* AI Workflows for Developers

* His work as a committer on Delta Lake

* Where lakehouse architecture works and where it falls short

* The transition into Developer Relations at Databricks

This is a grounded, practical conversation about what...

In this episode of Data Engineering Central, I sit down with Kevin, who spent 13 years working at Apple before walking away at the end of 2025.

* Not to jump to another job.

* Not to start a company.

* But to take a step back from everything.

Kevin shares his full journey—from growing up in the suburbs of Atlanta to building a career at Apple, and ultimately reaching the point where he could walk away financially and mentally.

You can follow along with Kevin below.

We dive deep into wh...

In this episode of Data Engineering Central, I sit down with the founder of DataFlint, Daniel Aronovich, to talk about the realities of working with Apache Spark, distributed data systems, and the future of data engineering.

We start with his early journey into tech—how he first discovered large-scale data systems and the lessons he learned from working with real-world Spark workloads.

* The conversation then turns toward the future of data engineering, particularly the growing role of AI in software development and data infrastructure. We discuss why generic AI coding assistants often struggle with complex di...

In this episode, I sit down with Matt Martin, Staff Engineer, data architect, ETL practitioner, and author of a new book on DuckDB coming soon, to talk about the past, present, and future of data engineering.

Matt has spent decades building and architecting data platforms across technologies such as SQL Server, Oracle, DB2, Hadoop, Redshift, and BigQuery, and now focuses on modern tools such as DuckDB and single-node analytics.

We discuss how the data industry has evolved, what actually makes data platforms succeed, and where tools like DuckDB, Polars, Databricks, and Snowflake fit into the f...

In this episode of Data Engineering Central, I sit down with a veteran Software Engineer John Crickett; with decades of experience in the industry to unpack what really matters in building a long and successful engineering career.

We talk about how he first got into software, the early jobs and tools that shaped his thinking, and the massive technology shifts he’s witnessed across decades of engineering—from early stacks and tools to today’s AI-assisted workflows.

* We also dive into the difference between coding and real-world software engineering, what separates junior, senior, and principal engine...

In this episode of the Data Engineering Central Podcast, I sit down with Yuki (Yuki Kakegawa) to talk about his journey into tech, the tools and platforms he’s worked with, and where he thinks data engineering and AI are headed next.

We cover:

• How Yuki got into tech• Early career lessons and pivots• Tools and technologies he’s worked with over the years• How data engineering has evolved• The impact of AI on software development• What engineers should focus on right now• Advice for those building their careers in data

Yuki shares practical insights...

In this episode of Data Engineering Central, I sit down with Bart Konieczny — data engineer, distributed systems expert, and well-known author in the Data and Spark ecosystem — for a deep technical conversation about modern data engineering.

We cover:

* How Bart got into tech and distributed systems

* His journey through different engineering roles

* Spark internals and why they still matter

* The realities of lakehouse architecture

* Streaming vs batch systems

* AI’s impact on data engineering

* What engineers should focus on in 2026

In a worl...

In this episode of the Data Engineering Central Podcast, I sit down with Maxine Meurer, DevOps engineer, author, and educator behind I Love DevOps, for a wide-ranging conversation about careers, infrastructure, automation, and what it actually means to build systems that last.

This isn’t a buzzword-heavy DevOps chat. It’s a grounded, honest discussion between two engineers about how people really get into tech, how careers evolve over time, and why modern infrastructure is as much about systems thinking and human judgment as it is about tools.

We talk through Maxine’s journey from early...

In this episode, I sit down with industry veteran Robin Moffatt — Sr. Principal Advisor in Streaming Data Technologies (Kafka, etc.) and a longtime voice in the data engineering community, to unpack the journey from old-school data architectures to today’s real-time streaming ecosystems. From early mainframe data processing and COBOL through the rise of Apache Kafka, streaming ETL, and event-driven systems, Robin shares lived experience from across decades of building, scaling, and evolving data platforms.

We dive into:

* 🧠 How the role of software engineering has shifted with the rise of distributed, real-time systems

* 📊 Why event strea...

In this episode of the Data Engineering Central Podcast, I sit down with R. Tyler Croy for a wide-ranging conversation on the present—and future—of modern data platforms.

Tyler is a long-time open-source contributor to projects such as delta-rs. You can watch him on YouTube, read his blog, or work directly with him through his consultancy, Buoyant Data.

Tyler has spent years deep in the open-source data ecosystem, contributing to projects such as Delta Lake and thinking critically about how real-world data systems are built and maintained. This isn’t a hype-driven conversation—it’s a grounded...

In this episode of the Data Engineering Central Podcast, I sit down with Hoyt Emerson, founder of The Full Data Stack and Early Signal, for a wide-ranging conversation on data, analytics, and creating content in the tech world.

We talk candidly about:

* What actually matters in modern data and analytics

* Why so much “data content” misses the mark

* The difference between noise and real signal

* What works (and doesn’t) when building a technical audience

* Writing, consistency, and credibility in the data space

* Why opinions + experi...

In this episode of the Data Engineering Central Podcast, I sit down with Andy Leonard — someone who’s been building systems long before “data engineering” was even a job title.

Andy’s career didn’t start in software at all. It started with physical circuits, literally wiring systems as an electrician, before moving into programming, databases, and eventually decades of hands-on data engineering work.

This conversation isn’t about trends or hype cycles. It’s about how the fundamentals of data work have evolved, what hasn’t changed, and what you only learn after years of building, break...

In this episode of the Data Engineering Central Podcast, I interview a Data OG, someone who’s been around the data space forever, and we talked about all things data, past, present, and future.

I’m joined by Thomas Horton a longtime friend and one of the most well-rounded data professionals I know. Over the course of his career, Tom has worn just about every hat in data: developer, DBA, analyst, and everything in between. He’s lived through the era of on-prem databases, the rise of analytics, and the constant reinvention that defines modern data engineering today...

In this episode, I sit down with Scott Haines — O’Reilly author, Databricks MVP, and veteran of Yahoo, Nike, and Twilio — for a wide-ranging conversation on the real state of modern data engineering.

We dig into open-source ecosystems, Lakehouse architectures, the evolution of Spark, streaming, what’s broken and what’s working in today’s data tooling, and the lessons Scott has learned scaling platforms at some of the biggest companies in the world.

If you care about data engineering, architecture, OSS, or the future of the modern data stack, you’ll love this one.

Thanks...

Hello! A new episode of the Data Engineering Central Podcast is dropping today. We will be covering a few hot topics!

* Cluster Fatigue

* The Death of Open Source

Going to be a great show, come along for the ride!

Thanks for reading Data Engineering Central! This post is public so feel free to share it.

This is a free preview of a paid episode. To hear more, visit dataengineeringcentral.substack.com

Hello! A new episode of the Data Engineering Central Podcast is dropping today, we will be covering a few hot topics!

* Apache Iceberg Catalogs

* new Boring Catalog

* new full Iceberg support from Databricks/Unity Catalog

* Databricks SQL Scripting

* DuckDB coming to a Lake House near you

* Lakebase from Databricks

Going to be a great show, come along for the ride!

Thanks …

Hello, my fair-weathered friends and readers! I am gone on vacation this week with my family, probably at this moment lying in the sand on a beach (Lord willing the creek don’t rise), not thinking of you all.

Anywho, be that as it may, I didn’t want you to miss my pretty face, so here is a video of me ranting about Apache Iceberg, something I’ve had a lot of practice doing and enjoy quite thoroughly.

For all you free-loaders out there, you can get 20% off to celebrate Memorial Day.

https...

This is a free preview of a paid episode. To hear more, visit dataengineeringcentral.substack.com

It’s time for another episode of the Data Engineering Central Podcast. In this episode, we cover …

* Rust-based tool called UV to replace pip and poetry etc

* Apache X-Table and the Future of the Lake House

* How is AI going to affect you?

Thanks for being a consumer of Data Engineering Central; your support means a lot. Please share this podcast with your friend…

It’s time for another episode of the Data Engineering Central Podcast. In this episode, we cover …

* AWS Lambda + DuckDB and Delta Lake (Polars, Daft, etc).

* IAC - Long Live Terraform.

* Databricks Data Quality with DQX.

* Unity Catalog releases for DuckDB and Polars

* Bespoke vs Managed Data Platforms

* Delta Lake vs. Iceberg and UinFORM for a single table.

Thanks for b…

In todays episode of Data Engineering Central Podcast we talk about a few hot topics, AWS S3 Tables, Databricks raising money, are Data Contracts Dead, and the Lake House Storage Format battle!

It's a good one, buckle up!

It’s time for another episode of the Data Engineering Central Podcast. In this episode we cover …

* Apache Airflow vs Databricks Workflows

* End-of-Year Engineering Planning for 2025

* 10 Billion Row Challenge with DuckDB vs Daft vs Polars

* Raw Data Ingestion.

As usual, the full episode is available to paid subscribers, and a shortened version to you free loaders out there, don’t worry, I still love you though.

It’s time for another episode of Data Engineering Central Podcast, our third one! Topics in this episode …

* Should you use DuckDB or Polars?

* Small Engineering Changes (PR Reviews)

* Daft vs Spark on Databricks with Unity Catalog (Delta Lake)

* Primary and Foreign keys in the Lake House

Enjoy!