Schema Evolution Without Downtime

Hey, welcome back to Deltaplex Live: The Real-time Enterprise Show. I'm your host Alex, and... yeah, if you've been around for a while, you know what we're about — real talk about real-time data, the messy stuff that happens in production, and the people building the systems that have to handle it. And today... today we're going to talk about something that I think every data engineer has a war story about. Schema changes. We've got a great guest back on the call — Ethan, Principal Data Engineer over at Deltaplex. Ethan, welcome back man, how are you?

Alex, good to be back, yeah. Doing well, doing well. Uh... we were just catching up before we hit record and I feel like it's been too long since we did one of these.

Right? And we always say we're gonna do it more regularly and then—

And then... something breaks in production and—

—and here we are, three months later. Okay, so look. Let's get into it. Schema evolution. Schema changes. This is... I mean, this is one of those things where— and I'll be honest, Ethan, when I was putting together the talking points for today's episode, I kind of thought, is this... is this unsexy? Like, is this the boring plumbing stuff? But then I actually started reading through the technical brief you all put out — Technical Brief 03 — and I was like, oh no. This is actually... this is where pipelines go to die.

Yeah. Yeah, I mean... it's one of those topics that sounds dry on the surface, right? Schema evolution. But the moment you've been on call at 2am because someone renamed a column in a source database and four dashboards broke and your CDC pipeline is just... throwing errors... it stops being dry real fast.

Right, right. Okay, so let's start from the top for folks who maybe aren't super deep in the weeds here. When we talk about a schema change — what does that actually mean in practice? What's happening?

Yeah, so... okay. Think about it this way. Every table in a database has a structure, right? The schema. The columns, the data types, the keys. And in a perfect world, that structure never changes once your pipeline is reading from it. But — we don't live in a perfect world.

Definitely not.

Right. So what happens is, the engineering team on the source side — maybe it's a product team, maybe it's a third-party vendor — they change something. They add a column, they rename a field, they change a data type, they drop a column entirely. And from their perspective, it might be a totally routine change. Like a five-minute migration. But on the data pipeline side, it can propagate everywhere. Your warehouse, your dashboards, your ML features, your downstream jobs... all of it can be affected.

And that's— yeah, that's what really stood out to me in the brief. It's like... the change itself isn't even necessarily the problem. It's the absence of a process around it.

Exactly. That's the key insight, right? It's not that schema changes are inherently catastrophic. It's that most pipelines aren't built to handle them gracefully. There's no schema change governance layer. So when it happens, you're in firefighting mode.

Okay, so let's walk through the different types of changes, because they're not all created equal. Some are more dangerous than others.

Yeah, for sure. So if I had to stack rank them by risk — adding a column is probably the lowest. You add a nullable column to a source table, existing records still work, downstream consumers aren't forced to do anything with the new field. It's pretty benign.

It's additive. You're not taking anything away.

Exactly. Now, modifying a data type — that's where it starts to get spicy. You widen a type, like int to bigint, or varchar 50 to varchar 255 — that might be fine if the target supports it. But if you're narrowing — going from larger precision to smaller — now you've got truncation risk, write failures...

And the pipeline doesn't know until it tries to write and just... explodes.

Right! And that's the worst case — when the first signal is a write failure. Ideally you want the system to classify the risk before it gets to that point.

Yeah. Okay, what about renames? Because I feel like renaming a column sounds so innocent—

Oh man. Column renames are... they're sneaky. Because at the source, it's a three-second migration. And then on your side, you've got dashboards referencing the old column name, SQL jobs, API outputs, maybe ML models that are pulling that feature... All of them are still expecting the old name. And if your pipeline passes through the rename without any aliasing or compatibility layer, everything downstream just... breaks.

And it's not just the breaking, it's the discovery. Like, figuring out what even depended on that field.

Yeah, lineage. Data lineage is huge here. If you don't have visibility into what's consuming a given field, a rename can turn into a days-long investigation.

Okay, and then on the other end of the spectrum — dropping a column. That's the big one, right?

Dropping is the most dangerous. Yeah. If someone drops a column and you don't catch it, you can end up with nulls in a field that downstream systems treat as required. Or jobs fail outright. The brief lays out a sensible staged approach — discover your consumers first, issue a deprecation notice, give people a migration window, then remove the field only after everyone's had time to move. It sounds obvious when you say it out loud, but...

It never happens that way in practice.

It basically never happens that way.

Okay, and then primary key changes. This one was interesting to me — I think of primary keys as pretty stable. Why would someone change those?

It happens more than you'd think. Composite keys getting collapsed into a surrogate key, UUID migrations, someone changes what constitutes a unique record. And for CDC systems — change data capture — this is really serious because the entire merge logic depends on the key. If the key changes and you haven't updated your pipeline logic, you can get duplicate records, missed deletes, broken upserts...

Yikes. Okay. So we've got all these failure modes. What do you actually do about it? What's the playbook?

So... the brief talks about four core strategies, and the way to think about it is — there's no single right answer for every change. You need to be able to apply different policies based on the risk level and the context.

Okay. Walk me through them.

Alright. First one — automatic sync. The pipeline detects a compatible change, checks that the target can support it, updates the target schema automatically, and just keeps going. Owners get notified, but nothing stops. Works best for additive changes and environments that prioritize speed.

So it's like... self-healing, almost.

Kind of, yeah. For the low-risk stuff. The key word is compatible — you're not blindly syncing everything, you're running a compatibility check first.

Okay. What's number two?

Ignore new fields until approved. The pipeline detects the new field but doesn't propagate it. Keeps delivering based on the last known good schema. Useful when a new source field might contain sensitive data, or it's an internal operational field that has no business being in your warehouse.

That makes sense. Governance gate on the way in.

Exactly. And then the third one is alert and pause. For the more dangerous changes — dropped columns, narrowed types, key changes — the pipeline pauses delivery and routes the change for human review. A controlled, short pause is way better than letting a breaking change propagate silently.

Right. Because then you're dealing with corrupted data or broken consumers, which is—

Which is a way bigger mess to clean up. Yeah.

Okay, and number four?

Custom transformation and compatibility rules. The most flexible option. For renames or restructuring — maintain aliases, write to both the old field and the new field simultaneously, or route data into versioned targets. Downstream consumers can keep using the old schema while they migrate to the new one at their own pace.

That dual-write thing — it seems powerful but also kind of scary. You're writing to two things at once.

Yeah, it's a trade-off. The benefit is you're not forcing everyone to migrate at the same time. You give downstream teams a transition window. But you've got to manage the complexity of running two representations in parallel, and you need a clear end date — otherwise the dual-write becomes permanent and now you've got tech debt.

Right, it becomes the new normal.

Exactly. So you need the governance around it. This dual-write window closes on this date. After that, consumers that haven't migrated are on their own.

Makes sense. And schema versioning is related to this, right? Publishing customers v1, customers v2...

Yeah. Schema versioning is a clean way to handle this for important shared datasets. Legacy consumers keep pointing at v1, new consumers adopt v2, at some point you deprecate v1 once everyone's moved. You can also layer in view-based abstraction — put a compatibility view on top of the underlying table so consumers don't even see the schema change happening beneath them.

That's elegant.

It's basically buying yourself time. Decoupling the rate of change at the source from the rate of migration downstream. In a large organization where every team moves at a different pace, that's really valuable.

Yeah. Okay, so let's zoom out. Because one of the things that comes through in the brief is that schema evolution isn't just a technical problem. It's an operational model question. Who owns this? How does it get managed?

Yeah, totally. And this is where a lot of teams get stuck. They solve the technical piece — okay, we'll pause on drops, we'll alias renames — but they don't build the operating model around it. Changes still fall through the cracks because nobody got notified, or someone gets notified but doesn't know what to do with the information.

What does a good operating model look like in your view?

A few things. First — detection and classification have to be automatic. You can't rely on humans to catch schema changes manually. By the time someone notices, the damage is often done. The platform needs to be watching DDL events, type changes, nullability changes, table renames, key changes... continuously.

And mapping that to risk levels automatically.

Right. And then second — policy by risk level. Not everything gets the same treatment. Safe, additive changes can be automated. Risky changes get paused for review. That separation is important.

Because otherwise you're either too aggressive — auto-syncing things you shouldn't — or too conservative and pausing on every tiny change and slowing everyone down.

Exactly. And then the third piece is notifications. Routing changes to the right owners. Not just blasting everyone with an alert that no one reads, but — this field changed, here's which downstream consumers are affected, here's the classification, here's the recommended action. Actionable information.

And data lineage is what makes that possible. You have to know what's consuming what.

You have to know the lineage. Yeah. And then the fourth piece is consumer protection. Giving downstream teams the tools they need — aliases, versioned targets, compatibility views, staged deprecation windows — so they're not left holding the bag every time the source changes.

Right. The cost of source changes doesn't fall entirely on the consumers.

The pain gets distributed appropriately instead of cascading downstream.

Okay. Real talk question. How mature is the industry actually on this? Is this a solved problem at most organizations?

Honestly? Most teams are somewhere between we handle it manually and pray and we've got some basic detection. Very few have the full operating model — detection, classification, policy, notifications, consumer protection — all working together. And the teams that do tend to be the ones that have been burned badly enough to invest in it.

The classic production incident as the forcing function.

Nothing like a sev-1 at 3am to get budget for schema governance tooling.

Alright. For a team listening to this and thinking — okay, we need to get better here — what's the first step?

Step one is visibility. Before you can do anything else, you need to know when schema changes are happening — proactively, not after they break something. Even basic detection and alerting buys you a lot.

Just get out of the dark.

Just get out of the dark, yeah. Step two — get your classification right. Not all changes are equal. Build a risk framework, even a simple one. Additive changes here, type changes here, drops and key changes here. And step three is actually talking to your downstream consumers. Do a lineage audit. Find out what's reading from your most critical tables. You can't protect consumers you don't know about.

That's practical. That's actionable. I appreciate that.

And honestly, the good news is tooling for this is getting better. Platforms like Deltaplex are building this into the pipeline lifecycle — the detection, the policy engine, the notifications, the consumer protection — as part of how the pipeline works, not a separate system you have to maintain.

Schema evolution as a first-class concern rather than an afterthought.

That's exactly the framing. Yeah.

Alright, Ethan — this has been great. Genuinely. I feel like I could keep going on this for another hour, but we should wrap.

Yeah, we've been known to go long.

We have. For folks who want to go deeper — the technical brief we've been referencing is Deltaplex Technical Brief 03: Schema Evolution Without Downtime. Full risk matrix, strategy framework, implementation checklists — all of it. We'll link it in the show notes. Ethan, as always, thanks for coming on. Great conversation.

Thanks for having me back, Alex. Always a good time.

And thank you all for listening to Deltaplex Live: The Real-time Enterprise Show. If this episode was useful, share it with someone on your team who's had a schema incident. They'll know why. We'll see you next time.