Lakebase + Lakeflow: the data-engineering stack just consolidated

I keep two completely different reflexes in my head for the same task, and last week they finally collided. The task is ordinary: an app needs a real database — proper transactions, low-latency reads and writes, not an analytics table playing dress-up. For years my hand went to the same place every time. Stand up a Postgres instance, wire it in, move on.

Watch which Postgres I reach for now, though — that's where the schism lives. If the work is 100% outside Databricks, on my own k3s cluster where I make the rules, I still hand-roll it: CloudNativePG, a Terraform module I keep around for exactly this, and every bit of toil that comes with owning your own database. Set the reclaim policy wrong and a stray terraform apply will cheerfully delete the data directory out from under you. I've done it. You learn fast to pin prevent_destroy = true on anything you'd cry over losing.

Then the database moves inside the Databricks ecosystem, and I don't do any of that anymore — I spin up a Lakebase instance instead. Same Postgres engine, none of the babysitting, and, more to the point, none of the seam I used to build between the database and the lakehouse. That's the consolidation this post is about: between Lakeflow and Lakebase, the pieces I used to stitch together by hand now ship as one platform. Let me walk you through what actually converged, and where it still bites.

The stack you used to stitch together

Think back to a "normal" data platform two or three years ago. You probably ran at least four moving parts: an ingestion tool to land raw data, a transformation-and-orchestration layer to shape it, a separate OLTP database to serve your application, and a warehouse to answer analytical questions. Four tools, four bills, four places for a credential to leak.

The failure mode wasn't any one of those tools. It was the glue between them. Every seam needed a sync job, and every sync job drifted. Your app database said one thing, your gold table said another, and you spent your Tuesday morning explaining to a stakeholder why the dashboard didn't match the product. I've debugged that exact mismatch more times than I'd like to admit, and the root cause was almost always the same: two sources of truth that were supposed to agree and quietly didn't.

Databricks spent the last two years collapsing those seams from both ends. Lakeflow came at it from the pipeline side. Lakebase came at it from the operational side. Put them together and the integration tax I described above largely disappears.

Lakeflow folded the pipeline into one surface

I wrote about Lakeflow when it went GA — go back and read that one if you want the full tour. The short version: Lakeflow unified the three things data engineers used to buy separately. Lakeflow Connect handles managed ingestion from your operational sources. Lakeflow Declarative Pipelines (the evolution of Delta Live Tables) handle transformation. Lakeflow Jobs (formerly Workflows) handle orchestration. One surface, ingest to gold.

What that buys you is fewer handoffs. Here's a declarative pipeline that lands rebel-fleet telemetry and rolls it up — notice there's no separate scheduler config, no airflow DAG, no glue:

-- Bronze: raw, append-only, straight off the source
CREATE OR REFRESH STREAMING TABLE fleet_events_raw
AS SELECT * FROM STREAM read_files('/volumes/telemetry/fleet/', format => 'json');

-- Silver: cleaned, typed, deduplicated
CREATE OR REFRESH STREAMING TABLE fleet_events
AS SELECT
     ship_id,
     CAST(event_ts AS TIMESTAMP) AS event_ts,
     status,
     sector
   FROM STREAM fleet_events_raw
   WHERE ship_id IS NOT NULL;

-- Gold: the question the commander actually asks
CREATE OR REFRESH MATERIALIZED VIEW ships_active_by_sector
AS SELECT sector, COUNT(DISTINCT ship_id) AS active_ships
   FROM fleet_events
   WHERE status = 'active'
   GROUP BY sector;

The pipeline declares its dependencies and Lakeflow figures out the order, the incremental refresh, and the backfill. You stopped writing orchestration code; you started declaring intent. That's the right trade — but it's only half the consolidation.

Lakebase put the operational database inside the lakehouse

The other half is the piece that retired that second reflex. Lakebase is a managed Postgres — real OLTP, real low-latency reads and writes — that lives inside the lakehouse platform instead of beside it. It's built on open-source Postgres, so your app talks to it with the same drivers, the same SQL, the same pg tooling you already use. No new query dialect to learn.

The piece I keep coming back to is database branching. You can branch a Lakebase database the way you branch a git repo — spin up an isolated copy for a test run, throw it away when you're done. If you've ever been afraid to run a migration against anything resembling production, sit with that for a second. You can branch the database, run the migration on the branch, point your integration tests at it, and only promote if it's green.

And because it's inside the platform, your operational tables and your analytical tables answer to the same Unity Catalog. One governance model. One place where access lives. The sync job I used to write by reflex — the one keeping a bolted-on Postgres in step with the gold layer — is the thing the architecture now removes.

What consolidation actually changes for your pipeline design

So what do you do differently? A few concrete shifts:

• Stop designing around the seam. If your reference architecture still has an arrow labeled "reverse-ETL back to the app database," question it. When the operational store lives in the lakehouse, the round trip from gold table to serving layer collapses into a read.
• Treat your database like code. Branch it in CI. Run the schema change against a throwaway branch, test against real shapes of data, and promote on green. This is the workflow application engineers have had for years and data engineers mostly haven't.
• Govern once. Put your access policy in Unity Catalog and let it cover both the operational and analytical sides. Stop maintaining two ACL models that have to agree.
• Push transformation into declarations. If you're still hand-writing orchestration, you're maintaining code the platform will now maintain for you. Spend that attention on the logic, not the plumbing.

The through-line: fewer integration points means fewer places to drift, and drift is where your Tuesday mornings go to die.

Where it still bites

I don't do vendor cheerleading, so let me name the gaps, because consolidation is never free.

First, this is more lock-in, full stop. Every seam you remove is a seam you can no longer swap out. When your ingestion, transformation, orchestration, serving database, and warehouse all carry one logo, your leverage at renewal time goes down. That's a real cost — price it in deliberately, the same way you'd price in the integration tax you're escaping.

Second, "managed Postgres" is not "every Postgres." If your app leans on an exotic extension or a very specific version, confirm it's supported before you bet the serving layer on it. Managed always means a curated subset.

Third, consolidation tempts you to put everything on the platform. Resist that. A high-throughput, latency-critical operational workload with its own scaling story may still belong on infrastructure you control. The win is removing the seams that hurt — not collapsing every system into one because you can.

My rule of thumb: consolidate the seams that cause drift, keep the boundaries that buy you real isolation or leverage. The architecture should follow the failure modes, not the marketing slide.

The stack is one thing now — design like it

For most of my career, "data engineering," "the app database," and "analytics" were three different jobs with three different tools and a pile of glue in between. That's the part that consolidated. Lakeflow took the pipeline; Lakebase took the operational store; Unity Catalog already had the governance. The seams I used to build by reflex are now design decisions I have to justify — and most of them I can't.

So go look at your own architecture diagram. Count the arrows that exist only to keep two systems in sync. Those are the ones on the chopping block in 2026. If you've already torn some of them out — or if you've hit a wall where the consolidation didn't deliver — I'd genuinely love to hear how it went. As always, I'm here to help.