What Databricks Cost Explainability Actually Requires

At some point, every team running Databricks at scale gets asked a question they can't answer cleanly. Not just why costs went up — that's one version. But also why did costs drop last month, and is that good news or a sign that something stopped running? Which team is driving the biggest share of spend and is it justified? Did that optimisation actually reduce total cost, or did it just move it somewhere else?

Getting to a clean answer means pulling DBU consumption from one system, cloud cost data from another, and stitching them together — which is harder than it sounds. Correlating them at the job level requires building the join logic from scratch: matching against resource tags that are rarely complete, allocating shared infrastructure costs across workloads, and making attribution decisions that need to hold up when finance asks how the figures were calculated.

Explainability isn't a dashboard feature. It's a structural capability with a specific definition: your cost data can answer any question that matters, at the granularity decisions require, on demand. Here's how to test whether you have it.

The Five Questions Your Cost Data Must Be Able to Answer

In practice, explainability means being able to answer five specific questions about any workload, on demand.

1. What did this job cost in total? 

Not DBUs in isolation. Not cloud infrastructure approximated at the workspace level. The combined spend: Databricks DBU charge correlated with the underlying compute, storage, and networking cost, for a specific workload, for a specific run. Without that combined number, every other cost question you ask is built on incomplete data.

2. Which team, project, or cost centre is responsible for that spend? 

Attribution is what makes cost governance possible. Without a consistent tagging model applied across both cost layers, allocating spend to a business dimension (team, project or department) is an estimation exercise. Chargeback built on estimates doesn’t survive scrutiny - and scrutiny comes from every direction. The team being charged will push back if they can’t see a clear link between their workload and the number on the invoice. Projects get disputes, costs get contested, and what should be routine allocation becomes a friction point nobody has time for.  Attribution only works when it’s defensible to everyone it touches. 

3. How has this workload's cost changed, and what drove the change? 

Cost drift is one of the most common causes of unexplained budget variance. A workload running measurably more expensive this month than last, with no change in business output, is compounding waste. Detecting it requires run-level cost baselines rather than monthly totals, so drift is visible before it accumulates into a number that requires explaining.

4. When we optimised this workload, did combined TCO actually fall?

Take migrating from classic or pro SQL warehouses to serverless. Azure infrastructure costs typically fall — serverless shifts compute management to Databricks, reducing the cloud footprint. But Databricks serverless consumption costs more per DBU. If you're only looking at the Azure bill, the migration looks like a win. If you're looking at both layers together, you might find the DBU increase partially or fully offsets the infrastructure saving. The correct validation isn't whether one bill improved — it's whether total spend across both layers fell. Reporting savings on one side without checking the other is answering an incomplete version of the question.

5. Can you see where costs are heading before they become a problem?

Knowing what something cost last month matters. Knowing it's trending 40% higher this month — before the billing cycle closes — is what gives you time to act. The same applies to reservations: if your DBU commitment is tracking to exhaust two weeks before period end, you need to know now, not when the on-demand rate kicks in. Forecasting isn't a nice-to-have. It's the difference between managing costs and reacting to them.

If your current setup can answer all five with actuals, you have explainability. If it can answer some but not others, you have partial visibility, useful but not sufficient when the question is asked at the wrong moment.

What Most Teams Are Working With Today

Most approaches to Databricks cost reporting fall into one of three categories. Each has a specific limitation that prevents it from meeting the five-question standard.

DBU-only reporting from system tables. Databricks provides billing data through system tables — accurate, accessible, and useful for understanding platform consumption. The limitation is that it covers one layer only. DBU cost in isolation is not a TCO. The cloud infrastructure component, often comparable in size to the DBU cost and sometimes larger, sits in a separate system and is not included.

Proportional allocation models. Some teams estimate the cloud infrastructure share by applying a multiplier to DBU consumption, or by dividing workspace-level infrastructure costs across jobs by usage proportion. The numbers add up and the model is reasonable. But the figures are estimates, not actuals, and estimates don't survive scrutiny when finance needs to validate them or when an optimisation claim needs to be confirmed.

Manual reconciliation notebooks. The most thorough custom approach is to combine DBU consumption from Databricks billing system tables with cloud cost exports, then build the logic needed to correlate the two using relevant identifiers, tags and timestamps. When implemented well, this can produce a detailed view of combined Databricks and cloud costs. The limitation is the engineering and maintenance burden: exports must be configured, differences in refresh timing and corrections handled, and tagging and attribution logic maintained as the environment evolves.

In February 2026, Databricks published a field solution specifically addressing this gap. A Cloud Infra Cost Field Solution for AWS and Azure that ingests, enriches, and joins Databricks and cloud cost data into a unified view. The fact that Databricks has now published guidance on closing this gap confirms it is a structural limitation of the native tooling, not an edge case, and that the engineering effort required to solve it is non-trivial even with that guidance in hand.

What Meeting the Standard Looks Like

Alinta Energy, one of Australia's largest energy retailers and generators, can answer all five of these questions today.

As Alinta’s data platform expanded across workloads, teams and environments, its existing approach to cost visibility reached a limit. Databricks and Azure costs could be viewed separately, but not in a single correlated view that consistently attributed spend to the projects and teams driving it. This made it difficult to understand total workload cost and validate whether optimisation efforts had reduced combined TCO or shifted cost between layers.

Closing that gap is what lumin8 was built to do. Working within Alinta’s Databricks environment, it correlated DBU consumption with underlying Azure infrastructure costs at the cluster, warehouse and job level, then mapped that spend to Alinta’s business dimensions, including project, environment and job name. The outcome was a correlated cost view that gave Alinta clearer workload-level attribution, supported forecasting and helped the team identify and validate optimisation actions. Using that visibility to guide a series of optimisation actions, Alinta reduced its combined annual Azure and Databricks costs by 32%.

Where Does Your Cost Visibility Stop?

Cost explainability is not about having another dashboard. It is about being able to trace costs and changes in spend back to the workloads and business dimensions that caused them – and understand what changed, why it changed and who owns it.

The five questions above provide a practical way to assess that capability. If your current setup cannot answer one or more of them clearly and consistently, that is where your explainability gap sits.

Closing that gap requires correlating Databricks and cloud infrastructure costs, applying defensible attribution and preserving enough detail to explain what changed and why.

lumin8 is a Databricks-native cost observability platform built by Furō. It correlates Databricks DBU consumption and cloud infrastructure costs at the job and query level — running entirely within your own Databricks environment, with no data leaving your infrastructure.

If you'd like to see what explainability looks like against your current environment, book a 30-minute session with the lumin8 team at lumin8.app.