PROBLEM
Lack of visibility and understanding of Databricks consumption across the organisaion, leading to spiralling Databricks costs.
SOLUTION
A unified solution providing capability to:
-
visualise Databricks consumption and cost at the most granular level
-
identify cost saving opportunities
-
accurately budget and forecast consumption and spend
-
catch silent "cost-creepers" and anomalies - no more bill shock
OUTCOMES
-
40% reduction in Alinta Energy's Databricks spend
INDUSTRY
Energy and Utilities
LOCATION
Australia
SERVICES
Data Engineering, DevOps
TECHNOLOGIES
Databricks (Azure), Azure Data Factory, Oracle EPM, PowerBI
“Furo’s solution has been a game-changer for us. The transparency it provides, has not only transformed how we manage our Databricks environment but also shifted our team’s entire approach to cloud spending. We now have full confidence in what we’re spending, knowing it is been proactively optimised.”
The Customer
Alinta Energy is one of Australia’s largest energy retailers, generators, investors, and developers. In the last decade they’ve grown from being the largest residential gas retailer in Western Australia to the preferred electricity and gas provider for more than 1.1 million homes and businesses Australia-wide and are determined to make energy better.
The Challenge
As the adoption of cloud platforms like Databricks accelerates, so too does data onboarding and consumption. Without a focus on platform efficiency, this leads to rapidly growing costs due to Databricks' consumption-based pricing model, as Alinta Energy were experiencing first hand.
With no existing off-the-shelf solutions that offer comprehensive cost monitoring, alerting, or insights for Databricks, there was very little visibility and understanding of what the costs were and where they were coming from.
Our objective was to develop a first-of-its-kind, purpose-built solution. This would provide granular visibility into Databricks costs, enabling stakeholders to accurately forecast and budget as well as manage spend more effectively by identifying anomalies, issuing timely notifications, and taking corrective actions.
Solution / Approach
Furo's approach to solving Alinta’s Databricks cost management challenge was a highly collaborative and innovative process. Furo partnered closely with Alinta to identify the key metrics they needed and to guide them on how to model and visualize their data for ease of consumption, creating a unified solution to consolidate this information into a single source for actionable insights. Using Databricks Unity Catalog billing tables, job data, and workspace naming, we developed a unified solution to consolidate this information into a single source for actionable insights.
This solution delivered timely insights into Databricks usage across workspaces/environments, projects, jobs and services. Through iterative development and frequent feedback loops, we quickly refined and improved the solution, making it more agile and responsive to the client’s evolving needs.
Outcomes Delivered
-
40% cost reduction in Databricks usage, realized within the first few months, due to new insights that were previously unattainable without the observability solution.
-
For the first time, stakeholders gained timely access to Databricks Unit (DBU) consumption data, enabling proactive decision-making and more efficient spend management.
Alinta Energy now has unprecedented control over their Databricks spend. They can analyse costs by project, environment, and job name, making changes based on usage patterns and tracking the impact before receiving monthly billing reports. This has unlocked a new level of cost optimisation and cultural transformation across their teams.
"These are outstanding reports, great work."
-- Alinta Energy - Executive Director of Technology
"Setting the bar high for data visualisation. Great work."
-- Alinta Energy - Business Intellience and Operations Manager
"You guys have set the bar pretty high for the IT team now."
-- Alinta Energy - Data Governance Manager