How TfNSW Lowered Their Data Lake Costs by 22% with CloudMonitor
Transport for NSW - Government Enterprise Customer Success Story
Transport for NSW (TfNSW) - Australia Government Agency
Transport for NSW (TfNSW) manages one of the largest fleets of vehicles in Australia, including Buses, Ferries, Light Rail, Trains and Metro. The real-time data generated by these vehicles provides valuable analytical opportunities to improve transport services by measuring service performance and optimising routes, however, the sheer volume of data presents data management challenges.
Data-Driven introduced CloudMonitor to TfNSW in an effort to improve cost visibility and enable chargeback to the various project teams within the program.
- Industry: Government
- Headquarters: Sydney, NSW, Australia
- Company Size: 10 000+
- Website: https://www.transport.nsw.gov.au/
Problem Statement
Historical operational data too large and costly to store efficiently and analyze
Historical GTFS data has always been too large and costly to store efficiently and analyse. Each TfNSW vehicle sends its location every 10 seconds which results in a huge stockpile of data containing valuable insights (over 5TB per week and growing).
In the past, this data was not stored which made insights from past vehicle trips almost impossible to obtain and ruled out the ability to predict trip delays or optimise trip routes. It also made it difficult to report on and analyse service performance and reduced the ability to improve customer services.
This scenario underscores the essential role of FinOps in balancing the operational need for continuous data with the financial imperatives of cost and efficiency, framing it not merely as a data storage issue but as an integrated challenge that intersects with several key domains within the FinOps practice.
Key FinOps Challenges
- Immature tagging policy which meant it was hard to allocate costs.
- The business had no tooling with which to view platform spend.
- IT would need to be involved to calculate and distribute cost reports.
- Many zombied and orphaned resources in DEV and UAT with no known owners
The FinOps Solution
CloudMonitor focused on "low-hanging fruit" and identified zombie resources and orphaned services.
The consultant conducted audits using CloudMonitor to find and remove orphaned resources — those resources that were no longer in use but still incurring costs. By systematically identifying and deleting these resources, they prevented unnecessary expenses.
It recommended right-sizing virtual machines and services based on actual demand. Additionally, it advised on the purchase of reserved instances, which offered notable discounts compared to pay-as-you-go options. This was done for Storage and Databricks compute.
The Program Manager was given the CloudMonitor Reporting suite to be able to subscribe to monthly Budget versus Forecasting reports with real-time cost anomaly alerting via the CloudMonitor Teams bot.
Results for the Business
A 22% Reduction in the Data Lake Azure Spend
TfNSW achieved a significant transformation by integrating FinOps principles into their data management practice. The impact of this alignment was profound, resulting in a 22% reduction in Data Lake costs, enabling more efficient data storage and analysis while improving transport customer services.
Right-sized Resources
Recommendations for over-provisioned resources ensure the system cost is minimized.
Cost Anomaly Detection
Real-Time cost anomaly detection catches cost spikes quicker, which means less bill shock.
Improved Cost Governance
The CloudMonitor reports allow the whole project team to subscribe to cost reports.
Related FinOps Domains
Arraya found that Customers first and foremost wanted to Understanding Cloud Usage and Cost.
As Azure expenses started to rise, the need for efficient cloud resource management became crucial. The goal was to match the real workload requirements with the cloud services used, ensuring that only appropriate resources were accessed when needed, to best serve the business. This highlighted the significance of Cloud Usage Optimisation.
Reservation/RI Recommendations are part of the Cloud Rate Optimisation domain.
More Success Stories
How Clinic to Cloud lowered Azure costs by 46% with CloudMonitor | Customer Success Story
About Transport for NSW
Transport for NSW is a government-run enterprise responsible for the delivery and development of safe, integrated and efficient transport systems for the people of NSW; including transport planning, strategy, policy, procurement and other non-service delivery functions across all modes of transport: Buses, Ferries, Light Rail, Trains and Metro.
NSW Transport works hand-in-hand with operating agencies, private operators and industry partners to deliver customer-focused services and projects in order to make NSW a better place to live, work and visit.
Learn more about Transport for NSW
