https://ift.tt/WvgojYb
Mantel Group and Austroads have detailed the beginning of a project on handling telematics data, particularly in relation to heavy vehicles like trucks, under national regulations.
During Databricks’ Data and AI summit held on 7 August at the Hilton Hotel in Sydney, Mantel Group told ARN how it was chosen by the organisation to modernise its data analytics pipeline, improve its technology infrastructure, and implement advanced tools like continuous integration/continuous deployment (CI/CD) and Terraform.
The organisation tapped Mantel Group as the increasing volume of data was pushing the limits of Austroads’ system, creating delays in processing and other inefficiencies, with the project starting in April.
Austroads is the association of the Australian and New Zealand transport agencies, representing all levels of government. The organisation works closely with transport operators, and their work is governed by strict regulations set by the National Telematics Framework. This ensures the data collected, including license plates, vehicle identification numbers (VINs) and GPS tracking as vehicles move through road sections, is compliant with national guidelines and laws.
Initially, only a few thousand vehicles were monitored, but now the numbers are projected to reach around 90,000 in a few years.
Currently, the platform Austroad uses for telematics data is based on SQL Server, a traditional database system. However, the increasing volume of data — now over a billion records per year — is pushing the limits of this system, creating delays in processing and other inefficiencies.
Payam Ghadirian, head of data and analytics at Austroads, said alongside its goal to build a robust, modern data platform are certain principles that should guide the technical deployment.
“With the vast amount of data we’re handling, validation is critical,” he said. “Automation needs to be at the heart of the design, from the ingestion and validation processes to the infrastructure itself.”
While some manual work will be involved, automation will be the focus, from the Terraform code that spins up the AWS infrastructure to the Databricks services and the data pipelines, said Ghadirian.
“Without it, we risk circling back to where we started,” he said. “Additionally, we’re working in an environment where we don’t control the data entirely.
“It’s provided by third parties, so we need to emphasise data governance, quality and validation.”
According to Ghadirian, Austroads approached this by embedding testing directly into the ingestion pipeline and using a modular approach to add new tests as issues with the data arise. This allowed the organisation to handle errors and loop them back to the third parties.
“Austroad’s strength lies in gaining a deep understanding of the business and recognising the value that data brings to operations and customer services,” he said.
“It’s important to step back from the technology sometimes and ask what are we trying to achieve? What are the true value drivers behind these technical requirements?”
At the same time, Ghadirian noted it must also anticipate the challenges both organisations might face and build with those potential obstacles in mind.
“One major challenge is not being in control of our data mastering process, which remains crucial,” he said.
“We also know that standard analytics alone won’t suffice in the future. We need to be prepared for machine learning and AI, and that means setting ourselves up to deploy machine learning use cases effectively.”
Marty Conneely, partner at Mantel Group, said Databricks resolves scalability issues, as it’s designed to handle increasing data loads, supports machine learning and enables automation and modular architecture.
“It’s also built for collaboration, allowing us to deliver value quickly,” he added. “It’s been the backbone of this project, and we’re excited to continue our partnership with Databricks and Austroads.