Accelerating Statewide Connected Vehicles Big (Sensor Fusion) Data ETL Pipelines on GPUs

TL;DR

This paper presents a GPU-accelerated ETL pipeline for statewide connected vehicle data, significantly reducing processing time and enabling real-time insights for transportation management.

Contribution

It introduces a GPU-based data processing framework using Nvidia Rapids and Dask that accelerates CV data ETL by 70 times, facilitating real-time analysis.

Findings

70x faster data processing for Missouri CV data

Reduced ETL time from 48 hours to 25 minutes

Enables real-time transportation infrastructure insights

Abstract

Real-time traffic and sensor data from connected vehicles have the potential to provide insights that will lead to the immediate benefit of efficient management of the transportation infrastructure and related adjacent services. However, the growth of electric vehicles (EVs) and connected vehicles (CVs) has generated an abundance of CV data and sensor data that has put a strain on the processing capabilities of existing data center infrastructure. As a result, the benefits are either delayed or not fully realized. To address this issue, we propose a solution for processing state-wide CV traffic and sensor data on GPUs that provides real-time micro-scale insights in both temporal and spatial dimensions. This is achieved through the use of the Nvidia Rapids framework and the Dask parallel cluster in Python. Our findings demonstrate a 70x acceleration in the extraction, transformation, and loading (ETL) of CV data for the State of Missouri for a full day of all unique CV journeys, reducing the processing time from approximately 48 hours to just 25 minutes. Given that these results are for thousands of CVs and several thousands of individual journeys with sub-second sensor data, implies that we can model and obtain actionable insights for the management of the transportation infrastructure.