FLAD: Federated Learning for LLM-based Autonomous Driving in Vehicle-Edge-Cloud Networks

TL;DR

This paper introduces FLAD, a federated learning framework for autonomous driving that leverages distributed multimodal data, optimizing training efficiency and privacy in vehicle-edge-cloud networks, with practical implementation on resource-constrained devices.

Contribution

FLAD presents a novel cloud-edge-vehicle architecture, an intelligent training scheduling mechanism, and a personalized knowledge distillation method for federated LLM-based autonomous driving.

Findings

FLAD achieves superior autonomous driving performance.

Efficiently utilizes distributed vehicular resources.

Successfully prototypes on resource-constrained devices.

Abstract

Large Language Models (LLMs) have impressive data fusion and reasoning capabilities for autonomous driving (AD). However, training LLMs for AD faces significant challenges including high computation transmission costs, and privacy concerns associated with sensitive driving data. Federated Learning (FL) is promising for enabling autonomous vehicles (AVs) to collaboratively train models without sharing raw data. We present Federated LLM-based Autonomous Driving (FLAD), an FL framework that leverages distributed multimodal sensory data across AVs in heterogeneous environment. FLAD has three key innovations: (1) a cloud-edge-vehicle collaborative architecture that reduces communication delay and preserving data privacy; (2) an intelligent parallelized collaborative training with a communication scheduling mechanism that optimizes training efficiency, leveraging end-devices otherwise having insufficient resources for model training; and (3) a knowledge distillation method that personalizes LLM according to heterogeneous edge data. In addition, we prototype FLAD in a testbed with NVIDIA Jetsons, overcoming practical implementation challenges including CPU/GPU memory sharing in resource-constrained devices, dynamic model partitions, and fault-tolerant training.Extensive experimental evaluation demonstrates that FLAD achieves superior end-to-end AD performance while efficiently utilizing distributed vehicular resources, opening up new possibilities for future collaborative AD model training and knowledge sharing.