OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework

TL;DR

OWLed introduces an outlier-weighted layerwise pruning method that significantly compresses large language models for autonomous driving, maintaining performance while reducing computational costs.

Contribution

The paper presents a novel pruning framework that leverages outlier features for model compression without fine-tuning, tailored for autonomous driving applications.

Findings

OWLed achieves superior compression with minimal performance loss.

The encoder is more sensitive to pruning than the LLM.

The method improves perception, action prediction, and language understanding.

Abstract

The integration of Large Language Models (LLMs) into autonomous driving systems offers promising enhancements in environmental understanding and decision-making. However, the substantial computational demands of deploying LLMs locally on vehicles render this approach unfeasible for real-world automotive applications. To address this challenge, we introduce OWLed, the Outlier-Weighed Layerwise Pruning for Efficient Autonomous Driving Framework that leverages outlier-weighted layerwise sparsity for model compression. Our method assigns non-uniform sparsity ratios to different layers based on the distribution of outlier features, significantly reducing the model size without the need for fine-tuning. To ensure the compressed model adapts well to autonomous driving tasks, we incorporate driving environment data into both the calibration and pruning processes. Our empirical studies reveal that the encoder component is more sensitive to pruning than the LLM, highlighting its critical role in the system. Experimental results demonstrate that OWLed outperforms existing methods in perception, action prediction, and language understanding while substantially lowering computational requirements. These findings underscore the potential of combining advanced pruning techniques with LLMs to develop efficient and robust autonomous driving systems capable of handling complex scenarios. Code will be made publicly available.