Precise Drive with VLM: First Prize Solution for PRCV 2024 Drive LM challenge

TL;DR

This paper presents a winning solution for the PRCV 2024 Drive LM challenge, utilizing an enhanced multi-modal model with refined input processing and training strategies to improve driving decision-making accuracy.

Contribution

The authors developed an improved multi-modal model with novel input formatting and a modified loss function, achieving first place in the PRCV 2024 Drive LM challenge.

Findings

Achieved a top score of 0.6064 in the challenge.

Enhanced model accuracy through input and training refinements.

Secured first prize in the competition.

Abstract

This technical report outlines the methodologies we applied for the PRCV Challenge, focusing on cognition and decision-making in driving scenarios. We employed InternVL-2.0, a pioneering open-source multi-modal model, and enhanced it by refining both the model input and training methodologies. For the input data, we strategically concatenated and formatted the multi-view images. It is worth mentioning that we utilized the coordinates of the original images without transformation. In terms of model training, we initially pre-trained the model on publicly available autonomous driving scenario datasets to bolster its alignment capabilities of the challenge tasks, followed by fine-tuning on the DriveLM-nuscenes Dataset. During the fine-tuning phase, we innovatively modified the loss function to enhance the model's precision in predicting coordinate values. These approaches ensure that our model possesses advanced cognitive and decision-making capabilities in driving scenarios. Consequently, our model achieved a score of 0.6064, securing the first prize on the competition's final results.