Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024

TL;DR

This paper presents a fine-tuned multimodal model, InternVL-1.5, for autonomous driving tasks involving language, achieving top performance in the CVPR 2024 Autonomous Grand Challenge by effectively handling multi-view images and automatic annotations.

Contribution

The work introduces a novel approach to adapt InternVL-1.5 for autonomous driving by multi-view image formatting and automatic annotation, leading to state-of-the-art results in the challenge.

Findings

Achieved a score of 0.6002 on the challenge leaderboard.

Effectively handled multi-view images with specific formatting.

Demonstrated the effectiveness of automatic annotation strategy.

Abstract

This technical report describes the methods we employed for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. We utilized a powerful open-source multimodal model, InternVL-1.5, and conducted a full-parameter fine-tuning on the competition dataset, DriveLM-nuScenes. To effectively handle the multi-view images of nuScenes and seamlessly inherit InternVL's outstanding multimodal understanding capabilities, we formatted and concatenated the multi-view images in a specific manner. This ensured that the final model could meet the specific requirements of the competition task while leveraging InternVL's powerful image understanding capabilities. Meanwhile, we designed a simple automatic annotation strategy that converts the center points of objects in DriveLM-nuScenes into corresponding bounding boxes. As a result, our single model achieved a score of 0.6002 on the final leadboard.