Solution for OOD-CV UNICORN Challenge 2024 Object Detection Assistance LLM Counting Ability Improvement

TL;DR

This paper presents ODAC, a method combining object detection with large language models to improve robustness and counting accuracy in out-of-distribution object detection tasks, achieving second place in the ECCV challenge.

Contribution

The paper introduces ODAC, a novel approach integrating object detection assistance with LLMs to enhance performance on challenging OOD datasets.

Findings

Ranked second in ECCV OOD-CV UNICORN Challenge 2024

Achieved a score of 0.86 on the test dataset

Demonstrated improved robustness on OOD and challenging datasets

Abstract

This report provide a detailed description of the method that we explored and proposed in the ECCV OOD-CV UNICORN Challenge 2024, which focusing on the robustness of responses from large language models. The dataset of this competition are OODCA-VQA and SketchyQA. In order to test the robustness of the model. The organizer extended two variants of the dataset OODCV-Counterfactual and Sketchy-Challenging. There are several difficulties with these datasets. Firstly, the Sketchy-Challenging dataset uses some rarer item categories to test the model's generalization ability. Secondly, in the OODCV-Counterfactual dataset, the given problems often have inflection points and computational steps, requiring the model to recognize them during the inference process. In order to address this issue, we propose a simple yet effective approach called Object Detection Assistance Large Language Model(LLM) Counting Ability Improvement(ODAC), which focuses on using the object detection model to assist the LLM. To clarify, our approach contains two main blocks: (1)Object Detection Assistance. (2) Counterfactual Specific prompt. Our approach ranked second in the final test with a score of 0.86.