Solution for Meta KDD Cup'25: A Comprehensive Three-Step Framework for Vision Question Answering

TL;DR

This paper presents a comprehensive three-step framework for vision question answering, utilizing multi-modal retrieval, data augmentation, and multi-task fine-tuning to improve accuracy and address hallucination issues in VLLMs.

Contribution

It introduces a unified solution for multi-task VQA challenges, incorporating novel data augmentation, reranking, and multi-source retrieval techniques, winning multiple rankings in the KDD Cup 2025 challenge.

Findings

Achieved top rankings in all three tasks of the KDD Cup 2025 CRAG-MM challenge.

Demonstrated effectiveness of multi-task fine-tuning and RAG in VQA.

Secured second place in Task 3 after human evaluation.

Abstract

Vision Large Language Models (VLLMs) have improved multi-modal understanding and visual question answering (VQA), but still suffer from hallucinated answers. Multi-modal Retrieval-Augmented Generation (RAG) helps address these issues by incorporating external information, yet challenges remain in visual context comprehension, multi-source retrieval, and multi-turn interactions. To address these challenges, Meta constructed the CRAG-MM benchmark and launched the CRAG-MM Challenge at KDD Cup 2025, which consists of three tasks. This paper describes the solutions of all tasks in Meta KDD Cup'25 from BlackPearl team. We use a single model for each task, with key methods including data augmentation, RAG, reranking, and multi-task fine-tuning. Our solution achieve automatic evaluation rankings of 3rd, 3rd, and 1st on the three tasks, and win second place in Task3 after human evaluation.