Winning Solution For Meta KDD Cup' 24

TL;DR

This paper presents the winning solutions for all tasks in Meta KDD Cup 24, focusing on building a Retrieval-Augmented Generation (RAG) system using web sources and knowledge graphs, with tuned LLMs and API interfaces.

Contribution

It introduces a comprehensive framework combining web retrieval, knowledge graph APIs, and tuned LLMs for improved RAG performance across multiple tasks.

Findings

Achieved 1st place in all three tasks with high scores.

Developed a specialized LLM tuning method to reduce hallucinations.

Created an API interface for effective knowledge graph integration.

Abstract

This paper describes the winning solutions of all tasks in Meta KDD Cup 24 from db3 team. The challenge is to build a RAG system from web sources and knowledge graphs. We are given multiple sources for each query to help us answer the question. The CRAG challenge involves three tasks: (1) condensing information from web pages into accurate answers, (2) integrating structured data from mock knowledge graphs, and (3) selecting and integrating critical data from extensive web pages and APIs to reflect real-world retrieval challenges. Our solution for Task #1 is a framework of web or open-data retrieval and answering. The large language model (LLM) is tuned for better RAG performance and less hallucination. Task #2 and Task #3 solutions are based on a regularized API set for domain questions and the API generation method using tuned LLM. Our knowledge graph API interface extracts directly relevant information to help LLMs answer correctly. Our solution achieves 1st place on all three tasks, achieving a score of 28.4%, 42.7%, and 47.8%, respectively.