RAGSys: Item-Cold-Start Recommender as RAG System

TL;DR

This paper investigates how retrieval-augmented generation can enhance in-context learning for domain-specific tasks by treating demonstration retrieval as an item-cold-start recommender problem, emphasizing diversity and quality.

Contribution

It introduces a novel evaluation method for demonstration retrieval in ICL and highlights the importance of diversity and quality bias, bridging recommender systems and LLMs.

Findings

Diversity and quality bias significantly impact ICL performance.

Recommender system techniques can improve demonstration retrieval for LLMs.

Proposed evaluation method correlates well with downstream task performance.

Abstract

Large Language Models (LLM) hold immense promise for real-world applications, but their generic knowledge often falls short of domain-specific needs. Fine-tuning, a common approach, can suffer from catastrophic forgetting and hinder generalizability. In-Context Learning (ICL) offers an alternative, which can leverage Retrieval-Augmented Generation (RAG) to provide LLMs with relevant demonstrations for few-shot learning tasks. This paper explores the desired qualities of a demonstration retrieval system for ICL. We argue that ICL retrieval in this context resembles item-cold-start recommender systems, prioritizing discovery and maximizing information gain over strict relevance. We propose a novel evaluation method that measures the LLM's subsequent performance on NLP tasks, eliminating the need for subjective diversity scores. Our findings demonstrate the critical role of diversity and quality bias in retrieved demonstrations for effective ICL, and highlight the potential of recommender system techniques in this domain.