Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models

TL;DR

This paper investigates how large language models perform in regression tasks by analyzing whether they rely more on internal knowledge retrieval or learning from in-context examples, providing insights for better prompt engineering.

Contribution

It introduces a framework to evaluate the balance between knowledge retrieval and in-context learning in LLMs for regression tasks, supported by extensive experiments.

Findings

LLMs can solve real-world regression problems

Performance depends on prior knowledge and example richness

Mechanisms range between retrieval and learning spectrum

Abstract

Generative Large Language Models (LLMs) are capable of being in-context learners. However, the underlying mechanism of in-context learning (ICL) is still a major research question, and experimental research results about how models exploit ICL are not always consistent. In this work, we propose a framework for evaluating in-context learning mechanisms, which we claim are a combination of retrieving internal knowledge and learning from in-context examples by focusing on regression tasks. First, we show that LLMs can solve real-world regression problems and then design experiments to measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. We argue that this process lies on a spectrum between these two extremes. We provide an in-depth analysis of the degrees to which these mechanisms are triggered depending on various factors, such as prior knowledge about the tasks and the type and richness of the information provided by the in-context examples. We employ three LLMs and utilize multiple datasets to corroborate the robustness of our findings. Our results shed light on how to engineer prompts to leverage meta-learning from in-context examples and foster knowledge retrieval depending on the problem being addressed.