LLMs as Function Approximators: Terminology, Taxonomy, and Questions for Evaluation

TL;DR

This paper proposes viewing large language models as function approximators for specific tasks, emphasizing evaluation of their approximation quality, stability, and discoverability, moving beyond vague general intelligence metaphors.

Contribution

It introduces a new framework for evaluating LLMs as function approximators, integrating practical and theoretical aspects of their capabilities and limitations.

Findings

Highlights the importance of approximation quality and stability.

Connects evaluation metrics with theoretical understanding.

Addresses secondary issues like prompt injection and jailbreaking.

Abstract

Natural Language Processing has moved rather quickly from modelling specific tasks to taking more general pre-trained models and fine-tuning them for specific tasks, to a point where we now have what appear to be inherently generalist models. This paper argues that the resultant loss of clarity on what these models model leads to metaphors like "artificial general intelligences" that are not helpful for evaluating their strengths and weaknesses. The proposal is to see their generality, and their potential value, in their ability to approximate specialist function, based on a natural language specification. This framing brings to the fore questions of the quality of the approximation, but beyond that, also questions of discoverability, stability, and protectability of these functions. As the paper will show, this framing hence brings together in one conceptual framework various aspects of evaluation, both from a practical and a theoretical perspective, as well as questions often relegated to a secondary status (such as "prompt injection" and "jailbreaking").