On Evaluating LLMs' Capabilities as Functional Approximators: A Bayesian Perspective

TL;DR

This paper introduces a Bayesian evaluation framework to assess LLMs' ability to model functions, revealing their strengths in leveraging prior knowledge but weaknesses in raw data pattern understanding.

Contribution

It presents a novel Bayesian-based evaluation method for LLMs' function modeling capabilities, providing new insights into their strengths and limitations.

Findings

LLMs excel at using prior domain knowledge.

LLMs are relatively weak at understanding raw data patterns.

The framework offers a comprehensive assessment of LLMs' function modeling abilities.

Abstract

Recent works have successfully applied Large Language Models (LLMs) to function modeling tasks. However, the reasons behind this success remain unclear. In this work, we propose a new evaluation framework to comprehensively assess LLMs' function modeling abilities. By adopting a Bayesian perspective of function modeling, we discover that LLMs are relatively weak in understanding patterns in raw data, but excel at utilizing prior knowledge about the domain to develop a strong understanding of the underlying function. Our findings offer new insights about the strengths and limitations of LLMs in the context of function modeling.