From task structures to world models: What do LLMs know?

TL;DR

This paper explores the nature of knowledge in large language models, proposing the concept of 'instrumental knowledge' and examining how it relates to human-like worldly understanding through structured world models.

Contribution

It introduces the idea of 'instrumental knowledge' in LLMs and analyzes how such models can approximate human-like worldly knowledge via structured world models.

Findings

LLMs possess 'instrumental knowledge' based on their abilities.

Recovery of worldly knowledge in LLMs depends on resource-rational tradeoffs.

Structured world models influence the extent of knowledge LLMs can exhibit.

Abstract

In what sense does a large language model have knowledge? The answer to this question extends beyond the capabilities of a particular AI system, and challenges our assumptions about the nature of knowledge and intelligence. We answer by granting LLMs "instrumental knowledge"; knowledge defined by a certain set of abilities. We then ask how such knowledge is related to the more ordinary, "worldly" knowledge exhibited by human agents, and explore this in terms of the degree to which instrumental knowledge can be said to incorporate the structured world models of cognitive science. We discuss ways LLMs could recover degrees of worldly knowledge, and suggest such recovery will be governed by an implicit, resource-rational tradeoff between world models and task demands.