Plant in Cupboard, Orange on Rably, Inat Aphone. Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environment

TL;DR

This paper evaluates large language models' ability to incrementally learn and adapt in a text-based environment, focusing on object discovery, in-context learning, and understanding synthetic words, revealing performance gaps especially with synthetic vocabulary.

Contribution

It introduces a novel benchmark for testing incremental learning and in-context adaptation of LLMs in a simulated environment with new objects and language.

Findings

Larger models outperform smaller ones in object discovery tasks.

All models struggle with understanding synthetic pseudo-English words.

In-context learning improves task efficiency but varies across models.

Abstract

Large Language Models (LLMs) serve not only as chatbots but as key components in agent systems, where their common-sense knowledge significantly impacts performance as language-based planners for situated or embodied action. We assess LLMs' incremental learning (based on feedback from the environment), and controlled in-context learning abilities using a text-based environment. We introduce challenging yet interesting set of experiments to test i) how agents can incrementally solve tasks related to every day objects in typical rooms in a house where each of them are discovered by interacting within the environment, ii) controlled in-context learning abilities and efficiency of agents by providing short info about locations of objects and rooms to check how faster the task can be solved, and finally iii) using synthetic pseudo-English words to gauge how well LLMs are at inferring meaning of unknown words from environmental feedback. Results show that larger commercial models have a substantial gap in performance compared to open-weight but almost all models struggle with the synthetic words experiments.