PragWorld: A Benchmark Evaluating LLMs' Local World Model under Minimal Linguistic Alterations and Conversational Dynamics

TL;DR

This paper evaluates how well language models can maintain and update internal world models during conversations, especially under minimal linguistic changes, revealing their struggles and proposing interpretability and fine-tuning solutions.

Contribution

It introduces benchmarks for assessing LLMs' ability to encode and update conversational world models and proposes interpretability and regularization methods to improve their robustness.

Findings

LLMs struggle to accurately track entities under linguistic alterations.

Transformer layers can be identified as useful or harmful for maintaining conversation details.

Layer-regularization fine-tuning improves LLM robustness in conversational tasks.

Abstract

Real-world conversations are rich with pragmatic elements, such as entity mentions, references, and implicatures. Understanding such nuances is a requirement for successful natural communication, and often requires building a local world model which encodes such elements and captures the dynamics of their evolving states. However, it is not well-understood whether language models (LMs) construct or maintain a robust implicit representation of conversations. In this work, we evaluate the ability of LMs to encode and update their internal world model in dyadic conversations and test their malleability under linguistic alterations. To facilitate this, we apply seven minimal linguistic alterations to conversations sourced from popular datasets and construct two benchmarks comprising yes-no questions. We evaluate a wide range of open and closed source LMs and observe that they struggle to maintain robust accuracy. Our analysis unveils that LMs struggle to memorize crucial details, such as tracking entities under linguistic alterations to conversations. We then propose a dual-perspective interpretability framework which identifies transformer layers that are useful or harmful and highlights linguistic alterations most influenced by harmful layers, typically due to encoding spurious signals or relying on shortcuts. Inspired by these insights, we propose two layer-regularization based fine-tuning strategies that suppress the effect of the harmful layers.