What if Othello-Playing Language Models Could See?

TL;DR

This paper investigates whether multi-modal learning, combining move sequences and board images, enhances understanding and robustness in Othello-playing language models compared to text-only models.

Contribution

The study introduces VISOTHELLO, a multi-modal model for Othello, demonstrating improved performance, robustness, and shared internal representations over text-only approaches.

Findings

Multi-modal training improves model performance.

Multi-modal models show increased robustness to irrelevant perturbations.

Shared internal representations emerge across different architectures.

Abstract

Language models are often said to face a symbol grounding problem. While some have argued the problem can be solved without resort to other modalities, many have speculated that grounded learning is more efficient. We explore this question in Othello, a simplified, rule-based world that offers a controlled and interpretable testbed for studying world understanding. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images. Using the Othello rule understanding task, we examine whether multi-modal learning provides advantages over text-only approaches. We further evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment. Our results suggest that multi-modal training not only improves performance and robustness but also promotes convergence toward shared internal representations across different model architectures.