Evaluating Contrast Localizer for Identifying Causal Units in Social & Mathematical Tasks in Language Models

TL;DR

This paper adapts a neuroscientific contrast localizer to identify causally relevant units in large language and vision-language models for Theory of Mind and mathematical reasoning tasks, revealing surprising results about unit importance.

Contribution

It introduces a contrast localizer method for pinpointing causally relevant units in LLMs and VLMs, and critically evaluates its effectiveness across multiple models and tasks.

Findings

Low-activation units sometimes cause larger performance drops.

Units from the mathematical localizer often impair ToM performance.

Contrast localizers may not reliably identify causally relevant units.

Abstract

This work adapts a neuroscientific contrast localizer to pinpoint causally relevant units for Theory of Mind (ToM) and mathematical reasoning tasks in large language models (LLMs) and vision-language models (VLMs). Across 11 LLMs and 5 VLMs ranging in size from 3B to 90B parameters, we localize top-activated units using contrastive stimulus sets and assess their causal role via targeted ablations. We compare the effect of lesioning functionally selected units against low-activation and randomly selected units on downstream accuracy across established ToM and mathematical benchmarks. Contrary to expectations, low-activation units sometimes produced larger performance drops than the highly activated ones, and units derived from the mathematical localizer often impaired ToM performance more than those from the ToM localizer. These findings call into question the causal relevance of contrast-based localizers and highlight the need for broader stimulus sets and more accurately capture task-specific units.