Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors

TL;DR

This study explores how transformer-based large language models internally recognize and correct typographical errors, revealing specific neurons and attention heads responsible for typo detection and correction across different layers.

Contribution

The paper introduces a method to identify neurons and heads that detect and fix typos, providing insights into internal mechanisms of LLMs for typo correction.

Findings

LLMs can correct typos using local context with specific neurons.

Middle layer neurons handle global context typo correction.

Typo heads consider broad context rather than specific tokens.

Abstract

This paper investigates how LLMs encode inputs with typos. We hypothesize that specific neurons and attention heads recognize typos and fix them internally using local and global contexts. We introduce a method to identify typo neurons and typo heads that work actively when inputs contain typos. Our experimental results suggest the following: 1) LLMs can fix typos with local contexts when the typo neurons in either the early or late layers are activated, even if those in the other are not. 2) Typo neurons in the middle layers are responsible for the core of typo-fixing with global contexts. 3) Typo heads fix typos by widely considering the context not focusing on specific tokens. 4) Typo neurons and typo heads work not only for typo-fixing but also for understanding general contexts.