An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs

TL;DR

This paper explores how neuron activation in large language models explains their arithmetic reasoning abilities with Chain-of-Thought prompts, offering a unified perspective on prior empirical observations.

Contribution

It introduces a neuron activation-based approach to explain LLM reasoning, specifically identifying reasoning neurons in feed-forward layers using GPT-4.

Findings

Activation of reasoning neurons correlates with LLM performance on arithmetic tasks.

Neuron analysis explains the importance of CoT prompt components.

Proposes an automatic method to identify reasoning neurons.

Abstract

Large language models (LLMs) have shown strong arithmetic reasoning capabilities when prompted with Chain-of-Thought (CoT) prompts. However, we have only a limited understanding of how they are processed by LLMs. To demystify it, prior work has primarily focused on ablating different components in the CoT prompt and empirically observing their resulting LLM performance change. Yet, the reason why these components are important to LLM reasoning is not explored. To fill this gap, in this work, we investigate ``neuron activation'' as a lens to provide a unified explanation to observations made by prior work. Specifically, we look into neurons within the feed-forward layers of LLMs that may have activated their arithmetic reasoning capabilities, using Llama2 as an example. To facilitate this investigation, we also propose an approach based on GPT-4 to automatically identify neurons that imply arithmetic reasoning. Our analyses revealed that the activation of reasoning neurons in the feed-forward layers of an LLM can explain the importance of various components in a CoT prompt, and future research can extend it for a more complete understanding.