Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps

TL;DR

This paper introduces a novel method called FUR to measure how faithfully language models' reasoning steps are embedded in their parameters, by selectively unlearning reasoning steps and observing prediction changes.

Contribution

The paper proposes a new framework and method, FUR, to quantify parametric faithfulness of chain of thought reasoning in language models, addressing a key gap in understanding model reasoning.

Findings

FUR can precisely alter model predictions by unlearning key reasoning steps.

Unlearning affects the model's reasoning, indicating faithfulness of the chain of thought.

Generated CoTs post-unlearning support different answers, revealing deeper model behavior.

Abstract

When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.