On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

TL;DR

This paper investigates the difficulty of ensuring that Large Language Models' Chain-of-Thought reasoning accurately reflects their true behavior, finding that current methods offer only limited improvements across various benchmarks.

Contribution

The study introduces novel strategies for in-context learning, fine-tuning, and activation editing aimed at improving CoT faithfulness, and provides extensive empirical analysis of their effectiveness.

Findings

Limited success of strategies in improving faithfulness

Marginal improvements from fine-tuning and in-context learning

Activation editing showed minimal impact

Abstract

As Large Language Models (LLMs) are increasingly being employed in real-world applications in critical domains such as healthcare, it is important to ensure that the Chain-of-Thought (CoT) reasoning generated by these models faithfully captures their underlying behavior. While LLMs are known to generate CoT reasoning that is appealing to humans, prior studies have shown that these explanations do not accurately reflect the actual behavior of the underlying LLMs. In this work, we explore the promise of three broad approaches commonly employed to steer the behavior of LLMs to enhance the faithfulness of the CoT reasoning generated by LLMs: in-context learning, fine-tuning, and activation editing. Specifically, we introduce novel strategies for in-context learning, fine-tuning, and activation editing aimed at improving the faithfulness of the CoT reasoning. We then carry out extensive empirical analyses with multiple benchmark datasets to explore the promise of these strategies. Our analyses indicate that these strategies offer limited success in improving the faithfulness of the CoT reasoning, with only slight performance enhancements in controlled scenarios. Activation editing demonstrated minimal success, while fine-tuning and in-context learning achieved marginal improvements that failed to generalize across diverse reasoning and truthful question-answering benchmarks. In summary, our work underscores the inherent difficulty in eliciting faithful CoT reasoning from LLMs, suggesting that the current array of approaches may not be sufficient to address this complex challenge.