Breaking the Pre-Sampling Barrier: Activation-Informed Difficulty-Aware Self-Consistency

TL;DR

This paper introduces ACTSC, a novel method that uses internal neuron activations to estimate problem difficulty, enabling adaptive sampling in Large Language Models without extra model calls, thus reducing inference costs.

Contribution

ACTSC leverages internal neuron activations for difficulty estimation, eliminating the need for pre-sampling and additional model calls, improving efficiency in self-consistency decoding.

Findings

Reduces inference costs significantly

Maintains accuracy comparable to existing methods

Applicable to new datasets without pre-sampling

Abstract

Self-Consistency (SC) is an effective decoding strategy that improves the reasoning performance of Large Language Models (LLMs) by generating multiple chain-of-thought reasoning paths and selecting the final answer via majority voting. However, it suffers from substantial inference costs because it requires a large number of samples. To mitigate this issue, Difficulty-Adaptive Self-Consistency (DSC) was proposed to reduce unnecessary token usage for easy problems by adjusting the number of samples according to problem difficulty. However, DSC requires additional model calls and pre-sampling to estimate difficulty, and this process is repeated when applying to each dataset, leading to significant computational overhead. In this work, we propose Activation-Informed Difficulty-Aware Self-Consistency (ACTSC) to address these limitations. ACTSC leverages internal difficulty signals reflected in the feed-forward network neuron activations to construct a lightweight difficulty estimation probe, without any additional token generation or model calls. The probe dynamically adjusts the number of samples for SC and can be applied to new datasets without requiring pre-sampling for difficulty estimation. To validate its effectiveness, we conduct experiments on five benchmarks. Experimental results show that ACTSC effectively reduces inference costs while maintaining accuracy relative to existing methods.