Counterfactual Likelihood Tests for Indirect Influence in Private Reasoning Channels

TL;DR

This paper introduces a counterfactual likelihood test to measure influence in private reasoning channels, demonstrating its effectiveness over traditional textual probes and highlighting the importance of separating direct and indirect influence in private AI systems.

Contribution

The paper presents a novel counterfactual likelihood method for evaluating private influence, controlling confounds, and providing more reliable influence measurement in reasoning models.

Findings

Counterfactual likelihood separates unmasked and masked influence conditions.

Reverse influence from B to A is near zero in masked validation.

The method reliably identifies private-to-public influence pathways.

Abstract

Reasoning systems increasingly separate intermediate computation into private and public channels, creating evaluation cases that look similar in transcripts: independent co-derivation, direct access to private content, and indirect influence through public communication. This paper presents a counterfactual likelihood test for measuring influence between private reasoning channels. The method replaces an upstream private block with a length-matched donor block, holds the public token sequence and downstream target fixed, and measures the downstream target's negative-log-likelihood shift. On a 7B role-channel reasoning model used for validation, textual probes are unreliable: raw n-gram overlap overstates leakage, corrected overlap remains noisy, and canary reproduction reports no discrimination. Counterfactual likelihood separates unmasked and masked conditions, while length matching controls a RoPE positional confound. In the hardened masked validation, reverse B-to-A influence is near zero, while A-to-B influence persists through public-speech hidden states. A multi-checkpoint validation across three checkpoints, five seeds, and 13,734 valid directional contrasts replicates this asymmetry. A graph-separation control that blocks private-to-public carrier edges produces bit-identical natural and counterfactual scores across all 13,734 control evaluations, identifying the tested public-channel pathway as the complete carrier of the measured counterfactual signal under the implemented role-visibility mask. The results show that private-channel evaluation should report direct and indirect influence separately, and that counterfactual likelihood probes provide a practical default for measuring these boundaries.