GATES: Self-Distillation under Privileged Context with Consensus Gating

TL;DR

This paper introduces GATES, a self-distillation method for document-grounded question answering that leverages consensus among multiple reasoning traces to improve training when supervision is unreliable.

Contribution

GATES proposes a novel consensus-gated self-distillation approach that uses agreement among multiple reasoning trajectories to supervise a student model without ground truth labels.

Findings

In-domain accuracy improved from 46.0% to 62.0%.

Math benchmark accuracy increased from 20.2% to 35.4%.

Consensus gating significantly enhances transfer to document-free settings.

Abstract

We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.