DuoTeach: Dual Role Self-Teaching for Coarse-to-Fine Decision Coordination in Vision--Language Models

TL;DR

This paper introduces DuoTeach, a self-distillation framework that enhances vision-language models' ability to make coherent, multi-level taxonomy decisions in a coarse-to-fine manner, significantly improving accuracy and zero-shot generalization.

Contribution

DuoTeach is a novel dual-role self-teaching distillation method that improves cross-level decision coordination in vision-language models without requiring ground-truth labels.

Findings

Up to 30.24-point improvement in DWPA on benchmarks.

Zero-shot performance on unseen taxonomies increased from 17.17% to 43.66%.

Enhanced within-call multi-level decision coordination.

Abstract

Coarse-to-fine path decision-making requires predicting a valid taxonomy path in which earlier decisions constrain later ones. However, existing benchmarks score each level independently, obscuring cross-level validity and consistency. To better align evaluation with this setting, we introduce a Joint Path Decision (JPD) protocol that requires predicting the full path in one call, together with Depth-Weighted Prefix Accuracy (DWPA), a metric family that measures path reliability with tunable emphasis on deeper levels. Under JPD, strong vision-language models (VLMs) frequently produce invalid parent-child pairs and brittle full-path predictions, suggesting that their failures stem not only from incomplete taxonomic knowledge but also from unstable cross-level decision coordination. To address this problem, we propose DuoTeach, a dual-role self-teaching distillation framework that requires no ground-truth labels and reuses the same pretrained VLM in two roles. Its Decision-Conditioned Rollout (DCR) generates more coherent teacher traces by conditioning each level on prior decisions, and distills this coordinated behavior into the student without additional test-time rollouts. Across multiple taxonomy-structured benchmarks and VLM base models, DuoTeach improves in-domain DWPA (alpha = 0.95) by up to 30.24 points and boosts zero-shot performance on unseen taxonomies from 17.17% to 43.66%. Further analyses attribute these gains to improved within-call multi-level decision coordination.