GTMA: Dynamic Representation Optimization for OOD Vision-Language Models

TL;DR

This paper introduces GTMA, a dynamic optimization method that enhances vision-language models' ability to handle out-of-distribution images by constructing adaptive pseudo-word embeddings at inference time.

Contribution

The paper proposes a novel inference-time optimization framework, GTMA, that creates continuous pseudo-words to improve OOD performance in vision-language models, surpassing existing methods.

Findings

GTMA improves zero-shot OOD accuracy by up to 20%.

GTMA maintains in-distribution performance.

Ablation confirms pseudo-word optimization is essential.

Abstract

Vision-language models (VLMs) struggle in open-world applications, where out-of-distribution (OOD) concepts often trigger cross-modal alignment collapse and severely degrade zero-shot performance. We identify the root cause as modal asymmetry: while the visual encoder can extract discriminative features from unseen images, the text encoder is constrained by a fixed discrete vocabulary and cannot synthesize new semantic anchors. Existing approaches such as CoOp or LoRA provide only partial remedies, as they remain confined to the pre-trained semantic space. To overcome this bottleneck, we propose dynamic representation optimization, realized through the Guided Target-Matching Adaptation (GTMA) framework. At inference time, GTMA constructs a continuous pseudo-word embedding that best aligns with an OOD image's visual anchor, effectively bypassing vocabulary limitations. The optimization is driven by an adaptive gradient-based representation policy optimization algorithm, which incorporates semantic regularization to preserve plausibility and compatibility with the model's prior knowledge. Experiments on ImageNet-R and the VISTA-Beyond benchmark demonstrate that GTMA improves zero-shot and few-shot OOD accuracy by up to 15-20 percent over the base VLM while maintaining performance on in-distribution concepts. Ablation studies further confirm the necessity of pseudo-word optimization.