What do your logits know? (The answer may surprise you!)

TL;DR

This paper investigates how vision-language models retain and leak information at different internal levels, revealing potential privacy risks from model internals through systematic comparison of representational bottlenecks.

Contribution

It provides the first systematic comparison of information retention at various internal levels in vision-language models, highlighting privacy risks from bottleneck representations.

Findings

Bottlenecks like top-k logits can leak task-irrelevant information.

Even simple bottlenecks reveal as much information as full residual streams.

Information leakage occurs at multiple representational levels.

Abstract

Recent work has shown that probing model internals can reveal a wealth of information not apparent from the model generations. This poses the risk of unintentional or malicious information leakage, where model users are able to learn information that the model owner assumed was inaccessible. Using vision-language models as a testbed, we present the first systematic comparison of information retained at different "representational levels'' as it is compressed from the rich information encoded in the residual stream through two natural bottlenecks: low-dimensional projections of the residual stream obtained using tuned lens, and the final top-k logits most likely to impact model's answer. We show that even easily accessible bottlenecks defined by the model's top logit values can leak task-irrelevant information present in an image-based query, in some cases revealing as much information as direct projections of the full residual stream.