Auditing the Reliability of Multimodal Generative Search

TL;DR

This paper audits the reliability of multimodal generative search systems, revealing a significant portion of claims are unsupported by cited videos, highlighting trustworthiness issues.

Contribution

It provides the first large-scale analysis of video-grounded claims in multimodal search, identifying common failure modes and factors linked to unsupported claims.

Findings

3.7% to 18.7% of claims are unsupported by sources

Unsupported claims often involve unverifiable details and overstated assertions

Claims with low semantic similarity to videos are more likely unsupported

Abstract

Multimodal Large Language Models (MLLMs) increasingly function as generative search systems that retrieve and synthesize answers from multimedia content, including YouTube videos. Although these systems project authority by citing specific videos as evidence, the extent to which these citations genuinely substantiate the generated claims remains unexamined. We present a large-scale audit of the Gemini 2.5 Pro multimodal search system, analyzing 11,943 claim-video pairs generated across Medical, Economic, and General domains. Through automated verification using three independent LLM judges (87.7% inter-rater agreement), validated against human annotations, we find that depending on the judge's strictness, between 3.7% and 18.7% of video-grounded claims are not supported by their cited sources. The dominant failure modes are not outright contradictions but rather unverifiable specificities and overstated claims, suggesting the system injects precise but ungrounded details from parametric knowledge while citing videos as evidence. Exploratory post-hoc analysis via logistic regression reveals properties associated with these failures: claims departing from source vocabulary ($\beta = -1.6$ to $-3.1$, $p < 0.01$) and claims with low semantic similarity to the video transcript ($\beta = -2.1$ to $-11.6$, $p < 0.01$) are significantly more likely to be unsupported. These findings characterize the current trustworthiness of video-based generative search and highlight the gap between the confidence these systems project and the fidelity of their outputs.