Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events

TL;DR

This paper introduces BlackSwanSuite, a benchmark for testing vision-language models' ability to reason about unexpected, out-of-distribution events in videos, revealing significant gaps compared to human performance.

Contribution

The paper presents a novel benchmark suite for evaluating abductive and defeasible reasoning in VLMs on atypical video events, highlighting current model limitations.

Findings

Current VLMs lag behind humans by up to 32% on these tasks.

Significant performance gaps indicate limitations in reasoning about unexpected events.

The benchmark enables targeted evaluation of reasoning capabilities beyond pattern recognition.

Abstract

The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus on typical visual scenarios, making it difficult to discern whether model performance stems from keen perception and reasoning skills, or reliance on pure statistical recall. We argue that by focusing on atypical events in videos, clearer insights can be gained on the core capabilities of VLMs. Explaining and understanding such out-of-distribution events requires models to extend beyond basic pattern recognition and regurgitation of their prior knowledge. To this end, we introduce BlackSwanSuite, a benchmark for evaluating VLMs' ability to reason about unexpected events through abductive and defeasible tasks. Our tasks artificially limit the amount of visual information provided to models while questioning them about hidden unexpected events, or provide new visual information that could change an existing hypothesis about the event. We curate a comprehensive benchmark suite comprising over 3,800 MCQ, 4,900 generative and 6,700 yes/no questions, spanning 1,655 videos. After extensively evaluating various state-of-the-art VLMs, including GPT-4o and Gemini 1.5 Pro, as well as open-source VLMs such as LLaVA-Video, we find significant performance gaps of up to 32% from humans on these tasks. Our findings reveal key limitations in current VLMs, emphasizing the need for enhanced model architectures and training strategies. Our data and leaderboard is available at blackswan.cs.ubc.ca.