[De|Re]constructing VLMs' Reasoning in Counting

TL;DR

This paper investigates the reasoning limitations of vision-language models in counting tasks, identifying causes of errors, and demonstrating that targeted fine-tuning of the output layer significantly enhances their counting accuracy.

Contribution

It provides a detailed analysis of VLMs' counting failures and introduces a simple yet effective fine-tuning approach to improve their reasoning capabilities.

Findings

VLMs are sensitive to object number, type, and arrangement.

Errors mainly stem from incorrect last-layer representations.

Fine-tuning the output layer improves counting accuracy by up to 21%.

Abstract

Vision-Language Models (VLMs) have recently gained attention due to their competitive performance on multiple downstream tasks, achieved by following user-input instructions. However, VLMs still exhibit several limitations in visual reasoning, such as difficulties in identifying relations (e.g., spatial, temporal, and among objects), understanding temporal sequences (e.g., frames), and counting objects. In this work, we go beyond score-level benchmark evaluations of VLMs by investigating the underlying causes of their failures and proposing a targeted approach to improve their reasoning capabilities. We study the reasoning skills of seven state-of-the-art VLMs in the counting task under controlled experimental conditions. Our experiments show that VLMs are highly sensitive to the number and type of objects, their spatial arrangement, and the co-occurrence of distractors. A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space. Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%. We corroborate these findings by achieving consistent improvements on real-world datasets.