BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models

TL;DR

BabyVLM-V2 is a developmentally inspired vision-language model that uses a longitudinal infant-centric dataset and a new benchmark suite to evaluate early childhood cognitive skills, aiming to improve sample efficiency and developmental relevance.

Contribution

It introduces a new pretraining framework and DevCV Toolbox benchmark for developmentally grounded vision-language modeling inspired by infant cognition.

Findings

A compact model pretrained from scratch performs competitively on DevCV Toolbox.

BabyVLM-V2 outperforms GPT-4o on some developmental tasks.

The framework accelerates research in developmentally plausible vision foundation models.

Abstract

Early children's developmental trajectories set up a natural goal for sample-efficient pretraining of vision foundation models. We introduce BabyVLM-V2, a developmentally grounded framework for infant-inspired vision-language modeling that extensively improves upon BabyVLM-V1 through a longitudinal, multifaceted pretraining set, a versatile model, and, most importantly, DevCV Toolbox for cognitive evaluation. The pretraining set maximizes coverage while minimizing curation of a longitudinal, infant-centric audiovisual corpus, yielding video-utterance, image-utterance, and multi-turn conversational data that mirror infant experiences. DevCV Toolbox adapts all vision-related measures of the recently released NIH Baby Toolbox into a benchmark suite of ten multimodal tasks, covering spatial reasoning, memory, and vocabulary understanding aligned with early children's capabilities. Experimental results show that a compact model pretrained from scratch can achieve competitive performance on DevCV Toolbox, outperforming GPT-4o on some tasks. We hope the principled, unified BabyVLM-V2 framework will accelerate research in developmentally plausible pretraining of vision foundation models.