CogVLM2: Visual Language Models for Image and Video Understanding

TL;DR

CogVLM2 is a new family of visual language models that significantly improve image and video understanding by supporting higher resolutions, integrating multi-frame inputs, and achieving state-of-the-art results on multiple benchmarks.

Contribution

The paper introduces CogVLM2, a novel generation of visual language models with enhanced architecture, training methods, and capabilities for both image and video understanding, including automated temporal grounding.

Findings

Achieved state-of-the-art results on multiple benchmarks.

Supported input resolution up to 1344x1344 pixels.

Integrated multi-frame input with timestamps for video understanding.

Abstract

Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in pursuit of enhanced vision-language fusion, efficient higher-resolution architecture, and broader modalities and applications. Here we propose the CogVLM2 family, a new generation of visual language models for image and video understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image understanding model, CogVLM2 inherits the visual expert architecture with improved training recipes in both pre-training and post-training stages, supporting input resolution up to $1344 \times 1344$ pixels. As a video understanding model, CogVLM2-Video integrates multi-frame input with timestamps and proposes automated temporal grounding data construction. Notably, CogVLM2 family has achieved state-of-the-art results on benchmarks like MMBench, MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4, contributing to the advancement of the field.