VisionPangu: A Compact and Fine-Grained Multimodal Assistant with 1.7B Parameters

TL;DR

VisionPangu is a compact 1.7B-parameter multimodal model that enhances detailed image captioning by combining efficient multimodal alignment, high-quality supervision, and instruction tuning, achieving competitive performance with more structured descriptions.

Contribution

The paper introduces VisionPangu, a novel compact multimodal model that improves detailed image captioning through efficient alignment and high-quality supervision, without large-scale architectures.

Findings

Achieves competitive captioning performance with only 1.7B parameters.

Produces more structured and detailed image captions.

Effectively incorporates dense human-authored descriptions for semantic richness.

Abstract

Large Multimodal Models (LMMs) have achieved strong performance in vision-language understanding, yet many existing approaches rely on large-scale architectures and coarse supervision, which limits their ability to generate detailed image captions. In this work, we present VisionPangu, a compact 1.7B-parameter multimodal model designed to improve detailed image captioning through efficient multimodal alignment and high-quality supervision. Our model combines an InternVL-derived vision encoder with the OpenPangu-Embedded language backbone via a lightweight MLP projector and adopts an instruction-tuning pipeline inspired by LLaVA. By incorporating dense human-authored descriptions from the DOCCI dataset, VisionPangu improves semantic coherence and descriptive richness without relying on aggressive model scaling. Experimental results demonstrate that compact multimodal models can achieve competitive performance while producing more structured and detailed captions. The code and model weights will be publicly available at https://www.modelscope.cn/models/asdfgh007/visionpangu.