Small Language Model Meets with Reinforced Vision Vocabulary

TL;DR

This paper introduces Vary-toy, a small vision-language model with an improved vocabulary that performs well on various visual question answering tasks, making advanced LVLM features accessible on limited hardware.

Contribution

It presents a compact LVLM with an enhanced vision vocabulary, utilizing positive samples for better visual encoding, enabling high performance on multiple benchmarks.

Findings

Achieves 65.6% ANLS on DocVQA

Attains 59.1% accuracy on ChartQA

Reaches 88.1% accuracy on RefCOCO

Abstract

Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI community. However, the relatively large number of parameters (more than 7B) of popular LVLMs makes it difficult to train and deploy on consumer GPUs, discouraging many researchers with limited resources. Imagine how cool it would be to experience all the features of current LVLMs on an old GTX1080ti (our only game card). Accordingly, we present Vary-toy in this report, a small-size Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we introduce an improved vision vocabulary, allowing the model to not only possess all features of Vary but also gather more generality. Specifically, we replace negative samples of natural images with positive sample data driven by object detection in the procedure of generating vision vocabulary, more sufficiently utilizing the capacity of the vocabulary network and enabling it to efficiently encode visual information corresponding to natural objects. For experiments, Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1% accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on the homepage.