VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models

TL;DR

VARCO-VISION is a new Korean-English vision-language model that effectively learns bilingual visual and linguistic information, demonstrating strong performance across diverse tasks and providing new datasets for evaluation.

Contribution

We introduce VARCO-VISION, an open-source bilingual VLM with a novel training strategy that preserves knowledge and expands capabilities, along with new Korean evaluation datasets.

Findings

Outperforms similar-sized models in bilingual image-text tasks

Capable of grounding, referring, and OCR functions

Provides new benchmarks for Korean vision-language understanding

Abstract

In this paper, we introduce an open-source Korean-English vision-language model (VLM), VARCO-VISION. We incorporate a step-by-step training strategy that allows a model learn both linguistic and visual information while preserving the backbone model's knowledge. Our model demonstrates outstanding performance in diverse settings requiring bilingual image-text understanding and generation abilities compared to models of similar size. VARCO-VISION is also capable of grounding, referring, and OCR, expanding its usage and potential applications for real-world scenarios. In addition to the model, we release five Korean evaluation datasets, including four closed-set and one openset benchmarks. We anticipate that our milestone will broaden the opportunities for AI researchers aiming to train VLMs. VARCO-VISION is available at https://huggingface.co/NCSOFT/VARCO-VISION-14B.