MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning

TL;DR

This paper introduces a lightweight image captioning model that replaces large language models with a compact one, combined with a self-refinement framework inspired by human visual perception, achieving competitive performance with reduced computational cost.

Contribution

The paper presents a novel multimodal self-refinement framework for lightweight image captioning, demonstrating that a small 125M-parameter model can match larger models' performance.

Findings

Compact model achieves comparable performance to large MLLMs.

Self-refinement improves captioning reliability and detail.

Model extends effectively to long-range video question answering.

Abstract

Systems such as video chatbots and navigation robots often depend on streaming image captioning to interpret visual inputs. Existing approaches typically employ large multimodal language models (MLLMs) for this purpose, but their substantial computational cost hinders practical application. This limitation motivates our development of a lightweight captioning model. Our investigation begins by replacing the large-scale language component in MLLMs with a compact 125M-parameter model. Surprisingly, this compact model, despite a 93x reduction in size, achieves comparable performance to MLLMs, suggesting that factual image captioning does not significantly require the complex reasoning abilities of LLMs. Despite this promising result, our lightweight model still lacks reliability. To address this, we draw inspiration from the human visual process: perceiving a global and coarse understanding of the scene before attending to finer details. Accordingly, we propose a multimodal self-refinement framework that guides the model to utilize features from salient regions, identified by referencing the previous coarse caption, and to produce a refined description. Experimental results demonstrate the superiority of our model in both single-sentence and detailed captioning, extending even to long-range video QA tasks.

Tables

Table 1. Table 1 : Available memory on edge devices and GPU usage of recent MLLMs by parameter size under FP16 precision.

Edge devices		LLaVA-1.5 7B,	LLaVA-NeXT 34B,	LLaVA-OA 72B,
iPhone 16	Galaxy S25	mPLUG-Owl3 8B	InternVL 40B	Qwen2-VL 72B
8GB	12GB	16GB $+$	68GB $+$	140GB $+$

Table 2. Table 2 : Comparison of captioning performance. We evaluate our model and existing captioning models. Despite not introducing any newly proposed methods, our model achieves strong performance. Here, ‘G’ represents a MLLM generalist, while ‘S’ denotes the small captioning model. ‘*’ indicates models that are fine-tuned in this study, as they were not trained for detailed captioning tasks.

MS COCO [mscoco]
	model	venue	# data	# params	B@4 [papineni2002bleu]	MET [meteor]	CIDEr [vedantam2015cider]	BERT [bertscore]	CLAIR [clair]	GPT [mllmjudge]
G	InstructBLIP [dai2023instructblip]	NeurIPS23	130M>	8.2B	38.0	29.4	127.8	69.1	-	-
	Unified-IOXL [lu2023unifiedio]	ICLR23	130M	7.3B	37.0	29.5	123.6	68.2	-	-
	Shikra [chen2023shikra]	arXiv23	-	7.2B	-	-	117.5	-	-	-
	Qwen-VL [bai2023qwenvl]	arXiv23	1.5B	9.6B	39.1	30.1	131.9	69.8	77.8$\pm$3.4	2.89$\pm$0.11
	LLaVA-1.5 [llava15]	CVPR24	1M	7.3B	39.4	29.5	133.7	69.4	78.1$\pm$3.8	2.93$\pm$0.10
	Cambrian [cambrian]	NeurIPS24	2M	10.5B	40.1	30.9	137.5	-	78.2$\pm$3.2	3.02$\pm$0.13
	I-Tuning [luo2023ituningtuningfrozenlanguage]	ICASSP23	0.5M	250M	35.5	28.8	120.0	-	-	2.50$\pm$0.10
	CapPa [tschannen2023image]	NeurIPS23	1B	650M	-	-	125.8	-	-	2.67$\pm$0.08
	LocCa [wan2024locca]	NeurIPS24	1B	600M	-	-	127.1	-	-	2.66$\pm$0.11
	SmallCap [ramos2023smallcap]	CVPR23	0.5M	450M	37.6	28.7	122.7	67.2	73.7$\pm$3.9	2.46$\pm$0.10
	Tag2Text [huang2023tag2text]	ICLR24	4M	900M	38.4	30.0	128.7	69.3	76.1$\pm$3.1	2.78$\pm$0.08
	ViPCap [vipcap]	AAAI25	4M	225M	37.7	28.6	122.9	-	-	-
S	Ours		1M	450M	39.4	30.3	129.6	69.4	76.3$\pm$3.0	2.74$\pm$0.06
ShareGPT4V [chen2024sharegpt4v] & DCI [dci]
	model	venue	# data	# params	B@4 [papineni2002bleu]	CIDEr [vedantam2015cider]	BERT [bertscore]	CAPT [dong2024benchmarking]	CLAIR [clair]	GPT [mllmjudge]
G	Qwen-VL [bai2023qwenvl]	arXiv23	1.5B	9.6B	10.8	35.6	37.2	48.4	57.5$\pm$3.2	3.05$\pm$0.08
	LLaVA-1.5 [llava15]	CVPR24	1M	7.3B	10.6	36.1	36.6	48.6	$\pm$	$\pm$
	EyesWideShut [eyeswideshut]	CVPR24	1M	7.6B	11.6	36.5	37.2	49.0	-	-
	Cambrian [cambrian]	NeurIPS24	2M	10.5B	13.3	38.7	38.1	50.1	58.2$\pm$3.1	3.03$\pm$0.09
	SmallCap* [ramos2023smallcap]	CVPR23	0.5M	450M	14.5	20.1	28.9	23.3	-	-
	Tag2Text* [huang2023tag2text]	ICLR24	4M	900M	17.8	32.5	40.7	40.1	54.2$\pm$3.1	2.72$\pm$0.14
S	Ours		1M	450M	18.0	40.5	43.1	45.9	54.6$\pm$3.4	2.74$\pm$0.12
GLaMM [rasheed2024glamm]
	model	venue	# data	# params	B@4 [papineni2002bleu]	CIDEr [vedantam2015cider]	BERT [bertscore]	CAPT [dong2024benchmarking]	CLAIR [clair]	GPT [mllmjudge]
G	LLaVA-1.5 [llava15]	CVPR24	1M	7.3B	8.8	23.4	35.1	40.0	53.8$\pm$4.0	3.02$\pm$0.10
S	Ours		1M	450M	16.5	29.1	38.7	42.0	51.8$\pm$4.1	2.64$\pm$0.09

Table 3. Table 3 : Quantitative results of MM-SeR. The results demonstrate the effectiveness of extending single-pass captioning with a self-refinement stage. Refinement relies on two key inputs: ① the initially generated caption and ② multi-layer features from the vision encoder. MM-SeR yields consistent performance gains on single-sentence and detailed captioning tasks.

MS COCO [mscoco]
model	#params	B@4 [papineni2002bleu]	gain	CIDEr [vedantam2015cider]	gain	CLAIR [clair]	gain	GPT [mllmjudge]	gain
LLaVA-1.5 [llava15]	7.3B	39.4	-	133.7	-	78.1$\pm$3.8	-	2.93$\pm$0.10	-
Our model	450M	39.4	-	129.6	-	76.3$\pm$3.0	-	2.74$\pm$0.06	-
+ MM-SeR with +	500M	39.9	+0.5	133.5	+3.9	77.6$\pm$2.9	+1.3	2.82$\pm$0.09	+0.08
+ MM-SeR with	500M	39.6	+0.2	131.9	+2.3	-	-	-	-
+ MM-SeR with	500M	39.6	+0.2	132.3	+2.7	-	-	-	-
Single pass with	450M	39.7	+0.3	130.6	+1.0	-	-	-	-
ShareGPT4V [chen2024sharegpt4v] & DCI [dci]
model	#params	CIDEr [vedantam2015cider]	gain	CAPT [dong2024benchmarking]	gain	CLAIR [clair]	gain	GPT [mllmjudge]	gain
Cambrian [cambrian]	10.5B	38.7	-	50.1	-	58.2$\pm$3.1	-	3.00$\pm$0.10	-
Our model	450M	40.5	-	45.9	-	54.6$\pm$3.4	-	2.74$\pm$0.12	-
+ MM-SeR with +	500M	43.6	+3.1	48.4	+2.5	57.7$\pm$3.0	+3.1	3.02$\pm$0.12	+0.28
+ MM-SeR with	500M	42.8	+2.3	47.1	+1.2	55.8$\pm$3.1	+1.2	2.78$\pm$0.11	+0.04
+ MM-SeR with	500M	43.1	+2.6	47.6	+1.7	56.8$\pm$3.4	+2.2	2.88$\pm$0.12	+0.14
Single pass with	450M	42.5	+2.0	46.5	+0.6	56.3$\pm$2.9	+1.7	2.90$\pm$0.11	+0.16
GLaMM [rasheed2024glamm]
model	#params	CIDEr [vedantam2015cider]	gain	CAPT [dong2024benchmarking]	gain	CLAIR [clair]	gain	GPT [mllmjudge]	gain
LLaVA-1.5 [llava15]	7.3B	23.4	-	40.0	-	53.8$\pm$4.0	-	3.02$\pm$0.10	-
Our model	450M	29.1	-	42.0	-	51.8$\pm$4.1	-	2.64$\pm$0.09	-
+ MM-SeR with +	500M	30.4	+1.3	42.8	+0.8	53.4$\pm$3.8	+1.6	2.88$\pm$0.11	+0.24

Table 4. Table 4 : Evaluation on Long-Range VideoQA. We follow the baseline setup of LLoVi  [ zhang2023simple ] , replacing the captioner with our model or MLLMs. Frame-level captions are aggregated and provided to Qwen2.5-14B for answering video-related questions. Our specialist delivers competitive accuracy with significantly fewer parameters, demonstrating its applicability even in long-range video understanding.

LLM	Captioner	LLaVA-1.0 [llava]	BLIP-2 [li2023blip2]	LLavA-1.5 [llava15]	SmallCap* [ramos2023smallcap]	Tag2text* [huang2023tag2text]	Our specialist	+ MM-SeR
Qwen2.5 14B [yang2024qwen2]	#params	7.3B	7.4B	7.3B	450M	900M	450M	500M
	accuracy	47.6	50.6	51.1	41.8	47.1	49.3	50.8
	time	29m 20s	29m 44s	29m 20s	4m 45s	7m 14s	4m 53s	5m 10s

Table 5. Table 5 : Inference time required to generate captions for 100 streaming images. Given the captioning performance in Table  3 , our model demonstrates remarkable efficiency.

	LLaVA-1.5 [llava15]	Ours	+ MM-SeR
time	274.49 s	5.55 s (97.97%$\downarrow$)	7.44 s (97.28%$\downarrow$)

Table 6. Table 6 : MM-SeR on larger language models. We evaluate detailed captioning performance on ShareGPT4V  [ chen2024sharegpt4v ] and DCI  [ dci ] using captioning specialists built from OPT-1.3B and LLaMA-2-7B. Across both models, MM-SeR consistently improves performance over their respective baselines, demonstrating that the refinement framework generalizes beyond lightweight LMs.

				CIDEr [vedantam2015cider]		CAPT [dong2024benchmarking]
Language model	vision encoder	LoRA [hu2022lora]	total # params	initial gen.	+ MM-SeR	initial gen.	+ MM-SeR
OPT-1.3B [zhang2022opt]	CLIP ViT-L	$\times$	1.6B	50.2	53.1 (+2.9)	49.0	50.2 (+1.2)
LLaMA-2-7B [touvron2023llama]	CLIP ViT-L	✓	7.3B	57.3	61.7 (+4.4)	52.7	53.6 (+0.9)

Table 7. Table 7 : Effect of iterative refinement. We compare initial captions and one to three refinement steps on the detailed captioning benchmarks ShareGPT4V  [ chen2024sharegpt4v ] and DCI  [ dci ] .

Language model	Stage	CAPT [dong2024benchmarking]	GPT [mllmjudge]
OPT-125M [zhang2022opt]	initial caption	45.9	2.74$\pm$0.12
	refinement $\times$1	48.4	3.02$\pm$0.12
	refinement $\times$2	47.8	3.01$\pm$0.10
	refinement $\times$3	48.1	3.03$\pm$0.10
OPT-1.3B [zhang2022opt]	initial caption	49.0	3.04$\pm$0.10
	refinement $\times$1	50.2	3.14$\pm$0.11
	refinement $\times$2	50.5	3.16$\pm$0.10
	refinement $\times$3	50.3	3.20$\pm$0.10

Table 8. Table 8 : Evaluation results across different hardware resources and datasets.

Resource (RAM)	Data	Model	Mem.	Inf. time	Power.	B@4 [papineni2002bleu]	CIDEr [vedantam2015cider]	CLAIR [clair]	GPT [mllmjudge]
Jetson Nano (4G)	All	LLaVA-1.5-7B	out-of-memory	N/A	N/A	N/A	N/A	N/A	N/A
RTX 3090 (24G)	MS COCO [mscoco]	Ours-500M	3.2G	5s	230 W	39.5	133.8	78.6$\pm$2.9	2.83$\pm$0.06
Jetson Nano (4G)			2.6G	20s	13 W	39.5	133.8	78.6$\pm$2.9	2.73$\pm$0.09
RTX 3090 (24G)	ShareGPT4V [chen2024sharegpt4v]	Ours-500M	3.2G	5s	230 W	22	43.2	57.9$\pm$3.0	3.01$\pm$0.10
Jetson Nano (4G)	& DCI [dci]		2.7G	21s	13 W	22.2	42.9	57.4$\pm$2.9	3.02$\pm$0.11

Table 9. Table 9 : Performance comparison of our lightweight captioner under different learning strategies: origin, more data, and distillation. Distillation from a strong teacher, especially when combined with MM-SeR, leads to the best results.

ShareGPT4V [chen2024sharegpt4v] & DCI [dci]
Metric	origin	+SeR	gain	more data	distillation	+SeR	gain
CIDEr [vedantam2015cider]	40.5	43.6	+3.1	36.8	42.6	43.6	+2.0
CAPT [dong2024benchmarking]	45.9	48.4	+2.5	43.2	46.5	47.4	+0.9

Table 10. Table 10 : For the captioning specialist, we evaluate different vision encoders and the corresponding selected layer indices.

Vision encoder	#params	indices of selected layers	CIDEr [vedantam2015cider]	CLAIR [clair]	GPT [mllmjudge]
CLIP [CLIP]	300M	{23}	42.8	55.8	2.78
	300M	{13, 18, 23}	43.3	57.7	3.02
CLIP [CLIP]+DINOv2 [oquab2024dinov]	600M	{23} + {23}	42.9	57.3	3.02
SigLIPv2 [siglip2]	300M	{23}	43.0	56.2	2.88
	300M	{15, 23}	45.9	57.7	3.05
	300M	{13, 18, 23}	45.5	58.2	3.07
DINOv2 [oquab2024dinov]	300M	{23}	32.8	48.6	2.55
	300M	{13, 18, 23}	33.0	50.1	2.66

Table 11. Table 11 : Token-level differences between pseudo-initial and ground-truth captions.

pseudo-initial caption $\hat{c}$	different tokens $E=t\mid \hat{c}\left(t\right)\ne c\left(t\right)$ in Section 4.2
A woman in a room with two dogs	two / dogs
A cat sitting on a chair in front of the window.	sitting / on a chair / in front of the window

Table 12. Table 12 : Ablation results evaluating connector types, refinement configurations, and ViT layer selections in SeR-Connector. Architectural variants are illustrated in Figure  9 (b) and (c).

(I) connectors for initial generation
	(b)	(c)	(d)	(e)	(f)
129.6 ✓	130.6	125.9	127.7	122.9	130.9

Table 13. Table 13 : Detailed captioning results on ShareGPT4V  [ chen2024sharegpt4v ] & DCI  [ dci ] with fine-tuning on four types of pseudo-initial captions. The comparison highlights how different levels of overlap or inclusion of GT captions affect model performance.

Detailed captioning	CIDEr [vedantam2015cider]	gain	CAPT [dong2024benchmarking]	gain	GPT [mllmjudge]	gain
Our specialist fine-tuned via Stage 1	40.5	-	45.9	-	2.74$\pm$0.12	-
finetuned w/Data1	42.5	+2.0	47.2	+1.3	2.86$\pm$0.11	+0.12
finetuned w/Data2	41	+0.5	46.6	+0.7	2.76$\pm$0.12	+0.02
finetuned w/Data3	43.6	+3.1	48.4	+2.5	3.02$\pm$0.12	+0.28
finetuned w/Data4	43.8	+3.3	48.2	+2.3	3.04$\pm$0.09	+0.30

Table 14. Table 14 : Few-shot examples of GT captions and corresponding pseudo-initial captions.

GT caption	pseudo-initial caption
A woman in a room with a cat	A woman in a room with two cats
	A woman in a room with a dog
	A woman in a room with a cat
A multicolored motorcycle rests outside of a sheep farm	A multicolored bicycle rests inside a sheep farm
	A bright red motorcycle rests outside of a sheep farm
	A multicolored motorcycle races around a sheep farm

Table 15. Table 15 : Statistics of datasets used for fine-tuning. Notably, the number of images in fine-tuning stages 1 and 2 are identical .

Dataset	#images	#GT captions per image	#pairs for fine-tuning stage 1	#pseudo-initial captions per GT caption	#pairs for fine-tuning stage 2
COCO [mscoco]	113K	5	565K	3	1.6M
ShareGPT4V [chen2024sharegpt4v]	100K	5	500K	3	1.5M
DCI [dci]	7.4K	10	74K	3	0.2M
GLaMM [rasheed2024glamm]	550K	1	550K	3	1.6M

Table 16. Table 16 : Hyperparameters used for model training. The settings to the left and right of the / correspond to those used in Section  4 and Section  3 , respectively.

	Pretraining	Fine-tuning
Dataset	LCS-558K [llava15]	MS COCO [mscoco] or SharedGPT [chen2024sharegpt4v] + DCI [dci]
Adapter	4-layer MLP with GELU	4-layer MLP with GELU / Deeplens
Trainable	Adapter layers only	Adapters + Language Model
Training Epochs	1	10 / 2
Learning Rate	$1\times {10}^{-4}$	$2\times {10}^{-5}$
Weight Decay	0	0
Warm-up Ratio	0.03	0.03
Learning Rate Scheduler	Cosine decay	Cosine decay

Table 17. Table 17 : Performance of LLaVA-1.5 under different prompt instructions.

LLaVA-1.5	CIDEr [vedantam2015cider]	BERTScore [bertscore]	CAPT [dong2024benchmarking]
“Describe the photo within 55 words”	36.1	36.6	48.6
“Describe the photo in detail”	12.8	17.6	40.6

Equations

$$\mathcal{L}(\theta )=-\mathbb{E}\left[\sum _j\log {\pi }_{\theta }\right(c_{k,j}\mid i_k,{\hat{c}}_k,c_{k,<j}\left)\right],$$

Full text

1 Introduction

Recent progress in image captioning has been driven by the remarkable capability of Multimodal Large Language models (MLLMs) [llava, li2023blip2]. Building on these advances, image captioning has become a crucial component in various applications. For example, video-based chatbot systems utilize frame-wise caption generation for temporal understanding [wang2023internvid, moment-gpt, videotree], while navigation robots construct graph-structured scene descriptions to operate in complex environments [xie2024embodied, jiang2024roboexp]. Despite this progress, the substantial computational demands of MLLMs [jin2024efficient] pose a significant barrier to their practical deployment.

In many industrial systems, detection and segmentation models with fewer than 500M parameters are commonly deployed. This gap led us to ask whether captioning is truly so difficult that it must rely on MLLMs. To address this question, we design a lightweight captioning model that follows the architectural pattern of recent MLLMs and evaluate it on several captioning tasks. Specifically, we implement our model within the LLaVA framework [llava] by replacing LLaMA-7B [touvron2023llama] with OPT-125M [zhang2022opt], a language model that is 56× smaller in parameter count. Surprisingly, the resulting model not only competes with MLLMs on the standard MS COCO [mscoco] benchmark but also performs strongly on more challenging detailed captioning tasks [dci], while outperforming existing small captioning models [ramos2023smallcap, huang2023tag2text]. These results reveal the insight that the complex capabilities of LLMs are less critical for tasks that focus on enumerating factual visual details. These results also suggest that applying modern multimodal architectural designs to small captioning models is required to unlock their potential for deployment in real applications.

Despite the promising performance, the resulting model still exhibits a reliability gap compared to MLLMs. We attribute this gap primarily to visual blindness; prior work [cambrian, eyeswideshut] has highlighted that MLLMs often suffer from ambiguous visual features, which limits their ability to capture fine-grained details. This finding necessitates a method to supply the model with clearer and more informative visual inputs. We address this limitation by adopting a process similar to human visual perception, which begins with a global understanding of the scene before attending to local details. This multi-stage human approach contrasts with conventional captioning models, which typically operate in a single pass, processing the image only once to produce a description. To this end, we design a new framework, Multimodal Self-Refinement (MM-SeR), which enables the lightweight model to emulate this multi-stage human process. Specifically, MM-SeR enables the model to first generate an initial caption. The model then leverages this caption to guide its attention toward salient visual regions and extract richer information from the multi-layer features of the vision encoder. This refinement process allows the model to produce a more accurate final description.

In our experiments, we evaluate the MM-SeR framework on diverse benchmarks, extending beyond standard MS COCO [mscoco] to include detailed captioning and long-form VideoQA tasks. We compare our model with MM-SeR against both existing small captioners and MLLMs, demonstrating comparable or even superior performance. Particularly in the challenging long-form VideoQA setting, our lightweight model not only outperforms other small specialists but also approaches the accuracy of MLLM generalists. This is accomplished while utilizing 93% fewer parameters and achieving 82% shorter inference time compared to the MLLMs. These results indicate that the proposed baseline and refinement framework offer a practical route toward lightweight captioning suitable for resource-constrained and on-device applications.

2 Motivation & Scope

Image Captioning as Foundational Technology. Image captioning converts visual content into natural language descriptions [mokady2021clipcap, hu2022scaling]. Beyond being a standalone task, it serves as a core component in various vision–language applications. In video-grounded chatbot systems [sarto2023positive, wang2024internvideo2, zhao2023learning, moment-gpt], captioning generates textual representations of multiple frames, which are then integrated into LLMs as prompts to guide instruction following. Similarly, exploration robots [xie2024embodied, jiang2024roboexp] operating in disaster environments rely on captioning to encode observed scenes, enabling navigation and human interaction. In this work, we study image captioning as a key enabler for these applications.

Real-World Deployment Challenges. Recent studies on the above applications often employ open-source MLLMs [wang2023internvid] or cloud-based APIs (e.g., OpenAI API) [xie2024embodied] as image captioners. In practice, these approaches face two major limitations: (1) open-source MLLMs demand computational resources beyond the capacity of edge devices [cai2020tinytl, song2023ecotta] as shown in Table 1, and (2) cloud-based APIs rely on stable network connectivity, which may be unavailable in disaster environments. Moreover, repeated captioning across multiple scenes [zhang2023simple] further increases the computational burden, making real-world deployment more difficult.

3 Exploring Lightweight Captioning

Motivated by the challenges discussed above, we explore a lightweight captioning model and examine its performance through extensive evaluation.

Model construction. To construct a lightweight captioning model, we aim to reduce the dependence on LLMs, which account for most of the computational cost in MLLMs (e.g., 96% in LLaVA-7B [llava] arises from LLaMA [touvron2023llama]). Accordingly, we replace the LLaMA-7B in LLaVA-1.5 with OPT-125M [zhang2022opt], a 56× smaller language model.

Experimental details. We adopt the publicly available LLaVA-1.5 [llava15] codebase. Except for replacing the language model, all training configurations remain consistent with the original setup, including batch size and learning rate. More implementation details are provided in Section I.1. and our source code111https://github.com/junha1125/Lightweight-Captioner. To train our model, we first pretrain the multimodal connector on the Caption Concept-balanced 558K dataset [llava15], followed by fine-tuning on task-specific datasets such as MS COCO [mscoco], DCI [dci], and ShareGPT4V [chen2024sharegpt4v]. We evaluate our model using standard metrics, BLEU [papineni2002bleu], CIDEr [vedantam2015cider], and BERTScore [bertscore], as well as MLLM-as-a-Judge [mllmjudge, clair] with GPT-4o-mini [achiam2023gpt].

Generalist vs. Specialist. Following prior studies [ye2024mplug, mome, wu2024visionllm], we define generalists as MLLMs trained on diverse datasets for broad objectives and evaluated in a zero-shot manner using publicly released weights. In contrast, specialists refer to compact captioning models such as SmallCap [ramos2023smallcap] and Tag2Text [huang2023tag2text], which are trained and optimized exclusively on task-specific data.

3.1 Results on Single Sentence Captioning

We use the MS COCO Captions dataset [mscoco], the most widely adopted benchmark for image captioning, where each caption consists of a single sentence, with an average length of about ten words. As shown in Table 2, our model outperforms previous small-scale captioning methods [huang2023tag2text, wan2024locca] that contain fewer than 1B parameters. In particular, it achieves a CIDEr score 6.9 points higher than SmallCap [ramos2023smallcap], which also employs OPT-125M as its language backbone. Despite the absence of any newly introduced techniques, the model demonstrates unexpectedly strong performance, which we further analyze in Section E.1. Moreover, when compared with generalist MLLMs [chen2023pali, bai2023qwenvl], our model attains comparable results while requiring far less computational resources.

3.2 Results on Detailed Captioning

For this experiment, we fine-tune our model on the DCI [dci] and ShareGPT4V [chen2024sharegpt4v] datasets, and additionally include GLaMM [rasheed2024glamm]. We initially expected the lightweight model to underperform on this task, as OPT-125M offers limited language capacity due to its small parameter size, as shown in Table 2. However, the results in Table 2 and Figure 3 show that our model achieves unexpectedly strong performance on detailed captioning, contrary to our assumption.

3.3 Key Insight

Although MLLMs depend on the advanced reasoning capabilities of large language models for tasks such as visual question answering and instruction following, our experiments reveal a contrasting trend in factual image captioning. We find that accurate caption generation can be achieved without such reasoning-intensive processes, as a lightweight 125M-parameter language model attains comparable performance to full-scale MLLMs. This result implies that image captioning primarily relies on perceptual grounding rather than abstract reasoning, suggesting that a compact model can serve as an efficient and practical alternative for captioning-based applications.

4 MM-SeR: Multimodal Self-Refinement

Despite the impressive performance of our lightweight captioner, we still observe a capacity gap relative to large-scale MLLMs in certain evaluations. To address this limitation, we introduce a new framework, Multimodal Self-Refinement (MM-SeR), illustrated in Figure 1, which improves caption quality through self-guided refinement. First, we draw from the human description process: forming an initial global understanding before attending to salient details. Following this principle, MM-SeR adopts a multi-stage generation procedure. The model first produces an initial caption capturing the overall scene, and then uses this possibly coarse output to guide the extraction of clearer and more informative visual features, which support subsequent refinement.

4.1 Proposed Framework

As illustrated in Figure 4, our framework extends the conventional single-pass captioning approach by incorporating a self-refinement stage. Although the same language model is used for both the initial caption and the refinement step, the refinement process operates with a dedicated connector (i.e., SeR-Connector). This connector is specifically tasked with integrating the following inputs: the previously generated caption and multi-layer features from the vision encoder. These inputs each serve a distinct purpose, as detailed below.

Looking at what matters. When asked to refine a caption such as “A cat relaxing on a brown chair,” humans naturally attend to the key elements referenced in the text, like the cat and the chair. Following this intuition, we feed the initial caption into the SeR-Connector and Language models, allowing the modules to identify visually relevant regions and direct their attention toward them during refinement.

Looking in detail. Coarse visual features often limit fine-grained description quality. While some prior studies [cambrian, eyeswideshut, fromcliptodino] address this by adding auxiliary vision encoders, this strategy increases model size; for instance, Interleaved MoF [eyeswideshut] introduces DINOv2 [oquab2024dinov], adding 300M parameters—a 66.7% increase relative to our 450M-parameter model. Instead of expanding the architecture, we aim to maximize the utility of the existing encoder by leveraging multi-layer features, which provide richer and more detailed representations. Although earlier works [fpn] have explored multi-layer features, our approach uses them specifically to supply finer-grained visual cues for the refinement stage.

Novelty of MM-SeR. This refinement paradigm parallels the concept of self-refinement explored in LLM research [selfrefine, le2022coderl, pan2023automatically]. By extending this idea to the multimodal domain, our framework integrates visual evidence directly into the refinement, to the best of our knowledge, the first attempt to realize self-refinement within multimodal models.

4.2 Training Strategy

For the proposed framework to operate effectively, the model needs to be trained to produce reliable initial captions and to extract the refined visual features described in Figure 4. To achieve this, we adopt a two-stage training strategy.

In the first stage, the model is trained to generate the initial caption following the standard LLaVA procedure [llava15]. In the second stage, the model learns how to perform refinement. Let the training set be $X=\{x_{1},x_{2},\dots,x_{N}\}$, where each $x_{k}=(i_{k},c_{k})$ contains an image and its ground-truth caption. A straightforward strategy might treat the model’s momentary first-pass caption as the refinement input and the corresponding ground-truth caption $c_{k}$ as the target. However, this setup often produces pairs with little semantic alignment. For example, if the initial caption is “a table in front of a window” while the $c_{k}$ is “a cat sitting on a table,” the two descriptions offer no meaningful basis for progressive refinement. Training on such mismatched pairs would likely lead the model to disregard the initial caption and simply regenerate a new one, rather than learn how to refine it. Further discussion is in Section E.5.

To address this issue, we generate pseudo-initial captions $\hat{c}_{k}$ by prompting GPT-4o-mini [achiam2023gpt] to introduce small perturbations to entities, attributes, or relations in the $c_{k}$. For example, given the $c_{k}$ “a cat sitting on a chair,” the pseudo-initial $\hat{c}_{k}$ version may become “a dog sitting on a chair.” During training, the SeR-Connector receives multi-layer visual features together with the $\hat{c}_{k}$ and learns to extract features that are more informative for caption refinement. The language model takes the visual embeddings and $\hat{c}_{k}$ as an additional textual prompt and predicts a refined caption, which is supervised to match $c_{k}$.

Rationale for refinement training. Let $\pi_{\theta}$ denote the language model. Each pseudo‑initial caption $\hat{c}_{k}$ deviates from the ground truth $c_{k}$ at only a few token positions $E_{k}=\{t\mid\hat{c}_{k,t}\neq c_{k,t}\}$. Under the sequence‑level objective

[TABLE]

gradients are likely to be primarily concentrated on tokens in $E_{k}$, leading to a form of targeted optimization, in which the model retains the correct parts of $\hat{c}_{k}$ while rewriting only the erroneous ones. Writing $\Delta_{k}(\theta)=\log\pi_{\theta}(c_{k}\mid i_{k},\hat{c}_{k})-\log\pi_{\theta}(\hat{c}_{k}\mid i_{k},\hat{c}_{k}),$ we have $\mathcal{L}(\theta)\propto-\mathbb{E}[\Delta_{k}]$; hence, minimizing $\mathcal{L}$ is equivalent to maximizing the expected margin $\Delta_{k}$, thereby directly increasing the likelihood of the refined caption relative to its flawed precursor. In effect, each gradient step encourages SeR-Connector to focus on visual regions likely responsible for errors in the initial caption and guides the language model to better interpret these refined features, resulting in more accurate captions. This targeted optimization resembles the philosophy of Direct Preference Optimization (DPO) [dpo], if we consider $c_{k}$ and $\hat{c}_{k}$ as preferred and less-preferred responses. In contrast to DPO, which treats both responses symmetrically during optimization, our method offers a new perspective by assigning them distinct roles as input and target.

5 Experiments

5.1 Implementation Details

We train the framework in two stages. The first stage runs for 10 epochs to learn initial caption generation, and the second stage runs for 2 epochs to train the refinement process. Both stages use a batch size of 256 and a learning rate of $2*10^{-5}$. All experiments are conducted on two NVIDIA A6000 GPUs. Additional training settings and hyperparameters are described in Section I.1.

SeR-Connector. The SeR-Connector is simply implemented with a set of Transformer blocks [bert]. As shown in Figure 9, it receives two inputs: ① the previously generated caption, encoded as token embeddings, and ② multi-layer features from the vision encoder. From the vision encoder, we collect $N$ visual token vectors of dimension $d$ from $m$ selected layers. The features are concatenated along the channel dimension, yielding a representation of size $N*(md)$ that preserves hierarchical visual information. The resulting output is then forwarded to the language model for refinement. The ablation study is discussed in Section D.1.

5.2 Results

Effect of MM-SeR. The results in Table 3 and Figure 6 demonstrate the effectiveness of our framework in improving caption quality. We present ablation studies on the two key inputs of our framework, ① the initial captions and ② the multi-layer visual features, to examine their individual contributions. The refinement stage improves the initial captions by +3.9 and +2.5 CIDEr points for the single-sentence and detailed captioning tasks, respectively. We further evaluate the use of ② without applying the refinement stage, observing that single-pass captioning provides only limited gains, indicating the necessity of the step. In our framework, adding the refinement stage inevitably introduces additional overhead, including roughly 50M parameters for the SeR-Connector and one extra inference step with the language model. Nevertheless, as shown in Table 5, the computational overhead remains minimal compared with MLLMs.

Evaluation on long-range video question answering. Since captioning models serve as core components in downstream applications, as discussed in Section 2, we assess their utility in a practical setting by using our captioner as the captioning baseline for the long-range VideoQA task introduced in LLoVi [zhang2023simple]. (Setup:) For fair comparison, all small specialists [ramos2023smallcap, huang2023tag2text], including ours, are trained on ShareGPT4V [chen2024sharegpt4v] and DCI [dci]. Following [zhang2023simple], we adopt Qwen2.5-14B as the backbone LLM for answer generation across all captioners to isolate the effect of caption quality. Inference time is measured end-to-end, accounting for both caption generation and LLM reasoning under a 10-minute video scenario. (Results:) As shown in Table 4, our specialist achieves 49.3 accuracy, outperforming prior small captioners such as SmallCap (41.8) and Tag2Text (47.1). When equipped with MM-SeR, performance further increases to 50.8, approaching the best generalist pipeline LLaVA-1.5 (51.1) despite using over 14× fewer parameters. In terms of efficiency, our specialist requires only 4m 53s, and 5m 10s with MM-SeR, which is substantially faster than generalist MLLMs ($\approx$ 29m). These results indicate that our lightweight captioner, combined with MM-SeR, offers strong suitability for real-world applications.

Visual analysis of initial caption utilization in MM-SeR. A key input to our MM-SeR framework is the initial caption, which may reflect a rough and first-pass view of the image. To understand how this caption guides refinement, we examine the model’s visual attention during the generation process. (Setup:) We analyze the regions the model attends to when generating specific words and visualize attention maps using code adapted from API [yu2024attention]. Further implementation details are provided in Section I.3. (Results:) As shown in Figure 7, the single-pass captioner often distributes its attention broadly across the image, struggling to localize fine-grained regions. This behavior reflects the limitation of describing an image in a single glance, where the model attempts to process all information at once without focusing on details. In contrast, when the model performs the refinement step using the previously generated caption, the attention patterns become more concentrated on the relevant regions associated with each word. This suggests that the model leverages the initial caption as a guide, enabling it to “look at what matters” during refinement.

MM-SeRwith larger language models.

We extend MM-SeR to larger language models (LMs), including OPT-1.3B and LLaMA-2-7B, to examine whether the framework generalizes beyond lightweight models. To build captioning specialists, we first trained these LMs on the ShareGPT and DCI datasets. We then integrated MM-SeR and applied an additional refinement stage using the procedure described in Section 4.2. Our results in Table 6 show that MM-SeR provides consistent gains over these stronger baselines. Specifically, the framework improves CAPT [dong2024benchmarking] scores by 1.2 points for OPT-1.3B and 0.9 points for LLaMA-2-7B, indicating that MM-SeR remains effective when applied to larger LMs. Although larger LMs such as OPT-1.3B and LLaMA-2-7B offer stronger captioning capability as observed in prior scaling studies [kaplan2020scaling, wei2022emergent, hu2022scaling], their scale, approximately 10× and 56× larger than OPT-125M, can limit practical deployment. Developing captioning models that balance accuracy and efficiency, therefore, remains an important direction.

Iterative self-refinement.

We also examine whether MM-SeR benefits from iterative refinement rather than a single refinement pass. As shown in Table 7, applying multiple refinement steps to the smallest LM, OPT-125M, provides no measurable improvement. In contrast, two or three refinement iterations yield meaningful gains for OPT-1.3B. This gap suggests that smaller LMs lack the capacity required to leverage multi-step refinement, whereas larger LMs are able to utilize the additional refinement signal effectively. The trend aligns with recent findings in LLM-based self-refinement [selfrefine, ranaldi2024self, wu2024large, kumar2024training], which report increasing benefits as model capacity grows. Future work could explore scalable strategies, such as dynamically adjusting the iteration count, which remains a challenge in LLM refinement. We note that our core contribution is the demonstration that self-refinement can be effective paradigm for MLLMs.

Remarks. We also evaluate our model on a resource-constrained device, Jetson Nano, in Section B.2. The strategy for generating pseudo-initial captions is detailed in Section G.1, and we discuss the limitations of our specialist captioner and existing MLLMs in Section E.

6 Related Work

Multimodal Large Language Models (MLLMs) have attracted considerable research attention due to their versatile applications, such as chat-bots [wu2023multimodal]. Early approaches integrated contrastive image-language pretrained models [wang2022git] with powerful LLMs, enabling complex reasoning. The development of instruction-based datasets [llava] and innovative training strategies [li2022mplug, ye2024mplug] has further accelerated progress, substantially improving MLLM performance and broadening their capabilities. Despite these achievements, most MLLMs heavily depend on large-parameter LLMs, making deployment on memory-constrained devices unfeasible. This limitation will likely restrict access for a significant portion of users worldwide.

Image Captioning Models. Recent advances in image captioning have improved training efficiency and descriptive fidelity. Approaches such as CaMEL [barraco2022camel] and SmallCap [ramos2023smallcap] emphasize minimizing trainable parameters by leveraging mean-teacher distillation and employing retrieval augmentation, while Tag2Text [huang2023tag2text] and LoCCa [wan2024locca] introduce novel mechanisms such as dedicated tagging and location-aware refinement to improve caption quality. Unlike existing approaches that focus on reducing trainable parameters, or rely on single-pass inference—potentially missing crucial details—our method prioritizes inference efficiency, considering on-device operation and systematically addressing the limitations of single-pass generation.

Self-Refinement in LLMs. Humans often refine their writing through iterative review to enhance clarity and precision [selfrefine, dou2024stepcoder]. Recent research has applied this refinement concept to LLMs, introducing techniques. For instance, Self-Refine [le2022coderl, ranaldi2024self, pan2023automatically] enabled models to autonomously critique and iteratively enhance their outputs. Unlike such approaches limited to the LLM domain, our method introduces refinement in a multimodal context, guided by both language and vision.

7 Conclusion & Broader Impact

We introduced a lightweight captioning approach with multimodal self-refinement, motivated by the need for efficient visual understanding systems on edge devices. Our study began with an OPT-125M–based captioner, which surprisingly achieved performance comparable to large MLLMs despite a substantial reduction in parameters (93%$\downarrow$) and inference time (97%$\downarrow$). To further enhance this lightweight model, we proposed a multimodal self-refinement framework, the first refinement-based approach explored in the MLLM community. Through extensive experiments, we demonstrated that the proposed architecture and framework provide more accurate and informative captions. These improvements hold consistently across single-sentence captioning, detailed captioning, and practical downstream tasks such as long-range video question answering. We hope that the model explored in this work serves as a practical solution for on-device applications and that our findings on visual self-refinement inspire deeper investigation in future multimodal research.

Broader Impact. Future research may explore integrating external tools (e.g., zoom or crop, as in GPT-o3 [openai2024o3o4]), and designing a unified multimodal connector for both the initial and secondary glance. More research questions that aim to enhance both captioning performance and efficiency in real-world applications are provided in Section E.6.

Appendix

Appendix A Novelty of our framework MM-SeR

• •

What is it? Humans first take in the overall scene, then refine at specific regions to notice finer details. Our MM-SeR framework mimics this human tendency, allowing the captioning specialist to revise its output.

• •

Why novel? To the best of our knowledge, this is the first work to introduce a multimodal refinement method that jointly utilizes visual features and the model’s output.

• $\circ$

Lightweight captioning community: This field has been gradually declining since the remarkable capabilities of LLMs were discovered. In this work, we revisit the practically important yet underexplored topic of lightweight captioning models.

• $@itemii$

Unlike ours, some methods [luo2023semantic, ramos2023smallcap, wang2022text] utilize prior captions obtained through a heavy image-text retrieval process. For example, SmallCap performs similarity matching between a given image and 500,000 candidate captions at each iteration.

• $@itemii$

Unlike ours, several works [huang2023tag2text, lu2018neural] incorporate object detection or tagging procedures. However, despite these extra components, their captioning performance has been limited.

• $@itemii$

Unlike ours, which refines text output generated via a full forward pass, certain works [luo2023semantic, chen2022analog] adopt Diffusion Transformers and denoise latent text embeddings. Moreover, they provide little motivation or analysis as to why such a process is important from the perspective of utilizing visual cues more effectively.

• $\circ$

NLP community: Humans often refine their writing, and coders revise their code through iterative review. Recent research has applied this refinement/correctness concept to LLMs [selfrefine, ranaldi2024self, wu2024large, kumar2024training]. Unlike such approaches limited to the LLM domain, our method introduces refinement in a multimodal context.

• $\circ$

Multimodal LLMs community: Existing models [llava, bai2023qwenvl, zhu2025internvl3] heavily rely on the complex reasoning capabilities of LLMs and typically generate outputs in a single pass. We uniquely demonstrate the effectiveness of revisiting and refining its own textual outputs.

• $\circ$

Visual Blindness community: Several studies [cambrian, eyeswideshut] believe the visual encoder is a critical bottleneck, necessitating adapting multiple vision encoders. Since our research focuses on a lightweight model, we maximize the utility of the existing vision encoder through multi-level utilization.

Appendix B Additional Experiments

B.1 When GPT meets ‘self-refinement’

We additionally examine how OpenAI’s GPT models, which show strong captioning performance, behave under this refinement process. Although our framework includes two components, an previous caption and visual representations from SeR-connector, we apply only the former, since modifying the internal architecture of GPT models is not feasible. The results, shown in Figure 8, are obtained using GPT-4-turbo. As a commercial model with a large parameter budget, GPT-4-turbo often generates accurate initial captions, though occasional inaccuracies still appear. To assess whether refinement is beneficial, we instruct the model to perform self-refinement and observe that it can revise its caption by re-examining the provided image. OpenAI’s GPT-o3 reflects similar ideas through the introduction of Thinking with Images [openai2024o3o4]. However, this direction remains relatively unexplored in the research community, indicating the need for further investigation.

B.2 Evaluation on an Actual Edge Device

o highlight the practical value of our approach, we evaluate our lightweight captioner on edge devices, where deploying large-scale MLLMs is often infeasible. Our motivation stems from the observation that, although MLLMs are powerful, their computational demands limit use in resource-constrained environments. In contrast, lightweight captioners remain relatively unexplored despite their suitability for real-world applications. We show that such captioners can be effectively deployed on devices including an RTX 3090 and the Jetson Nano. We assess captioning performance on the MS COCO and ShareGPT4V&DCI datasets, and additionally measure inference time, memory usage, and power consumption. All models are executed for 100 iterations with a batch size of 1. As shown in Table 8, our method performs consistently across devices and, importantly, remains fully operational in settings where models like LLaVA-1.5 cannot run. These results support the deployability of our framework on edge hardware and highlight lightweight captioning as a promising direction for real-world assistive technologies.

B.3 Comparison on different learning strategies

We examine how our lightweight specialist performs under different learning strategies. The first strategy trains the model with maximal data coverage by combining COCO, ShareGPT, DCI, and GLaMM (i.e., more data). The second strategy applies distillation, training on captions generated by LLaVA-1.6-34B [llava-recap] (i.e., distillation). Our original approach trains the model solely on the target datasets (i.e., origin). The results in Table 9 show that the more data strategy yields limited improvement, likely due to weakened task alignment when mixing heterogeneous datasets. In contrast, the distillation strategy benefits from learning from a strong teacher and improves performance even in a single-pass setting. When combined with our refinement framework (i.e., MM-SeR), it produces further gains, suggesting that distillation and MM-SeRcomplement each other. Additionally, as noted in Section 4.2, our datasets resemble those used in Direct Preference Optimization (DPO). Exploring reinforcement-learning-based training such as DPO may offer another promising direction for improving lightweight captioners.

B.4 Efficacy with Other Vision Encoders

We examine whether our strategy of selecting multi-level features generalizes to vision encoders beyond CLIP [CLIP], which serves as our original setup. As shown in Table 10, the approach consistently improves performance across different encoders. Both SigLIPv2 [siglip2] and DINOv2 [oquab2024dinov] benefit from incorporating multi-level features, indicating that our method is not restricted to CLIP-based models. Interestingly, CLIP with multi-level features surpasses the combination of CLIP and DINOv2 in several metrics while maintaining better parameter efficiency. In contrast, DINOv2 alone delivers lower performance, likely due to weaker alignment with language features. For all experiments, we pair OPT-125M with each vision encoder and train the resulting models on the ShareGPT [chen2024sharegpt4v]&DCI [dci] datasets.

Appendix C Additional Related Work

C.1 Visual Blindness in VLMs.

Despite significant advancements, MLLMs still face limitations in their visual capabilities, hindering their practical applications. Eyes Wide Shut [eyeswideshut] demonstrated that even GPT-4V [achiam2023gpt] struggles with basic visual questions. Research on this topic typically points to two main sources of failure: one relates to the language decoder, which can hallucinate details not present in the image [bai2024hallucination], while the other focuses on the visual encoder, which may provide ambiguous visual information. Several studies, including Cambrian [cambrian], suggest that the visual encoder provides ambiguous visual information and constitutes a critical bottleneck. We also concentrate on the visual issue, particularly within the context of our lightweight model, where the visual encoder accounts for a significant portion of the parameters. Furthermore, we introduce a novel operational framework to improve visual grounding.

Appendix D SeR-Connector

D.1 Ablation study

We conduct a series of ablation experiments to analyze the design choices behind the SeR-Connector, as summarized in Table 12, with the corresponding variants illustrated in Figure 9. Our analysis covers three aspects: connector selection, refinement configurations, and the layer-indexing strategy for visual feature extraction. (I) We first compare multimodal connectors used for initial caption generation. Prior work has explored designs such as Cross-Attention [ramos2023smallcap, barraco2022camel], Q-Former [li2023blip2bootstrappinglanguageimagepretraining], and Transformer-style modules, yet we find that the simple MLP connector from LLaVA [llava15] remains competitive. Configurations (b) and (f), which incorporate multi-level features and a BERT-style Transformer block, respectively, show slightly improved performance, but given the small gains, we preserve the lightweight MLP structure for efficiency. (II) We then assess different connectors for the refinement stage. Combining our base configuration (a) with structure (A), which uses both proposed inputs, achieves strong results. Although applying (A) to configuration (b) yields a minor improvement, it is insufficient to justify adopting it as the default. (III) Lastly, we examine the effect of selecting different layer sets from the ViT encoder. Across the tested combinations, using a diverse trio of layers 13, 18, 23 provides the best performance.

Appendix E Discussions

E.1 Why do previous small models fall short?

The development of captioning models can be viewed in two phases: before and after the integration of LLMs. Earlier approaches typically relied on architectures with relatively small parameter counts. For instance, CaMEL [barraco2022camel] and SmallCap [ramos2023smallcap] used GPT-2 models with 125M to 350M parameters, while Tag2Text [huang2023tag2text] and LoCCa [wan2024locca] employed BERT-based models ranging from 300M to 900M parameters. The emergence of ClipCap [mokady2021clipcap], BLIP [li2023blip2], and LLaVA [llava] shifted the field toward LLM-driven captioners, and recent research has largely centered on building MLLMs. In contrast to this trend, we revisit smaller captioning models and highlight an overlooked limitation. Many of these models inject visual features through cross-attention, a design that offers limited benefit when paired with small language models. Empirically, this is reflected in the performance of structure (c) in Figure 9, which yields a relatively low score of 125.9 in Table 12. Had the field not shifted so strongly toward LLMs, such architectural constraints in small models might have been recognized earlier. Building on this insight, we adopt the LLaVA architecture and inject visual features directly into the self-attention inputs, as shown in Figure 9 (a). This simple modification leads to stronger performance and demonstrates that small models can remain practical and effective. We hope this encourages reducing reliance on LLMs for tasks such as captioning and fosters greater interest in developing lightweight yet capable models.

E.2 Limitations of Existing MLLMs

We examine the broader challenges faced by existing multimodal large language models (MLLMs), particularly their susceptibility to visual blindness. Prior work, including Eyes Wide Shut [eyeswideshut] and Cambrian [cambrian], has identified this issue and attempted to mitigate it using multiple vision encoders such as DINOv2 [oquab2024dinov], SigLIPv2 [siglip2], and CLIP [CLIP]. However, as illustrated in Figure 10, even large-scale models continue to struggle with producing consistent long-form captions in complex, multi-object scenes. We further evaluate two recent MLLMs, LLaVA-Next [llavanext] and LLaVA-OneVision [llavaonevision]. Despite employing advanced techniques—such as partitioning the input into grids and processing features from each region—both models still generate incorrect captions. These observations indicate that visual blindness remains a persistent issue across different model sizes and architectures. In this context, our MM-SeR framework, which directs attention to key regions via initial captions and leverages multi-level features from a single vision encoder, offers an efficient and effective step toward addressing this limitation.

E.3 Limitations of lightweight captioners

While we have demonstrated the potential of small models in captioning tasks, the use of lightweight LMs unavoidably introduces some limitations. In particular, we observe that the model occasionally suffers from issues such as repetitive phrasing, reduced fluency, limited OCR capability, and a lack of general world knowledge. Examples of these cases are provided in Figure 11. These limitations may stem from two primary factors: (i) the small number of parameters, which can restrict the model’s capacity for complex reasoning and language generation [kaplan2020scaling], and (ii) the limited scale and quality of training data, as our model was trained on approximately 500K image-caption pairs from ShareGPT-4V [chen2024sharegpt4v] which contains machine-generated captions. A natural direction for future work is to investigate how far the capabilities of small models can be scaled with access to larger and higher-quality training datasets. In addition to the results in Section E.2, we observe similar issues in larger models such as LLaVA-1.5, suggesting that these challenges remain unresolved [bai2024hallucination, eyeswideshut] and require deeper investigation.

E.4 Limitations of existing evaluation methods

To ensure fair comparison across captioning models, we adopt seven evaluation metrics: BLEU@4 [papineni2002bleu], METEOR [meteor], CIDEr [vedantam2015cider], BERTScore [bertscore], CAPTURE [dong2024benchmarking], CLAIR [clair], and MLLM-as-judge [mllmjudge]. For CLAIR and MLLM-as-judge, we randomly sample 100 images and evaluate each with 10 different seeds to report both the mean and standard deviation (shown with the $\pm$ symbol). Despite these efforts, current evaluation metrics do not always correlate well with human judgment. Some models rank higher under one metric but lower under others, leading to inconsistent comparisons. Moreover, as discussed in Section E.3, small specialists sometimes exhibit reduced fluency, which existing metrics fail to capture. Among the metrics examined, MLLM-as-judge generally provides more stable and reliable assessments, while CLAIR shows higher variance across runs. These observations highlight the need for more robust evaluation methods for captioning. Future research should account for multiple aspects of quality, including fluency, coherence, faithfulness, relevance, informativeness, and completeness. Additionally, moving beyond n-gram matching, MLLM-based evaluators (e.g., OpenAI GPT) will be essential for producing consistent and trustworthy assessments. Such advancements can improve confidence in captioning-based applications.

E.5 Role of Pseudo-Initial Captions in Refinement

In this part, we discuss how incorporating pseudo-initial captions provides more effective supervision during training. We demonstrate the following points in the main paper: (i) The model is trained to generate the ground-truth (GT) caption given both the image and the pseudo-initial caption as input: $I+\hat{c}\;\rightarrow\;\text{Our model}\;\rightarrow\;c$. (ii) If the pseudo-initial caption is generated following the strategies in Section 4.2, then it is unlikely to include too many differing tokens from the GT caption. To illustrate the effect of pseudo-initial captions, we provide an example from Table 11: consider the pair in which both $c$ and $\hat{c}$ correspond to variations of “A woman in a room with a cat.” When the pseudo-initial caption and the GT caption share a substantial portion of tokens, the model is encouraged to consult $\hat{c}$ and revise only the mismatches to recover the GT caption. In contrast, if the two captions differ too greatly, the model tends to ignore the pseudo-initial caption and regenerate the GT caption independently. This behavior enables the model to correct localized errors rather than rewriting the entire caption, thereby preventing misleading supervision.

As an additional experiment, we fine-tune our captioner in Stage 2 of Figure 4 using pseudo-initial captions generated under four conditions: (Data,1) initial captions produced by the Stage 1–trained captioner; (Data,2) pseudo-initial captions that differ substantially from the GT caption; (Data,3) pseudo-initial captions with minor modifications from the GT caption (our default strategy); and (Data,4) two pseudo-initial captions per sample—one generated as in Data,3 and one identical to the GT caption (e.g., GT: “A woman in a room with a cat”; pseudo-initials: “A boy in a room with a dog” and “A woman in a room with a cat”). Table 13 shows that Data,1 yields reasonable improvements within our framework, while Data,2 produces the expected behavior in which the model disregards the pseudo-initial caption and regenerates a new one. Notably, Data,4 performs comparably to Data,3, and together with the effect observed in Data,1, suggests that our framework is robust to moderate variation in pseudo-initial caption quality.

E.6 Further Research Questions

To guide future exploration, we outline several research directions that may advance the field of lightweight captioning and multimodal learning more broadly:

What is the minimal model size required for a captioning specialist to be practically useful in real-world assistive technologies? At what point does the performance–efficiency trade-off stabilize? 2. 2.

Although supervised training is commonly used in multimodal training, recent work has shown clear benefits from reinforcement learning in LLMs [touvron2023llama]. Can similar reward-driven methods improve captioning models? 3. 3.

The LLaVA framework is widely adopted due to its simplicity and open-source accessibility. Could a unified Transformer-based multimodal architecture serve as a more effective foundation for practical captioners? 4. 4.

Multimodal performance depends heavily on the quality of visual representations. How reliable are features from encoders such as SigLIP [zhai2023sigmoid] or MAE [mae_survey], especially for fine-grained captioning? 5. 5.

Can our framework be applied effectively in simpler VQA settings, and are large language models necessary in such cases? 6. 6.

Do captioning models exhibit biases (e.g., gender, occupation, age), and how should such concerns be addressed across diverse cultural contexts and deployment domains such as indoor, outdoor, and robotic environments? Could domain adaptation with user feedback offer a viable solution?

We consider these questions promising directions for future research and believe that addressing them will contribute to more widely deployable multimodal captioning systems.

Appendix F Supplementary Results

Analysis of captioning operation.

We present additional results from the analysis experiments conducted in Section 3, as shown in Figure 12. In our supplementary analysis of the captioning operation, we further examine whether CLIP’s visual representations contain sufficiently fine-grained information. To this end, we adopt the decoder architecture from Masked Autoencoder (MAE) [mae], freeze the CLIP encoder, and train only the MAE decoder on MS-COCO images. The reconstructed outputs show that images generated from the CLIP embeddings deviate noticeably from the originals and lack visual clarity. This suggests that the final-layer CLIP features are relatively coarse, reinforcing our motivation for leveraging multi-level representations to support more detailed and faithful caption generation.

Comparison between our specialist and LLaVA-1.5-7B.

Additional comparisons between our lightweight specialist and the large multimodal generalist LLaVA-1.5-7B, discussed in Section 3, are provided in Figure 14. Despite its smaller parameter size and frozen vision encoder, our specialist delivers unexpectedly strong performance on captioning benchmarks, suggesting that compact models can still handle detailed captioning tasks effectively.

Impact of multimodal self-refinement.

Additional qualitative results related to our refinement framework are shown in Figure 13, supplementing the findings in Section 5. Our refinement method provides noticeable improvements in caption quality for both single-sentence and detailed captioning.

Appendix G Extensive Details on The Dataset

G.1 Pseudo-Initial Caption Generation

We describe our strategy for generating pseudo-initial captions as follows. The instruction prompt used for this process is shown in Figure 16.

• •

Pseudo-initial captions are created by modifying 0–3 elements of the ground-truth (GT) caption.

• •

Modifications fall into four categories: Entity (e.g., chair, cat), Attribute (e.g., color, material such as wooden, count such as three cups, texture, shape, size, inspired by DSG [1]), Relation (e.g., A in front of B), and Action (e.g., eating, blowing).

• •

Apart from these modifications, the overall sentence structure and style should be preserved.

• •

The pseudo-initial caption may occasionally be identical to the GT caption.

• •

Few-shot examples are provided, as shown in Table 14.

We consistently generate three pseudo-initial captions per GT caption. Hence, the dataset can be summarized as Table 15, where ‘#pairs’ indicates the number of image-caption pairs.

G.2 Datasets for Detailed Captioning

While MS COCO [mscoco] has long been the standard benchmark for image captioning, its single-sentence annotations often fail to capture the full richness of visual content. Recent works [ye2025painting, gabbay2021zerodim, li2023factual, dong2024benchmarking] highlight this limitation and underscore the need for datasets that support more detailed captioning. In this study, we use three datasets that provide higher-quality and more comprehensive image descriptions: ShareGPT-4V [chen2024sharegpt4v], DCI [dci], and GLaMM [rasheed2024glamm]. ShareGPT-4V captions are initially generated by GPT-4o and subsequently refined by human annotators. DCI consists of fully human-written captions. GLaMM, in contrast, produces detailed descriptions by combining outputs from multiple open-source tools, including object detectors and scene parsers, and composing them using an LLM.

G.3 ShareGPT & DCI

The DCI dataset [dci] contains 7.4K training images and 0.4K test images, each paired with 10 human-written captions averaging 55 words. The ShareGPT4V dataset [chen2024sharegpt4v] includes 100K images, with each image accompanied by a single long, human-verified caption of roughly 200 words. Directly using this format poses challenges for n-gram–based evaluation metrics, which benefit from multiple reference captions per image. To address this, we summarize each original caption into five shorter captions of approximately 50 words. Generating multiple long captions risks introducing hallucinations, whereas summarization preserves the original content faithfully. The prompt used for this process is shown in Figure 15, and the resulting dataset will be made publicly available. Because DCI is relatively small compared to MS COCO (118K images with five captions each), we combine DCI with the processed ShareGPT4V dataset to create a unified benchmark. This yields 102.4K training images (100K from ShareGPT4V and 7.4K from DCI), each with five or ten detailed captions, and 5K test images—making the combined dataset comparable in scale to MS COCO.

G.4 GLaMM

The GLaMM dataset [rasheed2024glamm] contains automatically generated captions produced using a combination of object detection models, scene-graph parsers, and LLMs. Each caption is approximately 45 words on average. In our experience, despite leveraging a wide range of visual tools, the caption quality is often inconsistent. We observed frequent factual inaccuracies, typically one or two incorrect words appearing every two to three captions. Each image in GLaMM is paired with a single caption, which poses challenges for n-gram–based evaluation, as previously discussed. Nevertheless, we include this dataset in our experiments. We randomly sample 600K image–caption pairs from the full dataset, allocating 30K for testing and using the remaining 570K for training. Possibly due to the quality issues noted above, models trained on GLaMM generally underperform compared to those trained on ShareGPT4V and DCI.

Appendix H Prompt Templates

We provide the prompt templates used when interacting with OpenAI’s GPT models throughout our study. The prompt for summarizing long captions into shorter ones, introduced in Section G.3, is shown in Figure 15. The prompt used to generate pseudo-initial captions for our refinement framework is presented in Figure 16, with example outputs shown in Figure 18. Finally, the prompt used in the MLLM-as-judge evaluation setup is provided in Figure 17, where we closely follow the template proposed in the original MLLM-as-judge papers [clair, mllmjudge].

Appendix I Experimental Details

I.1 Pretraining and Finetuning

Our implementation is based on the LLaVA-1.5 repository [llava15]. To reduce reliance on large language models, we replace LLaMA with the OPT series [zhang2022opt]. Training proceeds in three stages: pretraining, finetuning for caption generation (described in Section 3), and finetuning for multimodal self-refinement (described in Section 4.2). The hyperparameters used in each stage are summarized in Table 16. We closely follow the original LLaVA training configuration. As highlighted in Section E.1, our method differs from prior small-model approaches in that visual features are injected directly into the self-attention inputs of the language model.

I.2 Experiment setup of MLLMs

Table 2 compares specialist models with several generalist MLLMs, including InstructBLIP [dai2023instructblip], Unified-IO-XL [lu2023unifiedio], Shikra [chen2023shikra], Qwen-VL [bai2023qwenvl], and LLaVA-1.5 [llava15]. Although these generalist models were trained on MS COCO images, they were not trained on datasets such as ShareGPT4V, DCI, or GLaMM. Instead, they were instruction-tuned on large-scale multimodal datasets; for example, Qwen-VL and InstructBLIP were trained on approximately 1.5B and 130M instruction samples, respectively. For this reason, we categorize them as generalist models. We evaluate the generalist MLLMs using publicly available checkpoints from their official repositories, without additional fine-tuning. For the single-sentence captioning task, we use the prompt: Provide a one-sentence caption for the provided image.” For the detailed captioning task, we use the prompt: Describe the photo within 55 words.” We also include an additional evaluation of LLaVA-1.5 using an alternative instruction, as shown in Table 17. These results highlight two key observations: (1) evaluation metrics such as BERTScore [bertscore] tend to penalize long-form outputs, and (2) longer generations increase hallucination frequency, consistent with prior findings [jiang2024hallucination].

I.3 Attention map visualization

As part of our analysis in Section 5.2, we evaluate whether the model attends to the appropriate image regions when generating specific words in a caption. To this end, we adapt the visualization code provided by API [yu2024attention], originally designed to highlight attention maps between images and questions in VQA tasks. We modify the attention hooking module222https://github.com/yu-rp/apiprompting/blob/master/API/API_LLaVA/hook.py to visualize attention between image regions and selected words within captions. This enables us to examine which areas the language model focuses on when producing particular tokens.

I.4 Image reconstruction

To investigate whether CLIP’s visual representations are coarse or ambiguous, we conducted an image reconstruction experiment using a Masked AutoEncoder (MAE) framework (details in Section 4). In this setup, a Vision Transformer (ViT) encoder produces visual features, which the decoder subsequently uses to reconstruct the image. We adopt the same visual encoder as used in LLaVA-1.5, CLIP ViT-L/14-336, and keep its parameters frozen. The decoder receives the visual embeddings without masking and predicts the corresponding RGB image. We utilize two types of visual inputs: (i) Last-layer Features (LF), and (ii) Multi-Level Features (ML), where ML consists of outputs from layers 13, 18, and 23 of the encoder. The decoder was trained on 100K images from the MS COCO dataset. Further architectural details are available in the MAE repository333https://github.com/facebookresearch/mae/blob/main/models_mae.py.

I.5 Long Range Video Question Answering

We evaluate our model on the Long-Range Video Question Answering task [zhang2023simple]. This task requires the system to answer multiple-choice questions based on user queries about videos that are 10 minutes or longer in duration. We emphasize that currently, no video MLLMs (Multimodal Large Language Models) are capable of handling this task directly. Most existing models [li2024llava, bai2025qwen2, zhu2025internvl3] impose limits on the number of input video frames they can process, making it difficult to cover the full temporal span of such long videos. To address this limitation, recent approaches [ye2025re, wang2024videoagent] propose first extracting per-frame captions. Subsequently, both the generated captions and the question are injected into an LLM capable of processing over 100K tokens in a single prompt. For evaluation, we follow the setup provided in the official implementation of LLoVi444https://github.com/CeeZh/LLoVi. replacing only the captioning models. (i) When using the generalist model, we generate a caption for each frame using the prompt: "Describe this frame within 50 words.". (ii) For our specialist model, we utilize the same prompt and apply the version trained on the ShareGPT4V [chen2024sharegpt4v] and DCI [dci] datasets.