Multi-Modal LLM based Image Captioning in ICT: Bridging the Gap Between General and Industry Domain

TL;DR

This paper introduces a domain-specific multi-modal large language model for ICT image captioning, trained via a multi-stage strategy, outperforming larger models in accuracy and BLEU scores.

Contribution

It proposes a novel multi-stage training approach for a domain-specific image captioning model in ICT, combining synthetic and expert-annotated data for improved performance.

Findings

DICModel outperforms state-of-the-art models with fewer parameters.

BLEU score increases by approximately 56.8% over comparable models.

Achieves higher accuracy than Qwen2.5-VL 32B on objective questions.

Abstract

In the information and communications technology (ICT) industry, training a domain-specific large language model (LLM) or constructing a retrieval-augmented generation system requires a substantial amount of high-value domain knowledge. However, the knowledge is not only hidden in the textual modality but also in the image modality. Traditional methods can parse text from domain documents but dont have image captioning ability. Multi-modal LLM (MLLM) can understand images, but they do not have sufficient domain knowledge. To address the above issues, this paper proposes a multi-stage progressive training strategy to train a Domain-specific Image Captioning Model (DICModel) in ICT, and constructs a standard evaluation system to validate the performance of DICModel. Specifically, this work first synthesizes about 7K image-text pairs by combining the Mermaid tool and LLMs, which are used for the first-stage supervised-fine-tuning (SFT) of DICModel. Then, ICT-domain experts manually annotate about 2K image-text pairs for the second-stage SFT of DICModel. Finally, experts and LLMs jointly synthesize about 1.5K visual question answering data for the instruction-based SFT. Experimental results indicate that our DICModel with only 7B parameters performs better than other state-of-the-art models with 32B parameters. Compared to the SOTA models with 7B and 32B parameters, our DICModel increases the BLEU metric by approximately 56.8% and 20.8%, respectively. On the objective questions constructed by ICT domain experts, our DICModel outperforms Qwen2.5-VL 32B by 1% in terms of accuracy rate. In summary, this work can efficiently and accurately extract the logical text from images, which is expected to promote the development of multimodal models in the ICT domain.