ViTOC: Vision Transformer and Object-aware Captioner

TL;DR

ViTOC is a novel image captioning model that combines Vision Transformer and object detection to improve accuracy and diversity, utilizing an object-aware prompting strategy and CLIP-based evaluation.

Contribution

The paper introduces a dual-path architecture with object-aware prompting for enhanced captioning, and a reference-free CLIP-based evaluation method.

Findings

Outperforms baseline models on COCO dataset

Effective handling of long-tail data through object-aware prompting

Achieves efficient end-to-end training with pretrained models

Abstract

This paper presents ViTOC (Vision Transformer and Object-aware Captioner), a novel vision-language model for image captioning that addresses the challenges of accuracy and diversity in generated descriptions. Unlike conventional approaches, ViTOC employs a dual-path architecture based on Vision Transformer and object detector, effectively fusing global visual features and local object information through learnable vectors. The model introduces an innovative object-aware prompting strategy that significantly enhances its capability in handling long-tail data. Experiments on the standard COCO dataset demonstrate that ViTOC outperforms baseline models across all evaluation metrics. Additionally, we propose a reference-free evaluation method based on CLIP to further validate the model's effectiveness. By utilizing pretrained visual model parameters, ViTOC achieves efficient end-to-end training.