NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledge

TL;DR

NOC-REK introduces an end-to-end method for novel object captioning that retrieves vocabulary from external knowledge sources, enabling description of objects outside training data without retraining.

Contribution

It presents a novel approach that integrates vocabulary retrieval from external knowledge with caption generation, eliminating the need for retraining when new objects appear.

Findings

Outperforms state-of-the-art methods on COCO and Nocaps datasets.

Effectively describes novel objects outside training data.

Eliminates retraining by updating external knowledge.

Abstract

Novel object captioning aims at describing objects absent from training data, with the key ingredient being the provision of object vocabulary to the model. Although existing methods heavily rely on an object detection model, we view the detection step as vocabulary retrieval from an external knowledge in the form of embeddings for any object's definition from Wiktionary, where we use in the retrieval image region features learned from a transformers model. We propose an end-to-end Novel Object Captioning with Retrieved vocabulary from External Knowledge method (NOC-REK), which simultaneously learns vocabulary retrieval and caption generation, successfully describing novel objects outside of the training dataset. Furthermore, our model eliminates the requirement for model retraining by simply updating the external knowledge whenever a novel object appears. Our comprehensive experiments on held-out COCO and Nocaps datasets show that our NOC-REK is considerably effective against SOTAs.