Tuning-less Object Naming with a Foundation Model

TL;DR

This paper presents a real-time, tuning-less object naming system using a foundation model that associates image features with a growing vocabulary through attention, enabling one-shot object naming without model fine-tuning.

Contribution

The novel approach leverages transformer attention mechanisms to associate image features with vocabulary indices, supporting generalization and one-shot object naming without additional training.

Findings

System can correctly name objects in different contexts

Supports handling a large number of objects

Operates in real-time without model fine-tuning

Abstract

We implement a real-time object naming system that enables learning a set of named entities never seen. Our approach employs an existing foundation model that we consider ready to see anything before starting. It turns seen images into relatively small feature vectors that we associate with index to a gradually built vocabulary without any training of fine-tuning of the model. Our contribution is using the association mechanism known from transformers as attention. It has features that support generalization from irrelevant information for distinguishing the entities and potentially enable associating with much more than indices to vocabulary. As a result, the system can work in a one-shot manner and correctly name objects named in different contents. We also outline implementation details of the system modules integrated by a blackboard architecture. Finally, we investigate the system's quality, mainly how many objects it can handle in this way.