Online-PVLM: Advancing Personalized VLMs with Online Concept Learning

TL;DR

Online-PVLM introduces a train-free, hyperbolic representation-based framework for real-time personalized concept learning in visual language models, enabling scalable and efficient adaptation without retraining.

Contribution

It proposes a novel online concept learning method using hyperbolic embeddings, supporting real-time personalization in large-scale scenarios without additional training.

Findings

Achieves state-of-the-art performance on the OP-Eval benchmark.

Supports real-time, scalable personalized concept learning.

Demonstrates effectiveness across diverse question types.

Abstract

Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically require the learning of separate embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient retrieval of concept embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging hyperbolic representations. Our approach makes a train-free paradigm for concept embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and dataset will be made available.