Improving Personalized Search with Regularized Low-Rank Parameter Updates

TL;DR

This paper introduces a regularized low-rank parameter adaptation method for personalized vision-language retrieval, effectively recognizing new personal concepts with minimal data while preserving general knowledge, and achieves state-of-the-art results on benchmark datasets.

Contribution

The paper proposes a novel low-rank regularized adaptation technique for vision-language models to improve personalized retrieval of new concepts with few examples.

Findings

Achieves 4-22% improvement over prior methods on DeepFashion2 and ConCon-Chi benchmarks.

Regularized low-rank adaptation effectively balances personal concept learning and general knowledge retention.

Parameter addition strategy enhances the combination of multiple personal concepts.

Abstract

Personalized vision-language retrieval seeks to recognize new concepts (e.g. "my dog Fido") from only a few examples. This task is challenging because it requires not only learning a new concept from a few images, but also integrating the personal and general knowledge together to recognize the concept in different contexts. In this paper, we show how to effectively adapt the internal representation of a vision-language dual encoder model for personalized vision-language retrieval. We find that regularized low-rank adaption of a small set of parameters in the language encoder's final layer serves as a highly effective alternative to textual inversion for recognizing the personal concept while preserving general knowledge. Additionally, we explore strategies for combining parameters of multiple learned personal concepts, finding that parameter addition is effective. To evaluate how well general knowledge is preserved in a finetuned representation, we introduce a metric that measures image retrieval accuracy based on captions generated by a vision language model (VLM). Our approach achieves state-of-the-art accuracy on two benchmarks for personalized image retrieval with natural language queries - DeepFashion2 and ConCon-Chi - outperforming the prior art by 4%-22% on personal retrievals.