DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement

TL;DR

DegustaBot is a novel zero-shot visual preference estimation algorithm that personalizes multi-object household rearrangement tasks using pre-trained vision-language models and a new preference dataset.

Contribution

The paper introduces DegustaBot, leveraging vision-language foundation models with novel zero-shot prompting for personalized object rearrangement.

Findings

50% of model predictions are acceptable to at least 20% of users

Developed a large dataset of personal preferences in table setting

Proposed two new metrics for evaluating personalized preferences

Abstract

De gustibus non est disputandum ("there is no accounting for others' tastes") is a common Latin maxim describing how many solutions in life are determined by people's personal preferences. Many household tasks, in particular, can only be considered fully successful when they account for personal preferences such as the visual aesthetic of the scene. For example, setting a table could be optimized by arranging utensils according to traditional rules of Western table setting decorum, without considering the color, shape, or material of each object, but this may not be a completely satisfying solution for a given person. Toward this end, we present DegustaBot, an algorithm for visual preference learning that solves household multi-object rearrangement tasks according to personal preference. To do this, we use internet-scale pre-trained vision-and-language foundation models (VLMs) with novel zero-shot visual prompting techniques. To evaluate our method, we collect a large dataset of naturalistic personal preferences in a simulated table-setting task, and conduct a user study in order to develop two novel metrics for determining success based on personal preference. This is a challenging problem and we find that 50% of our model's predictions are likely to be found acceptable by at least 20% of people.