Break Out the Silverware -- Semantic Understanding of Stored Household Items
Michaela Levi-Richter, Reuth Mirsky, Oren Glickman

TL;DR
This paper introduces the Stored Household Item Challenge, a benchmark for evaluating service robots' ability to infer storage locations of household items, and presents NOAM, a hybrid vision-language model that demonstrates emergent commonsense reasoning.
Contribution
The paper proposes a new benchmark for household item storage prediction and introduces NOAM, a hybrid model combining scene understanding with large language models, achieving near-human performance.
Findings
NOAM outperforms baseline models and existing multimodal models.
The benchmark datasets enable realistic evaluation of household organization reasoning.
NOAM approaches human-level accuracy in storage location prediction.
Abstract
``Bring me a plate.'' For domestic service robots, this simple command reveals a complex challenge: inferring where everyday items are stored, often out of sight in drawers, cabinets, or closets. Despite advances in vision and manipulation, robots still lack the commonsense reasoning needed to complete this task. We introduce the Stored Household Item Challenge, a benchmark task for evaluating service robots' cognitive capabilities: given a household scene and a queried item, predict its most likely storage location. Our benchmark includes two datasets: (1) a real-world evaluation set of 100 item-image pairs with human-annotated ground truth from participants' kitchens, and (2) a development set of 6,500 item-image pairs annotated with storage polygons over public kitchen images. These datasets support realistic modeling of household organization and enable comparative evaluation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Social Robot Interaction and HRI · Language and cultural evolution
