Feasibility with Language Models for Open-World Compositional Zero-Shot Learning

TL;DR

This paper introduces FLM, a method leveraging large language models to assess the feasibility of state-object pairs, significantly improving open-world compositional zero-shot learning performance.

Contribution

The work demonstrates how external LLMs can effectively determine feasibility of unseen combinations, enhancing zero-shot learning in open-world scenarios.

Findings

FLM improves OW-CZSL performance across benchmarks.

Vicuna and ChatGPT are identified as top-performing LLMs.

In-context learning is crucial for LLM effectiveness in feasibility assessment.

Abstract

Humans can easily tell if an attribute (also called state) is realistic, i.e., feasible, for an object, e.g. fire can be hot, but it cannot be wet. In Open-World Compositional Zero-Shot Learning, when all possible state-object combinations are considered as unseen classes, zero-shot predictors tend to perform poorly. Our work focuses on using external auxiliary knowledge to determine the feasibility of state-object combinations. Our Feasibility with Language Model (FLM) is a simple and effective approach that leverages Large Language Models (LLMs) to better comprehend the semantic relationships between states and objects. FLM involves querying an LLM about the feasibility of a given pair and retrieving the output logit for the positive answer. To mitigate potential misguidance of the LLM given that many of the state-object compositions are rare or completely infeasible, we observe that the in-context learning ability of LLMs is essential. We present an extensive study identifying Vicuna and ChatGPT as best performing, and we demonstrate that our FLM consistently improves OW-CZSL performance across all three benchmarks.