PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts

TL;DR

PerceptionCLIP enhances zero-shot image classification by inferring and conditioning on contextual attributes, inspired by human perception, leading to improved robustness and generalization without additional training.

Contribution

It introduces a training-free, two-step method that leverages CLIP's ability to infer context, improving zero-shot classification performance.

Findings

Improves zero-shot classification accuracy

Enhances robustness to spurious features

Achieves better generalization and group robustness

Abstract

Vision-language models like CLIP are widely used in zero-shot image classification due to their ability to understand various visual concepts and natural language descriptions. However, how to fully leverage CLIP's unprecedented human-like understanding capabilities to achieve better performance is still an open question. This paper draws inspiration from the human visual perception process: when classifying an object, humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information. Inspired by it, we observe that providing CLIP with contextual attributes improves zero-shot image classification and mitigates reliance on spurious features. We also observe that CLIP itself can reasonably infer the attributes from an image. With these observations, we propose a training-free, two-step zero-shot classification method PerceptionCLIP. Given an image, it first infers contextual attributes (e.g., background) and then performs object classification conditioning on them. Our experiments show that PerceptionCLIP achieves better generalization, group robustness, and interoperability. Our code is available at https://github.com/umd-huang-lab/perceptionCLIP