Deep sprite-based image models: An analysis

TL;DR

This paper analyzes sprite-based image decomposition models, identifying their core components, and proposes a new deep method that improves interpretability, scalability, and performance on segmentation benchmarks.

Contribution

The paper provides an in-depth analysis of sprite-based models and introduces a deep decomposition method that matches state-of-the-art performance and enhances interpretability and scalability.

Findings

Proposed method performs on par with state-of-the-art segmentation models.

Scales linearly with the number of objects in images.

Explicitly identifies object categories and models images interpretably.

Abstract

While foundation models drive steady progress in image segmentation and diffusion algorithms compose always more realistic images, the seemingly simple problem of identifying recurrent patterns in a collection of images remains very much open. In this paper, we focus on sprite-based image decomposition models, which have shown some promise for clustering and image decomposition and are appealing because of their high interpretability. These models come in different flavors, need to be tailored to specific datasets, and struggle to scale to images with many objects. We dive into the details of their design, identify their core components, and perform an extensive analysis on clustering benchmarks. We leverage this analysis to propose a deep sprite-based image decomposition method that performs on par with state-of-the-art unsupervised class-aware image segmentation methods on the standard CLEVR benchmark, scales linearly with the number of objects, identifies explicitly object categories, and fully models images in an easily interpretable way.