MatteFormer: Transformer-Based Image Matting via Prior-Tokens

TL;DR

MatteFormer is a transformer-based image matting model that leverages prior-tokens representing trimap regions to improve alpha matte prediction, achieving state-of-the-art results.

Contribution

The paper introduces prior-tokens and a Prior-Attentive Swin Transformer (PAST) block to effectively incorporate trimap information into transformer-based image matting.

Findings

Achieves state-of-the-art performance on Composition-1k and Distinctions-646 datasets.

Utilizes prior-tokens for global trimap region representation.

Demonstrates significant margin improvements over previous methods.

Abstract

In this paper, we propose a transformer-based image matting model called MatteFormer, which takes full advantage of trimap information in the transformer block. Our method first introduces a prior-token which is a global representation of each trimap region (e.g. foreground, background and unknown). These prior-tokens are used as global priors and participate in the self-attention mechanism of each block. Each stage of the encoder is composed of PAST (Prior-Attentive Swin Transformer) block, which is based on the Swin Transformer block, but differs in a couple of aspects: 1) It has PA-WSA (Prior-Attentive Window Self-Attention) layer, performing self-attention not only with spatial-tokens but also with prior-tokens. 2) It has prior-memory which saves prior-tokens accumulatively from the previous blocks and transfers them to the next block. We evaluate our MatteFormer on the commonly used image matting datasets: Composition-1k and Distinctions-646. Experiment results show that our proposed method achieves state-of-the-art performance with a large margin. Our codes are available at https://github.com/webtoon/matteformer.