FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptation

TL;DR

This paper introduces a lightweight adaptation method for multiview diffusion models that enables explicit control over material, texture, and style transfer across multiple views, using few-shot self-attention feature aggregation.

Contribution

It presents a novel few-shot self-attention adaptation technique that enhances multiview diffusion models with appearance transfer capabilities while maintaining view coherence.

Findings

Effective appearance transfer with few examples

Preserves object geometry and view consistency

Enables diverse material and style manipulation

Abstract

Multiview diffusion models have rapidly emerged as a powerful tool for content creation with spatial consistency across viewpoints, offering rich visual realism without requiring explicit geometry and appearance representation. However, compared to meshes or radiance fields, existing multiview diffusion models offer limited appearance manipulation, particularly in terms of material, texture, or style. In this paper, we present a lightweight adaptation technique for appearance transfer in multiview diffusion models. Our method learns to combine object identity from an input image with appearance cues rendered in a separate reference image, producing multi-view-consistent output that reflects the desired materials, textures, or styles. This allows explicit specification of appearance parameters at generation time while preserving the underlying object geometry and view coherence. We leverage three diffusion denoising processes responsible for generating the original object, the reference, and the target images, and perform reverse sampling to aggregate a small subset of layer-wise self-attention features from the object and the reference to influence the target generation. Our method requires only a few training examples to introduce appearance awareness to pretrained multiview models. The experiments show that our method provides a simple yet effective way toward multiview generation with diverse appearance, advocating the adoption of implicit generative 3D representations in practice.