Sounding that Object: Interactive Object-Aware Image to Audio Generation

TL;DR

This paper introduces an interactive object-aware image-to-audio generation model that enables users to generate sounds for specific objects in images, leveraging object-centric learning and attention mechanisms for improved alignment.

Contribution

The paper presents a novel model integrating object-centric learning with diffusion models, allowing interactive, object-level sound generation grounded in image segmentation.

Findings

Outperforms baselines in object-sound alignment

Employs attention mechanism that approximates segmentation masks

Enables user interaction for targeted sound generation

Abstract

Generating accurate sounds for complex audio-visual scenes is challenging, especially in the presence of multiple objects and sound sources. In this paper, we propose an {\em interactive object-aware audio generation} model that grounds sound generation in user-selected visual objects within images. Our method integrates object-centric learning into a conditional latent diffusion model, which learns to associate image regions with their corresponding sounds through multi-modal attention. At test time, our model employs image segmentation to allow users to interactively generate sounds at the {\em object} level. We theoretically validate that our attention mechanism functionally approximates test-time segmentation masks, ensuring the generated audio aligns with selected objects. Quantitative and qualitative evaluations show that our model outperforms baselines, achieving better alignment between objects and their associated sounds. Project page: https://tinglok.netlify.app/files/avobject/