Multi-task View Synthesis with Neural Radiance Fields

TL;DR

This paper introduces MuvieNeRF, a novel framework for multi-task view synthesis that leverages multi-view and multi-task attention mechanisms to synthesize various scene properties, outperforming traditional models.

Contribution

The paper proposes MuvieNeRF, a new multi-task view synthesis framework with cross-task and cross-view attention modules, enabling versatile scene property synthesis across different NeRF backbones.

Findings

MuvieNeRF achieves high-quality multi-property synthesis on synthetic and real benchmarks.

It outperforms conventional discriminative models in multi-view scene property prediction.

The framework demonstrates universal applicability across various NeRF architectures.

Abstract

Multi-task visual learning is a critical aspect of computer vision. Current research, however, predominantly concentrates on the multi-task dense prediction setting, which overlooks the intrinsic 3D world and its multi-view consistent structures, and lacks the capability for versatile imagination. In response to these limitations, we present a novel problem setting -- multi-task view synthesis (MTVS), which reinterprets multi-task prediction as a set of novel-view synthesis tasks for multiple scene properties, including RGB. To tackle the MTVS problem, we propose MuvieNeRF, a framework that incorporates both multi-task and cross-view knowledge to simultaneously synthesize multiple scene properties. MuvieNeRF integrates two key modules, the Cross-Task Attention (CTA) and Cross-View Attention (CVA) modules, enabling the efficient use of information across multiple views and tasks. Extensive evaluation on both synthetic and realistic benchmarks demonstrates that MuvieNeRF is capable of simultaneously synthesizing different scene properties with promising visual quality, even outperforming conventional discriminative models in various settings. Notably, we show that MuvieNeRF exhibits universal applicability across a range of NeRF backbones. Our code is available at https://github.com/zsh2000/MuvieNeRF.