Reconstructing Animals and the Wild

TL;DR

This paper introduces a novel method for reconstructing complex natural scenes, including animals and their environments, from single images using synthetic data and large language model priors, advancing scene understanding beyond anthropocentric assumptions.

Contribution

It presents a new approach that combines large language models and synthetic datasets to reconstruct natural scenes with animals and environment from single images, without relying on real-world training data.

Findings

Successfully reconstructs natural scenes with animals and environment from single images.

Generalizes well to real-world images despite training solely on synthetic data.

Provides a large synthetic dataset and code for future research.

Abstract

The idea of 3D reconstruction as scene understanding is foundational in computer vision. Reconstructing 3D scenes from 2D visual observations requires strong priors to disambiguate structure. Much work has been focused on the anthropocentric, which, characterized by smooth surfaces, coherent normals, and regular edges, allows for the integration of strong geometric inductive biases. Here, we consider a more challenging problem where such assumptions do not hold: the reconstruction of natural scenes containing trees, bushes, boulders, and animals. While numerous works have attempted to tackle the problem of reconstructing animals in the wild, they have focused solely on the animal, neglecting environmental context. This limits their usefulness for analysis tasks, as animals exist inherently within the 3D world, and information is lost when environmental factors are disregarded. We propose a method to reconstruct natural scenes from single images. We base our approach on recent advances leveraging the strong world priors ingrained in Large Language Models and train an autoregressive model to decode a CLIP embedding into a structured compositional scene representation, encompassing both animals and the wild (RAW). To enable this, we propose a synthetic dataset comprising one million images and thousands of assets. Our approach, having been trained solely on synthetic data, generalizes to the task of reconstructing animals and their environments in real-world images. We will release our dataset and code to encourage future research at https://raw.is.tue.mpg.de/