SplaTraj: Camera Trajectory Generation with Semantic Gaussian Splatting

TL;DR

SplaTraj is a novel framework that generates photorealistic camera trajectories based on language instructions by formulating the problem as a continuous-time trajectory optimization over Gaussian Splatting models.

Contribution

It introduces a new method to generate image sequences from photorealistic models guided by language, using gradient-based optimization of camera trajectories.

Findings

Successfully generates photorealistic image sequences matching instructions.

Optimizes camera paths to smoothly view specified objects.

Demonstrates high-quality results across various environments.

Abstract

Many recent developments for robots to represent environments have focused on photorealistic reconstructions. This paper particularly focuses on generating sequences of images from the photorealistic Gaussian Splatting models, that match instructions that are given by user-inputted language. We contribute a novel framework, SplaTraj, which formulates the generation of images within photorealistic environment representations as a continuous-time trajectory optimization problem. Costs are designed so that a camera following the trajectory poses will smoothly traverse through the environment and render the specified spatial information in a photogenic manner. This is achieved by querying a photorealistic representation with language embedding to isolate regions that correspond to the user-specified inputs. These regions are then projected to the camera's view as it moves over time and a cost is constructed. We can then apply gradient-based optimization and differentiate through the rendering to optimize the trajectory for the defined cost. The resulting trajectory moves to photogenically view each of the specified objects. We empirically evaluate our approach on a suite of environments and instructions, and demonstrate the quality of generated image sequences.