T4DT: Tensorizing Time for Learning Temporal 3D Visual Data

TL;DR

This paper introduces a novel tensor-based compression method for 4D time-varying 3D scenes, significantly reducing memory requirements while maintaining geometric fidelity, using low-rank tensor formats with theoretical guarantees.

Contribution

It proposes a closed-form tensor compression approach for 4D signed distance functions, offering an efficient alternative to learning-based methods like DeepSDF and NeRF.

Findings

Achieves high compression ratios for 4D signed distance functions.

Maintains geometric quality despite significant memory reduction.

Provides theoretical guarantees for tensor truncation methods.

Abstract

Unlike 2D raster images, there is no single dominant representation for 3D visual data processing. Different formats like point clouds, meshes, or implicit functions each have their strengths and weaknesses. Still, grid representations such as signed distance functions have attractive properties also in 3D. In particular, they offer constant-time random access and are eminently suitable for modern machine learning. Unfortunately, the storage size of a grid grows exponentially with its dimension. Hence they often exceed memory limits even at moderate resolution. This work proposes using low-rank tensor formats, including the Tucker, tensor train, and quantics tensor train decompositions, to compress time-varying 3D data. Our method iteratively computes, voxelizes, and compresses each frame's truncated signed distance function and applies tensor rank truncation to condense all frames into a single, compressed tensor that represents the entire 4D scene. We show that low-rank tensor compression is extremely compact to store and query time-varying signed distance functions. It significantly reduces the memory footprint of 4D scenes while remarkably preserving their geometric quality. Unlike existing, iterative learning-based approaches like DeepSDF and NeRF, our method uses a closed-form algorithm with theoretical guarantees.