Adaptive Time-step Training for Enhancing Spike-Based Neural Radiance Fields

TL;DR

This paper introduces PATA, a dynamic training strategy for spike-based NeRF models that adaptively balances rendering quality and computational efficiency, significantly reducing inference time and power consumption.

Contribution

It proposes a novel adaptive time step training method for spike-based NeRFs, enabling scene-specific inference and resource savings without sacrificing rendering quality.

Findings

Reduces inference time steps by 64%

Decreases running power by 61.55%

Maintains high rendering fidelity

Abstract

Neural Radiance Fields (NeRF)-based models have achieved remarkable success in 3D reconstruction and rendering tasks. However, during both training and inference, these models rely heavily on dense point sampling along rays from multiple viewpoints, resulting in a surge in floating-point operations and severely limiting their use in resource-constrained scenarios like edge computing. Spiking Neural Networks (SNNs), which communicate via binary spikes over discrete time steps, offer a promising alternative due to their energy-efficient nature. Given the inherent variability in scene scale and texture complexity in neural rendering and the prevailing practice of training separate models per scene, we propose a spike-based NeRF framework with a dynamic time step training strategy, termed Pretrain-Adaptive Time-step Adjustment (PATA). This approach automatically explores the trade-off between rendering quality and time step length during training. Consequently, it enables scene-adaptive inference with variable time steps and reduces the additional consumption of computational resources in the inference process. Anchoring to the established Instant-NGP architecture, we evaluate our method across diverse datasets. The experimental results show that PATA can preserve rendering fidelity while reducing inference time steps by 64\% and running power by 61.55\%.