Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding

TL;DR

This paper introduces sinusoidal positional encoding (SPE), a hyperparameter-free method that adaptively learns high-frequency features, improving efficiency and accuracy in various neural tasks involving high-frequency functions.

Contribution

The paper proposes SPE, a novel sinusoidal positional encoding that automatically adapts to true frequencies, eliminating manual hyperparameter tuning and enhancing learning of high-frequency functions.

Findings

SPE improves fidelity in 3D view synthesis and TTS tasks.

SPE accelerates training across multiple high-frequency tasks.

SPE performs well without hyperparameter tuning.

Abstract

Fourier features based positional encoding (PE) is commonly used in machine learning tasks that involve learning high-frequency features from low-dimensional inputs, such as 3D view synthesis and time series regression with neural tangent kernels. Despite their effectiveness, existing PEs require manual, empirical adjustment of crucial hyperparameters, specifically the Fourier features, tailored to each unique task. Further, PEs face challenges in efficiently learning high-frequency functions, particularly in tasks with limited data. In this paper, we introduce sinusoidal PE (SPE), designed to efficiently learn adaptive frequency features closely aligned with the true underlying function. Our experiments demonstrate that SPE, without hyperparameter tuning, consistently achieves enhanced fidelity and faster training across various tasks, including 3D view synthesis, Text-to-Speech generation, and 1D regression. SPE is implemented as a direct replacement for existing PEs. Its plug-and-play nature lets numerous tasks easily adopt and benefit from SPE.