Random at First, Fast at Last: NTK-Guided Fourier Pre-Processing for Tabular DL

TL;DR

This paper introduces a Fourier pre-processing method for tabular deep learning that, guided by NTK analysis, accelerates training and improves performance by transforming inputs into a fixed, kernel-conditioned feature space.

Contribution

It repurposes random Fourier features as a parameter-free pre-processing step, providing theoretical insights and empirical evidence of faster convergence and better results in tabular deep learning.

Findings

Faster convergence of deep networks with Fourier pre-processing

Improved final performance with fewer hyperparameters

Theoretical NTK analysis explains the benefits of the method

Abstract

While random Fourier features are a classic tool in kernel methods, their utility as a pre-processing step for deep learning on tabular data has been largely overlooked. Motivated by shortcomings in tabular deep learning pipelines - revealed through Neural Tangent Kernel (NTK) analysis - we revisit and repurpose random Fourier mappings as a parameter-free, architecture-agnostic transformation. By projecting each input into a fixed feature space via sine and cosine projections with frequencies drawn once at initialization, this approach circumvents the need for ad hoc normalization or additional learnable embeddings. We show within the NTK framework that this mapping (i) bounds and conditions the network's initial NTK spectrum, and (ii) introduces a bias that shortens the optimization trajectory, thereby accelerating gradient-based training. These effects pre-condition the network with a stable kernel from the outset. Empirically, we demonstrate that deep networks trained on Fourier-transformed inputs converge more rapidly and consistently achieve strong final performance, often with fewer epochs and less hyperparameter tuning. Our findings establish random Fourier pre-processing as a theoretically motivated, plug-and-play enhancement for tabular deep learning.