Penalizing Localized Dirichlet Energies in Low Rank Tensor Products

TL;DR

This paper investigates the limitations of global Dirichlet energy regularization in low-rank tensor-product B-spline models and introduces local Dirichlet energy-based regularization to improve robustness and inference.

Contribution

The authors propose a novel local Dirichlet energy regularization method for TPBS models and develop estimators for inference from incomplete samples, enhancing model robustness.

Findings

TPBS models outperform neural networks in overfitting regimes

Global Dirichlet energy regularization can be ineffective due to perfect interpolation scenarios

Local Dirichlet energy regularization improves robustness and inference in TPBS models

Abstract

We study low-rank tensor-product B-spline (TPBS) models for regression tasks and investigate Dirichlet energy as a measure of smoothness. We show that TPBS models admit a closed-form expression for the Dirichlet energy, and reveal scenarios where perfect interpolation is possible with exponentially small Dirichlet energy. This renders global Dirichlet energy-based regularization ineffective. To address this limitation, we propose a novel regularization strategy based on local Dirichlet energies defined on small hypercubes centered at the training points. Leveraging pretrained TPBS models, we also introduce two estimators for inference from incomplete samples. Comparative experiments with neural networks demonstrate that TPBS models outperform neural networks in the overfitting regime for most datasets, and maintain competitive performance otherwise. Overall, TPBS models exhibit greater robustness to overfitting and consistently benefit from regularization, while neural networks are more sensitive to overfitting and less effective in leveraging regularization.