Predicting the Critical Number of Layers for Hierarchical Support Vector Regression

TL;DR

This paper introduces a method to predict the optimal number of layers in hierarchical support vector regression models by analyzing data spectra, enabling better model design before training.

Contribution

The paper proposes a novel approach to determine the critical number of layers in HSVR models using Fourier and DMD spectra analysis, addressing a key gap in model selection.

Findings

Identifies a phase transition in training error related to model depth.

Develops a spectral-based method to predict the critical scale for HSVR.

Enables pre-training determination of the necessary model complexity.

Abstract

Hierarchical support vector regression (HSVR) models a function from data as a linear combination of SVR models at a range of scales, starting at a coarse scale and moving to finer scales as the hierarchy continues. In the original formulation of HSVR, there were no rules for choosing the depth of the model. In this paper, we observe in a number of models a phase transition in the training error -- the error remains relatively constant as layers are added, until a critical scale is passed, at which point the training error drops close to zero and remains nearly constant for added layers. We introduce a method to predict this critical scale a priori with the prediction based on the support of either a Fourier transform of the data or the Dynamic Mode Decomposition (DMD) spectrum. This allows us to determine the required number of layers prior to training any models.