Symphony in the Latent Space: Provably Integrating High-dimensional Techniques with Non-linear Machine Learning Models

TL;DR

This paper introduces a novel additive influence model that decouples high-dimensional interactions from non-linear feature interactions, improving forecasting accuracy in complex multivariate time series applications.

Contribution

The paper proposes a new framework that separates the learning of high-dimensional interactions from non-linear features, with provable guarantees and novel algorithms for better forecasting.

Findings

Significantly improved forecasting accuracy over existing methods

Effective embedding of entities in low-dimensional latent space

Robust non-parametric and ensemble learning algorithms

Abstract

This paper revisits building machine learning algorithms that involve interactions between entities, such as those between financial assets in an actively managed portfolio, or interactions between users in a social network. Our goal is to forecast the future evolution of ensembles of multivariate time series in such applications (e.g., the future return of a financial asset or the future popularity of a Twitter account). Designing ML algorithms for such systems requires addressing the challenges of high-dimensional interactions and non-linearity. Existing approaches usually adopt an ad-hoc approach to integrating high-dimensional techniques into non-linear models and recent studies have shown these approaches have questionable efficacy in time-evolving interacting systems. To this end, we propose a novel framework, which we dub as the additive influence model. Under our modeling assumption, we show that it is possible to decouple the learning of high-dimensional interactions from the learning of non-linear feature interactions. To learn the high-dimensional interactions, we leverage kernel-based techniques, with provable guarantees, to embed the entities in a low-dimensional latent space. To learn the non-linear feature-response interactions, we generalize prominent machine learning techniques, including designing a new statistically sound non-parametric method and an ensemble learning algorithm optimized for vector regressions. Extensive experiments on two common applications demonstrate that our new algorithms deliver significantly stronger forecasting power compared to standard and recently proposed methods.