Understanding the Limits of Deep Tabular Methods with Temporal Shift

TL;DR

This paper investigates why deep tabular models struggle with temporal shifts, identifies issues in training protocols, and proposes a Fourier-based temporal embedding to improve robustness and generalization in evolving data environments.

Contribution

It introduces a new training protocol that reduces bias and lag, and a plug-and-play Fourier series-based temporal embedding to better capture temporal dependencies in tabular data.

Findings

Improved model performance with the new training protocol.

Enhanced capture of temporal patterns using Fourier embeddings.

Significant robustness gains under temporal distribution shifts.

Abstract

Deep tabular models have demonstrated remarkable success on i.i.d. data, excelling in a variety of structured data tasks. However, their performance often deteriorates under temporal distribution shifts, where trends and periodic patterns are present in the evolving data distribution over time. In this paper, we explore the underlying reasons for this failure in capturing temporal dependencies. We begin by investigating the training protocol, revealing a key issue in how model selection performs. While existing approaches use temporal ordering for splitting validation set, we show that even a random split can significantly improve model performance. By minimizing the time lag between training data and test time, while reducing the bias in validation, our proposed training protocol significantly improves generalization across various methods. Furthermore, we analyze how temporal data affects deep tabular representations, uncovering that these models often fail to capture crucial periodic and trend information. To address this gap, we introduce a plug-and-play temporal embedding method based on Fourier series expansion to learn and incorporate temporal patterns, offering an adaptive approach to handle temporal shifts. Our experiments demonstrate that this temporal embedding, combined with the improved training protocol, provides a more effective and robust framework for learning from temporal tabular data.