Differentiable Segmentation of Sequences

TL;DR

This paper introduces a differentiable approach to sequence segmentation that allows joint estimation of change points and model parameters using gradient descent, enabling integration into deep learning models.

Contribution

It proposes a novel family of TSP-based warping functions for differentiable segmentation, unifying various segmented models under a gradient-based optimization framework.

Findings

Effective change point detection in COVID-19 spread modeling

Successful application to concept drift in classification

Compatible with standard gradient descent algorithms

Abstract

Segmented models are widely used to describe non-stationary sequential data with discrete change points. Their estimation usually requires solving a mixed discrete-continuous optimization problem, where the segmentation is the discrete part and all other model parameters are continuous. A number of estimation algorithms have been developed that are highly specialized for their specific model assumptions. The dependence on non-standard algorithms makes it hard to integrate segmented models in state-of-the-art deep learning architectures that critically depend on gradient-based optimization techniques. In this work, we formulate a relaxed variant of segmented models that enables joint estimation of all model parameters, including the segmentation, with gradient descent. We build on recent advances in learning continuous warping functions and propose a novel family of warping functions based on the two-sided power (TSP) distribution. TSP-based warping functions are differentiable, have simple closed-form expressions, and can represent segmentation functions exactly. Our formulation includes the important class of segmented generalized linear models as a special case, which makes it highly versatile. We use our approach to model the spread of COVID-19 with Poisson regression, apply it on a change point detection task, and learn classification models with concept drift. The experiments show that our approach effectively learns all these tasks with standard algorithms for gradient descent.