Structured Sparse Modelling with Hierarchical GP

TL;DR

This paper introduces a Bayesian hierarchical Gaussian process model for structured sparse linear regression, utilizing Expectation Propagation for inference, and demonstrates its effectiveness on real spatio-temporal data.

Contribution

It presents a novel hierarchical GP prior for spike and slab coefficients in sparse regression, combined with an EP inference algorithm, advancing structured sparsity modeling.

Findings

Effective modeling of spatio-temporal sparsity

Successful application to real data

Improved inference with hierarchical GP prior

Abstract

In this paper a new Bayesian model for sparse linear regression with a spatio-temporal structure is proposed. It incorporates the structural assumptions based on a hierarchical Gaussian process prior for spike and slab coefficients. We design an inference algorithm based on Expectation Propagation and evaluate the model over the real data.

Tables

Table 1. TABLE I: Performance of signal reconstruction for background subtraction in video

Measure	Hierarchical GP	One-level GP
NMSE	$0.0153$	$0.0392$
F-measure	$0.9870$	$0.9403$

Equations

$${\mathbf{y}}_t\sim \mathcal{N}({\mathbf{y}}_t|\mathbf{X}{\mathbf{\beta }}_t,{\sigma }^2\mathbf{I}).$$

$${\beta }_{it}\sim {\omega }_{it}{\delta }_0({\beta }_{it})+(1-{\omega }_{it})\mathcal{N}({\beta }_{it}|0,{\sigma }_{\beta }^2),$$

$${\omega }_{it}$$

$${\mathbf{\gamma }}_t$$

$${\mathbf{\mu }}_t\sim \mathcal{N}({\mathbf{\mu }}_t|{\mathbf{\mu }}_{t-1},\mathbf{W}),\mathbf{W}(i,j)={\alpha }_W\exp \left(-\frac{(i-j{)}^2}{2l_W^2}\right).$$

Full text

I Introduction

Sparse regression problems arise often in various applications, e.g., model selection, compressive sensing, EEG source localisation and gene modelling [1, 2]. One of the Bayesian approaches to force the coefficients being zeros is the spike and slab prior [3]: each component is modelled as a mixture of spike, that is the delta-function in zero, and slab, that is some vague distribution. Following the Bayesian approach, latent variables that are indicators of spikes are added to the model [4] and the relevant distribution is placed over them [5].

In this model each component is modelled to be spike or slab independently. However, in many applications non-zero elements tend to appear in groups forming an unknown structure: wavelet coefficients of images are usually organised in trees [6], chromosomes have a spatial structure along the genome [2].

We propose an extension of the spike and slab model by imposing a hierarchical Gaussian process (GP) prior on the latent variables. Such hierarchical prior allows to model spatial structural dependencies for coefficients that can evolve in time. The new model is flexible as spatial and temporal dependencies are decoupled by different levels of the hierarchical GP prior.

II Proposed model

The observations $\mathbf{y}_{t}\in\mathbb{R}^{K}$ are collected with the design matrix $\mathbf{X}\in\mathbb{R}^{K\times N}$ from the unknown coefficients $\boldsymbol{\beta}_{t}\in\mathbb{R}^{N}$ at every time moment $t\in[1,\ldots,T]$, with independent noise:

[TABLE]

We consider the case when $K<N$, therefore the problem of recovery of $\boldsymbol{\beta}_{t}$ from $\mathbf{y}_{t}$ is ill-posed and regularisation is required.

Sparsity

The vectors $\boldsymbol{\beta}_{t}$ are assumed to be sparse, that is implemented in the model using the spike and slab approach:

[TABLE]

where $\omega_{it}$ are the latent indicators of spike and slab.

Spatial clustering

Non-zero elements in $\boldsymbol{\beta}_{t}$ are assumed to be clustered in groups at every timestamp. Therefore spatial dependencies for the positions of spikes in $\beta_{it}$ are modelled with the GP:

[TABLE]

GPs specify prior over an unknown structure. This is particularly useful as it allows to avoid a specification of any structural patterns — structural modelling is governed only by the GP covariance function.

Temporal evolution

Clusters of spikes in $\boldsymbol{\beta}_{t}$ are assumed to evolve in time. This evolution is addressed with the hierarchical GP dynamic system model [7]. The mean for the spatial GP changes over time according to the top-level temporal GP:

[TABLE]

This allows to implicitly specify the prior over the transition function of the structure. The rate of the evolution is controlled with the top-level GP covariance function.

The exact posterior of the parameters is intractable, therefore approximate inference methods are required. Inference is based on Expectation Propagation [8] in this paper.

The structural assumptions in sparse models are studied in the literature. The group lasso [9] provides sparse solutions for predefined groups of coefficients. Group constraints for sparse models include smooth relevance vector machines [10], Boltzmann machine prior [11]; spatio-temporal coupling of the parameters [12, 13]. In [14] a spatio-temporal structure is modelled with a one-level GP prior. In contrast to that model the new one introduces an additional level of a GP prior for temporal dependencies, therefore the temporal and spatial structures are decoupled, adding flexibility to the model. The high-level GP controls the change of spike groups in time while the low-level GP allows the local changes within each group.

III Numerical experiments

The performance of the proposed hierarchical GP algorithm is compared with the one-level GP prior introduced in [14]. We apply both algorithms to the problem of object detection in video sequences. The Convoy dataset [15] is used where the frame difference is applied for moving object detection. The sparse observations are obtained as $\mathbf{y}_{t}=\mathbf{X}\boldsymbol{\beta}_{t}$, where $\mathbf{X}$ is the matrix with i.i.d. Gaussian elements, the number of observations is $\approx 40\%$ of the dimension of the hidden signal $\boldsymbol{\beta}_{t}$. This procedure corresponds to compressive sensing observations [16].

The reconstruction results on the sample frames are presented in Figure 1. The obtained performance measures values can be found in Table I. The F-measure compares binary masks computed by thresholding the true vectors $\boldsymbol{\beta}$ and the posterior estimates $\widehat{\boldsymbol{\beta}}$. NMSE represents normalised mean squared error between $\boldsymbol{\beta}$ and $\widehat{\boldsymbol{\beta}}$. The proposed algorithm shows better results in terms of both measures.

IV Conclusion

In this work we propose a new approach for spatial structure modelling in sparse models that allows to capture complex temporal evolution of data patterns. We also develop an efficient inference method based on EP. The numerical experiments show superiority of the proposed model over the current state-of-the-art.