Alignment Distances on Systems of Bags

TL;DR

This paper introduces a kernelized alignment distance for Systems of Bags, improving classification of dynamic visual scenes by effectively handling ambiguities in linear dynamic system parameters.

Contribution

It develops a kernelized alignment distance with a convergent Jacobi-type algorithm, outperforming existing measures in classifying dynamic scenes and textures.

Findings

Outperforms Martin Distance and Maximum Singular Value Distance in classification tasks.

Effective in classifying abstract mean of video sets, surpassing state-of-the-art methods.

Converges to critical points, ensuring reliable computation of the alignment distance.

Abstract

Recent research in image and video recognition indicates that many visual processes can be thought of as being generated by a time-varying generative model. A nearby descriptive model for visual processes is thus a statistical distribution that varies over time. Specifically, modeling visual processes as streams of histograms generated by a kernelized linear dynamic system turns out to be efficient. We refer to such a model as a System of Bags. In this work, we investigate Systems of Bags with special emphasis on dynamic scenes and dynamic textures. Parameters of linear dynamic systems suffer from ambiguities. In order to cope with these ambiguities in the kernelized setting, we develop a kernelized version of the alignment distance. For its computation, we use a Jacobi-type method and prove its convergence to a set of critical points. We employ it as a dissimilarity measure on Systems of Bags. As such, it outperforms other known dissimilarity measures for kernelized linear dynamic systems, in particular the Martin Distance and the Maximum Singular Value Distance, in every tested classification setting. A considerable margin can be observed in settings, where classification is performed with respect to an abstract mean of video sets. For this scenario, the presented approach can outperform state-of-the-art techniques, such as Dynamic Fractal Spectrum or Orthogonal Tensor Dictionary Learning.

Tables

Table 1. TABLE I: Recognition rate on DynTex subsets: 1-NN

	Alpha	Beta	Gamma
LBP-TOP	96.7 %	85.8 %	84.9 %
ASF-TOP	91.7 %	86.4 %	89.4 %
st-TCoF	98.3 %	98.2 %	98.1 %
SoB + Martin	91.7 %	74.7 %	64.4 %
SoB + Max SV	96.7 %	84.0 %	78.4 %
SoB + Align	98.3 %	90.1 %	79.9 %

Table 2. TABLE II: Recognition Rate on DynTex subsets: NCC

	Alpha	Beta	Gamma
DFS	83.6 %	65.2 %	60.8 %
2D+T Curvelet	85.0 %	67.0 %	-
OTDL	86.6 %	69.0 %	64.2 %
SoB + Martin	83.3 %	51.9 %	41.7 %
SoB + Max SV	85.0 %	59.9 %	54.9 %
SoB + Align	88.3 %	75.3 %	67.1  %

Table 3. TABLE III: Confusion matrix for Dyntex Beta (NCC Classification)

	S	V	T	F	Cw	Fo	Sm	E	Tf	R
Sea	18				2
Vegetation		15	1			1		3
Trees		2	8			3	3		3	1
Flags				19						1
Calm water	1				18		1
Fountain		3	2		1	11	1		2
Smoke				4			12
Escalator								6	1
Traffic							1		8
Rotation						2			1	7

Table 4. TABLE IV: Recognition Rate on YUPENN dataset: SoB

Martin	Max SV	Align w/o tuning	Align best
83.3	80.7	86.4 %	88.8 %

Table 5. TABLE V: Recognition Rate on YUPENN dataset: State of The Art

SOE	TSVQ	BoST	st-TCoF
74 %	69 %	85 %	98 %

Equations

$$\begin{array}{rl}{\mathbf{x}}_{t+1}& =\mathbf{A}{\mathbf{x}}_t+{\mathbf{w}}_t,\\ {\mathbf{y}}_t& =\mathbf{\mu }+\mathbf{C}{\mathbf{x}}_t+{\mathbf{v}}_t,\end{array}$$

$$\kappa :{\mathbb{R}}^{p\times M}\times {\mathbb{R}}^{p\times N}\to {\mathbb{R}}^{M\times N}.$$

$$⟨\phi ({\mathbf{y}}_1),\phi ({\mathbf{y}}_2){⟩}_{\mathcal{H}}=\kappa ({\mathbf{y}}_1,{\mathbf{y}}_2).$$

$${\kappa }_{{\chi }^2}({\mathbf{y}}_1,{\mathbf{y}}_2)=\exp \left(-\frac{1}{2}\sum _{i\in {\mathcal{S}}_1\cup {\mathcal{S}}_2}\frac{(y_{1,i}-y_{2,i}{)}^2}{y_{1,i}+y_{2,i}}\right)$$

$$\begin{array}{rl}\mathbf{F}:{\mathbb{R}}^{M\times N}& \to {\mathcal{H}}^N,\\ \mathbf{D}& \mapsto \left[\begin{array}{c}{\sum }_{i=1}^M{d}_{i,1}{\mathbf{f}}_i\cdots {\sum }_{i=1}^M{d}_{i,N}{\mathbf{f}}_i\end{array}\right].\end{array}$$

$${\mathbf{F}}^{\top }\mathbf{G}=\left[\begin{array}{ccc}⟨{\mathbf{f}}_1,{\mathbf{g}}_1{⟩}_{\mathcal{H}}& \cdots & ⟨{\mathbf{f}}_1,{\mathbf{g}}_n{⟩}_{\mathcal{H}}\\ ⋮& \ddots & ⋮\\ ⟨{\mathbf{f}}_m,{\mathbf{g}}_1{⟩}_{\mathcal{H}}& \cdots & ⟨{\mathbf{f}}_m,{\mathbf{g}}_n{⟩}_{\mathcal{H}}\end{array}\right].$$

$$\Vert \mathbf{F}{\Vert }_F=\sqrt{\mathrm{tr}({\mathbf{F}}^{\top }\mathbf{F})}.$$

$$\begin{array}{rl}{\mathbf{x}}_{t+1}& =\mathbf{A}{\mathbf{x}}_t+{\mathbf{w}}_t,\\ \phi ({\mathbf{y}}_t)& =\mathbf{\mu }+\mathbf{C}{\mathbf{x}}_t+{\mathbf{v}}_t,\end{array}$$

$$\{{\mathbf{C}}_{:,1},\dots ,{\mathbf{C}}_{:,n}\}\cup \{\mathbf{\mu }\}\subset \mathrm{span}\left(\{\phi ({\mathbf{y}}_1),\dots ,\phi ({\text{y}}_N)\}\right)$$

$$\mathbf{\Phi }=\left[\begin{array}{c}\phi ({\mathbf{y}}_1)\cdots \phi ({\text{y}}_N)\end{array}\right]\in {\mathcal{H}}^N.$$

$$\mathbf{C}=\mathbf{\Phi }\mathbf{\alpha },\mathbf{\mu }=\mathbf{\Phi }\mathbf{\beta }.$$

$$\begin{array}{rl}{\mathbf{x}}_{t+1}& =\mathbf{A}{\mathbf{x}}_t+{\mathbf{w}}_t,\\ \phi ({\mathbf{y}}_t)& =\mathbf{\Phi }(\mathbf{\beta }+\mathbf{\alpha }{\mathbf{x}}_t)+{\mathbf{v}}_t.\end{array}$$

$$\begin{array}{rl}{\mathbf{C}}_i^{\top }{\mathbf{C}}_j& ={\mathbf{\alpha }}_i^{\top }\kappa ({\mathbf{Y}}_i,{\mathbf{Y}}_j){\mathbf{\alpha }}_j,\\ {\mathbf{\mu }}_i^{\top }{\mathbf{\mu }}_j& ={\mathbf{\beta }}_i^{\top }\kappa ({\mathbf{Y}}_i,{\mathbf{Y}}_j){\mathbf{\beta }}_j,i,j\in \{1,2\}\end{array}$$

$$\Vert \mathbf{A}{\Vert }_2<1.$$

$$\begin{array}{rl}{d}_F({\Theta }_1,{\Theta }_2){)}^2=& {\lambda }_A\Vert {\mathbf{A}}_1-{\mathbf{A}}_2{\Vert }_F^2+\Vert {\mathbf{C}}_1-{\mathbf{C}}_2{\Vert }_F^2\\ & +{\lambda }_{\mu }\Vert {\mathbf{\mu }}_1-{\mathbf{\mu }}_2{\Vert }^2\end{array}$$

$$d_F({\Theta }_1,{\Theta }_2{)}^2=\tau ({\Theta }_1,{\Theta }_2)-2\rho ({\Theta }_1,{\Theta }_2),$$

$$\begin{array}{rl}\tau ({\Theta }_1,{\Theta }_2)=& {\lambda }_A(\mathrm{tr}({\mathbf{A}}_1^{\top }{\mathbf{A}}_1)+\mathrm{tr}({\mathbf{A}}_2^{\top }{\mathbf{A}}_2))\\ & +\mathrm{tr}({\mathbf{\alpha }}_1^{\top }\kappa ({\mathbf{Y}}_1,{\mathbf{Y}}_1){\mathbf{\alpha }}_1)\\ & +\mathrm{tr}({\mathbf{\alpha }}_2^{\top }\kappa ({\mathbf{Y}}_2,{\mathbf{Y}}_2){\mathbf{\alpha }}_2)+{\lambda }_{\mu }\Vert {\mathbf{\mu }}_1-{\mathbf{\mu }}_2{\Vert }^2\end{array}$$

$$\rho ({\Theta }_1,{\Theta }_2)={\lambda }_A\mathrm{tr}({\mathbf{A}}_1^{\top }{\mathbf{A}}_2)+\mathrm{tr}({\mathbf{\alpha }}_1^{\top }\kappa ({\mathbf{Y}}_1,{\mathbf{Y}}_2){\mathbf{\alpha }}_2).$$

$$\mathbf{P}\cdot \left(\mathbf{A},\mathbf{Y},\mathbf{\alpha },\mathbf{\beta }\right)=\left({\mathbf{P}}^{-1}\mathbf{AP},\mathbf{Y},\mathbf{\alpha }\mathbf{P},\mathbf{\beta }\right).$$

$${\mathbf{C}}^{\top }\mathbf{C}={\mathbf{\alpha }}^{\top }\kappa (\mathbf{Y},\mathbf{Y})\mathbf{\alpha }={\mathbf{I}}_n.$$

$$\begin{array}{rl}{\mathcal{K}}_{n,p,\kappa }=& \{(\mathbf{A},\mathbf{Y},\mathbf{\alpha },\mathbf{\beta })\\ & \in {\mathbb{R}}^{n\times n}\times {\mathbb{R}}^{p\times N}\times {\mathbb{R}}^{N\times n}\times {\mathbb{R}}^N|N\in \mathbb{N},\\ & \Vert \mathbf{A}{\Vert }_2<1,{\mathbf{\alpha }}^{\top }\kappa (\mathbf{Y},\mathbf{Y})\mathbf{\alpha }={\mathbf{I}}_n\}.\end{array}$$

$$d_F({\Theta }_1,\mathbf{Q}\cdot {\Theta }_2{)}^2=d_F({\mathbf{Q}}^{\top }\cdot {\Theta }_1,{\Theta }_2{)}^2\forall \mathbf{Q}\in O(n)$$

$$\begin{array}{rl}\tau ({\Theta }_1,{\Theta }_2)=& {\lambda }_A(\mathrm{tr}({\mathbf{A}}_1^{\top }{\mathbf{A}}_1)+\mathrm{tr}({\mathbf{A}}_2^{\top }{\mathbf{A}}_2))\\ & +{\lambda }_{\mu }\Vert {\mathbf{\mu }}_1-{\mathbf{\mu }}_2{\Vert }^2+2n.\end{array}$$

$$\{({\Theta }_1,{\Theta }_2)|\exists \mathbf{Q}\in O(n)s.t.{\Theta }_1=\mathbf{Q}\cdot {\Theta }_2\}.$$

$$\begin{array}{rl}{d}_{\mathcal{O}}({\Theta }_1,{\Theta }_2{)}^2=& \underset{\mathbf{Q}\in O(n)}{\min }{d}_F({\Theta }_1,\mathbf{Q}\cdot {\Theta }_2{)}^2\\ =& \tau ({\Theta }_1,{\Theta }_2)-2\underset{\mathbf{Q}\in O(n)}{\max }\rho ({\Theta }_1,\mathbf{Q}\cdot {\Theta }_2).\end{array}$$

$$\begin{array}{rl}{d}_{\mathcal{O}}({\Theta }_1,{\Theta }_2)& =\underset{\mathbf{Q}\in O(n)}{\min }({\Theta }_1,\mathbf{Q}\cdot {\Theta }_2)\\ & =\underset{\mathbf{Q}\in O(n)}{\min }{d}_F({\mathbf{Q}}^{\top }\cdot {\Theta }_1,{\Theta }_2)\\ & =\underset{\mathbf{Q}\in O(n)}{\min }{d}_F({\Theta }_2,{\mathbf{Q}}^{\top }\cdot {\Theta }_1)\\ & =d_{\mathcal{O}}({\Theta }_2,{\Theta }_1).\end{array}$$

$$\begin{array}{rl}& d_{\mathcal{O}}({\Theta }_1,{\Theta }_2)+d_{\mathcal{O}}({\Theta }_2,{\Theta }_3)\\ =& d_F({\mathbf{Q}}_{12}^{\top }\cdot {\Theta }_1,{\Theta }_2)+d_F({\Theta }_2,{\mathbf{Q}}_{23}\cdot {\Theta }_3)\\ \ge & d_F({\mathbf{Q}}_{12}^{\top }\cdot {\Theta }_1,{\mathbf{Q}}_{23}\cdot {\Theta }_3)=d_F({\Theta }_1,{\mathbf{Q}}_{12}{\mathbf{Q}}_{23}\cdot {\Theta }_3)\\ \ge & d_{\mathcal{O}}({\Theta }_1,{\Theta }_3).\end{array}$$

$$\begin{array}{rl}\rho ({\Theta }_1,& \mathbf{Q}\cdot {\Theta }_2)={\lambda }_A\mathrm{tr}({\mathbf{A}}_1^{\top }{\mathbf{Q}}^{\top }{\mathbf{A}}_2\mathbf{Q})\\ & +\mathrm{tr}({\mathbf{\alpha }}_1^{\top }\kappa ({\mathbf{Y}}_1,{\mathbf{Y}}_2){\mathbf{\alpha }}_2\mathbf{Q}).\end{array}$$

$$\prod _{p=1}^n\prod _{q=p+1}^n{\mathbf{G}}_{p,q}(c,s),$$

$${\mathbf{G}}_{p,q}(c,s)=\left[\begin{array}{ccccccc}1& \cdots & 0& \cdots & 0& \cdots & 0\\ ⋮& \ddots & ⋮& & ⋮& & ⋮\\ 0& \cdots & c& \cdots & s& \cdots & 0\\ ⋮& & ⋮& \ddots & ⋮& & ⋮\\ 0& \cdots & -s& \cdots & c& \cdots & 0\\ ⋮& & ⋮& & ⋮& \ddots & ⋮\\ 0& \cdots & 0& \cdots & 0& \cdots & 1\end{array}\right].$$

Full text

I Introduction

Many of the most successful classification frameworks for videos employ generative models of visual processes, where videos are modeled as distributions of descriptors. Prominent examples are Local Binary Patterns in Three Orthogonal Plains (LBP-TOP) [1] and Bags of Systems (BoS) [2, 3, 4]. Typically, the descriptors in question are local in the spatiotemporal domain, but the distributions are global, neglecting their spatial and temporal order. This has proven successful in many classification problems, however, there are scenarios where such a procedure could turn out suboptimal for several reasons. Spatiotemporally local descriptors are supposed to capture the dynamics locally in space and time. This is sensible for the recognition of dynamic textures on a small scale, but for large-scale dynamic textures or real-world dynamic scenes, the temporal dynamics on a global scale is a more distinguishing feature: for instance, a traffic scene is more characterized by the appearance and disappearance of vehicles than by the movement of the trees on the roadside. Furthermore, breaking up the global temporal order of the overall visual process can be problematic in cases where the appearance of the frames changes as a whole over the course of time, e.g. when observing outdoor scenes under changing weather conditions. In such cases, the video can have semantic features that may get lost by destroying the temporal order. On the other hand, breaking up the global spatial order contradicts the everyday observation that looking at single frames of a visual process often suffices to distinguish between dynamic scenes or high-resolution textures. In such cases, employing well established still image feature extraction methods on the isolated frames can be a sensible step in the feature extraction of the overall visual process.

Remarkably, for still image textures and still image scenes, generative, distribution-based models have proven their efficacy at several occasions. Texture images, being often thought of as realizations of stochastic processes [5], have a long standing tradition of distribution based models [6, 7, 8, 9]. Meanwhile, the concept of BoS is inspired by the Bag of Words (BoW) [10] paradigm, where images are described by the frequency of previously learned features contained in them. BoW based methods have been successfully employed in the task of distinguishing still-image scenes [11, 12]. Moreover, outstanding performance on dynamic scene recognition can be achieved when ”bags” of spatiotemporal features are computed on a temporally local scale. For instance, the authors of [13] propose computing several temporally localized bags of oriented filter response features from videos and produce outstanding results. The classification is performed by a majority vote that encompasses all of the computed bags in a video.

We conclude that, employing generative, distribution based models for the individual frames, or alternatively, localized collections of frames of visual processes such as dynamic textures and dynamic scenes is a promising approach. In the classical case, including many BoW based approaches, these models are histograms. Alternatively, they can also be represented by Fisher Vectors [14] or statistical moments of a parametrized distribution model [9]. We will focus on the classical view in the following, even though the proposed methods can be easily generalized to the other perspectives. This leads to the assumption that individual frames of a visual process can be well modeled by histograms. However, a model treating visual processes as sequences of histograms neglects their temporal dynamics and thus fails to generalize from one sample to the whole process.

A remedy is to derive a dynamic model for the temporal evolution of the histograms.

We refer to such an approach as System of Bags (SoB) in reference to the successful Bags of Systems. Unlike Bags of Systems, where dynamic systems on spatially and temporally local scale are computed and ”bags” thereof on a global scale are created, we compute histograms on a temporally local but spatially global scale and model their evolution over time by means of a dynamic system. Fig. 1 visualizes this distinction. The left side of the picture shows the procedure of BoS and related generative models. A video is first divided into spatiotemporal cubes and for each cube, a word, e.g. a stack of dynamic system parameters $\Theta_{i},i\in\{1,2,3,\dots\}$, is computed. Afterwards, a descriptor distribution over the complete video is estimated as the final representation of the video. The right side of the figure shows the procedure of generating a SoB. Distributions of descriptors are computed on temporally local but a spatially global scale. The temporal progression of these distributions is successively modeled as a dynamic system $\Theta$. Typically, the terms bag and word imply the usage of a learned codebook. However, this technique can be applied independently of codebooks and thus will be referred to as SoBs even in cases where the histograms were not computed with respect to a codebook.

I-A Related Work and our Contribution

The temporal evolution of histograms can not be well described by linear dynamic systems. Instead, we employ the concept of Kernel Linear Dynamic Systems (KLDS) which model the observations in a kernel feature space. The parameters describing the KLDS are the representations of the visual processes employed in his work. Using these descriptors in the context of supervised learning requires the definition of a dissimilarity measure. The available literature on KLDSs offers a manageable number of dissimilarity measures that perform insufficiently on the SoB descriptors used in this work, as will be shown in the experimental Section IV, which may be due to the fact that they put too much emphasis on the dynamic part of the KLDS parameters, whereas in the setting discussed in this work, static information, i.e. the information not related to the temporal context has a considerable significance. The KLDS itself was introduced in [15] and motivated by the recognition of dynamic textures. As a dissimilarity measure, a kernelized version of the widely adopted Martin distance [16] was applied.

Modeling visual processes as SoBs i.e. KLDSs of histograms was employed in [17] for the classification of human actions. The authors modeled videos of human actions as streams of histograms of optical flow (HOOF). As a similarity measure, a Binet-Cauchy Maximum Singular Value kernel was applied. The work was further enhanced in [18], where human interactions were targeted. More generally, bags and histograms as representations of samples of multidimensional time signals were used both for human action [19] and dynamic scene [13] recognition, but the temporal order was neglected in both cases.

The novelty of this work is to explore the framework of SoB in the context of dynamic scene and large-scale dynamic texture recognition and to develop an appropriate dissimilarity measure. To this end, we adopt the framework of the alignment distance from [20] and develop a kernelized version suitable for the comparison of SoBs. Unlike the mentioned dissimilarity measures for SoBs, impact of the static and dynamic components can be chosen depending on the employed generative image model. Besides, its property of being the square of a metric and its simple definition based on the Frobenius distance allows for the definition of Fréchet means [21]. This is crucial for the classification via the nearest class center (NCC) [22]. A part of this work is dedicated to the computation of abstract means of sets of KLDSs for avoiding the memory burden of nearest neighbor (1-NN) classification.

The computation of alignment distances involves modeling the appearance of a visual process as an equivalence class of points on a Stiefel manifold. This is closely related to Grassmann based models. The authors of [23] model visual processes as points on kernelized Grassmann manifolds, while our approach employs kernelized Stiefel manifolds in a similar manner. In particular, the authors propose to model the spaces of video frames, or of temporally localized features extracted from them, as sparse codes via points on Grassmann manifolds. Furthermore, a kernel dictionary learning approach for dynamic texture recognition was employed in [24].

I-B Notation

Boldfaced uppercase letters, e.g. $\mathbf{A},\mathbf{C}$ denote matrices and boldfaced lowercase letters e.g $\mathbf{v},\mathbf{w}$ denote vectors. Bold italic letters like $\boldsymbol{a}$ or $\boldsymbol{\alpha}$ refer to any members of metric spaces. The identity matrix in $\mathbb{R}^{n\times n}$ is written as $\mathbf{I}_{n}$ and a vector of ones as $\mathbf{1}_{n}$. For submatrices, the colon notation of Matlab is adopted, e.g. $\mathbf{V}_{1:n,1:n}$ for the left-upper $n\times n$ square submatrix of $\mathbf{V}$. Singular value and Eigenvalue decompositions are assumed to be sorted in a descending manner.

II Systems of Bags

II-A Visual Processes as Streams of Histograms

Given an ordered set of vectorized samples $\{\mathbf{s}_{1}\cdots\mathbf{s}_{N}\}$ of a multidimensional signal in time, let us assume that temporally local dynamics are negligible for the assignment of a class. This assumption can not be kept up when it comes to determining the sense of rotation of a wheel or a windmill from video footage but can be usually assumed to be valid for telling one scene video apart from another. In such cases, it is sensible to convert the signal to a matrix $\begin{bmatrix}\mathbf{y}_{1}\cdots\mathbf{y}_{N}\end{bmatrix}\in\mathbb{R}^{p\times N}$ of temporally localized feature vectors that capture the distinguishable characteristics on a spatially global but temporally local scale.

Classification requires the generalization from one set of signals of one class to other signals of the same class. Since the entities in our model are temporally ordered sets of features, our aim is to learn a generative model that describes how the feature vectors develop over time. For many image classification scenarios, histogram-based feature vectors have proven successful. We do not pose any constraints on the histograms except that their entries are nonnegative and their $\ell_{1}$-norm is 1. Let $\mathbf{Y}=\begin{bmatrix}\mathbf{y}_{1}\cdots\mathbf{y}_{N}\end{bmatrix}\in\mathbb{R}^{p\times N}$ be a sample matrix of histogram vectors that were observed from a visual process over time. A common temporal model - one that is particularly popular in the modeling of dynamic textures [25] - is a linear dynamic system (LDS) typically modeled as

[TABLE]

where $\boldsymbol{\mu}\in\mathbb{R}^{p}$ is the expected value of the observations $\{\mathbf{y}_{1},\mathbf{y}_{2},\dots\}$ and $\mathbf{C}\in\mathbb{R}^{p\times n}$ is the observation matrix which, together with $\boldsymbol{\mu}$ maps the internal state space vector $\mathbf{x}_{t}\in\mathbb{R}^{n}$ to its respective observation $\mathbf{y}\in\mathbb{R}^{p}$ at time $t$, unperturbed by noise. The observation noise vector $\mathbf{v}_{t}$ describes the model error. In the state space, the state transition matrix $\mathbf{A}\in\mathbb{R}^{n\times n}$ models the expected temporal evolution of the state vector and the term $\mathbf{w}_{t}$ accounts for the process noise. The noise terms are assumed to be i.i.d. Gaussian. The parameters $\boldsymbol{\mu}$, $\mathbf{A}$ and $\mathbf{C}$ describe the predictable part of the dynamics of the system and are thus a natural choice for a feature representation of it.

II-B Kernelized Linear Dynamic Systems

Sets of histograms can not be well modeled by linear vector spaces due to the intrinsic structure of histogram manifolds [17] and thus a non-linear model is preferred. Let the function $\varphi:\mathbb{R}^{p}\rightarrow\mathcal{H}$ be a feature space mapping of histograms to a feature space corresponding to an appropriate histogram kernel

[TABLE]

In other words, for two histograms $\mathbf{y}_{1},\mathbf{y}_{2}\in\mathbb{R}^{p}$, the inner product fo their feature space mapping in the Hilbert space $\mathcal{H}$ can be written as

[TABLE]

We assume that $\kappa$ operates on matrices and returns a matrix of kernel values for each pair of columns of the input matrices. A number of kernels are available for probabilistic models and, in particular, histograms. Among the most popular are the Bhattacharrya kernel, the histogram intersection kernel [26] and the $\chi^{2}$-kernel. We will restrict ourselves to the $\chi^{2}$-kernel which is defined as

[TABLE]

for a pair of histograms $\mathbf{y}_{1},\mathbf{y}_{2}$, where $\mathcal{S}_{1},\mathcal{S}_{2}$ denote the supports of $\mathbf{y}_{1}$ and $\mathbf{y}_{2}$, respectively.

Since kernel feature spaces are separable [27], we can think of linear operators that map from real-valued euclidean vectors to $\mathcal{H}$ as matrices, where by matrix a touple of elements in $\mathcal{H}$ is meant. Specifically, a matrix $\mathbf{F}=\begin{bmatrix}\boldsymbol{f}_{1}\cdots\boldsymbol{f}_{M}\end{bmatrix}\in\mathcal{H}^{M}$ represents the operator

[TABLE]

Beyond that, for two matrices $\mathbf{F}\in\mathcal{H}^{m},\mathbf{G}\in\mathcal{H}^{n}$, we define the product

[TABLE]

From this follow the definitions of the respective Frobenius inner product $\mathrm{tr}(\mathbf{F}^{\top}\mathbf{G})$ for $M=N$ and the Frobenius norm

[TABLE]

A KLDS is defined as

[TABLE]

Unlike equation (1), the observation $\mathbf{y}_{t}$ is not modeled directly, but in terms of its feature space mapping $\varphi(\mathbf{y}_{t})\in\mathcal{H}$. The matrix $\mathbf{C}$, denoted feature space observer in the following, is typically not directly available, since $\varphi$ is usually not given and described implicitly via the kernel trick. The same holds for the feature space bias $\boldsymbol{\mu}$. However, given a set of observations $\{\mathbf{y}_{1},\dots\textbf{y}_{N}\}$ such that

[TABLE]

holds, let us define

[TABLE]

Then there exists a coefficient matrix $\boldsymbol{\alpha}\in\mathbb{R}^{N\times n}$ and a coefficient vector $\boldsymbol{\beta}\in\mathbb{R}^{N}$ for which the following relations are valid.

[TABLE]

Thus, the KLDS (8) can be equivalently described via $\mathbf{A}$ and $\boldsymbol{\alpha}$, along with $\boldsymbol{\beta}$ and a sample matrix $\mathbf{Y}=\begin{bmatrix}\mathbf{y}_{1}\cdots\mathbf{y}_{N}\end{bmatrix}$ as follows.

[TABLE]

Given two feature space observers $\mathbf{C}_{1},\mathbf{C}_{2}$ and feature space biases $\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}$, described by the sample matrices $\mathbf{Y}_{1},\mathbf{Y}_{2}$ and the coefficient parameters $\boldsymbol{\alpha}_{1},\boldsymbol{\alpha}_{2},\boldsymbol{\beta}_{1},\boldsymbol{\beta}_{2}$, the relations

[TABLE]

can be concluded. For the kernel (4), the set of feature representations of histograms is bounded. This follows directly from the fact that for any histogram, the canonical norm induced by (4) is $1$. We can thus assume, that the underlying system is stable. This is imposed by constraining the spectral norm of the state transition matrix to

[TABLE]

An algorithm based on Kernel PCA to extract $\mathbf{A}$ and $\boldsymbol{\alpha}$ from a set of observations of a system was provided in [15]. Algorithm 1 incorporates the procedure. The state space dimension $n$ has to be fixed manually. It should be large enough to capture the variation of the data which can be measured by observing the magnitudes of the eigenvalues of the Gram matrix $\mathbf{K}$, and small enough to make the computation of the state transition matrix $\mathbf{A}$ feasible.

III Kernelized Alignment Distances

III-A State Space Bases and Invariance

Given two KLDSs described by $\Theta_{1}=\left(\mathbf{A}_{1},\mathbf{Y}_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\beta}_{1}\right)$ and ${\Theta}_{2}=\left(\mathbf{A}_{2},\mathbf{Y}_{2},\boldsymbol{\alpha}_{2},\boldsymbol{\beta}_{2}\right)$, respectively, one of the most elementary machine learning tasks is comparing the two by means of a dissimilarity measure. A natural choice is a linear combination of the squared Frobenius distances of the dynamic parameters:

[TABLE]

Note that (15) differs from the formulation in [20] by the term $\|\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2}\|^{2}$ and the missing of the process noise covariance. Since we assume that the temporally local histograms are already quite discriminative for each visual process, it is sensible to consider the feature space bias in the distance measure. By contrast, the process noise is of little importance and will be neglected. The parameters $\lambda_{A}$ and $\lambda_{\mu}$ are real and positive. They incorporate the discriminatory power of each aspect of deterministic part of the dynamics. Their choice is always a matter of consideration and depends on the specific problem. When a sufficient number of training samples are available, cross validation can be used for finding the best values. Some general guidelines can be inferred from the roles the KLDS parameters play in the motion equation (8). In particular, large values for $\lambda_{A}$ should be used for scenarios, where the appearance has little discriminatory power, e.g. when similar objects with diverse movements are to be distinguished. A value well above $1$ for $\lambda_{\mu}$ should be employed, when isolated frames of the videos provide much discriminatory power and a value close to [math] for the opposite case.

To facilitate computations, we rewrite (15) in terms of the trace product and split it up as

[TABLE]

with

[TABLE]

and

[TABLE]

The advantage of (15) is that the Frobenius distance is well studied and easy to interpret. However, this choice has the drawback that it is ambiguous. To see this, let $\mathbf{P}\in\mathbb{R}^{n\times n}$ be invertible. For an arbitrary parameter touple $\left(\mathbf{A},\mathbf{Y},\boldsymbol{\alpha},\boldsymbol{\beta}\right)$, consider the transformation

[TABLE]

Substituting it into (12) and neglecting the noise terms indicates that it describes the same dynamics as $(\mathbf{A},\mathbf{Y},\boldsymbol{\alpha},\boldsymbol{\beta})$, but $d_{F}\left((\mathbf{A},\mathbf{Y},\boldsymbol{\alpha},\boldsymbol{\beta}),\mathbf{P}\cdot(\mathbf{A},\mathbf{Y},\boldsymbol{\alpha},\boldsymbol{\beta})\right)^{2}$ does not vanish in general. This is undesirable, since a distance measure should account for ambiguities of particular representations. It is possible to partially accommodate these ambiguities by imposing the constraint that the columns of the feature space observer $\mathbf{C}$ must be orthogonal, i.e

[TABLE]

For any representation $\left(\mathbf{A},\mathbf{Y},\boldsymbol{\alpha},\boldsymbol{\beta}\right)$, a change of state space basis $\mathbf{P}$ can be found such that the transformation (19) satisfies this constraint. Furthermore, this constraint is fulfilled for any representation extracted with Algorithm 1. We formalize it by defining the set of valid, stable KLDSs as

[TABLE]

For any $\Theta\in\mathcal{K}_{n,p,\kappa}$, a transformation $\mathbf{Q}\cdot\Theta$ is a member of $\mathcal{K}_{n,p,\kappa}$ if and only if $\mathbf{Q}$ is orthogonal. Furthermore, for any two descriptors $\Theta_{1}=(\mathbf{A}_{1},\mathbf{Y}_{1},\boldsymbol{\alpha}_{1},\boldsymbol{\beta}_{1})\in\mathcal{K}_{n,p,\kappa}$ and $\Theta_{2}=(\mathbf{A}_{2},\mathbf{Y}_{2},\boldsymbol{\alpha}_{2},\boldsymbol{\beta}_{2})\in\mathcal{K}_{n,p,\kappa}$, it can be shown that

[TABLE]

holds and equation (17) simplifies to

[TABLE]

III-B Alignment Distance for KLDSs

In the following we define a non-ambiguous dissimilarity measure on KLDSs. The remaining ambiguity of $d_{F}^{2}$ with respect to orthogonal transformations suggests operating on equivalence classes of $\mathcal{K}_{n,p,\kappa}$, rather than itself. Let $\mathcal{OK}_{n,p,\kappa}$ denote the quotient of $\mathcal{K}_{n,p,\kappa}$ induced by the equivalence relation

[TABLE]

A dissimilarity measure on $\mathcal{OK}_{n,p,\kappa}$ is

[TABLE]

The square root $d_{\mathcal{O}}$ of (25) is actually a metric on $\mathcal{OK}_{n,p,\kappa}$. To see the positive definiteness, we first observe that its square $d_{\mathcal{O}}^{2}$ is real and non-negative by definition. Neither can it be zero unless there is an orthogonal $\mathbf{Q}$, such that $\Theta_{1}=\mathbf{Q}\cdot\Theta_{2}$, since otherwise $2\rho$ would be smaller than $\tau$ in (25), due to the Cauchy-Schwarz inequality. But this makes $\Theta_{1}$ and $\Theta_{2}$ member of the same equivalence class of (24). The symmetry property follows directly form (22):

[TABLE]

As for the triangle inequality, consider the three systems $\Theta_{1},\Theta_{2}$, and $\Theta_{3}$, with $\mathbf{Q}_{12}$ and $\mathbf{Q}_{23}$ being the orthogonal matrices that solve (25) for the respective pairs. Since $d_{F}$ is a metric on $\mathcal{K}_{n,p,\kappa}$, we can conclude the relation

[TABLE]

We refer to $d_{\mathcal{O}}$ as the alignment metric, as opposed to its square $d_{\mathcal{O}}^{2}$, alignment distance. The reason for this distinction is that the metric is helpful for a mathematical interpretation, while for the definition and implementation of the algorithms only the distance is of interest.

III-C Jacobi-type Method for Computing the Alignment Distance

This subsection aims at solving (25), which boils down to finding an orthogonal maximizer of

[TABLE]

The set of $O(n)$ is not connected, but consists of the two connected components $SO(n)$ and $O(n)\setminus SO(n)$. For the sake of simplicity, we treat these two cases separately. The authors of [28] propose to compute the alignment distance of classical LDSs by writing matrices in $SO(n)$ as products of Givens rotations, i.e. as

[TABLE]

in which $\mathbf{G}_{p,q}(c,s)$ describes a matrix that performs a rotation in the plane spanned by the coordinate axes $p$ and $q$:

[TABLE]

The real numbers $s,c$ denote the sine and cosine of the rotation angle and appear in the rows and columns with the indexes $p$ and $q$, respectively. Consequentially, they conform

[TABLE]

Now the optimization problem (25) can be approached by maximizing $\rho$ for each $p$ and $q$ individually [28]:

In each iteration, Algorithm 2 sweeps through all possible combinations of two-dimensional rotations and determines the sine minimizing the cost function (25) for each one of them. It repeats the procedure until a complete sweep does not significantly alter the cost function. The scalar optimization problem

[TABLE]

can be solved analytically. With $\Theta=(\mathbf{A},\mathbf{Y},\boldsymbol{\alpha},\boldsymbol{\beta})$ and $\tilde{\Theta}=(\tilde{\mathbf{A}},\tilde{\mathbf{Y}},\tilde{\boldsymbol{\alpha}},\tilde{\boldsymbol{\beta}})$, it can be reformulated as

[TABLE]

It can be observed that (33) is quadratic in $c$ and $s$. Since constant offsets are irrelevant for maximization, we can write it as

[TABLE]

where the factors $k_{0},\dots,k_{4}$ can be determined by writing the trace products as sums of matrix elements [28] and eliminating the constant offset. Substituting the second constraint yields

[TABLE]

If the optimum is not at the boundaries of $[-1,1]$, it is at a critical point, i.e. at a point with a vanishing derivative. Setting the first derivative to 0 produces

[TABLE]

The reformulation is given by multiplying with $\pm\sqrt{1-s^{2}}$ and applying the third binomial formula. The resulting formula is the quartic equation

[TABLE]

for which closed-form solution formulas exist [28] or Newton type methods can be applied. If the global maximum is not unique, the candidate with the smallest value for $|s|$ must be selected. Once $\hat{s}$ is found, so is $|\hat{c}|=\sqrt{1-\hat{s}^{2}}$. The sign of $\hat{c}$ is finally determined by evaluating $\rho(\Theta,\mathbf{G}_{p,q}(c,\hat{s})\cdot\tilde{\Theta})$ for $c=|\hat{c}|$ and $c=-|\hat{c}|$.

While we need the optimizer on $O(n)$, Algorithm 2 searches only one of its connected components. Thus, a necessary condition for the solution of (25) is that the determinant of the determinant of the initialization $\mathbf{Q}$ has the correct sign. Practically, this implies that the algorithm needs to be run at least twice with initializations both in $SO(n)$ and $O(n)\setminus SO(n)$.

III-D Convergence Properties

The purpose of this subsection is to take a closer a look at Algorithm 2 from the point of view of convergence and numerical complexity. For a discussion of convergence properties of Jacobi-type methods in general, the reader is referred to [29]. The optimization problem (25) is non-convex due to the restriction of $\mathbf{Q}$ to be in $O(n)$. Although Jacobi methods perform reasonably well in practice, unfortunately we lack a proof of global convergence for this particular method.

Let

[TABLE]

be the cost function value at iteration $l$ of Algorithm 2 for any initialization. The sequence $(f(\mathbf{Q}^{(l)}))_{l\in\mathbb{N}}$ is bounded above and increases monotonically. Hence, it converges to a limit $\hat{f}$. In other words, the algorithm always terminates. The execution time is dictated by the solution of (37), for which fast and robust methods exist but which has to be performed $(n^{2}-n)/2$ times per iteration.

An essential question is, what the algorithm returns. Since we can not guarantee that it always returns a global maximum, we want at least to make sure that the result is (close to) a critical point of $f$. This is shown in Theorem 1. One claim that is made is that $f$ can be further increased at a non-critical point $\mathbf{Q}$ on $O(n)$ by multiplying $\mathbf{Q}$ with a Givens rotation. The claim follows from the observation that for any $\mathbf{Q}\in O(n)$, the set

[TABLE]

spans the tangent space of $O(n)$ at $\mathbf{Q}$ which consists of products of $\mathbf{Q}$ with skew-symmetric matrices[30].

Theorem 1**.**

Let $(\mathbf{Q}^{(l)})_{l\in\mathbb{N}}$ be the sequence of orthogonal matrices generated by the outer while loop of Algorithm 2. The sequence $(\mathbf{Q}^{(l)})_{l\in\mathbb{N}}$ is bounded with respect to the Frobenius norm. If the order of each sweep is chosen randomly, each accumulation point is almost certainly a critical point of the cost function $f$.

Proof.

Since $O(n)$ is bounded w.r.t the Frobenius norm, so is $(\mathbf{Q}^{(l)})_{l\in\mathbb{N}}$.

Let $\hat{f}$ be the limit of the cost function value sequence $(f(\mathbf{Q}^{(l)}))_{l\in\mathbb{N}}$ and $o(l)$ the monotonically increasing index mapping belonging to a convergent subsequence of $(\mathbf{Q}^{(l)})_{l\in\mathbb{N}}$. The sequence $(\mathbf{Q}^{(o(l))})_{l\in\mathbb{N}}$ converges to a limit point $\hat{\mathbf{Q}}\in{O(n)}$ with $f(\hat{\mathbf{Q}})=\hat{f}$. Assume that $\hat{\mathbf{Q}}$ is not a critical point of $f$. This implies, that we can find a Givens rotation $\mathbf{G}_{p,q}(c,s)$, such that

[TABLE]

Since the sequence $(\|\mathbf{Q}^{(o(l))}-\hat{\mathbf{Q}}\|_{F})_{l\in\mathbb{N}}$ converges to 0, so does the sequence

$(\|\mathbf{G}_{p,q}(c,s)\mathbf{Q}^{(o(l))}-\mathbf{G}_{p,q}(c,s)\hat{\mathbf{Q}}\|_{F})_{l\in\mathbb{N}}$, because the Frobenius norm is invariant under orthogonal transformations. The mapping $f$ is continuuous w.r.t. the Frobenius norm and thus there is an index $\eta\in\mathbb{N}$ such that

[TABLE]

holds for all $l\geq\eta$. However, at each iteration the algorithm starts the sweep by performing a Givens rotation. If the affected indexes for this first Givens rotation are chosen randomly, the algorithm almost certainly will eventually choose $p,q$ and thus yield the point $\mathbf{G}_{p,q}(\hat{c},\hat{s})\mathbf{Q}^{(o(\hat{l}))}$ with $\hat{l}\geq\eta$, where $\hat{c}$ and $\hat{s}$ are chosen such that $f$ is maximized, at the beginning of a sweep. For such a point, the inequality

[TABLE]

holds. This is not possible, because $f(\mathbf{Q}^{(o(\hat{l})+1)})\geq f(\mathbf{G}_{p,q}(\hat{c},\hat{s})\mathbf{Q}^{(o(\hat{l}))})$ and $\hat{f}$ is an upper bound for any element of $(f(\mathbf{Q}^{(l)}))_{l\in\mathbb{N}}$.

∎

III-E Fréchet Means of sets of KLDSs

Due to the metric property, the alignment distance allows for the employment Fréchet means on subsets of $\mathcal{K}_{n,p,\kappa}$ [20]. Consider a finite subset $\{\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{K}\}$ of a space $\mathcal{X}$ equipped with a metric $d(\cdot,\cdot)$. The Fréchet mean [21] of $\{\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{K}\}$ is the minimizer

[TABLE]

This may look like an abstract concept at first, but it is a natural way to choose a representative point out of a set of candidates. We want this representative point not to be far away of any of the reference points, which is why the squared sum of metrics to all these reference points is minimized. By doing so, we make sure through the triangle inequality, that the metric from a test sample to the representative point differs as little as possible from the metric from a test sample to any reference point. Fig. 2 visualizes this intuition: if we consider a reference point $\boldsymbol{x}_{\text{ref}}$, a testing point $\boldsymbol{x}_{\text{test}}$ and a representative point $\boldsymbol{x}_{\text{rep}}$, the triangle inequality yields

[TABLE]

A small value for $d(\boldsymbol{x}_{\text{rep}},\boldsymbol{x}_{\text{ref}})$ thus makes sure that $\boldsymbol{x}_{\text{rep}}$ is a sensible approximation of $\boldsymbol{x}_{\text{ref}}$ for proximity based classification tasks.

Consider the finite set $\{\Theta_{i}=(\mathbf{A}_{i},\mathbf{Y}_{i},\boldsymbol{\alpha}_{i},\boldsymbol{\beta}_{i})\}_{i\in\{1,\dots,K\}}\subset\mathcal{K}_{n,p,\kappa}$. The Fréchet mean of $\{\Theta_{i}\}_{i\in\{1,\dots,K\}}$ is the minimizer $\bar{\Theta}=(\bar{\mathbf{A}}_{i},\bar{\mathbf{Y}}_{i},\bar{\boldsymbol{\alpha}}_{i},\bar{\boldsymbol{\beta}}_{i})\in\mathcal{K}_{n,p,\kappa}$ of the average of alignment distances to the $K$ KLDSs:

[TABLE]

The minimization problem (45) can be approached iteratively, until the cost function

[TABLE]

can not be further reduced.

Assume that at a given iteration $l$, an approximate solution $\bar{\Theta}^{(l)}$ was determined and let $\mathbf{Q}_{i}^{(l)}$ be the minimizer of $d_{F}(\bar{\Theta}^{(l)},\mathbf{Q}\cdot\Theta_{i})^{2}$, i.e. the maximizer of $\rho(\bar{\Theta}^{(l)},\mathbf{Q}\cdot\Theta_{i})^{2}$, for each $i$. Substituting this into (45) yields

[TABLE]

This minimization problem can be solved separately for $\mathbf{A}$, and the rest of the parameters. With this in mind, let us split up the problem. For $\mathbf{A}$, this yields

[TABLE]

For the remaining parameters, note that both the feature space bias as well as the feature space observer can be represented as a linear combination of the feature space mappings of all the involved samples. We could thus choose $\mathbf{\bar{\mathbf{Y}}^{(l)}}$ to be fixed for all iterations $l$ as

[TABLE]

to make sure that the feature space bias and the feature space observer are determined exactly. However, in practice a sample matrix of size $N^{*}=\sum_{i=1}^{K}N_{i}$ can become quickly not handleable. Since the feature space bias and the feature space observer typically operate on a much smaller dimension, it is reasonable to assume that fewer samples are needed to model them, i.e. a sample matrix with a considerably lower number of columns $\bar{N}$ can be employed. The choice of $\bar{\mathbf{Y}}\in\mathbb{R}^{p\times\bar{N}}$ can be made heuristically, following insights of Nyström interpolation of kernel matrices [31]. It would go beyond the scope of this work to review different strategies for the choice of $\bar{\mathbf{Y}}\in\mathbb{R}^{p\times\bar{N}}$. In this work, we employ the k-means approach proposed in [32] and assume that $\kappa(\bar{\mathbf{Y}},\bar{\mathbf{Y}})$ has full rank.

Assuming that $\bar{\mathbf{Y}}\in\mathbb{R}^{p\times\bar{N}}$ is determined, let us fix

[TABLE]

for each iteration $l$ and

[TABLE]

for all iterations. This leads to the formulations

[TABLE]

The Equation (53) is an unconstrained quadratic minimization problem with the analytical solution

[TABLE]

for all iterations. Additionally, the multiplication from left or right with orthogonal matrices does not affect the spectral norm $\|\cdot\|_{2}$ of a matrix. Together with the triangle inequality, this unfolds that the solution of (48), yielded by the euclidean average,

[TABLE]

is unaffected by the constraint $\|\mathbf{A}\|_{2}<1$. For the solution of (52), the EVD of the symmetric kernel matrix is written as

[TABLE]

From the constraint $\boldsymbol{\alpha}^{\top}\kappa(\bar{\mathbf{Y}},\bar{\mathbf{Y}})\boldsymbol{\alpha}=\mathbf{I}_{n}$, we can conclude that the solution must be of the form $\boldsymbol{\alpha}=\mathbf{V}\mathbf{\Lambda}^{-\frac{1}{2}}\mathbf{Z}^{\prime}$, where $\mathbf{Z}^{\prime\top}\mathbf{Z}^{\prime}=\mathbf{I}_{n}$. Substituting this into (52) reduces the problem to finding

[TABLE]

which can be solved as follows. Let the singular value decomposition (SVD) of the known factor of the trace product in (57) be given by

[TABLE]

Substituting it into (57) yields $\mathrm{tr}(\mathbf{Z}^{\top}\mathbf{U}^{\prime}\mathbf{\Sigma}^{\prime}\mathbf{V}^{\prime\top})$ which is maximized by $\mathbf{Z}^{\prime}=\mathbf{U}^{\prime}{\mathbf{V}^{\prime}}_{:,1:n}^{\top}$. Finally, we arrive at

[TABLE]

Algorithm 3 summarizes the described procedure.

Fréchet means are in general not unique. Besides, we can not expect Algorithm 3 to find a global minimum due to the non-convexity of the whole problem. It should thus not be thought of as a method for finding the representative point of a finite subset of $\mathcal{K}_{n,p,\kappa}$, but rather for finding a point that is not far away of any of its elements. Note that the algorithm always terminates, since the cost function is non-negative and the algorithm iterations do not increase it.

IV Experiments

IV-A Overview

This section analyzes the performance of the presented alignment distance on dynamic texture and dynamic scene recognition tasks performed in Matlab on an Intel Core i5-2400 machine. To this end, we convert each frame to a histogram representation. LBP [8] and BoW [10] are chosen, due to their successful applications to still-image texture and scene recognition respectively. The performance of the KLDS parameters in combination with the presented alignment distance was compared against KLDS parameters in combination with the Maximum Singular Value distance proposed in [17] and the Martin distance [15]. In order to put the numbers into context, state-of-the art results from recent publications are included. Throughout all experiments, the alignment distance is computed via the Jacobi-type method which is initialized twice for each sign of the determinant of $\mathbf{Q}$ once. The initializations are determined by creating random sets of orthogonal matrices and choosing the element with the lowest cost function value within. The code for reproducing the NCC experiments will be made available on IEEE Xplore upon publication.

IV-B Dynamic Textures

IV-B1 Database

The DynTex database [33] is a collection of high-resolution RGB texture videos. Three different subsets of DynTex have been compiled and labeled for recognition benchmarking.

• •

DynTex Alpha: The Alpha subset is composed of 60 dynamic textures divided into 3 classes, each containing 20 videos of Sea, Grass, and Trees, respectively.

• •

DynTex Beta: The Beta dataset is composed of 162 dynamic textures divided into 10 classes. The classes with the number of samples indicated in brackets are Sea (20), Vegetation (20), Trees (20), Flags (20), Calm Water (20), Fountains (20), Smoke (16), Escalator (7), Traffic (9) and Rotation (10).

• •

DynTex Gamma: The Gamma dataset is composed of 264 dynamic textures divided into 10 classes. The classes with the number of samples indicated in brackets are Flowers (29), Sea (38), Naked trees (25), Foliage (35), Escalator (7), Calm water (30), Flags (31), Grass (23), Traffic (9) and Fountains (37).

Fig. 3 depicts frames from the DynTex Beta Collection.

IV-B2 Implementation Details

The videos were converted to grayscale. An LBP histogram was computed for each frame via a third-party toolbox [34]. The SoB KLDS parameters of order $n=5$ were computed by means of Algorithm 1 from each stream of LBP histograms. Besides 1-NN performance, we are interested in the ability of Algorithm 3 to represent sets of KLDSs via a representative point. To this end, we performed an NCC classification task additionally. Each class center is computed via Algorithm 3. In order to compare the performance to the Martin and the Maximum SV distance, the medoids were determined as class centers. For the NCC classification, $\lambda_{\mu}$ was set to [math] for the sake of simplicity. Technically, this makes the alignment metric lose its positive definiteness on $\mathcal{K}_{n,p,\kappa}$. Practically, this is not a problem, since we simply work on an appropriate set projection of $\mathcal{K}_{n,p,\kappa}$. The parameter $\lambda_{A}$ was set to $0.25$ for all experiments, which produced the best results on the Alpha Dataset for NCC evaluation. For the 1-NN evaluation, two experiments have been performed to investigate the impact of the parameters. At first $\lambda_{\mu}$ was fixed to 0 and $\lambda_{A}$ varied in the range $0.15,0.35,\dots,1.95$. Then, $\lambda_{A}=0.25$ was kept and $\lambda_{\mu}$ was varied in the range $0,10,20,\dots,100$.

IV-B3 Results

Table I shows the 1-NN classification results of SoB in comparison to LBP-TOP [1, 35], aggregated salient features in three orthogonal planes (ASF-TOP) [36] and Transferred Convolutional Net Features (st-TCoF) [35]. The performance of SoB in combination with the aligned distance is plotted against $\lambda_{A}$ in 4 and against $\lambda_{\mu}$ in Fig. 5.

SoB performs considerably better in combination with the alignment distance than in combination with the other two distance measures with a margin of at least 1.5 percentage points. Overall, it competes well with methods based on ”shallow” representations, such as LBP-TOP or ASF-TOP outperforming them both on DynTex Alpha as well as on DynTex Beta. Still, the deep features learned by st-TCoF yield significantly better results. Table II shows the results of the NCC classification, which is considerably more challenging than the 1-NN classification and is more meaningful with regards to intra-class variability. For the Max SV and the Martin distance, the medoid was computed as an equivalent of the mean. The alignment Fréchet mean of SoB sets provided significantly better result than the medoids with a margin of at least 3.3 percentage points. Beyond that, it yielded the best results published so far to our best knowledge, performing better than Dynamic Fractal Spectrum (DFS) [37], Spatiotemporal Curvelet Transform (2D+T Curvelet) [22], Orthogonal Tensor Dictionary Learning (OTDL), [38] on DynTex Alpha and Beta. Regarding the Gamma dataset, it should be noted that the authors of [38] wrongly describe it as a 10-class database of 275 samples, indicating that the experimental results for OTDL and DFS could have been produced on a different version from the one used in this work, while the evaluation of 2D+T Curvelet [22] was performed on an 11 class adaptation of this dataset, which is why these results are excluded.

Table III shows the confusion matrix for NCC classification on DynTex Beta.

IV-B4 Runtime

On average, the computation of the alignment distance for two KLDSs with state space dimension $n=5$ took 0.0316s. The overall runtimes for the NN (NCC) experiments for DynTex Alpha, Beta and Gamma with SoB + Align were 112.8s (539.47s), 828.98s (1,194.25s) and 2,205.2s (3,364.8s) respectively. The time used for feature computation was not considered.

IV-C Dynamic Scenes

IV-C1 Database

The YUPENN data set [39] is comprised of fourteen dynamic scene classes, Beach, Elevator, Forest fire, Fountain, Highway, Lightning Storm, Ocean, Railway, Rushing river, Clouds, Snowing, City street, Waterfalls and Windmill farm, each class containing 30 color videos of different durations and resolutions. Fig. 6 depicts the different classes.

IV-C2 Implementation Details

OpenCV was used to compute the SURF features for the BoW model of size $p=500$. For the 1-NN evaluation it needed to be made sure that the codebooks are not learned from the tested video samples. To this end, the dataset was divided into three equally large subsets and for each testing sample the codebook generated from the two subsets not containing the sample is used. That way each video sample in the dataset was evaluated by converting the dataset into a set of histogram streams with respect to one of three codebooks. The resulting classification rate was determined by averaging the classification rates for the three codebooks. The histogram streams were converted to KLDSs of state space dimension $n=10$ which is chosen higher than for the previous experiment in order to account for the higher dimensionality of the feature vectors. Initially, $\lambda_{A}$ is set to $0.25$ and $\lambda_{\mu}$ to [math], as was done for the NCC experiments on the DynTex database. The experiment was repeated for different values of $\lambda_{A}$ and $\lambda_{\mu}$.

IV-C3 Results

Table IV shows the classification result for SoB based approaches as described in this work. For the configuration that was used in the previous NCC experiments on the DynTex datasets, the alignment distance performs significantly better than the other two distance measure with 86.4 % correct classifications. Of all tested parameters, $\lambda_{\mu}=0.8$ and $\lambda_{A}=0.5$ yielded the best results and could further improve the classification rate up to 88.8 %. Table V shows the classification results of recently proposed and well-received approaches: Spatiotemporal Oriented Energy (SOE) [39], Tree-Structured Vector Quantization (TSVQ) [40], Bag of System Trees (BoST) [2] and Transferred Convolutional Net Features (st-TCoF) [35]. Like in the DynTex experiments, the st-TCoF descriptors outperform all other approaches in 1-NN classification with a considerable margin. These results are in line with the success of convolutional neural networks (CNNs) in the area of computer vision in the recent years. Hence, SoBs based on bags learned by CNNs could lead to a significant improvement in performance of the presented approach.

IV-C4 Runtime

On average, the computation of the alignment distance for two KLDSs with state space dimension $n=10$ took 0.1781s. The overall runtime for the experiment with non-tuned SoB + Align was 31,423s. The time used for feature computation was not considered.

V Conclusion

Generative and statistical models are widely used in recently presented video and image descriptors. This paper discusses the modeling of videos as streams of histograms generated by a KLDS. As a framework for recognition and classification, this work presents a distance measure on KLDS parameters that allows for computing the dissimilarity of pairs and procrustean means of sets of visual processes described by temporally evolving histograms. The resulting framework competes well with state-of-the-art approaches on widely used dynamic scene and dynamic texture benchmarks. In particular, employing procrustean alignment means for NCC classification outperforms state-of the art approaches on the task of classifying dynamic textures.