Topological signature for periodic motion recognition

TL;DR

This paper introduces a topological signature based on persistent homology for recognizing periodic motions, demonstrating robustness and effectiveness across various activities and partial body data.

Contribution

The paper presents a novel topological signature method for periodic motion recognition that is robust to perturbations and independent of the number of motion cycles.

Findings

Effective in gait recognition using only lower body silhouette

Applicable to various periodic motions like running and jumping

Proven robustness to data perturbations

Abstract

In this paper, we present an algorithm that computes the topological signature for a given periodic motion sequence. Such signature consists of a vector obtained by persistent homology which captures the topological and geometric changes of the object that models the motion. Two topological signatures are compared simply by the angle between the corresponding vectors. With respect to gait recognition, we have tested our method using only the lowest fourth part of the body's silhouette. In this way, the impact of variations in the upper part of the body, which are very frequent in real scenarios, decreases considerably. We have also tested our method using other periodic motions such as running or jumping. Finally, we formally prove that our method is robust to small perturbations in the input data and does not depend on the number of periods contained in the periodic motion sequence.

Figures

Tables

Table 1. Table 1: Cosine distance between the topological signatures obtained from the persistence diagrams showed in Figure 8 .

	$dgm\left(F_{S2}^{c1}\right)$	$dgm\left(F_{S2}^{c2}\right)$
$dgm\left(F_{S1}^{c1}\right)$	0.985	0.987
$dgm\left(F_{S1}^{c2}\right)$	0.981	0.990

Table 2. Table 2: Bottleneck distance between persistence diagrams showed in Figure 8 .

	$dgm\left(F_{S2}^{c1}\right)$	$dgm\left(F_{S2}^{c2}\right)$
$dgm\left(F_{S1}^{c1}\right)$	855727	1319872
$dgm\left(F_{S1}^{c2}\right)$	2273559	5446584

Table 3. Table 3: Bottleneck distance between persistence diagrams according to Figure 9 .

Table 4. Table 4: Cosine distance between the persistence diagrams showed in Figure 9 .

Table 5. Table 5: Accuracy (in % percent \% ) using a training dataset consisting of samples under natural conditions (i.e., without carrying a bag or wearing a coat).

Methods	CASIA-Bbg	CASIA-Bcl	CASIA-Bnm	Average
Tieniu.T [1]	52.0	32.73	97.6	60.8
Khalid.B [12]	78.3	44.0	100	74.1
Singh.S [22]	74.58	77.04	93.44	81.7
Imad.R et al. [5]	81.70	68.80	93.60	81.40
Lishani et al. [23]	76.90	83.30	88.70	83.00
Our Method
using cosine	80.5	81.7	92.4	84.9
using angle	84.2	87.6	94.1	88.6

Table 6. Table 6: Accuracy (in % percent \% ) using a training dataset consisting of samples under different conditions (natural-walking, carrying-bag and wearing-coat).

Methods	CASIA-Bbg	CASIA-Bcl	CASIA-Bnm	Average
Khalid.B [12]	55.6	34.7	69.1	53.1
Our Method	92.3	94.3	94.7	93.8

Table 7. Table 7: Total Accuracy (in %) according to the dataset provided by L. Gorelick et al. for each of the 7 experiments considered.

	A	B	C	D	E	F	G
	1-7	2-6	3-5	4-4	5-3	6-2	7-1
Rank 1	74.6	82.1	85.7	86.8	88.9	90.1	90.1
Rank 2	89.3	96.1	97.1	97.8	98.5	98.1	98.7

Table 8. Table 8: Accuracy (in %) reached by action, according to the dataset provided by L. Gorelick et al. for each of the 7 experiments considered.

	A	B	C	D	E	F	G
	1-7	2-6	3-5	4-4	5-3	6-2	7-1
jack	100.0	100.0	100.0	100.0	100.0	100.0	100.0
pjump	87.7	92.8	94.4	96.3	97.0	95.5	95.5
skip	77.4	83.7	86.8	86.8	83.7	87.5	87.5
run	32.7	40.0	48.8	53.3	61.0	65.0	65.0
walk	83.6	93.3	94.4	95.0	96.7	96.0	96.0
wave1	92.3	99.2	100.0	100.0	100.0	100.0	100.0
wave2	48.4	65.7	75.4	76.3	84.0	86.5	86.5

Table 9. Table 9: Confusion matrix among actions (in %) according to dataset provided by L. Gorelick et al. for each of the 7 experiments considered (part II).

	Actions	jack	pjump	skip	run	walk	wave1	wave2
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	1.6	87.7	0.1	1.40	0.0	8.6	0.6
	skip	0.0	0.7	77.4	13.0	9.3	0.0	0.0
A	run	5.4	0.7	48.0	32.7	14.0	0.1	0.0
1-7	walk	2.0	0.0	10.0	4.1	83.6	0.0	0.0
	wave1	1.4	3.4	1.0	0.7	0.0	92.3	1.1
	wave2	48.0	0.4	0.0	0.1	0.1	2.6	48.4
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	0.3	92.8	0.0	2.5	0.0	4.3	0.0
	skip	0.0	0.0	83.7	9.3	7.0	0.0	0.0
B	run	1.0	0.0	46.8	40.0	12.2	0.0	0.0
2-6	walk	0.0	0.0	4.2	2.5	93.3	0.0	0.0
	wave1	0.0	0.7	0.0	0.0	0.0	99.2	0.2
	wave2	31.8	0.0	0.0	0.0	0.0	2.5	65.7
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	0.0	94.4	0.0	1.2	0.0	4.4	0.0
	skip	0.0	0.0	86.8	8.4	4.8	0.0	0.0
C	run	0.0	0.0	41.8	48.8	9.4	0.0	0.0
3-5	walk	0.0	0.0	3.2	2.4	94.4	0.0	0.0
	wave1	0.0	0.0	0.0	0.0	0.0	100.0	0.0
	wave2	24.0	0.0	0.0	0.0	0.0	0.6	75.4
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	0.0	96.3	0.0	0.5	0.0	3.3	0.0
	skip	0.0	0.0	86.8	7.8	5.5	0.0	0.0
D	run	0.0	0.0	38.3	53.3	8.5	0.0	0.0
4-4	walk	0.0	0.0	3.0	2.0	95.0	0.0	0.0
	wave1	0.0	0.0	0.0	0.0	0.0	100.0	0.0
	wave2	23.3	0.0	0.0	0.0	0.0	0.5	76.3

Table 10. Table 10: Confusion matrix among actions (in %) according to dataset provided by L. Gorelick et al. for each of the 7 experiments considered (part II).

	Actions	jack	pjump	skip	run	walk	wave1	wave2
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	0.0	97.0	0.0	0.7	0.0	2.3	0.0
	skip	0.0	0.0	83.7	9.0	7.3	0.0	0.0
E	run	0.0	0.0	34.7	61.0	4.3	0.0	0.0
5-3	walk	0.0	0.0	0.0	3.3	96.7	0.0	0.0
	wave1	0.0	0.0	0.0	0.0	0.0	100.0	0.0
	wave2	16.0	0.0	0.0	0.0	0.0	0.0	84.0
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	0.0	95.5	0.0	0.0	0.0	4.50	0.0
	skip	0.0	0.0	87.5	8.0	4.50	0.0	0.0
F	run	0.0	0.0	31.5	65.0	3.50	0.0	0.0
6-2	walk	0.0	0.0	0.0	4.0	96.0	0.0	0.0
	wave1	0.0	0.0	0.0	0.0	0.0	100	0.0
	wave2	13.5	0.0	0.0	0.0	0.0	0.0	86.5
	jack	100	0.0	0.0	0.0	0.0	0.0	0.0
	pjump	0.0	95.5	0.0	0.0	0.0	4.5	0.0
	skip	0.0	0.0	88.0	8.0	4.5	0.0	0.0
G	run	0.0	0.0	32.0	65.0	3.5	0.0	0.0
7-1	walk	0.0	0.0	0.0	4.0	96.0	0.0	0.0
	wave1	0.0	0.0	0.0	0.0	0.0	100	0.0
	wave2	14.0	0.0	0.0	0.0	0.0	0.0	87.0

Table 11. Table 11: Cluster according to similar lower clothes used.

Cluster (code)	Lower clothes	Upper clothes
2349ABCXY	RP: regular pant	2) HS	HS: Half Shirt
		3) HS,Ht	FS: Full Shirt
		4) HS,Cs	Ht: Hat
		9) FS	PK: Parka
		A) PK	Cs: Casquette Cap
		B) DJ	DJ: Down Jacket
		C) DJ,Mf	Mf: Muffler
		X) FS,Ht
		Y) FS,Cs

Equations

$$\varnothing \subset K_{i_1}\subset \cdots \subset K_{i_{\ell }}=K$$

$$d_b(dgm(F),dgm(F^{\prime }))=\underset{\gamma }{\min }\{\underset{a}{\max }\{||a-\gamma (a)|{|}_{\infty }\}\}$$

$$C:W\Rightarrow W^{\prime }$$

$$d_b(dgm(F),dgm(F^{\prime }))\le ϵ.$$

$${\mathcal{V}}_{\pi }^i={\mathcal{X}}_{\pi }^i$$

$$d_b(dgm(\partial K_{\pi }),dgm(\partial K_{\pi }^{\prime }))\le ϵ.$$

$$\begin{array}[]{c| c c }&\,dgm(F_{008-nm2}^{c1})&\,dgm(F_{008-nm2}^{c2})\\ \hline\cr dgm(F_{001-nm1}^{c1})&0.06&0.128\\ dgm(F_{001-nm1}^{c2})&0.125&0.059\\ \hline\cr\end{array}$$

$$\begin{array}[]{c| c c }&\,dgm(F_{008-nm2}^{c1})&\,dgm(F_{008-nm2}^{c2})\\ \hline\cr dgm(F_{001-nm1}^{c1})&0.739&0.886\\ dgm(F_{001-nm1}^{c2})&0.832&0.905\\ \hline\cr\end{array}$$

Full text

K****eywords Feature extraction  $\cdot$ Periodic motion  $\cdot$ Video sequences  $\cdot$ Persistent Homology

1 Introduction

Person recognition at distance, without the subject cooperation, is an important task in video surveillance. Nevertheless, very few biometric techniques can be used in such scenario. Gait recognition is a technique with special potential under these circumstances since features can be extracted from any viewpoint and at bigger distances than other biometric approaches. Currently, there are good results in the state of the art for persons walking under natural conditions (without carrying a bag or wearing a coat). See, for example, [1, 2, 3]. However, it is common for people to walk carrying things that change their natural gait. The accuracy in gait recognition for persons carrying bag or using coat can be consulted, for example, in [1] for the CASIA-B gait dataset111http://www.cbsr.ia.ac.cn/GaitDatasetB-silh.zip. Moreover, people usually perform movements with the upper body part unrelated to the natural dynamic of the gait.

Up to now, the most successful approaches in gait recognition use silhouettes to get the features. Among the silhouette-based techniques, the best results have been obtained from the methods based in Gait Energy Images (GEI) [4, 1, 2, 5, 6]. Generally, these strategies are affected by a small number of silhouettes (one gait cycle or less). Moreover, the temporal order in which silhouettes appear is not captured in those representations, losing the relative relations of the movements in time. Besides, the features extracted by those methods are highly correlated with errors in the segmentation of the silhouettes [7] and these errors frequently appear in the existing algorithms for background segmentation. This implies that GEI methods are influenced by the shape of the silhouettes instead of the relative positions among the parts of the body while walking.

In our previous conference papers [3, 8, 9, 10], we concentrated our effort in overcoming most of the difficulties explained above. In these works, the gaits were modeled by a persistent-homology-based representation, called topological signature for the gait sequence, and used for human identification [3], gender classification [8], carried object detection [9] and monitoring human activities at distance [10]. Later, in [11], we computed our topological signature using only the lower part of the body (see Figure 1), avoiding many of the effects arising from the variability in the upper body part (related, for example, to hand gestures while talking on cell). This selection is endorsed by the result given in [12], which shows that this part of the body provides most of the necessary information for classification. The topological signature defined in our previous papers has been also used in [13] for recognizing 3D face expression and [14] for differentiating forehand and backhand strokes performed by a tennis player.

Other papers using persistent homology for action recognition are the followings. In [15], topological features of the attractor of the dynamical system are used for modelling human actions and a nearest neighbor classifier is trained with the persistence-based features. In [16], point clouds describing the oscillatory patterns of body joints from the principal components of their time series using Taken’s delay embedding are computed. In [17], a novel framework, based on persistent cohomology can automatically detect, parameterize and interpolate periodic motion patterns obtained from a motion sequence. In none of those papers, theoretical results on the correct behaviour of the designed methods are provided.

In this paper, in Section 2, we first recall the main background needed to understand it. We generalize the procedure given in our previous papers to obtain the topological signature for a periodic motion in Section 3. The input of the procedure is a sequence of silhouettes obtained from a video. A simplicial complex $\partial K(I)$ which represents the periodic motion is then constructed in Subsection 3.1. Sixteen persistence barcodes are then computed considering the distance to eight fixed planes: $2$ horizontal, $2$ vertical, $2$ oblique and $2$ depth planes, completely capturing, this way, the motion in the sequence (see Subsection 3.2). More concretely, for each plane $\pi$, we compute two persistence barcodes (graphical tools encoding the persistent homology information). One persistence barcode detects the variation of connected components and the other one detects the variation of tunnels, when we go through $\partial K(I)$ in a direction perpendicular to the plane $\pi$. Putting together all this information, we construct a vector called topological signature for each sequence in Subsection 3.3. We compare two topological signatures by the angle between the vectors forming the signatures. As an original contribution of this paper, we theoretically study the stability of the topological signature in Section 4. We formally prove, in terms of probabilities, that small perturbations in the input body silhouettes provoke small perturbations in the resulting topological signature. We also prove that the direction of each of the vectors that make up the topological signature for a sequence remains the same independently on the number of periods the sequence contains. Since we compare two topological signatures by the angle between the corresponding vectors, then the previous assertion implies that the topological signature is independent on the number of periods the sequence contains. Experimental results are showed in Section 5. Conclusions are given in Section 6.

2 Preliminaries

Let us now introduce the definitions and concepts used throughout the paper. The main one is the concept of persistent homology, which is an algebraic tool for measuring topological features of shapes and functions. It is built on top of homology, which is a topological invariant that captures the amount of connected components, tunnels, cavities and higher-dimensional counterparts of a shape. Small size features in persistent homology are often categorized as noise, while large size features describe topological properties of shapes. For a more detailed introduction on the theoretical concepts introduced in this section see, for example, [18, 19, 20].

A 3D binary image is a pair $I=(\mathbb{Z}^{3},B)$, where $B$ (called the foreground) is a finite set of points of $\mathbb{Z}^{3}$ and $B^{c}=\mathbb{Z}^{3}\backslash B$ is the background. The cubical complex $Q(I)$ associated to $I$ is a combinatorial structure constituted by a set of unit cubes with square faces parallel to the coordinate planes and vertices in $\mathbb{Z}^{3}$. More concretely, the set of vertices $V$ of any cube $c\in Q(I)$ satisfies that $V=\{(i,j,k),(i+1,j,k),(i,j+1,k),(i,j,k+1),(i+1,j+1,k),(i+1,j,k+1),(i,j+1,k+1),(i+1,j+1,k+1)\}$ for some $(i,j,k)\in\mathbb{Z}^{3}$, and $V\subseteq B$. The [math]-faces of $c$ are its $8$ corners (vertices), its $1$-faces are its $12$ edges, its $2$-faces are its $6$ squares and, finally, its $3$-face is the cube $c$ itself.

A $p$-simplex $\sigma$ in $\mathbb{R}^{n}$ is the set of $p+1$ affinely independent points in $\mathbb{R}^{n}$. Observe that always $p\leq n$. A set $\mu$ in $\mathbb{R}^{n}$ is a face of $\sigma$ if $\mu\subseteq\sigma$. The simplices considered in this paper are [math]-simplices (representing vertices), $1$-simplices (representing edges) and $2$-simplices (representing triangles), all of them embedded in $\mathbb{R}^{3}$. The formal definition of a simplicial complex $K$ is as follows [21, p. 7]: A simplicial complex $K$ is a collection of simplices such that: (1) every face of a simplex of $K$ is in $K$; and (2) the intersection of any two simplices of $K$ is a face of each of them. A simplicial complex is finite if it has a finite number of simplices.

Let $K$ be a simplicial complex. A $p$-chain on $K$ is a formal sum of $p$-simplices of $K$. The group of $p$-chains is denoted by $C_{p}(K)$. The $p$-boundary operator $\partial_{p}:C_{p}(K)\to C_{p-1}(K)$ is a homomorphism such that for each $p$-simplex $\sigma$ of $K$, $\partial_{p}(\sigma)$ is the sum of its $(p-1)$-faces. For example, if $\sigma$ is a triangle, $\partial_{2}(\sigma)$ is the sum of its edges. The kernel of $\partial_{p}$ is called the group of $p$-cycles in $C_{p}(K)$ and the image of $\partial_{p+1}$ is called the group of $p$-boundaries in $C_{p}(K)$. The $p$-homology $H_{p}(K)$ of $K$ is the quotient group of $p$-cycles relative to $p$-boundaries (see [21, Chapter 5]). The [math]-homology classes of $K$ (i.e. the classes in $H_{0}(K)$) represent the connected components of $K$, the $1$-homology classes its tunnels and the $2$-homology classes its cavities.

A filtration $F$ of a simplicial complex $K$ is an ordering of the simplices of $K$ dictated by a filter function $f:K\rightarrow\mathbb{R}$, satisfying that if a simplex $\sigma$ is a face of another simplex $\sigma^{\prime}$ in $K$ then $f(\sigma)\leq f(\sigma^{\prime})$ and $\sigma$ appears before $\sigma^{\prime}$ in the filtration. The associated filtered simplicial complex is the sequence:

[TABLE]

where $i_{1}<\cdots<i_{\ell}$ and $K_{i_{j}}=f^{-1}(-\infty,i_{j}]$ for $1\leq j\leq\ell$.

Consider a filtration $F=(\sigma_{1},\sigma_{2},\dots,\sigma_{m})$ of a simplicial complex $K$ obtained from a given filter function $f:K\to\mathbb{R}$. If $\sigma_{i}$ completes a $p$-cycle ($p$ being the dimension of $\sigma_{i}$) when $\sigma_{i}$ is added to $F_{i-1}=(\sigma_{1},\dots,\sigma_{i-1})$, then a $p$-homology class $\alpha$ is born at time $f(\sigma_{i})$; otherwise, a $(p-1)$-homology class dies at time $f(\sigma_{i})$. The difference between the birth and death times of a homology class $\gamma$ is called its persistence, which quantifies the significance of a topological attribute. If $\alpha$ never dies, we set its persistence to infinity.

For a $p$-homology class that is born at time $f(\sigma_{i})$ and dies at time $f(\sigma_{j})$, we draw a bar $[f(\sigma_{i}),f(\sigma_{j}))$ with endpoints $f(\sigma_{i})$ and $f(\sigma_{j})$. The set of bars $\{[f(\sigma_{i}),f(\sigma_{j}))\subset{\mathbb{R}}\}$ (resp. points $\{(f(\sigma_{i}),f(\sigma_{j}))\in{\mathbb{R}}^{2}\}$) representing birth and death times of homology classes is called the persistence barcode $B(F)$ (resp. persistence diagram $dgm(F)$) for the filtration $F$. Analogously, fixed $i$, the set of bars (resp. points) representing birth and death time of $i$-homology classes is called the $i$-persistence barcode (resp. the $i$-persistence diagram) for $F$.

For example, in Figure 2, the filtration $F=\{$ $b$, $c$ ,$bc$ ,$e$, $be$, $ec$, $a$, $ab$, $ac$, $abc$, $d$, $bd$, $de$, $bde$, $f$, $ef$, $cf$, $cef\}$ of the simplicial complex which consists of a composition of three triangles, can be read on the $x$-axis of the picture. Bars corresponding to the persistence of [math]-homology classes (i.e. the persistence of connected components) are colored in blue and bars corresponding to the persistence of $1$-homology classes (i.e., the persistence of tunnels) are colored in red. Observe that only two bars survive until the end: one corresponding to the connected component and one corresponding to the tunnel of the simplicial complex.

The bottleneck distance (see [18, page 229]) is classically used to compare two persistence diagrams $dgm(F)=\{a_{1},\dots,a_{k}\}$ and $dgm(F^{\prime})=\{a^{\prime}_{1},\dots,a^{\prime}_{k^{\prime}}\}$ for two filtrations $F$ and $F^{\prime}$ of, respectively, two finite simplicial complexes $K$ and $K^{\prime}$. The bottleneck distance between $dgm(F)$ and $dgm(F^{\prime})$ is:

[TABLE]

where $\gamma:dgm(F)\to dgm(F^{\prime})$ is a bijection that can associate a point off the diagonal with another point on or off the diagonal, where diagonal is the set $D=\{(x,x)\}\subset{\mathbb{R}}^{2}$. For points $a=(x,y)\in F\cup D$ and $\gamma(a)=(x^{\prime},y^{\prime})\in F^{\prime}\cup D$, the expression $||a-\gamma(a)||_{\infty}$ means $\max\{|x-x^{\prime}|,|y-y^{\prime}|\}$. Observe that since $K$ and $K^{\prime}$ are finite then so are $dgm(F)$ and $dgm(F^{\prime})$. Table 2 in page 2 shows the bottleneck distance between the persistence diagrams pictured in Figure 8.

The following definitions are taken from [20]. Let $W$ and $W^{\prime}$ be the vertex set of, respectively, two simplicial complexes $K$ and $K^{\prime}$. A correspondence $C:W\Rightarrow W^{\prime}$ from $W$ to $W^{\prime}$ is a subset of $W\times W^{\prime}$ satisfying that for any $v\in W$ there exists $v^{\prime}\in W^{\prime}$ such that $(v,v^{\prime})\in C$ and, conversely, for any $v^{\prime}\in W^{\prime}$ there exists $v\in W$ such that $(v,v^{\prime})\in C$. Besides, for a subset $\sigma$ of $W$, $C(\sigma)$ denotes the subset of $W^{\prime}$ satisfying that a vertex $v^{\prime}$ is in $C(\sigma)$ if and only if there exists a vertex $v\in\sigma$ such that $(v,v^{\prime})\in C$. Finally, given filter functions $f:K\to\mathbb{R}$ and $f^{\prime}:K^{\prime}\to\mathbb{R}$, and the corresponding filtrations $F$ and $F^{\prime}$, we say that $C:W\Rightarrow W^{\prime}$ is $\epsilon$-simplicial from $F$ to $F^{\prime}$ if for any $t\in\mathbb{R}$ and simplex $\sigma\in K$ such that $f(\sigma)\leq t$, every simplex $\mu\in K^{\prime}$ with vertices in $C(\sigma)$ satisfies that $f^{\prime}(\mu)\leq t+\epsilon$. The transpose of $C$, denoted by $C^{T}$, is the image of $C$ through the symmetry map $(x,y)\to(y,x)$.

The following results will be used later in the paper.

Proposition 1

[20, Proposition 4.2]** Let $S$ and $T$ be filtered complexes with vertex sets $X$ and $Y$ respectively. If $C:X\Rightarrow Y$ is a correspondence such that $C$ and $C^{T}$ are both $\epsilon$-simplicial, then together they induce a canonical $\epsilon$-interleaving between $H(S)$ and $H(T)$, the interleaving homomorphisms being $H(C)$ and $H(C^{T})$.

The homology groups of each subcomplex in a filtered complex together with the morphism between homology groups induced by the inclusion maps is an example of a $q$-tame module if all the subcomplexes are finite which is the case we deal with in this paper.

Theorem 1

[20, Theorem 2.3]** If $U$ is a $q$-tame module then it has a well-defined persistence diagram $dgm(U)$. If $U$, $V$ are $q$-tame persistence modules that are $\epsilon$-interleaved then there exists an $\epsilon$-matching between the multisets $dgm(U)$ and $dgm(V)$. Thus, the bottleneck distance between the diagrams satisfies the bound $d_{b}(dgm(U),dgm(V))\leq\epsilon$.

3 Topological signature for a periodic motion

In this section, we explain how to compute the topological signature for a sequence of silhouettes. First, in Subsection 3.1, a simplicial complex is built from the given sequence. Then, in Subsection 3.2, eight filtrations of the simplicial complex are computed in order to capture the movements that characterize the motion recorded in the sequence. In Subsection 3.3 we finally explain how to obtain the signature from the persistence barcodes for the given filtrations.

3.1 From sequences of silhouettes to simplicial complexes

In this subsection we introduce the construction of the simplicial complex $\partial K(I)$ which models the input sequence, which is a sequence of silhouettes obtained from a periodic motion sequence. See, for example, Figure 3.a. In our previous papers [3, 8, 9, 10, 11], we got the sequences from the background segmentation provided in CASIA-B dataset222http://www.cbsr.ia.ac.cn/GaitDatasetB-silh.zip for gait silhouettes.

We build a 3D binary image $I=(\mathbb{Z}^{3},B)$ by stacking $k$ consecutive silhouettes. The cubical complex $Q(I)$ associated to the 3D binary image $I$ is then computed. The height of each silhouette is set to $1$ and the width changes accordingly to preserve the original proportion between height and width. Besides, $z$-coordinates that represented the amount of silhouettes in the stack is also set to $1$. Then, by construction, $x$-, $y$- and $z$-coordinates of the vertices in $Q(I)$ have their values in the interval $[0,1]$. Finally, the squares that are faces of exactly one cube in $Q(I)$ are divided into two triangles. These triangles together with their faces (vertices and edges) form the simplicial complex $\partial K(I)$. See Figure 3.b and Figure 4.b. The different colors in the figures are used just to recognize each silhouette in the complex.

3.2 Filtration of the simplicial complex $\partial K(I)$

The next step in our process is to compute filtrations of the previously computed simplicial complex $\partial K(I)$, in order to capture the movements recorded in the sequence.

Eight different filtrations of $\partial K(I)$ are computed using, respectively, eight planes: two vertical planes ($x=0$ and $x=1$), two horizontal planes ($y=0$ and $y=1$) and four oblique planes ($x-y=1$, $y-x=1$, $x+y=0$ and $x+y=2$). See Figure 5.

More concretely, for each plane $\pi$, we define the filter function $f_{\pi}:\partial K(I)\rightarrow\mathbb{R}$ which assigns to each vertex of $\partial K(I)$ its distance to the plane $\pi$, and to any other simplex of $\partial K(I)$, the greatest distance of its vertices to $\pi$. A filtration $\partial K_{\pi}$ of $\partial K(I)$ is then computed dictated by the filter function $f_{\pi}$.

Observe that for a vertex $v\in\partial K(I)$, the value of $f_{\pi}(v)$ is: (a) less or equal than $1$ if $\pi$ is an horizontal o vertical plane; and (b) less or equal than $\sqrt{2}$ if $\pi$ is an oblique plane. Finally, observe that not any filtration involves $z$-coordinate. The reason for this is that our aim is to obtain a topological signature robust to the number of silhouettes (which was one of the weaknesses of previous approaches as mentioned in the introduction) used to compute the simplicial complex $\partial K(I)$.

3.3 Topological signature

The final step in our process is to compute the persistence barcode for each filtration $\partial K_{\pi}$ associated to each plane $\pi$ (see Figure 5).

We only consider bars in the persistence barcode with length strictly greater than [math]. This way, we do not take into account any topological event $e$ that is born and dies at the same distance to the reference plane. This is not a problem in the sense that we could lose information, since that event $e$ will be captured using a different reference plane.

As an example, the persistence barcodes for two filtrations dictated by the distance to the plane $x=0$ and obtained, respectively, from the video sequences 001-nm-01-090 and 002-nm-01-090 of the CAISA-B dataset, are shown in Figure 6. The set of red bars forms the [math]-persistence barcode and the set of blue bars forms the $1$-persistence barcode. Notice the green circle showing topological features that are born and die at the same time.

Now, for each plane $\pi$, the [math]- and $1$-persistence barcode for the filtration $\partial K_{\pi}$ are explored according to a uniform sampling. More concretely, given a positive integer $n$ (being $n=24$ in our experimental results, obtained by cross validation), we compute the value $h=\frac{k}{n}$, which represents the width of the “window” we use to analyze the persistence barcode, being $k$ the greatest distance of a vertex in $\partial K(I)$ to the plane $\pi$. Since the distance to the plane $\pi$ has been normalized then $k\leq\sqrt{2}$, so $h\leq\frac{\sqrt{2}}{24}$.

Procedure 1

A vector ${\cal V}_{\pi}^{0}$ (resp. ${\cal V}_{\pi}^{1}$) of $2n$ entries is constructed as follows. For $s=0,\dots,n-1$:

• (a)

entry $2s$ contains the number of [math]- (resp. $1$-) homology classes that are born before $s\cdot h$ and persist or die after $s\cdot h$;

• (b)

entry $2s+1$ contains the number of [math]- (resp. $1$-) homology classes that are born in $s\cdot h$ or later and before $(s+1)\cdot h$.

For example, suppose that a homology class is born and dies within the interval $[s\cdot h,(s+1)\cdot h)$. Then, in this case, we add $1$ in entry $2s+1$. On the other hand, suppose that a homology class is born in the interval $[s\cdot·h,(s+1)\cdot h)$ and dies in $[t\cdot h,(t+1)\cdot h)$ for some $s,t$ such that $s<t\leq n$. Then, in this case, we add $1$ in entry $2s+1$, and in entries $2j$ for $s<j\leq t$.

Dividing the entries in two categories (a) and (b), small details in the object are highlighted, which is crucial for distinguishing two different motions. For example, let us suppose a scenario in which $m$ [math]-homology classes are born in $[s\cdot h,(s+1)\cdot h)$ and persist or die at the end of $[(s+1)\cdot h,(s+2)\cdot h)$ and not any other [math]-homology class is born, persists or dies in these intervals. Then, we put [math] in entries $2s$ and $2s+3$ of ${\cal V}_{\pi}^{0}$, and $m$ in entries $2s+1$ and $2s+2$ of ${\cal V}_{\pi}^{0}$. On the other hand, let us suppose that $m$ [math]-homology classes are born and die in $[s\cdot h,(s+1)\cdot h)$ and in $[(s+1)\cdot h,(s+2)\cdot h)$ and not any other [math]-homology class is born, persists or dies in these intervals. Then, we put [math] in entries $2s$ and $2s+2$ of ${\cal V}_{\pi}^{0}$ and $m$ in entries $2s+1$ and $2s+3$ of ${\cal V}_{\pi}^{0}$. Therefore, considering (a) and (b) separately, we can distinguish both scenarios.

Fixed a plane $\pi$, we then obtain two $2n$-dimensional vectors for $\partial K_{\pi}$, one for the [math]-persistence barcode and the other for the $1$-persistence barcode for the filtration $\partial K_{\pi}$. Since we have eight planes, $\{\pi_{1},\dots\pi_{8}\}$, and two vectors per plane, $\{{\cal V}^{0}_{\pi_{i}},{\cal V}^{1}_{\pi_{i}}\}:$ $i=1,\dots,8$, we have a total of sixteen $2n$-dimensional vectors which form the topological signature for a periodic motion sequence.

Finally, to compare the topological signatures for two periodic motion sequences, we add up the angle between each pair of vectors in the signatures. Since a signature consists of sixteen vectors, the best comparison for two sequences is obtained when the total sum is [math] and the worst is $90^{o}\cdot 16=1440^{o}$. Observe that in our previous papers [3, 8, 9, 10], we used the cosine distance333The cosine distance between two vectors $v$ and $w$ is $\frac{v\cdot w}{||v||\cdot||w||}$. to compare two given topological signatures. In that case, the best comparison for two sequences is obtained when the total sum is $16$ and the worst comparison when it is [math].

Remark 1

We have noticed that using the angle instead of the cosine to compare two topological signatures, the efficiency (accuracy) of gait recognition increases by $5\%$. This comparison is made in Table 5. The reason for this phenomena is that angle is more discriminative than cosine when the angle between vectors is close to zero444Recall that $\lim_{x\to 0}\frac{x}{1-\cos(x)}=\infty$..

4 Stability of the topological signature for a periodic motion sequence

Once we have defined the topological signature for a periodic motion sequence, our aim is to prove its stability under small perturbations on the sequence (Theorem 2) and/or under variations on the number of periods in the sequence (Theorem 3).

The following technical statements will be used to prove Theorem 2.

Proposition 2

*Let $F$ (resp. $F^{\prime}$) be a filtration of a simplicial complex $K$ (resp. $K^{\prime})$ dictated by a filter function $f:K\to\mathbb{R}$ (resp. $f^{\prime}:K^{\prime}\to\mathbb{R}$). Let $W$ (resp. $W^{\prime}$) be the vertex set of $K$ (resp. $K^{\prime}$).

If $C:W\Rightarrow W^{\prime}$ is a correspondence from $W$ to $W^{\prime}$ satisfying that $f^{\prime}(v^{\prime})\leq f(v)+\epsilon$ for every $(v,v^{\prime})\in C$, then*

[TABLE]

Proof. Let $t\in\mathbb{R}$ and $\sigma\in K$ such that $f(\sigma)\leq t$. Then, every simplex $\mu\in K^{\prime}$ with vertices in $C(\sigma)$ satisfies that $f^{\prime}(\mu)\leq t+\epsilon$. Then, $C:W\Rightarrow W^{\prime}$ is $\epsilon$-simplicial from $F$ to $F^{\prime}$ by definition. $\square$

Proposition 3

*Let $F$ (resp. $F^{\prime}$) be a filtration of a simplicial complex $K$ (resp. $K^{\prime})$ dictated by a filter function $f:K\to\mathbb{R}$ (resp. $f^{\prime}:K^{\prime}\to\mathbb{R}$). Let $W$ (resp. $W^{\prime}$) be the vertex set of $K$ (resp. $K^{\prime}$).

If the correspondence $C:W\Rightarrow W^{\prime}$ is $\epsilon$-simplicial from $F$ to $F^{\prime}$, then*

[TABLE]

Proof. The statement is a direct consequence of Proposition 1 in page 1 and Theorem 1. $\square$

In the following theorem, we prove, in terms of probabilities, that the topological signature is stable under small perturbations on the input data (i.e., the input sequence).

Theorem 2

*Let $I$ (resp. $I^{\prime}$) be a 3D binary image. Let $W$ (resp. $W^{\prime}$) be the vertex set of $\partial K(I)$ (resp. $\partial K(I^{\prime})$). Let $\partial K_{\pi}$ (resp. $\partial K^{\prime}_{\pi}$) be the filtration of $\partial K(I)$ (resp. $\partial K(I^{\prime})$) dictated by the distance function $f_{\pi}:\partial K(I)\to\mathbb{R}$ (resp. $f^{\prime}_{\pi}:\partial K(I^{\prime})\to\mathbb{R}$) to a given plane $\pi$. Let ${\cal V}^{j}_{\pi}$ (resp. ${\cal X}^{j}_{\pi}$), where $j=0,1$, be the two vectors obtained by applying Proc. 1 to the persistence barcode for $\partial K_{\pi}$ (resp. $\partial K^{\prime}_{\pi}$). Let $m_{i}$ (resp. $m^{\prime}_{i}$) be the number of bars in the $i$-persistence barcodes for the filtration $\partial K_{\pi}$ (resp. $\partial K^{\prime}_{\pi}$).

If $C:W\Rightarrow W^{\prime}$ is a correspondence from $W$ to $W^{\prime}$ satisfying that $f^{\prime}_{\pi}(v^{\prime})\leq f_{\pi}(v)+\epsilon$ for every $(v,v^{\prime})\in C$, then*

[TABLE]

where:

• •

$k$* is the maximum distance of a point in $\partial K_{\pi}$ to the plane $\pi$;*

• •

$n$* is the number of subintervals (“windows") in which the interval $[0,k]$ is divided.*

Proof. By Proposition 2, we have that $C:$ $W\Rightarrow W^{\prime}$ is $\epsilon$-simplicial from $\partial K_{\pi}$ to $\partial K^{\prime}_{\pi}$. By Proposition 3, we have that

[TABLE]

Let $\gamma:dgm(\partial K_{\pi})\cup\{(x,x)\}\to dgm(\partial K^{\prime}_{\pi})\cup\{(x^{\prime},x^{\prime})\}$ be the bijection such that $\max_{a}\{||a-\gamma(a)||_{\infty}\}=d_{b}(dgm(\partial K_{\pi}),dgm(\partial K^{\prime}_{\pi}))$.

Now, let $a=(x,y)\in dgm(\partial K_{\pi})$ and $a^{\prime}=(x^{\prime},y^{\prime})\in dgm(\partial K^{\prime}_{\pi})$, being $x<y$ and $x^{\prime}<y^{\prime}$. On the one hand, if $\gamma(a)=a^{\prime}$, then $|x-x^{\prime}|\leq\epsilon$ and $|y-y^{\prime}|\leq\epsilon$. On the other hand, if $\gamma(a)$ lies in the diagonal, then $|x-y|\leq\epsilon$. Similarly, if $\gamma^{-1}(a^{\prime})$ lies in the diagonal, then $|x^{\prime}-y^{\prime}|\leq\epsilon$.

Now, observe that ${\cal V}^{i}_{\pi}$ can be different from ${\cal X}^{i}_{\pi}$ if there exists a point $(\alpha,\beta)$ in $dgm(\partial K_{\pi})$ or $dgm(\partial K^{\prime}_{\pi})$ satisfying that $\alpha$ or $\beta$ belongs to $(sh-\epsilon,sh+\epsilon)$ for $h=\lfloor\frac{k}{n}\rfloor$ and $s\in\{1,\dots,n-1\}$. Any of both situations ($\alpha$ or $\beta$ belongs to $(sh-\epsilon,sh+\epsilon)$) can occur with probability $\frac{2(n-1)\epsilon}{k}$.

Now, on the one hand, suppose that $\gamma(a)=a^{\prime}$. If $x$ and $y$ do not belong to $(sh-\epsilon,sh+\epsilon)$ then $x^{\prime}$ and $y^{\prime}$ either. On the other hand, suppose $\gamma(a)=(x^{\prime},x^{\prime})$ then $|x-y|\leq\epsilon$. If $x\not\in(sh-\epsilon,sh+\epsilon)$ then $y$ either. Respectively, suppose $\gamma^{-1}(a^{\prime})=(x,x)$. If $x^{\prime}\not\in(sh-\epsilon,sh+\epsilon)$ then $y^{\prime}$ either.

Therefore, ${\cal V}^{i}_{\pi}={\cal X}^{i}_{\pi}$ with probability greater or equal than $\left(1-\frac{2(n-1)\epsilon}{k}\right)^{m_{i}+m^{\prime}_{i}}$. $\square$

As we will see next, the result given in Theorem 2 only makes sense when $\epsilon$ is “small" enough.

Corollary 1

The probability of being ${\cal V}^{i}_{\pi}={\cal X}^{i}_{\pi}$ tends to $1$ when $\epsilon$ tends to [math]. Besides, such probability is non-negative when $\epsilon$ is less or equal than $\frac{k}{2(n-1)}$ (recall that $h=\frac{k}{n}$ is the width of the“window” we use to analyze the persistence barcode).

Proof. First, it is clear that if $\epsilon$ tends to [math] then $\left(1-\frac{2(n-1)\epsilon}{k}\right)^{m_{i}+m^{\prime}_{i}}$ tends to [math]. Second, $1-\frac{2(n-1)\epsilon}{k}\geq 0$ when $\epsilon\leq\frac{k}{2(n-1)}$. $\square$

Take two 3D binary images $I$ and $I^{\prime}$, and a plane $\pi$ such that there exists a correspondence $C$ between the vertices of $\partial K(I)$ and $\partial K(I^{\prime})$ satisfying that $f_{\pi}(v^{\prime})\leq f_{\pi}(v)+\epsilon$ for any pair of vertices $v\in\partial K(I)$ and $v^{\prime}\in\partial K(I^{\prime})$ matched by $C$. Recall that, by construction, $f_{\pi}(v)$ and $f^{\prime}_{\pi}(v^{\prime})$ are less or equal than $1$ (resp. $\sqrt{2}$) if $\pi$ is an horizontal or vertical plane (resp. oblique plane). Then $k\leq 1$ (resp. $k\leq\sqrt{2}$) if $\pi$ is an horizontal or vertical plane (resp. oblique plane). Now fix, for example, $k=0.9$, $n=24$ and $m_{0}=m^{\prime}_{0}=20$. Recall that $n=24$ is the one used in our experimentation obtained by cross validation. Half window size for this example is $0.9/(2\cdot 24)=0.01875$. We have that ${\cal V}^{0}_{\pi}={\cal X}^{0}_{\pi}$ with probability greater or equal than $P$ for $P=0.814672814702977$ if $\epsilon=0.0001$ and $P=0.979758005555892$ if $\epsilon=0.00001$.

Finally, the following result shows that the topological signature does not depend on the number of periods in a given sequence.

Theorem 3

The direction of the topological signature $\{{\cal V}^{0}_{\pi_{i}},{\cal V}^{1}_{\pi_{i}}\}_{i=1,\dots,8}$ is independent on the number of periods the sequence contains.

Proof. Let $I_{1}$ be a stack of consecutive silhouettes taken from a sequence of a periodic motion. Let $I_{2}$ be obtained by stacking twice the silhouettes of $I_{1}$. That is, for any point $v=(x,y,z)$ in $I_{1}$ there exist exactly two points $v=(x,y,z)$ and $v^{\prime}=(x,y,z+k)$ in $I_{2}$ for a fixed $k$. Let $\partial K(I_{1})$ (resp. $\partial K(I_{2})$ ) be the simplicial complex obtained from $I_{1}$ (resp. $I_{2})$). Let $\pi$ be the referred plane, as one of the eight planes considered in this paper, used to compute the respective filtrations $\partial K^{1}_{\pi}$ and $\partial K^{2}_{\pi}$. Since all the planes considered in this paper are perpendicular to plane $z=0$, then if the distance of a vertex $v$ in $\partial K(I_{1})$ to the plane $\pi$ is $d$, then the distance of vertices $v$ and $v^{\prime}$ in $\partial K(I_{2})$ to the plane $\pi$ is also $d$. Therefore, for any bar in the persistence barcode associated to the filtration $\partial K^{1}_{\pi}$, there exist exactly two bars in the persistence barcode associated to the filtration $\partial K^{2}_{\pi}$ having the same birth and death times. Consequently, the vector corresponding to the topological signature for $I_{2}$ is exactly twice the vector corresponding to the topological signature for $I_{1}$. $\square$

For instance, let $S1$ and $S2$ be the silhouettes extracted from two gait sequences of the same person from CASIA-B dataset, both $S1$ and $S2$ having two gait cycles.

Let $P_{S1}^{c1}$ and $P_{S2}^{c1}$ (resp. $P_{S1}^{c2}$ and $P_{S2}^{c2}$) be the silhouette sequences of exactly one gait cycle (resp. two gait cycles) on $S1$ and $S2$. See Figure 7. Then, use the same reference plane $\pi$ to obtain the four filtrations $F_{S1}^{c2}$, $F_{S2}^{c2}$, $F_{S1}^{c1}$ and $F_{S2}^{c1}$ for the simplicial complexes associated to the 3D binary images obtained from the sequences $P_{S1}^{c2}$, $P_{S2}^{c2}$, $P_{S1}^{c1}$ and $P_{S1}^{c1}$, respectively. The [math]-persistence diagrams $dgm(F_{S1}^{c2})$, $dgm(F_{S2}^{c2})$, $dgm(F_{S1}^{c1})$ and $dgm(F_{S2}^{c1})$ are showed in Figure 8. We can observe that the diagrams $dgm(F_{S1}^{c2})$ and $dgm(F_{S2}^{c2})$ have approximately the double of persistent points than the diagrams $dgm(F_{S1}^{c1})$ and $dgm(F_{S2}^{c1})$ (look at the area inside of the red circles in Figure 8). Since the topological signature is computed using “windows” in the persistence barcode (or equivalent, in the persistence diagram), then the modules of the topological signature obtained from the persistence diagrams $dgm(F_{S1}^{c2})$ and $dgm(F_{S2}^{c2})$ is approximately the double of the modules of the topological signature obtained from the persistence diagrams $dgm(F_{S1}^{c1})$ and $dgm(F_{S2}^{c1})$. Nevertheless, the direction remains approximately the same. Observe that we do not have exactness since we do not stack the same silhouettes twice, but we take one or two cycles from a sequence, so small errors and variations may appear.

In Table 1 we show the results of the comparison between the topological signatures obtained from the persistence diagrams $dgm(F_{S1}^{c2})$, $dgm(F_{S2}^{c2})$, $dgm(F_{S1}^{c1})$ and $dgm(F_{S2}^{c1})$ using the cosine distance. Observe that, in all the cases, the cosine distance is almost $1$ (i.e., all the vectors have almost the same direction), which makes sense since all the gait sequences correspond to the same person and the corresponding filtrations are computed using the same reference plane. Finally, in Table 2 we show that if we consider the classical bottleneck distance to compare the different persistence diagrams, we obtain different results depending on the number of gait cycles we consider to compute the topological signature. Therefore, the comparison using bottleneck distance does not provide useful information in this case.

We now repeat the experiment for gait sequences obtained from two different persons and considering only the lowest fourth part of the body silhouettes as in [11]. Let 001-nm1 and 001-nm2 be two gait sequences of person 001 and let 008-nm2 be a gait sequence of person 008 taken from CASIA-B dataset. Let $\partial K(I_{001-nm1}^{ci})$, $\partial K(I_{001-nm2}^{ci})$ and $\partial K(I_{008-nm2}^{ci})$ be the simplicial complexes associated to the silhouettes sequences of exactly $i$ gait cycles on 001-nm1, 001-nm2 and 008-nm2, for $i=1,2$.

A fixed reference plane $\pi$ is used to obtain the [math]-persistence diagrams $dgm($ $F_{001-nm1}^{ci})$, $dgm(F_{001-nm2}^{ci})$ and $dgm($ $F_{008-nm2}^{ci})$ of the simplicial complex associated to $\partial K(I_{001-nm1}^{ci})$, $\partial K(I_{001-nm2}^{ci})$ and $\partial K(I_{008-nm2}^{ci})$, respectively. Observe in Figure 9 that the persistence diagrams obtained from two gait cycles have twice as many points as the persistence diagrams obtained from one gait cycle. This can be noticed in the area inside of the red circles in Figure 9.

In Table 3 (resp. Table 4), we show the results for the comparison between the persistence diagrams pictured in Figure 9, using bottleneck distance (resp. cosine distance). The results show that the value of the bottleneck distance increases with the number of gait cycles, regardless of whether we compare sequences of the same person. However, the values of the cosine distance between the topological signature for sequences of the same person are similar regardless of whether we consider different number of cycles in the sequence.

5 Experimental Results

In this section we show several experiments to support our approach. First, results when applying our method on the CASIA-B dataset555http://www.cbsr.ia.ac.cn/GaitDatasetB-silh.zip for gait recognition are given. Second, we compare different periodic motions such as jump, jack movement, run, skip or walk applying our approach to the dataset of motion actions provided by L. Gorelick et al666http://www.wisdom.weizmann.ac.il/ vision/SpaceTimeActions.html. Finally, our method is evaluated on the OU-SIRT-B dataset777http://www.am.sanken.osaka-u.ac.jp/BiometricDB/GaitTM.html for gait recognition when people use different clothing types. In all the experiments, the dataset is divided into two sets: the training set and the test set. The topological signature for each person is obtained as the average of the topological signatures computed from each of the samples of such person in the training dataset.

The CASIA-B dataset contains samples for each of the 11 different angles from which each of a total of 124 people is recorded. For each angle, there are 6 samples per person walking under natural conditions i.e., without carrying a bag or wearing a coat (CASIA-Bnm), 2 samples per person carrying some sort of bag (CASIA-Bbg) and 2 samples per person wearing a coat (CASIA-Bcl). The CASIA-B dataset provides, for each sample, its background segmentation (called sequence). Summing up, there are 10 sequences (6 from CASIA-Bnm, 2 from CASIA-Bbg and 2 from CASIA-Bcl) per person-angle. We carried out two different experiments using the CASIA-B dataset.

In the first experiment, from the total of 10 sequences per person-angle, we used four sequences (taken from CASIA-Bnm dataset) to train, and the rest to test. Our results for lateral view (90 degrees) are shown in Table 5, where we took the cross validation average ($\binom{6}{4}$ = 15 combinations) of accuracy at rank 1 from the candidate list. The experiment was carried out using only the legs of the body silhouette as in [11]. As we have previously said in Remark 1, we obtain better results when using the angle between vectors, instead of the cosine distance, to compare topological signatures.

In the second experiment, we followed the protocol used in [12, Section 5.3]. This way, we considered a mixture of natural, carrying-bag and wearing-coat sequences, since it models a more realistic situation where persons do not collaborate while the samples are being taken. Specifically for lateral view (90 degrees), from the ten sequences per person, six sequences were used to train (four without carrying a bag or wearing a coat, one carrying a bag and one wearing a coat) and the rest was used for the test. Using this training dataset we generated $123$ topological signatures, one for each person in the dataset. We must point out that person labeled as $005$ in CAISA-B was removed from the experiment due to extremely poor quality. We used the remaining sequence of each person carrying a bag and the one wearing a coat for testing. This gave us $246$ sequences for testing: $123$ persons times $2$ sequences per person. Now:

• (1)

For each person, we compared its previously computed topological signature with the two topological signatures obtained from each of the two test sequences. This way we obtain a set, called True Positive (TP) that contains $246$ comparison values.

• (2)

For each person, we compared its previously computed topological signature with the two signatures obtained from each of the two test sequences of a different person. This way we obtain a set, called True Negative (TN) that contains $123\cdot(2\cdot 122)=30012$ values.

We restrict the TN sets to the $246$ smallest values, in order to balance the TP and TN sets.

Observe Figure 10 in which the $y$-axis represents percentages and the $x$-axis represents thresholds. Each point of the red curve (resp. blue curve) shows the percentage of values of the TP set (resp. TN set) lower than a threshold. Table 6 shows the accuracy of the results using the angle between vectors to compare two topological signatures. In Table 5 and Table 6 we can observe that the best result of our method is obtained for the set of natural sequences (CASIA-Bnm) and the worst for the set of persons carrying bags. This is due to that bags can affect on the lowest fourth part of the body silhouette (see Figure 1). Moreover, the weight of the bag can change the dynamic of the gait. Nevertheless, accuracy results in all cases are greater that $80\%$. Besides, the features obtained from the lowest fourth part of the body silhouette gave an accuracy for the sequences of persons walking under natural conditions of $94.1\%$, which only decreases $3.9\%$ with respect to our previous paper [3] in which we used the whole body silhouette ($98.0\%$) and only considered persons walking under natural condition both for training and testing. This confirms that the highest information in the gait is in the motion of the legs, which supports the results given in [12]. As we can see in Table 5 and Table 6, our method outperforms previous methods for gait recognition with or without carrying a bag or wearing a coat. Besides, as we can see in Table 6, the algorithm explained in [12] decreases considerably the accuracy obtained by training mixing the natural, carrying-bag and wearing-coat sequences. On the contrary, our algorithm improves the accuracy for the whole test set outperforming in more that 35% the results given in [12]. Comparing Table 5 and Table 6, we can as well arrive to the conclusion that training with more heterogeneous data gives to our method a more powerful representation for the classification step.

Another experiment was carried out using the motion action dataset provided by L. Gorelick et al888http://www.wisdom.weizmann.ac.il/ vision/SpaceTimeActions.html. The aim of this experiment is to see how discriminative is our topological signature between different periodic motions.

This dataset has 7 actions, 8 samples by action, taken from 9 different persons (56 sequences in total). Figure 11 shows silhouette samples of these actions. To know the performance of our topological signature, we made several experiments with different number of samples in the training and test dataset. Concretely, let $a-b$ denote the experiment which consists of $a$ samples per person to train and $b$ to test. This way, we made $7$ different experiments: (A) $1-7$, (B) $2-6$, (C) $3-5$, (D) $4-4$, (E) $5-3$, (F) $6-2$, and (G) $7-1$. For instance, for A) $1-7$, we have 7 samples to train, 1 per action, and 49 samples to test, 7 per action.

The samples to train were taken randomly in 100 iterations in order to compute the average accuracy. Table 7 shows the accuracy reached among all actions. Besides, Table 8 shows the independent accuracy reached by each action. Table 9 and Table 10 show the confusion matrix for each experiment. We can see the confusion between run and skip, which present similarities in their dynamics (as we can see in Figure 12, there exists similitude in run and skip posses).

Finally, we provide some computations made on the OU-SIRT-B dataset999http://www.am.sanken.osaka-u.ac.jp/BiometricDB/GaitTM.html, where each person has several samples with different combinations of clothes. This dataset has samples from 68 people, taken using a treadmill. It provides the silhouettes, that is, we do not have the videos to apply our own background subtraction algorithm. Each person may not have samples for all the clothes combinations. Given the number of samples per person (max $27$ and min $9$) it is possible to design several experiments. As an example, we made a group for similar lower clothes in order to organize the training and the test sets (see Table 11).

According to clusters in Table 11 we took three samples to train (the samples with clothes combination 2, 3 and 4) and two to test (the samples with clothes combination 9 and A) for each of the first 26 persons. Moreover, we took the samples with clothes combination 9, A and B and to train, and C and X to test for the rest of the persons, due to people do not have the same combination clothes. Considering only the lower part of the body, we obtain a rate of $79.4$% of accuracy.

More examples and the source code written in Matlab can be obtained visiting our web page101010http://grupo.us.es/cimagroup/.

6 Conclusion

In this paper, we have presented a persistent-homology-based signature successfully applied in the past to gait recognition. We have shown that such topological signature can be applied to any periodic motion (not only gait), such as running or jumping. We have formally proved that such signature is robust to noise and independent to the number of periods used to compute it.

Funding. This work has been partially funded by the Applied Math department and Institute of Mathematics at the University of Seville, Andalusian project FQM-369 and Spanish project MTM2015-67072-P.