Bilateral Operators for Functional Maps

TL;DR

This paper introduces bilateral operators within the functional maps framework for shape correspondence, leveraging pointwise descriptors and intrinsic geometry to improve local smoothing and correspondence accuracy.

Contribution

It develops a novel bilateral operator that combines pointwise descriptors and heat kernel information to enhance shape matching in the spectral domain.

Findings

Bilateral operators improve shape correspondence accuracy.

The method effectively exploits descriptor and geometry information.

Enhanced local smoothing leads to better matching results.

Abstract

A majority of shape correspondence frameworks are based on devising pointwise and pairwise constraints on the correspondence map. The functional maps framework allows for formulating these constraints in the spectral domain. In this paper, we develop a functional map framework for the shape correspondence problem by constructing pairwise constraints using point-wise descriptors. Our core observation is that, every point-wise descriptor allows for the construction a pairwise kernel operator whose low frequency eigenfunctions depict regions of similar descriptor values at various scales of frequency. By aggregating the pairwise information from the descriptor and the intrinsic geometry of the surface encoded in the heat kernel, we construct a hybrid kernel and call it the bilateral operator. Analogous to the edge preserving bilateral filter in image processing, the action of the bilateral operator on a function defined over the manifold yields a descriptor dependent local smoothing of that function. By forcing the correspondence map to commute with the Bilateral operator, we show that we can maximally exploit the information from a given set of pointwise descriptors in a functional map framework.

Tables

Table 1. Table 1 : Visualizing correspondence for varying number of descriptors for different algorithms. Functional Map: [ 17 ] , Diagonal refers to [ 16 ] and Kernel Functional Map [ 25 ]

	2 Descriptors	4 Descriptors	6 Descriptors
Functional Map
Diagonal
Kernel Functional Maps
Bilateral

Equations

$${\mathbf{\Pi }}^{\ast }=\arg \underset{\mathbf{\Pi }\in {\mathcal{P}}_n}{\min }h(\mathbf{\Pi })+\alpha g(\mathbf{\Pi })$$

$$h(\mathbf{\Pi })=\sum _{i=1}^{i=N}||\mathbf{\Pi }{f}_{\mathcal{X}}^{(i)}-f_{\mathcal{Y}}^{(i)}|{|}^2$$

$$g(\mathbf{\Pi })=\sum _{i=1}^{i=M}||\mathbf{\Pi }{O}_{\mathcal{X}}^{(i)}-O_{\mathcal{Y}}^{(i)}\mathbf{\Pi }|{|}^2$$

$$f(x)=\sum _{i\ge 1}{⟨{\varphi }_i,f⟩}_{\mathcal{X}}{\varphi }_i(x)$$

$$\mathbf{\Pi }=\Psi C{\Phi }^T$$

$$C^{\ast }=\arg \underset{C}{\min }$$

$$+\alpha$$

$${\hat{f}}_{\mathcal{X}}^{(i)}$$

$${\hat{O}}_{\mathcal{X}}^{(i)}$$

$${\hat{O}}_{\mathcal{Y}}^{(i)}$$

$$K_{f_{\mathcal{X}}^{(i)}}(x,x^{\prime })=e^{-\frac{(f_{\mathcal{X}}^{(i)}(x)-f_{\mathcal{X}}^{(i)}(x^{\prime }){)}^2}{2{\sigma }^2}}$$

$$\frac{\partial u(t,x)}{\partial t}={\Delta }_{\mathcal{X}}u(t,x)$$

$$u(t,x)={\int }_{\mathcal{X}}{H}_{\mathcal{X}}\left(t,x,x^{\prime }\right)u_0\left(x^{\prime }\right)d{x}^{\prime }$$

$$H_{\mathcal{X}}\left(t,x,x^{\prime }\right)=\sum _i{e}^{{\lambda }_{i}t}{\varphi }_i(x){\varphi }_i\left(x^{\prime }\right)$$

$$O_{\mathcal{X}}^{(i)}(x,x^{\prime })$$

$$O_{\mathcal{Y}}^{(i)}(y,y^{\prime })$$

$$\hat{I}({\mathbf{p}}_i)\propto \sum _{j\in {\mathcal{N}}_i}{e}^{\frac{-||{\mathbf{p}}_i-{\mathbf{p}}_j|{|}^2}{2{\sigma }^2}}I({\mathbf{p}}_j)$$

$$H_{2D}(t,{\mathbf{p}}_i,{\mathbf{p}}_j)=\frac{1}{4\pi t}{e}^{-||{\mathbf{p}}_i-{\mathbf{p}}_j|{|}^2/4t}$$

$$\hat{I}({\mathbf{p}}_i)\propto \sum _{j\in {\mathcal{N}}_i}\{e^{\frac{-||{\mathbf{p}}_i-{\mathbf{p}}_j|{|}^2}{2{\sigma }^2}}.e^{\frac{-|I({\mathbf{p}}_i)-I({\mathbf{p}}_j){|}^2}{2{\beta }^2}}\}I({\mathbf{p}}_j)$$

$${\hat{O}}_{\mathcal{X}}^{(i)}$$

$${\hat{O}}_{\mathcal{Y}}^{(i)}$$

$${\mathbf{K}}_{f_{\mathcal{X}}^{(i)}}$$

$${\Phi }^T{\mathbf{K}}_{f_{\mathcal{X}}^{(i)}}\Phi$$

$$f_{\mathcal{X}}^{(i)}=e^{\frac{-d(x_i,x{)}^2}{{\sigma }^2}}$$

Full text

1 Introduction

Shape correspondence between a pair of surfaces is the task of estimating a semantically meaningful map from one surface onto the other. In geometry processing, surfaces are typically represented as compact two-dimensional Riemannian manifolds. Given two such surfaces $\mathcal{X}$ and $\mathcal{Y}$, the correspondence map $\varphi:\mathcal{X}\rightarrow\mathcal{Y}$ is typically obtained as the solution to an optimization problem where the notion of a good or bad correspondence is explicitly modeled by the objective function.

Typically, such cost functions comprise of a combination of two components: pointwise descriptor preservation or function preservation constraints, and pairwise descriptor preservation or operator commutativity constraints. A point-wise descriptor associates a scalar value to every point on the surface. Therefore, any candidate for its corresponding point on the second shape must have an equivalent descriptor value. For a surface discretized by $n$ points, a single point-wise descriptor is a vector of size $n\times 1$. Notable examples are the heat kernel signature (HKS) [21], the wave kernel (WKS) [4], shape context signatures [12] etc.

In contrast, a pairwise descriptor associates a scalar value for every pair of points on each shape. Therefore, given a pair of points on the first shape, a correspondence map must result in choosing corresponding pairs of points on the second shape yielding equivalent pairwise descriptor values. For an $n$ point surface, a single pairwise descriptor is a matrix of size $n\times n$. Typical examples include, inter-geodesic distance matrices [7] and heat kernels [18, 24] of the surfaces. Pairwise descriptors can also be interpreted as operators acting on functions defined over the surface. Therefore the constraint of preserving a pairwise descriptor is equivalent to the argument that applying the operator to any function and transferring it by correspondence must be equivalent to transferring the function first and then operating on it, thus commuting with the correspondence map.

The functional maps paradigm [17], provides for an elegant linear algebraic method of estimating correspondence maps between surfaces. Under the assumption of approximate isometry, the main idea is to formulate the pairwise and pointwise constraints in the spectral domain of the shapes and represent the map using the eigenfunctions of the laplace beltrami operators of the surfaces. This formulation enables replacing a combinatorial non-convex optimization problem of finding permutations with a simple linear least squares estimation yielding a linear map between the functional spaces of the pair of manifolds.

In most practical applications, it is usually convenient to find reliable pointwise descriptors like point or part correspondences. Eg: shape segments [15], spectral descriptors [21, 4, 14], histograms of local features [23] etc. that are relatively easier to identify and compute. The authors of [16] demonstrate that the information from a point-wise descriptor can be extended to construct a linear operator from it. They propose an operator (or pairwise descriptor) that acts on a function through a point-wise multiplication of the descriptor. Imposing a commutativity constraint with this operator leads to an improved functional map estimation from the same set of known corresponding point-wise descriptors.

Advancing this idea of constructing a pairwise operator from a pointwise descriptor, we explore a novel operator which we call the bilateral operator. The bilateral operator is a smoothing operator, where the weights of averaging are obtained by a combination of intrinsic and descriptor similarities. Each pointwise descriptor can be used to define a pairwise bilateral operator whose action indicates descriptor dependent smoothing of a function over the manifold. By requiring that the correspondence map commute with such an operator, we show a considerable increase in correspondence quality in comparison to previous functional map approaches. The intuition behind our method is that combination of geodesic and descriptor closeness yields high quality correspondence. This means that pairs of points on one shape that are close geodesically or have similar descriptor values correspond very distinctly to pairs of such similar points on the other shape, given the availability of good quality corresponding pointwise descriptors.

2 Background

Without loss of generality, let us consider a pair of shapes $\mathcal{X}$ and $\mathcal{Y}$ that are sampled consistently by $n$ points each. We consider the discrete counterpart of the correspondence map $\varphi:\mathcal{X}\rightarrow\mathcal{Y}$ and observe that it can be represented as a permutation matrix $\boldsymbol{\Pi}\in\{0,1\}^{n\times n}$ satisfying $\mathbf{\Pi}^{\top}\mathbf{1}=\mathbf{\Pi}\mathbf{1}=\mathbf{1}$. where $\mathbf{1}$ is a column vector of ones. A majority of the shape correspondence algorithms can be expressed as an energy minimization problem having the following form [24]:

[TABLE]

where $\mathcal{P}_{n}$ denotes the space of $n\times n$ permutation matrices.

Given a set of corresponding point-wise descriptors: $\{f_{\mathcal{X}}^{(i)},f_{\mathcal{Y}}^{(i)}\in\mathbb{R}^{n\times 1}\}_{i=1}^{i=N}$, the pointwise descriptor preservation term can be written as:

[TABLE]

and similarly, for a set of corresponding pairwise descriptors $\{O_{\mathcal{X}}^{(i)},O_{\mathcal{Y}}^{(i)}\in\mathbb{R}^{n\times n}\}_{i=1}^{i=M}$, the pairwise descriptor preservation (or operator commutativity constraint) is given by:

[TABLE]

The functional maps paradigm [17] allows for solving this problem efficiently by leveraging the knowledge of a compact basis for representing functions on each surface. Typically, the eigenfunctions of the Laplace Beltrami operator of each surface are used, since they form a convenient basis to represent functions and more importantly, they are proven to be optimal for representing smooth functions over manifolds [1].

To fix notations, let $\mathcal{X}$ be a two-dimensional Riemannian manifold. Consider the space of functions defined on the manifold: $L^{2}(\mathcal{X})=\left\{f:\mathcal{X}\rightarrow\mathbb{R}|\langle f,f\rangle_{\mathcal{X}}<\infty\right\}$. The positive semi-definite Laplace-Beltrami operator (LBO) $\Delta_{\mathcal{X}}$ generalizes the notion of the Laplacian operator from Euclidean spaces to surfaces. $\Delta_{\mathcal{X}}$ admits an orthonormal eigendecomposition $\Delta_{\mathcal{X}}\phi_{i}=\lambda_{i}\phi_{i}$, where the eigenvalues form a discrete spectrum $0=\lambda_{1}\leq\lambda_{2}\leq\ldots$ and the eigenfunctions $\Phi=\{\phi_{1},\phi_{2},\dots$ } form an orthonormal basis for $L^{2}(\mathcal{X})$, thereby allowing us to expand any smooth function $f\in L^{2}(\mathcal{X})$ as a Fourier series

[TABLE]

Let $0=\mu_{1}\leq\mu_{2}\leq\ldots$ and $\Psi=\{\psi_{1},\psi_{2},\dots\}$ be the corresponding eigenvalues and eigenfunctions of the LBO $\Delta_{\mathcal{Y}}$ on $\mathcal{Y}$. Then the functional representation of the correspondence map in the spectral basis is given by

[TABLE]

Where the matrix C is a functional maps matrix, whose elements encode the coefficients of the correspondence map represented in the spectral domain. The main utility of the functional maps framework is that, a much larger $n\times n$ permutation can be approximated by a smaller $k\times k$ matrix C by using a few low frequency eigenfunctions $\Phi,\Psi\in\mathbb{R}^{n\times k}$ of the laplace beltrami operators $\Delta_{\mathcal{X}}$ and $\Delta_{\mathcal{Y}}$ respectively.

By rewriting equation 1 using the spectral approximation of 5, the functional maps matrix can be obtained as the solution to the problem:

[TABLE]

where

[TABLE]

Where $A_{\mathcal{X}}$ and $A_{\mathcal{Y}}$ are the area matrices of the respective shapes. Different shape correspondence algorithms in literature can be viewed as particular cases of the loss functions 1 or 6, depending on whether the correspondence was solved in the spatial domain as a permutation or in the spectral domain as a functional map. Notable examples include [16], which proposes a diagonal operator $O_{\mathcal{X}}^{(i)}=\textrm{diag}(f_{\mathcal{X}}^{(i)})\in\mathbb{R}^{n\times n}$ whos action is equivalent to a pointwise multiplication of the corresponding descriptors $f_{\mathcal{X}}^{(i)}$ and $f_{\mathcal{Y}}^{(i)}$. Similarly [9, 8, 2] show the application of geodesic distances and [24] proposed the use of heat-kernels of the surfaces as pairwise descriptors.

3 Bilateral Operators

Our main observation is that we can formulate a more informative pairwise constraint on the correspondence map from an input of point-wise descriptors. This means that in addition to requiring that the correspondence map preserve the descriptor values point-wise as in equation 2, the correspondence map must also preserve the similarities between every pair of points as measured by the descriptor itself. For a given descriptor $f_{\mathcal{X}}^{(i)}$ and a pair of points $(x,x^{\prime})\in\mathcal{X}$, we can associate a notion of similarity between them given by:

[TABLE]

Equation 10 is an operator constructed from a point-wise descriptor $f_{\mathcal{X}}^{(i)}$. One can also interpret it as a smoothing operator over the surface where the weights of averaging are determined by the kernel in equation 10.

A more intrinsic notion of similarity over the surface is given by the heat kernel. Heat kernels are fundamental solutions to the heat diffusion equation over the manifold.

[TABLE]

with the initial conditions $u(0,x)=u_{0}(x)$. Here $u:[0,\infty)\times\mathcal{X}\rightarrow\mathbb{R}$ represents the amount of heat at point $x$ at time $t$. The solution to is given by:

[TABLE]

The heat kernel can be represented in terms of the spectral decomposition of the LBO and is given by:

[TABLE]

For every pair of pointwise descriptors $\{f_{\mathcal{X}}^{(i)},f_{\mathcal{Y}}^{(i)}\}$, we can compute the corresponding bilateral operators that combine the descriptor similarity measure of 10 with an intrinsic notion of similarity encoded by the heat kernel:

[TABLE]

The bilateral operators of equation 14 are conceptually related to a promonient filtering technique used for the problem of image de-noising. Image de-noising is the problem of recovering an image signal from its noisy estimate. As in the case of any noise removal problem, the common intuition is to employ some form of smoothing to the image in which the value of the image function at each pixel is a weighted average of its neighbors. The classical signal processing view of smoothing employs a gaussian weighted averaging of the the image as in equation 15.

[TABLE]

Here $\mathbf{p}_{i}\in\mathbb{R}^{2}$ encodes the 2D spatial location of pixel $i$, and $\mathcal{N}_{i}$ refers to the pixel indices in the spatial neighborhood of the $i^{th}$ pixel. $\widehat{I}$ is the estimate of the de-noised image. The operation of Equation 15 is equivalent to a heat diffusion of on the standard 2D euclidean plane with the noisy image $I$ as the initial condition. This is validated by the fact that the heat kernel in planar domain is the standard gaussian kernel:

[TABLE]

However, an important property desirable from such smoothing operators is edge preservation. See figure 1. A gaussian smoothing of equation 15 will smooth away essential details of the image like dominant edges and textures, since the weights are independent of image intensity. This motivated certain edge preserving filters that allowed for a image-dependent heat diffusion. The beltrami flow framework [19, 20] and the the bilateral filter [22] introduced an additional component that weights a pair of pixels by the difference in their image values in addition to geometric closeness ensured by the heat kernel. Therefore the action of the bilateral filter was to average pixels that were close in the geometric and functional domain, leading to an improved image de-noising filter.

[TABLE]

In this paper, we assert that a similar argument can be extended to the recovery of correspondence maps. Given the knowledge of good quality point-wise descriptors, we assert that a desirable correspondence map must preserve a combination of both geodesic and descriptor closeness. This can be achieved by commuting with the bilateral operators of equation 14.

There is however a minor conceptual difference between the bilateral operators used in image smoothing and the operators of equation 14. An edge preserving kernel for image smoothing demands that the points be spatially close and also have similar image intensities. Hence the resulting bilateral operator is a pointwise multiplication of the kernels (or a Hadamard multiplication of two positive definite matrices). However, this property is appropriate for a de-noising scenario where both similarities need to be strong for the final weight to be significant.

In contrast, the bilateral operator proposed in this paper suggests an additive combination of geodesic and descriptor similarity. Hence either of the similarities need to be strong for the final kernel value to be significant. The constraint of commuting with such an operator leads to the following observation. Pairs of points on the source shape that are similar in a geodesic sense (hence a higher value for the heat kernel) or by way of having identical descriptor values must correspond distinctly to similar pairs on the target. The optimal correspondence ignores only those pairs that are neither geodesically close nor have similar descriptor values. Intuitively, this policy works favourably when the shapes are not strictly related by an isometry, since there exist some pairs of points that are relatively far apart on the surface but worthy of good matching. The heat kernel may not weight such pairs highly. However, such pairs can be rightly deemed close by an informative pointwise descriptor and this relationship is leveraged by the pairwise bilateral operator. Hence commuting with the bilateral operator enables a more informative utilization of each descriptor leading to a better correspondence quality for a smaller budget of point-wise descriptors.

Figure 2 provides a visualization on the action of the kernel operator of equation 10. The eigenfunctions show a harmonic progression in the descriptor space, that is, the shape is harmonically clustered into regions of similar descriptor values. This is in contrast to the global manifold harmonics that are the eigenfunctions of the heat kernel. Therefore, given that we have knowledge of a pair of corresponding descriptors, we require that, not only must the correspondence map preserve the value of the descriptor pointwise, but also the layered pairwise relationship between every pair of points as encoded by the descriptor centric kernel of Equation 10. This philosophy works best when we combine it with a geometric notion of similarity encoded in the heat kernel.

3.1 Numerical Computation

For each descriptor, an explicit construction of the bilateral operators $\{O_{\mathcal{X}}^{(i)},O_{\mathcal{Y}}^{(i)}\in\mathbb{R}^{n\times n}\}$ demands the population of a $n\times n$ matrix using equation 10. However, for a functional map estimation, we only need the spectral equivalents of the operators $\{\widehat{O}_{\mathcal{X}}^{(i)},\widehat{O}_{\mathcal{Y}}^{(i)}\in\mathbb{R}^{k\times k}\}$ for its use in the optimization 6. These can be computed as follows:

[TABLE]

For spectral projection of the descriptor kernels $\Phi^{T}\mathbf{K}_{f_{\mathcal{X}}^{(i)}}\Phi$ and $\Psi^{T}\mathbf{K}_{f_{\mathcal{Y}}^{(i)}}\Psi$, we use the nystrom extension for approximating large positive definite matrices [26]. In particular, the Nystrom approximation requires only a sparse set of columns of any kernel matrix. Let $\mathbf{R}\in\mathbb{R}^{n\times n_{0}},\;n_{0}\ll n$ be such a column stacked matrix of the columns of $\mathbf{K}_{f_{\mathcal{X}}^{(i)}}$. Let $\mathbf{R_{0}}\in\mathbb{R}^{n_{0}\times n_{0}}$ be the smaller intersecting submatrix of $\mathbf{R}$. Then we have the following expressions:

[TABLE]

and a similar expression may be obtained for the kernel matrices for shape $\mathcal{Y}$. We use the farthest point sampling strategy [11] for constructing the column matrix $\mathbf{R}$. By evaluating the kernel functions 10 for only a sparse subsampled set of points and use the approximations of equation 20 we arrive at the spectral coefficients for the pairwise bilateral operators to be used in a functional map pipeline.

4 Correspondence Algorithm

5 Experiments

The general philosophy of our experiments is to compare between various functional map algorithms by imputing all of them with the same set of descriptors and using the same linear optimizer with identical coefficients for the pairwise and pointwise constraints. We compare with three algorithms: the original functional maps framework [17] without any descriptor based regularizer, the diagonal operator proposed in [16] and kernel functional maps [25]. All of these algorithms have the same objective of maximizing the information from a fixed budget of pointwise descriptors.

First, we conduct a synthetic experiment to assess the variation in correspondence quality as a function of number of perfect input pointwise descriptors over 25 shape pairs of FAUST [5]. The objective of this experiment is to assess which functional map algorithm (or constraint) maximally exploits the information from a fixed budget of high quality descriptors. Most recent and arguably state-of-the-art algorithms on shape correspondence are tailored towards learning optimal descriptors with properties of localization, sensitivity and invariance. Recent results from deep-learning frameworks have demonstrated that such descriptors can be learned from examples [14, 6, 13, 10]. What we intend to answer with the first experiment is the question, Given that we have high-quality pointwise descriptors, which functional map algorithm achieves the best correspondence quality?

We compute descriptors by constructing gaussian kernels over ground-truth corresponding points (see Figure 2 (a)), and progressively increase the number of such descriptors as the input to the different algorithms.

[TABLE]

What we expect to see is a decay in the average correspondence geodesic error as the number of such descriptors increase for each algorithm. We use 60 eigenbasis functions for all shapes algorithms. The parameters for all algorithms we chosen as per the choices made in the respective papers. For the bilateral kernel we chose $t=1e-3$ and $\sigma=3$ and $\gamma=1$ for all experiments. Figure 3 shows the results. We see that for a few descriptors, the bilateral filter shows a much lower average correspondence error for the same number of input descriptors. However, all algorithms eventually converge similarly when the number of descriptors increase. Table 1 visualizes the correspondence for the first few landmarks of the plot.

However, general descriptors are far from perfect. This means that there will be a problem of noise leading to inaccurate correspondence. Therefore, in such scenarios, it is not surprising to expect that increasing the number of descriptors may not necessarily lead to an improved correspondence. Moreover the existence of outliers may even deteriorate correspondence quality.

Figure 6 shows the results of our approach compared to [17],[16],[25] on SCAPE and FAUST datasets. We evaluate each algorithm on randomly 100 chosen pairs, for each shape we used 100 discrete laplace-beltarami eigenfunctions as the functional basis. The results of figures 6 show that the bilateral kernel is very potent at lower number of descriptors, and compares favourably with other algorithms for larger number of descriptors, similar to the results in of figure 3. Figure 5 shows comparison of qualitative results of point to point mapping from the source to a target shape using 10 wave kernel signature/map [4] descriptors. The qualitative results are inline with the quantitative ones, see figure 6.

6 Conclusion

In this paper, we propose a novel operator constructed from a pointwise descriptor. We introduce the bilateral operator that combines a descriptor based similarity with the intrinsic geometry of the surface encoded in the heat kernel. By enforcing commutativity constraints with this operator, we demonstrate that we can achieve a better correspondence quality, given a fixed budget of pointwise descriptors.