Linear optimal transport subspaces for point set classification

TL;DR

This paper introduces a new method for classifying point sets using linear optimal transport to handle spatial deformations effectively.

Contribution

The novel framework uses Linear Optimal Transport to create a convex data space for efficient and robust point set classification.

Findings

The LOT transform simplifies point set classification by creating a convex embedding space.

The method achieves competitive accuracy without hyper-parameter tuning.

It shows robustness in out-of-distribution scenarios with varying deformation magnitudes.

Abstract

Learning from point sets is an essential component in many computer vision and machine learning applications. Native, unordered, and permutation invariant set structure space is challenging to model, particularly for point set classification under spatial deformations. Here we propose a framework for classifying point sets experiencing certain types of spatial deformations, with a particular emphasis on datasets featuring affine deformations. Our approach employs the Linear Optimal Transport (LOT) transform to obtain a linear embedding of set-structured data. Utilizing the mathematical properties of the LOT transform, we demonstrate its capacity to accommodate variations in point sets by constructing a convex data space, effectively simplifying point set classification problems. Our method, which employs a nearest-subspace algorithm in the LOT space, demonstrates label efficiency, non-iterative behavior, and requires no hyper-parameter tuning. It achieves competitive accuracies compared to state-of-the-art methods across various point set classification tasks. Furthermore, our approach exhibits robustness in out-of-distribution scenarios where training and test distributions vary in terms of deformation magnitudes.

Figures

Full text

Introduction

1

Point sets provide valuable insights about object geometry, making them useful for a variety of applications, including object detection, recognition, segmentation, and tracking in fields such as robotics, autonomous vehicles, virtual reality, and computer vision, among others [1–5]. They represent the surface geometry of an object in an N-dimensional space as a set of points, obtained using various scanning technologies such as LiDAR or photogrammetry [6, 7], or by sampling a continuous probability density function over an N-D space [8]. However, modeling the set structure space for classification presents significant challenges due to the sparsity and noise in data, the accumulation of spatial deformations (such as affine deformations) in real-world point set data, and the high dimensionality of point sets, among other factors [9, 10, 1, 11]. Furthermore, defining a metric or distance function for point set classification is challenging due to the permutation invariant nature of point sets resulting from the arbitrary order of points in a set [1, 12]. However, despite these challenges, there has been a growing interest in developing new algorithms and techniques for point set classification.

In recent years, several research efforts have focused on point set classification, resulting in the development of various methods to address challenges in this area. Over the last few decades, point set classification methods have evolved from relying on feature engineering [13–16] to utilizing deep neural networks to learn representations and use them in classification tasks [3, 1, 17]. Neural networks have emerged as a leading classification framework for point sets, providing end-to-end learning capabilities and eliminating the need for hand-crafted feature engineering. They have demonstrated to achieve high accuracy in several classification tasks, and are also suitable for parallel implementation using graphical processing units (GPUs) [3, 1, 17]. However, the effectiveness of neural network-based methods is often limited by their high data requirements [18], high computational costs [1], and vulnerability to out-of-distribution samples, e.g., adversarial attacks [19–21].

While the conventional approach to modeling point sets involves direct processing of their coordinates, an alternative and less commonly used method is to represent a point set as a deformation of another point set [22]. To address this challenge, point set deformation models have been developed utilizing the mathematics of optimal mass transport [22, 23]. These models treat a point set as a smooth, nonlinear, and invertible transformation of a reference point set structure. The estimation of such models can be facilitated through the use of the linear optimal transport (LOT) transform, which has found applications in various fields [23]. The LOT of a point set provides as a linear embedding for that point set which can be used to compare with other point set data [23]. The LOT transform has been combined with various machine learning techniques and has been used in many applications [23, 20].

This paper introduces a new method for classifying point sets by expanding upon the LOT-based modeling frameworks. We start by introducing a transport generative model to define point set classes, where class elements can be conceived as instances of an unknown template point set pattern under the effect of unknown spatial deformations. Using the mathematical properties of the LOT transform, we establish that these point set classes, under our generative model (with certain conditions on spatial deformations), can be constructed as convex subspaces in the LOT space, which are capable of accommodating the variations in point set data. Subsequently, we propose a nearest subspace-based classifier in the LOT space for classifying point sets under the given generative model. Our model is also capable of mathematically encoding invariances by integrating mathematical knowledge of deformations known to be present in the data. In our experiments, we particularly focus on datasets experiencing affine deformations and demonstrate the effectiveness of our method compared to several state-of-the-art methods. Our approach exhibits particular strength in situations characterized by limited training data and in the challenging out-of-distribution setting, where the training and test distributions differ in terms of deformation magnitudes.

Preliminaries

2

Linear optimal transport embeddings

2.1

The fundamental principle of optimal transport theory relies on quantifying the amount of effort (measured as the product of mass and distance) required to rearrange one distribution to another, which gives rise to the Wasserstein metric between distributions. In the present study, we utilize a linearized version of this metric, as outlined in [23], which is constructed formally through a tangent space approximation of the underlying manifold.

Following the construction in [23], we define the linear optimal transport transform for probability measures in $[eqn]$ , which is the set of absolutely continuous measures with bounded finite second moments and densities^1^. For simplicity, let us fix a reference measure $[eqn]$ as the Lebesgue measure on a convex compact set of $[eqn]$ . Thanks to Brenier’s theorem [24], there is a unique minimizer $[eqn]$ to the following optimal transportation problem

$$[eqn]$$

where the push-forward (transport) relation $[eqn]$ is defined via $[eqn]$ for any measurable set $[eqn]$ . The linear optimal transport (LOT) transform is given by the following correspondence

$$[eqn]$$

where each probability measure $[eqn]$ is identified with the optimal transport map $[eqn]$ from a fixed reference $[eqn]$ to $[eqn]$ , which lies in a linear space. This square-root of the minimum is called the Wasserstein-2 distance between $[eqn]$ and $[eqn]$ [25]. The LOT metric between two probability distributions $[eqn]$ is^2^

$$[eqn]$$

For simplicity, we denote $[eqn]$ as the LOT transform of $[eqn]$ , i.e., $[eqn]$ where $[eqn]$ is fixed.

It turned out the linearization ability of LOT is closely related to the scope of the following so-called composition property [26, 27]

$$[eqn]$$

where $[eqn]$ , and $[eqn]$ is the set of all diffeomorphisms from $[eqn]$ to $[eqn]$ . In particular, given a convex $[eqn]$ , the LOT embedding of deformed measures via maps in $[eqn]$ become convex^3^ if all $[eqn]$ satisfies the above composition property (4), which is shown more formally below.

Proposition 1 (Lemma A. 2 in [27]) Let $[eqn]$ be convex. Given $[eqn]$ , *define $[eqn]$ *. If $[eqn]$ , (4) holds, then $[eqn]$ is convex in the LOT transform domain.

When the dimension $[eqn]$ , it is shown in [26] that $[eqn]$ can only be “basic” transformations (more specifically, translations or isotropic scalings or their compositions) for the composition property (4) to hold for arbitrary $[eqn]$ ’s. Luckily, [27] proposes an approximate composition property for perturbations of the aforementioned basic transformations, the set of which we denote as $[eqn]$ .

Property 1 (Approximate composition, p. 388 in [27]^4^) Let $[eqn]$ and $[eqn]$ . Let $[eqn]$ such that $[eqn]$ for some $[eqn]$ . Then there exists some $[eqn]$ such that

$$[eqn]$$

Remark: Using the $[eqn]$ notation for LOT transform of $[eqn]$ , we have

$$[eqn]$$

With the above approximate composition property, one can show the following approximate convexity analog of Proposition 1 using Lemma A.3, A. 4 of [27]:

Proposition 2 Let $[eqn]$ and $[eqn]$ be convex such that for any $[eqn]$ , there exists some $[eqn]$ such that $[eqn]$ . Given $[eqn]$ , we have $[eqn]$ is $[eqn]$ -convex in the LOT transform domain, where $[eqn]$ is given in the above approximate composition property. In particular, for any $[eqn]$ [0, 1] and $[eqn]$ ,

$$[eqn]$$

where $[eqn]$ .

Discrete implementation for point sets

2.2

For the analysis of discrete point set data, a discrete version of the Linear Optimal Transport (LOT) embedding is required. In this particular case, both the reference $[eqn]$ and target $[eqn]$ are chosen as discrete probability measures, represented by point sets in $[eqn]$ . A point set in a $[eqn]$ -dimensional space is a finite set of points in $[eqn]$ . A point set $[eqn]$ with $[eqn]$ points can be thought as the image of an injective map $[eqn]$ ^5^. Given a point set $[eqn]$ with $[eqn]$ points, we define a discrete probability distribution associated with the point set as

$$[eqn]$$

Given a diffeomorphism $[eqn]$ , the push-forward distribution of $[eqn]$ under $[eqn]$ is given as

$$[eqn]$$

Let $[eqn]$ denote the collection of injective maps from $[eqn]$ to $[eqn]$ . Given $[eqn]$ , the optimal transportation (Wasserstein-2) distance between associated distributions $[eqn]$ and $[eqn]$ can be obtained by solving the linear programming problem given below:

$$[eqn]$$

where $[eqn]$ , and $[eqn]$ for all $[eqn]$ . Let us fix some $[eqn]$ and use $[eqn]$ as a reference. It turned out that any minimizer matrix $[eqn]$ to the optimal transport problem in (10) is a permutation matrix[25]. In other words, there is a permutation $[eqn]$ such that

$$[eqn]$$

Hence with $[eqn]$ being fixed, an optimal transport map between $[eqn]$ and $[eqn]$ can be determined by $[eqn]$ and $[eqn]$ . The LOT transform for $[eqn]$ is defined as [23]^6^

$$[eqn]$$

and the LOT distance between two point set measures is

$$[eqn]$$

where $[eqn]$ .

Transport based Classification Problem statement

3

In this section, we present a generative model-based problem statement for the type of classification problems we discuss in this paper, building upon the preliminaries established earlier. Our focus is on point set classification, where every class can be viewed as a collection of instances of a prototype point set pattern (a template) observed under unknown spatial deformations. To formalize this concept, we introduce a generative model that provides a formal approach to characterizing point set data of this type.

Generative model Let $[eqn]$ be a set of smooth one-to-one transformations in an $[eqn]$ -dimensional space. The mass-preserving generative model for the $[eqn]$ -th class is defined to be the set

$$[eqn]$$

where $[eqn]$ corresponds to the point set distribution of the prototype template pattern for the $[eqn]$ -th class and $[eqn]$ represents the point set distribution of the $[eqn]$ -th sample from the $[eqn]$ -th class in $[eqn]$ . With these definitions, we can now construct a formal mathematical description for the generative model-based problem statement for point set classification.

Classification problem: Let the set of point set distributions $[eqn]$ are given as in equation (13). Given training samples $[eqn]$ (class 1), $[eqn]$ (class 2),… as training data, determine the class of an unknown distribution $[eqn]$ .

Note that the generative model in equation (13) describes set-structured point set data, which makes it challenging to compare point sets due to their permutation-invariant nature. The generative model above is also not guaranteed to be convex, presenting challenges for effective classification using machine learning techniques. In the subsequent sections, we present solutions to the above classification problem at first by restructuring the point clouds by providing linear optimal transport (LOT) embeddings for them and then by approximating the resulting convex spaces with subspaces as done in many image [21, 28], signal [29, 30], and gradient distribution [31] classification problems.

Proposed solution

4

The LOT transform, which was previously described in section 2, can significantly simplify the classification problem described earlier by providing a convex linear embedding for the set-structured point set data. Let us first investigate the generative model in equation (13) in the LOT transform space. Applying the approximate composition property (equation (6)) to the generative model in equation (13), we have the LOT-space generative model as follows:

$$[eqn]$$

In this context, $[eqn]$ and $[eqn]$ refer to the LOT embeddings of $[eqn]$ and $[eqn]$ , respectively, with respect to a reference structure $[eqn]$ (see equation (11)). Based on the preliminary results presented in Section 2 (Property 1, Proposition2, and other results), it is possible to establish the convexity of the set $[eqn]$ up to a certain bound, subject to certain constraints. Furthermore, we can show that when $[eqn]$ , the intersection of $[eqn]$ with $[eqn]$ is empty [21].

Training phase

4.1

Based on the aforementioned theoretical discussions, we put forward a straightforward non-iterative training approach for the classification method. This involves computing a projection matrix that maps each sample in the LOT space onto the subspace $[eqn]$ (as outlined in [21]), generated by the $[eqn]$ -convex set $[eqn]$ . Specifically, we estimate the projection matrix by applying the following procedure:

$$[eqn]$$

Given a set of sample training data $[eqn]$ , the first step in our proposed method is to apply the LOT transform on them using a reference distribution $[eqn]$ . This results in the generation of transformed samples, denoted as $[eqn]$ . The reference distribution $[eqn]$ is obtained by selecting a point set at random from the training set, followed by the introduction of random perturbations. Subsequently, we estimate $[eqn]$ as follows:

$$[eqn]$$

The proposed method also provides a structure to mathematically encode invariances with respect to deformations that are known to be present in the data [21, 28]. In this paper, we prescribe methods to encode invariances with respect to a set of affine transformations: translation, isotropic and anisotropic scaling, and shear. Detailed descriptions of the deformation types used for encoding invariances and the corresponding methodologies are explained as follows:

Translation: Let $[eqn]$ be the translation by $[eqn]$ and $[eqn]$ . Using equation (6), we have that $[eqn]$ , where $[eqn]$ . Consequently, $[eqn]$ Therefore, as in [21, 28], we define the spanning set for translation as $[eqn]$ , where $[eqn]$ , $[eqn]$ .Isotropic scaling: Let $[eqn]$ be the normalized isotropic scaling of $[eqn]$ by $[eqn]$ , where $[eqn]$ and $[eqn]$ . Using equation (6), we have that $[eqn]$ . As in [21, 28], an additional spanning set for isotropic scaling is not required as the subspace containing $[eqn]$ naturally contains its scalar multiplication $[eqn]$ . Therefore, the spanning set for isotropic is defined as $[eqn]$ .Anisotropic scaling: Let $[eqn]$ be the normalized anisotropic scaling of $[eqn]$ , where $[eqn]$ , and $[eqn]$ . Using equation (6), we have that $[eqn]$ . Consequently, $[eqn]$ . Therefore, the spanning set for anisotropic scaling is defined as $[eqn]$ , where $[eqn]$ , $[eqn]$ .Shear: Let $[eqn]$ be the normalized shear of $[eqn]$ , where $[eqn]$ and $[eqn]$ . Here, the shear matrix $[eqn]$ is constructed using the shear factors $[eqn]$ , which are located at the non-diagonal positions of $[eqn]$ . Using equation (6), we have that $[eqn]$ . Consequently, $[eqn]$ . Therefore, the spanning set for shear is defined as $[eqn]$ , where $[eqn]$ , $[eqn]$ .

Finally, in light of the preceding discussion, we can approximate the enriched subspace $[eqn]$ as

$$[eqn]$$

where $[eqn]$ .

Testing phase

4.2

To classify a given test sample $[eqn]$ , we first apply the LOT transform to $[eqn]$ to obtain its corresponding LOT space representation $[eqn]$ with respect to the reference $[eqn]$ (which was pre-selected duing the training phase). Assuming that the test samples originate from the generative model presented in equation (13) (or equation (14)), we can determine the class of an unknown test sample $[eqn]$ using the following expression:

$$[eqn]$$

where $[eqn]$ is the distance between the test sample and the trained subspaces in the LOT transform space. We can estimate the distance between $[eqn]$ and the trained subspaces using $[eqn]$ , where the matrix $[eqn]$ contains the basis vectors of the subspace $[eqn]$ arranged in its columns.

Results

5

Experimental setup

5.1

Our objective is to analyze how the proposed method performs compared to state-of-the-art approaches in terms of classification accuracy, required training data, and robustness in out-of-distribution scenarios in limited training data setting. To achieve this, we created training sets of varying sizes from the original training set for each dataset under examination. We then trained the models using these training sets and assessed their performance on the original test set. Each train split was generated by randomly selecting (without replacement) samples from the original training set, and we repeated the experiments for each split size ten times. The same train-test data samples were used for all algorithms in each split.

In order to assess the effectiveness of the proposed approach, we utilized several comparison methods. These included PointNet [1], DGCNN [17], and multilayer perceptron (MLP) [32] in FSpool feature embedding space [16]. We also conducted a comparative analysis with various conventional machine learning techniques across different set feature embedding spaces. These included logistic regression (LR), kernel support vector machine (k-SVM), multilayer perceptron (MLP), and nearest subspace (NS) classifier models [32] in GeM1, GeM2, GeM4 [13], COVpool [14, 15], and FSpool [16] embedding spaces. The performance of the proposed method was evaluated in relation to these baselines. We conducted these evaluations in addition to performing out-of-distribution experiments. In the proposed method, we selected the number of basis vectors for the subspaces $[eqn]$ such that the total variance explained by the chosen basis vectors in the $[eqn]$ -th class captured up to 99% of the total variance explained by that class.

To assess the relative performance of the methods, we evaluated them on several datasets, including Point cloud MNIST [33, 34], ModelNet [35], and ShapeNet [36] datasets. We additionally applied random translations, anisotropic scaling, and shear transformations to both the training and test sets of the datasets. For the ShapeNet dataset, we tested the methods under two experimental setups: the regular setup, where both the training and test sets contained point sets at the same deformation magnitude level, and the out-of-distribution setup, where the training and test sets contained point sets at different deformation magnitude levels.

Accuracy in synthetic case

5.2

We first evaluated the effectiveness of the proposed method by comparing it with other state-of-the-art techniques on two synthetic datasets. The synthetic datasets were generated by selecting one sample per class from the point cloud MNIST and ShapeNet datasets, followed by introducing random translations, anisotropic scaling, and shear transformations to each selected sample to generate training and test sets. Specifically, the training set consisted of two samples per class, while the test set comprised 25 samples per class. The obtained comparative results are displayed in Fig. 1. As observed, the proposed method substantially outperformed the other methods in this synthetic scenario.

Accuracy and efficiency in real datasets

5.3

We conducted the performance evaluation of the proposed method by comparing it with several state-of-the-art techniques, including PointNet, DGCNN, and MLP in FSpool feature embedding space, on the MNIST, ShapeNet, and ModelNet datasets. Fig. 2 presents the average test accuracy values obtained for different numbers of training samples per class. The results demonstrate that our proposed method outperformed the other methods across the range of training sample sizes used to train the models. Notably, the proposed method’s accuracy vs. training size curves exhibited a smoother trend in most cases compared to the other methods.

Out-of-distribution robustness

5.4

To assess the effectiveness of the proposed method under the out-of-distribution setting, we introduced a gap between the magnitudes of deformations in the training and test sets. Specifically, we used $[eqn]$ as the deformation set for the ‘out-distribution’ test set, while $[eqn]$ was the deformation set for the ‘in-distribution’ training set. We trained the models using the ‘in-distribution’ data and tested using the ‘out-distribution’ data. For our out-of-distribution experiment, we used the ShapeNet dataset with small deformations as the ‘in-distribution’ training set and the ShapeNet dataset with larger deformations as the ‘out-distribution’ test set (see Fig. 3). The results show that the proposed method outperformed the other methods by an even more significant margin under the challenging out-of-distribution setup, as shown in Fig. 3. Under this setup, the proposed method obtained accuracy figures closer to that in the standard experimental setup (i.e., ShapeNet in Fig. 2). On the other hand, the accuracy of the other methods declined significantly under the out-of-distribution setup compared to the standard experimental setup (see ShapeNet results in Figs. 2 and 3).

Comparison with set-embedding-based methods

5.5

We further evaluated the proposed method against various set embedding-based approaches in combination with classical machine learning methods. The study involved comparing the proposed method with different classifier techniques, including LR, k-SVM, MLP, and NS [32], that were employed with various set-to-vector embedding methods, such as GeM (1, 2, 4) [13], COVpool [14, 15], and FSpool [16]. Fig. 4 illustrates the percentage test accuracy results obtained from these modified experiments, along with the results of the proposed method for comparison. As shown in Fig. 4, the proposed method outperformed all these models in terms of test accuracy.

Discussion

6

This paper presents a new method for classifying point sets using linear optimal transport (LOT) subspaces. Our method is appropriate for problems where the data at hand can be represented as instances of prototype template point set patterns observed under smooth, nonlinear, and one-to-one transformations. The results achieved in different experimental scenarios indicate that our proposed approach can deliver accuracy results comparable to state-of-the-art methods, provided that the data adheres to the generative model specified in equation (13). Additionally, the nearest LOT subspace technique was shown to be more data-efficient in these cases, meaning that it can attain higher accuracy levels using fewer training samples.

Our proposed method maintains high classification accuracy, even in challenging out-of-distribution experimental conditions, as depicted in Fig.3, whereas the accuracy figures of other methods decline sharply. These results indicate that our method provides a better overall representation of the underlying data distribution, resulting in robust classification performance. The key to achieving better accuracy under out-of-distribution conditions is that our method not only learns the deformations present in the data but also learns the underlying data model, including the types of the deformations, such as translation, scaling, and shear, and their respective magnitudes. These deformation types can be learned from just a few training samples containing those deformations, as well as potentially from the mathematically prescribed invariances proposed in [28].

Our proposed method, which utilizes the nearest subspace classifier in the LOT domain, is more suitable for classification problems in the above category compared with general set embedding methods in combination with classical machine learning classifiers, as demonstrated by its classification performance. Typically, point set data classes in their original domain do not constitute embeddings, and commonly used set-to-vector representation techniques are inadequate in generating effective embeddings for them, as indicated by the results. This presents a significant challenge for any machine learning approach to perform effectively. However, the subspace model is appropriate in the LOT domain since the LOT transform provides a linear embedding and convex data geometry. Moreover, considering the subspace model in the LOT space improves the generative nature of our proposed classification method by implicitly including the data points from the convex combination of the provided training data points.

Conclusions

7

In this paper, we propose an end-to-end classification system designed for a specific category of point set classification problems, where a data class can be considered as a collection of instances of a template pattern observed under a set of spatial deformations. If these deformations are appropriately modeled as a collection of smooth, one-to-one, and nonlinear transformations, then the data classes become easily separable in the transform space, specifically the LOT space, due to the properties outlined in the paper. These properties also enable the approximation of data classes as convex subspaces in the LOT space, resulting in a more suitable data model for the nearest subspace method. As we observed in our experiments, this approach yields high accuracy and robustness against out-of-distribution conditions. Many point set classification problems can be formulated in this way, and therefore, our proposed solution has wide applicability.

Finally, we note that there can be many potential adaptations of the proposed method. For instance, the linear subspace method in the presented LOT space could be adjusted to incorporate alternative assumptions regarding the set that best represents each class. While some problems might benefit from a linear subspace method similar to the one described earlier, where all linear combinations are allowed, other problems may be require constraining the model using linear convex hulls. Additionally, investigating the sliced-Wasserstein distance using discrete CDT transform (as proposed in [31]) in conjunction with subspace models is another promising avenue for future research.

Our proposed approach provides promising results in point set classification and serves as a basis for further exploration in this domain. As the amount of 3D (or N-D) data continues to increase and accurate object recognition and scene understanding become more crucial, we believe that the combination of linear optimal transport embeddings and subspace modeling in the transform space will become increasingly significant in this context. We anticipate that our proposed method will inspire further research in this direction and lead to novel developments in recognizing 3D (or N-D) objects or distributions.