KPConv: Flexible and Deformable Convolution for Point Clouds

TL;DR

KPConv introduces a flexible, deformable convolution method for point clouds that improves classification and segmentation performance by learning kernel point locations and adapting to local geometry.

Contribution

It proposes a novel point convolution method with learnable, deformable kernel points that operate directly on point clouds without intermediate representations.

Findings

Outperforms state-of-the-art on classification tasks

Effective for segmentation with complex geometries

Robust to varying point cloud densities

Abstract

We present Kernel Point Convolution (KPConv), a new design of point convolution, i.e. that operates on point clouds without any intermediate representation. The convolution weights of KPConv are located in Euclidean space by kernel points, and applied to the input points close to them. Its capacity to use any number of kernel points gives KPConv more flexibility than fixed grid convolutions. Furthermore, these locations are continuous in space and can be learned by the network. Therefore, KPConv can be extended to deformable convolutions that learn to adapt kernel points to local geometry. Thanks to a regular subsampling strategy, KPConv is also efficient and robust to varying densities. Whether they use deformable KPConv for complex tasks, or rigid KPconv for simpler tasks, our networks outperform state-of-the-art classification and segmentation approaches on several datasets. We also offer ablation studies and visualizations to provide understanding of what has been learned by KPConv and to validate the descriptive power of deformable KPConv.

Tables

Table 1. Table 1: 3D Shape Classification and Segmentation results. For generalizability to real data, we only consider scores obtained without shape normals on ModelNet40 dataset. The metrics are overall accuracy (OA) for Modelnet40, class average IoU (mcIoU) and instance average IoU (mIoU) for ShapeNetPart.

	ModelNet40	ShapeNetPart
Methods	OA	mcIoU	mIoU
SPLATNet [34]	-	$83.7$	$85.4$
SGPN [42]	-	$82.8$	$85.8$
3DmFV-Net [9]	$91.6$	$81.0$	$84.3$
SynSpecCNN [48]	-	$82.0$	$84.7$
RSNet [15]	-	$81.4$	$84.9$
SpecGCN [40]	$91.5$	-	$85.4$
PointNet++ [27]	$90.7$	$81.9$	$85.1$
SO-Net [19]	$90.9$	$81.0$	$84.9$
PCNN by Ext [2]	$92.3$	$81.8$	$85.1$
SpiderCNN [45]	$90.5$	$82.4$	$85.3$
MCConv [13]	$90.9$	-	$85.9$
FlexConv [10]	$90.2$	$84.7$	$85.0$
PointCNN [20]	$92.2$	$84.6$	$86.1$
DGCNN [43]	$92.2$	$85.0$	$84.7$
SubSparseCNN [9]	-	$83.3$	$86.0$
KPConv rigid	$92.9$	$85.0$	$86.2$
KPConv deform	$92.7$	$85.1$	$86.4$

Table 2. Table 2: 3D scene segmentation scores (mIoU). Scannet, Semantic3D and Paris-Lille-3D (PL3D) scores are taken from their respective online benchmarks (reduced-8 challenge for Semantic3D). S3DIS scores are given for Area-5 (see supplementary material for k-fold).

Methods	Scannet	Sem3D	S3DIS	PL3D
Pointnet [26]	-	-	$41.1$	-
Pointnet++ [27]	$33.9$	-	-	-
SnapNet [4]	-	$59.1$	-	-
SPLATNet [34]	$39.3$	-	-	-
SegCloud [37]	-	$61.3$	$48.9$	-
RF_MSSF [38]	-	$62.7$	$49.8$	$56.3$
Eff3DConv [50]	-	-	$51.8$	-
TangentConv [36]	$43.8$	-	$52.6$	-
MSDVN [30]	-	$65.3$	$54.7$	$66.9$
RSNet [15]	-	-	$56.5$	-
FCPN [28]	$44.7$	-	-	-
PointCNN [20]	$45.8$	-	$57.3$	-
PCNN [2]	$49.8$	-	-	-
SPGraph [17]	-	$73.2$	$58.0$	-
ParamConv [41]	-	-	$58.3$	-
SubSparseCNN [9]	$72.5$	-	-	-
KPConv rigid	$68.6$	$74.6$	$65.4$	$72.3$
KPConv deform	$68.4$	$73.1$	$67.1$	$75.9$

Table 3. Table 3: Model statistics on 4 datasets: ModelNet40, ShapeNetPart, Scannet, Semantic3D.

		MN40	SNP	Scannet	Sem3D
Avg pts/elem		$6800$	$2370$	$8950$	$3800$
Avg pts/batch		$109K$	$38K$	$90K$	$38K$
Params	rigid	$14.3M$	$14.2M$	$14.1M$	$14.1M$
	deform	$15.2M$	$15.0M$	$14.9M$	$14.9M$
Training	rigid	$3.5$	$5.5$	$4.3$	$8.8$
(batch/s)	deform	$3.1$	$4.3$	$3.9$	$7.1$
Inference	rigid	$8.7$	$16.7$	$9.3$	$17.5$
(batch/s)	deform	$8.0$	$12.2$	$8.1$	$15.0$

Table 4. Table 4: Stable dispositions of the kernel point positions when the center point is fixed. If a disposition has an axis of symmetry, we describe it by the successive groups of points sharing a plane perpendicular to this axis.

$K$	disposition	groups along
	name	symmetrical axis
$5$	Tetrahedron
$7$	Octahedron	1-4-1
$13$	Icosahedron	1-5-5-1
$15$	-	1-6-6-1
$18$	-	1-5-5-5-1
$19$	-	1-4-4-4-4-1
$21$	-	1-6-6-6-1
$25$	-	4-4-4-4-4-4

Table 5. Table 5: Segmentation mIoUs on ShapeNetPart.

Method	class	inst.	aero	bag	cap	car	chair	ear	guit	knif	lamp	lapt	moto	mug	pist	rock	skate	table
	avg.	avg.
Kd-Net [16]	$77.4$	$82.3$	$80.1$	$74.6$	$74.3$	$70.3$	$88.6$	$73.5$	$90.2$	$87.2$	$81.0$	$94.9$	$57.4$	$86.7$	$78.1$	$51.8$	$69.9$	$80.3$
SO-Net [19]	$81.0$	$84.9$	$82.8$	$77.8$	$88.0$	$77.3$	$90.6$	$73.5$	$90.7$	$83.9$	$82.8$	$94.8$	$69.1$	$94.2$	$80.9$	$53.1$	$72.9$	$83.0$
PCNN by Ext [2]	$81.8$	$85.1$	$82.4$	$80.1$	$85.5$	$79.5$	$90.8$	$73.2$	$91.3$	$86.0$	$85.0$	$95.7$	$73.2$	$94.8$	$83.3$	$51.0$	$75.0$	$81.8$
PointNet++ [27]	$81.9$	$85.1$	$82.4$	$79.0$	$87.7$	$77.3$	$90.8$	$71.8$	$91.0$	$85.9$	$83.7$	$95.3$	$71.6$	$94.1$	$81.3$	$58.7$	$76.4$	$82.6$
SynSpecCNN [48]	$82.0$	$84.7$	$81.6$	$81.7$	$81.9$	$75.2$	$90.2$	$74.9$	$93.0$	$86.1$	$84.7$	$95.6$	$66.7$	$92.7$	$81.6$	$60.6$	$82.9$	$82.1$
DGCNN [43]	$82.3$	$85.1$	$84.2$	$83.7$	$84.4$	$77.1$	$90.9$	$78.5$	$91.5$	$87.3$	$82.9$	$96.0$	$67.8$	$93.3$	$82.6$	$59.7$	$75.5$	$82.0$
SpiderCNN [45]	$82.4$	$85.3$	$83.5$	$81.0$	$87.2$	$77.5$	$90.7$	$76.8$	$91.1$	$87.3$	$83.3$	$95.8$	$70.2$	$93.5$	$82.7$	$59.7$	$75.8$	$82.8$
SubSparseCNN [9]	$83.3$	$86.0$	$84.1$	$83.0$	$84.0$	$80.8$	$91.4$	$78.2$	$91.6$	$89.1$	$85.0$	$95.8$	$73.7$	$95.2$	$84.0$	$58.5$	$76.0$	$82.7$
SPLATNet [34]	$83.7$	$85.4$	$83.2$	$84.3$	$89.1$	$80.3$	$90.7$	$75.5$	$92.1$	$87.1$	$83.9$	$96.3$	$75.6$	$95.8$	$83.8$	$64.0$	$75.5$	$81.8$
PointCNN [20]	$84.6$	$86.1$	$84.1$	$86.5$	$86.0$	$80.8$	$90.6$	$79.7$	$92.3$	$88.4$	$85.3$	$96.1$	$77.2$	$95.3$	$84.2$	$64.2$	$80.0$	$83.0$
FlexConv [10]	$85.0$	$84.7$	$83.6$	$91.2$	$96.7$	$79.5$	$84.7$	$71.7$	$92.0$	$86.5$	$83.2$	$96.6$	$71.7$	$95.7$	$86.1$	$74.8$	$81.4$	$84.5$
KPConv rigid	$85.0$	$86.2$	$83.8$	$86.1$	$88.2$	$81.6$	$91.0$	$80.1$	$92.1$	$87.8$	$82.2$	$96.2$	$77.9$	$95.7$	$86.8$	$65.3$	$81.7$	$83.6$
KPConv deform	$85.1$	$86.4$	$84.6$	$86.3$	$87.2$	$81.1$	$91.1$	$77.8$	$92.6$	$88.4$	$82.7$	$96.2$	$78.1$	$95.8$	$85.4$	$69.0$	$82.0$	$83.6$

Table 6. Table 6: Semantic segmentation IoU scores on S3DIS Area-5 . Additionally, we give the mean class recall, a measure that some previous works call mean class accuracy.

Method	mIoU	mRec	ceil.	floor	wall	beam	col.	wind.	door	chair	table	book.	sofa	board	clut.
Pointnet [26]	$41.1$	$49.0$	$88.8$	$97.3$	$69.8$	$0.1$	$3.9$	$46.3$	$10.8$	$52.6$	$58.9$	$40.3$	$5.9$	$26.4$	$33.2$
SegCloud [37]	$48.9$	$57.4$	$90.1$	$96.1$	$69.9$	$0.0$	$18.4$	$38.4$	$23.1$	$75.9$	$70.4$	$58.4$	$40.9$	$13.0$	$41.6$
Eff 3D Conv [50]	$51.8$	$68.3$	$79.8$	$93.9$	$69.0$	$0.2$	$28.3$	$38.5$	$48.3$	$71.1$	$73.6$	$48.7$	$59.2$	$29.3$	$33.1$
TangentConv [36]	$52.6$	$62.2$	$90.5$	$97.7$	$74.0$	$0.0$	$20.7$	$39.0$	$31.3$	$69.4$	$77.5$	$38.5$	$57.3$	$48.8$	$39.8$
RNN Fusion [46]	$57.3$	$63.9$	$92.3$	$98.2$	$79.4$	$0.0$	$17.6$	$22.8$	$62.1$	$74.4$	$80.6$	$31.7$	$66.7$	$62.1$	$56.7$
SPGraph [17]	$58.0$	$66.5$	$89.4$	$96.9$	$78.1$	$0.0$	$42.8$	$48.9$	$61.6$	$84.7$	$75.4$	$69.8$	$52.6$	$2.1$	$52.2$
ParamConv [41]	$58.3$	$67.1$	$92.3$	$96.2$	$75.9$	$0.3$	$6.0$	$69.5$	$63.5$	$66.9$	$65.6$	$47.3$	$68.9$	$59.1$	$46.2$
KPConv rigid	$65.4$	$70.9$	$92.6$	$97.3$	$81.4$	$0.0$	$16.5$	$54.5$	$69.5$	$90.1$	$80.2$	$74.6$	$66.4$	$63.7$	$58.1$
KPConv deform	$67.1$	$72.8$	$92.8$	$97.3$	$82.4$	$0.0$	$23.9$	$58.0$	$69.0$	$91.0$	$81.5$	$75.3$	$75.4$	$66.7$	$58.9$

Table 7. Table 7: Semantic segmentation IoU scores on S3DIS k-fold . Additionally, we give the mean class recall, a measure that some previous works call mean class accuracy.

Method	mIoU	mRec	ceil.	floor	wall	beam	col.	wind.	door	chair	table	book.	sofa	board	clut.
Pointnet [26]	$47.6$	$66.2$	$88.0$	$88.7$	$69.3$	$42.4$	$23.1$	$47.5$	$51.6$	$42.0$	$54.1$	$38.2$	$9.6$	$29.4$	$35.2$
RSNet [15]	$56.5$	$66.5$	$92.5$	$92.8$	$78.6$	$32.8$	$34.4$	$51.6$	$68.1$	$60.1$	$59.7$	$50.2$	$16.4$	$44.9$	$52.0$
SPGraph [17]	$62.1$	$73.0$	$89.9$	$95.1$	$76.4$	$62.8$	$47.1$	$55.3$	$68.4$	$73.5$	$69.2$	$63.2$	$45.9$	$8.7$	$52.9$
PointCNN [20]	$65.4$	$75.6$	$94.8$	$97.3$	$75.8$	$63.3$	$51.7$	$58.4$	$57.2$	$71.6$	$69.1$	$39.1$	$61.2$	$52.2$	$58.6$
KPConv rigid	$69.6$	$78.1$	$93.7$	$92.0$	$82.5$	$62.5$	$49.5$	$65.7$	$77.3$	$57.8$	$64.0$	$68.8$	$71.7$	$60.1$	$59.6$
KPConv deform	$70.6$	$79.1$	$93.6$	$92.4$	$83.1$	$63.9$	$54.3$	$66.1$	$76.6$	$57.8$	$64.0$	$69.3$	$74.9$	$61.3$	$60.3$

Equations

$$(\mathcal{F}\ast g)(x)=\sum _{x_i\in {\mathcal{N}}_x}g(x_i-x)f_i$$

$$g(y_i)=\sum _{k<K}h\left(y_i,{\tilde{x}}_k\right)W_k$$

$$h\left(y_i,{\tilde{x}}_k\right)=\max \left(0,1-\frac{\Vert y_i-{\tilde{x}}_k\Vert }{\sigma }\right)$$

$$(\mathcal{F}\ast g)(x)=\sum _{x_i\in {\mathcal{N}}_x}{g}_{deform}(x-x_i,\Delta (x))f_i$$

$$g_{deform}(y_i,\Delta (x))=\sum _{k<K}h\left(y_i,{\tilde{x}}_k+{\Delta }_k(x)\right)W_k$$

$${\mathcal{L}}_{reg}=\sum _x{\mathcal{L}}_{fit}(x)+{\mathcal{L}}_{rep}(x)\text{vspace}-2ex$$

$${\mathcal{L}}_{fit}(x)=\sum _{k<K}\underset{y_i}{\min }{\left(\frac{\Vert y_i-({\tilde{x}}_k+{\Delta }_k(x))\Vert }{\sigma }\right)}^2\text{vspace}-2ex$$

$${\mathcal{L}}_{rep}(x)=\sum _{k<K}\sum _{l\ne k}h{\left({\tilde{x}}_k+{\Delta }_k(x),{\tilde{x}}_l+{\Delta }_l(x)\right)}^2\text{vspace}-1ex$$

$$\forall x\in {\mathbb{R}}^3,E_k^{rep}(x)=\frac{1}{\Vert x-{\tilde{x}}_k\Vert }$$

$$\forall x\in {\mathbb{R}}^3,E^{att}(x)={\Vert x\Vert }^2$$

$$E^{tot}=\sum _{k<K}\left(E^{att}({\tilde{x}}_k)+\sum _{l\ne k}{E}_k^{rep}({\tilde{x}}_l)\right)$$

Full text

• •

Sec. A details our network architectures, the training parameters, and compares the model sizes and speeds.

• •

Sec. B presents the kernel point initialization method.

• •

Sec. C describes how our regularization strategy tackles the “lost” kernel point phenomenon.

• •

Sec. D enumerates more segmentation results with class scores.

• •

KPConv Method video222https://www.youtube.com/watch?v=uwuvp9mc˙0o&t=19s illustrates KPConv principle with animated diagrams, and shows some learned kernel deformations.

• •

KPConv Results video333https://www.youtube.com/watch?v=˙cFQxJorSAI shows indoor and outdoor scenes segmented by KP-FCNN.

1 Introduction

The dawn of deep learning has boosted modern computer vision with discrete convolution as its fundamental building block. This operation combines the data of local neighborhoods on a 2D grid. Thanks to this regular structure, it can be computed with high efficiency on modern hardware, but when deprived of this regular structure, the convolution operation has yet to be defined properly, with the same efficiency as on 2D grids.

Many applications relying on such irregular data have grown with the rise of 3D scanning technologies. For example, 3D point cloud segmentation or 3D simultaneous localization and mapping rely on non-grid structured data: point clouds. A point cloud is a set of points in 3D (or higher-dimensional) space. In many applications, the points are coupled with corresponding features like colors. In this work, we will always consider a point cloud as those two elements: the points $\mathcal{P}\in\mathbb{R}^{N\times 3}$ and the features $\mathcal{F}\in\mathbb{R}^{N\times D}$. Such a point cloud is a sparse structure that has the property to be unordered, which makes it very different from a grid. However, it shares a common property with a grid which is essential to the definition of convolutions: it is spatially localized. In a grid, the features are localized by their index in a matrix, while in a point cloud, they are localized by their corresponding point coordinates. Thus, the points are to be considered as structural elements, and the features as the real data.

Various approaches have been proposed to handle such data, and can be grouped into different categories that we will develop in the related work section. Several methods fall into the grid-based category, whose principle is to project the sparse 3D data on a regular structure where a convolution operation can be defined more easily [24, 29, 34]. Other approaches use multilayer perceptrons (MLP) to process point clouds directly, following the idea proposed by [49, 26].

More recently, some attempts have been made to design a convolution that operates directly on points [2, 45, 20, 14, 13]. These methods use the spatial localization property of a point cloud to define point convolutions with spatial kernels. They share the idea that a convolution should define a set of customizable spatial filters applied locally in the point cloud.

This paper introduces a new point convolution operator named Kernel Point Convolution (KPConv). KPConv also consists of a set of local 3D filters, but overcomes previous point convolution limitations as shown in related work. KPConv is inspired by image-based convolution, but in place of kernel pixels, we use a set of kernel points to define the area where each kernel weight is applied, like shown in Figure 1. The kernel weights are thus carried by points, like the input features, and their area of influence is defined by a correlation function. The number of kernel points is not constrained, making our design very flexible. Despite the resemblance of vocabulary, our work differs from [32], which is inspired from point cloud registration techniques, and uses kernel points without any weights to learns local geometric patterns.

Furthermore, we propose a deformable version of our convolution [7], which consists of learning local shifts applied to the kernel points (see Figure 3). Our network generates different shifts at each convolution location, meaning that it can adapt the shape of its kernels for different regions of the input cloud. Our deformable convolution is not designed the same way as its image counterpart. Due to the different nature of the data, it needs a regularization to help the deformed kernels fit the point cloud geometry and avoid empty space. We use Effective Receptive Field (ERF) [22] and ablation studies to compare rigid KPConv with deformable KPConv.

As opposed to [41, 2, 45, 20], we favor radius neighborhoods instead of k-nearest-neighbors (KNN). As shown by [13], KNN is not robust in non-uniform sampling settings. The robustness of our convolution to varying densities is ensured by the combination of radius neighborhoods and regular subsampling of the input cloud [38]. Compared to normalization strategies [13, 14], our approach also alleviates the computational cost of our convolution.

In our experiments section, we show that KPConv can be used to build very deep architectures for classification and segmentation, while keeping fast training and inference times. Overall, rigid and deformable KPConv both perform very well, topping competing algorithms on several datasets. We find that rigid KPConv achieves better performances on simpler tasks, like object classification, or small segmentation datasets. Deformable KPConv thrives on more difficult tasks, like large segmentation datasets offering many object instances and greater diversity. We also show that deformable KPConv is more robust to a lower number of kernel points, which implies a greater descriptive power. Last but not least, a qualitative study of KPConv ERF shows that deformable kernels improve the network ability to adapt to the geometry of the scene objects.

2 Related Work

In this section, we briefly review previous deep learning methods to analyze point clouds, paying particular attention to the methods closer to our definition of point convolutions.

Projection networks. Several methods project points to an intermediate grid structure. Image-based networks are often multi-view, using a set of 2D images rendered from the point cloud at different viewpoints [35, 4, 18]. For scene segmentation, these methods suffer from occluded surfaces and density variations. Instead of choosing a global projection viewpoint, [36] proposed projecting local neighborhoods to local tangent planes and processing them with 2D convolutions. However, this method relies heavily on tangent estimation.

In the case of voxel-based methods, the points are projected on 3D grids in Euclidean space [24, 30, 3]. Using sparse structures like octrees or hash-maps allows larger grids and enhanced performances [29, 9], but these networks still lack flexibility as their kernels are constrained to use $3^{3}=27$ or $5^{3}=125$ voxels. Using a permutohedral lattice instead of an Euclidean grid reduces the kernel to $15$ lattices [34], but this number is still constrained, while KPConv allows any number of kernel points. Moreover, avoiding intermediate structures should make the design of more complex architectures like instance mask detector or generative models more straightforward in future works.

Graph convolution networks. The definition of a convolution operator on a graph has been addressed in different ways. A convolution on a graph can be computed as a multiplication on its spectral representation [8, 48], or it can focus on the surface represented by the graph [23, 5, 33, 25]. Despite the similarity between point convolutions and the most recent graph convolutions [39, 43], the latter learn filters on edge relationships instead of points relative positions. In other words, a graph convolution combines features on local surface patches, while being invariant to the deformations of those patches in Euclidean space. In contrast, KPConv combines features locally according to the 3D geometry, thus capturing the deformations of the surfaces.

Pointwise MLP networks. PointNet [26] is considered a milestone in point cloud deep learning. This network uses a shared MLP on every point individually followed by a global max-pooling. The shared MLP acts as a set of learned spatial encodings and the global signature of the input point cloud is computed as the maximal response among all the points for each of these encodings. The network’s performances are limited because it does not consider local spatial relationships in the data. Following PointNet, some hierarchical architectures have been developed to aggregate local neighborhood information with MLPs [27, 19, 21].

As shown by [41, 20, 13], the kernel of a point convolution can be implemented with a MLP, because of its ability to approximate any continuous function. However, using such a representation makes the convolution operator more complex and the convergence of the network harder. In our case, we define an explicit convolution kernel, like image convolutions, whose weights are directly learned, without the intermediate representation of a MLP. Our design also offers a straightforward deformable version, as offsets can directly be applied to kernel points.

Point convolution networks. Some very recent works also defined explicit convolution kernels for points, but KPConv stands out with unique design choices.

Pointwise CNN [14] locates the kernel weights with voxel bins, and thus lacks flexibility like grid networks. Furthermore, their normalization strategy burdens their network with unnecessary computations, while KPConv subsampling strategy alleviates both varying densities and computational cost.

SpiderCNN [45] defines its kernel as a family of polynomial functions applied with a different weight for each neighbor. The weight applied to a neighbor depends on the neighbor’s distance-wise order, making the filters spatially inconsistent. By contrast, KPConv weights are located in space and its result is invariant to point order.

Flex-convolution [10] uses linear functions to model its kernel, which could limit its representative power. It also uses KNN, which is not robust to varying densities as discussed above.

PCNN [2] design is the closest to KPConv. Its definition also uses points to carry kernel weights, and a correlation function. However, this design is not scalable because it does not use any form of neighborhood, making the convolution computations quadratic on the number of points. In addition, it uses a Gaussian correlation where KPConv uses a simpler linear correlation, which helps gradient backpropagation when learning deformations [7].

We show that KPConv networks outperform all comparable networks in the experiments section. Furthermore, to the best of our knowledge, none of the previous works experimented a spatially deformable point convolution.

3 Kernel Point Convolution

3.1 A Kernel Function Defined by Points

Like previous works, KPConv can be formulated with the general definition of a point convolution (Eq. 1), inspired by image convolutions. For the sake of clarity, we call $x_{i}$ and $f_{i}$ the points from $\mathcal{P}\in\mathbb{R}^{N\times 3}$ and their corresponding features from $\mathcal{F}\in\mathbb{R}^{N\times D}$. The general point convolution of $\mathcal{F}$ by a kernel $g$ at a point $x\in\mathbb{R}^{3}$ is defined as:

[TABLE]

We stand with [13] advising radius neighborhoods to ensure robustness to varying densities, therefore, \mathcal{N}_{x}=\left\{\begin{array}[]{c|c}x_{i}\in\mathcal{P}&\left\|x_{i}-x\right\|\leqslant r\end{array}\right\} with $r\in\mathbb{R}$ being the chosen radius. In addition, [38] showed that hand-crafted 3D point features offer a better representation when computed with radius neighborhoods than with KNN. We believe that having a consistent spherical domain for the function $g$ helps the network to learn meaningful representations.

The crucial part in Eq. 1 is the definition of the kernel function $g$, which is where KPConv singularity lies. $g$ takes the neighbors positions centered on $x$ as input. We call them $y_{i}=x_{i}-x$ in the following. As our neighborhoods are defined by a radius $r$, the domain of definition of $g$ is the ball $\mathcal{B}_{r}^{3}=\left\{y\in\mathbb{R}^{3}\>|\>\left\|y\right\|\leqslant r\right\}$. Like image convolution kernels (see Figure 2 for a detailed comparison between image convolution and KPConv), we want $g$ to apply different weights to different areas inside this domain. There are many ways to define areas in 3D space, and points are the most intuitive as features are also localized by them. Let $\left\{\widetilde{x}_{k}\>|\>k<K\right\}\subset\mathcal{B}_{r}^{3}$ be the kernel points and $\left\{W_{k}\>|\>k<K\right\}\subset\mathbb{R}^{D_{in}\times D_{out}}$ be the associated weight matrices that map features from dimension $D_{in}$ to $D_{out}$. We define the kernel function $g$ for any point $y_{i}\in\mathcal{B}_{r}^{3}$ as :

[TABLE]

where $h$ is the correlation between $\widetilde{x}_{k}$ and $y_{i}$, that should be higher when $\widetilde{x}_{k}$ is closer to $y_{i}$. Inspired by the bilinear interpolation in [7], we use the linear correlation:

[TABLE]

where $\sigma$ is the influence distance of the kernel points, and will be chosen according to the input density (see Section 3.3). Compared to a gaussian correlation, which is used by [2], linear correlation is a simpler representation. We advocate this simpler correlation to ease gradient backpropagation when learning kernel deformations. A parallel can be drawn with rectified linear unit, which is the most popular activation function for deep neural networks, thanks to its efficiency for gradient backpropagation.

3.2 Rigid or Deformable Kernel

Kernel point positions are critical to the convolution operator. Our rigid kernels in particular need to be arranged regularly to be efficient. As we claimed that one of the KPConv strengths is its flexibility, we need to find a regular disposition for any $K$. We chose to place the kernel points by solving an optimization problem where each point applies a repulsive force on the others. The points are constrained to stay in the sphere with an attractive force, and one of them is constrained to be at the center. We detail this process and show some regular dispositions in the supplementary material. Eventually, the surrounding points are rescaled to an average radius of $1.5\sigma$, ensuring a small overlap between each kernel point area of influence and a good space coverage.

With properly initialized kernels, the rigid version of KPConv is extremely efficient, in particular when given a large enough $K$ to cover the spherical domain of $g$. However it is possible to increase its capacity by learning the kernel point positions. The kernel function $g$ is indeed differentiable with respect to $\widetilde{x}_{k}$, which means they are learnable parameters. We could consider learning one global set of $\left\{\widetilde{x}_{k}\right\}$ for each convolution layer, but it would not bring more descriptive power than a fixed regular disposition. Instead the network generates a set of $K$ shifts $\Delta(x)$ for every convolution location $x\in\mathbb{R}^{3}$ like [7] and define deformable KPConv as:

[TABLE]

[TABLE]

We define the offsets $\Delta_{k}(x)$ as the output of a rigid KPConv mapping $D_{in}$ input features to $3K$ values, as shown in Figure 3. During training, the network learns the rigid kernel generating the shifts and the deformable kernel generating the output features simultaneously, but the learning rate of the first one is set to $0.1$ times the global network learning rate.

Unfortunately, this straightforward adaptation of image deformable convolutions does not fit point clouds. In practice, the kernel points end up being pulled away from the input points. These kernel points are lost by the network, because the gradients of their shifts $\Delta_{k}(x)$ are null when no neighbors are in their influence range. More details on these “lost” kernel points are given in the supplementary. To tackle this behaviour, we propose a “fitting” regularization loss which penalizes the distance between a kernel point and its closest neighbor among the input neighbors. In addition, we also add a “repulsive” regularization loss between all pair off kernel points when their influence area overlap, so that they do not collapse together. As a whole our regularization loss for all convolution locations $x\in\mathbb{R}^{3}$ is:

[TABLE]

[TABLE]

[TABLE]

With this loss, the network generates shifts that fit the local geometry of the input point cloud. We show this effect in the supplementary material.

3.3 Kernel Point Network Layers

This section elucidates how we effectively put the KPConv theory into practice. For further details, we have released our code using Tensorflow library.

Subsampling to deal with varying densities. As explained in the introduction, we use a subsampling strategy to control the density of input points at each layer. To ensure a spatial consistency of the point sampling locations, we favor grid subsampling. Thus, the support points of each layer, carrying the features locations, are chosen as barycenters of the original input points contained in all non-empty grid cells.

Pooling layer. To create architectures with multiple layer scales, we need to reduce the number of points progressively. As we already have a grid subsampling, we double the cell size at every pooling layer, along with the other related parameters, incrementally increasing the receptive field of KPConv. The features pooled at each new location can either be obtained by a max-pooling or a KPConv. We use the latter in our architectures and call it “strided KPConv”, by analogy to the image strided convolution.

KPConv layer. Our convolution layer takes as input the points $\mathcal{P}\in\mathbb{R}^{N\times 3}$, their corresponding features $\mathcal{F}\in\mathbb{R}^{N\times D_{in}}$, and the matrix of neighborhood indices $\mathfrak{N}\in{[\![1,N]\!]}^{N^{\prime}\times n_{max}}$. $N^{\prime}$ is the number of locations where the neighborhoods are computed, which can be different from $N$ (in the case of “strided” KPConv). The neighborhood matrix is forced to have the size of the biggest neighborhood $n_{max}$. Because most of the neighborhoods comprise less than $n_{max}$ neighbors, the matrix $\mathfrak{N}$ thus contains unused elements. We call them “shadow” neighbors, and they are ignored during the convolution computations.

Network parameters. Each layer $j$ has a cell size ${dl}_{j}$ from which we infer other parameters. The kernel points influence distance is set as equal to $\sigma_{j}=\Sigma\times{dl}_{j}$. For rigid KPConv, the convolution radius is automatically set to $2.5\,\sigma_{j}$ given that the average kernel point radius is $1.5\,\sigma_{j}$. For deformable KPConv, the convolution radius can be chosen as $r_{j}=\rho\times{dl}_{j}$. $\Sigma$ and $\rho$ are proportional coefficients set for the whole network. Unless stated otherwise, we will use the following set of parameters, chosen by cross validation, for all experiments: $K=15$, $\Sigma=1.0$ and $\rho=5.0$. The first subsampling cell size ${dl}_{0}$ will depend on the dataset and, as stated above, ${dl}_{j+1}=2*{dl}_{j}$.

3.4 Kernel Point Network Architectures

Combining analogy with successful image networks and empirical studies, we designed two network architectures for the classification and the segmentation tasks. Diagrams detailing both architectures are available in the supplementary material.

KP-CNN is a 5-layer classification convolutional network. Each layer contains two convolutional blocks, the first one being strided except for the first layer. Our convolutional blocks are designed like bottleneck ResNet blocks [12] with a KPConv replacing the image convolution, batch normalization and leaky ReLu activation. After the last layer, the features are aggregated by a global average pooling and processed by the fully connected and softmax layers like in an image CNN. For the results with deformable KPConv, we only use deformable kernels in the last 5 KPConv blocks (see architecture details in the supplementary material).

KP-FCNN is a fully convolutional network for segmentation. The encoder part is the same as in KP-CNN, and the decoder part uses nearest upsampling to get the final pointwise features. Skip links are used to pass the features between intermediate layers of the encoder and the decoder. Those features are concatenated to the upsampled ones and processed by a unary convolution, which is the equivalent of a $1\times 1$ convolution in image or a shared MLP in PointNet. It is possible to replace the nearest upsampling operation by a KPConv, in the same way as the strided KPConv, but it does not lead to a significant improvement of the performances.

4 Experiments

4.1 3D Shape Classification and Segmentation

Data. First, we evaluate our networks on two common model datasets. We use ModelNet40 [44] for classification and ShapenetPart [47] for part segmentation. ModelNet40 contains 12,311 meshed CAD models from 40 categories. ShapenetPart is a collection of 16,681 point clouds from 16 categories, each with 2-6 part labels. For benchmarking purpose, we use data provided by [27]. In both cases, we follow standard train/test splits and rescale objects to fit them into a unit sphere (and consider units to be meters for the rest of this experiment). We ignore normals because they are only available for artificial data.

Classification task. We set the first subsampling grid size to ${dl}_{0}=2\mathrm{cm}$. We do not add any feature as input; each input point is assigned a constant feature equal to 1, as opposed to empty space which can be considered as 0. This constant feature encodes the geometry of the input points. Like [2], our augmentation procedure consists of scaling, flipping and perturbing the points. In this setup, we are able to process $2.9$ batches of 16 clouds per second on an Nvidia Titan Xp. Because of our subsampling strategy, the input point clouds do not all have the same number of points, which is not a problem as our networks accept variable input point cloud size. On average, a ModelNet40 object point cloud comprises 6,800 points in our framework. The other training parameters are detailed in the supplementary material, along with the architecture details. We also include the number of parameters and the training/inference speeds for both rigid and deformable KPConv.

As shown on Table 1, our networks outperform other state-of-the-art methods using only points (we do not take into account methods using normals as additional input). We also notice that rigid KPConv performances are slightly better. We suspect that it can be explained by the task simplicity. If deformable kernels add more descriptive power, they also increase the overall network complexity, which can disturb the convergence or lead to overfitting on simpler tasks like this shape classification.

Segmentation task. For this task, we use KP-FCNN architecture with the same parameters as in the classification task, adding the positions $(x,y,z)$ as additional features to the constant 1, and using the same augmentation procedure. We train a single network with multiple heads to segment the parts of each object class. The clouds are smaller (2,300 points on average), and we can process $4.1$ batches of 16 shapes per second. Table 1 shows the instance average, and the class average mIoU. We detail each class mIoU in the supplementary material. KP-FCNN outperforms all other algorithms, including those using additional inputs like images or normals. Shape segmentation is a more difficult task than shape classification, and we see that KPConv has better performances with deformable kernels.

4.2 3D Scene Segmentation

Data. Our second experiment shows how our segmentation architecture generalizes to real indoor and outdoor data. To this end, we chose to test our network on 4 datasets of different natures. Scannet [6], for indoor cluttered scenes, S3DIS [1], for indoor large spaces, Semantic3D [11], for outdoor fixed scans, and Paris-Lille-3D [31], for outdoor mobile scans. Scannet contains 1,513 small training scenes and 100 test scenes for online benchmarking, all annotated with 20 semantic classes. S3DIS covers six large-scale indoor areas from three different buildings for a total of 273 million points annotated with 13 classes. Like [37], we advocate the use of Area-5 as test scene to better measure the generalization ability of our method. Semantic3D is an online benchmark comprising several fixed lidar scans of different outdoor scenes. More than 4 billion points are annotated with 8 classes in this dataset, but they mostly cover ground, building or vegetation and there are fewer object instances than in the other datasets. We favor the reduced-8 challenge because it is less biased by the objects close to the scanner. Paris-Lille-3D contains more than 2km of streets in 4 different cities and is also an online benchmark. The 160 million points of this dataset are annotated with 10 semantic classes.

Pipeline for real scene segmentation. The 3D scenes in these datasets are too big to be segmented as a whole. Our KP-FCNN architecture is used to segment small subclouds contained in spheres. At training, the spheres are picked randomly in the scenes. At testing, we pick spheres regularly in the point clouds but ensure each point is tested multiple times by different sphere locations. As in a voting scheme on model datasets, the predicted probabilities for each point are averaged. When datasets are colorized, we use the three color channels as features. We still keep the constant 1 feature to ensure black/dark points are not ignored. To our convolution, a point with all features equal to zero is equivalent to empty space. The input sphere radius is chosen as $50\times{dl}_{0}$ (in accordance to Modelnet40 experiment).

Results. Because outdoor objects are larger than indoor objects, we use ${dl}_{0}=6\mathrm{cm}$ on Semantic3D and Paris-Lille-3D, and ${dl}_{0}=4\mathrm{cm}$ on Scannet and S3DIS. As shown in Table 2, our architecture ranks second on Scannet and outperforms all other segmentation architectures on the other datasets. Compared to other point convolution architectures [2, 20, 41], KPConv performances exceed previous scores by 19 mIoU points on Scannet and 9 mIoU points on S3DIS. SubSparseCNN score on Scannet was not reported in their original paper [9], so it is hard to compare without knowing their experimental setup. We can notice that, in the same experimental setup on ShapeNetPart segmentation, KPConv outperformed SubSparseCNN by nearly 2 mIoU points.

Among these 4 datasets, KPConv deformable kernels improved the results on Paris-Lille-3D and S3DIS while the rigid version was better on Scannet and Semantic3D. If we follow our assumption, we can explain the lower scores on Semantic3D by the lack of diversity in this dataset. Indeed, despite comprising 15 scenes and 4 billion points, it contains a majority of ground, building and vegetation points and a few real objects like car or pedestrians. Although this is not the case of Scannet, which comprises more than 1,500 scenes with various objects and shapes, our validation studies are not reflected by the test scores on this benchmark. We found that the deformable KPConv outperformed its rigid counterpart on several different validation sets (see Section 4.3). As a conclusion, these results show that the descriptive power of deformable KPConv is useful to the network on large and diverse datasets. We believe KPConv could thrive on larger datasets because its kernel combines a strong descriptive power (compared to other simpler representations, like the linear kernels of [10]), and great learnability (the weights of MLP convolutions like [20, 41] are more complex to learn). An illustration of segmented scenes on Semantic3D and S3DIS is shown in Figure 4. More results visualizations are provided in the supplementary material.

4.3 Ablation Study

We conduct an ablation study to support our claim that deformable KPConv has a stronger descriptive power than rigid KPConv. The idea is to impede the capabilities of the network, in order to reveal the real potential of deformable kernels. We use Scannet dataset (same parameters as before) and use the official validation set, because the test set cannot be used for such evaluations. As depicted in Figure 5, the deformable KPConv only loses $1.5\%$ mIoU when restricted to 4 kernel points. In the same configuration, the rigid KPConv loses $3.5\%$ mIoU.

As stated in Section 4.2, we can also see that deformable KPConv performs better than rigid KPConv with 15 kernel points. Although it is not the case on the test set, we tried different validation sets that confirmed the superior performances of deformable KPConv. This is not surprising as we obtained the same results on S3DIS. Deformable KPConv seem to thrive on indoor datasets, which offer more diversity than outdoor datasets. To understand why, we need to go beyond numbers and see what is effectively learned by the two versions of KPConv.

4.4 Learned Features and Effective Receptive Field

To achieve a deeper understanding of KPConv, we offer two insights of the learning mechanisms.

Learned features. Our first idea was to visualize the features learned by our network. In this experiment, we trained KP-CNN on ModelNet40 with rigid KPConv. We added random rotation augmentations around vertical axis to increase the input shape diversity. Then we visualize each learned feature by coloring the points according to their level of activation for this features. In Figure 6, we chose input point clouds maximizing the activation for different features at the first and third layer. For a cleaner display, we projected the activations from the layer subsampled points to the original input points. We observe that, in its first layer, the network is able to learn low-level features like vertical/horizontal planes (a/b), linear structures (c), or corners (d). In the later layers, the network detects more complex shapes like small buttresses (e), balls (f), cones (g), or stairs (h). However, it is difficult to see a difference between rigid and deformable KPConv. This tool is very useful to understand what KPConv can learn in general, but we need another one to compare the two versions.

Effective Receptive Field. To apprehend the differences between the representations learned by rigid and deformable KPConv, we can compute its Effective Receptive Field (ERF) [22] at different locations. The ERF is a measure of the influence that each input point has on the result of a KPConv layer at a particular location. It is computed as the gradient of KPConv responses at this particular location with respect to the input point features. As we can see in Figure 7, the ERF varies depending on the object it is centered on. We see that rigid KPConv ERF has a relatively consistent range on every type of object, whereas deformable KPConv ERF seems to adapt to the object size. Indeed, it covers the whole bed, and concentrates more on the chair that on the surrounding ground. When centered on a flat surface, it also seems to ignore most of it and reach for further details in the scene. This adaptive behavior shows that deformable KPConv improves the network ability to adapt to the geometry of the scene objects, and explains the better performances on indoor datasets.

5 Conclusion

In this work, we propose KPConv, a convolution that operates on point clouds. KPConv takes radius neighborhoods as input and processes them with weights spatially located by a small set of kernel points. We define a deformable version of this convolution operator that learns local shifts effectively deforming the convolution kernels to make them fit the point cloud geometry. Depending on the diversity of the datasets, or the chosen network configuration, deformable and rigid KPConv are both valuable, and our networks brought new state-of-the-art performances for nearly every tested dataset. We release our source code, hoping to help further research on point cloud convolutional architectures. Beyond the proposed classification and segmentation networks, KPConv can be used in any other application addressed by CNNs. We believe that deformable convolutions can thrive in larger datasets or challenging tasks such as object detection, lidar flow computation, or point cloud completion.

Acknowledgement. The authors gratefully acknowledge the support of ONR MURI grant N00014-13-1-0341, NSF grant IIS-1763268, a Vannevar Bush Faculty Fellowship, and a gift from the Adobe and Autodesk corporations.

A Network Architectures and Parameters

As explained in the main paper, our architectures are built with convolutional blocks, designed like bottleneck ResNet blocks [12]. This is the case whether we use a normal or strided KPConv, with rigid or deformable kernels. Figure 8 describes these blocks, and Figure 9 our two network architectures made from them. In Figure 9, we show an example of point cloud from ModelNet40 dataset, subsampled at every layer. It illustrates how the convolution radius (red sphere) grows proportionally to the subsampling grid size. In all our experiments with deformable KPConv, we use deformable kernels in the last 5 convolutional blocks (2nd block from layer 3, and both block from layer 4 and 5). The green number above layers in Figure 9 are the feature dimensions used in our blocks ($D$ in Figure 8).

Our layers process point clouds of variable sizes, so we cannot stack them along a new “batch” dimension. We thus stack our point and feature tensors along their first dimension (number of points). As the neighbor and pooling indices do not point from one input cloud to another, each batch element is processed independently without any implementation trick. We only need to keep track of the batch element indices in order to define the global pooling of KP-CNN. Since the number of points can vary a lot, we use a variable batch size by selecting as many elements as possible until a certain number of total of batch points is reached. This limit is chosen so that the average batch size correspond to the target batch size. A very similar batch strategy has already been described by [13].

KP-CNN training. We use a Momentum gradient Descent optimizer to minimize a cross-entropy loss, with a batch size of 16, a momentum of $0.98$ an initial learning rate of $10^{-3}$. Our learning rate is scheduled to decrease exponentially, and we choose the exponential decay to ensure it is divided by 10 every 100 epochs. A $0.5$ probability dropout is used in the final fully connected layers. The network converges in 200 epochs. In the case of deformable kernels, the regularization loss is added to the output loss with a multiplicative factor of $0.1$.

KP-FCNN training. We also use a Momentum gradient Descent optimizer to minimize a point-wise cross-entropy loss, with a batch size of 10, a momentum of $0.98$ an initial learning rate of $10^{-2}$. The same learning rate schedule is used and no dropout is used. Among all experiments, the network needs 400 epochs at most to converge. For real scene segmentation, we can generate any number of input spheres, so we define an epoch as 500 optimizer steps, which is equivalent to 5000 spheres seen by the network. The same deformable regularization loss is used.

Model sizes and speeds. Table 3 shows the statistics of our models on different datasets. First we notice that KP-FCNN and KP-CNN have similar number of parameters, because the decoder part of KP-FCNN only involves light 1x1 convolution. We see that the running speeds are different from one dataset to another, which is not surprising. Indeed, the number of operations performed during a forward pass of our network depends on the number of points of the current batch, and the maximum number of neighbors of these points. Our models have been prototyped with a RTX 2080Ti in this experiment, which explains the slight difference with the Titan Xp used in the main paper.

B Kernel Points Initialization

Our KPConv operates in a ball, and requires kernel points regularly placed in this domain. There is no obvious regular disposition of points in a sphere, so we chose to solve this issue by translating it into an optimization problem. The problem is simple, we want the $K$ points $\widetilde{x}_{k}$ to be as far from each other as possible inside a given sphere. We thus assign a repulsive potential to each point:

[TABLE]

And add an attractive potential to the sphere center to avoid them diverging indefinitely:

[TABLE]

The problem then consists of minimizing the global energy:

[TABLE]

The solution is found by gradient descent with the points initialized randomly and some optional constraints. In our case, we fix one of the points at the center of the sphere. For some values of $K$ (listed in Table 4), the points converge to a unique stable disposition. Those stable dispositions are in fact regular polyhedrons. Each polyhedron can be described by grouping points sharing a plane perpendicular to the polyhedron symmetrical axis. For a better understanding, some of these dispositions are shown in Figure 10.

In every layer of KP-CNN and KP-FCNN, the points locations are rescaled from the chosen stable disposition to the appropriate radius and randomly rotated. Note that $E^{tot}$ can also be used as a regularization loss in KP-CNN, when the kernel point positions are trained.

C Effect of the Kernel Point Regularization

When we designed deformable KPConv, we first used a straightforward adaptation of image deformable convolutions, but the network had very poor performances. We investigated the kernel deformations after the network convergence and noticed that the kernel points were often pulled away from the input points. This phenomenon comes from the sparse nature of point clouds, there is empty space around the points. We remind that the shifts are predicted by the network, thus, they depend on the input shape.

For a particular input during training, if a kernel point is shifted away from the input points, then the gradient of its shift $\Delta_{k}(x)$ is null. It is thus “lost” by the network and remains away for similar input shapes. Because of the stochastic nature of the network optimizer, this happens for many input shapes during convergence.

Figure 11 illustrates “lost” kernel points on the example of a room floor. First we see the rigid kernel in red, its scale gives an idea of the kernel points influence range. In the middle, the purple points depict a deformed kernel predicted by a network without any regularization loss. Most purple points are far from the floor plane and thus “lost”.

Our regularization strategy, described in the main paper, prevents this phenomenon, as shown in the bottom of Figure 11. We can notice that our regularization strategy does not only prevent the “lost” kernel points. It also helps to maximize the number of active kernel points in KPConv (those with input points in range). Almost every yellow point is close to the floor plane.

D More Segmentation Results

In this section, we provide more details on our segmentation experiments, for benchmarking purpose with future works. We give class scores for our experiments on ShapeNetPart (Table 5) and S3DIS (Tables 6 and 7) dataset. Scannet [6], Semantic3D [11] and NPM3D [31] are online benchmarks, the class scores can be found on their respective website.