Surface Networks via General Covers

TL;DR

This paper introduces a novel surface-to-image representation for sphere-type surfaces that enables the effective application of CNNs, achieving state-of-the-art results in shape analysis tasks.

Contribution

It proposes a low distortion covering map for surface-to-image representation, facilitating deep learning on geometric data with improved accuracy.

Findings

Achieves state-of-the-art results in shape retrieval and classification.

Provides a low distortion, single-image surface representation.

Enables effective CNN application to 3D surface data.

Abstract

Developing deep learning techniques for geometric data is an active and fruitful research area. This paper tackles the problem of sphere-type surface learning by developing a novel surface-to-image representation. Using this representation we are able to quickly adapt successful CNN models to the surface setting. The surface-image representation is based on a covering map from the image domain to the surface. Namely, the map wraps around the surface several times, making sure that every part of the surface is well represented in the image. Differently from previous surface-to-image representations, we provide a low distortion coverage of all surface parts in a single image. Specifically, for the use case of learning spherical signals, our representation provides a low distortion alternative to several popular spherical parameterizations used in deep learning. We have used the surface-to-image representation to apply standard CNN architectures to 3D models as well as spherical signals. We show that our method achieves state of the art or comparable results on the tasks of shape retrieval, shape classification and semantic shape segmentation.

Tables

Table 1. Table 1: Comparison of our method and the top results in each category of the SHREC17 shape retrieval task.

Method	P@N	R@N	F1@N	mAP	NDCG
FURUYA_DLAN	0.814	0.683	0.706	0.656	0.754
Tatsuma_ReVGG	0.705	0.769	0.719	0.696	0.783
SHREC16-Bai_GIFT	0.678	0.667	0.661	0.607	0.735
Deng_CM-VGG-6DB	0.412	0.706	0.472	0.524	0.642
Spherical CNN [7]	0.701	0.711	0.699	0.676	0.756
SO(3) Equivariant CNNs [12]	0.717	0.737	-	0.685	-
Ours	0.749 ($2^{nd}$)	0.741 ($2^{nd}$)	0.734	0.709	0.794

Table 2. Table 2: Results on ModelNet40 dataset.

Method	Inputs	Accuracy
Learning Gims [40]	mesh	83.9%
3DShapeNets [49]	voxels	$84.7\%$
VoxNet [29]	voxels	$85.9\%$
Pointnet[36]	points	$89.2\%$
Pointnet++ [37]	points	$91.9\%$
Dynamic graph CNN [47]	points	$92.2\%$
PCNN [2]	points	$92.3\%$
Spherical CNN [7]	spherical	$85.0\%$
SO(3) Equivariant CNNs [12]	spherical	$88.9\%$
Spherical on unstructured grid [19]	spherical	$90.5\%$
Octahedron unfolding (rot $z$)	spherical	$90.2\%$
Equirectangular projection (rot $z$)	spherical	$90.1\%$
Ours	spherical	$91.6\%$
Ours (rot $z$)	spherical	$91.0\%$

Table 3. Table 3: Results on the human segmentation dataset.

Method	Inputs	Accuracy
Toric CNN [27]	WKS,AGD,curv	$88.00\%$
Geodesic Conv [28]	3D coords	$76.49\%$
Pointnet++ [37]	3D coords	$90.77\%$
Dynamic graph CNN [47]	3D coords	$89.72\%$
Multi-directional Conv [34]	3D coords	$88.61\%$
Learning Gims [40]	3D coords	$84.53\%$
Ours	3D coords	$91.31\%$

Table 4. Table 4: Gluing instructions for choices of k , d , ρ 𝑘 𝑑 𝜌 k,d,\rho

$k$	$d$	$\rho$	Gluing instructions
3	3	$\left[{\left[3\right]}^3\right]$	$(1,2,3),(1,2,3),(1,2,3)$
3	6	$\left[{[1,5]}^3\right]$	$(1,2,3,4,5),(1,3,4,6,2),(2,6,3,5,4)$
3	9	$\left[{[1^2,7]}^3\right]$	$(1,2,3,4,5,6,7),(1,7,6,2,3,8,9),(1,9,8,2,5,4,3)$
4	2	$\left[{\left[2\right]}^4\right]$	$(1,2),(1,2),(1,2),(1,2)$
4	4	$\left[{[1,3]}^4\right]$	$(1,2,3),(2,3,4),(1,2,4),(1,2,3)$
4	6	$\left[{[1^2,4]}^4\right]$	$(1,2,3,4),(2,5,4,3),(1,5,6,4),(1,5,4,6)$
4	8	$\left[{[1^3,5]}^4\right]$	$(1,2,3,4,5),(3,6,8,5,4),(1,5,4,7,2),(2,7,4,8,6)$
4	10	$\left[{[1^4,6]}^4\right]$	$(1,2,3,4,5,6),(1,7,8,3,5,9),(2,10,8,7,6,5),(1,9,4,3,8,10)$
5	5	$\left[{[1^2,3]}^5\right]$	$(3,4,5),(2,3,5),(1,5,2),(1,2,5),(2,4,3)$
5	10	$\left[{[1^5,5]}^5\right]$	$(6,7,8,9,10),(1,7,3,4,9),(1,8,4,3,7),(2,5,4,7,6),(2,10,9,4,5)$
6	6	$\left[{[1^3,3]}^6\right]$	$(1,2,3),(2,5,3),(3,6,5),(3,5,6),(1,4,5),(3,5,4)$
6	9	$\left[{[1^5,4]}^6\right]$	$(1,9,3,5),(1,7,8,4),(3,7,5,6),(4,8,7,9),(1,3,6,2)$

Table 5. Table 5: Channel sizes of our U-Net architecture for surface segmentation

Spatial Dimensions	Layer	kernel size	# input channels	# output channels
512 x 512	Conv2d	5	3	128
	Conv2d	3	128	128
	MaxPool2d	2
256 x 256	Conv2d	3	128	128
	Conv2d	3	128	128
	MaxPool2d	2
128 x 128	Conv2d	3	128	128
	MaxPool2d	2
64 x 64	Conv2d	3	128	256
	MaxPool2d	2
32 x 32	Conv2d	3	256	512
	MaxPool2d	2
16 x 16	Conv2d	3	512	512
	Conv2d	3	512	512
	UpSample
32 x 32	Conv2d	3	1024	256
	Conv2d	3	256	256
	UpSample
64 x 64	Conv2d	3	512	128
	UpSample
128 x 128	Conv2d	3	256	128
	UpSample
256 x 256	Conv2d	3	256	128
	UpSample
	Conv2d	3	256	128
512 x 512	Conv2d	1	128	8

Equations

$$\sum _{j=1}^{l_i}{r}_{i,j}=d.$$

$$\sum _{i=1}^k\sum _{j=1}^{l_i}(r_{i,j}-1)=2d$$

$$E:I\to M$$

$$E$$

$$\Phi$$

$$\Psi$$

$$\Sigma =\left\{{\sigma }_1,\dots ,{\sigma }_k\right\}$$

$$\rho =\left[[1^{d-r},r{]}^k\right],$$

$$k(r-1)=2d.$$

$$\Sigma =\{$$

$$(3)(4)(125),(1)(5)(243)\}.$$

$$\begin{array}{rl}0& =\chi (T)=|V_T|-|E_T|+|F_T|\\ & =d\underset{\chi (M)}{\underbrace{(|V_M|-|E_M|+|F_M|)}}-\sum _{i=1}^k(d-l_i)\\ & =2d-\sum _{i=1}^k(d-l_i)\end{array}$$

$$\sum _{j=1}^{l_i}{r}_{i,j}=d$$

$$\sum _{i=1}^k\sum _{j=1}^{l_i}(r_{i,j}-1)=2d$$

$$Ax=b$$

$$\sum _{u\in N_v}{w}_{vu}\left(x_v-x_u\right)=0$$

$$v_1^{\prime }=[0,0{]}^T,v_2^{\prime }=[1,0{]}^T,v_3^{\prime }=[1,1{]}^T,v_4^{\prime }=[1,0{]}^T$$

$$\tilde{v}-v=a$$

$$\sum _{u\in N(v)}{w}_{uv}(x_v-x_u)+\sum _{u\in N(\tilde{v})}{w}_{u\tilde{v}}(x_{\tilde{v}}-x_u+a)=0$$

Full text

The surface-image representation is based on a covering map from the image domain to the surface. Namely, the map wraps around the surface several times, making sure that every part of the surface is well represented in the image. Differently from previous surface-to-image representations, we provide a low distortion coverage of all surface parts in a single image. Specifically, for the use case of learning spherical signals, our representation provides a low distortion alternative to several popular spherical parameterizations used in deep learning.

We have used the surface-to-image representation to apply standard CNN architectures to 3D models including spherical signals. We show that our method achieves state of the art or comparable results on the tasks of shape retrieval, shape classification and semantic shape segmentation.

1 Introduction

Adapting deep learning methods to geometric data (e.g., shapes) is a vibrant research area that has already produced state of the art algorithms for several geometric learning tasks (e.g., [36, 37, 42]).

Two prominent approaches are: (i) mapping the geometric data to tensors (e.g., images) and using off-the-shelf convolutional neural network (CNN) architectures and optimization techniques [42, 49, 40, 27]; and (ii) developing novel architectures and optimization techniques that are tailored to the geometric data [28, 36, 37]. An important benefit of (i) is in reducing the geometric learning task to an image learning one, allowing to harness the huge algorithmic progress of neural networks for images directly to geometric data.

Some previous attempts, following (i), to perform learning tasks on geometric data use projections to 2D planes, e.g., by rendering the shapes [42]. Such projections are not injective and suffer from occlusions, thus often require a collection of projections for a single shape. Other methods embed the shape in an encapsulating 3D grid [49, 29]; these methods require dealing with higher dimensional tensors and are usually less robust to deformations. Other methods [40, 27] try to find low distortion 2D mappings to an image domain. In this case the intrinsic dimensionality of the data is preserved, however, these maps suffer from high distortion and/or ignore the difference in the topologies of the surface (no boundary) and the image (with boundary).

In this paper, we advocate a novel 2D mapping method for representing sphere-type (genus zero, e.g., the human model in Figure 1a, left) surfaces as images. The challenge in using an image to represent a surface has two aspects: geometrical and topological. Geometrically, a general curved surface cannot be mapped to a flat domain (i.e., the image) without introducing a significant distortion. Topologically, an image has a boundary while sphere-type surfaces do not; hence, any mapping between the two will introduce cuts and discontinuities. Furthermore, a naive application of 2D convolution to the image would be ambiguous on the surface (see Figure 2 and Subsection 3.1).

To address these challenges we think of the image as a periodic domain (i.e., a torus) and relax the notion of a one-to-one mapping to that of a covering map from the image domain onto the surface. That is, we construct a mapping from the image domain to the surface that covers the surface several times. For example, Figure 1a visualizes a degree-$5$ covering map. Meaning, the surface appears $5$ times in the image; note how each part of the surface appears with low distortion at-least once in the image. The image generated by our covering map is periodic, namely its left and right boundaries as well as its bottom and top boundaries correspond, making the image boundaryless. Importantly, since image convolution is well defined on a torus, it will translate to a continuous convolution-like operator on the surface [27].

Applying our method to surface learning is easy: use a covering map to transfer functions of interest over the input surfaces (e.g., the coordinate functions) to images and apply one’s favorite CNN with periodic padding.

We tested our method in two scenarios: spherical signal learning [9, 7], and surface collection learning. For spherical signal learning, our approach provided state of the art results among all spherical methods on a shape retrieval dataset (SHREC17 [39]) and a shape classification dataset (ModelNet40 [49]). For surface collection learning, our method produced state of the art results on a surface segmentation dataset (Humans [27]). Our contributions are:

• •

We introduce a broad family of low distortion surface-to-toric image representations. The toric image representation allows applying off-the-shelf CNNs to general genus-zero surfaces.

• •

In particular, we provide a framework for learning spherical signals using CNNs.

• •

We introduce a practical algorithm for computing toric covers of genus zero surfaces.

Our code is available at https://github.com/nivha/surface_networks_covers

2 Previous work

Applying deep learning techniques to geometric data has proved to be a huge success in the last few years. A wide variety of methods were suggested, where the most popular approaches are: volumetric based methods (e.g., [49, 29]), rendering based methods (e.g., [42, 48, 51]), spectral based methods (e.g., [5, 10]) and methods that operate directly on the surface itself (e.g., [28]). A popular related problem is the problem of learning on point clouds which received a lot of attention lately (see e.g., [36, 2, 25]).

Here, we restrict our attention to intrinsic or parameterization-based surface methods and refer the reader to the above mentioned works and a recent survey [4] for further information.

Local parameterization. Such methods (e.g., [28, 3, 30]) extract local surface patches and use them in order to learn point representations. In [28] the authors use local polar coordinates as the patch operator. In a follow-up work, [3] use projections on oriented anisotropic diffusion kernels, where [30] learn the patch operator using a Gaussian mixture model. In contrast to these works, we employ a global parameterization which represents the shape using a single image.

Global parameterization. Other methods use global parametrization of the surface to a canonical domain. [40, 41] use an area-preserving parameterization and map surfaces to a planar domain (going through a sphere); the global area-preserving parameterization cannot cover the surface with low distortion everywhere and depends on the specific cut made on the surface.

The most similar method to ours is [27] that proposes gluing four copies of the surface into a torus and map it conformally (i.e., preserving angles) to a flat torus, where the convolution is well defined. Their map is defined by a choice of three points on the surface, and suffers from significant angle and scale distortion, see Figure 1b (e.g., the head, right arm and torso). In order to cover each point on the surface reasonably well, the authors sample multiple triplets of points from each surface where each triplet focuses on a different part of the surface. In a follow up work, [15] use the same parameterization as a surface representation for Generative Adversarial Networks (GANs) [13]. In order to deal with the high distortion of each single parameterization, the authors devise a multi-chart structure and rely on given sparse correspondences between the surfaces.

Convolutions on tangent planes. [46] define convolutions on surfaces by working on the tangent planes. [31] also define the convolutions on tangent planes and relate convolutions on nearby points using parallel transport. [34] define convolutions on surfaces by extending the notion of a signal on a surface into a directional signal and build layers that are equivariant to the choice of reference directions. [17] utilizes 4-rotational symmetric field to define a domain for convolution on a surface.

Convolutions of spherical signals. Our work targets learning of general genus zero surfaces. In particular, it can facilitate learning of spherical signals, a task that has received growing interest in the last few years. [43, 9, 52] note that an equirectangular projection of a spherical signal suffers from large distortions and suggest network architectures that try to compensate for these distortions. [6] perform 2D convolution on spherical strips extracted from the spherical signal. [19] suggest to define the convolution of a spherical signal as a linear combination of differential operators with learned weights. In a different line of work, [7, 12, 23] propose networks that are invariant to the natural action of $SO(3)$ on spherical signals. [8] advocate the notion of gauge equivariance as the correct equivariance notion on manifolds, and construct gauge equivariant networks on spheres.

Other methods. [50] tackle the shape segmentation problem by a novel architecture that operates on local features (such as normals) and global features (such as distances) and then fuses them together. [24] propose an improved graph neural network model based on the Dirac operator.

3 Preliminaries

In this section we discuss our choice of periodic images (i.e. images with toric topology) and introduce branched covering maps, the main mathematical tool used in our approach.

3.1 Convolutions on flattened spheres

A standard way to apply CNNs to a signal on a sphere-type surface is to represent it as an image and apply standard 2D convolution. Since representing a sphere as an image requires cutting and duplicating the cuts, different boundary segments in the image represent the same segment on the sphere.

In the case where the transformation in the image domain between the two duplicated boundary segments is a pure translation then the result of applying 2D convolution at any two matching points on these segments will result in exactly the same value. In other cases, such as equirectangular spherical projection [43] or octahedron spherical projection [35, 40], 2D convolution on two matching points result in two different values. Figure 2 shows an example where duplicated image boundary segments are marked with the same color arrows; a pair of matching points (marked $\mathrm{P}$) are shown in each example along with an illustration of a convolution kernel. Note that only in the toric topology the kernel is consistent at the duplicated points. A similar point of view for toric images was suggested in [27]. We extend it to a more general family for toric images of sphere-type surfaces.

3.2 Branched covering maps

This section provides a brief introduction to branched covering maps (for more details see [16]). We start with a formal definition:

Definition 1**.**

Let $X$ and $Y$ be topological spaces. A map $E:X\to Y$ is a branched covering map if every point $y\in Y$ except for a finite set of points $\{b_{1},\ldots,b_{k}\}$ has a neighborhood $U\subseteq Y$, such that $E^{-1}(U)$ is a disjoint union of homeomorphic 111A homeomorphism is continuous map with a continuous inverse. copies of $U$.

The set of points $\{b_{1},\ldots,b_{k}\}$ are called branch points.

A simple example for a branched covering map is ${E(z)=z^{d}}$, for $X=Y=\mathbb{C}$, and for some integer $d$. The function $E$ has one branch point at $b_{1}=0$. Every point $y\in Y\setminus\left\{0\right\}$, has $d$ distinct pre-images ${E^{-1}(y)=\{x_{1},\ldots,x_{d}\}\subset X}$. However, the point $y=0$ has a single pre-image $E^{-1}(y)=\{0\}$. We say that the point $y=0$ has $d$ pre-images located at [math], or that [math] is a pre-image with multiplicity $d$. The ramification index of $x$ over $E(x)$ is the multiplicity at $x$, namely $1$ for all $x\in E^{-1}(Y\setminus\left\{0\right\})$ and $d$ for $x=E^{-1}(b_{1})=0$. We denote it as $r(x|E(x))$. Figure 3b shows this example for $d=4$. In fact, this example captures all the local behaviors of covering maps: around a point $x\in X$ with $r(x|E(x))=r$ the map $E$ looks like the map $z\mapsto z^{r}$.

Let us give another example: Consider the function ${E(z)=z^{2}(z-7)}$. It has a branch point at $y=0$ with two distinct pre-images. Namely, $E^{-1}(0)=\{0,7\}$. Here, the ramification index of [math] over $E(0)$ is $2$ and the ramification index of $7$ over $E(7)$ is $1$. We say that the ramification structure of [math] is $[2,1]$, formally:

Definition 2**.**

Let $E:X\to Y$ be a branched covering map, $b_{i}$ a branch point and $l_{i}=|E^{-1}(b_{i})|$, the number of pre-images of $b_{i}$. The ramification structure of $b_{i}$ is the multi-set of ramification indices of its pre-images, denoted by $\rho_{i}=[r_{i,1},\ldots r_{i,l_{i}}]$. The ramification type of $E$ is the collection of its ramification structures, $\rho=[\rho_{1},\ldots,\rho_{k}]$.

Figure 3a depicts a branch point $b_{1}$ with three distinct pre-images, $l_{1}=3$, and ramification structure $\rho_{1}=[2,1,3]$. Note that the ramification structure of a non-branch point is a trivial multi-set of ones: $[1\dots 1]$, see e.g., the red dot in Figure 3a.

The sum of the ramification indices of any point in $X$ is independent of the choice of the point (see [11], page 44 Proposition 7), namely

[TABLE]

Lastly, $d$ is called the degree of the covering. Intuitively, the degree of the covering counts how many times $X$ covers $Y$, or alternatively how many copies of $Y$ can be found in $X$.

3.3 Riemann-Hurwitz formula

A key fact about ramification types of branched covering maps between (boundaryless) surfaces is the Riemann-Hurwitz formula (RH), which connects the genus (i.e., number of handles) of the surfaces with the ramification type. In our case, we map a torus to a sphere-type surface and get the corresponding RH formula:

[TABLE]

A quick derivation of this formula is given in Section E.1.

Therefore, the RH formula sets a necessary condition on the possible ramification types $\rho$ of such branched covering maps. For example, the ramification type $\left[[2],[2],[2],[2]\right]$ satisfies the RH equations but the ramification type $\left[[2],[2]\right]$ does not (in this case $d=2$, $k=2$, $l_{i}=1$), implying that there is no covering map with this ramification type. We note that Equations (1) and (2) are necessary but not sufficient conditions.

4 Approach

Our goal is transferring signals (i.e., functions) from a sphere-type surface $M$ to the image domain $I$ (i.e., the flat torus: unit square $[0,1]^{2}$ with opposite ends identified). This is done by constructing a branched covering map

[TABLE]

and pulling back the signals to the image using $E$. That is, given a signal $f:M\rightarrow\mathbb{R}^{n}$ that we want to transfer, the value of a pixel $p\in I$ is set to $f(E(p))$. We represent the surface $M$ using a triangular mesh.

We build the covering map $E:I\rightarrow M$ in two steps, as a composition of two functions:

[TABLE]

where $T$ is a torus-type surface built out of $d$ copies of $M$, $\Psi$ is a branched covering map, and $\Phi$ is a homeomorphism between the two tori $I$ and $T$ (see Figure 4 for illustration).

4.1 Computing the branched covering map $\Psi$

In this section we describe how we construct the mesh $T$ out of the mesh $M$ and the branched covering map $\Psi:T\to M$. The idea is to cut and glue together several copies of the input surface $M$ in a way that generates a toric covering space corresponding to a specific choice of $\rho$.

First, we choose $k$ branch points $b_{1},\ldots,b_{k}$ from the set of vertices of $M$ (using farthest point sampling), a degree $d$ and a valid ramification type $\rho$ satisfying Equations (1)-(2). Our algorithm then consists of the following steps:

Step (i): We cut the mesh along $k$ disjoint paths, all emanating from the same (arbitrary) vertex $v_{0}$ in $M$ and ending at the branch points $b_{i}$ for $i\in[k]$. Figure 5 shows this for $k=d=5$. Topologically, $M_{disk}$ is a disk, with all branch points at its boundary.

Step (ii): $M_{disk}$ is then duplicated $d$ times, to form copies $M_{disk}^{\scriptscriptstyle(1)},\ldots,M_{disk}^{\scriptscriptstyle(d)}$. Figure 5 shows the $5$ copies with $v_{0}$ as a white dot and the branch points as colored dots.

Step (iii): We glue the $d$ copies of $M_{disk}$ to create the surface $T$ as follows. Consider a branch point $b_{i}$; it has $d$ copies located in each of the copies of $M_{disk}$, see e.g., the blue dots in Figure 5. Denote by $B_{j}$ and $A_{j}$ the two boundary edges emanating from the $j$-th copy of $b_{i}$. Note that on the original surface $A_{j}$ is glued to $B_{j}$; since every $B_{j^{\prime}}$ is a duplicate of $B_{j}$, $A_{j}$ can be glued to any $B_{j^{\prime}}$, $j^{\prime}\in[d]$. Therefore, to describe the gluing of the edges emanating from $b_{i}$ we use a permutation $\sigma_{i}\in S_{d}$ (a permutation is a bijection $[d]\rightarrow[d]$): $A_{j}$ is glued to $B_{\sigma_{i}(j)}$. The collection of all permutations (one permutation per branch point)

[TABLE]

is called the gluing instructions. Given gluing instructions $\Sigma$ we use it to stitch the boundary of the $d$ copies of $M_{disk}$ to construct the toric surface $T$ (i.e., genus one). The mapping $\Psi:T\to M$ is then defined by: map $v\in T$ to its original version in $M$, and extend linearly in each triangle (i.e., face) of $T$. $\Psi$ is a well defined branched covering map. The gluing procedure is summarized in Algorithm 1. In Subsection 4.1.1 we describe the algorithm for computing the gluing instructions given the desired ramification type $\rho$.

4.1.1 Computing the gluing instructions

In this paper we limit our attention to ramification types of the form

[TABLE]

where $d$ is the cover degree, $k$ is the number of branch points, and $r$ is the maximal multiplicity of the branch points’ pre-images. The motivation in choosing these ramification types is two-fold: First, we want all branch points to be treated equally by the cover. Second, applying higher ramification order improves area distortion of protruding parts (see e.g., [21]); See Figure 1 and Subsection 4.3 for an example.

First, let us compute necessary conditions for $\rho$ defined in (5) to be a feasible ramification type. Equation (1) is automatically satisfied since $d-r+r=d$. Plugging $\rho$ in (2) we get

[TABLE]

This sets a trade-off between $r$ and $d$: higher values of $r$, while reducing distortion of protruding parts would force higher degree $d$ of the cover, which will produce more copies of $M$ in the image. Practically, we found that $k=5,r=5,d=10$ and $k=6,r=2,d=3$ are both good options that strike a good balance between $r$ and $d$.

To compute gluing instructions $\Sigma$ we start with $k,r,d$ satisfying (6). The next theorem (proved in Section E.2)provides a necessary and sufficient condition for the gluing instructions $\Sigma$ to furnish a cover with ramification type $\rho$:

Theorem 1**.**

A set of gluing instructions $\Sigma=\left\{\sigma_{1},\ldots,\sigma_{k}\right\}$ yields a branched covering map with ramification type $\rho$ if and only if the following conditions hold:

1. (i)

The cycle structure of $\sigma_{i}$ equals the ramification structure of $b_{i}$, i.e.**, $\rho_{i}=[r_{i,1},\ldots,r_{i,l_{i}}]$. 2. (ii)

$\Sigma$* is a product one tuple. That is, $\sigma_{1}\cdot\sigma_{2}\cdots\sigma_{k}=I_{d}$.* 3. (iii)

The group $G$ generated by $\Sigma$ is a transitive subgroup of $S_{d}$. Namely, for each $i,j\in[d]$ there exists $\sigma\in G$ so that $j=\sigma(i)$.

Theorem 1 indicates that we should search for permutations $\sigma_{i}$ with prescribed cycle structures. That is, the permutations $\sigma_{i}$, if exist, are in some prescribed conjugacy classes of the permutation group. Algorithm 2 performs such a search, more or less exhaustively, using conditions (ii) and (iii) to prune options that cannot lead to a solution $\Sigma$.

Since theoretically not all $k,r,d$ satisfying Equation (6) have a corresponding covering map, Algorithm 2 can terminate without finding gluing instructions. In this case, according to Theorem 1 we know that there is no covering map with ramification type $\rho$. Nevertheless, it is rare to find such examples in practice and indeed we did not encounter such a case in our experiments. Table 4 contains the results of Algorithm 2 for any permissible $k,r,d$ with $k\leq 6,d\leq 10$ so that they can be used as input to Algorithm 1.

4.2 Flattening the toric surface

The last part of our covering map computation is the computation of the map $\Phi:I\rightarrow T$. Equivalently, we compute $\Phi^{-1}$. To that end we use a version of the Orbifold-Tutte embedding [1]. We first cut $T$ along the two generating loops of the torus (using [20], Algorithm 5) to get a disk-type surface $T_{disk}$. Second, we compute a bijective piecewise affine map $\Phi^{-1}:T\rightarrow I$ by solving a sparse linear system of equations $Ax=b$, where $A\in\mathbb{R}^{m\times m}$ and $x,b\in\mathbb{R}^{m\times 2}$, and $m$ is the number of vertices in the disk-like mesh $T_{disk}$. This system is a discrete version of the Poisson equation [26], see Section G for details on how to construct $A,b$. We use $x$ to map the vertices of $T$ to $I$ and extend linearly to get the piecewise affine map $\Phi^{-1}$.

The resulting map is discrete harmonic [26], approximately conformal up to a linear transformation, and as proven in [1], a bijection.

4.3 Example

Figure 1a depicts the case $k=5$, $r=3$, $d=5$. Thus $\rho=\left[[1,1,3]^{5}\right]$; every branch point $b_{i}$ has three distinct pre-images, where two have ramification one, and one with order-$3$ ramification. The gluing instructions in this case, computed using Algorithm 2, are:

[TABLE]

Note that each of these permutations has a cycle structure $[1,1,3]$ as required in Theorem 1 (i); conditions (ii)-(iii) can be checked as well. These gluing instructions were used to glue the $5$ copies of $M_{disk}$ (as shown in Figure 5 and described in Algorithm 1) to generate the representation $E:I\to M$ shown in Figure 1a.

5 Experiments

To evaluate the efficacy of our method we tested it in two main scenarios: learning signals on the sphere, and learning sphere-type surface data.

5.1 Evaluation

In this section we compare the geometric properties of our representation to standard or existing techniques. Figure 6 shows the area and scale distortion of our method (right, in blue) and two other popular methods for sphere flattening: Equirectangular projection (see e.g., [43]) and octahedron unfolding projection, see [35]. Area distortion is computed as the determinant of the differential of the cover map $E$ (treated as affine over each triangle of $M$), and angle distortion is the condition number of the differential. Since our image representation contains several copies of each triangle of $M$ we use the least distorted one for the histogram, as we want each part of the surface to appear in the image at-least once with low distortion. As can be seen in Figure 6, our projection has better angle preservation with only a mild sacrifice to area distortion.

In Figure 7 we repeat this experiment with a sphere-type model of a human and compare the area and angle distortion of five different types of image representations. While the method of [40] (leftmost, in red) preserves area better, it suffers from significant angle distortion. The orbifold covering of [27] (second to the left, in red) is angle-preserving, but suffers from notable area shrinking. Our covering maps (green and blue) strike a balance between angle and area preservation. The covering of type $[[1^{5},5]^{5}]$ (middle, in blue) has the least area distortion and we chose it for the segmentation task (below).

The top row of Figure 7 compares the different image representations by reconstructing the original model. Specifically, for each vertex of the mesh we sampled its $x,y,z$ coordinates directly from the image at the vertex location (we used $512\times 512$ images here). In our representation, we take the coordinates from the vertex copy with the least area distortion. Note that the image representations of [40] and [27] do not represent well significant parts of the surface (e.g., the right leg and the head).

5.2 3D shape retrieval

The first application of our method is 3D shape retrieval. We use the SHREC2017 benchmark [39] that contains $51162$ 3D models from $55$ different categories. There are two separate challenges: (i) the shapes are consistently aligned (ii) the shapes are randomly rotated. We tackle the (harder) second challenge.

Since the shapes are not of genus zero we follow the protocol of [7] that project the meshes on a bounding sphere using ray casting, and record six functions on this sphere: distance to the model, $cos/sin$ of the model angles (this is done for both the model and its convex-hull). We then use our method to transfer these six spherical signals to periodic images (flat torus). See Figure 8 for an example of such shape representation.

We compare our method to the top methods in each category, the Spherical CNN method [7], and the recent SO-3 equivariant networks suggested in [12]. The results are summarized in Table 1; note that in the F1 measure we score first among all methods.

For this application we use a slight modification of the inception v3 architecture [45]. We train the network with ADAM optimizer [22] for $100$ epochs with learning rate $0.05$, batch size of $32$, and learning rate decay of $0.995$. Training took $15$ minutes per epoch on a Tesla V100 Nvidia GPU. In evaluation time we average the output of the network on $5$ randomly rotated copies of the query model.

5.3 Surface classification

We apply our method to the ModelNet40 surface classification benchmark [49] that contains $12311$ 3D models from 40 different categories. As in the shape retrieval task, we follow the protocol of [7] to generate input signals on a sphere. We then use our method to represent the spherical signals as periodic images and apply the same inception v3 model as in the shape retrieval task. We present peak performance results (following [19]) for two scenarios that are popular in the literature: (i) the shapes are rotated randomly about the $z$ axis; and (ii) the shapes are learned in their original orientation. We train the network with ADAM optimizer [22] for $100$ epochs for scenario (1) and $300$ for scenario (ii) with learning rate $0.0005$, batch size $16$, and learning rate decay $0.995$. Training took $19$ minutes per epoch for the first scenario (that contains $10$ rotation augmentations) and $3$ minutes per epoch for the second scenario on a Tesla V100 Nvidia GPU.

Table 2 compares our results with several recent methods including the baselines of equirectangular projection (e.g., [43]) and octahedron unfolding projection [35]. Our results are the best among all spherical learning methods.

5.4 Surface segmentation

While our first two application targeted spherical signals, our last applications learns signals defined on general sphere-type human models. In particular, we perform human model semantic segmentation. We use the benchmark from [27] that consists of 373 train models from multiple sources and 18 test models. $5\%$ randomly sampled train models were used as a validation set (18 models). All models are given as triangular meshes. For each model, each face is labeled according to a predefined partition of the human body (e.g., head, torso, hands, total of $8$ labels). The task is to label the triangles of a new unseen human model with these labels. For each model we generate an augmented set of $120$ images per mesh, by permuting the order of the branch points, multiplying the vertices by a random orthogonal matrix and a uniform scale sampled from $[0.85,1.15]$ as suggested in [34], and small periodic image translations of $\pm 15$ pixels. In evaluation, as the toric image contains $d$ values for each triangle on the original mesh, we use the label of the triangle with the largest area. Furthermore, we use 10 random augmentations of test images and label each mesh face using a majority vote. Table 3 summarizes the results of this experiment, where our method outperformed previous methods; Figure 9 shows typical segmentation results.

For this application we used the U-net architecture [38] with $16$ layers (see Table 5 for details). We used a weighted loss with equal probability labels, and trained the network using stochastic gradient descent with momentum [44] for $50$ epochs with learning rate $0.2$, batch size $2$, and learning rate decay of $0.995$. Training takes $\sim 3$ hours per epoch on a Tesla V100 Nvidia GPU.

6 Conclusions

In this paper, we introduce a new method for representing sphere-type surfaces as toric images that can be used in standard Convolutional Neural Network frameworks for shape learning tasks. The method allows faithful representation of all parts of the surface in a single image, thus alleviating the need to generate multiple maps to cover each surface. Our method is general and can target both spherical signal learning tasks as well as more general learning tasks that involve signals on different genus zero surfaces. Practically, we showed that off-the-shelf CNN models applied to images generated with our method lead to state of the art performance in the tasks of shape retrieval, shape classification and surface segmentation.

The main limitation of this work is its restriction to genus-zero surfaces. This kind of models are abundant, but certainly do not exhaust all 3D models. We would like to seek a generalization of this method to point clouds, depth images and more general topological types.

7 Acknowledgements

This research was supported in part by the European Research Council (ERC Consolidator Grant, ”LiftMatch” 771136) and the Israel Science Foundation (Grant No. 1830/17).

Appendix A Convolution on a spherical mesh

In Figure 10 we depict a cover map from the torus (texture square image on the left) to a human surface (middle); this map covers the human $5$ times. We further show how standard convolution stencil (in yellow) translates to a seamless convolution on the surface. Note that the texture seams on the human models are pretty arbitrary and just indicate when moving to a different copy of the surface.

Appendix B Guidelines on Choosing parameters

Adding branch points helps reducing the local distortion in protruding parts, therefore we recommend to choose as many branch points as there are protruding parts common in the dataset (e.g. $5$ for humans, $8$ for octopuses etc.). As we mentioned in section 4.1.1 we choose a ramification type of the form $[[r,1^{d-r}]]$ for each branch point.

As noted in Section 4.1.1, higher ramification ($r$) also improves area distortion of protruding parts. However, in that case, we are limited by the RH formula (Equation 6). So we would recommend choosing the highest $r$ possible (e.g. as appears in Table 4) and taking $d$ (number of copies) to satisfy Equation 6. Also note that higher $r$ implies higher $d$ (number of copies). Therefore, for a fixed image resolution we would like the highest number of branch points for which all relevant parts are still visible in the image.

Appendix C Gluing Instructions

As mentioned in Section 4.1.1, for each choice of number of branch points $k$ , degree $d$ and ramification type $\rho$ satisfying Equations (5) and (6) We need to compute a product one tuple of permutations satisfying the conditions of theorem 1. We note that this computation can be done in an offline step, before using Algorithm 1 to compute the toric parameterization. In Table 4 We provide gluing instructions corresponding to each valid choice of $k\leq 6$, $d\leq 10$ and $\rho$ that complies with Equations (5) and (6). Each of the gluing instructions in Table 4 can be used as input to Algorithm (1).

Appendix D Implementation Details

Learning.

We use Pytorch [32] for learning. All the experiments are done with toric images generated by our algorithm and off-the-shelf CNN architectures with a single change: we replace the standard zero padding with periodic padding.

Data generation.

For the surface segmentation task we use a cover of the type ${\rho=\left[[1^{5},5]^{5}\right]}$, that is, ${\rho_{i}=[1^{5},5],i\in[5]}$. For the spherical learning tasks (shape retrieval and classification) we use a cover of type ${\rho_{i}=[1,2],i\in[6]}$. The locations of the branch points are chosen using farthest points sampling. We use the shortest paths from an arbitrary base point to all branch points in order to cut the mesh. When the mesh does not allow such a path we subdivide it locally (without changing its geometry). This pre-processing step is implemented in Matlab. It takes $\sim 22$ seconds in average (relatively long running time due to a non-optimized mesh cutting code in Matlab) to generate a periodic (toric) image for a mesh with $6890$ vertices on a single CPU core in an Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz machine.

D.1 Segmentation Task

Prediction.

The network outputs per-pixel labels. In order to obtain a label for each face in the original mesh $M$, we first transfer the per-pixel logits to the faces $F_{T}$ of the toric mesh using bilinear interpolation sampled at the faces’ centers. Since each face $f$ in $M$ has $d$ duplicated faces in the toric mesh $T$ ($|\Psi^{-1}(f)|=d$), each face $f$ in $M$ has $d$ sets of logits. We use a weighted average of the $d$ sets of logits, where the weights are the area scales of the faces $\Psi^{-1}(f)$. The label of $f$ is the argmax of this weighted-average of logits. This means that better scaled faces (in the toric mesh) receive more weight when deciding how to label a face in the original mesh $M$.

Architecture.

We use a version of a U-Net [38]. The feature-channels sizes are given in Table 5. After each convolution we use $\mathrm{ReLU}$ with a Batch-Normalization layer [18]. Each UpSample layer is a nearest-neighbour interpolation with scale-factor 2.

Appendix E Proofs

E.1 Riemann-Hurwitz formula

Consider a branched covering map $E:T\to M$ of degree $d$ and $k$ branch points, from a toric mesh $T=(V_{T},E_{T},F_{T})$ to a spherical mesh $M=(V_{M},E_{M},F_{M})$. We prove that the ramification type of $E$ must satisfy the Riemann-Hurwitz formula (9).

Proof of Riemann-Hurwitz formula.

First, we note that the set of branch points $B=\{b_{1},\ldots b_{k}\}$ can always be chosen from $V_{M}$.

Every node $v\in V_{M}\setminus B$ has $d$ pre-images in $V_{T}$. However, a branch point $b_{i}$ has $l_{i}<d$ pre-images in $V_{T}$. Every edge $e\in E_{M}$ has exactly $d$ pre-images in $E_{T}$, that is ${|E_{T}|=d|E_{M}|}$. Similarly, ${|F_{T}|=d|F_{M}|}$.

By computing the Euler characteristic for a toric surface:

[TABLE]

Using

[TABLE]

and rewriting we obtain the Riemann-Hurwitz formula (RH), in its version for a map from a toric surface to a spherical surface:

[TABLE]

∎

E.2 Proof of Theorem 1

We recall the following topological facts. A degree $d$ branched covering map $E:T\to M$ from a torus to a genus [math] surface induces a group homomorphism, called the monodromy representation, from $\pi_{1},$ the fundamental group of $M\setminus\left\{b_{1},\ldots,b_{k}\right\}$ to $S_{d}$.

The homomorphism is given as follows: We take each loop $l\in\pi_{1},$ based at a point $p$, and lift it to $T$ starting from a preimage of $p$. This lift has to end at another preimage of $p$. Due to properties of the lifting, this induces a permutation on the preimages of $p$ in $T$, referred to as the fiber of $p$.

The group $\pi_{1}$ has $k$ generators and a single relation. The generators, $l_{1}\ldots,l_{k},$ are the $k$ loops around each of the branch points. The relation is $l_{1}*\ldots*l_{k}=1$.

Our gluing instructions, $\sigma_{1},\ldots,\sigma_{k}$, will be the images of $l_{1},\ldots,l_{k}$ under the monodromy representation. We shall now give a proof of Theorem 1 . Namely, that our algorithm produces a cover $T\to M$ with ramification $\rho$ if and only if the gluing instructions are a tuple of permutations satisfying the conditions of Theorem 1.

Proof of Theorem 1.

First we prove that the conditions in the theorem are necessary.

For $(i)$, we note that a lift of a loop around a branch point $l_{i}$ with a particular ramification structure induces a permutation with the same cycle structure.

For $(ii)$, the fact that $l_{1}*\ldots*l_{k}=1,$ implies (using group homomorphism) that $\sigma_{1}\cdot\ldots\cdot\sigma_{k}=I_{d}.$

For $(iii),$ fix $p_{1},p_{2}$ in the fiber of $p.$ Since $T$ is connected, there exists a path $\gamma$ connecting $p_{1}$ and $p_{2}$. The loop $E\circ\gamma$ is a loop starting and ending at $p$ whose lift takes $p_{1}$ to $p_{2}$. Thus, the action of group generated by $\Sigma=\left\{\sigma_{1},\ldots,\sigma_{n}\right\}$ is transitive.

Conversely, suppose we have a product one tuple $\sigma_{1},\ldots,\sigma_{k}$ satisfying the conditions of the theorem and $k$ branch points $b_{1},\ldots,b_{k}$. Then condition (i) allows us to define an action of the group $H:=\left\langle\sigma_{1},\sigma_{2},\ldots,\sigma_{k}\right\rangle$ on $[d]$. Following the construction in [16] pg 68-70 the space $\nicefrac{{U\times[d]}}{{\pi_{1}\times H}}$ is a covering space of $M$, where $U$ is the universal cover of $M$. The transitivity of $H$ implies that this covering space $C$ is connected. Condition $(iii)$ implies by the Riemann-Hurwitz formula that $C$ is topologically a torus.

Let $D$ be the space produced from Algorithm 1. Note that the construction in Algorithm 1 implies that lifting a loop circling each branch point $b_{i}$ induces the permutation $\sigma_{i}$ on the fiber of a generic point. Thus, the action of $\pi_{1}$ on $D$ coincides with the action of $\pi_{1}$ on $C$. Since every action of $\pi_{1}$ on $[d]$ (up to conjugation) produces a unique (up to homeomorphism) covering space, we deduce that $D$ is homeomorphic to $C$.

∎

Comment:

The equivalence between branched covering maps and tuples of permutations satisfying the conditions of Theorem 1 is well known. This equivalence is commonly referred to as Riemann’s existence theorem (RET). However, to the best of our knowledge, it was previously not known how to practically construct any given branched covering map (our Algorithm 1).

Appendix F Gluing Instructions

We now turn to describing an algorithm that finds tuples of permutations $\sigma_{1},\ldots,\sigma_{k}\in S_{d},$ corresponding to a prescribed ramification structure $\rho$, up to simultaneous conjugation (relabeling of the branch points). We call such a tuple a product one tuple. We implement our algorithm using Magma computational algebra system [14].

We denote the conjugacy class in $S_{d}$ associated with the cycle structure of $\rho_{i}$ by $C_{i}$. In the algorithm construction we use the following:

Claim 1**.**

$\left\langle\sigma_{1},\sigma_{2},\ldots,\sigma_{k-1}\right\rangle$* is a transitive permutation group and $\Pi_{i=1}^{k-1}\sigma_{i}\in C_{n},$ if and only if $\sigma_{1},\sigma_{2},\ldots,\sigma_{k}$, where $\sigma_{k}=\left(\sigma_{1}\sigma_{2}\cdots\sigma_{k-1}\right)^{-1}$ is a transitive product one tuple with $\sigma_{k}\in C_{k}$.* 2. 2.

The set $\left\{\sigma_{1},\ldots,\sigma_{i}\right\}$ can be completed to a transitive product one tuple compatible with a ramification structure $\rho$ if and only if $\left\{\sigma_{1},\ldots,\sigma_{i-1},g\sigma_{i}g^{-1}\right\}$, for any $g\in Z(\sigma_{1},\ldots,\sigma_{i-1})$ ($Z$* denotes the centralizer), can be completed to a transitive product one tuple compatible with $\rho$.*

Proof.

(1) follows from the observations that adding elements to a transitive generator set keeps the set transitive, and that for $g\in S_{d}$ the cycle structure of $g$ and $g^{-1}$ are the same. For (2), note that for any $g\in Z\left(\sigma_{1},\ldots,\sigma_{i-1}\right)$ and $j\in\left[i-1\right]$ it holds that $g\sigma_{j}g^{-1}=\sigma_{j}$. Thus, for any $g\in Z\left(\sigma_{1},\ldots,\sigma_{i-1}\right)$, we have that any tuple with $\sigma_{1},\ldots,\sigma_{i}$ is the same as a tuple with $g\sigma_{1}g^{-1},\ldots,g\sigma_{i}g^{-1}$, up to simultaneous conjugation. ∎

The main idea in the algorithm for finding all gluing instructions corresponding to a ramification type $\rho$ is to exhaustively go over all tuples $\sigma_{i}\in C_{i}$ and check whether they form a product one tuple. We use the claim above to prune this exhaustive search, as described in Algorithm 2. Note that this computation is done once for a given cover ramification type and is reused for all models using this type of cover.

Appendix G Orbifold-Tutte embedding of $T$

We compute $x$ by solving a sparse linear system following [1]:

[TABLE]

Here $A\in\mathbb{R}^{m\times m}$ and $x,b\in\mathbb{R}^{m\times 2}$, where $m$ is the number of vertices in the disk-like mesh $T_{disk}$. The linear system (10) is constructed by putting together four sets of linear equations as follows:

First, for all interior vertices we set the discrete harmonic equation:

[TABLE]

where $N_{v}$ is the set of vertices in $V_{T_{disk}}$ adjacent to $v$ and $w_{uv}$ are the cotangent weights [33].

Let $L_{1}$ and $L_{2}$ be the generators of the homotopy group of $T$. Denote by $v_{0}\in V_{T}$ the intersection of the two loops $L_{1}$ and $L_{2}$ . In $T_{disk}$, the vertex $v_{0}$ has four copies $v_{1}^{\prime},v_{2}^{\prime},v_{3}^{\prime},v_{4}^{\prime}$. next, we ensure that these four copies are mapped to the four corners of the unit square $[0,1]^{2}$. Explicitly,

[TABLE]

Each vertex $v\in\partial V_{T_{disk}}\setminus{\{v_{1}^{\prime},v_{2}^{\prime},v_{3}^{\prime},v_{4}^{\prime}\}}$ has a twin vertex $\tilde{v}$ such that $v$ and $\tilde{v}$ correspond to the same vertex in the uncut mesh $T$. Moreover, each such vertex $v$ has its origin in $V_{T}$ either in $L_{1}$ or in $L_{2}$.

We set the vertices whose origin is in $L_{1}$ to be different by a constant translation in $[0,1]^{T}$ and the vertices whose origin is in $L_{2}$ to be different by a constant translation in $[1,0]^{T}$. Namely:

[TABLE]

where $v$ and $\tilde{v}$ are twins, and $a$ is either $[0,1]^{T}$ or $[1,0]^{T}$, depending on whether the origin of $v$ belongs to $L_{1}$ or $L_{2}$.

Finally we set each vertex $v\in\partial V_{T_{disk}}\setminus{\{v_{1}^{\prime},v_{2}^{\prime},v_{3}^{\prime},v_{4}^{\prime}\}}$ to be the weighted average of both its neighbors and the translated neighbors of its twin.

[TABLE]

with $a$ as before.