A scaled space-filling curve index applied to tropical rain forest tree distributions

TL;DR

This paper introduces a scaled Gray-Hilbert space-filling curve index that improves data handling efficiency for high-dimensional spatial data, demonstrated on a large tropical rainforest dataset, outperforming static versions especially with varied data distributions.

Contribution

It presents a scalable, data-driven scaled Gray-Hilbert curve index that adapts to different data distributions, enhancing spatial data processing in high-dimensional contexts.

Findings

The scaled Gray-Hilbert index outperforms static versions in space efficiency.

The index adapts to different data distributions through a local sparsity measure.

Visualization of binary trees illustrates the influence of data tail distributions.

Abstract

In order to be able to process the increasing amount of spatial data, efficient methods for their handling need to be developed. One major challenge for big spatial data is access. This can be achieved through space-filling curves, as they have the property that nearby points on the curve are also nearby in space. They are able to handle higher dimensional data, too. Higher dimensional data is widely used e.g. in CityGML and is becoming more and more important. In a laboratory experiment on a tropical rain forest tree data set of 2.5 million points taken from an 18-dimensional space, it is demonstrated that the recently constructed scaled Gray-Hilbert curve index performs better than its standard static version, saving a significant amount of space for a projection of the data set onto 8 attributes. The implementation is based on a binary tree in a data-driven process, in a similar way as e.g. the R-tree. Its scalability allows the handling of different kinds of data distributions which are reflected in the tree structure of the index. The relative efficiency of the scaled Gray-Hilbert curve in comparison with the best static version is seen to depend on the distribution of the point cloud. A local sparsity measure derived from properties of the corresponding trees can distinguish point clouds with different tail distributions. The different resulting binary trees are visualised to illustrate the influences of the different tail distributions they have been built on.

Tables

Table 1. Table 1: Values of of the Gray-Hilbert capacity ratio for the projection onto all eight variables.

${\log }_{2}s$	bubble	ring
0	0.076	0.076
1	0.052	0.052
2	0.032	0.032
3	0.018	0.018
4	0.010	0.010
5	0.006	0.005
6	0.644	0.649
7	0.342	0.355
8	0.180	0.185
9	0.097	0.092
10	0.051	0.050
11	0.026	0.026
12	0.015	0.014
13	0.009	0.007
14	0.912	0.804

Table 2. Table 2: Values of ρ 𝜌 \rho for projections of the data onto certain sets of attributes, and different bucket capacitys s 𝑠 s . The values in boldface have almost identical tail distributions (cf. Figure 5 ).

	$s=1$		$s=2$		$s=4$
attributes	bubble	ring	bubble	ring	bubble	ring
all eight	0.928	0.928	0.744	0.744	0.729	0.729
$x,y$	0.288	0.288	0.519	0.519	0.521	0.521
$x,y,\mathrm{date}$	0.886	0.886	0.890	0.776	0.790	0.790
$x,y,\mathrm{DBH}$	0.858	0.858	0.691	0.691	0.680	0.680
$x,y,\mathrm{hom}$	0.857	0.857	0.745	0.745	0.702	0.702
$x,y,\mathrm{latin}$	0.437	0.437	0.547	0.547	0.573	0.573
$x,y,\mathrm{stemtag}$	0.732	0.732	0.738	0.739	0.734	0.736
$x,y,\mathrm{tag}$	0.693	0.693	0.698	0.698	0.678	0.678

Table 3. Table 3: Plausibility of assuming a log-normal distribution for x , y 𝑥 𝑦 x,y , and comparison with the power law distribution.

log-normal	log-normal vs. power law
$p$-value	log-likelihood ratio	$p$-value
0.77	0.84	0.20

Equations

$$x\mapsto x+(x▹1)$$

$$\bullet$$

$$(a_1,\dots ,a_k)$$

$$\left(\begin{array}{cccccccc}n-1& n-2& \dots & d& d-1& \dots & 1& 0\\ d& n-1& \dots & d+1& d-1& \dots & 1& 0\end{array}\right)$$

$$\left(\begin{array}{cccccc}n-1& n-2& \dots & n-(d+1)& \dots & 0\\ d& d-1& \dots & 0& \dots & d+1\end{array}\right)$$

$$x$$

$$y$$

$$\mathrm{date}$$

$$\mathrm{DBH}$$

$$\mathrm{hom}$$

$$\mathrm{latin}$$

$$\mathrm{stemtag}$$

$$\mathrm{tag}$$

$$\Omega (T,S)$$

$$\omega (T,S)$$

$$T_{\mathrm{static}}$$

$$k=\lceil \frac{{\log }_2\frac{\left|S\right|}{s}}{n}\rceil$$

$$R(S)=\frac{\Omega (T_{\mathrm{scaled}},S)}{\Omega (T_{\mathrm{static}},S)}$$

$$(2s{)}^{\rho (S,s)}=\frac{\left|S\right|}{\left|L(T_{\mathrm{scaled}})\right|}\left(1+\omega (T_{\mathrm{static}})\right)$$

Full text

1 Introduction

Big spatial data requires efficient methods for their processing. Not only is the amount of geographical data increasing, but also their dimensionality, as other geographical attributes found in e.g. forest cadastres, 3D City models or infrastructure models etc. become relevant. As the number of points becomes large and lives inside a multi-dimensional space, efficient access is becoming a major issue for big spatial data. In order to handle such issues, space-filling curves have turned out to be a suitable indexing method, as nearby points on the curve are nearby in the space. Hilbert curves are a special case with particularly good self-similarity properties which made them popular in the domain of computer science. With geo-spatial data, they also have been successfully used. However, only static space-filling curves have been used so far, and these reportedly cause storage problems as dimension increases. In [2], a scaled index based on a $p$-adic generalisation of higher-dimensional Hilbert curves was developed. The number of iterations there differs locally and depends on the local density of the point cloud. This index is applied in the present article to rain forest data consisting of 2.5 million data points in a multi-dimensional space, and is also compared with the traditional static Hilbert curve index.

The contributions of this article are

• •

the implementation of a data-driven index structure based on the index inherent in space-filling curves themselves

• •

to exploit the inherent tree-structure of space-filling curves for $nD$ point clouds

• •

an index which can handle different kinds of distributions through its scalability

• •

reduced storage by not storing space-filling curve keys

• •

a study of the influence of point cloud distributions on the tree structure of the index

The following section gives an overview of the state of the art of space-filling curve indices. That section is followed by a presentation of our methodology, in which the scaled Hilbert curve index is explained. Section 4 contains a description of the rain forest tree data set. The article concludes with a section presenting the experimental results.

2 State of the art

Motivated by Cantor’s counter-intuitive result which says that the unit interval and Euclidean $n$-dimensional space have the same cardinality, Peano proved in the year 1890 that there exists a continuous mapping of the unit interval onto the unit square. This means the discovery of a curve passing through all points of the unit square [16]. A year later, Hilbert published a variation of Peano’s construction by using a binary subdivision in order to approximate the curve [10]. This Hilbert curve is now widely used in computer science.

The above are examples of so-called space-filling curves which can be generalised to higher dimension. Analogues of Hilbert curves for higher dimension use the binary reflected Gray code which is a special re-ordering of the binary numbers [8, 9]. In [2], a systematic approach to $n$-dimensional Hilbert curves and to their $p$-adic generalisations is undertaken, and thus a scaled index for data taken from a space of high dimension is obtained, and tested for dimension up to 431.

In [15], the performance of various different space-filling curves with respect to different criteria is studied. Due to the continuity of the space-filling curve, nearby points on the curve correspond to nearby points in the space indexed by the curve. This makes space-filling curves interesting for spatial information. Applications in this domain include city models in 3D, where it has been observed that they allow for a more efficient information retrieval than the CityGML format [19]. The management of space-time point clouds using space-filling curves has been studied in [17], where a B-tree index is imposed onto space-filling curve keys of space-time data points, which is a 2-step approach. A similar two-step approach using an $n$-dimensional space-filling curve library was proposed and tested for point clouds in 4D using the Oracle Index Organized Table instead of the B-tree [7]. In geo-informatics, already a Hilbert curve index for data in 6D space has been observed as computationally challenging [20].

3 Methodology

The binary reflected Gray code is an ordering of the binary numbers in such a way that subsequent numbers differ only by one bit [6]. According to [14], it can be represented by the map

[TABLE]

on binary numbers, where $x\triangleright k$ means the right-shift of $x$ by $k$ places, and $+$ is addition modulo $2$ or, equivalently, the logical ${\rm XOR}$-operator.

If $I^{n}$ is the $n$-dimensional hypercube, then an $n$-digit binary number represents a subhypercube of $I^{n}$ obtained by applying a subdivision into two equal parts to each coordinate, and each [math] means the first, and $1$ the second half. The binary reflected Gray code then yields a traversal of all $2^{n}$ subhypercubes of $I^{n}$ by moving along neighbouring subhypercubes in such a way that the binary codes of the first and the last ones differ in precisely one coordinate. For $n=2$, this is depicted in Figure 1 (left).

Now, if $I^{n}$ is iteratively further subdivided, then there exist for each iteration $k$ transformations of the binary reflected Gray code for each subhypercube such that the union of all these transformed traversals form a curve which passes through all the $2^{kn}$ subhypercubes, i.e. the binary codes of any subsequent subhypercubes differ in precisely one coordinate. Such a curve is a so-called Hilbert curve, or Gray-Hilbert curve, as it is called in [2].

In the article above, only affine transformations are allowed. It turns out that these are given by a permutation of coordinates followed by a translation, so that there are $(n-1)!$ possible transformed Gray codes after fixing the start point and the end point, which are neighbours [2]. As each start and end point is determined by the previous iteration, we see from this that the $2$-dimensional Gray-Hilbert curve is uniquely determined by the Gray code and coincides with the well-known Hilbert curve whose second iteration is depicted in Figure 1 (right).

A point in the $k$-th iteration of a Gray-Hilbert curve can be represented by a sequence of $n$-digit binary numbers

[TABLE]

of length $k$, where each $a_{i}$ signifies the (transformed) Gray code value corresponding to a hypercube at iteration $i$. The binary number $a_{i}$ itself corresponds to an assignment of values [math] or $1$ to the coordinates in some specified ordering, depending on the transformation. Thus we can represent the hypercubes at level $k$ by a path from root to a leaf in a binary tree with $2^{nk}$ leaves. This tree is obtained by iteratively gluing the roots of copies of a binary tree with $n$ levels to the leaves of a tree starting with a root vertex. The gluing corresponds to the hypercube subdivision. The levels of each of the tree pieces are each labelled by a coordinate which is given by the local ordering of coordinates which is determined by the transformation used for building up the Gray-Hilbert curve at this particular hypercube subdivision. By taking an infinite iteration process, we call the resulting locally finite rooted tree the Gray-Hilbert tree in dimension $n$. Figure 2 shows a part of the Gray-Hilbert tree of dimension two, namely the tree obtained by performing the first two iterations.

A path in the Gray-Hilbert tree from root to a given vertex corresponds to a sequence of subdivisions of certain coordinates which yields a subcuboid of $I^{n}$ whose edges are one of two kinds: either short or long. The blue path in Figure 2 thus corresponds to the blue rectangle in Figure 1 (right).

Let $S\subset I^{n}$ be a finite point cloud. The Gray-Hilbert tree generated by $S$ is defined as being the smallest subtree of the Gray-Hilbert tree in dimension $n$ having the same root, and also whose leaves correspond to subcuboids of $I^{n}$ each containing precisely one element of $S$. In [2], an efficient algorithm is given for computing the Gray-Hilbert tree generated by any finite point cloud in $I^{n}$, which uses certain types of coordinate permutations. The linear ordering imposed on $S$ by identifying $S$ with the leaves of the Gray-Hilbert tree generated by $S$ is called the scaled Gray-Hilbert curve index for $S$.

In contrast to that, the traditional static Gray-Hilbert curve index is the linear ordering of a finite point cloud $S$ given by an iteration of a Gray-Hilbert curve which uses a subdivision of $I^{n}$ which separates all points of $S$.

In this article, we consider the two different Gray-Hilbert curve indices determined by the following permutations. The first one, called bubble is defined in the Gray-Hilbert tree at the corresponding local root vertex of the following $n$-level binary tree by the permutation

[TABLE]

where $d$ is the coordinate number coming in the first level after that particular local root vertex. The second type is called ring and is given by the permutation

[TABLE]

at each entry point into a new copy of the $n$-level binary tree.

4 The data set

The data set represents tree-specific information taken from ca. 938,000 trees in Barro Colorado Island (Panama) in a series of 7 censuses, beginning in 1982, in an area of 50 hectares [11, 5, 12]. There are $18$ attributes, from which are chosen for the indexing methods $8$ attributes which seem interesting from a geo-information point of view. These are namely the following:

[TABLE]

The non-numerical attribute $\operatorname{latin}$ is coded by assigning to each species a random value in the interval $[0,1]$.

Thus the data set used here consists of a point cloud of 2.45 million data points in an $8$-dimensional space, 368,000 of which have missing values. Whereas in [18], sphere packings were used, it is the aim of this article to compare the performance of space-filling curve indices with respect to the influence of certain coordinates on the resulting tree structures.

5 Results

The implementation of the index methods is based on a tree structure and is now part of the geo-spatio-temporal database architecture DB4GeO [3] which has been redesigned in order to be able to use OLTP and OLAP for big data issues, using property graph database methods [13]. The corresponding algorithm is presented in [2]. Tree structures are very well suited for modelling the scaled Gray-Hilbert curve and can also be applied to the static version. The length of the path from root to leaf node equals the number of bits used to address the sub-hypercuboid which corresponds to the leaf node. Traversing the tree in pre-order defines a linear ordering of leaf nodes which corresponds to the linear ordering of the hypercuboids given by the Gray-Hilbert curve.

Given some multi-dimensional point cloud, the leaf nodes of the Gray-Hilbert tree, or any sub-tree of it, are viewed as buckets which are respectively empty, filled, underfilled, or overfilled, depending on if the corresponding hypercuboid contains no, a specified number, less than, or more than a pre-specified number of data points. This specified parameter is denoted here as $s$, and is called the bucket capacity.

Following [2], we define the quantities

[TABLE]

where $T$ is a sub-tree of the Gray-Hilbert tree and $S$ is a point cloud.

The quantity $\Omega(T,S)$ is a measure which takes into account the capacity of $T$, which is the number of leaf nodes, and a factor $1+\omega(T,S)$ where $\omega(T,S)$ is the number of leaf node splittings which would occur for $S$, if the depth of the tree $T$ were not bounded. We call $\Omega(T,S)$ the capacity of $T$ for $S$.

We will write $T_{\operatorname{scaled}}$ for the Gray-Hilbert tree generated by $S$. The tree

[TABLE]

is defined as the finite Gray-Hilbert tree with maximal iteration number

[TABLE]

as in [2]. Here, $\left\lvert S\right\rvert$ means the number of elements of $S$, and $s$ is the bucket capacity. Choosing this value of $k$ yields an optimal static Gray-Hilbert curve index with the property that, on average, no leaf node is overfilled.

Figures 3 and 4 show the Gray-Hilbert tree for the projection onto the eight chosen attributes, and onto $x,y\operatorname{latin}$, respectively. In Figure 3, $T_{\operatorname{static}}$ is shown in black, whereas the tree $T_{\operatorname{scaled}}$ is coloured. The size of the green glyphs represents the number of objects inside a leaf node of $T_{\operatorname{static}}$. In Figure 4, $T_{\operatorname{static}}$ is shown on the left, and $T_{\operatorname{scaled}}$ on the right. Colours blue to green represent the levels, and colours yellow to red as well as the size of the glyphs represent the number of objects inside a leaf node. The levels are represented by colours.

The iteration numbers of $T_{\operatorname{static}}$ are $k=3$ for all 8 attributes, and $k=5$ for $x,y,\operatorname{latin}$.

In order to compare the scaled and the static Gray-Hilbert curve indices, we consider the Gray-Hilbert capacity ratio for $S$

[TABLE]

defined in [2]. If $R(S)<1$, then the scaled index is more storage efficient than the optimal static index, otherwise not.

In [2], it was mathematically proven that $R(S)$ is indeed never greater than $1$ if $n$ is sufficiently large, and also tends to zero for increasing dimension $n$. In Table 1, it is confirmed that for the projection of the data onto all eight chosen attributes, the Gray-Hilbert capacity ratio is indeed less than one, also for quite large values of the bucket capacity $s$ (although for $s=2^{14}$ it does get close to $1$).

The Gray-Hilbert local sparsity measure $\rho(S,s)$ is defined via the relationship

[TABLE]

where $\left\lvert L(T_{\operatorname{scaled}})\right\rvert$ is the number of leaf nodes of $T_{\operatorname{scaled}}$. This measure $\rho(S,s)$ provably takes values between [math] and $1$ [2]. If $\rho(S,s)$ is near zero, then $S$ is more likely to be uniformly distributed, whereas $\rho(S,s)$ near one indicates that this should not be the case.

Table 2 gives the values of $\rho(S,s)$ for projections of the data set onto different attribute sets: first $x,y$, then $x,y$ and a third attribute. The extra attribute leading to the closest value of $\rho$ for the attribute set $x,y$ is $\operatorname{latin}$. Figure 5 shows the tail distributions of $x,y$ and $x,y$ plus an additional attribute. It can be seen that adding $\operatorname{latin}$ to $x,y$ does not change the tail distribution, as could be expected from the $\rho$ values. All other such attribute sets, with the exception of $x,y,\operatorname{tag}$, yield tail distributions which are different from that of $x,y$. This means that $\rho(S,s)$ is able to distinguish between point clouds with different tail distributions, and the example of $x,y,\operatorname{latin}$ and $x,y,\operatorname{tag}$ shows that $\rho(S,s)$ can distinguish more point clouds than the tail distribution. A double logarithmic plot of the tail distribution reveals whether a uniform distribution is likely or not, because the uniform distribution yields a plot displaying a longer ‘survival time’ at probability $1$, followed by a sharp drop to [math]. As they are almost straight lines, a heavy-tailed distribution like a power law or a log-normal distribution might be likely.

The framework in [4] shows how to perform a goodness-of-fit test of heavy-tailed distributions together with a method for directly comparing two alternative distributions. In [1] a brief overview of these methods is given. Table 3 shows the results of a test for a log-normal distribution for the $x,y$ attribute set and a comparison with the power law distribution. As the $p$-value for the first test is high, a log-normal distribution cannot be ruled out. The log-likelihood ratio being positive shows that the log-normal distribution is preferred over the power law, and as the corresponding $p$-value is low this time, statistical fluctuations are not very likely to be the cause of this test result. This strengthens the hypothesis that the underlying distribution may be log-normal.

6 Conclusions

The distribution of a point cloud has a strong influence on the structure of an optimal tree-index for accessing it. When using Hilbert curves or their higher-dimensional generalisations which are called Gray-Hilbert curves (as they use the Gray code), then the optimal tree-index is given by the smallest sub-tree of the Gray-Hilbert tree whose leaves are the data points. It can be seen from the results that this scaled Gray-Hilbert curve index has the potential of saving a lot of superfluous nodes which are inevitable in the static Gray-Hilbert curve index, especially in higher dimension. This means that a large amount of the space-filling curve keys can be economised with this index. The Gray-Hilbert local sparsity measure is able to distinguish point clouds with differing tail distributions, and moreover can distinguish more than the tail distribution function. It would be interesting to find out how the distinguishing properties of data sets can be expressed by other means.

Acknowledgements

The authors of this work intend to acknowledge something in the published version of this article. They would also like to express their thanks to the Center for Tropical Forest Science of the Smithsonian Tropical Research Institute for providing the data set.