Maximum rectilinear convex subsets

TL;DR

This paper introduces efficient algorithms for finding subsets of points with maximum rectilinear convex properties, including boundary points, interior emptiness, and area, with applications to related polygon problems.

Contribution

It presents new algorithms with $O(n^3)$ time and $O(n^2)$ space for various maximum rectilinear convex subset problems, improving understanding and methods in computational geometry.

Findings

Algorithms for maximum boundary points in rectilinear convex hulls

Algorithms for maximum area rectilinear convex hulls with empty interior

Simplified algorithms for maximum-area orthoconvex and staircase polygons

Abstract

Let $P$ be a set of $n$ points in the plane. We consider a variation of the classical Erd\H{o}s-Szekeres problem, presenting efficient algorithms with $O(n^3)$ running time and $O(n^2)$ space complexity that compute: (1) A subset $S$ of $P$ such that the boundary of the rectilinear convex hull of $S$ has the maximum number of points from $P$, (2) a subset $S$ of $P$ such that the boundary of the rectilinear convex hull of $S$ has the maximum number of points from $P$ and its interior contains no element of $P$, (3) a subset $S$ of $P$ such that the rectilinear convex hull of $S$ has maximum area and its interior contains no element of $P$, and (4) when each point of $P$ is assigned a weight, positive or negative, a subset $S$ of $P$ that maximizes the total weight of the points in the rectilinear convex hull of $S$. We also revisit the problems of computing a maximum-area orthoconvex polygon and computing a maximum-area staircase polygon, amidst a point set in a rectangular domain. We obtain new and simpler algorithms to solve both problems with the same complexity as in the state of the art.

Figures

Equations

$$RCH(P)={\mathbb{R}}^2-\bigcup _{\text{Q is P-free}}Q.$$

$$M_i(P)=\bigcup _{p\in P}\overline{Q_i(p)},$$

$$RCH(P)=\bigcap _{i=1,2,3,4}{M}_i(P),$$

$$C_{p,q}~{}=~{}\left\{\begin{array}[]{ll}1&\mathbf{if~{}}p=q\\ \max\{1+C_{r,q}\}~{}\text{over~{}all}~{}r\in P(p,q)\cup\{q\}&\mathbf{if~{}}p\neq q.\\ \end{array}\right.$$

$$T_{p,q}~{}=~{}\max\left\{\begin{array}[]{lr}C_{p,q}&\textbf{\small(A)}\\ 1+T_{r,q}~{}\text{over~{}all}~{}r\in R_{p\setminus q}&\textbf{\small(B)}\\ 1+T_{p,r}~{}\text{over~{}all}~{}r\in R_{q\setminus p}&\textbf{\small(C)}\\ \alpha_{p,q}+T_{r,r}~{}\text{over~{}all}~{}r\in R_{p,q}&~{}~{}\textbf{\small(D)}\\ \alpha_{p,q}+U_{p,r}~{}\text{over~{}all}~{}r\in R_{p,q}&\textbf{\small(E)}\\ \end{array}\right.$$

$$U_{p,r}=\max \{T_{r,s}\}\text{over all}s\in R_{p\setminus r}^{\prime }.$$

$$|RCH(S_0)|=\max \{S_{p,s}\}\text{over all}p,s\in P\text{such that}{p}_x<s_x$$

$$S_{p,s}=\max \{T_{p,q}^{\prime }+T_{r,s}\}\text{over all}q,r\in P\text{such that (p,q,r,s) is a 4-separator}.$$

$$C_{p,q}~{}=~{}\left\{\begin{array}[]{ll}1&\mathbf{if~{}}p=q\\ \max\{1+C_{r,q}\}~{}\text{over~{}all}~{}r\in P(p,q)\cup\{q\}~{}\text{such~{}that}~{}P(r,u)=\emptyset&\mathbf{if~{}}p\neq q.\\ \end{array}\right.$$

$$T_{p,q}~{}=~{}\max\left\{\begin{array}[]{lr}C_{p,q}&\textbf{\small(A)}\\ 1+T_{r,q}~{}\text{over~{}all}~{}r\in R_{p\setminus q}~{}:~{}P\cap E(p,q,r)=\emptyset&\textbf{\small(B)}\\ 1+T_{p,r}~{}\text{over~{}all}~{}r\in R_{q\setminus p}~{}:~{}P\cap E(p,q,r)=\emptyset&\textbf{\small(C)}\\ \alpha_{p,q}+T_{r,r}~{}\text{over~{}all}~{}r\in R_{p,q}~{}:~{}P\cap E(p,q,r)=\emptyset&~{}~{}\textbf{\small(D)}\\ \alpha_{p,q}+U_{p,r}~{}\text{over~{}all}~{}r\in R_{p,q}~{}:~{}P\cap E(p,q,r)=\emptyset&\textbf{\small(E)}\\ \end{array}\right.$$

$$U_{p,r}=\max \{T_{r,s}\}\text{over all}s\in R_{p\setminus r}^{\prime }.$$

$$|RCH(S_0)|=\max \{S_{p,s}\}\text{over all}p,s\in P\text{with}{p}_x<s_x$$

$$S_{p,s}~{}=~{}\left\{\begin{array}[]{lr}\max\{T^{\prime}_{p,q}+T_{r,s}\}~{}\text{: $(p,q,r,s)$ is a $4$-separator}&p\prec^{\prime}s\\ \max\{T^{\prime}_{p,q}+T_{r,s}\}~{}\text{: $(p,q,r,s)$ is a $4$-separator, $P(u,v)=\emptyset$}&p\prec s.\\ \end{array}\right.$$

$$C_{p,q}~{}=~{}\left\{\begin{array}[]{ll}0&\mathbf{if~{}}p=q\\ \max_{r\in P(p,q)\cup\{q\}}\{\mathsf{Area}(B(r,u))+C_{r,q}\}~{}:~{}P(r,u)=\emptyset&\mathbf{if~{}}p\neq q.\\ \end{array}\right.$$

$$T_{p,q}~{}=~{}\max\left\{\begin{array}[]{lr}C_{p,q}&\textbf{\small(A)}\\ \mathsf{Area}(E(p,q,r))+T_{r,q}~{}\text{over~{}all}~{}r\in R_{p\setminus q}~{}:~{}P\cap E(p,q,r)=\emptyset&\textbf{\small(B)}\\ \mathsf{Area}(E(p,q,r))+T_{p,r}~{}\text{over~{}all}~{}r\in R_{q\setminus p}~{}:~{}P\cap E(p,q,r)=\emptyset&\textbf{\small(C)}\\ \mathsf{Area}(E(p,q,r))+T_{r,r}~{}\text{over~{}all}~{}r\in R_{p,q}~{}:~{}P\cap E(p,q,r)=\emptyset&~{}~{}\textbf{\small(D)}\\ \mathsf{Area}(E(p,q,r))+U_{p,r}~{}\text{over~{}all}~{}r\in R_{p,q}~{}:~{}P\cap E(p,q,r)=\emptyset&\textbf{\small(E)}\\ \end{array}\right.$$

$$U_{p,r}=\max \{T_{r,s}\}\text{over all}s\in R_{p\setminus r}^{\prime }.$$

$$|RCH(S_0)|=\max \{S_{p,s}\}\text{over all}p,s\in P\text{with}{p}_x<s_x,$$

$$S_{p,s}~{}=~{}\left\{\begin{array}[]{lr}\max_{\{\text{$4$-separators $(p,q,r,s)$\}}}\{T^{\prime}_{p,q}+T_{r,s}\}{}&p\prec^{\prime}s\\ \max_{\{\text{$4$-separators $(p,q,r,s)$\}}}\{U(p,q,s)+V(p,r,s))\}~{}\text{: $P(u(r),v(q))=\emptyset$}&p\prec s.\\ \end{array}\right.$$

$$C_{p,q}~{}=~{}\left\{\begin{array}[]{ll}w(p)&\mathbf{if~{}}p=q\\ w(p)+\max_{r\in P(p,q)\cup\{q\}}\left\{\mathsf{Weight}(B(r,u))+C_{r,q}\right\}&\mathbf{if~{}}p\neq q.\\ \end{array}\right.$$

$$T_{p,q}~{}=~{}\max\left\{\begin{array}[]{lr}C_{p,q}&\textbf{\small(A)}\\ w(p)+\mathsf{Weight}(E(p,q,r))+T_{r,q}~{}\text{over~{}all}~{}r\in R_{p\setminus q}&\textbf{\small(B)}\\ w(q)+\mathsf{Weight}(E(p,q,r))+T_{p,r}~{}\text{over~{}all}~{}r\in R_{q\setminus p}&\textbf{\small(C)}\\ \alpha_{p,q}+\mathsf{Weight}(E(p,q,r))+T_{r,r}~{}\text{over~{}all}~{}r\in R_{p,q}&~{}~{}\textbf{\small(D)}\\ \alpha_{p,q}+\mathsf{Weight}(E(p,q,r))+U_{p,r}~{}\text{over~{}all}~{}r\in R_{p,q}&\textbf{\small(E)}\\ \end{array}\right.$$

$$U_{p,r}=\max \{T_{r,s}\}\text{over all}s\in R_{p\setminus r}^{\prime }.$$

$$|RCH(S_0)|=\max \{S_{p,s}\}\text{over all}p,s\in P\text{such that}{p}_x<s_x$$

$$S_{p,s}~{}=~{}\left\{\begin{array}[]{lr}\max_{\{\text{$4$-separators $(p,q,r,s)$\}}}\{T^{\prime}_{p,q}+T_{r,s}\}{}&p\prec^{\prime}s\\ \max_{\{\text{$4$-separators $(p,q,r,s)$\}}}\{U(p,q,s)+V(p,r,s))\}&p\prec s.\\ \end{array}\right.$$

Full text

††

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 734922.

Keywords: Erdős-Szekeres problems, convex subsets, optimization, orthoconvexity, rectilinear convex hull.

1 Introduction

Let $P$ be a point set in general position in the plane. A subset $S$ of $P$ with $k$ elements is called a convex $k$-gon if the elements of $S$ are the vertices of a convex polygon, and it is called a convex $k$-hole of $P$ if the interior of the convex hull of $S$ contains no element of $P$. The study of convex $k$-gons and convex $k$-holes of point sets started in a seminal paper by Erdős and Szekeres [9] in 1935. Since then, numerous papers about both the combinatorial and the algorithmic aspects of convex $k$-gons and convex $k$-holes have been published. The reader can consult the two survey papers about so-called Erdős-Szekeres type problems [7, 11].

There are recent papers studying the existence and number of convex $k$-gons and convex $k$-holes for finite point sets in the plane [1, 2, 3]. Papers dealing with the algorithmic complexity of finding largest convex $k$-gons and convex $k$-holes are, respectively, Chvátal and Kincsek [8] and Avis and Rappaport [5], which solve these problems in $O(n^{3})$ time.

Erdős-Szekeres type problems have also been studied for colored point sets. Let $P$ be a point set such that each of its elements is assigned a color, say red or blue. Bautista-Santiago et al. [6] studied the problem of finding a monochromatic subset $S$ of $P$ of maximum cardinality such that all of the elements of $P$ contained in the convex hull of $S$ have the same color. As a generalization, they also studied the problem in which each element of $P$ has assigned a (positive or negative) weight. In this case, the goal is to find a subset $S$ of $P$ that maximizes the total weight of the points of $P$ contained in the convex hull of $S$. Each of these problems was solved in $O(n^{3})$ time and $O(n^{2})$ space. Further, their algorithm can easily be adapted to find a subset $S$ of $P$ such that the convex hull of $S$ is empty and of maximum area in $O(n^{3})$ time and $O(n^{2})$ space.

In this paper, we study Erdős-Szekeres type problems under a variation of convexity known as rectilinear convexity, or orthoconvexity: Let $P=\{p_{1},\ldots,p_{n}\}$ be a set of $n$ points in the plane in general position. A quadrant of the plane is the intersection of two open half-planes whose supporting lines are parallel to the $x$- and $y$-axes, respectively. We say that a quadrant $Q$ is $P$-free if it does not contain any point of $P$. The rectilinear convex hull of $P$, denoted as $RCH(P)$, initially defined by Ottmann et al. [12], is defined as:

[TABLE]

The rectilinear convex hull of a point set might be a simply connected set, yielding an intuitive and appealing structure (see Figure 1a). However, in other cases the rectilinear convex hull can have several connected components (see Figure 1b), some of which might be single points which we call pinched points. The size of $RCH(P)$ is the number of elements of $P$ on the boundary of $RCH(P)$. The sizes of the rectilinear convex hulls in Figures 1a and 1b are, respectively, thirteen and twelve.

Alegría-Galicia et al. [4] gave an optimal $\Theta(n\log n)$-time and $O(n)$-space algorithm to compute the orientation of the coordinate axes such that the rectilinear convex hull of a set $P$ of $n$ points in the plane has minimum area. The reader can refer to the literature for other results related to rectilinear convexity [4, 10, 14]. In this paper, we present efficient algorithms for the following geometric optimization problems:

MaxRCH: Given a set $P$ of $n$ points in the plane, find a subset $S\subseteq P$ such that the size of $RCH(S)$ is maximized.

MaxEmptyRCH: Given a set $P$ of $n$ points in the plane, find a subset $S\subseteq P$ such that the interior of $RCH(S)$ contains no point of $P$ and the size of $RCH(S)$ is maximized.

MaxAreaRCH: Given a set $P$ of $n$ points in the plane, find a subset $S\subseteq P$ such that the interior of $RCH(S)$ contains no point of $P$ and the area of $RCH(S)$ is maximized.

MaxWeightRCH: Given a set $P$ of $n$ points in the plane, such that each $p\in P$ is assigned a (positive or negative) weight $w(p)$, find a subset $S\subseteq P$ that maximizes $\sum_{p\in P\cap RCH(S)}w(p)$.

In Section 3, we give an $O(n^{3})$-time $O(n^{2})$-space algorithm to solve the MaxRCH problem. Then, in Section 4 we show how to adapt this algorithm to solve the other three problems, each in $O(n^{3})$ time and $O(n^{2})$ space. The complexities of our algorithms are the same as the complexities of the best-known algorithms to solve these problems with the usual definition of convexity.

2 Some notation and definitions

For the sake of simplicity, we assume that all point sets $P$ considered in this paper are in general position, which means that no two points of $P$ share the same $x$- or $y$-coordinate. Using a $O(n\log n)$-time preprocessing step, we can also assume when necessary that the points of a point set $P$ are ordered by $x$-coordinate or $y$-coordinate. Given a point set $P$ in the plane, we will use $a$, $b$, $c$, and $d$ to denote the leftmost, bottommost, rightmost, and topmost points of $P$, respectively, unless otherwise stated. Note that $a$, $b$, $c$, and $d$ are not necessarily different. In Figure 1b, we have $a=d$.

Given a point $p$ of the plane, let $p_{x}$ and $p_{y}$ denote the $x$- and $y$-coordinates of $p$, respectively. For $p,q\in\mathbb{R}^{2}$, $p\neq q$, we write $p\prec q$ to denote that $p_{x}<q_{x}$ and $p_{y}<q_{y}$, and $p\prec^{\prime}q$ to denote that $p_{x}<q_{x}$ and $p_{y}>q_{y}$. Let $p,q\in\mathbb{R}^{2}$, and consider a set $S=\{v_{1},\ldots,v_{k}\}$ of $k$ points such that $v_{1}=p$, $v_{k}=q$, and $v_{i}\prec^{\prime}v_{i+1}$ for $i=1,2,\dots,k-1$. A $1$-staircase joining $p$ to $q$ is an orthogonal polygonal chain, such that two consecutive elements of $S$ are joined by an elbow consisting of a horizontal segment followed by a vertical segment. For an illustration, see Figure 2a. A $3$-staircase joining $p$ to $q$ is defined in a similar way, but using elbows whose first segment is vertical. Analogously, we define $2$- and $4$-staircases, except that we require $v_{i}\prec v_{i+1}$. The first segment is vertical in the $2$-staircase and horizontal in the $4$-staircase. Points of $S$ are called the vertices of the staircase.

Any point $p$ in the plane defines four open axis-aligned quadrants $Q_{i}(p)$, $i=1,2,3,4$, as follows (see Figure 2b): $Q_{1}(p)~{}=~{}\{q\in\mathbb{R}^{2}\mid p\prec q\}$, $Q_{2}(p)~{}=~{}\{q\in\mathbb{R}^{2}\mid q\prec^{\prime}p\}$, $Q_{3}(p)~{}=~{}\{q\in\mathbb{R}^{2}\mid q\prec p\}$, and $Q_{4}(p)~{}=~{}\{q\in\mathbb{R}^{2}\mid p\prec^{\prime}q\}$. Given a point set $P$ in the plane, for $i=1,2,3,4$, let

[TABLE]

where $\overline{Q_{i}(p)}$ denotes the closure of $Q_{i}(p)$. The elements of $P$ that belong to the boundary of $M_{i}(P)$, are called the (rectilinear) $i$-extremal points of $P$ (see Figure 2c). Note that the $i$-extremal points of $P$ are the vertices of a $i$-staircase connecting all of them. This $i$-staircase, that we call the $i$-staircase associated with $P$, is the part of the boundary of $M_{i}(P)$ that connects all the $i$-extremal points of $P$ (see Figure 2c). For every $J\subseteq\{1,2,3,4\}$, we say that $p\in P$ is $J$-extremal if $p$ is $j$-extremal for every $j\in J$. The rectilinear convex hull of $P$ is the set111The notation $\mathcal{RH}(P)$ is also used for the rectilinear convex hull [4].

[TABLE]

see Figure 1. The boundary of $RCH(P)$ is (a part of) the union of the $1$-, $2$-, $3$- and $4$-staircases associated with $P$. Observe that the endpoints of these four staircases are $a,b,c$ and $d$, $a$ is $\{1,4\}$-extremal, $b$ is $\{1,2\}$-extremal, $c$ is $\{2,3\}$-extremal, and $d$ is $\{3,4\}$-extremal. In Figure 1b, as $a=d$, then $a$ is $\{1,3,4\}$-extremal and the $4$-staircase associated with $P$ consists of only point $a$.

Observe that $RCH(P)$ is disconnected when either the intersection of the complements $\mathbb{R}^{2}\setminus M_{1}(P)$ and $\mathbb{R}^{2}\setminus M_{3}(P)$ is not empty, as shown in Figure 1b, or the intersection of the complements $\mathbb{R}^{2}\setminus M_{2}(P)$ and $\mathbb{R}^{2}\setminus M_{4}(P)$ is not empty. In other words, when either the $1$- and $3$-staircases associated with $P$ cross or the $2$- and $4$-staircases associated with $P$ cross.

Definition 2.1**.**

A pinched point $u$ of $RCH(P)$ occurs when $u$ is either $\{1,3\}$-extremal, as shown in Figure 1b, or $\{2,4\}$-extremal.

Definition 2.2**.**

The size of $RCH(P)$ is the number of points of $P$ which are $i$-extremal for at least one $i\in\{1,2,3,4\}$.

From the definition of the staircases for $P$, the following observation is straightforward.

Observation 2.3**.**

Assume that the concatenation of the four $i$-staircases associated with $P$ is traversed counter-clockwise. For two consecutive $i$-extremal points $p$ and $p^{\prime}$, $Q_{i+2}(o)$ contains no element of $P$, where $i+2$ is taken modulo $4$ and $o=(p^{\prime}_{x},p_{y})$ for $i=1,3$ or $o=(p_{x},p^{\prime}_{y})$ for $i=2,4$.

Given two points $u\neq v$ in the plane, let $B(u,v)$ denote the smallest open axis-aligned rectangle containing $u$ and $v$, and let $P(u,v)=P\cap B(u,v)$. Note that $u$ and $v$ are two opposed corners of $B(u,v)$. If $u=v$, then we define $B(u,u)$ as point $u$.

Definition 2.4**.**

We say that $RCH(P)$ is vertically separable if rectangles $B(a,d)$ and $B(b,c)$ are separated by a vertical line. The two examples shown in Figure 1 are vertically separable.

Given a point set $S$, and a horizontal line $\ell$, let $S^{\prime}$ be the image of $S$ under a reflection around $\ell$. The following lemma is key for our algorithms:

Lemma 2.5**.**

For all $S\subseteq P$, $|S|\geq 2$, either $RCH(S)$ or $RCH(S^{\prime})$ is vertically separable.

Proof.

Note that $d_{x}<b_{x}$ is necessary and sufficient for vertical separability of $B(a,d)$ and $B(b,c)$, that is, $RCH(S)$ is vertically separable. Suppose then that $b_{x}<d_{x}$, and let $\ell$ be a horizontal line. It is straightforward to see that, if we reflect the point set $S$ around $\ell$, then $S$ becomes $S^{\prime}$ and we have that $RCH(S^{\prime})$ is vertically separable. ∎

We will assume in each of the problems MaxRCH, MaxEmptyRCH, MaxAreaRCH, and MaxWeightRCH that the optimal subset $S\subseteq P$ is such that $RCH(S)$ is vertically separable. To finish this section, we give one more definition. For every $p,q\in P$ such that $p\prec q$, let $R_{p\setminus q}$, $R_{q\setminus p}$, and $R_{p,q}$ be the subsets of $P$ in the regions $Q_{4}(p)\setminus\overline{Q_{4}(q)}$, $Q_{4}(q)\setminus\overline{Q_{4}(p)}$, and $Q_{4}(p)\cap Q_{4}(q)$, respectively (see Figure 3, left). Observe that if $r\in R_{p\setminus q}$ then $r\prec q$, if $r\in R_{q\setminus p}$ then $p\prec r$, and if $r\in R_{p,q}$ then $r\not\prec q$ and $p\not\prec r$. For every $p,q\in P$ such that $q\prec^{\prime}p$, we define $R^{\prime}_{p\setminus q}$, $R^{\prime}_{q\setminus p}$ and $R^{\prime}_{p,q}$ as the subsets of $P$ in the regions $Q_{4}(q)\cap Q_{3}(p)$, $Q_{4}(q)\cap Q_{1}(p)$ and $Q_{4}(p)$, respectively.

3 Rectilinear convex hull of maximum size

In this section, we solve the MaxRCH problem. Given $P$, our goal is to combine four staircases in order to obtain a subset $S$ of $P$ whose rectilinear convex hull is of maximum size. All of this has to be done carefully, since the occurrence of pinched points may lead to overcounting.

Our algorithm to solve the MaxRCH problem proceeds in three steps: In the first step, we compute the $2$-staircases of maximum size for every $p,q\in P$ such that $p\prec q$. In the second step, we compute what we call a triple $1$-staircase and a triple $3$-staircase of maximum sizes (yet to be defined). In the third and last step, we show how to combine a triple $1$-staircase and a triple $3$-staircase to solve the MaxRCH problem. In this step, we will make sure that the solution thus obtained is vertically separable. Our algorithm will run in $O(n^{3})$ time and $O(n^{2})$ space. We describe now in detail the steps of our algorithm.

The first step: For every $p,q\in P$ such that $p\prec q$ or $p=q$, let $\mathcal{C}_{p,q}$ be a $2$-staircase with endpoints $p$ and $q$ of maximum size, see Figure 3, right. Let $C_{p,q}$ be the number of elements of $P$ in $\mathcal{C}_{p,q}$. Note that $C_{p,q}$ equals the maximum number of $2$-extremal points over all $S\cup\{p,q\}$ with $S\subseteq P(p,q)$. We can easily calculate $C_{p,q}$, for all $p,q\in P$ with $p\prec q$ or $p=q$, in $O(n^{3})$ time and $O(n^{2})$ space, using dynamic programming with the following recurrence:

[TABLE]

Using the elements $C_{p,q}$, it is a routine matter to determine $\mathcal{C}_{p,q}$, for any $p\prec q$.222We note that using not so trivial methods, we can calculate all of the $C_{p,q}$’s in $O(n^{2}\log n)$ time. However, this yields no improvement on the overall time complexity of our algorithms.

Definition 3.1**.**

Given a point set $S\subseteq P$, we define the triple $1$-staircase (resp., triple $3$-staircase) associated with $S$ as the concatenation of the $1$-, $2$- and $3$-staircases (resp., the $3$-, $4$- and $1$-staircases) associated with $S$.

The second step: In this step, our goal is to obtain a triple $1$-staircase and a triple $3$- staircase of maximum cardinality, starting and ending at some pairs of points of $P$. Triple staircases allow us to conveniently manage pinched points and disconnections of the rectilinear convex hull. Notice that the boundary of $M_{1}(S)\cap M_{2}(S)\cap M_{3}(S)$ (except for its two infinite rays) always belongs to the triple $1$-staircase associated with $S$.

Consider $p,q\in P$ such that $p\prec q$ or $p=q$. Let $Z(p,q)=Q_{4}(u)$, where $u=(p_{x},q_{y})$, and let $z(p,q)=Z(p,q)\cap P$. Let $\mathcal{T}_{p,q}$ be the triple $1$-staircase of maximum cardinality among all subsets $S\cup\{p,q\}$ with $S\subseteq z(p,q)$. If $S^{\prime}\subseteq z(p,q)$ is the set associated with $\mathcal{T}_{p,q}$, observe that $M_{1}(S^{\prime}\cup\{p,q\})\cap M_{2}(S^{\prime}\cup\{p,q\})\cap M_{3}(S^{\prime}\cup\{p,q\})$ may contain points in $P(p,q)$, it may be disconnected, and it may have pinched points (see Figure 4). Note that $p$ and $q$ are always the endpoints of $\mathcal{T}_{p,q}$. Let $X_{p,q}$ denote the set of extreme vertices of $\mathcal{T}_{p,q}$ (that is, the set of $1$-, $2$- and $3$-extremal points of $S^{\prime}\cup\{p,q\}$), and let $T_{p,q}$ be the cardinality of $X_{p,q}$.

We calculate all of the $T_{p,q}$’s by dynamic programming using Equations (2) and (3). We store all of the $T_{p,q}$’s in a table $T$. If $\alpha_{p,q}=1$ when $p=q$, and $\alpha_{p,q}=2$ when $p\neq q$, then:

[TABLE]

where for every pair $p,r\in P$ such that $p\prec^{\prime}r$

[TABLE]

Values $U_{p,r}$ are stored in a table $U$. The next lemma shows the correctness of this recurrence.

Lemma 3.2**.**

The recurrence (2) correctly calculates $T_{p,q}$, the size of $X_{p,q}$, in $O(n^{3})$ time and $O(n^{2})$ space.

Proof.

Let $\mathcal{T}_{p,q}$ be an optimal triple $1$-staircase for a pair of points $p,q\in P$ such that $p\prec q$ and let $S^{\prime}\subseteq z(p,q)$ be the point set associated with $\mathcal{T}_{p,q}$. In the counter-clockwise traversal of the triple $1$-staircase, let $p^{-}$ and $q^{-}$ be the elements of $P$ that follow and precede $p$ and $q$, respectively. Hence, $\mathcal{T}_{p,q}$ can be obtained as an extension of $\mathcal{T}_{p,q^{-}}$, $\mathcal{T}_{p^{-},q}$, or $\mathcal{T}_{p^{-},q^{-}}$.

If $p^{-},q^{-}\in B(p,q)$, then necessarily $\mathcal{T}_{p,q}$ is a $2$-staircase (the $1$ and $3$-staircases of $\mathcal{T}_{p,q}$ consist of only points $p$ and $q$, respectively), so $T_{p,q}=C_{p,q}$ and case (A) is used to set $T_{p,q}$. Thus, we assume in the rest of the proof that at least one of $p^{-}$ and $q^{-}$ is not in $B(p,q)$. See Figure 5 for the cases.

If $p^{-}\in R_{p\setminus q}$, then we have $p^{-}\prec q$ and $p^{-}$ is not 3-extremal in $\mathcal{T}_{p^{-},q}$. We use case (B) to find a point $r$ that plays the role of $p^{-}$ to compute the value of $T_{p,q}$. If $q^{-}\in R_{q\setminus p}$, then we have $p\prec q^{-}$ and $q^{-}$ is a point that is not 1-extremal in $\mathcal{T}_{p,q^{-}}$. We use case (C) to find a point $r$ that plays the role of $q^{-}$ to compute the value of $T_{p,q}$.

Suppose now that $p^{-}$ is in $R_{p,q}$ (a similar reasoning applies if $q^{-}$ is in $R_{p,q}$). In this case, $q^{-}$ cannot be in $B(p,q)$ and if $q^{-}\in R_{q\setminus p}$ then we use case (C). If $q^{-}\in R_{p,q}$, there are two cases to analyze: $p^{-}=q^{-}$ and $p^{-}\neq q^{-}$. If $p^{-}=q^{-}$, then we can use case (D) to find a point $r$ that plays the role of $p^{-}$ to compute the value of $T_{p,q}$. When $p^{-}\neq q^{-}$, we prove that $p^{-}\prec q^{-}$, and then case (E) can be used to find a pair of points $r$ and $s$ playing the roles of $p^{-}$ and $q^{-}$, both in $R_{p,q}$, with maximum value $T_{r,s}$. By Observation 2.3, as $p$ and $p^{-}$ are consecutive $1$-extremal points, then $Q_{3}(o)$ contains no element in $S^{\prime}$, where $o=(p^{-}_{x},p_{y})$. In particular, since $q^{-}$ is in $R_{p,q}$, this implies that $q^{-}$ cannot be in either $Q_{2}(p^{-})$ or $Q_{3}(p^{-})$. Moreover, as $q$ and $q^{-}$ are consecutive $3$-extremal points, then $Q_{1}(o^{\prime})$ contains no element in $S^{\prime}$ again by Observation 2.3, where $o^{\prime}=(q_{x},q^{-}_{y})$. As a consequence, $q^{-}$ cannot be in $Q_{4}(p^{-})$, so we conclude that $q^{-}\in Q_{1}(p^{-})$.

To compute tables $T$ and $U$, we scan the elements of $P$ from right to left. Each time an element $p\in P$ is encountered, we scan all of the $q\in P$ such that $p_{x}<q_{x}$, again from right to left. When $p\prec q$ we compute $T_{p,q}$, and when $p\prec^{\prime}q$ we compute $U_{p,q}$. Each entry of $T$ and $U$ is determined in $O(n)$ time. Thus, $U$ and $T$ can be computed in overall $O(n^{3})$ time and $O(n^{2})$ space. Cases (A) to (D) are in $O(n)$. We charge the work done in case (E) to constructing table $U$, which can be done in $O(n)$ time per entry. Thus the entire complexity is in $O(n^{3})$ time and $O(n^{2})$ space. ∎

In a totally analogous way, we can calculate triple $3$-staircases of maximum size. For $p\prec q$ or $p=q$, let $T^{\prime}_{p,q}$ be the size of the triple $3$-staircase $\mathcal{T}^{\prime}_{p,q}$ of maximum cardinality among all subsets $S^{\prime}\cup\{p,q\}$, where $S^{\prime}$ is now a subset of points of $P$ in $Q_{2}(v)$ with $v=(q_{x},p_{y})$. After rotating the coordinates by $\pi$, observe that the triple $3$-staircase $\mathcal{T}^{\prime}_{p,q}$ is the triple $1$-staircase $\mathcal{T}_{q,p}$. Thus, by symmetry with the $T_{p,q}$’s, all the $T^{\prime}_{p,q}$’s can also be calculated in $O(n^{3})$ time and $O(n^{2})$ space.

The third step: In this step, we show how to combine a triple $1$-staircase and a triple $3$-staircase to solve the MaxRCH problem. Recall that the solution must be vertically separable. Next we give the definition of a $4$-separator and then we show that it is equivalent to vertical separability.

Definition 3.3**.**

Let $S\subseteq P$ be any subset with $|S|\geq 2$. Given four (not necessarily distinct) extremal points $p,q,r,s\in S$, we say that the tuple $(p,q,r,s)$ is a $4$-separator of $RCH(S)$ if the following five conditions are satisfied: $(1)$ $p\prec q$ or $p=q$; $(2)$ $q\prec^{\prime}r$; $(3)$ $r\prec s$ or $r=s$; $(4)$ $p$ and $r$ are consecutive points in the $1$-staircase of $S$; and $(5)$ $s$ and $q$ are consecutive points in the $3$-staircase of $S$ (see Figure 6).

The next lemma shows the equivalence between vertical separability and the existence of $4$-separators.

Lemma 3.4**.**

$RCH(S)$* is vertically separable if and only if $RCH(S)$ has a $4$-separator.*

Proof.

Let us first assume that $RCH(S)$ is vertically separable, that is, $B(a,d)$ and $B(b,c)$ are separated by a vertical line. Recall that $a$, $b$, $c$, and $d$ are the leftmost, bottommost, rightmost, and topmost points of $S$, respectively. Then, we can argue the following: If $a\prec d$, then $S$ has at least one $1$-extremal point to each side of the vertical line through $d$. Otherwise, if $a=d$, then $S$ has at least one $1$-extremal point to the right side. Thus, covering both cases, let $p$ and $r$ be the two consecutive $1$-extremal points of $S$ such that $p_{x}\leq d_{x}<r_{x}$. Now, given that $d_{x}<r_{x}$, we have that: If $r$ is also $3$-extremal, thus a pinched point, then $S$ has at least one $3$-extremal point to the left side of the vertical line through $r$. Otherwise, if $r$ is not $3$-extremal, then $S$ has at least one $3$-extremal point to each side of this line. Thus, we can define $s$ and $q$ as the two consecutive $3$-extremal points of $S$ such that $q_{x}<r_{x}\leq s_{x}$. It is straightforward to see now that $(p,q,r,s)$ is a $4$-separator of $RCH(S)$. Note that, when $p=a=d$, then necessarily $q=p=a=d$.

Assume now that $(p,q,r,s)$ is a $4$-separator of $RCH(S)$. We then have that: $d\prec^{\prime}q$ or $d=q$, and $r\prec^{\prime}b$ or $r=b$. These conditions, together with $q\prec^{\prime}r$, directly imply that $B(a,d)$ and $B(b,c)$ are separated by a vertical line, thus $RCH(S)$ is vertically separable. ∎

Using $4$-separators, we show how to find an optimal solution that is vertically separable. Among all subsets $S$ of $P$ such that $RCH(S)$ is vertically separable, let $S_{0}$ be a subset of $P$ such that $RCH(S_{0})$ has maximum size. Let $(p,q,r,s)$ be a $4$-separator of $RCH(S_{0})$. The key observation is that the vertices of $\mathcal{T}^{\prime}_{p,q}\cup\mathcal{T}_{r,s}$ are the set of extremal points of $S_{0}$. Note that $\mathcal{T}^{\prime}_{p,q}\cap\mathcal{T}_{r,s}=\emptyset$ and $|RCH(S_{0})|$, the size of $RCH(S_{0})$, is $T^{\prime}_{p,q}+T_{r,s}$.

Thus, we proceed as follows: For given $p,s\in P$ such that $p_{x}<s_{x}$, let $\mathcal{S}_{p,s}$ be the rectilinear convex hull of maximum size, among all subsets $S\subseteq P$ containing $p$ and $s$ such that there exist two points $q,r\in S$ with $(p,q,r,s)$ being a $4$-separator of $RCH(S)$. Let $S_{p,s}$ be the size of $\mathcal{S}_{p,s}$. Note that $S_{p,s}=T^{\prime}_{p,q}+T_{r,s}$ for some $4$-separator $(p,q,r,s)$. Then, the following equations allow us to calculate $|RCH(S_{0})|$ in $O(n^{3})$ time and $O(n^{2})$ space, as Theorem 3.5 proves:

[TABLE]

where for each pair of points $p,s\in P$ with $p_{x}<s_{x}$

[TABLE]

Theorem 3.5**.**

The MaxRCH problem can be solved in $O(n^{3})$ time and $O(n^{2})$ space.

Proof.

According to Equations (4) and (5), we only need to show how to compute $S_{p,s}$ in linear time, for given $p$ and $s$. Let $Q_{p}$ be the set of all points $q\in P$ such that $q\prec^{\prime}s$, and $p\prec q$ or $p=q$. Let $Q_{s}$ be the set of all points $r\in P$ such that $p\prec^{\prime}r$, and $r\prec s$ or $r=s$. Note that $Q_{p}\cap Q_{s}=\emptyset$ and that $p$ and $s$ belong to $Q_{p}$ and $Q_{s}$, respectively, only when $p_{y}>s_{y}$. Let $L_{p,s}$ be the list of the elements of $Q_{p}\cup Q_{s}$ sorted by $x$-coordinate. Observe that if $(p,q,r,s)$ is a $4$-separator of $RCH(S_{p,s})$, then $r\in Q_{s}$, $q\in Q_{p}$, and $q$ is the point $q^{*}\in Q_{p}$ from the beginning of $L_{p,s}$ to $r$ such that $T^{\prime}_{p,q^{*}}+T_{r,s}=\max\{T^{\prime}_{p,q^{\prime}}+T_{r,s}\}$ over all $q^{\prime}\in Q_{p}$ from the beginning of $L_{p,s}$ to $r$.

We calculate $S_{p,s}$ by processing the elements of $L_{p,s}$ in order. For an element $t$ in $L_{p,s}$, let $q_{t}^{*}$ be the point in $Q_{p}$ maximizing $T^{\prime}_{p,q^{\prime}}$ over all $q^{\prime}\in Q_{p}$ from the beginning of $L_{p,s}$ to $t$ (including $t$). When processing a point $t$, observe that if $t\in Q_{p}$, then $q_{t}^{*}$ is either $t$ or $q_{t-1}^{*}$. Otherwise, if $t\in Q_{s}$, then $q_{t}^{*}=q_{t-1}^{*}$. Moreover, if $t\in Q_{s}$, then we set $q=q_{t}^{*}$ and $r=t$, and consider $(p,q,r,s)$ as a feasible $4$-separator of $\mathcal{S}_{p,s}$. After processing the last element of $L_{p,s}$, among all the (linear number) feasible separators, we return the solution $S_{p,s}$ induced by the feasible separator that maximizes $T^{\prime}_{p,q}+T_{r,s}$. Thus, $S_{p,s}$ can be calculated in $O(n)$ time, once tables $T$ and $T^{\prime}$ have been constructed. ∎

4 Maximum size/area empty rectilinear convex hulls and maximum weight rectilinear convex hull

In this section, we show how to adapt the algorithm of Section 3 to solve the MaxEmptyRCH, the MaxAreaRCH and the MaxWeightRCH problems. The first observation is that, as the optimal solution for any of these problems is the rectilinear convex hull of a subset $S$ of $P$, then Lemmas 2.5 and 3.4 hold. This implies that we can assume that $B(a,d)$ and $B(b,c)$ are separated by a vertical line, so a 4-separator exists for the optimal solution in any of the problems. As a consequence, the algorithms to solve these three problems follow the same scheme as the algorithm described in the previous section, and we only need to show how to adapt in each problem the calculation of the $2$-staircases, the triple $1$- and $3$-staircases and the rectilinear convex hulls $\mathcal{S}_{p,s}$ to fulfill the requirements on emptiness, area or weight.

We start by solving the MaxEmptyRCH problem, we continue with the MaxAreaRCH problem and we finish with the MaxWeightRCH problem.

4.1 Maximum size empty rectilinear convex hull

To solve the MaxEmptyRCH problem in $O(n^{3})$ time and $O(n^{2})$ space, we modify the steps of our previous algorithm. These modifications ensure that the “interiors” of the triple $1$- and $3$-staircases and the rectilinear convex hulls are empty. Recall that in this problem we are looking for a subset $S\subseteq P$ such that $RCH(S)$ has maximum size and there is no element of $P$ in the interior of $RCH(S)$.

The first step: For a pair of points $p,q\in P$ such that $p\prec q$ or $p=q$, we say that the $2$-staircase associated with a subset $S$ of $P(p,q)$ is empty if no point of $P$ is in the interior of $B(p,q)\cap M_{2}(S\cup\{p,q\})$, see Figure 7, left.

Let $\mathcal{C}_{p,q}$ be the empty $2$-staircase of maximum cardinality over all subsets $S\cup\{p,q\}$ with $S\subseteq P(p,q)$, and let $C_{p,q}$ be the size of $\mathcal{C}_{p,q}$. Observe that if $u$ is the point $(p_{x},q_{y})$ and $r\in P$ is the vertex of the $2$-staircase that follows $p$, then $P(r,u)=\emptyset$. Thus, values $C_{p,q}$ can be computed using the following recurrence:

[TABLE]

As $q$ and $u$ are on the same horizontal line, $P(q,u)$ is not defined. In this case, we assume that $P(q,u)$ is the empty set. Using standard range counting techniques [13], we can preprocess the grid $G$ generated by the vertical and horizontal lines through the elements of $P$ in $O(n^{2})$ time and space, so that for every pair of vertices $u,v$ of $G$ we can query the number of points in $P(u,v)$ in $O(1)$ time. Thus, we can decide whether $P(u,v)=\emptyset$ in $O(1)$ time. Therefore, values $C_{p,q}$ can be calculated in $O(n^{3})$ time and $O(n^{2})$ space.

The second step: For every $p,q\in P$ such that $p\prec q$ or $p=q$, we say that the triple $1$-staircase $\mathcal{T}$ corresponding to a subset $S$ of $z(p,q)$ is empty if the (disconnected) region $\mathcal{O}_{\mathcal{T}}=\overline{Z(p,q)}\cap M_{1}(S\cup\{p,q\})\cap M_{2}(S\cup\{p,q\})\cap M_{3}(S\cup\{p,q\})$ associated with $\mathcal{T}$ contains no element of $P$. Please, refer to Figure 4, where the shaded areas correspond to $\mathcal{O}_{\mathcal{T}}$. Let $\mathcal{T}_{p,q}$ be the empty triple $1$-staircase of maximum size among all subsets $S\cup\{p,q\}$, with $S\subseteq z(p,q)$. Let $E(p,q,r)$ denote the interior of $B(p,q)\cap Q_{2}(r)$ if $p\neq q$, and the empty set if $p=q$. In Figure 5 examples of open rectangles $E(p,q,r)$ are shown as shaded rectangles . We show how to compute $T_{p,q}$, the cardinality of the set of extreme vertices of $\mathcal{T}_{p,q}$, using the following equations that are similar to Equations (2) and (3):

[TABLE]

where for every pair $p,r\in P$ such that $p\prec^{\prime}r$

[TABLE]

In case (A), $\mathcal{O}_{p,q}$ is empty as $\mathcal{C}_{p,q}$ is an empty 2-staircase. Equation (7) is obtained from Equation (2) by further constraining $r$ in the cases from (B) to (E) to satisfy $P\cap E(p,q,r)=\emptyset$. This guarantees that the interior of $\mathcal{O}_{p,q}$ is empty of elements of $P$ for all $p,q$. Verifying that $P\cap E(p,q,r)=\emptyset$ can be decided in $O(1)$ time by using a range counting query. The proof of correctness of Equations (7) and (8) follows the same steps as in Lemma 3.2. Hence, computing the new table $T$ can be done in $O(n^{3})$ time and $O(n^{2})$ space. By symmetry, values in $T^{\prime}_{p,q}$, the sizes of the empty triple $3$-staircases $\mathcal{T}^{\prime}_{p,q}$, can also be calculated in $O(n^{3})$ time and $O(n^{2})$ space.

The third step: For given $p,s\in P$ such that $p_{x}<s_{x}$, let $\mathcal{S}_{p,s}$ be the empty rectilinear convex hull of maximum size, among all subsets $S\subseteq P$ containing $p$ and $s$ such that $RCH(S)$ is empty and there exist two points $q,r\in S$ with $(p,q,r,s)$ being a $4$-separator of $RCH(S)$. Let $S_{p,s}$ denote the size of $\mathcal{S}_{p,s}$. To compute $S_{p,s}$ we have to distinguish whether $p\prec^{\prime}s$ or $p\prec s$. If $p\prec^{\prime}s$ (see Figure 6, top-right, or any in the bottom), then $S_{p,s}=\max\{T^{\prime}_{p,q}+T_{r,s}\}$ over all $4$-separators $(p,q,r,s)$. Otherwise, if $p\prec s$ (see Figure 7, right), then we must ensure that each $4$-separator $(p,q,r,s)$ satisfies $P(u,v)=\emptyset$, where $u=(r_{x},s_{y})$ and $v=(q_{x},p_{y})$.

If $S_{0}$ is a subset of $P$ such that $RCH(S_{0})$ is empty, vertically separable and of maximum size, the new equations to compute $|RCH(S_{0})|$ are:

[TABLE]

where for each pair of points $p,s\in P$ such that $p_{x}<s_{x}$

[TABLE]

Theorem 4.1**.**

The MaxEmptyRCH problem can be solved in $O(n^{3})$ time and $O(n^{2})$ space.

Proof.

Again, we only need to show that given points $p$ and $s$ $S_{p,s}$ can be computed in linear time. When $p\prec^{\prime}s$, we argue as in the proof of Theorem 3.5. However, when $p\prec s$, we need to only consider $4$-separators such that $P(u,v)=\emptyset$. Let $Q_{p,s}=\{t\in P:p\prec t\prec s\}$, which satisfies $Q_{p,s}\cap Q_{p}=\emptyset$ and $Q_{p,s}\cap Q_{s}=\emptyset$. Recall that $Q_{p}\cap Q_{s}=\emptyset$ and that, when $p\prec s$, $Q_{p}=\{q\in P:p\prec q,q\prec^{\prime}s\}$ and $Q_{s}=\{r\in P:r\prec s,p\prec^{\prime}r\}$. Let $L_{p,s}$ be the list of the points of $Q_{p}\cup Q_{s}\cup Q_{p,s}$ sorted by $x$-coordinate. Assuming that we have already sorted $P$ by $x$-coordinate $L_{p,s}$ is obtained in $O(n)$ time.

As before, we calculate $S_{p,s}$ by processing the elements of $L_{p,s}$ in order. For an element $t$ in $L_{p,s}$, let $q_{t}^{*}$ be the point in $Q_{p}$ maximizing $T^{\prime}_{p,q^{\prime}}$ over all $q^{\prime}\in Q_{p}$ from the beginning of $L_{p,s}$ to $t$ (including $t$) subject to there being no elements of $Q_{p,s}$ in $L_{p,s}$ from $q^{\prime}$ to $t$. When processing a point $t\in Q_{p,s}$, we set $q_{t}^{*}=\mathbf{nil}$ denoting that $q_{t}^{*}$ is undefined. Observe that, when processing a point $t\in Q_{p}$, if $q_{t-1}^{*}=\mathbf{nil}$ then $q_{t}^{*}=t$, and if $q_{t-1}^{*}\neq\mathbf{nil}$ then $q_{t}^{*}$ is either $t$ or $q_{t-1}^{*}$. When processing a point $t\in Q_{s}$, then $q_{t}^{*}=q_{t-1}^{*}$, and if $q_{t-1}^{*}\neq\mathbf{nil}$, then we set $q=q_{t}^{*}$ and $r=t$, and consider $(p,q,r,s)$ as a feasible $4$-separator of $\mathcal{S}_{p,s}$. Note that for this $4$-separator we have $P(u,v)=\emptyset$. After processing all elements in $L_{p,s}$, $S_{p,s}$ is determined by a feasible $4$-separator that maximizes $T^{\prime}_{p,q}+T_{r,s}$. ∎

4.2 Maximum area empty rectilinear convex hull

In the MaxAreaRCH problem, we determine an empty rectilinear convex hull of maximum area. To solve this problem, we proceed as in the previous subsection. The only difference is that we sum areas in all of our recurrences, instead of counting points. Given a bounded set $Z\subset\mathbb{R}^{2}$, we denote the area of $Z$ as $\mathsf{Area}(Z)$.

Now, $\mathcal{C}_{p,q}$, $\mathcal{T}_{p,q}$, $\mathcal{T}^{\prime}_{p,q}$ and $\mathcal{S}_{p,q}$ are as described in Section 4.1, with the difference that they maximize area instead of maximizing size. The areas are defined as follows. If $S=\{p,v_{2},\ldots,v_{k-1},q\}$ is the set of vertices of an empty $2$-staircase, we define the area of this staircase as $\mathsf{Area}(B(p,q)\cap M_{2}(S))$. For an empty triple $1$-staircase or an empty triple $3$-staircase $\mathcal{T}$, its area is the area of its associated region $\mathcal{O}_{\mathcal{T}}$. The area of a rectilinear convex hull is the area of its interior.

The first step: For a pair of points $p,q\in P$ such that $p\prec q$ or $q=p$, we compute $C_{p,q}$, the area of $\mathcal{C}_{p,q}$, using the following recurrence, which is a variant of Equation (6) maximizing area:

[TABLE]

where $u=(p_{x},q_{y})$. As $B(q,u)$ is not defined, we set $\mathsf{Area}(B(q,u))=0$.

The second step: For every $p,q\in P$ such that $p\prec q$ or $p=q$, let $T_{p,q}$ be the area of $\mathcal{T}_{p,q}$. All $T_{p,q}$’s can be calculated in $O(n^{3})$ time and $O(n^{2})$ space using the following equations, which are variants of Equations (7) and (8) maximizing area (recall that if $p=q$, then $E(p,q,r)=\emptyset$, so its area is [math]):

[TABLE]

where for every pair $p,r\in P$ such that $p\prec^{\prime}r$

[TABLE]

The areas $T^{\prime}_{p,q}$ for the empty triple $3$-staircases $\mathcal{T}^{\prime}_{p,q}$ can be calculated in a similar way.

The third step: Let $S_{p,s}$ be the area of $\mathcal{S}_{p,s}$. Recall that, for given $p,s\in P$ such that $p_{x}<s_{x}$, $\mathcal{S}_{p,s}$ is the empty rectilinear convex hull of maximum area, among all subsets $S\subseteq P$ containing $p$ and $s$ such that $RCH(S)$ is empty and there exist two points $q,r\in S$ with $(p,q,r,s)$ a $4$-separator of $RCH(S)$. Observe that if $p\prec^{\prime}s$, then $S_{p,s}=T^{\prime}_{p,q}+T_{r,s}$ for some $4$-separator $(p,q,r,s)$. Otherwise, if $p\prec s$ then $S_{p,s}=\mathsf{Area}(B(u,v))+T^{\prime}_{p,q}+T_{r,s}$ for some $4$-separator $(p,q,r,s)$, subject to $P(u,v)=\emptyset$, where $u=u(r)=(r_{x},s_{y})$ and $v=v(q)=(q_{x},p_{y})$ (see Figure 7, right).

Given that $\mathsf{Area}(B(u,v))$ depends on both $r$ and $q$, using the inclusion/exclusion principle, we can then calculate $S_{p,s}$ as $S_{p,s}=T^{\prime}_{p,q}+\mathsf{Area}(B(v,s))+T_{r,s}+\mathsf{Area}(B(p,u))-\mathsf{Area}(B(p,s)).$ Since $p$ and $s$ are fixed, note that $U(p,q,s)=T^{\prime}_{p,q}+\mathsf{Area}(B(v(q),s))$ depends only on $q$ and $V(p,r,s)=T_{r,s}+\mathsf{Area}(B(p,u(r)))-\mathsf{Area}(B(p,s))$ depends only on $r$. Each of these two values can be computed in $O(1)$ time, once $T$ and $T^{\prime}$ have been computed in the second step. If $S_{0}$ is a subset of $P$ such that $RCH(S_{0})$ is empty, vertically separable and of maximum area, the new equations to compute $RCH(S_{0})$ are:

[TABLE]

where for each pair of points $p,s\in P$ such that $p_{x}<s_{x}$

[TABLE]

Theorem 4.2**.**

The MaxAreaRCH problem can be solved in $O(n^{3})$ time and $O(n^{2})$ space.

Proof.

The proof follows the proof of Theorem 4.1. The only difference is that, when processing an element $t$ in $L_{p,s}$, $q_{t}^{*}$ is the point in $Q_{p}$ maximizing $T^{\prime}_{p,q^{\prime}}+\mathsf{Area}(B(v(q^{\prime}),s))$, instead of maximizing $T^{\prime}_{p,q^{\prime}}$. After processing all elements in $L_{p,s}$, $S_{p,s}$ is determined by a feasible $4$-separator that maximizes $T^{\prime}_{p,q}+\mathsf{Area}(B(v,s))+T_{r,s}+\mathsf{Area}(B(p,u))-\mathsf{Area}(B(p,s))$. ∎

4.3 Maximum weight rectilinear convex hull

In the MaxWeightRCH problem, each input point $p$ of $P$ is comes with a (positive or negative) weight $w(p)$. We determine a subset $S\subseteq P$ such that $RCH(S)$ has maximum weight, that is, such that $\sum_{p\in P\cap RCH(S)}{}w(p)$ is maximized. The algorithm to solve this problem combines the ideas of the previous algorithms and follows the same steps, however, now we add weights. We define $\mathsf{Weight}(Z)=\sum_{p\in P\cap Z}w(p)$ as the weight of a region $Z\subset\mathbb{R}^{2}$. Using the same range counting techniques [13] as in Section 3, we can preprocess the grid $G$ generated by the vertical and horizontal lines through the elements of $P$ in $O(n^{2})$ time and space, so that for every pair of vertices $u,v$ of $G$ we can query $\mathsf{Weight}(B(u,v))=\sum_{p\in P(u,v)}w(p)$ in $O(1)$ time.

Now, $\mathcal{C}_{p,q}$, $\mathcal{T}_{p,q}$, $\mathcal{T}^{\prime}_{p,q}$ and $\mathcal{S}_{p,q}$ are as described in Section 3, except that weight is maximized. The weights are defined as follows. If $S=\{p,v_{2},\ldots,v_{k-1},q\}$ is the set of vertices of a $2$-staircase, its weight is defined as $w(p)+w(q)+\mathsf{Weight}(B(p,q)\cap M_{2}(S))$. Note that the weights of all points in $S$ are included in this formulae. For a triple $1$-staircase or a triple $3$-staircase $\mathcal{T}$, its weight is the addition of the weights of the points of $P$ that appear on the boundary or in the interior of $\mathcal{O}_{\mathcal{T}}$, the region associated with $\mathcal{T}$. Finally, the weight of a rectilinear convex hull is the addition of the points of $P$ on its boundary or interior.

The first step: If $C_{p,q}$ is the weight of $\mathcal{C}_{p,q}$, for a pair of points $p,q\in P$ such that $p\prec q$ or $q=p$, all $C_{p,q}$’s can be computed in $O(n^{3})$ time and $O(n^{2})$ space using the following recurrence:

[TABLE]

where $u=(p_{x},q_{y})$. We set $\mathsf{Weight}(B(q,u))=0$ as $B(q,u)$ is not defined.

The second step: If $T_{p,q}$ is the weight of $\mathcal{T}_{p,q}$, for every $p,q\in P$ such that $p\prec q$ or $p=q$, then all $T_{p,q}$’s (and, by symmetry, all $T^{\prime}_{p,q}$’s) can be calculated in $O(n^{3})$ time and $O(n^{2})$ space using the following equations, where $\alpha_{p,q}=w(p)$ if $p=q$, and $\alpha_{p,q}=w(p)+w(q)$ if $p\neq q$:

[TABLE]

where for every pair $p,r\in P$ such that $p\prec^{\prime}r$

[TABLE]

The third step: For given $p,s\in P$ such that $p_{x}<s_{x}$, let $S_{p,s}$ be the weight of $\mathcal{S}_{p,s}$. Using similar reasoning as in the previous subsection, one can show that, if $S_{0}$ is a subset of $P$ such that $RCH(S_{0})$ is vertically separable of maximum weight, the following equations calculate $|RCH(S_{0})|$:

[TABLE]

where for each pair of points $p,s\in P$ such that $p_{x}<s_{x}$

[TABLE]

Now $U(p,q,s)=T^{\prime}_{p,q}+\mathsf{Weight}(B(v(q),s))$ and $V(p,r,s)=T_{r,s}+\mathsf{Weight}(B(p,u(r)))-\mathsf{Weight}(B(p,s))$, with $u=u(r)=(r_{x},s_{y})$ and $v=v(q)=(q_{x},p_{y})$. The proof of the next theorem is a straightforward adaptation of the previous arguments.

Theorem 4.3**.**

The MaxWeightRCH problem can be solved in $O(n^{3})$ time and $O(n^{2})$ space.