Maximum-Area Triangle in a Convex Polygon, Revisited

TL;DR

This paper critically examines and improves algorithms for finding maximum-area inscribed polygons in convex polygons, revealing flaws in previous methods and proposing more accurate algorithms with different time complexities.

Contribution

It demonstrates the failure of a 1979 linear-time algorithm for maximum-area inscribed triangles and introduces a corrected quadratic-time and an $O(n ext{log} n)$ divide-and-conquer algorithm.

Findings

The 1979 linear-time algorithm for maximum-area inscribed triangles fails in some cases.

A corrected quadratic-time algorithm is proposed for the problem.

The 1979 algorithm for maximum-area $k$-gon with $k=4$ also fails to find the optimal solution.

Abstract

We revisit the following problem: Given a convex polygon $P$, find the largest-area inscribed triangle. We show by example that the linear-time algorithm presented in 1979 by Dobkin and Snyder for solving this problem fails. We then proceed to show that with a small adaptation, their approach does lead to a quadratic-time algorithm. We also present a more involved $O(n\log n)$ time divide-and-conquer algorithm. Also we show by example that the algorithm presented in 1979 by Dobkin and Snyder for finding the largest-area $k$-gon that is inscribed in a convex polygon fails to find the optimal solution for $k=4$. Finally, we discuss the implications of our discoveries on the literature.

Equations

$$T(n)=\max \{T(\alpha (n+6))+T((1-\alpha )(n+6))+O(n+6),T(\alpha (n+6)+O(n+6)\}$$

$$T(m)=T(\alpha m)+T((1-\alpha )m)+m$$

Full text

1 Introduction

We revisit a classic problem in computational geometry: Given a convex polygon $P$, find the largest-area inscribed triangle. Figure 1 illustrates the problem. In 1979, Dobkin and Snyder [1] presented a linear-time algorithm to solve this problem. In this note, we present an example of a polygon on which their algorithm fails. There exists, however, another linear-time algorithm for computing the largest inscribed triangle is presented by Chandran and Mount [2], originally intended to solve the parallel version of the problem. Since the initial posting of this manuscript on arXiv [3], two new linear-time algorithms for solving the problem have already been claimed by Kallus [4] and Jin [5].

The counter example is shown in Figure 12 and requires careful placement of the vertices of the polygon: our coordinates are integers in the range $[0,5000]$, and a range of this order of magnitude seems to be necessary. We remark that in [6] the authors implemented the presented algorithm in [2], however, for the correctness of the implication, they produced a set of $10,000$ random convex polygons with range of vertices in $[0,1000]$, although our counter example clarifies that the presented algorithm by Dobkin and Snyder [1] always works correctly on any random convex polygon in the range $[0,1000]$.

Our counter example was found by solving a system of quadratic equations. We carefully analyze the underlying geometry and give insight into the reason for the failure of the original algorithm, and use this insight to create a new $O(n\log n)$ algorithm to solve the problem.

The study of geometric containment problems was initiated by Michael Shamos [7], who considered the question of finding the longest line segment in a convex polygon, also known in computational geometry as the diameter of the polygon. Shamos presented a linear-time algorithm in his thesis [8], based on a technique which is now known under the name rotating calipers [9]. Dobkin and Snyder [1] also claimed a linear-time algorithm for computing the diameter, which was later found to be incorrect by Avis et al. [10].

Dobkin and Snyder [1] originally claimed to have a generic linear-time algorithm for finding the largest inscribed $k$-gon inside a convex polygon (for constant $k$), but this claim was later retracted: Boyce et al. [11] observe that the algorithm by Dobkin and Snyder fails for $k=5$ and instead present a $O(kn\log n+n\log^{2}n)$ algorithm for finding the largest $k$-gon. Aggarwal et al. [12] improve their result to $O(kn+n\log n)$ time by using a matrix search method. However, both algorithms still rely on the correctness of the Dobkin and Snyder algorithm for triangles, which to this day remains uncontested. As we show here, this algorithm is also incorrect, and by extension, so are the solutions by Boyce et al. and Aggarwal et al.. In Section 8, we discuss the implications of our finding in more detail.

Furthermore, We also show by example that the presented algorithm [1] also fails to find the optimal solution for $k=4$. Our counter example is a polygon on 16 vertices in the range [0, 26500]. Our counter example, combined with the work in [10] and [11] would suggest that the problem of finding the largest-area quadrilateral in linear time is still open. Although Boyce et al. [11] claimed one is presented by Shamos [7], but the cited manuscript cannot be found on-line.

1.1 Related work

The largest-area triangle problem falls in a broader class of geometric optimization problems, where the goal is to find some object inscribed inside another object. During the past 40 years, many optimization measures have been studied:

Chang et al. [13] studied the problem of computing the largest-area or largest-perimeter convex polygon that is inscribed in a simple polygon $P$. They presented an $O(n^{7})$ time algorithm for the largest-area inscribed convex hull. They solved this problem by using the fact that a largest-area convex polygon must be the intersection of $P$ and a set of $m$ half-planes defined by $m$ chords of $P$, where $m$ is smaller than the number of reflex vertices of $P$. They also presented an $O(n^{6})$ time algorithm for the largest-perimeter inscribed convex hull. Obviously, when the given polygon is convex,the solution to both problems is unique and can be found in linear time. Cabello et al. [14] presented a randomized near-linear-time $(1-\varepsilon)$-approximation algorithm for this problem, with running time in $O(n(\log^{2}n+(1/\varepsilon^{3})\log n+1/\varepsilon^{4}))$ time, and with probability at least $2/3$, the result has an area of at least $(1-\varepsilon)$ times the area of an optimal solution..

Daniels et al. [15] presented an $O(n\log^{2}n)$ time algorithm for finding the largest-area axis-parallel rectangle that is inscribed in a simple polygon. They also presented an $\Omega(n\log n)$ lower bound for this problem. The running time also matches with the one of the best known algorithm for finding the largest-area axis-parallel rectangle that is inscribed in an orthogonal polygons.

Cabello et al. [16] studied the problem of finding the largest-area or largest-perimeter rectangle that is inscribed in a convex polygon. They presented an exact algorithm that runs in $O(n^{3})$ time, and a $(1-\varepsilon)$-approximation algorithm that runs in $O(\varepsilon^{-\frac{1}{2}}\log n+\varepsilon^{-\frac{3}{2}})$ time; see also [17, 18, 19, 20].

DePano, Ke and O’Rourke [21] studied the problem of computing the largest-area square and equilateral triangle contained in a convex polygon.

Jin and Matulel [22] studied the problem of computing the largest-area parallelogram that is inscribed in a convex polygon $P$, and presented an $O(n^{2})$ time algorithm, which is based on the fact that largest-area parallelogram must have all of its corners on the perimeter of $P$.

Melissaratos and Souvaine [23] studied the problem of computing the largest-area or perimeter triangle that is inscribed in a non-convex polygon. They presented an $O(n^{3})$ time algorithm for these problems.

When the input is an (unstructured) set of points in the plane, rather than a convex polygon, we may ask a similar question: what is the largest-area triangle or $k$-gon that uses only points of the given set as vertices? Clearly, we can attack this problem by first computing the convex hull of the given points. However, this takes $O(n\log n)$ time, and indeed Drysdale and Jaromczyk show that computing the largest-area or largest-perimeter $k$-gon must take at least $\Omega(n\log n)$ time for any $k\geq 2$, using a reduction from set disjointness [24].

1.2 Contribution

In this paper, we obtain the following results.

• •

We present a $9$-vertex polygon on which the algorithm by Dobkin and Snyder for computing the largest-area triangle [1] fails. By extensions, the algorithm by Boyce et al. and Aggarwal et al. for computing the largest-area $k$-gon [11, 12] also fail. In particular, the example disproves Lemma 2.5 of Dobkin and Snyder [1] and Lemma 3.2 of Boyce et al. [11] (Section 7.2).

• •

We analyze the geometry and present insight into the reason of the failure of these lemmas. We then use this insight to give a quadratic-time algorithm for computing the largest-area triangle in the same spirit as Dobkin and Snyder’s algorithm, which we prove is correct (Section 5).

• •

We then extend our analysis significantly and present a divide-and-conquer algorithm that works in $O(n\log n)$ time (Section 6).

• •

We present a $16$-vertex polygon on which the algorithm by Dobkin and Snyder for computing the largest-area quadrangle [1] fails.

As an effect of the problem’s central nature in computational geometry, a number of follow-up results that depend on largest-area triangles or $k$-gons, either directly by using the algorithm as a preprocessing step or indirectly by relying on false claimed properties, will have to be reevaluated [1, 11, 12, 25, 26]. We discuss several of these in some detail in Section 8.

2 Preliminaries

In this section, we first review several core concepts introduced by Dobkin and Snyder [1] and Boyce et al. [11], and then describe the algorithm by Dobkin and Snyder in detail. Although we are primarily concerned with triangles, we introduce some concepts more generally for $k$-gons; as we discuss in Section 8, the question of finding the largest-area $k$-gon inside a convex polygon is also impacted (and reopened) by our results.

2.1 Definitions

Let $P$ be a convex polygon with $n$ vertices. We say a convex polygon $Q$ is $P$-aligned if the vertices of $Q$ are a subset of the vertices of $P$. Note that there always exists a $P$-aligned largest-area $k$-gon inscribed in $P$ (assuming $k\leq n$). Boyce et al. [11] define a rooted polygon $Q$ with root $r\in P$ to be any $P$-aligned polygon that includes $r$. Let $P$ be a convex polygon. Two $P$-aligning polygons are said to interleave, if between every two successive vertices of one, there is a vertex of the other (possibly coinciding with one of them) [11]. Dobkin and Snyder [1] define a stable triangle to be a rooted triangle $T=pqr$, where $r$ is the root, such that any other $P$-aligned triangle $\triangle p^{\prime}qr$ or $\triangle pq^{\prime}r$ has area smaller than (or equal to) $T$. Henceforth, we will refer to such triangles as 2-stable, and to $p$ and $q$ as stable vertices. We also define a 3-stable triangle to be a (rooted or unrooted) $P$-aligned triangle $T=pqr$ such that any other $P$-aligned triangle $\triangle p^{\prime}qr$ or $\triangle pq^{\prime}r$ or $\triangle pqr^{\prime}$ has area smaller than (or equal to) $T$. Note that in degenerate cases, there could be multiple (stable) triangles with equal area. In the remainder, we denote by $\Lambda$ the largest-area triangle, or, if it is not unique, any triangle with maximum area. Note that $\Lambda$ is 3-stable.

2.2 Dobkin and Snyder’s triangle algorithm

We will now recall the triangle algorithm [1], outlined in Algorithm 1 and illustrated in Figure 10111For Simplification of the implication, our algorithm is presented in C++ pseudo-code, it is easy to observe that it is equivalent to the original presented algorithm in  [1]. The same is also hold for $k=4$. .

Let $P=\{p_{0},p_{1},\ldots,p_{n-1}\}$. We assume that $P$ is given in a clockwise orientation. Assume an arbitrary vertex of $P$ is the root, assign this vertex and its two subsequent vertices in the clockwise order on the boundary of $P$ to variables $a,b$ and $c$. We then “move $c$ forward” along the boundary of $P$ as long as this increases the area of $\triangle abc$. If we can no longer advance $c$, we advance $b$ if this increases the area of $\triangle abc$, then try again to advance $c$. If we cannot advance either $b$ or $c$ any further, we advance $a$. We keep track of the largest-area triangle found, and stop when $a$ returns to the starting position. Since $a$ visits $n$ vertices and $b$ and $c$ each visit fewer than $2n$ vertices, the algorithm runs in $O(n)$ time (assuming we are given the cyclic ordering of the points on $P$).

Dobkin and Snyder [1] claim that Algorithm 1 computes the largest-area triangle inscribed in $P$. Their argument hinges on the following key lemma.

Lemma 1** ([1, Lemma 2.5]).**

Let $P=\{p_{0},p_{1},\ldots,p_{n-1}\}$ be a convex polygon. There exists an $i$ ($0\leq i\leq n-1$) such that the $p_{i}$-anchored maximum triangle is the largest-area triangle inscribed in $P$.

We claim that Lemma 2.5 [1] is false. In the lemma, the $p_{i}$-anchored maximum triangle refers to the largest triangle found by the algorithm while $a=p_{i}$. Note that this is not necessarily the same as the largest-area rooted triangle at $p_{i}$. This essential observation lies at the heart of the following construction.

3 Counter-example to Algorithm 1

In Figure 12 we provide a polygon $P$ on $9$ vertices such that the largest-area inscribed triangle and the triangle computed by Algorithm 1 are not the same. We use the following points: $a_{1}=(4752,4262),a_{2}=(3383,413),b_{1}=(759,2927),b_{2}=(4745,4322),c_{1}=(1213,691),c_{2}=(2506,4423),a_{0}=(3040,4460),b_{0}=(1000,1000),c_{0}=(5000,1000)$. The largest-area triangle is $\triangle a_{0}b_{0}c_{0}$; however, Algorithm 1 reports triangle $\triangle c_{0}c_{1}c_{2}$ as the largest-area triangle.

We obtained $P$ by solving the following problem. Let $\triangle a_{0}b_{0}c_{0}$ be the globally maximum area triangle on $P$, find six vertices $a_{1}$, $a_{2}$, $b_{1}$, $b_{2}$, $c_{1}$ and $c_{2}$ such that each of $\triangle a_{1}a_{2}a_{0}$, $\triangle b_{1}b_{2}b_{0}$ and $\triangle c_{1}c_{2}c_{0}$ are 2-stable triangles with area strictly less than $\triangle a_{0}b_{0}c_{0}$, and there is no other triangle on $P=\{a_{0},b_{0},c_{0},a_{1},a_{2},b_{1},b_{2},c_{1},c_{2}\}$ with area equal or larger than $a_{0}b_{0}c_{0}$. This resulted in a set of ${{9}\choose{3}}-1$ nonlinear constraints (many of which are redundant by Lemma 8) which delineate a very small but non-empty solution space. We were able to find an integer solution with 4-digit integers, but none with 3-digit integers.

4 Some observations on the largest-area triangle

In this section, we argue that Algorithm 1 does work on convex polygons in which there exists only one 2-stable triangle per vertex. However, if there are multiple 2-stable triangles rooted at the same vertex, the algorithm only works if the first such triangle considered happens to be the largest one. We investigate how many 2- and 3-stable triangles there can be, both per vertex and in total.

4.1 The number of 2-stable triangles

Lemma 2**.**

Let $r$ be any vertex of $P$. All 2-stable triangles rooted at $r$ are interleaving.

Proof.

Figure 4(b,c) illustrates the proof. Assume for the sake of contradiction that $P$ is a convex polygon and $a$ is a vertex on $P$ with several 2-stable triangles, such that there exists at least one 2-stable triangle $\triangle abc$ rooted at $a$ that is not interleaving with another 2-stable triangle $\triangle ab^{\prime}c^{\prime}$. W.l.o.g. both $\triangle abc$ and $\triangle ab^{\prime}c^{\prime}$ are ordered counterclockwise.

First consider the case where $c$ occurs before $c^{\prime}$ and $b^{\prime}$ occurs before $b$ in counterclockwise order. Let $L$ and $L^{\prime}$ be the lines through $b$ that are parallel to the lines $ac$ and $ac^{\prime}$ respectively. As $\triangle ab^{\prime}c^{\prime}$ is 2-stable, $b^{\prime}$ has to be on the same side of $L$ than $a$. Additionally $b^{\prime}$ has to be on the other side of $L^{\prime}$ than $a$. It follows that $b^{\prime}$ has to be placed after $b$ in counterclockwise ordering, contradicting the assumption.

Second consider the case where $b^{\prime}$ and $c^{\prime}$ both occur before $b$ and $c$ in counterclockwise order. Let $L$ and $L^{\prime}$ be the lines through $c^{\prime}$ that are parallel to the lines $ac$ and $ab^{\prime}$ respectively. As $\triangle abc$ is 2-stable, $b$ is on the other side of $L$ than $a$. As $\triangle ab^{\prime}c^{\prime}$ is 2-stable, $b$ is on the same side of $L^{\prime}$ than $a$. This forces $b$ to be placed before $c^{\prime}$ in counterclockwise ordering which again is a contradiction. ∎

Observation 3**.**

The number of 2-stable triangles rooted at any given vertex of a convex polygon is at most $O(n)$.

It is possible that for a given vertex $p_{0}$ on a convex polygon $P$, each edge $p_{0}p_{i}$ is an edge of a 2-stable triangle, as illustrated in Figure 6. Furthermore, Figure 5 shows that any of these triangles could be the largest one; that is, the sequence of areas of the $2$-stable triangles rooted at $p_{0}$ is not necessarily increasing or decreasing. However, $p_{0}p_{i}$ can participate in at most two 2-stable triangles, namely, using the vertices that are farthest from the line through $p_{0}$ and $p_{i}$. So, the number of 2-stable triangles rooted at any vertex of $P$ is $O(n)$.

Furthermore, we can slightly alter the polygon in Figure 5(left) so that it becomes a polygon $P^{\prime}$ with $O(n^{2})$ $2$-stable triangles, see Figure 5(right). If we replace $p_{0}$ with $n$ new vertices $p_{0},p_{0_{1}},p_{0_{2}},\ldots,p_{0_{n-1}}$, all on the boundary of $P$ (ordered clock-wisely) and close to each other, and such that the other points $p_{i}$ are far enough from each other, the resulting polygon $P^{\prime}$ can have $O(n^{2})$ 2-stable triangles.

Implicitly, Algorithm 1 is based on the assumption that there is only a linear number of $2$-stable triangles that are comparable in size and to the size of the largest-area triangle. We can alter the example in Figure 5(right) to make any of the $O(n^{2})$ $2$-stable triangles the largest, but independent of such a change, Algorithm 1 will always report the maximum area triangle out of the first found 2-stable triangles among all the vertices as the largest-area triangle.

Clearly, Observation 3 shows that the total number of 2-stable triangles is at most quadratic.

Corollary 4**.**

The total number of 2-stable triangles on a convex polygon is bounded by $O(n^{2})$.

In [3] we show that correcting the Dobkin and Snyder algorithm to find all the 2-stable triangles results in a quadratic time algorithm.

4.2 The number of 3-stable triangles

Lemma 5**.**

Two 3-stable triangles on a convex polygon are always interleaving.

Proof.

Let $\triangle abc$ and $\triangle rst$ be 3-stable triangles. Assume for the sake of contradiction that they are not interleaving. Following Lemma 2 we know that the two triangles do not have a common vertex. So either two vertices of $\triangle rst$ are between two vertices of $\triangle abc$ in counterclockwise order or the all three vertices of $\triangle rst$ are between two vertices of $\triangle abc$ in counterclockwise order. W.l.o.g. we can assume that the counterclockwise order or the vertices is either $a,b,r,s,c,t$ or $a,b,r,s,t,c$. The proof is the same for both cases and can be seen in Figure 7. For the sake of simplicity we assume that the line $sa$ is horizontal and that the vertices $c$ and $t$ lie below that line. Let $L$ and $L^{\prime}$ be the lines through $b$ that are parallel to the lines $ac$ and $st$ respectively. The vertex $r$ is closer to $ac$ than $b$ and $r$ is further away from $st$ than $b$. However due to $L$ having an upward slope and $L^{\prime}$ having a downward slope, $r$ has to be before $b$ in cyclic ordering, which is a contradiction. ∎

Lemma 6**.**

The total number of 3-stable triangles on a given convex polygon is bounded by $O(n)$.

Proof.

Suppose $P=a_{0},...,a_{n-1}$ is the convex polygon. Let $P$ given in the clockwise ordering. Let $X_{0}$ be the number of 3-stable triangles rooted at $a_{0}$, and let $x_{0,1}$ and $x_{0,n_{a_{0}}}$ denote, respectively, the first and last 3-stable triangles rooted at $a_{0}$, etc. Interleaving property of 3-stable triangles follows that if we move $a_{0}$ forward to $a_{1}$, the second and third vertices of triangle $x_{1,1}$ can only be located after or on the second and third vertices of triangle $x_{0,1}$. Also exactly one triangle rooted at $a_{1}$ can share both of the second and third vertices with one 3-stable triangle rooted at $a_{0}$, otherwise contradict the convexity of $P$. Note that the same argument is hold for all the successive vertices of $a_{1}$.

If there is a triangle $x_{1,j}$ with a common second vertex with $x_{0,i}$, then the third vertex of $x_{1,j}$ either coincides with the third vertex of $x_{0,i}$, or moves forward on the clockwise ordering of $P$. Thus the range of movement of the second vertex of 3-stable triangles rooted at $a_{1}$ is bounded by the second vertices of triangles $x_{0,1}$ and $x_{0,n_{a_{0}}}$ in the clockwise ordering, and the range of movement of the third vertex of 3-stable triangles rooted at $a_{1}$ is bounded by the third vertices of triangles $x_{0,1}$ and $a_{n-1}$. Let $i_{0}$ denote the index of the second vertex of $x_{0,n_{a_{0}}}$ and so on.

Obviously we never passed through $a_{n-1}$. Also $i_{j}$ for $j=0,...,n-1$ are ordered by their indices. Also since the 3-stable triangles are interleaving, the range of the indices we trace by the third vertex of any 3-stable triangles do not overlap (or overlap with at most one vertex). All together $\sum_{j=0}^{{n-1}}(i_{j+1}-i_{j})$ is bounded by $O(n)$, and thus the total number of 3-stable triangles is bounded by $O(n)$.

∎

5 A quadratic-time triangle algorithm

In this section, we present a quadratic-time algorithm to find the largest-area inscribed triangle. Let $P=\{p_{0},p_{1},\ldots,p_{n-1}\}$ be a given convex polygon. Recall that the largest-area triangle is 3-stable. The idea of the algorithm is to find all 3-stable triangles in $P$: for each vertex $p_{i}$ we find all 2-stable triangles rooted at $p_{i}$ in a single linear pass; because all 3-stable triangles are also 2-stable for some vertex, we find all 3-stable triangles.

In step $i$ of Algorithm 2, we let $a=p_{i}$ be the root, and start searching from $a$ and its two subsequent vertices $b$ and $c$ on $P$. In contrast to Algorithm 1, each time we move $a$, we reset $b$ and $c$, but just like in Algorithm 1, each time we move $b$, $c$ stays where it is. This means each step of the algorithm now takes linear time, and the total algorithm takes quadratic time.

5.1 Correctness

We will start the correctness proof of Algorithm 2 by the following lemma.

Lemma 7**.**

Algorithm 2 considers all 2-stable triangles.

Proof.

Suppose the lemma is false. Then, there exists at least one 2-stable triangle $\triangle rst$ rooted at $r$ that the algorithm cannot find when $a=r$. First assume that during the algorithm, at some point, $b$ will reach $s$. While $b$ is at $s$, $c$ will traverse some sequence $X$ of vertices of $P$; let $x$ be the first vertex of $X$. If $t$ is not in $X$, there are two cases: $t$ comes before $X$, or $t$ comes after $X$.

If $t$ comes before $X$, then $c$ already passed $t$ before $b$ reached $s$. This means that $b$ was at some point $u$ when $c$ passed $t$. But now, $\triangle rux$ and $\triangle rst$ would both be $2$-stable, but not interleaved, contradicting Lemma 2 (see Figure 8(left)).

If $t$ comes after $X$, then $b$ already moved away from $s$ before $c$ reaches $t$, say, when $c$ was at another vertex $v$. But then, $\triangle rsv$ was $2$-stable. However, $\triangle rst$ is also $2$-stable, contradicting the definition of $2$-stability.

Now suppose we missed $\triangle rst$ because $b$ did not reach $s$ before $c$ reaches $a-1$. But then, $c$ passed both $s$ and $t$, so when this happens $\triangle rbc$ and $\triangle rst$ are not interleaved, a contradiction with Lemma 2 (see Figure 8(right)). ∎

Theorem 1**.**

Algorithm 2 will find the largest-area triangle in $O(n^{2})$ time.

Proof.

The correctness of the algorithm depends on three facts. First, the largest-area triangle is always a 3-stable triangle; second, in the above procedure we will find all 3-stable triangles; and third, the set of all 3-stable triangles is a subset of the set of all 2-stable triangles. The correctness of the first and third facts are obvious. In Lemma 2 we proved we will consider all the 2-stable triangles. Thus we can conclude the Algorithm 2 works correctly.

∎

6 A divide-and-conquer triangle algorithm

In this section, we will provide a more efficient algorithm for finding the largest-area inscribed triangle on a convex polygon. We will use the following previously established lemmas.

Lemma 8** ([11, Lemma 2.2]).**

A globally largest-area $k$-gon and a largest-area rooted $k$-gon interleave.

Lemma 9**.**

The largest-area rooted triangle can be found in linear time.

Proof.

The largest area triangle rooted at an arbitrary fixed vertex $a$ of $P$ can be found via one step of Algorithm 2. In the correctness proof of that algorithm we mentioned that we can find all the 2-stable triangles on any given root $a$ in linear time. The largest-area rooted triangle can also be found in linear time. ∎

6.1 Algorithm

In the first step of the algorithm we choose an arbitrary vertex $a$ on $P$ and compute the largest-area triangle rooted at $a$. We call this triangle ${T^{1}}_{a}$. ${T^{1}}_{a}$ decomposes the boundary of $P$ into three intervals that share their endpoints. Let $m$ be the median vertex on the largest of these intervals (largest in terms of complexity). We then compute the largest-area rooted triangle on $m$, ${T^{1}}_{m}$. We call ${T^{1}}_{a}$ and ${T^{1}}_{m}$ dividing triangles. The vertices of ${T^{1}}_{a}$ and ${T^{1}}_{m}$ subdivide $P$ into six intervals that share their endpoints; Figures 9a and 9b show two possible configurations.

Recall that, by Lemma 8, the largest-area triangle $\Lambda$ must interleave with both ${T^{1}}_{a}$ and ${T^{1}}_{m}$. This implies that, once we fix the interval that contains one vertex of $\Lambda$, the other two vertices are constrained to lie in two pairwise disjoint intervals. Depending on the configuration, there could be either one or two sets of three compatible intervals; if there are two, they must interleave.

As a result, we construct either one or two smaller polygons $P^{\prime}$ or $P^{\prime}$ and $P^{\prime\prime}$ by directly connecting these intervals.

In the second step, we will repeat the above procedure again by finding another largest triangle $T_{a}^{2}$ rooted on an arbitrary vertex $a$ of $P^{\prime}$ (or $P^{\prime\prime}$), and another largest triangle $T_{m}^{2}$ rooted on the median vertex $m$ of the largest sub-interval of $P^{\prime}$ (or $P^{\prime\prime}$) induced by $T_{a}^{2}$.

Recursively repeating this, we show that for some subproblem $P^{\star}$ in step $i$ of the algorithm (note that in step $i$ there may be up to $2^{i}$ separate subproblems) if ${T^{i}}_{a}$ and ${T^{i}}_{m}$ are interleaving triangles in $P^{\star}$, we decompose the problem into two smaller subproblems, and if they are not interleaving, we get a single smaller subproblem. In all cases, the size of each subproblem is between $\frac{1}{6}$ and $\frac{5}{6}$ times the size of the previous subproblem, and in the case where we have two subproblems, the sum of their sizes equals the size of the previous subproblem plus $6$.

We will repeat the procedure of constructing dividing triangles in each step on one or two smaller polygons, until our subproblems become triangles themselves; in this case we simply return the area of the triangle. This procedure is outlined in Algorithm 3.

6.2 Time complexity

We start analyzing the time complexity of the algorithm with the following lemma.

Lemma 10**.**

Let $P$ be a convex polygon with $n$ vertices. The (one or two) subproblems induced by $P$ have size at most $\frac{5}{6}(n+6)$.

Proof.

The dividing triangles $T_{a}$ and $T_{m}$ decompose the boundary of $P$ into six intervals. Let $I_{1}$ and $I_{2}$ be the two intervals incident to $m$. Only one of $I_{1}$ or $I_{2}$ can include a vertex of the largest-area triangle, otherwise the largest-area triangle would no longer interleave both $T_{a}$ and $T_{m}$. Consider the $n-6$ (between $n-6$ and $n-4$) vertices not part of $T_{a}$ or $T_{m}$. Because of the choice of $m$, both $I_{1}$ and $I_{2}$ contain at least a factor $\frac{1}{6}$ of these vertices, so $\frac{1}{6}(n-6)$ each (see Figure 9b). Since one subproblem does not contain $I_{1}$, and the other subproblem does not contain $I_{2}$, each subproblem has size at most $n-\frac{1}{6}(n-6)=\frac{5}{6}(n+6)$. ∎

Note that, if $P$ splits into two subproblems $P^{\prime}$ and $P^{\prime\prime}$, then $|P^{\prime}|+|P^{\prime\prime}|\leq n+6$.

So the recursive equation of the divide-and-conquer algorithm is

[TABLE]

where $\frac{1}{6}\leq\alpha\leq\frac{5}{6}$, by Lemma 10.

By using $T(3)=1$ and considering the maximum part of the above equation, and using the method of Akra and Bazzi [27], the recursion can be written as

[TABLE]

and as $m$ is in $O(n)$, $T(n)$ is bounded by $O(n\log n)$.

6.3 Correctness

For the correctness proof of Algorithm 3 we will show that in each step of the recursion we always transfer all three vertices of the largest-area triangle $\Lambda$ to the same subproblem (or to both subproblems, if and only if $\Lambda=T_{a}$ or $\Lambda=T_{m}$).

Lemma 11**.**

*In each step of Algorithm 3, there is always at least one subproblem containing all three vertices of the largest-area triangle $\Lambda$. *

Proof.

Let $P$ be a convex polygon, and consider the dividing triangles $T_{a}$ and $T_{m}$. If $\Lambda=T_{a}$ or $\Lambda=T_{m}$, clearly both subproblems of $P$ contain all three vertices of $\Lambda$.

Otherwise, consider a subproblem $P^{\prime}$ and suppose it does not contain all three vertices $p$, $q$, and $z$ of $\Lambda$; say (w.l.o.g.) $P^{\prime}$ contains $p$ but not $z$. We know $\Lambda=\triangle pqz$ must interleave both ${T_{a}}$ and ${T_{m}}$. But then $z$ must be in $P^{\prime}$. This is only possible if $z$ is a vertex of $T_{a}$ or $T_{m}$. ∎

Theorem 2**.**

Algorithm 3 finds the largest-area triangle in $O(n\log n)$ time.

7 Largest-area quadrangle

Let $P$ be a convex polygon with $n$ vertices. In the reminder, we denote by $\Lambda_{P,k}$ the largest (by area) $P$-aligned polygon with $k$ vertices. Also we denote $Q_{p,k}$ for $P$-aligned polygons with $k$ vertices. A polygon $Q$ is $k$-stable where it has $k$ stable vertices.

Note that all the vertices of $\Lambda_{P,k}$ are stable, but a $k$-stable $Q_{P,k}$ does not necessarily coincides with the $\Lambda_{P,k}$, as illustrated in Figure 11.

Indeed the idea of the presented method [1] was based on starting with a rooted $Q_{P,k}$ and moving the vertices of $Q_{P,k}$ around the given polygon $P$ where keeping the cyclic ordering of $Q_{P,k}$ and increasing the area, and updating the area while finding a larger $k-1$-stable rooted polygon.

This procedure will result in keeping the sequence of the area of the potential solution only increasing. The authors [1] named this attribute as the unimodality of the area, but we illustrated in Figure 13 that keeping the unimodality will not result in finding the optimal solution necessarily.

7.1 Dobkin and Snyder’s algorithm for $k=4$

We will now recall the quadrilateral algorithm [1], that is outlined in Algorithm 4 and illustrated in Figure 10.

Let $P=\{p_{0},p_{1},\ldots,p_{n-1}\}$. We assume that $P$ is given in a counter clockwise orientation. Assume an arbitrary vertex of $P$ is is the root of the algorithm, assign this vertex and its three subsequent vertices in the counter clockwise order on the boundary of $P$ to variables $a,b$, $c$ and $d$. We then “move $d$ forward” along the boundary of $P$ as long as this increases the area of $abcd$.

If we can no longer advance $d$, we advance $c$ if this increases the area of $abcd$, then try again to advance $d$. If we can no longer advance $d$ and $c$, we advance $b$ if this increases the area of $abcd$, then try again to advance $d$ and $c$.

If we cannot advance either $d$, $c$ or $b$ any further, we advance $a$. We keep track of the largest-area quadrilateral found, and stop when $a$ returns to the starting position. Since $a$ visits $n$ vertices and $d$, $c$ and $b$ each visit fewer than $2n$ vertices, the algorithm runs in $O(n)$ time (assuming we are given the cyclic ordering of the points on $P$).

Indeed, Algorithm 4 is based on an observation that the largest inscribed quadrilateral treats as a unimodal function, which is not correct. But, of course there is an quadrilateral with some vertices to be stable, that Algorithm 4 will find it in linear-time, but our counter example shows that the reported quadrilateral does not necessarily equal to $\Lambda_{4,P}$. Furthermore, the reported quadrilateral is not even 4-stable.

7.2 Counter-example to Algorithm 4

In Figure 12 we provide a polygon $P$ on $16$ vertices such that $\Lambda_{4,P}$ and the largest-area inscribed quadrilateral computed by Algorithm 4 are not the same.

We use the following points: $a_{1}=(26096,06750),a_{2}=(26130,9933),a_{3}=(25940,10728),a_{4}=(23090,22189),a_{5}=(18106,23681),a_{6}=(13484,24407),$ $a_{7}=(13174,24343),a_{8}=(3090,22189),a_{9}=(0,17308),a_{10}=(80,14350),a_{11}=(323,13331),a_{12}=(3090,2189),a_{13}=(8459,385),a_{14}=(12837,0),$ $a_{15}=(13392,114),a_{16}=(23090,2189)$. The largest-area quadrilateral is $a_{4}a_{8}a_{12}a_{16}$; however, Algorithm 4 reports $a_{1}a_{4}a_{8}a_{12}$ as the largest-area quadrilateral, while starting the algorithm from an arbitrary root. The results of running Algorithm 4 while starting on root $a_{1}$ are demonstrated on Figure 13. Thus, the algorithm fails to find $\Lambda_{P,4}$ on any possible root.

8 Implications

Our discovery directly or indirectly affects the results of the following studies.

8.1 Largest-area $k$-gon inscribed in a convex polygon

As mentioned, the Dynamic Programming method presented by Boyce et al. [1] for finding the largest-area $k$-gon starts looking for the optimal answer from the largest-area rooted triangle. Aggarwal et al. [12] improve their result to $O(kn+n\log n)$ time by using a matrix search method. The method that Boyce et al. [1] use to find the largest-area rooted triangle is again based on the assumption that there is only one stable triangle on each vertex of the polygon. So, they start their algorithm from a non-optimal answer. Also in their algorithm, they compute $k$ intervals and they look for one point per interval, and these intervals are computed according to the starting situation. So they fail to find the largest area $k$-gon that is inscribed in a convex polygon.

In Lemma 9 we proved that the globally largest-area rooted triangle can be found in linear time. As such, it is relatively straightforward to correct their algorithm by changing the first step of their algorithm.

8.2 Largest-Area Triangle inscribed in a set of imprecise points

Keikha et al. [25] consider the question of finding bounds on the area of the largest-area triangle in a set of imprecise points: points that are known to be in a given region in space. Their algorithm for computing the largest-area triangle on a given set of imprecise points modeled by parallel line segments was based on the Dobkin and Snyder algorithm [1], and thus also fails to find the optimal answer. Two cases in particular are impacted. When the imprecise points are modeled as unit-length segments, their algorithm directly applies the largest-area triangle algorithm. This algorithm can easily be fixed by using our new divide-and-conquer algorithm instead, but this will result in a running time of $O(n\log n)$, while a running time of $O(n)$ was reported in [25]. In contrast, when the imprecise points are modeled as segments of arbitrary length, their solution does not directly apply the largest-triangle algorithm but is rather based on the stability of certain triangles. With a slightly extended analysis, the reported running time of $O(n^{2})$ can still be achieved.

8.3 Convex hull of a simple polygon

Bhattacharia et al. [26] applied the idea of unimodality of the vertical distance of a moving vertex on the boundary of a convex polygon from one of its edges to compute the convex hull of a simple polygon $P$ in linear time. Specifically, they use the fact that there is one vertex $p_{j}$ on a convex polygon with maximum distance from an edge $p_{1}p_{k}$ of $P$. Using this fact, they decompose their problem into two subproblems by computing two half convex chains, one chain starting from $p_{1}$ and ending at $p_{j}$, and the other starting from $p_{k}$ and ending at $p_{j}$. On each subproblem, they use a stack $S$ which stores some vertices, such that the triangles consisting of one vertex of $S$ and a fixed edge of $P$ are only increasing in the order in which they are stored in the stack. But as there is only one non-fixed vertex (for considering the area of a triangle with a fixed base) in each step of the algorithm [26], the application of the idea of changing the area of an inscribed triangle by moving only one of its vertices [1] is still correct. Therefore, the correctness of the results of [26] is not impacted by our discovery.

8.4 Critical Triangle

Kallus [28] applied the Pentagon Lemma [1] to compute the critical triangle $T$ in a convex compact subset of $\mathcal{R}^{2}$; $K$, where the critical triangle is an inscribed triangle with maximum area that the critical determinant of $K$ is equal to twice the area of $T$. As the Pentagon Lemma [1] is still correct, the correctness of the results of [28] is not impacted by our discovery. They called $K$ is extensible if there is a domain $K^{\prime}$ containing $K$ but different from it that has the same critical determinant as $K$. Otherwise, $K$ is inextensible.

9 Discussion

To summarize, we disproved the linear-time algorithm presented by Dobkin and Snyder [1] for computing the largest-area inscribed triangle by presenting a $9$-vertex polygon, on which the algorithm fails to output the optimal solution. Dobkin and Schnyder’s algorithm also fails to find the largest-area 4-gon in a convex polygon. For $k=2$ and for $k\geq 5$, it was already known that their algorithm fails, so this now conclusively shows that the algorithm is wrong for all possible valies of k.

We also presented a divide-and-conquer algorithm for computing the largest inscribed $k$-gon which runs in $O(n\log n)$ time. Our findings reopen the question of whether the largest-area quadrangle in a convex polygon can be found in linear time.

There remains a significant gap between the best know algorithm by Boyce et al. [11], which runs in $O(kn+n\log n)$ time, and the lower bound of $\Omega(n\log n)$ [12].

Acknowledgments

The authors would like to thank everybody who has discussed this discovery and its implications with them over the past months, in particular Bahareh Banyassady, Ahmad Biniaz, Prosenjit Bose, Yoav Kallus, Kshitiz Kumar, Manish Kumar Bera, Benjamin Raichel, Lena Schlipf, and the participants of Dagstughl Seminar 17171 on Computional Geometry, for their useful comments and insights. Maarten Löffler was partially supported by the Netherlands Organization for Scientific Research (NWO) under project no. 614.001.504.