On Colourability of Polygon Visibility Graphs

TL;DR

This paper investigates the complexity of colouring visibility graphs of polygons, providing polynomial algorithms for 4-colouring simple polygons and proving NP-completeness for 5-colouring and polygons with holes.

Contribution

It introduces polynomial algorithms for 4-colouring simple polygons and establishes NP-completeness results for 5-colouring and polygons with holes.

Findings

Polynomial-time algorithm for 4-colouring simple polygons

NP-completeness of 5-colouring simple polygons

NP-completeness of 4-colouring polygons with holes

Abstract

We study the problem of colouring visibility graphs of polygons. In particular, for visibility graphs of simple polygons, we provide a polynomial algorithm for 4-colouring, and prove that the 5-colourability question is already NP-complete for them. For visibility graphs of polygons with holes, we prove that the 4-colourability question is NP-complete.

Equations

$$H$$

$$p_{ij}^1$$

$$a_1$$

Full text

{AMS}

68Q25, 68R10, 68U05

1 Introduction

Visibility graphs are widely studied graph classes in computational geometry. Geometric sets such as sets of points or line segments, polygons, polygons with obstacles, etc., all can correspond to specific visibility graphs, and have uses in robotics, signal processing, security paradigms, decomposing shapes into clusters [1, 2, 12, 8, 16]. We study the visibility graphs of simple polygons in the Euclidean plane, but we also mention polygons with (again polygonal) holes in Section 4. To make things clear, all polygons in the paper are simple unless stated otherwise.

Given an $n$-vertex polygon $P$ (not necessarily convex) in the plane, two points $p$ and $q$ of $P$ are said to be mutually visible if, and only if the line segment $\overline{pq}$ does not intersect the exterior of $P$. The $n$-vertex visibility graph $G(V,E)$ of $P$ is defined as follows. The vertex set $V$ of $G$ contains a vertex $v_{i}$ if, and only if, the polygon $P$ contains the point $p_{i}$ as its vertex. The edge set $E$ of $G$ contains an edge $\{v_{i},v_{j}\}$ if, and only if, the points $p_{i}$ and $p_{j}$ are mutually visible. Given a polygon $P$ in the plane, we can compute its visibility graph $G$ in $\mathcal{O}(n^{2})$ time using the polygon triangulation method [17, 9]. Hence, in this paper, we slightly abuse notation by not distinguishing between a polygon $P$ and its visibility graph $G$ and referring to a polygon vertex $p_{i}$ as to the corresponding $G$-vertex $v_{i}$.

Visibility graphs of polygons have been studied with respect to various theoretical and practical computational problems. The complexities of several popular optimization problems have been determined for visibility graphs of polygons. A geometric variation of the dominating set problem, namely polygon guarding, is one of the most studied problems in computational geometry and is known as the Art Gallery Problem [16]. It has been studied extensively for both polygons with and without holes and has been found to be NP-hard in both cases [13, 18]. Besides, given a polygon, computing a maximum independent set is known to be hard, due to Shermer [21] (see also [14] for other problems), while computing a maximum clique has been shown to be in polynomial time by Ghosh et al. [20].

A proper vertex colouring of a graph is an assignment of labels or colours to the vertices of the graph so that no two adjacent vertices have the same colours. Henceforth, when we say colouring a graph, we refer to proper vertex colouring. The chromatic number of a graph is defined as the minimum number of colours used in any proper colouring of the graph. Visibility graph colouring has been studied for various types of visibility graphs. Babbitt et al. gave upper bounds for the chromatic numbers of k-visibility graphs of arcs and segments [3]. Kára et al. characterized 3-colourable visibility graphs of point sets and described a super-polynomial lower bound on the chromatic number with respect to the clique number of visibility graphs of point sets [11]. Pfender showed that, as for general graphs, the chromatic number of visibility graphs of point sets is also not upper-bounded by their clique numbers [19]. Diwan and Roy showed that for visibility graphs of point sets, the 5-colouring problem is NP-hard, but 4-colouring is solvable in polynomial time [6].

The problem of colouring the visibility graphs of given polygons has been studied in the special context where each internal point of the polygon is seen by a vertex, whose colour appears exactly once among the vertices visible to that point [4, 10, 7]. However, little is known on colouring visibility graphs of polygons without such constraints. Although 3-colouring is NP-hard for general graphs [15], in particular it is rather trivial to solve it for visibility graphs of polygons in polynomial time using a greedy approach. With $4$ colours the same question has been open so far (precisely, until the conference paper [5]).

In this paper we completely settle the complexity question of the general problem of colouring polygonal visibility graphs, which was declared open in 1995 by Lin and Skiena [14]. In Section 2, we provide a polynomial-time algorithm to find a $4$-colouring of a given graph $G$ with the promise that $G$ is the visibility graph of some polygon, if $G$ is indeed $4$-colourable. On the other hand, in Section 3 we provide a reduction showing that the question of $k$-colourability of the visibility graph of a given simple polygon is NP-complete for any $k\geq 5$. We remark that in the conference version of this paper [5] we used a different reduction showing hardness only for $k\geq 6$. In Section 4, we additionally show that already the question of $4$-colourability of visibility graphs of polygons with holes is an NP-complete problem.

2 4-Colouring visibility graphs

In this section, we study the algorithmic question of $4$-colourability of the visibility graph of a given polygon. The full structure of $4$-colourable visibility graphs is not yet known and it seems to be non-trivial. For instance, if a visibility graph is planar, it is obviously $4$-colourable. Though, if such a graph contains $K_{5}$, then it is neither planar nor $4$-colourable, but a visibility graph not containing any $K_{5}$ may be non-planar yet $4$-colourable (Figure 1).

The related algorithmic problem of 3-colouring visibility graphs is rather easy to resolve as follows. Every simple polygon can be triangulated and, in such a triangulation, every non-boundary edge is contained in two triangles. One can then proceed greedily edge by edge: Suppose a triangle has already been coloured, and it shares an edge with a triangle that is not fully coloured. Then the two end vertices of the shared edge uniquely determine the colour of the third vertex of the uncoloured triangle.

Our algorithm essentially generalizes the 3-colouring method for 4-colouring. We first divide the polygon into reduced polygons. A polygon $P$ is called a reduced polygon, if every chord of $P$ (i.e., an internal diagonal) is intersected by another chord of $P$. After the division, we find and colour in each reduced subpolygon a triangle (a $K_{3}$ subgraph) with three distinct colours. Subsequently, whenever we find an uncoloured vertex $v$ adjacent to some three vertices coloured with three distinct colours (such as, to an already coloured triangle), we can uniquely colour also $v$, by the fourth colour. We will show that we can exhaust all vertices of a reduced subpolygon in this manner. Furthermore, we check for possible colouring conflicts – since the colouring process is unique, this suffices to solve $4$-colourability.

Altogether, this will lead to the following theorem.

Theorem 2.1**.**

*The 4-colourability problem is solvable in polynomial time for visibility graphs of simple polygons, and if a 4-colouring exists, then it can be computed in polynomial time from the given input graph (even without a visibility representation). *

2.1 Unique 4-colouring of reduced polygons

We first prove that if a reduced polygon is 4-colourable, then the 4-colouring is unique up to a permutation of colours. In the coming proof, consider a polygon $P$ and its visibility graph $G(V,E)$, embedded on $P$. Hereafter we slightly abuse notation by equating $P$ and $G$. Since we want to 4-colour $P$, we assume that $G$ has no $K_{5}$ (or we answer ‘no’). We denote the clockwise polygonal chain of $P$ from a vertex $u$ to a vertex $v$ as $\Gamma(u,v)$.

One can easily see that it is enough to focus on reduced $P$ in our proofs. Indeed, assume an edge $uv$ of $G$ which is a chord of $P$ and not crossed by any other chord. We can partition $P$ into subpolygons $P_{1}$ and $P_{2}$, where $P_{1}=(u\,\Gamma(u,v)\,v)$ and $P_{2}=(v\,\Gamma(v,u)\,u)$. Since no edge of $G$ has one end in $P_{1}\setminus P_{2}$ and the other in $P_{2}\setminus P_{1}$, the polygons $P_{1}$ and $P_{2}$ can be $4$-coloured separately and merged again (provided that $P_{1}$ and $P_{2}$ are $4$-colourable).

Let $u$ and $v$ be two vertices of $P$. The shortest path between $u$ and $v$ is a (graph) path from $u$ to $v$ in $G$ such that the sum of the Euclidean lengths of its edges is minimized. Such a shortest path between $u$ and $v$ is unique in $P$ and is denoted as $\Pi(u,v)$. Observe that all non-terminal vertices of a shortest path are non-convex [8]. We will assume an implicit ordering of vertices on $\Pi(u,v)$ from $u$ to $v$. When we say that some vertex $w$ is the first (or last) vertex on $\Pi(u,v)$ with a certain property, we mean that $w$ precedes (respectively, succeeds) all other vertices with that property on $\Pi(u,v)$.

For a proof of Theorem 2.1, we have got the following sequence of claims. Consider, in all of them, a $K_{5}$-free reduced polygon $P$ and its three vertices $t_{1},t_{2},t_{3}$ forming a triangle $T\subseteq G$. Assume that $T$ is already coloured (which is unique up to a permutation of the colours). Suppose that $v_{i}$ is an uncoloured vertex, such that an edge incident to $v_{i}$ intersects $T$. Then we have the following lemmas.

Lemma 2.2**.**

*Assume that two vertices $v_{i}\in\Gamma(t_{1},t_{2})$ and $v_{j}\in\Gamma(t_{2},t_{3})$ see each other, and the edge $v_{i}v_{j}$ intersects $t_{1}t_{2}$ and $t_{2}t_{3}$. Then the colours of all vertices on the four paths $\Pi(t_{1},v_{i})$, $\Pi(t_{2},v_{i})$, $\Pi(t_{2},v_{j})$ and $\Pi(t_{3},v_{j})$, including $v_{i},v_{j}$ themselves, are uniquely determined by the colours of $T$. *

Proof 2.3**.**

We prove the claim by induction on the four paths. As the base case, the first vertices of these paths are the vertices of $T$, which are already assigned different colours.

For the induction step, assume that $\Pi(t_{1},v_{i})$, $\Pi(t_{2},v_{i})$, $\Pi(t_{2},v_{j})$ and $\Pi(t_{3},v_{j})$ have been coloured till vertices $v_{a}$, $v_{b}$, $v_{c}$ and $v_{d}$ respectively. Also, their immediate uncoloured successors on $\Pi(t_{1},v_{i})$, $\Pi(t_{2},v_{i})$, $\Pi(t_{2},v_{j})$ and $\Pi(t_{3},v_{j})$ are $v_{p}$, $v_{q}$, $v_{r}$ and $v_{s}$ respectively. We aim to show that the colours of at least one of $v_{p}$, $v_{q}$, $v_{r}$ and $v_{s}$ is uniquely determined by the already coloured vertices.

We have the following cases (cf. Figure 2).

Case 1: $v_{p}$ sees $v_{b}$ or some predecessor of $v_{b}$ on $\Pi(t_{2},v_{i})$

By definition, $v_{p}$ is the immediate successor of $v_{a}$ on $\Pi(t_{1},v_{i})$, so $v_{p}$ must see $v_{a}$. The right tangent of $v_{a}$ to $\Pi(t_{2},v_{i})$ lies to the right of the right tangent of $v_{p}$ to $\Pi(t_{2},v_{i})$. So, if the right tangent of $v_{p}$ to $\Pi(t_{2},v_{i})$ touches $\Pi(t_{2},v_{i})$ at a vertex $v_{u}$, then $v_{a}$ sees $v_{u}$. Note that either $v_{u}=v_{b}$ or $v_{u}$ precedes $v_{b}$ on $\Pi(t_{2},v_{i})$. In any case, $v_{u}$ is already coloured. Since $v_{p}$, $v_{a}$ and $\Pi(t_{3},v_{j})$ lie on the same side of $v_{i}v_{j}$, and $v_{p}$ is nearer to $v_{i}v_{j}$ than $v_{a}$ is, $v_{p}$ and $v_{a}$ see a vertex $v_{t}$ of $\Pi(t_{3},v_{j})$. If $v_{u}$ also sees $v_{t}$, and $v_{t}$ is already coloured, then the claim is proved (Figure 2(a)). So we consider the other two cases, namely, that $v_{u}$ does not see $v_{t}$, or that $v_{t}$ is not yet coloured.

Subcase 1.a: $v_{u}$ does not see $v_{t}$

Since $v_{t}$ and $v_{u}$ lie on different sides of $\overline{v_{i}v_{j}}$ and of $\overline{t_{2}t_{3}}$, some vertex of $\Pi(t_{2},v_{j})$ must be blocking $v_{u}$ and $v_{t}$. Let $v_{w}$ be the first vertex of $\Pi(t_{2},v_{j})$ blocking $v_{u}$ and $v_{t}$. Then $v_{u}$ sees $v_{w}$. The vertex $v_{w}$ is closer to $\overline{v_{i}v_{j}}$ than $v_{u}$ is. Also, $v_{w}$ lies to the right of $\overrightarrow{v_{a}v_{u}}$ and $\overrightarrow{v_{p}v_{u}}$, and to the left of $\overrightarrow{v_{a}v_{t}}$ and $\overrightarrow{v_{p}v_{t}}$. Then the only possible blockers between $v_{w}$ and $v_{p}$ or $v_{a}$ can be from $\Pi(t_{2},v_{i})$. But all the vertices on $\Pi(t_{2},v_{i})$ preceding $v_{u}$ are farther from $v_{i}v_{j}$ than $v_{u}$ is. So, there can be no such blocker, and $v_{w}$ must be visible from both $v_{a}$ and $v_{p}$ (Figure 2(b)). If $v_{w}$ is already coloured, then the claim is proved. Suppose that $v_{w}$ is not yet coloured. Then consider $v_{r}$, which now precedes $v_{w}$ on $\Pi(t_{2},v_{j})$. The vertices $v_{r}$ and $v_{c}$ are consecutive on $\Pi(t_{2},v_{j})$ and hence see each other. Since $\Pi(t_{2},v_{j})$ and $\Pi(t_{1},v_{i})$ are on opposite sides of $v_{i}v_{j}$, the vertices $v_{c}$ and $v_{r}$ both see $v_{a}$ or some vertex preceding $v_{a}$ on $\Pi(t_{1},v_{i})$. Let $v_{x}$ be the last coloured vertex of $\Pi(t_{1},v_{i})$ seen by both $v_{c}$ and $v_{r}$. If $v_{x}\neq v_{a}$ then let $v_{y}$ be the last vertex of $\Pi(t_{2},v_{i})$ that blocks $v_{c}$ from the successor of $v_{y}$ on $\Pi(t_{1},v_{i})$. Then $v_{y}$ must be visible from $v_{x}$, $v_{r}$ and $v_{c}$. Since $v_{x}$ precedes $v_{a}$ on $\Pi(t_{1},v_{i})$, and $v_{y}$ precedes $v_{b}$ on $\Pi(t_{2},v_{i})$, both $v_{x}$ and $v_{y}$ must be already coloured. So, $T$ uniquely determines the colour of $v_{r}$. If $v_{x}=v_{a}$ then since $v_{u}$ is on the right tangent of $v_{a}$ to $\Pi(t_{2},v_{i})$, both $v_{c}$ and $v_{r}$ see $v_{u}$. Hence, $T$ uniquely determines the colour of $v_{r}$. Now we move to the second subcase.

Subcase 1.b: $v_{u}$ sees $v_{t}$, but $v_{t}$ is not yet coloured

Since $v_{u}$ sees $v_{t}$, $\Pi(t_{2},v_{j})$ is a concave chain and the edge $t_{1}t_{3}$ exists in $P$, $v_{u}$ must see every predecessor of $v_{t}$ on $\Pi(t_{2},v_{j})$. This means that both $v_{d}$ and $v_{s}$ see $v_{u}$ (Figure 2(c)). Let the right tangent from $v_{d}$ touches $\Pi(t_{1},v_{i})$ in a vertex $v_{y}$. Then $v_{s}$ must see $v_{y}$, because the last vertices $v_{i}$ and $v_{j}$ of concave chains $\Pi(t_{1},v_{i})$ and $\Pi(t_{3},v_{j})$ see each other. Also, the left tangent of $v_{u}$ to $\Pi(t_{1},v_{i})$ must touch $\Pi(t_{1},v_{i})$ at a vertex equal to or preceding $v_{y}$. Thus, all three of $v_{s}$, $v_{d}$ and $v_{u}$ see a common vertex $v_{z}$ on $\Pi(t_{1},v_{i})$ which precedes $v_{a}$, since $v_{u}$ and $v_{t}$ see $v_{a}$. Thus, $v_{z}$ is already coloured, and $v_{u}$, $v_{d}$ and $v_{z}$ form a $K_{4}$ with $v_{s}$ and uniquely determine the colour of $v_{s}$.

Case 2: $v_{p}$ does not see $v_{b}$ or any predecessor of $v_{b}$ on $\Pi(t_{2},v_{i})$

Since $\Pi(t_{2},v_{i})$ is a concave chain, this means that the tangent drawn from $v_{p}$ to $\Pi(t_{2},v_{i})$ in the direction of $t_{2}$, has whole $\Pi(t_{2},v_{b})$ to its left (refer to Figure 2(d)). Suppose that $v_{b}$ does not see $v_{a}$ or some other vertex of $\Pi(t_{1},v_{a})$. Since also $\Pi(t_{1},v_{i})$ is a concave chain, and $t_{1}$ sees $t_{2}$, all blockers between $v_{q}$ and $\Pi(t_{1},v_{a})$ must come from $\Pi(v_{p},v_{i})$, and must include $v_{p}$. But then, the aforementioned tangent drawn from $v_{p}$ to $\Pi(t_{2},v_{i})$ must have at least part of $\Pi(t_{2},v_{b})$ to the right, which is absurd.

So, $v_{b}$ must see $v_{a}$ or some other vertex of $\Pi(t_{1},v_{a})$. Let $v_{g}$ denote the last vertex of $\Pi(t_{1},v_{i})$ seen by $v_{b}$. Then $v_{g}$ exists, it belongs to $\Pi(t_{1},v_{a})$ since $v_{p}$ (the successor of $v_{a}$) does not see $v_{b}$, and $v_{g}$ is seen by $v_{q}$ (Figure 2(d)). Since the vertex $v_{g}$ is on $\Pi(t_{1},v_{a})$, it is already coloured.

*Let us now similarly consider a vertex, say $v_{t}$ on $\Pi(t_{2},v_{j})$, which is seen by both $v_{b}$ and $v_{q}$. Suppose that $v_{g}$ or other common coloured neighbour of $v_{b}$ and $v_{q}$ sees $v_{t}$. Then we are immediately done if $v_{t}$ is coloured, or we are in Subcase 1.b if $v_{t}$ is uncoloured. Otherwise, some vertex on $\Pi(t_{3},v_{j})$ blocks all visibilities between $v_{t}$ and all the common neighbours of $v_{q}$ and $v_{b}$. Then we finish as in Subcase 1.a. *

Corollary 2.4**.**

*If any vertex $v_{i}$ of $P$ sees a vertex of $T$ and their visibility edge crosses one of the edges of $T$, then the colour of $v_{i}$ is uniquely determined by the colours of $T$. *

Proof 2.5**.**

*Without loss of generality, suppose that $v_{i}$ sees $t_{1}$, and $v_{i}t_{1}$ crosses $t_{2}t_{3}$. Then $v_{j}=t_{1}$, $\Pi(t_{2},v_{j})=t_{2}t_{1}$ and $\Pi(t_{1},v_{j})=t_{1}$, and Lemma 2.2 proves the claim. *

Theorem 2.6**.**

*If a reduced polygon is 4-colourable, then it has a unique 4-colouring up to a permutation of colours. *

Proof 2.7**.**

Consider a triangle $T$ in a reduced polygon $P$. If $P$ is not just $T$, then at least one edge of $T$ is not a boundary edge of $P$. Without loss of generality, let $t_{1}t_{2}$ be such an edge. Since $P$ is reduced, there must be a vertex $v_{i}$ on the boundary chain $\Gamma(t_{1},t_{2})$ such that an edge incident to $v_{i}$ crosses $t_{1}t_{2}$. By Lemma 2.2 and Corollary 2.4, if $P$ is 4-colourable, then all vertices on the shortest paths $\Pi(t_{1},v_{i})$ and $\Pi(t_{2},v_{i})$, including $v_{i}$, have a 4-colouring uniquely determined by $T$. In case $t_{2}t_{3}$ or $t_{3}t_{1}$ are not boundary edges of $P$, we can similarly find $v_{j}$ on $\Gamma(t_{2},t_{3})$ and $v_{k}$ on $\Gamma(t_{3},t_{1})$ and uniquely 4-colour $\Pi(t_{2},v_{j})$, $\Pi(t_{3},v_{j})$, $\Pi(t_{3},v_{k})$ and $\Pi(t_{1},v_{k})$.

Now, all the remaining uncoloured vertices of $P$ are on boundary chains of the form $\Gamma(v_{a},v_{b})$, where $v_{a}$ and $v_{b}$ are two consecutive vertices in one of the six paths mentioned above. Furthermore, no vertex in the polygonal chain $\Gamma(v_{a},v_{b})$, other than $v_{a}$ and $v_{b}$, is coloured. Without loss of generality, let $v_{a}$ and $v_{b}$ be two consecutive vertices on $\Pi(t_{1},t_{2})$. If $v_{a}v_{b}$ is not a boundary edge of $P$, then since $P$ is reduced, there must be an uncoloured vertex $v_{u}$ in $\Gamma(v_{a},v_{b})$ such that an edge incident to $v_{u}$ crosses $v_{a}v_{b}$. This edge is either incident to a vertex of $\Pi(t_{2},v_{i})$, or crosses an edge of $\Pi(t_{2},v_{i})$.

Consider the case where such an edge from $v_{u}$ to a vertex of $\Pi(t_{2},v_{i})$ exists. Then consider a vertex $v_{w}$ that is closest to $\overline{v_{a}v_{b}}$ among all the vertices of $\Pi(t_{2},v_{i})$ that see an internal vertex (say, $v_{z}$) of $\Gamma(v_{a},v_{b})$ (top of Figure 3). Since the edge $v_{w}v_{z}$ exists, $v_{w}$ cannot be blocked by any vertex of $\Pi(t_{1},v_{i})$. Due to the choice of $v_{w}$, no vertex of $\Pi(t_{2},v_{i})$ can block $v_{w}$ from $v_{a}$ or $v_{b}$. So, $v_{w}$ sees both $v_{a}$ and $v_{b}$. Then, based on the triangle $v_{a}v_{b}v_{w}$, Lemma 2.2 and Corollary 2.4 can be used to uniquely determine a $4$-colouring for $\Pi(v_{a},v_{z})$ and $\Pi(v_{b},v_{z})$.

Now consider the case in which $v_{u}$ does not see any vertex of $\Pi(t_{2},v_{i})$, but an edge incident to $v_{u}$ crosses an edge $v_{c}v_{d}$ of $\Pi(t_{2},v_{i})$, where $v_{c}$ precedes $v_{d}$ (bottom of Figure 3). Then $v_{c}$ (as well as $v_{d}$) must see both $v_{a}$ and $v_{b}$, since there cannot be any blockers in $\Gamma(v_{a},v_{b})$ or $\Gamma(v_{c},v_{d})$ which are not on $\Pi(t_{1},v_{i})$ or $\Pi(t_{2},v_{i})$. Again, based on the triangle $v_{a}v_{b}v_{c}$, Lemma 2.2 can be used to uniquely determine a $4$-colouring for $\Pi(v_{a},v_{u})$ and $\Pi(v_{b},v_{u})$.

*Now we recurse the above procedure. Let $T_{1}=\{T\}$, and let $S_{1}=\{\Pi(t_{1},v_{i}),$ $\Pi(t_{2},v_{i}),\Pi(t_{2},v_{j}),\Pi(t_{3},v_{j}),\Pi(t_{3},v_{k}),\Pi(t_{1},v_{k})\}$. Note that we have assumed that none of the edges of $T$ are boundary edges. If some edges of $T$ are boundary edges then $S_{1}$ will have less elements. By the above procedure, we can uniquely 4-colour all vertices of all paths of $S_{1}$. Then, all the uncoloured vertices $v_{u}$ lie on $\Gamma(v_{a},v_{b})$, where $v_{a}$ and $v_{b}$ are consecutive vertices of some path of $S_{1}$. For each such $v_{a}v_{b}$, we find a new triangle based on $v_{a}v_{b}$ as above, and two new paths of the form $\Pi(v_{a},v_{u})$ and $\Pi(v_{b},v_{u})$. Let $T_{2}$ denote the set of all such new triangles, and $S_{2}$ denote the set of all newly coloured shortest paths obtained this way. In general, following the same method we can always construct $T_{i+1}$ and $S_{i+1}$ from $T_{i}$ and $S_{i}$, until all vertices of $P$ are coloured. Since in each step, the colours of vertices are uniquely determined, it follows that if $P$ has a 4-colouring, then it must be unique. *

2.2 Computing a 4-colouring without polygonal representation

In the previous section, we have proved that if a reduced polygon is 4-colourable, then its 4-colouring must be unique up to permutations. Now we use the property to derive a polynomial time 4-colouring algorithms for the visibility graph of a polygon, even when the polygonal embedding or boundary are not given. First we need to define a few structures and operations.

Definition 2.8**.**

*Call a pair of adjacent vertices whose removal disconnects a given graph $G$, a bottleneck pair. Consider removing all the bottleneck pairs from $G$. We are left with connected components of $G$. Now, consider any bottleneck pair $(x,y)$. Suppose that $x$ and $y$ were earlier adjacent to a set of vertices $S_{x}$ and $S_{y}$ of a connected component $C_{i}$. Then create a copy of $(x,y)$ and re-connect them with edges with the vertices of $S_{x}$ and $S_{y}$ respectively. Do this with every bottleneck pair of $G$. Call the subgraphs of $G$ so formed as reduced subgraphs of $G$. *

We have the following lemma.

Lemma 2.9**.**

*Let $G$ be the visibility graph of a polygon $P$. Each bottleneck pair of $G$ corresponds to an internal edge of $P$ that is not intersected by any other internal edge of $P$, and vice versa. *

Proof 2.10**.**

*We use the same notations for the vertices of $G$ and their corresponding vertices of $P$. Consider any internal edge $xy$ of $P$ such that no other internal edge of $P$ intersects it. Then disconnecting the edge $xy$ and the vertices $x$ and $y$ disconnects $G$. So $(x,y)$ is a bottleneck pair. Conversely, suppose that $(x,y)$ is a bottleneck pair. Then $xy$ is an internal edge of $P$ since deleting a boundary edge does not disconnect $G$. Let $P_{1}$ and $P_{2}$ be the two subpolygons of $P$ divided by $xy$. If there was a visibility edge from $P_{1}$ to $P_{2}$ not incident to $x,y$, then since the visibility graphs of $P_{1}$ and of $P_{2}$ are connected, deleting $xy$ would again not disconnect $G$. So, is an internal edge of $P$ not intersected by any other internal edge of $P$. *

The corollary below follows immediately from Lemma 2.9.

Corollary 2.11**.**

*Each reduced subgraph of $G$ is the visibility graph of some reduced subpolygon of $P$. Likewise, each reduced subpolygon of $P$ has a reduced subgraph of $G$ as its visibility graph. *

Now, in light of the above Algorithm 1 and Theorem 2.6, we prove Theorem 2.1.

Proof 2.12** (Proof of Theorem 2.1).**

Corollary 2.4 shows that the reduced subgraphs correspond to reduced polygons. By Theorem 2.6 (and its proof), 4-colourable reduced polygons have unique 4-colourings which can be found iteratively by colouring each time a vertex with some three previously distinctly coloured neighbours. Since the algorithm always chooses a colour for a vertex by this iterative scheme, the computed (partial) 4-colouring is the only one possible. So, the algorithm is correct.

*Let the number of vertices and edges in $G$ be $n$ and $m$ respectively. The bottleneck pairs that do not cross any other chord, can be found in $O(m^{2})$ time. Thus, the decomposition of $P$ into reduced subpolygons takes $O(m^{2})$ time. A vertex adjacent to every vertex of a coloured triangle can be found in $O(n)$ time. While computing the colouring on the shortest paths, a pointer can be kept on each of the shortest paths, and the colouring takes $O(n)$ time. The colouring step can be iterated at most once for each vertex, so the complexity for all vertices is $O(n^{2})$. Checking for conflict takes $O(m)$ time. Finally, rejoining the reduced subgraphs takes $O(n)$ time. Thus, the complexity of the algorithm is $O(m^{2})$. *

3 Hardness of 5-colourability

In this section we prove that the problem of deciding whether the visibility graph $G$ of a given simple polygon $P$ can be properly coloured with $5$ colours, is NP-complete.

Membership of our problem in NP is trivial (since $G$ can be efficiently computed from $P$ and then a colouring checked on $G$). We are going to present a polynomial reduction from the NP-hard problem of $3$-colourability of general graphs. Our reduction shares some common ideas with reductions on visibility graphs presented in [14, 5], but the main difference is in not using the SAT problem (which makes our case even simpler). The rough outline of the reduction is depicted in Figure 4.

Theorem 3.1**.**

*The problem – given a simple polygon $P$ in the plane, to decide whether the visibility graph of $P$ is properly $k$-colourable – is NP-complete for every $k\geq 5$. *

Proof 3.2**.**

As mentioned, the problem is in NP since one can construct the visibility graph $G$ of $P$ in polynomial time [17, 9] and then verify a colouring. In the opposite direction, we reduce from the NP-complete problem of $3$-colouring a given graph $H$.

Let $V(H)=\{v_{1},\ldots,v_{n}\}$. The polygon $P$ constructed from $H$ is shaped as in Figure 4. The top chain of $P$ consists of $3n+2$ vertices in a sawtooth configuration, such that the convex vertices of the teeth are marked by $v_{1},\ldots,v_{n}$. The picture is scaled such that each $v_{i}$ sees the whole bottom chain. The bottom chain contains, for each edge $v_{i}v_{j}\in E(H)$, $i<j$ (in an arbitrary order of edges), a “pocket” consisting of $5$ vertices $p_{ij}^{1},p_{ij}^{2},p_{ij}^{3},p_{ij}^{4},p_{ij}^{5}$ in order, as detailed in Figure 5. Importantly, $p_{ij}^{1}$ and $p_{ij}^{5}$ are mutually so close that the vertices $p_{ij}^{2},p_{ij}^{3}$ in the lower left corner can see only the vertex $v_{j}$ (of course, besides $p_{ij}^{1}$ and $p_{ij}^{5}$) and the vertex $p_{ij}^{4}$ in the lower right corner can see only the vertex $v_{i}$.

Assume now that we have got a proper $5$-colouring of the visibility graph $G$ of the constructed polygon $P$. We easily argue the following:

• •

Choose any edge $v_{i}v_{j}\in E(H)$. Then the vertices $p_{ij}^{1}$ and $p_{ij}^{5}$ of the corresponding pocket must receive distinct colours which we, up to symmetry, denote by $4$ and $5$. Since every vertex of the top chain sees $p_{ij}^{1}$ and $p_{ij}^{5}$, we get that every vertex $v_{k}$, $k=1,\ldots,n$, has a colour $1,2$ or $3$.

• •

For each edge $v_{i}v_{j}\in E(H)$, the $5$-tuple of vertices $(v_{j},p_{ij}^{1},p_{ij}^{2},p_{ij}^{3},p_{ij}^{5})$ of $P$ induces a $K_{5}$, and so does the nearly-identical $5$-tuple $(p_{ij}^{1},p_{ij}^{2},p_{ij}^{3},p_{ij}^{4},p_{ij}^{5})$. Consequently, in any proper $5$-colouring of $G$, the vertices $v_{j}$ and $p_{ij}^{4}$ get the same colour. And since $p_{ij}^{4}$ sees $v_{i}$, the colours of $v_{i}$ and $v_{j}$ must be distinct.

Altogether, any proper $5$-colouring of the visibility graph $G$ of $P$ implies a proper $3$-colouring of the graph $H$.

On the other hand, assume a proper colouring of the graph $H$ by colours $\{1,2,3\}$. We give the same colours to the vertices $v_{1},\ldots,v_{n}$ of the top chain of $P$, and we can always complete (e.g., greedily from left to right) this partial colouring to a proper $3$-colouring of the top chain of $P$. Then we assign alternate colours $4,5,4,5,\ldots$ to the exposed vertices of the bottom chain. Finally, we colour the lower corners of the bottom pockets as follows; for an edge $v_{i}v_{j}\in E(H)$, we give $p_{ij}^{4}$ the colour of $v_{j}$, and to $p_{ij}^{2},p_{ij}^{3}$ the remaining two colours among $1,2,3$. This gives a proper $5$-colouring of the visibility graph $G$ of $P$.

*The last bit is to show that the construction of $P$ can be realized in a grid of polynomial size in $n=|V(H)|$. Both the top and bottom concave shapes can be realized as “fat” parabolas, requiring only rough resolution of $\Theta(n^{2})$ in both horizontal and vertical directions. This is fully sufficient for the top chain, but realizing the pockets of the bottom chain is more delicate. Still, fine placement of the pocket of an edge $v_{i}v_{j}$ depends only on the vertices $v_{i}$ and $v_{j}$ of the top chain, and not on other pockets. Within the main scale, each pocket has dimensions $\Theta(n)$ and the pinhole opening is, say, $\frac{1}{n}$, and hence a sufficient precision for adjusting the pocket corners is $\Theta(\frac{1}{n})$. Altogether, the construction of $P$ is achieved on an $\mathcal{O}(n^{3})$ grid. *

4 Hardness of 4-colourability with holes

Consider a polygon $P$ together with a collection of pairwise disjoint polygons $Q_{i}$, $i=1,\ldots,k$, such that $Q_{i}\subseteq int(P)$. Then the set P\setminus int\big{(}Q_{1}\cup\ldots\cup Q_{k}\big{)} is called a polygon with holes. In this section we prove that, for polygons with holes, already $4$-colourability is an NP-complete problem. Given the algorithm for $4$-colouring from Section 2, it is natural that the proof we are going to present should be very different from the reduction in Section 3.

For better clarity, we present a construction of a polygon with holes as “digging polygonal corridors in solid mass”. These corridors (precisely, their topological closure) will then form the point set of our polygon, while the “mass trapped between” corridors will form the holes in the polygon. On a high level, our corridors will be composed of elementary channels, as depicted in Figures 6 and 7, placed along the lines of a large hexagonal (honeycomb) grid in the plane. More details follow next.

Theorem 4.1**.**

*The problem – given a polygon with holes $P$ in the plane, to decide whether the visibility graph of $P$ is properly $k$-colourable – is NP-complete for every $k\geq 4$. *

Proof 4.2**.**

The claim follows from Theorem 3.1 for $k\geq 5$, and so we consider only $k=4$ here. Again, the problem is clearly in NP. In the opposite direction, we reduce from the NP-complete problem of $3$-colouring a given planar graph $H$.

We first recall a folklore claim that every planar graph $H$ can be represented in a usual sufficiently large hexagonal grid in the following way: there is a collection of pairwise disjoint subtrees of the grid $T_{v}$: $v\in V(H)$ (representatives of the vertices of $H$) such that, for every edge $uv\in E(H)$, the grid contains an edge between $V(T_{u})$ and $V(T_{v})$ (called a representative edge of $uv$). (In other words, $H$ is a minor of the grid.) To simplify our construction, we may moreover assume that we always choose representative edges in the grid which are not of horizontal direction (out of the three directions $0^{\circ}$, $120^{\circ}$ and $240^{\circ}$).

Having such a representation of the given planar graph $H$ in the grid, we continue as follows. Let a vertex channel be the polygonal fragment shown in Figure 6, where the triples $a_{1}a_{2}a_{3}$ and $b_{1}b_{2}b_{3}$ are the triangle joins of the channel. Channels are composed, after suitable rotation, by gluing their triangle joins together, as illustrated in bottom part of Figure 6. When no further channel is glued to a join, then the dotted triangle edge(s) is “sealed” by a polygon edge. Let an edge channel be the polygonal fragment shown in Figure 7, again having two triangle joins $a_{1}a_{2}a_{3}$ and $b_{1}b_{2}b_{3}$ at its ends. Edge channels are used and composed in a same way with vertex channels, but edge channels cannot be rotated, only mirrored by the vertical axis (that is why we do not use them along horizontal grid edges). Altogether, we construct a polygon $P$ from $H$ by composing copies of the vertex channel along all the grid edges of each $T_{v}$, $v\in V(H)$, and by further composing in copies of the edge channel (possibly mirrored) along the representative edges of $H$ in the grid.

Assume now that we have got a proper $4$-colouring of the visibility graph of $P$. For each triangle join, let the vertex with the middle $y$-coordinate be called the flag vertex (it is the vertex which is extreme to the left or right). One can easily check from Figure 6 that, among all vertex channels of one $T_{v}$, $v\in V(H)$, all the triangle joins receive the same unordered triple of colours and, in particular, all the flag vertices have the same one colour. The same claim can also be derived from Theorem 2.6 applied to the standalone simple polygon formed by the vertex channels of $T_{v}$. Furthermore, one can check from Figure 7, that also the edge channel maintains the property that both its triangle joins must receive the same unordered triple of colours.

Naturally assuming connectivity of $H$, we hence conclude that every triangle join constructed in $P$ receives the same unordered triple of colours, say $\{1,2,3\}$. Now, to each vertex $v$ of $H$ we assign the unique colour from $\{1,2,3\}$ which occurs on the flag vertices of $T_{v}$. Since the two flag vertices of the edge channel see each other ($a_{1}$ and $b_{3}$ in Figure 7), this ensures that for every edge $uv\in E(H)$ the colours assigned to $u$ and $v$ are distinct, and so $H$ is $3$-colourable.

In the converse direction, we assume that $H$ has a proper $3$-colouring. We can routinely $4$-colour the polygonal fragments of each $T_{v}$, $v\in V(H)$, such that all the flag vertices of $T_{v}$ get the colour of $v$. Then, for each $uv\in E(H)$ with distinct colours on $u$ and $v$, we can complete proper $4$-colouring of the fragment of $P$ made by the representative edge channel of $uv$, as shown in the right part of Figure 7. Hence the visibility graph of $P$ is then $4$-colourable.

*Finally, the construction of $P$ is easily done (with negligible distortion of the angles of hexagonal grid) within polynomial resolution and so in polynomial time. *

5 Conclusions

In this paper we have shown that the problem of deciding 5-colourability for visibility graphs of simple polygons, is NP-complete. We have also proved that the 4-colouring problem can be solved for visibility graphs of simple polygons, in polynomial time, whereas for visibility graphs of polygons with holes, it becomes NP-complete. However, it still remains to be explored whether approximation algorithms could exist for the hard colouring problems on visibility graphs of polygons.