Column subset selection is NP-complete

TL;DR

This paper proves that the column subset selection problem (CSSP), a fundamental problem in numerical linear algebra, is NP-complete, resolving a long-standing open question about its computational complexity.

Contribution

The paper establishes that CSSP is NP-complete, providing the first proof of its computational hardness and clarifying the problem's algorithmic intractability.

Findings

CSSP is NP-complete.

Addresses a long-standing open problem.

Implications for algorithms in numerical linear algebra.

Abstract

Let $M$ be a real $r\times c$ matrix and let $k$ be a positive integer. In the column subset selection problem (CSSP), we need to minimize the quantity $\|M-SA\|$, where $A$ can be an arbitrary $k\times c$ matrix, and $S$ runs over all $r\times k$ submatrices of $M$. This problem and its applications in numerical linear algebra are being discussed for several decades, but its algorithmic complexity remained an open issue. We show that CSSP is NP-complete.

Equations

$${\delta }_k(M)=\underset{S,A}{\min }\Vert M-SA\Vert ,$$

$$t=\frac{1}{4(m+n{)}^3},$$

$$\left(\begin{array}[]{ccc|ccc|ccc|ccc}1&0&0&1&0&0&1&0&0&t^{2}&0&t^{2}\\ 0&1&0&0&1&0&0&1&0&0&t^{2}&t^{2}\\ 0&0&1&0&0&1&0&0&1&t^{2}&t^{2}&0\\ \hline\cr t^{3}&t^{3}&t^{3}&0&0&0&0&0&0&t^{5}&t^{5}&t^{5}\\ 0&0&0&t^{3}&t^{3}&t^{3}&0&0&0&t^{5}&t^{5}&t^{5}\\ 0&0&0&0&0&0&t^{3}&t^{3}&t^{3}&t^{5}&t^{5}&t^{5}\\ \hline\cr 0&0&0&0&0&0&0&0&0&t&t&t\end{array}\right),$$

$${\delta }_n\left(\mathcal{M}(G)\right)⩽\sqrt{m{t}^2+4n{t}^6+m{t}^{10}}.$$

$$\underset{A}{\min }\Vert \mathcal{M}-SA\Vert ⩽\sqrt{m{t}^2+4n{t}^6+m{t}^{10}},$$

$$\Vert (I-S{S}^{+})\mathcal{M}\Vert ⩽\sqrt{m{t}^2+4n{t}^6+m{t}^{10}}$$

$$V^{\ast }=\{v^{\phi (v)},v\in V\}$$

$$S=\left(\begin{array}[]{c|c|c}I_{n_{1}}&O&O\\ \hline\cr O&I_{n_{2}}&O\\ \hline\cr O&O&I_{n_{3}}\\ \hline\cr t^{3}\mathbf{j}_{n_{1}}&O&O\\ \hline\cr O&t^{3}\mathbf{j}_{n_{2}}&O\\ \hline\cr O&O&t^{3}\mathbf{j}_{n_{3}}\\ \hline\cr O&O&O\end{array}\right),$$

$$\left(\begin{array}[]{c|c|c|c|c|c|c}u_{1}t^{6}J_{n_{1}}&O&O&-u_{1}t^{3}\mathbf{j}^{\top}_{n_{1}}&O&O&O\\ \hline\cr O&u_{2}t^{6}J_{n_{2}}&O&O&-u_{2}t^{3}\mathbf{j}^{\top}_{n_{2}}&O&O\\ \hline\cr O&O&u_{3}t^{6}J_{n_{3}}&O&O&-u_{3}t^{3}\mathbf{j}^{\top}_{n_{3}}&O\\ \hline\cr-u_{1}t^{3}\mathbf{j}_{n_{1}}&O&O&u_{1}&0&0&0\\ \hline\cr O&-u_{2}t^{3}\mathbf{j}_{n_{2}}&O&0&u_{2}&0&0\\ \hline\cr O&O&-u_{3}t^{3}\mathbf{j}_{n_{3}}&0&0&u_{3}&0\\ \hline\cr O&O&O&0&0&0&1\end{array}\right),$$

$$m{t}^2+4n{\left(\frac{t^3}{1+n{t}^6}\right)}^2+(m+2){\left(\frac{t^5}{1+n{t}^6}\right)}^2,$$

Full text

1. Introduction

Throughout this paper, we denote by $\|M\|$ the Frobenius norm of a real matrix $M$, that is, the square root of the sum of squares of the elements of $M$. If $M$ has $r$ rows and $c$ columns, then we write $M\in\mathbb{R}^{r\times c}$. For any integer $k\leqslant c$, we denote

[TABLE]

where $A$ and $S$ run, respectively, over $\mathbb{R}^{k\times c}$ and over all $r\times k$ submatrices of $M$. If we fix $S$ in the definition of $\delta_{k}(M)$, then the optimal value of $A$ is known to equal $S^{+}M$, where $X^{+}$ denotes the pseudoinverse of $X$.

The column subset selection problem (CSSP) is the task of detecting a submatrix $S$ realizing the minimum in (1.1). The quantity $\delta_{k}(M)$ can be seen as a value of how close a matrix can be to the linear space spanned by a tuple of $k$ of its columns, which makes CSSP important for problems of low-rank matrix approximation. In fact, CSSP is being studied for more than twenty years (see [4]), and many new areas of its application have arisen since that time. Besides the low-rank approximation theory ([2, 3]), the subset selection problem arises naturally in statistical data analysis ([16]), large data analysis ([17]), optimal experiment design ([7]), statistical computing ([4]), artificial intelligence ([1]), networking ([10]), rank-deficient least squares computation ([8, 9]), and other branches of modern applied mathematics.

The CSSP problem has been widely discussed in the applied mathematics community, but its algorithmic complexity status remained unknown. In 2009, Boutsidis, Mahoney, and Drineas ([3]) pointed out that the NP-hardness of CSSP is an open problem, and this problem has been discussed in numerous sources since then ([1, 6, 13, 14, 15]). A recent paper [5] presents a first step towards a solution: Çivril proves that CSSP is UG-hard, that is, he gives a conditional NP-hardness proof subject to the validity of the so-called Unique Games conjecture. Since there is currently no clear evidence of whether this conjecture is true or false (see [12]), the paper by Çivril left the complexity status of CSSP wide open.

2. The result

The goal of our paper is to determine the complexity status of CSSP and to prove that this problem is NP-hard. Moreover, we prove that the CSSP problem restricted to rational matrices is NP-complete in the Turing model of computation. This problem can be stated formally in the following way.

Problem 2.1**.**

(COLUMN SUBSET SELECTION.)

Given: A matrix $M\in\mathbb{Q}^{r\times c}$, an integer $k\in\{1,\ldots,c\}$, and a positive rational threshold $\tau$.

Question: Is $\delta_{k}(M)\leqslant\sqrt{\tau}$?

Recall that the optimal matrix $A$ in (1.1) can be expressed as $S^{+}M$, where $X^{+}$ denotes the pseudoinverse of $X$. In fact, many authors prefer writing $SS^{+}M$ instead of $SA$ in (1.1), and this gives an equivalent definition of what we denote by $\delta_{k}(M)$. In particular, we see that Problem 2.1 belongs to NP: If we correctly choose the optimal submatrix $S$, then we are able to compute $\delta_{k}(M)$ in polynomial time. Our goal is to prove the following result.

Theorem 2.2**.**

Problem 2.1 is NP-complete.

As said above, Problem 2.1 belongs to NP, so we only need to construct a polynomial transformation to Problem 2.1 from some known NP-complete problem. To this end, we use the classical problem of graph three-coloring, see [11] for details. We consider simple graphs $G=(V,E)$, where $V$ is a finite non-empty set of vertices, and $E$ is a finite non-empty set of edges. A function $\varphi:V\to\{1,2,3\}$ is called a three-coloring of $G$ if $\varphi(u)\neq\varphi(v)$ holds whenever $\{u,v\}\in E$.

3. The reduction

We proceed with the description of our reduction. We denote by $n,m$ the cardinalities of $V,E$, respectively, we set

[TABLE]

and we denote by $V^{i}=\{v_{1}^{i},\ldots,v_{n}^{i}\}$ a copy of the set $V=\{v_{1},\ldots,v_{n}\}$. Our reduction from three-coloring to CSSP is the matrix $\mathcal{M}={\mathcal{M}}(G)$ defined as follows. The rows of ${\mathcal{M}}$ have indexes in $V\cup\{1,2,3\}\cup\{{\varepsilon}\}$, and the columns are indexed with $V^{1}\cup V^{2}\cup V^{3}\cup E$. We denote by ${\mathcal{M}}(\alpha,\beta)$ the entry of ${\mathcal{M}}$ at the intersection of a row with index $\alpha$ and a column with index $\beta$. For all distinct vertices $u,v\in V$, for all distinct numbers $i,j\in\{1,2,3\}$, and for all $e\in E$, we define the entries of ${\mathcal{M}}$ by

(1) ${\mathcal{M}}(u,v^{i})=0$, ${\mathcal{M}}(u,u^{i})=1$,

(2) ${\mathcal{M}}(u,e)=t^{2}$ if $u$ is adjacent to $e$ and ${\mathcal{M}}(u,e)=0$ otherwise,

(3) ${\mathcal{M}}(i,v^{j})=0$, ${\mathcal{M}}(i,v^{i})=t^{3}$, ${\mathcal{M}}(i,e)=t^{5}$,

(4) ${\mathcal{M}}({\varepsilon},v^{i})=0$, ${\mathcal{M}}({\varepsilon},e)=t$.

For instance, the matrix ${\mathcal{M}}(K_{3})$, which corresponds to the triangle graph $K_{3}$, looks like

[TABLE]

where the first three rows have indexes in $V$, the next three rows have indexes $\{1,2,3\}$, and the last row has index ${\varepsilon}$. Similarly, the columns of the above matrix are partitioned into triples corresponding to $V^{1},V^{2},V^{3}$, and $E$, respectively. Our goal in the rest of the paper is to show the following.

Theorem 3.1**.**

A graph $G$ has a three-coloring if and only if

[TABLE]

This theorem would complete the NP-hardness proof of Problem 2.1.

4. The proof. Part one

In the rest of our paper, we denote by ${\mathcal{V}}$ an $n$-element subset of the set of column indexes of ${\mathcal{M}}$. The submatrix consisting of the columns of ${\mathcal{M}}$ with indexes in ${\mathcal{V}}$ is to be denoted by $S$. Also, we write ${\mathcal{M}}(\square,\beta)$ to denote the column of ${\mathcal{M}}$ with index $\beta$. As said above, the two conditions

[TABLE]

[TABLE]

are equivalent, and Theorem 3.1 states that one of them is satisfied by some $S$ if and only if $G$ is a three-colorable graph. In this section, we point out several necessary conditions for $S$ to satisfy (4.1) and (4.2).

Lemma 4.1**.**

If the sets ${\mathcal{V}}$ and $E\cup\{v^{1},v^{2},v^{3}\}$ are disjoint for some $v\in V$, then the inequality (4.1) is false.

Proof.

We see that the $v$th entry of any column of $S$ is zero, so the matrix ${\mathcal{M}}-SA$ has ones at $(v^{1},v),(v^{2},v),(v^{3},v)$. We have $\|{\mathcal{M}}-SA\|\geqslant 3$, which contradicts (4.1). ∎

In order to prove one more result of similar kind, we need to recall the definition of a strongly column diagonally dominant matrix. Namely, this is a real square matrix $D$ any of whose diagonal elements is greater than the sum of the absolute values of all the elements in the same column. A classical result of matrix theory, sometimes called the Levy-Desplanques theorem, states that any such matrix is non-singular. (An interested reader is referred to [18] for a historical survey on this theorem.)

Lemma 4.2**.**

If there is $e\in{\mathcal{V}}\cap E$, then (4.1) is false.

Proof.

Denoting by $\mathcal{S}$ the linear space spanned by the columns of $S$, we see that ${\mathcal{M}}(\square,e)$ belongs to $\mathcal{S}$. Taking into account (3.1) and (4.1), we conclude that, for all $v\in V$, the space $\mathcal{S}$ contains a column $C_{v}$ such that $\|C_{v}-{\mathcal{M}}(\square,v^{1})\|<(n+1)^{-1}$. In particular, the $v$th element of $C_{v}$ is at least $n/(n+1)$, and the absolute value of the $u$th element of $C_{v}$ is less than $(n+1)^{-1}$ (where $u\in V\cup\{\varepsilon\}$ is different from $v$).

Since $\mathcal{S}$ is spanned by $n$ vectors, the matrix (we denote it by $C$) consisting of the $n+1$ columns $\{C_{v}\}$ and ${\mathcal{M}}(\square,e)$ must be of rank at most $n$. However, the submatrix of $C$ formed by taking the rows with indexes in $V\cup\{\varepsilon\}$ is strongly column diagonally dominant, so we get a contradiction. ∎

Lemma 4.3**.**

The inequalities (4.1), (4.2) imply that ${\mathcal{V}}=\{v^{\psi(v)},v\in V\}$, where $\psi$ is a mapping from $V$ to $\{1,2,3\}$.

Proof.

Follows immediately from Lemmas 4.1 and 4.2. ∎

5. The proof. Part two

In this section, we consider the indexing sets ${\mathcal{V}}$ as in Lemma 4.3. Namely, we will prove that the matrix $S$ satisfies (4.2) if and only if the corresponding mapping $\psi$ is a three-coloring of $G$. This would mean that, if $G$ does not admit a three-coloring, the inequality (4.2) is never satisfied. This would prove Theorem 3.1 and show that the task of determining an optimal submatrix $S$ in a CSSP instance is at least as hard as finding a three-coloring of a given graph.

Lemma 5.1**.**

If $G$ is three-colorable, then $\delta_{n}\left({\mathcal{M}}(G)\right)\leqslant\sqrt{mt^{2}+4nt^{6}+mt^{10}}$.

Proof.

Let $\varphi:V\to\{1,2,3\}$ be a three-coloring of $G$. We take

[TABLE]

to be the indexes of columns of ${\mathcal{M}}$ which form the matrix $S$ as in (4.1), and we proceed with the description of $A$. Its rows will have indexes in $V^{*}$, and the indexing of columns will correspond to that of ${\mathcal{M}}$. We set $A(\square,v^{i})$ to be the column with the one at the position $v^{\varphi(v)}$ and zeros everywhere else; for $e=\{u,v\}$ in $E$, the column $A(\square,e)$ has $t^{2}$ at the positions $u^{\varphi(u)},v^{\varphi(v)}$ and zeros elsewhere. It remains to check that $\|{\mathcal{M}}-SA\|=\sqrt{mt^{2}+4nt^{6}+mt^{10}}$. ∎

Lemma 5.2**.**

Assume ${\mathcal{V}}=\{v^{\psi(v)},v\in V\}$, where a mapping $\psi$ from $V$ to $\{1,2,3\}$ is not a three-coloring of $G$. Then (4.2) is not satisfied.

Proof.

We can write

[TABLE]

where the splitting of columns into blocks corresponds to the indexing sets $\psi^{-1}(1)$, $\psi^{-1}(2)$, $\psi^{-1}(3)$, $\{1\}$, $\{2\}$, $\{3\}$, $\{{\varepsilon}\}$. The $O$’s stand for zero matrices of relevant sizes, and $\mathbf{j}_{k}$ is the $1\times k$ matrix of all ones. Since the columns of $S$ are linearly independent, we can write $S^{+}=(S^{\top}S)^{-1}S^{\top}$, and a computation shows that $I-SS^{+}$ equals

[TABLE]

where $u_{i}=({1+n_{i}t^{6}})^{-1}$. Again, the splitting of the rows and columns into blocks corresponds to the partition $\psi^{-1}(1)$, $\psi^{-1}(2)$, $\psi^{-1}(3)$, $\{1\}$, $\{2\}$, $\{3\}$, $\{{\varepsilon}\}$.

By (5.1), the ${\varepsilon}$th row of $(I-SS^{+}){\mathcal{M}}$ equals the ${\varepsilon}$th row of ${\mathcal{M}}$, so it has $m$ entries equal to $t$. Now let $\Delta$ be the matrix formed by the rows of $(I-SS^{+}){\mathcal{M}}$ with indexes $1,2,3$. Using (5.1) once again, we note that, for $v\in V$ and $j\in\{1,2,3\}\setminus\{\psi(v)\}$, the $v^{j}$th column of $\Delta$ contains one zero, one element of the form $u_{i}t^{3}$, and one element of the form $-u_{i}t^{3}$. (Although it is not relevant for the proof, let us note that the $v^{\psi(v)}$th column of $\Delta$ consists of zeros.)

For $e=\{u,v\}\in E$, there are two possibilities. If $\psi(u)=\psi(v)$, then the $e$th column of $\Delta$ contains two zeros and one element of the form $u_{i}t^{5}$. Otherwise, all the three elements in that column are of the form $\pm u_{i}t^{5}$ (and since $\psi$ is not a three-coloring of $G$, this occurrence arises at least once). Therefore, the sum of squares of the entries of $(I-SS^{+}){\mathcal{M}}$ is greater than or equal to

[TABLE]

which is greater than $mt^{2}+4nt^{6}+mt^{10}$ as a comparison with (3.1) shows. ∎

This completes the proof of the main results of our paper. Namely, we have shown that the column subset selection problem is NP-hard, and this problem becomes NP-complete in the Turing model of computation when restricted to rational matrices. Recall that we were working with the Frobenius norm of matrices, which seems to be the most common way to measure the cost function in the CSSP problem (see [3, 5]). However, the same method will surely allow one to prove the same results for CSSP with respect to other norms like the $\ell_{p}$ operator norms for all $p>1$ or the spectral norm of matrices.