Text Extraction From Texture Images Using Masked Signal Decomposition

TL;DR

This paper introduces a novel method for extracting text from textured backgrounds in images by using masked signal decomposition, which effectively separates overlaid signals with similar colors.

Contribution

The work proposes a new approach that models text extraction as a masked signal decomposition problem, improving accuracy over traditional segmentation methods.

Findings

Proposed algorithm outperforms recent methods on challenging images.

Effective separation of text from textured backgrounds with similar colors.

Uses alternating optimization to solve a relaxed binary mask problem.

Abstract

Text extraction is an important problem in image processing with applications from optical character recognition to autonomous driving. Most of the traditional text segmentation algorithms consider separating text from a simple background (which usually has a different color from texts). In this work we consider separating texts from a textured background, that has similar color to texts. We look at this problem from a signal decomposition perspective, and consider a more realistic scenario where signal components are overlaid on top of each other (instead of adding together). When the signals are overlaid, to separate signal components, we need to find a binary mask which shows the support of each component. Because directly solving the binary mask is intractable, we relax this problem to the approximated continuous problem, and solve it by alternating optimization method. We show that the proposed algorithm achieves significantly better results than other recent works on several challenging images.

Figures

Tables

Table 1. TABLE I : Comparison of accuracy of different algorithms

Segmentation Algorithm	Precision	Recall	F1 score
SPEC [3]	67%	77%	71.6%
Hierarchical Clustering [2]	66.5%	92%	77.2%
Sparse Dec. with TV [7]	71%	91.7%	80%
The proposed algorithm	95%	92.5%	93.7%

Equations

$$x=(1-w)\circ x_1+w\circ x_2\text{s.t.}w\in \{0,1{\}}^n$$

$$\underset{w,x_1,x_2}{\min }{\varphi }_1(x_1)+{\lambda }_1{\varphi }_2(x_2)+{\lambda }_2{\varphi }_3(w)$$

$$\text{s.t.}w\in \{0,1{\}}^n,x=(1-w)\circ x_1+w\circ x_2$$

$$x=(1-w)\circ x_1+w\circ x_2,w\in \{0,1{\}}^n$$

$$x=(1-w)\circ (P_1{\alpha }_1)+w\circ (P_2{\alpha }_2)s.t.\Vert {\alpha }_i{\Vert }_0\le k_i$$

$$x=(I-W)P_1{\alpha }_1+W{P}_2{\alpha }_{2}s.t.\Vert {\alpha }_i{\Vert }_0\le k_i$$

$$\underset{w,{\alpha }_1,{\alpha }_2}{\min }\frac{1}{2}\Vert f-(1-w)\circ P_1{\alpha }_1-w\circ P_2{\alpha }_2{\Vert }_2^2+\lambda \Vert w{\Vert }_0$$

$$\text{s.t.}w\in \{0,1{\}}^n,\Vert {\alpha }_1{\Vert }_0\le k_1,\Vert {\alpha }_2{\Vert }_0\le k_2$$

$$\underset{w,{\alpha }_1,{\alpha }_2}{\min }\frac{1}{2}\Vert f-(1-w)\circ P_1{\alpha }_1-w\circ P_2{\alpha }_2{\Vert }_2^2+\lambda \Vert w{\Vert }_1$$

$$\text{s.t.}w\in [0,1{]}^n,\Vert {\alpha }_1{\Vert }_0\le k_1,\Vert {\alpha }_2{\Vert }_0\le k_2$$

$${\alpha }_2=\underset{{\alpha }_2}{\mathrm{argmin}}\Vert f-(I-W)P_1{\alpha }_1-W{P}_2{\alpha }_2{\Vert }_2^2\Rightarrow$$

$$\displaystyle\nabla_{\alpha_{2}}L=0\Rightarrow P_{2}^{t}W^{t}\big{(}WP_{2}\alpha_{2}+(I-W)P_{1}\alpha_{1}-f\big{)}=0$$

$$\Rightarrow {\alpha }_2=(P_2^t{W}^{t}W{P}_2{)}^{-1}{P}_2^t{W}^t(f-(I-W)P_1{\alpha }_1)$$

$$\underset{w}{\min }\frac{1}{2}\Vert f-\text{diag}(P_1{\alpha }_1)(1-w)-\text{diag}(P_2{\alpha }_2)w{\Vert }_2^2+\lambda \Vert w{\Vert }_1$$

$$\text{s.t.}w\in [0,1{]}^n$$

$$\underset{w}{\min }\frac{1}{2}\Vert h-Cw{\Vert }_2^2+\lambda \Vert w{\Vert }_1$$

$$\text{s.t.}w\in [0,1{]}^n$$

$$\underset{w}{\min }g(w)=\frac{1}{2}\Vert h-Cw{\Vert }_2^2+\lambda \Vert w{\Vert }_1$$

$$=\sum _{i=1}^n{g}_i(w_i)=\sum _{i=1}^n\frac{1}{2}|h_i-C_{ii}{w}_i{|}^2+\lambda |w_i{|}_1$$

$$w_i=\text{soft}(h_i/C_{ii},\lambda /C_{ii}^2)$$

$$\text{Soft}(x,\lambda )=\text{sign}(x)\text{max}(|x|-\lambda ,0)$$

$$\displaystyle\ w_{i}^{*}=\Pi_{[0,1]}\big{(}\text{soft}(h_{i}/C_{ii},\lambda/C^{2}_{ii})\big{)}$$

$$\text{Precision}=\frac{\text{TP}}{\text{TP+FP}},\text{Recall}=\frac{\text{TP}}{\text{TP+FN}}$$

$$\text{F1}=2\frac{\text{precision}\times \text{recall}}{\text{precision+recall}}$$

Full text

I Introduction

Text extraction is an important problem with many applications in image processing and computer vision, such as optical character recognition, license plate detection, road sign detection in autonomous driving. Text extraction could be very challenging when the background has complicated texture and has overlapping color distributions as the text. Text extraction is usually accomplished in two steps: text region detection which detects the regions where text is high likely to be present [1], and text segmentation to find a binary mask which shows the location of text. Our main focus in this work would be on the second step, which is to derive a binary mask for text segmentation in a detected region containing texts.

Different algorithms have been proposed in the past for text segmentation from images. In [2], Haffner et al proposed an algorithm for text image segmentation and document compression using a hierarchical clustering approach. In [3], an algorithm is proposed for segmentation of texts and graphics from screen content images and coding. In [4], Kumar proposes an algorithm for extracting texts from document images using matched wavelet filters. They used a clustering-based technique to estimate globally matched wavelet filters using a collection of groundtruth images. In [5], Saha proposed a text segmentation algorithm using Hough transform. Various algorithms based on sparse decomposition has also been proposed for text extraction, where the text extraction is achieved by assuming proper prior on the text component [6], [7]. There are also many other works based on histogram analysis, maximally stable extremal region (MSER), and appearance [8]-[12]. The reader is referred to [17] for a good survey of text recognition.

In this work we look at this problem from a signal decomposition perspective. We assume that the foreground texts $x_{2}$, and background $x_{1}$ are combined to create an image $x$. But instead of assuming an additive model ($x=x_{1}+x_{2}$) as in [6]-[7], we consider an overlaying model, which is a more truthful characterization of images with overlaid texts. Specifically, we assume any pixel value in the image comes either from the background component, or from foreground text. We can formulate this as:

[TABLE]

where $\circ$ denotes the element-wise product, and $w$ is the binary mask indicating foreground pixel locations.

If we have some prior knowledge about each component, we can solve the decomposition problem as the following optimization problem:

[TABLE]

where $\phi_{i}$ are some cost functions that should be minimized based on our prior knowledge about $x_{1}$, $x_{2}$, and $w$. After solving this optimization problem, $w$ will give us the location of texts. We will discuss further about the problem formulation and solution in the next section.

Figure 2 compares the segmentation results using hierarchical k-means approach, sparse decomposition, and the proposed algorithm in this work for a sample image. As we can see, each of the previous approaches have their own difficulties, for example clustering based scheme would have difficulty separating text from background in the case where text has a similar color to background. And sparse decomposition based model, misses some part of the text, while detecting some part of the background as text. On the other hand, the proposed algorithm performs significantly better. This approach can also be used for segmentation of medical images [14], and also texture extraction from biometrics [15]-[16].

The rest of this paper is structured as follows: Section II presents the problem formulation, and the iterative algorithm to solve the optimization problem. The experimental results and performance analysis are provided in Section III, and the paper is concluded in Section IV.

II Problem Formulation

As discussed earlier, we consider text as an overlaid component on the background image. Therefore we can denote an image as the masked summation of the background and text:

[TABLE]

To further simplify the problem, we assume that each component has a sparse representation using some properly designed dictionary or forming a subspace. Therefore we can write this problem as:

[TABLE]

where $P_{i}$ is n$\times\text{m}_{i}$ matrix, where each column denotes one of the basis functions from the corresponding subspace/dictionary. We would like to note that here we assume that both subspaces are known, in the applications where the choice of subspaces/dictionaries is not clear, we can use dictionary/subspace learning algorithms to learn them [18]-[23].

Note that, the model in (4) can be rewritten as:

[TABLE]

where $W$ is a diagonal matrix with the vector $w$ on its main diagonal (i.e. $W=\text{diag}(w)$). If a diagonal element is 1, the corresponding pixel belongs to the foreground, and otherwise to the background.

The decomposition problem in Eq. (4) is a highly ill-posed problem. Therefore we need to impose some prior on each component, and also on $w$ to be able to perform this decomposition. We assume that each component has a sparse representation with respect to its own subspace, but not with respect to the other one. We also assume that the second component is sparse, which is very desirable for text. To promote sparsity of the second component, we add the $\ell_{0}$ norm of $w$ to the cost function (note that $w$ corresponds to the support of the second component).

We can incorporate all these priors in an optimization problem as shown below:

[TABLE]

This problem is a combinatorial problem, and is not tractable, both because of the $\|w\|_{0}$ term in the cost function and also the binary nature of $w$. We relax these conditions to be able to solve this problem in an alternating optimization approach. We replace the $\|w\|_{0}$ in the cost function with $\|w\|_{1}$, and also relax the $w\in\{0,1\}^{n}$ condition to $w\in[0,1]^{n}$. Then we will get the following optimization problem:

[TABLE]

This problem can be solved with different approaches, such as majorization minimization, alternating direction method, and random sampling approach [24]-[29]. Here we present an algorithm based on alternating direction method, which simply solves this problem by updating each variable at a time, and setting the gradient of cost function ($L$) w.r.t. each variable to zero. The optimization steps with respect to $\alpha_{1}$ and $\alpha_{2}$ are symmetric, therefore we only show the solution for $\alpha_{2}$. We first ignore the constraint $\|\alpha_{2}\|_{0}\leq k_{2}$ and solve the unconstrained problem, and then keep the largest $k_{2}$ components of $\alpha_{2}$ to satisfy the constraint.

[TABLE]

Then by keeping the $k_{2}$ largest components of the above solution, we will derive the $\alpha_{2}$ update as $\alpha_{2}^{*}=\Pi_{top}(\alpha_{2};k_{2})$. We also provide the update step of $w$ here. By using the equality $\text{diag}(w)P_{2}\alpha_{2}=\text{diag}(P_{2}\alpha_{2})w$, we can re-write the optimization w.r.t. $w$ as below:

[TABLE]

First note that, we can simplify the above optimization problem as below:

[TABLE]

where $C=\text{diag}(P_{2}\alpha_{2})-\text{diag}(P_{1}\alpha_{1})=\text{diag}(P_{2}\alpha_{2}-P_{1}\alpha_{1})$, and $h=f-P_{1}\alpha_{1}$. We can first ignore the constraint, and then project the optimal solution of the cost function onto the feasible set ($w\in[0,1]^{n}$). Since $C$ is a diagonal matrix, the cost function can be decoupled in the components of $w$ as:

[TABLE]

where $C_{ii}$ denotes the i-th diagonal element of $C$ (which is the i-th diagonal element of the vector $P_{1}\alpha_{1}-P_{2}\alpha_{2}$). Now this problem can be easily solved with soft-thresholding as [30]:

[TABLE]

where $\text{soft}(x,\lambda)$ denotes the soft-thresholding operator applied element-wise and defined:

[TABLE]

Now if we denote the projection operator on [0,1] by $\Pi_{[0,1]}(w)$, then the optimization solution of $w$ step would be:

[TABLE]

The overall algorithm is summarized in Algorithm 1.

where $\bar{W}=I-W$.

III Experimental Results

In this section we provide the experimental study on the application of the proposed algorithm for text extraction from several challenging images. These images are manually generated by adding text on top of a relatively complicated background. Our dataset contains more than 300 image blocks.

We first define the parameters of our model. We apply our algorithm on blocks of 64x64. We first convert each block into a vector of 4096 dimension, and then use the proposed algorithm for background foreground separation. For the first component we use 40 dimensional low-frequency DCT subspace [31], and for the second component we use 10 dimensional Hadamard subspace [32]. The sparsity of coefficients are chosen as $k_{1}=5$ and $k_{2}=5$. The weight parameter for the sparsity term is chosen to be $\lambda=10$, which is tuned by testing on a validation set. The number of iterations for optimization algorithm is chosen to be $k_{max}=10$, and $w$ is initialized with uniform random variable in [0,1].

We now discuss two possible ways to perform the binarization. As mentioned earlier, the goal is to solve the binary optimization problem, but to make it tractable, we approximate the binary variables with a continuous variable in $[0,1]$, and binarize them after solving the optimization problem. In the algorithm 1, we first solve the optimization problem in Eq. (7), for a continuous $w$, and then binarize $w$ at the very end. An alternative approach is to binarize the variables $w$ after each update of $w$ in algorithm 1. We tested both approaches for two of the test images, and provided the results in Figure 2. As we can see, usually doing the binarization at the very end works better. One possible reason could be that with the second approach, the final solution is very sensitive to the initialization.

We now provide the comparison of the proposed algorithm with prior approaches on text extraction. We compare the proposed algorithm with three previous algorithms; hierarchical k-means clustering [2], shape primitive extraction, and sparsity based approach [7]. Figure 3 shows the comparison between the proposed algorithm performance compared with the previous approaches for 3 sample images from our dataset. As it can be seen, the proposed algorithm achieves more accurate result than previous methods.

We also provide the average precision, recall and F1 score [33] achieved by different algorithms for the above sample images. The average precision, recall and F1 score by different algorithms are given in Table 1. The precision, recall and F1 score are defined as in Eq. (12) and (13), where TP, FP and FN denote true positive, false positive and false negative respectively. In our evaluation, we treat the text pixels as positive. A pixel that is correctly identified as text (compared to the ground-truth) is considered true positive.

[TABLE]

[TABLE]

As it can be seen, the proposed scheme achieves much higher precision and recall than SPEC, hierarchical k-means clustering and sparse decomposition approach. It is worth mentioning that our algorithm is pretty robust to the initialized value of variables. A complete study of the initialization impact on the segmentation result is presented in [34].

IV conclusion

This paper proposes a text extraction algorithm from a signal decomposition perspective. We consider texts as an overlaying component on top of a natural scene image, where the pixel values at each point comes from one and only one of the components (in contrast with the traditional signal decomposition case, where the signal component at each point is assumed to be the summation of corresponding values from different components). Each component is assumed to have a sparse representation with respect to a suitable subspace. The text component is also assumed to be sparse. We then propose an optimization framework to separate the background and text components using the alternating direction method. Experimental results show that the proposed algorithm can provide significantly better text extraction when the background is textured and has similar color distribution to text. This algorithm could be further improved by learning the subspaces for the desired application.

Acknowledgment

The authors would like to thank Ivan Selesnick, Pablo Sprechmann, and Arian Maleki for their valuable comments regarding this work.