Water-Filling: An Efficient Algorithm for Digitized Document Shadow Removal

TL;DR

This paper introduces a new algorithm that effectively removes shading artifacts from digitized documents by modeling illumination correction as a diffusion process, leading to improved image quality under challenging lighting conditions.

Contribution

The paper presents a novel diffusion-based algorithm for shading artifact removal in digitized documents, outperforming existing methods in challenging lighting scenarios.

Findings

The proposed method effectively reduces shading artifacts.

It outperforms state-of-the-art illumination correction techniques.

The algorithm demonstrates robustness on smartphone-captured documents.

Abstract

In this paper, we propose a novel algorithm to rectify illumination of the digitized documents by eliminating shading artifacts. Firstly, a topographic surface of an input digitized document is created using luminance value of each pixel. Then the shading artifact on the document is estimated by simulating an immersion process. The simulation of the immersion process is modeled using a novel diffusion equation with an iterative update rule. After estimating the shading artifacts, the digitized document is reconstructed using the Lambertian surface model. In order to evaluate the performance of the proposed algorithm, we conduct rigorous experiments on a set of digitized documents which is generated using smartphones under challenging lighting conditions. According to the experimental results, it is found that the proposed method produces promising illumination correction results and outperforms the results of the state-of-the-art methods.

Figures

Equations

$$w(x,t)\ge 0,$$

$$w(x,t)=0,$$

$${}\begin{split}\frac{\partial w(x,t)}{\partial t}\bigg{|}_{x=x_{0}}&=-\nabla\cdot\mathbf{J_{w}}(x,t)\big{|}_{x=x_{0}}=-\frac{\partial J_{w}(x,t)}{\partial x}\bigg{|}_{x=x_{0}},\end{split}$$

$$J_w(x_0,t)\propto G(x_0,t)-G(x_0+\Delta ,t).$$

$${}\begin{split}w(x,y,t+\Delta)=&\eta\cdot\big{\{}G(x+\Delta,y,t)+G(x-\Delta,y,t)+G(x,y+\Delta,t)\\ &+G(x,y-\Delta,t)-4\cdot G(x,y,t)\big{\}}+w(x,y,t),\end{split}$$

$$\begin{array}{rl}{w}_{\varphi }(x_0,t)& \propto \min \{-G(x_0,t)+G(x_0+\Delta ,t),0\}\\ & +\min \{-G(x_0,t)+G(x_0-\Delta ,t),0\}.\end{array}$$

$$w_{\psi }(x_0,t)=(\hat{h}-G(x_0,t))\cdot e^{-t},$$

$$\begin{array}{rl}w(x,y,t& +\Delta )=(\hat{h}-G(x,y,t))\cdot e^{-t}+\eta \cdot \{\min \{G(x+\Delta ,y,t)-G(x,y,t),0\}\\ & +\min \{G(x-\Delta ,y,t)-G(x,y,t),0\}+\min \{G(x,y+\Delta ,t)-G(x,y,t),0\}\\ & +\min \{G(x,y-\Delta ,t)-G(x,y,t),0\}\}+w(x,y,t).\end{array}$$

$$I_r\left(x,y\right)=\ell \cdot I_f=\ell \cdot \frac{I\left(x,y\right)}{I_b\left(x,y\right)}\approx \underset{t\to \infty }{\lim }\frac{I\left(x,y\right)}{G\left(x,y,t\right)}\cdot \ell ,$$

Full text

1 Introduction

With the progressive development of built-in cameras in smart handheld devices, people are generating more amount of digital content than ever before. Aside from capturing moments, such powerful cameras are used for capturing digitized copies of printed documents (*e.g., *important notes, certificates, and visiting cards) for keeping personal records or for sharing the documents with others. This is a popular global trend for its ready availability and relatively easier operational methods than any other digitizing device, *e.g., *a scanner. Although generating a digitized document using a smart handheld devices is as easy as pointing and shooting the document, the quality of the captured digitized image is often a concern. The paper documents having folds cause to create uneven surfaces and result in illumination distorted documents. Additionally, the specular reflections and hard shadows contribute in rendering poor quality digitized documents. Figures 1(a)-1(c) show three examples of illumination distorted digitized documents captured in different lighting conditions using smartphone’s cameras. Even though the scanners are specialized device for producing digitized documents, it also suffer from its own limitations. For example, scanning a page from a thick and bound book suffer from illumination distortions because of the curled area in the spine region [17] (see Fig. 1(d)). Although one can attempt to remove these distortions by pressing the spine of the book harder, such distortions can hardly be eliminated [32, 28].

Correcting illumination distortion of digitized documents is an important task for not only enhancing readability for the readers but also it is crucial for applying optical character recognition (OCR) software on the digitized documents. Various studies in the literature show the importance of rectifying illumination distortion for a significantly better OCR performance [28, 2, 33, 35]. The traditional methods for correcting the illumination distortions of the digitized documents start with estimating the background shades followed by removing the shades using a document surface reconstruction model [34, 36, 13, 10, 19, 20, 18, 4, 30]. These strategies can be divided into two prominent categories: mask-and-interpolation approaches [34, 36, 13, 10], and without-mask-interpolation approaches [19, 20, 18, 4, 30]. The former approaches figure out the location of text, and photo regions on the documents using either edge detection or binarization methods, followed by applying a mask that covers the text regions along with its adjacent area on the surface of the documents. Finally, the masked regions are interpolated to correct the illumination of the background of the digitized documents. The latter approaches use color histograms of the document image patches to aggregate the local regions belonging to the background and interpolate the rest of the regions to identify the shaded regions. As removing the shading artifacts using the well-known surface models is a straightforward task, most of the works in the literature concentrated only on extracting the background layers as accurately as possible.

In this paper, we propose an efficient illumination correction algorithm for removing shading artifacts from the digitized images. The algorithm starts with constructing a topographic surface using the luminance values of the digitized image pixels irrespective of the contents of the documents (*i.e., *text and image). Then, a model representing the dynamics of fluids simulated by a diffusion equation is applied on the topographic surface to estimate the shading artifacts on the documents. After estimating the shading artifacts, the digitized documents are reconstructed using the Lambertian surface model to correct the illumination by removing the shades. An example of a topographic surface representation of a digitized image segment is visualized in Fig. 2. In designing the proposed algorithm, we are influenced by the techniques applied in watershed transform [31, 9, 23]. Accordingly, we named the proposed method called Water-filling algorithm for digitized document illumination correction.

The rest of the paper is organized as follows. In Section 2, we present a review of related works in the literature. In Section 3, the proposed Water-filling algorithm for digitized documents illumination correction is described. The experimental results and comparison studies are presented in Section 4. Finally, the conclusion is presented in Section 5.

2 Related Work

We review the related works in the literature and categorize them into three groups. Each of the groups is discussed in the following subsections.

2.1 Classical Image Binarization Based Approaches

One of the traditional approaches for removing shading artifacts from gray-scale digitized documents is binarization technique based methods [24, 5, 21, 22, 8, 25, 1]. In [24, 22], the authors proposed a locally adaptive binarization technique based method which computes a threshold using local information. Bukhari *et al. *[5] improved the method proposed in [24] by introducing an automatic parameter tuning algorithm using local information. However, it is inevitable that these methods suffer from significant degradation in photographic regions of the digitized documents as such methods convert a gray level pixel to either a black or a white pixel.

2.2 Parametric Reconstruction Based Approaches

Illumination distortion on camera generate digitized documents of uneven surface is addressed in the literature using the 3D reconstruction models. Lu *et al. *[14, 15] proposed a method for removing shading artifacts on digitized documents captured by cameras using the 3D shape reconstruction of the documents. In their method, the 3D shape of the document is reconstructed by fitting the illumination values to a polynomial surface. Meng *et al. *[16] proposed another 3D shape reconstruction based method by isometric mesh construction assuming that the page shape is a general cylindrical surface (GCS). Assuming GCS form for camera captured digitized document images are proven to work well for modeling 3D shape of real documents [11]. Tian *et al. *[29] proposed another 3D reconstruction based method using text region on the digitized documents. In their method, the perspective distortion of the text region is estimated using text orientation and horizontal text line. Although their proposed methods are effective for correctly identify illumination distortions due to uneven surface of the target document, the method has a limitation in rectifying the shading artifacts on the digitized documents in general (e.g., shadow from light occluding object).

Digitized documents generated by scanners experience relatively less shading artifacts as the scanners use a single light source with an uniform light direction. For such digitized documents, the surface of the document is rendered smooth and constant. Such type of digitized document can be considered to have a parametric surface with various assumptions [17, 28, 37]. Such assumptions lead to a straightforward reconstruction of the 3D shapes for digitized documents captured by scanners. The assumption based digitized document illumination correction techniques are not applicable for correcting generalized digitized documents.

2.3 Background Shading Estimation Based Approaches

In background shading estimation based methods, a digitized document is assumed to have two separate layers. The background layer–the shading layer which contains illumination distortions, and the foreground layer–the layer which contains the text and images.

2.3.1 Mask and Interpolation

In mask and interpolation approach, a mask is created to cover the text region by detecting the text on a digitized document. Zhang *et al. *[34, 36] proposed a method to create a mask using the Canny edge detector algorithm followed by using morphological closing operation. This method is effective only for documents which contain the text of a specific size. Documents containing varying font sizes or documents containing large photos result in poor illumination corrected documents. Lee *et al. *[13] proposed another mask and interpolation based method which detects the text and the photo regions with large rectangle masks using the connected edges. This approach is effective for closely captured digitized documents without having any surrounding information. Otherwise, the whole document would be identified as a separate rectangle, hence, that rectangle area would be processed separately. In [10], Gato *et al. *introduced a method to use the binarized image as a mask. However, the generated masks often fail to cover the photo regions on the digitized documents accurately.

2.3.2 Direct Interpolation

In direct interpolation based approaches, image local regions belonging to the background are detected for illumination correction. Oliveira *et al. *[19, 20, 18] assumed that a local region belonging to a digitized document background has a narrow Gaussian shape in its color histogram. With that assumption, initially, the local blocks belonging to the background were identified. Then those blocks were used to estimate the background layer using an interpolation method. Since an image with large size photo has piecewise constant color in the photo region, this method does not work correctly in such cases. In [3], a shadow map is directly generated using local background and text color estimation. Then the shadow map is applied on the document to rectify the shading artifacts, while document images having irregular shading patterns leads the algorithm to a wrong estimation of the global shadow map. Tsoi *et al. *[4, 30] assumed that boundary region of a digitized document has uniform color. The shading of the interior document was estimated by a linear interpolation of shading using document boundary regions. The method also suffers when shading of the boundary regions has irregular shading patterns. Fan *et al. *[7] used a watershed-transform segmentation method on the noise-filtered luminance image to segment the background regions. Then, interpolation is applied on the textual regions to estimate illuminant values throughout the image. Since the segmented region with the largest catchment basins is automatically classified as a background region, the performance of the method from this paper is limited when two background regions are separated.

3 Water-Filling Algorithm

The proposed digitized document illumination correction algorithm, called Water-Filling algorithm, is described in this section. We start the description with a number of terminologies and relevant observation. The topography of a local area depicts the detail physical construction of land which includes elevation and depression. A catchment basin on a topography surface is an extent of a land where the accumulated natural water converges to a meeting point, technically called a minima. An illustration of catchment basins and the corresponding minimum is displayed in Fig. 2 (c). A digitized document, having shading artifacts, can be represented as a topographic surface, where a depression on the surface is considered as a catchment basin. Figure 2 (d) shows an example of a catchment basin appeared on a digitized document due to shading artifact.

An illumination distorted region on a digitized document captured by a camera often occurred due to occluding light sources by the camera itself along with the camera holding hand(s) of the user. In case of a scanner, the illumination distorted region often spread from top to bottom or left to right of the document surface. In both device cases, the shaded regions touch the boundary of the digitized documents. Figure 3 shows a few examples where it can be seen that the shaded region meets the boundary. In the proposed algorithm, we exploit this general observation and summarize that the catchment basins which touch the image boundary correspond to the shading artifacts of the background regions. The other existing catchment basins which do not touch the boundary of the image are in general belonging to the text or photo regions of the documents. We solidify this idea empirically in supplementary material. Based on these observations, let us assume that $D_{I}\subset\mathbb{Z}^{2}$ be a domain of two-dimensional gray-scale image $I$.

In the following sub-sections, the digitized document illumination correction problem is formalized followed by the proposed solution mechanism discussion in detail.

3.1 Modeling by Diffusion Equation

In this sub-section, we model a filling mechanism of catchment basins using water. We use two different methods to simulate this water-filling task: incrementally filling of catchment basins method, and flood-and-effuse method. Ultimately, we link these two methods to provide the final version of the water-filling method.

Before modeling each method using diffusion equation, we need to set a few constraints on a few variables. Let $h$ be the altitude of the topographic surface, $w(x,t)$ be the water level on the topographic surface $x$ at a point of time $t$. Figure 3 illustrates an one-dimensional cartoon model of a catchment basin. Our objective is to restore the background of the digitized document affecting the structure of the document as low as possible. In order to meet the objective, we set two constraints on $w()$ given as follows.

[TABLE]

Equation (1) limits water level to non-negative value since water either be stored or flow out. Equation (2) describes the drop of water at the image boundary so that only the catchment basins at the interior region of the image can be filled. Under these constraints, the dynamics of water can be designed with the following diffusion equation.

[TABLE]

where $\mathbf{J_{w}}$ is the flux of the diffusing water. Equation (3) states that a change in water level in any part of the structure is due to inflow and outflow of water into and out of that part of the structure.

3.1.1 Incremental Filling of Catchment Basins

Let $G(x_{0},t)$ be the overall altitude of water at a location $x_{0}$ at time $t$, which can be written as $G(x_{0},t)=h(x_{0})+w(x_{0},t)$. Assuming that there is a continuous water supply source from the peak $\hat{h}$, the total flow of the water is directly proportional to its relative height at $x$. Formally, we write it as follows.

[TABLE]

Similarly, for the 2D case of the flow dynamics, the partial derivative of $w()$ with respect to time $t$ at $(x_{0},y_{0})$ is proportional to the relative difference of $G()$ at $(x_{0},y_{0})$ and its neighborhood, where $(x,y)$ represents a point on the 2D topographic surface. Following Equation (4), the 2D iterative update formula for $w()$ is given in Equation (5).

[TABLE]

where $\eta$ is a hyperparameter which decides the speed of the process. The $\eta$ has to be chosen carefully as an inappropriate value of $\eta$ might lead to slow convergence or divergence of the water level. Since we use $G()$ values from four-neighborhood at point $(x,y)$, the proper $\eta$ value is less than or equal to $0.25$. The value of $\eta$ greater than $0.25$ cause flooding in the whole structure. The value of $G()$ after convergence is used to estimate the background layer of an image. The $G()$ is converged at time $t$, if $G(x,y,t)-G(x,y,t-\Delta)<\delta$, $\forall x,\forall y$, where $\delta$ should be a small value. In this paper, we have set $\delta=0.01$.

3.1.2 Flood and Effuse

In this method, the diffusion equation can be decomposed into two independent processes: water effusion, and flood. For the effusion process, we consider the dynamics of water flow on the topographic surface without any external water supply. We only consider the effusion of water through the image border. Also, the amount of water at each location never be increased in order to further boost the speed of effusion process. The formulation of this idea for the 1D case is straightforward from Equation (3) and (4) is given as follows.

[TABLE]

The effusive term $w_{\phi}(x_{0},t)$ has non-positive value and represents the amount of water to be effused at each location. Conversely, in the flooding process $w_{\psi}(x_{0},t)$, the model be immersed to the same altitude for few initial moments.

[TABLE]

where $\hat{h}={\max}\ h(x),\forall x$. The partial derivative of the overall water level $w()$ with respect to time $t$ is the sum of $w_{\phi}$ and $w_{\psi}$. The iterative update formula for $w()$ in the 2D case is as follows.

[TABLE]

Once $G()$ converges, it can be used to reconstruct photometrically correct version of the digitized documnet. In Lambertian surface model, the illumination value of an image is equal to the product of foreground text layer and background shading layer [34]. Given the background layer $I_{b}$ and foreground layer $I_{f}$, the final photometrically correct image, $I_{r}$, can be computed as follows.

[TABLE]

where $\ell\in[0,1]$ for tuning the brightness of the output image. A few example of simulation results of the above two methods are shown in Fig. 4. It is notable that the results from the flood method show the desired dynamics precisely – once the overall image is filled with water, it gradually effuses water at the image boundary.

Although the incremental filling method removes shading artifacts quite well, the photo regions are not clearly restored because the filling process at the large catchment basins are too slow. Meanwhile, it is observed that the overall image is reconstructed correctly with the flood-and-effuse method.

3.1.3 Boosting with Downsampling

The time complexity of the proposed algorithm is $\mathcal{O}(I_{width}\times I_{height}\times t)$. Since both of the Equations (5) and (8) are pixel-wise operations, the processing time increases as the image size gets larger, which hinders these methods from being used in real-time applications. Although applying either equation after downsampling the input document size at a rate of $k_{s}$ can boost the processing time significantly, we observe that background shadings on small catchment basins may not be estimated correctly due to aliasing effect. However, shading artifacts in a small region do not have a drastic effect on the overall digitized document illumination correction. Thus, we apply the flood-and-effuse method on the downsampled input document only to attain the rough approximation quickly. After estimating the background artifacts, using bicubic interpolation the background layer is approximated to the original size. Then, the incremental filling method is used to reconstruct the details of the rectified digitized document. In the following section, we show that this combined scheme produces reliable results in a shorter time with acceptable peak signal-to-noise ratio (PSNR). The pseudocode of the proposed algorithm is given in Algorithm 1.

4 Experiments and Performance Evaluation

4.1 Datasets

Since there is no publicly available digitized document illumination distortion dataset, we have created our own to compare the performance of our proposed method with the state-of-the-art methods. Primarily our dataset consists of $159$ illumination distorted digitized documents, among them $109$ images are captured using two smartphone’s cameras and the rest $50$ images are captured using a scanner. The size of camera captured images are $3264\times 2448$, and the scanned images are $3455\times 2464$ ($72$ dpi). The digitized document captured by cameras are generated under different lighting conditions. For PSNR comparison, we captured $87$ ground-truth images along with its illumination distorted digitized images. In order to capture ground-truth images, a camera stand is used in a well-lit room to capture a ground-truth photo followed by capturing another image by adding shades on the document surface by occluding light sources intentionally. We present the benchmark dataset in our supplementary material. We further collected $22$ business card images from a publicly available dataset [6] for OCR edit-distance comparison. The proposed algorithm is implemented using Microsoft C++ development environment. For experiments, we use a workstation having an Intel Corei7-4790k, $64$ bits CPU with $16$ GB memory and $128$ SSD storage device.

4.2 Methods for Comparison

The performances of the proposed algorithm have been compared with several state-of-the-art methods in the literature. For comparing the performance on the digitized documents captured by smartphones, we select a binarization method (Sauvola *et al. *[24]), a mask and interpolation method (Zhang *et al. *[34]), two direct interpolation methods (Oliveira *et al. *[18], Bako *et al. *[3] and Kligler *et al. *[12]), and a rolling-ball based method (Sternberg *et al. *[27]). The rolling ball based method estimates a local background value for every pixel by taking an average over a large ball around the pixel. For comparing the performance on digitized documents captured by scanner devices, we select the methods proposed in [24], [27], [18], [17] and [34].

4.3 Parameters Tuning

There are two hyperparameters to be set in our algorithm for generating optimum results, namely sampling rate $k_{s}$ and the brightness factor $\ell$. In order to do that, first of all, we conduct an experiment using the first 20 distorted images along with their ground-truth images. We need to set an acceptable $k_{s}$ to correct shading artifacts fast by meeting our objective, that is we want estimate shadow as clearly as possible. Too large $k_{s}$ makes the algorithm preform faster at a cost of poor quality output. In order to produce acceptable results using the proposed algorithm, we need to look after two crucial components, the PSNR in the corrected images in comparison to the ground-truths, and the average elapsed time to accomplish illumination rectification. Accordingly, we conduct two experiments for selecting an acceptable $k_{s}$ which are reported in Fig. 5. It is obvious that, by selecting a larger $k_{s}$ makes the algorithm perform faster by a small margin. However, that would cost the algorithm performing poorly by decreasing significantly large amount of PSNR. Accordingly, we select $k_{s}=5$ for the rest of the experiments as the default sampling rate.

The brightness of the output image is also an important factor. The output document should be comfortable to the human eyes–too bright and too dark both are undesirable. To determine the correct brightness factor, we conduct another experiment to measure PSNR for changing brightness factor $\ell$. Figure 5 (c) shows the experiment results. As it can be seen, for brightness factor $\ell=0.85$, we achieve optimum results. Based on this experiment, we set $\ell=0.85$ for the rest of the experiments to be done.

4.4 Results Analysis

After tuning the hyperparameters of the proposed method, we conduct a visual comparison of the illumination corrected results of ours method and that of a number of existing methods. Figure 6 shows an example digitized documents along with the ground-truth and the corresponding illumination corrected results performed by several methods. For [18, 3, 12], false detection of background region results in unclear rectification of illumination distortions along the shadow borders. Moreover, their method often adds additional noises in the text and image regions which cause in destroying the structures of the documents. Another important concern of method [18] is that the photo regions on the documents render into dark patches. For the image region on the document, the method [34] fails to preserve the structure and color, in fact, it ruins the visual quality in those regions.

In order to have a fairer comparison, the methods are applied to the ground-truth data to compare the PSNR. Figure 7 summarizes the performances of the methods on ground-truth data where it can be seen that the proposed method outperforms the other compared methods in terms of PSNR. The closest PSNR to our method is achieved by method [27] with the parameter ball size $=50$. However, the PSNR difference between these two method is quite significant and our method is proved to be superior in producing better illumination corrected digitized documents.

The average time requires to produce a rectified document is another important criteria to be measured. From the experimental results shown in Fig. 7, the method [24] take least amount of average time ($3\times 10^{-2}$ sec.) to accomplish the task, while the method [3] takes the highest amount of average time to correct illumination of a document. Among the rest of the reported methods, as outputs produced by method [24] are severely distorted, we can disregard their performance. Among the rest, our proposed method, using sampling rate $k_{s}=5$, takes $1.4$ seconds to accomplish the task. Although the elapsed time ($1.4$ second) for this setting is higher than the elapsed time of the other remaining methods, by setting the sampling rate $k_{s}=14$, our method can accomplish the task in $0.1$ second. In that case, the documents have to suffer from more noise (PSNR 22.04, see Fig. 5) than that of sampling rate $k_{s}=5$. However, the PSNR level is still better than that of methods [18], [34], and [27] (see Fig. 7). That means, even if a coarse sampling rate is chosen for the proposed algorithm, the output quality of the documents is still better than the other compared methods.

While our method shows superior performance over other compared methods presented in the previous sub-section, it is also important to examine whether the structure of characters is well preserved after illumination correction. To this end, we measure the OCR performance for each illumination corrected output, as an indication of readability of the methods. Before applying OCR, the corrected digitized documents are geometrically de-warped using inverse perspective projective mapping by manually selecting four corner points of each document. Then, we apply an online OCR implementation of [26] to recognize texts on the documents. In order to compare the performance of our algorithm with the state-of-the-art methods, we measure the average edit-distance between the OCR results and the original text written on the cards. The edit-distance measure is reported in Fig. 9. As it can be seen, the average edit-distance measure of the text on the illumination corrected digitized business cards by our algorithm is $9.38$ while the nearest edit-distance measure is performed by method [12] having the performance score $10.74$. Besides, the proposed method outperforms the other reported method by a significantly large margin. Based on the experiments using OCR results, we strongly claim that the proposed method is not only capable of correcting illumination of the digitized documents but also keep the structure of the documents well preserved, which helps the user for experiencing better readability using the digitized document corrected by our proposed algorithm. We provide further experimental results (*e.g., *scanner) in supplementary material.

5 Conclusions

In this paper, we have proposed a novel algorithm, called water-filling algorithm, for illumination correction of shaded digitized documents. The proposed water-filling algorithm is based on simulating the immersion process of a topographic surface using water. We have implemented this idea using a diffusion equation. To the best of our knowledge, no previous method used diffusion equation based approach for digitized document illumination correction. Based on the experimental results, it is found that the proposed method can outperform the performance of the previous methods under various challenging lighting conditions. The limitation of the proposed algorithm is that any text or photo connected to image border is regarded as a shadow, and eliminated after the processing. However, we have observed that this is not a critical problem since the salient text or photo is usually not connected to image borders in a document image. We also need to mention that our method often produces unsatisfactory results for eliminating specular lights in an image, because once the original luminance value of a point in the foreground layer is significantly damaged due to overexposure, it is hard to reconstruct the point using the Lambertian surface model even if we precisely estimated the background layer. These limitations will be addressed in our future works.