Reweighted Infrared Patch-Tensor Model With Both Non-Local and Local Priors for Single-Frame Small Target Detection

TL;DR

This paper introduces a novel infrared patch-tensor model that combines local and non-local priors for improved single-frame small target detection in complex backgrounds, outperforming existing methods especially with dim targets and clutter.

Contribution

It proposes a reweighted infrared patch-tensor model utilizing both local structure and non-local self-correlation priors with an adaptive weighting scheme for enhanced detection.

Findings

Outperforms state-of-the-art methods on challenging images

Effective in detecting very dim targets and heavy clutter

Utilizes a reweighted robust PCA approach with adaptive weights

Abstract

Many state-of-the-art methods have been proposed for infrared small target detection. They work well on the images with homogeneous backgrounds and high-contrast targets. However, when facing highly heterogeneous backgrounds, they would not perform very well, mainly due to: 1) the existence of strong edges and other interfering components, 2) not utilizing the priors fully. Inspired by this, we propose a novel method to exploit both local and non-local priors simultaneously. Firstly, we employ a new infrared patch-tensor (IPT) model to represent the image and preserve its spatial correlations. Exploiting the target sparse prior and background non-local self-correlation prior, the target-background separation is modeled as a robust low-rank tensor recovery problem. Moreover, with the help of the structure tensor and reweighted idea, we design an entry-wise local-structure-adaptive and sparsity enhancing weight to replace the globally constant weighting parameter. The decomposition could be achieved via the element-wise reweighted higher-order robust principal component analysis with an additional convergence condition according to the practical situation of target detection. Extensive experiments demonstrate that our model outperforms the other state-of-the-arts, in particular for the images with very dim targets and heavy clutters.

Tables

Table 1. TABLE I: Mathematical Notations

Notation	Explanation
$𝓧,𝑿,𝒙,x$	tensor, matrix, vector, scalar.
${𝑿}_{\left(n\right)}$	mode-$n$ matricization of tensor $𝓧\in {ℝ}^{I_1\times I_2\times \cdots \times I_N}$, obtained by arranging the mode-$n$ fibers as the columns of the resulting matrix of size ${ℝ}^{I_n\times {\prod }_{k\ne n}{I}_k}$.
$\mathrm{vec}\left(𝓧\right)$	vectorization of tensor $𝓧$.
$⟨𝓧,𝓨⟩$	inner product of tensor $𝓧$ and $𝓨$, which is defined as $⟨𝓧,𝓨⟩:=\mathrm{vec}{\left(𝓧\right)}^{\top }\mathrm{vec}\left(𝓨\right)$.
${\Vert 𝓧\Vert }_0$	${\ell }_0$ norm of tensor $𝓧$ which counts the number of non-zero elements.
${\Vert 𝓧\Vert }_1$	${\ell }_1$ norm of tensor $𝓧$.
${\Vert 𝓧\Vert }_F$	Frobenius norm of tensor $𝓧$, which is defined by ${\Vert 𝓧\Vert }_F:=\sqrt{⟨𝓧,𝓧⟩}$.
${\mathrm{fold}}_i\left(𝑿\right)$	returns tensor $𝓩$ that ${𝒁}_{\left(i\right)}=𝑿$.
${\Vert 𝑿\Vert }_{\ast }$	nuclear norm of matrix $𝑿$, which is defined by ${\Vert 𝑿\Vert }_{\ast }={\sum }_i{𝝈}_i$, where the SVD of $𝑿=𝑼\mathrm{diag}\left(𝝈\right){𝑽}^{\top }$.
${𝒮}_{\mu }\left(𝒙\right)$	element-wise shrinkage operator is defined as ${𝒮}_{\mu }\left(𝒙\right):=\mathrm{sign}\left(𝒙\right)\max (\left|𝒙\right|-\mu ,0)$. ${𝒮}_{\mu }\left(𝒙\right)$ is the closed-form solution of the problem: $\widehat{𝒚}=\arg {\min }_{𝒚}{\Vert 𝒙-𝒚\Vert }_F^2+\lambda {\Vert 𝒚\Vert }_1$ [31].
${𝒟}_{\mu }\left(𝑿\right)$	matrix singular value thresholding operator: ${𝒟}_{\mu }\left(𝑿\right):=𝑼\mathrm{diag}\left(\overline{\sigma }\right){𝑽}^{\top }$, where $𝑿=𝑼\mathrm{diag}\left(\sigma \right){𝑽}^{\top }$ is the SVD of $𝑿$ and $\overline{\sigma }:=\max (\sigma -\mu ,0)$. ${𝒟}_{\mu }\left(𝑿\right)$ is the closed-form solution of the problem: $\widehat{𝒀}=\arg {\min }_{𝒀}{\Vert 𝑿-𝒀\Vert }_F^2+\frac{\mu }{2}{\Vert 𝒀\Vert }_{\ast }$ [32].
${𝒯}_{i,\mu }\left(𝓧\right)$	${𝒯}_{i,\mu }\left(𝓧\right):={\mathrm{fold}}_i\left({𝒟}_{\mu }\left({𝑿}_{\left(i\right)}\right)\right)$.

Table 2. TABLE II: Target and background characteristics of real infrared image sequences

	# Frame	Image Resolution	Target Shape	Target characteristics	Background characteristics
Sequence 1-4	400	$255\times 320$	Gaussian	⌑   Tiny, very dim with low contrast.    ⌑   Moves along the cloud edges or     buried in the cloud.	⌑   Strong undulant cloud backgrounds.    ⌑   Approximately noise-free.
Sequence 5	30	$200\times 256$	Rectangular	⌑   Small due to long imaging distance.    ⌑   Brightness, contrast, and size vary a lot.	⌑   Heavy cloudy-sky background clutters     in uniform backgrounds with noise.

Table 3. TABLE III: Detailed parameter settings for twelve tested methods

No.	Methods	Acronyms	Parameter settings
1	Max-Median filter [10]	Max-Median	Support size: $5\times 5$
2	Top-Hat method [48]	Top-Hat	Structure shape: square, structure size: $3\times 3$
3	Phase spectrum of Fourier Transform [49]	PFT	Disk radius: $3$
4	Multiscale Patch-based Contrast Measure [19]	MPCM	$N=1,3,\dots ,9$
5	Weighted Local Difference Measure [23]	WLDM	$L=4,m=2,n=2$
6	Two-Dimensional Least Mean Square filter [9]	TDLMS	Support size: $5\times 5$, step size: $\mu =5\times {10}^{-8}$
7	Infrared Patch-Image Model [26]	IPI	Patch size: $50\times 50$, sliding step: $10$, $\lambda =\frac{L}{\sqrt{\min (IJ,P)}}$, $L=3$, $\epsilon ={10}^{-7}$
8	Patch-level RPCA method [50]	PRPCA	Patch size: $50\times 50$, sliding step: $10$, $\lambda =\frac{L}{\sqrt{\min (IJ,P)}}$, $L=3$, $\epsilon ={10}^{-7}$
9	Weighted Infrared Patch-Tensor Model [29]	WIPI	Patch size: $51\times 51$, sliding step: $10$, smoothing parameter $h=15$, $\epsilon ={10}^{-7}$
10	Non-negative IPI model via Partial Sum minimization of singular values [30]	NIPPS	Patch size: $50\times 50$, sliding step: $10$, $\lambda =\frac{L}{\sqrt{\min (IJ,P)}}$, $L=2$, $r=5\times {10}^{-3}$
11	Infrared Patch-Tensor Model	IPT	Patch size: $50\times 50$, sliding step: $10$, $\lambda =\frac{L}{\sqrt{\max (I,J,P)}}$, $L=3$
12	Reweighted Infrared Patch-Tensor Model	RIPT	Patch size: $50\times 50$, sliding step: $10$, $\lambda =\frac{L}{\sqrt{\min (I,J,P)}}$, $L=1$, $h=10$, $ϵ=0.01$, $\epsilon ={10}^{-7}$

Table 4. TABLE IV: Algorithm Complexity and Computational Time Comparisons of Different Methods

	TDLMS	PFT	MPCM	WLDM	IPI	WIPI	NIPPS	IPT	WIPT	RIPT
Complexity	$𝒪\left(L^{2}MN\right)$	$𝒪\left(MN\log MN\right)$	$𝒪\left(L^{3}MN\right)$	$𝒪\left(L^{3}MN\right)$	$𝒪\left(m{n}^2\right)$	$𝒪\left(m{n}^2\right)$	$𝒪\left(m{n}^2\right)$	$𝒪\left(m{n}^2\right)$	$𝒪\left(m{n}^2\right)$	$𝒪\left(m{n}^2\right)$
Time (s)	0.162	0.025	0.083	6.059	16.998	52.995	15.515	8.598	6.932	3.169

Table 5. TABLE V: Quantitative comparison of the tested methods for the representative images of Sequence 1 1 1 - 4 4 4

	$65$th frame of Sequence $1$				$52$th frame of Sequence $2$				$53$th frame of Sequence $3$				$56$th frame of Sequence $4$
Method	LSNRG	SCRG	BSF		LSNRG	SCRG	BSF		LSNRG	SCRG	BSF		LSNRG	SCRG	BSF
Max-Median	5.49	10.69	12.10		1.87	5.46	7.37		2.96	6.21	11.27		7.54	9.81	16.66
Top-Hat	3.47	13.47	12.34		2.10	12.55	8.08		3.10	9.48	11.24		4.04	22.13	21.06
PFT	4.83	53.01	7.43		1.37	10.96	3.22		0.68	7.48	3.38		9.16	113.25	18.77
MPCM	1.48	7.69	3.46		1.62	11.84	15.98		0.38	1.91	2.15		1.88	14.17	4.68
WLDM	0.87	2.00	1.99		2.22	9.95	12.23		1.94	8.19	3.95		3.11	12.11	3.56
TDLMS	1.36	3.44	3.53		1.76	4.27	3.38		2.61	4.39	4.49		1.99	4.77	4.50
IPI	220.38	5215.82	19256.20		10.72	104.34	172.90		Inf	Inf	Inf		Inf	2788.19	4939.03
PRPCA	5.17	382.58	20179.12		1.30	26.88	2628.42		1.68	31.85	1982.80		2.83	267.31	20966.41
CWRPCA	2.67	36.77	602.45		4.62	40.59	58.23		7.69	98.57	201.89		52.92	441.65	2065.42
NIPPS	15.95	315.08	670.65		3.89	66.99	81.05		20.59	343.06	735.19		87.92	2280.13	3103.00
IPT	9.80	2096.70	87797.82		2.14	332.56	17488.27		3.22	Inf	Inf		2.86	Inf	Inf
RIPT	Inf	Inf	Inf		Inf	Inf	Inf		Inf	Inf	Inf		Inf	Inf	Inf

Equations

$$f_{\text{F}}=f_{\text{B}}+f_{\text{T}}+f_{\text{N}},$$

$$\mathcal{F}=\mathcal{B}+\mathcal{T}+\mathcal{N},$$

$$\text{rank}({\mathbf{B}}_{(1)})\le r_1,\text{leavevmode}\text{rank}({\mathbf{B}}_{(2)})\le r_2,\text{leavevmode}\text{rank}({\mathbf{B}}_{(3)})\le r_3,$$

$$\Vert \mathcal{T}{\Vert }_0\le k,$$

$$\Vert \mathcal{F}-\mathcal{B}-\mathcal{T}{\Vert }_{\text{F}}\le \delta .$$

$$\underset{\mathcal{B},\mathcal{T}}{\min }\text{rank}(\mathbf{B})+\lambda \Vert \mathcal{T}{\Vert }_0,\text{leavevmode}s.t.\text{leavevmode}\mathcal{B}+\mathcal{T}=\mathcal{F}.$$

$$\underset{\mathcal{B},\mathcal{T}}{\min }\sum _{i=1}^3\Vert {\mathbf{B}}_{(i)}{\Vert }_{\ast }+\lambda \Vert \mathcal{T}{\Vert }_1,\text{leavevmode}s.t.\text{leavevmode}\Vert \mathcal{F}-\mathcal{B}-\mathcal{T}{\Vert }_{\text{F}}\le \delta .$$

$${\mathbf{J}}_{\alpha }(\nabla u_{\sigma })=G_{\alpha }\ast (\nabla {\mathbf{u}}_{\sigma }\otimes \nabla {\mathbf{u}}_{\sigma })=\left(\begin{array}{cc}{J}_{11}& J_{12}\\ J_{21}& J_{22}\end{array}\right),$$

$${\lambda }_1,{\lambda }_2=\left(J_{11}+J_{22}\right)\pm \sqrt{{\left(J_{22}-J_{11}\right)}^2+4J_{12}^2}.$$

$${\mathcal{W}}_{\mathrm{LS}}=\text{exp}\left(h\cdot \frac{\left({\mathcal{L}}_1-{\mathcal{L}}_2\right)-d_{\min }}{d_{\max }-d_{\min }}\right),$$

$$\underset{\mathcal{B},\mathcal{T}}{\min }\sum _{i=1}^3\Vert {\mathbf{B}}_{(i)}{\Vert }_{\ast }+\lambda \Vert {\mathcal{W}}_{\mathrm{LS}}\odot \mathcal{T}{\Vert }_1,\text{leavevmode}s.t.\text{leavevmode}\Vert \mathcal{F}-\mathcal{B}-\mathcal{T}{\Vert }_{\text{F}}\le \delta ,$$

$${\mathcal{W}}_{\mathrm{SE}}^{k+1}(i,j,p)=\frac{1}{|{\mathcal{T}}^k(i,j,p)|+ϵ},$$

$${\mathcal{W}}_{\mathrm{SE}}^{k+1}(i,j,p)$$

$$=\{\begin{array}{ll}\frac{1}{{\mathcal{T}}^k(i,j,p)+ϵ},& \text{if }{\mathcal{T}}^k(i,j,p)>0;\\ \infty ,& \text{if }{\mathcal{T}}^k(i,j,p)\le 0,\end{array}$$

$${\mathcal{W}}^k={\mathcal{W}}_{\mathrm{LS}}\odot {\mathcal{W}}_{\mathrm{SE}}^k.$$

$$\underset{\mathcal{B},\mathcal{T}}{\min }\sum _{i=1}^3\Vert {\mathbf{B}}_{(i)}{\Vert }_{\ast }+\lambda \Vert \mathcal{W}\odot \mathcal{T}{\Vert }_1,\text{leavevmode}s.t.\text{leavevmode}\mathcal{B}+\mathcal{T}=\mathcal{F}.$$

$$\mathcal{L}=$$

$$\sum _{i=1}^N\frac{1}{2\mu }\Vert {\mathcal{B}}_i+\mathcal{T}-\mathcal{F}{\Vert }^2-⟨{\mathcal{Y}}_i,{\mathcal{B}}_i+\mathcal{T}-\mathcal{F}⟩,$$

$${\mathcal{B}}_i^{k+1}=\arg \underset{{\mathcal{B}}_i}{\min }{\Vert {\mathbf{B}}_{i,(i)}\Vert }_{\ast }+\frac{1}{2\mu }{\Vert {\mathcal{B}}_i-(\mathcal{F}+\mu {\mathcal{Y}}_i^k-{\mathcal{T}}^k)\Vert }_{\text{F}}^2$$

$${\mathcal{T}}^{k+1}=$$

$$\sum _{i=1}^N\frac{1}{2\mu }{\Vert \mathcal{T}-\left(\mathcal{F}+\mu {\mathcal{Y}}_i^k-{\mathcal{B}}_i^{k+1}\right)\Vert }_{\text{F}}^2$$

$${\mathcal{Y}}_i^{k+1}={\mathcal{Y}}_i^k+\frac{1}{{\mu }^k}\left(\mathcal{F}-{\mathcal{B}}_i^{k+1}-{\mathcal{T}}^{k+1}\right),\text{leavevmode}i=1,\cdots ,N.$$

$${\mathcal{B}}_i^{k+1}$$

$${\mathcal{T}}^{k+1}$$

$$t_{\text{up}}=\max (v_{\min },{\bar{f}}_{\text{T}}+k\sigma ),$$

$$\text{LSNRG}=\frac{{\text{LSNR}}_{\text{out}}}{{\text{LSNR}}_{\text{in}}},$$

$$\text{BSF}=\frac{{\sigma }_{\text{in}}}{{\sigma }_{\text{out}}},$$

$$\text{SCRG}=\frac{{\text{SCR}}_{\text{out}}}{{\text{SCR}}_{\text{in}}},$$

$$P_{\text{d}}$$

$$F_{\text{a}}$$

Full text

I Introduction

Infrared small target detection is a key technique for many applications, including early-warning system, precision guided weapon, missile tracking system, and maritime surveillance system [1, 2, 3]. Traditional sequential detection methods, such as 3D matched filter [4], improved 3D filter [5], and multiscan adaptive matched filter [6], are workable in the case of static background, exploiting the target spatial-temporal information. Nevertheless, with the recent fast development of high-speed aircrafts [7] like anti-ship missiles, the imaging backgrounds generally change quickly due to rapid relative motion between the imaging sensor and the target. The performance of the spatial-temporal detection method degrades rapidly. Therefore, the research of single-frame infrared small target detection is of great importance and has attracted a lot of attention in recent years.

Different from general object or saliency detection tasks, the main challenge of infrared small target detection is lacking enough information. Due to the long imaging distance, the target is always small without any other texture or shape features. As the target type, imaging distance, and neighboring environment differ a lot in real scenes, the target brightness could vary from extremely dim to very bright (see fig. 5 for example). In the absence of spatial-temporal information and the target features like shape and size, the characteristics of the background [8] and the relation between the background and target are very important priors for single-frame infrared small target detection. Thus how to design a model to incorporate and exploit these priors is vital for infrared small target detection in a single image.

I-A Prior work on single-frame infrared small target detection

The previously proposed single-frame infrared small target detection methods could be roughly classified into two categories. In the first type, a local background consistent prior is exploited, assuming the background is slowly transitional and nearby pixels are highly correlated. As a result, the target is viewed as the one that breaks this local correlation. Under this assumption, the classical methods, such as two-dimensional least mean square (TDLMS) filter [9] and Max-Median filter [10], enhance the small target by subtracting the predicted background from the original image. Unfortunately, besides the targets, they enhance the edges of the sky-sea surface or heavy cloud clutter as well, since these structures also break the background consistency as the target does. To differentiate the real target and high-frequency change, some edge analysis approaches [11, 12] have been proposed to extend these methods to estimate the edge direction in advance and preserve the edges. Bai et al. [13] designed a new Top-Hat transformation using two different but correlated structuring elements. Another class of local prior based methods exploits the local contrast, which is computed by comparing a pixel or a region only with its neighbors. The seminal work of Laplacian of Gaussian (LoG) filter based method [14] has motivated a broad range of studies on the Human Visual System (HVS), and has led to a series of HVS based methods, e.g., Difference of Gaussians (DoG) [15], second-order directional derivative (SODD) filter [16], local contrast measure (LCM) [17], improved local contrast measure (ILCM) [18], multiscale patch-based contrast measure (MPCM) [19], multiscale gray difference weighted image entropy [20], improved difference of Gabors (IDoGb) [21], local saliency map (LSM) [22], weighted local difference measure (WLDM) [23], local difference measure (LDM) [24], etc.

The second type of single-frame infrared small target detection methods which has not been explored extensively, exploits the non-local self-correlation property of background patches, assuming that all background patches come from a single subspace or a mixture of low-rank subspace clusters. Then, target-background separation can be realized with the low-rank matrix recovery [25]. Essentially, this type of methods attempts to model the infrared small target as an outlier in the input data. To this end, Gao et al. [26] generalized the traditional infrared image model to a new infrared patch-image model via local patch construction. Then the target-background separation problem is reformulated as a robust principal component analysis (RPCA) [27] problem of recovering low-rank and sparse matrices, achieving a state-of-the-art background suppressing performance. To correctly detect the small target located in a highly heterogeneous background, He et al. [28] proposed a low-rank and sparse representation model under the multi-subspace-cluster assumption.

I-B Motivation

Existing methods detect the infrared small target by either utilizing the local pixel correlation or exploiting the non-local patch self-correlation. From our observation, the unsatisfying performance of local prior methods [20, 23] in detecting the dim target under complicated background mainly lies in their imperfect grayscale based center-difference measures. The saliency of a dim but true target would be easily overwhelmed by the measured saliency of some rare structures. We call this phenomenon the rare structure effect. In contrast, the non-local prior methods [26, 29, 30] suffer from the salient edge residuals. Its intrinsic reason is because the strong edge is also a sparse component as the same as the target due to lack of sufficient similar samples. Since the target might be dimmer than the edge, they would simultaneously be wiped out by simply increasing the threshold.

Our key observation is that the non-local prior and local prior are not equivalent, and in fact they are complementary for the problem of infrared small target detection, as illustrated in fig. 1. The often appearing false alarm components in local (non-local) prior methods could be well suppressed by the non-local (local) prior methods. For example, the stubborn strong edges in the non-local prior based methods, can be easily identified by the local edge analysis approaches. Naturally, an intuitive way to solve above-mentioned dilemma is to extract the local structure information and merge it into the non-local prior based detection framework. Therefore, how to simultaneously and appropriately utilize both the local and non-local priors has been an important issue for improving the detection performance under very complex backgrounds. To the best of our knowledge, very few single-frame infrared small target detection methods concern this problem.

To address this issue, we propose a single-frame small target detection framework with reweighted infrared patch-tensor model (RIPT). Our main contributions consist of the following three folds:

To dig out more information from the non-local self-correlationship in patch space, we generalize the patch-image model to a novel infrared patch-tensor model (IPT) and formulate the target-background separation task as an optimization problem of recovering low-rank and sparse tensors. 2. 2.

To incorporate the local structure prior into the IPT model, an element-wise weight is designed based on structure tensor, which helps to suppress the remaining edges and preserve the dim target simultaneously. 3. 3.

To reduce the computing time, we adopt a reweighted scheme to enhance the sparsity of the target patch-tensor. Considering the particularity of infrared small target detection, an additional stopping criterion is used to avoid excessive computation.

The proposed RIPT model is validated on different real infrared image datasets. Compared with the state-of-the-art methods, our proposed model achieves a better background suppressing and target detection performance.

The remaining of this paper is organized as follows. We propose the non-local correlation based IPT model in Section II. The details of the local structure weight construction are described in Section III. In Section IV, we further propose the reweighted IPT model and its detailed optimization scheme is also provided. Section V presents detailed experimental results and discussion. Finally we conclude this paper in Section VI.

II Non-local correlation driven infrared patch-tensor model

To dig out more spatial correlationships, we develop a novel target-background separation framework named infrared patch-tensor model in this section. Before describing the details, several mathematical notations are defined first in table I.

Given an infrared image, it could be modeled as a linear superposition of target image, background image and noise image:

[TABLE]

where $f_{\textup{F}}$, $f_{\textup{B}}$, $f_{\textup{T}}$, and $f_{\textup{N}}$ represent the input image, background image, target image, and noise image, respectively. Via a window sliding from the top left to the bottom right over an image, we stack the patches into a 3D cube (see the construction step in fig. 4). Then eq. 1 is transferred to the patch space with spatial structure preserved:

[TABLE]

where $\bm{\mathcal{F}},\bm{\mathcal{B}},\bm{\mathcal{T}},\bm{\mathcal{N}}\in\mathbb{R}^{I\times J\times P}$ are the input patch-tensor, background patch-tensor, and target patch-tensor, respectively. $I$ and $J$ are the patch height and width, $P$ is the patch number.

Background patch-tensor $\bm{\mathcal{B}}$. Generally, the background is considered as slowly transitional, which implies high correlations among both the local and non-local patches in the image, as illustrated in fig. 2(a). Although patches $p_{1},p_{2},p_{3}$ locate in the different region of the image, they are equivalent. Based on this non-local correlationship, the state-of-the-art IPI model imposed the low-rank constraint to background patch-image. As a patch-image is the mode-$3$ unfolding matrix of a patch-tensor, the patch-image model could be viewed as a special case of the proposed patch-tensor model essentially. Since the main difficulty of detecting the infrared small target in a single image is lacking enough information, only considering the low-rank structure in one unfolding is insufficient to deal with highly difficult scenes. Naturally, it motivates us to think whether we can utilize the other two unfolding modes. Actually, the mode-$1$ and mode-$2$ unfolding matrices of the infrared patch-tensor are also low-rank. In fig. 2(b) – (d), the singular values of all the unfolding matrices rapidly decrease to zero, which demonstrates that every unfolding mode of the background patch-tensor is intrinsically low-rank. Therefore, we can consider the background patch-tensor $\bm{\mathcal{B}}$ as a low-rank tensor, and their unfolding matrices are also all low-rank defined as:

[TABLE]

where $r_{1}$, $r_{2}$, and $r_{3}$ all are constants, representing the complexity of the background image. The larger their values are, the more complex the background is.

Target patch-tensor $\bm{\mathcal{T}}$. Since the small target merely occupies several pixels in the whole image, the target patch-tensor is an extremely sparse tensor in fact. That is:

[TABLE]

where $k$ is a small integer determined by the number and size of the small target.

Noise patch-tensor $\bm{\mathcal{N}}$. In this paper, we just assume that the noise is additive white Gaussian noise and $\|\bm{\mathcal{N}}\|_{\textup{F}}\leq\delta$ for some $\delta>0$. Thus we have

[TABLE]

It should be noted that although the values of parameters $k,r,\delta$ are different depending on the images, we would not directly use them in the following sections.

Ideally, we would like to obtain a low-rank and sparse decomposition and solve the following problem:

[TABLE]

Unfortunately, the rank computation of a given tensor is a NP-hard problem in general [33]. In Ref. [34], Goldfarb and Qin proposed the Higher-order RPCA (robust tensor recovery) through replacing the rank by a convex surrogate Tucker-rank $\textup{CTrank}(\bm{\mathcal{B}})$, and $\|\bm{\mathcal{T}}\|_{0}$ by $\|\bm{\mathcal{T}}\|_{1}$ to make the above problem tractable. In the singleton model, the tensor rank regularization term is defined as the sum of all the nuclear norms of the mode-$i$ unfoldings, i.e., $\textup{CTrank}(\bm{\mathcal{B}})=\sum_{i}\|\bm{B}_{(i)}\|_{*},\ i=1,2,3$. With this relaxation, our proposed IPT model with random noise assumption can be solved by minimizing the following convex problem:

[TABLE]

$\lambda$ is a weighting parameter that controls the global trade-off between the background patch-tensor and the target patch-tensor. Larger $\lambda$ can shrink those non-target but sparse components to zeros in the target patch-tensor. Nevertheless, it will also over-shrink the dim target which should be preserved. On the contrary, a smaller $\lambda$ does retain the dim target, but it keeps the strong cloud edges as well. Therefore, adopting a global constant weighting parameter $\lambda$ is not a reasonable scheme for detecting the infrared small target in a complex scene. Naturally, it motivates us to design an entry-wise weighting scheme.

III Incorporating local structure prior

In this section, we focus our emphasis on combining the local structure prior and non-local correlation prior together into our model. we construct a local structure weight and interpret it as an edge salience measure. For the sake of simplicity, the local structure weight is designed on the basis of the image structure tensor. Structure tensor is widely used in many partial differential equation (PDE) based methods [35, 36] to estimate the local structure information in the image, including edge orientation. To integrate the local information, the structure tensor is constructed based on a local regularization of a tensorial product, which is defined as

[TABLE]

where $\bm{u}_{\sigma}$ is a Gaussian-smoothed version of a given image $\bm{u}$. $\sigma>0$ is the standard deviation of the Gaussian kernel which denotes the noise scale, making the edge detector ignorant of small details. $\bm{J}_{\alpha}$ is a symmetric and positive semi-definite matrix, which has two orthonormal eigenvectors denoted as $\bm{w}$ and $\bm{v}=\bm{w}^{\bot}$, where $\bm{w}=\left(2J_{12},J_{22}-J_{11}+\sqrt{\left(J_{22}-J_{11}\right)^{2}+4J_{12}^{2}}\right)^{\top},\bm{w}=\bm{w}/|\bm{w}|$. $\bm{w}$ points to the maximum contrast direction of the geometry structure while $\bm{v}$ points to the minimum direction [37]. Their corresponding eigenvalues $\lambda_{1}$ and $\lambda_{2}$ can be calculated via

[TABLE]

These two values can be used as two feature descriptors of the local geometry structure, where at the flat region, $\lambda_{1}\approx\lambda_{2}\approx 0$; at the edge region, $\lambda_{1}\gg\lambda_{2}\approx 0$; at the corner region, $\lambda_{1}\geq\lambda_{2}\gg 0$. For the sake of low computational cost, we take $\lambda_{1}-\lambda_{2}$ as the edge awareness feature since its value of the edge pixel is much larger than that belongs to the flat region and corner. By applying eq. 8 and eq. 9 to every pixel in the input image $f_{\text{F}}$, two matrices $\bm{L}_{1}$ and $\bm{L}_{2}$ can be obtained, which consists of the large and small structure tensor eigenvalues of all the pixels, respectively. Then we can transform $\bm{L}_{1}$ and $\bm{L}_{2}$ to their corresponding patch-tensors $\bm{\mathcal{L}}_{1}$ and $\bm{\mathcal{L}}_{2}$. Finally, we define the local structure weight patch-tensor as follows

[TABLE]

where $h$ is a weight stretching parameter, $d_{\max}$ and $d_{\min}$ are the maximum and minimum of $\bm{\mathcal{L}}_{1}-\bm{\mathcal{L}}_{2}$, respectively. fig. 3 displays the edge awareness maps of fig. 5, which demonstrates that the structure tensor based local structure weight has a good performance in identifying the edges. It should be noticed that for the sake of displaying effect, fig. 3 is created via a normalized version of $-\text{exp}\left(-h\cdot\frac{\left(\bm{\mathcal{L}}_{1}-\bm{\mathcal{L}}_{2}\right)-d_{\min}}{d_{\max}-d_{\min}}\right)$. In the proposed algorithm, we still calculate the local structure weight via eq. 10.

With the help of $\bm{\mathcal{W}}_{\mathrm{LS}}$, we can rewrite eq. 7 into a weighted IPT model:

[TABLE]

where $\odot$ is the notation of Hadamard product. Thanks to the weighted IPT model (11), the strong edges could be well suppressed in the target image.

IV Reweighted infrared patch-tensor model and its solution

IV-A Reweighted infrared patch-tensor model

The computing time is also a major concern in infrared small target detection. Generally, the stopping criterion of a RPCA algorithm is that the reconstruction error is less than a certain small value. To meet this criterion, WIPT needs dozens of iterations, which is still time-consuming. An interesting phenomenon we find is that the non-zero entry number in the target patch-tensor has ceased to grow before the algorithm converges. In fact, in the target image, true target merely occupies the brightest of the few. In the second half of the iteration, their values barely change. Therefore, considering the particularity of infrared small target, it is reasonable to replace the reconstruction error with the target patch-tensor sparsity in our proposed model. The algorithm stops iteration once the non-zero entry ceases to grow. Then the sparsity of the target patch-tensor becomes critical in reducing the computing time. We hope the non-zero entries keep decreasing as the iteration goes on, leaving the final target image the sparsest. Unfortunately, the real situation is against our expectation in IPT and WIPT, where the target image deteriorates as the algorithm converges. Naturally, it motivates us to take a sparsity enhancing approach to solve this problem.

In Ref. [38], Candès proposed a reweighted $\ell_{1}$ minimization for enhancing sparsity. By minimizing a sequence of weighted $\ell_{1}$ norm, a significant performance improvement is obtained on sparse recovery. Inspired by it, many improved RPCA models have been proposed [39, 40, 41, 42]. Motivated by these state-of-the-art models, we adopt a similar reweighted scheme for the values in the target patch-tensor. The large weights discourage non-zero entries, and the small weights preserve non-zero elements. The sparsity enhancement weight is defined as follows

[TABLE]

where $\epsilon>0$ is a preset constant to avoid division by zero. Then besides the relative error $\|\bm{\mathcal{F}}-\bm{\mathcal{B}}^{k+1}-\bm{\mathcal{T}}^{k+1}\|_{\text{F}}/\|\bm{\mathcal{F}}\|_{\text{F}}<\varepsilon$, we could add a new end condition that counts the non-zero entry element, namely $\|\bm{\mathcal{T}}^{k+1}\|_{0}=\|\bm{\mathcal{T}}^{k}\|_{0}$. With the help of this empirical observation, the computing time could be largely decreased, as illustrated in fig. 10 and table IV.

Another intrinsic characteristic that both Ref. [26] and Ref. [29] neglect is the fact that the small target is always brighter than its neighborhood environment in infrared images due to the physical imaging mechanism [43]. Therefore, besides the sparsity constraint [44, 45] of the target patch-tensor, it is reasonable to assume that all the entries in $\bm{\mathcal{T}}$ are non-negative. To this end, we incorporate this target non-negative prior into the reweighted IPT model via rewriting eq. 12 as follows

[TABLE]

where $\delta\left(\cdot\right)$ is an indicator function. We combine the local structure weight $\bm{\mathcal{W}}_{\mathrm{LS}}$ and sparsity enhancing weight $\bm{\mathcal{W}}_{\mathrm{SE}}^{k}$ to get the adaptive weight as follows

[TABLE]

Finally, we generalize the proposed IPT model and WIPT model to a novel reweighted infrared patch-tensor model (RIPT) as follows

[TABLE]

IV-B Solution of RIPT model

In this subsection, we show how to solve the proposed RIPT model as a reweighted robust tensor recovery problem via an Alternating Direction Method of Multipliers (ADMM) [46]. The augmented Lagrangian function of eq. 15 is defined as

[TABLE]

where $\bm{\mathcal{Y}}_{i}\in\mathbb{R}^{I\times J\times P},\ i=1,2,3$ are the Lagrange multiplier tensors, and $\mu$ is a positive penalty scalar. ADMM decomposes the minimization of $\mathcal{L}$ into two subproblems that minimize $\bm{\mathcal{B}}_{i}$ and $\bm{\mathcal{T}}$, respectively. More specifically, the iterations of ADMM go as follows:

Updating $\bm{\mathcal{B}}_{i}$ with the other terms fixed.

[TABLE]

Updating $\bm{\mathcal{T}}$ with the other terms fixed.

[TABLE]

Updating the multiplier $\bm{\mathcal{Y}}_{i}^{k+1}$ with the other terms fixed.

[TABLE]

The subproblems (17) and (18) can be solved via the following two operators, respectively.

[TABLE]

From eq. 21, it could be observed that the weighting parameter determines the soft-threshold, controlling the trade-off between the target patch-tensor and background patch-tensor. Therefore, our element-wise adaptive weight tensor could simultaneously preserve the small target and suppress the strong edges. Finally, the solution of the proposed model is given in algorithm 1.

IV-C Detection Procedure

In fig. 4, we present the whole procedure of detecting the infrared small target via the model proposed in this paper. The detailed steps are as follows:

Given an infrared image $f_{\textup{F}}$, its local structure feature map $f_{\textup{LS}}$ is calculated via eq. 9. 2. 2.

The original patch-tensor $\bm{\mathcal{F}}$ and local structure weight patch-tensor $\bm{\mathcal{W}}_{\mathrm{LS}}$ are constructed from $f_{\textup{F}}$ and $f_{\textup{LS}}$. 3. 3.

algorithm 1 is performed to decompose the patch-tensor $\bm{\mathcal{F}}$ into the background patch-tensor $\bm{\mathcal{B}}$ and target patch-tensor $\bm{\mathcal{T}}$. 4. 4.

The background image $f_{\textup{B}}$ and target image $f_{\textup{T}}$ are reconstructed from the background patch-tensor $\bm{\mathcal{B}}$ and target patch-tensor $\bm{\mathcal{T}}$, respectively. For the sake of implementation convenience, we adopt the uniform average of estimators (UAE) reprojection scheme [47]. 5. 5.

The target is segmented out as the same as Ref. [26]. The adaptive threshold is determined by:

[TABLE]

where $\bar{f}_{\textup{T}}$ and $\sigma$ are the average and standard deviation of the target image $f_{\textup{T}}$. $k$ and $v_{\min}$ are constants determined empirically.

V Experimental validation

To fully evaluate the proposed algorithm, we conduct a series of experiments using images of various scenarios and include ten state-of-the-art methods for comparison.

V-A Experimental setup

Datasets. We test the proposed model on extensive real infrared images to cover different scenarios, as illustrated in fig. 5, varying from the flat backgrounds with salient targets to complex backgrounds with heavy clutters and extremely dim targets. All targets are labeled with red boxes. Since some targets are so dim that could hardly be observed by human eyes directly, we enlarge the demarcated area. Taking into account that the biggest difficulty of current infrared small target detection is how to detect those very dim targets with strong clutters, a good detection performance on those extremely complex images is more convincing than the satisfying result on relatively simple images. Therefore, in the following experiments, our main focus is put on the datasets with complex scenes that fig. 5(a) – (d) and (l) belong to. The detailed characteristics of these five sequences are presented in table II.

Baselines and Parameter settings. The proposed algorithm is compared with ten state-of-the-art solutions, including three filtering based methods (Max-Median [10], Top-Hat [48], TDLMS [9]), three HVS based methods (PFT [49], MPCM [19], WLDM [23]), and four recently developed low-rank methods (IPI [26], PRPCA [50], WIPI [29], NIPPS [30]). table III summarizes all the methods involved in the experiments and their detailed parameter settings. For all the low-rank methods, i.e. IPI, PRPCA, WIPI, NIPPS, IPT, and RPIT, they are all solved via ADMM. All the algorithms are implemented in MATLAB 2016b on a PC of 3.4 GHz and 4GB RAM. The source code of our method is publicly available at https://github.com/YimianDai/DENTIST.

Evaluation Metrics. For a comprehensive evaluation, four metrics including the local signal to noise ratio gain (LSNRG), background suppression factor (BSF), signal to clutter ratio gain (SCRG), and receiver operating characteristic (ROC) curve are adopted in comparison of background suppressing performance. LSNRG measures the local signal to noise ratio (LSNR) gain, which is defined as

[TABLE]

where $\textup{LSNR}_{\textup{in}}$ and $\textup{LSNR}_{\textup{out}}$ are the LSNR values before and after background suppression. LSNR is given as $\textup{LSNR}=P_{\textup{T}}/P_{\textup{B}}$. $P_{\textup{T}}$ and $P_{\textup{B}}$ are the maximum grayscales of the target and neighborhood, respectively. BSF measures the background suppression quality using the standard deviation of the neighborhood region. It is defined as:

[TABLE]

where $\sigma_{\text{in}}$ and $\sigma_{\text{out}}$ are the standard variances of background neighborhood before and after background suppression. The most widely used SCRG is defined as the ratio of signal-to-clutter ratio (SCR) before and after processing:

[TABLE]

where SCR represents the difficulty of detecting the infrared small target, and it is defined by $\text{SCR}=|\mu_{\text{t}}-\mu_{\text{b}}|/\sigma_{\text{b}}$. $\mu_{\text{t}}$ is the average target grayscale. $\mu_{\text{b}}$ and $\sigma_{\text{b}}$ are the average grayscale and standard deviation of the neighborhood region. For all these three metrics, the higher their values are, the better background suppression performance the detection method has. All three metrics are computed in a local region, as illustrated in fig. 6. The target size is $a\times b$, and $d$ is the neighborhood width. we set $d=20$ in this paper.

Among all the existing metrics, the detection probability $P_{\text{d}}$ and false-alarm rate $F_{\text{a}}$ are the key performance indicators, which are defined as follows

[TABLE]

The ROC curve shows the trade-off between the true detections and false detections.

V-B Validation of the proposed method

In this subsection, we take a closer look at the proposed method by validating their robustness against various scenes and noisy cases. At last, the roles of the patch-tensor, local structure weight, and sparsity enhancement weight are examined in depth to evaluate each prior individually.

V-B1 Robustness to various scenes

In fig. 7, we show the separated target images for the images of fig. 5. Observing fig. 7, it can be clearly seen that the background clutters are completely wiped out, leaving the target the only sole component in the target image. Since fig. 5 contains a lot of different scenarios, it is fair to say that the proposed method is quite robust to various scenes.

V-B2 Robustness to noise

Noise is another key influence factor. In fig. 8, we evaluate the proposed method’s performance in the case of noise with different levels. When the noise standard deviation is $10$, the proposed method could well enhance the targets and suppress the clutters and noise. As the noise standard deviation increases to $20$, RIPT still detects the target in fig. 8(m) – (n) and (q) – (r), but fails in fig. 8(o) – (p). Nevertheless, this failure is acceptable, since the target is totally overwhelmed by the noise in fig. 8(o) – (p) (see the enlarged box). Therefore, the noise influence depends not only on the intensity of the noise itself but also on the original contrast of the target. As long as the polluted target can maintain a weak contrast like fig. 8(c) – (d), the proposed method is still able to detect it.

V-B3 Roles of components in the proposed model

To further understand the effects of the components in the proposed RIPT model, we evaluate each prior individually with experiment to investigate how these priors influence the final detection performance. The ROC curves of IPI, IPT, IPT with sparsity enhancement weight (SIPT), WIPT, and RIPT for Sequence 1 – 4 are given in fig. 9, leading to the following observations. (1) The four patch-tensor based methods outperform the state-of-the-art IPI method, which demonstrates that the patch-tensor model, involving mode-$1$ and mode-$2$ unfolding matrices, does contribute to the final detection performance. (2) By comparing WIPT and RIPT with IPT and SIPT, we see that incorporating local structure prior significantly improves the detection probability. (3) Although the sparsity enhancement weight does not contribute to the final detection performance, it significantly reduces the iteration number, as illustrated in fig. 10. These observations indicate that the introduced priors are effective, and, when combined together, lead to excellent performance as reported in the next subsections.

V-C *Algorithm Complexity and Computational Time *

The proposed model is solved via ADMM, which has been proved a $\mathcal{O}(1/k)$ convergence [51, 52]. Therefore, our solving algorithm is ensured to converge. The algorithm complexity and computational time for fig. 5(a) with various methods are given in table IV. The image size is $M\times N$, and $m,n$ are the rows and columns of the patch-image or mode-$3$ unfolding. Although the computational complexities of these methods seem the same, their computing time differs a lot. For the filtering and HVS based methods, the difference in computing time lies in whether the code could be vectorized. For the low-rank methods, the dominant factor is the iteration number. It can be seen from the data in table IV that the low-rank methods are generally slower than the filtering and HVS based methods. Nevertheless, considering low-rank method could handle more difficult scenes, this trade-off is acceptable. Among the low-rank methods, the RIPT costs the least time. The underlying reason is that both the local structure weight and sparsity enhancement weight help to reduce the iteration number. In addition, unlike the weight in WIPI, the time for constructing the weight is neglectable in RIPT.

V-D Parameters analysis

For the proposed model, the related parameters, such as the patch size, sliding step, weight stretching parameter $h$, weighting parameter $\lambda$, and penalty factor $\mu$, are all important factors, which usually affects the model fitness on the real databases. Therefore, a better performance can be obtained by finely tuning these parameters. Nevertheless, the optimal values are always related to the infrared image content. In fig. 11, we give the ROC curves for different model parameters on Sequence 1 – 4 to evaluate their influence. These parameters are validated to obtain a local optimal value with other parameters fixed. The stepped shape of our ROC curves might seem a bit odd. It is because we have adopted a much larger weighting parameter $\lambda$ than normal RPCA-based foreground-background separation tasks in order to better fit the actual situation of single-frame infrared small target detection.

V-D1 Patch size $I\times J$

It not only has a large impact on the separation, but also influences the computational complexity. The matrix size of mode-$1$ and mode-$2$ unfoldings of the patch-tensor is $I\times(J\cdot P)$; the matrix size of mode-$3$ unfolding is $J\times(I\cdot P)$. Obviously, a smaller patch size will lead to a smaller computational complexity. On one hand, we hope a larger patch size to ensure that the target is sparse enough. On the other hand, a larger patch size reduces the correlationships between the non-local patches, which degrades the separation results. To seek a balance between a low computational burden, target sparsity, and background correlationship, we vary the patch size $P$ from $10$ to $60$ with ten intervals and provide the ROC curves in the first row of fig. 11. By observing the ROC curves, we can have the following conclusions. Firstly, a smaller patch size is easier to raise false alarms while a larger patch size may lead to a relatively lower detection probability, which just demonstrates our above analysis about the patch size. Secondly, the proposed RIPT method is not very sensitive to the patch size. The detection result of the patch size among $20$ – $60$ is acceptable. Thirdly, $30$ seems a good choice for Sequence 1 – 4 since it achieves the best performance in ROC.

V-D2 Sliding step

The sliding step influences the patch-tensor size as well. To reduce the computational complexity, we prefer a larger sliding step, which means smaller matrices to perform SVD. Nevertheless, a larger sliding step also reduces the redundancy of the original patch-tensor $\bm{\mathcal{T}}$ and undermines the final detection results since our proposed model is based on the non-local repentance of correlated patches. To investigate its influence, we vary the sliding step $S$ from $8$ to $16$ with two intervals. The results are displayed in the second row of fig. 11. It could be observed that the ROC curve of small sliding step like $8$ tends to have a more sharp shape, but its overall detection probability remains relatively low. The best value for sliding step is among $12$ to $14$, here we pick $12$. In addition, by comparing the first row with the second row of fig. 11, we can conclude that the algorithm is quite robust to the variation of step length.

V-D3 Weight stretching parameter $h$

It controls the local structure weight’s suppression degree to the clutter edges. We vary $h$ from $6$ to $22$ in the experiment and illustrate the ROC curves in the third row of fig. 11. Generally, we would like a larger $h$ which suppresses the undesirable non-target components thoroughly. Nevertheless, since the target-clutter distinguishing scheme is not always perfect, an overlarge $h$ would also wipe out some targets. A typical example is the different performance of $h=18$ or $22$ among four sequences. For Sequence 2 and 3, $h=18$ or $22$ achieves the best performance. But, they perform the worst for Sequence 1 and 4. It is because the target moves along the cloud edge in many frames of Sequence 1 and 4, and an overlarge $h$ would easily mistake the target as the edge and suppress it completely, resulting in a lower detection probability. On the contrary, a smaller $h$ might preserve the small target, but it also retains some non-target components, making the false-alarm ratio relatively high. For Sequence 2 and 3, when the detection probability is fixed, the false-alarm ratio of $h=6$ is the largest. In order to seek a balance, we set the optimal $h$ as $10$ in the following experiment.

V-D4 Weighting parameter $\lambda$

Despite the usage of local structure weight, fine tuning of $\lambda$ is still of great importance. We show the effects of $\lambda$ in the fourth row of fig. 11. Since $\lambda$ is set as $L/\sqrt{\min(I,J,P)}$ in our model, instead of directly varying $\lambda$, we vary $L$ from $0.3$ to $1.5$. From the illustration, it can be observed that a large $\lambda$ does keep the false-alarm ratio being quite low like. For example, the ROC curves of $L=1.0$ and $L=1.5$ for Sequence 2 are straight line segments. But their detection probabilities are also low because many dim targets are suppressed by the overlarge threshold. On the contrary, when the detection probability is fixed, the false-alarm ratio of $L=0.3$ is always higher than the others, suggesting that a too small $L$ is also not a good choice.

V-D5 Penalty factor $\mu$

It is precisely the shrinking threshold of eq. 21, which influences the low-rank property of the background patch-tensor. With a smaller $\mu$, more details are preserved in the background patch-tensor. Thus fewer non-target components are left in the target patch-tensor. Nevertheless, the small target might be preserved in the background patch-tensor as well, resulting no target in the target image. On the contrary, a larger $\mu$ would lead to more non-target components lying in the target patch-tensor. Thus, it is necessary to choose an appropriate value for $\mu$ to keep the balance between detection probability and false-alarm ratio. Since we set $\mu=C_{\mu}\text{std}(\text{vec}(\bm{\mathcal{F}}))$, instead of varying $\mu$ directly, we investigate the influence of the penalty factor on Sequence 1 – 4 by varying $C_{\mu}$ from $0.5$ to $3$. The results are shown in the last row of fig. 11, from which we can observe that an overlarge or too small $\mu$ is not an optimal choice and the best value for our four sequences is about $0.7$.

V-E Comparison with State-of-the-Arts

In this subsection, we first compare the proposed model with the other state-of-the-art methods on the ability of clutter suppression. fig. 12 – fig. 15 show the separated target images by twelve tested methods for four representative frames of Sequence 1 – 4 in fig. 5. It can be seen that the classical Max-Median filter does enhance the tiny targets in fig. 12(b) - fig. 15(b). Nevertheless, many non-target pixels are also enhanced simultaneously, especially in fig. 13(b) and fig. 14(b), which would raise many false alarms. In fig. 12(a) - fig. 15(a) produced by TDLMS, the phenomenon of enhancing non-target isolated points does not exist, but the cloud edges are highlighted, making them much brighter than the small target. Since the given target size matches the real target size exactly, the Top-Hat operator succeeds to enhance the target region. If not match, the Top-Hat operator is likely to lose the target. No matter whether the given and real target sizes match, Top-Hat cannot well suppress the background clutters. Many strong clutters still remain in resulting images, as illustrated in fig. 12(c) - fig. 15(c). Although PFT can retain the target to a certain extent, the target is not necessarily the brightest and there are also many non-target salient residuals, as shown in fig. 12(d) - fig. 15(d). MPCM and WLDM failed to achieve good results because they suffered from a phenomenon we named rare structure effect which is caused by the inaccuracy of the local dissimilarity measure and often happens when the target is extremely dim. In next subsection, we will further discuss this phenomenon.

In fact, the common and intrinsic reason behind the unsatisfactory performance of all these six methods lies in their pre-set assumption about the target shape, namely a hot spot brighter than its neighborhood. Nevertheless, when the target is too dim to maintain its significant contrast over non-target components, just like fig. 5(a) – (d), they might not perform as well as they usually do.

The last six tested methods are all low-rank based methods. Comparing with above six methods, their results contain fewer background details. Relatively speaking, the effects of PRPCA and WIPI are not as good as the other four methods. Different from the other low-rank based methods that all build their low-rank assumptions on the data structure composed of patches, PRPCA supposes the individual patch is low-rank. Thus in PRPCA, each patch is applied to an individual RPCA process. Then all the separated target patches are synthesized into a target image. By comparing fig. 13(g) and fig. 15(g) with fig. 13(h) and fig. 15(h) , it can be seen that fewer edges were left by IPI than PRPCA. It is because the rare structure in a patch is not necessarily rare in the patch-image due to the redundancy of the whole image. Therefore, the results of IPI and IPT are much better than those of PRPCA. As to WIPI, considering the targets in Sequence 1 – 4 is much dimmer than those in Ref. [29], it is fair to say that the steering kernel based patch-level weight is still not robust enough to handle all of the complex infrared backgrounds. From fig. 12(l) and fig. 15(l), we can see that with the help of the local structure weight, the non-target components were suppressed completely via our proposed model. For example, the cloud residuals in fig. 12(g) by IPI is brighter than its target, while in fig. 12(l), it is wiped out thoroughly. Based on above comparisons, it is fair to conclude that the proposed RIPT model achieves the most satisfying performance in infrared background suppression among twelve tested methods.

For infrared small target detection, the biggest difficulty is the interference of complex backgrounds. These undesirable background clutters raise the false alarm rates, and might even overwhelm the dim targets. Therefore, the ability of successfully suppressing the background clutters is a major concern in evaluating an infrared small target detection method. Quantitative evaluating indices are also widely used to assess this ability. table V shows the experimental data of all twelve tested methods for fig. 5(a) – (d). The gray-scale of every separated target image is rescaled to the range $0--255$. It could be observed that our proposed method gets the highest scores among all indices and all tested images. Different from the filtering based methods, for the low-rank based methods, Inf, namely infinity, is quite common, which just means that the target neighborhood completely shrinks to zero. In addition, it should be noted that the high scores in these three quantitative indices merely reflect the good suppression performance in a local region, and not necessarily mean a good global suppression ability.

To further reveal the advantage of the proposed model, we display the ROC curves of Sequence 1 and Sequence 3 – 5 for comparison in fig. 16. The most interesting point is the performances of the state-of-the-art WLDM on Sequence 1, 3, 4 and Sequence 5 are very different. For Sequence 5, WLDM performs very well but fails in Sequence 1 – 4. We believe the reason lies in the rare structure effect which is a born problem for local contrast method. NIPPS’s performance is slightly better than the IPI model. Finally, the proposed algorithm achieves the highest detection probability for the same false-alarm ratio, meaning that the proposed RIPT model has a better performance than the other models.

VI Conclusion

To further suppress the strong edges while preserving the spatial correlation, a reweighted infrared patch-tensor model for small target detection is developed in this paper, simultaneously combining non-local redundant prior and local structure prior together. A local structure weight is designed based on the structure tensor and served as an edge indicator in the weighted model. In addition, a sparsity enhancement scheme is adopted to avoid the target image being contaminated. Then the target-background separation task is modeled as a reweighted robust tensor recovery problem, which can be efficiently solved via ADMM. Detailed experimental results show that our proposed model is robust to various scenarios and obtains the clearest separated target images compared with the state-of-the-art target-background separation methods.

Acknowledgments

The authors would like to thank the editor and anonymous reviewers for their helpful comments and suggestions. This work was supported in part by the National Natural Science Foundation of China under Grants no. 61573183, and Open Research Fund of Key Laboratory of Spectral Imaging Technology, Chinese Academy of Sciences under Grant no. LSIT201401.