Shape-Color Differential Moment Invariants under Affine Transformations

TL;DR

This paper introduces a new method for constructing shape-color differential moment invariants that are invariant under affine transformations, improving image classification and retrieval performance.

Contribution

It presents a general formula for shape-color primitives and constructs 50 shape-color differential moment invariants invariant to affine transforms.

Findings

SCDMIs outperform traditional color descriptors in image classification.

50 instances of shape-color differential moment invariants are developed.

Experimental results demonstrate improved retrieval accuracy.

Abstract

We propose the general construction formula of shape-color primitives by using partial differentials of each color channel in this paper. By using all kinds of shape-color primitives, shape-color differential moment invariants can be constructed very easily, which are invariant to the shape affine and color affine transforms. 50 instances of SCDMIs are obtained finally. In experiments, several commonly used color descriptors and SCDMIs are used in image classification and retrieval of color images, respectively. By comparing the experimental results, we find that SCDMIs get better results.

Tables

Table 1. Table 1: The construction methods of S ​ C ​ D ​ M ​ I ​ 50 𝑆 𝐶 𝐷 𝑀 𝐼 50 SCDMI50

Name	scCore	sCore
$SCDMI{s}_0^1/SCDMI{s}_1^1$	$C(1,2,3)/SC{P}_1(1,2,3)$	$\left(x_1{y}_2-x_2{y}_1\right){\left(x_1{y}_3-x_3{y}_1\right)}^2$
$SCDMI{s}_0^2/SCDMI{s}_1^2$	$C(1,2,3)/SC{P}_1(1,2,3)$	$\left(x_1{y}_2-x_2{y}_1\right){\left(x_1{y}_3-x_3{y}_1\right)}^3$
$SCDMI{s}_0^3/SCDMI{s}_1^3$	$C(1,2,3)/SC{P}_1(1,2,3)$	$\left(x_1{y}_2-x_2{y}_1\right)\left(x_1{y}_3-x_3{y}_1\right)\left(x_2{y}_3-x_3{y}_2\right)$
$SCDMI{s}_0^4/SCDMI{s}_1^4$	$C(1,2,3)/SC{P}_1(1,2,3)$	$\left(x_1{y}_2-x_2{y}_1\right)\left(x_1{y}_3-x_3{y}_1\right){\left(x_2{y}_3-x_3{y}_2\right)}^3$
$SCDMI{s}_0^5/SCDMI{s}_1^5$	$C(1,2,3)/SC{P}_1(1,2,3)$	${\left(x_1{y}_2-x_2{y}_1\right)}^2{\left(x_1{y}_3-x_3{y}_1\right)}^2\left(x_2{y}_3-x_3{y}_2\right)$
$SCDMI{s}_0^6/SCDMI{s}_1^6$	$C(1,2,4)/SC{P}_1(1,2,4)$	$\left(x_1{y}_2-x_2{y}_1\right)\left(x_2{y}_3-x_3{y}_2\right)\left(x_3{y}_4-x_4{y}_3\right)$
$SCDMI{s}_0^7/SCDMI{s}_1^7$	$C(1,2,4)/SC{P}_1(1,2,4)$	$\left(x_1{y}_2-x_2{y}_1\right)\left(x_2{y}_3-x_3{y}_2\right){\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^8/SCDMI{s}_1^8$	$C(1,3,4)/SC{P}_1(1,3,4)$	$\left(x_1{y}_2-x_2{y}_1\right)\left(x_2{y}_3-x_3{y}_2\right){\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^9/SCDMI{s}_1^9$	$C(1,2,3)/SC{P}_1(1,2,3)$	${\left(x_1{y}_2-x_2{y}_1\right)}^2\left(x_2{y}_3-x_3{y}_2\right)\left(x_3{y}_4-x_4{y}_2\right)$
$SCDMI{s}_0^{10}/SCDMI{s}_1^{10}$	$C(1,2,4)/SC{P}_1(1,2,4)$	${\left(x_1{y}_2-x_2{y}_1\right)}^2\left(x_2{y}_3-x_3{y}_2\right){\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^{11}/SCDMI{s}_1^{11}$	$C(2,3,4)/SC{P}_1(2,3,4)$	${\left(x_1{y}_2-x_2{y}_1\right)}^2\left(x_2{y}_3-x_3{y}_2\right){\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^{12}/SCDMI{s}_1^{12}$	$C(1,2,4)/SC{P}_1(1,2,4)$	${\left(x_1{y}_2-x_2{y}_1\right)}^3\left(x_2{y}_3-x_3{y}_2\right){\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^{13}/SCDMI{s}_1^{13}$	$C(1,2,3)/SC{P}_1(1,2,3)$	$\left(x_1{y}_2-x_2{y}_1\right){\left(x_2{y}_3-x_3{y}_2\right)}^2{\left(x_3{y}_4-x_4{y}_3\right)}^2$
$SCDMI{s}_0^{14}/SCDMI{s}_1^{14}$	$C(1,2,4)/SC{P}_1(1,2,4)$	$\left(x_1{y}_2-x_2{y}_1\right){\left(x_2{y}_3-x_3{y}_2\right)}^2{\left(x_3{y}_4-x_4{y}_3\right)}^2$
$SCDMI{s}_0^{15}/SCDMI{s}_1^{15}$	$C(1,2,4)/SC{P}_1(1,2,4)$	$\left(x_1{y}_2-x_2{y}_1\right){\left(x_2{y}_3-x_3{y}_2\right)}^3\left(x_3{y}_4-x_4{y}_3\right)$
$SCDMI{s}_0^{16}/SCDMI{s}_1^{16}$	$C(1,3,4)/SC{P}_1(1,3,4)$	$(x_1{y}_2-x_2{y}_1)(x_2{y}_3-x_3{y}_2){(x_3{y}_4-x_4{y}_3)}^2\cdot$ $\left(x_4{y}_1-x_1{y}_4\right)$
$SCDMI{s}_0^{17}/SCDMI{s}_1^{17}$	$C(1,2,3)/SC{P}_1(1,2,3)$	${(x_1{y}_2-x_2{y}_1)}^2(x_2{y}_3-x_3{y}_2){(x_3{y}_4-x_4{y}_3)}^3\cdot$ $\left(x_4{y}_1-x_1{y}_4\right)$
$SCDMI{s}_0^{18}/SCDMI{s}_1^{18}$	$C(1,2,4)/SC{P}_1(1,2,4)$	$\left(x_1{y}_2-x_2{y}_1\right)\left(x_1{y}_3-x_3{y}_1\right)\left(x_1{y}_4-x_4{y}_1\right)$
$SCDMI{s}_0^{19}/SCDMI{s}_1^{19}$	$C(1,2,4)/SC{P}_1(1,2,4)$	$(x_1{y}_2-x_2{y}_1)(x_1{y}_3-x_3{y}_1)(x_1{y}_4-x_4{y}_1)\cdot$ ${\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^{20}/SCDMI{s}_1^{20}$	$C(2,3,4)/SC{P}_1(2,3,4)$	$(x_1{y}_2-x_2{y}_1){(x_1{y}_3-x_3{y}_1)}^2(x_1{y}_4-x_4{y}_1)\cdot$ ${\left(x_3{y}_4-x_4{y}_3\right)}^2$
$SCDMI{s}_0^{21}/SCDMI{s}_1^{21}$	$C(1,2,3)/SC{P}_1(1,2,3)$	${(x_1{y}_2-x_2{y}_1)}^2(x_1{y}_3-x_3{y}_1)(x_1{y}_4-x_4{y}_1)\cdot$ $\left(x_3{y}_4-x_4{y}_3\right)$
$SCDMI{s}_0^{22}/SCDMI{s}_1^{22}$	$C(1,2,3)/SC{P}_1(1,2,3)$	${(x_1{y}_2-x_2{y}_1)}^2(x_1{y}_3-x_3{y}_1)(x_1{y}_4-x_4{y}_1)\cdot$ ${\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^{23}/SCDMI{s}_1^{23}$	$C(1,2,4)/SC{P}_1(1,2,4)$	${(x_1{y}_2-x_2{y}_1)}^2(x_1{y}_3-x_3{y}_1)(x_1{y}_4-x_4{y}_1)\cdot$ ${\left(x_3{y}_4-x_4{y}_3\right)}^3$
$SCDMI{s}_0^{24}/SCDMI{s}_1^{24}$	$C(1,2,4)/SC{P}_1(1,2,4)$	$(x_1{y}_2-x_2{y}_1)(x_2{y}_3-x_3{y}_2){(x_3{y}_4-x_4{y}_3)}^2\cdot$ $\left(x_4{y}_1-x_1{y}_4\right)\left(x_2{y}_4-x_4{y}_2\right)$
$SCDMI{s}_0^{25}/SCDMI{s}_1^{25}$	$C(1,2,4)/SC{P}_1(1,2,4)$	${(x_1{y}_2-x_2{y}_1)}^2(x_2{y}_3-x_3{y}_2){(x_3{y}_4-x_4{y}_3)}^2\cdot$ ${\left(x_4{y}_1-x_1{y}_4\right)}^2\left(x_2{y}_4-x_4{y}_2\right)$

Table 2. Table 2: The classification accuracies obtained by using different features

Descriptor	Accuracy	Descriptor	Accuracy
SCDMI50	98.67%	GPSOs	78.56%
AMIs	50.11%	Hu moments	25.00%
Color moments	74.77%	RGhistogram	80.56%
Transformed color distribution	95.78%

Equations

$$\left(\begin{array}[]{c}x^{{}^{\prime}}\\ y^{{}^{\prime}}\\ \end{array}\right)=SA\cdot\left(\begin{array}[]{c}x\\ y\\ \end{array}\right)+ST=\left(\begin{array}[]{cc}\alpha_{1}&\alpha_{2}\\ \beta_{1}&\beta_{2}\\ \end{array}\right)\cdot\left(\begin{array}[]{c}x\\ y\\ \end{array}\right)+\left(\begin{array}[]{c}O_{x}\\ O_{y}\\ \end{array}\right)$$

$$\left(\begin{array}[]{c}R^{{}^{\prime}}(x,y)\\ G^{{}^{\prime}}(x,y)\\ B^{{}^{\prime}}(x,y)\\ \end{array}\right)=CA\cdot\left(\begin{array}[]{c}R(x,y)\\ G(x,y)\\ B(x,y)\\ \end{array}\right)+CT=\left(\begin{array}[]{ccc}\ a_{1}&a_{2}&a_{2}\\ \ b_{1}&b_{2}&b_{2}\\ \ c_{1}&c_{2}&c_{2}\\ \end{array}\right)\cdot\left(\begin{array}[]{c}R(x,y)\\ G(x,y)\\ B(x,y)\\ \end{array}\right)+\left(\begin{array}[]{c}O_{R}\\ O_{G}\\ O_{B}\\ \end{array}\right)$$

$$S(p,q)=\left|\begin{array}[]{cc}(x_{p}-\bar{x})&(x_{q}-\bar{x})\\ (y_{p}-\bar{y})&(y_{q}-\bar{y})\\ \end{array}\right|$$

$$\begin{split}&C(p,q,r)=\left|\begin{array}[]{ccc}(R(x_{p},y_{p})-\bar{R})&(R(x_{q},y_{q})-\bar{R})&(R(x_{r},y_{r})-\bar{R})\\ (G(x_{p},y_{p})-\bar{G})&(G(x_{q},y_{q})-\bar{G})&(G(x_{r},y_{r})-\bar{G})\\ (B(x_{p},y_{p})-\bar{B})&(B(x_{q},y_{q})-\bar{B})&(B(x_{r},y_{r})-\bar{B})\\ \end{array}\right|\end{split}$$

$$sCore(n,m;d_{1},d_{2},...,d_{n})=\underbrace{S(1,2)S(k,l)...S(r,n)}_{m}\$$

$$cCore(N,M;D_{1},D_{2},...,D_{N})=\underbrace{C(1,2,3)C(G,K,L)...C(P,Q,N)}_{M}\$$

$$S^{{}^{\prime }}(p,q)=|SA|\cdot S(p,q)$$

$$C^{{}^{\prime }}(p,q,r)=|CA|\cdot C(p,q,r)$$

$$sCor{e}^{{}^{\prime }}(n,m;d_1,d_2,...,d_n)=|SA{|}^m\cdot sCore(n,m;d_1,d_2,...,d_n)$$

$$cCor{e}^{{}^{\prime }}(N,M;D_1,D_2,...,D_N)=|CA{|}^M\cdot cCore(N,M;D_1,D_2,...,D_N)$$

$$\begin{array}{rl}& SCAMIs(n,m,N,M;d_1,...,d_n;D_1,...,D_N)\\ & =\frac{I(sCore(n,m,d_1,...,d_n)\cdot cCore(N,M,D_1,...,D_N))}{I(sCore(1,0){)}^{max(n+N)+m-\frac{3M}{2}}\cdot I(cCore(3,2;2,2,2){)}^{\frac{M}{2}}}\end{array}$$

$$\begin{array}{rl}SCAMI{s}^{{}^{\prime }}(n,m,N,M;d_1,...,d_n;D_1,...,D_N)& \\ =SCAMIs(n,m,N,M;d_1,...,d_n;D_1,...,D_N)& \end{array}$$

$$SC{M}_{pq\alpha \beta \gamma }=\iint (x-\bar{x}{)}^p(y-\bar{y}{)}^q(R(x,y)-\bar{R}{)}^{\alpha }(G(x,y)-\bar{G}{)}^{\beta }(B(x,y)-\bar{B}{)}^{\gamma }dxdy$$

$$\begin{array}{rl}SC{M}_{pq\alpha \beta \gamma }^k=\iint & (x-\bar{x}{)}^p(y-\bar{y}{)}^q(R^{(k)}(x,y)-{\bar{R}}^{\delta (k)}{)}^{\alpha }(G^{(k)}(x,y)-{\bar{G}}^{\delta (k)}{)}^{\beta }\\ & (B^{(k)}(x,y)-{\bar{B}}^{\delta (k)}{)}^{\gamma }dxdy\end{array}$$

$$\delta (k)=\{\begin{array}{rlr}1& & (k=0)\\ 0& & (k\ne 0)\end{array}$$

$$\begin{split}SCP_{k}(p,q,r)=\left|\begin{array}[]{ccc}F^{k}_{R}(x_{p},y_{p})&F^{k}_{R}(x_{q},y_{q})&F^{k}_{R}(x_{r},y_{r})\\ F^{k}_{G}(x_{p},y_{p})&F^{k}_{G}(x_{q},y_{q})&F^{k}_{G}(x_{r},y_{r})\\ F^{k}_{B}(x_{p},y_{p})&F^{k}_{B}(x_{q},y_{q})&F^{k}_{B}(x_{r},y_{r})\\ \end{array}\right|\end{split}$$

$$F_C^k(x,y)=\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{k}{i}\right)(x-\bar{x}{)}^i(y-\bar{y}{)}^{k-i}\frac{{\partial }^{k}C(x,y)}{\partial x^i\partial y^{k-i}},C\in \{R,G,B\}.$$

$$scCore_{k}(N,M;D_{1},D_{2},...,D_{N})=\underbrace{SCP_{k}(1,2,3)SCP_{k}(G,K,L)...SCP_{k}(P,Q,N)}_{M}\$$

$$SC{P}_k^{{}^{\prime }}(p,q,r)=|CA|\cdot SC{P}_k(p,q,r)$$

$$\begin{split}SCP_{k}^{{}^{\prime}}(p,q,r)=\left|\begin{array}[]{ccc}F^{k}_{R^{{}^{\prime}}}(x_{p}^{{}^{\prime}},y_{p}^{{}^{\prime}})&F^{k}_{R^{{}^{\prime}}}(x_{q}^{{}^{\prime}},y_{q}^{{}^{\prime}})&F^{k}_{R^{{}^{\prime}}}(x_{r}^{{}^{\prime}},y_{r}^{{}^{\prime}})\\ F^{k}_{G^{{}^{\prime}}}(x_{p}^{{}^{\prime}},y_{p}^{{}^{\prime}})&F^{k}_{G^{{}^{\prime}}}(x_{q}^{{}^{\prime}},y_{q}^{{}^{\prime}})&F^{k}_{G^{{}^{\prime}}}(x_{r}^{{}^{\prime}},y_{r}^{{}^{\prime}})\\ F^{k}_{B^{{}^{\prime}}}(x_{p}^{{}^{\prime}},y_{p}^{{}^{\prime}})&F^{k}_{B^{{}^{\prime}}}(x_{q}^{{}^{\prime}},y_{q}^{{}^{\prime}})&F^{k}_{B^{{}^{\prime}}}(x_{r}^{{}^{\prime}},y_{r}^{{}^{\prime}})\\ \end{array}\right|\end{split}$$

$$scCor{e}_k^{{}^{\prime }}(N,M;D_1,D_2,...,D_N)=|CA{|}^M\cdot scCor{e}_k(N,M;D_1,D_2,...,D_N)$$

$$scCore_{k}^{{}^{\prime}}(N,M;D_{1},D_{2},...,D_{N})=\underbrace{SCP_{k}^{{}^{\prime}}(1,2,3)SCP_{k}^{{}^{\prime}}(G,K,L)...SCP_{k}^{{}^{\prime}}(P,Q,N)}_{M}\$$

$$\begin{array}{rl}& SCDMI{s}_k(n,m,N,M;d_1,...,d_n;D_1,...,D_N)\\ & =\frac{I(sCore(n,m,d_1,...,d_n)\cdot scCor{e}_k(N,M,D_1,...,D_N))}{I(sCore(1,0){)}^{max(n+N)+m-\frac{3M}{2}}\cdot I(scCor{e}_k(3,2;2,2,2){)}^{\frac{M}{2}}}\end{array}$$

$$\begin{array}{rl}SCDMI{s}_k^{{}^{\prime }}(n,m,N,M;d_1,...,d_n;D_1,...,D_N)& \\ =SCDMI{s}_k(n,m,N,M;d_1,...,d_n;D_1,...,D_N)& \end{array}$$

$$\begin{array}{rl}& SCDMI{s}_k^{{}^{\prime }}(n,m,N,M;d_1,...,d_n;D_1,...,D_N)\\ & =\frac{I(sCor{e}^{{}^{\prime }}(n,m,d_1,...,d_n)\cdot scCor{e}_k^{{}^{\prime }}(N,M,D_1,...,D_N))}{I(sCor{e}^{{}^{\prime }}(1,0){)}^{max(n+N)+m-\frac{3M}{2}}\cdot I(scCor{e}_k^{{}^{\prime }}(3,2;2,2,2){)}^{\frac{M}{2}}}\end{array}$$

$$SCP_{1}(p,q,r)=\left|\begin{array}[]{ccc}F^{1}_{R}(x_{p},y_{p})&F^{1}_{R}(x_{q},y_{q})&F^{1}_{R}(x_{r},y_{r})\\ F^{1}_{G}(x_{p},y_{p})&F^{1}_{G}(x_{q},y_{q})&F^{1}_{G}(x_{r},y_{r})\\ F^{1}_{B}(x_{p},y_{p})&F^{1}_{B}(x_{q},y_{q})&F^{1}_{B}(x_{r},y_{r})\\ \end{array}\right|$$

$$F_C^1(x,y)=(x-\bar{x})\frac{\partial C}{\partial x}+(y-\bar{y})\frac{\partial C}{\partial y}C\in \{R,G,B\}.$$

$$SCDMI50=[SCDMI{s}_0^1,...,SCDMI{s}_0^{25},SCDMI{s}_1^1,...,SCDMI{s}_1^{25}]$$

$$\begin{array}{rl}SCDM{I}_0^3=& \{6SC{M}_{02001}^{0}SC{M}_{11010}^{0}SC{M}_{20100}^0-6SC{M}_{02001}^{0}SC{M}_{11100}^{0}SC{M}_{20010}^0\\ & -6SC{M}_{02010}^{0}SC{M}_{11001}^{0}SC{M}_{20100}^0+6SC{M}_{02010}^{0}SC{M}_{11100}^{0}SC{M}_{20001}^0\\ & +6SC{M}_{02100}^{0}SC{M}_{11001}^{0}SC{M}_{20010}^0-6SC{M}_{02100}^{0}SC{M}_{11010}^{0}SC{M}_{20001}^0\}\\ & /\{6SC{M}_{00002}^{0}SC{M}_{00020}^{0}SC{M}_{00200}^0-6SC{M}_{00002}^0(SC{M}_{00110}^0{)}^2\\ & -6(SC{M}_{00011}^0{)}^{2}SC{M}_{00200}^0+12SC{M}_{00011}^{0}SC{M}_{00101}^{0}SC{M}_{00110}^0\\ & -6SC{M}_{00020}^0(SC{M}_{00101}^0{)}^2\}\end{array}$$

$$\begin{array}{rl}& \frac{\partial C(x,y)}{\partial x}=C(x-2,y)-8C(x-1,y)+8C(x+1,y)-C(x+2,y)\\ & \frac{\partial C(x,y)}{\partial y}=C(x,y-2)-8C(x,y-1)+8C(x,y+1)-C(x,y+2)\end{array}$$

Full text

1 Introduction

Image classification and retrieval for color images are two hotspots in pattern recognition. How to extract effective features, which are robust to color variations caused by the changes in the outdoor environment and geometric deformations caused by viewpoint changes, is the key issue. The classical approach is to construct invariant features for color images. Moment invariants are widely used invariant features.

Moment invariants were first proposed by Hu[1] in 1962. He defined geometric moments and constructed 7 geometric moment invariants which were invariant under the similarity transform(rotation, scale and translation). Researchers applied Hu moments to many fields of pattern recognition and achieved good results[2, 3]. Nearly 30 years later, Flusser et al.[4] constructed the affine moment invariants (AMIs) which are invariant under the affine transform. The geometric deformations of an object, which are caused by the viewpoint changes, can be represented by the projective transforms. However, general projective transforms are complex nonlinear transformations. So, it’s difficult to construct projective moment invariants. When the distance between the camera and the object is much larger than the size of the object itself, the geometric deformations can be approximated by the affine transform. AMIs have been used in many practical applications, such as character recognition[5] and expression recognition[6]. In order to obtain more AMIs, researchers designed all kinds of methods. Suk et al.[7] proposed graph method which can be used to construct AMIs of arbitrary orders and degrees. Xu et al.[8] proposed the concept of geometric primitives, including distance, area and volume. AMIs can be constructed by using various geometric primitives. This method made the construction of moment invariants have intuitive geometric meaning.

The above-mentioned moment invariants are all designed for gray images. With the popularity of color images, the moment invariants for color images began to appear gradually. Researchers wanted to construct moment invariants which are not only invariant under the geometric deformations but also invariant under the changes of color space. Geusebroek et al.[9] proved that the affine transform model was the best linear model to simulate changes in color resulting from changes in the outdoor environment. Mindru et al.[10] proposed moment invariants which were invariant under the shape affine transform and the color diagonal-offset transform. The invariants were constructed by using the related concepts of Lie group. Some complex partial differential equations had to be solved. Thus, the number of them was limited and difficult to be generalized. Also, Suk et al.[11] put forward affine moment invariants for color images by combining all color channels. But this approach was not intuitive and did not work well for the color affine transform. To solve these problems, Gong et al.[12, 13, 14] constructed the color primitive by using the concept of geometric primitive proposed in [8]. Combining the color primitive with some shape primitives, moment invariants that are invariant under the shape affine and color affine transforms can be constructed easily, which were named shape-color affine moment invariants(SCAMIs). In [14], they obtained 25 SCAMIs which satisfied the independency of the functions. However, we find that a large number of SCAMIs with simple structures and good properties are missed in [14].

In this paper, we propose the general construction formula of shape-color primitives by using partial differentials of each color channel. Then, we use two kinds of shape-color primitives to construct shape-color differential moment invariants(SCDMIs), which are invariant under the shape affine and color affine transforms. We find that the construction formula of SCAMIs proposed in [14] is a special case of our method. Finally, commonly used image descriptors and SCDMIs are used for image classification and retrieval of color images, respectively. By comparing the experimental results, we find that SCDMIs proposed in this paper get better results.

2 Related Work

In order to construct image features which are robust to color variations and geometric deformations, researchers have made various attempts. Among them, SCAMIs proposed in [14] are worthy of special attention. SCAMIs are invariant under the shape affine and color affine transforms. Two kinds of affine transforms are defined by

[TABLE]

[TABLE]

where SA and CA are nonsingular matrices.

For the color image $I(R(x,y),G(x,y),B(x,y))$, let $(x_{p},y_{p}),(x_{q},y_{q}),(x_{r},y_{r})$ be three arbitrary points in the domain of $I$. The shape primitive and the color primitive are defined by

[TABLE]

[TABLE]

where $\bar{A}$ represents the mean value of $A$, $A\in\{x,y,R,G,B\}$.

Then, using Eq.(3) and (4), the shape core can be defined by

[TABLE]

where n and m represent that the sCore is the product of m shape primitives which are constructed by N points $(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})$. $k<l$, $r<n$, $k,l,r\in\left\{1,2,...n\right\}$. $d_{i}$ represents the number of point $(x_{i},y_{i})$ in all shape primitives, $i=1,2,...,n$.

Similarly, the color core can be defined by

[TABLE]

Where N and M represent that the cCore is the product of M color primitives which are constructed by N points $(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{N},y_{N})$. $G<K<L$, $P<Q<N$, $G,K,L,P,Q\in\left\{1,2,...N\right\}$. $D_{i}$ represents the number of point $(x_{i},y_{i})$ in all color primitives, $i=1,2,...,N$.

Suppose the color image $I(R(x,y),G(x,y),B(x,y))$ is transformed into the image $I^{{}^{\prime}}(R^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}),G^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}),B^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}))$ by two transformations defined by Eq.(1) and (2), $(x^{{}^{\prime}}_{p},y^{{}^{\prime}}_{p}),(x^{{}^{\prime}}_{q},y^{{}^{\prime}}_{q}),(x^{{}^{\prime}}_{r},y^{{}^{\prime}}_{r})$ in $I^{{}^{\prime}}$ are the corresponding points of $(x_{p},y_{p}),(x_{q},y_{q}),(x_{r},y_{r})$ in $I$. Gong et al.[14] have proved

[TABLE]

[TABLE]

Further results can be concluded

[TABLE]

[TABLE]

Therefore, the SCAMIs are constructed by

[TABLE]

Then there is a relation

[TABLE]

It must be said that $\max\{n,N\}$, $\max\limits_{i}\{d_{i}\}$ and $\max\limits_{i}\{D_{i}\}$ are named the degree, the shape order and the color order of SCAMIs, respectively. In fact, Eq.(11) can be expressed as polynomial of shape-color moment. This moment was first proposed in [16] and defined by

[TABLE]

Gong et al.[14] proposed that they constructed all SCAMIs of which degrees $\leqslant 4$, shape orders $\leqslant 4$ and color orders $\leqslant 2$. They obtained 24 SCAMIs which are functional independencies using the method proposed by Brown [17]. However, we will point out in the Section 3 that they omitted many simple and well-behaved SCAMIs.

3 The Construction Framework of SCDMIs

In this section, we introduce the general definitions of shape-color differential moment and shape-color primitive, firstly. Then, using the shape-color primitive, the shape-color core can be constructed. Finally, according to Eq.(11) and the shape-color core, we obtain the general construction formula of SCDMIs. Also, 50 instances of SCDMIs are given for experiments in the Section 4.

3.1 The Definition of The General Shape-Color Moment

Definition 1.

Suppose the color image $I(R(x,y),G(x,y),B(x,y))$ have the k-order partial derivatives $(k=0,1,2,...)$. The general shape-color differential moment is defined by

[TABLE]

where $(R^{(k)}(x,y)$, $G^{(k)}(x,y),B^{(k)}(x,y))$ represent the k-order partial derivatives of $(R(x,y),G(x,y),B(x,y))$. $\bar{R},\bar{G},\bar{B}$ represent the mean values of $R,G,B$. $\delta(k)$ is the impact function which is defined by

[TABLE]

We can find that Eq.(14) and Eq.(13) are identical, when $k=0$. Therefore, the shape-color moment is a special case of the general shape-color differential moment.

3.2 The Construction of The General Shape-Color Primitive

Definition 2.

Suppose the color image $I(R(x,y),G(x,y),B(x,y))$ have the k-order partial derivatives $(k=0,1,2,...)$. $(x_{p},y_{p}),(x_{q},y_{q}),(x_{r},y_{r})$ are three arbitrary points in the domain of $I$. The general shape-color primitive is defined by

[TABLE]

where

[TABLE]

We can find that $C(p,q,r)$ defined by Eq.(4) is a special case of $SCP_{k}(p,q,r)$, when $k=0$.

3.3 The Construction of The General Shape-Color Core

Definition 3.

Using Definition 2, the general shape-color core is defined by

[TABLE]

Where $k=1,2,...$, N and M represent that the $scCore_{k}$ is the product of M shape-color primitives constructed by N points $(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{N},y_{N})$. $G<K<L$, $P<Q<N$, $G,K,L,P,Q\in\left\{1,2,...N\right\}$. $D_{i}$ represents the number of point $(x_{i},y_{i})$ in all shape-color primitives, $i=1,2,...,N$.

Obviously, $cCore(N,M;D_{1},D_{2},...,D_{N})$ defined by Eq.(6) is a special case of $scCore_{k}(N,M;D_{1},D_{2},...,D_{N})$, when $k=0$.

3.4 The Construction of SCDMIs

Theorem 1.

Let the color image $I(R(x,y),G(x,y),B(x,y))$ be transformed into the image $I^{{}^{\prime}}(R^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}),G^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}),B^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}))$ by Eq.(1) and Eq.(2), $(x_{p}^{{}^{\prime}},y_{p}^{{}^{\prime}}),(x_{q}^{{}^{\prime}},y_{q}^{{}^{\prime}}),(x_{r}^{{}^{\prime}},y_{r}^{{}^{\prime}})$ in $I^{{}^{\prime}}$ are corresponding points of $(x_{p},y_{p}),(x_{q},y_{q}),(x_{r},y_{r})$ in $I$, respectively. Suppose that $R(x,y),G(x,y),B(x,y),R^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}),G^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}}),B^{{}^{\prime}}(x^{{}^{\prime}},y^{{}^{\prime}})$ have the k-order partial derivatives $(k=0,1,2,...)$. Then there is a relation

[TABLE]

where

[TABLE]

Further, the following relation can be obtained

[TABLE]

where

[TABLE]

By using Maple2015, the proof of Theorem 1 is obvious. We can find that Eq.(10) is a special case of Eq.(20), when $k=0$. So, when we replace $cCore(N,M;\\ D_{1},D_{2},...,D_{N})$ in Eq.(11) with $scCore_{k}(N,M;D_{1},D_{2},...,D_{N})$, Eq.(12) is still tenable. Now, we can define SCDMIs.

Theorem 2.

[TABLE]

Then there is a relation

[TABLE]

where

[TABLE]

Eq.(23) can be expressed as polynomial of $SCM^{k}_{pq\alpha\beta\gamma}$. Eq.(11) is a special case of Eq.(23) when $k=0$. The proof of Eq.(24) is exactly the same as that of Eq.(12) proposed in [14].

3.5 The Instances of SCDMIs

We can use Eq.(24) to construct instances of $SCDMIs$ by setting different k values. However, color images are discrete data, the partial derivatives of each order can’t be accurately calculated. With the elevation of order, the error will be more and more, which will greatly affect the stability of SCDMIs. So, we set $k=0,1$ in this paper.

When $k=0$, $SCDMIs_{0}$ are equivalent to SCAMIs. We construct $SCDMIs_{0}$ of which degrees $\leqslant 4$, shape orders $\leqslant 4$ and color orders $\leqslant 1$. Gong et al. [14] thought that in order to obtain the $SCDMIs_{0}$ of which degrees $\leqslant 4$, shape orders $\leqslant 4$ and color orders $\leqslant 1$, $scCore_{0}(3,1;1,1,1)$ must be $C(1,2,3)$. This judgment is wrong. In fact, $scCore_{0}(3,1;1,1,1)$ can be $C(1,2,3)$, $C(1,2,4)$, $C(1,3,4)$ and $C(2,3,4)$. Thus, lots of $SCDMIs_{0}$ were missed in [14]. By correcting this shortcoming, we get 25 $SCDMIs_{0}$ that satisfy the independence of the function by using the method proposed by [17].

At the same time, when $k=1$, $SCP_{1}(p,q,r)$ is defined by

[TABLE]

where

[TABLE]

By replacing $C(1,2,3)$, $C(1,2,4)$, $C(1,3,4)$ and $C(2,3,4)$ with $SCP_{1}(1,2,3)$, $SCP_{1}(1,2,4)$, $SCP_{1}(1,3,4)$ and $SCP_{1}(2,3,4)$, 25 $SCDMIs_{1}$ can be obtained. Therefore, we can construct the feature vector SCDMI50, which is defined by

[TABLE]

The construction methods of 50 instances are shown in Table 1.

In order to more clearly explain that SCDMIs can be expanded into the polynomial of $SCM^{k}_{pq\alpha\beta\gamma}$, we give the shape-color moment polynomial of $SCMIs_{0}^{3}$ which is represented by

[TABLE]

4 Experimental Results

In this section, some experiments are designed to evaluate the performance of SCDMI50. Firstly, we verify the stability and discriminability of SCDMI50 by using synthetic images. Then, some retrieval experiments based on real image databases are performed. Also, we chose some commonly used image descriptors for comparison.

It is worth noting that we have to choose the method to calculate the first order partial differentials of $R(x,y)$, $G(x,y)$ and $B(x,y)$. The 5 points difference formulas are used for approximating the first partial derivatives of discrete color images, which are defined by

[TABLE]

where $C\in\{R,G,B\}$. We choose this method because it guarantees the computational accuracy of the first partial differential to a certain extent and also maintains a relatively fast calculation speed.

4.1 The Stability and Discriminability of SCDMI50

We select 50 different kinds of butterfly images, which are shown in the Fig.(1.a). Then, 5 shape affine transforms and 4 color affine transforms are applied to each image. So, one image can get 20 transformed versions which are shown in the Fig.(1.b). Thus, we obtain the database containing 1000 images. $10\%$ images are used as the training data and the rest make up the testing data.

For comparison with $SCDMI50$, we chose some commonly used features of gray images or color images.

• •

Hu moments, which were composed of 7 invariants under the shape similarity transform, proposed in [1]. Hu moments were designed for gray images.

• •

AMIs which were composed of 17 invariants under the shape affine transform proposed in [18]. AMIs were designed for gray images.

• •

RGhistogram, which consisted of 60-dimensional features and was scale-invariant in color space, proposed in [19].

• •

Transformed color distribution, which consisted of 60-dimensional features, was proposed in [19]. Transformed color distribution was scale-invariant and sift-invariant in color space.

• •

Color moments consisted of the first, second and third geometric moments of each channel of the color image, which were proposed in [15]. Color moments are sift-invariant in color space.

• •

GPSOs were invariant under the shape affine transforms and the color diagonal-offset transform, which were proposed in [10]. GPSOs consisted of 21 moment invariants.

Subsequently, image classification is performed on the butterfly database by using different kind of features. The Chi-Square distance is selected to measure the similarity of two images. Finally, we plot the classification accuracies obtained by using different features, which are shown in the Table 2.

On the one hand, we can find that the result obtained by using $SCDMI50$ is better than those obtained by using other features. And, color information is very important for the classification. On the other hand, the Table 2 show that $SCDMI50$ have good stability and distinguishability for the shape affine and color affine transforms, and demonstrate that the construction formula of SCDMIs designed in the Section 3 is correct.

4.2 Image Retrieval on Real Image Database

In order to further illustrate the performance of $SCDMI50$ better than the commonly used features listed in the Section 4.1, the database COIL-100 proposed in [20] is chosen for our experiment. COIL-100 contains 7202 images of 100 categories, each of which has 72 images taken from different angles. We choose 30 classes, each class contains 5 images. For each image, 6 color affine transformations are applied. Thus, 900 images of 30 categories are obtained, each of categories contains 30 images which are shown in the Fig.(2).

Similar to the Section 4.1, the image retrieval experiment is performed on this database. The Precision-Recall curves are shown in the Fig.(3).

Obviously, the result obtained by using SCDMI50 is far superior to those obtained by using other features. This is because that traditional image features are constructed by using only color information or shape information. However, we use two kinds of information while constructing SCDMI50. In addition, traditional image descriptors are only stable for simple changes in color space, such as the diagonal-offset transforms. When the color space changes drastically, they are less robust.

Finally, in order to observe the performance of $SCDMIs_{0}$ and $SCDMIs_{1}$ clearly, we compare the retrieval results of $SCDMI50$, $SCDMI_{0}25$ and $SCDMI_{1}25$ which are shown in the Fig.(4). Among them, $SCDMI_{0}25$ and $SCDMI_{1}25$ are defined by

[TABLE]

[TABLE]

Because of the error caused by the inaccuracy of partial derivative calculation, the result obtained by using $SCDMI_{1}25$ slightly worse than the result obtained $SCDMI_{0}25$, but far better than the results obtained by other commonly used features. Meanwhile, $SCDMI_{1}25$ increase the number of invariants which have simple structures and good properties. We combine $SCDMI_{1}25$ and $SCDMI_{0}25$ to get $SCDMI50$ which achieve the best retrieval result in our experiment.

5 Conclusion

In this paper, we proposed a kind of shape-color differential moment invariants (SCDMIs) of color images, which are invariant under the shape affine and color affine transforms, by using partial derivatives of each color channel. It is obvious that all SCAMIs proposed in [14] are the special cases of SCDMIs when $k=0$. Then, we correct a mistake in [14] and obtain 50 instances of SCDMIs, which have simple structures and good properties. Finally, several commonly used image descriptors and SCDMIs are used in image retrieval of color images, respectively. By comparing the experimental results, we find that SCDMIs get better results.