Inaccurate Label Distribution Learning

TL;DR

This paper addresses the challenge of inaccurate label distribution learning by proposing a graph-regularized low-rank and sparse decomposition method to recover true label distributions from noisy data, improving model robustness.

Contribution

It introduces a novel approach to handle noisy label distributions by modeling them as a combination of ideal distributions and sparse noise, utilizing low-rank and graph regularization techniques.

Findings

Effective recovery of true label distributions from noisy data.

Improved LDL model performance on real-world datasets.

Robustness against annotation errors in label distributions.

Abstract

Label distribution learning (LDL) trains a model to predict the relevance of a set of labels (called label distribution (LD)) to an instance. The previous LDL methods all assumed the LDs of the training instances are accurate. However, annotating highly accurate LDs for training instances is time-consuming and very expensive, and in reality the collected LD is usually inaccurate and disturbed by annotating errors. For the first time, this paper investigates the problem of inaccurate LDL, i.e., developing an LDL model with noisy LDs. We assume that the noisy LD matrix is a linear combination of an ideal LD matrix and a sparse noise matrix. Consequently, the problem of inaccurate LDL becomes an inverse problem, where the objective is to recover the ideal LD and noise matrices from the noisy LDs. We hypothesize that the ideal LD matrix is low-rank due to the correlation of labels and utilize the local geometric structure of instances captured by a graph to assist in recovering the ideal LD. This is based on the premise that similar instances are likely to share the same LD. The proposed model is finally formulated as a graph-regularized low-rank and sparse decomposition problem and numerically solved by the alternating direction method of multipliers. Furthermore, a specialized objective function is utilized to induce a LD predictive model in LDL, taking into account the recovered label distributions. Extensive experiments conducted on multiple datasets from various real-world tasks effectively demonstrate the efficacy of the proposed approach. \end{abstract}

Tables

Table 1. Table 1: Details of the datasets.

Index	Data sets	examples	features	labels
1	M2B	1240	250	5
2	RAF-ML	4908	200	6
3	SCUT-FBP	1500	300	5
4	fbp5500	5500	512	5
5	flickr-ldl	11150	200	8
6	twitter-ldl	10040	200	8
7	Yeast-cdc	2465	24	15
8	Yeast-alpha	2465	24	18
9	SBU-3DFE	2500	243	6
10	Movie	7755	1869	5
11	s-JAFFE	213	243	6
12	Nature-scene	2000	294	9

Table 2. Table 2: The distribution distance/similarity measures.

Measure	Formula
Chebyshev $\downarrow$	${\mathrm{Dis}}_1(𝒅,\widehat{𝒅})={\max }_j\left|d_j-{\hat{d}}_j\right|$
Clark	${\mathrm{Dis}}_2(𝒅,\widehat{𝒅})=\sqrt{{\sum }_{j=1}^c\frac{{\left(d_j-{\hat{d}}_j\right)}^2}{{\left(d_j+{\hat{d}}_j\right)}^2}}$
Canberra $\downarrow$	${\mathrm{Dis}}_3(𝒅,\widehat{𝒅})={\sum }_{j=1}^c\frac{\left|d_j-{\hat{d}}_j\right|}{d_j+{\hat{d}}_j}$
Kullback-Leibler$\downarrow$	${\mathrm{Dis}}_4(𝒅,\widehat{𝒅})={\sum }_{j=1}^c{d}_j\ln \frac{d_j}{{\hat{d}}_j}$
cosine $\uparrow$	${\mathrm{Sim}}_1(𝒅,\widehat{𝒅})=\frac{{\sum }_{j=1}^c{d}_j{\hat{d}}_j}{\sqrt{{\sum }_{j=1}^c{d}_j^2}\sqrt{{\sum }_{j=1}^c{\hat{d}}_j^2}}$
S$\varphi$rensen $\downarrow$	${\mathrm{Dis}}_5(𝒅,\widehat{𝒅})=\frac{2\left|𝒅\cap \widehat{𝒅}\right|}{\left|𝒅\right|+\left|\widehat{𝒅}\right|}$
intersection$\uparrow$	${\mathrm{Sim}}_1(𝒅,\widehat{𝒅})={\sum }_i\min (𝒅,\widehat{𝒅})$

Table 3. Table 3: Comparison results (mean ± plus-or-minus \pm std) measured by seven metrics.

data	algorithm	chebyshev	clark	canberra	kldist	cosine	intersection	S$\varphi$rensen
M2B	AA-BP	0.7020±.0888	1.6817±.0924	3.5446±.0007	1.7782±.0022	0.3246±.0003	0.2742±.0693	0.7258±.0693
	AA-KNN	0.5596±.0338	1.5662±.0518	3.2637±.0007	0.7447±.0022	0.7160±.0003	0.4404±.0757	0.5596±.0757
	CPNN	0.4919±.0005	1.6775±.0080	3.5298±.0148	0.8785±.0032	0.6083±.0015	0.4071±.0021	0.5929±.0021
	LDSVR	0.5421±.0000	1.6846±.0009	3.5139±.0025	1.6929±.0215	0.5722±.0014	0.3925±.0014	0.6075±.0003
	LCLR	0.4911±.0050	1.6459±.0055	3.3909±.0152	0.8058±.0323	0.6576±.0056	0.4523±.0043	0.5477±.0043
	LDLSF	0.5050±.0043	1.6575±.0027	3.4292±.0084	0.8306±.0060	0.6466±.0026	0.4392±.0024	0.5608±.0024
	LDLLC	0.4995±.0034	1.6489±.0028	3.3974±.0098	1.0626±.0338	0.6546±.0023	0.4499±.0024	0.5501±.0024
	PT-Bayes	0.5917±.0007	2.0498±.0020	4.4067±.0034	1.5719±.0006	0.5281±.0003	0.4041±.0008	0.5959±.0008
	OURS	0.4499±.0040	1.5743±.0040	3.2218±.0090	0.1592±.0085	0.7312±.0035	0.5246±.0021	0.4754±.0021
RAF	AA-BP	0.4080±.0006	1.6892±.0072	3.7083±.0146	0.2950±.0014	0.5491±.0010	0.4600±.0019	0.5400±.0688
	AA-KNN	0.3573±.0021	1.5698±.0011	3.3801±.0036	0.0803±.0458	0.7137±.0016	0.5421±.0021	0.4579±.0002
	CPNN	0.4000±.0216	1.1745±.0338	2.3595±.0463	0.5378±.0056	0.6301±.0047	0.6000±.0216	0.5612±.0974
	LDSVR	0.4733±.0015	2.1164±.0009	4.9538±.0125	0.0760±.0009	0.5561±.0002	0.3334±.0010	0.6666±.0782
	LCLR	0.3454±.0017	1.5577±.0050	3.3432±.0131	0.5786±.0047	0.7391±.0022	0.5550±.0020	0.4450±.0020
	LDLSF	0.3477±.0016	1.6051±.0036	3.3787±.0102	0.5882±.0048	0.7335±.0029	0.5511±.0021	0.4489±.0021
	LDLLC	0.4984±.0028	1.6526±.0021	3.4142±.0074	0.5834±.0024	0.6587±.0027	0.4490±.0022	0.5510±.0022
	PT-Bayes	0.5971±.0035	2.1911±.0857	5.2174±.0362	0.8998±.0105	0.7248±.0113	0.4029±.0236	0.5971±.1247
	OURS	0.2846±.0040	1.4816±.0040	3.0576±.0090	0.0211±.0085	0.8504±.0035	0.6518±.0021	0.3482±.0021
SCUT	AA-BP	0.3597±.1013	1.4320±.0014	2.8959±.0014	0.1961±.0017	0.7618±.0762	0.5443±.0748	0.4557±.0748
	AA-KNN	0.6356±.0338	1.6880±.0967	3.6875±.0014	0.1471±.0071	0.6253±.0719	0.3644±.0634	0.6356±.0634
	CPNN	0.6933±.1107	1.7230±.0807	3.7820±.0014	0.1903±.0021	0.5101±.0557	0.3067±.0744	0.6933±.0744
	LDSVR	0.9317±.0338	2.6361±.0014	7.3787±.0014	0.6446±.0054	0.5166±.0610	0.3722±.0551	0.8031±.0551
	LCLR	0.3501±.0040	1.4539±.0052	2.8097±.0132	0.5677±.0065	0.7434±.0032	0.5604±.0030	0.4396±.0030
	LDLSF	0.3412±.0033	1.4632±.0035	2.8269±.0106	0.5622±.0046	0.7499±.0023	0.5640±.0026	0.4360±.0026
	LDLLC	0.3521±.0026	1.4627±.0037	2.8317±.0095	0.5726±.0041	0.7425±.0021	0.5578±.0019	0.4422±.0019
	PT-Bayes	0.3927±.0014	1.5204±.0694	3.0080±.0014	0.2213±.0069	0.6617±.0436	0.5025±.0271	0.4975±.0271
	OURS	0.2468±.0029	1.3507±.0050	2.5022±.0140	0.1842±.0113	0.8469±.0023	0.7033±.0024	0.2967±.0024
fbp5500	AA-BP	0.1829±.0014	1.3551±.0014	2.4861±.0014	0.0988±.0017	0.8272±.0658	0.6421±.0859	0.3579±.0859
	AA-KNN	0.3295±.0759	1.4446±.0014	2.7604±.0014	0.0981±.0016	0.7862±.0499	0.5884±.0376	0.4116±.0376
	CPNN	0.3968±.0014	1.5044±.0940	2.9635±.0014	0.1819±.0014	0.6585±.0754	0.5022±.0745	0.4978±.0745
	LDSVR	0.3270±.0014	1.4421±.0812	2.7538±.2398	0.0900±.0014	0.7922±.0612	0.5905±.0609	0.4095±.0609
	LCLR	0.3377±.0180	1.4497±.0167	2.7787±.0556	0.5183±.0558	0.7805±.0373	0.5810±.0240	0.4190±.0240
	LDLSF	0.3326±.0026	1.4497±.0689	2.7798±.0014	0.5184±.0027	0.7854±.2293	0.5861±.3806	0.4139±.1711
	LDLLC	0.3334±.0016	1.4497±.0018	2.7808±.0053	0.5149±.0024	0.7832±.0012	0.5820±.0012	0.4180±.0012
	PT-Bayes	0.3424±.0848	1.5953±.0014	3.3559±.0014	0.3005±.0030	0.6586±.0747	0.4508±.0475	0.5492±.0475
	OURS	0.2733±.0014	1.3931±.0027	2.5892±.0072	0.0542±.0030	0.8688±.0017	0.6610±.0017	0.3390±.0017
flickr	AA-BP	0.1738±.0208	1.0952±.1007	2.5672±.3154	0.2797±.0017	0.7915±.1874	0.6963±.1936	0.3037±.0556
	AA-KNN	0.0680±.0147	0.3163±.0670	0.7203±.1924	0.0300±.0017	0.9662±.0859	0.9035±.1093	0.0965±.0181
	CPNN	0.8854±.0123	2.5295±.0570	7.0966±.1667	0.0257±.0017	0.7183±.0713	0.5095±.0937	0.7962±.0115
	LDSVR	0.0637±.0022	0.2537±.0860	0.5838±.2489	0.0198±.0017	0.9759±.1113	0.9212±.1403	0.0788±.0170
	LCLR	0.8761±.0001	2.5554±.0010	7.1757±.0029	7.6130±.0259	0.6780±.0013	0.4642±.0016	0.8042±.0002
	LDLSF	0.8797±.0001	2.5562±.0005	7.1775±.0014	7.5705±.0093	0.6779±.0008	0.4634±.0008	0.8037±.0002
	LDLLC	0.8761±.0001	2.5574±.0004	7.1816±.0011	4.3153±.3001	0.6750±.0004	0.4608±.0006	0.8048±.0001
	PT-Bayes	0.6004±.0029	2.2813±.0083	6.1057±.0269	1.3497±.0097	0.4749±.0050	0.3143±.0026	0.6857±.0026
	OURS	0.0673±.0678	0.2691±.2703	0.6256±.6275	0.0220±.0222	0.9733±.9730	0.9161±.9157	0.0839±.0843
twitter	AA-BP	0.1733±.0077	1.1054±.0200	2.5808±.0563	0.4290±.0189	0.7944±.0136	0.6997±.0097	0.3003±.0097
	AA-KNN	0.0825±.0004	0.3483±.0008	0.7879±.0019	0.1228±.0004	0.9558±.0003	0.8919±.0003	0.1081±.0003
	CPNN	0.0976±.0041	0.4117±.0271	0.9384±.0583	0.1468±.0112	0.9403±.0057	0.8715±.0079	0.1285±.0079
	LDSVR	0.0873±.0005	0.3250±.0080	0.7427±.0148	0.1112±.0032	0.9583±.0015	0.8979±.0021	0.1021±.0021
	LCLR	0.8754±.0048	2.6268±.0009	7.3826±.0025	5.6168±.0215	0.5876±.0014	0.3482±.0014	0.8177±.0003
	LDLSF	0.8754±.0001	2.6267±.0007	7.3824±.0022	4.7922±.0003	0.5867±.0011	0.3482±.0012	0.8181±.0002
	LDLLC	0.8755±.0000	2.6270±.0008	7.3833±.0024	4.7977±.0014	0.5861±.0012	0.3476±.0014	0.8183±.0002
	PT-Bayes	0.5947±.0035	2.2634±.0085	6.0445±.0292	1.4410±.0103	0.4692±.0044	0.3190±.0030	0.6810±.0030
	OURS	0.0854±.0048	0.3118±.0049	0.7175±.0144	0.0308±.0086	0.9610±.0083	0.9015±.0051	0.0985±.0051

Table 4. Table 4: Summary of the Friedman statistics F F subscript 𝐹 𝐹 F_{F} in terms of each evaluation metric and the critical value at 0.05 0.05 0.05 significance level (# comparing algorithms K = 9 𝐾 9 K=9 , # data sets N = 12 𝑁 12 N=12 ).

Evaluation metric	$F_F$	Critical value $\left(\alpha =0.05\right)$
Chebyshev	$29.841$
Clark	$28.9700$
KL-distance	$30.495$	$15.51$
Canberra	$32.376$
Cosine	$73.5598$
Intersection	$73.559$
S$\varphi$rensen	$30.102$

Table 5. Table 5: Comparison results (mean ± plus-or-minus \pm std) measured by seven metrics.

		chebyshev	clark	canberra	kldist	cosine	intersection	S$\varphi$rensen
alpha	AA-BP	0.0434±.0001	0.9215±.0005	3.1670±.0014	0.0885±.0093	0.9370±.0008	0.8393±.0002	0.1607±.0006
	AA-KNN	0.0275±.0401	0.5602±.0232	1.9409±.0964	0.0336±.0349	0.9691±.0252	0.8955±.0811	0.1045±.0028
	CPNN	0.0724±.0025	0.8629±.0018	3.2067±.0088	0.0933±.0045	0.9028±.0016	0.8121±.0015	0.1879±.0057
	LDSVR	0.0294±.0029	0.5970±.0041	1.9463±.0157	0.0390±.0096	0.9652±.0028	0.8947±.0021	0.1053±.0006
	LCLR	0.0151±.0016	0.2412±.0011	0.7946±.0031	0.0070±.0036	0.9930±.0013	0.9560±.0002	0.0440±.0054
	LDLSF	0.0151±.0023	0.2413±.0032	0.7953±.0109	0.0070±.0070	0.9930±.0093	0.9560±.0032	0.0440±.0033
	LDLLC	0.0145±.0029	0.2252±.0033	0.7344±.0083	0.4233±.0054	0.9939±.0019	0.9584±.0018	0.0406±.0083
	PT-Bayes	0.3875±.0015	2.0231±.0010	7.6577±.0039	0.6384±.0048	0.4843±.0016	0.5185±.0009	0.4815±.0018
	OURS	0.0143±.0006	0.2201±.0028	0.7446±.0057	0.0063±.0006	0.9938±.0006	0.9590±.0007	0.0411±.0080
cdc	AA-BP	0.0669±.0024	1.0440±.0036	2.5764±.0079	0.1639±.0069	0.9139±.0021	0.8433±.0017	0.1567±.0028
	AA-KNN	0.0448±.0106	0.6288±.0276	2.0304±.0790	0.0543±.0199	0.9484±.0213	0.8629±.0130	0.1371±.0076
	CPNN	0.0620±.0058	1.4701±.0024	4.6585±.0080	0.2909±.0207	0.8524±.0011	0.7283±.0013	0.2717±.0030
	LDSVR	0.0232±.0020	0.2772±.0036	0.8296±.0144	0.0094±.0091	0.9910±.0038	0.9459±.0020	0.0541±.0019
	LCLR	0.0168±.0020	0.2237±.0027	0.6752±.0085	0.0074±.0012	0.9928±.0011	0.9555±.0011	0.0445±.0021
	LDLSF	0.0168±.0020	0.2236±.0036	0.6748±.0144	0.0074±.0091	0.9928±.0038	0.9555±.0020	0.0445±.0211
	LDLLC	0.0164±.0024	0.2147±.0036	0.6435±.0079	0.9225±.0069	0.9933±.0021	0.9576±.0017	0.0424±.0028
	PT-Bayes	0.2408±.0042	1.9271±.0135	6.8458±.0342	0.5819±.0023	0.6639±.0026	0.5617±.0044	0.4383±.0029
	OURS	0.0161±.0028	0.2163±.0076	0.6493±.0211	0.0073±.0030	0.9942±.0019	0.9578±.0021	0.0428±.0019
sja	AA-BP	0.4391±.0001	1.3493±.0004	2.7821±.0011	0.7564±61.4300	0.5295±.0004	0.5365±.0006	0.4635±.0001
	AA-KNN	0.1646±.0001	0.5756±.0005	1.0271±.0014	0.1006±.0093	0.9024±.0008	0.8354±.0008	0.1926±.0002
	CPNN	0.1986±.0040	0.7276±.0040	1.3999±.0090	0.1796±.0085	0.8364±.0035	0.7681±.0021	0.2319±.0021
	LDSVR	0.1716±.0795	0.6218±.0031	1.1184±.0042	0.1175±.0061	0.8917±.0016	0.8284±.0015	0.1716±.0012
	LCLR	0.1277±.0039	0.4430±.0057	0.9331±.0080	0.0792±.0018	0.9248±.0016	0.8400±.0759	0.1600±.0748
	LDLSF	0.1240±.0051	0.4693±.0112	0.9814±.0232	0.0754±.0039	0.9309±.0037	0.8381±.0043	0.1619±.0043
	LDLLC	0.1129±.0227	0.4690±.0731	0.9822±.0878	4.4745±.0565	0.9308±.0357	0.8380±.0256	0.1620±.1174
	PT-Bayes	0.2714±.0584	0.7375±.0918	1.4657±.0192	0.2256±.0168	0.7842±.0190	0.7286±.0557	0.2714±.0782
	OURS	0.0940±.0227	0.1893±.0731	0.2903±.0878	0.0342±.0565	0.9721±.0357	0.9060±.0256	0.0940±.0002
SBU	AA-BP	0.3012±.0040	1.1851±.0080	2.7515±.0187	0.6154±1.5501	0.5961±.0037	0.5337±.0036	0.4663±.0036
	AA-KNN	0.2419±.0029	0.6145±.0033	1.3260±.0083	0.3355±.0054	0.8456±.0019	0.6884±.0018	0.2546±.0018
	CPNN	0.2689±.0053	0.8444±.0277	1.9320±.0656	0.3131±.0149	0.7359±.0077	0.6558±.0089	0.3442±.0089
	LDSVR	0.2324±.0279	1.1012±.0238	2.4310±.0621	0.4319±.1620	0.7189±.0093	0.6033±.0076	0.3967±.0112
	LCLR	0.1345±.0027	0.4134±.0068	0.9043±.0160	0.0848±.0021	0.9179±.0022	0.8384±.0028	0.1616±.0028
	LDLSF	0.1383±.0000	0.4112±.0008	0.8998±.0024	0.0840±.0014	0.9186±.0012	0.8392±.0014	0.1608±.0002
	LDLLC	0.1400±.0023	0.4139±.0032	0.9045±.0109	0.0859±.0070	0.9169±.0093	0.8381±.0032	0.1619±.0033
	PT-Bayes	0.3044±.0029	0.8913±.0033	1.9535±.0083	0.4238±.0054	0.6885±.0019	0.6276±.0018	0.3724±.0083
	OURS	0.1270±.0001	0.3919±.0007	0.8499±.0022	0.0672±.0003	0.9288±.0011	0.8485±.0012	0.1515±.0002
MOVIE	AA-BP	0.1743±.0024	0.7322±.0036	1.3920±.0079	0.4005±.0069	0.8671±.0021	0.7569±.0017	0.2431±.0083
	AA-KNN	0.1695±.0007	0.7123±.0022	1.3483±.0045	0.3992±.0018	0.8775±.0007	0.7596±.0008	0.2404±.0008
	CPNN	0.1786±.0030	0.7414±.0088	1.4142±.0182	0.4299±.0108	0.8629±.0053	0.7439±.0046	0.2561±.0046
	LDSVR	0.1807±.0016	0.7508±.0035	1.4321±.0077	0.4411±.0038	0.8593±.0020	0.7398±.0018	0.2602±.0018
	LCLR	0.1654±.0042	0.7093±.0135	1.3432±.0342	0.1683±.0023	0.8827±.0026	0.7615±.0044	0.2385±.0029
	LDLSF	0.1735±.0028	0.7322±.0076	1.3915±.0211	0.1829±.0030	0.8704±.0019	0.7501±.0021	0.2499±.0054
	LDLLC	0.1817±.0019	0.7552±.0063	1.4398±.0174	0.1976±.0135	0.8581±.0017	0.7386±.0023	0.2614±.0019
	PT-Bayes	0.1850±.0011	0.7627±.0025	1.4609±.0061	0.4506±.0025	0.8564±.0009	0.7357±.0011	0.2643±.0011
	OURS	0.1338±.0040	0.6105±.0080	1.1399±.0187	0.1464±1.5501	0.9222±.0037	0.8077±.0036	0.1923±.0036
NATURE	AA-BP	0.4040±.0104	2.5361±.0126	7.1280±.0630	3.9933±.1458	0.5036±.0254	0.3678±.0175	0.6322±.0175
	AA-KNN	0.3566±.0059	2.4758±.0096	6.9364±.0399	3.7662±.0269	0.6313±.0038	0.3948±.0045	0.6052±.0045
	CPNN	0.3818±.0064	2.5189±.0106	7.1490±.0474	4.1641±.0636	0.5343±.0166	0.3388±.0087	0.6612±.0087
	LDSVR	0.3642±.0061	2.4766±.0074	6.9645±.0277	3.9580±.0179	0.5798±.0032	0.3679±.0028	0.6321±.0028
	LCLR	0.3583±.0036	2.4808±.0155	6.8452±.0285	1.1300±.0068	0.6766±.0047	0.4662±.0055	0.5338±.0055
	LDLSF	0.3744±.0090	2.5772±.0234	7.1752±.0513	1.7217±.0090	0.6083±.0101	0.4530±.0100	0.5470±.0100
	LDLLC	0.6816±.0027	2.8773±.0068	8.4701±.0160	2.8957±.0021	0.4247±.0022	0.3069±.0028	0.6931±.0028
	PT-Bayes	0.4285±.0076	2.5436±.0073	7.2272±.0297	3.9983±.0667	0.5423±.0064	0.3399±.0052	0.6601±.0052
	OURS	0.3177±.0030	2.4525±.0050	6.7557±.0208	1.1300±.0067	0.7158±.0020	0.4749±.0023	0.5251±.0023

Table 6. Table 6: Average ranking of algorithms across 12 datasets under different metrics.

	AA-BP	AA-KNN	CPNN	LDSVR	LCLR	LDLSF	LDLLC	PT-Bayes	Ours
Chebyshev	6.17	4.67	5.67	5.33	4.00	4.54	5.38	7.83	1.42
Clark	5.79	4.00	6.08	5.50	4.00	5.13	5.33	7.67	1.50
Canberra	6.00	4.25	6.25	5.42	3.67	4.67	5.50	7.83	1.42
kl	6.00	3.67	5.42	4.92	5.33	4.58	6.67	7.17	1.25
Cosine	9.00	8.00	6.96	6.04	5.00	4.00	2.96	2.04	1.00
Intersection	9.00	8.00	6.92	6.08	5.00	3.96	3.04	2.00	1.00
Sorensen	5.92	4.67	6.58	5.58	3.92	4.17	5.33	7.42	1.42

Table 7. Table 7: Prediction results measured by (Chebyshev ↓ ↓ \downarrow , Clark ↓ ↓ \downarrow , Cosine ↑ ↑ \uparrow , S ϕ italic-ϕ \phi rensen ↓ ↓ \downarrow ) for each compared algorithm on the controlled dataset (with b=0.2). For the LDL algorithm ℱ ∈ { \mathcal{F}\in\{ AA-BP, AA-KNN, CPNN, LDSVR,PT-Bayes } } \} , the performance of the ℱ ℱ \mathcal{F} -LSag is compared against that of ℱ ℱ \mathcal{F} , ℱ ℱ \mathcal{F} (G), ℱ ℱ \mathcal{F} (I) represent LDL algorithm training with ground-truth LD and noise LD respectively.

Chebyshev$\downarrow$
	$ℱ$=AABP			$ℱ$=AA-KNN			$ℱ$=CPNN			$ℱ$=LDSVR			$ℱ$=PT-Bayes
Dataset	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(GT)	$ℱ$-LSag	$ℱ$(I)
Yeast-alpha	0.0750	0.0843	0.1207	0.0142	0.0217	0.0275	0.0126	0.0101	0.0724	0.0150	0.0117	0.0193	0.1269	0.0767	0.3875
Yeast-cdc	0.0696	0.0303	0.1147	0.0238	0.0242	0.0448	0.0249	0.0198	0.0620	0.0201	0.0204	0.0232	0.1422	0.1675	0.2408
s-JAFFE	0.0839	0.0796	0.4391	0.0991	0.0883	0.1646	0.0302	0.0667	0.1986	0.2197	0.0828	0.1716	0.1512	0.0828	0.2714
SBU 3DFE	0.2981	0.1679	0.3012	0.2376	0.1715	0.2419	0.0732	0.1475	0.2689	0.1983	0.2158	0.2324	0.0828	0.2179	0.3044
Average rank		1	2		1	2		1	2		1	2		1	2
clark$\downarrow$
	$ℱ$=AABP			$ℱ$=AA-KNN			$ℱ$=CPNN			$ℱ$=LDSVR			$ℱ$=PT-Bayes
Dataset	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(GT)	$ℱ$-LSag	$ℱ$(I)
Yeast-alpha	1.3956	1.7023	1.8428	0.2132	0.4554	0.5602	0.1927	0.1816	0.8629	0.2486	0.1765	0.4174	1.9051	0.0767	0.3875
Yeast-cdc	1.5694	0.4751	1.6540	0.2892	0.5043	0.6288	0.3007	0.2488	1.4701	0.2539	0.2424	0.2772	1.4234	1.4526	1.9271
s-JAFFE	1.3260	0.4982	1.3493	0.6934	0.4760	0.5756	0.3394	0.3026	0.7276	0.6493	0.3650	0.6218	0.3650	0.3650	0.7375
SBU 3DFE	0.6008	0.9449	1.1851	0.3962	0.4902	0.6145	0.4644	0.5130	0.8444	0.5602	0.6509	1.1012	0.6485	0.6485	0.8913
Average rank		1	2		1	2		1	2		1	2		1	2
cosine$\uparrow$
	$ℱ$=AABP			$ℱ$=AA-KNN			$ℱ$=CPNN			$ℱ$=LDSVR			$ℱ$=PT-Bayes
Dataset	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(GT)	$ℱ$-LSag	$ℱ$(I)
Yeast-alpha	0.8489	0.8195	0.7641	0.9949	0.9965	0.9691	0.9958	0.9963	0.9028	0.9937	0.9965	0.9814	0.7442	0.8520	0.4843
Yeast-cdc	0.8267	0.9717	0.7674	0.9902	0.9933	0.9484	0.9894	0.9929	0.8524	0.9925	0.9933	0.9910	0.7867	0.8703	0.6639
s-JAFFE	0.6994	0.9439	0.5295	0.8110	0.9726	0.9024	0.9759	0.9737	0.8364	0.8563	0.9710	0.8917	0.9710	0.9710	0.7842
SBU 3DFE	0.8251	0.8927	0.5961	0.9572	0.9139	0.8456	0.9265	0.9222	0.7359	0.8832	0.8456	0.7189	0.8454	0.8454	0.6885
Average rank		1	2		1	2		1	2		1	2		1	2
S$\varphi$rensen$\downarrow$
	$ℱ$=AABP			$ℱ$=AA-KNN			$ℱ$=CPNN			$ℱ$=LDSVR			$ℱ$=PT-Bayes
Dataset	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(G)	$ℱ$-LSag	$ℱ$(I)	$ℱ$(GT)	$ℱ$-LSag	$ℱ$(I)
Yeast-alpha	0.2340	0.2946	0.3569	0.0412	0.1010	0.1045	0.0370	0.0353	0.1879	0.0438	0.0327	0.0811	0.3447	0.2399	0.4815
Yeast-cdc	0.2898	0.0889	0.3179	0.0526	0.1248	0.1371	0.0510	0.0406	0.2717	0.0511	0.0442	0.0541	0.3070	0.2354	0.4383
s-JAFFE	0.4115	0.1699	0.4635	0.2376	0.1646	0.1926	0.0899	0.0912	0.2319	0.2017	0.0958	0.1716	0.0958	0.0958	0.2714
SBU 3DFE	0.2175	0.2562	0.4663	0.1256	0.1784	0.2546	0.1662	0.1862	0.3442	0.2214	0.2619	0.3967	0.2584	0.2584	0.3724
Average rank		1	2		1	2		1	2		1	2		1	2

Table 8. Table 8: Wilcoxon signed-rank test between ℱ ℱ \mathcal{F} -LSag and ℱ ℱ \mathcal{F} in terms of (Chebyshev ↓ ↓ \downarrow , clark ↓ ↓ \downarrow , Cosine ↑ ↑ \uparrow , S ϕ italic-ϕ \phi rensen ↓ ↓ \downarrow ). Significance level α 𝛼 \alpha =0.05.

$ℱ$-LSag		Evaluation metric
vs		Chebyshev$\downarrow$	clark$\downarrow$	cosine$\uparrow$	S$\varphi$ren$\downarrow$
$ℱ$	AA-BP	win[4.88e-04]	win[4.88e-04]	win[4.88e-04]	win[4.88e-04]
	AA-KNN	win[4.88e-04]	win[4.88e-04]	win[6.84e-03]	win[4.88e-04]
	CPNN	win[4.88e-04]	win[4.88e-04]	win[4.88e-04]	win[4.88e-04]
	LDSVR	win[1.46e-03]	win[9.77e-04]	win[9.77e-04]	win[9.77e-04]
	PT-Bayes	win[4.88e-04]	win[4.88e-04]	win[4.88e-04]	win[4.88e-04]

Equations

$$\mathbf{D}=\tilde{\mathbf{D}}+\mathbf{E},$$

$$\underset{\tilde{\mathbf{D}},\mathbf{E}}{\min }\mathrm{rank}(\tilde{\mathbf{D}})+\alpha \mathrm{card}(\mathbf{E})$$

$$\text{ s.t. }\mathbf{D}=\tilde{\mathbf{D}}+\mathbf{E},$$

$$\underset{a_i}{\min }\sum _{j=1}^n\left(\frac{1}{2}{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }_2^2{a}_{ij}+\gamma a_{ij}^2\right)$$

$$\text{ s.t. }{a_i}^T{1}_n=1_n,\forall i.j,0\le a_{ij}\le 1,$$

$$\sum _{i,j}\min a_{ij}{\Vert {\tilde{\mathbf{d}}}_i-{\tilde{\mathbf{d}}}_j\Vert }^2=\mathrm{Tr}\left(\tilde{\mathbf{D}}\mathbf{L}{\tilde{\mathbf{D}}}^{\mathrm{T}}\right),$$

$$\underset{\tilde{\mathbf{D}},\mathbf{E}}{\min }\mathrm{rank}(\tilde{\mathbf{D}})+\alpha \mathrm{card}(\mathbf{E})+\beta \mathrm{Tr}\left(\tilde{\mathbf{D}}\mathbf{L}{\tilde{\mathbf{D}}}^{\mathrm{T}}\right)$$

$$\text{ s.t. }\mathbf{D}=\tilde{\mathbf{D}}+\mathbf{E},$$

$$\underset{\tilde{\mathbf{D}},\mathbf{E}}{\min }\alpha \Vert \mathbf{E}{\Vert }_1+\Vert \tilde{\mathbf{D}}{\Vert }_{\ast }+\beta \mathrm{Tr}\left(\tilde{\mathbf{D}}\mathbf{L}{\tilde{\mathbf{D}}}^{\mathrm{T}}\right)$$

$$\text{ s.t. }\mathbf{D}=\tilde{\mathbf{D}}+\mathbf{E}.$$

$$\underset{(\mathbf{\Theta },\mathbf{b})}{\min }\frac{1}{2}\Vert \mathbf{\Theta }{\Vert }_F^2+\kappa \ell ((\mathbf{\Theta },\mathbf{b}))$$

$$\text{ s.t }\forall i,{\tilde{d}}_i^j\left({\mathbf{\theta }}_j^{\top }\phi \left({\mathbf{x}}_i\right)+b_j\right)⩾0,$$

$$\underset{\tilde{\mathbf{D}},\mathbf{E}}{\min }\alpha \Vert \mathbf{E}{\Vert }_1+\Vert \mathbf{Z}{\Vert }_{\ast }+\beta \mathrm{Tr}\left(\tilde{\mathbf{D}}\mathbf{L}{\tilde{\mathbf{D}}}^{\mathrm{T}}\right)$$

$$\text{ s.t. }\mathbf{D}=\tilde{\mathbf{D}}+\mathbf{E},\tilde{\mathbf{D}}=\mathbf{Z}.$$

$${\mathcal{L}}_{(\tilde{\mathbf{D}},\mathbf{E},\mathbf{Z},{\mathbf{\Gamma }}_1,{\mathbf{\Gamma }}_2)}=\beta \mathrm{Tr}\left(\tilde{\mathbf{D}}\mathbf{L}{\tilde{\mathbf{D}}}^{\mathbf{T}}\right)+\alpha \Vert \mathbf{E}{\Vert }_1+\Vert \mathbf{Z}{\Vert }_{\ast }+⟨{\mathbf{\Gamma }}_2,\tilde{\mathbf{D}}-\mathbf{Z}⟩$$

$$+⟨{\mathbf{\Gamma }}_1,\tilde{\mathbf{D}}+\mathbf{E}-\mathbf{D}⟩+\frac{\mu }{2}\left(\Vert {\tilde{\mathbf{D}}+\mathbf{E}-D\Vert }_F^2+{\Vert \tilde{\mathbf{D}}-Z\Vert }_F^2\right),$$

$$\underset{\mathbf{Z}}{\min }\Vert \mathbf{Z}{\Vert }_{\ast }+⟨{\mathbf{\Gamma }}_2,\tilde{\mathbf{D}}-\mathbf{Z}⟩+\frac{\mu }{2}\Vert \tilde{\mathbf{D}}-\mathbf{Z}{\Vert }_F^2.$$

$${\mathbf{Z}}^{k+1}={\mathcal{J}}_{1/\mu }\left({\tilde{\mathbf{D}}}^{k+1}+\frac{{\mathbf{\Gamma }}_2^k}{{\mu }^k}\right),$$

$$\underset{\tilde{\mathbf{D}}}{\min }\beta \mathrm{Tr}\left(\tilde{\mathbf{D}}\mathbf{L}{\tilde{\mathbf{D}}}^{\mathbf{T}}\right)+⟨{\mathbf{\Gamma }}_1,\tilde{\mathbf{D}}+\mathbf{E}-\mathbf{D}⟩$$

$$+⟨{\mathbf{\Gamma }}_2,\tilde{\mathbf{D}}-\mathbf{Z}⟩+\frac{\mu }{2}\left(\Vert {\tilde{\mathbf{D}}+\mathbf{E}-D\Vert }_F^2+{\Vert \tilde{\mathbf{D}}-Z\Vert }_F^2\right).$$

$${\tilde{\mathbf{D}}}^{k+1}=-{\mu }^k\left({\psi }_1+{\psi }_1\right)/\left(2{\mathbf{L}}^k+2{\mu }^k\mathbf{I}\right).$$

$$\underset{\mathbf{E}}{\min }\alpha \Vert \mathbf{E}{\Vert }_1+⟨{\mathbf{\Gamma }}_1,\tilde{\mathbf{D}}+\mathbf{E}-\mathbf{D}⟩+\frac{\mu }{2}\Vert \tilde{\mathbf{D}}+\mathbf{E}-\mathbf{D}{\Vert }_F^2.$$

$${\mathbf{E}}^{k+1}={\delta }_{\alpha /\mu }\left(\mathbf{D}-{\tilde{\mathbf{D}}}^{k+1}+\frac{{\mathbf{\Gamma }}_1^k}{{\mu }_k}\right),$$

$$\left\{\begin{array}[]{l}\mathbf{\Gamma}_{1}^{k+1}=\mathbf{\Gamma}_{1}^{k}+\mu^{k}\left(\mathbf{\widetilde{D}}^{k+1}+\mathbf{{E}}^{k+1}-\mathbf{D}^{k+1}\right)\\ \mathbf{\Gamma}_{2}^{k+1}=\mathbf{\Gamma}_{2}^{k}+\mu^{k}\left(\mathbf{\widetilde{D}}^{k+1}-\mathbf{Z}^{k+1}\right)\\ \mu^{k+1}=\min\left(1.1\mu,\mu_{\max}\right).\end{array}\right.$$

$$\underset{a_i}{\min }\sum _{j=1}^n\frac{1}{2}{\Vert a_i+\frac{1}{4r}{u}_i\Vert }_2^2{a}_{ij}$$

$$\text{ s.t. }{a_i}^T{1}_n=1_n,\forall i.j,0\le a_{ij}\le 1,$$

$$\mathcal{L}\left(a_i,\varpi ,\varrho \right)=\frac{1}{2}{\Vert a_i+\frac{1}{4r}{u}_i\Vert }_2^2{a}_{ij}$$

$$-\varpi \left(a_i^T{1}_n-1\right)-{\varrho }_i^T{a}_i,$$

$$\left\{\begin{array}[]{l}\forall j,\quad\frac{1}{4r}u_{j}+a_{ij}-\varpi-\varrho_{j}=0,\\ \forall j,\quad\varrho_{j}\geq 0,\quad 0\leq a_{ij}\leq 1,\\ \forall j,a_{ij}\varrho_{j}=0.\end{array}\right.$$

$$a_j={\left(f_j-\bar{\varrho }\right)}_{+}$$

$$f(\bar{\varrho })=\frac{1}{n}\sum _{j=1}^n{\left(\bar{\varrho }-f_{ij}\right)}_{+}-\bar{\varrho }=0.$$

$${\bar{\varrho }}_{t+1}={\bar{\varrho }}_t-\frac{f\left({\bar{\varrho }}_t\right)}{f^{\prime }\left({\bar{\varrho }}_t\right)},$$

$$\tilde{\ell }\left(u_i\right)=\ell \left(u_i^{(k)}\right)+{\frac{d\ell }{du}|}_{u_i^{(k)}}\frac{{\left({\mathbf{e}}_i^{(k)}\right)}^{\top }}{u_i^{(k)}}\left({\mathbf{e}}_i-{\mathbf{e}}_i^{(k)}\right)$$

$$\bar{\ell }\left(u_i\right)=\ell \left(u_i^{(k)}\right)+{\frac{d\ell \left(u_i\right)}{d{u}_i}|}_{u_i^{(k)}}\frac{u_i^2-{\left(u_i^{(k)}\right)}^2}{2u_i^{(k)}}$$

$$=\frac{1}{2}{\xi }_i{u}_i^2+\tau ,$$

$${\xi }_i={\frac{1}{u_i^{(k)}}\frac{\ell \left(u_i\right)}{d{u}_i}|}_{u_i^{(k)}}=\{\begin{array}{ll}0& u_i^{(k)}<\epsilon \\ \frac{2\left(u_i^{(k)}-\epsilon \right)}{u_i^{(k)}}& u_i^{(k)}\ge \epsilon \end{array}$$

Full text

{IEEEImpStatement}

LDL has gained popularity among researchers for addressing label ambiguity problems and yielding promising results. It provides precise supervision information for finer-grained predictions. However, accurate labeling of training instances is time-consuming and expensive, leading to inaccuracies and noise in real-world scenarios. To address this challenge, this paper introduces a novel method based on graph-regularized low-rank and sparse decomposition. Our method enhances model robustness against label distribution noise, ensuring reliable performance in challenging conditions. It has the potential to support various LDL methods, including facial expression recognition, facial age estimation, and other intelligent detection and recognition scenarios.

{IEEEkeywords}

Label distribution learning, Inaccurate label distribution learning, Multi-label learning, Noise label learning.

1 Introduction

\IEEEPARstart

Label distribution learning (LDL) is an emerging topic in machine learning. Different from the traditional single-label learning and multi-label learning, which use binary value to specify whether an instance is related to a certain label, LDL solves the problem of to what degree a label can describe an instance. This powerful learning paradigm is good at handling label ambiguity and has many real-world applications, like music classification [1], breast tumor cellularity assessment [2], and facial age estimation [3].

LDL was first proposed by Geng [4]. In LDL, the relative importance of each label to an instance is called the description degree, which is captured by a label distribution (LD). Fig. 1(a) shows a multi-label scene image, where “lake” has higher importance than “cloud”, and at the same time both of them are positive labels, so it makes sense to know the description degree of each label, which forms an LD as shown in Fig. 1(b). Similar to other machine learning paradigms, in the training phase of LDL, a training set with many instances and the annotated LDs are given to train an LDL model. In the test phase, the learned LDL model is used to predict the LD for an unseen sample.

To solve the LDL problem, different models have been proposed. For example, Jia et al. [5] used label correlations on local samples and proposed two new LDL algorithms, called GD-LDLSCL and Adam-LDL-SCL, respectively. In order to solve the objective mismatch and improve the classification performance of LDL, Wang and Geng [6] proposed the label distribution learning machine. To reduce the high computational overhead of LDL, Tan et al [7] developed an LDL algorithm based on stream learning with multiple output regression, called MDLRML. To avoid the problem that LDL treats data differently in the training stage and testing stage, Wang et al. [8] proposed the re-weighting large margin label distribution learning. Considering the label ranking relationships, Jia et al. [9] introduced a ranking loss function to the traditional LDL models.

Motivation: Although those LDL methods have achieved great success in many applications, all of them assumed that the LDs of the training instances are accurate, however, precisely assigning an accurate LD to an instance is extremely time-consuming and expensive. Therefore, in reality, the LDs collected in the training set are usually inaccurate with many noises, and inaccurate LDs become a common phenomenon in LDL. For example, Fig. 1(b) shows the ground truth LD of an instance, and Fig. 1(c) denotes the inaccurate LD, which puts higher description degree to the label “colud” and lower description degree to the label “lake”. It is very important to investigate how to construct a reliably LDL model with inaccurate LDs, which unfortunately has been overlooked by the previous researches.

In this paper, we study the problem of inaccurate label distribution learning (ILDL), i.e., design an LDL model with a training set annotated by inaccurate LD, for the first time. Specifically, we assume that the inaccurate LD is the linear combination of an ideal LD and a sparse noise. Then, the ILDL problem can be treated as an inverse problem to separate the ideal LD and the noise label from the inaccurate observations. To this end, we collect the LDs of all the instances to construct an LD matrix and assume that the ideal LD matrix is low-rank, since the labels are usually correlated to each other in multi-label learning [10], and the noise label matrix is sparse due to the fact that the coarse labeling usually generates only a small fraction of noise. Moreover, if two instances are similar enough, their LDs should also be similar to each other. Motivated by this observation, we use the local similarity structure of the instrances to assist the recovery of the ideal LD. Finally, we formalize the proposed model as a low-rank and sparse decomposition problem with a graph regularization (LSag) and solve it using the alternating direction method of multipliers (ADMM) [11]. The recovered LD are taken into consideration when inducing a LD predictive model for LDL, achieved through the utilization of a specialized objective function. Extensive experiments validate the advantage of our approach over the state-of-the-art approaches.

We organize the rest of the paper as follows. First, related works on LDL are briefly discussed. Second, technical details of the proposed approach are introduced. Third, experimental results of comparative studies are reported. Finally, we conclude this paper.

2 Related Work

2.1 Lable Distribution Learning

As a new learning paradigm, LDL can better describe the labeling degree of an instance than the traditional multi-label learning. Accordingly, LDL has attracted a lot of attention. In this section, we briefly review the researches in LDL.

The develop of LDL is inspired by solving various real-world applications. For example, in the early years, LDL shined in facial age recognition task [12]. After that, Geng [13] proposed an LDL-based head pose estimation algorithm, which makes full use of the multi-label distribution information. Zhou et al. [14] found that all facial expressions in nature cannot be defined by a binary label, and accordingly, they developed a facial emotion recognition algorithm based on LDL. In addition, the idea of LDL has been applied to the prediction of multi-component compositions of Martian craters [15], age estimation of the speaker [16], indoor crowd counting [17], and infant age estimation [18].

Apart from the real-world applications, many researches focus on developing an effective LDL model for general purposes. We roughly divide the existing LDL algorithms into three categories. The first category converts the LDL problem into a single-label learning problem, i.e. transforming the training samples into a set of weighted single-label samples. The representative algorithms are PT-SVM and PT-Bayes [4], which use the SVM algorithm and the Bayes classifier to solve the transformed weighted single-label learning problem. The second category is algorithm adaption, which extends the traditional machine learning algorithms to deal with the LDL problem. For example, the K-nearest neighbors (KNN) classifier finds the top $k$ neighbors of an instance and uses the average labels of the top $k$ neighbors as the prediction of the LD of that instance. Backpropagation (BP) neural networks can directly minimizes the descriptive degree of the final prediction through the BP algorithm. The last category is specialized algorithms, such as IIS-LDL and BFGS-LDL [12]. They formulated LDL as a regression problem and used an improved iterative scaling algorithm and a quasi-Newton method to solve the final regression problem, respectively.

As the LDs are usually annotated by different persons with diverse levels of experience, assigning a precise description degree to all instances is very challenging, and inaccurate LD is a common phenomenon in LDL. However, the previous researches all assumed the LD of the training set is accurate, which cannot handle the inaccurate LDL problem. This paper will investigate the inaccurate LDL problem for the first time.

3 The Proposed Method

Notations: Let $\mathbf{X}=\left[\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{n}\right]\in\mathbbm{R}^{n\times d}$ denote the feature matrix, and $Y=\left\{y_{1},y_{2},\ldots,y_{m}\right\}$ be the label space, where $n$, $m$, and $d$ denote the number of instances, the number of the labels and the dimension of features. The training set of the LDL problem is represented as: $\mathbbm{T}=\left\{\left(\mathbf{x}_{1},\mathbf{d}_{1}\right),\quad\left(\mathbf{x}_{2},\mathbf{d}_{2}\right),\ldots,\left(\mathbf{x}_{n},\mathbf{d}_{n}\right)\right\}$, where $\mathbf{d}_{i}=\left[d_{\mathbf{x}_{i}}^{y_{1}},d_{\mathbf{x}_{i}}^{y_{2}},\ldots,d_{\mathbf{x}_{i}}^{y_{m}}\right]$ is the label distribution vector to the $i$th sample $\mathbf{x}_{i}$. $d_{\mathbf{x}_{i}}^{y}$ indicates the importance degree of label $y$ to $\mathbf{x}_{i}$, which satisfies $d_{\mathbf{x}_{i}}^{y}\in[0,1]$ and $\sum_{y}d_{x}^{y}=1$. The LD matrix of all the instances is denoted as $\mathbf{D}=\left[\mathbf{d}_{1},\mathbf{d}_{2},\ldots,\mathbf{d}_{n}\right]\in\mathbbm{R}^{n\times m}$. LDL aims to learn a mapping function from $\mathbbm{T}$, which can predict the LD for unseen instances.

The traditional LDL approaches all assume that LD matrix $\mathbf{D}$ is accurate, but considering the fact that precisely annotating the LD for an instance is very costly, in reality, the collected LD matrix is usually not accurate, which is polluted by labeling noise. Directly training an LDL model with noisy LD will certainly result in unsatisfactory performance. To this end, this paper investigates the problem of inaccurate LDL, which can construct a reliable LDL model from the noisy LD.

3.1 Low-rank and Sparse Decomposition of the Noisy Label Distribution

To achieve inaccurate LDL, we assume that the observed noisy LD matrix is the linear combination of an ideal LD matrix and labeling error matrix, i.e.,

[TABLE]

where $\mathbf{\widetilde{D}}\in\mathbbm{R}^{n\times m}$ denotes the to be recovered ideal LD matrix, $\mathbf{E}\in\mathbbm{R}^{n\times m}$ represents the error term in the LD. Accordingly, the inaccurate LDL problem becomes an inverse problem, i.e., recovering the ideal LD matrix $\mathbf{{\widetilde{D}}}$ and the error matrix $\mathbf{E}$ from the inaccurate LD matrix $\mathbf{D}$.

To solve the inverse problem, we need to leverage the characteristics of the ideal LD matrix and error matrix. In LDL, each instance has multiple valid labels, and the label correlations exist in most multiple label learning problems. Due to label correlations, the ideal LD matrix is supposed to be low-rank. Note that the low-rankness of the LD matrix has been verified in [10]. Besides, although the given LD is not accurate, it is usually annotated by different persons with some training on annotation, we assume that only minority proportion of the LDs is inaccurate, and accordingly, the error matrix is sparse. Based on the above assumptions, the proposed ILDL problem is preliminarily formulated as

[TABLE]

where $\operatorname{rank}(\mathbf{\widetilde{D}})$ denotes the rank of the ideal LD matrix, $\operatorname{card(\mathbf{E})}$ records the number of non-zero elements in $\mathbf{E}$, and $\alpha$ is the trade-off parameter. By solving Eq. (2), the noisy LD matrix $\mathbf{D}$ will be decomposed to a low-rank ideal LD matrix $\mathbf{\widetilde{D}}$ and a sparse error term $\mathbf{E}$.

3.2 Exploiting Instances Correlations by Adaptive Graph Learning

The correlations among the instances are also important for recovering the ideal LD, i.e. if two instances are close in feature space, their ideal LDs should also be similar to each other. In order to capture the similarity relationships of the instances, we construct an adaptive graph $\mathbf{A}\in\mathbbm{R}^{n\times n}$ as:

[TABLE]

where $a_{ij}$ is the $(i,j)$-th element of $\mathbf{A}$, which represents the similarity between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$, $\mathbf{1}_{n}\in\mathbbm{R}^{1\times n}$ is an all ones vector with size $n$, and $\gamma>0$ is a trade-off parameter. The first term in Eq. (3) ensures that ${a}_{ij}$ is larger when $\mathbf{{x}}_{i}$ and $\mathbf{{x}}_{j}$ are similar to each other. The second term of Eq. (3) avoids the trivial solution, i.e., $\mathbf{A}$ becomes an identity matrix. The constraints of Eq. (3) guarantee that the similarities among instances are non-negative and the similarity matrix is normalized. After solving Eq. (3), we use the similarity relationship $\mathbf{A}$ of samples to guide the ideal LD recovery, i.e.,

[TABLE]

where $\operatorname{Tr}(\cdot)$ is the trace of a matrix. $\mathbf{L}=\mathbf{\hat{A}}+\left(\mathbf{A}+\mathbf{A}^{\top}\right)/2\in\mathbbm{R}^{n\times n}$ is the graph Laplacian matrix, $\mathbf{\hat{A}}$ is a diagonal matrix with the $(i,i)$-th element $\mathbf{\hat{A}}_{ii}=\sum_{j=1}^{n}\left[\left(a_{ij}+a_{ji}\right)/2\right]$. By minimizing Eq. (4), two instances with similar feature representations will tend to own the similar LDs.

3.3 Model Formulation

Combining the above priors, our model is formulated as:

[TABLE]

where $\beta$ is the trade-off parameter. As the rank function $\operatorname{rank}(\cdot$) and the card function $\operatorname{card}(\cdot$) are both non-convex and discrete, Eq. (5) is difficult to solve. Therefore, we relax those two terms by the associated convex surrogates, i.e. nuclear norm for $\operatorname{rank}(\cdot)$ and $\ell_{1}$ norm for $\operatorname{card}(\cdot)$, and our model finally becomes

[TABLE]

By solving Eq. (6), we can recover an ideal LD and a sparse error matrix from the noise LD. Then any LDL algorithms can be applied on $\widetilde{\mathbf{D}}$ to learn a reliable label distribution prediction model.

3.4 Making Prediction

After solving Eqs. (3) and (6), a clean LD is learned, since $\mathbf{d}_{i}$ is a real-valued quantity, multi-output support vector regression (MSVR) [19], [20] is utilized to address this scenario. In this approach, a kernel regression model is employed to parameterize the label distribution predictor:

[TABLE]

where $\bm{\Theta}=\left[\bm{\theta}_{1},\bm{\theta}_{2},\ldots,\bm{\theta}_{q}\right]$ and $\bm{b}=\left[b_{1},b_{2},\ldots,b_{q}\right]^{\top}$ signify the weight matrix and the bias vector of the regression model, respectively. As indicated in Eq. (7), the first term is responsible for regulating the complexity of the resulting model. The second term represents the hinge loss, and its specific definition is as follows: $\ell(\mathbf{(\Theta,b)}=\max(0,u_{i}-\varepsilon)$, here $u_{i}=\left\|\bm{e}_{i}\right\|=\sqrt{\bm{e}_{i}^{\top}\bm{e}_{i}}$ with $\bm{e}_{i}=\bm{\widetilde{d}}_{i}-\bm{\Theta}^{\top}\varphi\left(\bm{x}_{i}\right)-\mathbf{b}$. The hinge loss generates an insensitive zone around the estimation, determined by $\epsilon$. In other words, any loss of $u_{i}$ smaller than $\epsilon$ will be disregarded. The constraint is employed to maintain consistency between the signs of the prediction and the ideal LD matrix $\widetilde{d}_{i}^{j}$. In order to facilitate the optimization of the objective function, we relax the constraint to: $\forall i,\widetilde{d}_{i}^{j}\left(\bm{\theta}_{j}^{\top}\varphi\left(\bm{x}_{i}\right)+b_{j}\right)\geqslant 0$ = $-\sum_{i=1}^{n}\sum_{j=1}^{c}\widetilde{d}_{\bm{x}_{i}}^{j}\bm{\theta}_{j}^{\top}\bm{\phi}_{i}$ = $-\operatorname{tr}\left(\hat{\mathbf{L}}^{\top}\mathbf{\Theta\Phi}\right)$, where $\bm{\Phi}=\left[\phi_{1},\phi_{2},\ldots,\phi_{n}\right]$.

3.5 Numerical Solution of Eq. (6)

We use ADMM to solve problem (6), which is good at handling the equality constraints. First, we introduce an intermediate variable $\mathbf{Z}\in\mathbbm{R}^{n\times m}$, and rewrite Eq. (6) as :

[TABLE]

Then, the augmented Lagrangian form of Eq. (8) is:

[TABLE]

where $\mathbf{\Gamma}_{1}\!\in\!\mathbbm{R}^{n\times m}$, $\mathbf{\Gamma}_{2}\!\in\!\mathbbm{R}^{n\times m}$ denote the Lagrangian multipliers, $\mu$ is a positive penalty parameter, $\|\cdot\|_{F}^{2}$ is Frobenius norm, $\langle\cdot\rangle$ denotes the inner product of two vectors. Eq. (9) can be solved by alternately solving the following sub-problems:

$\mathbf{Z}$-Subproblem is formulated as:

[TABLE]

Eq. (10) is a nuclear norm minimization problem, with a closed-form solution, i.e, [21]

[TABLE]

where $\mathcal{J}(\cdot)$ is single value thresholding operator, which firstly performs singular value decomposition on $\mathbf{\widetilde{D}}^{k+1}+\mathbf{\Gamma}_{2}^{k}/\mu^{k}=\mathbf{U}\hat{\mathbf{\Sigma}}\mathbf{V}^{\top}$, and then the solution is given by $\mathbf{U}\hat{\mathbf{\Sigma}}\mathbf{V}^{\top}$, where $\hat{\Sigma}_{ii}=\max\left(0,\Sigma_{ii}-1/\mu\right)$. 2. 2.

$\mathbf{\widetilde{D}}$-Subproblem is formulated as:

[TABLE]

Eq. (12) can be solved by setting the first-order derivative to zero, i.e.,

[TABLE]

where $\psi_{1}=\mathbf{Z}-\mathbf{\Gamma}_{2}^{k}/\mu^{k}$, $\psi_{2}=\mathbf{D}-\mathbf{E}-\mathbf{\Gamma}_{1}^{k}/\mu^{k}$, $\mathbf{I}\in\mathbbm{R}^{n\times n}$ is an all ones matrix. 3. 3.

$\mathbf{E}$-Subproblem is represented as:

[TABLE]

Eq. (14) can be sloved by

[TABLE]

where $\delta_{\alpha/\mu}(\cdot)$ is the soft-thresholding operator [22]: $\delta_{\omega}(a)=\operatorname{sgn}(a)$ for $|a|\geq\omega$ and zero otherwise. 4. 4.

The update multipliers and penalty parameter are updated by

[TABLE]

3.6 Numerical Solution of Eq. (3)

To solve Eq. (3), we rewrite it as

[TABLE]

where $u_{ij}=\frac{\beta}{2}\left\|x_{i}-x_{j}\right\|^{2}$. Eq. (17) can be solved column-wisely, and the corresponding Lagrangian function of problem (17) regarding the $i$-th column is

[TABLE]

where $\varpi$ is a scalar and $\varrho$ is a Lagrangian coefficient vector. According to the KKT conditions [23], we have

[TABLE]

After solving the KKT conditions, we have

[TABLE]

where $\bar{\varrho}=\frac{\bm{1}^{\top}\bm{\varrho}}{n}$ and $f=\mathbf{p}-\frac{\mathbf{11}^{T}}{n}\mathbf{p}+\frac{1}{n}\mathbf{1}$, and $\bar{\varrho}$ is the root of the following equation

[TABLE]

Eq. (21) can be solved efficiently by the Newton method

[TABLE]

where $f^{\prime}$(X) represents the partial derivative of X.

3.7 Numerical Solution of Eq. (7)

To minimize the objective function, we opt for an iterative quasi-Newton method called Iterative Re-Weighted Least Square (IRWLS) [24]. Initially, the objective function is approximated by its first-order Taylor expansion at the solution of the current k-th iteration, denoted by $\mathbf{\Theta}^{(k)}$:

[TABLE]

where $\bm{e}_{i}^{(k)}$ and $u_{i}^{(k)}$ are calculated using $\mathbf{\Theta}^{(k)}$ and $\mathbf{b}^{(k)}$. Subsequently, a quadratic approximation is further constructed as:

[TABLE]

where

[TABLE]

and $\tau$ is a constant term that does not rely on either $\mathbf{\Theta}^{(k)}$ or $\mathbf{b}^{(k)}$. By combining Eq. (7) and (24), our objective function can be rewritten as:

[TABLE]

Here, $\mathbf{H}=\left[h_{ij}\right]_{n\times n}$, where $h_{ij}=\xi_{i}\delta_{ij}$, and $\delta_{ij}$ is the Kronecker’s delta function. By setting the corresponding gradient to zero:

[TABLE]

the solution is obtained as

[TABLE]

Then, the solution for the next iteration, $\mathbf{\Theta}^{(k+1)}$, is obtained using a line search algorithm with $\bm{\Theta}^{s}$ and $\bm{\Theta}^{(k)}$. Finally, after normalizing the prediction results, we obtain the predicted label distribution. In addition, our method can also cooperate with any LDL (Label Distribution Learning) algorithm. The overall algorithm flowchart is shown in Algorithm 1.

4 Experimental Results and Analyses

4.1 Datasets

In this section, we present the datasets employed in our experiments to assess the performance of our proposed method. A total of 15 datasets are utilized, and their details are provided in Table 1. The datasets encompass a broad spectrum of domains, such as biology, film, facial expression analysis, images from social media platforms, facial beauty perception, and natural scene classification. This diverse assortment of datasets enables us to evaluate the adaptability of our proposed method across various application contexts.

The first, third, and fourth datasets, M2B [25] , SCUT-FBP [26], and fbp5500 [27], focus on facial beauty perception. For M2B and SCUT-FBP, the features and label distributions are processed according to [28]. For fbp5500, we utilize the ResNet [29] trained by the authors to extract 512-dimensional features.

The second datasets, RAF-ML, pertains to facial expression recognition, with each image characterized by a 2000-dimensional DBM-CNN feature and a 6-dimensional expression distribution [30]. To reduce the feature dimensionality, we apply principal component analysis (PCA), resulting in 200-dimensional features.

The sixth and seventh datasets, flickr-ldl and twitter-ldl datasets [31]. These datasets comprise 10,045 and 10,700 images, respectively, annotated with 8 prevalent emotions. Both logical labels and label distributions are provided for these datasets. Image features are extracted utilizing VGGNet and subsequently dimensionality-reduced to 200 using PCA.

Seventh and Eighth datasets : These datasets are related to yeast, focusing on the budding yeast Saccharomyces cerevisiae. Each dataset represents the results of distinct biological experiments, involving a total of 2,465 yeast genes described by a phylogenetic profile vector with 24 features. The expression level of each gene at different time points is represented by the corresponding label’s normalized description degree.

The ninth and eleventh datasets are s-JAFFE and SBU-3DFE, are extended versions of widely-used facial expression databases, JAFFE [32] and BU 3DFE [33], respectively. SJAFFE contains 213 grayscale images with 243-dimensional LBP features [34]. Each image is scored by 60 individuals on six basic emotions, and the normalized average scores create the label distribution. Similarly, SBU 3DFE consists of 2,500 images scored by 23 individuals, resulting in a label distribution version of the dataset.

The tenth dataset is a movie genre dataset contains information about various movies and their associated genres. Features are extracted from movie metadata, such as cast, director, plot, and release year, resulting in a multi-dimensional feature vector for each movie. The label distribution is determined by calculating the proportions of each genre associated with the movie.

The last dataset is natural scene dataset, which contains 2,000 images with inconsistent multi-label rankings. Ten human annotators ranked the images using nine possible labels. A non-linear programming process transformed the inconsistent rankings into label distributions, and a 294-dimensional feature vector was extracted for each image.

4.2 Evaluation Metrics

In this study, we employ a combination of six metrics to evaluate the performance of the LDL algorithms. These metrics comprise five distance-based measures and one similarity-based measure, as follows:

Chebyshev$\downarrow$ Clark$\downarrow$ Kullback-Leibler (KL)$\downarrow$ Canberra$\downarrow$ S$\phi$ren$\downarrow$ Cosine$\uparrow$ intersection$\uparrow$. The formulas for these metrics are provided in Table 2. In these formulas, $\bm{d}$ represents the actual label distribution, and $\hat{\bm{d}}$ represents the predicted label distribution for the i-th element. Lower values indicate better performance for distance-based measures, while higher values signify better performance for similarity-based measures.

4.3 Inaccurate LD Matrix Generation

To simulate the inaccurate LD, we added a controlled Gaussian noise on the ground-truth LD matrix. Specifically, we used the Matlab function randn() to generate a random matrix of the same size as the ground-truth LD, and multiplied the generated random matrix by the variance (b) and added the mean (a) to construct the label error matrix. Then we added the error matrix to the LD matrix, and normalized the summarization as the noisy LD matrix.

4.4 Comparative Studies

4.4.1 Comparison with sota LDL algorithms

We compare our approach with seven state-of-the-art label distribution learning approaches, using parameter configurations suggested in their respective literature:

• •

AA-BP [4]: AA-BP is a structure with a three-layer network. The network outputs different units, and each output unit represents the descriptive degree of the label.

• •

AA-KNN [4]: For each new instance $\mathbf{x}$in AA-KNN, first find its k nearest neighbors in the training set. Then, calculate the mean of the label distribution of all k nearest neighbors as the label distribution of $\mathbf{x}$.

• •

PT-Bayes [4]: PT-Bayes transforms the LDL problem into a single-label learning problem, effectively converting the training samples into a set of weighted single-label samples. PT-Bayes then utilizes the Bayes classifier to address the transformed weighted single-label learning problem.

• •

LCLR [35]: LCLR reconstructs a new supervised label distribution with global and local label-related information. [$\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$, and K are set to 0.0001, 0.001, 0.001, 0.001, and 4, respectively.]

• •

LDLSF [36]: LDLSF uses label-specific features to improve label distribution learning performance. [M are diagonal matrices in which all diagonal elements are 0.5, $\rho$ is set as $10^{-3}$]

• •

LDLLC [37]: LDLLC utilizes local label correlation to make prediction distributions between similar instances as close as possible.

• •

CPNN [12]: Conditional Probability Neural Network, employs a three-layer neural network structure to learn the distribution of labels.

• •

LDSVR [38]: LDSVR is to simultaneously fit a sigmoid function to each component of the label distribution using a multi-output support vector machine.

For our approach, trade-off parameters $\alpha$ and $\beta$ are set as 0.05 and 0.05 in the recover part. In the predection part, trade-off parameters $\kappa$ and $\nu$ are set as 1 and 0.1. Table 3 and Table 5 present detailed experimental results comparing the algorithms using each evaluation metric. To analyze and statistically compare the performance differences between algorithms, we employ the Friedman test [39], which is a widely accepted statistical test for multiple algorithms and a specific number of datasets. For each evaluation metric, the average rank of the j-th algorithm is calculated as $R_{j}=\frac{1}{N}\sum_{i=1}^{N}r_{i}^{j}$, where $r_{i}^{j}$ represents the rank of the j-th algorithm on the i-th dataset. Subsequently, the Friedman statistics $F_{F}$, distributed according to the F-distribution with $(K-1)$ numerator degrees of freedom and $(K-1)(N-1)$ denominator degrees of freedom, are computed as follows:

[TABLE]

Table 4 summarizes the Friedman statistics $F_{F}$ for each evaluation metric and the corresponding critical value at significance level $\alpha=0.05$. As shown in Table 4, the indicators of all evaluation methods exceed the critical value, i.e., the hypothesis that all algorithms perform the same is rejected, indicating that the performance of the algorithms is significantly different.

To further distinguish the performance among the comparing algorithms, a post-hoc test is necessary at this stage. We employ the Bonferroni-Dunn test [40]. LSag is treated as the control algorithm, and the difference between the average ranks of IDI-LDL and one comparing algorithm is compared with the critical difference (CD). If their difference is larger than one CD (CD=2.994 with K = 9 and N = 12 at a significance level of $\alpha=0.05$), the performance of LSag is deemed to be significantly different from that of the comparing algorithm.

Fig. 5 presents the CD diagrams [39] for each evaluation metric. In each sub-figure, the average rank of each comparing algorithm is marked along the axis with lower ranks to the right, and a thick line connects LSag and any comparing algorithm if the difference between their average ranks is less than one CD. Additionally, the average ranking of the compared algorithms across 12 datasets is shown in Table 6. Based on the above results, observations can be made as follows:

• •

In 99.21% of the cases, our algorithm achieved the best results. This can be attributed to the fact that traditional algorithms such as AA-BP, AA-KNN, LDSVR, CPNN, and PT-Bayes do not consider the presence of noise in the label distribution, which can lead to incorrect guidance for classifier learning.

• •

On the other hand, although LDLLC, LDL-SF, and LCLR take label correlation into account, they also fail to address the noise issue in the label distribution. As a result, the models they learn underperform.

• •

Our algorithm consistently achieves the highest average ranking, as it takes into account the noise present in the label distribution, while other methods do not.

• •

Under the Chebyshev$\downarrow$ metric, our algorithm is significantly better than all other algorithms, except for LCLR. Similar results can be observed under the Canberra$\downarrow$ and S$\phi$ren$\downarrow$ metrics. For the Clark$\downarrow$ metric, our algorithm outperforms all algorithms except for AA-KNN, and this performance is also replicated under the Kullback-Leibler (KL)$\downarrow$ and Cosine$\uparrow$ metrics. Under the intersection$\uparrow$ metric, our algorithm is significantly better than LCLR, AA-BP, AA-KNN, CPNN and LDSVR.

4.5 Further Analyses

4.5.1 Parameter sensitivity analysis

: The impact of different hyper-parameters $\alpha$ and $\beta$ in Eq. (6) on the prediction of experimental results is shown in Figure 2. As depicted in Figure 3, the trade-off parameters $\alpha$ and $\beta$, which control the strength of error and preserving the instance correlation topological structure, respectively, do indeed influence the performance of our method. However, the proposed model is quite robust to the those two hyper-parameters, i.e., the Chebyshev distance and S$\phi$rensen distance are relatively stable as the parameter value changes within a reasonable range, which serves as a desirable property in using the proposed approach. Additionally, the impact of different parameters $\kappa$ and $\nu$ on the prediction results is illustrated in Figure 3. As shown in Figure 3, parameters $\kappa$ and $\nu$, which control the strength of the term between the error of the predicted results and the recovered LD, and the term controlling the alignment between the predicted LD and the ideal LD, respectively, do indeed affect the performance of our method. However, our method still remains stable within a certain range.

4.5.2 Visualizing Experimental Results

To better understand our algorithm, we have visualized a portion of the prediction results, as shown in Figure 4. The first column displays the representative Label Distribution (LD) samples from M2B, RAF-ML, Flickr, and Nature-Scene datasets. The second column to the last columns present the predicted LDs using different algorithms. In each figure, the x-axis represents various labels, and the y-axis indicates the descriptiveness of the corresponding labels. From Figure 4, we have the following observations:

• •

When training with Inaccurate Label Distributions (ILDs), algorithms such as AA-BP, CPNN, LSVR, and LDLSF struggle to accurately predict the Label Distribution (LD) for unseen instances. Specifically, they fail to capture the descriptiveness of each individual label and the relative importance ranking among the label descriptiveness.

• •

AA-KNN, LCLR, LDLLC, and PT-Bayes can capture the relative magnitudes between different labels in their predictions; however, the descriptiveness of each individual label is still inaccurate. This is because these algorithms do not consider the noise present in the Label Distribution (LD) while learning the model.

• •

Our algorithm not only accurately predicts the Label Distribution (LD) for each instance but also effectively predicts the ranking of descriptiveness corresponding to different labels. This is because we consider the noise present in the label distribution before learning the classification model.

4.6 Collaboration with ther LDL Algorithms

In this section, we discuss the scalability of our method, specifically whether the performance of different LDL algorithms can be improved when facing inaccurate label distributions by recovering the ideal LD through the recovery model. The experimental setup is as follows. Prior to training, we use our recovery model (Eq .(6)) to recover the ideal LD. Then, we use the recovered LD for training and finally test on the real data. Note that this setup is consistent with the previous settings. The experimental results are presented in Table 7. Here, we use 4 datasets to validate whether the recovered LD can help other LDL algorithms improve their performance when faced with ILD. Here, we used two human face datasets, SJAFFE and SBU-3DFE, as well as two yeast datasets, Yeast-alpha and Yeast-cdc. As shown in Table 7, $\mathcal{F}$(I) denotes the performance an LDL algorithm trained on the noise LD, and $\mathcal{F}$-LSag indicates the performance of that LDL algorithm trained on the recovered LD by our approach. We also show the performance of different LDL algorithms trained on the ground-truth label distribution (i.e., $\mathcal{F}$(G)), which can be regarded as the performance upper bound. To analyze whether there are statistical performance gaps among $\mathcal{F}$(I) and $\mathcal{F}$-LSag, Wilcoxon signed-rank test [41], which is a widely-accepted statistical test for comparisons of two algorithms over several datasets, is employed. Table 8 summarizes the statistical test results and the p-values. Based on the above results, observations can be made as follows:

• •

Noisy LD can cause a significant degradation in the performance for different LDL algorithms, so it is necessary to address the issue of learning with ILD.

• •

$\mathcal{F}$-LSag is statistically superior to the $\mathcal{F}$ in all cases (4 datasets and four metrics) , suggesting the effectiveness of our approach. This is because accurate supervision information can guide more precise model training.

• •

The performance of $\mathcal{F}$-LSag is quite close to the $\mathcal{F}$(I) on different LDL algorithms, indicating our approach can recover high-quality LD from the noisy LD.

5 Conclusion

This paper investigates the problem of inaccurate label distribution learning for the first time. To be specific, we treat the noisy LD matrix as the liner combination of an ideal LD matrix and an error label matrix, and separates them by a novel adaptive graph-regularized low-rank and sparse decomposition model. Then, we use ADMM to efficiently optimize the proposed model. The recovered LD are taken into consideration when inducing a LD predictive model for LDL, achieved through the utilization of a specialized objective function. Extensive experiments demonstrate that our method can effectively address the ILDL problem.