Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning
Xingwei He, Yeyun Gong, A-Long Jin, Weizhen Qi, Hang Zhang, Jian Jiao,, Bartuer Zhou, Biao Cheng, SM Yiu, Nan Duan

TL;DR
This paper introduces a metric-guided distillation approach to improve the relevance scoring of retrievers and rankers in commonsense generation, leading to state-of-the-art results on the CommonGen benchmark.
Contribution
It proposes a novel metric distillation rule to transfer knowledge from evaluation metrics to rankers and retrievers, enhancing relevance assessment in commonsense generation.
Findings
Achieved new state-of-the-art results on CommonGen.
Distilled ranker improves relevance scoring accuracy.
Distilled retriever surpasses previous SOTA performance.
Abstract
Commonsense generation aims to generate a realistic sentence describing a daily scene under the given concepts, which is very challenging, since it requires models to have relational reasoning and compositional generalization capabilities. Previous work focuses on retrieving prototype sentences for the provided concepts to assist generation. They first use a sparse retriever to retrieve candidate sentences, then re-rank the candidates with a ranker. However, the candidates returned by their ranker may not be the most relevant sentences, since the ranker treats all candidates equally without considering their relevance to the reference sentences of the given concepts. Another problem is that re-ranking is very expensive, but only using retrievers will seriously degrade the performance of their generation models. To solve these problems, we propose the metric distillation rule to distill…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Text and Document Classification Technologies
