Local Dense Logit Relations for Enhanced Knowledge Distillation

TL;DR

This paper introduces a novel knowledge distillation method called Local Dense Relational Logit Distillation (LDRLD) that captures fine-grained inter-class relationships and adaptively emphasizes critical class pairs to improve student model performance.

Contribution

The paper proposes LDRLD, a new method for logit distillation that models detailed class relationships and uses adaptive weighting strategies for enhanced knowledge transfer.

Findings

LDRLD outperforms existing logit distillation methods on CIFAR-100, ImageNet-1K, and Tiny-ImageNet.

The adaptive decay weight strategy effectively emphasizes important class relationships.

Recursive decoupling improves the clarity and detail of transferred knowledge.

Abstract

State-of-the-art logit distillation methods exhibit versatility, simplicity, and efficiency. Despite the advances, existing studies have yet to delve thoroughly into fine-grained relationships within logit knowledge. In this paper, we propose Local Dense Relational Logit Distillation (LDRLD), a novel method that captures inter-class relationships through recursively decoupling and recombining logit information, thereby providing more detailed and clearer insights for student learning. To further optimize the performance, we introduce an Adaptive Decay Weight (ADW) strategy, which can dynamically adjust the weights for critical category pairs using Inverse Rank Weighting (IRW) and Exponential Rank Decay (ERD). Specifically, IRW assigns weights inversely proportional to the rank differences between pairs, while ERD adaptively controls weight decay based on total ranking scores of category pairs. Furthermore, after the recursive decoupling, we distill the remaining non-target knowledge to ensure knowledge completeness and enhance performance. Ultimately, our method improves the student's performance by transferring fine-grained knowledge and emphasizing the most critical relationships. Extensive experiments on datasets such as CIFAR-100, ImageNet-1K, and Tiny-ImageNet demonstrate that our method compares favorably with state-of-the-art logit-based distillation approaches. The code will be made publicly available.