TL;DR
torchdistill is an open-source, modular framework built on PyTorch that simplifies the design, reproduction, and experimentation of knowledge distillation methods through declarative configuration files, promoting reproducibility and extensibility.
Contribution
It introduces a highly generalized, maintainable framework for knowledge distillation that supports diverse methods, datasets, and experiments with declarative configurations.
Findings
Successfully reproduced state-of-the-art results on ImageNet and COCO datasets.
Demonstrated efficient training strategies and flexible experiment design.
Provided publicly available source code and models for community use.
Abstract
While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work. Such frameworks, however, are usually neither well generalized nor maintained, thus researchers are still required to write a lot of code to refactor/build on the frameworks for introducing new methods, models, datasets and designing experiments. In this paper, we present our developed open-source framework built on PyTorch and dedicated for knowledge distillation studies. The framework is designed to enable users to design…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsKnowledge Distillation
