Accuracy on In-Domain Samples Matters When Building Out-of-Domain detectors: A Reply to Marek et al. (2021)

TL;DR

This paper emphasizes that high accuracy on in-domain samples is crucial for building effective out-of-domain detectors, highlighting that stronger in-domain classifiers improve out-of-domain detection robustness.

Contribution

It provides insights into the importance of in-domain classifier strength for out-of-domain detection and discusses experimental observations that can guide future research.

Findings

Stronger IND classifiers enhance OOD detection.

Discrepancies in results may stem from classifier robustness.

Experimental observations clarify factors affecting OOD detector performance.

Abstract

We have noticed that Marek et al. (2021) try to re-implement our paper Zheng et al. (2020a) in their work "OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation". Our paper proposes a model to generate pseudo OOD samples that are akin to IN-Domain (IND) input utterances. These pseudo OOD samples can be used to improve the OOD detection performance by optimizing an entropy regularization term when building the IND classifier. Marek et al. (2021) report a large gap between their re-implemented results and ours on the CLINC150 dataset (Larson et al., 2019). This paper discusses some key observations that may have led to such a large gap. Most of these observations originate from our experiments because Marek et al. (2021) have not released their codes1. One of the most important observations is that stronger IND classifiers usually exhibit a more robust ability to detect OOD samples. We hope these observations help other researchers, including Marek et al. (2021), to develop better OOD detectors in their applications.