A new approach for fine-tuning sentence transformers for intent classification and out-of-scope detection tasks

TL;DR

This paper introduces a novel fine-tuning method for sentence transformers that improves out-of-scope detection in virtual assistants by combining intent classification with embedding reconstruction, leading to better OOS rejection without harming classification accuracy.

Contribution

The paper proposes a regularization technique using auto-encoder based embedding reconstruction to enhance OOS detection in sentence transformers.

Findings

Achieved 1-4% improvement in OOS rejection performance.

Maintained comparable intent classification accuracy.

Enhanced embedding space separation for in-scope and OOS queries.

Abstract

In virtual assistant (VA) systems it is important to reject or redirect user queries that fall outside the scope of the system. One of the most accurate approaches for out-of-scope (OOS) rejection is to combine it with the task of intent classification on in-scope queries, and to use methods based on the similarity of embeddings produced by transformer-based sentence encoders. Typically, such encoders are fine-tuned for the intent-classification task, using cross-entropy loss. Recent work has shown that while this produces suitable embeddings for the intent-classification task, it also tends to disperse in-scope embeddings over the full sentence embedding space. This causes the in-scope embeddings to potentially overlap with OOS embeddings, thereby making OOS rejection difficult. This is compounded when OOS data is unknown. To mitigate this issue our work proposes to regularize the cross-entropy loss with an in-scope embedding reconstruction loss learned using an auto-encoder. Our method achieves a 1-4% improvement in the area under the precision-recall curve for rejecting out-of-sample (OOS) instances, without compromising intent classification performance.