Distributionally Robust Recurrent Decoders with Random Network Distillation

TL;DR

This paper introduces a method that enhances language models' robustness to out-of-distribution data by detecting and disregarding OOD context during inference, thereby improving performance under distribution shifts.

Contribution

It presents a novel approach combining OOD detection with Random Network Distillation to make autoregressive models more robust to distribution shifts.

Findings

Improved language modeling performance on multiple datasets.

Effective detection and handling of out-of-distribution contexts.

Smooth transition to less expressive models for OOD data.

Abstract

Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to "shortcut learning": relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.