Spectral Probing

TL;DR

This paper introduces a learnable spectral probing method that analyzes linguistic information at various timescales and levels, providing detailed, efficient, and multilingual insights into contextualized embeddings for NLP tasks.

Contribution

It develops a fully learnable frequency filter for spectral analysis, surpassing manual filters in granularity and efficiency, and demonstrates its effectiveness across multiple languages and NLP tasks.

Findings

Spectral profiles reveal cross-task similarities and differences.

Profiles are consistent across languages, indicating robustness.

Spectral probing outperforms manual filters in analysis depth.

Abstract

Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and communicative levels, such as syntax and semantics. Contextualized embeddings have analogously been found to capture these phenomena at distinctive layers and frequencies. Leveraging these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granular analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we investigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a linguistically intuitive manner, while remaining consistent across languages-highlighting their potential as robust, lightweight task descriptors.