Does My Representation Capture X? Probe-Ably

TL;DR

Probe-Ably is an extendable framework that automates and supports reliable probing methodologies to investigate neural model representations, addressing the complexity and evolving best practices in probing techniques.

Contribution

It introduces Probe-Ably, a flexible tool that simplifies conducting probing experiments following recent best practices for analyzing neural representations.

Findings

Supports multiple probing methods

Automates best-practice protocols

Facilitates reliable analysis of neural features

Abstract

Probing (or diagnostic classification) has become a popular strategy for investigating whether a given set of intermediate features is present in the representations of neural models. Probing studies may have misleading results, but various recent works have suggested more reliable methodologies that compensate for the possible pitfalls of probing. However, these best practices are numerous and fast-evolving. To simplify the process of running a set of probing experiments in line with suggested methodologies, we introduce Probe-Ably: an extendable probing framework which supports and automates the application of probing methods to the user's inputs.