Explaining Black-box Language Models with Knowledge Probing Systems: A Post-hoc Explanation Perspective

TL;DR

This paper introduces KnowProb, a post-hoc probing system that explains black-box language models by assessing their implicit knowledge understanding beyond surface content, revealing their limitations and aiding interpretability.

Contribution

The paper proposes a novel knowledge-guided probing method, KnowProb, to evaluate and explain the implicit knowledge in black-box language models from multiple perspectives.

Findings

Current PLMs primarily learn a single distribution of representations.

PLMs face challenges in capturing hidden knowledge behind text.

KnowProb effectively identifies limitations of black-box models.

Abstract

Pre-trained Language Models (PLMs) are trained on large amounts of unlabeled data, yet they exhibit remarkable reasoning skills. However, the trustworthiness challenges posed by these black-box models have become increasingly evident in recent years. To alleviate this problem, this paper proposes a novel Knowledge-guided Probing approach called KnowProb in a post-hoc explanation way, which aims to probe whether black-box PLMs understand implicit knowledge beyond the given text, rather than focusing only on the surface level content of the text. We provide six potential explanations derived from the underlying content of the given text, including three knowledge-based understanding and three association-based reasoning. In experiments, we validate that current small-scale (or large-scale) PLMs only learn a single distribution of representation, and still face significant challenges in capturing the hidden knowledge behind a given text. Furthermore, we demonstrate that our proposed approach is effective for identifying the limitations of existing black-box models from multiple probing perspectives, which facilitates researchers to promote the study of detecting black-box models in an explainable way.