A Theoretical Survey on Foundation Models

TL;DR

This paper reviews interpretable methods rooted in machine learning theory that aim to understand the inner workings of foundation models more accurately and resource-efficiently than traditional explainability techniques.

Contribution

It provides a comprehensive survey of interpretable methods for foundation models, emphasizing theoretical foundations and practical insights into their mechanisms and behaviors.

Findings

Identifies limitations of post-hoc explainability methods.

Highlights the importance of resource-light, accurate interpretability techniques.

Suggests future research directions for understanding foundation models.

Abstract

Understanding the inner mechanisms of black-box foundation models (FMs) is essential yet challenging in artificial intelligence and its applications. Over the last decade, the long-running focus has been on their explainability, leading to the development of post-hoc explainable methods to rationalize the specific decisions already made by black-box FMs. However, these explainable methods have certain limitations in terms of faithfulness and resource requirement. Consequently, a new class of interpretable methods should be considered to unveil the underlying mechanisms of FMs in an accurate, comprehensive, heuristic, and resource-light way. This survey aims to review those interpretable methods that comply with the aforementioned principles and have been successfully applied to FMs. These methods are deeply rooted in machine learning theory, covering the analysis of generalization performance, expressive capability, and dynamic behavior. They provide a thorough interpretation of the entire workflow of FMs, ranging from the inference capability and training dynamics to their ethical implications. Ultimately, drawing upon these interpretations, this review identifies the next frontier research directions for FMs.