Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting

TL;DR

This paper introduces a multimodal early exit model for document image classification that balances high accuracy with reduced latency, making it suitable for scalable, real-world VDU applications.

Contribution

It presents the first multimodal early exit design for VDU, combining training strategies and exit layer placements to optimize performance and efficiency.

Findings

Over 20% latency reduction while maintaining accuracy

Improved performance-efficiency trade-off over traditional methods

Calibration enhances confidence scores for early exiting

Abstract

This work addresses the need for a balanced approach between performance and efficiency in scalable production environments for visually-rich document understanding (VDU) tasks. Currently, there is a reliance on large document foundation models that offer advanced capabilities but come with a heavy computational burden. In this paper, we propose a multimodal early exit (EE) model design that incorporates various training strategies, exit layer types and placements. Our goal is to achieve a Pareto-optimal balance between predictive performance and efficiency for multimodal document image classification. Through a comprehensive set of experiments, we compare our approach with traditional exit policies and showcase an improved performance-efficiency trade-off. Our multimodal EE design preserves the model's predictive capabilities, enhancing both speed and latency. This is achieved through a reduction of over 20% in latency, while fully retaining the baseline accuracy. This research represents the first exploration of multimodal EE design within the VDU community, highlighting as well the effectiveness of calibration in improving confidence scores for exiting at different layers. Overall, our findings contribute to practical VDU applications by enhancing both performance and efficiency.