A Survey on Interpretable Cross-modal Reasoning

TL;DR

This survey reviews the current state of interpretable cross-modal reasoning, highlighting methods, datasets, challenges, and future directions to enhance transparency and understanding in AI systems across different modalities.

Contribution

It provides a comprehensive taxonomy of I-CMR methods, reviews datasets with explanations, and discusses challenges and future research opportunities.

Findings

Three-level taxonomy of I-CMR methods

Compilation of datasets with explanation annotations

Identification of key challenges and future directions

Abstract

In recent years, cross-modal reasoning (CMR), the process of understanding and reasoning across different modalities, has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics. As the deployment of AI systems becomes more ubiquitous, the demand for transparency and comprehensibility in these systems' decision-making processes has intensified. This survey delves into the realm of interpretable cross-modal reasoning (I-CMR), where the objective is not only to achieve high predictive performance but also to provide human-understandable explanations for the results. This survey presents a comprehensive overview of the typical methods with a three-level taxonomy for I-CMR. Furthermore, this survey reviews the existing CMR datasets with annotations for explanations. Finally, this survey summarizes the challenges for I-CMR and discusses potential future directions. In conclusion, this survey aims to catalyze the progress of this emerging research area by providing researchers with a panoramic and comprehensive perspective, illuminating the state of the art and discerning the opportunities. The summarized methods, datasets, and other resources are available at https://github.com/ZuyiZhou/Awesome-Interpretable-Cross-modal-Reasoning.