Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks

TL;DR

This paper introduces an Explicit Logic Channel to validate, select, and enhance Multimodal Large Language Models in zero-shot visual-language tasks, improving trustworthiness and explainability.

Contribution

The paper proposes a novel Explicit Logic Channel that mimics human reasoning, enabling validation and enhancement of MLLMs without ground-truth labels.

Findings

Explicit Logic Channel improves model validation and selection.

Cross-channel integration enhances zero-shot task performance.

Proposed methods increase trustworthiness and explainability of MLLMs.

Abstract

Frontier Multimodal Large Language Models (MLLMs) exhibit remarkable capabilities in Visual-Language Comprehension (VLC) tasks. However, they are often deployed as zero-shot solution to new tasks in a black-box manner. Validating and understanding the behavior of these models become important for application to new task. We propose an Explicit Logic Channel, in parallel with the black-box model channel, to perform explicit logical reasoning for model validation, selection and enhancement. The frontier MLLM, encapsulating latent vision-language knowledge, can be considered as an Implicit Logic Channel. The proposed Explicit Logic Channel, mimicking human logical reasoning, incorporates a LLM, a VFM, and logical reasoning with probabilistic inference for factual, counterfactual, and relational reasoning over the explicit visual evidence. A Consistency Rate (CR) is proposed for cross-channel validation and model selection, even without ground-truth annotations. Additionally, cross-channel integration further improves performance in zero-shot tasks over MLLMs, grounded with explicit visual evidence to enhance trustworthiness. Comprehensive experiments conducted for two representative VLC tasks, i.e., MC-VQA and HC-REC, on three challenging benchmarks, with 11 recent open-source MLLMs from 4 frontier families. Our systematic evaluations demonstrate the effectiveness of proposed ELC and CR for model validation, selection and improvement on MLLMs with enhanced explainability and trustworthiness.