Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation

TL;DR

This paper introduces a new dataset to evaluate foundation models on out-of-distribution reasoning tasks across multiple modalities, highlighting the importance of exceptional case benchmarking and proposing prompt engineering techniques to improve performance.

Contribution

It is the first study to create and validate a dataset for assessing foundation models on exceptional, out-of-distribution scenarios across diverse modalities and tasks.

Findings

Prompt engineering techniques like CoT improve model performance.

Validation shows enhanced reasoning abilities with proposed methods.

Dataset enables comprehensive evaluation of FMs in exceptional cases.

Abstract

Foundation models (FMs) have achieved significant success across various tasks, leading to research on benchmarks for reasoning abilities. However, there is a lack of studies on FMs performance in exceptional scenarios, which we define as out-of-distribution (OOD) reasoning tasks. This paper is the first to address these cases, developing a novel dataset for evaluation of FMs across multiple modalities, including graphic novels, calligraphy, news articles, and lyrics. It includes tasks for instance classification, character recognition, token prediction, and text generation. The paper also proposes prompt engineering techniques like Chain-of-Thought (CoT) and CoT+Few-Shot to enhance performance. Validation of FMs using various methods revealed improvements. The code repository is accessible at: https://github.com/MLAI-Yonsei/ExceptionalBenchmark