EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark

TL;DR

EEE-Bench is a comprehensive multimodal benchmark designed to evaluate large language and multimodal models' abilities to solve complex, real-world electrical and electronics engineering problems involving intricate visual and textual information.

Contribution

The paper introduces EEE-Bench, a new multimodal benchmark with 2860 engineering problems, and provides extensive evaluation revealing current models' limitations in practical engineering tasks.

Findings

Current models perform poorly on EEE-Bench, with accuracy from 19.48% to 46.78%.

Models tend to rely on text and overlook visual information, revealing a 'laziness' shortcoming.

EEE-Bench exposes critical gaps in models' ability to handle complex, multimodal engineering problems.

Abstract

Recent studies on large language models (LLMs) and large multimodal models (LMMs) have demonstrated promising skills in various domains including science and mathematics. However, their capability in more challenging and real-world related scenarios like engineering has not been systematically studied. To bridge this gap, we propose EEE-Bench, a multimodal benchmark aimed at assessing LMMs' capabilities in solving practical engineering tasks, using electrical and electronics engineering (EEE) as the testbed. Our benchmark consists of 2860 carefully curated problems spanning 10 essential subdomains such as analog circuits, control systems, etc. Compared to benchmarks in other domains, engineering problems are intrinsically 1) more visually complex and versatile and 2) less deterministic in solutions. Successful solutions to these problems often demand more-than-usual rigorous integration of visual and textual information as models need to understand intricate images like abstract circuits and system diagrams while taking professional instructions, making them excellent candidates for LMM evaluations. Alongside EEE-Bench, we provide extensive quantitative evaluations and fine-grained analysis of 17 widely-used open and closed-sourced LLMs and LMMs. Our results demonstrate notable deficiencies of current foundation models in EEE, with an average performance ranging from 19.48% to 46.78%. Finally, we reveal and explore a critical shortcoming in LMMs which we term laziness: the tendency to take shortcuts by relying on the text while overlooking the visual context when reasoning for technical image problems. In summary, we believe EEE-Bench not only reveals some noteworthy limitations of LMMs but also provides a valuable resource for advancing research on their application in practical engineering tasks, driving future improvements in their capability to handle complex, real-world scenarios.