Multi-modal Machine Learning in Engineering Design: A Review and Future Directions

TL;DR

This paper reviews the current state, challenges, and future directions of multi-modal machine learning in engineering design, emphasizing applications, data needs, and model development to advance intelligent design tools.

Contribution

It provides a comprehensive overview of MMML concepts, applications in engineering design, and outlines future research directions including dataset creation and model scalability.

Findings

MMML enables cross-modal synthesis and retrieval in engineering design.

Challenges include data scarcity and model interpretability.

Future efforts should focus on dataset development and scalable models.

Abstract

In the rapidly advancing field of multi-modal machine learning (MMML), the convergence of multiple data modalities has the potential to reshape various applications. This paper presents a comprehensive overview of the current state, advancements, and challenges of MMML within the sphere of engineering design. The review begins with a deep dive into five fundamental concepts of MMML:multi-modal information representation, fusion, alignment, translation, and co-learning. Following this, we explore the cutting-edge applications of MMML, placing a particular emphasis on tasks pertinent to engineering design, such as cross-modal synthesis, multi-modal prediction, and cross-modal information retrieval. Through this comprehensive overview, we highlight the inherent challenges in adopting MMML in engineering design, and proffer potential directions for future research. To spur on the continued evolution of MMML in engineering design, we advocate for concentrated efforts to construct extensive multi-modal design datasets, develop effective data-driven MMML techniques tailored to design applications, and enhance the scalability and interpretability of MMML models. MMML models, as the next generation of intelligent design tools, hold a promising future to impact how products are designed.