Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems

TL;DR

This paper explores the integration of multi-modal multi-task learning with hardware co-design to optimize autonomous systems' performance and power efficiency, addressing unique challenges and proposing a differentiable optimization framework.

Contribution

It introduces a co-design framework for MMMT models and heterogeneous hardware, formulated as a differentiable optimization problem to enhance performance and reduce power consumption.

Findings

Formulated MMMT and hardware co-design as a differentiable optimization problem.

Highlighted the importance of co-design for power-limited autonomous systems.

Discussed opportunities and challenges in applying MMMT in autonomous systems.

Abstract

Optimizing the quality of result (QoR) and the quality of service (QoS) of AI-empowered autonomous systems simultaneously is very challenging. First, there are multiple input sources, e.g., multi-modal data from different sensors, requiring diverse data preprocessing, sensor fusion, and feature aggregation. Second, there are multiple tasks that require various AI models to run simultaneously, e.g., perception, localization, and control. Third, the computing and control system is heterogeneous, composed of hardware components with varied features, such as embedded CPUs, GPUs, FPGAs, and dedicated accelerators. Therefore, autonomous systems essentially require multi-modal multi-task (MMMT) learning which must be aware of hardware performance and implementation strategies. While MMMT learning has been attracting intensive research interests, its applications in autonomous systems are still underexplored. In this paper, we first discuss the opportunities of applying MMMT techniques in autonomous systems and then discuss the unique challenges that must be solved. In addition, we discuss the necessity and opportunities of MMMT model and hardware co-design, which is critical for autonomous systems especially with power/resource-limited or heterogeneous platforms. We formulate the MMMT model and heterogeneous hardware implementation co-design as a differentiable optimization problem, with the objective of improving the solution quality and reducing the overall power consumption and critical path latency. We advocate for further explorations of MMMT in autonomous systems and software/hardware co-design solutions.