A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models

TL;DR

This survey reviews the co-design of hardware and software tailored for large language models, highlighting challenges, innovations, and future directions to optimize their development and deployment.

Contribution

It provides a comprehensive overview of system-level optimization strategies and co-design approaches specifically for large language models, guiding future research.

Findings

Analysis of hardware and software challenges in LLM deployment

Summary of existing co-design approaches for LLMs

Identification of future research directions in LLM system optimization

Abstract

The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.