Generative AI Systems: A Systems-based Perspective on Generative AI

TL;DR

This paper explores the design, building, and understanding of multimodal Generative AI Systems (GenAISys) from a systems perspective, emphasizing new research directions and cross-disciplinary approaches.

Contribution

It introduces a systems-based framework for GenAISys, highlighting design principles, training methods, and open research questions for multimodal generative AI.

Findings

Proposes a systems perspective for GenAISys design

Identifies key challenges in reliability and verifiability

Suggests cross-disciplinary approaches for understanding GenAI systems

Abstract

Large Language Models (LLMs) have revolutionized AI systems by enabling communication with machines using natural language. Recent developments in Generative AI (GenAI) like Vision-Language Models (GPT-4V) and Gemini have shown great promise in using LLMs as multimodal systems. This new research line results in building Generative AI systems, GenAISys for short, that are capable of multimodal processing and content creation, as well as decision-making. GenAISys use natural language as a communication means and modality encoders as I/O interfaces for processing various data sources. They are also equipped with databases and external specialized tools, communicating with the system through a module for information retrieval and storage. This paper aims to explore and state new research directions in Generative AI Systems, including how to design GenAISys (compositionality, reliability, verifiability), build and train them, and what can be learned from the system-based perspective. Cross-disciplinary approaches are needed to answer open questions about the inner workings of GenAI systems.