VTutor: An Open-Source SDK for Generative AI-Powered Animated Pedagogical Agents with Multi-Media Output

TL;DR

VTutor is an open-source SDK that integrates generative AI with advanced animation to create realistic, multi-media pedagogical agents for engaging and adaptive human-AI interactions in educational contexts.

Contribution

It introduces a comprehensive toolkit combining LLMs, animation, and web technologies to develop emotionally resonant, context-aware animated pedagogical agents for the first time.

Findings

Enhances learner engagement and feedback receptivity.

Supports real-time personalized feedback and natural speech synchronization.

Provides a scalable, open-source platform for research and development.

Abstract

The rapid evolution of large language models (LLMs) has transformed human-computer interaction (HCI), but the interaction with LLMs is currently mainly focused on text-based interactions, while other multi-model approaches remain under-explored. This paper introduces VTutor, an open-source Software Development Kit (SDK) that combines generative AI with advanced animation technologies to create engaging, adaptable, and realistic APAs for human-AI multi-media interactions. VTutor leverages LLMs for real-time personalized feedback, advanced lip synchronization for natural speech alignment, and WebGL rendering for seamless web integration. Supporting various 2D and 3D character models, VTutor enables researchers and developers to design emotionally resonant, contextually adaptive learning agents. This toolkit enhances learner engagement, feedback receptivity, and human-AI interaction while promoting trustworthy AI principles in education. VTutor sets a new standard for next-generation APAs, offering an accessible, scalable solution for fostering meaningful and immersive human-AI interaction experiences. The VTutor project is open-sourced and welcomes community-driven contributions and showcases.