# Exploring multimodal collaborative storytelling with Pepper: a preliminary study with zero-shot LLMs

## TL;DR

This paper explores using a social robot and large language models to co-create stories with users, combining physical objects and natural language.

## Contribution

The novel integration of zero-shot LLMs and a social robot for multimodal collaborative storytelling.

## Key findings

- Users showed high acceptance of the storytelling system with Pepper.
- The Llama model successfully adapted stories in a zero-shot setting.
- Incorporating physical objects enhanced user engagement and immersion.

## Abstract

With the rise of large language models (LLMs), collaborative storytelling in virtual agents or chatbots has gained popularity. Despite storytelling has long been employed in social robotics as a means to educate, entertain, and persuade audiences, the integration of LLMs into such platforms remains largely unexplored. This paper presents the initial steps for a novel multimodal collaborative storytelling system in which users co-create stories with the social robot Pepper through natural language interaction and by presenting physical objects. The robot employs a YOLO-based vision system to recognize these objects and seamlessly incorporate them into the narrative. Story generation and adaptation are handled autonomously using the Llama model in a zero-shot setting, aiming to assess the usability and maturity of such models in interactive storytelling. To enhance immersion, the robot performs the final story using expressive gestures, emotional cues, and speech modulation. User feedback, collected through questionnaires and semi-structured interviews, indicates a high level of acceptance.