MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf

TL;DR

This paper evaluates the effectiveness of large language models as meeting delegates, developing a benchmark with real transcripts and analyzing their performance, strengths, and limitations in practical delegation scenarios.

Contribution

It introduces a comprehensive benchmark for LLMs in meeting delegation and provides empirical insights into their capabilities and challenges in real-world applications.

Findings

GPT-4/4o balance engagement strategies effectively

Approximately 60% of responses cover key points

Identifies areas for improvement in relevance and error tolerance

Abstract

In contemporary workplaces, meetings are essential for exchanging ideas and ensuring team alignment but often face challenges such as time consumption, scheduling conflicts, and inefficient participation. Recent advancements in Large Language Models (LLMs) have demonstrated their strong capabilities in natural language generation and reasoning, prompting the question: can LLMs effectively delegate participants in meetings? To explore this, we develop a prototype LLM-powered meeting delegate system and create a comprehensive benchmark using real meeting transcripts. Our evaluation reveals that GPT-4/4o maintain balanced performance between active and cautious engagement strategies. In contrast, Gemini 1.5 Pro tends to be more cautious, while Gemini 1.5 Flash and Llama3-8B/70B display more active tendencies. Overall, about 60\% of responses address at least one key point from the ground-truth. However, improvements are needed to reduce irrelevant or repetitive content and enhance tolerance for transcription errors commonly found in real-world settings. Additionally, we implement the system in practical settings and collect real-world feedback from demos. Our findings underscore the potential and challenges of utilizing LLMs as meeting delegates, offering valuable insights into their practical application for alleviating the burden of meetings.