A multi-purpose automatic editing system based on lecture semantics for remote education

TL;DR

This paper introduces an automatic multi-camera editing system for remote education that uses lecture semantics to select the most relevant views, enhancing online students' learning experience by mimicking human directing rules.

Contribution

It presents a novel system that analyzes lecture semantics to automatically direct and edit multiple video streams in real-time and offline, surpassing existing speaker-tracking methods.

Findings

System effectively captures lecture key events

Qualitative and quantitative analyses confirm its accuracy

Improves information flow for remote students

Abstract

Remote teaching has become popular recently due to its convenience and safety, especially under extreme circumstances like a pandemic. However, online students usually have a poor experience since the information acquired from the views provided by the broadcast platforms is limited. One potential solution is to show more camera views simultaneously, but it is technically challenging and distracting for the viewers. Therefore, an automatic multi-camera directing/editing system, which aims at selecting the most concerned view at each time instance to guide the attention of online students, is in urgent demand. However, existing systems mostly make simple assumptions and focus on tracking the position of the speaker instead of the real lecture semantics, and therefore have limited capacities to deliver optimal information flow. To this end, this paper proposes an automatic multi-purpose editing system based on the lecture semantics, which can both direct the multiple video streams for real-time broadcasting and edit the optimal video offline for review purposes. Our system directs the views by semantically analyzing the class events while following the professional directing rules, mimicking a human director to capture the regions of interest from the viewpoint of the onsite students. We conduct both qualitative and quantitative analyses to verify the effectiveness of the proposed system and its components.