Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams

TL;DR

Augmented Physics is a novel tool that uses computer vision and language models to convert static textbook diagrams into interactive simulations, enhancing physics learning through personalized and engaging digital content.

Contribution

This work introduces a semi-automatic system that integrates computer vision and multimodal language models to create embedded interactive physics simulations from static textbook diagrams.

Findings

System enables semi-automatic extraction of diagrams from textbooks.

Interactive simulations improve engagement and personalization in physics education.

Usability and expert feedback indicate positive educational impact.

Abstract

We introduce Augmented Physics, a machine learning-integrated authoring tool designed for creating embedded interactive physics simulations from static textbook diagrams. Leveraging recent advancements in computer vision, such as Segment Anything and Multi-modal LLMs, our web-based system enables users to semi-automatically extract diagrams from physics textbooks and generate interactive simulations based on the extracted content. These interactive diagrams are seamlessly integrated into scanned textbook pages, facilitating interactive and personalized learning experiences across various physics concepts, such as optics, circuits, and kinematics. Drawing from an elicitation study with seven physics instructors, we explore four key augmentation strategies: 1) augmented experiments, 2) animated diagrams, 3) bi-directional binding, and 4) parameter visualization. We evaluate our system through technical evaluation, a usability study (N=12), and expert interviews (N=12). Study findings suggest that our system can facilitate more engaging and personalized learning experiences in physics education.