Combined Hapto-Visual and Auditory Rendering of Cultural Heritage Objects

TL;DR

This paper presents a multi-modal rendering framework that combines haptic, visual, and auditory data to enhance virtual cultural heritage objects with realistic textures, sounds, and scalable details, improving user perception and interaction.

Contribution

It introduces a novel multi-modal rendering system integrating fast haptic proxy updates, surface texture effects, and position-dependent sound synthesis for cultural heritage objects.

Findings

Haptic rendering runs 100 times faster than 1 kHz update rate

Surface textures are realistically simulated using bilateral filtering

Auditory properties are effectively integrated with position-dependent sound synthesis

Abstract

In this work, we develop a multi-modal rendering framework comprising of hapto-visual and auditory data. The prime focus is to haptically render point cloud data representing virtual 3-D models of cultural significance and also to handle their affine transformations. Cultural heritage objects could potentially be very large and one may be required to render the object at various scales of details. Further, surface effects such as texture and friction are incorporated in order to provide a realistic haptic perception to the users. Moreover, the proposed framework includes an appropriate sound synthesis to bring out the acoustic properties of the object. It also includes a graphical user interface with varied options such as choosing the desired orientation of 3-D objects and selecting the desired level of spatial resolution adaptively at runtime. A fast, point proxy-based haptic rendering technique is proposed with proxy update loop running 100 times faster than the required haptic update frequency of 1 kHz. The surface properties are integrated in the system by applying a bilateral filter on the depth data of the virtual 3-D models. Position dependent sound synthesis is incorporated with the incorporation of appropriate audio clips.