A Vision-Based Subtitle Generator: Text Reconstruction via Subtle Vibrations from Videos

TL;DR

This paper introduces a system that generates subtitles from subtle vibrations in objects caused by sound, using video and AI techniques.

Contribution

The first generative approach to recover text directly from sound-induced object vibrations using high-speed video.

Findings

VSG-Transformer variants achieved character error rates of 13.7% and 12.5% using vibrations from a bag of chips.

The system works effectively even with limited temporal sampling in videos.

Phase-based motion estimation and HuBERT encoder reduce reliance on large video datasets.

Abstract

Subtle vibrations induced in everyday objects by ambient sound, especially speech, carry rich acoustic cues that can potentially be transformed into meaningful text, with potential implications for monitoring and security-related scenarios. This paper presents a Vision-based Subtitle Generator (VSG). This is the first attempt to recover text directly from high-speed videos of sound-induced object vibrations using a generative approach. To this end, VSG introduces a phase-based motion estimation (PME) technique that treats each pixel as an “independent microphone”, and extracts thousands of pseudo-acoustic signals from a single video. Meanwhile, the pretrained Hidden-unit Bidirectional Encoder Representations from Transformers (HuBERT) serves as the encoder of the proposed VSG-Transformer architecture, effectively transferring large-scale acoustic representation knowledge to the vibration-to-text task. These strategies significantly reduce reliance on large volumes of video data. Experimentally, text was generated from vibrations induced in a bag of chips by AISHELL-1 audio samples. Two VSG-Transformer variants with different parameter scales (Base and Large) achieved character error rates of 13.7 and 12.5%, respectively, demonstrating the remarkable effectiveness of the proposed generative approach. Furthermore, experiments using signal upsampling techniques show that the VSG-Transformer maintains effective performance when operating on videos with limited temporal sampling, indicating robustness to lower sampling rates.

Figures

Full text

1. Introduction

When sound interacts with an object, the sound induces subtle vibrations on the object surface. The resulting motion patterns partially preserve informative features of the surrounding acoustic environment. Remarkably, if speech is a component of the ambient sound, the vibrations may encode semantic content that can be extracted and reconstructed into text. This is analogous to subtitle generation for a silent film, and we term the system that we built a Vision-based Subtitle Generator (VSG).

In recent years, subtle sound-induced vibrations of object surfaces have exhibited significant potential in terms of surveillance and security [1]. Such vibrations can reveal human activity in the environment and even enable remote eavesdropping, as speech-induced mechanical responses can be sensed without direct acoustic access. This capability raises important privacy and security considerations and motivates systematic investigation of vibration-based sensing [1,2,3,4,5,6,7,8,9]. Most prior work in this area falls into two main categories [2]: Audio recovery and acoustic source classification.

Audio recovery methods reconstruct acoustic signals in a form that can be played and recognized by the human ear [1,2,3,4,5,6,7,8,9]. One classical example involves the use of laser Doppler vibrometers (LDVs) [3] that recover sound by measuring the Doppler shift in laser light reflected as the surface of an object vibrates. In recent years, researchers have explored a range of alternative physical mechanisms for audio recovery. The Lamphone [2,4] use telescopes and optical sensors to detect minute optical variations of distant objects that are induced by sound-driven vibrations. The intensities of speaker light-emitting diode (LED) indicators [5] and vibrations of the read/write heads of hard disk drives [6] can also “accidentally” leak acoustic information that enables remote audio recovery. In addition, camera-based vibration measurement has attracted considerable research interest [10,11,12], demonstrating that subtle vibrations can be reliably extracted from video using image analysis, and thereby enabling new opportunities for audio reconstruction from visual signals. For example, Davis et al. [1] were the first to introduce a phase-based algorithm that retrieved acoustic signals from human-imperceptible motions in high-speed videos. Zhang et al. [7] developed a more computationally efficient method using singular value decomposition (SVD) techniques to enhance the robustness and accuracy of sound reconstruction from video data.

In contrast, acoustic source classification techniques are aimed at extracting information on the sound sources in a given scene. Such data include the number and/or gender of speakers. Also, the methods may link signals to particular words in a precompiled dictionary [13,14,15,16,17]. For example, Gyrophone [13] uses gyroscopes of the micro-electro-mechanical system embedded in modern smartphones to capture signals. This identifies speakers and enables digit-level speech recognition, albeit within a limited vocabulary (“one”, “two”, “three”, etc.). Side Eye [14] leverages the rolling shutter and movable lens mechanisms of mainstream smartphone cameras to create a point-of-view (POV), optical-acoustic side channel that accurately identifies spoken digits, speakers, and genders. As the latter technique does not focus on human auditory perception, a broad range of unintelligible or rough signals can be utilized. Table 1 presents a summary of related work.

However, both mainstream approaches are associated with distinct challenges [2]:

• (1)Audio recovery techniques focus on human auditory perception, imposing strict requirements on the complex signal processing pipelines that are often built using expert prior knowledge. Recognition performance can vary significantly by the hearing capacity and training level of the listener.
• (2)The principal limitation of classification-based techniques is the restricted output space of existing models. Classification is often limited to isolated words or digits, usually in precompiled dictionaries. This renders the compilation of task-specific word lists difficult.

Innovatively, VSG represents the first attempt to recover text directly from high-speed videos of sound-induced object vibrations via a generative approach. Compared to existing constrained approaches—either human-auditory-dependent audio recovery or fixed-vocabulary acoustic source classification—VSG transcends their limitations by formulating the task as an open-ended generative problem. To tackle Challenge 1, VSG introduces a phase-based motion estimation (PME) technique that treats each pixel as an “independent microphone”, extracting thousands of pseudo-acoustic signals (PASs) from a single video. Large volumes of human-unintelligible PASs are directly utilized during training. This also avoids any need for the complex, task-specific, audio preprocessing steps that were traditionally used to enhance the auditory experience of the listener. For Challenge 2, VSG employs an autoregressive generative architecture (termed VSG-Transformer) rather than classification-based designs. The state-of-the-art pretrained Hidden-unit Bidirectional Encoder Representations from Transformers (HuBERT) serves as the encoder in this architecture. In recent years, the original BERT [18] and variants thereof, such as HuBERT [19], XLNet [20], and RoBERTa [21], have become the dominant paradigms of natural language processing. Through transfer learning, the pretrained HuBERT model can be modified to handle downstream tasks, effectively transferring large-scale acoustic representation knowledge to the vibration-to-text conversion. Moreover, these strategies significantly reduce reliance on large volumes of video data.

Experimentally, the vibrational responses of everyday objects are converted to standard audio samples of the AISHELL-1 corpus. This is a widely used, open-source Mandarin speech dataset that serves as a speech recognition benchmark [22]. The evaluation metric is the character error rate (CER). This is the standard parameter employed when assessing automatic speech recognition (ASR) models. The results validate the effectiveness of the proposed VSG approach. Text is reconstructed from subtle object vibrations captured on video. Specifically, VSG-Transformer-Base and VSG-Transformer-Large (similar models but of different scales) exhibited CERs of 13.7 and 12.5%, respectively. In addition, performance variations and limiting cases under diverse acquisition conditions were analyzed to assess robustness. Such analysis helps characterize when vibration-based text reconstruction is more likely to be effective, providing a basis for informed consideration of its broader implications, such as potential monitoring- or eavesdropping-related applications. The applicability of the VSG-Transformer to videos with limited temporal sampling was further investigated. Signal upsampling techniques were employed to mitigate the reliance on high-speed imaging devices. When appropriate interpolation strategies were applied, the VSG-Transformer maintained acceptable recognition performance under limited temporal resolution. These results suggest that the proposed VSG approach remains applicable when video data are captured with lower temporal sampling.

The remainder of the paper is organized as follows: Section 2 describes the proposed method. Section 3 presents the experimental validation and discusses the results. Section 4 concludes the paper.

2. Methods

This section presents the general framework of the proposed VSG and the key related concepts, including PAS synthesis and the VSG-Transformer architecture. The theoretical background and implementation details follow.

2.1. General Framework of the Method

VSG evaluates only ambient sound. The semantic content encoded in pixel signals is used to reconstruct output text. Figure 1 illustrates the overall flow, which consists of the following steps, A full demonstration animation of VSG is available at [https://youtu.be/FLw-quDNizQ] (accessed on 9 February 2026) (Animation_S1 in Supplementary Materials).

1. Video capture: The response of any object to sound is purely physical. As the acoustic excitations vary, the resulting object surface vibrations are captured by a camera, effectively transforming physical displacements into pixel-level signals within the video frames;
2. PAS acquisition: PASs are obtained via phase-based processing of the pixel signals;
3. VSG-Transformer training and testing: Large-scale PASs that encode rich acoustic features are used to construct the PAS dataset employed to train and evaluate VSG-Transformer. A multi-stage transfer learning strategy effectively links the pretrained acoustic representations of HuBERT to the PAS-driven VSG task;
4. Text reconstruction: The trained VSG-Transformer reconstructs text based the PASs extracted from new videos.

2.2. Extraction of PAS

As outlined above, VSG requires audio-like signals that effectively encode acoustic features of the input video V(x,y,t). Crucially, such signals must be widely accessible. This section details the principles and procedures involved. The process can be broadly divided into two stages. First, local motion signals are computed at each pixel location. Next, these signals are transformed into pixel-level representations that approximate audio waveforms. These are the PASs.

• (a)Computation of Local Motion Signals

Fleet and Jepson were the first to use spatio-temporal bandpass filters for PME in image sequences [23]. The phase gradient of the complex-valued output is a reliable approximation of the motion field. Building on this, subsequent studies used more advanced filtering strategies [24,25,26]. For example, Gautama and Van Hulle [24] employed a set of quadrature Gabor filter pairs when computing the temporal phase gradient of a spatially bandpassed video to estimate the motion field. In recent years, many studies have bypassed the explicit computation of optical-flow vector fields by, rather, directly leveraging phase variations when estimating the displacements of image textures in video sequences [1,27].

Following the approach of [1], local motion is here computed by analyzing the phase variations within a complex, steerable pyramid representation of the input video V(x,y,t). A complex steerable pyramid is a multi-scale, multi-orientation filter bank (see [27] for details) that decomposes each video frame into complex-valued sub-bands, indexed by the scale r and orientation θ. At each pixel location (x,y), the sub-band output can be represented in terms of amplitude A and phase φ, as follows:

$$[eqn]$$

Traditionally, a complex steerable pyramid is subjected to downsampling-based decomposition as in [1,27]. Here, however, the resolution across all filters is uniform (Figure 2). This yields sub-band outputs that are perfectly aligned in terms of spatial resolution. The amplitude and phase at each pixel location (x,y) across the different sub-bands exhibit a direct correspondence. This facilitates later pixel-level signal synthesis.

The phase variations $[eqn]$ are then computed by subtracting the phase of each pixel in the reference frame—typically the first frame of the video—from the corresponding phase in the current frame. The formal expression is:

$$[eqn]$$

By [24], the computed phase variations afford very good approximations of image texture displacements, especially those of subtle motions, along the corresponding orientation and scale.

To provide intuition for the relationship between phase variations and physical displacement, a simple one-dimensional example is considered. Let $[eqn]$ denote a 1D image intensity profile, and suppose the signal undergoes a global translation over time, resulting in $[eqn]$ , where $[eqn]$ denotes the displacement. Using Fourier series decomposition, the translated signal can be expressed as a sum of complex sinusoids:

$$[eqn]$$

Each term corresponds to a frequency component ω, which can be interpreted as a particular scale $[eqn]$ of the signal. For a given frequency band, the signal can be written as:

$$[eqn]$$

Since $[eqn]$ is a complex sinusoid, its phase term $[eqn]$ varies linearly with the displacement $[eqn]$ . Therefore, temporal changes in phase directly encode the physical motion. In two-dimensional images, this principle allows pixel-wise phase differences to precisely describe local displacements, enabling each pixel to act as an independent vibration sensor.

• (b)PAS Synthesis

A large number of PASs can be independently constructed by utilizing phase variations $[eqn]$ at each pixel. Specifically, within each sub-band, each local motion signal is weighted by the squared amplitude at time $[eqn]$ . This can be formulated as:

$$[eqn]$$

where $[eqn]$ denotes the sub-band index. Subsequently, the outputs across all orientations $[eqn]$ and scales $[eqn]$ are aggregated via summation:

$$[eqn]$$

Finally, PAS is normalized by scaling and centering to within the range [−1, 1]. Figure 3 compares the spectrogram of the original audio [Figure 3b] to those of the PASs [Figure 3c–e] derived from different surface regions [ $[eqn]$ of Figure 3a] of the object. To provide an intuitive understanding of signal quality, the PASs were rendered as audible waveforms using the method proposed in [1]. Audio samples corresponding to Figure 3b–e are also available at [https://youtu.be/rMcX9ofYY68] (accessed on 9 February 2026) (Animation_S2 in Supplementary Materials), enabling subjective comparison of the reconstructed PASs with the original audio. Given the inherent ambiguities of local phases in regions where image texture is weak, motion signals extracted from such pixel locations tend to be very noisy and/or unreliable.

In most areas, the resulting PASs are too coarse to be intelligible to the human ear. Current end-to-end ASR models also exhibit limited recognition performance on these signals, as detailed in Section 3.3. However, the advantage of PASs lies in their abundance—they can be densely extracted at the pixel level and collectively capture rich semantic information embedded in surface vibrations. Experimental results confirm their substantial training potential when combined with dedicated network architectures and learning strategies, as further demonstrated in Section 3.3.

Note that variations in texture, brightness, and material properties across different pixel locations that correspond to distinct regions of the object surface introduce diverse noise characteristics and frequency attenuations. Such variations contribute to the robustness of model training using large-scale PAS inputs.

2.3. Proposed Model: The VSG-Transformer

Figure 4 shows the architecture of the VSG-Transformer. An overview of each component follows, and the design rationale is explained. In broad terms, the model features three key modules: (1) a convolutional shrinkage module that suppresses noise and enhances features; (2) a pretrained HuBERT-based encoder that captures high-level acoustic representations; and, (3) an attention-based decoder that generates autoregressive text.

• (a)The convolutional shrinkage module

As defined by Equation (6) of Section 2.2, PASs are combinations of $[eqn]$ derived from the amplitude and phase outputs across all orientations $[eqn]$ and scales $[eqn]$ of the corresponding filters. For a given pixel location, filters that are mismatched in terms of scale and/or orientation typically deliver weak sub-band amplitudes. If such amplitudes are used to compute squared weightings, the PASs contain noise components that are substantially weaker than the actual signal. Therefore, soft thresholding was here used to eliminate noise. Such thresholding is as a core feature of signal denoising algorithms [28]. Noise features with magnitudes near zero are suppressed, effectively reducing them to zero. The nonlinear nature of the transformation is formally defined by the following equation:

$$[eqn]$$

where x denotes the input feature, y is the output, and τ is a positive threshold parameter. However, determination of an appropriate threshold τ remains challenging in practice.

As shown in [29], soft thresholding can be integrated into the deep architecture as a nonlinear transformation layer. The threshold parameter τ is then automatically learned by the network, not manually specified. Here, soft thresholding was employed as the activation function that was integrated with convolutional layers to form a convolutional shrinkage module that suppressed noise-related information and enhanced feature discriminability.

Specifically, given a PAS dataset for a VSG task with N training samples, $[eqn]$ , where PAS is a one-dimensional waveform input and W a sequence of L tokens $[eqn]$ (Figure 4). The convolutional shrinkage module features two 1D convolutional layers that flank a shrinkage unit. The kernel configurations are 32 and 1, respectively, and both maintain a stride of 1. Initial processing through the first convolutional layer expands the feature channels of the waveform input to 32. The subsequent shrinkage layer first dynamically computes channel-specific τ values and generates a 32-channel threshold vector. Next, that layer subjects each feature channel to a nonlinear soft thresholding operation, effectively suppressing noise but preserving critical signal components. The residual connection integrates the features that are soft-thresholded. The final 1D convolutional layer compresses the feature channels back to a single channel. This ensures that the output is compatible with the subsequent processing stages.

• (b)The pretrained HuBERT encoder

In the second module, the pretrained HuBERT encoder extracts high-level acoustic representations from the input waveform. HuBERT has consistently attracted considerable research attention. HuBERT captures complex temporal dependencies and semantic features effectively because HuBERT was trained using large datasets, including 10,000 h of speech from the WenetSpeech [30]. The pretrained HuBERT model uses its extensive prior knowledge to engage in transfer learning. HuBERT therefore rapidly adapts to PAS inputs. In this study, two pretrained HuBERT models with different parameter scales—HuBERT-Base (95 million parameters) and HuBERT-Large (317 million parameters)—were integrated with VSG-Transformer, yielding VSG-Transformer-Base and VSG-Transformer-Large, respectively.

As illustrated in Figure 4, the pretrained HuBERT model features a convolutional neural network (CNN) encoder followed by a Transformer. The CNN encoder contains seven convolutional layers, each with 512 output channels. The first layer receives a single-channel input that directly matches the one-dimensional waveform output by the convolutional shrinkage module. The subsequent layers process the 512 channel inputs. The CNN encoder delivers a downsampled two-dimensional feature sequence $[eqn]$ wherein each feature vector $[eqn]$ is of dimensionality 512 and is computed by aggregating information across all input channels. The feature sequence is subjected to local masking using the strategy described in [31,32] before transmission to the Transformer module, the Transformer features a configurable number of blocks with predefined embedding dimensions, inner FFN dimensions, and attention heads. The detailed architectural specifications of the HuBERT-Base and HuBERT-Large models are listed in Table 2.

Both models output a sequence of hidden units $[eqn]$ that serve as high-level acoustic representations. These are next fed to the downstream decoder module to improve text generation.

• (c)The decoder

The decoder module of VSG-Transformer transforms the high-level acoustic representations produced by the HuBERT encoder into coherent text outputs. The decoder employs a self-attention mechanism to effectively capture long-range dependencies within the sequence. The decoder operates in an autoregressive manner through a stack of attention blocks, integrating both acoustic features from the HuBERT encoder and text-level information from previous decoding steps to enhance transcriptional accuracy.

The structure of the decoder is illustrated in Figure 4. A character-embedding layer is first used to convert the character sequence into an output encoding $[eqn]$ , which is then combined with a positional encoding $[eqn]$ . The dimensionality of both embeddings is that of HuBERT. The combined embeddings are then passed through a stack of $[eqn]$ decoder blocks that generate the final decoder outputs. Each decoder block features three sub-blocks, of which the first is a masked multi-head self-attention block wherein the queries, keys, and values are identical. Masking is used to ensure that the prediction at position $[eqn]$ depends only on the outputs at positions earlier than $[eqn]$ . The second sub-block is a multi-head attention block in which the keys and values are the encoder outputs $[eqn]$ . Here, the queries are derived from the output of the preceding sub-block. The third sub-block is a position-wise feed-forward network. Each sub-block in the decoder features a residual connection and layer normalization. The latter uses a pre-norm rather than the conventional post-norm structure. Given input $[eqn]$ to a sub-block, the corresponding output is:

$$[eqn]$$

Finally, the decoder outputs are computed via a linear projection followed by a softmax function.

In general, VSG-Transformer effectively integrates noise suppression, acoustic representation, and autoregressive text generation, leveraging transfer learning to fully exploit the capabilities of HuBERT and enable reliable PAS-to-text conversion. The combination of modules described above is well-suited to the VSG task, which demands accurate modeling of complex acoustic and semantic patterns embedded in input PAS signals.

Notably, VSG-Transformer does not rely heavily on semantic features within PASs per se. Rather, transfer learning enables the model to leverage knowledge acquired in the source domain (the speech corpora) using the HuBERT module and then apply that information to tasks in the target domain (the PAS dataset). This significantly reduces model dependency on large volumes of PAS data. The model can be trained using only a few high-speed videos. Data acquisition from such videos is computationally demanding.

3. Experimental Validation

The VSG technique was evaluated in a series of experiments. The setup included a common object (a bag of potato chips), a loudspeaker (KRK Rokit 5), and a high-speed camera (i-SPEED 230). All video recordings were captured in a typical meeting room (Figure 5). Videos were captured at high frame rates (about 16 kHz; the spatial resolution was 128 × 128 pixels). Standard audio files from the AISHELL-1 corpus—an open-source Mandarin speech dataset with a speech recognition baseline—were played through the loudspeaker to excite sound-induced vibrations on the object surface. A total of 500 video clips were recorded, each corresponding to a distinct speech utterance randomly sampled from the AISHELL-1 corpus, with no sentence or audio clip repeated across the dataset. The speech signals were played at an approximate sound pressure level (SPL) of 80 dB, comparable to that of an unamplified stage actor. Experimental evidence suggests that this quantity is adequate, as the model’s semantic understanding primarily originates from the pretrained module. The PASs extracted from the videos expose the model to variations in texture, brightness-related noise, and frequency attenuation—factors that enhance robustness during training. Subsequently, initial and final non-informative frames were manually truncated to retain only valid segments.

A chip bag was selected as the primary object for evaluation due to its strong and consistent vibration response under acoustic excitation. Compared to other common materials such as foam cups, plants, tissue boxes, and other everyday items, it exhibits more favorable motion characteristics, as demonstrated in [1]. Since the current experiment represents an initial feasibility study of VSG, the evaluation focused on objects with a higher likelihood of producing reliable results. Given the substantial time and computational demands of high-speed video acquisition and processing, large-scale testing across a broader range of materials is being conducted in follow-up work and will be published in future updates.

3.1. Dataset Generation

Figure 6 shows frames from videos captured during the experiments. Each frame is fully occupied by the object, effectively eliminating background interference. During recording, the object pose and the camera viewing angle were randomly varied but with maintenance of an approximately constant distance between the object and the camera. This significantly enhanced the robustness and generalizability of the experimental results.

PASs were extracted from the videos as described in Section 2.2. Motion signals from pixels on the object surface that exhibited clear textural and phase variations were employed. Specifically, all PASs within each video—128 pixels (height) × 128 pixels (width) = 16,384 pixels in total—were extracted for subsequent processing. A percentile-based pooling strategy was employed. This is analogous to the pooling layers employed during deep learning. Specifically, each input frame of size 128 (height) × 128 (width) pixels was partitioned into 256 regions, each measuring 8 (height) × 8 (width) pixels. Within each region, the PAS corresponding to the pixel, the amplitude of which was closest to $[eqn]$ , was selected, where $[eqn]$ and $[eqn]$ were the maximum and minimum amplitude within the region, respectively, and $[eqn]$ a user-defined threshold (here, 0.8). This identified a representative PAS within each local region, suppressed any effects of noise and outliers, and thereby improved the general quality of the selected PASs.

After application of the percentile-based pooling strategy, each 128 (height) × 128 (width) video frame yielded 256 PASs. A total of 500 videos were randomly split into training, validation, and test sets in a 70:15:15 ratio (350, 75, and 75 videos, respectively) at the video level, such that no video appears in more than one split. The speech samples were drawn from a subset of speakers in AISHELL-1, and individual speakers may appear in more than one split. However, the utterance content across different speakers is strictly non-overlapping, as the task focuses on text reconstruction rather than speaker-specific characteristics. In total, the PAS dataset contained 128,000 PAS samples (500 × 256). Detailed statistics are listed in Table 3.

Additionally, the publicly available AISHELL-1 corpus, which contains 178 h of Mandarin speech, was employed during certain stages of training. All recordings were sampled at 16 kHz. To facilitate comparisons, each training sample in the PAS dataset is here termed an “utterance”, following the AISHELL-1 convention. However, as detailed in Section 2.2, PASs are not equivalent to semantically intelligible speech in the traditional sense.

3.2. Training and Testing

Section 2.3 described the architecture of the proposed framework. There are three key modules: a convolutional shrinkage module for noise suppression, a pretrained HuBERT encoder for acoustic representation, and an attention-based decoder for text generation. A transfer learning strategy was used to retain as much as possible of the knowledge of the pretrained HuBERT when effectively training our model across all components. This involved stage-wise adjustments of the training parameters, the use of module freezing policies, and employment of different dataset inputs. This section outlines these in detail.

During training stage 1, both the shrinkage layer of the convolutional shrinkage module and the entire HuBERT encoder were frozen, as illustrated in Figure 7a. During this stage, only AISHELL-1 was used for training over 130 epochs. Two parallel experiments were conducted using HuBERT-Base and HuBERT-Large. Stage 1 sought to establish an automatic baseline ASR capability based on the AISHELL-1 corpus. In stage 2, the shrinkage layer was thawed but the HuBERT encoder remained frozen [Figure 7b]. Training in this stage employed only the PAS dataset (40 epochs). The two parallel training tracks were based on the respective models obtained during stage 1. Stage 2 aimed to optimize the shrinkage layer in terms of task-specific noise suppression while allowing transfer of knowledge from AISHELL-1 to the recognition tasks posed by the PAS dataset.

All VSG-Transformer modules were implemented in PyTorch 2.1.0 [33]. Each training batch ( $[eqn]$ ) contained approximately 100 s of speech or PAS and the corresponding transcriptions. The Adam optimizer [34] was employed; the parameters were $[eqn]$ and $[eqn]$ . The learning rate was varied during training using the warm-up schedule of:

$$[eqn]$$

where warmup was set to 12,000. To prevent overfitting, the neighborhood smoothing strategy of [35] was employed, with the probability of the correct label set to 0.8. The token vocabulary included 4231 characters from the training set with two special symbols: “<unk>” for unseen tokens and “<eos>” to pad the ends of token sequences. Both the residual dropout and attention dropout rates were 0.1. Residual dropout was applied to each sub-block prior to addition of the residual connections. Attention dropout was applied to the softmax activation within each attention mechanism. Finally, the model parameters from the last 10 epochs were averaged to obtain the final output. During inference, beam search decoding employed a beam width of 10 and a length penalty of 1.0 [36]. When PASs were extracted from the same video, a majority voting strategy was used to aggregate the recognition results. This featured token-wise voting across all sequences followed by selection of the most frequent token at each position when creating the final transcription. All experiments were conducted on an NVIDIA RTX 4090 GPU, and the reported CER results are the averages over five independent runs. The overall pipeline of VSG is visualized in Figure 8.

3.3. Results and Ablation Studies

The results are presented in Table 4. Using the transfer learning strategy, the VSG-Transformer successfully achieved PAS-to-text conversion using high-speed video recordings of common vibrating objects. Human auditory perception was not in play. Specifically, VSG-Transformer-Base achieved a CER of 13.7% on the VSG task; the VSG-Transformer-Large figure was 12.5%. In contrast, for the AISHELL-1 ASR task of stage 1, the CERs were 6.4% and 6.1%, respectively, indicating that VSG-Transformer can engage in end-to-end ASR. VSG-Transformer-Large performed better than VSG-Transformer-Base on both the ASR and VSG tasks, showing that an increase in the model scale enhanced the capacity to extract rich acoustic representations for downstream tasks.

After multi-stage training of the multi-module VSG-Transformer network, ablation studies were used to assess quantitatively the utilities of individual modules within the network architecture and to isolate and evaluate the impacts of the stage-wise training strategy by selectively removing certain training components.

The ablation study was designed to systematically examine the contributions of key components and design choices in the proposed VSG framework. Six ablation tests were conducted using the VSG-Transformer-Large model. The results are summarized in Table 5.

Specifically, freezing the convolutional shrinkage layer during stage-2 training results in a clear degradation in recognition accuracy, with the CER increasing from 12.5% to 19.1%. This observation indicates that adaptive noise suppression is essential for mitigating PAS-specific artifacts and stabilizing the representations prior to decoding. Varying the depth of the decoder shows that architectural capacity plays an important but non-linear role. Increasing the number of decoder blocks generally improves recognition performance, suggesting that deeper decoders better capture long-range semantic dependencies in PAS sequences. However, the gains become marginal once the number of blocks exceeds six, indicating diminishing returns from excessive depth. The aggregation strategy at inference time is also critical. Removing majority voting causes a substantial performance drop, with the CER rising to 24.5%. This result confirms that individual PASs often contain incomplete or noisy vibration cues due to spatial variability across pixels, and that majority voting effectively integrates complementary information from multiple PASs to suppress local disturbances and improve transcription robustness. Performance is further influenced by the PAS selection strategy. Both overly aggressive pooling and excessively coarse or fine spatial partitioning lead to degraded accuracy. The baseline configuration, using 8 × 8 regions with a percentile threshold of $[eqn]$ = 0.80, achieves the best balance between preserving informative vibration cues and suppressing noise, whereas weighted averaging and extreme region sizes introduce additional interference or dilute useful information. Comparisons with conventional baselines further reinforce the effectiveness of the proposed design. A visual-microphone-based pipeline directly followed by an ASR model, without stage-2 adaptation, performs poorly (CER = 40.3%). A PAS + ASR baseline without stage-2 training degrades even further (CER = 56.7%). These results indicate that the proposed VSG-Transformer, together with the stage-wise transfer learning strategy, is essential for reliable PAS-to-text conversion.

3.4. Computational Cost and Runtime Analysis

The proposed VSG framework focuses on reconstructing textual content directly from subtle vibrations captured in high-speed video. Beyond recognition accuracy, an understanding of the computational characteristics of the processing pipeline and its behavior under challenging acquisition conditions is important for assessing the practical scope of the proposed approach. Section 3.4 therefore analyzes the runtime of the main processing stages.

The overall processing pipeline consists of two major stages: (1) PME-based generation of PASs, and (2) inference using the VSG-Transformer. Among these stages, PME constitutes the dominant computational overhead, as it operates at the pixel level on high-frame-rate video sequences. The runtime of this stage scales approximately linearly with video duration and spatial resolution, and further increases with the number of pyramid scales and orientations employed in the steerable pyramid decomposition. As reported in Table 6, for a representative 5-s video clip with a spatial resolution of 128 × 128 pixels, the PME preprocessing time ranges from 129.79 s when using a shallow configuration (1 scale, 2 orientations) to 212.38 s for a deeper and more directional configuration (3 scales, 4 orientations). This increase reflects the additional computational burden introduced by multi-scale and multi-orientation filtering. All experiments were conducted on a workstation equipped with an NVIDIA RTX 4090 GPU and an Intel Core i9-12900K CPU. All runtimes reported in Table 6 are averaged over 10 independent runs and are provided as representative references.

In contrast, the inference stage based on the VSG-Transformer is comparatively lightweight once PASs have been extracted and cached offline. The inference time is approximately 0.16 s per PAS for the Base model and 0.43 s per PAS for the Large model. When aggregating 256 PASs per video using batch inference followed by majority voting, the total transcription time is approximately 27.12 s for the Base model and 75.51 s for the Large model per video.

These results indicate that, while PME preprocessing remains computationally intensive in the current implementation, the downstream text inference stage does not constitute a bottleneck.

3.5. Failure Modes and Operational Limitations

In addition to computational cost, the robustness of the proposed framework under diverse acquisition conditions is an important practical consideration. To this end, we analyzed the performance variations and limiting cases observed in the experiments, as summarized in Table 7. Figure 9 presents representative frames extracted from videos captured under different acquisition and scene conditions.

Performance degradation is primarily associated with conditions that weaken vibration-induced motion cues or reduce their reliability in PAS extraction. Specifically, increasing the camera–object distance and decreasing the SPL make vibration-induced motion cues harder to capture, resulting in lower signal-to-noise ratios in the extracted PASs and correspondingly higher CERs. Similarly, adverse imaging conditions—such as reduced illumination or defocus—negatively affect the quality of phase-based motion estimation by diminishing local texture contrast and phase stability, thereby introducing additional noise into PAS extraction. Scene composition also plays an important role. As the object occupancy ratio within the field of view decreases, the amount of usable vibration information becomes limited, which degrades transcription accuracy. Furthermore, under complex acoustic conditions involving overlapping speech, recognition performance deteriorates markedly as interference increases, reflecting the inherent difficulty of isolating target speech content from visually induced vibrations alone.

In extreme cases—including very low SPL, severe defocus, strong interfering speech, or minimal object occupancy—PAS extraction becomes unstable, leading to partial or complete recognition failure. These failure modes delineate the current operational boundaries of the proposed method and highlight scenarios in which visual vibration-based speech recovery is intrinsically challenging.

From a broader perspective, the identified performance boundaries also provide practical guidance for assessing the feasibility of vision-based vibration sensing in monitoring or eavesdropping-related applications. The results indicate that successful text reconstruction relies on sufficiently strong vibration responses, adequate visual resolution, and limited acoustic interference. These findings offer practical insights for determining when visual vibration analysis may yield usable semantic information, and when it is unlikely to be effective. As such, the present study contributes not only a proof of concept, but also a set of empirically grounded conditions that can inform future system design and feasibility assessment in surveillance-oriented scenarios.

3.6. VSG Operation on Videos with Limited Temporal Sampling

The VSG framework and the experimental validation thereof have been described above. There is one key constraint: the raw VSG input must be high-speed (16-kHz) video because both the HuBERT-Base and HuBERT-Large models [19] were pretrained on speech data encoded as 16-kHz mono WAV files. The PASs must match this sampling rate to ensure appropriate alignment with the expected input format. Otherwise, feature extraction errors that significantly compromise model performance are to be expected.

However, unlike audio recording, high-speed video recording is resource-intensive. Memory consumption is high, as is the computational cost. This section investigates whether VSG might accept lower-frame-rate videos. Audio waveform upsampling techniques were leveraged when seeking to build a lightweight VSG version.

The experiments investigated this issue using four different upsampling methods. One was a traditional signal processing technique (Bandlimited Sinc Interpolation (BSI) of the widely adopted PyTorch-based Torchaudio library [37]). Additionally, three deep learning approaches were assessed: a deep neural network (DNN) [38], attention-based feature-wise linear modulation (AFiLM) [39], and Phase-Net [40]. A DNN can implicitly predict missing high-frequency components in bandwidth-expanded speech, thereby improving recognition performance when speech signals are obtained under lower temporal sampling. A DNN also addresses the discontinuity between a narrowband input and a reconstructed high-frequency spectrum. AFiLM uses a self-attention mechanism to model long-range temporal dependencies. It is aimed at increasing the fidelity of waveform upsampling. Both of these methods operate directly on 1D PAS waveforms. However, Phase-net uses phase data as the primary inputs. Phase-net was originally designed for video frame interpolation, leveraging phase and amplitude features from complex-valued sub-bands obtained by decomposing each frame using a multi-scale, multi-orientation filter bank (Section 2.2 above). Phase-Net can integrate the obtained phase interpolations with PAS computations, thereby enabling effective PAS upsampling.

BSI upsampling used the parameters of the Torchaudio audio-processing guidelines [37]. These leverage GPU-based acceleration to optimize interpolation, effectively balancing computational efficiency with spectral fidelity. The structural configurations of the deep learning models were those of the original publications [38,39,40]. Hyperparameter tuning and the training details can be found in the respective references. All four upsampling methods were used to modify original high-speed videos sampled at 8, 4, and 2 kHz, generating 16-kHz PASs via 2×, 4×, and 8× interpolation, respectively. The upsampled PASs were used to construct the PASs dataset and then for stage 2 training and evaluation of VSG-Transformer-Large. The CER results are summarized in Table 8.

As shown in Table 8, at all upsampling ratios, Phase-Net consistently yielded the best VSG performance, with all CERs below 20%. Remarkably, Phase-Net achieved test CERs of 13.6 and 15.3% under 2× (original 8 kHz) and 4× (original 4 kHz) upsampling. Performance degradation was slight compared to direct training on native 16 kHz data. Performance did not collapse.

Both DNN and AFiLM performed very similarly in terms of the CERs, particularly when recognizing speech at original 4-kHz or higher sampling rates. In contrast, BSI did not perform well. After 8× upsampling, VSG-Transformer did not converge during stage 2 training. This may be because, during sampling at 2 kHz, the Nyquist limit restrains the signal bandwidth to below 1 kHz. Such a restricted bandwidth typically captures only the fundamental frequency ( $[eqn]$ ) and part of the first formants ( $[eqn]$ ) of vowels and voiced consonants, entirely missing the higher formants such as $[eqn]$ and $[eqn]$ . Moreover, high-frequency signals from unvoiced consonants such as /f/, /s/, and /k/, which are often above 2 kHz, are completely absent in such low-bandwidth signals [14], significantly impairing recognition accuracy.

Historically, all three deep learning-based upsampling methods sought to incorporate speech features embedded in a high-frequency spectrum into the narrowband speech signal. During training of Phase-Net, DNN, and AFiLM, 16-kHz signals served as the targets of training. In contrast, BSI—a pure numerical interpolation method—fails to recover lost high-frequency features during upsampling. This fundamental distinction largely explains the performance gap between BSI and the deep learning methods. Effective VSG extraction from video data with limited temporal sampling seems to require guidance from high-frame-rate signals during upsampling. However, deep networks featuring learnable upsampling parameters serve as promising directions toward lightweight VSG implementation.

4. Conclusions and Future Work

This paper shows that subtle vibrations of everyday objects, caused by ambient sound, can be extracted from video recordings and used to generate text, effectively transforming the objects into vision-based subtitle generators. The VSG pipeline initially acquires pixel-level PASs from the surfaces of objects by PME. These are scalable, abundant robust signals that effectively represent the underlying acoustic features. A pretrained HuBERT-based generative model, termed the VSG-Transformer, is then employed for the PAS-to-text task. The architecture effectively integrates three key components: a convolutional shrinkage module for noise suppression, a pretrained HuBERT-based encoder for extraction of high-level acoustic representations, and an attention-based decoder for autoregressive text generation. Training employs multi-stage transfer learning. The model leverages the knowledge embedded in the pretrained HuBERT and effectively adapts this to recognition of the PAS dataset.

Experimentally, the Base and Large variants of VSG-Transformer successfully accomplished VSG tasks. The CERs of the models were 13.7 and 12.5%, respectively. Both models also performed excellently on the conventional ASR task trained on AISHELL-1 with CERs of 6.4 and 6.1%, respectively. Comprehensive ablation studies quantitatively evaluated the individual contributions of each module and the overall effectiveness of multi-stage training. Finally, the possibility of lightweight VSG implementation was explored. Low-sampling-rate PASs were upsampled. Four upsampling methods—BSI, DNN, AFiLM, and Phase-Net—that covered both traditional signal processing and deep learning approaches, were tested. Phase-Net upsampling consistently yielded the best performance. Specifically, the VSG-Transformer-Large CERs were 13.6, 15.3, and 19.6% after training on and evaluation of PAS datasets upsampled from original 8, 4 and 2 kHz videos, respectively. In addition, this study systematically examined performance variations and limiting cases under diverse acquisition conditions. The results help characterize when vibration-based text reconstruction is more likely to be effective, providing a principled basis for considering its broader implications.

In conclusion, VSG fully leverages the flexibility of deep networks and the strength of transfer learning, effectively bridging computer vision-based measurement and natural language processing. The VSG represents a promising new research direction with considerable potential. Future work should explore signal combinations from multiple objects and the integration of inputs from multiple synchronized cameras. Such temporally aligned signals might contain implicit correlations across different sources, further enhancing VSG performance. The rapid advances in smartphone electronic components—particularly the significant increase in camera frame rates (approaching 2 kHz in certain devices)—suggest that VSG deployment in such smartphones is feasible.