Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition

TL;DR

This paper investigates integrating domain-adapted self-supervised learning models, specifically wav2vec2.0, into ASR systems to improve recognition accuracy for dysarthric and elderly speech, achieving significant WER reductions.

Contribution

It introduces novel methods for combining wav2vec2.0 with traditional ASR systems and applies multi-modal approaches, resulting in state-of-the-art performance on dysarthric and elderly speech datasets.

Findings

Significant WER reductions of 8.22% and 3.43% absolute on two datasets.

Achieved lowest published WERs of 22.56% and 18.17%.

Enhanced recognition of very low intelligibility and unseen words.

Abstract

Automatic recognition of disordered and elderly speech remains a highly challenging task to date due to the difficulty in collecting such data in large quantities. This paper explores a series of approaches to integrate domain adapted SSL pre-trained models into TDNN and Conformer ASR systems for dysarthric and elderly speech recognition: a) input feature fusion between standard acoustic frontends and domain adapted wav2vec2.0 speech representations; b) frame-level joint decoding of TDNN systems separately trained using standard acoustic features alone and with additional wav2vec2.0 features; and c) multi-pass decoding involving the TDNN/Conformer system outputs to be rescored using domain adapted wav2vec2.0 models. In addition, domain adapted wav2vec2.0 representations are utilized in acoustic-to-articulatory (A2A) inversion to construct multi-modal dysarthric and elderly speech recognition systems. Experiments conducted on the UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest TDNN and Conformer ASR systems integrated domain adapted wav2vec2.0 models consistently outperform the standalone wav2vec2.0 models by statistically significant WER reductions of 8.22% and 3.43% absolute (26.71% and 15.88% relative) on the two tasks respectively. The lowest published WERs of 22.56% (52.53% on very low intelligibility, 39.09% on unseen words) and 18.17% are obtained on the UASpeech test set of 16 dysarthric speakers, and the DementiaBank Pitt test set respectively.

Tables

Table 1. Table 1 : The performance of wav2vec2.0 based ASR, TDNN/Conformer based systems constructed without or with wav2vec2.0 (w2v) representations and optionally using the cross-domain inverted UTI articulatory features on 1) the UASpeech test set; AND 2) the DementiaBank Pitt development (Dev) and evaluation (Eval) sets; ``VL/L/M/H'' refer to intelligibility subgroups. ``INV'' and ``PAR'' refer to clinical investigator and elderly participant. ``ada. w2v'' and ``ada. BN'' are the abbreviations of cross-domain adapted wav2vec2.0 representations and bottleneck features from MLAN [ 35 ] respectively. ``+'' and `` → → \rightarrow '' stand for frame-level joint decoding and multi-pass decoding. † † {\dagger} and ∗ ∗ \ast denote statistical significant (MAPSSWE, α 𝛼 \alpha = 0.05) differences obtained against the baseline wav2vec2.0 and Conformer systems (Sys. 4,12).

Sys	model	acoustic feature	A2A input	UASpeech WER (%)							DBank WER(%)
				unseen	seen	VL	L	M	H	All	Dev.		Eval.		All
											PAR.	INV.	PAR.	INV.
$1$	TDNN-f	FBK	✗	51.62	16.98	62.53	31.92	23.12	13.67	30.56	47.93	19.91	36.66	19.76	33.80
$2$			ada. BN	54.95	17.64	63.35	33.14	25.76	15.78	32.27	45.82	19.21	34.89	18.42	32.35
$3$	+LHUC	FBK	✗	47.73	15.49	61.12	28.95	19.08	11.94	28.13	45.49	19.26	35.44	18.42	32.33
$4$	wav2vec2	raw audio	✗	55.75	14.67	62.76	36.91	25.24	9.11	30.78	29.71	14.29	21.27	15.32	21.60
$5$	TDNN-f	FBK+w2v	✗	55.01	14.65	60.59†	34.81†	23.80†	12.01	30.48	27.21†	13.20†	19.13†	12.87	19.73†
$6$			ada. w2v	53.90†	14.17	60.07†	33.58†	23.49†	11.33	29.75†	26.43†	12.77†	18.29†	12.10†	19.08†
$7$	Sys.3 + 5	-	-	43.77†	11.72†	53.19†	26.33†	16.47†	9.02	24.29†	26.98†	12.82†	18.06†	12.21	19.29†
$8$	Sys.3 + 6	-	-	43.68†	11.71†	53.39†	26.17†	16.57†	8.84	24.25†	26.11†	12.60†	17.80†	11.32†	18.78†
$9$	Sys.3+5+6	-	-	43.54†	11.68†	53.26†	26.13†	16.45†	8.81	24.18†	26.05†	12.52†	17.59†	11.43†	18.69†
$10$	Sys.3$\to$4	-	-	40.06†	11.72†	52.53†	25.00†	14.80†	7.08†	22.83†	34.17	15.41	25.49	15.98	24.55
$11$	Sys.9$\to$4	-	-	39.09†	11.90†	53.12†	25.03†	14.04†	6.32†	22.56†	25.27†	12.28†	16.84†	11.54†	18.17†
$12$	Conformer	FBK	✗	99.30	18.24	66.77	49.39	46.47	42.02	50.03	48.71	20.97	36.93	19.42	34.57
$13$		FBK+w2v	ada. w2v	72.80∗	14.87∗	64.24∗	43.09∗	36.41∗	17.28∗	37.59∗	28.38∗	14.53∗	19.40∗	13.10∗	20.79∗
$14$	Sys.13$\to$4	-	-	71.73∗	14.50∗	63.46∗	42.75∗	35.69∗	16.53∗	36.95∗	27.89∗	14.28∗	19.15∗	13.10∗	20.45∗

Table 2. Table 2 : WER (%) of published and our best systems on UASpeech. ``15 spk'' stands for only 15 speakers in the test set, but the same 4 speakers are used in ``very low'' (VL) intelligibility group.

System	VL	All
Sheffield-2020 Fine-tuning CNN-TDNN speaker adaptation [2] (15 spk)	68.24	30.76
CUHK-2021 NAS DNN + Data Aug. + LHUC-SAT + AV fusion [3]	60.30	25.21
CUHK-2022 DNN + Data Aug. + LHUC-SAT + AUV fusion [39]	60.14	24.82
CUHK-2022 DNN + Data Aug. + SBE Adapt + LHUC-SAT [6]	59.30	25.05
CUHK-2022 TDNN + spectral basic GAN + LHUC-SAT [13]	59.18	27.85
BUT-2022 Wav2vec2 + fMLLR + xvectors [31] (15 spk)	57.72	22.83
TDNN + wav2vec2 fea. + sys. combination (sys. 10 Table 1, ours)	52.53	22.83
TDNN + wav2vec2 fea. + sys. combination (sys. 11 Table 1, ours)	53.12	22.56

Table 3. Table 3 : WER (%) and AD detection accuracy (%) between published systems on DementiaBank and our system.

System	WER (%)		Acc
	All	Eval PAR.
TDNN + Data Aug. + domain adapt. + LHUC-SAT [4]	29.90	33.17	87.5
TDNN + Data Aug. + SBE Adapt + LHUC-SAT [6]	29.16	31.89	-
TDNN + spectral basic GAN + LHUC-SAT [13]	31.60	34.08	-
Conformer + NAS + domain & spk adapt + sys. combination [20]	24.2	25.5	87.5
TDNN + wav2vec2 fea. + sys. combination (sys. 11 table 1, ours)	18.17	16.84	91.7

Equations

$$L_m=-\log \frac{\exp (\mathrm{sim}(c_t,q_t)/\kappa )}{{\sum }_{\tilde{q}\in \tilde{Q}}\exp (\mathrm{sim}(c_t,\tilde{q})/\kappa )}$$

$$\text{vspace}-0.1cm\hat{P}(y_t|X_1^{1:T},X_2^{1:T})=\alpha P_{a+w}(y_t|X_1^{1:T})+\beta P_a(y_t|X_2^{1:T})$$

$${\hat{Y}}_{best}=\arg \underset{i}{\min }\{\alpha s_i^{w2v}+\beta s_i^{tdnn/cfm}\}\text{vspace}-0.1cm$$

Full text

Index Terms—  Dysarthric Speech, Elderly Speech, Wav2vec2.0, Pre-trained ASR System

1 Introduction

Accurate recognition of elderly and disordered speech remains a challenging task [1, 2, 3, 4, 5, 6, 7]. Speech disorders such as dysarthria are associated with neuro-motor conditions and often found among elderly adults experiencing neurocognitive disorders such as Alzheimer’s disease (AD). ASR technologies tailored for their needs not only improve their quality of life, but also support large scale automatic early diagnosis of neurocognitive impairment [8, 9, 10, 11, 12, 4].

Elderly and disordered speech bring many challenges to current deep learning based ASR technologies predominantly targeting non-aged, healthy adult users. First, a large mismatch between such data and non-aged, healthy adult voice is often observed. Second, the co-occurring disabilities and mobility limitations often found among elderly and disordered speakers lead to the difficulty in collecting large quantities of such data that are essential for data intensive ASR system development. To this end, a series of recent researches have been conducted to transfer knowledge from normal speech and ASR systems trained on such out-of-domain data [6, 4, 13, 14, 15, 16, 1, 2, 3, 5, 17, 18, 19, 20].

An alternative solution to address the above data scarcity problem is to use self-supervised learning (SSL) models [21, 22, 23] that are pre-trained on large quantities of unlabelled data. The associated neural speech representations produced by these pre-trained ASR systems are also inherently robust to domain mismatch [24, 25, 26]. Although they have been successfully applied to a range of normal speech processing tasks including speech recognition [21, 27, 22, 23], speech emotion recognition [28] and speaker recognition [29], very limited researches on SSL pre-trained models for disordered and elderly speech have been conducted [30, 31, 24]. Among these, wav2vec2.0 and WavLM models were applied to Japanese electrolaryngeal and English dysarthric speech in [30], where an overall WER of 51.8% was reported on the benchmark UASpeech task. Speaker adaptation of wav2vec2.0 models using fMLLR and x-vectors for dysarthric speech recognition was investigated in [31]. Cross-lingual SSL based dysarthric speech recognition was studied in [24].

Efforts on applying SSL pre-trained speech models to dysarthric and elderly speech are also confronted with the same data scarcity problem. Direct fine-tuning of a large number of model parameters on limited impaired or elderly speech data rapidly leads to overfitting and poor generalisation. This issue is further exasperated when the limited training data provides insufficient coverage of words in the test data. For example, approximately 39% of words in the benchmark UASpeech test set do not occur in the training data. This was found to produce a larger disparity in performance between seen and unseen words in dysarthric speech, and between impaired speakers with high and very low intelligibility, for end-to-end ASR systems including normal speech pre-trained models, than hybrid ASR systems that are trained using phonetic targets [3, 17, 30].

To this end, this paper explores a series of approaches to integrate SSL pre-trained wav2vec2.0 models into hybrid TDNN [32] and Conformer [33] ASR systems for dysarthric and elderly speech recognition in order to exploit the diversity and complementarity among them, including a) input feature fusion between standard acoustic front-ends and wav2vec2.0 speech representations; b) frame-level joint decoding [34, 3, 35] of TDNN systems separately trained using standard acoustic features alone and with additional wav2vec2.0 representations; and c) cross system multi-pass decoding [36] involving the TDNN or Conformer system N-best outputs to be rescored using domain adapted wav2vec2.0 models. Finally, wav2vec2.0 representations are utilized to facilitate neural network based acoustic-to-articulatory (A2A) inversion [35]. The resulting articulatory movement features are used to construct multi-modal dysarthric and elderly ASR systems.

Experiments conducted on the UASpeech [37] dysarthric and DementiaBank Pitt [38] elderly speech corpora suggest TDNN and Conformer ASR systems constructed by integrating domain adapted wav2vec2.0 models consistently outperform the standalone wav2vec2.0 models by statistically significant WER reductions of 8.22% and 3.43% absolute (26.71% and 15.88% relative) on the two tasks respectively. The lowest published WERs of 22.56% (52.53% on very low intelligibility, 39.09% on unseen words) and 18.17% are obtained on the UASpeech test set of 16 dysarthric speakers and the DementiaBank Pitt test set.

The main contributions of the paper are summarized below:

1) To the best of our knowledge, this paper presents the first work to systematically investigate the generalization capability of SSL pre-trained models for dysarthric and elderly speech recognition tasks including two objectives: a) minimize the overall WER with a focus on the most challenging portions of the very low (VL) intelligibility dysarthric or elderly speech data; and b) model generalization performance on unseen words that are unlikely to be sufficiently covered in practice for pathological and medical domain speech. In contrast, related previous works [30, 31, 24] analyzed their performance disparity across different speech pathology severity subsets but not their generalization to unseen words or sentences.

2) To address such generalization issues, this paper proposes novel approaches to integrate SSL pre-trained models into in-domain data constructed ASR systems. In contrast, prior researches focused on using stand-alone domain adapted pre-trained models [30, 31, 24].

3) This work also presents the first use of SSL based speech representations to cross-domain A2A inversion. In contrast, prior researches on cross-domain A2A inversion were conducted predominantly using non-SSL based domain adaptation methods [39, 35].

2 Pre-trained Wav2vec2.0 ASR Model

2.1 Model Architecture

The wav2vec2.0 model is an SSL pre-trained model that jointly learns latent contextualized speech representations and an inventory of discretized speech units. It consists of three components (left part of Fig. 1), including a speech feature encoder $f(\cdot)$ containing multi-layer 1D CNN based convolution blocks, a contextual transformer network $g(\cdot)$ and a quantization module $h(\cdot)$ consisting of 2 codebooks with 320 entries each. The input of wav2vec2.0 model is raw speech audio $\mathcal{X}$, which is first encoded into a sequence of representations $\mathcal{Z}=f(\mathcal{X})$ with a stride of 20 ms and a receptive field of 25 ms by the CNN-based speech feature encoder. These CNN extracted features are then fed into transformer based contextual encoder to produce contextualized speech representations $\mathcal{C}=g(\mathcal{Z})$. A logit $l\in\mathcal{R}^{2\times 320}$ is calculated from each vector $z_{i}\in\mathcal{Z}$ before fed into Gumbel-Softmax [40] operators to choose the best code among all entries of each codebook. The selected codes are concatenated together before a linear transformation is applied to produce the quantized vector $q_{i}$ during pretraining.

2.2 Model Pre-training And Fine-tuning

Wav2vec2.0 model is pre-trained using a weighted interpolation between the contrastive and diversity loss functions. For the speech representation of time step $t$ that is randomly masked, the contrastive loss based training aims to reproduce the quantized vector $q_{t}$ for the masked frame at the output of contextual transformer encoder, while discriminating against $K$ ``distractors'' representations ${\tilde{q}}\in{\tilde{Q}}$ that are randomly sampled from the other masked time steps within the same speech utterance. The contrastive loss function is defined as

[TABLE]

where ${\rm sim}(c_{t},q_{t})$ is the cosine similarity between the contextualized vector $c_{t}$ and the quantized representation $q_{t}$. $\kappa$ is the temperature and set $\widetilde{Q}$ consists of both the target quantization $q_{t}$ and $K$ distractors. The entropy based diversity loss is designed to regularize the use of the quantized codebook representations.

During the fine-tuning stage, a randomly initialized linear projection layer is added on top of contextual transformer network $g(\cdot)$ into $V$ classes representing the vocabulary entries of the task considered. The model then is fine-tuned with standard Connectionist Temporal Classification (CTC) loss [41] in a supervised fashion.

2.3 Speech Feature Extraction

The SSL pre-trained wav2vec2.0 model can be used as a standalone speech recognition system after task fine-tuning. Alternatively their contextual speech representations can be incorporated into various back-end target domain data trained ASR systems via feature fusion based on, e.g. TDNN or Conformer architectures, as considered in this paper. To this end, a Bottleneck module'' (circled in blue, top middle in Fig. 1) is introduced immediately above the highest positioned contextual transformer block to produce more compact speech representations111Alternative approaches of extracting bottleneck features, e.g. either from the CNN feature encoder outputs, contextual transformer network outputs, or a weighted average of the hidden layer outputs of each transformer block as suggested in [[28](#bib.bib28)] led to performance degradation.. The Bottleneck module'' contains a stack of four interleaving convolutional and feedforward layers: the first 1D transposed de-convolution CNN layer is used to change the stride length to 10ms to allow a frame rate synchronization with that of the back-end ASR systems; a fully connected (FC) block, which consists of a linear layer, rectified linear unit (ReLU) activation and dropout module, is used to change the dimension of extracted features from 1024 to 256222256 empirically outperforms other #dim, e.g. 128, 512 or 768. This is followed by a CNN layer and a final FC block which are used to revert the stride back to 20ms and restore the dimensionality to 1024. The final wav2vec2.0 speech representations are extracted from the first FC layer outputs and concatenated with the standard acoustic front-ends via feature fusion in hybrid TDNN or Conformer E2E ASR systems ((a) or (b) in Fig. 1).

3 Wav2vec2.0 features based A2A inversion

In this paper, the out-of-domain non-aged normal speech of the TaL dataset [42] containing parallel ultrasound, video data collected from 81 native speakers of English is used to construct mixture density networks (MDN) based neural A2A inversion models [39]. Due to the large acoustic domain mismatch, a direct cross-domain application of the A2A inversion model trained on normal, non-aged acoustic-articulatory parallel data to the elderly and dysarthric speech is problematic, as shown in the previous researches on cross-domain A2A inversion [39, 35]. To this end, the large acoustic domain mismatch is minimized using either baseline multi-level adaptive networks (MLAN) [3, 39, 35] or the above domain fine-tuned SSL wav2vec2.0 speech representations of Section 2.3.

The SSL based domain adaptation includes the following steps: 1) The pre-trained wav2vec2.0 model is fine-tuned on both out-of-domain normal speech and in-domain dysarthric or elderly speech simultaneously; 2) the resulting fine-tuned model is then used to produce more domain invariant speech representations that will exhibit smaller mismatch between these types of data. The resulting domain adapted speech representations are used in A2A inversion model training and articulatory feature generation for dysarthric or elderly audio data. More details of the MDN based A2A inversion models can be found in [39]. The articulatory movement features generated from dysarthric or elderly speech are concatenated with the standard acoustic features via feature fusion to construct multi-modal TDNN or Conformer systems ((c) or (d) in Fig. 1).

4 integration with TDNN/Conformer system

In order to exploit the diversity and complementarity among the SSL pre-trained wav2vec2.0 model and back-end TDNN or Conformer ASR systems, and to improve the generalization performance on unseen words as well as dysarthric speech of very low intelligibility, a series of integration approaches are explored in this paper.

1) input feature fusion is used to fuse standard acoustic frontends and domain adapted wav2vec2.0 speech representations via either feature concatenation before fed into the TDNN/Conformer system input layer, or optionally feature fusion at the TDNN hidden layer (connections (a) and (b) in Fig. 1, centre left).

2) time-synchronous frame-level joint decoding is used to combine two or more hybrid TDNN systems separately trained using either standard acoustic features alone, or with additional domain fine-tuned wav2vec2.0 representations, by a frame level linear interpolation of system specific acoustic scores (green dotted box (e) in Fig. 1, bottom right). Let $P_{a+w}(y_{t}|X_{1}^{1:T})$ and $P_{a}(y_{t}|X_{2}^{1:T})$ denote the frame level acoustic scores at $t$-th frame of TDNN systems constructed with or without additional wav2vec2.0 representations. Then the final frame-level score is obtained by

[TABLE]

where $\alpha$ and $\beta$ are the weights assigned to each system.

3) cross system multi-pass decoding involves a first decoding pass TDNN or Conformer system produced N-best outputs to be rescored using domain adapted wav2vec2.0 models. Consider a sequence of acoustic features $X$ and its corresponding TDNN or Conformer (cfm) decoded N-best recognition hypotheses in {$Y_{1},Y_{2},...,Y_{n}$}. Let ${\textbf{s}}^{w2v}=[s_{1}^{w2v},s_{2}^{w2v},...,s_{n}^{w2v}]^{T}$ and ${\textbf{s}}^{tdnn/cfm}=[s_{1}^{tdnn/cfm},s_{2}^{tdnn/cfm},...,s_{n}^{tdnn/cfm}]^{T}$ denote the N-best score vectors produced by the first pass TDNN/Conformer and the second pass wav2vec2.0 models. The final 1-best output produced in the second pass wav2vec2.0 rescoring is obtained by interpolating the system specific N-best entry level scores as

[TABLE]

where $\alpha$ and $\beta$ are the weights assigned to the second pass wav2vec2.0 and the first pass TDNN or Conformer systems. An example of two-pass rescoring based TDNN/Conformer and wav2vec2.0 system combination is shown in Fig. 1 (top right, red box (f)).

5 Experiments

Task Description: The UASpeech corpus is the largest publicly available dysarthric speech corpus [37] consisting of 16 dysarthric and 13 control speakers, and contains 155 common words and 300 uncommon words. The entire corpus is further divided into 3 subset blocks per speaker. The same set of 155 common words is used in all three blocks, while the uncommon words in each block are different. The data from Block 1 and 3 of all the 29 speakers are used as the training set, while the data of Block 2 collected from all the 16 dysarthric speakers serves as the test set. After removing excessive silence and speed perturbation based data augmentation [43], a total of 130.1 hours of audio data is used as the training set, while 9 hours of speech is used for performance evaluation. The DementiaBank Pitt [38] corpus contains about 33-hour audio recorded over interviews between the 292 elderly participants and clinical investigators. After silence stripping and data augmentation, the duration of training data is increased to 58.9 hours, while the development and evaluation sets contain 2.5 hours and 0.6 hours of audio respectively. The test data word coverage rates of UASpeech and DementiaBank are 61% and 98.7%.

Experimental Setup: The pre-trained wav2vec2.0 models on UASpeech and DementiaBank corpora are the Large Wav2Vec2.0 model333https://huggingface.co/facebook/wav2vec2-large-960h-lv60 and Conformer based Wav2Vec2.0 with relative position embeddings444https://huggingface.co/facebook/wav2vec2-conformer-rel-pos-large-960h-ft respectively which are pretrained with 60k hours of Libri-light data and fine-tuned with 960 hours of Librispeech data. During domain adaptation to UASpeech data, the wav2vec2.0 model is first fine-tuned with the non-augmented data to update all parameters except those of speech feature encoder, then further fine-tuned with the 130.1h augmented data to only update the parameters of the first 12 transformer blocks to ensure generalization to unseen words. The numbers of training epochs for the two fine-tuning stages are both 20. For the DementiaBank data, all the wav2vec2.0 parameters except those of the speech feature encoder are fine-tuned for 30 epochs on the 58.9h augmented data. The initial learning rate is 1e-5 and then linearly decays. As section 2.3 mentioned, the dimensionality of bottleneck wav2vec2.0 speech representations is 256 and that of ultrasound based articulatory features is 144. Experiment setups of back-end TDNN and Conformer systems follow [13].

Results on UASpeech: 1) Although there is no significant overall WER difference between the baseline TDNN with standard 40-dim filter-bank (FBK) features and the domain fine-tuned wav2vec2.0 model in Table 1 (Sys.1 vs. 4), the wav2vec2.0 model degrades the performance by 4.13% absolute on the unseen subset. 2) Fusing FBK front-ends with the fine-tuned wav2vec2.0 features (Fig. 1(a)-(b)) as TDNN system inputs (Sys. 5), or optionally further incorporating cross-domain A2A inverted features (Fig. 1(c)-(d)) for multimodal TDNN system training (Sys. 6), consistently reduce the overall WER by 0.3% and 1.03% absolute over the wav2vec2.0 model (Sys.5, 6 vs. 4). 3) Large overall WER reductions of 6.49-6.60% absolute (11.98-12.21% on unseen words) over the wav2vec2.0 model are obtained by frame-level joint decoding between the LHUC-SAT TDNN system trained using FBK features alone, and those trained with additional wav2vec2.0 representations and inverted articulatory features (Sys.7-9 vs. 4)555System weights empirically set as 3:2, 3:2 and 9:1:5 for Sys. 7, 8 and 9. 4) Further performance improvements are obtained by cross system multi-pass decoding (Sys.10, 11), which uses the wav2vec2.0 model to rescore the joint decoded TDNN systems outputs. The lowest published WER of 22.56% on 16 dysarthric speakers (Sys.10-11, 52.53% on very low intelligibility, 39.09% on unseen words) is obtained666System weights empirically set as 2:9 and 4:15 for Sys.10 and 11. 5) Similar experiments on Conformer systems also demonstrate the effectiveness of the wav2vec2.0 features via feature fusion (Sys.13 vs. 12), where smaller system combination improvements are obtained by multi-pass decoding (Sys.14 vs. 13), due to architecture similarity between the wav2vec2.0 and Conformer models leading to reduced cross system complementarity. Finally, the performance of our best systems (Sys.10, 11, Table 1) are contrasted against published state-of-the-art results on UASpeech in Table 2.

Results on DementiaBank: 1) Compared with the baseline TDNN system trained with 40-dim FBK features, the domain fine-tuned wav2vec2.0 model produces consistent and large WER reduction of 12.20% absolute (Sys.4 vs. 1). 2) Similar to the UASpeech experiments, the incorporation of wav2vec2.0 features alone, or plus their corresponding cross-domain articulatory features, produces significant WER reduction up to 2.52% (Sys.6 vs. 4). 3) Frame-level joint decoding between the LHUC-SAT TDNN system using FBK features (Sys. 3), and the TDNN systems using additional wav2vec2.0 features (Sys. 5) and articulatory features (sys. 6) consistently outperforms the adapted wav2vec2.0 model (sys. 4) by a significant WER reduction of 2.91% absolute (13.47% relative, Sys. 9 vs. 4). 4) After the cross system multi-pass decoding, the lowest published WER of 18.17% on DementiaBank Pitt test set is obtained (Sys.11). Using its final recognition outputs on evaluation set to extract textual features, a state-of-the-art speech recognition based AD detection [44] accuracy of 91.7% is also obtained. A comparison of WERs and AD detection accuracy on DementiaBank between previously published ASR systems and our system is shown in Table 3.

6 CONCLUSION

This paper explores a series of approaches to integrate cross-domain adapted SSL pre-trained models into TDNN and Conformer ASR systems for dysarthric and elderly speech including input feature fusion, frame-level joint decoding and multi-pass decoding. In addition, domain adapted wav2vec2.0 representations are utilized in acoustic-to-articulatory (A2A) inversion to construct multi-modal dysarthric and elderly ASR systems. The TDNN and Conformer ASR systems integrated domain adapted wav2vec2.0 models consistently outperform the standalone domain fine-tuned wav2vec2.0 models by statistically significant WER reductions of 8.22% and 3.43% absolute (26.71% and 15.88% relative) on the benchmark UASpeech and DementiaBank Pitt test set respectively, while producing the lowest published WERs of 22.56% and 18.17% on the two tasks. Future research will study the personalization of pre-trained ASR models for dysarthric and elderly speakers and pre-trained ASR model compression approaches to make it more practically applicable.

7 Acknowledgement

This research is supported by Hong Kong RGC GRF grant No. 14200220, 14200021, TRS T45-407/19N, Innovation & Technology Fund grant No. ITS/218/21, and National Natural Science Foundation of China (NSFC) Grant 62106255.