End-to-End Mispronunciation Detection and Diagnosis Using Transfer Learning

• Published in: Applied Sciences
• Main Authors: Linkai Peng, Yingming Gao, Rian Bao, Ya Li, Jinsong Zhang
• Format: Article in Journal/Newspaper
• Language: English
• Published: MDPI AG 2023
• Subjects: mispronunciation detection and diagnosis (MDD); computer-aided pronunciation training (CAPT); transfer learning; pretrained model; text modulation gate; Technology; T; Engineering (General). Civil engineering (General); TA1-2040; Biology (General); QH301-705.5; Physics; QC1-999; Chemistry; QD1-999; Arctic
• Online Access: https://doi.org/10.3390/app13116793; https://doaj.org/article/8a65852fffec4f2ba8a8235923ddd845

As an indispensable module of computer-aided pronunciation training (CAPT) systems, mispronunciation detection and diagnosis (MDD) techniques have attracted a lot of attention from academia and industry over the past decade. To train robust MDD models, this technique requires massive human-annotated speech recordings which are usually expensive and even hard to acquire. In this study, we propose to use transfer learning to tackle the problem of data scarcity from two aspects. First, from audio modality, we explore the use of the pretrained model wav2vec2.0 for MDD tasks by learning robust general acoustic representation. Second, from text modality, we explore transferring prior texts into MDD by learning associations between acoustic and textual modalities. We propose textual modulation gates that assign more importance to the relevant text information while suppressing irrelevant text information. Moreover, given the transcriptions, we propose an extra contrastive loss to reduce the difference of learning objectives between the phoneme recognition and MDD tasks. Conducting experiments on the L2-Arctic dataset showed that our wav2vec2.0 based models outperformed the conventional methods. The proposed textual modulation gate and contrastive loss further improved the F1-score by more than 2.88% and our best model achieved an F1-score of 61.75%.