Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms
Recently, end-to-end (E2E) models, which allow to take spectral vector sequences of L2 (second-language) learners' utterances as input and produce the corresponding phone-level sequences as output, have attracted much research attention in developing mispronunciation detection (MD) systems. How...
Main Authors: | , , |
---|---|
Format: | Article in Journal/Newspaper |
Language: | unknown |
Published: |
arXiv
2021
|
Subjects: | |
Online Access: | https://dx.doi.org/10.48550/arxiv.2110.08731 https://arxiv.org/abs/2110.08731 |
id |
ftdatacite:10.48550/arxiv.2110.08731 |
---|---|
record_format |
openpolar |
spelling |
ftdatacite:10.48550/arxiv.2110.08731 2023-05-15T15:11:22+02:00 Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms Lo, Tien-Hong Sung, Yao-Ting Chen, Berlin 2021 https://dx.doi.org/10.48550/arxiv.2110.08731 https://arxiv.org/abs/2110.08731 unknown arXiv Creative Commons Attribution 4.0 International https://creativecommons.org/licenses/by/4.0/legalcode cc-by-4.0 CC-BY Sound cs.SD Artificial Intelligence cs.AI Audio and Speech Processing eess.AS FOS Computer and information sciences FOS Electrical engineering, electronic engineering, information engineering Article CreativeWork article Preprint 2021 ftdatacite https://doi.org/10.48550/arxiv.2110.08731 2022-03-10T13:50:05Z Recently, end-to-end (E2E) models, which allow to take spectral vector sequences of L2 (second-language) learners' utterances as input and produce the corresponding phone-level sequences as output, have attracted much research attention in developing mispronunciation detection (MD) systems. However, due to the lack of sufficient labeled speech data of L2 speakers for model estimation, E2E MD models are prone to overfitting in relation to conventional ones that are built on DNN-HMM acoustic models. To alleviate this critical issue, we in this paper propose two modeling strategies to enhance the discrimination capability of E2E MD models, each of which can implicitly leverage the phonetic and phonological traits encoded in a pretrained acoustic model and contained within reference transcripts of the training data, respectively. The first one is input augmentation, which aims to distill knowledge about phonetic discrimination from a DNN-HMM acoustic model. The second one is label augmentation, which manages to capture more phonological patterns from the transcripts of training data. A series of empirical experiments conducted on the L2-ARCTIC English dataset seem to confirm the efficacy of our E2E MD model when compared to some top-of-the-line E2E MD models and a classic pronunciation-scoring based method built on a DNN-HMM acoustic model. : 7 pages, 2 figures, 4 tables, accepted to Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2021) Article in Journal/Newspaper Arctic DataCite Metadata Store (German National Library of Science and Technology) Arctic Pacific |
institution |
Open Polar |
collection |
DataCite Metadata Store (German National Library of Science and Technology) |
op_collection_id |
ftdatacite |
language |
unknown |
topic |
Sound cs.SD Artificial Intelligence cs.AI Audio and Speech Processing eess.AS FOS Computer and information sciences FOS Electrical engineering, electronic engineering, information engineering |
spellingShingle |
Sound cs.SD Artificial Intelligence cs.AI Audio and Speech Processing eess.AS FOS Computer and information sciences FOS Electrical engineering, electronic engineering, information engineering Lo, Tien-Hong Sung, Yao-Ting Chen, Berlin Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms |
topic_facet |
Sound cs.SD Artificial Intelligence cs.AI Audio and Speech Processing eess.AS FOS Computer and information sciences FOS Electrical engineering, electronic engineering, information engineering |
description |
Recently, end-to-end (E2E) models, which allow to take spectral vector sequences of L2 (second-language) learners' utterances as input and produce the corresponding phone-level sequences as output, have attracted much research attention in developing mispronunciation detection (MD) systems. However, due to the lack of sufficient labeled speech data of L2 speakers for model estimation, E2E MD models are prone to overfitting in relation to conventional ones that are built on DNN-HMM acoustic models. To alleviate this critical issue, we in this paper propose two modeling strategies to enhance the discrimination capability of E2E MD models, each of which can implicitly leverage the phonetic and phonological traits encoded in a pretrained acoustic model and contained within reference transcripts of the training data, respectively. The first one is input augmentation, which aims to distill knowledge about phonetic discrimination from a DNN-HMM acoustic model. The second one is label augmentation, which manages to capture more phonological patterns from the transcripts of training data. A series of empirical experiments conducted on the L2-ARCTIC English dataset seem to confirm the efficacy of our E2E MD model when compared to some top-of-the-line E2E MD models and a classic pronunciation-scoring based method built on a DNN-HMM acoustic model. : 7 pages, 2 figures, 4 tables, accepted to Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2021) |
format |
Article in Journal/Newspaper |
author |
Lo, Tien-Hong Sung, Yao-Ting Chen, Berlin |
author_facet |
Lo, Tien-Hong Sung, Yao-Ting Chen, Berlin |
author_sort |
Lo, Tien-Hong |
title |
Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms |
title_short |
Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms |
title_full |
Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms |
title_fullStr |
Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms |
title_full_unstemmed |
Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation Mechanisms |
title_sort |
improving end-to-end modeling for mispronunciation detection with effective augmentation mechanisms |
publisher |
arXiv |
publishDate |
2021 |
url |
https://dx.doi.org/10.48550/arxiv.2110.08731 https://arxiv.org/abs/2110.08731 |
geographic |
Arctic Pacific |
geographic_facet |
Arctic Pacific |
genre |
Arctic |
genre_facet |
Arctic |
op_rights |
Creative Commons Attribution 4.0 International https://creativecommons.org/licenses/by/4.0/legalcode cc-by-4.0 |
op_rightsnorm |
CC-BY |
op_doi |
https://doi.org/10.48550/arxiv.2110.08731 |
_version_ |
1766342234917044224 |