End-to-End Automatic Speech Recognition with Deep Mutual Learning
This paper is the first study to apply deep mutual learning (DML) to end-to-end ASR models. In DML, multiple models are trained simultaneously and collaboratively by mimicking each other throughout the training process, which helps to attain the global optimum and prevent models from making over-con...
Main Authors: | , , , , |
---|---|
Format: | Text |
Language: | unknown |
Published: |
2021
|
Subjects: | |
Online Access: | http://arxiv.org/abs/2102.08154 |
id |
ftarxivpreprints:oai:arXiv.org:2102.08154 |
---|---|
record_format |
openpolar |
spelling |
ftarxivpreprints:oai:arXiv.org:2102.08154 2023-09-05T13:19:04+02:00 End-to-End Automatic Speech Recognition with Deep Mutual Learning Masumura, Ryo Ihori, Mana Takashima, Akihiko Tanaka, Tomohiro Ashihara, Takanori 2021-02-16 http://arxiv.org/abs/2102.08154 unknown http://arxiv.org/abs/2102.08154 Computer Science - Computation and Language Computer Science - Machine Learning text 2021 ftarxivpreprints 2023-08-16T16:20:44Z This paper is the first study to apply deep mutual learning (DML) to end-to-end ASR models. In DML, multiple models are trained simultaneously and collaboratively by mimicking each other throughout the training process, which helps to attain the global optimum and prevent models from making over-confident predictions. While previous studies applied DML to simple multi-class classification problems, there are no studies that have used it on more complex sequence-to-sequence mapping problems. For this reason, this paper presents a method to apply DML to state-of-the-art Transformer-based end-to-end ASR models. In particular, we propose to combine DML with recent representative training techniques. i.e., label smoothing, scheduled sampling, and SpecAugment, each of which are essential for powerful end-to-end ASR models. We expect that these training techniques work well with DML because DML has complementary characteristics. We experimented with two setups for Japanese ASR tasks: large-scale modeling and compact modeling. We demonstrate that DML improves the ASR performance of both modeling setups compared with conventional learning methods including knowledge distillation. We also show that combining DML with the existing training techniques effectively improves ASR performance. Comment: Accepted at Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2020, pp.632-637 Text DML ArXiv.org (Cornell University Library) Pacific |
institution |
Open Polar |
collection |
ArXiv.org (Cornell University Library) |
op_collection_id |
ftarxivpreprints |
language |
unknown |
topic |
Computer Science - Computation and Language Computer Science - Machine Learning |
spellingShingle |
Computer Science - Computation and Language Computer Science - Machine Learning Masumura, Ryo Ihori, Mana Takashima, Akihiko Tanaka, Tomohiro Ashihara, Takanori End-to-End Automatic Speech Recognition with Deep Mutual Learning |
topic_facet |
Computer Science - Computation and Language Computer Science - Machine Learning |
description |
This paper is the first study to apply deep mutual learning (DML) to end-to-end ASR models. In DML, multiple models are trained simultaneously and collaboratively by mimicking each other throughout the training process, which helps to attain the global optimum and prevent models from making over-confident predictions. While previous studies applied DML to simple multi-class classification problems, there are no studies that have used it on more complex sequence-to-sequence mapping problems. For this reason, this paper presents a method to apply DML to state-of-the-art Transformer-based end-to-end ASR models. In particular, we propose to combine DML with recent representative training techniques. i.e., label smoothing, scheduled sampling, and SpecAugment, each of which are essential for powerful end-to-end ASR models. We expect that these training techniques work well with DML because DML has complementary characteristics. We experimented with two setups for Japanese ASR tasks: large-scale modeling and compact modeling. We demonstrate that DML improves the ASR performance of both modeling setups compared with conventional learning methods including knowledge distillation. We also show that combining DML with the existing training techniques effectively improves ASR performance. Comment: Accepted at Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2020, pp.632-637 |
format |
Text |
author |
Masumura, Ryo Ihori, Mana Takashima, Akihiko Tanaka, Tomohiro Ashihara, Takanori |
author_facet |
Masumura, Ryo Ihori, Mana Takashima, Akihiko Tanaka, Tomohiro Ashihara, Takanori |
author_sort |
Masumura, Ryo |
title |
End-to-End Automatic Speech Recognition with Deep Mutual Learning |
title_short |
End-to-End Automatic Speech Recognition with Deep Mutual Learning |
title_full |
End-to-End Automatic Speech Recognition with Deep Mutual Learning |
title_fullStr |
End-to-End Automatic Speech Recognition with Deep Mutual Learning |
title_full_unstemmed |
End-to-End Automatic Speech Recognition with Deep Mutual Learning |
title_sort |
end-to-end automatic speech recognition with deep mutual learning |
publishDate |
2021 |
url |
http://arxiv.org/abs/2102.08154 |
geographic |
Pacific |
geographic_facet |
Pacific |
genre |
DML |
genre_facet |
DML |
op_relation |
http://arxiv.org/abs/2102.08154 |
_version_ |
1776199879038599168 |