Extended study on using pretrained language models and YiSi-1 for machine translation evaluation
We present an extended study on using pretrained language models and YiSi-1 for machine translation evaluation. Although the recently proposed contextual embedding based metrics, YiSi-1, significantly outperform BLEU and other metrics in correlating with human judgment on translation quality, we hav...
Main Author: | |
---|---|
Format: | Article in Journal/Newspaper |
Language: | English |
Published: |
Association for Computational Linguistics
2020
|
Subjects: | |
Online Access: | https://nrc-publications.canada.ca/eng/view/ft/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 https://nrc-publications.canada.ca/eng/view/object/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 https://nrc-publications.canada.ca/fra/voir/objet/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 |
id |
ftnrccanada:oai:cisti-icist.nrc-cnrc.ca:cistinparc:cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 |
---|---|
record_format |
openpolar |
spelling |
ftnrccanada:oai:cisti-icist.nrc-cnrc.ca:cistinparc:cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 2023-07-16T03:59:17+02:00 Extended study on using pretrained language models and YiSi-1 for machine translation evaluation Lo, Chi-Kiu 2020-11-19 text https://nrc-publications.canada.ca/eng/view/ft/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 https://nrc-publications.canada.ca/eng/view/object/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 https://nrc-publications.canada.ca/fra/voir/objet/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 eng eng Association for Computational Linguistics Proceedings of the Fifth Conference on Machine Translation, Fifth Conference on Machine Translation, November 19-20, 2020, Online, Publication date: 2020-11-19, Pages: 895–902 Creative Commons, Attribution 4.0 Generic (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/) Creative Commons, Attribution 4.0 Générique (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/deed.fr) article 2020 ftnrccanada 2023-06-24T23:01:52Z We present an extended study on using pretrained language models and YiSi-1 for machine translation evaluation. Although the recently proposed contextual embedding based metrics, YiSi-1, significantly outperform BLEU and other metrics in correlating with human judgment on translation quality, we have yet to understand the full strength of using pretrained language models for machine translation evaluation. In this paper, we study YiSi-1’s correlation with human translation quality judgment by varying three major attributes (which architecture; which inter- mediate layer; whether it is monolingual or multilingual) of the pretrained language models. Results of the study show further improvements over YiSi-1 on the WMT 2019 Metrics shared task. We also describe the pretrained language model we trained for evaluating Inuktitut machine translation output. Peer reviewed: Yes NRC publication: Yes Article in Journal/Newspaper inuktitut National Research Council Canada: NRC Publications Archive |
institution |
Open Polar |
collection |
National Research Council Canada: NRC Publications Archive |
op_collection_id |
ftnrccanada |
language |
English |
description |
We present an extended study on using pretrained language models and YiSi-1 for machine translation evaluation. Although the recently proposed contextual embedding based metrics, YiSi-1, significantly outperform BLEU and other metrics in correlating with human judgment on translation quality, we have yet to understand the full strength of using pretrained language models for machine translation evaluation. In this paper, we study YiSi-1’s correlation with human translation quality judgment by varying three major attributes (which architecture; which inter- mediate layer; whether it is monolingual or multilingual) of the pretrained language models. Results of the study show further improvements over YiSi-1 on the WMT 2019 Metrics shared task. We also describe the pretrained language model we trained for evaluating Inuktitut machine translation output. Peer reviewed: Yes NRC publication: Yes |
format |
Article in Journal/Newspaper |
author |
Lo, Chi-Kiu |
spellingShingle |
Lo, Chi-Kiu Extended study on using pretrained language models and YiSi-1 for machine translation evaluation |
author_facet |
Lo, Chi-Kiu |
author_sort |
Lo, Chi-Kiu |
title |
Extended study on using pretrained language models and YiSi-1 for machine translation evaluation |
title_short |
Extended study on using pretrained language models and YiSi-1 for machine translation evaluation |
title_full |
Extended study on using pretrained language models and YiSi-1 for machine translation evaluation |
title_fullStr |
Extended study on using pretrained language models and YiSi-1 for machine translation evaluation |
title_full_unstemmed |
Extended study on using pretrained language models and YiSi-1 for machine translation evaluation |
title_sort |
extended study on using pretrained language models and yisi-1 for machine translation evaluation |
publisher |
Association for Computational Linguistics |
publishDate |
2020 |
url |
https://nrc-publications.canada.ca/eng/view/ft/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 https://nrc-publications.canada.ca/eng/view/object/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 https://nrc-publications.canada.ca/fra/voir/objet/?id=cd8d16f5-2b67-41aa-955f-84b4b6dc4e31 |
genre |
inuktitut |
genre_facet |
inuktitut |
op_relation |
Proceedings of the Fifth Conference on Machine Translation, Fifth Conference on Machine Translation, November 19-20, 2020, Online, Publication date: 2020-11-19, Pages: 895–902 |
op_rights |
Creative Commons, Attribution 4.0 Generic (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/) Creative Commons, Attribution 4.0 Générique (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/deed.fr) |
_version_ |
1771546878773559296 |