Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems
We present a collection of expanded human annotations of the WMT20 English-Inuktitut machine translation shared task, covering the Nunavut Hansard portion of the dataset. Additionally, we recompute News rankings to take into account the completed set of human annotations and certain irregularities i...
| Main Authors: | , |
|---|---|
| Format: | Article in Journal/Newspaper |
| Language: | English |
| Published: |
Association for Computational Linguistics
2022
|
| Subjects: | |
| Online Access: | https://nrc-publications.canada.ca/eng/view/ft/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 https://nrc-publications.canada.ca/eng/view/object/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 https://nrc-publications.canada.ca/fra/voir/objet/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 |
| _version_ | 1821561004097536000 |
|---|---|
| author | Knowles, Rebecca Lo, Chi-kiu |
| author_facet | Knowles, Rebecca Lo, Chi-kiu |
| author_sort | Knowles, Rebecca |
| collection | National Research Council Canada: NRC Publications Archive |
| description | We present a collection of expanded human annotations of the WMT20 English-Inuktitut machine translation shared task, covering the Nunavut Hansard portion of the dataset. Additionally, we recompute News rankings to take into account the completed set of human annotations and certain irregularities in the annotation task construction. We show the effect of these changes on the downstream task of the evaluation of automatic metrics. Finally, we demonstrate that character-level metrics correlate well with human judgments for the task of automatically evaluating translation into this polysynthetic language. Peer reviewed: Yes NRC publication: Yes |
| format | Article in Journal/Newspaper |
| genre | inuktitut Nunavut |
| genre_facet | inuktitut Nunavut |
| geographic | Nunavut |
| geographic_facet | Nunavut |
| id | ftnrccanada:oai:cisti-icist.nrc-cnrc.ca:cistinparc:d52a4c28-7e45-48ef-a575-998fc349eeb8 |
| institution | Open Polar |
| language | English |
| op_collection_id | ftnrccanada |
| op_relation | Proceedings of the Seventh Conference on Machine Translation (WMT), Seventh Conference on Machine Translation (WMT), December 2022, Abu Dhabi, United Arab Emirates (Hybrid), Publication date: 2022-12, Pages: 140–153 |
| op_rights | Creative Commons, Attribution 4.0 Generic (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/) Creative Commons, Attribution 4.0 Générique (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/deed.fr) |
| publishDate | 2022 |
| publisher | Association for Computational Linguistics |
| record_format | openpolar |
| spelling | ftnrccanada:oai:cisti-icist.nrc-cnrc.ca:cistinparc:d52a4c28-7e45-48ef-a575-998fc349eeb8 2025-01-16T22:44:23+00:00 Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems Knowles, Rebecca Lo, Chi-kiu 2022-12 text https://nrc-publications.canada.ca/eng/view/ft/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 https://nrc-publications.canada.ca/eng/view/object/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 https://nrc-publications.canada.ca/fra/voir/objet/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 eng eng Association for Computational Linguistics Proceedings of the Seventh Conference on Machine Translation (WMT), Seventh Conference on Machine Translation (WMT), December 2022, Abu Dhabi, United Arab Emirates (Hybrid), Publication date: 2022-12, Pages: 140–153 Creative Commons, Attribution 4.0 Generic (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/) Creative Commons, Attribution 4.0 Générique (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/deed.fr) article 2022 ftnrccanada 2023-03-12T00:01:33Z We present a collection of expanded human annotations of the WMT20 English-Inuktitut machine translation shared task, covering the Nunavut Hansard portion of the dataset. Additionally, we recompute News rankings to take into account the completed set of human annotations and certain irregularities in the annotation task construction. We show the effect of these changes on the downstream task of the evaluation of automatic metrics. Finally, we demonstrate that character-level metrics correlate well with human judgments for the task of automatically evaluating translation into this polysynthetic language. Peer reviewed: Yes NRC publication: Yes Article in Journal/Newspaper inuktitut Nunavut National Research Council Canada: NRC Publications Archive Nunavut |
| spellingShingle | Knowles, Rebecca Lo, Chi-kiu Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems |
| title | Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems |
| title_full | Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems |
| title_fullStr | Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems |
| title_full_unstemmed | Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems |
| title_short | Test set sampling affects system rankings: expanded human evaluation of WMT20 English-Inuktitut systems |
| title_sort | test set sampling affects system rankings: expanded human evaluation of wmt20 english-inuktitut systems |
| url | https://nrc-publications.canada.ca/eng/view/ft/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 https://nrc-publications.canada.ca/eng/view/object/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 https://nrc-publications.canada.ca/fra/voir/objet/?id=d52a4c28-7e45-48ef-a575-998fc349eeb8 |