SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages
This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being...
Published in: | Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology |
---|---|
Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
Other Authors: | |
Format: | Conference Object |
Language: | English |
Published: |
2022
|
Subjects: | |
Online Access: | http://hdl.handle.net/10138/340752 |
id |
ftunivhelsihelda:oai:helda.helsinki.fi:10138/340752 |
---|---|
record_format |
openpolar |
institution |
Open Polar |
collection |
HELDA – University of Helsinki Open Repository |
op_collection_id |
ftunivhelsihelda |
language |
English |
topic |
113 Computer and information sciences |
spellingShingle |
113 Computer and information sciences Pimentel, Tiago Ryskina, Maria Mielke, Sabrina J. Wu, Shijie Chodroff, Eleanor Leonard, Brian Nicolai, Garett Ate, Yustinus Ghanggo Khalifa, Salam Habash, Nizar El-Khaissi, Charbel Goldman, Omer Gasser, Michael Lane, William Coler, Matt Oncevay, Arturo Montoya Samame, Jaime Rafael Silva Villegas, Gema Celeste Ek, Adam Bernardy, Jean-Philippe Shcherbakov, Andrey Bayyr-ool, Aziyana Sheifer, Karina Ganieva, Sofya Plugaryov, Matvey Klyachko, Elena Salehi, Ali Krizhanovsky, Andrew Krizhanovsky, Natalia Vania, Clara Ivanova, Sardana Salchak, Aelita Straughn, Christopher Liu, Zoey Washington, Jonathan Ataman, Duygu Kieraś, Witold Woliński, Marcin Suhardijanto, Totok Stoehr, Niklas Nuriah, Zahroh Ratan, Shyam Tyers, Francis M. Ponti, Edoardo M. Aiton, Grant Hatcher, Richard J. Prud'hommeaux, Emily Kumar, Ritesh Hulden, Mans Barta, Botond Lakatos, Dorina Szolnok, Gábor Ács, Judit Raj, Mohit Yarowsky, David Cotterell, Ryan Ambridge, Ben Vylomova, Ekaterina SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages |
topic_facet |
113 Computer and information sciences |
description |
This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, Võro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Asháninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems' predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving >90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems' performance on previously unseen lemmas. Peer reviewed |
author2 |
Department of Computer Science |
format |
Conference Object |
author |
Pimentel, Tiago Ryskina, Maria Mielke, Sabrina J. Wu, Shijie Chodroff, Eleanor Leonard, Brian Nicolai, Garett Ate, Yustinus Ghanggo Khalifa, Salam Habash, Nizar El-Khaissi, Charbel Goldman, Omer Gasser, Michael Lane, William Coler, Matt Oncevay, Arturo Montoya Samame, Jaime Rafael Silva Villegas, Gema Celeste Ek, Adam Bernardy, Jean-Philippe Shcherbakov, Andrey Bayyr-ool, Aziyana Sheifer, Karina Ganieva, Sofya Plugaryov, Matvey Klyachko, Elena Salehi, Ali Krizhanovsky, Andrew Krizhanovsky, Natalia Vania, Clara Ivanova, Sardana Salchak, Aelita Straughn, Christopher Liu, Zoey Washington, Jonathan Ataman, Duygu Kieraś, Witold Woliński, Marcin Suhardijanto, Totok Stoehr, Niklas Nuriah, Zahroh Ratan, Shyam Tyers, Francis M. Ponti, Edoardo M. Aiton, Grant Hatcher, Richard J. Prud'hommeaux, Emily Kumar, Ritesh Hulden, Mans Barta, Botond Lakatos, Dorina Szolnok, Gábor Ács, Judit Raj, Mohit Yarowsky, David Cotterell, Ryan Ambridge, Ben Vylomova, Ekaterina |
author_facet |
Pimentel, Tiago Ryskina, Maria Mielke, Sabrina J. Wu, Shijie Chodroff, Eleanor Leonard, Brian Nicolai, Garett Ate, Yustinus Ghanggo Khalifa, Salam Habash, Nizar El-Khaissi, Charbel Goldman, Omer Gasser, Michael Lane, William Coler, Matt Oncevay, Arturo Montoya Samame, Jaime Rafael Silva Villegas, Gema Celeste Ek, Adam Bernardy, Jean-Philippe Shcherbakov, Andrey Bayyr-ool, Aziyana Sheifer, Karina Ganieva, Sofya Plugaryov, Matvey Klyachko, Elena Salehi, Ali Krizhanovsky, Andrew Krizhanovsky, Natalia Vania, Clara Ivanova, Sardana Salchak, Aelita Straughn, Christopher Liu, Zoey Washington, Jonathan Ataman, Duygu Kieraś, Witold Woliński, Marcin Suhardijanto, Totok Stoehr, Niklas Nuriah, Zahroh Ratan, Shyam Tyers, Francis M. Ponti, Edoardo M. Aiton, Grant Hatcher, Richard J. Prud'hommeaux, Emily Kumar, Ritesh Hulden, Mans Barta, Botond Lakatos, Dorina Szolnok, Gábor Ács, Judit Raj, Mohit Yarowsky, David Cotterell, Ryan Ambridge, Ben Vylomova, Ekaterina |
author_sort |
Pimentel, Tiago |
title |
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages |
title_short |
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages |
title_full |
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages |
title_fullStr |
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages |
title_full_unstemmed |
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages |
title_sort |
sigmorphon 2021 shared task on morphological reinflection: generalization across languages |
publishDate |
2022 |
url |
http://hdl.handle.net/10138/340752 |
long_lat |
ENVELOPE(-60.783,-60.783,-62.450,-62.450) ENVELOPE(132.817,132.817,59.683,59.683) |
geographic |
Aymara Evenki Sakha |
geographic_facet |
Aymara Evenki Sakha |
genre |
Chukchi Evenki Itelmen karelian Sakha |
genre_facet |
Chukchi Evenki Itelmen karelian Sakha |
op_relation |
10.18653/v1/2021.sigmorphon-1.25 Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology 978-1-954085-62-6 Pimentel , T , Ryskina , M , Mielke , S J , Wu , S , Chodroff , E , Leonard , B , Nicolai , G , Ate , Y G , Khalifa , S , Habash , N , El-Khaissi , C , Goldman , O , Gasser , M , Lane , W , Coler , M , Oncevay , A , Montoya Samame , J R , Silva Villegas , G C , Ek , A , Bernardy , J-P , Shcherbakov , A , Bayyr-ool , A , Sheifer , K , Ganieva , S , Plugaryov , M , Klyachko , E , Salehi , A , Krizhanovsky , A , Krizhanovsky , N , Vania , C , Ivanova , S , Salchak , A , Straughn , C , Liu , Z , Washington , J , Ataman , D , Kieraś , W , Woliński , M , Suhardijanto , T , Stoehr , N , Nuriah , Z , Ratan , S , Tyers , F M , Ponti , E M , Aiton , G , Hatcher , R J , Prud'hommeaux , E , Kumar , R , Hulden , M , Barta , B , Lakatos , D , Szolnok , G , Ács , J , Raj , M , Yarowsky , D , Cotterell , R , Ambridge , B & Vylomova , E 2021 , SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages . in Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology . The Association for Computational Linguistics , pp. 229–259 , SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , 05/08/2021 . https://doi.org/10.18653/v1/2021.sigmorphon-1.25 workshop efb6b952-8a5d-4c3b-adae-95460ec414b2 http://hdl.handle.net/10138/340752 |
op_rights |
cc_by openAccess info:eu-repo/semantics/openAccess |
container_title |
Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology |
container_start_page |
229 |
op_container_end_page |
259 |
_version_ |
1787423724401590272 |
spelling |
ftunivhelsihelda:oai:helda.helsinki.fi:10138/340752 2024-01-07T09:42:40+01:00 SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages Pimentel, Tiago Ryskina, Maria Mielke, Sabrina J. Wu, Shijie Chodroff, Eleanor Leonard, Brian Nicolai, Garett Ate, Yustinus Ghanggo Khalifa, Salam Habash, Nizar El-Khaissi, Charbel Goldman, Omer Gasser, Michael Lane, William Coler, Matt Oncevay, Arturo Montoya Samame, Jaime Rafael Silva Villegas, Gema Celeste Ek, Adam Bernardy, Jean-Philippe Shcherbakov, Andrey Bayyr-ool, Aziyana Sheifer, Karina Ganieva, Sofya Plugaryov, Matvey Klyachko, Elena Salehi, Ali Krizhanovsky, Andrew Krizhanovsky, Natalia Vania, Clara Ivanova, Sardana Salchak, Aelita Straughn, Christopher Liu, Zoey Washington, Jonathan Ataman, Duygu Kieraś, Witold Woliński, Marcin Suhardijanto, Totok Stoehr, Niklas Nuriah, Zahroh Ratan, Shyam Tyers, Francis M. Ponti, Edoardo M. Aiton, Grant Hatcher, Richard J. Prud'hommeaux, Emily Kumar, Ritesh Hulden, Mans Barta, Botond Lakatos, Dorina Szolnok, Gábor Ács, Judit Raj, Mohit Yarowsky, David Cotterell, Ryan Ambridge, Ben Vylomova, Ekaterina Department of Computer Science 2022-02-22T08:45:01Z application/pdf http://hdl.handle.net/10138/340752 eng eng 10.18653/v1/2021.sigmorphon-1.25 Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology 978-1-954085-62-6 Pimentel , T , Ryskina , M , Mielke , S J , Wu , S , Chodroff , E , Leonard , B , Nicolai , G , Ate , Y G , Khalifa , S , Habash , N , El-Khaissi , C , Goldman , O , Gasser , M , Lane , W , Coler , M , Oncevay , A , Montoya Samame , J R , Silva Villegas , G C , Ek , A , Bernardy , J-P , Shcherbakov , A , Bayyr-ool , A , Sheifer , K , Ganieva , S , Plugaryov , M , Klyachko , E , Salehi , A , Krizhanovsky , A , Krizhanovsky , N , Vania , C , Ivanova , S , Salchak , A , Straughn , C , Liu , Z , Washington , J , Ataman , D , Kieraś , W , Woliński , M , Suhardijanto , T , Stoehr , N , Nuriah , Z , Ratan , S , Tyers , F M , Ponti , E M , Aiton , G , Hatcher , R J , Prud'hommeaux , E , Kumar , R , Hulden , M , Barta , B , Lakatos , D , Szolnok , G , Ács , J , Raj , M , Yarowsky , D , Cotterell , R , Ambridge , B & Vylomova , E 2021 , SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages . in Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology . The Association for Computational Linguistics , pp. 229–259 , SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , 05/08/2021 . https://doi.org/10.18653/v1/2021.sigmorphon-1.25 workshop efb6b952-8a5d-4c3b-adae-95460ec414b2 http://hdl.handle.net/10138/340752 cc_by openAccess info:eu-repo/semantics/openAccess 113 Computer and information sciences Conference contribution publishedVersion 2022 ftunivhelsihelda 2023-12-14T00:15:21Z This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, Võro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Asháninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems' predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving >90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems' performance on previously unseen lemmas. Peer reviewed Conference Object Chukchi Evenki Itelmen karelian Sakha HELDA – University of Helsinki Open Repository Aymara ENVELOPE(-60.783,-60.783,-62.450,-62.450) Evenki ENVELOPE(132.817,132.817,59.683,59.683) Sakha Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology 229 259 |