ORCA: A Challenging Benchmark for Arabic Language Understanding
Due to their crucial role in all NLP, several benchmarks have been proposed to evaluate pretrained language models. In spite of these efforts, no public benchmark of diverse nature currently exists for evaluation of Arabic. This makes it challenging to measure progress for both Arabic and multilingu...
Main Authors: | , , |
---|---|
Format: | Text |
Language: | unknown |
Published: |
2022
|
Subjects: | |
Online Access: | http://arxiv.org/abs/2212.10758 |
id |
ftarxivpreprints:oai:arXiv.org:2212.10758 |
---|---|
record_format |
openpolar |
spelling |
ftarxivpreprints:oai:arXiv.org:2212.10758 2023-09-05T13:22:19+02:00 ORCA: A Challenging Benchmark for Arabic Language Understanding Elmadany, AbdelRahim Nagoudi, El Moatez Billah Abdul-Mageed, Muhammad 2022-12-20 http://arxiv.org/abs/2212.10758 unknown http://arxiv.org/abs/2212.10758 Computer Science - Computation and Language Computer Science - Artificial Intelligence text 2022 ftarxivpreprints 2023-08-16T17:27:14Z Due to their crucial role in all NLP, several benchmarks have been proposed to evaluate pretrained language models. In spite of these efforts, no public benchmark of diverse nature currently exists for evaluation of Arabic. This makes it challenging to measure progress for both Arabic and multilingual language models. This challenge is compounded by the fact that any benchmark targeting Arabic needs to take into account the fact that Arabic is not a single language but rather a collection of languages and varieties. In this work, we introduce ORCA, a publicly available benchmark for Arabic language understanding evaluation. ORCA is carefully constructed to cover diverse Arabic varieties and a wide range of challenging Arabic understanding tasks exploiting 60 different datasets across seven NLU task clusters. To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models. We also provide a public leaderboard with a unified single-number evaluation metric (ORCA score) to facilitate future research. Comment: All authors contributed equally. Accepted at ACL 2023, Toronto, Canada Text Orca ArXiv.org (Cornell University Library) Canada |
institution |
Open Polar |
collection |
ArXiv.org (Cornell University Library) |
op_collection_id |
ftarxivpreprints |
language |
unknown |
topic |
Computer Science - Computation and Language Computer Science - Artificial Intelligence |
spellingShingle |
Computer Science - Computation and Language Computer Science - Artificial Intelligence Elmadany, AbdelRahim Nagoudi, El Moatez Billah Abdul-Mageed, Muhammad ORCA: A Challenging Benchmark for Arabic Language Understanding |
topic_facet |
Computer Science - Computation and Language Computer Science - Artificial Intelligence |
description |
Due to their crucial role in all NLP, several benchmarks have been proposed to evaluate pretrained language models. In spite of these efforts, no public benchmark of diverse nature currently exists for evaluation of Arabic. This makes it challenging to measure progress for both Arabic and multilingual language models. This challenge is compounded by the fact that any benchmark targeting Arabic needs to take into account the fact that Arabic is not a single language but rather a collection of languages and varieties. In this work, we introduce ORCA, a publicly available benchmark for Arabic language understanding evaluation. ORCA is carefully constructed to cover diverse Arabic varieties and a wide range of challenging Arabic understanding tasks exploiting 60 different datasets across seven NLU task clusters. To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models. We also provide a public leaderboard with a unified single-number evaluation metric (ORCA score) to facilitate future research. Comment: All authors contributed equally. Accepted at ACL 2023, Toronto, Canada |
format |
Text |
author |
Elmadany, AbdelRahim Nagoudi, El Moatez Billah Abdul-Mageed, Muhammad |
author_facet |
Elmadany, AbdelRahim Nagoudi, El Moatez Billah Abdul-Mageed, Muhammad |
author_sort |
Elmadany, AbdelRahim |
title |
ORCA: A Challenging Benchmark for Arabic Language Understanding |
title_short |
ORCA: A Challenging Benchmark for Arabic Language Understanding |
title_full |
ORCA: A Challenging Benchmark for Arabic Language Understanding |
title_fullStr |
ORCA: A Challenging Benchmark for Arabic Language Understanding |
title_full_unstemmed |
ORCA: A Challenging Benchmark for Arabic Language Understanding |
title_sort |
orca: a challenging benchmark for arabic language understanding |
publishDate |
2022 |
url |
http://arxiv.org/abs/2212.10758 |
geographic |
Canada |
geographic_facet |
Canada |
genre |
Orca |
genre_facet |
Orca |
op_relation |
http://arxiv.org/abs/2212.10758 |
_version_ |
1776202843900870656 |