FarsTail: A Persian Natural Language Inference Dataset
Natural language inference (NLI) is known as one of the central tasks in natural language processing (NLP) which encapsulates many fundamental aspects of language understanding. With the considerable achievements of data-hungry deep learning methods in NLP tasks, a great amount of effort has been de...
Published in: | Soft Computing |
---|---|
Main Authors: | , , , , , |
Format: | Text |
Language: | unknown |
Published: |
2020
|
Subjects: | |
Online Access: | http://arxiv.org/abs/2009.08820 https://doi.org/10.1007/s00500-023-08959-3 |
id |
ftarxivpreprints:oai:arXiv.org:2009.08820 |
---|---|
record_format |
openpolar |
spelling |
ftarxivpreprints:oai:arXiv.org:2009.08820 2023-09-05T13:19:07+02:00 FarsTail: A Persian Natural Language Inference Dataset Amirkhani, Hossein AzariJafari, Mohammad Pourjafari, Zohreh Faridan-Jahromi, Soroush Kouhkan, Zeinab Amirak, Azadeh 2020-09-18 http://arxiv.org/abs/2009.08820 https://doi.org/10.1007/s00500-023-08959-3 unknown http://arxiv.org/abs/2009.08820 Soft Computing (2023) doi:10.1007/s00500-023-08959-3 Computer Science - Computation and Language text 2020 ftarxivpreprints https://doi.org/10.1007/s00500-023-08959-3 2023-08-16T16:05:46Z Natural language inference (NLI) is known as one of the central tasks in natural language processing (NLP) which encapsulates many fundamental aspects of language understanding. With the considerable achievements of data-hungry deep learning methods in NLP tasks, a great amount of effort has been devoted to develop more diverse datasets for different languages. In this paper, we present a new dataset for the NLI task in the Persian language, also known as Farsi, which is one of the dominant languages in the Middle East. This dataset, named FarsTail, includes 10,367 samples which are provided in both the Persian language as well as the indexed format to be useful for non-Persian researchers. The samples are generated from 3,539 multiple-choice questions with the least amount of annotator interventions in a way similar to the SciTail dataset. A carefully designed multi-step process is adopted to ensure the quality of the dataset. We also present the results of traditional and state-of-the-art methods on FarsTail including different embedding methods such as word2vec, fastText, ELMo, BERT, and LASER, as well as different modeling approaches such as DecompAtt, ESIM, HBMP, and ULMFiT to provide a solid baseline for the future research. The best obtained test accuracy is 83.38% which shows that there is a big room for improving the current methods to be useful for real-world NLP applications in different languages. We also investigate the extent to which the models exploit superficial clues, also known as dataset biases, in FarsTail, and partition the test set into easy and hard subsets according to the success of biased models. The dataset is available at https://github.com/dml-qom/FarsTail Text DML ArXiv.org (Cornell University Library) Soft Computing |
institution |
Open Polar |
collection |
ArXiv.org (Cornell University Library) |
op_collection_id |
ftarxivpreprints |
language |
unknown |
topic |
Computer Science - Computation and Language |
spellingShingle |
Computer Science - Computation and Language Amirkhani, Hossein AzariJafari, Mohammad Pourjafari, Zohreh Faridan-Jahromi, Soroush Kouhkan, Zeinab Amirak, Azadeh FarsTail: A Persian Natural Language Inference Dataset |
topic_facet |
Computer Science - Computation and Language |
description |
Natural language inference (NLI) is known as one of the central tasks in natural language processing (NLP) which encapsulates many fundamental aspects of language understanding. With the considerable achievements of data-hungry deep learning methods in NLP tasks, a great amount of effort has been devoted to develop more diverse datasets for different languages. In this paper, we present a new dataset for the NLI task in the Persian language, also known as Farsi, which is one of the dominant languages in the Middle East. This dataset, named FarsTail, includes 10,367 samples which are provided in both the Persian language as well as the indexed format to be useful for non-Persian researchers. The samples are generated from 3,539 multiple-choice questions with the least amount of annotator interventions in a way similar to the SciTail dataset. A carefully designed multi-step process is adopted to ensure the quality of the dataset. We also present the results of traditional and state-of-the-art methods on FarsTail including different embedding methods such as word2vec, fastText, ELMo, BERT, and LASER, as well as different modeling approaches such as DecompAtt, ESIM, HBMP, and ULMFiT to provide a solid baseline for the future research. The best obtained test accuracy is 83.38% which shows that there is a big room for improving the current methods to be useful for real-world NLP applications in different languages. We also investigate the extent to which the models exploit superficial clues, also known as dataset biases, in FarsTail, and partition the test set into easy and hard subsets according to the success of biased models. The dataset is available at https://github.com/dml-qom/FarsTail |
format |
Text |
author |
Amirkhani, Hossein AzariJafari, Mohammad Pourjafari, Zohreh Faridan-Jahromi, Soroush Kouhkan, Zeinab Amirak, Azadeh |
author_facet |
Amirkhani, Hossein AzariJafari, Mohammad Pourjafari, Zohreh Faridan-Jahromi, Soroush Kouhkan, Zeinab Amirak, Azadeh |
author_sort |
Amirkhani, Hossein |
title |
FarsTail: A Persian Natural Language Inference Dataset |
title_short |
FarsTail: A Persian Natural Language Inference Dataset |
title_full |
FarsTail: A Persian Natural Language Inference Dataset |
title_fullStr |
FarsTail: A Persian Natural Language Inference Dataset |
title_full_unstemmed |
FarsTail: A Persian Natural Language Inference Dataset |
title_sort |
farstail: a persian natural language inference dataset |
publishDate |
2020 |
url |
http://arxiv.org/abs/2009.08820 https://doi.org/10.1007/s00500-023-08959-3 |
genre |
DML |
genre_facet |
DML |
op_relation |
http://arxiv.org/abs/2009.08820 Soft Computing (2023) doi:10.1007/s00500-023-08959-3 |
op_doi |
https://doi.org/10.1007/s00500-023-08959-3 |
container_title |
Soft Computing |
_version_ |
1776199929499222016 |