Reducing Confusion in Active Learning for Part-Of-Speech Tagging
Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost. This is now an essential tool for building low-resource syntactic analyzers such as part-of-speech (POS) taggers. Existing AL heuristics are generally designed on the principle of sele...
Main Authors: | , , , |
---|---|
Format: | Text |
Language: | unknown |
Published: |
2020
|
Subjects: | |
Online Access: | http://arxiv.org/abs/2011.00767 |
id |
ftarxivpreprints:oai:arXiv.org:2011.00767 |
---|---|
record_format |
openpolar |
spelling |
ftarxivpreprints:oai:arXiv.org:2011.00767 2023-09-05T13:22:56+02:00 Reducing Confusion in Active Learning for Part-Of-Speech Tagging Chaudhary, Aditi Anastasopoulos, Antonios Sheikh, Zaid Neubig, Graham 2020-11-02 http://arxiv.org/abs/2011.00767 unknown http://arxiv.org/abs/2011.00767 Computer Science - Computation and Language text 2020 ftarxivpreprints 2023-08-16T16:10:29Z Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost. This is now an essential tool for building low-resource syntactic analyzers such as part-of-speech (POS) taggers. Existing AL heuristics are generally designed on the principle of selecting uncertain yet representative training instances, where annotating these instances may reduce a large number of errors. However, in an empirical study across six typologically diverse languages (German, Swedish, Galician, North Sami, Persian, and Ukrainian), we found the surprising result that even in an oracle scenario where we know the true uncertainty of predictions, these current heuristics are far from optimal. Based on this analysis, we pose the problem of AL as selecting instances which maximally reduce the confusion between particular pairs of output tags. Extensive experimentation on the aforementioned languages shows that our proposed AL strategy outperforms other AL strategies by a significant margin. We also present auxiliary results demonstrating the importance of proper calibration of models, which we ensure through cross-view training, and analysis demonstrating how our proposed strategy selects examples that more closely follow the oracle data distribution. Comment: To appear in TACL 2020. This is a pre-MIT Press publication version Text sami ArXiv.org (Cornell University Library) |
institution |
Open Polar |
collection |
ArXiv.org (Cornell University Library) |
op_collection_id |
ftarxivpreprints |
language |
unknown |
topic |
Computer Science - Computation and Language |
spellingShingle |
Computer Science - Computation and Language Chaudhary, Aditi Anastasopoulos, Antonios Sheikh, Zaid Neubig, Graham Reducing Confusion in Active Learning for Part-Of-Speech Tagging |
topic_facet |
Computer Science - Computation and Language |
description |
Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost. This is now an essential tool for building low-resource syntactic analyzers such as part-of-speech (POS) taggers. Existing AL heuristics are generally designed on the principle of selecting uncertain yet representative training instances, where annotating these instances may reduce a large number of errors. However, in an empirical study across six typologically diverse languages (German, Swedish, Galician, North Sami, Persian, and Ukrainian), we found the surprising result that even in an oracle scenario where we know the true uncertainty of predictions, these current heuristics are far from optimal. Based on this analysis, we pose the problem of AL as selecting instances which maximally reduce the confusion between particular pairs of output tags. Extensive experimentation on the aforementioned languages shows that our proposed AL strategy outperforms other AL strategies by a significant margin. We also present auxiliary results demonstrating the importance of proper calibration of models, which we ensure through cross-view training, and analysis demonstrating how our proposed strategy selects examples that more closely follow the oracle data distribution. Comment: To appear in TACL 2020. This is a pre-MIT Press publication version |
format |
Text |
author |
Chaudhary, Aditi Anastasopoulos, Antonios Sheikh, Zaid Neubig, Graham |
author_facet |
Chaudhary, Aditi Anastasopoulos, Antonios Sheikh, Zaid Neubig, Graham |
author_sort |
Chaudhary, Aditi |
title |
Reducing Confusion in Active Learning for Part-Of-Speech Tagging |
title_short |
Reducing Confusion in Active Learning for Part-Of-Speech Tagging |
title_full |
Reducing Confusion in Active Learning for Part-Of-Speech Tagging |
title_fullStr |
Reducing Confusion in Active Learning for Part-Of-Speech Tagging |
title_full_unstemmed |
Reducing Confusion in Active Learning for Part-Of-Speech Tagging |
title_sort |
reducing confusion in active learning for part-of-speech tagging |
publishDate |
2020 |
url |
http://arxiv.org/abs/2011.00767 |
genre |
sami |
genre_facet |
sami |
op_relation |
http://arxiv.org/abs/2011.00767 |
_version_ |
1776203499646746624 |