ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition
International audience This paper addresses the question of hierarchical named entity evaluation. In particular, we focus on metrics to deal with complex named entity structures as those introduced within the QUAERO project. The intended goal is to propose a smart way of evaluating partially correct...
Main Authors: | , , , , |
---|---|
Other Authors: | , , , , , , , , , , , , , , , , , , |
Format: | Conference Object |
Language: | English |
Published: |
HAL CCSD
2014
|
Subjects: | |
Online Access: | https://hal.science/hal-01134713 |
id |
ftuniparissaclay:oai:HAL:hal-01134713v1 |
---|---|
record_format |
openpolar |
spelling |
ftuniparissaclay:oai:HAL:hal-01134713v1 2023-11-12T04:19:20+01:00 ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition Ben Jannet, Mohamed Adda-Decker, Martine Galibert, Olivier Kahn, Juliette Rosset, Sophie LPP - Laboratoire de Phonétique et Phonologie - UMR 7018 (LPP) Université Sorbonne Nouvelle - Paris 3-Centre National de la Recherche Scientifique (CNRS) Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur (LIMSI) Université Paris-Sud - Paris 11 (UP11)-Sorbonne Université - UFR d'Ingénierie (UFR 919) Sorbonne Université (SU)-Sorbonne Université (SU)-Université Paris-Saclay-Centre National de la Recherche Scientifique (CNRS)-Université Paris Saclay (COmUE) LNE Université Paris-Sud - Paris 11 (UP11) This work was partially funded by the CIFRE convention 2012/0771 European Language Resources Association (ELRA) Khalid Choukri Thierry Declerck Hrafn Loftsson Bente Maegaard Joseph Mariani Asuncion Moreno Jan Odijk Stelios Piperi ANR-11-IDEX-0005,USPC,Université Sorbonne Paris Cité(2011) ANR-12-BS02-0006,VERA,Analyse d'erreurs avancée pour la reconnaissance de la parole(2012) Reykjavik, Iceland 2014-05-26 https://hal.science/hal-01134713 en eng HAL CCSD hal-01134713 https://hal.science/hal-01134713 Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) Ninth International Conference on Language Resources and Evaluation (LREC'14) https://hal.science/hal-01134713 Ninth International Conference on Language Resources and Evaluation (LREC'14), European Language Resources Association (ELRA), May 2014, Reykjavik, Iceland. pp.3987-3994 http://lrec2014.lrec-conf.org/en/ hierarchical named entity metrics evaluation [INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL] [SHS.LANGUE]Humanities and Social Sciences/Linguistics info:eu-repo/semantics/conferenceObject Conference papers 2014 ftuniparissaclay 2023-10-14T22:08:31Z International audience This paper addresses the question of hierarchical named entity evaluation. In particular, we focus on metrics to deal with complex named entity structures as those introduced within the QUAERO project. The intended goal is to propose a smart way of evaluating partially correctly detected complex entities, beyond the scope of traditional metrics. None of the existing metrics are fully adequate to evaluate the proposed QUAERO task involving entity detection, classification and decomposition.We are discussing the strong and weak points of the existing metrics. We then introduce a new metric, the Entity Tree Error Rate (ETER), to evaluate hierarchical and structured named entity detection, classification and decomposition. The ETER metric builds upon the commonly accepted SER metric, but it takes the complex entity structure into account by measuring errors not only at the slot (or complex entity) level but also at a basic (atomic) entity level. We are comparing our new metric to the standard one using first some examples and then a set of real data selected from the ETAPE evaluation results. Conference Object Iceland Archives ouvertes de Paris-Saclay The Slot ENVELOPE(155.083,155.083,-82.667,-82.667) |
institution |
Open Polar |
collection |
Archives ouvertes de Paris-Saclay |
op_collection_id |
ftuniparissaclay |
language |
English |
topic |
hierarchical named entity metrics evaluation [INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL] [SHS.LANGUE]Humanities and Social Sciences/Linguistics |
spellingShingle |
hierarchical named entity metrics evaluation [INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL] [SHS.LANGUE]Humanities and Social Sciences/Linguistics Ben Jannet, Mohamed Adda-Decker, Martine Galibert, Olivier Kahn, Juliette Rosset, Sophie ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition |
topic_facet |
hierarchical named entity metrics evaluation [INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL] [SHS.LANGUE]Humanities and Social Sciences/Linguistics |
description |
International audience This paper addresses the question of hierarchical named entity evaluation. In particular, we focus on metrics to deal with complex named entity structures as those introduced within the QUAERO project. The intended goal is to propose a smart way of evaluating partially correctly detected complex entities, beyond the scope of traditional metrics. None of the existing metrics are fully adequate to evaluate the proposed QUAERO task involving entity detection, classification and decomposition.We are discussing the strong and weak points of the existing metrics. We then introduce a new metric, the Entity Tree Error Rate (ETER), to evaluate hierarchical and structured named entity detection, classification and decomposition. The ETER metric builds upon the commonly accepted SER metric, but it takes the complex entity structure into account by measuring errors not only at the slot (or complex entity) level but also at a basic (atomic) entity level. We are comparing our new metric to the standard one using first some examples and then a set of real data selected from the ETAPE evaluation results. |
author2 |
LPP - Laboratoire de Phonétique et Phonologie - UMR 7018 (LPP) Université Sorbonne Nouvelle - Paris 3-Centre National de la Recherche Scientifique (CNRS) Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur (LIMSI) Université Paris-Sud - Paris 11 (UP11)-Sorbonne Université - UFR d'Ingénierie (UFR 919) Sorbonne Université (SU)-Sorbonne Université (SU)-Université Paris-Saclay-Centre National de la Recherche Scientifique (CNRS)-Université Paris Saclay (COmUE) LNE Université Paris-Sud - Paris 11 (UP11) This work was partially funded by the CIFRE convention 2012/0771 European Language Resources Association (ELRA) Khalid Choukri Thierry Declerck Hrafn Loftsson Bente Maegaard Joseph Mariani Asuncion Moreno Jan Odijk Stelios Piperi ANR-11-IDEX-0005,USPC,Université Sorbonne Paris Cité(2011) ANR-12-BS02-0006,VERA,Analyse d'erreurs avancée pour la reconnaissance de la parole(2012) |
format |
Conference Object |
author |
Ben Jannet, Mohamed Adda-Decker, Martine Galibert, Olivier Kahn, Juliette Rosset, Sophie |
author_facet |
Ben Jannet, Mohamed Adda-Decker, Martine Galibert, Olivier Kahn, Juliette Rosset, Sophie |
author_sort |
Ben Jannet, Mohamed |
title |
ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition |
title_short |
ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition |
title_full |
ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition |
title_fullStr |
ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition |
title_full_unstemmed |
ETER: a New Metric for the Evaluation of Hierarchical Named Entity Recognition |
title_sort |
eter: a new metric for the evaluation of hierarchical named entity recognition |
publisher |
HAL CCSD |
publishDate |
2014 |
url |
https://hal.science/hal-01134713 |
op_coverage |
Reykjavik, Iceland |
long_lat |
ENVELOPE(155.083,155.083,-82.667,-82.667) |
geographic |
The Slot |
geographic_facet |
The Slot |
genre |
Iceland |
genre_facet |
Iceland |
op_source |
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) Ninth International Conference on Language Resources and Evaluation (LREC'14) https://hal.science/hal-01134713 Ninth International Conference on Language Resources and Evaluation (LREC'14), European Language Resources Association (ELRA), May 2014, Reykjavik, Iceland. pp.3987-3994 http://lrec2014.lrec-conf.org/en/ |
op_relation |
hal-01134713 https://hal.science/hal-01134713 |
_version_ |
1782335796903149568 |