A hierarchical taxonomy for classifying hardness of inference tasks

• Main Authors: Gleize, Martin, Grau, Brigitte
• Other Authors: Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur (LIMSI), Université Paris Saclay (COmUE)-Centre National de la Recherche Scientifique (CNRS)-Sorbonne Université - UFR d'Ingénierie (UFR 919), Sorbonne Université (SU)-Sorbonne Université (SU)-Université Paris-Saclay-Université Paris-Sud - Paris 11 (UP11), European Language Resources Association
• Format: Conference Object
• Language: English
• Published: HAL CCSD 2014
• Subjects: inference; question answering; textual entailment; [INFO]Computer Science [cs]; [INFO.INFO-CL]Computer Science [cs]/Computation and Language [cs.CL]; Iceland
• Online Access: https://hal.archives-ouvertes.fr/hal-02281982; https://hal.archives-ouvertes.fr/hal-02281982/document; https://hal.archives-ouvertes.fr/hal-02281982/file/GleizeGrauLREC2014.pdf

International audience Exhibiting inferential capabilities is one of the major goals of many modern Natural Language Processing systems. However, if attempts have been made to define what textual inferences are, few seek to classify inference phenomena by difficulty. In this paper we propose a hierarchical taxonomy for inferences, relatively to their hardness, and with corpus annotation and system design and evaluation in mind. Indeed, a fine-grained assessment of the difficulty of a task allows us to design more appropriate systems and to evaluate them only on what they are designed to handle. Each of seven classes is described and provided with examples from different tasks like question answering, textual entailment and coreference resolution. We then test the classes of our hierarchy on the specific task of question answering. Our annotation process of the testing data at the QA4MRE 2013 evaluation campaign reveals that it is possible to quantify the contrasts in types of difficulty on datasets of the same task.