Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems
International audience The unsupervised discovery of linguistic terms from either continuous phoneme transcriptions or from raw speech has seen an increasing interest in the past years both from a theoretical and a practical standpoint. Yet, there exists no common accepted evaluation method for the...
Main Authors: | , , , , , , |
---|---|
Other Authors: | , , , , , , , , , , , , , , , |
Format: | Conference Object |
Language: | English |
Published: |
HAL CCSD
2014
|
Subjects: | |
Online Access: | https://inria.hal.science/hal-01026368 https://inria.hal.science/hal-01026368/document https://inria.hal.science/hal-01026368/file/320_Paper.pdf |
id |
ftunivrennes1hal:oai:HAL:hal-01026368v1 |
---|---|
record_format |
openpolar |
spelling |
ftunivrennes1hal:oai:HAL:hal-01026368v1 2024-04-14T08:13:46+00:00 Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems Ludusan, Bogdan Versteegh, M. Jansen, Aren Gravier, Guillaume Cao, Xuan-Nga Johnson, Mark Dupoux, Emmanuel Laboratoire de sciences cognitives et psycholinguistique (LSCP) Département d'Etudes Cognitives - ENS Paris (DEC) École normale supérieure - Paris (ENS-PSL) Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École normale supérieure - Paris (ENS-PSL) Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École des hautes études en sciences sociales (EHESS)-Centre National de la Recherche Scientifique (CNRS) Human Language Technology Center of Excellence Baltimore (HLTCOE) Johns Hopkins University (JHU) Center for Language and Speech Processing Baltimore (CLSP) Multimedia content-based indexing (TEXMEX) Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Inria Rennes – Bretagne Atlantique Institut National de Recherche en Informatique et en Automatique (Inria) Department of computing (Department of computing) Macquarie University Reykyavik, Iceland 2014-05 https://inria.hal.science/hal-01026368 https://inria.hal.science/hal-01026368/document https://inria.hal.science/hal-01026368/file/320_Paper.pdf en eng HAL CCSD hal-01026368 https://inria.hal.science/hal-01026368 https://inria.hal.science/hal-01026368/document https://inria.hal.science/hal-01026368/file/320_Paper.pdf info:eu-repo/semantics/OpenAccess Language Resources and Evaluation Conference https://inria.hal.science/hal-01026368 Language Resources and Evaluation Conference, May 2014, Reykyavik, Iceland evaluation spoken term discovery word segmentation [INFO.INFO-MM]Computer Science [cs]/Multimedia [cs.MM] info:eu-repo/semantics/conferenceObject Conference papers 2014 ftunivrennes1hal 2024-03-21T16:26:24Z International audience The unsupervised discovery of linguistic terms from either continuous phoneme transcriptions or from raw speech has seen an increasing interest in the past years both from a theoretical and a practical standpoint. Yet, there exists no common accepted evaluation method for the systems performing term discovery. Here, we propose such an evaluation toolbox, drawing ideas from both speech technology and natural language processing. We first transform the speech-based output into a symbolic representation and compute five types of evaluation metrics on this representation: the quality of acoustic matching, the quality of the clusters found, and the quality of the alignment with real words (type, token, and boundary scores). We tested our approach on two term discovery systems taking speech as input, and one using symbolic input. The latter was run using both the gold transcription and a transcription obtained from an automatic speech recognizer, in order to simulate the case when only imperfect symbolic information is available. The results obtained are analysed through the use of the proposed evaluation metrics and the implications of these metrics are discussed. Conference Object Iceland Université de Rennes 1: Publications scientifiques (HAL) |
institution |
Open Polar |
collection |
Université de Rennes 1: Publications scientifiques (HAL) |
op_collection_id |
ftunivrennes1hal |
language |
English |
topic |
evaluation spoken term discovery word segmentation [INFO.INFO-MM]Computer Science [cs]/Multimedia [cs.MM] |
spellingShingle |
evaluation spoken term discovery word segmentation [INFO.INFO-MM]Computer Science [cs]/Multimedia [cs.MM] Ludusan, Bogdan Versteegh, M. Jansen, Aren Gravier, Guillaume Cao, Xuan-Nga Johnson, Mark Dupoux, Emmanuel Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
topic_facet |
evaluation spoken term discovery word segmentation [INFO.INFO-MM]Computer Science [cs]/Multimedia [cs.MM] |
description |
International audience The unsupervised discovery of linguistic terms from either continuous phoneme transcriptions or from raw speech has seen an increasing interest in the past years both from a theoretical and a practical standpoint. Yet, there exists no common accepted evaluation method for the systems performing term discovery. Here, we propose such an evaluation toolbox, drawing ideas from both speech technology and natural language processing. We first transform the speech-based output into a symbolic representation and compute five types of evaluation metrics on this representation: the quality of acoustic matching, the quality of the clusters found, and the quality of the alignment with real words (type, token, and boundary scores). We tested our approach on two term discovery systems taking speech as input, and one using symbolic input. The latter was run using both the gold transcription and a transcription obtained from an automatic speech recognizer, in order to simulate the case when only imperfect symbolic information is available. The results obtained are analysed through the use of the proposed evaluation metrics and the implications of these metrics are discussed. |
author2 |
Laboratoire de sciences cognitives et psycholinguistique (LSCP) Département d'Etudes Cognitives - ENS Paris (DEC) École normale supérieure - Paris (ENS-PSL) Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École normale supérieure - Paris (ENS-PSL) Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-École des hautes études en sciences sociales (EHESS)-Centre National de la Recherche Scientifique (CNRS) Human Language Technology Center of Excellence Baltimore (HLTCOE) Johns Hopkins University (JHU) Center for Language and Speech Processing Baltimore (CLSP) Multimedia content-based indexing (TEXMEX) Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-Inria Rennes – Bretagne Atlantique Institut National de Recherche en Informatique et en Automatique (Inria) Department of computing (Department of computing) Macquarie University |
format |
Conference Object |
author |
Ludusan, Bogdan Versteegh, M. Jansen, Aren Gravier, Guillaume Cao, Xuan-Nga Johnson, Mark Dupoux, Emmanuel |
author_facet |
Ludusan, Bogdan Versteegh, M. Jansen, Aren Gravier, Guillaume Cao, Xuan-Nga Johnson, Mark Dupoux, Emmanuel |
author_sort |
Ludusan, Bogdan |
title |
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
title_short |
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
title_full |
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
title_fullStr |
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
title_full_unstemmed |
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
title_sort |
bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems |
publisher |
HAL CCSD |
publishDate |
2014 |
url |
https://inria.hal.science/hal-01026368 https://inria.hal.science/hal-01026368/document https://inria.hal.science/hal-01026368/file/320_Paper.pdf |
op_coverage |
Reykyavik, Iceland |
genre |
Iceland |
genre_facet |
Iceland |
op_source |
Language Resources and Evaluation Conference https://inria.hal.science/hal-01026368 Language Resources and Evaluation Conference, May 2014, Reykyavik, Iceland |
op_relation |
hal-01026368 https://inria.hal.science/hal-01026368 https://inria.hal.science/hal-01026368/document https://inria.hal.science/hal-01026368/file/320_Paper.pdf |
op_rights |
info:eu-repo/semantics/OpenAccess |
_version_ |
1796311828081410048 |