Performance metrics for marine mammal signal detection and classification.
Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-c...
Published in: | The Journal of the Acoustical Society of America |
---|---|
Main Authors: | , , , |
Format: | Article in Journal/Newspaper |
Language: | unknown |
Published: |
eScholarship, University of California
2022
|
Subjects: | |
Online Access: | https://escholarship.org/uc/item/6vc7j5sn https://doi.org/10.1121/10.0009270 |
id |
ftcdlib:oai:escholarship.org:ark:/13030/qt6vc7j5sn |
---|---|
record_format |
openpolar |
spelling |
ftcdlib:oai:escholarship.org:ark:/13030/qt6vc7j5sn 2024-09-15T18:00:03+00:00 Performance metrics for marine mammal signal detection and classification. Hildebrand, John A Frasier, Kaitlin E Helble, Tyler A Roch, Marie A 414 2022-01-01 https://escholarship.org/uc/item/6vc7j5sn https://doi.org/10.1121/10.0009270 unknown eScholarship, University of California qt6vc7j5sn https://escholarship.org/uc/item/6vc7j5sn doi:10.1121/10.0009270 public The Journal of the Acoustical Society of America, vol 151, iss 1 Animals Whales Sound Spectrography Echolocation Vocalization Animal Acoustics Benchmarking article 2022 ftcdlib https://doi.org/10.1121/10.0009270 2024-06-28T06:28:22Z Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-characteristic (ROC) curves, detection-error-trade-off (DET) curves, precision-recall (PR) curves, and cost curves. These metrics were applied to the generalized power law detector for blue whale D calls [Helble, Ierley, D'Spain, Roch, and Hildebrand (2012). J. Acoust. Soc. Am. 131(4), 2682-2699] and the click-clustering neural-net algorithm for Cuvier's beaked whale echolocation click detection [Frasier, Roch, Soldevilla, Wiggins, Garrison, and Hildebrand (2017). PLoS Comp. Biol. 13(12), e1005823] using data prepared for the 2015 Detection, Classification, Localization and Density Estimation Workshop. Detection class imbalance, particularly the situation of rare occurrence, is common for long-term passive acoustic monitoring datasets and is a factor in the performance of ROC and DET curves with regard to the impact of false positive detections. PR curves overcome this shortcoming when calculated for individual detections and do not rely on the reporting of true negatives. Cost curves provide additional insight on the effective operating range for the detector based on the a priori probability of occurrence. Use of more than a single metric is helpful in understanding the performance of a detection algorithm. Article in Journal/Newspaper Blue whale University of California: eScholarship The Journal of the Acoustical Society of America 151 1 414 427 |
institution |
Open Polar |
collection |
University of California: eScholarship |
op_collection_id |
ftcdlib |
language |
unknown |
topic |
Animals Whales Sound Spectrography Echolocation Vocalization Animal Acoustics Benchmarking |
spellingShingle |
Animals Whales Sound Spectrography Echolocation Vocalization Animal Acoustics Benchmarking Hildebrand, John A Frasier, Kaitlin E Helble, Tyler A Roch, Marie A Performance metrics for marine mammal signal detection and classification. |
topic_facet |
Animals Whales Sound Spectrography Echolocation Vocalization Animal Acoustics Benchmarking |
description |
Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-characteristic (ROC) curves, detection-error-trade-off (DET) curves, precision-recall (PR) curves, and cost curves. These metrics were applied to the generalized power law detector for blue whale D calls [Helble, Ierley, D'Spain, Roch, and Hildebrand (2012). J. Acoust. Soc. Am. 131(4), 2682-2699] and the click-clustering neural-net algorithm for Cuvier's beaked whale echolocation click detection [Frasier, Roch, Soldevilla, Wiggins, Garrison, and Hildebrand (2017). PLoS Comp. Biol. 13(12), e1005823] using data prepared for the 2015 Detection, Classification, Localization and Density Estimation Workshop. Detection class imbalance, particularly the situation of rare occurrence, is common for long-term passive acoustic monitoring datasets and is a factor in the performance of ROC and DET curves with regard to the impact of false positive detections. PR curves overcome this shortcoming when calculated for individual detections and do not rely on the reporting of true negatives. Cost curves provide additional insight on the effective operating range for the detector based on the a priori probability of occurrence. Use of more than a single metric is helpful in understanding the performance of a detection algorithm. |
format |
Article in Journal/Newspaper |
author |
Hildebrand, John A Frasier, Kaitlin E Helble, Tyler A Roch, Marie A |
author_facet |
Hildebrand, John A Frasier, Kaitlin E Helble, Tyler A Roch, Marie A |
author_sort |
Hildebrand, John A |
title |
Performance metrics for marine mammal signal detection and classification. |
title_short |
Performance metrics for marine mammal signal detection and classification. |
title_full |
Performance metrics for marine mammal signal detection and classification. |
title_fullStr |
Performance metrics for marine mammal signal detection and classification. |
title_full_unstemmed |
Performance metrics for marine mammal signal detection and classification. |
title_sort |
performance metrics for marine mammal signal detection and classification. |
publisher |
eScholarship, University of California |
publishDate |
2022 |
url |
https://escholarship.org/uc/item/6vc7j5sn https://doi.org/10.1121/10.0009270 |
op_coverage |
414 |
genre |
Blue whale |
genre_facet |
Blue whale |
op_source |
The Journal of the Acoustical Society of America, vol 151, iss 1 |
op_relation |
qt6vc7j5sn https://escholarship.org/uc/item/6vc7j5sn doi:10.1121/10.0009270 |
op_rights |
public |
op_doi |
https://doi.org/10.1121/10.0009270 |
container_title |
The Journal of the Acoustical Society of America |
container_volume |
151 |
container_issue |
1 |
container_start_page |
414 |
op_container_end_page |
427 |
_version_ |
1810437164096290816 |