Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators ope...
Main Authors: | , , , , , , |
---|---|
Other Authors: | |
Format: | Article in Journal/Newspaper |
Language: | English |
Published: |
ACM, Association for Computing Machinery
2024
|
Subjects: | |
Online Access: | http://hdl.handle.net/10138/576331 |
id |
ftunivhelsihelda:oai:helda.helsinki.fi:10138/576331 |
---|---|
record_format |
openpolar |
spelling |
ftunivhelsihelda:oai:helda.helsinki.fi:10138/576331 2024-09-09T19:16:40+00:00 Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams Radeta, Marko Freitas, Ruben Rodrigues, Claudio Zuniga, Agustin Nguyen, Ngoc Thi Flores, Huber Nurmi, Petteri Department of Computer Science 2024-05-30T09:54:05Z application/pdf http://hdl.handle.net/10138/576331 eng eng ACM, Association for Computing Machinery 10.1145/3649457 339614 The research was supported by the Foundation for Science and Technology (FCT) projects: (i) INTERWHALE - Advancing Interactive Technology for Responsible Whale-Watching (grant agreement: PTDC/CCI-COM/0450/2020), (ii) MARE - The Marine and Environmental Sciences Centre (grant agreement: UIDB/04292/2020), (iii) ARNET - Aquatic Research Network (grant agreement: LA/P/0069/2020), and (iv) PhD scholarship (grant agreement: 2022.09961.BD). It has been also financed by the EU Horizon Europe project CLIMAREST: Coastal Climate Resilience and Marine Restoration Tools for the Arctic Atlantic basin (grant agreement: 101093865), the Academy of Finland (grant number: 339614), the European Social Fund via “ICT programme” measure, Estonian Center of Excellence in ICT Research (TK148 EXCITE), and the Nokia Foundation (grant number: 20220138) Radeta , M , Freitas , R , Rodrigues , C , Zuniga , A , Nguyen , N T , Flores , H & Nurmi , P 2024 , ' Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams ' , ACM Transactions on Interactive Intelligent Systems (TiiS) , vol. 14 , no. 2 . https://doi.org/10.1145/3649457 ORCID: /0000-0003-1011-8937/work/160774768 ORCID: /0000-0001-8262-6434/work/161727452 http://hdl.handle.net/10138/576331 794ee9b9-8d37-48b7-8812-1d141af63650 info:eu-repo/semantics/openAccess openAccess Computer and information sciences computer vision object detection machine learning deep learning annotation videos man-machine human-in-the-loop intelligent user interface AI-assisted interface Article acceptedVersion 2024 ftunivhelsihelda 2024-06-18T14:26:52Z AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labelling techniques (with and without AI-assistance). In a controlled study with N=34 annotators and a follow up study with 51963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases. Peer reviewed Article in Journal/Newspaper Arctic HELDA – University of Helsinki Open Repository |
institution |
Open Polar |
collection |
HELDA – University of Helsinki Open Repository |
op_collection_id |
ftunivhelsihelda |
language |
English |
topic |
Computer and information sciences computer vision object detection machine learning deep learning annotation videos man-machine human-in-the-loop intelligent user interface AI-assisted interface |
spellingShingle |
Computer and information sciences computer vision object detection machine learning deep learning annotation videos man-machine human-in-the-loop intelligent user interface AI-assisted interface Radeta, Marko Freitas, Ruben Rodrigues, Claudio Zuniga, Agustin Nguyen, Ngoc Thi Flores, Huber Nurmi, Petteri Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams |
topic_facet |
Computer and information sciences computer vision object detection machine learning deep learning annotation videos man-machine human-in-the-loop intelligent user interface AI-assisted interface |
description |
AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labelling techniques (with and without AI-assistance). In a controlled study with N=34 annotators and a follow up study with 51963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases. Peer reviewed |
author2 |
Department of Computer Science |
format |
Article in Journal/Newspaper |
author |
Radeta, Marko Freitas, Ruben Rodrigues, Claudio Zuniga, Agustin Nguyen, Ngoc Thi Flores, Huber Nurmi, Petteri |
author_facet |
Radeta, Marko Freitas, Ruben Rodrigues, Claudio Zuniga, Agustin Nguyen, Ngoc Thi Flores, Huber Nurmi, Petteri |
author_sort |
Radeta, Marko |
title |
Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams |
title_short |
Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams |
title_full |
Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams |
title_fullStr |
Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams |
title_full_unstemmed |
Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams |
title_sort |
man and the machine: effects of ai-assisted human labeling on interactive annotation of real-time video streams |
publisher |
ACM, Association for Computing Machinery |
publishDate |
2024 |
url |
http://hdl.handle.net/10138/576331 |
genre |
Arctic |
genre_facet |
Arctic |
op_relation |
10.1145/3649457 339614 The research was supported by the Foundation for Science and Technology (FCT) projects: (i) INTERWHALE - Advancing Interactive Technology for Responsible Whale-Watching (grant agreement: PTDC/CCI-COM/0450/2020), (ii) MARE - The Marine and Environmental Sciences Centre (grant agreement: UIDB/04292/2020), (iii) ARNET - Aquatic Research Network (grant agreement: LA/P/0069/2020), and (iv) PhD scholarship (grant agreement: 2022.09961.BD). It has been also financed by the EU Horizon Europe project CLIMAREST: Coastal Climate Resilience and Marine Restoration Tools for the Arctic Atlantic basin (grant agreement: 101093865), the Academy of Finland (grant number: 339614), the European Social Fund via “ICT programme” measure, Estonian Center of Excellence in ICT Research (TK148 EXCITE), and the Nokia Foundation (grant number: 20220138) Radeta , M , Freitas , R , Rodrigues , C , Zuniga , A , Nguyen , N T , Flores , H & Nurmi , P 2024 , ' Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams ' , ACM Transactions on Interactive Intelligent Systems (TiiS) , vol. 14 , no. 2 . https://doi.org/10.1145/3649457 ORCID: /0000-0003-1011-8937/work/160774768 ORCID: /0000-0001-8262-6434/work/161727452 http://hdl.handle.net/10138/576331 794ee9b9-8d37-48b7-8812-1d141af63650 |
op_rights |
info:eu-repo/semantics/openAccess openAccess |
_version_ |
1809756851836813312 |