Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams

AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators ope...

Full description

Bibliographic Details
Main Authors: Radeta, Marko, Freitas, Ruben, Rodrigues, Claudio, Zuniga, Agustin, Nguyen, Ngoc Thi, Flores, Huber, Nurmi, Petteri
Other Authors: Department of Computer Science
Format: Article in Journal/Newspaper
Language:English
Published: ACM, Association for Computing Machinery 2024
Subjects:
Online Access:http://hdl.handle.net/10138/576331
id ftunivhelsihelda:oai:helda.helsinki.fi:10138/576331
record_format openpolar
spelling ftunivhelsihelda:oai:helda.helsinki.fi:10138/576331 2024-09-09T19:16:40+00:00 Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams Radeta, Marko Freitas, Ruben Rodrigues, Claudio Zuniga, Agustin Nguyen, Ngoc Thi Flores, Huber Nurmi, Petteri Department of Computer Science 2024-05-30T09:54:05Z application/pdf http://hdl.handle.net/10138/576331 eng eng ACM, Association for Computing Machinery 10.1145/3649457 339614 The research was supported by the Foundation for Science and Technology (FCT) projects: (i) INTERWHALE - Advancing Interactive Technology for Responsible Whale-Watching (grant agreement: PTDC/CCI-COM/0450/2020), (ii) MARE - The Marine and Environmental Sciences Centre (grant agreement: UIDB/04292/2020), (iii) ARNET - Aquatic Research Network (grant agreement: LA/P/0069/2020), and (iv) PhD scholarship (grant agreement: 2022.09961.BD). It has been also financed by the EU Horizon Europe project CLIMAREST: Coastal Climate Resilience and Marine Restoration Tools for the Arctic Atlantic basin (grant agreement: 101093865), the Academy of Finland (grant number: 339614), the European Social Fund via “ICT programme” measure, Estonian Center of Excellence in ICT Research (TK148 EXCITE), and the Nokia Foundation (grant number: 20220138) Radeta , M , Freitas , R , Rodrigues , C , Zuniga , A , Nguyen , N T , Flores , H & Nurmi , P 2024 , ' Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams ' , ACM Transactions on Interactive Intelligent Systems (TiiS) , vol. 14 , no. 2 . https://doi.org/10.1145/3649457 ORCID: /0000-0003-1011-8937/work/160774768 ORCID: /0000-0001-8262-6434/work/161727452 http://hdl.handle.net/10138/576331 794ee9b9-8d37-48b7-8812-1d141af63650 info:eu-repo/semantics/openAccess openAccess Computer and information sciences computer vision object detection machine learning deep learning annotation videos man-machine human-in-the-loop intelligent user interface AI-assisted interface Article acceptedVersion 2024 ftunivhelsihelda 2024-06-18T14:26:52Z AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labelling techniques (with and without AI-assistance). In a controlled study with N=34 annotators and a follow up study with 51963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases. Peer reviewed Article in Journal/Newspaper Arctic HELDA – University of Helsinki Open Repository
institution Open Polar
collection HELDA – University of Helsinki Open Repository
op_collection_id ftunivhelsihelda
language English
topic Computer and information sciences
computer vision
object detection
machine learning
deep learning
annotation
videos
man-machine
human-in-the-loop
intelligent user interface
AI-assisted interface
spellingShingle Computer and information sciences
computer vision
object detection
machine learning
deep learning
annotation
videos
man-machine
human-in-the-loop
intelligent user interface
AI-assisted interface
Radeta, Marko
Freitas, Ruben
Rodrigues, Claudio
Zuniga, Agustin
Nguyen, Ngoc Thi
Flores, Huber
Nurmi, Petteri
Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
topic_facet Computer and information sciences
computer vision
object detection
machine learning
deep learning
annotation
videos
man-machine
human-in-the-loop
intelligent user interface
AI-assisted interface
description AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labelling techniques (with and without AI-assistance). In a controlled study with N=34 annotators and a follow up study with 51963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases. Peer reviewed
author2 Department of Computer Science
format Article in Journal/Newspaper
author Radeta, Marko
Freitas, Ruben
Rodrigues, Claudio
Zuniga, Agustin
Nguyen, Ngoc Thi
Flores, Huber
Nurmi, Petteri
author_facet Radeta, Marko
Freitas, Ruben
Rodrigues, Claudio
Zuniga, Agustin
Nguyen, Ngoc Thi
Flores, Huber
Nurmi, Petteri
author_sort Radeta, Marko
title Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
title_short Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
title_full Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
title_fullStr Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
title_full_unstemmed Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams
title_sort man and the machine: effects of ai-assisted human labeling on interactive annotation of real-time video streams
publisher ACM, Association for Computing Machinery
publishDate 2024
url http://hdl.handle.net/10138/576331
genre Arctic
genre_facet Arctic
op_relation 10.1145/3649457
339614
The research was supported by the Foundation for Science and Technology (FCT) projects: (i) INTERWHALE - Advancing Interactive Technology for Responsible Whale-Watching (grant agreement: PTDC/CCI-COM/0450/2020), (ii) MARE - The Marine and Environmental Sciences Centre (grant agreement: UIDB/04292/2020), (iii) ARNET - Aquatic Research Network (grant agreement: LA/P/0069/2020), and (iv) PhD scholarship (grant agreement: 2022.09961.BD). It has been also financed by the EU Horizon Europe project CLIMAREST: Coastal Climate Resilience and Marine Restoration Tools for the Arctic Atlantic basin (grant agreement: 101093865), the Academy of Finland (grant number: 339614), the European Social Fund via “ICT programme” measure, Estonian Center of Excellence in ICT Research (TK148 EXCITE), and the Nokia Foundation (grant number: 20220138)
Radeta , M , Freitas , R , Rodrigues , C , Zuniga , A , Nguyen , N T , Flores , H & Nurmi , P 2024 , ' Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams ' , ACM Transactions on Interactive Intelligent Systems (TiiS) , vol. 14 , no. 2 . https://doi.org/10.1145/3649457
ORCID: /0000-0003-1011-8937/work/160774768
ORCID: /0000-0001-8262-6434/work/161727452
http://hdl.handle.net/10138/576331
794ee9b9-8d37-48b7-8812-1d141af63650
op_rights info:eu-repo/semantics/openAccess
openAccess
_version_ 1809756851836813312