Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning

Microsoft Windows Feedback Hub is designed to receive customer feedback on a wide variety of subjects including critical topics such as power and battery. Feedback is one of the most effective ways to have a grasp of users' experience with Windows and its ecosystem. However, the sheer volume of...

Full description

Bibliographic Details
Main Authors: Abdali, Sara, Parikh, Anjali, Lim, Steve, Kiciman, Emre
Format: Text
Language:unknown
Published: 2023
Subjects:
Computer Science - Artificial Intelligence
Computer Science - Computation and Language
Computer Science - Machine Learning
Statistics - Methodology
DML
Online Access:http://arxiv.org/abs/2312.06820
id ftarxivpreprints:oai:arXiv.org:2312.06820
record_format openpolar
spelling ftarxivpreprints:oai:arXiv.org:2312.06820 2024-01-14T10:06:27+01:00 Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning Abdali, Sara Parikh, Anjali Lim, Steve Kiciman, Emre 2023-12-11 http://arxiv.org/abs/2312.06820 unknown http://arxiv.org/abs/2312.06820 Computer Science - Artificial Intelligence Computer Science - Computation and Language Computer Science - Machine Learning Statistics - Methodology text 2023 ftarxivpreprints 2023-12-17T02:07:43Z Microsoft Windows Feedback Hub is designed to receive customer feedback on a wide variety of subjects including critical topics such as power and battery. Feedback is one of the most effective ways to have a grasp of users' experience with Windows and its ecosystem. However, the sheer volume of feedback received by Feedback Hub makes it immensely challenging to diagnose the actual cause of reported issues. To better understand and triage issues, we leverage Double Machine Learning (DML) to associate users' feedback with telemetry signals. One of the main challenges we face in the DML pipeline is the necessity of domain knowledge for model design (e.g., causal graph), which sometimes is either not available or hard to obtain. In this work, we take advantage of reasoning capabilities in Large Language Models (LLMs) to generate a prior model that which to some extent compensates for the lack of domain knowledge and could be used as a heuristic for measuring feedback informativeness. Our LLM-based approach is able to extract previously known issues, uncover new bugs, and identify sequences of events that lead to a bug, while minimizing out-of-domain outputs. Text DML ArXiv.org (Cornell University Library)
institution Open Polar
collection ArXiv.org (Cornell University Library)
op_collection_id ftarxivpreprints
language unknown
topic Computer Science - Artificial Intelligence
Computer Science - Computation and Language
Computer Science - Machine Learning
Statistics - Methodology
spellingShingle Computer Science - Artificial Intelligence
Computer Science - Computation and Language
Computer Science - Machine Learning
Statistics - Methodology
Abdali, Sara
Parikh, Anjali
Lim, Steve
Kiciman, Emre
Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning
topic_facet Computer Science - Artificial Intelligence
Computer Science - Computation and Language
Computer Science - Machine Learning
Statistics - Methodology
description Microsoft Windows Feedback Hub is designed to receive customer feedback on a wide variety of subjects including critical topics such as power and battery. Feedback is one of the most effective ways to have a grasp of users' experience with Windows and its ecosystem. However, the sheer volume of feedback received by Feedback Hub makes it immensely challenging to diagnose the actual cause of reported issues. To better understand and triage issues, we leverage Double Machine Learning (DML) to associate users' feedback with telemetry signals. One of the main challenges we face in the DML pipeline is the necessity of domain knowledge for model design (e.g., causal graph), which sometimes is either not available or hard to obtain. In this work, we take advantage of reasoning capabilities in Large Language Models (LLMs) to generate a prior model that which to some extent compensates for the lack of domain knowledge and could be used as a heuristic for measuring feedback informativeness. Our LLM-based approach is able to extract previously known issues, uncover new bugs, and identify sequences of events that lead to a bug, while minimizing out-of-domain outputs.
format Text
author Abdali, Sara
Parikh, Anjali
Lim, Steve
Kiciman, Emre
author_facet Abdali, Sara
Parikh, Anjali
Lim, Steve
Kiciman, Emre
author_sort Abdali, Sara
title Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning
title_short Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning
title_full Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning
title_fullStr Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning
title_full_unstemmed Extracting Self-Consistent Causal Insights from Users Feedback with LLMs and In-context Learning
title_sort extracting self-consistent causal insights from users feedback with llms and in-context learning
publishDate 2023
url http://arxiv.org/abs/2312.06820
genre DML
genre_facet DML
op_relation http://arxiv.org/abs/2312.06820
_version_ 1788060928569245696

Similar Items