Artificial intelligence in the GPs office: a retrospective study on diagnostic accuracy

Objective Machine learning (ML) is expected to play an increasing role within primary health care (PHC) in coming years. No peer-reviewed studies exist that evaluate the diagnostic accuracy of ML models compared to general practitioners (GPs). The aim of this study was to evaluate the diagnostic acc...

Full description

Bibliographic Details
Published in:Scandinavian Journal of Primary Health Care
Main Authors: Steindor Ellertsson, Hrafn Loftsson, Emil L. Sigurdsson
Format: Article in Journal/Newspaper
Language:English
Published: Taylor & Francis Group 2021
Subjects:
Online Access:https://doi.org/10.1080/02813432.2021.1973255
https://doaj.org/article/d0322ac110a74b778a4e10d4fbba8cbf
Description
Summary:Objective Machine learning (ML) is expected to play an increasing role within primary health care (PHC) in coming years. No peer-reviewed studies exist that evaluate the diagnostic accuracy of ML models compared to general practitioners (GPs). The aim of this study was to evaluate the diagnostic accuracy of an ML classifier on primary headache diagnoses in PHC, compare its performance to GPs, and examine the most impactful signs and symptoms when making a prediction. Design A retrospective study on diagnostic accuracy, using electronic health records from the database of the Primary Health Care Service of the Capital Area (PHCCA) in Iceland. Setting Fifteen primary health care centers of the PHCCA. Subjects All patients that consulted a physician, from 1 January 2006 to 30 April 2020, and received one of the selected diagnoses. Main outcome measures Sensitivity, Specificity, Positive Predictive Value, Matthews Correlation Coefficient, Receiver Operating Characteristic (ROC) curve, and Area under the ROC curve (AUROC) score for primary headache diagnoses, as well as Shapley Additive Explanations (SHAP) values of the ML classifier. Results The classifier outperformed the GPs on all metrics except specificity. The SHAP values indicate that the classifier uses the same signs and symptoms (features) as a physician would, when distinguishing between headache diagnoses. Conclusion In a retrospective comparison, the diagnostic accuracy of the ML classifier for primary headache diagnoses is superior to GPs. According to SHAP values, the ML classifier relies on the same signs and symptoms as a physician when making a diagnostic prediction.Keypoints Little is known about the diagnostic accuracy of machine learning (ML) in the context of primary health care, despite its considerable potential to aid in clinical work. This novel research sheds light on the diagnostic accuracy of ML in a clinical context, as well as the interpretation of its predictions. If the vast potential of ML is to be utilized in primary health care, ...