Reliability of trachoma clinical grading--assessing grading of marginal cases.
Clinical examination of trachoma is used to justify intervention in trachoma-endemic regions. Currently, field graders are certified by determining their concordance with experienced graders using the kappa statistic. Unfortunately, trachoma grading can be highly variable and there are cases where e...
Published in: | PLoS Neglected Tropical Diseases |
---|---|
Main Authors: | , , , , , , , , , , , |
Format: | Article in Journal/Newspaper |
Language: | English |
Published: |
Public Library of Science (PLoS)
2014
|
Subjects: | |
Online Access: | https://doi.org/10.1371/journal.pntd.0002840 https://doaj.org/article/13c0fc08525b473e8305d8ecb6d9fcd0 |
id |
ftdoajarticles:oai:doaj.org/article:13c0fc08525b473e8305d8ecb6d9fcd0 |
---|---|
record_format |
openpolar |
spelling |
ftdoajarticles:oai:doaj.org/article:13c0fc08525b473e8305d8ecb6d9fcd0 2023-05-15T15:14:53+02:00 Reliability of trachoma clinical grading--assessing grading of marginal cases. Salman A Rahman Sun N Yu Abdou Amza Sintayehu Gebreselassie Boubacar Kadri Nassirou Baido Nicole E Stoller Joseph P Sheehan Travis C Porco Bruce D Gaynor Jeremy D Keenan Thomas M Lietman 2014-05-01T00:00:00Z https://doi.org/10.1371/journal.pntd.0002840 https://doaj.org/article/13c0fc08525b473e8305d8ecb6d9fcd0 EN eng Public Library of Science (PLoS) http://europepmc.org/articles/PMC4006735?pdf=render https://doaj.org/toc/1935-2727 https://doaj.org/toc/1935-2735 1935-2727 1935-2735 doi:10.1371/journal.pntd.0002840 https://doaj.org/article/13c0fc08525b473e8305d8ecb6d9fcd0 PLoS Neglected Tropical Diseases, Vol 8, Iss 5, p e2840 (2014) Arctic medicine. Tropical medicine RC955-962 Public aspects of medicine RA1-1270 article 2014 ftdoajarticles https://doi.org/10.1371/journal.pntd.0002840 2022-12-31T08:43:52Z Clinical examination of trachoma is used to justify intervention in trachoma-endemic regions. Currently, field graders are certified by determining their concordance with experienced graders using the kappa statistic. Unfortunately, trachoma grading can be highly variable and there are cases where even expert graders disagree (borderline/marginal cases). Prior work has shown that inclusion of borderline cases tends to reduce apparent agreement, as measured by kappa. Here, we confirm those results and assess performance of trainees on these borderline cases by calculating their reliability error, a measure derived from the decomposition of the Brier score.We trained 18 field graders using 200 conjunctival photographs from a community-randomized trial in Niger and assessed inter-grader agreement using kappa as well as reliability error. Three experienced graders scored each case for the presence or absence of trachomatous inflammation-follicular (TF) and trachomatous inflammation-intense (TI). A consensus grade for each case was defined as the one given by a majority of experienced graders. We classified cases into a unanimous subset if all 3 experienced graders gave the same grade. For both TF and TI grades, the mean kappa for trainees was higher on the unanimous subset; inclusion of borderline cases reduced apparent agreement by 15.7% for TF and 12.4% for TI. When we assessed the breakdown of the reliability error, we found that our trainees tended to over-call TF grades and under-call TI grades, especially in borderline cases.The kappa statistic is widely used for certifying trachoma field graders. Exclusion of borderline cases, which even experienced graders disagree on, increases apparent agreement with the kappa statistic. Graders may agree less when exposed to the full spectrum of disease. Reliability error allows for the assessment of these borderline cases and can be used to refine an individual trainee's grading. Article in Journal/Newspaper Arctic Directory of Open Access Journals: DOAJ Articles Arctic PLoS Neglected Tropical Diseases 8 5 e2840 |
institution |
Open Polar |
collection |
Directory of Open Access Journals: DOAJ Articles |
op_collection_id |
ftdoajarticles |
language |
English |
topic |
Arctic medicine. Tropical medicine RC955-962 Public aspects of medicine RA1-1270 |
spellingShingle |
Arctic medicine. Tropical medicine RC955-962 Public aspects of medicine RA1-1270 Salman A Rahman Sun N Yu Abdou Amza Sintayehu Gebreselassie Boubacar Kadri Nassirou Baido Nicole E Stoller Joseph P Sheehan Travis C Porco Bruce D Gaynor Jeremy D Keenan Thomas M Lietman Reliability of trachoma clinical grading--assessing grading of marginal cases. |
topic_facet |
Arctic medicine. Tropical medicine RC955-962 Public aspects of medicine RA1-1270 |
description |
Clinical examination of trachoma is used to justify intervention in trachoma-endemic regions. Currently, field graders are certified by determining their concordance with experienced graders using the kappa statistic. Unfortunately, trachoma grading can be highly variable and there are cases where even expert graders disagree (borderline/marginal cases). Prior work has shown that inclusion of borderline cases tends to reduce apparent agreement, as measured by kappa. Here, we confirm those results and assess performance of trainees on these borderline cases by calculating their reliability error, a measure derived from the decomposition of the Brier score.We trained 18 field graders using 200 conjunctival photographs from a community-randomized trial in Niger and assessed inter-grader agreement using kappa as well as reliability error. Three experienced graders scored each case for the presence or absence of trachomatous inflammation-follicular (TF) and trachomatous inflammation-intense (TI). A consensus grade for each case was defined as the one given by a majority of experienced graders. We classified cases into a unanimous subset if all 3 experienced graders gave the same grade. For both TF and TI grades, the mean kappa for trainees was higher on the unanimous subset; inclusion of borderline cases reduced apparent agreement by 15.7% for TF and 12.4% for TI. When we assessed the breakdown of the reliability error, we found that our trainees tended to over-call TF grades and under-call TI grades, especially in borderline cases.The kappa statistic is widely used for certifying trachoma field graders. Exclusion of borderline cases, which even experienced graders disagree on, increases apparent agreement with the kappa statistic. Graders may agree less when exposed to the full spectrum of disease. Reliability error allows for the assessment of these borderline cases and can be used to refine an individual trainee's grading. |
format |
Article in Journal/Newspaper |
author |
Salman A Rahman Sun N Yu Abdou Amza Sintayehu Gebreselassie Boubacar Kadri Nassirou Baido Nicole E Stoller Joseph P Sheehan Travis C Porco Bruce D Gaynor Jeremy D Keenan Thomas M Lietman |
author_facet |
Salman A Rahman Sun N Yu Abdou Amza Sintayehu Gebreselassie Boubacar Kadri Nassirou Baido Nicole E Stoller Joseph P Sheehan Travis C Porco Bruce D Gaynor Jeremy D Keenan Thomas M Lietman |
author_sort |
Salman A Rahman |
title |
Reliability of trachoma clinical grading--assessing grading of marginal cases. |
title_short |
Reliability of trachoma clinical grading--assessing grading of marginal cases. |
title_full |
Reliability of trachoma clinical grading--assessing grading of marginal cases. |
title_fullStr |
Reliability of trachoma clinical grading--assessing grading of marginal cases. |
title_full_unstemmed |
Reliability of trachoma clinical grading--assessing grading of marginal cases. |
title_sort |
reliability of trachoma clinical grading--assessing grading of marginal cases. |
publisher |
Public Library of Science (PLoS) |
publishDate |
2014 |
url |
https://doi.org/10.1371/journal.pntd.0002840 https://doaj.org/article/13c0fc08525b473e8305d8ecb6d9fcd0 |
geographic |
Arctic |
geographic_facet |
Arctic |
genre |
Arctic |
genre_facet |
Arctic |
op_source |
PLoS Neglected Tropical Diseases, Vol 8, Iss 5, p e2840 (2014) |
op_relation |
http://europepmc.org/articles/PMC4006735?pdf=render https://doaj.org/toc/1935-2727 https://doaj.org/toc/1935-2735 1935-2727 1935-2735 doi:10.1371/journal.pntd.0002840 https://doaj.org/article/13c0fc08525b473e8305d8ecb6d9fcd0 |
op_doi |
https://doi.org/10.1371/journal.pntd.0002840 |
container_title |
PLoS Neglected Tropical Diseases |
container_volume |
8 |
container_issue |
5 |
container_start_page |
e2840 |
_version_ |
1766345286185123840 |