NEW ROC METHODOLOGY FOR EVALUATION OF DIAGNOSTIC SYSTEMS

University Of Iowa, Iowa City IA

Investigators

Linked publications & trials

Abstract

Evaluation of diagnostic imaging systems and procedures is an important factor in the cost and quality of health care. Dorfman, Berbaum & Metz (1992) developed a new methodology that provides a comprehensive framework within which to analyze receiver operating characteristic (RQC) studies with multiple readers that takes into account both reader-sample and case-sample variation. The broad, long-term objective is to assess, perfect, and extend the useful range of application of the Dorfman/Berbaum/Metz (DBM) multireader ROC methodology for the evaluation of accuracy of diagnostic imaging systems. Significant problems remain to be solved before this approach can be broadly adopted with confidence to evaluate diagnostic systems. The specific aims of the proposed research project are: (1) determine why empirical and nominal error rates sometimes differ under the null hypothesis for the discrete rating format, and investigate procedures for reducing this difference in a broad range of experimental conditions to improve the validity and increase power; (2) determine why empirical and nominal error rates sometimes differ for continuous rating format and to investigate procedures for reducing this difference in a broad range of experimental conditions to improve validity and increase power; (3) evaluate the validity of the DBM multireader methodology in estimating confidence intervals for various effect-size differences; (4) develop and evaluate procedures for estimating the minimum reader and case sample sizes needed to detect a clinically meaningful difference with a fair degree of power and a small probability of a Type l error; and (5) incorporate covariates into the DBM multireader methodology, and evaluate the validity of this extension. These aims will be addressed by both Monte Carlo simulations and theoretical analysis.