New Methods for Evaluating Interobserver Agreement

Emory University, Atlanta GA

Investigators

Linked publications & trials

Abstract

[unreadable] DESCRIPTION (provided by applicant): Behavioral, social and biomedical researchers use a variety of instruments to measure a wide range of personal, environmental, behavioral and biomedical variables. Before a new measurement scale is adopted for practical use, one needs to evaluate inter, intra and total agreement among various observers and/or measurement methods. In this proposal we take a fresh look into the field of assessing observer agreement for continuous measurement scales. Existing indices of agreement, such as the intra-class correlation coefficient, indices based on generalizability theory (GT) and the concordance correlation coefficient measure the total observer agreement, which combines the inter and intra observer components of agreement. Currently there is no proper inter-observer agreement index for assessing the "true" agreement among observers in terms of true (i.e., error-free) values of different observers on the same subject. The first major objective of this project is to develop two new indices of interobserver agreement defined in terms of the distances among the true values assigned to each subject by several observers. In the proposed research, we will develop methods for statistical inference on the new interobserver agreement indices and investigate their properties via simulations. [unreadable] The second major objective of this project is to develop and investigate innovative methods for statistical inference on the GT-based indices. In GT it is usually assumed that the observations follow an analysis of variance (ANOVA) model and inference is made under the normality assumption. These assumptions are often unrealistic. We will extend GT concepts to more general settings and develop nonparametric, Semi-parametric and parametric methods for estimation and inference. Existing and new agreement indices will be compared and linked. All the proposed methods will be applied to real data sets from the behavioral and biomedical sciences. [unreadable] [unreadable]