Methods for evaluating bioterrorism surveillance tools

Harvard Pilgrim Health Care, Inc., Boston MA

Investigators

Linked publications & trials

Abstract

DESCRIPTION: How will we know when a bioterrorist attacks us? Since the terrorist dissemination of anthrax in October of 2001, there has been concern about the nation's vulnerability to bioterrorism, that is, to terrorism by the spread of biological agents. Increasing resources have thus been devoted to surveillance systems intended to detect such attacks. One promising detection method is to monitor visits to primary care physicians. This approach relies on the fact that many potential bioterrorism agents cause early symptoms that are non-specific. If an attack can be detected through these early symptoms, then treatment, prophylaxis, and containment can be started earlier than if definitive diagnoses are required. However, detection through surveillance of early symptoms is difficult since the 'signal' of the bioterrorism must be detected against the 'noise' of naturally occurring disease. Several statistical algorithms have been proposed for this signal detection; however, relatively little is known about the relative performance of the methods. This is an important question, as substantially improved detection may be achieved by some statistical techniques relative to others, given the same data. There are two major obstacles impeding the comparison of techniques: First, bioterrorist attacks must be simulated, since they have traits that rule out simplification and generalization. We have addressed this problem by creating a complex microsimulation to describe the effects of an anthrax attack and how it would appear in a surveillance system we operate. Second, viable metrics for comparison must be created. In this case, relatively simple methods such as ARLs or ROC curves, are insufficient, since they ignore crucial features, such as timeliness of detection, variable rankings of methods for different false positive rates, and/or the number of people affected. In this application, we propose developing tools to compare statistical methods for detection of bioterrorist attack. We will explore: 1) weighted ROC curves; 2) generalized multidimensional ROC surfaces; and 3) cost-based evaluation incorporating investigation and false positive costs as well as the value of mortality and morbidity incurred and averted by each method.