Topics in Predictive and Descriptive Data Mining

Stanford University, Stanford CA

Investigators

Abstract

Abstract PI: Jerry Friedman DMS-0204029 This proposal seeks support for research in predictive and descriptive data mining (DM). Decision tree methods are the most popular predictive DM tools. Research under this grant will investigate ways to overcome their most serious limitation: severe over fitting in the presence of categorical (factorial) predictor variables with very large numbers of values (factors). Cluster analysis is often used as a tool in descriptive DM. In most DM applications a large number of variables are measured on each observation. Usually clustering, if it exists, occurs only within (often small) unknown subsets of all the measured variables. Moreover, individual clusters may represent groupings on (possibly overlapping) variable subsets. The goal is to identify the clustered groups as well as the particular variable subsets on which each one preferentially clusters. Traditional clustering algorithms are not well suited for this task. Research under this grant will investigate new approaches for solving this problem, especially in situations where there are a very large number of measured variables. Data mining is used to discover patterns and relationships in data, with an emphasis on very large data bases. It has had a major impact in business, industry, science, medicine, and most recently homeland security. Data mining activities divide into two types: predictive and descriptive. Predictive DM involves using past observational data from a system to build a mathematical model of that system. The model is used to predict some future unknown property (attribute or variable) of the system, given other properties that will be known in the future. Descriptive DM seeks to construct compact, interpretable summaries of the data in order to understand patterns and relationships, without focusing on the prediction of particular attributes. This research will investigate new methodologies for increasing the power of both descriptive and predictive DM in problems for which they have been traditionally weak.