Optimizing Classification Models to Application-Specific Performance Metrics

Cornell University, Ithaca NY

Investigators

Abstract

Many different criteria can be used to train and evaluate classifiers. Different criteria are appropriate in different settings, and learning methods that perform well on one criterion may not perform well on other. If a user must use a specific learning algorithm or model class, but needs to optimize to a performance measure for which that model class is not designed, they cannot do so. For example, neural nets are easy to train for continuous measures such as squared error and cross-entropy, but are difficult to train for discontinuous measures such as accuracy, Lift, and ROC area. SVMs are designed to optimize accuracy, but not squared error or cross-entropy. Decision trees typically optimize information-theoretic measures or accuracy, but are not designed to maximize ROC area or to minimize squared error. Moreover, for some performance metrics such as Lift we do not yet have any effective learning procedures. We are developing general-purpose cross-optimization methods for training learning algorithms to any performance measure. More specifically, we are developing meta-algorithms for optimizing the performance of different types of classifiers to metrics other than the one for which they were designed. We are using two meta-learning methods to accomplish this. The first is an ensemble learning method can optimize the performance of an ensemble of base-level classifiers to the user's criterion. The second method is a model adaption procedure that starts with a model optimized to one metric, and then iteratively transforms it into a model that is near-optimal with respect to a different user specified criterion. Both methods are designed to be compatible with most existing supervised learning methods. Our work has the potential to substantially improve classifiers by dealing up-front with the performance requirements of real-world applications. The work will have broad impact by giving machine learning users the flexibility to apply the performance metric that best fits their scientific, governmental or commercial needs. Our plans for outreach include distribution of software to aid classifier evaluation on multiple metrics, distributing software for ensemble selection, creating a multiple-metric competition at a conference such as KDD, organizing workshops on multi-metric learning, and building educational modules that demonstrate the importance of performance metrics in application-specific classifier design.