Probabilistic and Statistical Methods in Machine Learning

University Of New Mexico, Albuquerque NM

Investigators

Abstract

The main focus of this research is to take further the theory of data-dependent bounds on generalization error of learning algorithms, especially, in the context of classification problems. One of the main features of modern classification techniques is that they take into account the distribution of the so called classification margins (large margin methods), the quantities that characterize the reliability of classification. However, the margins alone do not give a satisfactory explanation of the superb performance of these methods. To provide such an explanation one has to combine margin type parameters with complexities of the classification rules in rather sophisticated upper confidence bounds. The goal of the research is to use concentration inequalities and various tools from the theory of Gaussian, empirical and Rademacher processes to develop new, much more subtle and powerful bounds of this type, that would lead to a much better understanding of the performance of the existing large margin classification methods and suggest ways to develop statistically optimal large margin procedures. The research includes the study of limit theorems and inequalities for ratio type empirical processes; the study of localized complexities of function classes involved in learning algorithms and the development of new bounds on generalization performance in terms of individual complexities of combined classifiers; the investigation of convergence rates of the empirical margin distribution to the true margin distribution of classifiers in terms of localized and individual complexities; the study of convergence rates of learning algorithms and the development of adaptive classification algorithms with optimal convergence rates; and the investigation of spectral properties of random matrices that play important role in learning theory for kernel machines. Learning Theory is a rapidly growing area between Computer Science, Mathematics, and Statistics that deals with modeling the process of learning and generalization in both biological and artificial neural networks and other learning machines. The results of the research are likely to facilitate further development of boosting, kernel machines, and other learning techniques and lead to new probabilistic bounds and asymptotic results in the theory of empirical processes with potential applications to many problems in statistical learning theory and other areas of Statistics. Methods of statistical learning theory have been penetrating many important areas of applications ranging from biotechnology to computer security. One of the topics of the proposed research is to develop applications of large margin learning methods in the area of robust control, in particular, to the problem of congestion control in communication networks. The results of the research are likely to impact this and other areas of applications.