Information Theoretic Validity of Penalized Likelihood Procedures: Application to Sparse Regression and Gaussian Graphical Models

Title

 

Information Theoretic Validity of Penalized Likelihood Procedures: Application to Sparse Regression and Gaussian Graphical Models

 

Abstract

We review theory for the Minimum Description Length principle, penalized likelihood and its statistical risk. An information theoretic condition on a penalty pen(f) yields the conclusion that the optimizer of the penalized log lokelihood criterion log 1/likelihood(f) + pen(f) has risk not more than the index of resolvability, corresponding to the accuracy of the optimizer of the expected value of the criterion.  For the linear span of a dictionary of candidate terms, we develop the information theoretic validity of penalties based on the l_1 norm of the coefficients in regression and log-density estimation settings. New results are presented for Gaussian graphical models.  This represents joint work with Xi Luo and Sabyasachi Chatterjee.

 

About the presenter
 

Prof. Andrew Barron will be visiting HIIT from September 9 until September 27, hosted by Teemu Roos. Please contact either Prof. Barron ([email protected]) or Teemu Roos ([email protected]) to discuss with him during his visit.

 

Andrew R. Barron is a professor of statistics at Yale University, where he has served as Chair of Statistics from 1999-2006. His research interests include the study of information-theoretic properties in the topics of probability limit theory, statistical inference, high-dimensional function estimation, neural networks, model selection, communication, universal data compression, prediction, and investment theory.

 

Prof. Barron received (jointly with Bertrand S. Clarke) the 1991 Browder J. Thompson Prize (best paper in all IEEE TRANSACTIONS in 1990 by authors age 30 or under) for the paper “Information-Theoretic Asymptotics of Bayes Methods.” Dr. Barron was an Institute of Mathematical Statistics Medallion Award recipient in 2005. He served on the Board of Governors of the IEEE Information Theory Society from 1995 to 1999, and was Secretary of the Board of Governors during 1989-1990. He has served as an Associate Editor for the IEEE Transactions on Information Theory from 1993 to 1995, and the Annals of Statistics for 1995-1997.