An information approach to regularization parameter selection under model misspecification

We review the information approach to regularization parameter selection and its information complexity extension for the solution of discrete ill posed problems. An information criterion for regularization parameter selection was first proposed by Shibata in the context of ridge regression as an extension of Takeuchi's information criterion. In the information approach, the regularization parameter value is chosen to maximize the mean expected log likelihood (MELL) of a model whose parameters are estimated using the maximum penalized likelihood method. Under the Gaussian noise assumption such a choice coincides with the minimum of mean predictive error choice. Maximization of the MELL corresponds to minimization of the mean Kullback-Leibler information, that measures the deviation of the approximating (model) distribution from the true one. The resulting regularization parameter selection methods can handle possible functional and distributional misspecifications when the usual assumptions of Gaussian noise and/or linear relationship have been made but not met. We also suggest that in engineering applications it is beneficial to find ways of lowering the risk of getting grossly under-regularized solutions and that the new information complexity regularization parameter selection method (RPSM) is one of the possibilities. Several examples of applying the reviewed RPSMs are given.