Prof. Stephen J. Wright conducted an excellent tutorial in NIPS 2010. This tutorial peeks into several important aspects of algorithms that are useful to practical and large-scale optimization problems in machine learning. Besides high-level overview of each aspect, the talk provides pointers to key references. Topics covered are:
Stochastic and Incremental Gradient Methods
Shrinking/Thresholding for Regularized Formulations
Optimal Manifolds Identification and High-Order Methods
Decomposition and Coordinate Relaxation
Also some tutorials/talks of interest from the long-term program “Modern Trends in Optimization and Its Application” (Sep – Dec 2010) in UCLA (provided the slides are released).
(Tutorial) Algorithms for Sparse Optimization
(Tutorial) Introduction to Robust Optimization