What are the state-of-the-art machine learning methods and why do they work? This graduate-level introductory course to machine learning focuses on the foundations of modern machine learning. We will cover selected topics from supervised learning, unsupervised learning, generative learning, and interactive learning. For each topic, we will describe key algorithmic ideas/intuitions and basic theoretical insights. By the end of the course, students will master main machine learning techniques, and apply/optimize/develop them for specific applications.

Syllabus: Syllabus

Who: Professor Ju Sun (Instructor) Email: jusun AT umn.edu (Office Hours: 4–6pm Thur)
            Leon Luo Email: luo00042 AT umn.edu (Office Hours: 3–4pm Mon)

When/Where: Tue/Thur 2:30 – 3:45pm @ Mechanical Engineering 108 (in-person only with UNITE option)

Lecture Schedule

Date Topics Notes
Sep 03 Overview [Slides]
Sep 05 Review of high-dimensional calculus [Notes]
Three Ingredients of Machine Learning; Linear Predictions
Sep 10 Linear regression and least-squares problem [Notes]
Sep 12 Gradient descent for unconstrained optimization
Sep 17 More on gradient descent; review of subspaces and hyperplanes
Sep 19 Linear classification: Perceptron, SVM, and logistic regression
Support Vector Machines and Kernel Methods
Sep 24 Margin-maximization principle and hard-margin SVM [Notes]
Sep 26 Review of convex analysis; KKT conditions for convex problems
Oct 01 Analysis of hard-margin SVMs; soft-margin SVM
Oct 03 Property of soft-margin SVMs; solving SVM problem via stochastic gradient descent
Oct 08 Kernel methods
Oct 10 Beyond binary classification: multiclass and other learning settings
Elements of Statistical Learning Theory
Oct 15 Warmup: finite hypothesis class, realizable case [Notes]
Oct 17 General learning setup with finite hypothesis class
Oct 22 Infinite hypothesis class with Rademacher complexity
Oct 24 Infinite hypothesis class with growth function and VC dimension
Ensemble Methods: from Simple to Powerful
Oct 29 Decision stumps and their linear combinations; Adaboost [Notes]
Oct 31 Why Adaboost works; Adaboost as greedy method to find a linear combination
Nov 05 Gradient boosting
Nov 07 Decision trees
Nov 12 Random forests
Nov 14 Generalization of SVM and Boosting via margin bounds
Beyond Supervised Learning
Nov 19 Three interpretation of PCA [Notes]
Nov 21 Random projection and compressed sensing
Nov 26 Nonlinear dimension reduction
Nov 28 HAPPY THANKSGIVING! NO LECTURE
Dec 03 Clustering with non-density-based methods
Dec 05 Clustering with density-based methods
Dec 10 Generative models


Homework Assignments
HW1 (Due: Oct 06, 2024)
HW2 (Due: Oct 27, 2024)
HW3 (Due: Nov 30, 2024)
HW4 (Due: Dec 11 2024)
HW5 (Due: Dec 23 2024)