Think Deep Learning
Over the last few years, deep neural networks (DNN) have fundamentally transformed the way people think of machine learning and approach practical problems. Successes around DNN have ranged from traditional AI fields such as computer vision, natural language processing, interactive games, to health care, physical sciences—touching each and every corner of theoretical and applied domains. On the other hand, DNN still largely operate as black-boxes and we only have very limited understanding as for when and why they work. This course introduces basic ingredients of DNN, samples important applications, and throws around open problems. Emphasis is put on thinking from first principles, as the field is still evolving rapidly and there is nothing there that cannot be changed.
Instructor: Professor Ju Sun Email: jusun AT umn.edu
When/Where: T/Th 2:30PM–3:45PM, Akerman Hall 225
TA’s: Yuan Yao Email: yaoxx340 AT umn.edu Taihui Li Email: lixx5027 AT umn.edu
The detailed syllabus, containing the office hours, recommended references, assessment, homework and project requirements, programming and computing, and other resources, can be found here: Syllabus.pdf
References
- (D2L) Dive into Deep Learning by Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. Livebook.
- (DL) Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville. MIT Press, 2016.
- (MNDL) Neural Networks and Deep Learning by Charu Aggarwal. Springer, 2018.
- (DLR) The Deep Learning Revolution by Terrence J. Sejnowski. MIT Press, 2018.
- (DLP) Deep Learning with Python by François Chollet. Livebook.
- (HDML) Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (2ed) by Aurélien Géron. O’Reilly Media, 2019.
Lectures
slides: slides with transitions handout: slides without transitions
| Date | Topics | Notes | Reading |
|---|---|---|---|
| 01/21 | Overview | Slides Handout | |
| 01/23 | Neural networks: old and new | Slides Handout | DLP Ch 1, D2L Ch 3–4, MNDL Ch 2 |
| 01/28 | Fundamental belief: universal approximation theorems | Slides Handout | Visual proof of UAT |
| 01/30 | UAT: from shallow to deep | Slides Handout | |
| 02/04 | (Tutorial) Numpy, Scipy, Colab [Guest: Dr. Ben Lynch of MSI] | HTML Notebook | |
| 02/06 | (Discussion) Project ideas | Slides | |
| 02/11 | Basics of numerical optimization: preliminaries | Slides Handout | Zorich MA I (Ch 8) Nocedal & Wright NA (Chs 1&2) |
| 02/13 | Basics of numerical optimization: iterative methods – I | Slides Handout | Nocedal & Wright NA (Chs 3–7) |
| 02/18 | (Tutorial) Tensorflow, Pytorch, MSI GPU cluster [Guest: Dr. Ben Lynch of MSI] | HTML Notebook | |
| 02/20 | Basics of numerical optimization: iterative methods – II | Same as 02/13 | |
| 02/25 | Basics of numerical optimization: computing derivatives – I | Slides Handout | Nocedal & Wright NA (Ch 8), AD in ML |
| 02/27 | Basics of numerical optimization: computing derivatives – II | same as 02/25 | Why bother understanding AD? |
| 03/03 | Optimization for DNNs: basic methods | Slides Handout | OPT for DL Course by Ruoyu Sun , Survey paper, Stanford CS231n Notes: 1, 2, 3 |
| 03/05 | Optimization for DNNs: tricks | Slides Handout | OPT for DL Course by Ruoyu Sun , Survey paper, Stanford CS231n Notes: 1, 2, 3 |
| 03/10 | SPRING BREAK – NO CLASS | ||
| 03/12 | SPRING BREAK – NO CLASS | ||
| 03/17 | Unsupervised representation learning: autoencoders, factorization, and sparse Coding | Slides Handout | Representation learning review , DL Chs 13–15 |
| 03/19 | From fully connected to convolutional neural networks | ||
| 03/24 | Convolutional neural networks: basic models and computation | ||
| 03/26 | Convolutional neural networks: advanced models and applications | ||
| 03/31 | Project progress presentation | ||
| 04/02 | Project progress presentation | ||
| 04/07 | Project progress presentation | ||
| 04/09 | Sequential 1 | ||
| 04/14 | Sequential 2 | ||
| 04/16 | Sequential 3 | ||
| 04/21 | Generative models 1 | ||
| 04/23 | Generative models 2 | ||
| 04/28 | Reinforcement learning 1 | ||
| 04/30 | Reinforcement learning 2 | ||
| 05/05 | Scattering transform |
Homework assignments
- Assignment 0 (Due: Feb 07)
- Assignment 1 (Due: Mar 15)
- Assignment 2 (Due: Apr 18)
- Assignment 3 (Due: May 02)
- Assignment 4 (Due: May 14)