Lectures

Here is a provisional list of the course's lecture titles and content. The order in which I present the material may change  as may the content. I will make the slides for each lecture available (probably just) before I give the lecture. 

Lecture 1

Title:  The Deep Learning Revolution

Topics covered: Review of the impact deep networks have had in the application fields of speech, computer vision and NLP. Review of the course's syllabus, Review of the course's assignments, project, written exam and assessment.

Slides: 

• Lecture1.pdf, Lecture1_2by2.pdf
• CourseAdmin.pdf

Lecture 2

Title:  Learning Linear Binary & Linear Multi-class Classifiers from Labelled Training Data

 (mini-batch gradient descent optimization applied to "Loss + Regularization" cost functions)

Topics covered: Binary SVM classifiers as an unconstrained optimization problem, supervised learning = minimizing loss + regularization, Gradient Descent, SGD, mini-batch optimization,  multi-class classification with one layer networks, Different loss-functions.

Slides: Lecture2.pdf, Lecture2_2by2.pdf

Suggested Reading Material: Sections 5.1.4, 5.2, 5.2.2, 5.7.2 from "Deep Learning" by Goodfellow, Bengio and Courville. Link to Chapter 5 of Deep Learning

Sections 8.1.3, 8.3.1 from the book give amore detailed description and analysis of mini-batch gradient descent and SGD than given in the lecture notes. Link to Chapter 8 of Deep Learning.

The suggested readings from chapter 5 should be familiar to those who have already taken courses in ML. Lecture 2 should be more-or-less self-contained. But the reading material should flesh out some of the concepts referred to in the lecture

Lecture 3

Title:  Back Propagation

Topics covered:  Chain rule of differentiation, Computational graph, Back propagation (More detail then you probably ever expected!)

Slides: Lecture3.pdf, Lecture3_2by2.pdf

Suggested Reading Material:

Section 6.5 from the deep learning book.

I'm going to go into very explicit detail about the back-propagation algorithm.  It was not my original intention to have such an involved description but condensing the explanation make things less clear. My hope, though, is that everybody will have a good understanding of the theory and the mechanics of the algorithm after this lecture. I go into more specific detail (but not as generic) than in the deep learning book. So my recommendation is that you read my lecture notes to get a good understanding for the concrete example(s)  I explain and then you can read the deep learning book for a broader description.  Section 6.5 also assume you know about networks with more than 1 layer! So it may be better to hold off reading it until after lecture 4 (where I will talk about n-layer networks, activation functions, etc..) 

Lecture 4

Title:  From fully connected networks to Convolutional Networks (ConvNets)

Topics covered:  Backprop with Batch Normalization, Trouble of overfitting, Logistics of training:
Training, Validation \& Test sets, Cross-validation, Limitations of fully connected neural networks, Introducing Convolutional Networks

Slides:

Lecture 5

Title:  ConvNets Ctd

Topics covered:  max-pooling layers, history of ConvNets, Backprop for convolutional and max-pooling layers, viewing the fully connected layers as convolutional layers, Initialization of the parameters, Regularization, Batch normalization again, Evaluation of the models - ensembles

Slides:

Lecture 6

Title:  ConvNets in Practice

Topics covered:  Specification of architecture (big or many small filters, wide or deep), Training ConvNets in practice, Transfer learning, Distributed training, CPU/GPU bottlenecks 

Slides:

Lecture 7

Title:  Modern deep ConvNets + Variety of Applications

Topics covered:  Review of the modern top performing deep ConvNets - VGG, Inception, ResNet, Object detection with ConvNets, Speech Recognition with ConvNets, Encode - Decode ConvNets (estimating quantities for every pixel) 

Slides:

Lecture 8

Title:  Recurrent Neural Networks

Topics covered:  Coping with sequential data of variable length, Recurrent neural networks, GRUS, LSTMS, Using these for video analysis, RNNs for text synthesis

Slides:

Lecture 9

Title:  Incorporating Attention Mechanisms in Deep Networks

Topics covered:  Including attention mechanism into neural networks, Describe applications Machine translation, Generating text descriptions of images, Q&A systems 

Slides:

Lecture 10

Title:  Incorporating Explicit Memory Mechanisms in Deep Networks or Reinforcement Learning

Topics covered:  

Slides:

Lecture 11

Title:  Generative Adversarial networks (guest lecture)

Topics covered:  The hot and very exciting area of generative learning via generative adversarial networks.

Slides: