EP232U Deep Neural Networks 5.0 credits

1. Data-limited scenarios
- Relevant non-linear transformations
- Kernel tricks and kernel regression
- Connection with Support Vector Machine
- Random features and neural network
- Few-shot learning
2. Implementation issues and strategies in deep neural network
- Non-convex problem, gradient search and backpropagation,
- Training issues in DNNs, vanishing and exploding gradient problems
- Training using different optimization methods, e.g., SGD, RMSprop, AdaDelta, Adam, Dropout, data augmentation, etc.
- Unbalanced data problem and useful tricks, such as data augmentation
- Cross-validation techniques and model optimization to address over-fitting, for example, grid search, random search, k-fold, Stratified k-fold, early-stopping, drop out, bias-variance trade-off for model monitoring.
3. Structured deep neural networks
- AlexNet, VGG-16, U-Net, ResNet, DenseNet, SciNet, etc
4. Generative models.
- Implicit and explicit models
- Generative Adversarial Networks (GANs)
- Auto encoders, such as Variational auto-encoder (VAE), Denoising auto-encoder
- DC GAN, Cycle GAN
- Normalized flow models and likelihood computation
- Real NVP and Glow models
- Mixture flow model with expectation-maximization and gradient search
5. Deep neural networks for dynamic signals
- Recursive neural networks (RNN), e.g. LSTM, reservoir computing,
- Hidden Markov models (HMM)
- Normalised Flow networks based HMM
- Attention mechanism
6. Incremental learning, transfer learning
- Incremental learning, Learning without forgetting
- Transfer learning via pretraining, Transfer learning with GANs

After passing the course, the student shall be able to

• give an account of the ideas behind training and testing of deep neural networks
• explain when neural network can and cannot be used
• choose appropriate neural network for different practical problems.

• Knowledge in the equivalent IX1304 of one variable calculus Mathematics 7.5 credits
• Knowledge in linear algebra equivalent SF1672 Linear Algebra 7.5 credits
• Knowledge in probability theory equivalent SF2940 Probability Theory 7.5 credits
• Knowledge in Programming equivalent DD1315 programming and Matlab 7.5 credits
• The upper secondary course English B/6

If the course is discontinued, students may request to be examined during the following two academic years.

• DEL1 - Workshop, 1.0 credits, grading scale: P, F
• INL1 - Home assignments, 4.0 credits, grading scale: P, F

Based on recommendation from KTH’s coordinator for disabilities, the examiner will decide how to adapt an examination for students with documented disability.

The examiner may apply another examination format when re-examining individual students.

Attendance and presentation at final workshop are compulsory to pass the course. The workshop is arranged in a virtual way. If a student can not participate in the workshop, they must send their presentation, and the teacher will make a short oral examination.

Saikat Chatterjee

• All members of a group are responsible for the group's work.
• In any assessment, every student shall honestly disclose any help received and sources used.
• In an oral assessment, every student shall be able to present and answer questions about the entire assignment and solution.