Skip to main content

Peter Brohan: Quasi-Newtonian Optimisation for Deep Neural Networks

MSc Thesis Presentation

Time: Thu 2020-08-27 09.15 - 10.15

Lecturer: Peter Brohan

Location: Zoom, meeting ID: 61236446099

Supervisor: Yishao Zhou


In his 2010 paper, 'Deep learning via hessian-free optimization', Martens suggested techniques for the application of Quasi-Newtonian optimisation methods as a Neural Network learning algorithm. Here we examine the background of the paper, beginning with the structure and function of a neural network itself. We move on to consider a number of current popular alternative learning algorithms, considering their variety of approaches in approximating the optimisation problem in a tractable manner.

We then move on to look at Quasi-Newtonian methods in general, examining the Gauss-Newton, Levenberg-Marquardt and Truncated Newtonian approximations, each of which allows us make some approximation of the curvature of a given function, and to find approximate optimal solutions more practically than using the full Newton's Method.

We then consider the application of these methods to Neural Networks themselves, discuss further adaptations and run some small experiments to allow us some comparison of the Quasi-Newtonian approach to those methods in popular use today.

Belongs to: Department of Mathematics
Last changed: Aug 20, 2020