Peter Brohan: Quasi-Newtonian Optimisation for Deep Neural Networks
MSc Thesis Presentation
Time: Thu 2020-08-27 09.15 - 10.15
Location: Zoom, meeting ID: 61236446099
Participating: Peter Brohan
Supervisor: Yishao Zhou
In his 2010 paper, 'Deep learning via hessian-free optimization', Martens suggested techniques for the application of Quasi-Newtonian optimisation methods as a Neural Network learning algorithm. Here we examine the background of the paper, beginning with the structure and function of a neural network itself. We move on to consider a number of current popular alternative learning algorithms, considering their variety of approaches in approximating the optimisation problem in a tractable manner.
We then move on to look at Quasi-Newtonian methods in general, examining the Gauss-Newton, Levenberg-Marquardt and Truncated Newtonian approximations, each of which allows us make some approximation of the curvature of a given function, and to find approximate optimal solutions more practically than using the full Newton's Method.
We then consider the application of these methods to Neural Networks themselves, discuss further adaptations and run some small experiments to allow us some comparison of the Quasi-Newtonian approach to those methods in popular use today.