NOC:Optimization Theory and Algorithms

Lecture 1 - Introduction to the course - 1 - Prerequisites, key elements

Lecture 2 - Introduction to the course - 2 - Types of problems

Lecture 3 - Introduction to the course - 3 - An optimization example to live longer

Lecture 4 - Summary of background material - Linear Algebra - I

Lecture 5 - Summary of background material - Linear Algebra - II

Lecture 6 - Summary of background material - Analysis - I

Lecture 7 - Summary of background material - Analysis - II

Lecture 8 - Summary of background material - Analysis - III

Lecture 9 - Summary of background material - Calculus - I

Lecture 10 - Summary of background material - Calculus - II

Lecture 11 - Summary of background material - Calculus - III

Lecture 12 - Example of Multivariate Differentiation

Lecture 13 - Gradient of Quadratic form and product rule

Lecture 14 - Directional derivative, hessian, and mean value theorem

Lecture 15 - Unconstrained optimization - 1 - Roadmap of the course and Taylor’s theorem

Lecture 16 - Unconstrained optimization - 2 - Identifying a local minima - 1st and 2nd order conditions

Lecture 17 - Unconstrained optimization - 3 - Proof of 1st Order Condition

Lecture 18 - Unconstrained optimization - 4 - overview of algorithms and choosing a descent direction

Lecture 19 - Unconstrained optimization - 5 - properties of descent directions steepest descent direction

Lecture 20 - Unconstrained optimization - 6 - properties of descent directions newton direction

Lecture 21 - Unconstrained optimization - 7 - Trust Region Methods

Lecture 22 - A MATLAB session

Lecture 23 - Introduction to Line Search

Lecture 24 - Wolfe Conditions

Lecture 25 - Strong Wolfe Conditions

Lecture 26 - Backtracking Line Search

Lecture 27 - Line Search - Analysis

Lecture 28 - Line Search - Convergence and Rate - 1

Lecture 29 - Line Search - Convergence and Rate - 2

Lecture 30 - Convergence analysis of a descent algorithm - 1

Lecture 31 - Convergence analysis of a descent algorithm - 2

Lecture 32 - Implementation of an optimization algorithm in MATLAB

Lecture 33 - Conjugate Gradient Methods - Introduction and Proof

Lecture 34 - Visualizing Quadratic Forms

Lecture 35 - Orthogonality and Conjugacy

Lecture 36 - Conjugate Directions Method - Introduction and Proof

Lecture 37 - Discussion on doubts

Lecture 38 - More on Conjugate Directions Method

Lecture 39 - Ways of Generating Conjugate Directions

Lecture 40 - Expanding Subspace Theorem

Lecture 41 - Discussion on doubts

Lecture 42 - Conjugate Gradient Method

Lecture 43 - MATLAB implementation on CGM

Lecture 44 - Discussion on doubts

Lecture 45 - Preconditioned Conjugate Gradient - Part 1

Lecture 46 - Preconditioned Conjugate Gradient - Part 2

Lecture 47 - Preconditioned Conjugate Gradient - Part 3

Lecture 48 - Non Linear Conjugate Gradient method

Lecture 49 - Intro to Newton methods

Lecture 50 - Newton methods and convergence

Lecture 51 - Checks for positive definite matrices

Lecture 52 - Hessian Modification

Lecture 53 - Quasi newton methods

Lecture 54 - BFGS method

Lecture 55 - Least squares problems

Lecture 56 - Linear least squares - Part 1

Lecture 57 - Linear least squares - Part 2

Lecture 58 - Solving least squares using SVD

Lecture 59 - Non linear least squares

Lecture 60 - Constrained Optimisation

Lecture 61 - Single equality constraint

Lecture 62 - Single inequality constraint - Part 1

Lecture 63 - Single inequality constraint - Part 2

Lecture 64 - Two inequality constraints example

Lecture 65 - Linearised feasible directions

Lecture 66 - Feasible sequences and tangent cone

Lecture 67 - LICQ conditions

Lecture 68 - KKT conditions (First order necessary conditions)

Lecture 69 - Proof sketch for KKT conditions - Part 1

Lecture 70 - Proof sketch for KKT conditions - Part 2

Lecture 71 - Introduction to Projected gradient descent

Lecture 72 - Projected gradient descent and proof of convergence

Lecture 73 - Proof of convergence - Part 2

Lecture 74 - Subgradients and Subdifferential

Lecture 75 - Projection onto l1 ball

Lecture 76 - Soft thresholding example

Lecture 77 - Recap of Projection onto l1 ball

Lecture 78 - KKT and duality introduction

Lecture 79 - Intuition of duality and dual problem

Lecture 80 - Proof of concavity of the dual problem - Part 1

Lecture 81 - Proof of concavity of the dual problem - Part 2

Lecture 82 - Proof of concavity of the dual problem - Part 3