Course teached as: B031301 - OPTIMIZATION TECHNIQUES FOR MACHINE LEARNING Second Cycle Degree in ARTIFICIAL INTELLIGENCE
Teaching Language
Italian (additional lecture notes are in English)
Course Content
Optimality conditions;
Methods for unconstrained local optimization;
Methods for constrained local optimization;
Optimization methods for machine learning problems.
Metodi di ottimizzazione non vincolata, L. Grippo, M. Sciandrone, Springer-Verlag, 2011
Additional Lecture Notes
Learning Objectives
This course gives the student a theoretical background on non linear optimization. It has the objective of giving the student a sufficiently deep knowledge on continuous optimization theory, optimization algorithms and their main characteristics. Specific attention is devoted to optimization methods for Machine Lerning
CA2: Applying knowledge and understanding related to the analysis and optimization of systems, as well as to their innovation also through the development and improvement of design methods, constantly confronting with the rapid evolution of engineering.
CA3: Applying knowledge and understanding related to the choice and application of appropriate analytical and modelling methods, based on mathematical and numerical analysis, in order to better simulate the behavior of components and plants in order to predict and improve their performance.
CA6: Applying knowledge and understanding related to the identification, location and retrieval of data and information necessary for the assessment.
CA8: Applying knowledge and understanding related to the appropriate interpretation of the results of experimental tests, verification calculations and complex theoretical simulation processes, through the use of the computer, applying the acquired experimental, modeling, mathematical and informatics bases.
CA12: Applying adequate knowledge and understanding to understand English texts.
CC1: In-depth knowledge and understanding of the theoretical-scientific aspects of engineering, in which students are able to identify, formulate and solve, even in an innovative way, complex and/or interdisciplinary problems. The ability to understand a multidisciplinary context in the engineering field and to work with a problem solving approach
Prerequisites
Elementary knowledge of calculus (Taylor expansions, gradients, Hessian matrix)
Linear algebra
A course on operations Research / linear programming might prove useful
Teaching Methods
Front lectures.
Type of Assessment
Oral exam on all the course subjects
The exam consists in checking, through theooretical questions:
- knowledge of the theory of optimization (optimality conditions);
- knowledge of non linear optimization algorithms and their theoretical properties;
- knowledge of optimization methods applied to machine learning.
Course program
- Introduction: mathematical optimization problems, examples
- Basic notions and definitions
- The empirical risk minimization problem
- Optimality conditions for unconstrained problems
- Convergence of iterative algorithms
- Gradient descent methods
- Conjugate directions method
- Least squares linear regression
- Newton method
-Quasi-Newton methods
- Logistic Regression
- Trust region methods
- Derivative-free methods
- Optimality conditions for constrained problems: KKTs
- Support Vector Machines: Definition and solution of the training problem
- Problems with convex constraints
- Methods for constrained problems
- Incremental methods for finite-sum problems
- Complexity of optimization algorithms
- Optimization algorithms for clustering