Optimization for Data Science

Overview

This course provides an in-depth theoretical treatment of classical and modern optimization methods that are relevant in data science. After a general discussion about the role that optimization has in the process of learning from data, we give an introduction to the theory of convex optimization. Based on this, we present and analyze algorithms in the following four categories: first-order methods, second-order methods, non-convexity, and min-max optimization.

The emphasis is on the motivations and design principles behind the algorithms, on provable performance bounds, and on the mathematical tools and techniques to prove them. The goal is to equip students with a fundamental understanding about why optimization algorithms work, and what their limits are.

Course information

Lecturers

Prof. Dr. Niao He (OAT Y21.1), Prof. Dr. Bernd Gaertner (OAT Z15).

Classes

Mon 13-14 (NO C60), Tue 10-12 (ETF C1).

Useful Links

Course Catalogue (261-5110-00L). All materials are available at Moodle including announcements and Q&A.

Prerequisites

A solid background in analysis and linear algebra; some background in theoretical computer science (computational complexity, analysis of algorithms); the ability to understand and write mathematical proofs.

Exercises

Group A, surnames A - L: Tue 14-16, CAB G51; Group B, surnames M - Z: Fri 14-16, ML H44. The exercises are discussed in classes. Students are expected to try to solve the problems beforehand. Your assistant is happy to look at your solutions and correct or comment on them. Students are assigned to classes according to surnames. Attendance according to these assignments is not compulsory but encouraged.

Assistants

Xiang Li (OAT Y23), Zebang Shen (OAT Y21.2), Anas Barakat (OAT Y21.2), Ilyas Fatkhullin (OAT X14), Liang Zhang (OAT Y23), Linfei Pan (CNB G100.9), Fangyuan Sun, Weixuan Yuan

Additional reading

Stephen Boyd and Lieven Vandenberghe, Convex Optimization; Sebastien Bubeck, Convex Optimization: Algorithms and Complexity; Martin J. Wainwright, High-Dimensional Statistics: A Non-Asymptotic Viewpoint.

Schedule and materials