Convex Optimization Live Blog

I’ll be live blogging my graduate course on convex optimization this semester (Fall 2024). The course is based on the text Convex Optimization by Stephen Boyd and Lieven Vandenberghe.

Lecture 1: Introduction

1. Convex Optimization in the Age of LLMs

Table of Contents

Part I: Modeling

Lecture 2: Programmable convexity. Examples of convex sets and how to generate them.

Lecture 3: Separating hyperplanes.

1. Think local, act global. Convex optimization problems are the ones where local search finds global solutions.

2. Degrees of separation. Separating hyperplane theorems and their implications.

Lecture 4: Convex functions. Convex functions and how to generate them.

Lecture 5: More examples of convex and quasiconvex functions

1. Stretching the Definitions. A taste of the weird world of quasiconvexity: nonconvex models that aren’t quite convex but are close enough.

2. Linear Doesn't Mean Easy. One of the best ways to learn linear algebra is to suffer through all of the exercises in this class.

Lecture 6: Convex Optimization Problems

1. Declarative Mathematical Programming. Thinking about optimization modeling as a form declarative programming.

2. The Cost of Modeling. Optimization models are always idealizations. Know when this is a problem.

Lecture 7: Cone Programming

1. Modeling Dystopia. The paradox of choice in modeling problems as convex optimizations.

2. Cone Programming. A technical post on the expressivity of modeling with self-dual cones.

Lecture 8: Optimality Conditions. How to think about certificates that proving optimality.

Lecture 9: Duality

1. Duality. Trying to motivate why we should care about duality and explain some connections to von Neumann’s minimax theory.

2. Looking above and below. Lagrangian duality is a systematic way to generate lower bounds of optimization problems.

Lecture 10: Ends in a Draw. A geometric view of strong duality.

Part II: Applications

Lecture 11: Inverse Problems.

1. Inverse Problems - An introduction.

2. An Inversion Cookbook - A modular way to pose convex inverse problems.

3. Inverse Frontiers - Inverse problems I don’t know how to model as convex optimization problems.

Lecture 12: Function Fitting. Interpolation Is All You Need

Lecture 13: Decision Theory. The Shape of Stats to Come.

Lecture 14: Experiment Design. Designed Interactions.

Midterm Reflections

Lecture 15: Convex Optimization at the Midpoint.

Part III: Algorithms

Lecture 16: Solving linear equations at scale. Basic Linear Algebra Subprogramming.

Lecture 17: Unconstrained Minimization Algorithms. All paths point downhill.

Lecture 18: Convergence Analysis. Halo in a Haystack.

Lecture 19: Primal Dual Interior Point Methods.

1. The Central Path.

2. A note about DSLs and their limitations. Indiscipline.

Lecture 20: Other approaches to constrained optimization. The Art of Desconstraining.

Lecture 21: Dual Decomposition.

Coda: Optimal Control

Lecture 22: The Optimal Part of Control.

Lecture 23: Duality, LQR, the Method of Adjoints, and Backpropagation. Twirling Towards Freedom.

End of Semester Reflections

Unbounded Above.