Feedback, Learning, and Adaptation - A Syllabus

A plan for this semester's class on learning, dynamics, and control.

Let’s take a stab at a semester’s worth of material on learning and control. As I said yesterday, this is a class about feedback. It will be about designed feedback, where we transform measurements of a system into actions to steer it. It will also be about modeled feedback, where we identify the input of some abstracted component with the output of another. Feedback is a simple idea that is perplexingly difficult to understand, analyze, and robustly deploy. We’ll start from simple, static examples and work our way to more complicated dynamical systems. We’ll look at designed feedback systems in control and “observed” feedback systems in the wild, and discuss how we model each.¹

We then turn to stability. Stability is a funny topic. Most people who have taken a class in control think that stability analysis is all of control theory. It’s just a lot of math belaboring conditions for all trajectories return to the origin. This makes control theory feel like a study of dead things. But in feedback systems, stability is a means of understanding active disturbance rejection far from equilibrium. Stability is really a theory of homeostasis, exploring the conditions that guarantee a system remains in a desired operating point. We’ll talk about the different ways in which we prove stability and relate them to the ways we think about convergence in optimization algorithms and generalization errors in machine learning models.

We’ll then discuss PID control. There is nothing more underappreciated in control theory than PID control. Most engineered control systems are PID. Most machine-learning and optimization algorithms are PID. And when you start looking at biological systems, you find that lots of regulatory mechanisms can be understood through the lens of PID. We can talk about “general controllers,” but this narrow, three-parameter PID model tells us a shocking amount about control analysis and design, as well as machine learning generalization.

We’ll then turn to optimization-based control. I mentioned yesterday that I don’t want to get trapped by optimal control. We can’t avoid it entirely, but I have a specific focus in mind for this class. PID control design tunes parameters. Optimization design, by contrast, is nonparametric. The parameters of optimization-based controllers emerge indirectly from how we tune costs and constraints. Optimization-based design enables us to incorporate concepts from stability theory into our cost function. We’ll cover the basics of optimal control and its approximations through Model Predictive Control and Approximate Dynamic Programming.²

We’ll then move to robustness and fragility. We’ll try to understand the basics of what makes a control system robust and how to design to recover from failures. We’ll look at the interplay between prediction quality, model misspecification, and robustness. Control design is confusing because negative feedback will often “work” even when your models are mostly wrong. The issue, of course, is that sometimes they won’t work. Here’s where we really get into the crux of what it means to learn, to adapt, to generalize. When models are wrong and measurements are noisy, how can you keep systems functioning at high levels?

This will let us pivot to what most people consider “learning.” We’ll discuss system identification, the parametric and nonparametric tools for estimating the dynamical properties of systems in open and closed loops. We’ll discuss how to identify systems for regulation and feedback. We’ll talk about estimation, how it works, and how it connects to control. This will let us get at the issue of partial observation and hidden states. We’ll talk about forecasting and how to build reliable forecasts, or at least create an illusion of reliability. And then we’ll talk about what happens when you put all of these “learned” components back into feedback loops, and the weird fragilities that can pop up.

Finally, I want to spend time on monster models. How can we understand and manipulate real complex systems with these ideas? When can we replace complex models with simple machine learning predictions? How do we use these in loops with complex systems? I don’t know if we can get there, but we’ll check back in April.

The class is going to be very much a “let’s figure this out as we go together.” I reserve the right to completely change this syllabus mid-semester! While I love the control perspective on feedback, I think it hits walls pretty quickly and can’t explain everything. I want to understand those walls without getting bogged down in functional analysis or noncommutative geometry. We’ll see if we can do it.

Along those same lines, while I wish I had a clean reading list to share, I’m not quite there yet. I have a rough idea of what I want to do, but I’m guessing this will change and morph as the semester goes on. Some of the key control references are: Feedback Control Theory by Doyle, Francis, and Tannenbaum; Advanced PID Control by Astrom and Hagglund; “The Behavioral Approach to Open and Interconnected Systems” by Willems; Stephen Boyd’s EE363 notes.³ For machine learning, we’ll frequently refer to the last two chapters in Patterns, Predictions, and Actions, along with the associated cited references. And all of this will be supplemented by a bunch of papers and blogs that I will pull out for each lecture.

And now I have the blog beg: What are your favorite writings on the foundations of feedback, learning, and homeostasis? Let me know in the comments. I’m hoping to draw in many perspectives this semester, and I promise to deliver an annotated bibliography at the end.

1. For my hardcore control-theorist readers: yes, this means the behavioral view will serve as a running reference point throughout the course.

2. Let it be known that most people have decided to call approximate dynamic programming “deep reinforcement learning.”

3. I really wish these were a fully fleshed out monograph. These notes are so good.