Cool Kids Keep

On the academic imperialism of reinforcement learning.

Argmin blog is back and feeling the AGI. We’re in the homestretch of this class on machine learning and we’re going to do a lecture on reinforcement learning. I guess.

“Wait, how can you do reinforcement learning justice in a single lecture?” you ask in bewilderment. The question depends on whether you are an RL Maximalist or an RL Minimalist.

RL Maximalism

Sarah Dean introduced me to the idea of RL Maximalism. For the RL Maximalist, reinforcement learning encompasses all decision making under uncertainty. The RL Maximalist Creed is promulgated in the introduction of Sutton and Barto:

Reinforcement learning is learning what to do--how to map situations to actions--so as to maximize a numerical reward signal.

Sutton and Barto highlight the breadth of the RL Maximalist program through examples: 

A good way to understand reinforcement learning is to consider some of the examples and possible applications that have guided its development.

A master chess player makes a move. The choice is informed both by planning--anticipating possible replies and counterreplies--and by immediate, intuitive judgments of the desirability of particular positions and moves.

An adaptive controller adjusts parameters of a petroleum refinery's operation in real time. The controller optimizes the yield/cost/quality trade-off on the basis of specified marginal costs without sticking strictly to the set points originally suggested by engineers.

A gazelle calf struggles to its feet minutes after being born. Half an hour later it is running at 20 miles per hour.

A mobile robot decides whether it should enter a new room in search of more trash to collect or start trying to find its way back to its battery recharging station. It makes its decision based on how quickly and easily it has been able to find the recharger in the past.

Phil prepares his breakfast. Closely examined, even this apparently mundane activity reveals a complex web of conditional behavior and interlocking goal-subgoal relationships: walking to the cupboard, opening it, selecting a cereal box, then reaching for, grasping, and retrieving the box. Other complex, tuned, interactive sequences of behavior are required to obtain a bowl, spoon, and milk jug. Each step involves a series of eye movements to obtain information and to guide reaching and locomotion. Rapid judgments are continually made about how to carry the objects or whether it is better to ferry some of them to the dining table before obtaining others. Each step is guided by goals, such as grasping a spoon or getting to the refrigerator, and is in service of other goals, such as having the spoon to eat with once the cereal is prepared and ultimately obtaining nourishment.

That’s casting quite a wide net there, gentlemen! And other than chess, current reinforcement learning methods don’t solve any of these examples. But based on researcher propaganda and credulous reporting, you’d think reinforcement learning can solve all of these things. For the RL Maximalists, as you can see from their third example, all of optimal control is a subset of reinforcement learning. Sutton and Barto make that case a few pages later:

In this book, we consider all of the work in optimal control also to be, in a sense, work in reinforcement learning. We define reinforcement learning as any effective way of solving reinforcement learning problems, and it is now clear that these problems are closely related to optimal control problems, particularly those formulated as MDPs. Accordingly, we must consider the solution methods of optimal control, such as dynamic programming, also to be reinforcement learning methods.

My friends who work on stochastic programming, robust optimization, and optimal control are excited to learn they actually do reinforcement learning. Or at least that the RL Maximalists are claiming credit for their work.

This RL Maximalist view resonates with a small but influential clique in the machine learning community. At OpenAI, an obscure hybrid non-profit org/startup in San Francisco run by a religious organization, even supervised learning is reinforcement learning. So yes, for the RL Maximalist, we have been studying reinforcement learning for an entire semester, and today is just the final Lecunian cherry.

RL Minimalism

The RL Minimalist views reinforcement learning as the solution of short-horizon policy optimization problems by a sequence of random randomized controlled trials. For the RL Minimalist working on control theory, their design process for a robust robotics task might go like this:

1. Design a complex policy optimization problem. This problem will include an intricate dynamics model. This model might only by accessible through a simulator. The formulation will explicitly quantify model and environmental uncertainties as random processes. 

2. Posit an explicit form for the policy that maps observations to actions. A popular choice for the RL Minimalist is some flavor of neural network.

3. The resulting problem is probably hard to optimize, but it can be solved by iteratively running random searches. That is, take the current policy, perturb it a bit, and if the perturbation improves the policy, accept the perturbation as a new policy.

This approach can be very successful. RL Minimalists have recently produced demonstrations of agile robot dogs, superhuman drone racing, and plasma control for nuclear fusion. The funny thing about all of these examples is there’s no learning going on. All just solve policy optimization problems in the way I described above. 

I am totally fine with this RL Minimalism. Honestly, it isn’t too far a stretch from what people already do in academic control theory. In control, we frequently pose optimization problems for which our desired controller is the optimum. We’re just restricted by the types of optimization problems we know how to solve efficiently. RL Minimalists propose using inefficient but general solvers that let them pose almost any policy optimization problem they can imagine. The trial-and-error search techniques that RL Minimalists use are frustratingly slow and inefficient. But as computers get faster and robotic systems get cheaper, these crude but general methods have become more accessible. 

The other upside of RL Minimalism is it’s pretty easy to teach. For the RL Minimalist, after a semester of preparation, the theory of reinforcement learning only needs one lecture. The RL Minimalist doesn’t have to introduce all of the impenetrable notation and terminology of reinforcement learning, nor do they need to teach dynamic programming. RL Minimalists have a simple sales pitch: “Just take whatever derivative-free optimizer you have and use it on your policy optimization problem.” That’s even more approachable than control theory!

Indeed, embracing some RL Minimalism might make control theory more accessible. Courses could focus on the essential parts of control theory: feedback, safety, and performance tradeoffs. The details of frequency domain margin arguments or other esoteric minutiae could then be secondary. 

Whose view is right?

I created this split between RL Minimalism and Maximalism in response to an earlier blog where I asserted that “reinforcement learning doesn’t work.” In that blog, I meant something very specific. I distinguished systems where we have a model of the world and its dynamics against those we could only interrogate through some sort of sampling process. The RL Maximalists refer to this split as “model-based” versus “model-free.” I loathe this terminology, but I’m going to use it now to make a point.

RL Minimalists are solving model-based problems. They solve these problems with Monte Carlo methods, but the appeal of RL Minimalism is it lets them add much more modeling than standard optimal control methods. RL Minimalists need a good simulator of their system. But if you have a simulator, you have a model. RL Minimalists also need to model parameter uncertainty in their machines. They need to model environmental uncertainty explicitly. The more modeling that is added, the harder their optimization problem is to solve. But also, the more modeling they do, the better performance they get on the task at hand.

The sad truth is no one can solve a “model-free” reinforcement learning problem. There are simply no legitimate examples of this. When we have a truly uncertain and unknown system, engineers will spend months (or years) building models of this system before trying to use it. Part of the RL Maximalist propaganda suggests you can take agents or robots that know nothing, and they will learn from their experience in the wild. Outside of very niche demos, such systems don’t exist and can’t exist.

This leads to my main problem with the RL Minimalist view: It gives credence to the RL Maximalist view, which is completely unearned. Machines that “learn from scratch” have been promised since before there were computers. They don’t exist. You can’t solve how a giraffe works or how the brain works using temporal difference learning. We need to separate the engineering from the science fiction.