Guest speaker series on Graphical Models and Neural Networks

Optimal control as graphical model inference

In this lecture we will be particularly interested in a setting where an agent needs to make a sequence of decisions under uncertainty. We will review the framework of optimal control as probabilistic inference using probabilistic graphical models. After a brief introduction to Markov decision processes, I will show the equivalence between the Belief Propagation algorithm and the Bellman optimality operator for a certain class of control problems that are efficiently solvable. I will then illustrate this equivalence using simple examples and using exact as well as approximate inference methods that you have seen in this course, e.g. junction tree and variational methods. Finally, we will also consider the Reinforcement Learning setting, where one needs to learn the model in addition to selecting optimal actions.