Reinforcement learning notes

Read these in order: foundations first (MDP, POMDP, HMM), then tabular RL, then function approximation and policy methods, and finally model-based RL. Mainly based on Probabilistic AI (A. Krause) and Sutton & Barto.

1. Ch. 1Markov decision processesFinite MDPs, policies, value functions; Bellman expectation and optimality; policy and value iteration.
2. Ch. 2POMDPsPartial observability, belief states and the Bayes update, reduction to an enlarged MDP.
3. Ch. 3Hidden Markov modelsHidden states and emissions, connection to POMDPs.
4. Ch. 4Tabular RLOn/off-policy, model-based vs model-free, ε-greedy and optimism (R_max), TD-learning, SARSA, Q-learning.
5. Ch. 5RL with function approximationDQN and Double DQN, policy gradients, REINFORCE, actor-critic, TRPO/PPO, DDPG, SAC.
6. Ch. 6Model-based approximate RLPlanning with MPC, learning dynamics and rewards, exploration with Thompson sampling and optimism.