I thought I would write about some theory behind Reinforcement Learning, with eligibility traces, contraction mapping, POMDP and so on, but then I realized if I go down that rabbit hole, I would probably never finish this. So here are some practical stuff that people are actually using these days.
Two main approaches in model-free RL are policy-based, where we build a model of the policy function, and value-based, where we learn the value function. For many practical problems, the structure of the (unknown) policy function might be easier to learn than the (unknown) value function, in which case it makes more sense to use policy-based methods. Moreover, with policy-based methods, we will have an explicit policy, which is the ultimate goal of Reinforcement learning to control (the other type being learn-to-predict, more on that some other time). With value-based methods, like DQN, we will need to do an additional inference step to get the optimal policy.
The hybrid of policy-based and value-based is called Actor-Critic methods, which hopefully will be mentioned in another post.
One of the most straightforward approach in policy-based RL is, unsurprisingly, evolutionary algorithms. In this approach, a population of policies is maintained and evolutionized over time. People show that this works pretty well, e.g. for the Tetris game. However, due to the randomness, this is apparently only applied to problems where the number of parameters of the policy function is small.
A big part of policy-based methods, however, is based on Policy Gradient, where an exact estimate of the gradient of the expected future reward can be computed. Roughly speaking, there is an exact formulation for the gradient of the policy, which we can then use to optimize the policy. Since Deep Learning people basically worship gradients (pun intended), this method suites very well and became trendy about 2 years ago.
So, what is it all about? It is based on a very simple trick called Likelihood Ratio.
Let’s say we have a sequence of states and actions 
be the immediate reward at timestep t after the agent be in state
and performs action
.
be the total reward of the whole trajectory
be the transition model (for presentation purpose only, we won’t need to know this quantity at the end of the day, because we are in model-free RL).
be the policy function, which is basically a distribution of actions given state. We will build a model with parameters
and learn it from the observed sequences
‘s.
be the expected future reward.
Using the definition of the expectation, we have, where
is the probability of trajectory
under the policy
.
be the immediate reward at timestep t after the agent be in state 
and performs action 
.
be the total reward of the whole trajectory 
be the transition model (for presentation purpose only, we won’t need to know this quantity at the end of the day, because we are in model-free RL).
be the policy function, which is basically a distribution of actions given state. We will build a model with parameters ![U\left(\theta\right) = \mathbb{E}\left[\sum_{t=0}^{T} R\left(s_t,a_t\right);\pi_\theta\right]](http://s0.wp.com/latex.php?latex=U%5Cleft%28%5Ctheta%5Cright%29+%3D+%5Cmathbb%7BE%7D%5Cleft%5B%5Csum_%7Bt%3D0%7D%5E%7BT%7D+R%5Cleft%28s_t%2Ca_t%5Cright%29%3B%5Cpi_%5Ctheta%5Cright%5D&bg=ffffff&fg=333333&s=0 "U\left(\theta\right) = \mathbb{E}\left[\sum_{t=0}^{T} R\left(s_t,a_t\right);\pi_\theta\right]")
be the expected future reward.![U\left(\theta\right) = \mathbb{E}\left[\sum_{t=0}^{T} R\left(s_t,a_t\right);\pi_\theta\right] = \sum_\tau P\left(\tau;\theta\right)R\left(\tau\right)](http://s0.wp.com/latex.php?latex=U%5Cleft%28%5Ctheta%5Cright%29+%3D+%5Cmathbb%7BE%7D%5Cleft%5B%5Csum_%7Bt%3D0%7D%5E%7BT%7D+R%5Cleft%28s_t%2Ca_t%5Cright%29%3B%5Cpi_%5Ctheta%5Cright%5D+%3D+%5Csum_%5Ctau+P%5Cleft%28%5Ctau%3B%5Ctheta%5Cright%29R%5Cleft%28%5Ctau%5Cright%29&bg=ffffff&fg=333333&s=0 "U\left(\theta\right) = \mathbb{E}\left[\sum_{t=0}^{T} R\left(s_t,a_t\right);\pi_\theta\right] = \sum_\tau P\left(\tau;\theta\right)R\left(\tau\right)")
, where 
is the probability of trajectory 
.Now, the whole purpose of learning is to estimate the parameters 
Just like any good mathematician, we would just compute the derivative of 
This might look too trivial, but this simple trick makes the right hand side becomes the expectation of the function 
During training, what we do is we take the episodes one at a time, compute the above estimation of the gradient (i.e. using 
However, this estimation is extremely noisy (say high-variance), and data-inefficient. It is high variance mainly because of the sum of total rewards 
We will see later how to address those issues. For now, let’s finish computing the only missing piece on the right hand side:
![\displaystyle\begin{array}{rl} \nabla_\theta\log P\left(\tau^{(i)};\theta\right) & = \displaystyle \nabla \log\left[\prod_{t=0}^{T-1}P\left(s_{t+1}^{(i)}\vert s_t^{(i)},a_t^{(i)}\right)\pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\right] \\ \; & = \displaystyle \nabla_\theta\left[\sum_{t=0}^{T-1}\log P\left(s_{t+1}^{(i)}\vert s_t^{(i)},a_t^{(i)}\right) + \sum_{t=0}^{T-1}\log \pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\right] \\ \; & = \displaystyle \nabla_\theta\sum_{t=0}^{T-1}\log \pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\\ \; & = \displaystyle \sum_{t=0}^{T-1}\nabla_\theta\log \pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\end{array}](http://s0.wp.com/latex.php?latex=%5Cdisplaystyle%5Cbegin%7Barray%7D%7Brl%7D+%5Cnabla_%5Ctheta%5Clog+P%5Cleft%28%5Ctau%5E%7B%28i%29%7D%3B%5Ctheta%5Cright%29+%26+%3D+%5Cdisplaystyle+%5Cnabla+%5Clog%5Cleft%5B%5Cprod_%7Bt%3D0%7D%5E%7BT-1%7DP%5Cleft%28s_%7Bt%2B1%7D%5E%7B%28i%29%7D%5Cvert+s_t%5E%7B%28i%29%7D%2Ca_t%5E%7B%28i%29%7D%5Cright%29%5Cpi_%5Ctheta%5Cleft%28a_t%5E%7B%28i%29%7D%5Cvert+s_t%5E%7B%28i%29%7D%5Cright%29%5Cright%5D+%5C%5C+%5C%3B+%26+%3D+%5Cdisplaystyle+%5Cnabla_%5Ctheta%5Cleft%5B%5Csum_%7Bt%3D0%7D%5E%7BT-1%7D%5Clog+P%5Cleft%28s_%7Bt%2B1%7D%5E%7B%28i%29%7D%5Cvert+s_t%5E%7B%28i%29%7D%2Ca_t%5E%7B%28i%29%7D%5Cright%29+%2B+%5Csum_%7Bt%3D0%7D%5E%7BT-1%7D%5Clog+%5Cpi_%5Ctheta%5Cleft%28a_t%5E%7B%28i%29%7D%5Cvert+s_t%5E%7B%28i%29%7D%5Cright%29%5Cright%5D+%5C%5C+%5C%3B+%26+%3D+%5Cdisplaystyle+%5Cnabla_%5Ctheta%5Csum_%7Bt%3D0%7D%5E%7BT-1%7D%5Clog+%5Cpi_%5Ctheta%5Cleft%28a_t%5E%7B%28i%29%7D%5Cvert+s_t%5E%7B%28i%29%7D%5Cright%29%5C%5C+%5C%3B+%26+%3D+%5Cdisplaystyle+%5Csum_%7Bt%3D0%7D%5E%7BT-1%7D%5Cnabla_%5Ctheta%5Clog+%5Cpi_%5Ctheta%5Cleft%28a_t%5E%7B%28i%29%7D%5Cvert+s_t%5E%7B%28i%29%7D%5Cright%29%5Cend%7Barray%7D&bg=ffffff&fg=333333&s=0 "\displaystyle\begin{array}{rl} \nabla_\theta\log P\left(\tau^{(i)};\theta\right) & = \displaystyle \nabla \log\left[\prod_{t=0}^{T-1}P\left(s_{t+1}^{(i)}\vert s_t^{(i)},a_t^{(i)}\right)\pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\right] \\ \; & = \displaystyle \nabla_\theta\left[\sum_{t=0}^{T-1}\log P\left(s_{t+1}^{(i)}\vert s_t^{(i)},a_t^{(i)}\right) + \sum_{t=0}^{T-1}\log \pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\right] \\ \; & = \displaystyle \nabla_\theta\sum_{t=0}^{T-1}\log \pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\\ \; & = \displaystyle \sum_{t=0}^{T-1}\nabla_\theta\log \pi_\theta\left(a_t^{(i)}\vert s_t^{(i)}\right)\end{array}")
We could drop the first part of the sum because that is the transition model, and does not depends on
In practice, people would use a deep neural net to model the policy function 
The annoying thing though, is there is no explicit loss function in this case. We can only compute the derivative, while the true objective function is still intractable. Therefore, monitoring the training process will be quite manual.
In tensorflow, since there is no explicit loss function, we would need to use tf.gradients() and apply_gradients() separately to train the policy net.
There are couple of approaches for reducing the variance of the gradient estimate. One simple way is to use a baseline, which is the expected rewards over each episode. This, combined with a discount rate for the discounted future rewards, is used in the well-known REINFORCE paper. Using a discount rate though, will make the estimate become bias. A rate of 
More sophisticated approaches include Natural Gradients and Trust-Region Policy Optimization (TRPO), where a surrogate objective function is used instead of the
Yet another way is to pre-train the policy net to mimic (human) expert behavior in a supervised setting, of course only when such a labelled dataset is available. This is one of the trick used in the AlphaGo paper.
However, the trend nowadays is to use some form of Actor-Critic, which, as David Silver said, is the most general form of RL.
