Bellman Update and Synthetic Data in Q-Transformer
Notes on Q-learning, temporal difference, Monte Carlo, and others methods related to Q-Transformer.
Here are my notes on Q-learning and Q-transformer. Take it with grain of salt, as I am new in this area.
The Q-transformer is important paper, because it describes successful application of suboptimal synthetic (autonomously collected) data and transformer architecture in a robotic reinforcement learning problem.
Before Q-transformer let’s first talk about a bigger topic: Bellman Update in Reinforcement Learning.