What is a decision transformer and how does it translate RL into supervised learning?

Decision Transformer is an approach in Reinforcement Learning (RL) that formulates the agent's training task as a supervised learning problem. Instead of traditional training through interaction with the environment and policy updates based on rewards, Decision Transformer is trained on historical data (trajectories) using a transformer architecture.

Main idea:

• The input to the model consists of sequences of states, actions, and target returns (return-to-go).
• The model learns to predict the next action, considering the desired level of reward.

Thus, the RL task reduces to predicting the next action based on precedents, allowing the use of supervised learning methods by training the transformer on pre-collected data without the need for interactive training.

This enables:

• Using large datasets without online interaction.
• Training models capable of generalization and planning.

Approximately, the Decision Transformer takes as input:

(states, actions, returns_to_go) -> next_action

and is trained to minimize the prediction error of the action.