A Survey of Reinforcement Learning Informed by Natural Language
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktäschel | IJCAI 2019
In a Nutshell 🥜
Luketina et al.1 survey the state of the field in the use of natural language for reinforcement learning tasks. The paper is motivated by that most traditional reinforcement learning approaches learn only from rewards or demonstrations. However, this results in low sample efficiency and RL agents that do not generalize well to unseen environments. This paper first surveys existing work on utilizing natural language in reinforcement learning then highlights key areas that could benefit from further investigation.
The paper divides the settings of natural language in reinforcement learning as language-conditional, in which language is part of the task formulation such as natural language instructions towards the goal or reward, and language-assisted, in which language is helpful but not necessary to solve a task such as information about the environment dynamics. In the language-conditional setting, the paper reviews instruction following, induction of reward from language, and environments that contain text in the action or observation space. In the language-assisted setting, the paper reviews transferring domain-specific textual resources such as game manuals, and as means of representing policies. In addition, the paper also describes that language information can be task-dependent such as tutorials or instructions for the task, or task-independent such as general priors about the world.
The paper then summarizes that most existing works focus on limited instruction following benchmarks that operate in closed-task domains and closed worlds. The language used are also often fairly simple and often synthetic, with small vocabulary sizes and multiple pieces of evidence to ground each word. Conversely, real natural language has beneficial properties such as the power-law distribution of word frequencies, complex composition, and “long tail” lexicon entities that can help force RL agents to generalize better. The paper particularly argues that pre-trained language models trained on large-scale unstructured or descriptive language corpora can be especially useful.
Some Thoughts 💭
This paper presents a succinct survey of natural language informed reinforcement learning and poses several interesting points for further exploration.
Given the recent advances in pre-trained language models and representation learning, incorporating them into reinforcement learning seems promising.
Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas, J., Grefenstette, E., ... & Rocktäschel, T. (2019). A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926.