Reinforcement Learning from Human Feedback

How to scale RL to combinatorially sized language action spaces and messy human preference rewards?

The ultimate aim of language technology is to interact with humans. However, most such systems are trained without direct signals of human preference, with supervised target strings serving as (a sometimes crude) proxy. This work focuses on using reinforcement learning to interact and align to human preferences.



  1. Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
    Rajkumar Ramamurthy*, Prithviraj Ammanabrolu*, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi
    In International Conference on Learning Representations (ICLR), 2023
  2. Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
    Ximing Lu, Faeze Brahman, Peter West, Jaehun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Yuchen Lin, Skyler Hallinan, Xiang Ren, Sean Welleck, and Yejin Choi
    arXiv preprint arXiv:2305.15065, 2023
  3. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi
    In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023


  1. How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds
    Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur Szlam, Tim Rocktäschel, and Jason Weston
    In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun 2021


  1. Interactive fiction games: A colossal adventure
    Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan
    In Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2020


  1. Improvisational storytelling agents
    Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, Shruti Singh, Brent Harrison, Murtaza Dhuliawala, Pradyumna Tambwekar, Animesh Mehta, Richa Arora, Nathan Dass, and  others
    In Workshop on Machine Learning for Creativity and Design (NeurIPS 2017), Jun 2017