Multimodal Language Grounding

Grounding language to other modalities

How do we effectively synergize learning from language by grounding to other modalities such as vision and motor control?



  1. Multimodal Knowledge Alignment with Reinforcement Learning
    Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Rowan Zellers, Prithviraj Ammanabrolu, Ronan Le Bras, Gunhee Kim, and Yejin Choi
    In Conference on Computer Vision and Pattern Recognition (CVPR), 2023
  2. Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
    Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox
    In International Conference on Machine Learning (ICML), 2023