Learning Generalizable Feature Fields for Mobile Manipulation

Roger Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, JitengMu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, Xiaolong Wang

Learning Generalizable Feature Fields for Mobile Manipulation: 7 upvotes on Hugging Face Daily Papers, #8 of 8 papers on 2024-03-13. Day-by-day upvote history.

An open problem in mobile manipulation is how to represent objects and scenes in a unified manner, so that robots can use it both for navigating in the environment and manipulating objects. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherit to an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's ability to generalize to open-set objects as well as running time, when performing open-vocabulary mobile manipulation in dynamic scenes.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.