Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, Amanpreet Singh

Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language: 29 upvotes on Hugging Face Daily Papers, #1 of 6 papers on 2023-06-29. Day-by-day upvote history.

We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs). Our system uses a language model to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever. We open-source our code at https://github.com/ContextualAI/lens and provide an interactive demo.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.