DisCLIP: Open-Vocabulary Referring Expression Generation
Lior Bracha (Bar Ilan University),* Eitan Shaar (bar Ilan University), Aviv Shamsian (Bar Ilan University), Ethan Fetaya (Bar Ilan University), Gal Chechik (NVIDIA)
The 34th British Machine Vision Conference
Abstract
Referring Expressions Generation (REG) aims to produce textual descriptions that unambiguously identifies specific objects within a visual scene. Traditionally, this has been achieved through supervised learning methods, which perform well on specific data distributions but often struggle to generalize to new images and concepts. To address this issue, we present a novel approach for REG, named DisCLIP, short for discriminative CLIP. We build on CLIP, a large-scale visual-semantic model, to guide an LLM to generate a contextual description of a target concept in an image while avoiding other distracting concepts. Notably, this optimization happens at inference time and does not require additional training or tuning of learned parameters. We measure the quality of the generated text by evaluating the capability of a receiver model to accurately identify the described object within the scene. To achieve this, we use a frozen zero-shot comprehension module as a critique of our generated referring expressions. We evaluate DisCLIP on multiple referring-expression benchmarks through human evaluation and show that it significantly outperforms previous methods on out-of-domain datasets. Our results highlight the potential of using pre-trained visual-semantic models for generating high-quality contextual descriptions in new visual domains.Video
Citation
@inproceedings{Bracha_2023_BMVC,
author = {Lior Bracha and Eitan Shaar and Aviv Shamsian and Ethan Fetaya and Gal Chechik},
title = {DisCLIP: Open-Vocabulary Referring Expression Generation},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0670.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by