Language-driven Grasp Detection

Abstract

Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work.

1 Introduction

Imagine we want an assistant robot to grasp a cup among a clutter of daily objects such as a knife, a fork, a cup, and a pair of scissors. Conventionally, to convey the idea of grasping this specific object, humans use the natural language command, “give me the cup”, for instance. Although humans intuitively know how to grasp the cup given the linguistic command, determining specific grasp actions for objects based on natural language instructions or language-driven grasp detection remains challenging for robots [78]. First, natural language is usually overlooked in existing grasp datasets [69] while training vision-and-language neural networks necessitates an excessive number of labeled examples [80]. Second, recent works usually focus on particular manipulation tasks with limited objects [28], imposing a bottleneck for in-the-wild robot execution [77]. Finally, despite recent developments, bridging the gap between language, vision, and control for real-world robotic experiments remains a challenging task [100].

Recently, language-driven robotic frameworks are gaining traction, offering the potential for robots to process natural language, and bridging the gap between robotic manipulations and real-world human-robot interaction [63]. PaLM-E [24], EgoCOT [63], and ConceptFusion [38] are some notable embodied robots with the ability to comprehend natural language by harnessing the power of large foundation models such as ChatGPT [67]. However, most works assume the high-level actions of robots and ignore the fundamental grasping actions, restricting the structure for generalization across robotic domains, tasks, and skills [62]. In this paper, we explore training a language-driven agent to implement low-level actions, focusing on the task of object grasping via image observations. Specifically, our hypothesis is centered around the establishment of a robotic system that can execute grasping actions following a given language instruction for any universal object.

We first present Grasp-Anything++ to serve as a large-scale dataset for language-driven grasp detection. Our dataset is based on the Grasp-Anything [87], and is synthesized from foundation models. Compared to the original Grasp-Anything dataset, we provide more than 10M grasp prompts and 3M associated object masks, 6M ground truth poses at the object part level. Our dataset showcases the ability to facilitate grasp detection using language instructions. We label the ground truth at both the object level and part level, providing a comprehensive understanding of real-world scenarios. For example, our ground truth includes both general instructions “give me the knife” and detail ones such as “grasp the handle of the steak knife”. We empirically show that our large-scale dataset successfully facilitates zero-shot grasp detection on both vision-based tasks and real-world robotic experiments.

To tackle the challenging language-driven grasp detection task, we propose a new diffusion model-based method. Our selection of diffusion models is motivated by their proven efficacy in conditional generation tasks [33]. These models have shown efficiency beyond image synthesis, including other image-based tasks such as image segmentation [92], and visual grounding [51]. Despite achieving notable success, integrating visual and text features effectively remains a challenge [14] as the majority of existing literature employs latent strategies to combine visual and text features [46]. We address this challenge by employing a new training strategy for learning text and image features, focusing on the use of feature maps as guidance information for grasp pose generation. Our main contribution is a new training objective that incorporates the feature maps and explicitly contributes to the denoising process. In summary, our contributions are three-fold:

• We propose Grasp-Anything++, a large-scale language-driven dataset for grasp detection tasks.

• We propose a diffusion model with a training objective that explicitly contributes to the denoising process to detect the grasp poses.

• We demonstrate that our Grasp-Anything++ dataset and the proposed method outperform other approaches and enable successful robotic applications.

2 Related Work

Grasp Detection. Grasp detection is a popular task in both computer vision and robotic community [18, 65, 78, 2, 27]. Recently, establishing robotic systems with the ability to follow natural commands has been actively researched [100, 78, 96]. The prevalent solution to the language-driven grasp detection task is to split into two stages: one for grounding the target object, and the other is to synthesize grasp poses from the grounding visual-text correlations [96, 1]. Training in two stages may result in longer inference time [53]. In addition, several works [100, 98, 107] adopt foundation models, such as GroundDINO [54] and GPT-3 [8]. Accessing such commercial foundation models is not always available [85], especially on robotic systems with limited resources or unstable internet connection [47]. In our work, we directly train the model on the large-scale Grasp-Anything++ dataset to inherit the power of a foundation-based dataset, while ensuring a straightforward inference process for the downstream robotic applications.

Language-driven Grasp Detection Datasets. While there are many grasp datasets have been introduced [39, 18, 68, 49, 94, 57, 61, 25, 27, 60, 26, 10], the majority of them overlook the text modality. Therefore, the grasping of objects out of a clutter typically experiences ambiguities in what object to grasp [105]. DailyGrasp [100] is one of the first grasp datasets employing natural language for scene descriptions; however, the scene description corpus in this dataset is relatively small and does not specify which part of the object should be grasped. In our work, we present Grasp-Anything++, which is a large-scale language-driven grasping dataset. Furthermore, Grasp-Anything++ describes the grasping object at both the part level and object level, providing more information for the robot to execute the grasping [103].

Step	Description
Scene Generation	User	Please help me generate scene descriptions for natural arrangements of daily objects. Each description has the following form: <Object_1><Object_2>…<Verb><Container_Object>. Please also ensure the incorporation of a rich and varied lexicon in the scene descriptions.
	Sample	A steel knife, a polished fork and a pristine ceramic plate on a wooden table.
	Text-to- Image	We use Stable Diffusion [74] to proceed text-to-image generation.
Object Masking	User	For object part-level description, given an input list {<Object_1>, <Object_2>, …}, the output will be a list that describes the parts of objects as: {<Object_1>: [<Part_1.1>, <Part_1.2>, …], <Object_2>: [<Part_2.1>, <Part_2.2>, …]}.
	Sample	{knife: [handle, blade], fork: [handle, neck, stem, tines], plate: [rim, base]}
	Post Process	We use OFA [89] and SAM [42] to locate the region describing the objects.
Part Masking	User	Given the object list and part lists of each scene description, you will generate for me all prompts with the following format: {<Manipulation_Action><Object_ID><Part_ID>}. The part that is more suitable for human grasping is positioned at the start of the list to represent the grasping actions.
	Sample	Give me the steel knife; Grasp the knife at its handle.
	Post Process	We leverage VLPart [81] to locate the region describing the parts of objects.
Grasp Generation	User	Generate for me a scene description with grasp instructions following the templates.
	Sample	Scene description: A steel knife, a polished fork and a pristine ceramic plate on a wooden table. Object list: {knife, fork, plate}. Part lists: {knife: [handle, blade], fork: [handle, neck, stem, tines], plate: [rim, base]}. Prompts: Give me the steel knife; Grasp the knife at its handle.
	Grasp Labelling	We utilize a pretrained RAGT-3/3 [10] to generate grasp poses corresponding to the located region.

Table 1: Grasp-Anything++ creation pipeline. We utilize ChatGPT to generate scene descriptions and grasp instructions from the user input. We generate images given scene descriptions and automatically synthesize the grasp poses.

Diffusion Models for Robotic Applications. Diffusion models [33] have emerged as the new state-of-the-art method of generative tasks [101]. Recently, we have witnessed growing attention for utilizing diffusion models in robotic applications [75]. Liu et al. [55] propose a diffusion model to handle the language-guided object rearrangement task. Diffusion models are also applied to other robotic tasks such as motion planning [11], and trajectory optimization [37]. The authors in [84] present a diffusion model to determine grasp poses by minimizing a SDF loss. Overall, the diffusion models employed in previous works often combine visual and text features in a latent mechanism [7], which may cause interpretability problems [100] for robotic systems that require low-level controls [22]. To tackle this challenge, we propose a training objective that explicitly contributes to the denoising process. We demonstrate that our proposed strategy is theoretically supported and is more effective than the latent strategy.

3 The Grasp-Anything++ Dataset

We utilize large-scale foundation models to create the Grasp-Anything++. Our dataset offers open-vocabulary grasping commands and images with associated groundtruth. There are three key steps in establishing our dataset: i) prompting procedure, ii) image synthesis and grasp poses annotation, and iii) post-processing.

3.1 Prompting Procedure

We first establish prompt-based procedures to generate a large-scale scene description corpus as well as grasp prompt instructions. In particular, we utilize ChatGPT to generate the prompts for two tasks: i) Scene descriptions: Sentences capturing the scene arrangement, including the extracted object and part lists, and ii) Grasp instructions: Prompts directing the robot to grasp specific objects or parts.

We follow a procedure in Table 1 to implement ChatGPT’s output templates. The reference target in the grasp instruction may be either an object or an object’s part. When the reference is an object’s part, that part is directly selected as the reference in the grasp instruction sentence. If the reference is an object, we determine the grasping region on the part of the object that is likely to be grasped in everyday scenarios as described in affordance theory [66].

3.2 Image Synthesis and Grasp Annotation

Image Synthesis. Given the scene description corpus, we first utilize a large-scale pretrained text-to-image model, namely, Stable Diffusion [74] to generate images from scene descriptions. Next, we perform a series of visual grounding and image segmentation using OFA [89], Segment-Anything [42], and VLPart [81] to locate the referenced object or part to the grasp instruction.

(a) Number of categories

(b) Scene description length

(c) Number of object/part references.

Figure 2: Data statistics. We analyze object categories and the use of natural language in the Grasp-Anything++ dataset.

Grasp Annotation. Grasp poses are represented as 2D rectangles, consistent with prior research and practical compatibility with real-world parallel grippers [39, 18]. Utilizing a pretrained network [10], we annotate grasp poses based on part segmentation masks. Since potential inaccuracies in these candidate poses could occur, we follow the procedure as defined in [87] to evaluate the quality of generated grasp poses to discard unreasonable grasp poses.

Specifically, grasp quality is evaluated through net torque $\mathcal{T}=\left(\tau_{1}+\tau_{2}\right)-RMg\text{~{},}$ where resistance at contact points is $\tau_{i}=K\mu_{s}F\cos\alpha_{i}$. With constants such as $M$ (mass), $g$ (gravitational acceleration), $K$ (geometrical characteristics), $\mu_{s}$ (static friction coefficient), and $F$ (applied force), accurately determining $\mathcal{T}$ directly is challenging due to the physical difficulties in precisely measuring $M$, $K$, and $\mu_{s}$. Thus, we employ a surrogate measure, $\tilde{\mathcal{T}}=\dfrac{\cos\alpha_{1}+\cos\alpha_{2}}{R}\text{~{},}$ as an alternative. As a result in [13], antipodal grasps score higher on $\tilde{\mathcal{T}}$, indicating better quality. Consequently, grasps are evaluated based on $\tilde{\mathcal{T}}$, with positive values indicating positive grasps and others considered as negative.

Post Processing. Despite training on extensive datasets, Stable Diffusion [74] may produce subpar content, commonly termed as hallucination[36] when generating images from the text prompts. To address this, we perform manual reviews to filter out such images, with qualitative examples in our figures. Our process includes checks at every stage to prevent duplicate or hallucinated content. However, manual inspection introduces biases, which we counter with guidelines focusing on abnormal structures or implausible gravity (Fig. 3), aligns with approach in the literature [76].

Figure 3: Failure image generation cases. Images generated by Stable Diffusion [74] may exhibit hallucinatory artifacts such as sunglasses lacking a lens, scissors with an anomalous structure, and a spoon not resting properly on a table.

Additionally, ChatGPT-generated scene prompts often duplicate [56]. To address this, we use duplication checking, filtering out identical prompts with BERTScore [106], which assesses sentence similarity through cosine similarities of token embeddings. We remove sentences with a BERTScore above 0.85 as in prior study [90].

3.3 Data Statistics

Number of Categories. To evaluate the object category diversity, we apply a methodology akin to that in [17]. Utilizing 300 categories from LVIS dataset [29], we employ a pretrained model [108] to identify 300 candidate objects from our dataset for each category. We then curate a subset comprising 90,000 objects, refining it by excluding items that do not align semantically with their designated categories. A category is considered significant if it has more than 40 objects. Fig. 2(a) shows the results. Overall, our dataset spans over $236$ categories from LVIS dataset, indicating a notable degree of object diversity in our dataset.

Scene Descriptions. Fig. 2(b) shows the distribution of scene descriptions based on sentence length. The analysis reveals a wide range of sentence lengths, spanning from 10 words to 100 words per sentence. On average, each scene description consists of approximately 54 words, indicative of detailed and descriptive sentences. These scene descriptions correspond to sets of grasp instructions. Fig. 2(c) further shows the objects and object parts in scene descriptions.

Diversity Analysis. We assess the diversity of occlusion and lighting conditions in the dataset. Regarding occlusion, we use a pretrained YOLOv5 model to identify objects within images. The results indicate that $93.8\%$ of images have a substantial overlap of five or more bounding boxes, which suggests a diverse range of occlusion within the Grasp-Anything++ dataset. Regarding lighting conditions, we convert images to YCbCr to analyze Y channel (luminance) and find that GraspL1M has the most diverse lighting conditions, identifying by the lowest Gini coefficient (a metric to measure the inequality of a distribution) of $0.26$, compared to VMRD [104] ($0.31$), OCID-grasp [2] ($0.32$), Cornell [39] ($0.62$), Jacquard [18] ($0.91$).

4 Language-driven Grasp Detection

Motivation. The use of diffusion model for language-driven grasp detection is motivated by its efficiency in various generative tasks [33, 101, 44, 55, 16]. Conditional generation, such as our language-driven grasp detection task, aligns seamlessly with diffusion models’ capabilities [34]. Moreover, language-driven grasp detection represents a fine-grained problem in which the outputs strongly depend on the text input [4]. For example, “grasp the steak knife” and “grasp the kraft knife” refer to two different objects on the image. To this end, we propose using contrastive loss with diffusion model to tackle this task, as contrastive learning is a popular solution for fine-grained tasks [9, 21, 102].

4.1 Constrastive Loss for Diffusion Model

We represent the target grasp pose as $\mathbf{x}_{0}$ in the diffusion model. The objective of our diffusion process of language-driven grasp detection involves denoising from a noisy state $\mathbf{x}_{T}$ to the original grasp pose $\mathbf{x}_{0}$, conditioned on the input image and grasp instruction represented by $y$.

In a diffusion process [33], assume that $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})$ is the forward process and we parameterize the reverse process by $p_{\theta}(\mathbf{x}_{0:T})$. The conditional diffusion process [20] assumes $\hat{q}$ is the forward process but with the inclusion of a condition $y$. The goal of the reverse process is to optimize the variational bound on negative log likelihood [33]

$$\mathcal{L}=\mathbb{E}\left[-\log p_{\theta}(\mathbf{x}_{T})-\sum_{t\geq 1}\log\frac{p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}{q(\mathbf{x}_{t}|\mathbf{x}_{t-1})}\right]\text{~{}\@.}$$ (1)

We prove in the Appendix that

$$\begin{gathered}\mathcal{L}=\mathbb{E}\bigg{[}\underbrace{C-\log p_{\theta}(\mathbf{x}_{T})}_{\textrm{Constant}}+\underbrace{\log\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)}_{\textrm{Contrastive}}+\\ \underbrace{\sum_{t>1}D_{\text{KL}}(\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)\|p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}))-\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1},y)}_{\textrm{Denoising score}}\bigg{]}\text{~{}\@.}\end{gathered}$$ (2)

The terms $D_{\text{KL}}(\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)\|p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}))$ and $\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1},y)$ in Equation 2 are similar to the concept of denoising score used in [33]. Thus, we can represent the quantity $\sum_{t>1}D_{\text{KL}}(\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)\|p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}))-\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1},y)$ by the loss utilized in [82, 83]

$$\mathcal{L}_{\rm{diffusion}}=\mathbb{E}_{\mathbf{x}_{0}\sim q(\mathbf{x}_{0}|y),t\sim[1,T]}\left[\mathbf{x}_{0}-f(\mathbf{x}_{t+1},t+1,\tilde{\mathbf{x}}_{0})\right]^{2}\text{~{}\@.}$$ (3)

Since $q(\mathbf{x}_{T}|\cdot)$ is equivalent to an isotropic Gaussian distribution as $T\rightarrow+\infty$, the estimation quantity $\log p_{\theta}(\mathbf{x}_{T})$ converges to a constant when $\theta\rightarrow\theta^{\ast}$. Therefore, we can ignore the first term of Equation 2.

Finally, the term ${\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)}$ of Equation 2 provides more information about the relation between $\mathbf{x}_{T}$ and $\mathbf{x}_{0}$. As this term is intractable [79], we parameterize ${\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)}$ as $p_{\psi,y}(\mathbf{x}_{0},\mathbf{x}_{T})$. This estimation resembles the noise-contrastive estimation [30], where the $\mathbf{x}_{T}$, $\mathbf{x}_{0}$ can be considered as a pair of contrastive estimation and $\psi$ can be estimated by a contrastive loss.

In [88, 20, 82], the authors indicate that predicting $\mathbf{x}_{0}$ is often infeasible but predicting an estimation $\tilde{\mathbf{x}}_{0}$ is tractable and can be used as a ‘pseudo’ estimation of $\mathbf{x}_{0}$. We denote $\tilde{\mathbf{x}}_{0}$ as an estimation of $\mathbf{x}_{0}$. The loss term ${\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)}$ can be approximated by using the following contrastive loss

$$\mathcal{L}_{\rm{contrastive}}=\max\left(0,\left\|\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}}\right\|_{2}^{2}-M\right),$$ (4)

where $M$ is the number of dimension of $\mathbf{x}_{0}$, and $\alpha_{t}$ is the variance schedule at timestep $t$ ($t=\overline{1;T}$).

Proposition 1.

Suppose that $\tilde{\mathbf{x}}_{0},\,\mathbf{x}_{0}$ and $\epsilon$ are independent, and that

$$\left\|\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}}\right\|_{2}^{2}\geq M\text{~{}\@.}$$

Then there exists $C>0$ such that: for arbitrary $\delta>0$, if $\mathcal{L}_{\rm{contrastive}}<\delta$, then

$$\mathbb{E}\left[\|\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}\|_{2}^{2}\right]<C\delta\text{~{}\@.}$$

Proof.

See Supplementary Material. ∎

Remark 1.1.

• Proposition 1 suggests that if the contrastive loss $\mathcal{L}_{\rm{contrastive}}$ tends to zero, then the prediction $\tilde{\mathbf{x}}_{0}$ will approach the ground truth $\mathbf{x}_{0}$.

Remark 1.2.

• The condition $\left\|\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}}\right\|_{2}^{2}\geq M$ is suitable for our language-driven grasp detection task as $\tilde{\mathbf{x}}_{0}$ and $\mathbf{x}_{T}$ are two contrastive quantities, therefore, we can assume there is a minimum distance between $\tilde{\mathbf{x}}_{0}$ and $\mathbf{x}_{T}$. In addition, in the proof of Proposition 1, we see that $\mathbb{E}\left[\left\|\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}}\right\|_{2}^{2}-M\right]=\beta^{2}\mathbb{E}\left[\|\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}\|_{2}^{2}\right]$, which is always nonnegative. Therefore, it is both theoretically and experimentally reasonable to add this assumption.

Figure 4: Language-drive Grasp Detection (LGD) network. We present the network architecture (left) and the proposed training objectives of the denoising process (right).

4.2 Language-driven Grasp Detection Network

Figure 5: Language-driven grasp detection results visualization.

Network. Our network operates on two conditions: an image denoted as I and a corresponding text prompt represented as $e$. To process these conditions, we employ a vision encoder to extract visual features from I and a text encoder to derive textual embeddings from $e$. The resulting feature vectors, denoted as $\textit{{I}}^{\prime}$ and $e^{\prime}$, are subsequently subjected to a fusion module, ALBEF [50]. We leverage the attention mask generated by the ALBEF fusion module as the estimation $\tilde{\mathbf{x}}_{0}$ of $\mathbf{x}_{0}$. Next, we aggregate three elements: the estimation region $\tilde{\mathbf{x}}_{0}$, the grasp pose at the current timestep $\mathbf{x}_{t+1}$, and the timestep $t+1$. These inputs are combined using MLP layers, similar to the approach outlined in [82]. Specifically, the output operation can be expressed as: $\mathbf{x}_{t}=f(\mathbf{x}_{t+1},t+1,\tilde{\mathbf{x}}_{0})$, where function $f$ encompasses a composition of multiple MLP layers. Additional specifics regarding these universal MLP layers are provided in the Supplementary Material.

Training Objective. In our context, conditioned grasp detection models the distribution $p(\mathbf{x}_{0}|y)$ as the reversed diffusion process of gradually cleaning $\mathbf{x}_{t+1}$. Instead of predicting $\mathbf{x}_{t}$ as formulated by [33], we follow Ramesh et al. [72] and predict the signal itself, i.e., $\mathbf{x}_{t}=f(\mathbf{x}_{t+1},t+1,\tilde{\mathbf{x}}_{0})$ with the simple objective [33]. To this end, we utilize the contrastive loss as in Equation 4 to explicitly improve the learning objective of the denoising process:

$$\mathcal{L}_{\rm{total}}=\mathcal{L}_{\rm{contrastive}}+\mathcal{L}_{\rm{diffusion}}\text{~{}\@.}$$ (5)

5 Experiments

We conduct experiments to evaluate our proposed method and Grasp-Anything++ dataset using both the vision-based metrics and real robot experiments. We then demonstrate zero-shot grasp results and discuss the challenges and open questions for future works.

5.1 Language-driven Grasp Detection Results

Baselines. We compare our language-driven grasp detection method (LGD) with the linguistically supported versions of GR-CNN [43], Det-Seg-Refine [2], GG-CNN [59], CLIPORT [78] and CLIP-Fusion [96]. In all cases, we employ a pretrained CLIP [71] or BERT [19] as the text embedding. The implementation details of all baselines can be found in our Supplementary Material.

Setup. To assess the generalization of all methods trained on Grasp-Anything++, we utilize the concept of base and new labels [109] in zero-shot learning. We categorize LVIS labels from Section 3.3 to form labels for our experiment. In particular, we select 70% of these labels by frequency for ‘Base’ and assign the remaining 30% to ‘New’. We also use the harmonic mean (‘H’) to measure the overall success rates [109]. Our primary evaluation metric is the success rate, defined similarly to [43], necessitating an IoU score of the predicted grasp exceeding $25\%$ with the ground truth grasp and an offset angle less than $30^{\circ}$.

Baseline	Seen	Unseen	H
GR-ConvNet [43] + CLIP [71]	0.37	0.18	0.24
Det-Seg-Refine [2] + CLIP [71]	0.30	0.15	0.20
GG-CNN [59] + CLIP [71]	0.12	0.08	0.10
CLIPORT [78]	0.36	0.26	0.29
CLIP-Fusion [96]	0.40	0.29	0.33
LGD (ours) + BERT [19]	0.44	0.38	0.41
LGD (ours) + CLIP [71]	0.48	0.42	0.45

Table 2: Language-driven grasp detection results.

Main Results. Table 2 shows the results of language-driven grasp detection on the Grasp-Anything++ dataset. The findings indicate a notable performance advantage of our LGD over other baseline approaches, with LGD outperforming the subsequent best-performing baselines (CLIP-Fusion) by margins of $0.14$ on Grasp-Anything++ dataset.

Baseline	Seen	Unseen	H
LGD w/o predicting $\tilde{\mathbf{x}}_{0}$	0.15	0.08	0.10
LGD w/o contrastive loss	0.45	0.40	0.42
LGD w contrastive loss	0.48	0.42	0.45

Table 3: Contrastive loss analysis.

Figure 6: Loss visualization.

Figure 7: t-SNE visualization. We apply t-SNE to cluster the vision-and-language features with and without contrastive loss.

Contrastive Loss Analysis. Table 3 presents the performance of LGD under varied configurations. The outcomes emphasize the substantial influence of the training objective (contrastive loss) and the importance of language instructions in enhancing LGD performance on both seen and unseen classes in the Grasp-Anything++ dataset.

Fig. 6 shows the contrastive loss $\mathcal{L}_{\text{contrastive}}$ approaching towards $0$ during training, indicating the grasp pose estimation $\tilde{\mathbf{x}}_{0}$ aligns with the ground truth $\mathbf{x}_{0}$, as anticipated by Proposition 1. The subsequent attention maps visualization in Fig. 8 shows the attention region is meaningful and improves the results when employing our proposed contrastive loss compared to its absence. Moreover, we employ t-SNE for vision-and-language embedding visualization, as in [58], by processing $2,000$ samples from the Grasp-Anything++ dataset through the ALBEF module. The outcomes reveal that our contrastive loss facilitates better object classification, as evidenced in Fig.7 by clearer segregation of pixel embeddings across various semantic classes, underscoring contrastive loss’s role in refining embeddings’ differentiation for improved class distinctions.

Qualitative Results. Fig. 5 presents qualitative results of the language-driven grasp detection task, suggesting that our LGD method generates more semantically plausible than other baselines. Despite satisfactory performance, LGD occasionally predicts incorrect results, with a detailed analysis of these cases available in our Appendix.

Figure 8: Attention map visualization. We compare the attention map when utilizing our proposed contrastive loss and when not.

Baseline	Single	Cluttered
GR-ConvNet [43] + CLIP [71]	0.33	0.30
Det-Seg-Refine [2] + CLIP [71]	0.30	0.23
GG-CNN [59] + CLIP [71]	0.10	0.07
CLIPORT [78]	0.27	0.30
CLIP-Fusion [96]	0.40	0.40
LGD (ours)	0.43	0.42

Table 4: Robotic language-driven grasp detection results.

Robotic Validation. We provide quantitative results by integrating our language-driven grasp detection pipeline for a robotic grasping application with a KUKA LBR iiwa R820 robot. Using the RealSense D435i camera, the grasp pose inferred from approaches in Table 4 is transformed into the 6DoF grasp pose, similar to [43]. The optimization-based trajectory planner in [86, 6] is employed to execute the grasps. Experiments are conducted for two scenarios, i.e., the single object scenario and the cluttered scene scenario, of a set of $20$ real-world daily objects. In each scenario, we run $30$ experiments using baselines listed in Table 4 and a predefined grasping prompt corpus. The results exhibit that our LGD outperforms other baselines. Furthermore, although LGD is trained on our Grasp-Anything++ which is a solely synthesis dataset created by foundation models, it still shows reasonable results on real-world objects.

	Grasp-Anything++ (ours)			Jacquard [18]			Cornell [39]			VMRD [104]			OCID-grasp [2]
Baseline	Base	New	H	Base	New	H	Base	New	H	Base	New	H	Base	New	H
GR-ConvNet [43]	0.71	0.59	0.64	0.88	0.66	0.75	0.98	0.74	0.84	0.77	0.64	0.70	0.86	0.67	0.75
Det-Seg-Refine [2]	0.62	0.57	0.59	0.86	0.60	0.71	0.99	0.76	0.86	0.75	0.60	0.66	0.80	0.62	0.70
GG-CNN [59]	0.68	0.57	0.62	0.78	0.56	0.65	0.96	0.75	0.84	0.69	0.53	0.59	0.71	0.63	0.67
LGD (no text) (ours)	0.74	0.63	0.68	0.89	0.69	0.77	0.97	0.76	0.85	0.79	0.66	0.72	0.88	0.68	0.76

Table 5: Base-to-new zero-shot grasp detection results.

5.2 Zero-shot Grasp Detection

Our proposed Grasp-Anything++ is a large-scale dataset. Apart from the language-driven grasp detection task, we believe it can be used for other purposes. In this experiment, we seek to answer the question: Can Grasp-Anything++ be useful in the traditional grasp detection task without text? Consequently, we verify our Grasp-Anything++ and LGD (no text) with other existing datasets and grasping methods.

Setup. We setup an LGD (no text) version, and other state-of-the-art grasp detection methods GR-ConvNet [43], Det-Seg-Refine [2], GG-CNN [59]. We use five datasets: our Grasp-Anything++, Jacquard [18], Cornell [39], VMRD [104], and OCID-grasp [2] in this experiment.

Zero-shot Results. Table 5 summarizes the base-to-new grasp detection results on five datasets. Overall, the performance of LGD even without the language branch is better than other baselines across all datasets. Furthermore, this table also shows that our Grasp-Anything++ dataset is more challenging to train as the detection results are lower than related datasets using the same approaches due to the greater coverage of unseen objects in the testing phase.

	Jacquard	Cornell	VMRD	OCID-grasp	Grasp-Anything++
Jacquard [18]	0.87	0.51	0.13	0.21	0.17
Cornell [39]	0.07	0.98	0.20	0.12	0.13
VMRD [104]	0.06	0.21	0.79	0.11	0.10
OCID-grasp [2]	0.09	0.12	0.20	0.74	0.11
Grasp-Anything++ (ours)	0.41	0.63	0.30	0.39	0.65

Table 6: Cross-dataset grasp detection results.

Cross-dataset Evaluation. To further verify the usefulness of our Grasp-Anything++ dataset, we conduct the cross-dataset validation in Table 6. We use the GR-ConvNet [43] to reuse its results on existing grasp datasets. GR-ConvNet is trained on a dataset (row) and evaluated on another dataset (column). For example, training on Jacquard and testing on Cornell yields an accuracy of $0.51$. Notably, training with our dataset improves performance by approximately $10-33\%$ compared to other datasets.

In the wild grasp detection. Fig. 9 shows visualization results using LGD (no text) trained on our Grasp-Anything++ dataset on random internet images and other datasets images. We can see that the detected grasp poses are adequate in quality and quantity. This demonstrates that although our Grasp-Anything++ is fully created by foundation models without having any real images, models trained on our Grasp-Anything++ dataset still generalize well on real-world images.

Figure 9: In the wild grasp detection. Top row images are from GraspNet [27], YCB-Video [94], NBMOD [10] datasets; bottom row shows internet images.

5.3 Discussion

Our experiments indicate that Grasp-Anything++ can serve as a foundation dataset for both language-driven and traditional grasp detection tasks. However, there are certain limitations. First, our dataset lacks depth images for directly being applied to robotic applications [64]. Second, we remark that the creation of our dataset is time-consuming and relies on access to the ChatGPT API. Fortunately, future research can reuse our provided assets (images, prompts, etc.) without starting from scratch. Furthermore, our experiments show that adding language to the grasp detection task (Table 2) poses a more challenging problem compared to standard grasp detection task (Table 5).

We see several interesting future research directions. First, future work could investigate the use of text or image-to-3D models [95] or image-to-depth [73] and reuse our dataset’s prompts and images to construct 3D language-driven grasp datasets. Additionally, beyond linguistic grasp instruction adherence, our dataset holds potential for varied applications, including scene understanding [97] and scene generation [3], hallucination analysis [36], and human-robot interaction [12].

6 Conclusion

We introduce Grasp-Anything++, a large-scale dataset with 1M images and 10M grasp prompts for language-driven grasp detection tasks. We propose LGD, a diffusion-based method to tackle the language-driven grasp detection task. Our diffusion model employs a contrastive training objective, which explicitly contributes to the denoising process. Empirically, we have shown that Grasp-Anything++ serves as a foundation grasp detection dataset. Finally, our LGD improves the performance of other baselines, and the real-world robotic experiments further validate the effectiveness of our dataset and approach.

Appendix A Theoretical Findings

In this section, we first show the derivation of Equation 4 in our main paper. We then show the proof of Proposition 1 in the main paper.

A.1 Derivation of Equation 4

It was indicated in [20] that $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{t-1},y)$, therefore, the loss in Equation 3 in our main paper can be written as

$$\mathcal{L}=\mathbb{E}\left[-\log p_{\theta}(\mathbf{x}_{T})-\sum_{t\geq 1}\log\frac{p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}{\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{t-1},y)}\right]\text{~{}\@.}$$ (6)

Using Bayes’ Theorem, we can further derive the term $\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{t-1},y)$ of Equation 6 as follows

$$\hat{q} (\mathbf{x}_{t}|\mathbf{x}_{t-1},y)=\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{t-1},y)}{\hat{q}(\mathbf{x}_{t-1},y)}$$

$$=\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{t-1},y)}{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{t-1},\mathbf{x}_{0},y)}\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{t-1},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1},y)}$$

$$=\frac{1}{\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},\mathbf{x}_{t},y)}\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{t-1},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{0},y)}\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1},y)}$$

$$=\frac{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},\mathbf{x}_{t},y)}\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1},y)}$$

$$=\frac{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},\mathbf{x}_{t},y)}\frac{\hat{q}(\mathbf{x}_{t},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1},\mathbf{x}_{0},y)}\frac{\hat{q}(\mathbf{x}_{t-1},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1},y)}$$

$$=\frac{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},\mathbf{x}_{t},y)}\frac{\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{0},y)}\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},y)\text{~{}\@.}$$

Follow by Ho et al. [33], we can assume that $\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},\mathbf{x}_{t},y)=\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{t-1},y)$ due to the Markov chain. Thus, $\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{t-1},y)$ can be further derived as follows

$$\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{t-1},y)=\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)\frac{\hat{q}(\mathbf{x}_{t}|\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{0},y)}\text{~{}\@.}$$ (7)

From Equation 6 and Equation 7, we can express the negative log likelihood loss as follows

$$\begin{gathered}L=\mathbb{E}\bigg{[}-\log\frac{p_{\theta}(\mathbf{x}_{T})}{\hat{q}(\mathbf{x}_{T}|\mathbf{x}_{0},y)}-\sum_{t>1}\log\frac{p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)}\\ -\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1},y)\bigg{]}\text{~{}\@.}\end{gathered}$$ (8)

By using Bayes’ Theorem again, we can formulate $\hat{q}(\mathbf{x}_{T}|\mathbf{x}_{0},y)$ of Equation 8 as follows

$$\hat{q}(\mathbf{x}_{T}|\mathbf{x}_{0},y) =\frac{\hat{q}(\mathbf{x}_{T},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{0},y)}$$

$$=\frac{\hat{q}(\mathbf{x}_{T},\mathbf{x}_{0},y)}{\hat{q}(\mathbf{x}_{0})}\frac{\hat{q}(\mathbf{x}_{0})}{\hat{q}(\mathbf{x}_{0},y)}$$

$$=\frac{\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)}{\hat{q}(y|\mathbf{x}_{0})}\text{~{}\@.}$$ (9)

Since $\hat{q}(y|\mathbf{x}_{0})$ is known labels per sample [20], thus, can be treated as a constant $C$. We conclude with the final derivation of Equation 8 by

$$L =\mathbb{E}\bigg{[}-\log\frac{p_{\theta}(\mathbf{x}_{T})}{\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)}+\log\hat{q}(y|\mathbf{x}_{0})$$

$$\qquad-\sum_{t>1}\log\frac{p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}{\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)}-\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1},y)\bigg{]}$$

$$=\mathbb{E}\bigg{[}C-\log p_{\theta}(\mathbf{x}_{T})+\log\hat{q}(\mathbf{x}_{0}|\mathbf{x}_{T},y)+$$

$$\qquad\sum_{t>1}D_{\text{KL}}(\hat{q}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0},y)\|p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}))$$

$$\qquad\qquad-\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1},y)\bigg{]}\text{~{}\@.}$$ (10)

A.2 Proof of Proposition 1

Proof.

The correlation between $\mathbf{x}_{0}$ and $\mathbf{x}_{T}$ is given by [33]

$$\mathbf{x}_{T}=\sqrt{\overline{\alpha}_{T}}\mathbf{x}_{0}+\sqrt{1-\overline{\alpha}_{T}}\mathbf{\epsilon},\mathbf{\epsilon}\sim\mathcal{N}(0,\mathbf{I})\text{~{}\@.}$$ (11)

It follows from Equation 11 that

$$\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}} =\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\left(\sqrt{\overline{\alpha}_{T}}\mathbf{x}_{0}+\sqrt{1-\overline{\alpha}_{T}}\mathbf{\epsilon}\right)}{\sqrt{1-\overline{\alpha_{T}}}}$$

$$=\beta(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0})-\epsilon\text{~{},}$$

with $\beta=\sqrt{\frac{\overline{\alpha}_{T}}{1-\overline{\alpha}_{T}}}$. Thus, under the condition $\|\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}}\|^{2}_{2}\geq M$, we have

$$\mathbb{E}[\mathcal{L}_{\text{contrastive}}] =\mathbb{E}\left[\left\|\frac{\sqrt{\overline{\alpha_{T}}}\tilde{\mathbf{x}}_{0}-\mathbf{x}_{T}}{\sqrt{1-\overline{\alpha_{T}}}}\right\|_{2}^{2}-M\right]$$

$$=\mathbb{E}\left[\|\beta(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0})-\epsilon\|_{2}^{2}-\|\epsilon\|_{2}^{2}\right]$$

$$=\mathbb{E}\left[\|\beta(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0})\|_{2}^{2}-2\left<\beta(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}),\epsilon\right>\right]$$

$$=\mathbb{E}\left[\|\beta(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0})\|_{2}^{2}\right]-2\beta\mathbb{E}\left[\left<(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}),\epsilon\right>\right]\text{~{}\@.}$$

However, since $\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}$ and $\epsilon$ are independent and $\mathbb{E}[\epsilon]=0$, we have

$$\mathbb{E}\left[\left<(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}),\epsilon\right>\right]=\left<\mathbb{E}\left[\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}\right],\mathbb{E}\left[\epsilon\right]\right>=0\text{~{}\@.}$$

Thus,

$$\mathbb{E}[\mathcal{L}_{\text{contrastive}}]=\mathbb{E}\left[\|\beta(\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0})\|_{2}^{2}\right]=\beta^{2}\mathbb{E}\left[\|\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}\|_{2}^{2}\right]\text{~{}\@.}$$

Hence, with $C=\beta^{-2}$, we have $\mathbb{E}\left[\|\tilde{\mathbf{x}}_{0}-\mathbf{x}_{0}\|_{2}^{2}\right]\leq C\delta$ as desired. ∎

Appendix B Remark on Related Works

Grasp Datasets. Numerous grasp datasets have been introduced recently [64], each with varying characteristics such as data representation (RGB-D or 3D point clouds), grasp labels (rectangle-based or 6-DoF), and quantity [70]. Our Grasp-Anything++ dataset differs primarily in its universality, contrasting the limited object selection in existing benchmarks. It covers a wide range of everyday objects and includes natural scene descriptions, facilitating research in language-driven grasp detection. Furthermore, the Grasp-Anything++ dataset uniquely presents natural object arrangements, in contrast to the more strictly controlled configurations in previous datasets [69]. Grasp-Anything++ outperforms other benchmarks in both the number of objects and the number of samples.

Contrastive Loss for Diffusion Models. Recent advancements in contrastive learning have become a prominent attraction in diffusion model research, as evidenced in [31]. While most studies in the diffusion literature regard contrastive learning primarily as a method of data augmentation [35] for improving the performance of models on fine-grained prediction [15], several notable works, including [93, 31, 45, 102]. For instance, Yang et al. [102] leverage intermediate layer features for calculating contrastive loss in negative sample pairs. Contrary to these perspectives, our paper considers contrastive learning as an integral aspect of the training objective, explicitly contributing to the denoising process of diffusion models.

Grasp Detection. Deep learning has significantly advanced grasp detection, with initial efforts by Lenz et al.[48] employing deep learning for grasp pose detection. Following this, deep learning-based approaches[99, 52, 40, 91, 2, 43, 10] have become the predominant methodology in the field. Despite extensive research, the real-world application of deep learning for robotic grasping remains a challenge, primarily due to the limited size and diversity of grasp datasets [28, 69].

Figure 10: Rectangle representation of grasp poses. The groundtruth rectangle is defined with $5$ parameters $\{x,y,w,h,\theta\}$ which $(x,y)$ is the center point of the rectangle, $(w,h)$ is the width and height of the rectangle, and $\theta$ is the rotational angle of the rectangle with respect to the image plane.

Grasp Ground Truth Definition. 6-DoF and rectangle representations are the most prevalent in grasp detection literature. While 6-DoF poses offer greater flexibility and adaptability [27] for complex tasks, rectangle grasp poses are advantageous for their simplicity [39], efficiency in specific scenarios, and lower hardware and computational requirements [18]. Considering that objects in synthesized images from foundation models often lack 3D information [41], the rectangle representation of grasp poses appears more suitable for our task. The ground truth rectangle is defined in Fig. 10.

Appendix C Grasp-Anything++ Analysis

Additional Visualization. Fig. 11 provides further examples from the Grasp-Anything++ dataset, illustrating its diverse and extensive representation of everyday objects. The additional samples of Grasp-Anything++ showcase a diverse collection of objects typically found in everyday environments, such as home offices, kitchens, and living spaces. The collection includes a variety of shapes, sizes, and types of objects such as writing instruments, electronic devices, and household items, each situated within its own designated space. Furthermore, the annotated grasp poses, produced by the Grasp-Anything++’s pipeline, demonstrate high fidelity, thereby offering a foundation for both qualitative and quantitative grasp detection research.

Figure 11: More samples of our Grasp-Anything++ dataset.

Appendix D LGD Implementation Details

In this section, we first discuss the observation about the attention mask during the diffusion process which led to our motivation for using the contrastive diffusion. We then provide the implementation details of our LGD network.

D.1 Observation

In Fig. 12, we present a visualization of the attention mask generated by the vision transformer backbone and the grasp pose throughout the diffusion process. It is evident that during the initial time steps, there is a substantial overlap between the guiding region and the grasp pose, which diminishes as time progresses. This finding suggests that, by the end of the forward process, the guiding region and the noisy grasp pose can be regarded as a contrasting pair. We mathematically express this contrastive relationship between the guiding region $\tilde{\mathbf{x}}_{0}$ and $\mathbf{x}_{t}$. This contrastive relationship forms a central role in our network design, as depicted in the overview of our method in the main paper.

Figure 12: Observation. Comparison between the grasp poses and attention map output by the network backbone.

D.2 LGD Implementation Details

For an image I of resolution $W\times H$, we employ a ResNet-50 vision encoder backbone [32] to derive feature representations $\textit{{I}}^{\prime}\in\mathbb{R}^{w\times h}$ with latent dimensions $w$ and $h$. Similarly, we obtain text embedding $e^{\prime}\in\mathbb{R}^{|D|}$ using a text encoder, such as CLIP [71] or BERT [19]. The dimensionality of these latent features depends on the text encoder’s architecture.

Component	Description	Input size	Output size
(i)	A vision encoder (ResNet-50 [32])	$[H,W,3]$	$[h,w,3]$
(ii-a)	A text encoder (CLIP [71] or BERT [19])	Any	$[|D|]$
(ii-b)	MLP layers	$[|D|]$	$[N+1,d_{\text{text}}]$
(iii)	ALBEF [50]	$[h,w,3],[N+1,d_{\text{text}}]$	$[W,H],[\ell]$
(iv)	MLP layers of Equation 12	$[1]$	$[d_{\text{ts}}]$
(v)	MLP layers of Equation 13	$[\ell]$	$[d_{\text{vl}}]$
(vi)	MLP layers of Equation 14	$[M]$	$[d_{\text{df}}]$
(vii)	MLP layers to output denoising state	$[d_{\text{ts}}+d_{\text{vl}}+d_{\text{df}}]$	$[M]$

Table 7: Architecture specifications of our method.

Hyperparameter	Value
$W$	224
$H$	224
$N$	196
$M$ (number of grasp parameters)	5
$|D_{\text{CLIP}}|$ of (ii-a)	512
$|D_{\text{BERT}}|$ of (ii-a)	768
$d_{\text{text}}$ of (ii-b)	128
$\ell$ of (iii)	1024
$d_{\text{ts}}$ of (iv)	32
$d_{\text{vl}}$ of (v)	256
$d_{\text{df}}$ of (vi)	256
Num. attention layers (ALBEF)	6

Table 8: Hyperparameter details.

Utilizing the ALBEF architecture [50], we integrate vision and language embeddings. Specifically, we encode each intermediate feature $\textit{{I}}^{\prime}$ into a set $\textit{{v}}=\{\textit{{v}}_{\text{cls}},\textit{{v}}_{1},\textit{{v}}_{2},\dots,\textit{{v}}_{N}\}$, where $N$ denotes the number of segmented patches, similar to the approach in [23]. We feed text embeddings $e^{\prime}$ through MLP layers, producing a sequence of embeddings $\textit{{u}}=\{\textit{{u}}_{\text{cls}},\textit{{u}}_{1},\ldots,\textit{{u}}_{N}\}$. Cross-attention mechanisms in the multimodal encoder integrate image features v with text features u. The multimodal attention layer outputs an attention map $\tilde{\mathbf{x}}_{0}\in\mathbb{R}^{W\times H}$ and a final text-image representation $z_{\text{vl}}^{\ast}\in\mathbb{R}^{\ell}$.

Using a sequence of MLP layers, we integrate features from timestep $t+1$, the current state $\mathbf{x}_{t+1}$, and the final text-image representation $z\in\mathbb{R}^{\ell}$ as follows

$$z_{\text{ts}}=\text{MLP}(t+1)\in\mathbb{R}^{d_{\text{ts}}}\text{~{}\@.}$$ (12)

$$z_{\text{vl}}=\text{MLP}(z_{\text{vl}}^{\ast})\in\mathbb{R}^{d_{\text{vl}}}\text{~{}\@.}$$ (13)

$$z_{\text{df}}=\text{MLP}(\mathbf{x}_{t+1})\in\mathbb{R}^{d_{\text{df}}}\text{~{}\@.}$$ (14)

Finally, we concatenate $z_{\text{ts}},z_{\text{vl}},z_{\text{df}}$ to form $z$, which is then processed through an additional MLP to yield the decoded state for the denoising process.

$$\mathbf{x}_{t}=\text{MLP}(z)\in\mathbb{R}^{d_{\text{df}}}\text{~{}\@.}$$ (15)

Architecture Summarization. As outlined in the main paper, our network architecture contains: (i) a vision encoder, (ii) a text encoder followed by MLP layers, (iii) ALBEF module, (iv) MLP layers to encode timestep information, (v) MLP layers to encode text-image features $z_{\text{vl}}^{\ast}$, (vi) MLP layers to encode noisy state $\mathbf{x}_{t}$, and (vii) MLP layers to output denoising state. We summarize the architecture and hyperparameters of LGD in Table 7 and Table 8.

Appendix E Experimental Setups

We present implementation details of other baselines in the language-driven grasp detection task.

E.1 Baseline Setups

Linguistic versions of GR-ConvNet [43], Det-Seg-Refine [2], GG-CNN [59]. We make slight modifications to these baselines by adding a component to fuse image and text features from the input. Specifically, we utilize the CLIP text encoder [71] to extract text embeddings $e^{\prime}$. To ensure a fair comparison between methods, we also utilize ALBEF architecture [50] to do the fusion between the text embedding and the visual features. The remaining training loss and parameter are inherited from the original work.

CLIPORT [78]. The original CLIPORT architecture learns a policy $\pi$, which does not directly solve our task. We modify the CLIPORT architecture’s final layers with appropriately sized MLPs to output grasp poses defined by five parameters $(x,y,w,h,\theta)$. This adaptation ensures consistency with our grasp detection baselines, diverging from CLIPORT’s original policy $\pi$ learning framework.

CLIP-Fusion [96]. In our re-implementation of the architecture from [96], we follow the cross-attention module in CLIP-Fusion with constructed MLP layers. The final MLP layers in the architecture is modified to output five parameters, corresponding to predicted grasp poses.

Figure 13: Overview of the robotic experiment setup.

Figure 14: Additional language-driven grasp detection visualizations. We provide fine-grained grasp detection cases, our method successfully differentiates between objects of similar structure but varying color, such as a green bottle and a blue bottle.

E.2 Robotic Setup

In Figure 13, we present the robotic evaluation conducted on a KUKA robot. Our grasp detection leverages our proposed LGD and other methods listed in the real robot experiments (Table 4 of the main paper), and the results are translated into a 6DOF grasp pose through depth images captured by an Intel RealSense D435i depth camera, as in [43]. The trajectory planner [86, 5] is employed for the execution of the grasp. We use two computers for the experiment. The first computer (PC1) runs the real-time control software Beckhoff TwinCAT, the Intel RealSense D435i camera, and the Robotiq 2F-85 gripper, while the second computer (PC2) runs ROS on Ubuntu Noetic 20.04. PC1 communicates with the robot via a network interface card (NIC) using the EtherCAT protocol. The inference process is performed on PC2 with an NVIDIA 3080 GPU. Our assessment encompasses both single-object and cluttered scenarios, involving a diverse set of $20$ real-world daily objects (Fig. 17). To ensure robustness and reliability, we repeat each experiment for all methods a total of $30$ times.

Appendix F Extra Experiments

Number of Parameter Comparison. Table 9 shows the number of parameters in all methods. This table illustrates that the results from the language-driven grasp detection studies in the main paper reveal a consistent trade-off between performance and the number of parameters across all baselines. Notably, LGD emerges as the balanced baseline, offering a good balance of performance efficiency and computational resource utilization.

Baseline	$\#$Parameters	Success rate
GR-ConvNet [43] + CLIP [71]	2.07M	0.24
Det-Seg-Refine [2] + CLIP [71]	1.82M	0.20
GG-CNN [59] + CLIP [71]	1.24M	0.10
CLIPORT [78]	10.65M	0.29
CLIP-Fusion [96]	13.51M	0.33
LGD (ours)	5.18M	0.45

Table 9: Number of parameter comparison.

Figure 15: Prediction failure cases. In cases where objects have similar structures, such as jars and bottles, our LGD occasionally fails to detect correct grasp poses.

Failure Cases. Though achieving satisfactory results, our method still predicts incorrect grasp poses. A large number of objects and grasping prompts in our dataset suggest a significant challenge for the tasks. Some failure cases are depicted in Fig. 15. From this figure, we can see that the correlation between the text and the attention map of the visual features is not well-aligned, which leads to incorrect prediction of the grasp poses.

(a)

(b)

Figure 16: Detection results in robotic experiments. Images are captured from a RealSense camera with experiments in Fig 18.

Figure 17: Set of 20 objects used in the robotic experiment.

Additional Detection Visualization. Fig. 14 illustrates additional language-driven grasp pose detection using the LGD method. The result demonstrates our method’s capability in reasonably aligning grasp poses with linguistic instructions in fine-grained scenarios, as seen with a green and blue bottle. Remarkably, these qualitative examples demonstrate the effectiveness of our proposed LGD method in fine-grained cases, in line with the contrastive loss objectives outlined in our main paper.

Robotic Demonstration. In Fig. 18, we show a sequence of actions when the KUKA robot grasps different objects in cluttered scenes. Fig. 16 further shows the detection result of our LGD method on an image captured by a RealSense camera mounted on the robot. The robotic experiments demonstrate that although our LGD method is trained on a synthesis Grasp-Anything++ dataset, it still be able to generalize to detect grasp pose in real-world images. More illustrations can be found in our Demonstration Video.

Figure 18: Snapshots of two example robotic experiments.