Synthetic Data Generation with Generative AI Reference Workflow#
Generative AI is helping developers who rely on 3D scenes to simplify and accelerate their workflows.
In this workflow guide, we demonstrate how developers can use NVIDIA APIS to quickly create and augment complex digital twins for building generative physical AI.
For this workflow, we’ll specifically explore NVIDIA USD APIs that enable developers to take advantage of USD for 3D application and workflow development. Afterward, we’ll use the Cosmos WFMs to augment a dataset. These include:
USD Code API : Utilizes a state-of-the-art large language model that answers OpenUSD knowledge queries and generates USD Python code.
USD Search API : An AI-powered search for OpenUSD data, 3D models, images, and assets using text or image-based inputs.
Cosmos World Foundation Models : With Cosmos Transfer, a perception-based Physical AI, augment datasets using AOVs for precise realistic variations.
Both microservices are currently available as a preview on the NVIDIA API Catalog , where developers can make API calls for evaluation.
You will be able to make your own warehouse dataset and extend the warehouses as you see fit. Here are a few examples from our team to help inspire you to make your own variations:
Follow this step-by-step guide to start your development journey with these NVIDIA APIs for digital twin and physical AI development!
Workflow Diagram#
Nvidia Omniverse and NIMs are used to generate scenes and assets for domain randomization. Physically based data is rendered into ground truth AOVs, such as RGB, Segmentation, Normals.
These AOVs are used in the Data Augmentation step to condition Cosmos WFMs. From these initial AOVs incredible variety can be generated following randomization of the input prompts, and retain scene and segmentation accuracy.
Using the augmented datasets, training then proceeds using Pytorch and NeMo.
