The YellowDog workload management platform schedules and provisions compute, both on premise and in the cloud. Using YellowDog, customers can build HPC clusters in the cloud, either in specific regions or scaling across the globe.
Our platform is cloud native and schedules compute instances based on application characteristics, rather than the constraints of a fixed on-premises HPC cluster. Customers manage workloads using the YellowDog scheduler or third-party schedulers, such as Slurm, through the web portal. This gives a single view on cost for users, teams and department consumption across applications. In addition to provisioning resources across cloud regions YellowDog is also able to provision based on instance type and machine size.
Security is of primary concern for enterprise cloud customers. This is especially true when running workloads that contain or generate intellectual property fundamental to the success of their business.
Data is encrypted when at rest and in transit. But what happens when your data is being processed? Step forward Confidential Computing.
Confidential Computing uses hardware-based technology to isolate data, specific functions, or an entire application from the operating system, hypervisor, and other privileged processes. Data is stored in the trusted execution environment (TEE), outside of which it is impossible to view the data or operations performed on it, even with a debugger. The TEE ensures that only authorised software can access the data. If the software is altered or tampered with, the TEE denies the operation.
Google Cloud recently announced the release of Confidential Computing VMs based on 3rd Gen AMD EPYC(™) processors (codenamed “Milan”) on C2D instance types. Ahead of release, YellowDog was given access to the C2D VMs to evaluate the technology at scale using the award winning YellowDog Platform.
A Confidential Virtual Machine (Confidential VM) is a type of Compute Engine VM that provides hardware-based encryption in-use for workloads.
The required configuration for creating a Confidential VM in Google Cloud is straightforward. Users can select the supported instance type simply by choosing the Confidential VM checkbox. No code changes are required to workloads to use these instance types.
YellowDog launched 1,100 Confidential VMs (C2D-standard-112 instances) across 4 regions (Asia-East, Asia-Southeast, Europe, and Central US) on Google Cloud.
This created one of the World’s largest secure C2D cloud clusters to date with 123,200 vCPUs fully provisioned in 19 minutes. As each instance was brought online they were issued tasks by the YellowDog Scheduler and commenced work.
Spot Virtual Machines were used, these are provided at a lower cost than regular instances, but can be withdrawn (pre-emption) at short notice.
YellowDog is self-healing and handles pre-emptions automatically by discovering and provisioning additional virtual machines to make up for the shortfall and re-distributing incomplete tasks to ensure that deadlines are met, whilst also giving customers the benefit of the lowest compute costs. The two graphs below show the VM locations, and the time taken to provision, schedule and start running a workload.
YellowDog also used Google Cloud Storage for both seeding and collecting workload data across these regions.
The key to creating clusters at scale is to ensure that VMs start working on tasks as soon as they have provisioned. Throughout the workload execution, YellowDog ensures the cluster is used to its maximum capacity and any instances lost due to pre-emptions are automatically replaced.
The workload used in this cluster is Molecular Docking, a key stage in the Drug Discovery pipeline in Life Sciences which lends itself to being run at scale. Using Confidential VMs makes perfect sense, given the large investment Life Sciences companies make in new drug lines. This compute option massively increases the security options available and is of huge benefit.
Harvard University’s Real Database contains over 1.4 billion commercially available on-demand molecules. This database can be used by scientists to find potential hit compounds for the development of new drugs and therapeutics. YellowDog used this database, AutoDock Vina, an open-source application for molecular docking and Open Babel (also open-source) to convert and store data from the docking simulations.
Prior to job execution files representing “virtual” molecules are uploaded to Google Cloud Storage (GCS) whilst after completing each analysis the docking scoring results, and job outputs are returned to GCS using YellowDog’s Object Store service. Note, GCS is mounted in the working directory of the instance to reduce data transfer activity.
During execution of the workload, instances were provisioned and issued tasks by the YellowDog Scheduler which utilised each vCPU at 100% until completion.
Separately, YellowDog also executed a smaller workload to compare job duration between confidential and non-confidential VMs using a single region and instance type. There was no discernible difference in performance as identified in job durations. This is important as in this use case there is minimal additional cost in deploying this significantly increased level of security.
In a world where customers pay per second for compute, the time to provision, schedule, run a workload and then de-provision becomes an important cost factor. The YellowDog platform operates seamlessly across multiple geographies and regions, allowing customers to take full advantage of discounted spot pricing anywhere in the world. On Google Cloud, spot instances can be up to 91% cheaper than on-demand instances so the saving can be very significant.
The combination of YellowDog, Google Cloud and Confidential Compute technology from AMD represents a great partnership for secure cloud computing. Organisations of all sizes can scale workloads securely, cost effectively and on-demand using YellowDog on Google Cloud Confidential VMs enabled with AMD security features.
YellowDog is a SaaS cloud management platform and intelligent cloud scheduler which transforms the way workloads are run, whether directly in the cloud or bursting from on-premise. Providing the fastest, easiest cloud access in the world, provisioning in minutes, scaling to millions of cores, YellowDog gives unprecedented choice for HPC in the cloud. See us in action, book a demo now.
To book a demo of the YellowDog Platform, or to ask our team for further information, you're just a click away!
To learn more about AMD based VMs on Google Cloud, visit https://buildboldcloud.com/en/
Or, find more information about AMD’s Infinity Guard secure processing technology at https://www.amd.com/en/technologies/infinity-guard and download a White Paper here.
Google Cloud accelerates every organization’s ability to digitally transform its business. We deliver enterprise-grade solutions that leverage Google’s cutting-edge technology – all on the cleanest cloud in the industry. Customers in more than 200 countries and territories turn to Google Cloud as their trusted partner to enable growth and solve their most critical business problems.
Please upgrade your browser
You are seeing this because you are using a browser that is not supported. The YellowDog website is built using modern technology and standards. We recommend upgrading your browser with one of the following to properly view our website:
Windows MacPlease note that this is not an exhaustive list of browsers. We also do not intend to recommend a particular manufacturer's browser over another's; only to suggest upgrading to a browser version that is compliant with current standards to give you the best and most secure browsing experience.