Automated Cleanup of Unused Google Cloud Projects

As an organization increasingly relies on cloud, its number of cloud projects can also increase. Over time, project sprawl creeps in, and an organization can be left with tens, or even hundreds, of unnecessary projects. While these projects can be deleted in bulk, it becomes challenging to determine which projects are no longer needed. As manual efforts to understand each project are undertaken, valuable resources go wasted performing this arduous task. Even worse, resources running in the superfluous projects could be increasing your costs, carbon footprint, and security risk.

Introducing Remora

Remora is a serverless solution that helps limit the number of unused projects in your organization. It works with the Unattended Project Recommender to notify project owners of their unused projects, escalate notifications, and then delete those projects if no action is taken after a predetermined period.

How it works

The way the solution works is straightforward. The unattended project recommender analyzes usage activity on projects in your organization to make recommendations about reclaiming or removing unattended projects. Taking those recommendations a step further, Remora was designed to identify owners of the unattended projects and then send them a customizable email notification or assign them a Jira ticket. You can establish a predefined cadence to send notifications, designating an Essential Contact to be copied after the first email (e.g., the folder owner). After three emails are sent for any given project, a time-to-live (TTL) can be set to determine how long the unused project can stay unused before it is removed. Remora labels each project with an impending deletion date.

1. Cloud Scheduler kicks off a Cloud Workflow that runs on a scheduled cadence
2. A Cloud Workflow will check for active or claimed unattended project recommendations for the organization via the Recommender API
3. A BigQuery table containing each recommendation and the number of times it has been processed is queried and updated
4. A Cloud Workflow will determine what actions need to be taken (e.g., deleting the project as well as which contacts should be notified) 
5. A Cloud Workflow will publish a message to a Pub/Sub topic
6. A Pub/Sub subscription will push the message to a Cloud Function
7. A Cloud Function will retrieve the Jira or Sendgrid credentials from Secret Manager in order to notify the designated contacts
8. Jira or Sendgrid will notify project contacts via a ticket or email, respectively, regarding the unattended projects
9. The project contacts will be able to act on the unattended project recommendations, either by applying or dismissing them
10. Applying or dismissing the recommendation will update its state on the Recommender API

Remora’s core capabilities

Remora was built with several essential capabilities to ensure it could be customized to help meet each organization’s unique requirements:

• Dry-run mode: dry-run mode is enabled by default, which prevents Remora from deleting projects. Dry-run mode must be turned off in order for projects to be deleted by the solution.

• Multiple notifications: owners of unused projects should have multiple opportunities to act on the recommendations. Remora notifies owners every time it runs, and Cloud Scheduler can be used to set up periodic Remora runs (e.g., once a week).

• Summary notifications: an owner on multiple unused projects receives a single email notification with all the projects identified.

• Escalation of notifications: the first notification is always sent directly to the project owner(s). We’ve implemented two mechanisms for escalations of subsequent notifications:

• Essential Contacts: Remora escalates to the specified category of Essential Contacts for the project. If your identities are different from your email addresses, configuring Essential Contacts will inform Remora of the correct escalation email addresses.

• Folder or organization admins: when an Essential Contacts category is not specified, Remora escalates to the admin of the project’s parent folder or organization (whichever is the parent in the resource hierarchy).

• Time-to-live: Organization admins can set the number of days during which an unused project can remain in their organization. Remora will label the projects with their impending deletion date and delete the projects after the designated period of time and three notifications.

• Notification mechanisms: Remora sends email notifications using Sendgrid or creates Jira tickets. 

• Deployment using Google Cloud CLI or Terraform: Remora can be deployed manually using gcloud commands or as a Terraform module.

The holistic solution

The entire solution is made up by combining the components below. 

Unattended Project Recommender

The unattended project recommender analyzes project usage and provides recommendations to remove unused projects. Generally, a project will be recommended for deletion when it has low usage for 30 days and no OAuth tokens used in the last 180 days. Remora will then label the unattended project for deletion.

Google Cloud Workflows and Scheduler 

Workflows is a service that lets you connect different Google Cloud services and APIs to create pipelines and process automation. Workflows are configured with a YAML or JSON file that lists a series of steps in their order of execution. For this solution, Workflows are used to create the initial BigQuery dataset and tables where recommendations will be tracked, retrieve the latest unattended project recommendations from the Recommender API, and call Pub/Sub to initiate the notification process to the owners of the identified unattended projects. The workflows execute on a schedule configured using Cloud Scheduler, Google Cloud’s crontab-as-a-service solution. Cloud Scheduler is where you configure how often you want Remora to process unattended project recommendations.

Cloud Functions

Cloud Functions is Google Cloud’s function-as-a-service offering that lets you execute lightweight functions without the need to manage any servers. Cloud Functions can execute programmatically when triggered by events from Cloud Storage, Pub/Sub, Firebase or HTTP requests. Here, a Cloud Function is triggered via Pub/Sub to alert the project owner via email using Sendgrid or via an issue in Jira. 

Terraform

To simplify and streamline the deployment of Remora, we compiled the individual Google Cloud CLI commands into a Terraform module that creates all the resources needed to get Remora running. As a Terraform module, Remora can be deployed to the provided Google Cloud project and customized with just a few variables. 

The module will handle the creation and configuration of Workflows, Cloud Scheduler, Cloud Functions, and a service account with custom role assignments on the project and organization IAM policy. The code used for the Cloud Functions is included in the module and is uploaded as an archive file to a Cloud Storage bucket.

Check out the documentation in the repository for more detailed usage information and examples. Here’s one simple example of what a module might look like:

The example above will retrieve recommendations from the unattended project recommender every Sunday night, then use Sendgrid to send an email to the unattended project owner. 

Putting it into practice

As soon as Remora is deployed in your organization, Workflows will query the Recommender API based on a specified interval. With Sendgrid configured as the notifier, the project owner will receive a message like this:

After being notified, the unattended project owner will have two options: delete the project right away, or dismiss the recommendation so that it won’t be picked up by the Recommender API again. 

If no action is taken after the first notification, the next notification will include your specified category of Essential Contacts. If the Essential Contacts category is not set, the next owner in the resource hierarchy (i.e., the folder or organization) is included instead. The second message will look like this:

Finally, if no action is taken after three notifications and the TTL has expired, the project is automatically shut down and marked for deletion when Remora runs. Just like shutting down a project manually, there is a 30-day period where the project can be restored in case it was deleted in error. 

By leveraging the intelligence of Active Assist’s recommendation APIs, Workflows, and Cloud Functions, Remora will prune unattended projects to potentially resolve security risks, reduce your carbon footprint, and lower the associated costs of your cloud infrastructure without the overhead incurred from frequent manual auditing. Additionally, since Remora is an open-source project, you can examine and customize the logic used in the Workflows and Cloud Functions to tailor the solution to your organization’s needs. You can get started by checking out the project repository on GitHub and deploying Remora using the provided Terraform module. If you would like to learn more about Active Assist, please take a look at this YouTube playlist covering Active Assist and its intelligent features.