GCP Billing Monitor

By Drew Rothstein, President

Google Cloud Billing

If you operate on Google Cloud Platform (or Amazon Web Services, or Microsoft Azure) as one of your Public Cloud providers you will likely have hundreds of projects as you grow your business. Separating services into dedicated projects is a best practice to ensure isolation of permissions and responsibilities.

Creation of projects can be fairly quickly automated and can help with standardization of organization management / structure. For example, having a provisioning script or UI that creates a project, sets the default permissions, and removes any cloud provider defaults that are unsafe or not best practice for your organization (e.g. Asgard by Netflix (dead), Control Tower by AWS, et cetera).

But sometimes, projects are created outside of standard process or tools for a variety of reasons and that is totally okay. When this does occur though, having automated tools to let you know something isn’t as you expect is critical before things get all cattywampus.

Billing

One of the standardizations you will likely choose to make is to have a specific billing account within your organization that you would like to use to centralize billing. You may have several of these depending on your structure.

When projects are created outside of standard tools they may end up getting set to all sorts of billing projects - maybe the wrong one, maybe someone’s personal account, or maybe not have billing set up at all!

This can create really strange side effects with most cloud providers where certain services may not work as you expect or maybe even their automated systems that detect potential fraud don’t work correctly and your project gets erroneously flagged as a cryptocurrency mining operation and is thus disabled instantly!

Monitoring

We ran into parts of this scenario and implemented a very small tool that runs in a Google Cloud Function to alert us each day if there are any projects not set to the appropriate billing account that we would expect.

We chose Cloud Functions because this is one API call that runs once a day. There is no infrastructure needed beyond provisioning the function and Cloud Scheduler to call it.

As many businesses, we utilize notifications in Slack as our output method given the low criticality for most scenarios and not needing a higher escalation path.

Implementation

The implementation is fairly straightforward as a small JavaScript function. This is abstracted / generalized a bit for the copy / paste use on the Intertubes.

const express = require('express');
const {CloudBillingClient} = require('@google-cloud/billing');
const {IncomingWebhook} = require('@slack/webhook');

const DEBUG           = false;
const BILLING_ACCOUNT = 'billingAccounts/YOUR_BILLING_ID';
const BILLING_UI      = 'https://console.cloud.google.com/billing/projects?organizationId=YOUR_BILLING_ORG'

const billingClient = new CloudBillingClient();


async function runner() {
  let response_data = [];

  const request = {
    name: BILLING_ACCOUNT,
  };

  const [projects] = await billingClient.listProjectBillingInfo(request);

  for (const project of projects) {
    response_data.push(project);
  }

  const data_to_post = await parse_request(response_data);

  if (data_to_post.length !== 0) {
    post_to_slack(data_to_post);
  }
}

// parse
function parse_request(data) {
  if (DEBUG) {
    console.log(data);
  }

  let projects_to_review = [];

  const number_of_billing_projects = data.length;

  if (number_of_billing_projects !== 0) {
    console.log(`There are ${number_of_billing_projects} projects to review`);

    for (const project of data) {
      projects_to_review.push(project['projectId']);
    }
  } else {
    console.log('No projects; As expected');
  }

  return projects_to_review
}

// post
function post_to_slack(data) {
  const webhook = new IncomingWebhook(process.env.SLACK_TOKEN);
  (async () => {
    await webhook.send({
      text: `*GCP Billing Projects to Review*

:powerup:
\`${'Projects'.padEnd('10', '.')}\` \`${data.toString()}\`

*Link*: ${BILLING_UI}
      `,
    });
  })();
}

const app = express();
app.get('/run', (req, res) => {
  runner();
  res.status(200).send('Completed');
});

module.exports = { main: app };

And an associated package.json:

{
  "name": "billing-account-monitor",
  "main": "index.js",
  "version": "0.0.1",
  "private": true,
  "engines": {
    "node": ">=16.0.0"
  },
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@google-cloud/billing": "^2.3.0",
    "@slack/webhook": "^6.0.0",
    "axios": "^0.21.1",
    "dotenv": "^10.0.0",
    "express": "^4.17.1"
  }
}

The details of how this has the one environment variable injected and is deployed is fairly specific to every environment and is thus not included here. In addition, how to configure Cloud Scheduler to trigger is also not included as this varies by organization.

If you were seeking to deploy this manually, it would look something like:

gcloud functions deploy $(FUNCTION_NAME) \
    --region=$(GCP_REGION) \
    --entry-point=main \
    --trigger-http \
    --quiet \
    --service-account=<YOUR DEDICATED SERVICE ACCOUNT> \
    --project=$(GCP_PROJECT) \
    --memory=128MB \
    --runtime=nodejs16 \
    --timeout=30 \
    --max-instances=1

Output

The output that this provides is clear an actionable by a human. We have found that this can detect far more than just a billing linkage problem but stray or abandoned projects.

GCP Billing Monitor Slack Output

Conclusion

Being able to quickly detect, alert, and then remediate when infrastructure is not configured as you expect is a critical part of running an engineering organization. While manual approaches using standardized interfaces can work, things can frequently slip through your process. The approach outlined above was a very quick solution that has already paid off saving us both time and money.

This is a short example of an afternoon project on the Infrastructure side of our Engineering team here at Unit 410. If you are interested in this type of work, feel free to get in touch with us via dev [at] unit410 [dot] com.

Header Image Sources