Debugging Google Application Default Credentials

Obligatory picture of keys, which work a bit like tokens. Photo by regularguy.eth on Unsplash

Today, the following error surfaced while I was running code accessing Google Cloud Platform (GCP) resources from my local computer:

ACCESS_TOKEN_SCOPE_INSUFFICIENT 403 PERMISSION_DENIED

My identity did have permission to access the resource, but somehow, insufficient scopes were associated with my access token. This, I had to see for myself ;-).

The default credentials file

The code that I was running authenticated itself according to Google’s application default credential flow. Consider reading my post on application default credentials if this does not sound familiar to you:

Authentication on GCP with Docker: Application Default Credentials

28 Apr 2021 · How applications magically authenticate themselves with GCP through their environment, and how to make locally running containers magic too

When running locally, the default credentials flow usually ends up reading credentials from your local application default credentials file. On Mac and Linux, you can find this file at ~/.config/gcloud/application_default_credentials.json. Provided that you have generated default credentials, the file has contents similar to this:

{
  "client_id": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "client_secret": "d-FL95Q19q7MQmFpd7hHD0Ty",
  "quota_project_id": "your-project-id",
  "refresh_token": "your-refresh-token",
  "type": "authorized_user"
}

Refresh token grant

The application default credentials file contains a refresh token. As far as I know, it is impossible to inspect the scopes associated with Google’s refresh tokens directly. However, refresh tokens can be exchanged for access tokens, and those can be inspected.

Exchanging refresh tokens for access tokens is called the refresh token grant. A single call to Google’s token endpoint suffices to obtain a fresh access token and ID token:

curl --silent --request POST 'https://oauth2.googleapis.com/token' --header 'Content-Type: application/json' --data-raw "$(jq '. | .grant_type = "refresh_token" '  ~/.config/gcloud/application_default_credentials.json)"

The result of this call should look something like this:

{
  "access_token": "an_access_token",
  "expires_in": 3599,
  "scope": "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform openid https://www.googleapis.com/auth/accounts.reauth",
  "token_type": "Bearer",
  "id_token": "an_id_token"
}

Sure enough, in my case, the refresh token in the application default credentials file did not include the https://www.googleapis.com/auth/cloud-platform scope.

Now… I have no clue how this happened, as the CLI usually includes the cloud-platform scope by default 🤷‍♂️. Creating new application default credentials resulted in a refresh token that did include the required scope, solving my issue:

gcloud auth application-default login

Although the above sufficed for me today, I have had to investigate Google’s access tokens or ID tokens further. Tokens can originate from default credentials as above or from another source, such as the metadata service or service account credentials.

Inspecting access and ID tokens

You can inspect access and ID tokens by presenting them to Google’s token introspection endpoint.†

† Google’s token endpoint is meant for debugging and is not meant to be used by production resource servers or other internal servers — it is subject to throttling and does not implement the OAuth token introspection spec.

Access tokens

Store your access token in the ACCESS_TOKEN variable and review it as follows:

curl --silent --request GET "https://oauth2.googleapis.com/tokeninfo?access_token=${ACCESS_TOKEN}"

The result looks similar to this:

{
  "azp": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "aud": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "sub": "your-sub-id",
  "scope": "openid https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/accounts.reauth",
  "exp": "1664891540",
  "expires_in": "3137",
  "email": "your-email",
  "email_verified": "true",
  "access_type": "offline"
}

ID tokens

Similarly, check ID tokens by running:

curl --silent --request GET "https://oauth2.googleapis.com/tokeninfo?id_token=${ID_TOKEN}"

The result looks something like this:

{
  "iss": "https://accounts.google.com",
  "azp": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "aud": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "sub": "your-sub-id",
  "hd": "your-domain",
  "email": "your-email",
  "email_verified": "true",
  "at_hash": "some-hash",
  "iat": "1664887940",
  "exp": "1664891540",
  "alg": "RS256",
  "kid": "...",
  "typ": "JWT"
}

Unlike Google’s refresh and access tokens, Google ID tokens are, nowadays, plain JWT tokens. You can, therefore, also inspect them more directly without making a call to the introspection endpoint. The most accessible approach is to paste your token and review its contents at https://jwt.io/.

You can quickly inspect the header, payload, and signatures of JWT tokens on https://jwt.io/.

Alternatively, decode the payload locally:

echo "$ID_TOKEN" | cut -d. -f2 | base64 -d | jq

The output looks similar to:

{
  "iss": "https://accounts.google.com",
  "azp": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "aud": "764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com",
  "sub": "your-sub-id",
  "hd": "your-domain",
  "email": "your-email",
  "email_verified": true,
  "at_hash": "some-hash",
  "iat": 1664887940,
  "exp": 1664891540
}