Debugging OAuth: The mysterious case of missing scopes on refresh tokens
I hate OAuth. There I said it. It has some advantages, but it can make non-interactive programmatic access to APIs that use it more complex where something like REST or API keys being sent as a HTTP header just work. I’m possibly jaded by my experiences with OAuth over the years but maybe after reading this you’ll understand why.
For certain APIs OAuth is the only option, specifically with various Google APIs many require OAuth, so you have no choice.
The Google Business Profile APIs only support OAuth you cannot use service accounts with this API product. The Google Business Profile APIs are also by request only, you have to go through an approval process to have access to use them.
The first sign of OAuth issues
Things started going wrong on 11th December 2024. Up until this point, our OAuth client had been working for many months with no issues. Now all of sudden we appear to no longer be able call Google Business Profile API endpoints reliably after about an hour with our OAuth client. We hadn’t made any changes to our OAuth setup or configuration, and nothing changed in our Google Cloud Console in terms of the Client ID or Client Secret.
Looking at the debug logs/errors reported from Guzzle, we find “403 Request had insufficient authentication scopes”. That is an interesting error I hadn’t seen before. Typically, if an invalid token was provided it would be a 401 Unauthorized error, but we have specifically got a 403 error back from Google. With debug headers enabled, the error is more specifically “ACCESS_TOKEN_SCOPE_INSUFFICIENT”. An even more interesting error to get, because the scope requested is the required scope needed for these APIs:
https://www.googleapis.com/auth/business.manageIt is poorly documented and doesn’t appear on the OAuth 2.0 scopes list, but the business.manage scope is the correct scope to use for the Google Business Profile APIs.
That one-hour scenario immediately suggested something wrong with the token, as the token validity period is 3600 seconds aka one hour before a token is expired and either a new token is needed, or a refresh token needs to be used.
We were setting all the right parameters in our initial request i.e. offline access type to get a refresh token, so it wasn’t as if our token is just expiring with no refresh. It would appear as soon as the refresh token was used, this then triggered the 403 error, but why? It took a bit of time but after inspecting the token details of an initially valid token and then another that was refreshed, the problem became clear.
Working access token from the initial OAuth request (private info masked):
{
"issued_to": "clientId",
"audience": "clientId",
"user_id": "userId",
"scope": "https://www.googleapis.com/auth/business.manage https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid",
"expires_in": 3570,
"email": "email@example.com",
"verified_email": true,
"access_type": "offline"
}The access token obtained by the refresh token (private info masked):
{
"issued_to": "clientId",
"audience": "clientId",
"user_id": "userId",
"scope": "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid",
"expires_in": 3591,
"email": "email@example.com",
"verified_email": true,
"access_type": "offline"
}For some reason on the new token generated by use of the refresh token, the business.manage scope has been removed entirely from the “scope” value. This certainly explains the 403 error and more specifically “ACCESS_TOKEN_SCOPE_INSUFFICIENT” but a more puzzling question now is why the scope has been removed to begin with? This certainly doesn’t seem like normal OAuth practice/behaviour. We are just using the provided refresh token given to us, we aren’t influencing the scopes at this point, we already made on our initial request with that scope in place, why has it been removed?
It took a bit of time to find evidence of a reproducible problem, but this at least illustrates why we are getting the error, but it doesn’t really provide a solution. It was also determined that it doesn’t matter if a refresh of the token happens after the original token expires or it is forced before the expiry period, as soon as the refresh token is used, the token loses the valid scope needed and the 403 error will occur on any request going forward until an entirely fresh token is obtained. This however requires going through the OAuth process, with the consent screen etc so it’s not a long-term solution, given we need this for unattended programmatic access.
Caching of valid responses to work around the problem
By chance, our use of Google Business Profile generally uses long cached responses, so even though we can’t query the APIs after one hour without going through the full OAuth consent process again, we have long cached data durations on various calls which means they can survive for 24 hours or even longer, reducing the impact of the issue we are facing. This means that while our OAuth client is broken, providing the requests were made when the client has a valid token, we would maintain the integrations and services using the API data from cached responses without too much noticeable impact. We just had to ensure that requests for this data happened when the client was working and cached a valid response.
This did mean that we’d have to keep going through the OAuth consent process manually requiring human interaction at minimum once every 24 hours. This is not ideal, but better than nothing and certainly better than every hour.
Refresh tokens are specifically broken, what’s that about?!
Knowing that the context is specifically with refresh tokens, I decided to check the behaviour of manually refreshing a token outside of any client library and simply with a plain HTTP request, doing a very barebones API call to the https://oauth2.googleapis.com/token endpoint with a valid refresh token in hand.
curl --location 'https://oauth2.googleapis.com/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'client_secret={clientSecret}' \
--data-urlencode 'grant_type=refresh_token' \
--data-urlencode 'client_id={clientId}' \
--data-urlencode 'refresh_token={refreshToken}'Upon getting a new token and checking the scope value:
{
"access_token": "{accessToken}",
"expires_in": 3599,
"scope": "https://www.googleapis.com/auth/userinfo.email openid https://www.googleapis.com/auth/business.manage https://www.googleapis.com/auth/userinfo.profile",
"token_type": "Bearer",
"id_token": "{id_token}"
}The business.manage scope is there, and it hasn’t been removed from this refresh token. This now rules out anything on the Google side, because the scope is there, it works and is what we would expect to see.
Finding the broken dependency
After now seeing with my own eyes that a plain HTTP request to the token endpoint does not behave the same way, there had to be a dependency issue somewhere. A quick workaround was to bypass our OAuth client refresh process and send a separate HTTP request specifically for refreshing the token, but ultimately it is just working around the problem, and we shouldn’t need to do that.
In the case of this project, we are using a client library which itself has several dependencies. The chain of dependencies is:
The two Verbb packages are Craft CMS plugins/modules which are related to each other to do the heavy lifting of OAuth client handling. There hadn’t been any major changes in these plugins that could be correlated to the behaviour I was seeing. However, looking at league/oauth2-client the 2.8.0 release immediately got my attention, and it didn’t take long to see the exact issue describing the problem I had been facing for several weeks as an open issue: 2.8.0 breaks exiting scope handling · Issue #1052 · thephpleague/oauth2-client
It turns out, on 11th December the 2.8.0 release was published and our project’s composer.lock had been updated and pulled in this release alongside routine package updates. Because it was a minor point release, it wasn’t considered to be major or breaking, however unfortunately, this version has shipped with breaking refresh tokens pretty hard. It isn’t specific to Google APIs, but Google APIs are one of the main areas that have been caught up in the crossfire.
The issue is specifically handling of scopes on refresh tokens, the code in 2.8.0 basically nukes any scopes other than the default values provided on the class, which in the case of Google is openid, email and profile.
The easiest fix currently is just downgrade back to 2.7.0 and everything is fine again. There is a pull request pending to resolve the issue: Only provide scopes when set in options by barryvdh · Pull Request #1053 · thephpleague/oauth2-client but it has yet to be merged, so at the moment if you have any dependency on the oauth2-client PHP package, using 2.8.0 will potentially cause broken OAuth client behaviour and you might end up chasing yourself for a while!
