At approximately 20:00 UTC a routine deploy was initiated to improve the reliability and stability of the /valhalla/v1 endpoints of our Directions API service in the ap-southeast-2, us-west-2, and eu-west-1 AWS regions.
At 20:04 UTC a customer reported an increased level HTTP 401 errors from the Directions API, an error code that is normally returned when a request is authenticated with an invalid token.
At 20:15 UTC our team acknowledged the incident report and opened an investigation. Immediate action was taken to rollback the deploy.
By 20:30 UTC the deploy had been rolled back and service was fully restored.
After stabilizing the service, a root cause analysis uncovered a server misconfiguration related to access token decryption. The misconfiguration was compounded by a bug that obscured a validation error and prevented normal health checks and acceptance tests from identifying the misconfiguration.
We resolved the issue by:
As the /valhalla/v1 endpoints are a newer addition to Mapbox API, we will be examining our monitoring systems to proactively detect this type of misconfiguration in the future. These endpoints are now part of the scope of services that will be included on status.mapbox.com updates.