Elevated error rates on Customer API

Timeline

1. ResolvedJun 5, 2026 at 3:25PM

   Impact window: 2026-06-05, 15:00–15:04 UTC (~3 minutes) On June 5, 2026, the Customer API experienced a brief disruption of approximately three minutes. During this window, a portion of API requests may have failed or timed out. The service recovered automatically and has been fully operational since 15:04 UTC. What happened A backend service process encountered an unexpected fault in an underlying runtime component and automatically restarted. While the replacement instance was starting up and passing health checks, incoming requests during that short window were not served successfully. Resolution The service self-healed once the new instance became healthy. We confirmed full recovery and normal request success rates immediately afterward. Data impact None. No data was lost or affected. Requests that failed during the window were not partially processed and can be safely retried. Next steps To reduce the likelihood and impact of similar events, we're adding redundancy to this service so a single process restart cannot affect availability, and updating the underlying runtime component involved.

2. MonitoringJun 5, 2026 at 3:17PM

   Replacement instance is healthy and serving traffic. Request success rates have returned to normal. Monitoring to confirm full recovery before resolving.

3. IdentifiedJun 5, 2026 at 3:12PM

   Identified the cause: a backend service process encountered an unexpected fault in an underlying runtime component and was automatically replaced. While the replacement instance was starting up and passing health checks, a portion of incoming requests were not served successfully. Impact appears bounded to ~15:00–15:04 UTC.

4. InvestigatingJun 5, 2026 at 3:08PM

   We're seeing elevated error rates on the Customer API starting around 15:00 UTC. Investigating now.