Investigating slower metric collection

Timeline

1. ResolvedJun 23, 2026 at 2:14AM

   We're fully caught up. Metric collection is processing in real time again and dashboards are showing current data, so the customer-facing impact from this incident is over. As a final follow-up, we'll reprocess the small number of jobs that timed out earlier during the backlog — no data will be lost in the meantime. Thanks for your patience.

2. MonitoringJun 23, 2026 at 1:58AM

   We're almost caught up. Processing is healthy again and we're currently working through over 1 million metric-collection jobs per minute. Once we're fully caught up, we'll reprocess any jobs that were terminated or timed out during the incident. Things are looking healthy and we expect to be fully recovered shortly.

3. MonitoringJun 23, 2026 at 1:28AM

   We're working through the queue as quickly as we can and scaling up to handle the extra load. No data is lost — each metric-collection job is currently just taking a few minutes longer to process than usual. We'll continue monitoring as the backlog clears.

4. MonitoringJun 19, 2026 at 12:31PM

   New workers have been deployed and we are now processing the backlog at approximately 3x normal throughput. Dashboards should begin showing fresher metrics shortly. We expect the queue to be fully caught up within the hour. Continuing to monitor.

5. IdentifiedJun 19, 2026 at 11:27AM

   We've identified the cause. A background service responsible for processing metric-collection jobs has been failing to keep up, creating a backlog. As a result, dashboards may show delayed or stale metrics. No data has been lost — pending metrics are queued and will be processed once the service is restored. Our team is actively working on a fix and we'll post another update shortly.

6. InvestigatingJun 19, 2026 at 11:12AM

   We're looking into reports of slower-than-usual metric collection. Dashboards may show delayed or stale metrics while we investigate. The underlying services remain operational and no data is being lost — metrics are queued and will catch up. We'll post an update as soon as we know more.