Sansform - Elevated Management API error rates across all regions – Incident details

Experiencing partially degraded performance

Elevated Management API error rates across all regions

Resolved
Partial outage
Started 4 months agoLasted about 2 hours

Affected

Supabase

Operational from 2:48 PM to 3:04 PM, Degraded performance from 3:04 PM to 4:16 PM, Operational from 4:16 PM to 4:46 PM

Supabase → Dashboard

Operational from 2:48 PM to 3:04 PM, Degraded performance from 3:04 PM to 4:16 PM, Operational from 4:16 PM to 4:46 PM

Updates
  • Resolved
    Resolved

    Error rates have returned to baseline and all regions are operating as expected.

  • Monitoring
    Monitoring

    We've seen error rates return to normal levels, and all regions are now holding steady. We will continue to monitor to make sure things remain stable.

  • Update
    Update

    We are continuing to see some periodic spikes in error rates; however, total rate of errors are decreasing across most regions. We're continuing efforts to get all systems back to green.

    When error rates spike, management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions remain impacted.

    Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

  • Update
    Update

    The team is continuing to deploy mitigations, and we are seeing error rates improve, but are not back to baseline yet.

    All management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions remain impacted.

    Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

  • Identified
    Identified

    We have identified the source of the increased error rates, and are working on a fix now.

    All management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions remain impacted.

    Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

  • Update
    Update

    We've expanded this to all management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions would also be impacted. Error rates are currently improving, but the team is investigating the root cause to ensure continued stability.

    Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

  • Investigating
    Investigating

    We are currently investigating the cause, but users may see issues creating projects across all regions. Existing projects are not affected and remain up and stable.