CIVP Construct API and Mobile Service Interruption

Incident Report for City of Pleasanton Systems

Resolved

SERVICE RESTORED as of 1:20 PM PT - pending closed loop verification with customers.

This is was mitigated ahead of Azure's reported full restoration by 4:20PM PT
Posted Oct 30, 2025 - 07:06 PDT

Monitoring

SERVICE RESTORED as of 1:20 PM PT - pending closed loop verification with customers.

This is was mitigated ahead of Azure's reported full restoration by 4:20PM PT - please report any issues you experience to Support, but no further impact is expected.

Impact was limited to API (Construct) and Mobile during this incident.

Monitoring will continue until Close of Business today.

Time posted

Oct 29, 13:36 PDT
Posted Oct 29, 2025 - 13:57 PDT

Update

An update has been posted

Update from MS now:

"We are continuing to monitor progress closely and will provide an ETA for full mitigation within the next 20 minutes as we assess recovery across the AFD service."

NOTE that the Accela failover plan for this service is being prepped and validated in parallel, and service is expected to be restored regardless of Microsoft updates.

Updates as they become available.

Time posted

Oct 29, 12:14 PDT
Posted Oct 29, 2025 - 12:21 PDT

Update

We are continuing to work on a fix for this issue.
Posted Oct 29, 2025 - 11:44 PDT

Identified

Accela is actively moving to fail over services in parallel with Azure service restoration.

It's also been determined that CRM services are impacted, in addition to CIVP Mobile and Construct.

"Starting at approximately 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.

We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.

We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.

This message was last updated at 17:18 UTC on 29 October 2025"
Posted Oct 29, 2025 - 11:41 PDT