City of Pleasanton Systems Status

All Systems Operational

Uptime over the past 90 days. View historical uptime.

www.cityofpleasantonca.gov Operational

90 days ago

100.0 % uptime

Today

Pleasanton Library Systems Operational

90 days ago

99.97 % uptime

Today

Pleasanton Library Web Page Operational

90 days ago

100.0 % uptime

Today

Utility Billing Payment Portal Operational

90 days ago

100.0 % uptime

Today

Pleasanton Recreation (CivicRec) Operational

90 days ago

100.0 % uptime

Today

Public Meetings - Agenda and LiveStream (CivicClerk) Operational

90 days ago

100.0 % uptime

Today

Laserfiche - Public Records Search Operational

90 days ago

100.0 % uptime

Today

Internet Service Operational

90 days ago

99.74 % uptime

Today

GovQA Public Records Request Operational

90 days ago

100.0 % uptime

Today

City Council Video and Audio Streaming Operational

90 days ago

100.0 % uptime

Today

www.inpleasanton.com Operational

90 days ago

100.0 % uptime

Today

www.firehousearts.org Operational

90 days ago

100.0 % uptime

Today

www.pleasantonpermits.com Operational

90 days ago

100.0 % uptime

Today

Land Management / Permitting System Operational

90 days ago

100.0 % uptime

Today

Voice Permit System Operational

90 days ago

100.0 % uptime

Today

www.trivalleyjobs.com Operational

90 days ago

100.0 % uptime

Today

E-Mail Communications Operational

90 days ago

100.0 % uptime

Today

Telephone and VoiceMail Operational

90 days ago

99.14 % uptime

Today

Online Business License Renewal Operational

90 days ago

100.0 % uptime

Today

Police Online Reporting System Operational

90 days ago

100.0 % uptime

Today

GIS City Viewer Operational

90 days ago

99.96 % uptime

Today

Local and Wide Area Network Operational

90 days ago

99.36 % uptime

Today

CityShare - SharePoint System Operational

90 days ago

100.0 % uptime

Today

FileTrail Operational

90 days ago

100.0 % uptime

Today

Tyler Munis - Selfservice Portal Operational

90 days ago

100.0 % uptime

Today

Tyler Munis - Financial/HRMS System Operational

90 days ago

99.93 % uptime

Today

Help Center - Helpdesk System Operational

90 days ago

98.65 % uptime

Today

Microsoft Teams Operational

90 days ago

100.0 % uptime

Today

Cashiering System - BridgePay Operational

90 days ago

100.0 % uptime

Today

File & Data Server Operational

90 days ago

100.0 % uptime

Today

Fileshare (Accellion Kiteworks) Operational

90 days ago

100.0 % uptime

Today

SharePoint Online (Progress Intranet) Operational

90 days ago

100.0 % uptime

Today

Zoom Telephone Services Operational

Zoom Zoom Meetings Operational

Zoom Zoom Video Webinars Operational

Global Protect (remote access) Operational

90 days ago

100.0 % uptime

Today

OpenText - XMEDIUS Fax Services Operational

90 days ago

100.0 % uptime

Today

SeamlessDocs E-Form System Operational

90 days ago

100.0 % uptime

Today

Veoci Operational

90 days ago

100.0 % uptime

Today

Telestaff Operational

90 days ago

100.0 % uptime

Today

Splashtop Remote Desktop Operational

90 days ago

100.0 % uptime

Today

DigEplan Operational

90 days ago

100.0 % uptime

Today

Utility Billing IVR Operational

90 days ago

100.0 % uptime

Today

Adobe Creative Cloud Operational

90 days ago

100.0 % uptime

Today

DocuSign Operational

90 days ago

100.0 % uptime

Today

Dog License System Operational

90 days ago

100.0 % uptime

Today

IT Servers Operational

90 days ago

98.28 % uptime

Today

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

www.cityofpleasantonca.gov - Uptime

Fetching

Library Systems - Uptime

Fetching

Utility Billing Payment Portal - Uptime

Fetching

Pleasanton Recreation (CivcRec)

Fetching

firehousearts.org

Fetching

Pleasanton Online Permit System - Uptime

Fetching

inpleasanton.com

Fetching

Tyler Munis - Selfservice Portal

Fetching

Laserfiche - Public Records Search - Uptime

Fetching

Internet Connectivity

Fetching

Past Incidents

Jul 27, 2024

No incidents reported today.

Jul 26, 2024

No incidents reported.

Jul 25, 2024

No incidents reported.

Jul 24, 2024

No incidents reported.

Jul 23, 2024

Server Outage

Resolved - While there may still be remnants of anomalies or outages to an endpoint computer, we now consider this incident resolved. The IT department will release a full post-mortem report on this incident. This will have an executive summary but also include details regarding the timeline, our response, impact analysis, and lessons learned, among other details.
Jul 23, 11:43 PDT

Update - Monday Morning Blues - screen of death. See what I did there? While, for the most part, we have remediated the most critical systems that urgently need to be restored, we are dealing with the remaining group of knocked-out-of-commission computers.

Please be patient; the team is trying to get through these quickly. Meanwhile, it has been learned that some have been able to recover from the BSOD by rebooting their systems up to 15 times (that is not a typo).

Thanks again for your continued support.
Jul 22, 08:48 PDT

Update - Making great progress—There has been a great team of people helping with this effort, from our external partners here in town—ENT Networks—to deputized IT folks in Public Works and other areas of this great organization assisting IT.

- Servers: We believe we have restored service to the most critical and urgent servers needing recovery

- Workstations: We continue to work on restoring services to computers that have been stubborn about coming back online. The encouraging news is that while it has often been a case-by-case remedy, we have had 100% success. However, it will take continued patience if we have not visited your system yet.

If you have an urgent need that we should recognize in our prioritization - please don't be shy about taking another bite of the apple and letting us know via our service desk.

In-Progress: The team is hitting remote sites - such as LPFD stations, DBAC, Gingerbread, etc.
Jul 19, 13:28 PDT

Update - So... How's your day? Me? Oh, I'm just looking to see who will take my shares of Crowdstrike stock. Here is where we are as we assess this morning's damage, including collateral damage.

We still have about 16 servers that need their services restored. The "fix" does not work for some, and a full restore from our backup system is required.

We have multiple endpoint systems that require service. The Crowdstike "fix" success rate is low and, in some cases, not applicable.

We have external partners in the same position as us—some that we integrate with, some that we employ as Software-as-a-Service, and some that we rely on for the resources they provide.

So, thank you so much for your patience and understanding and for not being offended if it appears you are not the priority. You are! But it takes a bit to get from the pool to your desk.
Jul 19, 09:45 PDT

Update - Continue to Monitor
Jul 19, 09:38 PDT

Monitoring - Many Servers have had their services restored, and many more are pending the team's attention. We continue to work through these.

The more significant challenge is that many will be met with a computer screen with a BSOD (blue screen of death). Crowdstrike has released an update that should bring your PC or laptop back online upon a reboot.

However, this has been a bit of a hit-or-miss for us. Here are the Workaround steps for individual hosts:

1. Reboot the Computer:
- Restart your computer to allow it to download the necessary files.

2. If the Computer Crashes Again:
- Start in Safe Mode or Recovery Mode:
---- Restart your computer and press the appropriate key to enter Safe Mode or the Windows Recovery Environment (usually F8, F11, or a similar key).

- Navigate to the Specific Folder:
---- Open File Explorer.
---- Go to: C:\Windows\System32\drivers\CrowdStrike.

- Delete the Problematic File:
---- Look for a file that starts with “C-00000291” and ends with “.sys”.
---- Delete this file.

3. Restart Normally:
- Restart your computer as usual.

If you continue to have boot issues - please contact the Service Desk via any of these channels:

• Web: https://help.cityofpleasantonca.gov (access requires your city network login account)
• eMail: itrequest@cityofpleasantonca.gov
• Phone: 925.931.5083—if no one answers, leave a message. Doing so will auto-generate a ticket with your voicemail as an attachment.
• SMS Text: 925.931.8350
Jul 19, 02:40 PDT

Identified - We want to inform you about a global issue currently affecting Microsoft systems running CrowdStrike endpoint protection security—systems we also utilize. This issue is causing widespread disruptions, resulting in the offline status of many of our Windows desktops and servers.

Current Impact:
Authentication and Network Services: Some of our servers, which authenticate logins, provide IP addresses to endpoints, and use DNS to resolve hostnames and URLs, are affected. This disruption may present conditions that appear to be network-related.

Critical Applications: Servers hosting critical software applications and services are also offline, impacting their availability and performance.

Our Response:
Disaster Recovery Effort: We are treating this situation with the urgency of a Disaster Recovery effort to ensure business continuity. We focus on restoring our most critical systems, starting with servers and extending to desktops.

Monitoring and Mitigation: We are closely monitoring the situation and actively seeking remedies to address the root cause of the disruption.

Next Steps:
Workarounds and Potential Remedies: We are discovering workarounds and potential solutions, though neither Microsoft nor CrowdStrike has officially released any.

Ongoing Updates: We will continue to provide updates via this System Status Page. We encourage you to subscribe for the latest updates, which may not be forwarded to the "All Users" email distribution list.

Support:
If you have any immediate concerns or require assistance, please contact our support team via the Service Desk

We apologize for any inconvenience this may cause and appreciate your understanding as we work through this global incident.

Thank you for your patience and cooperation.
Jul 19, 01:05 PDT

Update - This incident involves network outages. Staff is heading onsite.
Jul 18, 23:25 PDT

Investigating - Our monitoring system is alerting to server outages - We are investigating now
Jul 18, 23:01 PDT

Jul 22, 2024

Jul 21, 2024

No incidents reported.

Jul 20, 2024

No incidents reported.

Jul 19, 2024

Jul 18, 2024

Jul 17, 2024

No incidents reported.

Jul 16, 2024

No incidents reported.

Jul 15, 2024

No incidents reported.

Jul 14, 2024

No incidents reported.

Jul 13, 2024

No incidents reported.

Related