While we continue to receive isolated, random tickets related to this event. It is mainly from those who are returning to the office. IT considers this matter resolved.
Thank you so much to our great user community for your support and patience while we navigated the issues we faced.
Posted Aug 20, 2024 - 13:02 PDT
Update
Incident Update - Domain Controller and Active Directory Restoration
The IT team, along with support from a contractor, has been working to resolve issues with our domain controllers. After initial attempts to fix the problems, we decided the best and really only path forward was to restore from a backup from July 18th—yes, the day before the Crowstrike incident—you can do the math here. We then rebuilt supporting servers using this restored state. All domain controllers are now fully synced, and DNS services have been refreshed.
However, since the restoration point is from July 18th, some recent changes (e.g., new users, shared-drive access, computers, password resets, and account deletions) must be re-applied. Additionally, we have temporarily disabled a sync with this on-premise environment and Office 365 to prevent cloud data loss and further sync issues, and we are working to resolve this as soon as possible. However, there may be some anomalies in access for some users.
We continue monitoring the situation and are addressing the remaining issues, including performing those tasks described as quickly as possible.
Please submit any issues regarding your login account, access to a shared drive, computer not joining the domain, or any issues related to access privileges to the IT Service Desk. We have an Omni-channel helpdesk, so you can submit a ticket, even if you don't have access to the Help Center:
You may submit requests for service via these channels:
- Web: https://help.cityofpleasantonca.gov (access requires your city network login account); Submitting a ticket via the web is the best and most preferred way to request help. This method allows you to enter more pertinent data (including the requested Priority level and due date).
We are back up but are continuing to do some additional configuration. If we have any other changes that might interrupt services we will update this page. Thank you.
Posted Aug 10, 2024 - 20:15 PDT
Update
The changes are taking longer than expected. We hope to have it completed shortly.
Posted Aug 10, 2024 - 19:57 PDT
Update
We will be performing some additional system changes at 6:45 pm. These changes will cause network disruptions while servers are swapped, but the downtime should be minimal (15-30 minutes). We will update here when the system is back up again."
Posted Aug 10, 2024 - 18:32 PDT
Update
We are continuing to work on a fix for this issue.
Posted Aug 10, 2024 - 18:24 PDT
Identified
This root cause has been identified, and we will be implementing a fix - Here are the details:
Emergency Maintenance on Domain Controller Today at 12:30 PM
IT will perform critical changes on our primary domain controller, which is essential to our network's functionality.
Key Details: - Scheduled Start Time: 12:30 PM today - Maintenance Window is 1 hour - Expected Downtime: Minimal to none. However, as with any critical system change, extended downtime is potential. - IT is recommending that you do not reboot your systems during this time to avoid potential issues.
We are taking every precaution to minimize any disruption, but please be aware of potential interruptions in service and the possibility of extended downtime. We will update the status page (status.cityofpleasantonca.gov) and send a follow-up email once the maintenance is complete.
Posted Aug 10, 2024 - 11:49 PDT
Update
Incident Update: Temporary Network Interruption Due to DNS Service Issue
During our recent efforts to address the ongoing domain controller issue, DNS services were unintentionally disabled momentarily, resulting in a temporary network interruption.
We have restored DNS services, and network connectivity has returned to normal. We apologize for any inconvenience this may have caused and appreciate your patience as we continue to work on the domain controller issue.
Please contact our support team if you experience any further issues.
Posted Aug 09, 2024 - 15:31 PDT
Update
This ongoing incident has manifested in other areas requiring identity access and authorization, such as printing and VPN access, among other areas we are discovering.
The root cause has not yet been identified, and the "fixes" implemented seem to have been temporary. The issue is not impacting all users but a small random subset, which makes it all the harder to isolate.
We have reached out to a couple of resources to escalate this matter for expert analysis. It is not that we don't have experts in-house; it's just that the ones we need now have propellers on their hats.
Posted Aug 09, 2024 - 12:37 PDT
Investigating
For some staff, the Pleasdata X:, Y:, and U: drives become inaccessible for periods of time. We are currently investigating this issue.
Posted Aug 07, 2024 - 09:24 PDT
This incident affected: File & Data Server and Global Protect (remote access).