How a WSFC Default Setting Triggered a P0 SQL Server Outage

This article describes a critical SQL Server Failover Cluster incident caused by a default “Add all eligible storage” setting during node addition and subsequent run validate cluster tool, which led to a DB crash and failover failure, and outlines a detailed step-by-step recovery process and preventive method to avoid mission critical outages. Incident overview. In enterprise architecture, a Severity 1 / Priority 0 (P0) incident represents the highest level of business disruption—a total system blackout where all end-users are completely down and revenue-generating operations grind to a halt. High-availability design is meant to protect systems from failure, but configuration oversights during infrastructure scaling can inadvertently cause the exact outages they are built to prevent. In this article, I break down a real-world production incident involving a SQL Server Failover Cluster Instance (FCI) running on Windows Server Failover Clustering (WSFC). Discover why an administrative wizard checkbox triggered an immediate crash into a SUSPECT / RECOVERY_PENDING state, how the database was forced back online, and how to safeguard your cluster topologies against this issue. What Happened?During a SQL server migration from 2022 to 2025, added a newly provisioned SQL Virtual Machine (VM) node into an active, multi-node SQL Server Failover Cluster. To ensure the new node was completely healthy and integrated correctly, an engineer immediately triggered a Cluster Validation Report. Within seconds of starting the validation tests, the entire active database instance crashed hard. Both the primary and secondary nodes completely lost access to the storage volumes. The primary database slid into a catastrophic RECOVERY_PENDING and SUSPECT state. Attempts to failover to the secondary node failed entirely, rendering the application offline. Also, attempted to restart the DB instance in the management studio, but an error occurred. “Unable to restart service MSSQLSERVER. Additional information. Unable to stop services on server. The object involved has disconnected from its client.” The Error LogsUpon inspecting the SQL Server Error Log, identified a critical failure. Log stated File activation failure. The physical file name "L:\” Databasename”_log1.ldf" may be incorrect. The log cannot be rebuilt because there were open transactions/users when the database was shutdown, no checkpoint occurred to the database, or the database was read-only. This error could occur if the transaction log file was manually deleted or lost due to a hardware or environment failure. Database 'ABC' (database ID 6) startup failed with error 5105, severity 25, state 6. **Root Cause Analysis (RCA) The root cause points directly to a default wizard configuration coupled with aggressive validation checks on live production hardware.1. Checkbox: "Add all eligible storage" When running the Add Node Wizard in Windows Server Failover Clustering, there is a notoriously dangerous default option: "Add all eligible storage to the cluster." Because this box was left checked, the cluster manager aggressively scanned the new VM node, mapped its local or unassigned presentation paths, and swept the existing shared production SAN disks into a state of flux. 2. The Storage Validation Sabotage To verify the cluster's health, ran the full Cluster Validation suite. By default, this suite executes destructive Storage Validation Tests. Because the shared disks holding the live database were now part of the wider, un-stabilized cluster footprint across the new node, the validation tests forcefully broke the active SCSI commands, unmounted the volumes, and dropped persistent reservations to test multi-node failover capabilities. 3. The Mid-Transaction Rip The validation test essentially yanked the L:\ drive (holding the transaction logs) away from the SQL Server engine mid-transaction. Because no clean checkpoints could be written before the storage handles disappeared, the .ldf file became disconnected and desynchronized. SQL Server threw Error 5105 (severe physical file activation crash), preventing database startup and breaking automatic failover. The Technical Solution: Step-by-Step Recovery To recover from a structural file-lock and transaction-log mismatch caused by a cluster validation crash, follow these sequential Steps: Step 1: Stabilize and Clean Up the Cluster Storage Layer Do not attempt database-level repairs until the underlying storage fabric is locked down and stable. 1. Halt Validation: Cancel or wait for any active Cluster Validation tests to stop running completely. 2. Evict Node or Disks: Open Failover Cluster Manager. 2.1 go to Nodes -> right click on newly added node-> more Actions->Evict 2.2 go to Storage -> Disks, locate any disks that were accidentally pulled in by the wizard, and select Remove from Failover Cluster. 3. Clear Persistent Locks: If the production shared disks are stuck in a locked "Offline" state, clear locks via an administrative PowerShell window on the primary node: powershell Clear-ClusterDiskReservation -Disk 4. Bring Resources Online**:** Right-click your SQL Server shared disk resources within Failover Cluster Manager and manually select Bring Online on your primary node. Step 2: Clear SQL Service and File Handles. Because the disk layer was abruptly torn away, SQL Server might maintain dead file latches. Open SQL Server Configuration Manager. Restart the SQL Server instance service to release dead OS handles and attempt a fresh crash recovery cycle. Note: - In our case, restart of SQL Server instance brings up the database back online and resolved the issue. (Most of the case this will work. Otherwise, go for Step 3) Step 3: Force Emergency Database Repair. Because SQL Server cannot rebuild the log file automatically due to the abrupt interruption of open transactions, you must explicitly force the database online and use REPAIR_ALLOW_DATA_LOSS to make the file structures consistent. Execute the following script in SQL Server Management Studio (SSMS): sql USE [master]; GO 1. Force the database into EMERGENCY mode to bypass startup file validation ALTER DATABASE [DB_name] SET EMERGENCY; GO 2. Kick out any orphan connections or user handles to lock exclusive access ALTER DATABASE [DB_name] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; GO 3. Rebuild the corrupted transaction log and repair structural allocation errors NOTE: This will drop uncommitted mid-flight transactions to bring the DB to a transactionally consistent state. If you must run this command, stop the SQL Server instance first (if possible) and physically copy the underlying .mdf and .ldf database files to a backup drive. This preserves an undo path if the repair damages constraints or deletes vital business records. DBCC CHECKDB ('DB_name', REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS, ALL_ERRORMSGS; GO 4. Restore the database back to standard production operation ALTER DATABASE [DB_name] SET MULTI_USER; GO Key Takeaways: - This severe P0 incident provides several critical rules of thumb for database administrators and infrastructure engineers: Always Uncheck Default Storage Options**:** When running the "Add Node Wizard" or "Create Cluster Wizard," always uncheck the default option to "Add all eligible storage to the cluster." Storage deployment should always be an explicit, manual process Never Run Storage Validation in Production: Cluster validation tests are inherently aggressive and intrusive. Never execute a full validation test (especially storage tests) on an active, live production cluster. If you must run validation to verify a new node, explicitly isolate the tests by omitting storage testing from the validation wizard. when choosing to run the storage piece of the CVR. From the SQL side of the equation, this should be doing during a downtime when SQL Server can be stopped before the validation is run. Isolate Provisioning Pipelines: Do not let a brand-new virtual machine (VM) connect to live, critical company data until the VM is fully set up and secure.