SQL Server Backup Failure Reveals Disk Corruption and VSS Issues
Replace the failing disk, run HDD Regenerator to recover, verify backup integrity, test restore, document incident.
Replace the failing disk, run HDD Regenerator to recover, verify backup integrity, test restore, document incident.
Summary
The backup system failed at the end of 2023 because a VSS provider could not read a snapshot of the MS SQL database server.
Active Business Backup from Synology was unable to read the VSS volume, and the SQL patch that added heavy I/O on audit pages exposed weakened magnetic sectors on the disk.
The disk had bad sectors; running HDD Regenerator rewrote the weak sectors and allowed the database to be read again.
After replacing the failing disk with a new Dell drive, the backup process restored successfully, but the incident highlighted that backups must be verified and that heavy I/O patches can trigger silent corruption.
Key lessons include verifying backup restores, monitoring I/O during patches, and documenting incidents for future reference.
Key changes
- Backup failed due to VSS provider unable to read snapshot
- Active Business Backup from Synology couldn’t read VSS volume
- SQL patch caused heavy I/O on audit pages exposing weakened magnetic sectors
- Disk had bad sectors; HDD Regenerator rewrote them
- Backup verification revealed data loss; offline repair needed
- Replaced disk with new Dell drive, but warranty didn’t cover repair
- Backup process restored after disk replacement and data recovery
- Lesson: verify backup restores and monitor heavy I/O patches