Recently I was working on a problem for a Windows Server 2008 system where the daily incremental backups were failing. These backups had never worked, but the weekly full backups were successful.
First step was reviewing the failed backup logs (these were CommVault logs), and you could see the following:
and
So, the next thing you do is run up a command prompt and check the vss writers on that server:
The system writer was not listed.
Next step was to check if there were any conflicting tasks such as AV scans running at the same time as the backup, there weren’t any.
I then went on to review the system and application event logs to see if there were any VSS errors. Interestingly there were none of these either.
Of course I had to try the reboot option (just to see if it would work), but nope, this didn’t help.
Researching lead me to the usual Microsoft articles on missing vss system writer:
KB 2009272 and a blog article on missing system writer case explained.
Although I didn’t see any VSS errors in the event logs, I decided to double check the permissions for the two directories mentioned in KB 2009272 anyway just to be on the safe side. As I did not have direct access to the end system I tried to be “smart” about this and decided to use PowerShell to check the permissions. With a couple of lines of PowerShell I could get the File Permissions and the Directory Permissions.
File Permissions:
Get-ChildItem “c:\windows\winsxs\filemaps” | Get-Acl | export-csv c:\filemaps_fileperms.csv
Get-ChildItem “C:\Windows\winsxs\Temp\PendingRenames” | Get-Acl | export-csv c:\pendingrenames_fileperms.csv
Directory Permissions:
Get-ChildItem -Path ‘C:\Windows\winsxs’ -directory -Filter “*filemap*” | get-acl | export-csv c:\filemaps_permissions.csv
Get-ChildItem -path ‘C:\Windows\winsxs\Temp’ -directory -filter “*pendingnames*” | Get-Acl | export-csv c:\pendingrenames_permissions.csv
Unfortunately I was being a bit too clever for my own good. Simple screenshots would have been much easier in the long run, as I hadn’t taken into account the way PowerShell formats the output it generates. Ultimately all the permissions were correct, which I kind of suspected given that I didn’t have any of the corresponding VSS errors in the event logs.
Proceeding onwards I reviewed the second Microsoft article and got the Microsoft.Net cache folders to be checked and if any files present to clear them out and restart the cryptographic service. There were not however any temp files in Microsoft.Net cache directories, so another dead end.
I was out of ideas, I had done all the suggestions in the articles (without running Procmon), but it had all been covered off. So I bit the bullet and raised the case with Microsoft.
What did they do? Confirm that the system writer was missing by running vssadmin list writers, checked the event logs for VSS errors (none present) then use Procmon.
- Start Procmon logging.
- Run command vssadmin list writers at an elevated command prompt.
- Stop Procmon logging.
On reviewing the log file they saw the following:
So the setupapi.ev1, setupapi.ev2 and setupapi.ev3 files are locked.
Referring back to the Microsoft blog I mentioned earlier, you can see the registry key entry it mentions.
Next the Microsoft engineer had a look at the 3 locked files in the c:\windows\inf directory and compared them to a system with the same OS version to check file sizes. There was a difference. Microsoft said that these 3 files were corrupt.
To resolve the problem, all we had to do was copy setupapi.ev1, setupapi.ev2 and setupapi.ev3 from the working system (same OS level) to the problem one and then restart the cryptographic service.
Testing with an incremental backup was then successful!