ID 30115: VssError: Insufficient storage available to create either the shadow copy storage file or other shadow copy data.(0x8004231F)

Hi folks,

Quick post about a small problem you may ran into from time to time!

Error Messages
You recevie the following error message in the DPM-console:

DPM failed to synchronize changes for SQL Server 2012 database SQL01\SQLIN03\DPMDB on SQL01.contoso.local because the snapshot volume did not have sufficient storage space to hold the churn on the protected computer (ID 30115 Details: VssError:Insufficient storage available to create either the shadow copy storage file or other shadow copy data.(0x8004231F))

And the following event is logged in the System log on the protected server:

When preparing a new volume shadow copy for volume E:, the shadow copy storage on volume C:\...\{volume-GUID} could not be located in non-critical space. Consider using a shadow copy storage volume that does not have any shadow copies. 

Cause
The Shadow Copy Service is allocated limited amount of space which is not sufficient for the creation of a snap.

Solution

• Right click the volume (in my case E:) in disk management.
• Go to the Shadow Copies-tab and located the volume (still E:) in the "Select a volume"list.
• Press settings and increase the "Maximum size:" or configure it with "No limit:"
• Press OK in the Settings-window and re run your job from the DPM-console

Best Regards

Markus Eliasson

Update Rollup 8 for System Center 2012 R2 Data Protection Manager

Hi folks,

DPM Update Rollup 8 has been released! 
https://support.microsoft.com/en-us/kb/3086084

Unfortunately there is no new features but something worth mention is the following information:

"Note If the DPM server and protected servers are running DPM 2012 R2 Update Rollup 5 (version 4.2.1292.0) or earlier update rollups, the Agent update will require a restart of protected servers."

So what does this mean? Well, are you running UR6 and UR7 DPM-agents in your enviroment. There is no need for restart when you update them to UR8!

Below is the Issues that are fixed in this update rollup:

• The DPM Agent crashes intermittently during a backup.
• If you are trying to recover data from an imported tape, DPM may crash with a "Connection to the DPM service has been lost" error.
• If you try to back up a SharePoint site that uses SQL Always On as a content database, SQL logs are not truncated as expected.
• You cannot verify tape library compatibility for tapes that use RSMCompatmode settings such as IBM 35xx, 2900, and so on.
• If you have multiple SharePoint farms hosted on the same SQL cluster with different instances but the same database names, DPM cannot back up the correct SharePoint farm content.
• If you run Update Rollup 7 for Data Protection Manager 2012 R2, and you have already configured online protection for one or more protection groups, trying to change the protection group populates the default DPM settings for the "Select long-term goals" wizard instead of the previous configured values.
• When you try to protect a SQL failover cluster, the Data Protection Manager UI crashes for every backup or synchronization operation.
• If you install Update Rollup 7 for Data Protection Manager 2012 R2, self-service recovery for SQL databases may not work, and you receive the following error message:
  
  

Kind Regards

Markus Eliasson

Event ID 10028: DCOM was unable to communicate with the computer...

Hi,

A small post regarding my solution to a problem I encountered today.

Issue:

Backup-jobs have been failing and your DPM-agent is listed as unavalible.

In the Event Viewer on your DPM-server event id 10028 is logged in the System log:

DCOM was unable to communicate with the computer contoso-app01.contoso.local using any of the configured protocols; requested by PID a24 (C:\Program Files\Microsoft System Center 2012\DPM\DPM\bin\msdpm.exe

The first thing you would troubleshoot is if there are any firewalls that are blocking the communication but for my case I knew nothing had change in the communication.

Resolution

1. Open up an elevated command promt and run ipconfig /flushdns.

2. Refresh your agent on the Management-tab in your DPM-console.

3. Re-run any failed DPM-jobs.

A very simple solution in my case to get the jobs working again but there is still some work to find out if there is an issue in the DNS registration in Active Directory.

Take care!

//Markus

Restart VSS-writers manually without server reboot

Hi,

Have you ever experienced problem with snapshot-backup due to failed VSS-writers and you cannot restart your server until after business hour or the next service window?

An alternative solution other than restarting the server is to restart each of the associated services for each of the VSS writers!

Find failed VSS-writers and restart service:

• Find each of the VSS writers in a failed state by issuing this command in an elevated command prompt - vssadmin list writers.
• Make a list of all the failed VSS writers or take a screenshot.
• Find the VSS writer's associated Service Display Name in the table below and restart the service.
• Issue the command again to confirm the status has changed Stable with no Error - vssadmin list writers.
• Attempt to perform another snapshot.
• If the VSS writers fail again, it will be necessary to restart the server.

VSS Writer	Service Name	Service Display Name
ASR Writer	VSS	Volume Shadow Copy
BITS Writer	BITS	Background Intelligent Transfer Service
COM+ REGDB Writer	VSS	Volume Shadow Copy
DFS Replication service writer	DFSR	DFS Replication
DHCP Jet Writer	DHCPServer	DHCP Server
FRS Writer	NtFrs	File Replication
FSRM writer	srmsvc	File Server Resource Manager
IIS Config Writer	AppHostSvc	Application Host Helper Service
IIS Metabase Writer	IISADMIN	IIS Admin Service
Microsoft Exchange Writer	MSExchangeIS	Microsoft Exchange Information Store
Microsoft Hyper-V VSS Writer	vmms	Hyper-V Virtual Machine Management
NTDS	NTDS	Active Directory Domain Services
OSearch VSS Writer	OSearch	Office SharePoint Server Search
OSearch14 VSS Writer	OSearch14	SharePoint Server Search 14
Registry Writer	VSS	Volume Shadow Copy
Shadow Copy Optimization Writer	VSS	Volume Shadow Copy
SPSearch VSS Writer	SPSearch	Windows SharePoint Services Search
SPSearch4 VSS Writer	SPSearch4	SharePoint Foundation Search V4
SqlServerWriter	SQLWriter	SQL Server VSS Writer
System Writer	CryptSvc	Cryptographic Services
TermServLicensing	TermServLicensing	Remote Desktop Licensing
WINS Jet Writer	WINS	Windows Internet Name Service (WINS)
WMI Writer	Winmgmt	Windows Management Instrumentation

Kind Regards
Markus Eliasson

The replica is inconsistent with the protected data source (ID30111)

Hi,

Today I ran into a virtual machine which after months of successful jobs started to fail while trying to protect it with via Online Snapshot.

ID 30111

The event logs on the Hyper-V host for the virtual machine did not give me any additional information so I logged onto the virtual machine itself and ran VSSADMIN List Writers and it showed almost all VSS-writers in a "[10] Failed" state with the error "Timed Out".

I restarted all failed VSS-writers but as soon as I restarted the backup-job, the VSS-writers timed out again. The application event log on the virtual machine had the following errors: 

• Event ID 17 "Could not find a matching original volume for shadow volume \\?\Volume{GUID}"

• Event ID 12293: "Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PreFinalCommitSnapshots({a2709227-7262-48ab-987b-5dc13bac4ab9}, 1) [hr = 0x80070490, Element not found.].
  Operation: Executing Asynchronous Operation.
  Context: Current State: DoSnapshotSet.

Resolution

As Event ID 17 was referring to a volume, I opened up an elevated PowerShell-window and ran Diskpart followed by List disk.

From the output, I could see was that one of my volumes were in an Offline state:

Disk 0 Online 60 GB 0 B *
  Disk 1 Offline 20GB 1024KB
  Disk 2 Online 100GB 0 B

I opened up Disk Management (diskmgmt.msc), right-clicked disk 2 and made it Online.

Without further actions, I tried to perform a consistency check on the virtual machine from the DPM-console and the job completed successfully.

Kind Regards

Markus Eliasson

Expired Disk Based Recovery Points are not deleted after installing DPM 2012 R2 UR7

Hi Everyone,

After you install DPM 2012 R2 UR7 update (KB3065246), DPM cannot delete expired recovery points. This will lead to extra recovery point volume usage.  

Please follow the link and instructions below to download and install the fix for this issue:
Fix for a known issue with Update Rollup 7 for System Center 2012 R2 Data Protection Manager

Install Instructions:

Please ensure you are currently running DPM 2012 R2 UR7 before applying the fix.
The issue is not applicable to older updates and will be fixed automatically in later updates. Please perform the below steps on all DPM servers running DPM 2012 R2 UR7. Steps to verify that you are running DPM 2012 R2 UR7

1. Open DPM UI and click on help menu option and click on "About Data Protection Manager" option.

2. A popup will be displayed with version information.

3. Make sure version is 4.2.1338.0, else no need to apply the fix.

Updating pruneshadowcopiesDpm2010.ps1 script:

4. Backup original pruneshadowcopiesDpm2010.ps1 from System Drive:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\bin to another location to make sure you have a copy saved with you.

5. Download pruneshadowcopiesDpm2010.ps1 file from the link above.

6. Replace the original pruneshadowcopiesDpm2010.ps1 at the location in step 4 with the new downloaded pruneshadowcopiesDpm2010.ps1 file.

7. Monitor DPM for one day and it should clean up the expired recovery points, the cleanup happens at regular intervals once a day during midnight.

Shrinking the recovery point volume (optional):

8. In case the Recovery Point Volume already increased and DPM is consuming a lot of storage you can try the option of shrinking the recovery point volume using the below steps: 

     a. Click on the Protection Tab

     b. Right click on the Protection Group and select “Modify Disk Allocation”

     c. Click on the Shrink link next to each Recovery Point Volume, it will display the new size of the volume.

     d. Click on OK it will open a new popup showing the progress of DPM Shrink operation.

Note: Use the Shrink feature judiciously, repeated small shrinks cause free space fragmentation and increases the number of extents.

Kind Regards
Markus Eliasson

Troubleshoot ID 30111: VssError: The specified object was not found

Hi folks,

Here's a quick solution how to fix the following error in DPM in a scenario where you have configured Online Snapshot-protection of a virtual machine:

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError: The specified object was not found. (0x80042308))

Troubleshooting steps:

1. Verify that the virtual machine is not deleted in Hyper-V/VMM.
2. Modify the protection group where you configured protection for the virtual machine.
3. Expand your Hyper-V cluster and locate your virtal machine in the "Modify Group"-wizard.
4. Expand the virtual machine and make sure the checkbox for "Hyper-V\Online\ServerName" is checked.
5. Complete the "Modify Group"-wizard by clicking Next untill it updates the configuration.
6. When you succesfully have created a recovery point for the virtual machine; Stop the protection of the failing datasource.

Problem cause:

This problem will occur when a virtual machine gets clooned or deleted and then recreated again with the same name.

//Markus

Update Rollup 6 for System Center 2012 R2 Data Protection Manager

Update Rollup 6 for DPM 2012 R2 has been realesed!

And Microsoft gives us a lot of bugg fixes as well as two new features:

• Option to keep online backup data while deleting a protection group
• Support for sQL Server 2012 as DPMDB

For more information and download:

https://support.microsoft.com/en-us/kb/3030574

//Markus

Restore DPM DB from Azure Backup Vault

Recently a company contacted me and needed help to restore a failed DPM server. The challenge in the restore was to recover their DPM database as it was protected via Online Protection by the failed DPM-server.

This blogpost will cover how you restore the DPM DB from an Azure Backup Vault when the DPM-server protecting the database has failed.

Prerequisites

·         A clean installation of DPM 2012 R2 with a local SQL 2012-instance on a Windows Server 2012 R2 with the same computer name as the failed DPM-server.

·         The storage connected to the new server as the failed DPM-server had.

·         The Azure encryption passphrase for the failed DPM-server.

Restore the DPM DB from Windows Azure

1.     Log into your Azure Portal: https://manage.windowsazure.com

2.     In the Azure Portal, navigate to Recovery Services – “Your Backup Vault-name”.

3.     Download the Microsoft Azure Recovery Service (MARS)-agent.

4.     From the same page, download the Vault credentials to your Backup Vault.

5.     Install the MARS-agent with your preferred settings on a machine not running DPM.

6.     Open Microsoft Azure Backup that you just installed.

7.     In the right table, press “Recover Data”.

8.     On the Getting Started page choose “Another server” and specify your Vault Credentials.

9.     Select the failed DPM-server and press ext.

10.  Choose “Browse for files” and press next.

11.  Select the volume and choose date and time from which you want to recover, press next.

12.  In the “Available items:” window, select the root-folder and press next.

13.  Choose the path to where you want to recover and keep the other settings as default.

14.  Specify the same passphrase as you had configured on the failed DPM-server and click Recover.

Restore and synchronize the DPM database

1.     Make sure the restored DPM-database is placed on your newly installed DPM-server.

2.     Stop the SQL-Agent service and all DPM-services on the new DPM-server:

·         DPM

·         DPM AccessManager Service

·         DPM Writer

·         DPM RA

3.     Open SQL Server Management Studio and detach the new/empty DPM-database.

4.     Right-click “Databases” in the SQL-instance and choose “Attach…”

5.     Click “Add…” then browse to your restored DPM-database and click OK twice.

6.     Start the SQL-Agent service and all DPM-services.

7.     Open the DPM Management Shell and run:

DpmSync –sync

8.     After you have run the DPMSync –sync command, you must perform a consistency check for all data sources from the DPM-console.

Your DPM-server should now be in a functional state!

//Markus

Troubleshoot ID 32008: DPM cannot protect this SharePoint farm as it cannot detect the configuration of the dependent SQL databases.

Today I helped a customer that was trying to protect a SharePoint-farm with DPM 2012 R2 UR5. 

The SharePoint was configured as followed:

• 2 Front-End web (WFE) servers running on Windows Server 2012 R2
• The SharePoint-databases resided on a SQL 2014 AlwaysOn-cluster.

When we tried to add the SharePoint form to a protection group, we were prompted with the following error:

DPM cannot protect this SharePoint farm as it cannot detect the configuration of the dependent SQL databases. (ID: 32008)

1) Ensure that the SharePoint VSS writer is reporting content databases on the front-end web server.

2) Verify that the SQL Server VSS writer is enabled and running in a healthy state on all the back-end SQL servers machines.

3) Verify if some of the databases that are part of this SharePoint farm are already being protected by DPM. To protect the SharePoint Farm, remove these databases from protection and delete their existing recovery points.

Additional details of the error could not be found in any logs on the DPM-server, WFE-servers or the SQL-servers…

Troubleshooting steps:

1. Verify that the SharePoint VSS writer is running and the status is stable on the WFE-server. From an elevated command prompt: vssadmin list writers 
2. Make sure that DPM are not protecting any SharePoint Config- or Content-databases that belongs to the SharePoint-farm. (Don’t forget to remove any inactive protection as well!)
3. Run the ConfigureSharePoint command on the front-end web server from an elevated command prompt (%program files%\Microsoft Data Protection Manager\DPM\Bin\):
   ConfigureSharePoint.exe –EnableSharePointProtection 
4. (For SQL-cluster using SQL Alias)
   Make sure your SQL aliases is configured properly on your WFEs (both 32bits & 64bits):
   http://mmman.itgroove.net/2013/11/03/setting-up-a-sql-server-alias-with-sharepoint-server-2013/ 

Additional troubleshooting:

However, even after performing the steps above we were still encountered with the same error message and DPM could not enumerate all SharePoint-content databases that belonged to the farm.

What we had to do (even as we could see that the SharePoint VSS writer was up and running) were to run the Stsadm command-line tool on the WFE-server. 

The Stsadm.exe is found together with the other SharePoint products and technologies in the default installation folder: %CommonProgramFiles%\microsoft shared\web server extensions\12\bin.

To register the SharePoint VSS writer you will have to run the following command from an elevated command prompt: stsadm -o –registerwsswriter.

After the operation completed successfully, we could go back to my DPM-server and configure protection for the SharePoint-farm with any errors.

//Markus

List unproteced volumes on your DPM-server

Hi folks,

Last week I encountered a customer that had a challenge that backup operators would add a production server to a protection group in DPM but not protect all necessary volumes on the production server.

To solve their challenge I created a PowerShell-script that queries all protected servers and compare what volumes they have with the protected volumes in DPM. The result of the script is a list with the server name and drive letter on the volume that is not protected.

# Create folder for output-files

New-item C:\Temp\DPM\Scripts -type directory -force | Out-Null

# Clears all content in output-files, if exists

Clear-Content C:\Temp\DPM\Scripts\AllVolumes.txt

Clear-Content C:\Temp\DPM\Scripts\ProtectedVolumes.txt

Clear-Content C:\Temp\DPM\Scripts\NotProtectedVolumes.txt

# Get all servers that is present in any Protection Groupp

$Allps = Get-ProductionServer | where { $_.ServerProtectionState -eq "HasDataSourcesProtected" }

# Gets all Protection Groups

$pg=Get-ProtectionGroup -DPMServerName (&hostname)

# Gets all protected Volumes

foreach ($ps in $pg) { GET-DataSource -ProtectionGroup $ps | where { $_.ObjectType -eq "Volume" } | Select Computer,Name | ft -HideTableHeaders | out-file -Width 50 C:\Temp\DPM\Scripts\ProtectedVolumes.txt -Append }

# Gets all available Volumes

foreach ($Allps1 in $Allps) { Get-DPMDatasource -ProductionServer $Allps1 -Inquire | where { $_.ObjectType -eq "Volume" } | Select Computer,Name | ft -HideTableHeaders | out-file -Width 50 C:\Temp\DPM\Scripts\AllVolumes.txt -Append }

# Removes empty lines and sort data from output-files

(gc C:\Temp\DPM\Scripts\ProtectedVolumes.txt) | ? {$_.trim() -ne "" } | Sort-Object | Set-Content C:\Temp\DPM\Scripts\ProtectedVolumes.txt

(gc C:\Temp\DPM\Scripts\AllVolumes.txt) | ? {$_.trim() -ne "" } | Sort-Object | Set-Content C:\Temp\DPM\Scripts\AllVolumes.txt

# Compare protected volumes with all volumes and exports not proteced volumes

Compare-Object $(Get-Content C:\Temp\DPM\Scripts\ProtectedVolumes.txt) $(Get-Content C:\Temp\DPM\Scripts\AllVolumes.txt) -PassThru | out-file -Width 50 C:\Temp\DPM\Scripts\NotProtectedVolumes.txt -Append

Remember to run the code in an elevated DPM Shell and if you want to use the script for another data-source rather then volumes: Change the "Volume" to for example "System Protection" in line 14 & 17. Example: { $_.ObjectType -eq "System Protection" }

Take care! //Markus

Contribute to the development of DPM

Do you want to have your opinion heard about development of SCDPM? 

Then complete the following survey: https://www.surveymonkey.com/s/DPMdeploymentblockers

Take care all!