We run a 3 node Server 2012 R2 cluster and backup using DPM 2012 R2

Everytime a VM is live migrated to another node, when DPM next tries to back it up, the recovery point fails with the "Change Tracking has been marked inconsistent" error and a consistency check needs to be done.

This is fine if one VM is Live Migrated but when we have to put a host in Maintenance mode for some reason and 50 VM's need to be migrated off, it them means 50 consistency checks and our backups go from 2 hours to 8 hours.

Is there a way around this or is this just the way DPM works?

Simon Holman
Expeed Technology
Australian Web Hosting

hi,

I had a 2012 Hyper-v cluster which i upgraded to 2012 R2. As part of this you need to make a new cluster and then migrate the VM's across. 

This obviously breaks DPM protection as the VM's it was protecting are listed as oldcluster\vmname. Is there anyway to avoid starting the protection from scratch? I tried stopping protection on the VM and leaving the replica on disk and then re-adding it, but it sees it as a new protection so creates a new replica.

We have TB's of data and while this is ok for the primary DPM site, we have a secondary DPM protection site for disaster recovery which is on a much slower link, which will takes weeks to resync.

is there a way to avoid having to resync all the data again that it already has?

thanks

Hi All,

First of all, I know it's a beta and these are the perils of being an early adopter, but I've got a serious problem.

I've upgraded our production Hyper-V cluster to Server 2012. The setup is a 4 node cluster running CSVs on an ISCSI SAN with MPIO via dual gigabit Ethernet networks. The SAN storage is provided byOpen-E DSS7 and replicated to another server in a different building.

Post the upgrade everything about the cluster seemed stable and to work as expected - live migrations etc all working. I then turned my attention to backups, and I discovered that Server 2012 wasn't supported by DPM. Fortunately there is a beta of DPM 2012 SP1 which adds support for Server 2012, unfortunately there is no upgrade path from the beta to RTM of SP1. Not wanting to upgrade our production DPM server to a beta, I installed a copy of DPM 2012 SP1 beta on a VM to provide a stopgap backup solution for VM level backups of certain machines that couldn't be backed up in other ways. I realise that running the backup server on the same cluster / SAN as the stuff that's being backed up is an odd thing to do, but this at least serves to provide snapshots, SAN replication provides resilience, and like I say, this is a stopgap.

Then I started noticing problems. First symptom was that on starting / rebooting VMs, sometimes other VMs would hang for perhaps 30s - 2m, people would start complaining that SharePoint had gone unresponsive etc. However, they would come back to life in a minute or two.On a couple of occasions we came in in the morning to find a number of VMs off or paused (backups ran overnight). Both of these problems occurred only when the DPM server was turned on. I thought the issue might be general load on the SAN, having both the backup server and the machines being backed up living on the same CSV / hardware. I moved the DPM server to a different ISCSI box and put on aggressive throttling (200Mbps) to try to reduce load, but the problem continues.

The event logs on the Hyper-V cluster suggest I/O timeouts to the SAN at the times of the backups. Lot's of event ID 1069, 1205, 1146, 1230,  (various cluster resources failed). The interesting one I think is 5120 Cluster Shared Volume 'Volume5' ('VOLUME NAME') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

Is anyone else using SP1 beta to successfully backup a 2012 Hyper-V cluster? 

Is anyone seeing the same problem?

Is it likely that this is a problem with SP1 beta, will it be fixed at RTM?

Any suggestions for a stopgap solution?

I think I might try setting up a test physical DPM server to check the issue isn't in someway related to the fact that the DPM server sits on the same cluster it's backing up. I'm also happy to consider the problem could lie elsewhere i.e. with the SAN storage (this was upgraded from v6 to v7 at the same time as the 2012 upgrade, but as soon as I tell the vendor that the problem relates to running a beta of DPM they will be pointing fingers at that.

Thanks,

Tim

I'm in the process of setting up a 2012 R2 CU2 DPM Secondary Server - need help or step by step direction

Installed the DPM Agent on the primary DPM server, when i created the protection group on the secondary I selected a server from the (Cluster) no problem, backed up the server. 23GB in 3min.

Trying to add more servers the DPM app crashed and lost communication with DPM.

Trying to figure out what i should be backing up?  Virtual machines or the Primary DPM server?

Also my SQL Database is on a remote SQL which i installed the Agent to this from the secondary server. Backed up no issue.

Thanks

ANNCEX

Hi everyone!

There is a hyper-v cluster built on Windows server 2012 datacenter, and SC DPM 2012 as a backup software. After we updated DPM to Rollup 5 backups of VM that using two or more cluster disks as a storage began to fail with an error he replica of Microsoft Hyper-V \Backup Using Saved State\xxxx on SCVMM xxxx Resources.xxxxx is not consistent with the protected data source. DPM error ID = 93. (ID 33123).  Even if I detach all disks, exept one from VM backup still fails with the same error.  

Hi,

we are running a 6 Node 2012 R2 HyperV Cluster with two CSV Volumes hostet on an emc VNX5300 connected with FC. Everytime our DPM 2012 R2 is starting a backup of more than two vm machines the performance of the complete cluster is going down. The Server all fully patched....

We are getting event ID 5120 and 5217 in the failover Cluster manager, does anybody know a solution?

Thanks

Moelli

Hello,

I currently have the problem that my DPM Backup of 2 of my VMs is not working.

The VM is a 2008 R2 and the Hyper-V is a 2012R2. The newest Integration tools are already installed.
DPM is a 2012 R2 and both VMs are SQL Server.

When DPM tries to backup the VM after about 2 Minutes I get the following error on the DPM:

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
 (0x800423F4))

Within the VM itself I get the following error in the application log:

Level: Error
Source: vmicvss
EventID: 19
Description:
Not all the shadow volumes arrived in the guest operating system.

------------

Level: Error
Source: VSS
EventID: 12293
Description:
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PreFinalCommitSnapshots({36da5a5a-d516-4683-9cb7-c62fbdac9e1d}, 3) [hr = 0x800705b4, This operation returned because the timeout period expired.
]. 

Operation:
   Executing Asynchronous Operation

Context:
   Current State: DoSnapshotSet

------------

Level: Error
Source: SQLVDI
EventID: 1
Description:
SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=4132. Thread=7344. Client. Instance=. VD=Global\{5E902890-836C-4CFD-A83A-55F63BF7E628}1_SQLVDIMemoryName_0. 

Could someone help me resolve this issue.

Thanks in advance

Paul

Hi All,

• Server 2012 r2
• DPM2012
• Cluster Backups 

We have seen an issue I've read in a few forums where VHDs are getting locked when performing a cluster backup.  Unfortunately we're unable to reproduce the issue and it happens irregularly (once every month or so).

Within HyperV manager the status of the VM is "Backing Up" or "Creating VSS Snapshot", however the VM is unable to be managed at all.  When connecting via the console, the VM looks up, but you're unable to get to a log on screen.

I've found that by restarting the DPM agent on the host, restarting the job from DPM (which forces a failed state), you're then able to start the VM from the failover cluster manager.  Within Virtual Machine Manager, the VM is listed as "Missing", though once you start through the cluster, it's all good.  

Once the VM is up and you log on to it, there is nothing in the event log for the time the backup has been running.  The last event ID in the log, references VSS issues.

We've seen the issue on 2 VMs, which both happen to be XenDesktop Controllers, so unsure if there is something local causing it.

Wondering if anyone was able to get a result for this.  I've patched the servers and logged a job with MS, but they are being painful with the troubleshooting process and the fact it's an irregular issue which we can't replicate doesn't help.

As I learned from many blogs, starting from Windows Server 2012, there is a single VSS snapshot created for backup multiple VMs located at the same CSV disk.

For example, I have a Hyper-V 2012 R2 cluster with multiple CSV disks. I choose one of these CSVs to backup. I add half of VMs into one PG and another half into another PG. No other resources in these PGs. Schedules for both PGs are the same.

In this case, how many snapshots will be created for that CSV? One, because both PGs start backup at the same time, or two, because DPM requests snapshot creation for each PG separately?

http://about.me/exchange12rocks

Hello

I'm trying to add a passthrough disk to DPM 2012 R2 VM, i can see the disk from the disk management in the DPM VM, and when i want to add it to the storage pool the DPM cannot detect it. When i try with a vhdx, i can add it perfectly but i had a very slow performance.

Additional info:
- Disks are initialized as GBT
- Disks are 1 TB LUN .
- San is FC (Dell compellent SC8000)
- OS used: Windows 2012 R2 Datacenter for HyperV nodes and VMs,
- Disk type used for passthrough disk from VM is SCSI

Anyone a suggestion?

Thanks in advance,

BAKIR

I am using DPM 2012 with a Windows Server 2008 R2 based Hyper-V cluster.

The SAN which holds the CSV is a Dell Equallogic, so I guess I could use hardware VSS, but I really do not need the parallel backups, so I just went the simpler route with the system VSS provider.

So far I have been maintaining DataSourceGroups.xml more or less by hand by running DSConfig.ps1 when there are changes to which VMs must be protected, but I came across thisarticle: http://robertanddpm.blogspot.dk/2010/07/enabling-serialized-backup-of-hyper-v.html, which suggests a scheduled task for always keeping DataSourceGroups.xml up to date.

Running the Powershell script generates a DataSourceGroups.xml with every VM in the cluster, but many of them do not need backing up at all.

Does it present a problem for DPM that there will be several VMs listed in DataSourceGroups.xml, which are not in any protection group?

Hi

We have a 2 Node Hyper V 2012 Cluster. We are using DPM 2012. The Protection Group Number 1 we have is backing up SQL on our SSO Server. I tried creating another Protection Group Number 2, to complete a backup of the Hyper V of the same SSO Server. I am not able to complete a successful back of Protection Group #2 (Hyper V).

Does any know if it is possible to complete SQL and a Hyper V Backups on the same server?

Thanks

Shane M

Hey guys, I'm trying to restore a folder from a server which runs on a Hyper-V cluster. The cluster is managed by SCVMM. We're using DPM 2012 R2. In DPM, I go to the Recovery tab, select the "SCVMM servername Resources (clustername.domainname.com)", open the protected Hyper-V data, select a date, and open the VHD. In the VHD I browse to the folder I want to restore and select "Recover". So far, so good.

A new wizard pops up, the only option I have is to restore to a network location. In the next window, I can select a server. I select the fileserver where the file came from, but the volumes on the server are not loading. I'm just looking at a loading symbol. After 10 minutes, the hourglass disappears, and I can't select any volumes on the server.

I tried to select the DPM server where I'm working on, and I get exactly the same results. I can't select a volume to restore the data to. I tried to restore the data using Powershell, but that didn't work either. There are no other users logged on to the server. The fileserver where the files came from is located in another domain, but the backups are performed without any issues. Does anyone have a clue how I can restore these files?

We have a 5 node cluster.  All nodes are running fully patched versions of Windows Server2012 Datacenter (including hotfixes KB2813630 and KB2796995).  Storage is EqualLogic running firmware 6.0.2.  All nodes have EqualLogic HIT 4.5 installed and we are using the hardware provider.  We have two 3TB thin provisioned CSVs setup.  One is not in use.  The other currently contains the first 14 VMs that have been moved from our existing stand-alone Windows Server 2008 R2 SP1 Hyper-V servers.  Only 5 of the 14 VMs are being backed up.  Protection was stopped and started for the move and the required consistency check was performed after the move.  The DPM server is a physical server running SCDPM 2012 SP1 RU2.  All Hyper-V servers have had their agent updated after RU2.  The SCDPM server only has a single protection group setup for all Hyper-V servers (legacy 2008 R2 servers and 2012 cluster).  All backups are succeeding on the legacy servers which are running the same EqualLogic HIT version and are storing their VMs on the same SAN.  Overnight, some backups will fail and others will succeed.  When I fix them up the next day, they will sometimes fail as well even if I tell it to resume backups on one VM at a time.  I can see the hardware snapshots being created on the SAN.  The SAN doesn't report any errors.  SCDPM fails and reports the following:

Type:    Recovery point
Status:    Failed
Description:    The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:A function call was made when the object was in an incorrect state
for that function
 (0x80042301))
    More information
End time:    4/23/2013 3:37:09 PM
Start time:    4/23/2013 3:34:44 PM
Time elapsed:    00:02:25
Data transferred:    0 MB
Cluster node    xxxxx.xxxx.xxx
Recovery Point Type    Express Full
Source details:    \Backup Using Child Partition Snapshot\vm1
Protection group:    Hyper-V VMs - Daily

It leaves the Micrsoft Hyper-V VSS Writer in a failed state with a Timed Out error.  All other VSS writers are fine.  I am also intermittently seeing the following in Application log on some nodes only when backups fail:

Event: 12363

Source: VSS

An expected hidden volume arrival did not complete because this LUN was not detected.

 LUN ID            {350f0b61-0244-4708-abab-a413fb710e7b}
 
 Version            0x0000000000000001
 Device Type        0x0000000000000000
 Device TypeModifier    0x0000000000000000
 Command Queueing    0x0000000000000001
 Bus Type        0x0000000000000009
 Vendor Id        EQLOGIC
 Product Id        100E-00
 Product Revision        6.0
 Serial Number        6090A0881074D4686E17059B9F4365CA
 
 Storage Identifiers
 Version        16
 Identifier Count    2

    Identifier        0
    CodeSet        "VDSStorageIdCodeSetBinary" (1)
    Type        "VDSStorageIdTypeFCPHName" (3)
    Byte Count    16

    60 90 A0 88  10 74 D4 68   6E 17 05 9B  9F 43 65 CA    `....t.hn....Ce.

    Identifier        1
    CodeSet        "VDSStorageIdCodeSetBinary" (1)
    Type        "VDSStorageIdTypeVendorSpecific" (0)
    Byte Count    16

    01 00 00 00  1F BF 0E 6A   00 00 00 3F  00 00 10 54    .......j...?...T
 
 

Operation:
   Exposing Volumes
   Locating shadow-copy LUNs
   PostSnapshot Event
   Executing Asynchronous Operation

Context:
   Execution Context: Provider
   Provider Name: Dell EqualLogic VSS HW Provider
   Provider Version: 4.5.0
   Provider ID: {d4689bdf-7b60-4f6e-9afb-2d13c01b12ea}
   Current State: DoSnapshotSet

Event: 8194

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {d70791b2-f0fe-416e-bbea-e631878ee313}

Hi,

I have read the MS documentation regarding excluding pagefile churns from virtual machines to increase performance of backups and reduce storage space required and I would like to implement this within our environment.

http://technet.microsoft.com/en-us/library/jj721498.aspx

However, when I move the pagefiile from the system disk onto a seperate VHD I get the following warning:

"If you disable the paging file or set the initial size to less than 400 megabytes and a system error occurs, Windows might not record details that could help identify the problem. Do you want to continue?"

As I understand things this means that I may not get a crash dump if a BSOD occurs. Is this correct, and if so how can I mitigate? How have others managed this scenario, do you just live with the fact that you won't get a dump file in the event of a problem?

Thanks in advance.

Microsoft Partner

We have a Server 2012 R2 cluster that connects to an Equallogic SAN and backs up to a DPM server with a dedicated JBOD array

When we backup a VM using DPM 2012 R2, there is a large amount of network traffic over both the Cluster Heartbeat and Live Migration networks. Both networks on the host running the VM show usage of aprox 500Mbps inbound and both networks on the Host that owns the CSV volume show usage of 500Mbps outbound. That almost seems to me to be CSV Redirected IO but when I run the Powershell command Get-ClusterSharedVolumeState it shows all CSV's in Direct mode.

My 2 question is, why is it using both networks (Cluster and VM) on each host, with almost an identical throughput?

The data transfer to the Management network on the DPM server is about 1/4 of the throughput between the hosts.

Simon Holman
Expeed Technology
Australian Web Hosting

We have a DPM 2012 R2 backing up a Server 2012 R2 3 node cluster running Hyper-V (CSV) "ClusterA".

We have a VM "ServerA" which we migrated using VMM from a Windows Server 2008 R2 stand alone Hyper-V machine to "ClusterA" without issues.

Inside DPM we edit the existing protection group for ClusterA and see the new ServerA in the list. It is not possible to expand that server object to see the Hyper-V checkbox underneath it. Which we can for the other servers in that protection group.

We don't get any error messages or event logs, just no expansion. Anyone knows how to troubleshoot this kind of behaviour?

Thanks!

Erik, with many, many certs! ;-)

Hi All,

I'm trying to backup a VM on a 2012 Cluster. I can do it by using the system VSS provider, but when I try to use the hardware provider, (Dell equalogic) it doesn't work. DPM will sit for a while trying and then report a retryable VSS error. 

The only error I'm seeing on the Host is the following:

Event ID 12297
Volume Shadow Copy Service error: The I/O writes cannot be flushed during the shadow copy creation period on volume \\?\Volume{3312155e-569a-42f3-ab3a-baff892a2681}\. The volume index in the shadow copy set is 0. Error details: Open[0x00000000, The operation completed successfully.
], Flush[0x80042313, The shadow copy provider timed out while flushing data to the volume being shadow copied. This is probably due to excessive activity on the volume. Try again later when the volume is not being used so heavily.
], Release[0x00000000, The operation completed successfully.
], OnRun[0x00000000, The operation completed successfully.
]. 

Operation:
   Executing Asynchronous Operation

Context:
   Current State: DoSnapshotSet

I don't know where to go from here - There is no activity on the CSV (this is the only VM on it, and both the CSV and VM were created specifically for testing this issue

Does anyone have any ideas? I'm desperate. 

Update:

Ok, so I can Take DPM out of the picture. Trying to do a snapshot from the Dell Auto-Snapshot manager, I get the same errors. But I also get a bit more information:

Started at 3:02:47 PM
Gathering Information...
Phase 1: Checking pre-requisites... (3:02:47 PM)
Phase 2: Initializing Smart Copy Operation (3:02:47 PM)
Adding components from cluster node SB-BLADE01
Adding components from cluster node SB-BLADE04
Adding components from cluster node SB-BLADE02
Retrieving writer information
Phase 3: Adding Components and Volumes (3:02:52 PM)
Adding components to the Smart Copy Set
Adding volumes to the Smart Copy Set
Phase 4: Creating Smart Copy (3:02:52 PM)
Creating Smart Copy Set

An error occurred:

An error occurred during phase: Creating Smart Copy
    
    Exception from HRESULT: 0x80042313.
  
Creating Smart Copy Set

An error occurred:

An error occurred during phase: Creating Smart Copy
    
    Exception from HRESULT: 0x80042313.
  

An error occurred:

Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_FREEZE'. Check the application component to verify it is in a valid state for the operation.

An error occurred:

One or more errors occurred during the operation.  Check the detailed progress updates for details.

An error occurred:

Smart Copy creation failed.

Source:  Creating Smart Copy Set

An error occurred:

An error occurred during phase: Creating Smart Copy
    
    Exception from HRESULT: 0x80042313.
  

An error occurred:

Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_FREEZE'. Check the application component to verify it is in a valid state for the operation.

An error occurred:

One or more errors occurred during the operation.  Check the detailed progress updates for details.
Error:  VSS can no longer flush I/O writes.

Thanks,

John