Hi,

I hope someone can advise. I have a problem of my own making. I cannot resolve it and would appreciate some guidance if possible.

I have a clustered environment protected with DPM 2012 R2 UR3. As part of routine testing I recently restored a VM to a non-clustered hyper-V host (HV2). All went well with the restore, but it ran overnight. Once the restore completed it attached to the Hyper-V manager on HV2, but it did not turn on the VM. HV2 has a DPM Agent installed.

When my nightly DPM job ran, the PG that the VM was in decided that the VM was hosted on HV2 and changed the PG to reflect that.

The next morning, I checked the restores, everything worked fine, so I deleted the VM from the HV2 and deleted the VM files. At which point, DPM couldn't find the Server, and backups started to fail. When I realised what was happening, I removed the VM from the PG and attempted to re-add it. It kept coming up as a VM on HV2, not the cluster, and wouldn't re-add as a cluster member. When I expand the cluster in the Modify Group wizard, I can see the VM, but it does not have a checkbox bedside it. So I was stuck with a VM outside the PG and couldn't re-add it. I decided to create a new PG for that VM and it still thought it was on HV2 until I clicked the CLEAR CACHE button. Hey presto, the VM was no longer on HV2 and it appeared in the cluster with a checkbox. So I added it to the PG, set everything up to run out of hours and left it to run. The next morning, the new PG had failed to back up the VM and again it thought the VM was on HV2.

DPM keeps failing to find the VM correctly. The DPM error code is:

The operation failed for Microsoft Hyper-V \Backup Using Child Partition Snapshot\VM1 on HV2.domain.net because the data source is not available. (ID 30169 Details: VssError:The specified object was not found. (0x80042308))

I can only think that this is an entry in the SQL Tables that keeps resetting the PG settings, since everything else appears to be OK. Has anyone else had a similar issue? Or does anyone know how I can resolve this? I have been without a good backup for a few days and I'm getting nervous!

Many Thanks

John

Regards, John

We have a 5 node cluster.  All nodes are running fully patched versions of Windows Server2012 Datacenter (including hotfixes KB2813630 and KB2796995).  Storage is EqualLogic running firmware 6.0.2.  All nodes have EqualLogic HIT 4.5 installed and we are using the hardware provider.  We have two 3TB thin provisioned CSVs setup.  One is not in use.  The other currently contains the first 14 VMs that have been moved from our existing stand-alone Windows Server 2008 R2 SP1 Hyper-V servers.  Only 5 of the 14 VMs are being backed up.  Protection was stopped and started for the move and the required consistency check was performed after the move.  The DPM server is a physical server running SCDPM 2012 SP1 RU2.  All Hyper-V servers have had their agent updated after RU2.  The SCDPM server only has a single protection group setup for all Hyper-V servers (legacy 2008 R2 servers and 2012 cluster).  All backups are succeeding on the legacy servers which are running the same EqualLogic HIT version and are storing their VMs on the same SAN.  Overnight, some backups will fail and others will succeed.  When I fix them up the next day, they will sometimes fail as well even if I tell it to resume backups on one VM at a time.  I can see the hardware snapshots being created on the SAN.  The SAN doesn't report any errors.  SCDPM fails and reports the following:

Type:    Recovery point
Status:    Failed
Description:    The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:A function call was made when the object was in an incorrect state
for that function
 (0x80042301))
    More information
End time:    4/23/2013 3:37:09 PM
Start time:    4/23/2013 3:34:44 PM
Time elapsed:    00:02:25
Data transferred:    0 MB
Cluster node    xxxxx.xxxx.xxx
Recovery Point Type    Express Full
Source details:    \Backup Using Child Partition Snapshot\vm1
Protection group:    Hyper-V VMs - Daily

It leaves the Micrsoft Hyper-V VSS Writer in a failed state with a Timed Out error.  All other VSS writers are fine.  I am also intermittently seeing the following in Application log on some nodes only when backups fail:

Event: 12363

Source: VSS

An expected hidden volume arrival did not complete because this LUN was not detected.

 LUN ID            {350f0b61-0244-4708-abab-a413fb710e7b}
 
 Version            0x0000000000000001
 Device Type        0x0000000000000000
 Device TypeModifier    0x0000000000000000
 Command Queueing    0x0000000000000001
 Bus Type        0x0000000000000009
 Vendor Id        EQLOGIC
 Product Id        100E-00
 Product Revision        6.0
 Serial Number        6090A0881074D4686E17059B9F4365CA
 
 Storage Identifiers
 Version        16
 Identifier Count    2

    Identifier        0
    CodeSet        "VDSStorageIdCodeSetBinary" (1)
    Type        "VDSStorageIdTypeFCPHName" (3)
    Byte Count    16

    60 90 A0 88  10 74 D4 68   6E 17 05 9B  9F 43 65 CA    `....t.hn....Ce.

    Identifier        1
    CodeSet        "VDSStorageIdCodeSetBinary" (1)
    Type        "VDSStorageIdTypeVendorSpecific" (0)
    Byte Count    16

    01 00 00 00  1F BF 0E 6A   00 00 00 3F  00 00 10 54    .......j...?...T
 
 

Operation:
   Exposing Volumes
   Locating shadow-copy LUNs
   PostSnapshot Event
   Executing Asynchronous Operation

Context:
   Execution Context: Provider
   Provider Name: Dell EqualLogic VSS HW Provider
   Provider Version: 4.5.0
   Provider ID: {d4689bdf-7b60-4f6e-9afb-2d13c01b12ea}
   Current State: DoSnapshotSet

Event: 8194

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {d70791b2-f0fe-416e-bbea-e631878ee313}

Hyper VM guest running Windows 2012. Hyper VM Host is a node in a Windows 2008 R2 2-node HyperV Cluster with CSV

I was running Azure Backup of my on-prem virtual machine (from the inside of the VM). My G Drive vanished and there are these errors in the event log. notice the times for each

Log Name:      System
Source:        disk
Date:          7/24/2015 6:49:24 PM
Event ID:      157
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      CHI-PRODSPSQL.bridgenet.lan
Description:
Disk 3 has been surprise removed.

====================

Log Name:      CloudBackup
Source:        CloudBackup
Date:          7/24/2015 6:49:30 PM
Event ID:      11
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      CHI-PRODSPSQL.bridgenet.lan
Description:
The backup operation has completed with errors.

So, in the VM in Disk Manager, I have 5 drives... C, E, F,H, I. There used to be a G Drive. But it's gone.

In Hyper V , the properties of the VM has 6 drives attached.  From the way they are named I really don't know which VHD file corresponds to which drive letter.. but one of these HDs is attached to the VM and the VM doesn't seem to know about it.

1 - Why the hell would a VM disk be removed? I am so angry right now that I can just explode.  This is my SQL Server and now i'm missing all the DBs on the G Drive.

2 - I'm afraid to even touch the VM now cuz I have no idea what state it is.

I'm so fed up with this crappy software.

http://flemmingriis.com/dpm-2012-r2-windows-server-2012-r2-disable-odx/ and the issue persists.

Specifically, the error is: Event 1 VDS Basic Provider, Unexpected failure. Error code: 48F@01000003.

These occur several times then the CSV goes offline and all the VM's crash. Needless to say, this is catastrophic and a bullet proof solution needs to be found.

I'd appreciate any help :)

Hi,

we are trying to protect Session and VDI profiles stored on a csv disk, if we just protect the folder it gives an error

Hello,

I have a new SC DPM 2012 R2 install that we are in the process of rolling out.  I have been searching for a couple of days now for an answer, but with no luck.

We have a number of Redhat 5.5 VM's running on our SCVMM environment.  We would like to get them backed up with DPM.  We have an 8 host cluster of hosts that these servers are guests on.  I have had much success with the rollout of DPM.  However I am now at a point where I cannot figure out why some of the Redhat 5.5 VM's are only offering Offline backups.

I have successfully configure 9 of the VM's that are configured exactly the same and they appropriately backup with Online backup.  However 5 of them will not seem to show Online as an option when configuring a protection group.

I have verified that they have the LIS tools.  They have SCSI disks.  They are on some of the same hosts as others that are working.  All of the HV daemons are running. etc.   

What could be causing these 5 VM's to not offer online backup?

Appreciate any help.

Posted over in the RDS forum and got pointed over here.

I just want to know what is the best way to backup my RDS environment is. Its a 4 node hyper v cluster(compute) and a 2 node scale out file server(Storage). 

We have another hyper v cluster (for servers) that we backup using DPM. That works but that would be a TON of data transfer for doing the 150 virtual machines we have in our RDS setup.

Please advise.

I have set up a two host Hyper-V failover cluster using CSVs and everything is fine but, when setting up the backups, the DPM server, running System Center DPM 2012 R2, states the backup succeeds but it only takes up a very small amount of data and finishes way too quickly.  So, I performed a restore to an alternate location to see what would be restored and it was only the VM config files and no VHD file.  I am wondering why the server only backed up the config files and not the VM's VHD files?  Both hosts have the DPM agent and can even be backed up themselves but no VM?  The DPM server is on 2008 R2 and it's backing up Server 2012 R2 VMs and the hosts are also 2012 R2.  The same DPM server can backup a lab host server with 15 VMs on it just fine but not the cluster.

To clarify, I am using a Dell VRTX with two blades and a shared storage backplane.  After the initial backup, there is no VHD in the backup as seen after doing a restore to a file share--all that was there was the XML config files which makes sense since the amount of "Allocated Disk Space" during the set up of the protection group was only 75KB!

I wasn't sure where to post this as the problem is affecting everything but we mostly backup Hyper-V.Nothing has been backing up since 14th August and no alerts appeared in the monitoring area in that period to indicate that backups weren't running.

What I have seen is a warning alert in that the database had reached a 2GB size threshold. Even though there was enough disk space on the database server it looks like it stopped trying to write to it because this limit had been reached.  The DPM Alerts Windows event log didn’t have any events since the 17^th, the DPM Backup Events didn’t have anything since 14^th.As this is a configurable threshold I increased it to 3GB and performed a consistency check on some VMs that hadn’t been backing up since 14^th August. These backups worked and new events were written to the DPM event logs as normal.

I expected the recovery points to be created to schedule after this as normal but even though consistency checks I initiate manually work it appears that DPM isn't even trying to back up anything at the times scheduled in the PG config.

Hi,

I have installed DPM Server 2012 R2 on Server 2008 R2 Enterprise x64 bit OS and uses BAckup on Disk.

MY All  hyper-V Machines are also hosted on  Server 2008 R2 Enterprise x64 bit  hyper-V Host, Some Virtual Machiness are of server 2008 R2 and some of Server 2012 R2..

While I am creating the Protection Group for taking the Backup of Virtual Machines  I am getting the Description for Server 2008 R2 VM as Backup using Child Partitions Snapshots   and the description for Server 2012 R2 as Backup using Saved State..

hence i am looking for the differences between two uses terms of Hyper-V VMs Backup  as Backup using Child Partitions Snapshots  and Backup using Saved State here. also how can i solve the description Backup using Saved State

Shailendra Dev

Hi

We are running a 3 node Hyper-V failover cluster with a CSV stored on a new SAS connected SAN.

Hyper-V hosts run Server 2012 R2 with all current updates and DPM is 2012 R2 and also with latest updates.

The question is whether we should use the hardware VSS provider and let the SAN do hardware based volume snapshots or if we should just use the VSS Null Provider which uses Hyper-V Checkpoint's to make online VM backup's.

We currently have some issues with the hardware VSS provider which seems both slower when having to negotiate and start the backup's and which seems to have issues with the mapped shadow data volumes getting stuck on the hosts. When using the VSS null provider everything works fine.

I read some places that Microsoft stopped relying on hardware VSS providers with DPM 2012 R2 U3 where they introduced the VSS null provider and made it the default provider, so that in fact you need to disable it in registry to use a hardware provider.

But is that the case ? Is it now recommended by Microsoft to use the VSS Nulll provider to backup VM's hosted on CSV's and what would be the benefits and drawbacks compared to hardware based providers.

I fail to find proper information about that.

Thanks for any links or answers that can help shed some light on to this.

Best Regards

Martin

Hi,

I have installed DPM Server 2012 R2 on Server 2008 R2 Enterprise x64 bit OS and uses BAckup on Disk.

MY All  hyper-V Machines are also hosted on  Server 2008 R2 Enterprise x64 bit  hyper-V Host, Some Virtual Machiness are of server 2008 R2 and some of Server 2012 R2..

While I am creating the Protection Group for taking the Backup of Virtual Machines  I am getting the Description for Server 2008 R2 VM asBackup using Child Partitions Snapshots   and the description for Server 2012 R2 asBackup using Saved State..

hence i am looking for the differences between two uses terms of Hyper-V VMs Backup  as Backup using Child Partitions Snapshots and Backup using Saved State here. also how can i solve the description Backup using Saved State

Shailendra Dev

Hello All,

I am having issues performing a restore of a VM Guest back into a failover Hyper-V Cluster.

Current Setup:
DPM 2012 R2
Windows Server 2012 R2 (Physical Server)
DPM is connected to a Tape Library and Dedicated LUN on a SAN for backups
3 Hyper-V Hosts configured with Failover Clustering

I installed the DPM agent on all 3 hosts, performed a backup of a VM Guest which displays "Online\Servername". I deleted the server from the failover cluster then removed from Hyper-V Manager. I then proceeded to perform a restore of the server but it does not give me an option to restore to original location so it can be re-integrated into the failover cluster.

What am I doing wrong here? I am not an expert in Hyper-V so when I looked at the prerequisites for Hyper-V backups of VMs on CSV it stated a VSS Hardware Provider needs to be installed. I didn't setup the VMHosts so I am not sure if or what it is.

I did run vssadmin list providers on one of the hosts so I am assuming I have one?

C:\Windows\system32>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
   Provider type: Software
   Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
   Version: 1.0.0.1

Provider name: 'Microsoft CSV Shadow Copy Provider'
   Provider type: Software
   Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
   Version: 1.0.0.1

Provider name: 'VSS Null Provider'
   Provider type: Software
   Provider Id: {8202aeda-45bd-48c4-b38b-ea1b7017aec3}
   Version: 4.2.1338.0

Provider name: 'Microsoft File Share Shadow Copy provider'
   Provider type: Fileshare
   Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
   Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

Provider name: 'VSS Null Provider'
   Provider type: Fileshare
   Provider Id: {f4a69dd4-f712-40e3-a6b3-faeff03cb2b8}
   Version: 4.2.1338.0

Any guidance would greatly be appreciated. Thank you!

Hello,

I have this situation:

1. Several standalone (no cluster) production Hyper-V server with VM's on it. Lets call them original VM's. 

2. Backup server, with Hyper-V role on it. One VM on this server is DPM. And on this server is target for VM replication of original VM's (replica VMs)

DPM is configured with VMM for support for live migration. 

I configure DPM to protect original VM. But if there is replication of VM on backup server, backup job fails with error that there is no DPM agent on backup server. If I install DPM agent on backup server - it back up not the original VM, but replicated VM.

One workaround is to uninstall VMM agent on backup server, but i need real solution.

How to tell DPM that it should backup original VM, but not the replica ?

I would like to know whether is there a way to know what is being replicated for hyper-v recovery point, lately I realize DPM is replicating huge amount data for my recovery point.

Sometime it happen everyday, file server that used to replicate 20-30 GB daily now can replicate over 1TB. The point I want to bring up is that on the recovery point for 30 days retention for the recovery point only used about 120GB which does not portion to the daily replicated data.

Just wonder anyone is facing the same issue and whether there is a work around for the issue, this large amount of replication is affecting my replication to my Secondary DPM Server.

Thanks you.

Hi

I'm having issues with successfully changing some VM servers which are in a clustered Hyper V 2012 R2 environment from a DPM 2012 R2 server located in one site to the other. 

The VMs themselves were recently migrated over from one site to the other in Hyper-V and are working fine. This required the Protection Group in DPM which was backing up successfully from the original site to be stopped (ORIGINALSITEDPM). I kept the restore points. Then I set it up again on the DPM server on the other site for DR. 

Two of the VM servers are now backing up on DPM just fine, but 7 out of the 9 have not. I get the following errors on the failing serversUnable to configure protection :-

Details:-

The bit that puzzles me is why it is using the ORIGINALSITEDPM reference at all, since these devices are located on the production cluster and are now in the Inactive Protection on the ORIGINALSITEDPM server, just like the other ones which did work fine. So far I have tried rebooting both DPM servers in both sites. The only thing I have not tried is rebooting the actual VMs as they are 24/7 production boxes and I don't wish to do that unless I'm fairly sure it will fix the issue.

Thanks in advance 

Host DPM Server OS: Windows Server 2012 R2

DPM 2012 R2 - Physical Server

I am trying to restore a VM to a prior date and I get this error when selecting restore to original location

DPM was unable to access the cluster resource group ServerName.ClusterName.Domain for Online\ServerName of protection group Hyper-V Guests. Recover the datasource using the alternate location workflow. (ID: 32579)

Has anyone come across this issue before and what did you do to fix it? Thanks.

Hi All,

I have a DPM 2012 R2 (Rollup 7) server running on Windows Server 2012 R2, and I have constant issues protecting Hyper-V workloads. The DPM server is backing up physical DC's and other Windows servers without issue but has constant issues protecting Hyper-V servers. I have tried to protect two stand alone 2012 R2 Hyper-V nodes and each of them seems to be stuck at 0% for hours. Only using the Microsoft VSS providers. Once when I rebooted a Hyper-V server and the DPM server it backed up for a day and then stopped working again with the same symptoms. Can anyone offer advice on this? As can be seen from the attached image it stays on 0MB data transfer. I am losing confidence in DPM to protect Hyper-V workloads!

Carl Smith MCITP-EA

Hi,

I am currently configuring DPM 2012 R2 in a hosted environment. As part of this I am trying to test the recover of a Hyper-V VM to a different physical host from its original location using the "Recover as virtual machine to any host" option. However when using this option all it seems to do is move the virtual machine from 1 physical host to another and does not apply the recovery point.

Current environment configuration:

3 x Physical Server hosts (all Windows Server 2012 R2) configured as a failover cluster (inc storage)

Hyper-V 2012 R2 role installed on all 3 hosts

System Center VMM 2012 R2 used to manage Virtual Machines

Steps used in the process are as follows:

1. In the DPM console select the Recovery option

2. Browse to the VM you want to restore

3. Expand the VM node and select "All Protected Hyper-V Data"

4. Select Recoverable item in the Path list

5. Right-Click and select "Show all recovery points"

6. Select the recovery point to restore and click Recovery

7. On the Review Recovery Selection page click Next

8. Select the "Recovery as virtual machine to any host"

9. Click Next

10. Click Browse

11. Select the Hyper-V physical host and clustered storage location for where the VM is going to be recovered to.

12. Click Ok.

13. Click Next

14. Click Next again

15. On the Summary page click Recover

16. The recovery process with proceed and restore the virtual machine to a different physical host but will not recovery the virtual machine as per the recovery point.

Can anyone please confirm if I am using the correct process or whether I am doing something obviously wrong in this process?