Removing a protection group after storage failure

August 14, 2018, 7:17 am

≫ Next: DPM 2019 insists offering non-existing VM a for Hyper-V host

≪ Previous: Access Denied - Trying to protect a VM stored on SOFS

Hello Dear all,

We experienced a storage failure of our DPM 2016 server - lost the Disk for a configured protection group data is down the drain. A replacement drive came in and i have configured it as available storage. I am attempting to rebuild the protection groups - aka restart backups but i have to stop the old ones first/ The protection group is saying volume missing. When i Attempt to stop it i get

Removing C:\ failed:
Error 40005: The specified storage is unusable as it is a system volume, has existing data, is in an unhealthy state or formatted incorrectly.
Recommended action: Ensure that the storage is healthy and mounted and then retry the operation."

I attempted stooping the group without deleting the data and again no go.

How do we force stop and remove the protection group ?

Thanks in advance.

↧

DPM 2019 insists offering non-existing VM a for Hyper-V host

April 17, 2019, 10:29 am

≫ Next: Error 40002

≪ Previous: Removing a protection group after storage failure

Hello everybody,

a fresh install of DPM 2019 on Windows Server 2019 Datacenter shows all VMs on a freshly installed Windows Server 2019 Datacenter Hyper-V host, and it backs them up just fine. However, DPM offers a strange VM named "Th-VM" which does not exist, which we did never create, and whose name would be far away from our VM naming standards. So if we choose Hyper-V auto-protection for that host, DPM tries to backup that VM and, of course, fails doing so. We can run backups fine if we uncheck both Auto-Protection and that strange VM. A refresh of the host in the DPM protection group wizard (having the host not expanded in the tree) still shows up this "Th-VM" VM.

Why does DPM offer us a VM which does not and did never exist, and how can we get rid of it? Thanks!

Best Regards, Stefan Falk

↧

Error 40002

November 3, 2017, 4:23 am

≫ Next: Fail to backup Hyper-V 2016 VM

≪ Previous: DPM 2019 insists offering non-existing VM a for Hyper-V host

Hi,

We have DPM 2016 running on WS2016 the major PG are VM running on hyper-v cluster 2012 R2, I install DPM, create PG and online protect, all run ok, for prower problems (Puerto Rico hurricane Maria), we need off the datacenter every 2 weekends, but the Monday when the datacenter power on, DPM cannot run any PG and this error is show for the jobs:

The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)

More information
Recommended action: Ensure that the storage containing the VHD is healthy, mounted, and the VHD is not in use by other processes, and retry the operation.

Isaias

↧

Fail to backup Hyper-V 2016 VM

May 16, 2019, 2:49 am

≫ Next: Backup Hyper-VM including Checkpoints

≪ Previous: Error 40002

I had receive error as below when backup my Hyper 2016 VM

pls advise.

either the servername needs reboot , is part of a cluster then some/all nodes in the cluster need t be rebooted.

↧

Backup Hyper-VM including Checkpoints

May 17, 2019, 12:58 am

≫ Next: HCI Node Error 5120 on Reboot with DPM Agent Installed

≪ Previous: Fail to backup Hyper-V 2016 VM

Hi..

we are running DPM 2016 RU6 to protect our clustered Hyper-V (2016) VMs.

Now I've tried to restore some of them and I'm wondering why the recovered VM lost all her Checkpoints...

Only one vhdx file was restored. I've tried also a network folder target. One vhdx and two config files (vmcx, vmrs) are the only one that were restored.

Does somebody know if this is the default behavior, that only the current selected/running checkpoint will be used for the backup?

Is it possible to enable DPM to backup any VM Source?

Use a file system backup to backup the VM folder are no option for us because the VMs are running on a S2D Cluster and so the storage location of the VM can dynamically changed...

Thanks

Alex

↧

HCI Node Error 5120 on Reboot with DPM Agent Installed

May 21, 2019, 12:59 am

≫ Next: cannot backup VM - The DPM service was unable to communicate with the protection agent

≪ Previous: Backup Hyper-VM including Checkpoints

Hi,

I have newly deployed, 3 node, Windows 2016 (Build 2724) Hyper-Converged Hyper-V/S2D cluster which is operating normally. The intention is to use DPM 2016 LTSB UR6 to perform VM level backups hosted on the cluster.

After deploying the DPM agents (5.0.375.0) to the cluster nodes, i've started to see cluster errors generated when rebooting a node, CSV auto pause errors for each CSV in the cluster:

"CSVName has entered a pause state because of 'STATUS_UNEXPECTED_NETWORK_ERROR (c00000c4)"

When rebooting a node, there are no backup jobs running, if I remove the DPM agents the nodes reboot without error.

Also, it may be a red herring, but there are entries in the cluster logs generated during a node reboot that relate to an error closing a file named ' DpmFilterTrace.txt' which is stored on the CSVs. Not sure if this is normal or an indication of an issue.

Would appreciate any input from people protecting workloads on HCI clusters using DPM.

Thanks in advance,

James

↧

cannot backup VM - The DPM service was unable to communicate with the protection agent

May 24, 2019, 7:04 am

≫ Next: Problem moving Hyper-V host/guests to new protection server

≪ Previous: HCI Node Error 5120 on Reboot with DPM Agent Installed

Hello.

New day, new DPM problem:

Cannot protect one particular VM using host-level protection. It's the only VM having this problem. The error message (redacted names):

Affected area: RCT\<VMNAME>
Occurred since: 24.05.2019 15:52:46
Description: The replica of Microsoft Hyper-V RCT\<VMNAME> on <HVHOST> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with
 consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
The DPM service was unable to communicate with the protection agent on <HVHOST>. (ID 52 Details: The remote computer refused the network connection (0x800704C9))
More information
Recommended action: 
1) Restart the DPM Replication Agent service on <HVHOST>.
2) If <HVHOST> is configured using certificates, ensure that the DPM CPWrapper service is running on DPM Server and aof0078.ad.apponfly.com. Also ensure that certificates used by both the computers are valid.
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution: To dismiss the alert, click below

Can backup all other VMs on the Hyper-V host without issues. Hyper-V 2016 Server protected by DPM 2019. Previously protected by DPM 2016 UR7, where backup of this particular VM worked. HV hosts and DPM are in the same domain.

What I've tried:

- CC job - no good

- removed the protected DS and re-add it to DPM - no good. Actually the initial backup 'finishes' without a problem, but the backup has ~100MB (instead of ~40GB). RP count 0. CC job doesn't help.

- migrated the VM to different host - same issue

- 'reset' RCT (by shutting down the VM and deleting the .RCT .MRT files) - no good

- tried monitoring the DPMRA log during the backup attempt: this error comes out, however doesn't makes it any clearer:

1250 16A4 
05/24 13:51:13.117
03 cbackupsubtask.cpp(653)
[000001F7BE491E40] C4FBC1EF-FB64-41D5-9F10-5A44C6B4D7B0
WARNING <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147023671" Reason="Error" CommandID="RAReadDatasetFixup" CommandInstanceID="044cde5b-e0d9-4fd3-8c37-a787048583e2"
 GuidWorkItem="e8436d7b-8204-4910-b7a6-d08637ec1f45" TETaskInstanceID="c4fbc1ef-fb64-41d5-9f10-5a44c6b4d7b0"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147023671"
 DetailedSource="2"/><RAStatus><RAReadDatasetFixup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><LWVStatus BytesTransferred="1272" NumberOfFilesTransferred="7" NumberOfFilesFailed="0"
 DataCorruptionDetected="false"/><FixupStatus BytesTransferred="0" NumberOfFilesTransferred="0" NumberOfFilesFailed="0" DataCorruptionDetected="false"/></RAReadDatasetFixup></RAStatus></Status>

Any ideas are welcome

↧

Problem moving Hyper-V host/guests to new protection server

October 15, 2018, 3:44 am

≫ Next: DPM deduplication on standalone Hyper-V machine

≪ Previous: cannot backup VM - The DPM service was unable to communicate with the protection agent

We have a problem regarding Data Protection Manager and backup of a virtual SQL machine running on Hyper-V.

The setup is as follows:

We have 4 DPM servers running Windows 2012 R2 and DPM 2016 (5.0.361.0) - DPM01, DPM02, DPM03 and DPM04.

Then we have a Hyper-V Production server running Windows 2012 R2 - HPV12.

Finally HPV12 is running 6 Hyper-V virtual machines - among those SQL02.

The problem is this:

We moved protection of HPV12 from DPM04 to DPM02 - no problem.

Then we moved protection from DPM02 to DPM01 - but now we only see 5 of 6 VMs under HPV12 - SQL02 is not showing up (and we did the refresh, in the correct way).

To troublshoot we moved protection of HPV12 back to DPM02 - now we see SQL02. Then to DPM01 again - now we don't see SQL02.

↧

DPM deduplication on standalone Hyper-V machine

October 9, 2015, 6:32 am

≫ Next: Problem moving Hyper-V host/guests to new protection server

≪ Previous: Problem moving Hyper-V host/guests to new protection server

Hi,

we've been (mostly) following this article to set up deduplication for DPM, adapted for

standalone machines. The article also contains instructions to set a few registry values:

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name DeepGCInterval -Value 0xFFFFFFFF

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name HashIndexFullKeyReservationPercent -Value 70

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name EnablePriorityOptimization -Value 1

Trouble is, "HKLM:\Cluster" doesn't exist when you don't have a cluster. Searching

the web for the settings' names showed me a few people who seem to have a similar

problem, but no definitive solution, only a guess.

"DeepGCInterval" is the only setting that appears in a Microsoft document. The

setting would be made in "HKLM\System\CurrentControlSet\Services\ddpsvc\Settings".

My question: can I assume that this is the right place to do ALL three of the

above registry settings? Is this the place where the dedup service looks for

settings when it's not running on a clustered machine? The path seems to imply that.

Cheers,

↧

Problem moving Hyper-V host/guests to new protection server

October 15, 2018, 3:44 am

≫ Next: HCI Node Error 5120 on Reboot with DPM Agent Installed

≪ Previous: DPM deduplication on standalone Hyper-V machine

↧

HCI Node Error 5120 on Reboot with DPM Agent Installed

May 21, 2019, 12:59 am

≫ Next: Replica is inconsistent

≪ Previous: Problem moving Hyper-V host/guests to new protection server

Hi,

After deploying the DPM agents (5.0.375.0) to the cluster nodes, i've started to see cluster errors generated when rebooting a node, CSV auto pause errors for each CSV in the cluster:

"CSVName has entered a pause state because of 'STATUS_UNEXPECTED_NETWORK_ERROR (c00000c4)"

When rebooting a node, there are no backup jobs running, if I remove the DPM agents the nodes reboot without error.

Would appreciate any input from people protecting workloads on HCI clusters using DPM.

Thanks in advance,

James

↧

Replica is inconsistent

June 19, 2019, 11:37 am

≫ Next: DPM 2019 - Consistency Check

≪ Previous: HCI Node Error 5120 on Reboot with DPM Agent Installed

Hi Everyone!

I am kinda stuck on an issue and have not been able to find an answer to that solves the problem. I was wondering if anyone can point me in the right direction.

I currently have a Hyper-V 2016 server that I am backing up using DPM 2016, Now the Hyper-V host its self has roughly about 50 VM's and about 35 of them are backing up without issue however I have about 15 guests that are all in a "Replica is inconsistent" State with the following error in even log :

The replica of <Guest> on <Server> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
An unexpected error occurred while the job was running. (ID: 104)

Performing a consistency check does nothing and the alert pops back into the console. Any ideas or help ?

Stuart.

↧

DPM 2019 - Consistency Check

July 2, 2019, 1:09 am

≫ Next: DPM 2016 backups failing - Unknown error ID 104 - Leaving Checkpoints behind

≪ Previous: Replica is inconsistent

Hi,

I have a file server backed up using DPM 2019, that is in an inconsistent state. When I run the consistency check, after a while access to shares on the server is interrupted to the point that files cannot be opened. When I stop the check, access is fine again.

I have ran 2010 and 2019 and have had no issue with this before.

The server that I am backing up is on a 2012 Hyper V Fail-over Cluster connected to iSCSI SAN. My backup server is on a dedicated server.

Any help with this would be much appreciated.

Thanks,

Mark

↧

DPM 2016 backups failing - Unknown error ID 104 - Leaving Checkpoints behind

July 3, 2019, 3:14 am

≫ Next: NX / DX bit in windows 7 and Bitlocker failure

≪ Previous: DPM 2019 - Consistency Check

Hi,

I have been having a recent issue with DPM 2016 UR7 where my VM backups have been failing and leaving behind "checkpoints" that don't actually exist in Hyper-V.

If I then try to run the backup job again I get faced with Unknown error ID 104. I've seen a few posts regarding this however no fixes that have worked for me.

I have tried rebooting the virtual machines in question to find that they fail there merge. I have updated all protection agents to use UR7 agent (5.0.409.0). Some of the virtual machines will merge once shutdown and backup again successfully but a day or two later I will be faced with the same issue.

Has any experienced anything similar recently?

I would love some ideas on this one, in some cases if I shut down the VM to merge and try to start again I will be faced with power on error, virtual hard disk chain is corrupt of which I then run SET-VHD command to point back to the parent disk. It will then boot and backup.

Just seems like a really odd issue and has only been happening over the course of the past week or 2. I have also noticed that DPM has stopped deleting expired recovery points outside of the retention range but I believe that is a different issue.

Cheers.

↧

NX / DX bit in windows 7 and Bitlocker failure

November 14, 2018, 7:25 pm

≫ Next: Consistency checks failing on one cluster, succeeding on another

≪ Previous: DPM 2016 backups failing - Unknown error ID 104 - Leaving Checkpoints behind

On windows 7 machine after turning on NX in BIOS and turning on "Data Execution Prevention" option per below link the laptop failed to boot up. It is asking for Bitlocker key which is lost.

Can you revert back to "Last Good Configuration" or is there any option left to boot up the system?

https://social.technet.microsoft.com/Forums/en-US/2395b730-5d3d-4245-9470-5673f2ec5fc0/how-to-enable-nx-dx-bit-in-windows-7-without-bios-in-windows-10-this-automatically-enabled?forum=win10itprohardware

Thanks.

↧

Consistency checks failing on one cluster, succeeding on another

July 29, 2019, 12:01 am

≫ Next: VMGS files missing in backup - unable to recover VMs

≪ Previous: NX / DX bit in windows 7 and Bitlocker failure

Hi @ all,

[all servers involved are 2016, patched up to July CU]

trying to back up Hyper-V VMs running on three separate clusters. All VM configs and disks reside on the same SOFS living on an S2D cluster. The clusters are as follows:

Production: 10 nodes on HP Synergy blades, production VLAN and SMB VLAN sharing the same converged networking infrastructure.

DMZ: 2 nodes on HP rack servers, production VLAN is 4x1GB Cu, SMB VLAN is 10GB optic

Management: 2 nodes on HP rack servers, production VLAN is 4x1GB Cu, SMB VLAN is 10GB optic. This is where the DPM VM resides.

Shares on the SOFS, permissions, delegation and so on are configured in an identical way and differ only in regard to the machine accounts actually granted permissions. All permissions are granted via Active Directory groups.

Virtual machines can be created, managed and live migrated within their respective clusters without any issues.

DPM 2019 10.19.58.0 on a VM with a local SQL2016 instance.

DPM agent is installed on all hosts in all clusters, connectivity shows OK. Replicas of both Windows and Linux VMs can be created without warnings. And here's where it gets interesting:

Any VMs on the Management Cluster (where the DPM VM is running) can be backed up without issue regardless of OS or power state.
On the DMZ cluster, an initial replica can be created and shows OK. On attempt to create a recovery point, there is an error saying "Replica is inconsistent". Consistency check ends in error state saying ID 2033 (a required privilege is not held by the client (0x80070522). The UNC path and host shown in the error message are where the VM is stored and registered. This behaviour does not change with the power state of the VM.
On the production cluster, an initial replica stays green for several seconds then goes red. From there on, the behaviour is identical to taht on the DMZ cluster.

All hosts have been rebooted at least once since DPM agent rollout.

What am I missing?

added the machine account of the DPM server to Full Access on one of the shares, to no avail
enabled File Server VSS writer on S2D cluster nodes, to no avail (backup of Management cluster worked before I did that)

Evgenij Smirnov

http://evgenij.smirnov.de

↧

VMGS files missing in backup - unable to recover VMs

August 2, 2019, 10:03 am

≫ Next: Hyper-V host level protection

≪ Previous: Consistency checks failing on one cluster, succeeding on another

Hello,

I have a trouble with VM backup in our DPM 2016: it looks like the DPM is not backing VMGS files for VMs.
I am using Hyper-V backup feature of DPM, DPMs are hosted on Windows Server 2019, with resilient change tracking
(RCT), all VMs are stored on ReFS volumes.

The backup contains vhdx, VMCX and VMRS file, but ***VMGS*** file is missing, this behavior is identical for all backed up VMs.

The restore or VM import fails with following Event Error in Hyper-V-VMMS, ID 18440

Import failed. Import task failed to copy file from 'F:\SomeVM-2_8-1-2019_18.23.2\d-Vol\vm\SomeVM\Virtual Machines\E51373A3-3030-452F-996E-0B09E8ECCF1C.vmgs' to 'C:\ProgramData\Microsoft\Windows\Hyper-V\Planned Virtual Machines\E51373A3-3030-452F-996E-0B09E8ECCF1C.vmgs': The system cannot find the file specified. (0x80070002).

(because VMGS file is really not present in the backup set restored on disk).

DPM Version 5.0.409.0, Update Rollup 7 for SCDPM 2016
DPM OS Version : Microsoft Windows Server 2019 Standard 10.0.17763.557

Hypervisor OS Version : Microsoft Windows Server 2019 Standard 10.0.17763

Any ideas?
Thank you,
Jan

↧

Hyper-V host level protection

November 25, 2018, 12:58 pm

≫ Next: hyperv replica

≪ Previous: VMGS files missing in backup - unable to recover VMs

I have a Hyper-V guest playing fileserver role and which happens to play other roles as well (AD/DS, DHCP, DNS).

Further I have a (virtual) DPM server (dedicated machine with a physical disk passed-through) and I want to do use this to make host level protection of the fileserver/ad/ds/dns/dhcp machine with retention range of e.g. 15 rec. points /days).

For the fileserver part (one dedicated VHDX) I want to make backups with retention range of e.g. 60 rec. points.

Is this possible without massive duplication? (It seems like the host level backup is always includes all attached HDD's)

↧

hyperv replica

August 22, 2019, 11:35 pm

≫ Next: How to fix Event ID 5217 and 5120 when backup up a VM on CSV with DPM

≪ Previous: Hyper-V host level protection

Dears,

can someone advise on how to configure hyperv replication between a cluster of 2 hyper-v in main site and one standalone hyperv in the DR.

many thanks

↧

How to fix Event ID 5217 and 5120 when backup up a VM on CSV with DPM

July 11, 2013, 10:47 am

≫ Next: Backup multiple Hyper-V clusters with 2 DPM 2016 servers setup with Scale-Out Protection

≪ Previous: hyperv replica

I have a 4 Node Cluster with 2 Shared Cluster Volumes on an ISCSI SAN. Each node is up-to-date with Updates and is running Server 2012 DataCenter. I have a seperate 2012 server with Microsoft DPM on it I use to backup my VMs. The cluster has 24 diffrent VMs on it all diffrent sizes, OSs, and configurations. Recently DPM has been getting hung on one of the VMs. When the backup gets hung, I get events

5217

Software snapshot creation on Cluster Shared Volume(s) ('\\?\Volume{f3c25587-f296-481e-b2fc-606c4eef1e92}\') with snapshot set id '{dd4a47b9-e645-4303-9df6-c978b29b3cd5}' failed with error 'HrError(0x00000001)(1)'. Please check the state of the CSV resources and the system events of the resource owner nodes.

and

5120

Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_FILE_CLOSED(c0000128)'. All I/O will temporarily be queued until a path to the volume is reestablished.

And naturally I cannot backup the VM. All of the other VMs backup fine.

Anyone got any Ideas?

↧