Quantcast
Channel: Data Protection Manager - Hyper-V and CSV Clusters forum
Viewing all 575 articles
Browse latest View live

Host-level backup failing

0
0

Hi. We have a 2012R2 Hyper-V cluster that is being backed up by DPM 2012 R2 UR6. We've originally setup the protection groups for host-level backups (using the hyper-v snapshots, vss null provider) and that worked fine.

Due to problems with the cluster nodes, we were forced to reinstall them. After reinstalling, we've reinstalled DPM agent, updated it, configured it to connect to the right DPM server (setdpmserver.exe) and reattached in DPM.

But since then the host-level protection is mostly failing. It occasionally works for some VMs, but for most it fails. We've tried removing all the protection groups and then removed the nodes (agents) from DPM, added them back and reconfigured the protection, but it's still failing for most VMs (I'd say 1/10 works).

The error is always the same:

Affected area:\Online\VM01
Occurred since:10. 10. 2015 19:52:59
Description:The replica of Microsoft Hyper-V \Online\VM01 on VM01 Resources.hvclus.domain.local is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
An unexpected error occurred while the job was running. (ID 104 Details: The RPC server is unavailable (0x800706BA))
More information
Recommended action:Retry the operation.
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution:To dismiss the alert, click below
Inactivate

I've observed that when I run the consistency check (or configure new protection), the VHDX snapshots are created (including the _autorecovery one), DPM pretends to be running the backup for about 2 minutes, but then it fails with the above error and 0MB transferred. The .avhdx files are removed automatically.

Connectivity with the hosts (nodes) is fine. I've also tried moving the VMs to different cluster nodes, changing CSV ownership to different nodes, nothing helps.

There's little information in the node's eventlogs:

 - EventID 10170, source Hyper-V-VMMS: Requester reported unsuccessful backup for the virtual machine 'VM01'. (Virtual machine ID xxx)

 - followed by eventID 16010, source Hyper-V-VMMS: The operation failed.

 - followed by the 2 events for disk merge (start and finish)

Can anyone help?


DPM deduplication on standalone machine

0
0

Hi,


we've been (mostly) following this article to set up deduplication for DPM, adapted for

standalone machines. The article also contains instructions to set a few registry values:

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name DeepGCInterval -Value 0xFFFFFFFF

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name HashIndexFullKeyReservationPercent -Value 70

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name EnablePriorityOptimization -Value 1

Trouble is, "HKLM:\Cluster" doesn't exist when you don't have a cluster. Searching

the web for the settings' names showed me a few people who seem to have a similar

problem, but no definitive solution, only a guess.

"DeepGCInterval" is the only setting that appears in a Microsoft document. The

setting would be made in "HKLM\System\CurrentControlSet\Services\ddpsvc\Settings".

My question: can I assume that this is the right place to do ALL three of the

above registry settings? Is this the place where the dedup service looks for

settings when it's not running on a clustered machine? The path seems to imply that.

--

Cheers,

pk



Backup of VM fails after enabling replica..

0
0

Hi,

I have hyper-v failover cluster running on 2012 R2,
and using DPM 2012 R2 to protects the vm's.

When i enabling the vm to replicate to another host,
that use a nas to Storage. The backup of the primary vm fails With this error:

Starting synchronization on \Online\VM1 failed:

Error 33343: DPM has detected that protection agents are
not installed on the following server(s) : NAS

Recommended action: 1) To install agents on these
servers, on the Agents tab in the Management task area, click Install in the
Action pane.

2) If any of the above servers correspond to a cluster or
a mirror, you need to install the DPM protection agent on all the physical
nodes of that cluster/mirror.

So why does the dpm need to have an agent on the storage
that the replicated vm is using? When I’m trying to backup the primary vm.


Best Regards


DPM deduplication on standalone Hyper-V machine

0
0

Hi,


we've been (mostly) following this article to set up deduplication for DPM, adapted for

standalone machines. The article also contains instructions to set a few registry values:

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name DeepGCInterval -Value 0xFFFFFFFF

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name HashIndexFullKeyReservationPercent -Value 70

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name EnablePriorityOptimization -Value 1

Trouble is, "HKLM:\Cluster" doesn't exist when you don't have a cluster. Searching

the web for the settings' names showed me a few people who seem to have a similar

problem, but no definitive solution, only a guess.

"DeepGCInterval" is the only setting that appears in a Microsoft document. The

setting would be made in "HKLM\System\CurrentControlSet\Services\ddpsvc\Settings".

My question: can I assume that this is the right place to do ALL three of the

above registry settings? Is this the place where the dedup service looks for

settings when it's not running on a clustered machine? The path seems to imply that.

--

Cheers,

pk




Was doing a backup and now my virtual HD is gone!

0
0

Hyper VM guest running Windows 2012. Hyper VM Host is a node in a Windows 2008 R2 2-node HyperV Cluster with CSV

I was running Azure Backup of my on-prem virtual machine (from the inside of the VM). My G Drive vanished and there are these errors in the event log. notice the times for each

Log Name:      System
Source:        disk
Date:          7/24/2015 6:49:24 PM
Event ID:      157
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      CHI-PRODSPSQL.bridgenet.lan
Description:
Disk 3 has been surprise removed.

====================

Log Name:      CloudBackup
Source:        CloudBackup
Date:          7/24/2015 6:49:30 PM
Event ID:      11
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      CHI-PRODSPSQL.bridgenet.lan
Description:
The backup operation has completed with errors.

So, in the VM in Disk Manager, I have 5 drives... C, E, F,H, I. There used to be a G Drive. But it's gone.

In Hyper V , the properties of the VM has 6 drives attached.  From the way they are named I really don't know which VHD file corresponds to which drive letter.. but one of these HDs is attached to the VM and the VM doesn't seem to know about it.

1 - Why the hell would a VM disk be removed? I am so angry right now that I can just explode.  This is my SQL Server and now i'm missing all the DBs on the G Drive.

2 - I'm afraid to even touch the VM now cuz I have no idea what state it is.

I'm so fed up with this crappy software.

DPM failed to clean up data of old incremental backups on the replica for Microsoft Hyper-V

0
0

I keep getting these errors on some hyper-v vms on a windows 2012 r2 cluster. What does this mean and how can i fix them?

DPM failed to clean up data of old incremental backups on the replica for Microsoft Hyper-V \Offline\Server1 on Server1.Cluster1.brazos-ra.dst.tx.us. Synchronization will fail until the replica cleanup succeeds. (ID 30134 Details: The operation completed successfully (0x0))

Hyper-V performence

0
0

Hi

I have 1 DPM server a physical HP DL380.

I backup 1 cluster with 5 nodes. All nodes have the DPM agent installed.

I see the following perfomence from this setup (or so I think). Please see the below picture is this normal?

I have seen this setup flatline at 95-98 % when I was testing the system from a standalone Hyper-V server.

So my question is is this normal? - Is this just because DPM is using some dedupe, and therefore dosn't need to copy everything?
Or do this look low, and what information can I provide to you?

My fear is that it's the receiving storage that can't receive fast enough, as it is some slow disks.

Error 157 - Virtual Machine Somehow in 2 Protection Groups

0
0
I am Running DPM 2012 SP 1. I have a virtual machine that somehow ended up in 2 Protection Groups and now I cannot modify either of those protection groups. It just errors with the following error:

Removing \Backup Using Child Partition Snapshot\ServerX failed:

Error 157: Backup Using Child Partition Snapshot\ServerY cannot be added to protection because it is already a member of a protection group

ServerY is the VM that’s in 2 Protection Groups.

Please advise how I can properly get this removed from one of the protection groups.

I've tried the command shell with no luck.


Best Practice for DPM + Hyper-V Cluster + CSV + Fibre Channel Backups?

0
0

Hi,

we´re evaluating Hyper-V and DPM at the moment. The test setup is as follows:

- 1 FC SAN, created various Volumes, added to the Hyper-V Cluster as CSVs

- 3 Hyper-V Hosts running as a Hyper-V Cluster with FC connection to the SAN

- 1 Physical DPM Server with local Backup storage and also FC connection to the SAN

As I understand, DPM installs the DPM Agent on the 3 Hyper-V Hosts. Within DPM I create a Protection group, selecting the Cluster itself and the VMs it should back up. The Backup process then connects via LAN to the DPM Agents on the Hosts and pulls the Data for the backups from the CSVs.

Now since the DPM also has a FC connection to the SAN, I wonder if I could utilize that one somehow. This will be a huge performance boost (SAN is 8GBit/s, LAN is only 2GBit/s in a Team). I may be completely off with that. But could I just join the DPM Server to the Hyper-V Cluster also, so it has access to the CSVs? DPM will not be used to place VMs of course. But with direct access to the CSVs, would it pull the Data directly from the CSVs via FC?

Or is this idea just plain stupid and it might work, but will break things when going into production? I just wonder if I could utilize that FC connection from the DPM Server to the SAN in some way to speed up backups and restore operations.

Any input to this topic is appreciated. Thanks in advance :-)

Intermittent volume shadow copy Service error: Unexpected error DeviceIoControl 0x80070001

0
0

I keep seeing the following error on 2012 R2 Hyper-V servers which is causing the backing up of the system protection to be very unreliable.

When I look at the shadow copies settings for the C drive I'm trying to protect I get: Error 0,80042306: The shadow copy provider had an error.

Log Name:      Application
Source:        VSS
Date:          10/12/2015 09:39:07
Event ID:      12289
Level:         Error
Description:
Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{8a65780c-2737-4bdb-a2ae-7de2b4060df7} - 0000000000000180,0x00530024,0000000000000000,0,000000DFE0128F20,4096,[0]).  hr = 0x80070001, Incorrect function.
.

Operation:
   Query diff areas on this volume

Context:
   Volume Name: C:\

Most of the posts I've found have been to do with virtual floppy drives on VMware servers but the Hyper-V servers I'm seeing this on have never had VMware installed.

Replica is inconistent

0
0

Hi all,

We are running DPM 2012 R2 on Windows Server 2012 R2

We have a Protection Group: Hyper-V Servers (36 members) where just 2 servers always give the same error message, when we try to run a synchronization job with consistency check.

All servers have "Windows Server Backup" feature installed.

I´ve checked all requirements from this list

  • The Backup integration service must be enabled, which means that the OS running in the VM must support Hyper-V integration services. 
  • The Windows guest OS must support VSS (Windows 2003 or later).
  • Dynamic disks must not be present within the VM.
  • All volumes must be NTFS—even when Microsoft Application Virtualization (App-V), which might create a non-NTFS volume, is used.
  • The VM must be running.
  • VSS storage assignment for the volumes must not be modified.
  • If the VM is part of a cluster configuration, then the cluster resource group must be online.


error details:

The replica of Online\server15 on server23.domain.local is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106) An unexpected error occurred while the job was running. (ID: 104)
   3106
   84de8511-a080-470a-8b32-4d7164bc8ece
   server31.domain.local
   server23.domain.local
   84de8511-a080-470a-8b32-4d7164bc8ece



   Replica inconsistent
   9147bcc5-2fac-4bc6-90c5-7a87d7dce392
   The replica of Online\server15 on server23.domain.local is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106) An unexpected error occurred while the job was running. (ID: 104)
   104
   2
   21744196
   27
   HyperV
   -1
   -1
   -1
   -1
   27446501
   -1
   -1
   -1
   a3bf1f00-fd8a-4554-b8e7-003dfb940004
   c23c04cd-4aa8-48bf-9f3e-37a601677402
   1
   Retry the operation.
   Execute resume DPM backups 


Not working 'exclude VHDX'

0
0

Hi,

I have some VM's with dev databases, which not need to backup.

I run set-DPMGlobalProperty -DPMServerName dpm.mydomain.local -HyperVPagefileExclusions "Sql_server_E.vhdx"

if run: 

Get-DPMGlobalProperty –PropertyName HyperVPagefileExclusions

I get good results, but in DPM server recovery points I see this vhdx is backuped

Servers info:

DPM 2012 R2

Host - W2012R2

VM - W2012R2

VSS timing out when backing up certain large VMs

0
0

It looks very similar to this problem but on Server 2012 R2.

 

Time-out errors occur in Volume Shadow Copy service writers, and shadow copies are lost during backup and during times when there are high levels of input/output

https://support.microsoft.com/en-us/kb/826936

 

Looking at the event logs of the cluster node the VMs are running on I see the following events (all from VSS) repeated from around the time of the VMs with the problem was being backed up:

 

12298 - Volume Shadow Copy Service error: The I/O writes cannot be held during the shadow copy creation period on volume \\?\Volume{GUID}\. The volume index in the shadow copy set is 0. Error details: Open[0x00000000, The operation completed successfully.

], Flush[0x00000000, The operation completed successfully.

], Release[0x00000000, The operation completed successfully.

], OnRun[0x80042314, The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.

].

12297 - Volume Shadow Copy Service error: The I/O writes cannot be flushed during the shadow copy creation period on volume \\?\Volume{GUID}\. The volume index in the shadow copy set is 1. Error details: Open[0x00000000, The operation completed successfully.

], Flush[0x80042313, The shadow copy provider timed out while flushing data to the volume being shadow copied. This is probably due to excessive activity on the volume. Try again later when the volume is not being used so heavily.

], Release[0x00000000, The operation completed successfully.

], OnRun[0x00000000, The operation completed successfully.

].

12341 - Volume Shadow Copy Warning: VSS spent 0x000000000000003c seconds trying to flush and hold the volume \\?\Volume{GUID}\. This might cause problems when other volumes in the shadow-copy set timeout waiting for the release-writes phase, and it can cause the shadow-copy creation to fail. Trying again when disk activity is lower may solve this problem.

12340 - Volume Shadow Copy Error: VSS waited more than 40 seconds for all volumes to be flushed. This caused volume \\?\Volume{GUID}\ to timeout while waiting for the release-writes phase of shadow copy creation. Trying again when disk activity is lower may solve this problem.

8229 - A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.

Interestingly I have another VM that's larger than the failing ones but which doesn't see this issue.

Is there a way to increase the Hyper-V VSS timeout? I've found references to
HKLM\Software\\Microsoft\Windows NT\CurrentVersion\SPP\CreateTimeout but the string isn't present on my system and I've only seen it mentioned in relation to Server 2008 R2, not 2012 R2. http://kb.backupassist.com/articles.php?aid=2997

Edit: Changing the registry key to the 20 minute value mentioned in the article didn't make any difference. While doing some digging to try and find where the 40 second timeout was coming from I found that it's the default value for the NewPathRecoveryTime in MPIO settings. Is this significant?

Linux server shows Online\ServerName while the others are Offline\ServerName, Backups fail/error out on server

0
0

Background Info:

DPM Info:
DPM 2012 R2 Ver: 4.2.1373.0
OS: Windows Server 2012 R2

Linux Info:

Server is hosted on Windows Server 2012 R2 Hyper-V Failover Cluster
OS Version: Ubuntu Version: 14.04.3
2GB RAM
CPU: 1 AMD 6386 SE 2.7GHz

Issue:

I am trying to back up several Linux/Unix servers that are all hosted on Hyper-V FA hosts and 1 server constantly fails to be backed up. All the Linux/Unix VMs are the same OS versions so I am confused they are all listed as Offline\ServerName except for the 1 that is giving me issues. It is labeled as Online\ServerName. If I shutdown the server completely then run a consistency check/recovery point it works as designed. Once the server is turned back on DPM cannot backup the server.

We didn't have LIS tools installed, then we tried with LIS Tools installed. Same issue, no change.

When DPM starts its consistency check the CPU load keeps growing to the point the server becomes unstable. The only way to resolve the CPU load issue is to restart the machine.

An unexpected error occurred while the job was running. (ID 104 Details: Access is denied (0x80070005))

0
0

We have a two node failover hyperV cluster running with CSV. We can backup hyperv servers on one server, but not on the other. Network is working fine, and we can backup normal files from both servers. SCDPM server is W2008R2 with SCDPM 2012R2 UR6, both clients are 4.2.1312.0. A virtual server can be backuped on the first server, then we live migrate it to the other server. The same virtual server then fails to backup.

We get this message in the SCDPM console:

Recovery point creation jobs for Microsoft Hyper-V \Offline\server on server.cluster.domain.com have been failing. The number of failed recovery point creation jobs = 1.
 If the data source protected has some dependent data sources (like a SharePoint Farm), then click on the Error Details to view the list of dependent data sources for which recovery point creation failed. (ID 3114)

An unexpected error occurred while the job was running. (ID 104 Details: Access is denied (0x80070005))

And at the same time, we get in the system eventlog the following DCOM event 10006 message:

DCOM got error "2147942405" from the computer xxxxx.domain.com when attempting to activate the server:
{DA6AA17A-D61C-4E9C-8CEA-DB25DEA52A95}

We backup only to disk. Normal backups of other servers are just working fine. The is no sharepoint farm on the server.

I tried changing the MSDTC rights (http://www.eventid.net/display-eventid-10006-source-DCOM-eventno-272-phase-1.htm), but that didn't solve it.

http://serverfault.com/questions/563088/dpm2010-windows-2012-dcom-errors-communicating-with-all-agents suggests to remove the node from the domain, and back again. This is not something I want to do because it might break the hyperv cluster, and I don't know how to fix that.

Do you have the same problem ? What actions did you do to solve this ?


Hyper-V Virtual Machines backup

0
0

I am looking into best approach for backing up Hyper-V machines running on premise. I have System Center 2012 DPM running and would like to utilize it to make reliable backups from which we can recreate the VMs if needed. 

My plan is to install DPM agent on the Hyper-V host machines and backup each VMs VHD or VHDX to local iSCSI drive mounted in SCDPM server and also to push recovery points to Azure storage for off site backup resolution. 

Please let me know if you have any experience with this setup or perhaps a better idea on how to approach that. 

Thanks,

Chris

DPM Hyper-V backup, with TSM on the windows guest system

0
0

Hi

We have 150+ VM's in our environment, which we have started to backup with DPM.

All servers have a TSM agent installed and running, doing offsite backup of user data.

After we introduced DPM, we see more serveres fail there TSM backup.

I run start my DPM Protection Group at 17:30 and TSM backups start at different times, but we have some cases where we can confirm that the backups have been running at the same time, and is failed with TSM.

So I would like to hear some thoughts about this,
 - Is this really bad practice?
 - Should this not be a problem, and therefor a TSM completely issue?
 - I know I can run pre/post scripts with TSM, so could I stop a service, or otherwise force DPM not to run when TSM is running?
 - The opposite of above, can DPM run a script or something on the VM it is backing up?
 - Or is there something else I don't think of?

Best regards, and thanks in advance

Lars Mortensen

DPM Integration, How do I query cluster?

0
0

I cant figure out how to run the DPM integration pack against a hyper-v cluster. I want to automatically create backups of new VMs but whenever I add the name HVCLUS01 (the name of one my cluster) the job just spins forever and never shows my datasources. If I add a non clustered server to the name, the job runs fast showing me exactly what I'm looking for; all non protected hyper-v VMs.

I've been banging my head against the wall for two nights trying to get this, Any insight would be much appreciated. Thanks

Unable to perform item level recovery on some converted VHD to VHDX files

0
0

Hi

We run a 2012 R2 failover cluster with 2 Hyper-V hosts and we use DPM 2012 R2 to do VM backup's.

When moving machines from the old 2008 R2 failvoer cluster to the new 2012 R2 cluster, many of the VHD files were also converted to VHDX.

It seem's that for many of those we can't do ILR in DPM and browse those VHDX files. All 2003 VM's are affected and many but not all the 2008 R2 VM's. All 2012 R2 VM's Work fine. All 2003 and 2008 R2 VM's taht didn't have their VHD files converted to VHDX, also Work fine.

I checked those suggestions that the error (ID 958) provides and its not because of the use of dynamic disks inside the VM or any of the other suggestions. The only suggestion we dont comply with is that Hyper-V role is not intalled on the DPM server, as the error suggests that we do. But i believe this is not nessesary with Server 2012 R2 and DPM 2012 R2 and since we can open some and other's not, it can't be because of missing Hyper-V role and the functionality to open VHD/VHDX files.

We also tried the suggestion to change the registry value for FsDepends, Start DWORD from 3 to 0.

Has anyone else experienced this and found a solution ?

Best Regards

Martin

VM Backup with Linux Fedora 23 not working

0
0

We have a 2012 R2 Hyper-V cluster and we are using DPM 2012 R2 UR8 to backup our VMs.  We just updated 3 VMs from Linux Fedora 19 to Fedora 23 and now after updating to Fedora 23 when the VM backups run, they are failing with this error in DPM: "The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The writer experienced a non-transient error.  If the backup process is retried, the error is likely to reoccur. (0x800423F4))"

The VSS writer "Microsoft Hyper-V VSS Writer" also goes to a failed state and has a last error of "Non-retryable error"

In the Windows Application Event log this is the error: "A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error.  If the backup process is retried, the error is likely to reoccur.  Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:
   PrepareForSnapshot Event

Context:
   Execution Context: Writer
   Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Name: Microsoft Hyper-V VSS Writer
   Writer Instance ID: {739f04ef-2257-4f4a-b6ba-a4cf451b77bf}
   Command Line: C:\Windows\system32\vmms.exe
   Process ID: 24236

I've reset the VSS writer but that hasn't fixed it.  I have also removed it from DPM and re-added it with no luck.

Does DPM 2012 R2 UR8 support this kind of backup with Linux Fedora 23?

Any ideas how to make this work?

Viewing all 575 articles
Browse latest View live




Latest Images