A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://learn.microsoft.com/en-us/azure/backup/backup-azure-vms-troubleshoot below:

Troubleshoot backup errors with Azure VMs - Azure Backup

You can troubleshoot errors encountered while using Azure Backup with the information listed below:

Backup

This section covers backup operation failure of Azure Virtual machine.

Basic troubleshooting Common issues

The following are common issues with backup failures on Azure virtual machines.

VMRestorePointInternalError - Antivirus configured in the VM is restricting the execution of backup extension

Error code: VMRestorePointInternalError

If at the time of backup, the Event Viewer Application logs displays the message Faulting application name: IaaSBcdrExtension.exe then it's confirmed that the antivirus configured in the VM is restricting the execution of backup extension. To resolve this issue, exclude the directories below in the antivirus configuration and retry the backup operation.

CopyingVHDsFromBackUpVaultTakingLongTime - Copying backed up data from vault timed out

Error code: CopyingVHDsFromBackUpVaultTakingLongTime
Error message: Copying backed up data from vault timed out

This could happen due to transient storage errors or insufficient storage account IOPS for backup service to transfer data to the vault within the timeout period. Configure VM backup using these best practices and retry the backup operation.

UserErrorVmNotInDesirableState - VM is not in a state that allows backups

Error code: UserErrorVmNotInDesirableState
Error message: VM is not in a state that allows backups.

The backup operation failed because the VM is in Failed state. For a successful backup, the VM state should be Running, Stopped, or Stopped (deallocated).

UserErrorFsFreezeFailed - Failed to freeze one or more mount-points of the VM to take a file-system consistent snapshot

Error code: UserErrorFsFreezeFailed
Error message: Failed to freeze one or more mount-points of the VM to take a file-system consistent snapshot.

Step 1:

If you can't un-mount the devices then you can update the VM backup configuration to ignore certain mount points. For example, if '/mnt/resource' mount point can't be un-mounted and causing the VM backup failures, you can update the VM backup configuration files with the MountsToSkip property as follows.

  1. Check if there is the vmbackup.conf file under the /etc/azure/ directory.

  2. If there's no /etc/azure/vmbackup.conf, you can copy file from the /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.XXX.0/main/tempPlugin/vmbackup.conf.

  3. In the /etc/azure/vmbackup.conf file, add the following configuration for Azure VM Backup to skip fsfreeze and take snapshot for the /mnt/resource mount point.

    cat  /etc/azure/vmbackup.conf[SnapshotThread]
    fsfreeze: True
    MountsToSkip = /mnt/resource
    SafeFreezeWaitInSeconds=600
    
    

Step 2:

Identify the failed to freeze mount points from the extension log file.
For example: /boot, /usr/sap in the below sample output.

    2017-11-02 11:22:56 Thawing: /boot
    2017-11-02 11:22:56 Failed to FITHAW: /boot
    2017-11-02 11:22:56 Thawing: /sapshare
    2017-11-02 11:22:56 Thawing: /usr/sap
    2017-11-02 11:22:56 Failed to FITHAW: /usr/sap

On the Linux VM execute 'mount' command and check if the failed mount points have multiple entries. If yes, remove the old entries or rename the mount path and retry the backup operation.

ExtensionSnapshotFailedCOM / ExtensionInstallationFailedCOM / ExtensionInstallationFailedMDTC - Extension installation/operation failed due to a COM+ error

Error code: ExtensionSnapshotFailedCOM
Error message: Snapshot operation failed due to COM+ error

Error code: ExtensionInstallationFailedCOM
Error message: Extension installation/operation failed due to a COM+ error

Error code: ExtensionInstallationFailedMDTC
Error message: Extension installation failed with the error "COM+ was unable to talk to the Microsoft Distributed Transaction Coordinator

The Backup operation failed due to an issue with Windows service COM+ System application. To resolve this issue, follow these steps:

ExtensionFailedVssWriterInBadState - Snapshot operation failed because VSS writers were in a bad state

Error code: ExtensionFailedVssWriterInBadState
Error message: Snapshot operation failed because VSS writers were in a bad state.

This error occurs because the VSS writers were in a bad state. Azure Backup extensions interact with VSS Writers to take snapshots of the disks. To resolve this issue, follow these steps:

Step 1: Check the Free Disk Space, VM resources as RAM and page file, and CPU utilization percentage.

Step 2: Restart VSS writers that are in a bad state.

Note

Restarting some services can have an impact on your production environment. Ensure the approval process is followed and the service is restarted at the scheduled downtime.

Step 3: If restarting the VSS writers did not resolve the issue, then run the following command from an elevated command-prompt (as an administrator) to prevent the threads from being created for blob-snapshots.

REG ADD "HKLM\SOFTWARE\Microsoft\BcdrAgentPersistentKeys" /v SnapshotWithoutThreads /t REG_SZ /d True /f

Step 4: If steps 1 and 2 did not resolve the issue, then the failure could be due to VSS writers timing out due to limited IOPS.

To verify, navigate to System and Event Viewer Application logs and check for the following error message:
The shadow copy provider timed out while holding writes to the volume being shadow copied. This is probably due to excessive activity on the volume by an application or a system service. Try again later when activity on the volume is reduced.

Solution:

ExtensionFailedVssServiceInBadState - Snapshot operation failed due to VSS (Volume Shadow Copy) service in bad state

Error code: ExtensionFailedVssServiceInBadState
Error message: Snapshot operation failed due to VSS (Volume Shadow Copy) service in bad state.

This error occurs because the VSS service was in a bad state. Azure Backup extensions interact with VSS service to take snapshots of the disks. To resolve this issue, follow these steps:

Restart VSS (Volume Shadow Copy) service.

If the issue still persists, restart the VM at the scheduled downtime.

UserErrorSkuNotAvailable - VM creation failed as VM size selected is not available

Error code: UserErrorSkuNotAvailable Error message: VM creation failed as VM size selected is not available.

This error occurs because the VM size selected during the restore operation is an unsupported size.

To resolve this issue, use the restore disks option during the restore operation. Use those disks to create a VM from the list of available supported VM sizes using PowerShell cmdlets.

UserErrorMarketPlaceVMNotSupported - VM creation failed due to Market Place purchase request being not present

Error code: UserErrorMarketPlaceVMNotSupported

Error message: VM creation failed due to Market Place purchase request being not present.

Azure Backup supports backup and restore of VMs that are available in Azure Marketplace. This error occurs when you try to restore a VM (with a specific Plan/Publisher setting), which is no longer available in Azure Marketplace. Learn more here.

In this scenario, a partial failure happens where the disks are restored, but the VM isn't restored. This is because it's not possible to create a new VM from the restored disks.

If the publisher doesn't have any Marketplace information, you can attach the restored disk(s) (that were created during partial failure) as data disks to an existing VM.

ExtensionConfigParsingFailure - Failure in parsing the config for the backup extension

Error code: ExtensionConfigParsingFailure
Error message: Failure in parsing the config for the backup extension.

This error happens because of changed permissions on the MachineKeys directory: %systemdrive%\programdata\microsoft\crypto\rsa\machinekeys. Run the following command and verify that permissions on the MachineKeys directory are default ones: icacls %systemdrive%\programdata\microsoft\crypto\rsa\machinekeys.

Default permissions are as follows:

If you see permissions in the MachineKeys directory that are different than the defaults, follow these steps to correct permissions, delete the certificate, and trigger the backup:

  1. Fix permissions on the MachineKeys directory. By using Explorer security properties and advanced security settings in the directory, reset permissions back to the default values. Remove all user objects except the defaults from the directory and make sure the Everyone permission has special access as follows:

  2. Delete all certificates where Issued To is the classic deployment model or Windows Azure CRP Certificate Generator:

  3. Trigger a VM backup job.

ExtensionStuckInDeletionState - Extension state is not supportive to the backup operation

Error code: ExtensionStuckInDeletionState
Error message: Extension state is not supportive to the backup operation

The Backup operation failed due to inconsistent state of Backup Extension. To resolve this issue, follow these steps:

ExtensionFailedSnapshotLimitReachedError - Snapshot operation failed as snapshot limit is exceeded for some of the disks attached

Error code: ExtensionFailedSnapshotLimitReachedError
Error message: Snapshot operation failed as snapshot limit is exceeded for some of the disks attached

The snapshot operation failed as the snapshot limit has exceeded for some of the disks attached. Complete the following troubleshooting steps and then retry the operation.

ExtensionFailedTimeoutVMNetworkUnresponsive - Snapshot operation failed due to inadequate VM resources

Error code: ExtensionFailedTimeoutVMNetworkUnresponsive
Error message: Snapshot operation failed due to inadequate VM resources.

The backup operation on the VM failed due to delay in network calls while performing the snapshot operation. To resolve this issue, perform Step 1. If the issue persists, try steps 2 and 3.

Step 1: Create snapshot through Host

From an elevated (admin) command-prompt, run the following command:

REG ADD "HKLM\SOFTWARE\Microsoft\BcdrAgentPersistentKeys" /v SnapshotMethod /t REG_SZ /d firstHostThenGuest /f
REG ADD "HKLM\SOFTWARE\Microsoft\BcdrAgentPersistentKeys" /v CalculateSnapshotTimeFromHost /t REG_SZ /d True /f

This will ensure the snapshots are taken through host instead of Guest. Retry the backup operation.

Step 2: Try changing the backup schedule to a time when the VM is under less load (like less CPU or IOPS)

Step 3: Try increasing the size of the VM and retry the operation

320001, ResourceNotFound - Could not perform the operation as VM no longer exists / 400094, BCMV2VMNotFound - The virtual machine doesn't exist / An Azure virtual machine wasn't found

Error code: 320001, ResourceNotFound
Error message: Could not perform the operation as VM no longer exists.

Error code: 400094, BCMV2VMNotFound


Error message: The virtual machine doesn't exist
An Azure virtual machine wasn't found.

This error happens when the primary VM is deleted, but the backup policy still looks for a VM to back up. To fix this error, take the following steps:

UserErrorBCMPremiumStorageQuotaError - Could not copy the snapshot of the virtual machine, due to insufficient free space in the storage account

Error code: UserErrorBCMPremiumStorageQuotaError
Error message: Could not copy the snapshot of the virtual machine, due to insufficient free space in the storage account

For premium VMs on VM backup stack V1, we copy the snapshot to the storage account. This step makes sure that backup management traffic, which works on the snapshot, doesn't limit the number of IOPS available to the application using premium disks.

We recommend that you allocate only 50 percent, 17.5 TB, of the total storage account space. Then the Azure Backup service can copy the snapshot to the storage account and transfer data from this copied location in the storage account to the vault.

380008, AzureVmOffline - Failed to install Microsoft Recovery Services extension as virtual machine is not running

Error code: 380008, AzureVmOffline
Error message: Failed to install Microsoft Recovery Services extension as virtual machine is not running

The VM Agent is a prerequisite for the Azure Recovery Services extension. Install the Azure Virtual Machine Agent and restart the registration operation.

  1. Check if the VM Agent is installed correctly.
  2. Make sure that the flag on the VM config is set correctly.

Read more about installing the VM Agent and how to validate the VM Agent installation.

ExtensionSnapshotBitlockerError - The snapshot operation failed with the Volume Shadow Copy Service (VSS) operation error

Error code: ExtensionSnapshotBitlockerError
Error message: The snapshot operation failed with the Volume Shadow Copy Service (VSS) operation error This drive is locked by BitLocker Drive Encryption. You must unlock this drive from the Control Panel.

Turn off BitLocker for all drives on the VM and check if the VSS issue is resolved.

VmNotInDesirableState - The VM isn't in a state that allows backups

Error code: VmNotInDesirableState
Error message: The VM isn't in a state that allows backups.

ExtensionSnapshotFailedNoSecureNetwork - The snapshot operation failed because of failure to create a secure network communication channel

Error code: ExtensionSnapshotFailedNoSecureNetwork
Error message: The snapshot operation failed because of failure to create a secure network communication channel.

ExtensionVCRedistInstallationFailure - The snapshot operation failed because of failure to install Visual C++ Redistributable for Visual Studio 2012

Error code: ExtensionVCRedistInstallationFailure
Error message: The snapshot operation failed because of failure to install Visual C++ Redistributable for Visual Studio 2012.

UserErrorRequestDisallowedByPolicy - An invalid policy is configured on the VM which is preventing Snapshot operation

Error code: UserErrorRequestDisallowedByPolicy
Error message: An invalid policy is configured on the VM which is preventing Snapshot operation.

If you have an Azure Policy that governs tags within your environment, either consider changing the policy from a Deny effect to a Modify effect, or create the resource group manually according to the naming schema required by Azure Backup.

UserErrorUnableToOpenMount

Error code: UserErrorUnableToOpenMount

Cause: Backups failed because the backup extensions on the VM were unable to open the mount points in the VM.

Recommended action: The backup extension on the VM must be able to access all mount points in the VM to determine the underlying disks, take snapshot, and calculate the size. Ensure that all mount points are accessible.

Jobs Error details Workaround Cancellation isn't supported for this job type:
Wait until the job finishes. None The job isn't in a cancelable state:
Wait until the job finishes.
or
The selected job isn't in a cancelable state:
Wait for the job to finish. It's likely that the job is almost finished. Wait until the job is finished. Backup can't cancel the job because it isn't in progress:
Cancellation is supported only for jobs in progress. Try to cancel an in-progress job. This error happens because of a transitory state. Wait a minute and retry the cancel operation. Backup failed to cancel the job:
Wait until the job finishes. None Restore Disks appear offline after File Restore

If after restore, you notice the disks are offline then:

Folder is missing when a Linux VM is recovered as a new VM

This issue can occur if disks are mounted to a directory using the device name (e.g., /dev/sdc1) instead of UUID. When the VM reboots or when it is recovered as a new VM, the device names are assigned in a random order. To ensure that the right drive is mounted to your directory, always mount drives using UUID obtained from the blkid utility. Learn more.

UserErrorInstantRpNotFound - Restore failed because the Snapshot of the VM was not found

Error code: UserErrorInstantRpNotFound
Error message: Restore failed because the snapshot of the VM was not found. The snapshot could have been deleted, please check.

This error occurs when you are trying to restore from a recovery point that was not transferred to the vault and was deleted in the snapshot phase.
To resolve this issue, try to restore the VM from a different restore point.

Common errors Error details Workaround Restore failed with a cloud internal error.
  1. The cloud service to which you're trying to restore is configured with DNS settings. You can check:
    $deployment = Get-AzureDeployment -ServiceName "ServiceName" -Slot "Production" Get-AzureDns -DnsSettings $deployment.DnsSettings.
    If Address is configured, then DNS settings are configured.
  2. The cloud service to which to you're trying to restore is configured with ReservedIP, and existing VMs in the cloud service are in the stopped state. You can check that a cloud service has reserved an IP by using the following PowerShell cmdlets: $deployment = Get-AzureDeployment -ServiceName "servicename" -Slot "Production" $dep.ReservedIPName.
  3. You're trying to restore a virtual machine with the following special network configurations into the same cloud service:
  4. Select a new cloud service in the UI or see restore considerations for VMs with special network configurations.
The selected DNS name is already taken:
Specify a different DNS name and try again. This DNS name refers to the cloud service name, usually ending with .cloudapp.net. This name needs to be unique. If you get this error, you need to choose a different VM name during restore.

This error is shown only to users of the Azure portal. The restore operation through PowerShell succeeds because it restores only the disks and doesn't create the VM. The error will be faced when the VM is explicitly created by you after the disk restore operation.

The specified virtual network configuration isn't correct:
Specify a different virtual network configuration and try again. None The specified cloud service is using a reserved IP that doesn't match the configuration of the virtual machine being restored:
Specify a different cloud service that isn't using a reserved IP. Or choose another recovery point to restore from. None The cloud service has reached its limit on the number of input endpoints:
Retry the operation by specifying a different cloud service or by using an existing endpoint. None The Recovery Services vault and target storage account are in two different regions:
Make sure the storage account specified in the restore operation is in the same Azure region as your Recovery Services vault. None The storage account specified for the restore operation isn't supported:
Only Basic or Standard storage accounts with locally redundant or geo-redundant replication settings are supported. Select a supported storage account. None The type of storage account specified for the restore operation isn't online:
Make sure that the storage account specified in the restore operation is online. This error might happen because of a transient error in Azure Storage or because of an outage. Choose another storage account. The resource group quota has been reached:
Delete some resource groups from the Azure portal or contact Azure Support to increase the limits. None The selected subnet doesn't exist:
Select a subnet that exists. None The Backup service doesn't have authorization to access resources in your subscription. To resolve this error, first restore disks by using the steps in Restore backed-up disks. Then use the PowerShell steps in Create a VM from restored disks. UserErrorMigrationFromTrustedLaunchVM ToNonTrustedVMNotAllowed

Error code: UserErrorMigrationFromTrustedLaunchVMToNonTrustedVMNotAllowed

Error message: Backup cannot be configured for the VM which has migrated from Trusted Launch mode to non Trusted Launch mode.

Scenario 1: Migration of Trusted Launch VM to Generation 2 VM is blocked.

Migration of Trusted Launch VM to Generation 2 VM is not supported. This is because the VM Guest State (VMGS) blob created for Trusted Launch VMs isn't present for Generation 2 VM. Therefore, the VM won't start.

Scenario 2: Unable to protect a Standard VM with the same name as of Trusted Launch VM that was previously deleted.

To resolve this issue:

  1. Disable soft delete.
  2. Stop VM protection with delete backup data.
  3. Re-enable soft delete.
  4. Configure VM protection again with the appropriate policy after the old backup data deletion is complete from the Recovery Services vault.

Note

You can also create a VM:

UserErrorCrossSubscriptionRestoreNotSupportedForOLR 

Error code: UserErrorCrossSubscriptionRestoreNotSupportedForOLR 

Error message: Operation failed as Cross Subscription Restore is not supported for Original Location Recovery.

Resolution: Ensure that you select Create New/ Restore Disk for restore operation.

UserErrorCrossSubscriptionRestoreNotSupportedForUnManagedAzureVM  

Error code: UserErrorCrossSubscriptionRestoreNotSupportedForUnManagedAzureVM  

Error message: Operation failed as Cross Subscription Restore is not supported for Azure VMs with Unmanaged Disks.

Resolution: Perform standard restores within the same subscription instead.

UserErrorCrossSubscriptionRestoreNotSupportedForCRR

Error code: UserErrorCrossSubscriptionRestoreNotSupportedForCRR  

Error message: Operation failed as Cross Subscription Restore is not supported along-with Cross Region Restore.

Resolution: Use either Cross Subscription Restore' or Cross Region Restore.   

UserErrorCrossSubscriptionRestoreNotSupportedFromSnapshot 

Error code: UserErrorCrossSubscriptionRestoreNotSupportedFromSnapshot 

Error message: Operation failed as Cross Subscription Restore is not supported when restoring from a Snapshot recovery point.

Resolution: Select a different recovery point where Tier 2 (Vault-Tier) is available.  

UserErrorCrossSubscriptionRestoreInvalidTenant 

Error code: UserErrorCrossSubscriptionRestoreInvalidTenant 

Error message: Operation failed as the tenant IDs for source and target subscriptions don't match.

Resolution: Ensure that the source and target subscriptions belong to the same tenant.

UserErrorCrossSubscriptionRestoreInvalidTargetSubscription 

Error code: UserErrorCrossSubscriptionRestoreInvalidTargetSubscription 

Error message: Operation failed as the target subscription specified for restore is not registered to the Azure Recovery Services Resource Provider. 

Resolution: Ensure the target subscription is registered to the Recovery Services Resource Provider before you attempt a cross subscription restore. Creating a vault in the target Subscription should register the Subscription to Recovery Services Resource Provider.

UserErrorCrossSubscriptionRestoreNotSupportedForEncryptedAzureVM

Error code: UserErrorCrossSubscriptionRestoreNotSupportedForEncryptedAzureVM

Error message: Operation failed as Cross Subscription Restore is not supported for Encrypted Azure VMs.

Resolution: Use the same subscription for Restore of Encrypted AzureVMs.

UserErrorCrossSubscriptionRestoreNotSupportedForTrustedLaunchAzureVM

Error code: UserErrorCrossSubscriptionRestoreNotSupportedForTrustedLaunchAzureVM

Error message: Operation failed as Cross Subscription Restore is not supported for Trusted Launch Azure VMs (TVMs).

Resolution: Use the same subscription for Restore of Trusted Launch Azure VMs.

UserErrorCrossSubscriptionRestoreInvalidTargetSubscription

Error code: UserErrorCrossSubscriptionRestoreInvalidTargetSubscription

Error message: Operation failed as the target subscription specified for restore is not registered to the Azure Recovery Services Resource Provider.

Recommended action: Ensure that the target subscription is registered to the Recovery Services Resource Provider before you attempt a cross subscription restore. Creating a vault in the target Subscription should typically register the Subscription to Recovery Services vault Provider.

Backup or restore takes time

If your backup takes more than 12 hours, or restore takes more than 6 hours, review best practices, and performance considerations

VM Agent Set up the VM Agent

Typically, the VM Agent is already present in VMs that are created from the Azure gallery. But virtual machines that are migrated from on-premises datacenters won't have the VM Agent installed. For those VMs, the VM Agent needs to be installed explicitly.

Windows VMs - Set up the agent Linux VMs - Set up the agent Update the VM Agent Windows VMs - Update the agent Linux VMs - Update the agent Validate VM Agent installation

Verify the VM Agent version on Windows VMs:

  1. Sign in to the Azure virtual machine and navigate to the folder C:\WindowsAzure\Packages. You should find the WaAppAgent.exe file.
  2. Right-click the file and go to Properties. Then select the Details tab. The Product Version field should be 2.6.1198.718 or higher.
Troubleshoot VM snapshot issues

VM backup relies on issuing snapshot commands to underlying storage. Not having access to storage or delays in a snapshot task run can cause the backup job to fail. The following conditions can cause snapshot task failure:

Networking

DHCP must be enabled inside the guest for IaaS VM backup to work. If you need a static private IP, configure it through the Azure portal or PowerShell. Make sure the DHCP option inside the VM is enabled. Get more information on how to set up a static IP through PowerShell:

Migration from Standard to Enhanced policy issue BackUpOperationFailedV2

Error code: BackUpOperationFailedV2

Error message: The backups of VMs have failed after the policy migration from Standard to Enhanced policy.

Cause: Snapshot retention of the Enhanced policy is applied on the older recovery points that were originally created under the Standard policy when you move a VM from a Standard to Enhanced policy.

Recommended action: Contact Microsoft Support for resolution.


RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4