Thursday, December 8, 2016

VMware vSphere 5.5 sysprep customization never completes and fails for virtual machine

I recently ran into an extremely strange issue at a client’s environment that a VMware support engineer never managed to resolve and it was only by chance that I was able to figure out a workaround.  The likelihood of running into this issue again is probably quite low but I thought I’d write this blog post in case anyone else comes across it as I did.


You’ve noticed that deploying virtual machines from template with customization specifications no longer complete and result with the following error in VMware View when attempting to deploy full clone VDIs:

Customization operation timed out

Pariing state: In pairing…


Reviewing the full clone virtual machine logs show the start of customization:

Started customization of virtualMachine.

Customization log located at C:\Windows\TEMP\vmware-imc\guestcust.log in the guest OS.


Reviewing the guestcust.log does not show any errors:



While the problem described above can be caused by many reasons, the reason I encountered a few weeks ago was one that I would have never guessed.  What caused the customization process to fail in this environment I had to troubleshoot in was that the ESXi version was from a release date that was later than the vCenter version.

The ESXi version in this environment was 5.5.0, 3568722

The vCenter version in this environment was 5.5.0, 3252642

Using the information provided by the following KBs:

Build numbers and versions of VMware ESXi/ESX (2143832)

Build numbers and versions of VMware vCenter Server (2143838)

Will show the following information for the ESXi host and vCenter:

Version: ESXi 5.5 Express Patch 10
Release Date: 2016-02-22
Build Number: 3568722

Version: vCenter Server 5.5 Update 3b
Release Date: 2015-12-08
Build Number: 3252642

To correct the issue, update vCenter to a version with a later release date than the ESXi host and note that having the vCenter version newer than the ESXi host is best practice.

If upgrading the host is not an option, a strange workaround I noticed was that the customization would complete successfully if I disable HA on the cluster:


I’m not sure if this is specific to this environment but if there are no other options, try temporarily disabling HA and see if a new cloned VM would complete the customization.

Wednesday, December 7, 2016

Unable to reconfigure HA on VMware ESXi 5.5.0 hosts after failed vCenter upgrade


You’ve recently attempted to upgrade vCenter 5.5 from 3b to 3e but had to rollback the upgrade to a previous snapshot due to an issue.  Since the upgraded vCenter server was briefly started and had upgraded the HA agents on the ESXi hosts, you now see alarms thrown for the cluster as well as the hosts. The following is one of those errors:

Configuration Issues

There was an error unconfiguring the vSphere HA agent on this host. To solve this problem, connect the host to a vCenter Server of version 5.0 or later.


Attempting to use the Reconfigure for vSphere HA feature on the hosts fails and disabling then re-enabling HA on the cluster also fails.


To resolve this issue, the first step to try is disable HA on the cluster then follow instructions for reinstalling the HA agent as outlined in the following KB:

Verifying and reinstalling the correct version of the VMware vCenter Server agents (1003714)

Since the environment shown above is ESXi 5.5, the following commands were executed:

~ # cd opt/vmware/uninstallers/

/opt/vmware/uninstallers # ls

/opt/vmware/uninstallers # cp /opt/vmware/uninstallers/

/opt/vmware/uninstallers # chmod +x /tmp/

/opt/vmware/uninstallers # /tmp/

/opt/vmware/uninstallers #


If completing the above continues to throw the same error, proceed and try to use the following commands to manually remove the FDM VIB:

~ # esxcli software vib list | grep -i fdm

vmware-fdm 5.5.0-3252642 VMware VMwareCertified 2016-06-23

~ # esxcli software vib remove -n vmware-fdm

Removal Result

Message: Operation finished successfully.

Reboot Required: false

VIBs Installed:

VIBs Removed: VMware_bootbank_vmware-fdm_5.5.0-3252642

VIBs Skipped:

~ #

~ # esxcli software vib list | grep -i fdm

~ #


Once completed, restart the host and attempt to re-enable HA on the cluster to configure the hosts.

Tuesday, December 6, 2016

Attempting to run Setup or Remove Skype for Business Server Components again throws the error: “HostLocalActivateTask execution failed on an unrecoverable error.”


You’ve recently made a change to your Skype for Business topology and need to run the Setup or Remove Skype for Business Server Components again to apply the changes:


… but notice that it fails with the error:

HostLocalActivateTask execution failed on an unrecoverable error.

The following is the output commands executed:

> Bootstrap-CsComputerLogging status to: C:\Users\tluk\AppData\Local\Temp\BootstrapFull-[2015_09_04][11_33_12].htmlChecking prerequisites for bootstrapper...Checking prerequisite WMIEnabled...prerequisite satisfied.Checking prerequisite NoBootstrapperOnBranchOfficeAppliance...prerequisite satisfied.Checking prerequisite SupportedOS...prerequisite satisfied.Checking prerequisite NoOtherVersionInstalled...prerequisite satisfied.Host name: svrskypestd01.ariel.internalDisabling unused roles...Executing PowerShell command: Disable-CSComputer -Confirm:$false -Verbose -Report "C:\Users\tluk\AppData\Local\Temp\Disable-CSComputer-[2015_09_04][11_33_16].html"Checking prerequisites for roles...Checking prerequisite SupportedOS...prerequisite satisfied.Checking prerequisite SupportedOSNoDC...prerequisite satisfied.Checking prerequisite DotNet35...prerequisite satisfied.Checking prerequisite SupportedSqlRtcLocal...prerequisite satisfied.Checking prerequisite WMIEnabled...prerequisite satisfied.Checking prerequisite NoOtherVersionInstalled...prerequisite satisfied.Checking prerequisite PowerShell...prerequisite satisfied.Checking prerequisite SupportedServerOS...prerequisite satisfied.Checking prerequisite KB2533623Installed...prerequisite satisfied.Checking prerequisite SupportedSqlLyncLocal...prerequisite satisfied.Checking prerequisite SupportedSqlRtc...prerequisite satisfied.Checking prerequisite IIS...prerequisite satisfied.Checking prerequisite IIS7Features...prerequisite satisfied.Checking prerequisite KB2982006Installed...prerequisite satisfied.Checking prerequisite ASPNet...prerequisite satisfied.Checking prerequisite KB2646886Installed...prerequisite satisfied.Checking prerequisite BranchCacheBlock...prerequisite satisfied.Checking prerequisite WCF...prerequisite satisfied.Checking prerequisite WindowsMediaFoundation...prerequisite satisfied.Checking prerequisite SqlUpgradeInstanceRtcLocal...prerequisite satisfied.Checking prerequisite SqlInstanceRtcLocal...prerequisite satisfied.Checking prerequisite VCredist...prerequisite satisfied.Checking prerequisite SqlNativeClient...prerequisite satisfied.Checking prerequisite SqlClrTypes...prerequisite satisfied.Checking prerequisite SqlSharedManagementObjects...prerequisite satisfied.Checking prerequisite UcmaRedist...prerequisite satisfied.Checking prerequisite KB2858668Installed...prerequisite satisfied.Checking prerequisite VcRedistForWinFab...prerequisite satisfied.Checking prerequisite WinFabForWin2012...prerequisite satisfied.Checking prerequisite MicrosoftIdentityExtensions...prerequisite satisfied.Checking prerequisite SqlUpgradeInstanceLyncLocal...prerequisite satisfied.Checking prerequisite SqlInstanceLyncLocal...prerequisite satisfied.Checking prerequisite SqlUpgradeInstanceRtc...prerequisite satisfied.Checking prerequisite SqlInstanceRtc...prerequisite satisfied.Checking prerequisite RewriteModule...prerequisite satisfied.Checking prerequisite TenantPowershell...prerequisite satisfied.Installing any collocated databases...Executing PowerShell command: Install-CSDatabase -Confirm:$false -Verbose -LocalDatabases -Report "C:\Users\tluk\AppData\Local\Temp\Install-CSDatabase-[2015_09_04][11_33_28].html"Enabling new roles...This step will configure services, apply permissions, create firewall rules, etc.Executing PowerShell command: Enable-CSComputer -Confirm:$false -Verbose -Report "C:\Users\tluk\AppData\Local\Temp\Enable-CSComputer-[2015_09_04][11_33_39].html"HostLocalActivateTask execution failed on an unrecoverable error.


The following output is found in the bootstrap log: 

Error: HostLocalActivateTask execution failed on an unrecoverable error.

▼ Details

└ Type: PowerShellException

└ ▼ Stack Trace

at Microsoft.Rtc.Internal.Tools.Bootstrapper.Dependencies.PowerShellProvider.RunCmdlet(String cmdlet)
at Microsoft.Rtc.Management.Internal.Utilities.LogWriter.InvokeAndLog(Action action)

└ ▼ Additional Details

Error: HostLocalActivateTask execution failed on an unrecoverable error.

▼ Details

└ Type: PowerShellException

└ ▼ Stack Trace

at Microsoft.Rtc.Internal.Tools.Bootstrapper.Dependencies.PowerShellProvider.HandlePowerShellErrors(Collection`1 errors)
at Microsoft.Rtc.Internal.Tools.Bootstrapper.Dependencies.PowerShellProvider.RunCmdlet(String cmdlet)


You’ve confirmed that all the services are started:



To determine why this error is thrown during the setup process, execute the cmdlet Enable-CSComputer:

PS C:\Users\tluk> Enable-CSComputer

Enable-CSComputer : HostLocalActivateTask execution failed on an unrecoverable


At line:1 char:1

+ Enable-CSComputer

+ ~~~~~~~~~~~~~~~~~

+ CategoryInfo : InvalidOperation: ([0] Microsoft.R....Core.Servi


:SourceCollection) [Enable-CsComputer], DeploymentException

+ FullyQualifiedErrorId : TaskFailed,Microsoft.Rtc.Management.Deployment.A


WARNING: No patterns found. Skipping rewrite rules creation for Web Scheduler

WARNING: No patterns found. Skipping rewrite rules creation for Web Scheduler

WARNING: Enable-CSComputer encountered errors. Consult the log file for a

detailed analysis, and ensure all errors (6) and warnings (2) are addressed

before continuing.

WARNING: Detailed results can be found at



PS C:\Users\tluk>


Review the log Enable-CSComputer-xxxxx-xxx-xxx.html that the cmdlet generates and look for an error similar to the following:

Error: The service name “<serviceName>” is already in use

An example of this could be:

Error: The service name “RTCATS” is already in use

Once the service is confirmed, execute the cmdlet sc delete RTCATS to remove the service:


Restart the server then proceed to rerun the Setup or Remove Skype for Business Server Components again:


Monday, December 5, 2016

Attempting to upgrade Citrix StoreFront (Xenpp 7.6) to (XenApp 7.11) fails with: “An error occurred during installation. Please ensure all the required prerequisites have been installed an run the installer again.”


You’re attempting to upgrade your XenApp 7.6 environment to 7.11 but noticed that the upgrade for Citrix StoreFront


… to fails with:

StoreFront did not install

An error occurred during the installation.

StoreFront failed.

Citrix StoreFront failed.

An error occurred during installation. Please ensure all the required prerequisites have been installed an run the installer again.


Reviewing the Application log events show the following error logged:

Log Name: Application

Source: Citrix Extensible Meta-Install

Event ID: 0

Level: Error


Timestamp: 12/3/2016 11:41:18 AM

Category:Error, WinError

Message:Installation of '..\CitrixStoreFront-x64.msi' failed with error code 1603. Fatal error during installation


Reviewing the installation log CitrixMsi-CitrixStoreFront-x64-2016-12-03-11-41-06.log file does not provide a reason why the installation failed other than indicating that it did:

Property(S): OutOfNoRbDiskSpace = 0

Property(S): PrimaryVolumeSpaceAvailable = 0

Property(S): PrimaryVolumeSpaceRequired = 0

Property(S): PrimaryVolumeSpaceRemaining = 0

Property(S): INSTALLLEVEL = 1

MSI (s) (98:E8) [11:41:18:642]: Note: 1: 1708

MSI (s) (98:E8) [11:41:18:642]: Product: Citrix StoreFront -- Installation failed.

MSI (s) (98:E8) [11:41:18:643]: Windows Installer installed the product. Product Name: Citrix StoreFront. Product Version: Product Language: 1033. Manufacturer: Citrix Systems, Inc.. Installation success or error status: 1603.

MSI (s) (98:E8) [11:41:18:650]: Deferring clean up of packages/files, if any exist

MSI (s) (98:E8) [11:41:18:650]: MainEngineThread is returning 1603

MSI (s) (98:F0) [11:41:18:653]: RESTART MANAGER: Session closed.

MSI (s) (98:F0) [11:41:18:655]: RESTART MANAGER: Session closed.

MSI (s) (98:F0) [11:41:18:655]: No System Restore sequence number for this installation.

=== Logging stopped: 12/3/2016 11:41:18 ===

MSI (s) (98:F0) [11:41:18:662]: User policy value 'DisableRollback' is 0

MSI (s) (98:F0) [11:41:18:662]: Machine policy value 'DisableRollback' is 0

MSI (s) (98:F0) [11:41:18:662]: Incrementing counter to disable shutdown. Counter after increment: 0

MSI (s) (98:F0) [11:41:18:662]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2

MSI (s) (98:F0) [11:41:18:663]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2

MSI (s) (98:F0) [11:41:18:663]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied. Counter after decrement: -1

MSI (s) (98:F0) [11:41:18:663]: Destroying RemoteAPI object.

MSI (s) (98:10) [11:41:18:663]: Custom Action Manager thread ending.

MSI (c) (A8:D8) [11:41:18:665]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied. Counter after decrement: -1

MSI (c) (A8:D8) [11:41:18:666]: MainEngineThread is returning 1603

=== Verbose logging stopped: 12/3/2016 11:41:18 ===



While there could be various reasons as to why this error would be thrown during the upgrade of StoreFront to 3.7, one of the reasons that could cause this is if you have applied customizations to the UI of the legacy 2.6 StoreFront.  The environment above environment had the following customization applied so that allow the applications were displayed and placed into appropriate folders:

The links to the package in the above blog post no longer work so if you come across an environment with these customizations applied and not sure how it is configured, the following are the files bundled in the package:


The ReadMe.txt file contains the following instructions:

This folder contains customization files for Citrix Receiver for Web 2.5 to provide a folder view for a mandatory store:





If you have not customized your Receiver for Web site, simply copy all the files to the contrib folder of your site.

If you have modified any of custom.script.js, and custom.wrstrings.en.js, you have to manually merge the files in the folder with your modified files and copy the remaining files.

In addition, if you need to support other languages, you have to translate the string "Main" and insert the translated string into custom.wrstrings.<lang-code>.js.





The following files located in C:\inetpub\wwwroot\Citrix\<yourCitrixSite>Web\contrib  are the ones that were replaced:

  • custom.script.js
  • custom.wrstrings.en.js
  • FolderClosed32.png
  • folderview.min.js

The easiest way to get the original UI back is simply navigate to the \contrib folder of a Citrix StoreFront site that wasn’t customized and copy the files back over.  If there is no additional StoreFront site to use then create a new temporary one and copy the files over:


Once the files are over, restart the StoreFront server (an iisreset does not appear to be enough), rerun the upgrade and process should now complete:


Wednesday, November 30, 2016

Attempting to connect to a VMware Horizon View virtual desktop from external throws the error: “The display protocol for this desktop is currently not available. Please contact your system administrator.”


You attempt to connect to a virtual desktop from the internet with the VMware View Horizon Client which passes the traffic to VMware Horizon View Security server which then contacts the View Connection server which then attempts to establish a connection to the VDI but fails with the following message:

The display protocol for this desktop is currently not available. Please contact your system administrator.



While there are various reasons why this error would be thrown, one of them is if your View Connection server has a stale DNS entry to the virtual desktop. A client recently contacted me to troubleshoot this error message and what the problem ended up to be was that the virtual desktops were changed to another port group and while the DNS entries were updated on the domain controllers in the same Active Directory site, the View Connection servers were using DNS servers from a different Active Directory Site so the changes to the DNS entries were not immediately available causing this error to be thrown.

Wednesday, November 23, 2016

Server Manger displays the following message for Remote Desktop Services: "There are no RD Connection Broker servers in the server pool"

I’ve been asked several times this year about a seemingly trivial task that I tend to forget myself so I thought I’d write a quick blog post about it in case someone encounters the same situation and for myself to refer to in the future.


You’ve in one of the following situations:

1. You’ve deployed a new RDS server and added it to an existing collections and would like to perform administrative tasks from that server

2. You’ve asked another administrator to administer an existing RDS server but this administrator uses an account that has never performed these tasks

The problem encountered here in either of the cases above is that when you log onto an RDS server that was recently added to a collection or with an account that has never administered the RDS collections before, you will notice that the options to administer the servers are not available in Server Manager:


Navigating to the Remote Desktop Services section will display the following message:

There are no RD Connection Broker servers in the server pool.

To manage a deployment, you must add all the servers in the deployment to the server pool.

To create a new deployment, run the Add Roles and Features Wizard and select the Remote Desktop Services installation option.



I find that most administrators including myself would eventually figure this out but I tend to forget a few months afterwards. To get the Remote Desktop Services collection to show up in the Server Manager console, click on the Manage button on the top right hand corner and then Add Servers:


You will need to add all of the RDS servers in the collection into this Window but if you only know of one because you’re not familiar with the environment, you can go ahead and add just the one you know of:



Hitting the OK button will bring you back to the Dashboard:


Clicking on the Remote Desktop Services section will display a message telling you that you need to add the rest of the servers in the deployment:

The following servers in this deployment are not part of the server pool:

The servers must be added to the server pool.


Proceed by adding the servers listed:



You should now see the collections in the deployment once the servers have been added:



Thursday, November 17, 2016

Security Alert with certificate error presented when users launch Outlook 2013 or 2016 in an Exchange Server 2016 environment


You have recently migrated from an earlier version of Exchange 2010 or 2013 to Exchange Server 2016 and noticed that users are now receiving a Security Alert with a certificate error when they launch Outlook:

Information you exchange with this site cannot be viewed or changed by others. However, there is a problem with the site’s security certificate.


Clicking on the View Certificate button will show the certificate you’ve assigned to the Exchange server and the reason why it is showing an error is because the certificate does not have a SAN entry for the internal server FQDN as shown in the screenshot.


While some administrators would choose to proceed by adding the internal server FQDN onto the certificate to get around this issue, it is actually not required.  The first and most important reason is that public Certificate Authorities no longer issue certificates with internal domain names such as .local which means if your internal domain used a non-routable suffix then there would be no way to get the FQDN into the certificate as a SAN entry.

One of the reasons why this error would be thrown when users launch their Outlook is that the Service Connection Point (SCP) for Autodiscover has not been updated yet.  Begin by executing the following cmdlet to review the AutoDiscoverServiceInternalUri configuration:

Get-ClientAccessService -identity <serverName> | FL


Note how the AutoDiscoverServiceInternalUri references FQDN of the internal server name which would cause this error to be thrown if your internal certificate did not have the internal server’s FQDN as a SAN entry.  To correct this issue, simply use the following cmdlet to configure it to use the autodiscover URL that you would have added as a SAN entry in the certificate and usually resolves to a record that load balances traffic to the Exchange servers in the organization or directly to an Exchange server is there is only one in the organization:

Set-ClientAccessService -identity <serverName> -AutoDiscoverServiceInternalUri

With the above configuration corrected, users will now be directed to the autodiscover URL that has an entry on the certificate assigned to the Exchange servers.

As stated earlier, note that this error can be thrown by various reasons and the solution stated here may not apply to every instance of this message.