AskAresh

In my previous blog post Upgrade from VMware Horizon 7.13.1 to 8.6 (2206) fails with VMware Horizon View Blast Secure Gateway (VMBlastSG) could not be installed. In this post, I will go into the details on the manual uninstall process and installation of the latest version of the VMware Horizon.

Why inplace upgrade fails?

The core reason for the in-place upgrade failing was that the VMware Service around Horizon was not getting deleted during the uninstall performed by the setup, which was rolling back the change.

What is the solution?

The quick solution is to uninstall the VMware Horizon 7 HTML Access, followed by VMware Horizon 7 Connection Server. Perform a reboot on the broker virtual machine, and this step will delete the services from about that were not getting deleted automatically. Install the VMware Horizon 8.x setup, and the installation will go smoothly.

What is the alternate method detailed steps?

The in-place upgrade method described here – Upgrade Connection Servers in a Replicated Group works 99.99% of the time in corner cases like mine. You will have to perform this method. Ensure you follow the basics of a full backup of the brokers, ADAM database, SQL database, backup of the locked.properties file and Disable vCenter provisioning.

Don’t perform this method without seeking proper VMware Support guidance. If you run into issues, you will be in unsupported territory and might ask yourself how you ended up here.

Uninstall existing VMware Horizon

• Login to the broker you are going to perform the upgrade and open the Programs and Features
• First, uninstall the VMware Horizon 7 HTML Access
• Second, uninstall the VMware Horizon 7 Connection Server
• You will be left with the AD LDS Instances (Local and CloudPod partitions). Make sure you leave them as-is.

Reboot

To get rid of the ghosted services perform an reboot of the broker VM.

Install the latest release 8.x

• Login to the broker you are going to perform the upgrade
• Validate whether the above services are deleted
• Right click and run as administrators on the Connection server.exe of the 8.x setup
• Make sure you select – Standard Server and click on HTML access and IPv4

• It will detect the instance of the Horizon and you need to click on OK

• Select Configure the Firewall within Windows
• The Horizon 8.x will be installed successfully on the virtual machine
• Repeat this steps on the other brokers within your POD

Validations

You must wait approximately 5-7 mins for the Horizon Administrator console to come online. Validate the Health dashboard for any errors and check the desired 8.x version is present.

I hope you will find this information useful if you encounter the issue and it should help you save time. If you manage to tweak or improvise further on this solution, please don’t forget to keep me posted.

Thanks,
Aresh Sarkari

Tags: Desktop Virtualization, Horizon, Upgrade, VDI, VMware, VMware EUC, VMware Horizon

Many customers are already in the process of upgrading from VMware Horizon 7.x to 8.x or will soon upgrade as the End Of Life dates are upcoming in April 2023. I want to share a rare experience wherein the Horizon upgrade from 7.13.1 to 8.6 version failed. In the rare occasion where in upgrade fails in the below mentioned manner, the workaround steps will come in handy.

Issue

During the upgrade of the first connection server in the POD1, we encountered the following error five mins into the upgrade. Note before starting the upgrade, the entire health dashboard for the POD was green and included backup and snapshots.

Environment Overview

Let take a look at the environment details to provide an high-level overview:
Active Site (POD1)

• 5 VMware Horizon Connection Servers 7.13.1
• SQL Database on Microsoft SQL 2016 Always-on – EventsDB
• The 5 Brokers are behind an NSX Load balancer

Active Site (POD2)

• 5 VMware Horizon Connection Servers 7.13.1
• SQL Database on Microsoft SQL 2016 Always-on – EventsDB
• The 5 Brokers are behind an NSX Load balancer

Observations

The logs from the installer had the following message it’s complaining about insufficient privileges, which the installer is run with admin privileges already.

MSI (s) (A0:E0) [06:12:07:977]: Executing op: ActionStart(Name=InstallServices,Description=Installing new services,Template=Service: [2])
Action 6:12:07: InstallServices. Installing new services
MSI (s) (A0:E0) [06:12:07:977]: Executing op: ProgressTotal(Total=9,Type=1,ByteEquivalent=1300000)
MSI (s) (A0:E0) [06:12:07:977]: Executing op: ServiceInstall(Name=PCOIPSG,DisplayName=VMware Horizon View PCoIP Secure Gateway,ImagePath="C:\Program Files\VMware\VMware View\Server\bin\SecurityGateway.exe",ServiceType=16,StartType=3,ErrorControl=1,,Dependencies=[~],,,Password=**********,Description=Provides VMware Horizon View PCoIP gateway services.,,)
InstallServices: Service:
MSI (s) (A0:E0) [06:12:08:228]: Executing op: ServiceInstall(Name=VMBlastSG,DisplayName=VMware Horizon View Blast Secure Gateway,ImagePath="C:\Program Files\VMware\VMware View\Server\appblastgateway\nssm.exe",ServiceType=16,StartType=3,ErrorControl=0,,Dependencies=[~],,,Password=**********,Description=Provides VMware Horizon View Blast gateway services.,,)
InstallServices: Service:
Info 1923.Service VMware Horizon View Blast Secure Gateway (VMBlastSG) could not be installed. Verify that you have sufficient privileges to install system services.
Action ended 6:12:08: InstallFinalize. Return value 3.

Also noticed during the upgrade, it should uninstall the services from version 7.13.1, and new services would be created for version 8.6. In this case, they were listed as disabled.

Workaround

Later during the ongoing RCA investigation, a workaround provided was working well. The reason for socializing the workaround is that we spent a tremendous amount of time and effort in a revert operation, which was very time consuming and cumbersome. Only if we had known about this workaround during the failed upgrade would we have saved tremendous effort.

Prerequisite / Rollback Plan 

1. Power off all the Connection Servers part of Cloud POD Federation 
2. Take a powered-off snapshot
3. In case of an incident during the change activity which requires recovering Horizon Environment, Snapshots will be used as a fallback plan as the last option if break/fix, troubleshooting steps performed is not resolving the issue. 
4. In troubleshooting scenarios of a failed upgrade 

Workaround Steps 

1. The following steps need to be performed on each Connection Server at a time 
2. Uninstall HTML and Horizon Connection Server 7.13.1 component keeping ADLDS and ADLDSG Instances intact. Check services.msc, and Horizon Services will appear in the Disabled state 
3. Perform a reboot of the Connection Server, and Horizon Services will be cleared from services.msc
4. Install Horizon 8.6 as Standard, which will pick the residing ADLDS and ADLDSG instance
5. After successful install, it can be verified using ldp utility as instructed in KB https://kb.vmware.com/s/article/2064157 and look for the fields whenChanged , and whenCreated . This step can be performed prior upgrade for comparing the state of ADLDS

The VMware GSS case handled by Jezill Asharaf (A very helpful support engineer) and a few of the backend engineering team has been instrumental. I hope you will find this information useful if you encounter the issue and it should help you save time. If you manage to tweak or improvise further on this solution, please don’t forget to keep me posted.

Thanks,
Aresh Sarkari

Tags: Desktop Virtualization, Horizon, Upgrade, VDI, VMware, VMware EUC, VMware Horizon