Monday, July 06, 2020

vCenter Server Appliance - Update installation is in progress

A few days ago, I have updated my home-lab VCSA vCenter Server 7.0 GA (15952498) to vCenter Server 7.0.0a (16189094). Everything seemed ok from the vCenter (vSphere Client) perspective. I was seeing there vCenter build 16189207, which is obviously VCSA 16189094.

The only problem I had was the fact, that I was not able to log in to VCSA VAMI.

After user authentication into VAMI, I was getting a message ... "Update installation is in progress". During the problem isolation, I tested the REST API and was getting the same message through Appliance REST API.

To be honest, I had no idea, how to check VCSA update status and move it forward, therefore I did the log files troubleshooting.

In /var/log/vmware/applmgmt/vami.log I had regularly repeating messages... "Executing VMware vCenter Server SNMP post upgrade actions..."

The next step was parsing PatchRunner.log. There was nothing remarkable except info about the patch stageDir located at "/storage/core/software-update/updates/" where i had two empty directories and file "patch_phase_context.json"

Normally, the directory /storage/core/software-update/updates/ is empty, however I have there still subdirectory with patch … /storage/core/software-update/updates/

I tried to delete the whole directory /storage/core/software-update/updates/ but it did not help.

It seemed, that post-upgrade actions (specifically VMware vCenter Server SNMP) cannot finish.

I HAD NO IDEA ... I WAS STUCK ... the next step was the research on the internal VMware slack channel. I have not found anything, therefore asked for help. In a few days, somebody else had the same issue in his home lab and he got the tip from VMware Engineering guys.

The trick was in file /etc/applmgmt/appliance/software_update_state.conf

I had there the following content

    "state": "INSTALL_IN_PROGRESS",
    "version": "",
    "latest_query_time": "2020-06-12T17:30:46Z",
    "operation_id": "/storage/core/software-update/install_operation"

so the trick was to change the content into

    "state": "UP_TO_DATE"

After the change, I was able to log in to VAMI and use it as usual. Well, almost as usual.
I realized, that I was not able to start another update as buttons [STAGE ONLY] and [STAGE AND INSTALL] are greyed out.

Nevertheless, this is only GUI problem and VCSA CLI update procedure works like a charm, so the update can be done through SSH session and command software-packages install --url or software-packages install --iso

Another positive thing is, that I was able to perform vCenter native backup, so if needed, I have the option to do a VCSA backup and restore, which would probably solve this cosmetic GUI issue.

Disclaimer: This exercise was done in my home lab. If you will experience a similar issue in a production system, please, contact VMware support before any changes in VCSA.

No comments: