NSX-T/vSphere Blank vSphere ESXi DCUI screen

Submitted by Robin van Altena on Mon, 01/02/2023 - 08:18
 
 
Follow your favourite author

Leave us your email address and be the first to receive a notification when Robin posts a new blog.

NSX-T/vSphere Blank vSphere ESXi DCUI screen
Mon 02 Jan, 2023
Playing around and experimenting with things is always fun in a lab environment. The best way to learn for me is to do and try-out the features myself. Fortunately, we have a great Lab environment and the vExpert program provides the licenses to allow me to try all the good stuff. Although sometimes you run into the strangest issues. This time I ran into some issues with an ESXi host, the distributed switch, and the NSX-T configuration. Somehow the ESXi host had a blank DCUI screen. Which I had seen before….
Textarea

I have seen this screen before, but that was usually when I was messing around with the Lab environment or I’m trying to uninstall NSX-T forcefully. True, that maybe not the best way, but hey it’s a Lab environment. What I did, most of the time, was resetting the network configuration or reset the host to factory defaults and reconfigure the host again. But since I didn’t do anything specific in my Lab. I wanted to do a little more digging.

The reason I was looking at this screen in the first place was because I found some errors in NSX-T that I wanted to investigate. I saw a lot more errors, but I found these the most relevant.

TEP errors in NSX-T
Textarea

The error is also visible in the vCenter GUI, with an alert on the vSwitch and HA status.

Distributed switch state errors
Textarea

And finally, the vSwitch can be seen from the ESXi host view, but there aren’t any VMkernel adapters available on the host. Which is kind of strange for a host that is available in vCenter.

No VMkernel ports available for a ESXi host
Textarea

So, I started to dig a little deeper.

From a putty/SSH session towards the host, I could see the NICs and the distributed switch was available.
esxcli network nic list
esxcli network vswitch dvs vmware list

NICs and vDS as seen from the command line
Textarea

Then the command for listing the VMkernel ports gave me the following output:
esxcli network ip interface list

Node not found for VMkernel ports
Textarea

That was an error I had not seen before. So, I asked Google. Usually when I search for these issues on the internet there isn’t much to be found, but this time it appears to be a bug in vSphere ESXi 7.0.2. See KB 84148
Since I encountered some SD-card issues with version 7.0.2. I haven’t been a fan of that release, but so far it had been running in my nested Lab without any issues. Up till now.
The permanent solution is to upgrade to version 7.0.3 which I will start to do in all my lab environments, but for now I’ll show the workaround mentioned in the KB article.

If you encounter this specific issue, you can restore functionality by placing the host in maintenance mode and running the following command:

/usr/lib/vmware/sqlite/bin/sqlite3 /etc/vmware/configstore/current-store-1 "UPDATE config SET Success=1 WHERE name='vmknics';"

Workaround for KB 84148
Textarea

Yes, there is still an issue with the NSX-T TEP configuration, but at least the other VMkernel adapters are available again in the GUI.

The VMkernel adapters are available again
Textarea

And the DCUI wasn’t blank anymore. So, we're getting somewhere.

The DCUI is available again
Textarea

The error about the missing TEP adapter and the available VTEP adapter showed me the hosts might also be prepared for NSX-V somehow. Not a real surprise since I have been testing a lot of V2T migrations in this lab. But looking at the NSX-V configuration I found no evidence of any NSX-V configuration left on the hosts.

The next step in the work around is to disconnect and reconnect the host while it is in maintenance mode. Before I did that, I decided to remove the vmk3 port from the ESXi host since I could not find the configuration in NSX-V anymore. Again, this is a Lab environment, so I could do this without any risk.

After disconnecting and reconnecting the ESXi hosts there was still an error visible in NSX-T.

The vDS sync issue that remained
Textarea

But after performing the workaround as described in KB 84148, removing the obsolete VMkernel adapters for NSX-V and entering & exiting the host in NSX maintenance mode the issues were resolved.

vSphere vDS without configuration issues
NSX-T Transport nodes without configuration issues
Textarea

And the VMkernel ports were also available again in from the vSphere GUI and the command line.

VMkernel ports seen from the vCenter GUI line
VMkernel ports seen from the command line
Textarea

So, I think my issue was a double problem on the one hand I was suffering from a bug in vSphere ESXi version 7.0.2. on the other hand, I had some leftovers from a previous V2T upgrade or other configuration issues. Before trying to implement this work around know your environment and check the description in the VMware KB to see if it matches your errors.

Anyway, I’m upgrading my lab to 7.0.3. or 8.x as soon as possible.

Hopefully, you have enjoyed reading this blog. If you have any questions or would like to see some more, please leave them at the bottom.

Questions, Remarks & Comments

If you have any questions and need more clarification, we are more than happy to dig deeper. Any comments are also appreciated. You can either post it online or send it directly to the author, it’s your choice.
Let us know  

 
 
Questions, Remarks & Comments

Message Robin directly, in order to receive a quick response.

More about RedLogic