NSX-T/vSphere Blank vSphere ESXi DCUI screen

I have seen this screen before, but that was usually when I was messing around with the Lab environment or I’m trying to uninstall NSX-T forcefully. True, that maybe not the best way, but hey it’s a Lab environment. What I did, most of the time, was resetting the network configuration or reset the host to factory defaults and reconfigure the host again. But since I didn’t do anything specific in my Lab. I wanted to do a little more digging.

The reason I was looking at this screen in the first place was because I found some errors in NSX-T that I wanted to investigate. I saw a lot more errors, but I found these the most relevant.

The error is also visible in the vCenter GUI, with an alert on the vSwitch and HA status.

And finally, the vSwitch can be seen from the ESXi host view, but there aren’t any VMkernel adapters available on the host. Which is kind of strange for a host that is available in vCenter.

So, I started to dig a little deeper.

From a putty/SSH session towards the host, I could see the NICs and the distributed switch was available.
esxcli network nic list
esxcli network vswitch dvs vmware list

Then the command for listing the VMkernel ports gave me the following output:
esxcli network ip interface list

That was an error I had not seen before. So, I asked Google. Usually when I search for these issues on the internet there isn’t much to be found, but this time it appears to be a bug in vSphere ESXi 7.0.2. See KB 84148
Since I encountered some SD-card issues with version 7.0.2. I haven’t been a fan of that release, but so far it had been running in my nested Lab without any issues. Up till now.
The permanent solution is to upgrade to version 7.0.3 which I will start to do in all my lab environments, but for now I’ll show the workaround mentioned in the KB article.

If you encounter this specific issue, you can restore functionality by placing the host in maintenance mode and running the following command:

/usr/lib/vmware/sqlite/bin/sqlite3 /etc/vmware/configstore/current-store-1 "UPDATE config SET Success=1 WHERE name='vmknics';"

Yes, there is still an issue with the NSX-T TEP configuration, but at least the other VMkernel adapters are available again in the GUI.

And the DCUI wasn’t blank anymore. So, we're getting somewhere.

The error about the missing TEP adapter and the available VTEP adapter showed me the hosts might also be prepared for NSX-V somehow. Not a real surprise since I have been testing a lot of V2T migrations in this lab. But looking at the NSX-V configuration I found no evidence of any NSX-V configuration left on the hosts.

The next step in the work around is to disconnect and reconnect the host while it is in maintenance mode. Before I did that, I decided to remove the vmk3 port from the ESXi host since I could not find the configuration in NSX-V anymore. Again, this is a Lab environment, so I could do this without any risk.

After disconnecting and reconnecting the ESXi hosts there was still an error visible in NSX-T.

But after performing the workaround as described in KB 84148, removing the obsolete VMkernel adapters for NSX-V and entering & exiting the host in NSX maintenance mode the issues were resolved.

And the VMkernel ports were also available again in from the vSphere GUI and the command line.

So, I think my issue was a double problem on the one hand I was suffering from a bug in vSphere ESXi version 7.0.2. on the other hand, I had some leftovers from a previous V2T upgrade or other configuration issues. Before trying to implement this work around know your environment and check the description in the VMware KB to see if it matches your errors.

Anyway, I’m upgrading my lab to 7.0.3. or 8.x as soon as possible.

Hopefully, you have enjoyed reading this blog. If you have any questions or would like to see some more, please leave them at the bottom.