A malicious IP filter

What came before

For a service provider in the Netherlands, we were tasked to migrate the current NSX-V environment to NSX-T 4.1. Added complexity came in the form of VMware Cloud Director and a varied setup of; private and public networks, VPNs, and NAT rules that were connected to Organization vDCs and vApps in a number of different ways. Ways that specifically required NSX-T version 4.1 to keep the tenant networking setup intact. For this NSX-V to NSX-T 4.1 migration we deployed a greenfield NSX-T Manager. The building and testing of the new cluster went well. Test migrations were successful, so we proceeded to the actual migration phase of the project. We started with the first handful of customers and all was well in the world. Packets were flowing, being transported, and translated.

The surprise came the morning after a major tenant migration from the NSX-V cluster to the NSX-T cluster. Note, that nothing strange happened the night of the migration. The tooling functioned flawless, the checklist was all checked, and monitoring was green after the migration tasks were completed. It seemed like a job well done.

Early in the morning a call came in that a certain important Azure resource was not responding to any kind of network traffic. Of the IP block allocated to this Azure tenant, only one IP address was being stubborn and refusing to respond to any of our packets, probes and attempts to connect. The packets did not seem to get any further than the internal network, but the neighboring IP addresses responded timely and error-free.

The search

Initially, we ignored the Distributed Firewall as the source of all this commotion because none of the engineers had implemented an additional firewall rule for this customer. The evening before, we migrated all existing firewall rules from NSX-V to NSX-T. Checked, verified and triple-checked to make sure we didn’t miss any rules and port forwards. This morning the number of rules was still the same and there was no reason for the migrated rules to block any traffic to this specific IP address. We asked the networking team if there was a known issue at this moment.

The networking team reported that the upstream router was able to connect without any issues, so we were pointed to our own virtual infrastructure as the culprit for dropping customer traffic. Lo and behold, the Traceflow tool showed a miraculous firewall rule that appeared seemingly out of nowhere. Surprise!

The reveal

This twin set of firewall rules have the name “Default Malicious IP Block Rules". It uses a list of malicious IP addresses that is automatically downloaded and applied from NTICS (NSX Threat Intelligence Cloud Service) to block traffic to and from IP addresses that have been marked as harmful for various reasons.
Knowing the name of the firewall rule, we quickly found the note in the NSX-T documentation: “If you are the Greenfield customer, this feature is by default enabled for you with the appropriate license. If you are the Brownfield customer, you will have to perform the steps mentioned in the procedure to enable this feature.” (source: Configure Malicious IP Feeds (vmware.com))

The required license for this feature was present and the rule had been automatically enabled during NSX Manager deployment. Looking at the Malicious IPs Filtering and Analysis Dashboard (Malicious IPs Filtering and Analysis Dashboard (vmware.com)), we were able to find the blocked IP address and add it to the exception list with a single click. According to the Dashboard, this IP address had been blocked because of phishing reports.

With a single click, traffic finally arrives back at its destination in Azure.

Looking through the objects in the interface, it seems that it is not possible to see the contents of the Malicious IP set to check if it will block an IP address before enabling the two rules. It would be nice to see a quick and easy to find contact method for the NTICS in case of a false positive result.

At this moment, I have not found a way to get an IP address marked as non-malicious with the NTICS. Do you know a method of contact? Please let us know in the comments below this blog or contact us.

Lessons learned

Sometimes a small note can trip you up. Share seemingly small things with your colleagues and as we say in Dutch: a donkey does not hit the same stone twice.
I certainly won’t forget this little new feature in the Distributed Firewall. It’s worth noting that the Malicious Ips Filtering and Analysis Dashboard shows a lot more dropped packages than I expected. You might find more unwanted traffic when you enable this firewall rule.