Nesting NSX-T (lab) inside NSX-T (infra)

When we build nested labs, we typically use VLANS to provide an L2 broadcast domain where we can provide networking services across. I also did this for a long time in my home lab, when I needed a new network with a new gateway I typically configured the gateway on a routed (SVI) VLAN subinterface with a new tagged VLAN. With this, I only had the gateway and then I needed to configure the other 4 switches in my home network with this new VLAN. This took too much time and effort and since I am lazy, I did not want to do this anymore, so I installed NSX-v and started to play with NSX-v. Eventually, I upgraded to NSX-T, I used the V2T migration coordinator but this went horribly wrong, so I did a reinstall. Bust this is another story for another time.

All my (nested) lab components need a network. So I decided to use NSX-T Segments for these lab VM's. This all works well, but what if I want to build a nested NSX-T lab? This means that the “GENEVE” lab network that is required to provide NSX-T "overlay" Networking services inside my nested lab is using an NSX-T "underlay" network as the transport layer.

Yes, I know … There is not a simple way to explain this better so I decided to illustrate this in a drawing:

When you look at the picture you can divide the picture into two constructs, INFRA and NESTED.

• INFRA
  • Infra switch (physical)
    • My physical switch where the infra ESXi server is connected to
    • The interfaces of this switch connecting to the infra ESXi server are trunk interfaces and allow all VLANS across the interfaces
  • infra ESXi Host (physical)
    • One of my physical ESX servers that are connected to the physical switch/
    • This physical host also hosts the virtual machines that will behave as nested ESXi hosts
  • infra vmnic interfaces
    • The physical NICs of the ESXi host connected to the physical switch
  • infra N-VDS -> The N-VDS offered by the NSX-T Manager that is part of my lab infrastructure
    • infra NSX-T VLAN segments -> The VLAN Segments offered by my infra NSX-T Manager
      • infra vmk interfaces -> The infra vmkernel interfaces, connected to a VLAN backed segment
    • infra NSX-T GENEVE overlay segments #1 -> The GENEVE (overlay) Segments offered by my infra NSX-T Manager
      • infra Web, App and DB VM’s -> Regular VM’s connected to NSX-T VLAN backed segments
    • infra NSX-T GENEVE overlay segments #2 -> The GENEVE (overlay) Segments offered by my infra NSX-T Manager that are going to be used as UNDERLAY for the Nested parts (explained below)
• NESTED
  • nested ESXi Host (virtual)
    • One of my virtual nested ESX servers that are connected to the infra N-VDS switch
  • nested vmnic interfaces
    • The virtual NICs of the nested ESXi host connected to the infra N-NDS switch on an infra overlay segment
  • nested N-VDS -> The N-VDS offered by the NSX-T Manager that is part of my NESTED lab infrastructure
    • infra NSX-T GENEVE overlay segments -> The GENEVE (overlay) Segments offered by my NESTED NSX-T Manager
      • nested vmk interfaces -> The nested vmkernel interfaces, connected to a (nested) GENEVE overlay backed segment
      • nested Web, App and DB vVM’s -> Double nested vVM’s connected to NSX-T GENEVE overlay backed segment

I know this is mind-blowing and pretty challenging to write about, but I did my best and hope you understand what is going on.