Routing with NSX using multiple sites using Locale ID (with eBGP all the way)

From Iwan
Jump to: navigation, search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Routing with NSX using multiple sites using Locale ID (with eBGP all the way)

Introduction

In one of my previous articles I demonstrated the routing between two Data Centers with using eBGP and iBGP. These previous articles focussed on using only one path northbound as the primary exit point. So workloads that are in DC1 and DC2 will either choose the DC1 WAN or the DC2 WAN as the primary path (where DC1 is always prefered and DC2 is always secondary)

This article will be about doing almost the same, but now with the use of the NSX Locale ID feature. With this feature the workloads that are in DC1 will exit trough the DC1 WAN and the workloads that are in DC2 will exit trough the DC2 WAN.

Diagram

The following diagram will be used for our setup.

>> LAB100_-_NSX_Routing_based_on_eBGP_and_locale_ID.pdf

Nsx-routing-blog-locale-1.png

Components

I am still using the same componenents as in this previous article but in this article we will actively use the UDLR-02 and leave the UDLR-1 alone.

Use–Cases

I have thought of the following use-cases below.

As it is not possible to test active/passive and active/active with one UDLR I am using two UDLR instances here.

For testing purpose, I will use three tenants with each having their own Web, App and DB tiers.

Tenant one and two will be using the first UDLR (UDLR-01) with active/passive site egress. Tenant three will be using the second UDLR (UDLR-02) with active/active site egress.

Tenant three

Tenant three will have workloads in DC1 and in DC2 and because it is using local egress the traffic will exit from the Cisco CSR1000V routers local to that site. When the primary site is down the traffic should exit from the secondary site and when the secondary is down traffic should exit from the primary.

Routing protocol options

All possible routing options are described in the previous article.

The routing option in this article will be:

Option 1

  1. eBGP peering between the UDLR and the ESG’s
  2. eBGP peering between the ESG’s and the external routers
  3. eBGP peering between the external routers and the CORE

Option 1

Because of the many options available in the setup I want to start with the following: Tenant 3 workloads with the use of UDLR-02 and option 1 routing.

UDLR configuration

When Locale ID is used the UDLR Control VM needs to be deployed in both DC's. After the deployment is done on the primary site we need to browse to the secondary NSX Manager and manually deploy an instance of the UDLR control VM in DC2. When this is done some DC2 site specific configurations are needed.

  1. DC1 and DC2 UDLR-02 instances will have their own interfaces with own IP addresses
  2. DC1 and DC2 UDLR-02 instances will have their own eBGP peerings
  3. DC1 and DC2 UDLR-02 instances will have their own site (or NSX Manager) specific Locale ID

DC1

Manage --> Firewall

  1. Make sure the firewall is disabled

Nsx-routing-blog-locale-2.png

  1. Get the Locale ID from the NSX Manager

Nsx-routing-blog-locale-3.png

Manage --> Routing --> Global Configuration

  1. Configure Locale ID and make sure this is the same as the NSX Manager
  2. Enable ECMP
  3. Make sure that the default gateway is not configured
  4. Configure a router ID

Nsx-routing-blog-locale-4.png

Manage --> Routing --> BGP

  1. Enable BGP
  2. Configure the Local AS
  3. Disable Graceful Restart
  4. Configure the BGP peers towards ESG-A

Nsx-routing-blog-locale-5.png

Manage --> Routing --> Route Redistribution

  1. Enable route redistribution for BGP
  2. Specify what you allow to redistribute

Nsx-routing-blog-locale-6.png

DC2

  1. Make sure the firewall is disabled

Nsx-routing-blog-locale-7.png

  1. Get the Locale ID from the NSX Manager

Nsx-routing-blog-locale-8.png

Manage --> Routing --> Global Configuration

  1. Configure Locale ID and make sure this is the same as the NSX Manager
  2. Enable ECMP
  3. Make sure that the default gateway is not configured
  4. Configure a router ID

Nsx-routing-blog-locale-9.png

Manage --> Routing --> BGP

  1. Enable BGP
  2. Configure the Local AS
  3. Disable Graceful Restart
  4. Configure the BGP peers towards ESG-A

Nsx-routing-blog-locale-10.png

Manage --> Routing --> Route Redistribution

  1. Enable route redistribution for BGP
  2. Specify what you allow to redistribute

Nsx-routing-blog-locale-11.png

Compute Cluster configuration

The same locale ID (per site) also need to be configured on the Compute Clusters. We need to verify this if this happened automatically and if this is not the case we need to configure this manually.

DC1

Nsx-routing-blog-locale-12.png

DC2

Nsx-routing-blog-locale-13.png

ESG configuration

DC1

Manage --> Firewall

  1. Make sure the firewall is disabled

Nsx-routing-blog-locale-14.png

Manage --> Routing --> Global Configuration

  1. Enable ECMP
  2. Make sure that the default gateway is not configured
  3. Configure a router ID

Nsx-routing-blog-locale-15.png

Manage --> Routing --> BGP

  1. Enable BGP
  2. Configure the Local AS
  3. Disable Graceful Restart
  4. Configure the BGP peers towards the external routers and the UDLR.

Nsx-routing-blog-locale-16.png

Manage --> Routing --> Route Redistribution

  1. Enable route redistribution for BGP
  2. Specify what you allow to redistribute

Nsx-routing-blog-locale-17.png

DC2

Manage --> Firewall

  1. Make sure the firewall is disabled

Nsx-routing-blog-locale-18.png

Manage --> Routing --> Global Configuration

  1. Enable ECMP
  2. Make sure that the default gateway is not configured
  3. Configure a router ID

Nsx-routing-blog-locale-19.png

Manage --> Routing --> BGP

  1. Enable BGP
  2. Configure the Local AS
  3. Disable Graceful Restart
  4. Configure the BGP peers towards the external routers and the UDLR

Nsx-routing-blog-locale-20.png

Manage --> Routing --> Route Redistribution

  1. Enable route redistribution for BGP
  2. Specify what you allow to redistribute

Nsx-routing-blog-locale-21.png

Cisco1000V configuration

DC1

!
root ##bl##hostname rt-a-01
!
router bgp 65511
 bgp router-id 10.11.11.31
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.100.19.2 remote-as 65521
 neighbor 10.100.19.2 description ESG-A
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.100.19.2 activate
 exit-address-family
!
ip route 10.200.19.0 255.255.255.0 10.11.11.253
ip route 10.200.21.0 255.255.255.0 10.11.11.253
!
!
root ##bl##hostname rt-a-02
!
router bgp 65511
 bgp router-id 10.11.11.32
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.100.21.2 remote-as 65521
 neighbor 10.100.21.2 description ESG-A
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.100.21.2 activate
 exit-address-family
!
ip route 10.200.19.0 255.255.255.0 10.11.11.253
ip route 10.200.21.0 255.255.255.0 10.11.11.253
!

DC2

!
root ##bl##hostname rt-b-01
!
router bgp 65512
 bgp router-id 10.11.11.33
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.200.19.2 remote-as 65522
 neighbor 10.200.19.2 description ESG-B
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.200.19.2 activate
 exit-address-family
!
ip route 10.100.19.0 255.255.255.0 10.11.11.253
ip route 10.100.21.0 255.255.255.0 10.11.11.253
!
!
root ##bl##hostname rt-b-02
!
router bgp 65512
 bgp router-id 10.11.11.34
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.200.21.2 remote-as 65522
 neighbor 10.200.21.2 description ESG-B
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.200.21.2 activate
 exit-address-family
!
ip route 10.100.19.0 255.255.255.0 10.11.11.253
ip route 10.100.21.0 255.255.255.0 10.11.11.253
!

Core configuration

!
root ##bl##hostname cs-01
!
router bgp 65510
 bgp router-id 10.11.11.253
 bgp log-neighbor-changes
 neighbor 10.11.11.31 remote-as 65511
 neighbor 10.11.11.31 description RT-A-01
 neighbor 10.11.11.32 remote-as 65511
 neighbor 10.11.11.32 description RT-A-02
 neighbor 10.11.11.33 remote-as 65512
 neighbor 10.11.11.33 description RT-B-01
 neighbor 10.11.11.34 remote-as 65512
 neighbor 10.11.11.34 description RT-B-02
 !
 address-family ipv4
  network 10.11.11.0 mask 255.255.255.0
  neighbor 10.11.11.31 activate
  neighbor 10.11.11.31 next-hop-self
  neighbor 10.11.11.32 activate
  neighbor 10.11.11.32 next-hop-self
  neighbor 10.11.11.33 activate
  neighbor 10.11.11.33 next-hop-self
  neighbor 10.11.11.34 activate
  neighbor 10.11.11.34 next-hop-self
  no auto-summary
 exit-address-family
!

UDLR route peering verification

DC1

Nsx-routing-blog-locale-22.png

DC2

root ##bl##NSX-edge-71b9e24f-9a47-423d-b8e7-392dc8b3e47a-0> show ip bgp neighbors

BGP neighbor is 172.39.39.33,   remote AS 65522,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 3194 messages, Sent 4728 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x3740211c
         Route refresh request:received 0 sent 0
         Prefixes received 13 sent 5 advertised 5
Connections established 1, dropped 2468
Local host: 172.39.39.45, Local port: 179
Remote host: 172.39.39.33, Remote port: 53355

NSX-edge-71b9e24f-9a47-423d-b8e7-392dc8b3e47a-0>

ESG route peering verification

DC1

root ##bl##REGA-ESG01-0> show ip bgp neighbors

BGP neighbor is 10.100.19.1,   remote AS 65511,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2203 messages, Sent 2301 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0xa87fda9c
         Route refresh request:received 0 sent 0
         Prefixes received 3 sent 14 advertised 14
Connections established 1, dropped 1
Local host: 10.100.19.2, Local port: 12895
Remote host: 10.100.19.1, Remote port: 179


BGP neighbor is 10.100.21.1,   remote AS 65511,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2201 messages, Sent 2300 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0xa87fda9c
         Route refresh request:received 0 sent 0
         Prefixes received 3 sent 14 advertised 14
Connections established 1, dropped 1
Local host: 10.100.21.2, Local port: 31006
Remote host: 10.100.21.1, Remote port: 179


BGP neighbor is 172.39.39.13,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2340 messages, Sent 2323 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 3 Identifier 0xa87fda9c
         Route refresh request:received 0 sent 0
         Prefixes received 7 sent 7 advertised 7
Connections established 2, dropped 42
Local host: 172.39.39.1, Local port: 13445
Remote host: 172.39.39.13, Remote port: 179


BGP neighbor is 172.39.39.29,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2286 messages, Sent 2282 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 4 Identifier 0xa87fda9c
         Route refresh request:received 0 sent 0
         Prefixes received 5 sent 7 advertised 7
Connections established 1, dropped 1
Local host: 172.39.39.17, Local port: 37920
Remote host: 172.39.39.29, Remote port: 179

REGA-ESG01-0>

DC2

root ##bl##REGB-ESG01-0> show ip bgp neighbors

BGP neighbor is 10.200.19.1,   remote AS 65512,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2192 messages, Sent 2287 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x637a1aec
         Route refresh request:received 0 sent 0
         Prefixes received 9 sent 8 advertised 8
Connections established 1, dropped 1
Local host: 10.200.19.2, Local port: 57360
Remote host: 10.200.19.1, Remote port: 179


BGP neighbor is 10.200.21.1,   remote AS 65512,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2191 messages, Sent 2280 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0x637a1aec
         Route refresh request:received 0 sent 0
         Prefixes received 9 sent 8 advertised 8
Connections established 1, dropped 1
Local host: 10.200.21.2, Local port: 17297
Remote host: 10.200.21.1, Remote port: 179


BGP neighbor is 172.39.39.13,   remote AS 65530,
BGP state = Idle, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: None
         Address family none
         Graceful restart Capability:none
                 Restart remain time: 0
Received 15751 messages, Sent 11421 messages
Default minimum time between advertisement runs is 30 seconds
For Address family none
         Index 3 Identifier 0x637a1aec
         Route refresh request:received 0 sent 0
         Prefixes received 0 sent 0 advertised 0
Connections established 0, dropped 15751
Local host: 10.200.21.2, Local port: 0
Remote host: 172.39.39.13, Remote port: 0


BGP neighbor is 172.39.39.45,   remote AS 65530,
BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2278 messages, Sent 2279 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 4 Identifier 0x637a1aec
         Route refresh request:received 0 sent 0
         Prefixes received 5 sent 13 advertised 13
Connections established 1, dropped 1
Local host: 172.39.39.33, Local port: 53355
Remote host: 172.39.39.45, Remote port: 179

REGB-ESG01-0>

Cisco1000V route peering verification

DC1

root ##bl##rt-a-01#show ip bgp summary
BGP router identifier 10.11.11.31, local AS number 65511
BGP table version is 54, main routing table version 54
17 network entries using 4216 bytes of memory
21 path entries using 2520 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7544 total bytes of memory
BGP activity 30/13 prefixes, 47/26 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510    2218    2206       54    0    0 1d09h           7
10.100.19.2     4        65521    2304    2205       54    0    0 1d09h          14
rt-a-01#
root ##bl##rt-a-02#show ip bgp summary
BGP router identifier 10.11.11.32, local AS number 65511
BGP table version is 54, main routing table version 54
17 network entries using 4216 bytes of memory
21 path entries using 2520 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7544 total bytes of memory
BGP activity 35/18 prefixes, 47/26 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510    2209    2210       54    0    0 1d09h           7
10.100.21.2     4        65521    2303    2204       54    0    0 1d09h          14
rt-a-02#

DC2

root ##bl##rt-b-01#show ip bgp summary
BGP router identifier 10.11.11.33, local AS number 65512
BGP table version is 76, main routing table version 76
17 network entries using 4216 bytes of memory
19 path entries using 2280 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7304 total bytes of memory
BGP activity 30/13 prefixes, 50/31 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510    5385    5387       76    0    0 3d09h          11
10.200.19.2     4        65522    2290    2194       76    0    0 1d09h           8
rt-b-01#
root ##bl##rt-b-02#show ip bgp summary
BGP router identifier 10.11.11.34, local AS number 65512
BGP table version is 49, main routing table version 49
17 network entries using 4216 bytes of memory
19 path entries using 2280 bytes of memory
3/3 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 7304 total bytes of memory
BGP activity 17/0 prefixes, 35/16 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.253    4        65510    2210    2204       49    0    0 1d09h          11
10.200.21.2     4        65522    2284    2195       49    0    0 1d09h           8
rt-b-02#

Core route peering verification

root ##bl##cs-01#show ip bgp summary 
BGP router identifier 10.11.11.253, local AS number 65510
BGP table version is 83, main routing table version 83
17 network entries using 2312 bytes of memory
45 path entries using 2340 bytes of memory
3/3 BGP path/bestpath attribute entries using 372 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5072 total bytes of memory
BGP activity 30/13 prefixes, 109/64 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.11.11.31     4        65511    2209    2221       83    0    0 1d09h          14
10.11.11.32     4        65511    2212    2212       83    0    0 1d09h          14
10.11.11.33     4        65512    5389    5387       83    0    0 3d09h           8
10.11.11.34     4        65512    2205    2210       83    0    0 1d09h           8
cs-01#

UDLR routing tables

DC1

Nsx-routing-blog-locale-23.png

DC2

root ##bl##NSX-edge-71b9e24f-9a47-423d-b8e7-392dc8b3e47a-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 17

B       10.11.11.0/24        [20/0]        via 172.39.39.33
B       10.100.19.0/24       [20/0]        via 172.39.39.33
B       10.100.21.0/24       [20/0]        via 172.39.39.33
B       10.200.19.0/24       [20/0]        via 172.39.39.33
B       10.200.21.0/24       [20/0]        via 172.39.39.33
B       172.20.1.0/24        [20/0]        via 172.39.39.33
B       172.20.2.0/24        [20/0]        via 172.39.39.33
B       172.20.3.0/24        [20/0]        via 172.39.39.33
B       172.20.8.0/24        [20/0]        via 172.39.39.33
B       172.20.9.0/24        [20/0]        via 172.39.39.33
B       172.20.10.0/24       [20/0]        via 172.39.39.33
C       172.30.1.0/24        [0/0]         via 172.30.1.254
C       172.30.2.0/24        [0/0]         via 172.30.2.254
C       172.30.3.0/24        [0/0]         via 172.30.3.254
B       172.39.39.0/28       [20/0]        via 172.39.39.33
C       172.39.39.16/28      [0/0]         via 172.39.39.30
C       172.39.39.32/28      [0/0]         via 172.39.39.45
NSX-edge-71b9e24f-9a47-423d-b8e7-392dc8b3e47a-0>

ESG routing tables

DC1

root ##bl##REGA-ESG01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 17

B       10.11.11.0/24        [20/0]        via 10.100.19.1
B       10.11.11.0/24        [20/0]        via 10.100.21.1
C       10.100.19.0/24       [0/0]         via 10.100.19.2
C       10.100.21.0/24       [0/0]         via 10.100.21.2
B       10.200.19.0/24       [20/0]        via 10.100.19.1
B       10.200.19.0/24       [20/0]        via 10.100.21.1
B       10.200.21.0/24       [20/0]        via 10.100.19.1
B       10.200.21.0/24       [20/0]        via 10.100.21.1
B       172.20.1.0/24        [20/0]        via 172.39.39.14
B       172.20.2.0/24        [20/0]        via 172.39.39.14
B       172.20.3.0/24        [20/0]        via 172.39.39.14
B       172.20.8.0/24        [20/0]        via 172.39.39.14
B       172.20.9.0/24        [20/0]        via 172.39.39.14
B       172.20.10.0/24       [20/0]        via 172.39.39.14
B       172.30.1.0/24        [20/0]        via 172.39.39.30
B       172.30.2.0/24        [20/0]        via 172.39.39.30
B       172.30.3.0/24        [20/0]        via 172.39.39.30
C       172.39.39.0/28       [0/0]         via 172.39.39.1
C       172.39.39.16/28      [0/0]         via 172.39.39.17
B       172.39.39.32/28      [20/0]        via 172.39.39.30
REGA-ESG01-0>

DC2

root ##bl##REGB-ESG01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 17

B       10.11.11.0/24        [20/0]        via 10.200.19.1
B       10.11.11.0/24        [20/0]        via 10.200.21.1
B       10.100.19.0/24       [20/0]        via 10.200.19.1
B       10.100.19.0/24       [20/0]        via 10.200.21.1
B       10.100.21.0/24       [20/0]        via 10.200.19.1
B       10.100.21.0/24       [20/0]        via 10.200.21.1
C       10.200.19.0/24       [0/0]         via 10.200.19.2
C       10.200.21.0/24       [0/0]         via 10.200.21.2
B       172.20.1.0/24        [20/0]        via 10.200.19.1
B       172.20.1.0/24        [20/0]        via 10.200.21.1
B       172.20.2.0/24        [20/0]        via 10.200.19.1
B       172.20.2.0/24        [20/0]        via 10.200.21.1
B       172.20.3.0/24        [20/0]        via 10.200.19.1
B       172.20.3.0/24        [20/0]        via 10.200.21.1
B       172.20.8.0/24        [20/0]        via 10.200.19.1
B       172.20.8.0/24        [20/0]        via 10.200.21.1
B       172.20.9.0/24        [20/0]        via 10.200.19.1
B       172.20.9.0/24        [20/0]        via 10.200.21.1
B       172.20.10.0/24       [20/0]        via 10.200.19.1
B       172.20.10.0/24       [20/0]        via 10.200.21.1
B       172.30.1.0/24        [20/0]        via 172.39.39.46
B       172.30.2.0/24        [20/0]        via 172.39.39.46
B       172.30.3.0/24        [20/0]        via 172.39.39.46
C       172.39.39.0/28       [0/0]         via 172.39.39.2
B       172.39.39.16/28      [20/0]        via 172.39.39.46
C       172.39.39.32/28      [0/0]         via 172.39.39.33
REGB-ESG01-0>

Cisco1000V routing tables

DC1

root ##bl##rt-a-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.31/32 is directly connected, GigabitEthernet2
C        10.100.1.0/24 is directly connected, GigabitEthernet1
L        10.100.1.101/32 is directly connected, GigabitEthernet1
C        10.100.19.0/24 is directly connected, GigabitEthernet3
L        10.100.19.1/32 is directly connected, GigabitEthernet3
B        10.100.21.0/24 [20/0] via 10.100.19.2, 1d09h
S        10.200.19.0/24 [1/0] via 10.11.11.253
S        10.200.21.0/24 [1/0] via 10.11.11.253
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.100.19.2, 1d09h
B        172.20.2.0 [20/0] via 10.100.19.2, 1d09h
B        172.20.3.0 [20/0] via 10.100.19.2, 1d09h
B        172.20.8.0 [20/0] via 10.100.19.2, 1d09h
B        172.20.9.0 [20/0] via 10.100.19.2, 1d09h
B        172.20.10.0 [20/0] via 10.100.19.2, 1d09h
      172.30.0.0/24 is subnetted, 3 subnets
B        172.30.1.0 [20/0] via 10.100.19.2, 1d09h
B        172.30.2.0 [20/0] via 10.100.19.2, 1d09h
B        172.30.3.0 [20/0] via 10.100.19.2, 1d09h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.100.19.2, 1d09h
B        172.39.39.16 [20/0] via 10.100.19.2, 1d09h
B        172.39.39.32 [20/0] via 10.100.19.2, 1d09h
rt-a-01#
root ##bl##rt-a-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.32/32 is directly connected, GigabitEthernet2
C        10.100.1.0/24 is directly connected, GigabitEthernet1
L        10.100.1.102/32 is directly connected, GigabitEthernet1
B        10.100.19.0/24 [20/0] via 10.100.21.2, 1d09h
C        10.100.21.0/24 is directly connected, GigabitEthernet3
L        10.100.21.1/32 is directly connected, GigabitEthernet3
S        10.200.19.0/24 [1/0] via 10.11.11.253
S        10.200.21.0/24 [1/0] via 10.11.11.253
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.100.21.2, 1d09h
B        172.20.2.0 [20/0] via 10.100.21.2, 1d09h
B        172.20.3.0 [20/0] via 10.100.21.2, 1d09h
B        172.20.8.0 [20/0] via 10.100.21.2, 1d09h
B        172.20.9.0 [20/0] via 10.100.21.2, 1d09h
B        172.20.10.0 [20/0] via 10.100.21.2, 1d09h
      172.30.0.0/24 is subnetted, 3 subnets
B        172.30.1.0 [20/0] via 10.100.21.2, 1d09h
B        172.30.2.0 [20/0] via 10.100.21.2, 1d09h
B        172.30.3.0 [20/0] via 10.100.21.2, 1d09h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.100.21.2, 1d09h
B        172.39.39.16 [20/0] via 10.100.21.2, 1d09h
B        172.39.39.32 [20/0] via 10.100.21.2, 1d09h
rt-a-02#

DC2

root ##bl##rt-b-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.33/32 is directly connected, GigabitEthernet2
S        10.100.19.0/24 [1/0] via 10.11.11.253
S        10.100.21.0/24 [1/0] via 10.11.11.253
C        10.200.1.0/24 is directly connected, GigabitEthernet1
L        10.200.1.101/32 is directly connected, GigabitEthernet1
C        10.200.19.0/24 is directly connected, GigabitEthernet3
L        10.200.19.1/32 is directly connected, GigabitEthernet3
B        10.200.21.0/24 [20/0] via 10.200.19.2, 1d09h
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.2.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.3.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.8.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.9.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.10.0 [20/0] via 10.11.11.253, 1d09h
      172.30.0.0/24 is subnetted, 3 subnets
B        172.30.1.0 [20/0] via 10.200.19.2, 1d09h
B        172.30.2.0 [20/0] via 10.200.19.2, 1d09h
B        172.30.3.0 [20/0] via 10.200.19.2, 1d09h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.200.19.2, 1d09h
B        172.39.39.16 [20/0] via 10.200.19.2, 1d09h
B        172.39.39.32 [20/0] via 10.200.19.2, 1d09h
rt-b-01#
root ##bl##rt-b-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.34/32 is directly connected, GigabitEthernet2
S        10.100.19.0/24 [1/0] via 10.11.11.253
S        10.100.21.0/24 [1/0] via 10.11.11.253
C        10.200.1.0/24 is directly connected, GigabitEthernet1
L        10.200.1.102/32 is directly connected, GigabitEthernet1
B        10.200.19.0/24 [20/0] via 10.200.21.2, 1d09h
C        10.200.21.0/24 is directly connected, GigabitEthernet3
L        10.200.21.1/32 is directly connected, GigabitEthernet3
      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.2.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.3.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.8.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.9.0 [20/0] via 10.11.11.253, 1d09h
B        172.20.10.0 [20/0] via 10.11.11.253, 1d09h
      172.30.0.0/24 is subnetted, 3 subnets
B        172.30.1.0 [20/0] via 10.200.21.2, 1d09h
B        172.30.2.0 [20/0] via 10.200.21.2, 1d09h
B        172.30.3.0 [20/0] via 10.200.21.2, 1d09h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.200.21.2, 1d09h
B        172.39.39.16 [20/0] via 10.200.21.2, 1d09h
B        172.39.39.32 [20/0] via 10.200.21.2, 1d09h
rt-b-02#

Core routing table

root ##bl##cs-01#show ip route bgp 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 10.11.11.254 to network 0.0.0.0

      172.20.0.0/24 is subnetted, 6 subnets
B        172.20.1.0 [20/0] via 10.11.11.31, 1d09h
B        172.20.2.0 [20/0] via 10.11.11.31, 1d09h
B        172.20.3.0 [20/0] via 10.11.11.31, 1d09h
B        172.20.8.0 [20/0] via 10.11.11.31, 1d09h
B        172.20.9.0 [20/0] via 10.11.11.31, 1d09h
B        172.20.10.0 [20/0] via 10.11.11.31, 1d09h
      172.30.0.0/24 is subnetted, 3 subnets
B        172.30.1.0 [20/0] via 10.11.11.33, 1d09h
B        172.30.2.0 [20/0] via 10.11.11.33, 1d09h
B        172.30.3.0 [20/0] via 10.11.11.33, 1d09h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.11.11.31, 1d09h
B        172.39.39.16 [20/0] via 10.11.11.31, 1d09h
B        172.39.39.32 [20/0] via 10.11.11.34, 1d09h
cs-01#

UDLR BGP tables

DC1

Nsx-routing-blog-locale-24.png

DC2

root ##bl##NSX-edge-71b9e24f-9a47-423d-b8e7-392dc8b3e47a-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
  > 10.11.11.0/24      172.39.39.33      0     100      60   65522 i
  > 10.100.19.0/24     172.39.39.33      0     100      60   65522 ?
  > 10.100.21.0/24     172.39.39.33      0     100      60   65522 ?
  > 10.200.19.0/24     172.39.39.33      0     100      60   65522 ?
  > 10.200.21.0/24     172.39.39.33      0     100      60   65522 ?
  > 172.20.1.0/24      172.39.39.33      0     100      60   65522 ?
  > 172.20.2.0/24      172.39.39.33      0     100      60   65522 ?
  > 172.20.3.0/24      172.39.39.33      0     100      60   65522 ?
  > 172.20.8.0/24      172.39.39.33      0     100      60   65522 ?
  > 172.20.9.0/24      172.39.39.33      0     100      60   65522 ?
  > 172.20.10.0/24     172.39.39.33      0     100      60   65522 ?
  > 172.30.1.0/24      0.0.0.0           0     100   32768   ?
  > 172.30.2.0/24      0.0.0.0           0     100   32768   ?
  > 172.30.3.0/24      0.0.0.0           0     100   32768   ?
  > 172.39.39.0/28     172.39.39.33      0     100      60   65522 ?
  > 172.39.39.16/28    0.0.0.0           0     100   32768   ?
    172.39.39.32/28    172.39.39.33      0     100      60   65522 ?
  > 172.39.39.32/28    0.0.0.0           0     100   32768   ?
NSX-edge-71b9e24f-9a47-423d-b8e7-392dc8b3e47a-0>


ESG BGP tables

DC1

root ##bl##REGA-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
  > 10.11.11.0/24      10.100.19.1       0     100      60   65511 65510 i
    10.11.11.0/24      10.100.21.1       0     100      60   65511 65510 i
  > 10.100.19.0/24     0.0.0.0           0     100   32768   ?
  > 10.100.21.0/24     0.0.0.0           0     100   32768   ?
  > 10.200.19.0/24     10.100.19.1       0     100      60   65511 65510 65512 65522 ?
    10.200.19.0/24     10.100.21.1       0     100      60   65511 65510 65512 65522 ?
  > 10.200.21.0/24     10.100.19.1       0     100      60   65511 65510 65512 65522 ?
    10.200.21.0/24     10.100.21.1       0     100      60   65511 65510 65512 65522 ?
  > 172.20.1.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.2.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.3.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.8.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.9.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.10.0/24     172.39.39.14      0     100      60   65530 ?
  > 172.30.1.0/24      172.39.39.30      0     100      60   65530 ?
  > 172.30.2.0/24      172.39.39.30      0     100      60   65530 ?
  > 172.30.3.0/24      172.39.39.30      0     100      60   65530 ?
    172.39.39.0/28     172.39.39.14      0     100      60   65530 ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
    172.39.39.16/28    172.39.39.30      0     100      60   65530 ?
  > 172.39.39.16/28    0.0.0.0           0     100   32768   ?
  > 172.39.39.32/28    172.39.39.30      0     100      60   65530 ?
REGA-ESG01-0>

DC2

root ##bl##REGB-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
  > 10.11.11.0/24      10.200.19.1       0     100      60   65512 65510 i
    10.11.11.0/24      10.200.21.1       0     100      60   65512 65510 i
  > 10.100.19.0/24     10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    10.100.19.0/24     10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 10.100.21.0/24     10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    10.100.21.0/24     10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 10.200.19.0/24     0.0.0.0           0     100   32768   ?
  > 10.200.21.0/24     0.0.0.0           0     100   32768   ?
  > 172.20.1.0/24      10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    172.20.1.0/24      10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 172.20.2.0/24      10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    172.20.2.0/24      10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 172.20.3.0/24      10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    172.20.3.0/24      10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 172.20.8.0/24      10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    172.20.8.0/24      10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 172.20.9.0/24      10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    172.20.9.0/24      10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 172.20.10.0/24     10.200.19.1       0     100      60   65512 65510 65511 65521 ?
    172.20.10.0/24     10.200.21.1       0     100      60   65512 65510 65511 65521 ?
  > 172.30.1.0/24      172.39.39.46      0     100      60   65530 ?
  > 172.30.2.0/24      172.39.39.46      0     100      60   65530 ?
  > 172.30.3.0/24      172.39.39.46      0     100      60   65530 ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
  > 172.39.39.16/28    172.39.39.46      0     100      60   65530 ?
    172.39.39.32/28    172.39.39.46      0     100      60   65530 ?
  > 172.39.39.32/28    0.0.0.0           0     100   32768   ?
REGB-ESG01-0>

Cisco1000V BGP tables

DC1

root ##bl##rt-a-01#show ip bgp
BGP table version is 54, local router ID is 10.11.11.31
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.100.19.2                            0 65521 ?
 *>  10.100.21.0/24   10.100.19.2                            0 65521 ?
 r>  10.200.19.0/24   10.11.11.253                           0 65510 65512 65522 ?
 r>  10.200.21.0/24   10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.1.0/24    10.100.19.2                            0 65521 ?
 *>  172.20.2.0/24    10.100.19.2                            0 65521 ?
 *>  172.20.3.0/24    10.100.19.2                            0 65521 ?
 *>  172.20.8.0/24    10.100.19.2                            0 65521 ?
 *>  172.20.9.0/24    10.100.19.2                            0 65521 ?
 *>  172.20.10.0/24   10.100.19.2                            0 65521 ?
 *>  172.30.1.0/24    10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.30.2.0/24    10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.30.3.0/24    10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.39.39.0/28   10.100.19.2                            0 65521 ?
 *>  172.39.39.16/28  10.100.19.2                            0 65521 ?
 *>  172.39.39.32/28  10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
rt-a-01#
root ##bl##rt-a-02#show ip bgp
BGP table version is 54, local router ID is 10.11.11.32
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.100.19.0/24   10.100.21.2                            0 65521 ?
 r>  10.100.21.0/24   10.100.21.2                            0 65521 ?
 r>  10.200.19.0/24   10.11.11.253                           0 65510 65512 65522 ?
 r>  10.200.21.0/24   10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.1.0/24    10.100.21.2                            0 65521 ?
 *>  172.20.2.0/24    10.100.21.2                            0 65521 ?
 *>  172.20.3.0/24    10.100.21.2                            0 65521 ?
 *>  172.20.8.0/24    10.100.21.2                            0 65521 ?
 *>  172.20.9.0/24    10.100.21.2                            0 65521 ?
 *>  172.20.10.0/24   10.100.21.2                            0 65521 ?
 *>  172.30.1.0/24    10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.30.2.0/24    10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.30.3.0/24    10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.39.39.0/28   10.100.21.2                            0 65521 ?
 *>  172.39.39.16/28  10.100.21.2                            0 65521 ?
 *>  172.39.39.32/28  10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
rt-a-02#

DC2

root ##bl##rt-b-01#show ip bgp
BGP table version is 76, local router ID is 10.11.11.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>  10.100.21.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>  10.200.19.0/24   10.200.19.2                            0 65522 ?
 *>  10.200.21.0/24   10.200.19.2                            0 65522 ?
 *>  172.20.1.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.2.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.3.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.8.0/24    10.11.11.253                           0 65510 65511 65521 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.20.9.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.10.0/24   10.11.11.253                           0 65510 65511 65521 ?
 *>  172.30.1.0/24    10.200.19.2                            0 65522 ?
 *>  172.30.2.0/24    10.200.19.2                            0 65522 ?
 *>  172.30.3.0/24    10.200.19.2                            0 65522 ?
 *>  172.39.39.0/28   10.200.19.2                            0 65522 ?
 *                    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.39.39.16/28  10.200.19.2                            0 65522 ?
 *                    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.39.39.32/28  10.200.19.2                            0 65522 ?
rt-b-01#
root ##bl##rt-b-02#show ip bgp
BGP table version is 49, local router ID is 10.11.11.34
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>  10.100.21.0/24   10.11.11.253                           0 65510 65511 65521 ?
 *>  10.200.19.0/24   10.200.21.2                            0 65522 ?
 r>  10.200.21.0/24   10.200.21.2                            0 65522 ?
 *>  172.20.1.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.2.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.3.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.8.0/24    10.11.11.253                           0 65510 65511 65521 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.20.9.0/24    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.20.10.0/24   10.11.11.253                           0 65510 65511 65521 ?
 *>  172.30.1.0/24    10.200.21.2                            0 65522 ?
 *>  172.30.2.0/24    10.200.21.2                            0 65522 ?
 *>  172.30.3.0/24    10.200.21.2                            0 65522 ?
 *>  172.39.39.0/28   10.200.21.2                            0 65522 ?
 *                    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.39.39.16/28  10.200.21.2                            0 65522 ?
 *                    10.11.11.253                           0 65510 65511 65521 ?
 *>  172.39.39.32/28  10.200.21.2                            0 65522 ?
rt-b-02#

Core BGP tables

root ##bl##cs-01#show ip bgp
BGP table version is 83, local router ID is 10.11.11.253
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.11.11.0/24    0.0.0.0                  0         32768 i
r  10.100.19.0/24   10.11.11.32                            0 65511 65521 ?
r>                  10.11.11.31                            0 65511 65521 ?
r  10.100.21.0/24   10.11.11.32                            0 65511 65521 ?
r>                  10.11.11.31                            0 65511 65521 ?
r  10.200.19.0/24   10.11.11.33                            0 65512 65522 ?
r>                  10.11.11.34                            0 65512 65522 ?
r  10.200.21.0/24   10.11.11.33                            0 65512 65522 ?
r>                  10.11.11.34                            0 65512 65522 ?
*  172.20.1.0/24    10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.20.2.0/24    10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.20.3.0/24    10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.20.8.0/24    10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.20.9.0/24    10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.20.10.0/24   10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.30.1.0/24    10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*                   10.11.11.34                            0 65512 65522 ?
*>                  10.11.11.33                            0 65512 65522 ?
*  172.30.2.0/24    10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*                   10.11.11.34                            0 65512 65522 ?
*>                  10.11.11.33                            0 65512 65522 ?
*  172.30.3.0/24    10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*                   10.11.11.34                            0 65512 65522 ?
*>                  10.11.11.33                            0 65512 65522 ?
*  172.39.39.0/28   10.11.11.33                            0 65512 65522 ?
*                   10.11.11.34                            0 65512 65522 ?
*                   10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.39.39.16/28  10.11.11.34                            0 65512 65522 ?
*                   10.11.11.33                            0 65512 65522 ?
*                   10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.39.39.32/28  10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*                   10.11.11.33                            0 65512 65522 ?
*>                  10.11.11.34                            0 65512 65522 ?
cs-01#

Routing path verifications

Routing path verification from the external client VM to T3–WEB–1, T3–WEB–2, T3–WEB–3 and T3–WEB–4

This is an INGRESS trafficflow. As we did not make any configuration changes to manipulate the INGRESS trafficflow the INGRESS traffic can either choose to select the DC1 path or the DC2 path. This is currently determined by BGP.

root ##bl##C:\Users\Administrator>tracert 172.30.1.1

Tracing route to 172.30.1.1 over a maximum of 30 hops

  1     1 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3     1 ms    <1 ms    <1 ms  10.200.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.46
  5     1 ms     1 ms     1 ms  172.30.1.1

Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.30.1.2

Tracing route to 172.30.1.2 over a maximum of 30 hops

  1     2 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.46
  5   944 ms     1 ms    <1 ms  172.30.1.2

Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.30.1.3

Tracing route to 172.30.1.3 over a maximum of 30 hops

  1     1 ms     1 ms     6 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.46
  5   723 ms     1 ms    <1 ms  172.30.1.3

Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.30.1.4

Tracing route to 172.30.1.4 over a maximum of 30 hops

  1     1 ms     1 ms     4 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.46
  5    82 ms     1 ms    <1 ms  172.30.1.4

Trace complete.

Routing path verification from the T3–WEB–1, T3–WEB–2, T3–WEB–3 and T3–WEB–4 to the external client VM

This is an EGRESS trafficflow. The expectation is that the traffic towards T3-WEB-1 and T3-WEB-2 will fo trough the DC1 routers and the traffic towards T3-WEB-3 and T3-WEB-4 will fo trough the DC2 routers. The reason for this is that Locale ID is configured.

root ##bl##root@Web01:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.082 ms  2.689 ms  2.687 ms
 2  172.39.39.17 (172.39.39.17)  0.666 ms  0.527 ms  0.524 ms
 3  10.100.21.1 (10.100.21.1)  1.067 ms  1.091 ms  1.072 ms
 4  10.11.11.50 (10.11.11.50)  1.347 ms * *
root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.172 ms  0.149 ms  0.131 ms
 2  * * 172.39.39.17 (172.39.39.17)  0.455 ms
 3  10.100.21.1 (10.100.21.1)  0.758 ms  0.599 ms  0.494 ms
 4  10.11.11.50 (10.11.11.50)  0.918 ms * *
root@Web02:~#
root ##bl##root@Web03:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.120 ms  0.109 ms  0.060 ms
 2  172.39.39.33 (172.39.39.33)  0.387 ms  0.345 ms  0.445 ms
 3  10.200.21.1 (10.200.21.1)  1.249 ms  1.371 ms  1.383 ms
 4  10.11.11.50 (10.11.11.50)  1.210 ms * *
root@Web03:~#
root ##bl##root@Web04:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.109 ms  0.181 ms  0.132 ms
 2  172.39.39.33 (172.39.39.33)  0.471 ms  0.470 ms  0.466 ms
 3  10.200.21.1 (10.200.21.1)  0.900 ms  0.871 ms  0.868 ms
 4  10.11.11.50 (10.11.11.50)  1.063 ms * *
root@Web04:~#

Change the Locale ID manually on DC2 to match DC1

DC1

The locale ID will not change here and these settings are kept te same

DC2

Compute Cluster

Nsx-routing-blog-locale-25.png

Routing path verifications

Routing path verification from the external client VM to T3–WEB–1, T3–WEB–2, T3–WEB–3 and T3–WEB–4

This is an INGRESS trafficflow. As we did not make any configuration changes to manipulate the INGRESS trafficflow the INGRESS traffic can either choose to select the DC1 path or the DC2 path. This is currently determined by BGP.

root ##bl##C:\Users\Administrator>tracert 172.30.1.1

Tracing route to 172.30.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms     3 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3     1 ms    <1 ms    <1 ms  10.200.21.2
  4   231 ms    <1 ms    <1 ms  172.39.39.30
  5   966 ms     1 ms    <1 ms  172.30.1.1

Trace complete.


root ##bl##C:\Users\Administrator>tracert 172.30.1.2

Tracing route to 172.30.1.2 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.30
  5     1 ms     1 ms    <1 ms  172.30.1.2

Trace complete.


root ##bl##C:\Users\Administrator>tracert 172.30.1.3

Tracing route to 172.30.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms     3 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.30
  5     2 ms    <1 ms    <1 ms  172.30.1.3

Trace complete.


root ##bl##C:\Users\Administrator>tracert 172.30.1.4

Tracing route to 172.30.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
  3    <1 ms    <1 ms    <1 ms  10.200.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.30
  5   761 ms     1 ms     1 ms  172.30.1.4

Trace complete.

Routing path verification from the T3–WEB–1, T3–WEB–2, T3–WEB–3 and T3–WEB–4 to the external client VM

This is an EGRESS trafficflow. The expectation is that the traffic towards T3-WEB-1, T3-WEB-2, T3-WEB-3 and T3-WEB-4 will go trough the DC1 routers. The reason for this is that Locale ID is configured and tweaked.

root ##bl##root@Web01:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.209 ms  0.086 ms  0.091 ms
 2  172.39.39.17 (172.39.39.17)  0.612 ms  0.603 ms  0.594 ms
 3  10.100.21.1 (10.100.21.1)  1.129 ms  1.234 ms  1.184 ms
 4  10.11.11.50 (10.11.11.50)  1.583 ms * *
root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.223 ms  0.098 ms  0.107 ms
 2  172.39.39.17 (172.39.39.17)  0.559 ms  0.564 ms  0.546 ms
 3  10.100.21.1 (10.100.21.1)  1.179 ms  1.226 ms  1.221 ms
 4  10.11.11.50 (10.11.11.50)  2.281 ms * *
root@Web02:~#
root ##bl##root@Web03:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.250 ms  0.126 ms  0.130 ms
 2  * * 172.39.39.17 (172.39.39.17)  0.991 ms
 3  10.100.19.1 (10.100.19.1)  1.303 ms  1.323 ms  1.350 ms
 4  10.11.11.50 (10.11.11.50)  1.345 ms * *
root@Web03:~#
root ##bl##root@Web04:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.30.1.254 (172.30.1.254)  0.147 ms  0.093 ms  0.079 ms
 2  * * 172.39.39.17 (172.39.39.17)  1.076 ms
 3  10.100.19.1 (10.100.19.1)  1.951 ms  1.948 ms  2.100 ms
 4  10.11.11.50 (10.11.11.50)  1.799 ms * *
root@Web04:~#

Lets now put the Locale ID back to how it was and have the egress traffic exiting locally again. What happens in case of any failures on DC2. Does DC1 takes over the path for the DC2 workloads when the ESG-B goes down automatically?

Take ESG–B down and verify the paths

After I have verified if the locale ID is working as it should I did a power-off on the ESG-B to see what happens. The expectation is that all the traffic that was exiting the DC2 WAN routers will now go trough the DC1 WAN routers. After the power-off of the ESG-B the route was not rerouted automatically. A change of the Locale ID of the DC2 compute cluster (manually) forced a re-route to the DC1 WAN routers.