VxLAN BGP EVPN on Nexus 9000v – Part 1

Last Thursday i got a notification that there are new images out for the Nexus 9k. First thing i checked was if there is one too for the Nexus 9000v. And yes, there was it: 7.0(3)I6(1). Next thing to check was if Cisco integrated the highly anticipated feature VxLAN BGP EVPN. And after a bit of searching (since there were no release notes) i found it in the already updated NX-OSv 9000 Guide:

Of course I immediately wanted to test it, but i had a real lab available over the weekend, so that had to wait. But now i had the time for a simple setup to test it. And I’m so glad -> It does work great! In the following days i will do a test with a bigger setup and also do a VxLAN Routing setup.

Here is the configuration i used. Since (beside of the IPs) the config is the same for every Leaf and Spine. I did just put in one for each of them:

nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature vn-segment-vlan-based
feature nv overlay

vlan 1,192
ip pim rp-address 10.0.0.255 group-list 224.0.0.0/4
ip pim anycast-rp 10.0.0.255 10.0.255.1
ip pim anycast-rp 10.0.0.255 10.0.255.2
vlan 192
  vn-segment 100192

interface nve1
  no shutdown
  source-interface loopback1
  host-reachability protocol bgp
  member vni 100192
    suppress-arp
    mcast-group 224.1.1.192

interface Ethernet1/1
  no switchport
  medium p2p
  ip unnumbered loopback0
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  no switchport
  medium p2p
  ip unnumbered loopback0
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/9
  switchport access vlan 192

 interface loopback0
  description Underlay
  ip address 10.0.255.11/32
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  description Overlay
  ip address 10.1.255.11/32
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode

router ospf 255
  router-id 10.0.255.11
router bgp 255
  router-id 10.1.255.11
  neighbor 10.1.255.1
    remote-as 255
    update-source loopback1
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.1.255.2
    remote-as 255
    update-source loopback1
    address-family l2vpn evpn
      send-community
      send-community extended
evpn
  vni 100192 l2
    rd auto
    route-target import auto
    route-target export auto
nv overlay evpn
feature ospf
feature bgp
feature pim


ip pim rp-address 10.0.0.255 group-list 224.0.0.0/4
ip pim anycast-rp 10.0.0.255 10.0.255.1
ip pim anycast-rp 10.0.0.255 10.0.255.2
vlan 1

interface Ethernet1/1
  no switchport
  medium p2p
  ip unnumbered loopback0
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  no switchport
  medium p2p
  ip unnumbered loopback0
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback0
  description Underlay
  ip address 10.0.255.1/32
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  description Overlay
  ip address 10.1.255.1/32
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode

interface loopback255
  ip address 10.0.0.255/32
  ip router ospf 255 area 0.0.0.0
  ip pim sparse-mode

router ospf 255
  router-id 10.0.255.1
router bgp 255
  router-id 10.1.255.1
  template peer Leaf
    update-source loopback1
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client
  neighbor 10.1.255.11
    inherit peer Leaf
    remote-as 255
  neighbor 10.1.255.12
    inherit peer Leaf
    remote-as 255

Heres the verification:

Leaf_1# show system internal l2fwder mac 
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   192    0050.7966.6800    static   -          F     F  (0x47000001) nve-peer1
 10.1.25  
*   192    0050.7966.6801   dynamic   00:00:48   F     F     Eth1/9  
    1           1         -00:00:22:22:33:33         -             1

Leaf_1# show nve peers
Interface Peer-IP          State LearnType Uptime   Router-Mac       
--------- ---------------  ----- --------- -------- -----------------
nve1      10.1.255.12      Up    CP        00:01:32 n/a              

Leaf_1# show nve vni
Codes: CP - Control Plane        DP - Data Plane          
       UC - Unconfigured         SA - Suppress ARP
       
Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      100192   224.1.1.192       Up    CP   L2 [192]           SA   

Leaf_1# show vxlan interface
Interface       Vlan    VPL Ifindex     LTL             HW VP
=========       ====    ===========     ===             =====
Eth1/9          192     0x530c07f8      0x1801          2050

Leaf_1# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.1.255.11, local AS number 255
BGP table version is 44, L2VPN EVPN config peers 2, capable peers 2
6 network entries and 8 paths using 1296 bytes of memory
BGP attribute entries [3/468], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.255.1      4   255      61      65       44    0    0 00:47:35 2         
10.1.255.2      4   255      61      65       44    0    0 00:47:18 2         

Leaf_1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 44, local router ID is 10.1.255.11
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.1.255.11:32959    (L2VNI 100192)
*>i[2]:[0]:[0]:[48]:[0050.7966.6800]:[0]:[0.0.0.0]/216
                      10.1.255.12                       100          0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6801]:[0]:[0.0.0.0]/216
                      10.1.255.11                       100      32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6800]:[32]:[192.168.1.2]/248
                      10.1.255.12                       100          0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.1]/248
                      10.1.255.11                       100      32768 i

Route Distinguisher: 10.1.255.12:32959
* i[2]:[0]:[0]:[48]:[0050.7966.6800]:[0]:[0.0.0.0]/216
                      10.1.255.12                       100          0 i
*>i                   10.1.255.12                       100          0 i
* i[2]:[0]:[0]:[48]:[0050.7966.6800]:[32]:[192.168.1.2]/248
                      10.1.255.12                       100          0 i
*>i                   10.1.255.12                       100          0 i

Leaf_1# show l2route evpn mac all

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link 
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete(D):Del Pending (S):Stale
 (C):Clear
(Ps):Peer Sync (O):Re-Originated 

Topology    Mac Address    Prod   Flags         Seq No     Next-Hops      
----------- -------------- ------ ------------- ---------- ----------------
192         0050.7966.6800 BGP    SplRcv        0          10.1.255.12    
192         0050.7966.6801 Local  L,            0          Eth1/9         

Leaf_1# show l2route evpn mac-ip all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link 
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated 
Topology    Mac Address    Prod   Flags         Seq No     Host IP         Next-
Hops      
----------- -------------- ------ ---------- --------------- ---------------
192         0050.7966.6801 ARP    L,            0          192.168.1.1    Eth1/9
         
192         0050.7966.6800 BGP    --            0          192.168.1.2    10.1.2
55.12
Leaf_2#  show system internal l2fwder mac 
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   192    0050.7966.6800   dynamic   00:05:03   F     F     Eth1/9  
*   192    0050.7966.6801    static   -          F     F  (0x47000001) nve-peer1
 10.1.25  
    1           1         -00:00:22:22:33:33         -             1

Leaf_2# show nve peers
Interface Peer-IP          State LearnType Uptime   Router-Mac       
--------- ---------------  ----- --------- -------- -----------------
nve1      10.1.255.11      Up    CP        00:05:20 n/a              

Leaf_2# show nve vni
Codes: CP - Control Plane        DP - Data Plane          
       UC - Unconfigured         SA - Suppress ARP
       
Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      100192   224.1.1.192       Up    CP   L2 [192]           SA   

Leaf_2# show vxlan interface
Interface       Vlan    VPL Ifindex     LTL             HW VP
=========       ====    ===========     ===             =====
Eth1/9          192     0x530c07f8      0x1801          2050

Leaf_2# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.1.255.12, local AS number 255
BGP table version is 45, L2VPN EVPN config peers 2, capable peers 2
6 network entries and 8 paths using 1296 bytes of memory
BGP attribute entries [3/468], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.255.1      4   255      64      67       45    0    0 00:51:08 2         
10.1.255.2      4   255      64      67       45    0    0 00:51:06 2         

Leaf_2# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 45, local router ID is 10.1.255.12
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.1.255.11:32959
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[0]:[0.0.0.0]/216
                      10.1.255.11                       100          0 i
* i                   10.1.255.11                       100          0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.1]/248
                      10.1.255.11                       100          0 i
* i                   10.1.255.11                       100          0 i

Route Distinguisher: 10.1.255.12:32959    (L2VNI 100192)
*>l[2]:[0]:[0]:[48]:[0050.7966.6800]:[0]:[0.0.0.0]/216
                      10.1.255.12                       100      32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[0]:[0.0.0.0]/216
                      10.1.255.11                       100          0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6800]:[32]:[192.168.1.2]/248
                      10.1.255.12                       100      32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.1]/248
                      10.1.255.11                       100          0 i

Leaf_2# show l2route evpn mac all

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link 
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete(D):Del Pending (S):Stale
 (C):Clear
(Ps):Peer Sync (O):Re-Originated 

Topology    Mac Address    Prod   Flags         Seq No     Next-Hops      
----------- -------------- ------ ------------- ---------- ----------------
192         0050.7966.6800 Local  L,            0          Eth1/9         
192         0050.7966.6801 BGP    SplRcv        0          10.1.255.11

 

 

 

Related posts

10 Thoughts to “VxLAN BGP EVPN on Nexus 9000v – Part 1”

  1. Nestor

    How much RAM did you spend to set up this test topology/

    1. Hi Nestor!

      I’ve testet it with 8GB and 4GB of Ram per Nexus 9000v and it worked fine with both. Then you should have about one CPU Core per Nexus.

      You can just add up the RAM and CPU per device you want to simulate. From my experience, it’s safer to overbook the CPU then the RAM.

  2. Roland Pet?

    Hi!

    I tried to build a very similar lab, except that I used VPC pairs as leaf. After a reboot the nve interface is simple lost from the running configuration, but it is present in the startup. I cannot add the interface again, becouse I do not have nve option after interface keyword. I also tried to reenable the VXLAN relevant features, but it did not help either. Have you ever seen something like this?

    Thanks

    1. Hi!

      Which features were enabled after the reboot? Which one did you reenable? Which gns3 version are you running and which 9000v version?

      I saw similar behavior with gns 1.x, but since 2.0 it looks much better …

  3. syn

    Noticed the same issue, using UNL,. Everything was configured working fine, after some reeboots int NVE was not present . I will try to make a new lab setup but It smells like a bug.

    1. I’ve never tried UNL. But since it already did work, could really be a bug …

  4. John Taylor

    Martin,

    Where did you get the 9000v code from CCO? If so, how did you make it work on GNS3?

  5. George Solorzano

    Hi great stuff man !!

    One question, do you know if multicast is supported in Nexus 9000v on the data plane ?

Leave a Reply to George Solorzano Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.