Last Thursday i got a notification that there are new images out for the Nexus 9k. First thing i checked was if there is one too for the Nexus 9000v. And yes, there was it: 7.0(3)I6(1). Next thing to check was if Cisco integrated the highly anticipated feature VxLAN BGP EVPN. And after a bit of searching (since there were no release notes) i found it in the already updated NX-OSv 9000 Guide:
Of course I immediately wanted to test it, but i had a real lab available over the weekend, so that had to wait. But now i had the time for a simple setup to test it. And I’m so glad -> It does work great! In the following days i will do a test with a bigger setup and also do a VxLAN Routing setup.
Here is the configuration i used. Since (beside of the IPs) the config is the same for every Leaf and Spine. I did just put in one for each of them:
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature vn-segment-vlan-based
feature nv overlay
vlan 1,192
ip pim rp-address 10.0.0.255 group-list 224.0.0.0/4
ip pim anycast-rp 10.0.0.255 10.0.255.1
ip pim anycast-rp 10.0.0.255 10.0.255.2
vlan 192
vn-segment 100192
interface nve1
no shutdown
source-interface loopback1
host-reachability protocol bgp
member vni 100192
suppress-arp
mcast-group 224.1.1.192
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/9
switchport access vlan 192
interface loopback0
description Underlay
ip address 10.0.255.11/32
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Overlay
ip address 10.1.255.11/32
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
router ospf 255
router-id 10.0.255.11
router bgp 255
router-id 10.1.255.11
neighbor 10.1.255.1
remote-as 255
update-source loopback1
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.1.255.2
remote-as 255
update-source loopback1
address-family l2vpn evpn
send-community
send-community extended
evpn
vni 100192 l2
rd auto
route-target import auto
route-target export auto
nv overlay evpn
feature ospf
feature bgp
feature pim
ip pim rp-address 10.0.0.255 group-list 224.0.0.0/4
ip pim anycast-rp 10.0.0.255 10.0.255.1
ip pim anycast-rp 10.0.0.255 10.0.255.2
vlan 1
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Underlay
ip address 10.0.255.1/32
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Overlay
ip address 10.1.255.1/32
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
interface loopback255
ip address 10.0.0.255/32
ip router ospf 255 area 0.0.0.0
ip pim sparse-mode
router ospf 255
router-id 10.0.255.1
router bgp 255
router-id 10.1.255.1
template peer Leaf
update-source loopback1
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.1.255.11
inherit peer Leaf
remote-as 255
neighbor 10.1.255.12
inherit peer Leaf
remote-as 255
Heres the verification:
Leaf_1# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 192 0050.7966.6800 static - F F (0x47000001) nve-peer1
10.1.25
* 192 0050.7966.6801 dynamic 00:00:48 F F Eth1/9
1 1 -00:00:22:22:33:33 - 1
Leaf_1# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.1.255.12 Up CP 00:01:32 n/a
Leaf_1# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 100192 224.1.1.192 Up CP L2 [192] SA
Leaf_1# show vxlan interface
Interface Vlan VPL Ifindex LTL HW VP
========= ==== =========== === =====
Eth1/9 192 0x530c07f8 0x1801 2050
Leaf_1# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.1.255.11, local AS number 255
BGP table version is 44, L2VPN EVPN config peers 2, capable peers 2
6 network entries and 8 paths using 1296 bytes of memory
BGP attribute entries [3/468], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.255.1 4 255 61 65 44 0 0 00:47:35 2
10.1.255.2 4 255 61 65 44 0 0 00:47:18 2
Leaf_1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 44, local router ID is 10.1.255.11
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.1.255.11:32959 (L2VNI 100192)
*>i[2]:[0]:[0]:[48]:[0050.7966.6800]:[0]:[0.0.0.0]/216
10.1.255.12 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6801]:[0]:[0.0.0.0]/216
10.1.255.11 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6800]:[32]:[192.168.1.2]/248
10.1.255.12 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.1]/248
10.1.255.11 100 32768 i
Route Distinguisher: 10.1.255.12:32959
* i[2]:[0]:[0]:[48]:[0050.7966.6800]:[0]:[0.0.0.0]/216
10.1.255.12 100 0 i
*>i 10.1.255.12 100 0 i
* i[2]:[0]:[0]:[48]:[0050.7966.6800]:[32]:[192.168.1.2]/248
10.1.255.12 100 0 i
*>i 10.1.255.12 100 0 i
Leaf_1# show l2route evpn mac all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete(D):Del Pending (S):Stale
(C):Clear
(Ps):Peer Sync (O):Re-Originated
Topology Mac Address Prod Flags Seq No Next-Hops
----------- -------------- ------ ------------- ---------- ----------------
192 0050.7966.6800 BGP SplRcv 0 10.1.255.12
192 0050.7966.6801 Local L, 0 Eth1/9
Leaf_1# show l2route evpn mac-ip all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated
Topology Mac Address Prod Flags Seq No Host IP Next-
Hops
----------- -------------- ------ ---------- --------------- ---------------
192 0050.7966.6801 ARP L, 0 192.168.1.1 Eth1/9
192 0050.7966.6800 BGP -- 0 192.168.1.2 10.1.2
55.12
Leaf_2# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 192 0050.7966.6800 dynamic 00:05:03 F F Eth1/9
* 192 0050.7966.6801 static - F F (0x47000001) nve-peer1
10.1.25
1 1 -00:00:22:22:33:33 - 1
Leaf_2# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 10.1.255.11 Up CP 00:05:20 n/a
Leaf_2# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 100192 224.1.1.192 Up CP L2 [192] SA
Leaf_2# show vxlan interface
Interface Vlan VPL Ifindex LTL HW VP
========= ==== =========== === =====
Eth1/9 192 0x530c07f8 0x1801 2050
Leaf_2# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.1.255.12, local AS number 255
BGP table version is 45, L2VPN EVPN config peers 2, capable peers 2
6 network entries and 8 paths using 1296 bytes of memory
BGP attribute entries [3/468], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.255.1 4 255 64 67 45 0 0 00:51:08 2
10.1.255.2 4 255 64 67 45 0 0 00:51:06 2
Leaf_2# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 45, local router ID is 10.1.255.12
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.1.255.11:32959
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[0]:[0.0.0.0]/216
10.1.255.11 100 0 i
* i 10.1.255.11 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.1]/248
10.1.255.11 100 0 i
* i 10.1.255.11 100 0 i
Route Distinguisher: 10.1.255.12:32959 (L2VNI 100192)
*>l[2]:[0]:[0]:[48]:[0050.7966.6800]:[0]:[0.0.0.0]/216
10.1.255.12 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[0]:[0.0.0.0]/216
10.1.255.11 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.7966.6800]:[32]:[192.168.1.2]/248
10.1.255.12 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.7966.6801]:[32]:[192.168.1.1]/248
10.1.255.11 100 0 i
Leaf_2# show l2route evpn mac all
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete(D):Del Pending (S):Stale
(C):Clear
(Ps):Peer Sync (O):Re-Originated
Topology Mac Address Prod Flags Seq No Next-Hops
----------- -------------- ------ ------------- ---------- ----------------
192 0050.7966.6800 Local L, 0 Eth1/9
192 0050.7966.6801 BGP SplRcv 0 10.1.255.11
How much RAM did you spend to set up this test topology/
Hi Nestor!
I’ve testet it with 8GB and 4GB of Ram per Nexus 9000v and it worked fine with both. Then you should have about one CPU Core per Nexus.
You can just add up the RAM and CPU per device you want to simulate. From my experience, it’s safer to overbook the CPU then the RAM.
Hi!
I tried to build a very similar lab, except that I used VPC pairs as leaf. After a reboot the nve interface is simple lost from the running configuration, but it is present in the startup. I cannot add the interface again, becouse I do not have nve option after interface keyword. I also tried to reenable the VXLAN relevant features, but it did not help either. Have you ever seen something like this?
Thanks
Hi!
Which features were enabled after the reboot? Which one did you reenable? Which gns3 version are you running and which 9000v version?
I saw similar behavior with gns 1.x, but since 2.0 it looks much better …
Noticed the same issue, using UNL,. Everything was configured working fine, after some reeboots int NVE was not present . I will try to make a new lab setup but It smells like a bug.
I’ve never tried UNL. But since it already did work, could really be a bug …
Martin,
Where did you get the 9000v code from CCO? If so, how did you make it work on GNS3?
Here you go -> https://nfvguy.mas-net.at/2017/03/13/nexus-9000v-on-gns3/
Hi great stuff man !!
One question, do you know if multicast is supported in Nexus 9000v on the data plane ?
Hi George,
as of now the documentation says no:
L2 Switching Multicast -> Supported as Broadcast (not explicit Mcast) , No PIM or Mcast Group support
-> https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/nx-osv/configuration/guide/b_Cisco_Nexus_9000v/b_Cisco_Nexus_9000v_chapter_011.html#concept_43BBF174020E4B59ACA983D802522853