After getting VXLAN F&L to work on the Nexus 9000v. I thought why not trying the “easier one”, which was already tested a lot of times over the last 1-2 years. And well, i works out of the box.
How is it done?
- CSR1000v as Leafs
- IOU L3 Router (you can basically use any router that is capable of OSPF/ISIS/EIGRP and PIM)
- VPCS als clients to ping
- GNS3 to capture the Ethernet frames for a deeper understanding of what’s going on
At first you need basic connectivity and dynamic routing in your fabric. Since i played a lot with OSPF lately, it was a welcomed chance to freshen up the command syntax of ISIS:
interface Loopback0 ip address 10.0.255.11 255.255.255.255 interface GigabitEthernetx description to Spine_x ip address 10.x.x.2 255.255.255.0 ip router isis 255 router isis 255 net 49.0001.0000.0000.0011.00 is-type level-1 log-adjacency-changes passive-interface Loopback0
interface Loopback0 ip address 10.0.255.1 255.255.255.255 interface Ethernet0/x description to Spine_x ip address 10.x.x.1 255.255.255.0 ip router isis 255 router isis 255 net 49.0001.0000.0000.0001.00 is-type level-1 log-adjacency-changes passive-interface Loopback0
Leaf_1#sh isis neighbors
Tag 255:
System Id Type Interface IP Address State Holdtime Circuit Id
Spine_1 L1 Gi1 10.1.1.1 UP 9 Spine_1.01
Spine_2 L1 Gi2 10.2.1.1 UP 7 Spine_2.01
Spine_3 L1 Gi3 10.3.1.1 UP 8 Spine_3.01
Spine_4 L1 Gi4 10.4.1.1 UP 9 Spine_4.01
Leaf_1#sh ip route
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 28 subnets, 2 masks
i L1 10.0.255.1/32 [115/10] via 10.1.1.1, 00:00:36, GigabitEthernet1
i L1 10.0.255.2/32 [115/10] via 10.2.1.1, 00:00:36, GigabitEthernet2
i L1 10.0.255.3/32 [115/10] via 10.3.1.1, 00:00:36, GigabitEthernet3
i L1 10.0.255.4/32 [115/10] via 10.4.1.1, 00:00:36, GigabitEthernet4
C 10.0.255.11/32 is directly connected, Loopback0
i L1 10.0.255.12/32 [115/20] via 10.4.1.1, 00:00:36, GigabitEthernet4
[115/20] via 10.3.1.1, 00:00:36, GigabitEthernet3
[115/20] via 10.2.1.1, 00:00:36, GigabitEthernet2
[115/20] via 10.1.1.1, 00:00:36, GigabitEthernet1
i L1 10.0.255.13/32 [115/20] via 10.4.1.1, 00:00:36, GigabitEthernet4
[115/20] via 10.3.1.1, 00:00:36, GigabitEthernet3
[115/20] via 10.2.1.1, 00:00:36, GigabitEthernet2
[115/20] via 10.1.1.1, 00:00:36, GigabitEthernet1
i L1 10.0.255.14/32 [115/20] via 10.4.1.1, 00:00:36, GigabitEthernet4
[115/20] via 10.3.1.1, 00:00:36, GigabitEthernet3
[115/20] via 10.2.1.1, 00:00:36, GigabitEthernet2
[115/20] via 10.1.1.1, 00:00:36, GigabitEthernet1
C 10.1.1.0/24 is directly connected, GigabitEthernet1
L 10.1.1.2/32 is directly connected, GigabitEthernet1
i L1 10.1.2.0/24 [115/20] via 10.1.1.1, 00:00:36, GigabitEthernet1
i L1 10.1.3.0/24 [115/20] via 10.1.1.1, 00:00:36, GigabitEthernet1
i L1 10.1.4.0/24 [115/20] via 10.1.1.1, 00:00:36, GigabitEthernet1
C 10.2.1.0/24 is directly connected, GigabitEthernet2
L 10.2.1.2/32 is directly connected, GigabitEthernet2
i L1 10.2.2.0/24 [115/20] via 10.2.1.1, 00:00:36, GigabitEthernet2
i L1 10.2.3.0/24 [115/20] via 10.2.1.1, 00:00:36, GigabitEthernet2
i L1 10.2.4.0/24 [115/20] via 10.2.1.1, 00:00:36, GigabitEthernet2
C 10.3.1.0/24 is directly connected, GigabitEthernet3
L 10.3.1.2/32 is directly connected, GigabitEthernet3
i L1 10.3.2.0/24 [115/20] via 10.3.1.1, 00:00:36, GigabitEthernet3
i L1 10.3.3.0/24 [115/20] via 10.3.1.1, 00:00:36, GigabitEthernet3
i L1 10.3.4.0/24 [115/20] via 10.3.1.1, 00:00:36, GigabitEthernet3
C 10.4.1.0/24 is directly connected, GigabitEthernet4
L 10.4.1.2/32 is directly connected, GigabitEthernet4
i L1 10.4.2.0/24 [115/20] via 10.4.1.1, 00:00:36, GigabitEthernet4
i L1 10.4.3.0/24 [115/20] via 10.4.1.1, 00:00:36, GigabitEthernet4
i L1 10.4.4.0/24 [115/20] via 10.4.1.1, 00:00:36, GigabitEthernet4
Then it’s time for the pretty forward Multicast part.
ip multicast-routing distributed interface Loopback0 ip pim sparse-mode interface GigabitEthernetx ip pim sparse-mode ip pim bidir-enable
ip multicast-routing interface Loopback0 ip pim sparse-mode interface Ethernet0/x ip pim sparse-mode ip pim bidir-enable ip pim bsr-candidate Loopback0 0 ip pim rp-candidate Loopback0 group-list Multicast bidir ! ip access-list standard Multicast permit 224.0.0.0 0.255.255.255
Leaf_1#sh ip pim neighbor
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable, G - GenID Capable,
L - DR Load-balancing Capable
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.1.1.1 GigabitEthernet1 06:38:11/00:01:38 v2 1 / B S P G
10.2.1.1 GigabitEthernet2 00:01:18/00:01:25 v2 1 / B S P G
10.3.1.1 GigabitEthernet3 00:01:18/00:01:25 v2 1 / B S P G
10.4.1.1 GigabitEthernet4 00:01:16/00:01:26 v2 1 / B S P G
Leaf_1#sh ip mroute
IP Multicast Routing Table
(*,224.0.0.0/8), 06:28:39/-, RP 10.0.255.1, flags: B
Bidir-Upstream: GigabitEthernet1, RPF nbr: 10.1.1.1
Incoming interface list:
GigabitEthernet4, Accepting/Sparse
GigabitEthernet3, Accepting/Sparse
GigabitEthernet2, Accepting/Sparse
Loopback0, Accepting/Sparse
GigabitEthernet1, Accepting/Sparse
## 224.0.1.4 = AUTO-RP-DISCOVERY
(*, 224.0.1.40), 06:34:41/00:02:19, RP 0.0.0.0, flags: DCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
GigabitEthernet4, Forward/Sparse, 00:02:08/00:02:09
GigabitEthernet2, Forward/Sparse, 00:02:08/00:02:05
GigabitEthernet3, Forward/Sparse, 00:02:09/00:02:05
GigabitEthernet1, Forward/Sparse, 06:34:41/00:02:19
Now it’s time for the VXLAN part
bridge-domain 1 member vni 74656 member GigabitEthernet6 service-instance 1 interface GigabitEthernet6 description to PC1 no ip address service instance 1 ethernet encapsulation untagged interface nve1 no ip address source-interface Loopback0 member vni 74656 mcast-group 224.1.2.3
nothing to do here ;) The spines are just used for IGP routing, multicast and forwarding the traffic from the leafs.
Leaf_1#show nve peers
Interface VNI Type Peer-IP Router-RMAC eVNI state flags UP time
nve1 74656 L2DP 10.0.255.13
nve1 74656 L2DP 10.0.255.12
nve1 74656 L2DP 10.0.255.14
Leaf_1#show bridge-domain 1
Bridge-domain 1 (2 ports in all)
State: UP Mac learning: Enabled
Aging-Timer: 300 second(s)
GigabitEthernet6 service instance 1
vni 74656
AED MAC address Policy Tag Age Pseudoport
0 0050.7966.6801 forward dynamic 267 nve1.VNI74656, VxLAN
src: 10.0.255.11 dst: 10.0.255.13
0 0050.7966.6800 forward dynamic 276 GigabitEthernet6.EFP1
0 0050.7966.6803 forward dynamic 271 nve1.VNI74656, VxLAN
src: 10.0.255.11 dst: 10.0.255.12
0 0050.7966.6802 forward dynamic 276 nve1.VNI74656, VxLAN
src: 10.0.255.11 dst: 10.0.255.14
To get a good packet capture via Wireshark i did disable the links from Leaf_1 to Spine2-4. Heres the screenshot from Wireshark:
And that’s it, VXLAN F&L on the CSR1000v is working!
Thanks here to Jon Major and his great Blogpost about VXLAN and CSR1kv