Background
Free Range Routing is a really cool fork of the Quagga project by Cumulus Linux, both are implementations of a number of network routing protocols for Linux. While Quagga implemented the classic protocols such as OSPF, BGP, RIP, and PIMD that have been in use for decades, FRR implements some new ones and adds features to the old protocols.
Free Range Routing is a bit tricky to build from source at the moment (February 2020) due to some newer packages such as libyang not being readily available in major distributions such as Ubuntu. However Ubuntu’s Snap technology makes installing a very recent version of Free Range Routing, well, a snap. I have another post that shows you how to install it and run EIGRP on Linux which is pretty cool because it’s Cisco proprietary (yes, Cisco still owns it despite publishing an RFC).
Another protcol that Free Range Routing implements is NHRP, which is the key component in another Cisco-originated technology, Dynamic Multipoint Virtual Private Network, or DMVPN for short. NHRP serves as the protocol that determines GRE tunnel endpoints.
For this post, I’m going to skip the encryption part of DMVPN that IPsec provides. It’s done with Strongswan on Linux, and it’s tricky to get right and I’ll cover it in another post. But you can get a nice unencrypted overlay network running with NHRP and GRE before you ever start encrypting. I don’t think there’s an official term for DMVPN without IPsec, but I’ve heard networking folk refer to it as “naked” DMVPN. Obviously you don’t want to do this on the Internet (despite the “Internet” node in my topology), but there are some viable use cases for naked DMVPN on a private network and I’ve personally seen it on production networks. Free Range Routing’s NHRP implementation is still experimental, so please be careful.
Topology
I don’t like how cluttered this drawing is, but I’m pretty sure all the info on there is relevant to the lab. The idea here is to get PC’s 1, 2 and 3 to be able to ping each other. The 2 Ubuntu routers and 1 Cisco router can’t establish routing with “The Internets” so they’ll need a tunneling mechanism to learn about the networks of the other routers that have PC’s connected. First I’ll get NHRP connected and get GRE tunnels working. Then I’ll run BGP to advertise 192.168.0.0/24, 172.16.0.0/24 and 172.17.0.0/24 over the “overlay” GRE network.
Configuration
This post assumes the basic IP connectivity between the Ubuntu and Cisco routers has been set up, as well as LAN connectivity for the PC’s. Specifically, networks 11.0.0.0/30, 22.0.0.0/30, 33.0.0.0/30, 192.168.0.0/24, 172.16.0.0/24, and 172.17.0.0/24 have all been configured. With all that boring stuff done, we’ll jump into more interesting things.
Ubuntu18.04-FRR-1
This is the DMVPN hub. First order of business is to create a GRE tunnel interface:
$ip tunnel add james_gre mode gre key 42 ttl 64 dev eth0 $ip addr add 10.0.0.1/32 dev james_gre $ip link set james_gre up
You might be wondering why the GRE IP address is 10.0.0.1/32 and not 10.0.0.1/24. Well apparently it has something to do with the way the good folks at Cumulus Linux implemented it. I’m sure they had their reasons, and they outline a couple in their documentation and on their Github NHRP readme page. That’s just the way it’s done with FRR, so I don’t argue.
Next we need to jump into the FRR CLI and configure NHRP. Remember to type the full path as it didn’t get added to $PATH when FRR was installed (or you can add it to your $PATH, or cd to /snap/bin):
$/snap/bin/frr.vtysh Hello, this is FRRouting (version 7.2.1). Copyright 1996-2005 Kunihiro Ishiguro, et al. # conf t (config)#int james_gre (config-if)#ip nhrp network-id 1 (config-if)#ip nhrp nhs dynamic nbma 11.0.0.1 (config-if)#ip nhrp registration no-unique (config-if)#exit (config)#
I’ve skipped setting up “NHRP redirect” (direct spoke-to-spoke communication) as I was unable to get it working in my lab between a Cisco spoke and Ubuntu spoke. I can make it work between two Ubuntu spokes or two Cisco spokes but that’s not really much fun. As I said this implementation is still a work in progress.
Next I’ll set up BGP routing to advertise the networks where the PC’s are connected. I’m using BGP because it lends itself to DMVPN for a number of reasons and also FRR’s NHRP implementation doesn’t support multicast yet, so no EIGRP or OSPF. The Hub will serve as a BGP “route reflector”, which basically means you can avoid a full mesh of BGP neighborships in iBGP (BGP where the AS is all the same). Sounds fancy but it actually simplifies things here, I promise.
(config)#router bgp 1 (config-router)#neighbor 10.0.0.2 remote-as 1 (config-router)#neighbor 10.0.0.3 remote-as 1 (config-router)#address-family ipv4 unicast (config-router-af)#network 192.168.0.0/24 (config-router-af)#neighbor 10.0.0.2 route-reflector-client (config-router-af)#neighbor 10.0.0.3 route-reflector-client #
Ubuntu18.04-FRR-2
The Ubuntu spoke is pretty similar, except without any route reflector stuff.
$ip tunnel add james_gre mode gre key 42 ttl 64 dev eth0 $ip addr add 10.0.0.2/32 dev james_gre $ip link set james_gre up $/snap/bin/frr.vtysh Hello, this is FRRouting (version 7.2.1). Copyright 1996-2005 Kunihiro Ishiguro, et al. # conf t (config)#int james_gre (config-if)#ip nhrp network-id 1 (config-if)#ip nhrp nhs dynamic nbma 11.0.0.1 (config-if)#ip nhrp registration no-unique (config-if)#exit (config)#ip route 10.0.0.0/24 10.0.0.1 (config)#router bgp 1 (config-router)#neighbor 10.0.0.1 remote-as 1 (config-router)#address-family ipv4 unicast (config-router-af)#network 172.16.0.0/24 (config-router-af)#exit #
You may have noticed an additional command – I added a static route via “10.0.0.0/24 10.0.0.1”. This was for compatibility with the Cisco device. I could avoid this step and use NHRP redirect with another Ubuntu spoke and they would communicate directly. However with a Cisco spoke in play I had to add the route and all traffic goes through the hub. Maybe it’ll be better next release. Or if there’s a configuration I’m missing, please comment or shoot me an email, I’d love to know because I practically broke my keyboard trying different configurations.
CiscoIOSv15.6(2)T-1
This is the tunnel and BGP configuration for my Cisco IOSv router:
interface Tunnel0 ip address 10.0.0.3 255.255.255.0 no ip redirects ip nhrp network-id 1 ip nhrp nhs dynamic nbma 11.0.0.1 ip nhrp registration no-unique ip nhrp shortcut tunnel source GigabitEthernet0/0 tunnel mode gre multipoint tunnel key 42 ! router bgp 1 bgp log-neighbor-changes network 172.17.0.0 mask 255.255.255.0 neighbor 10.0.0.1 remote-as 1 !
You probably noticed that the IP address on the tunnel interface is not /32 like the Ubuntu routers, it’s a /24 which is ultimately the size of the DMVPN network. That’s the way Cisco does it, so I don’t argue.
Verification
You can run some useful show commands from the FRR CLI on the Ubuntu routers as well as the Cisco CLI. “show ip route” is really useful, you’ll notice “N” routes on the Ubuntu routers which is how FRR NHRP advertises other spoke IP addresses (Read the FRR docs for more detail):
# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 11.0.0.2, eth0, 1d01h01m
C>* 10.0.0.1/32 is directly connected, james_gre, 1d00h57m
N>* 10.0.0.2/32 [10/0] is directly connected, james_gre, 02:56:29
N>* 10.0.0.3/32 [10/0] is directly connected, james_gre, 12:37:57
C>* 11.0.0.0/30 is directly connected, eth0, 1d01h02m
B> 172.16.0.0/24 [200/0] via 10.0.0.2 (recursive), 01:17:14
* via 10.0.0.2, james_gre onlink, 01:17:14
B> 172.17.0.0/24 [200/0] via 10.0.0.3 (recursive), 01:16:53
* via 10.0.0.3, james_gre onlink, 01:16:53
C>* 192.168.0.0/24 is directly connected, eth1, 12:10:26
#
You can also run “show dmvpn” which should show the other endpoints:
# show dmvpn Src Dst Flags SAs Identity <br> 11.0.0.1 22.0.0.1 n 0 <br> 11.0.0.1 33.0.0.1 n 0
And a good old fashioned ping should confirm everything for you. Here I’m pinging from the PC behind the Ubuntu spoke to the PC behind the Cisco spoke:
PC2> ping 172.17.0.2 84 bytes from 172.17.0.2 icmp_seq=1 ttl=62 time=3.523 ms 84 bytes from 172.17.0.2 icmp_seq=2 ttl=62 time=3.828 ms
Troubleshooting
You can apparently send NHRP debugs to syslog, but I didn’t find them particularly helpful when I was troubleshooting. I found a packet capture to be the most useful. The the key is to get NHRP going, in a packet capture you should see a NHRP registration request and reply with Code=Success. If you see registration requests with no request at all, you have a problem. On my cisco I originally forgot to put in the tunnel key command, and it kept sending a request and getting an ICMP error response. Don’t forget your tunnel key.
DMVPN between Ubuntu and Cisco, pretty cool right?
Hi,
I see you noted that you got NHRP redirect working between multiple ubuntu spokes.
Can you share how you got this to work as I’m very much struggling with this?
HI Alistair! Thanks for visiting and sorry for the slow reply – can you show me your configuration? I’d be glad to help any way I can 🙂
Pretty sure it was as simple as “ip nhrp redirect” in the tunnel config in frr.vtysh
Hi
Playing second day with it.
tunnels are UP.
NHRP working.
Trying to play with eigrp on frr – no luck. Maybe you know some trick for eigrp on frr (like /32 on gre tunnel) ?
my config:
interface gre1
ip hello-interval eigrp 30
ip hold-time eigrp 90
ip nhrp mtu 1400
ip nhrp network-id 123456
ip nhrp nhs dynamic nbma 172.26.5.43
ip nhrp registration no-unique
tunnel source eth0
!
router eigrp 1
network 192.168.168.192/27
Unfortunately that is a limitation of FRR for now – you will not be able to get routing protocols that use multicast to work over GRE, and DMVPN uses GRE for its tunneling mechanism. Should have mentioned that at the top. There is a Github issue logged here https://github.com/FRRouting/frr/issues/6276. The reason I used BGP in the lab here is that it’s technically a Layer 7 protocol – it uses a tcp session to send its messages and everything is unicast. I’m hoping FRR will support multicast over GRE soon, maybe I’ll see if I can find where they need help in their implementation and contribute some code. It’s open source after all 🙂
Thanks James.
Understood.
so far only strongswan left to add to this setup.
Agreed! I’ll see if I can get it working, and put my steps in a new post. It requires a special build of strongswan, which is why I didn’t include it in this post.
Hi!
Have you tried using Cisco as a hub in this scheme?
No I haven’t – it’s on my to-do list 🙂
Have you ever used a Cisco router as the hub? I am unable to get it to work.
Haven’t! I will see if I can give it a try. Thanks for commenting 🙂