EVPN: All roads lead to the firewall

Fifi · telcokwaks.com ·

8 min read Original article ↗

Introduction

In the past, I have worked with proprietary firewalls (Stormshield, Arkoon, Netasq). The experience was rather mixed: the whole thing is hard to debug, updates are gated behind licenses, prices are excessive, and there are many technical limitations.

So many reasons to rule out proprietary gear for this kind of design, in favor of a Linux-based solution (FRR, BGP, nftables): open source, innovation, and independence.

The idea: a hub-and-spoke design where the firewalls play the role of hub, and nobody talks to anyone without going through them.

Overview

Here is a simplified diagram of this design:

evpn-vrf-route-propagation

On the network’s routers, we instantiate a hub services VRF (HUBSVC in the configuration). This VRF holds only the default routes originated by the firewalls.

A second VRF is present: the spoke VRF. It serves as the home VRF for the client interfaces we want to isolate. It lets us export, over EVPN, the various destinations reachable through the firewall. When machines in this VRF want to talk to other machines, their traffic is drawn in thanks to a leak of the hub services default routes.

In our diagram, the interface between r1 and riri is in the spoke VRF: when riri wants to communicate with loulou, it follows the hub services default route.

The firewall brings up IPv4 and IPv6 unicast eBGP sessions for spoke services (SPKSVC) and for hub services: EVPN stops at the spine. We could have used a single EVPN session between the firewall and the spine; keeping separate unicast sessions decouples network management from firewall management, for instance when different teams handle each.

Here is an example of traffic between riri and loulou:

evpn-hub-spoke-firewall

For redundancy, this design can include several firewalls: the return traffic may then not take the same firewall as the outbound path. This kind of case rules out stateful firewalling: you would have to either synchronize conntrack across all firewalls (with conntrackd), or give up on state. The second option is the one chosen here, detailed in the Filtering section.

evpn-hub-spoke-firewall-redondance

The EVPN mechanics

The whole isolation trick lies in the route-targets of the spoke VRF. Here is the relevant FRR configuration on r1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

! spoke VRF: hosts the client interfaces, bound to VNI 110 (L3VNI)
vrf spoke
 vni 110
exit-vrf
!
! BGP instance for the spoke VRF, in the fabric AS (65010)
router bgp 65010 vrf spoke
 !
 address-family ipv4 unicast
  ! advertise the directly connected networks (the client subnets)
  redistribute connected
  ! import the default from the HUBSVC VRF, filtered by the route-map (0.0.0.0/0 only)
  import vrf route-map LEAK_DEFAULT4
  import vrf HUBSVC
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
  ! same in IPv6: only ::/0 is allowed to leak from HUBSVC
  import vrf route-map LEAK_DEFAULT6
  import vrf HUBSVC
 exit-address-family
 !
 address-family l2vpn evpn
  rd 192.0.2.3:110
  ! sentinel import RT: matches nothing, so r1 learns no other spoke
  route-target import 65010:65000
  ! export RT: used to generate the spoke services VRF
  route-target export 65010:110
  ! advertise the VRF's IPv4/IPv6 prefixes into EVPN (type-5 routes)
  advertise ipv4 unicast
  advertise ipv6 unicast
 exit-address-family
!
ip prefix-list DEFAULT4 seq 5 permit 0.0.0.0/0
ipv6 prefix-list DEFAULT6 seq 5 permit ::/0
!
route-map LEAK_DEFAULT4 permit 10
 match ip address prefix-list DEFAULT4
!
route-map LEAK_DEFAULT6 permit 10
 match ipv6 address prefix-list DEFAULT6

Three things to note:

  • The spoke VRF exports its networks with route-target 65010:110. The spine imports them into a spoke services VRF and advertises them over eBGP to the firewall: the firewall therefore knows every spoke destination.
  • The spoke VRF imports 65010:65000, a “sentinel” route-target that deliberately matches nothing. Without it, FRR would automatically import its own export route-target, and each spoke would learn the other spokes’ networks. This line is what guarantees there is no direct route between r1 and r2.
  • The default route comes in via import vrf HUBSVC, filtered by a route-map that only lets the default through: 0.0.0.0/0 in IPv4 (LEAK_DEFAULT4) and ::/0 in IPv6 (LEAK_DEFAULT6).

As a result, on r1, the spoke VRF’s routing table is minimal:

1
2
3
4
5
6
7
8
9

r1# show ip route vrf spoke
VRF spoke:
B>* 0.0.0.0/0 [20/0] via 192.0.2.1 (vrf HUBSVC) onlink, weight 1, 00:13:37
C>* 198.51.100.0/28 is directly connected, eth2, 00:13:42

r1# show ipv6 route vrf spoke
VRF spoke:
B>* ::/0 [20/0] via 2001:db8:ffff::1 (vrf HUBSVC) onlink, weight 1, 00:13:37
C>* 2001:db8:a::/64 is directly connected, eth2, 00:13:42

loulou’s networks (198.51.100.16/28 and 2001:db8:b::/64) appear nowhere: the only way out is the default route, and it leads to the firewall.

Filtering

Here, the choice is to work with stateless filtering only. This lets us add as many firewalls as we want, without deploying any conntrack synchronization: since the return traffic may go through a different firewall than the outbound path, stateful filtering would force us to replicate state across every machine.

The downside is that each rule has to be written in both directions: swapping sources / destinations for the return traffic.

Chaining nftables rules the classic way quickly runs into performance issues, with significant jitter: each packet walks the chain rule by rule, and the processing time depends on the position of the matching rule.

Conversely, nftables sets (the successors to ipset) are far better suited to this need: the lookup cost stays nearly constant, whatever the number of entries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

root@fw1:~# nft list ruleset
table inet filter {
	set setipv4 {
		type ipv4_addr . ipv4_addr . inet_service
		flags interval
		elements = { 198.51.100.0/28 . 198.51.100.16/28 . 443,
			     198.51.100.0/28 . 198.51.100.16/28 . 22 }
	}

	set setipv6 {
		type ipv6_addr . ipv6_addr . inet_service
		flags interval
		elements = { 2001:db8:a::/64 . 2001:db8:b::/64 . 443,
			     2001:db8:a::/64 . 2001:db8:b::/64 . 22 }
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
		icmp type { echo-request, echo-reply } accept
		icmpv6 type { echo-request, echo-reply } accept
		ip saddr . ip daddr . tcp dport @setipv4 accept
		ip daddr . ip saddr . tcp sport @setipv4 accept
		ip6 saddr . ip6 daddr . tcp dport @setipv6 accept
		ip6 daddr . ip6 saddr . tcp sport @setipv6 accept
	}
}

The first two rules allow ICMP and ICMPv6: handy for reachability tests. The rest of the filtering goes through the sets, one per address family.

A TCP connection from riri to loulou on port 443 generates two frames with symmetric headers:

1
2
3

          source        destination
outbound  riri:40000  →  loulou:443
return    loulou:443  →  riri:40000

The set stores only one orientation, the outbound one: riri . loulou . 443 (client, server, server port). The ephemeral port (40000) is not in it.

So the return frame lands on that same entry, the second rule reads its fields in reverse:

1
2

outbound  ip saddr . ip daddr . tcp dport  =  riri . loulou . 443
return    ip daddr . ip saddr . tcp sport  =  riri . loulou . 443

The saddr and daddr fields are swapped (source and destination trade places on the way back), and we match on sport instead of dport (the server port moves from destination to source). A single set entry therefore covers both directions.

Two sets, a few fixed rules, and the forward chain never grows: only the sets change.

Another operational advantage: sets can be modified on the fly, without reloading the ruleset.

1

root@fw1:~# nft add element inet filter setipv4 { 198.51.100.0/28 . 198.51.100.16/28 . 5432 }

Verification

From riri, we check that loulou is reachable:

1
2
3
4
5

riri:~# ping -c1 198.51.100.18
64 bytes from 198.51.100.18: icmp_seq=1 ttl=61 time=1.42 ms

riri:~# ping -c1 2001:db8:b::18
64 bytes from 2001:db8:b::18: icmp_seq=1 ttl=61 time=1.51 ms

And on the firewall, a tcpdump shows the traffic does go through it, in both directions:

1
2
3
4
5
6
7
8
9

root@fw1:~# tcpdump -ni any 'icmp or icmp6'
eth1  In  IP 198.51.100.2 > 198.51.100.18: ICMP echo request, id 42, seq 1
eth2  Out IP 198.51.100.2 > 198.51.100.18: ICMP echo request, id 42, seq 1
eth1  In  IP 198.51.100.18 > 198.51.100.2: ICMP echo reply, id 42, seq 1
eth2  Out IP 198.51.100.18 > 198.51.100.2: ICMP echo reply, id 42, seq 1
eth1  In  IP6 2001:db8:a::2 > 2001:db8:b::18: ICMP6, echo request, id 43, seq 1
eth2  Out IP6 2001:db8:a::2 > 2001:db8:b::18: ICMP6, echo request, id 43, seq 1
eth1  In  IP6 2001:db8:b::18 > 2001:db8:a::2: ICMP6, echo reply, id 43, seq 1
eth2  Out IP6 2001:db8:b::18 > 2001:db8:a::2: ICMP6, echo reply, id 43, seq 1

The packet comes in through the hub services VRF (the default route) and leaves through the spoke services VRF. If the nftables rule does not exist, the packet dies here: policy drop.

Conclusion

With this setup, all traffic between machines is drawn to the firewalls. Redundancy and scaling are easy.

All of it with Linux, FRR, and nftables. Zero proprietary appliances, zero licenses, and debugging done with the tools we already know. And when something is missing or blocks us, we can read the code and contribute, rather than opening a ticket with a vendor and waiting.

Deploying the configuration stays simple: no need for proprietary controllers or web interfaces to manage a large number of devices.

A complete containerlab lab reproducing this design (dual-stack IPv4/IPv6) is available on GitHub.

Further reading