A Linux based firewall sandwich

While Linux has well documented server based load balancing features for both Layer-7 and Layer-4, there is little documentation on how one can make a firewall sandwich. Somehow it seems that this is a hard enough problem that even long time LVS contributor Roberto Nibali can be quoted to say on the topic “Buy a commercial load balancer and be done with it. Spend the spare time with your wife and kids or go to the pub with your buddies”. Problem is I don’t have a wife, and girlfriend is too far away so …

The 40,000ft. view

The following diagram (click to view in full size) captures the 40,000 feet view of the testbed in use. In short we have:

  1. a home-brew load generator simulating thousands of HTTP clients and servers (courtesy of my colleagues Alex and Leonidas)
  2. a couple of Solaris based firewalls
  3. a Linux load balancer running LVS in direct routing mode
  4. a Linux router

The firewalls setup

While the Solaris boxes run a proprietary dayjob product they can be treated as a regular firewall. That is assuming an HTTP request going through firewall-1, the HTTP response should also come back through firewall-1. In the event the response returns through firewall-2, firewall-2 will drop it altogether. Hence one should be able to reproduce the setup with an arbitrary linux firewall in place of the firewalls that:

  1. accepts new connections on the ingress interface (VLAN 162)
  2. reject packets on the egress interface (VLAN 165) that do not belong to an existing connection

Putting aside the above, the networking setup of the firewalls is pretty straightforward, configure the interfaces and add the routes to the HTTP client and server subnets.

firewall-1# more /etc/hostname.e1000g16*
firewall-1# cat /etc/inet/static_routes
# File generated by route(1M) - do not edit.

The load balancer

Linux Virtual Server was chosen for the load balancer setup. I installed Ubuntu 11.10 and then a couple of extra packages:

lvs# apt-get -y install vlan ipvsadm

Once done I configured the ingress interface to communicate with the HTTP clients:

lvs# tail -10 /etc/network/interfaces
iface eth1 inet manual
up ifconfig eth1 up

auto eth1.162
iface eth1.162 inet static
vlan_raw_device eth1

Then I proceeded to the load balancer setup. I want my firewalls to handle all web traffic (rather than a traffic to a specific VIP), so I had to use the firewall mark approach. I mark all HTTP related packets:

lvs# iptables -t mangle -N DIVERT
lvs# iptables -t mangle -A PREROUTING -i eth1.162 -p tcp --dport 80 -j DIVERT
lvs# iptables -t mangle -A DIVERT -j MARK --set-mark 1
lvs# iptables -t mangle -A DIVERT -j ACCEPT

then load balance them to my firewalls:

lvs# ipvsadm -A -f 1 -s sh
lvs# ipvsadm -a -f 1 -r -g -w 100
lvs# ipvsadm -a -f 1 -r -g -w 100

then make sure that my packets do get delivered to IPVS, even if their destination IP is not on a local interface (this caused a never-ending frustration till I found it):

lvs# ip rule add fwmark 1 lookup 100
lvs# ip route add local dev lo table 100

then add a route back to the HTTP clients via the appropriate interface (beats me why it’s needed, perhaps a Linux networking expert can explain why):

lvs# ip route add via

That’s it. Firing up my load generator I get a bunch of packets on eth1.162 which get load balanced to my two firewalls. The firewalls propagate the packet to the egress router, the egress router gets a response …

The egress router

And all it has to do is return the response via the originating firewall. It’s pretty much impossible to determine the originating load balancer through L4 criteria; sure enough if you use source-IP hashing on ingress you can do destination-IP hashing-load balancing on the egress. But what if you’re doing round robin distribution?

Towards this end the most simple way to solve this issue is via MAC persistence. If a request came through MAC-address “11:22:33:44:55:66” then return it through the same MAC. So all I need to do is write down the MAC addresses of the firewalls’ egress interfaces. Damn, I love iptables & iproute2:

egress-router# iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
egress-router# iptables -A PREROUTING -t mangle -m mark ! --mark 0 -j ACCEPT
egress-router# iptables -t mangle -A PREROUTING -i eth2.165 -m mac --mac-source 00:0c:29:ef:a3:29 -j MARK --set-mark 1
egress-router# iptables -t mangle -A PREROUTING -i eth2.165 -m mac --mac-source 00:0c:29:c7:93:96 -j MARK --set-mark 2
egress-router# iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark
egress-router# echo "101 firewall1" >> /etc/iproute2/rt_tables
egress-router# echo "102 firewall2" >> /etc/iproute2/rt_tables
egress-router# ip rule add fwmark 1 table firewall1
egress-router# ip rule add fwmark 2 table firewall2
egress-router# ip route add default via table firewall1
egress-router# ip route add via table firewall1
egress-router# ip route add default via table firewall2
egress-router# ip route add via table firewall2

What does the above snippet do:

  1. it uses a separate connection mark for traffic coming from each firewall; mark-1 for firewall-1, mark-2 for firewall-2
  2. it adds a couple of iproute2 routing tables, one for each firewall
  3. it adds a couple of iproute2 rules to look up the appropriate routing table, depending on the connection mark
  4. it adds all appropriate routes to the firewall specific routing tables

That’s it!

The above are enough to make the firewall sandwich work. Sure enough my load generator reports no errors, my firewall logs report traffic in both firewall with no errors, ipvsadm reports requests being load balanced:

lvs# ipvsadm -l --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
FWM 1 674 9579 0 609892 0
-> 367 3853 0 261303 0
-> 307 5726 0 348589 0

Everything looks great!

What’s missing

Actually a lot. This was just a prototype to make a point to a C-level exec. Notably missing are server healthchecking and loadbalancing failover. But these are problems well understood with a robust solution one can easily google for.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: