1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
|
= Setting up active-active load-sharing hash-based stateful firewall =
by Pablo Neira Ayuso <pablo@netfilter.org> in 2010
If you want to know more about this configuration and other firewall
architectures, please read:
* Demystifying cluster-based fault-tolerant firewalls.
IEEE Internet Computing, 13(6):31-38, December 2009.
Available at: https://perso.ens-lyon.fr/laurent.lefevre/pdf/IC2009_Neira_Gasca_Lefevre.pdf
== 0x0 intro ==
Under this directory you can find a script that allows you to setup a simple
active-active hash-based load-sharing firewall cluster based on the iptables'
cluster match.
== 0x1 testbed ==
My testbed looks like the following:
---------- eth1 eth2 ----------
client A ------| |--- firewall 1 ----| |
(192.168.0.2) | switch | (.0.5) (.1.5) | switch |--- server
| | | | (192.168.1.2)
client B ------| |--- firewall 2 ----| |
(192.168.0.11) ---------- (.0.5) (.1.5) ----------
eth1 eth2
The firewalls perform SNAT to masquerade clients. Note that both cluster
firewall have the same IP addresses. For administrative purposes, it is
a good idea that each firewall has its one IP address to SSH them, make
sure you add the appropriate rule to skip the cluster match rule-set!
More comments: although the picture shows two switches, I'm actually
using one and I separated the clients and the server in two different
VLANs.
The script also sets a multicast MAC address that is the same for both
firewalls so that the switch floods the same packets to both firewalls.
Using a multicast MAC address is a RFC violation [1], since network node
must not include multicast MAC address in ARP replies, but:
a) it is the only way I found so far to obtain the behaviour from my
HP procurve switches.
b) the VRRP MAC address range is not supported appropritely by switch
vendors, at least by my HP procurve switches. If switch vendors
support this MAC address range appropriately, they will handle them
as multicast MAC address. As of 2011 I did not find any switch handling
VRRP MAC address range as multicast ports (they still handle them as
normal unicast MAC addresses, therefore my solution does not work with
two nodes with the same VRRP MAC address).
The cluster match relies upon the Connection Tracking System (conntrack).
Thus, traffic coming in the reply direction which does not belong this node
is labeled as INVALID for TCP and ICMP protocols. The scripts add a rule to
drop this traffic to avoid possible packet duplication. For UDP traffic,
you will have to add a rule to drop NEW traffic in the reply direction
because conntrack considers it valid. If you don't do this, both nodes
may accept reply traffic, thus, sending duplicated packets to the client,
which is not what you want.
During my last experiments, I was using the Linux kernel 2.6.37 in the
firewalls and the server. Everything you need to setup this configuration
is available in stock Linux kernels. No external patches with new features
are required.
== 0x2 running scripts ==
Copy the script to each node, then adjust the script variables to your
configuration.
On firewall 1:
firewall1# ./clusterip-node1.sh start
On firewall 2:
firewall2# ./clusterip-node2.sh start
== 0x3 trouble-shooting ==
Some troubleshooting may help to understand how this setup works. Check
the following if you experience problems:
1) Check that Multicast MAC address are assigned to the NICs:
firewall1$ ip maddr
[...]
2: eth1
[...]
link 01:00:5e:00:01:01 static
3: eth2
[...]
link 01:00:5e:00:01:02 static
The scripts add the multicast MAC addresses to the NICs, if this
is not done the traffic will be discarded by the firewalls'
networking stack.
2) ICMP ping the server from one the clients:
client$ ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=63 time=0.220 ms
--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.220/0.220/0.220/0.000 ms
If this does not work, make sure the firewalls are including the
multicast MAC address in their ARP replies, you can check this
by looking at the neigbour cache:
client$ ip neighbour
[...]
192.168.0.5 dev eth1 lladdr 01:00:5e:00:01:01 REACHABLE
server$ ip neighbour
[...]
192.168.1.5 dev eth1 lladdr 01:00:5e:00:01:02 REACHABLE
firewall$ ip neighbour
[...]
192.168.0.5 dev eth1 lladdr 01:00:5e:00:01:01 REACHABLE
192.168.1.5 dev eth2 lladdr 01:00:5e:00:01:02 REACHABLE
3) Test TCP connections: you can use netcat to start simple connections
between the client and the server.
You can also use intensive HTTP traffic generation to test performance
like injectX.c and httpterm from Willy Tarreau:
http://1wt.eu/tools/inject/
http://1wt.eu/tools/httpterm/
clientA:~/http-client-benchmark# ./client -t 60 -u 200 -G 192.168.1.2:8000
# hits hits/s ^h/s ^bytes kB/s errs rst tout mhtime
266926 26692 26766 3881270 3779 0 0 0 0.237
294067 26733 27141 3935621 3785 0 0 0 0.176
clientB~/http-client-benchmark# ./client -t 30 -u 40 -G 192.168.1.2:8020
# hits hits/s ^h/s ^bytes kB/s errs rst tout mhtime
53250 17750 17368 2518448 2513 0 0 0 0.240
70766 17691 17516 2539907 2505 0 0 0 0.297
^h/s is the current number of HTTP petitions per second. This means
that you get ~45000 HTTP petitions per second. In my setup, with only
one firewall active I get ~27000 HTTP petitions per second. We obtain
extra performance of ~66%, not that bad 8-).
I have configured httpterm to send object of 0 bytes over HTTP
to obtain the maximum number of HTTP flows. This is the worst case
scenario in firewall load.
I forgot to mention that I set CPU affinity for NICs IRQs. I've got
two cores, one for each firewall NIC.
== 0x4 report sucessful setups ==
My testbed is composed of low-cost basic five years old HP proliant
systems, you can see that the numbers are not great. I like knowing
about numbers, I'd appreciate if you drop me a line to tell me the
numbers that you get and your experience.
== 0x5 conclusions and future works ==
The cluster match allows to setup load-sharing hash-based stateful
firewalls that is a way to avoid having a spare backup firewall as
it happens in classical Primary-Backup setups.
Still, there is some pending work to fully integrate conntrackd and HA
managers with it (in case that you want high availability, of course).
-o-
[1] More specifically, it's a RFC 1812 (section 3.3.2) violation.
It's been reported that this is a problem for CISCO routers:
http://marc.info/?l=netfilter&m=128810399113170&w=2
Michele Codutti: "The problem is the multicast MAC address that these
routers doesn't "like". They discard any incoming packet with MAC
multicast address to be compliant with RFC1812. The only documented
(by Cisco) workaround is to put a fixed arp entry with the multicast
address that maps the clustered IP in the router."
If you keep reading the mailing thread, the reported problem affected
Cisco 7200 VXR.
--02/02/2010
|
