LVS/TUN mode with FreeBSD and Solaris realserver

Revision as of 08:05, 16 August 2006 by Maluyao (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

LVS/TUN mode with FreeBSD and Solaris

Author: Ma Luyao RHCE maluyao at gmail dot com


The Linux Virtual Server Project (LVS) is a project started by Dr. Wensong Zhang. LVS implements three load balance technology.

Network Address Translation (VS/NAT) Direct Routing (VS/DR) IP Tunneling (VS/TUN)

VS/NAT is easy to setup. The load balancer may be a bottleneck of the whole system where the number of servers is more than 20, because both the request packets and response packets need to be rewritten by the load balancer.

VS/DR has the best performance. VS/DR uses MAC-spoofing technology, so it requires that one of the load balancer's NIC and the real server's NIC must be in the same IP network segment, and physical segment as well.

VS/TUN is the most scalable. The advantage is since servers connect each other by using IP-tunneling, the load balance and real servers can reside on different LAN , or even WAN.

I plan to setup a set of servers with load balance technique. Realserver run FreeBSD and Solaris system in different places on our business. So it is the only choice for us to use VS/TUN.

According to the official document of LVS, FreeBSD and Solaris system can not be a realserver, but I did do it after some hard testing.

We install Fedora Core 5 Linux on our Load Balance server. Kernel-2.6.15 is default.

FreeBSD version is 5.4, we recompile the kernel in order to improve the performance. The default kernel can work too.

Solaris box runs Solaris 10 x86.

Normal Linux Solaris IP-tunneling

IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram, which allows datagram destined for one IP address to be wrapped and redirected to another IP address. Many modern OS support IP tunneling, such as Linux, FreeBSD and Solaris.

Let's do some test to setup normal Linux-Solaris and Linux-FreeBSD IP-tunneling without LVS. OS | IP Address | Tunnel IP Address

Fedora Core 5 Linux | | Solaris 10 x86 | | FreeBSD 5.4 | |

Config Linux box

In order to create an IP-Tunneling between Linux box and Solaris box, run command:

ip tunnel add tun0 mode ipip remote local ifconfig tun0 pointopoint ip tunnel add tun1 mode ipip remote local ifconfig tun1 pointopoint

The "tun0" is a virtual NIC device name. It can be replaced by other names, such as "abc0" or "def0".

Config Solaris box

To create an IP-tunneling, we have two methods, First method of creating tunnel is using three commands as below.

/sbin/ifconfig ip.tun0 plumb /sbin/ifconfig ip.tun0 tsrc tdst /sbin/ifconfig ip.tun0

It takes effect immediately.

Second method is creating /etc/hostname.ip.tun0 file which contains two lines as below.

tsrc tdst netmask up

If we use the second method, the tunnel will exist automatically after we reboot system.

Config FreeBSD box On FreeBSD box, we also have two methods to create an IP-Tunneling, one way is to run three commands:

ifconfig gif0 create ifconfig gif0 tunnel ifconfig gif0 inet netmask

If we use the second method, the tunnel will exist automatically after we reboot system.

Add three lines on /etc/rc.conf file as below, the tunnel will exist automatically after we reboot system.

gif_interfaces="gif0" gifconfig_gif0="" ifconfig_gif0=" netmask"

Using IP-tunneling

In all the cases on Linux/FreeBSD/Solaris, the netmask value will be default if we ignore to set it.

After creating two tunnels, we can ping and at Linux box successfully, and ping at Solaris or FreeBSD box successfully. All applications layers protocol will work correctly.

Working example of VS/TUN

In VS/TUN mode, because realserver don't send any datagram to Load balancer, we need not create any tunnel on Linux box and must create proper tunnel on FreeBSD/Solaris box.

Topological graph


We can treat all machines on internet and have their real ip. The client can send their requests to VIP of LB Server. LB Server redirects the requests realserver through IP-tunneling. Realserver can send datagram to client.

Because both the gif0 of FreeBSD and ip.tun0 both are NO-ARP device, they are invisible to client.

Config FreeBSD box On FreeBSD box, edit /etc/rc.conf as below:

... ifconfig_lnc0="inet netmask 0xffffff00" gif_interfaces="gif0" gifconfig_gif0="" ifconfig_gif0=" netmask 0xffffffff" apache_enable="YES" ... looks like a terminal of this tunnel. It can be any idle IP and never use.

Config Solaris box

On Solaris , edit /etc/hostname.ip.tun0 as below:

tsrc tdst netmask up looks like a terminal of this tunnel. It can be any IP address and never use it.

Config Load balancer

Run ipvsadm command on Linux Box: Linux kernel must have ip_forward enable. Edit /etc/sysctl.conf and set to:

net.ipv4.ip_forward = 1

While running command:

sysctl -p

It will take effect.

Run the scripts:

ipvsadm -C ipvsadm -A -t -s wlc ipvsadm -a -t -r -i ipvsadm -a -t -r -i

In this script, the port of LB Server and realerver must be same. Good idea is not setting the port of realserver.

Now VS/TUN is finished. When a client access at port 80, the datagram will be sent to or 192.168.180 port 80.

Conclusions and Future Work

I believe AIX and HP-UX can also work with VS/TUN, I have no Aix or HP-UX servers. Who can help to provide me with such an environment?