Difference between revisions of "LVS/TUN mode with FreeBSD and Solaris realserver"

Revision as of 08:47, 18 March 2008

Author: Ma Luyao RHCE maluyao at gmail dot com

August 16, 2006

Introduction

The Linux Virtual Server Project (LVS) is a project started by Dr. Wensong Zhang. LVS implements three load balance technology.

a. Network Address Translation (VS/NAT)

b. Direct Routing (VS/DR)

c. IP Tunneling (VS/TUN)

VS/NAT is easy to setup. The load balancer may be a bottleneck of the whole system where the number of servers is more than 20, because both the request packets and response packets need to be rewritten by the load balancer.

VS/DR has the best performance. VS/DR uses MAC-spoofing technology, so it requires that one of the load balancer’s NIC and the real server’s NIC must be in the same IP network segment, and physical segment as well.

VS/TUN is the most scalable. The advantage is that since servers connect each other by using IP-tunneling, the load balancer and the real servers can reside on different LAN , or even WAN.

I plan to setup a set of servers with load balancing technique. Real servers run FreeBSD and Solaris system in different places on our business. So it is the only choice for us to use VS/TUN.

We install Fedora Core 5 Linux on our Load Balance server. Kernel-2.6.15 is default.

FreeBSD version is 5.4, we recompile the kernel in order to improve the performance. The default kernel can work too.

Solaris box runs Solaris 10 x86.

Normal Linux Solaris/FreeBSD IP-tunneling

IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram, which allows datagram destined for one IP address to be wrapped and redirected to another IP address. Many modern OS support IP tunneling, such as Linux, FreeBSD and Solaris.

Let’s do some test to setup normal Linux-Solaris and Linux-FreeBSD IPtunneling without LVS.

Testing environment

Config Linux box

In order to create an IP-Tunneling between Linux box and Solaris box, run command:

ip tunnel add tun0 mode ipip remote 10.0.0.2 local 10.0.0.1

ifconfig tun0 172.16.0.1 pointopoint 172.16.0.2

ip tunnel add tun1 mode ipip remote 10.0.0.3 local 10.0.0.1

ifconfig tun1 172.16.0.1 pointopoint 172.16.0.3

The ”tun0” is a virtual NIC device name. It can be replaced by other names, such as ”abc0” or ”def0”.

Config Solaris box

To create an IP-tunneling, we have two methods, First method of creating tunnel is using three commands as below.

/sbin/ifconfig ip.tun0 plumb

/sbin/ifconfig ip.tun0 tsrc 10.0.0.2 tdst 10.0.0.1

/sbin/ifconfig ip.tun0 172.16.0.2 172.16.0.1

It takes effect immediately.

Second method is creating /etc/hostname.ip.tun0 file which contains two lines as below.

tsrc 10.0.0.2 tdst 10.0.0.1

172.16.0.2 172.16.0.1 netmask 255.255.255.255 up

If we use the second method, the tunnel will exist automatically after we reboot system.

Config FreeBSD box

On FreeBSD box, we also have two methods to create an IP-Tunneling, one way is to run three commands:

ifconfig gif0 create

ifconfig gif0 tunnel 10.0.0.3 10.0.0.1

ifconfig gif0 inet 172.16.0.3 172.16.0.1 netmask 255.255.255.0

If we use the second method, the tunnel will exist automatically after we reboot system.

Add three lines on /etc/rc.conf file as below, the tunnel will exist automatically after we reboot system.

gif_interfaces=”gif0”

gifconfig_gif0=”10.0.0.3 10.0.0.1”

ifconfig_gif0=”172.16.0.3 172.16.0.1 netmask 255.255.255.0”

Using IP-tunneling

In all the cases on Linux/FreeBSD/Solaris, the netmask value will be default if we ignore to set it.

After creating two tunnels, we can ping 10.0.0.2 and 10.0.0.3 at Linux box successfully, and ping 10.0.0.1 at Solaris or FreeBSD box successfully. All applications layers protocol will work correctly.

Working example of VS/TUN

In VS/TUN mode, because realserver don’t send any datagram to Load balancer, we need not create any tunnel on Linux box and must create proper tunnel on FreeBSD/Solaris box.

Topological graph

We can treat all machines on internet and have their real ip. The client can send their requests to VIP of LB Server. LB Server redirects the requests realserver through IP-tunneling. Realserver can send datagram to client.

Because both the gif0 of FreeBSD and ip.tun0 both are NO-ARP device, they are invisible to client.

Config FreeBSD box

On FreeBSD box, edit /etc/rc.conf as below:

...

ifconfig_lnc0=”inet 192.168.0.170 netmask 0xffffff00”

gif_interfaces=”gif0”

gifconfig_gif0=”192.168.0.170 192.168.10.100”

ifconfig_gif0=”192.168.10.200 192.168.10.99 netmask 0xffffffff”

apache_enable=”YES”

...

192.168.10.99 looks like a terminal of this tunnel. It can be any idle IP and never use.

Config Solaris box

On Solaris , edit /etc/hostname.ip.tun0 as below:

tsrc 192.168.0.180 tdst 192.168.10.100

192.168.10.200 192.168.10.99 netmask 255.255.255.255 up

192.168.10.99 looks like a terminal of this tunnel. It can be any IP address and never use it.

Config Load balancer

Run ipvsadm command on Linux Box: Linux kernel must have ip forward enable.

Edit /etc/sysctl.conf and set to:

net.ipv4.ip_forward = 1

While running command:

sysctl -p

It will take effect.

Run the scripts:

ipvsadm -C

ipvsadm -A -t 192.168.10.200:80 -s wlc

ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.170 -i

ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.180 -i

In this script, the port of LB Server and realerver must be same. Good idea is not setting the port of realserver.Now VS/TUN is finished. When a client access 192.168.10.200 at port 80, the datagram will be sent to 192.168.0.170 or 192.168.180 port 80.

Conclusions and Future Work

I believe AIX and HP-UX can also work with VS/TUN, I have no Aix or HP-UX servers. Who can help to provide me with such an environment?

References:

[1] http://www.linuxvirtualserver.org

[2] http://www.triload.com/en/manual/index.html

[3] http://blog.kovyrin.net/2006/03/17/how-to-create-ip-ip-tunnel-between-freebsd-and-linux/

[4] http://docs.sun.com/app/docs/doc/817-0573/6mgc65bcq?a=view