Difference between revisions of "LVS/TUN mode with FreeBSD and Solaris realserver"

From LVSKB
Jump to: navigation, search
 
(lidomtatroc)
 
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
LVS/TUN mode with FreeBSD and Solaris
+
dardrone
 +
Author: Ma Luyao RHCE  maluyao at gmail dot com
  
Author: Ma Luyao RHCE  maluyao at gmail dot com
+
August 16, 2006
  
Introduction
+
== Introduction ==
  
The Linux Virtual Server Project (LVS) is a project started by Dr. Wensong Zhang. LVS implements three load balance technology.  
+
The Linux Virtual Server Project ([[LVS]]) is a project started by Dr. Wensong Zhang. LVS implements three load balance technology.
  
Network Address Translation (VS/NAT)
+
a. Network Address Translation ([[LVS/NAT|VS/NAT]])
Direct Routing (VS/DR)
+
IP Tunneling (VS/TUN)
+
  
VS/NAT is easy to setup. The load balancer may be a bottleneck of the whole system where the number of servers is more than 20, because both the request packets and response packets need to be rewritten by the load balancer.
+
b. Direct Routing ([[LVS/DR|VS/DR]])
  
VS/DR has the best performance. VS/DR uses MAC-spoofing technology, so it requires that one of the load balancer's NIC and the real server's NIC must be in the same IP network segment, and physical segment as well.
+
c. IP Tunneling ([[LVS/TUN|VS/TUN]])
  
VS/TUN is the most scalable. The advantage is since servers connect each other by using IP-tunneling, the load balance and real servers can reside on different LAN , or even WAN.
+
[[LVS/NAT|VS/NAT]] is easy to setup. The load balancer may be a bottleneck of the whole system where the number of servers is more than 20, because both the request packets and response packets need to be rewritten by the [[load balancer]].
  
I plan to setup a set of servers with load balance technique. Realserver run FreeBSD and Solaris system in different places on our business. So it is the only choice for us to use VS/TUN.
+
[[LVS/DR|VS/DR]] has the best performance. VS/DR uses MAC-spoofing technology, so it requires that one of the load balancer’s NIC and the [[real server]]’s NIC must be in the same IP network segment, and physical segment as well.
  
According to the official document of LVS, FreeBSD and Solaris system can not be a realserver, but I did do it after some hard testing.  
+
[[LVS/TUN|VS/TUN]] is the most scalable. The advantage is that since servers connect each other by using IP-tunneling, the load balancer and the real servers can reside on different LAN , or even WAN.
 +
 
 +
I plan to setup a set of servers with [[load balancing]] technique. [[Real server]]s run FreeBSD and Solaris system in different places on our business. So it is the only choice for us to use VS/TUN.
  
 
We install Fedora Core 5 Linux on our Load Balance server. Kernel-2.6.15 is default.
 
We install Fedora Core 5 Linux on our Load Balance server. Kernel-2.6.15 is default.
Line 27: Line 28:
 
Solaris box runs Solaris 10 x86.
 
Solaris box runs Solaris 10 x86.
  
Normal Linux Solaris IP-tunneling
+
== Normal Linux Solaris/FreeBSD IP-tunneling ==
  
 
IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram, which allows datagram destined for one IP address to be wrapped and redirected to another IP address. Many modern OS support IP tunneling, such as Linux, FreeBSD and Solaris.
 
IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram, which allows datagram destined for one IP address to be wrapped and redirected to another IP address. Many modern OS support IP tunneling, such as Linux, FreeBSD and Solaris.
  
Let's do some test to setup normal Linux-Solaris and Linux-FreeBSD IP-tunneling without LVS.
+
Let’s do some test to setup normal Linux-Solaris and Linux-FreeBSD IPtunneling without LVS.
OS                  | IP Address | Tunnel IP Address
+
  
Fedora Core 5 Linux  | 10.0.0.1  | 172.16.0.1
+
=== Testing environment ===
Solaris 10 x86      | 10.0.0.2  | 172.16.0.2
+
FreeBSD 5.4          | 10.0.0.3  | 172.16.0.3
+
  
Config Linux box
+
{|
 +
|-
 +
|| OS IP      ||        Address  || Tunnel IP Address
 +
|-
 +
|| Fedora Core 5 Linux || 10.0.0.1 || 172.16.0.1
 +
|-
 +
|| Solaris 10 x86  || 10.0.0.2 || 172.16.0.2
 +
|-
 +
|| FreeBSD 5.4  || 10.0.0.3 || 172.16.0.3
 +
|}
 +
 
 +
=== Config Linux box ===
  
 
In order to create an IP-Tunneling between Linux box and Solaris box, run command:
 
In order to create an IP-Tunneling between Linux box and Solaris box, run command:
  
ip tunnel add tun0 mode ipip remote 10.0.0.2 local 10.0.0.1
+
ip tunnel add tun0 mode ipip remote 10.0.0.2 local 10.0.0.1
ifconfig tun0 172.16.0.1 pointopoint 172.16.0.2
+
ip tunnel add tun1 mode ipip remote 10.0.0.3 local 10.0.0.1
+
ifconfig tun0 172.16.0.1 pointopoint 172.16.0.2
ifconfig tun1 172.16.0.1 pointopoint 172.16.0.3
+
 +
ip tunnel add tun1 mode ipip remote 10.0.0.3 local 10.0.0.1
 +
 +
ifconfig tun1 172.16.0.1 pointopoint 172.16.0.3
  
The "tun0" is a virtual NIC device name. It can be replaced by other names, such as "abc0" or "def0".
+
The ”tun0” is a virtual NIC device name. It can be replaced by other names, such as ”abc0” or ”def0”.
  
Config Solaris box
+
=== Config Solaris box ===
  
 
To create an IP-tunneling, we have two methods, First method of creating tunnel is using three commands as below.
 
To create an IP-tunneling, we have two methods, First method of creating tunnel is using three commands as below.
  
/sbin/ifconfig ip.tun0 plumb
+
/sbin/ifconfig ip.tun0 plumb
/sbin/ifconfig ip.tun0 tsrc 10.0.0.2 tdst 10.0.0.1
+
/sbin/ifconfig ip.tun0 172.16.0.2 172.16.0.1
+
/sbin/ifconfig ip.tun0 tsrc 10.0.0.2 tdst 10.0.0.1
 +
 +
/sbin/ifconfig ip.tun0 172.16.0.2 172.16.0.1
  
 
It takes effect immediately.
 
It takes effect immediately.
Line 61: Line 75:
 
Second method is creating /etc/hostname.ip.tun0 file which contains two lines as below.
 
Second method is creating /etc/hostname.ip.tun0 file which contains two lines as below.
  
 +
tsrc 10.0.0.2 tdst 10.0.0.1
 +
 +
172.16.0.2 172.16.0.1 netmask 255.255.255.255 up
  
tsrc 10.0.0.2 tdst 10.0.0.1
+
If we use the second method, the tunnel will exist automatically after we reboot system.
172.16.0.2 172.16.0.1 netmask 255.255.255.255 up
+
  
 +
=== Config FreeBSD box ===
  
If we use the second method, the tunnel will exist automatically after we reboot system.
 
 
Config FreeBSD box
 
 
On FreeBSD box, we also have two methods to create an IP-Tunneling, one way is to run three commands:
 
On FreeBSD box, we also have two methods to create an IP-Tunneling, one way is to run three commands:
  
ifconfig gif0 create
+
ifconfig gif0 create
ifconfig gif0 tunnel 10.0.0.3 10.0.0.1
+
ifconfig gif0 inet 172.16.0.3 172.168.0.1 netmask 255.255.255.0
+
ifconfig gif0 tunnel 10.0.0.3 10.0.0.1
 +
 +
ifconfig gif0 inet 172.16.0.3 172.16.0.1 netmask 255.255.255.0
  
 
If we use the second method, the tunnel will exist automatically after we reboot system.
 
If we use the second method, the tunnel will exist automatically after we reboot system.
Line 79: Line 95:
 
Add three lines on /etc/rc.conf file as below, the tunnel will exist automatically after we reboot system.
 
Add three lines on /etc/rc.conf file as below, the tunnel will exist automatically after we reboot system.
  
gif_interfaces="gif0"
+
gif_interfaces=”gif0”
gifconfig_gif0="10.0.0.3 10.0.0.1"
+
ifconfig_gif0="172.16.0.3 172.16.0.1 netmask 255.255.255.0"
+
gifconfig_gif0=”10.0.0.3 10.0.0.1”
 +
 +
ifconfig_gif0=”172.16.0.3 172.16.0.1 netmask 255.255.255.0”
  
 
+
===  Using IP-tunneling ===
Using IP-tunneling
+
  
 
In all the cases on Linux/FreeBSD/Solaris, the netmask value will be default if we ignore to set it.
 
In all the cases on Linux/FreeBSD/Solaris, the netmask value will be default if we ignore to set it.
Line 90: Line 107:
 
After creating two tunnels, we can ping 10.0.0.2 and 10.0.0.3 at Linux box successfully, and ping 10.0.0.1 at Solaris or FreeBSD box successfully. All applications layers protocol will work correctly.
 
After creating two tunnels, we can ping 10.0.0.2 and 10.0.0.3 at Linux box successfully, and ping 10.0.0.1 at Solaris or FreeBSD box successfully. All applications layers protocol will work correctly.
  
Working example of VS/TUN  
+
== Working example of VS/TUN ==
  
In VS/TUN mode, because realserver don't send any datagram to Load balancer, we need not create any tunnel on Linux box and must create proper tunnel on FreeBSD/Solaris box.  
+
In VS/TUN mode, because realserver don’t send any datagram to Load balancer, we need not create any tunnel on Linux box and must create proper tunnel on FreeBSD/Solaris box.
  
Topological graph
+
=== Topological graph ===
  
topo.eps
+
[[Image:Topo.png|center|600px]]
  
We can treat all machines on internet and have their real ip. The client can send their requests to VIP of LB Server. LB Server redirects the requests realserver through IP-tunneling. Realserver can send datagram to client.
+
We can treat all machines on internet and have their real ip. The client can send their requests to [[VIP]] of LB Server. LB Server redirects the requests realserver through IP-tunneling. Realserver can send datagram to client.
  
 
Because both the gif0 of FreeBSD and ip.tun0 both are NO-ARP device, they are invisible to client.
 
Because both the gif0 of FreeBSD and ip.tun0 both are NO-ARP device, they are invisible to client.
  
Config FreeBSD box
+
=== Config FreeBSD box ===
 +
 
 
On FreeBSD box, edit /etc/rc.conf as below:
 
On FreeBSD box, edit /etc/rc.conf as below:
  
...
+
...
ifconfig_lnc0="inet 192.168.0.170 netmask 0xffffff00"
+
gif_interfaces="gif0"
+
ifconfig_lnc0=”inet 192.168.0.170 netmask 0xffffff00”
gifconfig_gif0="192.168.0.170 192.168.10.100"
+
ifconfig_gif0="192.168.10.200 192.168.10.99 netmask 0xffffffff"
+
gif_interfaces=”gif0”
apache_enable="YES"
+
...
+
gifconfig_gif0=”192.168.0.170 192.168.10.100”
 
+
 +
ifconfig_gif0=”192.168.10.200 192.168.10.99 netmask 0xffffffff”
 +
 +
apache_enable=”YES”
 +
 +
...
  
 
192.168.10.99 looks like a terminal of this tunnel. It can be any idle IP and never use.
 
192.168.10.99 looks like a terminal of this tunnel. It can be any idle IP and never use.
  
Config Solaris box
+
=== Config Solaris box ===
 
+
 
On Solaris , edit /etc/hostname.ip.tun0 as below:
 
On Solaris , edit /etc/hostname.ip.tun0 as below:
  
tsrc 192.168.0.180 tdst 192.168.10.100
+
tsrc 192.168.0.180 tdst 192.168.10.100
192.168.10.200 192.168.10.99 netmask 255.255.255.255 up
+
 
+
192.168.10.200 192.168.10.99 netmask 255.255.255.255 up
  
 
192.168.10.99 looks like a terminal of this tunnel. It can be any IP address and never use it.
 
192.168.10.99 looks like a terminal of this tunnel. It can be any IP address and never use it.
  
Config Load balancer
+
=== Config Load balancer ===
 +
 
 +
Run ipvsadm command on Linux Box: Linux kernel must have ip forward enable.
  
Run ipvsadm command on Linux Box:
+
Edit /etc/sysctl.conf and set to:
Linux kernel must have ip_forward enable. Edit /etc/sysctl.conf and set to:
+
  
net.ipv4.ip_forward = 1
+
net.ipv4.ip_forward = 1
  
 
While running command:
 
While running command:
  
sysctl -p  
+
sysctl -p
  
 
It will take effect.
 
It will take effect.
Line 141: Line 164:
 
Run the scripts:
 
Run the scripts:
  
ipvsadm -C
+
ipvsadm -C
ipvsadm -A -t 192.168.10.200:80 -s wlc
+
ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.170 -i
+
ipvsadm -A -t 192.168.10.200:80 -s wlc
ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.180 -i
+
 +
ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.170 -i
 +
 +
ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.180 -i
  
 +
In this script, the port of LB Server and realerver must be same. Good idea is not setting the port of realserver.Now VS/TUN is finished. When a client access 192.168.10.200 at port 80, the datagram will be sent to 192.168.0.170 or 192.168.180 port 80.
  
In this script, the port of LB Server and realerver must be same. Good idea is not setting the port of realserver.
+
== Conclusions and Future Work ==
  
Now VS/TUN is finished. When a client access 192.168.10.200 at port 80, the
 
datagram will be sent to 192.168.0.170 or 192.168.180 port 80.
 
  
Conclusions and Future Work
+
I believe AIX and HP-UX can also work with VS/TUN, I have no Aix or HP-UX servers. Who can help to provide me with such an environment?  
 
+
I believe AIX and HP-UX can also work with VS/TUN, I have no Aix or HP-UX servers. Who can help to provide me with such an environment?
+
  
 
References:
 
References:
  
http://www.linuxvirtualserver.org
+
[1] http://www.linuxvirtualserver.org
http://www.triload.com/en/manual/index.html
+
 
 +
[2] http://www.triload.com/en/manual/index.html
 +
 
 +
[3] http://blog.kovyrin.net/2006/03/17/how-to-create-ip-ip-tunnel-between-freebsd-and-linux/
 +
 
 +
[4] http://docs.sun.com/app/docs/doc/817-0573/6mgc65bcq?a=view
 +
 
 +
[[Category:LVS Examples|Tunnel]]

Latest revision as of 02:39, 25 December 2008

dardrone Author: Ma Luyao RHCE maluyao at gmail dot com

August 16, 2006

Introduction

The Linux Virtual Server Project (LVS) is a project started by Dr. Wensong Zhang. LVS implements three load balance technology.

a. Network Address Translation (VS/NAT)

b. Direct Routing (VS/DR)

c. IP Tunneling (VS/TUN)

VS/NAT is easy to setup. The load balancer may be a bottleneck of the whole system where the number of servers is more than 20, because both the request packets and response packets need to be rewritten by the load balancer.

VS/DR has the best performance. VS/DR uses MAC-spoofing technology, so it requires that one of the load balancer’s NIC and the real server’s NIC must be in the same IP network segment, and physical segment as well.

VS/TUN is the most scalable. The advantage is that since servers connect each other by using IP-tunneling, the load balancer and the real servers can reside on different LAN , or even WAN.

I plan to setup a set of servers with load balancing technique. Real servers run FreeBSD and Solaris system in different places on our business. So it is the only choice for us to use VS/TUN.

We install Fedora Core 5 Linux on our Load Balance server. Kernel-2.6.15 is default.

FreeBSD version is 5.4, we recompile the kernel in order to improve the performance. The default kernel can work too.

Solaris box runs Solaris 10 x86.

Normal Linux Solaris/FreeBSD IP-tunneling

IP tunneling (IP encapsulation) is a technique to encapsulate IP datagram within IP datagram, which allows datagram destined for one IP address to be wrapped and redirected to another IP address. Many modern OS support IP tunneling, such as Linux, FreeBSD and Solaris.

Let’s do some test to setup normal Linux-Solaris and Linux-FreeBSD IPtunneling without LVS.

Testing environment

OS IP Address Tunnel IP Address
Fedora Core 5 Linux 10.0.0.1 172.16.0.1
Solaris 10 x86 10.0.0.2 172.16.0.2
FreeBSD 5.4 10.0.0.3 172.16.0.3

Config Linux box

In order to create an IP-Tunneling between Linux box and Solaris box, run command:

ip tunnel add tun0 mode ipip remote 10.0.0.2 local 10.0.0.1

ifconfig tun0 172.16.0.1 pointopoint 172.16.0.2

ip tunnel add tun1 mode ipip remote 10.0.0.3 local 10.0.0.1

ifconfig tun1 172.16.0.1 pointopoint 172.16.0.3

The ”tun0” is a virtual NIC device name. It can be replaced by other names, such as ”abc0” or ”def0”.

Config Solaris box

To create an IP-tunneling, we have two methods, First method of creating tunnel is using three commands as below.

/sbin/ifconfig ip.tun0 plumb

/sbin/ifconfig ip.tun0 tsrc 10.0.0.2 tdst 10.0.0.1

/sbin/ifconfig ip.tun0 172.16.0.2 172.16.0.1

It takes effect immediately.

Second method is creating /etc/hostname.ip.tun0 file which contains two lines as below.

tsrc 10.0.0.2 tdst 10.0.0.1

172.16.0.2 172.16.0.1 netmask 255.255.255.255 up

If we use the second method, the tunnel will exist automatically after we reboot system.

Config FreeBSD box

On FreeBSD box, we also have two methods to create an IP-Tunneling, one way is to run three commands:

ifconfig gif0 create

ifconfig gif0 tunnel 10.0.0.3 10.0.0.1

ifconfig gif0 inet 172.16.0.3 172.16.0.1 netmask 255.255.255.0

If we use the second method, the tunnel will exist automatically after we reboot system.

Add three lines on /etc/rc.conf file as below, the tunnel will exist automatically after we reboot system.

gif_interfaces=”gif0”

gifconfig_gif0=”10.0.0.3 10.0.0.1”

ifconfig_gif0=”172.16.0.3 172.16.0.1 netmask 255.255.255.0”

Using IP-tunneling

In all the cases on Linux/FreeBSD/Solaris, the netmask value will be default if we ignore to set it.

After creating two tunnels, we can ping 10.0.0.2 and 10.0.0.3 at Linux box successfully, and ping 10.0.0.1 at Solaris or FreeBSD box successfully. All applications layers protocol will work correctly.

Working example of VS/TUN

In VS/TUN mode, because realserver don’t send any datagram to Load balancer, we need not create any tunnel on Linux box and must create proper tunnel on FreeBSD/Solaris box.

Topological graph

Topo.png

We can treat all machines on internet and have their real ip. The client can send their requests to VIP of LB Server. LB Server redirects the requests realserver through IP-tunneling. Realserver can send datagram to client.

Because both the gif0 of FreeBSD and ip.tun0 both are NO-ARP device, they are invisible to client.

Config FreeBSD box

On FreeBSD box, edit /etc/rc.conf as below:

...

ifconfig_lnc0=”inet 192.168.0.170 netmask 0xffffff00”

gif_interfaces=”gif0”

gifconfig_gif0=”192.168.0.170 192.168.10.100”

ifconfig_gif0=”192.168.10.200 192.168.10.99 netmask 0xffffffff”

apache_enable=”YES”

...

192.168.10.99 looks like a terminal of this tunnel. It can be any idle IP and never use.

Config Solaris box

On Solaris , edit /etc/hostname.ip.tun0 as below:

tsrc 192.168.0.180 tdst 192.168.10.100

192.168.10.200 192.168.10.99 netmask 255.255.255.255 up

192.168.10.99 looks like a terminal of this tunnel. It can be any IP address and never use it.

Config Load balancer

Run ipvsadm command on Linux Box: Linux kernel must have ip forward enable.

Edit /etc/sysctl.conf and set to:

net.ipv4.ip_forward = 1

While running command:

sysctl -p

It will take effect.

Run the scripts:

ipvsadm -C

ipvsadm -A -t 192.168.10.200:80 -s wlc

ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.170 -i

ipvsadm -a -t 192.168.10.200:80 -r 192.168.0.180 -i

In this script, the port of LB Server and realerver must be same. Good idea is not setting the port of realserver.Now VS/TUN is finished. When a client access 192.168.10.200 at port 80, the datagram will be sent to 192.168.0.170 or 192.168.180 port 80.

Conclusions and Future Work

I believe AIX and HP-UX can also work with VS/TUN, I have no Aix or HP-UX servers. Who can help to provide me with such an environment?

References:

[1] http://www.linuxvirtualserver.org

[2] http://www.triload.com/en/manual/index.html

[3] http://blog.kovyrin.net/2006/03/17/how-to-create-ip-ip-tunnel-between-freebsd-and-linux/

[4] http://docs.sun.com/app/docs/doc/817-0573/6mgc65bcq?a=view