我们的前端负载均衡已经切换到了LVS FULLNAT模式,它比LVS DR 模式 的好处是 LVS和后端机器不用在一个二层网络里面,非常好扩展,不好的地方是后端机器要装一台TOA模块来识别 HTTP请求的源IP。
想了解LVS FULLNAT的话可以参考这个链接:
http://kb.linuxvirtualserver.org/wiki/IPVS_FULLNAT_and_SYNPROXY
用LVS fullnat的时候其实需要注意一些TCP option的设置,比如 timestamps 和 windows scaling,而且 TCP option 和 synproxy 还有关系,在这篇文章我们手动测试,得出 怎么在LVS fullnat机器上 设置TCP option 。
环境:
LVS VIP : 60.28.208.10
LVS fullnat localip 段:10.0.19.0/24
后端IP : 10.0.11.12
探测的抓包
LVS 检测后端时LVS的抓包记录:
18:45:36.023248 IP 10.0.19.226.63923 > 10.0.11.12.http: Flags [S], seq 1537556744, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 18:45:36.023270 IP 10.0.19.226.19042 > 10.0.11.12.https: Flags [S], seq 520304659, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 18:45:36.023544 IP 10.0.11.12.http > 10.0.19.226.63923: Flags [S.], seq 3628316982, ack 1537556745, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 18:45:36.023560 IP 10.0.19.226.63923 > 10.0.11.12.http: Flags [.], ack 1, win 115, length 0 18:45:36.023565 IP 10.0.11.12.https > 10.0.19.226.19042: Flags [S.], seq 458068257, ack 520304660, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 18:45:36.023572 IP 10.0.19.226.19042 > 10.0.11.12.https: Flags [.], ack 1, win 115, length 0 18:45:36.023589 IP 10.0.19.226.63923 > 10.0.11.12.http: Flags [R.], seq 1, ack 1, win 115, length 0 18:45:36.023602 IP 10.0.19.226.19042 > 10.0.11.12.https: Flags [R.], seq 1, ack 1, win 115, length 0
LVS 检测后端时后端的抓包记录:
18:46:26.041901 IP 10.0.19.226.61028 > 10.0.11.12.http: Flags [S], seq 3032456781, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
18:46:26.041937 IP 10.0.11.12.http > 10.0.19.226.61028: Flags [S.], seq 1104080475, ack 3032456782, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
18:46:26.041951 IP 10.0.19.226.40349 > 10.0.11.12.https: Flags [S], seq 849356025, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
18:46:26.041960 IP 10.0.11.12.https > 10.0.19.226.40349: Flags [S.], seq 1050279505, ack 849356026, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
18:46:26.042113 IP 10.0.19.226.61028 > 10.0.11.12.http: Flags [.], ack 1, win 115, length 0
18:46:26.042147 IP 10.0.19.226.40349 > 10.0.11.12.https: Flags [.], ack 1, win 115, length 0
18:46:26.042310 IP 10.0.19.226.61028 > 10.0.11.12.http: Flags [R.], seq 1, ack 1, win 115, length 0
18:46:26.042373 IP 10.0.19.226.40349 > 10.0.11.12.https: Flags [R.], seq 1, ack 1, win 115, length 0
上面可以看到,LVS 从localip 段里面选一个IP,起两个端口分别探测 80 和 443,建立连接之后 里面发送 RST 给后端,这样探测也成功了。
请求的抓包
在一台机器上执行下面命令:
# curl -H “Host:10.0.11.12” http://60.28.208.10/test
LVS 上面抓VIP 的包如下:
# tcpdump -n -i em1 host 60.28.208.10 and port 80
17:08:00.122008 IP 111.206.15.146.15405 > 60.28.208.10.http: Flags [S], seq 1679917190, win 14600, options [mss 1460,sackOK,TS val 1412716450 ecr 0,nop,wscale 7], length 0
17:08:00.122018 IP 60.28.208.10.http > 111.206.15.146.15405: Flags [S.], seq 2689133917, ack 1679917191, win 14600, options [mss 1452,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,wscale 1], length 0
17:08:00.132352 IP 111.206.15.146.15405 > 60.28.208.10.http: Flags [.], ack 1, win 115, length 0
17:08:00.132439 IP 111.206.15.146.15405 > 60.28.208.10.http: Flags [P.], seq 1:169, ack 1, win 115, length 168
17:08:00.132877 IP 60.28.208.10.http > 111.206.15.146.15405: Flags [.], ack 169, win 123, length 0
17:08:01.138207 IP 60.28.208.10.http > 111.206.15.146.15405: Flags [P.], seq 1:215, ack 169, win 123, length 214
17:08:01.148585 IP 111.206.15.146.15405 > 60.28.208.10.http: Flags [.], ack 215, win 123, length 0
17:08:01.148741 IP 111.206.15.146.15405 > 60.28.208.10.http: Flags [F.], seq 169, ack 215, win 123, length 0
17:08:01.149060 IP 60.28.208.10.http > 111.206.15.146.15405: Flags [F.], seq 215, ack 170, win 123, length 0
17:08:01.159344 IP 111.206.15.146.15405 > 60.28.208.10.http: Flags [.], ack 216, win 123, length 0
LVS 上面抓连接后端机器的包:
# tcpdump -n -i em2 net 10.0.19.0/24 and host 10.0.11.12
17:08:00.132468 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [S], seq 3159822860, win 5000, options [Unknown Option 2003c2d6fce0f92,mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
17:08:00.132656 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [S.], seq 2205177579, ack 3159822861, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
17:08:00.132686 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [P.], seq 1:169, ack 1, win 115, options [Unknown Option 2003c2d6fce0f92], length 168
17:08:00.132855 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [.], ack 169, win 123, length 0
17:08:01.138194 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [P.], seq 1:215, ack 169, win 123, length 214
17:08:01.148598 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [.], ack 215, win 123, length 0
17:08:01.148749 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [F.], seq 169, ack 215, win 123, length 0
17:08:01.149046 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [F.], seq 215, ack 170, win 123, length 0
17:08:01.159357 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [.], ack 216, win 123, length 0
上面的包是在开启syn_proxy的时候抓的,可以看出(注意看时间点),Client 请求来的时候,LVS 会先和Client 建好连接,然后在 Client 请求数据的时候 LVS 选择一个localip端口 和 后端 建立连接,建立成功后LVS 直接 向后端请求数据,获取数据后 再通过 VIP 传给 Client。
下面我们把 syn_proxy 关掉之后重新抓一下看看。
# tcpdump -n -i em1 host 60.28.208.10 and port 80
17:10:54.142439 IP 111.206.15.146.15408 > 60.28.208.10.http: Flags [S], seq 50780707, win 14600, options [mss 1460,sackOK,TS val 1412890473 ecr 0,nop,wscale 7], length 0
17:10:54.142802 IP 60.28.208.10.http > 111.206.15.146.15408: Flags [S.], seq 629114004, ack 50780708, win 14600, options [mss 1452,nop,nop,sackOK,nop,wscale 7], length 0
17:10:54.153112 IP 111.206.15.146.15408 > 60.28.208.10.http: Flags [.], ack 1, win 115, length 0
17:10:54.153180 IP 111.206.15.146.15408 > 60.28.208.10.http: Flags [P.], seq 1:169, ack 1, win 115, length 168
17:10:54.153360 IP 60.28.208.10.http > 111.206.15.146.15408: Flags [.], ack 169, win 123, length 0
17:10:55.158651 IP 60.28.208.10.http > 111.206.15.146.15408: Flags [P.], seq 1:215, ack 169, win 123, length 214
17:10:55.168912 IP 111.206.15.146.15408 > 60.28.208.10.http: Flags [.], ack 215, win 123, length 0
17:10:55.168940 IP 111.206.15.146.15408 > 60.28.208.10.http: Flags [F.], seq 169, ack 215, win 123, length 0
17:10:55.169284 IP 60.28.208.10.http > 111.206.15.146.15408: Flags [F.], seq 215, ack 170, win 123, length 0
17:10:55.179620 IP 111.206.15.146.15408 > 60.28.208.10.http: Flags [.], ack 216, win 123, length 0
# tcpdump -n -i em2 net 10.0.19.0/24 and host 10.0.11.12
17:10:54.142475 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [S], seq 1583761902, win 14600, options [Unknown Option 2003c306fce0f92,mss 1460,sackOK,TS val 1412890473 ecr 0,nop,wscale 7], length 0
17:10:54.142766 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [S.], seq 629114004, ack 1583761903, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
17:10:54.153128 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [.], ack 1, win 115, options [Unknown Option 2003c306fce0f92], length 0
17:10:54.153192 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [P.], seq 1:169, ack 1, win 115, options [Unknown Option 2003c306fce0f92], length 168
17:10:54.153348 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [.], ack 169, win 123, length 0
17:10:55.158639 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [P.], seq 1:215, ack 169, win 123, length 214
17:10:55.168925 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [.], ack 215, win 123, length 0
17:10:55.168944 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [F.], seq 169, ack 215, win 123, length 0
17:10:55.169272 IP 10.0.11.12.http > 10.0.19.227.commplex-link: Flags [F.], seq 215, ack 170, win 123, length 0
17:10:55.179630 IP 10.0.19.227.commplex-link > 10.0.11.12.http: Flags [.], ack 216, win 123, length 0
通过上面的抓包可以看到(同样看时间点),关闭syn_proxy 的时候,LVS 接受到请求,此时开启一个localip的一个端口 给后端发送 SYN,收到 后端的SYN+ACK之后,LVS通过VIP 给Client 返回 SYN+ACK,然后 Client 会再发送 ACK,VIP 会接收到这个ACK,然后再通过localip 传给 后端,后端收到后 建立连接。看这个流程,LVS 纯粹扮演 Proxy 的角色。
syn_proxy 机制会影响 Client 到 Server 的TCP 连接的Option 选项,下面细看。
关于timestamps 的测试
Client —> LVS —> Server
LVS 上面有三个和 timestamps 相关的参数 :
net.ipv4.tcp_timestamps
net.ipv4.vs.fullnat_timestamp_remove_entry
net.ipv4.vs.synproxy_timestamp
(LVS 默认 net.ipv4.vs.fullnat_timestamp_remove_entry = 1 ,net.ipv4.vs.synproxy_timestamp = 0)
在开启 syn_proxy 情况下的测试:
1. 即使 Client 和 Server 的 timestamps 都设置为1 ,只要 LVS net.ipv4.vs.synproxy_timestamp 不设置为1 ,timestamps 就不会启用。此时 LVS 的 net.ipv4.tcp_timestamps 和 net.ipv4.vs.fullnat_timestamp_remove_entry 完全没用。
2. 如果Client 关闭了 net.ipv4.tcp_timestamps ,timestamps 选项不会启用。
3. 比较诧异的是,无论 Server 的 net.ipv4.tcp_timestamps 是否设置为1,只要LVS 开启 net.ipv4.vs.synproxy_timestamp 和 Client 开启 net.ipv4.tcp_timestamps,建立的TCP连接 就会启用 timestamps (此时在Server 上抓包 会看到 TS val 和 ecr )
在关闭 syn_proxy 情况下的测试:
1. 此时 net.ipv4.vs.fullnat_timestamp_remove_entry 参数起作用,即使 Client 和 Server 的 timestamps 都设置为1 ,只要 LVS net.ipv4.vs.fullnat_timestamp_remove_entry 不设置为0 ,timestamps 就不会启用。
2. 如果Client 关闭了 net.ipv4.tcp_timestamps ,timestamps 选项不会启用。
3. 如果Server关闭了 net.ipv4.tcp_timestamps ,timestamps 选项不会启用。
总结:
1. syn_proxy 开启,net.ipv4.vs.synproxy_timestamp 决定LVS 是否启用timestamps。
2. syn_proxy 关闭,net.ipv4.vs.fullnat_timestamp_remove_entry 决定LVS 是否启用timestamps。
3. LVS FULLNAT 模式 的 net.ipv4.tcp_timestamps 参数 完全没用,设置与不设置都不会影响timestamps
4. 简单点,如果想开启LVS的 timestamps,把 net.ipv4.vs.synproxy_timestamp 设置成1,同时把 net.ipv4.vs.fullnat_timestamp_remove_entry 设置成 0
关于windows scaling 的测试
Client —> LVS —> Server
在开启 syn_proxy 情况下的测试:
1. net.ipv4.vs.synproxy_wscale 决定连接 是否 wscale,即使 Client 和 Server 都开启了wscale,只要LVS 不开启 net.ipv4.vs.synproxy_wscale,连接就不会开启wscale。
2. 如果Client 不开启 net.ipv4.tcp_window_scaling ,连接就不会开启 wscale。
3. Server 的 net.ipv4.tcp_window_scaling 变的没用,即使没打开,只要 Client 打开了net.ipv4.tcp_window_scaling 而且LVS 打开了 net.ipv4.vs.synproxy_wscale ,连接就会打开 wscale
在关闭 syn_proxy 情况下的测试:
1. 只要 Client 和 Server 同时开启 wscale ,连接就会开启 wscale,不管 LVS 如何设置。
总结:
1. LVS 的 net.ipv4.tcp_window_scaling 完全没用,可忽略这个参数。
2. 想开启LVS 的wscale, 把 net.ipv4.vs.synproxy_wscale 设置成1