使用 Nginx 做翻墙代理

配置

/etc/nginx/nginx.conf 配置,可以采用自签名证书,客户端连的时候不 verify 证书即可。

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
    worker_connections 10240;
}

http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" '
    '"upstream=$upstream_addr" "scheme=$scheme"     "X-Remote-App=$http_x_remote_app" "reqtime=$request_time"     "upstream_resptime=$upstream_response_time"     "$upstream_cache_status" "host=$host"';
    ssl_certificate /etc/nginx/ssl/52.77.252.184.crt;
    ssl_certificate_key /etc/nginx/ssl/52.77.252.184.key;
    ssl_session_cache shared:SSL:100m;
    ssl_session_timeout 10m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SH    A256:ECDHE-RSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AE    S128-SHA:DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    access_log /var/log/nginx/access.log main;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # Load modular configuration files from the     /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/server.conf 配置。

#
# A virtual host using mix of IP-, name-, and port-based configuration
#

server {
    listen 443 ssl;
    server_name _;

    access_log /var/log/nginx/443_access.log main;
    resolver 8.8.8.8;

    set $upstream_endpoint $host;

    location / {
        proxy_pass https://$upstream_endpoint;
    }
}

server {
    listen 80;
    server_name _;

    access_log /var/log/nginx/80_access.log main;
    resolver 8.8.8.8;

    set $upstream_endpoint $host;

    location / {
        proxy_pass http://$upstream_endpoint;
    }
}

测试

要修改 header,而且 curl 要加 -k 不 verify 证书。

$ curl -k -I -H "Host:play.google.com" "https://52.77.252.184/store/apps/details?hl=en&id=tr.com.fugo.kelimeavi2.en"
HTTP/1.1 200 OK

$ curl -k -I -H "Host:www.baidu.com" http://52.77.252.184
HTTP/1.1 200 OK

这样翻墙的好处是可以有多台 Nginx,程序能够控制访问哪台 Nginx。

一次 CDN 源站流量暴增的问题分析

上周经历了一次 CDN 源站流量暴增的问题,原因是有 CDN 有一批文件过期,大量回源。

CDN 回源过程:

  1. 第一次回源,如果源站返回 Last-Modified 头,CDN 会记录下来;

  2. CDN 的文件过期后,如果源站之前已经返回了 Last-Modified 头,就以 If-Modified-Since 头(内容是 源站返回的Last-Modified 值) 来访问源站,如果文件未变更,则返回 304;如果之前源站没返回  Last-Modified 头,则会下载整个文件,源站返回 200。

流量暴增的问题就出现在源站不返回 Last-Modified 头上,解决方法是源站返回 Last-Modified 头。

但是需要注意几点,先看看源站前面的 Nginx 配置文件:

location ~ /(?<name>.+?)/.* {
    expires 100d;

    rewrite ^/.+?/(.*)$ /$1 break;

    proxy_cache m_wdjcdn_com;
    proxy_cache_key &quot;$request_uri&quot;;
    proxy_cache_valid 200 30d;

    proxy_set_header Host $name.cdn.xxxx.com;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_pass http://new-source-nodes;
}
  1. 如果源站文件放在本地硬盘(使用 root 指令) 而不是通过 proxy_pass 打到后端,就不会出现这个问题,因为 Nginx 对静态文件完全支持 Last-Modified 和 If-Modified-Since 语义;

  2. Nginx 有一个指令 if_modified_since,默认是 exact,表示 If-Modified-Since 头的值必须和上次获取的 Last-Modified 值一致,否则还会 200;可以把 if_modified_since 改成 before,表示 If-Modified-Since 值可以比 Last-Modified 值大,也就是更新。

请求中的 Accept-Encoding Header 出错

请求类似:

GET /api/v1/feed HTTP/1.1
model: Android
If-Modified-Since: Thu, 10 Sep 2015 08:42:41 GMT+00:00
If-None-Match: “03e6c6bafcf335f108dd2fdc8359976cc”
Host: baobab.xxx.com
Connection: Keep-Alive
Accdpt-Encoding: gzip
User-Agent: okhttp/2.4.0

在 LVS 外网网卡上抓包: lvs.pcap

在 wireshark 里通过 http.request and http contains “Accdpt-Encoding” 过滤,然后再根据 ip.addr 和 tcp.port 过滤,看到如下结果:

111132432432423h1

说明进来的请求已经是有问题的了。

 

好吧,这篇文章目前为止没有价值,先 mark,如果有后续的话继续更新。

 

Tengine request_time 和 upstream_response_time 相等的问题排查

最近一段时间我们把 Tengine 从 1.4.6(nginx/1.2.9) 升级到了 Tengine 2.1.0(nginx/1.6.2),发现 request_time 出奇的小,大部分情况下都和 upstream_response_time 相等,特别对于 https 请求,一般要 0.x 秒,但是日志记录的只有 0.0x 秒。

 

搜遍了 Tengine,也没搜到有关系的内容,由于 Tengine 是 fork 的 Nginx,所以在 Nginx 测试一下,发现 nginx/1.6.2 同样有问题,继续测试 nginx/1.2.9 发现没问题。

所以是在 nginx/1.2.9 和 nginx/1.6.2 中间的某个版本引文的问题,实在没辙,只有用二分法找到了出问题的版本:

nginx-1.4.7    NOTOK
nginx-1.4.3    NOTOK
nginx-1.4.0    NOTOK
ngnx-1.3.16   NOTOK
nginx-1.3.15  NOTOK
ngnx-1.3.14   OK
nginx-1.3.9    OK
nginx-1.2.9    OK

nginx-1.3.15 的 changlog:

Changes with nginx 1.3.15 26 Mar 2013 *) Change: opening and closing a connection without sending any data in
it is no longer logged to access_log with error code 400. *) Feature: the ngx_http_spdy_module.
Thanks to Automattic for sponsoring this work.

*) Feature: the “limit_req_status” and “limit_conn_status” directives.
Thanks to Nick Marden.

*) Feature: the “image_filter_interlace” directive.
Thanks to Ian Babrou.

*) Feature: $connections_waiting variable in the
ngx_http_stub_status_module.

*) Feature: the mail proxy module now supports IPv6 backends.

*) Bugfix: request body might be transmitted incorrectly when retrying a
request to the next upstream server; the bug had appeared in 1.3.9.
Thanks to Piotr Sikora.

*) Bugfix: in the “client_body_in_file_only” directive; the bug had
appeared in 1.3.9.

*) Bugfix: responses might hang if subrequests were used and a DNS error
happened during subrequest processing.
Thanks to Lanshun Zhou.

*) Bugfix: in backend usage accounting.

changlog 都没提,擦,搜了一下,搜到了这个:

http://trac.nginx.org/nginx/ticket/356

解释如下:

As of 1.3.15, $request_time calculation was slightly changed due to some optimizations introduced, and for a first request in a connection it no longer includes time between opening a connection and reading a first byte from a client. It still works correctly though.

看样子是从收到请求数据的第一个字节开始算的,而不是按照之前建立连接时的时间(之前是否是建立时间我没真正测过,属猜测),在官方文档中看到 request_time 的说明:

request processing time in seconds with a milliseconds resolution; time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client.

官方文档写的是第一个字节。

 

抓个包看看:request_time.pcap

日志中记录:”request_time=0.013″ “upstream_response_time=0.013”

E06FF4CC-110F-4739-B015-DBD55B0D9F75

10.0.23.74 是 Nginx,Nginx 接受到真实的请求数据是在 16,17 便是 Nginx 发送完数据的时间点,两者的时间差正好是 0.013,确认。

 

参考:

http://nginx.org/en/CHANGES-1.4

http://trac.nginx.org/nginx/ticket/356

redis 连接一直是 ESTABLISHED 的问题排查

昨天想删一台机器,发现上面还有 redis 连接:

# netstat -nat |grep ESTABLISHED |grep 6379
tcp 0 0 10.0.27.92:6379 10.0.27.157:24044 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.96.27:28975 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.69.47:58511 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.16.29.9:44571 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.29.49:48137 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.69.46:8854 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.70.67:42271 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.70.67:42269 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.24.30:17776 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.22.91:17823 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.23.79:59200 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.24.30:46296 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.23.98:31277 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.0.22.118:40458 ESTABLISHED
tcp 0 0 10.0.27.92:6379 10.16.29.9:44548 ESTABLISHED

这些连接一直都在,更奇怪的是,10.0.70.67 这个 IP 已经 ping 不通了,连接却不会断,正常情况下 tcp keepalive 会隔段时间检查一次,发现不通之后会发送 reset 。

 

为了看看到底有没有发送 tcp keepalive ack 包,抓个包看看,命令是 tcpdump -i em2 port 6379,下面是抓了一夜的包:redis.cpap

用 wireshark 分析,发现基本都是 keepalive ack 的包,取 10.0.96.27 这个IP,截个图看看:

4888351E-2D1D-4E6F-8062-F2DA259CF9C7

可以看到,10.0.96.27 主动 发送 ACK(SEQ: M、ACK: N)给 redis,redis 回复 ACK(SEQ: N、ACK:M+1),且 Len 都是0。

这能够解释大部分 IP 一直在 ESTABLISHED,因为一直有 tcp keepalive,但是 10.0.70.67 解释不通了,而且上面根本没抓到 10.0.70.67 的包,这只有一种可能: redis 不主动发送 keepalive。

 

找了下文档,发现 redis 确实默认关闭 tcp keepalive,所以对于已经建立的连接,不会发送 tcp keepalive ack 来确认对方存活,而如果对方突然死机或者关电源导致对方不主动关闭连接,那么 redis 就一直认为对方是活的,就不会去关闭连接了。

redis 提供了配置文件来更改这一默认行为:

# TCP keepalive.
#
# If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
# of communication. This is useful for two reasons:
#
# 1) Detect dead peers.
# 2) Take the connection alive from the point of view of network
# equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 60 seconds.
tcp-keepalive 0

 

事实上,如果 redis 对 idle 有时间限制,我遇到的情况也不会存在,但是 redis 也确实默认对 idle 的连接不加时间限制, 只是提供了 timeout 参数来更改。