Emqtt优化相关

作者:新溪-gordon <programfan.info#gmail.com>
时间:2018-07-03

EMQ 消息服务器1.x版本 MQTT 连接压力测试到130万,在一台8核心、32G内存的 CentOS 服务器上

Emqtt优化步骤

一、Linux Kernel Tuning:

//The system-wide limit on max opened file handles:
# 2M:系统所有进程可打开的文件数量
sysctl -w fs.file-max=2097152
sysctl -w fs.nr_open=2097152
echo 2097152 > /proc/sys/fs/nr_open

# 1M:系统允许当前进程打开的文件数量
ulimit -n 1048576


## 上面临时操作,这儿是持久化的
// 文件/etc/sysctl.conf
fs.file-max = 1048576

// 文件/etc/security/limits.conf
*      soft   nofile      1048576
*      hard   nofile      1048576

二、Network Tuning:

//Increase number of incoming connections backlog::
# backlog,Socket 监听队列长度
sysctl -w net.core.somaxconn=32768
sysctl -w net.ipv4.tcp_max_syn_backlog=16384
sysctl -w net.core.netdev_max_backlog=16384

//Local Port Range:
# 可用知名端口范围
sysctl -w net.ipv4.ip_local_port_range="1000 65535"

// Read/Write Buffer for TCP connections:
# TCP Socket 读写 Buffer 设置:
sysctl -w net.core.rmem_default=262144
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=16777216

#sysctl -w net.ipv4.tcp_mem='16777216 16777216 16777216'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216'

Connection Tracking:

# TCP 连接追踪设置:
sysctl -w net.nf_conntrack_max=1000000
sysctl -w net.netfilter.nf_conntrack_max=1000000
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

// The TIME-WAIT Buckets Pool, Recycling and Reuse:
# TIME-WAIT Socket 最大数量、回收与重用设置:
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576
# 注意: 不建议开启該设置,NAT模式下可能引起连接RST
# net.ipv4.tcp_tw_recycle = 1
# net.ipv4.tcp_tw_reuse = 1

// Timeout for FIN-WAIT-2 sockets:
# FIN-WAIT-2 Socket 超时设置:
sysctl -w net.ipv4.tcp_fin_timeout = 15

Erlang VM Tuning:

# 优化设置 Erlang 虚拟机启动参数,配置文件 emqttd/etc/emq.conf:
>> emqttd/etc/emq.conf:
## Erlang Process Limit(重要)
## Max number of Erlang proccesses.
## A MQTT client consumes two proccesses.
## The value should be larger than max_clients * 2
node.process_limit = 2097152

## Sets the maximum number of simultaneously existing ports for this system(重要)
## Max number of Erlang Ports. A MQTT client consumes one port. The value should be larger than max_clients.
node.max_ports = 1048576

The EMQ Broker(EMQ消息服务器参数):

// Tune the acceptor pool, max_clients limit and sockopts for TCP listener in etc/emqttd.config:
## TCP Listener
# 设置 TCP 监听器的 Acceptor 池大小,最大允许连接数。配置文件 emqttd/etc/emq.conf:
listener.tcp.external = 0.0.0.0:1883
listener.tcp.external.acceptors = 64
listener.tcp.external.max_clients = 1000000

Test Client(另一台专门测试的服务器):

sysctl -w net.ipv4.ip_local_port_range="500 65535"
sysctl -w fs.file-max=1000000  # 注:这个在上面已经解释
echo 1000000 > /proc/sys/fs/nr_open
ulimit -n 100000

Benchmark使用

并发连接测试工具: http://github.com/emqtt/emqtt_benchmark:

安装:

make

emqtt_bench_sub的使用:

// -c, --count: 最大client数量(最高60K)
// -i, --interval: 连接broker的间隔,默认为10
// -t, --topic
// -q, --qos: 默认为0
./emqtt_bench_sub -h 10.140.2.6 -c 50000 -i 10 -t bench/%i -q 2

emqtt_bench_pub的使用:

// -s, --size: payload size [default: 256]
// -I, --interval_of_msg: 发布消息的间隔,默认为1000
./emqtt_bench_pub -h 10.140.2.6 -c 100 -I 10 -t bench/%i -s 256

认证相关:

-u, --username     username for connecting to server
-P, --password     password for connecting to server
-k, --keepalive    keep alive in seconds [default: 300]
-C, --clean        clean session [default: true]

Local interface:

// --ifaddr: local ipaddress or interface address
./emqtt_bench_pub --ifaddr 192.168.1.10
./emqtt_bench_sub --ifaddr 192.168.2.10

One-way SSL Socket:

// -h, --host[default: localhost]
// -p, --port[default: 1883]
./emqtt_bench_sub -c 100 -i 10 -t bench/%i -p 8883 -S
./emqtt_bench_pub -c 100 -I 10 -t bench/%i -p 8883 -s 256 -S

Two-way SSL Socket:

./emqtt_bench_sub -c 100 -i 10 -t bench/%i -p 8883 --certfile path/to/client-cert.pem --keyfile path/to/client-key.pem
./emqtt_bench_pub -c 100 -i 10 -t bench/%i -s 256 -p 8883 --certfile path/to/client-cert.pem --keyfile path/to/client-key.pem