FastDFS 5.0.1 Nginx+fastdfs_nginx_model 1.15 vs HDFS httpFS hadoop2.7.1 http get 小文件性能对比

访问同一个文件

[root@vm7 ~]# ab -n 10000 -c 100 http://192.168.18.229/group1/M00/00/00/wKgS5VW7HyOABKiVAAARVIZ0lBE144.xml
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.18.229 (be patient)

Completed 1000 requests

Completed 2000 requests

Completed 3000 requests

Completed 4000 requests

Completed 5000 requests

Completed 6000 requests

Completed 7000 requests

Completed 8000 requests

Completed 9000 requests

Completed 10000 requests

Finished 10000 requests

Server Software: nginx/1.9.2

Server Hostname: 192.168.18.229

Server Port: 80

Document Path: /group1/M00/00/00/wKgS5VW7HyOABKiVAAARVIZ0lBE144.xml

Document Length: 4436 bytes

Concurrency Level: 100

Time taken for tests: 0.903 seconds

Complete requests: 10000

Failed requests: 0

Write errors: 0

Total transferred: 46220000 bytes

HTML transferred: 44360000 bytes

Requests per second: 11077.56 [#/sec] (mean)

Time per request: 9.027 [ms] (mean)

Time per request: 0.090 [ms] (mean, across all concurrent requests)

Transfer rate: 50000.46 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max

Connect: 0 3 1.4 3 7

Processing: 1 6 1.7 6 13

Waiting: 0 4 1.1 4 12

Total: 6 9 1.0 9 14

Percentage of the requests served within a certain time (ms)

50% 9

66% 9

75% 9

80% 9

90% 10

95% 11

98% 12

99% 14

100% 14 (longest request)

[root@vm7 ~]# ab -n 10000 -c 100 http://192.168.18.229:50070/webhdfs/v1/input/capacity-scheduler.xml?op=OPEN
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.18.229 (be patient)

Completed 1000 requests

Completed 2000 requests

Completed 3000 requests

Completed 4000 requests

Completed 5000 requests

Completed 6000 requests

Completed 7000 requests

Completed 8000 requests

Completed 9000 requests

Completed 10000 requests

Finished 10000 requests

Server Software: Jetty(6.1.26)

Server Hostname: 192.168.18.229

Server Port: 50070

Document Path: /webhdfs/v1/input/capacity-scheduler.xml?op=OPEN

Document Length: 0 bytes

Concurrency Level: 100

Time taken for tests: 15.164 seconds

Complete requests: 10000

Failed requests: 0

Write errors: 0

Non-2xx responses: 10000

Total transferred: 4570000 bytes

HTML transferred: 0 bytes

Requests per second: 659.45 [#/sec] (mean)

Time per request: 151.643 [ms] (mean)

Time per request: 1.516 [ms] (mean, across all concurrent requests)

Transfer rate: 294.30 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max

Connect: 0 3 6.8 2 86

Processing: 1 148 65.7 131 680

Waiting: 1 145 65.3 128 679

Total: 1 151 65.2 135 680

Percentage of the requests served within a certain time (ms)

50% 135

66% 152

75% 173

80% 182

90% 231

95% 262

98% 316

99% 424

100% 680 (longest request)

对比

Requests per second

11077.56/659.45=16.798180302

Nginx+fastdfs_nginx_model的 http get 处理速度接近HDFS httpFS的17倍.