hadoop.terasort测试

硬件配置：

node configuration: 2*4-core 16GB-ram 4*1T-storage

node number: 11
软件配置(其他是默认设置)：
replication:
1
---------------------------------

测试过程中调节的参数：
mapred.tasktracker.map.tasks.maximum=4(共八个cores,
留一个给datanode和tasktracker使用)
mapred.tasktracker.reduce.tasks.maximum=3

---------------------------------

测试性能的参数：

调节文件块大小：64MB->128MB

调节:

<property>

<name>mapred.map.tasks</name>

<value>2</value>

<description>The default number of
map tasks per job.

Ignored when mapred.job.tracker is "local".

</description>

</property>

[bin/hadoop fs -rmr terasort/input-GB001]
bin/hadoop jar hadoop-0.20.2-examples.jar teragen
10000000
terasort/input-GB001

Generating 10000000 using 2 maps with step of 5000000

10/07/27 12:27:39 INFO mapred.JobClient: Running job:
job_201007271223_0003

10/07/27 12:27:40 INFO mapred.JobClient: map 0%
reduce 0%

10/07/27 12:27:54 INFO mapred.JobClient: map 53%
reduce 0%

10/07/27 12:28:00 INFO mapred.JobClient: map 100%
reduce 0%

10/07/27 12:28:02 INFO mapred.JobClient: Job complete:
job_201007271223_0003

10/07/27 12:28:02 INFO mapred.JobClient: Counters: 6

10/07/27 12:28:02 INFO
mapred.JobClient: Job
Counters

10/07/27 12:28:02 INFO
mapred.JobClient:
Launched map tasks=2

10/07/27 12:28:02 INFO
mapred.JobClient:
FileSystemCounters

10/07/27 12:28:02 INFO
mapred.JobClient:
HDFS_BYTES_WRITTEN=1000000000

10/07/27 12:28:02 INFO
mapred.JobClient: Map-Reduce
Framework

10/07/27 12:28:02 INFO
mapred.JobClient:
Map input records=10000000

10/07/27 12:28:02 INFO
mapred.JobClient:
Spilled Records=0

10/07/27 12:28:02 INFO
mapred.JobClient:
Map input bytes=10000000

10/07/27 12:28:02 INFO
mapred.JobClient:
Map output records=10000000
tersgen测试：

hadoop jar hadoop/hadoop-*-examples.jar
teragen
10 terasort/input-KB001

15s

hadoop jar hadoop/hadoop-*-examples.jar
teragen
10000 terasort/input-MB001

13s

hadoop jar hadoop/hadoop-*-examples.jar teragen
10000000 terasort/input-GB001

22s

hadoop jar hadoop/hadoop-*-examples.jar teragen
20000000 terasort/input-GB002

34s

hadoop jar hadoop/hadoop-*-examples.jar teragen
30000000 terasort/input-GB003

46s

hadoop jar hadoop/hadoop-*-examples.jar teragen
40000000 terasort/input-GB004

55s

hadoop jar hadoop/hadoop-*-examples.jar teragen
50000000 terasort/input-GB005

70s

hadoop jar hadoop/hadoop-*-examples.jar teragen 100000000
terasort/input-GB010

122s(mapred.map.tasks=02)

066s(mapred.map.tasks=04)

048s(mapred.map.tasks=06)

045s(mapred.map.tasks=08)

041s(mapred.map.tasks=09)

038s(mapred.map.tasks=10)

034s(mapred.map.tasks=11)Node
number

034s(mapred.map.tasks=12)

034s(mapred.map.tasks=13)

030s(mapred.map.tasks=14)

030s(mapred.map.tasks=15)

030s(mapred.map.tasks=16)

028s(mapred.map.tasks=20)

028s(mapred.map.tasks=22)2CPU*11Node=22CPU

028s(mapred.map.tasks=23)

028s(mapred.map.tasks=24)

028s(mapred.map.tasks=25)

028±1s(mapred.map.tasks=26)

028±1s(mapred.map.tasks=27)

028±1s(mapred.map.tasks=28)

028±1s(mapred.map.tasks=28)

028±1s(mapred.map.tasks=28)

028s(mapred.map.tasks=30)

029s(mapred.map.tasks=35)

030±1s(mapred.map.tasks=44) available
map number=4Map*11Node

043s(mapred.map.tasks=100)

067s(mapred.map.tasks=200)

------------------------------------------------------------------------------------

bin/hadoop fs -cat terasort/input-GB001/part-00000

.t^#\|v$2\
0AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDDEEEEEEEEEEFFFFFFFFFFGGGGGGGGGGHHHHHHHH

75@~?'WdUF
1IIIIIIIIIIJJJJJJJJJJKKKKKKKKKKLLLLLLLLLLMMMMMMMMMMNNNNNNNNNNOOOOOOOOOOPPPPPPPP

w[o||:N&H,
2QQQQQQQQQQRRRRRRRRRRSSSSSSSSSSTTTTTTTTTTUUUUUUUUUUVVVVVVVVVVWWWWWWWWWWXXXXXXXX

------------------------------------------------------------------------------------

bin/hadoop jar hadoop-0.20.2-examples.jar terasort
terasort/input-GB001 terasort/output-GB001

10/07/27 00:11:05 INFO
terasort.TeraSort: starting

10/07/27 00:11:05 INFO mapred.FileInputFormat: Total input paths to
process : 2

10/07/27 00:11:06 INFO util.NativeCodeLoader: Loaded the
native-hadoop library

10/07/27 00:11:06 INFO zlib.ZlibFactory: Successfully loaded
& initialized native-zlib library

10/07/27 00:11:06 INFO compress.CodecPool: Got brand-new
compressor

Making 1 from 100000 records

Step size is 100000.0

10/07/27 00:11:06 INFO mapred.JobClient: Running job:
job_201007270004_0003

10/07/27 00:11:07 INFO mapred.JobClient: map 0%
reduce 0%

10/07/27 00:11:21 INFO mapred.JobClient: map 50%
reduce 0%

10/07/27 00:11:24 INFO mapred.JobClient: map 100%
reduce 0%

10/07/27 00:11:33 INFO mapred.JobClient: map 100%
reduce 14%

10/07/27 00:11:36 INFO mapred.JobClient: map 100%
reduce 25%

10/07/27 00:11:39 INFO mapred.JobClient: map 100%
reduce 33%

10/07/27 00:11:54 INFO mapred.JobClient: map 100%
reduce 69%

10/07/27 00:11:57 INFO mapred.JobClient: map 100%
reduce 74%

10/07/27 00:12:00 INFO mapred.JobClient: map 100%
reduce 79%

10/07/27 00:12:03 INFO mapred.JobClient: map 100%
reduce 83%

10/07/27 00:12:06 INFO mapred.JobClient: map 100%
reduce 88%

10/07/27 00:12:09 INFO mapred.JobClient: map 100%
reduce 93%

10/07/27 00:12:15 INFO mapred.JobClient: map 100%
reduce 100%

10/07/27 00:12:17 INFO mapred.JobClient: Job complete:
job_201007270004_0003

10/07/27 00:12:17 INFO mapred.JobClient: Counters: 19

10/07/27 00:12:17 INFO
mapred.JobClient: Job
Counters

10/07/27 00:12:17 INFO
mapred.JobClient:
Launched reduce tasks=1

10/07/27 00:12:17 INFO
mapred.JobClient:
Rack-local map tasks=4

10/07/27 00:12:17 INFO
mapred.JobClient:
Launched map tasks=16

10/07/27 00:12:17 INFO
mapred.JobClient:
Data-local map tasks=12

10/07/27 00:12:17 INFO
mapred.JobClient:
FileSystemCounters

10/07/27 00:12:17 INFO
mapred.JobClient:
FILE_BYTES_READ=2382257412

10/07/27 00:12:17 INFO
mapred.JobClient:
HDFS_BYTES_READ=1000057358

10/07/27 00:12:17 INFO
mapred.JobClient:
FILE_BYTES_WRITTEN=3402255956

10/07/27 00:12:17 INFO
mapred.JobClient:
HDFS_BYTES_WRITTEN=1000000000

10/07/27 00:12:17 INFO
mapred.JobClient: Map-Reduce
Framework

10/07/27 00:12:17 INFO
mapred.JobClient:
Reduce input groups=10000000

10/07/27 00:12:17 INFO
mapred.JobClient:
Combine output records=0

10/07/27 00:12:17 INFO
mapred.JobClient:
Map input records=10000000

10/07/27 00:12:17 INFO
mapred.JobClient:
Reduce shuffle bytes=951549114

10/07/27 00:12:17 INFO
mapred.JobClient:
Reduce output records=10000000

10/07/27 00:12:17 INFO
mapred.JobClient:
Spilled Records=33355441

10/07/27 00:12:17 INFO
mapred.JobClient:
Map output bytes=1000000000

10/07/27 00:12:17 INFO
mapred.JobClient:
Map input bytes=1000000000

10/07/27 00:12:17 INFO
mapred.JobClient:
Combine input records=0

10/07/27 00:12:17 INFO
mapred.JobClient:
Map output records=10000000

10/07/27 00:12:17 INFO
mapred.JobClient:
Reduce input records=10000000

10/07/27 00:12:17 INFO terasort.TeraSort: done

hadoop jar
hadoop-0.20.2-examples.jar terasort terasort/input~KB001
terasort/output~KB001
22s(2个map)

hadoop jar
hadoop-0.20.2-examples.jar terasort terasort/input~MB001
terasort/output~MB001

22s(2个map因为是批处理，所以省去了网络连接的1s)

hadoop jar hadoop-0.20.2-examples.jar terasort terasort/input~GB001
terasort/output~GB001

76s=22s+54s(16个map)

hadoop jar hadoop-0.20.2-examples.jar terasort
terasort/input~GB002 terasort/output~GB002

136s=22s+114s(30个map)

hadoop jar hadoop-0.20.2-examples.jar terasort
terasort/input~GB003 terasort/output~GB003

187s=22s+165s(46个map)

hadoop jar hadoop-0.20.2-examples.jar terasort
terasort/input~GB004 terasort/output~GB004

250s=22s+228s(60个map)

hadoop jar hadoop-0.20.2-examples.jar terasort
terasort/input~GB005 terasort/output~GB005

307s=22s+285s(76个map)

hadoop jar hadoop-0.20.2-examples.jar terasort terasort/input~GB010
terasort/output~GB010

793s=22s+771s(150个map)