大数据IMF传奇行动 scala IDE 内存不够问题解决

1、scalaIDE 运行提示报错：Please use a larger heap size

2、修改jdk 使用内存

找到eclispe 中window->preferences->Java->Installed JRE ，点击右侧的Edit 按钮，在编辑界面中的 “Default VM Arguments ”选项中，填入如下值即可。

-Xms128m -Xmx512m

3、修改读入windows本地文件的路径

val conf = new SparkConf() //创建SparkConf对象

conf.setAppName("Wow,My First Spark App!") //设置应用程序的名称，在程序运行的监控界面可以看到名称

conf.setMaster("local") //此时，程序在本地运行，不需要安装Spark集群

val lines = sc.textFile("G://IMFBigDataSpark2016//Bigdata_Software//spark-1.6.0-bin-hadoop2.6//spark-1.6.0-bin-hadoop2.6//spark-1.6.0-bin-hadoop2.6//README.md", 1) //读取本地文件并设置为一个Partion

4、运行搞定！

16/01/14 11:58:36 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,NODE_LOCAL, 1894 bytes)

16/01/14 11:58:36 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)

16/01/14 11:58:36 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks

16/01/14 11:58:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 17 ms

package : 1

For : 2

Programs : 1

processing. : 1

Because : 1

The : 1

cluster. : 1

its : 1

[run : 1

APIs : 1

have : 1

Try : 1

computation : 1

through : 1

5、hadoop 找不到的问题，将导入winutils.exe包来解决

16/01/14 11:58:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/01/14 11:58:26 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.