java.io.IOException: No FileSystem for scheme: file spark hadoop

问题描述

// 创建一个Scala版本的Spark Context
val conf = new     SparkConf().setAppName("wordCount").setMaster("local")
val sc = new SparkContext(conf)
// 读取我们的输入数据
val input = sc.textFile(logFile)
// 把它切分成一个个单词
val words = input.flatMap(line => line.split(" "))
// 转换为键值对并计数
val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}
// 将统计出来的单词总数存入一个文本文件，引发求值
println(counts.collect().mkString(","))

结果报错如下：

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: file
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:658)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:436)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:409)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at scala.Option.map(Option.scala:145)
......

解决方案(加入了三句话)：

// 创建一个Scala版本的Spark Context
val conf = new SparkConf().setAppName("wordCount").setMaster("local")
val sc = new SparkContext(conf)

val h_conf = sc.hadoopConfiguration

h_conf.set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName)
h_conf.set("fs.file.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getName)

// 读取我们的输入数据
val input = sc.textFile(logFile)
// 把它切分成一个个单词
val words = input.flatMap(line => line.split(" "))
// 转换为键值对并计数
val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}
// 将统计出来的单词总数存入一个文本文件，引发求值
println(counts.collect().mkString(","))

报错原因：