IDEA，SparkSql读取HIve中的数据

传统Hive计算引擎为MapReduce，在Spark1.3版本之后，SparkSql正式发布，并且SparkSql与apache hive基本完全兼容，基于Spark强大的计算能力，使用Spark处理hive中的数据处理速度远远比传统的Hive快。
在idea中使用SparkSql读取HIve表中的数据步骤如下
1、首先，准备测试环境，将hadoop集群conf目录下的core-site.xml、hdfs-site.xml和Hive中conf目录下hive-site.xml拷贝在resources目录下

2、pom依赖

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.1</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.1</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.0.1</version>
</dependency>

<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.25</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.1</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.1</version>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.1</version>
</dependency>

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.4.11</version>
</dependency>

<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.41</version>
</dependency>

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.4.11</version>
</dependency>

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>2.2.0</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.1.0</version>
</dependency>

<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.3</version>
</dependency>

<dependency>
<groupId>com.github.sgroschupf</groupId>
<artifactId>zkclient</artifactId>
<version>0.1</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>2.3.0</version>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.11</artifactId>
<version>1.7.2</version>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.11</artifactId>
<version>1.7.2</version>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table_2.11</artifactId>
<version>1.7.2</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.1</version>
</dependency>

3、代码开发

import org.apache.spark.sql.{DataFrame, SparkSession}

object SparkSql_Hive {
def main(args: Array[String]): Unit = {
//创建SparkSession对象
val spark = SparkSession.builder()
.appName(this.getClass.getSimpleName)
.master("local[*]")
.config("dfs.client.use.datanode.hostname", "true")
.enableHiveSupport()
.getOrCreate()
//指定库名
val sql1 = "use mydb"
spark.sql(sql1)
//查看该库下的表结构
val sql2 = "show tables"
spark.sql(sql2).show()
//读取hivemydb库下per表
val sql3 = "select * from mydb.per"
spark.sql(sql3).show()
}
}

4、查看打印结果
显示mydb库下的所有表

可以看到SparkSql已经读取了Hive中的数据
5、测试中遇到的问题
（1）、找不到HIve相关的类

Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:778)
at com.yangshou.SparkSql_Hive$.main(SparkSql_Hive.scala:12)
at com.yangshou.SparkSql_Hive.main(SparkSql_Hive.scala)

通过查阅相关资料，最后认为是Spark版本不对，把pom文件中Spark2.1.0的版本改为2.0.1，最终解决问题