How to access HBase from spark-shell using YARN as the master on CDH 5.3 and Spark 1.2
2016-06-20 10:51
871 查看
How to access HBase from spark-shell using YARN as the master on CDH 5.3 and Spark 1.2
http://somelittlebits.blogspot.com/2015/01/how-to-access-hbase-from-spark-shell.html
How to access HBase from spark-shell using YARN as the master on CDH 5.3 and Spark 1.2
From terminal:# export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-server.jar:/etc/hbase/conf/hbase-site.xml
# spark-shell --master yarn-client
Now you can access HBase from the Spark shell prompt:
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
val tableName = "My_HBase_Table_Name"
val hconf = HBaseConfiguration.create()
hconf.set(TableInputFormat.INPUT_TABLE, tableName)
val admin = new HBaseAdmin(hconf)
if (!admin.isTableAvailable(tableName)) {
val tableDesc = new HTableDescriptor(tableName)
admin.createTable(tableDesc)
}
val hBaseRDD = sc.newAPIHadoopRDD(hconf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
val result = hBaseRDD.count()
Thanks to these refs for pointers:
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/44744
http://apache-spark-user-list.1001560.n3.nabble.com/HBase-and-non-existent-TableInputFormat-td14370.html
相关文章推荐
- hdfs shell 命令
- Windows 7上执行Cake 报错原因是Powershell 版本问题
- powershell1.0简体中文版下载
- WinExec,ShellExecute ,CreateProcess 区别
- 编写"优美"的SHELLCODE
- Shell的命令行参数
- shell下的基础程序题练习
- [shell基础]——read命令
- shell脚本:练习题
- Shell 脚本知识回顾 (一) —— 基础篇
- [shell基础]——echo命令
- Shell 脚本知识回顾 (二) —— Shell变量
- Shell基本知识
- Shell 脚本知识回顾 (三) —— 替换、运算符、字符串、数组
- Shell 脚本知识回顾 (四) —— Shell 命令及Shell 相关语句
- Shell 脚本知识回顾 (五) —— Shell 循环
- Shell 脚本知识回顾 (六) —— Shell 函数
- Shell 脚本中如何使用make命令
- linux shell 流程控制
- shell脚本应用小程序