spark SQL Running the Thrift JDBC/ODBC server
2015-03-04 17:22
393 查看
Running the Thrift JDBC/ODBC server
1:运行./sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.server2.thrift.bind.host=feng02 --master spark://feng02:7077 --driver-class-path /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/lib/mysql-connector-java-5.1.32-bin.jar --executor-memory
1g
端口:10000
服务器:feng02
spark master:spark://feng02:7077
driver-class-path:mysql驱动包(hive配置的)
[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.server2.thrift.bind.host=feng02 --master spark://feng02:7077 --driver-class-path /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/lib/mysql-connector-java-5.1.32-bin.jar starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/sbin/../logs/spark-jifeng-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-feng02.out
2:运行beeline
Now you can use beeline to test the Thrift JDBC/ODBC server:
./bin/beeline
[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/beeline Spark assembly has been built with Hive, including Datanucleus jars on classpath Beeline version ??? by Apache Hive
3:连接server
参考:https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineExample
beeline> !connect jdbc:hive2://feng02:10000 jifeng jifeng org.apache.hive.jdbc.HiveDriver Connecting to jdbc:hive2://feng02:10000 log4j:WARN No appenders could be found for logger (org.apache.thrift.transport.TSaslTransport). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Connected to: Spark SQL (version 1.2.0) Driver: null (version null) Transaction isolation: TRANSACTION_REPEATABLE_READ
4:查询
0: jdbc:hive2://feng02:10000> show tables; +----------------+ | result | +----------------+ | course | | hbase_table_1 | | pokes | | student | +----------------+ 4 rows selected (2.723 seconds)
0: jdbc:hive2://feng02:10000> select * from student; +-----+----------+------+ | id | name | age | +-----+----------+------+ | 1 | nick | 24 | | 2 | doping | 25 | | 3 | caizhi | 26 | | 4 | liaozhi | 27 | | 5 | wind | 30 | +-----+----------+------+ 5 rows selected (10.554 seconds) 0: jdbc:hive2://feng02:10000> select a.*,b.* from student a join course b where a.id=b.id ; +-----+----------+------+-----+-----+-----+-----+-----+ | id | name | age | id | c1 | c2 | c3 | c4 | +-----+----------+------+-----+-----+-----+-----+-----+ | 1 | nick | 24 | 1 | 英语 | 中文 | 法文 | 日文 | | 2 | doping | 25 | 2 | 中文 | 法文 | | | | 3 | caizhi | 26 | 3 | 中文 | 法文 | 日文 | | | 4 | liaozhi | 27 | 4 | 中文 | 法文 | 拉丁 | | | 5 | wind | 30 | 5 | 中文 | 法文 | 德文 | | +-----+----------+------+-----+-----+-----+-----+-----+ 5 rows selected (2.33 seconds)
4:Java JDBC连接
package demo.test; import java.sql.*; public class Pretest { public static void main( String args[] ) throws SQLException , ClassNotFoundException { String jdbcdriver="org.apache.hive.jdbc.HiveDriver"; String jdbcurl="jdbc:hive2://feng02:10000"; String username="scott"; String password="tiger"; Class.forName(jdbcdriver); Connection c = DriverManager.getConnection(jdbcurl,username,password); Statement st = c.createStatement(); print( "num should be 1 " , st.executeQuery("select * from student")); // TODO indexing } static void print( String name , ResultSet res ) throws SQLException { System.out.println( name); ResultSetMetaData meta=res.getMetaData(); //System.out.println( "\t"+res.getRow()+"条记录"); String str=""; for(int i=1;i<=meta.getColumnCount();i++){ str+=meta.getColumnName(i)+" "; //System.out.println( meta.getColumnName(i)+" "); } System.out.println("\t"+str); str=""; while ( res.next() ){ for(int i=1;i<=meta.getColumnCount();i++){ str+= res.getString(i)+" "; } System.out.println("\t"+str); str=""; } } }
上面是运行参数
结果显示:
num should be 1 id name age 1 nick 24 2 doping 25 3 caizhi 26 4 liaozhi 27 5 wind 30
相关文章推荐
- SparkSQL使用之JDBC代码访问Thrift JDBC Server
- sparkSQL里 sql语句,dataframe,Thrift Server JDBC都可以实现对数据的查询,过滤等操作, 哪这3种情况分别是什么情况下使用
- SparkSQL ThriftServer服务的使用和程序中JDBC的连接
- SparkSQL ThriftServer服务的使用和程序中JDBC的连接
- SparkSQL使用之Thrift JDBC server
- SQL Server set nocount on 与JDBC: The statement did not return a result
- sparkSQL1.1入门之七:ThriftServer和CLI
- Spark SQL Thrift Server 配置 Kerberos身份认证和权限管理
- Spark SQL thrift server 部署
- tomcat The web application [/zzti] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServe
- How to move the databases that are used by SharePoint Portal Server 2003 to a computer that is running SQL Server
- Caused by: java.sql.SQLException: The MySQL server is running with the --read-only option so it cann
- registered the JDBC driver [com.microsoft.jdbc.sqlserver.SQLServerDriver] but failed to unregister
- The web application [/zzti] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver
- com.microsoft.sqlserver.jdbc.SQLServerException: The server version is not suppo
- spark-thrift-server 执行spark-sql 的OOM GC异常.
- SQL server, get all running sessions on the SQL server
- Unable to list the columns. [Microsoft][ODBC SQL Server Driver][SQL Server]无法预定义语句。
- sparkSQL1.2.1入门之七:ThriftServer和CLI
- 分布式sparkSQL引擎应用:从远程通过thriftServer连接spark集群处理hive中的数据