.NET ODBC与JAVA JDBC连接HIVE查询数据的性能比较
2016-12-23 10:31
816 查看
最近在做大数据相关项目，大数据平台编程一般用JAVA，但由于项目组中成员用C#较熟，因此，准备做两者连接HIVE读取数据的性能比较测试，如果相差较大，就考虑整个项目用JAVA开发，这样学习成本较高。比较后发现，两者性能差距不大，这样至少前端的开发可以利用C#的原有资源了。好了，废话不多说，上代码。
C#通过ODBC连接HIVE读取数据
C#连接HIVE使用MapRHive ODBC Connector，该软件可以下载地址 http://archive.mapr.com/tools/MapR-ODBC/MapR_Hive/。
1.安装完成后，打开 "ODBCData Source Administrator",新建 ODBC 连接，选择 "MapR Hive ODBC Connector"
2.DNS 重要参数配置如下图：
数据源名：自己定义一个
（1）host: hiveServer2 服务器的IP地址
（2）Port: hive Server 监听的服务端口，默认为10000，可以在 hive-site.xml 配置文件中查到
（3）hive server type:hiveserver2
（4）Authentication：如上图选择按 "user name" 访问的方式，并输入用户名
（5）点击 ”Test", 如下图连接成功，则OK
配置完成，测试成功后。编写控制台测试程序
private void btnConnectHive_Click(object sender, EventArgs e)
{
string dns = "DSN=MYHIVE;UID=hive;PWD=";
System.Diagnostics.Stopwatch stopwatch = new System.Diagnostics.Stopwatch();
stopwatch.Start(); // 开始监视代码
string sql = "select count(*) from wgmes.tq_zbs_if";
DataSet dsHivedata = HiveOdbcClient.Current.Query(dns, sql);
stopwatch.Stop(); // 停止监视
TimeSpan timeSpan = stopwatch.Elapsed; // 获取总时间
double hours = timeSpan.TotalHours; // 小时
double minutes = timeSpan.TotalMinutes; // 分钟
double seconds = timeSpan.TotalSeconds; // 秒数
double milliseconds = timeSpan.TotalMilliseconds; // 毫秒数
this.dgvHiveData.DataSource = dsHivedata.Tables[0];
this.txtBTime.Text = hours.ToString()+"h"+"\r\n"+minutes.ToString()+"m" + "\r\n" + seconds.ToString()+"s" + "\r\n" + milliseconds.ToString()+"ms";
}
其中，wgmes是数据库名，表tq_zbs_if是一个包含图片的11万条数据的表。
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data.Odbc;
using System.Data;
namespace hiveform
{
public class HiveOdbcClient
{
public static HiveOdbcClient Current
{
get { return new HiveOdbcClient(); }
}
/// <summary>
///
/// </summary>
/// <param name="context"></param>
public void ExcuteNoQuery(string dns, string sql)
{
OdbcConnection conn = null;
try
{
conn = new OdbcConnection(dns);
conn.Open();
OdbcCommand cmd = new OdbcCommand(sql, conn);
cmd.ExecuteNonQuery();
}
catch (Exception ex)
{
throw ex;
}
finally
{
if (null != conn)
{
conn.Close();
}
}
}
/// <summary>
///
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
public DataSet Query(string dns, string sql, string tblName = "tbl")
{
DataSet set = new DataSet();
OdbcConnection conn = null;
try
{
conn = new OdbcConnection(dns);
conn.Open();
OdbcCommand cmd = conn.CreateCommand();
cmd.CommandText = sql;
OdbcDataAdapter adapter = new OdbcDataAdapter(cmd);
adapter.Fill(set, tblName);
}
catch (Exception ex)
{
throw ex;
}
finally
{
if (null != conn)
{
conn.Close();
}
}
return set;
}
}
}
执行结果如下：
Java JDBC连接HIve
通过JDBC连接HIVE只需要两个JAR包：hadoop-common-2.7.2.jar和hive-jdbc-2.0.1-standalone.jar，hive-jdbc-2.0.1-standalone.jar可在相应的HIVE安装包中的jdbc文件夹下，
至于网上转载的资料说需要一大堆其他的Jar包都被hive-jdbc-2.0.1-standalone.jar集成了。注意，hive-jdbc-2.0.1-standalone.jar的版本号要与连接的Hive版本对应，第一次下载了最新的
2.1.1结果报错说连接无法解析。代码如下：
package testHive;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class testHive {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
private static String url = "jdbc:hive2://10.0.83.21:10000/default";
private static String user = "hive";
private static String password = "";
private static String sql = "";
private static ResultSet res;
public static void main(String[] args) {
Connection conn = null;
Statement stmt = null;
try {
conn = getConn();
stmt = conn.createStatement();
String tableName = "wgmes.tq_zbs_if";
// 执行 regular hive query 统计操作
long starTime=System.currentTimeMillis();
String result=countData(stmt, tableName);
long endTime=System.currentTimeMillis();
long TimeSpan=(endTime-starTime)/1000;
System.out.println("执行“regular hive query”运行结果:");
System.out.println("count ------>" + result+"\r\n"+"TimeCost=" + TimeSpan+"s");
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
} catch (SQLException e) {
e.printStackTrace();
System.exit(1);
} finally {
try {
if (conn != null) {
conn.close();
conn = null;
}
if (stmt != null) {
stmt.close();
stmt = null;
}
} catch (SQLException e) {
e.printStackTrace();
}
}
}
private static String countData(Statement stmt, String tableName)
throws SQLException {
sql = "select count(*) from " + tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
String results="";
while (res.next()) {
results= res.getString(1);
break;
}
return results;
}
private static void selectData(Statement stmt, String tableName)
throws SQLException {
sql = "select * from " + tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
System.out.println("执行 select * query 运行结果:");
while (res.next()) {
System.out.println(res.getInt(1) + "\t" + res.getString(2));
}
}
private static void loadData(Statement stmt, String tableName)
throws SQLException {
String filepath = "/home/hadoop01/data";
sql = "load data local inpath '" + filepath + "' into table "
+ tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
}
private static void describeTables(Statement stmt, String tableName)
throws SQLException {
sql = "describe " + tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
System.out.println("执行 describe table 运行结果:");
while (res.next()) {
System.out.println(res.getString(1) + "\t" + res.getString(2));
}
}
private static void showTables(Statement stmt, String tableName)
throws SQLException {
sql = "show tables '" + tableName + "'";
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
System.out.println("执行 show tables 运行结果:");
if (res.next()) {
System.out.println(res.getString(1));
}
}
private static void createTable(Statement stmt, String tableName)
throws SQLException {
sql = "create table "
+ tableName
+ " (key int, value string) row format delimited fields terminated by '\t'";
stmt.executeQuery(sql);
}
private static String dropTable(Statement stmt,String tableName) throws SQLException {
sql = "drop table " + tableName;
stmt.executeQuery(sql);
return tableName;
}
private static Connection getConn() throws ClassNotFoundException,
SQLException {
Class.forName(driverName);
Connection conn = DriverManager.getConnection(url, user, password);
return conn;
}
}
执行结果如下图：
C#通过ODBC连接HIVE读取数据
C#连接HIVE使用MapRHive ODBC Connector，该软件可以下载地址 http://archive.mapr.com/tools/MapR-ODBC/MapR_Hive/。
1.安装完成后，打开 "ODBCData Source Administrator",新建 ODBC 连接，选择 "MapR Hive ODBC Connector"
2.DNS 重要参数配置如下图：
数据源名：自己定义一个
（1）host: hiveServer2 服务器的IP地址
（2）Port: hive Server 监听的服务端口，默认为10000，可以在 hive-site.xml 配置文件中查到
（3）hive server type:hiveserver2
（4）Authentication：如上图选择按 "user name" 访问的方式，并输入用户名
（5）点击 ”Test", 如下图连接成功，则OK
配置完成，测试成功后。编写控制台测试程序
private void btnConnectHive_Click(object sender, EventArgs e)
{
string dns = "DSN=MYHIVE;UID=hive;PWD=";
System.Diagnostics.Stopwatch stopwatch = new System.Diagnostics.Stopwatch();
stopwatch.Start(); // 开始监视代码
string sql = "select count(*) from wgmes.tq_zbs_if";
DataSet dsHivedata = HiveOdbcClient.Current.Query(dns, sql);
stopwatch.Stop(); // 停止监视
TimeSpan timeSpan = stopwatch.Elapsed; // 获取总时间
double hours = timeSpan.TotalHours; // 小时
double minutes = timeSpan.TotalMinutes; // 分钟
double seconds = timeSpan.TotalSeconds; // 秒数
double milliseconds = timeSpan.TotalMilliseconds; // 毫秒数
this.dgvHiveData.DataSource = dsHivedata.Tables[0];
this.txtBTime.Text = hours.ToString()+"h"+"\r\n"+minutes.ToString()+"m" + "\r\n" + seconds.ToString()+"s" + "\r\n" + milliseconds.ToString()+"ms";
}
其中，wgmes是数据库名，表tq_zbs_if是一个包含图片的11万条数据的表。
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data.Odbc;
using System.Data;
namespace hiveform
{
public class HiveOdbcClient
{
public static HiveOdbcClient Current
{
get { return new HiveOdbcClient(); }
}
/// <summary>
///
/// </summary>
/// <param name="context"></param>
public void ExcuteNoQuery(string dns, string sql)
{
OdbcConnection conn = null;
try
{
conn = new OdbcConnection(dns);
conn.Open();
OdbcCommand cmd = new OdbcCommand(sql, conn);
cmd.ExecuteNonQuery();
}
catch (Exception ex)
{
throw ex;
}
finally
{
if (null != conn)
{
conn.Close();
}
}
}
/// <summary>
///
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
public DataSet Query(string dns, string sql, string tblName = "tbl")
{
DataSet set = new DataSet();
OdbcConnection conn = null;
try
{
conn = new OdbcConnection(dns);
conn.Open();
OdbcCommand cmd = conn.CreateCommand();
cmd.CommandText = sql;
OdbcDataAdapter adapter = new OdbcDataAdapter(cmd);
adapter.Fill(set, tblName);
}
catch (Exception ex)
{
throw ex;
}
finally
{
if (null != conn)
{
conn.Close();
}
}
return set;
}
}
}
执行结果如下：
Java JDBC连接HIve
通过JDBC连接HIVE只需要两个JAR包：hadoop-common-2.7.2.jar和hive-jdbc-2.0.1-standalone.jar，hive-jdbc-2.0.1-standalone.jar可在相应的HIVE安装包中的jdbc文件夹下，
至于网上转载的资料说需要一大堆其他的Jar包都被hive-jdbc-2.0.1-standalone.jar集成了。注意，hive-jdbc-2.0.1-standalone.jar的版本号要与连接的Hive版本对应，第一次下载了最新的
2.1.1结果报错说连接无法解析。代码如下：
package testHive;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class testHive {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
private static String url = "jdbc:hive2://10.0.83.21:10000/default";
private static String user = "hive";
private static String password = "";
private static String sql = "";
private static ResultSet res;
public static void main(String[] args) {
Connection conn = null;
Statement stmt = null;
try {
conn = getConn();
stmt = conn.createStatement();
String tableName = "wgmes.tq_zbs_if";
// 执行 regular hive query 统计操作
long starTime=System.currentTimeMillis();
String result=countData(stmt, tableName);
long endTime=System.currentTimeMillis();
long TimeSpan=(endTime-starTime)/1000;
System.out.println("执行“regular hive query”运行结果:");
System.out.println("count ------>" + result+"\r\n"+"TimeCost=" + TimeSpan+"s");
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
} catch (SQLException e) {
e.printStackTrace();
System.exit(1);
} finally {
try {
if (conn != null) {
conn.close();
conn = null;
}
if (stmt != null) {
stmt.close();
stmt = null;
}
} catch (SQLException e) {
e.printStackTrace();
}
}
}
private static String countData(Statement stmt, String tableName)
throws SQLException {
sql = "select count(*) from " + tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
String results="";
while (res.next()) {
results= res.getString(1);
break;
}
return results;
}
private static void selectData(Statement stmt, String tableName)
throws SQLException {
sql = "select * from " + tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
System.out.println("执行 select * query 运行结果:");
while (res.next()) {
System.out.println(res.getInt(1) + "\t" + res.getString(2));
}
}
private static void loadData(Statement stmt, String tableName)
throws SQLException {
String filepath = "/home/hadoop01/data";
sql = "load data local inpath '" + filepath + "' into table "
+ tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
}
private static void describeTables(Statement stmt, String tableName)
throws SQLException {
sql = "describe " + tableName;
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
System.out.println("执行 describe table 运行结果:");
while (res.next()) {
System.out.println(res.getString(1) + "\t" + res.getString(2));
}
}
private static void showTables(Statement stmt, String tableName)
throws SQLException {
sql = "show tables '" + tableName + "'";
System.out.println("Running:" + sql);
res = stmt.executeQuery(sql);
System.out.println("执行 show tables 运行结果:");
if (res.next()) {
System.out.println(res.getString(1));
}
}
private static void createTable(Statement stmt, String tableName)
throws SQLException {
sql = "create table "
+ tableName
+ " (key int, value string) row format delimited fields terminated by '\t'";
stmt.executeQuery(sql);
}
private static String dropTable(Statement stmt,String tableName) throws SQLException {
sql = "drop table " + tableName;
stmt.executeQuery(sql);
return tableName;
}
private static Connection getConn() throws ClassNotFoundException,
SQLException {
Class.forName(driverName);
Connection conn = DriverManager.getConnection(url, user, password);
return conn;
}
}
执行结果如下图：
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- java中使用JDBC连接mysql数据库查询数据
- java 利用jdbc连接hive查询示例
- java jdbc连接mysql 查询数据查找不到
- java连接memcached，并对比从数据库中查询数据与从memcached查询的性能
- JDBC/ODBC/.NET连接数据库/数据文件的连接字符串大全
- Hive连接MYSQL数据OK,查询出现in thread "Thread-18" java.lang.IllegalArgumentException:Does not contain a vali
- java连接memcached，并对比从数据库中查询数据与从memcached查询的性能
- java使用JDBC连接数据库，提高查询大量数据速度
- java项目中使用JDBC连接mysql数据库查询数据
- java JDBC 连接数据库查询数据与直接使用sql的疑问
- Java/jdbc连接数据库预处理只能查询一条数据？
- C#(.NET)数据访问连接、查询、插入等操作的封装类
- 使用JDBC-ODBC连接SQLServer数据库后读取数据库中表的数据代码
- ADO.NET实体数据模型多表连接查询中的实体键问题
- Java JDBC操作Image类型数据 添加和查询
- JAVA查询Oracle数据库集群连接字符串及其JDBC jar包选择.
- Java JDBC-ODBC 连接 Access 数据库 中文乱码解决
- om.microsoft.sqlserver.jdbc.SQLServerException: 到主机 的 TCP/IP 连接失败。 java.net.ConnectException: Connection refused: connect
- ADO.NET 的几个示例二，带连接数据查询方法
- ado.net entity framework 查询性能比较