Spark snappy

Spark Snappy 问题

In my experience, if you cd into the

/sparkDir/conf

and rename the spark-env.sh.template to spark-env.sh, and then set the JAVA_OPTSand hadoop_DIR, it works.

You will also have to edit this

/etc/profile

line:

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH

hive on spark：

[Error 30017]: Skipping stats aggregation by error org.apache.hadoop.hive.ql.metadata.HiveException: [Error 30001]: StatsPublisher cannot be initialized. There was a error in the initialization of StatsPublisher, and retrying might help. If you dont want the query to fail because accurate statistics could not be collected, set hive.stats.reliable=false

Table tmp.app_fin_assign_debt stats: [numFiles=2, numRows=-1, totalSize=80885949, rawDataSize=-1]

yarn logs 发现：

16/06/12 17:43:14 ERROR jdbc.JDBCStatsPublisher: Error during JDBC connection to jdbc:mysql://127.0.0.1:3306;databaseName=TempStatsStore;create=true.

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Cannot load connection class because of underlying exception: 'java.lang.NumberFormatException: For input string: "3306;databaseName=TempStatsStore;create=true"'.

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

at com.mysql.jdbc.Util.getInstance(Util.java:386)

连接Hive元数据库 配置hive-site

<property>

<name>hive.stats.jdbcdriver</name>

<value>com.mysql.jdbc.Driver</value>

<description>The JDBC driver for the database that stores temporary hive statistics.</description>

</property>

<property>

<name>hive.stats.dbconnectionstring</name>

<value>jdbc:mysql://******:3306;databaseName=TempStatsStore;create=true</value>

<description>The default connection string for the database that stores temporary hive statistics.</description>

</property>