spark-sql之 Caused by: MetaException(message:Version information not found in metastore. )
2017-09-12 17:06
555 查看
试着使用spark-sql访问hive表,启动后spark-sql --driver-class-path /home/hadoop/bigdata/hive/lib/mysql-connector-java.jar
报如下错误:
17/09/12 16:47:44 INFO metastore.ObjectStore: Initialized ObjectStore
17/09/12 16:47:45 WARN metadata.Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:114)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
..........................................
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 17 more
Caused by: MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 22 more
17/09/12 16:47:45 INFO util.ShutdownHookManager: Shutdown hook called
17/09/12 16:47:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-69f57d6a-4a3d-4826-b0a0-5e3549aa2951
元数据是hive中存的,应该还是与hive相关的配置有关。经查,把hive-site.xml中如下配置项改为false可归避此问题。(启动spark-sql前曾将hive下的hive-site.xml copy至spark的conf下,并在spark-env.sh中设了$HIVE_HOME)
<property>
<name>hive.metastore.schema.verification</name>
<value>true</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures proper metastore schema migration.
(Default)
False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
将此选配设为false后将不再检查metastore的schema的一致性,先前在hive升级后此选项是true时访问是正常的,到spark-sql中报错,猜测可能是spark中支持的hive版本与metastore对应的hive版本不一致导致的。
系统中hive是2.1.1, spark-sql启动后显示的hive版本是1.2.1,看来确实是版本兼容的问题。
17/09/12 16:59:09 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is hdfs://master:9000/hive/warehouse
报如下错误:
17/09/12 16:47:44 INFO metastore.ObjectStore: Initialized ObjectStore
17/09/12 16:47:45 WARN metadata.Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:114)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
..........................................
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 17 more
Caused by: MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664)
at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.verifySchema(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 22 more
17/09/12 16:47:45 INFO util.ShutdownHookManager: Shutdown hook called
17/09/12 16:47:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-69f57d6a-4a3d-4826-b0a0-5e3549aa2951
元数据是hive中存的,应该还是与hive相关的配置有关。经查,把hive-site.xml中如下配置项改为false可归避此问题。(启动spark-sql前曾将hive下的hive-site.xml copy至spark的conf下,并在spark-env.sh中设了$HIVE_HOME)
<property>
<name>hive.metastore.schema.verification</name>
<value>true</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures proper metastore schema migration.
(Default)
False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
将此选配设为false后将不再检查metastore的schema的一致性,先前在hive升级后此选项是true时访问是正常的,到spark-sql中报错,猜测可能是spark中支持的hive版本与metastore对应的hive版本不一致导致的。
系统中hive是2.1.1, spark-sql启动后显示的hive版本是1.2.1,看来确实是版本兼容的问题。
17/09/12 16:59:09 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is hdfs://master:9000/hive/warehouse
相关文章推荐
- Hive 0.12 Caused by: MetaException(message:Version information not found in metastore. )解决方法
- spark MetaException(message:Version information not found in metastore. )
- hive :MetaException(message:Version information not found in metastore. )
- 启动hive2.1.1遇到的异常,Caused by: MetaException(message:Version information not found in metastore. )
- hive :MetaException(message:Version information not found in metastore. )
- 启动spark-sql报错Caused by: MetaException(message:Version information not found
- Caused by: MetaException(message:Hive Schema version 2.1.0 does not match metastore's schema version
- 解决:ould not find a getter for porDate in class com.bc.bean.PortalSheet Caused by: org.hibernate.PropertyNotFoundException: Could
- Caused by: java.lang.ClassNotFoundException: *** in loader dalvik.sy
- Android studio报:Caused by: java.lang.ClassNotFoundException: Didn't find class "xhs.com.view.ParentV
- Caused by: java.lang.ClassNotFoundException: Didn't find class "com.***.client.v1.Mai
- Caused by: Android.content.res.Resources$NotFoundException: Resource ID #0x7f0c0001
- Caused by: java.lang.ClassNotFoundException: org.hibernate.service.jta.platform.spi.JtaPlatform
- Caused by: java.lang.ClassNotFoundException: Didn't find class "com.zyh.testswiperefresh.MainActivit
- Phoenix 连接hbase 异常:Caused by: java.lang.ClassNotFoundException: org.iq80.snappy.CorruptionException
- Caused by: android.content.res.Resources$NotFoundException: String resource ID #0x0
- 【报错】spring整合activeMQ,pom.xml文件缺架包,启动报错:Caused by: java.lang.ClassNotFoundException: org.apache.xbean.spring.context.v2.XBeanNamespaceHandler
- Android项目编译的时候出现:Caused by: java.lang.ClassNotFoundException: com.example.aaa.MainActivity
- selenium之 坑(StaleElementReferenceException: Message: Element not found in the cache...)
- Caused by: java.lang.ClassNotFoundException: com.alibaba.fastjson.JSONObject