Hive ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.RegexSerDe
2012-02-02 17:10
405 查看
在Hive的测试中,发现使用正则表达式的字段分隔方式时,总是进行不下去,创建表的语句如下:
通过查看Tracking URL,发现抛出类找不到的异常,类
解决方法:
将下面的配置语句,加在配置文件: $HIVE_INSTALL/conf/hive-site.xml中,value中hive-contrib-*.jar的路径为你机器上实际的放置,在$HIVE_INSTALL/lib目录下寻找。
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/develop/hive-0.8.0/lib/hive-contrib-0.8.0.jar</value>
<description>Added by tiger.zeng on 20120202.These JAR file are available to all users for all jobs</description>
</property>
退出hive shell, 重新启动hive shell, 可运行正常。
运行结果如下:
hive> select time,clause from mylog2;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201202021642_0003, Tracking URL = http://hdp0:50030/jobdetails.jsp?jobid=job_201202021642_0003
Kill Command = /home/hadoop/hadoop-0.20.203.0/bin/hadoop job -Dmapred.job.tracker=hdp0:9001 -kill job_201202021642_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-02-02 16:58:43,327 Stage-1 map = 0%, reduce = 0%
2012-02-02 16:58:49,375 Stage-1 map = 100%, reduce = 0%
2012-02-02 16:58:55,437 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201202021642_0003
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 1286 HDFS Write: 564 SUCESS
Total MapReduce CPU Time Spent: 0 msec
OK
NULL NULL
NULL NULL
NULL NULL
NULL NULL
2012-01-17 00:14:19,301 org.apache.hadoop.conf.Configuration -0 [main] DEBUG org.apache.hadoop.conf.Configuration
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
2012-01-17 00:14:19,314 org.apache.hadoop.util.NativeCodeLoader -13 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader
2012-01-17 00:14:19,314 org.apache.hadoop.util.NativeCodeLoader -13 [main] INFO org.apache.hadoop.util.NativeCodeLoader
2012-01-17 00:14:19,316 org.apache.hadoop.io.compress.zlib.ZlibFactory -15 [main] INFO org.apache.hadoop.io.compress.zlib.ZlibFactory
NULL NULL
Time taken: 21.28 seconds
hive>
参考:
create table mylog2(time string,clause string) row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' with serdeproperties ( "input.regex" = "(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}) - (.*) -(.*)" );
通过查看Tracking URL,发现抛出类找不到的异常,类
org.apache.hadoop.hive.contrib.serde2.RegexSerDe没有找到。
解决方法:
将下面的配置语句,加在配置文件: $HIVE_INSTALL/conf/hive-site.xml中,value中hive-contrib-*.jar的路径为你机器上实际的放置,在$HIVE_INSTALL/lib目录下寻找。
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/develop/hive-0.8.0/lib/hive-contrib-0.8.0.jar</value>
<description>Added by tiger.zeng on 20120202.These JAR file are available to all users for all jobs</description>
</property>
退出hive shell, 重新启动hive shell, 可运行正常。
运行结果如下:
hive> select time,clause from mylog2;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201202021642_0003, Tracking URL = http://hdp0:50030/jobdetails.jsp?jobid=job_201202021642_0003
Kill Command = /home/hadoop/hadoop-0.20.203.0/bin/hadoop job -Dmapred.job.tracker=hdp0:9001 -kill job_201202021642_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2012-02-02 16:58:43,327 Stage-1 map = 0%, reduce = 0%
2012-02-02 16:58:49,375 Stage-1 map = 100%, reduce = 0%
2012-02-02 16:58:55,437 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201202021642_0003
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 1286 HDFS Write: 564 SUCESS
Total MapReduce CPU Time Spent: 0 msec
OK
NULL NULL
NULL NULL
NULL NULL
NULL NULL
2012-01-17 00:14:19,301 org.apache.hadoop.conf.Configuration -0 [main] DEBUG org.apache.hadoop.conf.Configuration
NULL NULL
NULL NULL
NULL NULL
NULL NULL
NULL NULL
2012-01-17 00:14:19,314 org.apache.hadoop.util.NativeCodeLoader -13 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader
2012-01-17 00:14:19,314 org.apache.hadoop.util.NativeCodeLoader -13 [main] INFO org.apache.hadoop.util.NativeCodeLoader
2012-01-17 00:14:19,316 org.apache.hadoop.io.compress.zlib.ZlibFactory -15 [main] INFO org.apache.hadoop.io.compress.zlib.ZlibFactory
NULL NULL
Time taken: 21.28 seconds
hive>
参考:
Jasper Knulst 13020438802011年04月06日 星期三 06时51分20秒 | ![]() Link Here Print view |
Hi(ve), I created a table like this; create table testtable (veld1 STRING,veld2 STRING,veld3 STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' > WITH SERDEPROPERTIES ("input.regex" = "([a-z]{4}[0-9])þ([a-z]{4}[0-9])þ([a-z]{4}[0-9])") STORED AS TEXTFILE; The table is OK, select * from testtable shows the contents of the underlying HDFS file. However when I invoke a MR job by select veld2 from testtable, the MR job starts but I get mapper errors saying: "Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.RegexSerDe" I already copied the hive serde jar (in my case hive-serde-0.7.0-CDH3B4.jar) to $HADOOP_HOME/lib and restarted jobtracker/tasktrackers but that doesn't help. Cheers JasperShow footer-- Kind Regards \ Met Vriendelijke Groet, Jasper Knulst BI Consultant VLC Den Haag Gildeweg 5B 2632 BD Nootdorp M: +31 (0)6 19 66 75 11 T: +31 (0)15 764 07 50
Loren Siebert 13020443192011年04月06日 星期三 06时58分39秒 | ![]() Link Here Print view |
You need to tell Hive about the JAR. This is how I do it in hive-site.xml: <property> <name>hive.aux.jars.path</name> <value>file:///usr/lib/hive/lib/hive-contrib-0.7.0-CDH3B4.jar</value> <description>These JAR file are available to all users for all jobs</description> </property>Show quoted text
相关文章推荐
- Hive 启动报错 ClassNotFoundException: org.apache.hadoop.hive.service.HiveServer
- Hive ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found
- hive利用jdbc连接时报错:java.lang.ClassNotFoundException:org.apache.hadoop.hive.jdbc.HiveDriver
- hive启动报错ClassNotFoundException: org.apache.hadoop.hive.service.HiveServer
- Hive1.2.1 启动报错 ClassNotFoundException: org.apache.hadoop.hive.service.HiveServer
- java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HBaseSplit
- 安装sqoop-1.99.7报caused by java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
- java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableOutputFormat解决办法
- eclipse运行hadoop1版本错误如下 ClassNotFoundException:org.apache.hadoop.io.nativeio.NativeIO$POSIX$Stat
- 解决kylin sync table报错:MetaException(message:java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found
- jdbc链接hive报错:java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransport
- JDBC 连接 hive2 遇到的java.lang.ClassNotFoundException: org.apache.hive.jdbc.HiveDriver
- 解决Eclipse中运行WordCount出现 java.lang.ClassNotFoundException: org.apache.hadoop.examples.WordCount$Token
- 解决Eclipse中运行WordCount出现 java.lang.ClassNotFoundException: org.apache.hadoop.examples.WordCount$TokenizerMapper问题【转】
- java.lang.ClassNotFoundException: org.apache.hadoop.fs.CanUnbuffer
- 解决使用libhdfs.so连接hdfs出错问题java.lang.ClassNotFoundException: org.apache.hadoop.fs.F
- hbase无法启动Regionserver:ClassNotFoundException: org.apache.hadoop.util.PlatformName问题解决
- java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
- java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
- ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory