利用eclipse编写自定义hive udf函数

在做日志分析的过程中，用到了hadoop框架中的hive，不过有些日志处理用hive中的函数处理显得力不从心，就需要用udf来进行扩展处理了

1  在eclipse中新建java project   hiveudf   然后新建class  package(com.afan)  name(UDFLower)

2  添加jar library  hadoop-0.20.2-core.jar   hive-exec-0.7.0-cdh3u0.jar两个文件到project

3  编写代码

    package com.afan;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class UDFLower extends UDF{
public Text evaluate(final Text s){
if (null == s){
return null;
}
return new Text(s.toString().toLowerCase());
}
}
4  编译输出打包文件为udf_hive.jar
5 将udf_hive.jar放入配置好的linux系统的文件夹中路径为/home/udf/udf_hive.jar

6 打开hive命令行测试

   hive> add jar /home/udf/udf_hive.jar;

Added
udf_hive.jar to class path

Added resource: udf_hive.jar

创建udf函数

hive> create temporary function my_lower as 'com.afan.UDFLower';

创建测试数据

hive> create table dual (info string);

导入数据文件data.txt

data.txt文件内容为

WHO

AM

I

HELLO

hive>
load data local inpath '/home/data/data.txt' into table dual;

hive>
select info from dual;

Total
MapReduce jobs = 1

Launching Job 1 out of 1

Number of reduce tasks is set to 0 since there's no reduce operator

Starting Job = job_201105150525_0003, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201105150525_0003
Kill Command = /usr/local/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201105150525_0003

2011-05-15 06:46:05,459 Stage-1 map = 0%,  reduce = 0%

2011-05-15 06:46:10,905 Stage-1 map = 100%,  reduce = 0%

2011-05-15 06:46:13,963 Stage-1 map = 100%,  reduce = 100%

Ended Job = job_201105150525_0003

OK

WHO

AM

I

HELLO

使用udf函数

hive> select my_lower(info) from dual;

Total MapReduce jobs = 1

Launching Job 1 out of 1

Number of reduce tasks is set to 0 since there's no reduce operator

Starting Job = job_201105150525_0002, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201105150525_0002
Kill Command = /usr/local/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201105150525_0002

2011-05-15 06:43:26,100 Stage-1 map = 0%,  reduce = 0%

2011-05-15 06:43:34,364 Stage-1 map = 100%,  reduce = 0%

2011-05-15 06:43:37,484 Stage-1 map = 100%,  reduce = 100%

Ended Job = job_201105150525_0002

OK

who

am

i

hello

经测试成功通过

参考文章http://landyer.iteye.com/blog/1070377