thrift安装使用——python客户端通过thrift调用hive
2011-09-09 09:53
801 查看
Thrift是一个跨语言服务部署框架,最初由Facebook于2007年开发,后于2008年进入Apache孵化器(Apache Incubator)。
类似于SOAP,google protobuf,COM 和CORBA,Thrift通过定义一个中间定义语言和Thrift代码生成工具,生成指定语言的代码。
目前,Thrift支持C++,Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk和OCaml的代码生成。
1安装thrift
安裝需要使用到的套件:
sudo apt-get install libboost-dev libevent-dev python-dev automake pkg-config libtool flex bison g++
編譯與安裝thrift:
wget http://ftp.tcc.edu.tw/pub/Apache//incubator/thrift/0.5.0-incubating/thrift-0.5.0.tar.gz
tar -xvf thrift-0.5.0.tar.gz
mv thrift-0.5.0 thrift
cd thrift
./configure
make
sudo make install
2安装好python thrift模块,但是无法加载
>>> from thrift.transport import TTransport
Traceback (most recent call last):
from thrift.transport import TTransport
ImportError: No module named thrift.transport
然而模块文件存在,/usr/lib/python2.6/site-packages/thrift/transport/TTransport.py
$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/Numeric',
'/usr/lib/python2.6/dist-packages/PIL',
'/var/lib/python-support/python2.6',
'/var/lib/python-support/python2.6/gtk-2.0',
'/usr/local/lib/python2.6/dist-packages']
(note that "site-packages" is not in the list). I assume that I configured my thrift code when the paths were different in my system, and some upgrade changed python directories. So when I "make install" thrift, they are still copied into site-packages. Now, having thrift already installed there, it is a matter of adding this dir to the search path. This is done with sys.path.append, according to python's documentation:
>>> sys.path.append('/usr/lib/python2.6/site-packages')
>>> sys.path
['',
'/usr/lib/python2.6',
(...)
'/usr/local/lib/python2.6/dist-packages',
'/usr/lib/python2.6/site-packages']
>>> from thrift.transport import TTransport
(No error message now)
在.bashrc文件中加入:
export PYTHONPATH=/usr/lib/python2.6/site-packages
3启动Hive Thrift Server
(hive --service hiveserver)
afan@ubuntu:/usr/local/hadoop$ hive --service hiveserver
Starting Hive Thrift Server
Hive histroy file=/tmp/afan/hive_job_log_afan_201106272202_1106483111.txt
OK
4代码
#!/usr/bin/env python
import sys
sys.path.append('/usr/local/hadoop/hive/lib/py')
from hive_service import ThriftHive
from hive_service.ttypes import HiveServerException
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
def hiveExe(sql):
try:
transport = TSocket.TSocket('127.0.0.1', 10000)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ThriftHive.Client(protocol)
transport.open()
client.execute(sql)
print "The return value is : "
print client.fetchAll()
print "............"
transport.close()
except Thrift.TException, tx:
print '%s' % (tx.message)
if __name__ == '__main__':
hiveExe("select * from t_afan_test")
5执行
afan@ubuntu:~/py-impl$ python py-thrift.py
The return value is:
['[12,23,23,34]\t["what","are","this"]','[34,45,34,23,12]\t["who","am","i","are"]']
...................
afan@ubuntu:/usr/local/hadoop/hive/lib/py$ ls
fb303 hive_metastore hive_service thrift
fb303_scripts hive_serde queryplan
from:http://blog.sina.com.cn/s/blog_61c463090100tlt9.html
类似于SOAP,google protobuf,COM 和CORBA,Thrift通过定义一个中间定义语言和Thrift代码生成工具,生成指定语言的代码。
目前,Thrift支持C++,Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk和OCaml的代码生成。
1安装thrift
安裝需要使用到的套件:
sudo apt-get install libboost-dev libevent-dev python-dev automake pkg-config libtool flex bison g++
編譯與安裝thrift:
wget http://ftp.tcc.edu.tw/pub/Apache//incubator/thrift/0.5.0-incubating/thrift-0.5.0.tar.gz
tar -xvf thrift-0.5.0.tar.gz
mv thrift-0.5.0 thrift
cd thrift
./configure
make
sudo make install
2安装好python thrift模块,但是无法加载
>>> from thrift.transport import TTransport
Traceback (most recent call last):
from thrift.transport import TTransport
ImportError: No module named thrift.transport
然而模块文件存在,/usr/lib/python2.6/site-packages/thrift/transport/TTransport.py
$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/Numeric',
'/usr/lib/python2.6/dist-packages/PIL',
'/var/lib/python-support/python2.6',
'/var/lib/python-support/python2.6/gtk-2.0',
'/usr/local/lib/python2.6/dist-packages']
(note that "site-packages" is not in the list). I assume that I configured my thrift code when the paths were different in my system, and some upgrade changed python directories. So when I "make install" thrift, they are still copied into site-packages. Now, having thrift already installed there, it is a matter of adding this dir to the search path. This is done with sys.path.append, according to python's documentation:
>>> sys.path.append('/usr/lib/python2.6/site-packages')
>>> sys.path
['',
'/usr/lib/python2.6',
(...)
'/usr/local/lib/python2.6/dist-packages',
'/usr/lib/python2.6/site-packages']
>>> from thrift.transport import TTransport
(No error message now)
在.bashrc文件中加入:
export PYTHONPATH=/usr/lib/python2.6/site-packages
3启动Hive Thrift Server
(hive --service hiveserver)
afan@ubuntu:/usr/local/hadoop$ hive --service hiveserver
Starting Hive Thrift Server
Hive histroy file=/tmp/afan/hive_job_log_afan_201106272202_1106483111.txt
OK
4代码
#!/usr/bin/env python
import sys
sys.path.append('/usr/local/hadoop/hive/lib/py')
from hive_service import ThriftHive
from hive_service.ttypes import HiveServerException
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
def hiveExe(sql):
try:
transport = TSocket.TSocket('127.0.0.1', 10000)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ThriftHive.Client(protocol)
transport.open()
client.execute(sql)
print "The return value is : "
print client.fetchAll()
print "............"
transport.close()
except Thrift.TException, tx:
print '%s' % (tx.message)
if __name__ == '__main__':
hiveExe("select * from t_afan_test")
5执行
afan@ubuntu:~/py-impl$ python py-thrift.py
The return value is:
['[12,23,23,34]\t["what","are","this"]','[34,45,34,23,12]\t["who","am","i","are"]']
...................
afan@ubuntu:/usr/local/hadoop/hive/lib/py$ ls
fb303 hive_metastore hive_service thrift
fb303_scripts hive_serde queryplan
from:http://blog.sina.com.cn/s/blog_61c463090100tlt9.html
相关文章推荐
- thrift之python客户端安装和使用 以及卸载
- Thrift安装配置及使用python通过thrift连接HBase测试
- netty学习九:(window7上)python客户端通过thrift调用java服务端
- 使用Python通过Thrift接口访问HBase和Hive
- 哈工大语言云(LTP)本地安装使用及Python调用
- Python 中__init__.py的使用(通过包调用方法)
- Centos7.2 thrift-0.9.3 安装使用(cpp服务端,go客户端)
- 在windows7上,使用python通过thrift访问hbase
- 使用Thrift0.9.1实现跨语言调用Golang、Php、Python、Java
- 使用国内镜像通过pip安装python的一些包(转)
- java通过thrift调用python远程执行shell
- 使用国内镜像通过pip安装python 包
- MYSQL 5.7.17 安装到使用,用python调用mysql
- 使用python来操作hive(通过 pyhs2 和 HiveServer2)
- 使用国内镜像通过pip安装python包
- Hive 安装 python thrift 连接hiserver2
- python通过thrift方式连接hive
- [zz]使用thrift做c++,java和python的相互调用
- [python] linux安装mysql,设置mysql允许外网访问,并使用sqlalchemy在python中调用
- windows下python调用R语言的模块rpy2安装使用