sqoop安装
2018-01-02 21:50
33 查看
一 概述
sqoop是apache下一款“Hadoop和关系数据库服务器之间传送数据”的工具。
导入数据:Mysql、Oracle导入数据到hadoop的HDFS、Hive、HBase等数据存储系统。
导出数据:从Hadoop的文件系统中导出数据到关系数据库mysql等。
Sqoop的本质还是一个命令行工具,和HDFS、Hive相比,并没有什么高深的理论。
把数据从非hadoop导到hadoop系统叫导入。
把数据从hadoop导到非hadoop系统叫导出。
二 工作机制
将导入或导出命令翻译成mapreduce程序来实现。
在翻译出的mapreduce中主要是对inputformat和outputformat进行定制。
三 Sqoop安装
1 下载Sqoop
sqoop是apache下一款“Hadoop和关系数据库服务器之间传送数据”的工具。
导入数据:Mysql、Oracle导入数据到hadoop的HDFS、Hive、HBase等数据存储系统。
导出数据:从Hadoop的文件系统中导出数据到关系数据库mysql等。
Sqoop的本质还是一个命令行工具,和HDFS、Hive相比,并没有什么高深的理论。
把数据从非hadoop导到hadoop系统叫导入。
把数据从hadoop导到非hadoop系统叫导出。
二 工作机制
将导入或导出命令翻译成mapreduce程序来实现。
在翻译出的mapreduce中主要是对inputformat和outputformat进行定制。
三 Sqoop安装
1 下载Sqoop
[root@centos opt]# wget http://archive.apache.org/dist/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz[/code]
2 安装Sqoop[root@centos opt]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz [root@centos opt]# mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop-1.4.6
3 修改sqoop-env.sh[root@centos conf]# vi sqoop-env.sh [root@centos conf]# cat sqoop-env.sh # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # included in all the hadoop scripts with source command # should not be executable directly # also should not be passed any arguments, since we need original $* # Set Hadoop-specific environment variables here. #Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/opt/hadoop-2.7.4 #Set path to where hadoop-*-core.jar is available xport HADOOP_MAPRED_HOME=/opt/hadoop-2.7.4 #set the path to where bin/hbase is available export HBASE_HOME=/opt/hbase-1.2.6 #Set the path to where bin/hive is available #export HIVE_HOME= #Set the path for where zookeper config dir is export ZOOCFGDIR=/opt/zookeeper-3.4.6/conf
4 配置环境变量etc/profileexport SQOOP_HOME=/opt/sqoop-1.4.6 export PATH=$PATH:$SQOOP_HOME/bin
四 上传mysql的驱动
[root@centos lib]# ll mysql-connector-java-5.1.30-bin.jar
-rw-r--r-- 1 root root 954041 Mar 11 2014 mysql-connector-java-5.1.30-bin.jar
五 简单测试[root@centos lib]# sqoop help /opt/sqoop-1.4.6/conf/sqoop-env.sh: line 26: xport: command not found Warning: /opt/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /opt/sqoop-1.4.6/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 18/01/01 11:52:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS import-mainframe Import datasets from a mainframe server to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version information See 'sqoop help COMMAND' for information on a specific command. [root@centos lib]# sqoop version /opt/sqoop-1.4.6/conf/sqoop-env.sh: line 26: xport: command not found Warning: /opt/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /opt/sqoop-1.4.6/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 18/01/01 11:52:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 Sqoop 1.4.6 git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25 Compiled by root on Mon Apr 27 14:38:36 CST 2015
六 参考
https://chuanke.baidu.com/v7662331-230792-1635549.html
相关文章推荐
- Sqoop笔记一之简介安装
- sqoop-1.4.6安装与使用
- sqoop的安装和原理
- hadoop学习第十一节: Sqoop介绍、安装与操作
- sqoop1.99.4安装与应用
- hadoop2.2.0(单节点)下Sqoop-1.4.6安装与配置(数据读取涉及hadoop、hbase和hive)
- sqoop1.99.3安装与教程
- Sqoop1.4.6 安装配置
- Sqoop1.99.7安装、配置和使用(二)
- sqoop的安装与使用
- Mapr 安装hadoop的组件(十)——安装Sqoop
- CentOS下sqoop配置安装
- Sqoop的安装与测试
- 【配置】sqoop安装配置
- Sqoop安装及操作以及oozie&sqoop
- CentOS下sqoop1.4.4的安装
- sqoop安装和简单实用
- Sqoop1 安装
- hadoop-2.5.0,hbase,hive,pig,sqoop,zookeeper 集群安装
- Ubunut12.04安装Sqoop(三)