Sqoop增量同步Oracle数据到hive:merge-key再次详解
2019-03-25 19:01
351 查看
版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/qq_28356739/article/details/88803284
对于sqoop增量同步Oracle数据到hive的命令参数以及如何定制自动增量job的测试已经再前面几篇文章详细测试过了,连接:
1、sqoop避免输入密码自动增量job脚本介绍
这篇文章是基于上面连接的文章继续做的拓展,上篇文章结尾说了如果一个表很大。我第一次初始化一部分最新的数据到hive表,如果没初始化进来的历史数据今天发生了变更,那merge-key的增量方式会不会报错呢?之所以会提出这个问题,是因为笔者真的有这个测试需求,接下来先对oracle端的库表数据做下修改,来模拟这种场景。
第一步:先插入一条数据
当前时间为:
SQL> select sysdate from dual; SYSDATE ------------------- 2019-03-25 18:20:26
为了模拟我是有一部分历史数据没有导入到hive表,我这里先给oracle表插入一条历史数据:
SQL> select * from inr_job; EMPNO ENAME JOB SAL ETLTIME ---------- ---------- --------- ---------- ------------------- 1 er CLERK 800 2019-03-22 17:24:42 2 ALLEN SALESMAN 1600 2019-03-22 17:24:42 3 WARD SALESMAN 1250 2019-03-22 17:24:42 4 JONES MANAGER 2975 2019-03-22 17:24:42 5 MARTIN SALESMAN 1250 2019-03-22 17:24:42 6 zhao DBA 1000 2019-03-22 17:24:42 7 yan BI 100 2019-03-22 17:24:42 8 dong JAVA 400 2019-03-22 17:24:42 8 rows selected. SQL> insert into inr_job values(9,'test','test',200,sysdate-20); 1 row created. SQL> commit; Commit complete. SQL> select * from inr_job; EMPNO ENAME JOB SAL ETLTIME ---------- ---------- --------- ---------- ------------------- 1 er CLERK 800 2019-03-22 17:24:42 2 ALLEN SALESMAN 1600 2019-03-22 17:24:42 3 WARD SALESMAN 1250 2019-03-22 17:24:42 4 JONES MANAGER 2975 2019-03-22 17:24:42 5 MARTIN SALESMAN 1250 2019-03-22 17:24:42 6 zhao DBA 1000 2019-03-22 17:24:42 7 yan BI 100 2019-03-22 17:24:42 8 dong JAVA 400 2019-03-22 17:24:42 9 test test 200 2019-03-05 18:53:23--模仿没初始化到hive表的his数据 9 rows selected.
接下来手动更新一下这个历史数据:
SQL> update inr_job set sal=999,etltime=sysdate where empno=9; 1 row updated. SQL> commit; Commit complete. SQL> select * from inr_job; EMPNO ENAME JOB SAL ETLTIME ---------- ---------- --------- ---------- ------------------- 1 er CLERK 800 2019-03-22 17:24:42 2 ALLEN SALESMAN 1600 2019-03-22 17:24:42 3 WARD SALESMAN 1250 2019-03-22 17:24:42 4 JONES MANAGER 2975 2019-03-22 17:24:42 5 MARTIN SALESMAN 1250 2019-03-22 17:24:42 6 zhao DBA 1000 2019-03-22 17:24:42 7 yan BI 100 2019-03-22 17:24:42 8 dong JAVA 400 2019-03-22 17:24:42 9 test test 999 2019-03-25 18:54:39 9 rows selected.
现在数据发生了变动,然后去执行一下增量脚本:
[root@hadoop hadoop]# sqoop job --exec auto_job Warning: /hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 19/03/25 18:55:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/hbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19/03/25 18:55:51 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled. 19/03/25 18:55:51 INFO manager.SqlManager: Using default fetchSize of 1000 19/03/25 18:55:51 INFO tool.CodeGenTool: Beginning code generation 19/03/25 18:55:52 INFO manager.OracleManager: Time zone has been set to GMT 19/03/25 18:55:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM INR_JOB t WHERE 1=0 19/03/25 18:55:52 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /hadoop Note: /tmp/sqoop-root/compile/f64e34273a58459369885b96fe46a1ad/INR_JOB.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 19/03/25 18:55:56 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f64e34273a58459369885b96fe46a1ad/INR_JOB.jar 19/03/25 18:55:56 INFO manager.OracleManager: Time zone has been set to GMT 19/03/25 18:55:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM INR_JOB t WHERE 1=0 19/03/25 18:55:56 INFO tool.ImportTool: Incremental import based on column ETLTIME 19/03/25 18:55:56 INFO tool.ImportTool: Lower bound value: TO_TIMESTAMP('2019-03-25 18:50:07.0', 'YYYY-MM-DD HH24:MI:SS.FF') 19/03/25 18:55:56 INFO tool.ImportTool: Upper bound value: TO_TIMESTAMP('2019-03-25 18:55:56.0', 'YYYY-MM-DD HH24:MI:SS.FF') 19/03/25 18:55:56 INFO manager.OracleManager: Time zone has been set to GMT 19/03/25 18:55:56 INFO mapreduce.ImportJobBase: Beginning import of INR_JOB 19/03/25 18:55:56 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 19/03/25 18:55:56 INFO manager.OracleManager: Time zone has been set to GMT 19/03/25 18:55:56 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 19/03/25 18:55:56 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.66:8032 19/03/25 18:55:59 INFO db.DBInputFormat: Using read commited transaction isolation 19/03/25 18:55:59 INFO mapreduce.JobSubmitter: number of splits:1 19/03/25 18:56:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553503985304_0013 19/03/25 18:56:00 INFO impl.YarnClientImpl: Submitted application application_1553503985304_0013 19/03/25 18:56:00 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1553503985304_0013/ 19/03/25 18:56:00 INFO mapreduce.Job: Running job: job_1553503985304_0013 19/03/25 18:56:10 INFO mapreduce.Job: Job job_1553503985304_0013 running in uber mode : false 19/03/25 18:56:10 INFO mapreduce.Job: map 0% reduce 0% 19/03/25 18:56:19 INFO mapreduce.Job: map 100% reduce 0% 19/03/25 18:56:20 INFO mapreduce.Job: Job job_1553503985304_0013 completed successfully 19/03/25 18:56:20 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of 2d31a bytes read=0 FILE: Number of bytes written=144777 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=38 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=5870 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=5870 Total vcore-milliseconds taken by all map tasks=5870 Total megabyte-milliseconds taken by all map tasks=6010880 Map-Reduce Framework Map input records=1 Map output records=1 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=100 CPU time spent (ms)=3220 Physical memory (bytes) snapshot=189059072 Virtual memory (bytes) snapshot=2147303424 Total committed heap usage (bytes)=102236160 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=38 19/03/25 18:56:20 INFO mapreduce.ImportJobBase: Transferred 38 bytes in 23.7426 seconds (1.6005 bytes/sec) 19/03/25 18:56:20 INFO mapreduce.ImportJobBase: Retrieved 1 records. 19/03/25 18:56:20 INFO tool.ImportTool: Final destination exists, will run merge job. 19/03/25 18:56:20 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 19/03/25 18:56:20 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.66:8032 19/03/25 18:56:22 INFO input.FileInputFormat: Total input paths to process : 2 19/03/25 18:56:23 INFO mapreduce.JobSubmitter: number of splits:2 19/03/25 18:56:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553503985304_0014 19/03/25 18:56:23 INFO impl.YarnClientImpl: Submitted application application_1553503985304_0014 19/03/25 18:56:23 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1553503985304_0014/ 19/03/25 18:56:23 INFO mapreduce.Job: Running job: job_1553503985304_0014 19/03/25 18:56:37 INFO mapreduce.Job: Job job_1553503985304_0014 running in uber mode : false 19/03/25 18:56:37 INFO mapreduce.Job: map 0% reduce 0% 19/03/25 18:56:46 INFO mapreduce.Job: map 100% reduce 0% 19/03/25 18:56:56 INFO mapreduce.Job: map 100% reduce 100% 19/03/25 18:56:57 INFO mapreduce.Job: Job job_1553503985304_0014 completed successfully 19/03/25 18:56:57 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=614 FILE: Number of bytes written=435819 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=657 HDFS: Number of bytes written=361 HDFS: Number of read operations=9 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=11103 Total time spent by all reduces in occupied slots (ms)=7376 Total time spent by all map tasks (ms)=11103 Total time spent by all reduce tasks (ms)=7376 Total vcore-milliseconds taken by all map tasks=11103 Total vcore-milliseconds taken by all reduce tasks=7376 Total megabyte-milliseconds taken by all map tasks=11369472 Total megabyte-milliseconds taken by all reduce tasks=7553024 Map-Reduce Framework Map input records=9 Map output records=9 Map output bytes=590 Map output materialized bytes=620 Input split bytes=296 Combine input records=0 Combine output records=0 Reduce input groups=9 Reduce shuffle bytes=620 Reduce input records=9 Reduce output records=9 Spilled Records=18 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=263 CPU time spent (ms)=3980 Physical memory (bytes) snapshot=670138368 Virtual memory (bytes) snapshot=6394978304 Total committed heap usage (bytes)=508559360 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=361 File Output Format Counters Bytes Written=361 19/03/25 18:56:57 INFO tool.ImportTool: Saving incremental import state to the metastore 19/03/25 18:56:57 INFO tool.ImportTool: Updated data for job: auto_job
发现没有报错唉,然后去看看hive表:
hive> select * from inr_job; OK 1 er CLERK 800.0 2019-03-22 17:24:42.0 2 ALLEN SALESMAN 1600.0 2019-03-22 17:24:42.0 3 WARD SALESMAN 1250.0 2019-03-22 17:24:42.0 4 JONES MANAGER 2975.0 2019-03-22 17:24:42.0 5 MARTIN SALESMAN 1250.0 2019-03-22 17:24:42.0 6 zhao DBA 1000.0 2019-03-22 17:24:42.0 7 yan BI 100.0 2019-03-22 17:24:42.0 8 dong JAVA 400.0 2019-03-22 17:24:42.0 9 test test 999.0 2019-03-25 18:54:39.0 Time taken: 0.336 seconds, Fetched: 9 row(s)
没初始化进来的历史数据在近期变动之后,如果符合增量条件的话,也会append进来并不会报错,完全符合笔者需求,其实看看merge-key参数大致原理,也是知道这样是可行的,毕竟是通过主键和最后修改时间去做增量合并。
相关文章推荐
- 通过sqoop增量传送oracle数据到hive
- Sqoop将MySQL和Oracle的数据导入HIVE和Hbase
- sqoop导入oracle数据至hive
- 阿里巴巴去Oracle数据迁移同步工具(全量+增量,目标支持MySQL/DRDS)
- sqoop导oracle数据到hive中并动态分区
- 利用ogg实现oracle到kafka的增量数据实时同步
- Sqoop增量从MySQL中向hive导入数据
- SQOOP从Oracle导入数据到Hive时hang up在MapReduce作业过程
- sqoop从oracle导入数据导hive表不支持的数据类型
- Sqoop增量从MySQL中向hive导入数据
- Sqoop是一款开源的工具,主要用于在HADOOP(Hive)与传统的数据库(mysql、oracle...)间进行数据的传递
- 利用Sqoop从oracle同步数据到HDFS出现The Network Adapter could not establish the connection的解决方法
- 通过sqoop导入Oracle数据到Hive时异常.IOException: Cannot run program "hive": error=2, No such file or directory
- 使用KETTLE从mysql同步增量数据到oracle
- sqoop导入数据到Base并同步hive与impala
- Java api 调用Sqoop2进行MySQL-->Hive的数据同步
- 定时从大数据平台同步HIVE数据到oracle
- Hadoop数据工具sqoop,导入HDFS,HIVE,HBASE,导出到oracle
- Oracle不同数据库同步Merge方法,可以部分替代高级复制。对小数据量同步问题不大。转载http://blog.chinaunix.net/u1/55091/showart_430716.html
- Sqoop_具体总结 使用Sqoop将HDFS/Hive/HBase与MySQL/Oracle中的数据相互导入、导出