how to use Sqoop to import/ export data

[b]Sqoop [/b]is a tool designed for efficiently transferring data between RDBMS and HDFS, we can import data from mysql, oracle, and other data bases into HDFS very easily; meanwhile we can dump data into data base from HDFS. For detailed documentation, please refer to sqoop documentation.

Before using Sqoop, please follow steps to setup it correctly.

Sqoop - Import

the following command is used for import



given a table named stock_info, and the schema is:



Case 1: we can use below command to import stock_info data to hadoop hdfs file system:


and the result looks like:



we can verify result in hdfs by running command



Case 2: sepcify the target directory in hdfs by running the following import command


then we can verify result by executing the same command as above

Case 3: imcremental import by specifying --incremental, --check-column and --append arguments. Note we should change 'last_chg_date' when applying other tables.



Case 4: specify target file format as parquet format by adding argument '--as-parquetfile'



Case 5: import all tables



Sqoop - Export

export means to dump data from hdfs to mysql, oracle or other data bases, command syntax is like


given there are many parquet files under stock_info folder which is imported by sqoop import command incrementally



then we want to dump data back into mysql data base, using the following command


finally verify data in mysql command line



Incremental importing data

by using linux timer 'crontab' to schedule a job to execute importing periodically.


if it works, you will receive email in '/var/spool/mail/hlli'; meanwhile we can verify data by running command



Commonly used Sqoop commands



References:
http://sqoop.apache.org/ http://man.linuxde.net/crontab