Apache Hadoop生态组件部署分享-Sqoop

1、上传压缩包并解压

apache 复制代码

tar -xf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/apache/

2、将mysql驱动包放置在其lib下

apache 复制代码

cp /opt/softs/mysql-connector-java-5.1.47.jar /opt/apache/sqoop-1.4.7.bin__hadoop-2.6.0/lib

3、配置SQOOP环境

bash 复制代码

export SQOOP_HOME=/opt/apache/sqoop-1.4.7.bin__hadoop-2.6.0export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:${SQOOP_HOME}/bin

4、验证使用

cs 复制代码

[root@apache230 ~]# sqoop list-databases --connect jdbc:mysql://apache230.hadoop.com:3306/ --username root --password 123456

5、验证抽取数据

mysql数据准备

sql 复制代码

create table test.testsqoopb(name VARCHAR(100)) DEFAULT CHARSET=utf8;
INSERT INTO test.testsqoopb (name)  VALUES ('john_doe');INSERT INTO test.testsqoopb (name)  VALUES ('张三');

直接抽取到HDFS上

cpp 复制代码

sqoop import \--connect jdbc:mysql://apache230.hadoop.com:3306/test \--username root \--password 123456 \--table testsqoopb \--target-dir /test/sqoop/test_sqoop \--delete-target-dir \--fields-terminated-by "\t" \--num-mappers 1 
注: 如果要多个map执行,则需要加--split-by 字段

查看验证

直接抽取到HIVE中

apache 复制代码

sqoop import \--connect jdbc:mysql://apache230.hadoop.com:3306/test \--username root \--password 123456 \--table testsqoopb \--num-mappers 1 \--delete-target-dir \--hive-import \--fields-terminated-by "\t" \--hive-overwrite \--hive-table test_sqoop

抽取到hbase

apache 复制代码

sqoop import \--connect jdbc:mysql://apache230.hadoop.com:3306/test \--username root \--password 123456 \--table sqoophbase \--hbase-create-table \--hbase-row-key "name" \--hbase-table "test_sqoopb" \--column-family "info" \--num-mappers 1 \--split-by id

当前遇到问题,后续再看,版本兼容问题

6、导出数据到mysql

apache 复制代码

sqoop export \--connect 'jdbc:mysql://apache230.hadoop.com:3306/test?useUnicode=true&characterEncoding=utf-8' \--username root \--password 123456 \--table testsqoopb \--num-mappers 1 \--export-dir /warehouse/hive/warehouse/test_sqoop \--input-fields-terminated-by "\t"

验证查看