hadoop测试环境sqoop使用

提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档

[Sqoop看这篇文章就够了_must contain 'conditions' in where clause._SoWhat1412的博客-CSDN博客](https://blog.csdn.net/qq_31821675/article/details/106682454 "Sqoop看这篇文章就够了_must contain 'conditions' in where clause._SoWhat1412的博客-CSDN博客")
大数据环境

C:\Windows\System32\drivers\etc

修改ip和hostname的对应关系

127.0.0.1 view-localhost # view localhost server

192.168.52.150 hadoop01 hadoop01.itcast.cn

192.168.52.151 hadoop02 hadoop02.itcast.cn

检查环境

ping 通环境测试

检查hadoop的版本

root@hadoop01 \~\]# hadoop version Hadoop 3.0.0-cdh6.2.1 Source code repository http://github.com/cloudera/hadoop -r 7ce7dcdb76fd7f700cd7749bbc6390f76ff2abae Compiled by jenkins on 2019-09-11T08:04Z Compiled with protoc 2.5.0 From source with checksum b6cc25ff15ff575f159489abbee34c5 This command was run using /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hadoop-common-3.0.0-cdh6.2.1.jar 检查hive的版本 \[root@hadoop01 \~\]# hive --version WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in \[jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Hive 2.1.1-cdh6.2.1

Subversion file:///container.redhat7/build/cdh/hive/2.1.1-cdh6.2.1/rpm/BUILD/hive-2.1.1-cdh6.2.1 -r fc0582dc4d2e2342d1eef8cb702bad68e1e2cdd2

Compiled by jenkins on Wed Sep 11 01:31:00 PDT 2019

From source with checksum 6bf22b68335469e1b010bcfecc8ea570

检查 hive服务是否启动了

root@hadoop01 \~\]# hive WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in \[jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hive-common-2.1.1-cdh6.2.1.jar!/hive-log4j2.properties Async: false

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive>

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive>

>

>

> show databases;

OK

default

test

Time taken: 4.039 seconds, Fetched: 2 row(s)

hive>

>

检查 sqoop是否安装

root@hadoop01 \~\]# sqoop help Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in \[jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

23/09/20 13:48:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

usage: sqoop COMMAND [ARGS]

Available commands:

codegen Generate code to interact with database records

create-hive-table Import a table definition into Hive

eval Evaluate a SQL statement and display the results

export Export an HDFS directory to a database table

help List available commands

import Import a table from a database to HDFS

import-all-tables Import tables from a database to HDFS

import-mainframe Import datasets from a mainframe server to HDFS

job Work with saved jobs

list-databases List available databases on a server

list-tables List available tables in a database

merge Merge results of incremental imports

metastore Run a standalone Sqoop metastore

version Display version information

See 'sqoop help COMMAND' for information on a specific command.

检查mysql数据库

root@hadoop01 \~\]# mysql -uroot -p123456 mysql: \[Warning\] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \\g. Your MySQL connection id is 70 Server version: 5.7.26 MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement. mysql\> mysql\> show databases; +--------------------+ \| Database \| +--------------------+ \| information_schema \| \| hive \| \| hue \| \| mysql \| \| nev \| \| oozie \| \| performance_schema \| \| scm \| \| sys \| \| test \| +--------------------+ 10 rows in set (0.00 sec) sqoop list-databases --connect jdbc:mysql://192.168.52.150:3306/ --username root --password 123456 测试sqoop数据库的连接 \[root@hadoop01 \~\]# sqoop list-databases --connect jdbc:mysql://192.168.52.150:3306/ --username root --password 123456 Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in \[jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

23/09/20 13:54:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

23/09/20 13:54:17 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

23/09/20 13:54:17 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

Wed Sep 20 13:54:18 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

information_schema

hive

hue

mysql

nev

oozie

performance_schema

scm

sys

test

root@hadoop01 \~\]# 练习环境部分信息 页面部分的补充 [http://hadoop01:7180/cmf/home](http://hadoop01:7180/cmf/home "http://hadoop01:7180/cmf/home") admin/admin ![](https://file.jishuzhan.net/article/1705841464606461954/beacf7205a52f32dc6eaa0c05690eecb.webp) [http://hadoop02:8889/hue/](http://hadoop02:8889/hue/ "http://hadoop02:8889/hue/") [http://hadoop02:8889/hue/](http://hadoop02:8889/hue/ "http://hadoop02:8889/hue/") hue/hue ![](https://file.jishuzhan.net/article/1705841464606461954/bd194fc4bec89b60d4bf9eb2afbda5b6.webp)

相关推荐
rui锐rui5 小时前
大数据学习6:Sqoop数据迁移工具
大数据·学习·sqoop
王小王-1232 天前
基于Hadoop的用户购物行为可视化分析系统设计与实现
大数据·hadoop·分布式·用户购物行为·电商日志分析
爱吃面的猫3 天前
大数据Hadoop之——Flink1.17.0安装与使用(非常详细)
大数据·hadoop·分布式
Edingbrugh.南空3 天前
Hadoop MapReduce 入门
大数据·hadoop·mapreduce
大数据CLUB4 天前
基于spark的奥运会奖牌变化数据分析
大数据·hadoop·数据分析·spark
Edingbrugh.南空4 天前
Hadoop高可用集群搭建
大数据·hadoop·分布式
无级程序员5 天前
hive2服务启动报错:/tmp/hive on HDFS should be writable(不是chmod 777能解决的)
hive·hadoop·hdfs
rui锐rui5 天前
大数据学习2:HIve
大数据·hive·学习
凌辰揽月5 天前
Servlet学习
hive·学习·servlet
weixin_307779136 天前
Hive集群之间迁移的Linux Shell脚本
大数据·linux·hive·bash·迁移学习