Conda 安装Jupyter:使用Pyhive(Kerberos)

安装基本环境

conda create -n bigdata python=3.10

conda activate bigdata

conda install -y pandas numpy pyhive

yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

pip install sasl

Jupyter Notebook

安装jupyter notebook配置自动提示

复制代码
conda install nb_conda_kernels

配置jupyter,添加密码,允许root,设置启动目录

环境配置-为linux服务器配置可以远程访问的Jupyter - 知乎 (zhihu.com)

jupyter notebook启动,以下报错可忽略

AttributeError: 'NotebookApp' object has no attribute 'io_loop'

PyHive连接开启Kerberos的Hive

jupyter notebook部署的服务器为集群已配置Kerberos认证的服务器,否则需要配置客户端认证

python 复制代码
from pyhive import hive
import pandas as pd
import numpy as np
import time

def func_time(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()

        execution_time = end_time - start_time
        print(f"函数 {func.__name__} 的执行时间为:{execution_time} 秒")

        return result

    return wrapper


class HiveCursor:
    def __init__(self, host, port, db, auth, username, configuration={'hive.execution.engine': 'spark'}):
        self.host = host
        self.port = port
        self.db = db
        self.username = username
        self.auth = auth
        self.conn = None
        self.cursor = None
        self.configuration = configuration

    def __enter__(self):
        # 建立与 Hive 的连接
        if self.auth == "KERBEROS":
            self.conn = hive.Connection(host=self.host, port=self.port, database=self.db,
                                        kerberos_service_name=self.username,
                                        auth=self.auth, configuration=self.configuration)
        else:
            self.conn = hive.Connection(host=self.host, port=self.port, database=self.db, username=self.username,
                                        auth=self.auth, configuration=self.configuration)
        self.cursor = self.conn.cursor()
        return self

    def query(self, sql):
        self.cursor.execute(sql)
        # 将查询结果读取到 DataFrame
        df = pd.DataFrame(self.cursor.fetchall())
        # 设置 DataFrame 列名
        df.columns = [desc[0] for desc in self.cursor.description]
        return df

    def __exit__(self, exc_type, exc_val, exc_tb):
        # 关闭连接
        self.conn.close()


@func_time
def hive_kerberos():
    query = "show databases"

    with HiveCursor(host="master", port=10000, db="default", auth='KERBEROS',
                    username='hive') as _hive:
        result = _hive.query(sql)
        print(result)


@func_time
def hive_on_spark():
    query = "show databases"

    with HiveCursor(host="master", port=10000, db="default", auth='KERBEROS',
                    username='hive', configuration={'hive.execution.engine': 'spark'}) as _hive:
        df = _hive.query(sql)
        print(df)
相关推荐
yumgpkpm6 小时前
CMP (类ClouderaCDP7.3(404次编译) )华为鲲鹏Aarch64(ARM)信创环境多个mysql数据库汇聚的操作指南
大数据·hive·hadoop·zookeeper·big data·cloudera
woshihonghonga17 小时前
停止Conda开机自动运行方法
linux·人工智能·conda
liliangcsdn17 小时前
如何使用命令修改conda虚拟环境目录
conda
夫唯不争,故无尤也1 天前
Maven创建Java项目实战全流程
java·数据仓库·hive·hadoop·maven
林炳然1 天前
在 VS Code 中使用 Jupyter:创建自定义内核完整指南
jupyter
JANGHIGH2 天前
YOLO系列——Ubuntu20.04下通过conda虚拟环境安装Labelme
yolo·conda
yumgpkpm2 天前
CMP (类Cloudera) CDP7.3(400次编译)在华为鲲鹏Aarch64(ARM)信创环境中的性能测试过程及命令
大数据·hive·hadoop·python·elasticsearch·spark·cloudera
yumgpkpm2 天前
大数据综合管理平台(CMP)(类Cloudera CDP7.3)有哪些核心功能?
hive·hadoop·elasticsearch·zookeeper·big data
板凳坐着晒太阳2 天前
Hive 删除分区语句卡死问题
数据仓库·hive·hadoop
Stestack3 天前
Linux - conda 环境安装教程
linux·conda