Conda 安装Jupyter:使用Pyhive(Kerberos)

安装基本环境

conda create -n bigdata python=3.10

conda activate bigdata

conda install -y pandas numpy pyhive

yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

pip install sasl

Jupyter Notebook

安装jupyter notebook配置自动提示

复制代码
conda install nb_conda_kernels

配置jupyter,添加密码,允许root,设置启动目录

环境配置-为linux服务器配置可以远程访问的Jupyter - 知乎 (zhihu.com)

jupyter notebook启动,以下报错可忽略

AttributeError: 'NotebookApp' object has no attribute 'io_loop'

PyHive连接开启Kerberos的Hive

jupyter notebook部署的服务器为集群已配置Kerberos认证的服务器,否则需要配置客户端认证

python 复制代码
from pyhive import hive
import pandas as pd
import numpy as np
import time

def func_time(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()

        execution_time = end_time - start_time
        print(f"函数 {func.__name__} 的执行时间为:{execution_time} 秒")

        return result

    return wrapper


class HiveCursor:
    def __init__(self, host, port, db, auth, username, configuration={'hive.execution.engine': 'spark'}):
        self.host = host
        self.port = port
        self.db = db
        self.username = username
        self.auth = auth
        self.conn = None
        self.cursor = None
        self.configuration = configuration

    def __enter__(self):
        # 建立与 Hive 的连接
        if self.auth == "KERBEROS":
            self.conn = hive.Connection(host=self.host, port=self.port, database=self.db,
                                        kerberos_service_name=self.username,
                                        auth=self.auth, configuration=self.configuration)
        else:
            self.conn = hive.Connection(host=self.host, port=self.port, database=self.db, username=self.username,
                                        auth=self.auth, configuration=self.configuration)
        self.cursor = self.conn.cursor()
        return self

    def query(self, sql):
        self.cursor.execute(sql)
        # 将查询结果读取到 DataFrame
        df = pd.DataFrame(self.cursor.fetchall())
        # 设置 DataFrame 列名
        df.columns = [desc[0] for desc in self.cursor.description]
        return df

    def __exit__(self, exc_type, exc_val, exc_tb):
        # 关闭连接
        self.conn.close()


@func_time
def hive_kerberos():
    query = "show databases"

    with HiveCursor(host="master", port=10000, db="default", auth='KERBEROS',
                    username='hive') as _hive:
        result = _hive.query(sql)
        print(result)


@func_time
def hive_on_spark():
    query = "show databases"

    with HiveCursor(host="master", port=10000, db="default", auth='KERBEROS',
                    username='hive', configuration={'hive.execution.engine': 'spark'}) as _hive:
        df = _hive.query(sql)
        print(df)
相关推荐
理想国的女研究僧2 天前
Jupyter Notebook操作指南(1)
ide·python·学习·jupyter
Neverfadeaway2 天前
Jupyter Notebook 介绍、安装及使用
jupyter·markdown·ipython·jupyter详解·jupyter快捷键
cyforkk2 天前
Anaconda配置环境变量和镜像
conda
❀搜不到3 天前
查询 conda + pip 装的包
windows·conda·pip
core5123 天前
Hive实战(三)
数据仓库·hive·hadoop
程序员小羊!3 天前
大数据电商流量分析项目实战:Hive 数据仓库(三)
大数据·数据仓库·hive
core5123 天前
Hive实战(一)
数据仓库·hive·hadoop·架构·实战·配置·场景
智海观潮3 天前
Spark SQL解析查询parquet格式Hive表获取分区字段和查询条件
hive·sql·spark
cxr8284 天前
基于Claude Code的 规范驱动开发(SDD)指南
人工智能·hive·驱动开发·敏捷流程·智能体
996终结者4 天前
同类软件对比(四):Jupyter vs PyCharm vs VS Code:Python开发工具终极选择指南
vscode·python·jupyter·pycharm·visual studio code