Python 操作大数据使用 Hadoop

静听山水2024-09-04 10:35

参考：https://blog.csdn.net/wuShiJingZuo/article/details/135620018

python 复制代码

from hdfs import InsecureClient
hdfs_client = InsecureClient("http://localhost:9000", user="xiaokkk")
# 测试连接
print(hdfs_client.status("/"))

复制代码

{'accessTime': 0, 'blockSize': 0, 'childrenNum': 9, 'fileId': 16385, 'group': 'supergroup', 'length': 0, 'modificationTime': 1725326811876, 'owner': 'xiaokkk', 'pathSuffix': '', 'permission': '755', 'replication': 0, 'snapshotEnabled': True, 'storagePolicy': 0, 'type': 'DIRECTORY'}

python 复制代码

# 查看目录
files_and_dirs = hdfs_client.list("/")
print(files_and_dirs)

复制代码

['apps', 'backup', 'data', 'ks', 'logs', 'tmp', 'user']

python 复制代码

# 上传本地文件到HDFS
local_file_path = "/Users/xiaokkk/Desktop/local_projects/hadoop/input.txt"
hdfs_file_path = "/data/input.txt"

hdfs_client.upload(hdfs_file_path, local_file_path)
print(f"File {local_file_path} uploaded to {hdfs_file_path}")

复制代码

File /Users/xiaokkk/Desktop/local_projects/hadoop/input.txt uploaded to /data/input.txt

python 复制代码

# 下载HDFS文件到本地
hdfs_client.download(hdfs_file_path, "/Users/xiaokkk/Desktop/local_projects/hadoop/test_download.txt")

python 复制代码

# 删除HDFS文件
client.delete('/path/to/your/file', recursive=False)  # 如果文件是一个目录并且包含文件，则需要设置

python 复制代码

# 连接到HDFS
from hdfs import InsecureClient
client = InsecureClient("http://localhost:9000", user="xiaokkk")
# 指定HDFS上的文件路径
hdfs_path = '/data/input.txt'
# 打开HDFS上的文件
with client.read(hdfs_path, encoding='utf-8') as reader:
    # 读取文件内容
    content = reader.read()
    print(content)

# # 对文件内容进行处理，例如统计行数
# line_count = content.count('\n')
# print(f"文件 {hdfs_path} 包含 {line_count + 1} 行")
# 
# # 如果你想处理每一行，可以这样做
# for line in content.splitlines():
#     # 处理每一行
#     print(line)

复制代码

Hello world, this is a simple example of a text file.
This file will be used to demonstrate how the word count works.
Every word will be counted, and thpython word_count.py input.txt another_input.txt > output.txte results will show the frequency of each word.