ViewFs And Federation On HDFS

序言

ViewFs 是在Federation的基础上提出的,用于通过一个HDFS路径来访问多个NameSpace,同时与ViewFs搭配的技术是client-side mount table(这个就是具体的规则配置信息可以放置在core.xml中,也可以放置在mountTable.xml中).

总的来说ViewFs的其实就是一个中间层,用于去连接不同的Namenode,然后返还给我们的客户端程序. 所以ViewFs必须要实现HDFS的所有接口,这样才能来做转发管理. 这样就会有一些问题,比如不同的NameNode版本带来的问题,就没法解决cuiyaonan2000@163.com

Federation Of HDFS

只是单纯的搭建联盟其实比较简单.

core.xml

XML 复制代码
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>


    <property>
        <name>hadoop.tmp.dir</name>
        <value>/soft/hadoop/data_hadoop</value>
    </property>


    <!-- 这里的ip要改成对应的namenode地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop:9000/</value>
    </property>

 

</configuration>

hfds-site.xml

XML 复制代码
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>


    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>


    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/soft/hadoop/data_hadoop/datanode</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/soft/hadoop/data_hadoop/namenode</value>
    </property>


    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>


    <property>
        <name>dfs.nameservices</name>
        <value>ns1,ns2</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.ns1</name>
        <value>hadoop:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.ns1</name>
        <value>hadoop:50070</value>
    </property>

    <property>
        <name>dfs.namenode.secondaryhttp-address.ns1</name>
        <value>hadoop:50090</value>
    </property>


    <property>
        <name>dfs.namenode.rpc-address.ns2</name>
        <value>hadoop1:9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address.ns2</name>
        <value>hadoop1:50070</value>
    </property>

    <property>
        <name>dfs.namenode.secondaryhttp-address.ns2</name>
        <value>hadoop1:50090</value>
    </property>

</configuration>

启动

首先就是需要格式化namenode,这个很常规 hdfs namenode -format

关于联盟版本的创建则需要设置联盟的id,所以需要再格式化namenode 的时候指定

hdfs namenode -format -clusterId cui

验证

最主要的就是共用DataNode,即他们的DataNode 信息一样

ViewFs Of HDFS

相关推荐
青云交1 天前
Java 大视界 -- 基于 Java 的大数据分布式存储在智慧城市时空大数据管理与应用中的创新实践(408)
java·hdfs·flink·智慧城市·hbase·java 分布式存储·时空大数据
计算机毕设残哥3 天前
HDFS存储农业大数据的秘密是什么?高级大豆数据分析与可视化系统架构设计思路
大数据·hadoop·python·hdfs·数据分析·spark·django
dessler5 天前
Hadoop HDFS-JournalNode(jn)详细介绍
linux·运维·hdfs
dessler6 天前
Hadoop HDFS-SecondaryNameNode(2nn)详细介绍
大数据·hadoop·hdfs
張萠飛9 天前
hive on tez为什么写表时,要写临时文件到hdfs目录
hive·hadoop·hdfs
小白不想白a15 天前
【Hadoop】HDFS 分布式存储系统
hadoop·分布式·hdfs
dessler25 天前
Hadoop HDFS-部署和基本操作
linux·运维·hdfs
程序员小羊!1 个月前
Hadoop HDFS 3.3.4 讲解~
大数据·hadoop·hdfs
码字的字节1 个月前
深入解析Hadoop高频面试题:HDFS读/写流程的RPC调用链
hadoop·hdfs·rpc
码字的字节2 个月前
Hadoop安全机制深度剖析:Kerberos认证与HDFS ACL细粒度权限控制
hadoop·安全·hdfs·kerberos·acl