clickhouse数据字典

前言：

clickhouse的安装见

https://blog.csdn.net/weixin_41463944/article/details/142534145?spm=1001.2014.3001.5501

1、需要在路径/etc/clickhouse-serve下定义*_dictionary.xml结尾的配置文件

2、需要准备organization.csv和sales.csv本地测试文件（注意不是windows下的xlsx表格，是linux下的文件）

2.1 organization.csv文件

1,"a0001","研发部"
2,"a0002","产品部"
3,"a0003","数据部"
4,"a0004","测试部"
5,"a0005","运维部"
6,"a0006","规划部",
7,"a0007","市场部"

2.2 sales.csv文件

1,2016-01-01,2017-01-10,100

2,2016-05-01,2017-07-01,200

3,2014-03-05,2018-01-20,300

4,2018-08-01,2019-10-01,400

5,2017-03-01,2017-06-01,500

6,2017-04-09,2018-05-30,600

7,2018-06-01,2019-01-25,700

8,2019-08-01,2019-12-12,800

3、关于每个xml字典文件的格式

<?xml version="1.0"?>
<dictionaries>
  <dictionary>
    <name>字典名称（全局唯一）</name>

    <source>
      数据源配置
    </source>

    <layout>
      字典数据在内存中的格式
    </layout>

    <structure>
      字典的数据结构

    </structure>

    <lifetime>
    字典的自动更新频率
      <min>300</min>
      <max>360</max>
    </lifetime>

  </dictionary>
</dictionaries>

4、关于扩展字典的几种类型配置

一共7种类型，其中flat、hashed、sparse_hashed、range_hashed、cache的主键为数值型UInt64类型的，其他类型如complex_key_hashed、complex_key_cache为复合类型的key

4.1）flat类型的字典，在/etc/clickhouse-serve目录下新建test_flat_dictionary.xml

<?xml version="1.0"?>
<dictionaries>
  <dictionary>
    <name>test_flat_dict</name>

    <source>
      <file>
        <path>/home/clickhouse-server/etc/clickhouse-server/dictionary/organization.csv</path>
        <format>CSV</format>
      </file>
    </source>

    <layout>
      <flat/>
    </layout>

    <structure>
      <id>
        <name>id</name>
      </id>

      <attribute>
        <name>code</name>
        <type>String</type>
        <null_value></null_value>
      </attribute>

      <attribute>
        <name>name</name>
        <type>String</type>
        <null_value></null_value>
      </attribute>

    </structure>

    <lifetime>
      <min>300</min>
      <max>360</max>
    </lifetime>

  </dictionary>
</dictionaries>

4.2）hashed类型的字典，在/etc/clickhouse-serve目录下新建test_hashed_dictionary.xml

<?xml version="1.0"?>
<dictionaries>
  <dictionary>
    <name>test_hashed_dict</name>

    <source>
      <file>
        <path>/home/clickhouse-server/etc/clickhouse-server/dictionary/organization.csv</path>
        <format>CSV</format>
      </file>
    </source>

    <layout>
      <hashed/>
    </layout>

    <structure>
      <id>
        <name>id</name>
      </id>

      <attribute>
        <name>code</name>
        <type>String</type>
        <null_value></null_value>
      </attribute>

      <attribute>
        <name>name</name>
        <type>String</type>
        <null_value></null_value>
      </attribute>

    </structure>

    <lifetime>
      <min>300</min>
      <max>360</max>
    </lifetime>

  </dictionary>
</dictionaries>

4.3）complex_key_hashed类型的字典，在/etc/clickhouse-serve目录下新建complex_key_hashed_dictionary.xml

<?xml version="1.0"?>
<dictionaries>
  <dictionary>
    <name>test_complex_key_hashed_dict</name>

    <source>
      <file>
        <path>/home/clickhouse-server/etc/clickhouse-server/dictionary/organization.csv</path>
        <format>CSV</format>
      </file>
    </source>

    <layout>
      <complex_key_hashed/>
    </layout>

    <structure>
      <key>
        <attribute>
          <name>id</name>
          <type>UInt64</type>
        </attribute>
        <attribute>
          <name>code</name>
          <type>String</type>
        </attribute>
      </key>

      <attribute>
        <name>name</name>
        <type>String</type>
        <null_value></null_value>
      </attribute>

    </structure>

    <lifetime>
      <min>300</min>
      <max>360</max>
    </lifetime>

  </dictionary>
</dictionaries>

4.4）range_hashed类型的字典，在/etc/clickhouse-serve目录下新建test_range_hashed_dictionary.xml

<?xml version="1.0"?>
<dictionaries>
  <dictionary>
    <name>test_range_hashed_dict</name>

    <source>
      <file>
        <path>/home/clickhouse-server/etc/clickhouse-server/dictionary/sales.csv</path>
        <format>CSV</format>
      </file>
    </source>

    <layout>
      <range_hashed/>
    </layout>

    <structure>
      <id>
        <name>id</name>
      </id>

      <range_min>
        <name>start</name>
      </range_min>

      <range_max>
        <name>end</name>
      </range_max>

      <attribute>
        <name>price</name>
        <type>Float32</type>
        <null_value></null_value>
      </attribute>

    </structure>

    <lifetime>
      <min>300</min>
      <max>360</max>
    </lifetime>

  </dictionary>
</dictionaries>

5、关于字典的数据源

5.1 文件类型，格式通常为CSV或者TabSeparated

1）本地文件

<source>
      <file>
        <path>/home/clickhouse-server/etc/clickhouse-server/dictionary/organization.csv</path>
        <format>CSV</format>
      </file>
    </source>

2）可执行文件

<source>
      <executable>
        <command>/home/clickhouse-server/etc/clickhouse-server/dictionary/organization.csv</command>
        <format>CSV</format>
      </executable>
    </source>

3）远程文件

<source>
      <http>
        <url>xxxxx.csv</url>
        <format>CSV</format>
      </http>
    </source>

5.2 数据库类型的数据源
1）mysql

<source>
      <mysql>
        <port>3306</port>
        <user>xxxx</user>
        <password>xxxx</password>
        <replica>
          <host>localhost</host>
          <priority>1</priority>
        </replica>
        <db>db_mybatis</db>
        <table>userList</table>
      </mysql>
    </source>

其中<mysql> 标签的下一级可添加 <where>is_deleted=0</where>做为筛选条件

2）clickhouse

 <source>
      <clickhouse>
        <host>localhost</host>
        <port>9000</port>
        <user>xxxx</user>
        <password>xxxxx</password>
        <db>my_database</db>
        <table>test_table</table>
      </clickhouse>
    </source>

6、关于DDL语句

dictGet('字典名称','字典的具体字段','key键值')，如果字典为复合key的话，则在字典函数中的key用元组tuple('a','b','c')表示

7、关于手动刷新数据字典

7.1 ）全局字典刷新

system reload dictionaries

7.2）刷新单个字典（字典名称不加单引号）

system reload dictionary test_mysql_dict