基于 DataX + DataX-Web 生成报表数据

Lin_Miao_092025-12-21 8:55

背景

目标

架构

测试示例

背景

目前数据库数据量增长，导致部分报表查询和导出操作变慢。为此，建议采用DataX结合DataX-Web构建报表数据生成平台。

目标

提供 Web 可视化界面进行任务配置、调度、监控与日志查看；
支持定时执行、失败重试、告警通知等运维能力；
建立标准化、可复用的报表数据生成流程，降低开发与维护成本。

架构

测试示例

复制代码

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      },
      "errorLimit": {
        "record": 5
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "yRjwDFuoPKlqya9h9H2Amg==",
            "password": "XCYVpFosvZBBWobFzmLWvA==",
            "connection": [
              {
                "querySql": [
                  "SELECT\n    o.order_id,\n    o.customer_id,\n    c.customer_name,\n    c.city,\n    o.product_name,\n    o.amount,\n    o.order_date\nFROM orders o\nJOIN customers c ON o.customer_id = c.customer_id"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://192.168.234.128:3306/DataX?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "username": "yRjwDFuoPKlqya9h9H2Amg==",
            "password": "XCYVpFosvZBBWobFzmLWvA==",
            "column": [
              "`order_id`",
              "`customer_id`",
              "`customer_name`",
              "`city`",
              "`product_name`",
              "`amount`",
              "`order_date`"
            ],
            "connection": [
              {
                "table": [
                  "fact_order_summary"
                ],
                "jdbcUrl": "jdbc:mysql://192.168.234.128:3306/DataX_1?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai"
              }
            ]
          }
        }
      }
    ]
  }
}