开源免费ETL工具==PYTHON实现

方便自己快速处理一些基于文件的聚合计算,比如CSV。

https://github.com/hebian1994/etl_react_flow

🛠️ ETL Flow Builder

A powerful, visual ETL (Extract-Transform-Load) tool built with React , React Flow , Material-UI , and a Python Flask backend. Design and manage complex data pipelines with a user-friendly interface and DAG-based execution.


📸 Preview

📸 Structure


🚀 Features

🔄 Flow Management

  • Create, edit, and delete ETL flows
  • Flow version history and versioning
  • Real-time configuration and validation

🧩 Node System

  • Modular nodes for various ETL operations:
    • File Input
    • Data Viewer
    • Filter
    • Left Join
  • Custom node configuration panels
  • Node connection validation and schema enforcement

⚙️ Data Processing

  • DAG-based flow execution
  • Schema propagation and management
  • Preview intermediate data at any node
  • Configuration validation before execution

🧱 System Architecture

🖥️ Frontend

  • Framework: React + TypeScript
  • Visualization: React Flow
  • UI Library: Material-UI (MUI)
  • Engine Library: Polars
  • Core Components:
    • FlowList: Dashboard for managing flows
    • Designer: Drag-and-drop interface for building flows
    • History: View and restore previous versions
    • Custom Node UI and Config Panels

🔧 Backend

  • Framework: Python + Flask
  • Database: SQLite (via SQLAlchemy ORM)
  • API: RESTful endpoints for flow and node operations
  • Core Services:
    • FlowService: Handles flow CRUD and metadata
    • NodeService: Manages node lifecycle and configurations
    • ETLService: Executes DAGs and manages schema propagation

🧬 Data Models

复制代码
Flow    ──>  stores flow structure and metadata
Node    ──>  represents each ETL operation
Config  ──>  holds node-specific configuration
Schema  ──>  manages and validates data schema

🔒 State & Validation

  • Node configuration status tracking
  • Flow validation before execution
  • Schema-aware transformations and previews
  • UI-managed configuration state

🛠️ Tech Stack

Layer Technology
Frontend React, TypeScript, React Flow, MUI
Backend Polars, Python, Flask, SQLAlchemy
Database SQLite
Architecture REST API + DAG Executor

📦 Getting Started

🔧 Prerequisites

  • Node.js (v16+)
  • Python (v3.8+)
  • pipenv or virtualenv

🖥️ Frontend Setup

复制代码
cd frontend
npm install
npm run dev

🐍 Backend Setup

复制代码
cd backend
pipenv install
pipenv run flask run
相关推荐
特立独行的猫A3 小时前
别再用 nohup 了:让应用在后台常驻的正确姿势与 systemd 服务详解
java·后端·python
冬奇Lab5 小时前
一天一个开源项目(第262篇):AG-UI —— 把 AI Agent 接进前端的开放协议,和 MCP、A2A 并列的第三块拼图
人工智能·开源
吃我一窝jio5 小时前
Python与Go线程模型深度对比
java·开发语言·python
zwd20056 小时前
Manim 怎么放大画面的某一部分?
python·动画·可视化·manim·数学动画
屿华AI7 小时前
AUTOSAR 实战-诊断工具 | UDS 0x27 安全访问的配置化实现:种子密钥算法插件化设计
python·测试工具·汽车
RobinDevNotes7 小时前
Claude Code 剪视频,不上传、不要 API Key
人工智能·python·音视频·开发工具·ipython
孟健8 小时前
用T3 Code手机远程控制Claude Code完成周报总结实测
开源·ai编程·claude
如夏倾城8 小时前
VRRP(虚拟路由器冗余协议Virtual Router Redundancy Protocol)
网络·python·网络协议·安全·智能路由器·负载均衡·信息与通信
IvanCodes9 小时前
LangChain(04):Runnable 与 LCEL 的执行与组合机制
python·langchain·agent
特创数字科技9 小时前
纯本地离线运行的文档脱敏工具(技术分享)
前端·驱动开发·python