开源免费ETL工具==PYTHON实现

一个java开发2025-06-17 10:29

方便自己快速处理一些基于文件的聚合计算，比如CSV。

https://github.com/hebian1994/etl_react_flow

🛠️ ETL Flow Builder

A powerful, visual ETL (Extract-Transform-Load) tool built with React , React Flow , Material-UI , and a Python Flask backend. Design and manage complex data pipelines with a user-friendly interface and DAG-based execution.

📸 Preview

📸 Structure

🚀 Features

🔄 Flow Management

Create, edit, and delete ETL flows
Flow version history and versioning
Real-time configuration and validation

🧩 Node System

Modular nodes for various ETL operations:
- File Input
- Data Viewer
- Filter
- Left Join
Custom node configuration panels
Node connection validation and schema enforcement

⚙️ Data Processing

DAG-based flow execution
Schema propagation and management
Preview intermediate data at any node
Configuration validation before execution

🧱 System Architecture

🖥️ Frontend

Framework: React + TypeScript
Visualization: React Flow
UI Library: Material-UI (MUI)
Engine Library: Polars
Core Components:
- FlowList: Dashboard for managing flows
- Designer: Drag-and-drop interface for building flows
- History: View and restore previous versions
- Custom Node UI and Config Panels

🔧 Backend

Framework: Python + Flask
Database: SQLite (via SQLAlchemy ORM)
API: RESTful endpoints for flow and node operations
Core Services:
- FlowService: Handles flow CRUD and metadata
- NodeService: Manages node lifecycle and configurations
- ETLService: Executes DAGs and manages schema propagation

🧬 Data Models

复制代码

Flow    ──>  stores flow structure and metadata
Node    ──>  represents each ETL operation
Config  ──>  holds node-specific configuration
Schema  ──>  manages and validates data schema

🔒 State & Validation

Node configuration status tracking
Flow validation before execution
Schema-aware transformations and previews
UI-managed configuration state

🛠️ Tech Stack

Layer	Technology
Frontend	React, TypeScript, React Flow, MUI
Backend	Polars, Python, Flask, SQLAlchemy
Database	SQLite
Architecture	REST API + DAG Executor

📦 Getting Started

🔧 Prerequisites

Node.js (v16+)
Python (v3.8+)
pipenv or virtualenv

🖥️ Frontend Setup

复制代码

cd frontend
npm install
npm run dev

🐍 Backend Setup

复制代码

cd backend
pipenv install
pipenv run flask run