中国电网Java面试被问：流批一体架构的实现和状态管理

一、流批一体架构概览

1. 核心设计原则

text

复制

下载

同一套API：一套代码既能处理流也能处理批
统一状态管理：流批共享状态存储和访问方式
统一计算引擎：底层引擎同时支持流批处理
统一元数据：一致的schema管理和数据目录

2. 架构演进对比

图表

代码

复制

下载

全屏

graph TD
    A[传统Lambda架构] --> B[流批一体架构]
    
    subgraph A [Lambda架构：两套系统]
        A1[批处理层] --> A2[批处理视图]
        A3[速度层] --> A4[实时视图]
        A2 --> A5[服务层]
        A4 --> A5
    end
    
    subgraph B [流批一体：统一系统]
        B1[统一计算层] --> B2[统一状态存储]
        B2 --> B3[统一服务层]
    end

二、核心实现技术栈

1. 主流实现框架

python

复制

下载

# Apache Flink（最成熟的流批一体）
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import StreamTableEnvironment

env = StreamExecutionEnvironment.get_execution_environment()
t_env = StreamTableEnvironment.create(env)

# 同一套Table API处理流和批
# 流处理
t_env.execute_sql("""
    CREATE TABLE orders_stream (
        order_id STRING,
        user_id BIGINT,
        amount DOUBLE,
        order_time TIMESTAMP(3)
    ) WITH (
        'connector' = 'kafka',
        'scan.startup.mode' = 'earliest-offset'
    )
""")

# 批处理
t_env.execute_sql("""
    CREATE TABLE orders_batch (
        order_id STRING,
        user_id BIGINT,
        amount DOUBLE,
        order_time TIMESTAMP(3)
    ) WITH (
        'connector' = 'filesystem',
        'format' = 'parquet'
    )
""")

2. 统一API层设计

java

复制

下载

// 以Apache Beam为例的跨引擎API
PipelineOptions options = PipelineOptionsFactory.create();

// 批处理模式
options.setRunner(DirectRunner.class);

// 流处理模式  
options.setRunner(FlinkRunner.class);
options.as(FlinkPipelineOptions.class).setStreaming(true);

// 同一套Pipeline定义
Pipeline p = Pipeline.create(options);
p.apply(TextIO.read().from("input.txt"))
 .apply(ParDo.of(new DoFn<String, String>() {
     @ProcessElement
     public void processElement(ProcessContext c) {
         c.output(c.element().toUpperCase());
     }
 }))
 .apply(TextIO.write().to("output.txt"));

三、统一状态管理实现

1. 状态存储架构

text

复制

下载

┌─────────────────────────────────────────┐
│          应用层（流/批处理）              │
├─────────────────────────────────────────┤
│    统一状态API（KeyedState/OperatorState）│
├─────────────────────────────────────────┤
│  状态后端抽象（StateBackend Interface）    │
├─────────────────────────────────────────┤
│  RocksDB      │  Heap      │  Filesystem │
│  (流式优化)    │ (批处理优化)│ (检查点存储)  │
└─────────────────────────────────────────┘

篇幅限制下面就只能给大家展示小册部分内容了。整理了一份核心面试笔记包括了：Java面试、Spring、JVM、MyBatis、Redis、MySQL、并发编程、微服务、Linux、Springboot、SpringCloud、MQ、Kafc

需要全套面试笔记及答案
【点击此处即可/免费获取】

2. 状态后端实现

java

复制

下载

// Flink状态后端配置示例
StreamExecutionEnvironment env = StreamExecutionEnvironment.get_execution_environment();

// RocksDBStateBackend - 适合大规模状态
env.setStateBackend(new RocksDBStateBackend(
    "hdfs://namenode:40010/flink/checkpoints",
    true  // 异步快照
));

// FsStateBackend - 适合中小规模状态
env.setStateBackend(new FsStateBackend(
    "hdfs://namenode:40010/flink/checkpoints",
    false  // 是否异步
));

// HeapStateBackend - 适合测试和小规模
env.setStateBackend(new MemoryStateBackend());

// 统一状态访问
DataStream<Tuple2<String, Integer>> stream = ...;
stream.keyBy(0)
    .flatMap(new RichFlatMapFunction<Tuple2<String, Integer>, Tuple2<String, Integer>>() {
        
        private transient ValueState<Integer> sumState;
        
        @Override
        public void open(Configuration parameters) {
            // 定义状态描述符
            ValueStateDescriptor<Integer> descriptor = 
                new ValueStateDescriptor<>("sum", Integer.class);
            sumState = getRuntimeContext().getState(descriptor);
        }
        
        @Override
        public void flatMap(Tuple2<String, Integer> value, Collector<Tuple2<String, Integer>> out) {
            Integer currentSum = sumState.value();
            if (currentSum == null) {
                currentSum = 0;
            }
            currentSum += value.f1;
            sumState.update(currentSum);
            out.collect(Tuple2.of(value.f0, currentSum));
        }
    });

3. 状态持久化和恢复

java

复制

下载

// 检查点配置
env.enableCheckpointing(60000); // 每60秒做一次checkpoint
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointTimeout(30000);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
env.getCheckpointConfig().enableExternalizedCheckpoints(
    ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

// 保存点（Savepoint）管理
String savepointPath = "hdfs:///savepoints/savepoint-123";
// 触发保存点
String triggerSavepoint = env.triggerSavepoint(savepointPath);

// 从保存点恢复
StreamExecutionEnvironment restoredEnv = 
    StreamExecutionEnvironment.get_execution_environment();
restoredEnv.setStateBackend(...);
restoredEnv.restoreState(savepointPath);

四、流批统一的计算模型

1. 统一窗口机制

sql

复制

下载

-- Apache Flink SQL的窗口语法
-- 流处理窗口
SELECT 
    user_id,
    TUMBLE_START(order_time, INTERVAL '1' HOUR) as window_start,
    COUNT(*) as order_count,
    SUM(amount) as total_amount
FROM orders_stream
GROUP BY 
    user_id,
    TUMBLE(order_time, INTERVAL '1' HOUR)

-- 批处理窗口（历史数据重计算）
SELECT 
    user_id,
    TUMBLE_START(order_time, INTERVAL '1' HOUR) as window_start,
    COUNT(*) as order_count,
    SUM(amount) as total_amount
FROM orders_batch
GROUP BY 
    user_id,
    TUMBLE(order_time, INTERVAL '1' HOUR)

-- 事件时间处理统一
CREATE TABLE orders_with_watermark (
    order_id STRING,
    user_id BIGINT,
    amount DOUBLE,
    order_time TIMESTAMP(3),
    WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND
) WITH (...);

2. 时间语义统一

python

复制

下载

# 使用同一套时间处理API
from pyflink.common import WatermarkStrategy, TimeCharacteristic
from pyflink.common.watermark_strategy import TimestampAssigner

# 事件时间处理（流批统一）
class OrderTimestampAssigner(TimestampAssigner):
    def extract_timestamp(self, value, record_timestamp):
        return value.order_time

# 流处理
stream_env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
watermark_strategy = WatermarkStrategy.for_bounded_out_of_orderness(
    Duration.of_seconds(5)
).with_timestamp_assigner(OrderTimestampAssigner())

# 批处理也使用相同的时间提取逻辑
batch_env.set_parallelism(1)

五、增量计算与状态维护

1. 增量处理模式

java

复制

下载

// 使用MapState实现增量聚合
public class IncrementalAggregator extends 
    KeyedProcessFunction<String, Order, AggregationResult> {
    
    private transient MapState<Long, Double> hourlyState; // key: hour, value: sum
    
    @Override
    public void open(Configuration parameters) {
        MapStateDescriptor<Long, Double> descriptor = 
            new MapStateDescriptor<>("hourly-amount", Long.class, Double.class);
        hourlyState = getRuntimeContext().getMapState(descriptor);
    }
    
    @Override
    public void processElement(Order order, Context ctx, 
                              Collector<AggregationResult> out) throws Exception {
        
        long hour = order.getTimestamp() / (60 * 60 * 1000);
        
        // 增量更新
        Double current = hourlyState.get(hour);
        if (current == null) {
            current = 0.0;
        }
        current += order.getAmount();
        hourlyState.put(hour, current);
        
        // 输出当前聚合结果
        out.collect(new AggregationResult(order.getUserId(), hour, current));
    }
}

2. 状态TTL管理

java

复制

下载

// 状态生存时间配置
StateTtlConfig ttlConfig = StateTtlConfig
    .newBuilder(Time.hours(24))  // 24小时TTL
    .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)  // 更新时刷新TTL
    .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired) // 不返回过期数据
    .cleanupInBackground()  // 后台清理
    .build();

ValueStateDescriptor<Integer> descriptor = 
    new ValueStateDescriptor<>("user-session", Integer.class);
descriptor.enableTimeToLive(ttlConfig);

六、数据一致性保障

1. 端到端Exactly-Once实现

java

复制

下载

// 两阶段提交实现端到端一致性
public class TwoPhaseCommitSink extends RichSinkFunction<Order> 
    implements CheckpointedFunction, CheckpointListener {
    
    private transient Connection connection;
    private transient PreparedStatement statement;
    private transient List<Order> pendingRecords;
    
    @Override
    public void invoke(Order value, Context context) throws Exception {
        // 添加到待提交列表
        pendingRecords.add(value);
    }
    
    @Override
    public void snapshotState(FunctionSnapshotContext context) throws Exception {
        // 第一阶段：预提交
        connection.setAutoCommit(false);
        for (Order order : pendingRecords) {
            statement.setString(1, order.getId());
            statement.setDouble(2, order.getAmount());
            statement.addBatch();
        }
        statement.executeBatch();
        // 不提交，等待checkpoint完成
    }
    
    @Override
    public void notifyCheckpointComplete(long checkpointId) throws Exception {
        // 第二阶段：正式提交
        connection.commit();
        pendingRecords.clear();
    }
    
    @Override
    public void initializeState(FunctionInitializationContext context) throws Exception {
        // 状态恢复
        if (context.isRestored()) {
            // 从检查点恢复状态
            ListState<Order> restoredState = context.getOperatorStateStore()
                .getListState(new ListStateDescriptor<>("pending-records", Order.class));
            for (Order order : restoredState.get()) {
                pendingRecords.add(order);
            }
        }
    }
}

2. 幂等性写入

python

复制

下载

# 基于主键的幂等写入
class IdempotentDatabaseSink:
    def __init__(self, connection_params):
        self.conn = create_connection(connection_params)
        self.processed_ids = set()  # 已处理ID集合
        
    def write_with_idempotency(self, records):
        with self.conn.cursor() as cursor:
            for record in records:
                # 检查是否已处理
                if record.id in self.processed_ids:
                    continue
                    
                # 使用INSERT ON CONFLICT或MERGE实现幂等
                sql = """
                    INSERT INTO orders (id, user_id, amount, timestamp)
                    VALUES (%s, %s, %s, %s)
                    ON CONFLICT (id) DO NOTHING
                """
                cursor.execute(sql, (record.id, record.user_id, 
                                    record.amount, record.timestamp))
                
                self.processed_ids.add(record.id)
            
            self.conn.commit()

七、架构实践案例

1. 电商实时数仓流批一体实现

yaml

复制

下载

# 架构配置
architecture:
  processing_engine: flink
  state_backend: rocksdb
  metastore: hive
  
layers:
  ods:
    source: kafka
    format: json
    watermark: event_time
    
  dwd:
    processing: 
      - deduplication
      - dimension_join
      - normalization
    state: 
      ttl: 7d
      backend: rocksdb
      
  dws:
    windows:
      - tumbling_1h
      - sliding_5m
      - session_10m
    aggregations:
      - count
      - sum
      - uv
      
  ads:
    serving: 
      - mysql
      - redis
      - clickhouse
    refresh: 
      streaming: incremental
      batch: full

2. Flink + Iceberg实现湖仓一体

sql

复制

下载

-- 创建Iceberg表
CREATE CATALOG hive_catalog WITH (
  'type'='iceberg',
  'catalog-type'='hive',
  'uri'='thrift://localhost:9083',
  'warehouse'='hdfs://localhost:9000/warehouse'
);

USE CATALOG hive_catalog;

-- 流式写入
INSERT INTO user_behavior_iceberg
SELECT user_id, item_id, action, event_time
FROM kafka_user_behavior;

-- 批量读取（同一张表）
SELECT user_id, COUNT(*) as action_count
FROM user_behavior_iceberg
WHERE event_date = '2024-01-01'
GROUP BY user_id;

-- 时间旅行查询
SELECT * FROM user_behavior_iceberg 
FOR SYSTEM_TIME AS OF timestamp '2024-01-01 10:00:00';

-- 增量读取
SELECT * FROM user_behavior_iceberg
WHERE event_date >= '2024-01-01' 
  AND __change_type != 'DELETE';

八、性能优化策略

1. 状态优化

java

复制

下载

// 1. 状态序列化优化
env.getConfig().registerTypeWithKryoSerializer(
    UserBehavior.class, 
    CustomKryoSerializer.class
);

// 2. 状态分区优化
DataStream<Tuple2<String, Integer>> partitionedStream = 
    stream.keyBy(new KeySelector<Tuple2<String, Integer>, String>() {
        @Override
        public String getKey(Tuple2<String, Integer> value) {
            // 自定义分区逻辑
            return value.f0.substring(0, 3); // 前3字符作为key
        }
    });

// 3. RocksDB优化配置
RocksDBStateBackend backend = new RocksDBStateBackend("hdfs://checkpoints");
backend.setDbStoragePath("/tmp/rocksdb");  // 本地存储路径
backend.setPredefinedOptions(PredefinedOptions.SPINNING_DISK_OPTIMIZED);

2. 检查点优化

java

复制

下载

// 对齐优化
env.getCheckpointConfig().enableUnalignedCheckpoints();
env.getCheckpointConfig().setAlignedCheckpointTimeout(Duration.ofMillis(100));

// 增量检查点（RocksDB）
RocksDBStateBackend backend = (RocksDBStateBackend) env.getStateBackend();
backend.enableIncrementalCheckpointing(true);

// 检查点压缩
env.getCheckpointConfig().enableExternalizedCheckpoints(
    ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(5000);

篇幅限制下面就只能给大家展示小册部分内容了。整理了一份核心面试笔记包括了：Java面试、Spring、JVM、MyBatis、Redis、MySQL、并发编程、微服务、Linux、Springboot、SpringCloud、MQ、Kafc

需要全套面试笔记及答案
【点击此处即可/免费获取】

九、监控与运维

1. 状态监控指标

prometheus

复制

下载

# Prometheus监控指标
flink_taskmanager_job_task_stateSize_bytes
flask_taskmanager_job_task_numRegisteredStates
flink_taskmanager_job_task_lastCheckpointSize
flink_jobmanager_job_lastCheckpointDuration
flink_taskmanager_job_task_checkpointAlignmentTime

# 自定义状态监控
public class StateMonitor extends RichMapFunction<String, String> {
    
    @Override
    public String map(String value) throws Exception {
        RuntimeContext ctx = getRuntimeContext();
        
        // 获取状态统计
        MetricGroup metrics = ctx.getMetricGroup();
        metrics.gauge("state-size", new Gauge<Long>() {
            @Override
            public Long getValue() {
                return estimateStateSize();
            }
        });
        
        return value;
    }
}

2. 自动扩缩容

yaml

复制

下载

# Flink on K8s自动扩缩配置
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: stream-batch-unified
spec:
  flinkVersion: v1_17
  image: flink:1.17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    kubernetes.operator.job.autoscaler.enabled: "true"
    kubernetes.operator.job.autoscaler.metric: "cpu"
    kubernetes.operator.job.autoscaler.target: "70"
    kubernetes.operator.job.autoscaler.stabilization.interval: "5m"

十、最佳实践总结

1. 状态设计原则

• 最小化：只存储必要状态

• 分区化：合理设计key避免数据倾斜

• 生命周期管理：设置合适的TTL

• 序列化优化：选择高效序列化方式

2. 流批统一要点

• 使用事件时间作为统一时间语义

• 实现增量计算而非全量重算

• 设计幂等的输出结果

• 保持流批处理逻辑一致性

3. 实施路径建议

text

复制

下载

1. 评估阶段：分析现有Lambda架构痛点
2. 试点阶段：选择非核心业务进行验证
3. 扩展阶段：逐步迁移核心业务
4. 优化阶段：性能调优和监控完善
5. 治理阶段：建立标准和最佳实践

流批一体架构是现代数据处理的必然趋势，正确实现状态管理和统一计算模型是关键。建议从Flink或Spark Structured Streaming开始实践，逐步构建完整的流批一体数据平台。