pyflink task并行度问题

复制代码
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.functions import RuntimeContext, FlatMapFunction, MapFunction
import json
import re
import logging
import sys
from pyflink.datastream.state import ValueStateDescriptor, MapStateDescriptor
from pyflink.datastream.connectors.kafka import FlinkKafkaConsumer, TypeInformation,FlinkKafkaProducer
from pyflink.common.typeinfo import Types
from pyflink.datastream.connectors.elasticsearch import Elasticsearch7SinkBuilder, ElasticsearchEmitter, FlushBackoffType
from  pyflink.datastream.connectors import  DeliveryGuarantee
from pyflink.common.serialization import SimpleStringSchema
from datetime import datetime




logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(asctime)s-%(levelname)s-%(message)s")
logger = logging.getLogger(__name__)

# ���� StreamExecutionEnvironment ����
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
env.add_jars("file:///root/flink-sql-connector-kafka_2.11-1.14.4.jar")
from pyflink.datastream import DataStream, StreamExecutionEnvironment
from pyflink.datastream.functions import RuntimeContext, FlatMapFunction, MapFunction
from pyflink.common.typeinfo import Types

env = StreamExecutionEnvironment.get_execution_environment()
data = DataStream(env._j_stream_execution_environment.socketTextStream('192.168.137.201', 8899))
#调用map算子,封装成一个task,并行度为8,有8个subtask
ds1=data.map(lambda s: s.upper()).set_parallelism(8)
##sink算子,并行度为4
ds1.print().set_parallelism(4)
相关推荐
勘察加熊人22 分钟前
Python+Streamlit实现登录页
开发语言·python
DavieLau26 分钟前
Python开发后端InfluxDB数据库测试接口
服务器·数据库·python·时序数据库
文人sec1 小时前
接口自动化测试设计思路--设计实战
python·https·单元测试·自动化·pytest
子燕若水1 小时前
Flask 调试的时候进入main函数两次
后端·python·flask
编程有点难1 小时前
Python训练打卡Day23
开发语言·python
qq_14182697322 小时前
python通过curl访问deepseek的API调用案例
java·数据库·python
红衣小蛇妖3 小时前
Python基础学习-Day23
开发语言·python·学习
Hello world.Joey4 小时前
数据挖掘入门-二手车交易价格预测
人工智能·python·数据挖掘·数据分析·conda·pandas
刘延林.4 小时前
树莓5安装 PyCharm 进行python脚本开发
ide·python·pycharm
小洛~·~4 小时前
多模态RAG与LlamaIndex——1.deepresearch调研
人工智能·python·深度学习·神经网络·chatgpt