pyflink task并行度问题

复制代码
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.functions import RuntimeContext, FlatMapFunction, MapFunction
import json
import re
import logging
import sys
from pyflink.datastream.state import ValueStateDescriptor, MapStateDescriptor
from pyflink.datastream.connectors.kafka import FlinkKafkaConsumer, TypeInformation,FlinkKafkaProducer
from pyflink.common.typeinfo import Types
from pyflink.datastream.connectors.elasticsearch import Elasticsearch7SinkBuilder, ElasticsearchEmitter, FlushBackoffType
from  pyflink.datastream.connectors import  DeliveryGuarantee
from pyflink.common.serialization import SimpleStringSchema
from datetime import datetime




logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(asctime)s-%(levelname)s-%(message)s")
logger = logging.getLogger(__name__)

# ���� StreamExecutionEnvironment ����
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
env.add_jars("file:///root/flink-sql-connector-kafka_2.11-1.14.4.jar")
from pyflink.datastream import DataStream, StreamExecutionEnvironment
from pyflink.datastream.functions import RuntimeContext, FlatMapFunction, MapFunction
from pyflink.common.typeinfo import Types

env = StreamExecutionEnvironment.get_execution_environment()
data = DataStream(env._j_stream_execution_environment.socketTextStream('192.168.137.201', 8899))
#调用map算子,封装成一个task,并行度为8,有8个subtask
ds1=data.map(lambda s: s.upper()).set_parallelism(8)
##sink算子,并行度为4
ds1.print().set_parallelism(4)
相关推荐
程序员清风12 分钟前
Python 数据分析环境搭建:从 Jupyter 到 pandas
python·jupyter·数据分析
月光船幽幽25 分钟前
加性偏移外推提升参数识别可靠性
python·算法
BD_Marathon1 小时前
消息对象中字段的说明
java·前端·python
Java后端的Ai之路1 小时前
Python进阶探索23 - Python中的Time与Datetime模块
开发语言·数据库·python·datetime·time
Java后端的Ai之路2 小时前
Python进阶探索24_应用案例_linux系统的监控
linux·开发语言·python·应用·探索
happylifetree2 小时前
Python22-26:核心语法-流程控制语句-if条件判断
python
程序员清风2 小时前
PydanticAI 实战:用类型安全构建可靠的 Python Agent
开发语言·python·安全
梦想画家3 小时前
SQLMesh Python 模型入门(三):前后置语句、蓝图建模与避坑指南
大数据·python·sqlmesh
迅猛龙办公室3 小时前
Python实现绘制同切圆
开发语言·python
言乐64 小时前
Python语音检索
开发语言·python·django·virtualenv·pygame