记一次spark在docker本地启动报错

1,背景

在docker中部署spark服务和调用spark服务的微服务,微服务之间通过fegin调用

2,问题,docker容器中服务器来后,注册中心都有,调用服务也正常,但是调用spark启动任务后报错,报错信息

java 复制代码
exception in thread "main" org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@pinganan_datastudio_pinganan-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net:35159
	at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
	at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
	at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:184)
	at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
	at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:127)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:183)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
	at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.<init>(SparkContext.java:78)
	at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.getInstance(SparkContext.java:104)
	at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.getInstance(SparkContext.java:99)
	at com.itcom.pinganan.standmodel.engine.spark.Launcher.main(Launcher.java:49)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
25/06/05 15:34:43 INFO ShutdownHookManager: Shutdown hook called
25/06/05 15:34:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-98ea3d8e-e10e-4766-8f54-e9e4bd4499d8
25/06/05 15:34:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-abd80629-1d00-4f9d-8ab0-f337e6de09ab
复制代码
3,分析
HeartbeatReceiver@pinganan_datastudio_pinganan-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net这个是中pinganan-scheduler-worker这个服务在注册中心的虚拟域名,解析不到这个域名,
注册中心的服务信息

```java
pinganan_datastudio_everdc-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net:pinganan-scheduler-worker:8080 【11.10.33.67】

4,尝试将这个服务注册的ip在编排文件中指定这个服务的ip

java 复制代码
      environment:
      - eureka.instance.prefer-ip-address=true
      - eureka.instance.ip-address=192.168.10.224
      - eureka.instance.non-secure-port=8080

5,运行后报错

java 复制代码
调用5park执行引擎服务出错:
feign.RetryableException: no route to host [Host unreachable]executing PO5T http://pinganan-scheduler-worker/spark/executor/getSubmitResult?applicationПame=
pinganan-data-studio&modelRunld=12 at feign.FeignException.errorExecuting[FeignExcep
tion.java:249]~

6,分析上面问题

这样制定后,其他服务fegin调用失败,找不到这个服务,只能再改回去

7,尝试在spark提交任务中指定driver的ip,spark.driver.host=11.10.33.67

提交运行正常

8,但是如果重启docker服务后这个ip会变化,所以将这个服务注册的ip在编排文件中更改指定一下

将 - net:

  • db2_net

替换为

networks:

db2_net:

ipv4_address: 11.10.13.67

9,重启任务后,提交任务正常

相关推荐
数据与人工智能律师4 小时前
数字迷雾中的安全锚点:解码匿名化与假名化的法律边界与商业价值
大数据·网络·人工智能·云计算·区块链
勇哥的编程江湖4 小时前
starrocks官网docker部署mysql无法连接
运维·docker·容器
mykyle6 小时前
Elasticsearch-ik分析器
大数据·elasticsearch·jenkins
小码过河.7 小时前
CentOS 搭建 Docker 私有镜像仓库
linux·docker·centos
weixin_lynhgworld7 小时前
淘宝扭蛋机小程序系统开发:重塑电商互动模式
大数据·小程序
RPA+AI十二工作室9 小时前
影刀RPA_Temu关键词取数_源码解读
大数据·自动化·源码·rpa·影刀
飞询9 小时前
部署 coze-loop
docker·coze
龙卷风040510 小时前
使用本地IDEA连接服务器远程构建部署Docker服务
后端·docker
Sui_Network10 小时前
探索 Sui 上 BTCfi 的各类资产
大数据·人工智能·科技·游戏·区块链
大数据张老师11 小时前
用 AI 做数据分析:从“数字”里挖“规律”
大数据·人工智能