记一次spark在docker本地启动报错

1，背景

在docker中部署spark服务和调用spark服务的微服务，微服务之间通过fegin调用

2，问题，docker容器中服务器来后，注册中心都有，调用服务也正常，但是调用spark启动任务后报错，报错信息

exception in thread "main" org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@pinganan_datastudio_pinganan-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net:35159
	at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
	at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:134)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
	at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:32)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:184)
	at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
	at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:127)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:183)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
	at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.<init>(SparkContext.java:78)
	at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.getInstance(SparkContext.java:104)
	at com.itcom.pinganan.standmodel.engine.spark.context.SparkContext.getInstance(SparkContext.java:99)
	at com.itcom.pinganan.standmodel.engine.spark.Launcher.main(Launcher.java:49)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
25/06/05 15:34:43 INFO ShutdownHookManager: Shutdown hook called
25/06/05 15:34:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-98ea3d8e-e10e-4766-8f54-e9e4bd4499d8
25/06/05 15:34:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-abd80629-1d00-4f9d-8ab0-f337e6de09ab

3，分析
HeartbeatReceiver@pinganan_datastudio_pinganan-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net这个是中pinganan-scheduler-worker这个服务在注册中心的虚拟域名，解析不到这个域名，
注册中心的服务信息

```java
pinganan_datastudio_everdc-scheduler-worker.1.n9zf1yk8znety8x2lw2pzn9a8.db2_net:pinganan-scheduler-worker:8080 【11.10.33.67】

4，尝试将这个服务注册的ip在编排文件中指定这个服务的ip

      environment:
      - eureka.instance.prefer-ip-address=true
      - eureka.instance.ip-address=192.168.10.224
      - eureka.instance.non-secure-port=8080

5，运行后报错

调用5park执行引擎服务出错:
feign.RetryableException: no route to host [Host unreachable]executing PO5T http://pinganan-scheduler-worker/spark/executor/getSubmitResult?applicationПame=
pinganan-data-studio&modelRunld=12 at feign.FeignException.errorExecuting[FeignExcep
tion.java:249]~

6，分析上面问题

这样制定后，其他服务fegin调用失败，找不到这个服务，只能再改回去

7，尝试在spark提交任务中指定driver的ip，spark.driver.host=11.10.33.67

提交运行正常

8，但是如果重启docker服务后这个ip会变化，所以将这个服务注册的ip在编排文件中更改指定一下

将 - net:

• db2_net

替换为

networks:

db2_net:

ipv4_address: 11.10.13.67

9，重启任务后，提交任务正常