记一次运行spark报错

提交spark任务运次报错

java 复制代码
06/03 18:27:50 INFO Client: Setting up container launch context for our AM
25/06/03 18:27:50 INFO Client: Setting up the launch environment for our AM container
25/06/03 18:27:50 INFO Client: Preparing resources for our AM container
25/06/03 18:27:50 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
25/06/03 18:27:59 INFO Client: Uploading resource file:/tmp/spark-27957b63-3767-457f-a931-cc7f84917c3e/__spark_libs__545909112269949051.zip -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/__spark_libs__545909112269949051.zip
25/06/03 18:28:02 INFO Client: Uploading resource file:/opt/apps/everdc/spark_jars/everdc-stand-model-engine-spark-4.9.0.jar -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/everdc-stand-model-engine-spark-4.9.0.jar
25/06/03 18:28:02 INFO Client: Uploading resource file:/app/logs/model_run_2.json -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/model_run_2.json
25/06/03 18:28:02 INFO Client: Uploading resource file:/tmp/spark-27957b63-3767-457f-a931-cc7f84917c3e/__spark_conf__4210376678626642445.zip -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/__spark_conf__.zip
25/06/03 18:28:03 INFO SecurityManager: Changing view acls to: root,hdfs
25/06/03 18:28:03 INFO SecurityManager: Changing modify acls to: root,hdfs
25/06/03 18:28:03 INFO SecurityManager: Changing view acls groups to: 
25/06/03 18:28:03 INFO SecurityManager: Changing modify acls groups to: 
25/06/03 18:28:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root, hdfs); groups with view permissions: Set(); users  with modify permissions: Set(root, hdfs); groups with modify permissions: Set()
25/06/03 18:28:03 INFO Client: Deleted staging directory hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPassword(Ljava/lang/String;)[C
	at org.apache.spark.SSLOptions$.$anonfun$parse$8(SSLOptions.scala:194)
	at scala.Option.orElse(Option.scala:447)
	at org.apache.spark.SSLOptions$.parse(SSLOptions.scala:194)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:98)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:1137)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:220)
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1327)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)

报错中org.apache.hadoop.conf.Configuration.getPassword,这个方法找不到,hadoop在2.7以上版本中才有这个方法,查看运行hadoop环境hadoop是2.4.5的版本的,升级到hadoop2.7后正常

相关推荐
mykyle25 分钟前
Elasticsearch-ik分析器
大数据·elasticsearch·jenkins
itLaity28 分钟前
基于Kafka实现简单的延时队列
spring boot·分布式·kafka
qq_5298353538 分钟前
Zookeeper的简单了解
分布式·zookeeper·云原生
weixin_lynhgworld2 小时前
淘宝扭蛋机小程序系统开发:重塑电商互动模式
大数据·小程序
smileNicky2 小时前
RabbitMQ有多少种Exchange?
分布式·rabbitmq
你我约定有三2 小时前
RabbitMQ--消息丢失问题及解决
java·开发语言·分布式·后端·rabbitmq·ruby
Java初学者小白2 小时前
秋招Day19 - 分布式 - 分布式事务
java·分布式
RPA+AI十二工作室4 小时前
影刀RPA_Temu关键词取数_源码解读
大数据·自动化·源码·rpa·影刀
程序员小羊!4 小时前
Zookeeper 3.6.3【详细技术讲解】整
分布式·zookeeper·云原生
Sui_Network4 小时前
探索 Sui 上 BTCfi 的各类资产
大数据·人工智能·科技·游戏·区块链