spark-submit \
--master yarn \
--deploy-mode client \
--executor-memory 20g \
--executor-cores 4 \
--num-executors 3 \
--class com.cpic.sparkdemo.downTest.downTxtSelect /cpic/cpicapp/lm_test/jav
报错log
Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:86)at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)at org.apache.spark.SparkContext.<init>(SparkContext.scala:505)at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2508)at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:939)at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)at scala.Option.getOrElse(Option.scala:121)at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:930)at com.cpic.sparkdemo.downTest.downTxtSelect.main(downTxtSelect.java:19)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:920)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:195)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:220)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:140)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[cpicapp@Cloud System!! DT~/lm_test]$timed out waiting for input: auto-logout
同时 管理平台日志是:
Uncaught exception: | org.apache.spark.internal.Logging$class.logError(Logging.scala:91)
org.apache.spark.SparkException: Failed to connect to driver!at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:630)at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:490)at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:304)at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:242)at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:242)at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:242)at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:786)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:785)at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:241)at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:812)at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:844)at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
2020-06-23 13:28:28,898 | INFO | [main] | Final app status: FAILED, exitCode: 13, (rea
暂时还没有找到合适的解决方法 只能记录问题