首页 > 其他 > 详细

spark WARNTaskSchedulerImpl:Initial job has not accepted any resources; check your cluster UI to

时间:2015-03-26 23:26:10      阅读:1096      评论:0      收藏:0      [点我收藏+]

spark在提交任务时,出现如下错误:

15/03/26 22:29:36 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/03/26 22:29:51 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/03/26 22:30:06 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/03/26 22:30:21 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

技术分享

从警告信息上看,初始化job时没有获取到任何资源;提示检查集群,确保workers可以被注册并有足够的内存资源。


如上问题产生的原因是多方面的,可能原因如下:


1.因为提交任务的节点不能和spark工作节点交互,因为提交完任务后提交任务节点上会起一个进程,展示任务进度,大多端口为4044,工作节点需要反馈进度给该该端口,所以如果主机名或者IP在hosts中配置不正确。所以检查下主机名和ip是否配置正确


2.有可能是内存不足

  检查内存

   conf.set("spark.executor.memory", "3000m")

  Make sure to set SPARK_LOCAL_IP andSPARK_MASTER_IP.

  查看8080端口,确保一些workers保持Alive状态,确保 some cores 是可利用的。

 





spark WARNTaskSchedulerImpl:Initial job has not accepted any resources; check your cluster UI to

原文:http://blog.csdn.net/u013361361/article/details/44657205

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!