project 中遇到
log file & project file
build.sbt
startup.sh
> sh startup.sh
example:
.sbt
name := "Simple Project" //project name version := "1.0" //project version scalaVersion := "2.11.8" //scala version libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % "2.1.0" libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.2.11"
//all Dependencies should be aware of the version compatibility
sh
/usr/local/spark/bin/spark-submit --driver-class-path /usr/local/spark/jars/*:/usr/local/spark/jars/kafka/* --class "org.apache.spark.examples.streaming.KafkaWordCount" /usr/local/spark/mycode/kafka/target/scala-2.11/simple-project_2.11-1.0.jar 127.0.0.1:2181 1 sex 1
spark-submit目录
参数：
--driver-class-path spark中的所有jars以及kafka中所有jars
--class 工程文件main所在class + 生成的jar的位置
127.0.0.1:2181 ： zookeeper-server kafka-server ip+port
1 sex 1： consumer group；topics； number of consumers
REFERENCE:
http://dblab.xmu.edu.cn
原文：http://www.cnblogs.com/yumanman/p/7599548.html