spark 类对象去重

override def equals(obj: Any): Boolean = obj match {
  case User(age,_) => this.age == age
  case _ => false
}

}

//Uer 对象重写 hashCode 和equals 方法 判断对象是否相等

sc.parallelize(List(User(11,"op"),User(12,"pp"),User(11,"lp")))
.distinct()
.foreach(println)

结果显示

或者

sc.parallelize(List(User(11,"op"),User(12,"pp"),User(11,"lp")))
.map((x,null))
.reduceByKey((x,)=>x)
.map(._1)
.foreach(println)

spark 类对象去重

原文：https://blog.51cto.com/u_15084467/2720506