Spark:intersection算子

intersection是Tranformation算子,具有shuffle

它是去二个rdd的交集,也就是共同存在的

源码:
让二个rdd经过map阶段形成元组,value为null(1,null).
在调cogroup阶段让有相同key的value都没null,
filter过滤value,如果value1,和value2不为空说明.第一个rdd出现的值在第二个rdd也出现了
keys,取出所有的key/和values一样object IntersectionDemo {def main(args: Array[String]): Unit = {val conf = new SparkConf().setAppName("IntersectionDemo").setMaster("local[*]")val sc = new SparkContext(conf)val rdd1 = sc.parallelize(List(5,6,4,7))val rdd2 = sc.parallelize(List(1,2,3,4))val rdd3: RDD[(Int, Null)] = rdd1.map((_, null))val rdd4: RDD[(Int, Null)] = rdd2.map((_, null))val rdd5: RDD[(Int, (Iterable[Null], Iterable[Null]))] = rdd3.cogroup(rdd4)val keys = rdd5.filter(it =>it._2._1.nonEmpty && it._2._2.nonEmpty).keyskeys.saveAsTextFile("IntersectionDemo-out2")sc.stop()}}