Spark ShuffleDependency Shuffle依赖关系
- Represents a dependency on the output of a shuffle stage. Note that in the case of shuffle,the RDD is transient since we don’t need it on the executor side.
更多资源
- github: https://github.com/opensourceteams/spark-scala-maven
- csdn(汇总视频在线看): https://blog.csdn.net/thinktothings/article/details/84726769
youtub视频演示
- https://youtu.be/8T6PyHuf_wQ (youtube视频)
- https://www.bilibili.com/video/av37442139/?p=5 (bilibili视频)
- github: https://github.com/opensourceteams/spark-scala-maven
输入数据
List(('c',1),('b',1),('a',1),('a',1)
处理程序scala
package com.opensource.bigdata.spark.local.rdd.operation.dependency.shuffle.n_01_ShuffleDependency
import com.opensource.bigdata.spark.local.rdd.operation.base.BaseScalaSparkContext
object Run extends BaseScalaSparkContext{
def main(args: Array[String]): Unit = {
val sc = pre()
val rdd1 = sc.parallelize(List(('c',1),('b',1),('a',1),('a',1)),2)
val rdd2 =rdd1.reduceByKey((a,b) => a + b)
println("rdd2\n" + rdd2.collect().mkString("\n"))
sc.stop()
}
}
数据处理图
