Spark - 技术栈

1. Spark概述

2. Spark运行模式及安装部署

3. RDD概述

|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ### 1.1 什么是RDD RDD（Resilient Distributed Dataset）叫做弹性分布式数据集，是Spark中最基本的数据抽象。代码中是一个抽象类，它代表一个弹性的、不可变、可分区、里面的元素可并行计算的集合。 |
| ### 1.2 RDD五大特性 |
| data 类型为RDD（分布式数据集） RDD 算子 collect 收集完装到数组中，数组函数如下 glom repartition 和coalesce的区别 两个都能调整分区数,但repartition的底层依然是调用了coalesce coalesce 的语法: coalesce(num,shuffle=False) 默认不启动shuffle repartition 的语法: repartition(num) 默认启动shuffle repartition 中将shuffle改成了ture,且参数不可修改 因此,repartition常用于增加分区,coalesce常用于减小分区 关键就在于shuffle是否启动 重新分区的根本是通过hash取模后再分区,因此必须通过shuffle 分区数据重新分区时会出现1个分区数据分配到其他多个分区的情况,也就形成了「宽依赖」 减小分区的根本是将1个分区完整归类到另一个分区中,属于1对1的情况,也就形成「窄依赖」 创建RDD的第二种方式mkRDD Spark wordcount 案例 sc.textFile("hdfs://kb129:9000/kb23/tmp/*.txt").flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey(+).glom.collect |
| WordCount 工作流程及 API Scala import org.apache.spark.sql.SparkSession import org.apache.spark.{SparkConf, SparkContext} object WordCountDemo { def main(args: Array[String]): Unit = { //创建SparkConf并设置App名称 val conf = new SparkConf().setMaster("local[*]").setAppName("wordCount") //创建Spark context，该对象是提交Spark App的入口 val sc = SparkContext.getOrCreate(conf) //创建Spark session val spark = SparkSession.builder().config(conf).getOrCreate() //val rdd = sc.textFile("hdfs://kb129:9000/kb23/tmp/*.txt") //val rdd2 = rdd.flatMap(x => x.split(" ")) //val rdd3 = rdd2.map(x => (x, 1)) //val result = rdd3.reduceByKey((x, y) => x + y) //等价于 val result = sc.textFile("hdfs://kb129:9000/kb23/tmp/*.txt") .flatMap(x => x.split(" ")).map(x => (x, 1)) .reduceByKey((x, y) => x + y) //控制台输出结果 result.collect.foreach(println) //结果保存到hdfs中 //result.saveAsTextFile("hdfs://kb129:9000/kb23/tmp/wordcount") } } |
| RDD 算子练习 --- 求出"张"姓学生总分最高的学生姓名、总分 Scala //1 sortBy方式求"张"姓总分最高 sc.parallelize(Array(("zhangsan", 100, 88, 80), ("lisi", 80, 66, 99), ("zhangxuefeng", 99, 8, 100), ("zhangwuji", 9, 80, 10))) .filter(x => x._1.startsWith("zhang")) .map(x => (x._1, x._2 + x._3 + x._4)) .sortBy(x => - x._2) .take(1) .foreach(println) //(zhangsan,279) //2 reduce方式求"张"姓总分最高 println( sc.parallelize(Array(("zhangsan", 100, 88, 80), ("lisi", 80, 66, 99), ("zhangxuefeng", 99, 8, 100), ("zhangwuji", 9, 80, 10))) .filter(x => x._1.startsWith("zhang")) .map(x => (x._1, x._2 + x._3 + x._4)) .reduce((x, y) => {if (x._2 > y._2) x else y}) ) //(zhangsan,279) //3 reduceByKey方式求"张"姓总分最高 sc.parallelize(Array(("zhangsan", 100, 88, 80), ("lisi", 80, 66, 99), ("zhangxuefeng", 99, 8, 100), ("zhangwuji", 9, 80, 10))) .filter(x => x._1.startsWith("zhang")) .map(x => ("zhang",(x._1, x._2 + x._3 + x._4))) .reduceByKey((x, y) => {if (x._2 >y._2) x else y}) .map(x => x._2) .collect().foreach(println) //(zhangsan,279) //4 max方式求"张"姓总分最高 println( sc.parallelize(Array(("zhangsan", 100, 88, 80), ("lisi", 80, 66, 99), ("zhangxuefeng", 99, 8, 100), ("zhangwuji", 9, 80, 10))) .filter(x => x._1.startsWith("zhang")) .map(x => (x._2 + x._3 + x._4, x._1)) .max .swap ) //(zhangsan,279) //5 groupBy方式求"张"姓总分最高 sc.parallelize(Array(("zhangsan", 100, 88, 80), ("lisi", 80, 66, 99), ("zhangxuefeng", 99, 8, 100), ("zhangwuji", 9, 80, 10))) .filter(x => x._1.startsWith("zhang")) .map(x => (x._1, x._2 + x._3 + x._4)) .groupBy(x => x._1.substring(0, 5)) .map(x => { var name = ""; var sumscore = 0; val itor = x._2.iterator; for (e <- itor) { if (e._2 > sumscore) { name = e._1; sumscore = e._2 } }; (name, sumscore) }).collect().foreach(println) //(zhangsan,279) |
| ### 2.6 RDD依赖关系 #### 2.6.1 查看血缘关系 RDD只支持粗粒度转换，即在大量记录上执行的单个操作。将创建RDD的一系列Lineage（血统）记录下来，以便恢复丢失的分区。RDD的Lineage会记录RDD的元数据信息和转换行为，当该RDD的部分分区数据丢失时，它可以根据这些信息来重新运算和恢复丢失的数据分区。 #### 2.6.3 窄依赖窄依赖表示每一个父RDD的Partition最多被子RDD的一个Partition使用(一对一or多对一)，窄依赖我们形象的比喻为独生子女。 #### 2.6.4 宽依赖宽依赖表示同一个父RDD的Partition被多个子RDD的Partition依赖(只能是一对多)，会引起Shuffle，总结：宽依赖我们形象的比喻为超生。具有宽依赖的transformations 包括：sort 、reduceByKey 、groupByKey 、join 和调用rePartition 函数的任何操作。宽依赖对Spark去评估一个transformations有更加重要的影响，比如对性能的影响。在不影响业务要求的情况下，要尽量避免使用有宽依赖的转换算子，因为有宽依赖,就一定会走shuffle，影响性能 #### 2.6.5 Stage任务划分（面试重点） 1）DAG有向无环图 DAG（Directed Acyclic Graph）有向无环图是由点和线组成的拓扑图形，该图形具有方向，不会闭环。例如，DAG记录了RDD的转换过程和任务的阶段。 2）任务运行的整体流程 3）RDD任务切分中间分为：Application、Job、Stage和Task （1）Application：初始化一个SparkContext即生成一个Application；（2）Job：一个Action算子就会生成一个Job；（3）Stage：Stage等于宽依赖的个数加1；（4）Task：一个Stage阶段中，最后一个RDD的分区个数就是Task的个数。注意：Application->Job->Stage->Task每一层都是1对n的关系。 |
| ### 2.3 Transformation转换算子（面试开发重点） RDD整体上分为Value类型、双Value类型和Key-Value类型 #### 2.3.1 Value类型 map() 映射 mapPartitions() 以分区为单位执行Map mapPartitionsWithIndex() 带分区号 flatMap() 扁平化 glom() 分区转换数组 groupBy() 分组 filter() 过滤 sample() 采样 distinct() 去重 coalesce() 合并分区 repartition() 重新分区（执行Shuffle） sortBy() 排序 pipe() 调用脚本 #### 2.3.2 双Value类型交互 intersection() 交集 union() 并集不去重 subtract() 差集 zip() 拉链 #### 2.3.3 Key-Value类型 partitionBy() 按照K重新分区 reduceByKey() 按照K聚合V groupByKey() 按照K重新分组 aggregateByKey() 按照K处理分区内和分区间逻辑 foldByKey() 分区内和分区间相同的aggregateByKey() combineByKey() 转换结构后分区内和分区间操作 sortByKey() 按照K进行排序 mapValues() 只对V进行操作 join() 等同于sql里的内连接,关联上的要,关联不上的舍弃 cogroup() 类似于sql的全连接，但是在同一个RDD中对key聚合 |
| ### 2.4 Action行动算子行动算子是触发了整个作业的执行。因为转换算子都是懒加载，并不会立即执行。 #### 2.4.1 reduce()聚合 #### 2.4.2 collect()以数组的形式返回数据集 #### 2.4.3 count()返回RDD中元素个数 #### 2.4.4 first()返回RDD中的第一个元素 #### 2.4.5 take()返回由RDD前n个元素组成的数组 #### 2.4.6 takeOrdered()返回该RDD排序后前n个元素组成的数组 #### 2.4.7 aggregate()案例 #### 2.4.8 fold()案例 #### 2.4.9 countByKey()统计每种key的个数 #### 2.4.10 save相关算子 #### 2.4.11 foreach(f)遍历RDD中每一个元素 |
| ### 2.7 RDD持久化 #### 2.7.1 RDD Cache缓存 RDD通过Cache或者Persist方法将前面的计算结果缓存，默认情况下会把数据以序列化的形式缓存在JVM的堆内存中。但是并不是这两个方法被调用时立即缓存，而是触发后面的action算子时，该RDD将会被缓存在计算节点的内存中，并供后面重用。缓存有可能丢失，或者存储于内存的数据由于内存不足而被删除，RDD的缓存容错机制保证了即使缓存丢失也能保证计算的正确执行。通过基于RDD的一系列转换，丢失的数据会被重算，由于RDD的各个Partition是相对独立的，因此只需要计算丢失的部分即可，并不需要重算全部Partition。 3）自带缓存算子 Spark会自动对一些Shuffle操作的中间数据做持久化操作（比如：reduceByKey）。这样做的目的是为了当一个节点Shuffle失败了避免重新计算整个输入。但是，在实际使用的时候，如果想重用数据，仍然建议调用persist或cache。 #### 2.7.2 RDD CheckPoint检查点 1）检查点：是通过将RDD中间结果写入磁盘。 2）为什么要做检查点？由于血缘依赖过长会造成容错成本过高，这样就不如在中间阶段做检查点容错，如果检查点之后有节点出现问题，可以从检查点开始重做血缘，减少了开销。 3）检查点存储路径：Checkpoint的数据通常是存储在HDFS等容错、高可用的文件系统 4）检查点数据存储格式为：二进制的文件 5）检查点切断血缘：在Checkpoint的过程中，该RDD的所有依赖于父RDD中的信息将全部被移除。 6）检查点触发时间：对RDD进行Checkpoint操作并不会马上被执行，必须执行Action操作才能触发。但是检查点为了数据安全，会从血缘关系的最开始执行一遍。 #### 2.7.3 缓存和检查点区别 1）Cache缓存只是将数据保存起来，不切断血缘依赖。Checkpoint检查点切断血缘依赖。 2）Cache缓存的数据通常存储在磁盘、内存等地方，可靠性低。Checkpoint的数据通常存储在HDFS等容错、高可用的文件系统，可靠性高。 3）建议对checkpoint()的RDD使用Cache缓存，这样checkpoint的job只需从Cache缓存中读取数据即可，否则需要再从头计算一次RDD。 4）如果使用完了缓存，可以通过unpersist()方法释放缓存 |

4. 广播变量

5. 累加器

|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| 累加器：分布式共享只写变量。（Executor和Executor之间不能读数据）累加器用来把Executor端变量信息聚合到Driver端。在Driver中定义的一个变量，在Executor端的每个task都会得到这个变量的一份新的副本，每个task更新这些副本的值后，传回Driver端进行合并计算。 |

6.Spark优化

7. Spark SQL概述

8. Spark SQL编程

9. SparkSQL数据的加载与保存

10.Spark数据分析及处理

11. Spark高级操作之json复杂和嵌套数据结构的操作

|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Json 使用参考 https://www.cnblogs.com/tomato0906/articles/7291178.html |
| 练习1---解析Json字符串及转DataFrame转JSON Scala import org.apache.spark.sql.SparkSession import org.apache.spark.sql.types._ object JsonStu { // 定义样例类 case class DeviceData(id: Int, device: String) def main(args: Array[String]): Unit = { val spark = SparkSession.builder().appName("retention").master("local[*]").getOrCreate() val sc = spark.sparkContext import spark.implicits._ import org.apache.spark.sql.functions._ //创建schema结构 val jsonSchema = new StructType() .add("battery_level", LongType) .add("c02_level",LongType) .add("cca3",StringType) .add("cn", StringType) .add("device_id",LongType) .add("device_type", StringType) .add("signal", LongType) .add("ip",StringType) .add("temp", LongType) .add("timestamp", TimestampType) val eventsFromJSONDS = Seq( (0, """{"device_id": 0, "device_type": "sensor-ipad", "ip": "68.161.225.1", "cca3": "USA", "cn": "United States", "temp": 25, "signal": 23, "battery_level": 8, "c02_level": 917, "timestamp" :1475600496 }"""), (1, """{"device_id": 1, "device_type": "sensor-igauge", "ip": "213.161.254.1", "cca3": "NOR", "cn": "Norway", "temp": 30, "signal": 18, "battery_level": 6, "c02_level": 1413, "timestamp" :1475600498 }"""), (2, """{"device_id": 2, "device_type": "sensor-ipad", "ip": "88.36.5.1", "cca3": "ITA", "cn": "Italy", "temp": 18, "signal": 25, "battery_level": 5, "c02_level": 1372, "timestamp" :1475600500 }"""), (3, """{"device_id": 3, "device_type": "sensor-inest", "ip": "66.39.173.154", "cca3": "USA", "cn": "United States", "temp": 47, "signal": 12, "battery_level": 1, "c02_level": 1447, "timestamp" :1475600502 }"""), (4, """{"device_id": 4, "device_type": "sensor-ipad", "ip": "203.82.41.9", "cca3": "PHL", "cn": "Philippines", "temp": 29, "signal": 11, "battery_level": 0, "c02_level": 983, "timestamp" :1475600504 }"""), (5, """{"device_id": 5, "device_type": "sensor-istick", "ip": "204.116.105.67", "cca3": "USA", "cn": "United States", "temp": 50, "signal": 16, "battery_level": 8, "c02_level": 1574, "timestamp" :1475600506 }"""), (6, """{"device_id": 6, "device_type": "sensor-ipad", "ip": "220.173.179.1", "cca3": "CHN", "cn": "China", "temp": 21, "signal": 18, "battery_level": 9, "c02_level": 1249, "timestamp" :1475600508 }"""), (7, """{"device_id": 7, "device_type": "sensor-ipad", "ip": "118.23.68.227", "cca3": "JPN", "cn": "Japan", "temp": 27, "signal": 15, "battery_level": 0, "c02_level": 1531, "timestamp" :1475600512 }"""), (8, """ {"device_id": 8, "device_type": "sensor-inest", "ip": "208.109.163.218", "cca3": "USA", "cn": "United States", "temp": 40, "signal": 16, "battery_level": 9, "c02_level": 1208, "timestamp" :1475600514 }"""), (9, """{"device_id": 9, "device_type": "sensor-ipad", "ip": "88.213.191.34", "cca3": "ITA", "cn": "Italy", "temp": 19, "signal": 11, "battery_level": 0, "c02_level": 1171, "timestamp" :1475600516 }""")) .toDF("id", "json") //eventsFromJSONDS.printSchema() //eventsFromJSONDS.show(3) //直接解析 val df = eventsFromJSONDS.select( get_json_object($"json", "$.device_id").as("device_id"), get_json_object($"json", "$.device_type").as("device_type"), get_json_object($"json", "$.ip").as("ip"), get_json_object($"json", "$.cca3").as("cca3"), get_json_object($"json", "$.cn").as("cn"), get_json_object($"json", "$.temp").as("temp"), get_json_object($"json", "$.signal").as("signal"), get_json_object($"json", "$.battery_level").as("battery_level"), get_json_object($"json", "$.c02_level").as("c02_level"), get_json_object($"json", "$.timestamp").as("timestamp") ) df.printSchema() df.show(3) //使用样例类格式创建DF val eventsDS = Seq( (0, """{"device_id": 0, "device_type": "sensor-ipad", "ip": "68.161.225.1", "cca3": "USA", "cn": "United States", "temp": 25, "signal": 23, "battery_level": 8, "c02_level": 917, "timestamp" :1475600496 }"""), (1, """{"device_id": 1, "device_type": "sensor-igauge", "ip": "213.161.254.1", "cca3": "NOR", "cn": "Norway", "temp": 30, "signal": 18, "battery_level": 6, "c02_level": 1413, "timestamp" :1475600498 }"""), (2, """{"device_id": 2, "device_type": "sensor-ipad", "ip": "88.36.5.1", "cca3": "ITA", "cn": "Italy", "temp": 18, "signal": 25, "battery_level": 5, "c02_level": 1372, "timestamp" :1475600500 }"""), (3, """{"device_id": 3, "device_type": "sensor-inest", "ip": "66.39.173.154", "cca3": "USA", "cn": "United States", "temp": 47, "signal": 12, "battery_level": 1, "c02_level": 1447, "timestamp" :1475600502 }"""), (4, """{"device_id": 4, "device_type": "sensor-ipad", "ip": "203.82.41.9", "cca3": "PHL", "cn": "Philippines", "temp": 29, "signal": 11, "battery_level": 0, "c02_level": 983, "timestamp" :1475600504 }"""), (5, """{"device_id": 5, "device_type": "sensor-istick", "ip": "204.116.105.67", "cca3": "USA", "cn": "United States", "temp": 50, "signal": 16, "battery_level": 8, "c02_level": 1574, "timestamp" :1475600506 }"""), (6, """{"device_id": 6, "device_type": "sensor-ipad", "ip": "220.173.179.1", "cca3": "CHN", "cn": "China", "temp": 21, "signal": 18, "battery_level": 9, "c02_level": 1249, "timestamp" :1475600508 }"""), (7, """{"device_id": 7, "device_type": "sensor-ipad", "ip": "118.23.68.227", "cca3": "JPN", "cn": "Japan", "temp": 27, "signal": 15, "battery_level": 0, "c02_level": 1531, "timestamp" :1475600512 }"""), (8, """ {"device_id": 8, "device_type": "sensor-inest", "ip": "208.109.163.218", "cca3": "USA", "cn": "United States", "temp": 40, "signal": 16, "battery_level": 9, "c02_level": 1208, "timestamp" :1475600514 }"""), (9, """{"device_id": 9, "device_type": "sensor-ipad", "ip": "88.213.191.34", "cca3": "ITA", "cn": "Italy", "temp": 19, "signal": 11, "battery_level": 0, "c02_level": 1171, "timestamp" :1475600516 }""")) .toDF("id", "device").as[DeviceData] val frame = eventsDS.select(from_json($"device", jsonSchema) as "devices") frame.printSchema() frame.show(3,false) //找出DF中的所有Struct 包含的所有子列 frame.select($"devices.*").show(3) //找出DF中的Struct 指定的cn子列 frame.select($"devices.device_id",$"devices.cn").show(3) //DataFrame转Json字符串 val frame2 = frame.select(to_json( struct($"devices.device_id", $"devices.battery_level")).as("jsonstr") ) frame2.show(false) spark.close() } } |
| 练习2---- 解析如下json {"name":"njzb","roomInfo":{"area":2000,"employee":98,"roomNum":30},"students":[{"classes":"kb01","stuId":"1","stuName":"zhangsan","teacher":"xingxing"},{"classes":"kb02","stuId":"2","stuName":"lisi","teacher":"gree"},{"classes":"kb03","stuId":"3","stuName":"wangwu","teacher":"gree"},{"classes":"kb05","stuId":"4","stuName":"zhaoliu","teacher":"xingxing"}],"teachers":[{"name":"gree","skill":"bigdata&java","yearNum":7},{"name":"xingxing","skill":"bigdata&java","yearNum":3}]} Scala import nj.zb.kb23.etl.utils.JdbcUtils import org.apache.spark.sql.SparkSession import org.apache.spark.sql.types._ object JsonStu4 { def main(args: Array[String]): Unit = { val spark = SparkSession.builder().appName("retention").master("local[*]").getOrCreate() val sc = spark.sparkContext import spark.implicits._ import org.apache.spark.sql.functions._ val rdd = sc.textFile("in/school.txt") //rdd.collect().foreach(println) //"name", "roomInfo", "students", "teachers" val sc1DF = rdd.toDF("school") //sc1DF.printSchema() //sc1DF.show(false) val sc2DF = sc1DF.select( get_json_object($"school", "$.name").as("name"), get_json_object($"school", "$.roomInfo").as("roomInfo"), get_json_object($"school", "$.students").as("students"), get_json_object($"school", "$.teachers").as("teachers") ) //sc2DF.printSchema() //sc2DF.show(false) val sc3DF = sc2DF.select( $"name", get_json_object($"roomInfo", "$.area").as("area"), get_json_object($"roomInfo", "$.employee").as("employee"), get_json_object($"roomInfo", "$.roomNum").as("roomNum"), $"students", $"teachers" ) //sc3DF.printSchema() //sc3DF.show(false) val stuSchema = StructType( Array( StructField("classes", StringType), StructField("stuId", StringType), StructField("stuName", StringType), StructField("teacher", StringType) ) ) val teaSchema = StructType( Array( StructField("name", StringType), StructField("skill", StringType), StructField("yearNum", StringType) ) ) val sc4StuDF = sc3DF.select( $"name", $"area", $"employee", $"roomNum", from_json($"students", ArrayType(stuSchema)).as("students")) //sc4StuDF.printSchema() //sc4StuDF.show(false) val sc4TeaDF = sc3DF.select( $"name", $"area", $"employee", $"roomNum", from_json($"teachers", ArrayType(teaSchema)).as("teachers")) //sc4TeaDF.printSchema() //sc4TeaDF.show(false) val sc5StuDF = sc4StuDF .withColumn("students", explode($"students")) //sc5StuDF.printSchema() //sc5StuDF.show(false) val sc5TeaDF = sc4TeaDF .withColumn("teachers", explode($"teachers")) //sc5TeaDF.printSchema() //sc5TeaDF.show(false) val sc6StuDF = sc5StuDF .withColumn("classes", $"students.classes") .withColumn("stuId", $"students.stuId") .withColumn("stuName", $"students.stuName") .withColumn("teacher", $"students.teacher") .drop("students") //sc6StuDF.show(false) val sc6TeaDF = sc5TeaDF .withColumn("teacherName", $"teachers.name") .withColumn("skill", $"teachers.skill") .withColumn("yearNum", $"teachers.yearNum") .drop("name") .drop("area") .drop("employee") .drop("roomNum") .drop("teachers") //sc6TeaDF.show(false) val schoolDF = sc6StuDF .join(sc6TeaDF, $"teacher" === $"teacherName") .drop("teacherName") schoolDF.show() //使用工具类将DF导入到Mysql中 JdbcUtils.dataFrameToMysql(schoolDF,"school") } } |