Spark读取MySQL数据库表

官方地址:JDBC To Other Databases - Spark 4.0.0 Documentation

官方案例:

复制代码
// Note: JDBC loading and saving can be achieved via either the load/save or jdbc methods
// Loading data from a JDBC source
val jdbcDF = spark.read
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver")
  .option("dbtable", "schema.tablename")
  .option("user", "username")
  .option("password", "password")
  .load()

val connectionProperties = new Properties()
connectionProperties.put("user", "username")
connectionProperties.put("password", "password")
val jdbcDF2 = spark.read
  .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties)
// Specifying the custom data types of the read schema
connectionProperties.put("customSchema", "id DECIMAL(38, 0), name STRING")
val jdbcDF3 = spark.read
  .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties)

// Saving data to a JDBC source
jdbcDF.write
  .format("jdbc")
  .option("url", "jdbc:postgresql:dbserver")
  .option("dbtable", "schema.tablename")
  .option("user", "username")
  .option("password", "password")
  .save()

jdbcDF2.write
  .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties)

// Specifying create table column data types on write
jdbcDF.write
  .option("createTableColumnTypes", "name CHAR(64), comments VARCHAR(1024)")
  .jdbc("jdbc:postgresql:dbserver", "schema.tablename", connectionProperties)

验证:

复制代码
<dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.13.16</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.13</artifactId>
      <version>4.0.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.13</artifactId>
      <version>4.0.0</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
      <version>8.0.15</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.13</artifactId>
      <version>4.0.0</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-10_2.13</artifactId>
      <version>4.0.0</version>
    </dependency>
  </dependencies>

import org.apache.spark.SparkConf

object SparkCase04 {


  import org.apache.spark.sql.SparkSession


  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setMaster("local[1]").setAppName("WC")

    val spark = SparkSession.builder().config(conf).getOrCreate()


   var df = spark.read.format("jdbc")
    .option("url", "jdbc:mysql://node11:3306/wjobs?useSSL=false")
    .option("dbtable", "user")
    .option("user", "root")
    .option("password", "root123")
    .load()

    df.show()
  }


}
相关推荐
Decay__4 小时前
PostgreSQL 数据库全栈运维实战系列 — 合集索引(24 篇)
运维·数据库·sql·mysql·架构
田超凡5 小时前
深入理解MySQL_经典问题总结2
mysql·java-ee
FungLeo8 小时前
FengCMS · PHP+MySQL 升级实录 · 危险默认值之殇:root/123456、认证码 8888 写进配置加载器
android·mysql·php·安全加固·fengcms
琳少_9 小时前
java程序开发笔记
java·开发语言·数据库·笔记
ZhengEnCi9 小时前
覆盖索引怎么设计:从回表代价到 MySQL 与 PostgreSQL 的落地写法
sql·mysql·postgresql
FungLeo11 小时前
FengCMS · PHP+MySQL升级实录 ·并发一致性连环坑:模板缓存、排队保存、备份恢复的竞态
mysql·缓存·php·工程实践·fengcms
Pocker_Spades_A12 小时前
工业物联网时序数据库选型:一份能直接抄走的技术规格书
数据库·物联网·时序数据库
苜蓿muxu12 小时前
缓存三大问题:穿透、击穿、雪崩
数据库·redis
FungLeo13 小时前
FengCMS · PHP+MySQL 升级实录·模型导入从“加密 SQL 码“到.fengmodel.json
sql·mysql·php·工程实践·fengcms
正在走向自律13 小时前
MySQL兼容性深度实践:建表、增删改查与迁移避坑全记录
数据库·sql·mysql·信创·数据库迁移