Spark Catalog

#iceberg catalog

https://iceberg.apache.org/docs/latest/spark-configuration/

相关接口

复制代码
  /**
   * (Scala-specific)
   * Create a table from the given path based on a data source, a schema and a set of options.
   * Then, returns the corresponding DataFrame.
   *
   * @param tableName is either a qualified or unqualified name that designates a table.
   *                  If no database identifier is provided, it refers to a table in
   *                  the current database.
   * @since 2.0.0
   */
  @deprecated("use createTable instead.", "2.2.0")
  def createExternalTable(
      tableName: String,
      source: String,
      schema: StructType,
      options: Map[String, String]): DataFrame = {
    createTable(tableName, source, schema, options)
  }

  /**
   * (Scala-specific)
   * Create a table based on the dataset in a data source, a schema and a set of options.
   * Then, returns the corresponding DataFrame.
   *
   * @param tableName is either a qualified or unqualified name that designates a table.
   *                  If no database identifier is provided, it refers to a table in
   *                  the current database.
   * @since 2.2.0
   */
  def createTable(
      tableName: String,
      source: String,
      schema: StructType,
      options: Map[String, String]): DataFrame

hive metastore

The default implementation of the Hive metastore in Apache Spark uses Apache Derby for its database persistence. This is available with no configuration required but is limited to only one Spark session at any time for the purposes of metadata storage. This obviously makes it unsuitable for use in multi-user environments, such as when shared on a development team or used in Production.

相关推荐
武子康33 分钟前
大数据-98 Spark 从 DStream 到 Structured Streaming:Spark 实时计算的演进
大数据·后端·spark
武子康8 小时前
大数据-100 Spark DStream 转换操作全面总结:map、reduceByKey 到 transform 的实战案例
大数据·后端·spark
武子康1 天前
大数据-99 Spark Streaming 数据源全面总结:原理、应用 文件流、Socket、RDD队列流
大数据·后端·spark
计算机毕业设计木哥2 天前
计算机毕设选题推荐:基于Java+SpringBoot物品租赁管理系统【源码+文档+调试】
java·vue.js·spring boot·mysql·spark·毕业设计·课程设计
IT毕设梦工厂3 天前
大数据毕业设计选题推荐-基于大数据的客户购物订单数据分析与可视化系统-Hadoop-Spark-数据可视化-BigData
大数据·hadoop·数据分析·spark·毕业设计·源码·bigdata
大数据CLUB3 天前
基于spark的澳洲光伏发电站选址预测
大数据·hadoop·分布式·数据分析·spark·数据开发
计算机编程小央姐3 天前
跟上大数据时代步伐:食物营养数据可视化分析系统技术前沿解析
大数据·hadoop·信息可视化·spark·django·课程设计·食物
孟意昶3 天前
Spark专题-第一部分:Spark 核心概述(2)-Spark 应用核心组件剖析
大数据·spark·big data
智海观潮3 天前
Spark SQL | 目前Spark社区最活跃的组件之一
大数据·spark
盛源_013 天前
hadoop的api操作对象存储
hdfs·spark