Spark Catalog

#iceberg catalog

https://iceberg.apache.org/docs/latest/spark-configuration/

相关接口

复制代码
  /**
   * (Scala-specific)
   * Create a table from the given path based on a data source, a schema and a set of options.
   * Then, returns the corresponding DataFrame.
   *
   * @param tableName is either a qualified or unqualified name that designates a table.
   *                  If no database identifier is provided, it refers to a table in
   *                  the current database.
   * @since 2.0.0
   */
  @deprecated("use createTable instead.", "2.2.0")
  def createExternalTable(
      tableName: String,
      source: String,
      schema: StructType,
      options: Map[String, String]): DataFrame = {
    createTable(tableName, source, schema, options)
  }

  /**
   * (Scala-specific)
   * Create a table based on the dataset in a data source, a schema and a set of options.
   * Then, returns the corresponding DataFrame.
   *
   * @param tableName is either a qualified or unqualified name that designates a table.
   *                  If no database identifier is provided, it refers to a table in
   *                  the current database.
   * @since 2.2.0
   */
  def createTable(
      tableName: String,
      source: String,
      schema: StructType,
      options: Map[String, String]): DataFrame

hive metastore

The default implementation of the Hive metastore in Apache Spark uses Apache Derby for its database persistence. This is available with no configuration required but is limited to only one Spark session at any time for the purposes of metadata storage. This obviously makes it unsuitable for use in multi-user environments, such as when shared on a development team or used in Production.

相关推荐
蒙特卡洛的随机游走20 小时前
Spark核心数据(RDD、DataFrame 和 Dataset)
大数据·分布式·spark
蒙特卡洛的随机游走21 小时前
Spark的宽依赖与窄依赖
大数据·前端·spark
Lansonli1 天前
大数据Spark(六十九):Transformation转换算子intersection和subtract使用案例
大数据·分布式·spark
励志成为糕手1 天前
宽依赖的代价:Spark 与 MapReduce Shuffle 的数据重分布对比
大数据·spark·mapreduce·分布式计算·sortshuffle
weixin_525936332 天前
部分Spark SQL编程要点
大数据·python·sql·spark
智海观潮2 天前
学好Spark必须要掌握的Scala技术点
大数据·spark·scala
数智顾问3 天前
破解 Shuffle 阻塞:Spark RDD 宽窄依赖在实时特征工程中的实战与未来
大数据·分布式·spark
想ai抽3 天前
吃透大数据算法-算法地图(备用)
大数据·数据库·spark
一个java开发3 天前
spark热点key导致的数据倾斜复现和加盐处理
大数据·spark
IT研究室3 天前
大数据毕业设计选题推荐-基于大数据的商店购物趋势分析与可视化系统-大数据-Spark-Hadoop-Bigdata
大数据·spark·课程设计