Spark Delta Lake

rm -r dp-203 -f

git clone https://github.com/MicrosoftLearning/dp-203-azure-data-engineer dp-203

cd dp-203/Allfiles/labs/07

./setup.ps1

python 复制代码
%%pyspark
df = spark.read.load('abfss://files@datalakexxxxxxx.dfs.core.windows.net/products/products.csv', format='csv'
## If header exists uncomment line below
##, header=True
)
display(df.limit(10))
python 复制代码
%%pyspark
df = spark.read.load('abfss://files@datalakexxxxxxx.dfs.core.windows.net/products/products.csv', format='csv'
## If header exists uncomment line below
, header=True
)
display(df.limit(10))
python 复制代码
 delta_table_path = "/delta/products-delta"
 df.write.format("delta").save(delta_table_path)
  1. On the files tab, use the ↑ icon in the toolbar to return to the root of the files container, and note that a new folder named delta has been created. Open this folder and the products-delta table it contains, where you should see the parquet format file(s) containing the data.
python 复制代码
from delta.tables import *
from pyspark.sql.functions import *

 # Create a deltaTable object
deltaTable = DeltaTable.forPath(spark, delta_table_path)

 # Update the table (reduce price of product 771 by 10%)
deltaTable.update(
     condition = "ProductID == 771",
     set = { "ListPrice": "ListPrice * 0.9" })

 # View the updated data as a dataframe
deltaTable.toDF().show(10)
python 复制代码
 new_df = spark.read.format("delta").load(delta_table_path)
 new_df.show(10)
python 复制代码
 new_df = spark.read.format("delta").option("versionAsOf", 0).load(delta_table_path)
 new_df.show(10)
python 复制代码
deltaTable.history(10).show(20, False, True)
python 复制代码
 spark.sql("CREATE DATABASE AdventureWorks")
 spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION '{0}'".format(delta_table_path))
 spark.sql("DESCRIBE EXTENDED AdventureWorks.ProductsExternal").show(truncate=False)

This code creates a new database named AdventureWorks and then creates an external tabled named ProductsExternal in that database based on the path to the parquet files you defined previously. It then displays a description of the table's properties. Note that the Location property is the path you specified.

sql 复制代码
%%sql

 USE AdventureWorks;

 SELECT * FROM ProductsExternal;
python 复制代码
 df.write.format("delta").saveAsTable("AdventureWorks.ProductsManaged")
 spark.sql("DESCRIBE EXTENDED AdventureWorks.ProductsManaged").show(truncate=False)

This code creates a managed tabled named ProductsManaged based on the DataFrame you originally loaded from the products.csv file (before you updated the price of product 771). You do not specify a path for the parquet files used by the table - this is managed for you in the Hive metastore, and shown in the Location property in the table description (in the files/synapse/workspaces/synapsexxxxxxx/warehouse path).

sql 复制代码
%%sql

 USE AdventureWorks;

 SELECT * FROM ProductsManaged;
sql 复制代码
%%sql

 USE AdventureWorks;

 SHOW TABLES;
sql 复制代码
%%sql

 USE AdventureWorks;

 DROP TABLE IF EXISTS ProductsExternal;
 DROP TABLE IF EXISTS ProductsManaged;
  1. Return to the files tab and view the files/delta/products-delta folder. Note that the data files still exist in this location. Dropping the external table has removed the table from the metastore, but left the data files intact.
  2. View the files/synapse/workspaces/synapsexxxxxxx/warehouse folder, and note that there is no folder for the ProductsManaged table data. Dropping a managed table removes the table from the metastore and also deletes the table's data files.
sql 复制代码
%%sql

 USE AdventureWorks;

 CREATE TABLE Products
 USING DELTA
 LOCATION '/delta/products-delta';
sql 复制代码
%%sql

 USE AdventureWorks;

 SELECT * FROM Products;
python 复制代码
 from notebookutils import mssparkutils
 from pyspark.sql.types import *
 from pyspark.sql.functions import *

 # Create a folder
 inputPath = '/data/'
 mssparkutils.fs.mkdirs(inputPath)

 # Create a stream that reads data from the folder, using a JSON schema
 jsonSchema = StructType([
 StructField("device", StringType(), False),
 StructField("status", StringType(), False)
 ])
 iotstream = spark.readStream.schema(jsonSchema).option("maxFilesPerTrigger", 1).json(inputPath)

 # Write some event data to the folder
 device_data = '''{"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"ok"}
 {"device":"Dev2","status":"error"}
 {"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"error"}
 {"device":"Dev2","status":"ok"}
 {"device":"Dev2","status":"error"}
 {"device":"Dev1","status":"ok"}'''
 mssparkutils.fs.put(inputPath + "data.txt", device_data, True)
 print("Source stream created...")

Ensure the message Source stream created... is printed. The code you just ran has created a streaming data source based on a folder to which some data has been saved, representing readings from hypothetical IoT devices.

python 复制代码
 # Write the stream to a delta table
 delta_stream_table_path = '/delta/iotdevicedata'
 checkpointpath = '/delta/checkpoint'
 deltastream = iotstream.writeStream.format("delta").option("checkpointLocation", checkpointpath).start(delta_stream_table_path)
 print("Streaming to delta sink...")
python 复制代码
 # Read the data in delta format into a dataframe
 df = spark.read.format("delta").load(delta_stream_table_path)
 display(df)
python 复制代码
 # create a catalog table based on the streaming sink
 spark.sql("CREATE TABLE IotDeviceData USING DELTA LOCATION '{0}'".format(delta_stream_table_path))
sql 复制代码
 %%sql

 SELECT * FROM IotDeviceData;
python 复制代码
 # Add more data to the source stream
 more_data = '''{"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"ok"}
 {"device":"Dev1","status":"error"}
 {"device":"Dev2","status":"error"}
 {"device":"Dev1","status":"ok"}'''

 mssparkutils.fs.put(inputPath + "more-data.txt", more_data, True)
sql 复制代码
%%sql

 SELECT * FROM IotDeviceData;
python 复制代码
 deltastream.stop()
sql 复制代码
 -- This is auto-generated code
 SELECT
     TOP 100 *
 FROM
     OPENROWSET(
         BULK 'https://datalakexxxxxxx.dfs.core.windows.net/files/delta/products-delta/',
         FORMAT = 'DELTA'
     ) AS [result]
sql 复制代码
 USE AdventureWorks;

 SELECT * FROM Products;

Run the code and observe that you can also use the serverless SQL pool to query Delta Lake data in catalog tables that are defined the Spark metastore.

相关推荐
大大大大晴天8 小时前
每天认识一个组件:VictoriaMetrics
大数据
龙腾AI白云9 小时前
具身智能的空间导航与未知环境探索
大数据·数据库·人工智能·机器学习·知识图谱
演艺大赏10 小时前
三个坑连环套:一次 NameNode 起不来,从 JournalNode 未格式化查到配置里多了个挂载点
大数据·hadoop
Aloudata11 小时前
服饰零售数据治理如何统一款色码与库存口径,支撑门店售罄率和补货分析?
大数据·数据分析·零售
躺柒11 小时前
读数据分析思维通识课07数据思维图
大数据·数据挖掘·数据分析·大数据分析·数据思维图
西木Qi12 小时前
Kafka消息中间件单机版安装配置
分布式·kafka
OriginCoding13 小时前
一个 JDK 跑起六个分布式挑战:Kotlin 版 Gossip Glomers(一)
分布式·kotlin
回眸&啤酒鸭13 小时前
【回眸】Hy4 模型效果展示与能力评测大纲
大数据·人工智能·机器学习
伞伞悦读14 小时前
【第86期】分类指标与错误成本对照的实验与排查
大数据·人工智能·机器学习
金融小师妹19 小时前
多因子关联模型:黄金与铜价同步回落背后的金融变量及PPI传导分析
大数据·人工智能·金融