Spark SQL 操作实战

一缕阳光lyz2024-05-22 12:04

Spark SQL 基础

Spark SQL 是 Apache Spark 处理结构化数据的模块。

配置spark环境

python 复制代码 
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q www-us.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz  
!tar xf spark-2.4.8-bin-hadoop2.7.tgz
!pip install -q findspark
python 复制代码 
import os
os.environ["JAVA_HOME"]="/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"]="/content/spark-2.4.8-bin-hadoop2.7"
python 复制代码 
import findspark
findspark.init()

初始化SparkSession

初始化

python 复制代码 
 SparkSession 用于创建数据帧，将数据帧注册为表，执行 SQL 查询，缓存表及读取 Parquet 文件。
python 复制代码 
from pyspark.sql import SparkSession
python 复制代码 
spark = SparkSession \
       .builder \
       .appName("Python Spark SQL basic example") \
       .config("spark.some.config.option", "some-value") \
       .getOrCreate()

创建数据帧

从 RDD 创建

python 复制代码 
from pyspark.sql.types import *
from pyspark.sql import *
from pyspark.sql.functions import *

推断 Schema

python 复制代码 
sc = spark.sparkContext
python 复制代码 
!wget https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/people.txt
python 复制代码 
lines = sc.textFile("people.txt")
python 复制代码 
parts = lines.map(lambda l: l.split(","))
python 复制代码 
people = parts.map(lambda p: Row(name=p[0],age=int(p[1])))
python 复制代码 
peopledf = spark.createDataFrame(people)

指定 Schema

python 复制代码 
people = parts.map(lambda p: Row(name=p[0], age=int(p[1].strip())))
python 复制代码 
schemaString = "name age"
python 复制代码 
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split()]
python 复制代码 
schema = StructType(fields)
python 复制代码 
spark.createDataFrame(people, schema).show()
+-------+---+
|   name|age|
+-------+---+
|Michael| 29|
|   Andy| 30|
| Justin| 19|
+-------+---+

从Spark数据源创建

JSON

python 复制代码 
!wget -q https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/people.json
!wget -q https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/employees.json
!wget -q https://github.com/awesome-AI-cheatsheets/tree/main/Spark/data/customers.json
python 复制代码 
df = spark.read.json("customers.json")
python 复制代码 
df.show()
+--------------------+---+---------+----------+--------------------+
|             address|age|firstName|  lastName|         phoneNumber|
+--------------------+---+---------+----------+--------------------+
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...|
+--------------------+---+---------+----------+--------------------+
python 复制代码 
df2 = spark.read.load("people.json", format="json")

Parquet文件

python 复制代码 
!wget -q https://github.com/apache/spark/raw/master/examples/src/main/resources/users.parquet
python 复制代码 
df3 = spark.read.load("users.parquet")

文本文件

python 复制代码 
df4 = spark.read.text("people.txt")

查阅数据信息

查阅spark Dataframe的信息

python 复制代码 
df.dtypes   #返回 df 的列名与数据类型
[('address', 'struct<city:string,state:string,street:string,zip:string>'),
 ('age', 'bigint'),
 ('firstName', 'string'),
 ('lastName', 'string'),
 ('phoneNumber', 'struct<number:string,type:string>')]
python 复制代码 
df.show()   #显示 df 的内容
+--------------------+---+---------+----------+--------------------+
|             address|age|firstName|  lastName|         phoneNumber|
+--------------------+---+---------+----------+--------------------+
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...|
+--------------------+---+---------+----------+--------------------+
python 复制代码 
df.head(3)   #返回前 n 行数据
[Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile')),
 Row(address=Row(city='Brighton', state='MI', street='4 B Blue Ridge Blvd', zip='48116'), age=25, firstName='Josephine', lastName='Darakjy', phoneNumber=Row(number='210-319-103', type='mobile')),
 Row(address=Row(city='Bridgeport', state='NJ', street='8 W Cerritos Ave #54', zip='08014'), age=32, firstName='Boris', lastName='Chemel', phoneNumber=Row(number='210-322-007', type='mobile'))]
python 复制代码 
df.first()   #返回第 1 行数据
Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile'))
python 复制代码 
df.take(3)   #返回前 n 行数据
[Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile')),
 Row(address=Row(city='Brighton', state='MI', street='4 B Blue Ridge Blvd', zip='48116'), age=25, firstName='Josephine', lastName='Darakjy', phoneNumber=Row(number='210-319-103', type='mobile')),
 Row(address=Row(city='Bridgeport', state='NJ', street='8 W Cerritos Ave #54', zip='08014'), age=32, firstName='Boris', lastName='Chemel', phoneNumber=Row(number='210-322-007', type='mobile'))]
python 复制代码 
df.schema   #返回 df 的 Schema
StructType(List(StructField(address,StructType(List(StructField(city,StringType,true),StructField(state,StringType,true),StructField(street,StringType,true),StructField(zip,StringType,true))),true),StructField(age,LongType,true),StructField(firstName,StringType,true),StructField(lastName,StringType,true),StructField(phoneNumber,StructType(List(StructField(number,StringType,true),StructField(type,StringType,true))),true)))
python 复制代码 
df.describe().show()   #汇总统计数据
+-------+-----------------+---------+----------+
|summary|              age|firstName|  lastName|
+-------+-----------------+---------+----------+
|  count|                4|        4|         4|
|   mean|             28.5|     null|      null|
| stddev|6.027713773341708|     null|      null|
|    min|               22|    Boris|Butterburg|
|    max|               35|Josephine|     Smith|
+-------+-----------------+---------+----------+
python 复制代码 
df.columns   #返回 df 的列名
['address', 'age', 'firstName', 'lastName', 'phoneNumber']
python 复制代码 
df.count()   #返回 df 的行数
4
python 复制代码 
df.distinct().count()   #返回 df 中不重复的行数
4
python 复制代码 
df.printSchema()   #返回 df的 Schema
root
 |-- address: struct (nullable = true)
 |    |-- city: string (nullable = true)
 |    |-- state: string (nullable = true)
 |    |-- street: string (nullable = true)
 |    |-- zip: string (nullable = true)
 |-- age: long (nullable = true)
 |-- firstName: string (nullable = true)
 |-- lastName: string (nullable = true)
 |-- phoneNumber: struct (nullable = true)
 |    |-- number: string (nullable = true)
 |    |-- type: string (nullable = true)
python 复制代码 
df.explain()   #返回逻辑与实体方案
== Physical Plan ==
*(1) FileScan json [address#452,age#453L,firstName#454,lastName#455,phoneNumber#456] Batched: false, Format: JSON, Location: InMemoryFileIndex[file:/content/customers.json], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<address:struct<city:string,state:string,street:string,zip:string>,age:bigint,firstName:str...

重复值

dropDuplicates函数

python 复制代码 
df = df.dropDuplicates()

查询

自定义函数F

python 复制代码 
from pyspark.sql import functions as F

Select

python 复制代码 
df.select("firstName").show()   #显示 firstName 列的所有条目
+---------+
|firstName|
+---------+
|Josephine|
|     Jane|
|    Boris|
|  Johnson|
+---------+
python 复制代码 
df.select("firstName","lastName").show()
+---------+----------+
|firstName|  lastName|
+---------+----------+
|Josephine|   Darakjy|
|     Jane|Butterburg|
|    Boris|    Chemel|
|  Johnson|     Smith|
+---------+----------+
python 复制代码 
df.show()
+--------------------+---+---------+----------+--------------------+
|             address|age|firstName|  lastName|         phoneNumber|
+--------------------+---+---------+----------+--------------------+
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...|
+--------------------+---+---------+----------+--------------------+
python 复制代码 
df.select("firstName", "age", df.phoneNumber \
            .alias("contactInfo")) \
            .select("contactInfo", "firstName", "age") \
            .show()   #显示 firstName、age 的所有条目和类型
+--------------------+---------+---+
|         contactInfo|firstName|age|
+--------------------+---------+---+
|[210-319-103, mob...|Josephine| 25|
|[210-332-578, mob...|     Jane| 22|
|[210-322-007, mob...|    Boris| 32|
|[210-303-029, mob...|  Johnson| 35|
+--------------------+---------+---+
python 复制代码 
df.select(df["firstName"],df["age"]+ 1).show()   # 显示 firstName 和 age 列的所有 记录，并对 age 记录添加1
+---------+---------+
|firstName|(age + 1)|
+---------+---------+
|Josephine|       26|
|     Jane|       23|
|    Boris|       33|
|  Johnson|       36|
+---------+---------+
python 复制代码 
df.select(df['age'] > 24).show()   #显示所有小于24岁的记录
+----------+
|(age > 24)|
+----------+
|      true|
|     false|
|      true|
|      true|
+----------+

When

python 复制代码 
df.select("firstName", F.when(df.age > 30, 1) \
            .otherwise(0)) \
            .show()   #显示 firstName，且大于30岁显示 1，小于30岁显示0
+---------+--------------------------------------+
|firstName|CASE WHEN (age > 30) THEN 1 ELSE 0 END|
+---------+--------------------------------------+
|Josephine|                                     0|
|     Jane|                                     0|
|    Boris|                                     1|
|  Johnson|                                     1|
+---------+--------------------------------------+
python 复制代码 
df[df.firstName.isin("Jane","Boris")].collect()   # 显示符合指定条件的 firstName 列 的记录
[Row(address=Row(city='New Orleans', state='LA', street='6649 N Blue Gum St', zip='70116'), age=22, firstName='Jane', lastName='Butterburg', phoneNumber=Row(number='210-332-578', type='mobile')),
 Row(address=Row(city='Bridgeport', state='NJ', street='8 W Cerritos Ave #54', zip='08014'), age=32, firstName='Boris', lastName='Chemel', phoneNumber=Row(number='210-322-007', type='mobile'))]

Like

python 复制代码 
df.select("firstName", df.lastName.like("Smith")) \
  .show()   # 显示 lastName 列中包含 Smith 的 firstName 列的记录
+---------+-------------------+
|firstName|lastName LIKE Smith|
+---------+-------------------+
|Josephine|              false|
|     Jane|              false|
|    Boris|              false|
|  Johnson|               true|
+---------+-------------------+

Startswith - Endswith

python 复制代码 
df.select("firstName", df.lastName \
  .startswith("Sm")) \
  .show()   # 显示 lastName 列中以 Sm 开头的 firstName 列的记录
+---------+------------------------+
|firstName|startswith(lastName, Sm)|
+---------+------------------------+
|Josephine|                   false|
|     Jane|                   false|
|    Boris|                   false|
|  Johnson|                    true|
+---------+------------------------+
python 复制代码 
df.select(df.lastName \
  .endswith("th")) \
  .show()   # 显示以 th 结尾的 lastName
+----------------------+
|endswith(lastName, th)|
+----------------------+
|                 false|
|                 false|
|                 false|
|                  true|
+----------------------+

Substring

python 复制代码 
df.select(df.firstName.substr(1, 3) \
 .alias("name")) \
 .collect()   #返回 firstName 的子字符串
[Row(name='Jos'), Row(name='Jan'), Row(name='Bor'), Row(name='Joh')]

Between

python 复制代码 
df.select(df.age.between(22, 24)) \
        .show()   #显示介于22岁至24岁之间的 age 列的记录
+-----------------------------+
|((age >= 22) AND (age <= 24))|
+-----------------------------+
|                        false|
|                         true|
|                        false|
|                        false|
+-----------------------------+

添加、修改、删除列

添加列

python 复制代码 
df.select(df.phoneNumber.number).show()
+------------------+
|phoneNumber.number|
+------------------+
|       210-319-103|
|       210-332-578|
|       210-322-007|
|       210-303-029|
+------------------+
python 复制代码 
df = df.withColumn('city',df.address.city) \
             .withColumn('postalCode',df.address.zip) \
             .withColumn('state',df.address.state) \
             .withColumn('streetAddress',df.address.street) \
             .withColumn('telePhoneNumber', df.phoneNumber.number) \
             .withColumn('telePhoneType', df.phoneNumber.type)
python 复制代码 
df.show()
+--------------------+---+---------+----------+--------------------+-----------+----------+-----+--------------------+---------------+-------------+
|             address|age|firstName|  lastName|         phoneNumber|       city|postalCode|state|       streetAddress|telePhoneNumber|telePhoneType|
+--------------------+---+---------+----------+--------------------+-----------+----------+-----+--------------------+---------------+-------------+
|[Brighton, MI, 4 ...| 25|Josephine|   Darakjy|[210-319-103, mob...|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|    210-319-103|       mobile|
|[New Orleans, LA,...| 22|     Jane|Butterburg|[210-332-578, mob...|New Orleans|     70116|   LA|  6649 N Blue Gum St|    210-332-578|       mobile|
|[Bridgeport, NJ, ...| 32|    Boris|    Chemel|[210-322-007, mob...| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|    210-322-007|       mobile|
|[Bridgeport, NJ, ...| 35|  Johnson|     Smith|[210-303-029, mob...| Bridgeport|     08112|   NJ|     5 W Blue Ave St|    210-303-029|       mobile|
+--------------------+---+---------+----------+--------------------+-----------+----------+-----+--------------------+---------------+-------------+

修改列

python 复制代码 
df = df.withColumnRenamed('telePhoneNumber', 'phoneNumber')

删除列

python 复制代码 
df = df.drop("address", "phoneNumber")
# 等价于
# df = df.drop(df.address).drop(df.phoneNumber)

分组

groupBy操作

python 复制代码 
df.groupBy("age")\
      .count() \
      .show()   #按 age 列分组，统计每组人数
+---+-----+
|age|count|
+---+-----+
| 22|    1|
| 32|    1|
| 25|    1|
| 35|    1|
+---+-----+

筛选

filter筛选

python 复制代码 
df.filter(df["age"]>24).show()   #按 age 列筛选，保留年龄大于24岁的
+---+---------+--------+----------+----------+-----+--------------------+-------------+
|age|firstName|lastName|      city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+--------+----------+----------+-----+--------------------+-------------+
| 25|Josephine| Darakjy|  Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 32|    Boris|  Chemel|Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|   Smith|Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+--------+----------+----------+-----+--------------------+-------------+

排序

sort与orderBy操作

python 复制代码 
peopledf.sort(peopledf.age.desc()).collect()
[Row(age=30, name='Andy'),
 Row(age=29, name='Michael'),
 Row(age=19, name='Justin')]
python 复制代码 
df.sort("age", ascending=False).collect()
[Row(age=35, firstName='Johnson', lastName='Smith', city='Bridgeport', postalCode='08112', state='NJ', streetAddress='5 W Blue Ave St', telePhoneType='mobile'),
 Row(age=32, firstName='Boris', lastName='Chemel', city='Bridgeport', postalCode='08014', state='NJ', streetAddress='8 W Cerritos Ave #54', telePhoneType='mobile'),
 Row(age=25, firstName='Josephine', lastName='Darakjy', city='Brighton', postalCode='48116', state='MI', streetAddress='4 B Blue Ridge Blvd', telePhoneType='mobile'),
 Row(age=22, firstName='Jane', lastName='Butterburg', city='New Orleans', postalCode='70116', state='LA', streetAddress='6649 N Blue Gum St', telePhoneType='mobile')]
python 复制代码 
df.orderBy(["age","city"],ascending=[0,1]).collect()
[Row(age=35, firstName='Johnson', lastName='Smith', city='Bridgeport', postalCode='08112', state='NJ', streetAddress='5 W Blue Ave St', telePhoneType='mobile'),
 Row(age=32, firstName='Boris', lastName='Chemel', city='Bridgeport', postalCode='08014', state='NJ', streetAddress='8 W Cerritos Ave #54', telePhoneType='mobile'),
 Row(age=25, firstName='Josephine', lastName='Darakjy', city='Brighton', postalCode='48116', state='MI', streetAddress='4 B Blue Ridge Blvd', telePhoneType='mobile'),
 Row(age=22, firstName='Jane', lastName='Butterburg', city='New Orleans', postalCode='70116', state='LA', streetAddress='6649 N Blue Gum St', telePhoneType='mobile')]

替换缺失值

replace操作

python 复制代码 
df.na.fill(50).show()   #用一个值替换空值
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
python 复制代码 
df.na.drop().show()   #去除 df 中为空值的行
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
python 复制代码 
df.na.replace(10, 20).show()
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+

重分区

repartition重分区

python 复制代码 
df.repartition(10)\
      .rdd \
      .getNumPartitions()   #将 df 拆分为10个分区
10
python 复制代码 
df.coalesce(1).rdd.getNumPartitions()   #将 df 合并为1个分区
1

运行 SQL 查询

将数据帧注册为视图

python 复制代码 
peopledf.createGlobalTempView("people")
python 复制代码 
df.createTempView("customer")
python 复制代码 
df.createOrReplaceTempView("customer")

查询视图

python 复制代码 
df5 = spark.sql("SELECT * FROM customer").show()
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
|age|firstName|  lastName|       city|postalCode|state|       streetAddress|telePhoneType|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
| 25|Josephine|   Darakjy|   Brighton|     48116|   MI| 4 B Blue Ridge Blvd|       mobile|
| 22|     Jane|Butterburg|New Orleans|     70116|   LA|  6649 N Blue Gum St|       mobile|
| 32|    Boris|    Chemel| Bridgeport|     08014|   NJ|8 W Cerritos Ave #54|       mobile|
| 35|  Johnson|     Smith| Bridgeport|     08112|   NJ|     5 W Blue Ave St|       mobile|
+---+---------+----------+-----------+----------+-----+--------------------+-------------+
python 复制代码 
peopledf2 = spark.sql("SELECT * FROM global_temp.people").show()
+---+-------+
|age|   name|
+---+-------+
| 29|Michael|
| 30|   Andy|
| 19| Justin|
+---+-------+

输出

数据结构

python 复制代码 
rdd1 = df.rdd   #将 df 转换为 RDD
python 复制代码 
df.toJSON().first()   #将 df 转换为 RDD 字符串
'{"age":25,"firstName":"Josephine","lastName":"Darakjy","city":"Brighton","postalCode":"48116","state":"MI","streetAddress":"4 B Blue Ridge Blvd","telePhoneType":"mobile"}'
python 复制代码 
df.toPandas()   #将 df 的内容转为 Pandas 的数据帧

| | age | firstName | lastName | city | postalCode | state | streetAddress | telePhoneType |
| 0 | 25 | Josephine | Darakjy | Brighton | 48116 | MI | 4 B Blue Ridge Blvd | mobile |
| 1 | 22 | Jane | Butterburg | New Orleans | 70116 | LA | 6649 N Blue Gum St | mobile |
| 2 | 32 | Boris | Chemel | Bridgeport | 08014 | NJ | 8 W Cerritos Ave #54 | mobile |

3 35 Johnson Smith Bridgeport 08112 NJ 5 W Blue Ave St mobile

保存至文件

python 复制代码 
df.select("firstName", "city")\
      .write \
      .save("nameAndCity.parquet")
python 复制代码 
df.select("firstName", "age") \
      .write \
      .save("namesAndAges.json",format="json")

终止SparkSession

终止spark session

python 复制代码 
spark.stop()
相关推荐
成富4 分钟前
文本转SQL（Text-to-SQL），场景介绍与 Spring AI 实现
数据库·人工智能·sql·spring·oracle
songqq275 分钟前
SQL题：使用hive查询各类型专利top 10申请人，以及对应的专利申请数
数据库·sql
时差9533 小时前
【面试题】Hive 查询：如何查找用户连续三天登录的记录
大数据·数据库·hive·sql·面试·database
Mephisto.java4 小时前
【大数据学习 | kafka高级部分】kafka的优化参数整理
大数据·sql·oracle·kafka·json·database
山海青风4 小时前
第七篇: BigQuery中的复杂SQL查询
sql·googlecloud
lzhlizihang6 小时前
【Hive sql 面试题】求出各类型专利top 10申请人，以及对应的专利申请数（难）
大数据·hive·sql·面试题
威哥爱编程7 小时前
SQL Server 数据太多如何优化
数据库·sql·sqlserver
Mephisto.java8 小时前
【大数据学习 | kafka高级部分】kafka的kraft集群
大数据·sql·oracle·kafka·json·hbase
Mephisto.java8 小时前
【大数据学习 | kafka高级部分】kafka的文件存储原理
大数据·sql·oracle·kafka·json
W Y9 小时前
【架构-37】Spark和Flink
架构·flink·spark
热门推荐
01【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总02(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04【AIGC】重塑未来的科技巨轮05全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）06【经验分享】Ubuntu22.04安装微信(linux官方版)07基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测08Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO09RAG 实践- Ollama+RagFlow 部署本地知识库10【TC3xx芯片】TC3xx芯片电源管理系统PMS详解