大数据开发学习Day29

一、Linux

查看磁盘整体占用
只查看 inode 使用情况
查找 7 天前修改过的文件

bash 复制代码

df -h
df -i
find . -mtime +7

df -h 查看磁盘分区容量、已用、剩余，集群磁盘告警常用
df -i 查看 inode 占用，排查inode 耗尽导致无法创建文件
find -mtime +7 查找 7 天前修改文件，日志归档、清理历史文件刚需

二、SQL

1440. 布尔表达式计算

sql 复制代码

SELECT
    left_operand,
    operator,
    right_operand,
    CASE
        WHEN operator = '>' AND left_operand > right_operand THEN 'true'
        WHEN operator = '<' AND left_operand < right_operand THEN 'true'
        WHEN operator = '=' AND left_operand = right_operand THEN 'true'
    ELSE 'false'
    END AS value
FROM Expressions;

CASE 多条件组合判断

字段逻辑运算 + 条件分支输出标签

数据校验、规则判定类 SQL 通用模板

1484. 按日期分组销售产品

sql 复制代码

SELECT
    sell_date,
    COUNT(DISTINCT product) AS num_sold,
    GROUP_CONCAT(DISTINCT product ORDER BY product SEPARATOR ',') AS products
FROM Activities
GROUP BY sell_date
ORDER BY sell_date;

GROUP_CONCAT 行转字符串拼接

支持去重、排序、指定分隔符

数仓每日商品汇总、标签拼接高频用法

1527. 患某种疾病的患者

sql 复制代码

SELECT *
FROM Patients
WHERE conditions LIKE 'DIAB1%' 
   OR conditions LIKE '% DIAB1%';

LIKE 模糊匹配

匹配开头或中间带空格的关键字

日志、标签字段模糊检索标准写法

三、Pyspark

python 复制代码

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, countDistinct, collect_list, concat_ws, lower

spark = SparkSession.builder.master("local[*]").appName("Day29").getOrCreate()

# 1. 布尔表达式判断
expr = spark.createDataFrame([
    (5,">",3),(2,"<",4)
], ["left_operand","operator","right_operand"])

expr.withColumn("value",
    expr.casewhen((col("operator")==">") & (col("left_operand")>col("right_operand")), "true")
         .when((col("operator")=="<") & (col("left_operand")<col("right_operand")), "true")
         .otherwise("false")
).show()

# 2. 日期分组+产品拼接
act = spark.createDataFrame([
    ("2025-05-01","Banana"),
    ("2025-05-01","Apple")
], ["sell_date","product"])

act.groupBy("sell_date")
   .agg(
       countDistinct("product").alias("num_sold"),
       concat_ws(",", collect_list("product")).alias("products")
   ).show()

spark.stop()

Spark casewhen 对应 SQL CASE
collect_list + concat_ws 实现 GROUP_CONCAT 效果

分组聚合 + 字符串拼接，报表汇总常用

四、算法

27. 移除元素

python 复制代码

def removeElement(nums, val):
    slow = 0
    for fast in range(len(nums)):
        if nums[fast] != val:
            nums[slow] = nums[fast]
            slow += 1
    return slow

双指针原地移除指定值

不开新数组、空间 O (1)

数组基础双指针必背模板