Machine Learning ---- Gradient Descent

目录

[一、The concept of gradient:](#一、The concept of gradient:)

[① In a univariate function:](#① In a univariate function:)

[②In multivariate functions:](#②In multivariate functions:)

[二、Introduction of gradient descent cases:](#二、Introduction of gradient descent cases:)

[三、Gradient descent formula and its simple understanding:](#三、Gradient descent formula and its simple understanding:)

[四、Formula operation precautions:](#四、Formula operation precautions:)


一、The concept of gradient:

① In a univariate function:

gradient is actually the differentiation of the function, representing the slope of the tangent of the function at a given point

②In multivariate functions:

a gradient is a vector with a direction, and the direction of the gradient indicates the direction in which the function rises the fastest at a given point

二、Introduction of gradient descent cases:

Do you remember the golf course inside the cat and mouse? It looks like this in the animation:

Let's take a look at these two pictures. You can easily see the distant hill, right? We can take it as the most typical example, and the golf course can also be abstracted into a coordinate map:

So in this coordinate, we will correspond the following (x, y) to (w, b) respectively. Then, when J (w, b) is at its maximum, which is the peak in the red area of the graph, we start the gradient descent process.

Firstly, we rotate one circle from the highest point to find the direction with the highest slope. At this point, we can take a small step down. The reason for choosing this direction is actually because it is the steepest direction. If we walk down the same step length, the height of descent will naturally be the highest, and we can also walk faster to the lowest point (local minimum point). At the same time, after each step, we look around and choose. Finally, we can determine this path:Finally reaching the local minimum point A, is this the only minimum point? Of course not:

It is also possible to reach point B, which is also a local minimum point. At this point, we have introduced the implementation process of gradient descent, and we will further understand its meaning through mathematical formulas.

三、Gradient descent formula and its simple understanding:

We first provide the formula for gradient descent:

In the formula, corresponds to what we call the learning rate, and the equal sign is the same as the assignment symbol in computer program code. J (w, b) can be found in the regression equation blog in the previous section. As for the determination of the learning rate, we will share it with you next time. Here, we will first understand the meaning of the formula:

Firstly, let's simplify the formula and takeb equal to 0as an example. This way, we can better understand its meaning through a two-dimensional Cartesian coordinate system:

In this J (w, b) coordinate graph, which is a quadratic function, since we consider b in the equation to be 0,So we can assume that = ,So, such a partial derivative can be seen as the derivative in the unary case. At this point, it can be seen that when >0 and the corresponding w value is in the right half, the derivative is positive, that is, its slope is positive. This is equivalent to subtracting a positive number from w, and its w point will move to the left, which is the closest to its minimum value, which is the optimal solution. Similarly, when in the left half of the function, its w will move to the right, which is close to the minimum value, So the step size for each movement is .

This is a simple understanding of the gradient descent formula.


四、Formula operation precautions:

This is a simple understanding of the gradient descent formula

just like this:

The following is an incorrect order of operations that shouldbe avoided:

This is the understanding of the formula and algorithm implementation for gradient descent. As for the code implementation, we will continue to explain it in future articles.

Machine Learning ---- Cost function-CSDN博客

相关推荐
如夏倾城19 分钟前
入侵检测系统(IDS)原理、部署模式与 IPS/防火墙对比
大数据·网络·人工智能·网络安全·云原生·架构·kubernetes
Dawson Zhu25 分钟前
【模型量化⑤】Weight-Only INT4:AWQ与GPTQ两大算法深度解析
人工智能·语言模型·架构·aigc·agi
水如烟1 小时前
孤能子视角:通用智能 vs 分工协同智能群系统–––集中结晶 vs 分布式结晶
人工智能
罗西的思考1 小时前
【轻量化复现大模型 / KIMI K3】训练模块梳理 & Muon P2P 优化器
人工智能·深度学习·机器学习
lilihuigz1 小时前
权限配置:WordPress 7.2 Abilities API 权限、输入验证与 JSON 输出指南
人工智能
杨凯凡1 小时前
IT日报|2026-11-20:欧盟 AI 网络攻击、Valkey 9.2、GitHub 企业迁移
人工智能·github
鲁邦通物联网2 小时前
机器人梯控到站状态机设计:楼层匹配、事件时序与任务判断
人工智能·机器人·机器人梯控·agv梯控·非侵入式采集·机器人乘梯·机器人自主乘梯
静水深码2 小时前
文生图提示词的工程师写法:把玄学话术拆成结构化模板
人工智能·后端·代码规范
Seiran2 小时前
Claude Fable 5.1写技术方案,ZEEKEAI提效实录
人工智能
JonaHu2 小时前
GPT Image 2.6 多语言海报改版:保留商品细节,分离文案与图像验收
图像处理·人工智能·aigc