Machine Learning ---- Gradient Descent

目录

[一、The concept of gradient:](#一、The concept of gradient:)

[① In a univariate function:](#① In a univariate function:)

[②In multivariate functions:](#②In multivariate functions:)

[二、Introduction of gradient descent cases:](#二、Introduction of gradient descent cases:)

[三、Gradient descent formula and its simple understanding:](#三、Gradient descent formula and its simple understanding:)

[四、Formula operation precautions:](#四、Formula operation precautions:)


一、The concept of gradient:

In a univariate function

gradient is actually the differentiation of the function, representing the slope of the tangent of the function at a given point

In multivariate functions

a gradient is a vector with a direction, and the direction of the gradient indicates the direction in which the function rises the fastest at a given point

二、Introduction of gradient descent cases:

Do you remember the golf course inside the cat and mouse? It looks like this in the animation:

Let's take a look at these two pictures. You can easily see the distant hill, right? We can take it as the most typical example, and the golf course can also be abstracted into a coordinate map:

So in this coordinate, we will correspond the following (x, y) to (w, b) respectively. Then, when J (w, b) is at its maximum, which is the peak in the red area of the graph, we start the gradient descent process.

Firstly, we rotate one circle from the highest point to find the direction with the highest slope. At this point, we can take a small step down. The reason for choosing this direction is actually because it is the steepest direction. If we walk down the same step length, the height of descent will naturally be the highest, and we can also walk faster to the lowest point (local minimum point). At the same time, after each step, we look around and choose. Finally, we can determine this path:Finally reaching the local minimum point A, is this the only minimum point? Of course not:

It is also possible to reach point B, which is also a local minimum point. At this point, we have introduced the implementation process of gradient descent, and we will further understand its meaning through mathematical formulas.

三、Gradient descent formula and its simple understanding:

We first provide the formula for gradient descent:

In the formula, corresponds to what we call the learning rate, and the equal sign is the same as the assignment symbol in computer program code. J (w, b) can be found in the regression equation blog in the previous section. As for the determination of the learning rate, we will share it with you next time. Here, we will first understand the meaning of the formula:

Firstly, let's simplify the formula and takeb equal to 0as an example. This way, we can better understand its meaning through a two-dimensional Cartesian coordinate system:

In this J (w, b) coordinate graph, which is a quadratic function, since we consider b in the equation to be 0,So we can assume that = ,So, such a partial derivative can be seen as the derivative in the unary case. At this point, it can be seen that when >0 and the corresponding w value is in the right half, the derivative is positive, that is, its slope is positive. This is equivalent to subtracting a positive number from w, and its w point will move to the left, which is the closest to its minimum value, which is the optimal solution. Similarly, when in the left half of the function, its w will move to the right, which is close to the minimum value, So the step size for each movement is .

This is a simple understanding of the gradient descent formula.


四、Formula operation precautions:

This is a simple understanding of the gradient descent formula

just like this:

The following is an incorrect order of operations that shouldbe avoided:

This is the understanding of the formula and algorithm implementation for gradient descent. As for the code implementation, we will continue to explain it in future articles.

Machine Learning ---- Cost function-CSDN博客

相关推荐
美狐美颜sdk44 分钟前
跨平台直播美颜SDK集成实录:Android/iOS如何适配贴纸功能
android·人工智能·ios·架构·音视频·美颜sdk·第三方美颜sdk
DeepSeek-大模型系统教程1 小时前
推荐 7 个本周 yyds 的 GitHub 项目。
人工智能·ai·语言模型·大模型·github·ai大模型·大模型学习
郭庆汝1 小时前
pytorch、torchvision与python版本对应关系
人工智能·pytorch·python
IT古董1 小时前
【第二章:机器学习与神经网络概述】03.类算法理论与实践-(3)决策树分类器
神经网络·算法·机器学习
小雷FansUnion3 小时前
深入理解MCP架构:智能服务编排、上下文管理与动态路由实战
人工智能·架构·大模型·mcp
资讯分享周3 小时前
扣子空间PPT生产力升级:AI智能生成与多模态创作新时代
人工智能·powerpoint
叶子爱分享4 小时前
计算机视觉与图像处理的关系
图像处理·人工智能·计算机视觉
鱼摆摆拜拜4 小时前
第 3 章:神经网络如何学习
人工智能·神经网络·学习
一只鹿鹿鹿4 小时前
信息化项目验收,软件工程评审和检查表单
大数据·人工智能·后端·智慧城市·软件工程
张较瘦_5 小时前
[论文阅读] 人工智能 | 深度学习系统崩溃恢复新方案:DaiFu框架的原位修复技术
论文阅读·人工智能·深度学习