深度学习入门(9) - Reinforcement Learning 强化学习

Reinforcement Learning

an agent performs actions in environment, and receives rewards

goal: Learn how to take actions that maximize reward

Stochasticity: Rewards and state transitions may be random

Credit assignment : Reward r t r_t rt may not directly depend on action a t a_t at

Nondifferentiable: Can't backprop through the world

Nonstationary: What the agent experiences depends on how it acts

Markov Decision Process (MDP)

Mathematical formalization of the RL problem: A tuple ( S , A , R , P , γ ) (S,A,R,P,\gamma) (S,A,R,P,γ)

S S S: Set of possible states

A A A: Set of possible actions

R R R: Distribution of reward given (state, action) pair

P P P: Transition probability: distribution over next state given (state, action)

γ \gamma γ: Discount factor (trade-off between future and present rewards)

Markov Property: The current state completely characterizes the state of the world. Rewards and next states depend only on current state, not history.

Agent executes a policy π \pi π giving distribution of actions conditioned on states.

Goal : Find best policy that maximizes cumulative discounted reward ∑ t γ t r t \sum_t \gamma^tr_t ∑tγtrt

We will try to find the maximal expected sum of rewards to reduce the randomness.

Value function V π ( s ) V^{\pi}(s) Vπ(s): expected cumulative reward from following policy π \pi π from state s s s

Q function Q π ( s , a ) Q^{ \pi}(s,a) Qπ(s,a) : expected cumulative reward from following policy π \pi π from taking action a a a in state s s s

Bellman Equation

After taking action a in state s, we get reward r and move to a new state s'. After that, the max possible reward we can get is max ⁡ a ′ Q ∗ ( s ′ , a ′ ) \max_{a'} Q^*(s',a') maxa′Q∗(s′,a′)

Idea: find a function that satisfy Bellman equation then it must be optimal

start with a random Q, and use Bellman equation as an update rule.

But if the state is large/infinite, we can't iterate them.

Approximate Q(s, a) with a neural network, use Bellman equation as loss function.

-> Deep q learning

Policy Gradients

Train a network π θ ( a , s ) \pi_{\theta}(a,s) πθ(a,s) that takes state as input, gives distribution over which action to take

Objective function: Expected future rewards when following policy π θ \pi_{\theta} πθ

Use gradient ascent -> play some tricks to make it differentiable

Other approaches:

Actor-Critic

Model-Based

Imitation Learning

Inverse Reinforcement Learning

Adversarial Learning

...

Stochastic computation graphs

相关推荐
zxsz_com_cn7 小时前
设备健康管理大数据平台:工业智能化的核心数据引擎
运维·人工智能
算家计算7 小时前
破5亿用户!国产AI模型成功逆袭,成为AI普及浪潮主角
人工智能·开源·资讯
Jolie_Liang7 小时前
国内金融领域元宇宙金融特殊需求与技术挑战研究报告
人工智能·元宇宙
算家计算7 小时前
SAIL-VL2本地部署教程:2B/8B参数媲美大规模模型,为轻量级设备量身打造的多模态大脑
人工智能·开源·aigc
Costrict7 小时前
解锁新阵地!CoStrict 现已支持 JetBrains 系列 IDE
大数据·ide·人工智能·深度学习·自然语言处理·ai编程·visual studio
姚家湾7 小时前
MAC mini /绿联NAS 上安装本地AFFiNE
人工智能·affine
Python智慧行囊8 小时前
图像处理-opencv(二)-形态学
人工智能·计算机视觉
阿里云大数据AI技术8 小时前
云栖实录|阿里云 Milvus:AI 时代的专业级向量数据库
大数据·人工智能·搜索引擎
机器之心8 小时前
太强了!DeepSeek刚刚开源新模型,用视觉方式压缩一切
人工智能·openai
救救孩子把8 小时前
18-机器学习与大模型开发数学教程-第1章 1-10 本章总结与习题
人工智能·数学·机器学习