系列文章目录
文章目录
- 系列文章目录
- 前言
- [Definition of optimal policy](#Definition of optimal policy)
- [Bellman optimality equation](#Bellman optimality equation)
-
- Introduction
- [Maximization on the right-hand side](#Maximization on the right-hand side)
- [Contraction mapping theorem](#Contraction mapping theorem)
- Solution
- Optimality
- [Analyzing optimal policies](#Analyzing optimal policies)
- 总结
前言
本系列文章主要用于记录 B站 赵世钰老师的【强化学习的数学原理】的学习笔记，关于赵老师课程的具体内容，可以移步：
B站视频：【【强化学习的数学原理】课程：从零开始到透彻理解（完结）】
GitHub 课程资料：Book-Mathematical-Foundation-of-Reinforcement-Learning
Definition of optimal policy
The state value could be used to evaluate if a policy is good or not: if
v π 1 ( s ) ≥ v π 2 ( s ) , ∀ s ∈ S v_{\pi_1}(s) \geq v_{\pi_2}(s), \quad \forall s \in \mathcal{S} vπ1(s)≥vπ2(s),∀s∈S
then π 1 \pi_1 π1 is "better" than π 2 \pi_2 π2.
A policy π ∗ \pi^* π∗ is optimal if
v π ∗ ( s ) ≥ v π ( s ) , ∀ s ∈ S , ∀ π . v_{\pi^*}(s) \geq v_\pi(s), \quad \forall s \in \mathcal{S}, \; \forall \pi. vπ∗(s)≥vπ(s),∀s∈S,∀π.
Bellman optimality equation
Introduction
Bellman optimality equation (elementwise form):
v ( s ) = max π ∑ a π ( a ∣ s ) ( ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v ( s ′ ) ) , ∀ s ∈ S v(s) = \max_{\pi} \sum_a \pi(a \mid s) \left( \sum_r p(r \mid s,a) r + \gamma \sum_{s'} p(s' \mid s,a) v(s') \right), \quad \forall s \in \mathcal{S} v(s)=maxπ∑aπ(a∣s)(∑rp(r∣s,a)r+γ∑s′p(s′∣s,a)v(s′)),∀s∈S
= max π ∑ a π ( a ∣ s ) q ( s , a ) , s ∈ S = \max_{\pi} \sum_a \pi(a \mid s) q(s,a), \quad s \in \mathcal{S} =maxπ∑aπ(a∣s)q(s,a),s∈S
Remarks:
- p ( r ∣ s , a ) , p ( s ′ ∣ s , a ) p(r \mid s,a), p(s' \mid s,a) p(r∣s,a),p(s′∣s,a) are known.
- v ( s ) , v ( s ′ ) v(s), v(s') v(s),v(s′) are unknown and to be calculated.
- π ( s ) \pi(s) π(s) is known!
结合 Bellman 方程依赖于已知策略，解释为什么在 Bellman 最优性方程 里要取 max π \max_\pi maxπ，以及它和最优策略 π ∗ \pi^* π∗ 的关系
回顾：Bellman 方程（依赖已知策略）
对于一个固定的策略 π \pi π，状态价值函数定义为：
v π ( s ) = ∑ a π ( a ∣ s ) ∑ r p ( r ∣ s , a ) r + γ ∑ s ' p ( s ' ∣ s , a ) v π ( s ' ) v_\pi(s) = \sum_a \pi(a \mid s) \Bigg \\sum_r p(r \\mid s,a) r + \\gamma \\sum_{s'} p(s' \\mid s,a) v_\\pi(s') \\Bigg vπ(s)=∑aπ(a∣s)∑rp(r∣s,a)r+γ∑s'p(s'∣s,a)vπ(s')
- 这里 π ( a ∣ s ) \pi(a|s) π(a∣s) 是 已知的动作分布 ，所以 Bellman 方程在这种情况下是 策略评估 (policy evaluation) 工具。
- 策略 = 对每个状态 s s s**，给所有可能动作** a a a 分配概率，即一个策略对应了一组"所有状态的动作概率分布"。
- 不同策略对应的就是 不同的动作概率分布集合 。
策略 π 1 \pi_1 π1**：**
在状态 s s s，可能给动作 a 1 a_1 a1 的概率高一些，给动作 a 2 a_2 a2 的概率低一些。
策略 π 2 \pi_2 π2**：**
在相同状态 s s s，可能恰好相反，给 a 1 a_1 a1 的概率低，给 a 2 a_2 a2 的概率高。
从策略依赖到最优性
如果我们不想只评估一个具体策略，而是想找到 最优策略 ，那就需要考虑：对于每个状态 s s s，什么样的动作选择（或策略）能最大化长期回报？
于是，状态价值的定义变成：
v ∗ ( s ) = max π v π ( s ) , ∀ s ∈ S v^*(s) = \max_\pi v_\pi(s), \quad \forall s \in \mathcal{S} v∗(s)=maxπvπ(s),∀s∈S
这里的 max π \max_\pi maxπ 表示：在所有可能的策略中，找到一个能使价值函数最大的策略。
Bellman 最优性方程
把 max π \max_\pi maxπ 引入 Bellman 方程，得到：
v ( s ) = max a ∑ r p ( r ∣ s , a ) r + γ ∑ s ' p ( s ' ∣ s , a ) v ( s ' ) v^(s) = \max_a \Bigg \\sum_r p(r \\mid s,a) r + \\gamma \\sum_{s'} p(s' \\mid s,a) v\^(s') \\Bigg v(s)=maxa∑rp(r∣s,a)r+γ∑s'p(s'∣s,a)v(s')
关键点：
- 在普通 Bellman 方程里： π ( a ∣ s ) \pi(a|s) π(a∣s) 是已知的分布。
- 在 Bellman 最优性方程里：我们不固定策略，而是 直接在动作层面取最大化，等价于"选择最优动作"。
- 因此，最优价值函数 v ∗ ( s ) v^*(s) v∗(s) 不再依赖于某个具体的 π \pi π，而是内含了 策略优化的过程。
和最优策略 π ∗ \pi^* π∗ 的关系
定义：
π ( s ) = arg max a ∑ r p ( r ∣ s , a ) r + γ ∑ s ' p ( s ' ∣ s , a ) v ( s ' ) \pi^(s) = \arg\max_a \Bigg \\sum_r p(r \\mid s,a) r + \\gamma \\sum_{s'} p(s' \\mid s,a) v\^(s') \\Bigg π(s)=argmaxa∑rp(r∣s,a)r+γ∑s'p(s'∣s,a)v(s')
即最优策略就是在每个状态下选择能使未来回报最大的动作。
换句话说：
- 普通 Bellman 方程 = 已知策略的价值评估。
- Bellman 最优性方程 = 在所有策略中选择最优的价值函数，从而定义了最优策略。
Bellman optimality equation (matrix-vector form):
v = max π ( r π + γ P π v ) v = \max_\pi (r_\pi + \gamma P_\pi v) v=maxπ(rπ+γPπv)
where the elements corresponding to s s s or s ′ s' s′ are
r π s ≜ ∑ a π ( a ∣ s ) ∑ r p ( r ∣ s , a ) r , r_\\pi_s \triangleq \sum_a \pi(a \mid s) \sum_r p(r \mid s,a) r, rπs≜∑aπ(a∣s)∑rp(r∣s,a)r,
P π s , s ′ = p ( s ′ ∣ s ) ≜ ∑ a π ( a ∣ s ) ∑ s ′ p ( s ′ ∣ s , a ) P_\\pi{s,s'} = p(s' \mid s) \triangleq \sum_a \pi(a \mid s) \sum{s'} p(s' \mid s,a) Pπs,s′=p(s′∣s)≜∑aπ(a∣s)∑s′p(s′∣s,a)
Here max π \max_\pi maxπ is performed elementwise.
Maximization on the right-hand side
Solve the Bellman optimality equation
必须先考虑右边的式子，即先有某个最优策略 π \pi π，然后才有最优的状态价值 v ( s ) v(s) v(s)
-
elementwise form
v ( s ) = max π ∑ a π ( a ∣ s ) ( ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v ( s ′ ) ) , ∀ s ∈ S v(s) = \max_{\pi} \sum_a \pi(a \mid s) \left( \sum_r p(r \mid s,a) r + \gamma \sum_{s'} p(s' \mid s,a) v(s') \right), \quad \forall s \in \mathcal{S} v(s)=maxπ∑aπ(a∣s)(∑rp(r∣s,a)r+γ∑s′p(s′∣s,a)v(s′)),∀s∈S
-
Fix v ′ ( s ) v'(s) v′(s) first and solve π \pi π:
v ( s ) = max π ∑ a π ( a ∣ s ) q ( s , a ) v(s) = \max_\pi \sum_a \pi(a \mid s) q(s,a) v(s)=maxπ∑aπ(a∣s)q(s,a)
-
Inspired by the above example, considering that ∑ a π ( a ∣ s ) = 1 \sum_a \pi(a \mid s) = 1 ∑aπ(a∣s)=1, we have
max π ∑ a π ( a ∣ s ) q ( s , a ) = max a ∈ A ( s ) q ( s , a ) , \max_\pi \sum_a \pi(a \mid s) q(s,a) = \max_{a \in \mathcal{A}(s)} q(s,a), maxπ∑aπ(a∣s)q(s,a)=maxa∈A(s)q(s,a),
-
where the optimality is achieved when
π ( a ∣ s ) = { 1 , a = a ∗ 0 , a ≠ a ∗ \pi(a \mid s) = \begin{cases} 1, & a = a^* \\ 0, & a \neq a^* \end{cases} π(a∣s)={1,0,a=a∗a=a∗
-
where a ∗ = arg max a q ( s , a ) a^* = \arg\max_a q(s,a) a∗=argmaxaq(s,a).
-
-
matrix-vector form
v = max π ( r π + γ P π v ) v = \max_\pi (r_\pi + \gamma P_\pi v) v=maxπ(rπ+γPπv)
-
Let
f ( v ) : = max π ( r π + γ P π v ) . f(v) := \max_\pi (r_\pi + \gamma P_\pi v). f(v):=maxπ(rπ+γPπv).
-
Then, the Bellman optimality equation becomes
v = f ( v ) v = f(v) v=f(v)
-
where
f ( v ) s = max π ∑ a π ( a ∣ s ) q ( s , a ) , s ∈ S . f(v)s = \max\pi \sum_a \pi(a \mid s) q(s,a), \quad s \in \mathcal{S}. f(v)s=maxπ∑aπ(a∣s)q(s,a),s∈S.
-
Contraction mapping theorem
Fixed point:
x ∈ X x \in X x∈X is a fixed point of f : X → X f : X \to X f:X→X if f ( x ) = x f(x) = x f(x)=x
Contraction mapping (or contractive function):
f f f is a contraction mapping if ∥ f ( x 1 ) − f ( x 2 ) ∥ ≤ γ ∥ x 1 − x 2 ∥ \| f(x_1) - f(x_2) \| \leq \gamma \| x_1 - x_2 \| ∥f(x1)−f(x2)∥≤γ∥x1−x2∥, where γ ∈ ( 0 , 1 ) \gamma \in (0,1) γ∈(0,1).
- γ \gamma γ must be strictly less than 1 1 1 so that many limits such as γ k → 0 \gamma^k \to 0 γk→0 as k → ∞ k \to \infty k→∞ hold.
- Here ∥ ⋅ ∥ \| \cdot \| ∥⋅∥ can be any vector norm.
Contraction mapping theorem
For any equation that has the form of x = f ( x ) x = f(x) x=f(x), if f f f is a contraction mapping, then
- Existence: There exists a fixed point x ∗ x^* x∗ satisfying f ( x ∗ ) = x ∗ f(x^*) = x^* f(x∗)=x∗.
- Uniqueness: The fixed point x ∗ x^* x∗ is unique.
- Algorithm: Consider a sequence { x k } \{x_k\} {xk} where x k + 1 = f ( x k ) x_{k+1} = f(x_k) xk+1=f(xk), then x k → x ∗ x_k \to x^* xk→x∗ as k → ∞ k \to \infty k→∞. Moreover, the convergence rate is exponentially fast.
Solution
Let's come back to the Bellman optimality equation:
v = f ( v ) = max π ( r π + γ P π v ) v = f(v) = \max_\pi (r_\pi + \gamma P_\pi v) v=f(v)=maxπ(rπ+γPπv)
Contraction Property:
-
f ( v ) f(v) f(v) is a contraction mapping satisfying
∥ f ( v 1 ) − f ( v 2 ) ∥ ≤ γ ∥ v 1 − v 2 ∥ \| f(v_1) - f(v_2) \| \leq \gamma \| v_1 - v_2 \| ∥f(v1)−f(v2)∥≤γ∥v1−v2∥
-
where γ \gamma γ is the discount rate!
Existence, Uniqueness, and Algorithm:
-
For the BOE v = f ( v ) = max π ( r π + γ P π v ) v = f(v) = \max_\pi (r_\pi + \gamma P_\pi v) v=f(v)=maxπ(rπ+γPπv), there always exists a solution v ∗ v^* v∗ and the solution is unique. The solution could be solved iteratively by
v k + 1 = f ( v k ) = max π ( r π + γ P π v k ) v_{k+1} = f(v_k) = \max_\pi (r_\pi + \gamma P_\pi v_k) vk+1=f(vk)=maxπ(rπ+γPπvk)
-
This sequence { v k } \{v_k\} {vk} converges to v ∗ v^* v∗ exponentially fast given any initial guess v 0 v_0 v0. The convergence rate is determined by γ \gamma γ.
Optimality
Suppose v ∗ v^* v∗ is the solution to the Bellman optimality equation. It satisfies
v ∗ = max π ( r π + γ P π v ∗ ) v^* = \max_\pi (r_\pi + \gamma P_\pi v^*) v∗=maxπ(rπ+γPπv∗)
Suppose
π ∗ = arg max π ( r π + γ P π v ∗ ) \pi^* = \arg\max_\pi (r_\pi + \gamma P_\pi v^*) π∗=argmaxπ(rπ+γPπv∗)
Then
v ∗ = r π ∗ + γ P π ∗ v ∗ v^* = r_{\pi^*} + \gamma P_{\pi^*} v^* v∗=rπ∗+γPπ∗v∗
Therefore, π ∗ \pi^* π∗ is a policy and v ∗ = v π ∗ v^* = v_{\pi^*} v∗=vπ∗ is the corresponding state value.
Policy Optimality
-
Suppose that v ∗ v^* v∗ is the unique solution to v = max π ( r π + γ P π v ) , v = \max_\pi (r_\pi + \gamma P_\pi v), v=maxπ(rπ+γPπv),
and v π v_\pi vπ is the state value function satisfying v π = r π + γ P π v π v_\pi = r_\pi + \gamma P_\pi v_\pi vπ=rπ+γPπvπ for any given policy π \pi π, then
v ≥ v π , ∀ π v^ \geq v_\pi, \quad \forall \pi v≥vπ,∀π
Greedy Optimal Policy
-
For any s ∈ S s \in \mathcal{S} s∈S, the deterministic greedy policy
π ( a ∣ s ) = { 1 , a = a ( s ) 0 , a ≠ a ( s ) \pi^(a \mid s) = \begin{cases} 1, & a = a^(s) \\ 0, & a \neq a^(s) \end{cases} π(a∣s)={1,0,a=a(s)a=a(s)
-
is an optimal policy solving the BOE. Here,
∗ a ( s ) = arg max a q ( s , a ) , ∗ *a^(s) = \arg\max_a q^(s,a),* ∗a(s)=argmaxaq(s,a),∗
-
where q ( s , a ) : = ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v ∗ ( s ′ ) q^(s,a) := \sum_r p(r \mid s,a) r + \gamma \sum_{s'} p(s' \mid s,a) v^*(s') q(s,a):=∑rp(r∣s,a)r+γ∑s′p(s′∣s,a)v∗(s′)
-
Proof
π ( s ) = arg max π ∑ a π ( a ∣ s ) ( ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v ( s ′ ) ) \pi^(s) = \arg\max_\pi \sum_a \pi(a \mid s) \left( \sum_r p(r \mid s,a) r + \gamma \sum_{s'} p(s' \mid s,a) v^(s') \right) π(s)=argmaxπ∑aπ(a∣s)(∑rp(r∣s,a)r+γ∑s′p(s′∣s,a)v(s′))
对公式的概括
v ∗ v^* v∗ 的意义
v ∗ v^* v∗ 是一个"理想值表"，它告诉你：从每个状态出发，如果以后一直做出最优选择，能拿到的总回报是多少。
它满足一个自洽的关系式（Bellman 最优性方程）：
v ∗ ( s ) = max a 即时奖励 + γ × 未来价值 v^*(s) = \max_a \Big\\text{即时奖励} + \\gamma \\times \\text{未来价值}\\Big v∗(s)=maxa即时奖励+γ×未来价值
π ∗ \pi^* π∗ 的意义
- π ∗ \pi^* π∗ 是"最优策略"，它规定了：在每个状态下应该采取哪个动作，才能保证总回报不比任何其他策略差。
- 从公式上看， π ∗ \pi^* π∗ 就是选择能让 v ∗ v^* v∗ 达到最大的那个动作（也就是"贪心选择"）。
Policy Optimality 定理
- 对于任意其他策略 π \pi π，它的价值函数 v π v_\pi vπ 都不会超过 v ∗ v^* v∗。
- v ∗ v^* v∗ 是所有策略里能实现的最高水平，它一定支配所有其他策略的价值表。
Greedy Optimal Policy 定理
- 只要你已经有了 v ∗ v^* v∗，那么直接在每个状态里"选那个让回报最大化的动作"就能得到 π ∗ \pi^* π∗。
- 最优策略其实就是"贪心地"选动作，但前提是这个贪心是基于正确的 v ∗ v^* v∗。
更进一步的解释
- 为什么要先有 v ∗ v^* v∗ 才能得到 π ∗ \pi^* π∗？
- 因为 π ∗ \pi^* π∗ 的定义依赖于"未来回报"，而未来回报就是由 v ∗ v^* v∗ 描述的。
- 一旦知道了 v ∗ v^* v∗，最优策略就能"顺理成章"地通过贪心法则推出来。
- 为什么 v ∗ v^* v∗ 比任何 v π v_\pi vπ 都大？
- v π v_\pi vπ 是"固定策略下"的表现。
- v ∗ v^* v∗ 是在每一步都挑选最优动作的表现。
- 显然，如果你随时都能选最好的动作，你的表现不可能比其他任何固定策略差。
- 为什么贪心策略一定最优？
- 因为 Bellman 方程已经保证了：在 v ∗ v^* v∗ 下，每个状态的最优价值都等于"选择最优动作"得到的回报。
- 所以只要你在每个状态都执行这个"最优动作"，整个过程的价值函数自然等于 v ∗ v^* v∗。
- 也就是说：贪心 + 正确的价值表 = 全局最优策略。
Analyzing optimal policies
What factors determine the optimal policy?
-
It can be clearly seen from the BOE
v ( s ) = max π ∑ a π ( a ∣ s ) ( ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v ( s ′ ) ) v(s) = \max_\pi \sum_a \pi(a \mid s) \left( \sum_r p(r \mid s,a) r \;+\; \gamma \sum_{s'} p(s' \mid s,a) v(s') \right) v(s)=maxπ∑aπ(a∣s)(∑rp(r∣s,a)r+γ∑s′p(s′∣s,a)v(s′))
-
that there are three factors:
- Reward design: r r r
- System model: p ( s ′ ∣ s , a ) , p ( r ∣ s , a ) p(s' \mid s,a), \; p(r \mid s,a) p(s′∣s,a),p(r∣s,a)
- Discount rate: γ \gamma γ
-
In this equation, v ( s ) , v ( s ′ ) , π ( a ∣ s ) v(s), v(s'), \pi(a \mid s) v(s),v(s′),π(a∣s) are unknowns to be calculated.
Optimal Policy Invariance
-
Consider a Markov decision process with v ∗ ∈ R ∣ S ∣ v^* \in \mathbb{R}^{|\mathcal{S}|} v∗∈R∣S∣ as the optimal state value satisfying
v ∗ = max π ( r π + γ P π v ∗ ) . v^* = \max_\pi (r_\pi + \gamma P_\pi v^*). v∗=maxπ(rπ+γPπv∗).
-
If every reward r r r is changed by an affine transformation to a r + b ar + b ar+b, where a , b ∈ R a, b \in \mathbb{R} a,b∈R and a ≠ 0 a \neq 0 a=0, then the corresponding optimal state value v ′ v' v′ is also an affine transformation of v ∗ v^* v∗:
v ′ = a v ∗ + b 1 − γ 1 , v' = a v^* + \frac{b}{1-\gamma} \mathbf{1}, v′=av∗+1−γb1,
- where γ ∈ ( 0 , 1 ) \gamma \in (0,1) γ∈(0,1) is the discount rate and 1 = 1 , ... , 1 T \mathbf{1} = 1, \\ldots, 1^T 1=1,...,1T.
-
Consequently, the optimal policies are invariant to the affine transformation of the reward signals.
总结
Bellman 最优性方程刻画了在所有策略中选择最优策略的价值函数，它保证存在唯一的最优状态价值 v ∗ v^* v∗，并且通过对每个状态下的动作取最大化（贪心原则）即可导出最优策略 π ∗ \pi^* π∗，同时最优策略的性质只依赖于奖励设计、环境转移模型和折扣因子，而对奖励的仿射变换保持不变。