Value iteration

English Version pytorch
1

https://d2l.ai/chapter_reinforcement-learning/value-iter.html

1 Like
2

Typo: the left hand side of the equation (17.2.9) should be pi*(s).

3

Great articles to clarify some basic ideas behind RL. I think it’s really a good starting point to begin RL learning.
But there are some small errors:

  1. The expectation over $r(s_0, a_0)$ is also needed for Eq. 17.2.2;
  2. In Eq. 17.2.9, it should be max rather than arg_max.
4

Seems like a typo in Equation 17.2.2

Shouldn’t Expectation over a_0, also include the first term?

5

do these code blocks still work? or do you need to follow the order of the book for the code to work? i haven’t been able to get the first chunk running for this and i’d really like to try out these exercises for reinforcement learning

6

There are two problems with this article here.
The first is equation 17.2.9, where ‘argmax’ should be corrected to ‘max’.
The second is that in the code implementation of the value_iteration function, as it says in its own comments, “Calculate \sum_{s‘} p(s’\mid s,a) [r + \gamma v_k(s’)]”, which should be fixed to “Calculate [r + \sum_{s’} p(s’\mid s,a) \gamma v_k(s’)]”, based on the previous equation 17.2.13.

Q[k,s,a] += pr * (reward + gamma * V[k - 1, nextstate])

should be fixed to:

Q[k,s,a] += (reward + pr * gamma * V[k - 1, nextstate])
7

Created a PR for fixing equation 17.2.2:

Current formula:

V^pi(s_0) = r(s_0, a_0) + gamma * E_{a_0 ~ pi(s_0)}[
    E_{s_1 ~ P(s_1 | s_0, a_0)}[V^pi(s_1)]
]

Proposed formula:

V^pi(s_0) = E_{a_0 ~ pi(s_0)}[
    r(s_0, a_0) + gamma * E_{s_1 ~ P(s_1 | s_0, a_0)}[V^pi(s_1)]
]

Reason: the immediate reward term r(s_0, a_0) depends on a_0. Since a_0 is sampled from the policy pi(s_0), this reward term should be inside the expectation over a_0. Otherwise the formula looks as if a_0 is already fixed in r(s_0, a_0), while the following expectation still averages over a_0.

8

upd: also formula 17.2.7 here on the left we have stohastic policy, but argmax seems to return a set of actions, those are different mathematical objects. If we asume we can describe stohastic policy as a series of actions taken from every state, then we are save. But i am afraid it’s easy to get puzzled there.

upd 2: the definition of 17.2.9 is the same as 17.2.7, which almost defines stohastic/determenistic policy as value function. be careful. Othe contibutors stated to pay atenntion to where to use argmax, or just ‘max’ which returns different objects besides those operations similarities