List of errata (as for September 2011) Last paragraph of section 4.2 (thanks to Javier Insa for spotting these errata): - "Definition 3.3 in Section 6" must be "Definition 6 in Section 3.3". - There's another reference to "Definition 3.3", which is again "Definition 6". - At the end of the paragraph it says: "That is the reason why we will typically specify a small n in the definition of n-actions reward-sensitive environment, which implies that there must be some reward `spent' after each n or more actions" should be "That is the reason why we will typically specify a small n in the definition of n-actions reward-sensitive environment, which implies that there must be some reward `spent' after each n (or less) actions" Example 4: - Page 1524 (last formula, second robot): there's a missing 1/4 for the last term before the last equality. - Page 1524. The same for the third robot. Footnote 8: ``and this choice determines a constant term which may very important for small environments'' is missing the word ``be'' and should be ``and this choice determines a constant term which may be very important for small environments''.