I did the Reinforcement learning specialization on Coursera (https://www.coursera.org/specializations/reinforcement-learn...), which used this book. The specialization was great, but I would say that I had a lot of trouble following this book. I would find that there was a concept I didn't quite get, and then other things would build on top of that concept. I ended up feeling like I didn't quite grasp everything, even though I read most of the chapters more than once.
I do feel like there are a lot of things that aren't fully explained in this book, and maybe it is expected that the reader has some prior knowledge that I didn't have. For example, they never really explain the basic notation for backup diagrams. They just show one and mention that they are a thing. The mathematical notation they use also isn't really explained - they just start showing formulas of increasing complexity. It's possible to look these things up elsewhere, but they probably could have spent just a little bit more time explaining some of the basics and made it easier to follow.
This book seems to be very theoretical. Can someone recoment more practical books with code samples using some modern ML framework, probably something like _Hands-On Machine Learning_ by Geron Aurelien
To be honest I think this may be actually an advantage: it explains concepts which otherwise are just weird parameters in code. Since it is pretty lengthy I would actually recommend to read the chapters relevant to a specific method you are interested in (maybe going a backwards to build the right context).
"You’ll see that fast.ai’s way of teaching is very different to what you might be used to, if you did a technical degree at university. Nearly all technical subjects at university are taught “bottom up”: start with basic foundations, and gradually work up to complete useful solutions to real world problems. But we go “top down”: start with complete useful solutions to real world problems, and gradually work down to the basic foundations. Education experts recommend this approach for more effective learning."
I don't think this discusses reinforcement learning (not sure about the comment you responded to). To my recollection, the first course goes through basic architectures and concepts, and the second discusses autoenvoders.
"Grokking Deep Reinforcement Learning" by Miguel Morales and "Deep Reinforcement Learning in Action" by Alexander Zai and Brandon Brown both look promising, though the code might be outdated. Looks like they use the OpenAI Gym environment, which has since been forked and maintained as Gymnasium.
Do we have any real world applications where the policy satisfies strictly some constraints (think physics: conservation of mass etc)? There is research in the field but not sure if anything is in production.
Interesting to read the last section of the last chapter (17.6 Reinforcement Learning and the Future of Artificial Intelligence) given that the book is from 2020 and ChatGPT (in which RL plays a key role) was published in 2022.
Basic probability/stats and some basic fundamentals around dynamic programming/recursion would be very helpful.
The big problem I found with this field is that the core ideas are very subtly built on top of each other. Without a proper teacher or an environment to study, self-study is much much harder.
(Past chapter 5, it should be a breeze as the foundation would have been strongly set)
I did the Reinforcement learning specialization on Coursera (https://www.coursera.org/specializations/reinforcement-learn...), which used this book. The specialization was great, but I would say that I had a lot of trouble following this book. I would find that there was a concept I didn't quite get, and then other things would build on top of that concept. I ended up feeling like I didn't quite grasp everything, even though I read most of the chapters more than once.
I do feel like there are a lot of things that aren't fully explained in this book, and maybe it is expected that the reader has some prior knowledge that I didn't have. For example, they never really explain the basic notation for backup diagrams. They just show one and mention that they are a thing. The mathematical notation they use also isn't really explained - they just start showing formulas of increasing complexity. It's possible to look these things up elsewhere, but they probably could have spent just a little bit more time explaining some of the basics and made it easier to follow.
What made the specialization great? Did they explain things well where the book didn’t? Are the courses self-contained? What are the prerequisites?
Related. Others?
Reinforcement Learning: An Introduction (2018) [pdf] - https://news.ycombinator.com/item?id=19191746 - Feb 2019 (23 comments)
Reinforcement Learning: An Introduction, Second Edition - https://news.ycombinator.com/item?id=18547998 - Nov 2018 (6 comments)
New Draft of “Reinforcement Learning: An Introduction, Second Edition” - https://news.ycombinator.com/item?id=12568414 - Sept 2016 (33 comments)
Reinforcement Learning: An Introduction - https://news.ycombinator.com/item?id=1083662 - Jan 2010 (4 comments)
This book seems to be very theoretical. Can someone recoment more practical books with code samples using some modern ML framework, probably something like _Hands-On Machine Learning_ by Geron Aurelien
To be honest I think this may be actually an advantage: it explains concepts which otherwise are just weird parameters in code. Since it is pretty lengthy I would actually recommend to read the chapters relevant to a specific method you are interested in (maybe going a backwards to build the right context).
If you I'd combine it with e.g. https://spinningup.openai.com/en/latest/ or doing some toy projects with https://stable-baselines3.readthedocs.io/en/master/ it would probably render the most value.
https://course.fast.ai/
"You’ll see that fast.ai’s way of teaching is very different to what you might be used to, if you did a technical degree at university. Nearly all technical subjects at university are taught “bottom up”: start with basic foundations, and gradually work up to complete useful solutions to real world problems. But we go “top down”: start with complete useful solutions to real world problems, and gradually work down to the basic foundations. Education experts recommend this approach for more effective learning."
I don't think this discusses reinforcement learning (not sure about the comment you responded to). To my recollection, the first course goes through basic architectures and concepts, and the second discusses autoenvoders.
"Grokking Deep Reinforcement Learning" by Miguel Morales and "Deep Reinforcement Learning in Action" by Alexander Zai and Brandon Brown both look promising, though the code might be outdated. Looks like they use the OpenAI Gym environment, which has since been forked and maintained as Gymnasium.
There's lecture notes on this book available from David Silver at UCL: https://www.davidsilver.uk/teaching/
Do we have any real world applications where the policy satisfies strictly some constraints (think physics: conservation of mass etc)? There is research in the field but not sure if anything is in production.
This is what I'd like to know as well.
Interesting to read the last section of the last chapter (17.6 Reinforcement Learning and the Future of Artificial Intelligence) given that the book is from 2020 and ChatGPT (in which RL plays a key role) was published in 2022.
i pretty much have read this book. pretty boring tbh but still good. i would recommend doing hands on implementations with the cartpole environment
What are the math prerequisites for this book? What other reinforcement learning books or courses are recommended for beginners?
Basic probability/stats and some basic fundamentals around dynamic programming/recursion would be very helpful.
The big problem I found with this field is that the core ideas are very subtly built on top of each other. Without a proper teacher or an environment to study, self-study is much much harder.
(Past chapter 5, it should be a breeze as the foundation would have been strongly set)
This is THE book. It is suitable for beginners with basic math background equivalent to CS undergrad I’d say