Count Bayesie's Recommended Books in Probability and Statistics
A question I often get is "How did you learn all this stuff?" and the honest answer is: reading. This page is a list of books I've read over the years. I've broken them down by category to help you find what you may need, this includes various mathematical prerequisites. Most of these contain Amazon Associate links to help fund me reading even more books!
Calculus Made Easy (Thompson, Gardner) - You can get pretty far using just discrete probability, but you'll inevitably need to use calculus. Whether you never properly learned calculus or just forgot everything this is an amazing book. This book is over 100 years old now, but the updates by Martin Gardner do a great job of keeping it modern.
Head First Statistics (Griffiths) - This book is heavily focused on Classical Statistics, but as hip (and useful) as Bayesian analysis is, it is still important to have a solid foundation in Classical methods. This is a very clear tutorial that will leave you with a solid understanding of "stats 101".
Data Analysis with Open Source Tools (Janert) - A very pragmatic book that is Bayesian at heart but not religious about it. If you need to get something done or solve a problem with real world data this is the book to get you started. Many of the statistical tools presented in this book are things I use professionally every week but are often not touched upon in other text books/courses.
What is a P-Value Anyway? (Vickers) - The first book on Classical statistics you should read once you get the basic "stats 101" understanding. It does a phenomenal job a building a deep intuition about the use and abuse of p-values. Even if you plan to only do Bayesian analysis it is still essential to understand and be able to interpret results from Classical Null Hypothesis Testing.
Doing Bayesian Data Analysis (Kruschke) - Does an excellent job of teaching Bayesian methods as essentially a drop-in replacement for Classical statistics. Very easy to understand introduction, and will answer any questions you might have about "How do I approach topic X in a Bayesian manner?"
Data Analysis: a Bayesian Tutorial (Sivia, Skilling) - A very concise but more mathematically demanding book than Doing Bayesian Data Analysis. If you prefer reading equation to long, text descriptions you'll find this more appealing. I recommend reading both this and Doing Bayesian Data Analysis side by side to get a thorough introduction.
Probability: The Logic of Science (Jaynes) - This is the book on Bayesian analysis. I really recommend getting a strong foundation in probability and statistics before diving in, only because you'll enjoy it that much more. Jaynes doesn't assume that Bayesian analysis is just an evolution of Classical statistics, but rather starts from first principles and builds it up as a form of logic. This is one of the most important books I have read, period. It is also in that category of books that are never truly "finished" because you could easily spend a life time on a single chapter.
Probability with Measure Theory
Understanding Analysis (Abbott) - A basic grounding in Real Analysis is a must if you want to move on to learning rigorous Probability with Measure Theory. This is a very gentle introduction to the subject that should give you a deep enough understanding of Real Analysis to work through the other books on this topic.
Principles of Mathematical Analysis (Rudin) - Rudin's book is extremely concise, but packed full of great insights into Real Analysis. If you don't have a background in Real Analysis this book is pretty intense, so I recommend using it in conjunction with another book if you aren't already familiar with the subject. It is definitely one of those books that you can continually return to and get more insights out of.
A First Look at Rigorous Probability Theory (Rosenthal) - Rigorous Probability theory is heavy in abstract mathematics so there's no way around it being difficult to get started with, but this book is probably the best place to start. A Measure Theoretic understanding of Probability provides many insights into some of the mysteries of the subject and opens the door for the study of more advanced topics like Stochastic Processes.
Probability & Measure Theory (Ash) - Robert B. Ash is brilliant at expository writing in mathematics (and has a really great essay on it). I'm not done with this book yet, but so far find it to be an excellent place to follow A First Look at Rigorous Probability Theory. Ash believes strongly in the values of working through exercises and so he includes many, with solutions, in this book. A great place for self-study.
Elements of Statistical Learning (Hastie, Tibshirani) - Written from an often Frequentist perspective this book is a stellar catalog of machine learning techniques. The list of techniques covered is very extensive, and the exposition on each method is very clear and easy to understand. The only downside is that, given the rising popularity of Bayesian techniques in machine learning, some of the intuitions will feel out-of-date if you read contemporary papers. Still, this remains one of my favorite books on the topic.
Machine Learning: A Probabilistic Perspective (Murphy)- This is the first book on machine learning I've found that is truly the Bayesian answer to Elements of Statistical Learning. There is no dearth of truly excellent machine learning books out there, but few cover such a broad range of topics. There's no doubt that this book is more of a reference than a tutorial, but it is my favorite reference when I need a thoroughly Bayesian understanding of a particular topic in machine learning.
Probabilistic Graphical Models (Koller, Friedman) - Graphical models are a method of solving problems in probability by representing your probability distribution as a graph. I'm really torn on this book: I use it probably every other week professionally, it is well written and gives a very thorough explanation of what it covers, but it is not a friendly book if you are new to the subject. This is only a problem because, to my knowledge, there are no good books for beginners in PGMs. Luckily Koller has an amazing series of lectures that follow the book on coursera.
50 Challenging Problems in Probability with Solutions (Mosteller) - Just get this book! It's super cheap, incredibly thin, and has probably enough challenging questions to keep you thinking for a life time. Mosteller is a great writer and these problems even include some that have no concrete answer. I'm sure you can find a free copy online, but I find the paper copy invaluable for all the notes you'll take in the margins.
Digital Dice: Computational Solutions to Practical Probability Problems (Nahin) - This book is similar to 50 Challenging Problems only focuses strictly on using Monte Carlo Simulations to solve everything! It's a really fun approach that shows how much easier it can often be to take the simulation approach instead of the analytic approach. I stumbled across this in a local books store and really wish it was more popular.
Choice and Chance (Whitworth) - I just got this book the other day, so I haven't read enough to give a thorough review but it is fascinating. It's mentioned in the preface to 50 Challenging Problems as essentially the first probability puzzle book. It was written in 1870 and doesn't even make the assumption that the reader is familiar with addition and multiplication!