Bayesian statistics is one of my favorite topics on this blog. But, the more I write the harder it is to find everything! I've decided to build this guide the organize these posts. This should make it easier for everyone to learn the basics. I plan to continually update this guide as I add new content. For updates follow @willkurt on twitter or subscribe to the email list.
Getting Started with Bayes' Theorem and Prior Probability
This guide only assumes you have a basic familiarity with probability. If you can work out the chance of flipping a coin and getting 3 heads in a row, then you should be all set. If you do need a refresher the Khan Academy has some great lectures on the topic. There is no need for a background in Statistics. The beauty of Bayesian Statistics is it can be built up from basic probability.
What to read to get you excited?
Nate Silver's The Signal and the Noise is a fun read and will get you very excited about Bayesian statistics. There are no strong math requirements for this book. If equations still make you a bit nervous this is a great place to build up your enthusiasm to dive in!
Recommended posts for getting started!
The best place to start learning is with Bayes' Theorem. In this popular post, we'll cover the basics using Lego bricks. This allows you to use visual learning to build a deep intuition for the theorem. We won't be ignoring the mathematical detail either. By the end of this post, you'll be able to derive this formula from scratch!
The next key topic is Bayesian Priors. Prior Probabilities allow us to model our beliefs about the world. This post focuses on C3POs classic underestimation of Han Solo's ability to navigate an asteroid field. Rather than simply dismissing C3PO, we'll see how using a Prior could help him.
Ever wonder if your friend is really allergic to gluten? We can use Bayes' Theorem and prior information to find out! We'll also discuss how being more certain than not is not all that certain.
Thinking like a Bayesian
With the basics down, it's time to start really thinking like a Bayesian! In these posts, we focus on how we can model our beliefs about the world, learn about the beliefs of others and compare hypotheses the Bayesian way!
Recommended Posts to get you thinking like a Bayesian!
Are you superstitious? Do you believe that a dinner novelty could have mystic powers? How much would it take to convince you? Using a classic episode of the Twilight Zone we'll learn how to answer these questions mathematically. We'll learn about Bayes' Factor and how we can use it to understand everyday reasoning.
We further explore Bayes' Factor by building the Voight-Kampff Test from Blade Runner! In this post, we explore how to make Bayes' Factor easier to interpret using decibel-based 'evidence'. Evidence allows us to very easily interpret the results of our test. No more confusing p-values!
Recommend Reading to really understand Bayesian thinking
Probability Theory: The Logic of Science is the book to read if you're fascinated with Bayesian thinking. Though there is plenty of math, this book is about philosophy. Jaynes is probably the most radical Bayesian there is. A few pages often provide me with food for thought for months. Also, if you need insults to sling at Frequentists this is the place to get them!
The catch is that this is a very challenging book. To help guide you through it Aubrey Clayton has put together a very nice set of lectures.
Bayesian analysis is where we put what we've learned to practical use. In my experience, there are two major benefits to Bayesian statistics over classical statistics:
The first is that you very easy model existing information. This can very often lead to better results since the model has more to work with. Additionally, you can account for inherent bias in your analysis. By modeling the subjective parts of analysis, you can better control them.
The second is that is no "mystery meat" in your tools. All of Bayesian statistics is built from the basics of probability. If you understand the posts so far, you have the tools to build nearly everything you'll need for practical analysis.
Recommend Posts to Learn Bayesian Analysis
Most probability books tell us how likely events are. We'll be told that a biased coin has a 0.6 probability of heads. In practice, we often don't know these probabilities. Determining whether or not the data comes from a biased coin is the real challenge. Parameter estimation deals with how we fill in these missing probabilities.
We've seen that priors can be used for subjective information. But quite often prior beliefs are based on hard data. In this post, we see how we can use existing data to develop priors. With data backed priors we can develop much better parameter estimates.
In all of the examples we've seen so far we've had very nice probability distributions to work with. Quite often we aren't so lucky. In this post, we learn one of the simpler techniques for working difficult prior probabilities and still getting useful results.
Can you derive a classical t-test from scratch? My guess is not without some difficulty. One of the benefits of Bayesian statistics is everything can be built from the basics. In this post, we combine everything covered so far to build an A/B test from scratch.
Recommend Books for Bayesian Analysis
If you've followed the post up to this point Doing Bayesian Data Analysis is an excellent next step. It will cover much of the material we have so far and then take you much further. The approach is mathematical, but never too challenging.
For practical Bayesian statistics, nobody gets me more excited than Andrew Gelman! This is not an easy book to work through but it is an absolute gem. The text is filled with wonderful, real world example that will alway renew your love of Bayesian Statistics.
Here's a great video that shows off Gelman's enthusiasm for Bayesian Analysis:
It's never the end! Look out for updates!
There's always more to explore in Bayesian Statistics! I plan to continually update this guide as I write relevant posts or come across other amazing books. Please subscribe the to email list or follow me on twitter for updates. Feel free to leave a comment on any topics you'd like to see covered!