A Guide to Bayesian Statistics

Bayesian statistics is one of my favorite topics on this blog. I love the topic so much I wrote a book on Bayesian Statistics to help anyone learn: Bayesian Statistics the Fun Way! The following post is the original guide to Bayesian Statistics that eventually became a the book! If you enjoy the resources below, please consider ordering a copy of “Bayesian Statistics the Fun Way!” It includes reworked versions of these posts and tons of original content.

Getting Started with Bayes' Theorem and Prior Probability

This guide only assumes you have a basic familiarity with probability. If you can work out the chance of flipping a coin and getting 3 heads in a row, then you should be all set. If you do need a refresher the Khan Academy has some great lectures on the topic. There is no need for a background in Statistics. The beauty of Bayesian Statistics is it can be built up from basic probability.

What to read to get you excited?

The Signal and The Noise

This is a great book to get you excited about Bayesian Stats!

This is a great book to get you excited about Bayesian Stats!

 

Nate Silver's The Signal and the Noise is a fun read and will get you very excited about Bayesian statistics. There are no strong math requirements for this book. If equations still make you a bit nervous this is a great place to build up your enthusiasm to dive in!

Recommended posts for getting started!

Bayes' Theorem with Lego

Bayes' Theorem is much easier to understand visually

Bayes' Theorem is much easier to understand visually

The best place to start learning is with Bayes' Theorem. In this popular post, we'll cover the basics using Lego bricks. This allows you to use visual learning to build a deep intuition for the theorem. We won't be ignoring the mathematical detail either. By the end of this post, you'll be able to derive this formula from scratch!


Han Solo and Bayesian Priors

"Never tell me the Odds!"

"Never tell me the Odds!"

The next key topic is Bayesian Priors. Prior Probabilities allow us to model our beliefs about the world. This post focuses on C3POs classic underestimation of Han Solo's ability to navigate an asteroid field. Rather than simply dismissing C3PO, we'll see how using a Prior could help him.

Use Bayes' Theorem to Investigate Food Allergies

Let's put Bayes' theorem to use!

Let's put Bayes' theorem to use!

Ever wonder if your friend is really allergic to gluten? We can use Bayes' Theorem and prior information to find out! We'll also discuss how being more certain than not is not all that certain.


Thinking like a Bayesian

With the basics down, it's time to start really thinking like a Bayesian! In these posts, we focus on how we can model our beliefs about the world, learn about the beliefs of others and compare hypotheses the Bayesian way!

Recommended Posts to get you thinking like a Bayesian!

Bayesian Reasoning in The Twilight Zone!

That glimmer in his eye is from Bayesian Statistics!

That glimmer in his eye is from Bayesian Statistics!

Are you superstitious? Do you believe that a dinner novelty could have mystic powers? How much would it take to convince you? Using a classic episode of the Twilight Zone we'll learn how to answer these questions mathematically. We'll learn about Bayes' Factor and how we can use it to understand everyday reasoning.

Using Bayes' Factor to Build a Voight-Kampff Test!

Just say no to p-values

Just say no to p-values

We further explore Bayes' Factor by building the Voight-Kampff Test from Blade Runner! In this post, we explore how to make Bayes' Factor easier to interpret using decibel-based 'evidence'. Evidence allows us to very easily interpret the results of our test. No more confusing p-values!

Recommend Reading to really understand Bayesian thinking

Probability Theory: The Logic of Science

You could easily spend a lifetime pondering this Jaynes' masterpiece.

You could easily spend a lifetime pondering this Jaynes' masterpiece.

 

Probability Theory: The Logic of Science is the book to read if you're fascinated with Bayesian thinking. Though there is plenty of math, this book is about philosophy. Jaynes is probably the most radical Bayesian there is. A few pages often provide me with food for thought for months. Also, if you need insults to sling at Frequentists this is the place to get them!


The catch is that this is a very challenging book. To help guide you through it Aubrey Clayton has put together a very nice set of lectures. 

Introductory lecture on Probability Theory: The Logic of Science by E.T. Jaynes. Aubrey Clayton, March 2015

 

Bayesian Analysis

Bayesian analysis is where we put what we've learned to practical use. In my experience, there are two major benefits to Bayesian statistics over classical statistics: 

The first is that you very easy model existing information. This can very often lead to better results since the model has more to work with. Additionally, you can account for inherent bias in your analysis. By modeling the subjective parts of analysis, you can better control them.

The second is that is no "mystery meat" in your tools. All of Bayesian statistics is built from the basics of probability. If you understand the posts so far, you have the tools to build nearly everything you'll need for practical analysis. 

Recommend Posts to Learn Bayesian Analysis

Parameter Estimation

Understanding CDF plots is very important

Understanding CDF plots is very important


Most probability books tell us how likely events are. We'll be told that a biased coin has a 0.6 probability of heads. In practice, we often don't know these probabilities. Determining whether or not the data comes from a biased coin is the real challenge. Parameter estimation deals with how we fill in these missing probabilities. 

Bayesian Priors for Parameter Estimation

Three priors from the same data

Three priors from the same data

 

We've seen that priors can be used for subjective information. But quite often prior beliefs are based on hard data. In this post, we see how we can use existing data to develop priors. With data backed priors we can develop much better parameter estimates.

Rejection Sampling and Tricky Priors

Rejection sampling looks pretty fun!

Rejection sampling looks pretty fun!

In all of the examples we've seen so far we've had very nice probability distributions to work with. Quite often we aren't so lucky. In this post, we learn one of the simpler techniques for working difficult prior probabilities and still getting useful results.


Bayesian A/B Testing: A Hypothesis Test that Makes Sense

The beauty of two competing posteriors!

The beauty of two competing posteriors!

Can you derive a classical t-test from scratch? My guess is not without some difficulty. One of the benefits of Bayesian statistics is everything can be built from the basics. In this post, we combine everything covered so far to build an A/B test from scratch.


Recommend Books for Bayesian Analysis

Doing Bayesian Data Analysis.

The next logic step after the posts here

The next logic step after the posts here

If you've followed the post up to this point Doing Bayesian Data Analysis is an excellent next step. It will cover much of the material we have so far and then take you much further. The approach is mathematical, but never too challenging. 

Bayesian Data Analysis.

This book is full of wonderful, practical examples

This book is full of wonderful, practical examples

For practical Bayesian statistics, nobody gets me more excited than Andrew Gelman! This is not an easy book to work through but it is an absolute gem. The text is filled with wonderful, real world example that will alway renew your love of Bayesian Statistics.

Here's a great video that shows off Gelman's enthusiasm for Bayesian Analysis:

Delivered by Andrew Gelman, Director, Applied Statistics Center, Columbia University, at the inaugural New York R Conference in New York City at Work-Bench on Friday, April 24th, and Saturday, April 25th.

 

It's never the end! Look out for updates!

There's always more to explore in Bayesian Statistics! I plan to continually update this guide as I write relevant posts or come across other amazing books. Please subscribe the to email list or follow me on twitter for updates. Feel free to leave a comment on any topics you'd like to see covered!