Standard probability theory is based on the idea that although certain phenomena are random and unpredictable when taken in isolation, we can make predictions based on aggregate observations of individual phenomena. For example, suppose we have a biased coin. We flip it and it comes up heads. Based on this single observation, we cannot determine the bias of the coin or predict the outcome of the next toss. But by flipping the coin many, many times, we can expect the ratio of heads to tails to approach the actual bias of the coin, and we can make predictions about the likelihood of certain events, like the outcome of the next toss.

The Bayesian approach treats probability as a mechanism for modeling incomplete knowledge about some event or process or phenomenon. A model for some phenomenon is proposed and each observation of the phenomenon is used to refine the model. Again, suppose we have a biased coin that we flip many times. We imagine that a parameter, call it theta, determines the bias of the coin. As we observe the outcome of the flips, we refine our estimation of theta.

The fundamental difference between the two is this:

• Standard probability theory treats randomness as a physical property.
• Bayesian probability theory treats randomness as an information-theoretical quantity.