The Bayesian approach treats probability as a mechanism for modeling incomplete knowledge about some event or process or phenomenon. A model for some phenomenon is proposed and each observation of the phenomenon is used to refine the model. Again, suppose we have a biased coin that we flip many times. We imagine that a parameter, call it theta, determines the bias of the coin. As we observe the outcome of the flips, we refine our estimation of theta.
The fundamental difference between the two is this:
- Standard probability theory treats randomness as a physical property.
- Bayesian probability theory treats randomness as an information-theoretical quantity.