# Bayes Factors#

Bayes factors are central quantities in both Bayesian hypothesis testing and Bayesian model comparison. A Bayes factor allows to assess which of two hypotheses or two competing models is in favour of the other.

## Bayes Factors in Bayesian Hypothesis Testing#

In the situation of classical hypothesis testing, there is one unknown parameter \(\theta\in\Theta\) lying in a given set \(\Theta\). The set \(\Theta\) can be decomposed as \(\Theta=\Theta_0\cup\Theta_1\) such that \(\Theta_0\cap\Theta_1=\emptyset\). The null hypothesis \(H_0\) states that \(H_0:~\theta\in\Theta_0\) and the alternative hypothesis is given by \(H_1:~\theta\in\Theta_1\). In a Bayesian setting, unknown parameters are assumed to follow a distribution. Therefore, probabilities are assigned to the hypotheses given by the prior likelihoods

such that \(\pi_0+\pi_1=1\). Taking into account a data set \(x\), the corresponding posterior likelihoods are defined by

where again \(p_0+p_1=1\). The definition of the posterior likelihoods is based on Bayes’ theorem. The Bayes factor \(B\) in favour of \(H_0\) against \(H_1\) is then defined by

Thus, a Bayes factor represents the ratio of the posterior odds \(p_0/p_1\) and the prior odds \(\pi_0/\pi_1\). It reflects the evidence in the data in favour of \(H_0\) as opposed to \(H_1\). By substituting the above definitions into Equation (11), we obtain by applying Bayes’ theorem

where \(\pi(x|\theta\in\Theta_i)=\mathbb{P}(x|\theta\in\Theta_i)\), for \(i\in\{0,1\}\), denotes the likelihood of alternative \(i\).

The informative value of \(B\) clearly depends on its magnitude. The following table provides established thresholds to interpret a concrete value of \(B\) [Kass and Raftery, 1995]:

\(B\) |
\(2\ln(B)\) |
Evidence against \(H_1\) |
---|---|---|

\([1,3]\) |
\([0,2]\) |
not worth more than a bare mention |

\((2,6]\) |
\((3,20]\) |
positive |

\((6,10]\) |
\((20,150]\) |
strong |

\(>10\) |
\(>150\) |
very strong |

The application of Bayes factors has several advantages compared with the approach of classical hypothesis testing. Instead of considering point estimates, parameter values are assumed to underlie a dristibution and a Bayes factor accounts for the whole distribution of the parameter space. Moreover, a Bayes factor is able to provide evidence in favour of \(H_0\) as well as against \(H_0\). However, by definition, Bayes factors only serve as a quantity for pairwise comparisons of alternatives. Besides, as for Bayesian methods in general, Bayes factors can be sensitive to the choice of prior distributions [Morey *et al.*, 2016].

The concrete form of \(\Theta_0\) and \(\Theta_1\) affects the corresponding Bayes factor and the evaluation of a hypothesis test. A detailed account on this topic can be found in [Lee, 2012].

## Bayes Factors for Model Selection#

Bayes factors cannot only be used to assess the plausibility of parameter values for a given model (or, e.g., for a distribution), but also for competing inherently different models [Hug, 2014]. Therefore, the focus is now on selecting one model among competing models \(M_1,\dots,M_m\) parameterized by parameters \(\theta_i\in\Theta_i\), for \(i=1,\dots,m\) that best explains the data. Analogous to Equation (12), the Bayes factor in favour of \(M_k\) over \(M_l\) is defined by

The quantity \(\pi(x|M_j)\), for \(j\in\{1,\dots,m\}\), is called the marginal likelihood of model \(M_j\) that can be written as

Equation (13) reflects that the marginal likelihood \(\pi(x|M_j)\) defines the likelihood of \(x\) for model \(M_j\) where the impact of the parameter \(\theta_j\) is marginalized (integrated) out. More formally, the marginal likelihood for model \(M_j\) is defined as the integral over the parameter space \(\Theta_j\) of the likelihood of \(M_j\) times the prior distribution of \(M_j\).

In order to determine Bayes factors, the integrals from Equation (13) need to be computed for the models of interest. Since the integral for model \(M_j\) is defined over the whole parameter space \(\Theta_j\), an analytic solution is, in practice, not feasible in many cases. However, various estimation techniques have been developed to address this issue. Different procedures are reviewed in [Friel and Wyse, 2012]. In the last decade, the thermodynamic integration approach has been established as the state-of-the-art method to approximate marginal likelihoods. Its application goes back to [Friel and Pettitt, 2008, Lartillot and Philippe, 2006].

In contrast to this article of a Bayesian model selection method, the article model selection provides an introduction to model selection tools for frequentist statistics.

## References#

- FP08
Nial Friel and Anthony N. Pettitt. Marginal likelihood estimation via power posteriors.

*Journal of the Royal Statistical Society. Series B (Statistical Methodology)*, 70:589–607, 2008.- FW12
Nial Friel and Jason Wyse. Estimating the evidence – a review.

*Statistica Neerlandica*, 66:288–308, 11 2012.- Hug14
Sabine C. Hug.

*From low-dimensional model selection to high-dimensional inference: tailoring Bayesian methods to biological dynamical systems*. TU München (PhD Thesis), 2014.- KR95
Robert E. Kass and Adrian E. Raftery. Bayes factors.

*Journal of the American Statistical Association*, 90:773–795, 1995.- LP06
Nicolas Lartillot and Hervé Philippe. Computing Bayes factors using thermodynamic integration.

*Systematic Biology*, 55:195–207, 4 2006.- Lee12
Peter M. Lee.

*Bayesian Statistics: An Introduction*. Wiley, 4 edition, 2012.- MRR16
Richard D. Morey, Jan Willem Romeijn, and Jeffrey N. Rouder. The philosophy of Bayes factors and the quantification of statistical evidence.

*Journal of Mathematical Psychology*, 72:6–18, 2016.

## Contributors#

Tamadur Albaraghtheh