The Bayesian Network is an easy-to-understand graphical notation representing the conditional inter-dependence of variables within a system. This simple graphical formalism can leverage conditional probability distributions to describe relationships between variables in a system. How can Bayesian networks compute the probabilities of specific events given other facts? Humans also use this kind of reasoning to render decisions in uncertain environments.
Given data and parameter , a simple Bayesian analysis starts with a prior probability (prior) and likelihood to compute a posterior probability .
Often the prior on depends in turn on other parameters that are not mentioned in the likelihood. So, the prior must be replaced by a likelihood , and a prior on the newly introduced parameters is required, resulting in a posterior probability
This is the simplest example of a hierarchical Bayes model.
The process may be repeated; for example, the parameters may depend in turn on additional parameters , which require their own prior. Eventually the process must terminate, with priors that do not depend on unmentioned parameters.
Given the measured quantities each with normally distributed errors of known standard deviation ,
Suppose we are interested in estimating the . An approach would be to estimate the using a maximum likelihood approach; since the observations are independent, the likelihood factorizes and the maximum likelihood estimate is simply
However, if the quantities are related, so that for example the individual have themselves been drawn from an underlying distribution, then this relationship destroys the independence and suggests a more complex model, e.g.,
with improper priors , . When , this is an identified model (i.e. there exists a unique solution for the model's parameters), and the posterior distributions of the individual will tend to move, or shrink away from the maximum likelihood estimates towards their common mean. This shrinkage is a typical behavior in hierarchical Bayes models.
Some care is needed when choosing priors in a hierarchical model, particularly on scale variables at higher levels of the hierarchy such as the variable in the example. The usual priors such as the Jeffreys prior often do not work, because the posterior distribution will not be normalizable and estimates made by minimizing the expected loss will be inadmissible.