Jeffreys' prior

TL;DR

• A rule for choosing priors so that the assigned probabilities do not change when the parameter’s units change.
• For a normal mean, the prior density is proportional to the square root of the variance; for the precision (inverse variance), it is proportional to the reciprocal of the square root of the precision.
• Avoids arbitrary choices of parameter ranges or measurement units, reducing potential bias in estimates.

Definition

Jeffreys’ prior is a method of assigning probabilities to different possible values of a parameter in a statistical model that is based on the idea that the probabilities should be chosen in a way that is invariant to changes in the units of measurement of the parameter.

Explanation

A uniform prior over a parameter range can produce different implied probabilities when the parameter is expressed in different units (for example, inches versus centimeters). Jeffreys’ prior addresses this by assigning a prior density that remains invariant under reparameterization or changes of measurement units. In the examples given for the normal distribution, this leads to a prior density proportional to the square root of the variance for the mean, and to a prior density proportional to the reciprocal of the square root of the precision for the inverse-variance parameter. Using Jeffreys’ prior avoids making arbitrary choices about ranges or units and aims to provide a consistent, objective prior.

Examples

Estimating the mean of a normal distribution

Choosing a uniform prior on the range of possible values of the mean is not invariant to changes in units. For example, measuring the mean in inches might give a range from 0 to 10 inches, while measuring in centimeters would give a range from 0 to 254 centimeters. Jeffreys’ prior assigns a probability density that is proportional to the square root of the variance of the distribution:

$$\pi(\mu) \propto \sqrt{\sigma^2}$$

This choice ensures the probability of any given range of values is the same regardless of the units of measurement.

Estimating the precision (inverse variance) of a normal distribution

For the precision parameter (inverse variance), Jeffreys’ prior assigns a density proportional to the reciprocal of the square root of the precision:

$$\pi(\tau) \propto \tau^{-1/2}$$

Use cases

• Provides a way to assign probabilities in a manner that is consistent and objective, without needing arbitrary choices about ranges or units.
• Can help to reduce bias and improve the accuracy of statistical estimates.

Notes or pitfalls

• A uniform prior over a parameter range is not invariant under change of units; Jeffreys’ prior is designed specifically to avoid this problem by producing priors that are invariant to re-scaling or reparameterization.

Related terms

• Uniform prior
• Normal distribution
• Mean
• Variance
• Precision (inverse variance)
• Prior probability