Generalized P Values

TL;DR

Generalized p-values extend the traditional p-value approach by allowing a tailored evaluation for each hypothesis rather than relying on a single fixed cutoff.
They are useful when the null hypothesis is not well-defined or when multiple hypotheses are tested simultaneously.
Decisions are still made by comparing p-values to a threshold (commonly 0.05); below the threshold one rejects the null, above it one does not.

Definition

A p-value is the probability of obtaining a result at least as extreme as the observed one, assuming the null hypothesis is true. It is commonly used in hypothesis testing to decide whether to reject the null hypothesis in favor of an alternative hypothesis.

Generalized p-values are a more flexible approach to hypothesis testing than the traditional p-value approach. Instead of setting a fixed threshold for the p-value, generalized p-values allow for a more tailored approach to hypothesis testing.

Explanation

The p-value quantifies how likely the observed data (or something more extreme) would be if the null hypothesis were true. In standard practice, a threshold (typically 0.05) is chosen: if the p-value is below that threshold, the result is considered statistically significant and the null hypothesis is rejected; if it is above, the null hypothesis is not rejected.

Generalized p-values adapt this process by permitting tailored evaluation criteria for different hypotheses or situations. This flexibility can be applied when the null hypothesis is ill-defined or when multiple hypotheses are evaluated concurrently, allowing separate assessments rather than a single uniform cutoff.

Examples

Drug effect on blood pressure

A researcher studies the effects of a new drug on blood pressure. The null hypothesis is that the drug has no effect on blood pressure, while the alternative hypothesis is that the drug does have an effect on blood pressure. The researcher could use a generalized p-value to determine the likelihood of observing the observed results, given that the null hypothesis is true. If the p-value is below a certain threshold, the researcher could reject the null hypothesis in favor of the alternative hypothesis. If the p-value is above the threshold, the researcher could conclude that the observed results are not statistically significant and the null hypothesis is not rejected.

Multiple hypotheses for a treatment

A researcher studies the effects of a new treatment on patients with a certain medical condition and has multiple hypotheses (for example: the treatment improves symptoms, reduces the risk of complications, and improves overall quality of life). The researcher could use generalized p-values to determine the likelihood of observing the observed results for each hypothesis, given that the null hypothesis is true. For example, the researcher could calculate a p-value for the hypothesis that the treatment improves symptoms and a separate p-value for the hypothesis that the treatment reduces complications. If the p-value for improving symptoms is below the threshold, the researcher could reject the null hypothesis that there is no improvement. If the p-value for reducing complications is above the threshold, the researcher could conclude that the observed result is not statistically significant and the null hypothesis is not rejected.

Use cases

Situations where the null hypothesis is not well-defined.
Scenarios involving multiple hypotheses being tested simultaneously.

Notes or pitfalls

A commonly used threshold for declaring statistical significance is 0.05.
If a p-value is below the chosen threshold, the null hypothesis is typically rejected; if it is above the threshold, it is typically not rejected.
Results described as “not statistically significant” correspond to p-values above the threshold.

p-value
null hypothesis
alternative hypothesis
multiple hypotheses (multiple hypothesis testing)