Gaussian Process

TL;DR

• A Gaussian process defines a distribution over functions using a mean function and a covariance function, enabling probabilistic predictions.
• It explicitly models uncertainty and can capture complex, non-linear relationships via the covariance function.
• It typically requires large amounts of data and is not well suited for discontinuous or non-smooth phenomena.

Definition

A Gaussian process is a stochastic process characterized by a joint distribution of all possible outcomes, where each outcome is a normally distributed random variable. The process is completely specified by its mean function and covariance function, which describe the expected value and variability of the process at any given point in time.

Explanation

• The mean function gives the expected value of the process at each input (for example, a point in time).
• The covariance function (kernel) describes how values at different inputs co-vary, determining smoothness, variability, and how information is shared across inputs.
• Given the mean and covariance functions, a Gaussian process allows probabilistic predictions about future outcomes by using the joint normal distribution over function values.
• Strengths described in the source: handling uncertainty with incomplete data and modeling complex, non-linear relationships through appropriate choice of covariance function.
• Limitations described in the source: substantial data requirements for accurate estimation of mean and covariance functions, and poor fit for discontinuous or non-smooth phenomena.

Examples

Daily temperature of a city

• Mean function: the average temperature for a particular day of the year.
• Covariance function: describes expected variability of temperature between different days (for example, showing that temperatures may be more variable in the summer months than in the winter months).
• Use: predict the probability of a given temperature on a particular day given the mean and covariance functions.

Stock price of a company

• Mean function: the expected value of the stock price.
• Covariance function: describes the expected variability of the stock price over time (for example, showing that the stock price may be more variable in the short term than in the long term).
• Use: predict the probability of a given stock price at a given point in time given the mean and covariance functions.

Use cases

• Making probabilistic predictions about future outcomes (e.g., temperatures, stock prices) using mean and covariance functions.
• Modeling complex, non-linear relationships by selecting covariance functions that capture the relationship structure.
• Handling uncertainty and making predictions when data are incomplete, assuming an underlying normal distribution.

Notes or pitfalls

• Gaussian processes require a large amount of data to accurately estimate mean and covariance functions.
• They are not well suited for modeling discontinuous or non-smooth phenomena (for example, a stock price undergoing a sudden change in business strategy), because they assume continuity and smoothness in the underlying distribution.

Related terms

• Mean function
• Covariance function (kernel)
• Stochastic process
• Normally distributed / Normal distribution