Bias-Variance Tradeoff

TL;DR

The bias-variance tradeoff describes how model complexity affects prediction error.
Simplicity of a model is said to lead to low bias; flexibility of a model is said to lead to low variance.
Regularization and ensembling are common techniques to find the right balance and improve accuracy.

Definition

The bias-variance tradeoff is a fundamental concept in machine learning and statistics that refers to the balancing act between the complexity of a model and the amount of error in its predictions.

Explanation

The tradeoff rests on the idea that model simplicity and model flexibility affect different components of error: simplicity is associated with low bias, while flexibility is associated with low variance.
A model with low bias accurately represents the underlying relationship between input variables and the target variable. The source uses linear regression as an example of a model with low bias because it assumes a linear relationship between inputs and the target.
A model with low variance produces consistent predictions for similar inputs. The source uses decision trees as an example of a model with low variance because they split the data into homogeneous groups, producing similar outputs for similar inputs.
The tradeoff is illustrated with a house-price prediction scenario: a simple model such as linear regression is described as having low bias but possibly high variance because it may not capture complex relationships; a more complex model such as a decision tree is described as having low variance but may have high bias if it oversimplifies the relationship.
The goal of machine learning algorithms, per the source, is to find a model with the right balance between bias and variance to minimize overall prediction error.

Examples

Linear regression

Described in the source as a simple model that “has low bias because it assumes a linear relationship between the input variables and the target variable.”

Decision tree

Described in the source as a model that “has low variance because it splits the data into homogeneous groups, leading to consistent predictions for similar inputs.”

House price prediction scenario

Used to illustrate the tradeoff: a simple model (linear regression) is presented as low bias but may have high variance; a more complex model (decision tree) is presented as low variance but may have high bias.

Regularization (in linear regression and decision trees)

Regularization is described as a technique that penalizes complex models to reduce overfitting. In linear regression, this is applied by adding a penalty term to the cost function to encourage smaller coefficients. In decision trees, regularization can be applied by limiting tree depth to reduce the number of splits.

Ensembling (random forest)

Ensembling combines multiple models to improve overall accuracy. The random forest example trains multiple decision trees on different subsets of the data and combines their predictions (majority vote). According to the source, this reduces variance by averaging predictions of individual trees and can also reduce bias by using a diverse set of models.

Notes or pitfalls

According to the source, a model with low bias and low variance produces accurate predictions, while a model with high bias and high variance produces inaccurate predictions.
The source emphasizes that finding the right balance between bias and variance is essential for model accuracy.

Regularization
Ensembling
Linear regression
Decision tree
Random forest