Ensemble Methods

TL;DR

Combine multiple models’ predictions to improve overall prediction accuracy.
Common approaches include bagging (training on different data subsets) and boosting (training models sequentially to correct previous errors).
Often used to reduce overfitting, improve generalization, and increase model performance.

Definition

Ensemble methods are a type of machine learning algorithm that combines the predictions of multiple individual models to make more accurate predictions than any of the individual models alone. This is achieved by training multiple models on the same dataset and then combining their predictions through some mathematical function, such as a weighted average or majority vote.

Explanation

Ensembles train several models and aggregate their outputs so the combined prediction is typically more reliable than any single model. Aggregation can use functions like a weighted average or majority vote. By varying how the individual models are trained and how their outputs are combined, ensembles can address issues such as variance and difficult-to-classify examples.

Two main approaches described are bagging and boosting. Bagging trains separate models on different random subsets of the training data and combines their predictions (often by majority vote). Boosting trains models sequentially, with each model focusing on correcting errors made by the previous ones; for example, AdaBoost increases the weights of training examples misclassified by earlier models so later models emphasize those cases.

Examples

Bagging (Bootstrapped Aggregation)

Bagging involves training multiple models on different subsets of the training data. Each model is trained on a different random subset of the data, and the final prediction is made by combining the predictions of all the individual models, often using a majority vote where the predicted class is the one chosen by the most individual models.

Boosting (including AdaBoost)

Boosting trains multiple models in a sequential manner. Each model is trained to correct the mistakes of the previous model to improve the ensemble’s overall performance. AdaBoost is a boosting algorithm that assigns higher weights to training examples misclassified by previous models so that the next model focuses more on those difficult cases.

Use cases

Reduce overfitting by decreasing variance through training on different subsets of data (bagging).
Improve generalization by giving more weight to difficult cases (boosting).
Increase overall model performance compared to individual models.

Bagging (bootstrapped aggregation)
Boosting
AdaBoost