Backfitting

TL;DR

• Iterative fitting method that alternates between variables, updating one set of parameters while keeping the others fixed.
• Commonly applied when models have many predictors or when predictors exhibit collinearity.
• Repeat the alternating fits until parameter estimates converge.

Definition

Backfitting is a technique used in regression analysis to estimate the parameters of a model by iteratively fitting each variable in the model while holding the remaining variables fixed.

Explanation

Backfitting proceeds by repeatedly fitting parts of a regression model one at a time while treating the other parts as fixed. Start by fitting a simple model using one predictor to obtain initial parameter estimates. Then hold those estimates fixed and fit the remaining predictors to update their parameters. Alternate which predictor(s) are fit and which are held fixed, iterating this process until the parameter estimates for all predictors converge (i.e., they do not change significantly from one iteration to the next). This approach is often used when the number of predictors is large or when predictors are highly correlated, which can make parameter estimation difficult using traditional simultaneous fitting methods.

Examples

Dataset

Price	Size	Bedrooms	Bathrooms
300000	1000	3	2
325000	1500	4	2.5
350000	2000	3	3
400000	2500	5	3.5

Step 1 — fit size alone:

$$\text{Price} = a + b * \text{Size}$$

System:

$$\begin{aligned} 300000 &= a + b * 1000 \\ 325000 &= a + b * 1500 \\ 350000 &= a + b * 2000 \\ 400000 &= a + b * 2500 \end{aligned}$$

Solved:

$$a = 17500 \\ b = 75$$

Initial estimate:

$$\text{Price} = 17500 + 75 * \text{Size}$$

Step 2 — hold (b=75) fixed, fit bedrooms and bathrooms:

$$\text{Price} = c + d * \text{Bedrooms} + e * \text{Bathrooms}$$

System:

$$\begin{aligned} 300000 &= c + d * 3 + e * 2 \\ 325000 &= c + d * 4 + e * 2.5 \\ 350000 &= c + d * 3 + e * 3 \\ 400000 &= c + d * 5 + e * 3.5 \end{aligned}$$

Solved:

$$c = -10000 \\ d = 12500 \\ e = 2500$$

Updated estimate:

$$\text{Price} = -10000 + 12500 * \text{Bedrooms} + 2500 * \text{Bathrooms}$$

Step 3 — hold updated estimates of (c) and (d) fixed, fit size:

$$\text{Price} = f + g * \text{Size}$$

System:

$$\begin{aligned} 300000 &= f + g * 1000 \\ 325000 &= f + g * 1500 \\ 350000 &= f + g * 2000 \\ 400000 &= f + g * 2500 \end{aligned}$$

Solved:

$$f = 23000 \\ g = 75$$

Updated estimate:

$$\text{Price} = 23000 + 75 * \text{Size}$$

Repeat the alternating fitting steps until parameter estimates for all variables have converged.

Use cases

• Used when the number of variables in the model is large.
• Used when there is collinearity among variables, which can hinder parameter estimation by traditional simultaneous methods.
• Useful for large or complex models to improve the accuracy of parameter estimates and predictions.

Notes or pitfalls

• The process is repeated until the parameter estimates converge (i.e., they do not change significantly between iterations).

Related terms

• Regression analysis
• Collinearity