Gauss Newton Method

TL;DR

• Iterative algorithm to solve nonlinear least squares by updating model parameters until convergence.
• Uses the Jacobian matrix to compute gradients and form parameter updates.
• Starts from an initial guess and repeatedly updates parameters using the Jacobian and gradient information.

Definition

The Gauss-Newton method is an iterative method used to solve nonlinear least squares problems. It was developed in the early 19th century and is named in the source after Carl Friedrich Gauss and Adrien-Marie Legendre, with Joseph Louis Lagrange later generalizing the method. The method refines parameter estimates by using the Jacobian matrix and the gradient of the objective function (the sum of squared residuals) until the estimates converge.

Explanation

• Least squares refers to minimizing the sum of the squares of the differences between observed values and model predictions, commonly used in regression analysis.
• The Gauss-Newton method applies to nonlinear models by iteratively improving parameter estimates:
  1. Make an initial guess for the model parameters.
  2. Compute predicted outputs for each data point using the current parameter estimates.
  3. Compute the Jacobian matrix, which contains partial derivatives of the model output with respect to the model parameters and describes sensitivity of outputs to parameter changes.
  4. Compute the gradient of the objective function (the sum of squared residuals) with respect to the parameters.
  5. Update the parameter estimates using the gradient and the Jacobian (the source states the update is the current estimates plus the inverse of the Jacobian matrix times the gradient).
  6. Repeat these steps until the estimates converge to a solution.

Examples

Quadratic fit example

Given a set of data points to fit with a quadratic function of the form:

$$y = a*x^2 + b*x + c$$

• Initial guess: (a=1), (b=1), (c=1).
• For a data point ((x=2, y=3)), the source computes the predicted value as:

$$12^2 + 12 + 1 = 7$$

• The source presents the Jacobian matrix for the quadratic function as a 3x3 matrix with partial derivatives of (y) with respect to (a), (b), and (c):

$$\begin{bmatrix} 2x^2 & x & 1 \end{bmatrix}$$ $$\begin{bmatrix} 4x^3 & 2x & 1 \end{bmatrix}$$ $$\begin{bmatrix} 6x^4 & 3x^2 & 1 \end{bmatrix}$$

• The gradient of the objective function is a (3 \times 1) vector with partial derivatives with respect to (a), (b), and (c).
• According to the source, the updated estimates are the current estimates plus the inverse of the Jacobian matrix times the gradient. The process is repeated until convergence.

Use cases

• Regression analysis (fitting model parameters by minimizing sum of squared residuals)

Related terms

• Jacobian matrix
• Least squares
• Regression analysis