The Levenberg-Marquardt method is a method of non-linear optimization. It minimizes the following functions: . Such a problem could be solved as a general non-linear optimization problem (for example, using a L-BFGS algorithm, but it is reasonable to use the information about the function F structure to solve a problem more effectively.

Levenberg-Marquardt method

Many methods use the gradient of the function to be minimized. The Levenberg-Marquardt method uses Jacobian instead of gradient, that is, a matrix A with elements . Jacobian provides more information about the modular surface than the gradient does.

Why is Jacobian more informative? There is a not quite rigorous, but intuitively obvious explanation for it: by knowing the function F gradient, we can generate its linear approximation, but if we know its Jacobian, we can generate the linear approximations of functions f_i  which let us obtain a quadratic approximation of function F. f_i  are usually nonlinear functions, but F is nonlinear to a greater extent. Therefore, the approximation of function F as a sum of squares of linear approximations is more precise than a simple linear approximation of function F as a general nonlinear function.

Jacobian usage in combination with some techniques which choose an optimal step length on the basis of the modular surface information allows to solve a problem by using less number of iterations than any other method. It is also considered that in multiextremal problems the Levenberg-Marquardt method tends to converge to better local minima than the other optimization methods. This property hasn't been sufficiently investigated in theory yet, but in practice this method had become a classical way to solve a problem with subjective functions in the form of sum of squares.

Description of the LevenbergMarquardtMinimize subroutine

This subroutine uses the user-defined FuncVecJac subroutine which calculates either the vector of the function values FVec or Jacobian FJac subject to the value of IFlag, in point X. It should be noted that the FuncVecJac subroutine doesn't need to waste time for memory allocation of the arrays FVec and FJac, because the memory is allocated in calling the subroutine before the first call of FuncVecJac. Setting a dimension of array G each time when calling a subroutine will excessively slow down an algorithm.

Report a bug

C#

C# 1.0 source.
levenbergmarquardt.csharp.zip - Levenberg-Marquardt algorithm for multivariate optimization

C++

C++ source.
levenbergmarquardt.cpp.zip - Levenberg-Marquardt algorithm for multivariate optimization
ablas.zip - optimized basic linear algebra subroutines with SSE2 support (for C++ sources only)

Delphi

Delphi source.
Can be compiled under FPC (in Delphi compatibility mode).
levenbergmarquardt.delphi.zip - Levenberg-Marquardt algorithm for multivariate optimization

Visual Basic 6

Visual Basic 6 source.
levenbergmarquardt.vb6.zip - Levenberg-Marquardt algorithm for multivariate optimization

Zonnon ^beta

Zonnon source.
Zonnon is an experimental language developed at ETH Zurich.
See www.zonnon.ethz.ch for more information.
levenbergmarquardt.zonnon.zip - Levenberg-Marquardt algorithm for multivariate optimization