Which optimizer to use#

This is a very very very short and oversimplifying guide on selecting an optimization algorithm based on a minimum of information.

To select an optimizer, you need to answer two questions:

Is your criterion function differentiable?
Do you have a nonlinear least squares structure (i.e. do you sum some kind of squared residuals at the end of your criterion function)?

Define some inputs#

Again, we use versions of the sphere function to illustrate how to select these algorithms in practice

import numpy as np
import optimagic as om

@om.mark.least_squares
def sphere(params):
    return params


def sphere_gradient(params):
    return params * 2


start_params = np.arange(5)

Differentiable criterion function#

Use scipy_lbfsgsb as optimizer and provide the closed form derivative if you can. If you do not provide a derivative, optimagic will calculate it numerically. However, this is less precise and slower.

res = om.minimize(
    fun=sphere,
    params=start_params,
    algorithm="scipy_lbfgsb",
    jac=sphere_gradient,
)
res.n_fun_evals

Note that this solves a 5 dimensional problem with just 3 criterion evaluations. For higher dimensions, you will need more, but it scales very well to dozens and hundreds of parameters.

If you are worried about being stuck in a local optimum, use multistart optimization.

Not differentiable, only scalar output#

Use nag_pybobyqa. Note that for this you need to install the PyBOBYQA package if you do not already have it:

pip install Py-BOBYQA

Then you select the algorithm as follows:

res = om.minimize(
    fun=sphere,
    params=start_params,
    algorithm="nag_pybobyqa",
)
res.n_fun_evals

Not differentiable, least squares structure#

Use nag_dfols. To use nag_dfols, you need to install it via:

pip install DFO-LS

This optimizer will only work if your criterion function returns a dictionary that contains the entry root_contributions. This needs to be a numpy array or pytree that contains the residuals of the least squares problem.

res = om.minimize(
    fun=sphere,
    params=start_params,
    algorithm="nag_dfols",
)
res.n_fun_evals